Re: [tsvwg] I-D Action: draft-ietf-tsvwg-l4sops-00.txt

[Greg White <[e-mail]>]

Sebastian,

On the L4S Interim call yesterday, you mentioned that the data on ECN usage in this Roddav paper for ports 80 and 443 were more realistic.  I'm not sure that I agree!

The Port 80 traffic shows 0.26% having a non-zero ECN field.  Ok, fine.   But, then it shows the breakdown of that 0.26% as:  0.22% ECT(1), 88.22% ECT(0) and 11.55% CE.  Disregarding the small amount of ECT1 traffic, this is 11.6% CE-marking!  While not as obviously broken as the overall statistic of 94% CE marking, it does still seem awfully high to be actual CE-marking for Classic ECN.  

For Port 443, the picture is a little bit more believable, but I'm not sure what we learn from it:  0.3% prevalence of non-zero ECN field, and, of that, 1.15% CE.   
We don't know what the end-to-end RTT is for these connections, but I would estimate the RTT for the NYC-SaoPaulo link itself to be about 130ms, so maybe end-to-end it's 150ms RTT typically? 
If 100% of the ECT connections were traversing an RFC3168 bottleneck (unlikely), I think a 1.15% CE marking rate would mean on the order of a few hundred kbps per flow.  Fairly low, but believable.  If it is more like 10% of them (aligning with Pete's data) that were experiencing a 3168 AQM, then that means 11.5% CE marking for those flows (and a throughput more like ~100 kbps - pretty low, but still believable).  So, is this "evidence" that the prevalence of 3168 AQMs is between 10% and 100%?  I doubt it.

Is it possible that this link was carrying traffic from Buenos Aires?

Currently, I plan to write:

Other studies (e.g. [Trammel], [Bauer], [Mandalari]) have examined ECN traversal, but have not reported data on prevalence of CE-marking by networks. Another [Roddav] examined traces from a Tier 1 ISP link in 2018 and observed that 94% of the non-zero ECN marked packets were CE, which appears to reflect a misconfiguration of equipment using that link, as opposed to providing evidence of RFC3168 AQM deployment.

Let me know if you think there is something more insightful we can conclude from that paper.


-Greg




On 5/7/21, 9:28 AM, "tsvwg on behalf of Greg White" <[e-mail] on behalf of [e-mail]> wrote:

    Hi Sebastian,

    Thanks for pointing this paper out.  I had not seen it previously, and will include a mention of it in the draft.  The results seem a bit puzzling though.  They report that, of the ~5% of IPv4 packets with a non-zero ECN field, 94% of them were CE!  It is hard to believe that this is actual evidence of ECN marking in the wild.  It seems more like evidence of some kind of ToS munging happening for traffic on this ISP's link between NYC and Sao Paulo.  

    While in terms of total packets that CAIDA dataset is huge, in terms of visibility to traffic in the wild (roughly 80-90% of which is directly peered locally as I understand it) it might not be that representative.  Does anyone have further insight into this paper, or the peculiarities of the CAIDA equinix-nyc trace?

    So far, I think the data from Akamai and Apple have been the most useful in understanding the situation.

    -Greg



    On 5/6/21, 3:53 AM, "Sebastian Moeller" <[e-mail]> wrote:

        Hi List, hi Greg,

        sorry for being late, but...

        ...speaking of evidence for ECN marking in the wild; here an analysis of CAIDA internet "back-bone link " traffic from 2018 https://ieeexplore.ieee.org/document/8909187 that has not beed discussed as far as I can tell. The data showed between ~5% of IPv4 packets carrying something other than NotECT in the ECN bitfield, a more detailed look at port resolution showed 0.26% (port 80) or 0.3% (port 443) in IPv4. I do not think this was broken down by network, but indicates non-zero ECN usage in the wild in 2018. IMHO the percentage of traffic might be a better proxy for ECN prevalence than number of networks. That said for figuring out how many/which operators should be pro-actively informed of the coming changes identifying the networks seems helpful.



        > On May 6, 2021, at 05:54, Greg White <[e-mail]> wrote:
        > 
        > All,
        > 
        > I had hoped to upload this further in advance of the interim meeting next week so that participants would have more chance to review it beforehand. 
        > 
        > There are quite a few changes from the last individual draft, as can be seen in the diff: https://www.ietf.org/rfcdiff?url1=draft-white-tsvwg-l4sops-02&url2=draft-ietf-tsvwg-l4sops-00 
        > And, there remain a few TODO items as well.  These changes/TODOs were the mainly result of review comments on the mailing list, though I received some offline comments and suggested text as well.
        > 
        > I'll do a relatively quick summary of the changes during the timeslot on Monday, but for now, the most notable changes are:
        > 
        > New Section 3.1 summarizing recent studies of RFC3168 deployment (Jake, Pete, please review and see if the text is an accurate summary).

        [SM] See above for an additional relevant study with 2018 data that might be worth adding in the next revision.

        > New Section 6 discussing actions that can be taken by an operator of FQ bottlenecks.

        [SM] There I find:
        "This unfairness is compounded by the fact that the FQ	
         	   scheduler will already be causing unfairness to flows within the	
         	   tunnel relative to flows that are not tunneled."

        This argues that the AQM "should" divine the number of true capacity seeking flows in the compound tunnel aggregate and assign the proper share. I argue that this is impossible to robustly solve in general, and, in the case of encrypted tunnels also arguably the wrong thing to do, as doing so would introduce additional, potential severe re-ordering in the encrypted flow. But a partial solutions already exist, on IPv6 a tunnel could make use of the flow-label field to give additional flow identifier to AQMs. Arguably that is what tunnels should do, as that keeps the policy decision how much reordering one will accept in the hand of the tunnel operators.



        > New Section 7 discussing conclusion of the L4S experiment (Sebastian, for now I went with the text proposed on the mailing list, we can revise it in the next draft if needed).

        [SM] Much appreciated, while I have my quibbles, getting this section in feels quite helpful (and I do not pretend my initial draft was in any way complete and/or perfect). But for completeness, I still think it a good idea to lay out the gamut of potentially required/recommended actions at network termination. The argument will be better the more objective it is, and it already carries a noticeable pro-L4S bias.


        Regards
        	Sebastian


        > 
        > -Greg
        > 
        > 
        > On 5/5/21, 6:34 PM, "tsvwg on behalf of [e-mail]" <[e-mail] on behalf of [e-mail]> wrote:
        > 
        > 
        >    A New Internet-Draft is available from the on-line Internet-Drafts directories.
        >    This draft is a work item of the Transport Area Working Group WG of the IETF.
        > 
        >            Title           : Operational Guidance for Deployment of L4S in the Internet
        >            Author          : Greg White
        >    	Filename        : draft-ietf-tsvwg-l4sops-00.txt
        >    	Pages           : 18
        >    	Date            : 2021-05-05
        > 
        >    Abstract:
        >       This document is intended to provide guidance in order to ensure
        >       successful deployment of Low Latency Low Loss Scalable throughput
        >       (L4S) in the Internet.  Other L4S documents provide guidance for
        >       running an L4S experiment, but this document is focused solely on
        >       potential interactions between L4S flows and flows using the original
        >       ('Classic') ECN over a Classic ECN bottleneck link.  The document
        >       discusses the potential outcomes of these interactions, describes
        >       mechanisms to detect the presence of Classic ECN bottlenecks, and
        >       identifies opportunities to prevent and/or detect and resolve
        >       fairness problems in such networks.  This guidance is aimed at
        >       operators of end-systems, operators of networks, and researchers.
        > 
        > 
        >    The IETF datatracker status page for this draft is:
        >    https://datatracker.ietf.org/doc/draft-ietf-tsvwg-l4sops/
        > 
        >    There is also an HTML version available at:
        >    https://www.ietf.org/archive/id/draft-ietf-tsvwg-l4sops-00.html
        > 
        > 
        >    Please note that it may take a couple of minutes from the time of submission
        >    until the htmlized version and diff are available at tools.ietf.org.
        > 
        >    Internet-Drafts are also available by anonymous FTP at:
        >    ftp://ftp.ietf.org/internet-drafts/
        > 
        > 
        >