From nobody Fri May 28 03:03:33 2021 Return-Path: X-Original-To: tsvwg@ietfa.amsl.com Delivered-To: tsvwg@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9C9813A22F1 for ; Fri, 28 May 2021 03:03:31 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.098 X-Spam-Level: X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=heistp.net Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yv-tMGvCH2Bn for ; Fri, 28 May 2021 03:03:26 -0700 (PDT) Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id B086E3A22EF for ; Fri, 28 May 2021 03:03:25 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id r13so1305232wmq.1 for ; Fri, 28 May 2021 03:03:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heistp.net; s=google; h=message-id:subject:from:to:cc:date:in-reply-to:references :user-agent:mime-version; bh=05MJIhWaE1frUPHwZgkir5pK2q1/SYIA4hmzzf+fUbo=; b=QyPcqynZGCwW0P8Vo2orgktL6jsVOJCWo0pZHyTacS8VudWVAqNz4uvcG0/3Nz5hY8 N0gLXOp17lZjv3xB9RWGyD8X++9hzFcYhtr4Hwtljp1jR3mOy8LufOnlTHx9OWofQ2f7 m5OiUibA3hE3J2rkvzQ+1MMw7o70ZhdvF3D3xLEu/iuRHThrQgljUi5H4eUS9VCmMdK2 1gHdRZfShbvKEMEPb+XMivB7G/Wm7EgYQRrf58KvjDwHzgXAP5uMfvBCYYkLd0WflPas Vmf/PjOPU/UwTyG1Us2Q6YadGcKmV9dEjKvOCIKIoBvUxJl4uI4L6uekBy2HB6cJFn8f mgpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:user-agent:mime-version; bh=05MJIhWaE1frUPHwZgkir5pK2q1/SYIA4hmzzf+fUbo=; b=DACf7+oqsRstVEKPGTwHoqR0lDwBxo8HoLI9sxZ+80C87iKbtaohxBfy0FQKeY4anO tUTfpcRI3nHJXlGf3+qS1FNlwmd6rdLgUPgOO4DhDiNhSpHgMpCPHz+1fxpMUPVtvDVo AEYLrmBU30VkNyAv7sWN4ZFUv8XSN/i1Lz6Gl5jtg55uZlyy7jvWDhMhLQ6a9gYTP/oD M1H4/Fd+KeY7ERBq3rZzu5YUph7BU4nkdw+VzuyokIXowAYNqhkf1SeV+uDWbgIliFoQ YVTEhllGPuYVlivg78pLqEsXs21sRL9R4VcOKiaOUSG809C0e/xhPlbZCmvGp/8yO1d3 Tv+Q== X-Gm-Message-State: AOAM531WmWgApcL85Vzt3p3aNGZFhDz2pE3R0IPrmn6GY2ygx9FLiR2V 3je9HAjf0YKav06sIngMT2OBpw== X-Google-Smtp-Source: ABdhPJz8EkE9+znM9azson38jpokzjjTAInbm/8m3PCJRzzKu5ajxGLYl64Op6kVl1o2sWdEOcbvbg== X-Received: by 2002:a1c:a944:: with SMTP id s65mr4891915wme.181.1622196202065; Fri, 28 May 2021 03:03:22 -0700 (PDT) Received: from [10.72.0.88] (h-1169.lbcfree.net. [185.193.85.130]) by smtp.gmail.com with ESMTPSA id u3sm6279317wre.76.2021.05.28.03.03.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 May 2021 03:03:21 -0700 (PDT) Message-ID: <2783a54145a2f1a3afda15ecdb31b741e999b1a0.camel@heistp.net> From: Pete Heist To: "De Schepper, Koen (Nokia - BE/Antwerp)" Cc: "tsvwg@ietf.org" Date: Fri, 28 May 2021 12:03:20 +0200 In-Reply-To: References: <162158815765.22731.15608328324211025925@ietfa.amsl.com> <3F147A3D-BD68-4F0A-89FF-9A92284FF0A5@gmx.de> <0b7edf59-5bce-3189-8745-324083c98ce4@bobbriscoe.net> Content-Type: multipart/alternative; boundary="=-rNcZk+cFx63hIvtwkf4Z" User-Agent: Evolution 3.40.1 MIME-Version: 1.0 Archived-At: Subject: Re: [tsvwg] New rev of draft-ietf-tsvwg-ecn-l4s-id-17 X-BeenThere: tsvwg@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Transport Area Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 May 2021 10:03:32 -0000 --=-rNcZk+cFx63hIvtwkf4Z Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit Hi Koen, On Thu, 2021-05-27 at 08:46 +0000, De Schepper, Koen (Nokia - BE/Antwerp) wrote: > I agree that larger (adaptive) replay window is the main solution, > and removes all other needs. The adaptive window could easily grow > based on the gap that an early packet is creating, allowing the next > packets to fill the gap (optionally shrinking the window afterwards > back to gradually smaller values). >   > As an alternative, another smarter reordering-safe mechanism would > help as well. I think a symmetric window that also drops (or “keeps > aside without moving the window”) sporadic early packets, will > perform better… It would drop (or delay) only the expedited CE(s). I'm just double checking, we're on the same page that the reordering that affects tunnel anti-replay mechanisms doesn't require CE marks, right? It isn't just expedited CEs that can advance the window past the tunnel's valid traffic, but any traffic marked ECT(1), legitimately or not. David wrote a crystal clear explanation earlier, in which CE marks are only mentioned in order to clear up that prior misconception: https://mailarchive.ietf.org/arch/msg/tsvwg/PEVCuDJfbrel74ud8kNJtmVwhHA/ Regards, Pete > This all avoids the need for different SA’s and/or DCSPs… >   > Koen. >   > From: tsvwg On Behalf Of Black, David > Sent: Wednesday, May 26, 2021 9:34 PM > To: Bob Briscoe ; Gorry Fairhurst > ; Wesley Eddy > Cc: tsvwg@ietf.org > Subject: Re: [tsvwg] New rev of draft-ietf-tsvwg-ecn-l4s-id-17 >   > Inline … --David >   > From: Bob Briscoe > Sent: Wednesday, May 26, 2021 1:16 PM > To: Black, David; Gorry Fairhurst; Wesley Eddy > Cc: tsvwg@ietf.org > Subject: Re: [tsvwg] New rev of draft-ietf-tsvwg-ecn-l4s-id-17 >   > [EXTERNAL EMAIL] > David, > On 25/05/2021 21:22, Black, David wrote: > > Two comments: > >   > > [1] This VPN discussion appears to be assuming a remote access VPN, > > so it does not align well with site-to-site VPNs: > >   > > > This particular path divergence occurs iff an ECT0 flow is CE- > > marked /prior/ to the VPN ingress: > > > * In general where the VPN ingress is on the end-system > > (transport mode) that doesn't happen. > > > * There is however a chance that CE could be introduced before a > > 'bump in the wire' VPN ingress (tunnel mode).  > >   > > For site-to-site VPNs, the last line above is a significant > > understatement.  A scenario that helps to understand this is ROBO, > > specifically a sizeable remote office or branch office such as a > > large retail store, where the remote or branch VPN gateway is a > > potential bottleneck node for outbound traffic courtesy of fan-in > > from multiple end systems and the LAN having more bandwidth than > > the (purchased) VPN bandwidth. > > [BB] Wouldn't that particular bottleneck apply marking at the egress > of the VPN gateway after encap? > [David>] That depends on the details of how the VPN is integrated > with the network, e.g., if queue is upstream of encap. >   > > [2] I think the whole approach of parallel SAs is not a good idea – > > it would be better to recommend larger replay windows, perhaps much > > larger, and/or use DSCPs to direct traffic to parallel SAs (which > > would disambiguate CE across L4S and non-L4S traffic). > > [BB] You haven't given any reasons. >   > [David>] Sure: > * See RFC 7657 for general discussion about why variable QoS > treatment > within a single reliable transport protocol session is not a good > idea – that discussion is couched in Diffserv terms, but it's > applicable to L4S ECN-based service differentiation. > * Asking VPNs to classify traffic based on ECN field values is also > asking for the possibility of discard of traffic marked with some of > those values, which is not a good place to go given the desire for > the ECN field to remain end-to-end. > * Using DSCPs is a "reduce to previous case" approach that doesn't > involve modification of any security specs and aligns with existing > VPN implementations that are cognizant of DSCPs. > * Larger replay windows are a lower-effort approach that ought to > apply to deployed VPNs whose replay window is configurable.  In > practice, this is the place to start if L4S causes VPN problems. > > > Bob > > >   > > Thanks, --David > >   > > From: Bob Briscoe > > Sent: Sunday, May 23, 2021 6:00 PM > > To: Black, David; Gorry Fairhurst; Wesley Eddy > > Cc: tsvwg@ietf.org > > Subject: Re: [tsvwg] New rev of draft-ietf-tsvwg-ecn-l4s-id-17 > >   > > [EXTERNAL EMAIL] > > David, > > See inline, tagged [BB]... > > On 21/05/2021 23:02, Black, David wrote: > > > > section 6.2 with its, "use two SAs, one for ECT(1) and one for > the rest" seems a bit limited > > >   > > > The situation is more severe than that, and it's problematic if > there's any non-L4S TCP traffic that uses ECN involved.  The two SAs > are actually for 2 ECN values each: > > >   > > >    To avoid this limitation, a VPN ingress participating in the > L4S > > >    experiment SHOULD map packets onto two parallel SAs indexed by > the > > >    lowest significant bit of the IP-ECN field. > > >   > > > That indexing puts Not-ECT [00b] and ECT(0) [10b] packets into > one SA, and puts ECT(1) [01b] and CE [11b] packets into another.  > Non-L4S TCP uses ECT(0) and CE, hence a flow that carries any > congestion indications is split across those two SAs.  ECMP based on > SPI (deployed technology) that puts those parallel SAs on different > network paths is not assured to preserve packet order across those > SAs, and packet reordering is not a good thing to do to a non-L4S TCP > flow. > > > > [BB] Yes, this would be a potential problem. > > > > This particular path divergence occurs iff an ECT0 flow is CE- > > marked /prior/ to the VPN ingress: > > * In general where the VPN ingress is on the end-system (transport > > mode) that doesn't happen. > > * There is however a chance that CE could be introduced before a > > 'bump in the wire' VPN ingress (tunnel mode). > > > > For a VPN participating in the L4S experiment with two SAs, if > > there was any CE before the ingress, we did think carefully about > > which way to classify it: > > * If CE goes with ECT0, then if a subsequent L4S node gives ECT0 > > more queue delay than CE, the VPN anti-replay function could > > discard some ECT0 packets. > > * If CE goes with ECT1, it avoids anti-replay discard completely. > > We had tried to think of possible problems with splitting Classic > > flows across SAs,  but we hadn't thought of your point where ECMP > > on the SPI makes CE and ECT0 from the same flow taking different > > paths. > > > > I believe it is still best to classify CE with ECT1, based on the > > following reasoning. I think the scenario would occur with very low > > probability {Note 1}, and if it did, the CE might either arrive a > > little earlier or a little later than data packets around them. > > * If earlier, the benefit of ECN would not be lost, but there would > > be an extremely small chance (p_s * p_r * p_N) of an occasional > > spurious re-xmt {Note 2}. > > * If later, the chance that a delayed CE would be mistaken for a > > drop and trigger a spurious re-xmt and a congestion response would > > probably be higher, but still very small (p_s * p_r * p_l) {Note > > 3}. That would lose the benefit of ECN but, at least for Classic > > flows, a single CE is meant to trigger a large congestion response > > as if it was a loss anyway. > > > > I don't want to belittle the point you've made, because this is > > certainly a new problem. However, I do think the probabilities need > > to be put in perspective. > > > > I suspect the ECMP problem with any ECN (3168 or L4S) is likely to > > be a greater concern. You might recall that João Taveira from > > Fastly pointed this out during a tsvwg discussion on ECN back in > > 2017. This link should jump to 14:25 mins into a 2017 NANOG talk > > from Lorenzo Saino of Fastly about how they had to disable ECN > > negotiation when Apple clients started to request ECN in 2015 - > > because a number of different ECMP vendors still hash on the ToS > > byte for ECMP and load balancing, so the data would have been > > routed to a different server from the handshake: > >     https://youtu.be/ciClZdwHelU?t=805 [youtu.be] > > > > Regards > > > > > > > > Bob > > > > {Note 1}: Reasoning for scenario occurring with low probability > > (we'll call it p_s): > > Usually, an institution provides a tunnel-mode VPN between > > physically secured networks (e.g. between their intranet and an > > employee's home laptop). AQM deployment would be unusual in the > > interior of corporate and institutional intranets, because the > > bottleneck is more likely elsewhere (e.g. in the access link > > between the site and the Internet). However, there are bound to be > > some cases of tunnel-mode VPNs where the traffic arriving could > > have already been ECN-marked (for instance Jonathan Morton's > > example of gamers providing a VPN end-point on the public Internet > > to hide their own IP address). > > Then, to experience this problem, there also has to be some load > > balancing or ECMP after the VPN ingress but before the egress. I'm > > sure that will occur sometime somewhere (let's say with probability > > p_r). The likelihood of the scenario occurring will then be p_s * > > p_r. > > > > {Note 2}: > > * N packets in a row within a Classic flow have to be CE-marked for > > early CE packets to result in a spurious re-xmt, let's say that > > happens with probability p_N. As already explained in Appx B of > > ecn-l4s-id, N was traditionally 3, but RACK is making it larger. > > And the main sources of ECN marking for Classic flows are FQ_CoDel > > and CAKE, both of which take great care not to mark multiple > > packets in a row. So p_N is going to be tiny. > > And the probability of a spurious re-xmt with early reordering will > > be the vanishingly small product (p_s * p_r * p_N). > > > > {Note 3}: > > * Let's say p_l (for late) is the probability that the reordering > > between the two SAs is sufficient to make each CE packet arrive > > late enough  to be deemed a loss. p_l is unlikely to be as small as > > p_N, but the overall probability of this occurring is still reduced > > because the probability that CE marking precedes that VPN (p_s), > > and that there's routing keyed on flow IDs and SPIs (p_r). So the > > overall probability of a spurious rexmt due to late reordering is > > (p_s * p_r * p_l). > > > > > > > > Bob > > > > > > >   > > > Thanks, --David > > >   > > > -----Original Message----- > > > From: Sebastian Moeller > > > Sent: Friday, May 21, 2021 5:27 PM > > > To: Bob Briscoe > > > Cc: Gorry Fairhurst; Black, David; Wesley Eddy; tsvwg@ietf.org > > > Subject: Re: [tsvwg] New rev of draft-ietf-tsvwg-ecn-l4s-id-17 > > >   > > >   > > > [EXTERNAL EMAIL] > > >   > > > Bob, chairs, > > >   > > > section 6.2 with its, "use two SAs, one for ECT(1) and one for > the rest" seems a bit limited since it ignores that VPNs might > propagate both DSCPs and ECN bits between the layers, so IMHO a > better approach might be to recommend to treat DSCP+ECN bits as one > aggregate byte (let's cal it TOS ;) ) as the extra ECT(1)-SA seems to > be required for all SAs that already exist to deal with multiple > supported DSCPs. So in a sense the recommendation would be to double > the number of SAs. > > > > [BB] Yes, we ought to reword it to say that the VPN ingress should > > use two SAs indexed on the LSB of the ECN field, and, if it was > > also classifying on DSCPs, it could also consider classifying any > > low latency DSCP(s) with the L4S packets. To avoid the anti-replay > > problem, there would only need to be one SA configured per each > > degree of queuing delay, not one for every ECN x DSCP combination. > > > > > > >   > > >   > > > Also: > > > "and the current draft of DTLS 1.3 says "The receiver       > > >      SHOULD pick a window large enough to handle any plausible > reordering,      > > >      which depends on the data rate."  However, in practice, the > size of  > > >      the VPN's anti-replay window is not always scaled > appropriately." > > >   > > > L4S on a 10 ms path under load can introduce re-ordering in the > range of 50 ms (roughly twice the difference between the L- and C- > queue delay targets), re-ordering tolerance 5 times of the path RTT > seems to be a bit on the high side to expect, no? > > > > [BB] The above text that I quoted from the DTLS spec. is > > reasonable, both practically (see below) and in terms of taking > > responsibility for the problem. Beyond its window, the anti-replay > > function presumes a packet is guilty of a replay attack with no > > evidence, purely because it chooses not to hold that amount of > > evidence. Therefore it's proper that it holds a sufficient window > > of evidence for any plausible reordering. > > > > BTW, the C-queue target has never been 25ms. I noticed JM said that > > incorrectly as well recently. > > * A default C queue delay target of 15ms has always been > > recommended in aqm-dualq-coupled. That results in PI2 Qdelay of > > about 25ms at the 99%ile or 30ms at the 99.9%ile. We have been > > considering whether to change the default target to 10ms for some > > time, but not done so yet. > > * Low Latency DOCSIS specifies a default C queue delay target of > > 10ms. > > > > So a replay window allowing for 30ms of packets at the interface > > rate would be sufficient. > > At 1Gb/s (say) using 1500B packets, that's a replay window of 2500 > > packets. > > > > Quoting Pete Heist's info here https://github.com/heistp/l4s- > > tests/#dropped-packets-for-tunnels-with-replay-protection-enabled > > [github.com] : > > > "Modern Linux kernels have a default maximum replay window size > > > of 4096 (XFRMA_REPLAY_ESN_MAX inxfrm.h [elixir.bootlin.com]). > > > Wireguard uses a hardcoded value of 8192 with no option for > > > runtime configuration, increased from 2048 in May 2020 bythis > > > commit [git.zx2c4.com]." > > Regards > > > > > > > > Bob > > > > > > >   > > >   > > >   > > >   > > >   > > > Regards > > >   Sebastian > > >   > > >   > > > > On May 21, 2021, at 11:21, Bob Briscoe > wrote: > > > >   > > > > Chairs, list, > > > >   > > > > We've posted a new rev of draft-ietf-tsvwg-ecn-l4s-id-17 > attempting to address all the discussion since the last posting just > before the interim. In particular: > > > > * review comments on a careful read from Gorry and the chairs > > > > * the VPN anti-replay problem > > > > * added an out-of-band test for an RFC3168 ECN AQM in a shared > queue. > > > >   > > > > There are a couple of outstanding discussions, which I'm sure > will continue on the list, e.g. the role of RFC4774 and whether to > remove any of Appx C. But it was considered better to get the queued > up changes out, to re-base the discussions. > > > >   > > > > This is quite an extensive set of changes, so pls check and > pass any comments to the list. > > > >   > > > > Thanks for everyone who is contributing, and particularly to > the chairs for continuing to referee this all. We've added > appropriate thanks in the Acks section. > > > >   > > > >   > > > > Bob > > > >   > > > >   > > > > On 21/05/2021 10:09, internet-drafts@ietf.org wrote: > > > > > A New Internet-Draft is available from the on-line Internet- > Drafts directories. > > > > > This draft is a work item of the Transport Area Working Group > WG of the IETF. > > > > >   > > > > >         Title           : Explicit Congestion Notification > (ECN) Protocol for Very Low Queuing Delay (L4S) > > > > >         Authors         : Koen De Schepper > > > > >                           Bob Briscoe > > > > > Filename        : draft-ietf-tsvwg-ecn-l4s-id-17.txt > > > > > Pages           : 57 > > > > > Date            : 2021-05-21 > > > > >   > > > > > Abstract: > > > > >    This specification defines the protocol to be used for a > new network > > > > >    service called low latency, low loss and scalable > throughput (L4S). > > > > >    L4S uses an Explicit Congestion Notification (ECN) scheme > at the IP > > > > >    layer that is similar to the original (or 'Classic') ECN > approach, > > > > >    except as specified within.  L4S uses 'scalable' > congestion control, > > > > >    which induces much more frequent control signals from the > network and > > > > >    it responds to them with much more fine-grained > adjustments, so that > > > > >    very low (typically sub-millisecond on average) and > consistently low > > > > >    queuing delay becomes possible for L4S traffic without > compromising > > > > >    link utilization.  Thus even capacity-seeking (TCP-like) > traffic can > > > > >    have high bandwidth and very low delay at the same time, > even during > > > > >    periods of high traffic load. > > > > >   > > > > >    The L4S identifier defined in this document distinguishes > L4S from > > > > >    'Classic' (e.g. TCP-Reno-friendly) traffic.  It gives an > incremental > > > > >    migration path so that suitably modified network > bottlenecks can > > > > >    distinguish and isolate existing traffic that still > follows the > > > > >    Classic behaviour, to prevent it degrading the low queuing > delay and > > > > >    low loss of L4S traffic.  This specification defines the > rules that > > > > >    L4S transports and network elements need to follow with > the intention > > > > >    that L4S flows neither harm each other's performance nor > that of > > > > >    Classic traffic.  Examples of new active queue management > (AQM) > > > > >    marking algorithms and examples of new transports (whether > TCP-like > > > > >    or real-time) are specified separately. > > > > >   > > > > >   > > > > > The IETF datatracker status page for this draft is: > > > > > > https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9qj70HfR$ > [datatracker[.]ietf[.]org] > > > > >   > > > > > There is also an htmlized version available at: > > > > > > https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-17__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9vC50L2Y$ > [datatracker[.]ietf[.]org] > > > > >   > > > > > A diff from the previous version is available at: > > > > > > https://urldefense.com/v3/__https://www.ietf.org/rfcdiff?url2=draft-ietf-tsvwg-ecn-l4s-id-17__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9oZoDRib$ > [ietf[.]org] > > > > >   > > > > >   > > > > > Internet-Drafts are also available by anonymous FTP at: > > > > > > https://urldefense.com/v3/__ftp://ftp.ietf.org/internet-drafts/__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9rKXcwvA$ > [ftp[.]ietf[.]org] > > > > >   > > > > >   > > > >   > > > > -- > > > > > ________________________________________________________________ > > > > Bob Briscoe                               > https://urldefense.com/v3/__http://bobbriscoe.net/__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9gj_uPXC$ > [bobbriscoe[.]net] > > > >   > > >   > > > > > > -- > > ________________________________________________________________ > > Bob Briscoe                               http://bobbriscoe.net/ > [bobbriscoe.net] >   > -- > ________________________________________________________________ > Bob Briscoe                               http://bobbriscoe.net/ > [bobbriscoe.net] --=-rNcZk+cFx63hIvtwkf4Z Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable
Hi Koen,

On Thu, 2021-05-27 at 08:46 = +0000, De Schepper, Koen (Nokia - BE/Antwerp) wrote:

I agree that = larger (adaptive) replay window is the main solution, and removes all other= needs. The adaptive window could easily grow based on the gap that an earl= y packet is creating, allowing the next packets to fill the gap (optionally= shrinking the window afterwards back to gradually smaller values).

 

A= s an alternative, another smarter reordering-safe mechanism would help as w= ell. I think a symmetric window that also drops (or =E2=80=9Ckeeps aside wi= thout moving the window=E2=80=9D) sporadic early packets, will perform bett= er=E2=80=A6 It would drop (or delay) only the expedited CE(s).


I'm just double checking, we're on the same p= age that the reordering that affects tunnel anti-replay mechanisms doesn't = require CE marks, right? It isn't just expedited CEs that can advance the w= indow past the tunnel's valid traffic, but any traffic marked ECT(1), legit= imately or not. David wrote a crystal clear explanation earlier, in which C= E marks are only mentioned in order to clear up that prior misconception:


Regards,
=
Pete

This all avoids the need for different SA= =E2=80=99s and/or DCSPs=E2=80=A6

=  

Koen.

 

From: tsvwg <tsvwg-bounces@ietf.org> On Behalf Of Black, DavidSent: Wednesday, May 26, 2021 9:34 PM
To: Bob Briscoe <= ;ietf@bobbriscoe.net>; Gorry Fairhurst <gorry@erg.abdn.ac.uk>; Wes= ley Eddy <wes@mti-systems.com>
Cc: tsvwg@ietf.org
Sub= ject: Re: [tsvwg] New rev of draft-ietf-tsvwg-ecn-l4s-id-17<= /p>

 

Inline =E2=80=A6 --David

 

 

[EXTERNAL EMAI= L]

David,

On 25/05/2021 21= :22, Black, David wrote:

Two comments:=

 

[1] Th= is VPN discussion appears to be assuming a remote access VPN, so it does no= t align well with site-to-site VPNs:

 

> This particular path divergence occurs iff an ECT0 flow is= CE-marked /prior/ to the VPN ingress:
> * In general where the VPN i= ngress is on the end-system (transport mode) that doesn't happen.
> *= There is however a chance that CE could be introduced before a 'bump in th= e wire' VPN ingress (tunnel mode). 

 = ;

For site-to-site VPNs, the last line above is a significant understatement= .  A scenario that helps to understand this is ROBO, specifically a si= zeable remote office or branch office such as a large retail store, where t= he remote or branch VPN gateway is a potential bottleneck node for outbound= traffic courtesy of fan-in from multiple end systems and the LAN having mo= re bandwidth than the (purchased) VPN bandwidth.


[BB] Wouldn't that particular bottleneck= apply marking at the egress of the VPN gateway after encap?

[David>] That depends on the details of how the VPN i= s integrated with the network, e.g., if queue is upstream of encap.

 

<= span style=3D"color:#1F497D">[2] I think the whole approach of parallel SAs= is not a good idea =E2=80=93 it would be better to recommend larger replay= windows, perhaps much larger, and/or use DSCPs to direct traffic to parall= el SAs (which would disambiguate CE across L4S and non-L4S traffic).=


[BB] You haven't giv= en any reasons.

 =

[David&= gt;] Sure:

  • See RFC 7657 for general discussion about why va= riable QoS treatment within a single reliable transport protocol session is= not a good idea =E2=80=93 that discussion is couched in Diffserv terms, bu= t it's applicable to L4S ECN-based service differentiation.<= /li>
  • Asking VPNs to classify traffic based on ECN fie= ld values is also asking for the possibility of discard of traffic marked w= ith some of those values, which is not a good place to go given the desire = for the ECN field to remain end-to-end.
  • Using DSCPs is a "reduce to previous case" approach that doesn't inv= olve modification of any security specs and aligns with existing VPN implem= entations that are cognizant of DSCPs.
  • Larger replay windows are a lower-effort approach that ought to apply= to deployed VPNs whose replay window is configurable.  In practice, t= his is the place to start if L4S causes VPN problems.



    • Bob

     

    Thanks, --David

    =

     

    From: Bob Briscoe <ietf@bobbriscoe.net>
    Sent= : Sunday, May 23, 2021 6:00 PM
    To: Black, David; Gorry Fairhu= rst; Wesley Eddy
    Cc: tsvwg@ietf= .org
    Subject: Re: [tsvwg] New rev of draft-ietf-tsvwg-ecn-l4s= -id-17

     

    [EXTERNAL EMAIL] =

    David,
    = See inline, tagged [BB]...

    On 21/= 05/2021 23:02, Black, David wrote:

    section 6.2 with its, "use two SAs, =
one for ECT(1) and one for the rest" seems a bit limited
     
    The situation is more severe th=
an that, and it's problematic if there's any non-L4S TCP traffic that uses =
ECN involved.  The two SAs are actually for 2 ECN values each:
     
       To avoid this limita=
tion, a VPN ingress participating in the L4S
     &nb=
sp; experiment SHOULD map packets onto two parallel SAs indexed by the=

       lowest significant bit of the IP-ECN field.
     
    That indexing puts Not-ECT =
[00b] and ECT(0) [10b] packets into one SA, and puts ECT(1) [01b] and CE [1=
1b] packets into another.  Non-L4S TCP uses ECT(0) and CE, hence a flo=
w that carries any congestion indications is split across those two SAs.&nb=
sp; ECMP based on SPI (deployed technology) that puts those parallel SAs on=
 different network paths is not assured to preserve packet order across tho=
se SAs, and packet reordering is not a good thing to do to a non-L4S TCP fl=
ow.


    [BB] Yes, this would be a potential problem.

    This pa= rticular path divergence occurs iff an ECT0 flow is CE-marked /prior/ to th= e VPN ingress:
    * In general where the VPN ingress is on the end-system (= transport mode) that doesn't happen.
    * There is however a chance that CE= could be introduced before a 'bump in the wire' VPN ingress (tunnel mode).=

    For a VPN participating in the L4S experiment with two SAs, if ther= e was any CE before the ingress, we did think carefully about which way to = classify it:
    * If CE goes with ECT0, then if a subsequent L4S node gives= ECT0 more queue delay than CE, the VPN anti-replay function could discard = some ECT0 packets.
    * If CE goes with ECT1, it avoids anti-replay discard= completely. We had tried to think of possible problems with splitting Clas= sic flows across SAs,  but we hadn't thought of your point where ECMP = on the SPI makes CE and ECT0 from the same flow taking different paths.
    =
    I believe it is still best to classify CE with ECT1, based on the follo= wing reasoning. I think the scenario would occur with very low probability = {Note 1}, and if it did, the CE might either arrive a little earlier or a l= ittle later than data packets around them.
    * If earlier, the benefit of= ECN would not be lost, but there would be an extremely small chance (p_s *= p_r * p_N) of an occasional spurious re-xmt {Note 2}.
    * If later, the c= hance that a delayed CE would be mistaken for a drop and trigger a spurious= re-xmt and a congestion response would probably be higher, but still very = small (p_s * p_r * p_l) {Note 3}. That would lose the benefit of ECN but, a= t least for Classic flows, a single CE is meant to trigger a large congesti= on response as if it was a loss anyway.

    I don't want to belittle the= point you've made, because this is certainly a new problem. However, I do = think the probabilities need to be put in perspective.

    I suspect the= ECMP problem with any ECN (3168 or L4S) is likely to be a greater concern.= You might recall that Jo=C3=A3o Taveira from Fastly pointed this out durin= g a tsvwg discussion on ECN back in 2017. This link should jump to 14:25 mi= ns into a 2017 NANOG talk from Lorenzo Saino of Fastly about how they had t= o disable ECN negotiation when Apple clients started to request ECN in 2015= - because a number of different ECMP vendors still hash on the ToS byte fo= r ECMP and load balancing, so the data would have been routed to a differen= t server from the handshake:
        https://youtu.be/ciClZ= dwHelU?t=3D805 [youtu.be]

    Regards



    Bob

    {Not= e 1}: Reasoning for scenario occurring with low probability (we'll call it = p_s):
    Usually, an institution provides a tunnel-mode VPN between physica= lly secured networks (e.g. between their intranet and an employee's home la= ptop). AQM deployment would be unusual in the interior of corporate and ins= titutional intranets, because the bottleneck is more likely elsewhere (e.g.= in the access link between the site and the Internet). However, there are = bound to be some cases of tunnel-mode VPNs where the traffic arriving could= have already been ECN-marked (for instance Jonathan Morton's example of ga= mers providing a VPN end-point on the public Internet to hide their own IP = address).
    Then, to experience this problem, there also has to be some l= oad balancing or ECMP after the VPN ingress but before the egress. I'm sure= that will occur sometime somewhere (let's say with probability p_r). The l= ikelihood of the scenario occurring will then be p_s * p_r.

    {Note 2}= :
    * N packets in a row within a Classic flow have to be CE-marked for ea= rly CE packets to result in a spurious re-xmt, let's say that happens with = probability p_N. As already explained in Appx B of ecn-l4s-id, N was tradit= ionally 3, but RACK is making it larger. And the main sources of ECN markin= g for Classic flows are FQ_CoDel and CAKE, both of which take great care no= t to mark multiple packets in a row. So p_N is going to be tiny.
    And the= probability of a spurious re-xmt with early reordering will be the vanishi= ngly small product (p_s * p_r * p_N).

    {Note 3}:
    * Let's say p_l (= for late) is the probability that the reordering between the two SAs is suf= ficient to make each CE packet arrive late enough  to be deemed a loss= . p_l is unlikely to be as small as p_N, but the overall probability of thi= s occurring is still reduced because the probability that CE marking preced= es that VPN (p_s), and that there's routing keyed on flow IDs and SPIs (p_r= ). So the overall probability of a spurious rexmt due to late reordering is= (p_s * p_r * p_l).



    Bob


     
    Thanks, --David
     
    -----Original Message-----
    From: Sebastian Moeller <=
;moeller0@gmx.de> 
    Sent: Friday, May 21, 2021 5=
:27 PM
    To: Bob Briscoe
    Cc: Gorry =
Fairhurst; Black, David; Wesley Eddy; tsv=
wg@ietf.org
    Subject: Re: [tsvwg] New rev of draft-=
ietf-tsvwg-ecn-l4s-id-17
     
    &=
nbsp;
    [EXTERNAL EMAIL] 
     
    Bob, chairs,
     
    section 6.2 with its, "use two SAs, one for ECT(1) and one for the re=
st" seems a bit limited since it ignores that VPNs might propagate both DSC=
Ps and ECN bits between the layers, so IMHO a better approach might be to r=
ecommend to treat DSCP+ECN bits as one aggregate byte (let's cal it TOS ;) =
) as the extra ECT(1)-SA seems to be required for all SAs that already exis=
t to deal with multiple supported DSCPs. So in a sense the recommendation w=
ould be to double the number of SAs.


    [BB] Yes, we ought to rew= ord it to say that the VPN ingress should use two SAs indexed on the LSB of= the ECN field, and, if it was also classifying on DSCPs, it could also con= sider classifying any low latency DSCP(s) with the L4S packets. To avoid th= e anti-replay problem, there would only need to be one SA configured per ea= ch degree of queuing delay, not one for every ECN x DSCP combination.

     
     
    Also:
    "and the curre=
nt draft of DTLS 1.3 says "The receiver      =
 
         SHOULD pick a window l=
arge enough to handle any plausible reordering,    &nbs=
p; 
         which depends on the=
 data rate."  However, in practice, the size of  
         the VPN's anti-replay window is not alw=
ays scaled appropriately."
     
    L4S on a 10 ms path under load can introduce re-ordering in the range of 5=
0 ms (roughly twice the difference between the L- and C-queue delay targets=
), re-ordering tolerance 5 times of the path RTT seems to be a bit on the h=
igh side to expect, no?


    [BB] The above text that I quoted from the DTLS spec. is reasonable, b= oth practically (see below) and in terms of taking responsibility for the p= roblem. Beyond its window, the anti-replay function presumes a packet is gu= ilty of a replay attack with no evidence, purely because it chooses not to = hold that amount of evidence. Therefore it's proper that it holds a suffici= ent window of evidence for any plausible reordering.

    BTW, the C-queu= e target has never been 25ms. I noticed JM said that incorrectly as well re= cently.
    * A default C queue delay target of 15ms has always been recomme= nded in aqm-dualq-coupled. That results in PI2 Qdelay of about 25ms at the = 99%ile or 30ms at the 99.9%ile. We have been considering whether to change = the default target to 10ms for some time, but not done so yet.
    * Low Lat= ency DOCSIS specifies a default C queue delay target of 10ms.

    So a = replay window allowing for 30ms of packets at the interface rate would be s= ufficient.
    At 1Gb/s (say) using 1500B packets, that's a replay window of= 2500 packets.

    Quoting Pete Heist's info here https://github.com/heistp/l4s-tests= /#dropped-packets-for-tunnels-with-replay-protection-enabled [github.com] :

    "Moder= n Linux kernels have a default maximum replay window size of 4096 (XFRMA_REPLAY_ESN_MAX inxfrm.h [elixir.bootlin.com]). Wireguard u= ses a hardcoded value of 8192 with no option for runtime configuration, inc= reased from 2048 in May 2020 bythis commit [git.zx2c4.com]."

    Re= gards



    Bob


     
     
     =

     
     
    Re=
gards
      Sebastian
     
     
    On=
 May 21, 2021, at 11:21, Bob Briscoe <ietf@bobbriscoe.net> wrote:
     
    Chairs, list,
     
    We've posted a new rev of draft-ietf-tsvwg-ecn-l4s-id-17 attempting to ad= dress all the discussion since the last posting just before the interim. In= particular:
    * review comments on a careful read from =
Gorry and the chairs
    * the VPN anti-replay problem
    * added an out-of-band test for an RFC3168 ECN AQM in a s=
hared queue.
     
    There are a c=
ouple of outstanding discussions, which I'm sure will continue on the list,=
 e.g. the role of RFC4774 and whether to remove any of Appx C. But it was c=
onsidered better to get the queued up changes out, to re-base the discussio=
ns.
     
    This is quite an exten=
sive set of changes, so pls check and pass any comments to the list.
     
    Thanks for everyone who is contr=
ibuting, and particularly to the chairs for continuing to referee this all.=
 We've added appropriate thanks in the Acks section.
    &=
nbsp;
     
    Bob
    = 
     
     
    On 21/05/2021 =
10:09, internet-drafts@ietf.org=
 wrote:
    A New Internet-=
Draft is available from the on-line Internet-Drafts directories.=

    This draft is a work item of the Transport Area Working Group WG=
 of the IETF.
     
      =
      Title      &nb=
sp;    : Explicit Congestion Notification (ECN) Protocol for=
 Very Low Queuing Delay (L4S)
        =
    Authors         =
: Koen De Schepper
          =
;            &n=
bsp;       Bob Briscoe
     =
Filename        : draft-ietf-tsvwg-ecn-l=
4s-id-17.txt
     Pages      =
;     : 57
     Date   =
         : 2021-05-21
     
    Abstract:
     &n=
bsp; This specification defines the protocol to be used for a new network
       service called low latency, low loss and s=
calable throughput (L4S).
       L4S uses an Exp=
licit Congestion Notification (ECN) scheme at the IP
    &=
nbsp;  layer that is similar to the original (or 'Classic') ECN approa=
ch,
       except as specified within.  L4S=
 uses 'scalable' congestion control,
       whic=
h induces much more frequent control signals from the network and
       it responds to them with much more fine-grained ad=
justments, so that
       very low (typically su=
b-millisecond on average) and consistently low
     &=
nbsp; queuing delay becomes possible for L4S traffic without compromising
       link utilization.  Thus even capacity=
-seeking (TCP-like) traffic can
       have high=
 bandwidth and very low delay at the same time, even during
       periods of high traffic load.
     =
;
       The L4S identifier defined in this docu=
ment distinguishes L4S from
       'Classic' (e.=
g. TCP-Reno-friendly) traffic.  It gives an incremental
       migration path so that suitably modified network bottle=
necks can
       distinguish and isolate existin=
g traffic that still follows the
       Classic =
behaviour, to prevent it degrading the low queuing delay and
       low loss of L4S traffic.  This specification defin=
es the rules that
       L4S transports and netw=
ork elements need to follow with the intention
     &=
nbsp; that L4S flows neither harm each other's performance nor that of=

       Classic traffic.  Examples of new active=
 queue management (AQM)
       marking algorithm=
s and examples of new transports (whether TCP-like
    &nb=
sp;  or real-time) are specified separately.
    &nbs=
p;
     
    The IETF datatracker st=
atus page for this draft is:
    https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-ietf-ts=
vwg-ecn-l4s-id/__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU=
68p06V9qj70HfR$ [datatracker[.]ietf[.]org]
     <=
o:p>
    There is also an htmlized version available at:
    https://urldefense.com/v3/_=
_https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-17__;!!L=
pKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9vC50L2Y$ [=
datatracker[.]ietf[.]org]
     
    =
A diff from the previous version is available at:
    https://urldefense.com/v3/__https://www.ietf.org=
/rfcdiff?url2=3Ddraft-ietf-tsvwg-ecn-l4s-id-17__;!!LpKI!xeGDaVTRL33QRdX3Tos=
2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9oZoDRib$ [ietf[.]org]<=
/pre>
     
     
    Internet-=
Drafts are also available by anonymous FTP at:
    https:/=
/urldefense.com/v3/__ftp://ftp.ietf.org/internet-drafts/__;!!LpKI!xeGDaVTRL=
33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68p06V9rKXcwvA$ [ftp[.]ietf[.]=
org]
     
     
     
    -- 
    __=
______________________________________________________________
    Bob Briscoe         &=
nbsp;           &nbs=
p;         https://urldefense.com/v3/__http://bo=
bbriscoe.net/__;!!LpKI!xeGDaVTRL33QRdX3Tos2AURXirtYtZXHcEP8W5a6OO_m4gWSHU68=
p06V9gj_uPXC$ [bobbriscoe[.]net]
     =

     



    -- =

    __________________________________________________________=
______
    Bob Briscoe      =
            &nb=
sp;            http://bobbriscoe.=
net/ [bobbriscoe.net]

     

    -- <=
/pre>
    ________________________________________________________________<=
o:p>
    Bob Briscoe       &=
nbsp;           &nbs=
p;           http://bobbriscoe.net/ [b=
obbriscoe.net]

<= span>
--=-rNcZk+cFx63hIvtwkf4Z--