Re: [tsvwg] New rev of draft-ietf-tsvwg-ecn-l4s-id-17

[Bob Briscoe <[e-mail]>]

Pete,

On 24/05/2021 08:54, Pete Heist wrote:
> Hi Bob/Sebastian,
>
> On Sun, 2021-05-23 at 21:54 +0100, Bob Briscoe wrote:
>> Sebastian, inline [BB]
>>
>> On 21/05/2021 22:26, Sebastian Moeller wrote:
>>
>>> Bob, chairs,
>>> section 6.2 with its, "use two SAs, one for ECT(1) and one for the rest" seems a bit limited since it ignores that VPNs might propagate both DSCPs and ECN bits between the layers, so IMHO a better approach might be to recommend to treat DSCP+ECN bits as one aggregate byte (let's cal it TOS ;) ) as the extra ECT(1)-SA seems to be required for all SAs that already exist to deal with multiple supported DSCPs. So in a sense the recommendation would be to double the number of SAs.
>>
>> [BB] Yes, we ought to reword it to say that the VPN ingress should 
>> /at least/ use two SAs indexed on the LSB of the ECN field, and, if 
>> it is also classifying on DSCPs, it could also consider classifying 
>> any low latency DSCP(s) with the L4S packets. To avoid the 
>> anti-replay problem, there would only need to be one SA configured 
>> per each degree of queuing delay, not one for every ECN x DSCP 
>> combination.
>>
>> We'll have to see how common multiple combinations are in practice. 
>> As ecn-l4s-id says, L4S with just best efforts...
>>     "is expected to be the most common and useful
>>     arrangement.  But, more generally, an operator might choose to
>>     control bandwidth allocation through a hierarchy of Diffserv PHBs"
>>
>> So the ECN field could be the only field that gives a delay delta. 
>> However different networks will have their own view on which 
>> technology they want to use for low latency. So VPNs will probably 
>> need to cater for both DSCPs and the ECN field being used for low 
>> delay in different networks.
>>
>>
>>> Also:
>>> "and the current draft of DTLS 1.3 says "The receiver	
>>>   	   SHOULD pick a window large enough to handle any plausible reordering,	
>>>   	   which depends on the data rate."  However, in practice, the size of	
>>>   	   the VPN's anti-replay window is not always scaled appropriately."
>>> L4S on a 10 ms path under load can introduce re-ordering in the range of 50 ms (roughly twice the difference between the L- and C-queue delay targets), re-ordering tolerance 5 times of the path RTT seems to be a bit on the high side to expect, no?
>>
>> [BB] IMO, the above text that I quoted from the DTLS spec. is 
>> reasonable, both practically (see below) and in terms of taking 
>> responsibility for the problem. Beyond its window, the anti-replay 
>> function presumes a packet is guilty of a replay attack with no 
>> evidence, purely because it chooses not to hold that amount of 
>> evidence. Therefore it's proper that it holds a sufficient window of 
>> evidence for any plausible reordering.
>>
>> BTW, the C-queue target has never been 25ms. I noticed JM said that 
>> incorrectly as well recently.
>> * A default C queue delay target of 15ms has always been recommended 
>> in aqm-dualq-coupled. Under a heavy load of short and long flow 
>> arrivals in both the L&C queues, that results in PI2 Qdelay of about 
>> 25ms at the 99%ile or 30ms at the 99.9%ile. We have been considering 
>> whether to change the default target to 10ms for some time, but not 
>> done so yet.
>> * Low Latency DOCSIS specifies a default C queue delay target of 10ms.
>>
>> So a replay window allowing for 30ms of packets at the interface rate 
>> would probably be sufficient.
>> At 1Gb/s (say) using 1500B packets, that's a replay window of 2500 
>> packets.
>>
>> Quoting Pete Heist's info 
>> herehttps://github.com/heistp/l4s-tests/#dropped-packets-for-tunnels-with-replay-protection-enabled 
>> :
>>
>>> "Modern Linux kernels have a default maximum replay window size of 
>>> 4096 (|XFRMA_REPLAY_ESN_MAX| in xfrm.h 
>>> <https://elixir.bootlin.com/linux/latest/source/include/uapi/linux/xfrm.h>). 
>>> Wireguard uses a hardcoded value of 8192 with no option for runtime 
>>> configuration, increased from 2048 in May 2020 by this commit 
>>> <https://git.zx2c4.com/wireguard-linux/commit/drivers/net/wireguard?id=c78a0b4a78839d572d8a80f6a62221c0d7843135>."
>
> Just noting that I updated the above text to change "default maximum" 
> to "fixed maximum" as that's the hard limit. To allow for 30ms, that 
> would place a tunnel bandwidth limit at ~1.6Gbps until drops occur 
> that can't be avoided without increasing this xfrm limit.

[BB] Yes, everyone will have to keep scaling the replay window as link 
rates continue to increase. But we all know that older deployments hang 
around. That's why in the latest ecn-l4s-id draft we recommended 
splitting the traffic into two SAs, to side-step the problem altogether. 
But it's still important that implementations scale the anti-replay 
window for a reasonable degree of reordering as well.

>
> Regarding the sojourn times, Flent plots TCP RTT using periodic 
> sampled values from running the 'ss' utility, which gets it from the 
> tcpi_rtt field of the tcp_info struct. tcpi_rtt is set to 
> tcp_sock->srtt_us >> 3 in tcp.c, and srtt_us is the "smoothed round 
> trip time << 3 in usecs", according to tcp.h.

[BB] As I have said for 6 years now, by using the *smoothed* RTT, Flent 
is already filtering out important queue delay variability metrics. And, 
I've also said that, if you want the higher percentiles of delay 
measurements, *sampling* becomes increasingly inaccurate to conclude 
anything about queuing delay percentiles for any reasonable experiment 
duration. For certain you cannot say anything at all about the peak if 
you're only taking samples.

Higher percentile queuing delay is not only important for assessing 
anti-replay. It's important for real-time applications, because they 
have to choose a buffering time before play-out, given any packet that 
arrives after play-out is useless (equivalent to a loss). E.g. a 
real-time app that waits for the 99.9%-ile delay will effectively 
discard 0.1% of packets itself.

That is why, 6 years ago now, we developed a way to measure higher 
percentiles of queuing delay. An AQM measures the sojourn of every 
packet anyway, so we encoded that qDelay into the IPID field of the IP 
header (v4 only), so that it could be decoded at the receiver. We could 
have just logged the measurements at the AQM, but we wanted to have the 
AQM delay data in synch with the e2e measurement data, so we could 
verify that they match, e.g. when debugging.

The code for this is open-source, and available from 
https://github.com/L4STeam/l4sdemo .

>
> Are we sure that we have a good handle on what the *peak* differences 
> in sojourn times between C and L can be?

It's much more useful to measure high percentiles than the peak, which 
might be just a weird outlier.

You will have seen the plots we produce of the higher queuing delay 
percentiles. For instance this one is for the worst-case test we do with 
an extremely demanding traffic pattern in each queue (L and C): 300 web 
flows arriving per second as well as a long-running flow:
https://datatracker.ietf.org/meeting/106/materials/slides-106-tsvwg-sessb-31-tcp-prague-status-of-implementation-and-evaluation-00#page=3

The plot linked above shows 34ms at the 99.9%-ile. I said 30ms based on 
the 99.9%-iles of the most recent test runs we've done (we run all the 
tests after every code change).

Even measuring every packet, you can see that the plot becomes less 
regular by the 99.999%- ile, because we only run the experiments long 
enough to have about an order of magnitude more packets than the highest 
percentile we're looking for (e.g. 10^6 packets for a 5-nines 
percentile). If you were sampling, you would have to run an experiment 
for days to get reliable percentiles like this.




Bob

>
> Regards,
> Pete
>
>> Regards
>>
>>
>>
>> Bob
>>
>>
>>> Regards
>>> 	Sebastian
>>>
>>>> On May 21, 2021, at 11:21, Bob Briscoe<[e-mail]>  wrote:
>>>> Chairs, list,
>>>> We've posted a new rev of draft-ietf-tsvwg-ecn-l4s-id-17 attempting to address all the discussion since the last posting just before the interim. In particular:
>>>> * review comments on a careful read from Gorry and the chairs
>>>> * the VPN anti-replay problem
>>>> * added an out-of-band test for an RFC3168 ECN AQM in a shared queue.
>>>> There are a couple of outstanding discussions, which I'm sure will continue on the list, e.g. the role of RFC4774 and whether to remove any of Appx C. But it was considered better to get the queued up changes out, to re-base the discussions.
>>>> This is quite an extensive set of changes, so pls check and pass any comments to the list.
>>>> Thanks for everyone who is contributing, and particularly to the chairs for continuing to referee this all. We've added appropriate thanks in the Acks section.
>>>> Bob
>>>> On 21/05/2021 10:09,[e-mail]  wrote:
>>>>
>>>>> A New Internet-Draft is available from the on-line Internet-Drafts directories.
>>>>> This draft is a work item of the Transport Area Working Group WG of the IETF.
>>>>>          Title           : Explicit Congestion Notification (ECN) Protocol for Very Low Queuing Delay (L4S)
>>>>>          Authors         : Koen De Schepper
>>>>>                            Bob Briscoe
>>>>> 	Filename        : draft-ietf-tsvwg-ecn-l4s-id-17.txt
>>>>> 	Pages           : 57
>>>>> 	Date            : 2021-05-21
>>>>> Abstract:
>>>>>     This specification defines the protocol to be used for a new network
>>>>>     service called low latency, low loss and scalable throughput (L4S).
>>>>>     L4S uses an Explicit Congestion Notification (ECN) scheme at the IP
>>>>>     layer that is similar to the original (or 'Classic') ECN approach,
>>>>>     except as specified within.  L4S uses 'scalable' congestion control,
>>>>>     which induces much more frequent control signals from the network and
>>>>>     it responds to them with much more fine-grained adjustments, so that
>>>>>     very low (typically sub-millisecond on average) and consistently low
>>>>>     queuing delay becomes possible for L4S traffic without compromising
>>>>>     link utilization.  Thus even capacity-seeking (TCP-like) traffic can
>>>>>     have high bandwidth and very low delay at the same time, even during
>>>>>     periods of high traffic load.
>>>>>     The L4S identifier defined in this document distinguishes L4S from
>>>>>     'Classic' (e.g. TCP-Reno-friendly) traffic.  It gives an incremental
>>>>>     migration path so that suitably modified network bottlenecks can
>>>>>     distinguish and isolate existing traffic that still follows the
>>>>>     Classic behaviour, to prevent it degrading the low queuing delay and
>>>>>     low loss of L4S traffic.  This specification defines the rules that
>>>>>     L4S transports and network elements need to follow with the intention
>>>>>     that L4S flows neither harm each other's performance nor that of
>>>>>     Classic traffic.  Examples of new active queue management (AQM)
>>>>>     marking algorithms and examples of new transports (whether TCP-like
>>>>>     or real-time) are specified separately.
>>>>> The IETF datatracker status page for this draft is:
>>>>> https://datatracker.ietf.org/doc/draft-ietf-tsvwg-ecn-l4s-id/
>>>>> There is also an htmlized version available at:
>>>>> https://datatracker.ietf.org/doc/html/draft-ietf-tsvwg-ecn-l4s-id-17
>>>>> A diff from the previous version is available at:
>>>>> https://www.ietf.org/rfcdiff?url2=draft-ietf-tsvwg-ecn-l4s-id-17
>>>>> Internet-Drafts are also available by anonymous FTP at:
>>>>> ftp://ftp.ietf.org/internet-drafts/
>>>> -- 
>>>> ________________________________________________________________
>>>> Bob Briscoehttp://bobbriscoe.net/
>>
>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/