Re: [tsvwg] Robustness to packet reordering

[Sebastian Moeller <[e-mail]>]

> On 8. Feb 2025, at 11:38, Michael Eriksson <michael.eriksson=[e-mail]> wrote:
> 
> Well, QUIC receivers can easily handle an almost unlimited amount of reordering.

Quick note, if we are not talking about multiplexing within one 5 tuple*, then I believe there is quite some use cases where the data is only usable if all segments have been received. So out of order delivery might not actually give any big advantage in application latency. E.g. if we consider the case with a slow bottleneck (e.g. WiFi) in an otherwise fast path, there the resequencing will mostly result in creating line speed bursts from the receiver side of the bottleneck, if the ratio between bottleneck rate and rate on the reminder of the path the latency gained for doing out of order delivery instead of re-sequencing is rather mild.

*) I understand the rationale for adding multiplexing to (each) protocol layer, but occasionally maybe using multiplexing via the lower layer's mechanism is a viable option...


> The problem is the senders, and in particular the loss detection and congestion controller. As has already been mentioned in the thread, the default packet reordering threshold (kPacketThreshold) is only three packets.
> 
> It is of course possible to adapt the threshold dynamically as the RFC mentions, and with Ack Frequency you can also reduce the number of (spurious) acks.
> 
> Until it is verified that "all" production QUIC stacks implement dynamic packet reordering thresholds, it seems unwise to completely turn off L2 in-order delivery.

I note that endpoints (EP) and intermediate hops (IH) have different desires in regards to reordering... EPs really want as little reordering as possible even if they can deal with it (as resequencing will translate into latency), while IHs want as much freedom as possible and as little work*.
So IMHO if the IETF truly feels that more re-ordering is tolerable we should write an explicit RFC giving guidance how much reordering is deemed acceptable, "completely turn off L2 in-order delivery" seems not in the interest of end points... I guess giving guidance in units of time might be helpful unless tied to something an IH might not be privy to like path-RTT/OWD.

*) This is why IHs typically do not seem to restrict reordering to flows that actually are affected by a link layer packet loss but apply it unconditionally to all traffic. Getting information about which packet aggregates are fate-sharing and hence would gain something from re-seqiencing can be computationally costly to impossible.

Sebastian

> 
> /me
> 
> From: David Schinazi <[e-mail]>
> Subject: [tsvwg] Re: Robustness to packet reordering
> Date: Fri, 7 Feb 2025 00:14:28 +0100 (Central European Standard Time)
> 
>> Having the link-layer delay packets to provide them in-order to the transport layer was helpful multiple decades ago when TCP implementations had more naive algorithms and were very resource-constrained. Given how modern implementations work, reordering at the link-layer is almost always harmful. TCP and QUIC stacks can handle reordering well, thanks to both protocol features and implementations having more memory to work with. The delay induced by this reordering will generally cause more harm than good. Furthermore, one of the biggest motivators for QUIC was to break head-of-line blocking and allow out-of-order delivery to the application layer. We have large bodies of data showing that this improves performance. Please disable reordering at the 5G layer. 
>> 
>> David
>> 
>> On Thu, Feb 6, 2025 at 2:15 PM Christian Huitema <[e-mail]> wrote:
>> 
>> On 2/6/2025 12:55 PM, Martin Thomson wrote:
>> >
>> > On Fri, Feb 7, 2025, at 04:59, Greg White wrote:
>> >> This is an important topic relating to the expectations and
>> >> requirements that transport protocols place on layer 2 protocols.  In
>> >> layer 2 standards bodies that I've been involved in, it has been
>> >> understood that "the upper layers" expect in-order delivery,
>> > As far as QUIC goes, it is sensitive to reordering in the network.  Some reordering will be interpreted as damage (Christian cited the relevant parts) and performance suffers in a few minor way when things arrive out of order (ACKs are less efficient, data needs to be held, memory accesses are less likely to be contiguous, etc...).
>> >
>> > However, the idea that the network might seek to "fix" these problems, when doing so necessarily involves extra work and delays, is not a good trade.  Stuff that is delayed to "fix" a reordering that happened might delay signals that the QUIC stack could use, even if some data needs to be held at the endpoint.  QUIC packets contain many things, some of which don't need to be strictly ordered to be useful.
>> 
>> Applications that are sensitive to delays will break their traffic into 
>> multiple QUIC streams. In case of packet loss, only one of those streams 
>> will be blocked, the others will be delivered without "head of queue 
>> blocking". Implementing L2 correction will make the response worse, not 
>> better, because all the streams will be delayed for the duration of the 
>> L2 correction instead of just one.
>> 
>> -- Christian Huitema
>> 
> 
>