Re: [tsvwg] Cross-area alignment on "L4S and RACK"

[Bob Briscoe <[e-mail]>]

Michael,

That is the point I thought you were making, and my point /is/ about that.

If an application needs all the packets (i.e. a reliable protocol), it's 
not useful to reduce the latency of some packets but not all. That's 
where the in-order requirement comes from.

However, if a stream-based transport is used (i.e. TCP) the 
/application/ still gets in-order delivery from TCP. It just doesn't 
have to require links to provide in order delivery as well.

Note that there can be multiple flows over one link. So 'in-order 
delivery' for a link does not mean in order of TCP sequence number. It 
means 'delivered in the order that packets arrived at the link' (the 
link ingress adds its own sequence numbers, and the link egress buffers 
them until it can send out in the same order, or until a time-out or max 
no. of link retransmissions).

So, if the link loses a packet from flow A (the link isn't aware of 
microflows, I'm just saying the packet happens to be from flow A), 
link-layer in-order delivery will hold back all packets sent after that 
packet (from all flows, not just flow A) until the link repairs the gap 
or gives up.

So hop-by-hop in-order delivery gives every flow the same delay as the 
worst delayed flow.

Of course, someone building a network for DETNET would avoid link 
technologies like WiFi or LTE that frequently lose and retransmit 
packets. But wherever there is variable delay in a link, guaranteed 
in-order delivery per-hop means every flow is guaranteed to always get 
the worst delay from every link, which would have been experienced by 
only one flow without per-hop in-order delivery.

And many of the applications of DETNET involve dumb industrial machinery 
that just expects everything to arrive in order and to a clocked 
schedule. But it's always as effective to deploy a resequencing buffer 
as the penultimate hop, screwed onto the receiving machine if necessary, 
rather than require per-hop resequencing in the network.

________________
So, why do Internet links often ensure in-order delivery even though TCP 
puts everything in order for the application?

Short answer: the 3 DupACK rule (which is to do with loss-recovery, a 
different issue from the discussion above).

Long answer: As flow rates have scaled up, the typical time between 3 
packet arrivals has become so small that TCP's 3-DupACK rule makes links 
have to provide delivery with only a very small re-ordering degree - so 
small that it's easier for a link to just deliver everything in the 
order it received it; not because TCP doesn't put packets back in order, 
but just so as not to trigger TCP into generating spurious retransmissions.



Bob

On 08/11/2018 16:35, Scharf, Michael wrote:
>
> Inline [ms]
>
> *From:*Bob Briscoe [mailto:[e-mail]]
> *Sent:* Thursday, November 8, 2018 12:56 PM
> *To:* Scharf, Michael; [e-mail]
> *Cc:* [e-mail]
> *Subject:* Re: [tsvwg] Cross-area alignment on "L4S and RACK"
>
> Michael,
>
> This is merely a symptom of a difference in opinion on where the 
> resequencing function should be located.
>
>   * L4S takes the end-to-end approach saying that it is sufficient to
>     do resequencing in one place (the receiving host) if it is needed.
>     Then any resequencing delay only affects that flow.
>   * The DETNET approach is saying the resequencing must be done
>     hop-by-hop. This means there appears to be no resequencing needed
>     on the receiver (except if there's re-ordering on the final hop).
>
>   * In order to guarantee no resequencing in the network (DETNET
>     approach), only the the worst case latency can be guaranteed, and
>     every packet has to have that same worst-case latency.
>   * When you leave resequencing to the receiver (end-to-end approach),
>     most of the packets arrive earlier than they would with DETNET,
>     but out of order ones might not. Then the application chooses
>     whether it wants to wait.
>
> [ms] My point is different: There seem to be other working groups in 
> the IETF that do talk about “low latency” as well but apparently they 
> also do need in-order delivery of packets, too. So the assumption that 
> e.g. a link layer technology can simply disable in-order delivery for 
> all “low latency” applications seems not correct. The reality may be a 
> bit more complex.
>
> The 3 DupACK rule in TCP led the Internet not to follow the end-to-end 
> principle on re-sequencing. Now we're removing the 3 DupACK rule, we 
> can take advantage of the e2e principle.
>
> [ms] For TCP, three DupACKS are a SHOULD in RFC 5681 and RFC 5681 is 
> standards track. So the bar for “removing” that rule for TCP is not low…
>
> [ms] Also, the end-to-end principle comes with tradeoffs. For 
> instance, relying on the end-to-end principle for recovery of bit 
> errors is not efficient. For in-order delivery there are tradeoffs as 
> well, see e.g. Section 4 in draft-ietf-tcpm-rack-04. There may be no 
> free lunch.
>
> Of course, an application might choose to use TCP and L4S, rather than 
> UDP and L4S (e.g QUIC). Then there could still be HoL blocking in the 
> receiving TCP stack. But at least there's no HoL blocking in the 
> network, and the app can choose between stream (TCP) or datagram (UDP).
>
> [ms] If LRO/GRO in the receiving TCP stack gets messed up, I fail to 
> see the benefit (see draft-ietf-tcpm-rack-04).
>
> Michael
>
>
> Bob
>
> On 06/11/2018 19:20, Scharf, Michael wrote:
>
>     A comment on the TCPM presentation "L4S and RACK":
>
>     As far as I understand, the DETNET WG in RTG area has quite some uses cases for ultra-low latency transport - in particular also with bounded jitter. And some of these applications (e.g. using UDP) apparently may not be able to tolerate *any* out-of-order packet delivery.
>
>     So perhaps some cross-area alignment on ulta-low-latency application requirements would be useful?
>
>     Michael
>
>     (who recently had to review draft-ietf-detnet-architecture)
>
>
>
> -- 
> ________________________________________________________________
> Bob Briscoehttp://bobbriscoe.net/

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/