Re: [tcpm] [tsvwg] Cross-area alignment on "L4S and RACK"

[Bob Briscoe <[e-mail]>]

Michael,

On 09/11/2018 18:03, Bob Briscoe wrote:
> Michael,
>
> Please address the scaling rationale in the appendix for why it does 
> matter how a sender realizes this internally.
>
> The list of access links types I mentioned are rarely connected 
> directly to routers.
Correction to my own statement: Only LTE links are rarely connected to 
routers , WiFi are, and DOCSIS are (the CMTS).

The radio link (ARQ) issue is not to do with routing.
One can think of bonded links as a form of L2 multipath, and therefore 
broadly related to routing.
But it's access link expertise that's needed here, not routing 
expertise, I would say.


Bob

>
>
>
> Bob
>
> On 09/11/2018 17:44, Scharf, Michael wrote:
>>
>> I’ll not comment on link technology implementation, scheduler and 
>> queue design assumptions in Appendix A.1.7 of 
>> draft-ietf-tsvwg-ecn-l4s-id-05. People e.g. in RTG area may be more 
>> familiar with the implications of internal designs e.g. inside a 
>> router. IMHO the best way to get routing experts into the loop before 
>> further discussing what is currently listed in Appendix A.1.7.
>>
>> Thus, I limit my response to the actual wording in the main part of 
>> draft-ietf-tsvwg-ecn-l4s-id-05:
>>
>>     o  A scalable congestion control MUST detect loss by counting in
>>
>>       units of time, which is scalable, and MUST NOT count in units of
>>
>>       packets (as in the 3 DupACK rule of traditional TCP), which is not
>>
>>       scalable (see Appendix A.1.7 
>> <https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05#appendix-A.1.7>for 
>> rationale).
>>
>> As individual contributor, I disagree with this requirement. I would 
>> probably be OK with a wording that a TCP sender must be able to 
>> tolerate reordering beyond 3 Duplicate ACKs. I believe it does not 
>> matter how a sender realizes this internally.
>>
>> And if that requirement is not just removed, I believe the wording 
>> “traditional TCP” has to be replaced by “standard TCP”.
>>
>> BTW, also, in the rest of the document there has to be a clear 
>> distinction between experiments and standards. History tells us that 
>> some experiments can evolve into standards track specifications, 
>> while other experiments may fail. And there is no way of knowing that 
>> in advance.
>>
>> Michael
>>
>> *From:*Bob Briscoe [mailto:[e-mail]]
>> *Sent:* Friday, November 9, 2018 11:08 AM
>> *To:* Scharf, Michael; [e-mail]
>> *Cc:* [e-mail]
>> *Subject:* Re: [tsvwg] Cross-area alignment on "L4S and RACK"
>>
>> Michael,
>>
>> On 09/11/2018 06:21, Scharf, Michael wrote:
>>
>>     I don’t see the fundamental difference between a link technology
>>     that guarantees in-order delivery within a flow (say by flow
>>     hashing, see ECMP) vs. some link scheme that guarantees in-order
>>     delivery, say, only for some fraction of the traffic
>>     characterized by some other means (say, other header bits).
>>
>> [The "other means" and "other bits" wording is pretty ambiguous here. 
>> I reckon I can guess what you mean but then I don't know why you're 
>> saying this, 'cos I don't think I've disagreed with you on this. So 
>> perhaps you could be more specific and we might be able to see why 
>> you've brought this up.]
>>
>> Also I wasn't really talking about all "link technology that 
>> guarantees in-order delivery within a flow". {Note 1} I was talking 
>> about link technology where all flows are blocked while waiting to 
>> get the aggregate back in order that it was in when it arrived. I 
>> specifically said:
>>
>> Note that there can be multiple flows over one link. So 'in-order 
>> delivery' for a link does not mean in order of TCP sequence number. 
>> It means 'delivered in the order that packets arrived at the link' 
>> (the link ingress adds its own sequence numbers, and the link egress 
>> buffers them until it can send out in the same order, or until a 
>> time-out or max no. of link retransmissions).
>>
>> For example:
>>
>>   * A packet resequencing buffer after allowing for link-layer
>>     retransmissions is common for radio link technologies such as LTE
>>     (PDCP layer) and 802.11 WiFi.
>>   * A packet resequencing buffer after merging multiple bonded link
>>     channels is common for other access link technologies such as
>>     downstream DOCSIS bonded channels.
>>
>>
>>
>> Clearly, reordering is a problem that needs to be discussed in the 
>> IETF as a whole. And we could discuss whether a revision of RFC 3366 
>> is needed.
>>
>> No. There is no proposal to alter general guidance for all links like 
>> RFC 3366. No-one is saying this.
>>
>> There is not even a proposal to require or even recommend that 
>> L4S-specific links do anything.
>>
>> The specific proposal is to require L4S /sources/ to use a RACK-like 
>> scheme, as a condition for setting the ECT(1) codepoint. See the last 
>> bullet in S.4.3 of draft-ietf-tsvwg-ecn-l4s-id-05 
>> <https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05#section-4.3>. 
>>
>>
>> This says nothing about what L4S links MUST, SHOULD or MAY do. It 
>> just gives an L4S link an opportunity not to have to do resequencing.
>>
>>
>> But I am not convinced that this topic has to be tied into an 
>> experiment such as L4S.
>>
>> It is an opportunity that currently only applies to L4S.
>>
>> L4S is merely an example of a case where we can mandate that there 
>> will be no non-RACK traffic in a link, which creates an opportunity 
>> for those who are implementing the L4S DualQ Coupled AQM to ignore 
>> the RFC3366 resequencing advice for the L4S queue (but we are not 
>> changing the advice, because L4S is only an experiment). The prize is 
>> to remove the head-of-line blocking problem for their users.
>>
>> This is not L4S-specific, altho there are no other examples at the 
>> moment where this opportunity is possible (so in that sense it is 
>> /currently/ L4S-specific).
>>
>>
>> The message has to be handled really carefully, because the 
>> opportunities for misunderstanding are enormous (as evidenced by this 
>> thread, and we haven't even started talking to non-transport people yet).
>>
>> So Gorry and David B want me to draft a short I-D that they will 
>> word-smith and author as tsvwg chairs to make the situation clear.
>>
>> I have said I don't want this discussion to be officially raised 
>> outside the transport area until we are sure that a variant of RACK 
>> is even feasible without any packet counting at all (i.e. without the 
>> initial 3-DupACK rule in current RACK). So I'll prioritize working 
>> that out for the next week or so, rather than drafting anything.
>>
>>
>>
>> Bob
>>
>> {Note 1}: I hadn't considered that this would affect technologies 
>> like ECMP that maintain order without a resequencing buffer and 
>> without introducing HoL blocking. But actually, the consequence there 
>> would be that L4S traffic could be sprayed by ECMP without per-flow 
>> hashing.
>>
>>
>> Michael
>>
>> *From:*Bob Briscoe [mailto:[e-mail]]
>> *Sent:* Friday, November 9, 2018 6:42 AM
>> *To:* Scharf, Michael; [e-mail] <mailto:[e-mail]>
>> *Cc:* [e-mail] <mailto:[e-mail]>
>> *Subject:* Re: [tsvwg] Cross-area alignment on "L4S and RACK"
>>
>> Michael,
>>
>> That is the point I thought you were making, and my point /is/ about 
>> that.
>>
>> If an application needs all the packets (i.e. a reliable protocol), 
>> it's not useful to reduce the latency of some packets but not all. 
>> That's where the in-order requirement comes from.
>>
>> However, if a stream-based transport is used (i.e. TCP) the 
>> /application/ still gets in-order delivery from TCP. It just doesn't 
>> have to require links to provide in order delivery as well.
>>
>> Note that there can be multiple flows over one link. So 'in-order 
>> delivery' for a link does not mean in order of TCP sequence number. 
>> It means 'delivered in the order that packets arrived at the link' 
>> (the link ingress adds its own sequence numbers, and the link egress 
>> buffers them until it can send out in the same order, or until a 
>> time-out or max no. of link retransmissions).
>>
>> So, if the link loses a packet from flow A (the link isn't aware of 
>> microflows, I'm just saying the packet happens to be from flow A), 
>> link-layer in-order delivery will hold back all packets sent after 
>> that packet (from all flows, not just flow A) until the link repairs 
>> the gap or gives up.
>>
>> So hop-by-hop in-order delivery gives every flow the same delay as 
>> the worst delayed flow.
>>
>> Of course, someone building a network for DETNET would avoid link 
>> technologies like WiFi or LTE that frequently lose and retransmit 
>> packets. But wherever there is variable delay in a link, guaranteed 
>> in-order delivery per-hop means every flow is guaranteed to always 
>> get the worst delay from every link, which would have been 
>> experienced by only one flow without per-hop in-order delivery.
>>
>> And many of the applications of DETNET involve dumb industrial 
>> machinery that just expects everything to arrive in order and to a 
>> clocked schedule. But it's always as effective to deploy a 
>> resequencing buffer as the penultimate hop, screwed onto the 
>> receiving machine if necessary, rather than require per-hop 
>> resequencing in the network.
>>
>> ________________
>> So, why do Internet links often ensure in-order delivery even though 
>> TCP puts everything in order for the application?
>>
>> Short answer: the 3 DupACK rule (which is to do with loss-recovery, a 
>> different issue from the discussion above).
>>
>> Long answer: As flow rates have scaled up, the typical time between 3 
>> packet arrivals has become so small that TCP's 3-DupACK rule makes 
>> links have to provide delivery with only a very small re-ordering 
>> degree - so small that it's easier for a link to just deliver 
>> everything in the order it received it; not because TCP doesn't put 
>> packets back in order, but just so as not to trigger TCP into 
>> generating spurious retransmissions.
>>
>>
>>
>> Bob
>>
>> On 08/11/2018 16:35, Scharf, Michael wrote:
>>
>>     Inline [ms]
>>
>>     *From:*Bob Briscoe [mailto:[e-mail]]
>>     *Sent:* Thursday, November 8, 2018 12:56 PM
>>     *To:* Scharf, Michael; [e-mail] <mailto:[e-mail]>
>>     *Cc:* [e-mail] <mailto:[e-mail]>
>>     *Subject:* Re: [tsvwg] Cross-area alignment on "L4S and RACK"
>>
>>     Michael,
>>
>>     This is merely a symptom of a difference in opinion on where the
>>     resequencing function should be located.
>>
>>       * L4S takes the end-to-end approach saying that it is
>>         sufficient to do resequencing in one place (the receiving
>>         host) if it is needed. Then any resequencing delay only
>>         affects that flow.
>>       * The DETNET approach is saying the resequencing must be done
>>         hop-by-hop. This means there appears to be no resequencing
>>         needed on the receiver (except if there's re-ordering on the
>>         final hop).
>>
>>       * In order to guarantee no resequencing in the network (DETNET
>>         approach), only the the worst case latency can be guaranteed,
>>         and every packet has to have that same worst-case latency.
>>       * When you leave resequencing to the receiver (end-to-end
>>         approach), most of the packets arrive earlier than they would
>>         with DETNET, but out of order ones might not. Then the
>>         application chooses whether it wants to wait.
>>
>>     [ms] My point is different: There seem to be other working groups
>>     in the IETF that do talk about “low latency” as well but
>>     apparently they also do need in-order delivery of packets, too.
>>     So the assumption that e.g. a link layer technology can simply
>>     disable in-order delivery for all “low latency” applications
>>     seems not correct. The reality may be a bit more complex.
>>
>>     The 3 DupACK rule in TCP led the Internet not to follow the
>>     end-to-end principle on re-sequencing. Now we're removing the 3
>>     DupACK rule, we can take advantage of the e2e principle.
>>
>>     [ms] For TCP, three DupACKS are a SHOULD in RFC 5681 and RFC 5681
>>     is standards track. So the bar for “removing” that rule for TCP
>>     is not low…
>>
>>     [ms] Also, the end-to-end principle comes with tradeoffs. For
>>     instance, relying on the end-to-end principle for recovery of bit
>>     errors is not efficient. For in-order delivery there are
>>     tradeoffs as well, see e.g. Section 4 in draft-ietf-tcpm-rack-04.
>>     There may be no free lunch.
>>
>>     Of course, an application might choose to use TCP and L4S, rather
>>     than UDP and L4S (e.g QUIC). Then there could still be HoL
>>     blocking in the receiving TCP stack. But at least there's no HoL
>>     blocking in the network, and the app can choose between stream
>>     (TCP) or datagram (UDP).
>>
>>     [ms] If LRO/GRO in the receiving TCP stack gets messed up, I fail
>>     to see the benefit (see draft-ietf-tcpm-rack-04).
>>
>>     Michael
>>
>>
>>     Bob
>>
>>     On 06/11/2018 19:20, Scharf, Michael wrote:
>>
>>         A comment on the TCPM presentation "L4S and RACK":
>>
>>           
>>
>>         As far as I understand, the DETNET WG in RTG area has quite some uses cases for ultra-low latency transport - in particular also with bounded jitter. And some of these applications (e.g. using UDP) apparently may not be able to tolerate *any* out-of-order packet delivery.
>>
>>           
>>
>>         So perhaps some cross-area alignment on ulta-low-latency application requirements would be useful?
>>
>>           
>>
>>         Michael
>>
>>         (who recently had to review draft-ietf-detnet-architecture)
>>
>>           
>>
>>           
>>
>>
>>
>>
>>
>>     -- 
>>
>>     ________________________________________________________________
>>
>>     Bob Briscoehttp://bobbriscoe.net/
>>
>>
>>
>>
>> -- 
>> ________________________________________________________________
>> Bob Briscoehttp://bobbriscoe.net/
>>
>>
>>
>> -- 
>> ________________________________________________________________
>> Bob Briscoehttp://bobbriscoe.net/
>
> -- 
> ________________________________________________________________
> Bob Briscoehttp://bobbriscoe.net/

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/