Re: [tcpm] [tsvwg] Cross-area alignment on "L4S and RACK"

Bob Briscoe <ietf@bobbriscoe.net> Fri, 09 November 2018 10:08 UTC

Return-Path: <ietf@bobbriscoe.net>
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id F018A12777C; Fri, 9 Nov 2018 02:08:40 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.99
X-Spam-Level:
X-Spam-Status: No, score=-1.99 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, T_KAM_HTML_FONT_INVALID=0.01] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MI9QnEJ_Y8Qv; Fri, 9 Nov 2018 02:08:37 -0800 (PST)
Received: from server.dnsblock1.com (server.dnsblock1.com [85.13.236.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 68A5C12007C; Fri, 9 Nov 2018 02:08:36 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=b9amIgq49JPEiP/GBOsu4T46ApgvNaIvGPl1rYhzHaQ=; b=itmkkf+uMxoq5+f36/f3ejP0b CJ5QYDz2EgnCEM25Rq7ZLkfFm42iBVKOSgpAKuMi/lId3/aDSZ6syuYf1j1ZgSd8hTxVb2x4OAba2 kdAFaYKG3Vd8wgeyL5HOEhArCNc2riHJsvIT8JvR++hy70whAgE8yztZAanFwgPkxPDohqUx+dQ43 SxbAMuXRtOrsJ6nJjeiKWOwqrNm7mQm146rVrUfFg4QMa2lEDhoVo+9hVnshc3aT++HSRHVD3afri 8VJ6SMNVl5v4Uqek1NPW6g3XpAakE9So+hgvFIRsWvnybygTOUtnihvOp6LTGcsC6kGZ2jN6svzDI lGNm951OA==;
Received: from dhcp-995f.meeting.ietf.org ([31.133.153.95]:36236) by server.dnsblock1.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.91) (envelope-from <ietf@bobbriscoe.net>) id 1gL3iJ-0007zI-PW; Fri, 09 Nov 2018 10:08:33 +0000
To: "Scharf, Michael" <Michael.Scharf@hs-esslingen.de>, "tcpm@ietf.org" <tcpm@ietf.org>
Cc: "tsvwg@ietf.org" <tsvwg@ietf.org>
References: <6EC6417807D9754DA64F3087E2E2E03E2D14F3F0@rznt8114.rznt.rzdir.fht-esslingen.de> <110030da-11f7-c9c2-c059-2abdf6178864@bobbriscoe.net> <6EC6417807D9754DA64F3087E2E2E03E2D15201A@rznt8114.rznt.rzdir.fht-esslingen.de> <1f7ccdda-0e0c-4241-3cfb-b4fe4cc00b47@bobbriscoe.net> <6EC6417807D9754DA64F3087E2E2E03E2D152905@rznt8114.rznt.rzdir.fht-esslingen.de>
From: Bob Briscoe <ietf@bobbriscoe.net>
Message-ID: <53f76b72-67e8-9c95-0ea5-a5621f378dd0@bobbriscoe.net>
Date: Fri, 9 Nov 2018 10:08:29 +0000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1
MIME-Version: 1.0
In-Reply-To: <6EC6417807D9754DA64F3087E2E2E03E2D152905@rznt8114.rznt.rzdir.fht-esslingen.de>
Content-Type: multipart/alternative; boundary="------------51A1A3204F8D85971AB682C1"
Content-Language: en-GB
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - server.dnsblock1.com
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - bobbriscoe.net
X-Get-Message-Sender-Via: server.dnsblock1.com: authenticated_id: in@bobbriscoe.net
X-Authenticated-Sender: server.dnsblock1.com: in@bobbriscoe.net
Archived-At: <https://mailarchive.ietf.org/arch/msg/tcpm/1T0qX3KGqAwNwT0wxnhPC-_Z8YM>
Subject: Re: [tcpm] [tsvwg] Cross-area alignment on "L4S and RACK"
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group <tcpm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tcpm>, <mailto:tcpm-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tcpm/>
List-Post: <mailto:tcpm@ietf.org>
List-Help: <mailto:tcpm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tcpm>, <mailto:tcpm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 09 Nov 2018 10:08:41 -0000

Michael,

On 09/11/2018 06:21, Scharf, Michael wrote:
>
> I don’t see the fundamental difference between a link technology that 
> guarantees in-order delivery within a flow (say by flow hashing, see 
> ECMP) vs. some link scheme that guarantees in-order delivery, say, 
> only for some fraction of the traffic characterized by some other 
> means (say, other header bits).
>
[The "other means" and "other bits" wording is pretty ambiguous here. I 
reckon I can guess what you mean but then I don't know why you're saying 
this, 'cos I don't think I've disagreed with you on this. So perhaps you 
could be more specific and we might be able to see why you've brought 
this up.]

Also I wasn't really talking about all "link technology that guarantees 
in-order delivery within a flow". {Note 1} I was talking about link 
technology where all flows are blocked while waiting to get the 
aggregate back in order that it was in when it arrived. I specifically said:
> Note that there can be multiple flows over one link. So 'in-order 
> delivery' for a link does not mean in order of TCP sequence number. It 
> means 'delivered in the order that packets arrived at the link' (the 
> link ingress adds its own sequence numbers, and the link egress 
> buffers them until it can send out in the same order, or until a 
> time-out or max no. of link retransmissions).
For example:

  * A packet resequencing buffer after allowing for link-layer
    retransmissions is common for radio link technologies such as LTE
    (PDCP layer) and 802.11 WiFi.
  * A packet resequencing buffer after merging multiple bonded link
    channels is common for other access link technologies such as
    downstream DOCSIS bonded channels.


> Clearly, reordering is a problem that needs to be discussed in the 
> IETF as a whole. And we could discuss whether a revision of RFC 3366 
> is needed.
>
No. There is no proposal to alter general guidance for all links like 
RFC 3366. No-one is saying this.

There is not even a proposal to require or even recommend that 
L4S-specific links do anything.

The specific proposal is to require L4S /sources/ to use a RACK-like 
scheme, as a condition for setting the ECT(1) codepoint. See the last 
bullet in S.4.3 of draft-ietf-tsvwg-ecn-l4s-id-05 
<https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05#section-4.3>;.

This says nothing about what L4S links MUST, SHOULD or MAY do. It just 
gives an L4S link an opportunity not to have to do resequencing.

> But I am not convinced that this topic has to be tied into an 
> experiment such as L4S.
>
It is an opportunity that currently only applies to L4S.

L4S is merely an example of a case where we can mandate that there will 
be no non-RACK traffic in a link, which creates an opportunity for those 
who are implementing the L4S DualQ Coupled AQM to ignore the RFC3366 
resequencing advice for the L4S queue (but we are not changing the 
advice, because L4S is only an experiment). The prize is to remove the 
head-of-line blocking problem for their users.

This is not L4S-specific, altho there are no other examples at the 
moment where this opportunity is possible (so in that sense it is 
/currently/ L4S-specific).


The message has to be handled really carefully, because the 
opportunities for misunderstanding are enormous (as evidenced by this 
thread, and we haven't even started talking to non-transport people yet).

So Gorry and David B want me to draft a short I-D that they will 
word-smith and author as tsvwg chairs to make the situation clear.

I have said I don't want this discussion to be officially raised outside 
the transport area until we are sure that a variant of RACK is even 
feasible without any packet counting at all (i.e. without the initial 
3-DupACK rule in current RACK). So I'll prioritize working that out for 
the next week or so, rather than drafting anything.



Bob

{Note 1}: I hadn't considered that this would affect technologies like 
ECMP that maintain order without a resequencing buffer and without 
introducing HoL blocking. But actually, the consequence there would be 
that L4S traffic could be sprayed by ECMP without per-flow hashing.

> Michael
>
> *From:*Bob Briscoe [mailto:in@bobbriscoe.net]
> *Sent:* Friday, November 9, 2018 6:42 AM
> *To:* Scharf, Michael; tcpm@ietf.org
> *Cc:* tsvwg@ietf.org
> *Subject:* Re: [tsvwg] Cross-area alignment on "L4S and RACK"
>
> Michael,
>
> That is the point I thought you were making, and my point /is/ about that.
>
> If an application needs all the packets (i.e. a reliable protocol), 
> it's not useful to reduce the latency of some packets but not all. 
> That's where the in-order requirement comes from.
>
> However, if a stream-based transport is used (i.e. TCP) the 
> /application/ still gets in-order delivery from TCP. It just doesn't 
> have to require links to provide in order delivery as well.
>
> Note that there can be multiple flows over one link. So 'in-order 
> delivery' for a link does not mean in order of TCP sequence number. It 
> means 'delivered in the order that packets arrived at the link' (the 
> link ingress adds its own sequence numbers, and the link egress 
> buffers them until it can send out in the same order, or until a 
> time-out or max no. of link retransmissions).
>
> So, if the link loses a packet from flow A (the link isn't aware of 
> microflows, I'm just saying the packet happens to be from flow A), 
> link-layer in-order delivery will hold back all packets sent after 
> that packet (from all flows, not just flow A) until the link repairs 
> the gap or gives up.
>
> So hop-by-hop in-order delivery gives every flow the same delay as the 
> worst delayed flow.
>
> Of course, someone building a network for DETNET would avoid link 
> technologies like WiFi or LTE that frequently lose and retransmit 
> packets. But wherever there is variable delay in a link, guaranteed 
> in-order delivery per-hop means every flow is guaranteed to always get 
> the worst delay from every link, which would have been experienced by 
> only one flow without per-hop in-order delivery.
>
> And many of the applications of DETNET involve dumb industrial 
> machinery that just expects everything to arrive in order and to a 
> clocked schedule. But it's always as effective to deploy a 
> resequencing buffer as the penultimate hop, screwed onto the receiving 
> machine if necessary, rather than require per-hop resequencing in the 
> network.
>
> ________________
> So, why do Internet links often ensure in-order delivery even though 
> TCP puts everything in order for the application?
>
> Short answer: the 3 DupACK rule (which is to do with loss-recovery, a 
> different issue from the discussion above).
>
> Long answer: As flow rates have scaled up, the typical time between 3 
> packet arrivals has become so small that TCP's 3-DupACK rule makes 
> links have to provide delivery with only a very small re-ordering 
> degree - so small that it's easier for a link to just deliver 
> everything in the order it received it; not because TCP doesn't put 
> packets back in order, but just so as not to trigger TCP into 
> generating spurious retransmissions.
>
>
>
> Bob
>
> On 08/11/2018 16:35, Scharf, Michael wrote:
>
>     Inline [ms]
>
>     *From:*Bob Briscoe [mailto:in@bobbriscoe.net]
>     *Sent:* Thursday, November 8, 2018 12:56 PM
>     *To:* Scharf, Michael; tcpm@ietf.org <mailto:tcpm@ietf.org>
>     *Cc:* tsvwg@ietf.org <mailto:tsvwg@ietf.org>
>     *Subject:* Re: [tsvwg] Cross-area alignment on "L4S and RACK"
>
>     Michael,
>
>     This is merely a symptom of a difference in opinion on where the
>     resequencing function should be located.
>
>       * L4S takes the end-to-end approach saying that it is sufficient
>         to do resequencing in one place (the receiving host) if it is
>         needed. Then any resequencing delay only affects that flow.
>       * The DETNET approach is saying the resequencing must be done
>         hop-by-hop. This means there appears to be no resequencing
>         needed on the receiver (except if there's re-ordering on the
>         final hop).
>
>       * In order to guarantee no resequencing in the network (DETNET
>         approach), only the the worst case latency can be guaranteed,
>         and every packet has to have that same worst-case latency.
>       * When you leave resequencing to the receiver (end-to-end
>         approach), most of the packets arrive earlier than they would
>         with DETNET, but out of order ones might not. Then the
>         application chooses whether it wants to wait.
>
>     [ms] My point is different: There seem to be other working groups
>     in the IETF that do talk about “low latency” as well but
>     apparently they also do need in-order delivery of packets, too. So
>     the assumption that e.g. a link layer technology can simply
>     disable in-order delivery for all “low latency” applications seems
>     not correct. The reality may be a bit more complex.
>
>     The 3 DupACK rule in TCP led the Internet not to follow the
>     end-to-end principle on re-sequencing. Now we're removing the 3
>     DupACK rule, we can take advantage of the e2e principle.
>
>     [ms] For TCP, three DupACKS are a SHOULD in RFC 5681 and RFC 5681
>     is standards track. So the bar for “removing” that rule for TCP is
>     not low…
>
>     [ms] Also, the end-to-end principle comes with tradeoffs. For
>     instance, relying on the end-to-end principle for recovery of bit
>     errors is not efficient. For in-order delivery there are tradeoffs
>     as well, see e.g. Section 4 in draft-ietf-tcpm-rack-04. There may
>     be no free lunch.
>
>     Of course, an application might choose to use TCP and L4S, rather
>     than UDP and L4S (e.g QUIC). Then there could still be HoL
>     blocking in the receiving TCP stack. But at least there's no HoL
>     blocking in the network, and the app can choose between stream
>     (TCP) or datagram (UDP).
>
>     [ms] If LRO/GRO in the receiving TCP stack gets messed up, I fail
>     to see the benefit (see draft-ietf-tcpm-rack-04).
>
>     Michael
>
>
>     Bob
>
>     On 06/11/2018 19:20, Scharf, Michael wrote:
>
>         A comment on the TCPM presentation "L4S and RACK":
>
>           
>
>         As far as I understand, the DETNET WG in RTG area has quite some uses cases for ultra-low latency transport - in particular also with bounded jitter. And some of these applications (e.g. using UDP) apparently may not be able to tolerate *any* out-of-order packet delivery.
>
>           
>
>         So perhaps some cross-area alignment on ulta-low-latency application requirements would be useful?
>
>           
>
>         Michael
>
>         (who recently had to review draft-ietf-detnet-architecture)
>
>           
>
>           
>
>
>
>
>     -- 
>
>     ________________________________________________________________
>
>     Bob Briscoehttp://bobbriscoe.net/
>
>
>
> -- 
> ________________________________________________________________
> Bob Briscoehttp://bobbriscoe.net/

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/