Re: [tsvwg] L4S issue #22: CE Ambiguity and Reordering

Luca Muscariello <> Tue, 25 February 2020 11:36 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 22F603A0A1B for <>; Tue, 25 Feb 2020 03:36:39 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.099
X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (1024-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 8EpjO5bZVadY for <>; Tue, 25 Feb 2020 03:36:36 -0800 (PST)
Received: from ( [IPv6:2a00:1450:4864:20::331]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 3458D3A0A17 for <>; Tue, 25 Feb 2020 03:36:36 -0800 (PST)
Received: by with SMTP id c84so2782161wme.4 for <>; Tue, 25 Feb 2020 03:36:36 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=qU9vtlAVMuV58M7+wlg1p9+ByfnX/F8l0sUzdvdkxqM=; b=QoFzdXOgKAi5fqstNdyNfa9Efok9OVfJvYJhJp4QaLzijSuTz9GwypdI/bJT3Z8Q0o dAGvYOLj8lTge7mdNhjeauijEobkHKGwrTSaIKzUKHBMekL46GS8SulmY7EyU6nmco3y 01ZrG+4qEu9nhsSBGoEx+C1ZXVQb7EJ7Ki7KY=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=qU9vtlAVMuV58M7+wlg1p9+ByfnX/F8l0sUzdvdkxqM=; b=rCcNLX7K1FfBV3BEp05/Ivdx3r/x3c7m7MTDI9R020TC11r+eNFVLhjpPOQRik6U0k qFKFAq18kZmb3DaQG3gK6phPpzZZwY4b1s+oJUPtbPdTHLE9bnWs8YUMiT4MbKvLO8Tx y5TzMrordV5hnUdmXljYVHqbm68DVK4Ivw2sIq7ExS1Ezc+dNfclBhTl5YLlsw3Que+J VC0n9F0d8vDwGCOwoewb7HUpO3G8q0gQjUqCGcIkGxBDpZW5jBaDLFPXdxjQMnCu0pya J62xNo3cRs9Nah7dLJx4m9dABxsLcRQikndiBhegpGxrzJHiBlsrFlYWOMiqo4Yt98cu h/iQ==
X-Gm-Message-State: APjAAAUx5QMkfWDWcM9aFHKkuo8IUrPECDwQFtyK2KcbEwrJFCZZ48Oa kLVtQd9LqTCsBOeCaGBQoPffLzDgA7Ao9ebdN9JuRA==
X-Google-Smtp-Source: APXvYqyngPuiTXwKpmzr65gGEym0UwQDWl6nYLrwzuC80J4fbIz7iKtNjlOtkvgqotkCZ9B4h5L0xKfZjSEAKIMUjjc=
X-Received: by 2002:a1c:6408:: with SMTP id y8mr4844126wmb.130.1582630593643; Tue, 25 Feb 2020 03:36:33 -0800 (PST)
MIME-Version: 1.0
References: <> <> <> <>
In-Reply-To: <>
From: Luca Muscariello <>
Date: Tue, 25 Feb 2020 12:36:22 +0100
Message-ID: <>
To: Jonathan Morton <>
Cc: "Holland, Jake" <>, tsvwg IETF list <>
Content-Type: multipart/alternative; boundary="000000000000586603059f64e63a"
Archived-At: <>
Subject: Re: [tsvwg] L4S issue #22: CE Ambiguity and Reordering
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Transport Area Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Tue, 25 Feb 2020 11:36:39 -0000

I think issue #22 ( is more than
just out of order.
It is related to issue #28 and #26.

I did not follow the WebEx recording (not yet shared I think) so I do not
know what
has been discussed during the virtual meeting. Please share the recording.

Out of order is an effect of mis-classification that can happen frequently
in presence of
unresponsive traffic in either the priority or non-priority queues (see
issue #26).
Depending on the rate of unresponsive traffic, not only average rate but
instantaneous rate,
out-of-order events can oscillate frequently. I still have not found any
reference that
proves this does not happen. I tried to raise this point in the past but
it's been ignored
or dismissed by saying that unresponsive traffic is negligible. I disagree.

There are several potential issues coming from that phenomenon. One in
particular is that (issue #28)
the overall system will not be able to guarantee the latency targets at

It looks like most of the discussion on this issue is about how this
affects TCP, but the applications
that would mostly benefit from the priority queue are not really using TCP.
So out-of-order/mis-classification for real-time traffic would be worse.

I think that this issue is also related to how much dualQ can provide
predictable performance
under a general traffic mix which constitutes today and future Internet

The problem of predictability looks fundamental here, even to deploy an
The risk is that we build something to satisfy customers' low latency
request that won't meet the
expectation. Building dualQ has a cost and we better know what we get out
of it before building it
in silicon (issues #22 #26 #28 at least).


On Mon, Feb 24, 2020 at 6:12 PM Jonathan Morton <>

> > On 24 Feb, 2020, at 5:15 pm, Holland, Jake <jholland=
>> wrote:
> >
> > What I was aiming toward when asking the question originally was that if
> there’s something that already exists and could be cited that demonstrates
> the scope of the problem justifying a normative SHOULD for RACK-like
> behavior, it would be beneficial to include it.  If there is no such source
> to cite, I didn’t mean to say it’s necessary to create one before moving
> forward.
> >
> > To me the reordering issue itself seems so minor that I’m confused why
> there’s so much text and meeting time discussion about it.  I was worried I
> was missing something.  If the whole issue is addressed by the appendix
> text, it seems like even the “SHOULD” for RACK-like behavior from the
> transport is not justified, since there’s basically no harm even without it?
> My understanding here is that the root of this issue is that CE-marked
> packets originating upstream of the DualQ may be classified into a
> different queue than the rest of their flow, and may therefore be delivered
> out of order.  DualQ treats CE packets as always belonging to the L4S
> queue, so this issue affects Classic flows only.
> The maximum expected quantity of reordering is easiest to define in time
> units; the CE packet would be about 0.5ms late if the L4S queue was at its
> target depth while the Classic queue was empty, while it would be 15ms
> early if the reverse were true.  The behaviour of a RACK-type transport can
> be directly predicted based on those values, while predicting the behaviour
> of a traditional stack may require those times to be converted into packet
> serialisation times by way of the flow's throughput.
> The "RACK requirement" applies only to L4S flows, so I believe it is not
> relevant to resolving this.  Classic flows will continue to use a mixture
> of RACK (eg. Linux) and traditional (eg. default FreeBSD at this time)
> stacks.  It is the latter that are most likely to be affected, so testing
> (and thought experiments) should focus there.
>  - Jonathan Morton