Re: [tsvwg] Reasons for WGLC/RFC asap

Pete Heist <pete@heistp.net> Thu, 19 November 2020 20:03 UTC

Return-Path: <pete@heistp.net>
X-Original-To: tsvwg@ietfa.amsl.com
Delivered-To: tsvwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 919473A10F2 for <tsvwg@ietfa.amsl.com>; Thu, 19 Nov 2020 12:03:00 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.098
X-Spam-Level:
X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=heistp.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id D_pcPKrRCF3s for <tsvwg@ietfa.amsl.com>; Thu, 19 Nov 2020 12:02:58 -0800 (PST)
Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 9051B3A1106 for <tsvwg@ietf.org>; Thu, 19 Nov 2020 12:02:44 -0800 (PST)
Received: by mail-wm1-x335.google.com with SMTP id c198so6440112wmd.0 for <tsvwg@ietf.org>; Thu, 19 Nov 2020 12:02:44 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heistp.net; s=google; h=message-id:subject:from:to:cc:date:in-reply-to:references :user-agent:mime-version; bh=8yKoY5kDx4OA/2mGceZCp1RhWbp+F9C1ALLOC7afwW8=; b=ItkL1j8PoAmtkn9NHaPupDNuDoqGmgJBJUFRxG8nh2DQoCmMjhJ2cGb4dNSqbKXnzS OwXYnvk3zpbaVtBvsjmwnVL7LbXnnK3cVb9N3hXvZqWThCfKnW9NW02O+l2QNvteuMI6 ul+1gS+K+PvaLQuTBpTbsvF6nOLWyXbjdsJFZav2HEeR2h+i3rPg7hR3J+HeJ6DT1N07 HZBx7EUU/u4B5lWd6QBOlVqBmvmvIa4Qwr90lF+mEF5N76VDJvC9a10X3uhvXpr9swi9 oG6YSoo5aoRZJF54iEddrjGcrWSAoiPLh1roNXRhaEACjXFFNTGBbKZF2EtfovWN0+Vl p9MA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:user-agent:mime-version; bh=8yKoY5kDx4OA/2mGceZCp1RhWbp+F9C1ALLOC7afwW8=; b=IDTKL29nwhTZ7AcinYEg1NaU+jmnJvxFbTHc4mwbb/ekii7WRVxvmRVDUm2Fuj6idm iVVw56dHY8R3wzEn6psc8yUq5XYQIMswyGJ9aomxK6XL6FlvhhgbaCNeBVCCAcd2Atl/ TkZDADn8k7xn16gNTjOHHOISyiWiZtxoVdV/5RFu5On3/rv7+MyjjEB0W4GP6r825yFa WsERH552BOBTQ+QZsu+XQdKbtN3c6u6E5LXRFYaDIAzogC9woQq4L4dhw8MTvHZge7Wd Ys8b4K/LoqHT7jcBP4FIb4BHovtPHCKF9SZceGnh8q7jsKJWuNe1HbQQ4rZ0XJaKiZXc 6PMw==
X-Gm-Message-State: AOAM53364jplv1cy1NaV0fIOHvtufd/WGoKP+XnhRIg6RUXb1+0ebNf9 eL/+YDucqYBOb0ux5Og9fw1gYHJKw8QDMg==
X-Google-Smtp-Source: ABdhPJyW7X9oiRSaD7zfwZm+Tc/qZ+b6cvE6piPhelwTo0tMseSVDScfWR0cP1RD9NqoifxIXK7GQA==
X-Received: by 2002:a1c:66d5:: with SMTP id a204mr5886349wmc.184.1605816162852; Thu, 19 Nov 2020 12:02:42 -0800 (PST)
Received: from [10.72.0.88] (h-1169.lbcfree.net. [185.193.85.130]) by smtp.gmail.com with ESMTPSA id n67sm1463476wmf.25.2020.11.19.12.02.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 Nov 2020 12:02:42 -0800 (PST)
Message-ID: <559f0accc589e902590f17d78dde719802d5dc96.camel@heistp.net>
From: Pete Heist <pete@heistp.net>
To: Gorry Fairhurst <gorry@erg.abdn.ac.uk>
Cc: tsvwg IETF list <tsvwg@ietf.org>
Date: Thu, 19 Nov 2020 21:02:39 +0100
In-Reply-To: <b1d743be-4df0-c9c4-2839-3156df509628@erg.abdn.ac.uk>
References: <AM8PR07MB747626CB7622CB89209018A8B9E10@AM8PR07MB7476.eurprd07.prod.outlook.com> <507f049aba1d228cb122545702f1ab9e51467745.camel@heistp.net> <b1d743be-4df0-c9c4-2839-3156df509628@erg.abdn.ac.uk>
Content-Type: multipart/alternative; boundary="=-008kjhdQK/TZHDE33HL3"
User-Agent: Evolution 3.38.1
MIME-Version: 1.0
Archived-At: <https://mailarchive.ietf.org/arch/msg/tsvwg/U6Xijm3CAefJaDyiGcXOFc17gQM>
Subject: Re: [tsvwg] Reasons for WGLC/RFC asap
X-BeenThere: tsvwg@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Transport Area Working Group <tsvwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tsvwg>, <mailto:tsvwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tsvwg/>
List-Post: <mailto:tsvwg@ietf.org>
List-Help: <mailto:tsvwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tsvwg>, <mailto:tsvwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 19 Nov 2020 20:03:04 -0000

On Thu, 2020-11-19 at 16:34 +0000, Gorry Fairhurst wrote:
> 
> On 19/11/2020 16:22, Pete Heist wrote:
> 
> > 
> > Hi Koen,
> > 
> > 
> > 
> > Rather than thinking of this as advantages and disadvantages to
> > waiting, I see it as an engineering process. It was decided earlier
> > this year that the L4S proposal has enough support to continue, so
> > we're on that path now. Part of that decision, as I understood it,
> > also recognized that there are valid safety concerns around
> > compatibility with existing AQMs, and some solution needs to be
> > devised.
> > 
> > 
> > 
> > RFC3168 bottleneck detection was added to TCP Prague, which appears
> > to be difficult to do reliably when there is jitter or cross-flow
> > traffic, and it has since been disabled in the reference
> > implementation. The l4s-ops draft was started, but isn't complete
> > yet and may need WG adoption as part of a LC. We can then decide
> > how effective the proposed mitigations are against the risks and
> > prevalence.
> > 
> > 
> > 
> > To start a WGLC now would circumvent that earlier recognition that
> > a safety case needs to be made. Meanwhile, since testing showed
> > that tunnels through RFC3168 FQ AQMs are a straightforward path to
> > unsafe flow interaction, along with other issues relative to the
> > goals, it doesn't seem like the engineering process is done just
> > yet.
> > 
> > 
> > 
> 
> By the way, I liked your data - and it helped me a lot to look at
> this, thanks very much for doing this.
I'm glad, as I think we're at our best when we're doing engineering and
producing data. I wish it were easier to do!
> 
> It would help me if you clarify what you mean by  "unsafe" - to me
> "safety" relates to traffic unresponsive to drop, as in CBR traffic,
> etc. I've not understood how CE-marked traffic can erode safety, but
> maybe I missed something?
Sure, so the existing RFC3168 CE signal in use on the Internet today
indicates an MD (multiplicative decrease), whereas the redefined CE
signal in L4S indicates an AD (additive decrease). Two congestion
controls responding to CE in a different way, or one that responds to
CE with an AD and one that responds only to drop (i.e. all standard
congestion controls that advertise Not-ECT), will not interact safely
in the same RFC3168 signaling queue. We're probably on the same page
here already, but I'll refer to section 5 of RFC8257.

That is one of the reasons why ECT(1) is used in L4S to place L4S flows
in the L queue- to keep them separate from conventional flows in the C
queue. As long as flows have advertised their capability correctly,
that works.

However, existing RFC3168 queues do not have knowledge of L4S,
therefore will not know that ECT(1) means that traffic needs to be
segregated and signaled in a different way. They will signal a Prague
flow, which sets ECT(1), with CE, expecting the flow to respond with an
MD, rather than AD. Meanwhile they'll signal an RFC3168 or non-ECN flow
with either CE or drop, and in either case the flow will respond with
an MD, causing conventional flows to yield to Prague flows to varying
degrees depending on the AQM in use.

Here's an example of CUBIC and Prague when they end up in the same
fq_codel queue:
http://sce.dnsmgr.net/results/l4s-2020-11-11T120000-final/l4s-s6-rfc3168-1q/l4s-s6-rfc3168-1q/l4s-s6-rfc3168-1q-ns-prague-vs-cubic-fq_codel-50Mbit-20ms_tcp_delivery_with_rtt.svg

Here's a more extreme example of Reno and Prague sharing a single PIE
queue with ECN enabled (less common):
http://sce.dnsmgr.net/results/l4s-2020-11-11T120000-final/l4s-s6-rfc3168-1q/l4s-s6-rfc3168-1q/l4s-s6-rfc3168-1q-ns-prague-vs-reno-pie-50Mbit-20ms_tcp_delivery_with_rtt.svg

In the example with PIE, Reno appears to be driven at or close to
minimum cwnd. In the fq_codel example, the steady state throughput of
Prague:CUBIC is around 19:1. We've seen a range in the Codel case from
around 12:1 to 20:1. In my opinion, we could use the word "unsafe" here
in both cases.
> I'm not sure why "tunnels have crept in here. There have always been
> side-effects with classification (and hence scheduling), but I don't
> see new issues relating to "tunnels" with ECN.
Tunnels are relevant because they provide an easy practical path to the
unsafe flow interaction described above. The widely used fq_codel qdisc
has ECN enabled by default. Fortunately, because it has flow-fair
queueing, Prague flows and conventional flows are usually placed in a
separate queue (hash collisions aside), causing Prague to only affect
itself with additional delay (TCP RTT). However, a tunnel's
encapsulated packets all share the same fq_codel queue because they all
have the same 5-tuple, so there is unsafe interaction between the
tunnel's flows. Here we use Wireguard through fq_codel:

http://sce.dnsmgr.net/results/l4s-2020-11-11T120000-final/l4s-s5-tunnel/l4s-s5-tunnel-phys-wireguard-prague-vs-cubic-fq_codel-50Mbit-20ms_tcp_delivery_with_rtt.svg

I'll leave it to the WG to come up with examples of what types of
tunnels and traffic scenarios could lead to this, but one example is a
user who has a privacy VPN on their PC, and fq_codel on their home
gateway. Let's say one flow connects to an L4S capable server, and
another flow to a non-L4S, conventional server. The L4S flow will
dominate the non-L4S one (whether it's ECN capable or not), probably
causing some level of poor service, perhaps for a video stream,
download, or whatever.
> I'm not commenting on when the Chairs think a WGLC will provide
> useful information, we'll say that in due course.
Ok, I trust that we'll engage enough disinterested people into
congestion control who will add their input.

Thanks Gorry for looking this over. :)
> 
> Best wishes,
> 
> Gorry
> 
> > 
> > Regards,
> > 
> > Pete
> > 
> > 
> > 
> > On Wed, 2020-11-18 at 10:31 +0000, De Schepper, Koen (Nokia -
> > BE/Antwerp) wrote:
> > 
> > > 
> > > Hi all,
> > >  
> > > To continue on the discussions in the meeting, a recap and some
> > > extra thoughts. Did I miss some arguments?
> > >  
> > > Benefits to go for WGLC/RFC asap:
> > > * There is NOW a big need for solutions that can support Low
> > > Latency for new Interactive applications
> > > * The big L4S benefits were a good reason to justify the extra
> > > network effort to finally implement ECN in general and AQMs in
> > > network equipment
> > > * Timing is optimal now: implementations in NW equipment are
> > > coming
> > > and deployment can start now
> > > * Deployment of L4S support will include deployment of Classic
> > > ECN
> > > too! So even for the skeptics among us, that consider that the
> > > experiment can fail due to CCs not performing to expectations, we
> > > will fall back to having Classic ECN support
> > > * Current drafts are about the network part, and are ready and
> > > stable for a very long time now.
> > > * Only dependency to CCs in the drafts are the mandatory Prague
> > > requirements (only required input/review from future CC
> > > developers: are they feasible for you)
> > > * We have a good baseline for a CC (upstreaming to Linux is
> > > blocked
> > > by the non-RFC status)
> > > * Larger scale (outside the lab) experiments are blocked by non-
> > > RFCs status
> > > * It will create the required traction within the CC community to
> > > come up with improvements (if needed at all for the applications
> > > that would benefit from it; applications that don’t benefit from
> > > it yet, can/will not use it)
> > > * NW operators have benefits now (classic ECN and good AQMs) and
> > > in
> > > the future can offer their customers better Low Latency
> > > experience for the popular interactive applications
> > > * When more L4S CCs are developed, the real independent
> > > evaluation
> > > of those can start
> > >  
> > > Disadvantages to wait for WGLC/RFC:
> > > * We’ll gets stuck in an analysis paralysis (aren’t we already?)
> > > * Trust in L4S will vanish
> > > * No signs that we can expect more traction in CC development;
> > > trust and expectations of continuous delays will not attract
> > > people working on it, as there will be plenty of time before
> > > deployments are materializing
> > > * Product development of L4S will stall and die due to
> > > uncertainty
> > > on if L4S will finally materialize
> > > * Product development of Classic ECN will stall and die due to
> > > uncertainty on how L4S will finally materialize
> > >  
> > > What are the advantages to wait? Do they overcome these
> > > disadvantages?
> > >  
> > > Regards,
> > > Koen.
> > >  
> > > 
> > 
> > 
> > 
> > 
> 
> 
>