Re: [tsvwg] FQ & VPNs

Dave Taht <> Sun, 21 February 2021 12:00 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 250943A1A8E for <>; Sun, 21 Feb 2021 04:00:25 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.095
X-Spam-Status: No, score=-2.095 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_IMAGE_RATIO_06=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id ozVAHUxs_OKX for <>; Sun, 21 Feb 2021 04:00:22 -0800 (PST)
Received: from ( [IPv6:2607:f8b0:4864:20::132]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id D55333A1A8D for <>; Sun, 21 Feb 2021 04:00:21 -0800 (PST)
Received: by with SMTP id e2so8472044ilu.0 for <>; Sun, 21 Feb 2021 04:00:21 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=sxo7QKFvflxJhRISsaJGl/mOGxXvRSM6pDEs09Wm+LI=; b=mwlW2v/cNTx0a54voslgdWtPw+rwhB+BeK6mGzLiO4cYok4YzTv3gAWnojhR0y3jz3 gv2Krf6jOuqq06T0hTEPjBrkEn79igyQq/VzW4vSVdtsMu0LmKndV4+XVJAXUsVPqDCq JQPKvPG/ps1McPHDZCUF4cr81t+4/oi1HIuojyHaNO+6bDigxLJoilI6/mb5m00oRJsI 9uiKQ9P+Q1Ks+eKvz41gaCvP+nVMh5t929P6BywBYHK0GkZf+b+hLOT3+7JacEZE/bka gWL8OrpZtMHZM/fxennpzb+ioyJCcOGa4XwuEv0VV/pbLWNyU3YeHzxXm1wSf9a16Qfe SHZg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=sxo7QKFvflxJhRISsaJGl/mOGxXvRSM6pDEs09Wm+LI=; b=kXHehwYxk1t2u9m3KXqVEOWihSNdOMAaZbXsAWgxzlgd97J5SWS/kOSjxLe4/lvZDu 4GFNj1fjHmS8xPdzSmme8LuIFz+RBTJWN6pzBjGWtIgT6obGN7KBnyNyE3bDrei7od5a c9zfZ00gyPa/9iomvjeyLvq0aFtxijiOsFIBCdU1Z2kg2nUYvipyZAPG0lsaYtqIhwy1 1XaH8fXQeGBoFIpGTe1LGUacrNCjZVSCxmnHuOV6WyUizX176Qb5utsvidbFindZScVs 5TkNk+2jbh/WjWLeXaPDtec3Osxfrhafs/6Z2aqtfep9i+Uaje/IbfQpVzzVoq7a3U+v OMcA==
X-Gm-Message-State: AOAM533oNzjlN2q2mFVqGMqlFO7KfNDxxgG2632AlxX5rHacZwQGIuSb anpx34BZfXuSFpPH3e+ayxzY26vkrnbNeZHFlF28GLKNCSw=
X-Google-Smtp-Source: ABdhPJy0meX89kAB9UcxMUV+d/QrOKscOf4Bvfn3BGmji8zfau8O1KZbQvQUcEd8ITsjV7t8gLp/6uKNUW0oEZtCiMQ=
X-Received: by 2002:a05:6e02:1d85:: with SMTP id h5mr11592197ila.246.1613908820675; Sun, 21 Feb 2021 04:00:20 -0800 (PST)
MIME-Version: 1.0
References: <> <> <> <> <> <> <> <> <> <> <> <>
In-Reply-To: <>
From: Dave Taht <>
Date: Sun, 21 Feb 2021 04:00:08 -0800
Message-ID: <>
To: Ingemar Johansson S <>
Cc: Jonathan Morton <>, Bob Briscoe <>, TSVWG <>
Content-Type: multipart/related; boundary="000000000000f54d1105bbd76d82"
Archived-At: <>
Subject: Re: [tsvwg] FQ & VPNs
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Transport Area Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Sun, 21 Feb 2021 12:00:25 -0000

Participating reluctantly, as usual.

On Sat, Feb 20, 2021 at 11:04 PM Ingemar Johansson S <> wrote:

> Hi
> Trying to summarize. Your concern is that the L4S flows starve out the
> non-L4S flows when they share the same VPN tunnel.
> You proposed the solution yourself below:
> “The VPN user is thus in a position to predict, understand, and possibly
> mitigate the effect by splitting the traffic across multiple VPN flows, if
> performance turns out to be more important than hiding that piece of
> information”.
> The same rationale can be used to put the L4S flows in a separate VPN
> tunnel, problem solved.

In terms of VPNs there are multiple other problems. Site-site, bare metal,
running fq_codel over ipsec

> And then, there is always a possibility to upgrade the AQMs to support L4S
> as well ?. It always appear from your argumentation that fq-codel is some
> kind of untouchable monolith that cannot at any cost be modified, unless of
> course you see a possibility that VPN tunnels can somehow leak inner header
> information (see separate discussion around
> draft-ietf-tsvwg-transport-encrypt)

It's not an untouchable monolith. It is merely the default qdisc in nearly
every linux distribution at this point, ios, and osx. It is used by 10s,
maybe 100s of millions of devices at this point. It is part of billions of
containers. With nearly 9 years of deployment into an ever increasing
number of embedded devices that typically lag 5 or more years behind the
mainline OSes. Making changes to how ecn is interpreted would take 8+ years
to flush out of the field.

Total (documented) L4S market penetration of its mis-interpretation of the
CE field - 0. I might be kind and say "alternate", rather than mis, but
slamming L4S through a rfc3168 fq_codel'd system does have the documented
effects so often discussed here.

The ce_threshold parameter has been part of fq_codel for many years also.
Tweaking that to instead trigger off of ECT(1) is straightforward. Without
clear data - which we didn't have at the time of this debate hitting public
mailing lists - doing that seemed dicy - and there was indeed a
encapsulation bug regarding that transition that was found and fixed a
while back which will take years to propagate to the field, also.

The SCE alternative is a backward compatible interpretation of the ECN
related bits - and fails safe -  and could have gone into wider
distribution 2 years ago with near zero impact on end-user OSes, and as
gradual upgrade, and been (well, actually, has, as fq_codel_fast and the
alternate cake have been tested by a few parties at this point) tested as
to other impacts. Coupled also with a DSCP field a la "nqb" (and preferably
a few other dscp bits) it could be effective, sooner.

I have been tempted a lot to at least make SCE available as an option in
the upcoming openwrt release, but neither side has made a convincing
argument that doing this to ECN at all makes sense, and in fact the flood
of negative data on it (rtt unfairness in particular), makes me more
inclined to just recommend ecn be disabled entirely.

> With that said.. I believe that Bob is right. It benefits the TSVWG group
> better that we continue this discussion off list. It is just the same old
> arguments over and over again, albeit packaged in different wording.
> /Ingemar
> *From:* Jonathan Morton <>
> *Sent:* den 20 februari 2021 22:29
> *To:* Ingemar Johansson S <>
> *Cc:* Dave Taht <>om>; Bob Briscoe <>et>;
> TSVWG <>
> *Subject:* Re: [tsvwg] FQ & VPNs
> On 20 Feb, 2021, at 8:42 pm, Ingemar Johansson S <
>> wrote:
> Quoting Jonathan " However, I would remind you that neither SFQ nor
> fq_codel can distinguish between flows carried inside an encrypted tunnel,
> so this cannot be relied upon alone to make L4S safe.  Pete's data shows
> significant tunnelled traffic, much of which is probably due to people
> working from home and using VPNs to access a corporate network."
> This line of reasoning makes me confused.
> Section 6.2 in RFC8290 (
> clearly says that fq-codel will not deliver its intended benefits for
> encrypted tunnels. If ISP's still rely on fq-codel to do it´s job well even
> though a substantial share of the traffic is VPN, then that is of course a
> problem. But I see that only as an fq-codel problem.
> I will just note here that although treating a VPN containing multiple
> flows as a single flow results in a reduction in throughput, when it has to
> compete against other traffic at a saturated bottleneck, this is a
> relatively minor and infrequent event at the ISP in question.
> The degree of impairment experienced by the VPN is also strictly bounded
> by the number of saturating flows carried within it.  The VPN user is thus
> in a position to predict, understand, and possibly mitigate the effect by
> splitting the traffic across multiple VPN flows, if performance turns out
> to be more important than hiding that piece of information.
> So what I cannot at all understand.. Why should a known problem (or read
> design feature) in fq-codel (or SFQ) be interpreted as something that makes
> L4S unsafe ?.
> The problem is entirely due to L4S flows' harm to conventional AIMD flows
> when passing through the same queue and AQM instance, unless the AQM is of
> a type that specifically understands L4S.  This is a predictable and
> inevitable consequence of L4S flows responding qualitatively less (by what
> is effectively an Additive Decrease, despite claims to the contrary) to
> each CE mark than defined in either RFC-3168 or RFC-8511 (which both
> mandate a Multiplicative Decrease to a single CE mark).
> Consequently, if L4S and AIMD flows share a single VPN tunnel, which
> passes through a conventional ECN AQM at the bottleneck, the AIMD flows
> will collapse to a very low throughput.  It is crucial to understand here
> that all existing ECN-marking AQMs deployed in the Internet today - most of
> which are fq_codel or similar - fall into this category.
> I attach an example of the likely consequences.  The red trace at the top
> is L4S throughput; the light blue trace at the very bottom is the AIMD flow
> starting ten seconds later.  You may refer to previous data sets published
> by Pete Heist and myself for further examples.
>  - Jonathan Morton

"For a successful technology, reality must take precedence over public
relations, for Mother Nature cannot be fooled" - Richard Feynman <Dave Täht> CTO, TekLibre, LLC Tel: 1-831-435-0729