Re: [iccrg] Will Fair Queuing change Congestion Control?

Sebastian Moeller <moeller0@gmx.de> Wed, 04 January 2023 11:38 UTC

Return-Path: <moeller0@gmx.de>
X-Original-To: iccrg@ietfa.amsl.com
Delivered-To: iccrg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 27931C14CF0E for <iccrg@ietfa.amsl.com>; Wed, 4 Jan 2023 03:38:25 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.547
X-Spam-Level:
X-Spam-Status: No, score=-2.547 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmx.de
Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UmiQ4aulI1nx for <iccrg@ietfa.amsl.com>; Wed, 4 Jan 2023 03:38:20 -0800 (PST)
Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 5DE96C14CF19 for <iccrg@irtf.org>; Wed, 4 Jan 2023 03:38:20 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de; s=s31663417; t=1672832292; bh=0AaOEf2qmk30Qg9HCN2yNscQd4pqRcT8VLGkrX8qYVE=; h=X-UI-Sender-Class:Subject:From:In-Reply-To:Date:Cc:References:To; b=kp8DdkWCvUsRRpledlHkb4FkGtnuzeq0gqKzdPMEwir+Ei/uGFXVbasxkzoEqekId mc5VmMk0u0UfYXI7JOcJP1Fw+IosG7W1SGpjSBHmVbnWh5EshJc8yPEGhVgQ0106fx fCenV8tbvTgU99qwAjqZvCPs2J5r+b9nb3WsuxQGFx238Dncr00LbkgDht0Agdd/53 1nddP9Oc/B1bAOPnZcrKf41fC27F+dc6UbsZIweB0TefamtnUU1XpMVUA5PucRKo/u T9uexjRQiuRY4hkpeVd5wjvDY72uzNpSackvwbdNzNgYLhfPAfjPDFog3c5l8L+pJm dnfmfiZSx/AcA==
X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a
Received: from smtpclient.apple ([134.76.241.253]) by mail.gmx.net (mrgmx104 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MD9XF-1p3vHB2LK5-0098ve; Wed, 04 Jan 2023 12:38:12 +0100
Content-Type: text/plain; charset="utf-8"
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\))
From: Sebastian Moeller <moeller0@gmx.de>
In-Reply-To: <IA1PR11MB724614E5325ADBE41322D8FBC3F59@IA1PR11MB7246.namprd11.prod.outlook.com>
Date: Wed, 04 Jan 2023 12:38:08 +0100
Cc: Maximilian Bachl <maximilian.bachl@gmail.com>, "iccrg@irtf.org" <iccrg@irtf.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <D14C9723-3BE3-4281-B58A-1D4C9FB68724@gmx.de>
References: <CAEXAmbs1WwvJ7761eSkSeT_w1GfwnbPL9Tcja8K7fxZHxAiMWA@mail.gmail.com> <E8CE0E38-BC35-416D-8D56-FEA1416968D6@apple.com> <CAEXAmbtjjjwqApdeXKL30x5WKaJ7Fs1_tWvbBdFUScRmqM9cXA@mail.gmail.com> <C3C1EA84-E8E5-4DCB-9EB7-BDAABDC4BE11@gmx.de> <IA1PR11MB724614E5325ADBE41322D8FBC3F59@IA1PR11MB7246.namprd11.prod.outlook.com>
To: "Annapareddy, Narasimha" <reddy@tamu.edu>
X-Mailer: Apple Mail (2.3696.120.41.1.1)
X-Provags-ID: V03:K1:y8dusgwyZ5KfDRBUZ77smaCeYq67HxWJdItm/vXGFSJKyZt+NI/ IAwSOX7N5fM5jbEKqUXLhXRKkRNJkb62h4idRklwzaORqMCBhcF6kUeLAm4te/d7o30462l lMok1ffva8gyGcu1tJDGxf3E7AeTorVTclDa8pW4LjNRzV/TvfGZANgkuaaGtt120Tt69/o ZL2jcrLc5cGFmtZ7clCdA==
UI-OutboundReport: notjunk:1;M01:P0:hs6GbDrbHS0=;BqCcMRHxOKFGj5XGzo8UXgg0C1Q BTwMclXz1BV+ufVV2J/Nryw9lwHnTa0MEDU+8batbCIS4qTsUN4V1KmpahoojJuZjGRkZXJzk 4Y+Yu/uB/B/U3sVuptLChYic0L834/cX/XXfbLhepdpvrlDhcjs5QQIIyP6xPNIJVk/3jOW9d Y31LMZDdVmJ8v0Y7QysidR38xnJgLn6MBNeLtiqLjixeyo65uzd3j85wCIVr1WpoY+OpMiO58 4h4ltkjHXB6hnpH3Klbt6eYFs/XepsboMlMmm5vo4hpuAdeNLn9qScrusxNlPuOMDqCvpOiAp TjKpmG1HlG+B1ZP46sLHSqYtLgqT0v+88/XRk6b62mGdTxLtL1pgNyzOrTGGGg+LSEC1pkORP Wxvxd5JXKRQvQ6iH6jSKjcAzA7PxswG50d2YlX2WjJBgxu4apmndfzRpCMtkPm/FlcpCMXAdP LttM5a4kyZEAU+v9louqx4WrUg6jdKeVIdJBtrXy1dWPgrENIxq1q1M5QvMA8jc3ctbYm8Z6P b+5xYssE/X42Ego09Vm6pQqnScvP2WV4fPlTMvtUJGe0EjmddLmBw9GtQsYeWuFNXrBLSJ1Nh RdI7Y4bNccwrWwgzBHCuXN3uuFvxI7VYt5BAM0WZTDZ6YTBDcbuRVWcp70rm5pJRBDyso4EfJ 33E/9atJpUNgbMOGVPiLZaJnrBJY8AbGSr6NimwW1P5n7266IZGsIVaUtshHgfKthXriQj/gW b23/yU1x9X3OcTMtWY/PjhEVvwUJ9/diWrwrhOJ6ltqHX0+UxsG4ih6fN4bo9U+i6aJ5vRjRT sBD84mGq8VrJbGxrYEOMjJsRraqsyepcg3xUnRNh9dQ6ePcZK/i7r5Rfs7/I3O6L3rqg4fxN/ ZZdEryt+BkcgEgFu+e35NPv6RTOyfRfEZXtoHOI16Wv2bDbe6dcCbtQaopwW4fEervcSNfMxw CeQAFojNd9p7ffypMplbom3m3yI=
Archived-At: <https://mailarchive.ietf.org/arch/msg/iccrg/2lHfZvgt7WP29Mdt4b2iXUtDAM4>
Subject: Re: [iccrg] Will Fair Queuing change Congestion Control?
X-BeenThere: iccrg@irtf.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: "Discussions of Internet Congestion Control Research Group \(ICCRG\)" <iccrg.irtf.org>
List-Unsubscribe: <https://www.irtf.org/mailman/options/iccrg>, <mailto:iccrg-request@irtf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/iccrg/>
List-Post: <mailto:iccrg@irtf.org>
List-Help: <mailto:iccrg-request@irtf.org?subject=help>
List-Subscribe: <https://www.irtf.org/mailman/listinfo/iccrg>, <mailto:iccrg-request@irtf.org?subject=subscribe>
X-List-Received-Date: Wed, 04 Jan 2023 11:38:25 -0000

Dear Narasimha,



> On Jan 4, 2023, at 10:09, Annapareddy, Narasimha <reddy@tamu.edu> wrote:
> 
> There are many ways delay-based congestion protocols can compete with loss-based protocols.
>  
> Here is an example approach:
>  
> Kiran Kotla and A. L. Narasimha Reddy, Making a delay-based protocol adaptive to heterogenous environments, Proc. of IWQOS, June 2008.
>  
> The basic idea is to claim a bit more bandwidth than TCP per successful transmission of window for the extra/early backoff on delay signals.

	[SM] Thanks for the link! To clarify, there appears to be no widely-used delay based congestion control algorithm in the unconstrained internet.
Any opinion on why PERT or something PERT-like has such issues of deployment? Also do you have any data/anecdotes about using PERT over the existing internet? I understand the value of simulations, but in the end with simulations one is constraint to the scenarios that are "well-simulatable"...

Again thanks, this paper as well as your other paper "Emulating AQM from End Hosts" are quite interesting for one of the projects I volunteer in.

Regards
	Sebastian




>  
> Best Regards
> Narasimha Reddy
>  
> From: iccrg <iccrg-bounces@irtf.org> On Behalf Of Sebastian Moeller
> Sent: Wednesday, January 4, 2023 9:55 AM
> To: Maximilian Bachl <maximilian.bachl@gmail.com>
> Cc: Vidhi Goel <vidhi_goel@apple.com>; iccrg@irtf.org
> Subject: Re: [iccrg] Will Fair Queuing change Congestion Control?
>  
> Hi Maximilian, > On Jan 3, 2023, at 23: 18, Maximilian Bachl <maximilian. bachl@ gmail. com> wrote: > > Hi Vidhi, > > Thank you for your comments! > > Few things to note, FQ expands to Flow Queue (not fair queueing) which 
> ZjQcmQRYFpfptBannerStart
> This Message Is From an External Sender
> This message came from outside your organization.
> ZjQcmQRYFpfptBannerEnd
> Hi Maximilian,
>  
> > On Jan 3, 2023, at 23:18, Maximilian Bachl <maximilian.bachl@gmail.com> wrote:
> > 
> > Hi Vidhi,
> > 
> > Thank you for your comments! 
> > 
> > Few things to note, FQ expands to Flow Queue (not fair queueing) which means every flow (4 tuple) has a different queue. Priorities are implemented based on service classes, for example. Also, Apple devices are not the most common bottleneck on an end to end path.
> > I assumed "flow queuing" and "fair queueing" to be mostly synonymous, as for example the fq_codel manual page in Linux calls it “fair queuing” (https://urldefense.com/v3/__https://man.archlinux.org/man/tc-fq_codel.8.en__;!!KwNVnqRv!AvRSZqVzJ0HZCTW441y9nS3-h3dVte7uXtQhbgK7GiwyS9IT73P139SBuqDEw32-SdnN3slTsLgr4DvCL9Y$). To clarify, what I meant is “flow queuing”. 
>  
>                [SM] I would argue that both flow and fair are expansions of the Fi FQ that are commonly used, one could even argue for calling this flow-fair queueing ;). It just turns out that a number of folk seem to be allergic about the term "fair" hence a recent tendency to call this flow queueing (which admittedly is more precise anyway).
>  
>  
> > 
> > Even if we assume there is enough deployment of flow queueing in the network, flows can hurt themselves by not responding to growing latency. This problem has been addressed in L4S drafts and I highly recommend reading them. 
> > Endpoints could use delay-minimizing congestion control if they could be certain that fair queuing is deployed at the bottleneck of a path. Thus, I think they wouldn’t necessarily hurt themselves. 
>  
>                [SM] Here is the rub, for delay based-CC to work flows need to be sure not to be in competition with "drop-based" CCs as otherwise they will loose. What L4S actually tries to deliver is a separation for "old" and "new" CCs so that new signaling and CC-response behavior can be used without getting into competition with old-style CC.
> However while on principle this sounds like a viable way forward, the way this was designed and implemented again confounds old and new CC-signaling, as L4S in a hare-brined move worthy only of ridicule decided to re-use the exiting ECN signaling, and especially to re-define the meaning of what a CE mark means in a non-backward-compatible manner. IMHO this is not a sign of solid engineering, but what do I know, I am a wet-ware biologist... 
>  
>  
> > 
> > ECN combined with an immediate AQM combined with Prague congestion control is basically the idea of L4S.
> > I think that L4S can improve how congestion control is done. However, from what I understand, it doesn't necessarily isolate flows.
>  
>                [SM] Indeed, in its default form it only classifies traffic into onl and new-style CC and schedules flows in one of two queues in which all packets of a class are essentially in a FIFO, so class-isolation, but no flow-separation. Historically the L4S drafts have been full of arguments why flow-queueing sucks* and needs to be avoided at all costs. These philippics have been excised in later versions and replaced with text that describes as FQ as one way to implement a L4S AQM. IMHO this is clearly an improvement over earlier drafts, but given my experience with SQM I would argue that L4S should heavily recommend FQ scheduling and only offer dual-queue scheduling as band-aid for situations where FQ is impossible for one way or the other. (My rationale for that change is that the most likely deployment of L4S schedulers is at the ISP to end-user edge, where as the SQM project has shown, FQ-scheduling is a viable option that actually works even when just using old-style "dropped-based" CC).
>  
>  
>  
> *) Mostly boiling down to that an attacker that can split its traffic in multiple flows can get more than his/her fair share of the capacity, a failure mode that FQ shares with a FIFO I would argue so, yes an attack surface FQ does not remedy, but hardly a novel attack surface... to be expolicit, this is IMHO not a convincing argument, but again as biologist I have apparently little insiight in what "engineers" find convincing.
>  
>  
> > Thus a bad faith actor could get an unfairly large share of bandwidth, which couldn't happen when FQ were deployed.
>  
>                [SM] Mostly yes, except if the attacker can split his/he traffic into multiple different flows. Cake offers an elegant way around this dilemma by offering an additional layer of isolation, where it first splits capacity equitably between internal IP addresses, thereby restricting the flow-inflation problem to the traffic mix sent to individual computers.
>  
>  
> > With FQ being deployed on a path, each flow could also choose its own congestion control and whether it wants to prioritize low delay or maximum throughput. That's why I think that exciting possibilities open up when more vendors deploy FQ like Apple did. 
>  
>                [SM] However Apple's fq_codel seems to differ from RFC fq_codel in several ways, so I am not sure (in that I personally do not know about the details) whether Apple's deployment actually helps.
>  
> Regards
>                Sebastian
>  
> > 
> > Regards,
> > Max
> > 
> > On Tue 3. Jan 2023 at 20:45, Vidhi Goel <vidhi_goel@apple.com> wrote:
> > Hello Max,
> > 
> > 
> > Few things to note, FQ expands to Flow Queue (not fair queueing) which means every flow (4 tuple) has a different queue. Priorities are implemented based on service classes, for example. Also, Apple devices are not the most common bottleneck on an end to end path.
> > 
> > Even if we assume there is enough deployment of flow queueing in the network, flows can hurt themselves by not responding to growing latency. This problem has been addressed in L4S drafts and I highly recommend reading them. ECN combined with an immediate AQM combined with Prague congestion control is basically the idea of L4S.
> > 
> > Vidhi
> > 
> >> On Jan 3, 2023, at 9:20 AM, Maximilian Bachl <maximilian.bachl@gmail.com> wrote:
> >> 
> >> Apple introduced fair queuing (FQ) on more than a billion of devices in the last years (https://urldefense.com/v3/__https://blog.cerowrt.org/post/state_of_fq_codel/__;!!KwNVnqRv!AvRSZqVzJ0HZCTW441y9nS3-h3dVte7uXtQhbgK7GiwyS9IT73P139SBuqDEw32-SdnN3slTsLgrihq1oYY$). I wonder if this could have an impact on congestion control. When there’s fair queuing at the bottleneck link of a path, the endpoints can use delay-minimizing congestion control and don’t have to worry about more aggressive senders. 
> >> 
> >> However, currently, there’s no established mechanism for endpoints to know of the existence of fair queuing on a path. There are some possibilities, though, how endpoints can be made aware of the existence of fair queuing: 
> >>           • A dedicated bit in some packet header. For example, some ECN or DSCP bits could be repurposed to indicate the existence of fair queuing at the bottleneck. Drawback: It’s hard to convince people to repurpose/introduce new bits and switches/routers have to support it. 
> >>           • An end-host-based mechanism to detect fair queuing without the involvement of switches/routers. I created a proof-of-concept implementation of such a mechanism, which can detect the presence/absence of fair queuing with an accuracy of > 95% (https://urldefense.com/v3/__https://arxiv.org/abs/2206.10561__;!!KwNVnqRv!AvRSZqVzJ0HZCTW441y9nS3-h3dVte7uXtQhbgK7GiwyS9IT73P139SBuqDEw32-SdnN3slTsLgra2Jc9rk$).
> >>           • Closed ecosystems in which all components are aware that there is fair queuing: As a hypothetical example, let’s assume a user plays a game on nvidia’s cloud-gaming platform on an iPhone. The phone is connected to T-Mobile’s 5G network. Both the client on the iPhone as well as the server in nvidia’s network know that there’s fair queuing on the path between the client and the server, since they have an agreement with T-Mobile, which controls the entire path. No extra bits in packet headers are needed, just mutual agreement between the user’s device, the cellular network provider as well as the server on the internet. 
> >> 
> >> If you have some opinions on whether fair queuing is going to change how we do congestion control, I’d be curious. 
> >> 
> >> Regards,
> >> Max
> >> _______________________________________________
> >> iccrg mailing list
> >> iccrg@irtf.org
> >> https://urldefense.com/v3/__https://www.irtf.org/mailman/listinfo/iccrg__;!!KwNVnqRv!AvRSZqVzJ0HZCTW441y9nS3-h3dVte7uXtQhbgK7GiwyS9IT73P139SBuqDEw32-SdnN3slTsLgr4bkQSXU$
> > 
> > _______________________________________________
> > iccrg mailing list
> > iccrg@irtf.org
> > https://urldefense.com/v3/__https://www.irtf.org/mailman/listinfo/iccrg__;!!KwNVnqRv!AvRSZqVzJ0HZCTW441y9nS3-h3dVte7uXtQhbgK7GiwyS9IT73P139SBuqDEw32-SdnN3slTsLgr4bkQSXU$
>  
> _______________________________________________
> iccrg mailing list
> iccrg@irtf.org
> https://urldefense.com/v3/__https://www.irtf.org/mailman/listinfo/iccrg__;!!KwNVnqRv!AvRSZqVzJ0HZCTW441y9nS3-h3dVte7uXtQhbgK7GiwyS9IT73P139SBuqDEw32-SdnN3slTsLgr4bkQSXU$