Re: [tcpm] [tsvwg] [Ecn-sane] ECN CE that was ECT(0) incorrectly classified as L4S

<> Mon, 05 August 2019 12:17 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 1FACD1201E3; Mon, 5 Aug 2019 05:17:02 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -4.298
X-Spam-Status: No, score=-4.298 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id WjyzheC-YPFa; Mon, 5 Aug 2019 05:16:59 -0700 (PDT)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 74E611201DE; Mon, 5 Aug 2019 05:16:57 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;;; q=dns/txt; s=dtag1; t=1565007417; x=1596543417; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=1p6NBs9VlTWdj9etn3VEV2k3YHMRtK/2N+TK3g6b3gY=; b=ZSjDotI3Nh2EV/wylyk+YjFJniO5pg5cKJWf++0I+vIUoibY+Hhg+e0f ab3/EOyNhY/nkpfelLtE42Wv+ly3f2lwwZYUAXhXvO07lNhhkF+wX2YV0 5aXWyI6F2Sq9vcDqkn60pa/+ZfguO/J38MgpsoU2GLU291YmWiu6tDW+T iBIcPBUD7VHrke0wFmWqbjeqVyCjs6mx8pF62Ul5sAMjqUMmmJY8M35sr 3QssS1Kkj7k2ui9osGM9k9XrHbFLP0w2kRDVoaFANZbsXYsM4WETqeK2h NI5Q548islDj5cqDUhbJSs8fJnRlMr+yF/MlV6w9iFI9ewVurvYqyX8nW w==;
Received: from ([]) by with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Aug 2019 14:16:54 +0200
X-IronPort-AV: E=Sophos;i="5.64,349,1559512800"; d="scan'208";a="473401192"
X-MGA-submission: MDHMV5cBsQt9ixhT7iyFOImqe0l1nwqEyAPAC7tRa7eTZ0zbCQ12mZJ5uJK6j1fDkNdZ5X94IyUzSO9HFXFtS++NCRqcuQSISZy5tee3WBDJl4TGES0259sVpHjb+hOoCLi+xkNMYCYhnT7Z06db85WS0whcOUMhmNqU7QtDqPVf4w==
Received: from ([]) by with ESMTP/TLS/AES256-SHA; 05 Aug 2019 14:16:49 +0200
Received: from ( by ( with Microsoft SMTP Server (TLS) id 15.0.1473.3; Mon, 5 Aug 2019 14:16:38 +0200
Received: from ( by ( with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Mon, 5 Aug 2019 14:16:38 +0200
Received: from ( by ( with Microsoft SMTP Server (TLS) id 15.0.1473.3; Mon, 5 Aug 2019 14:16:37 +0200
Received: from LEXPR01MB0463.DEUPRD01.PROD.OUTLOOK.DE ( by LEXPR01MB1247.DEUPRD01.PROD.OUTLOOK.DE ( with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2136.17; Mon, 5 Aug 2019 12:16:37 +0000
Received: from LEXPR01MB0463.DEUPRD01.PROD.OUTLOOK.DE ([fe80::2d90:342b:b54e:462b]) by LEXPR01MB0463.DEUPRD01.PROD.OUTLOOK.DE ([fe80::2d90:342b:b54e:462b%6]) with mapi id 15.20.2136.018; Mon, 5 Aug 2019 12:16:37 +0000
Thread-Topic: [tsvwg] [Ecn-sane] ECN CE that was ECT(0) incorrectly classified as L4S
Thread-Index: AQHVNmzjJ0yZ6qt6w0KBBMUQQa2wF6bnpeTwgAAbvwCABKfngIAAI2aAgAABO+A=
Date: Mon, 05 Aug 2019 12:16:37 +0000
References: <> <> <FRXPR01MB0710085AE4AEFA896819AFA69CD90@FRXPR01MB0710.DEUPRD01.PROD.OUTLOOK.DE> <> <LEXPR01MB046306842E5AB407A7BFC6619CDA0@LEXPR01MB0463.DEUPRD01.PROD.OUTLOOK.DE> <>
In-Reply-To: <>
Accept-Language: de-DE, en-US
Content-Language: de-DE
authentication-results: spf=none (sender IP is );
x-originating-ip: []
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 810bba89-af6e-4f1c-87fb-08d7199ec3cb
x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600148)(711020)(4605104)(1401327)(2017052603328)(7193020); SRVR:LEXPR01MB1247;
x-ms-traffictypediagnostic: LEXPR01MB1247:
x-ms-exchange-purlcount: 1
x-microsoft-antispam-prvs: <LEXPR01MB124760C803F09B4BE55B096F9CDA0@LEXPR01MB1247.DEUPRD01.PROD.OUTLOOK.DE>
x-ms-oob-tlc-oobclassifiers: OLM:10000;
x-forefront-prvs: 01208B1E18
x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(39860400002)(396003)(346002)(376002)(366004)(136003)(189003)(199004)(5660300002)(186003)(26005)(8936002)(8676002)(486006)(6916009)(53936002)(52396003)(305945005)(508600001)(7736002)(3846002)(66946007)(6116002)(446003)(11346002)(66446008)(66476007)(66556008)(64756008)(9686003)(476003)(54906003)(76116006)(6306002)(55016002)(33656002)(2906002)(4326008)(7696005)(256004)(14454004)(14444005)(71190400001)(45776006)(76176011)(102836004)(966005)(66066001)(86362001)(66574012)(81166006)(71200400001)(81156014)(1411001)(68736007); DIR:OUT; SFP:1101; SCL:1; SRVR:LEXPR01MB1247; H:LEXPR01MB0463.DEUPRD01.PROD.OUTLOOK.DE; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1;
received-spf: None ( does not designate permitted sender hosts)
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam-message-info: H5HJeX2IabJzpN5DMxaAo3k8DHXi51HvSGgKbUVgcK25yK3eEOFs/xGaNmORlTGG5UHqjor8jaDfjbjqtUE5Ab+eD0YQ6M1FkLIOBPF/j68t8Xr+HBDpQdBtFbJ6PzjI7qGYnYo6VdoRbERzzGl7HcSIh7yhfjSuMiboQNefT4oWwYEU4PuMvwingoXTtmWkFDhtNbs9viNUD8bPQsERX6vJKAvXFGruMsSw+pDi7w/egJWAybmDL0TtkENHUxyQgB3A7ZCs7TlV3zP5tVPqdpqBwXIVvYff+gEyI8vSPPxfRJ5s7zjTW8Js5NU47kr067ENAC3WOeyPX/L263Fm928Fgbtu7VKNr7pEnu3rlOlVTel3fHuiGyAy7rWPDMvB24t3owPizzpanQSxwkqu4tFFbLyth1jXACSQLBgx2CQ=
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 810bba89-af6e-4f1c-87fb-08d7199ec3cb
X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Aug 2019 12:16:37.5847 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: bde4dffc-4b60-4cf6-8b04-a5eeb25f5c4f
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-Transport-CrossTenantHeadersStamped: LEXPR01MB1247
Archived-At: <>
Subject: Re: [tcpm] [tsvwg] [Ecn-sane] ECN CE that was ECT(0) incorrectly classified as L4S
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Mon, 05 Aug 2019 12:17:12 -0000

As I said, no corner-case engineering. On my vacation site abroad, Netflix worked as well as it does at home. Germany is often criticised as a laggard in developing a competitive broadband access market. My interest in IETF work being set up to work around commercial problems between ISPs and ISPs or CDNs in a rich country like the UK or US is low. 

If your technical standardization activities help to make using the Internet more convenient in African countries without raising extra cost or requiring extra skills, I'm sure that's a fair market. I know that these countries are struggling to improve the services operated in their networks. Conditions there (and some Asian countries) differ strongly from those in many other parts of the world.
It might be a good thing to clarify under which scenarios the problem solutions you work on create the most significant benefit.

In the market that I am aware of, there's a single regular bottleneck, which is the consumer access or terminal.

-----Ursprüngliche Nachricht-----
Von: Jonathan Morton <> 
Gesendet: Montag, 5. August 2019 13:00
An: Geib, Rüdiger <>
Betreff: Re: [tsvwg] [Ecn-sane] ECN CE that was ECT(0) incorrectly classified as L4S

> [JM] A progressive narrowing of effective link capacity is very common in consumer Internet access.  Theoretically you can set up a chain of almost unlimited length of consecutively narrowing bottlenecks, such that a line-rate burst injected at the wide end will experience queuing at every intermediate node.  In practice you can expect typically three or more potentially narrowing points:
> [RG] deleted. Please read , first two sentences. That's a sound starting point, and I don't think much has changed since 2005. 

As I said, that reference is *usually* true for *responsible* ISPs.  Not all ISPs, however, are responsible vis a vis their subscribers, as opposed to their shareholders.  There have been some high-profile incidents of *deliberately* inadequate peering arrangements in the USA (often involving Netflix vs major cable networks, for example), and consumer ISPs in the UK *typically* have diurnal cycles of general congestion due to under-investment in the high-speed segments of their network.

To say nothing of what goes on in Asia Minor and Africa, where demand routinely far outstrips supply.  In those areas, solutions to make the best use of limited capacity would doubtless be welcomed.

> [RG] About the bursts to expect, it's probably worth noting that today's most popular application generating traffic bursts is watching video clips streamed over the Internet. Viewers dislike the movies to stall. My impression is, all major CDNs are aware of that and try their best to avoid this situation. In particular, I don't expect streaming bursts to overwhelm access link shaper buffers by design. And that, I think, limits burst sizes of the majority of traffic.

In my personal experience with YouTube, to pick a major video streaming service not-at-all at random, the bursts last several seconds and are essentially ack-clocked.  It's just a high/low watermark system in the receiving client's buffer; when it's full, it tells the server to stop sending, and after it drains a bit it tells the server to start again.  When traffic is flowing, it's no different from any other bulk flow (aside from the use of BBR instead of CUBIC or Reno) and can be managed in the same way.

The timescale I'm talking about, on the other hand, is sub-RTT.  Packet intervals may be counted in microseconds at origin, then gradually spaced out into the millisecond range as they traverse the successive bottlenecks en route.  As I mentioned, there are several circumstances when today's servers emit line-rate bursts of traffic; these can also result from aggregation in certain link types (particularly wifi), and hardware offload engines which try to treat multiple physical packets from the same flow as one.  This then results in transient queuing delays as the next bottleneck spaces them out again.

When several such bursts coincide at a single bottleneck, moreover, the queuing required to accommodate them may be as much as their sum.  This "incast effect" is particularly relevant in datacentres, which routinely produce synchronised bursts of traffic as responses to distributed queries, but can also occur in ordinary web traffic when multiple servers are involved in a single page load.  IW10 does not mean you only need 10 packets of buffer space, and many CDNs are in fact using even larger IWs as well.

These effects really do exist; we have measured them in the real world, reproduced them in lab conditions, and designed qdiscs to accommodate them as cleanly as possible.  The question is to what extent they are relevant to the design of a particular technology or deployment; some will be much more sensitive than others.  The only way to be sure of the answer is to be aware, and do the appropriate maths.

> [RG] Other people use their equipment to communicate and play games

These are examples of traffic that would be sensitive to the delay from transient queuing caused by other traffic.  The most robust answer here is to implement FQ at each such queue.  Other solutions may also exist.

> Any solution for Best Effort service which is TCP friendly and support scommunication expecting no congestion at the same time should be easy to deploy and come with obvious benefits. 

Well, obviously.  Although not everyone remembers this at design time.

> [RG] I found Sebastian's response sound. I think, there are people interested in avoiding congestion at their access.

> the access link is the bottleneck, that's what's to be expected.

It is typically *a* bottleneck, but there can be more than one from the viewpoint of a line-rate burst.

> [RG] I'd like to repeat again what's important to me: no corner case engineering. Is there something to be added to Sebastian's scenario?

He makes an essentially similar point to mine, from a different perspective.  Hopefully the additional context provided above is enlightening.

 - Jonathan Morton