Re: [tsvwg] Reasons for WGLC/RFC asap

Jonathan Morton <> Thu, 19 November 2020 18:07 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 9DA6F3A0DF4 for <>; Thu, 19 Nov 2020 10:07:52 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.848
X-Spam-Status: No, score=-1.848 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id E6XYspx668uX for <>; Thu, 19 Nov 2020 10:07:50 -0800 (PST)
Received: from ( [IPv6:2a00:1450:4864:20::131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 806103A1110 for <>; Thu, 19 Nov 2020 10:07:37 -0800 (PST)
Received: by with SMTP id d17so9553160lfq.10 for <>; Thu, 19 Nov 2020 10:07:37 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=7Nl/JGUaF6tkY3JdrOSDmYOGDegnS9I/4xiXC566LVY=; b=joqdPMCpcXZM08VV2FesszvXuZxM/vEVJfpwSSVXGVSdFMNQQDfi3S6Kk9Dbh1GkcW g3llqHw78n6y0PXjdKwLe/uJo0HCFswnZKesMoPKK6S0X/DnsZSytLqxRH7UiAallJ7g 3QGztplL4R8E7mEc+WIM5fkT4yc8UPI/DGdsn5T3iC2v9bHeVKYtOGK3JoFSTSjUEuF1 L4gjzhaSUo+Fti7cgypm/7IgdLC+beGrpdmFNZgMoc4gUN8uPwGVLkObZmyK5690oGRJ XoK9/T1jUPZcgtCL1KlzrAucvmZUEgz8GiRUArN300K+g98Ggh2FLL7BL6hyQOZvZUmO 4yhA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=7Nl/JGUaF6tkY3JdrOSDmYOGDegnS9I/4xiXC566LVY=; b=sIeH5vT9keTSTFB6ySGw9BnkSQvyjzyfFo63KZYMzOjoJtBNTI2+culv78PMnd5DL6 NyCkzDAaDjDAQ+LuggF/xyFunWN/6yNgrTTr0bob+/BeSOR5r7GEQ/93aTEIIwO3QK4i MbtpwiVkTfY84MVaf0OrwU/0Z6nXyr1yXqOfFGJ+SJ55Jy74h0+eq2BTok+RjVLNMpDG bfHb6zAZB/dnYMrRFgWB9eV0g85QQxhS2M0WRo7sJGK43MS14H64Kdnp+pND/M8RG1zX SjIBlv0ELSgjMDZboivS7QyYIxEftIdQ+4nYNrprApgNET8LbD5VJo7g5IYKnLEjnc2B UHZA==
X-Gm-Message-State: AOAM533MZTNuOGWiET+4QnmnHX1COQ4sypXgzY8MvoAN+E+NkrbfleSg SzJ4GAdKEfg2E5/wsD2sZSE=
X-Google-Smtp-Source: ABdhPJzJCDFkzeP+lKaU9SlEL4uqbRF2nLOGs5OT+6nD329i6IycWupBtLJQmTZmb08/NnbTyczTFQ==
X-Received: by 2002:a19:4297:: with SMTP id p145mr7117970lfa.481.1605809255463; Thu, 19 Nov 2020 10:07:35 -0800 (PST)
Received: from jonathartonsmbp.lan ( []) by with ESMTPSA id q16sm35682lfb.299.2020. (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 19 Nov 2020 10:07:34 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.7\))
From: Jonathan Morton <>
In-Reply-To: <>
Date: Thu, 19 Nov 2020 20:07:33 +0200
Cc: Lars Eggert <>, tsvwg IETF list <>
Content-Transfer-Encoding: quoted-printable
Message-Id: <>
References: <> <> <> <> <> <> <> <>
To: Ingemar Johansson S <>
X-Mailer: Apple Mail (2.3445.9.7)
Archived-At: <>
Subject: Re: [tsvwg] Reasons for WGLC/RFC asap
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Transport Area Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Thu, 19 Nov 2020 18:07:53 -0000

> On 19 Nov, 2020, at 4:24 pm, Ingemar Johansson S <> wrote:
> Recall from the discussion yesterday that Stuart Cheshire and
> his team has not need evidence of ECN marking AQMs. So I fear that we spend
> a lot of time discussing problems that are very rare.

Okay, I'm going to interject on this.  I don't know precisely how Apple is collecting their data, but you can see the data collected by a MacOS X machine by entering the following in Terminal:

	sudo netstat -sp tcp

About two-thirds of the way down the output, you should find a group of statistics related to ECN.  Here's mine:

	917 client connections attempted to negotiate ECN
		789 client connections successfully negotiated ECN
		97 times graceful fallback to Non-ECN connection
		1 time lost ECN negotiating SYN, followed by retransmission
		0 server connection attempted to negotiate ECN
		0 server connection successfully negotiated ECN
		0 time lost ECN negotiating SYN-ACK, followed by retransmission
		162 times received congestion experienced (CE) notification
		0 time CWR was sent in response to ECE
		1986 times sent ECE notification
		32 connections received CE atleast once
		0 connection received ECE atleast once
		298 connections using ECN have seen packet loss but no CE
		17 connections using ECN have seen packet loss and CE
		15 connections using ECN received CE but no packet loss
		0 connection fell back to non-ECN due to SYN-loss
		14 connections fell back to non-ECN due to reordering
		0 connection fell back to non-ECN due to excessive CE-markings

Of the connections where ECN negotiation was attempted, most (almost 90%) succeeded, and of these some did show ECN activity in practice - noting that many connections probably did not reach the shaper limit set by my ECN-enabled router, or reached the capacity of my rural LTE connection at a time when it was less than the shaper limit.

But to put this into context, further up is:

	117038 connection requests

This is a machine with ECN support forced on by appropriate entries in /etc/sysctl.conf.  Even so, less than 1% of TCP connections initiated even *attempted* to negotiate ECN.  This led me to discover that MacOS X had unexpectedly and erroneously designated my network as an ECN blackhole, a condition that was temporarily resolved by turning wifi off and on again, but soon recurred for unknown reasons.  (This effect does not occur with my Linux-based machines.)

If this erroneous ECN blackholing phenomenon is common in Apple devices - not only desktops and laptops but also iPhones and iPads - then that would certainly explain why Stuart Cheshire sees so little ECN activity in the data available to him.  I would encourage him to double-check that with appropriate technical employees at Apple.

I note also that Jake Holland's data from Akamai depends on client endpoints initiating ECN negotiation, which is by no means universal thus far, and may therefore understate the deployment of RFC-3168 compliant AQMs (which exert congestion signalling via packet drops for Not-ECT traffic) by a large factor, especially in the small number of ASNs that already show significant ECN activity in that data.

In short, I believe there's a lot more ECN deployment in practice than most people realise.

 - Jonathan Morton