Re: [tsvwg] Fwd: New Version Notification for draft-heist-tsvwg-ecn-deployment-observations-02.txt

Bob Briscoe <> Mon, 22 March 2021 16:53 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 3B7C33A0D02 for <>; Mon, 22 Mar 2021 09:53:20 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: 0.773
X-Spam-Status: No, score=0.773 tagged_above=-999 required=5 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, SPF_SOFTFAIL=0.972, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 3LimNYFQtWH3 for <>; Mon, 22 Mar 2021 09:53:16 -0700 (PDT)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 3E7AB3A0CFC for <>; Mon, 22 Mar 2021 09:53:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;; s=default; h=Content-Transfer-Encoding:Content-Type: In-Reply-To:MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender :Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=pYhEOrZUOd78A2iPaZKMz4RuEQ2oXs3nku7kBXlS9ik=; b=B8KScMbKl/O8OffqWcB0fDpZ9I RBdeX3qxTTNtWDJex8Jd+V4lkQh2X4p22+4gMU5WXwH6Olda5kqMKRfEPrrX6m4wDVfO83R/cfh9g olNvrdSIo0yTVisJtdD7RuTteB5xKNcf1gP8SKEbvLVyXmqmrtkK74UhkM7wQ3zWtg5YyKlitzoU+ +Wvk2MWGXoVtAMaI8r5B8G7G95TMN56Zw8MyOcgi7nr4LvENyVOIOnc7E2hgqWKlZuvV+TBLlJzlN UkWrqniJYFH8E+kIOwmISpiGrJXquvbNluwbFWnvFuaiqt+ELS/WFWGHKAlp1wOkxwlvo1WGp23pp Hu+cyUvQ==;
Received: from ([]:46734 helo=[]) by with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.93) (envelope-from <>) id 1lONnm-0005vG-FC; Mon, 22 Mar 2021 16:53:14 +0000
To: Pete Heist <>
Cc: tsvwg IETF list <>,
References: <> <> <> <> <>
From: Bob Briscoe <>
Message-ID: <>
Date: Mon, 22 Mar 2021 16:53:12 +0000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.7.1
MIME-Version: 1.0
In-Reply-To: <>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-GB
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname -
X-AntiAbuse: Original Domain -
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain -
X-Get-Message-Sender-Via: authenticated_id:
Archived-At: <>
Subject: Re: [tsvwg] Fwd: New Version Notification for draft-heist-tsvwg-ecn-deployment-observations-02.txt
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Transport Area Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Mon, 22 Mar 2021 16:53:20 -0000


On 09/03/2021 01:13, Pete Heist wrote:
> Hi Bob, comment below...
> On Tue, 2021-03-09 at 00:25 +0000, Bob Briscoe wrote:
>> Pete,
>> Thanks for the updated draft a fortnight ago, with the extra columns
>> I
>> asked for.
>> I've been excluding the rows with obvious pathologies.
>> Of the 90 IPs in your TCP results, 74 have at least one obvious
>> pathology, leaving only 16 clean.
>> And actually, there was a pathology suggested in your slides today
>> that
>> isn't in the above,
>>       If I check for that too, 83 IPs have at least one pathology,
>> leaving only 7 clean.
>> Any idea what is going on?
>> ==Details==
>> In the TCP results, unlike with QUIC, the reverse feedback is
>> visible.
>> So I can check for more pathologies.
>> I checked for this obvious pathology, that wasn't directly mentioned
>> in
>> the slides today:
>> * Congestion experienced (CE) in one direction but no echo
>> congestion experienced (ECE) feedback in the other.
>> * Or conversely, echo congestion experienced (ECE) feedback in one
>> direction but no congestion experienced (CE) in the other.
> I believe I've got your question right, but let's see. From the vantage
> point of the gateway, we don't always get to see CE in one direction
> and ECE in the other. In fact, it seems a lot of the time we don't.
> Say downstream traffic passes through the gateway as ECT(0) and gets
> marked in an AQM later with CE, before arriving at the user endpoint.
> We haven't seen that CE at the gateway, but we still see the ECEs in
> the opposite direction. The same can happen upstream, if CE is marked
> somewhere after packets leave the gateway. That's why we don't require
> the presence of CE marks to mark an entry "possible AQM", but we use
> them if they're present.

[BB] OK that's a feasible explanation... but the numbers still seem weird...

Can you provide more insight into the topology of this ISP. You say it 
is a co-operative wireless ISP. Are all the access links symmetric?

I'm asking because, in the direction from the LAN, it seems highly 
unlikely that 38 of the 90 IPs experienced no ECN marking in the 
upstream access link or the backhaul, but they did experience ECN 
marking somewhere further into the network (which could include another 
access link for p2p traffic).

The results for the 38 IPs backhauled via the FQ-CoDel nodes (known to 
support ECN) are even more unexpected (again taking only the TCP 
results, because they show both directions):
* In the direction from the LAN, 10 of these 38 IPs saw no ECN marking 
from the LAN side (including none from those FQ-CoDel nodes), but they 
did see the feedback of ECN marking from somewhere deeper into the network.
* The results in the direction from the WAN are more what I would 
expect: 30 of the 38 IPs saw no ECN marking from the WAN side, but they 
did see the feedback of ECN marking that must have occurred on the LAN 
side (where the FQ-CoDel nodes are).


> Pete
>> Of the 90 IPs in the TCP results, 40 from the WAN have this pathology
>> and 38 from the LAN
>> (taking the direction as that of the data)
>> Only 16 IPs, don't have this pathology in either direction.
>> I would have thought you guys would have picked up these when
>> checking
>> for the ECE/CE >= 2
>> (and/or zeros).
>> If I include the pathology you mention where ECE/CE < 2
>> In all, of the 90 IPs in the TCP results, after I exclude any IP with
>> a
>> pathology, I only have 7 IPs left.
>> Bob
>> PS. I agree it's v strange for an ISP to share capacity between its
>> members by how many flows they are using.
>> On 08/03/2021 22:19, Pete Heist wrote:
>>> I'm following up in this thread to one question that's come up- why
>>> is
>>> an ISP deploying fq_codel in its backhaul?
>>> In short, this ISP encourages research and experiments by its
>>> members,
>>> and this was a way to both see how much congestion exists, and
>>> attempt
>>> to manage it with AQM when needed. Experiments with fq_codel were
>>> started a couple of years ago, and I helped add support for it to
>>> their
>>> custom router interface early last year. However, it's only
>>> actually
>>> been enabled on a couple of routers, because:
>>> - it takes a while to update routers in this environment
>>> - some of the routers run kernels prior to fq_codel support
>>> - congestion doesn't usually last long enough to give it much
>>> urgency,
>>> and sometimes it's fixed by just adding capacity
>>> IMO, fq_codel isn't ideal in this situation. When there actually is
>>> congestion to manage, flow-fairness means that one user can
>>> dominate
>>> the queue with many flows, which is exactly one of the problems
>>> we'd
>>> like to avoid. Host fairness could improve on this, or better yet,
>>> member fairness using the member database, as members can have more
>>> than one IP/MAC. But, fq_codel is what's available now, and
>>> generally
>>> usable enough in this case to get started with AQM.
>>> On Mon, 2021-03-08 at 11:09 +0100, Pete Heist wrote:
>>>> -------- Forwarded Message --------
>>>> Subject: New Version Notification for draft-heist-tsvwg-ecn-
>>>> deployment-
>>>> observations-02.txt
>>>> Date: Mon, 08 Mar 2021 01:57:07 -0800
>>>>> A new version of I-D, draft-heist-tsvwg-ecn-deployment-
>>>>> observations-
>>>>> 02.txt
>>>>> has been successfully submitted by Peter G. Heist and posted to
>>>>> the
>>>>> IETF repository.
>>>>> Name:           draft-heist-tsvwg-ecn-deployment-observations
>>>>> Revision:       02
>>>>> Title:          Explicit Congestion Notification (ECN)
>>>>> Deployment
>>>>> Observations
>>>>> Document date:  2021-03-08
>>>>> Group:          Individual Submission
>>>>> Pages:          30
>>>>> URL:
>>>>> Status:
>>>>> Html:
>>>>> Htmlized:
>>>>> Diff:
>>>>> Abstract:
>>>>>      This note presents data gathered at an Internet Service
>>>>> Provider's
>>>>>      gateway on the observed deployment and usage of ECN.
>>>>> Relevant
>>>>> IP
>>>>>      counter and flow tracking data was collected and analyzed
>>>>> for
>>>>> TCP
>>>>> and other protocols.
>>>> A few updates in this version:
>>>> * Improved statistics for the separated "known" and "unknown"
>>>> AQMs.
>>>> * Clarified the detection method for possible AQMs.
>>>> * Added possibility of QUIC-ECN traffic in non-TCP data.
>>>> Pete

Bob Briscoe