Re: [tcpm] [EXTERNAL] Re: Seeking WG opinions on ACKing ACKs with good cause

Bob Briscoe <> Mon, 12 July 2021 12:30 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 8BB763A12E3 for <>; Mon, 12 Jul 2021 05:30:27 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -0.196
X-Spam-Status: No, score=-0.196 tagged_above=-999 required=5 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, NICE_REPLY_A=-0.001, RCVD_IN_DNSWL_BLOCKED=0.001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id JAlQdLrU0B99 for <>; Mon, 12 Jul 2021 05:30:22 -0700 (PDT)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 0EBBD3A12DF for <>; Mon, 12 Jul 2021 05:30:21 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=fYMXPBSNFzfIPm65WmKUWP2gIiHVCnT69bSi5VL4CPk=; b=0eJta7zUVHjv5zOWZCpMASUT01 UUZYxjUtfkH48i87ztliOEnLN9L2OsTyujyoYQhD1fcBAQZnUH80IIIyCyvWw2O2swfgxejwlM3l4 SWTRhIoSfIEHp0chuQe12xTOVd0LkOdVN2dGYiqXlot8s9cp9ihSXPTWxoNm5q+A7RDwMn1OTDljd IvLo1F9Xb2mijBcPHn6p1ofkGwAbQUPwmwh/fWlbLvnMlmjy5izg29c5FdF9vsMXVGquYKywc1UzH 80AettSryiyqivOCehYdL0B2xEgzrVAFE9VI5SV83vyZ34YWXclCel39Acl64fi75GSHG/uBjlT6m mAjstSoQ==;
Received: from ([]:33826 helo=[]) by with esmtpsa (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from <>) id 1m2v4n-0007MR-1R; Mon, 12 Jul 2021 13:30:19 +0100
To: Neal Cardwell <>
Cc:, Mirja Kuehlewind <>, Yuchung Cheng <>, Richard Scheffenegger <>, Christian Huitema <>
References: <> <> <>
From: Bob Briscoe <>
Message-ID: <>
Date: Mon, 12 Jul 2021 13:30:17 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0
MIME-Version: 1.0
In-Reply-To: <>
Content-Type: multipart/alternative; boundary="------------8D5073207E3D337646AE5982"
Content-Language: en-GB
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname -
X-AntiAbuse: Original Domain -
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain -
X-Get-Message-Sender-Via: authenticated_id:
Archived-At: <>
Subject: Re: [tcpm] [EXTERNAL] Re: Seeking WG opinions on ACKing ACKs with good cause
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Mon, 12 Jul 2021 12:30:28 -0000


On 11/07/2021 19:28, Neal Cardwell wrote:
> On Sun, Jul 11, 2021 at 11:55 AM Christian Huitema 
> < <>> wrote:
>     > On Jul 11, 2021, at 4:42 AM, Bob Briscoe <
>     <>> wrote:
> ...
>     > The implementation will have its own ACK ratio - that's out of
>     scope of AccECN, except to set this max of 6 CEs, which is to
>     mitigate wrap of the 3-bit counter of CE-marks (which in the worst
>     case of 100% marking in the data direction could then induce 1 ACK
>     per 6 packets). This shouldn't limit forward performance, because
>     it only increases the reverse ACK rate if there is heavy
>     congestion in the forward path, when the Data Sender should be
>     reducing the forward rate anyway.
>     1 ACK per 6 packets would cause performance issues on high speed
>     links. Common setup is "4 to 8 ACK per RTT", which means intervals
>     much larger than 6 packets.
> I share this concern about the AccECN requirement of one ACK per 6 
> CE-marked data segments potentially causing performance issues with 
> high-speed links. Today, with most high-speed last-hop link 
> technologies (wifi, cellular, Ethernet) TCP receivers often receive 
> (from lower layers of the hardware and software networking stack) 
> large aggregates  (often up to 64KBytes or around 44 packets), and 
> generate a single ACK for that aggregate. AFAICT changing this 
> scenario to require 1 packet per 6 CE-marked data segments means that 
> during CE-marked scenarios receiving such an aggregate would require 
> ceil(44/6) = 8 ACKs, increasing the number of ACKs by up to 8x.
> I have two main concerns in that scenario:
> (1) CPU load: This seems like it would impose much higher CPU load on 
> the data receiver, at just the moment when the receiver may already be 
> under stress due to receiving data near the maximum rate of the link.
> (2) Congestion and ACK loss: This requirement for generating up to 8 
> ACKs per aggregate seems like it is likely to produce a tight burst of 
> 8 ACKs, which is going to increase congestion and increase the odds of 
> losing at least one ACK, which is going to cause accuracy problems for 
> "Accurate" ECN. :-)

[BB] I hadn't intended this wording to apply in response to a large 
burst. I'd want to alter the wording to allow just one ACK in this case.

> I understand that this proposed "ACK per 6 CE-marked data segments" 
> rule is necessary to avoid issues with the 3-bit ACE field wrapping. 
> So IMHO this is one of the good arguments against including the ACE 
> field in the AccECN design.

[BB] I think the subtext here is a preference for the DCTCP style of 
feedback vs ACE (when there is no AccECN TCP Option). I don't think this 
particular issue is any different between the two.

With DCTCP feedback, when a large data burst like this arrives, if there 
are transitions to and from CE marking within the burst, does Linux 
DCTCP generate an ACK for each transition, and for every n repetitions 
of CE not ECT, like the RFC says it should? Or does the implementation 
just pop out one ACK at the end?

Ilpo & I have been doing experiments with high levels of ACK coalescing 
causing the ACE field seen by the Data Sender to wrap multiple times 
under high congestion. For example, 1 ACK per ~4ms is one common scheme, 
which would result in about 1 ACK for 34 data packets at just 100Mb/s.

> The other main concerns I have with the ACE field are:
> o Complexity

[BB] I understand there could be a tradeoff between complexity and 
accuracy. But I think the complexity of ACE and DCTCP feedback are 
pretty much the same but, with even low levels of ACK coalescing, the 
accuracy of ACE is superior. Nonetheless, I understand that deploying 
something different to DCTCP when you've already got DCTCP could involve 
deployment complexity.

> o Redundancy with respect to the AccECN counter options

[BB] The redundancy is intended. For cases where the TCP Option doesn't 
traverse the path, or where TCP option space is limited.

> o Potential problems with middleboxes

[BB] Well, we've tested the 3 header bits over millions of paths without 
problems. But yes, there are billions more to test.

> o Known problems with NICs and drivers (based on Ilpo's nice talk, 
> "Accurate ECN Linux Implementation: Experiences and Challenges", at 
> the April 2020 TCPM interim meeting)

[BB] I understood that talk as concluding there weren't really problems. 
What specific problem do you have in mind?

> I realize that if AccECN does not have the ACE field feature, then 
> AccECN and TCP L4S will not be usable on paths with middleboxes that 
> strip the AccECN counter options. But IMHO living without the ACE 
> field is preferable. IMHO it's acceptable to say that L4S can only be 
> used with (a) QUIC, or (b) TCP connections where no middleboxes are 
> stripping AccECN options.

[BB] When you say 'living without the ACE field', alongside the AccECN 
TCP Option, would you leave classic ECN feedback? Or put DCTCP feedback 
in its place?

I don't think there's any case for using classic ECN feedback within 
AccECN. I see the competition as between ACE and DCTCP feedback, each 
optionally with the AccECN TCP Option.

If DCTCP feedback were a stop-gap until we could get good traversal of 
the TCP Option, it might have some merit, but I think DCTCP isn't going 
to work well with the level of ACK coalescing in the Internet. However, 
this is very difficult to judge unless we do large scale A-B experiments 
on the real Internet (and potentially within DCs). I'll think further 
about enabling some way to do that, possibly within AccECN's negotiation 
framework. But today, I have to prioritize for the draft deadline.



> cheers,
> neal

Bob Briscoe