Re: [tcpm] [EXTERNAL] Re: Seeking WG opinions on ACKing ACKs withgood cause

[Ilpo Järvinen <[e-mail]>]

Hi Neal,

I'm trying to understand your scenarios and I fail to see consistency here 
and there (which makes me wonder if I got you right at all).

On Sun, 11 Jul 2021, Neal Cardwell wrote:
> On Sun, Jul 11, 2021 at 11:55 AM Christian Huitema <[e-mail]>
> wrote:
>        
> 
>       > On Jul 11, 2021, at 4:42 AM, Bob Briscoe <[e-mail]>
>       wrote:
> 
> ... 
>       > The implementation will have its own ACK ratio - that's out of
>       scope of AccECN, except to set this max of 6 CEs, which is to
>       mitigate wrap of the 3-bit counter of CE-marks (which in the
>       worst case of 100% marking in the data direction could then
>       induce 1 ACK per 6 packets). This shouldn't limit forward
>       performance, because it only increases the reverse ACK rate if
>       there is heavy congestion in the forward path, when the Data
>       Sender should be reducing the forward rate anyway.
> 
>       1 ACK per 6 packets would cause performance issues on high speed
>       links. Common setup is "4 to 8 ACK per RTT", which means
>       intervals much larger than 6 packets.
> 
> 
> I share this concern about the AccECN requirement of one ACK per 6 CE-marked
> data segments potentially causing performance issues with high-speed links.
> Today, with most high-speed last-hop link technologies (wifi, cellular,
> Ethernet) TCP receivers often receive (from lower layers of the hardware and
> software networking stack) large aggregates  (often up to 64KBytes or around
> 44 packets), and generate a single ACK for that aggregate. AFAICT changing
> this scenario to require 1 packet per 6 CE-marked data segments means that
> during CE-marked scenarios receiving such an aggregate would require
> ceil(44/6) = 8 ACKs, increasing the number of ACKs by up to 8x. 
> 
> I have two main concerns in that scenario:
> 
> (1) CPU load: This seems like it would impose much higher CPU load on the
> data receiver, at just the moment when the receiver may already be under
> stress due to receiving data near the maximum rate of the link.

Those would be back-to-back ACKs in response to a 44 seg blob? At that 
point with warm caches & context, essentially just duplicating the same 
ACK with a few fields varied a little? Is "much higher" perhaps an 
overstatement?

I'd also tend to think link and CPU are not bottleneck at the same time 
(but I admit I might have dealt with too idealized environments to 
experience that in practice).

This next doesn't relate directly to your concern about CPU load but only
to effect of extra ACK on link performance. I remember discussing with 
Yuchung (perhaps you were there at that time too) some years back in some 
IETF about making more ACKs appear on the wire using some kind of ACK 
sending offloading (back then, main motivation for me was to increase SACK 
reduncancy). Yuchung told me that it would be really bad for things like 
Wifi. I now wonder if it really relates more to the cases where ACKs are 
roughly evenly spread out over the RTT preventing efficient use of the 
wireless channel or does that apply also to cases where they're very 
unevenly spread (back-to-back)?

And for the record, sending x ACKs was not the way I interpreted the
particular draft text. I took it just a rule when at latest, the ACK
should be generated. I just took it as "Since we got 44 CE marks, we
MUST send ACK now". (It seems to have be slightly differently worded
in -09 where my interpretation originates from).

> (2) Congestion and ACK loss: This requirement for generating up to 8 ACKs
> per aggregate seems like it is likely to produce a tight burst of 8 ACKs,
> which is going to increase congestion and increase the odds of losing at
> least one ACK, which is going to cause accuracy problems for "Accurate" ECN.
> :-)

So instead of 44 CE marks, 44 or 36 are detected (depending on how the 
sender treats safe/unsafe on that ACK after the loss)?

To me more challenging are the cases where ACE delta is zero but a wrap 
has occurred because zero usually means there's no CE event going on 
at all.

> I understand that this proposed "ACK per 6 CE-marked data segments" rule is
> necessary to avoid issues with the 3-bit ACE field wrapping. So IMHO this is
> one of the good arguments against including the ACE field in the AccECN
> design. 
> 
> The other main concerns I have with the ACE field are:
> 
> o Complexity

I hope you don't really mean ACE field itself is complex? Most of the 
complexity related to the header bits is due to the negotiation.

> o Redundancy with respect to the AccECN counter options

I ended up using ACE field to improve byte counter estimation between 
AccECN option arrivals when AccECN option is not put into every packet.
It makes the corrections to sender's estimation of CE (delivered_ce) 
smoother as the option eventually arrives. It is of course not necessary
if AccECN option is always present.

> o Potential problems with middleboxes
> 
> o Known problems with NICs and drivers (based on Ilpo's nice talk, "Accurate
> ECN Linux Implementation: Experiences and Challenges", at the April 2020
> TCPM interim meeting)

Are you saying that the offloading engine don't have a toggle for the
CWR clearing logic? (I didn't claim to know).

I've since then thought more about the offloading cases.
For sending, the headers remain the same so the feedback doesn't matter.
On receiving, I first looked into GRO expecting some impact, however, it 
turns out that advancing ACK sequence triggers a flush. As no cumulative 
ACKs get combined, ACE field changes did not trigger additional flushing 
in practice.

What I don't know, is if HW has ACK receiver offloading which combines ACK 
with different ACK sequence numbers (in the end, this works same as ACK 
coalescing). That might be the only case where clearly more ACKs would 
be passed upwards to processing when ACE field changes for a continuous 
run of CEs. In randomly spread CEs case, however, DCTCP feedback produces 
edges that is roughly similar in number so it's hard to see how ACE field 
would result in unbearable number of ACKs to process compared with DCTCP 
feedback.

> I realize that if AccECN does not have the ACE field feature, then AccECN
> and TCP L4S will not be usable on paths with middleboxes that strip the
> AccECN counter options. But IMHO living without the ACE field is preferable.
> IMHO it's acceptable to say that L4S can only be used with (a) QUIC, or (b)
> TCP connections where no middleboxes are stripping AccECN options.

This is just trading off "potential problems with middleboxes" with 
other potential problems with middleboxes.

I'm not saying here AccECN with option isn't good but it seems that you 
say you have concern x on ACE and then accept AccECN option as the 
solution but it has rather similar challenges related to x.


-- 
 i.