Re: [tcpm] draft-ietf-tcpm-accurate-ecn-24 addressing all WGLC comments

[Bob Briscoe <[e-mail]>]

Markku,

Sorry, it's taken a week to build a comprehensive reply to this long 
email. See inline tagged [BB]...

On 17/05/2023 12:24, Markku Kojo wrote:
> Hi Michael, all,
>
> On Sun, 14 May 2023, [e-mail] wrote:
>
>>> On 30. Mar 2023, at 16:53, Bob Briscoe <[e-mail]> wrote:
>>>
>>> Michael, Yoshi, Ian (as tcpm chairs),
>>>
>>> To close off the WGLC, I have just posted a new rev of accurate-ecn. 
>>> Hyperlinks quoted at the end.
>>> You will see the diff is rather extensive. I won't give a summary of 
>>> all the diffs like I usually do. Instead I can just refer to the 
>>> summary I gave in the presentation on Monday:
>>> https://datatracker.ietf.org/meeting/116/materials/slides-116-tcpm-draft-ietf-tcpm-accurate-ecn 
>>>
>>>
>>> Thank you again to the people who reviewed this during the WGLC:
>>> Michael Tüxen, Alex Burr, Gorry Fairhurst and Markku Kojo.
>>>
>>> All changes are editorial, apart from removing the para about not 
>>> mistaking certain ACKs of ACKs for DupACKs, which I will add to a 
>>> rev of the ECN++ draft, hopefully later this week.
>>>
>>> On the list, we have seen agreement from all the reviewers to these 
>>> changes, except no response from Markku yet.
>>> On Monday, I told Markku that I would post the draft in a few days, 
>>> so everyone can see the updates and diff.
>> Anyone having additional comments? In particular Markku regarding 
>> loss recovery?
>
> My apologies for being late with my reply to the author's comments on 
> my review (I've been extremly busy with other issues since the wg mtng 
> in Yokohama, including the rest of mtng week).
>
> I don't have much new comments but it seems that my major concern 
> regarding the problem of sending ACKs of ACKs was not fully understood.
>
> The first thing where I think I was not quite clear is that the major 
> problem with ACKs of ACKs is not that a pure Ack is made ECN-capable. 
> Instead, the problem is in generating an Ack of an pure Ack and that 
> is what one should prohibit to avoid problems. I understand that it 
> might be problematic to formulate rules whether generating an Ack of 
> an Ack is allowed (and when), instead of just disabling sending 
> ECN-capable ACKs.
> I don't have a strong opinion which way the problems with ACKs of ACKs 
> is avoided as long as they are avoided.

[BB] See later after your similar point (following your 'Why?' heading)...

>
> I am preparing a few scenarios to illustriate the problems ACKs of 
> ACks raise and will send them shortly once I have formulated a more 
> thorough reasoning why sending ACKs of ACKs is not really a good idea 
> and even seems to be unnecessary in most if not all cases, i.e., it 
> just results in sending unnecessary packets with not much useful 
> effect but creates a notable number of problems.

[BB] Having waited this long, it's rather disappointing to still hear 
you say "I have an argument, but I'll tell you later."

> It also seems not have been carefully enough considered in terms of 
> the very basic rubustness principle of "be conservative in what you 
> send ..."

[BB] The WG has been careful to ensure that ACKs of ACKs are unambiguous 
(cannot be mistaken for a DupACK), which is what the robustness 
principle requires. It's just that you think we have missed cases where 
they will be ambiguous. If you think that, we need to hear them all.

The robustness principle does not advocate sending nothing just in case 
some unknown factor might make it ambiguous. Especially given /not/ 
feeding back congestion notifications has potential to cause harm to 
others. Also, "no feedback" is much more ambiguous.

>
> Given that this draft is intended to become a stds track RFC I am 
> concerned of any text in this document that indicates (or even hints) 
> that TCP could acknowledge pure ACKS (this holds particularly the 
> rules and text in Sec 3.2.2.5.1 for the "Increment-Triggered ACKs"). 
> If it is seen necessary that this doc should have such pieces of rules 
> and text, I am fine if any such text is moved to an appendix as long 
> as the appendix makes it cristal clear that the text is valid only in 
> case one is implementing an experiment as per 
> [I-D.ietf-tcpm-generalized-ecn].

[BB] See point below about "Generic (Mechanistic) Reflector".

>
> Why?
>
> 1) It is well known that TCP does not acknowledge ACKs and Standards 
> track TCP has not been specified to acknowledge ACKs. This means that 
> a reader/implementer of this doc cannot correctly understand the rule 
> for "Increment-Triggered ACKs" unless there is a normative reference 
> to a spec that specifies ACKs of ACKs (or tells that it is even 
> possible).

[BB] ACKs of ACKs can indeed be tricky. But there's no need to consider 
not ACKing ACKs as an architectural principle. Not Acking ACKs on 
principle certainly avoids some tricky problems. However, we have a new 
situation here where, in limited circumstances, ACKs of ACKs are 
necessary. So the WG has already worked through the tricky problems and 
they have been addressed in the draft (e.g. mistaking ACKs of ACKs for 
DupACks, infinite ping-pong, etc). We'll discuss below whether you've 
found some more trickiness.

What is the new situation?

  * Until ECN was introduced, TCP ACKs only acknowledged data. So there
    was no need to acknowledge pure ACKs, which contain no data.
  * When ECN was introduced in RFC3168, TCP ACKs also acknowledged ECN
    markings. However, because RFC3168 precluded pure ACKs from being
    ECN-capable, there was still no need to acknowledge pure ACKs.
  * RFC5690, and now the ECN++ draft introduce the possibility of
    ECN-capable pure ACKs. So, in the limited circumstances described in
    the AccECN draft, ECN-capable pure ACKs now need to be acknowledged,
    because they contain new information - their ECN field.

Similarly, even though the final ACK of TCP's 3WHS is an ACK of an ACK , 
it is sent because it is needed (to prove that the SYN wasn't from a 
spoofed address).

It is true that not ACKing ACKs is well-known. However, whether it's 
well-known as a /principle/, or just as a current /feature/ of TCP is 
not clear. Anyway, the IETF's job is to update RFCs that are 
"well-known". We don't have to jump through any special procedural hoops 
to do something different from what is "well-known". Even if it were 
prohibited in a stds track RFC, we just have to specify what has to be 
done instead; in another stds track RFC.

If there are any tutorials, course notes or text books out there that 
say that not ACKing ACKs is a well-known principle, that's not the 
IETF's problem. It is the job of the tutors, lecturers and text book 
authors who wrote those materials to update them.

>
> 2) ACKs of ACKs tend to trigger duplicate Acks. There are tons of 
> algorithms that rely on the packet conservation principle and the fact 
> that TCP never injects a dupAck unless a *data* packet has arrived and 
> left the network. This is enforced with "MUST NOT" in RFC 5681, Sec 
> 4.2, because not conforming to this rule makes any algorithm that rely 
> on the rule to work incorrectly. These algorithms include (triggering) 
> Fast Retransmit, (controlling packet rate during) Fast Recovery, 
> (detecting spurious RTOs in) F-RTO, (calculating PipeAck in) RFC 7661, 
> (calculating DeliveredData in) PRR, etc. Furthermore, it would make 
> imposible to come up with any new algorihms that rely on this 
> important basic rule. In most cases such extra dupAcks make these 
> algorithms too aggressive because any extra dupAck is likely to inject 
> extra packet(s) to the network.
>
> So, it should be cristal clear that without SACK (or Timestamps) a TCP 
> *MUST NOT* send ACKs of ACKs.

[BB] Constraining the /Data Receiver/ as you propose would create an 
interop problem.
Explanation: Consider host A and B are not using SACK or timestamps. 
Nonetheless, with your approach, host A can still send ECN-capable pure 
ACKs to host B. Then, your rule puts host B in an impossible position, 
where it gets congestion notifications on ECN-capable pure ACKs, but it 
is not allowed to send any feedback about them.

Instead, if neither timestamps nor SACK are in use for the connection, 
we need to constrain the /Data Sender/ of a half connection from sending 
ECN-capable ACKs in the first place. This is the approach the WG has 
adopted in the AccECN and ECN++ specs.

Specifically:

  * The WG makes sure that RFCs about the /Data Sender/ of a half
    connection (e.g. the ECN++ experiment or other future RFCs) specify
    that sending ECN-capable pure ACKs is conditional on having another
    way to distinguish DupACKs, e.g. negotiating SACK or timestamps (and
    I will respond to your later points on the details of these).
  * The AccECN spec (which primarily specifies the feedback behaviour of
    a /Data Receiver/ in a half-connection) then only needs to define
    the Increment-triggered ACK rule.

The two together lead to the same outcome you want. But without the 
interop hole of your approach.

This is consistent with the "Generic (Mechanistic) Reflector" approach 
of the AccECN spec which says:
"AccECN is designed to be a generic reflector of whatever ECN markings 
it sees, whether or not they are compliant with a current standard."

These ACKs of ACKs are generically necessary to feed back congestion 
notifications from possible incoming packet patterns, not specifically 
for ECN++ or AckCC [RFC5690], or any other future RFC (forward 
compatibility). We'll edit the reference to ECN++ to make it clearer 
that it's one example, not the only case.

Here's another example of the generic reflector approach, already in the 
draft:
"Although RFC 3168 prohibits an ECN-capable SYN, providing feedback of 
ECN marking on the SYN supports future scenarios in which SYNs might be 
ECN-enabled (without prejudging whether they ought to be). ... "


> So, it should be cristal clear that without SACK (or Timestamps) a TCP 
> *MUST NOT* send ACKs of ACKs. 

I understand that you want this but, as just explained, without SACK or 
timestamps, the correct approach is to prevent the Data Sender putting 
the Data Receiver in the position where it would have to ACK ACKs in the 
first place.

In a connection without SACK or timestamps, if the Data Receiver were to 
get lots of congestion notifications on ECN-capable ACKs, it would face 
a difficult dilemma. Which would be more important: Signalling 
congestion by ACKing ACKs? or ensuring the performance improvements like 
Fast Retransmit, Fast Recovery etc. work well? The former would prevent 
harm to others, the latter would prevent harm to self.

Nonetheless, I will add some text to the AccECN draft that explains why 
it is important for other RFCs not to put a Data Receiver in the 
position where it has to ACK ACKs iff there is no way to distinguish 
them from DupACKs.

>
> 3) Even with SACK or Timestamps enabled it is not clear what an
> implementer should do. With SACK the AckECN authors seem to make an 
> assumption, which seems obvious but is not, that an ACK of ACK would 
> would never include SACK option and hence it could be distinguished 
> from a dupAck. However, RFC 2018 specifies: "If sent at all, SACK 
> options SHOULD be included in all ACKs which do not ACK the highest 
> sequence number in the data receiver's queue. So, if there is a hole 
> in the receiver's queue, the assumption is incorrect and it is unclear 
> which SACK info to include into the SACK option. Whatever one selects 
> to include, it makes DSACK (RFC 2883] void and breaks any DSACK-based 
> algorithms unless RFC 2018 is updated.

[BB]
Reading the draft, it is very clear that there is no such assumption. 
The text said solely what it meant:

    ... a host in AccECN
    mode that is sending ECN-capable pure ACKs SHOULD add one of the
    following additional checks when it tests whether an incoming pure
    ACK is a duplicate:

    o  If SACK has been negotiated for the connection, but there is no
       SACK option on the incoming pure ACK, it is not a duplicate;

That is, if an incoming ACK were a duplicate, it would have a SACK 
option on it. This /relies/ on the rule in RFC2018 that you quote. So 
there is no need (nor intention) to change SACK behaviour in any way.

(Note: to check this text, you'll need to refer to the previous AccECN 
draft here:
https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-accurate-ecn-23#section-3.2.2.5.1
It had been there since Jul 2021 (since -15). But, as explained above, 
it is longer in the latest AccECN draft (-24), because it has been moved 
to the editor's copy of the ECN++ draft, which I wrote on 4 Apr, but 
I've been waiting for your reply before submitting it.)

>
> With Timestamps some algorithms like Eifel detection breaks. Moreover, 
> there are other existing or potentially to-be-created heauristics, 
> including various measurement tools, that rely on the fact that TCP 
> does not echo a later Timestamp in a pure Ack than what arrived with 
> the latest data packet. Any such mechanisms are subject to break.

[BB] I don't fully understand what you're saying here. Can you be 
clearer please?
And can you please bear in mind that we are in the WGLC processing stage 
now. So review comments ought to be suggesting very specific changes to 
the text under review.

Again, this text about extra DupACK checks has now been removed from 
AccECN and will shortly appear in the ECN++ draft instead. I shall post 
the new ECN++ draft shortly.

>
> It might be good to hold discussing any details on what breaks and 
> how/why and what are the consequences until I have sent my reply with 
> scenarios to Richard.

[BB] Please try to prioritize any comments about the text that is now 
left in the AccECN draft. The WGLC of AccECN is waiting for no-one else 
at the moment.

But also bear in mind that the chairs plan to take ECN++ into WGLC once 
AccECN WGLC has been cleared. So we need to hear your actual argument 
about the DupACK text that has been moved to ECN++ urgently too. We 
can't work with "I have an argument, but I'll tell you later".

>
> One additional comment regarding the "Change-Triggered ACKs" rule is 
> that it would be useful to make it more clear how this plays with 
> delayed Acks and how it alters acknowledgement rate.
> I am not sure that what the draft currently says is quite correct:
>
>  "The approach can lead to some additional ACKs but it feeds back
>   the timing and the order in which ECN marks are received with minimal
>   additional complexity. If CE marks are infrequent, as is the case for
>   most AQMs at the time of writing, or there are multiple marks in a row,
>   the additional load will be low.
>
> For example, consider a scenario with bidirectional traffic between A 
> and B where B has a hole in sequence resulting in every data packet in 
> the current RTT to become acked (pure duplicate Acks). This may result 
> in a packet flow from A to B where every second packet is a pure 
> (duplicate) Ack. If there is congestion on the path from A to B such 
> that a significant number of (data) packets get marked, it may result 
> in acking every data packet from A to B. This does not necessarily 
> result in low additional load?

[BB] I don't quite understand how every second packet from A to B is a 
pure duplicate ACK, but I don't think I need to - I'll assume it's 
somehow possible.

Then I think you've somehow assumed that the data packets get CE-marked, 
but the interspersed pure ACKs don't (perhaps you're assuming that the 
pure ACKs in this case are not ECN-capable? Or perhaps you're assuming 
size-based packet marking?). Whatever, I agree that, if this scenario 
did occur, then the change-triggered ACK rule would indeed lead to B 
ACKing every data packet that arrives, with no delayed ACKs. {Note 1}

Nonetheless, the draft is quite open about the implications of the 
change-triggered ACK rule on ACK rate. In the sentence straight after 
the ones you quote, it says:
     "However, marking patterns with numerous non-contiguous CE marks 
could increase the load significantly."
And a little earlier it starts out by saying:
     "...the 'Change-Triggered ACKs' rule could sometimes cause the ACK 
rate to be problematic for high performance"

I don't think we need to include examples of how non-contiguous CE 
marking could occur. And if we did, I'd prefer to use one that was less 
complex to explain, e.g. a high level of probabilistic AQM marking. But 
thank you for this point anyway.


{Note 1}: It's ironic that the existing behaviour "where B has a hole in 
sequence" also results "in every data packet in the current RTT to 
become acked" (by A). I'm not giving this as an excuse for introducing 
another case with the same bad behaviour. I'm just highlighting the irony.


Regards


Bob

>
> Best regards,
>
> /Markku
>
>
>> Best regards
>> Michael
>>>
>>> Cheers
>>>
>>>
>>> Bob
>>>
>>> On 30/03/2023 15:15, [e-mail] wrote:
>>>> A New Internet-Draft is available from the on-line Internet-Drafts
>>>> directories. This Internet-Draft is a work item of the TCP 
>>>> Maintenance and
>>>> Minor Extensions (TCPM) WG of the IETF.
>>>>
>>>>    Title           : More Accurate Explicit Congestion Notification 
>>>> (ECN) Feedback in TCP
>>>>    Authors         : Bob Briscoe
>>>>                      Mirja Kühlewind
>>>>                      Richard Scheffenegger
>>>>    Filename        : draft-ietf-tcpm-accurate-ecn-24.txt
>>>>    Pages           : 64
>>>>    Date            : 2023-03-30
>>>>
>>>> Abstract:
>>>>    Explicit Congestion Notification (ECN) is a mechanism where network
>>>>    nodes can mark IP packets instead of dropping them to indicate
>>>>    incipient congestion to the endpoints.  Receivers with an 
>>>> ECN-capable
>>>>    transport protocol feed back this information to the sender.  
>>>> ECN was
>>>>    originally specified for TCP in such a way that only one feedback
>>>>    signal can be transmitted per Round-Trip Time (RTT). Recent new TCP
>>>>    mechanisms like Congestion Exposure (ConEx), Data Center TCP 
>>>> (DCTCP)
>>>>    or Low Latency, Low Loss, and Scalable Throughput (L4S) need more
>>>>    accurate ECN feedback information whenever more than one marking is
>>>>    received in one RTT.  This document updates the original ECN
>>>>    specification in RFC 3168 to specify a scheme that provides more 
>>>> than
>>>>    one feedback signal per RTT in the TCP header.  Given TCP header
>>>>    space is scarce, it allocates a reserved header bit previously
>>>>    assigned to the ECN-Nonce.  It also overloads the two existing ECN
>>>>    flags in the TCP header.  The resulting extra space is exploited to
>>>>    feed back the IP-ECN field received during the 3-way handshake as
>>>>    well.  Supplementary feedback information can optionally be 
>>>> provided
>>>>    in two new TCP option alternatives, which are never used on the TCP
>>>>    SYN.  The document also specifies the treatment of this updated TCP
>>>>    wire protocol by middleboxes.
>>>>
>>>> The IETF datatracker status page for this Internet-Draft is:
>>>> https://datatracker.ietf.org/doc/draft-ietf-tcpm-accurate-ecn/
>>>>
>>>> There is also an htmlized version available at:
>>>> https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-accurate-ecn-24
>>>>
>>>> A diff from the previous version is available at:
>>>> https://author-tools.ietf.org/iddiff?url2=draft-ietf-tcpm-accurate-ecn-24 
>>>>
>>>>
>>>> Internet-Drafts are also available by rsync at 
>>>> rsync.ietf.org::internet-drafts
>>>>
>>>>
>>>> _______________________________________________
>>>> tcpm mailing list
>>>> [e-mail]
>>>> https://www.ietf.org/mailman/listinfo/tcpm
>>>
>>> -- 
>>> ________________________________________________________________
>>> Bob Briscoe http://bobbriscoe.net/
>>>
>>
>>

-- 
________________________________________________________________
Bob Briscoehttp://bobbriscoe.net/