Re: [tcpm] draft-ietf-tcpm-accurate-ecn-24 addressing all WGLC comments

[Bob Briscoe <[e-mail]>]

Markku,

I'm top-posting 4 responses to all your points, in order to consolidate 
everything and avoid all the repetition:

#1) Generic should not be confused with experimental.
#2) Completeness
#3) Does not specify the necessary details?
#4) Not generic enough?


    #1) Generic should not be confused with experimental.


The AccECN spec is the IETF's PS spec for how TCP feeds back ECN on any 
packet. The requirement for AccECN to provide forward compatibility by 
providing generic feedback was agreed by this WG and set down in RFC7560 
(nearly 8 years ago). Generic mechanistic reflection is a central design 
principle of AccECN - to reflect the ECN field of SYNs, SYN-ACKs, ACKs, 
data packets, retransmissions, everything (§2.5 of the draft has said 
this since its inception). AccECN provides accurate feedback 
irrespective of whether or how it is currently used, so that it can be 
used if needed.
Mechanistic reflection is generic, not experimental.
     Analogy: Senders have conducted many CC experiments using feedback 
from stds-track rcvrs. But that doesn't make the rcvrs experimental; 
rather, it proves the f/b is generic.

You say an AccECN receiver automatically becomes part of an experiment 
as soon as the its peer enables ECN-capable ACKs. More constructively, 
one can say that the peer conducts the experiment by building on the 
generic behaviour of the receiver. The latter is more correct because 
the behaviour of the receiver was already specified (even though it was 
not exercised); so it predated the experiment, and could be analysed and 
checked prior to any experiments.

This is protocol layering. Feedback is a 'sub-layer' of the transport 
layer that lies logically beneath the congestion control function of the 
transport layer and is intended to allow future evolution of sender 
behaviour. The RTCP protocol demonstrates this well, because it was 
specified separately from RTP.

You express concern that specifying a case where a stds track AccECN 
implementation has to ack ACKs gives it no control over whether it is 
dragged into an experiment being conducted by its peer.  But that's the 
deal for every generic capability. It gets used. Our task is to make 
sure that it can be used in this way. That's why we have to correctly 
specify the SACK condition. That's what we do in the IETF.


    #2) Completeness


A protocol spec should say what an implementation has to do in response 
to all possible inputs. This ensures that there is just one behaviour to 
be checked for flaws (including security flaws), rather than 
implementers inventing their own behaviours, each of which might lead to 
bugs or vulnerabilities if it receives unexpected inputs.


    #3) Does not specify the necessary details?


You say this spec doesn't define the necessary safety features. We can't 
work with "it needs a careful review for potential problems" then do 
nothing while we wait in case we might see another email from you. If 
you think a safety feature is missing, the only way the WG works is 
constructively: participants identify what's missing, and (ideally) 
define it.

You repeatedly refer forward to apparent problems with SACK, which 
sounds like you have identified that the protocol is broken. But, having 
read forward to the end, you agree that the SACK condition is more solid 
than using TSOpt. And the only concrete criticism I can find is that we 
don't specify what DSACK to send. Richard & I (and others) have worked 
through all the cases you have proposed (and others we have contrived 
ourselves), and we cannot find any case where an implementation has to 
do anything different from what the DSACK RFCs already say.

I had independently found the rule you point out in RFC 5681, Sec 2, 
against generating more than one ACK for every incoming [data] segment. 
We can say that the potential requirement to ack ACKs contravenes that 
rule, which is why the SACK condition is necessary. I know you are 
concerned that it might not be as simple as that. But, without evidence, 
how do we distinguish this from over-dramatisation? In the absence of 
anything specific, we can't just sit on our hands. As Yoshi says, these 
will probably be corner cases. If new problems surface, we will have to 
deal with them as they arise.


    #4) Not generic enough?


To your point about feedback from pure ACKs and data being combined, 
that's not driven by any particular CCA. If ACKs are ECT, AccECN is 
designed to make it possible for the Data Sender to estimate how much of 
the combined per-packet feedback is from ACKs and data, by using the 
per-byte feedback in the option to estimate the number of data packets. 
This might not be perfect, but it was recognised from the start that we 
would need to decide on compromises,, particularly given very limited 
header space.

Nothing in the increment-triggered ACK rule was designed around ECN++. 
Requiring the receiver to emit an ACK after 'n' CE marks is a fairly 
obvious way of  ensuring that the sender  gets timely feedback of 
multiple CE markings. 'n' has to be less than 7 to prevent AccECN's 
3-bit ACE field wrapping. And greater than 2 to avoid too many ACKs. 
Then, as I said, we noticed that could cause ping-pong, so we raised the 
min to 3 if the ACK would be in response to an ACK. That would be 
necessary for protocol completeness (see above) whether or not an ECN++ 
draft existed.

You raise a concern that allowing Acks of Acks might preclude future 
algos that rely on the absence of 'fake dupacks'. Indeed, you say that 
contravening the one ACK per data segment rule of RFC5681 "will break 
all mechanisms that rely on the behavior specified in RFC 2883, 
including RFC 3708 and (TLP of) RACK (RFC 8985) as well as any potential 
new and useful algos that rely on this rule." But I think this is all 
stated rather over-dramatically, because you admit that the SACK 
condition allows an implementation to identify 'fake dupacks'. So it can 
exclude them from all such present and future algos. Then this all 
becomes a non-issue.

_______________
I haven't added any further discussion to the thread below (except a 
couple of minor points). To check that the above 4 points 
comprehensively address all your [MK2] points. I've tagged each with 
[BB3] and pointer(s) to the relevant response(s) above.


On 11/07/2023 16:48, Markku Kojo wrote:
> Bob, please see [MK2] inline.
>
> And apologies for the long delay in replying, again.
>
> On Mon, 5 Jun 2023, Bob Briscoe wrote:
>
>> Markku, pls see [BB2] inline
>>
>> On 26/05/2023 22:56, Markku Kojo wrote:
>>       Bob,
>>
>>       My apologies you had to wait for the scenarios as it took much 
>> longer with my limited cycles than I thought.
>>       Anyways, please see my reply to Richard, some scenarios are 
>> also included there.
>>
>>       To keep things easier, it might be good to try to keep the 
>> discussion on Acks of Acks (mainly) in the thread
>>       with my reply to Richard.
>>
>>       However, see inline tagged [MK].
>>
>>       On Wed, 24 May 2023, Bob Briscoe wrote:
>>
>>             Markku,
>>
>>             Sorry, it's taken a week to build a comprehensive reply 
>> to this long email. See inline tagged
>>             [BB]...
>>
>>             On 17/05/2023 12:24, Markku Kojo wrote:
>>                   Hi Michael, all,
>>
>>                   On Sun, 14 May 2023, [e-mail] wrote:
>>
>>                               On 30. Mar 2023, at 16:53, Bob Briscoe 
>> <[e-mail]>
>>                               wrote:
>>
>>                               Michael, Yoshi, Ian (as tcpm chairs),
>>
>>                               To close off the WGLC, I have just 
>> posted a new rev of
>>                               accurate-ecn. Hyperlinks quoted at the 
>> end.
>>                               You will see the diff is rather 
>> extensive. I won't give a
>>                               summary of all the diffs like I usually 
>> do. Instead I can just
>>                               refer to the summary I gave in the 
>> presentation on Monday:
>>
>> https://datatracker.ietf.org/meeting/116/materials/slides-116-tcpm-draft-ietf-tcpm-accurate-ecn
>>
>>                               Thank you again to the people who 
>> reviewed this during the WGLC:
>>                               Michael Tüxen, Alex Burr, Gorry 
>> Fairhurst and Markku Kojo.
>>
>>                               All changes are editorial, apart from 
>> removing the para about
>>                               not mistaking certain ACKs of ACKs for 
>> DupACKs, which I will add
>>                               to a rev of the ECN++ draft, hopefully 
>> later this week.
>>
>>                               On the list, we have seen agreement 
>> from all the reviewers to
>>                               these changes, except no response from 
>> Markku yet.
>>                               On Monday, I told Markku that I would 
>> post the draft in a few
>>                               days, so everyone can see the updates 
>> and diff.
>>
>>                         Anyone having additional comments? In 
>> particular Markku regarding loss
>>                         recovery?
>>
>>
>>                   My apologies for being late with my reply to the 
>> author's comments on my review (I've
>>                   been extremly busy with other issues since the wg 
>> mtng in Yokohama, including the rest
>>                   of mtng week).
>>
>>                   I don't have much new comments but it seems that my 
>> major concern regarding the problem
>>                   of sending ACKs of ACKs was not fully understood.
>>
>>                   The first thing where I think I was not quite clear 
>> is that the major problem with ACKs
>>                   of ACKs is not that a pure Ack is made ECN-capable. 
>> Instead, the problem is in
>>                   generating an Ack of an pure Ack and that is what 
>> one should prohibit to avoid problems.
>>                   I understand that it might be problematic to 
>> formulate rules whether generating an Ack
>>                   of an Ack is allowed (and when), instead of just 
>> disabling sending ECN-capable ACKs.
>>                   I don't have a strong opinion which way the 
>> problems with ACKs of ACKs is avoided as
>>                   long as they are avoided.
>>
>>
>>             [BB] See later after your similar point (following your 
>> 'Why?' heading)...
>>
>>
>>                   I am preparing a few scenarios to illustriate the 
>> problems ACKs of ACks raise and will
>>                   send them shortly once I have formulated a more 
>> thorough reasoning why sending ACKs of
>>                   ACKs is not really a good idea and even seems to be 
>> unnecessary in most if not all
>>                   cases, i.e., it just results in sending unnecessary 
>> packets with not much useful effect
>>                   but creates a notable number of problems.
>>
>>
>>             [BB] Having waited this long, it's rather disappointing 
>> to still hear you say "I have an
>>             argument,
>>             but I'll tell you later."
>>
>>
>>       [MK] I understand. My sincere apologies again.
>>
>>                   Given that this draft is intended to become a stds 
>> track RFC I am concerned of any text
>>                   in this document that indicates (or even hints) 
>> that TCP could acknowledge pure ACKS
>>                   (this holds particularly the rules and text in Sec 
>> 3.2.2.5.1 for the
>>                   "Increment-Triggered ACKs"). If it is seen 
>> necessary that this doc should have such
>>                   pieces of rules and text, I am fine if any such 
>> text is moved to an appendix as long as
>>                   the appendix makes it cristal clear that the text 
>> is valid only in case one is
>>                   implementing an experiment as per 
>> [I-D.ietf-tcpm-generalized-ecn].
>>
>>
>>             [BB] See point below about "Generic (Mechanistic) 
>> Reflector".
>>
>>
>>                   Why?
>>
>>                   1) It is well known that TCP does not acknowledge 
>> ACKs and Standards track TCP has not
>>                   been specified to acknowledge ACKs. This means that 
>> a reader/implementer of this doc
>>                   cannot correctly understand the rule for 
>> "Increment-Triggered ACKs" unless there is a
>>                   normative reference to a spec that specifies ACKs 
>> of ACKs (or tells that it is even
>>                   possible).
>>
>>
>>             [BB] ACKs of ACKs can indeed be tricky. But there's no 
>> need to consider not ACKing ACKs as an
>>             architectural principle. Not Acking ACKs on principle 
>> certainly avoids some tricky problems.
>>             However, we have a new situation here where, in limited 
>> circumstances, ACKs of ACKs are
>>             necessary.
>>             So the WG has already worked through the tricky problems 
>> and they have been addressed in the
>>             draft
>>             (e.g. mistaking ACKs of ACKs for DupACks, infinite 
>> ping-pong, etc). We'll discuss below whether
>>             you've found some more trickiness.
>>
>>
>>       (MK] I did not mean to refer to any principle but, as I said, 
>> that a reader/implementer cannot correctly
>>       understand the rule for "Increment-Triggered ACKs" because it 
>> is well-known to her/him that TCP does not Ack
>>       ACKs. This fact is that one can ack ACKs is not specified in 
>> this doc
>>
>>
>> [BB2] The fact that one /can/ ack ACKs is specified in the AccECN 
>> draft. It clearly says:
>>     "Although TCP normally only ACKs newly delivered data, in this 
>> case it is mandatory for A to emit ACKs of ACKs
>> because they feed back new congestion state (useful in case B starts 
>> sending)."
>>
>> We'll also make it clearer that "in this case" means "in the case of 
>> ECN-capable ACKs". And we'll make it clear that it's
>> conditional on carefully following all the conditions in any draft 
>> that specifies ECN-capable ACKs.
>
> [MK2] Right, but this (acking Acks) is experimental behaviour, needed 
> only for those participating the experiment. However, this doc 
> specifies the behaviour in the Increment-Triggered ACKs rule: MUST 
> emit an ACK".
> If one implements this draft but is not interested in ECN++ nor any 
> other similar experiment that allows ECN-capable ACKs, one must still 
> implement these Acks of Acks but there is no specification nor 
> normative ref how to do it (correctly, including all the conditions). 
> If then this implementation is deployed in the wild, it automatically 
> becomes part of the experiment once the other endpoint enables 
> ECN-capable ACKs but without the necessary safety procedures. This doc 
> cannot depend on an experimental spec which explains how to do it and 
> what conditions to follow.

[BB3] See
#1) Generic should not be confused with experimental.
#2) Completeness
#3) Does not specify the necessary details?

>>       nor does this doc give a (normative) reference where it is 
>> specified,
>>
>>
>> [BB2] As already explained, there is no normative ref where acking 
>> ACKs is specified in general, because they are not
>> needed in general. They are specified whenever they are needed, 
>> including in the examples Yoshi pointed to (keepalive,
>> etc).
>
> [MK2] An implementation of this draft send Acks of Acks "randomly" and 
> possibly multiple of them in a row but this draft does not specify the 
> necessary details unlike you claim above.

[BB3] See #4) Does not specify the necessary details

>
> [MK2] Pls see also my reply to Yoshi. The specific examples Yoshi 
> mentions are sent in very limited cases and result in sending only a 
> single Ack of an Ack which is not causing any harm to others unlike 
> sending multiple "random" Acks of Acks.
>
>>       including the details on which TSecr value to add or which SACK 
>> info if any to include when acking a pure
>>       Ack.
>>
>>
>> [BB2] OK.  But I've had to read all the through SACK and TS specs to 
>> guess what complication you might have seen. If you
>> know what made you ask this question, it would be much easier if you 
>> gave us a clue, or even just told us explicitly.
>
> [MK2] Sorry for not being clear/detailed enough. Maybe this was too 
> obvious to me. See details below for SACK (and my reply to Richard for 
> Timestamps for which I believe you already figured out the problem).

[BB3] See #3) Does not specify the necessary details?

>
>> * For TS, I'm pretty sure the normal rules (for which TSecr value to 
>> include) apply.
>> * Similarly, for SACK, I can't see that anything new needs to be said.
>>
>>       It is easy to misinterpret the "Increment-Triggered ACKs", if 
>> one doesn't realize that pure Acks may be
>>       acked.
>>
>>
>> [BB2] We'll make it clearer in the rule that 'packet' intentionally 
>> includes packets without data.
>
> [MK2] But as I said earlier, it would specify experimental behavior in 
> a PS doc.

[BB3]  See: #1) Generic should not be confused with experimental.

>
>>             What is the new situation?
>>              *  Until ECN was introduced, TCP ACKs only acknowledged 
>> data. So there was no need to
>>             acknowledge
>>                 pure ACKs, which contain no data.
>>              *  When ECN was introduced in RFC3168, TCP ACKs also 
>> acknowledged ECN markings. However, because
>>                 RFC3168 precluded pure ACKs from being ECN-capable, 
>> there was still no need to acknowledge
>>             pure
>>                 ACKs.
>>              *  RFC5690, and now the ECN++ draft introduce the 
>> possibility of ECN-capable pure ACKs. So, in
>>             the
>>                 limited circumstances described in the AccECN draft, 
>> ECN-capable pure ACKs now need to be
>>                 acknowledged, because they contain new information - 
>> their ECN field.
>>             Similarly, even though the final ACK of TCP's 3WHS is an 
>> ACK of an ACK , it is sent because it is
>>             needed (to prove that the SYN wasn't from a spoofed 
>> address).
>>
>>
>>       [MK] All otherwise clear, but I disagree that the final ACK of 
>> TCP's 3WHS is an ACK of an ACK. It is required
>>       because SYNACK contains control data that eats one sequence 
>> number, i.e., it advances RCV.NXT at the client
>>       end and when the ACK arrives at the server it is needed to 
>> advance SND.UNA. Very different from Acks of Acks
>>       in this draft.
>>
>>
>> [BB2] Yes, not the same, but similar - ECN information is also 
>> control data. But agreed that it doesn't consume a
>> sequence no. Neither do keepalive probes or their ACKs.
>
> [MK2] Right, but the actual purpose of Acks is to acknowledge sequence 
> numbers (that have arrived) and the logic of many crucial TCP algos 
> depend on how packets are acked.

[BB3] Good protocol specs never define or limit the uses of the protocol 
(it would be pointless to try anyway).

If you only wanted ACKs to be used to acknowledge sequence numbers, you 
should have objected to RFC3168, which used ACKs to feed back ECN as 
well. Or you could go further back and object to SACK because it 
acknowledges gaps, not sequence numbers that have arrived.

Can we please constructively focus on what is necessary to make generic 
reflection work, rather than keep repeating that the sky might fall if 
we change anything.

> TCP keep-alives are sent in a very specific case only (pls see my 
> reply to Yoshi). And, keep-alives could include data but that is not 
> recommended (I don't quite know/remember why it was recommended like 
> this though but that is not impotant here).
>
>>             It is true that not ACKing ACKs is well-known. However, 
>> whether it's well-known as a /principle/,
>>             or
>>             just as a current /feature/ of TCP is not clear. Anyway, 
>> the IETF's job is to update RFCs that
>>             are
>>             "well-known". We don't have to jump through any special 
>> procedural hoops to do something
>>             different
>>             from what is "well-known". Even if it were prohibited in 
>> a stds track RFC, we just have to
>>             specify
>>             what has to be done instead; in another stds track RFC.
>>
>>
>>       [MK] Again, I didn't mention it as a /principle/ but as a 
>> crucial piece of information that the reader needs
>>       to be noted, that is, the things are now different from what is 
>> well-known.
>>
>>       Sure IETF's job is to update RFCs, but if one changes what is 
>> prohibited in a stds track RFC, one needs to
>>       understand the consequences and explain them as well as give 
>> the justification why the change can be done
>>       (without problems), instead of just specifying the change.
>>
>>
>> [BB2] If ack'ing ACKs is prohibited, pls say where. Otherwise, pls 
>> confirm that you are not aware of anywhere that
>> explicitly prohibits it.
>
> [MK2] Apologies for not repeating. Please see my original item 2) a 
> few rows below with my explanation included (i.e., RFC 5681, Sec 2, 
> last para).  Prohibiting a receiver from generating more than one ACK 
> for every incoming [data] segment is to guarantee the fundamental 
> well-known behavior of TCP which is used as a prerequisite in tons of 
> loss-recovery and CC mechanisms for TCP. If it is not honored, all 
> these mechanism stop working correctly. That's why it is "MUST NOT".

[BB3] See: #3) Does not specify the necessary details?

>
>> Whatever, I have no problem with explaining the consequences. As I've 
>> said, that's not relevant in AccECN, 'cos the
>> increment-triggered ACK rule doesn't cause ACKs of ACKs unless you're 
>> also making ACKs ECN-capable. So only relevant in
>> the ECN++ draft
>
> [MK2] Please note it is the other (remote) endpoint of the TCP 
> connection that makes ACKs ECN-capable and that remote endpoint 
> implementation is totally independent of the local endpoint 
> implementing the increment-triggered ACK rule and thereby completely 
> out of control for the local endpoint.

[BB3] See:
#1) Generic should not be confused with experimental.

>
>>             If there are any tutorials, course notes or text books 
>> out there that say that not ACKing ACKs is
>>             a
>>             well-known principle, that's not the IETF's problem. It 
>> is the job of the tutors, lecturers and
>>             text
>>             book authors who wrote those materials to update them.
>>
>>
>>                   2) ACKs of ACKs tend to trigger duplicate Acks. 
>> There are tons of algorithms that rely
>>                   on the packet conservation principle and the fact 
>> that TCP never injects a dupAck unless
>>                   a *data* packet has arrived and left the network. 
>> This is enforced with "MUST NOT" in
>>                   RFC 5681, Sec 4.2, because not conforming to this 
>> rule makes any algorithm that rely on
>>                   the rule to work incorrectly. These algorithms 
>> include (triggering) Fast Retransmit,
>>                   (controlling packet rate during) Fast Recovery, 
>> (detecting spurious RTOs in) F-RTO,
>>                   (calculating PipeAck in) RFC 7661, (calculating 
>> DeliveredData in) PRR, etc. Furthermore,
>>                   it would make imposible to come up with any new 
>> algorihms that rely on this important
>>                   basic rule. In most cases such extra dupAcks make 
>> these algorithms too aggressive
>>                   because any extra dupAck is likely to inject extra 
>> packet(s) to the network.
>>
>>                   So, it should be cristal clear that without SACK 
>> (or Timestamps) a TCP *MUST NOT* send
>>                   ACKs of ACKs.
>>
>>
>>             [BB] Constraining the /Data Receiver/ as you propose 
>> would create an interop problem.
>>             Explanation: Consider host A and B are not using SACK or 
>> timestamps. Nonetheless, with your
>>             approach, host A can still send ECN-capable pure ACKs to 
>> host B. Then, your rule puts host B in
>>             an
>>             impossible position, where it gets congestion 
>> notifications on ECN-capable pure ACKs, but it is
>>             not
>>             allowed to send any feedback about them.
>>
>>             Instead, if neither timestamps nor SACK are in use for 
>> the connection, we need to constrain the
>>             /Data Sender/ of a half connection from sending 
>> ECN-capable ACKs in the first place. This is the
>>             approach the WG has adopted in the AccECN and ECN++ specs.
>>
>>
>>       [MK] I think I said in the beginning that I have no strong 
>> opinion which way Acks of Acks are disabled.
>>       However, I apologize that I didn't explain why I phrased MUST 
>> NOT send ACKs of ACKs. This is because it might
>>       be still useful to allow CE-marked pure Acks and take care of 
>> Ack CC by some other means than Acks of Acks.
>>       Currently the draft mandates Acks of Acks as the only way to 
>> report Ack congestion and I think it is too
>>       restrictive in a stds track doc, e.g., it rules out reducing 
>> Ack rate simply by reducing data send rate which
>>       would solve the interop problem in a very simple way. Moreover, 
>> When B gets congestion notifications on
>>       ECN-capable pure ACKs, not sending Acks of Acks does not 
>> prevent sending feedback; such feedback need not to
>>       be delivered immediately but by the time needed. Please see 
>> more on this in my reply to Richard.
>>
>>
>> [BB2] I think you've got the wrong end of the stick. The 
>> increment-triggered ACK rule is not, and never was, motivated by
>> AckCC. We wrote the 2 rules (Change-triggered and 
>> Increment-triggered) before someone pointed out that we'd used the
>> wording 'packet', which included pure ACKs. We changed one occurrence 
>> to data packet, but realized that the other
>> occurrence was correctly stated as just 'packet', because it fed back 
>> more timely congestion information.
>
> [MK2] I understand that the rule was not motivated nor written by 
> AckCC, but the result of the currently written rule is that it 
> prevents from providing feedback for AckCC purposes in some other 
> reasonable way. That is, it mandates sending the Acks of Acks even in 
> case one would have alternative/better way to provide the feedback (or 
> does not require any feedback).
>
>> Congestion information on pure ACKs doesn't mean the pure ACKs are 
>> the sole cause of congestion; it just means that they
>> are part of an aggregate that is congesting a bottleneck. CE on pure 
>> ACKs conveys information that is useful for
>> maintaining cwnd when no data is being sent (much better than CWV 
>> which relies on magic numbers to guess how to decay
>> information during idle periods).
>
> [MK2] I think I well understand the thinking behind this but it seems 
> to me that these Acks of Acks are still mostly unnecessary or late 
> (and probably stale) information. Please see what I replied to 
> Richard, trying to explain why.
>
>> I'm not particularly interested in AckCC, altho I suppose I am 
>> separately interested in draft-ietf-tcpm-ack-rate-request
>> and the equivalent in QUIC, because I want QUIC (and TCP) to be able 
>> to thin their ACK stream themselves, rather than
>> relying on ACK thinning in the network.
>
> [MK2] Agreed that it's better that TCP (and QUIC) thin their ACK 
> stream themselves. This said, I think that this doc should not 
> restrict (or decide in advance) as a stds track doc the way how 
> ECN-capable and CE-marked pure Acks are best fed back to a data 
> receiver. How to (best) implement CC with ECN-capable Acks is 
> obviously experimental (if not research). I believe we could agree on 
> this?

[BB3] We can agree that implementing CC with ECN-capable ACKs is 
experimental. But that doesn't stop us defining a stds track feedback 
capability that is generic enough for both current usage and future 
experiments.

See: #1) Generic should not be confused with experimental.

>
> [MK2] The current text mandates one particular way to feed back 
> congestion on pure Acks by bundling it together with the feedback for 
> data segments that experience congestion. This may be workable for a 
> spesific (experimental) CC that seemingly has been guiding the 
> development of this feature but not necessarily as a general approach 
> that suits for various other CC algos. A stds track protocol should 
> serve various different approaches, or at least not make them 
> unreasonable to implement, otherwise the protocol should be made 
> experimental.

[BB3] See #4) Not generic enough?


>
>>             Specifically:
>>              *  The WG makes sure that RFCs about the /Data Sender/ 
>> of a half connection (e.g. the ECN++
>>                 experiment or other future RFCs) specify that sending 
>> ECN-capable pure ACKs is conditional on
>>                 having another way to distinguish DupACKs, e.g. 
>> negotiating SACK or timestamps (and I will
>>                 respond to your later points on the details of these).
>>              *  The AccECN spec (which primarily specifies the 
>> feedback behaviour of a /Data Receiver/ in a
>>                 half-connection) then only needs to define the 
>> Increment-triggered ACK rule.
>>             The two together lead to the same outcome you want. But 
>> without the interop hole of your
>>             approach.
>>
>>             This is consistent with the "Generic (Mechanistic) 
>> Reflector" approach of the AccECN spec which
>>             says:
>>             "AccECN is designed to be a generic reflector of whatever 
>> ECN markings it sees, whether or not
>>             they
>>             are compliant with a current standard."
>>
>>             These ACKs of ACKs are generically necessary to feed back 
>> congestion notifications from possible
>>             incoming packet patterns, not specifically for ECN++ or 
>> AckCC [RFC5690], or any other future RFC
>>             (forward compatibility). We'll edit the reference to 
>> ECN++ to make it clearer that it's one
>>             example,
>>             not the only case.
>>
>>             Here's another example of the generic reflector approach, 
>> already in the draft:
>>             "Although RFC 3168 prohibits an ECN-capable SYN, 
>> providing feedback of ECN marking on the SYN
>>             supports future scenarios in which SYNs might be 
>> ECN-enabled (without prejudging whether they
>>             ought
>>             to be). ... "
>>
>>
>>                   So, it should be cristal clear that without SACK 
>> (or Timestamps) a TCP *MUST NOT* send
>>                   ACKs of ACKs.
>>
>>
>>             I understand that you want this but, as just explained, 
>> without SACK or timestamps, the correct
>>             approach is to prevent the Data Sender putting the Data 
>> Receiver in the position where it would
>>             have
>>             to ACK ACKs in the first place.
>>
>>             In a connection without SACK or timestamps, if the Data 
>> Receiver were to get lots of congestion
>>             notifications on ECN-capable ACKs, it would face a 
>> difficult dilemma. Which would be more
>>             important:
>>             Signalling congestion by ACKing ACKs? or ensuring the 
>> performance improvements like Fast
>>             Retransmit,
>>             Fast Recovery etc. work well? The former would prevent 
>> harm to others, the latter would prevent
>>             harm
>>             to self.
>>
>>
>>       [MK] Please see my reply to Richard to understand why Acks of 
>> Acks cause Fast Retransmit, Fast Recovery, etc
>>       to cause harm to others in the first place. And also why most 
>> of the Acks of Acks seem to be just unnecessary
>>       load to the network, that is, they harm others without (much) 
>> benefits.
>>
>>
>> [BB2] I am preparing my own reply to that email. Briefly, in the 
>> FR/FR case that you highlight more harm to others
>> certainly results, but I don't believe that generalizes to all cases. 
>> And I don't agree that the ACKs of ACKs don't have
>> much benefit for their cost. For instance, compare them with the cost 
>> of all the ACKs that are currently sent with
>> ACK-ratio 2, relative to their benefit.
>
> [MK2] Of course, the harm caused to others depends on the case; there 
> are millions of different variations of scenarios where harm is done 
> and another set of millions of scenario variations where harm is not 
> done. But if harm is done even in one case it must be addressed unless 
> such a scenario can be proved to be a (random) corner case.
> One fundamental problem in allowing Acks of Acks is that it 
> potentially prevents developping any future algos that rely on the 
> fact that fake dupacks should not occur. FR/FR and other problems I 
> raised are just something that immediately came into my mind and 
> threreby they probably are just the tip of the iceberg; there are 
> potentially many other existing algos that may also break. We need to 
> be very careful to not introduce Acks of Acks before all potential 
> disadvantages have been considered much more closely.

[BB3] See
#3) Does not specify the necessary details?
#4) Not generic enough?


>
>>             Nonetheless, I will add some text to the AccECN draft 
>> that explains why it is important for other
>>             RFCs not to put a Data Receiver in the position where it 
>> has to ACK ACKs iff there is no way to
>>             distinguish them from DupACKs.
>>
>>
>>                   3) Even with SACK or Timestamps enabled it is not 
>> clear what an
>>                   implementer should do. With SACK the AckECN authors 
>> seem to make an assumption, which
>>                   seems obvious but is not, that an ACK of ACK would 
>> would never include SACK option and
>>                   hence it could be distinguished from a dupAck. 
>> However, RFC 2018 specifies: "If sent at
>>                   all, SACK options SHOULD be included in all ACKs 
>> which do not ACK the highest sequence
>>                   number in the data receiver's queue. So, if there 
>> is a hole in the receiver's queue, the
>>                   assumption is incorrect and it is unclear which 
>> SACK info to include into the SACK
>>                   option. Whatever one selects to include, it makes 
>> DSACK (RFC 2883] void and breaks any
>>                   DSACK-based algorithms unless RFC 2018 is updated.
>>
>>
>>             [BB]
>>             Reading the draft, it is very clear that there is no such 
>> assumption. The text said solely what
>>             it
>>             meant:
>>
>>                ... a host in AccECN
>>                mode that is sending ECN-capable pure ACKs SHOULD add 
>> one of the
>>                following additional checks when it tests whether an 
>> incoming pure
>>                ACK is a duplicate:
>>
>>                o  If SACK has been negotiated for the connection, but 
>> there is no
>>                   SACK option on the incoming pure ACK, it is not a 
>> duplicate;
>>             That is, if an incoming ACK were a duplicate, it would 
>> have a SACK option on it. This /relies/ on
>>             the rule in RFC2018 that you quote. So there is no need 
>> (nor intention) to change SACK behaviour
>>             in
>>             any way.
>>
>>
>>       [MK] Sorry I don't understand your claim. Assume A sends data 
>> pkts 0,1,2,3,..,10 in the current window to B
>>       and pkt 1 is dropped. Simultaneously, B sends data to A such 
>> that the data pkts arrive at A after A has
>>       injected pkt 10 to the network. These pkts trigger pure 
>> cumulative Acks from A to B that follow A's data pkts
>>       1..10 and enough of the Acks get CE-marked due to congestion 
>> path A to B. When pkts 2..10 arrive at B, each
>>       of them trigger a valid dupAck with a SACK block included. When 
>> the CE-marked Acks that follow data pkts
>>       2..10 arrive at B, B needs to feedback congestion info on them 
>> in Acks of Acks. These Acks of Acks cannot
>>       cumulatively ACK the highest sequence number in the data 
>> receiver's queue (pkt 10) since pkt 0 is the highest
>>       pkt arrived insequence, so the Ack of Ack must include a SACK 
>> block as per RFC 2018. What is the SACK block
>>       info that the implementer, who follows carefully advice in RFC 
>> 2018 and there is no other advice, should
>>       include in these Acks unless RFC 2018 is not changed? How does 
>> the implementer know how to Ack, if it not
>>       specified?
>>
>>
>> [BB2] Surely those ACKs of ACKs from B to A are just further valid 
>> DupACKs with a SACK block included as defined in
>> RFC2018, still cumulatively ack'ing packet 0 delivered before the loss.
>>
>> As per our proposal, ACKs with SACK blocks that repeat a cumulative 
>> ACK /can/ be counted as DupACKS, which is correct in
>> this case, because they really are valid DupACKs.
>
> [MK2] Sorry for not repeating DSACK (RFC 2883] that I originally 
> mentioned in item 3) above. As per RFC 2883, these DupAcks should 
> report the sequence number range of the packet that triggered the 
> acknowledgement. So, which sequence number range will be reported in 
> the first SACK block of these DupAcks (Acks of Acks)?
> This falls under the same rule of RFC 5681 that one must not ack more 
> than once an arrival of a data segment. Otherwise, it will break all 
> mechanisms that rely on the behavior specified in RFC 2883, including 
> RFC 3708 and (TLP of) RACK (RFC 8985) as well as any potential new and 
> useful algos that rely on this rule.
>
> So, I believe a random implementer cannot understand how to implement 
> such an Ack of Ack when SACK is enabled unless it is clearly 
> specified. E.g., your suggestion above is incorrect as it would 
> include a (random) DSACK-block and would harm any DSACK-based algo. It 
> is important to understand that all TCP specifications have been 
> written with an assumption that TCP does not ack pure Acks and 
> therefore trying to interpret the specs for the case where one would 
> ack an Ack easily results in an incorrect implementation and potential 
> harm to others or other mechanisms (pls see my reply w.r.t Timestamps 
> to Richerd, which is an apt example of misinterpretation).

[BB3] See:
#3) Does not specify the necessary details?

>
>> Now, let's see what happens if the retransmission of packet 1 arrives 
>> at B before some of those ACKs of ACKs have left B
>> (unlike your scenario the retransmission reached B after all the ACKs 
>> of ACKs had left). Once the retransmission of 1 has
>> arrived and fills the hole, any ACKs of ACKs sent after it arrives 
>> will cumulatively ACK 10, and they will not need a
>> SACK block. So, even if more than one ACK cumulatively ACKs 10, they 
>> will not be counted as DupACKs, because they have no
>> SACK block.
>
> [MK2] Sure, only when all packets have arrived in order and there is 
> no hole in the suquence space, the rule works just fine. But it must 
> work allways. Otherwise, it would only work partially and we don't 
> want algorithms or software that almost works.

[BB3] See: #3) Does not specify the necessary details?


>
>> In summary, if I've understood your scenario correctly, I believe it 
>> supports the proposed additional condition (for the
>> ECN++ draft) that an ACK without SACK cannot be a DupACK.
>
> [MK2] Yes, but in order to make it implementable we need to separately 
> specify how to Ack pure Acks (w/ SACK) and make an 
> exception/clarification for the RFC 2018 rule. Note also that RFC 6675 
> (current SACK-based FR algo) changed the definition of DupAck from 
> that of previous version (RFC 3517). This is likely to make it a bit 
> triggier and the SACK-enhanced version of F-RTO thet was based on RFC 
> 3517 definition seems to require modeification/clarification, maybe 
> some other specs too.
>
>>             (Note: to check this text, you'll need to refer to the 
>> previous AccECN draft here:
>> https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-accurate-ecn-23#section-3.2.2.5.1
>>             It had been there since Jul 2021 (since -15). But, as 
>> explained above, it is longer in the latest
>>             AccECN draft (-24), because it has been moved to the 
>> editor's copy of the ECN++ draft, which I
>>             wrote
>>             on 4 Apr, but I've been waiting for your reply before 
>> submitting it.)
>>
>>
>>                   With Timestamps some algorithms like Eifel 
>> detection breaks. Moreover, there are other
>>                   existing or potentially to-be-created heauristics, 
>> including various measurement tools,
>>                   that rely on the fact that TCP does not echo a 
>> later Timestamp in a pure Ack than what
>>                   arrived with the latest data packet. Any such 
>> mechanisms are subject to break.
>>
>>
>>             [BB] I don't fully understand what you're saying here. 
>> Can you be clearer please?
>>             And can you please bear in mind that we are in the WGLC 
>> processing stage now. So review comments
>>             ought to be suggesting very specific changes to the text 
>> under review.
>>
>>
>>       [MK] Again, the same problem as with SACK. It is unspecified 
>> which TSecr value to put in Acks of Acks. That
>>       has not been specified anywhere, so how can I or anyone else 
>> check what breaks if anything, and how can an
>>       implementer know which TSecr value to include in Acks of Acks?
>>
>>       It is hard to propose specific changes to text that does not 
>> exist or to a problem that seems to be a missing
>>       piece of design or a design flaw, until the text exists or the 
>> intended design is known or the potentiel
>>       design flaw is mutually agreed whether there is a flaw or not.
>>
>>       (Please note also that the Eifel problem seems not to be 
>> serious, but instead the timestamps rule you
>>       proposed to distinguish dupAcks from Acks of Acks seems 
>> suspicious and calls for clarification).
>>
>>       Please see more details in my reply to Richard.
>>
>>
>> [BB2] In your other email to Richard, I think you were right that the 
>> TS option cannot always be used to filter out
>> DupACKs. That's because timestamps echo the oldest unack'd packet, 
>> not the newest. But I believe the SACK condition is
>> more solid.
>
> [MK2] Agreed, but needs a careful review of all potential implications.

[BB3] see #3) Does not specify the necessary details?

>
>> In the reply I'm preparing to that email to Richard, I've suggested 
>> that the rule in the ECN++ draft might be that a Data
>> Sender can only set ECT on pure ACKs:
>>  *  if SACK has been negotiated
>>  *  else if TS is in use and DupACKs are not used to detect losses 
>> (e.g. once the RACK algorithm is not using DupACKs).
>> But pls wait for that reply - it won't be immediately clear here why 
>> I'm saying this.
>>
>>
>>             Again, this text about extra DupACK checks has now been 
>> removed from AccECN and will shortly
>>             appear
>>             in the ECN++ draft instead. I shall post the new ECN++ 
>> draft shortly.
>>
>>
>>                   It might be good to hold discussing any details on 
>> what breaks and how/why and what are
>>                   the consequences until I have sent my reply with 
>> scenarios to Richard.
>>
>>
>>             [BB] Please try to prioritize any comments about the text 
>> that is now left in the AccECN draft.
>>             The
>>             WGLC of AccECN is waiting for no-one else at the moment.
>>
>>
>>       [MK] Unfortunately I was not able to do this, sorry. I got 
>> confused already earlier when I was reviewing
>>       AccECN and ended up checking ECN++ quickly as I noted that 
>> AccECN draft cited it for these issues. When
>>       reading ECN++ I found Sec 3.3.3 and read:
>>
>>        "The question of whether and how the receiver of pure ACKs is 
>> required to
>>         feed back any CE marks on them is outside the scope of the 
>> present
>>         specification because it is a matter for the relevant feedback
>>         specification ([RFC3168] or [I-D.ietf-tcpm-accurate-ecn])."
>>
>>       This got me totally confused together with the later discussion 
>> on the mic at Yokohama mtng because that was
>>       what I had read, and I was not able to understand what was 
>> intended to go where. And I still not quite know
>>       but I think I have a better hunch. The above text still reads 
>> in ECN++ draft, but hopefully not in editor's
>>       copy?
>>
>>
>> [BB2] The split between the drafts is essentially as follows:
>>  *  AccECN deals primarily with what feedback the data receiver of a 
>> half-connection emits, and how the data sender of a
>>     half-connection interprets that feedback. Plus, of course, the 
>> negotiation to set up the feedback capability for a
>>     connection.
>>  *  ECN++ primarily gives conditions for the data sender of a TCP 
>> half-connection to set the IP-ECN field.
>> So the scopes looks something like this half connection:
>>
>>           ______________________
>>        ,-'     Feedback          `-
>> TCP   (     <-----------------     \
>>        `-.___________________AccECN |
>>       .---.                  `.     |
>> IP   (ECN++) ---------------> |     ;
>>       '---'                    \   /
>>                                 "-"
>>       Snd                       Rcv
>>
>> Although, with ACK packets, it's even more confusing, because they 
>> are normally considered only at the TCP layer, but
>> here, we are thinking of feedback of the ECN field in the IP header 
>> of the ACK (which is usually fed back in the TCP
>> header of data packets, except in these more unusual ACK of ACK cases).
>>
>> This is why
>> * the increment-triggered ACK rule is in AccECN,
>> * but the condition on whether to set the IP/ECN field of a TCP ACK 
>> is in ECN++.
>>
>>
>>
>>       Anyways, the major problem is not with any certain text phrases 
>> but whether DupAck vs. Ack of Ack problem is
>>       solvable and whether injecting Acks of Acks really is a needed
>>
>>
>> [BB2] Agreed
>>
>>       and mature enough feature that can be part of a stds track 
>> protocol.
>>
>>
>> [BB2] I hope you can see that, without ECN-capable ACKs, the 
>> mechanistic reflection rules in the AccECN draft would still
>> apply, but never lead to ACKs of ACKs.
>
> [MK2] Thank you for the clarification and apologies for still mixing 
> comments for AccECN and ECN++ drafts. This seems  all clear but to 
> reiterate: enabling ECN-capable ACKs at the remote end is out of 
> control for the local end emitting the ACKs of ACKs.

[BB3] See: #1) Generic should not be confused with experimental.

>> So if ECN-capable ACKs never mature, everything in the AccECN draft 
>> still sits
>> correctly on the standards track.
>
> [MK2] Sorry to reiterate: the problem here I'd like to solve is that 
> AccECN mandates (MUST) Increment-Triggered ACKs -rule in stds track. 
> It means that if someone implements this stds track rule in some stack 
> without the intent to participate in the ECN++ experiment, the 
> behavior still sits in that stack. If someone sometime later on 
> specifies and enables ECN-capable ACKs (at the other end) for some 
> other purpose than ECN++, we will get these Acks of Acks even if we 
> don't want them (maybe the ECN++ experiment never matured for some 
> reason). That's why I seriously hesitate nailing down anything on 
> Acking of Acks in a stds track doc at this point.

[BB3] See: #1) Generic should not be confused with experimental.

BTW, DCTCP has widely used ECN++ for 10 years in DCs with significant 
performance benefit, because it sets ECT on all packets. So the idea 
that ECN++ won't mature is moot. Even so, the point about getting acks 
of ACKs even if we don't want them doesn't stand up. Even if the ECN++ 
experiment never matured, then the ack of ACKs capability would still be 
there for some future use of ECN on pure ACKs, but it just wouldn't be 
exercised if there were never any ECN-capable pure ACKs. That's the 
whole point of generic reflection - it might not be used in certain 
ways, but it's there if it is used.

>
> Wouldn't it be possible to move the Ack of Ack part of the rule into 
> an experimental doc because that part is needed for experimental 
> purposes only and reguires the necessary safeguards?

[BB3] Putting the generic feedback part on the stds track is the 
deliberate intention of the WG.
See: #1) Generic should not be confused with experimental.

>
>> This is not surprising, because we originally wrote the 
>> increment-triggered ACK rule without realizing that it could
>> apply to pure ACKs.
>>
>>
>>             But also bear in mind that the chairs plan to take ECN++ 
>> into WGLC once AccECN WGLC has been
>>             cleared. So we need to hear your actual argument about 
>> the DupACK text that has been moved to
>>             ECN++
>>             urgently too. We can't work with "I have an argument, but 
>> I'll tell you later".
>>
>>
>>       [MK] Apologies for the delay again. Please see my reply to 
>> Richard for my arguments.
>>
>>
>> [BB2] Yes, thank you for all the work you've put into that reply.
>>
>>

[snipped side conversation about delayed ACKs, which had completed]

Regards


Bob

> Best regards,
>
> /Markku
>
>> Regards
>>
>>
>> Bob
>>
>>
>>       Best regards,
>>
>>       /Markku
>>
>>
>>             Regards
>>
>>
>>             Bob
>>
>>
>>                   Best regards,
>>
>>                   /Markku
>>
>>
>>                         Best regards
>>                         Michael
>>
>>                               Cheers
>>
>>
>>                               Bob
>>
>>                               On 30/03/2023 15:15, 
>> [e-mail] wrote:
>>                                     A New Internet-Draft is available 
>> from the on-line
>>                                     Internet-Drafts
>>                                     directories. This Internet-Draft 
>> is a work item of
>>                                     the TCP Maintenance and
>>                                     Minor Extensions (TCPM) WG of the 
>> IETF.
>>
>>                                        Title           : More 
>> Accurate Explicit
>>                                     Congestion Notification (ECN) 
>> Feedback in TCP
>>                                        Authors         : Bob Briscoe
>>                                                          Mirja Kühlewind
>>                                                          Richard 
>> Scheffenegger
>>                                        Filename        :
>> draft-ietf-tcpm-accurate-ecn-24.txt
>>                                        Pages           : 64
>>                                        Date            : 2023-03-30
>>
>>                                     Abstract:
>>                                        Explicit Congestion 
>> Notification (ECN) is a
>>                                     mechanism where network
>>                                        nodes can mark IP packets 
>> instead of dropping
>>                                     them to indicate
>>                                        incipient congestion to the 
>> endpoints.  Receivers
>>                                     with an ECN-capable
>>                                        transport protocol feed back 
>> this information to
>>                                     the sender.  ECN was
>>                                        originally specified for TCP 
>> in such a way that
>>                                     only one feedback
>>                                        signal can be transmitted per 
>> Round-Trip Time
>>                                     (RTT).  Recent new TCP
>>                                        mechanisms like Congestion 
>> Exposure (ConEx), Data
>>                                     Center TCP (DCTCP)
>>                                        or Low Latency, Low Loss, and 
>> Scalable Throughput
>>                                     (L4S) need more
>>                                        accurate ECN feedback 
>> information whenever more
>>                                     than one marking is
>>                                        received in one RTT. This 
>> document updates the
>>                                     original ECN
>>                                        specification in RFC 3168 to 
>> specify a scheme
>>                                     that provides more than
>>                                        one feedback signal per RTT in 
>> the TCP header.
>>                                     Given TCP header
>>                                        space is scarce, it allocates 
>> a reserved header
>>                                     bit previously
>>                                        assigned to the ECN-Nonce.  It 
>> also overloads the
>>                                     two existing ECN
>>                                        flags in the TCP header. The 
>> resulting extra
>>                                     space is exploited to
>>                                        feed back the IP-ECN field 
>> received during the
>>                                     3-way handshake as
>>                                        well.  Supplementary feedback 
>> information can
>>                                     optionally be provided
>>                                        in two new TCP option 
>> alternatives, which are
>>                                     never used on the TCP
>>                                        SYN.  The document also 
>> specifies the treatment
>>                                     of this updated TCP
>>                                        wire protocol by middleboxes.
>>
>>                                     The IETF datatracker status page 
>> for this
>>                                     Internet-Draft is:
>> https://datatracker.ietf.org/doc/draft-ietf-tcpm-accurate-ecn/
>>
>>                                     There is also an htmlized version 
>> available at:
>> https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-accurate-ecn-24
>>
>>                                     A diff from the previous version 
>> is available at:
>> https://author-tools.ietf.org/iddiff?url2=draft-ietf-tcpm-accurate-ecn-24
>>
>>                                     Internet-Drafts are also 
>> available by rsync at
>> rsync.ietf.org::internet-drafts
>>
>>
>> _______________________________________________
>>                                     tcpm mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/tcpm
>>
>>
>>                               --
>> ________________________________________________________________
>>                               Bob Briscoe http://bobbriscoe.net/
>>
>>
>>
>>
>>             --
>> ________________________________________________________________
>>             Bob Briscoe http://bobbriscoe.net/
>>
>>
>>
>> -- 
>> ________________________________________________________________
>> Bob Briscoe http://bobbriscoe.net/
>>
>>

-- 
________________________________________________________________
Bob Briscoehttp://bobbriscoe.net/