Re: [tcpm] Fw: draft-ietf-tcpm-accurate-ecn-24 addressing allWGLCcomments

[Markku Kojo <[e-mail]>]

Bob,

TL;DR

1.  It seems that you missed the point with SACK/DSACK;

2. Even when SACK is in use there is no reliable way to distinguish real 
DupAcks from Acks of Acks ('fake' DupAcks). Therefore, existing stds track 
mechanism like RACK/TLP and F-RTO do no work correctly if one uses the 
SACK rule in ECN++ draft for trying to distinguish real DupAcks from Acks 
of Acks.

Please see [MK4] below for details.

On Thu, 27 Jul 2023, Bob Briscoe wrote:

> Markku, pls see [BB4]
>
> On 27/07/2023 08:26, Markku Kojo wrote:
>> Bob,
>> 
>> thank you for resending this with another email address as my regular 
>> address bounced (my account erreneously expired last Fri but is now back 
>> and working).
>> 
>> Please see below tagged [MK3].
>> 
>> On Wed, 26 Jul 2023, Markku P I Kojo wrote:
>> 
>>> 
>>> 
>>> ____________________________________________________________________________________________________________________________ 
>>> From: Bob Briscoe <[e-mail]>
>>> Sent: 23 July 2023 00:01
>>> To: Kojo, Markku P I <[e-mail]>
>>> Subject: Fwd: draft-ietf-tcpm-accurate-ecn-24 addressing all WGLC comments
>>> Retrying with an address which appears not to bounce.
>>> Pls add back mail distribution below in any reply.
>>> 
>>> 
>>> -------- Forwarded Message --------
>>> Subject:
>>> Re: draft-ietf-tcpm-accurate-ecn-24 addressing all WGLC comments
>>> Date:
>>> Sat, 22 Jul 2023 13:33:51 -0700
>>> From:
>>> Bob Briscoe <[e-mail]>
>>> To:
>>> Markku Kojo <[e-mail]>
>>> CC:
>>> [e-mail], Yoshifumi Nishida <[e-mail]>, Ian Swett 
>>> <[e-mail]>, [e-mail], Bob Briscoe
>>> [Apple] <[e-mail]>
>>> 
>>> Markku,
>>> 
>>> I'm top-posting 4 responses to all your points, in order to consolidate 
>>> everything and avoid all the repetition:
>>> 
>>> #1) Generic should not be confused with experimental.
>>> #2) Completeness
>>> #3) Does not specify the necessary details?
>>> #4) Not generic enough?
>>> 
>>> #1) GENERIC SHOULD NOT BE CONFUSED WITH EXPERIMENTAL.
>>> 
>>> 
>>> The AccECN spec is the IETF's PS spec for how TCP feeds back ECN on any 
>>> packet. The requirement for AccECN to provide
>>> forward compatibility by providing generic feedback was agreed by this WG 
>>> and set down in RFC7560 (nearly 8 years ago).
>>> Generic mechanistic reflection is a central design principle of AccECN - 
>>> to reflect the ECN field of SYNs, SYN-ACKs, ACKs,
>>> data packets, retransmissions, everything (§2.5 of the draft has said this 
>>> since its inception). AccECN provides accurate
>>> feedback irrespective of whether or how it is currently used, so that it 
>>> can be used if needed.
>>> Mechanistic reflection is generic, not experimental.
>>>     Analogy: Senders have conducted many CC experiments using feedback 
>>> from stds-track rcvrs. But that doesn't make the
>>> rcvrs experimental; rather, it proves the f/b is generic.
>> 
>> [MK3] All these CC experiments that I am aware of have used stds track 
>> receiver feedback that had already existed for a long time before the 
>> experiment and had been matured for years (decades). This draft intends to 
>> introduce new behaviour (acking ACKs randomly in scale) for upcoming 
>> experiments at the same time when the experiments are being specified. That 
>> is very different because the new feature has not matured. Besides, the 
>> Acks of Acks issue appeared only after the wg had decided this document to 
>> become stds track.
>
> [BB4] I think the WG understands your point. Where we are talking about 
> possible undefined future problems, it is for the WG to decide whether they 
> want to proceed.
>
>> 
>>> You say an AccECN receiver automatically becomes part of an experiment as 
>>> soon as the its peer enables ECN-capable ACKs.
>>> More constructively, one can say that the peer conducts the experiment by 
>>> building on the generic behaviour of the receiver.
>>> The latter is more correct because the behaviour of the receiver was 
>>> already specified (even though it was not exercised);
>>> so it predated the experiment, and could be analysed and checked prior to 
>>> any experiments.
>> 
>> [MK3] It can be well phrases both ways. "it was not exercised" is the key 
>> concern here. We do not know enough about potential problems; you say it 
>> "could be analysed and checked prior to any experiments" but that IMHO is 
>> not good enough to introduce such a new stds track behaviour. Even though 
>> it was carefully analysed, the TCP Timestamp condition was not found being 
>> incorrect, for example. This clearly shows how tricky issue we are handling 
>> and I (and I guess anyone else either) cannot come up with all potential 
>> problems easily. That's why we should be evry careful here.
>> Pls see also my reply to Richard where I explained how this feature of Acks 
>> of Acks may become a problem much later (maybe after decade or more), when 
>> AccECN possibly have been widely deployed as a stds track spec but ECN++ 
>> somehow faded away. How is it possible/guaranteed that another experiment 
>> that wants to enable pure Acks with non-SACK connections can get rid of the 
>> Acks of Acks it does not need and all harm they produce?
>> They are coded and deployed all over and it would be hard to get rid of 
>> them.
>
> [BB4] The code for acking ACKs will indeed be in stds track implementations 
> of AccECN, but it will not be exercised unless a peer is running an 
> experiment like ECN++. Any of the problems you mention can only occur during 
> one of those experiments, where they can also be resolved (or the experiment 
> ended).

[MK4] You did not answer the question. You say that any of the problems 
can also be resolved in the (upcoming) experiments. I disagree.
Could you please explain how would it be possible to resolve all the 
problems that arise in an experiment where an endpoint want's to take 
advantage of ECN-capable pure ACKs over a non-SACK TCP connection (for 
ACK CC purposes, for example) but the other end implements the stds track 
behaviour as currently specified in the draft?

If the goal of AccECN is to provide "generic" feedback for any CC 
purposes, I find it not very good/mature design if it excludes in advance 
certain CC approaches and makes the protocol unusable without options 
(e.g., non-SACK connections cannot leverage from ECN-capable pure ACKs).
IMO, even this is strong enough reason why I don't think that including 
Acks of Acks in this stds track spec is a reasonable idea.

>> 
>>> This is protocol layering. Feedback is a 'sub-layer' of the transport 
>>> layer that lies logically beneath the congestion
>>> control function of the transport layer and is intended to allow future 
>>> evolution of sender behaviour. The RTCP protocol
>>> demonstrates this well, because it was specified separately from RTP.
>>> 
>>> You express concern that specifying a case where a stds track AccECN 
>>> implementation has to ack ACKs gives it no control over
>>> whether it is dragged into an experiment being conducted by its peer.  But 
>>> that's the deal for every generic capability. It
>>> gets used. Our task is to make sure that it can be used in this way. 
>>> That's why we have to correctly specify the SACK
>>> condition. That's what we do in the IETF.
>>> 
>>> #2) COMPLETENESS
>>> 
>>> 
>>> A protocol spec should say what an implementation has to do in response to 
>>> all possible inputs. This ensures that there is
>>> just one behaviour to be checked for flaws (including security flaws), 
>>> rather than implementers inventing their own
>>> behaviours, each of which might lead to bugs or vulnerabilities if it 
>>> receives unexpected inputs.
>> 
>> [MK3] The incompleteness is exactly the problem that I have tried to 
>> explain. See (*) under item #3 below.
>> 
>>> #3) DOES NOT SPECIFY THE NECESSARY DETAILS?
>>> 
>>> 
>>> You say this spec doesn't define the necessary safety features. We can't 
>>> work with "it needs a careful review for potential
>>> problems" then do nothing while we wait in case we might see another email 
>>> from you. If you think a safety feature is
>>> missing, the only way the WG works is constructively: participants 
>>> identify what's missing, and (ideally) define it.
>> 
>> [MK3] (*) My apologies if I have not been clear enough. What I have tried 
>> to say is that the present spec does not specify how to ack an Ack when 
>> SACK is enabled. I was now able interpret the intended way to ack an Ack (I 
>> guessed this myself already earlier and you hinted it as well but I was not 
>> sure because there were contradictory comments on this matter earlier) from 
>> the most recent ECN++ spec which implies that an ECN++ peer expects that an 
>> Ack of Ack does not carry SACK option:
>>  "If there is no SACK option on the incoming pure ACK despite SACK
>>   having  been negotiated, it is not a duplicate;"
>> 
>> This way of acking, however, has not been specified in this spec. Instead, 
>> you claim that this would be the case if the implementer follows SACK (RFC 
>> 2018) and DSACK (RFC 2883). I disagee with this. Instead, the implied way 
>> from ECN++ spec and the SACK spec (and DSACK) are in conflict.
>> 
>> An implementer who implements this spec following the rules in this spec as 
>> well as SACK (RFC 2018) and DSACK (RFC 2883) has a serious dilemma to solve 
>> which is contrary to what you claim in item #2 above.
>> Let's walk through what the implementer has at hand:
>> 
>> 1. When enough pure ACKs with CE have arrived, this spec requires that the 
>> receiving end MUST send an ACK.
>> 
>> 2. The implementer reads SACK spec (RFC 2018) that requires that a SACK 
>> block is included in the ACK in certain cases:
>> 
>>  "SACK options SHOULD be included in all ACKs which do not ACK
>>  the highest sequence number in the data receiver's queue."
>> 
>> 2. a) If there is no hole in the sequence space, everything is fine;
>>       No SACK block is included as per RFC 2018 and these Acks of
>>       Acks are not considered being DupAcks.
>> 
>> 2. b) If there is hole in the sequence space, RFC 2018 requires a SACK
>>       block being included. The implementer includes a SACK block, but
>>       has a dilemma to solve: which SACK block to indicate. Whichever
>>       SACK block is included will result these Acks to be considered
>>       valid DupAcks as per ECN++ which they are not (they do not
>>       indicate an arrival of an out-of-order data packet but incorrectly
>>       repeat some old SACK info. This is against the rule in RFC 5681
>>       that prohibits for a good reason to send more than one ACK per
>>       arriving data packet and will result in problems (see below).
>>       That is, this spec misses the rule for acking ACKs without the
>>       SACK option and indicating this is different from RFC 2018
>>       specification. And your completeness reuirement in item #2 is
>>       not fulfilled.
>
> [BB4] There is no dilemma. The receiver of the ACKs of ACKs would i) have 
> already entered loss recovery or, ii) if the ACK of the data was lost, it 
> would enter loss recovery on receipt of the ACKs of ACKs.

[MK4] Of course the receiver of the ACKs of ACKs would either have entered 
loss recovery or would enter loss recovery. Entering loss recovery is not 
the problem but what happens during the loss recovery (fast recovery). Pls 
see below.

> RFC2018 allows no ambiguity on this. The state of the sender is either still 
> in loss recovery, when observing an ACK-with-updated-ACE-plus-SACK, or it 
> should enter loss recovery. There is nothing extra needed or possible in the 
> SACK RFCs.

[MK4] Any SACK block included in the Ack of an Ack as the first SACK 
block will incorrectly indicate arrival of a duplicate /data/ segment 
even though no such data segment has arrived. This is forbidden both in 
RFC 5681 as well as in RFC 2883 as I have several times indicated. See 
more below tagged (**).

>
>> 
>> 3. If the implementation also implements DSACK, the implementer has even
>>    harder dilemma to solve in case there is a hole in the sequence space.
>>    RFC 2018 requires that a SACK block is included and DSACK specifies
>>    that the first SACK block reports "sequence of data received by the
>>    receiver in the most recent packet".
>>    Because the pure ACK does not include any data, what should the
>>    implementer do. For sure there will be various different
>>    interpretations because this has not been specified in this spec
>>    (and your item #2 is not fulfilled).
>> 
>>   a) An impelmenter may select a "random" block, maybe the most
>>      recently SACKED which is incorrect because it breaks DSACK idea
>>      and is in conflict with RFC 2883.
>>   b) Another implementer may be innovative and includes some "random"
>>      sequence number and includes a zero length SACK block with the
>>      same sequence number as the left and right edge. The consequences
>>      of this are unknown but certainly not correct.
>>   c) A more clever inplementer that reads the thoughts of authors
>>      for this spec (or has followed the discusions or even read ECN++)
>>      may send an ACK without any SACK info but this is in conflict
>>      RFC 2018.
>>   d) Maybe some other interpretation ...
>> 
>> You said earlier (below) that a) is the correct way with DSACK:
>> 
>>  "[BB2] Surely those ACKs of ACKs from B to A are just further
>>   valid DupACKs with a SACK block included as defined in
>>   RFC2018, still cumulatively ack'ing packet 0 delivered
>>   before the loss."
>
> [BB4] The purpose of DSACK is to report the sequence numbers of the packet 
> that triggered the acknowledgement, which in these cases is a pure ACK.

[MK4] That is not quite correct. RFC 2018 specifies that the first SACK 
block reports the sequence numbers of the packet that triggered the 
acknowledgement. DSACK extends RFC 2018 by specifying and clarifying how 
an arrival of a duplicate data segment is reported.

> So, it just doesn't make sense to use DSACK on an ACK of an ACK at all (I'll 
> call this option e).
>
> We can say "no DSACK on these ACKs of ACKs" explicitly in the draft, but I 
> don't see how anyone would think  to use DSACK on one of these ACKs of ACKs.
>
> BTW, I can't see how my words could have been interpreted as option 'a' 
> ('random').

[MK4] I'm afraid you seem to make the same mistake as Richard did with 
TCP timestamps; you suggested that "ACKs of ACKs from B to A are just 
further valid DupACKs with a SACK block included as defined in RFC2018." 
As I have pointed out earlier, RFC 2018 does not specify how to ack on 
arrival of a pure ACK (nor does any other TCP spec). The rules in RFC 
2018 are correct only when acking data segments. However, you suggest 
following the rules in RFC 2018 when acking a CE-marked pure ACK.
That is, if you include a SACK block as defined in RFC 2018, you will 
include in the ACK of the ACK SACK blocks by repeating the most recently 
reported SACK blocks. So, you explicitly suggest using DSACK on ACKs of 
ACKs; the first SACK block would be incorrectly interpreted as DSACK by 
the receiver of the ACK of the ACK even though no duplicate data segment 
never arrived. The first SACK block (DSACK) will repeat the most recent 
SACK block and is therefore a "random" block because one cannot know in 
advance what is the latest data segment that arrived out-of-order. It 
depends on recent packet dynamics of the connection that is more or less 
random. So, if an author of the draft does not know how to ACK an ACK how 
would a "random" implementer know if the #COMPLETENESS requirement you 
brought up is not fulfilled?

[MK4] RFC 5681 and RFC 2883 clearly forbid sending an ACK unless a data 
segment has arrived. In other words, the rule "no DSACK on these ACKs of 
ACKs" as you suggest is completely vague and unworkable.

[MK4] The ECN++ rule does not work reliably.
The major remaining problem is the one you ignored in your recent reply. 
The suggested rule in ECN++ says: 
 "If there is no SACK option on the incoming pure ACK despite SACK
  having  been negotiated, it is not a duplicate;"
This rule does not reliably distinguish an ACK that indicates an arrival 
of duplicate data segment from arrival of a CE-marked pure ACK. Stds 
track specifications RACK/TLP (RFC (8985) and F-RTO (RFC 6582) do handle 
arrival of a duplicate ACK (as per cumulate Ack in the Ack field) without 
SACK option and with DSACK the same in certain cases to work correctly.
These specs have been carefully specified to work correctly and they 
specify their algos for completeness and the suggested ECN++ rule would 
make these algos work incorrectly. 
Please see Sec. 7.4.2 for RACK and my earlier ignored text just here 
below your previous signiture.
In addition, also the spurious retransmit detection on RFC 3708 seems not 
to work correctly with the ECN++ rule even though the spec is a bit vague 
for checking non-SACK DupAcks (Sec. 3, last para).

Hope I was able the describe the problems clear enough this time.

Thanks,

/Markku

> Bob
>
>> 
>> This would result in incorrect behaviour with DSACK and will confuse any 
>> algo that depends on correct behavior of DSACK in correctly reporting 
>> arrival of duplicate data segments, including RFC 3708 and (TLP of) RACK 
>> (RFC 8985) as I have earlier indicated.
>> 
>> The intention of DSACK is to inform an arrival of duplicate data segment 
>> and the spec explicitely prohibits repeating that info in the spirit of the 
>> one ACK per data segment rule of RFC5681. RFC 2883, Sec 4, item (2):
>> 
>>  "Each duplicate contiguous sequence of data received is reported
>>  in at most one D-SACK block.  (I.e., the receiver sends two identical
>>  D-SACK blocks in subsequent packets only if the receiver receives two
>>  duplicate segments.)"
>> 
>> Even if one selects the intended option (c above), it does not necessarily 
>> result in correct outcome. SACK specification in RFC 3517 used the same 
>> definition for DupAck as RFC 5681.
>> The DupAck definition in RFC 6675 holds for the purposes of RFC 6675 only 
>> as clearly stated in that specification. That is, it depends on an RFC spec 
>> which definition of DupAck (RFC 5681 of RFC 6675) it adopts regardless of 
>> whether the spec involves SACK or not. For example, F-RTO deliberately uses 
>> the DupAck definition of RFC 5681 for both non-SACK version and the 
>> SACK-enhanced version of F-RTO. In other words, the SACK-enhanced version 
>> of F-RTO uses an arriving DupAck without SACK option (or DupAck with SACK 
>> block but with no new SACK information) to declare RTO being not spurious 
>> similar to the non-SACK version does (Sec 3.1, step 3 a). Therefore, an 
>> arriving Ack of an Ack without SACK info incorrectly declares an RTO not 
>> spurious even if it actually was spurious RTO.
>> (This needs a bit different scenario than what I draw in Fig 3 b) of my 
>> scenarios that was for the non-SACK version of F-RTO)
>> 
>> To sum up: In order to partially solve the ambiguity on how to ack ACKs and 
>> avoid triggering false fast retransmits, this spec must update RFC 2018 by 
>> making an exception to the rule that specifies when a SACK block should be 
>> included. However, this does not solve the entire problem with Acks of Acks 
>> because there are algos like F-RTO that use the RFC 5681 definition of 
>> DupAck. Therefore, IMHO sending Acks of Acks should not be included in a 
>> stds track spec.
>> 
>>> You repeatedly refer forward to apparent problems with SACK, which sounds 
>>> like you have identified that the protocol is
>>> broken. But, having read forward to the end, you agree that the SACK 
>>> condition is more solid than using TSOpt. And the only
>>> concrete criticism I can find is that we don't specify what DSACK to send. 
>>> Richard & I (and others) have worked through all
>>> the cases you have proposed (and others we have contrived ourselves), and 
>>> we cannot find any case where an implementation
>>> has to do anything different from what the DSACK RFCs already say.
>> 
>> [MK3] Please see the explanation above. Hope it clarifies the issue now.
>> 
>>> I had independently found the rule you point out in RFC 5681, Sec 2, 
>>> against generating more than one ACK for every incoming
>>> [data] segment. We can say that the potential requirement to ack ACKs 
>>> contravenes that rule, which is why the SACK condition
>>> is necessary. I know you are concerned that it might not be as simple as 
>>> that. But, without evidence, how do we distinguish
>>> this from over-dramatisation? In the absence of anything specific, we 
>>> can't just sit on our hands. As Yoshi says, these will
>>> probably be corner cases. If new problems surface, we will have to deal 
>>> with them as they arise.
>> 
>> [MK3] Please see above. Definitely not a corner case.
>> 
>>> #4) NOT GENERIC ENOUGH?
>>> 
>>> 
>>> To your point about feedback from pure ACKs and data being combined, 
>>> that's not driven by any particular CCA. If ACKs are
>>> ECT, AccECN is designed to make it possible for the Data Sender to 
>>> estimate how much of the combined per-packet feedback is
>>> from ACKs and data, by using the per-byte feedback in the option to 
>>> estimate the number of data packets. This might not be
>>> perfect, but it was recognised from the start that we would need to decide 
>>> on compromises,, particularly given very limited
>>> header space.
>>> 
>>> Nothing in the increment-triggered ACK rule was designed around ECN++. 
>>> Requiring the receiver to emit an ACK after 'n' CE
>>> marks is a fairly obvious way of  ensuring that the sender  gets timely 
>>> feedback of multiple CE markings.
>> 
>> [MK3] Please see my reply to Richard where I explained why this feedback 
>> with Acks of Ack is in almost all scenarious unnecessary or not timely. In 
>> a single case when the other end first is idle but starts sending data it 
>> might be useful but even in such scenarious the feedback often comes too 
>> late or possibly is stale (if there is longish delay before the other end 
>> starts sending).
>> 
>>> 'n' has to be less
>>> than 7 to prevent AccECN's 3-bit ACE field wrapping. And greater than 2 to 
>>> avoid too many ACKs. Then, as I said, we noticed
>>> that could cause ping-pong, so we raised the min to 3 if the ACK would be 
>>> in response to an ACK. That would be necessary for
>>> protocol completeness (see above) whether or not an ECN++ draft existed.
>> 
>> [MK3] Raising the min to 3 helps in dampening the ping-pong effect. But 
>> still the ping-pong may last several RTTs if the cwnd is large and lot of 
>> pkts (most) get CE marked. Not sending feedback for CE-marked DupAcks (and 
>> not counting them in the counters) would be much more efficient way to 
>> dampen the ping-pong effect. In addition, it would avoid making unnecessary 
>> rate reductions in such cases (when DupAcks are arriving) where there will 
>> be a notable rate reduction anyway (the increased Ack rate due to DupAcks 
>> being sent for every data pkt goes away roughly in one RTT and the end 
>> receiving DupAcks will react to congestion and reduce cwnd, resulting in 
>> lower ack rate for the next RTTs). Pls see my reply to Richard.
>> This is another reason why this part of the rule is experimental in my 
>> opinion: we don't have any evidence/experience what is the best way to 
>> provide feedback/react to CE-marked pure Acks.
>> It would be useful if you could give one scenario where acking CE-marked 
>> DupAcks would give some benefit that cannot be (easily) solved(gained in 
>> some other means.
>> 
>>> You raise a concern that allowing Acks of Acks might preclude future algos 
>>> that rely on the absence of 'fake dupacks'.
>>> Indeed, you say that contravening the one ACK per data segment rule of 
>>> RFC5681 "will break all mechanisms that rely on the
>>> behavior specified in RFC 2883, including RFC 3708 and (TLP of) RACK (RFC 
>>> 8985) as well as any potential new and useful
>>> algos that rely on this rule." But I think this is all stated rather 
>>> over-dramatically, because you admit that the SACK
>>> condition allows an implementation to identify 'fake dupacks'. So it can 
>>> exclude them from all such present and future
>>> algos. Then this all becomes a non-issue.
>> 
>> [MK3] Please see above. My apologies if I was not clear enough for this 
>> part. I intended to say that SACK would help in distinguishing "fake 
>> dupacks" that will otherwise trigger a false fast rexmit. However, SACK 
>> seemingly does not solve all problems (e.g., breaks f-RTO). Unfortunately I 
>> have no time to pass through all potential algos that may break. Even less 
>> time to consider all potential new algos that could count on the 
>> fundamental rule for generating only one DupAck per arriving data segment. 
>> I just try to point out the ones that I found being problematic. Even if 
>> one existing and (widely) deployed algo gets broken, that should be strong 
>> enough warning signal to not proceed as a stds track feature.
>> 
>>> _______________
>>> I haven't added any further discussion to the thread below (except a 
>>> couple of minor points). To check that the above 4
>>> points comprehensively address all your [MK2] points. I've tagged each 
>>> with [BB3] and pointer(s) to the relevant response(s)
>>> above.
>> 
>> [MK3] I hope the above clarifications help. I didn't add anything below 
>> except one comment (taggd [MK3]). I may pass through the comments below 
>> later and comment if need be.
>> 
>> Thanks,
>> 
>> /Markku
>> 
>>> On 11/07/2023 16:48, Markku Kojo wrote:
>>>       Bob, please see [MK2] inline.
>>> 
>>>       And apologies for the long delay in replying, again.
>>> 
>>>       On Mon, 5 Jun 2023, Bob Briscoe wrote:
>>> 
>>>             Markku, pls see [BB2] inline
>>> 
>>>             On 26/05/2023 22:56, Markku Kojo wrote:
>>>                   Bob,
>>> 
>>>                   My apologies you had to wait for the scenarios as it 
>>> took much longer with my limited cycles
>>>             than I thought.
>>>                   Anyways, please see my reply to Richard, some scenarios 
>>> are also included there.
>>> 
>>>                   To keep things easier, it might be good to try to keep 
>>> the discussion on Acks of Acks (mainly)
>>>             in the thread
>>>                   with my reply to Richard.
>>> 
>>>                   However, see inline tagged [MK].
>>> 
>>>                   On Wed, 24 May 2023, Bob Briscoe wrote:
>>> 
>>>                         Markku,
>>> 
>>>                         Sorry, it's taken a week to build a comprehensive 
>>> reply to this long email. See inline tagged [BB]...
>>> 
>>>                         On 17/05/2023 12:24, Markku Kojo wrote:
>>>                               Hi Michael, all,
>>> 
>>>                               On Sun, 14 May 2023, [e-mail] 
>>> wrote:
>>> 
>>>                                           On 30. Mar 2023, at 16:53, Bob 
>>> Briscoe <[e-mail]>
>>>                                           wrote:
>>> 
>>>                                           Michael, Yoshi, Ian (as tcpm 
>>> chairs),
>>> 
>>>                                           To close off the WGLC, I have 
>>> just posted a new rev of
>>>                                           accurate-ecn. Hyperlinks quoted 
>>> at the end.
>>>                                           You will see the diff is rather 
>>> extensive. I won't give a
>>>                                           summary of all the diffs like I 
>>> usually do. Instead I can just
>>>                                           refer to the summary I gave in 
>>> the presentation on Monday:
>>> 
>>> 
>>> https://datatracker.ietf.org/meeting/116/materials/slides-116-tcpm-draft-ietf-tcpm-accurate-ecn
>>> 
>>>                                           Thank you again to the people 
>>> who reviewed this during the WGLC:
>>>                                           Michael Tüxen, Alex Burr, Gorry 
>>> Fairhurst and Markku Kojo.
>>> 
>>>                                           All changes are editorial, apart 
>>> from removing the para about
>>>                                           not mistaking certain ACKs of 
>>> ACKs for DupACKs, which I will add
>>>                                           to a rev of the ECN++ draft, 
>>> hopefully later this week.
>>> 
>>>                                           On the list, we have seen 
>>> agreement from all the reviewers to
>>>                                           these changes, except no 
>>> response from Markku yet.
>>>                                           On Monday, I told Markku that I 
>>> would post the draft in a few
>>>                                           days, so everyone can see the 
>>> updates and diff.
>>> 
>>>                                     Anyone having additional comments? In 
>>> particular Markku regarding loss
>>>                                     recovery?
>>> 
>>> 
>>>                               My apologies for being late with my reply to 
>>> the author's comments on my review
>>>             (I've
>>>                               been extremly busy with other issues since 
>>> the wg mtng in Yokohama, including the
>>>             rest
>>>                               of mtng week).
>>> 
>>>                               I don't have much new comments but it seems 
>>> that my major concern regarding the
>>>             problem
>>>                               of sending ACKs of ACKs was not fully 
>>> understood.
>>> 
>>>                               The first thing where I think I was not 
>>> quite clear is that the major problem with
>>>             ACKs
>>>                               of ACKs is not that a pure Ack is made 
>>> ECN-capable. Instead, the problem is in
>>>                               generating an Ack of an pure Ack and that is 
>>> what one should prohibit to avoid
>>>             problems.
>>>                               I understand that it might be problematic to 
>>> formulate rules whether generating an
>>>             Ack
>>>                               of an Ack is allowed (and when), instead of 
>>> just disabling sending ECN-capable
>>>             ACKs.
>>>                               I don't have a strong opinion which way the 
>>> problems with ACKs of ACKs is avoided
>>>             as
>>>                               long as they are avoided.
>>> 
>>> 
>>>                         [BB] See later after your similar point (following 
>>> your 'Why?' heading)...
>>> 
>>> 
>>>                               I am preparing a few scenarios to 
>>> illustriate the problems ACKs of ACks raise and
>>>             will
>>>                               send them shortly once I have formulated a 
>>> more thorough reasoning why sending
>>>             ACKs of
>>>                               ACKs is not really a good idea and even 
>>> seems to be unnecessary in most if not all
>>>                               cases, i.e., it just results in sending 
>>> unnecessary packets with not much useful
>>>             effect
>>>                               but creates a notable number of problems.
>>> 
>>> 
>>>                         [BB] Having waited this long, it's rather 
>>> disappointing to still hear you say "I have an
>>>                         argument,
>>>                         but I'll tell you later."
>>> 
>>> 
>>>                   [MK] I understand. My sincere apologies again.
>>> 
>>>                               Given that this draft is intended to become 
>>> a stds track RFC I am concerned of any
>>>             text
>>>                               in this document that indicates (or even 
>>> hints) that TCP could acknowledge pure
>>>             ACKS
>>>                               (this holds particularly the rules and text 
>>> in Sec 3.2.2.5.1 for the
>>>                               "Increment-Triggered ACKs"). If it is seen 
>>> necessary that this doc should have
>>>             such
>>>                               pieces of rules and text, I am fine if any 
>>> such text is moved to an appendix as
>>>             long as
>>>                               the appendix makes it cristal clear that the 
>>> text is valid only in case one is
>>>                               implementing an experiment as per 
>>> [I-D.ietf-tcpm-generalized-ecn].
>>> 
>>> 
>>>                         [BB] See point below about "Generic (Mechanistic) 
>>> Reflector".
>>> 
>>> 
>>>                               Why?
>>> 
>>>                               1) It is well known that TCP does not 
>>> acknowledge ACKs and Standards track TCP has
>>>             not
>>>                               been specified to acknowledge ACKs. This 
>>> means that a reader/implementer of this
>>>             doc
>>>                               cannot correctly understand the rule for 
>>> "Increment-Triggered ACKs" unless there
>>>             is a
>>>                               normative reference to a spec that specifies 
>>> ACKs of ACKs (or tells that it is
>>>             even
>>>                               possible).
>>> 
>>> 
>>>                         [BB] ACKs of ACKs can indeed be tricky. But 
>>> there's no need to consider not ACKing ACKs
>>>             as an
>>>                         architectural principle. Not Acking ACKs on 
>>> principle certainly avoids some tricky
>>>             problems.
>>>                         However, we have a new situation here where, in 
>>> limited circumstances, ACKs of ACKs are
>>>                         necessary.
>>>                         So the WG has already worked through the tricky 
>>> problems and they have been addressed in
>>>             the
>>>                         draft
>>>                         (e.g. mistaking ACKs of ACKs for DupACks, infinite 
>>> ping-pong, etc). We'll discuss below
>>>             whether
>>>                         you've found some more trickiness.
>>> 
>>> 
>>>                   (MK] I did not mean to refer to any principle but, as I 
>>> said, that a reader/implementer cannot
>>>             correctly
>>>                   understand the rule for "Increment-Triggered ACKs" 
>>> because it is well-known to her/him that
>>>             TCP does not Ack
>>>                   ACKs. This fact is that one can ack ACKs is not 
>>> specified in this doc
>>> 
>>> 
>>>             [BB2] The fact that one /can/ ack ACKs is specified in the 
>>> AccECN draft. It clearly says:
>>>                 "Although TCP normally only ACKs newly delivered data, in 
>>> this case it is mandatory for A to
>>>             emit ACKs of ACKs
>>>             because they feed back new congestion state (useful in case B 
>>> starts sending)."
>>> 
>>>             We'll also make it clearer that "in this case" means "in the 
>>> case of ECN-capable ACKs". And we'll
>>>             make it clear that it's
>>>             conditional on carefully following all the conditions in any 
>>> draft that specifies ECN-capable ACKs.
>>> 
>>> 
>>>       [MK2] Right, but this (acking Acks) is experimental behaviour, 
>>> needed only for those participating the
>>>       experiment. However, this doc specifies the behaviour in the 
>>> Increment-Triggered ACKs rule: MUST emit an ACK".
>>>       If one implements this draft but is not interested in ECN++ nor any 
>>> other similar experiment that allows
>>>       ECN-capable ACKs, one must still implement these Acks of Acks but 
>>> there is no specification nor normative ref
>>>       how to do it (correctly, including all the conditions). If then this 
>>> implementation is deployed in the wild, it
>>>       automatically becomes part of the experiment once the other endpoint 
>>> enables ECN-capable ACKs but without the
>>>       necessary safety procedures. This doc cannot depend on an 
>>> experimental spec which explains how to do it and what
>>>       conditions to follow.
>>> 
>>> 
>>> [BB3] See
>>> #1) Generic should not be confused with experimental.
>>> #2) Completeness
>>> #3) Does not specify the necessary details?
>>> 
>>>                   nor does this doc give a (normative) reference where it 
>>> is specified,
>>> 
>>> 
>>>             [BB2] As already explained, there is no normative ref where 
>>> acking ACKs is specified in general,
>>>             because they are not
>>>             needed in general. They are specified whenever they are 
>>> needed, including in the examples Yoshi
>>>             pointed to (keepalive,
>>>             etc).
>>> 
>>> 
>>>       [MK2] An implementation of this draft send Acks of Acks "randomly" 
>>> and possibly multiple of them in a row but
>>>       this draft does not specify the necessary details unlike you claim 
>>> above.
>>> 
>>> 
>>> [BB3] See #4) Does not specify the necessary details
>>> 
>>> 
>>>       [MK2] Pls see also my reply to Yoshi. The specific examples Yoshi 
>>> mentions are sent in very limited cases and
>>>       result in sending only a single Ack of an Ack which is not causing 
>>> any harm to others unlike sending multiple
>>>       "random" Acks of Acks.
>>> 
>>>                   including the details on which TSecr value to add or 
>>> which SACK info if any to include when
>>>             acking a pure
>>>                   Ack.
>>> 
>>> 
>>>             [BB2] OK.  But I've had to read all the through SACK and TS 
>>> specs to guess what complication you
>>>             might have seen. If you
>>>             know what made you ask this question, it would be much easier 
>>> if you gave us a clue, or even just
>>>             told us explicitly.
>>> 
>>> 
>>>       [MK2] Sorry for not being clear/detailed enough. Maybe this was too 
>>> obvious to me. See details below for SACK
>>>       (and my reply to Richard for Timestamps for which I believe you 
>>> already figured out the problem).
>>> 
>>> 
>>> [BB3] See #3) Does not specify the necessary details?
>>> 
>>> 
>>>             * For TS, I'm pretty sure the normal rules (for which TSecr 
>>> value to include) apply.
>>>             * Similarly, for SACK, I can't see that anything new needs to 
>>> be said.
>>> 
>>>                   It is easy to misinterpret the "Increment-Triggered 
>>> ACKs", if one doesn't realize that pure
>>>             Acks may be
>>>                   acked.
>>> 
>>> 
>>>             [BB2] We'll make it clearer in the rule that 'packet' 
>>> intentionally includes packets without data.
>>> 
>>> 
>>>       [MK2] But as I said earlier, it would specify experimental behavior 
>>> in a PS doc.
>>> 
>>> 
>>> [BB3]  See: #1) Generic should not be confused with experimental.
>>> 
>>> 
>>>                         What is the new situation?
>>>                          *  Until ECN was introduced, TCP ACKs only 
>>> acknowledged data. So there was no need to
>>>                         acknowledge
>>>                             pure ACKs, which contain no data.
>>>                          *  When ECN was introduced in RFC3168, TCP ACKs 
>>> also acknowledged ECN markings.
>>>             However, because
>>>                             RFC3168 precluded pure ACKs from being 
>>> ECN-capable, there was still no need to
>>>             acknowledge
>>>                         pure
>>>                             ACKs.
>>>                          *  RFC5690, and now the ECN++ draft introduce the 
>>> possibility of ECN-capable pure ACKs.
>>>             So, in
>>>                         the
>>>                             limited circumstances described in the AccECN 
>>> draft, ECN-capable pure ACKs now need
>>>             to be
>>>                             acknowledged, because they contain new 
>>> information - their ECN field.
>>>                         Similarly, even though the final ACK of TCP's 3WHS 
>>> is an ACK of an ACK , it is sent
>>>             because it is
>>>                         needed (to prove that the SYN wasn't from a 
>>> spoofed address).
>>> 
>>> 
>>>                   [MK] All otherwise clear, but I disagree that the final 
>>> ACK of TCP's 3WHS is an ACK of an ACK.
>>>             It is required
>>>                   because SYNACK contains control data that eats one 
>>> sequence number, i.e., it advances RCV.NXT
>>>             at the client
>>>                   end and when the ACK arrives at the server it is needed 
>>> to advance SND.UNA. Very different
>>>             from Acks of Acks
>>>                   in this draft.
>>> 
>>> 
>>>             [BB2] Yes, not the same, but similar - ECN information is also 
>>> control data. But agreed that it
>>>             doesn't consume a
>>>             sequence no. Neither do keepalive probes or their ACKs.
>>> 
>>> 
>>>       [MK2] Right, but the actual purpose of Acks is to acknowledge 
>>> sequence numbers (that have arrived) and the logic
>>>       of many crucial TCP algos depend on how packets are acked.
>>> 
>>> 
>>> [BB3] Good protocol specs never define or limit the uses of the protocol 
>>> (it would be pointless to try anyway).
>>> 
>>> If you only wanted ACKs to be used to acknowledge sequence numbers, you 
>>> should have objected to RFC3168, which used ACKs to
>>> feed back ECN as well. Or you could go further back and object to SACK 
>>> because it acknowledges gaps, not sequence numbers
>>> that have arrived.
>> 
>> [MK3] With both RFC 3168 and SACK the ACKS ack the sequence numbers in the 
>> first place. These specs just piggyback additional information on ACKS that 
>> would be sent anyway. They do not introduce extra pkts to be sent like this 
>> spec does. I think this makes a major difference.
>> 
>> [snipped the rest of msg as the msg was too long for the wg list]
>
> -- 
> ________________________________________________________________
> Bob Briscoe                               http://bobbriscoe.net/
>
>