Re: [tcpm] draft-ietf-tcpm-accurate-ecn-24 addressing all WGLC comments

[Markku Kojo <[e-mail]>]

Bob, please see [MK2] inline.

And apologies for the long delay in replying, again.

On Mon, 5 Jun 2023, Bob Briscoe wrote:

> Markku, pls see [BB2] inline
> 
> On 26/05/2023 22:56, Markku Kojo wrote:
>       Bob,
>
>       My apologies you had to wait for the scenarios as it took much longer with my limited cycles than I thought.
>       Anyways, please see my reply to Richard, some scenarios are also included there.
>
>       To keep things easier, it might be good to try to keep the discussion on Acks of Acks (mainly) in the thread
>       with my reply to Richard.
>
>       However, see inline tagged [MK].
>
>       On Wed, 24 May 2023, Bob Briscoe wrote:
>
>             Markku,
>
>             Sorry, it's taken a week to build a comprehensive reply to this long email. See inline tagged
>             [BB]...
>
>             On 17/05/2023 12:24, Markku Kojo wrote:
>                   Hi Michael, all,
>
>                   On Sun, 14 May 2023, [e-mail] wrote:
>
>                               On 30. Mar 2023, at 16:53, Bob Briscoe <[e-mail]>
>                               wrote:
>
>                               Michael, Yoshi, Ian (as tcpm chairs),
>
>                               To close off the WGLC, I have just posted a new rev of
>                               accurate-ecn. Hyperlinks quoted at the end.
>                               You will see the diff is rather extensive. I won't give a
>                               summary of all the diffs like I usually do. Instead I can just
>                               refer to the summary I gave in the presentation on Monday:
>                              
>             https://datatracker.ietf.org/meeting/116/materials/slides-116-tcpm-draft-ietf-tcpm-accurate-ecn
>
>                               Thank you again to the people who reviewed this during the WGLC:
>                               Michael Tüxen, Alex Burr, Gorry Fairhurst and Markku Kojo.
>
>                               All changes are editorial, apart from removing the para about
>                               not mistaking certain ACKs of ACKs for DupACKs, which I will add
>                               to a rev of the ECN++ draft, hopefully later this week.
>
>                               On the list, we have seen agreement from all the reviewers to
>                               these changes, except no response from Markku yet.
>                               On Monday, I told Markku that I would post the draft in a few
>                               days, so everyone can see the updates and diff.
>
>                         Anyone having additional comments? In particular Markku regarding loss
>                         recovery?
> 
>
>                   My apologies for being late with my reply to the author's comments on my review (I've
>                   been extremly busy with other issues since the wg mtng in Yokohama, including the rest
>                   of mtng week).
>
>                   I don't have much new comments but it seems that my major concern regarding the problem
>                   of sending ACKs of ACKs was not fully understood.
>
>                   The first thing where I think I was not quite clear is that the major problem with ACKs
>                   of ACKs is not that a pure Ack is made ECN-capable. Instead, the problem is in
>                   generating an Ack of an pure Ack and that is what one should prohibit to avoid problems.
>                   I understand that it might be problematic to formulate rules whether generating an Ack
>                   of an Ack is allowed (and when), instead of just disabling sending ECN-capable ACKs.
>                   I don't have a strong opinion which way the problems with ACKs of ACKs is avoided as
>                   long as they are avoided.
> 
>
>             [BB] See later after your similar point (following your 'Why?' heading)...
> 
>
>                   I am preparing a few scenarios to illustriate the problems ACKs of ACks raise and will
>                   send them shortly once I have formulated a more thorough reasoning why sending ACKs of
>                   ACKs is not really a good idea and even seems to be unnecessary in most if not all
>                   cases, i.e., it just results in sending unnecessary packets with not much useful effect
>                   but creates a notable number of problems.
> 
>
>             [BB] Having waited this long, it's rather disappointing to still hear you say "I have an
>             argument,
>             but I'll tell you later."
> 
>
>       [MK] I understand. My sincere apologies again.
>
>                   Given that this draft is intended to become a stds track RFC I am concerned of any text
>                   in this document that indicates (or even hints) that TCP could acknowledge pure ACKS
>                   (this holds particularly the rules and text in Sec 3.2.2.5.1 for the
>                   "Increment-Triggered ACKs"). If it is seen necessary that this doc should have such
>                   pieces of rules and text, I am fine if any such text is moved to an appendix as long as
>                   the appendix makes it cristal clear that the text is valid only in case one is
>                   implementing an experiment as per [I-D.ietf-tcpm-generalized-ecn].
> 
>
>             [BB] See point below about "Generic (Mechanistic) Reflector".
> 
>
>                   Why?
>
>                   1) It is well known that TCP does not acknowledge ACKs and Standards track TCP has not
>                   been specified to acknowledge ACKs. This means that a reader/implementer of this doc
>                   cannot correctly understand the rule for "Increment-Triggered ACKs" unless there is a
>                   normative reference to a spec that specifies ACKs of ACKs (or tells that it is even
>                   possible).
> 
>
>             [BB] ACKs of ACKs can indeed be tricky. But there's no need to consider not ACKing ACKs as an
>             architectural principle. Not Acking ACKs on principle certainly avoids some tricky problems.
>             However, we have a new situation here where, in limited circumstances, ACKs of ACKs are
>             necessary.
>             So the WG has already worked through the tricky problems and they have been addressed in the
>             draft
>             (e.g. mistaking ACKs of ACKs for DupACks, infinite ping-pong, etc). We'll discuss below whether
>             you've found some more trickiness.
> 
>
>       (MK] I did not mean to refer to any principle but, as I said, that a reader/implementer cannot correctly
>       understand the rule for "Increment-Triggered ACKs" because it is well-known to her/him that TCP does not Ack
>       ACKs. This fact is that one can ack ACKs is not specified in this doc
> 
> 
> [BB2] The fact that one /can/ ack ACKs is specified in the AccECN draft. It clearly says:
>     "Although TCP normally only ACKs newly delivered data, in this case it is mandatory for A to emit ACKs of ACKs
> because they feed back new congestion state (useful in case B starts sending)."
>
> We'll also make it clearer that "in this case" means "in the case of ECN-capable ACKs". And we'll make it clear that it's
> conditional on carefully following all the conditions in any draft that specifies ECN-capable ACKs.

[MK2] Right, but this (acking Acks) is experimental behaviour, needed only 
for those participating the experiment. However, this doc specifies the 
behaviour in the Increment-Triggered ACKs rule: MUST emit an ACK".
If one implements this draft but is not interested in ECN++ nor any other 
similar experiment that allows ECN-capable ACKs, one must still implement 
these Acks of Acks but there is no specification nor normative ref how to 
do it (correctly, including all the conditions). If then this 
implementation is deployed in the wild, it automatically becomes part of 
the experiment once the other endpoint enables ECN-capable ACKs but 
without the necessary safety procedures. This doc cannot depend on 
an experimental spec which explains how to do it and what conditions to 
follow.

>       nor does this doc give a (normative) reference where it is specified,
> 
> 
> [BB2] As already explained, there is no normative ref where acking ACKs is specified in general, because they are not
> needed in general. They are specified whenever they are needed, including in the examples Yoshi pointed to (keepalive,
> etc).

[MK2] An implementation of this draft send Acks of Acks "randomly" and 
possibly multiple of them in a row but this draft does not specify the 
necessary details unlike you claim above.

[MK2] Pls see also my reply to Yoshi. The specific examples Yoshi mentions 
are sent in very limited cases and result in sending only a single Ack of 
an Ack which is not causing any harm to others unlike sending multiple 
"random" Acks of Acks.

>       including the details on which TSecr value to add or which SACK info if any to include when acking a pure
>       Ack.
> 
> 
> [BB2] OK.  But I've had to read all the through SACK and TS specs to guess what complication you might have seen. If you
> know what made you ask this question, it would be much easier if you gave us a clue, or even just told us explicitly.

[MK2] Sorry for not being clear/detailed enough. Maybe this was too 
obvious to me. See details below for SACK (and my reply to Richard for 
Timestamps for which I believe you already figured out the problem).

> * For TS, I'm pretty sure the normal rules (for which TSecr value to include) apply.
> * Similarly, for SACK, I can't see that anything new needs to be said.
>
>       It is easy to misinterpret the "Increment-Triggered ACKs", if one doesn't realize that pure Acks may be
>       acked.
> 
> 
> [BB2] We'll make it clearer in the rule that 'packet' intentionally includes packets without data.

[MK2] But as I said earlier, it would specify experimental behavior in a 
PS doc.

>             What is the new situation?
>              *  Until ECN was introduced, TCP ACKs only acknowledged data. So there was no need to
>             acknowledge
>                 pure ACKs, which contain no data.
>              *  When ECN was introduced in RFC3168, TCP ACKs also acknowledged ECN markings. However, because
>                 RFC3168 precluded pure ACKs from being ECN-capable, there was still no need to acknowledge
>             pure
>                 ACKs.
>              *  RFC5690, and now the ECN++ draft introduce the possibility of ECN-capable pure ACKs. So, in
>             the
>                 limited circumstances described in the AccECN draft, ECN-capable pure ACKs now need to be
>                 acknowledged, because they contain new information - their ECN field.
>             Similarly, even though the final ACK of TCP's 3WHS is an ACK of an ACK , it is sent because it is
>             needed (to prove that the SYN wasn't from a spoofed address).
> 
>
>       [MK] All otherwise clear, but I disagree that the final ACK of TCP's 3WHS is an ACK of an ACK. It is required
>       because SYNACK contains control data that eats one sequence number, i.e., it advances RCV.NXT at the client
>       end and when the ACK arrives at the server it is needed to advance SND.UNA. Very different from Acks of Acks
>       in this draft.
> 
> 
> [BB2] Yes, not the same, but similar - ECN information is also control data. But agreed that it doesn't consume a
> sequence no. Neither do keepalive probes or their ACKs.

[MK2] Right, but the actual purpose of Acks is to acknowledge sequence 
numbers (that have arrived) and the logic of many crucial TCP algos depend 
on how packets are acked.
TCP keep-alives are sent in a very specific case only (pls see my reply 
to Yoshi). And, keep-alives could include data but that is not 
recommended (I don't quite know/remember why it was recommended like this 
though but that is not impotant here).

>             It is true that not ACKing ACKs is well-known. However, whether it's well-known as a /principle/,
>             or
>             just as a current /feature/ of TCP is not clear. Anyway, the IETF's job is to update RFCs that
>             are
>             "well-known". We don't have to jump through any special procedural hoops to do something
>             different
>             from what is "well-known". Even if it were prohibited in a stds track RFC, we just have to
>             specify
>             what has to be done instead; in another stds track RFC.
> 
>
>       [MK] Again, I didn't mention it as a /principle/ but as a crucial piece of information that the reader needs
>       to be noted, that is, the things are now different from what is well-known.
>
>       Sure IETF's job is to update RFCs, but if one changes what is prohibited in a stds track RFC, one needs to
>       understand the consequences and explain them as well as give the justification why the change can be done
>       (without problems), instead of just specifying the change.
> 
> 
> [BB2] If ack'ing ACKs is prohibited, pls say where. Otherwise, pls confirm that you are not aware of anywhere that
> explicitly prohibits it.

[MK2] Apologies for not repeating. Please see my original item 2) a few 
rows below with my explanation included (i.e., RFC 5681, Sec 2, last 
para).  Prohibiting a receiver from generating more than one ACK for every 
incoming [data] segment is to guarantee the fundamental well-known 
behavior of TCP which is used as a prerequisite in tons of loss-recovery 
and CC mechanisms for TCP. If it is not honored, all these mechanism stop 
working correctly. That's why it is "MUST NOT".

> Whatever, I have no problem with explaining the consequences. As I've said, that's not relevant in AccECN, 'cos the
> increment-triggered ACK rule doesn't cause ACKs of ACKs unless you're also making ACKs ECN-capable. So only relevant in
> the ECN++ draft

[MK2] Please note it is the other (remote) endpoint of the TCP connection 
that makes ACKs ECN-capable and that remote endpoint implementation is 
totally independent of the local endpoint implementing the 
increment-triggered ACK rule and thereby completely out of control for the 
local endpoint.

>             If there are any tutorials, course notes or text books out there that say that not ACKing ACKs is
>             a
>             well-known principle, that's not the IETF's problem. It is the job of the tutors, lecturers and
>             text
>             book authors who wrote those materials to update them.
> 
>
>                   2) ACKs of ACKs tend to trigger duplicate Acks. There are tons of algorithms that rely
>                   on the packet conservation principle and the fact that TCP never injects a dupAck unless
>                   a *data* packet has arrived and left the network. This is enforced with "MUST NOT" in
>                   RFC 5681, Sec 4.2, because not conforming to this rule makes any algorithm that rely on
>                   the rule to work incorrectly. These algorithms include (triggering) Fast Retransmit,
>                   (controlling packet rate during) Fast Recovery, (detecting spurious RTOs in) F-RTO,
>                   (calculating PipeAck in) RFC 7661, (calculating DeliveredData in) PRR, etc. Furthermore,
>                   it would make imposible to come up with any new algorihms that rely on this important
>                   basic rule. In most cases such extra dupAcks make these algorithms too aggressive
>                   because any extra dupAck is likely to inject extra packet(s) to the network.
>
>                   So, it should be cristal clear that without SACK (or Timestamps) a TCP *MUST NOT* send
>                   ACKs of ACKs.
> 
>
>             [BB] Constraining the /Data Receiver/ as you propose would create an interop problem.
>             Explanation: Consider host A and B are not using SACK or timestamps. Nonetheless, with your
>             approach, host A can still send ECN-capable pure ACKs to host B. Then, your rule puts host B in
>             an
>             impossible position, where it gets congestion notifications on ECN-capable pure ACKs, but it is
>             not
>             allowed to send any feedback about them.
>
>             Instead, if neither timestamps nor SACK are in use for the connection, we need to constrain the
>             /Data Sender/ of a half connection from sending ECN-capable ACKs in the first place. This is the
>             approach the WG has adopted in the AccECN and ECN++ specs.
> 
>
>       [MK] I think I said in the beginning that I have no strong opinion which way Acks of Acks are disabled.
>       However, I apologize that I didn't explain why I phrased MUST NOT send ACKs of ACKs. This is because it might
>       be still useful to allow CE-marked pure Acks and take care of Ack CC by some other means than Acks of Acks.
>       Currently the draft mandates Acks of Acks as the only way to report Ack congestion and I think it is too
>       restrictive in a stds track doc, e.g., it rules out reducing Ack rate simply by reducing data send rate which
>       would solve the interop problem in a very simple way. Moreover, When B gets congestion notifications on
>       ECN-capable pure ACKs, not sending Acks of Acks does not prevent sending feedback; such feedback need not to
>       be delivered immediately but by the time needed. Please see more on this in my reply to Richard.
> 
> 
> [BB2] I think you've got the wrong end of the stick. The increment-triggered ACK rule is not, and never was, motivated by
> AckCC. We wrote the 2 rules (Change-triggered and Increment-triggered) before someone pointed out that we'd used the
> wording 'packet', which included pure ACKs. We changed one occurrence to data packet, but realized that the other
> occurrence was correctly stated as just 'packet', because it fed back more timely congestion information.

[MK2] I understand that the rule was not motivated nor written by AckCC, 
but the result of the currently written rule is that it prevents from 
providing feedback for AckCC purposes in some other reasonable way. That 
is, it mandates sending the Acks of Acks even in case one would have 
alternative/better way to provide the feedback (or does not require any 
feedback).

> Congestion information on pure ACKs doesn't mean the pure ACKs are the sole cause of congestion; it just means that they
> are part of an aggregate that is congesting a bottleneck. CE on pure ACKs conveys information that is useful for
> maintaining cwnd when no data is being sent (much better than CWV which relies on magic numbers to guess how to decay
> information during idle periods).

[MK2] I think I well understand the thinking behind this but it seems to 
me that these Acks of Acks are still mostly unnecessary or late (and 
probably stale) information. Please see what I replied to Richard, trying 
to explain why.

> I'm not particularly interested in AckCC, altho I suppose I am separately interested in draft-ietf-tcpm-ack-rate-request
> and the equivalent in QUIC, because I want QUIC (and TCP) to be able to thin their ACK stream themselves, rather than
> relying on ACK thinning in the network.

[MK2] Agreed that it's better that TCP (and QUIC) thin their ACK stream 
themselves. This said, I think that this doc should not restrict (or 
decide in advance) as a stds track doc the way how ECN-capable and 
CE-marked pure Acks are best fed back to a data receiver. How to 
(best) implement CC with ECN-capable Acks is obviously experimental (if 
not research). I believe we could agree on this?

[MK2] The current text mandates one particular way to feed back 
congestion on pure Acks by bundling it together with the feedback for 
data segments that experience congestion. This may be workable for a 
spesific (experimental) CC that seemingly has been guiding the 
development of this feature but not necessarily as a general approach 
that suits for various other CC algos. A stds track protocol should serve 
various different approaches, or at least not make them unreasonable to 
implement, otherwise the protocol should be made experimental.

>             Specifically:
>              *  The WG makes sure that RFCs about the /Data Sender/ of a half connection (e.g. the ECN++
>                 experiment or other future RFCs) specify that sending ECN-capable pure ACKs is conditional on
>                 having another way to distinguish DupACKs, e.g. negotiating SACK or timestamps (and I will
>                 respond to your later points on the details of these).
>              *  The AccECN spec (which primarily specifies the feedback behaviour of a /Data Receiver/ in a
>                 half-connection) then only needs to define the Increment-triggered ACK rule.
>             The two together lead to the same outcome you want. But without the interop hole of your
>             approach.
>
>             This is consistent with the "Generic (Mechanistic) Reflector" approach of the AccECN spec which
>             says:
>             "AccECN is designed to be a generic reflector of whatever ECN markings it sees, whether or not
>             they
>             are compliant with a current standard."
>
>             These ACKs of ACKs are generically necessary to feed back congestion notifications from possible
>             incoming packet patterns, not specifically for ECN++ or AckCC [RFC5690], or any other future RFC
>             (forward compatibility). We'll edit the reference to ECN++ to make it clearer that it's one
>             example,
>             not the only case.
>
>             Here's another example of the generic reflector approach, already in the draft:
>             "Although RFC 3168 prohibits an ECN-capable SYN, providing feedback of ECN marking on the SYN
>             supports future scenarios in which SYNs might be ECN-enabled (without prejudging whether they
>             ought
>             to be). ... "
> 
>
>                   So, it should be cristal clear that without SACK (or Timestamps) a TCP *MUST NOT* send
>                   ACKs of ACKs.
> 
>
>             I understand that you want this but, as just explained, without SACK or timestamps, the correct
>             approach is to prevent the Data Sender putting the Data Receiver in the position where it would
>             have
>             to ACK ACKs in the first place.
>
>             In a connection without SACK or timestamps, if the Data Receiver were to get lots of congestion
>             notifications on ECN-capable ACKs, it would face a difficult dilemma. Which would be more
>             important:
>             Signalling congestion by ACKing ACKs? or ensuring the performance improvements like Fast
>             Retransmit,
>             Fast Recovery etc. work well? The former would prevent harm to others, the latter would prevent
>             harm
>             to self.
> 
>
>       [MK] Please see my reply to Richard to understand why Acks of Acks cause Fast Retransmit, Fast Recovery, etc
>       to cause harm to others in the first place. And also why most of the Acks of Acks seem to be just unnecessary
>       load to the network, that is, they harm others without (much) benefits.
> 
> 
> [BB2] I am preparing my own reply to that email. Briefly, in the FR/FR case that you highlight more harm to others
> certainly results, but I don't believe that generalizes to all cases. And I don't agree that the ACKs of ACKs don't have
> much benefit for their cost. For instance, compare them with the cost of all the ACKs that are currently sent with
> ACK-ratio 2, relative to their benefit.

[MK2] Of course, the harm caused to others depends on the case; there are 
millions of different variations of scenarios where harm is done and 
another set of millions of scenario variations where harm is not done. 
But if harm is done even in one case it must be addressed unless such a 
scenario can be proved to be a (random) corner case.
One fundamental problem in allowing Acks of Acks is that it 
potentially prevents developping any future algos that rely on the fact 
that fake dupacks should not occur. FR/FR and other problems I raised are 
just something that immediately came into my mind and threreby they 
probably are just the tip of the iceberg; there are potentially many other 
existing algos that may also break. We need to be very careful to not 
introduce Acks of Acks before all potential disadvantages have been 
considered much more closely.

>             Nonetheless, I will add some text to the AccECN draft that explains why it is important for other
>             RFCs not to put a Data Receiver in the position where it has to ACK ACKs iff there is no way to
>             distinguish them from DupACKs.
> 
>
>                   3) Even with SACK or Timestamps enabled it is not clear what an
>                   implementer should do. With SACK the AckECN authors seem to make an assumption, which
>                   seems obvious but is not, that an ACK of ACK would would never include SACK option and
>                   hence it could be distinguished from a dupAck. However, RFC 2018 specifies: "If sent at
>                   all, SACK options SHOULD be included in all ACKs which do not ACK the highest sequence
>                   number in the data receiver's queue. So, if there is a hole in the receiver's queue, the
>                   assumption is incorrect and it is unclear which SACK info to include into the SACK
>                   option. Whatever one selects to include, it makes DSACK (RFC 2883] void and breaks any
>                   DSACK-based algorithms unless RFC 2018 is updated.
> 
>
>             [BB]
>             Reading the draft, it is very clear that there is no such assumption. The text said solely what
>             it
>             meant:
>
>                ... a host in AccECN
>                mode that is sending ECN-capable pure ACKs SHOULD add one of the
>                following additional checks when it tests whether an incoming pure
>                ACK is a duplicate:
>
>                o  If SACK has been negotiated for the connection, but there is no
>                   SACK option on the incoming pure ACK, it is not a duplicate;
>             That is, if an incoming ACK were a duplicate, it would have a SACK option on it. This /relies/ on
>             the rule in RFC2018 that you quote. So there is no need (nor intention) to change SACK behaviour
>             in
>             any way.
> 
>
>       [MK] Sorry I don't understand your claim. Assume A sends data pkts 0,1,2,3,..,10 in the current window to B
>       and pkt 1 is dropped. Simultaneously, B sends data to A such that the data pkts arrive at A after A has
>       injected pkt 10 to the network. These pkts trigger pure cumulative Acks from A to B that follow A's data pkts
>       1..10 and enough of the Acks get CE-marked due to congestion path A to B. When pkts 2..10 arrive at B, each
>       of them trigger a valid dupAck with a SACK block included. When the CE-marked Acks that follow data pkts
>       2..10 arrive at B, B needs to feedback congestion info on them in Acks of Acks. These Acks of Acks cannot
>       cumulatively ACK the highest sequence number in the data receiver's queue (pkt 10) since pkt 0 is the highest
>       pkt arrived insequence, so the Ack of Ack must include a SACK block as per RFC 2018. What is the SACK block
>       info that the implementer, who follows carefully advice in RFC 2018 and there is no other advice, should
>       include in these Acks unless RFC 2018 is not changed? How does the implementer know how to Ack, if it not
>       specified?
> 
> 
> [BB2] Surely those ACKs of ACKs from B to A are just further valid DupACKs with a SACK block included as defined in
> RFC2018, still cumulatively ack'ing packet 0 delivered before the loss.
> 
> As per our proposal, ACKs with SACK blocks that repeat a cumulative ACK /can/ be counted as DupACKS, which is correct in
> this case, because they really are valid DupACKs.

[MK2] Sorry for not repeating DSACK (RFC 2883] that I originally mentioned 
in item 3) above. As per RFC 2883, these DupAcks should report the 
sequence number range of the packet that triggered the acknowledgement. 
So, which sequence number range will be reported in the first SACK block 
of these DupAcks (Acks of Acks)?
This falls under the same rule of RFC 5681 that one must not ack more 
than once an arrival of a data segment. Otherwise, it will break all 
mechanisms that rely on the behavior specified in RFC 2883, including RFC 
3708 and (TLP of) RACK (RFC 8985) as well as any potential new and useful 
algos that rely on this rule.

So, I believe a random implementer cannot understand how to implement such 
an Ack of Ack when SACK is enabled unless it is clearly specified. E.g., 
your suggestion above is incorrect as it would include a (random) 
DSACK-block and would harm any DSACK-based algo. It is important to 
understand that all TCP specifications have been written with an 
assumption that TCP does not ack pure Acks and therefore trying to 
interpret the specs for the case where one would ack an Ack easily 
results in an incorrect implementation and potential harm to others or 
other mechanisms (pls see my reply w.r.t Timestamps to Richerd, which is 
an apt example of misinterpretation).

> Now, let's see what happens if the retransmission of packet 1 arrives at B before some of those ACKs of ACKs have left B
> (unlike your scenario the retransmission reached B after all the ACKs of ACKs had left). Once the retransmission of 1 has
> arrived and fills the hole, any ACKs of ACKs sent after it arrives will cumulatively ACK 10, and they will not need a
> SACK block. So, even if more than one ACK cumulatively ACKs 10, they will not be counted as DupACKs, because they have no
> SACK block.

[MK2] Sure, only when all packets have arrived in order and there is no 
hole in the suquence space, the rule works just fine. But it must work 
allways. Otherwise, it would only work partially and we don't want 
algorithms or software that almost works.

> In summary, if I've understood your scenario correctly, I believe it supports the proposed additional condition (for the
> ECN++ draft) that an ACK without SACK cannot be a DupACK.

[MK2] Yes, but in order to make it implementable we need to separately 
specify how to Ack pure Acks (w/ SACK) and make an 
exception/clarification for the RFC 2018 rule. Note also that RFC 6675 
(current SACK-based FR algo) changed the definition of DupAck from that of 
previous version (RFC 3517). This is likely to make it a bit triggier and 
the SACK-enhanced version of F-RTO thet was based on RFC 3517 
definition seems to require modeification/clarification, maybe some other 
specs too.

>             (Note: to check this text, you'll need to refer to the previous AccECN draft here:
>                 https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-accurate-ecn-23#section-3.2.2.5.1
>             It had been there since Jul 2021 (since -15). But, as explained above, it is longer in the latest
>             AccECN draft (-24), because it has been moved to the editor's copy of the ECN++ draft, which I
>             wrote
>             on 4 Apr, but I've been waiting for your reply before submitting it.)
> 
>
>                   With Timestamps some algorithms like Eifel detection breaks. Moreover, there are other
>                   existing or potentially to-be-created heauristics, including various measurement tools,
>                   that rely on the fact that TCP does not echo a later Timestamp in a pure Ack than what
>                   arrived with the latest data packet. Any such mechanisms are subject to break.
> 
>
>             [BB] I don't fully understand what you're saying here. Can you be clearer please?
>             And can you please bear in mind that we are in the WGLC processing stage now. So review comments
>             ought to be suggesting very specific changes to the text under review.
> 
>
>       [MK] Again, the same problem as with SACK. It is unspecified which TSecr value to put in Acks of Acks. That
>       has not been specified anywhere, so how can I or anyone else check what breaks if anything, and how can an
>       implementer know which TSecr value to include in Acks of Acks?
>
>       It is hard to propose specific changes to text that does not exist or to a problem that seems to be a missing
>       piece of design or a design flaw, until the text exists or the intended design is known or the potentiel
>       design flaw is mutually agreed whether there is a flaw or not.
>
>       (Please note also that the Eifel problem seems not to be serious, but instead the timestamps rule you
>       proposed to distinguish dupAcks from Acks of Acks seems suspicious and calls for clarification).
>
>       Please see more details in my reply to Richard.
> 
> 
> [BB2] In your other email to Richard, I think you were right that the TS option cannot always be used to filter out
> DupACKs. That's because timestamps echo the oldest unack'd packet, not the newest. But I believe the SACK condition is
> more solid.

[MK2] Agreed, but needs a careful review of all potential implications.

> In the reply I'm preparing to that email to Richard, I've suggested that the rule in the ECN++ draft might be that a Data
> Sender can only set ECT on pure ACKs:
>  *  if SACK has been negotiated
>  *  else if TS is in use and DupACKs are not used to detect losses (e.g. once the RACK algorithm is not using DupACKs).
> But pls wait for that reply - it won't be immediately clear here why I'm saying this.
> 
>
>             Again, this text about extra DupACK checks has now been removed from AccECN and will shortly
>             appear
>             in the ECN++ draft instead. I shall post the new ECN++ draft shortly.
> 
>
>                   It might be good to hold discussing any details on what breaks and how/why and what are
>                   the consequences until I have sent my reply with scenarios to Richard.
> 
>
>             [BB] Please try to prioritize any comments about the text that is now left in the AccECN draft.
>             The
>             WGLC of AccECN is waiting for no-one else at the moment.
> 
>
>       [MK] Unfortunately I was not able to do this, sorry. I got confused already earlier when I was reviewing
>       AccECN and ended up checking ECN++ quickly as I noted that AccECN draft cited it for these issues. When
>       reading ECN++ I found Sec 3.3.3 and read:
>
>        "The question of whether and how the receiver of pure ACKs is required to
>         feed back any CE marks on them is outside the scope of the present
>         specification because it is a matter for the relevant feedback
>         specification ([RFC3168] or [I-D.ietf-tcpm-accurate-ecn])."
>
>       This got me totally confused together with the later discussion on the mic at Yokohama mtng because that was
>       what I had read, and I was not able to understand what was intended to go where. And I still not quite know
>       but I think I have a better hunch. The above text still reads in ECN++ draft, but hopefully not in editor's
>       copy?
> 
> 
> [BB2] The split between the drafts is essentially as follows:
>  *  AccECN deals primarily with what feedback the data receiver of a half-connection emits, and how the data sender of a
>     half-connection interprets that feedback. Plus, of course, the negotiation to set up the feedback capability for a
>     connection.
>  *  ECN++ primarily gives conditions for the data sender of a TCP half-connection to set the IP-ECN field.
> So the scopes looks something like this half connection:
> 
>           ______________________
>        ,-'     Feedback          `-
> TCP   (     <-----------------     \
>        `-.___________________AccECN |
>       .---.                  `.     |
> IP   (ECN++) ---------------> |     ;
>       '---'                    \   /
>                                 "-"
>       Snd                       Rcv
> 
> Although, with ACK packets, it's even more confusing, because they are normally considered only at the TCP layer, but
> here, we are thinking of feedback of the ECN field in the IP header of the ACK (which is usually fed back in the TCP
> header of data packets, except in these more unusual ACK of ACK cases).
> 
> This is why
> * the increment-triggered ACK rule is in AccECN,
> * but the condition on whether to set the IP/ECN field of a TCP ACK is in ECN++.
> 
> 
>
>       Anyways, the major problem is not with any certain text phrases but whether DupAck vs. Ack of Ack problem is
>       solvable and whether injecting Acks of Acks really is a needed
> 
> 
> [BB2] Agreed
>
>       and mature enough feature that can be part of a stds track protocol.
> 
> 
> [BB2] I hope you can see that, without ECN-capable ACKs, the mechanistic reflection rules in the AccECN draft would still
> apply, but never lead to ACKs of ACKs.

[MK2] Thank you for the clarification and apologies for still mixing 
comments for AccECN and ECN++ drafts. This seems  all clear but to 
reiterate: enabling ECN-capable ACKs at the remote end is out of control 
for the local end emitting the ACKs of ACKs.

> So if ECN-capable ACKs never mature, everything in the AccECN draft still sits
> correctly on the standards track.

[MK2] Sorry to reiterate: the problem here I'd like to solve is that 
AccECN mandates (MUST) Increment-Triggered ACKs -rule in stds track. It 
means that if someone implements this stds track rule in some stack 
without the intent to participate in the ECN++ experiment, the behavior 
still sits in that stack. If someone sometime later on specifies and 
enables ECN-capable ACKs (at the other end) for some other purpose than 
ECN++, we will get these Acks of Acks even if we don't want them (maybe 
the ECN++ experiment never matured for some reason). 
That's why I seriously hesitate nailing down anything on Acking of Acks 
in a stds track doc at this point.

Wouldn't it be possible to move the Ack of Ack part of the rule into an 
experimental doc because that part is needed for experimental purposes 
only and reguires the necessary safeguards?

> This is not surprising, because we originally wrote the increment-triggered ACK rule without realizing that it could
> apply to pure ACKs.
> 
>
>             But also bear in mind that the chairs plan to take ECN++ into WGLC once AccECN WGLC has been
>             cleared. So we need to hear your actual argument about the DupACK text that has been moved to
>             ECN++
>             urgently too. We can't work with "I have an argument, but I'll tell you later".
> 
>
>       [MK] Apologies for the delay again. Please see my reply to Richard for my arguments.
> 
> 
> [BB2] Yes, thank you for all the work you've put into that reply.
> 
>
>                   One additional comment regarding the "Change-Triggered ACKs" rule is that it would be
>                   useful to make it more clear how this plays with delayed Acks and how it alters
>                   acknowledgement rate.
>                   I am not sure that what the draft currently says is quite correct:
>
>                    "The approach can lead to some additional ACKs but it feeds back
>                     the timing and the order in which ECN marks are received with minimal
>                     additional complexity. If CE marks are infrequent, as is the case for
>                     most AQMs at the time of writing, or there are multiple marks in a row,
>                     the additional load will be low.
>
>                   For example, consider a scenario with bidirectional traffic between A and B where B has
>                   a hole in sequence resulting in every data packet in the current RTT to become acked
>                   (pure duplicate Acks). This may result in a packet flow from A to B where every second
>                   packet is a pure (duplicate) Ack. If there is congestion on the path from A to B such
>                   that a significant number of (data) packets get marked, it may result in acking every
>                   data packet from A to B. This does not necessarily result in low additional load?
> 
>
>             [BB] I don't quite understand how every second packet from A to B is a pure duplicate ACK, but I
>             don't think I need to - I'll assume it's somehow possible.
>
>             Then I think you've somehow assumed that the data packets get CE-marked, but the interspersed
>             pure
>             ACKs don't (perhaps you're assuming that the pure ACKs in this case are not ECN-capable? Or
>             perhaps
>             you're assuming size-based packet marking?). Whatever, I agree that, if this scenario did occur,
>             then the change-triggered ACK rule would indeed lead to B ACKing every data packet that arrives,
>             with no delayed ACKs. {Note 1}
>
>             Nonetheless, the draft is quite open about the implications of the change-triggered ACK rule on
>             ACK
>             rate. In the sentence straight after the ones you quote, it says:
>                 "However, marking patterns with numerous non-contiguous CE marks could increase the load
>             significantly."
>             And a little earlier it starts out by saying:
>                 "...the 'Change-Triggered ACKs' rule could sometimes cause the ACK rate to be problematic for
>             high performance"
>
>             I don't think we need to include examples of how non-contiguous CE marking could occur. And if we
>             did, I'd prefer to use one that was less complex to explain, e.g. a high level of probabilistic
>             AQM
>             marking. But thank you for this point anyway.
> 
>
>       [MK] Thanks for pointing these additional sentences. I somehow missed them and/or did not manage to relate
>       them to the text I quoted. I think this is good enough to address the case I raised, particularly "numerous
>       non-contiguous CE marks could increase the load significantly."
>
>             {Note 1}: It's ironic that the existing behaviour "where B has a hole in sequence" also results
>             "in
>             every data packet in the current RTT to become acked" (by A). I'm not giving this as an excuse
>             for
>             introducing another case with the same bad behaviour. I'm just highlighting the irony.
> 
>
>       [MK] Maybe ironic but the fact that a hole in sequence results in every data packet in RTT to become acked
>       has a very good reason being as it is because those dupAcks directly control the data rate per the packet
>       conservation principle in various important algos at the data sender,
>       such as Fast Recovery, and this behaviour is therefore a crucial part of such congestion control algos.
> 
> 
> [BB2] What if the bottleneck moves to the reverse path, due to all the extra ACKs...?

[MK2] That would be transient behavior because these extra Acks are sent 
for around one RTT only. But nothing is perfect ;)

Best regards,

/Markku

> Regards
> 
> 
> Bob
> 
>
>       Best regards,
>
>       /Markku
> 
>
>             Regards
> 
>
>             Bob
> 
>
>                   Best regards,
>
>                   /Markku
> 
>
>                         Best regards
>                         Michael
>
>                               Cheers
> 
>
>                               Bob
>
>                               On 30/03/2023 15:15, [e-mail] wrote:
>                                     A New Internet-Draft is available from the on-line
>                                     Internet-Drafts
>                                     directories. This Internet-Draft is a work item of
>                                     the TCP Maintenance and
>                                     Minor Extensions (TCPM) WG of the IETF.
>
>                                        Title           : More Accurate Explicit
>                                     Congestion Notification (ECN) Feedback in TCP
>                                        Authors         : Bob Briscoe
>                                                          Mirja Kühlewind
>                                                          Richard Scheffenegger
>                                        Filename        :
>                                     draft-ietf-tcpm-accurate-ecn-24.txt
>                                        Pages           : 64
>                                        Date            : 2023-03-30
>
>                                     Abstract:
>                                        Explicit Congestion Notification (ECN) is a
>                                     mechanism where network
>                                        nodes can mark IP packets instead of dropping
>                                     them to indicate
>                                        incipient congestion to the endpoints.  Receivers
>                                     with an ECN-capable
>                                        transport protocol feed back this information to
>                                     the sender.  ECN was
>                                        originally specified for TCP in such a way that
>                                     only one feedback
>                                        signal can be transmitted per Round-Trip Time
>                                     (RTT).  Recent new TCP
>                                        mechanisms like Congestion Exposure (ConEx), Data
>                                     Center TCP (DCTCP)
>                                        or Low Latency, Low Loss, and Scalable Throughput
>                                     (L4S) need more
>                                        accurate ECN feedback information whenever more
>                                     than one marking is
>                                        received in one RTT.  This document updates the
>                                     original ECN
>                                        specification in RFC 3168 to specify a scheme
>                                     that provides more than
>                                        one feedback signal per RTT in the TCP header. 
>                                     Given TCP header
>                                        space is scarce, it allocates a reserved header
>                                     bit previously
>                                        assigned to the ECN-Nonce.  It also overloads the
>                                     two existing ECN
>                                        flags in the TCP header.  The resulting extra
>                                     space is exploited to
>                                        feed back the IP-ECN field received during the
>                                     3-way handshake as
>                                        well.  Supplementary feedback information can
>                                     optionally be provided
>                                        in two new TCP option alternatives, which are
>                                     never used on the TCP
>                                        SYN.  The document also specifies the treatment
>                                     of this updated TCP
>                                        wire protocol by middleboxes.
>
>                                     The IETF datatracker status page for this
>                                     Internet-Draft is:
>                                     https://datatracker.ietf.org/doc/draft-ietf-tcpm-accurate-ecn/
>
>                                     There is also an htmlized version available at:
>                                     https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-accurate-ecn-24
>
>                                     A diff from the previous version is available at:
>                                     https://author-tools.ietf.org/iddiff?url2=draft-ietf-tcpm-accurate-ecn-24
>
>                                     Internet-Drafts are also available by rsync at
>                                     rsync.ietf.org::internet-drafts
> 
>
>                                     _______________________________________________
>                                     tcpm mailing list
>                                     [e-mail]
>                                     https://www.ietf.org/mailman/listinfo/tcpm
> 
>
>                               --
>                               ________________________________________________________________
>                               Bob Briscoe                               http://bobbriscoe.net/
> 
> 
> 
>
>             -- 
>             ________________________________________________________________
>             Bob Briscoe                               http://bobbriscoe.net/
> 
> 
> 
> -- 
> ________________________________________________________________
> Bob Briscoe                               http://bobbriscoe.net/
> 
>