Re: [tcpm] Comments on draft-ietf-tcpm-accurate-ecn

[Mirja Kühlewind <[e-mail]>]

Hi Yuchung,

please see below.

> Am 12.07.2018 um 11:41 schrieb Yuchung Cheng <[e-mail]>:
> 
> Yes packets with ACE options and (different) ACE-counter header would
> break GRO. There're many legacy h/w and s/w.

I’m not the expert here but my understanding is that is would not break GRO but could not make actual use or it if the ACE counter changed or the option is present. However, usually your will only have a small number of CE marks every couple of RTTs, and the counter only changes if you CE marks/the option is only present a few times per RTT. So the impact should be rather low. But this clear something to evaluate for the experiment.

> 
> Even without option, ACE header only allows reflecting up to 8 packets
> for receiver segmentation offload and ACK suppression?

Same as above, it can only up to 8 CE marked packets per ACK, however, CE marking rater are expected to be rather low. ACK suppression should probably not be used if the ACE counter has changes, however, usually the ACE counter stays stable for multiple RTTs and then ACK suppression is not a problem. However, not sure I understood you question here correctly…?

> 
> The appendix on ack loss causes ambiguity is good: but the pattern
> drops specifically the state-switch (CE<->noCE) ACK which only happens
> on delayed ACK. Is that pattern common? - or can we address most of
> that by delaying less ACKs?

I guess you have a 50% chance today to hit a delayed ACK. I guess delaying less (where you delay every second ACK today) would mean ACK very packet, and thus double ACK load on the network. I guess on today networks that actually in most cases not a problem, however, there might be specially cases where it is. However, that’s problem an independent question to evaluate.

> 
> I am all for making ECN more accurate for wide area beyond DCTCP, but
> am evaluating the pros/cons. What if we just
> 1. negotiate 'better' ECN via new SYN option

Not sure I understand your proposal correctly, but I assume you mean negotiate via SYN option and then just use the ACE counter in the header? If so, I don’t understand why you see the negotiation part in the TCP header as the problem?

> 2. mark all packets like ECN++

That would be nice but here you actually need AccECN because you want to have feedback for control packets as well. AccECN is providing this feedback. AccECN does not change the „use of ECN“; that’s what we have ECN++ for; both thing ideally would be deployed together however. Was that your questions?

Mirja


> 
> 
> 
> 
> 
> 
> On Thu, Jul 12, 2018 at 3:23 AM, Mirja Kühlewind
> <[e-mail]> wrote:
>> Hi Yuchung,
>> 
>> the question if you need this „more“ on accuracy really depends on the use case. If you use DCTCP as today, the ACE counter in the TCP is probably sufficient and you might not want to pay the additional overhead in your data center (that why the option is actually optional).
>> 
>> If you however, e.g., have very differently sized packets, then the byte counter in the option could give you a more accurate signal. Or if you are also interested in the ECT(1) counter, you need the option. Further the ACE counter also give your feedback on control packet and the option enables you to distinguish between CE-amrked payload and control packets, which can also become important when experimenting with making all packets ECN-capable.
>> 
>> Given accECN is a general feedback mechanism that in fact is designed to enable new future uses of the ECN signal, we wanted to keep all these option available while making is still as simple as possible and as flexible as possible.
>> 
>> Mirja
>> 
>> 
>> 
>>> Am 11.07.2018 um 14:35 schrieb Yuchung Cheng <[e-mail]>:
>>> 
>>> Hi Bob,
>>> 
>>> Neal and I evaluated the earlier draft. It is well-thought out but
>>> we're concerned about the options. Option is not mandatory but the
>>> lack of it also reduces accuracy. Option runs into space issues w/
>>> SACK and offload issues w/ TSO/GRO. They can be addressed for sure but
>>> aren't easy.
>>> 
>>> We're curious how much more "accuracy" it buys over current
>>> DCTCP-style ECN. Is there any study to show trade-offs of
>>> full-ACE-w-options vs ACE-wo-options vs current DCTCP-ECN?
>>> 
>>> 
>>> On Wed, Jul 11, 2018 at 11:00 AM, Bob Briscoe <[e-mail]> wrote:
>>>> Michael, tcpm list,
>>>> 
>>>> As well as addressing your points, as Mirja has already mentioned below, we
>>>> added a whole new appendix giving the rationale for the bits and codepoints
>>>> that AccECN has proposed to use on 1) the SYN and 2) SYN/ACK. A 3rd
>>>> subsection also identifies space for future evolution. It also points to
>>>> where rationale was already given in the body of the draft.
>>>> 
>>>> The appendix is in the draft submitted last week, available here:
>>>> https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-07#appendix-B
>>>> 
>>>> We'd be interested to hear whether this allays your concerns.
>>>> 
>>>> We have asked to present this in Montreal as well.
>>>> 
>>>> Cheers
>>>> 
>>>> 
>>>> Bob
>>>> 
>>>> 
>>>> On 02/07/18 16:54, Mirja Kühlewind wrote:
>>>>> 
>>>>> Hi Micheal,
>>>>> 
>>>>> I addressed a couple of your comments below.
>>>>> 
>>>>> For the other, bigger comments regarding extensibility, that I did not yet
>>>>> address below, we plan to add a new section to the appendix to explain
>>>>> extensibility options as previously discussed by mail. We will probably send
>>>>> a separate email on that part.
>>>>> 
>>>>> Mirja
>>>>> 
>>>>> 
>>>>>> Am 12.03.2018 um 01:59 schrieb Scharf, Michael (Nokia - DE/Stuttgart)
>>>>>> <[e-mail]>:
>>>>>> 
>>>>>> Hi Mirja,
>>>>>> 
>>>>>> Thanks a lot for the explanation. I won't follow-up on some of the
>>>>>> editorial suggestions.
>>>>>> 
>>>>>> Yet, I continue to believe that some formal wording in the document needs
>>>>>> to change, as explained below.
>>>>>> 
>>>>>> Thanks
>>>>>> 
>>>>>> Michael (with no hat on)
>>>>>> 
>>>>>> 
>>>>>>> -----Original Message-----
>>>>>>> From: Mirja Kühlewind [mailto:[e-mail]]
>>>>>>> Sent: Monday, March 05, 2018 1:54 PM
>>>>>>> To: Scharf, Michael (Nokia - DE/Stuttgart) <[e-mail]>
>>>>>>> Cc: [e-mail]; [e-mail]
>>>>>>> Subject: Re: Comments on draft-ietf-tcpm-accurate-ecn
>>>>>>> 
>>>>>>> Hi Micheal,
>>>>>>> 
>>>>>>> thanks for your feedback and sorry for my late reply.
>>>>>>> 
>>>>>>> Please see inline.
>>>>>>> 
>>>>>>>> Am 03.12.2017 um 20:17 schrieb Scharf, Michael (Nokia - DE/Stuttgart)
>>>>>>> 
>>>>>>> <[e-mail]>:
>>>>>>>> 
>>>>>>>> Hi all,
>>>>>>>> 
>>>>>>>> I have read draft-ietf-tcpm-accurate-ecn-05 (without the appendix). I
>>>>>>> 
>>>>>>> believe this document needs further work before moving forward.
>>>>>>>> 
>>>>>>>> Please find below my comments marked as [ms]. I have read the
>>>>>>> 
>>>>>>> document independent of the review from Gorry. I apologize if there is
>>>>>>> duplication.
>>>>>>>> 
>>>>>>>> Thanks
>>>>>>>> 
>>>>>>>> Michael (with no hat on)
>>>>>>>> 
>>>>>>>> 
>>>>>>>> ******************************
>>>>>>>> 
>>>>>>>> * Abstract:
>>>>>>>> 
>>>>>>>>  Recently, new TCP mechanisms like Congestion Exposure (ConEx) or
>>>>>>>> Data
>>>>>>> 
>>>>>>> Center TCP
>>>>>>>> 
>>>>>>>>  (DCTCP) need more accurate ECN feedback information whenever more
>>>>>>>>  than one marking is received in one RTT.
>>>>>>>> 
>>>>>>>> [ms] I don't think this statement is fully backed by RFC 8257. I
>>>>>>>> suggest to
>>>>>>> 
>>>>>>> remove this, or replace it by a more generic statement that more
>>>>>>> accurate
>>>>>>> information can be useful for several TCP extensions.
>>>>>>> 
>>>>>>> I disagree. Both ConEx and DCTCP need more accurate information. They do
>>>>>>> not need the mechanism that is specified in this draft, however, this is
>>>>>>> not
>>>>>>> what the sentences is saying.
>>>>>> 
>>>>>> In my understanding (as a non-native speaker), the use of the word "need"
>>>>>> is not correct here. DCTCP as specified in RFC 8257 can be implemented
>>>>>> without any such mechanism.
>>>>>> 
>>>>>> What would work for me is something of the form "... Data Center TCP
>>>>>> cannot get precise ECN feedback whenever more than one marking is received
>>>>>> in one RTT“.
>>>>> 
>>>>> This is not correct. DCTP need more than one feedback signal per RTT and
>>>>> therefore cannot use RFC3168; instead it implement it’s own feedback
>>>>> mechanism. However, to avoid confusion such that people could assume DCTP
>>>>> would not work without the accECN scheme as specified in this doc, I
>>>>> rephrased to:
>>>>> 
>>>>> "Recently, proposed
>>>>>      mechanisms like Congestion Exposure (ConEx <xref
>>>>> target="RFC7713"/>),
>>>>>      DCTCP <xref target="RFC8257"/> or L4S <xref
>>>>>      target="I-D.ietf-tsvwg-l4s-arch"/> need to know when more than one
>>>>>      marking is received in one RTT which is
>>>>>      information that cannot be provided by the feedback scheme as
>>>>> specified in
>>>>>      <xref target="RFC3168"/>."
>>>>> 
>>>>> 
>>>>>>>>  This document specifies an
>>>>>>>>  experimental scheme to provide more than one feedback signal per RTT
>>>>>>>>  in the TCP header.  Given TCP header space is scarce, it overloads
>>>>>>>>  the three existing ECN-related flags in the TCP header and provides
>>>>>>>>  additional information in a new TCP option.
>>>>>>>> 
>>>>>>>> [ms] This statement needs to be rewritten to correctly reflect what is
>>>>>>> 
>>>>>>> requested from IANA. My understanding is that this experimental document
>>>>>>> asks for allocation of a reserved TCP header flag. This needs to be
>>>>>>> called out
>>>>>>> prominently, IMHO. In addition, since this is not a standard, the
>>>>>>> suggested
>>>>>>> experimentation with the main TCP header must IMHO be explicitly
>>>>>>> mentioned. I also suggest to have later in a document a section that
>>>>>>> explicitly
>>>>>>> explains why it is appropriate to modify the main TCP header in an
>>>>>>> experiment.
>>>>>>> 
>>>>>>> I don’t know if any requirement that IANA assignment need to be called
>>>>>>> out
>>>>>>> in the abstract but we can do that. However, I believe the question if
>>>>>>> this
>>>>>>> document should or should not assign the bit is still not completely
>>>>>>> solved, or
>>>>>>> is it?
>>>>>> 
>>>>>> I believe this question will have to be reviewed during WGLC and, more
>>>>>> importantly, IETF last call. For the moment, my concern is that the document
>>>>>> correctly describes the IANA allocation.
>>>>>> 
>>>>>> I would like to see here a statement such as : "Given TCP header space is
>>>>>> scarce, this specification allocates a reserved header bit and overloads the
>>>>>> two ECN flags in the TCP header ...“.
>>>>> 
>>>>> A bit lengthy but now:
>>>>> 
>>>>> "Given TCP header space is
>>>>>      scarce, it allocates a reserved header bit, that was previously
>>>>> used for
>>>>>      ECN-Nonce which was recently declared historic, and overloads the
>>>>>      two existing ECN flags in the TCP header. Further, additional
>>>>>      information can be provided in a new TCP option that however is not
>>>>> used
>>>>>      on the TCP SYN."
>>>>> 
>>>>>>>> * 1.  Introduction
>>>>>>>> 
>>>>>>>>  Recently, proposed mechanisms like Congestion Exposure (ConEx
>>>>>>>>  [RFC7713]), DCTCP [RFC8257] or L4S [I-D.ietf-tsvwg-l4s-arch] need
>>>>>>>>  more accurate ECN feedback information whenever more than one
>>>>>>> 
>>>>>>> marking
>>>>>>>> 
>>>>>>>>  is received in one RTT.
>>>>>>>> 
>>>>>>>> [ms] At least for RFC 8257 seems to be implementable withoit this.
>>>>>>>> Instead
>>>>>>> 
>>>>>>> of stating a "need", it would IMHO make more sense to discuss the
>>>>>>> benefits
>>>>>>> of the suggested mechanism in this document of its own, independent of
>>>>>>> other proposals. To me, this document should be independent of other
>>>>>>> documents and specifically other experiments. We have to think about
>>>>>>> cases
>>>>>>> where not all experiments are successful. Then independent documents
>>>>>>> will
>>>>>>> be more future-proof in future.
>>>>>>> 
>>>>>>> This is a naming collision… The sentence was meant to say that these
>>>>>>> mechanisms new more accurate ECN feedback than provided today by
>>>>>>> RFC3168 but it was not meant to say that these mechanism have to use the
>>>>>>> scheme as specified in this document.
>>>>>>> 
>>>>>>> I added the following part sentence:
>>>>>>> 
>>>>>>> „Recently, proposed mechanisms like Congestion Exposure (ConEx
>>>>>>> [RFC7713]), DCTCP [RFC8257] or L4S [I-D.ietf-tsvwg-l4s-arch] need more
>>>>>>> accurate ECN feedback information than provided by the feedback scheme
>>>>>>> as specified in [RFC3168] whenever more than one marking is received in
>>>>>>> one
>>>>>>> RTT. This document specifies an alternative feedback scheme that
>>>>>>> provides
>>>>>>> more accurate information and could be used by these new TCP
>>>>>>> extensions.“
>>>>>>> 
>>>>>>> Does this help?
>>>>>> 
>>>>>> See my proposal for the abstract. I continue to disagree with the term
>>>>>> "need" but I think this can be sorted out by another term.
>>>>>> 
>>>>>>>>  If AccECN progresses from experimental to the standards
>>>>>>>>  track, it is intended to be a complete replacement for classic TCP/
>>>>>>>>  ECN feedback, not a fork in the design of TCP.
>>>>>>>> 
>>>>>>>> [ms] This sentence should be removed, as this is speculation.
>>>>>>> 
>>>>>>> Why? It states an intent… and that’s the intent that we have.
>>>>>>> 
>>>>>>>>  Until the AccECN experiment succeeds, [RFC3168] will remain as the
>>>>>>>>  standards track specification for adding ECN to TCP.
>>>>>>>> 
>>>>>>>> [ms] This sentence should be removed (or reworded)
>>>>>>> 
>>>>>>> Why? Does it help to add an only here:
>>>>>>> 
>>>>>>> "Until the AccECN experiment succeeds, [RFC3168] will remain as the only
>>>>>>> standards track specification for adding ECN to TCP.“
>>>>>> 
>>>>>> This wording is better.
>>>>>> 
>>>>>>>>  AccECN feedback overloads flags and fields in the main TCP header
>>>>>>>>  with new definitions, so both ends have to support the new wire
>>>>>>>>  protocol before it can be used.
>>>>>>>> 
>>>>>>>> [ms] In my reading this experimental document asks for *new* allocation
>>>>>>> 
>>>>>>> of a reserved TCP header flag.
>>>>>>> 
>>>>>>> Is this better?
>>>>>>> 
>>>>>>> "AccECN feedback overloads the two existing ECN flags as well as the
>>>>>>>     currently reserved and previously called NS flag in the main TCP
>>>>>>> header
>>>>>>>     with new definitions, so both ends have to support the new wire
>>>>>>> protocol
>>>>>>>     before it can be used.“
>>>>>>> 
>>>>>>> I understand that you are not happy with the word „overload“ here but
>>>>>>> the
>>>>>>> point of this sentence really is that the flags can/could be used
>>>>>>> differently
>>>>>>> and therefore we need a new negotiation before we can use them.
>>>>>> 
>>>>>> For me the following would work: "AccECN feedback overloads the two
>>>>>> existing ECN flags and
>>>>>> allocates the currently reserved and previously called NS flag in the
>>>>>> main TCP header.
>>>>>> Given the new definitions, both ends have to support the new wire
>>>>>> protocol
>>>>>> before it can be used."
>>>>>> 
>>>>>> I believe the wording has to be crystal clear on the reservation of bit 7
>>>>>> when it is discussed the first time in the text. In follow-up sections,
>>>>>> maybe shorter terms could be used.
>>>>> 
>>>>> Okay, now:
>>>>> 
>>>>> "AccECN feedback overloads the two existing ECN flags and
>>>>>    allocates the currently reserved and previously called NS flag in the
>>>>>    TCP header, to be used as one field indicating the number of
>>>>> congestion
>>>>>    experienced marked packets. Given the new definitions of these three
>>>>> bits,
>>>>>    both ends     have to support the new wire protocol before it can be
>>>>> used.
>>>>>    Therefore during the TCP handshake the two ends use these three bit
>>>>> in
>>>>>    the TCP header to negotiate the most advanced feedback protocol
>>>>>    that they can both support in a backward compatible way to
>>>>>    <xref target="RFC3168"/>."
>>>>>>> 
>>>>>>> If you prefer, we can also remove the NS flag in this list, as ECN Nonce
>>>>>>> was
>>>>>>> anyway never deployed.
>>>>>>> 
>>>>>>>>  For that we refer to [RFC3168] or any RFC that
>>>>>>>>  specifies a different response to TCP ECN feedback, for example:
>>>>>>>>  [RFC8257]; or the ECN experiments referred to in
>>>>>>>>  [I-D.ietf-tsvwg-ecn-experimentation], namely: a TCP-based Low
>>>>>>>> Latency
>>>>>>>>  Low Loss Scalable (L4S) congestion control
>>>>>>>> [I-D.ietf-tsvwg-l4s-arch];
>>>>>>>>  ECN-capable TCP control packets [I-D.ietf-tcpm-generalized-ecn], or
>>>>>>>>  Alternative Backoff with ECN (ABE)
>>>>>>>>  [I-D.ietf-tcpm-alternativebackoff-ecn].
>>>>>>>> 
>>>>>>>> [ms] At least ABE seems orthogonal. Anyway, I think this paragraph can
>>>>>>>> just
>>>>>>> 
>>>>>>> be deleted. If other experiments need more accurate feedback, it is up
>>>>>>> to
>>>>>>> them to explain how they would use this mechanism. This document should
>>>>>>> focus on how to signal the feedback, not how to use that.
>>>>>>> 
>>>>>>> Yes, that is what the paragraph says. Isn’t it better to be explicit
>>>>>>> about this?
>>>>>>> 
>>>>>>>>  It is likely (but not required) that the AccECN protocol will be
>>>>>>>>  implemented along with the following experimental additions to the
>>>>>>>>  TCP-ECN protocol: ECN-capable TCP control packets and
>>>>>>>> retransmissions
>>>>>>>>  [I-D.ietf-tcpm-generalized-ecn], which includes the ECN-capable SYN/
>>>>>>>>  ACK experiment [RFC5562]; and testing receiver non-compliance
>>>>>>>>  [I-D.moncaster-tcpm-rcv-cheat].
>>>>>>>> 
>>>>>>>> [ms] I am a big fan of simple, standalone documents. In my view, the
>>>>>>>> TCPM
>>>>>>> 
>>>>>>> working group should publish draft-ietf-tcpm-accurate-ecn and
>>>>>>> draft-ietf-
>>>>>>> tcpm-generalized-ecn independent documents, which probably implies that
>>>>>>> draft-ietf-tcpm-generalized-ecn does not use AccECN. If experimentation
>>>>>>> with ECT in SYN requires a combination, this could be done in a new,
>>>>>>> third
>>>>>>> document. Apart from having simpler focused documents, this could
>>>>>>> significantly help later with moving forward documents to standards
>>>>>>> track.
>>>>>>> 
>>>>>>> I disagree, however, this is a discussion to have on draft-ietf-tcpm-
>>>>>>> generalized-ecn. I don’t see a problem in  providing a reference here
>>>>>>> that
>>>>>>> says „it is likely…“ and nothing more.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> * 1.1.  Document Roadmap
>>>>>>>> 
>>>>>>>> [ms] A macroscopic comment is that this document has a lot of
>>>>>>>> introduction
>>>>>>> 
>>>>>>> and tutorial text with lot's of redundancy towards other documents. I
>>>>>>> think
>>>>>>> the document can be made much easier to read by shorten it. In many
>>>>>>> cases
>>>>>>> this is just an editorial change as there is redundancy. As one such
>>>>>>> example,
>>>>>>> just remove this section.
>>>>>>> 
>>>>>>> I guess this a matter of taste. As an AD, I’m a big fan of short and
>>>>>>> concise
>>>>>>> documents, however, some redundancy can also help understanding,
>>>>>>> especially if you explain things multiple times but with a different
>>>>>>> level of
>>>>>>> detail. I personally would not need the roadmap but I know many people
>>>>>>> who find these things helpful and to be honest I don’t see how removing
>>>>>>> this
>>>>>>> part makes the doc any better. If you don’t want it, don’t read it.
>>>>>>> 
>>>>>>>> 
>>>>>>>> * 1.2.  Goals
>>>>>>>> 
>>>>>>>> [ms] I think this section can also just be removed.
>>>>>>> 
>>>>>>> I have to say I also don’t see the point of removing this part. Given
>>>>>>> we’ve
>>>>>>> done the work on requirements, I think we should also link to this doc
>>>>>>> somewhere.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> * 1.3.  Experiment Goals
>>>>>>>> 
>>>>>>>>  TCP is critical to the robust functioning of the Internet, therefore
>>>>>>>>  any proposed modifications to TCP need to be thoroughly tested. The
>>>>>>>>  present specification describes an experimental protocol that adds
>>>>>>>>  more accurate ECN feedback to the TCP protocol.  The intention is to
>>>>>>>>  specify the protocol sufficiently so that more than one
>>>>>>>>  implementation can be built in order to test its function,
>>>>>>>> robustness
>>>>>>>>  and interoperability (with itself and with previous version of ECN
>>>>>>>>  and TCP).
>>>>>>>> 
>>>>>>>> [ms] I think all what is written in this paragraph is obvious, no?
>>>>>>>> Can't we just
>>>>>>> 
>>>>>>> delete this?
>>>>>>> 
>>>>>>> Sure, however, I don’t think it hurts to spell it out. For me both is
>>>>>>> fine, keep it
>>>>>>> or remove it.
>>>>>>> 
>>>>>>>>  The experimental protocol will be considered successful if it is
>>>>>>>>  deployed and if it satisfies the requirements of [RFC7560] in the
>>>>>>>>  consensus opinion of the IETF tcpm working group.  In short, this
>>>>>>>>  requires that it improves the accuracy and timeliness of TCP's ECN
>>>>>>>>  feedback, as claimed in Section 5, while striking a balance between
>>>>>>>>  the conflicting requirements of resilience, integrity and
>>>>>>>>  minimisation of overhead.  It also requires that it is not unduly
>>>>>>>>  complex, and that it is compatible with prevalent equipment
>>>>>>>>  behaviours in the current Internet (e.g. hardware offloading and
>>>>>>>>  middleboxes), whether or not they comply with standards.
>>>>>>>> 
>>>>>>>>  Testing will mostly focus on fall-back strategies in case of
>>>>>>>>  middlebox interference.  Current recommended strategies are
>>>>>>>> specified
>>>>>>>>  in Sections 3.1.2, 3.2.3, 3.2.4 and 3.2.7.  The effectiveness of
>>>>>>>>  these strategies depends on the actual deployment situation of
>>>>>>>>  middleboxes.  Therefore experimental verification to confirm large-
>>>>>>>>  scale path traversal in the Internet is needed before finalizing
>>>>>>>> this
>>>>>>>>  specification on the Standards Track.
>>>>>>>> 
>>>>>>>> [ms] These two paragraphs must be entirely rewritten. As I have
>>>>>>> 
>>>>>>> mentioned before, I don't think an RFC should speculate about TCPM and
>>>>>>> its
>>>>>>> consensus opinion. I would suggest a wording along the lines of:
>>>>>>>> 
>>>>>>>> <ms>
>>>>>>>>  The experimental protocol will be considered successful if
>>>>>>>>  testing confirms that the proposed mechanism can be deployed at
>>>>>>>> large
>>>>>>> 
>>>>>>> scale.
>>>>>>>> 
>>>>>>>>  Testing will mostly focus on fall-back strategies in case of
>>>>>>>>  middlebox interference.  Current recommended strategies are
>>>>>>>> specified
>>>>>>>>  in Sections 3.1.2, 3.2.3, 3.2.4 and 3.2.7.  The effectiveness of
>>>>>>>>  these strategies depends on the actual deployment situation of
>>>>>>>>  middleboxes.  Therefore experimental verification to confirm large-
>>>>>>>>  scale path traversal in the Internet is needed, e.g., by support in
>>>>>>>>  major TCP stacks.
>>>>>>>> </ms>
>>>>>>>> 
>>>>>>> I don’t understand your point here. I don’t think that the paraphrase
>>>>>>> speculates about the consensus of tcpm, in contrast it say tcpm has to
>>>>>>> decided if the requirements previously specified by tcpm are
>>>>>>> sufficiently
>>>>>>> fulfilled. I don’t see a reason to not mention the requirement draft as
>>>>>>> this
>>>>>>> draft as tcpm consensus and was written for this purpose.
>>>>>> 
>>>>>> My suggested wording uses the expression "can be deployed at large scale"
>>>>>> and I believe this is relevant.
>>>>>> 
>>>>>> The document already describes in Section 5 how the protocol satisfies
>>>>>> the agreed requirements for a more accurate ECN feedback protocol [RFC7560].
>>>>>> So, if the TCPM working group publishes this document with the content of
>>>>>> Section 5, I believe the TCPM working group already has reached consensus
>>>>>> that the protocol meets requirements. In addition, it is possible that new
>>>>>> requirements would be identified in future, e.g., as an outcome of the
>>>>>> experiment, and that would obviously have to be considered by TCPM. In that
>>>>>> case, for the success of the experiment not only RFC 7560 would matter, but
>>>>>> also further requirements. My proposed wording does not have all these
>>>>>> problems.
>>>>>> 
>>>>>> In a nutshell, I continue to believe that this section has to change.
>>>>> 
>>>>> 
>>>>> Okay, used your proposed wording. You have a point about the requirement
>>>>> and I mis-read you proposal earlier as „has to be deployed large-scale“.
>>>>> 
>>>>>>>> * 1.5.  Recap of Existing ECN feedback in IP/TCP
>>>>>>>> 
>>>>>>>> [ms] This section could probably be shortened as well.
>>>>>>>> 
>>>>>>>>  The last bit in byte 13 of the TCP header was defined as the Nonce
>>>>>>>>  Sum (NS) for the ECN Nonce [RFC3540].  RFC 3540 was never deployed
>>>>>>>> so
>>>>>>>>  it is being reclassified as historic, making this TCP flag available
>>>>>>>>  for use by the AccECN experiment instead.
>>>>>>>> 
>>>>>>>> [ms] This wording, as well as Figure 1, needs to take into account the
>>>>>>>> IANA
>>>>>>> 
>>>>>>> status when draft-ietf-tsvwg-ecn-experimentation is published.
>>>>>>> 
>>>>>>> Is does. However, I can explicitly say that is has be re-clssified as
>>>>>>> reserved.
>>>>>>> 
>>>>>>> "RFC 3540 was never deployed so it is being reclassified as historic
>>>>>>> [I-D.ietf-
>>>>>>> tsvwg-ecn-experimentation] and the respective flag has been marked as
>>>>>>> „reserved“ in the IANA TCP Header Flags registry, making this TCP flag
>>>>>>> available for use by the AccECN experiment instead.“
>>>>>>> 
>>>>>>> Better?
>>>>>>> 
>>>>>>>> In my understanding, this experimental document asks for new assignment
>>>>>>> 
>>>>>>> of a reserved TCP header flag.
>>>>>>> 
>>>>>>> As I said I’m not sure if we have fully concluded this discussion yet.
>>>>>>> However,
>>>>>>> what we really would want to is mention somewhere that this experiment
>>>>>>> with this flags is running. I guess there are three options:
>>>>>>> 1) keep it in the registry as reserved and conserve the knowledge in
>>>>>>> tcpm
>>>>>>> that this experiment is running and no other experimental RFC such use
>>>>>>> this
>>>>>>> flags as long as this experiment is running.
>>>>>>> 2) Keep is marked as reserved but add a note about this experiment in
>>>>>>> the
>>>>>>> IANA registry
>>>>>>> 3) Or assign it right away with IESG approval. I guess in this case tcpm
>>>>>>> could
>>>>>>> also consider to change the registration policy to „IETF Review“.
>>>>>> 
>>>>>> The current registration policy for the TCP header flags is "standards
>>>>>> action". I understand that the IESG could approve exceptions. But given the
>>>>>> policy, I believe the document has to be very precise on the request
>>>>>> regarding bit 7.
>>>>> 
>>>>> Okay, it now says:
>>>>> 
>>>>> "[TO BE REMOVED: IANA is requested to update the existing entry in the
>>>>> Transmission Control Protocol (TCP) Header Flags registration
>>>>> (https://www.iana.org/assignments/tcp-header-flags/tcp-header-flags.xhtml#tcp-header-flags-1)
>>>>> for Bit 7 to "AE (Accurate ECN), previously used by Historic as NS (Nonce
>>>>> Sum) [RFC3540, RFC8311]" and change the reference to this RFC-to-be instead
>>>>> of RFC8311.]“
>>>>> 
>>>>> I guess we could also ask IANA to add an additional comment column instead
>>>>> (but not sure if we then have to update RFC3168, which I think we really
>>>>> don’t want. Should be fine now.
>>>>> 
>>>>>>>> * 2.  AccECN Protocol Overview and Rationale
>>>>>>>> 
>>>>>>>>  o  an essential part that re-uses ECN TCP header bits to feed back
>>>>>>>>     the number of arriving CE marked packets.  This provides more
>>>>>>>>     accuracy than classic ECN feedback, but limited resilience
>>>>>>>> against
>>>>>>>>     ACK loss;
>>>>>>>> 
>>>>>>>> [ms] The word "re-use" is IMHO not correct.
>>>>>>> 
>>>>>>> I think this is nit picking. Using a different phrasing here makes the
>>>>>>> sentence
>>>>>>> unnecessary complicated. We don’t try to some how get a round the fact
>>>>>>> that we need to handle the flag registration correctly. However, here
>>>>>>> the
>>>>>>> point really is to explain how the protocol word. The main point of
>>>>>>> using the
>>>>>>> work „re-use“ here is really that we say that these flags are or have
>>>>>>> been
>>>>>>> used different by other TCP extension (and we therefore need a proper
>>>>>>> negotiation scheme).
>>>>>> 
>>>>>> If the allocation of a reserved flag is correctly explained in the
>>>>>> abstract and introduction, I think these sentences can use a bit relaxed
>>>>>> terminology.
>>>>>> 
>>>>>>>>  The two part design was necessary, given limitations on the space
>>>>>>>>  available for TCP options and given the possibility that certain
>>>>>>>>  incorrectly designed middleboxes prevent TCP using any new options
>>