Re: [tcpm] Comments on draft-ietf-tcpm-accurate-ecn

[Yuchung Cheng <[e-mail]>]

On Fri, Jul 13, 2018 at 11:53 AM, Mirja Kühlewind
<[e-mail]> wrote:
> Hi Yucheng,
>
> please see below.
>
>> Am 13.07.2018 um 14:38 schrieb Yuchung Cheng <[e-mail]>:
>>
>> hi --
>>
>> I agree:
>> 1. delayed/streched ACK aren't going away (in fact will be more common)
>> 2. GRO isn't and should not be a show-stopper
>> 3. SYN option is running tight
>> 4. HW opt comes after SW
>>
>> I worry:
>> 1. GRO is a unavoidable issue in deployment (let's not produce an
>> undeployable RFC). the ACE counter won't work as GRO can pack up to
>> 64KB/MTU =~ 45 pkts under heavy congestion.
>>
>> 2. ACE's benefit over DCTCP is when delayed ACKs and ack losses
>> happen. Now delayed ACKs happen frequently when the messages are
>> small, which means they don't solicit too many ACKs to be dropped in
>> the reverse path. Also that under heavy downstream loss or reordering,
>>
>> 3 SACK options become dominant to have ACE option.
>>
>> 4. If SYN option space is a concern, ACE header exchange is ok. But I
>> suspect the latter has more middlebox issue based on my TFO deployment
>> experience.
>>
>>
>> So with all that said, is it possible to have a "minimal"-ACE that
>> 1. Does ACE handshake as proposed
>> 2. Mark ECT on all (or at least SYN & data & rtx) packets like ECN++
>
> This is ECN++ and independent of AccECN. We on purposed have split the feedback and usage of ECN (which is not the case in RFC3168) to be able to change things in future independently
sure - IMO marking all packets just improve the accuracy significantly
vs the counters and options. For example, ECN on SYN is very useful on
incast / loaded link.

>
>> 3. Leave ACE-count and ACE option optional (i.e. MAY)
>
> I don’t understand this. If both is optional, you don’t have any feedback. Or what do you mean by „leave ACE-count optional“?
use-case: We can negotiate DCTCP-style ECN for the internet.

Then interested parties can progressively experiment on more accurate
"options" (!= TCP-option)

>
> Mirja
>
>
>
>>
>> as a "fairly safe" to deployment compromise. And I am happy to draft a
>> kernel patch for that :-)
>>
>>
>>
>> On Fri, Jul 13, 2018 at 1:18 AM, Bob Briscoe <[e-mail]> wrote:
>>> Yuchung,
>>>
>>> On 12/07/18 17:03, Mirja Kühlewind wrote:
>>>>
>>>> Hi Yuchung,
>>>>
>>>> please see below.
>>>>
>>>>> Am 12.07.2018 um 11:41 schrieb Yuchung Cheng <[e-mail]>:
>>>>>
>>>>> Yes packets with ACE options and (different) ACE-counter header would
>>>>> break GRO. There're many legacy h/w and s/w.
>>>>
>>>> I’m not the expert here but my understanding is that is would not break
>>>> GRO but could not make actual use or it if the ACE counter changed or the
>>>> option is present.
>>>
>>> [BB] Yuchung's criticism applies to any scheme that feeds back a continually
>>> changing signal like ECN. For instance, like AccECN feedback, DCTCP ECN
>>> feedback continually changes the TCP header flags, and I believe DCTCP is
>>> widely used in DCs.
>>>
>>> I believe, to make GRO support ECN feedback (whether DCTCP or AccECN), it
>>> would be necessary for GRO to know which bits to mask when determining
>>> whether a packet is mergeable with others. I believe GRO already collects
>>> header fields separately for the TCP logic to be able to work on them, but
>>> I'm also not an expert.
>>>>
>>>> However, usually your will only have a small number of CE marks every
>>>> couple of RTTs, and the counter only changes if you CE marks/the option is
>>>> only present a few times per RTT. So the impact should be rather low. But
>>>> this clear something to evaluate for the experiment.
>>>
>>> [BB] With DCTCP as currently designed, you tend to get runs of 100% CE if
>>> the load of short flows is very small. In that case, DCTCP feedback will
>>> change the header flags less often than AccECN. However, with more short
>>> flows, the feedback is more on-off. Then the flags change more often with
>>> DCTCP than with AccECN feedback.
>>>
>>> In general, hardware optimization is not going to optimize a protocol that
>>> didn't exist when the hardware was designed. It would be ideal to design a
>>> new protocol that takes advantage of existing hardware optimization.
>>> However, as long as an experimental protocol still works with existing
>>> hardware, I think it's reasonable to assume that hardware optimization will
>>> only arrive once a protocol has become well-established.
>>>
>>>>
>>>>> Even without option, ACE header only allows reflecting up to 8 packets
>>>>> for receiver segmentation offload and ACK suppression?
>>>>
>>>> Same as above, it can only up to 8 CE marked packets per ACK, however, CE
>>>> marking rater are expected to be rather low. ACK suppression should probably
>>>> not be used if the ACE counter has changes, however, usually the ACE counter
>>>> stays stable for multiple RTTs and then ACK suppression is not a problem.
>>>> However, not sure I understood you question here correctly…?
>>>>
>>>>> The appendix on ack loss causes ambiguity is good: but the pattern
>>>>> drops specifically the state-switch (CE<->noCE) ACK which only happens
>>>>> on delayed ACK. Is that pattern common? - or can we address most of
>>>>> that by delaying less ACKs?
>>>>
>>>> I guess you have a 50% chance today to hit a delayed ACK. I guess delaying
>>>> less (where you delay every second ACK today) would mean ACK every packet,
>>>> and thus double ACK load on the network. I guess on today networks that
>>>> actually in most cases not a problem, however, there might be specially
>>>> cases where it is. However, that’s problem an independent question to
>>>> evaluate.
>>>
>>> [BB] If there were no delayed ACKs, the problem with DCTCP feedback would
>>> largely disappear. But delayed ACKs are not going away. Which is why we
>>> proposed replacing DCTCP feedback with AccECN feedback.
>>>>
>>>>
>>>>> I am all for making ECN more accurate for wide area beyond DCTCP, but
>>>>> am evaluating the pros/cons. What if we just
>>>>> 1. negotiate 'better' ECN via new SYN option
>>>>
>>>> Not sure I understand your proposal correctly, but I assume you mean
>>>> negotiate via SYN option and then just use the ACE counter in the header? If
>>>> so, I don’t understand why you see the negotiation part in the TCP header as
>>>> the problem?
>>>
>>> [BB] @Yuchung, as Mirja says, your point #1 seems to be proposing a solution
>>> without a problem. In fact an option on the SYN would create a new problem
>>> 'cos of the severe shortage of space for SYN options (for this reason, the
>>> AccECN TCP option was designed not to be needed on the SYN).
>>>>
>>>>
>>>>> 2. mark all packets like ECN++
>>>>
>>>> That would be nice but here you actually need AccECN because you want to
>>>> have feedback for control packets as well. AccECN is providing this
>>>> feedback. AccECN does not change the „use of ECN“; that’s what we have ECN++
>>>> for; both thing ideally would be deployed together however. Was that your
>>>> questions?
>>>
>>> Cheers
>>>
>>>
>>> Bob
>>>
>>>> Mirja
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jul 12, 2018 at 3:23 AM, Mirja Kühlewind
>>>>> <[e-mail]> wrote:
>>>>>>
>>>>>> Hi Yuchung,
>>>>>>
>>>>>> the question if you need this „more“ on accuracy really depends on the
>>>>>> use case. If you use DCTCP as today, the ACE counter in the TCP is probably
>>>>>> sufficient and you might not want to pay the additional overhead in your
>>>>>> data center (that why the option is actually optional).
>>>>>>
>>>>>> If you however, e.g., have very differently sized packets, then the byte
>>>>>> counter in the option could give you a more accurate signal. Or if you are
>>>>>> also interested in the ECT(1) counter, you need the option. Further the ACE
>>>>>> counter also give your feedback on control packet and the option enables you
>>>>>> to distinguish between CE-amrked payload and control packets, which can also
>>>>>> become important when experimenting with making all packets ECN-capable.
>>>>>>
>>>>>> Given accECN is a general feedback mechanism that in fact is designed to
>>>>>> enable new future uses of the ECN signal, we wanted to keep all these option
>>>>>> available while making is still as simple as possible and as flexible as
>>>>>> possible.
>>>>>>
>>>>>> Mirja
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Am 11.07.2018 um 14:35 schrieb Yuchung Cheng <[e-mail]>:
>>>>>>>
>>>>>>> Hi Bob,
>>>>>>>
>>>>>>> Neal and I evaluated the earlier draft. It is well-thought out but
>>>>>>> we're concerned about the options. Option is not mandatory but the
>>>>>>> lack of it also reduces accuracy. Option runs into space issues w/
>>>>>>> SACK and offload issues w/ TSO/GRO. They can be addressed for sure but
>>>>>>> aren't easy.
>>>>>>>
>>>>>>> We're curious how much more "accuracy" it buys over current
>>>>>>> DCTCP-style ECN. Is there any study to show trade-offs of
>>>>>>> full-ACE-w-options vs ACE-wo-options vs current DCTCP-ECN?
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jul 11, 2018 at 11:00 AM, Bob Briscoe <[e-mail]>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Michael, tcpm list,
>>>>>>>>
>>>>>>>> As well as addressing your points, as Mirja has already mentioned
>>>>>>>> below, we
>>>>>>>> added a whole new appendix giving the rationale for the bits and
>>>>>>>> codepoints
>>>>>>>> that AccECN has proposed to use on 1) the SYN and 2) SYN/ACK. A 3rd
>>>>>>>> subsection also identifies space for future evolution. It also points
>>>>>>>> to
>>>>>>>> where rationale was already given in the body of the draft.
>>>>>>>>
>>>>>>>> The appendix is in the draft submitted last week, available here:
>>>>>>>> https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-07#appendix-B
>>>>>>>>
>>>>>>>> We'd be interested to hear whether this allays your concerns.
>>>>>>>>
>>>>>>>> We have asked to present this in Montreal as well.
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>>
>>>>>>>>
>>>>>>>> Bob
>>>>>>>>
>>>>>>>>
>>>>>>>> On 02/07/18 16:54, Mirja Kühlewind wrote:
>>>>>>>>>
>>>>>>>>> Hi Micheal,
>>>>>>>>>
>>>>>>>>> I addressed a couple of your comments below.
>>>>>>>>>
>>>>>>>>> For the other, bigger comments regarding extensibility, that I did
>>>>>>>>> not yet
>>>>>>>>> address below, we plan to add a new section to the appendix to
>>>>>>>>> explain
>>>>>>>>> extensibility options as previously discussed by mail. We will
>>>>>>>>> probably send
>>>>>>>>> a separate email on that part.
>>>>>>>>>
>>>>>>>>> Mirja
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Am 12.03.2018 um 01:59 schrieb Scharf, Michael (Nokia -
>>>>>>>>>> DE/Stuttgart)
>>>>>>>>>> <[e-mail]>:
>>>>>>>>>>
>>>>>>>>>> Hi Mirja,
>>>>>>>>>>
>>>>>>>>>> Thanks a lot for the explanation. I won't follow-up on some of the
>>>>>>>>>> editorial suggestions.
>>>>>>>>>>
>>>>>>>>>> Yet, I continue to believe that some formal wording in the document
>>>>>>>>>> needs
>>>>>>>>>> to change, as explained below.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>> Michael (with no hat on)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>> From: Mirja Kühlewind [mailto:[e-mail]]
>>>>>>>>>>> Sent: Monday, March 05, 2018 1:54 PM
>>>>>>>>>>> To: Scharf, Michael (Nokia - DE/Stuttgart)
>>>>>>>>>>> <[e-mail]>
>>>>>>>>>>> Cc: [e-mail]; [e-mail]
>>>>>>>>>>> Subject: Re: Comments on draft-ietf-tcpm-accurate-ecn
>>>>>>>>>>>
>>>>>>>>>>> Hi Micheal,
>>>>>>>>>>>
>>>>>>>>>>> thanks for your feedback and sorry for my late reply.
>>>>>>>>>>>
>>>>>>>>>>> Please see inline.
>>>>>>>>>>>
>>>>>>>>>>>> Am 03.12.2017 um 20:17 schrieb Scharf, Michael (Nokia -
>>>>>>>>>>>> DE/Stuttgart)
>>>>>>>>>>>
>>>>>>>>>>> <[e-mail]>:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>
>>>>>>>>>>>> I have read draft-ietf-tcpm-accurate-ecn-05 (without the
>>>>>>>>>>>> appendix). I
>>>>>>>>>>>
>>>>>>>>>>> believe this document needs further work before moving forward.
>>>>>>>>>>>>
>>>>>>>>>>>> Please find below my comments marked as [ms]. I have read the
>>>>>>>>>>>
>>>>>>>>>>> document independent of the review from Gorry. I apologize if there
>>>>>>>>>>> is
>>>>>>>>>>> duplication.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>>
>>>>>>>>>>>> Michael (with no hat on)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ******************************
>>>>>>>>>>>>
>>>>>>>>>>>> * Abstract:
>>>>>>>>>>>>
>>>>>>>>>>>>  Recently, new TCP mechanisms like Congestion Exposure (ConEx) or
>>>>>>>>>>>> Data
>>>>>>>>>>>
>>>>>>>>>>> Center TCP
>>>>>>>>>>>>
>>>>>>>>>>>>  (DCTCP) need more accurate ECN feedback information whenever
>>>>>>>>>>>> more
>>>>>>>>>>>>  than one marking is received in one RTT.
>>>>>>>>>>>>
>>>>>>>>>>>> [ms] I don't think this statement is fully backed by RFC 8257. I
>>>>>>>>>>>> suggest to
>>>>>>>>>>>
>>>>>>>>>>> remove this, or replace it by a more generic statement that more
>>>>>>>>>>> accurate
>>>>>>>>>>> information can be useful for several TCP extensions.
>>>>>>>>>>>
>>>>>>>>>>> I disagree. Both ConEx and DCTCP need more accurate information.
>>>>>>>>>>> They do
>>>>>>>>>>> not need the mechanism that is specified in this draft, however,
>>>>>>>>>>> this is
>>>>>>>>>>> not
>>>>>>>>>>> what the sentences is saying.
>>>>>>>>>>
>>>>>>>>>> In my understanding (as a non-native speaker), the use of the word
>>>>>>>>>> "need"
>>>>>>>>>> is not correct here. DCTCP as specified in RFC 8257 can be
>>>>>>>>>> implemented
>>>>>>>>>> without any such mechanism.
>>>>>>>>>>
>>>>>>>>>> What would work for me is something of the form "... Data Center TCP
>>>>>>>>>> cannot get precise ECN feedback whenever more than one marking is
>>>>>>>>>> received
>>>>>>>>>> in one RTT“.
>>>>>>>>>
>>>>>>>>> This is not correct. DCTP need more than one feedback signal per RTT
>>>>>>>>> and
>>>>>>>>> therefore cannot use RFC3168; instead it implement it’s own feedback
>>>>>>>>> mechanism. However, to avoid confusion such that people could assume
>>>>>>>>> DCTP
>>>>>>>>> would not work without the accECN scheme as specified in this doc, I
>>>>>>>>> rephrased to:
>>>>>>>>>
>>>>>>>>> "Recently, proposed
>>>>>>>>>      mechanisms like Congestion Exposure (ConEx <xref
>>>>>>>>> target="RFC7713"/>),
>>>>>>>>>      DCTCP <xref target="RFC8257"/> or L4S <xref
>>>>>>>>>      target="I-D.ietf-tsvwg-l4s-arch"/> need to know when more than
>>>>>>>>> one
>>>>>>>>>      marking is received in one RTT which is
>>>>>>>>>      information that cannot be provided by the feedback scheme as
>>>>>>>>> specified in
>>>>>>>>>      <xref target="RFC3168"/>."
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>  This document specifies an
>>>>>>>>>>>>  experimental scheme to provide more than one feedback signal per
>>>>>>>>>>>> RTT
>>>>>>>>>>>>  in the TCP header.  Given TCP header space is scarce, it
>>>>>>>>>>>> overloads
>>>>>>>>>>>>  the three existing ECN-related flags in the TCP header and
>>>>>>>>>>>> provides
>>>>>>>>>>>>  additional information in a new TCP option.
>>>>>>>>>>>>
>>>>>>>>>>>> [ms] This statement needs to be rewritten to correctly reflect
>>>>>>>>>>>> what is
>>>>>>>>>>>
>>>>>>>>>>> requested from IANA. My understanding is that this experimental
>>>>>>>>>>> document
>>>>>>>>>>> asks for allocation of a reserved TCP header flag. This needs to be
>>>>>>>>>>> called out
>>>>>>>>>>> prominently, IMHO. In addition, since this is not a standard, the
>>>>>>>>>>> suggested
>>>>>>>>>>> experimentation with the main TCP header must IMHO be explicitly
>>>>>>>>>>> mentioned. I also suggest to have later in a document a section
>>>>>>>>>>> that
>>>>>>>>>>> explicitly
>>>>>>>>>>> explains why it is appropriate to modify the main TCP header in an
>>>>>>>>>>> experiment.
>>>>>>>>>>>
>>>>>>>>>>> I don’t know if any requirement that IANA assignment need to be
>>>>>>>>>>> called
>>>>>>>>>>> out
>>>>>>>>>>> in the abstract but we can do that. However, I believe the question
>>>>>>>>>>> if
>>>>>>>>>>> this
>>>>>>>>>>> document should or should not assign the bit is still not
>>>>>>>>>>> completely
>>>>>>>>>>> solved, or
>>>>>>>>>>> is it?
>>>>>>>>>>
>>>>>>>>>> I believe this question will have to be reviewed during WGLC and,
>>>>>>>>>> more
>>>>>>>>>> importantly, IETF last call. For the moment, my concern is that the
>>>>>>>>>> document
>>>>>>>>>> correctly describes the IANA allocation.
>>>>>>>>>>
>>>>>>>>>> I would like to see here a statement such as : "Given TCP header
>>>>>>>>>> space is
>>>>>>>>>> scarce, this specification allocates a reserved header bit and
>>>>>>>>>> overloads the
>>>>>>>>>> two ECN flags in the TCP header ...“.
>>>>>>>>>
>>>>>>>>> A bit lengthy but now:
>>>>>>>>>
>>>>>>>>> "Given TCP header space is
>>>>>>>>>      scarce, it allocates a reserved header bit, that was previously
>>>>>>>>> used for
>>>>>>>>>      ECN-Nonce which was recently declared historic, and overloads
>>>>>>>>> the
>>>>>>>>>      two existing ECN flags in the TCP header. Further, additional
>>>>>>>>>      information can be provided in a new TCP option that however is
>>>>>>>>> not
>>>>>>>>> used
>>>>>>>>>      on the TCP SYN."
>>>>>>>>>
>>>>>>>>>>>> * 1.  Introduction
>>>>>>>>>>>>
>>>>>>>>>>>>  Recently, proposed mechanisms like Congestion Exposure (ConEx
>>>>>>>>>>>>  [RFC7713]), DCTCP [RFC8257] or L4S [I-D.ietf-tsvwg-l4s-arch]
>>>>>>>>>>>> need
>>>>>>>>>>>>  more accurate ECN feedback information whenever more than one
>>>>>>>>>>>
>>>>>>>>>>> marking
>>>>>>>>>>>>
>>>>>>>>>>>>  is received in one RTT.
>>>>>>>>>>>>
>>>>>>>>>>>> [ms] At least for RFC 8257 seems to be implementable withoit this.
>>>>>>>>>>>> Instead
>>>>>>>>>>>
>>>>>>>>>>> of stating a "need", it would IMHO make more sense to discuss the
>>>>>>>>>>> benefits
>>>>>>>>>>> of the suggested mechanism in this document of its own, independent
>>>>>>>>>>> of
>>>>>>>>>>> other proposals. To me, this document should be independent of
>>>>>>>>>>> other
>>>>>>>>>>> documents and specifically other experiments. We have to think
>>>>>>>>>>> about
>>>>>>>>>>> cases
>>>>>>>>>>> where not all experiments are successful. Then independent
>>>>>>>>>>> documents
>>>>>>>>>>> will
>>>>>>>>>>> be more future-proof in future.
>>>>>>>>>>>
>>>>>>>>>>> This is a naming collision… The sentence was meant to say that
>>>>>>>>>>> these
>>>>>>>>>>> mechanisms new more accurate ECN feedback than provided today by
>>>>>>>>>>> RFC3168 but it was not meant to say that these mechanism have to
>>>>>>>>>>> use the
>>>>>>>>>>> scheme as specified in this document.
>>>>>>>>>>>
>>>>>>>>>>> I added the following part sentence:
>>>>>>>>>>>
>>>>>>>>>>> „Recently, proposed mechanisms like Congestion Exposure (ConEx
>>>>>>>>>>> [RFC7713]), DCTCP [RFC8257] or L4S [I-D.ietf-tsvwg-l4s-arch] need
>>>>>>>>>>> more
>>>>>>>>>>> accurate ECN feedback information than provided by the feedback
>>>>>>>>>>> scheme
>>>>>>>>>>> as specified in [RFC3168] whenever more than one marking is
>>>>>>>>>>> received in
>>>>>>>>>>> one
>>>>>>>>>>> RTT. This document specifies an alternative feedback scheme that
>>>>>>>>>>> provides
>>>>>>>>>>> more accurate information and could be used by these new TCP
>>>>>>>>>>> extensions.“
>>>>>>>>>>>
>>>>>>>>>>> Does this help?
>>>>>>>>>>
>>>>>>>>>> See my proposal for the abstract. I continue to disagree with the
>>>>>>>>>> term
>>>>>>>>>> "need" but I think this can be sorted out by another term.
>>>>>>>>>>
>>>>>>>>>>>>  If AccECN progresses from experimental to the standards
>>>>>>>>>>>>  track, it is intended to be a complete replacement for classic
>>>>>>>>>>>> TCP/
>>>>>>>>>>>>  ECN feedback, not a fork in the design of TCP.
>>>>>>>>>>>>
>>>>>>>>>>>> [ms] This sentence should be removed, as this is speculation.
>>>>>>>>>>>
>>>>>>>>>>> Why? It states an intent… and that’s the intent that we have.
>>>>>>>>>>>
>>>>>>>>>>>>  Until the AccECN experiment succeeds, [RFC3168] will remain as
>>>>>>>>>>>> the
>>>>>>>>>>>>  standards track specification for adding ECN to TCP.
>>>>>>>>>>>>
>>>>>>>>>>>> [ms] This sentence should be removed (or reworded)
>>>>>>>>>>>
>>>>>>>>>>> Why? Does it help to add an only here:
>>>>>>>>>>>
>>>>>>>>>>> "Until the AccECN experiment succeeds, [RFC3168] will remain as the
>>>>>>>>>>> only
>>>>>>>>>>> standards track specification for adding ECN to TCP.“
>>>>>>>>>>
>>>>>>>>>> This wording is better.
>>>>>>>>>>
>>>>>>>>>>>>  AccECN feedback overloads flags and fields in the main TCP
>>>>>>>>>>>> header
>>>>>>>>>>>>  with new definitions, so both ends have to support the new wire
>>>>>>>>>>>>  protocol before it can be used.
>>>>>>>>>>>>
>>>>>>>>>>>> [ms] In my reading this experimental document asks for *new*
>>>>>>>>>>>> allocation
>>>>>>>>>>>
>>>>>>>>>>> of a reserved TCP header flag.
>>>>>>>>>>>
>>>>>>>>>>> Is this better?
>>>>>>>>>>>
>>>>>>>>>>> "AccECN feedback overloads the two existing ECN flags as well as
>>>>>>>>>>> the
>>>>>>>>>>>     currently reserved and previously called NS flag in the main
>>>>>>>>>>> TCP
>>>>>>>>>>> header
>>>>>>>>>>>     with new definitions, so both ends have to support the new
>>>>>>>>>>> wire
>>>>>>>>>>> protocol
>>>>>>>>>>>     before it can be used.“
>>>>>>>>>>>
>>>>>>>>>>> I understand that you are not happy with the word „overload“ here
>>>>>>>>>>> but
>>>>>>>>>>> the
>>>>>>>>>>> point of this sentence really is that the flags can/could be used
>>>>>>>>>>> differently
>>>>>>>>>>> and therefore we need a new negotiation before we can use them.
>>>>>>>>>>
>>>>>>>>>> For me the following would work: "AccECN feedback overloads the two
>>>>>>>>>> existing ECN flags and
>>>>>>>>>> allocates the currently reserved and previously called NS flag in
>>>>>>>>>> the
>>>>>>>>>> main TCP header.
>>>>>>>>>> Given the new definitions, both ends have to support the new wire
>>>>>>>>>> protocol
>>>>>>>>>> before it can be used."
>>>>>>>>>>
>>>>>>>>>> I believe the wording has to be crystal clear on the reservation of
>>>>>>>>>> bit 7
>>>>>>>>>> when it is discussed the first time in the text. In follow-up
>>>>>>>>>> sections,
>>>>>>>>>> maybe shorter terms could be used.
>>>>>>>>>
>>>>>>>>> Okay, now:
>>>>>>>>>
>>>>>>>>> "AccECN feedback overloads the two existing ECN flags and
>>>>>>>>>    allocates the currently reserved and previously called NS flag in
>>>>>>>>> the
>>>>>>>>>    TCP header, to be used as one field indicating the number of
>>>>>>>>> congestion
>>>>>>>>>    experienced marked packets. Given the new definitions of these
>>>>>>>>> three
>>>>>>>>> bits,
>>>>>>>>>    both ends     have to support the new wire protocol before it can
>>>>>>>>> be
>>>>>>>>> used.
>>>>>>>>>    Therefore during the TCP handshake the two ends use these three
>>>>>>>>> bit
>>>>>>>>> in
>>>>>>>>>    the TCP header to negotiate the most advanced feedback protocol
>>>>>>>>>    that they can both support in a backward compatible way to
>>>>>>>>>    <xref target="RFC3168"/>."
>>>>>>>>>>>
>>>>>>>>>>> If you prefer, we can also remove the NS flag in this list, as ECN
>>>>>>>>>>> Nonce
>>>>>>>>>>> was
>>>>>>>>>>> anyway never deployed.
>>>>>>>>>>>
>>>>>>>>>>>>  For that we refer to [RFC3168] or any RFC that
>>>>>>>>>>>>  specifies a different response to TCP ECN feedback, for example:
>>>>>>>>>>>>  [RFC8257]; or the ECN experiments referred to in
>>>>>>>>>>>>  [I-D.ietf-tsvwg-ecn-experimentation], namely: a TCP-based Low
>>>>>>>>>>>> Latency
>>>>>>>>>>>>  Low Loss Scalable (L4S) congestion control
>>>>>>>>>>>> [I-D.ietf-tsvwg-l4s-arch];
>>>>>>>>>>>>  ECN-capable TCP control packets [I-D.ietf-tcpm-generalized-ecn],
>>>>>>>>>>>> or
>>>>>>>>>>>>  Alternative Backoff with ECN (ABE)
>>>>>>>>>>>>  [I-D.ietf-tcpm-alternativebackoff-ecn].
>>>>>>>>>>>>
>>>>>>>>>>>> [ms] At least ABE seems orthogonal. Anyway, I think this paragraph
>>>>>>>>>>>> can
>>>>>>>>>>>> just
>>>>>>>>>>>
>>>>>>>>>>> be deleted. If other experiments need more accurate feedback, it is
>>>>>>>>>>> up
>>>>>>>>>>> to
>>>>>>>>>>> them to explain how they would use this mechanism. This document
>>>>>>>>>>> should
>>>>>>>>>>> focus on how to signal the feedback, not how to use that.
>>>>>>>>>>>
>>>>>>>>>>> Yes, that is what the paragraph says. Isn’t it better to be
>>>>>>>>>>> explicit
>>>>>>>>>>> about this?
>>>>>>>>>>>
>>>>>>>>>>>>  It is likely (but not required) that the AccECN protocol will be
>>>>>>>>>>>>  implemented along with the following experimental additions to
>>>>>>>>>>>> the
>>>>>>>>>>>>  TCP-ECN protocol: ECN-capable TCP control packets and
>>>>>>>>>>>> retransmissions
>>>>>>>>>>>>  [I-D.ietf-tcpm-generalized-ecn], which includes the ECN-capable
>>>>>>>>>>>> SYN/
>>>>>>>>>>>>  ACK experiment [RFC5562]; and testing receiver non-compliance
>>>>>>>>>>>>  [I-D.moncaster-tcpm-rcv-cheat].
>>>>>>>>>>>>
>>>>>>>>>>>> [ms] I am a big fan of simple, standalone documents. In my view,
>>>>>>>>>>>> the
>>>>>>>>>>>> TCPM
>>>>>>>>>>>
>>>>>>>>>>> working group should publish draft-ietf-tcpm-accurate-ecn and
>>>>>>>>>>> draft-ietf-
>>>>>>>>>>> tcpm-generalized-ecn independent documents, which probably implies
>>>>>>>>>>> that
>>>>>>>>>>> draft-ietf-tcpm-generalized-ecn does not use AccECN. If
>>>>>>>>>>> experimentation
>>>>>>>>>>> with ECT in SYN requires a combination, this could be done in a
>>>>>>>>>>> new,
>>>>>>>>>>> third
>>>>>>>>>>> document. Apart from having simpler focused documents, this could
>>>>>>>>>>> significantly help later with moving forward documents to standards
>>>>>>>>>>> track.
>>>>>>>>>>>
>>>>>>>>>>> I disagree, however, this is a discussion to have on
>>>>>>>>>>> draft-ietf-tcpm-
>>>>>>>>>>> generalized-ecn. I don’t see a problem in  providing a reference
>>>>>>>>>>> here
>>>>>>>>>>> that
>>>>>>>>>>> says „it is likely…“ and nothing more.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> * 1.1.  Document Roadmap
>>>>>>>>>>>>
>>>>>>>>>>>> [ms] A macroscopic comment is that this document has a lot of
>>>>>>>>>>>> introduction
>>>>>>>>>>>
>>>>>>>>>>> and tutorial text with lot's of redundancy towards other documents.
>>>>>>>>>>> I
>>>>>>>>>>> think
>>>>>>>>>>> the document can be made much easier to read by shorten it. In many
>>>>>>>>>>> cases
>>>>>>>>>>> this is just an editorial change as there is redundancy. As one
>>>>>>>>>>> such
>>>>>>>>>>> example,
>>>>>>>>>>> just remove this section.
>>>>>>>>>>>
>>>>>>>>>>> I guess this a matter of taste. As an AD, I’m a big fan of short
>>>>>>>>>>> and
>>>>>>>>>>> concise
>>>>>>>>>>> documents, however, some redundancy can also help understanding,
>>>>>>>>>>> especially if you explain things multiple times but with a
>>>>>>>>>>> different
>>>>>>>>>>> level of
>>>>>>>>>>> detail. I personally would not need the roadmap but I know many
>>>>>>>>>>> people
>>>>>>>>>>> who find these things helpful and to be honest I don’t see how
>>>>>>>>>>> removing
>>>>>>>>>>> this
>>>>>>>>>>> part makes the doc any better. If you don’t want it, don’t read it.
>>>>>>>>>>>
>>>>>>>>>>>> * 1.2.  Goals
>>>>>>>>>>>>
>>>>>>>>>>>> [ms] I think this section can also just be removed.
>>>>>>>>>>>
>>>>>>>>>>> I have to say I also don’t see the point of removing this part.
>>>>>>>>>>> Given
>>>>>>>>>>> we’ve
>>>>>>>>>>> done the work on requirements, I think we should also link to this
>>>>>>>>>>> doc
>>>>>>>>>>> somewhere.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> * 1.3.  Experiment Goals
>>>>>>>>>>>>
>>>>>>>>>>>>  TCP is critical to the robust functioning of the Internet,
>>>>>>>>>>>> therefore
>>>>>>>>>>>>  any proposed modifications to TCP need to be thoroughly tested.
>>>>>>>>>>>> The
>>>>>>>>>>>>  present specification describes an experimental protocol that
>>>>>>>>>>>> adds
>>>>>>>>>>>>  more accurate ECN feedback to the TCP protocol.  The intention
>>>>>>>>>>>> is to
>>>>>>>>>>>>  specify the protocol sufficiently so that more than one
>>>>>>>>>>>>  implementation can be built in order to test its function,
>>>>>>>>>>>> robustness
>>>>>>>>>>>>  and interoperability (with itself and with previous version of
>>>>>>>>>>>> ECN
>>>>>>>>>>>>  and TCP).
>>>>>>>>>>>>
>>>>>>>>>>>> [ms] I think all what is written in this paragraph is obvious, no?
>>>>>>>>>>>> Can't we just
>>>>>>>>>>>
>>>>>>>>>>> delete this?
>>>>>>>>>>>
>>>>>>>>>>> Sure, however, I don’t think it hurts to spell it out. For me both
>>>>>>>>>>> is
>>>>>>>>>>> fine, keep it
>>>>>>>>>>> or remove it.
>>>>>>>>>>>
>>>>>>>>>>>>  The experimental protocol will be considered successful if it is
>>>>>>>>>>>>  deployed and if it satisfies the requirements of [RFC7560] in
>>>>>>>>>>>> the
>>>>>>>>>>>>  consensus opinion of the IETF tcpm working group.  In short,
>>>>>>>>>>>> this
>>>>>>>>>>>>  requires that it improves the accuracy and timeliness of TCP's
>>>>>>>>>>>> ECN
>>>>>>>>>>>>  feedback, as claimed in Section 5, while striking a balance
>>>>>>>>>>>> between
>>>>>>>>>>>>  the conflicting requirements of resilience, integrity and
>>>>>>>>>>>>  minimisation of overhead.  It also requires that it is not
>>>>>>>>>>>> unduly
>>>>>>>>>>>>  complex, and that it is compatible with prevalent equipment
>>>>>>>>>>>>  behaviours in the current Internet (e.g. hardware offloading and
>>>>>>>>>>>>  middleboxes), whether or not they comply with standards.
>>>>>>>>>>>>
>>>>>>>>>>>>  Testing will mostly focus on fall-back strategies in case of
>>>>>>>>>>>>  middlebox interference.  Current recommended strategies are
>>>>>>>>>>>> specified
>>>>>>>>>>>>  in Sections 3.1.2, 3.2.3, 3.2.4 and 3.2.7.  The effectiveness of
>>>>>>>>>>>>  these strategies depends on the actual deployment situation of
>>>>>>>>>>>>  middleboxes.  Therefore experimental verification to confirm
>>>>>>>>>>>> large-
>>>>>>>>>>>>  scale path traversal in the Internet is needed before finalizing
>>>>>>>>>>>> this
>>>>>>>>>>>>  specification on the Standards Track.
>>>>>>>>>>>>
>>>>>>>>>>>> [ms] These two paragraphs must be entirely rewritten. As I have
>>>>>>>>>>>
>>>>>>>>>>> mentioned before, I don't think an RFC should speculate about TCPM
>>>>>>>>>>> and
>>>>>>>>>>> its
>>>>>>>>>>> consensus opinion. I would suggest a wording along the lines of:
>>>>>>>>>>>>
>>>>>>>>>>>> <ms>
>>>>>>>>>>>>  The experimental protocol will be considered successful if
>>>>>>>>>>>>  testing confirms that the proposed mechanism can be deployed at
>>>>>>>>>>>> large
>>>>>>>>>>>
>>>>>>>>>>> scale.
>>>>>>>>>>>>
>>>>>>>>>>>>  Testing will mostly focus on fall-back strategies in case of
>>>>>>>>>>>>  middlebox interference.  Current recommended strategies are
>>>>>>>>>>>> specified
>>>>>>>>>>>>  in Sections 3.1.2, 3.2.3, 3.2.4 and 3.2.7.  The effectiveness of
>>>>>>>>>>>>  these strategies depends on the actual deployment situation of
>>>>>>>>>>>>  middleboxes.  Therefore experimental verification to confirm
>>>>>>>>>>>> large-
>>>>>>>>>>>>  scale path traversal in the Internet is needed, e.g., by support
>>>>>>>>>>>> in
>>>>>>>>>>>>  major TCP stacks.
>>>>>>>>>>>> </ms>
>>>>>>>>>>>>
>>>>>>>>>>> I don’t understand your point here. I don’t think that the
>>>>>>>>>>> paraphrase
>>>>>>>>>>> speculates about the consensus of tcpm, in contrast it say tcpm has
>>>>>>>>>>> to
>>>>>>>>>>> decided if the requirements previously specified by tcpm are
>>>>>>>>>>> sufficiently
>>>>>>>>>>> fulfilled. I don’t see a reason to not mention the requirement
>>>>>>>>>>> draft as
>>>>>>>>>>> this
>>>>>>>>>>> draft as tcpm consensus and was written for this purpose.
>>>>>>>>>>
>>>>>>>>>> My suggested wording uses the expression "can be deployed at large
>>>>>>>>>> scale"
>>>>>>>>>> and I believe this is relevant.
>>>>>>>>>>
>>>>>>>>>> The document already describes in Section 5 how the protocol
>>>>>>>>>> satisfies
>>>>>>>>>> the agreed requirements for a more accurate ECN feedback protocol
>>>>>>>>>> [RFC7560].
>>>>>>>>>> So, if the TCPM working group publishes this document with the
>>>>>>>>>> content of
>>>>>>>>>> Section 5, I believe the TCPM working group already has reached
>>>>>>>>>> consensus
>>>>>>>>>> that the protocol meets requirements. In addition, it is possible
>>>>>>>>>> that new
>>>>>>>>>> requirements would be identified in future, e.g., as an outcome of
>>>>>>>>>> the
>>>>>>>>>> experiment, and that would obviously have to be considered by TCPM.
>>>>>>>>>> In that
>>>>>>>>>> case, for the success of the experiment not only RFC 7560 would
>>>>>>>>>> matter, but
>>>>>>>>>> also further requirements. My proposed wording does not have all
>>>>>>>>>> these
>>>>>>>>>> problems.
>>>>>>>>>>
>>>>>>>>>> In a nutshell, I continue to believe that this section has to
>>>>>>>>>> change.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Okay, used your proposed wording. You have a point about the
>>>>>>>>> requirement
>>>>>>>>> and I mis-read you proposal earlier as „has to be deployed
>>>>>>>>> large-scale“.
>>>>>>>>>
>>>>>>>>>>>> * 1.5.  Recap of Existing ECN feedback in IP/TCP
>>>>>>>>>>>>
>>>>>>>>>>>> [ms] This section could probably be shortened as well.
>>>>>>>>>>>>
>>>>>>>>>>>>  The last bit in byte 13 of the TCP header was defined as the
>>>>>>>>>>>> Nonce
>>>>>>>>>>>>  Sum (NS) for the ECN Nonce [RFC3540].  RFC 3540 was never
>>>>>>>>>>>> deployed
>>>>>>>>>>>> so
>>>>>>>>>>>>  it is being reclassified as historic, making this TCP flag
>>>>>>>>>>>> available
>>>>>>>>>>>>  for use by the AccECN experiment instead.
>>>>>>>>>>>>
>>>>>>>>>>>> [ms] This wording, as well as Figure 1, needs to take into account
>>>>>>>>>>>> the
>>>>>>>>>>>> IANA
>>>>>>>>>>>
>>>>>>>>>>> status when draft-ietf-tsvwg-ecn-experimentation is published.
>>>>>>>>>>>
>>>>>>>>>>> Is does. However, I can explicitly say that is has be re-clssified
>>>>>>>>>>> as
>>>>>>>>>>> reserved.
>>>>>>>>>>>
>>>>>>>>>>> "RFC 3540 was never deployed so it is being reclassified as
>>>>>>>>>>> historic
>>>>>>>>>>> [I-D.ietf-
>>>>>>>>>>> tsvwg-ecn-experimentation] and the respective flag has been marked
>>>>>>>>>>> as
>>>>>>>>>>> „reserved“ in the IANA TCP Header Flags registry, making this TCP
>>>>>>>>>>> flag
>>>>>>>>>>> available for use by the AccECN experiment instead.“
>>>>>>>>>>>
>>>>>>>>>>> Better?
>>>>>>>>>>>
>>>>>>>>>>>> In my understanding, this experimental document asks for new
>>>>>>>>>>>> assignment
>>>>>>>>>>>
>>>>>>>>>>> of a reserved TCP header flag.
>>>>>>>>>>>
>>>>>>>>>>> As I said I’m not sure if we have fully concluded this discussion
>>>>>>>>>>> yet.
>>>>>>>>>>> However,
>>>>>>>>>>> what we really would want to is mention somewhere that this
>>>>>>>>>>> experiment
>>>>>>>>>>> with this flags is running. I guess there are three options:
>>>>>>>>>>> 1) keep it in the registry as reserved and conserve the knowledge
>>>>>>>>>>> in
>>>>>>>>>>> tcpm
>>>>>>>>>>> that this experiment is running and no other experimental RFC such
>>>>>>>>>>> use
>>>>>>>>>>> this
>>>>>>>>>>> flags as long as this experiment is running.
>>>>>>>>>>> 2) Keep is marked as reserved but add a note about this experiment
>>>>>>>>>>> in
>>>>>>>>>>> the
>>>>>>>>>>> IANA registry
>>>>>>>>>>> 3) Or assign it right away with IESG approval. I guess in this case
>>>>>>>>>>> tcpm
>>>>>>>>>>> could
>>>>>>>>>>> also consider to change the registration policy to „IETF Review“.
>>>>>>>>>>
>>>>>>>>>> The current registration policy for the TCP header flags is
>>>>>>>>>> "standards
>>>>>>>>>> action". I understand that the IESG could approve exceptions. But
>>>>>>>>>> given the
>>>>>>>>>> policy, I believe the document has to be very precise on the request
>>>>>>>>>> regarding bit 7.
>>>>>>>>>
>>>>>>>>> Okay, it now says:
>>>>>>>>>
>>>>>>>>> "[TO BE REMOVED: IANA is requested to update the existing entry in
>>>>>>>>> the
>>>>>>>>> Transmission Control Protocol (TCP) Header Flags registration
>>>>>>>>>
>>>>>>>>> (https://www.iana.org/assignments/tcp-header-flags/tcp-header-flags.xhtml#tcp-header-flags-1)
>>>>>>>>> for Bit 7 to "AE (Accurate ECN), previously used by Historic as NS
>>>>>>>>> (Nonce
>>>>>>>>> Sum) [RFC3540, RFC8311]" and change the reference to this RFC-to-be
>>>>>>>>> instead
>>>>>>>>> of RFC8311.]“
>>>>>>>>>
>>>>>>>>> I guess we could also ask IANA to add an additional comment column
>>>>>>>>> instead
>>>>>>>>> (but not sure if we then have to update RFC3168, which I think we
>>>>>>>>> really
>>>>>>>>> don’t want. Should be fine now.
>>>>>>>>>
>>>>>>>>>>>> * 2.  AccECN Protocol Overview and Rationale
>>>>>>>>>>>>
>>>>>>>>>>>>  o  an essential part that re-uses ECN TCP header bits to feed
>>>>>>>>>>>> back
>>>>>>>>>>>>     the number of arriving CE marked packets.  This provides more
>>>>>>>>>>>>     accuracy than classic ECN feedback, but limited resilience
>>>>>>>>>>>> against
>>>>>>>>>>>>     ACK loss;
>>>>>>>>>>>>
>>>>>>>>>>>> [ms] The word "re-use" is IMHO not correct.
>>>>>>>>>>>
>>>>>>>>>>> I think this is nit picking. Using a different phrasing here makes
>>>>>>>>>>> the
>>>>>>>>>>> sentence
>>>>>>>>>>> unnecessary complicated. We don’t try to some how get a round the
>>>>>>>>>>> fact
>>>>>>>>>>> that we need to handle the flag registration correctly. However,
>>>>>>>>>>> here
>>>>>>>>>>> the
>>>>>>>>>>> point really is to explain how the protocol word. The main point of
>>>>>>>>>>> using the
>>>>>>>>>>> work „re-use“ here is really that we say that these flags are or
>>>>>>>>>>> have
>>>>>>>>>>> been
>>>>>>>>>>> used different by other TCP extension (and we therefore need a
>>>>>>>>>>> proper
>>>>>>>>>>> negotiation scheme).
>>>>>>>>>>
>>>>>>>>>> If the allocation of a reserved flag is correctly explained in the
>>>>>>>>>> abstract and introduction, I think these sentences can use a bit
>>>>>>>>>> relaxed
>>>>>>>>>> terminology.
>>>>>>>>>>
>>>>>>>>>>>>  The two part design was necessary, given limitations on the
>>>>>>>>>>>> space
>>>>>>>>>>>>  available for TCP options and given the possibility that certain
>>>>>>>>>>>>  incorrectly designed middleboxes prevent TCP using any new
>>>>>>>>>>>> options
>>>
>>>
>>> --
>>> ________________________________________________________________
>>> Bob Briscoe                               http://bobbriscoe.net/
>>>
>>
>
>