[tcpm] Comments on draft-ietf-tcpm-accurate-ecn

["Scharf, Michael (Nokia - DE/Stuttgart)" <[e-mail]>]

Hi all,

I have read draft-ietf-tcpm-accurate-ecn-05 (without the appendix). I believe this document needs further work before moving forward.

Please find below my comments marked as [ms]. I have read the document independent of the review from Gorry. I apologize if there is duplication.

Thanks

Michael (with no hat on)


******************************

* Abstract:

   Recently, new TCP mechanisms like Congestion Exposure (ConEx) or Data Center TCP
   (DCTCP) need more accurate ECN feedback information whenever more
   than one marking is received in one RTT.

[ms] I don't think this statement is fully backed by RFC 8257. I suggest to remove this, or replace it by a more generic statement that more accurate information can be useful for several TCP extensions.

   This document specifies an
   experimental scheme to provide more than one feedback signal per RTT
   in the TCP header.  Given TCP header space is scarce, it overloads
   the three existing ECN-related flags in the TCP header and provides
   additional information in a new TCP option.

[ms] This statement needs to be rewritten to correctly reflect what is requested from IANA. My understanding is that this experimental document asks for allocation of a reserved TCP header flag. This needs to be called out prominently, IMHO. In addition, since this is not a standard, the suggested experimentation with the main TCP header must IMHO be explicitly mentioned. I also suggest to have later in a document a section that explicitly explains why it is appropriate to modify the main TCP header in an experiment.


* 1.  Introduction

   Recently, proposed mechanisms like Congestion Exposure (ConEx
   [RFC7713]), DCTCP [RFC8257] or L4S [I-D.ietf-tsvwg-l4s-arch] need
   more accurate ECN feedback information whenever more than one marking
   is received in one RTT.

[ms] At least for RFC 8257 seems to be implementable withoit this. Instead of stating a "need", it would IMHO make more sense to discuss the benefits of the suggested mechanism in this document of its own, independent of other proposals. To me, this document should be independent of other documents and specifically other experiments. We have to think about cases where not all experiments are successful. Then independent documents will be more future-proof in future.

   If AccECN progresses from experimental to the standards
   track, it is intended to be a complete replacement for classic TCP/
   ECN feedback, not a fork in the design of TCP.

[ms] This sentence should be removed, as this is speculation.

   Until the AccECN experiment succeeds, [RFC3168] will remain as the
   standards track specification for adding ECN to TCP.

[ms] This sentence should be removed (or reworded)

   AccECN feedback overloads flags and fields in the main TCP header
   with new definitions, so both ends have to support the new wire
   protocol before it can be used.

[ms] In my reading this experimental document asks for *new* allocation of a reserved TCP header flag.

   For that we refer to [RFC3168] or any RFC that
   specifies a different response to TCP ECN feedback, for example:
   [RFC8257]; or the ECN experiments referred to in
   [I-D.ietf-tsvwg-ecn-experimentation], namely: a TCP-based Low Latency
   Low Loss Scalable (L4S) congestion control [I-D.ietf-tsvwg-l4s-arch];
   ECN-capable TCP control packets [I-D.ietf-tcpm-generalized-ecn], or
   Alternative Backoff with ECN (ABE)
   [I-D.ietf-tcpm-alternativebackoff-ecn].

[ms] At least ABE seems orthogonal. Anyway, I think this paragraph can just be deleted. If other experiments need more accurate feedback, it is up to them to explain how they would use this mechanism. This document should focus on how to signal the feedback, not how to use that.

   It is likely (but not required) that the AccECN protocol will be
   implemented along with the following experimental additions to the
   TCP-ECN protocol: ECN-capable TCP control packets and retransmissions
   [I-D.ietf-tcpm-generalized-ecn], which includes the ECN-capable SYN/
   ACK experiment [RFC5562]; and testing receiver non-compliance
   [I-D.moncaster-tcpm-rcv-cheat].

[ms] I am a big fan of simple, standalone documents. In my view, the TCPM working group should publish draft-ietf-tcpm-accurate-ecn and draft-ietf-tcpm-generalized-ecn independent documents, which probably implies that draft-ietf-tcpm-generalized-ecn does not use AccECN. If experimentation with ECT in SYN requires a combination, this could be done in a new, third document. Apart from having simpler focused documents, this could significantly help later with moving forward documents to standards track.


* 1.1.  Document Roadmap

[ms] A macroscopic comment is that this document has a lot of introduction and tutorial text with lot's of redundancy towards other documents. I think the document can be made much easier to read by shorten it. In many cases this is just an editorial change as there is redundancy. As one such example, just remove this section.


* 1.2.  Goals

[ms] I think this section can also just be removed.


* 1.3.  Experiment Goals

   TCP is critical to the robust functioning of the Internet, therefore
   any proposed modifications to TCP need to be thoroughly tested. The
   present specification describes an experimental protocol that adds
   more accurate ECN feedback to the TCP protocol.  The intention is to
   specify the protocol sufficiently so that more than one
   implementation can be built in order to test its function, robustness
   and interoperability (with itself and with previous version of ECN
   and TCP).

[ms] I think all what is written in this paragraph is obvious, no? Can't we just delete this?

   The experimental protocol will be considered successful if it is
   deployed and if it satisfies the requirements of [RFC7560] in the
   consensus opinion of the IETF tcpm working group.  In short, this
   requires that it improves the accuracy and timeliness of TCP's ECN
   feedback, as claimed in Section 5, while striking a balance between
   the conflicting requirements of resilience, integrity and
   minimisation of overhead.  It also requires that it is not unduly
   complex, and that it is compatible with prevalent equipment
   behaviours in the current Internet (e.g. hardware offloading and
   middleboxes), whether or not they comply with standards.

   Testing will mostly focus on fall-back strategies in case of
   middlebox interference.  Current recommended strategies are specified
   in Sections 3.1.2, 3.2.3, 3.2.4 and 3.2.7.  The effectiveness of
   these strategies depends on the actual deployment situation of
   middleboxes.  Therefore experimental verification to confirm large-
   scale path traversal in the Internet is needed before finalizing this
   specification on the Standards Track.

[ms] These two paragraphs must be entirely rewritten. As I have mentioned before, I don't think an RFC should speculate about TCPM and its consensus opinion. I would suggest a wording along the lines of:

<ms>
   The experimental protocol will be considered successful if
   testing confirms that the proposed mechanism can be deployed at large scale.
   Testing will mostly focus on fall-back strategies in case of
   middlebox interference.  Current recommended strategies are specified
   in Sections 3.1.2, 3.2.3, 3.2.4 and 3.2.7.  The effectiveness of
   these strategies depends on the actual deployment situation of
   middleboxes.  Therefore experimental verification to confirm large-
   scale path traversal in the Internet is needed, e.g., by support in
   major TCP stacks.
</ms>


* 1.5.  Recap of Existing ECN feedback in IP/TCP

[ms] This section could probably be shortened as well.

   The last bit in byte 13 of the TCP header was defined as the Nonce
   Sum (NS) for the ECN Nonce [RFC3540].  RFC 3540 was never deployed so
   it is being reclassified as historic, making this TCP flag available
   for use by the AccECN experiment instead.

[ms] This wording, as well as Figure 1, needs to take into account the IANA status when draft-ietf-tsvwg-ecn-experimentation is published. In my understanding, this experimental document asks for new assignment of a reserved TCP header flag.


* 2.  AccECN Protocol Overview and Rationale

   o  an essential part that re-uses ECN TCP header bits to feed back
      the number of arriving CE marked packets.  This provides more
      accuracy than classic ECN feedback, but limited resilience against
      ACK loss;

[ms] The word "re-use" is IMHO not correct.

   The two part design was necessary, given limitations on the space
   available for TCP options and given the possibility that certain
   incorrectly designed middleboxes prevent TCP using any new options.

[ms] IMHO it would make sense to more explicitly mention the downsides of only specifying an option and not allocating a TCP header flag, in this experimental document. The obvious  alternative would be to postpone the header flag allocation to a follow-up standards track document and just keep it reserved.

   The essential part overloads the previous definition of the three
   flags in the TCP header that had been assigned for use by ECN.  This
   design choice deliberately replaces the classic ECN feedback
   protocol, rather than leaving classic ECN feedback intact and adding
   more accurate feedback separately because:

[ms] Similar like previous comments, in my reading there are only _two_ ECN header flags. And, in addition, I think care is needed with wording such "replaces the classic ECN feedback". I don't think this experiment replaces the ECN standards. That would be up to a follow-up PS.


2.1.  Capability Negotiation

   AccECN is a change to the wire protocol of the main TCP header,
   therefore it can only be used if both endpoints have been upgraded to
   understand it.  The TCP client signals support for AccECN on the
   initial SYN of a connection and the TCP server signals whether it
   supports AccECN on the SYN/ACK.  The TCP flags on the SYN that the
   client uses to signal AccECN support have been carefully chosen so
   that a TCP server will interpret them as a request to support the
   most recent variant of ECN feedback that it supports.  Then the
   client falls back to the same variant of ECN feedback.

[ms] As this is an experimental specification, I would really like to see a discussion how a future standards track version of more accurate ECN could be negotiated. How could both endpoints detect whether the other one implements the future standards track version? For instance, would the only safe variant be that we allocate yet another reserved TCP header flag in a proposed standard to negotiate the standards track version, thus investing another reserved bit in the TCP header? I may be wrong, but to me it is too early to speculate how the PS version would look like, and whether it would have to be different to the experimental version, due to lessons learnt. I believe in the IETF we typically design protocols that allow future extension, and it is not exactly clear to be how AccECN could be extended later.

   An AccECN TCP client does not send the new AccECN Option on the SYN
   as SYN option space is limited and successful negotiation using the
   flags in the main header is taken as sufficient evidence that both
   ends also support the AccECN Option.  The TCP server sends the AccECN
   Option on the SYN/ACK and the client sends it on the first ACK to
   test whether the network path forwards the option correctly.

[ms] For what it is worth, I would personally be quite fine with allowing (or even mandating) an option in the SYN in the experimental version of this protocol. For instance, saving the SYN option space would then an excellent reason for moving towards the PS specification. I am also fine with being in the rough part of the consensus here.
* 2.3.  Delayed ACKs and Resilience Against ACK Loss

   If the AccECN Option is not available, e.g. it is being stripped by a
   middlebox, the AccECN protocol will only feed back information on CE
   markings (using the ACE field).  Although not ideal, this will be
   sufficient, because it is envisaged that neither ECT(0) nor ECT(1)
   will ever indicate more severe congestion than CE, even though future
   uses for ECT(0) or ECT(1) are still unclear
   [I-D.ietf-tsvwg-ecn-experimentation].

[ms] This needs to be reworded


* 2.4.  Feedback Metrics

   The CE packet counter in the ACE field and the CE byte counter in the
   AccECN Option both provide feedback on received CE-marks.  The CE
   packet counter includes control packets that do not have payload
   data, while the CE byte counter solely includes marked payload bytes.
   If both are present, the byte counter in the option will provide the
   more accurate information needed for modern congestion control and
   policing schemes, such as DCTCP or ConEx.

[ms] I suggest to write in the last sentence only "... the option will provide the more accurate information needed for congestion control". In general, I would prefer to have references to other mechanisms at only few (ideally a *single*) places in the document, instead of mixing them together.

   Feedback in bytes is recommended in order to protect against the
   receiver using attacks similar to 'ACK-Division' to artificially
   inflate the congestion window, which is why [RFC5681] now recommends
   that TCP counts acknowledged bytes not packets.

[ms] At least the last part and the reference to RFC 5681 is IMHO not needed here.


* 2.5.  Generic (Dumb) Reflector

   The ACE field provides information about CE markings on both data and
   control packets.  According to [RFC3168] the Data Sender is meant to
   set control packets to Not-ECT.  However, mechanisms in certain
   private networks (e.g. data centres) set control packets to be ECN
   capable because they are precisely the packets that performance
   depends on most.

   For this reason, AccECN is designed to be a generic reflector of
   whatever ECN markings it sees, whether or not they are compliant with
   a current standard.  Then as standards evolve, Data Senders can
   upgrade unilaterally without any need for receivers to upgrade too.
   It is also useful to be able to rely on generic reflection behaviour
   when senders need to test for unexpected interference with markings
   (for instance [I-D.kuehlewind-tcpm-ecn-fallback] and
   [I-D.moncaster-tcpm-rcv-cheat]).

   The initial SYN is the most critical control packet, so AccECN
   provides feedback on whether it is CE marked.  Although RFC 3168
   prohibits an ECN-capable SYN, providing feedback of CE marking on the
   SYN supports future scenarios in which SYNs might be ECN-enabled
   (without prejudging whether they ought to be).  For instance,
   [I-D.ietf-tsvwg-ecn-experimentation] updates this aspect of RFC 3168
   to allow experimentation with ECN-capable TCP control packets.

[ms] To me, the only thing that matters in this document that AccECN can provide feedback on whether the SYN is CE marked. The discussion on how to experiment with ECT e.g. in SYNs IMHO does not belong into this document. So it seems sufficient here to note that one of the benefits of AccECN is that CE marks in SYNs can be fed back.


* 3.1.1.  Negotiation during the TCP handshake

   Given the ECN Nonce [RFC3540] is being reclassified as historic, the
   present specification renames the TCP flag at bit 7 of the TCP header
   flags from NS (Nonce Sum) to AE (Accurate ECN) (see IANA
   Considerations in Section 6).

[ms] As mentioned before, this needs to be rewritten to ask for new IANA allocation of bit 7 in the TCP header flags.

   Table 2: ECN capability negotiation between Client (A) and Server (B)

[ms] As far as I can see, in -05 this table allocates all existing codepoints, while -03 had two currently unused codepoints. Not having any codepoints left seems to me not really future proof, e.g., regarding future proposed standards in this space (and I personally believe that TCP header flags must be allocated in a PS). And I don't fully see a need of feeding back ECT0 and specifically ECT1 in the TCP header flags as part of the experiment. Do we know for sure that this is the only possible use case of these two unallocated header bits? And why can't e.g. this be done in a TCP option instead? Or do I miss something?


* 3.1.2.  Retransmission of the SYN

   However,
   current measurements imply that a drop is less likely to be due to
   middlebox interference than other intermittent causes of loss, e.g.
   congestion, wireless interference, etc.

[ms] Such wording IMHO doesn't belong into normative text. This may actually also apply to other heuristics discussed in this section, which are not really important for interoperability.


3.2.7.  Path Traversal of the AccECN Option

3.2.7.1.  Testing the AccECN Option during the Handshake

   The TCP client MUST NOT include the AccECN TCP Option on the SYN.

[ms] I am not sure if I really understand the motivation for not allowing a option in the SYN. If the sender has space in the SYN left, what is the harm in an experimental version of the protocol? And I may miss something, but what would prevent the use of 2-byte option to negotiate the use of AccECN, e.g., to avoid experimental allocation of bit 7 in the initial SYN? While I think many tutorial text in this document could be shortened, I believe the use of a reserved TCP header flag should be reasoned.


* 3.2.8.  Usage of the AccECN TCP Option

   The following rules determine when a Data Receiver in AccECN mode
   sends the AccECN TCP Option, and which fields to include:

   Change-Triggered ACKs:  If an arriving packet increments a different
      byte counter to that incremented by the previous packet, the Data
      Receiver MUST immediately send an ACK with an AccECN Option,
      without waiting for the next delayed ACK (this is in addition to
      the safety recommendation in Section 3.2.5 against ambiguity of
      the ACE field).

      This is stated as a "MUST" so that the data sender can rely on
      change-triggered ACKs to detect transitions right from the very
      start of a flow, without first having to detect whether the
      receiver complies.  A concern has been raised that certain offload
      hardware needed for high performance might not be able to support
      change-triggered ACKs, although high performance protocols such as
      DCTCP successfully use change-triggered ACKs.

[ms] To me this sounds like a perfect example for a SHOULD with additional guidance why implementing this SHOULD is really important.

   For the avoidance of doubt, the change-triggered ACK mechanism is
   deliberately worded to ignore the arrival of a control packet with no
   payload, which therefore does not alter any byte counters, because it
   is important that TCP does not acknowledge pure ACKs.  The change-
   triggered ACK approach will lead to some additional ACKs but it feeds
   back the timing and the order in which ECN marks are received with
   minimal additional complexity.

[ms] The additional acks create network load. I think some wording is needed on the tradeoff between information accuracy and network load. There are network environments in which any additional packet is very expensive (e.g., energy) and it is not clear to me how the protocol design takes into account the potential overhead of additional ACKs. Maybe this could be another reason for a SHOULD.


* 4.2.  Compatibility with Other TCP Options and Experiments

   AccECN is compatible (at least on paper) with the most commonly used
   TCP options: MSS, time-stamp, window scaling, SACK and TCP-AO.  It is
   also compatible with the recent promising experimental TCP options
   TCP Fast Open (TFO [RFC7413]) and Multipath TCP (MPTCP [RFC6824]).

[ms] I would suggest the wording "... compatible with the experimental TCP options ..." or even "... compatible with the TCP options ...".


* 4.3.  Compatibility with Feedback Integrity Mechanisms

[ms] Quite a bit in this section is experimental work, which IMHO should be clearly emphasized. The one exception is...

      However, TCP-AO is often too brittle to use on many end-to-end
      paths, where middleboxes can make verification fail in their
      attempts to improve performance or security, e.g. by
      resegmentation or shifting the sequence space.

[ms] I am not sure if deployment challenges of other options need to be discussed in this document.

   Originally the ECN Nonce [RFC3540] was proposed to ensure integrity
   of congestion feedback.  With minor changes AccECN could be optimised
   for the possibility that the ECT(1) codepoint might be used as an ECN
   Nonce . However, given RFC 3540 is being reclassified as historic,
   the AccECN design has been generalised so that it ought to be able to
   support other possible uses of the ECT(1) codepoint, such as a lower
   severity or a more instant congestion signal than CE.

[ms] The discussion of RFC 3540 can probably be removed to a large extent.


* 6.  IANA Considerations

[ms] I think this section needs to be rewritten to request a new allocation of bit 7 of the TCP header flags. At least for the process I think it would make sense to have somewhere in the document a comprehensive explanation of why an experimental document requests a change of the main TCP header, and why this cannot be avoided (most notably in the initial SYN) by an alternative protocol design.


* 9.  Comments Solicited

   Comments and questions are encouraged and very welcome.  They can be
   addressed to the IETF TCP maintenance and minor modifications working
   group mailing list <[e-mail]>, and/or to the authors.

[ms] This section is not needed IMHO


10.  References

   [I-D.ietf-tsvwg-ecn-experimentation]
              Black, D., "Relaxing Restrictions on Explicit Congestion
              Notification (ECN) Experimentation", draft-ietf-tsvwg-ecn-
              experimentation-07 (work in progress), October 2017.

[ms] Normative reference?