[tcpm] Comments on draft-ietf-tcpm-generalized-ecn

["Scharf, Michael (Nokia - DE/Stuttgart)" <[e-mail]>]

Hi all,

I have read draft-ietf-tcpm-generalized-ecn-02. Some high-level thoughts:

- I believe that this experiment should be specified independently from other experiments, including AccECN. I believe this is by and large possible. A separation may perhaps require moving the SYN handling to a new I-D that is more closely tied into AccECN, but at first sight I don't see a major problem with this. For instance, I could think of scenarios in which it may make sense to *only* add ECN support to TCP control packets without deploying any other experiment.

- Apart from the unnecessary complexity resulting from the dependencies on AccECN, I think this document has a lot of other text (most notably in Section 4) that is not really relevant to implementers. Maybe one could consider moving more non-normative discussion to an appendix.

- I would suggest to move all discussion on implementation guidance, such as caching strategies, to one (non-normative) section.

Please find below my comments marked as [ms]. I have read the document independent of the review from Gorry. I apologize if there is duplication.

Thanks

Michael (with no hat on)


******************************

* Abstract

   This document describes an experimental modification to ECN when used
   with TCP.  It allows the use of ECN on the following TCP packets:
   SYNs, pure ACKs, Window probes, FINs, RSTs and retransmissions.

[ms] Obsoletion of RFC 5562 should probably be mentioned here. To me, it is not super-important to have the SYN specification in this document, it could e.g. be moved elsewhere into a separate document.


* 1.  Introduction

   RFC 5562 [RFC5562] is an experimental modification to ECN that enables
   ECN support for TCP SYN-ACK packets.

[ms] If the RFC is obsoleted, it should probably be mentioned here, too.

   ECN++ is a sender-side change.  It works whether the two ends of the
   TCP connection use classic ECN feedback [RFC3168] or experimental
   Accurate ECN feedback (AccECN [I-D.ietf-tcpm-accurate-ecn]).
   Nonetheless, if the client does not implement AccECN, it cannot use
   ECN++ on the one packet that offers most benefit from it - the
   initial SYN.  Therefore, implementers of ECN++ are RECOMMENDED to
   also implement AccECN.

[ms] I don't think that this coupling of this experiment and the AccECN experiment is future-proof. We have to foresee e.g. the case that only one of the two experiments suceeds, e.g., that ECN gets enabled on control packets w/o AccECN. As a result, I think this specification should be independent of AccECN. This may require some changes for the description of the SYN pricessing, e.g., moving the SYN handling to a separate document. Such a smaller, focused document may possibly depend on this experiment and AccECN. In a nutshell, I disagree with this section and the following content that creates a dependency between this document and AccECN.

   ECN++ is designed for compatibility with a number of latency
   improvements to TCP such as TCP Fast Open (TFO [RFC7413]), initial
   window of 10 SMSS (IW10 [RFC6928]) and Low latency Low Loss Scalable
   Transport (L4S [I-D.ietf-tsvwg-l4s-arch]), but they can all be
   implemented and deployed independently.

[ms] I don't understand why this text is then needed.

   [I-D.ietf-tsvwg-ecn-experimentation] is a standards track procedural
   device that relaxes requirements in RFC 3168 and other standards
   track RFCs that would otherwise preclude the experimental
   modifications needed for ECN++ and other ECN experiments.

[ms] I am not sure if such tutorial text is really needed.


* 1.1.  Motivation

   The absence of ECN support on TCP control packets and retransmissions
   has a potential harmful effect.  In any ECN deployment, non-ECN-
   capable packets suffer a penalty when they traverse a congested
   bottleneck.

[ms] I don't understand why this is true in "any" ECN deployment. To me, it is a network policy how to handle non-ECN capable packets, e.g., during congestion, and at least theoretically different options are possible. I would rephrase this e.g. as follows: "There is a risk that non-ECN-capable packets suffer a penalty...".

   Non-ECN control packets particularly harm performance in environments
   where the ECN marking level is high.  For example, [judd-nsdi] shows
   that in a controlled private data centre (DC) environment where ECN
   is used (in conjunction with DCTCP [RFC8257]), the probability of
   being able to establish a new connection using a non-ECN SYN packet
   drops to close to zero even when there are only 16 ongoing TCP flows
   transmitting at full speed.  The issue is that DCTCP exhibits a much
   more aggressive response to packet marking (which is why it is only
   applicable in controlled environments).  This leads to a high marking
   probability for ECN-capable packets, and in turn a high drop
   probability for non-ECN packets.  Therefore non-ECN SYNs are dropped
   aggressively, rendering it nearly impossible to establish a new
   connection in the presence of even mild traffic load.

[ms] I think this section describes a particular network deployment case, and not all assumptions are clearly spelt out. For instance, if the DCTCP traffic was assigned to another DiffServ class, it is not clear if the problem would be exactly the same. Instead of describing this specific case, I think the introduction could more generally emphasize the benefit of ECT in different cases, beyond DCTCP.

   Finally, there are ongoing experimental efforts to promote the
   adoption of a slightly modified variant of DCTCP (and similar
   congestion controls) over the Internet to achieve low latency, low
   loss and scalable throughput (L4S) for all communications
   [I-D.ietf-tsvwg-l4s-arch].  In such an approach, L4S packets identify
   themselves using an ECN codepoint [I-D.ietf-tsvwg-ecn-l4s-id].  With
   L4S and potentially other similar cases, preventing TCP control
   packets from obtaining the benefits of ECN would not only expose them
   to the prevailing level of congestion loss, but it would also
   classify control packets into a different queue with different
   network treatment, which may also lead to reordering, further
   degrading TCP performance.

[ms] It is understood that the L4S experiment may benefit from this I-D. For this, it would be sufficient to referent to this document from the L4S spec. We don't know the outcome of various experiments, and as a result I prefer that published RFCs are valid independent of other experiments.


* 1.2.  Experiment Goals

[ms] This section looks good, but I suggest to merge it with the "MEASURMENT NEEDED" discussion. Is it not possible to have a *single* list of what experiments and measurements are needed, in one place in the I-D?


* 1.3.  Document Structure

[ms] This section can hopefully be just removed. The fact that it is present shows that the document may still have room for editorial improvement, and, most notably, shortening.


* 2.  Terminology

   The keywords MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD,
   SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL, when they appear in this
   document, are to be interpreted as described in [RFC2119].

[ms] RFC 8174?

   ECT: ECN-Capable Transport.  One of the two codepoints ECT(0) or
   ECT(1) in the ECN field [RFC3168] of the IP header (v4 or v6).  An
   ECN-capable sender sets one of these to indicate that both transport
   end-points support ECN.  When this specification says the sender sets
   an ECT codepoint, by default it means ECT(0).  Optionally, it could
   mean ECT(1), which is in the process of being redefined for use by
   L4S experiments [I-D.ietf-tsvwg-ecn-experimentation]
   [I-D.ietf-tsvwg-ecn-l4s-id].

[ms] I don't understand why this has to be mentioned here. To me, it seems sufficient to reference related work at one single place in the document, in order to avoid dependencies.


* 3.  Specification

3.1.  Network (e.g.  Firewall) Behaviour

[ms] This could perhaps be moved to a dedicated section targeting switch/router/firewall/... implementers, who may not look for guidance in a section targeting TCP stack implementers. Just as a random thougt.

   This is likely to only involve
   a firewall rule change in a fraction of cases (at most 0.4% of paths
   according to the tests reported in Section 4.2.2).

[ms] This sort of data can easily get oudated and does not really fit into a normative section, IMHO.

* 3.2.  Endpoint Behaviour

   It can be seen that the sender can set ECT in all cases, except if it
   is not requesting AccECN feedback on the SYN.  Therefore it is
   RECOMMENDED that the experimental AccECN specification
   [I-D.ietf-tcpm-accurate-ecn] is implemented (as well as the present
   specification), because it is expected that ECT on the SYN will give
   the most significant performance gain, particularly for short flows.
   Nonetheless, this specification also caters for the case where AccECN
   feedback is not implemented.

[ms] As mentioned before, I would prefer that *this* document only discusses the case without AccECN. Possibly this requires that the SYN handling is moved to a separate document. To me, such a split would both be more future-proof and result in simpler documents. For instance, table 1 would be much simpler in this document. Such a change would impact quite a number of the following sections, but at first sight I don't see a reason that would prevent such a reorganization.


* 3.2.1.  SYN

[ms] As mentioned already, that whole section could move to another I-D. I am not sure if SYN would then be discussed in detail in this document. The statement in this I-D could perhaps be as simple as "Therefore, a TCP initiator MUST NOT set ECT on a SYN unless it also negotiates a mechanism for feedback to CE marks on SYNs. An example for such a feedback scheme is draft-ietf-tcpm-accurate-ecn. The specification can be found in draft-ietf-tcpm-foo."


* 3.2.1.1.  Setting ECT on the SYN

   With classic [RFC3168] ECN feedback, the SYN was never expected to be
   ECN-capable, so the flag provided to feed back congestion was put to
   another use (it is used in combination with other flags to indicate
   that the responder supports ECN).  In contrast, Accurate ECN (AccECN)
   feedback [I-D.ietf-tcpm-accurate-ecn] provides two codepoints in the
   SYN-ACK for the responder to feed back whether or not the SYN arrived
   marked CE.

   Therefore, a TCP initiator MUST NOT set ECT on a SYN unless it also
   attempts to negotiate Accurate ECN feedback in the same SYN.

[ms] Well, I don't think that Accurate ECN would be the only potential experiment to deal with CE marks on SYNs. For instance, one could design another feedback for CE marks in SYNs that leaves other parts of ECN as it is today. ECT-on-SYNs clearly requires experimentation, but that specific protocol aspect is to me by and large orthogonal to the other TCP control packets. Which is why I believe the spec depending on AccECN belongs into a separate I-D.


* 3.2.1.2.  Caching Lack of AccECN Support for ECT on SYNs

[ms] The whole caching discussion IMHO belongs into one (non-normative) place and is IMHO confusing so early in the document. The text can also be shortened IMHO.


* 3.2.1.3.  SYN Congestion Response

   If the SYN-ACK returned to the TCP initiator confirms that the server
   supports AccECN, it will also indicate whether or not the SYN was CE-
   marked.  If the SYN was CE-marked, the initiator MUST reduce its
   Initial Window (IW) and SHOULD reduce it to 1 SMSS (sender maximum
   segment size).

   If ECT has been set on the SYN and if the SYN-ACK shows that the
   server does not support AccECN, the TCP initiator MUST conservatively
   reduce its Initial Window and SHOULD reduce it to 1 SMSS.  A
   reduction to greater than 1 SMSS MAY be appropriate (see
   Section 4.2.1).  Conservatism is necessary because a non-AccECN SYN-
   ACK cannot show whether the SYN was CE-marked.

[ms] This congestion control algorithm may be one example where experimentation is needed prior to a proposed standard, and as congestion control evolves, it is possible that a PS would have different guidance. But this experiment would be quite independent from the TCP other control packets that raise much less questions on the congestion control interaction. To me, this is yet another reason to separate the experiments.

3.2.1.4.  Fall-Back Following No Response to an ECT SYN

   An ECT SYN might be lost due to an over-zealous path element (or
   server) blocking ECT packets that do not conform to RFC 3168.  Some
   evidence of this was found in a 2014 study [ecn-pam], but in a more
   recent study using 2017 data [Mandalari18] extensive measurements
   found no case where ECT on TCP control packets was treated any
   differently from ECT on TCP data packets.  Loss is commonplace for
   numerous other reasons, e.g. congestion loss at a non-ECN queue on
   the forward or reverse path, transmission errors, etc.

[ms] Such data gets easily outdated. IMHO does not belong into normative sections of an RFC (the same comment applies also elsewhere). Typically, measurements can be discussed in an appendix.

   Other fall-back strategies MAY be adopted where applicable (see
   Section 4.2.2 for suggestions, and the conditions under which they
   would apply).

[ms] Given that, a lot of content seems to be not normative and can probably be moved elsewhere (or even be removed).


* 3.2.2.1.  Setting ECT on the SYN-ACK

   Some classic ECN implementations might ignore a CE-mark on a SYN-ACK,
   or even ignore a SYN-ACK packet entirely if it is set to ECT or CE.
   This is a possibility because an RFC 3168 implementation would not
   necessarily expect a SYN-ACK to be ECN-capable.

      FOR DISCUSSION: To eliminate this problem, the WG could decide to
      prohibit setting ECT on SYN-ACKs unless AccECN has been
      negotiated.  However, this issue already came up when the IETF
      first decided to experiment with ECN on SYN-ACKs [RFC5562] and it
      was decided to go ahead without any extra precautionary measures

[ms] As there seem to be widely deployed "congestion control" algorithms that decide to ignore loss, I am not too worried about some cases where a TCP endpoint may ignore a single CE-mark. I may miss something, but I don't see how that could really result in congestion collapse or significant unfairness in today's Internet. A router that is really congested will drop *many* packets and thereby trigger the congestion control. Of course, I am here assuming that any transport protocol indeed reacts to loss by reducing the load immediately.

* 3.2.2.2.  SYN-ACK Congestion Response

   A host that sets ECT on SYN-ACKs MUST reduce its initial window in
   response to any congestion feedback, whether using classic ECN or
   AccECN.  It SHOULD reduce it to 1 SMSS.
[ms] I think this document can be simplified by avoiding this sort distinction wherever there is no difference.

   This is different to the behaviour specified in an earlier experiment that set ECT on the SYN-
   ACK [RFC5562].  This is justified in Section 4.3.

[ms] If RFC 5562 is obsoleted, I think a better wording would be of the form "This experiments updates the behavior defined in RFC 5562 as follow: ..."


* 3.2.2.3.  Fall-Back Following No Response to an ECT SYN-ACK

   This fall-back strategy attempts to use ECT one more time than the
   strategy for ECT SYN-ACKs in [RFC5562] (which is made obsolete, being
   superseded by the present specification).

[ms] I suggest to simplify such text by explicitly stating how RFC 5562 is updated.

   Other fall-back strategies
   MAY be adopted if found to be more effective, e.g. fall-back to not-
   ECT on the first retransmission attempt.

[ms] In general, implementation guidance e.g. on heuristics could IMHO be moved to one place in the document, outside the normative part.


* 3.2.3.  Pure ACK

   For the experiments proposed here, the TCP implementation will set
   ECT on pure ACKs.  It can ignore the requirement in section 6.1.4 of
   RFC 3168 to set not-ECT on a pure ACK.

   A host that sets ECT on pure ACKs MUST reduce its congestion window
   in response to any congestion feedback, in order to regulate any data
   segments it might be sending amongst the pure ACKs. {ToDo: Write-up
   reconsideration of this requirement in the light of WG comments.} It
   MAY also implement AckCC [RFC5690] to regulate the pure ACK rate, but
   this is not required.  Note that, in comparison, TCP Congestion
   Control [RFC5681] does not require a TCP to detect or respond to loss
   of pure ACKs at all; it requires no reduction in congestion window or
   ACK rate.

   The question of whether the receiver of pure ACKs is required to feed
   back any CE marks on them is a matter for the relevant feedback
   specification ([RFC3168] or [I-D.ietf-tcpm-accurate-ecn]).  It is
   outside the scope of the present specification.  Currently AccECN
   feedback is required to count CE marking of any control packet
   including pure ACKs.  Whereas RFC 3168 is silent on this point, so
   feedback of CE-markings might be implementation specific (see
   Section 4.4.1).

      DISCUSSION: An AccECN deployment or an implementation of RFC 3168
      that feeds back CE on pure ACKs will be at a disadvantage compared
      to an RFC 3168 implementation that does not.  To solve this, the
      WG could decide to prohibit setting ECT on pure ACKs unless AccECN
      has been negotiated.  If it does, the penultimate sentence of the
      Introduction will need to be modified.

[ms] For what it is worth, I would personally go for a more liberal specification, albeit I may here be on the rough side of the consensus. Will the world end if a TCP sender ignores a CE mark on a pure ACK? We somehow know that ignoring ACK loss in congestion control will not immediately cause congestion collapse in the Internet... So in which case is this really a relevant problem? But, sure, I may miss something here...


* 3.2.8.  General Fall-back for any Control Packet or Retransmission

[ms] I don't understand why this content is needed in a normative section. (And I also struggle with understanding the need for Section 4.9.)


* 4.  Rationale

[ms] This is a very long section with limited added value to implementers. An alternative would be to move the arguments to an appendix, and to clearly separate content that would matter to implementers.

* 4.1.  The Reliability Argument

[ms] I somehow miss in this section (and following ones) the question whether a sender really has to react to a CE mark in the same way like to a loss. There is ongoing experimentation in this space (draft-ietf-tcpm-alternativebackoff-ecn). Of course, we don't know the outcome of that other experiment, and I certainly don't want to create a dependency. Anyway, my high-level feeling is that for an experiment, this document is pretty convervative, as compared to what the Internet is as of today. But this is just my observation, this comment does not ask for any text change.

* 4.2.  SYNs

[ms] To me, this discussion is too much tied into AccECN. As mentioned before, for example, if we defined another way to feed back CE marks on SYNs in future, would this section still apply?

* 4.2.1.  Argument 1a: Unrecognized CE on the SYN

[ms] I have quite some doubts on this section and later subsections, but it may be better to sort out the high-level questions on SYNs first. As an editorial nit, I am not sure if the explanation for "S3" should dig into different data center designs.


* 5.  Interaction with popular variants or derivatives of TCP

   The following subsections discuss any interactions between setting
   ECT on all packets and using the following popular variants of TCP:
   IW10 and TFO.  It also briefly notes the possibility that the
   principles applied here should translate to protocols derived from
   TCP.  This section is informative not normative, because no
   interactions have been identified that require any change to
   specifications.  The subsection on IW10 discusses potential changes
   to specifications but recommends that no changes are needed.

   The designs of the following TCP variants have also been assessed and
   found not to interact adversely with ECT on TCP control packets: SYN
   cookies (see Appendix A of [RFC4987] and section 3.1 of [RFC5562]),
   TCP Fast Open (TFO [RFC7413]) and L4S [I-D.ietf-tsvwg-l4s-arch].

[ms] These two paragraphs can IMHO be removed without any loss of information.


* 5.1.  IW10

   If the initiator implements IW10, it seems rather over-conservative
   to reduce IW from 10 to 1 just in case a congestion marking was
   missed.  Nonetheless, the reduction to 1 SMSS will rarely harm
   performance, because:

   o  as long as the initiator is caching failures to negotiate AccECN,
      subsequent attempts to access the same server will not use ECT on
      the SYN anyway, so there will no longer be any need to
      conservatively reduce IW;

   o  currently it is not common for a TCP initiator (client) to have
      more than one data segment to send {ToDo: evidence/reference?} -
      IW10 is primarily exploited by TCP servers.

[ms] As IW10 seems pretty widely deployed, I wonder if that statement is indeed true for the broad set of use cases of TCP, most notably outside the WWW. I don't know if such text will indeed convince implementers. For sure we can publish whatever requirement we want in an RFC, but I see the risk that implementers will consider this guidance overly restrictive and e.g. just ignore this wording entirely, if it just focuses e.g. on the WWW.

   If a responder receives feedback that the SYN-ACK was CE-marked,
   Section 3.2.2.2 mandates that it reduces its initial window to 1
   SMSS.  When the responder also implements IW10, it is particularly
   important to adhere to this requirement in order to avoid overflowing
   a queue that is clearly already congested.

[ms] If the queue was "clearly already suggested", it might drop packets instead of marking them. I think it somehow depends on the AQM whether one can conclude from an ECN mark to a risk of "overflowing a queue". Instead of just stating normative limits, I think it could be valuable to implementers to explain that if the initial window is not reduced, there will be a significant risk of getting packet drops in that first window, which the corresponding impact on performance. Reacting to the CE mark reduces this risk and may thus result in better performance for this connection.


* 5.3.  TCP Derivatives

[ms] I think this can be removed


* 6.  Security Considerations

[ms] Aren't there privacy implications, e.g., for fingerprinting?