Re: [PANRG] New ECN section for

[Spencer Dawkins at IETF <[e-mail]>]

So, Martin had one question about
https://github.com/panrg/draft-dawkins-panrg-what-not-to-do/pull/6, and
that was the very reasonable

"You get one chance to get it right" may be true, but it seems like a
shallow recommendation. What could the ECN proponents have done
differently? Would wider testing have uncovered this? More involvement of
router vendors in standards development and interop?


I think that this is a really important question, although the answer might
be better placed in the "what TO do" document we've talked about producing
after "what NOT to do" was finished. But just to get us thinking about that
...

Martin asked about "more testing", but ECN went through a fair amount of
testing - the original proposal in IETF was produced as Experimental,
before advancing to Proposed Standard, which was once pretty common for TSV
documents that touched congestion avoidance behavior, and Sally Floyd's ECN
page contains a lot of information about simulations and testing results.
It's not obvious that the dreaded router described in {vista-impl],

      Intermediate Gateway Device problem #1: one of the most popular
      versions from one of the most popular vendors.  When a data packet
      arrives with either ECT(0) or ECT(1) (indicating successful ECN
      capability negotiation) indicated, router crashed.  Cannot be
      recovered at TCP layer

even existed when ECN was being tested (we could chase more details to find
out, but that doesn't help us with future protocols).

Martin also asked about "more involvement of router vendors in standards
development and interop". Again, we could chase details, but my experience
has been that

   - router vendors are some of the most consistent participants in
   standards development, and
   - the IETF is one of the most cross-area-review-oriented SDOs we know of.

Although (again, I don't know the details), my experience has been that
"intermediate device vendors" may not have much of a plan for upgrades
(check your home gateway device for a worked example). So maybe that would
have helped.

What I'm thinking of, in addition to Martin's questions, are these things.

   - ECN may be a FABULOUS example of what happens when protocol designers
   didn't manage to "grease" unused fields in their protocols ("whatever we
   thought the TOS bits were in those positions meant, they'll always be zero,
   until they weren't"). I'm referencing "use it or lose it" as
   https://datatracker.ietf.org/doc/draft-thomson-quic-bit-grease/, since
   that seems most current.
   - It may be that we get more chances to get something right if our
   protocols are deployed on devices that know how to install updates without
   human intervention (Windows 10 yes, most IoT devices no).

Other thoughts?

Best,

Spencer

On Sun, Feb 14, 2021 at 2:02 PM Spencer Dawkins at IETF <
[e-mail]> wrote:

> Dear PANRG.
>
> I have https://github.com/panrg/draft-dawkins-panrg-what-not-to-do/pull/6
> in Github, to add this material, as we've talked about on the mailing list.
>
> I would love to have comments from anyone familiar with the history of
> ECN, or its current practice.
>
> If you don't Github (there may be a few), here's what's new:
>
> I added this material to Section 4,
>
> 4.13.  Ability to Recover From Missteps
>
>    If early implementers discover problems with a new feature, that
>    feature is likely to be disabled, and convincing implementers to re-
>    enable that feature can be very difficult, and can require years or
>    decades.  (See Section 6.9).
>
> I asserted that "One Chance" was invariant, in Table 1, as
>
>     +-----------------------------------------------------+-----------+
>     | One Chance to Achieve Deployment (Section 4.13)     | Invariant |
>     +-----------------------------------------------------+-----------+
>
> I added this material to Section 6.
>
> 6.9.  Explicit Congestion Notification (ECN)
>
>    The suggested references for Explicit Congestion Notification (ECN)
>    are:
>
>    *  Recommendations on Queue Management and Congestion Avoidance in
>       the Internet [RFC2309]
>
>    *  A Proposal to add Explicit Congestion Notification (ECN) to IP
>       [RFC2481]
>
>    *  The Addition of Explicit Congestion Notification (ECN) to IP
>       [RFC3168]
>
>    *  Implementation Report on Experiences with Various TCP RFCs
>       [vista-impl], slides 6 and 7
>
>    *  Implementation and Deployment of ECN [SallyFloyd]
>
>    In the early 1990s, the large majority of Internet traffic used TCP
>    as its transport protocol, but TCP had no way to detect path
>    congestion before the path was so congested that packets were being
>    dropped, and these congestion events could affect all senders using a
>    path, either by "lockout", where long-lived flows monopolized the
>    queues along a path, or by "full queues", where queues remain full,
>    or almost full, for a long period of time.
>
>    In response to this situation, "Active Queue Management" (AQM) was
>    deployed in the network.  A number of AQM disciplines have been
>    deployed, but one common approach was that routers dropped packets
>    when a threshold buffer length was reached, so that transport
>    protocols like TCP that were responsive to loss would detect this
>    loss and reduce their sending rates.  Random Early Detection (RED)
>    was one such proposal in the IETF.  As the name suggests, a router
>    using RED as its AQM discipline that detected time-averaged queue
>    lengths passing a threshold would choose incoming packets
>    probablistically to be dropped [RFC2309].
>
>    Researchers suggested that providing "explicit congestion
>    notifications" to senders when routers along the path detected their
>    queues were building, giving them an indication that router queues
>    along the path were building, so that some senders would "slow down"
>    as if a loss had occurred, so that the path queues had time to drain,
>    and the path still had sufficient buffer capacity to accommodate
>    bursty arrivals of packets from other senders.  This was proposed as
>    an Experiment in [RFC2481], and standardized in [RFC3168].
>
>    A key aspect of ECN was the use of IP header fields rather than IP
>    options to carry explicit congestion notifications, since the
>    proponents recognized that
>
>       Many routers process the "regular" headers in IP packets more
>       efficiently than they process the header information in IP
>       options.
>
> 6.9.1.  Reasons for Non-deployment
>
>    The proponents of ECN did so much right, anticipating many of the
>    Lessons Learned now recognized in Section 4.  They recognized the
>    need to support incremental deployment (Section 4.2).  They
>    considered the impact on router throughput (Section 4.8).  They even
>    considered trust issues between end nodes and the network, both for
>    non-compliant end nodes (Section 4.10) and non-compliant routers
>    (Section 4.9).
>
>    They were rewarded with ECN being implemented in major operating
>    systems, both for end nodes and for routers.  A number of
>    implementations are listed under "Implementation and Deployment of
>    ECN" at [SallyFloyd].
>
>    What they did not anticipate, was routers that would crash, when they
>    saw bits 6 and 7 in the IPv4 TOS octet [RFC0791]/IPv6 Traffic Class
>    field [RFC2460], which [RFC2481] redefined to be "currently unused",
>    being set to a non-zero value.
>
>    As described in [vista-impl],
>
>       Intermediate Gateway Device problem #1: one of the most popular
>       versions from one of the most popular vendors.  When a data packet
>       arrives with either ECT(0) or ECT(1) (indicating successful ECN
>       capability negotiation) indicated, router crashed.  Cannot be
>       recovered at TCP layer
>
>    This implementation, which would be run on a significant percentage
>    of Internet end nodes, was shipped with ECN disabled, as was true for
>    several of the other implementations listed under "Implementation and
>    Deployment of ECN" at [SallyFloyd].  Even if subsequent router
>    vendors fixed these implementations, ECN was still disabled on end
>    nodes, and given the tradeoff between the benefits of enabling ECN
>    (somewhat better behavior during congestion) and the risks of
>    enabling ECN (possibly crashing a router somewhere along the path),
>    ECN tended to stay disabled on implementations that supported ECN for
>    decades afterwards.
>
> 6.9.2.  Lessons Learned
>
>    Of the contributions included in Section 6, ECN may be unique in
>    providing these lessons:
>
>    *  Even if you do everything right, you may trip over implementation
>       bugs in devices you know nothing about, that will cause severe
>       problems that prevent successful deployment of your path aware
>       technology.
>
>    *  After implementations disable your path aware technology, it may
>       take years, or even decades, to convince implementers to re-enable
>       it by default.
>
>    These two lessons, taken together, could be summarized as "you get
>    one chance to get it right".
>
> Best,
>
> Spencer
>