Re: [tcpm] Does this fix sufficiently de-ossify overstrict ECN negotiation by Linux TCP servers?

["Brian Trammell (IETF)" <[e-mail]>]

Hi Bob,

This seems eminently reasonable. I’m curious, would there be equivalent pushback on _only_ adding the SNMP counter? One could always use observation to debug once the counter starts going up...

Cheers,

Brian

Sent from my iPhone

> On 16 Mar 2019, at 12:38, Bob Briscoe <[e-mail]> wrote:
> 
> Folks,
> 
> After offline discussion with the Linux maintainers, we have agreed on option #2, which is to make overstrict ECN test more specific to RFC3168 by only disabling ECN if a SYN has a non-zero IP-ECN field and the first six TCP flags in the main header are 0b000011.
> 
> This will be a simple 1-line patch that maximizes the likelihood that it will be back-ported as far back as possible, hopefully back to the introduction of the over-strict test in May 2012. This should fix a large proportion of the installed base of Linux servers reasonably quickly.
> 
> ==Logging==
> Separately we will submit a patch to add logging of each event that disables ECN, but only for future kernel versions, not for back-porting. However, there is resistance to any logging, so this might not be accepted. The proposal is 
> * at default logging level, increment an SNMP count when ECN disabled for a connection due to a protocol violation
> * at increased logging level, on each event also write rate-limited kernel messages that record the client IP.
> 
> Then, if a sysadmin wants to, they can get any network mangling fixed.
> 
> 
> Bob
> 
>> On 20/01/2019 23:31, Bob Briscoe wrote:
>> TCPM folks,
>> 
>> RFC8311 and draft-ietf-tcpm-generalized-ecn enable ECN-capable SYNs. However, if a client sends an ECN-capable SYN, existing Linux servers disable ECN for the rest of the connection.
>> 
>> To fix as many existing Linux servers as possible, as soon as possible, I would recommend short-term solution #2 (see 'Three Fixes' below). An alternative would be short-term solution #3.
>> 
>> Anyone got a better fix? Anyone disagree with recommending #2? If not, pls confirm your agreement.
>> 
>> Summary of Pros and Cons
>> 
>> #2 pro: No longer ossifies non-ECN SYN
>> #2 pro: If a path does "unbleach" IP-ECN, protects the connection by disabling ECN
>> #2 con: A residual degree of ossification remains
>> 
>> #3 pro: No longer ossifies non-ECN SYN
>> #3 pro: No ossification at all
>> #3 con: If a path does "unbleach" IP-ECN, connection still attempts to use ECN even though the path is mangling the ECN field.
>> 
>> #1 is the full solution, but it should proceed in parallel as a solution for the future, not to fix the past.
>> Background
>> 
>> To negotiate ECN for a connection the client sets flag bits 8-9 in the TCP header to 11 [RFC3168 Section 6.1.1]. For your convenience, I've included a picture of the TCP header flags at the end.
>> 
>> In 2012 an additional test was patched into Linux servers. It disables ECN for the connection unless the IP-ECN field of the SYN is also zero (I'll call this the "overstrict ECN test").
>> 
>> This was because Section 6.1.1. of RFC3168 also said "A host MUST NOT set ECT on SYN or SYN-ACK packets." Unfortunately, the overstrict ECN test ossifies this requirement into TCP. 
>> 
>> The Linux patch was intended to protect a connection against the network mangling the IP-ECN field to a non-zero value. Unfortunately, it also disables ECN completely if a client tries to set ECT in the IP-ECN field of the SYN.
>> 
>> RFC8311 has now updated RFC3168 to allow ECT on a SYN, in conjunction with the ECN++ experiment. But few developers are going to set ECT on a SYN if it causes about 84% of servers to completely disable ECN (see Section 4.2.2. of the ECN++ draft for further details and the source of these numbers).
>> Three Fixes
>> 
>> I can think of three ways to fix existing Linux servers:
>> Deploy Accurate ECN on existing Linux servers.
>> AccECN inherently replaces the overstrict ECN test with a pair of proper two-ended tests for network mangling, in which each end feeds back the arriving IP ECN field in its TCP feedback. In contrast, the overstrict ECN test is one-ended, with the server only assuming what the client set.
>> Altho AccECN will solve the problem, it should not be relied on to fix the last six years of Linux server deployment. AccECN is a non-trivial update so it is unlikely to get back-ported and auto-deployed to most existing Linux servers any time soon. 
>> Make the overstrict ECN test more specific to RFC3168, and add logging.
>> Currently, the only proposal for allowing ECT on a SYN requires the client to also request Accurate ECN feedback, which requires TCP header bits 7-9 to be 111.
>> Therefore, the overstrict ECN test in a Linux server could be altered to solely apply if bits 7-9 == 011 (rather than just testing if bits 8-9 == 11).
>> However, I am concerned that this ossifies around the solution based on ECN++ and AccECN. What if ECN++ needs to be superseded by some future protocol that allows ECT on a SYN when some other field is different (e.g. one of TCP's three remaining reserved flags, or a new TCP option)? 
>> Unfortunately, we cannot test for the absence of a yet-to-be-invented TCP option. Nonetheless, I suggest the overstrict ECN test in a Linux server should at least be altered to solely apply if TCP header bits 4-9 == 000011. 
>> I suggest Linux also logs whenever it disables ECN due to this test, so that broken paths can be identified and fixed.
>> An extremely simple patch like this would be much more likely than #1 (AccECN) to be back-ported and deployed to most existing servers via regular Linux auto-updates.
>> As #2, but do not disable ECN when the test fails, just log those paths suspected of mangling ECN.
>> Our most recent measurements of millions of Internet paths found no occurrences where zero IP-ECN on a SYN was changed to non-zero ("unbleaching"). So we could just remove the test.
>> However, no occurrences found does not mean none exist. Altho we tested millions of paths, we only tested from a few dozen vantage points, albeit we did test both mobile and fixed networks (see Section 4.2.2. of the ECN++ draft for summary of tests).
>> If the number of problematic paths is small, it would still make sense to run the ECN test as proposed under item #2, but not disable ECN if it fails. Instead, just log any negative results, so the broken network elements can be identified and fixed.
>> Cheers
>> 
>> 
>> Bob
>> 
>> PS. For convenience, here's the TCP header flags (assuming the experimental AccECN protocol is approved as RFC):
>>        0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15
>>      +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
>>      |               |           | A | C | E | U | A | P | R | S | F |
>>      | Header Length | Reserved  | E | W | C | R | C | S | S | Y | I |
>>      |               |           |   | R | E | G | K | H | T | N | N |
>>      +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
>> 
>> 
>> -- 
>> ________________________________________________________________
>> Bob Briscoe                               http://bobbriscoe.net/
> 
> -- 
> ________________________________________________________________
> Bob Briscoe                               http://bobbriscoe.net/
> _______________________________________________
> tcpm mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/tcpm