Re: [tcpm] Does this fix sufficiently de-ossify overstrict ECN negotiation by Linux TCP servers?

[Bob Briscoe <[e-mail]>]

Brian,

Looks like adding only an SNMP counter could be acceptable. We'll submit 
a patch for net-next on that basis (as well as a patch without logging 
for net-stable).


Bob

On 17/03/2019 02:05, Bob Briscoe wrote:
> Brian,
> I've asked - I'll let you know once I hear.
>
> Bob
>
> On 16/03/2019 14:32, Brian Trammell (IETF) wrote:
>> Hi Bob,
>>
>> This seems eminently reasonable. I’m curious, would there be 
>> equivalent pushback on _only_ adding the SNMP counter? One could 
>> always use observation to debug once the counter starts going up...
>>
>> Cheers,
>>
>> Brian
>>
>> Sent from my iPhone
>>
>> On 16 Mar 2019, at 12:38, Bob Briscoe <[e-mail] 
>> <mailto:[e-mail]>> wrote:
>>
>>> Folks,
>>>
>>> After offline discussion with the Linux maintainers, we have agreed 
>>> on option #2, which is to make overstrict ECN test more specific to 
>>> RFC3168 by only disabling ECN if a SYN has a non-zero IP-ECN field 
>>> and the first six TCP flags in the main header are 0b000011.
>>>
>>> This will be a simple 1-line patch that maximizes the likelihood 
>>> that it will be back-ported as far back as possible, hopefully back 
>>> to the introduction of the over-strict test in May 2012. This should 
>>> fix a large proportion of the installed base of Linux servers 
>>> reasonably quickly.
>>>
>>> ==Logging==
>>> Separately we will submit a patch to add logging of each event that 
>>> disables ECN, but only for future kernel versions, not for 
>>> back-porting. However, there is resistance to any logging, so this 
>>> might not be accepted. The proposal is
>>> * at default logging level, increment an SNMP count when ECN 
>>> disabled for a connection due to a protocol violation
>>> * at increased logging level, on each event also write rate-limited 
>>> kernel messages that record the client IP.
>>>
>>> Then, if a sysadmin wants to, they can get any network mangling fixed.
>>>
>>>
>>> Bob
>>>
>>> On 20/01/2019 23:31, Bob Briscoe wrote:
>>>> TCPM folks,
>>>>
>>>> RFC8311 and draft-ietf-tcpm-generalized-ecn enable ECN-capable 
>>>> SYNs. However, if a client sends an ECN-capable SYN, existing Linux 
>>>> servers disable ECN for the rest of the connection.
>>>>
>>>> To fix as many existing Linux servers as possible, as soon as 
>>>> possible, I would recommend short-term solution #2 (see 'Three 
>>>> Fixes' below). An alternative would be short-term solution #3.
>>>>
>>>> Anyone got a better fix? Anyone disagree with recommending #2? If 
>>>> not, pls confirm your agreement.
>>>>
>>>>
>>>>       Summary of Pros and Cons
>>>>
>>>> #2 pro: No longer ossifies non-ECN SYN
>>>> #2 pro: If a path does "unbleach" IP-ECN, protects the connection 
>>>> by disabling ECN
>>>> #2 con: A residual degree of ossification remains
>>>>
>>>> #3 pro: No longer ossifies non-ECN SYN
>>>> #3 pro: No ossification at all
>>>> #3 con: If a path does "unbleach" IP-ECN, connection still attempts 
>>>> to use ECN even though the path is mangling the ECN field.
>>>>
>>>> #1 is the full solution, but it should proceed in parallel as a 
>>>> solution for the future, not to fix the past.
>>>>
>>>>
>>>>       Background
>>>>
>>>> To negotiate ECN for a connection the client sets flag bits 8-9 in 
>>>> the TCP header to 11 [RFC3168 Section 6.1.1] 
>>>> <https://tools.ietf.org/html/rfc3168#section-6.1.1>. For your 
>>>> convenience, I've included a picture of the TCP header flags at the 
>>>> end.
>>>>
>>>> In 2012 an additional test was patched into Linux servers. It 
>>>> disables ECN for the connection unless the IP-ECN field of the SYN 
>>>> is also zero (I'll call this the "overstrict ECN test").
>>>>
>>>> This was because Section 6.1.1. of RFC3168 also said "A host MUST 
>>>> NOT set ECT on SYN or SYN-ACK packets." Unfortunately, the 
>>>> overstrict ECN test ossifies this requirement into TCP.
>>>>
>>>> The Linux patch was intended to protect a connection against the 
>>>> network mangling the IP-ECN field to a non-zero value. 
>>>> Unfortunately, it also disables ECN completely if a client tries to 
>>>> set ECT in the IP-ECN field of the SYN.
>>>>
>>>> RFC8311 has now updated RFC3168 to allow ECT on a SYN, in 
>>>> conjunction with the ECN++ experiment. But few developers are going 
>>>> to set ECT on a SYN if it causes about 84% of servers to completely 
>>>> disable ECN (see Section 4.2.2. of the ECN++ draft 
>>>> <https://tools.ietf.org/html/draft-ietf-tcpm-generalized-ecn-03#section-4.2.2.2> 
>>>> for further details and the source of these numbers).
>>>>
>>>>
>>>>       Three Fixes
>>>>
>>>> I can think of three ways to fix existing Linux servers:
>>>>
>>>>  1. Deploy Accurate ECN
>>>>     <https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn> on
>>>>     existing Linux servers.
>>>>       * AccECN inherently replaces the overstrict ECN test with a
>>>>         pair of proper two-ended tests for network mangling, in
>>>>         which each end feeds back the arriving IP ECN field in its
>>>>         TCP feedback. In contrast, the overstrict ECN test is
>>>>         one-ended, with the server only assuming what the client set.
>>>>       * Altho AccECN will solve the problem, it should not be
>>>>         relied on to fix the last six years of Linux server
>>>>         deployment. AccECN is a non-trivial update so it is
>>>>         unlikely to get back-ported and auto-deployed to most
>>>>         existing Linux servers any time soon.
>>>>  2. Make the overstrict ECN test more specific to RFC3168, and add
>>>>     logging.
>>>>       * Currently, the only proposal for allowing ECT on a SYN
>>>>         requires the client to also request Accurate ECN feedback,
>>>>         which requires TCP header bits 7-9 to be 111.
>>>>       * Therefore, the overstrict ECN test in a Linux server could
>>>>         be altered to solely apply if bits 7-9 == 011 (rather than
>>>>         just testing if bits 8-9 == 11).
>>>>       * However, I am concerned that this ossifies around the
>>>>         solution based on ECN++ and AccECN. What if ECN++ needs to
>>>>         be superseded by some future protocol that allows ECT on a
>>>>         SYN when some other field is different (e.g. one of TCP's
>>>>         three remaining reserved flags, or a new TCP option)?
>>>>       * Unfortunately, we cannot test for the absence of a
>>>>         yet-to-be-invented TCP option. Nonetheless, I suggest *the
>>>>         overstrict ECN test in a Linux server should at least be
>>>>         altered to solely apply if TCP header bits 4-9 == 000011*.
>>>>       * I suggest Linux also *logs whenever it disables ECN due to
>>>>         this test*, so that broken paths can be identified and fixed.
>>>>       * An extremely simple patch like this would be much more
>>>>         likely than #1 (AccECN) to be back-ported and deployed to
>>>>         most existing servers via regular Linux auto-updates.
>>>>  3. As #2, but do not disable ECN when the test fails, just log
>>>>     those paths suspected of mangling ECN.
>>>>       * Our most recent measurements of millions of Internet paths
>>>>         found no occurrences where zero IP-ECN on a SYN was changed
>>>>         to non-zero ("unbleaching"). So we could just remove the test.
>>>>       * However, no occurrences found does not mean none exist.
>>>>         Altho we tested millions of paths, we only tested from a
>>>>         few dozen vantage points, albeit we did test both mobile
>>>>         and fixed networks (see Section 4.2.2. of the ECN++ draft
>>>>         <https://tools.ietf.org/html/draft-ietf-tcpm-generalized-ecn-03#section-4.2.2.2>
>>>>         for summary of tests).
>>>>       * If the number of problematic paths is small, it would still
>>>>         make sense to run the ECN test as proposed under item #2,
>>>>         but not disable ECN if it fails. Instead, just log any
>>>>         negative results, so the broken network elements can be
>>>>         identified and fixed.
>>>>
>>>> Cheers
>>>>
>>>>
>>>> Bob
>>>>
>>>> PS. For convenience, here's the TCP header flags (assuming the 
>>>> experimental AccECN protocol is approved as RFC):
>>>>         0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15
>>>>       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
>>>>       |               |           | A | C | E | U | A | P | R | S | F |
>>>>       | Header Length | Reserved  | E | W | C | R | C | S | S | Y | I |
>>>>       |               |           |   | R | E | G | K | H | T | N | N |
>>>>       +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
>>>>
>>>>
>>>> -- 
>>>> ________________________________________________________________
>>>> Bob Briscoehttp://bobbriscoe.net/
>>>
>>> -- 
>>> ________________________________________________________________
>>> Bob Briscoehttp://bobbriscoe.net/
>>> _______________________________________________
>>> tcpm mailing list
>>> [e-mail] <mailto:[e-mail]>
>>> https://www.ietf.org/mailman/listinfo/tcpm
>
> -- 
> ________________________________________________________________
> Bob Briscoehttp://bobbriscoe.net/

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/