Re: [tcpm] Asymmetric Middleboxes and Extended Data Offset (EDO)

[Joe Touch <[e-mail]>]

Hi, Bob and Wes (et al.),

First, IMO "option-removing" middleboxes ought to be considered broken
and fixed. Let's not treat this otherwise on that end.

Second, it's trivial to protect against that event by simply:

	a) not using the EDO space until after the SYN/ACK
	b) requiring it on every segment once enabled

(b) makes EDO robust to erroneous middleboxes, regardless of asymmetry.

(a) is required to differentiate asymmetric errors like this and
legitimate legacy servers.

FWIW, I don't think we need to do anything in the final ACK of the TWHS.
We treat it like any other segment - once the option makes it in both
directions, removal of the option effectively kills the packet (and
terminates the connection).

IF this is a significant concern, we can add it to EDO trivially, but
I'd like to know how prevalent this is and whether it can be corrected
by fixing the error first.

However, I don't quite see a way around avoiding SYN/ACK use of any
extension unless we pull the same trick with SYN/ACKs we use with SYNs
(OOB segment or dual-SYN/ACK).

Joe

On 10/2/2014 7:31 AM, Bob Briscoe wrote:
> Wes,
> 
> I agree that combining EDO with generic middlebox hardening would be
> nice. However, it's hard to provide a general-purpose middlebox
> hardening solution for all cases; most generic solutions introduce
> overhead - complexity, processing or extra round-trip latency - and each
> case has different sensitivities to different types of overhead.
> 
> For instance, in the design of MPTCP, there's a 3-way check to protect
> against asymmetric option removal
> <http://tools.ietf.org/html/rfc6824#section-2.1>. It's MPTCP-specific
> because MPTCP is perceived as more relevant to longer flows, so the
> extra start-up latency of this 3-way check is less of a concern. Also,
> if EDO used a 3-way check, it would not be able to extend the option
> space on any of the 3WHS segments, which would reduce its usefulness.
> 
> Section 6 of the above MPTCP RFC
> <http://tools.ietf.org/html/rfc6824#section-6> discusses asymmetric
> option removal (see Fig 16) and how MPTCP was hardened against it. It is
> also a useful enumeration of other possible middlebox problems (and how
> MPTCP was designed to work round them and to work with them).
> 
> 
> ==Segment Coalescing==
> 
> A colleague (Phil Eardley) has just pointed out that segment coalescing
> could be a problem for EDO. I suspect segments in the 3WHS won't get
> coalesced because there's nothing to coalesce with. But then later
> segments might be coalesced.
> 
> If this were the case, it would mean even a 3-way EDO capability
> negotiation would appear to have worked. But then subsequent segments
> could end up with a corrupted coalesced payload, with TCP options
> embedded at regular intervals throughout.
> 
> Regards
> 
> 
> Bob
> 
> At 14:29 02/10/2014, Eddy, Wesley M. (GRC-LCI0)[MTI SYSTEMS, INC.] wrote:
>> Hi Bob,  I think there's a generic problem that many options share of
>> detecting middlebox compatibility issues, and that for some subset of
>> them (EDO included) those issues could cause corruption of user data.
>>
>> This motivates stronger ways to validate segments (e.g. the form of
>> TCP-AO that's compatible with NAT, with the TCP option flag set ...
>> possibly with some new unkeyed algorithm).  It should be done in
>> common and not re-invented on a per-option basis IMHO.
>>
>>
>>
>> -----Original Message-----
>> From: Bob Briscoe [mailto:[e-mail]]
>> Sent: Thursday, October 02, 2014 7:50 AM
>> To: Joe Touch; Eddy, Wesley M. (GRC-LCI0)[MTI SYSTEMS, INC.]
>> Cc: tcpm IETF list
>> Subject: Asymmetric Middleboxes and Extended Data Offset (EDO)
>>
>> Joe, Wes,
>>
>> It is known that new TCP options are removed on x% of paths.
>> It is also known that y% of paths are asymmteric Let's assume z% of
>> the TCP-option-removing middleboxes are on the asymmetric part of a path.
>>
>> (x y and z are left as an exercise for the experimenter. I'm offline
>> at the mo, but I know there are papers that give good estimates of x
>> and y at least).
>>
>> Also some middleboxes behave asymmetrically, for instance firewalls
>> have stricter rules for traffic arriving from the public side vs the
>> private side. Such a device might remove unknown TCP options in one
>> direction, but not the other.
>>
>> ==Legend==
>> C               = TCP client
>> S               = TCP server
>> EDO             = EDO delivered
>> !X-EDO-X!       = Middlebox removes EDO TCP option
>> Extra-opts      = TCP options beyond the Data Offset (preserved)
>> ???!!!  = confused or crashes
>>
>> ==Scenarios==
>>
>> 1. EDO not removed from SYN, but middlebox removes EDO from SYN/ACK
>> 1.1 Extra Options on SYN/ACK
>>
>>    C ->                    EDO   -> S
>>    C <-   !X-EDO-X!   Extra-opts <- S
>>    C ???!!!
>>
>> 1.2 Extra Options on subsequent segment from server
>>
>>    C ->                   EDO    -> S
>>    C <-   !X-EDO-X!              <- S
>>      ...
>>    C <-   !X-EDO-X!   Extra-opts <- S
>>    C ???!!!
>>
>> 2. EDO not removed from SYN or SYN/ACK
>> 2.1 Middlebox removes EDO from subsequent segment from server
>>
>>    C ->                   EDO    -> S
>>    C <-      EDO      Extra-opts <- S
>>      ...
>>    C <-   !X-EDO-X!   Extra-opts <- S
>>    C ???!!!
>>
>> 2.2 Middlebox removes EDO from subsequent segment from client
>>
>>    C ->                   EDO    -> S
>>    C <-      EDO      Extra-opts <- S
>>      ...
>>    C ->   Extra-opts   !X-EDO-X! -> S
>>                              ???!!! S
>>
>>
>> ==Commentary==
>>
>> Case #1:
>> The passive opener (server) thinks that the connection supports EDO,
>> so it sends a SYN/ACK confirming this. But before it reaches the
>> client, a middlebox removes unknown TCP options, but forwards the
>> payload unchanged, including the extra TCP options.
>>
>> Without the EDO option, the client believes everything beyond the Data
>> Offset in the SYN/ACK is payload, so it passes both the extra TCP
>> options and the actual payload to the app.
>>
>> Outcome in both cases under #1:
>>         TCP delivers corrupt user-data to the client.
>>
>> Case #2:
>> The 3WHS negotiating EDO completes correctly, so both ends correctly
>> think that the connection supports EDO. But later in the connection, a
>> middlebox starts removing unknown TCP options, but continues to
>> forward the payload unchanged. This is probably much less likely than
>> case 1.1, but might be due to a path change, or the behaviour of a
>> middlbox being different for segments with SYN=0.
>>
>> Again, without the EDO option, the receiving end believes everything
>> beyond the Data Offset is payload, so it passes both the extra TCP
>> options and the actual payload to the app.
>>
>> Outcome in case #2.1:
>>         TCP delivers corrupt user-data to the client.
>> Outcome in case #2.2:
>>         TCP delivers corrupt user-data to the server.
>>
>> ==Implications==
>>
>> I found the above problem when I was working through all the possible
>> problems if I was to integrate EDO with draft-briscoe-tcpm-syn-op-sis
>> (to extend TCP option space on non-SYN packets subsequent to extending
>> space on the SYN).
>>
>> In the light of the above, I won't be integrating EDO with syn-op-sis.
>> I don't think we should introduce anything to TCP with a known risk of
>> delivering corrupt user-data, even if the risk is low. If there were a
>> middlebox behaviour that 'just' blocked the new feature, it would be
>> another matter. But if asymmetry does cause TCP to deliver corrupt
>> data to certain users, the same users will continually experience
>> bizarre and unreliable behaviour - very expensive for customer support
>> people to diagnose.
>>
>> I shall be adding a section to the syn-op-sis draft to define how
>> syn-op-sis can be used to extend the TCP option space on any packet as
>> well as the SYN (it's fairly obvious). I believe it will be immune to
>> the above problem. (Joe, I'll also add an Appendix for the alternative
>> protocol structure I mentioned in response to you not liking the
>> trailer-based structure.)
>>
>> As you know, I was amazed when it was proposed that EDO should go
>> forward for proposed standard. I said in the TCPM meeting that it
>> should be at least experimental, because these days any new TCP
>> options need to be tested against a wide range of middlebox
>> behaviours. With the above problems, I think we should revisit the
>> decision even to adopt EDO.
>>
>> Sorry guys, EDO looked promising, but our job is to ensure TCP
>> continues to be robust.
>>
>>
>> Regards
>>
>>
>> Bob
>>
>>
>> ________________________________________________________________
>> Bob Briscoe,                                                  BT
> 
> ________________________________________________________________
> Bob Briscoe,                                                  BT