[tcpm] Asymmetric Middleboxes and Extended Data Offset (EDO)

[Bob Briscoe <[e-mail]>]

Joe, Wes,

It is known that new TCP options are removed on x% of paths.
It is also known that y% of paths are asymmteric
Let's assume z% of the TCP-option-removing middleboxes are on the asymmetric part of a path.

(x y and z are left as an exercise for the experimenter. I'm offline at the mo, but I know there are papers that give good estimates of x and y at least).

Also some middleboxes behave asymmetrically, for instance firewalls have stricter rules for traffic arriving from the public side vs the private side. Such a device might remove unknown TCP options in one direction, but not the other.

==Legend==
C               = TCP client
S               = TCP server
EDO             = EDO delivered
!X-EDO-X!       = Middlebox removes EDO TCP option
Extra-opts      = TCP options beyond the Data Offset (preserved)
???!!!  = confused or crashes

==Scenarios==

1. EDO not removed from SYN, but middlebox removes EDO from SYN/ACK 
1.1 Extra Options on SYN/ACK

   C ->                    EDO   -> S
   C <-   !X-EDO-X!   Extra-opts <- S
   C ???!!!

1.2 Extra Options on subsequent segment from server

   C ->                   EDO    -> S
   C <-   !X-EDO-X!              <- S
     ...
   C <-   !X-EDO-X!   Extra-opts <- S
   C ???!!!

2. EDO not removed from SYN or SYN/ACK
2.1 Middlebox removes EDO from subsequent segment from server

   C ->                   EDO    -> S
   C <-      EDO      Extra-opts <- S
     ...
   C <-   !X-EDO-X!   Extra-opts <- S
   C ???!!!

2.2 Middlebox removes EDO from subsequent segment from client

   C ->                   EDO    -> S
   C <-      EDO      Extra-opts <- S
     ...
   C ->   Extra-opts   !X-EDO-X! -> S
                             ???!!! S


==Commentary==

Case #1:
The passive opener (server) thinks that the connection supports EDO, so it sends a SYN/ACK confirming this. But before it reaches the client, a middlebox removes unknown TCP options, but forwards the payload unchanged, including the extra TCP options.

Without the EDO option, the client believes everything beyond the Data Offset in the SYN/ACK is payload, so it passes both the extra TCP options and the actual payload to the app. 

Outcome in both cases under #1: 
        TCP delivers corrupt user-data to the client.

Case #2:
The 3WHS negotiating EDO completes correctly, so both ends correctly think that the connection supports EDO. But later in the connection, a middlebox starts removing unknown TCP options, but continues to forward the payload unchanged. This is probably much less likely than case 1.1, but might be due to a path change, or the behaviour of a middlbox being different for segments with SYN=0.

Again, without the EDO option, the receiving end believes everything beyond the Data Offset is payload, so it passes both the extra TCP options and the actual payload to the app. 

Outcome in case #2.1: 
        TCP delivers corrupt user-data to the client.
Outcome in case #2.2: 
        TCP delivers corrupt user-data to the server.

==Implications==

I found the above problem when I was working through all the possible problems if I was to integrate EDO with draft-briscoe-tcpm-syn-op-sis (to extend TCP option space on non-SYN packets subsequent to extending space on the SYN).

In the light of the above, I won't be integrating EDO with syn-op-sis. I don't think we should introduce anything to TCP with a known risk of delivering corrupt user-data, even if the risk is low. If there were a middlebox behaviour that 'just' blocked the new feature, it would be another matter. But if asymmetry does cause TCP to deliver corrupt data to certain users, the same users will continually experience bizarre and unreliable behaviour - very expensive for customer support people to diagnose.

I shall be adding a section to the syn-op-sis draft to define how syn-op-sis can be used to extend the TCP option space on any packet as well as the SYN (it's fairly obvious). I believe it will be immune to the above problem. (Joe, I'll also add an Appendix for the alternative protocol structure I mentioned in response to you not liking the trailer-based structure.)

As you know, I was amazed when it was proposed that EDO should go forward for proposed standard. I said in the TCPM meeting that it should be at least experimental, because these days any new TCP options need to be tested against a wide range of middlebox behaviours. With the above problems, I think we should revisit the decision even to adopt EDO.

Sorry guys, EDO looked promising, but our job is to ensure TCP continues to be robust. 


Regards


Bob


________________________________________________________________
Bob Briscoe,                                                  BT