Re: [tcpm] On the implementation of TCP urgent data (IETF Internet Draft)

[Jerry Leichter <[e-mail]>]

I haven't seen a copy of the paper, so I have to respond indirectly.

Microsoft confirmed the problem.  It was present in all their stacks  
prior to Vista.  Based on our repeated complaints, they eventually  
produced a patch for XP.  For some time, it was one of their "if you  
know to ask we'll give it to you" patches.  For all I know, they  
eventually decided to roll it in to some regular patch release.

The "OOB data leaking in line" has many causes.  I know of at least  
two completely different mechanisms:

	- On the sender, the problem can occur because you can only send
		one OOB byte per segment, but there is no correlation
		between segments and OOB write requests.  Consider the
		situation where the segmentation code wants to send a
		segment, and within the range of bytes it wants to send,
		there are two that were marked OOB.  The only way to handle
		this correctly is to send two segments, even if one of them
		is artificially short - perhaps even a single byte.  I
		haven't checked implementations in detail, but I've
		never seen this mentioned and I would be surprised if
		implementations get this right.

	- On the receiver, there is a messy case, the details of which
		I forget, in which the implementation doesn't keep enough
		state to properly handle two "close" OOB bytes.  If you
		look in Stevens, he actually reproduces a comment from
		the original BSD code which describes the problem, but
		basically says "you lose".  It's not clear anyone ever
		bothered to fix this.

The fundamental issue is this:  OOB bytes exist at the socket API  
level, but there is no such concept at the TCP level.  The urgent  
pointer exists at the TCP level, but it's basically invisible at the  
socket API level.  The theory is that you can use the urgent pointer  
to implement OOB bytes - and, in fact, that's true.  I've convinced  
myself of this by writing out a semi-formal definition of the  
theoretical semantics at the two levels and sketching an  
implementation that actually presents the right semantics at the  
socket API level.  I'm pretty sure that no one actually implements  
this right other than Microsoft (ironically) because it requires the  
ability to allocate space for OOB bytes at the receiver dynamically,  
and I know of no implementations that do so other than Microsoft's.   
By counting the allocated bytes against the window size you are  
willing you offer, you can avoid the problem that MS ran into, where  
there's no way to bound the amount of space allocated.  The  
implementation is potentially expensive, but it's only expensive for  
applications that actually send many OOB bytes - which is a reasonable  
tradeoff.

What actually happens is that the TCP stack implementers seem never to  
have believed in OOB on general principles, and in any case it seems  
that the TCP stack implementors (the network guys) and the socket API  
implementers (the OS guys) don't seem to talk to each other much.  So  
this delicate implementation issue, which lives exactly at the  
interface between the two, gets lost in the shuffle.  (Note that the  
multiple-OOB-bytes-in-a-segment problem probably cannot be solved  
without changing the TCP stack/socket API interface, since typically  
there is no way for the segmentation layer even to *see* that there  
are multiple outstanding OOB bytes:  The only available interface is a  
single urgent pointer value that gets passed across.)

There's an even more fundamental problem that it's worth pointing  
out:  The actual urgent pointer protocol has a flaw.  The UP is a 16- 
bit offset into the stream.  As long as no segment can be more than  
2^16 bytes - as was true when TCP was designed, because the windows  
size is represented in 16 bits - this works.  But if you use scaled  
windows, it's possible to be in a situation where you need to send a  
UP value that cannot be represented, because it's too far into a large  
segment.  There's an obvious work-around - send a short segment when  
this happens - but I doubt anyone does this.  In fact, I would guess  
that the offset gets calculated ignoring the 16-bit limitation and the  
actual UP sent is the lower 16 bits of the true UP.  Obviously, this  
can cause severe problems to any use of TCP that actually relies on  
the UP being correct!

Finally, there's yet another level of problem:  Some router-like  
devices (Cisco PIX firewalls - in their default configuration! - are a  
known example) simply turn off the Urgent bit!  This is to block a  
very old (10+ years?) Windows bug.  However, this is a disaster for  
any program that actually relies on OOB data/the UP.
                                                         -- Jerry


On Feb 28, 2009, at 3:36 PM, Fernando Gont wrote:

> Bill Squier wrote:
>
> (Added Andrew Yourtchenko (draft co-author) to the recipient's list)
>
> Comments inline...
>
>
>> I haven't had time to read the article completely, but I did skim the
>> Windows section, and Christos is correct.
>
> IIRC, we did the windows tests with cygwin. Maybe this is what lead to
> different results?
>
> That said, the next version of our Internet Draft will include text
> describing the buggy implementations you are referring to.
>
>
>> Further, I noticed that some of your analysis discusses OOB data
>> bleeding in-line.  This is almost certainly caused by an  
>> interaction (on
>> the _sender_) of Nagle and the fact that TCP defines only a single  
>> OOB
>> pointer.  The receiver is not returning bytes which it knows to be  
>> OOB
>> bytes inline, the _sender_ is accidentally placing more than a single
>> byte of OOB in each packet that it sends.
>
> There is no problem with that. TCP just provides a mechanism for  
> marking
> the end of urgent data. Just a mark.
>
>
>
>> The receiver has no way to know that, as the only means of  
>> communication
>> about OOB data between sender and receiver is a single pointer.
>
> Exactly. Any data that's before the pointer should be considered  
> "urgent".
>
> Thanks!
>
> Kind regards,
> -- 
> Fernando Gont
> e-mail: [e-mail] || [e-mail]
> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
>
>
>
>
>