Fun and surprises with IPv6 fragmentation

[Christian Huitema <[e-mail]>]

Yesterday, I was mentioning bugs of the interop. This morning, I woke up
to find an interesting message from Patrick McManus. Something is weird,
he said. The first data message that your server sends, with sequence
number N, always arrives before the final handshake message, with
sequence number N-1. That inversion appears to happen systematically.

It took us the best part of a day to explore blind alleys and finally
understand what was happening. The exchange was over IPv6. Upon
receiving a connection request from Patrick’s implementation, Picoquic
was sending back a handshake packet. Immediately after that, Picoquic
was sending its first data packet, which happens to be an MTU probe. And
it turns out that the probe was 1518 bytes, a bit longer than what the
AWS routers could accept. So some router inserted an IPv6 fragmentation
header and split the packet in two: a large initial fragment, 1496 byte
long, and a small second fragment 78 bytes long. You could think that
this is no big deal, since fragments would just be reassembled at the
destination, but you would be wrong.

Some routers on the path try to be helpful. They have learned from past
experience that short packets often carry important data, and so they
try to route them faster than long data packets. And here is what
happens in our case:

·         * The server prepares and send a Handshake packet, 590 bytes long.

·         * The server then prepares the MTU probe, 1518 bytes long.

·         * The MTU probe is split into fragment 1, 1496 bytes, and
fragment 2, 78 bytes.

·         * The handshake and the long fragment are routed on the normal
path, but the small fragment is routed at a higher priority level.

·         * The Linux driver at the destination receives the small
fragment first. It queues everything behind that until it receives the
long fragment.

·         * The Linux driver passes the reassembled packet to the
application, which cannot do anything with it because the encryption
keys can only be obtained from the handshake packet.

·         * The Linux driver then passes the handshake packet to the
application.

Which confirms an old opinion. When routers try to be smart and helpful,
they end up being dumb and harmful. Please just send the packets in the
order you get them!

I tried to work around the issue by setting the "don't fragment" bit on
the socket, but somehow that doesn't work. So I simply programmed the
server to not use payloads larger than 1440 bytes. Still, I can see that
pattern happening in other circumstances, such as a long Connection
Initial message followed by a short 0-RTT packet. isn't networking fun?

-- Christian Huitema