Re: Fun and surprises with IPv6 fragmentation

Christian Huitema <> Sun, 04 March 2018 18:45 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id F2943126DFB for <>; Sun, 4 Mar 2018 10:45:16 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.6
X-Spam-Status: No, score=-2.6 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 4IDmonSDVVHS for <>; Sun, 4 Mar 2018 10:45:15 -0800 (PST)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id D088512D77E for <>; Sun, 4 Mar 2018 10:45:14 -0800 (PST)
Received: from ([]) by with esmtps (TLSv1:AES256-SHA:256) (Exim 4.89) (envelope-from <>) id 1esYdE-0009Ne-FQ for; Sun, 04 Mar 2018 19:45:13 +0100
Received: from [] ( by with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.63) (envelope-from <>) id 1esYd8-0003Kf-6N for; Sun, 04 Mar 2018 13:45:10 -0500
Received: (qmail 14037 invoked from network); 4 Mar 2018 18:45:03 -0000
Received: from unknown (HELO []) ([]) (envelope-sender <>) by (qmail-ldap-1.03) with ESMTPA for <>; 4 Mar 2018 18:45:03 -0000
To: "Lubashev, Igor" <>, "" <>, "" <>
Cc: "" <>
References: <> <> <> <>
From: Christian Huitema <>
Message-ID: <>
Date: Sun, 04 Mar 2018 10:45:01 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0
MIME-Version: 1.0
In-Reply-To: <>
Content-Type: multipart/alternative; boundary="------------148B0E5735CF8FDA850C7AF2"
Content-Language: en-US
Subject: Re: Fun and surprises with IPv6 fragmentation
Authentication-Results:; auth=pass smtp.auth=
X-AntiSpamCloud-Outgoing-Class: unsure
X-AntiSpamCloud-Outgoing-Evidence: Combined (0.21)
X-Recommended-Action: accept
X-Filter-ID: EX5BVjFpneJeBchSMxfU5lImCNn1Fa9+r24lr0SvBk9602E9L7XzfQH6nu9C/Fh9KJzpNe6xgvOx q3u0UDjvO37pNwwF1lRXh5rzvPzo9Jts1ujulqUFmMITHM77eiViXOgOoOvt0FlwrzuWdHvU9c7i TvJ2/ZGzVWB9scFAaCdIFaUvXN+CI+RGy3Me16pByF1cJza2OTfUZp7q1vrgSR/TBCf6oYXAWGet lavcAjD9ytQxIHf9lN5jjLJaPK8lRJSPf/SXbEnDSsal/zZzc4n9VZdr7RAFD5mRwooUYhwMPaBP aKeQW+/QlaOdv8isl/qMm08Zpim2AHUKEWvQ6G/bWfgucjnNmABpGhD9TTttrFCuZ0NkwnSz2Luu o1u9uevuNfM1HjkNEFwape+IgNezYqxGMqsKjARq8PBC4qjSYb8Ll5Ew7esaVIVXxqL4mdySlZou 9qHIGOZDEEo7Oyc1nq0gsY582CWqKjiRB3upW940lL8kAcN44/h+EKQYy0BtzoAV9afvwClt5UF6 BikKngqPM5kg6DLO/jczwf3oeHyJXUyQKOQY08Sqhy3rAvtb2n/8ygNUZpILeWOV/dm1LR/Rkhms gi1bNni1Nx10xK5dOvm4pbg8CBiJcsFLIaKLZaIWHErhz1UEr981hLCx4eCSemGsHMs75OfoLqfl nGtGdG3g+ozMfeTuFOSvpyI1J519bLUrM3neslGWzoQMdcG0AKlI0DaFCuzWsrjouROe77KOt1v8 CMoiiVQVf/qOH8le3Zo2iD646pXyZ6OCuRoF/r1Z8lyFNSWRnwoKcZIuviiTlhP5iKsKcxHRGLKn 1meRR/jcOZCTAzGMwToUd/Fji/4E0A1Wfl7CNnoniG/Mx18wh299NVDnPvmw8t3Xj/LzZ5s/OJg1 L2asZ1FPmtYUPwECbZov76lmox9FrrkXXOH9yk2+X2eiUHXZfks5suSVIgIg9uJ3gfzn7g==
Archived-At: <>
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Sun, 04 Mar 2018 18:45:20 -0000

Thanks to all of you for the comments. I updated my blog entry on the
with the following new conclusion:

/It seems clear now that the fragmentation happened at the source, in
the Linux kernel. This leaves one remaining issue, the out of order

/There are in fact two separate out of order delivery issues. One is
having the second fragment arrive between the first one, and the second
is having the MTU probe arrive before the previously sent Handshake
packet. The inversion between the segments may be due to code in the
Linux kernel that believes that sending the last segment first speeds up
reassembly at the receiver. The inversion between the fragmented MTU
probe and the Handshake packet has two plausible causes:/

  * /Some router on the path may be forwarding the small fragment at a
    higher priority level./
  * /Some routers may be implementing equal cost multipath, and then
    placing fragmented packets and regular packets into separate hash

/The summary for developers, and for QUIC in particular, is that we
should really avoid triggering IPv6 fragmentation. It can lead to packet
losses when NATs and firewalls cannot find the UDP payload type and the
port numbers in the fragments. And it can also lead to out of order
delivery as we just saw. And for my own code, the lesson is simple. I
really need to set up the IPv6 Don’t Fragment option when sending MTU
probes, per section 11.2 of RFC 3542

/-- Christian Huitema