Re: Fun and surprises with IPv6 fragmentation

Christian Huitema <huitema@huitema.net> Sun, 04 March 2018 18:45 UTC

Return-Path: <huitema@huitema.net>
X-Original-To: quic@ietfa.amsl.com
Delivered-To: quic@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id F2943126DFB for <quic@ietfa.amsl.com>; Sun, 4 Mar 2018 10:45:16 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.6
X-Spam-Level:
X-Spam-Status: No, score=-2.6 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4IDmonSDVVHS for <quic@ietfa.amsl.com>; Sun, 4 Mar 2018 10:45:15 -0800 (PST)
Received: from mx43-out1.antispamcloud.com (mx43-out1.antispamcloud.com [138.201.61.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id D088512D77E for <quic@ietf.org>; Sun, 4 Mar 2018 10:45:14 -0800 (PST)
Received: from xsmtp01.mail2web.com ([168.144.250.230]) by mx68.antispamcloud.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.89) (envelope-from <huitema@huitema.net>) id 1esYdE-0009Ne-FQ for quic@ietf.org; Sun, 04 Mar 2018 19:45:13 +0100
Received: from [10.5.2.52] (helo=xmail12.myhosting.com) by xsmtp01.mail2web.com with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.63) (envelope-from <huitema@huitema.net>) id 1esYd8-0003Kf-6N for quic@ietf.org; Sun, 04 Mar 2018 13:45:10 -0500
Received: (qmail 14037 invoked from network); 4 Mar 2018 18:45:03 -0000
Received: from unknown (HELO [192.168.1.103]) (Authenticated-user:_huitema@huitema.net@[172.56.42.241]) (envelope-sender <huitema@huitema.net>) by xmail12.myhosting.com (qmail-ldap-1.03) with ESMTPA for <quic@ietf.org>; 4 Mar 2018 18:45:03 -0000
To: "Lubashev, Igor" <ilubashe=40akamai.com@dmarc.ietf.org>, "ek=40google.com@dmarc.ietf.org" <ek@google.com>, "mikkelfj@gmail.com" <mikkelfj@gmail.com>
Cc: "quic@ietf.org" <quic@ietf.org>
References: <681fcc96-4cf9-100d-9ad6-b3c7be9189a5@huitema.net> <CAAedzxq5G_gAaBzdizv5x-yobomW5+8sSZ_1rn4ApGYpQZmpvg@mail.gmail.com> <CAN1APdcF0DeKQSDd_=CPbSJRuvSWQRF_K+EUpzCrp7OMytzvNw@mail.gmail.com> <1f0f0060365f4d60a73cf68b7344a271@usma1ex-dag1mb5.msg.corp.akamai.com>
From: Christian Huitema <huitema@huitema.net>
Message-ID: <6eddf6ee-e9a9-da20-276c-c724a36bd33a@huitema.net>
Date: Sun, 4 Mar 2018 10:45:01 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0
MIME-Version: 1.0
In-Reply-To: <1f0f0060365f4d60a73cf68b7344a271@usma1ex-dag1mb5.msg.corp.akamai.com>
Content-Type: multipart/alternative; boundary="------------148B0E5735CF8FDA850C7AF2"
Content-Language: en-US
Subject: Re: Fun and surprises with IPv6 fragmentation
X-Originating-IP: 168.144.250.230
X-AntiSpamCloud-Domain: xsmtpout.mail2web.com
X-AntiSpamCloud-Username: 168.144.250.0/24
Authentication-Results: antispamcloud.com; auth=pass smtp.auth=168.144.250.0/24@xsmtpout.mail2web.com
X-AntiSpamCloud-Outgoing-Class: unsure
X-AntiSpamCloud-Outgoing-Evidence: Combined (0.21)
X-Recommended-Action: accept
X-Filter-ID: EX5BVjFpneJeBchSMxfU5lImCNn1Fa9+r24lr0SvBk9602E9L7XzfQH6nu9C/Fh9KJzpNe6xgvOx q3u0UDjvO37pNwwF1lRXh5rzvPzo9Jts1ujulqUFmMITHM77eiViXOgOoOvt0FlwrzuWdHvU9c7i TvJ2/ZGzVWB9scFAaCdIFaUvXN+CI+RGy3Me16pByF1cJza2OTfUZp7q1vrgSR/TBCf6oYXAWGet lavcAjD9ytQxIHf9lN5jjLJaPK8lRJSPf/SXbEnDSsal/zZzc4n9VZdr7RAFD5mRwooUYhwMPaBP aKeQW+/QlaOdv8isl/qMm08Zpim2AHUKEWvQ6G/bWfgucjnNmABpGhD9TTttrFCuZ0NkwnSz2Luu o1u9uevuNfM1HjkNEFwape+IgNezYqxGMqsKjARq8PBC4qjSYb8Ll5Ew7esaVIVXxqL4mdySlZou 9qHIGOZDEEo7Oyc1nq0gsY582CWqKjiRB3upW940lL8kAcN44/h+EKQYy0BtzoAV9afvwClt5UF6 BikKngqPM5kg6DLO/jczwf3oeHyJXUyQKOQY08Sqhy3rAvtb2n/8ygNUZpILeWOV/dm1LR/Rkhms gi1bNni1Nx10xK5dOvm4pbg8CBiJcsFLIaKLZaIWHErhz1UEr981hLCx4eCSemGsHMs75OfoLqfl nGtGdG3g+ozMfeTuFOSvpyI1J519bLUrM3neslGWzoQMdcG0AKlI0DaFCuzWsrjouROe77KOt1v8 CMoiiVQVf/qOH8le3Zo2iD646pXyZ6OCuRoF/r1Z8lyFNSWRnwoKcZIuviiTlhP5iKsKcxHRGLKn 1meRR/jcOZCTAzGMwToUd/Fji/4E0A1Wfl7CNnoniG/Mx18wh299NVDnPvmw8t3Xj/LzZ5s/OJg1 L2asZ1FPmtYUPwECbZov76lmox9FrrkXXOH9yk2+X2eiUHXZfks5suSVIgIg9uJ3gfzn7g==
X-Report-Abuse-To: spam@quarantine5.antispamcloud.com
Archived-At: <https://mailarchive.ietf.org/arch/msg/quic/I5_ZbvNnLd-KJunm7Vt4X5xVq8c>
X-BeenThere: quic@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <quic.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/quic>, <mailto:quic-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/quic/>
List-Post: <mailto:quic@ietf.org>
List-Help: <mailto:quic-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/quic>, <mailto:quic-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sun, 04 Mar 2018 18:45:20 -0000

Thanks to all of you for the comments. I updated my blog entry on the
subject
(https://huitema.wordpress.com/2018/03/03/having-fun-and-surprises-with-ipv6/)
with the following new conclusion:

/It seems clear now that the fragmentation happened at the source, in
the Linux kernel. This leaves one remaining issue, the out of order
delivery./

/There are in fact two separate out of order delivery issues. One is
having the second fragment arrive between the first one, and the second
is having the MTU probe arrive before the previously sent Handshake
packet. The inversion between the segments may be due to code in the
Linux kernel that believes that sending the last segment first speeds up
reassembly at the receiver. The inversion between the fragmented MTU
probe and the Handshake packet has two plausible causes:/

  * /Some router on the path may be forwarding the small fragment at a
    higher priority level./
  * /Some routers may be implementing equal cost multipath, and then
    placing fragmented packets and regular packets into separate hash
    buckets/

/The summary for developers, and for QUIC in particular, is that we
should really avoid triggering IPv6 fragmentation. It can lead to packet
losses when NATs and firewalls cannot find the UDP payload type and the
port numbers in the fragments. And it can also lead to out of order
delivery as we just saw. And for my own code, the lesson is simple. I
really need to set up the IPv6 Don’t Fragment option when sending MTU
probes, per section 11.2 of RFC 3542
<https://tools.ietf.org/html/rfc3542#section-11.2>.

/-- Christian Huitema