Re: [multipathtcp] how to deal with GRO when we use MPTCP?

Olivier Bonaventure <> Wed, 22 March 2017 08:54 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 0F6AD126FDC for <>; Wed, 22 Mar 2017 01:54:23 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -4.301
X-Spam-Status: No, score=-4.301 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (1024-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id jBiF-RArQP8J for <>; Wed, 22 Mar 2017 01:54:20 -0700 (PDT)
Received: from ( []) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id A0B21129689 for <>; Wed, 22 Mar 2017 01:54:15 -0700 (PDT)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: by (Postfix) with ESMTPSA id 2A33D67DBD2; Wed, 22 Mar 2017 09:54:08 +0100 (CET)
DKIM-Filter: OpenDKIM Filter v2.9.2 2A33D67DBD2
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;; s=selucl; t=1490172848; bh=AT0OXWQb1NkaNas9Ft34Cgucx10fgG4VYu8TqYydy8Q=; h=Reply-To:Subject:References:To:From:Date:In-Reply-To; b=qIpnICG9Tbcfz+VM8/vEgGE/d0GpLdJf0w8zdmZZd+3HPE1Ob5c3ZyYgk8xFlYb9b G9/4fV3nPlbL3Oq5uYUDmkGI9YWn15ywOGfeVgBiHYZOIFaJmDarS2HDPlIkRou5mA QGgDnaqqQznVPa5NhbIfmjEFKvw8uBBfNW1SGZq4=
X-Virus-Status: Clean
X-Virus-Scanned: clamav-milter 0.99.2 at smtp-5
References: <>
To: 王帅 <>, multipathtcp <>
From: Olivier Bonaventure <>
Message-ID: <>
Date: Wed, 22 Mar 2017 09:54:08 +0100
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.8.0
MIME-Version: 1.0
In-Reply-To: <>
Content-Type: text/plain; charset="windows-1252"; format="flowed"
Content-Transfer-Encoding: 7bit
X-Sgsi-Spamcheck: SASL authenticated,
X-SGSI-MailScanner-ID: 2A33D67DBD2.A3AEC
X-SGSI-MailScanner: Found to be clean
X-SGSI-Spam-Status: No
Archived-At: <>
Subject: Re: [multipathtcp] how to deal with GRO when we use MPTCP?
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Multi-path extensions for TCP <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 22 Mar 2017 08:54:23 -0000

>      Some NICs has GRO(generic-receive-offload), which works by
> aggregating multiple incoming packets
> <> from a
> single stream <> into a
> larger buffer before they are passed higher up the networking stack,
> thus reducing the number of packets that have to be processed. But when
> we use MPTCP, GRO may aggregate packets whose subflow sequence number is
> continuous but data sequence number is discontinuous. This is not
> expected. How can we deal with this problem? Is it necessary to disable GRO?

GRO should not aggregate packets that contain different TCP options that 
are unknown to the network interface. Our experience with GRO and 
Multipath TCP has been positive until now with the Linux MPTCP stack.

GRO is typically combined with TSO. What we usually observe is the 
following behaviour :

Sender creates a large packet with DSN, Timestamp, data
TSO splits the packet with each packet containing a copy of the DSN and 
the Timestamp. All DSNs are the same in the packets sent by the network 

On the receiver side, GRO sees a sequence of packets with the same DSN 
and Timestamp options. These packets are combined together to present a 
large packet containing DSN, Timestamp, data to the stack

GRO should not coalesce packets from an MPTCP sender that does not use 
TSO and thus send packets with different DSN values. If you see a 
network interface that behaves badly, please report this on the 
mptcp-dev mailing list