Re: [multipathtcp] MPTCP implementation feedback for RFC6824bis

Christoph Paasch <> Mon, 02 December 2019 17:28 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id DFE38120099 for <>; Mon, 2 Dec 2019 09:28:45 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -7.001
X-Spam-Status: No, score=-7.001 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id MU3dusWem5f6 for <>; Mon, 2 Dec 2019 09:28:44 -0800 (PST)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id E904F1200C3 for <>; Mon, 2 Dec 2019 09:28:43 -0800 (PST)
Received: from pps.filterd ( []) by ( with SMTP id xB2HMIGY008937; Mon, 2 Dec 2019 09:27:58 -0800
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; h=mime-version : content-transfer-encoding : content-type : sender : date : from : to : cc : subject : message-id : references : in-reply-to; s=20180706; bh=jSkB5kFSA5M34CrAKadAZmA8NHNHL/8h7XVVEdE5qjM=; b=rnSyavzBWOGhgaFxaoOvhb7de8Wr1l6tfL2Ep6l8tUwGtHKae53UfOXhVHQJZjaDj+Wn XSN+FySvKcNB0IVqmlnV03FeJG3gqE8ZKyV0gxuk+PPbxcJkPOubM7GjfCKNb1GrDT6+ /lhWXWMKpGCGp/rSmASfTWVFwaFxtWSNrSFLBhaIavn3D1ZzVOfuLG62FrxBhzXh2Y72 Xhjz9ggOYyylTm4wjvxXzDZFxTEL+vYk0Krg+3FGDsa5UFqcs996m+UD6JfMxdUvBEwn TeCqlFmOtpenT0UpuMofLnOijrsiriz1BOtDtZKbDbbacEOTrqu6Ps2k5EG+M1eE4W0L uA==
Received: from ( []) by with ESMTP id 2wknywg7sp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Mon, 02 Dec 2019 09:27:58 -0800
Received: from ( []) by (Oracle Communications Messaging Server 64bit (built May 7 2019)) with ESMTPS id <>; Mon, 02 Dec 2019 09:27:58 -0800 (PST)
Received: from by (Oracle Communications Messaging Server 64bit (built May 7 2019)) id <>; Mon, 02 Dec 2019 09:27:58 -0800 (PST)
X-Va-T-CD: 3617872cb080ac0182d9b2453d56adc6
X-Va-E-CD: 5a5a60e6f040669a01f467acbb8066aa
X-Va-R-CD: 3858de275c7eacb246074420dbf98e80
X-Va-CD: 0
X-Va-ID: a4da6220-828b-4021-889f-8d96979612ce
X-V-T-CD: 3617872cb080ac0182d9b2453d56adc6
X-V-E-CD: 5a5a60e6f040669a01f467acbb8066aa
X-V-R-CD: 3858de275c7eacb246074420dbf98e80
X-V-CD: 0
X-V-ID: cfb8dc35-abbc-4fdc-9bfe-172d3fbff272
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-12-02_04:,, signatures=0
MIME-version: 1.0
Content-transfer-encoding: 8BIT
Content-disposition: inline
Content-type: text/plain; charset=utf-8
Received: from localhost ([]) by (Oracle Communications Messaging Server 64bit (built May 7 2019)) with ESMTPSA id <>; Mon, 02 Dec 2019 09:27:57 -0800 (PST)
Date: Mon, 02 Dec 2019 09:27:57 -0800
From: Christoph Paasch <>
To: Alan Ford <>
Cc: MultiPath TCP - IETF WG <>, Yoshifumi Nishida <>, Philip Eardley <>, Mirja Kuehlewind <>, mptcp Upstreaming <>
Message-id: <20191202172757.GA84163@MacBook-Pro-64.local>
References: <> <> <> <>
In-reply-to: <>
User-Agent: Mutt/1.12.2 (2019-09-21)
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-12-02_04:, , signatures=0
Archived-At: <>
Subject: Re: [multipathtcp] MPTCP implementation feedback for RFC6824bis
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Multi-path extensions for TCP <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Mon, 02 Dec 2019 17:28:46 -0000

Hello Alan,

On 29/11/19 - 21:13:38, Alan Ford wrote:
> > On 28 Nov 2019, at 19:49, Christoph Paasch <> wrote:
> >> On Nov 28, 2019, at 8:16 AM, Alan Ford < <>> wrote:
> >>> On 27 Nov 2019, at 19:29, Christoph Paasch < <>> wrote:
> >>> Section 3.3.1, page 32 & 33, "A data sequence mapping does not need..."
> >>> 
> >>> This paragraph states that it is permissive to send a mapping in advance. Late-mapping is specified a bit higher around the sentence 
> >>> "Implementations MAY hold onto such unmapped data for a short while in the expectation that a mapping will arrive shortly"
> >>> 
> >>> This kind of early/late mapping announcements are difficult to handle in an implementation. The Linux Kernel implementation of <> has always disallowed such kind of mappings. Meaning, whenever a DSS-option is received such that the range specified by the relative subflow-sequence number in the DSS-option and the data-length does not (partially) cover the TCP sequence number of the packet itself, the subflow will be killed with a TCP-RST. The problem around handling such early/late mappings is that it is unclear for how long the stack needs to remember these mappings (in the early-mapping case), or for how long he needs to hold on to the data (in the late-mapping case).
> >>> 
> >>> We thus suggest to change this to the following:
> >>> Whenever a DSS-option is received on a packet such that the mapping of the subflow-sequence space does not partially cover the TCP-sequence number of  the packet itself, the host MUST discard this mapping and MAY destroy the subflow with a TCP-RST. It should be noted that a DATA_FIN that does not come with data has a relative subflow-sequence number of 0 and thus should be handled separately.
> >> 
> >> This one I do have an issue with:
> >> 
> >> - It is a technical change
> >> - Wording to this effect has been in the document since pretty much the beginning
> >> - It is a MAY which might as well say “there is no guarantee this would work”
> > 
> > The problem with the MAY is that the sender can't really know if the receiver accepts it (more regarding this below)
> > 
> >> Most importantly, the replacement text seems not to address this issue at all. If I read it correctly, it says that the data sequence mapping option MUST partially cover the subflow sequence space of the packet itself. But that has nothing to do with late or early mapping, both could partially cover the subflow sequence space and preceding data.
> >> 
> >> Can you clarify exactly what you want to permit and prevent, here?
> > 
> > Let me try to clarify what exactly we mean with early/late mapping so that we are all on the same terms here:
> > 
> > Early mapping:
> > 
> > A TCP-segment with sequence-number 1 holds a DSS-option with subflow-sequence number 1001 and data-length 100. This means we need to allocate space to store this DSS-option so that when the TCP-segment with seqno 1001 arrives we can know the mapping. There may be coming more of these DSS-options which all need to be stored in allocated memory. It is unclear what the limit to this is and there is no way to communicate this limit to the sender.
> I don’t think we have ever intended to support a mapping like this. If the text is not clear here then yes, we might have an issue.

Yes, I do think that the text is not very clear on that  - we should clarify

> We intend only to support: A TCP segment with sequence-number 1 holds a DSS option for SSN 1 and length 10000 (so multiple segments in the future).

Sounds good!

> Or, slightly more convoluted, a TCP segment with SN 100, length 100, which holds a DSS option for 151-250, and bytes 50-150 were already covered by a previous mapping.

Also good.

> I do not believe we have ever intended mappings on data segments where the segments do not include any of the mapped data. We did, however, intend to support mappings on pure ACKs in order to avoid any option space limits.

Mappings on a pure ACK are an unlikely use-case. The problem is that in the
end the mapping needs to reliably make it to the receiver as otherwise he
needs to throw the data away. Combining it with data implicitly makes it

In case of option-space limits, it is better to send the options that do
consume a lot of space (ADD_ADDR,...) on the pure ACK.

> > Late mapping:
> > 
> > The receiver receives data without DSS-options with TCP-sequence 1 to 1001. The corresponding DSS-option however arrives with the TCP-segment with seqno 2001. Here, the receiver needs to hold on to this data, waiting for the TCP-segment with the DSS-option. At one point the receiver needs to drop the data due to memory limits. Again, the sender has no way for knowing what this limit is.
> So this is slightly more problematic, and this is what the MAY in the text is designed to discuss.
> I believe we had intended to support a situation where you could have a segment 1-1000 without a DSS option, and 1001-2000 also without, and then 2001-3000 with the option then providing a mapping for 1 to 2001 or higher.

I think this scenario is fine to some extend with a slight change that the
provided mapping should be for 1 to 2002 or higher (thus, including the byte
of the 2001-3000 segment).

> This would allow a sender to start pushing out data before it knew what a mapping might look like.
> It does, however, seem an unlikely situation but you as a receiver could of course reduce the subflow window size in order to limit buffering. The text does recognise the memory issue and point out that this won’t be DATA ACKed and as such a sender should soon realise and retransmit on a separate subflow and then this subflow may eventually be closed.

Yes, it seems like an unlikerly situation and I'm not sure about the
use-case for doing this.

> Another situation would involve two mappings on the same TCP segment. If the first 50 bytes of a segment are covered by the mapping provided in that segment, but there are then also another 50 bytes, then the mapping can’t be provided until the next segment.

Yes, two mappings on a single packet are problematic simply because of the
lack of option-space.

> I guess my initial thought here is that this was intended to cover a number of corner cases but if it does not work for you then there are several compliant ways of dealing with it.
> > When the DSS-option comes together with the corresponding TCP-sequence it is straight-forward to store it together with the data. There are no issues with memory-allocation,... as all of this is accounted together with the announced window (yes, the memory is not counted against the window, but the receiver can foresee the DSS-option overhead when computing what window he should announce).
> > 
> > When a receiver gets data without a DSS-option, he can store it for up to 64KB of data as that is the maximum data-level length and the last segment of the 64KB-train could be holding the DSS-option. After that he has to drop the data.
> > 
> > When the mapping partially covers the segment it also isn't a problem as the unmapped part can safely be dropped and the mapped part can be passed on to the MPTCP-layer.
> > 
> > All of this does not imply that every segment of a mapping needs to hold the DSS-option. Just one of them needs to have it.
> That last statement aligns with what was intended in the text. But are you really saying that, or are you saying that the first one needs to have it? Because that would be a change.

No, the first one does not need to have it. Just one of them.