Re: [multipathtcp] RFC6824bis edits based on implementation feedback

Joe Touch <> Sun, 02 February 2020 17:39 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id BF4EC1200E5 for <>; Sun, 2 Feb 2020 09:39:15 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.218
X-Spam-Status: No, score=-1.218 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_NEUTRAL=0.779, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id BGpfx2v6q64W for <>; Sun, 2 Feb 2020 09:39:12 -0800 (PST)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 99BF3120033 for <>; Sun, 2 Feb 2020 09:39:12 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=0IwyaZhj+sijoCvjkPGwu9woE2aILcV0jaFMDr2BXbo=; b=HFec+iBp5LScA83SuGiXkWM33 Hz6ddj2l6QK6wzBoXqLZhn2Uxff+Ze7H4UqCp90/bUNZ8PTbhtQkXZvj8Rkd6hvSWJazt3c/9ETJc vB+Y9YwV1prQv6XEpwAlhlsloCVZ16zkTeCxYucNJWG0JdKv8tj8xJXkqhUqk/EyYjs8PdDVl6vlc UY82iFU3cZB1N0b3CbsYyHIVOAWfyCZ3asJ4VnatWW+mCdShsN6mVSqDl+VQFZJMS2RbD4DAxUui5 cMQHBm05C+aEK4JHiarMdcYMKXsKG5xgrwqqFm8w8X2tLwhWRwdHiS9TOkHXUd3ZGdAkcXe5ieWBf snFQZz0EA==;
Received: from ([]:57674 helo=[]) by with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.92) (envelope-from <>) id 1iyJD7-000IyE-SX; Sun, 02 Feb 2020 12:39:11 -0500
To: V Anil Kumar <>, Yoshifumi Nishida <>
Cc: multipathtcp <>
References: <> <> <> <>
From: Joe Touch <>
Message-ID: <>
Date: Sun, 2 Feb 2020 09:39:05 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.4.2
MIME-Version: 1.0
In-Reply-To: <>
Content-Type: multipart/alternative; boundary="------------225E470ECD052918FE74F6C3"
Content-Language: en-US
X-OutGoing-Spam-Status: No, score=-1.0
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname -
X-AntiAbuse: Original Domain -
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain -
X-Get-Message-Sender-Via: authenticated_id:
X-From-Rewrite: unmodified, already matched
Archived-At: <>
Subject: Re: [multipathtcp] RFC6824bis edits based on implementation feedback
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Multi-path extensions for TCP <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Sun, 02 Feb 2020 17:39:16 -0000

RST doesn't "close" a connection; it aborts it. If a RST is sent, it 
should be sent *consistent with how TCP treats sending a RST* and not 
defined anew for this extension.


On 2/2/2020 8:39 AM, V Anil Kumar wrote:
> Hi Yoshi,
> Thanks for this point. In fact, I had initially not thought of a 
> scenario, where the map is being delivered through a retransmitted 
> data packet while its first transmission did not include the map. Now 
> I am just seeing the document (RFC 6824-bis) in this context.
> My understanding is that in scenarios like what I described in my 
> previous mail, RST is likely to happen whether we explicitly state so 
> or not. Please see the paragraph containing the below text in RFC 
> 6824-bis.
> "If a mapping for that subflow-level sequence space does not arrive 
> within a receive window of data, that subflow SHOULD be treated as 
> broken, closed with a RST, and any unmapped data silently discarded."
> if we assume that the map is included while retransmitting the data 
> (even though the first transmission did not contain the map for some 
> reasons),  we could argue that RST could be avoided provided that the 
> retransmission is triggered within a receive window of data. But the 
> question here would be how and when will the retransmission take 
> place. In this case, the subflow may not initiate the retransmission 
> of data by its own (i.e., no retransmission due to three duplicate 
> ACKs or RTO expiry at subflow level) as there is no segment loss at 
> subflow level sequence space. So there could be a high possibility of 
> RST happening even before the map delivery through retransmission.
> With regards,
> Anil
> ------------------------------------------------------------------------
> *From: *"Yoshifumi Nishida" <>
> *To: *"V Anil Kumar" <>
> *Cc: *"alan ford" <>om>, "multipathtcp" 
> <>
> *Sent: *Saturday, February 1, 2020 3:39:51 AM
> *Subject: *Re: [multipathtcp] RFC6824bis edits based on implementation 
> feedback
> Hi Anil,
> I have a question about your proposed text.
> I am actually wondering if we really want to terminate connection here.
> The packets without proper mappings will be treated as invalid and 
> will be discarded.
> If an implementation failed to attach proper mapping for some reasons 
> (e.g. option space), it might be able to attach the proper one when it 
> retransmits the packets. This also looks ok to me.
> I don't have strong preference for this. But, do we have a reason to 
> terminate connection?
> Thanks,
> --
> Yoshi
> On Mon, Jan 13, 2020 at 10:28 AM V Anil Kumar < 
> <>> wrote:
>     Hi,
>     I have some points related to the  modifications (Change 2) being
>     proposed on data sequence map. Please see them inline. Though I am
>     putting forward the below points, if the consensus is in favour of
>     the proposed change for reducing implementation complexity, I am
>     also OK with that as well.
>     ------------------------------------------------------------------------
>     *From: *"alan ford" < <>>
>     *To: * <>
>     *Sent: *Thursday, January 2, 2020 4:21:32 AM
>     *Subject: *[multipathtcp] RFC6824bis edits based on implementation
>     feedback
>     Hi all,
>     We’d love to get this to a state of completion as soon as
>     possible, and to this end I am starting a new thread on this
>     topic. In discussion with the chairs, it /is /possible to make the
>     desired changes in AUTH48 as long as there is WG consensus. The
>     discussion so far has been fairly limited in terms of participation.
>     I would ask the chairs please if it was possible to specify a time
>     bound for this discussion and a default conclusion.
>     Regarding the changes, in summary, there are two areas where
>     changes have been requested by the implementation community. As we
>     are the IETF we obviously have strong focus on “running code” and
>     so ease of implementing standards-compliant code is strongly
>     desirable. However, we do not wish to reduce functionality agreed
>     by the IETF community if it is considered a required feature by
>     the community.
>     *Change 1*
>     *
>     *
>     Change the sentence reading:
>     /   If B has data to send first, then the reliable delivery of the
>     ACK + MP_CAPABLE can be inferred by the receipt of this data with
>     an MPTCP Data Sequence Signal (DSS) option (Section 3.3).
>     /
>     To:
>     /   If B has data to send first, then the reliable delivery of the
>     ACK + MP_CAPABLE is ensured by the receipt of this data with an
>     MPTCP Data Sequence Signal (DSS) option (Section 3.3) containing a
>     DATA_ACK for the MP_CAPABLE (which is the first octet of the data
>     sequence space).
>     /
>     What this means:
>     The current text is concerned only with ensuring a path is MPTCP
>     capable, and so only cares that DSS option occurs on a data
>     packet. However, the MP_CAPABLE option is defined to occupy the
>     first octet of data sequence space and thus, if analogous to TCP,
>     must be acknowledged. >From an implementation point of view it
>     would make sense not to have this hanging around forever and
>     instead define it is acknowledged at the connection level as soon
>     as received. This change ensures the first data packet also
>     DATA_ACKs this MP_CAPABLE octet.
>     *Change 2*
>     *
>     *
>     Change the sentence reading:
>     /   A Data Sequence Mapping does not need to be included in every
>     MPTCP packet, as long as the subflow sequence space in that packet
>     is covered by a mapping known at the receiver./
>     To:
>     /   The mapping provided by a Data Sequence Mapping MUST apply to
>     some or all of the subflow sequence space in the TCP segment which
>     carries the option. It does not need to be included in every MPTCP
>     packet, as long as the subflow sequence space in that packet is
>     covered by a mapping known at the receiver.
>     /
>     What this means:
>     The current text does not place any restrictions on where a
>     mapping could appear. In theory a sender could define a thousand
>     different mappings up front, send them all, and expect a receiver
>     to store this and reassemble data according to these mappings as
>     it arrives. Indeed, this was never explicitly disallowed since it
>     “might have been useful”. The implementation community, however,
>     has expressed concerns over the difficulty of implementing this
>     open-endedly. How many mappings is it reasonable to store? Is
>     there a DoS risk here? Instead, it has been requested that thee
>     specification restricts the placement of the DSS option to being
>     within the subflow sequence space to which it applies.
>     Below are my comments on this. I had shared some of these points
>     in a previous thread that you had initiated in the same context.
>     Transmitting large number of non-contiguous data sequence maps
>     could be a misbehaviour (map-flooding), though it is not clear
>     whether this can go to the extent of causing a potential DoS to
>     the data receiver. So some sort of restriction on this could be
>     useful. One approach could be to insist that the data sender
>     should ensure that the map being transmitted is for in-window
>     data, as per the receiver advertised window. A receiver should
>     anyhow be willing to store the maps for in-window data to deal
>     with packet loss.. For example, when a window of data segments
>     (say 1 to 64) is transmitted, each carrying its corresponding map,
>     and segment-1 is lost, the maps for the remaining 63 need to be
>     stored till the lost segment is retransmitted. Of course, in this
>     case the maps will be stored at the receiver side along with their
>     corresponding data. But the need to store multiple maps for
>     in-window data would still be there.
>     The problem with the proposed change (restriction) is that a data
>     sender may find it difficult, in case a need arise to slightly
>     delay the map delivery by few segments, i.e., sending some data
>     first without map, and then send the corresponding map in a later
>     segment, as shown below:
>     subflow-1:      segment-1 segment-3 segment-4 segment-7
>                           bytes:1-100 bytes:201-300  bytes:301-400
>     bytes:601-700
>                           no map  map for 1-100            map for
>     201-400         map for 601-700
>     subflow-2:       segment-2 segment-5 segment-6 segment-8
>                            bytes: 101-200 bytes:401-500         bytes:
>     501-600  bytes:701-800
>                            map for 101-200 map for 401-600          
>      no map  map for 701-800
>     In the above case, segment-1 goes without map and its map is
>     included later in segment-3, the next data segment in the same
>     subflow. Further,  in the above scheduling pattern, the map in
>     segment-3 cannot cover the  data in segment-1 and segment-3, as
>     some  data in between (segment-2) is transmitted through another
>     subflow..  With the proposed change, the map in segment-3 will
>     become invalid and this will eventually break subflow-1, though
>     this could be a corner case.
>     The question at this stage is why would segment-1 be transmitted
>     without its map. In the case of bidirectional data transfer, there
>     could be a need to pack both timestamp and SACK  options in a data
>     segment, i.e., piggybacking of  SACK with data. If we consider
>     that timestamp takes 12 bytes and SACK, even with single block,
>      takes another 10 bytes, the remaining 18 bytes option space is
>     not adequate to carry data sequence signal with map, especially
>     when DSN is 64 bit long. So the delivery of either of the two
>     (SACK or map) would be delayed.
>     As far as I understand, RFC 2018 (TCP Selective Acknowledgement
>     Options) implies that SACK should not be delayed. It states "If
>     sent at all, SACK options SHOULD be included in all ACKs which do
>     not ACK the highest sequence number in the data receiver's queue".
>     It also says "If data receiver generates SACK options under any
>     circumstance, it SHOULD generate them under all permitted
>     circumstances". So, as part of meeting the RFC 2018 requirements,
>     if the combination of SACK and timestamp is given preference over
>     DSS, data segments could be transmitted without their map.
>     Another case of delaying map could arise if the data sender
>     prefers to send ADD_ADDR option, instead of map, in a data
>     segment. It is nice that ADD_ADDR option can be delivered reliably
>     in a pure ACK, but I think this is not the case with DSS at present.
>     If we adopt the proposed change, I think it might also be helpful
>     to spell out how the receiver is supposed to behave, if it gets
>     maps not meeting the MUST condition in the proposed change.  For
>     example termination of the subflow with MP_TCPRST option (section
>     3.6 in RFC 6824-bis) with appropriate reason code and T flag value
>     to intimate the data sender the cause for subflow termination.
>     With regards,
>     Anil
>     Please can members of the WG express whether they are happy with
>     these changes, or concerned.
>     Best regards,
>     Alan
>     _______________________________________________
>     multipathtcp mailing list
> <>
>     _______________________________________________
>     multipathtcp mailing list
> <>
> _______________________________________________
> multipathtcp mailing list