[multipathtcp] RFC6824bis - New Handshake - Implementation feedback

Christoph Paasch <cpaasch@apple.com> Mon, 10 April 2017 06:18 UTC

Return-Path: <cpaasch@apple.com>
X-Original-To: multipathtcp@ietfa.amsl.com
Delivered-To: multipathtcp@ietfa.amsl.com
Received: from localhost (localhost []) by ietfa.amsl.com (Postfix) with ESMTP id 68F9B127775 for <multipathtcp@ietfa.amsl.com>; Sun, 9 Apr 2017 23:18:06 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -4.322
X-Spam-Status: No, score=-4.322 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=apple.com
Received: from mail.ietf.org ([]) by localhost (ietfa.amsl.com []) (amavisd-new, port 10024) with ESMTP id 1ANnoz1Y93ac for <multipathtcp@ietfa.amsl.com>; Sun, 9 Apr 2017 23:18:05 -0700 (PDT)
Received: from mail-in2.euro.apple.com (mail-in.euro.apple.com []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id CBCB6120727 for <multipathtcp@ietf.org>; Sun, 9 Apr 2017 23:18:04 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; d=apple.com; s=mailout2048s; c=relaxed/simple; q=dns/txt; i=@apple.com; t=1491805082; h=From:Sender:Reply-To:Subject:Date:Message-id:To:Cc:MIME-version:Content-type: Content-transfer-encoding:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=uezirBddUteDuXMWWKk9JYogTt8ZOag9qcblukFeLAs=; b=OxYuKupFl/GGYqqlI2gzBM0+bwF27OffTvZEI22hq3U5Ke1DrOsfy5bMxJmF2tdF f47BIq0cRk/ZaxFcI1d9gVZzm4ut7ErP+uhIsJUH5gzuuRq9YW5OxKyNNDC0i7h5 Z8ZlG9Sy7s5kKMp6N3ZDYkCo6eQLezBvz4XDHqESx9mk3+m38wUmrXGhVO2mTGkB kc+nwDEWsipdOb9ELeGJwQN58t7D6Qc9yZTniUFoea+GikhZHQNNgfcGKKBRZaPc ZIG+3cvtDcQPic/TjOuBds18PYQrH3BRu2ST59lerAFLS0azVmQlhuq5unusHhUX /DUtKWgBRZIfz2f+zT+zWQ==;
Received: from relay2.euro.apple.com (relay2.euro.apple.com []) (using TLS with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mail-in2.euro.apple.com (Symantec Mail Security) with SMTP id A2.1F.22441.A932BE85; Mon, 10 Apr 2017 07:18:02 +0100 (BST)
X-AuditID: 1148940c-e67de9a0000057a9-51-58eb239a2b4d
Received: from crk-mmpp-sz03 ( []) by relay2.euro.apple.com (Symantec Mail Security) with SMTP id B6.2F.04605.8932BE85; Mon, 10 Apr 2017 07:18:01 +0100 (BST)
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-disposition: inline
Content-type: text/plain; CHARSET=US-ASCII
Received: from localhost ([]) by crk-mmpp-sz03.euro.apple.com (Oracle Communications Messaging Server 64bit (built Feb 22 2017)) with ESMTPSA id <0OO600EBKK60IO20@crk-mmpp-sz03.euro.apple.com> for multipathtcp@ietf.org; Mon, 10 Apr 2017 07:18:00 +0100 (IST)
Sender: cpaasch@apple.com
Date: Mon, 10 Apr 2017 08:17:59 +0200
From: Christoph Paasch <cpaasch@apple.com>
To: MultiPath WG <multipathtcp@ietf.org>
Message-id: <20170410061759.GO11821@Chimay.local>
User-Agent: Mutt/1.7.1 (2016-10-04)
X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrCLMWRmVeSWpSXmKPExsUi6GTOoztL+XWEwYfnhhafV19nc2D0WLLk J1MAYxSXTUpqTmZZapG+XQJXxpTfb1kKLkhW7G7+xNLAOE2ki5GTQ0LARGLXg6PMXYxcHEIC 25kkVr54yAiTWNl5CSqxmlHi38OFrCAJXgFBiR+T77F0MXJwMAvISxw8LwsSZhaQlnj0dwY7 hK0l8f1RKwtE7zImibOX3jKDJIQFJCW679wBs1kEVCUuPP4K1sAG1PD2djvYfBEBDYnbbcuZ IOptJT6+OAa2i1fAUOLTbFWQsKiAssTfw/fA5ksIPGWVmH+kiXkCo+AsJOfNQjhvFpLzZiE5 bwEjyypG8dzEzBzdzDwjvdTSony9xIKCnFS95PzcTYygsPWYwrOD8eJBw0OMAhyMSjy8HHKv I4RYE8uKK3MPMUpwMCuJ8N6QBgrxpiRWVqUW5ccXleakFh9ilOZgURLnrU0xjxASSE8sSc1O TS1ILYLJMnFwSjUwcm8wWZ5hZm197vmRt3MUZOpKmrcXXZZMcHFrubZtQkmyCMMX238HL9gd 2RombfjduLgqVyGdPeeNnvCPy8YbpMX23tJ05thueVg+9EGv4LKYvCkWXpXbduQImDXmvDE1 Lk8Xqf2RfM/57PamxOfLQl99UF//nvfY+cA5/3+/a1bhjRYsO9ClxFKckWioxVxUnAgA+5Oc 0lcCAAA=
X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprDLMWRmVeSWpSXmKPExsUi6MSzVHem8usIg2ev2Cw+r77O5sDosWTJ T6YAxigum5TUnMyy1CJ9uwSujCm/37IUXJCs2N38iaWBcZpIFyMnh4SAicTKzkvMXYxcHEIC qxkl/j1cyAqS4BUQlPgx+R5LFyMHB7OAvMTB87IgYWYBaYlHf2ewQ9haEt8ftbJA9C5jkjh7 6S0zSEJYQFKi+84dMJtFQFXiwuOvYA1sQA1vb7eDzRcR0JC43bacCaLeVuLji2Ngu3gFDCU+ zVYFCYsKKEv8PXyPZQIj3ywkF81CuGgWkotmIbloASPLKkbRotScxEojvdTSony9xIKCnFS9 5PzcTYzgMDPn2cH46qDhIUYBDkYlHt7/cq8jhFgTy4orcw8xSnAwK4nw3pAGCvGmJFZWpRbl xxeV5qQWH2KU5mBREuctPXouQkggPbEkNTs1tSC1CCbLxMEp1cDovaXEYW5k2IWF64vzll+0 W5Wh/oL/W03Gt09aMSdzSl54FYtEs6z5NnfVtjvKYvPy2aPWhRcpLf8uP3HB/gnpe64/Prn/ Ucf6xBlbPHeUL99suL3p/vsyrqBZnZ0Nx73fneR6dXRCprNb3ObN7dcuzZi7QHTN3d8tM5/H eq358yGMS0Vga8v8g0osxRmJhlrMRcWJAIBglB4vAgAA
Archived-At: <https://mailarchive.ietf.org/arch/msg/multipathtcp/Mo-bvEcMEodPph8rTpylFMQ85IE>
Subject: [multipathtcp] RFC6824bis - New Handshake - Implementation feedback
X-BeenThere: multipathtcp@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Multi-path extensions for TCP <multipathtcp.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/multipathtcp>, <mailto:multipathtcp-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/multipathtcp/>
List-Post: <mailto:multipathtcp@ietf.org>
List-Help: <mailto:multipathtcp-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/multipathtcp>, <mailto:multipathtcp-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 10 Apr 2017 06:18:06 -0000


as mentioned during IETF'97 meeting, I implemented the new
MP_CAPABLE-handshake from RFC6824bis in the Linux Kernel.

Overall, it is not that complex to implement the new MP_CAPABLE-option. I am
currently still fighting with a bug, but once this is fixed I will work on
open-sourcing my code.

Nevertheless, I have some feedback that might be good to add
into RFC6824bis and also some notes of the style "watch out, this is tricky
in the implementation!" :

page 16:
* "If, however, A..." - it might be good to show here again the new
  MP_CAPABLE-option (or refer to Figure 4 two pages higher). I was looking
  for the figure but didn't really found it right away.

* "If B has data to send first, then the reliable delivery of the ACK
   can be inferred by the receipt of this data with an appropriate MPTCP
   Data Sequence Signal (DSS) option (Section 3.3)."

   Actually, it is the reception of a DATA_ACK that tells A that B has
   received the MP_CAPABLE. Not (as the above sentence states), the
   reception of a DSS-option.

* It might be good to go through some scenarios with the new
  MP_CAPABLE-option. For example:
  A sends MP_CAPABLE in the third ACK (B receives this thid ACK) and then
  (a bit later) sends data + the new MP_CAPABLE-option (this latter packet gets
  lost). Then, B sends data to A with a DATA_ACK.

  A receives the data sent by B with the DATA_ACK. Now, A's
  retransmission-timer expires and it tries to resend the data. The
  question now is: Should it use a regular DSS-option in the retransmission,
  or should it use the new MP_CAPABLE-option.

  It ultimately does not matter (the server should support receiving both),
  but it's an interesting sceanrio that might be worth describing IMO.

* We should describe, how this works with TSO. In particular, when using
  TSO, the new MP_CAPABLE-option will be copied to all segments (by the
  TSO-NIC). Now, when we have to retransmit one of these segments due to
  packet-loss, the sender might use a DSS-option this time or the new
  MP_CAPABLE-option (depending on whether or not he received a DATA_ACK).

  Basically, these last two points mean that the new MP_CAPABLE-option and
  DSS-option are "interchangeable". Meaning, if at one point I start sending
  data (sequence-number 1 to 100) and thus use the new MP_CAPABLE-option,
  later on I might decide to retransmit any byte within this sequence-range
  (1 to 100) with a regular DSS-option (if the server sent me a DATA_ACK
  back). Doing it that way, makes things on the sender-side a bit easier
  from an implementation-point-of-view.

* When using TFO, it might be worth to specify that the server cannot send a
  DATA_ACK as long as he hasn't received the MP_CAPABLE with the key by the

* What if the client wants to send an ADD_ADDR before sending the new
  MP_CAPABLE? It should be fine, but again the server should be graceful to

* The server still has to support receiving out-of-order data (aka, data
  with a DSS-option). So, he will have to queue this and wait for the new
  MP_CAPABLE-option. This is nothing special from an implementation
  point-of-view, but might be worth mentioning in the RFC.