Re: [multipathtcp] Regarding rate control at a subflow level

Gregory Detal <> Thu, 06 June 2019 10:24 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 6C0D1120219 for <>; Thu, 6 Jun 2019 03:24:46 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.889
X-Spam-Status: No, score=-1.889 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_REMOTE_IMAGE=0.01] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id GSdlHdWykDcl for <>; Thu, 6 Jun 2019 03:24:43 -0700 (PDT)
Received: from ( [IPv6:2607:f8b0:4864:20::344]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 76895120192 for <>; Thu, 6 Jun 2019 03:24:43 -0700 (PDT)
Received: by with SMTP id j19so1467252otq.2 for <>; Thu, 06 Jun 2019 03:24:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=co88utbG0ANKR9MMg9Nw7vJWdrDEDhE4OasguTgEjKs=; b=gjLLO2AIeca/82XS1UyzLTAELlzlQ1jnln9fApF4N8hUDMNdQeMWFppwjqmvl8UQ8A oRgOfCMmjBSqgu+msSz6zjUuwRDUfYOtVyW5B0oKWvRPXH1TTcSsEFyB5GnhQbqM1BuS EAs5YaJ6uipnquLkb7iCDWpZfjFsrDcm4Oq2v+jRh3pY/o/ioHKA1bjHAgwgTuH1o18J tQeuRRfw95xnpqQLaGyVCNy/GtQK2j2D3v8Crz3oFIMRXAbIaC5heDJ35GJgsiDkvtcP 72GmhbEOigWY74XycZotk+RRkf9MUmEpSNsRYQUb3tpJOrgoJaZ1iQLvwverh0yOLHVI RvPg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=co88utbG0ANKR9MMg9Nw7vJWdrDEDhE4OasguTgEjKs=; b=qkqN7iggB9Q7YeenemxW50C84fUBiQgtfdFAmWq4G0QkgElNzOOOhOuN7pP6Ql2Bq0 /knlJNUETAfzVpXztBTtyO4uGKXYQN0JOwW26cMLUwidBC+KnK6Ds+jQCHtVK8kuvJjk FV1V1zy+qs6odjVht/N2afzMZjUGQESeiANasaWu+0WK9lK7oeBq5VeXURWKDCCtchnz jLLfAhIp36BUNvbRxEYKnH6U6BAGOkjBb6ha6+7sYayQvxH6YFrJE7GMvtHFMBBMnrhM r/QTeijGdhgBigTslBIZ4/LG6qQpWpiwmKpvUKjKTCA9CjDH8nE7BEqFzyTtNatt0tyB tW0g==
X-Gm-Message-State: APjAAAWG9WxjV6X9UPgyxCBHLYevg1x/IX48YAIpgIwqRFMQBABpCLMc lL1eyCbc42Idg2b1W0UXsfjo20jHk9tinI1TidWRFKR4Ea03LwIvJ/RuTc1aAPg0nML/jVXaLJN K8WrwZj7LXvrkbIjuj/n3WtRN
X-Google-Smtp-Source: APXvYqxiCt+xBsmYu7o7dhKg9BwSvJeJtZarommBgpvzgXbD6nvS95PcjO8cl2tS5jB0q0O74LFjBbX8vGB2yCKVLxU=
X-Received: by 2002:a9d:7147:: with SMTP id y7mr13201295otj.152.1559816682539; Thu, 06 Jun 2019 03:24:42 -0700 (PDT)
MIME-Version: 1.0
References: <> <> <20190520135014.GG41806@MacBook-Pro-64.local> <LO2P123MB1965EA246F2652E9D5C47731EB180@LO2P123MB1965.GBRP123.PROD.OUTLOOK.COM>
In-Reply-To: <LO2P123MB1965EA246F2652E9D5C47731EB180@LO2P123MB1965.GBRP123.PROD.OUTLOOK.COM>
From: Gregory Detal <>
Date: Thu, 06 Jun 2019 12:24:06 +0200
Message-ID: <>
To: Philip Eardley <>
Cc:, Olivier Bonaventure <>, multipathtcp <>,,
Content-Type: multipart/alternative; boundary="000000000000472282058aa51f89"
Archived-At: <>
Subject: Re: [multipathtcp] Regarding rate control at a subflow level
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Multi-path extensions for TCP <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Thu, 06 Jun 2019 10:24:46 -0000

Hi Phil,

On Thu, May 30, 2019 at 7:53 PM <> wrote:

> I think the use case a hybrid operator could be interested in is the
> following. Similar to Alexander's use case.
> We want to provide a particular customer with broadband access that is
> faster than their DSL alone can provide, for instance to meet some minimum
> rate to meet a target of say 20Mbps. The rate can't quite be met by DSL,
> therefore cellular or even satellite is used as a top-up. DSL capacity is
> much cheaper than cellular /satellite, therefore the ideal scheduler would
> favour DSL.

This is exactly the type of deployments that we have on the field.

> It would react reasonably quickly to increased total load above the DSL
> rate (but not so fast that the instantaneous rate from a variable rate
> source 'kicks in' the cellular when the rate over a slightly longer time
> can be met by DSL).

There are 2 possible levels: (1) macro: create a subflow when the load
reaches the DSL rate and (2) micro: schedule packets first on the DSL
For (1) the CPE monitors the load on the DSL link  If the load is lower
than a minimum threshold, then no subflow is established over the LTE
network. If the load is above a maximum threshold, the CPE creates an
additional subflow over the LTE network. Note that (2) is always enabled as
as soon as the 2 subflows are available, the packet scheduler becomes
important. It first tries to fill the DSL link and once it is full, it
overflows over the LTE subflow.
We have deployments where (1) and (2) are enabled and others where only (2)
is enables. What is always important for a customer is that a Speedtest
always display the aggregated speed. In the first deployment mode, the
speed is increased in the middle of the test while with the second mode the
boost is observed directly.

> Note that the DSL rate is not completely static.  It supports deployments
> with a proxy in the home gateway and in the network (under the control of
> the operator),

This corresponds to the currently deployed Hybrid Access Networks.

> and deployments where there's only the proxy in the network (ie MPTCP is
> in the multi-interface phone).

This corresponds to the future ATSSS service in 5G networks that
will leverage the 0-rtt convert protocol that was started in the mptcp
working group and is now discussed within the tcpm working group.

> Possible to ensure that an individual flow goes over only one access.
> Downward direction (from network to house) more important than upward.  In
> a long-running session using a lot of bandwidth, because the amount of
> other traffic varies, it may be that this session sometimes uses just the
> DSL and sometimes is spread over both accesses.

Our experience shows that the scheduler that prioritises the DSL link is
sufficient to ensure that the traffic automatically moves to the DSL
network when the load decreases.

> On a minor point, ideally a speed test should measure the rate correctly
> (I don’t mean the scheduler identifies a speed test and favours it; rather,
> ideally the speed test shouldn’t eg measure just the DSL rate -
> alternatively re-define the speed test).

The Ookla speedtest is considered as the reference tool by customer and
typically opens n parallel TCP connections to saturate the available
network. Each of these connections typically creates and additional subflow
and both the DSL and the LTE link are used and tested. As said above, when
using the second deployment mode, both the DSL and the aggregated speed can
be seen but is is often tricky to tune properly. It could therefore be
interesting to think about a different and more reliable methodology to
measure hybrid access networks.

> So in terms of Olivier's question below, I think this means that the limit
> is about the total and not per subflow.

Looking at the downstream traffic, it seems that you need to limit the rate
of the incoming traffic towards a specific IP address. This can be
implemented on the internet facing side of a HAG. This way, the total rate
of the DSL+LTE is limited because the split of the traffic is done in the
HAG (DSL is filled first). This can be done locally on the HAG and does not
require any MPTCP signalling. If IPv4 and IPv6 are used, then the rate
should apply to all the addresses assigned to the same CPE.

> In terms of mechanism, I'm open to whatever is most suitable. I'm very
> interested if the best method would mean that ideally some new
> functionality is added to the MPTCP standard (and by implication to

It is possible to add signaling to MPTCP and MPQUIC, but those signals
operate on a per-connection basis. It is difficult to have a signal that
affects a group of connections. Another point that needs to be addressed is
the trust of the device that sends the signal. In an hybrid access network,
should the CPE send inside an MPTCP option the maximum rate at which the
HAG should deliver data ? If you trust all CPEs, maybe, but if customers
can select their own CPE or modify them, there are some risks which such

> Best wishes,
> phil
> -----Original Message-----
> From: multipathtcp [] On Behalf Of
> Christoph Paasch
> Sent: 20 May 2019 14:50
> To: Olivier Bonaventure <>
> Cc:; Ashutosh prakash <>;
> Nagesh shamnur <>
> Subject: Re: [multipathtcp] Regarding rate control at a subflow level
> Hello,
> On 17/05/19 - 09:59:54, Olivier Bonaventure wrote:
> > Dear Nagesh,
> > >
> > >                  Greetings. In case of Mobile deployments of MPTCP,
> > > though the data rates are getting cheaper, still it would be wise
> > > not to run the cellular path to full limit but to throttle to a
> > > certain extent considering cost in mind or if server wants to limit
> > > the client at subflow level, then I couldn’t find the support for
> > > the same in the specification.
> >
> > We have developed several prototypes that include this capability in
> > the Linux kernel.
> >
> > > So, while going through the discussion archives, could only find
> > > that, the peer(server) can throttle the speed for the entire
> > > connection by publishing a smaller receiver window rather than for a
> > > particular subflow. I feel, it would be a good idea if the peers can
> > > exchange this information using the control packets.
> >
> > We could imagine an MPTCP option that provides the maximum rate on a
> > per-subflow level, but I was wondering whether the use case is not to
> > limit the bandwidth on the smartphone at the link level (i.e. multiple
> > tcp connections or udp flows) and not at the subflow level. Could you
> > precise your use case for the subflow level ?
> another use-case for rate-control I see is when a client wants to tell a
> sender to gracefully close a subflow.
> - Sending a TCP-RST results in packet-loss of the in-flight data.
> - Reducing the window is going to stall the whole connection because the
> window is shared.
> - Setting MP_PRIO also won't work because none might want to drain a
> secondary
>   subflow even if the "primary" subflow is having severe packet-loss.
> Thus, a "maximum-rate" option on a per-subflow level would allow to send
> the rate to 0, which would drain the subflow.
> Christoph
> _______________________________________________
> multipathtcp mailing list
> _______________________________________________
> multipathtcp mailing list

[image: Tessares SA] <> Gregory Detalº | Co-Founder
& CTO | +32 486 83 50 51 <>
Tessares SA | Hybrid Access Solutions
1 Avenue Jean Monnet, 1348 Louvain-la-Neuve, Belgium

ºTRAAL Consulting SPRL