Re: Preparing for discussion on what to do about the multipath extension milestone

[Jana Iyengar <[e-mail]>]

After going through this thread, I'm still finding the framing of the
conversation to be: How do we get "multipath" into QUIC? Perhaps I'm not
being very charitable, but the title of this email thread isn't helping.

This framing is problematic and leads to the false scarcity of options we
see in this discussion, but I am still stuck at the premise. Let me explain.

Transport multipath is not a _problem_, it is a _class of solutions_ that
can be implemented in a transport. Multipath is simply the use of multiple
paths. The manner in which multiple paths are used depends on the problem
you are trying to solve.

For example, if you want to solve the handoff problem for reliability,
connection migration is a transport-level multipath solution for it. Note
that there are other ways to solve this problem. For example, if the
application is RESTful and simply fetching data (e.g. YouTube), then you
can implement this in the application quite easily. However, for other
applications, such as live transcription, where context is super important
and transport state is tied to application state, maintaining connection
state is necessary. We agreed that the problem was important -- I believe
that was in Melbourne -- and we then went about solving it with connection
migration with which, after many iterations, we now have a fragile peace.

So echoing what others have been saying on this thread: What is the problem
we are trying to solve? Do we agree on the importance of it? Is this
bandwidth aggregation across WiFi/Cellular (is that an important enough use
case?)? Is it latency reduction for HTTP? Is it increased reliability? In
what ways is it more than connection migration?

I don't want a laundry list of things that "multipath" can do. But an
analysis of the most important problem and why it needs to be solved. This
is where I'd like to engage, not around adoption of a solution, let alone
adoption of a draft.

I've still not heard folks rallying around one problem that we need to
solve right now, and I'm not hearing people saying that we need to do
this before we gain any experience with the existing multipath capabilities
of QUICv1. I definitely believe that we ought to wait for experience with
QUICv1.

Note that the problems MPTCP has solved for TCP can be quite different from
the problems it will need to solve with QUIC. For one, I agree with Lucas
that not talking about streams is like not talking about ordered delivery
with QUIC. Streams are QUIC's primary API model, and we need to understand
how an endpoint does streams across paths. This isn't an MPTCP problem, but
it is one we will need to solve when talking about QUIC. And I like to
remind myself that we spend time in working groups on engineering and
bit-coloring, and I suspect that there's plenty of that in just this one
single problem.

It's a fallacy to think that "this won't take too much time". While that is
never a good answer to why we should work on anything, it rings especially
hollow in the context of our experience building connection migration in
QUIC: it took us a couple of years and we kept stumbling and fixing things,
especially security vulnerabilities.

Because I like to hammer the point in: let's motivate the problem first and
agree on the shape of the problem and its importance. And let's see if we
agree to prioritize it/them over the other things that we are already
planning to work on (datagrams, version negotiation, deploying QUICv1 and
digesting experience.)

And I'll note this because the point has been raised. I agree with
Christian that _if we were to build "general-purpose multipath", I am
thinking of a far simpler (and potentially incremental) design -- maybe
using a single PN space...

... but trying to solve the protocol problem is putting the cart before the
horse. We are all good at protocol design, and we are all great at coloring
each bit just the right shade, even though we are terrible at agreeing on
what the right shade is. Before we dive into coloring the bits, let's
please adequately motivate spending engineering time and resources doing it.

- jana

On Mon, Oct 5, 2020 at 6:24 AM Olivier Bonaventure <
[e-mail]> wrote:

> Matt,
>
>
> >
> >     Let me try with a simple example on a moving smartphone. The
> >     application
> >     will send small amounts of data and receive variable amounts of data
> >     (depending on the type of requests).
> >
> > I want to start by saying this is a real usecase, and a problem we see
> > very obviously through quality of experience outliers for people using
> > our products.
> >
> >
> >     We create a sending and a receiving uniflow on both the Wi-Fi and the
> >     cellular interface. The smartphone has two sending uniflows and the
> >     server as well.
> >
> >     To send a short request, the client duplicates it over its two
> sending
> >     uniflows since it does not know which of the two uniflows will be
> >     the best.
> >
> >     To return the response, the server could use the same scheduler if
> the
> >     response is short. However, if the response is long, this is not very
> >     efficient since data is sent over two paths. It could then use both
> >     paths to send the data and get the lowest delay to deliver the
> response.
> >     This could be modulated by policies if the user pays on a per volume
> >     basis over one path and not over the other.
> >
> > This somewhat hand-waves away the scheduling problem. Most Internet
> > traffic flows in the direction of server -> client, where typical mobile
> > clients are obviously the ones with potentially multiple interfaces to
> > the Internet. The client also has the most information about the likely
> > quality of the first radio hop into the access network (e.g. signal
> > quality). The client also _may_ know about things like data pricing.
>
> Agreed
>
> > However, the server is the one which is responsible for scheduling most
> > data across these paths. To make good, proactive (rather than reactive)
> > scheduling decisions the server needs to be fed this information as
> > input to its scheduler. This seems a difficult thing to achieve in
> > practice, and without it I wonder whether the complexity of "full"
> > multipath will be worth it until we solve the signalling problem.
>
> There are several techniques that allow the server to learn about the
> different performance of the paths with MPTCP. Those can be applied to
> MPQUIC as well.
>
> If the application uses short requests and short responses, then a
> simple heuristic for the server is to send the response on the path
> where the last packet (data or ack) has been received. The reception of
> a packet is a good indication that a link works.
>
> If the application uses longer responses, then the congestion control
> used by the server over the two paths will enable it to easily find the
> best way to spread the load. If one path has a longer rtt or losses,
> then its congestion window will be slower than the other one and packets
> will naturally flow on the best path, but not only on this path. This
> does not require a specific scheduler and would work better than a
> strict weighted-round-robin scheduler where the client would indicate
> that it wants to receive 2/5 of the bytes over WiFi and 3/5 over cellular.
>
> > What
> > Ian is suggesting above, I think, is essentially an Active/Passive
> > extension to the existing connection migration mechanisms we have today.
> > What are your thoughts on this as an initial direction? It would allow
> > operators a way to solve this particular problem with only mild
> > modifications to the core protocol. Said another way, I think in theory
> > an omniscient packet scheduler can make very intelligent decisions which
> > would definitely benefit application quality. In practice though I have
> > concerns about how effective these schedulers will be versus the naive
> > approach which could be thought of as "Active/Passive" or "Failover",
> > eschewing bandwidth aggregation entirely. I'd also like to echo what
> > Kazuho said, which is that "Multipath" can mean many things, and I'd
> > prefer we narrow down the problem we want to solve in the WG, which will
> > drive our design direction.
>
> When coupled with congestion control, a simple packet scheduler such as
> lowest-rtt first (if paths have the same cost) or priority (for the
> lowest cost path) works well with MPTCP. The same would apply for MPQUIC
>
> >
> >     Another example is the hybrid access network scenario with a DSL and
> an
> >     LTE path. There, the objective is to send data over the LTE path only
> >     when the DSL is full.
> >
> >     In this case, the solution would differ. The client would first
> create
> >     sending and receiving uniflows over the DSL path. It then monitors
> the
> >     usage of this path. As long as the DSL is not fully used for some
> >     period
> >     of time (e.g. one or a few seconds), all data flows over the DSL
> path.
> >     Once the DSL path is saturated, the client creates a receiving
> uniflow
> >     (and possibly a sending one if the DSL upstream is saturated) over
> the
> >     LTE path. The second path can be used to offload traffic. In
> practice,
> >     the client and the server use a priority scheduler to always prefer
> the
> >     DSL over the LTE path, see
> >
> >
> https://inl.info.ucl.ac.be/publications/increasing-broadband-reach-hybrid-access-networks.html
> >
> > For these usecases, are you imagining that they would largely be used
> > internally for Internet operators? I always struggle with the hybrid
> > access examples, as they seem to assume a lot of knowledge about the
> > underlying networks that typical endpoints can't simply intuit from the
> > ether. The linked paper seems to suggest MPTCP as a proxy solution,
>
> Yes, that's deployed with MPTCP proxies running on home routers.
>
> > which I gather is largely the usecase things like ATSSS have for MPQUIC.
> > However, as a mobile application how do I know the DSL link is "full"
> > and thus that I should create a uniflow over the LTE path? The same
> > question applies to the server. It would be awesome if clients and
> > servers had more active information about the underlying network's state
> > rather than being reactive, but beyond things like ECN this seems
> > difficult to achieve in practice. If the Hybrid Access Network usecase
> > of multipath presupposes a deployment in an operator's network then I
> > would argue it is somewhat antithetical to the goals of QUIC, which
> > deliberately puts more control at the endpoints.
>
> On endpoints, you can get the same result by having a target bandwidth
> on the Wi-Fi interface. For example, a smartphone application could
> assume that if the current Wi-Fi bandwidth is below x Mbps then it
> should enable the LTE interface to boost a long download. Similarly, a
> videostreaming application could have a target that corresponds to the
> default quality chosen by the user and enable the LTE when it cannot
> receive video at the expected quality. The same applies for a radio
> streaming application.
>
>
> Olivier
>
>