Re: Preparing for discussion on what to do about the multipath extension milestone

[Spencer Dawkins at IETF <[e-mail]>]

Hi, Jana,

On Mon, Oct 5, 2020, 21:37 Jana Iyengar <[e-mail]> wrote:

> After going through this thread, I'm still finding the framing of the
> conversation to be: How do we get "multipath" into QUIC? Perhaps I'm not
> being very charitable, but the title of this email thread isn't helping.
>

Thanks for your helpful thoughts below.

Since the title of this thread was mine, it's fair to say that what I was
hoping for, was to identify the reasons people were thinking that dropping
the multipath charter deliverable was the right thing to do.

https://mailarchive.ietf.org/arch/msg/quic/cjjI9-uP6WXVI8p28pcTcq8vS-E/

Obviously, we moved far beyond what I was hoping for, without changing the
subject line ...

Best,

Spencer

This framing is problematic and leads to the false scarcity of options we
> see in this discussion, but I am still stuck at the premise. Let me explain.
>
> Transport multipath is not a _problem_, it is a _class of solutions_ that
> can be implemented in a transport. Multipath is simply the use of multiple
> paths. The manner in which multiple paths are used depends on the problem
> you are trying to solve.
>
> For example, if you want to solve the handoff problem for reliability,
> connection migration is a transport-level multipath solution for it. Note
> that there are other ways to solve this problem. For example, if the
> application is RESTful and simply fetching data (e.g. YouTube), then you
> can implement this in the application quite easily. However, for other
> applications, such as live transcription, where context is super important
> and transport state is tied to application state, maintaining connection
> state is necessary. We agreed that the problem was important -- I believe
> that was in Melbourne -- and we then went about solving it with connection
> migration with which, after many iterations, we now have a fragile peace.
>
> So echoing what others have been saying on this thread: What is the
> problem we are trying to solve? Do we agree on the importance of it? Is
> this bandwidth aggregation across WiFi/Cellular (is that an important
> enough use case?)? Is it latency reduction for HTTP? Is it increased
> reliability? In what ways is it more than connection migration?
>
> I don't want a laundry list of things that "multipath" can do. But an
> analysis of the most important problem and why it needs to be solved. This
> is where I'd like to engage, not around adoption of a solution, let alone
> adoption of a draft.
>
> I've still not heard folks rallying around one problem that we need to
> solve right now, and I'm not hearing people saying that we need to do
> this before we gain any experience with the existing multipath capabilities
> of QUICv1. I definitely believe that we ought to wait for experience with
> QUICv1.
>
> Note that the problems MPTCP has solved for TCP can be quite different
> from the problems it will need to solve with QUIC. For one, I agree with
> Lucas that not talking about streams is like not talking about ordered
> delivery with QUIC. Streams are QUIC's primary API model, and we need to
> understand how an endpoint does streams across paths. This isn't an MPTCP
> problem, but it is one we will need to solve when talking about QUIC. And I
> like to remind myself that we spend time in working groups on engineering
> and bit-coloring, and I suspect that there's plenty of that in just this
> one single problem.
>
> It's a fallacy to think that "this won't take too much time". While that
> is never a good answer to why we should work on anything, it rings
> especially hollow in the context of our experience building connection
> migration in QUIC: it took us a couple of years and we kept stumbling and
> fixing things, especially security vulnerabilities.
>
> Because I like to hammer the point in: let's motivate the problem first
> and agree on the shape of the problem and its importance. And let's see if
> we agree to prioritize it/them over the other things that we are already
> planning to work on (datagrams, version negotiation, deploying QUICv1 and
> digesting experience.)
>
> And I'll note this because the point has been raised. I agree with
> Christian that _if we were to build "general-purpose multipath", I am
> thinking of a far simpler (and potentially incremental) design -- maybe
> using a single PN space...
>
> ... but trying to solve the protocol problem is putting the cart before
> the horse. We are all good at protocol design, and we are all great at
> coloring each bit just the right shade, even though we are terrible at
> agreeing on what the right shade is. Before we dive into coloring the bits,
> let's please adequately motivate spending engineering time and resources
> doing it.
>
> - jana
>
> On Mon, Oct 5, 2020 at 6:24 AM Olivier Bonaventure <
> [e-mail]> wrote:
>
>> Matt,
>>
>>
>> >
>> >     Let me try with a simple example on a moving smartphone. The
>> >     application
>> >     will send small amounts of data and receive variable amounts of data
>> >     (depending on the type of requests).
>> >
>> > I want to start by saying this is a real usecase, and a problem we see
>> > very obviously through quality of experience outliers for people using
>> > our products.
>> >
>> >
>> >     We create a sending and a receiving uniflow on both the Wi-Fi and
>> the
>> >     cellular interface. The smartphone has two sending uniflows and the
>> >     server as well.
>> >
>> >     To send a short request, the client duplicates it over its two
>> sending
>> >     uniflows since it does not know which of the two uniflows will be
>> >     the best.
>> >
>> >     To return the response, the server could use the same scheduler if
>> the
>> >     response is short. However, if the response is long, this is not
>> very
>> >     efficient since data is sent over two paths. It could then use both
>> >     paths to send the data and get the lowest delay to deliver the
>> response.
>> >     This could be modulated by policies if the user pays on a per volume
>> >     basis over one path and not over the other.
>> >
>> > This somewhat hand-waves away the scheduling problem. Most Internet
>> > traffic flows in the direction of server -> client, where typical
>> mobile
>> > clients are obviously the ones with potentially multiple interfaces to
>> > the Internet. The client also has the most information about the likely
>> > quality of the first radio hop into the access network (e.g. signal
>> > quality). The client also _may_ know about things like data pricing.
>>
>> Agreed
>>
>> > However, the server is the one which is responsible for scheduling most
>> > data across these paths. To make good, proactive (rather than reactive)
>> > scheduling decisions the server needs to be fed this information as
>> > input to its scheduler. This seems a difficult thing to achieve in
>> > practice, and without it I wonder whether the complexity of "full"
>> > multipath will be worth it until we solve the signalling problem.
>>
>> There are several techniques that allow the server to learn about the
>> different performance of the paths with MPTCP. Those can be applied to
>> MPQUIC as well.
>>
>> If the application uses short requests and short responses, then a
>> simple heuristic for the server is to send the response on the path
>> where the last packet (data or ack) has been received. The reception of
>> a packet is a good indication that a link works.
>>
>> If the application uses longer responses, then the congestion control
>> used by the server over the two paths will enable it to easily find the
>> best way to spread the load. If one path has a longer rtt or losses,
>> then its congestion window will be slower than the other one and packets
>> will naturally flow on the best path, but not only on this path. This
>> does not require a specific scheduler and would work better than a
>> strict weighted-round-robin scheduler where the client would indicate
>> that it wants to receive 2/5 of the bytes over WiFi and 3/5 over cellular.
>>
>> > What
>> > Ian is suggesting above, I think, is essentially an Active/Passive
>> > extension to the existing connection migration mechanisms we have
>> today.
>> > What are your thoughts on this as an initial direction? It would allow
>> > operators a way to solve this particular problem with only mild
>> > modifications to the core protocol. Said another way, I think in theory
>> > an omniscient packet scheduler can make very intelligent decisions
>> which
>> > would definitely benefit application quality. In practice though I have
>> > concerns about how effective these schedulers will be versus the naive
>> > approach which could be thought of as "Active/Passive" or "Failover",
>> > eschewing bandwidth aggregation entirely. I'd also like to echo what
>> > Kazuho said, which is that "Multipath" can mean many things, and I'd
>> > prefer we narrow down the problem we want to solve in the WG, which
>> will
>> > drive our design direction.
>>
>> When coupled with congestion control, a simple packet scheduler such as
>> lowest-rtt first (if paths have the same cost) or priority (for the
>> lowest cost path) works well with MPTCP. The same would apply for MPQUIC
>>
>> >
>> >     Another example is the hybrid access network scenario with a DSL
>> and an
>> >     LTE path. There, the objective is to send data over the LTE path
>> only
>> >     when the DSL is full.
>> >
>> >     In this case, the solution would differ. The client would first
>> create
>> >     sending and receiving uniflows over the DSL path. It then monitors
>> the
>> >     usage of this path. As long as the DSL is not fully used for some
>> >     period
>> >     of time (e.g. one or a few seconds), all data flows over the DSL
>> path.
>> >     Once the DSL path is saturated, the client creates a receiving
>> uniflow
>> >     (and possibly a sending one if the DSL upstream is saturated) over
>> the
>> >     LTE path. The second path can be used to offload traffic. In
>> practice,
>> >     the client and the server use a priority scheduler to always prefer
>> the
>> >     DSL over the LTE path, see
>> >
>> >
>> https://inl.info.ucl.ac.be/publications/increasing-broadband-reach-hybrid-access-networks.html
>> >
>> > For these usecases, are you imagining that they would largely be used
>> > internally for Internet operators? I always struggle with the hybrid
>> > access examples, as they seem to assume a lot of knowledge about the
>> > underlying networks that typical endpoints can't simply intuit from the
>> > ether. The linked paper seems to suggest MPTCP as a proxy solution,
>>
>> Yes, that's deployed with MPTCP proxies running on home routers.
>>
>> > which I gather is largely the usecase things like ATSSS have for
>> MPQUIC.
>> > However, as a mobile application how do I know the DSL link is "full"
>> > and thus that I should create a uniflow over the LTE path? The same
>> > question applies to the server. It would be awesome if clients and
>> > servers had more active information about the underlying network's
>> state
>> > rather than being reactive, but beyond things like ECN this seems
>> > difficult to achieve in practice. If the Hybrid Access Network usecase
>> > of multipath presupposes a deployment in an operator's network then I
>> > would argue it is somewhat antithetical to the goals of QUIC, which
>> > deliberately puts more control at the endpoints.
>>
>> On endpoints, you can get the same result by having a target bandwidth
>> on the Wi-Fi interface. For example, a smartphone application could
>> assume that if the current Wi-Fi bandwidth is below x Mbps then it
>> should enable the LTE interface to boost a long download. Similarly, a
>> videostreaming application could have a target that corresponds to the
>> default quality chosen by the user and enable the LTE when it cannot
>> receive video at the expected quality. The same applies for a radio
>> streaming application.
>>
>>
>> Olivier
>>
>>