Re: What to do about multipath in QUIC

[Roberto Peon <[e-mail]>]

What the application can do, or what it can cause to be done depends on the API.
Does the application get a callback or have opportunity to take action when data has been received/when you’ve received and ack or nack’d?
Does it have a way to express to the lower level stack what it should do?

In recent memory, I’ve said that H2 priorities in their original form are dead or should be.
Within the past few days we’re seeing Ian mention Chrome killing server push.
I believe that we’ve not had the appropriate APIs to make these features useful in the browsers and sometimes in servers, and that has resulted in their death on some platforms, because there has been no reasonable way to use them.

I believe we’re likely to suffer exactly the same thing here—a feature that has obvious theoretical advantages, but which has tradeoffs in use.
Because it has tradeoffs, it will fail in applications or browsers without the appropriate API which allows the application to deal with those tradeoffs and play/innovate in the space.

-=R

From: Mikkel Fahnøe Jørgensen <[e-mail]>
Date: Wednesday, November 11, 2020 at 7:47 AM
To: "[e-mail]" <[e-mail]>, Roberto Peon <[e-mail]>, Lucas Pardue <[e-mail]>
Cc: QUIC WG <[e-mail]>, Olivier Bonaventure <[e-mail]>, Christian Huitema <[e-mail]>
Subject: Re: What to do about multipath in QUIC

You can’t do that at the app layer. You said yourself, wait for ACK on one path before sending on the other.

I can see a protocol that uses both QUIC and TPC to overcome UDP blocks, but then you could also argue that QUIC should support TCP as a transport fallback. I think that would go down the wrong path so to speak. It is better to let QUIC motivate passing UDP in more cases. So I’d rather have a super QUIC doing multiple paths.

There is of course WebRTC that uses all sorts of UDP/TCP patterns, but it is fairly complex to get right in all scenarios. Perhaps it is best to let WebRTC handle TCP and move forward with new tech for QUIC. Then you can hack WebRTC to also support QUIC if you can afford it.

On a related note:

I would be useful to be able to connect to an endpoint through multiple different public access points, i.e. load balancers. The LBs have a tendency to cut idle connections after a while, so being able to connect through multiple would avoid downtime on that account. This is sort of complicated because the connection establishment would see different IPs to the same endpiont, but I guess that is what multipath is?


Kind Regards,
Mikkel Fahnøe Jørgensen


On 10 November 2020 at 18.33.44, Roberto Peon ([e-mail]<mailto:[e-mail]>) wrote:
I believe it does solve the serialization problem, but let’s talk it out! 😊

My example of TCP+QUIC is the kind of thing you’d do when you were uncertain about one failing (perhaps UDP is being blocked).

What I’m talking about is a likely combination of two things:
1) Ability to get some new path or connection to a session (likely a single “server”, but that is the decision of the “server”).
- can also be within the connection, so long as the path is addressable by the application
- can be external to the app, where different connections are used.
2) Ability of application to schedule data to each path or session (i.e. to mux/demux onto the paths).
- This could be via a standardized config, or
- It could be up to the application to figure it out
- An API for determining how packets are scheduled seems necessary regardless of any other multipath implementation path.

An example standardized config could be something like:
  (race s=1 (path A) (delay 0.1 (path B)))
This would say that data should flow on path A by default (it is first), and if you’ve not gotten an ack in 100ms (0.1 s), try path B.

If one wanted to send data in duplicate on all paths:
  (race s=2 (path A)  (path B))
If one wanted to send the first X bytes on path A, and the rest on path B:
(partition X (path A) (path B))

This kind of config is something we’ve been using for a couple of years for video things, to good effect.
It is most certainly not perfect, but it does allow quite a bit of flexibility with a fairly minimal non-Turing complete config.
I’m not claiming we should use it, just showing it absolutely can be done in a way that supports about any application need without needing to write code everywhere.

-=R

From: Mikkel Fahnøe Jørgensen <[e-mail]<mailto:[e-mail]>>
Date: Tuesday, November 10, 2020 at 3:37 AM
To: Lucas Pardue <[e-mail]<mailto:[e-mail]>>, Roberto Peon <[e-mail]<mailto:[e-mail]>>, "[e-mail]<mailto:[e-mail]>" <[e-mail]<mailto:[e-mail]>>
Cc: Olivier Bonaventure <[e-mail]<mailto:[e-mail]>>, Christian Huitema <[e-mail]<mailto:[e-mail]>>, QUIC WG <[e-mail]<mailto:[e-mail]>>
Subject: Re: What to do about multipath in QUIC

But this doesn’t solve the serialization of a single stream over multiple paths.

Also, it doesn’t really make sense to mix a video stream on QUIC on one path and TCP on another. That would cause all kinds of problems, not to mention privacy.

I agree that there is a risk of a complex unnecesary feature being poorly implemented in QUIC. But true multipath cannot really be solved outiside of QUIC. I’m fine with making it optional or giving it dedicated version.

Someone suggested building multipath on top of multiple QUIC connections. I think that is viable if it is a newer QUIC version that delagates work to older QUIC versions. The key point is that externally this happens transparently, and there are optionas for the QUIC stacks to coordinate locally or remotely via a signalling path. Not sure which solution is ultimately the best, but you build upon what you already have.


Kind Regards,
Mikkel Fahnøe Jørgensen


On 9 November 2020 at 18.32.06, Roberto Peon ([e-mail]<mailto:[e-mail]>) wrote:
I’m still concerned that we’re looking at solving this inside the connection, instead of providing a way for this to be solved irrespective of the connection.
There is a fundamental routing problem we have here that we could address (addressing a session), but we’re not addressing with what I’m seeing discussed (addressing a session within the same connection object).

If we consider this problem as making the session addressable, then applications can do it the way that makes sense for them, without having to put everything in every stack everywhere, plus new APIs to actually make them work.

I’m afraid if we add multipath, it’ll be like what happened with server push. The lack of appropriate APIs made using it with the browser fraught with tradeoffs with no reasonable way for an application to fix.

Solve the addressing-of-a-session problem, however, and we make it easier to solve the likely API problem that will accompany multipath.

Example:
I could have a virtual connection which is composed of a TCP connection on path A, and a QUIC connection on path B.
.. or maybe I want to try out a new version/extension on QUIC, so I have a virtual connection with QUIC and QUIC+extension.
I could declare that I’d like for data to flow down QUIC+extension path unless that is too slow, then duplicate the data onto the QUIC path.

In my mind, the application should establishes the virtual connection, and provide at least one path, and can optionally add (and remove) subsequent paths.
This is something we do already in “storage-land”, where diversity and separable failure domains are important, and where the use-cases are extremely diverse in latency, data-amounts, and cost.
-=R

From: QUIC <[e-mail]<mailto:[e-mail]>> on behalf of Behcet Sarikaya <[e-mail]<mailto:[e-mail]>>
Reply-To: "[e-mail]<mailto:[e-mail]>" <[e-mail]<mailto:[e-mail]>>
Date: Monday, November 9, 2020 at 8:58 AM
To: Lucas Pardue <[e-mail]<mailto:[e-mail]>>
Cc: Christian Huitema <[e-mail]<mailto:[e-mail]>>, Behcet Sarikaya <[e-mail]<mailto:[e-mail]>>, QUIC WG <[e-mail]<mailto:[e-mail]>>, Olivier Bonaventure <[e-mail]<mailto:[e-mail]>>, Mikkel Fahnøe Jørgensen <[e-mail]<mailto:[e-mail]>>
Subject: Re: What to do about multipath in QUIC

Hi Lucas, Olivier,


On Mon, Nov 9, 2020 at 10:51 AM Lucas Pardue <[e-mail]<mailto:[e-mail]>> wrote:
Hey Olivier,

On Mon, Nov 9, 2020 at 4:31 PM Olivier Bonaventure <[e-mail]<mailto:[e-mail]>> wrote:
Lucas,
>
> On Mon, Nov 9, 2020 at 3:55 PM Behcet Sarikaya <[e-mail]<mailto:[e-mail]>
> <mailto:[e-mail]<mailto:[e-mail]>>> wrote:
>
>     Hi Folks,
>     I agree with Mikkel's points.
>     To Lucas: I meant my short mail sometime ago I think it was before
>     the interim (?) where I explained that connection migration is
>     mobility support which should (from layering point of view) be in IP
>     layer. In fact if IP layer has this support then then no need for
>     connection migration in QUIC, so those procedures in the code do not
>     get executed.
>
>     Multipath is multiple interface support. It seems more and more like
>     multipath probably better belongs in transport layer. Traffic in
>     each interface may go over different networks (in my case on over T
>     Mobile and the other AT&T). I believe a different PN is well
>     justified in multipath as we have it in the base draft because of
>     these traffic conditions (no offense to Christian).
>
>
> I still don't see why the current features of connection migration are
> not in some way a form of multipath.

You are right, connection migration is the weakest form of multipath.

Thanks. We heard use cases that would like stronger forms. I think it will help continue to move the discussion forward if we can establish some common ground on terms and capabilities.

This paragraph of RFC6824 then continues as follows :

    However, to the network layer, each MPTCP subflow looks
    like a regular TCP flow whose segments carry a new TCP option type.
    Multipath TCP manages the creation, removal, and utilization of these
    subflows to send data.  The number of subflows that are managed
    within a Multipath TCP connection is not fixed and it can fluctuate
    during the lifetime of the Multipath TCP connection.

This is not really connection migration and MPTCP provides much more
multipath capabilities than connection migration.

Yeah I follow. As someone coming from QUIC, the first sentence is kind of easily negated (which is a benefit IIUC). I think the remainder of the paragraph is partially satisfied by QUIC v1 if we consider PATH_CHALLENGE/PATH_RESPONSE and NEW_CONNECTION_ID/RETIRE_CONNECTION_ID. But it starts to fall apart when you want to do more complicated things. I think understanding the gaps in the transport signalling would be useful to document in isolation to any specific solution. draft-deconinck-quic-multipath has done some of that work already but it gets a little too tied up with the solution IMO.



I don't think Olivier would wish to undermine the most important feature of multipath: multiple paths going over concurrently possible over different networks.
Then he can not justify many features in draft-deconinck-quic-multipath.

Behcet
Cheers
Lucas