Re: Multi-path QUIC Extension Experiments

[Yunfei Ma <[e-mail]>]

Hi Mikkel,

Thanks for the question. Let me clarify below:

>
> Sorry, but this doesn’t make sense to me.
> In earlier mails it was stated that HoL happens even if all packets of one
> stream is always sent on the same path. If you have time dependent data
> like a video frame, and the frame is not, say, interlaced, then all the
> data should be placed on the same stream or at least the same path. Of
> course you can using interlacing on multiple streams and choose to display
> a lower quality frame if HoL is detected. So if you want to use bandwidth
> sharing across paths, then yes, there might be HoL blocking.
>

In our case, each stream is allowed to transmit on multiple paths (Please
note that this is an implementation choice. The draft does allow you to
choose other strategies. I remember that Spencer Dawkins also had a draft
about the difference between switching, steering and aggregation.). We use
aggregation for a reason, if all packets of one stream are always sent on
the same path, the stream that is unfortunate to get a bad path(small
bandwidth, high loss or high latency) will suffer a lot. In such a
scenario, the overall robustness is not improved. I think an important goal
of using multi-path for wireless transport is to make your network more
stable. The elastic resource pooling ability when allowing each stream to
transmit on multiple paths is quite helpful; You are right. If you want to
enable bandwidth sharing, then, you need to deal with HoL blocking.

But I am interested in understanding what MP-HoL means when streams are not
> distributed on multiple paths because then at least the transport layer
> should not exhibit blocking beyond flow control. It can be that the
> application protocol has stream dependencies that introduce HoL blocking,
> but that is kind of a separe discussion, relevant as it may be.
>

I think the stream dependencies you mentioned here is a great point. In our
implementation, we introduced a stream-priority based reinjection which
tries to address such dependency (There is a figure in the material that
Yanmei sent). But we haven't tried when each stream is limited to a single
path. In our case, streams are distributed on multiple paths. I would
definitely want to hear more about the application you are dealing with,
and maybe for wired transport, such a design is needed.

Cheers,
Yunfei

>
> On 18 Jul 2021, at 10.16, Yunfei Ma <[e-mail]> wrote:
>
> Hi Charles, Roberto, and Mirja:
>
> Thanks a lot for your questions. As all three of you are curious about the
> definition of MP-HoL, I am putting my answer into one reply.
>
> Short answer: the MP-HoL is not because of flow control, but rather, it is
> related to the nature of path heterogeneity. In other words, MP-HoL can
> happen when flow control limit is not reached (as pointed out by Charles,
> you can set a large limit on the client side).
>
> More specifically, when you want to send out packets on different paths at
> the same time, there is a scheduler to decide how to split your packets and
> put them on different paths. However, in mobile networks, the network paths
> could have very different path delays. MP-HoL blocking arises when the
> packets sent earlier at the slow path arrive later than the packets sent
> later at the fast path, causing out-of-order arrival. As a consequence, the
> out-of-order packets are not eligible to be submitted to applications, so
> the fast path has to wait.
>
> For example, say we want to send out two packets that belong to the same
> video frame with a min-RTT scheduler, which is default in MPTCP. For
> each packet, the scheduler selects a path for that packet to transmit. The
> selection has two criterias: (1) the path's congestion window is not full
> and (2) the path selected has a smaller RTT than the other. If somehow, at
> the moment of transmitting, the fast path's cwnd is full (some traffic has
> been sent before), the first packet is then put on the slow path by the
> scheduler. Later, an ACK is received and the fast path becomes available,
> so the scheduler puts the second packet on the fast path. As a result,
> there is an out-of-order arrival.
>
> What makes the problem even more difficult is that in mobile networks, the
> RTTs can change quickly, which makes accurate prediction very difficult.
> Worst case is that when the scheduler thinks it is using the fast path, it
> is actually using the slow path instead. As you can see, in order to make
> multi-path transport efficient, it is important to solve this problem and
> that's what we are doing in this project .
>
> I hope I have answered your questions. If not, please let me know.
>
> Cheers,
> Yunfei
>
>
>
> On Fri, Jul 16, 2021 at 12:51 PM Charles 'Buck' Krasic <
> [e-mail]> wrote:
>
>> "don't overcommit" includes the common practice of setting very large
>> limits on the client side, where in aggregate the case of server being flow
>> control limited is effectively non-existent.
>>
>> I am curious to hear clarification of the precise definition of MP-HoL
>> blocking here.  is it not flow control, but rather path aliasing where
>> distinct paths are actually sharing some physical link(s)?
>>
>> On Fri, Jul 16, 2021 at 12:13 PM Roberto Peon <fenix=
>> [e-mail]> wrote:
>>
>>> I too am curious!
>>> There are only two ways to handle flow control—overcommit, or don’t
>>> overcommit.
>>>
>>> The “don’t overcommit” choice leads to blocking, since any of that
>>> resource allocated to one path can’t be used by the other.
>>>
>>> The “overcommit” choice either leads to OOM, or throwing out some
>>> successfully transmitted and received data.
>>>
>>>
>>> Underlying this is a fun question: Which inefficiency is worse? Not
>>> using resources that should be used (i.e. from choosing to not overcommit),
>>> or sometimes redundantly using a resource (from choosing to overcommit)?
>>> I’m curious too about what implementation strategies we end up doing in
>>> general around this, and.. if enough implementations are choosing
>>> overcommit, if we need some different protocol mechanisms to bound the
>>> redundancy?
>>> -=R
>>>
>>>
>>>
>>> *From: *QUIC <[e-mail]> on behalf of Mirja Kuehlewind
>>> <mirja.kuehlewind=[e-mail]>
>>> *Date: *Friday, July 16, 2021 at 6:15 AM
>>> *To: *"Ma, Yunfei" <yunfei.ma=[e-mail]>, Robin
>>> MARX <[e-mail]>, Yanmei Liu <[e-mail]>
>>> *Cc: *"matt.joras" <[e-mail]>, 李振宇 <[e-mail]>,
>>> Christian Huitema <[e-mail]>, "lucaspardue.24.7" <
>>> [e-mail]>, quic <[e-mail]>, Qing An <
>>> [e-mail]>
>>> *Subject: *Re: Multi-path QUIC Extension Experiments
>>>
>>>
>>>
>>> Hi Yunfei,
>>>
>>>
>>>
>>> thanks as well for you sharing your results! Can you explain even a bit
>>> more what you mean by MP-HoL Blocking? Is this because of the flow control
>>> limits? If so wouldn’t it make sense to reserve a certain “space” for each
>>> path?
>>>
>>>
>>>
>>> Mirja
>>>
>>>
>>>
>>>
>>>
>>> *From: *QUIC <[e-mail]> on behalf of "Ma, Yunfei" <
>>> yunfei.ma=[e-mail]>
>>> *Date: *Thursday, 15. July 2021 at 04:18
>>> *To: *Robin MARX <[e-mail]>, Yanmei Liu <
>>> [e-mail]>
>>> *Cc: *"matt.joras" <[e-mail]>, 李振宇 <[e-mail]>,
>>> Christian Huitema <[e-mail]>, "lucaspardue.24.7" <
>>> [e-mail]>, quic <[e-mail]>, Qing An <
>>> [e-mail]>
>>> *Subject: *Re: Re: Multi-path QUIC Extension Experiments
>>>
>>>
>>>
>>> Hi Robin,
>>>
>>>
>>>
>>> Thanks so much for your questions!
>>>
>>>
>>>
>>> First, the head of line
>>> blocking discussed here is called multi-path head-of-line blocking or MP-HoL blocking, and its root cause is quite different from the stream HoL blocking usually discussed in
>>> QUICv1. The MP-HoL blocking happens when one path blocks the other path, not when one stream blocks the other stream. Please note that we indeed
>>> use multiple streams, for example, different video requests are carried in different QUIC streams. QUIC’s stream multiplexing ability and its benefits still hold in this scenario.
>>>
>>>
>>>
>>> Second, regarding packet scheduling mode,
>>> right now, in our Taobao A/B test, we transmit packets on multiple paths simultaneously. However, you can definitely use
>>> traffic switching only and choose to switch when one path could not meet
>>> your bandwidth requirement. Basically, if you use multiple paths
>>> simultaneously, you get the most elasticity from a resource pooling
>>> perspective.
>>> It really comes down on what your application needs. We will also update the packet scheduling section
>>> soon in a newer version of the
>>> draft, in which we plan to include more discussions on the packet scheduling
>>> policy.
>>>
>>>
>>>
>>> Third, regarding the benefits of more bandwith versus the "downsides".
>>> Whether you want more bandwidth depends on your application. For videos, yes, more bandwidth is
>>> extremely helpful in improving the long tail QoE, which is an important target for Taobao. We find multi-path QUIC helps us improve two important metrics, rebuffer rate and video start-up delays.
>>> In the past, if you work on multi-path scheduling that does not collaborate
>>> close enough with applications such as MPTCP, the MP-HoL blocking becomes
>>> the downside that cripples the
>>> performance. However, the user space nature of QUIC provides us the opportunity to solve this problem,
>>> so now our conclusion is that
>>> you can enjoy the benefits of more bandwidth and more reliable connectivity
>>> from multi-path without much of the “downsides”.
>>>
>>>
>>>
>>> I hope my answer is helpful, but feel free to let me know if you have
>>> any additional comments.
>>>
>>>
>>>
>>> Cheers,
>>>
>>> Yunfei
>>>
>>>
>>>
>>> from Alimail macOS
>>> <https://protect2.fireeye.com/v1/url?k=7cc82aa7-2353138a-7cc86a3c-8692dc8284cb-e08a325a5c75cf95&q=1&e=de295b4f-9105-4e32-980f-779c711eaa62&u=https://mail.alibaba-inc.com/>
>>>
>>> ------------------Original Mail ------------------
>>>
>>> *Sender:*Robin MARX <[e-mail]>
>>>
>>> *Send Date:*Wed Jul 14 07:39:37 2021
>>>
>>> *Recipients:*Yanmei Liu <[e-mail]>
>>>
>>> *CC:*quic <[e-mail]>, Ma, Yunfei <[e-mail]>,
>>> Christian Huitema <[e-mail]>, Qing An <
>>> [e-mail]>, 李振宇 <[e-mail]>, matt.joras <
>>> [e-mail]>, lucaspardue.24.7 <[e-mail]>
>>>
>>> *Subject:*Re: Multi-path QUIC Extension Experiments
>>>
>>> Hello Yanmei,
>>>
>>>
>>>
>>> Thanks for the additional results on an interesting topic. I'm looking
>>> forward to reading the SIGCOMM paper.
>>>
>>>
>>>
>>> I was a bit surprised to (apparently) see HOL blocking mentioned as a
>>> major issue, as that's one of the things QUIC aims to be better at than TCP.
>>>
>>> It's a bit difficult to understand from the slides, but it seems like
>>> you're sending packets for a single stream (Stream ID 1 in the diagrams) on
>>> both the slow and fast path, which would indeed induce HOL blocking.
>>>
>>> Consequently, I was wondering what the practical reasons are for you to
>>> multiplex packets for a single stream over multiple paths, as opposed to
>>> for example attaching a single stream to a single path (say: high priority
>>> streams use the fast path for all their packets).
>>>
>>>
>>>
>>> I see this mentioned a bit in the draft under "packet scheduling", where
>>> it talks about switching paths once the cwnd is full for one. That indeed
>>> leads to the behaviour seen in the slides, but that's my question: why
>>> would you take those approaches then?
>>>
>>> Are there so many cases where the additional "bandwidth" from using
>>> multiple path's cwnd for a single stream outweigh the downsides of HOL
>>> blocking? Relatedly: what are the packet loss rates you've observed on
>>> real networks?
>>>
>>> Have you experimented with e.g., tying streams to paths more closely?
>>> Does that work better or worse? Why?
>>>
>>>
>>>
>>> I'm mainly wondering how these tradeoffs evolve depending on the type of
>>> paths available and if it's possible to make a model to drive this logic.
>>>
>>> I assume there is much existing work on this for MPTCP, but I also
>>> assume some of that changes due to QUIC's independent streams / stream
>>> prioritization flexibility.
>>>
>>>
>>>
>>> Thank you in advance and with best regards,
>>>
>>> Robin
>>>
>>>
>>>
>>>
>>>
>>> On Sun, 11 Jul 2021 at 20:48, Yanmei Liu <miaoji.lym=
>>> [e-mail]> wrote:
>>>
>>> Hi everyone,
>>>
>>> We have finished some experiments about deploying multi-path quic
>>> extension(https://datatracker.ietf.org/doc/draft-liu-multipath-quic/) in
>>> Alibaba Taobao short-form video streaming, and the experiment results are
>>> concluded in the slides (attached file).
>>> If anyone is interested in the experimental details about multi-path
>>> quic, please let us know.
>>> All the feedbacks and suggestions are appreciated!
>>>
>>> Best regards,
>>> Yanmei
>>>
>>>
>>>
>>>
>>> --
>>>
>>>
>>>
>>> *dr. Robin Marx*
>>>
>>> Postdoc researcher - Web protocols
>>>
>>> Expertise centre for Digital Media
>>>
>>>
>>>
>>> *Cellphone *[phone]
>>>
>>>
>>>
>>> www.uhasselt.be
>>> <https://protect2.fireeye.com/v1/url?k=37557dd4-68ce44f9-37553d4f-8692dc8284cb-fe608437d16ed9d9&q=1&e=de295b4f-9105-4e32-980f-779c711eaa62&u=http://www.uhasselt.be/>
>>>
>>> Universiteit Hasselt - Campus Diepenbeek
>>>
>>> Agoralaan Gebouw D - B-3590 Diepenbeek
>>>
>>> Kantoor EDM-2.05
>>>
>>>
>>>
>>> [image: Image removed by sender.]
>>>
>>>
>>>
>>>
>