From nobody Thu Sep 16 23:39:00 2021 Return-Path: X-Original-To: quic@ietfa.amsl.com Delivered-To: quic@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 511573A005E for ; Thu, 16 Sep 2021 23:38:57 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.097 X-Spam-Level: X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id C75NCT0wbjGY for ; Thu, 16 Sep 2021 23:38:53 -0700 (PDT) Received: from mail-lf1-x129.google.com (mail-lf1-x129.google.com [IPv6:2a00:1450:4864:20::129]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 064233A003F for ; Thu, 16 Sep 2021 23:38:52 -0700 (PDT) Received: by mail-lf1-x129.google.com with SMTP id y28so28695436lfb.0 for ; Thu, 16 Sep 2021 23:38:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=6EBnFlp56jNRXfTxDlaLSIeRDxTl5tjt6FakIKnoX/4=; b=Qrd2sSyUCD5/NBEBxiE8KUrUACqzr/vVL5LsjeqDUaOTyaPQ6KhiZThn3FMtSVzhr3 E28tYGqw19d7nPJDSP+QFiEZabTg2SidltqC+cqDtwv5UYpM+n894HNggi0U71XyhwF+ Dj2fLDIpMjbFm3ie6jrg7mBKhZELTgvcFv5FJvt85Do37i2cTsN8AKqxfTGvkLZgHaQB 8HRaOKHfIiID/5N6ZzTbco6ocO0XPdZwH74UKIQlDgVc6L9ixE/6EMN6snwmP8i99ayB JpeLRXafjluSV/yAyG20ZbG3GA8zVXB5mWnLTSljbG85eHjjGSWqrFMX1nyWO26/vERO EXTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=6EBnFlp56jNRXfTxDlaLSIeRDxTl5tjt6FakIKnoX/4=; b=IWUfUuAnc0RBxvLT8jJwF/rSAf8bugTzUWJoZnY7RYJR968IsC8kdr0Ne5LUKKuUpg sJgIxC7xH584cZaaVrETuaWzeHLWaxwDpA94nDb5/mAhDSMwXWBtcAil1W43ioAAaziO ruQNtQze4q1Z3tcZ5H+SFcIx7+UarPpLii0FzlXSpAiQmyVzOGKbEOtJgpzpcGyTUadR 2UcGvuxmmoiKpMxQs1e+y5OKYtbXIfd/TJ/CIE22oUswnj+MgYxoglfvDBGfA37Sp83J iaRbAxBu6gDilVbnTnCIrZ1I2ObSM7oW6F2jNpVwtbusyot2dJxhFrlGojW69JCi6xvf wFjQ== X-Gm-Message-State: AOAM5302I3X5YNagpz4scdp0l7E4w6qeGRyP00StDdVglzKjIpq5sUtQ RTwd1u7lL5zdftUU2HOFBAI15WXXQE6ioqRs+eg= X-Google-Smtp-Source: ABdhPJxwhZgXZlovFa7UvQmDg8yzQAMCN30GLfpDVR5pIGjnzk6hzWaHACGNSSK+CgbYPkHw0agVc6uIkwSw6SXil0I= X-Received: by 2002:a05:651c:3dd:: with SMTP id f29mr8173945ljp.69.1631860728538; Thu, 16 Sep 2021 23:38:48 -0700 (PDT) MIME-Version: 1.0 References: <8C2E8EFB-756B-449B-84E0-11CD6B57E541@ericsson.com> <0334A48E-B6C6-464C-A48C-4512A453DA81@fb.com> <0B9EAA73-802A-49F9-AEA3-A4F6C574A3F9@fb.com> <41414E77-B01A-4EC1-9FF7-37A03DA5EF5A@eggert.org> In-Reply-To: From: Yunfei Ma Date: Thu, 16 Sep 2021 23:38:11 -0700 Message-ID: Subject: Re: Multi-path QUIC Extension Experiments To: Robin MARX Cc: Lars Eggert , Roberto Peon , quic , "matt.joras" , =?UTF-8?B?5p2O5oyv5a6H?= , Christian Huitema , "Charles 'Buck' Krasic" , "lucaspardue.24.7" , Yanmei Liu , Yunfei Ma , Qing An , =?UTF-8?Q?Mirja_K=C3=BChlewind?= Content-Type: multipart/alternative; boundary="0000000000000c276c05cc2b2f37" Archived-At: X-BeenThere: quic@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Main mailing list of the IETF QUIC working group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Sep 2021 06:38:57 -0000 --0000000000000c276c05cc2b2f37 Content-Type: text/plain; charset="UTF-8" Hi Robin, Thanks for the question and sorry for the late reply. With regard to the experiment, the goal was to achieve bandwidth, latency and reliability for a video stream at the same time. Intuitively, adding more paths should give you immediate improvement. But, as you pointed out, HoL might undo the benefit. So what we found was a way to overcome this, and the technique was to use QoE feedback in conjunction with scheduling techniques. What the results tell us is if you have a stream that is time critical and bandwidth intensive, with the user-space nature of QUIC, you now have a way to go forward using multipath. But please keep in mind that you can choose to use whatever scheduling algorithm you like with the draft. In the experiment, we are trying to push the limit, but it is indeed one of the many possibilities. If you want to send streams that do not require every high reliability or high bandwidth, you can definitely use the fixed-path-per-stream strategy. Recently, we also have also done testing with fixed-path-per-stream with draft-liu-multipath-quic for certain scenarios. Here are some observations, if you have a large number of streams, having an assigned path for each can actually give you more combined bandwidth utilization, but not every stream benefits and some could suffer because of a bad path assignment. If you really care about the long tail performance, then, I would recommend sending a stream on multiple paths. How to use multi-path and which scheduling mode depends on what you try to optimize. I hope the draft can provide a starting point, and please feel free to go beyond. Cheers, Yunfei > Thanks for your extended explanations on Multipath HoL-blocking and > especially: > > > I think the stream dependencies you mentioned here is a great point. In > our implementation, we introduced a stream-priority based reinjection which > tries to address such dependency (There is a figure in the material that > Yanmei sent). But we haven't tried when each stream is limited to a single > path. In our case, streams are distributed on multiple paths. I would > definitely want to hear more about the application you are dealing with, > and maybe for wired transport, such a design is needed. > > This is exactly what I was trying to explore in my previous mail. You're > basically intentionally causing (or perhaps risking?) HOL blocking because > you split a single stream over multiple paths. > As noted by Christian with the 'equal cost multipath', this can have > bandwidth usage benefits, but only if paths are usable/similar. If not, HOL > blocking might undo all the benefits you get from this setup (and using a > single path per stream would be better). > So my question was: where is the inflection point where you might decide > to switch modes? At which parameters is one better than the other? > I'd hoped you would have experimented with the fixed-path-per-stream setup > to get some insight into this. > > In my mind, the idea of doing a purely transport-level multipath scheduler > (i.e., without taking into account application layer streams / data > dependencies / etc.) > has historically made some sense for TCP / for completely separated > stacks, as the transport didn't have that type of information available. > It is however utterly strange to me that this approach would continue for > QUIC (at least in endpoint multipath, not things like in-network > aggregators that have been discussed), > where we have clear splits between streams and (hopefully) already some > type of prioritization information for each stream. > For QUIC, I'd expect one-path-per-stream to be the default, with > multiple-paths-per-stream to be an edge case if you have a single, > high-traffic stream (which I do assume is your situation with a video > stream). > > With best regards, > Robin > > > > > On Tue, 20 Jul 2021 at 09:15, Lars Eggert wrote: > >> On 2021-7-20, at 1:19, Roberto Peon >> wrote: >> > >> > If we have to send data along a path in order to discover properties >> about that path, then sending less data on the path means discovering less >> about that path. >> > >> > The ideal would be to send *enough* data on any one path to maintain an >> understanding of its characteristics (including variance), and no more than >> that, and then to schedule the rest of the data to whichever path(s) are >> best at the moment. >> >> ^^^ This. >> >> Because the Internet has no explicit network-to-endpoint signaling, an >> endpoint must build its understanding of the properties of a path by >> exercising it, and specifically exercising it to a degree that causes >> queues to form (to obtain "under load" RTTs, see bufferbloat) and >> congestion loss to happen (to obtain an understanding of available path >> capacity.) Some people have called this "putting pressure on a path". >> >> There has been a long-standing assumption that if you exercised a path in >> the (recent) past you can probably assume that the properties haven't >> changed much if you want to start exercising it again. This is why >> heuristics like caching path properties (RTTs, etc.) are often of benefit - >> often, but not always, and maybe never in some scenarios (e.g., >> overcommitted CGNs.) >> >> There has been some work on this in the past for MPTCP. For example, on >> mobile devices - which most often have multiple possible paths to a >> destination via WiFi and cellular - exercising multiple paths comes at a >> distinct increase in energy usage. So you need a heuristic to determine if >> the potential benefit of going multipath is worth the energy cost of >> probing multiple paths before you do so. >> >> Thanks, >> Lars >> >> > > -- > > dr. Robin Marx > Postdoc researcher - Web protocols > Expertise centre for Digital Media > > *Cellphone *+32(0)497 72 86 94 > > www.uhasselt.be > Universiteit Hasselt - Campus Diepenbeek > Agoralaan Gebouw D - B-3590 Diepenbeek > Kantoor EDM-2.05 > > > --0000000000000c276c05cc2b2f37 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Robin,

Thanks for t= he question and sorry for the late reply. With regard to the experiment, th= e goal was to achieve bandwidth, latency and reliability for a video stream= at the same time. Intuitively, adding more paths should give you immediate= improvement. But, as you pointed out, HoL might undo the benefit. So what = we found was a way to overcome this, and the technique was to use QoE feedb= ack in conjunction with scheduling techniques. What the results tell us is = if you have a stream that is time critical and bandwidth intensive, with th= e=C2=A0user-space nature of QUIC, you now have a way to go forward using mu= ltipath.

But please keep in mind that you can choo= se to use whatever scheduling algorithm you like with the draft. In the=C2= =A0experiment, we are trying to push the limit, but it is indeed one of the= many possibilities. If you want to send=C2=A0streams that do not require e= very high reliability or high bandwidth, you can definitely use the fixed-p= ath-per-stream strategy. Recently, we also have also done testing with fixe= d-path-per-stream with draft-liu-multipath-quic for certain scenarios. Here= are some observations, if you have a large number of=C2=A0streams, having = an assigned path for each can actually give you more combined bandwidth uti= lization, but not every stream benefits and some could suffer because of a = bad path assignment. If you really care about the long tail performance, th= en, I would recommend sending a stream on multiple paths.=C2=A0
<= br>
How to use multi-path and which scheduling mode depends on wh= at you try to optimize. I hope the draft can provide a starting point, and = please feel free to go beyond.

Cheers,
Y= unfei


<= div>=C2=A0
Thanks for your extended explanations on Multipath= HoL-blocking and especially:

> I think the str= eam dependencies you mentioned here is a great point. In our implementation= , we introduced a stream-priority based reinjection which tries to address = such dependency (There is a figure in the material that Yanmei sent). But w= e haven't tried when each stream is limited to a single path. In our ca= se, streams are distributed on multiple paths. I would definitely want to h= ear more about the application you are dealing with, and maybe for wired tr= ansport, such a design is needed.=C2=A0

This is ex= actly what I was trying to explore in my previous mail. You're basicall= y intentionally causing (or perhaps risking?) HOL blocking because you spli= t a single stream over multiple=C2=A0paths.=C2=A0
As noted by Chr= istian with the 'equal cost multipath', this can have bandwidth usa= ge benefits, but only if paths are usable/similar. If not, HOL blocking mig= ht undo all the benefits you get from this setup (and using a single path p= er stream would be better).=C2=A0
So my question was: where is th= e inflection point where you might decide to switch modes? At which paramet= ers is one better than the other?
I'd hoped you would have ex= perimented with the fixed-path-per-stream setup to get some insight into th= is.=C2=A0

In my mind, the idea of doing a purely t= ransport-level multipath scheduler (i.e., without taking into account appli= cation layer streams / data dependencies / etc.)
has historically= made some sense for TCP / for completely separated stacks, as the transpor= t didn't have that type of information available.
It is howev= er utterly strange to me that this approach would continue for QUIC (at lea= st in endpoint multipath, not things like in-network aggregators that have = been discussed),
where we have clear splits between streams and (= hopefully) already some type of prioritization information for each stream.= =C2=A0
For QUIC, I'd expect one-path-per-stream to be the def= ault, with multiple-paths-per-stream to be an edge case if you have a singl= e, high-traffic stream (which I do assume is your situation with a video st= ream).=C2=A0

With best regards,
Robin




On Tue, 20 Jul 2021 at 09:1= 5, Lars Eggert <lar= s@eggert.org> wrote:
On 2021-7-20, at 1:19, Roberto Peon <fenix=3D40fb.com@dmarc.ietf.org&g= t; wrote:
>
> If we have to send data along a path in order to discover properties a= bout that path, then sending less data on the path means discovering less a= bout that path.
>
> The ideal would be to send *enough* data on any one path to maintain a= n understanding of its characteristics (including variance), and no more th= an that, and then to schedule the rest of the data to whichever path(s) are= best at the moment.

^^^ This.

Because the Internet has no explicit network-to-endpoint signaling, an endp= oint must build its understanding of the properties of a path by exercising= it, and specifically exercising it to a degree that causes queues to form = (to obtain "under load" RTTs, see bufferbloat) and congestion los= s to happen (to obtain an understanding of available path capacity.) Some p= eople have called this "putting pressure on a path".

There has been a long-standing assumption that if you exercised a path in t= he (recent) past you can probably assume that the properties haven't ch= anged much if you want to start exercising it again. This is why heuristics= like caching path properties (RTTs, etc.) are often of benefit - often, bu= t not always, and maybe never in some scenarios (e.g., overcommitted CGNs.)=

There has been some work on this in the past for MPTCP. For example, on mob= ile devices - which most often have multiple possible paths to a destinatio= n via WiFi and cellular - exercising multiple paths comes at a distinct inc= rease in energy usage. So you need a heuristic to determine if the potentia= l benefit of going multipath is worth the energy cost of probing multiple p= aths before you do so.

Thanks,
Lars



--

--0000000000000c276c05cc2b2f37--
dr. Rob= in=C2=A0Marx
Postdoc researcher - Web protocols
Expertise centre for Digital Media
=C2=A0
Cellphone=C2=A0+32(0)497 72 86 94 <= /td>
=C2=A0
ww= w.uhasselt.be
Universiteit Hasselt - Campus Diepenbeek
Agoralaan Gebouw D - B-3= 590 Diepenbeek
Kantoor=C2=A0EDM-2.05
=C2=A0
3D""