Re: More details about multipath quic in Alibaba

Yunfei Ma <yfmascgy@gmail.com> Mon, 26 October 2020 02:06 UTC

Return-Path: <yfmascgy@gmail.com>
X-Original-To: quic@ietfa.amsl.com
Delivered-To: quic@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D5C3C3A17C1 for <quic@ietfa.amsl.com>; Sun, 25 Oct 2020 19:06:35 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.097
X-Spam-Level:
X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id iBYAOOe3QfI0 for <quic@ietfa.amsl.com>; Sun, 25 Oct 2020 19:06:34 -0700 (PDT)
Received: from mail-vs1-xe2d.google.com (mail-vs1-xe2d.google.com [IPv6:2607:f8b0:4864:20::e2d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id E975C3A17BB for <quic@ietf.org>; Sun, 25 Oct 2020 19:06:33 -0700 (PDT)
Received: by mail-vs1-xe2d.google.com with SMTP id s15so4034019vsm.0 for <quic@ietf.org>; Sun, 25 Oct 2020 19:06:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=bLjp6gDn5YsQyoVBOdDuV2xTeFUkNef+dElgH1AoPiw=; b=AXF7/8m15L2d3vrDXtxGXU+n42U/5kAFmefhbew/i5aL3/7Trv0J7Rmfy9fIKAnSrz J7x5VfwHqgPx9E79LcB39A7QawrHreZerEmstCKXIj0SevwOQbRjArBgBzVKIMKnP+X+ 41Gu3nXD/avNvzdXeJnBKqPqSapl5a6U4tKgNl+WCv5GGExXuZx66VoVAwnLVAFcMuvs 2RvafahdNWbjwBy3D9k1hvCbGE9Vv9OtQhi9bRYwD44AoLlWLUof8cH4qqiORV/ovLXm pcP0FjOhUy3p/D8TjyRCq/KatT4ypwxuAdJoCG+w5yo6ot8ansZUdizZemYpQ0d8V+1n VaFQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=bLjp6gDn5YsQyoVBOdDuV2xTeFUkNef+dElgH1AoPiw=; b=hfMUXUCvT7hj1yWvffWihQuUsSM0Na893JPOHPAG4IYcjeBaBzUrcxKZ6bB5/tOV48 sW2vUjp47s64X8Yu2rsIjRfzEKgAVcVCgfKWHn4UmWVY6HjpreRs3HJslyi5eDPYuFP0 erpZ2ZPAKQ0Ef2Wcpum9iDPuk3Nl19xr8A/DGsSUHMHi7Hb+wVfszk/Othnorpv21qEf Yku3BdXA1LJAf/8te7rl+fFPeXTtI2nG/XBCutotRf6ed6I+2DxZ1ZuMijIgrMmePz0O GSaPbauTQwbVvIYgACFMzulYF/ZjUHdadjVrDXZuJ3+1mmbDgTUenz4xmzO5/RO1UaWK /2sA==
X-Gm-Message-State: AOAM532xs051DjYqoRc62mzZ9fSkZISiTJFm/0nxnmhaCoT0xJpdDuW+ UdovOW2clZuSf0DzCf1z2vQkCvbCBbFApgkcAlQ=
X-Google-Smtp-Source: ABdhPJyoporXFYnJqcAfUUwdavW8/Qi57UybTEuX6rJIuFbte68bcx1UdBrrY8sXNcWRymX/Hgoo5YaMZvjwa6xDgDU=
X-Received: by 2002:a67:fe12:: with SMTP id l18mr15280665vsr.35.1603677992909; Sun, 25 Oct 2020 19:06:32 -0700 (PDT)
MIME-Version: 1.0
References: <CAMDWRAb5pWdYHwc5h6MSc6uprtUQ+bWniMWamiv+mVnX32yQXA@mail.gmail.com> <CAPDSy+7Tx-wSG6+z97XdYGt6zxQfU_Lfp7axHjfz3xrAR43iDg@mail.gmail.com>
In-Reply-To: <CAPDSy+7Tx-wSG6+z97XdYGt6zxQfU_Lfp7axHjfz3xrAR43iDg@mail.gmail.com>
From: Yunfei Ma <yfmascgy@gmail.com>
Date: Sun, 25 Oct 2020 19:05:56 -0700
Message-ID: <CAHgerOFPJACJuY3WQbQYP3gMCGYrt0xHvwe+oLkC94+WX6f60g@mail.gmail.com>
Subject: Re: More details about multipath quic in Alibaba
To: David Schinazi <dschinazi.ietf@gmail.com>
Cc: Yanmei Liu <healing4d@gmail.com>, "Liu, Hongqiang" <hongqiang.liu@alibaba-inc.com>, QUIC <quic@ietf.org>, miaoji.lym@alibaba-inc.com, anqing.aq@alibaba-inc.com, Yunfei Ma <yunfei.ma@alibaba-inc.com>
Content-Type: multipart/alternative; boundary="00000000000019fccd05b2896155"
Archived-At: <https://mailarchive.ietf.org/arch/msg/quic/7gGWmAYdPqsz868u1I7rf5He5-Q>
X-BeenThere: quic@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <quic.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/quic>, <mailto:quic-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/quic/>
List-Post: <mailto:quic@ietf.org>
List-Help: <mailto:quic-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/quic>, <mailto:quic-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 26 Oct 2020 04:41:41 -0000

Hi David,

Thanks for the comments. Please find my answers below:

>
>> - Q1: Do we care about cellular cost?
>> - Sure. Cellular cost is a very important factor that influences our
>> packet scheduler design. Some of our customers are sensitive about the
>> cellular cost and some do not, so they need different scheduling policies.
>> For customers who are sensitive about cellular cost, Alibaba have
>> collaborated with several mobile carriers, so customers can get enrolled in
>> a special data plan, which allows them to enjoy unlimited free cellular
>> data when using apps from Alibaba eco-system(i.e. alibaba traffic costs the
>> user nothing). For customers who are not sensitive about the data cost, for
>> example, professional streamers/Vloggers and customers who want to get high
>> quality services for business collaborations, they can turn on the
>> multi-path feature if they want.
>>
>
David: That's really cool. Out of curiosity, how does your application know
that the user has unlimited free cellular data or not?

Answer: We collaborate with mobile carriers to issue a sim card called
AliCard so users who use the sim card automatically get unlimited free
cellular data. For other users, they can decide if they want to turn on
this option.

>
>
>> - Q2: If someone who wants this could go off an cook up an extension and
>> get experience with it then come back with that experience?
>> - We completely agree that a large-scale deployment experience is needed
>> to draw a conclusion and inspire new innovations. Right now, we are doing
>> the extension described in
>> https://tools.ietf.org/html/draft-an-multipath-quic-00, which is
>> integrated in Taobao for short-form video streaming. The feature is already
>> made available for certain customers in China and we are working to expand
>> the scope.
>>
>
David: This is great! If by any chance you have the time to run an
experiment between connection migration and full multipath that would be
incredibly valuable.

Answer: Thanks, it is a really great suggestion. We have been doing some
experiments in several use cases we care about and have got some early
observations. However, to reach a solid conclusion, we would like to
conduct a more thorough study, so It would still take us sometime to
collect more data as well as our customers' feedback, but we would
definitely love to share our findings later on.

>
>
>> - Q3: How to solve the linkability problem in multipath ?
>> - QUICv1 solves the linkablity problem with Connection ID’s
>> re-negotiation when connection migration happens. We reuse the mechanism
>> from QUICv1 in our multipath quic implementation. For each new
>> sub-connection, client and server need to exchange available unused
>> Connection IDs before a client tries to create a new sub-connnection. When
>> client creates a new sub-connection and sends packets on the new path, it
>> uses the new Connection IDs. As the process of the negotiation is
>> encrypted, it would not link the packets sent on new path with the previous
>> path.
>>
>
David: That's a great plan.

Answer: Thanks. We would also like to learn if anyone has a simple way of
doing this.

>
> - Q4: Difference from early multipath QUIC proposals?
>> - First of all, we really appreciate the work of early proposals such as
>> [I-D.deconinck-quic-multipath]. However, in our implementation, we’ve found
>> that two things are extremely important and should impact the protocol
>> design. First, we build multi-paths based on the concept bi-directional
>> “sub-connections” in each new path, and find it’s the easiest way to
>> implement multipath because it readily fits into the nature of both
>> cellular and wifi links that cover the majority of multi-path applications.
>> In doing so, you can almost reuse all the design / implementations for
>> QUICv1 connections. Second,  the multi-path QUIC design needs dynamic
>> scheduling strategy. As is pointed in the answer to Q1 above, our customers
>> who want multipath have very different needs, so making the scheduler
>> dynamic allows us to fulfill the job.
>>
>> We would like to know what people think about the draft, and all
>> feedbacks and suggestions are welcome.
>>
>
David: I'd have to implement this to form a full opinion, but I think I
like this design - placing the complexity on top of QUIC instead of inside
QUIC sounds like a great way to simplify implementations.

Answer: We really appreciate it if you can try and let us know how you
think. To make things simple to understand and easy to use is always an
important goal for us. Given that QUIC will be used for most of the web
traffic, the ease of maintenance is very important for our  day-to-day
operations.

Cheers,
Yunfei