Re: Multi-path QUIC Extension Experiments
Roberto Peon <fenix@fb.com> Sun, 18 July 2021 21:52 UTC
Return-Path: <prvs=6833880767=fenix@fb.com>
X-Original-To: quic@ietfa.amsl.com
Delivered-To: quic@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 019483A0EA4 for <quic@ietfa.amsl.com>; Sun, 18 Jul 2021 14:52:42 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.548
X-Spam-Level:
X-Spam-Status: No, score=-2.548 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.452, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=fb.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BvWmLCzPsuMY for <quic@ietfa.amsl.com>; Sun, 18 Jul 2021 14:52:36 -0700 (PDT)
Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 333A73A0EA9 for <quic@ietf.org>; Sun, 18 Jul 2021 14:52:35 -0700 (PDT)
Received: from pps.filterd (m0148460.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16ILnriW018191; Sun, 18 Jul 2021 14:52:00 -0700
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : mime-version; s=facebook; bh=Dl7/zc7Crd2hyHx4Y3+T0gnLak1pAdQvNzYExvEYNm0=; b=UnpI/jphYa0QI4E/nYAaDb8SDyDansGz1yfk8o4OveiLNG7I07djXlk4iDkKAkb8QxGl jxVPpajpmOlldpg00bmmPisBC1iB/iQGeqbLhWlcQGiaHprmJmVtEaCOI/3YAdFojQFI ISSWN1gEqCjn1USLqfkj/cR0AbOUN+xkeSE=
Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com with ESMTP id 39uvr1p5p8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Sun, 18 Jul 2021 14:51:59 -0700
Received: from NAM12-DM6-obe.outbound.protection.outlook.com (100.104.98.9) by o365-in.thefacebook.com (100.104.94.198) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Sun, 18 Jul 2021 14:51:58 -0700
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UZX5SQA57mmIHc7QKWz8x0/LxM7F67/4k3Hr1vRlS7Zr1MOZxFuuyxpuXCM6zx72roTmD5PxAlrqqpqNiLt9AELZ5CynHonkCDDo9GjSZFuAcuo8snwiXSxY5JOh52N+67F7EB39ceFLfrRIxq9q5qWtYzJB8mxscXPIvovxgu87J1NRrlNShbbUWpDoOLRepuK9QsHSFcpQ3YVnBqtg0FMGxZvLaQmksxeM6Zq+EERQZKQFqIaooXdTQaNNeL5JEz5/jphemjjH/N6VlaNnnEgQtokYnXKZcdF8UBIxVAH5aXJRV0d4fWo9ywdSVfZ8k1y/uII3HFUA2d5MERygCQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1ln6Uz7bg0NRmqZAcQIr4PYgryyhSZEY23qFMYORqAE=; b=L+/KAdXCJnarpt82iBW19azp2R6tq11Qydptk6vvzKrv26gvtM4R2OwpugpVKCFq/tKysPfnc7REvCLtKNeDjGIR3Fw9LJtTDj5qaQg2RjEuV0dVcgxJ3abmheeStncM240Gc7D63/UZinTpY++j/Q6cVuF4S+g1E5q/NmdE+GBWf+yQvK/7dLR7tCMbZ56noE6VJelMvE3kdnPiSWfKQIBQ0I7d1fYSDil6tukbedcZuvJkUH2vlHBDSt1YkBZrmUAWLrSYa0pMa0wBHbayfN4TdMmv58kVk0EOBnFY3qsB3KLL3izzr0k7AxZlDxPKw0wP3wULmL62lugwIF996Q==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=fb.com; dmarc=pass action=none header.from=fb.com; dkim=pass header.d=fb.com; arc=none
Received: from DM6PR15MB2681.namprd15.prod.outlook.com (2603:10b6:5:1aa::28) by DM6PR15MB2474.namprd15.prod.outlook.com (2603:10b6:5:8e::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.25; Sun, 18 Jul 2021 21:51:55 +0000
Received: from DM6PR15MB2681.namprd15.prod.outlook.com ([fe80::c5:d5e1:fad0:5deb]) by DM6PR15MB2681.namprd15.prod.outlook.com ([fe80::c5:d5e1:fad0:5deb%6]) with mapi id 15.20.4331.032; Sun, 18 Jul 2021 21:51:55 +0000
From: Roberto Peon <fenix@fb.com>
To: Yunfei Ma <yfmascgy@gmail.com>, Charles 'Buck' Krasic <charles.krasic@gmail.com>, Mirja Kuehlewind <mirja.kuehlewind@ericsson.com>
CC: "matt.joras" <matt.joras@gmail.com>, 李振宇 <zyli@ict.ac.cn>, Christian Huitema <huitema@huitema.net>, Yanmei Liu <miaoji.lym@alibaba-inc.com>, "lucaspardue.24.7" <lucaspardue.24.7@gmail.com>, quic <quic@ietf.org>, Qing An <anqing.aq@alibaba-inc.com>, Yunfei Ma <yunfei.ma@alibaba-inc.com>
Subject: Re: Multi-path QUIC Extension Experiments
Thread-Topic: Multi-path QUIC Extension Experiments
Thread-Index: AQHXekR/rwb9xtCPfE2fLUknTCKSfqtFgxqAgAB/+ACAAmLTAIAAbnaA
Date: Sun, 18 Jul 2021 21:51:55 +0000
Message-ID: <B98E91A9-0E29-44E3-9F0C-06B2DA38DDBD@fb.com>
References: <8C2E8EFB-756B-449B-84E0-11CD6B57E541@ericsson.com> <0334A48E-B6C6-464C-A48C-4512A453DA81@fb.com> <CAPhuoz0vz2k63_ZaWmUg_XgSHUopid7vf+JY=JVFm_VqQJY87w@mail.gmail.com> <CAHgerOGhX3G_aBMrwZ0zXjN8tu9dqtu-9tu4z7YU80qfqaZkzQ@mail.gmail.com>
In-Reply-To: <CAHgerOGhX3G_aBMrwZ0zXjN8tu9dqtu-9tu4z7YU80qfqaZkzQ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-MacOutlook/16.50.21061301
authentication-results: gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=none action=none header.from=fb.com;
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: e336e333-1cd0-4151-5d0e-08d94a3642ba
x-ms-traffictypediagnostic: DM6PR15MB2474:
x-microsoft-antispam-prvs: <DM6PR15MB2474F5A7B632A265F2260372CDE09@DM6PR15MB2474.namprd15.prod.outlook.com>
x-fb-source: Internal
x-ms-oob-tlc-oobclassifiers: OLM:10000;
x-ms-exchange-senderadcheck: 1
x-ms-exchange-antispam-relay: 0
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: kMFfOQLm8x4rATIPHE9rzATeoVGVLvnCxea/kNy4S78CJNGzmQdiL7uMEnQ9aRUZ90yFx0qkt5AElIEfeIwd3RGJlKdMZ9NcTNhg4XD1zIeD3fohx1fLC3S5wRydLqy2Pb/y1ToqCppik6z5Ip9wuUc2D9jW5zmBV1lPpdjqk/frGLgrFhsB3FTUanZJM50Py21bbZAMfwmSHXj6F1Uo/lguEmxa2caf+NwG53kPW+RFA6EM0tB6m4t4x2+60aa88DuFvBS3FYFaF8Z7+L06aoTpGJZaxxrMgOL0qqXukygrpAz/En3SwOLkO1e3gocNVh1xpE6lv8622vwI6+8CEhKAyJPdxIDVhE/VVUk7KmzNGpUSHG2N152FFkSSK/nhQhTtWwoYATpwfJ22hTt9DC4Cz9oiSQe2UIYQF7KMsZXhsUpfJlTyAR1+Vzrjj5zDOMYyIAFsa9CxPSZOZSWUdDT258BlZKI0tLdN6JDUEG3uJitr93jjPB43qYYj1AIuh7NCK2RVdud6uHIKq3LeOoawlbo2msJgHRHWj/SeqOga9LQb5TC/RjIxBQcZ5z0L7G2K5oiboDaxcUt8+cV+1osMiiq7Fa0Mu+nAymLDRplvO/Z5osPQZuu9RuWf1jj+FuV5Qu9Pdfqv67+Wt45+J00LpG9DJ2y8Q0rX3J2dV+UaWU1CudWulovHBYl21osdCfdaBtQBr2TovZHEw+v5l9auxq1zKJAh1fkd7ivlT8iUyVCvuOhOis37lvu/nTMIaYX8o61y8N9Ea7T/78SOomZijG2aQJP+Yun/6pSuqBNs44/xo2Kijio4kcC3x/v061Uf/S9R/l+JxuqZCBMIIjkpjzDzuZM/6PXdRc9scQ4=
x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM6PR15MB2681.namprd15.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(346002)(366004)(39860400002)(136003)(376002)(86362001)(8676002)(71200400001)(64756008)(2616005)(66556008)(66476007)(316002)(66946007)(8936002)(166002)(33656002)(76116006)(38100700002)(91956017)(66446008)(26005)(30864003)(7416002)(36756003)(122000001)(83380400001)(186003)(478600001)(2906002)(6512007)(4326008)(6486002)(6506007)(53546011)(110136005)(5660300002)(3480700007)(54906003)(45980500001)(38070700004); DIR:OUT; SFP:1102;
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: ub0xr5L+HJgxhrJbka5qdu9GAbIJHuBZQZPWOFMLUQVqbpEC0QmMz7kecvvjpVvQ10EQgtN7v+lNewFbY0/4emtiR9GQc3IxPB1qD9jmlAq1jA4tOnFH9JRmAPgJ/f4JkXvpboYkWQNMGqYoM0fsxmWMV7cIUyR9aP0tn8Xlfy40KGQTCUAqd1OA1NzwtjejJwuPTZvdGPHXgeyweXCLii/pfGnsbWvUCd67OEaOSH6G0juMn4usEpzrNl+76b3umahPi9j2akR7infjvTcmFV2YqprIQ6dxyIBWeTjXlNfFYZTT6Hlx/sHM6Rc7HGVBzGI2w3jqPjhIyptLTLnEcEAIoicQ756ypeTnrX37uoMTXquVsYccVyYrMxLHVJxm4v5EDrB9pF151yzBnxA3F5KFpsFPjZ/fRZqpC2kxitP+uXPKmnN9TYdxU40zHgivgDDVLDHRZbQVL+9w5Dqv763NBb4pQLF8dU1UHtIkLNQNXa2cnJ9XBzNNlHSgjO+155UttEd+/RIL6Z2zNe31XQxjRCX4rt0fZbK4CTPxOalBH4VmummQSWQ+YBPNvq+gC5U50T7gjfL2HCFjIqZZ9jmITCg9X7Izfe4Du2lczPFmzkdmImUSSrvaYxaBaBpzXp6z4uqC8ugUZCpnukbhAqsPbvdwbNKrKvsD3EvBFUyVvAwriBe2kI2AnG9NpssaHAKb1yGt2a+eDqnchL6nwx3YuPZBt5J6zztmPJp9C7GymjueN+KB7sE96sc8VuQjv+C6AgzuL6+X7aSv8h47E0xvqU3X78dWVexDgdliAfsX8x6FcwsVEQx5qrmeKfQyqCn52VTMn941NngyuzZlqI9KoWW0wLOu2956u1tTmeqm8dC9HT9hkU6wfuljwRoKn5hDlgN+m20PcGYQn0Mqe6PvOHjFmkufJ9AMjma2qrhYaV8JLR6CmkW3cEs5JsRkiUQrkklt5rOBoQ14mk7+lUHVM6EMlqRUh3Je3oD/+qGsydIP+nLyqlnDsViGLzq6QpHSM2GXrBg2QwzVY4smF+a9/dOdsAARBkSdchsj4qkZv/fx82Oo89+/fDYMunYpP0p9Q1pR8zz/jmR4yeNiXj4c0otr3m0FF3mXInY9LLmkAsxTLNv6qRfGQFuUtbBHdLJ5QgVBwgnT/STyvt/RhUFn30Q6EjRXvIizofjNxz62iLntuU5yOpHIJ1JAQGdMFjar6FjBsC9cMGljoxMo99aQTeroFlhXzEQ8/unLJ6aeB2ZqzkhCgb3pOHWRaN0sdYSgWcl7NaI8fPmkJD1gaJh9gnwuYYIEoUqb68FROYnyqf6buwzKBhwDdFK3eL37
x-ms-exchange-transport-forked: True
Content-Type: multipart/alternative; boundary="_000_B98E91A90E2944E39F0C06B2DA38DDBDfbcom_"
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: DM6PR15MB2681.namprd15.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e336e333-1cd0-4151-5d0e-08d94a3642ba
X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Jul 2021 21:51:55.6475 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: HR6fompESbw+dF0VQyJFSx0hBeoX//IqYBqAr36k2Q63T/InealaPqhlx66GsxMP
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR15MB2474
X-OriginatorOrg: fb.com
X-Proofpoint-ORIG-GUID: u5C3k49LDdi83kuTVyO6HcHwznVhRjYq
X-Proofpoint-GUID: u5C3k49LDdi83kuTVyO6HcHwznVhRjYq
X-Proofpoint-UnRewURL: 10 URL's were un-rewritten
MIME-Version: 1.0
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.391, 18.0.790 definitions=2021-07-18_10:2021-07-16, 2021-07-18 signatures=0
X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 spamscore=0 adultscore=0 impostorscore=0 bulkscore=0 lowpriorityscore=0 mlxlogscore=999 suspectscore=0 clxscore=1011 priorityscore=1501 mlxscore=0 phishscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107180149
X-FB-Internal: deliver
Archived-At: <https://mailarchive.ietf.org/arch/msg/quic/L8gAgSfktGef54rpfTF86CyuYnQ>
X-BeenThere: quic@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <quic.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/quic>, <mailto:quic-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/quic/>
List-Post: <mailto:quic@ietf.org>
List-Help: <mailto:quic-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/quic>, <mailto:quic-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sun, 18 Jul 2021 21:52:42 -0000
It sounds like this problem is not inherent to single-connection multi-path, but will be present in any multi-path implementation, including multiple-tcp-connections used with application-layer muxing. If this is correct, then it isn’t really a ‘QUIC problem, but rather an implementation/scheduling/CC problem. That isn’t saying that it isn’t interesting or important to solve, but rather that the protocol itself need not change to solve the problem for generic-QUIC transport. H3, OTOH, may suffer from this at L7+ proxies without some changes to QUIC or H3, but that is a much longer conversation that doesn’t require multi-path to happen. -=R From: Yunfei Ma <yfmascgy@gmail.com> Date: Sunday, July 18, 2021 at 1:17 AM To: Charles 'Buck' Krasic <charles.krasic@gmail.com>, Mirja Kuehlewind <mirja.kuehlewind@ericsson.com>, Roberto Peon <fenix@fb.com> Cc: "matt.joras" <matt.joras@gmail.com>, 李振宇 <zyli@ict.ac.cn>, Christian Huitema <huitema@huitema.net>, Yanmei Liu <miaoji.lym@alibaba-inc.com>, "lucaspardue.24.7" <lucaspardue.24.7@gmail.com>, quic <quic@ietf.org>, Qing An <anqing.aq@alibaba-inc.com>, Yunfei Ma <yunfei.ma@alibaba-inc.com> Subject: Re: Multi-path QUIC Extension Experiments Hi Charles, Roberto, and Mirja: Thanks a lot for your questions. As all three of you are curious about the definition of MP-HoL, I am putting my answer into one reply. Short answer: the MP-HoL is not because of flow control, but rather, it is related to the nature of path heterogeneity. In other words, MP-HoL can happen when flow control limit is not reached (as pointed out by Charles, you can set a large limit on the client side). More specifically, when you want to send out packets on different paths at the same time, there is a scheduler to decide how to split your packets and put them on different paths. However, in mobile networks, the network paths could have very different path delays. MP-HoL blocking arises when the packets sent earlier at the slow path arrive later than the packets sent later at the fast path, causing out-of-order arrival. As a consequence, the out-of-order packets are not eligible to be submitted to applications, so the fast path has to wait. For example, say we want to send out two packets that belong to the same video frame with a min-RTT scheduler, which is default in MPTCP. For each packet, the scheduler selects a path for that packet to transmit. The selection has two criterias: (1) the path's congestion window is not full and (2) the path selected has a smaller RTT than the other. If somehow, at the moment of transmitting, the fast path's cwnd is full (some traffic has been sent before), the first packet is then put on the slow path by the scheduler. Later, an ACK is received and the fast path becomes available, so the scheduler puts the second packet on the fast path. As a result, there is an out-of-order arrival. What makes the problem even more difficult is that in mobile networks, the RTTs can change quickly, which makes accurate prediction very difficult. Worst case is that when the scheduler thinks it is using the fast path, it is actually using the slow path instead. As you can see, in order to make multi-path transport efficient, it is important to solve this problem and that's what we are doing in this project . I hope I have answered your questions. If not, please let me know. Cheers, Yunfei On Fri, Jul 16, 2021 at 12:51 PM Charles 'Buck' Krasic <charles.krasic@gmail.com<mailto:charles.krasic@gmail.com>> wrote: "don't overcommit" includes the common practice of setting very large limits on the client side, where in aggregate the case of server being flow control limited is effectively non-existent. I am curious to hear clarification of the precise definition of MP-HoL blocking here. is it not flow control, but rather path aliasing where distinct paths are actually sharing some physical link(s)? On Fri, Jul 16, 2021 at 12:13 PM Roberto Peon <fenix=40fb.com@dmarc.ietf.org<mailto:40fb.com@dmarc.ietf.org>> wrote: I too am curious! There are only two ways to handle flow control—overcommit, or don’t overcommit. The “don’t overcommit” choice leads to blocking, since any of that resource allocated to one path can’t be used by the other. The “overcommit” choice either leads to OOM, or throwing out some successfully transmitted and received data. Underlying this is a fun question: Which inefficiency is worse? Not using resources that should be used (i.e. from choosing to not overcommit), or sometimes redundantly using a resource (from choosing to overcommit)? I’m curious too about what implementation strategies we end up doing in general around this, and.. if enough implementations are choosing overcommit, if we need some different protocol mechanisms to bound the redundancy? -=R From: QUIC <quic-bounces@ietf.org<mailto:quic-bounces@ietf.org>> on behalf of Mirja Kuehlewind <mirja.kuehlewind=40ericsson.com@dmarc.ietf.org<mailto:40ericsson.com@dmarc.ietf.org>> Date: Friday, July 16, 2021 at 6:15 AM To: "Ma, Yunfei" <yunfei.ma<http://yunfei.ma>=40alibaba-inc.com@dmarc.ietf.org<mailto:40alibaba-inc.com@dmarc.ietf.org>>, Robin MARX <robin.marx@uhasselt.be<mailto:robin.marx@uhasselt.be>>, Yanmei Liu <miaoji.lym@alibaba-inc.com<mailto:miaoji.lym@alibaba-inc.com>> Cc: "matt.joras" <matt.joras@gmail.com<mailto:matt.joras@gmail.com>>, 李振宇 <zyli@ict.ac.cn<mailto:zyli@ict.ac.cn>>, Christian Huitema <huitema@huitema.net<mailto:huitema@huitema.net>>, "lucaspardue.24.7" <lucaspardue.24.7@gmail.com<mailto:lucaspardue.24.7@gmail.com>>, quic <quic@ietf.org<mailto:quic@ietf.org>>, Qing An <anqing.aq@alibaba-inc.com<mailto:anqing.aq@alibaba-inc.com>> Subject: Re: Multi-path QUIC Extension Experiments Hi Yunfei, thanks as well for you sharing your results! Can you explain even a bit more what you mean by MP-HoL Blocking? Is this because of the flow control limits? If so wouldn’t it make sense to reserve a certain “space” for each path? Mirja From: QUIC <quic-bounces@ietf.org<mailto:quic-bounces@ietf.org>> on behalf of "Ma, Yunfei" <yunfei.ma<http://yunfei.ma>=40alibaba-inc.com@dmarc.ietf.org<mailto:40alibaba-inc.com@dmarc.ietf.org>> Date: Thursday, 15. July 2021 at 04:18 To: Robin MARX <robin.marx@uhasselt.be<mailto:robin.marx@uhasselt.be>>, Yanmei Liu <miaoji.lym@alibaba-inc.com<mailto:miaoji.lym@alibaba-inc.com>> Cc: "matt.joras" <matt.joras@gmail.com<mailto:matt.joras@gmail.com>>, 李振宇 <zyli@ict.ac.cn<mailto:zyli@ict.ac.cn>>, Christian Huitema <huitema@huitema.net<mailto:huitema@huitema.net>>, "lucaspardue.24.7" <lucaspardue.24.7@gmail.com<mailto:lucaspardue.24.7@gmail.com>>, quic <quic@ietf.org<mailto:quic@ietf.org>>, Qing An <anqing.aq@alibaba-inc.com<mailto:anqing.aq@alibaba-inc.com>> Subject: Re: Re: Multi-path QUIC Extension Experiments Hi Robin, Thanks so much for your questions! First, the head of line blocking discussed here is called multi-path head-of-line blocking or MP-HoL blocking, and its root cause is quite different from the stream HoL blocking usually discussed in QUICv1. The MP-HoL blocking happens when one path blocks the other path, not when one stream blocks the other stream. Please note that we indeed use multiple streams, for example, different video requests are carried in different QUIC streams. QUIC’s stream multiplexing ability and its benefits still hold in this scenario. Second, regarding packet scheduling mode, right now, in our Taobao A/B test, we transmit packets on multiple paths simultaneously. However, you can definitely use traffic switching only and choose to switch when one path could not meet your bandwidth requirement. Basically, if you use multiple paths simultaneously, you get the most elasticity from a resource pooling perspective. It really comes down on what your application needs. We will also update the packet scheduling section soon in a newer version of the draft, in which we plan to include more discussions on the packet scheduling policy. Third, regarding the benefits of more bandwith versus the "downsides". Whether you want more bandwidth depends on your application. For videos, yes, more bandwidth is extremely helpful in improving the long tail QoE, which is an important target for Taobao. We find multi-path QUIC helps us improve two important metrics, rebuffer rate and video start-up delays. In the past, if you work on multi-path scheduling that does not collaborate close enough with applications such as MPTCP, the MP-HoL blocking becomes the downside that cripples the performance. However, the user space nature of QUIC provides us the opportunity to solve this problem, so now our conclusion is that you can enjoy the benefits of more bandwidth and more reliable connectivity from multi-path without much of the “downsides”. I hope my answer is helpful, but feel free to let me know if you have any additional comments. Cheers, Yunfei from Alimail macOS<https://protect2.fireeye.com/v1/url?k=7cc82aa7-2353138a-7cc86a3c-8692dc8284cb-e08a325a5c75cf95&q=1&e=de295b4f-9105-4e32-980f-779c711eaa62&u=https://mail.alibaba-inc.com/> ------------------Original Mail ------------------ Sender:Robin MARX <robin.marx@uhasselt.be<mailto:robin.marx@uhasselt.be>> Send Date:Wed Jul 14 07:39:37 2021 Recipients:Yanmei Liu <miaoji.lym@alibaba-inc.com<mailto:miaoji.lym@alibaba-inc.com>> CC:quic <quic@ietf.org<mailto:quic@ietf.org>>, Ma, Yunfei <yunfei.ma@alibaba-inc.com<mailto:yunfei.ma@alibaba-inc.com>>, Christian Huitema <huitema@huitema.net<mailto:huitema@huitema.net>>, Qing An <anqing.aq@alibaba-inc.com<mailto:anqing.aq@alibaba-inc.com>>, 李振宇 <zyli@ict.ac.cn<mailto:zyli@ict.ac.cn>>, matt.joras <matt.joras@gmail.com<mailto:matt.joras@gmail.com>>, lucaspardue.24.7 <lucaspardue.24.7@gmail.com<mailto:lucaspardue.24.7@gmail.com>> Subject:Re: Multi-path QUIC Extension Experiments Hello Yanmei, Thanks for the additional results on an interesting topic. I'm looking forward to reading the SIGCOMM paper. I was a bit surprised to (apparently) see HOL blocking mentioned as a major issue, as that's one of the things QUIC aims to be better at than TCP. It's a bit difficult to understand from the slides, but it seems like you're sending packets for a single stream (Stream ID 1 in the diagrams) on both the slow and fast path, which would indeed induce HOL blocking. Consequently, I was wondering what the practical reasons are for you to multiplex packets for a single stream over multiple paths, as opposed to for example attaching a single stream to a single path (say: high priority streams use the fast path for all their packets). I see this mentioned a bit in the draft under "packet scheduling", where it talks about switching paths once the cwnd is full for one. That indeed leads to the behaviour seen in the slides, but that's my question: why would you take those approaches then? Are there so many cases where the additional "bandwidth" from using multiple path's cwnd for a single stream outweigh the downsides of HOL blocking? Relatedly: what are the packet loss rates you've observed on real networks? Have you experimented with e.g., tying streams to paths more closely? Does that work better or worse? Why? I'm mainly wondering how these tradeoffs evolve depending on the type of paths available and if it's possible to make a model to drive this logic. I assume there is much existing work on this for MPTCP, but I also assume some of that changes due to QUIC's independent streams / stream prioritization flexibility. Thank you in advance and with best regards, Robin On Sun, 11 Jul 2021 at 20:48, Yanmei Liu <miaoji.lym=40alibaba-inc.com@dmarc.ietf.org<mailto:40alibaba-inc.com@dmarc.ietf.org>> wrote: Hi everyone, We have finished some experiments about deploying multi-path quic extension(https://datatracker.ietf.org/doc/draft-liu-multipath-quic/)<https://datatracker.ietf.org/doc/draft-liu-multipath-quic/)> in Alibaba Taobao short-form video streaming, and the experiment results are concluded in the slides (attached file). If anyone is interested in the experimental details about multi-path quic, please let us know. All the feedbacks and suggestions are appreciated! Best regards, Yanmei -- dr. Robin Marx Postdoc researcher - Web protocols Expertise centre for Digital Media Cellphone +32(0)497 72 86 94 www.uhasselt.be<https://protect2.fireeye.com/v1/url?k=37557dd4-68ce44f9-37553d4f-8692dc8284cb-fe608437d16ed9d9&q=1&e=de295b4f-9105-4e32-980f-779c711eaa62&u=http://www.uhasselt.be/> Universiteit Hasselt - Campus Diepenbeek Agoralaan Gebouw D - B-3590 Diepenbeek Kantoor EDM-2.05 Error! Filename not specified.
- Multi-path QUIC Extension Experiments Yanmei Liu
- Re: Multi-path QUIC Extension Experiments Robin MARX
- Re: Re: Multi-path QUIC Extension Experiments Ma, Yunfei
- Re: Multi-path QUIC Extension Experiments Mirja Kuehlewind
- Re: Multi-path QUIC Extension Experiments Roberto Peon
- Re: Multi-path QUIC Extension Experiments Charles 'Buck' Krasic
- Re: Multi-path QUIC Extension Experiments Yunfei Ma
- Re: Multi-path QUIC Extension Experiments Mikkel Fahnøe Jørgensen
- Re: Multi-path QUIC Extension Experiments Roberto Peon
- Re: Multi-path QUIC Extension Experiments Christian Huitema
- Multi-path QUIC Extension Experiments Alexis Norech
- Re: Multi-path QUIC Extension Experiments Charles 'Buck' Krasic
- Re: Multi-path QUIC Extension Experiments Roberto Peon
- Re: Multi-path QUIC Extension Experiments Lucas Pardue
- Re: Multi-path QUIC Extension Experiments Yunfei Ma
- Re: Multi-path QUIC Extension Experiments Yunfei Ma
- Re: Multi-path QUIC Extension Experiments Yunfei Ma
- Re: Multi-path QUIC Extension Experiments Christian Huitema
- Re: Multi-path QUIC Extension Experiments Lucas Pardue
- Re: Multi-path QUIC Extension Experiments Lars Eggert
- Re: Multi-path QUIC Extension Experiments Robin MARX
- Re: Multi-path QUIC Extension Experiments Mikkel Fahnøe Jørgensen
- Re: Multi-path QUIC Extension Experiments Mikkel Fahnøe Jørgensen
- Re: Multi-path QUIC Extension Experiments Behcet Sarikaya
- Re: Multi-path QUIC Extension Experiments Christian Huitema
- Re: Multi-path QUIC Extension Experiments Yunfei Ma
- Re: Multi-path QUIC Extension Experiments Ian Swett
- Re: Multi-path QUIC Extension Experiments Yunfei Ma
- Re: Multi-path QUIC Extension Experiments Spencer Dawkins at IETF
- Re: Multi-path QUIC Extension Experiments Christian Huitema
- Re: Multi-path QUIC Extension Experiments Spencer Dawkins at IETF
- Re: Multi-path QUIC Extension Experiments Yunfei Ma