From nobody Fri Dec 4 07:55:35 2020 Return-Path: X-Original-To: quic@ietfa.amsl.com Delivered-To: quic@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id BCC503A0E59 for ; Fri, 4 Dec 2020 07:55:26 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.087 X-Spam-Level: X-Spam-Status: No, score=-2.087 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_PASS=-0.001, T_SPF_HELO_TEMPERROR=0.01, UNPARSEABLE_RELAY=0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NbPgp-NUieyS for ; Fri, 4 Dec 2020 07:55:20 -0800 (PST) Received: from mail-yb1-xb2a.google.com (mail-yb1-xb2a.google.com [IPv6:2607:f8b0:4864:20::b2a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id B49063A0E15 for ; Fri, 4 Dec 2020 07:55:19 -0800 (PST) Received: by mail-yb1-xb2a.google.com with SMTP id 2so5765755ybc.12 for ; Fri, 04 Dec 2020 07:55:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:in-reply-to:references:mime-version:date:message-id:subject:to :cc; bh=HfPSVIB1gWIQYse5KoSbtLUPiHY+JwLAIg0cjHCJ5pw=; b=lVkV1XyDoSERhH5HKHeC91NhtPnfsqnQUH01hXoR0mDA6dp66YD4fjXOO+gMqeGtak PgBrnIpAz30xcLzzTl+g4+8zVyj4ibkODOVkFhKyaH7ndWWyZ1hFdzq9Xn27ACAH5yLU cPeSbmxO4ImRmZkFEjM//Z9UMQrq5WOq4ejJON5vIh9B0kwUmrSKWgSPolkqTMiPXp2T gLknjtHwhHwcZZVysvp5ujgmmzTbCEZ5sFYFTzjmktyYsc6mVXKuV6U4IB1BY9sRHjMO 1+gXVquUzWbX+rEzleGZ+P4d2lZmwEX7s8mZQaz1yT8O03ZO0XBF2b5A0AH87jgrJm+6 8VAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:in-reply-to:references:mime-version:date :message-id:subject:to:cc; bh=HfPSVIB1gWIQYse5KoSbtLUPiHY+JwLAIg0cjHCJ5pw=; b=AcBF63dlRxOrDpGT/4Gx/pqs74t9avFM6aJySIvklHlTCHSWXv51kt9FU4UvMTzEJG fyA8ixpVXiGCXRLjw9LbWyseUo7b6DsEzGIakj623GbEry7CtlaMyb/ABZLZLINF9aBM UmCtGOGCu0FbeHPsasYoeZt2HI2A3ZTk9K9H2Doz2uUeDnN0Zt7t2qx5Nt0WU724IQ3E SJ4uDHQiYc1gdk41qyIDpLwKWqw9Woy6fGMuNNem3GcQm3Hf2WuVGre1HhkHQhYsaFqT WhESjVPeB9SjzklkfzAbsbaEY5Jw52ExCuTLvo9246vDMziInKvXRY/VLbt4/Wn7FcHB OVnw== X-Gm-Message-State: AOAM5332QW15mc4Qvk2Ejp6ZuSMCJx6O8plwkznImTCUM4cmZzxMGt/a p3XT2oe5ALlEyQ1vYkyzukQ/bccCuhq3xC2WgSE= X-Google-Smtp-Source: ABdhPJz11gMIU/WhEWPxx1EeXl2U+yB2UZRad5GwBKE2OTY/gGkzp7TK/WJkQ8J7GbV6sqcGwo9otTCxyKbgX2igUmI= X-Received: by 2002:a25:5382:: with SMTP id h124mr6152131ybb.263.1607097318885; Fri, 04 Dec 2020 07:55:18 -0800 (PST) Received: from 1058052472880 named unknown by gmailapi.google.com with HTTPREST; Fri, 4 Dec 2020 07:55:18 -0800 From: =?UTF-8?Q?Mikkel_Fahn=C3=B8e_J=C3=B8rgensen?= In-Reply-To: <8dcdb29c-bb94-8eda-32c9-9d147ca4fa6e@uclouvain.be> References: <538215d1-3b9e-4784-920d-03be4c3a503a.miaoji.lym@alibaba-inc.com> <062fe812-8afb-d946-8336-1f4dc5ebeaaf@uclouvain.be> <7540ef46-9948-c76c-3617-5755be3cdf37@huitema.net> <3bb7f359-ebe5-7a54-0224-bb1f5f1754af@huitema.net> <651e0ae1-0a5e-89e9-55c0-c33439599da6@huitema.net> <4b96dbb8-e72c-7f99-0bb3-9ee27b7bda78@huitema.net> <850c5bdd-948e-269a-1488-77a77843d5e6@huitema.net> <3A1F078F-6F88-47E8-AEB0-5E8C889AC28C@ericsson.com> <8dcdb29c-bb94-8eda-32c9-9d147ca4fa6e@uclouvain.be> MIME-Version: 1.0 Date: Fri, 4 Dec 2020 07:55:18 -0800 Message-ID: Subject: Re: Packet number spaces in multipath (was Re: What to do about multipath in QUIC) To: Jana Iyengar , Quentin De Coninck , Mirja Kuehlewind Cc: Kazuho Oku , Christian Huitema , IETF QUIC WG Content-Type: multipart/alternative; boundary="000000000000cfce3505b5a58008" Archived-At: X-BeenThere: quic@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Main mailing list of the IETF QUIC working group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Dec 2020 15:55:34 -0000 --000000000000cfce3505b5a58008 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable If there is a unique CID per path, that problem would go away, right? Using the same CID on multiple paths could both cause privacy and routing issues. Mikkel On 4 December 2020 at 10.21.33, Quentin De Coninck ( quentin.deconinck@uclouvain.be) wrote: Jana and all, I just have a concern about such a design using sequence number of the connection ID as the path ID. Assume that the server is willing to offer two available connection IDs at any time, and that the client is using both at the same time for multipath transfer. It uses the CID with seq num 0 on path A and the CID with seq num 1 on path B. Then, for any reason (like privacy), the client wants to change the connection ID used on path B (using seq num 1). However, the host cannot retire the CID with seq num 1 without retiring the CID with seq num 0 too. If the server does not want to provide additional connection IDs and the client is not willing to reuse CID with seq num 1 on path A, the client is stuck with the CID with seq num 0 on path A and cannot use the path B anymore. This is why I believe we should not link the path ID to the sequence number of the Connection ID (because it is a monotonically increasing sequence number), and rather have a separate space for them. Best regards, Quentin Mirja, I'm referring to what Christian was summarizing below. Separate PN spaces but path ID is implicit as the sequence number of the connection ID, and ACKs reflect this sequence number. - jana On Wed, Dec 2, 2020 at 3:02 AM Mirja Kuehlewind < mirja.kuehlewind@ericsson.com> wrote: > Hi Jana, > > > > can you maybe confirm what you mean by =E2=80=9Cthe design=E2=80=9D below= just to make > sure we are all on the same page: Is that different PN spaces per path, b= ut > using the same key on all paths with CIDs as part of the nonce? > > > > Thanks! > > Mirja > > > > > > *From: *QUIC on behalf of Jana Iyengar < > jri.ietf@gmail.com> > *Date: *Wednesday, 25. November 2020 at 04:35 > *To: *Christian Huitema > *Cc: *IETF QUIC WG , Kazuho Oku > *Subject: *Packet number spaces in multipath (was Re: What to do about > multipath in QUIC) > > > > (I'm taking Spencer's suggestion to spin off a new thread.) > > > > Christian, Kazuho, > > > > Slowly catching up on this, and apologies if I'm missing anything that wa= s > previously discussed in the centi-thread earlier. > > > > If I understand the design correctly, it makes sense to me, and is very > close to what we had implemented in Chromium a while ago. > > > > Having thought about this problem several times in the past, I'd like to > share a few points that come to mind. > > > > First though, a point on terminology: the receiver maintains a separate > "ReceivedPackets" for each CID, probably for each CID sequence number > (CSN). Let's please not call this a SACK Dashboard, to avoid confusion. > > > > On the question of sending more than 2^32 packets, I think that resetting > the packet number (PN) is ok on new CIDs. I don't see why a sender would > need to maintain continuity across multiple paths anyways, since the CC a= nd > loss recovery contexts are going to be different across paths. A sender > _could_ still maintain these packets in the same "SentPackets" structure = if > it wants to, it would need an internal representation of CSN+PN to key of= f. > > > > ACK Frames: > > ------------------ > > Kazuho pointed out that when acknowledging, the ACK frame format should > include CSN. I agree. I would argue for a design where a receiver uses an > ACK frame per CSN (and encodes the CSN explicitly). There are multiple > values for doing this, the primary being that you benefit from compressio= n > when PNs are contiguous within a CSN. > > > > Return Path: > > ----------------- > > There are other ways to decide which return path to send an ACK on this, > but I would propose that a receiver respond on the most recently active > forward path. That is, the path on which a packet was most recently > received. This has the natural effect that a sender that wants to > distribute traffic in a particular way also causes ACKs to be distributed > similarly across the corresponding reverse paths. > > > > RTT measurements: > > --------------------------- > > The return path for ACK frames will impact RTT measurements. That is fine= . > It is more important that information reach the sender as soon as possibl= e > than that it should not affect RTT measurements; we can fix the sender > to measure and compensate as necessary. The estimated RTT statistics > reflect the distribution of samples, and if both paths are being used, th= en > the SmoothedRTT will reflect the expected value based on the traffic > distribution across paths. > > > > That said, it might be useful to track some new stats, especially about > how much later is a "late ack" -- an ACK frame that contains no useful > information -- is received. I'd have to think a bit more about this, but = I > think we can devise a stat here. This gives us useful information on the > longest return path, which we can then explicitly use as part of the PTO > computations, to compensate for the fact that the RTT is based on the > shortest return path. (I would _not_ use this stat in the time-based loss > detection timer, but PTO ought to be fine.) > > > > - jana > > > > On Tue, Nov 17, 2020 at 9:42 AM Christian Huitema > wrote: > > I have been thinking about variations of that. I think we are making > progress here. > > If we follow your design, we get two constraints: > > 1) That the receive maintains an acknowledgement list based on the CID > through which the packets are received. > > 2) That the senders guarantee that the same sequence number will not be > used more than once with a specific CID. > > The main implementation cost is for receivers. They have to allocate and > maintain a "SACK Dashboard" in the context of each CID that they issue. > > Senders have lots of control. For example, the "only once" condition is > also met if a simple sender uses a single number space, as long as it doe= s > not send more than 2^32 packets. That makes the design reasonably easy to > implement for constrained implementations. > > Zero length CID are still possible, but that means the receiver supports > only one PN space per sender. Multipath is not impossible, but you end up > managing a single RTT and a single recovery structure. Not very good, but > similar to what happens if multipath is implemented at the IP level. > > There is still an issue for coordinating the take down of a path. Suppose > that a client was using both Wi-Fi and LTE, and moves out of Wi-Fi range. > The server will find out eventually that the packets sent to the Wi-Fi pa= th > are never acknowledged, but that may take some time. It would be better i= f > the client could send a message saying something like "Abandon this path"= . > That's not the same semantic as "retire this CID". We need a new frame fo= r > that. > > "Abandon this path" is an extreme case. There are half-way steps, like > manage the relative priority of a path. > > --000000000000cfce3505b5a58008 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =
If t= here is a unique CID per path, that problem would go away, right?

Using the same CID on multi= ple paths could both cause privacy and routing issues.
=
Mikkel


On 4 December 2020 at 10.21.33, Quentin De Coninck (quentin.deconinck@uclouvain.= be) wrote:

=
=20 =20

Jana and all,

I just have a concern about such a design using sequence number of the connection ID as the path ID.

Assume that the server is willing to offer two available connection IDs at any time, and that the client is using both at the same time for multipath transfer. It uses the CID with seq num 0 on path A and the CID with seq num 1 on path B. Then, for any reason (like privacy), the client wants to change the connection ID used on path B (using seq num 1). However, the host cannot retire the CID with seq num 1 without retiring the CID with seq num 0 too. If the server does not want to provide additional connection IDs and the client is not willing to reuse CID with seq num 1 on path A, the client is stuck with the CID with seq num 0 on path A and cannot use the path B anymore.

This is why I believe we should not link the path ID to the sequence number of the Connection ID (because it is a monotonically increasing sequence number), and rather have a separate space for them.

Best regards,

Quentin

=20
Mirja,

I'm referring to what Christian was summarizing below. Separate PN spaces but path ID is implicit as the sequence number of the connection ID, and ACKs reflect this sequence number.

- jana

On Wed, Dec 2, 2020 at 3:02 A= M Mirja Kuehlewind <mirja.kuehlewind@ericsson.com> wrote:

Hi Jana,

=C2=A0

can you maybe confirm what you mean by =E2=80=9Cthe design=E2=80=9D bel= ow just to make sure we are all on the same page: Is that different PN spaces per path, but using the same key on all paths with CIDs as part of the nonce?

=C2=A0

Thanks!

Mirja

=C2=A0

=C2=A0

From: Q= UIC <quic-bounces@ietf.org> on behalf of Jana Iyengar <jri.ietf@gmail.com>
Date: Wednesday, 25. November 2020 at 04:35
To: Christian Huitema <huitema@huitema.net>
Cc: IETF QUIC WG <quic@ietf.org>, Kazuho Oku <kazuhooku@gmail.com>
Subject: Packet number spaces in multipath (was Re: What to do about multipath in QUIC)

=C2=A0

(I'= m taking Spencer's suggestion to spin off a new thread.)

=C2=A0<= /p>

Christi= an, Kazuho,

=C2=A0<= /p>

Slowly catching up on this, and apologies if I'm missing anything that was previously discussed in the centi-thread earlier.

=C2=A0<= /p>

If I understand the design correctly, it makes sense to me, and is very close to what we had implemented in Chromium a while ago.

=C2=A0<= /p>

Having thought about this problem several times in the past, I'd like to share a few points that come to mind.

=C2=A0<= /p>

First though, a point on terminology: the receiver maintains a separate "ReceivedPackets" for = each CID,=C2=A0probably for each CID sequence number (CSN)= . Let's please not call this a SACK Dashboard, to avoid confusion.

=C2= =A0

On th= e question of sending more than 2^32 packets, I think that resetting the packet number (PN) is ok on=C2=A0new CIDs. I don't see why a sender would = need to maintain continuity across multiple paths anyways, since the CC and loss recovery=C2=A0contexts are going to be different across paths. A sender _could_ still maintain these packets in the same "SentPackets" structure if it wants to, it = would need an internal representation of CSN+PN to key off.

=C2= =A0

ACK Frames:

-----= -------------

Kazuh= o pointed out that when acknowledging, the ACK frame format should include CSN. I agree. I would argue for=C2=A0a design where a receiver uses an ACK frame per CSN (and encodes the CSN explicitly). There are multiple values for doing this, the primary being that you benefit from compression when PNs are contiguous within a CSN.=C2=A0

=C2= =A0

Ret= urn Path:

---= --------------

The= re are other ways to decide which return path to send an ACK on this, but I would propose that=C2=A0= a receiver respond on the most recently active forward path. That is, the path on which a packet was most recently received. This has the natural effect that a sender that wants to distribute traffic=C2=A0in a particular way also causes ACKs to be distributed similarly across the corresponding reverse paths.

=C2= =A0

RTT measurements:

---= ------------------------

The return path for ACK frames will impact RTT measurements. That is fine. It is more important that information reach the sender as soon as possible than that it should not affect RTT measurements; we can fix the sender to=C2=A0measure and compensate as necessary. The estimated RTT statistics reflect the distribution of samples, and if both paths are being used, then the SmoothedRTT will reflect the expected value based on the traffic distribution across paths.

=C2= =A0

Tha= t said, it might be useful to track some new stats, especially about how much later is a "late ack" -- an ACK=C2=A0frame=C2=A0that= contains no useful information -- is received. I'd have to think a bit more about this, but I think we can devise a stat here. This gives us useful information on the longest return path, which we can then explicitly use as part of the PTO computations, to compensate for the fact that the RTT is based on the shortest return path. (I would _not_ use this stat in the time-based loss detection timer,=C2=A0 but PTO ought to be fine.)

=C2= =A0

- jana

=C2= =A0

On Tue, Nov 17, 2020 at 9:42 AM Christian Huitema <huitema@huitema.net> wrote:

I have been thinking about variations of that. I think we are making progress here.

If we follow your design, we get two constraints:

1) That the receive maintains an acknowledgement list based on the CID through which the packets are received.

2) That the senders guarantee that the same sequence number will not be used more than once with a specific CID.

The main implementation cost is for receivers. They have to allocate and maintain a "SACK Dashboard" in the context of each CID th= at they issue.

Senders have lots of control. For example, the "only once" condition is also met if a simple sender uses a single number space, as long as it does not send more than 2^32 packets. That makes the design reasonably easy to implement for constrained implementations.

Zero length CID are still possible, but that means the receiver supports only one PN space per sender. Multipath is not impossible, but you end up managing a single RTT and a single recovery structure. Not very good, but similar to what happens if multipath is implemented at the IP level.

There is still an issue for coordinating the take down of a path. Suppose that a client was using both Wi-Fi and LTE, and moves out of Wi-Fi range. The server will find out eventually that the packets sent to the Wi-Fi path are never acknowledged, but that may take some time. It would be better if the client could send a message saying something like "Abandon this path".= That's not the same semantic as "retire this CID". We need a new frame for that.

"Abandon thi= s path" is an extreme case. There are half-way steps, like manage the relative priority of a path.

=20
--000000000000cfce3505b5a58008--