Re: Packet number spaces in multipath (was Re: What to do about multipath in QUIC)

[Christian Huitema <[e-mail]>]

If you want to send unreliable data with sequencing information, it 
might be simpler to use STREAM frame in "unreliable" mode than to use 
Datagram frames.

On 11/26/2020 3:34 AM, [e-mail] wrote:
>
> Dear all,
>
> sry for hijacking this conversation. I’m not very familiar with the 
> different multipath designs for QUIC, however I want to draw attention 
> to multipath re-ordering which probably becomes important when 
> multipath is combined with DATAGRAM.
>
> As long as multipath QUIC is operated with strict reliability (similar 
> to TCP), re-ordering on receiver side is a simple process known from 
> MPTCP. Introducing unreliable DATAGRAM transmission makes it more 
> challenging on receiver side to maintain the packet order, because it 
> is not easy to differentiate between delayed and lost packets. To 
> avoid HoL, a multipath re-ordering process may benefit from having 
> connection and path sequencing. In 
> https://tools.ietf.org/html/draft-amend-iccrg-multipath-reordering-01 
> <https://tools.ietf.org/html/draft-amend-iccrg-multipath-reordering-01> 
> we intend to describe this in section 5.6, how fast packet loss 
> detection can be applied using these different packet sequence spaces. 
> Still the description is meaningless und will be updated until next 
> IETF, however we have successfully implemented this approach in a 
> MP-DCCP prototype, which faces similar challenges in terms of 
> re-ordering. That means, fast packet loss detection is very beneficial 
> for the receiver re-ordering process to not lose time until an 
> outstanding packet is assumed lost.
>
> Br
>
> Markus
>
> **
>
> *From:*QUIC <[e-mail]> *On Behalf Of *Jana Iyengar
> *Sent:* Mittwoch, 25. November 2020 04:35
> *To:* Christian Huitema <[e-mail]>
> *Cc:* IETF QUIC WG <[e-mail]>; Kazuho Oku <[e-mail]>
> *Subject:* Packet number spaces in multipath (was Re: What to do about 
> multipath in QUIC)
>
> (I'm taking Spencer's suggestion to spin off a new thread.)
>
> Christian, Kazuho,
>
> Slowly catching up on this, and apologies if I'm missing anything that 
> was previously discussed in the centi-thread earlier.
>
> If I understand the design correctly, it makes sense to me, and is 
> very close to what we had implemented in Chromium a while ago.
>
> Having thought about this problem several times in the past, I'd like 
> to share a few points that come to mind.
>
> First though, a point on terminology: the receiver maintains a 
> separate "ReceivedPackets" for each CID, probably for each CID 
> sequence number (CSN). Let's please not call this a SACK Dashboard, to 
> avoid confusion.
>
> On the question of sending more than 2^32 packets, I think that 
> resetting the packet number (PN) is ok on new CIDs. I don't see why a 
> sender would need to maintain continuity across multiple paths 
> anyways, since the CC and loss recovery contexts are going to be 
> different across paths. A sender _could_ still maintain these packets 
> in the same "SentPackets" structure if it wants to, it would need an 
> internal representation of CSN+PN to key off.
>
> ACK Frames:
>
> ------------------
>
> Kazuho pointed out that when acknowledging, the ACK frame format 
> should include CSN. I agree. I would argue for a design where a 
> receiver uses an ACK frame per CSN (and encodes the CSN explicitly). 
> There are multiple values for doing this, the primary being that you 
> benefit from compression when PNs are contiguous within a CSN.
>
> Return Path:
>
> -----------------
>
> There are other ways to decide which return path to send an ACK on 
> this, but I would propose that a receiver respond on the most recently 
> active forward path. That is, the path on which a packet was most 
> recently received. This has the natural effect that a sender that 
> wants to distribute traffic in a particular way also causes ACKs to be 
> distributed similarly across the corresponding reverse paths.
>
> RTT measurements:
>
> ---------------------------
>
> The return path for ACK frames will impact RTT measurements. That is 
> fine. It is more important that information reach the sender as soon 
> as possible than that it should not affect RTT measurements; we can 
> fix the sender to measure and compensate as necessary. The estimated 
> RTT statistics reflect the distribution of samples, and if both paths 
> are being used, then the SmoothedRTT will reflect the expected value 
> based on the traffic distribution across paths.
>
> That said, it might be useful to track some new stats, especially 
> about how much later is a "late ack" -- an ACK frame that contains no 
> useful information -- is received. I'd have to think a bit more about 
> this, but I think we can devise a stat here. This gives us useful 
> information on the longest return path, which we can then explicitly 
> use as part of the PTO computations, to compensate for the fact that 
> the RTT is based on the shortest return path. (I would _not_ use this 
> stat in the time-based loss detection timer,  but PTO ought to be fine.)
>
> - jana
>
> On Tue, Nov 17, 2020 at 9:42 AM Christian Huitema <[e-mail] 
> <mailto:[e-mail]>> wrote:
>
>     I have been thinking about variations of that. I think we are
>     making progress here.
>
>     If we follow your design, we get two constraints:
>
>     1) That the receive maintains an acknowledgement list based on the
>     CID through which the packets are received.
>
>     2) That the senders guarantee that the same sequence number will
>     not be used more than once with a specific CID.
>
>     The main implementation cost is for receivers. They have to
>     allocate and maintain a "SACK Dashboard" in the context of each
>     CID that they issue.
>
>     Senders have lots of control. For example, the "only once"
>     condition is also met if a simple sender uses a single number
>     space, as long as it does not send more than 2^32 packets. That
>     makes the design reasonably easy to implement for constrained
>     implementations.
>
>     Zero length CID are still possible, but that means the receiver
>     supports only one PN space per sender. Multipath is not
>     impossible, but you end up managing a single RTT and a single
>     recovery structure. Not very good, but similar to what happens if
>     multipath is implemented at the IP level.
>
>     There is still an issue for coordinating the take down of a path.
>     Suppose that a client was using both Wi-Fi and LTE, and moves out
>     of Wi-Fi range. The server will find out eventually that the
>     packets sent to the Wi-Fi path are never acknowledged, but that
>     may take some time. It would be better if the client could send a
>     message saying something like "Abandon this path". That's not the
>     same semantic as "retire this CID". We need a new frame for that.
>
>     "Abandon this path" is an extreme case. There are half-way steps,
>     like manage the relative priority of a path.
>