Re: Alibaba's Technical Report of single packet number space implementation (SPNS) for multi-path QUIC

[Quentin De Coninck <[e-mail]>]

Hi Yunfei,

Thanks for the report! This is an interesting in-the-wild experience paper complementary to an experience I performed some months ago with picoquic in NS3 environment exploring various 2-path setups (“The Packet Number Space Debate in Multipath QUIC. In: ACM SIGCOMM Computer Communication Review, July 2022 issue“, attached here for convenience). Most of your findings (2, 3 and 4) are consistent with my previous observations.

As an individual, my main concern regarding single packet number space (SPNS) solution relates to the receiver implementation, especially regarding the packet number duplicate detection. Given that paths may have very different latencies, a receiver might eventually drop all packets coming from slow paths because their packet numbers are too low compared to other paths. Assume for instance that your receiver tracks the last 150 packets before the largest PN seen. Assume 1-200 are sent on a slow path and 201-400 on a fast one. If all of 201-400 arrive at the receiver before any of 1-200, all the packets of the 1-200 burst will be dropped by the receiver (due to duplicate prevention). Of course, one could adopt more clever receiver’s heuristics (e.g., timing-based). But in any case, I don’t see how a receiver could always make the right distinction between a very delayed packet (number) on a slow path and an actually lost one. Because the sender does not advertise, in advance, how it will spread packet number over paths, neither its current lowest in-flight packet number. In multiple packet number space (MPNS), you have monotonically increasing per-path packet numbers, so there is no such here there. I wonder if this is somewhat related to your finding (3.). Did you adapted the receiver algorithm in some way to tackle this?

In general, multipath transport performance (e.g. MPTCP) depends on the sender (scheduling,...) and the network. With SPNS, you will also have the receiver implementation that will interfere, especially if it does not correctly advertise all the packets it received, which is IMHO annoying. This would make some traffic patterns hard to benefit from multipath in some network situations. Of course, with MPNS the receiver still needs to send ACK_MP frames “in a correct way”, but this is the case for any SPNS/MPNS variant anyway.

Also, as your report outlines as well, providing good SPNS performance also requires specific changes in the recovery, RTT, ACK generation, ECN,… algorithms, while with MPNS the usual QUIC ones can be applied per-path.

From your experience, would you recommend going one variant only or keeping both is still valuable?

Best regards,
Quentin


On 21 Oct 2022, at 06:22, Yunfei Ma <[e-mail]<mailto:[e-mail]>> wrote:

Hi All,

At the last IETF meeting (114), one of the key questions regarding multipath QUIC was on the choice of packet number space. The current working group draft (02) has the multiple packet number space (MPNS) as the required and the single packet number space (SPNS) as the optional. While MPNS is relatively straightforward to implement, SPNS is not. To help the community better understand the implementation complexity and performance tradeoff, we have wrapped up a report on our SPNS implementation and experiments with real-world deployments at Alibaba's Taobao RPC scenario. Please check the attached report at the end of this email.

Some of our findings:
1. We outline our algorithm changes and show that SPNS could achieve similar RTT estimation accuracy as MPNS (Fig. 3b).
2. There is an ACK size bloating issue for SPNS. We find that even without packet loss, in SPNS the number of ACK range holes can still be large (Fig. 3c).
3. When the request size is small, SPNS achieves almost the same speed as MPNS. However, when the request size becomes large, we have observed speed degradation of SPNS compared to MPNS (Fig. 5a).
4. The ACK size can be suppressed. We set two limits in ACK ranges. One is the default_range and the other is the max_range. (See sec. 2.4). However, reducing ACK size may sacrifice performance.

We hope this report can serve as a useful reference for engineers and researchers who are interested in multipath QUIC, and we encourage more testing and experiments from the community, and together, move this technology forward.

I would like to acknowledge my two coauthors Yingqi and Yanmei for this report. We also want to thank Christian Huitema, Mirja Küehlewind, Quentin De Coninck and Olivier Bonaventure for helpful discussions on the implementation details and tradeoffs of SPNS.

Cheers,
Yunfei


<Alibaba_SPNS_Report.pdf>