Re: Alibaba's Technical Report of single packet number space implementation (SPNS) for multi-path QUIC

[Yunfei Ma <[e-mail]>]

Hi Quentin,

Please see my response below:

On Fri, Oct 21, 2022 at 12:20 AM Quentin De Coninck <
[e-mail]> wrote:

> Hi Yunfei,
>
> Thanks for the report! This is an interesting in-the-wild experience paper
> complementary to an experience I performed some months ago with picoquic in
> NS3 environment exploring various 2-path setups (“The Packet Number Space
> Debate in Multipath QUIC. In: ACM SIGCOMM Computer Communication Review,
> July 2022 issue“, attached here for convenience). Most of your findings
> (2, 3 and 4) are consistent with my previous observations.
>

We read your report and it was very nice! Thanks for your experiment!


> As an individual, my main concern regarding single packet number space
> (SPNS) solution relates to the receiver implementation, especially
> regarding the packet number duplicate detection. Given that paths may have
> very different latencies, a receiver might eventually drop all packets
> coming from slow paths because their packet numbers are too low compared to
> other paths. Assume for instance that your receiver tracks the last 150
> packets before the largest PN seen. Assume 1-200 are sent on a slow path
> and 201-400 on a fast one. If all of 201-400 arrive at the receiver before
> any of 1-200, all the packets of the 1-200 burst will be dropped by the
> receiver (due to duplicate prevention). Of course, one could adopt more
> clever receiver’s heuristics (e.g., timing-based). But in any case, I don’t
> see how a receiver could always make the right distinction between a very
> delayed packet (number) on a slow path and an actually lost one. Because
> the sender does not advertise, in advance, how it will spread packet number
> over paths, neither its current lowest in-flight packet number. In multiple
> packet number space (MPNS), you have monotonically increasing per-path
> packet numbers, so there is no such here there. I wonder if this is
> somewhat related to your finding (3.). Did you adapted the receiver
> algorithm in some way to tackle this?
>

What you can do is to record the largest PN seen by each path at the
receiver and track the packet number back to the minimum of the largest
received PN among all paths minus N (In your above case, N is 150). You are
right that it requires the receiver to make algorithm changes and if you
don’t do that, it will cause packet loss. Another issue is that, due to the
large path delay difference, when the number of holes is larger than the
maximum number of ACK ranges in the ACK frame, it will also cause spurious
loss. Both of these issues could lead to reduced performance when data rate
is large.

>
>
In general, multipath transport performance (e.g. MPTCP) depends on the
> sender (scheduling,...) and the network. With SPNS, you will also have the
> receiver implementation that will interfere, especially if it does not
> correctly advertise all the packets it received, which is IMHO annoying.
> This would make some traffic patterns hard to benefit from multipath in
> some network situations. Of course, with MPNS the receiver still needs to
> send ACK_MP frames “in a correct way”, but this is the case for any
> SPNS/MPNS variant anyway.
>
> Also, as your report outlines as well, providing good SPNS performance
> also requires specific changes in the recovery, RTT, ACK generation, ECN,…
> algorithms, while with MPNS the usual QUIC ones can be applied per-path.
>

Yes, SPNS implementation needs some effort and MPNS is more straightforward
to implement.

>
>
>From your experience, would you recommend going one variant only or keeping
> both is still valuable?
>

My feeling is to use MPNS as much as possible, unless one really needs
zero-cid support. The support of zero-cid in multipath quic is what we want
more feedback and opinion from the working group.

>
> Best regards,
> Quentin
>

Cheers,
Yunfei

>
>
> On 21 Oct 2022, at 06:22, Yunfei Ma <[e-mail]> wrote:
>
>
> Hi All,
>
> At the last IETF meeting (114), one of the key questions regarding
> multipath QUIC was on the choice of packet number space. The current
> working group draft (02) has the multiple packet number space (MPNS) as the
> required and the single packet number space (SPNS) as the optional. While
> MPNS is relatively straightforward to implement, SPNS is not. To help the
> community better understand the implementation complexity and performance
> tradeoff, we have wrapped up a report on our SPNS implementation and
> experiments with real-world deployments at Alibaba's Taobao RPC scenario.
> Please check the attached report at the end of this email.
>
> Some of our findings:
> 1. We outline our algorithm changes and show that SPNS could achieve
> similar RTT estimation accuracy as MPNS (Fig. 3b).
> 2. There is an ACK size bloating issue for SPNS. We find that even without
> packet loss, in SPNS the number of ACK range holes can still be large (Fig.
> 3c).
> 3. When the request size is small, SPNS achieves almost the same speed as
> MPNS. However, when the request size becomes large, we have observed speed
> degradation of SPNS compared to MPNS (Fig. 5a).
> 4. The ACK size can be suppressed. We set two limits in ACK ranges. One is
> the default_range and the other is the max_range. (See sec. 2.4). However,
> reducing ACK size may sacrifice performance.
>
> We hope this report can serve as a useful reference for engineers and
> researchers who are interested in multipath QUIC, and we encourage more
> testing and experiments from the community, and together, move this
> technology forward.
>
> I would like to acknowledge my two coauthors Yingqi and Yanmei for this
> report. We also want to thank Christian Huitema, Mirja Küehlewind,
> Quentin De Coninck and Olivier Bonaventure for helpful discussions on the
> implementation details and tradeoffs of SPNS.
>
> Cheers,
> Yunfei
>
>
> <Alibaba_SPNS_Report.pdf>
>
>
>