Re: Multipath acknowledgments

[Michael Eriksson <[e-mail]>]

Hi Yunfei,

I agree that cross-path ACKs is a powerful tool and we seem to agree that for them to work well, an implementation needs to understand the peer's ACK strategy. That is doable if you control both sides of the connection, but then you don't need a protocol standard at all :-).

Until cross-path ACKs are better understood, I argue that they should be opt-in, and the default mechanism should be to always send ACKs in-path, a strategy that the spec already recommends:

   The recommended default behaviour consists in
   sending ACK(_MP) frames on the path they acknowledge packets.

Thinking a bit more, the use of cross-path ACKs should probably not only be a binary opt-in but a negotiation about the ACK strategy to use. It is then possible to experiment with different cross-path ACK strategies to see which works best.

/me

________________________________
From: Yunfei Ma <[e-mail]><mailto:[e-mail]>
To: Michael Eriksson <[e-mail]><mailto:[e-mail]>
Cc: IETF QUIC WG <[e-mail]><mailto:[e-mail]>
Subject: Multipath acknowledgments
Date: Sat, 19 Nov 2022 10:28:04 +0100 (Central European Standard Time)

Hi Michael,

You have a good point. Yes, the scheduling strategy on ACKs affects the performance, especially if you are using delay-based congestion control where putting ACKs on different paths could induce jitters in the inter-packet delay. Due to the fact that the congestion controller is on the sender-side but the scheduler for ACKs is on the receiver-side, there is a lack of sender-receiver coordination. One way to solve this problem is through the negotiation method you mentioned above, and the other way is to use a feedback signal that informs the peer on how to schedule packets.

In fact, for scheduling ACK-eliciting data packets, such a lack of coordination also proves to be a problem. For example, in XLINK, we use reinjection to enhance performance, however, aggressively reinjecting packets increases redundancy, and thus, traffic costs. Our solution is to let the receiver provide some feedback so that the aggressiveness of the sender-side scheduler can be dynamically adjusted. In an early draft, https://datatracker.ietf.org/doc/html/draft-liu-multipath-quic-04, we define a QoE_Control_Signals frame which serves this coordination purpose. The QoE_Control_Signals is designed to be flexible and it can be used to carry various semantics. In your case, if the sender wants the receiver to schedule ACKs in an in-path way, it can convey this message through QoE_Control_Signals.

In my opinion, the good thing about ACK_MP is that it is more capable than normal ACK, so even if you prefer to do in-path acknowledgement most of the time, ACK_MP still provides you an "emergency exit" if one of the return paths is completely blocked. On the other hand, we do want the ability to better coordinate a peer's scheduler (for both data and ack packets). Therefore, in addition to the core multipath draft, we may want to think about a separate draft discussing some advanced scheduling strategies that focus on how to achieve the best performance under various conditions.

Cheers,
Yunfei

On Fri, Nov 18, 2022 at 7:03 AM Michael Eriksson <michael.eriksson=[e-mail]<mailto:[e-mail]>> wrote:
Greetings fellow multipathers,

TL;DR: Multipath QUIC should by default use the same in-path acknowledgment design as unipath QUIC. Cross-path acknowledgments are interesting but complex, not yet fully understood and should be an opt-in extension.


The current multipath draft specifies that acknowledgment signaling should be done with a new frame type, ACK_MP, when the (potential) use of multiple paths has been agreed for a connection. This means that all implementations need to be able to handle incoming cross-path acknowledgments (XP ACKs), where acknowledgments of packets that have arrived on one path is sent on other paths. Sending XP ACKs is voluntary.

XP ACKs have some attractive properties, in particular these two:

  *   Some "self-clocking" congestion control algorithms, such as the popular CUBIC, can grow the congestion window faster if the ACK is sent on a path with lower latency
  *   The stream data retransmission and receive buffers can be smaller if the flow control feedback is faster

However, XP ACKs also have a few drawbacks, mainly because the experienced round-trip time varies depending on which path is used for each ACK. This implies that:

  *   Latency-based congestion control algorithms can be confused
  *   Loss detection is harder; both the packet loss time threshold and the probe timeout will have problems

I argue that these problems are not well understood, and forcing all multipath implementations to handle them is not a good idea. However, there are interesting possible optimizations that should be researched, so there should be a mechanism to negotiate the use of XP ACKs. If this should be specified in the main multipath spec or another document is a separate but minor question.

When the use of XP ACKs is not negotiated, each path should use the acknowledgment mechanism (including the use of the normal ACK frame) that is already implemented and proven to work with unipath QUIC.


Summary

Cross-path acknowledgments are so difficult to handle that implementations should not be forced to do that. The default should be to only do in-path acknowledgments exactly like unipath QUIC does.


/me