Re: [iccrg] Updates to draft-briscoe-iccrg-prague-congestion-control-03

[Marten Seemann <[e-mail]>]

> With L4S, you can get consecutive CE marks and I don’t think receivers
need to ACK every CE marked packet. So, probably it’s best to not assume
that every CE would illicit an immediate ACK. Another reason is ACK packets
can get dropped or reordered and then there is no guarantee of the order
(by looking at the ACK at sender) in which packets were received at the
receiver.

That's what RFC 9000 defines though. A compliant QUIC implementation will
send a new ACK for every CE-marked packet (ignoring batching optimizations
and packet loss of packets containing ACK frames). It seems like this is
suboptimal if L4S regularly causes frequent consecutive CE markings.
We used to have an "Ignore CE" field in the ACK_FREQUENCY frame. Maybe we
need to bring back that field, or add a field that allows for richer
signaling about which ECN feedback would be useful? This QUIC extension
just entered WGLC a few days ago. I've opened
https://github.com/quicwg/ack-frequency/issues/244 to discuss this.

On Wed, 1 Nov 2023 at 05:17, Vidhi Goel <[e-mail]> wrote:

>
>    1. Section 2.4.3: Similarly, what's the correct order to process an
>    ACK that reports an ECN marking: For example, an ACK might acknowledge 20
>    new packets, and report one ECN marking. I think the correct order would be
>    applying the additive increase for 19 packets first, and then applying the
>    multiplicative decrease afterwards. This is because receiving a CE-marked
>    packet would elicit an immediate ACK frame from a QUIC receiver (RFC 9000,
>    section 13.2.1). The draft should probably be explicit about this.
>
>
> [BB] Good point.
> I agree with your logic, and I've added this to the list of edits too.
> However, I think I'll word it as a SHOULD, 'cos it makes sense when CE
> triggers feedback, but the implementer might have better info in some
> scenarios.
>
>
> With L4S, you can get consecutive CE marks and I don’t think receivers
> need to ACK every CE marked packet. So, probably it’s best to not assume
> that every CE would illicit an immediate ACK. Another reason is ACK packets
> can get dropped or reordered and then there is no guarantee of the order
> (by looking at the ACK at sender) in which packets were received at the
> receiver.
>
> Instead of thinking about CC response based on guessing the order of
> packets being received at the receiver, we should think about it as,
> whatever maybe the order of CE or non-CE packets at the receiver, is
> applying additive increase before multiplicative decrease better than doing
> the other way.
>
> 1. AI before MD - This will apply reduction on a larger cwnd, probably
> safer.
> 2. MD before AI - this will apply reduction on existing cwnd (which is
> smaller than 1.) and then do additive increase. Final cwnd will be higher
> than option 1.
>
> I think option 1. is probably safer to alleviate the congestion faster at
> the bottleneck.
>
>
> Vidhi
>
> On Oct 31, 2023, at 9:33 AM, Bob Briscoe <ietf=
> [e-mail]> wrote:
>
> Marten,
>
> On 31/10/2023 10:50, Marten Seemann wrote:
>
> I read the draft and I'm trying to figure out how I'd implement Prague in
> my QUIC stack. There are a couple of things I've noticed:
>
>    1. Section 2.3.2: It’s unclear to me when exactly *alpha* is updated.
>    I assume that once I receive the first ACK, I save the timestamp. When I
>    receive a new ACK, there are two code paths: if it’s received within one
>    *rtt_virt*, just accumulate the counters used to calculate *frac*. If
>    it’s received after *rtt_virt*, update *alpha *according to the
>    equation given in this section, reset the counters for *frac* and save
>    the timestamp as the beginning of the next *rtt_virt* epoch. However,
>    this would mean that the *alpha* value used for multiplicative
>    decrease (section 2.4.2) would always be slightly outdated, which seems
>    suboptimal for an immediate response to a growing queue. Is there a better
>    way?
>
> [BB] Indeed. It's actually lot worse than "slightly outdated". As it says
> at the end of the section you refer to:
>
> However, another approach is being investigated because these per-RTT
> updates introduce 1--2 rounds of delay into the congestion response on top
> of the inherent round of feedback delay (see Section 3.1.3
> <https://datatracker.ietf.org/doc/html/draft-briscoe-iccrg-prague-congestion-control-03#pracc_faster_response>
>  in the section on variants and future work).
>
>
> Section 3.1.3 goes on to summarize the brief tech report I posted on arXiv
> that explains and defines an alternative approach that removes all this
> lag, cited as [PerAckEWMA]:
>     Removing the Clock Machinery Lag from DCTCP/Prague
> <https://arxiv.org/abs/2101.07727>
>
> Joakim Misund was evaluating it about a year ago, when he decided to get a
> proper job ;) instead of doing his PhD with me. I am just getting round to
> working on it myself again. But if you wanted to try it yourself as wel, we
> could certainly compare notes.
>
>
>    1.
>
>    Section 2.3.3: QUIC uses both packet- and time-threshold loss
>    detection (see sections 6.1.1 and 6.1.2 of RFC 9002). I’m not sure what
>    exactly the recommendation of this draft is.
>
> [BB] I hadn't appreciated that QUIC deems there's been a loss if /either/
> of these conditions is met [RFC9002; §6.1]:
>
> The packet was sent kPacketThreshold packets before an acknowledged packet
> (Section 6.1.1
> <https://datatracker.ietf.org/doc/html/rfc9002#packet-threshold>), or it
> was sent long enough in the past (Section 6.1.2
> <https://datatracker.ietf.org/doc/html/rfc9002#time-threshold>).
>
>
> Whereas TCP RACK [RFC8985], only uses DupThresh at the start of a flow
> (when the RTT is likely to be inaccurate) until a decent reordering window
> is established (if reordering is detected at all):
>
> if some reordering has been observed, then RACK does not trigger fast
> recovery based on DupThresh.
>
> Assuming the QUIC RFC is not just badly written, it doesn't seem right.
> Once a flow has got going, a time threshold is generally considered more
> robust than a packet-threshold. So, even if the packet-threshold is adapted
> in parallel to the time-threshold, using the logical OR of them both will
> often override the time-threshold with the less robust packet threshold.
>
>
>    1. [cont] It would certainly be possible to turn off packet-threshold
>    loss detection, and rely on time-threshold altogether. Is that what QUIC
>    implementations should do?
>
>
> [BB] Well, that's certainly what I thought RACK was meant to do.
> This might end up requiring an erratum to RFC9002.
>
>
>    1.
>
>    Section 2.4.2: Is the suppression of further decreases after one
>    ECN-triggered decrease for one *srtt*, or is it one *rtt_virt*?
>    Reading section 2.4.4 it sounds like it’s *rtt_virt*, but this could
>    probably be clarified in this section.
>
>
> [BB] Yes. I will check that the Linux code does that though.
>
> We should have caught that mistake in the draft recently when we went
> through the draft checking all the places where it had said 'RTT' before we
> introduced rtt_virt. I've added this to my list of edits to make for the
> next rev.
>
>
>
>    1. Section 2.4.3: The QUIC ACK frame acknowledges (multiple) ranges of
>    packets at the same time, together with cumulative ECN counts. It’s
>    therefore not possible to tell which packet was ECN-marked. This means that
>    a QUIC stack will be able to determine *acked_sacked*, but not
>    *ece_delta*. Is it valid to approximate it by assuming that all
>    packets had the same average size? Either way, this is pretty awkward to
>    fit into the pseudo-code given in appendix B.5 of RFC 9002.
>
>
> [BB] Yes. It has to be, given the current QUIC protocol.
>
> Despite having the opportunity to fully integrate ECN into the design of
> QUIC, it seems it was still a bit of an afterthought (I shouldn't complain,
> 'cos I volunteered to help with adding ECN, but then didn't get involved
> 'cos of other pressing work at the time).
>
>
>
>    1. Section 2.4.3: Similarly, what's the correct order to process an
>    ACK that reports an ECN marking: For example, an ACK might acknowledge 20
>    new packets, and report one ECN marking. I think the correct order would be
>    applying the additive increase for 19 packets first, and then applying the
>    multiplicative decrease afterwards. This is because receiving a CE-marked
>    packet would elicit an immediate ACK frame from a QUIC receiver (RFC 9000,
>    section 13.2.1). The draft should probably be explicit about this.
>
>
> [BB] Good point.
> I agree with your logic, and I've added this to the list of edits too.
> However, I think I'll word it as a SHOULD, 'cos it makes sense when CE
> triggers feedback, but the implementer might have better info in some
> scenarios.
>
>
>    1. Section 2.4.4: I'm struggling to follow how exactly cwnd is
>    supposed to change for small RTTs. Most important from an implementation
>    perspective: section 2.4.3 says that *ai_per_rtt* will have a
>    different value for small RTTs. It would be helpful if section 2.4.4 would
>    contain an equation for *ai_per_rtt*.
>
>
> [BB] I think the equation you want is already in §2.4.4, altho I agree
> that the text could be clearer, because it states rules you might think are
> right then explains why they're wrong, rather than saying what is right
> first, then explaining why:
>
> Therefore, the increase in cwnd per packet has to be (1/M^2) * (1/cwnd).
>
> This gives the increase per packet, not per RTT, but that's what is needed
> in an implementation isn't it? Or am I misunderstanding why you want the
> increase per RTT in particular?
>
> Thanks for all these useful comments and questions.
>
>
>
> Bob
>
>
>
> On Sat, 14 Oct 2023 at 19:45, Bob Briscoe <ietf=
> [e-mail]> wrote:
>
>> iccrg,
>>
>> We've just posted an update to prague-congestion-control.
>> Links to diffs are quoted below.
>> The main technical changes:
>>
>>    - the Apple implementation falls back to CUBIC behaviour on loss
>>    (both the reduction and the subsequent increase). Currently the Linux
>>    implementation still falls back to Reno on loss, but that is being changed.
>>    - how the Apple implementation over QUIC behaves when the path or the
>>    remote peer fails to support ECN properly
>>    - the items already discussed on this list in response to Neal's
>>    review, some of which were editorial, but others were technical, e.g.
>>       - pseudocode for removing integer rounding bias
>>       - clarifying the RTT-independence approach
>>
>> Cheers
>>
>>
>> Bob & co-authors
>>
>>
>>
>> -------- Forwarded Message --------
>> Subject:  New Version Notification for
>> draft-briscoe-iccrg-prague-congestion-control-03.txt
>> Date:  Sat, 14 Oct 2023 05:07:58 -0700
>> From:  [e-mail]
>> To:  Bob Briscoe <[e-mail]> <[e-mail]>, Koen De
>> Schepper <[e-mail]> <[e-mail]>,
>> Olivier Tilmans <[e-mail]>
>> <[e-mail]>, Vidhi Goel <[e-mail]>
>> <[e-mail]>
>>
>> A new version of Internet-Draft
>> draft-briscoe-iccrg-prague-congestion-control-03.txt has been successfully
>> submitted by Bob Briscoe and posted to the
>> IETF repository.
>>
>> Name: draft-briscoe-iccrg-prague-congestion-control
>> Revision: 03
>> Title: Prague Congestion Control
>> Date: 2023-10-14
>> Group: Individual Submission
>> Pages: 34
>> URL:
>> https://www.ietf.org/archive/id/draft-briscoe-iccrg-prague-congestion-control-03.txt
>> Status:
>> https://datatracker.ietf.org/doc/draft-briscoe-iccrg-prague-congestion-control/
>> HTML:
>> https://www.ietf.org/archive/id/draft-briscoe-iccrg-prague-congestion-control-03.html
>> HTMLized:
>> https://datatracker.ietf.org/doc/html/draft-briscoe-iccrg-prague-congestion-control
>> Diff:
>> https://author-tools.ietf.org/iddiff?url2=draft-briscoe-iccrg-prague-congestion-control-03
>>
>> Abstract:
>>
>> This specification defines the Prague congestion control scheme,
>> which is derived from DCTCP and adapted for Internet traffic by
>> implementing the Prague L4S requirements. Over paths with L4S
>> support at the bottleneck, it adapts the DCTCP mechanisms to achieve
>> consistently low latency and full throughput. It is defined
>> independently of any particular transport protocol or operating
>> system, but notes are added that highlight issues specific to certain
>> transports and OSs. It is mainly based on experience with the
>> reference Linux implementation of TCP Prague and the Apple
>> implementation over QUIC, but it includes experience from other
>> implementations where available.
>>
>> The implementation does not satisfy all the Prague requirements (yet)
>> and the IETF might decide that certain requirements need to be
>> relaxed as an outcome of the process of trying to satisfy them all.
>> Future plans that have typically only been implemented as proof-of-
>> concept code are outlined in a separate section.
>>
>>
>>
>> The IETF Secretariat
>>
>>
>> _______________________________________________
>> iccrg mailing list
>> [e-mail]
>> https://www.irtf.org/mailman/listinfo/iccrg
>>
>
> --
> ________________________________________________________________
> Bob Briscoe                               http://bobbriscoe.net/
>
> _______________________________________________
> iccrg mailing list
> [e-mail]
> https://mailman.irtf.org/mailman/listinfo/iccrg
>
>
>