[tcpm] Re: PRR behaviour on detecting loss of a retransmission(WAS:I-D Action: draft-ietf-tcpm-prr-rfc6937bis-06.txt)

[Neal Cardwell <[e-mail]>]

Hi TCPM-ers,

The slides for our PRR presentation tomorrow at IETF are now posted here:
  https://datatracker.ietf.org/meeting/121/materials/slides-121-tcpm-prr-00

In the slides:

+ points (1) and (2) from the email discussion above are discussed in the
slides as "Open issue 1"

+ points (3) and (4) from the email discussion above are discussed in the
slides as "Open issue 2"

Our thoughts and the proposed new text may be easier to read in slide form.

Looking forward to the discussion tomorrow.

best regards,
neal



On Fri, Nov 1, 2024 at 4:55 PM Neal Cardwell <[e-mail]> wrote:

>
>
> On Tue, Oct 29, 2024 at 10:45 AM Markku Kojo <kojo=
> [e-mail]> wrote:
>
>> Hi Neal, all,
>>
>> I just noted that I have missed a bunch of replies as I had been out of
>> email for about one month starting in mid July. Many thanks for the
>> replies and clarifications and my apologies for not having time to track
>> and search the mailing list until now.
>>
>> I just now reply to this thread wrt lost retransmission handling. The
>> others are of less importance and possibly fine with the latest
>> adjustments. I'll pass through the rest of the threads ASAP within a few
>> coming days to see if there was anything important left.
>>
>> The major point with the suggested way of reinitializing the PRR state
>> using the same steps as in the beginning of recovery is that it
>> often results in incorrect result if one follows the current CC RFCs on
>> how to do multiplicative decrease. Reason: both FlightSize and cwnd are
>> badly off during Fast Recovery and must not be used to compute new
>> ssthresh (and cwnd).
>>
>> Pls see inline tagged [MK2].
>>
>> On Tue, 16 Jul 2024, Neal Cardwell wrote:
>>
>> >
>> >
>> > On Tue, Jun 25, 2024 at 10:25 AM Markku Kojo <kojo=
>> [e-mail]> wrote:
>> >       Hi Neal, all,
>> >
>> >       I changed the subject line for discussing this specific topic of
>> PRR
>> >       behaviour when loss of a retransmission is detected.
>> >
>> >       Please see below tagged [MK].
>> >
>> >       On Mon, 18 Mar 2024, Neal Cardwell wrote:
>> >
>> >       >       In addition, it seems that the algorithm in the latest
>> version does not
>> >       >       address my WGLC comment on reducing send rate (ssthresh)
>> again if
>> >       >       RACK-TLP detects loss of a retransmission. The sender
>> must reduce
>> >       >       ssthresh again as loss of a rexmit occurs on another RTT.
>> If it is not
>> >       >       done, the fast recovery keeps on sending at the same rate
>> until the end
>> >       >       of recovery regardless of how many times a segment has to
>> be
>> >       >       retransmitted. This sounds very bad behaviour to me in
>> front of heavy
>> >       >       congestion that drops a lot of pkts (rexmits) and the PRR
>> sender does not
>> >       >       react at all.
>> >       >
>> >       >
>> >       > I would argue that the question of whether a connection should
>> reduce ssthresh when
>> >       RACK-TLP detects the
>> >       > loss of a retransmission, while important, is outside the scope
>> of PRR. PRR is taking
>> >       loss detection and
>> >       > congestion control decisions as externally provided inputs into
>> PRR. When to mark a
>> >       packet as lost is a
>> >       > loss detection question, and whether to reduce ssthresh upon a
>> particular packet loss
>> >       is a congestion
>> >       > control decision. PRR is focused on taking the ssthresh output
>> from congestion
>> >       control, and loss detection
>> >       > decisions from the loss detection algorithm, and deciding how
>> to evolve the cwnd to
>> >       try to smoothly and
>> >       > safely converge the volume of in-flight data toward the given
>> ssthresh.
>> >
>> >       [MK] Yes, agreed that loss detection (including detecting the
>> loss of
>> >       a retransmission) is outside of scope of PRR (i.e., detecting
>> loss of
>> >       rexmit is currently RACK-TLP).
>> >
>> >       However, PRR is a congestion control algorithm defining the
>> congestion
>> >       control behaviour of the sender during a fast recovery. Currently
>> it
>> >       borrows only the multiplicative decrease factor from other
>> congestion
>> >       control algos, that is, either from RFC 5681 or RFC 9438) but
>> defines
>> >       everything else in controlling the send rate (= congestion ctrl)
>> during
>> >       fast recovery.
>> >
>> >       PRR does not need to define the multiplicative decrease factor to
>> be used
>> >       when a loss of rexmitted segment is detected. It may borrow it
>> from
>> >       another doc like it currently does for entering loss recovery.
>> >       However, I don't quite see how some other document possibly could
>> define
>> >       how the other PRR-specific variables are reinitialized,
>> >       e.g., RecoverFS. Maybe I am missing something but the algo seems
>> >       not to work correctly with a lowered ssthresh after detection of
>> >       lost rexmit unless RecoverFS (and prr_deliverd and prr_out too?)
>> is also
>> >       adjusted. Could you explain how the algo is supposed to work upon
>> >       detecting loss of a rexmit with multiplicative decrease factor of
>> 0.5,
>> >       for example.
>> >
>> >       Thanks,
>> >
>> >       /Markku
>> >
>> >
>> > Hi Markku,
>> >
>> > In regards to the way PRR is intended to work after a data sender
>> detects a lost retransmit, I have
>> > chatted with Matt about this, and I think Matt and I have a similar
>> perspective on this. Let me
>> > offer my thoughts:
>> >
>> > + I agree it doesn't make sense for other IETF documents to try to
>> define how PRR-specific variables
>> > are reinitialized. To do so would invite a combinatorial explosion of
>> standards, as every congestion
>> > control algorithm doc would need to specify how PRR-enabled and non-PRR
>> implementations should work.
>> > And then those documents would need to be updated every time the PRR
>> algorithm specification
>> > changes.
>>
>> [MK2] Yes, agreed.
>>
>> > + Likewise, I would argue that it doesn't make sense for the PRR
>> document to attempt to define (a)
>> > when a congestion control algorithm decides to slow down (reduce
>> ssthresh and/or cwnd),
>>
>> [MK2] Sure PRR document does not need to define (a). Lowering ssthresh
>> and cwnd on the loss of a retransmission is already MUST in RFC 5681. The
>> long term tradition in CC RFCs has been to repeat crucial MUSTs with a
>> normative reference. So, I thing this document should also explicitly
>> repeat that on detecting loss of a retransmission, the TCP sender MUST
>> lower ssthresh once per RTT (and give enough details on how to do it
>> correctly with PRR to avoid pitfalls with flightsize and cwnd, but not
>> saying how much to lower).
>>
>> > or (b) what
>> > the exact ssthresh value is as a result of the slow-down decision. For
>> the PRR document to attempt
>> > to document (a) and (b) would likewise result in a combinatorial
>> explosion of text, and dependencies
>> > between documents, and document updates.
>>
>> [MK2] Agreed, no need to define the exact ssthresh value, but needs to
>> advise how to compute it correctly (see below).
>>
>> > Instead, the model we are advocating with PRR is a separation of
>> concerns:
>> >
>> > + a congestion control algorithm decides:
>> >   (a) when a congestion control algorithm decides to slow down (reduce
>> ssthresh and/or cwnd)
>> >   (b) what the exact ssthresh value is as a result of the slow-down
>> decision
>> >
>> > + PRR decides: given (a) and (b) decisions made by a separate
>> congestion control algorithm, how to
>> > set the cwnd using each ACK
>> >
>> > How should that work, in practice, after a data sender detects a lost
>> retransmit?
>> >
>> > Each time the data sender detects a lost retransmit, the congestion
>> control algorithm should decide
>> > whether or not to slow down. I would think that, ideally, a
>> well-designed congestion control
>> > algorithm should slow down multiplicatively once per round trip, for
>> every round trip in which there
>> > is any loss detected (whether it is a lost retransmit or a lost
>> original retransmission). Any time
>> > the congestion control algorithm decides to slow down (for whatever
>> reason), it would initiate a new
>> > PRR episode, and invoke the PRR initialization code. That would take
>> care of initializing RecoverFS,
>> > prr_deliverd, and prr_out, to ensure that the PRR behavior over the
>> next round trip and beyond works
>> > correctly.
>>
>> [MK2] To ensure that multiplicative decrease becomes implemented
>> correctly
>> this document should give exact advise how to compute the new
>> ssthresh value. If one follows current RFCs (e.g., RFC 5681 or RFC 9438)
>> and computes the new value of ssthresh (and cwnd) either using flightsize
>> (ssthresh = 0.5 * FlightSize or ssthresh = 0.7 *FlightSize) or cwnd
>> (ssthresh = 0.5 * cwnd or ssthresh = 0.7 * cwnd) the result is often not
>> correct (or is more or less random).
>>
>> This is because fligthsize becomes inflated during fast recovery as the
>> TCP sender sends new data during the recovery (before any
>> cumulative/partial ACKs arrive). Similarly, with PRR cwnd reaches the
>> target (=correctly reduced) value only in the end of recovery, meaning
>> that cwnd is too big during the recovery.
>>
>> A typical, simple scenario with flightsize, for example:
>>
>> Amount of outstanding data is 100 segments and a loss is detected ->
>> ssthresh = 50 or 70 (assume CC algo is Reno or Cubic) and recovery
>> starts. Assume the fast rexmitted (1st lost) segment becomes dropped.
>>
>> During the first RTT the TCP sender injects 50 or 70 new data segments ->
>> FlightSize = 150 or 170. Soon after the first RTT, the TCP sender detects
>> the loss of the rexmitted segment and computes: ssthresh = 0.5 * 150 = 75
>> (Reno) or ssthresh = 0.7 * 170 = 119 (Cubic). This results in three times
>> higher ssthress value than expected with Reno (= 25) and ~ 2.5 times
>> higher than expected with Cubic (= 49).
>>
>> A simple scenario with cwnd, for example:
>>
>> Amount of outstanding data is 100 segments and a loss is detected ->
>> ssthresh = 50 or 70 (assume CC algo is Reno or Cubic) and recovery
>> starts. Assume there is significant number of losses in the current
>> window of data and the fast rexmitted (=1st lost) segment becomes
>> dropped. In addition, there may be significant Ack loss.
>> That is, a typical case with very heavy congestion and it would be
>> crucial to reduce ssthresh (and cwnd) correctly.
>>
>> During the first RTT only little date gets delivered (i.e., hardly any
>> SACKed data and little additional lost segments are detected, keeping
>> cwnd ~ flightsize (= cwnd before entering recovery). When lost rexmit
>> becomes detected after one RTT, the TCP sender computes new ssthresh =
>> 0.5 * cwnd or 0.7 * cwnd and the results is only minimal reduction from
>> the ssthresh used during the first RTT of recovery, instead lowering
>> twice
>> with the same multiplicative decrease factor.
>>
>> I hope this clarifies the issue and the need to define that ssthresh MUST
>> NOT be reinitialized using flightsize or cwnd.
>>
>> Maybe on detecting loss of a rexmit, ssthresh could be reinitialized
>>
>>   ssthresh = multiplicatice_decrease_factor * ssthresh?
>>
>> In addition, when rereading the prr algo I found also an additonal
>> problem with the PRR algo that uses pipe (RFC 6675 pipe algorithm)
>> together with RACK-TLP loss detection. The algo uses pipe as the
>> (quite accurate) estimate of outstanding data. However, the definition of
>> pipe depends on loss detection in RFC 6675 that defines a lost segment
>> different from RACK-TLP. The pipe algo depends of RFC 6675 IsLost()
>> function that requires three SACked segments above the lost segment to
>> declare the segment lost, while this is often not the case with RACK-TLP.
>>
>> Shouldn't the "pipe" estimate in the algo be based on the loss detection
>> algorithm in use? Otherwise, pipe may be badly off in number of
>> scenarios.
>>
>> The same holds for non-SACK fast recovery, that is NewReno. I think the
>> document/algo should clarify that, without SACK, the SACKed segments for
>> pipe calculation are estimated in the similar way as they are estimated
>> for DeliveredData (i.e., one SACKd segment = one duplicate ACK).
>>
>> Hope this is helpful.
>>
>> Best regards,
>>
>> /Markku
>>
>> > I would propose that we make that more clear in the PRR document.
>> >
>> > In section 6, "Algorithm", I would propose we change the existing text:
>> >
>> >   At the beginning of recovery, initialize the PRR state.
>> > to:
>> >
>> >    At the beginning of a congestion control response episode initiated
>> >    by the congestion control algorithm, a TCP data sender using PRR
>> >    MUST initialize the PRR state. The timing of the start of a
>> >    congestion control response episode is entirely up to the
>> >    congestion control algorithm, and (for example) could correspond to
>> >    the start of a fast recovery episode, or a once-per-round-trip
>> >    reduction when lost retransmits or lost original transmissions are
>> >    detected after fast recovery is already in progress.
>> >
>> > How does that sound to everyone?
>> >
>> > neal
>>
>
> Hi Markku,
>
> Markku, thanks once again for the detailed feedback!
>
> If I understand correctly, you are basically making four high-level points:
>
> (1) [when to reduce ssthresh on lost retransmits]
>
> > ...I thin[k] this document should also explicitly
> > repeat that on detecting loss of a retransmission, the TCP sender MUST
> > lower ssthresh once per RTT (and give enough details on how to do it
> > correctly with PRR to avoid pitfalls with flightsize and cwnd, but not
> > saying how much to lower).
>
> (2) [how to set ssthresh on lost retransmits]
>
> > The major point with the suggested way of reinitializing the PRR state
> > using the same steps as in the beginning of recovery is that it
> > often results in incorrect result if one follows the current CC RFCs on
> > how to do multiplicative decrease. Reason: both FlightSize and cwnd are
> > badly off during Fast Recovery and must not be used to compute new
> > ssthresh (and cwnd).
> > ...ssthresh MUST NOT be reinitialized using flightsize or cwnd.
> > ...Maybe on detecting loss of a rexmit, ssthresh could be reinitialized
> >
> >   ssthresh = multiplicative_decrease_factor * ssthresh?
>
> IMHO the (1)/(2) questions of exactly when to set ssthresh (sequence
> window? time window? lost retransmits?) and how to set ssthresh (based on
> FlightSize? cwnd?) is tricky, and opens up a whole can of worms.
>
> Many of the tricky issues are nicely described in detail in this nice IETF
> 120 deck from Matt Mathis:
>
> https://datatracker.ietf.org/meeting/120/materials/slides-120-ccwg-cc-response-while-application-limited-00
>
> The issue is so thorny that, AFAIK, all the major TCP implementations
> (Linux, FreeBSD, Windows, MacOS/iOS) diverge from the Reno/RFC5681 and
> CUBIC/RFC9438 standards, and do not set ssthresh based on FlightSize, but
> rather based on cwnd.
>
> IMHO this is a much deeper and wider problem than what can be reasonably
> tackled in a PRR draft.
>
> So for (1) and (2) my sense is still that it is best to leave the question
> of when/how to set ssthresh as something to be documented outside the PRR
> draft.
>
> So I will repeat my proposal: In section 6, "Algorithm", I would propose
> we change the existing text:
>
>   At the beginning of recovery, initialize the PRR state.
>
> to:
>
>    At the beginning of a congestion control response episode initiated
>    by the congestion control algorithm, a TCP data sender using PRR
>    MUST initialize the PRR state. The timing of the start of a
>    congestion control response episode is entirely up to the
>    congestion control algorithm, and (for example) could correspond to
>    the start of a fast recovery episode, or a once-per-round-trip
>    reduction when lost retransmits or lost original transmissions are
>    detected after fast recovery is already in progress.
>
> We will discuss this in the PRR slides for TCPM next week, and hopefully
> the TCPM chairs can moderate a discussion about whether to specify how/when
> to set ssthresh in the PRRbis draft.
>
> (3) ["pipe" and current loss detection algorithm]
> > Shouldn't the "pipe" estimate in the algo be based on the loss detection
> > algorithm in use?
>
> (4) ["pipe" and current loss detection algorithm for non-SACK connections]
> > The same holds for non-SACK fast recovery, that is NewReno. I think the
> > document/algo should clarify that, without SACK, the SACKed segments for
> > pipe calculation are estimated in the similar way as they are estimated
> > or DeliveredData (i.e., one SACKd segment = one duplicate ACK).
>
> I agree that your points (3) and (4) make sense. To address your points
> (3) and (4), where the PRRbis draft currently uses "pipe" (as defined in
> RFC 6675) it should probably instead use a name referring to the estimated
> volume of data in-flight in the network as estimated by the *current* loss
> detection algorithm (which is often not RFC 6675, e.g. in Linux TCP).
>
> I would propose that the PRRbis draft use "inflight" since (a) this is the
> term used for this concept in Linux TCP, BBR (CACM article and Internet
> draft), and the Prague draft, and (b) there should not be confusion because
> "inflight" is not used anywhere currently in PRR (RFC6937 or PRRbis),
> RFC6675, Reno/RFC5681, or CUBIC/RFC9438.
>
> In Sec 5, "Definitions", I would propose to add:
>
> inflight: The transport connection's sender-side estimate of the number of
> bytes still outstanding in the network. Connections with SACK and using RFC
> 6675 loss detection MAY use "pipe" as specified in RFC 6675. For
> connections using RACK-TLP loss detection [RFC8985] or other loss detection
> algorithms, they MUST calculate inflight by starting with SND.NXT -
> SND.UNA, subtracting out bytes SACKed in the SACK scoreboard, subtracting
> out bytes marked lost in the SACK scoreboard, and adding bytes in the
> scoreboard that have been retransmitted since they were last marked lost.
>
> Then I would propose to replace occurrences of "pipe" in the
> current draft-ietf-tcpm-prr-rfc6937bis-12 with "inflight".
>
> We will also add a slide about this in the PRR presentation at TCPM on
> Tuesday, to gather feedback on this proposal.
>
> Markku, thanks again for the detailed feedback!
>
> best,
> neal
>
>
>