Re: [tcpm] Last Call: <draft-ietf-tcpm-rack-13.txt>(TheRACK-TLPlossdetectionalgorithm for TCP) to Proposed Standard

[Markku Kojo <[e-mail]>]

Hi Martin,

On Tue, 15 Dec 2020, Martin Duke wrote:

> Hi Markku,
> 
> Thanks for the comments. The authors will incorporate many of your
> suggestions after the IESG review.
> 
> There's one thing I don't understand in your comments:
> 
> " That is,
> where can an implementer find advice for correct congestion control
> actions with RACK-TLP, when:
> 
> (1) a loss of rexmitted segment is detected
> (2) an entire flight of data gets dropped (and detected),
>      that is, when there is no feedback available and a timeout
>      is needed to detect the loss "
> 
> Section 9.3 is the discussion about CC, and is clear that the
> implementer should use either 5681 or 6937.

Just a cite nit: RFC 5681 provides basic CC concepts and some useful CC 
guidelines but given that RACK-TLP MUST implement SACK the algorithm in 
RFC 5681 is not that useful and an implementer quite likely follows 
mainly the algorithm in RFC 6675 (and not RFC 6937 at all if not 
implementing PRR).
And RFC 6675 is not mentioned in Sec 9.3, though it is listed in the 
Sec. 4 (Requirements).

> You went through the 6937 case in detail.

Yes, but without correct CC actions.

> If 5681, it's pretty clear to me that in (1) this is a new loss
> detection after acknowledgment of new data, and therefore requires a
> second halving of cwnd.

Hmm, not sure what you mean by "this is a new loss detection after 
acknowledgment of new data"?
But anyway, RFC 5681 gives the general principle to reduce cwnd and 
ssthresh twice if a retransmission is lost but IMHO (and I believe many 
who have designed new loss recovery and CC algorithms or implemented them 
agree) that it is hard to get things right if only congestion control 
principles are available and no algorithm.
That's why ALL mechanisms that we have include a quite detailed algorithm 
with all necessary variables and actions for loss recovery and/or CC 
purposes (and often also pseudocode). Like this document does for loss 
detection.

So the problem is that we do not have a detailed enough algorithm or 
rule that tells exactly what to do when a loss of rexmit is detected.
Even worse, the algorithms in RFC 5681 and RFC 6675 refer to 
equation (4) of RFC 5681 to reduce ssthresh and cwnd when a loss 
requiring a congestion control action is detected:

  (cwnd =) ssthresh = FlightSize / 2)

And RFC 5681 gives a warning not to halve cwnd in the equation but 
FlightSize.

That is, this equation is what an implementer intuitively would use 
when reading the relevant RFCs but it gives a wrong result for 
outstanding data when in fast recovery (when the sender is in 
congestion avoidance and the equation (4) is used to halve cwnd, it 
gives a correct result).
More precisely, during fast recovery FlightSize is inflated when new 
data is sent and reduced when segments are cumulatively Acked. 
What the outcome is depends on the loss pattern. In the worst case, 
FlightSize is signficantly larger than in the beginning of the fast 
recovery when FlightSize was (correctly) used to determine the halved 
value for cwnd and ssthresh, i.e., equation (4) may result in 
*increasing* cwnd upon detecting a loss of a rexmitted segment, instead 
of further halving it.

A clever implementer might have no problem to have it right with some 
thinking but I am afraid that there will be incorrect implementations 
with what is currently specified. Not all implementers have spent 
signicicant fraction of their career in solving TCP peculiarities.

> For (2), the RTO timer is still operative so
> the RTO recovery rules would still follow.

In short:
When with a non-RACK-TLP implementation timer (RTO) expires: cwnd=1 MSS, 
and slow start is entered.
When with a RACK_TLP implementation timer (PTO) expires, 
normal fast recovery is entered (unless implementing 
also PRR). So no RTO recovery as explicitly stated in Sec. 7.4.1.

This means that this document explicitly modifies standard TCP congestion 
control when there are no acks coming and the retransmission timer 
expires

from: RTO=SRTT+4*RTTVAR (RTO used for arming the timer)
       1. RTO timer expires
       2. cwnd=1 MSS; ssthresh=FlightSize/2; rexmit one segment
       3. Ack of rexmit sent in step 2 arrives
       4. cwnd = cwnd+1 MSS; send two segments
       ...

to:   PTO=min(2*SRTT,RTO) (PRO used for arming the timer)
       1. PTO times expires
       2. (cwnd=1 MSS); (re)xmit one segment
       3. Ack of (re)xmit sent in srep 2 arrives
       4. cwnd = ssthresh = FlightSize/2; send N=cwnd segments

For example, if FlightSize is 100 segments when timer expires,
congestion control is the same in steps 1-3, but in step 4 the 
current standard congestion control allows transmitting 2 segments, 
while RACK-TLP would allow blasting 50 segments.

Question is: what is the justification to modify standard TCP 
congestion control to use fast recovery instead of slow start for a 
case where timeout is needed to detect loss because there is no 
feedback and ack clock is lost? The draft does not give any 
justification. This clearly is in conflict with items (0) and (1) 
in BCP 133 (RFC 5033).

Furthermore, there is no implementation nor experimental experience 
evaluating this change. The implementation with experimental experience 
uses PRR (RFC 6937) which is an Experimental specification including a 
novel "trick" that directs PRR fast recovery to effectively use slow 
start in this case at hand.


> In other words, I am not seeing a case that requires new congestion
> control concepts except as discussed in 9.3.

See above. The change in standard congestion control for (2).
The draft intends not to change congestion control but effectively it 
does without any operational evidence.

What's also is missing and would be very useful:

- For (1), a hint for an implementer saying that because RACK-TLP is
   able to detect a loss of a rexmit unlike any other loss detection
   algorithm, the sender MUST react twice to congestion (and cite
   RFC 5681). And cite a document where necessary correct actions
   are described.

- For (1), advise that an implementer needs to keep track when it
   detects a loss of a retransmitted segment. Current algorithms
   in the draft detect a loss of retransmitted segment exactly in
   the same way as loss of any other segment. There seems to be
   nothing to track when a retransmission of a retransmitted segment
   takes place. Therefore, the algorithms should have additional
   actions to correctly track when such a loss is detected.

- For (1), discussion on how many times a loss of a retransmission
   of the same segment may occur and be detected. Seems that it
   may be possible to drop a rexmitted segment more than once and
   detect it also several times?  What are the implications?

- If previous is possible, then the algorithm possibly also
   may detect a loss of a new segment that was sent during fast
   recovery? This is also loss in two successive windows of data,
   and cwnd MUST be lowered twice. This discussion and necessary
   actions to track it are missing, if such scenario is possible.

> What am I missing?

Hope the above helps.

/Markku

> On Tue, Dec 15, 2020 at 10:57 AM Markku Kojo <[e-mail]>
> wrote:
>       Hi Neal, all,
>
>       (my apologies for this message getting so long but I didn't
>       want to drop
>       any parts of it in case someone wants to get the entire
>       discussion in one
>       piece. Also, apologies for repeating some of my concerns
>       inline below;
>       Those who are clear with the problems may well opt skipping
>       those parts.)
>
>       Thanks Neal for the reply. I believe we are slowly
>       converging with many
>       of my comments but two of the important issues still remain
>       open. That is,
>       where can an implementer find advice for correct congestion
>       control
>       actions with RACK-TLP, when:
>       (1) a loss of rexmitted segment is detected
>       (2) an entire flight of data gets dropped (and detected),
>            that is, when there is no feedback available and a
>       timeout
>            is needed to detect the loss
>
>       I fully understand the intent of the RACK-TLP draft not to
>       include
>       congestion control actions in this document (or the intent
>       to minimise
>       such advise). However, when new ways to detect loss are
>       introduced and
>       published (as this document would do), IMHO it is
>       imperative to have such
>       congestion control actions readily available for an
>       implementer. AFAIK
>       we do not have in the RFC series correct actions available
>       for these two
>       cases.
>
>       I do not have a strong opinion whether the necessary
>       actions should be in
>       this document or in another document, as long as we have
>       them. This is
>       more or less a process decision, I think.
>
>       Please see more details inline.
>
>       I also investigated a bit more on RACK-TLP as specified in
>       the draft and
>       it seems that there is a problem with the pseudocode as
>       currently
>       presented (some sequence numbers seem to be off by one,
>       resulting in
>       wrong outcome in some/many? cases). Please see inline in
>       the end of my
>       reply.
> 
>
>         On Sat, 12 Dec 2020, Neal Cardwell wrote:
>
>       > Hi Markku,
>       >
>       > Thanks for your many detailed and thoughtful comments.
>       Please see
>       > below in-line for our comments based on discussions among
>       RACK-TLP
>       > draft authors.
>       >
>       > On Tue, Dec 8, 2020 at 11:21 AM Markku Kojo
>       <[e-mail]> wrote:
>       >>
>       >> Hi,
>       >>
>       >> please see inline.
>       >>
>       >> On Mon, 7 Dec 2020, Yuchung Cheng wrote:
>       >>
>       >>> On Mon, Dec 7, 2020 at 8:06 AM Markku Kojo
>       <[e-mail]> wrote:
>       >>>>
>       >>>> Hi Yuchung,
>       >>>>
>       >>>> thanks for your reply. My major point is that IMHO
>       this RACK-TLP
>       >>>> specification should give the necessary advice w.r.t
>       congestion control
>       >>>> in cases when such advice is not available in the RFC
>       series or is not
>       >>>> that easy to interpreted correctly from the existing
>       RFCs.
>       >>>>
>       >>>> Even though the CC principles were available in RFC
>       series I believe you
>       >>>> agree with me that getting detailed CC actions correct
>       is sometimes quite
>       >>>> hard, especially if one needs to interpret and apply
>       them from another
>       >>>> spec than what one is implementing. Please see more
>       details inline below.
>       >>>>
>       >>>> In addition, we need to remember that this document is
>       not meant only for
>       >>>> TCP experts following the tcpm list and having deep
>       understanding of
>       >>>> congestion control but also those implementing TCP,
>       for the very first
>       >>>> time, for various appliances, for example. They do not
>       have first hand
>       >>>> experience in implementing TCP congestion control and
>       deserve clear
>       >>>> advice what to do.
>       >>>>
>       >>>> On Fri, 4 Dec 2020, Yuchung Cheng wrote:
>       >>>>
>       >>>>> On Fri, Dec 4, 2020 at 5:02 AM Markku Kojo
>       <[e-mail]> wrote:
>       >>>>>>
>       >>>>>> Hi all,
>       >>>>>>
>       >>>>>> I know this is a bit late but I didn't have time
>       earlier to take look at
>       >>>>>> this draft.
>       >>>>>>
>       >>>>>> Given that this RFC to be is standards track and
>       RECOMMENDED to replace
>       >>>>>> current DupAck-based loss detection, it is important
>       that the spec is
>       >>>>>> clear on its advice to those implementing it.
>       Current text seems to
>       >>>>>> lack important advice w.r.t congestion control, and
>       even though
>       >>>>>> the spec tries to decouple loss detection from
>       congestion control
>       >>>>>> and does not intend to modify existing standard
>       congestion control
>       >>>>>> some of the examples advice incorrect congestion
>       control actions.
>       >>>>>> Therefore, I think it is worth to correct the
>       mistakes and take
>       >>>>>> yet another look at a few implications of this
>       specification.
>       >>>>> As you noted, the intention is to decouple the two as
>       much as possible
>       >>>>>
>       >>>>> Unlike the 20 years ago where TCP loss detection and
>       congestion
>       >>>>> control are essentially glued in one piece, the
>       decoupling of the two
>       >>>>> (including modularizing congestion controls in
>       implementations) has
>       >>>>> helped fueled many great inventions of new congestion
>       controls.
>       >>>>> Codifying so-called-default C.C. reactions in the
>       loss detection is a
>       >>>>> step backward that the authors try their best to
>       avoid.
>       >>>>
>       >>>> While I fully agree with the general principle of
>       decoupling loss
>       >>>> detection from congestion control when it is possible
>       without leaving
>       >>>> open questions, I find it hard to get congestion
>       control right with this
>       >>>> spec in certain cases I raised just by following the
>       current standards
>       >>>> track CC specifications. The reason for this is that
>       RACK-TLP introduces
>       >>>> new ways to detect loss (i.e., not present in any
>       earlier standard track
>       >>>> RFC) and the current CC specifications do not provide
>       correct CC actions
>       >>>> for such cases as I try to point out below.
>       >>>>
>       >>>>> To keep the
>       >>>>> document less "abstract / unclear" as many WGLC
>       reviewers commented,
>       >>>>> we use examples to illustrate that includes CC
>       actions. But the
>       >>>>> details of these CC actions are likely to become
>       obsolete as CC
>       >>>>> continues to advance hopefully.
>       >>>>
>       >>>> Agreed. But I would appreciate if the CC actions in
>       the examples would
>       >>>> correctly follow what is specified in the the current
>       CC RFCs. And, I
>       >>>> would suggest explicitly citing the RFC(s) that each
>       of the examples is
>       >>>> illustriating so that there is no doubt which CC
>       variant the example
>       >>>> is valid with. Then there is no problem with the
>       correctness of the
>       >>>> example either even if the cited RFC becomes later
>       obsoleted.
>       >>>>
>       >>>>>>
>       >>>>>> Sec. 3.4 (and elsewhere when discussing recovering a
>       dropped
>       >>>>>> retransmission):
>       >>>>>>
>       >>>>>> It is very useful that RACK-TLP allows for
>       recovering dropped rexmits.
>       >>>>>> However, it seems that the spec ignores the fact
>       that loss of a
>       >>>>>> retransmission is a loss in a successive window that
>       requires reacting
>       >>>>>> to congestion twice as per RFC 5681. This advice
>       must be included in
>       >>>>>> the specification because with RACK-TLP recovery of
>       dropped rexmit
>       >>>>>> takes place during the fast recovery which is very
>       different
>       >>>>>> from the other standard algorithms and therefore
>       easy to miss
>       >>>>>> when implementing this spec.
>       >>>>>
>       >>>>> per RFC5681 sec 4.3
>       https://tools.ietf.org/html/rfc5681#section-4.3
>       >>>>> "Loss in two successive windows of data, or the loss
>       of a
>       >>>>>  retransmission, should be taken as two indications
>       of congestion and,
>       >>>>>   therefore, cwnd (and ssthresh) MUST be lowered
>       twice in this case."
>       >>>>>
>       >>>>> RACK-TLP is a loss detection algorithm. RFC5681 is
>       crystal clear on
>       >>>>> this so I am not sure what clause you suggest to add
>       to RACK-TLP.
>       >>>>
>       >>>> Right, this is the CC *principle* in RFC 5681 I am
>       refering to but I am
>       >>>> afraid it is not enough to help one to correctly
>       implement such lowering
>       >>>> of cwnd (and ssthresh) twice when a loss of a
>       retransmission is detected
>       >>>> during Fast Recovery. Nor do RFCs clearly advice
>       *when* this reduction
>       >>>> must take place.
>       >>>>
>       >>>> Congestion control principles tell us that congestion
>       must be reacted
>       >>>> immediately when detected. But at the same time,
>       standards track CC
>       >>>> specifications react to congestion only once during
>       Fast Recovery
>       >>>> because the losses in the current window, which Fast
>       Recovery repairs,
>       >>>> occured during the same RTT. That is, the current CC
>       specifications do
>       >>>> not handle lost rexmits during Fast Recovery but,
>       instead, the correct CC
>       >>>> reaction to a loss of a rexmit is automatically
>       achieved by those
>       >>>> specifications via RTO when cwnd is explicitly reset
>       upon RTO.
>       >>>>
>       >>>> Furthermore, the problem I am trying to point out is
>       that there is no
>       >>>> correct rule/formula available in the standards track
>       RFCs that would
>       >>>> give the correct way to reduce cwnd when the loss of
>       rexmit is detected
>       >>>> with RACK-TLP.
>       >>>>
>       >>>> I suggest everyone reading this message stops reading
>       at this point
>       >>>> for a while before continuing reading and figures out
>       themselves what
>       >>>> they think would be the correct equation to use in the
>       standards RFC
>       >>>> series to find the new halved value for cwnd when
>       RACK-TLP detects a loss
>       >>>> of a rexmit during Fast Recovery. I would appreciate a
>       comment on the
>       >>>> tcpm list from those who think they found the correct
>       answer immediately
>       >>>> and easily, and what was the formula to use.
>       >>>>
>       >>>> ...
>       >>>>
>       >>>> I think the best advice one may find by looking at RFC
>       6675 (and RFC
>       >>>> 5681) is to set
>       >>>>
>       >>>>   ssthresh = cwnd = (FlightSize / 2) (RFC 6675, Sec 5,
>       algorithm step 4.2)
>       >>>>
>       >>>> Now, let's modify the example in Sec 3.4 of the draft:
>       >>>>
>       >>>> 1. Send P0, P1, P2, ..., P20
>       >>>>    [Assume P1, ..., P20 dropped by network]
>       >>>>
>       >>>> 2.   Receive P0, ACK P0
>       >>>> 3a.  2RTTs after (2), TLP timer fires
>       >>>> 3b.  TLP: retransmits P20
>       >>>> ...
>       >>>> 5a.  Receive SACK for P20
>       >>>> 5b.  RACK: marks P1, ..., P20 lost
>       >>>>       set cwnd=ssthresh=FlightSize/2=10
>       >>>> 5c.  Retransmit P1, P2 (and some more depending on how
>       CC implemented)
>       >>>>       [P1 retransmission dropped by network]
>       >>>>
>       >>>>       Receive SACK P2 & P3
>       >>>> 7a.  RACK: marks P1 retransmission lost
>       >>>>       As per RFC 6675: set
>       cwnd=ssthresh=FlightSize/2=20/2=10
>       >>>> 7b.  Retransmit P1
>       >>>>       ...
>       >>>>
>       >>>> So, is the new value of the cwnd (10MSS) correct and
>       halved twice? If not,
>       >>>> where is the correct formula to do it?
>       >>>>
>       >>>> Before RFC 5681 was published we had a discussion on
>       FlighSize and that
>       >>>> during Fast Recovery it does not reflect the amount of
>       segments in
>       >>>> flight correctly but may also have a too large value.
>       It was decided not
>       >>>> to try to correct it because it only has an impact
>       when RTO fires during
>       >>>> Fast Recovery and in such a case cwnd is reset to 1
>       MSS. Having too large
>       >>>> ssthresh for RTO recovery in some cases was not
>       considered that bad
>       >>>> because a TCP sender anyway takes to most conservative
>       CC action with the
>       >>>> cwnd and would slow start from cwnd = 1 MSS. But now
>       when RACK-TLP enables
>       >>>> detecting loss of a rexmit during Fast Recovery we
>       have an unresolved
>       >>>> problem.
>       >>>
>       >>>> 1. Send P0, P1, P2, ..., P20
>       >>>>    [Assume P1, ..., P20 dropped by network]
>       >>>>
>       >>>> 2.   Receive P0, ACK P0
>       >>>> 3a.  2RTTs after (2), TLP timer fires
>       >>>> 3b.  TLP: retransmits P20
>       >>>> ...
>       >>>> 5a.  Receive SACK for P20
>       >>>> 5b.  RACK: marks P1, ..., P20 lost
>       >>>>       set cwnd=ssthresh=FlightSize/2=10
>       >>>> 5c.  Retransmit P1, P2 (and some more depending on how
>       CC implemented)
>       >>>>       [P1 retransmission dropped by network]
>       >>>>
>       >>>>       Receive SACK P2 & P3
>       >>>> 7a.  RACK: marks P1 retransmission lost
>       >>>>       As per RFC 6675: set
>       cwnd=ssthresh=FlightSize/2=20/2=10
>       >>>> 7b.  Retransmit P1
>       >>>
>       >>> To account for your points, IMO clearly stating the
>       existing CC RFC
>       >>> interactions w/o mandating any C.C. actions are the
>       best way to move
>       >>> forward. Here are the text diff I proposed:
>       >>>
>       >>> "Figure 1, above, illustrates  ...
>       >>> Notice a subtle interaction with existing congestion
>       control actions
>       >>> on event 7a. It essentially starts another new episode
>       of congestion
>       >>> due to the detection of lost retransmission. Per
>       RFC5681 (section 4.3)
>       >>> that loss in two successive windows of data, or the
>       loss of a
>       >>> retransmission, should be taken as two indications of
>       congestion as a
>       >>> principle. But RFC6675 that introduces the pipe concept
>       does not
>       >>> specify such a second window reduction. This document
>       reminds RACK-TLP
>       >>
>       >> This is not quite correct characterization of RFC 6675.
>       >> RFC 6675 does not repeat all guidelines of RFC 5681. RFC
>       6675 DOES specify
>       >> the two indications of congestion implicitly by clearly
>       stating in the
>       >> intro that it follows the guidelines set in RFC  5681.
>       >> And RFC 5681 articulates a set of MUSTs which ALL
>       alternative loss
>       >> recovery algorithms MUST follow in order to become RFCs.
>       >>
>       >> (*) RFC 5681, Sec 4.3 (pp. 12-13):
>       >>
>       >>   "While this document does not standardize any of the
>       >>    specific algorithms that may improve fast
>       retransmit/fast recovery,
>       >>    ...
>       >>    That is, ...
>       >>    Loss in two successive windows of data, or the loss
>       of a
>       >>    retransmission, should be taken as two indications of
>       congestion and,
>       >>    therefore, cwnd (and ssthresh) MUST be lowered twice
>       in this case."
>       >>
>       >>
>       >>> implementation to carefully consider the new multiple
>       congestion
>       >>> episode cases in the corresponding congestion control."
>       >>
>       >> I am sorry to say that this is very fuzzy and leaves an
>       implementer
>       >> all alone what to do.
>       >>
>       >> More inportantly, AFAIK we have no discussion nor
>       consensus in the IETF
>       >> on what is the correct CC action when lost rexmit is
>       detected.
>       >> Should one reset cwnd like current CCs do, or would
>       "halving again" be
>       >> fine, or something else? IMHO this requires
>       experimentation to decide.
>       >>
>       >> I assumed there is an implementation of RACK-TLP with
>       corresponding CC
>       >> actions and experimental results to devise what are the
>       implications of
>       >> the selected CC action(s) with various levels of
>       congestion. But it seems
>       >> we do not have such implementation nor experimental
>       evidence? Am I wrong?
>       >
>       > [RACK-TLP-team:]
>       >
>       > We do have such implementations of RACK-TLP with
>       corresponding CC
>       > actions, and experimental results were reported in
>       RACK-TLP
>       > presentations and earlier revisions of the RACK-TLP
>       draft. With
>
>       Yes, that was my understanding what is implemented. Given
>       there is this
>       implementation that reacts appropriately to congestion when
>       RACK-TLP
>       detects loss of a retransmitted segment, could you please
>       illuminate the
>       congestion control actions taken in the implementation with
>       which there
>       is experimental experience.
>
>       If I understood it correctly, the RACK-TLP in question is
>       implemented
>       with PRR. Let's assume another implementer wants to add
>       RACK-TLP with PRR
>       in the stack she/he is working on. I believe that the
>       implementer would
>       think that detecting a loss of a rexmit during fast
>       recovery would start
>       a new congestion episode in PRR (and that's what Yuchung
>       also indicated
>       in the new proposed text in his previeous reply shown
>       above). With
>       that advice and reading RFC 6937, an implementer would
>       reinitialize PRR
>       state for the new congestion episode. So how should the
>       implementer
>       calculate new values for ssthresh and RecoverFS? Should
>       she/he use what
>       is currently described in RFC 6937, Sec 3 (to keep
>       discussion simple
>       assume the implementer wants to have an IETF Standards
>       Track comformant
>       implementation, so use CongCtrlAlg() = FlightSize/2):
>
>         ssthresh = CongCtrlAlg()  // Target cwnd after recovery
>         prr_delivered = 0         // Total bytes delivered during
>       recovery
>         prr_out = 0               // Total bytes sent during
>       recovery
>         RecoverFS = snd.nxt-snd.una // FlightSize at the start of
>       recovery
>
>       Or should one use something else? AFAIK depending on the
>       loss pattern
>       neither FlightSize nor "snd.nxt-snd.una" would give correct
>       values for
>       the intended behavior. Depending on the loss pattern, at
>       the time
>       of detecting a loss of rexmit FlighSize may be ~ 50% larger
>       than in the
>       beginning of fast recovery. This would result in increasing
>       ssthresh
>       (send rate), instead of decreasing it. It seems that
>       something else in
>       PRR algo would also need to be modified to get sndcnt
>       correct but I
>       cannot figure it out right now.
>
>       If one wants to implement RACK-TLP without PRR, that is,
>       with RFC 6675,
>       we have the same probelm with FlightSize in determining a
>       new value for
>       cwnd and ssthresh.
>
>       My point is that using what we currently have out there in
>       the RFC
>       series gives incorrect CC advice. And, having correct CC
>       actions in this
>       case is far from trivial so IMHO high quality standards
>       specifications
>       should not leave an implementer of RACK-TLP alone to figure
>       it out.
>
>       In addition, if a new episode of fast recovery is started
>       for CC purposes
>       how would that play together with RACK-TLP loss detection?
>       In Sec 7.1 the
>       text says initialize (reset) TLP.end_seq and TLP.is_retrans
>       when
>       initiating fast recovery. But detecting a lost rexmit
>       seemingly should
>       not reinitialize TLP because PTO is disabled during fast
>       recovery as per
>       Sec 7.2. If CC actions are in another document and loss
>       detection in this
>       document, it would be of high importance to clarify when
>       each of the
>       fast recovery episodies (CC vs. loss detection) are
>       supposed to end: is
>       the end condition the same or different?
>
>       > respect to implementations, there is the Linux TCP stack,
>       which has
>       > been using RACK-TLP as the default loss recovery
>       algorithm since Linux
>       > v4.18 in August 2018. The exact commit is:
>       >
>       >  b38a51fec1c1 tcp: disable RFC6675 loss detection
>       >
>       > With Linux, the default sender behavior since that commit
>       has been
>       > governed by CUBIC, PRR, and RACK-TLP.
>       >
>       > RACK-TLP is used by all Google TCP traffic, including
>       internal RPC
>       > traffic and external YouTube and google.com traffic over
>       the public
>       > Internet. At Google we have years of high-traffic-volume
>       experience
>       > with RACK-TLP with three different CC schemes: CUBIC+PRR,
>       BBRv1, and
>       > BBRv2.
>       >
>       > My understanding is that Netflix FreeBSD TCP traffic uses
>       RACK-TLP,
>       > and Microsoft Windows TCP traffic has also been using
>       RACK-TLP.
>
>       And I wonder what would be the CC actions taken in each of
>       these
>       implementations when a loss of rexmited segment is
>       detected?
>
>       >> I sincerely apologize that this got raised this late in
>       the prosess and
>       >> I know how irritating it may be. I like the idea of
>       RACK-TLP and by no
>       >> means my intention is not to hold up the process but the
>       lack of evidence
>       >> makes me quite concerned. In particular, when this
>       document says that
>       >> the current loss detection SHOULD be replaced in all
>       implementations with
>       >> RACK-TLP.
>       >>
>       >>> and to emphasize in section 9.3 Interaction with
>       congestion control
>       >>>
>       >>> "9.3.  Interaction with congestion control
>       >>>
>       >>> RACK-TLP intentionally decouples loss detection ...
>       this appropriate.
>       >>> As mentioned in Figure 1 caption, RFC5681 mandates a
>       principle that
>       >>> Loss in two successive windows of data, or the loss of
>       a
>       >>> retransmission, should be taken as two indications of
>       congestion, and
>       >>> therefore reacted separately. However implementation of
>       RFC6675 pipe
>       >>> algorithm may not directly account for this newly
>       detected congestion
>       >>> events properly. Therefore the documents reminds
>       RACK-TLP
>       >>> implementation to carefully consider these implications
>       in its
>       >>> corresponding congestion control.
>       >>>
>       >>> ..."
>       >>>
>       >>>
>       >>>
>       >>>>
>       >>>>
>       >>>>>> Sec 9.3:
>       >>>>>>
>       >>>>>> In Section 9.3 it is stated that the only
>       modification to the existing
>       >>>>>> congestion control algorithms is that one
>       outstanding loss probe
>       >>>>>> can be sent even if the congestion window is fully
>       used. This is
>       >>>>>> fine, but the spec lacks the advice that if a new
>       data segment is sent
>       >>>>>> this extra segment MUST NOT be included when
>       calculating the new value
>       >>>>>> of ssthresh as per the equation (4) of RFC 5681.
>       Such segment is an
>       >>>>>> extra segment not allowed by cwnd, so it must be
>       excluded from
>       >>>>>> FlightSize, if the TLP probe detects loss or if
>       there is no ack
>       >>>>>> and RTO is needed to trigger loss recovery.
>       >>>>>
>       >>>>> Why exclude TLP (or any data) from FlightSize? The
>       congestion control
>       >>>>> needs precise accounting of the flight size to react
>       to congestion
>       >>>>> properly.
>       >>>>
>       >>>> Because FlightSize does not always reflect the correct
>       amount of data
>       >>>> allowed by cwnd. When a TCP sender is not already in
>       loss recovery and
>       >>>> it detects loss, this loss indicates the congestion
>       point for the TCP
>       >>>> sender, i.e., how much data it can have outstanding.
>       It is this amount of
>       >>>> data that it must use in calculating the new value of
>       cwnd (and
>       >>>> ssthresh), so it must not include any data sent beyond
>       the congestion
>       >>>> point. When TLP sends a new data segment, it is beyond
>       the congestion
>       >>>> point and must not be included. Same holds for the
>       segments sent via
>       >>>> Limited Transmit: they are allowed to be send out by
>       the packet
>       >>>> conservation rule (DupAck indicates a pkt has left the
>       network, but does
>       >>>> not allow increasing cwnd), i.e., the actual amount of
>       data in flight
>       >>>> remains the same.
>       >>>>
>       >>>>>> In these cases the temporary over-commit is not
>       accounted for as DupAck
>       >>>>>> does not decrease FlightSize and in case of an RTO
>       the next ACK comes too
>       >>>>>> late. This is similar to the rule in RFC 5681 and
>       RFC 6675 that prohibits
>       >>>>>> including the segments transmitted via Limitid
>       Transmit in the
>       >>>>>> calculation of ssthresh.
>       >>>>>>
>       >>>>>> In Section 9.3 a few example scenarios are used to
>       illustriate the
>       >>>>>> intended operation of RACK-TLP.
>       >>>>>>
>       >>>>>>   In the first example a sender has a congestion
>       window (cwnd) of 20
>       >>>>>>   segments on a SACK-enabled connection.  It sends
>       10 data segments
>       >>>>>>   and all of them are lost.
>       >>>>>>
>       >>>>>> The text claims that without RACK-TLP the ending
>       cwnd would be 4 segments
>       >>>>>> due to congestion window validation. This is
>       incorrect.
>       >>>>>> As per RFC 7661 the sender MUST exit the
>       non-validated phase upon an
>       >>>>>> RTO. Therefore the ending cwnd would be 5 segments
>       (or 5 1/2 segments if
>       >>>>>> the TCP sender uses the equation (4) of RFC 5681).
>       >>>>>>
>       >>>>>> The operation with RACK-TLP would inevitably result
>       in congestion
>       >>>>>> collapse if RACK-TLP behaves as described in the
>       example because
>       >>>>>> it restores the previous cwnd of 10 segments after
>       the fast recovery
>       >>>>>> and would not react to congestion at all! I think
>       this is not the
>       >>>>>> intended behavior by this spec but a mistake in the
>       example.
>       >>>>>> The ssthresh calculated in the beginning of loss
>       recovery should
>       >>>>>> be 5 segments as per RFC 6675 (and RFC 5681).
>       >>>>> To clarify, would this text look more clear?
>       >>>>>
>       >>>>> 'an ending cwnd set to the slow start threshold of 5
>       segments (half of
>       >>>>> the original congestion window of 10 segments)'
>       >>>>
>       >>>> This is correct, but please replace:
>       >>>>
>       >>>>   (half of the original congestion window of 10
>       segments)
>       >>>> -->
>       >>>>   (half of> the original FlightSize of 10 segments)
>       >>>
>       >>> sure will do
>       >>>
>       >>>>
>       >>>> cwnd in the example was 20 segments.
>       >>>>
>       >>>> Please also correct the ending cwnd for "without RACK"
>       scenario.
>       >>>> I poined out wrong equation number IN RFC 5681 and
>       INCORRECT cwnd value,
>       >>>> my apologies. It MAENT equation (3) AND that results
>       in ending cwnd of 5
>       >>>> and 2/5 MSS (not 5 and 1/2 MSS).
>       >>>> NB: and if a TCP sender implements entering CA when
>       cwnd > ssthresh, then
>       >>>> ending cwnd would be 6 and 1/6 MSS).
>       >>>>
>       >>>>>>
>       >>>>>> Furthermore, it seems that this example with
>       RACK-TLP refers to using
>       >>>>>> PRR_SSRB which effectively implements regular slow
>       start in this
>       >>>>>> case(?). From congestion control point of view this
>       is correct because
>       >>>>>> the entire flight of data as well as ack clock was
>       lost.
>       >>>>>>
>       >>>>>> However, as correctly discussed in Sec 2, congestion
>       window must be reset
>       >>>>>> to 1 MSS when an entire flight of data is and Ack
>       clock is lost. But how
>       >>>>>> can an implementor know what to do if she/he is not
>       implementing the
>       >>>>>> experimental PRR algrorithm? This spec articulates
>       specifying an
>       >>>>>> alternative for DupAck counting, indicating that TLP
>       is used to trigger
>       >>>>>> Fast Retransmit & Fast Recovery only, not a loss
>       recovery in slow start.
>       >>>>>> This means that without an additional advise an
>       implementation of this
>       >>>>>> spec would just halve the cwnd and ssthresh and send
>       a potentially very
>       >>>>>> large burst of segments in the beginning of the Fast
>       Recovery because
>       >>>>>> there is no ack clock. So, this spec begs for an
>       advise (MUST) when to
>       >>>>>> slow start and reset cwnd and when not, or at least
>       a discussion of
>       >>>>>> this problem and some sort of advise what to do and
>       what to avoid.
>       >>>>>> And, maybe a recommendation to implement it with
>       PRR?
>       >>>>>
>       >>>>> It's wise to decouple loss detection (RACK-TLP) vs
>       congestion/burst
>       >>>>> control (when to slow-start). The use of PRR is just
>       an example to
>       >>>>> illustrate and not meant for a recommendation.
>       >>>>
>       >>>> I understand the use of PRR was just an example, but
>       my point is that if
>       >>>> one wants to implement RACK-TLP but does not intend to
>       implement PRR but
>       >>>> RFC 6675 then we do not have a rule in RFC 6675 to
>       correctly implement
>       >>>> CC for the case when an entire flight is lost and loss
>       is detected with
>       >>>> TLP. Congestion control principle for this is clear
>       and also stated in
>       >>>> this draft but IMHO it is not enough to ensure correct
>       implementation.
>       >>>>
>       >>>> To my understanding we only have implementation
>       experience for RACK-TLP
>       >>>> only togerher with PRR, which has the necessary rule
>       to handle this kind
>       >>>> of scenario correctly.
>       >>>>
>       >>>> So, my question is how can one implement CC correctly
>       without PRR such
>       >>>> a scenario where entire inflight is lost?
>       >>>> Which rule and where in the RFC series gives the
>       necessary guidance to
>       >>>> reset cwnd and slow start when TCP detets loss of an
>       entire flight?
>       >>>
>       >>> I think we're going in loops. To move forward it'd help
>       if you suggest
>       >>> some text you like to change.
>       >>
>       >> Unfortunately I do not have an immediate solution. I
>       assumed there is an
>       >> implementation and you could enlighten what was the
>       solution there and
>       >> experimental results showing how it seems to work. If I
>       have understood
>       >> it correctly there is a solution/implementation and
>       experience of RACK-TLP
>       >> that works together with PRR but no
>       solution/implementation nor experience
>       >> how it works without PRR. Am I correct?
>       >
>       > [RACK-TLP-team:]
>       >
>       > The implementation solutions and experiments were
>       mentioned above. Our
>       > team's experience is with RACK-TLP in combination with
>       either
>       > CUBIC+PRR, BBRv1, and BBRv2. We are not sure if there are
>       other TCP
>       > implementations that implement RACK-TLP without PRR.
>
>       So, in addition to the lack of experimental evidence using
>       RACK-TLP
>       without PRR we have a specification problem for which we
>       have no answer:
>
>       The RACK-TLP text suggests that if PTO fires and TLP
>       detects a loss, the
>       TCP sender enters fast recovery. But
>
>       PTO = timeout that expires if there is no feedback (ACKs)
>       from
>              the network, i.e., entire flight of data is lost.
>
>       TLP = a single segment that is sent when timer expires.
>       What is
>              different compared to RTO is that PTO may expire
>       faster
>              and a different segment is (re)transmitted, allowing
>       to
>              detect more than one lost segment with a single Ack
>       w/ SACK
>              triggered by TLP (with regular RTO recovery only the
>       next
>              unacknowledged segment can reliably be considered
>       lost)
>
>       >From congestion control point of view this means that PTO
>       is no different
>       from RTO. The congestion control principle as stated in Sec
>       2.2 of this
>       document tells us that a loss of entire flight deserves a
>       cautious
>       response, i.e., reset cwnd and slow start.
>
>       When RACK-TLP is implemented with PRR we have appropriate
>       CC actions
>       specified in RFC 6937 that effectively reset cwnd and force
>       the TCP sender
>       to slow start, even though it tecnically enters fast
>       recovery. This is
>       safe.
>
>       However, if an implementer wants to implement RACK-TLP
>       without PRR,
>       she/he has to consult RFC 6675 which gives an incorrect CC
>       response for
>       this case, resulting in an unsafe implementation. The
>       problem is not in
>       the fast recovery algorithm in RFC 6675 because it was not
>       intended to be
>       initiated with TLP-kind of loss detection.
>
>       So the question is how to ensure an implementer has the
>       necessary advice
>       to implement correct congestion control actions for
>       RACK-TLP once this
>       document is published. The lack of correct advice becomes
>       even more
>       prominent as this document requires (all SACK capable) TCPs
>       to implement
>       RACK-TLP.
>
>       >> Given my understanding of PRR, the problem of an entire
>       flight being
>       >> dropped is quite nicely solved with it. However,
>       implementing RACK-TLP
>       >> without PRR begs for a solution. Here is my current
>       understanding:
>       >>
>       >> (1) RACK part is what can be called to "replace" DupAck
>       counting
>       >> (2) TLP part is effectively a new retransmission timeout
>       to detect
>       >>      a loss of an entire flight of data (i.e., it is
>       only invoked
>       >>      when a tail of the current flight becomes lost
>       which equals to
>       >>      the case of losing an entire flight since all
>       segments before
>       >>      the tail loss will get cumulatively Acked and hence
>       these segments
>       >>      are not anymore a part of the current flight at the
>       time the loss
>       >>      is detected. And we an assume application limited
>       TCP sender.
>       >> (3) Current CC principles require resetting cwnd in such
>       a case
>       >>      and entering slow start (and so effectively does
>       PRR though it is
>       >>      not explicitly stated in RFC 6937). Slow start
>       avoids a big burst
>       >>      if the lost flight is big.
>       >> (4) RACK-TLP would possibly like to allow that cwnd is
>       set to a half
>       >>      of the lost flight and not to slow start. This
>       means that the bigger
>       >>      the lost flight is, the bigger is the burst that
>       gets transmitted at
>       >>      the beginning of the recovery which is bad. So,
>       this approach would
>       >>      need at least a rule/advice for burst avoidance
>       (slow start, pacing,
>       >>      ...).
>       >> (5) When the lost flight is small (<= 4 segments) there
>       is no difference
>       >>      in recovery efficiency between (3) and (4). If the
>       lost flight is
>       >>     > 5 segments, then (4) takes less RTTs to complete
>       the recovery
>       >>      but generates a burst. Note, even if pacing over
>       one SRTT would be
>       >>      used, it is still a burst.
>       >>
>       >> Now, it would be useful to have experimental data to
>       know how the size of
>       >> the lost flight is distributed. Is it typically just a
>       few segments as
>       >> illustriated in the examples of the draft or is it often
>       larger?
>       >>
>       >> My advice would be to add a rule that cwnd MUST be reset
>       to 1 MSS
>       >> and the sender MUST enter slow start if TLP detects loss
>       of an entire
>       >> flight. This would be safe. Otherwise, without
>       experimental evidence from
>       >> a wide range of different network conditions and
>       workloads it feels
>       >> unsafe to allow more aggressive approach.
>       >
>       > [RACK-TLP-team:]
>       >
>       > Our team's position is that the RACK-TLP draft documents
>       a loss
>       > detection algorithm, and normative declarations about
>       congestion
>       > control behavior (like your "cwnd MUST be reset to 1
>       MSS", above)
>       > should be in a separate document focused on congestion
>       control.
>
>       I do not disagree with you what becomes to the role of
>       congestion control
>       in the RACK-TLP draft. It may well be in another document
>       but IMHO such
>       document must be available at the time this docment gets
>       published.
>       How can we otherwise ensure correct implementations?
>
>       So this is more of a process question than critique towards
>       the draft.
> 
>
>       >>>>> Section 3 has a lengthy section to elaborate the key
>       point of RACK-TLP
>       >>>>> is to maximize the chance of fast recovery. How C.C.
>       governs the
>       >>>>> transmission dynamics after losses are detected are
>       out of scope of
>       >>>>> this document in our authors' opinions.
>       >>>>>
>       >>>>>
>       >>>>>>
>       >>>>>> Another question relates to the use of TLP and
>       adjusting timer(s) upon
>       >>>>>> timeout. In the same example discussed above, it is
>       clear that PTO
>       >>>>>> that fires TLP is just a more aggressive retransmit
>       timer with
>       >>>>>> an alternative data segment to (re)transmit.
>       >>>>>>
>       >>>>>> Therefore, as per RFC 2914 (BCP 41), Sec 9.1, when
>       PTO expires, it is in
>       >>>>>> effect a retransmission timout and the timer(s) must
>       be backed-off.
>       >>>>>> This is not adviced in this specification. Whether
>       it is the TCP RTO
>       >>>>>> or PTO that should be backed-off is an open
>       question.  Otherwise,
>       >>>>>> if the congestion is persistent and further
>       transmission are also lost,
>       >>>>>> RACK-TLP would not react to congestion properly but
>       would keep
>       >>>>>> retransmitting with "constant" timer value because
>       new RTT estimate
>       >>>>>> cannot be obtained.
>       >>>>>> On a buffer bloated and heavily congested bottleneck
>       this would easily
>       >>>>>> result in sending at least one unnecessary
>       retransmission per one
>       >>>>>> delivered segment which is not advisable (e.g., when
>       there are a huge
>       >>>>>> number of applications sharing a constrained
>       bottleneck and these
>       >>>>>> applications are sending only one (or a few)
>       segments and then
>       >>>>>> waiting for an reply from the peer before sending
>       another request).
>       >>>>>
>       >>>>> Thanks for pointing to the RFC.  After TLP, RTO
>       timers will
>       >>>>> exp-backoff (as usual) for stability reasons
>       mentioned in sec 9.3
>       >>>>> (didn't find 9.1 relevant).
>       >>>>
>       >>>> My apologies for refering to the wrong section of RFC
>       2914, Yes, I meant
>       >>>> Sec 9.3.
>       >>>>
>       >>>>> In your scenario, you presuppose the
>       >>>>> retransmission is unnecessary so obviously TLP is not
>       good. Consider
>       >>>>> what happens without TLP where all the senders fire
>       RTO spuriously and
>       >>>>> blow up the network. It is equally unfortunate
>       behavior. "bdp
>       >>>>> insufficient of many flows" is a congestion control
>       problem
>       >>>>
>       >>>> If (without TLP) RTO is spurious, it may result in
>       unnecessary
>       >>>> retransmissions. But we have F-RTO (RFC 5682) and
>       Eifel (RFC 3522) to
>       >>>> detect and resolve it without TLP, so I don't find it
>       as a problem.
>       >>>>
>       >>>> To clarify more, what I am concerned about. Think
>       about a scenario where a
>       >>>> (narrow) bottleneck becomes heavily congested by a
>       huge number of
>       >>>> competing senders such that the available capacity per
>       sender is less
>       >>>> than 1 segment (or << 1 MSS).
>       >>>> This is a situation that network first enters before
>       congestion collapse
>       >>>> gets realized. So, it is extremely important that all
>       CC and timer
>       >>>> mechanisms handle it properly. Regular TCP handles it
>       via RFC 6298 by
>       >>>> backing off RTO expotentially and keeping this
>       backed-off RTO until an
>       >>>> new ACK is received for new data. This saves RACK-TLP
>       from full congestion
>       >>>> collapse. But consider what happens: even though RTO
>       is backed off, each
>       >>>> time a TCP sender manages to get one segment through
>       (with cwnd = 1 MSS)
>       >>>> it always first arms PTO with more or less constant
>       value of 2*SRTT. If
>       >>>> the bottleneck is buffer bloated the actual RTT easily
>       exceeds 2*SRTT and
>       >>>> TLP becomes spurious. After a spurious TLP, RTO
>       expires (maybe more than
>       >>>> once before exponential back-off of RTO results in
>       large enough value)
>       >>>> and a new RTT sample is not received. So, SRTT remains
>       unchanged and even
>       >>>> if sometimes a new sample is received, SRTT gets very
>       slowly adjusted. As
>       >>>> a result, each TCP sender would keep on sending a
>       spurious TLP for each
>       >>>> new segment resulting in at least 50% of the packets
>       being unnecessary
>       >>>> rexmitted and the utilization of the bottleneck is <
>       50%. This would not
>       >>>> be a full congestion collapse but has unwanted
>       symptoms towards
>       >>>> congestion collapse (Note: there is no clear line for
>       the level of
>       >>>> reduction in delivery of useful data is considered as
>       congestion
>       >>>> collapse).
>       >>>
>       >>> AFAIK you are saying: under extreme congestion shared
>       by many short
>       >>> flows, RACK-TLP can cause more packet losses because of
>       the more
>       >>> aggressive PTO timer. I agree and can add this to the
>       "section 9.3".
>       >>
>       >> No, that was not what I meant. This happens simply when
>       enough flows are
>       >> competing on the same bottleneck. It may be long flows
>       or maybe
>       >> applications with more or less continuous request-reply
>       exhanges.
>       >> E.g., if the bottleneck bit rate is 1 Mbits/s, RTT is
>       500msecs and PMTU
>       >> is 1500B, then a bit over 40 simultaneous flows would
>       mean that the
>       >> bottleneck becomes fully utilized with the equal share
>       of roughly 1 MSS
>       >> per flow. With a quite typical bottleneck buffer size
>       roughly equaling to
>       >> BDP, about 80+ flows would fill up the buffer as well
>       and increase
>       >> RTT >= 1 secs.  A buffer bloated bottleneck buffer would
>       mean even larger
>       >> RTT and allow more flows sharing the bottleneck without
>       loss.
>       >>
>       >> If the number of TCP flows is > 90 (or >> 90) RACK-TLP
>       would ends up
>       >> (almost) always unnecessarily retransmitting each new
>       segment once (PTO
>       >> being 1 sec or around 1 sec). RTO back-off after a few
>       rounds saves each
>       >> TCP sender from additional unnecessary rexmits but still
>       ~ 50% of the
>       >> deivered packets are not making any useful progress.
>       >>
>       >> If the bottleneck buffer is small, it would result in
>       more losses as you
>       >> suggest but would not be that much of problem because
>       majority of the
>       >> unnecessary rexmits would not get delivered over the
>       bottleneck link.
>       >> Instead, they wuold get dropped at the bottleneck.
>       Unnecessary rexmits
>       >> would just create extra load on the poth before the
>       bottleneck (which of
>       >> course is not a non-problem either).
>       >
>       > [RACK-TLP-team:]
>       >
>       > If we are correctly understanding the scenario you are
>       outlining, the
>       > PTO would likely not be 1 sec in this case. In this
>       scenario, the RTT
>       > is around 1sec, so the PTO would be around 2*srtt, which
>       would be
>       > around 2 secs. With a 1sec RTT, and even considering
>       delayed ACKs, it
>       > seems very unlikely that a 2 sec PTO would cause a
>       significant number
>       > of unnecessary TLP probes.
>
>       Well. Roughly the first initiated 40 flows will have SRTT ~
>       0.5 sec. If
>       next ~ 40 flows arrive within an RTT or so they will get
>       RTT samples
>       roughly between 0.5 - 1.x secs. The rest 80 - N hundreds
>       flows will not
>       get an RTT sample and will use initial/default PTO = 1 sec
>       (unless using
>       timestamps). The larger the buffer bloat the worse things
>       get.
>       Once ~200 flows are competing over the bottleneck, RTT will
>       be well
>       beyond 2 secs and grows when more flows join, guaranteeing
>       spurious TLP
>       for each new segment (unless timestamps are in use).
>
>       > Perhaps your postulated PTO of 1 sec is coming from the
>       pseudocode
>       > path in TLP_calc_PTO() where, if SRTT is not available,
>       the PTO is 1
>       > second? However, this case where there is no SRTT only
>       happens if (a)
>       > the connection is *not* using TCP timestamps (enabled for
>       most
>       > connections on the public Internet, due to
>       default-enabled support in
>       > Linux/Android and iOS/MacOS), and (b) the connection has
>       *never* had a
>       > single RTT sample from a non-retransmitted packet. To
>       meet both of
>       > those conditions is quite rare in practice. Is this the
>       path that is
>       > causing your concerns about a PTO of 1 sec?
>
>       Sure. Having timestamps helps a lot in case the bottleneck
>       buffer is
>       buffer bloated (though RTT sample will not include delayed
>       Ack effect for
>       rexmits that are immediately acked). But RACK-TLP does not
>       require
>       timestamps and there are a lot of TCP stacks out there in
>       addition to
>       Linux/Android and iOS/MacOS. And more are coming with IoT
>       expansion,
>       for example. In addition, it is quite suprising how old
>       (mobile) devices
>       many people are using, some of which are old enough and
>       have not been
>       eligible for an OS upgrade for many years.
>
>       But my quick example scenario was just to illustriate the
>       problem, maybe
>       typical in some areas where the only Internet access is
>       over a
>       geostationary satellite connection and people quite likely
>       not using
>       mobile/cellular devices. And, I selected a buffer-bloated
>       scenario
>       because that generates unnecessary rexmits which are a
>       prerequisite for
>       classical congestion collapse.
>
>       It may well be in some environments that these hundreds of
>       competing
>       flows are application limited (request-reply continuously)
>       and get
>       started over a longer period of time, each getting RTT
>       sample(s) over a
>       lightly loaded bottleneck and then becoming idle or sending
>       data only
>       sporadically. Then later they all become active more or
>       less at the same
>       time (maybe by a common trigger). Base RTT may then be much
>       lower than
>       500 ms as long as there is enough buffer bloat.
>
>       Furthermore, timestamps are not that helpful in case of a
>       heavily
>       congested small buffer bottleneck. Having approximately
>       accurate SRTT
>       does not alone help in reducing the sending rate below 1
>       segment/RTT in
>       heavily congested scenarios. Exponential backoff of timer
>       is a must.
>
>       I believe that either of us know what's really out there
>       employing
>       Internet for communication purposes. It does not matter
>       whether it is a
>       notable percentage or not as long there is a group of users
>       who are
>       encountering the envisioned problem.
>       It may very well be a small minority, but if this kind of a
>       problem
>       escalates for this group very often (maybe almost always),
>       it means we
>       are disabling use of Internet technology for them.
>
>       > Furthermore, in cases like this where the number of flows
>       is roughly
>       > equal to or greater than the BDP of the path,
>       Reno/[RFC5681] and CUBIC
>       > will spend the majority of their time in loss recovery
>       (the minimum
>       > ssthresh from [RFC5681] being 2*SMSS).
>
>       Sure there is surprisingly lot to improve with TCP when the
>       window needs
>       to be kept lower than 2 MSS or when the available share
>       would be even
>       less than 1 MSS/RTT. RFC 5681 would exponentially back-off
>       on every RTO,
>       leaving opportunity for other flows to make progress during
>       the
>       backed-off interval. Once the sender is lucky and gets an
>       Ack for a new
>       segment, min ssthresh of 2*MSS unfortunately allows it to
>       accelerate to
>       2*MSS/RTT and maybe more until RTO expires again. We cannot
>       avoid some
>       drops or unnecessary rexmits. Actually, in such a heavily
>       congested
>       scenario the combination of Karn's exponential back-off and
>       slow start
>       results in Multiplicative Increase Multiplicative Decrease
>       (MIMD)
>       behaviorwhich does not converge that well to fairness. But
>       it avoids
>       congestion collapse. It is kind a tradeoff between safe and
>       stable
>       behavior during heavy congestion and being able to ramp up
>       fast when the
>       congestion fades away. Even though the current behavior is
>       not ideal
>       for fairness, we should not make it even worse.
>
>       Currently only way to avoid spending most of the time in
>       loss
>       recovery seems to be ECN that would make a TCP sender to
>       become rate
>       controlled once cwnd hits the minimum cwnd of 1 MSS(*). But
>       there is a
>       lot to optimise also with ECN in this kind of scenario ...
>
>       (*) to avoid anyone repeating the misconception that
>       minimum cwnd is
>            2 MSS per equation (4) in RFC 5681: equation (4) sets
>       ssthresh,
>            not cwnd. RFC 3168 clearly states that cwnd must be
>       halved on
>            an arrival of ECE; this holds until cwnd = 1 MSS.
>       After this,
>            the sending rate of new segments is (RTO) timer
>       controlled and
>            the timer avlue exponentially backed-off if more ECEs
>       arrive.
>            At least two major OSs implement this incorrectly
>       unless now
>            already fixed.
>
>       > And the RACK-TLP ID specifies
>       > that TLP probes are not sent while in loss recovery.
>
>       Maybe I missed it but I couldn't find an end condition of
>       RTO recovery
>       for RACK-TLP purposes in the document. Regular RTO does not
>       have such end
>       condition because it is not needed. Congestion control and
>       loss recovery
>       are quite decoupled during  RTO recovery; slow start
>       follows its own
>       rules and ends when cwnd > ssthresh and loss recovery has
>       its own
>       rules to select segments to rexmit and just moves on
>       sending new data
>       once all lost segments have been retransmitted. Some
>       implementations may
>       have such variable/condition but it didn't help me.
>
>       Without timestamps, if TLP is not backed-off, a TLP is
>       likely to be sent
>       for each new segment (assuming certain conditions). Each
>       not backed-off
>       rexmit is potentially stealing an opportunity from another
>       flow to get
>       its segment being delivered, thereby contributing to unfair
>       use of
>       bottleneck resources.
>
>       >>> What authors disagree is that RTO must be back-off on
>       the first
>       >>> instance if TLP is not acked. While your suggestion
>       helps the
>       >>> congestion case, it may also hurt the recovery in other
>       cases when the
>       >>> TLP is dropped due to light/burst/transient congestion.
>       Arguing which
>       >>> scenarios matter more subjectively is not productive.
>       >>
>       >> When RACK-TLP is required to be implemented by all TCP
>       stacks, it
>       >> is extremely important that it always works in a safe
>       way and congestion
>       >> is always the primary concern to get properly handled.
>       Without arguing
>       >> more, I just want to point out that TCP must work
>       reasonably for all
>       >> Internet users. That means it must not generate a
>       situation where even a
>       >> small minority of the Internet users often or almost
>       always encounter
>       >> severe problems with their connectivity.
>       >>
>       >> For example, also users in developing countries where
>       possibly an entire
>       >> village shares just one mobile (cellular, maybe 3G
>       possibly only GPRS)
>       >> connection for their Internet access and pays per amount
>       of data should
>       >> get reasonable TCP behavior. In such a case, I cannot
>       agree with
>       >> engineering that results in almost always getting only
>       half of the
>       >> already scarce bandwidth whle paying double price fot
>       the useful data.
>       >>
>       >> But thinking about a way forward. Karn's algorithm would
>       require backing
>       >> off PTO even if you get Ack of PTO. Relaxing this does
>       not sound me that
>       >> bad at all, because there is often a pause before TLP
>       and a TCP sender
>       >> gets feedback, so apparently conditions are not that bad
>       and loss
>       >> recovery gets triggered (hopefully in slow start).
>       >> If TLP is not acked, RTO is needed and recovery is
>       completed in using RTO
>       >> recovery in slow start. Now, if the RTO recovery is
>       successful (no
>       >> losses during RTO recovery), it should be quite likely
>       that the TCP
>       >> sender is able to successfully send also one new
>       segment, because it
>       >> enters CA when it is sending at the half of the previous
>       rate. So, once
>       >> you get Ack for the new segment, the TLP back off can be
>       removed and it
>       >> is unlikely that the TLP back off slowed down next loss
>       detection.
>       >> On the other hand, if the RTO recovery after an
>       unsuccessful TLP is not
>       >> successful (more losses are detected), it is quite
>       likely that congestion
>       >> has not been resolved. So, it is important to be
>       conservative and have a
>       >> backed-off PTO (or even turn it off) to avoid (further)
>       unnecessary
>       >> rexmits. If PTO is not backed of, I'd envision PTO
>       mainly failing to get
>       >> an Ack in such a case and thereby it not being that
>       useful.
>       >>
>       >> This certainly would require experimental data in a
>       heavily congested
>       >> setting to really figure out the actual impact of
>       different alternatives.
>       >>
>       >>> So the question we need to look at is if RACK-TLP
>       2RTT-PTO + regular
>       >>> RTO-backoff is going to cause major stability issues in
>       extreme
>       >>> congestion. My understanding based on my officemate Van
>       Jacobson's
>       >>> explanation is, as long as there's eventual exponential
>       backoff, we'll
>       >>> avoid the repeated shelling the network.
>       >>
>       >> Right. But the problem is that with TLP as specified we
>       do not have full
>       >> exponential back off. It lacks Karn's clamped retransmit
>       backoff (as it
>       >> is called in Van Jacobson's seminal paper) which
>       requires keeping
>       >> backed-off timer of a retransmitted segment for the next
>       (new) segment and
>       >> ensures that there is eventual exponential backoff.
>       >> Backing off just RTO is not enough, because "fixed" PTO
>       for each new
>       >> segment breaks this "back-off chain", that is, the
>       exponential backoff
>       >> is not continueed until there is evidence that
>       congestion has been
>       >> resolved (a cumulative Ack arrives for new data). But as
>       I said, backing
>       >> off RTO saves RACK-TLP from a full congestion collapse.
>       Still, wasting
>       >> 50% of the available network capacity in certain usage
>       scenarios does not
>       >> sound acceptable for me.
>       >
>       > [RACK-TLP-team:]
>       >
>       > To address concerns related to Karn's algorithm, perhaps
>       it would be
>       > sufficient to specify that after a RACK-TLP sender sends
>       a TLP probe
>       > that is a retransmitted segment, it temporarily disables
>       further TLP
>       > retransmissions until the sender obtains a valid RTT
>       sample (i.e. from
>       > a non-retransmitted segment, or a segment with [RFC7323]
>       TCP
>       > timestamps)?
>
>       As a basic rule: disabling TLP probes for the period of
>       loss recovery as
>       currently specified sounds like a correct approach provided
>       that the end
>       condition
>
>         "until the sender obtains a valid RTT sample"
>
>       is added (this would be without timestamps, see some
>       considerations with
>       timestamps later below).
>
>       Then some justification and maybe some exeptions can also
>       be specified
>       (all these without timestamps):
>
>       If fast recovery is not entered via detecting loss using
>       TLP, the
>       sender does not need to wait for a valid RTT sample. It
>       should be enough
>       that RecoveryPoint is reached, i.e., use normal exit
>       condition for fast
>       recovery. However, I am not sure whether detecting a loss
>       of a rexmit
>       during fast recovery might change this?
>
>       I think it does not matter whether a TLP probe is a
>       retransmitted segment
>       or not:
>
>       (a) if there is no ack after TLP probe and RTO expires it
>       should be clear
>       that TLP must be disabled until an ack for a new previously
>       not rexmitted
>       segment is received, regardless of the probe segment sent
>       by TLP.
>       This is Karn's algorithm.
>
>       (b) if TLP triggers an Ack that is used to detect a loss of
>       an entire
>       flight of data, this may later during loss recovery result
>       in an RTO
>       which definitely requires disabling TLP as per the basic
>       rule. There
>       might be some exeptions when the basic rule is not
>       necessary but it is
>       hard for me to figure one right now, except
>
>       (c) if TLP rexmit detects and repairs a single loss, then
>       CC is anyway
>       invoked and Ack is received. If the arriving Ack is
>       triggered by the TLP
>       probe segment, it indicates that the conditions are likely
>       not that bad
>       (RTO not needed). So it might be ok, to rearm PTO for the
>       next time
>       without any backoff (without disbling it). However,
>
>       (d) if TLP was spurious rexmit, then PTO value obviously
>       was too low
>       (effective RTT may have increased suddenly and Acks got
>       just delayed).
>       This is possibly detected only one RTT later when the
>       DupAck of the TLP
>       rexmit arrives. TLP may have got rearmed by that time again
>       with too
>       low value, so it should be rearmed with doubled value to
>       avoid another
>       spurious TLP PTO (or disabled).
>
>       Then with timestamps. I think the simple rule for disabling
>       TLP "until
>       the sender obtains a valid RTT sample" is not necessarily
>       giving the
>       desired outcome in all cases. For example, if a TLP probe
>       is send and
>       then a late ack arrives for a segment sent before the TLP
>       probe and gives
>       a valid RTT sample with help of timestamps. This would
>       allow a new TLP
>       even though the sender has not got an Ack for a new segment
>       sent after
>       the TLP probe?
>
>       So, also with timestamps the requirement must be that the
>       sender obtains
>       a valid RTT sample for a segment sent after the TLP probe.
>       And, at least,
>       if RTO recovery is entered, then the Ack must be for a new
>       segment.
>       Cannot figure out now all potential exceptions when a valid
>       RTT sample
>       with timestamps would allow re-enabling TLP earlier. Too
>       complex
>       algorithm with too many possible scenarios to investigate
>       right now ;)
>
>       A single, simple rule would be advisable here. Otherwise,
>       this may end up
>       being too complex to get it right.
> 
>
>       >>> As a matter of fact some
>       >>> major TCP (!= Linux) implementation has implemented
>       linear backoff of
>       >>> first N RTOs before exp-backoff.
>       >>
>       >> But that's quite ok as long as there is eventual
>       exponential backoff,
>       >> including Karn's clamped retransmit backoff. Linear
>       back-off in the
>       >> beginning just makes resolving (heavy) congestion a bit
>       slower.
>       >>
>       >>>>
>       >>>>>>
>       >>>>>> Additional notes:
>       >>>>>>
>       >>>>>> Sec 2.2:
>       >>>>>>
>       >>>>>> Example 2:
>       >>>>>> "Lost retransmissions cause a  resort to RTO
>       recovery, since
>       >>>>>>   DUPACK-counting does not detect the loss of the
>       retransmissions.
>       >>>>>>   Then the slow start after RTO recovery could cause
>       burst losses
>       >>>>>>   again that severely degrades performance
>       [POLICER16]."
>       >>>>>>
>       >>>>>> RTO reovery is done in slow start. The last sentence
>       is confusing as
>       >>>>>> there is no (new) slow-start after RTO recovery (or
>       more precisely
>       >>>>>> slow start continues until cwnd > ssthresh). Do you
>       mean: if/when slow
>       >>>>>> start still continues after RTO Recovery has
>       repaired lost segments,
>       >>>>>> it may cause burst losses again?
>       >>>>> I mean the slow start after (the start of) RTO
>       recovery. HTH
>       >>>>
>       >>>> Tnx. I'd appreciate if the text could be clarified to
>       reflect this more
>       >>>> accurately. Maybe something along the lines(?):
>       >>>>
>       >>>>   "Then the RTO recovery in slow start could cause
>       burst
>       >>>>   losses again that severely degrades performance
>       [POLICER16]."
>       >>>>
>       >>>>>>
>       >>>>>> Example 3:
>       >>>>>>   "If the reordering degree is beyond DupThresh, the
>       DUPACK-
>       >>>>>>    counting can cause a spurious fast recovery and
>       unnecessary
>       >>>>>>    congestion window reduction.  To mitigate the
>       issue, [RFC4653]
>       >>>>>>    adjusts DupThresh to half of the inflight size to
>       tolerate the
>       >>>>>>    higher degree of reordering.  However if more
>       than half of the
>       >>>>>>    inflight is lost, then the sender has to resort
>       to RTO recovery."
>       >>>>>>
>       >>>>>> This seems to be somewhat incorrect description of
>       TCP-NCR specified in
>       >>>>>> RFC 4653. TCP-NCR uses Extended Limited Transmit
>       that keeps on sending
>       >>>>>> new data segments on DupAcks that makes it likely to
>       avoid an RTO in
>       >>>>>> the given example scenario, if not too many of the
>       the new data
>       >>>>>> segments triggered by Extended Limited Transmit are
>       lost.
>       >>>>> sorry I don't see how the text is wrong describing
>       RFC4653,
>       >>>>> specifically the algorithm in adjusting ssthresh
>       >>>>
>       >>>> To my understanding RFC4653 initializes DupThresh to
>       half of the inflight
>       >>>> size in the beginning of the Extended Limited
>       Transmit. Then on each
>       >>>> DupAck it adjusts (recalculates) DupThresh again such
>       that ideally a cwnd
>       >>>> worth of DupAcks are received before packet loss is
>       declared (or
>       >>>> reordering detected). So, if I am not incorrect, loss
>       of a half of the
>       >>>> inflight does not necessarily result in RTO recovery
>       with TCP-NCR.
>       >>> Could you suggest the text you'd like on NCR
>       description.
>       >>
>       >> I'm not an expert nor closely acquainted with NCR. There
>       might be many
>       >> different packet loss patterns that may affect the
>       behavior. So, my
>       >> advice is to simply drop the last sentence starting with
>       "However ...",
>       >> because it seems incorrect and replace the second but
>       last sentence:
>       >>
>       >>    To mitigate the issue, [RFC4653]
>       >>    adjusts DupThresh to half of the inflight size to
>       tolerate the
>       >>    higher degree of reordering.
>       >>
>       >> -->
>       >>
>       >>    To mitigate the issue, TCP-NCR [RFC4653]
>       >>    increases the DupThresh from the current fixed value
>       of three duplicate
>       >>    ACKs [RFC5681] to approximately a congestion window
>       of data having left
>       >>    the network.
>       >
>       > [RACK-TLP-team:]
>       >
>       > Sure, we will revise accordingly.
>
>       Tnx.
>
>       >
>       >>>>>>
>       >>>>>> Sec. 3.5:
>       >>>>>>
>       >>>>>>   "For example, consider a simple case where one
>       >>>>>>   segment was sent with an RTO of 1 second, and then
>       the application
>       >>>>>>   writes more data, causing a second and third
>       segment to be sent right
>       >>>>>>   before the RTO of the first segment expires. 
>       Suppose only the first
>       >>>>>>   segment is lost.  Without RACK, upon RTO
>       expiration the sender marks
>       >>>>>>   all three segments as lost and retransmits the
>       first segment.  When
>       >>>>>>   the sender receives the ACK that selectively
>       acknowledges the second
>       >>>>>>   segment, the sender spuriously retransmits the
>       third segment."
>       >>>>>>
>       >>>>>> This seems incorrect. When the sender receives the
>       ACK that selectively
>       >>>>>> acknowledges the second segment, it is a DupAck as
>       per RFC 6675 and does
>       >>>>>> not increase cwnd and cwnd remains as 1 MSS and pipe
>       is 1 MSS. So, the
>       >>>>>> rexmit of the third segment is not allowad until the
>       cumulative ACK of
>       >>>>>> the first segment arrives.
>       >>>>> I don't see where RFC6675 forbids growing cwnd. Even
>       if it does, I
>       >>>>> don't think it's a good thing (in RTO-slow-start) as
>       DUPACK clearly
>       >>>>> indicates a delivery has been made.
>       >>>>
>       >>>> SACKed sequences with DUpAcks indicate that those
>       sequences were
>       >>>> delivered but it does not tell when they were sent.
>       The basic principle
>       >>>> of slow start is to reliably determine the available
>       network capacity
>       >>>> during slow start. Therefore, slow start must ensure
>       it uses only
>       >>>> segments sent during the slow start to increase cwnd.
>       Otherwise, a TCP
>       >>>> sender may encounter exactly the problem of
>       unnecessary retransmission
>       >>>> envisioned in this example of RACK-TCP draft (and
>       increase cwnd on not
>       >>>> valid Acks).
>       >>>>
>       >>>> RFC 6675 does re-specify DupAck with SACK option but
>       it does not include
>       >>>> the rule for slow start. Slow start is specified in
>       RFC 5681. It is
>       >>>> crystal clear in allowing increase in cwnd only on
>       cumulative
>       >>>> Acks, i.e., forbidding to increase cwnd on DupAcks
>       (RFC 5681, Sec 3,
>       >>>> page 6:
>       >>>>
>       >>>>    During slow start, a TCP increments cwnd by at most
>       SMSS bytes for
>       >>>>    each ACK received that cumulatively acknowledges
>       new data.
>       >>>>
>       >>>> Maybe this example in the RACK-TLP draft was inspired
>       by a incorrect
>       >>>> implementation of SACK-based loss recovery?
>       >>>>
>       >>>> FYI: when we were finalizing RFC 6675 I suggested
>       including also an
>       >>>> algorithm for RTO recovery with SACK in RFC 6675. The
>       reason was exactly
>       >>>> that it might be not easy to gather info from multiple
>       documents and
>       >>>> hence help the implementor to have all necessary
>       advice in a single
>       >>>> document. This unfortunately did not get realized
>       though.
>       >>>
>       >>> I am honestly lost quibbling these RFC6675
>       implementations and feeling
>       >>> pedantic to these standards largely serve as guiding
>       principles
>       >>> instead of line-by-line code. Is cwnd one packet bigger
>       or smaller in
>       >>> this example making any different in advancing the
>       Internet's
>       >>> capability to do better loss detection. I do not think
>       so.
>       >>
>       >> Sorry, I don't understand what was quibbling here. I
>       believe I did not
>       >> argue anything about cwnd size with this example at hand
>       in Sec. 3.5.
>       >> My point is that it describes incorrect slow start
>       behavior. Correctly
>       >> implemented slow start does not have the problem
>       illustriated in the
>       >> example.
>       >>
>       >>> At this point please suggest a text you like to change.
>       >>
>       >> That is simple. Please remove the description of the
>       behavior without
>       >> RACK. Or correct it: the behavior would be exactly the
>       same as with RACK,
>       >> but the reason for not unnecessarily retransmitting
>       third segment can be
>       >> described to be different.
>       >> And, please correct also the description with RACK. With
>       RACK, third
>       >> segment does not get unnecessarly rexmitted for the
>       reason already
>       >> indicated and also because cwnd=1 MSS. And, no new
>       segments are allowed
>       >> either by cwnd=1.
>       >>
>       >
>       > [RACK-TLP-team:]
>       >
>       > Thanks. Perhaps it makes sense to consider a slightly
>       different RTO
>       > scenario to illustrate the point about how RACK RTO
>       behavior can avoid
>       > spurious retransmissions.
>       >
>       > Here is the proposed new text:
>       >
>       > "3.5.  An Example of RACK-TLP in Action: RTO
>       >
>       >   In addition to enhancing fast recovery, RACK improves
>       the accuracy of
>       >   RTO recovery by reducing spurious retransmissions.
>       >
>       >   Without RACK, upon RTO timer expiration the sender
>       marks all the
>       >   unacknowledged segments lost.  This approach can lead
>       to spurious
>       >   retransmissions.  For example, consider a simple case
>       where one
>       >   segment was sent with an RTO of 1 second, and then the
>       application
>       >   writes more data, causing a second and third segment to
>       be sent right
>       >   before the RTO of the first segment expires.  Suppose
>       none of the
>       >   segments were lost.  Without RACK, if there is a
>       spurious RTO then
>       >   the sender marks all three segments as lost and
>       retransmits the
>       >   first segment. If the ACK for the original copy of the
>       first segment
>       >   arrives right after the spurious RTO retransmission,
>       then the
>       >   sender continues slow start and spuriously retransmits
>       the second
>       >   and third segments, since it (erroneously) presumed
>       they are lost.
>
>       That seems fine. Would it be useful to add an additional
>       note saying that
>       if F-RTO is in use, then further unnecessary
>       retransmissions can be
>       avoided?
>
>       >   With RACK, upon RTO timer expiration the only segment
>       automatically
>       >   marked lost is the first segment (since it was sent an
>       RTO ago); for
>       >   all the other segments RACK only marks the segment lost
>       if at least
>       >   one round trip has elapsed since the segment was
>       transmitted.
>       >   Consider the previous example scenario, this time with
>       RACK.  With
>       >   RACK, when the RTO expires the sender only marks the
>       first segment as
>       >   lost, and retransmits that segment.  The other two very
>       recently sent
>       >   segments are not marked lost, because they were sent
>       less than one
>       >   round trip ago and there were no ACKs providing
>       evidence that they
>       >   were lost. Upon receiving the ACK for the RTO
>       retransmission the
>       >   RACK sender would not yet retransmit the second or
>       third segment. but
>       >   rather would rearm the RTO timer and wait for a new RTO
>       interval to elapse
>       >   before marking the second or third segments as lost.
>       > "
>
>       This behavior with RACK-TLP does not quite match with my
>       reading of
>       the draft in all parts, so I have some questions.
>
>       First, wouldn't the PTO timer be first running for the
>       first segment
>       and when the sender sends the 2nd and 3rd segment it would
>       rearm PTO for
>       the 3rd segment? My understanding based on Sec 8 is that
>       RTO cannot be
>       armed until PTO fires? And when the PTO fires, the sender
>       would
>       retransmit the 3rd segment? Maybe I am missing something?
>
>       Anyways, an RTO timer may expire in this scenario (e.g., if
>       there is no
>       Ack for TLP).
>       Let's assume the RTO timer expires as described. When the
>       first Ack after
>       the rexmit of the 1st segment arrives, isn't that the late
>       ACK for the
>       original copy of the first segment, not the ACK for the RTO
>       retransmission? Anyways, when this first Ack arrives after
>       the RTO, the
>       second and third segment are not marked as lost as
>       described, which
>       nicely prevents unnecessary retransmissions of those
>       segments.
>       However, if the 2nd and 3rd segments were lost, wouldn't
>       rearming RTO
>       with exponentially backed-off timer value (2*RTO) make the
>       recovery of
>       the 2nd and 3rd segment very slow? After 2*RTO the timer
>       expires and
>       the 2nd segment is rexmitted and only after one additional
>       RTT when the
>       ack for the 2nd segment arrives, the sender rexmits the 3rd
>       segment?
>       Wouldn't it be much more efficient to disable RACK loss
>       detection during
>       an RTO recovery and employ F-RTO? Maybe I am missing
>       something again?
> 
>
>       Then to the problem with the pseudocode I mentioned in the
>       beginning of
>       this message. If I am not incorrect it seems to me that
>       setting a value
>       to TLP.end_seq in TLP_send_probe() is off by one byte?
>       Maybe something
>       similar is wrong somewhere else too, didn't check. Please
>       apologise me if
>       I missed something.
>
>       Let's use an example with RACK-TLP recovery:
>
>       Send P1, ..., P9, P10
>       [P1,.., P9 delayed, P10 dropped]
>       /* I believe P1-9 delayed is not important here */
>
>       After 2*SRTT PTO expires
>       P10 rexmitted by TLP
>         [ TLP_send_probe(): set TLP.end_seq = SND.NXT (= SEG.P11)
>       ]
>
>         [after TLP, the  application possibly provides more data
>       to send]
>
>       Ack for original P1 arrives
>       (P11 sent, if more data by application)
>       Ack for original P2 arrives
>       (P12 sent, if more data by application)
>       ...
>       Ack for original P9 arrives
>       (P19 sent, if more data by application)
>
>       Ack for P10 triggered by TLP rexmit of P10 arrives
>         [TLP_process_ack(ACK):
>          ...
>          If ACK's ack. number > TLP.end_seq
>           returns False
>           since ACK's ack. number = SEQ.ACK = TLP.end_seq =
>       SEG.P11]
>
>          Loss of P10 is undetected and no reaction to congestion]
> 
>
>       My understanding of draft text is that a loss of P10 is not
>       concealed.
>       Please advise.
>
>       Best regards,
>
>       /Markku
>
>       > best regards,
>       >
>       > neal
>       >
>       >> Please remove also the last paragraph of the Sec. 3.5,
>       it being also
>       >> incorrect description of behavior.
>       >>
>       >> BR,
>       >>
>       >> /Markku
>       >>
>       >>>>
>       >>>> BR,
>       >>>>
>       >>>> /Markku
>       >>>>
>       >>>>>>
>       >>>>>> Best regards,
>       >>>>>>
>       >>>>>> /Markku
>       >>>>>>
>       >>>>>>
>       >>>>>>
>       >>>>>> On Mon, 16 Nov 2020, The IESG wrote:
>       >>>>>>
>       >>>>>>>
>       >>>>>>> The IESG has received a request from the TCP
>       Maintenance and Minor Extensions
>       >>>>>>> WG (tcpm) to consider the following document: -
>       'The RACK-TLP loss detection
>       >>>>>>> algorithm for TCP'
>       >>>>>>>  <draft-ietf-tcpm-rack-13.txt> as Proposed Standard
>       >>>>>>>
>       >>>>>>> The IESG plans to make a decision in the next few
>       weeks, and solicits final
>       >>>>>>> comments on this action. Please send substantive
>       comments to the
>       >>>>>>> [e-mail] mailing lists by 2020-11-30.
>       Exceptionally, comments may
>       >>>>>>> be sent to [e-mail] instead. In either case,
>       please retain the beginning
>       >>>>>>> of the Subject line to allow automated sorting.
>       >>>>>>>
>       >>>>>>> Abstract
>       >>>>>>>
>       >>>>>>>
>       >>>>>>>   This document presents the RACK-TLP loss
>       detection algorithm for TCP.
>       >>>>>>>   RACK-TLP uses per-segment transmit timestamps and
>       selective
>       >>>>>>>   acknowledgements (SACK) and has two parts: RACK
>       ("Recent
>       >>>>>>>   ACKnowledgment") starts fast recovery quickly
>       using time-based
>       >>>>>>>   inferences derived from ACK feedback.  TLP ("Tail
>       Loss Probe")
>       >>>>>>>   leverages RACK and sends a probe packet to
>       trigger ACK feedback to
>       >>>>>>>   avoid retransmission timeout (RTO) events. 
>       Compared to the widely
>       >>>>>>>   used DUPACK threshold approach, RACK-TLP detects
>       losses more
>       >>>>>>>   efficiently when there are application-limited
>       flights of data, lost
>       >>>>>>>   retransmissions, or data packet reordering
>       events.  It is intended to
>       >>>>>>>   be an alternative to the DUPACK threshold
>       approach.
>       >>>>>>>
>       >>>>>>>
>       >>>>>>>
>       >>>>>>>
>       >>>>>>> The file can be obtained via
>       >>>>>>>
>       https://datatracker.ietf.org/doc/draft-ietf-tcpm-rack/
>       >>>>>>>
>       >>>>>>>
>       >>>>>>>
>       >>>>>>> No IPR declarations have been submitted directly on
>       this I-D.
>       >>>>>>>
>       >>>>>>>
>       >>>>>>>
>       >>>>>>>
>       >>>>>>>
>       >>>>>>> _______________________________________________
>       >>>>>>> tcpm mailing list
>       >>>>>>> [e-mail]
>       >>>>>>> https://www.ietf.org/mailman/listinfo/tcpm
>       >>>>>>>
>       >>>>>
>       >>>
>       >
> 
> 
>