Re: [tcpm] [EXTERNAL] apparent ambiguities in draft spec for Hystart++ window start/end?

[Yi Huang <[e-mail]>]

Hi Neal,

Thanks for the review and sorry for the delay!

1. Our Hystart++ implementation actually does (b) and the draft intends (b) too.
>> Normally when I read that a sequence number X "is ACKed" I think of the (b) case, i.e. SND.UNA >= X.
I am somewhat confused by this because SND.UNA means the first byte of data that has not been ACKed. If it equals X, then X has not been ACKed yet.

We tried (a) in the past and we had issues with it. With ACK compression, it's common (at least for Windows) to see only 1 or a few ACKs for a whole window of packets especially in the first couple of RTT rounds. In this case, a round will very likely last for 2 RTT rounds (when it's waiting for the only ACK for the next window worth of data.)

We are not happy with (b) either. We set windowEnd to SndNxt when a round ends but SndNxt was updated by the previous ACK that triggered new data, which means the ACK that covers windowEnd (AckNo >= windowEnd) might come in anytime leading to marking rounds inaccurately.

We have been experimenting with a new method to mark rounds- set windowEnd to SndUna + min(CWnd, advertised window from the peer) when a round ends. We have been using it in LEDBAT++ already and rounds are aligned with RTTs more consistently. How does this sound?

2. We have not done any targeted testing of app-limited scenarios unfortunately. Our implementation does assume that the end of a previous round is the start of next round. I understand the situation you described where the next round might actually start a few days later and we will be comparing current round's minRTT with a stale last round's minRTT. One mitigation for this issue would be to clear last round's minRTT when idle is detected. E.g. if the time diff between two consecutive rounds is greater than a threshold, we skip the current round.

3. I agree it's not true in app-limited scenarios and we should remove it.

Thanks,

Yi
________________________________
From: Neal Cardwell <[e-mail]>
Sent: Monday, November 29, 2021 3:23 PM
To: Praveen Balasubramanian <[e-mail]>; [e-mail] <[e-mail]>
Cc: tcpm <[e-mail]>
Subject: [EXTERNAL] apparent ambiguities in draft spec for Hystart++ window start/end?

Hi,

Thanks again for the nice Hystart++ draft!

Below are three quick comments about what AFAICT are some ambiguities about the spec for detecting and handling window starts/ends, plus a suggestion for wording that section:

(1) Currently, the draft at:
  https://datatracker.ietf.org/doc/html/draft-ietf-tcpm-hystartplusplus#section-4.2<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fhtml%2Fdraft-ietf-tcpm-hystartplusplus%23section-4.2&data=04%7C01%7Chuanyi%40microsoft.com%7Cc36d089c6c1747a652fe08d9b38f5212%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637738251037372864%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=p3NISukUXfgsSbNFq6CTmjwhX8jdbXCr98MYgSQrWz0%3D&reserved=0>
says:

      When windowEnd is ACKed, the current round ends and windowEnd is
      set to SND.NXT

IMHO this is a bit ambiguous as to whether it means:
(a)  if (SND.UNA > windowEnd)
or
(b)  if (SND.UNA >= windowEnd)

Normally when I read that a sequence number X "is ACKed" I think of the (b) case, i.e. SND.UNA >= X. But for application-limited flights of data, I think that (b) interpretation would lead to the Hystart++ logic for the start/end of rounds potentially being executed twice for each flight of data: once for the first segment in the flight, and once for the last segment in the flight (because while processing ACKs for an application-limited flight the value of SND.NXT may not change).

So I imagine the Hystart++ algorithm actually intends the (a) semantics. The (a) approach is what Linux TCP CUBIC Hystart does, and the AFAICT  (a) semantics give the correct once-per-round behavior even for application-limited flights. I think it would be good to express that aspect of the algorithm more precisely to indicate the approach that is intended.

(2) A second, related, issue: for per-round behavior section 4.2 of the algorithm specifies how to detect the end of a round ("When windowEnd is ACKed, the current round ends"), but only specifies what action is to be taken at the start of a round ("At the start of each round during standard slow start ([RFC5681]) and CSS:").

I suppose the text is implicitly considering that the start of one round is the same as the end of the previous round. For bulk data that assumption might be acceptable; it might be fine to think of the start of one round as the same as the end of the previous round. But for application-limited flights of data that assumption does not hold, and the start of a round and end of a round are very different events with very different timing and characteristics. So for the common case of application-limited flights of data currently the draft does not specify how to detect the "start of each round" event that triggers the per-round actions; it only defines how to detect the "end" of a round.

(3)  A third, related, issue: the text in section 4.2 says:


   A round is chosen to be approximately the Round-Trip Time (RTT).

For application-limited flights this text is a poor or confusing fit. For application-limited flights,  the duration of time between two successive rounds, aka the length of a round (IMHO), can be arbitrarily long; it can be much longer than the RTT and perhaps hours or even days. I would suggest just dropping that sentence, to avoid confusion.

---

(a suggestion):

To clear up these ambiguities I would suggest tweaking the existing text from:

(before):
"""

   A round is chosen to be approximately the Round-Trip Time (RTT).  We
   recommend that rounds be measured using sequence numbers.  Round can
   be approximated using sequence numbers as follows:

      Define windowEnd as a sequence number initialize to SND.UNA

      When windowEnd is ACKed, the current round ends and windowEnd is
      set to SND.NXT

   At the start of each round during standard slow start ([RFC5681<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdatatracker.ietf.org%2Fdoc%2Fhtml%2Frfc5681&data=04%7C01%7Chuanyi%40microsoft.com%7Cc36d089c6c1747a652fe08d9b38f5212%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637738251037372864%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=d8QjJJDqAlTEi4xYVNByJm3Dk7tYjY%2BA2wv7w%2BanKck%3D&reserved=0>]) and
   CSS:

"""

...to be something more like the following:

(after):
"""

   Hystart++ measures rounds using sequence numbers, as follows:

      Define windowEnd as a sequence number initialized to SND.UNA.

      Upon receiving an ACK, if (SND.UNA > windowEnd) then
      a new round has started and the sender sets windowEnd to
      SND.NXT.

   At the start of each new round during standard slow start ([RFC5681])
   and CSS:
"""

best regards,
neal