[tcpm] Re: using SACK info for RTTM?

[Neal Cardwell <[e-mail]>]

On Wed, Jun 5, 2024 at 2:21 PM Yoshifumi Nishida <[e-mail]> wrote:

> Hi Yuchung, Neal, thank you so much. It’s interesting.
>
> So, it might be a good time to revive
> https://datatracker.ietf.org/doc/draft-yang-tcpm-ets/ ?
>

Yes, reviving the work for some kind of standardization of more precise TCP
timestamps is something we would like to do when time permits.


> OTOH, I’m thinking why DSACK is not sufficient here.
>

DSACK can work sometimes, however it is not as good as timestamp undo, for
at least a few reasons:

(1) DSACK undo is slower: DSACK undo takes about two round trips longer
than timestamp undo. With DSACK undo the data sender has one extra round
trip to make a full set of spurious retransmissions for apparent holes in
the sequence space, and then a second round trip to receive all the DSACKs
for the spurious retransmissions. Only then can the data sender undo the
congestion control response. With timestamp-based undo, usually a data
sender will receive an ACK very shortly (say, O(1ms)) after its spurious
retransmit that covers the spuriously retransmitted sequence and has a TS
ECR from before the retransmit, allowing the undo to happen immediately.

(2) DSACK undo is unreliable: if even a single ACK packet containing a
DSACK is lost, then the data sender cannot undo the congestion control
response. For example, if a flow is fully utilizing a 10Gbps * 100ms path,
and thus (1500B MTU) its cwnd is at least 82,562, then the loss rate in the
direction of returning ACKs needs to be zero in that round trip, or on
average less than 1/82,562, or < 0.0012%. Not impossible, but a stringent
requirement. With timestamp-based undo every incoming ACK has a TS ECR
value that allows undo, so ACK loss is not a problem.

neal



> Thanks,
> —
> Yoshi
>
> On Wed, Jun 5, 2024 at 10:06 AM Yuchung Cheng <[e-mail]> wrote:
>
>> Also TCP timestamp needs to really move to usec level for today's
>> data-center networks, which Eric Dumazet finally upstreamed that feature
>> (to opt-in). anything beyond 10us can't be used in Eifel
>>
>> On Wed, Jun 5, 2024 at 6:41 AM Neal Cardwell <[e-mail]>
>> wrote:
>>
>>> IMHO by far the biggest benefit of TCP timestamps is not in RTT
>>> measurement or PAWS, but in using them for "Eifel" undo (a la RFC 3522, RFC
>>> 4015): quickly detecting spurious loss detection events due to reordering,
>>> and quickly undoing the spurious congestion control slow-down response.
>>> This is important since reordering is increasingly common due to many
>>> increasingly common network mechanisms: link-layer retransmissions for
>>> wifi/cellular links, traffic engineering, multipathing and ECMP/WCMP
>>> load-balancing, protective load balancing (SIGCOMM 2022), protective
>>> reroute (SIGCOMM 2023), multi-queue NICs, etc. Those factors make the 12
>>> bytes of TCP option space overwhelmingly worth it.
>>>
>>> best regards,
>>> neal
>>>
>>>
>>> On Wed, Jun 5, 2024 at 3:03 AM Yoshifumi Nishida <[e-mail]>
>>> wrote:
>>>
>>>> Hi Yuchung,
>>>>
>>>> Thanks for the explanation.
>>>> I thought a bit about the trade-off between using 12 bytes options
>>>> space and giving up measuring RTTs for retransmitted packets.
>>>> But, I am included to prefer measuring RTTs for now.
>>>>
>>>> --
>>>> Yoshi
>>>>
>>>> On Mon, Jun 3, 2024 at 1:57 PM Yuchung Cheng <[e-mail]> wrote:
>>>>
>>>>> hi Yoshifumi,
>>>>>
>>>>> Linux only uses TS-opts if needed to disambiguate on RTT samples
>>>>> covering sequences that have been retransmitted. This applies to SACK or
>>>>> non-SACK. In order words, if an S/ACK covers a sequence range that has
>>>>> never been retransmitted, Linux does not use timestamp options.
>>>>>
>>>>> On Mon, Jun 3, 2024 at 1:29 PM Yoshifumi Nishida <[e-mail]>
>>>>> wrote:
>>>>>
>>>>>> Hi Neal, thank you so much for the comments.
>>>>>>
>>>>>> The linux algorithm you've described makes sense to me and it seems
>>>>>> the scheme doesn't require timestamp options.
>>>>>> However, as far as I've read linux code, it seems that linux still
>>>>>> uses timestamp options for RTT measurement to some extent.
>>>>>> I'm curious why linux is mixing two schemes for RTTM.
>>>>>> --
>>>>>> Yoshi
>>>>>>
>>>>>> On Mon, Jun 3, 2024 at 8:57 AM Neal Cardwell <[e-mail]>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jun 3, 2024 at 11:02 AM Yoshifumi Nishida <
>>>>>>> [e-mail]> wrote:
>>>>>>>
>>>>>>>> Hi folks,
>>>>>>>>
>>>>>>>> While I was checking RFC7323, I found the following sentence.
>>>>>>>>
>>>>>>>> RTTM update processing explicitly excludes segments not updating
>>>>>>>> SND.UNA.  The original text could be interpreted to allow taking
>>>>>>>> RTT samples when SACK acknowledges some new, non-continuous
>>>>>>>> data.
>>>>>>>>
>>>>>>>> I am a bit curious about the rationale of this sentence.
>>>>>>>> It seems to me that we cannot measure RTT when we have a gap in packet sequence with this rule.
>>>>>>>>
>>>>>>>>
>>>>>>> Yes, that rule forbids using RFC7323 timestamps for calculating RTT
>>>>>>> samples for SACKed sequence ranges.
>>>>>>>
>>>>>>> The rationale: AFAIK this rule is a necessary consequence of the
>>>>>>> conditions under which TS.Recent is updated.
>>>>>>>
>>>>>>> The rules for updating TS.Recent are in sec 4.3, "Which Timestamp to
>>>>>>> Echo":
>>>>>>>   https://datatracker.ietf.org/doc/html/rfc7323#section-4.3
>>>>>>> Rule (2) in sec 4.3 says:
>>>>>>>   If:
>>>>>>>     SEG.TSval >= TS.Recent and SEG.SEQ <= Last.ACK.sent
>>>>>>>   then SEG.TSval is copied to TS.Recent; otherwise, it is ignored.
>>>>>>>
>>>>>>> Since out-of-order sequence ranges that are SACKed will fail the
>>>>>>> SEG.SEQ <= Last.ACK.sent check, SACKed sequence ranges will not update
>>>>>>> TS.Recent. So using TS.Recent to calculate an RTT sample for a SACKed
>>>>>>> sequence range could, in general, give a vastly overestimated RTT sample.
>>>>>>> So that's why it's forbidden by the RFC.
>>>>>>>
>>>>>>> However, in practice usually this does not need to be a big deal.
>>>>>>> For example, Linux TCP still obtains an RTT sample for every
>>>>>>> non-retransmitted SACKed sequence range, by:
>>>>>>>
>>>>>>> (a) recording the transmit time of every sequence range
>>>>>>> (b) recording whether that sequence range was retransmitted, and then
>>>>>>> (c) using those two pieces of information when that sequence range
>>>>>>> is cumulatively or selectively ACKed, to calculate an RTT sample
>>>>>>> (rtt_sample = now - transmit_timestamp) if the sequence range was never
>>>>>>> retransmitted.
>>>>>>>
>>>>>>> So, in Linux TCP, SACKed sequence ranges fail to generate an RTT
>>>>>>> sample only when they were previously retransmitted.
>>>>>>>
>>>>>>> best regards,
>>>>>>> neal
>>>>>>>
>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> --
>>>>>>>> Yoshi
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> tcpm mailing list -- [e-mail]
>>>>>>>> To unsubscribe send an email to [e-mail]
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>> tcpm mailing list -- [e-mail]
>>>>>> To unsubscribe send an email to [e-mail]
>>>>>>
>>>>>