Re: [tcpm] TLP questions

[Neal Cardwell <[e-mail]>]

On Thu, May 10, 2018 at 7:29 PM Praveen Balasubramanian
<[e-mail]>
wrote:

> Thanks Neal for the detailed response along with the historical context.
Looking forward to draft 04 updates.



> A few more questions and comments.

Thanks again for another round of excellent questions and comments. :-)

> > then typically RACK will install a timer based on the reordering
> > window, and when that timer fires it will mark some packets lost
> > and enter fast recovery

> The "reordering settling" timer is defined as optional in the draft.
> The Windows implementation currently does not use this timer.
> Until we add such a timer, we plan to not prevent TLP even if SACK
> scoreboard is not empty. However if recovery is triggered, we'll
> cancel the PTO and arm an RTO. Do you see any issues with this
> approach? Since the draft makes the "reordering settling"
> timer optional, I think it should suggest this alternative approach.

That sounds to me like it would work OK. But IMHO it sounds like
it's missing out on the opportunity to use a reordering timer to
speed up fast recovery quite a bit by initiating a RACK-based
recovery using the reordering timer.

That scenario sounds like:

+ send the original flight of data
+ 1*srtt passes
+ receive 1 or 2 SACKs for packets in that flight, schedule TLP
+ wait 2*srtt
+ TLP timer fires, send TLP
+ 1*srtt passes
+ receive SACK of TLP, initiate RACK-based fast recovery

If there's a reordering timer, this could be:

+ send the original flight of data
+ 1*srtt passes
+ receive 1 or 2 SACKs for packets in that flight, schedule TLP
+ wait 0.25*min_rtt
+ reordering timer fires,  initiate RACK-based fast recovery

With the reordering timer, AFAICT it takes about 2.75*srtt less time before
the recovery begins, if I correctly noted all the details.

> > So the 2ms is to allow for real-world jitter in the network and end
hosts

> Currently the Windows implementation doesn't use TLP and RACK for
> connections with < 10 msec RTT. So until we change this logic,
> we will skip adding the 2 ms jitter protection.

Skipping the 2ms jitter protection sounds OK in that case.

Are TLP and RACK skipped in the <10ms case due to the granularity of
the send and ACK timestamps, or the granularity of the timers, or both?
I could imagine that there could be some nice latency improvements from
using TLP and RACK at those low RTTs, as long as any timers that
were scheduled were rounded up appropriately (e.g. by 10ms-20ms).
But I haven't fully thought through all the implications of this kind
of scenario.

> > By "a previously unsent segment" we mean basically
> > "the next segment (of MSS or fewer bytes) that the sender
> > would normally send if it had available cwnd at this time

> This is better but still not crisp enough to determine what
> exactly the Linux implementation does. Since you said that
> Linux does not roll back SND.NXT upon RTO, does this imply
> that "a previously unsent segment", is the one starting at SND.NXT?

Yes, for the Linux TLP sender, "a previously unsent segment" is the one
starting at SND.NXT.

BTW, F-RTO RFC - https://tools.ietf.org/html/rfc5682 - in
section 2.1 2) b), in describing its similar "forward" probe packets,
uses the phrase "transmit up to two new (previously unsent) segments".
So for consistency with the F-RTO text I have tentatively
changed that line in our internal draft of -04 from:
  "Transmit that new segment"
to:
  "Transmit one new (previously unsent) segment"

It is probably worth noting here that in the Linux TCP stack the sender
would not send a TLP when in the middle of an RTO-triggered recovery.
This corresponds to one of the conditions in the draft, in section
"5.4.1. Phase 1: Scheduling a loss probe": the condition that says:
    " 2.  The connection is not in loss recovery"

So perhaps because of this the different SND.NXT "rewind" behavior of
different TCP stacks upon RTO may not matter ultimately? Or perhaps
there are more implications to unpack. :-)

> > By "the last segment" we mean "the highest-sequence segment
> > (of MSS or fewer bytes) that has already been transmitted
> > and not ACKed or SACKed

> Again given that SND.NXT is not rolled back this would presumably
> the first UN(S)ACKed MSS or fewer bytes walking back from SND.NXT
> (I am not suggesting an actual walk back, just for illustration).

Yes, exactly.

> I am a bit confused that you used the "or SACKed" since you
> previously required that PTO not be armed if any SACK blocks
> were present. If that is followed, then "the last segment" would
> pretty much always be MSS or fewer bytes just before
> (non-rolled back) SND.NXT.

Yes, you are exactly right. I should have left the "or SACKed" part out of
that line in my e-mail. :-)

By the way, regarding retransmitting the "last segment", this discussion
made me realize that AFAICT the draft did not yet discuss why the TLP
retransmission is the last segment. I have added some proposed text to our
internal draft of the -04 rev to try to describe the rationale:

"When the loss probe is a retransmission, the sender uses the
highest-sequence segment sent so far. This is in order to deal with the
retransmission ambiguity problem in TCP. Suppose a sender sends N segments,
and then retransmits the last segment (segment N) as a loss probe, and then
the sender receives a SACK for segment N. As long as the sender waits for
any required RACK reordering settling timer to then expire, it doesn't
matter if that SACK was for the original transmission of segment N or the
TLP retransmission; in either case the arrival of the SACK for segment N
provides evidence that the segments preceding segment N were likely lost."

Of course welcome comments/suggestions about this proposed paragraph as
well.

Thanks!

neal