Re: [tcpm] draft-ietf-tcpm-rack-05 review

[Yuchung Cheng <[e-mail]>]

Hi Emmanuel,

Sorry for the late reply and thanks for testing RACK. Some questions:
1) what's the congestion control? I assume it's the default (i.e. Cubic)

2) what's the value of sysctl net.ipv4.tcp_no_metrics_save? I assume it's
default 0 meaning TCP save some metrics to be reused by future flows
towards same IP

3) what's the bandwidth of the bottleneck, or is BDP generally beyond the
5000 packets in your test

If my assumptions above are correct, I speculate that the performance gap
is caused by spurious early exit TCP Cubic slow-start. This is caused by
RACK not catching the some reordering well initially with its init RTT/4
reordering window. Many RACK parameters are tuned based on Google's traffic
where nominal RTT is much lower than 500ms.

In contrast, the Linux dupack implementation (referred by 3DUPACK in your
graph) has two features:
a) it dynamically extends the dupthresh to a maximum of 300 packets (sysctl
net.ipv4.tcp_max_reordering), hence makes TCP very resilient if only OOO is
reordering not drops.
b) the high dupthresh is cached and reused for new connection toward the
same dst ip by (2).
Thus it's possible all subsequent flows in one test are starting with
dupthresh to the highest possible, making it bullet-proof to any reordering
by never entering fast recovery falsely.

To check my theory, could you provide some tcpdump of each test (RACK vs
non-RACK)? only header captures are needed.



On Sat, Sep 7, 2019 at 4:03 AM Emmanuel Lochin <
[e-mail]> wrote:

> Hi Yuchung,
>
> Thanks for your answer, I've asked this question as we observed a decrease
> of the performance of TCP transfer in some reordering scenario when RACK is
> enabled. In brief, the problem has been solved by disabling RACK.
> I am involved in a CNES project (French space agency) where SATCOM systems
> are used (GEO or LEO) and in particular I work with Nicolas Kuhn from CNES
> (in copy of this email) .
>
> To assess whether or not the problem effectively came from RACK, we have
> started a bunch of experiments using Mininet or simply using the loopback
> interface. The results presented below are done with Mininet except the
> last one done in localhost
>
> The x-axis reports the size of the flows, basically we send 5 TCP flows in
> the same time of a given size and we repeat the experiments hundred of
> times over a Netem queue. We then compute the following statistics over
> these 500 flows for each flow length.
>
> There is no artificial losses introduced with Netem, we only use it only
> to reorder packets. The nominal RTT is 500ms (equiv. to a GEO system) where
> some packets are reordered every 5, 10, 15 or 20 packets sent. These
> reordered packets have then an RTT=250ms. As you can see, in all cases when
> RACK is enabled, the completion time (time difference between the first
> packet received at destination and the last one) is always worse with RACK.
> The kernel is 4.15.0-60-generic.
>
>
>
> We thought, may be, it was due to the reordering pattern which is too
> deterministic. We thus applied several other random patterns.
> Below we present the Netem config documented in the TC manpage for
> simplicity (i.e. reorder 25% 50% gap 5 and 10)
>
>
> But results obtained where the same.
>
> In all these previous experiments the reorder delay is 50% of the RTT, and
> indeed, the RTT is important.
> If we decrease it to 100ms, when the reorder delay corresponds to 90% of
> the RTT (meaning that RTT=100ms and reordered packets get 90ms), in that
> case, RACK performs much more better as shown below.
>
>
>
>
> Using the loopback interface, even with low RTT (i.e. 100ms), and when the
> reordering delay is half the RTT (50ms), RACK still does not perform always
> better than 3DUPACK.
>
>
> We really believe that RACK is a good idea, but it seems that the kernel
> implementation does not lead to what we expected. Do you have experienced
> such behavior or do you have measurements that thwart mine ?
>
> Thanks
>
> Emmanuel
>
>
>
> Le 07/09/2019 à 00:32, Yuchung Cheng a écrit :
>
> Hi Emmanuel,
>
> Yes setting the sysctl to 0 will completely disable RACK. Although by
> default it supports DUPTHRESH-based detection (as a simple extension)
> beside time-based detection. You can disable that extension w/ 0x4. But if
> you want absolutely old-style pure DUPTHRESH-based then you can set that to
> 0.
>
>
>
> On Fri, Sep 6, 2019 at 3:24 AM Emmanuel Lochin <
> [e-mail]> wrote:
>
>> Hi Yuchung, all,
>>
>> I have a question concerning the GNU/Linux kernel implementation of RACK.
>> There are 3 sysctl options to enable RACK (0x1 0x2 and 0x4) but none to
>> disable it (or may be 0x0 is not documented). As tcp_input.c implements
>> this boolean function:
>>
>> static bool tcp_is_rack(const struct sock *sk)
>> {
>>         return sock_net(sk)->ipv4.sysctl_tcp_recovery &
>> TCP_RACK_LOSS_DETECTION;
>> }
>>
>> can I consider that I can disable RACK by typing sysctl -w
>> net.ipv4.tcp_recovery=0 ?
>> I ask this question because I've seen several update done inside the
>> kernel to implement it and I want to be sure that I can really go back to
>> 3DUPACK scheme this way.
>>
>> Cheers,
>>
>> Emmanuel
>>
>> Le 05/09/2019 à 04:44, Martin Duke a écrit :
>>
>> I think the proposed text is fine.
>>
>> On Wed, Sep 4, 2019 at 5:55 PM Yuchung Cheng <[e-mail]> wrote:
>>
>>> On Tue, Sep 3, 2019 at 3:31 PM Martin Duke <[e-mail]>
>>> wrote:
>>>
>>> <snip>
>>
>>
>>> > (3) sec 7.3
>>> >
>>> > We have evaluated using the smoothed RTT (SRTT from
>>> >    [RFC6298] RTT estimation) or the most recently measured RTT
>>> >    (RACK.rtt) using an experiment similar to that in the Performance
>>> >    Evaluation section.  They do not make any significant difference in
>>> >    terms of total recovery latency.
>>> >
>>> > If there is truly no difference, then why not use SRTT as the standard?
>>> > Every TCP implementation has to store this, while min_rtt is unneeded
>>> for many (most?) congestion controls.
>>> >
>>> > Alternatively, you could strengthen this paragraph to not sound like
>>> it makes no difference..
>>> That's a good point -- our experiment at Google servers indeed didn't
>>> show much difference. But I think it's still better to use min_RTT
>>> than SRTT. On buffer-bloat friendly C.C. the SRTT could be orders of
>>> magnitude longer than the actual paths' RTT -- in my opinion factoring
>>> this network queuing delay in reordering window is not a good idea. So
>>> how about adding
>>>
>>> 'While the experiment does not show a difference between min RTT and
>>> SRTT, SRTT is less desirable to size the reordering window as it
>>> includes network congestion or delayed ACKs effects."
>>>
>>>
>>>
>> _______________________________________________
>> tcpm mailing [e-mail]://www.ietf.org/mailman/listinfo/tcpm
>>
>>
>> --
>> Emmanuel LOCHIN
>> Professeur ISAE
>> ISAE SUPAERO - Institut Supérieur de l'Aéronautique et de l'Espace
>> 10 avenue Edouard Belin - BP 54032 - 31055 TOULOUSE CEDEX 4 FRANCE -http://www.isae-supaero.fr
>> Tel [phone] <+33%205%2061%2033%2084%2085> - Fax [phone] <+33%205%2061%2033%2083%2045>
>> Web : http://personnel.isae.fr/emmanuel-lochin/
>>
>>
> --
> Emmanuel LOCHIN
> Professeur ISAE
> ISAE SUPAERO - Institut Supérieur de l'Aéronautique et de l'Espace
> 10 avenue Edouard Belin - BP 54032 - 31055 TOULOUSE CEDEX 4 FRANCE -http://www.isae-supaero.fr
> Tel [phone] <+33%205%2061%2033%2084%2085> - Fax [phone] <+33%205%2061%2033%2083%2045>
> Web : http://personnel.isae.fr/emmanuel-lochin/
>
>