Re: [tcpm] Thank you for the QUIC session in tcpm

[Neal Cardwell <[e-mail]>]

On Wed, Nov 14, 2018 at 11:18 PM Jana Iyengar <[e-mail]> wrote:

> Thanks for the clarification, Eric. I'm a bit surprised to see srtt be
> used by both Cubic and BBR, but that discussion's not for this thread.
>

Just to add context (sorry to dilute the thread!), the spot in Linux TCP
BBR that Eric pointed to that is using srtt is the initialization code
(bbr_init_pacing_rate_from_rtt()). The code is using srtt here to compute a
pacing rate (via pacing_rate = gain * cwd / srtt) largely because Linux TCP
can hot-swap congestion control modules. Because of that possibility, the
cwnd and srtt that BBR inherits at initialization may have been from a
buffer-bloated CUBIC/Reno connection, where srtt may have bloated in
proportion to cwnd, so that cwd / srtt is generally a better approximation
to the available bandwidth than cwnd / min_rtt.  As you would imagine,
after that initialization, Linux TCP BBR does not use srtt (or min_rtt) to
update the pacing rate; it uses a function of the estimated bandwidth.

best,
neal



> On Wed, Nov 14, 2018 at 9:44 PM Eric Dumazet <[e-mail]>
> wrote:
>
>>
>>
>> On 11/12/2018 11:13 PM, Jana Iyengar wrote:
>> > On what to do with an unconfirmed RTO: So, my thought is that if you've
>> hit an RTO, it's been a REALLY LONG time, because we've gone through TLP(s)
>> by now and failed. (This might change if we decide to combine TLP and RTO,
>> but that's for a later discussion.)  It's not fully clear to me what the
>> right responses here are, but some thoughts follow.
>> >
>> > 1/ If this was an actual increase in the RTT, it would naturally roll
>> into the SRTT and RTTVAR. We might even consider resetting the RTT
>> estimator, but we already know that the EWMA estimator we love takes just a
>> while to catch on, and I don't want to litigate changing the estimator
>> fundamentally here.
>> >
>> > 2/ Linux fq uses min_rtt IIRC, which shouldn't change as a result of an
>> increase in RTT. This *might* be a good time to kick the min_rtt estimator
>> to pace more conservatively.
>>
>> linux fq does not use min_rtt, but the pacing rate set by TCP.
>>
>> pacing rate for Cubic is using SRTT (
>> https://elixir.bootlin.com/linux/latest/source/net/ipv4/tcp_input.c#L806
>> )
>>
>> For BBR, pacing rate depends on various factors and phases, but SRTT is
>> used, not min_rtt
>>
>> https://elixir.bootlin.com/linux/latest/source/net/ipv4/tcp_bbr.c#L231
>>
>>
>> >
>> > 3/ I'm not convinced that it makes sense to change the cwnd. It seems
>> clear that an RTT increase with the same BW should increase the cwnd if
>> anything, so it seems reasonable to continue with the same cwnd at least.
>> If the BW changed, hopefully that'll be reflected in a congestion signal,
>> but in the absence of a congestion signal, I think we should keep the cwnd
>> as is.
>> >
>> >
>> > On Tue, Nov 13, 2018 at 12:35 AM Praveen Balasubramanian <
>> [e-mail] <mailto:[e-mail]>> wrote:
>> >
>> >     Yuchung this might be worth experimenting with and getting more
>> data on. Maybe there should be some reaction to an unconfirmed RTO even if
>> it is not reduction all the way to LW? I like Jana's suggestion that the
>> implementation SHOULD drop the cwnd even for unconfirmed RTO if not pacing.
>> >
>> >     I checked the latest draft and pacing is a RECOMMENDED. I am
>> noticing that transport algorithms in Linux and in QUIC are evolving
>> assuming pacing which is a function usually implemented and configured
>> outside of the transport.. So the transport is potentially making an
>> assumption which may result in safety issues. For example BBR congestion
>> control may not work very well if pacing was disabled or misconfigured? I
>> don’t know of any robust solution to this other than build pacing into the
>> transport itself. Pacing is also very challenging for low latency flows
>> because of lack of support for fine grained timers.
>> >
>> >     -----Original Message-----
>> >     From: Yuchung Cheng <[e-mail] <mailto:[e-mail]>>
>> >     Sent: Monday, November 12, 2018 8:37 AM
>> >     To: Jana Iyengar <[e-mail] <mailto:[e-mail]>>
>> >     Cc: Praveen Balasubramanian <[e-mail] <mailto:
>> [e-mail]>>; Bob Briscoe <[e-mail] <mailto:
>> [e-mail]>>; Ian Swett <[e-mail] <mailto:
>> [e-mail]>>; [e-mail] <mailto:[e-mail]> Extensions <
>> [e-mail] <mailto:[e-mail]>>
>> >     Subject: Re: [tcpm] Thank you for the QUIC session in tcpm
>> >
>> >     On Mon, Nov 12, 2018 at 4:13 AM, Jana Iyengar <[e-mail]
>> <mailto:[e-mail]>> wrote:
>> >     > Praveen,
>> >     >
>> >     > The point you're raising -- that we've lost the ack clock after
>> an RTO
>> >     > -- is a reasonable point. My argument is that with pacing, cwnd
>> >     > reduction is unwarranted because in the extreme case, this
>> collapses
>> >     > to restart after idle, where sending at cwnd with pacing is
>> reasonable
>> >     I agree w/ Jana as well - the most generic way to treat burst due
>> to prior_inflight << cwnd is pacing. With QUIC's approach, when RTO fires,
>> cwnd remains unchanged until the ACK of the first retransmission (i.e. a
>> probe packet) comes back. Therefore the delay is one RTT and the potential
>> damage is an additional cwnd-worth of burst.
>> >
>> >     Yes the worst case is more aggressive, and it can be too much for
>> DC incast case if pacing isn't supported - one idea is to selective enable
>> that if RTT variance is very large vs RTT.
>> >
>> >     >
>> >     > The draft does not say that a sender should reduce the cwnd if it
>> is
>> >     > not pacing, which we should add. Does that make sense?
>> >     >
>> >     > - jana
>> >     >
>> >     > On Fri, Nov 9, 2018 at 8:34 AM Praveen Balasubramanian
>> >     > <pravb=[e-mail] <mailto:
>> [e-mail]>> wrote:
>> >     >>
>> >     >> Yuchung I brought that difference up in an email to the quic wg
>> >     >> earlier this week.
>> >     >>
>> >     >> In app send limited case, inflight could be very small compared
>> to cwnd.
>> >     >> So in QUIC there is potential to send a burst out after a long
>> idle
>> >     >> period (with outstanding data) where TCP wouldn't. The draft
>> claims
>> >     >> this is okay to do because RTO may have been a result of RTT
>> increase
>> >     >> instead of loss. Is there data to suggest on which side we should
>> >     >> err? i.e. data on what are the chances that an RTO is due to an
>> RTT increase versus loss.
>> >     >>
>> >     >> Do you see any safety concerns with delayed reduction of cwnd in
>> case
>> >     >> where the RTO is not spurious?
>> >     >>
>> >     >> -----Original Message-----
>> >     >> From: tcpm <[e-mail] <mailto:[e-mail]>>
>> On Behalf Of Yuchung Cheng
>> >     >> Sent: Thursday, November 8, 2018 4:38 PM
>> >     >> To: Bob Briscoe <[e-mail] <mailto:[e-mail]
>> >>
>> >     >> Cc: Ian Swett <[e-mail] <mailto:[e-mail]>>;
>> tcpm IETF list <[e-mail] <mailto:[e-mail]>>
>> >     >> Subject: Re: [tcpm] Thank you for the QUIC session in tcpm
>> >     >>
>> >     >> On Thu, Nov 8, 2018 at 3:14 AM, Bob Briscoe <[e-mail]
>> <mailto:[e-mail]>> wrote:
>> >     >> >
>> >     >> > I just wanted to thank Jana for explaining QUIC loss recovery
>> to us
>> >     >> > (and QUIC CC as far as it goes).
>> >     >> > And thank you Jana, Ian, the chairs of both WGs (and anyone
>> else
>> >     >> > involved) for setting it up.
>> >     >> >
>> >     >> > If one is not full-time on QUIC, it's very difficult to keep up
>> >     >> > with all the changes. But now we have a checkpoint to start
>> from, I
>> >     >> > feel I will not be wasting people's time if I try to get
>> involved -
>> >     >> > at least I only might say something un-QUIC occasionally,
>> rather
>> >     >> > than nearly always. This has allowed people who understand how
>> TCP
>> >     >> > cold be improved to help with QUIC, when working on QUIC isn't
>> their day job.
>> >     >> >
>> >     >> > Again, Thank you.
>> >     >> I like particularly that QUIC only reduces cwnd to 1 after the
>> loss
>> >     >> is confirmed not upon RTO fires. It should be very feasible for
>> TCP
>> >     >> (at least
>> >     >> Linux) w/ TCP timestamps. It'll save a lot of spurious cwnd
>> reductions!
>> >     >>
>> >     >> Also IMHO TCP w/ quality timestamps are almost as good as QUIC
>> pkt-ids.
>> >     >> Google internally uses usec. We wish we could upstream it but RFC
>> >     >> needs to be updated.
>> >     >>
>> >     >> >
>> >     >> >
>> >     >> > Bob
>> >     >> >
>> >     >> >
>> >     >> > --
>> >     >> >
>> ________________________________________________________________
>> >     >> > Bob Briscoe
>> >     >> >
>> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbob
>> >     >> > briscoe.net <http://briscoe.net>%2F&amp;data=02%7C01%7Cpravb%
>> 40microsoft.com <http://40microsoft.com>%7C3c4d22866
>> >     >> >
>> 8444691c16808d648bd2ce9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%
>> >     >> >
>> 7C636776374678321324&amp;sdata=T%2BWTEc42VC%2Bz%2BsmMFyjlm37hmAwfee
>> >     >> > buPcuMYlVhgDY%3D&amp;reserved=0
>> >     >> >
>> >     >> > _______________________________________________
>> >     >> > tcpm mailing list
>> >     >> > [e-mail] <mailto:[e-mail]>
>> >     >> >
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fww
>> >     >> > w.i
>> >     >> > etf.org <http://etf.org
>> >%2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40micr
>> >     >> > oso
>> >     >> > ft.com <http://ft.com
>> >%7Cf0911eeb74d7446f424508d645dbb779%7C72f988bf86f141af91ab2d7
>> >     >> > cd0
>> >     >> >
>> 11db47%7C1%7C0%7C636773207316711149&amp;sdata=K667a3IQG4rarQ%2FOfAl
>> >     >> > yhK
>> >     >> > QQ05Cea421rgb64DlEMvs%3D&amp;reserved=0
>> >     >>
>> >     >> _______________________________________________
>> >     >> tcpm mailing list
>> >     >> [e-mail] <mailto:[e-mail]>
>> >     >>
>> >     >>
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
>> >     >> ietf.org <http://ietf.org
>> >%2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40micro
>> >     >> soft.com <http://soft.com
>> >%7C3c4d228668444691c16808d648bd2ce9%7C72f988bf86f141af91ab2d7
>> >     >>
>> cd011db47%7C1%7C0%7C636776374678321324&amp;sdata=33WxKh5c6qL4ln5jerpx
>> >     >> Rhytfrj1iTK284pEqv5fWKY%3D&amp;reserved=0
>> >     >>
>> >     >> _______________________________________________
>> >     >> tcpm mailing list
>> >     >> [e-mail] <mailto:[e-mail]>
>> >     >>
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
>> >     >> ietf.org <http://ietf.org
>> >%2Fmailman%2Flistinfo%2Ftcpm&amp;data=02%7C01%7Cpravb%40micro
>> >     >> soft.com <http://soft.com
>> >%7C3c4d228668444691c16808d648bd2ce9%7C72f988bf86f141af91ab2d7
>> >     >>
>> cd011db47%7C1%7C0%7C636776374678321324&amp;sdata=33WxKh5c6qL4ln5jerpx
>> >     >> Rhytfrj1iTK284pEqv5fWKY%3D&amp;reserved=0
>> >
>> >
>> >
>> > _______________________________________________
>> > tcpm mailing list
>> > [e-mail]
>> > https://www.ietf.org/mailman/listinfo/tcpm
>> >
>>
>> _______________________________________________
>> tcpm mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/tcpm
>>
> _______________________________________________
> tcpm mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/tcpm
>