Re: [tcpm] AD Review of draft-ietf-tcpm-hystartplusplus-09

[Neal Cardwell <[e-mail]>]

On Thu, Sep 8, 2022 at 10:03 PM Praveen Balasubramanian <
[e-mail]> wrote:

> Thanks for the review and the comments. I agree with Neal that burst
> control, pacing, and L limits should be separated from Hystart++, and
> should have their own RFC. One path forward would be to still refer to
> RFC3465 for context on ABC and origin of the L parameter but provide an
> updated recommendation for L based on deployment experience. Would the
> following updates be acceptable?
>
> In Section 4.2
>
> The following pseudocode integrates Appropriate Byte Counting as
>    described in [RFC3465 <https://datatracker.ietf.org/doc/html/rfc3465>]. See section 4.3 for an updated recommendation of variable L based on deployment experience.
>
>
I think Martin is suggesting that it would be nice to not have the
Hystart++ RFC depend on ABC in any direct way, and I like that direction.

What do folks think about something like the following text, which Martin
and I have been collaborating on:
"""
4.2. Algorithm Details

The following pseudocode uses a limit, L, to control the aggressiveness of
the cwnd increase during both standard slow start and CSS. While an
arriving ACK may newly acknowledge an arbitrary number of bytes, the
Hystart++ algorithm limits the number of those bytes applied to increase
the cwnd to L*MSS bytes.
"""


> In Section 4.3
>
> A paced TCP implementation SHOULD use L=Infinity. Burst concerns are mitigated by pacing and this setting allows for optimal cwnd growth on modern networks.
>
> A non-paced TCP implementation SHOULD use L >=1 and L <=8. L values larger than 8 can cause increase in burstiness and loss rates and result in poor performance.
>
>
This sounds good to me, although I would recommend "L >=2 and L <=8".

RFC3465 already says: "This document specifies that TCP implementations MAY
use L=2*SMSS bytes", so based on that and experience with Linux's
L=infinity I think it's definitely fine to have Hystart++ say stacks MAY
use L=2. I would argue that stacks SHOULD use at least 2, given delayed
ACKs.

In Section 4.3, how about something like:


"""

A paced TCP implementation SHOULD use L=Infinity. Burst concerns are
mitigated by pacing and this setting allows for optimal cwnd growth on
modern networks.


A non-paced TCP implementation SHOULD use L >=2 and L <=8. L values smaller
than 2 can suffer performance problems from the negative impact of the
standard delayed acknowledgment algorithm, which often sends one
acknowledgment per 2*SMSS bytes. L values larger than 8 can cause increases
in burstiness and loss rates and result in poor performance.
"""


> This draft is already updating slow start behavior from RFC 5681, so I
> think its within scope to update the recommendation on use of L based on
> deployment experience on the Internet and modern networks.
>
> Suggestions are welcome.
>

I agree it makes sense to consider an update to the recommendation on L
values, but I like Martin's idea to try to avoid dependencies on ABC, so
that ABC can be retired at some point.

cheers,
neal



>
> On Thu, Sep 8, 2022 at 2:24 PM Martin Duke <[e-mail]>
> wrote:
>
>> I am happy with that formulation, but want community consensus for the
>> numbers (1 and infinity)
>>
>> On Thu, Sep 8, 2022 at 2:12 PM <[e-mail]> wrote:
>>
>>> > On 8. Sep 2022, at 18:53, Martin Duke <[e-mail]> wrote:
>>> >
>>> > Hi Neal, thanks for the data.
>>> >
>>> > Given its wide deployment, I'm happy for the spec to allow L_max =
>>> infinity, assuming the community is fine with it.
>>> So would you be happy with:
>>>
>>> The positive integer L SHOULD be 1 and MAY be larger than 1.
>>>
>>> Best regards
>>> Michael
>>> >
>>> > I don't care that much, in theory, whether L limits are defined in
>>> this document or in another one. However, in practice adding about 2
>>> paragraphs to this document would allow us to obsolete 3465. If we can
>>> rapidly reach consensus on this, it seems like the expeditious thing to do.
>>> >
>>> > The alternative is to simply eliminate L from Hystart++, and we
>>> "really know" what people will do in production.
>>> >
>>> > On Thu, Sep 8, 2022 at 9:42 AM Neal Cardwell <[e-mail]>
>>> wrote:
>>> >
>>> >
>>> > On Wed, Sep 7, 2022 at 5:46 PM <[e-mail]> wrote:
>>> > > On 7. Sep 2022, at 22:44, Martin Duke <[e-mail]>
>>> wrote:
>>> > >
>>> > >
>>> > >
>>> > > On Wed, Sep 7, 2022 at 1:15 PM <[e-mail]> wrote:
>>> > > > On 7. Sep 2022, at 21:40, Martin Duke <[e-mail]>
>>> wrote:
>>> > > >
>>> > > > I would be happier with some sort of limit on L. Maybe: the
>>> positive integer L SHOULD be 2 and MUST be no more than 8(?) But whatever
>>> numbers the community is comfortable with are fine with me.
>>> > > Hi Martin,
>>> > >
>>> > > the crucial point here is that we know that MS used L = 8. But we
>>> don't have any information,
>>> > > whether other values of L are worse or better, what the tradeoffs
>>> are and in particular
>>> > > what traffic patterns impact the choice of L. Wouldn't we need such
>>> an analysis / results
>>> > > from an experiment for selecting an upper limit L_MAX? I would
>>> assume that it might
>>> > > have an impact whether pacing is used or not.
>>> > >
>>> > > Well the community has at least some data that L = 8 is safe on the
>>> internet. Maybe some other practitioners have data for 16 or 32 or
>>> whatever. It would be reasonable to set some sort of limit based on the
>>> limits of our empirical limit.
>>> > Sure. My point was just that the results provided my MS show that
>>> L_MAX <= 8 seems to
>>> > be safe in the scenario they looked at. But there is no statement that
>>> L_MAX > 8 is
>>> > bad...
>>> > >
>>> > > It would be fine to say "one SHOULD NOT exceed L = foo if you're not
>>> pacing"
>>> > >
>>> > >
>>> > > >
>>> > > > I'd also be happier with a cut-and-paste of those considerations
>>> from 3465, which should not be a lot of work.
>>> > > >
>>> > > > The lower-effort approach is just to strike L from the document
>>> and let people do what they've doing, which is use Ls well in excess of 2.
>>> But I'd prefer we'd actually write it down because
>>> > > > (1) our standards are better if they reflect widespread behavior
>>> in the internet; and
>>> > > I agree with the above.
>>> > > > (2) it means we can finally make 3465 historic, as we have
>>> standards-track documents that cover virtually all of its material.
>>> > > >
>>> > > > If the community really cannot converge on sensible limits for L,
>>> then I'd rather publish with no L than wait for a deadlock to resolve.
>>> > > In my view, the selection of L is not part of the core of the
>>> document.
>>> > >
>>> > > This document standardizes a new slow start algorithm, which
>>> includes L as a parameter, for which there is no standard.
>>> > My point was that there is no L_MAX given here.
>>> > >
>>> > > >
>>> > > > But as a way forward, I'd suggest:
>>> > > > 1) the authors propose text that sets bounds on L (a SHOULD and a
>>> MUST) and cut-and-pastes the considerations.
>>> > > Aren't the consideration in RFC 3465 coming to the conclusion: L
>>> SHOULD be 1, MAY be 2, MUST NOT be larger than 2?
>>> > > How to use the same considerations to come to a different
>>> conclusion: L SHOULD be 2, MUST NOT be larger than L_MAX
>>> > > for L_MAX > 2?
>>> > >
>>> > > Re: SHOULD 1 or 2, I don't have a strong opinion but I feel like the
>>> 3465 experiment shows that 2 is safe, and somewhat beneficial given the
>>> prevalence of ack thinning? If there is no consensus for this, I'm happy to
>>> go with 1.
>>> > >
>>> > > L_MAX: My preference is to set a value that is widely deployed with
>>> good data that it is safe to do so
>>> > Let us see if people provide data which values are good and which
>>> values are bad...
>>> >
>>> > FWIW, Linux TCP has been using L_MAX=infinity since 2013:
>>> >
>>> >
>>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9f9843a751d0a2
>>> >
>>> > IMHO trying to limit bursts with an L parameter is misguided:
>>> >
>>> > + AN L limit is only a partial solution; TCP RFCs still allow massive
>>> line-rate bursts of cwnd when restarting from idle, which is super-common
>>> in some very common Internet workloads: web traffic, streaming video, RPC.
>>> >
>>> > + To avoid such bursts, whether restarting from idle or not, TCP
>>> stacks should enable pacing.
>>> >
>>> > + Once pacing is enabled, having an L limit purely shoots your flow in
>>> the foot, causing it to fail to double cwnd each round trip in slow start
>>> in the very common case of the many ubiquitous aggregation mechanisms that
>>> cause ACKs to arrive for dozens of packets at a time (e.g.. Linux GRO/LRO
>>> commonly aggregate 45 packets into a single unit that is processed and
>>> ACKed as a unit).
>>> >
>>> > IMHO ideally burst control, pacing, and L limits should be separated
>>> from Hystart++, and should have their own RFC.
>>> >
>>> > And whatever is done, at a minimum Hystart++ should not depend on
>>> RFC3465 (ABC) (though it could mention it in passing to provide context).
>>> >
>>> > my two cents,
>>> > neal
>>> >
>>> > _______________________________________________
>>> > tcpm mailing list
>>> > [e-mail]
>>> > https://www.ietf.org/mailman/listinfo/tcpm
>>>
>>>