Re: [iccrg] [tcpm] A description of Linux pacing

[Yuchung Cheng <[e-mail]>]

my 2c: Linux is open source. nothing is better than the code itself to
fully understand how it works in the most current form.

A BCP on pacing design could be informational on pros and cons of various
SW/HW approaches on pacing. But it should not be Linux-specific or
generally implementation specific. In that regard, TSO/GRO BCP seems more
critical than pacing, as I continue to see HW or smart-NIC support has bugs.

On Fri, Mar 22, 2024 at 12:52 PM Scharf, Michael <
[e-mail]> wrote:

> Hi all,
>
>
>
> While this is probably not really useful to understand the current
> state-of-the-art, here is an old pointer:
>
>
>
> Almost 17 years ago there was a TSVAREA presentation about an experiment
> with pacing in the Linux kernel:
> https://www.ietf.org/proceedings/69/slides/tsvarea-3.pdf
>
>
>
> (More details were published later in
> https://content.ikr.uni-stuttgart.de/en/Content/Publications/Archive/Sf_QuickStartTCP-LNCS_36636.pdf
> )
>
>
>
> Quick-Start TCP was IMHO one of the first proposals that mandated pacing
> (only in the first RTT). At least in the Linux kernel we were not aware of
> much related work on pacing back then.
>
>
>
> Well, this is very old stuff. But +1 that documenting the current
> state-of-the-art could make sense.
>
>
>
> Michael
>
>
>
> *From:* tcpm <[e-mail]> *On Behalf Of *Michael Welzl
> *Sent:* Friday, March 22, 2024 7:47 AM
> *To:* Yuchung Cheng <ycheng=[e-mail]>
> *Cc:* Eric Dumazet <[e-mail]>; [e-mail]; [e-mail]
> *Subject:* Re: [tcpm] A description of Linux pacing
>
>
>
> This is useful indeed, many thanks for the pointer!  (*I* knew about it,
> but still).
>
>
>
> This said, I found this reference, and others, to be too "high-level" for
>  my own purposes. I wanted to understand what I see at a finer detail, and
> I think that this is the kind of documentation that’s missing IMO.  (e.g.,
> the initial burst limit of 10 (not IW, but hard-coded 10) packets, which
> took me by surprise, and will probably surprise others who were unaware;
> how micro bursts are used to implement a rate at a ~ 1ms granularity, with
> upper and lower bounds; the divergence from this for hosts that are less
> than 3ms away, …).
>
>
>
> Cheers,
>
> Michael
>
>
>
>
>
>
>
> On Mar 20, 2024, at 7:55 PM, Yuchung Cheng <
> ycheng=[e-mail]> wrote:
>
>
>
> Neal and I have a short paper on Linux pacing, which integrates TSQ, and
> TSO autosizing. it has evolved over the years but the core design remains.
>
> https://netdevconf.info/1.2/papers/bbr-netdev-1.2.new.new.pdf
>
>
>
> On Wed, Mar 20, 2024 at 12:42 AM Michael Welzl <[e-mail]> wrote:
>
> Hi,
>
>
>
> and thanks for your feedback!
>
>
>
> I agree; I can write this up in time for the Vancouver IETF. As for
> documenting other implementations, I don’t know internals of any others…
> but I’m very willing to involve others in such an I-D - if any expert on a
> different OS is interested, get in touch!
>
>
>
> Else, I could just make a start with this and see where it goes. It’s not
> much work, I can keep it short and simple.
>
>
>
> Cheers,
>
> Michael
>
>
>
>
>
> On 12 Mar 2024, at 23:08, Martin Duke <[e-mail]> wrote:
>
>
>
> Speaking as an individual, it's a bummer we don't have a real RFC about
> pacing, this would be a decent Informational RFC for a way to do pacing,
> that is also a useful reference for how an important implementation does it.
>
>
>
> Even more ambitious, one could document the approaches of several major
> implementations.
>
>
>
> On Mon, Feb 19, 2024 at 6:06 AM Michael Welzl <[e-mail]> wrote:
>
> Dear TCPM and ICCRG (I assume that there’s so much overlap with people in
> CCWG that it would just be spamming to send it there too?),
>
>
>
> Over the last two weeks or so, I have put some effort into trying to
> understand how Linux pacing *really* works  (I had some descriptions that
> were somewhat high-level; I wanted to obtain a more precise lower-level
> understanding).
>
> I wrote a document, just for myself, explaining what goes on in the code,
> as I can’t even try to follow the Linux kernel without taking notes.
>
>
>
> After a first iteration, I shared it with the bufferbloat mailing list, in
> the hope of getting corrections.
>
> I did!  Most notably (but not only), Neal Cardwell helped me a ton - and
> now the document should be quite thorough and hopefully correct.
>
>
>
> While I only did this for myself and just asked the list for help, several
> people have in the meantime told me that this document is actually valuable
> for the community - and so I thought I should share it here, too.
>
> It lives as a Google doc at this URL:
>
>
> https://docs.google.com/document/d/1-uXnPDcVBKmg5krkG5wYBgaA2yLSFK_kZa7xGDWc7XU/edit?usp=sharing
>
>
>
> Comments or fixes are very welcome!
>
> Please feel free to forward this as you want.
>
>
>
> I know that Google docs is not the format that we people here normally use
>  :-)   well, if someone thinks that it would indeed be useful to write this
> up as an I-D (probably skipping all the code details though), please let me
> know - I can do that, and I can also present it if there’s interest (but in
> Vancouver, not Brisbane).
>
>
>
> Cheers,
>
> Michael
>
>
>
> _______________________________________________
> tcpm mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/tcpm
>
>
>
> _______________________________________________
> tcpm mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/tcpm
>
> _______________________________________________
> tcpm mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/tcpm
>
>
>