Re: [iccrg] [tcpm] A description of Linux pacing

[Martin Duke <[e-mail]>]

Code is great, but for licensing reasons many developers cannot simply look
at the Linux code and port it over to their platform.

On Fri, Mar 22, 2024 at 1:07 PM Yuchung Cheng <ycheng=
[e-mail]> wrote:

> my 2c: Linux is open source. nothing is better than the code itself to
> fully understand how it works in the most current form.
>
> A BCP on pacing design could be informational on pros and cons of various
> SW/HW approaches on pacing. But it should not be Linux-specific or
> generally implementation specific. In that regard, TSO/GRO BCP seems more
> critical than pacing, as I continue to see HW or smart-NIC support has bugs.
>
> On Fri, Mar 22, 2024 at 12:52 PM Scharf, Michael <
> [e-mail]> wrote:
>
>> Hi all,
>>
>>
>>
>> While this is probably not really useful to understand the current
>> state-of-the-art, here is an old pointer:
>>
>>
>>
>> Almost 17 years ago there was a TSVAREA presentation about an experiment
>> with pacing in the Linux kernel:
>> https://www.ietf.org/proceedings/69/slides/tsvarea-3.pdf
>>
>>
>>
>> (More details were published later in
>> https://content.ikr.uni-stuttgart.de/en/Content/Publications/Archive/Sf_QuickStartTCP-LNCS_36636.pdf
>> )
>>
>>
>>
>> Quick-Start TCP was IMHO one of the first proposals that mandated pacing
>> (only in the first RTT). At least in the Linux kernel we were not aware of
>> much related work on pacing back then.
>>
>>
>>
>> Well, this is very old stuff. But +1 that documenting the current
>> state-of-the-art could make sense.
>>
>>
>>
>> Michael
>>
>>
>>
>> *From:* tcpm <[e-mail]> *On Behalf Of *Michael Welzl
>> *Sent:* Friday, March 22, 2024 7:47 AM
>> *To:* Yuchung Cheng <ycheng=[e-mail]>
>> *Cc:* Eric Dumazet <[e-mail]>; [e-mail]; [e-mail]
>> *Subject:* Re: [tcpm] A description of Linux pacing
>>
>>
>>
>> This is useful indeed, many thanks for the pointer!  (*I* knew about it,
>> but still).
>>
>>
>>
>> This said, I found this reference, and others, to be too "high-level" for
>>  my own purposes. I wanted to understand what I see at a finer detail, and
>> I think that this is the kind of documentation that’s missing IMO.  (e.g.,
>> the initial burst limit of 10 (not IW, but hard-coded 10) packets, which
>> took me by surprise, and will probably surprise others who were unaware;
>> how micro bursts are used to implement a rate at a ~ 1ms granularity, with
>> upper and lower bounds; the divergence from this for hosts that are less
>> than 3ms away, …).
>>
>>
>>
>> Cheers,
>>
>> Michael
>>
>>
>>
>>
>>
>>
>>
>> On Mar 20, 2024, at 7:55 PM, Yuchung Cheng <
>> ycheng=[e-mail]> wrote:
>>
>>
>>
>> Neal and I have a short paper on Linux pacing, which integrates TSQ, and
>> TSO autosizing. it has evolved over the years but the core design remains.
>>
>> https://netdevconf.info/1.2/papers/bbr-netdev-1.2.new.new.pdf
>>
>>
>>
>> On Wed, Mar 20, 2024 at 12:42 AM Michael Welzl <[e-mail]>
>> wrote:
>>
>> Hi,
>>
>>
>>
>> and thanks for your feedback!
>>
>>
>>
>> I agree; I can write this up in time for the Vancouver IETF. As for
>> documenting other implementations, I don’t know internals of any others…
>> but I’m very willing to involve others in such an I-D - if any expert on a
>> different OS is interested, get in touch!
>>
>>
>>
>> Else, I could just make a start with this and see where it goes. It’s not
>> much work, I can keep it short and simple.
>>
>>
>>
>> Cheers,
>>
>> Michael
>>
>>
>>
>>
>>
>> On 12 Mar 2024, at 23:08, Martin Duke <[e-mail]> wrote:
>>
>>
>>
>> Speaking as an individual, it's a bummer we don't have a real RFC about
>> pacing, this would be a decent Informational RFC for a way to do pacing,
>> that is also a useful reference for how an important implementation does it.
>>
>>
>>
>> Even more ambitious, one could document the approaches of several major
>> implementations.
>>
>>
>>
>> On Mon, Feb 19, 2024 at 6:06 AM Michael Welzl <[e-mail]> wrote:
>>
>> Dear TCPM and ICCRG (I assume that there’s so much overlap with people in
>> CCWG that it would just be spamming to send it there too?),
>>
>>
>>
>> Over the last two weeks or so, I have put some effort into trying to
>> understand how Linux pacing *really* works  (I had some descriptions that
>> were somewhat high-level; I wanted to obtain a more precise lower-level
>> understanding).
>>
>> I wrote a document, just for myself, explaining what goes on in the code,
>> as I can’t even try to follow the Linux kernel without taking notes.
>>
>>
>>
>> After a first iteration, I shared it with the bufferbloat mailing list,
>> in the hope of getting corrections.
>>
>> I did!  Most notably (but not only), Neal Cardwell helped me a ton - and
>> now the document should be quite thorough and hopefully correct.
>>
>>
>>
>> While I only did this for myself and just asked the list for help,
>> several people have in the meantime told me that this document is actually
>> valuable for the community - and so I thought I should share it here, too.
>>
>> It lives as a Google doc at this URL:
>>
>>
>> https://docs.google.com/document/d/1-uXnPDcVBKmg5krkG5wYBgaA2yLSFK_kZa7xGDWc7XU/edit?usp=sharing
>>
>>
>>
>> Comments or fixes are very welcome!
>>
>> Please feel free to forward this as you want.
>>
>>
>>
>> I know that Google docs is not the format that we people here normally
>> use  :-)   well, if someone thinks that it would indeed be useful to write
>> this up as an I-D (probably skipping all the code details though), please
>> let me know - I can do that, and I can also present it if there’s interest
>> (but in Vancouver, not Brisbane).
>>
>>
>>
>> Cheers,
>>
>> Michael
>>
>>
>>
>> _______________________________________________
>> tcpm mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/tcpm
>>
>>
>>
>> _______________________________________________
>> tcpm mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/tcpm
>>
>> _______________________________________________
>> tcpm mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/tcpm
>>
>>
>>
> _______________________________________________
> iccrg mailing list
> [e-mail]
> https://mailman.irtf.org/mailman/listinfo/iccrg
>