[v6ops] Re: [EXTERNAL] Re: Fwd: The V6OPS WG has placed draft-cc-v6ops-wlcg-flow-label-marking in state "Call For Adoption By WG Issued"

[Tim Chown <[e-mail]>]

Hi Tom,

From: Tom Herbert <tom=[e-mail]>
Date: Thursday, 15 August 2024 at 01:19
To: Brian E Carpenter <[e-mail]>
Cc: Tommy Jensen <[e-mail]>, IPv6 Operations <[e-mail]>
Subject: [v6ops] Re: [EXTERNAL] Re: Fwd: The V6OPS WG has placed draft-cc-v6ops-wlcg-flow-label-marking in state "Call For Adoption By WG Issued"
On Wed, Aug 14, 2024 at 4:16 PM Brian E Carpenter
<[e-mail]> wrote:
>
> Tom,
>
> On 15-Aug-24 10:56, Tom Herbert wrote:
> > On Wed, Aug 14, 2024 at 3:25 PM Brian E Carpenter
> > <[e-mail]> wrote:
> >>
> >> Hi,
> >>
> >> I am very strongly convinced that this draft should be published as an RFC. I am on the fence whether it should be published as an IETF stream Informational RFC or as an Independent stream Informational RFC. Since we know that most people don't read the boilerplate anyway, it hardly matters.
> >>
> >> Although this work bends the rules of RFC 6437, the implementers were (to my personal knowledge) aware of the issues from the start, allowed for some entropy bits, and most important have actually shown a use case for 20 largely underused bits in the IPv6 header. This is a use case that is worth publishing, and I can imagine it leading to reconsideration of some of the rules in RFC 6437. That's why it belongs in the RFC series.
> >
> > Hi Brian,
> >
> > I don't agree that the 20 bit flow label is underused. By default,
> > Linux will set a flow label in a packet using the full twenty bits of
> > entropy.
>
> I'm aware of that and glad of it.
>
> On the receive Linux path there are algorithms that assume
> > that the full twenty bits of entropy are available. For instance,
> > Receive Flow Steering (RFS) classifies packets by the hash that
> > includes the flow label. Usually, we configure at least 1024 hash
> > buckets (10 bits) so only getting five bits might create a bunch of
> > hash collisions. Hosts might also be using the flow hash for other
> > forms of filtering involving the flow label as well.
>
> I'm not sure what the use cases are for more than a handful of hash
> buckets.

Brian,

Take a look at RFS (https://lwn.net/Articles/382428/) The idea is
pretty simple, we get the tuple hash for received packets from the
device or computer in the host for every packet. The hash may include
the flow label as input. When we process the flow (e.g. TCP
connection), we note what CPU is handling the connection and create an
entry for that connection in rps_sock_flow_table which is the table
that maps a hash to a CPU (specifically to an RX queue associated with
that CPU). On subsequent packets we use the table to "steer" packets
to the queue for the CPU the flow is being processed on, thereby
creating affinity and cache locality of flow related data structures.
There is both a pure software variant of this, as well as a hardware
variant called RFS. We have shown very good server performance using
these techniques, and RFS has been in Linux for more than ten years
and there are some NICs that support aRFS. It's definitely deployed.

The algorithm essentially assumes a minimal number of hash collisions.
If two active flows are colliding in the map table then potentially
we're thrashing cache lines and hence performance can drop
precipitously-- as I mentioned, the default recommendation is 1024
entries in the map table, bu on a loaded system we've went as high as
32768 entries (each entry is only eight bytes, so the memory footprint
is just a bit more than negligible on a server). It probably doesn't
make sense to go much higher than that since in a UDP tunnel we
normally expect to get 15 bits of entropy in the source port
(essentially we can set the UDP source port like a flow label in a UDP
tunnel). So by this logic, we'd like at least fifteen bits of entropy
in the flow label.

An interesting thing here is that the WLCG data transfers use quite highly parallelised streams between compute / storage clusters around the 170+ sites in 40+ countries. In the two-week data challenge phase earlier this year (as a pre-test for the upcoming much higher luminosity LHC output to come) I recall over 150PB of LHC data was moved over around 10 days between sites. There is significant network tuning at the endpoints, but I don’t *think* anything like what you describe above.


The problem of having insufficient entropy in the flow label would
happen when other inputs also lack entropy. For instance, a single
IPv4/IPv6 tunnel might have high load, or even a single encrypted
tunnel. In other words, the problems would happen in cases where the
only entropy in the flow tuple is coming from the flow label.

Well, there is the “don’t rely on the flow label alone for hashing” rule, so you need to bear that in mind however many bits of entropy there are.


>
> IMHO, what matters in terms of the intended purpose of the RFC 6437
> version of the flow label is whether routers, ECMP/LAG systems, and
> load balancers are using it. That's where I have seen no indication
> of widespread deployment.

There's enough deployment that when we changed the flow label
midstream in connection, people complained that packets were being
dropped because they were being routed around stateful firewalls :-).
Also, we can also modulate the flow label to perform a type of source
routing, for instance if we notice a failing TCP connection we can
change the flow label to some other random value in hopes of finding a
better route. In the most extreme case, we could change the flow label
on every packet to perform Random Packet Steering across a network
which seems like where the high performance fabric folks want to go
(Falcon, Ulta Ethernet).

And that’s breaking a different rule – the “no change along the path”, rather than the “no meaning in the bits”.  There are different implications.

Tim


Tom

>
> >
> > Also, pertaining to routers, the draft states "The 5 entropy bits are
> > used to still support a level of conformance with the requirement
> > stated in RFC 6437 to support traffic distribution in ECMP and LAG
> > scenarios". I believe that is making an implicit assumption that we
> > have no more than 32 ports for ECMP or LAG.
>
> Yes.
>
> > In short, I am very leary about publishing an RFC that essentially
> > says that we really didn't need twenty bits of flow label, we just
> > needed five bits after all. Five bits might be sufficient in a one
> > limited domain, but maybe could cause problems in others. We really
> > don't know at this point...
>
> The WLCG really is a limited domain, even if it's world-wide. But you're
> correct; the draft could use a short discussion of that.
>
>      Brian
>
> >
> > Tom
> >
> >>
> >> Note that RFC 6294 was published in the Independent Stream; that's a precedent. For more of the tortured history of the flow label, see RFC 6436.
> >>
> >> Regards
> >>      Brian Carpenter
> >>
> >> On 15-Aug-24 06:05, Tommy Jensen wrote:
> >>> +1, I oppose the WG adopting this.
> >>>
> >>> I do think having the I-D published individually would make for a nice reference for any standards updates ("this is an example of where this document's mechanism is needed and why"). The concrete numbers are especially insightful to justify design trade-offs and defaults values. That said, maybe it does not even need to make it to RFC publication if, as a draft, it can be used by the WG to motivate a standards-friendly solution that can be adopted and published by the WG instead.
> >>>
> >>> This seems like something we should discuss at 121 either way.
> >>>
> >>> Thanks,
> >>> Tommy
> >>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >>> *From:* Tom Herbert <tom=[e-mail]>
> >>> *Sent:* Tuesday, August 13, 2024 3:21 PM
> >>> *To:* Mark Smith <[e-mail]>
> >>> *Cc:* IPv6 Operations <[e-mail]>
> >>> *Subject:* [EXTERNAL] [v6ops] Re: Fwd: The V6OPS WG has placed draft-cc-v6ops-wlcg-flow-label-marking in state "Call For Adoption By WG Issued"
> >>> [You don't often get email from tom=[e-mail]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification <https://aka.ms/LearnAboutSenderIdentification> ]
> >>>
> >>> On Tue, Aug 13, 2024 at 3:06 PM Mark Smith <[e-mail]> wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>> I don't support adoption for a number of reasons.
> >>>>
> >>>> Firstly, and the main reason, v6ops working group adoption means the
> >>>> document becomes a product of the WG, rather than just the authors.
> >>>>
> >>>> As this ID is documenting violations of IETF Standard Track RFC 6437,
> >>>> it becoming a v6ops WG document means that the v6ops WG is tacitly
> >>>> endorsing RFC 6437 violation, even if published as Informational.
> >>>>
> >>>> It becoming a WG document also suggests there is further work to be
> >>>> done on it by the WG, not just the authors. What further work on this
> >>>> ID is there the v6ops WG to do?
> >>>>
> >>>> If the IETF is the best place to publish it, why can't it be published
> >>>> as an Independent Submission, avoiding v6ops tacit endorsement and any
> >>>> WG publication overheads.
> >>>
> >>> Mark,
> >>>
> >>> I tend to agree. The proposal is for a very limited use case, and the
> >>> draft acknowledges that the correct mechanism to carry such data would
> >>> be Destination Options. IMO, the community would be better served by
> >>> the WG working on fixing the problems of extension header deployment
> >>> instead of pursuing workarounds like this. Independent Submission
> >>> seems appropriate.
> >>>
> >>> Tom
> >>>
> >>>>
> >>>> Why can't it be published as an academic paper outside of the IETF,
> >>>> further avoiding the IETF RFC publication costs?
> >>>>
> >>>> Regards,
> >>>> Mark.
> >>>>
> >>>> On Wed, 14 Aug 2024 at 04:27, Nick Buraglio
> >>>> <[e-mail]> wrote:
> >>>>>
> >>>>>
> >>>>> This call for adoption is wrapping up. If anyone else would like to comment, please read the draft and provide feedback by tomorrow.
> >>>>> Below is the current state:
> >>>>>
> >>>>> | [draft-cc-v6ops-wlcg-flow-label-marking](https://datatracker.ietf.org/doc/draft-cc-v6ops-wlcg-flow-label-marking/)  |             | Adoption Called 05-Aug-2024                        | Adoption Ending 19-August-2024 |
> >>>>> | ------------------------------------------------------------------------------------------------------------------ | ----------- | -------------------------------------------------- | ------------------------------ |
> >>>>> | Support                                                                                                            | Opposed     | Comments                                           | Comments Addressed             |
> >>>>> | Brian Carpenter                                                                                                    |             | discussion of deviations from RFC 6437 is helpful. |                                |
> >>>>> | Tim Winters                                                                                                        |             |                                                    |                                |
> >>>>> | Nick Buraglio                                                                                                      |             |                                                    |                                |
> >>>>> |                                                                                                                    | Tom Herbert | Feels is too specific                              | Yes                            |
> >>>>> | David Farmer                                                                                                       |             |                                                    |
> >>>>>
> >>>>>
> >>>>> I believe Tom had his concerns addressed but I never saw explicit support after.
> >>>>> If anyone else would like to comment, please read the draft and provide feedback by tomorrow.
> >>>>>
> >>>>> nb
> >>>>>
> >>>>>
> >>>>> On Mon, Aug 5, 2024 at 2:13 PM David Farmer <[e-mail]> wrote:
> >>>>>>
> >>>>>>
> >>>>>> I support adoption.
> >>>>>>
> >>>>>> On Mon, Aug 5, 2024 at 13:35 Nick Buraglio <[e-mail]> wrote:
> >>>>>>>
> >>>>>>> All,
> >>>>>>> We'd like to wrap this adoption call up. As of now we don't have a lot
> >>>>>>> of input, but it is mostly positive. Please read and comment.  Current
> >>>>>>> state:
> >>>>>>>
> >>>>>>> ### draft-cc-v6ops-wlcg-flow-label-marking
> >>>>>>> #### Adoption Called 06-July-2024
> >>>>>>> * Support - Comments - Comments Addressed
> >>>>>>> Brian Carpenter - discussion of deviations from RFC 6437 is helpful.
> >>>>>>> Tim Winters
> >>>>>>> Nick Buraglio
> >>>>>>>
> >>>>>>>   * Opposed - Comments - Comments Addressed
> >>>>>>> Tom Herbert - Feels is too specific - no
> >>>>>>>
> >>>>>>>
> >>>>>>> ---------- Forwarded message ---------
> >>>>>>> From: IETF Secretariat <[e-mail]>
> >>>>>>> Date: Sat, Jul 6, 2024 at 5:09 PM
> >>>>>>> Subject: The V6OPS WG has placed
> >>>>>>> draft-cc-v6ops-wlcg-flow-label-marking in state "Call For Adoption By
> >>>>>>> WG Issued"
> >>>>>>> To: <[e-mail]>,
> >>>>>>> <[e-mail]>, <[e-mail]>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> The V6OPS WG has placed draft-cc-v6ops-wlcg-flow-label-marking in state
> >>>>>>> Call For Adoption By WG Issued (entered by Nick Buraglio)
> >>>>>>>
> >>>>>>> The document is available at
> >>>>>>> https://datatracker.ietf.org/doc/draft-cc-v6ops-wlcg-flow-label-marking/ <https://datatracker.ietf.org/doc/draft-cc-v6ops-wlcg-flow-label-marking/>
> >>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> v6ops mailing list -- [e-mail]
> >>>>>>> To unsubscribe send an email to [e-mail]
> >>>>>
> >>>>> _______________________________________________
> >>>>> v6ops mailing list -- [e-mail]
> >>>>> To unsubscribe send an email to [e-mail]
> >>>>
> >>>> _______________________________________________
> >>>> v6ops mailing list -- [e-mail]
> >>>> To unsubscribe send an email to [e-mail]
> >>>
> >>> _______________________________________________
> >>> v6ops mailing list -- [e-mail]
> >>> To unsubscribe send an email to [e-mail]
> >>>
> >>> _______________________________________________
> >>> v6ops mailing list -- [e-mail]
> >>> To unsubscribe send an email to [e-mail]

_______________________________________________
v6ops mailing list -- [e-mail]
To unsubscribe send an email to [e-mail]