[v6ops] Re: [EXTERNAL] Re: Fwd: The V6OPS WG has placed draft-cc-v6ops-wlcg-flow-label-marking in state "Call For Adoption By WG Issued"

[Tom Herbert <[e-mail]>]

On Fri, Aug 16, 2024 at 9:41 AM Tim Chown
<Tim.Chown=[e-mail]> wrote:
>
> Hi Tom,
>
>
>
> From: Tom Herbert <tom=[e-mail]>
> Date: Thursday, 15 August 2024 at 01:19
> To: Brian E Carpenter <[e-mail]>
> Cc: Tommy Jensen <[e-mail]>, IPv6 Operations <[e-mail]>
> Subject: [v6ops] Re: [EXTERNAL] Re: Fwd: The V6OPS WG has placed draft-cc-v6ops-wlcg-flow-label-marking in state "Call For Adoption By WG Issued"
>
> On Wed, Aug 14, 2024 at 4:16 PM Brian E Carpenter
> <[e-mail]> wrote:
> >
> > Tom,
> >
> > On 15-Aug-24 10:56, Tom Herbert wrote:
> > > On Wed, Aug 14, 2024 at 3:25 PM Brian E Carpenter
> > > <[e-mail]> wrote:
> > >>
> > >> Hi,
> > >>
> > >> I am very strongly convinced that this draft should be published as an RFC. I am on the fence whether it should be published as an IETF stream Informational RFC or as an Independent stream Informational RFC. Since we know that most people don't read the boilerplate anyway, it hardly matters.
> > >>
> > >> Although this work bends the rules of RFC 6437, the implementers were (to my personal knowledge) aware of the issues from the start, allowed for some entropy bits, and most important have actually shown a use case for 20 largely underused bits in the IPv6 header. This is a use case that is worth publishing, and I can imagine it leading to reconsideration of some of the rules in RFC 6437. That's why it belongs in the RFC series.
> > >
> > > Hi Brian,
> > >
> > > I don't agree that the 20 bit flow label is underused. By default,
> > > Linux will set a flow label in a packet using the full twenty bits of
> > > entropy.
> >
> > I'm aware of that and glad of it.
> >
> > On the receive Linux path there are algorithms that assume
> > > that the full twenty bits of entropy are available. For instance,
> > > Receive Flow Steering (RFS) classifies packets by the hash that
> > > includes the flow label. Usually, we configure at least 1024 hash
> > > buckets (10 bits) so only getting five bits might create a bunch of
> > > hash collisions. Hosts might also be using the flow hash for other
> > > forms of filtering involving the flow label as well.
> >
> > I'm not sure what the use cases are for more than a handful of hash
> > buckets.
>
> Brian,
>
> Take a look at RFS (https://lwn.net/Articles/382428/) The idea is
> pretty simple, we get the tuple hash for received packets from the
> device or computer in the host for every packet. The hash may include
> the flow label as input. When we process the flow (e.g. TCP
> connection), we note what CPU is handling the connection and create an
> entry for that connection in rps_sock_flow_table which is the table
> that maps a hash to a CPU (specifically to an RX queue associated with
> that CPU). On subsequent packets we use the table to "steer" packets
> to the queue for the CPU the flow is being processed on, thereby
> creating affinity and cache locality of flow related data structures.
> There is both a pure software variant of this, as well as a hardware
> variant called RFS. We have shown very good server performance using
> these techniques, and RFS has been in Linux for more than ten years
> and there are some NICs that support aRFS. It's definitely deployed.
>
> The algorithm essentially assumes a minimal number of hash collisions.
> If two active flows are colliding in the map table then potentially
> we're thrashing cache lines and hence performance can drop
> precipitously-- as I mentioned, the default recommendation is 1024
> entries in the map table, bu on a loaded system we've went as high as
> 32768 entries (each entry is only eight bytes, so the memory footprint
> is just a bit more than negligible on a server). It probably doesn't
> make sense to go much higher than that since in a UDP tunnel we
> normally expect to get 15 bits of entropy in the source port
> (essentially we can set the UDP source port like a flow label in a UDP
> tunnel). So by this logic, we'd like at least fifteen bits of entropy
> in the flow label.
>
>
>
> An interesting thing here is that the WLCG data transfers use quite highly parallelised streams between compute / storage clusters around the 170+ sites in 40+ countries. In the two-week data challenge phase earlier this year (as a pre-test for the upcoming much higher luminosity LHC output to come) I recall over 150PB of LHC data was moved over around 10 days between sites. There is significant network tuning at the endpoints, but I don’t *think* anything like what you describe above.

That might be true in WLCG, but not necessarily for other deployments.

>
>
>
> The problem of having insufficient entropy in the flow label would
> happen when other inputs also lack entropy. For instance, a single
> IPv4/IPv6 tunnel might have high load, or even a single encrypted
> tunnel. In other words, the problems would happen in cases where the
> only entropy in the flow tuple is coming from the flow label.
>
>
>
> Well, there is the “don’t rely on the flow label alone for hashing” rule, so you need to bear that in mind however many bits of entropy there are.

Hi Tm,

"rely" is an interesting word here. Nobody relies on the flow label
for correct delivery of a packet, but we might rely on it as an
optimization for flow steering or routing. Per RFC6437 it's
RECOMMENDED that the flow label is set with a uniform distribution. I
take that to mean a uniform distribution across the full twenty bits.
So it makes sense that to build optimizations on the basis of that,
and considering that the vast majority of packets being sent now have
their flow labels set with a uniform distribution, it is a valid
assumption and valuable optimizations. If a host decides to set the
flow label to zero or use a non-uniform distribution then that's okay,
that shouldn't result in drop packets but we may lose some
optimizations.

>
>
>
> >
> > IMHO, what matters in terms of the intended purpose of the RFC 6437
> > version of the flow label is whether routers, ECMP/LAG systems, and
> > load balancers are using it. That's where I have seen no indication
> > of widespread deployment.
>
> There's enough deployment that when we changed the flow label
> midstream in connection, people complained that packets were being
> dropped because they were being routed around stateful firewalls :-).
> Also, we can also modulate the flow label to perform a type of source
> routing, for instance if we notice a failing TCP connection we can
> change the flow label to some other random value in hopes of finding a
> better route. In the most extreme case, we could change the flow label
> on every packet to perform Random Packet Steering across a network
> which seems like where the high performance fabric folks want to go
> (Falcon, Ulta Ethernet).
>
>
>
> And that’s breaking a different rule – the “no change along the path”, rather than the “no meaning in the bits”.  There are different implications.

There is no change of flow label along the path in this model, it's
only the source host that sets flow label. The practical implication
is that changing the flow label mid-flow even by the host might make
stateful devices in the path unhappy. So by default the flow label is
static for the lifetime of a flow, but within a limited domain where
it's known there's no stateful devices in the path we can use flow
label for interesting things like source routing for a failing
connection or the Random Packet Spraying that the HPC fabric folks
want.

Tom

>
>
>
> Tim
>
>
>
> Tom
>
> >
> > >
> > > Also, pertaining to routers, the draft states "The 5 entropy bits are
> > > used to still support a level of conformance with the requirement
> > > stated in RFC 6437 to support traffic distribution in ECMP and LAG
> > > scenarios". I believe that is making an implicit assumption that we
> > > have no more than 32 ports for ECMP or LAG.
> >
> > Yes.
> >
> > > In short, I am very leary about publishing an RFC that essentially
> > > says that we really didn't need twenty bits of flow label, we just
> > > needed five bits after all. Five bits might be sufficient in a one
> > > limited domain, but maybe could cause problems in others. We really
> > > don't know at this point...
> >
> > The WLCG really is a limited domain, even if it's world-wide. But you're
> > correct; the draft could use a short discussion of that.
> >
> >      Brian
> >
> > >
> > > Tom
> > >
> > >>
> > >> Note that RFC 6294 was published in the Independent Stream; that's a precedent. For more of the tortured history of the flow label, see RFC 6436.
> > >>
> > >> Regards
> > >>      Brian Carpenter
> > >>
> > >> On 15-Aug-24 06:05, Tommy Jensen wrote:
> > >>> +1, I oppose the WG adopting this.
> > >>>
> > >>> I do think having the I-D published individually would make for a nice reference for any standards updates ("this is an example of where this document's mechanism is needed and why"). The concrete numbers are especially insightful to justify design trade-offs and defaults values. That said, maybe it does not even need to make it to RFC publication if, as a draft, it can be used by the WG to motivate a standards-friendly solution that can be adopted and published by the WG instead.
> > >>>
> > >>> This seems like something we should discuss at 121 either way.
> > >>>
> > >>> Thanks,
> > >>> Tommy
> > >>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > >>> *From:* Tom Herbert <tom=[e-mail]>
> > >>> *Sent:* Tuesday, August 13, 2024 3:21 PM
> > >>> *To:* Mark Smith <[e-mail]>
> > >>> *Cc:* IPv6 Operations <[e-mail]>
> > >>> *Subject:* [EXTERNAL] [v6ops] Re: Fwd: The V6OPS WG has placed draft-cc-v6ops-wlcg-flow-label-marking in state "Call For Adoption By WG Issued"
> > >>> [You don't often get email from tom=[e-mail]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification <https://aka.ms/LearnAboutSenderIdentification> ]
> > >>>
> > >>> On Tue, Aug 13, 2024 at 3:06 PM Mark Smith <[e-mail]> wrote:
> > >>>>
> > >>>> Hi,
> > >>>>
> > >>>> I don't support adoption for a number of reasons.
> > >>>>
> > >>>> Firstly, and the main reason, v6ops working group adoption means the
> > >>>> document becomes a product of the WG, rather than just the authors.
> > >>>>
> > >>>> As this ID is documenting violations of IETF Standard Track RFC 6437,
> > >>>> it becoming a v6ops WG document means that the v6ops WG is tacitly
> > >>>> endorsing RFC 6437 violation, even if published as Informational.
> > >>>>
> > >>>> It becoming a WG document also suggests there is further work to be
> > >>>> done on it by the WG, not just the authors. What further work on this
> > >>>> ID is there the v6ops WG to do?
> > >>>>
> > >>>> If the IETF is the best place to publish it, why can't it be published
> > >>>> as an Independent Submission, avoiding v6ops tacit endorsement and any
> > >>>> WG publication overheads.
> > >>>
> > >>> Mark,
> > >>>
> > >>> I tend to agree. The proposal is for a very limited use case, and the
> > >>> draft acknowledges that the correct mechanism to carry such data would
> > >>> be Destination Options. IMO, the community would be better served by
> > >>> the WG working on fixing the problems of extension header deployment
> > >>> instead of pursuing workarounds like this. Independent Submission
> > >>> seems appropriate.
> > >>>
> > >>> Tom
> > >>>
> > >>>>
> > >>>> Why can't it be published as an academic paper outside of the IETF,
> > >>>> further avoiding the IETF RFC publication costs?
> > >>>>
> > >>>> Regards,
> > >>>> Mark.
> > >>>>
> > >>>> On Wed, 14 Aug 2024 at 04:27, Nick Buraglio
> > >>>> <[e-mail]> wrote:
> > >>>>>
> > >>>>>
> > >>>>> This call for adoption is wrapping up. If anyone else would like to comment, please read the draft and provide feedback by tomorrow.
> > >>>>> Below is the current state:
> > >>>>>
> > >>>>> | [draft-cc-v6ops-wlcg-flow-label-marking](https://datatracker.ietf.org/doc/draft-cc-v6ops-wlcg-flow-label-marking/)  |             | Adoption Called 05-Aug-2024                        | Adoption Ending 19-August-2024 |
> > >>>>> | ------------------------------------------------------------------------------------------------------------------ | ----------- | -------------------------------------------------- | ------------------------------ |
> > >>>>> | Support                                                                                                            | Opposed     | Comments                                           | Comments Addressed             |
> > >>>>> | Brian Carpenter                                                                                                    |             | discussion of deviations from RFC 6437 is helpful. |                                |
> > >>>>> | Tim Winters                                                                                                        |             |                                                    |                                |
> > >>>>> | Nick Buraglio                                                                                                      |             |                                                    |                                |
> > >>>>> |                                                                                                                    | Tom Herbert | Feels is too specific                              | Yes                            |
> > >>>>> | David Farmer                                                                                                       |             |                                                    |
> > >>>>>
> > >>>>>
> > >>>>> I believe Tom had his concerns addressed but I never saw explicit support after.
> > >>>>> If anyone else would like to comment, please read the draft and provide feedback by tomorrow.
> > >>>>>
> > >>>>> nb
> > >>>>>
> > >>>>>
> > >>>>> On Mon, Aug 5, 2024 at 2:13 PM David Farmer <[e-mail]> wrote:
> > >>>>>>
> > >>>>>>
> > >>>>>> I support adoption.
> > >>>>>>
> > >>>>>> On Mon, Aug 5, 2024 at 13:35 Nick Buraglio <[e-mail]> wrote:
> > >>>>>>>
> > >>>>>>> All,
> > >>>>>>> We'd like to wrap this adoption call up. As of now we don't have a lot
> > >>>>>>> of input, but it is mostly positive. Please read and comment.  Current
> > >>>>>>> state:
> > >>>>>>>
> > >>>>>>> ### draft-cc-v6ops-wlcg-flow-label-marking
> > >>>>>>> #### Adoption Called 06-July-2024
> > >>>>>>> * Support - Comments - Comments Addressed
> > >>>>>>> Brian Carpenter - discussion of deviations from RFC 6437 is helpful.
> > >>>>>>> Tim Winters
> > >>>>>>> Nick Buraglio
> > >>>>>>>
> > >>>>>>>   * Opposed - Comments - Comments Addressed
> > >>>>>>> Tom Herbert - Feels is too specific - no
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> ---------- Forwarded message ---------
> > >>>>>>> From: IETF Secretariat <[e-mail]>
> > >>>>>>> Date: Sat, Jul 6, 2024 at 5:09 PM
> > >>>>>>> Subject: The V6OPS WG has placed
> > >>>>>>> draft-cc-v6ops-wlcg-flow-label-marking in state "Call For Adoption By
> > >>>>>>> WG Issued"
> > >>>>>>> To: <[e-mail]>,
> > >>>>>>> <[e-mail]>, <[e-mail]>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> The V6OPS WG has placed draft-cc-v6ops-wlcg-flow-label-marking in state
> > >>>>>>> Call For Adoption By WG Issued (entered by Nick Buraglio)
> > >>>>>>>
> > >>>>>>> The document is available at
> > >>>>>>> https://datatracker.ietf.org/doc/draft-cc-v6ops-wlcg-flow-label-marking/ <https://datatracker.ietf.org/doc/draft-cc-v6ops-wlcg-flow-label-marking/>
> > >>>>>>>
> > >>>>>>> _______________________________________________
> > >>>>>>> v6ops mailing list -- [e-mail]
> > >>>>>>> To unsubscribe send an email to [e-mail]
> > >>>>>
> > >>>>> _______________________________________________
> > >>>>> v6ops mailing list -- [e-mail]
> > >>>>> To unsubscribe send an email to [e-mail]
> > >>>>
> > >>>> _______________________________________________
> > >>>> v6ops mailing list -- [e-mail]
> > >>>> To unsubscribe send an email to [e-mail]
> > >>>
> > >>> _______________________________________________
> > >>> v6ops mailing list -- [e-mail]
> > >>> To unsubscribe send an email to [e-mail]
> > >>>
> > >>> _______________________________________________
> > >>> v6ops mailing list -- [e-mail]
> > >>> To unsubscribe send an email to [e-mail]
>
> _______________________________________________
> v6ops mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]