Re: [v6ops] draft-cc-v6ops-wlcg-flow-label-marking-00.txt

[Tom Herbert <[e-mail]>]

Hi,

Thanks for the draft. I have a few comments.

We've seen attempts before to repurpose Flow Label bits. This proposal
does that, however it looks like the data is still identifying
information that would be consistent with the spirit of the flow
label; in particular, I believe that the flow label would be
consistent for the lifetime of a connection so that it wouldn't break
ECMP or stateful firewalls.

>From the draft: "Entropy bits are 5 bits in positions 1, 2, 12, 19 and
20, and are set at random once per network flow for the duration of
its lifetime"

Is there a reason the flow entropy bits aren't contiguous? It's
conceivable, that some router might choose to mask the lower bits of
the flow label when selecting an ECMP port, in which case it might be
better for the entropy bits to occupy the low order bits

>From the draft: "Extension headers are known to be problematic in that
they have a history of being filtered or dropped in transit, as
measured in [RFC7872] with substantual further discussion in
[RFC9098]."

This is a perfect example of the conundrum we face with IPv6 extension
headers! Some providers claim that they won't allow them until there's
no deployed use case for them, but developers can't deploy a valid use
case because too many networks don't allow them. By definition,
extension headers are the right mechanism to carry the information
being conveyed. There are also extensible to allow more than twenty
bytes, HBH Options can be inspected and processed by intermediate
devices (or completely ignored per RFC8200), and they provide a
definitive code point (an encoded flow label only works if everyone in
the network understands the encoding).

"It might be that such issues are less common in Research and
Education networks."

Have you measured this? I would hope that a Research network would
want to be on the forefront of the technology, and it seems likely
that 20 bits of information won't scale to other interesting use cases
(fine grained identification, QOS, etc.)

"Given the size of IPv6 addresses, it is possible to mark or "color"
packets by using specific site network prefixes"

IMO, this is equates to a hack and probably would have more
ramifications that overloading the flow label

>From the draft: "A recently published IETF personal draft documents
the concept of "Network Tokens", see [I-D.yiakoumis-network-tokens]."
"A network token is a small piece of data that end users attach to
their packets." "The draft proposes a 28-bit token ID field."

A "small piece of data", but not small enough to fit into twenty bits.
Yet another example of useful network signaling that should be in an
HBH option. You might want to look at
https://www.ietf.org/archive/id/draft-herbert-fast-04.txt also

"The Destination option header could therefore be a logical choice to
place application-specific telemetry identifiers''.

If the expectation is that intermediate nodes will need to process
this information, the Destination Options really aren't appropriate
(except in the case of an intermediate node with Dest Opts before the
routing header).This should be Hop-by-Hop Options

"However, at present, the linux implementation in particular requires
either setuid 0 or CAP_NET_RAW capability to be able to call
setsockopt(s, IPPROTO_IPV6, IPV6_DSTOPTS, ext_hdr_p, ext_hdr_size,
making it unusable by typical userspace applications. There has been a
set of patches made that could address this as well as extend the
functionality, though they have not been met with support from the
linux network maintainers."

Are you referring to patches I posted to netdev a while back? :-)
Generally, I don't believe that we should define on-the-wire protocols
on the basis of limitations or deficiencies in implementation,
especially in software. It is far easier to fix implementation than to
standardize and deploy a new protocol!  If you'd like to pursue these
changes let me know

"Marking in the payload has been considered to be out of scope given
the prevalence of TLS/SSL/etc, which means that payloads cannot be
inspected on path."

Good!

"As usage of IPv4 has been superseded by IPv6, it was determined that
further effort for IPv4 was unwarranted."

Good! In fact, I don't think you need to mention IPv4 at all other
than to say it's not supported for this use case

"If there are concerns about preserving entropy and reducing the
possible collisions with the standard use of the IPv6 Flow Label, we
could use the "entropy" bits defined above to instead calculate a
Hamming Code. A Hamming Code calculates a set of Parity Bits to be
used to extend a set of Message (Data) Bits, that will maximize the
number of bits that are different between "valid" messages."

The flow labels are set either by simply assigning a random value to a
flow or by hash the 4-tuple typically with Jenkins or Toeplitz hash.
In either case, the value is assigned with a uniform distribution and
has good waterfall properties (i.e. flipping a single bit of hash
input flips 1/2 of the bits of output). Regardless of the algorithm,
if we only have five bits of entropy to work with then there will be a
lot of collisions for flows between the same two endpoints. This could
create load imbalances in ECMP. I think this would practically only be
a problem for something like tunnels where many flow are encapsulate
over a tunnel and the only entropy is the flow label

Tom



On Fri, Jun 30, 2023 at 2:02 AM Tim Chown <[e-mail]> wrote:
>
> Hi Brian,
>
> > On 29 Jun 2023, at 23:50, Brian E Carpenter <[e-mail]> wrote:
> >
> > Hi Dale,
> >
> > Thanks for documenting this.
> >
> > As well as the impact on ECMP/LAG (where you might add a reference to
> > RFC 6438), could you also mention the impact on server load balancing
> > (RFC 7098)? The real issue here is that with only 5 random bits, the
> > probability of two very large flows hashing to the same path or server
> > is higher than we'd like.
>
> We can add that pointer.
>
> > But since the WLCG flows are *all* very large,
> > this presumably doesn't matter so much in your use case?
>
> Well, actually they’re not. Many transfers are aggregates of a high number of lower throughout flows, moreso perhaps that there’s been a shift away from GridFTP to XRootD.
>
> > I'd never thought about using a Hamming code to generate flow labels.
> > But since a given flow must use the same flow label throughout, you can
> > only use the regular 5-tuple as input to the Hamming code. I don't
> > believe that a Hamming code will give you a discrete uniform distribution,
> > but I'm not an expert in that area.
>
> This would be good to get some expert feedback on.
>
> Note that traffic for a single excitement (community) and application will share 15 common bits, so it’s the other 5 bits we would like to see as effectively randomised and distributed as possible.
>
> > An expert known as ChatGPT says:
> > "No, a Hamming code does not produce a discrete uniform distribution...
> > The non-uniformity of the distribution arises from the fact that Hamming
> > codes prioritize error detection and correction capabilities over
> > uniformity of codeword distribution."
>
> :)
>
> > So I think that simply using a stateless hash is the best way. These
> > days I like FNV (draft-eastlake-fnv) which is easy and cheap to implement.
>
> This is all good discussion, more feedback very welcome.
>
> We’d like to produce a -01 before the cutoff.
>
> Thanks,
> Tim
>
> > Regards
> >   Brian Carpenter
> >
> > On 30-Jun-23 07:52, Dale W. Carder wrote:
> >> Hi folks,
> >> We have submitted this draft documenting our experimental use of the
> >> IPv6 flow label for packet marking and the considerations we had made
> >> along the way that sort of got us to this point.
> >> I'm hoping we could get some initial discussion underway here before
> >> IETF 117.  Thanks!
> >> Dale
> >> ----- Forwarded message from [e-mail] -----
> >>> Date: Thu, 29 Jun 2023 12:18:53 -0700
> >>> From: [e-mail]
> >>> Subject: New Version Notification for draft-cc-v6ops-wlcg-flow-label-marking-00.txt
> >>>
> >>>
> >>> A new version of I-D, draft-cc-v6ops-wlcg-flow-label-marking-00.txt
> >>> has been successfully submitted by Dale W. Carder and posted to the
> >>> IETF repository.
> >>>
> >>> Name:               draft-cc-v6ops-wlcg-flow-label-marking
> >>> Revision:   00
> >>> Title:              Use of the IPv6 Flow Label for WLCG Packet Marking
> >>> Document date:      2023-06-29
> >>> Group:              Individual Submission
> >>> Pages:              15
> >>> URL:            https://www.ietf.org/archive/id/draft-cc-v6ops-wlcg-flow-label-marking-00.txt
> >>> Status:         https://datatracker.ietf.org/doc/draft-cc-v6ops-wlcg-flow-label-marking/
> >>> Html:           https://www.ietf.org/archive/id/draft-cc-v6ops-wlcg-flow-label-marking-00.html
> >>> Htmlized:       https://datatracker.ietf.org/doc/html/draft-cc-v6ops-wlcg-flow-label-marking
> >>>
> >>>
> >>> Abstract:
> >>>    This document describes an experimentally deployed approach currently
> >>>    used within the Worldwide Large Hadron Collider Computing Grid (WLCG)
> >>>    to mark packets with their project (experiment) and application.  The
> >>>    marking uses the 20-bit IPv6 Flow Label in each packet, with 15 bits
> >>>    used for semantics and 5 bits for entropy.  Alternatives, in
> >>>    particular use of IPv6 Extension Headers (EH), were considered but
> >>>    found to not be practical.  The WLCG is one of the largest worldwide
> >>>    research communities and has adopted IPv6 heavily for movement of
> >>>    many tens of PB of data annually, with the ultimate goal of running
> >>>    IPv6 only.
> >>>
> >>>
> >>>
> >>> The IETF Secretariat
> >>>
> >>>
> >> ----- End forwarded message -----
> >> _______________________________________________
> >> v6ops mailing list
> >> [e-mail]
> >> https://www.ietf.org/mailman/listinfo/v6ops
> >
> > _______________________________________________
> > v6ops mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/v6ops
>
> _______________________________________________
> v6ops mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/v6ops