Re: [IPv6] Flow label BCP [was: looking for feedback to draft-eckert-6man-qos-exthdr-discuss]

[Tom Herbert <[e-mail]>]

On Thu, Mar 7, 2024 at 1:32 PM Toerless Eckert <[e-mail]> wrote:
>
> On Thu, Mar 07, 2024 at 12:08:35PM -0800, Tom Herbert wrote:
> >
> > I think that makes sense. By default the flow label should be fixed
> > for the lifetime of a transport layer connection for consistent
> > routing of packets of the connection. The default behavior could be
> > changed by configuration, and the behavior could differ per connection
> > or per destination.
>
> That's a recommendation for crappy old TCP stacks that
> couldn't deal with packet reorder. If you're worried about link loss in DC ECMP,
> then  what you mentioned as spreading might be the better recommendation.

Toerless,

The problem isn't with TCP, it's with stateful firewalls. If we change
flow labels midway through a TCP connection then packets may be routed
differently so that they hit another firewall that doesn't have the
state for the connection so it drops the packets.

We actually hit this problem in Linux when flow labels were first
deployed. When a TCP connection moves past Established state, Linux
will compress the PCB. The problem was that the flow label was not
being saved in the compressed so subsequent packets sent for the
connection (final ACKs/FIN) were using a different flow label, routed
differently in the network, and a stateful firewall never saw the
final packets. The fix was easy, we just needed to preserve the flow
label in the compressed PCB. Now, by default, Linux will use the same
flow label over the lifetime of a TCP connection.

Tom


>
> Cheers
>     Toerless
>
> > Maybe one goal of the BCP is to assuage concerns over using the flow
> > label for ECMP instead of ports for input to the hash. Whenever we
> > suggest this, someone inevitably responds that the flow label isn't
> > reliable or secure. I think their concerns are based on the Security
> > Considerations of RFC6437-- might be worth revisiting these
> > considerations in a BCP.
>
> >
> > Tom
> >
> > >
> > >      Brian
> > >
> > > On 07-Mar-24 16:34, Tom Herbert wrote:
> > > >
> > > >
> > > > On Wed, Mar 6, 2024, 7:12 PM Brian E Carpenter <[e-mail] <mailto:[e-mail]>> wrote:
> > > >
> > > >     On 07-Mar-24 11:27, Toerless Eckert wrote:
> > > >      > On Wed, Mar 06, 2024 at 01:05:08PM -0800, Tom Herbert wrote:
> > > >      >> Toerless,
> > > >      >>
> > > >      >> The flow label is not usable and not a waste of header bits as some would
> > > >      >
> > > >      > Do you meaan "unusable" ?
> > > >      >
> > > >      >> seem to believe. It carries flow entropy that we use anywhere in the
> > > >      >> Internet (for ECMP, RSS, filtering, etc.,)
> > > >      >
> > > >      > I know. But see Joel's Pechakucha re. problems arising when differnt devices in the
> > > >      > same nework have different understandings what exactly to do with a flow label.
> > > >
> > > >     What exactly to do with the flow label is well defined. So non-conforming devices
> > > >     are non-conforming. That isn't exactly a new problem. For the IETF to look at the
> > > >     problem, we need an I-D, not an .mp4.
> > > >
> > > >      > That's why i am agreeing with his conclusion from back then to stop using it
> > > >      > in this "randomn interpretation" fashion (i do ECMP, you do RSS, he does filtering *aaarrrgghh*).
> > > >
> > > >     So, 2 of those 3 apparently violate the RFCs. (I doubt that RSS does what RFC 7098 says).
> > > >
> > > >
> > > > Brian,
> > > >
> > > > RSS has the same properties as ECMP, it's just another form of packet steering based on tuple hash with flow label as input. Filtering on flow label can be done as much as any header field is filtered.
> > > >
> > > > There is a question as to whether the flow label needs to be fixed for the lifetime of a connection. Because changing it can effect the route and trip up stateful devices in the network, we (Linux) makes the flow label fixed over the lifetime of a connection by default.
> > > >
> > > > If there's no stateful devices in the network we can modulate it for a connection when packets are dropped in an attempt to find a better route (this actually works pretty well an I believe at least on hyperscaler has deployed it). In the most extreme case, the flow label could be changed to a random value in every packet to perform Random Packet Spraying (RPS in high performance data centers is coming!).
> > > >
> > > > I don't believe any of these violate RFCs.
> > > >
> > > > Tom
> > > >
> > > >
> > > >
> > > >           Brian
> > > >
> > > >      >
> > > >      >> It think what you want is a generic way to repurpose the flow label bits.
> > > >      >> IMO, that's going to be difficult.
> > > >      >
> > > >      > I'd be happy to have an argument about that technically. I don't think it's true.
> > > >      > IETF stance re. standardizing something like that is of course a different issue,
> > > >      > but if the inability to find a technical issue counts...
> > > >      >
> > > >      > But i don't think that for my QoS interests i would even need to go down to such
> > > >      > big changes. It would suffice for the QoS header to determine and override any of
> > > >      > the actions associated with the flow-label in any pre-existing (and inconsistent)
> > > >      > implementations. And of course, one could think of the flow-label then not
> > > >      > be solely a random big number without specific semantic other than hoping to
> > > >      > be an as unique as possible ID, but more like a structured classification of the
> > > >      > flow (which i am sure has also been done in before).
> > > >      >
> > > >      > Cheers
> > > >      >      Toerless
> > > >      >>
> > > >      >> Tom
> > > >      >>
> > > >      >>
> > > >      >>> Cheers
> > > >      >>>      Toerless
> > > >      >>>
> > > >      >>>> Tom
> > > >      >>>>
> > > >      >>>>>
> > > >      >>>>> Cheers
> > > >      >>>>>      Toerless
> > > >      >>>>>
> > > >      >>>>>
> > > >      >>>>>> Tim
> > > >      >>>>>>
> > > >      >>>>>>> Cheers
> > > >      >>>>>>>     Toerless
> > > >      >>>>>>>
> > > >      >>>>>>> On Tue, Mar 05, 2024 at 10:31:09AM -0800, Tom Herbert wrote:
> > > >      >>>>>>>> On Tue, Mar 5, 2024 at 6:41 AM Tim Chown
> > > >      >>>>>>>> <Tim.Chown=[e-mail] <mailto:[e-mail]>> wrote:
> > > >      >>>>>>>>>
> > > >      >>>>>>>>> Hi,
> > > >      >>>>>>>>>
> > > >      >>>>>>>>> On 4 Mar 2024, at 23:02, Tom Herbert <tom=
> > > >      >>> [e-mail] <mailto:[e-mail]>> wrote:
> > > >      >>>>>>>>>
> > > >      >>>>>>>>> On Mon, Mar 4, 2024 at 12:37 PM Toerless Eckert <[e-mail] <mailto:[e-mail]>>
> > > >      >>> wrote:
> > > >      >>>>>>>>>
> > > >      >>>>>>>>>
> > > >      >>>>>>>>> Dear 6MAN-WG:
> > > >      >>>>>>>>>
> > > >      >>>>>>>>> I have just posted an extremely rough draft
> > > >      >>> draft-eckert-6man-qos-exthdr-discuss, to help start a discussion
> > > >      >>>>>>>>> about common IPv6 extension headers for (mostly) stateless QoS
> > > >      >>> beyond what we can do with just DSCP.
> > > >      >>>>>>>>>
> > > >      >>>>>>>>>
> > > >      >>>>>>>>> Hi Toerless,
> > > >      >>>>>>>>>
> > > >      >>>>>>>>> You might want to look at draft-herbert-fast and
> > > >      >>>>>>>>> draft-herbert-host2netsig. It looks like these have similar
> > > >      >>> goals.
> > > >      >>>>>>>>>
> > > >      >>>>>>>>>
> > > >      >>>>>>>>> And that is similar in spirit to what the CERN experiments are
> > > >      >>> doing with flow label semantics, which would/could be HbH header
> > > >      >>> information if then insertion penalty were not so high.
> > > >      >>>>>>>>
> > > >      >>>>>>>> Hi Tim,
> > > >      >>>>>>>>
> > > >      >>>>>>>> The CERN experiment might be okay as an experiment, but
> > > >      >>> overloading
> > > >      >>>>>>>> the twenty bit information of flow label is neither scalable nor
> > > >      >>>>>>>> standardizable. This is especially true for those proposals that
> > > >      >>> want
> > > >      >>>>>>>> to set some bits differently within the same flow and expect that
> > > >      >>>>>>>> routers will ignore those bits for ECMP hash.
> > > >      >>>>>>>>
> > > >      >>>>>>>> I am interested in what you mean by " if then insertion penalty
> > > >      >>> were
> > > >      >>>>>>>> not so high".
> > > >      >>>>>>>>
> > > >      >>>>>>>> Tom
> > > >      >>>>>>>>
> > > >      >>>>>>>>
> > > >      >>>>>>>>>
> > > >      >>>>>>>>>
> > > >      >>> https://www.ietf.org/archive/id/draft-cc-v6ops-wlcg-flow-label-marking-02.html <https://www.ietf.org/archive/id/draft-cc-v6ops-wlcg-flow-label-marking-02.html>
> > > >      >>>>>>>>>
> > > >      >>>>>>>>> And there are others, each doing something slightly different,
> > > >      >>> when we’d ideally have one EH to rule them all.
> > > >      >>>>>>>>>
> > > >      >>>>>>>>> Tim
> > > >      >>>>>>>>>
> > > >      >>>>>>>>>
> > > >      >>>>>>>>> Right now this is a discussion draft not intended to become RFC
> > > >      >>> because it's my impression that the
> > > >      >>>>>>>>> 6MAN community might benefit from some useful summary of how
> > > >      >>> DetNet (and potentially other WGs) might
> > > >      >>>>>>>>> use this work, but this would not be part of a final spec draft,
> > > >      >>> and likewise i have a wide range of
> > > >      >>>>>>>>> open questions instead of answers, and i included those
> > > >      >>> questions into the draft seeking for feedback from
> > > >      >>>>>>>>> 6MAN.
> > > >      >>>>>>>>>
> > > >      >>>>>>>>> Overall, i didn't want to go down a possible rabbit hole of
> > > >      >>> working on details of the spec if it just
> > > >      >>>>>>>>> turns out to involve insurmountable IETF process obtacles to go
> > > >      >>> this route. For example, we could continue to
> > > >      >>>>>>>>> standardize all advanced forwarding functions only into MPLS and
> > > >      >>> ignore IPv6 as DetNet has done so far
> > > >      >>>>>>>>> (*mumble ;-).
> > > >      >>>>>>>>>
> > > >      >>>>>>>>> The lack of such extension headers has IMHO held back innovation
> > > >      >>> into better (stateless) QoS, especially
> > > >      >>>>>>>>> in many controlled networks since at least 25 years, for example
> > > >      >>> when draft-stoica-diffserv-dps
> > > >      >>>>>>>>> was abandomed because it was too painfull trying to get to
> > > >      >>> through all the IETF IPv6 bureaucracy -
> > > >      >>>>>>>>> for just one algorithm, when there are so many that would
> > > >      >>> deserve experimentation in specific
> > > >      >>>>>>>>> networks. But given the good recent/ongoing work for example
> > > >      >>> into  I-D.ietf-6man-hbh-processing,
> > > >      >>>>>>>>> i would hope that we're closer now to actually wanting our
> > > >      >>> extensibility of IPv6 actually be used
> > > >      >>>>>>>>> by the industry (instead of all this happening only in MPLS).
> > > >      >>>>>>>>>
> > > >      >>>>>>>>> With DetNet we are too in the situation that we have multiple
> > > >      >>> candidates on the table and IMHO
> > > >      >>>>>>>>> it will not be very useufl trying to run a lottery for a single
> > > >      >>> "winner" and standardize just that.
> > > >      >>>>>>>>>
> > > >      >>>>>>>>> I have seen a lot more success in the industry by just letting
> > > >      >>> different algorithms compete with
> > > >      >>>>>>>>> each othrer in products and let the market decide. That was
> > > >      >>> quite a lot happening in e.g.: packet
> > > >      >>>>>>>>> scheduling in routers at least since the end of the 90th when in
> > > >      >>> my impression every new
> > > >      >>>>>>>>> hardware forwarding router implemented it's own new packet
> > > >      >>> scheduler based on the just hired lead
> > > >      >>>>>>>>> engineers PhD thesis. And over a period of 20 years, a lot of
> > > >      >>> commonality and industry
> > > >      >>>>>>>>> knowledge evolved in that space. For this type of scheduling,
> > > >      >>> this innovation was possible because it did not
> > > >      >>>>>>>>> require new packet headers, but just a lot of (ab)use of DSCP
> > > >      >>> and/or more or less horrenduous
> > > >      >>>>>>>>> QoS configurations. But for those solutions that do require
> > > >      >>> additional in-packet-QoS metadata,
> > > >      >>>>>>>>> we never created a viable method where it was easy for the
> > > >      >>> innovators/implementers to concentrate
> > > >      >>>>>>>>> on the novelties of the algorithm in question and get all the
> > > >      >>> knucklehead "how to packetize and what generic
> > > >      >>>>>>>>> requirements/functionalities" be provided as much as possible by
> > > >      >>> an existing framework/RFC.
> > > >      >>>>>>>>>
> > > >      >>>>>>>>> So, i'd be very happy to find interest to help progress this
> > > >      >>> work, aka: writing something
> > > >      >>>>>>>>> that ultimately would become a draft-ietf-6man-common-qos-exthr
> > > >      >>> or the like. I have tentatively
> > > >      >>>>>>>>> asked for a slot for IETF119 6MAN to present and get feedback,
> > > >      >>> if you think that would be time well
> > > >      >>>>>>>>> spent, pls. chime in.
> > > >      >>>>>>>>>
> > > >      >>>>>>>>> Cheers
> > > >      >>>>>>>>>    Toerless, for the authors
> > > >      >>>>>>>>>
> > > >      >>>>>>>>> On Mon, Mar 04, 2024 at 12:30:53PM -0800,
> > > >      >>> [e-mail] <mailto:[e-mail]> wrote:
> > > >      >>>>>>>>>
> > > >      >>>>>>>>> A new version of Internet-Draft
> > > >      >>> draft-eckert-6man-qos-exthdr-discuss-00.txt
> > > >      >>>>>>>>> has been successfully submitted by Toerless Eckert and posted to
> > > >      >>> the
> > > >      >>>>>>>>> IETF repository.
> > > >      >>>>>>>>>
> > > >      >>>>>>>>> Name:     draft-eckert-6man-qos-exthdr-discuss
> > > >      >>>>>>>>> Revision: 00
> > > >      >>>>>>>>> Title:    Considerations for common QoS IPv6 extension header(s)
> > > >      >>>>>>>>> Date:     2024-03-04
> > > >      >>>>>>>>> Group:    Individual Submission
> > > >      >>>>>>>>> Pages:    27
> > > >      >>>>>>>>> URL:
> > > >      >>> https://www.ietf.org/archive/id/draft-eckert-6man-qos-exthdr-discuss-00.txt <https://www.ietf.org/archive/id/draft-eckert-6man-qos-exthdr-discuss-00.txt>
> > > >      >>>>>>>>> Status:
> > > >      >>> https://datatracker.ietf.org/doc/draft-eckert-6man-qos-exthdr-discuss/ <https://datatracker.ietf.org/doc/draft-eckert-6man-qos-exthdr-discuss/>
> > > >      >>>>>>>>> HTMLized:
> > > >      >>> https://datatracker.ietf.org/doc/html/draft-eckert-6man-qos-exthdr-discuss <https://datatracker.ietf.org/doc/html/draft-eckert-6man-qos-exthdr-discuss>
> > > >      >>>>>>>>>
> > > >      >>>>>>>>>
> > > >      >>>>>>>>> Abstract:
> > > >      >>>>>>>>>
> > > >      >>>>>>>>>   This document is written to start a discussion and collect
> > > >      >>> opinions
> > > >      >>>>>>>>>   and ansers to questions raised in this document on the issue of
> > > >      >>>>>>>>>   defining IPv6 extension headers for DETNET-WG functionality with
> > > >      >>>>>>>>>   IPv6.
> > > >      >>>>>>>>>
> > > >      >>>>>>>>>
> > > >      >>>>>>>>>
> > > >      >>>>>>>>> The IETF Secretariat
> > > >      >>>>>>>>>
> > > >      >>>>>>>>>
> > > >      >>>>>>>>>
> > > >      >>> --------------------------------------------------------------------
> > > >      >>>>>>>>> IETF IPv6 working group mailing list
> > > >      >>>>>>>>> [e-mail] <mailto:[e-mail]>
> > > >      >>>>>>>>> Administrative Requests:
> > > >      >>> https://www.ietf.org/mailman/listinfo/ipv6 <https://www.ietf.org/mailman/listinfo/ipv6>
> > > >      >>>>>>>>>
> > > >      >>> --------------------------------------------------------------------
> > > >      >>>>>>>>>
> > > >      >>>>>>>>>
> > > >      >>>>>>>>>
> > > >      >>> --------------------------------------------------------------------
> > > >      >>>>>>>>> IETF IPv6 working group mailing list
> > > >      >>>>>>>>> [e-mail] <mailto:[e-mail]>
> > > >      >>>>>>>>> Administrative Requests:
> > > >      >>> https://www.ietf.org/mailman/listinfo/ipv6 <https://www.ietf.org/mailman/listinfo/ipv6>
> > > >      >>>>>>>>>
> > > >      >>> --------------------------------------------------------------------
> > > >      >>>>>>>>>
> > > >      >>>>>>>>>
> > > >      >>>>>>>>
> > > >      >>>>>>>
> > > >      >>>>>>> --
> > > >      >>>>>>> ---
> > > >      >>>>>>> [e-mail] <mailto:[e-mail]>
> > > >      >>>>>>
> > > >      >>>>>
> > > >      >>>>> --
> > > >      >>>>> ---
> > > >      >>>>> [e-mail] <mailto:[e-mail]>
> > > >      >>>>
> > > >      >>>
> > > >      >>> --
> > > >      >>> ---
> > > >      >>> [e-mail] <mailto:[e-mail]>
> > > >      >>>
> > > >      >
> > > >     --------------------------------------------------------------------
> > > >     IETF IPv6 working group mailing list
> > > >     [e-mail] <mailto:[e-mail]>
> > > >     Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6 <https://www.ietf.org/mailman/listinfo/ipv6>
> > > >     --------------------------------------------------------------------
> > > >
> >
>
> --
> ---
> [e-mail]