Re: [tcpm] [v6ops] Flow Label Load Balancing

[Alexander Azimov <[e-mail]>]

Brian,

May I ask you to give more details on this issue:

I am convinced that it is often unsafe, because that depends on exactly how
> a server farm load balancer handles such cases. If it maintains the TCP
> session on the same actual server if there is SYN(-ACK) retransmission, the
> flow label must not be changed.

If a retransmitted SYN goes to another server/anycast point this won't
bring much harm. At least we haven't experienced such problems in our
datacenters.

чт, 26 нояб. 2020 г. в 22:53, Brian E Carpenter <[e-mail]
>:

> Alexander,
>
> > 1. In the case of SYN or SYN-ACK retransmission flow label SHOULD be
> recalculated.
> ...
> > The first point stands for redirecting connection from the degraded path
> before the connection is established, and it looks safe.
>
> I am convinced that it is often unsafe, because that depends on exactly
> how a server farm load balancer handles such cases. If it maintains the TCP
> session on the same actual server if there is SYN(-ACK) retransmission, the
> flow label must not be changed.
> The benefit for possible broken routing is quite hypothetical, whereas the
> risk to
> session persistence at the server is real. (Remember that the source
> cannot know by magic that there is server load balancing at the far end,
> but it's the common case today.)
>
> > 2. In the case of RTO timeout expiration in the established TCP session
> the flow label MAY be recalculated. This setting MUST be switched off by
> default.
>
> As noted before, this is only safe in a limited domain where it is known
> with certainty that server load balancing is not in use. I think it's
> therefore a corner case and specifically, the appropriate normative
> statement would be SHOULD NOT, which is defined very carefully in RFC2119
> exactly for this sort of case.
>
> But in fact I think we should not attempt to legislate on this point. The
> value is marginal at best. As others have said, this is not the way to
> tackle operational routing problems.
>
> Regards
>    Brian Carpenter
>
> On 26-Nov-20 22:34, Alexander Azimov wrote:
> > Dear colleagues,
> >
> > We started discussing an incorrect default behavior, and Tom has already
> confirmed that it will be fixed.
> >
> > Later the thread turned into an argument if the flow label can be
> changed during connection/flow lifetime according to current RFC documents,
> though these documents can be updated. This looks a bit weird for me
> because I always thought that it is the IETF community's responsibility to
> document proper solutions. If something undocumented but worthy happens in
> the industry - IETF should catch up. So, I would like to get back to the
> discussion of reasons and their safety.
> >
> > The general idea of changing the routing path upon network
> outage/degradation looks obvious. Getting kind of source-based routing can
> significantly reduce the reaction time and improve end-user experience. The
> flow label has a perfect match here: transparent for the application, set
> by the source, not part of 5-tuple, while it can be used in the load
> balancing. IMO poorly documented. I would like to learn your feedback for
> the next wording:
> >
> > Let say that the flow label is a hash of values from the IP packet's
> 5-tuple and random number. Then
> >
> > 1. In the case of SYN or SYN-ACK retransmission flow label SHOULD be
> recalculated.
> > 2. In the case of RTO timeout expiration in the established TCP session
> the flow label MAY be recalculated. This setting MUST be switched off by
> default.
> > 3. Otherwise flow label SHOULD be preserved unchanged.
> >
> > The first point stands for redirecting connection from the degraded path
> before the connection is established, and it looks safe. The second one can
> also improve performance if it is set on the server-side. Please comment if
> you see security flaws in such a design.
> >
> > чт, 26 нояб. 2020 г. в 03:16, Tom Herbert <[e-mail] <mailto:
> [e-mail]>>:
> >
> >     On Wed, Nov 25, 2020 at 3:33 PM Brian E Carpenter
> >     <[e-mail] <mailto:[e-mail]>>
> wrote:
> >     >
> >     > I'm not Joel, but I did once spend some time grepping RFCs to find
> out whether
> >     > "flow" or "microflow" was the preferred term. In RFC2474, which is
> normative,
> >     > we have:
> >     >
> >     >    Microflow: a single instance of an application-to-application
> flow of
> >     >    packets which is identified by source address, destination
> address,
> >     >    protocol id, and source port, destination port (where
> applicable).
> >     >
> >     > But in the flow label work we explicitly avoided being that
> precise, and
> >     > did not use the term "microflow". There might be some load
> balancing
> >     > scenarios where you want a broader definition, even including
> bidirectional
> >     > flows. There are expired drafts on that topic:
> >     > draft-tarreau-extend-flow-label-balancing
> >     > draft-wang-6man-flow-label-reflection
> >     >
> >     Brian,
> >
> >     Random Packet Spraying
> >     (
> https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.297.529&rep=rep1&type=pdf
> )
> >     is an interesting idea where packets for a single connection are
> >     purposely distributed across multiple paths for load distribution.
> Per
> >     packet randomized flow labels with flow label aware ECMP makes this
> >     quite easy to do without requiring any special support in switches
> >     like you'd need with IPv4. I'm not necessarily advocating this, but
> it
> >     does highlight one potential use case of having a flow label that
> >     doesn;t have rigidly defined requirements on the host.
> >
> >     Tom
> >
> >     > Regards
> >     >    Brian
> >     >
> >     > On 26-Nov-20 11:13, Tom Herbert wrote:
> >     > > Joel, is there a normative definition of a flow?
> >     > >
> >     > > On Wed, Nov 25, 2020, 1:27 PM Joel M. Halpern <
> [e-mail] <mailto:[e-mail]> <mailto:
> [e-mail] <mailto:[e-mail]>>> wrote:
> >     > >
> >     > >     No, as Brian says, there are escape clauses in the flow
> definitions.
> >     > >
> >     > >     But changing the flow label due to traffic problems does not
> correspond
> >     > >     to packets being in actually different flows.
> >     > >     If one were using UDP, and mixing loss sensitive packets
> with loss
> >     > >     insensitive packets for a special application, sure, one
> could use two
> >     > >     flow labels.  But that is not what you are describing.
> >     > >
> >     > >     Yours,
> >     > >     Joel
> >     > >
> >     > >     On 11/25/2020 3:05 PM, Tom Herbert wrote:
> >     > >     > Joel,
> >     > >     >
> >     > >     > Is there an RFC that clearly and unambiguously states that
> a host MUST
> >     > >     > use the same flow label for the lifetime _and_ clearly
> defines exactly
> >     > >     > what a flow is with respect to such a requirement (for
> instance, how
> >     > >     > would you define a flow and enforce such a requirement in
> UDP? IPsec?
> >     > >     > other encapsulations?). If there is such a requirement
> then we'll
> >     > >     > change the code to be conformant.
> >     > >     >
> >     > >     > Tom
> >     > >     >
> >     > >     > On Wed, Nov 25, 2020 at 12:39 PM Joel M. Halpern <
> [e-mail] <mailto:[e-mail]> <mailto:
> [e-mail] <mailto:[e-mail]>>> wrote:
> >     > >     >>
> >     > >     >> This kind of thing is why, as I understand it, MPTCP has
> discovery
> >     > >     >> mechanisms ot know if both sides use it, and can select
> alternative
> >     > >     >> addresses for communication.
> >     > >     >>
> >     > >     >> Trying to guess flow labels that might avoid a problem
> because it might
> >     > >     >> be an ECMP problem, is just flailing about.  Not a good
> design for
> >     > >     >> operational protocols.
> >     > >     >>
> >     > >     >> And in general, designing protocols around "I know
> exactly what is going
> >     > >     >> on"  (the requirement for what you describe that goes
> well beyond just
> >     > >     >> "limited domains") is also a recipe for failure.
> >     > >     >>
> >     > >     >> The Flow Label RFCs are actually very explicit that a
> flow label is
> >     > >     >> supposed to be stable for the life of the flow.
> Otherwise, it isn't a
> >     > >     >> flow label.
> >     > >     >>
> >     > >     >> Yours,
> >     > >     >> Joel
> >     > >     >>
> >     > >     >> On 11/25/2020 2:35 PM, Tom Herbert wrote:
> >     > >     >>> Hi Fernando, comments in line...
> >     > >     >>>
> >     > >     >>> On Wed, Nov 25, 2020 at 12:13 AM Fernando Gont <
> [e-mail] <mailto:[e-mail]> <mailto:
> [e-mail] <mailto:[e-mail]>>> wrote:
> >     > >     >>>>
> >     > >     >>>> Hi, Tom,
> >     > >     >>>>
> >     > >     >>>> On 24/11/20 16:43, Tom Herbert wrote:
> >     > >     >>>> [....]
> >     > >     >>>>> Modulating the flow label is a means to affect the
> routing of packets
> >     > >     >>>>> through the network that uses flow labels as input to
> the ECMP hash.
> >     > >     >>>>
> >     > >     >>>> What's the point?
> >     > >     >>>>
> >     > >     >>>> 1) You cannot tell *if* the FL is being used.
> >     > >     >>>>
> >     > >     >>> Generally true, but in a limited domain this information
> could be
> >     > >     >>> discerned. I'd note that it's also generally true that
> we don't know
> >     > >     >>> if there is a load balancer or stateful firewall in the
> path that
> >     > >     >>> requires consistent routing, but in a limited domain we
> could know
> >     > >     >>> that also.
> >     > >     >>>
> >     > >     >>>> 2) Changing the FL does not necessarily mean that
> packets will employ a
> >     > >     >>>> different link.
> >     > >     >>>
> >     > >     >>> It's an opportunistic mechanism. If a connection is
> failing and we get
> >     > >     >>> a better path that fixes it by simply changing the flow
> label then
> >     > >     >>> what's the harm?
> >     > >     >>>
> >     > >     >>>>
> >     > >     >>>> 3) If the network is failing, shouldn't you handle this
> via routing?
> >     > >     >>>>
> >     > >     >>> Sure, but then that requires an out of band feedback
> loop from a TCP
> >     > >     >>> implementation to the network infrastructure to indicate
> there is a
> >     > >     >>> problem and then the network needs to respond. That's
> significant
> >     > >     >>> infrastructure and higher reaction time than doing
> something in TCP
> >     > >     >>> and IP. Think of modulating the flow label is an
> inexpensive form of
> >     > >     >>> source routing within a limited domain that doesn't need
> any
> >     > >     >>> infrastructure or heavyweight protocols or something
> like segment
> >     > >     >>> routing.
> >     > >     >>>
> >     > >     >>>>
> >     > >     >>>>
> >     > >     >>>>> The basic idea is that the flow label associated with
> a connection is
> >     > >     >>>>> randomly changed when the stack observes that the
> connection is
> >     > >     >>>>> failing (e.g. and an RTO). There is nothing in the
> specs that prevents
> >     > >     >>>>> this since the source is at liberty to set the flow
> label as it sees
> >     > >     >>>>> fit.
> >     > >     >>>>
> >     > >     >>>> The FL is expected to remain constant for the life of a
> flow. A
> >     > >     >>>> retransmitted packet is part of the same flow as the
> >     > >     >>>> originally-transmitted packet. So this seems to be
> contradicting the
> >     > >     >>>> very specification of the FL.
> >     > >     >>>>
> >     > >     >>>> For instance, If a RTO for a flow causes the FL to
> change, then one may
> >     > >     >>>> possibly argue that the FL is not naming/labeling what
> is said/expected
> >     > >     >>>> to be anming/labeling.
> >     > >     >>>
> >     > >     >>> Specifically, RFC6437 states:
> >     > >     >>>
> >     > >     >>> "It is therefore RECOMMENDED that source hosts support
> the flow label
> >     > >     >>> by setting the flow label field for all packets of a
> given flow to the
> >     > >     >>> same value chosen from an approximation to a discrete
> uniform
> >     > >     >>> distribution."
> >     > >     >>>
> >     > >     >>> So that is clearly a just recommendation, and not a
> requirement (and
> >     > >     >>> definitely not a MUST). Furthermore, RFC6437 states:
> >     > >     >>>
> >     > >     >>> "A forwarding node MUST either leave a non-zero flow
> label value
> >     > >     >>> unchanged or change it only for compelling operational
> security
> >     > >     >>> reasons as described in Section 6.1."
> >     > >     >>>
> >     > >     >>> So there's no guarantee in the protocol specs that flow
> labels are
> >     > >     >>> consistent for the life of the connection, which means
> that the
> >     > >     >>> network cannot assume that and thus it would be
> incorrect if the
> >     > >     >>> network tried to enforce flow label consistency as a
> protocol
> >     > >     >>> requirement. As I said, it is prudent to try to be
> consistent with
> >     > >     >>> flow labels and the default behavior in Linux should be
> changed,
> >     > >     >>> however I do not believe there's a valid claim of
> non-conformance that
> >     > >     >>> motivates removal of the feature that is already
> deployed.
> >     > >     >>>
> >     > >     >>> Tom
> >     > >     >>>
> >     > >     >>>
> >     > >     >>>
> >     > >     >>>
> >     > >     >>>>
> >     > >     >>>>
> >     > >     >>>>
> >     > >     >>>>> The feature is useful in large datacenter networks,
> like
> >     > >     >>>>> pparently Facebook where the patches originate, since
> information
> >     > >     >>>>> discerned by TCP can opportunistically be applied to
> route selection.
> >     > >     >>>>> The practical issue is that there are stateful devices
> like firewalls
> >     > >     >>>>> that require consistent routing in the network in
> which case changing
> >     > >     >>>>> the flow label can confuse them. As I mentioned, the
> original intent
> >     > >     >>>>> was that the flow label randomization feature should
> be opt-in instead
> >     > >     >>>>> of on by default.
> >     > >     >>>>
> >     > >     >>>> So... where is the "source" of the packet that would be
> "modulating" the FL?
> >     > >     >>>>
> >     > >     >>>> Thanks,
> >     > >     >>>> --
> >     > >     >>>> Fernando Gont
> >     > >     >>>> e-mail: [e-mail] <mailto:
> [e-mail]> <mailto:[e-mail] <mailto:
> [e-mail]>> || [e-mail] <mailto:
> [e-mail]> <mailto:[e-mail] <mailto:
> [e-mail]>>
> >     > >     >>>> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE
> A9EF D076 FFF1
> >     > >     >>>>
> >     > >     >>>>
> >     > >     >>>>
> >     > >     >>>
> >     > >     >>> _______________________________________________
> >     > >     >>> v6ops mailing list
> >     > >     >>> [e-mail] <mailto:[e-mail]> <mailto:
> [e-mail] <mailto:[e-mail]>>
> >     > >     >>> https://www.ietf.org/mailman/listinfo/v6ops
> >     > >     >>>
> >     > >
> >     > >
> >     > > _______________________________________________
> >     > > v6ops mailing list
> >     > > [e-mail] <mailto:[e-mail]>
> >     > > https://www.ietf.org/mailman/listinfo/v6ops
> >     > >
> >     >
> >
> >     _______________________________________________
> >     v6ops mailing list
> >     [e-mail] <mailto:[e-mail]>
> >     https://www.ietf.org/mailman/listinfo/v6ops
> >
> >
> >
> > --
> > Best regards,
> > Alexander Azimov
>
>

-- 
Best regards,
Alexander Azimov