Re: [v6ops] Flow Label Load Balancing

Brian E Carpenter <> Wed, 25 November 2020 20:04 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 7456B3A1C64; Wed, 25 Nov 2020 12:04:50 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.099
X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id FGikAEzXeGsZ; Wed, 25 Nov 2020 12:04:47 -0800 (PST)
Received: from ( [IPv6:2607:f8b0:4864:20::534]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id A31523A1C63; Wed, 25 Nov 2020 12:04:47 -0800 (PST)
Received: by with SMTP id t37so3360256pga.7; Wed, 25 Nov 2020 12:04:47 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=42KFVBc8+ZWoF+JqgQiO4syNYF9y+7EKA+BfDce5prs=; b=p0nwqbnzoS0Kzdv0SJYKxq8qfCOJIx2wTDYVQ6sNBIYtMQ7DOk7KF+GAm2c3VcSzcW J5PX4LeXKtll/YPNybnblPgzEaBacea+w0eVSBbDiNtcuQYtWLeJmI3WRfsdfdJ1DZRq mSOshJh5ZiPp4s8uay9uD+dbqhOaotX9fgTc4V7fiXONnieN+l1ow57Dm1gTy1o3RxOS W8VEJNBgrrJdicJSu1SXlkiHzcqW3GiD+ZBDF2UoFbSwA3CSG+F6gQJnCFbl5zfDk7Cg sXbDJVWoHZp36m7yBx2OFTzC914DSIslXWdxRMjmc6GI1clsBpPyT40v1Xn9tFmxhSiN 7QpA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=42KFVBc8+ZWoF+JqgQiO4syNYF9y+7EKA+BfDce5prs=; b=pG2N5BUkXGxqYJgcWj65TvIbnx9E4s6RiI4pdtQUBUsF4SkbrC1ds1Vd1QFbl6ku2W mXOfMjrbYuggcrKVolPgF/KVAjdIJgZKGg997kecW8hxAZ25OR5AAcrZwnSHBB6nakpA NFoMCblx4zlQ3Xf3JPbSY5uc84WVteJQ0mFrYSbtipuvSYMerI8pllCC8pYViZalHl1x Du1r84pR8BfUmGzp+vYA7TU6BySzYp9SxrKIUIKfSkiOmgy+f6B7NoaDuTG5YXK3NCgK 0jlIdQ/fRch6FVGexFTFCa+W5z2fSuLBHqbmAzAzgSRhcCZ+ncfGIMYIz724iKPq+nBG pUfA==
X-Gm-Message-State: AOAM530S38nfO9+rmWQyjje1dUeoT1kHQhQc6lJ67xZ5pY+nFeol73Rw TZkuuqkjLYMCR1pmAxaI0qL3EoKkiWwlNQ==
X-Google-Smtp-Source: ABdhPJyMBMxaHkYZ66l0Y2iytk3PaXXjZ9PdYsd+8ReFqvo7Lq/Qz2hVMJgQ+dWTNMF8bIufpr2T7g==
X-Received: by 2002:a62:5e81:0:b029:197:baa5:1792 with SMTP id s123-20020a625e810000b0290197baa51792mr4501345pfb.80.1606334686622; Wed, 25 Nov 2020 12:04:46 -0800 (PST)
Received: from [] ([]) by with ESMTPSA id y5sm3834165pjt.42.2020. (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Nov 2020 12:04:45 -0800 (PST)
To: "Joel M. Halpern" <>, Tom Herbert <>
Cc: IPv6 Operations <>, tcpm <>
References: <> <> <> <> <> <>
From: Brian E Carpenter <>
Message-ID: <>
Date: Thu, 26 Nov 2020 09:04:42 +1300
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1
MIME-Version: 1.0
In-Reply-To: <>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Archived-At: <>
Subject: Re: [v6ops] Flow Label Load Balancing
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: v6ops discussion list <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 25 Nov 2020 20:04:51 -0000

Hi Joel,

On 26-Nov-20 08:39, Joel M. Halpern wrote:
> This kind of thing is why, as I understand it, MPTCP has discovery 
> mechanisms ot know if both sides use it, and can select alternative 
> addresses for communication.
> Trying to guess flow labels that might avoid a problem because it might 
> be an ECMP problem, is just flailing about.  Not a good design for 
> operational protocols.
> And in general, designing protocols around "I know exactly what is going 
> on"  (the requirement for what you describe that goes well beyond just 
> "limited domains") is also a recipe for failure.
> The Flow Label RFCs are actually very explicit that a flow label is 
> supposed to be stable for the life of the flow.  Otherwise, it isn't a 
> flow label.

That's correct. However, the first paragraph of RFC6437 tells you that
it's the prerogative of the source of a flow to decide what is a flow.
It even says "a flow is not necessarily 1:1 mapped to a transport
connection." Therefore, Tom is correct that a source is conformant even
if it changes its mind halfway through a TCP connection. However, it's
only safe to do that if the source also knows that no harm will result,
which it cannot know in the general case. In a limited domain, the source
could know that there is no server load balancing in place, in which case
changing the label is not automatically damaging.

Thus, this is not something that should ever become a default behaviour,
and I think this thread began with a complaint that in some o/s release
it was the default.


> Yours,
> Joel
> On 11/25/2020 2:35 PM, Tom Herbert wrote:
>> Hi Fernando, comments in line...
>> On Wed, Nov 25, 2020 at 12:13 AM Fernando Gont <> wrote:
>>> Hi, Tom,
>>> On 24/11/20 16:43, Tom Herbert wrote:
>>> [....]
>>>> Modulating the flow label is a means to affect the routing of packets
>>>> through the network that uses flow labels as input to the ECMP hash.
>>> What's the point?
>>> 1) You cannot tell *if* the FL is being used.
>> Generally true, but in a limited domain this information could be
>> discerned. I'd note that it's also generally true that we don't know
>> if there is a load balancer or stateful firewall in the path that
>> requires consistent routing, but in a limited domain we could know
>> that also.
>>> 2) Changing the FL does not necessarily mean that packets will employ a
>>> different link.
>> It's an opportunistic mechanism. If a connection is failing and we get
>> a better path that fixes it by simply changing the flow label then
>> what's the harm?
>>> 3) If the network is failing, shouldn't you handle this via routing?
>> Sure, but then that requires an out of band feedback loop from a TCP
>> implementation to the network infrastructure to indicate there is a
>> problem and then the network needs to respond. That's significant
>> infrastructure and higher reaction time than doing something in TCP
>> and IP. Think of modulating the flow label is an inexpensive form of
>> source routing within a limited domain that doesn't need any
>> infrastructure or heavyweight protocols or something like segment
>> routing.
>>>> The basic idea is that the flow label associated with a connection is
>>>> randomly changed when the stack observes that the connection is
>>>> failing (e.g. and an RTO). There is nothing in the specs that prevents
>>>> this since the source is at liberty to set the flow label as it sees
>>>> fit.
>>> The FL is expected to remain constant for the life of a flow. A
>>> retransmitted packet is part of the same flow as the
>>> originally-transmitted packet. So this seems to be contradicting the
>>> very specification of the FL.
>>> For instance, If a RTO for a flow causes the FL to change, then one may
>>> possibly argue that the FL is not naming/labeling what is said/expected
>>> to be anming/labeling.
>> Specifically, RFC6437 states:
>> "It is therefore RECOMMENDED that source hosts support the flow label
>> by setting the flow label field for all packets of a given flow to the
>> same value chosen from an approximation to a discrete uniform
>> distribution."
>> So that is clearly a just recommendation, and not a requirement (and
>> definitely not a MUST). Furthermore, RFC6437 states:
>> "A forwarding node MUST either leave a non-zero flow label value
>> unchanged or change it only for compelling operational security
>> reasons as described in Section 6.1."
>> So there's no guarantee in the protocol specs that flow labels are
>> consistent for the life of the connection, which means that the
>> network cannot assume that and thus it would be incorrect if the
>> network tried to enforce flow label consistency as a protocol
>> requirement. As I said, it is prudent to try to be consistent with
>> flow labels and the default behavior in Linux should be changed,
>> however I do not believe there's a valid claim of non-conformance that
>> motivates removal of the feature that is already deployed.
>> Tom
>>>> The feature is useful in large datacenter networks, like
>>>> pparently Facebook where the patches originate, since information
>>>> discerned by TCP can opportunistically be applied to route selection.
>>>> The practical issue is that there are stateful devices like firewalls
>>>> that require consistent routing in the network in which case changing
>>>> the flow label can confuse them. As I mentioned, the original intent
>>>> was that the flow label randomization feature should be opt-in instead
>>>> of on by default.
>>> So... where is the "source" of the packet that would be "modulating" the FL?
>>> Thanks,
>>> --
>>> Fernando Gont
>>> e-mail: ||
>>> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
>> _______________________________________________
>> v6ops mailing list
> _______________________________________________
> v6ops mailing list