Re: [v6ops] [tcpm] Flow Label Load Balancing

Tom Herbert <> Thu, 03 December 2020 20:38 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 723FA3A0C35 for <>; Thu, 3 Dec 2020 12:38:54 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.898
X-Spam-Status: No, score=-1.898 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, SPF_HELO_NONE=0.001, SPF_NONE=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 1Pft6cIdT2DT for <>; Thu, 3 Dec 2020 12:38:52 -0800 (PST)
Received: from ( [IPv6:2a00:1450:4864:20::633]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 8CB6D3A0C46 for <>; Thu, 3 Dec 2020 12:38:52 -0800 (PST)
Received: by with SMTP id 7so5515343ejm.0 for <>; Thu, 03 Dec 2020 12:38:52 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=NIrav4tdAcakDCg0n7gnAvqr1Mn5aNrsxiqiFjmNwOU=; b=Eldg9cCt3VsCZru0t/MYk6TnPGRoZ0z9YbeRjy+CxTrIV1HR1Cla/V4roks76YPScc cdUuEx3cS/LvcKftzwEG6xXLrUhZAPCVteNlHMgLtC2ZEK4muNLxO6FVkqGRWkzvVMyS 1GKH8wGFvRUbgZnZFZX58Y/PsUjvfyjwnmunF1u0pjPE50cnRWQ0EE1GwqXOgtJUaT46 4MsR7Hv6NXy5f+eUT2wn4NbO5iWZcAYb3VsPuBG3MIftTFr9KRTpdrrrM6ILGLVoJ9ow StxAGPGBYvolI5FeERFXYVZX3TN9Z11pawKEm2b1OZCTC4uzRz5udT3f8+u8WBhLYzFY MUSg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=NIrav4tdAcakDCg0n7gnAvqr1Mn5aNrsxiqiFjmNwOU=; b=MwnD64AAZD3kYEuxT6ddUGY8CuFUOiIjkRC86ezMX/F/6KahHOjQTDxvWtVu+JZeTu 2DidKRgPCw/TOOit8/qmlY+fMdAagxtzmXLvofAW2iDY2usP7JeJJ36AzpkaLHG8jv5/ iWP3uMlG5Bzl9a7+QG2KyLDJZfba7iwdfWeBvfhwOZDu8If3KxORTrpvxW9KTJh6uuub 2ij/mSiDhuAeOKAf+yMhiALobpBOnh3cx4CWmebi/f/wXRE6efsRdiV7nGXi9TeFaQj2 xWXALT6iQ7KB5UJWdOertgK0sAo5qM13RDXVcSaUdTAvuLUIBYrGisfjMatFfZAVCGFO 3mvA==
X-Gm-Message-State: AOAM533/WI8SnF1yOHQzqQ+v77kM3hfWX4PdDRZ9vebwtt6H1Ia9Ubd/ TaI5MUlUNENuTbh8B6uHDIpspzz1+kauuZJ+Sum1Uw==
X-Google-Smtp-Source: ABdhPJzZzjuN8kyMl/2KzkfB5VN9VLQdYIkL7Ss22z65BHIUJXYdebynZmKFfPdsHOskTBjN54JrM0Dd+JC55vvg1Ow=
X-Received: by 2002:a17:906:3294:: with SMTP id 20mr4211114ejw.239.1607027930742; Thu, 03 Dec 2020 12:38:50 -0800 (PST)
MIME-Version: 1.0
References: <> <> <> <> <> <> <> <> <> <> <> <> <> <> <> <>
In-Reply-To: <>
From: Tom Herbert <>
Date: Thu, 3 Dec 2020 13:38:39 -0700
Message-ID: <>
To: Brian E Carpenter <>
Cc: Erik Kline <>, Fernando Gont <>, IPv6 Operations <>, tcpm <>, Fernando Gont <>
Content-Type: text/plain; charset="UTF-8"
Archived-At: <>
Subject: Re: [v6ops] [tcpm] Flow Label Load Balancing
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: v6ops discussion list <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Thu, 03 Dec 2020 20:38:55 -0000

On Thu, Dec 3, 2020 at 1:17 PM Brian E Carpenter
<> wrote:
> On 04-Dec-20 06:30, Tom Herbert wrote:
> > On Thu, Dec 3, 2020 at 8:43 AM Erik Kline <> wrote:
> >>
> >>>> Please include requirements for both hosts and network nodes. If there
> >>>> is going to the be a requirement that hosts MUST make a 1:1 mapping to
> >>>> a transport connection and the flow label MUST be consistent for the
> >>>> life of connection, then I would expect there to also be a requirement
> >>>> that the flow label MUST be immutable by all devices in the path-- in
> >>>> particular the requirement of RFC6437 would need to be updated: "A
> >>>> forwarding node MUST either leave a non-zero flow label value
> >>>> unchanged or change it only for compelling operational security
> >>>> reasons..."-- if hosts are not allowed to change a field they are
> >>>> primarily responsible for setting, then the network should not be
> >>>> allowed to modify it either.
> >>>
> >>> I don't recall _(of the top of my head) why we ended up making the field
> >>> mutable ("SHOULD NOT  be modified" rather than "MUST NOT.."). IIRC, it
> >>> had to do with allowing middle-boxes to mitigate FL-based covert
> >>> channels.  If that's the case (Brian CC'ed) then, in retrospective, that
> >>> seems to have been a bad idea.  -- I was part of the discussion, so I
> >>> take my share of blame. :-)
> >>
> >> My understanding/recollection is that since there were never any
> >> restrictions on modification of the flow label from the very beginning
> >> it wasn't super meaningful to try to close the barn door after all
> >> these years.
> >
> > Erik,
> >
> > There are two apropos statements in RFC6437:
> >
> > "There is no way to verify whether a flow label has been modified en
> > route or whether it belongs to a uniform distribution.  Therefore, no
> > Internet-wide mechanism can depend mathematically on unmodified and
> > uniformly distributed flow labels; they have a "best effort" quality."
> Yes, a MUST NOT is futile because it cannot be enforced.
> Also, the firewall folk  seemed very insistent that they would change
> the flow label (or zero it) because paranoia. So specifying that if they
> changed it, they should do it consistently, seemed better than a MUST NOT
> which they would definitely ignore.
> > "From an upper-layer viewpoint, a flow could consist of all packets in
> > one direction of a specific transport connection or media stream.
> > However, a flow is not necessarily 1:1 mapped to a transport
> > connection."
> >
> > So building a load balancer, or any other intermediate network device,
> > that *depends* on the flow label to be consistent for the life of a
> > transport connection and 1:1 mapped to a transport connection is
> > contrary to this guidance. The ship has already sailed on this in
> > deployment, and even if the protocol requirements were to be
> > sufficiently tightened that would take years to reach full effect and
> > that would preclude other use cases where modifying the flow label is
> > useful. If a stable 1:1 mapped identifier for transport ports is
> > needed in the network layer, it might make sense to create a new HBH
> > option for that. Personally, if I were building a load balancer I'd
> > use neither the flow label nor transport ports for entropy, I'd look
> > at using the destination address since 128 bits allows plenty of bits
> > of entropy, it's consistent per transport flow and always set in an
> > IPv6 packet.
> But that doesn't help with server load balancing. For ECMP/LAG in a
> transit network, the DA alone doesn't help either, if a particular
> destination is prone to very heavy flows for some reason.


Specifically, that is a problem for a host that uses a _single_ IPv6
address for all its connections. What I am suggesting is to give hosts
many addresses, which we already do anyway like when mobile devices
and data center servers are assigned /64s. If a host then randomly
chooses an address from that block for each connection it creates
thenl the address 2-tuple contains all the entropy needed for purposes
of load balancing, ECMP, etc. is contained in the source address.
Addresses are guaranteed to be present and plaintext in each packet so
no DPI required, they're as stable as any flow identifying information
in a packet can be and they offer much more entropy than sixteen bits
of port number of twenty bit flow label.

Since the block address assignments or already happening, this seems
more to be a host implementation matter where we perform address
selection for a new connection by random selection in the block. It's
on my wish list of things to implement :-)


> And of course
> the ports don't help if they're encrypted.
> There's no simple answer here.
>    Brian