Re: [v6ops] RFC 6437 - IPv6 flow label usage in context of ECMP

Brian E Carpenter <> Wed, 09 December 2020 04:20 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 89F583A0B0C; Tue, 8 Dec 2020 20:20:01 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.099
X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, NICE_REPLY_A=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id I0-ZI7oigeOU; Tue, 8 Dec 2020 20:19:59 -0800 (PST)
Received: from ( [IPv6:2607:f8b0:4864:20::429]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 65B943A0B0B; Tue, 8 Dec 2020 20:19:59 -0800 (PST)
Received: by with SMTP id s21so159335pfu.13; Tue, 08 Dec 2020 20:19:59 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language:content-transfer-encoding; bh=VdixiaI29dXdFnUyvvdEeJPfwyOCKLuW+lesoN5pigc=; b=uRO0Cy2G+icQvQ7DvThgBhzauQJ+hOQdWkefjKXjvfIa8NKaG4NN+OSeRgMUr/eDu4 rHADxjXhD16+Ck1RMCtWnaJGcaNw1LnhUXSVnOte62mNAprtOIG7TL5fDvzWiOrhZ7ph R+FHXuz+kpuUuuSr+hIIeXdqlAQFbNVOAetZYZ2cpsZCZsNWDALhb591ur49yu9Ly+lT 2JBI2Oo/Z5Fbp/xH2dju8X/a4U/zcrtX7RZ5in8+JcmL3tLvm6WBWOo5ij5Vk41mS6bM 3I6WZx6hPQzzn3zpgTrlcrQ4iYuz2X6pzGgOQcorEhhchPo8bDV+bTytnZSE/IzvMijo u4Sw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=VdixiaI29dXdFnUyvvdEeJPfwyOCKLuW+lesoN5pigc=; b=GazV/blofOrc+BS4i2empLnfdzOS7zxacAFBgixiGiiAghO3j4YOuIqxd0ceaG3fdQ //OgBfLK0bK3K+H9YkIf6rVOqwqoikZ0cZPY1iJUZBIEvz8jjIZZLBGCP4a3A9tI6gu1 3Ssrpqz1OC7eO5sAz+BMT2J6KzrxL79QDQA+qhVG76nGSgoTVptO3ELkfaT6hpbuzsGd Xtgw0AGwoCmEPaVkG2ZIu7fhmxL4c44NH3J+uQUn0gLSnl545UdneZaO/u3xFCRdx9tQ H3cNt5rp4azPQGzwCa7eY65W0YjSkta2woWCzJ6dVXhbZbFw6TIRm0InW0Wmq7x53XS5 5Jdw==
X-Gm-Message-State: AOAM532coAgq1bCiS9r3auS+g5L5o5NrK+IVWtmny4LIaXwNH4y4Xgu+ wJW5c7bD7AbZo3iQrYSNGV6vDEO9rIZ3Uw==
X-Google-Smtp-Source: ABdhPJwd4pg58Z4GKEvaWI2FuHfUBrDgfhG3t4TXKXGC0hRcYFejK3AXDBQSPv4+EUhE7fBuYzckpA==
X-Received: by 2002:aa7:9af4:0:b029:19d:975a:3ef2 with SMTP id y20-20020aa79af40000b029019d975a3ef2mr406439pfp.5.1607487598241; Tue, 08 Dec 2020 20:19:58 -0800 (PST)
Received: from [] ([]) by with ESMTPSA id j19sm408368pff.74.2020. (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 08 Dec 2020 20:19:57 -0800 (PST)
To: Gyan Mishra <>, IPv6 IPv6 List <>, IPv6 Operations <>
References: <>
From: Brian E Carpenter <>
Message-ID: <>
Date: Wed, 9 Dec 2020 17:19:52 +1300
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.1
MIME-Version: 1.0
In-Reply-To: <>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
Archived-At: <>
Subject: Re: [v6ops] RFC 6437 - IPv6 flow label usage in context of ECMP
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: v6ops discussion list <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 09 Dec 2020 04:20:02 -0000


I'm sorry, I don't understand what your question is. The reason that RFC6437 calls for a hash algorithm with certain properties when setting the flow label is precisely so that the reulting flow label is suitable input to any further ECMP or LAG hash, with or without the traditional n-tuple being available.

There is background info in RFC6436, and specific uses cases in RFC6438 and RFC7098. Maybe you can refine your question after reviewing those RFCs?

   Brian Carpenter

On 09-Dec-20 09:22, Gyan Mishra wrote:
> Dear v6ops & 6man,
> I have a question related to how IPv6 flow label works in the context of router ECMP load balancing characteristics resulting in even or uneven load balancing based on the understanding flow based or n-tuple ECMP hash based per packet algorithm.
> My understanding of RFC 6437 is that the IPv6 flow label is a means of classification of packets that are part of the same flow from a particular flow for use in QOS marking.
> From a load balancing perspective the flow key could be used to identify a flow and path taken on routes in a case of load balancing ECMP hash.  
> The goal of IPv6 flow label is for ordering of packets to be part of the same network path in an ECMP hash resulting in flow based load balancing identical to what exists today with IPv4.  
> What is confusing is if flow label 5 tuple is input to a ECMP hash algorithm similar to LAG and MLAG which yields  per packet load balancing based on n-tuple hash implemented by most vendors.  This would be preferable if as it would yield even load balancing.
> Or is the flow label goal to maintain the per flow load balancing and use for QOS packet classification so all packets that are part of a particular flow use the same ECMP path.  In this case the n-tuple hash key generated is just to identify which path the flow had taken on a router but not an input to a hash algorithm function as exists with LAG or MLAG.  
> The difference is significant as far as load balancing as a n-tuple input to ECMP hash algorithm achieves close to 50/50 load balancing where per flow load balancing can be uneven as it’s based on flows and thus from a network standpoint high bandwidth flows may hash to one ECMP path and low bandwidth flows may hash to another ECMP path.
> So in general with parallel links bundling LAG and MLAG is always preferred method for load balancing distribution over ECMP.  However L3 ECMP always exists and is optimal to assist further load balancing which often times exists in conjunction with LAG and MLAG.
> Excerpt from a doc I found online related to IPv6 flow label operation:
> A flow is a sequence of packets sent from a particular source to a  particular (unicast or multicast) destination for which the source  desires special handling by the intervening routers. The nature of  that special handling might be conveyed to the routers by a control  protocol, such as a resource reservation protocol, or by information  within the flow's packets themselves, e.g., in a hop-by-hop option.  The details of such control protocols or options are beyond the scope of this document.
> There may be multiple active flows from a source to a destination, as well as traffic that is not associated with any flow. A flow is uniquely identified by the combination of a source address and a  non-zero flow label. Packets that do not belong to a flow carry a  flow label of zero.
> A flow label is assigned to a flow by the flow's source node. New flow labels must be chosen (pseudo-)randomly and uniformly from the  range 1 to FFFFF hex. The purpose of the random allocation is to  make any set of bits within the Flow Label field suitable for use as  a hash key by routers, for looking up the state associated with the  flow.
> All packets belonging to the same flow must be sent with the same  source address, destination address, and flow label. If any of those packets includes a Hop-by-Hop Options header, then they all must be  originated with the same Hop-by-Hop Options header contents  (excluding the Next Header field of the Hop-by-Hop Options header).  If any of those packets includes a Routing header, then they all must be originated with the same contents in all extension headers up to  and including the Routing header (excluding the Next Header field in  the Routing header). The routers or destinations are permitted, but  not required, to verify that these conditions are satisfied. If a  violation is detected, it should be reported to the source by an ICMP Parameter Problem message, Code 0, pointing to the high-order octet  of the Flow Label field (i.e., offset 1 within the IPv6 packet). 
> The maximum lifetime of any flow-handling state established along a flow's path must be specified as part of the description of the  state-establishment mechanism, e.g., the resource reservation  protocol or the flow-setup hop-by-hop option. A source must not re-  use a flow label for a new flow within the maximum lifetime of any  flow-handling state that might have been established for the prior  use of that flow label.
> Excerpt from Section 2 below:
>    The 20-bit Flow Label field in the IPv6 header [RFC2460 <>] is used by a
>    node to label packets of a flow.  A Flow Label of zero is used to
>    indicate packets that have not been labeled.  Packet classifiers can
>    use the triplet of Flow Label, Source Address, and Destination
>    Address fields to identify the flow to which a particular packet
>    belongs.  Packets are processed in a flow-specific manner by nodes
>    that are able to do so in a stateless manner or that have been set up
>    with flow-specific state.  The nature of the specific treatment and
>    the methods for flow state establishment are out of scope for this
>    specification.
>    Flow label values should be chosen such that their bits exhibit a
>    high degree of variability, making them suitable for use as part of
>    the input to a hash function used in a load distribution scheme.  At
>    the same time, third parties should be unlikely to be able to guess
>    the next value that a source of flow labels will choose.
>    o  Forwarding nodes such as routers and load distributors MUST NOT
>       depend only on Flow Label values being uniformly distributed.  In
>       any usage such as a hash key for load distribution, the Flow Label
>       bits MUST be combined at least with bits from other sources within
>       the packet, so as to produce a constant hash value for each flow
>       and a suitable distribution of hash values across flows.
>       Typically, the other fields used will be some or all components of
>       the usual 5-tuple.  In this way, load distribution will still
>       occur even if the Flow Label values are poorly distributed.
>    Although uniformly distributed flow label values are recommended
>    below, and will always be helpful for load distribution, it is unsafe
>    to assume their presence in the general case, and the use case needs
>    to work even if the flow label value is zero.
>    As a general practice, packet flows should not be reordered, and the
>    use of the Flow Label field does not affect this.  In particular, a
>    Flow label value of zero does not imply that reordering is
>    acceptable.
> Kind Regards 
> Gyan
> -- 
> <>
> *Gyan Mishra*
> /Network Solutions A//rchitect /
> /M 301 502-1347
> 13101 Columbia Pike 
> /Silver Spring, MD
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> Administrative Requests:
> --------------------------------------------------------------------