Re: [v6ops] [tcpm] Flow Label Load Balancing
Yuchung Cheng <ycheng@google.com> Thu, 26 November 2020 16:39 UTC
Return-Path: <ycheng@google.com>
X-Original-To: v6ops@ietfa.amsl.com
Delivered-To: v6ops@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D0C783A14AB for <v6ops@ietfa.amsl.com>; Thu, 26 Nov 2020 08:39:39 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -15.6
X-Spam-Level:
X-Spam-Status: No, score=-15.6 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, URI_DOTEDU=1.999, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=no autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id aj1s5SlIRG1E for <v6ops@ietfa.amsl.com>; Thu, 26 Nov 2020 08:39:37 -0800 (PST)
Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 81B803A14A9 for <v6ops@ietf.org>; Thu, 26 Nov 2020 08:39:36 -0800 (PST)
Received: by mail-wm1-x330.google.com with SMTP id a186so2695198wme.1 for <v6ops@ietf.org>; Thu, 26 Nov 2020 08:39:36 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=edWfoAmVLZbYlmpbCJONOdlNJLI3bMwOAVJSH9ZaLCQ=; b=iUoi9IwLGLbG5Rpoemq2UQDon2zWNuLsqbnW3WmSzRecaosWeumRIntXHtuIjTBcsR duEZLqgZXuBCkGjGSD8BYjUjPYO87CU8nfbxKqzya4QP04zGCwq8y24OlhwQzOQh7dIL EYpnVfzIe7Jx2jh3f0HxJtU2WUKyDxJLJJxS0thyA94UfP9+FXCqvcZl4wJ/cYqI+2mO Rx02+uRRwYDbHzUMhU1ypuVKlagMuwJHIkGY/l37kRz3iM4XdtxS3NohJZnavzy1z3ub MYeDooXhpqTSmrYa/+FdOnruWCWJY95yiG9TLGyDOKrUbAXXLCk71ejAcjZ4VqSytsjn I+2w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=edWfoAmVLZbYlmpbCJONOdlNJLI3bMwOAVJSH9ZaLCQ=; b=lEIbojvHH6NS9d9HN1oqoSEq0qy5gq+lJKSlIsqdQ6ssAO9o5q161CMZxsviwQvFzk t5n2TmCbGiE5nqTrVTyR6CAbz5EgldUh0JU6P4nXR4CgJx2Fpm+KQSKlw3eskXQAdosS QFa4Vl/hH/Jzwjh0qWPrrlC7k1KrtH8TOMo1BMfaBvhJX0POBZ/e5lGcFGL9A4c/6p87 aLZqPE27sQXoDXq/ftkwOxfhxSOGpF6yM06CaP8Dx+2jt5BTalVdmM2+Yc3m+DQMfVk8 /sR8MbPxa9p7JLySVckxWH/XdFxIGztYOPlwl+52u2fy5QAbKu972nTUzc/ObL4uv5XU 9gpA==
X-Gm-Message-State: AOAM532ibAZME8D9WCkmo+y+tLcWSqMcnh3gZQDNmjhRkA/Gc/jV8JyP 37xIOsNq27x++CB7zAwlSwc2SzULB7IvErqSpAor2Q==
X-Google-Smtp-Source: ABdhPJzqXxadN+VVfaKqKqKd6ElReYkMJOQ8O5ba/SnFB+r0DbboFc/wdwFUdnF2KFMXogyMhIABcSyHr2oNVDNz2sY=
X-Received: by 2002:a7b:cd11:: with SMTP id f17mr4215376wmj.127.1606408774663; Thu, 26 Nov 2020 08:39:34 -0800 (PST)
MIME-Version: 1.0
References: <CAEGSd=DY8t8Skor+b6LSopzecoUUzUZhti9s0kdooLZGxPEt+w@mail.gmail.com> <d29042a7-742b-a445-cf60-2773e5515ae5@gont.com.ar> <CALx6S37+1duoNGR3dZWesHsZvx15kX9wCWufPMh=esvMaSMF_g@mail.gmail.com> <63e7aad3-7094-7492-dbe4-3eefb5236de3@gont.com.ar> <CALx6S37t4jump6S-R5_xdo5DF+RnHtT4rU5-RuiC-2GQ0PXxkQ@mail.gmail.com> <239c4b67-1d9a-da00-7bb0-52019be1b7c1@joelhalpern.com> <CALx6S34uSAne_LyhrWDcjkR5p7MO6ggm_Ua_h+6nkX41S=Ge=A@mail.gmail.com> <a8aad80c-1a4b-4a86-4c13-7391e8513049@joelhalpern.com> <CALx6S36xYADqNrPp1A_Ohx48d7SdV2oFOgVFVV+y_tDbGQG6ug@mail.gmail.com> <abf9c63a-2f7e-6f28-34e8-b3e9598cd2b9@gmail.com> <CALx6S36PTVT49CQHdJNx88PHyYQS23WYP3A7Xw1-+f_tt4H3Gg@mail.gmail.com> <CAEGSd=BGqFTygTiAt1v-71W3RTpdyVqyYzD1vi9uKebPPMoE5Q@mail.gmail.com>
In-Reply-To: <CAEGSd=BGqFTygTiAt1v-71W3RTpdyVqyYzD1vi9uKebPPMoE5Q@mail.gmail.com>
From: Yuchung Cheng <ycheng@google.com>
Date: Thu, 26 Nov 2020 08:38:57 -0800
Message-ID: <CAK6E8=eB-aCxehme5z9zvvx68dZKgSp=P_psh=78Dy6PBAjq6Q@mail.gmail.com>
To: Alexander Azimov <a.e.azimov@gmail.com>
Cc: Tom Herbert <tom@herbertland.com>, IPv6 Operations <v6ops@ietf.org>, tcpm <tcpm@ietf.org>, Brian E Carpenter <brian.e.carpenter@gmail.com>
Content-Type: multipart/alternative; boundary="00000000000061231205b5053001"
Archived-At: <https://mailarchive.ietf.org/arch/msg/v6ops/qfARpZk2vjfRG3skvVXaBjBfdBQ>
Subject: Re: [v6ops] [tcpm] Flow Label Load Balancing
X-BeenThere: v6ops@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: v6ops discussion list <v6ops.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/v6ops>, <mailto:v6ops-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/v6ops/>
List-Post: <mailto:v6ops@ietf.org>
List-Help: <mailto:v6ops-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/v6ops>, <mailto:v6ops-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 26 Nov 2020 16:39:40 -0000
On Thu, Nov 26, 2020 at 1:34 AM Alexander Azimov <a.e.azimov@gmail.com> wrote: > Dear colleagues, > > We started discussing an incorrect default behavior, and Tom has already > confirmed that it will be fixed. > > Later the thread turned into an argument if the flow label can be changed > during connection/flow lifetime according to current RFC documents, though > these documents can be updated. This looks a bit weird for me because I > always thought that it is the IETF community's responsibility to document > proper solutions. If something undocumented but worthy happens in the > industry - IETF should catch up. So, I would like to get back to the > discussion of reasons and their safety. > > The general idea of changing the routing path upon network > outage/degradation looks obvious. Getting kind of source-based routing can > significantly reduce the reaction time and improve end-user experience. The > flow label has a perfect match here: transparent for the application, set > by the source, not part of 5-tuple, while it can be used in the load > balancing. IMO poorly documented. I would like to learn your feedback for > the next wording: > > Let say that the flow label is a hash of values from the IP packet's > 5-tuple and random number. Then > > 1. In the case of SYN or SYN-ACK retransmission flow label SHOULD be > recalculated. > 2. In the case of RTO timeout expiration in the established TCP session > the flow label MAY be recalculated. This setting MUST be switched off by > default. > 3. Otherwise flow label SHOULD be preserved unchanged. > For completion, we might want to discuss reverse-path failure that RTO-triggered rehash in (1)(2) alone can not repair the connection in established or handshake phases. https://patchwork.ozlabs.org/project/netdev/patch/20180829215356.235336-1-ycheng@google.com/ I also agree with Tom that the general feature is very useful in keeping TCP connected with partial route failures, but the default should be an opt-in. > > The first point stands for redirecting connection from the degraded path > before the connection is established, and it looks safe. The second one can > also improve performance if it is set on the server-side. Please comment if > you see security flaws in such a design. > > чт, 26 нояб. 2020 г. в 03:16, Tom Herbert <tom@herbertland.com>: > >> On Wed, Nov 25, 2020 at 3:33 PM Brian E Carpenter >> <brian.e.carpenter@gmail.com> wrote: >> > >> > I'm not Joel, but I did once spend some time grepping RFCs to find out >> whether >> > "flow" or "microflow" was the preferred term. In RFC2474, which is >> normative, >> > we have: >> > >> > Microflow: a single instance of an application-to-application flow of >> > packets which is identified by source address, destination address, >> > protocol id, and source port, destination port (where applicable). >> > >> > But in the flow label work we explicitly avoided being that precise, and >> > did not use the term "microflow". There might be some load balancing >> > scenarios where you want a broader definition, even including >> bidirectional >> > flows. There are expired drafts on that topic: >> > draft-tarreau-extend-flow-label-balancing >> > draft-wang-6man-flow-label-reflection >> > >> Brian, >> >> Random Packet Spraying >> ( >> https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.297.529&rep=rep1&type=pdf >> ) >> is an interesting idea where packets for a single connection are >> purposely distributed across multiple paths for load distribution. Per >> packet randomized flow labels with flow label aware ECMP makes this >> quite easy to do without requiring any special support in switches >> like you'd need with IPv4. I'm not necessarily advocating this, but it >> does highlight one potential use case of having a flow label that >> doesn;t have rigidly defined requirements on the host. >> >> Tom >> >> > Regards >> > Brian >> > >> > On 26-Nov-20 11:13, Tom Herbert wrote: >> > > Joel, is there a normative definition of a flow? >> > > >> > > On Wed, Nov 25, 2020, 1:27 PM Joel M. Halpern <jmh@joelhalpern.com >> <mailto:jmh@joelhalpern.com>> wrote: >> > > >> > > No, as Brian says, there are escape clauses in the flow >> definitions. >> > > >> > > But changing the flow label due to traffic problems does not >> correspond >> > > to packets being in actually different flows. >> > > If one were using UDP, and mixing loss sensitive packets with loss >> > > insensitive packets for a special application, sure, one could >> use two >> > > flow labels. But that is not what you are describing. >> > > >> > > Yours, >> > > Joel >> > > >> > > On 11/25/2020 3:05 PM, Tom Herbert wrote: >> > > > Joel, >> > > > >> > > > Is there an RFC that clearly and unambiguously states that a >> host MUST >> > > > use the same flow label for the lifetime _and_ clearly defines >> exactly >> > > > what a flow is with respect to such a requirement (for >> instance, how >> > > > would you define a flow and enforce such a requirement in UDP? >> IPsec? >> > > > other encapsulations?). If there is such a requirement then >> we'll >> > > > change the code to be conformant. >> > > > >> > > > Tom >> > > > >> > > > On Wed, Nov 25, 2020 at 12:39 PM Joel M. Halpern < >> jmh@joelhalpern.com <mailto:jmh@joelhalpern.com>> wrote: >> > > >> >> > > >> This kind of thing is why, as I understand it, MPTCP has >> discovery >> > > >> mechanisms ot know if both sides use it, and can select >> alternative >> > > >> addresses for communication. >> > > >> >> > > >> Trying to guess flow labels that might avoid a problem because >> it might >> > > >> be an ECMP problem, is just flailing about. Not a good design >> for >> > > >> operational protocols. >> > > >> >> > > >> And in general, designing protocols around "I know exactly >> what is going >> > > >> on" (the requirement for what you describe that goes well >> beyond just >> > > >> "limited domains") is also a recipe for failure. >> > > >> >> > > >> The Flow Label RFCs are actually very explicit that a flow >> label is >> > > >> supposed to be stable for the life of the flow. Otherwise, it >> isn't a >> > > >> flow label. >> > > >> >> > > >> Yours, >> > > >> Joel >> > > >> >> > > >> On 11/25/2020 2:35 PM, Tom Herbert wrote: >> > > >>> Hi Fernando, comments in line... >> > > >>> >> > > >>> On Wed, Nov 25, 2020 at 12:13 AM Fernando Gont < >> fernando@gont.com.ar <mailto:fernando@gont.com.ar>> wrote: >> > > >>>> >> > > >>>> Hi, Tom, >> > > >>>> >> > > >>>> On 24/11/20 16:43, Tom Herbert wrote: >> > > >>>> [....] >> > > >>>>> Modulating the flow label is a means to affect the routing >> of packets >> > > >>>>> through the network that uses flow labels as input to the >> ECMP hash. >> > > >>>> >> > > >>>> What's the point? >> > > >>>> >> > > >>>> 1) You cannot tell *if* the FL is being used. >> > > >>>> >> > > >>> Generally true, but in a limited domain this information >> could be >> > > >>> discerned. I'd note that it's also generally true that we >> don't know >> > > >>> if there is a load balancer or stateful firewall in the path >> that >> > > >>> requires consistent routing, but in a limited domain we could >> know >> > > >>> that also. >> > > >>> >> > > >>>> 2) Changing the FL does not necessarily mean that packets >> will employ a >> > > >>>> different link. >> > > >>> >> > > >>> It's an opportunistic mechanism. If a connection is failing >> and we get >> > > >>> a better path that fixes it by simply changing the flow label >> then >> > > >>> what's the harm? >> > > >>> >> > > >>>> >> > > >>>> 3) If the network is failing, shouldn't you handle this via >> routing? >> > > >>>> >> > > >>> Sure, but then that requires an out of band feedback loop >> from a TCP >> > > >>> implementation to the network infrastructure to indicate >> there is a >> > > >>> problem and then the network needs to respond. That's >> significant >> > > >>> infrastructure and higher reaction time than doing something >> in TCP >> > > >>> and IP. Think of modulating the flow label is an inexpensive >> form of >> > > >>> source routing within a limited domain that doesn't need any >> > > >>> infrastructure or heavyweight protocols or something like >> segment >> > > >>> routing. >> > > >>> >> > > >>>> >> > > >>>> >> > > >>>>> The basic idea is that the flow label associated with a >> connection is >> > > >>>>> randomly changed when the stack observes that the >> connection is >> > > >>>>> failing (e.g. and an RTO). There is nothing in the specs >> that prevents >> > > >>>>> this since the source is at liberty to set the flow label >> as it sees >> > > >>>>> fit. >> > > >>>> >> > > >>>> The FL is expected to remain constant for the life of a >> flow. A >> > > >>>> retransmitted packet is part of the same flow as the >> > > >>>> originally-transmitted packet. So this seems to be >> contradicting the >> > > >>>> very specification of the FL. >> > > >>>> >> > > >>>> For instance, If a RTO for a flow causes the FL to change, >> then one may >> > > >>>> possibly argue that the FL is not naming/labeling what is >> said/expected >> > > >>>> to be anming/labeling. >> > > >>> >> > > >>> Specifically, RFC6437 states: >> > > >>> >> > > >>> "It is therefore RECOMMENDED that source hosts support the >> flow label >> > > >>> by setting the flow label field for all packets of a given >> flow to the >> > > >>> same value chosen from an approximation to a discrete uniform >> > > >>> distribution." >> > > >>> >> > > >>> So that is clearly a just recommendation, and not a >> requirement (and >> > > >>> definitely not a MUST). Furthermore, RFC6437 states: >> > > >>> >> > > >>> "A forwarding node MUST either leave a non-zero flow label >> value >> > > >>> unchanged or change it only for compelling operational >> security >> > > >>> reasons as described in Section 6.1." >> > > >>> >> > > >>> So there's no guarantee in the protocol specs that flow >> labels are >> > > >>> consistent for the life of the connection, which means that >> the >> > > >>> network cannot assume that and thus it would be incorrect if >> the >> > > >>> network tried to enforce flow label consistency as a protocol >> > > >>> requirement. As I said, it is prudent to try to be consistent >> with >> > > >>> flow labels and the default behavior in Linux should be >> changed, >> > > >>> however I do not believe there's a valid claim of >> non-conformance that >> > > >>> motivates removal of the feature that is already deployed. >> > > >>> >> > > >>> Tom >> > > >>> >> > > >>> >> > > >>> >> > > >>> >> > > >>>> >> > > >>>> >> > > >>>> >> > > >>>>> The feature is useful in large datacenter networks, like >> > > >>>>> pparently Facebook where the patches originate, since >> information >> > > >>>>> discerned by TCP can opportunistically be applied to route >> selection. >> > > >>>>> The practical issue is that there are stateful devices like >> firewalls >> > > >>>>> that require consistent routing in the network in which >> case changing >> > > >>>>> the flow label can confuse them. As I mentioned, the >> original intent >> > > >>>>> was that the flow label randomization feature should be >> opt-in instead >> > > >>>>> of on by default. >> > > >>>> >> > > >>>> So... where is the "source" of the packet that would be >> "modulating" the FL? >> > > >>>> >> > > >>>> Thanks, >> > > >>>> -- >> > > >>>> Fernando Gont >> > > >>>> e-mail: fernando@gont.com.ar <mailto:fernando@gont.com.ar> >> || fgont@si6networks.com <mailto:fgont@si6networks.com> >> > > >>>> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF >> D076 FFF1 >> > > >>>> >> > > >>>> >> > > >>>> >> > > >>> >> > > >>> _______________________________________________ >> > > >>> v6ops mailing list >> > > >>> v6ops@ietf.org <mailto:v6ops@ietf.org> >> > > >>> https://www.ietf.org/mailman/listinfo/v6ops >> > > >>> >> > > >> > > >> > > _______________________________________________ >> > > v6ops mailing list >> > > v6ops@ietf.org >> > > https://www.ietf.org/mailman/listinfo/v6ops >> > > >> > >> >> _______________________________________________ >> v6ops mailing list >> v6ops@ietf.org >> https://www.ietf.org/mailman/listinfo/v6ops >> > > > -- > Best regards, > Alexander Azimov > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm >
- [v6ops] Flow Label Load Balancing Alexander Azimov
- Re: [v6ops] Flow Label Load Balancing Tom Herbert
- Re: [v6ops] Flow Label Load Balancing Alexander Azimov
- Re: [v6ops] Flow Label Load Balancing Fernando Gont
- Re: [v6ops] Flow Label Load Balancing Alexander Azimov
- Re: [v6ops] Flow Label Load Balancing Tom Herbert
- Re: [v6ops] Flow Label Load Balancing Brian E Carpenter
- Re: [v6ops] Flow Label Load Balancing Tom Herbert
- Re: [v6ops] Flow Label Load Balancing Fernando Gont
- Re: [v6ops] Flow Label Load Balancing Fernando Gont
- Re: [v6ops] Flow Label Load Balancing Tom Herbert
- Re: [v6ops] Flow Label Load Balancing Joel M. Halpern
- Re: [v6ops] Flow Label Load Balancing Brian E Carpenter
- Re: [v6ops] Flow Label Load Balancing Tom Herbert
- Re: [v6ops] Flow Label Load Balancing Joel M. Halpern
- Re: [v6ops] Flow Label Load Balancing Tom Herbert
- Re: [v6ops] Flow Label Load Balancing Brian E Carpenter
- Re: [v6ops] Flow Label Load Balancing Tom Herbert
- Re: [v6ops] Flow Label Load Balancing Alexander Azimov
- Re: [v6ops] [tcpm] Flow Label Load Balancing Yuchung Cheng
- Re: [v6ops] Flow Label Load Balancing Brian E Carpenter
- Re: [v6ops] Flow Label Load Balancing Alexander Azimov
- Re: [v6ops] Flow Label Load Balancing Brian E Carpenter
- Re: [v6ops] Flow Label Load Balancing Fernando Gont
- Re: [v6ops] Flow Label Load Balancing Fernando Gont
- Re: [v6ops] Flow Label Load Balancing Fernando Gont
- Re: [v6ops] Flow Label Load Balancing Tom Herbert
- Re: [v6ops] Flow Label Load Balancing Mark Smith
- Re: [v6ops] Flow Label Load Balancing Joseph Touch
- Re: [v6ops] Flow Label Load Balancing Fernando Gont
- Re: [v6ops] Flow Label Load Balancing Fernando Gont
- Re: [v6ops] Flow Label Load Balancing Mark Smith
- Re: [v6ops] Flow Label Load Balancing Fernando Gont
- Re: [v6ops] [tcpm] Flow Label Load Balancing Alexander Azimov
- Re: [v6ops] [tcpm] Flow Label Load Balancing Michael Tuexen
- Re: [v6ops] [tcpm] Flow Label Load Balancing Tom Herbert
- Re: [v6ops] [tcpm] Flow Label Load Balancing Fernando Gont
- Re: [v6ops] [tcpm] Flow Label Load Balancing Tom Herbert
- Re: [v6ops] [tcpm] Flow Label Load Balancing Fernando Gont
- Re: [v6ops] [tcpm] Flow Label Load Balancing Erik Kline
- Re: [v6ops] [tcpm] Flow Label Load Balancing Fernando Gont
- Re: [v6ops] [tcpm] Flow Label Load Balancing Tom Herbert
- Re: [v6ops] [tcpm] Flow Label Load Balancing Fernando Gont
- Re: [v6ops] [tcpm] Flow Label Load Balancing Tom Herbert
- Re: [v6ops] [tcpm] Flow Label Load Balancing Brian E Carpenter
- Re: [v6ops] [tcpm] Flow Label Load Balancing Tom Herbert
- Re: [v6ops] [tcpm] Flow Label Load Balancing Fernando Gont