Re: [v6ops] Flow Label Load Balancing

Alexander Azimov <a.e.azimov@gmail.com> Fri, 20 November 2020 09:06 UTC

Return-Path: <a.e.azimov@gmail.com>
X-Original-To: v6ops@ietfa.amsl.com
Delivered-To: v6ops@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D9D6F3A1AEA; Fri, 20 Nov 2020 01:06:01 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.097
X-Spam-Level:
X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AmiyMDZw35W1; Fri, 20 Nov 2020 01:06:00 -0800 (PST)
Received: from mail-ot1-x32f.google.com (mail-ot1-x32f.google.com [IPv6:2607:f8b0:4864:20::32f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 9B00D3A1B1F; Fri, 20 Nov 2020 01:05:59 -0800 (PST)
Received: by mail-ot1-x32f.google.com with SMTP id 92so5033909otd.5; Fri, 20 Nov 2020 01:05:59 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=agEg0jGErrQU2uilzY71nL9vrFGMYJJmt9ZMWE4illA=; b=eWvGGkdwyHMOt7yaqDvOt7D1r5r6784HDjt0JMbI24RIzakYNV1303KlBF+k/kgb+9 uMssSGKp25lMcCjQiltLnXU3K0OB4kjbtAfglZXTtIdT7qkdn2R691fJ6WpvHBbespeV mO8xLchVxO++UviJHTw1py3JZWIvmAqkxZZuqT9tx6ERB2Ta9cOm8NVwsNIJmRV/MOTi +ZqIAN97YgBe+5zRKtydu/xaJy+lAKqIDDaFVyoZIkBFS6+ZT3YgUiIVok530hiApUtY gx+Oq6C7lgELu8FdV3ZXPnXNS2HIxqmhn/UMAkOUDIbzBpZm/PTMi9fGIi3MLJLkUWp5 eaqg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=agEg0jGErrQU2uilzY71nL9vrFGMYJJmt9ZMWE4illA=; b=NtRPn+q8enWDQ9pmc1NeVpLuTXzQWZIDZtqmC6PfMsVhOUayzgr4S6nPV3XYKCdOGF syW5bsUT6anR7XrM1jyIOJ+s5mn2WVBubSCGBsscAxXtiAvep8gBFhNRyV6n8icwIZsz b00BECP5DNHiuNDLJY8gV9s0Bs7BbpAwvYSJ9me+2qwDy/MmvXTK366QflhxaX3PAPg4 7bT2dnsr+tX/o6ti4njnS8+AjstEwmR/sYTTE0c3mW/VbPfn7yz1pLfoQOq6XnU+tTvF aqOCLojnMB7v6Rjrm6M8U6camcKctx0vMfdVNIq+HbFN7SFTd3tGxm8C1RnJ89z+OwNs Rb9g==
X-Gm-Message-State: AOAM532+f2HfKgfb4/6O/a5dQBfV4gTG1mx7FOtubVcTK0unPC28pNXR n4/StRUDMaSQc0xWtDmEwOlmvEOsXjTU3cE3+HBI60lkjPg=
X-Google-Smtp-Source: ABdhPJzaQgRtXz4bVzyB3AOgxLbJjdAx98U5VdH4vO+hPDjg/7jkOljQKU223cG2vFug/+MXdEXlcNNWZwSGeTsT23c=
X-Received: by 2002:a9d:4c92:: with SMTP id m18mr13553418otf.248.1605863158810; Fri, 20 Nov 2020 01:05:58 -0800 (PST)
MIME-Version: 1.0
References: <CAEGSd=DY8t8Skor+b6LSopzecoUUzUZhti9s0kdooLZGxPEt+w@mail.gmail.com> <CALx6S35uEeWQK5-b4JTjE27T0CCwdaUESJJ585cC=OgR-D3j1w@mail.gmail.com>
In-Reply-To: <CALx6S35uEeWQK5-b4JTjE27T0CCwdaUESJJ585cC=OgR-D3j1w@mail.gmail.com>
From: Alexander Azimov <a.e.azimov@gmail.com>
Date: Fri, 20 Nov 2020 12:05:46 +0300
Message-ID: <CAEGSd=D=adP7wAHhKte7QSTXq5GVL7-1FBEBCGDs4CS-Q3O_vA@mail.gmail.com>
To: Tom Herbert <tom@herbertland.com>
Cc: tcpm <tcpm@ietf.org>, IPv6 Operations <v6ops@ietf.org>
Content-Type: multipart/alternative; boundary="0000000000002389c305b4862791"
Archived-At: <https://mailarchive.ietf.org/arch/msg/v6ops/hzoCTsHDkWSHmGg2TYQlYXSMqcM>
Subject: Re: [v6ops] Flow Label Load Balancing
X-BeenThere: v6ops@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: v6ops discussion list <v6ops.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/v6ops>, <mailto:v6ops-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/v6ops/>
List-Post: <mailto:v6ops@ietf.org>
List-Help: <mailto:v6ops-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/v6ops>, <mailto:v6ops-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 20 Nov 2020 09:06:06 -0000

Tom,

As far as I know, the auto_flowlabels default is 1, which stands for:

> 1: automatic flow labels are enabled by default, they can be disabled on a
> per socket basis using the IPV6_AUTOFLOWLABEL socket option


And you get it all-in-one: FL changes after both SYN_RTO and RTO events.
The change after SYN_RTO looks safe and can bring benefit for the end-user
experience, while the FL change after the RTO outside of a controlled
environment is causing a problem.

And yes, we were able to detect it on a working system. One of the
developers contacted NOC team that he has constant connection failures
while pushing a significant amount of data, and it began when he started
working from home. It proved out that there was RTO happening during the
lifetime of the TCP session, the session jumped to another stateful L4
balancer and, as a result, the session got a timeout. At the moment we had
to disable FL load balancing at our border routers and DCI - this solved
the issue at the moment.

If it is possible to remove from the default behavior change of flow label
after RTO event, while keeping it for SYN_RTO would be great.

чт, 19 нояб. 2020 г. в 18:36, Tom Herbert <tom@herbertland.com>om>:

> On Thu, Nov 19, 2020 at 3:49 AM Alexander Azimov <a.e.azimov@gmail.com>
> wrote:
> >
> > Dear colleagues,
> >
> > I have added in the cc both v6op and tcpm for a reason and let me
> explain why.
> >
> > It's clear that we are moving forward with load balancing that uses flow
> label (FL). And the pressure will increase with SRv6 adoption. But at the
> moment wide adoption of FL-based load-balancing may create significant
> issues for TCP Anycast services.
> >
> > RFC6437 suggests putting hash from 5-tuple into FL value. And as far as
> I know, there is no document that updates this behavior. This description
> is perfectly fine, but what is implemented in the Linux kernel is
> different: FL is carrying hash from 5-tuple with an additional seed, and
> this seed is randomly changed after each RTO/SYN_RTO event. Here are
> related patches:
> >
> >
> https://lore.kernel.org/netdev/alpine.DEB.2.02.1407012100290.20628@tomh.mtv.corp.google.com/
> >
> https://lore.kernel.org/netdev/1438124526-2129341-1-git-send-email-tom@herbertland.com/
> > https://lore.kernel.org/netdev/20160928020337.3057238-1-brakmo@fb.com/
> >
> Alexander,
>
> The initial intent of the Linux flow label code was that the hash was
> persistent for the lifetime of the connection; in fact we fixed a
> nasty bug where the hash would change when a connection moved to TW
> state causing some firewalls not to see the final ACK so they can
> remove state. The feature where the flow hash is recalculated for a
> failing connection should be optional and not the default behavior. If
> this isn't what the implementation is doing I can take it up on Linux
> netdev to fix it.
>
> Have you been able to show this behavior is actually happening on a
> running system?
>
> Tom
>
> >
> > This is a great thing by the way because in the data center environment
> with multiple equal paths it gives a way to have pseudo-multipath TCP which
> jumps between paths in case of an outage. There might be interest to write
> down an informational document for this.
> >
> > But the problem happens when TCP flows start jumping between anycast
> points. If anycast service provides connection tracking (L7/TCP proxy,
> stateful L4 balancer) the 'jump' may redirect traffic to another point, and
> with the adoption of FL load balancing by the transit providers - to
> another location, where no TCP state is available. In other words, an
> established TCP session breaks after RTO event. And it's not just a theory
> - we already faced this kind of issue in our network, and it is really hard
> to debug.
> >
> > I wonder what you think is a proper solution:
> >
> > Making FL related RTO change as knob instead of default behavior;
> > Adding negotiation behavior in TCP;
> > Something else?
> >
> > I'm looking forward to your advice. If there is a document that
> describes the above problem - please give me a reference.
> >
> > --
> > Best regards,
> > Alexander Azimov
> > _______________________________________________
> > v6ops mailing list
> > v6ops@ietf.org
> > https://www.ietf.org/mailman/listinfo/v6ops
>


-- 
Best regards,
Alexander Azimov