[v6ops] Flow Label Load Balancing

Alexander Azimov <a.e.azimov@gmail.com> Thu, 19 November 2020 10:48 UTC

Return-Path: <a.e.azimov@gmail.com>
X-Original-To: v6ops@ietfa.amsl.com
Delivered-To: v6ops@ietfa.amsl.com
Received: from localhost (localhost []) by ietfa.amsl.com (Postfix) with ESMTP id 590133A005D; Thu, 19 Nov 2020 02:48:20 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.098
X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([]) by localhost (ietfa.amsl.com []) (amavisd-new, port 10024) with ESMTP id YJMLFOME9dyz; Thu, 19 Nov 2020 02:48:18 -0800 (PST)
Received: from mail-oi1-x233.google.com (mail-oi1-x233.google.com [IPv6:2607:f8b0:4864:20::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A14F03A0045; Thu, 19 Nov 2020 02:48:18 -0800 (PST)
Received: by mail-oi1-x233.google.com with SMTP id m143so5801877oig.7; Thu, 19 Nov 2020 02:48:18 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=pGNWkFYae1qTiPyvVV66IAP5WPPEXhwnMjgPELWIyxU=; b=jI4F4oWJkorYmqrT8uPwp3PKwQ8ZyTl2TY3B8QPfFYFgBweiFvnB/jHmBZ5V375gbD bP7ALwPsLA7fAmcYmwWjDSTkMs3ajiWxa/ZjPRg4w5q5Q4dD9S+eFzwyWEMDxkJWcpS+ cc16b2E2D3px+zuZMBJlpsAzpZ+IcTaOmqgFdQBE7eo7r3m3y/4A7L1BCBJ1bNYrkf2N bprDjGsrMezBOodTq0J1Buuya+2/V0SKEYt/lgik4Rz/L1iSyQo7YQdfvCPq2qu5dA3R lymely3XuqUsa9fdIlz3WE7bhu4FUNoSX0hYEJgTQfd5lOUoTsWmSVysiexET78543ns XrnQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=pGNWkFYae1qTiPyvVV66IAP5WPPEXhwnMjgPELWIyxU=; b=Wa7FMgQQ306A0B9NEpfILPpBYGHmEjosfNjQZ8w5ueh7vpJvG3+myou6671k4Kn5Mj Dm/hMF5vjsP0BFq5k6zhA6Rq1NKNFYm5TWMGrtuNDwnmqLQUqqhBnx9o71oZ0kY/QsTt 0KVYY4b+RKuO5KepX2vTDRVEj2vIcOJxfIjPI4IMyqfOTKCzBROi9Uw/ay5A4wS+u3Ae VgLrhgmyeloggsQxRZdlGgMp3tvIziYO3q+FgUELuef2jldLQPwpxxzfcpHJthA8tVN5 +zIMEzGHek9exCp6fdEwaIw/mwTI2FbEOdW1TIsdiBUs7EguXxMvAYgWqvStEegxuYmK KPOQ==
X-Gm-Message-State: AOAM532edMDJdnaJ+ZNN1N/YoTNf31PdDUpTI7zQfqctb7aU7UftykN1 n2vNiTZfdc0C9Eu6BhHMeOIMP1LDSiG/B6lt1YP5D86nYdw=
X-Google-Smtp-Source: ABdhPJyWdlgBLChqsWqrunfmFa6JmjU6LpUGzV6Nlm9W8mC2QJppj6qr1QHyAwufGHjbdf5p/Ma92g2SRY5OinzKjzQ=
X-Received: by 2002:aca:c4c3:: with SMTP id u186mr2476326oif.4.1605782897451; Thu, 19 Nov 2020 02:48:17 -0800 (PST)
MIME-Version: 1.0
From: Alexander Azimov <a.e.azimov@gmail.com>
Date: Thu, 19 Nov 2020 13:48:06 +0300
Message-ID: <CAEGSd=DY8t8Skor+b6LSopzecoUUzUZhti9s0kdooLZGxPEt+w@mail.gmail.com>
To: tcpm <tcpm@ietf.org>, v6ops@ietf.org
Content-Type: multipart/alternative; boundary="00000000000030651505b473773b"
Archived-At: <https://mailarchive.ietf.org/arch/msg/v6ops/V9ZljDjcqKVraHouhwsOWNxHcmU>
Subject: [v6ops] Flow Label Load Balancing
X-BeenThere: v6ops@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: v6ops discussion list <v6ops.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/v6ops>, <mailto:v6ops-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/v6ops/>
List-Post: <mailto:v6ops@ietf.org>
List-Help: <mailto:v6ops-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/v6ops>, <mailto:v6ops-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 19 Nov 2020 10:48:20 -0000

Dear colleagues,

I have added in the cc both v6op and tcpm for a reason and let me explain

It's clear that we are moving forward with load balancing that uses flow
label (FL). And the pressure will increase with SRv6 adoption. But at the
moment wide adoption of FL-based load-balancing may create significant
issues for TCP Anycast services.

RFC6437 suggests putting hash from 5-tuple into FL value. And as far as I
know, there is no document that updates this behavior. This description is
perfectly fine, but what is implemented in the Linux kernel is different:
FL is carrying hash from 5-tuple with an additional seed, and this seed is
randomly changed after each RTO/SYN_RTO event. Here are related patches:

   - https://lore.kernel.org/netdev/20160928020337.3057238-1-brakmo@fb.com/

This is a great thing by the way because in the data center environment with
multiple equal paths it gives a way to have pseudo-multipath TCP which
jumps between paths in case of an outage. There might be interest to write
down an informational document for this.

But the problem happens when TCP flows start jumping between anycast
points. If anycast service provides connection tracking (L7/TCP proxy,
stateful L4 balancer) the 'jump' may redirect traffic to another point, and
with the adoption of FL load balancing by the transit providers - to
another location, where no TCP state is available. In other words, an
established TCP session breaks after RTO event. And it's not just a theory
- we already faced this kind of issue in our network, and it is really hard
to debug.

I wonder what you think is a proper solution:

   - Making FL related RTO change as knob instead of default behavior;
   - Adding negotiation behavior in TCP;
   - Something else?

I'm looking forward to your advice. If there is a document that describes
the above problem - please give me a reference.

Best regards,
Alexander Azimov