Re: UDP source ports for HTTP/3 and QUIC

Stefan Eissing <stefan.eissing@greenbytes.de> Fri, 16 July 2021 07:51 UTC

Return-Path: <stefan.eissing@greenbytes.de>
X-Original-To: quic@ietfa.amsl.com
Delivered-To: quic@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3F2513A2B6A for <quic@ietfa.amsl.com>; Fri, 16 Jul 2021 00:51:12 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.898
X-Spam-Level:
X-Spam-Status: No, score=-1.898 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4fz4DElXQXj4 for <quic@ietfa.amsl.com>; Fri, 16 Jul 2021 00:51:07 -0700 (PDT)
Received: from mail.greenbytes.de (mail.greenbytes.de [217.91.35.233]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 4C6823A2B6D for <quic@ietf.org>; Fri, 16 Jul 2021 00:51:06 -0700 (PDT)
Received: by mail.greenbytes.de (Postfix, from userid 119) id 4492C9882FA; Fri, 16 Jul 2021 09:51:04 +0200 (CEST)
Received: from smtpclient.apple (unknown [84.157.188.252]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mail.greenbytes.de (Postfix) with ESMTPSA id 4C75A9802F8; Fri, 16 Jul 2021 09:51:02 +0200 (CEST)
Content-Type: text/plain; charset="us-ascii"
Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.22\))
Subject: Re: UDP source ports for HTTP/3 and QUIC
From: Stefan Eissing <stefan.eissing@greenbytes.de>
In-Reply-To: <20210716071640.GB3469@1wt.eu>
Date: Fri, 16 Jul 2021 09:51:01 +0200
Cc: Poul-Henning Kamp <phk@phk.freebsd.dk>, Mikkel Fahnøe Jørgensen <mikkelfj@gmail.com>, Toerless Eckert <tte@cs.fau.de>, Mark Nottingham <mnot@mnot.net>, IETF QUIC WG <quic@ietf.org>, HTTP Working Group <ietf-http-wg@w3.org>
Content-Transfer-Encoding: 7bit
Message-Id: <332D3142-DD7A-4E9F-BEA2-22C98CF05E1F@greenbytes.de>
References: <3985895D-D420-4995-831E-332E33693B79@mnot.net> <6F79A78A-1DF8-4A48-9B7F-334B309C9C26@gmail.com> <20210715092937.GC27830@1wt.eu> <20210716014010.GL24216@faui48e.informatik.uni-erlangen.de> <20210716060644.GA3469@1wt.eu> <202107160634.16G6YWat014024@critter.freebsd.dk> <20210716071640.GB3469@1wt.eu>
To: Willy Tarreau <w@1wt.eu>
X-Mailer: Apple Mail (2.3654.100.0.2.22)
Archived-At: <https://mailarchive.ietf.org/arch/msg/quic/4VrOpn6D20JRL5882TpVUhwJbgQ>
X-BeenThere: quic@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <quic.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/quic>, <mailto:quic-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/quic/>
List-Post: <mailto:quic@ietf.org>
List-Help: <mailto:quic-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/quic>, <mailto:quic-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 16 Jul 2021 07:51:12 -0000


> Am 16.07.2021 um 09:16 schrieb Willy Tarreau <w@1wt.eu>:
> 
> On Fri, Jul 16, 2021 at 06:34:32AM +0000, Poul-Henning Kamp wrote:
>> --------
>> Willy Tarreau writes:
>> 
>>> Stefan made a good point about the problem that might result, with
>>> inbound load balancing between multiple listeners (typically what's
>>> achieved by L3 switches doing L3+L4 hash between multiple servers,
>>> and operating systems hashing the source+destination port to pick a
>>> different listening socket). Thus a suggestion might be to possibly
>>> save resources by using a small amount of sockets, with "small" left
>>> to the appreciation of the implementation.
>> 
>> We should run the question "few or many UDP ports?" past some some
>> friendly 100G and 400G device driver maintainers.
>> 
>> The NICs I have been working with for the ESO ELT project all
>> included UDP ports in the hash they used to decide which CPU core
>> to deliver packets/interrupts to and we had to spread across UDP
>> ports to or all the traffic would hit one single core.
>> 
>> I dont know if QUIC has registered with the NIC designers and device
>> drivers writers yet, but given the opacity of QUIC packets, it is
>> very hard to see what else than the UDP port they can feed into
>> their hash, so I expect the answer to be "many".
> 
> That's true but queues are not the only parameter, as using many
> sockets also results in a massive cache miss ratio and many more
> syscalls. Even with TCP it's quite visible that using a few hundred
> connections tends to be incredibly more efficient at 100Gbps than
> using tens of thousands.
> 
> Ideally we want all queues to be used with a good balance while
> limiting the amount of operations to transport these data.
> 
> Probably that some wording evocating that possibility offered by
> the protocol and a suggestion to take various factors into account
> (L4 LB, NIC queues, listeners, syscalls etc) is sufficient to let
> implementors ask the relevant people for advice in their context.

It seems a good idea to actively pull in people knowledgable in 
these implementation and hear their recommendations/expectations.

As far as my limited knowledge goes, having a QUIC application
share a single source port for many QUIC connections does not
pose any problems for servers.

The concern is more about NAT (and esp. CGNAT) to coalesce
many Source:address+port into a few NAT-address+ports,
using QUIC connection IDs. On the extreme, once these NATs
use connection IDs, they could theoretically live with a single
NAT port. That would not work well with any UDP load balancing.

The advice for QUIC aware NAT implementations could then be
something like (IANAUBE):
- distribute QUIC connections across a minimum of N ports
- The N ports would ideally be random or a complete interval
  (they should not all give the same number module a prime)
- distribute new NATed connections in some random fashion
  or round-robin across your port pool.

- Stefan