Re: UDP source ports for HTTP/3 and QUIC
Stefan Eissing <stefan.eissing@greenbytes.de> Fri, 16 July 2021 07:51 UTC
Return-Path: <stefan.eissing@greenbytes.de>
X-Original-To: quic@ietfa.amsl.com
Delivered-To: quic@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3F2513A2B6A for <quic@ietfa.amsl.com>; Fri, 16 Jul 2021 00:51:12 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.898
X-Spam-Level:
X-Spam-Status: No, score=-1.898 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4fz4DElXQXj4 for <quic@ietfa.amsl.com>; Fri, 16 Jul 2021 00:51:07 -0700 (PDT)
Received: from mail.greenbytes.de (mail.greenbytes.de [217.91.35.233]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 4C6823A2B6D for <quic@ietf.org>; Fri, 16 Jul 2021 00:51:06 -0700 (PDT)
Received: by mail.greenbytes.de (Postfix, from userid 119) id 4492C9882FA; Fri, 16 Jul 2021 09:51:04 +0200 (CEST)
Received: from smtpclient.apple (unknown [84.157.188.252]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mail.greenbytes.de (Postfix) with ESMTPSA id 4C75A9802F8; Fri, 16 Jul 2021 09:51:02 +0200 (CEST)
Content-Type: text/plain; charset="us-ascii"
Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.22\))
Subject: Re: UDP source ports for HTTP/3 and QUIC
From: Stefan Eissing <stefan.eissing@greenbytes.de>
In-Reply-To: <20210716071640.GB3469@1wt.eu>
Date: Fri, 16 Jul 2021 09:51:01 +0200
Cc: Poul-Henning Kamp <phk@phk.freebsd.dk>, Mikkel Fahnøe Jørgensen <mikkelfj@gmail.com>, Toerless Eckert <tte@cs.fau.de>, Mark Nottingham <mnot@mnot.net>, IETF QUIC WG <quic@ietf.org>, HTTP Working Group <ietf-http-wg@w3.org>
Content-Transfer-Encoding: 7bit
Message-Id: <332D3142-DD7A-4E9F-BEA2-22C98CF05E1F@greenbytes.de>
References: <3985895D-D420-4995-831E-332E33693B79@mnot.net> <6F79A78A-1DF8-4A48-9B7F-334B309C9C26@gmail.com> <20210715092937.GC27830@1wt.eu> <20210716014010.GL24216@faui48e.informatik.uni-erlangen.de> <20210716060644.GA3469@1wt.eu> <202107160634.16G6YWat014024@critter.freebsd.dk> <20210716071640.GB3469@1wt.eu>
To: Willy Tarreau <w@1wt.eu>
X-Mailer: Apple Mail (2.3654.100.0.2.22)
Archived-At: <https://mailarchive.ietf.org/arch/msg/quic/4VrOpn6D20JRL5882TpVUhwJbgQ>
X-BeenThere: quic@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <quic.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/quic>, <mailto:quic-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/quic/>
List-Post: <mailto:quic@ietf.org>
List-Help: <mailto:quic-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/quic>, <mailto:quic-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 16 Jul 2021 07:51:12 -0000
> Am 16.07.2021 um 09:16 schrieb Willy Tarreau <w@1wt.eu>: > > On Fri, Jul 16, 2021 at 06:34:32AM +0000, Poul-Henning Kamp wrote: >> -------- >> Willy Tarreau writes: >> >>> Stefan made a good point about the problem that might result, with >>> inbound load balancing between multiple listeners (typically what's >>> achieved by L3 switches doing L3+L4 hash between multiple servers, >>> and operating systems hashing the source+destination port to pick a >>> different listening socket). Thus a suggestion might be to possibly >>> save resources by using a small amount of sockets, with "small" left >>> to the appreciation of the implementation. >> >> We should run the question "few or many UDP ports?" past some some >> friendly 100G and 400G device driver maintainers. >> >> The NICs I have been working with for the ESO ELT project all >> included UDP ports in the hash they used to decide which CPU core >> to deliver packets/interrupts to and we had to spread across UDP >> ports to or all the traffic would hit one single core. >> >> I dont know if QUIC has registered with the NIC designers and device >> drivers writers yet, but given the opacity of QUIC packets, it is >> very hard to see what else than the UDP port they can feed into >> their hash, so I expect the answer to be "many". > > That's true but queues are not the only parameter, as using many > sockets also results in a massive cache miss ratio and many more > syscalls. Even with TCP it's quite visible that using a few hundred > connections tends to be incredibly more efficient at 100Gbps than > using tens of thousands. > > Ideally we want all queues to be used with a good balance while > limiting the amount of operations to transport these data. > > Probably that some wording evocating that possibility offered by > the protocol and a suggestion to take various factors into account > (L4 LB, NIC queues, listeners, syscalls etc) is sufficient to let > implementors ask the relevant people for advice in their context. It seems a good idea to actively pull in people knowledgable in these implementation and hear their recommendations/expectations. As far as my limited knowledge goes, having a QUIC application share a single source port for many QUIC connections does not pose any problems for servers. The concern is more about NAT (and esp. CGNAT) to coalesce many Source:address+port into a few NAT-address+ports, using QUIC connection IDs. On the extreme, once these NATs use connection IDs, they could theoretically live with a single NAT port. That would not work well with any UDP load balancing. The advice for QUIC aware NAT implementations could then be something like (IANAUBE): - distribute QUIC connections across a minimum of N ports - The N ports would ideally be random or a complete interval (they should not all give the same number module a prime) - distribute new NATed connections in some random fashion or round-robin across your port pool. - Stefan
- UDP source ports for HTTP/3 and QUIC Mark Nottingham
- Re: UDP source ports for HTTP/3 and QUIC Martin Thomson
- Re: UDP source ports for HTTP/3 and QUIC Willy Tarreau
- Re: UDP source ports for HTTP/3 and QUIC Mikkel Fahnøe Jørgensen
- Re: UDP source ports for HTTP/3 and QUIC Willy Tarreau
- Re: UDP source ports for HTTP/3 and QUIC Stefan Eissing
- Re: UDP source ports for HTTP/3 and QUIC Willy Tarreau
- Re: UDP source ports for HTTP/3 and QUIC Töma Gavrichenkov
- Re: UDP source ports for HTTP/3 and QUIC Töma Gavrichenkov
- Re: UDP source ports for HTTP/3 and QUIC Nick Banks
- Re: UDP source ports for HTTP/3 and QUIC David Schinazi
- Re: UDP source ports for HTTP/3 and QUIC Töma Gavrichenkov
- Re: UDP source ports for HTTP/3 and QUIC Erik Nygren
- Re: UDP source ports for HTTP/3 and QUIC Mark Nottingham
- Re: UDP source ports for HTTP/3 and QUIC Toerless Eckert
- Re: UDP source ports for HTTP/3 and QUIC Willy Tarreau
- Re: UDP source ports for HTTP/3 and QUIC Poul-Henning Kamp
- Re: UDP source ports for HTTP/3 and QUIC Willy Tarreau
- Re: UDP source ports for HTTP/3 and QUIC Stefan Eissing
- Re: UDP source ports for HTTP/3 and QUIC Toerless Eckert
- Re: UDP source ports for HTTP/3 and QUIC Mark Nottingham
- Re: UDP source ports for HTTP/3 and QUIC Töma Gavrichenkov