Re: QUIC Invariants and QUIC-LB

Martin Thomson <> Thu, 21 May 2020 23:40 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id C38353A0CEF for <>; Thu, 21 May 2020 16:40:49 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.098
X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key) header.b=L4gXST9l; dkim=pass (2048-bit key) header.b=hyDcU3Xf
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 6MNO6aiiEQAS for <>; Thu, 21 May 2020 16:40:47 -0700 (PDT)
Received: from ( []) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 63EBB3A0903 for <>; Thu, 21 May 2020 16:40:47 -0700 (PDT)
Received: from compute2.internal (compute2.nyi.internal []) by mailout.nyi.internal (Postfix) with ESMTP id 75DBF5C0166 for <>; Thu, 21 May 2020 19:40:46 -0400 (EDT)
Received: from imap2 ([]) by compute2.internal (MEProxy); Thu, 21 May 2020 19:40:46 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; h=mime-version:message-id:in-reply-to:references:date:from:to :subject:content-type; s=fm2; bh=54Gkqb2aqG67QbdJ9fKVAZd5SfEWODg hhkDTPQFrOsk=; b=L4gXST9lYf41wxwfCybmuUt2vAorFmJ48LQMTdsz44vkREO TswBTojiNPByduH0xrHsS5ljhiBAswHZGTev7yj9Xephl3lC+AFN3QyAg5AAAlSK ptTDIvFFb7b0vEbhJEsmDCBYvBwzyytf0wVxo5FbYwn12vnCm8A1cSVrP1vtj15p GvIuiwwFW6WUaAZSXuWz2dsP880FaGCPUnsokU4jeKSjsd6/9dKbsAOnXURbOgfo D82t9MfcLqBNFrwK1YRE2aqLKzq2okDNFSlTJ4ngB0JRaGtFsolg6Ag8DW22T0I3 0EsoFO4OA34Evb+ya7mV88JDGZlZfjj7mjiSfdQ==
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=54Gkqb 2aqG67QbdJ9fKVAZd5SfEWODghhkDTPQFrOsk=; b=hyDcU3Xf5vXEoP9xitzgss 9GrQGlCCJjJC/Iz2I3inV17Jifw4NaCFbuXUMkYHFQnQKJQ4Sc8CzaQ5MrZDQSNV lERBRrSVGj4V1JNy71T6L76r/pxxOfLSWqUe0bO+99FyTuSb2Zz1wHsHs6f6laWQ tA2w3FFeMTNHfMSaHm8IgYJdjGBa3ZsYnY1BY9Um+TYloOPwaeFlrqNp0Y0jdc4q qkTOwCW+XABYhjXHPCbalco+f1BMNWItiwtwc+ljPZ2fSssBc8zDGKeFN22tmd/s hUHoziNZCe3nDXCNYdMXDYvVTaxOAatynET77mxyzZCRg3zmj1+RYzFHOUYMf5MQ ==
X-ME-Sender: <xms:fhHHXr-45b_6rYzbyzmu4TQ7TihbzxuOvsdNR2k-CczGxVfxyicQHw>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduhedrudduvddgvdefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefofgggkfgjfhffhffvufgtsehttd ertderredtnecuhfhrohhmpedfofgrrhhtihhnucfvhhhomhhsohhnfdcuoehmtheslhho figvnhhtrhhophihrdhnvghtqeenucggtffrrghtthgvrhhnpeeitefgfedukeevffethe ekudetjeeitdffveeutefgveejueffgfehuefhgfeigfenucffohhmrghinhepqhhuihgt fihgrdhorhhgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrh homhepmhhtsehlohifvghnthhrohhphidrnhgvth
X-ME-Proxy: <xmx:fhHHXntf1VI7y7boyeI-NUz9BfwyzKl7ldtKlLLWigm_2svwE2hZ8Q> <xmx:fhHHXpCBZp8meY-ATi2vWj9eemYt4al7vf7pQvU-nQ7sbJhrwx9kZQ> <xmx:fhHHXnf2r4wOCOH2OrRFLLHp-FF8XVSdmucsZrs-0SJroAK6BushJg> <xmx:fhHHXqsxxlwfsjEtajve-9B4PGRswq-t_A2F_AEez_7xmIow-RVfwA>
Received: by mailuser.nyi.internal (Postfix, from userid 501) id 272E7E0124; Thu, 21 May 2020 19:40:46 -0400 (EDT)
X-Mailer: Webmail Interface
User-Agent: Cyrus-JMAP/3.3.0-dev0-487-g38013f6-fm-20200519.001-g38013f6c
Mime-Version: 1.0
Message-Id: <>
In-Reply-To: <>
References: <> <>
Date: Fri, 22 May 2020 09:40:21 +1000
From: "Martin Thomson" <>
Subject: Re: QUIC Invariants and QUIC-LB
Content-Type: text/plain
Archived-At: <>
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Thu, 21 May 2020 23:40:50 -0000

That's a good point.

The set of assumptions here are fairly small, which is good.  Essentially, these turn into requirements for a new version of QUIC that is intended to replace v1, if that version also needs to work with a load balancer (which seems quite likely).  As the set is so small, and as I don't think that any replacement would be likely to make changes at so fundamental a level, it seems like those requirements wouldn't end up being too onerous.

Basically, this boils down to whether you think the ability to load balance is invariant, and while it might be inconceivable for something like HTTP, it is not generally so.

On Fri, May 22, 2020, at 00:31, Ian Swett wrote:
> Thanks for pointing these out. To me, these constraints seem like 
> natural consequences of using QUIC with a load balancer. But there are 
> use cases for QUIC where a load balancer isn't required(ie: P2P and 
> many datacenter use cases), so I'd prefer not to add more constraints 
> to the invariants.
> On Wed, May 20, 2020 at 7:28 PM Martin Duke <> wrote:
> > TL;DR We are trying to make QUIC-LB as independent of QUIC version as possible, but there are a few gaps in the invariants that make that harder. If any of these are oversights, it would be great to add it to the invariants.
> > 
> > *******
> > A few draft versions ago we did a pass on QUIC-LB to make it truly QUIC-version invariant, except for the Retry Services portion.
> > 
> > After a little more thought, I added this section to the draft:
> >
> > which describes a bunch of QUIC behaviors that are not explicitly described in the invariants draft, but that if changed might cause QUIC-LB not to be fully robust to new versions. An obvious one is a version that described a maximum CID length too small for some of the encodings.
> > 
> > A more subtle one is the handling of "non-compliant CIDs" (Sec 4). These are CIDs that are (1) too short for the configured algorithm, (2) fail authentication, or (3) don't map to a valid server ID. In v1, these will only occur in the client's first flight of Initial and 0RTT packets. The rules we have for version-independent LBs to process packets with non-compliant CIDs are as follows:
> > * if a short header, drop it.
> > * if a long header, use any hash algorithm that will consistently get you the same result over short time scales (i.e. a couple of RTTs, by which time a server-generated CID will be in use)
> > Compliant CIDs are always processed using the decoding in the spec.
> > 
> > There are some assumptions baked into this advice. 
> > (1) Short headers never contain client-generated CIDs. [QUIC-LB servers will always provide new CIDs when given the opportunity, but a new version could have short 0RTT headers or something] 
> > (2) Use of client-generated DCIDs is limited to a few RTTs at most (any more and you run the risk of the number of servers changing in this interval, which will generally cause the hash to get a different result)
> > (3) There's a bunch of stuff that remains constant in the packet headers over this short interval. I probably need to tighten up the language on what is a suitable input for the LB's hash, but in v1 you're currently safe using some combination of Client IP, Client Port, SCID, and DCID. If there of these that we're willing to lock down in the invariants, I could rewrite the guidance accordingly.
> > 
> > For example, if the invariants could include a statement like "A client will use no more than one DCID that it didn't receive from the server", that would simplify the problem space.
> > 
> > ****
> > 
> > I'm fine with having this text in the QUIC-LB draft, if need be, but if there are things that should be in the invariants but we didn't think about them, that would be even better.
> > 
> > Martin