Re: Is "Version Greasing" a new benfit or a new obstacle?

Roberto Peon <fenix@fb.com> Fri, 12 April 2019 16:42 UTC

Return-Path: <prvs=9005e680f9=fenix@fb.com>
X-Original-To: quic@ietfa.amsl.com
Delivered-To: quic@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3B5311202F1 for <quic@ietfa.amsl.com>; Fri, 12 Apr 2019 09:42:56 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.336
X-Spam-Level:
X-Spam-Status: No, score=-1.336 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, KHOP_DYNAMIC=1.363, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=fb.com header.b=Tp/PToxT; dkim=pass (1024-bit key) header.d=fb.onmicrosoft.com header.b=kdSE4l9B
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6R0N31WG8dxl for <quic@ietfa.amsl.com>; Fri, 12 Apr 2019 09:42:54 -0700 (PDT)
Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 0E274120072 for <quic@ietf.org>; Fri, 12 Apr 2019 09:42:54 -0700 (PDT)
Received: from pps.filterd (m0044012.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3CGeMos009803; Fri, 12 Apr 2019 09:42:37 -0700
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : mime-version; s=facebook; bh=Ojl4yBPH57qBWJ4C4gM6X/J/EDt0d4hWa3KN/JNqVug=; b=Tp/PToxTuvs0zQd5eU2dL066WSX6ROC3TWnT/QNvzwtYuhiIuf+nDW5Kn1ec7iURDtKC be5+o7NYSoteLtXzHie8009tppW3cp4pO9uiNohvauWvqL5ChEkN6YYgylxfQkFBQqpE TCraeuMTkaPcT6bXpjGBxkJsV7ARjyVU1Rw=
Received: from maileast.thefacebook.com ([199.201.65.23]) by mx0a-00082601.pphosted.com with ESMTP id 2rtv7q8g6g-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Fri, 12 Apr 2019 09:42:37 -0700
Received: from frc-mbx07.TheFacebook.com (2620:10d:c0a1:f82::31) by frc-hub02.TheFacebook.com (2620:10d:c021:18::172) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.1.1713.5; Fri, 12 Apr 2019 09:42:30 -0700
Received: from frc-hub02.TheFacebook.com (2620:10d:c021:18::172) by frc-mbx07.TheFacebook.com (2620:10d:c0a1:f82::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.1.1713.5; Fri, 12 Apr 2019 09:41:51 -0700
Received: from NAM05-DM3-obe.outbound.protection.outlook.com (192.168.183.28) by o365-in.thefacebook.com (192.168.177.72) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.1.1713.5 via Frontend Transport; Fri, 12 Apr 2019 09:41:51 -0700
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.onmicrosoft.com; s=selector1-fb-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ojl4yBPH57qBWJ4C4gM6X/J/EDt0d4hWa3KN/JNqVug=; b=kdSE4l9Bcu0IVnFMgVFYZcVrCL0xmK43DERXTjFM+bUYb/9H4DHIQ9ZJzJClfavUo+/KKpWFovDLXHWi+9p3CFWMUPAKO5MXE/suTAtU3/HS8dMnOmRmisvuXlubjuOVWrgOj8ZW7QiVg8DlXoDiSRlm0GhlZqjdhfM80NwpwbI=
Received: from BYAPR15MB2312.namprd15.prod.outlook.com (52.135.197.146) by BYAPR15MB2408.namprd15.prod.outlook.com (52.135.198.148) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1771.21; Fri, 12 Apr 2019 16:41:48 +0000
Received: from BYAPR15MB2312.namprd15.prod.outlook.com ([fe80::91d3:bc32:a475:76d7]) by BYAPR15MB2312.namprd15.prod.outlook.com ([fe80::91d3:bc32:a475:76d7%2]) with mapi id 15.20.1771.021; Fri, 12 Apr 2019 16:41:48 +0000
From: Roberto Peon <fenix@fb.com>
To: Praveen Balasubramanian <pravb@microsoft.com>, Christian Huitema <huitema@huitema.net>, Ian Swett <ianswett@google.com>
CC: "Gorry (erg)" <gorry@erg.abdn.ac.uk>, Mirja Kuehlewind <mirja.kuehlewind@ericsson.com>, Mikkel Fahnøe Jørgensen <mikkelfj@gmail.com>, "quic@ietf.org" <quic@ietf.org>, "Border, John" <john.border@hughes.com>
Subject: Re: Is "Version Greasing" a new benfit or a new obstacle?
Thread-Topic: Is "Version Greasing" a new benfit or a new obstacle?
Thread-Index: AQHU73o3xTufubPn9E2URSuto2uLa6Y1JwYAgAB3W4CAAAdsgIAADYwAgAAcxQCAAMrxAIAABNWAgAIQ3wCAAACXAIAAAIaAgAAEgoD//4tNAIAAeNGA//+Ob4A=
Date: Fri, 12 Apr 2019 16:41:48 +0000
Message-ID: <BDEE786D-9CCE-457F-946A-8CC195E3CDA8@fb.com>
References: <5CADADDD.7010005@erg.abdn.ac.uk> <EBF1BF30-62A5-4659-8AEC-0D5B3F2D65C6@fb.com> <BL0PR11MB3394294313F8F54A3D0CF4A3902E0@BL0PR11MB3394.namprd11.prod.outlook.com> <CAN1APdcm0hnT_Mu7D7x5QM6pApOQw1RdWCBkgY16bd5YWNtFkA@mail.gmail.com> <9084B09D-5E13-49FA-BA93-0D7276CDE420@erg.abdn.ac.uk> <CAN1APdeSF0-_N=mb1xkoe_qLwoVqP+X9_Wawi=Zu__6wdHtbOQ@mail.gmail.com> <699E2135-A3CE-4D33-91F6-D3C96E66674F@ericsson.com> <CAN1APde2SO6fkNzyznbv2-xNuXkkuC=bN3p8xRgwmRAmsZxrgA@mail.gmail.com> <MW2PR2101MB104902B6BC7C67D8688EA852B6280@MW2PR2101MB1049.namprd21.prod.outlook.com> <MW2PR2101MB104996C29680DF7B79230C8CB6280@MW2PR2101MB1049.namprd21.prod.outlook.com> <CAKcm_gMwcwZ0VzK4vDt-sCQctHVrwOm9ekPPi3O3Y2UnRrZaBw@mail.gmail.com> <02f27c4e-2d1e-29ab-6891-236442436a3c@huitema.net> <9C6D92FF-7635-48BD-A65B-FAB4B544476D@fb.com> <MW2PR2101MB10498605DC0F1FC3905865F4B6280@MW2PR2101MB1049.namprd21.prod.outlook.com>
In-Reply-To: <MW2PR2101MB10498605DC0F1FC3905865F4B6280@MW2PR2101MB1049.namprd21.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-MacOutlook/10.17.1.190326
x-originating-ip: [2620:10d:c090:200::1ab6]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 499389cf-f903-4edc-219f-08d6bf65c1dc
x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600139)(711020)(4605104)(2017052603328)(7193020); SRVR:BYAPR15MB2408;
x-ms-traffictypediagnostic: BYAPR15MB2408:
x-ms-exchange-purlcount: 2
x-microsoft-antispam-prvs: <BYAPR15MB240810B9C773983B23CC80FECD280@BYAPR15MB2408.namprd15.prod.outlook.com>
x-forefront-prvs: 0005B05917
x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(346002)(39860400002)(366004)(396003)(136003)(376002)(199004)(189003)(82746002)(58126008)(8676002)(186003)(2906002)(6116002)(446003)(2616005)(105586002)(46003)(256004)(81156014)(76176011)(14444005)(8936002)(486006)(476003)(54896002)(102836004)(316002)(106356001)(99286004)(66574012)(7736002)(6512007)(229853002)(97736004)(81166006)(6306002)(33656002)(93886005)(86362001)(68736007)(6506007)(54906003)(478600001)(45080400002)(11346002)(236005)(53936002)(4326008)(53546011)(110136005)(6486002)(6246003)(6436002)(25786009)(14454004)(36756003)(1511001)(71200400001)(71190400001)(5660300002)(83716004); DIR:OUT; SFP:1102; SCL:1; SRVR:BYAPR15MB2408; H:BYAPR15MB2312.namprd15.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1;
received-spf: None (protection.outlook.com: fb.com does not designate permitted sender hosts)
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam-message-info: k9cS2elridwdQnb2MKA0avpsYVQYALG7Yj6xiRu0474UDl7cWxtd34drzbTZmKFMxSjRIurAL7OQncYn4RDGvqaaXFHcpCiC82cYJBHPzu71VzolqaXy5KpQQxydKAEAkQc3iazFduMev0fjPsUDtfp3P4/hRwvsXDrR44HU1gbpGtYVTC4kSx9rZM+bSJXvxsjvNAhz7KiTeGcNo74m/cauWnlH/do8zPemuaPyFOd7ht12zJBs/MK6lLb82Batfi0vXCpWfQoBQpRfynodCZMNbfddlwsA/UWZTr+zNiIwc2OTbrGaakwKgExgd+hgaV0kSOc8M1CBBITDBpK9sUOXd/arKFPCewP81DpOcvVkY3uEXt1dyu+pCsM0Cf+3JrXRGCtmHbWz8w877Br3/ZjysDekITqgRwSIakinDA0=
Content-Type: multipart/alternative; boundary="_000_BDEE786D9CCE457F946A8CC195E3CDA8fbcom_"
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 499389cf-f903-4edc-219f-08d6bf65c1dc
X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Apr 2019 16:41:48.3554 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR15MB2408
X-OriginatorOrg: fb.com
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-04-12_09:, , signatures=0
X-Proofpoint-Spam-Reason: safe
X-FB-Internal: Safe
Archived-At: <https://mailarchive.ietf.org/arch/msg/quic/UybQaAi5G_gjG1FpgNagfotabPs>
X-BeenThere: quic@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <quic.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/quic>, <mailto:quic-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/quic/>
List-Post: <mailto:quic@ietf.org>
List-Help: <mailto:quic-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/quic>, <mailto:quic-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 12 Apr 2019 16:42:56 -0000

I’m not sure that I agree there—there is no need to have it standardized so long as the LB is involved in picking it. Each LB system can choose whatever it wants. Thus, every LB *deployment* could use a different scheme with different grease. Every LB deployment could change the scheme it uses pseudo-randomly or according to the lunar calendar, etc.

Because there is a dynamic (and controlling) component (namely the part of the LB system which vends and then later routes CIDs), there is no need for anything else.

-=R

From: Praveen Balasubramanian <pravb@microsoft.com>
Date: Friday, April 12, 2019 at 9:28 AM
To: Roberto Peon <fenix@fb.com>, Christian Huitema <huitema@huitema.net>, Ian Swett <ianswett@google.com>
Cc: "Gorry (erg)" <gorry@erg.abdn.ac.uk>, Mirja Kuehlewind <mirja.kuehlewind@ericsson.com>, Mikkel Fahnøe Jørgensen <mikkelfj@gmail.com>, "quic@ietf.org" <quic@ietf.org>, "Border, John" <john.border@hughes.com>
Subject: RE: Is "Version Greasing" a new benfit or a new obstacle?

Re. Encoding in CID. Then it needs to be standardized and if it is standardized what’s the point in greasing? Let’s not ignore the smaller services who use the cloud i.e. people deploying say NGINX web servers in AWS, Azure or GCP. Not every QUIC deployment is a large first party service which controls all aspects how LB and DDoS are done. In public clouds LB and DDoS protection are network functions (services) and are middleboxes under a completely different administrative domain.

Christian, this is about source address validation by the middlebox. This is how it works for TCP (think SYN cookies) and we need to have parity for QUIC. When under attack, short header packets will be dropped for addresses that have not been validated, but this is the same for non-SYN packets for TCP. I don’t understand how invariants can be used here. If we don’t have parity for DoS protection in network then QUIC traffic will be rate limited which is going to be really bad for performance. FWIW there is already DoS traffic that looks like QUIC.

From: Roberto Peon <fenix@fb.com>
Sent: Friday, April 12, 2019 9:16 AM
To: Christian Huitema <huitema@huitema.net>; Ian Swett <ianswett@google.com>; Praveen Balasubramanian <pravb@microsoft.com>
Cc: Gorry (erg) <gorry@erg.abdn.ac.uk>; Mirja Kuehlewind <mirja.kuehlewind@ericsson.com>; Mikkel Fahnøe Jørgensen <mikkelfj@gmail.com>; quic@ietf.org; Border, John <john.border@hughes.com>
Subject: Re: Is "Version Greasing" a new benfit or a new obstacle?

For any large deployment, I suspect they’ll need LB.

For most kinds of LB-ing, I’d expect the LB system to either directly or indirectly be involved in choosing/vending connection IDs.

If it is involved in vending/choosing connection IDs, it can encode version information into the CID.

-=R

From: Christian Huitema <huitema@huitema.net<mailto:huitema@huitema.net>>
Date: Friday, April 12, 2019 at 9:13 AM
To: Ian Swett <ianswett=40google.com@dmarc.ietf.org<mailto:ianswett=40google.com@dmarc.ietf.org>>, Praveen Balasubramanian <pravb=40microsoft.com@dmarc.ietf.org<mailto:pravb=40microsoft.com@dmarc.ietf.org>>
Cc: "Gorry (erg)" <gorry@erg.abdn.ac.uk<mailto:gorry@erg.abdn.ac.uk>>, Roberto Peon <fenix@fb.com<mailto:fenix@fb.com>>, Mirja Kuehlewind <mirja.kuehlewind@ericsson.com<mailto:mirja.kuehlewind@ericsson.com>>, Mikkel Fahnøe Jørgensen <mikkelfj@gmail.com<mailto:mikkelfj@gmail.com>>, "quic@ietf.org<mailto:quic@ietf.org>" <quic@ietf.org<mailto:quic@ietf.org>>, "Border, John" <john.border@hughes.com<mailto:john.border@hughes.com>>
Subject: Re: Is "Version Greasing" a new benfit or a new obstacle?



On 4/12/2019 8:57 AM, Ian Swett wrote:
From: Praveen Balasubramanian
...
Version greasing will create problems for DDoS protection. The DDoS protection middlebox will have to treat all traffic with unknown version numbers as plain old QUIC and rate limit it. Any solution that requires per connection state to track version numbers doesn’t scale. Especially in the cloud scenario customers bring their own web servers so this makes any opt-out solution not good enough. So I do think this is an obstacle for many deployments.

Don't we have an example of middle-box ossification just there?

Praveen's example points to the perils of the "shortest path" approach to middle-box design. Taking the shortest path means finding something that works now, and once you have found it deploy it without worry about consequences -- which very often include ossification. If the DDOS protection box looks for "QUIC versions that it knows" and just rate limits the other types of packets, then the box will by design prevent deployment of versions that it does not know.

I assume that big organizations like Azure will update their protections once a new version is out, but the experience shows that many other organizations don't. And even when they do, the focus on published versions will block experimenting with new versions, not to mention interfering with other features like connection migration. I take Praveen's example as an argument why we should in fact standardize version greasing, precisely to avoid this "shortest path middle-box design".

In the case of DDOS protection, I will point out that DDOS protection also needs to work with short header packets, which do not carry any version number. In order to avoid DDOS by short header packets, the boxes have to be able to distinguish between packets that belong to authorized connections and packets that don't. And if you do that, then you don't really need to look at the version numbers, you can simply rely on the invariants.

-- Christian Huitema