Re: UDP send costs in Linux

Subodh Iyengar <subodh@fb.com> Wed, 04 April 2018 02:50 UTC

Return-Path: <prvs=763250ffbd=subodh@fb.com>
X-Original-To: quic@ietfa.amsl.com
Delivered-To: quic@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id CBE2212946D for <quic@ietfa.amsl.com>; Tue, 3 Apr 2018 19:50:37 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.711
X-Spam-Level:
X-Spam-Status: No, score=-0.711 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, HTTPS_HTTP_MISMATCH=1.989, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=fb.com header.b=qlRxBTS9; dkim=pass (1024-bit key) header.d=fb.onmicrosoft.com header.b=Poei2XLf
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wbmYKGN10KxQ for <quic@ietfa.amsl.com>; Tue, 3 Apr 2018 19:50:36 -0700 (PDT)
Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 31D111289B0 for <quic@ietf.org>; Tue, 3 Apr 2018 19:50:36 -0700 (PDT)
Received: from pps.filterd (m0109332.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w342nD6l022353; Tue, 3 Apr 2018 19:50:34 -0700
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=from : to : subject : date : message-id : references : in-reply-to : content-type : mime-version; s=facebook; bh=O4edPnMptyFisse2fTshiAalvyCQ/enH79CL0qpxatM=; b=qlRxBTS9rNhqAwVmH7pYwQC1MSZUcyE/tlI7mPbd25S3rixUauvpNyA42VMm4YpJgHEV rQjJU2ZPZDpaXNd5dLQ0guIPJzriimzSnqTi3l6NRu9qgSFzB9vsE8atTq5LxlMLt2A+ Mm9d6CpRigD4G6ig1lWeRZ0pyg2djKOVAsg=
Received: from maileast.thefacebook.com ([199.201.65.23]) by mx0a-00082601.pphosted.com with ESMTP id 2h4m8h86yh-1 (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=NOT); Tue, 03 Apr 2018 19:50:34 -0700
Received: from NAM02-BL2-obe.outbound.protection.outlook.com (192.168.183.28) by o365-in.thefacebook.com (192.168.177.34) with Microsoft SMTP Server (TLS) id 14.3.361.1; Tue, 3 Apr 2018 22:50:33 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.onmicrosoft.com; s=selector1-fb-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=O4edPnMptyFisse2fTshiAalvyCQ/enH79CL0qpxatM=; b=Poei2XLf6wOST4+yGrPvqubXPJ+0c66nXq8aUpxTyXu7iUo3HWTSlh3aodZSEO+EGq+50NbBQ8L2a2AAFhWd1TFt9LnyOL8bmzy/QOUTalmhW4f8DkeyTCgbnUIxbPWWqDPdJAIq62fhtY8N/JIFnm/eZo6j5w9RwjRDQi1fEwE=
Received: from MWHPR15MB1821.namprd15.prod.outlook.com (10.174.255.137) by MWHPR15MB1517.namprd15.prod.outlook.com (10.173.235.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.631.10; Wed, 4 Apr 2018 02:50:31 +0000
Received: from MWHPR15MB1821.namprd15.prod.outlook.com ([fe80::4569:2fbb:3aab:6126]) by MWHPR15MB1821.namprd15.prod.outlook.com ([fe80::4569:2fbb:3aab:6126%17]) with mapi id 15.20.0631.013; Wed, 4 Apr 2018 02:50:31 +0000
From: Subodh Iyengar <subodh@fb.com>
To: Ian Swett <ianswett=40google.com@dmarc.ietf.org>, IETF QUIC WG <quic@ietf.org>
Subject: Re: UDP send costs in Linux
Thread-Topic: UDP send costs in Linux
Thread-Index: AQHTy6rd/yijFMm6G0acwigUqceJdaPv5xg8
Date: Wed, 04 Apr 2018 02:50:31 +0000
Message-ID: <MWHPR15MB18215781CA00A71CF1AD0137B6A40@MWHPR15MB1821.namprd15.prod.outlook.com>
References: <CAKcm_gP4zz1bW5T-_N2Oxy6o5Sw2mEs3DFU9_HrmfkuaJyLz0A@mail.gmail.com>
In-Reply-To: <CAKcm_gP4zz1bW5T-_N2Oxy6o5Sw2mEs3DFU9_HrmfkuaJyLz0A@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [2620:10d:c090:180::1:d273]
x-ms-publictraffictype: Email
x-microsoft-exchange-diagnostics: 1; MWHPR15MB1517; 7:9+9SM8RTiiFCZSxcGpuqo8CDW7TNXxHBK0eGzbCtQfSUIv9osNBX2PJyPKtzvx/zSqSJW4FotsttERIFSWQVd92fjTbQyNHrjDHNitZHxKqEXvLUuzqaM8tmwjIe4CMbF6ryH48nTZS7+1HjqjzWhAdOj0XM3A/4b5AUF6sw3UtP+vCUK0+OMKVBifA7fr0CGLbTbZFaEBvBnUBUXqdGKqy60Ge40SLIggYSWqu0XuUL5lZfiGtnL0kqQ/92PRnZ; 20:0ApLdUFyaFi2PdUUIPNZW9egRW3KBoUt7ayOMC34+sahVu13snWbDsFgwsoJ8GzI3rSqU36GG1yGCuAbNItHHGT0aot/HtgTQqS3rK+zJMJlN6T188LCB0YrU4mg8YhjFft5KDyoeb5ukUFc9ShiG7wbgZxeS8DnDnq5gAnF9Bc=
x-ms-exchange-antispam-srfa-diagnostics: SOS;
x-ms-office365-filtering-correlation-id: 3e351d57-2e91-4f16-0412-08d599d6d4f3
x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4604075)(3008032)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7153060)(7193020); SRVR:MWHPR15MB1517;
x-ms-traffictypediagnostic: MWHPR15MB1517:
x-microsoft-antispam-prvs: <MWHPR15MB1517ADA341A7F004642249F6B6A40@MWHPR15MB1517.namprd15.prod.outlook.com>
x-exchange-antispam-report-test: UriScan:(10436049006162)(9452136761055);
x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(3231221)(11241501184)(944501327)(52105095)(3002001)(93006095)(93001095)(10201501046)(6041310)(20161123562045)(20161123558120)(20161123564045)(20161123560045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(6072148)(201708071742011); SRVR:MWHPR15MB1517; BCL:0; PCL:0; RULEID:; SRVR:MWHPR15MB1517;
x-forefront-prvs: 0632519F33
x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(979002)(346002)(366004)(376002)(396003)(39380400002)(39860400002)(189003)(199004)(6506007)(8936002)(86362001)(14454004)(105586002)(25786009)(81166006)(606006)(81156014)(478600001)(5660300001)(966005)(476003)(97736004)(19627405001)(8676002)(55016002)(54896002)(9686003)(229853002)(2900100001)(76176011)(3660700001)(11346002)(33656002)(5250100002)(186003)(99286004)(6436002)(2906002)(446003)(6246003)(6116002)(7696005)(110136005)(3280700002)(316002)(7736002)(46003)(106356001)(53546011)(102836004)(6306002)(74316002)(68736007)(6606003)(236005)(53936002)(486006)(969003)(989001)(999001)(1009001)(1019001); DIR:OUT; SFP:1102; SCL:1; SRVR:MWHPR15MB1517; H:MWHPR15MB1821.namprd15.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1;
received-spf: None (protection.outlook.com: fb.com does not designate permitted sender hosts)
x-microsoft-antispam-message-info: skaqOTuPGilOHPVpo0m3Btk6JriDKiZnma9azkk1nkvKLN5tdp3BZhk+yJBfKy0EBehPM9/Bw602TJ5ABzkBa/QvEkUHy1DhiTm4h0PRMi6SDqTK1PUxG5eUGa4skrWTlj7CnFOusKQ+fqZ4OZ9nV2jrkJev7DW2TvOf2E+RDA4Dd8kYYxOMpPK4I8H9KPaR+k/ZF54Ewuh/RavwfQvaSoT7zA5p00W6ySRjwudX5SUzE2bfYPt7j+r1AYQ/JWxw0uxOyoV0QS9JXccf0mEs3YqAadUX6IW4PEhOjjB+YZcNAQGYU7kmnrunS9fRID91Q9VoyNp01SqotuqK6ANQiRlXEoV988k/qQU0jvdo1DYuQN2tegUDL1w8arUPmfFizgydIUB6SQF2wvuPdmS7CWm7jlPoBhddCJ0YibziB/k=
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative; boundary="_000_MWHPR15MB18215781CA00A71CF1AD0137B6A40MWHPR15MB1821namp_"
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 3e351d57-2e91-4f16-0412-08d599d6d4f3
X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Apr 2018 02:50:31.5977 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR15MB1517
X-OriginatorOrg: fb.com
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2018-04-03_11:, , signatures=0
X-Proofpoint-Spam-Reason: safe
X-FB-Internal: Safe
Archived-At: <https://mailarchive.ietf.org/arch/msg/quic/BJxE_EQZXVk5PkhByygGLbyWR6s>
X-BeenThere: quic@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <quic.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/quic>, <mailto:quic-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/quic/>
List-Post: <mailto:quic@ietf.org>
List-Help: <mailto:quic-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/quic>, <mailto:quic-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 04 Apr 2018 02:50:38 -0000

Thanks for sharing this Ian.


This definitely matches some of the observations we've seen as well in the UDP write path. Some of the other paths that we saw that added overhead was the route table lookup in linux udp stack. Connected UDP sockets did amortize that.


I'm looking forward to a smarter sendmmsg with GSO and zero copy. Is there any indication of the timeline for these patches to make it to linux? Would be happy to try any of these out to help iron out the API.


Subodh

________________________________
From: QUIC <quic-bounces@ietf.org> on behalf of Ian Swett <ianswett=40google.com@dmarc.ietf.org>
Sent: Tuesday, April 3, 2018 5:20:08 PM
To: IETF QUIC WG
Subject: UDP send costs in Linux

One challenge with QUIC at the moment is the increased CPU cost of sending UDP packets vs TCP payloads.  I've seen this across every platform Google has deployed QUIC on, so it's a widespread issue.

Here's an excellent presentation on what's causing the increased CPU consumption on Linux from Willem de Bruijn(UDP starts on slide 9).
http://vger.kernel.org/netconf2017_files/rx_hardening_and_udp_gso.pdf

And while you're thinking of CPU usage, it's worth looking at the presentation on timing wheel based packet pacing(which is minimum release time based) and is ideal for QUIC(and TCP for that matter): https://conferences.sigcomm.org/sigcomm/2017/files/program/ts-9-4-carousel.pdf<https://urldefense.proofpoint.com/v2/url?u=https-3A__conferences.sigcomm.org_sigcomm_2017_files_program_ts-2D9-2D4-2Dcarousel.pdf&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=h3Ju9EBS7mHtwg-wAyN7fQ&m=tgghgvFkps7jYaFNdNyZBNFf0epVxFZbOGhhybFwPiE&s=87gOGfz3S0lLbw8jy-lz3M9vPGChkmgtiJVzUxbMfvY&e=>

-Ian