RE: UDP send costs in Linux

Praveen Balasubramanian <> Wed, 04 April 2018 17:57 UTC

Return-Path: <>
Received: from (localhost [IPv6:::1]) by (Postfix) with ESMTP id 69B1C12D779 for <>; Wed, 4 Apr 2018 10:57:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple;; s=ietf1; t=1522864649; bh=23+wXo1YEQpHG1QP2RyLEpec7LDRpIbcPoGHQpDJBmE=; h=From:To:Subject:Date:References:In-Reply-To:Cc:Cc:Cc; b=u5AC4sQtE9vV9lZlY7LSMaLeTIuzo7wF6XEvSwVYcs6tKgt74w+TvyE/9N/XT/0FT qVZncf4KSfkL750/3mRYw8AZbUW6v3VRS2Z9DHK8kuRe0MDJdDY5M3XCFFRgs8O5po otRPl6bAWOm4HmPIOBbR2U2U941hW8ZYcZWYP/oo=
X-Mailbox-Line: From Wed Apr 4 10:57:29 2018
Received: from (localhost [IPv6:::1]) by (Postfix) with ESMTP id 0750A129C6E; Wed, 4 Apr 2018 10:57:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple;; s=ietf1; t=1522864649; bh=23+wXo1YEQpHG1QP2RyLEpec7LDRpIbcPoGHQpDJBmE=; h=From:To:Subject:Date:References:In-Reply-To:Cc:Cc:Cc; b=u5AC4sQtE9vV9lZlY7LSMaLeTIuzo7wF6XEvSwVYcs6tKgt74w+TvyE/9N/XT/0FT qVZncf4KSfkL750/3mRYw8AZbUW6v3VRS2Z9DHK8kuRe0MDJdDY5M3XCFFRgs8O5po otRPl6bAWOm4HmPIOBbR2U2U941hW8ZYcZWYP/oo=
Received: from localhost (localhost []) by (Postfix) with ESMTP id D2E1412AF84 for <>; Wed, 4 Apr 2018 10:57:28 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -0.011
X-Spam-Status: No, score=-0.011 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, HTTPS_HTTP_MISMATCH=1.989, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (1024-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id bEgluOj11zl2 for <>; Wed, 4 Apr 2018 10:57:25 -0700 (PDT)
Received: from ( [IPv6:2a01:111:f400:fe49::71d]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 5BF81129C6E for <>; Wed, 4 Apr 2018 10:57:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=23+wXo1YEQpHG1QP2RyLEpec7LDRpIbcPoGHQpDJBmE=; b=oBeeUWpYDQwOrnhagtTYdQFNXH4klW9bZ7HD2SdGyEiCnP4QpUT09DwiovQpHq7vlpebwjECsN4bWxnO93+4LslH/MOB+Zy1hUEdjxkGn1Y9xJN4LNkKQxuqyAhQ8YZLLGds8gza3+NkSxgwP+UgAuyFKxGbcxKBoDI1PxmbAJw=
Received: from ( by ( with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.675.1; Wed, 4 Apr 2018 17:57:23 +0000
Received: from ([fe80::de:ba33:4748:51da]) by ([fe80::de:ba33:4748:51da%6]) with mapi id 15.20.0675.003; Wed, 4 Apr 2018 17:57:23 +0000
From: Praveen Balasubramanian <>
To: Phillip Hallam-Baker <>, Mikkel Fahnøe Jørgensen <>
Subject: RE: UDP send costs in Linux
Thread-Topic: UDP send costs in Linux
Thread-Index: AQHTy6q+3JsicRIxhkabCq/gojpid6Pv6A6AgACtVICAACezAIAAD5iAgAAXAKA=
Date: Wed, 04 Apr 2018 17:57:22 +0000
Message-ID: <>
References: <> <> <> <> <>
In-Reply-To: <>
Accept-Language: en-US
Content-Language: en-US
x-originating-ip: [2001:4898:80e8:a::712]
x-ms-publictraffictype: Email
x-microsoft-exchange-diagnostics: 1; CY4PR21MB0280; 7:B6jF4JioNq6WAfpbfQJqTF4j4p4BppF+kMctuwp8WKbhUM6j5zG/jUkuktTIkhHJyCAdiQDcWqWcZOA6diq5xY0cSNz2brtpxwfv8lbLmYBxcFYP+ky+Db9L4CQrsGmqHhZZWLK0uTsky4HBZ63WSJSlWPUXa4ePJKxjeqnbL7tAfo4uvPHPeITAOpn8YZUwyOLN3rSpDybFGLM0iwcSyDZIUMW9fwElBJQv0D1DyArR7n2p7H2rVCzWuzOWI3cF; 20:jPAldTD8BG80LgDn/2QRPZLxqKvMEiu3Z9gMKULcdflTPZgGaoZ2cfCy9LYm6OUkz/ehovJFk+KKi1F/tIpXYNw3N35tQA0/IU3F2i+vwOvSMEscFPJpdAUTaDL7tmQIhRXhEZMSq/3IfY+HiIU/o+2Si9J2ZX/utBTOkGAdc6U=
x-ms-exchange-antispam-srfa-diagnostics: SOS;
x-ms-office365-filtering-correlation-id: 8e9258f4-1e1e-4618-5105-08d59a55849a
x-ms-office365-filtering-ht: Tenant
x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4604075)(3008032)(48565401081)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7193020); SRVR:CY4PR21MB0280;
x-ms-traffictypediagnostic: CY4PR21MB0280:
authentication-results: spf=none (sender IP is );
x-microsoft-antispam-prvs: <>
x-exchange-antispam-report-test: UriScan:(28532068793085)(158342451672863)(9452136761055)(189930954265078)(85827821059158)(67672495146484)(219752817060721)(21748063052155);
x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(61425038)(6040522)(2401047)(5005006)(8121501046)(10201501046)(3002001)(3231221)(944501327)(52105095)(93006095)(93001095)(6055026)(61426038)(61427038)(6041310)(20161123560045)(20161123564045)(20161123562045)(20161123558120)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(6072148)(201708071742011); SRVR:CY4PR21MB0280; BCL:0; PCL:0; RULEID:; SRVR:CY4PR21MB0280;
x-forefront-prvs: 0632519F33
x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(396003)(366004)(376002)(39860400002)(346002)(39380400002)(199004)(189003)(2900100001)(39060400002)(6246003)(53936002)(186003)(46003)(97736004)(11346002)(446003)(476003)(99286004)(86362001)(86612001)(25786009)(19609705001)(10290500003)(966005)(478600001)(606006)(3660700001)(10090500001)(4326008)(2906002)(8990500004)(3280700002)(68736007)(8936002)(8676002)(106356001)(105586002)(9686003)(55016002)(14454004)(54896002)(6306002)(81156014)(5250100002)(81166006)(6436002)(93886005)(7736002)(236005)(316002)(33656002)(22452003)(7696005)(76176011)(229853002)(6116002)(486006)(59450400001)(102836004)(6506007)(53546011)(74316002)(5660300001)(110136005)(54906003)(790700001); DIR:OUT; SFP:1102; SCL:1; SRVR:CY4PR21MB0280;; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1;
received-spf: None ( does not designate permitted sender hosts)
x-microsoft-antispam-message-info: pv4Lv0ARfLZqnPepQvN/DcsLoMk/G8qOMuibTCbAW/G1U9J5s87JXrJz1kH/AgJp6ttBQ3xD0E/6fHte1dRHOlCW18/NKfb8nc09w0xktwSkwxk6A0JwA5x6BIyq1Tkj5ACcnXzVSYaBiGyNOMKQwUNJe3Ex3wlqjYLJ4caDoSKwbJKesKAkBB6mncxnTjjUnDUcDoyL9rRNZLsvth03lejfJ+8MTpXemvLwF6HIhwnjR3TyFAohaydQ7jP8v3pyWFd/kb2rJG9Z1SDhco+AyHrLhQtpMYQ6k8aKf28LQzKY2ax/QeUWwfgxta5wRJIG0/UpM6OfVDvRAb71J1/632XLs4vmbn58pxvL516/ssYz5EEGq9EAxoMp0z/rmGDv7A67XDNTiE+Qyx00sLGGYOMFLMkmH7B9F2E5vzPh5yA=
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative; boundary="_000_CY4PR21MB06302D6BC1865A5889759548B6A40CY4PR21MB0630namp_"
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 8e9258f4-1e1e-4618-5105-08d59a55849a
X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Apr 2018 17:57:22.9535 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR21MB0280
Cc: Subodh Iyengar <>
Cc: Ian Swett <>
Archived-At: <>
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 04 Apr 2018 17:57:29 -0000

The need to worry is not on the client side – at least not immediately. 802.11ad, 802.11ax, and 5G LTE will bring Gigabit rates to client side so it is certainly going to become a problem longer term.

The current worry is on the server side. A large part of the work we do is performance optimization of the network stack. Google’s Sigcomm QUIC paper shows a 2x CPU increase going from TCP to QUIC. Most web services will NOT be able to make that trade off to get latency improvements for 90th percentile.

IMO improving UDP performance and hardware offloads are absolutely a necessity for QUIC to become widely adopted (not just by the biggest corporations with a large budget). We have work under way on both these fronts and I am very happy to see Linux also investing here in preparation for what’s coming. We need to do our best democratize this technology - multiple implementations and performance will both play a big role.

From: QUIC [] On Behalf Of Phillip Hallam-Baker
Sent: Wednesday, April 4, 2018 9:29 AM
To: Mikkel Fahnøe Jørgensen <>
Cc: Subodh Iyengar <>; IETF QUIC WG <>; Ian Swett <>
Subject: Re: UDP send costs in Linux

I would not worry too much at this point.

The reason we want to be able to work at the application level is backwards compatibility. It has to be possible to deploy QUIC on any machine even without OS support or it won't be deployable.

It does not have to be performant on every platform. If people are using QUIC, whatever needs to be moved into the kernel for performance reasons will move there.

On Wed, Apr 4, 2018 at 11:32 AM, Mikkel Fahnøe Jørgensen <<>> wrote:
I have no data to add on the Linux UDP stack, but another issue is the lack of netmap support in cloud hosting environments.
I have not yet been working with this, but have looked into the problem and asked around.

netmap is default in FreeBSD and optional in Linux. But neither works efficiently without a hypervisor patch that is also available for netmap. With support for netmap, the user space application can send directly to the network adapter with very little overhead. There is also dpdk and some other interfaces that might be slightly faster but more vendor specific.

Assuming an application has access to optimized netmap, the only hurdle is address lookup, but if the application also manages that, or at least does the caching, there shouldn’t be much in the way of OS interference.

Of course, netmap blocks the entire network stack, so no PING or SSH. CloudFlare added a netmap patch so only some traffic would be routed fra the network interface to netmap, and netmap also supports efficient packet forwarding to the OS or other applications.

None of this works well in general, but for a cloud host that can be bootet automatically and destroyed rather than serviced, there is some opportunity.

but only if cloud service providers starts adding support their supported images and hypervisors. Not sure if any are working on this now.


On 4 April 2018 at 15.11.17, Ian Swett (<>) wrote:
I hope some of these patches will be available soon, but I'm not sure if soon is a month or 6.

On Tue, Apr 3, 2018 at 10:50 PM Subodh Iyengar <<>> wrote:

Thanks for sharing this Ian.

This definitely matches some of the observations we've seen as well in the UDP write path. Some of the other paths that we saw that added overhead was the route table lookup in linux udp stack. Connected UDP sockets did amortize that.

I'm looking forward to a smarter sendmmsg with GSO and zero copy. Is there any indication of the timeline for these patches to make it to linux? Would be happy to try any of these out to help iron out the API.


From: QUIC <<>> on behalf of Ian Swett <<>>
Sent: Tuesday, April 3, 2018 5:20:08 PM
Subject: UDP send costs in Linux

One challenge with QUIC at the moment is the increased CPU cost of sending UDP packets vs TCP payloads.  I've seen this across every platform Google has deployed QUIC on, so it's a widespread issue.

Here's an excellent presentation on what's causing the increased CPU consumption on Linux from Willem de Bruijn(UDP starts on slide 9).<>

And while you're thinking of CPU usage, it's worth looking at the presentation on timing wheel based packet pacing(which is minimum release time based) and is ideal for QUIC(and TCP for that matter):<>