Re: UDP send costs in Linux

Frederick Kautz <> Wed, 04 April 2018 16:46 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 1C73712DA3E for <>; Wed, 4 Apr 2018 09:46:00 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: 0.09
X-Spam-Status: No, score=0.09 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, HTTPS_HTTP_MISMATCH=1.989, RCVD_IN_DNSWL_NONE=-0.0001] autolearn=unavailable autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id BptDrdwymD0t for <>; Wed, 4 Apr 2018 09:45:53 -0700 (PDT)
Received: from ( [IPv6:2607:f8b0:4001:c0b::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 88F3E120724 for <>; Wed, 4 Apr 2018 09:45:53 -0700 (PDT)
Received: by with SMTP id f6-v6so17818584ita.2 for <>; Wed, 04 Apr 2018 09:45:53 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=O7IE8sFb0z829C0MLDVuuJFo+O62yY/b3fofrkCZXJA=; b=nGpLOUneBMlz7NQc+WFWVq4WFAs/NpUmslvcLK3ixrBkEKXQ8y8hjMxqzGrVCoK0sY DUB6R6lHmcFA9LC2WGZ3axohYDa8LQzvQHDIz+TBZTWvOMED9U44bcnzfyMsgUXbbrnT qi1gTkoASFRxJ7iluase21NdiEVjk0yvI0CjUmq/1SyNhZaoJRGq/ocaU1QgqlcSMMDw biWm4//kHeMp/yM5MKsKw7gXbJhZJvqwE3gLJiXvDPa3h1nbxHGWqFcT8/5bxjWB8H1r S8UZjLhau0/GOirOagJ99dnG2KwNv/Z47Nj/mx//XDoqDhyvwWV2VDIJGyn/qMR5REy9 f8Hg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=O7IE8sFb0z829C0MLDVuuJFo+O62yY/b3fofrkCZXJA=; b=fD55UuEfxSPMb6Db+0D6D6gNTgd7XDg6DPSLRZLNcxZWkyL13FVQz+MiA5b15sQ+Km MwDq1oeXGCVuyCH4Zdp0mmPdy4rDaNARNSR559ti1qHGP5jPprJ4QsXlV750Xaa0nt5t uk7rwO13PxELTjkUqHk8OY5H6U/rjLx0knwwcER53DUQhW/e1myMdgo5rZgyK/Wp0I6u p/8OuTJgGlp9Tr8KdIu9bqpwW8ZbLtZy/e1GlmtM35VMIFEtxN0Jv2fre4aexMSAOCML omorcWRMoQUE1v5aLVwMv8YGFn1oMI4tGRkP9n4Cr73rlLWZphYCxOcxTry2vK/A0dEb pWGQ==
X-Gm-Message-State: ALQs6tCExoW3GN1L7rzDg15GDYXX/JNtXtnxi8HVY4nt+99ciW5E+fI6 Ulo3WGIM4ISbqCzk2TEYmmt0eQ9TY6e9XxVmAXjqWg==
X-Google-Smtp-Source: AIpwx4/e6A5TulfK/ddIocj1vBTPGmy5wTZzlzDz5q/PSMethTvzmJZ9bRa+pTS5WdJukrcr29t1l6S78o0DcnrpERI=
X-Received: by 2002:a24:ec8:: with SMTP id 191-v6mr10999601ite.42.1522860352816; Wed, 04 Apr 2018 09:45:52 -0700 (PDT)
MIME-Version: 1.0
References: <> <> <> <> <>
In-Reply-To: <>
From: Frederick Kautz <>
Date: Wed, 04 Apr 2018 16:45:41 +0000
Message-ID: <>
Subject: Re: UDP send costs in Linux
To: Phillip Hallam-Baker <>
Cc: Mikkel Fahnøe Jørgensen <>, Subodh Iyengar <>, IETF QUIC WG <>, Ian Swett <>
Content-Type: multipart/alternative; boundary="0000000000005f97c10569088e99"
Archived-At: <>
X-Mailman-Approved-At: Wed, 04 Apr 2018 12:39:47 -0700
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 04 Apr 2018 16:52:34 -0000

I agree with this sentiment. If we see traction, then we should see better
kernel support and offloading.

It should be trivial to get this working in something like VPP which can
use DPDK to hw offload, entirely skipping the kernel in the data path.

There are also techniques to bypass the initial memory allocation in the
kernel such as ebpf used by cilium and memif used by vpp.

My main concern at this point would be with cloud native environments with
limited hw offloading support. E.g. if we run in AWS or GCE, better kernel
support will probably be necessary.

On Wed, Apr 4, 2018, 9:28 AM Phillip Hallam-Baker <>

> I would not worry too much at this point.
> The reason we want to be able to work at the application level is
> backwards compatibility. It has to be possible to deploy QUIC on any
> machine even without OS support or it won't be deployable.
> It does not have to be performant on every platform. If people are using
> QUIC, whatever needs to be moved into the kernel for performance reasons
> will move there.
> On Wed, Apr 4, 2018 at 11:32 AM, Mikkel Fahnøe Jørgensen <
>> wrote:
>> I have no data to add on the Linux UDP stack, but another issue is the
>> lack of netmap support in cloud hosting environments.
>> I have not yet been working with this, but have looked into the problem
>> and asked around.
>> netmap is default in FreeBSD and optional in Linux. But neither works
>> efficiently without a hypervisor patch that is also available for netmap.
>> With support for netmap, the user space application can send directly to
>> the network adapter with very little overhead. There is also dpdk and some
>> other interfaces that might be slightly faster but more vendor specific.
>> Assuming an application has access to optimized netmap, the only hurdle
>> is address lookup, but if the application also manages that, or at least
>> does the caching, there shouldn’t be much in the way of OS interference.
>> Of course, netmap blocks the entire network stack, so no PING or SSH.
>> CloudFlare added a netmap patch so only some traffic would be routed fra
>> the network interface to netmap, and netmap also supports efficient packet
>> forwarding to the OS or other applications.
>> None of this works well in general, but for a cloud host that can be
>> bootet automatically and destroyed rather than serviced, there is some
>> opportunity.
>> but only if cloud service providers starts adding support their supported
>> images and hypervisors. Not sure if any are working on this now.
>> Mikkel
>> On 4 April 2018 at 15.11.17, Ian Swett (
>> wrote:
>> I hope some of these patches will be available soon, but I'm not sure if
>> soon is a month or 6.
>> On Tue, Apr 3, 2018 at 10:50 PM Subodh Iyengar <> wrote:
>>> Thanks for sharing this Ian.
>>> This definitely matches some of the observations we've seen as well in
>>> the UDP write path. Some of the other paths that we saw that added overhead
>>> was the route table lookup in linux udp stack. Connected UDP sockets did
>>> amortize that.
>>> I'm looking forward to a smarter sendmmsg with GSO and zero copy. Is
>>> there any indication of the timeline for these patches to make it to linux?
>>> Would be happy to try any of these out to help iron out the API.
>>> Subodh
>>> ------------------------------
>>> *From:* QUIC <> on behalf of Ian Swett
>>> <>
>>> *Sent:* Tuesday, April 3, 2018 5:20:08 PM
>>> *To:* IETF QUIC WG
>>> *Subject:* UDP send costs in Linux
>>> One challenge with QUIC at the moment is the increased CPU cost of
>>> sending UDP packets vs TCP payloads.  I've seen this across every platform
>>> Google has deployed QUIC on, so it's a widespread issue.
>>> Here's an excellent presentation on what's causing the increased CPU
>>> consumption on Linux from Willem de Bruijn(UDP starts on slide 9).
>>> And while you're thinking of CPU usage, it's worth looking at the
>>> presentation on timing wheel based packet pacing(which is minimum release
>>> time based) and is ideal for QUIC(and TCP for that matter):
>>> <>
>>> -Ian