Re: UDP send costs in Linux

Mikkel Fahnøe Jørgensen <> Wed, 04 April 2018 15:33 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 8437B12DA3E for <>; Wed, 4 Apr 2018 08:33:03 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -0.709
X-Spam-Status: No, score=-0.709 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, HTTPS_HTTP_MISMATCH=1.989, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, UNPARSEABLE_RELAY=0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id VV0NCW-F_NWz for <>; Wed, 4 Apr 2018 08:33:00 -0700 (PDT)
Received: from ( [IPv6:2607:f8b0:4001:c0b::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id EBDE512DA0A for <>; Wed, 4 Apr 2018 08:32:59 -0700 (PDT)
Received: by with SMTP id h143-v6so28701166ita.4 for <>; Wed, 04 Apr 2018 08:32:59 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=from:in-reply-to:references:mime-version:date:message-id:subject:to :cc; bh=aTyqhELuVKSysofcwoyosAhA84iAj62sjx6gmHV/dbE=; b=VIDm+YjYuT4H/kqpoMoJlsSVHhch7y1fx4ylY17ESlm5GdMK+A9sY8BBtOZP0ee++W 0yMeqpr3gquxt+uVBbdBLep1Srqte5BTY/qRHUCraI2Rit0nUp5kKS7Xqu7jg1SHuvAM 1Vhw9xjKXUV/ZBnQXwIxYdLWJggfMNL4Fbm7zAubJNckgu6O5+N4d4sOY/wFKfppyGb6 CoTtKHsxIu4GE+OO9xEdBaBmFRtr47dZVcIETetKw5E4lvwIsTaJm/HaMF361Xuiuu4K aQs7x8CXzQQ3Q5Qa61ZRfLgjL9/ON5u5HvzXVK3wsQ/B2Z1J6AKHvTfJtTaI7F11oycm ZFTQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:from:in-reply-to:references:mime-version:date :message-id:subject:to:cc; bh=aTyqhELuVKSysofcwoyosAhA84iAj62sjx6gmHV/dbE=; b=cTUQXOYGYCfagG5p4GqSUsZ0+O+N2fhdOH0ZfxewSR4LknXIkk9VSa0/VzVFikFQKH qC5xieP8I1RHtY10K1GzKgDO8/p+DhIXx5GbrrAQ/4/WN0DygH9NFvKHJ8alQ6qkFjQO /0QxcPFrFFvfX/LbKSqbDtOTQOYx8iMLf7qe0JdtV+dRwJaiBFvYFg4ZpjqnDVEkUBeI ECCQXRBdEXbP6mVGK7R42PvuexQ8TLp7FF24fHb4yPDkA5Coc2B0+pFA0Ze1Qe84TG/R X/bugMAmltYeSMaWmxki5bItydLj/DI8rsI9X78baLfTTFO2aPQUg3z/+kmU0B6Ec5k/ P/NA==
X-Gm-Message-State: ALQs6tCPyrHYbiyX8d8VTv9jy6d6SFv4s+MXENVnU3/BTUYawdIjSqTm prCvY+MgEcTjYxOHyXAtqBlvWx567azLpQuUy2jw8A==
X-Google-Smtp-Source: AIpwx4/ERgy0GVTes83al1WtvUuRq7msBGC+iPcaQS97ai0MimgxDmgjNWYA804oGYmjltl2EVh/mIMxDV76Top6UtI=
X-Received: by 2002:a24:7088:: with SMTP id f130-v6mr9856271itc.39.1522855979244; Wed, 04 Apr 2018 08:32:59 -0700 (PDT)
Received: from 1058052472880 named unknown by with HTTPREST; Wed, 4 Apr 2018 08:32:58 -0700
From: Mikkel Fahnøe Jørgensen <>
In-Reply-To: <>
References: <> <> <>
X-Mailer: Airmail (420)
MIME-Version: 1.0
Date: Wed, 04 Apr 2018 08:32:58 -0700
Message-ID: <>
Subject: Re: UDP send costs in Linux
To: Ian Swett <>, Subodh Iyengar <>
Content-Type: multipart/alternative; boundary="000000000000afe04c05690789c5"
Archived-At: <>
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 04 Apr 2018 15:33:03 -0000

I have no data to add on the Linux UDP stack, but another issue is the lack
of netmap support in cloud hosting environments.
I have not yet been working with this, but have looked into the problem and
asked around.

netmap is default in FreeBSD and optional in Linux. But neither works
efficiently without a hypervisor patch that is also available for netmap.
With support for netmap, the user space application can send directly to
the network adapter with very little overhead. There is also dpdk and some
other interfaces that might be slightly faster but more vendor specific.

Assuming an application has access to optimized netmap, the only hurdle is
address lookup, but if the application also manages that, or at least does
the caching, there shouldn’t be much in the way of OS interference.

Of course, netmap blocks the entire network stack, so no PING or SSH.
CloudFlare added a netmap patch so only some traffic would be routed fra
the network interface to netmap, and netmap also supports efficient packet
forwarding to the OS or other applications.

None of this works well in general, but for a cloud host that can be bootet
automatically and destroyed rather than serviced, there is some opportunity.

but only if cloud service providers starts adding support their supported
images and hypervisors. Not sure if any are working on this now.


On 4 April 2018 at 15.11.17, Ian Swett (

I hope some of these patches will be available soon, but I'm not sure if
soon is a month or 6.

On Tue, Apr 3, 2018 at 10:50 PM Subodh Iyengar <> wrote:

> Thanks for sharing this Ian.
> This definitely matches some of the observations we've seen as well in the
> UDP write path. Some of the other paths that we saw that added overhead was
> the route table lookup in linux udp stack. Connected UDP sockets did
> amortize that.
> I'm looking forward to a smarter sendmmsg with GSO and zero copy. Is there
> any indication of the timeline for these patches to make it to linux? Would
> be happy to try any of these out to help iron out the API.
> Subodh
> ------------------------------
> *From:* QUIC <> on behalf of Ian Swett
> <>
> *Sent:* Tuesday, April 3, 2018 5:20:08 PM
> *Subject:* UDP send costs in Linux
> One challenge with QUIC at the moment is the increased CPU cost of sending
> UDP packets vs TCP payloads.  I've seen this across every platform Google
> has deployed QUIC on, so it's a widespread issue.
> Here's an excellent presentation on what's causing the increased CPU
> consumption on Linux from Willem de Bruijn(UDP starts on slide 9).
> And while you're thinking of CPU usage, it's worth looking at the
> presentation on timing wheel based packet pacing(which is minimum release
> time based) and is ideal for QUIC(and TCP for that matter):
> <>
> -Ian