Re: UDP send costs in Linux

Ian Swett <ianswett@google.com> Wed, 04 April 2018 13:11 UTC

Return-Path: <ianswett@google.com>
X-Original-To: quic@ietfa.amsl.com
Delivered-To: quic@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6E0FF1289B0 for <quic@ietfa.amsl.com>; Wed, 4 Apr 2018 06:11:08 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.721
X-Spam-Level:
X-Spam-Status: No, score=-0.721 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, HTTPS_HTTP_MISMATCH=1.989, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lfe1yVYFsi7J for <quic@ietfa.amsl.com>; Wed, 4 Apr 2018 06:11:06 -0700 (PDT)
Received: from mail-io0-x22b.google.com (mail-io0-x22b.google.com [IPv6:2607:f8b0:4001:c06::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 0663E127333 for <quic@ietf.org>; Wed, 4 Apr 2018 06:11:06 -0700 (PDT)
Received: by mail-io0-x22b.google.com with SMTP id m83so26261614ioi.8 for <quic@ietf.org>; Wed, 04 Apr 2018 06:11:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=W4O5XNvU9C6PAWsBKfN/Dz6Zx/DnejLe5WdXRb9npKI=; b=Jxg8ak2BiqyH++x/I9RfQejzYza3UnPq15Xl0+kzM3Ty0uHWGutxtjFoM1o1BhFnRM qRe2rs2X0TDBokNxPkOrzau8vkhuJUGr25N2yczYAmGbVMMLKYdIqWbAVNschAKJnsJe xvvyBeBYKOEC1I9C7acfLrsEg4aDwgAkgtAkXajmsDrGfYOaJzzEtrCds2P7vk87XFcG 9rNa4IqdA4HdRFuuiHVFx3ccFBjUvf9MBvT30uMsKbommfhnwesYm1pk3LIfjbQDYkyy uCcgzbevzgG+/r9dHgbQQqNXXT1xtZr33b215fsa5c4k9W9429wJjZVNNFqmhJB7IiYF YECQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=W4O5XNvU9C6PAWsBKfN/Dz6Zx/DnejLe5WdXRb9npKI=; b=dR9wrwFR0zVUDM6DQj5o4TBkTWqG4X8F+Eoc5krvHqaZj5q4IOzVIkTPzh8v3llKvx YZx0jAHuEDqJxgi92JJl2Xb9rasAV/9R/orx6dUKGrCD28quUF8HF/lv4MoJ0qaMjlpW yTpp8etH1hbdgEPZemsi1SBOSePKm6V+amgFss74JLiUN1n0ngRzJx88KqXsMcO4Koyc dXUVOuZkQ+xrG2N2Z6uIlUbebvvc9K7DJB71oEC8/0/LgwTX0zqWIJIph+F32Y/ypsly CO7rKkRMLjUCJs1UFAP2mT42GZSpxnJYu+/LBWa5dvg02TiiEx16gKaq9r0oyLeVZ9o5 ZiRw==
X-Gm-Message-State: ALQs6tAkYMlWSXNE/TERxG+dhbB09BqHp8r1Kt/ZEvfB/nURchED0Xyi oGde3V+Vjcgsh8EUPfg0IHTQtBmz81gDXd3UcZeanA==
X-Google-Smtp-Source: AIpwx4+wswoNIcBXpJI1WWIWLCGhMjx63FCw8yysHxlufFey/N66XhqNPBo1M068gbOd2ExFnHUxhQghqilNyTk2M6Y=
X-Received: by 10.107.12.202 with SMTP id 71mr16011999iom.63.1522847465141; Wed, 04 Apr 2018 06:11:05 -0700 (PDT)
MIME-Version: 1.0
References: <CAKcm_gP4zz1bW5T-_N2Oxy6o5Sw2mEs3DFU9_HrmfkuaJyLz0A@mail.gmail.com> <MWHPR15MB18215781CA00A71CF1AD0137B6A40@MWHPR15MB1821.namprd15.prod.outlook.com>
In-Reply-To: <MWHPR15MB18215781CA00A71CF1AD0137B6A40@MWHPR15MB1821.namprd15.prod.outlook.com>
From: Ian Swett <ianswett@google.com>
Date: Wed, 04 Apr 2018 13:10:53 +0000
Message-ID: <CAKcm_gO1BdLJOfyQeURWU0jmJo7q9Zft4U9fu1o9py9Bys5NeA@mail.gmail.com>
Subject: Re: UDP send costs in Linux
To: Subodh Iyengar <subodh@fb.com>
Cc: IETF QUIC WG <quic@ietf.org>
Content-Type: multipart/alternative; boundary="001a113fc42c3588bd0569058ea4"
Archived-At: <https://mailarchive.ietf.org/arch/msg/quic/DEJeiNiev_geFXi4r5lD0BvfxSU>
X-BeenThere: quic@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <quic.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/quic>, <mailto:quic-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/quic/>
List-Post: <mailto:quic@ietf.org>
List-Help: <mailto:quic-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/quic>, <mailto:quic-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 04 Apr 2018 13:11:08 -0000

I hope some of these patches will be available soon, but I'm not sure if
soon is a month or 6.

On Tue, Apr 3, 2018 at 10:50 PM Subodh Iyengar <subodh@fb.com>; wrote:

> Thanks for sharing this Ian.
>
>
> This definitely matches some of the observations we've seen as well in the
> UDP write path. Some of the other paths that we saw that added overhead was
> the route table lookup in linux udp stack. Connected UDP sockets did
> amortize that.
>
>
> I'm looking forward to a smarter sendmmsg with GSO and zero copy. Is there
> any indication of the timeline for these patches to make it to linux? Would
> be happy to try any of these out to help iron out the API.
>
>
> Subodh
> ------------------------------
> *From:* QUIC <quic-bounces@ietf.org>; on behalf of Ian Swett
> <ianswett=40google.com@dmarc.ietf..org>
> *Sent:* Tuesday, April 3, 2018 5:20:08 PM
> *To:* IETF QUIC WG
> *Subject:* UDP send costs in Linux
>
> One challenge with QUIC at the moment is the increased CPU cost of sending
> UDP packets vs TCP payloads.  I've seen this across every platform Google
> has deployed QUIC on, so it's a widespread issue.
>
> Here's an excellent presentation on what's causing the increased CPU
> consumption on Linux from Willem de Bruijn(UDP starts on slide 9).
> http://vger.kernel.org/netconf2017_files/rx_hardening_and_udp_gso.pdf
>
> And while you're thinking of CPU usage, it's worth looking at the
> presentation on timing wheel based packet pacing(which is minimum release
> time based) and is ideal for QUIC(and TCP for that matter):
> https://conferences.sigcomm.org/sigcomm/2017/files/program/ts-9-4-carousel.pdf
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__conferences.sigcomm.org_sigcomm_2017_files_program_ts-2D9-2D4-2Dcarousel.pdf&d=DwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=h3Ju9EBS7mHtwg-wAyN7fQ&m=tgghgvFkps7jYaFNdNyZBNFf0epVxFZbOGhhybFwPiE&s=87gOGfz3S0lLbw8jy-lz3M9vPGChkmgtiJVzUxbMfvY&e=>
>
> -Ian
>