Re: Impact of hardware offloads on network stack performance

craigt <c@gryning.com> Thu, 10 May 2018 15:39 UTC

Return-Path: <c@gryning.com>
X-Original-To: quic@ietfa.amsl.com
Delivered-To: quic@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B8E9D12EB21 for <quic@ietfa.amsl.com>; Thu, 10 May 2018 08:39:41 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.609
X-Spam-Level:
X-Spam-Status: No, score=-2.609 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gryning-com.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gcrpjfJNvtzT for <quic@ietfa.amsl.com>; Thu, 10 May 2018 08:39:39 -0700 (PDT)
Received: from mail-lf0-x22e.google.com (mail-lf0-x22e.google.com [IPv6:2a00:1450:4010:c07::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 2753412762F for <quic@ietf.org>; Thu, 10 May 2018 08:39:39 -0700 (PDT)
Received: by mail-lf0-x22e.google.com with SMTP id r2-v6so3617746lff.4 for <quic@ietf.org>; Thu, 10 May 2018 08:39:39 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gryning-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=CQQhPpg+I4Ab2OCkvCarRUrxK62N8dsZRy432STQPcY=; b=TQlvOcJ+vB6Pk/jt95e66a+uOhqstFyWXueYC4DXFqCfpm7gjACg4GlLZSYOPH+ZY0 bqPOo4OVomgLXlBQSjDN6R/V5t/4/yFxYFnxCpwH1UCYmw6+7bmPBPex6expqbQfC4wz qaMZeuxJ0F6Svb5ouYfGtOeKvxD+SO3sSr/+Xav3BFjai30eStz7xxJ6BDUcUwo4pXT+ noEF6tmirzPl4CTh63uMJxbgqm8CsN0Pm3hESi94DfcMs96KnLKPZ/9VaRFtsVUzQmSK TFQFmrc14m7fXsIzNeI3AWNDcS/1RLovgkAkmYSahXwLbsV9xBXOsStyebayIoVhWiqQ Y69w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=CQQhPpg+I4Ab2OCkvCarRUrxK62N8dsZRy432STQPcY=; b=Q6sJI+gWW98LwPOGTo4BGkFGWNp7jeAwxQ/4zq/xoGxinL2GkT/VJSYa7E5liKFU6I EwqlM8aQy6vcoj/IMUVrhYweiaBPUsTeOz+/i03I9+pLizsMQywM35WVutED0Kfju+dG 1SH62y7+85iyOB2m5hiX/svDk8tEh0VicVTaberOyltsbSoG4L/6VGdZIlD7RUf79NZ5 ieE4yk9sqsp79bCuiUEp0GJzmPZR2eN5uPkiVNcYthSEtN1mQJZmdCsQ/GEYIYqp90Tg KtMVQUbIolKqazFPoKWzuDA9vnIVgjqVLfE/PyvmeCqmJZmxUUhHFQx0eceBuWs2Qfa2 8Ztw==
X-Gm-Message-State: ALKqPwcNTPPL3UKmkO+n/gU6WDF/IecVhtzzfnrJr7j34lNwx94Jd94Y vujG89UJ/ZJ1z463XhaJ1Z/+Lk5LNrM14yemK75fFg==
X-Google-Smtp-Source: AB8JxZro5GiboK1A9xIp77kQSllNzuoYfzri6Ti05v2LqVK7agXswInCPJfuARlFADR4ZOtGH9Xg5Oul3tGPjx0JHRs=
X-Received: by 2002:a2e:89d7:: with SMTP id c23-v6mr1536985ljk.22.1525966777321; Thu, 10 May 2018 08:39:37 -0700 (PDT)
MIME-Version: 1.0
Received: by 2002:a19:d012:0:0:0:0:0 with HTTP; Thu, 10 May 2018 08:39:36 -0700 (PDT)
X-Originating-IP: [2001:41c1:4:8000:e554:161e:d4c5:8156]
In-Reply-To: <CAN1APdejvqPWi-E14PiO6Pts7BpBv9BODMBjRoSDzwRD_hiruw@mail.gmail.com>
References: <CY4PR21MB0630CE4DE6BA4EDF54A1FEB1B6A40@CY4PR21MB0630.namprd21.prod.outlook.com> <1F436ED13A22A246A59CA374CBC543998B60C7AD@ORSMSX111.amr.corp.intel.com> <CAN1APdc3y0EwFqeYVvZs7MtBHhS_9CzwGmcwRqi_6GHWzF3_2Q@mail.gmail.com> <1F436ED13A22A246A59CA374CBC543998B60C851@ORSMSX111.amr.corp.intel.com> <CAN1APdcPkO-HfXqqvjeee6K8U8KSZQdtx=6fW1vZo+H6pKfzzQ@mail.gmail.com> <CAKcm_gP8TRPT7yi1y=mU5Fvq7xB5-1ieyKFLQMPomfabYbkcxA@mail.gmail.com> <bf540ec1f6f045aca1cf2379380630b5@usma1ex-dag1mb5.msg.corp.akamai.com> <SN1PR08MB185446750FCDA2B58C9833C8DA890@SN1PR08MB1854.namprd08.prod.outlook.com> <CACsn0cmJcAxORo4Cd-qyGJL3ZOT5Yz0WgdMqv_AGi4DjedyzFA@mail.gmail.com> <A6F55755-D7E1-40E4-8EC3-D5FE77A2EDBE@fb.com> <CAK-1kemWAW+2Sk5aKJ+RXKUAyjFii4MRH=xK7SE0mr=tbCiRfw@mail.gmail.com> <CAN1APddgX6=CxKdUA718=95EeCUPS60vVxbGFDHa7-eRjw7BxA@mail.gmail.com> <98A67023-AE1D-48EE-BD75-7D03E05E980D@netapp.com> <CAN1APdf2VKXB-sO97S=n5rCC4t6YZL22T=xdxPba9SgRAPKwLA@mail.gmail.com> <5213b2f5-c795-9d54-72b8-01a4a6db03ee@mellanox.com> <9E549481-9B80-4DED-AC37-DC0C91AEBCDE@netapp.com> <CAN1APdcZfs5UwNsyeqb=C3oodC6GzcoPM2EOY8nK1ofV+yqevA@mail.gmail.com> <CAN1APdejvqPWi-E14PiO6Pts7BpBv9BODMBjRoSDzwRD_hiruw@mail.gmail.com>
From: craigt <c@gryning.com>
Date: Thu, 10 May 2018 16:39:36 +0100
Message-ID: <CAK-1kemP1GTVaa-n03OS+BiCDqMEqMYctyPQ+F+Y2S7q1k8Ovg@mail.gmail.com>
Subject: Re: Impact of hardware offloads on network stack performance
To: Mikkel Fahnøe Jørgensen <mikkelfj@gmail.com>
Cc: Boris Pismenny <borisp@mellanox.com>, "Eggert, Lars" <lars@netapp.com>, Roberto Peon <fenix@fb.com>, Mike Bishop <mbishop@evequefou.be>, IETF QUIC WG <quic@ietf.org>, "Lubashev, Igor" <ilubashe@akamai.com>, Praveen Balasubramanian <pravb@microsoft.com>, Watson Ladd <watsonbladd@gmail.com>, Ian Swett <ianswett=40google.com@dmarc.ietf.org>, "Deval, Manasi" <manasi.deval@intel.com>
Content-Type: multipart/alternative; boundary="000000000000b3a07b056bdbd378"
Archived-At: <https://mailarchive.ietf.org/arch/msg/quic/0VY0Qj0HwGz8Ygq_mZlpaEdPuUs>
X-BeenThere: quic@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <quic.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/quic>, <mailto:quic-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/quic/>
List-Post: <mailto:quic@ietf.org>
List-Help: <mailto:quic-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/quic>, <mailto:quic-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 10 May 2018 15:39:42 -0000

The BBC folk involved in that work aren't on this list. If there are
questions about the implementation you would like answered, get in touch
with me and I'll try to get some time.

c

On 10 May 2018 at 16:02, Mikkel Fahnøe Jørgensen <mikkelfj@gmail.com> wrote:

> Some of you probably have inside knowledge of this:
>
> https://www.bbc.co.uk/rd/blog/2018-04-high-speed-networking-
> open-source-kernel-bypass
>
>
> On 10 May 2018 at 16.57.35, Mikkel Fahnøe Jørgensen (mikkelfj@gmail.com)
> wrote:
>
> Netmap is has a fairly low line count and complexity for what it achieves,
> it is not tied to any vendor specific hardware, it patches do seem to
> arrive for various cards.
>
> I recently did a user space emulation of netmap where separate user
> threads drains and fills the tx and rx rings respectively. I’m not sure
> this is worth it in terms of performance since it adds more synchronization
> (netmap makes certain assumptions because some updates are in the kernel).
> However, it does provide a uniform buffering layer and it makes it easier
> to plug in proper netmap when available. It does move some per message
> latency out of the way.
>
>
> On 10 May 2018 at 16.03.28, Eggert, Lars (lars@netapp.com) wrote:
>
> On 2018-5-10, at 14:57, Boris Pismenny <borisp@mellanox.com> wrote:
> > Also, is there a reason to prefer netmap over other userspace network
> stacks like DPDK/VPP/etc?
>
> For me personally, the netmap API is a lot more minimal and therefore
> easier to integrate. DPDK really wants to take over scheduling, memory
> management and all kinds of other stuff besides packet I/O. YMMV.
>
> Lars
>
>