Re: Impact of hardware offloads on network stack performance

craigt <c@gryning.com> Thu, 10 May 2018 12:50 UTC

Return-Path: <c@gryning.com>
X-Original-To: quic@ietfa.amsl.com
Delivered-To: quic@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A1E4A12D883 for <quic@ietfa.amsl.com>; Thu, 10 May 2018 05:50:18 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.609
X-Spam-Level:
X-Spam-Status: No, score=-2.609 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gryning-com.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cXAK9dusNDL9 for <quic@ietfa.amsl.com>; Thu, 10 May 2018 05:50:14 -0700 (PDT)
Received: from mail-lf0-x235.google.com (mail-lf0-x235.google.com [IPv6:2a00:1450:4010:c07::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 3A78E1275AB for <quic@ietf.org>; Thu, 10 May 2018 05:50:14 -0700 (PDT)
Received: by mail-lf0-x235.google.com with SMTP id t129-v6so2807959lff.3 for <quic@ietf.org>; Thu, 10 May 2018 05:50:14 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gryning-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=0s/vCrkmQGVG9bnxM6blDr3P4Q97LYzmEdCDZgFtXYs=; b=Kcaj/YfwC0jMMQGqo1PxX9tp5wUnY2I2e6OhhIgEOgceClQN46pgm9xS76pxK3asDi lW0uncXLcl0FLbrrq07xiMY8Mhx8KvkMmt/+JcgQQFe6IndxcfzJ2AU2k9+VhOhu6tcd nHenNVoCsjyLXvB7ESGJ+aSOSEPddhyHWjwqFyjitV690l0P4OILBZQBHcHS+bLEts2D jCaSdJxfWp8cOS1pYooufzE0XIueaCX5U/6jTdaYpsoaZ0I/CCwhAhpOMl5W7XshnYxV i+jlz0G2qnYwf2wF8N0Ra+k9eGK+hHL5PGQCtNvlC6w8wEdRV+5/Rql2CA+VJHbXrfmB BiqA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=0s/vCrkmQGVG9bnxM6blDr3P4Q97LYzmEdCDZgFtXYs=; b=gNGQkhhGoekssM248/p56AkpiaXxOiBpaToPnEDaySpGleG4l+nykd4poqtw/pky0M EtX0M8m+J7CDle2xaQQmvp1XbppSfnNqGSmeKlyvnKOCUjm5sFgKBLb5yQ2h/zhWF/6F nTY0iWQTF+dCj5+W1jMt+sWD69Ab7ShHuR0lKAqEsPOSV0jQvd6mrzA6O26xwHciLv04 UWI88jEsl4UahaR3w2K9+QiLCwQ8nZ89Xr5skAR+bKRtza4S+hWi7xKZPpMd5rtS56TO xCW7Uvo+QWfFfla92WI2srqx8mA2Rpk+bsUEc+NJ+nOTPpn8jTa6l0MrRvPdCj7UkkfB SRnw==
X-Gm-Message-State: ALKqPwct5xbZACRJiCIhApBpT/fBqzBx5QsOZFtlbOBnB7TD6tlrGERl zIp40e19yfgac1Tq9IAxQdYsAtcNTWhCs9RLSIcOnw==
X-Google-Smtp-Source: AB8JxZptleoyHeF2lusWWbh9NQBpvs4c8NVt674zZkgBpOIo6qFKSjXit653VmqOp/Ek4SG5g7Q3OVHzq7aC9ROEbbg=
X-Received: by 2002:a19:d853:: with SMTP id p80-v6mr1019026lfg.36.1525956612330; Thu, 10 May 2018 05:50:12 -0700 (PDT)
MIME-Version: 1.0
Received: by 2002:a19:d012:0:0:0:0:0 with HTTP; Thu, 10 May 2018 05:50:11 -0700 (PDT)
X-Originating-IP: [2001:41c1:4:8000:186e:9af4:4c85:721a]
In-Reply-To: <A6F55755-D7E1-40E4-8EC3-D5FE77A2EDBE@fb.com>
References: <CY4PR21MB0630CE4DE6BA4EDF54A1FEB1B6A40@CY4PR21MB0630.namprd21.prod.outlook.com> <CAN1APde93+S8CP-KcZmqPKCCvsGRiq6ECPUoh_Qk0j9hqs8h0Q@mail.gmail.com> <SN1PR08MB1854E64D7C370BF0A7456977DABB0@SN1PR08MB1854.namprd08.prod.outlook.com> <CAKcm_gNhsHxXM77FjRj-Wh4JxA21NAWKXX3KBT=eZJsCdacM7Q@mail.gmail.com> <DB6PR10MB1766789B25E31EBE70564BD3ACBA0@DB6PR10MB1766.EURPRD10.PROD.OUTLOOK.COM> <SN1PR08MB18548155722C665FFD45CC4DDABA0@SN1PR08MB1854.namprd08.prod.outlook.com> <CAKcm_gMYf6AbHgvg+WKv-woQCtzz9WztiEDf-iRUjKw63Qx=9Q@mail.gmail.com> <1F436ED13A22A246A59CA374CBC543998B60C7AD@ORSMSX111.amr.corp.intel.com> <CAN1APdc3y0EwFqeYVvZs7MtBHhS_9CzwGmcwRqi_6GHWzF3_2Q@mail.gmail.com> <1F436ED13A22A246A59CA374CBC543998B60C851@ORSMSX111.amr.corp.intel.com> <CAN1APdcPkO-HfXqqvjeee6K8U8KSZQdtx=6fW1vZo+H6pKfzzQ@mail.gmail.com> <CAKcm_gP8TRPT7yi1y=mU5Fvq7xB5-1ieyKFLQMPomfabYbkcxA@mail.gmail.com> <bf540ec1f6f045aca1cf2379380630b5@usma1ex-dag1mb5.msg.corp.akamai.com> <SN1PR08MB185446750FCDA2B58C9833C8DA890@SN1PR08MB1854.namprd08.prod.outlook.com> <CACsn0cmJcAxORo4Cd-qyGJL3ZOT5Yz0WgdMqv_AGi4DjedyzFA@mail.gmail.com> <A6F55755-D7E1-40E4-8EC3-D5FE77A2EDBE@fb.com>
From: craigt <c@gryning.com>
Date: Thu, 10 May 2018 13:50:11 +0100
Message-ID: <CAK-1kemWAW+2Sk5aKJ+RXKUAyjFii4MRH=xK7SE0mr=tbCiRfw@mail.gmail.com>
Subject: Re: Impact of hardware offloads on network stack performance
To: Roberto Peon <fenix@fb.com>
Cc: Watson Ladd <watsonbladd@gmail.com>, Mike Bishop <mbishop@evequefou.be>, Praveen Balasubramanian <pravb@microsoft.com>, "Lubashev, Igor" <ilubashe@akamai.com>, Ian Swett <ianswett=40google.com@dmarc.ietf.org>, Mikkel Fahnøe Jørgensen <mikkelfj@gmail.com>, IETF QUIC WG <quic@ietf.org>, "Deval, Manasi" <manasi.deval@intel.com>
Content-Type: multipart/alternative; boundary="000000000000d22b22056bd97523"
Archived-At: <https://mailarchive.ietf.org/arch/msg/quic/JXWQV1yEQiSJzSrP4zbKQr3Mj6M>
X-BeenThere: quic@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <quic.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/quic>, <mailto:quic-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/quic/>
List-Post: <mailto:quic@ietf.org>
List-Help: <mailto:quic-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/quic>, <mailto:quic-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 10 May 2018 12:50:19 -0000

Has anyone experimented with the zerocopy socket interfaces for exactly
this use case?
  https://netdevconf.org/2.1/papers/netdev.pdf

It would seem that is a more natural fit to the assertions made by several
of the authors surrounding packetisation.
I'm unaware if Windows/BSD have similar interfaces/implementations.

c


On 30 April 2018 at 21:37, Roberto Peon <fenix@fb.com> wrote:

> QUIC (mostly) doesn’t do that (rexmit a packet verbatim)—you’d need a new
> packet# for the rexmitted packet.
>
> -=R
>
>
>
> *From: *QUIC <quic-bounces@ietf.org> on behalf of Watson Ladd <
> watsonbladd@gmail.com>
> *Date: *Monday, April 23, 2018 at 1:56 PM
> *To: *Mike Bishop <mbishop@evequefou.be>
> *Cc: *Praveen Balasubramanian <pravb@microsoft.com>, "Lubashev, Igor" <
> ilubashe@akamai.com>, Ian Swett <ianswett=40google.com@dmarc.ietf.org>,
> Mikkel Fahnøe Jørgensen <mikkelfj@gmail.com>, IETF QUIC WG <quic@ietf.org>,
> "Deval, Manasi" <manasi.deval@intel.com>
> *Subject: *Re: Impact of hardware offloads on network stack performance
>
>
>
>
>
>
>
> On Mon, Apr 23, 2018 at 1:06 PM, Mike Bishop <mbishop@evequefou.be> wrote:
>
> I agree – the point of offloading the crypto is not only to spare that
> cost on the CPU, though that can be useful, it’s to enable further offloads
> that can’t be done on the encrypted packet.
>
>
>
> LRO that could simply group packets by CID and deliver a single batch of
> packets from a single CID in a single invocation seems like a great first
> step, and doesn’t even require crypto offload.  Once you have crypto
> offload, of course those packets per CID could be delivered already
> decrypted.  I’d want to see data on whether there’s value in making the
> received data look like one huge packet when there’s not a single
> bytestream, but that’s not something we need to specify here.
>
>
>
> LSO is more interesting, and the challenge there is that the transport
> stack still needs to know what data was in which packet so that it can do
> loss recovery properly.  QUIC explicitly disallows fragmentation of the UDP
> datagram, for the “painful and potentially non-deterministic” reasons you
> referenced.
>
>
>
> Forgive my complete ignorance, but doesn't it just need to know what
> packet it sent?
>
> More explicitly the sender would maintain a queue of sendable packets, and
> mark which have been sent. Acknowledged packets are removed from the queue.
> In event of loss, they simply retransmit the same packet. It's up to higher
> layers to pick the right size of packet and take data from the individual
> QUIC streams. If you wanted to expose the stream structure, that sounds a
> lot more complicated.
>
>
> Sincerely,
>
> Watson
>
>
>
>
>