RE: Impact of hardware offloads on network stack performance

"Deval, Manasi" <> Thu, 10 May 2018 13:38 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id C5324124217 for <>; Thu, 10 May 2018 06:38:29 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -6.899
X-Spam-Status: No, score=-6.899 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-5, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=unavailable autolearn_force=no
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id xOdXzeq1-7kb for <>; Thu, 10 May 2018 06:38:26 -0700 (PDT)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 7ABEC12DA14 for <>; Thu, 10 May 2018 06:38:26 -0700 (PDT)
X-Amp-Result: SKIPPED(no attachment in message)
X-Amp-File-Uploaded: False
Received: from ([]) by with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 10 May 2018 06:38:25 -0700
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos; i="5.49,385,1520924400"; d="scan'208,217"; a="40178803"
Received: from ([]) by with ESMTP; 10 May 2018 06:38:25 -0700
Received: from ( by ( with Microsoft SMTP Server (TLS) id 14.3.319.2; Thu, 10 May 2018 06:38:25 -0700
Received: from ([]) by ([]) with mapi id 14.03.0319.002; Thu, 10 May 2018 06:38:24 -0700
From: "Deval, Manasi" <>
To: craigt <>, Roberto Peon <>
CC: Watson Ladd <>, Mike Bishop <>, Praveen Balasubramanian <>, "Lubashev, Igor" <>, Ian Swett <>, Mikkel Fahnøe Jørgensen <>, IETF QUIC WG <>
Subject: RE: Impact of hardware offloads on network stack performance
Thread-Topic: Impact of hardware offloads on network stack performance
Date: Thu, 10 May 2018 13:38:24 +0000
Message-ID: <>
References: <> <> <> <> <DB6PR10MB1766789B25E31EBE70564BD3ACBA0@DB6PR10MB1766.EURPRD10.PROD.OUTLOOK.COM> <> <> <> <> <> <> <> <> <> <> <> <>
In-Reply-To: <>
Accept-Language: en-US
Content-Language: en-US
dlp-product: dlpe-windows
dlp-reaction: no-action
x-originating-ip: []
Content-Type: multipart/alternative; boundary="_000_1F436ED13A22A246A59CA374CBC543998B64E589ORSMSX111amrcor_"
MIME-Version: 1.0
Archived-At: <>
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Thu, 10 May 2018 13:38:30 -0000

The current socket scheme uses the kernel stack to implement the Ethernet – IP – UDP stack processing for QUIC packets. The AF_XDP provides a raw frame so one would need a policy on where to run the protocol stack processing.

Another comment on offloads -
Thanks for the insightful point of how crypto offload enables other offloads like segmentation. A segmentation offload for QUIC would need to increment packet numbers for each of the individual packets put on the wire. The transmitted packet numbers for a transmit scatter gather list would be deterministic. Therefore, this should not adversely impact any retransmits.


From: craigt []
Sent: Thursday, May 10, 2018 5:50 AM
To: Roberto Peon <>
Cc: Watson Ladd <>; Mike Bishop <>; Praveen Balasubramanian <>; Lubashev, Igor <>; Ian Swett <>; Mikkel Fahnøe Jørgensen <>; IETF QUIC WG <>; Deval, Manasi <>
Subject: Re: Impact of hardware offloads on network stack performance

Has anyone experimented with the zerocopy socket interfaces for exactly this use case?

It would seem that is a more natural fit to the assertions made by several of the authors surrounding packetisation.
I'm unaware if Windows/BSD have similar interfaces/implementations.


On 30 April 2018 at 21:37, Roberto Peon <<>> wrote:
QUIC (mostly) doesn’t do that (rexmit a packet verbatim)—you’d need a new packet# for the rexmitted packet.

From: QUIC <<>> on behalf of Watson Ladd <<>>
Date: Monday, April 23, 2018 at 1:56 PM
To: Mike Bishop <<>>
Cc: Praveen Balasubramanian <<>>, "Lubashev, Igor" <<>>, Ian Swett <<>>, Mikkel Fahnøe Jørgensen <<>>, IETF QUIC WG <<>>, "Deval, Manasi" <<>>
Subject: Re: Impact of hardware offloads on network stack performance

On Mon, Apr 23, 2018 at 1:06 PM, Mike Bishop <<>> wrote:
I agree – the point of offloading the crypto is not only to spare that cost on the CPU, though that can be useful, it’s to enable further offloads that can’t be done on the encrypted packet.

LRO that could simply group packets by CID and deliver a single batch of packets from a single CID in a single invocation seems like a great first step, and doesn’t even require crypto offload.  Once you have crypto offload, of course those packets per CID could be delivered already decrypted.  I’d want to see data on whether there’s value in making the received data look like one huge packet when there’s not a single bytestream, but that’s not something we need to specify here.

LSO is more interesting, and the challenge there is that the transport stack still needs to know what data was in which packet so that it can do loss recovery properly.  QUIC explicitly disallows fragmentation of the UDP datagram, for the “painful and potentially non-deterministic” reasons you referenced.

Forgive my complete ignorance, but doesn't it just need to know what packet it sent?

More explicitly the sender would maintain a queue of sendable packets, and mark which have been sent. Acknowledged packets are removed from the queue. In event of loss, they simply retransmit the same packet. It's up to higher layers to pick the right size of packet and take data from the individual QUIC streams. If you wanted to expose the stream structure, that sounds a lot more complicated.