IETF Logo Mail Archive

Impact of hardware offloads on network stack performance

Praveen Balasubramanian <pravb@microsoft.com> Wed, 04 April 2018 20:15 UTC

Return-Path: <pravb@microsoft.com>
X-Original-To: quic@ietfa.amsl.com
Delivered-To: quic@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 4045F12D869 for <quic@ietfa.amsl.com>; Wed, 4 Apr 2018 13:15:43 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2
X-Spam-Level:
X-Spam-Status: No, score=-2 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=microsoft.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id a5ALzAukvkH8 for <quic@ietfa.amsl.com>; Wed, 4 Apr 2018 13:15:41 -0700 (PDT)
Received: from NAM03-BY2-obe.outbound.protection.outlook.com (mail-by2nam03on0132.outbound.protection.outlook.com [104.47.42.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 88207126DCA for <quic@ietf.org>; Wed, 4 Apr 2018 13:15:41 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=xpmmWNbwYcMpDgSM+JnuiesMZRBsRoKUn9YeRgPp6gc=; b=KwUnd8vMN4NXqvHk7aZDoxn17p9hbDSBWsDqaK1O5mBIwRjLfq9IrHffrxQuLYXLFGz6VfHaKoI0UoHSGDDf0pFKe+GNDBF7CzUCoHBkV3Bc8OpEAZNZ0PUUoa20O5o4iD63yTJ8zjOQkudD4maz+70CjAkWgCa8GoFiLHbI6UU=
Received: from CY4PR21MB0630.namprd21.prod.outlook.com (10.175.115.20) by CY4PR21MB0184.namprd21.prod.outlook.com (10.173.193.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.675.1; Wed, 4 Apr 2018 20:15:40 +0000
Received: from CY4PR21MB0630.namprd21.prod.outlook.com ([fe80::de:ba33:4748:51da]) by CY4PR21MB0630.namprd21.prod.outlook.com ([fe80::de:ba33:4748:51da%6]) with mapi id 15.20.0675.003; Wed, 4 Apr 2018 20:15:40 +0000
From: Praveen Balasubramanian <pravb@microsoft.com>
To: IETF QUIC WG <quic@ietf.org>
Subject: Impact of hardware offloads on network stack performance
Thread-Topic: Impact of hardware offloads on network stack performance
Thread-Index: AdPMUbL7T8nZAW2ySiO289IPTC0ttg==
Date: Wed, 04 Apr 2018 20:15:40 +0000
Message-ID: <CY4PR21MB0630CE4DE6BA4EDF54A1FEB1B6A40@CY4PR21MB0630.namprd21.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [2001:4898:80e8:a::712]
x-ms-publictraffictype: Email
x-microsoft-exchange-diagnostics: 1; CY4PR21MB0184; 7:7NJWJVrqBGr3Lx3lwO3eiOwU1ZNR/KFZZJb7AbZZzUCYGnU5mQmvk4TlVLimA8zhPeMvhH3+KocsnbcOsk2/G088z1YuIF2nQ0Wps3j3N2Wn7Co9YNL0Ug5Pin1MW2zEZhWbuiuBtEpHG3Vl2gGWkyx9D90QpDjSysLRWqvFd8w5dxEInusQeMXWtODe2t7U1m3EEfLDuH8UabVHibZL0IenJrB0eulUBaopbITrLHwG2np0AexqgJ945+kJbvYz; 20:VT0BlZdgWGqe3OJaI8FoaRwavbbIEmFYALr77Y6+4FqahkYvs8hKgRptNBp+3njnfIl5qgP14HoOjtH4e5Uhc+YGfySvL7fuAyLZwFx2qXNi61RdgD36aLygTcHdfKbl++vXm2n9XfN0qpiwGlNYevWTCXmMh7IHdmNn2tFoiBQ=
x-ms-exchange-antispam-srfa-diagnostics: SOS;
x-ms-office365-filtering-correlation-id: 940e2884-9fc8-4afe-6c2b-08d59a68d648
x-ms-office365-filtering-ht: Tenant
x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4604075)(3008032)(48565401081)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7193020); SRVR:CY4PR21MB0184;
x-ms-traffictypediagnostic: CY4PR21MB0184:
authentication-results: spf=none (sender IP is ) smtp.mailfrom=pravb@microsoft.com;
x-microsoft-antispam-prvs: <CY4PR21MB0184B1B2B03C4BBB073D3439B6A40@CY4PR21MB0184.namprd21.prod.outlook.com>
x-exchange-antispam-report-test: UriScan:(28532068793085)(158342451672863)(21748063052155);
x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(61425038)(6040522)(2401047)(8121501046)(5005006)(3231221)(944501327)(52105095)(93006095)(93001095)(10201501046)(3002001)(6055026)(61426038)(61427038)(6041310)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(20161123564045)(20161123558120)(20161123562045)(6072148)(201708071742011); SRVR:CY4PR21MB0184; BCL:0; PCL:0; RULEID:; SRVR:CY4PR21MB0184;
x-forefront-prvs: 0632519F33
x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(346002)(366004)(39860400002)(376002)(396003)(39380400002)(199004)(189003)(99286004)(6916009)(7696005)(476003)(55016002)(54896002)(33656002)(81156014)(8676002)(53936002)(486006)(86612001)(7736002)(5250100002)(81166006)(2906002)(3660700001)(106356001)(68736007)(6116002)(790700001)(46003)(3280700002)(74316002)(6506007)(186003)(105586002)(102836004)(316002)(10090500001)(22452003)(97736004)(8990500004)(6306002)(6436002)(25786009)(8936002)(5660300001)(14454004)(478600001)(10290500003)(2900100001)(9686003)(86362001); DIR:OUT; SFP:1102; SCL:1; SRVR:CY4PR21MB0184; H:CY4PR21MB0630.namprd21.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1;
received-spf: None (protection.outlook.com: microsoft.com does not designate permitted sender hosts)
x-microsoft-antispam-message-info: lRkjMXDx4WzxxvSUUAwJVN96gOQVpz62oB0zZdWCTMw7u0v6Pidw9a6OdZNgQkxLR8aQtQ4xru+BYCnjTCxvZ04gRPHKcBrHRhjAwkXlx2/sNYg12ZgoAVZL0SodDxL/ri3+2U1cgiBA5XVd8Fvg86eI/000+A9FoMedQs3uG+a7FICnzsA8ww2gTE8hm2IsKOT8IF1S10anthsv2jTmM2SWSf4n33GsOxcbhLyCAti0E5OOGHHAXl2YTQtgj/I0iR8v/+nEVJ+u+BoFNLTyxN47TinlG7dOuYgJ81EAo2mYKQAIAhyXDN73D8/spvfh76d7zc3586ONfoiEZlFt+fApaXc/f+jzrad5k2WLXhk3YXl5hb/sCesIlLq9s+Q3Pt7Cvt9Jxk9JoMJBBqUsS48BJlpI5Zim7VDbvKIWYz4=
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative; boundary="_000_CY4PR21MB0630CE4DE6BA4EDF54A1FEB1B6A40CY4PR21MB0630namp_"
MIME-Version: 1.0
X-OriginatorOrg: microsoft.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 940e2884-9fc8-4afe-6c2b-08d59a68d648
X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Apr 2018 20:15:40.3898 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR21MB0184
Archived-At: <https://mailarchive.ietf.org/arch/msg/quic/Lnd7LsKFloqvWTVBD-eNDd8bLNo>
X-BeenThere: quic@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Main mailing list of the IETF QUIC working group <quic.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/quic>, <mailto:quic-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/quic/>
List-Post: <mailto:quic@ietf.org>
List-Help: <mailto:quic-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/quic>, <mailto:quic-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 04 Apr 2018 20:15:43 -0000

Some comparative numbers from an out of box default settings Windows Server 2016 (released version) for a single connection with a microbenchmark tool:

Offloads enabled

TCP gbps

UDP gbps

LSO + LRO + checksum

24

3.6

Checksum only

7.6

3.6

None

5.6

2.3


This is for a fully bottlenecked CPU core -- if you run lower data rates there is still a significant difference in Cycles/byte cost. Same increased CPU cost applies for client systems going over high data rate Wifi and cellular.

This is without any crypto. Once you add crypto the numbers become much worse with crypto cost becoming dominant. Adding another crypto step further exacerbates the problem. Hence crypto offload gains in importance followed by these batch offloads.

If folks need any more numbers I'd be happy to provide them.

Thanks

v2.23.0 | Report a Bug | By Email