[tsvwg] On packet reordering in 5G

Ingemar Johansson S <ingemar.s.johansson@ericsson.com> Sat, 03 August 2019 00:06 UTC

Return-Path: <ingemar.s.johansson@ericsson.com>
X-Original-To: tsvwg@ietfa.amsl.com
Delivered-To: tsvwg@ietfa.amsl.com
Received: from localhost (localhost []) by ietfa.amsl.com (Postfix) with ESMTP id DA03012002F for <tsvwg@ietfa.amsl.com>; Fri, 2 Aug 2019 17:06:48 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.001
X-Spam-Status: No, score=-2.001 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=ericsson.com
Received: from mail.ietf.org ([]) by localhost (ietfa.amsl.com []) (amavisd-new, port 10024) with ESMTP id 1zSZPhHh-ZSM for <tsvwg@ietfa.amsl.com>; Fri, 2 Aug 2019 17:06:44 -0700 (PDT)
Received: from EUR04-VI1-obe.outbound.protection.outlook.com (mail-vi1eur04on0631.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe0e::631]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 1EFB5120018 for <tsvwg@ietf.org>; Fri, 2 Aug 2019 17:06:44 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JiDqgZVE2MXHXMoEEmK5dhhnqKAswJL/Pcbb4f14VqZ/9jiJ1EZjoaoDQ/xJiucRc8G3LYqZhkqv5vaF16BXWD7RVsTMR86PzkOBjvLgVdmQ9Yv/DiJ57dtWM89dJit/uUjM8BsGu5nXY4oGbSbygTqkaOGAlNpI7kabY/CuLPy46JSW8ZKcvvzAwvTIeK6lUTTRbpAC50FDCOk4Fjbw5z0wF+OZHZca4O/lqLCHBgMCNlwGM0o6fpHU3UADvBpbC3trdlvcy0I+97ulTVmbsRbdv5Cz7R8ExIcZbscOUQVw80EiLChWrKWYG/0QwHwv8l8AkiPlP8mKXYpeySxS0A==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FJ9rE4jkXVVtIYkOtTKsRktqpaH+pUMe3xT5d52+JSg=; b=DOk5zg4YoXPwhHfcgCwNaEIpKNvkmHVj7hV9mRQKxUEHvfLVti5w2Nb/FUdeobpwbUalNFNorePvmY0KjuyVQ0dFbPRUvNZAtlYaHgcbRm2GJ71Q52Jqt49+TqqU1iHJQ0UnhIomBNkt8EbXhsGOb4oWqbOVo47iaUh+owlc4n9ATRNwkzYY23AhNuR6MLwa68Z50hS9QOIMOGN5PLO8fK7IKgmtt0q8e+FuGXSFNbNoIcVyUpYoddSRDXoHtAjsrHsqjtqPFxgqdMAELx+yGoO4r5ojyx0+8+aMv177/7EWfmMMwpuNZg3d8IN5RWy4t7dHMZGqh6LI853iTSnakg==
ARC-Authentication-Results: i=1; mx.microsoft.com 1;spf=pass smtp.mailfrom=ericsson.com;dmarc=pass action=none header.from=ericsson.com;dkim=pass header.d=ericsson.com;arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FJ9rE4jkXVVtIYkOtTKsRktqpaH+pUMe3xT5d52+JSg=; b=bON1wFuHWsQlQTggHHwle2nIoIC4CxlyJ4PurtK/Qj6w8PyLGKgosM8/MaU61xCIHcdjNyGtH7NRONvtOi6vnpcIsgX5l4hQYhbrg2qLwqKWA82xkfHbE7mXNyMz3gNTad77T29Z9U+i86nrZGHRRZJ/3aSis78Lt4rIqRZb+jc=
Received: from HE1PR07MB4425.eurprd07.prod.outlook.com ( by HE1PR07MB4425.eurprd07.prod.outlook.com ( with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2136.11; Sat, 3 Aug 2019 00:06:41 +0000
Received: from HE1PR07MB4425.eurprd07.prod.outlook.com ([fe80::fcb3:30af:89e8:99c9]) by HE1PR07MB4425.eurprd07.prod.outlook.com ([fe80::fcb3:30af:89e8:99c9%7]) with mapi id 15.20.2136.010; Sat, 3 Aug 2019 00:06:41 +0000
From: Ingemar Johansson S <ingemar.s.johansson@ericsson.com>
To: "tsvwg@ietf.org" <tsvwg@ietf.org>
CC: "Bob Briscoe (ietf@bobbriscoe.net)" <ietf@bobbriscoe.net>, Greg White <g.white@CableLabs.com>, "Black, David" <David.Black@dell.com>, "koen.de_schepper@nokia.com" <koen.de_schepper@nokia.com>, Ingemar Johansson S <ingemar.s.johansson@ericsson.com>
Thread-Topic: On packet reordering in 5G
Thread-Index: AdVJezMWHe80LFBhR0ud3FZKJHKXFQ==
Date: Sat, 3 Aug 2019 00:06:41 +0000
Message-ID: <HE1PR07MB4425D1A25AAD4C4F65D7330AC2D80@HE1PR07MB4425.eurprd07.prod.outlook.com>
Accept-Language: sv-SE, en-US
Content-Language: en-US
X-MS-Has-Attach: yes
authentication-results: spf=none (sender IP is ) smtp.mailfrom=ingemar.s.johansson@ericsson.com;
x-originating-ip: []
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: cd9f9795-9d83-465b-3ee7-08d717a6764b
x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(5600148)(711020)(4605104)(1401327)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(2017052603328)(49563074)(7193020); SRVR:HE1PR07MB4425;
x-ms-traffictypediagnostic: HE1PR07MB4425:
x-microsoft-antispam-prvs: <HE1PR07MB4425B02315FB675EB5FC25B5C2D80@HE1PR07MB4425.eurprd07.prod.outlook.com>
x-ms-oob-tlc-oobclassifiers: OLM:546;
x-forefront-prvs: 0118CD8765
x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(4636009)(396003)(376002)(346002)(39860400002)(366004)(136003)(189003)(199004)(476003)(9686003)(54896002)(2501003)(6306002)(55016002)(2351001)(236005)(107886003)(52536014)(2906002)(8936002)(8676002)(6436002)(1730700003)(81156014)(5640700003)(53936002)(7696005)(66476007)(966005)(66556008)(66616009)(64756008)(66446008)(76116006)(68736007)(33656002)(14454004)(5660300002)(74316002)(81166006)(86362001)(71190400001)(486006)(6916009)(3846002)(790700001)(6116002)(66574012)(99936001)(4326008)(21615005)(99286004)(66946007)(6506007)(71200400001)(186003)(54906003)(66066001)(14444005)(256004)(25786009)(316002)(102836004)(7736002)(26005)(478600001)(606006); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR07MB4425; H:HE1PR07MB4425.eurprd07.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1;
received-spf: None (protection.outlook.com: ericsson.com does not designate permitted sender hosts)
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam-message-info: qtHL8hQACaNj5EiBleOPUMX50+/WBZJKCqRLu1dGX5GBioh6UaBe8vY/yJtKdsS0pLnVyAe9qiw5t6xogOfqwLAtOLDQPPIGGyLLHmzlnkbn04U9de8DWCK6U8sO3IXEG6hDmIH/V98bK84Wmer1ufJS9HMvpxX9BWLyC9UhbXm225A/gGQw8vuQAVDXCNw7IfZcVXUm8Gw7veqhZ+PcQ/U6HXnDrcMeSZm+B6/RL8Kdymf4qanQuETdTuxGPGwGYrMs9hQ73kQwbS7eUBYfIbJCEJkF4Ix9ESuJ0LzVwFsrcdSDoRnCk8M5JfAxNAgkD4APiXYR5oNVGMcpVRpMQ3W2iNHwLkzb2A/1Qb2bqdbSk/kgiuUECkAGvAz0yU94kKvjf6MbsMPPtqlYpKSg+GhN4dUE4oBx8qS2Z+OaT/Q=
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=SHA1; boundary="----=_NextPart_000_0294_01D549A0.10DFB8F0"
MIME-Version: 1.0
X-OriginatorOrg: ericsson.com
X-MS-Exchange-CrossTenant-Network-Message-Id: cd9f9795-9d83-465b-3ee7-08d717a6764b
X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Aug 2019 00:06:41.1078 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 92e84ceb-fbfd-47ab-be52-080c6b87953f
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: ingemar.s.johansson@ericsson.com
X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR07MB4425
Archived-At: <https://mailarchive.ietf.org/arch/msg/tsvwg/_umRW05Ky-afwACEJCItprRicgI>
Subject: [tsvwg] On packet reordering in 5G
X-BeenThere: tsvwg@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Transport Area Working Group <tsvwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tsvwg>, <mailto:tsvwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tsvwg/>
List-Post: <mailto:tsvwg@ietf.org>
List-Help: <mailto:tsvwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tsvwg>, <mailto:tsvwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 03 Aug 2019 00:06:49 -0000



Long post and I probably missed something.. 

I was asked to clarify various aspects of packet reordering in 5G. I mentioned a little on this in  <https://lists.bufferbloat.net/pipermail/bloat/2019-March/008976.html> https://lists.bufferbloat.net/pipermail/bloat/2019-March/008976.html

I’ll try to keep the discussion around default bearers which are the typical bearers allocated for general mobile broadband (MBB) connectivity. Acknowledged mode (AM) transmission is used.

4G networks (and before this 3G) ensured in sequence delivery. The main reason behind this was that the TCP DupACK mechanism was sensitive to packet reordering. At that time it was likely a sensible thing to do as most of the traffic was mainly file transfers (and RACK was not discovered) 

Things have however changed over time.
1) Increased share of interactive services that require low latency. In-sequence delivery leads to HoL blocking issues and that often for no big use as transport protocols are better off to decide how to deal with out of order packets. This applies especially to QUIC, in which case many streams may be transmitted over one connection.

2) Higher link speeds means more memory is required in network nodes for the sole purpose of in-sequence delivery. For downlink traffic this means more memory in UE (User Equipment = USB modems, CPEs and other cellular devices). One can here argue that if in-sequence delivery is not enforced in on the RLC or PDCP layer, then it is likely needed to happen on the transport layer (e.g. TCP). But then again, for some transport protocols like QUIC where multiple streams are transmitted over the same connection, then it is likely better to get the packets delivered out of order.


First a short recap of the 3GPP user plane protocol stack (GTP stuff omitted), it goes like
| IP | PDCP | RLC | MAC | PHY | 


Where can packet reordering occur in 5G?

A.	On the MAC layer: Data from the RLC layer is put on transport blocks. If a transmission fails, a new transmission of the same data is repeated (but with a slightly different error control coding, incremental redundancy is used here to improve the spectrum efficiency. The max number of retransmits can be for instance 4, but this is configurable and vendor specific. Up to 16 HARQ (Hybrid AQR) processes can be run in NR (LTE has a limit of 8), this means that one chunk of data can be successfully transmitted before a previous chunk. This causes packet reordering. In LTE the reordering depth could be 8ms if one retransmission occurred, 16ms in case the same chunk of data is retransmitted twice and so on. In NR this reordering jitter is shorter, both as a result of a faster HARQ processing and shorter transmission intervals. As an example, with the highest numerology (0.125ms TTI=transmission interval) the jitter on the MAC layer is typically less than 1ms.
B.	On the RLC layer: The MAC layer transmission can fail if the max number of retransmission attempts is reached, this can occur if for instance the link adaptation (adjustment of the modulation and coding) is not optimal, in this case it is up to the RLC layer to retransmit the missing data. 
C.	Dual Connectivity: There can be cases where LTE is used for wide area coverage and NR is used for small cell coverage. One example is LTE running in the 2.4MHz band with 20MHz bandwidth and NR running on the 28GHz band with a 100MHz bandwidth. With Dual Connectivity, the data can be transmitted over both legs. Flow control is used to portion the data between the two legs. Depending on the efficiency of the  flow control and how the link bandwidth over the two legs vary, there will be packet reordering to a smaller or larger degree. I cannot give any example of typical reordering depths here as this reordering depends on the flow control algorithms in use.


In LTE it was up to the RLC layer to ensure in-sequence delivery up to higher layers. In 5G, this role is moved up to the PDCP layer, one reason is to make Dual Connectivity easier to implement. In addition, it is optional to implement in-sequence delivery. It is currently not possible for me to say if in-sequence delivery is implemented as it is vendor specific.


For the interested, this book gives more insight to the NR standard. 

 <https://www.elsevier.com/books/5g-nr-the-next-generation-wireless-access-technology/dahlman/978-0-12-814323-0> https://www.elsevier.com/books/5g-nr-the-next-generation-wireless-access-technology/dahlman/978-0-12-814323-0

OK, looks like shameless marketing but I can assure you that the authors (Ericsson employees) will pay me nothing for this ☹. Still the book is quite comprehensible and explains many of the design considerations.


Hope that this sheds at least some light



Ingemar Johansson  M.Sc. 

Master Researcher


Ericsson Research

Network Protocols & E2E Performance

Labratoriegränd 11

971 28, Luleå, Sweden

Phone +46-1071 43042

SMS/MMS +46-73 078 3289




            You’ve got to live with 

         What you can’t rise above
                Bruce Springsteen