Re: [tsvwg] On packet reordering in 5G

Bob Briscoe <ietf@bobbriscoe.net> Sat, 03 August 2019 13:20 UTC

Return-Path: <ietf@bobbriscoe.net>
X-Original-To: tsvwg@ietfa.amsl.com
Delivered-To: tsvwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id A2E1A1201EC for <tsvwg@ietfa.amsl.com>; Sat, 3 Aug 2019 06:20:49 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.998
X-Spam-Level:
X-Spam-Status: No, score=-1.998 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=bobbriscoe.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FORYL0ZvFJ9W for <tsvwg@ietfa.amsl.com>; Sat, 3 Aug 2019 06:20:46 -0700 (PDT)
Received: from server.dnsblock1.com (server.dnsblock1.com [85.13.236.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 6C87812003F for <tsvwg@ietf.org>; Sat, 3 Aug 2019 06:20:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bobbriscoe.net; s=default; h=Content-Type:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=SGBrD93SgqqxRg1y7/HedKjzBcyvR5Zq5jugqv9pDrs=; b=YdC9xlsNdWqfc6wqgVqfjev8k f9WqyduuWQ1eHMgvCxG1VOzDvXlrDgxuyvoAHLyX5Hf0HoL0RttGHI5QuD1G9HILl8DY91pt58XmJ iNODJr5+foFDBVxdgJDCI1Gb9mQrO57qRQOMC2YE4cTKPQP6zO7ZAImCeZgtNMuQ+Y0zajpMRfLL8 ld1jRuEs5+rgZSO0pJvE39pfD3pHUZzFHk8MXSVKvRMRoO7xo+IlvKv1ZnSv52D43gBvOkV2MuWdV 1IEJU8iu7Q/jrJiVludpIHZMMY2AvvEKuFLCjbVxGwkeW9ORxbMMRHmgyDR2+La0zXqZQVBu5Ym5A To8qL0WUQ==;
Received: from [31.185.128.31] (port=56902 helo=[192.168.0.6]) by server.dnsblock1.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.92) (envelope-from <ietf@bobbriscoe.net>) id 1httxh-0003VU-C2; Sat, 03 Aug 2019 14:20:41 +0100
To: Ingemar Johansson S <ingemar.s.johansson@ericsson.com>, "tsvwg@ietf.org" <tsvwg@ietf.org>
Cc: Greg White <g.white@CableLabs.com>, "Black, David" <David.Black@dell.com>, "koen.de_schepper@nokia.com" <koen.de_schepper@nokia.com>
References: <HE1PR07MB4425D1A25AAD4C4F65D7330AC2D80@HE1PR07MB4425.eurprd07.prod.outlook.com>
From: Bob Briscoe <ietf@bobbriscoe.net>
Message-ID: <e28c7b6f-8bfd-8f86-4724-85f3b98c3536@bobbriscoe.net>
Date: Sat, 3 Aug 2019 14:20:40 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1
MIME-Version: 1.0
In-Reply-To: <HE1PR07MB4425D1A25AAD4C4F65D7330AC2D80@HE1PR07MB4425.eurprd07.prod.outlook.com>
Content-Type: multipart/alternative; boundary="------------646AE4F1C7DF97DB8A28F647"
Content-Language: en-GB
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - server.dnsblock1.com
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - bobbriscoe.net
X-Get-Message-Sender-Via: server.dnsblock1.com: authenticated_id: in@bobbriscoe.net
X-Authenticated-Sender: server.dnsblock1.com: in@bobbriscoe.net
Archived-At: <https://mailarchive.ietf.org/arch/msg/tsvwg/LgWxXL-h9jhCxoPw722ejbTlFv0>
Subject: Re: [tsvwg] On packet reordering in 5G
X-BeenThere: tsvwg@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Transport Area Working Group <tsvwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tsvwg>, <mailto:tsvwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tsvwg/>
List-Post: <mailto:tsvwg@ietf.org>
List-Help: <mailto:tsvwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tsvwg>, <mailto:tsvwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 03 Aug 2019 13:20:50 -0000

Ingemar,

Thanks muchly for this comprehensive explanation of the state of the art.
I have posed a question for any folks from mobile operators watching 
this thread - inline.

On 03/08/2019 01:06, Ingemar Johansson S wrote:
>
> Hi
>
> Long post and I probably missed something..
>
> I was asked to clarify various aspects of packet reordering in 5G. I 
> mentioned a little on this in 
> https://lists.bufferbloat.net/pipermail/bloat/2019-March/008976.html
>
> I’ll try to keep the discussion around default bearers which are the 
> typical bearers allocated for general mobile broadband (MBB) 
> connectivity. Acknowledged mode (AM) transmission is used.
>
> 4G networks (and before this 3G) ensured in sequence delivery. The 
> main reason behind this was that the TCP DupACK mechanism was 
> sensitive to packet reordering. At that time it was likely a sensible 
> thing to do as most of the traffic was mainly file transfers (and RACK 
> was not discovered)
>
> Things have however changed over time.
> 1) Increased share of interactive services that require low latency. 
> In-sequence delivery leads to HoL blocking issues and that often for 
> no big use as transport protocols are better off to decide how to deal 
> with out of order packets. This applies especially to QUIC, in which 
> case many streams may be transmitted over one connection.
>
> 2) Higher link speeds means more memory is required in network nodes 
> for the sole purpose of in-sequence delivery. For downlink traffic 
> this means more memory in UE (User Equipment = USB modems, CPEs and 
> other cellular devices). One can here argue that if in-sequence 
> delivery is not enforced in on the RLC or PDCP layer, then it is 
> likely needed to happen on the transport layer (e.g. TCP). But then 
> again, for some transport protocols like QUIC where multiple streams 
> are transmitted over the same connection, then it is likely better to 
> get the packets delivered out of order.
>
> First a short recap of the 3GPP user plane protocol stack (GTP stuff 
> omitted), it goes like
> | IP | PDCP | RLC | MAC | PHY |
>
> Where can packet reordering occur in 5G?
>
>  1. On the MAC layer: Data from the RLC layer is put on transport
>     blocks. If a transmission fails, a new transmission of the same
>     data is repeated (but with a slightly different error control
>     coding, incremental redundancy is used here to improve the
>     spectrum efficiency. The max number of retransmits can be for
>     instance 4, but this is configurable and vendor specific. Up to 16
>     HARQ (Hybrid AQR) processes can be run in NR (LTE has a limit of
>     8), this means that one chunk of data can be successfully
>     transmitted before a previous chunk. This causes packet
>     reordering. In LTE the reordering depth could be 8ms if one
>     retransmission occurred, 16ms in case the same chunk of data is
>     retransmitted twice and so on. In NR this reordering jitter is
>     shorter, both as a result of a faster HARQ processing and shorter
>     transmission intervals. As an example, with the highest numerology
>     (0.125ms TTI=transmission interval) the jitter on the MAC layer is
>     typically less than 1ms.
>  2. On the RLC layer: The MAC layer transmission can fail if the max
>     number of retransmission attempts is reached, this can occur if
>     for instance the link adaptation (adjustment of the modulation and
>     coding) is not optimal, in this case it is up to the RLC layer to
>     retransmit the missing data.
>  3. Dual Connectivity: There can be cases where LTE is used for wide
>     area coverage and NR is used for small cell coverage. One example
>     is LTE running in the 2.4MHz band with 20MHz bandwidth and NR
>     running on the 28GHz band with a 100MHz bandwidth. With Dual
>     Connectivity, the data can be transmitted over both legs. Flow
>     control is used to portion the data between the two legs.
>     Depending on the efficiency of the  flow control and how the link
>     bandwidth over the two legs vary, there will be packet reordering
>     to a smaller or larger degree. I cannot give any example of
>     typical reordering depths here as this reordering depends on the
>     flow control algorithms in use.
>
> In LTE it was up to the RLC layer to ensure in-sequence delivery up to 
> higher layers. In 5G, this role is moved up to the PDCP layer, one 
> reason is to make Dual Connectivity easier to implement.
>
And this makes more sense once the RLC frames map 1:1 to IP packets in 5G.

> In addition, it is optional to implement in-sequence delivery. It is 
> currently not possible for me to say if in-sequence delivery is 
> implemented as it is vendor specific.
>
Assuming equipment vendors implement in-sequence delivery, the real 
question is whether mobile operators will still want it enabled on the 
default (MBB) bearer. Does anyone (e.g. from an operator) have a feel 
for an answer?

I assume that operators will keep PDCP in-sequence delivery enabled 
until the prevalence of modern re-ordering tolerant transports combined 
with the benefits to them outweigh the prevalence and detriments to 
conventional (3DupACK) transports.

This messy compromise is why we took the opportunity to draw a line in 
the sand with L4S, by requiring sender algorithms that have to be 
modified to support L4S to also have modern reordering tolerance from day-1.

I know there are objections on editorial, architectural and procedural 
grounds, so I shall be working through that minefield next week, because 
I've seen a lot of support on technical and pragmatic grounds - the two 
types are hard to weigh against each other.


> For the interested, this book gives more insight to the NR standard.
>
> https://www.elsevier.com/books/5g-nr-the-next-generation-wireless-access-technology/dahlman/978-0-12-814323-0
>
> OK, looks like shameless marketing but I can assure you that the 
> authors (Ericsson employees) will pay me nothing for this ☹. Still the 
> book is quite comprehensible and explains many of the design 
> considerations.
>
> Hope that this sheds at least some light
>
Indeed it does. Thank you again.



Bob


> /Ingemar
>
> ================================
>
> Ingemar Johansson  M.Sc.
>
> Master Researcher
>
> Ericsson Research
>
> Network Protocols & E2E Performance
>
> Labratoriegränd 11
>
> 971 28, Luleå, Sweden
>
> Phone +46-1071 43042
>
> SMS/MMS +46-73 078 3289
>
> ingemar.s.johansson@ericsson.com
>
> www.ericsson.com
>
> You’ve got to live with
>
>          What you can’t rise above
>                 Bruce Springsteen
>
> =================================
>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/