Re: [6lo-fragmentation-dt] Performance report for fragment forwarding

"Pascal Thubert (pthubert)" <> Thu, 20 September 2018 13:00 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id D5C88130F35 for <>; Thu, 20 Sep 2018 06:00:34 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -14.51
X-Spam-Status: No, score=-14.51 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-5, SPF_PASS=-0.001, T_DKIMWL_WL_HIGH=-0.01, USER_IN_DEF_DKIM_WL=-7.5] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (1024-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id KJriLL3p33Eg for <>; Thu, 20 Sep 2018 06:00:30 -0700 (PDT)
Received: from ( []) (using TLSv1.2 with cipher DHE-RSA-SEED-SHA (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 7590F130F36 for <>; Thu, 20 Sep 2018 06:00:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple;;; l=45924; q=dns/txt; s=iport; t=1537448430; x=1538658030; h=from:to:cc:subject:date:message-id:references: in-reply-to:mime-version; bh=0TsFZJBHejTmUZvN8UtjOv/rVaF2z7Iqx9eUuRYYfDI=; b=dyDOGelxUwuQC4Py4I9n5gCjVcs4XiP2kWZVgsZsXr/11Y4PHR2nzXEI ioqa0fbB3Dt52I6c6oFNf8yAZx0GH1EJ7O0Xuu/xvjh2kzrvF2von/nig W0poOvKPqqhWgmghdjO4Un6GCKTCDW2yTXxZpjjGrzCxu8FXoi3YOh5LX 0=;
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: =?us-ascii?q?A0AlAAA6maNb/5FdJa1bGQEBAQEBAQE?= =?us-ascii?q?BAQEBAQcBAQEBAYFRgRBIBSplfygKg2mURYINiF6NcIF6C4RsAheDKSE1FwE?= =?us-ascii?q?DAQECAQECbSiFOAEBAQECASMKOAISBQsCAQYCEQQBASEBBgMCAgIfERQJCAI?= =?us-ascii?q?EDgUIgxkBgR1MAw0IiC+bTYEuhzsNgk+KbxeBQT+BEQGCFH6BJoEwggEBBgE?= =?us-ascii?q?BRBCCS4JXAogihUCOYCwJAolHg0SDDx+BRIRQiQ6MX4dvAhEUgSUeATaBVXA?= =?us-ascii?q?VgyeDNwEHjRVviygBAQ0XB4EBgR4BAQ?=
X-IronPort-AV: E=Sophos;i="5.53,398,1531785600"; d="scan'208,217";a="173682271"
Received: from ([]) by with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 20 Sep 2018 13:00:29 +0000
Received: from ( []) by (8.15.2/8.15.2) with ESMTPS id w8KD0TLn023397 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=FAIL); Thu, 20 Sep 2018 13:00:29 GMT
Received: from ( by ( with Microsoft SMTP Server (TLS) id 15.0.1395.4; Thu, 20 Sep 2018 08:00:28 -0500
Received: from ([]) by ([]) with mapi id 15.00.1395.000; Thu, 20 Sep 2018 08:00:28 -0500
From: "Pascal Thubert (pthubert)" <>
To: Rahul Jadhav <>
CC: Carsten Bormann <>, "" <>, "" <>, "" <>
Thread-Topic: [6lo-fragmentation-dt] Performance report for fragment forwarding
Thread-Index: AQHUT2lvdrEYlBd9b0+UYF3V1708sqT2yRmAgACNbQCAAAhwAIAAEbcA///l//CAAgl/AP//sWrAgABjrAD//6xq4A==
Date: Thu, 20 Sep 2018 12:59:59 +0000
Deferred-Delivery: Thu, 20 Sep 2018 12:59:54 +0000
Message-ID: <>
References: <> <> <> <> <> <> <> <> <>
In-Reply-To: <>
Accept-Language: fr-FR, en-US
Content-Language: en-US
x-ms-exchange-transport-fromentityheader: Hosted
x-originating-ip: []
Content-Type: multipart/alternative; boundary="_000_82ed39709c3e421694f4ec241a83ca5cXCHRCD001ciscocom_"
MIME-Version: 1.0
Archived-At: <>
Subject: Re: [6lo-fragmentation-dt] Performance report for fragment forwarding
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: 6lo Fragmentation Design Team <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Thu, 20 Sep 2018 13:00:43 -0000

Hello Rahul

From: Rahul Jadhav <>;
Sent: jeudi 20 septembre 2018 14:48
To: Pascal Thubert (pthubert) <>;
Cc: Carsten Bormann <>;;;;
Subject: Re: [6lo-fragmentation-dt] Performance report for fragment forwarding

Please find my comments inline..

All in all, some of the concerns mentioned in this mail thread should be reflected in the draft as well. A naive implementation or implementers without any background on mac layer may seriously goof up the performance if they simply go by the draft currently.
[PT>] makes sense. Note that a burst of IP packets for a same destination have the same issue. The problem has to do with transmitting frames at L2 not that they are individual packets or multiple fragments at upper layer. If the upper layer makes 5 IP packets to avoid fragmentation of a larger packet in 5 fragments, and sends the 5 IP packets together, you’ll get the exact same result as if using fragments – unless I miss something always interesting to dig deeper..

6TiSCH protects against fragments interfering with one another, but it does not protect against a Hop1 sending frag N+1 to Hop2 while Hop2 is sending frag N to Hop3. Even 6TiSCH needs to pace its traffic to be forwarded by a same next hop, and that’s true for all packets not just fragments. Do you have any form of listen before talk in your simulation?

[RJ] Yes, there is carrier sensing before transmission in my simulation.
[PT>] Then other strategies are possible, like a longer CCA for the first sender.

If you do not delay the transmissions of intermediate fragments, your test certainly proves that the system cannot work satisfactorily. This much was predictable on paper. Say it take 10ms for a transmit + ack, and most transmissions succeed within 3-4 tries, it makes sense to do your testing injecting 50 to 100ms between fragment transmissions. Every node along the path must validate that fragments are thus spaced. Could you please relaunch your test with a rule like that?

[RJ] Yes we ll try this and get the data again. Carsten also suggested the same before in this mail chain.
[PT>] very cool, looking forward to seeing the change. Great research!

For the rest please see below:

If you could instrument the intermediate nodes, I’d like to see if for reason of retries a fragment may catch up with another one. This can be tested by making a hop in the middle very far and unreliable.

[RJ] Sorry but I m not sure what you mean here.

[PT>] The idea is to make a link less reliable then others in order to cause retries so a virtually slower link. You’ll see that packets pile up there even if paced at the source. That’s what I mean by other packets catching up.

[RJ] Ok got it. Thanks

Also, even if you pace at the source by inserting gaps, fragments may still pile up at the ingress of the slowest link on the way and your initial problem will be recreated there. This is only visible if you effectively have a slower hop, but is a very classical effect in window based networks, the full window ends up piling at ingress of the slowest link. You may recreate that by injecting additional traffic via a node on your path.

[RJ] Agreed that a bottleneck link may still create a problem. But if you see the network in our report, there is no such bottleneck node/link ... the paths are sufficiently balanced.

[PT>] So the space must be inserted at the bottleneck too. In theory, a transmitter needs to ensure that every packet to a same next hop is sufficiently spaced with the next packet, unless the next hop is the destination, so the next hop can forward.
To avoid that you would need to pace at the source to a rate that is slower what the slowest hop would do if it was the source. Really hard to do. So ideally, each node would ensure that there is a minimum gap between fragments, which should be no-op everywhere along a path but at the slowest hop where fragments will pile up.

Finally: you can limit the amount of queueing at the slowest hop if you use the recoverable frags draft, using windows of a few fragments for flow control.
[RJ] Amount of queueing can be limited in our case by using L2 feedback as well i.e. if the fragment is lost (and the sender knows this based on L2 feedback) then we can as well stop transmitting all subsequent fragments. For 802.15.4 the recovery procedure can be applied based on L2 feedback without any explicit ACK.
But right now we arent using any of these advanced techniques.

Our aim for this experiment was to check how much of an impact was it when fragment forwarding was used as mentioned in the VRB draft in single channel mode of operation.

[PT>] Without any pacing, the simulation is not so realistic and though interesting, conclusions there would be premature. I’d be very interested in seeing more tests varying the inter frame space and see how that impacts the delivery.

[RJ] Yes we ll try this.

Bottom line: the more you test with one frequency only the more you’ll find it’s a bad idea : )
[RJ] Yes we have realized (it the hard way) that using fragment forwarding in single channel mode is a very bad idea.
[PT>] Note that the same would happen with a series of packets in a row. I do not think that the fact these are fragments have much to do with it.

[RJ] With series of packets, do you mean per-hop reassembly? Well, per-hop reassembly does not suffer from this issue. But it suffers from buffering issue. So if the traffic is sufficiently sparse then using per-hop reassembly in this scenario makes more sense.
[PT>] No just different IP packet same destinations to be sent in a row, no fragmenting. Unless you do ECMP, they will be routed along a same path and have the same fate as fragments. This is why my expectation is that the the problem you point out is not limited to fragments.

 [PT>] : )

Take care,


All the best;


From: 6lo-fragmentation-dt <<>> On Behalf Of Rahul Jadhav
Sent: mercredi 19 septembre 2018 07:59
To: Carsten Bormann <<>>
Subject: Re: [6lo-fragmentation-dt] Performance report for fragment forwarding

Doing pacing on origin-node only should be easy to change/hack and might improve the forwarding efficiency. But this may also shoot up the mean latency.
We ll try this anways (add randomized delay between 10ms to 500ms) to check the impact on PDR. Thanks for this suggestion.

On Wed, 19 Sep 2018 at 10:25, Carsten Bormann <<>> wrote:
Hi Rahul,

> Carsten also mentioned a pacing mechanism .. while it might improve fwding efficiency it will add to buffer requirement. Also such a scheme might be non-trivial to be implemented.

Pacing is something the origin of the datagram does; I wasn’t suggesting the forwarders to add that (which indeed essentially puts the datagram into buffers there).
Clearly, pacing is counterproductive when forwarders do full reassembly, as it increases the time the partially assembled datagram is vulnerable to fragments of other datagrams hijacking the buffer — you want to capture the channel as much as possible then (to the level that you’d want to send following fragments with a higher MAC priority).  Of course, as the datagram travels through the network, it will become more and more smeared over time.

It would be interesting to see how your numbers change with some pacing at the origin; it should be easy to hack that in if you are not trying to do this in a general way.

> Regarding keeping higher mac-retry, ... We have chosen mac-retry of 3 after some experimentation (considering expected node densities and tx frequency). increasing mac-retry might not necessarily help, in fact it may backfire in terms of both PDR as well as mean latency. Would you still suggest to give it a try and what mac-retry do you think makes sense ?

Yeah, yet another tunable that in a robust protocol really should be self-tuning.
A MAC-layer retransmission is not useful if the layers above have already timed out.
So the latency introduced by the backoff mechanism is really the determining factor here.

Grüße, Carsten