Re: [rmcat] Packet reordering

[Lennart Schulte <[e-mail]>]

Hi,
while measuring 3G and 4G networks I also saw reordering: 35% of all
connections (12392 in 3G and 676 in 4G) had reordering. In about 2% of
these reordering events the reordering extent was at least 3 packets
large (which caused TCP to spuriously enter fast recovery).

We are currently working on 2 drafts about TCP robustness to reordering,
maybe they are of interest:
- Detection and quantification
  http://tools.ietf.org/html/draft-zimmermann-tcpm-reordering-detection
- Reaction and prevention
  http://tools.ietf.org/html/draft-zimmermann-tcpm-reordering-reaction

Regards,
Lennart


On 05/05/2014 02:17 AM, Keith Winstein wrote:
> Hello Ingemar,
> 
> I'd have to do a more careful (and recent!) measurement to answer you
> clearly, but my recollection is that it was generally not deep enough
> to cause a triple dupack except perhaps very rarely. Generally
> speaking, as I recall the typical behavior was to see groups of 3-4
> MSS-sized TCP segments randomly swizzled.
> 
> My experience matches yours re: commercial service on 1xEV-DO, UMTS
> and LTE networks.
> 
> Best regards,
> Keith
> 
> On Sun, May 4, 2014 at 3:47 PM, Ingemar Johansson S
> <[e-mail]> wrote:
>> Hi
>>
>> Change subject line a bit.
>>
>> Thanks for intersting input Keith. Do you have any data on how deep the repordering is?. Is it deep enough to cause 3-duplicate ACKs in TCP ?, I would belive that people would start to complain about poor TCP performance ?. Admit that I occasionally experince poor TCP performance in my home network but I have sofar blamed it not normal packet losses but perhaps there could be other causes.
>>
>> The precense of reordering in 3GPP networks depend. If we consider just default (best effort) bearers and look at the radio interface only then the answer is that reordering will not happen, the reason is that the default bearers use Acknowledged mode transmission with guaranteed in order delivery to higher layers. There are other e.g QoS bearers that use Unacknowledged mode transmission in which case reordering may occur.
>> Going to the backhaul. As I mentioned in an earlier email, reordering may occur here due to load balancing and I have seen examples of that, however only enough to trigger one dupACK. In teory GTP-U sequence numbers can be enabled and the packet ordering can be restored at the tunnel egress, doubt however that this is used.
>>
>> I don't argue against testing robustness against reordering (within reason) but perhaps it should then be a separate test case if necessary ? Somehow I believe however that reordering is a bigger challenge for audio and video rendering (necessiates a receiver buffer) than for the actual rate adaptation algorithms.
>>
>> /Ingemar
>>
>>> -----Original Message-----
>>> From: Keith Winstein [mailto:[e-mail]]
>>> Sent: den 2 maj 2014 22:09
>>> To: [e-mail]
>>> Subject: Re: [rmcat] rmcat Digest, Vol 21, Issue 2
>>>
>>> Our experience with packet reordering has actually been quite different. We
>>> (Mosh) have found that packet reordering can be extremely common on
>>> 802.11n networks when the "block acks" feature is used and a link is
>>> saturated, at least with some receiving 802.11n chipsets, including the ones in
>>> MacBooks.
>>>
>>> In Mosh version 1.2.2, we turned on an advisory warning when packets were
>>> reordered in the network, because we thought that in practice reordering
>>> was rare enough to be notable and might signify an attempted attack.
>>>
>>> After the release, we got a lot of complaints that the warning was showing
>>> up frequently for users on enterprise 802.11n networks using the MacBook
>>> and a few other devices. On a full-throttle TCP download to a MacBook Pro,
>>> we found that something like 30-50% of all TCP segments were being
>>> delivered out of order! And Mosh's UDP datagrams were seeing similar
>>> treatment when there was a back-to-back flight of them. We ended up
>>> disabling the warning in v1.2.3.
>>>
>>> As I understand, the block acks feature means that a sender may transmit a
>>> whole chunk of frames (some of which will be received without error and
>>> some won't), and then the receiving station can tell the sender which frames
>>> were successfully received and which need to be retransmitted. In the
>>> meantime, the receiver can deliver the successfully-received datagrams to IP
>>> or wait.
>>>
>>> RFC 3366 3.1 essentially advises link designers that they should use a receive-
>>> side reordering buffer to try to put datagrams back in order before delivery
>>> to IP, but our testing suggests that common 802.11n receive chipsets don't
>>> do this in practice.
>>>
>>> So our experience has been that in the contemporary Internet (at least on
>>> some 802.11n networks), reordering can be much more common than we
>>> had initially believed.
>>>
>>> On Fri, May 2, 2014 at 3:00 PM,  <[e-mail]> wrote:
>>>> ---------- Forwarded message ----------
>>>> From: Randell Jesup <[e-mail]>
>>>> To: [e-mail]
>>>> Cc:
>>>> Date: Fri, 02 May 2014 05:49:16 -0400
>>>> Subject: Re: [rmcat] draft-sarker-rmcat-eval-test-00, test case 4.4
>>>> RMCAT Flow competing with a long TCP Flow On 4/30/2014 1:51 PM, Colin
>>> Perkins wrote:
>>>>>
>>>>> On 30 Apr 2014, at 07:50, Zaheduzzaman Sarker
>>> <[e-mail]> wrote:
>>>>>>
>>>>>> Are you saying re-ordering is not a phenomena that we can observe in
>>> the current Internet? And we should not have test cases to handle re-
>>> ordering of packets in RMCAT?
>>>>>
>>>>> One of my students collected some UDP streaming measurements from
>>> our university to hosts on residential networks, covering all the main UK ISPs,
>>> a couple of years ago. With everything using wired connections and UDP we
>>> saw a couple of hundred packets delivered out of order, from ~250 million
>>> packets sent. I can dig into the details of the re-ordering patterns if there’s
>>> interest. We didn’t analyse them too closely at the time because packet loss
>>> and delay spikes were much more common.
>>>>
>>>>
>>>> Having it not fall over on (some) reordering: good.  My experience has
>>> been only specific unusual network/hardware situations cause more than
>>> 0.0x% reordering of packet delivery (at the IP level, ignoring things like TCP
>>> retransmit, which is not an IP-level reordering in the way I'm discussing).
>>>>
>>>> I've seen significant reordering only on a network (an old office network of
>>> mine!) where we had multiple bonded T1's and the packets went across one
>>> of them randomly (I presume).  I think I got to low-single-digit percent or
>>> more likely 0.n% (like 0.1-0.9%) reordered.  There may also be some wireless
>>> networks where this is more likely than on normal wired; I haven't noticed it
>>> on WiFi though.
>>>>
>>>> It's easy to accidentally set up DummyNet/etc to cause massive packet
>>> reordering, however.  I do not think that should be a primary or even
>>> secondary testcase, and (to be honest) I don't think we should care much
>>> about degenerate reordering cases.  I'm not even sure we should bother
>>> spending much energy in our evaluation tests making it happen at all (which
>>> isn't to say that the algorithm authors shouldn't consider it and if they feel it
>>> would affect them, test it or discuss it - plus they will see a small amount of it
>>> on real networks).
>>>>
>>>
>>
>