Re: [87attendees] IETF wireless

[Jim Gettys <[e-mail]>]

On Fri, Aug 9, 2013 at 1:11 PM, joel jaeggli <[e-mail]> wrote:

> On 8/9/13 8:18 AM, Jim Gettys wrote:
> >
> >
> >
> > On Mon, Aug 5, 2013 at 5:48 AM, Tim Chown <[e-mail]
> > <mailto:[e-mail]>> wrote:
> >
> >     Hi,
> >
> >     Others have given the wireless network at this IETF due praise.
> >      Indeed the wireless has been excellent for many IETFs now, such
> >     that we pretty much take it for granted.  Well done to everyone
> >     involved.
> >
> >     Is there any information available on the solution that was
> >     delivered?  From memory the APs were Cisco Aironet, but it would be
> >     interesting to know about density, controllers, etc.  Usage info,
> >     especially for IPv6, would be interesting too.
> >
> >     If the info isn't available, so be it, but it might be of interest
> >     to many...
> >
> >
> > Bufferbloat kills you in many/most conference venues; IETF is one of the
> > few where it does not.
>
> unicast packets are delivered "reliably" on 802.11 networks. really high
> latency on the first hop is generally indicative of high noise
> floor/lots of contention/pdu's getting clobbered. limiting
> retransmission there (equivalent to a shallow buffer) may help you but
> it just pushes the cost of retransmission (due to packet loss) to the
> end-points rather than eliminating the source of restransmission.
>

Actually, there is more to this than first appears; see below.


>
> there are performance tweaks you can make to the RF domain, and the link
> layer that can help you there. dispensing with low speed rates, backward
> compatibility, increasing the multicast rate and so forth. as a
> practical matter if you want to keep the 2.4ghz band usable pushing as
> many clients as possible to 802.11a/n.
>

Yup.  Very important.  A little broadcast/multicast can ruin your whole
day, along with low speed hosts at the edge of an area.  So by turning them
off, you are preserving the performance of the WiFi hop (at the cost of
usable range; a very good tradeoff in a conference venue, but less so
elsewhere).


>
> while it's probably a tautology not having crappy wireless is in fact
> the first step in not having a crappy wireless network.
>

Actually, in Dave Taht and Felix Fietkau digging into the Linux Atheros
driver, there are
a bunch of other issues.  I would expect that commercial AP vendors have
been similarly digging and fixing busted implementations.

If you have a noisy environment, damaged PDU's can easily cause a form of
input bufferbloat.

TCP currently does not like out of order delivery of packets.  So when a
damaged aggregate arrives, the Linux wifi stack does not forward packets
successfully received after the damaged packet immediately. You get a
number of attempts to fix the damaged packet in that transmit opportunity
(Andrew McGregor can explain in detail).  Once the damaged packet is
successfully received, all the queued packets can be released (until the
next damaged packet).

One of the changes Felix and Dave made in the Linux ath9k driver is to
minimize this receive buffering (it was much larger than it needed to be),
but also fix an infinite loop bug (also present in other Linux drivers), so
that a single damaged packet can't cause very long delays and ensure the
damaged packet gets dropped in a somewhat timely fashion.  At one point,
Dave Taht had seen > 1 second delay on simple ping packets.

So it isn't just a crappy wireless network; it's also been crappy
software/firmware.

Ultimately, lots more work needs to be done in the Linux WiFi stack to
rework its queuing entirely; that work is just underway.

I note this because all the bufferbloat issues apply equally to hosts and
home routers (most of which run Linux).  While you may not be able to
control your hosts (unless you are a Linux user), you can do something
about your home router...  And do so immediately, if you want to bother.


> > I first noticed at the Denver NANOG a working
> > wireless network; but the same people are involved in both NANOG and
> > IETF.
>
> the contractor is the same
>
> > So I poked into the topic to understand why. I was amazed to see a
> > room of 500 heavy duty network people and observe latencies from Denver
> > to California consistently under 50ms under load, and investigated why
> > the network was usable.
> >
> > At past IETF's I've urged the operation teams that they should document
> > what they do.  I did not have time in Berlin to repeat the request again.
> >
> > Besides the layout of AP's that other good operators do, there are a
> > number of "tricks" that they play:
> >
> > 1) the AP's have minimal internal buffering, and forward their packets
> > back to a big router.
>
> chris will chime in here I imagine. the APs are cisco 1142s which are
> nice high-end dual radio APs but not in an exceptional sense.
>
> the big router in this case is a gibson arch juniper so it's also not
> exceptional or particularly high capacity, nor is it doing a lot of
> shaping of any variety becuase none of the interfaces are terribly
> heavily utilized.
>
> in both cases overprovisioning works rather well.
>
> > 2) the big router, having lots of aggregated traffic, is able/does run
> > (W)RED to keep the wireless traffic queues from being too much of an
> > issue.  This is hopeless in a home environment where you have little
> > aggregation of traffic (but there, you can now get help from fq_codel if
> > you run OpenWrt/CeroWrt on your home router; but to fix the situation
> > properly will still take a lot of work in Linux's WiFi stack).
> > 3) There is some QOS/Diffserv tricks being played to prioritize certain
> > operations (e.g. DNS lookup, TCP opens, acks, and so on).   An early
> > version of the marking they do was documented at a NANOG meeting; but I
> > gather some later rules have been added since then. Those of you who
> > have looked at fq_codel in detail might recognize that fq_codel happens
> > to have similar behavior (without needing the hair of such explicit
> > marking/classification rules).
>
> Anton Capella was responsible for this in it's initial iterations. It's
> not that uncommon for wireless carriers to employ a similar bag of tricks
>

Yup; I could not remember Anton's name.


>
> afaik we don't do any qos, since we don't have any queues to speak off.
>

IIRC, it was mostly diffserv marking to help certain packet types along.

>
> most traffic in the IETF network is north-south due to the hosts on the
> network having little to do with each other. That's not really the case
> in a homenet. Where some relatively huge flows may be between devices on
> the same wireless lan e.g. timemachine backups occuring while you're
> trying to stream video or maintain the responsiveness of your ssh session.
>

Yup.  Homes are a great place to suffer from bufferbloat.


>
>
> >
> > Anyone setting up an large event today should take a careful look at
> > what is being done, and it would be good to get the knowledge captured
> > in enough detail for others to reproduce their work, as it can be done
> > today.
> >                              - Jim
> >
> >
> >
> >     Tim
> >     _______________________________________________
> >     87attendees mailing list
> >     [e-mail] <mailto:[e-mail]>
> >     https://www.ietf.org/mailman/listinfo/87attendees
> >
> >
> >
> >
> > _______________________________________________
> > 87attendees mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/87attendees
> >
>
>