Re: [87attendees] IETF wireless

[Jim Gettys <[e-mail]>]

On Fri, Aug 9, 2013 at 12:37 PM, Chris Elliott <[e-mail]> wrote:

> On Friday, August 9, 2013, Jim Gettys wrote:
>>
>> On Mon, Aug 5, 2013 at 5:48 AM, Tim Chown <[e-mail]> wrote:
>>
>>> Hi,
>>
>> ...
>
>> If the info isn't available, so be it, but it might be of interest to
>>> many...
>>>
>>
> We're in active discussions about documenting this, especially why we make
> certain decisions in light of the rather special conditions of the IETF
> meetings: relatively high network usage per attendee vs. many other
> meetings like IEEE and access to donated high speed internet connectivity,
> being just two of the parameters.
>
> Also, site selection is a huge part--the network team is part of the site
> evaluation and the technical recommendations on the suitability of venues
> to host the network you all know and love is taken very seriously, and has
> nixed many venues.
>
> Bufferbloat kills you in many/most conference venues; IETF is one of the
>> few where it does not.   I first noticed at the Denver NANOG a working
>> wireless network; but the same people are involved in both NANOG and IETF.
>>
>
> I haven't been involved with the NANOG network on years. My understanding
> is that our philosophy and design differ fundamentally. We over-provision,
> they tune, is an over-simplification...
>
>
>> So I poked into the topic to understand why. I was amazed to see a room
>> of 500 heavy duty network people and observe latencies from Denver to
>> California consistently under 50ms under load, and investigated why the
>> network was usable.
>>
>> At past IETF's I've urged the operation teams that they should document
>> what they do.  I did not have time in Berlin to repeat the request again.
>>
>
> Part of it is, we're volunteers. We love doing it. And many of us are
> quite active in standards, not just the network. At the end of two intense
> weeks, we're ready to get back to our "real" lives.
>
> And, as noted above, we hear you, and are working on documenting things.
>
>
>> Besides the layout of AP's that other good operators do, there are a
>> number of "tricks" that they play:
>>
>> 1) the AP's have minimal internal buffering, and forward their packets
>> back to a big router.
>>
>
> Agreed.
>
>
>>  2) the big router, having lots of aggregated traffic, is able/does run
>> (W)RED to keep the wireless traffic queues from being too much of an issue.
>>  This is hopeless in a home environment where you have little aggregation
>> of traffic (but there, you can now get help from fq_codel if you run
>> OpenWrt/CeroWrt on your home router; but to fix the situation properly will
>> still take a lot of work in Linux's WiFi stack).
>>
>
> I agree that bufferbloat is a huge issue in many environments. Addressing
> it fixed my son's company's network, with >20 people sharing a 35M down/5M
> up link, and is just one datum in that story.
>
> However, at the IETF, we have the luxury of "simply" over provisioning and
> relying on the enterprise routers to do the right thing without much tuning
> at all. Most of our traffic transits the Internet. The internal traffic is
> small and not an issue. I address wireless at the end briefly...
>
> Our traffic peaks on the external links hit a max of 200Mbps this meeting.
> The highest we've seen was in Hiroshima, where WIDE did the network and we
> basically were advisors without much to do, hit 300Mbps.
>

> Now, consider that we had two 1Gbps connections to a high-tier service
> provider (DT, in this case) with very low latency, and DT's connectivity to
> the rest of the Internet is very good.
>

> To way over-simplify, and maybe trolling a bit, who needs queueing or
> QoS when any buffers you have are empty? 1500 byte packets just don't take
> enough time to transmit on a 1G link to affect other traffic, including,
> for example, small voice packets, especially when they're only carrying
> 200Mbps peak.
>

I was only referring to the AP/router hop: beyond the IETF router,
classification is pretty much irrelevant given overprovisioned uplinks:
your bottlenecks will typically be in the WiFi hop, not in the back haul.

Remember, a single modern TCP stack and only a few wireless sessions can
fill even a 200-300Mbps pipe.  This wasn't true in the days of Windows XP
(which did not have TCP window scaling turned on, and crawled).

Over provisioning really just cuts the effective size (in time) of buffers
that you have.  Filling any buffers at a bottleneck is what TCP loves to
do. So it's easy to saturate the 802.11 hop, and on the IETF network, I
betcha it's often there, not in the backhaul.

Again, the AP's you are using don't sport giant buffers.

So it can/does really matter to run AQM (on the wireless links), even so
(when people download big files).  And if you aren't running any these
days, you are winning due to the lack of buffering in the AP's.


> And we often don't try to share the load across our two links--it's
> reliability we're after, not more bandwidth.
>
>  3) There is some QOS/Diffserv tricks being played to prioritize certain
>> operations (e.g. DNS lookup, TCP opens, acks, and so on).   An early
>> version of the marking they do was documented at a NANOG meeting; but I
>> gather some later rules have been added since then. Those of you who have
>> looked at fq_codel in detail might recognize that fq_codel happens to have
>> similar behavior (without needing the hair of such explicit
>> marking/classification rules).
>>
>
> Note that we don't do any QoS/Diffserv outside of what would have to be
> explicitly turned off in the gear we use.
>

At one point, I tracked down the rules/marking in use on the routers/AP's
when I looked into it.  It was all sensible stuff.


> Once again, I'm not dismissing bufferbloat as a real issue, just that I
> don't need to address it in the IETF network.
>

And the AP's aren't horribly over-buffered in the first place.

It's people who set up conference networks using consumer devices that
really get bit hard by bufferbloat (or many cheap motels that do likewise).
 I've seen this on many, many occasions.


>
>
>> Anyone setting up an large event today should take a careful look at what
>> is being done, and it would be good to get the knowledge captured in enough
>> detail for others to reproduce their work, as it can be done today.
>>
>
> Agreed.
>
> I think a large part of our success has been from how we build the
> wireless side of things. This is almost always the only constrained part of
> the net. More details later, but we've actively worked to encourage folks
> to use the 5Ghz spectrum, and we build the wireless net to maximize the
> available bandwidth while managing a low noise level.
>
> And, all equipment, from the Juniper and Cisco gear and other
> enterprise/carrier grade equipment, to the wireless drivers in everyone's
> client devices, have improved markedly over the past few years. I
> especially see the improvement in wireless.
>
> Wireless clients rarely act like the burro between two (or 100...) bales
> of hay
> these days. Clients used to roam frequently to a slightly "better" AP
> (BSSID). Now they stick much better, which leads to other, much easier to
> deal with, issues.
>
> I heard a story at this meeting that one of our own directly affected this
> (years ago) by simply moving around in front of the desk of a developer as they were tuning
> their code, and wondering why performance sucked and their code kept
> triggering roaming...they got the point, and fixed the code.
>

Yeah, and bufferbloat encourages this behavior: when your throughput drops
(either due to physical environment, or competing traffic), your latency
can go through the roof if the network (and host!) has lots of unmanaged
buffers, and you either die a slow death or move to a different ESSID. And
IETF wireless network doesn't have a pile of unmanged buffers, so you win
(in the downstream).  Saturating the wireless link in the upstream
direction you'll still have problems in your devices; bloat is in our
operating system stacks and device drivers...


> Thanks for the kind words about the net, and your continued encouragement
> for us to document what we do.
>

It's excellent!  Keep up the great work.

Jim


> Chris.
>
>
>>                               - Jim
>>
>>
>>
>>> Tim
>>> _______________________________________________
>>> 87attendees mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/87attendees
>>>
>>
>>
>
> --
> Chris Elliott
> [e-mail]
>
>