Re: Naive question on multiple TCP/IP channels and please dont start a uS NN debate here unless you really want to.

Jim Gettys <> Fri, 06 February 2015 13:47 UTC

Return-Path: <>
Received: from localhost ( []) by (Postfix) with ESMTP id BA0251A010F; Fri, 6 Feb 2015 05:47:31 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.277
X-Spam-Status: No, score=-1.277 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FM_FORGED_GMAIL=0.622, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_PASS=-0.001] autolearn=no
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 2p6y2Cqu1Hla; Fri, 6 Feb 2015 05:47:28 -0800 (PST)
Received: from ( [IPv6:2607:f8b0:4003:c01::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 4141C1A1AAB; Fri, 6 Feb 2015 05:47:28 -0800 (PST)
Received: by with SMTP id va2so13227850obc.6; Fri, 06 Feb 2015 05:47:27 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=Zr/Veo93SHFBF9yTItSvM2eLz1sKgNSktkN8URn7BgY=; b=TOW35JuvrOKHdrqnMK81vOkMn0VoMs9nv5x31K9ZMx/TwKsVZH8+QWBGpPO+CnedyL NHEgaU385SNDnwu1tiWAUTKmkepb5Vll1PmcvJrZY7c8LZfBZ/rQKe++jaO+2gV9GnWn yVGCxLSNt7o8vAjZNyQmbZd1sdhYE+0NqAyzUZ2xGNqckNJAJ3LBT0GsalMi5j1sZzDt t3iaxIuO8Yzms6AMDmSray4FNfiISDP9PFRPNw4WJuyBwp7NQ/viwcfq8gNomP1M4xE8 n2Oyyxveh6uqc+FzwHTsr4U0PbeXTj5flINOGwnVCziBldFsxG796FcJitkZWCT+Zulk bZHQ==
MIME-Version: 1.0
X-Received: by with SMTP id l7mr2539983oep.37.1423230447339; Fri, 06 Feb 2015 05:47:27 -0800 (PST)
Received: by with HTTP; Fri, 6 Feb 2015 05:47:27 -0800 (PST)
In-Reply-To: <>
References: <>
Date: Fri, 6 Feb 2015 08:47:27 -0500
X-Google-Sender-Auth: GIupkUx1d_0kkgRGXHShL1u2OXQ
Message-ID: <>
Subject: Re: Naive question on multiple TCP/IP channels and please dont start a uS NN debate here unless you really want to.
From: Jim Gettys <>
To: Richard Shockey <>
Content-Type: multipart/alternative; boundary=089e0115ffbc3f4fc4050e6ba962
Archived-At: <>
Cc: Phillip Hallam-Baker <>, "" <>, IETF Discussion Mailing List <>
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: IETF-Discussion <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Fri, 06 Feb 2015 13:47:31 -0000

On Thu, Feb 5, 2015 at 6:34 PM, Richard Shockey <> wrote:

> OK yes, My Netflix download is going to kill your VOIP call.

​Yes, it may, though pacing traffic may help (see the sched_fq work in

> RS > Well ..its always been more subtle than you think. You have to
> distinguish between a Voice OVER IP call aka Skype etc vs a “managed
> service” like Voice USING IP or VuIP which is an entirely different beast.
> Modern VuIP which is all of Cable Voice in Europe and the US  VZ FIOS etc
> and VoLTE is managed and  uses IP technologies such as SIP/IMS but the
> routing may or may not have anything to  to do with BGP.  And BTW you have
> to still do the first order number translation as well.  AKA RFC 6116 ENUM
> or something new which we are actually debating in RAI.
> Its segmented managed traffic. Its not Netfilx killing Skype its Microsoft
> Apple Android Updates as well.   We have no visibility on how the OS
> actually queues application packets if at all.

​Yup; this is a problem in the upstream direction (so is not the case you
state above, but the inverse case, typified by events such as some one
emailing a pile of images to your friends/family, backup and similar

One of the most common bottlenecks is the WiFi or cellular hop, and if the
operating system does a single bloated FIFO drop tail queue discipline, you
get into trouble.​

​ On Linux, this queue discipline has historically been one called
PFIFO_FAST, and implemented​ a large (typically 1024 packet) FIFO drop tail
queue (with a little bit of diffserv thrown in for good measure).

​Turning on a different queue discipline​ is a single configuration line,
and it appears that people have been deciding to make fq_codel the default
in various Linux distributions as of last fall (it has been the default in
OpenWrt on routers for a while).

At some point, it may make sense to use a different queue discipline
(sched_fq) as the default, but I think more testing is needed.  That's a
bit of a discussion better left to a different message.

> Can't fix the queuing algorithm for just one interaction...

​There is one important point about fq_codel that needs explanation, and
part of why we call it "flow queuing" rather than "fair queuing".  Here's
my best stab at a simple description of fq_codel:

*Flows are identified. The first packet(s) from new flows are scheduled
before packets from flows that have built a queue. If a flow’s queue
empties, it is eligible to again be considered a new flow. CoDel is applied
to control the queue length in any flow. Result: timely delivery, and
mixing of flows.*

​What effect does this algorithm have in practice? Here are some examples:
   o real time isochronous traffic​ (such as VOIP, skype, etc) won't build
a queue, so will be scheduled in preference to your bulk data.
   o your DNS traffic will be prioritized.
   o your TCP open handshakes will be prioritized
   o your DHCP & RA handshakes will be prioritized
   o your handshakes for TLS will be prioritized
   o any simple request/response protocol with small messages.
  o the first packet or so of a TCP transfer will be prioritized: remember,
that packet may have the size information needed for web page layout in it.
  o There is a *positive* incentive for flows to pace their traffic (i.e.
to be a good network citizen, rather than always transmitting at line rate).

*All without needing any explicit classification.  No identification of
what application is running is being performed at all in this algorithm.*

You can still do explicit classification in addition to fq_codel if you
want (it's need is greatly reduced and most people don't bother except when
at very low bandwidths, such as low speed DSL connections).

We don't care if the audio packets are SIP VOIP, or Skype, or a webrtc
audio stream; anything that is behaves as a network "good citizen" gets
best treatment. A well controlled paced TCP stream will get the same
treatment.  And it reduces latency exactly for the packets that are most
latency sensitive.

> The reason I started pushing on this is that in the wake of Title II I am
> expecting a lot of people to be asking me to explain how the Internet
> worketh and this is precisely the sort of example that shows how (1) things
> are more complex than they appear, (2) how some sort of coordination is
> needed and (3) how the coordination needs to take less than five years to
> come to a decision.
> RS>  You really really don’t want to go into the US NN debate.  Been there
> done that.  The people who would have called you already called someone
> else.  The outcome here in the US after February 26 when the FCC Order
> drops is already predetermined. The Zombies take over (aka the lawyers) it
> goes back to the courts. Its totally partisan. Kabuki Theatre.  Nothing
> happens. But Nothing is perhaps a reasonable outcome here. First do no harm.
​I hope the explanation above helps...
                                 - Jim

> Richard Shockey
> Shockey Consulting LLC
> Chairman of the Board SIP Forum
> richard<at>
> Skype-Linkedin-Facebook rshockey101
> PSTN +1 703-593-2683