Re: Naive question on multiple TCP/IP channels

Jim Gettys <jg@freedesktop.org> Thu, 05 February 2015 16:14 UTC

Return-Path: <gettysjim@gmail.com>
X-Original-To: ietf@ietfa.amsl.com
Delivered-To: ietf@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E814A1A8850 for <ietf@ietfa.amsl.com>; Thu, 5 Feb 2015 08:14:54 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.277
X-Spam-Level:
X-Spam-Status: No, score=-1.277 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FM_FORGED_GMAIL=0.622, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_PASS=-0.001] autolearn=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UnbMWKWjN9xU for <ietf@ietfa.amsl.com>; Thu, 5 Feb 2015 08:14:52 -0800 (PST)
Received: from mail-ob0-x233.google.com (mail-ob0-x233.google.com [IPv6:2607:f8b0:4003:c01::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id D14201A88E2 for <ietf@ietf.org>; Thu, 5 Feb 2015 08:14:49 -0800 (PST)
Received: by mail-ob0-f179.google.com with SMTP id wp4so7836039obc.10 for <ietf@ietf.org>; Thu, 05 Feb 2015 08:14:49 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=Ad/wazAanBte33XH+L8OaE+pxUyygWLpZArWrSGv3cU=; b=DY4ieOeiPliugFv+4dt37pUmj9eNaQH2oTI85zmcpbXdd02bS6YldJW3W+Kc2T0osZ lMbYR5o62IKkhOkAmncYPfyw+4Dv5Z/Du7B9TTgm9F9grVQmRjvC91JqYH19t6sUiEi0 GnugqUC+ReFDZ0TmrIXhQ8agag705pgKyJ/o/ZMMlveKl9Sfnj5pVHLcc34cLilYOCoe GV+gWal2uNUKc9ktrYBntzj+5tfu5w5Ai/Z/5p+F1ysRrrzob6kAKjoNARFM88oZfejK CtRPFsJbLDciyk8nf8GQuzupjue4HIy7Ap1XARBoh4eZIZEVv+/b4pPlqMTCbmiOauzC nWPQ==
MIME-Version: 1.0
X-Received: by 10.182.22.198 with SMTP id g6mr2966945obf.16.1423152889122; Thu, 05 Feb 2015 08:14:49 -0800 (PST)
Sender: gettysjim@gmail.com
Received: by 10.76.76.5 with HTTP; Thu, 5 Feb 2015 08:14:49 -0800 (PST)
In-Reply-To: <CAMm+Lwgb9L9bUG6ommBDYJzQTCU1cC_zLSEf_5JPeJ+c=yrYmA@mail.gmail.com>
References: <CAMm+Lwgb9L9bUG6ommBDYJzQTCU1cC_zLSEf_5JPeJ+c=yrYmA@mail.gmail.com>
Date: Thu, 05 Feb 2015 11:14:49 -0500
X-Google-Sender-Auth: sj0uFYGNW57WiSGGBfZH_iy2lZ8
Message-ID: <CAGhGL2D_GsU29tz6B5HkL_PhO3Vr7_nN64hfW6y014b8xwLVfA@mail.gmail.com>
Subject: Re: Naive question on multiple TCP/IP channels
From: Jim Gettys <jg@freedesktop.org>
To: Phillip Hallam-Baker <phill@hallambaker.com>
Content-Type: multipart/alternative; boundary="001a11c2eb2e6ad6ad050e599afc"
Archived-At: <http://mailarchive.ietf.org/arch/msg/ietf/vhFv-hTkU21J64Y2SnpS5yKC4bE>
Cc: IETF Discussion Mailing List <ietf@ietf.org>
X-BeenThere: ietf@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: IETF-Discussion <ietf.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/ietf>, <mailto:ietf-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/ietf/>
List-Post: <mailto:ietf@ietf.org>
List-Help: <mailto:ietf-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ietf>, <mailto:ietf-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 05 Feb 2015 16:14:55 -0000

On Wed, Feb 4, 2015 at 2:22 PM, Phillip Hallam-Baker <phill@hallambaker.com>
wrote:
> Today most Web browsers attempt to optimize download of images etc. by
> opening multiple TCP/IP streams at the same time. This is actually done
for
> two reasons, first to reduce load times and second to allow the browser to
> optimize page layout by getting image sizes etc up front.
>
> This approach first appeared round about 1994. I am not sure whether
anyone
> actually did a study to see if multiple TCP/IP streams are faster than one
> but the approach has certainly stuck.

They did study it; there was/is a point of diminishing returns...  In that
era, the diminishing returns were in part due to insufficient buffering at
dialup modem banks.

>
> But looking at the problem from the perspective of the network it is
really
> hard to see why setting up five TCP/IP streams between the same endpoints
> should provide any more bandwidth than one. If the narrow waist is
observed,
> then the only parts of the Internet that are taking note of the TCP part
of
> the packet are the end points. So having five streams should not provide
any
> more bandwidth than one unless the bandwidth bottleneck was at one or
other
> endpoint.

Most web objects are small.  In fact, given even IW4, much less IW10, most
objects will fit in the initial window of packets for that connection.

Note, this means that the increase in IW means that there is increasing
head of line blocking occurring at (most) bottleneck links.

Also note that the web is *not* a bulk data transfer system; it is very
latency sensitive.
If you increase the effective RTT, you kill performance.

On (almost all) TCP implementations today, those packets
​from all of those objects ​
will be transmitted at *line rate* in the data center, just to arrive
effectively simultaneously at the bottleneck near the user (typically the
broadband last mile, or in the customer's home router just before the WiFi
hop), where they cause head of line blocking on those devices, as
​almost all
 lack any sort of flow queuing.

So you get a burst of possibly up to hundreds of packets (due to web site
sharding) transmitted at the data center arriving *splat* at the bottleneck
link(s).

I've measured transients of *up to hundreds of milliseconds* on 50Mbps
links.

*So TCP congestion control/avoidance has effectively been defeated
entirely.​  It just isn't in play at all!​*

I call this "transient bufferbloat".  It is the cause of a lot of the
VOIP/teleconferencing drop outs you hear if you surf the web.


>
> Now there are some parts of the deployed Internet that do actually perform
> statefull inspection. But I would expect increasing the number of channels
> to degrade performance at a firewall or any other middle boxen.
>
> So we have a set of behavior that seems at odd with the theory. Has anyone
> done any experiments recently that would show which is right?
>
>
> The reason it makes a difference is that it is becoming clear that modern
> applications are not best served by an application API that is limited to
> one bi-directional stream. There are two possible ways to fix this
> situation. The first is to build something on top of TCP/IP the second is
to
> replace single stream TCP with multi-stream.

You generally find the size information needed for page layout in the first
*packet* of data.  Might be nice if those packets from different flows got
scheduled before the later packets, might it not?  Might be nice if other
traffic, such as other TCP opens, or DNS lookups, or your voip packets got
scheduled for transmission before those additional packets from that bunch
of flows started when you visited that crappy web page with tons of little
objects on it (median is
​of order​
10 objects, btw).

Oh, wait, you say that fq_codel does that?  Well, you might consider using
it.
On
​any​
 current Linux system, that is "one line" of configuration.  And you might
want to replace your home router too, and use firmware on it that
implements fq_codel.... Life gets much better, indeed.
​  And you might want to beat up your ISP, as the bottleneck may be your
 broadband head-end and not under your control.​ And ask Microsoft and
Apple when their OS's will do something sensible too....


https://tools.ietf.org/html/draft-hoeiland-joergensen-aqm-fq-codel-00
                                    - Jim