[tsvwg] Bi-directional asymmetric link test results, and Prague self-starvation

Pete Heist <pete@heistp.net> Sat, 09 May 2020 18:30 UTC

Return-Path: <pete@heistp.net>
X-Original-To: tsvwg@ietfa.amsl.com
Delivered-To: tsvwg@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 971E23A100E for <tsvwg@ietfa.amsl.com>; Sat, 9 May 2020 11:30:06 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.099
X-Spam-Level:
X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=heistp.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Hn0MBXy4M2YI for <tsvwg@ietfa.amsl.com>; Sat, 9 May 2020 11:30:03 -0700 (PDT)
Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 51E313A0CDC for <tsvwg@ietf.org>; Sat, 9 May 2020 11:29:54 -0700 (PDT)
Received: by mail-wr1-x430.google.com with SMTP id x17so5787300wrt.5 for <tsvwg@ietf.org>; Sat, 09 May 2020 11:29:54 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heistp.net; s=google; h=from:content-transfer-encoding:mime-version:subject:message-id:date :to; bh=otbyzWGDzK8pXOj/KSkIfguPlCgFLyTXhSYLLLUmnCY=; b=FM7krkerGzcvZ7NE2MHyU3NZm0El4GLwmP0VHvjMXCpH61lpAOII0M5/a43gn5m70y g4d2l/jL5AnELZt9QJ9/q3+lF2ygq2KArLeDfSBG8owRNDximYFTziH7NdDlLYp/GdK/ DiHjuuPHW/WwQI39ySY2DVchtr07jLbzq6hu+XNCnfbd6GlKs0v9Wa3zeZWQUKkWbZrk qPM6++9aZeNOBGILuaiVeEBwSR4+IYH1AsD1SrTj0vusMXyIStAMJgwmoOIKNSqlExgJ z7BkRysTkhtuMd0I76EM/Pvz7hz7ZTTH4l2yNZXMkFMvYpJMY24ZXN9Wr3qdKMM4Xq7l S1Qg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:message-id:date:to; bh=otbyzWGDzK8pXOj/KSkIfguPlCgFLyTXhSYLLLUmnCY=; b=RNQ8uOuKUNWr+yH7Mw7YocZ6PzQnteBTizwZozRGXkwTzEq3lmxxcN3dnDtZa6SBRZ tOG3ZfJxE73YQzCHyYCtS5V7UgrdkVCggk4xqOrsUxVG6HosLT7nYk47MXAsaBuSiy0p 5BCIsoCwtq/If2yOYG2aaTD+Yu7m/I3/q6Z0yVUMv2Mt1vtJQpCEFiLql4+XmvIucp7/ SlB4ObtT7QXowClUATq+oDMaE8C+lShMSdYu2eMQlxxrdBhT9CjKSO6ZfxZ2DjiWLeVc AkVWUYxs0TEK0RTIVCattQfH9pFyKPl7HKS4yQbnFH/tziRw2TpfnvjIDk8Lsvttt28Z WsyA==
X-Gm-Message-State: AGi0PuYcJ789sK+XuqXkXauX5p3ofV8k73juK3WKCUgLcnDRwN4PrQdw 7XHRx6LBdAbqOPh4FXXhVs87DEQeYaE=
X-Google-Smtp-Source: APiQypIA6G6GVhcZBH/Fa2BP9j2DvbGu3qgVd0NHsqKkK61x31If8CS1c3qPnQOfngI3m1RENvvzgg==
X-Received: by 2002:a05:6000:18c:: with SMTP id p12mr9150163wrx.335.1589048992408; Sat, 09 May 2020 11:29:52 -0700 (PDT)
Received: from yoda.luk.heistp.net (h-1169.lbcfree.net. [185.193.85.130]) by smtp.gmail.com with ESMTPSA id f7sm8631835wrt.10.2020.05.09.11.29.51 for <tsvwg@ietf.org> (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sat, 09 May 2020 11:29:51 -0700 (PDT)
From: Pete Heist <pete@heistp.net>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.5\))
Message-Id: <8750052B-47EE-4BAE-B821-F4160AE4EAD3@heistp.net>
Date: Sat, 9 May 2020 20:29:50 +0200
To: tsvwg IETF list <tsvwg@ietf.org>
X-Mailer: Apple Mail (2.3445.9.5)
Archived-At: <https://mailarchive.ietf.org/arch/msg/tsvwg/-gx8R1jcwiCsX1_5cdnvl3kKlOs>
Subject: [tsvwg] Bi-directional asymmetric link test results, and Prague self-starvation
X-BeenThere: tsvwg@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Transport Area Working Group <tsvwg.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tsvwg>, <mailto:tsvwg-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tsvwg/>
List-Post: <mailto:tsvwg@ietf.org>
List-Help: <mailto:tsvwg-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tsvwg>, <mailto:tsvwg-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 09 May 2020 18:30:12 -0000

We received some valid criticism at the last interim meeting that we hadn’t published any bi-directional traffic tests, on asymmetric links in particular. In response to that, I’ve added scenarios 8 and 9 to our latest test batch.

The results raise more questions about classic bottleneck detection, and the performance of Prague under conditions we’ve not tested before. What we appear to see is, it’s possible for Prague flows to also starve other Prague flows, because the bottleneck detection heuristic can arrive at different conclusions at different times when there are competing flows, even without the presence of induced jitter.

One of our concerns about heuristic based classic bottleneck detection is that it will be turned off due to its unreliability. From there, the path to starvation of competing flows is pretty straightforward. As an example, if there is a Prague flow that hashes to the same fq_codel bucket as an RFC3168 ECN flow, starvation of the RFC3168 flow *will* occur. fq_codel has 1024 buckets by default. In high capacity backhaul links that use fq_codel (like my ISP has/does), hash collisions will happen reasonably often.

We hope the potential for starvation is understood and taken seriously. So, without further ado...


*** Scenario 8: Four flows each direction over typical cable modem link rates, 10/200Mbit, and 35/1000Mbit

* In the test below with TCP Prague and dualpi2, 10/200Mbit at 80ms RTT, we can see that one Prague flow (the violet trace) has detected the bottleneck as classic (which it’s not). It is then driven to starvation by the other Prague flows some time after t=40s, before those flows also decide it’s a classic bottleneck shortly before t=60s. They then proceed, in phase, as classic flows, underutilizing the link:

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/l4s-s8-bidir-asym/l4s-s8-bidir-asym-ns-prague-dualpi2-10Mbit-200Mbit-80ms_all_scaled_delivery.svg

Run to run, this behavior can differ, as we can see in the second run of the same test:

http://sce.dnsmgr.net/results/ect1-2020-05-08T084306-s8-s9/l4s-s8-bidir-asym/l4s-s8-bidir-asym-ns-prague-dualpi2-10Mbit-200Mbit-80ms_all_scaled_delivery.svg

* Here is the same test as above with CUBIC-SCE and Cake, followed by LFQ, SCE's FQ emulation using two queues:

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/sce-s8-bidir-asym/sce-s8-bidir-asym-ns-cubic-sce-cake-10Mbit-200Mbit-80ms_all_scaled_delivery.svg

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/sce-s8-bidir-asym/sce-s8-bidir-asym-ns-cubic-sce-lfq_cobalt-10Mbit-200Mbit-80ms_all_scaled_delivery.svg


* In the test below with TCP Prague and dualpi2, 35/1000Mbit at 20ms RTT, we can also see flows that appear to detect the bottleneck as classic, and note some under-utilization in the box plot:

http://sce.dnsmgr.net/results/ect1-2020-05-08T084306-s8-s9/l4s-s8-bidir-asym/l4s-s8-bidir-asym-ns-prague-dualpi2-35Mbit-1Gbit-20ms_all_scaled_delivery.svg

http://sce.dnsmgr.net/results/ect1-2020-05-08T084306-s8-s9/l4s-s8-bidir-asym/l4s-s8-bidir-asym-ns-prague-dualpi2-35Mbit-1Gbit-20ms_box_totals.svg

* Here is the same test as above with CUBIC-SCE and cake, followed by LFQ. In the LFQ case as well as some other Twin-CodelAF plots not linked in this email, we’re aware of some throughput drops that are under investigation, however overall utilization is close to capacity:

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/sce-s8-bidir-asym/sce-s8-bidir-asym-ns-cubic-sce-cake-35Mbit-1Gbit-20ms_all_scaled_delivery.svg

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/sce-s8-bidir-asym/sce-s8-bidir-asym-ns-cubic-sce-cake-35Mbit-1Gbit-20ms_box_totals.svg

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/sce-s8-bidir-asym/sce-s8-bidir-asym-ns-cubic-sce-lfq_cobalt-35Mbit-1Gbit-20ms_all_scaled_delivery.svg

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/sce-s8-bidir-asym/sce-s8-bidir-asym-ns-cubic-sce-lfq_cobalt-35Mbit-1Gbit-20ms_box_totals.svg


* In the second run of scenario 8, we included some tests with pfifo and Linux’s default limit of 1000 packets, since those are still in common use on the Internet. Here is TCP Prague in a pfifo at 35/1000Mbit with 20ms RTT:

http://sce.dnsmgr.net/results/ect1-2020-05-08T084306-s8-s9/l4s-s8-bidir-asym/l4s-s8-bidir-asym-ns-prague-pfifo_1000_-35Mbit-1Gbit-20ms_all_scaled_delivery.svg

http://sce.dnsmgr.net/results/ect1-2020-05-08T084306-s8-s9/l4s-s8-bidir-asym/l4s-s8-bidir-asym-ns-prague-pfifo_1000_-35Mbit-1Gbit-20ms_box_totals.svg

* Here is the same test as above with CUBIC-SCE, which is about the same as plain CUBIC:

http://sce.dnsmgr.net/results/ect1-2020-05-08T084306-s8-s9/sce-s8-bidir-asym/sce-s8-bidir-asym-ns-cubic-sce-pfifo_1000_-35Mbit-1Gbit-20ms_all_scaled_delivery.svg

http://sce.dnsmgr.net/results/ect1-2020-05-08T084306-s8-s9/sce-s8-bidir-asym/sce-s8-bidir-asym-ns-cubic-sce-pfifo_1000_-35Mbit-1Gbit-20ms_box_totals.svg


*** Scenario 9: Four flows each direction over a 1Mbit uplink, 1/10Mbit and 1/50Mbit

* In the test below with TCP Prague and dualpi2, 1/10Mbit at 20ms RTT, we see large drop in utilization for some reason, accompanied by a significant increase in the ping time (reflecting the queue depth) over the RTT:

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/l4s-s9-bidir-asym-1mbit/l4s-s9-bidir-asym-1mbit-ns-prague-dualpi2-1Mbit-10Mbit-20ms_all_scaled_delivery.svg

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/l4s-s9-bidir-asym-1mbit/l4s-s9-bidir-asym-1mbit-ns-prague-dualpi2-1Mbit-10Mbit-20ms_box_totals.svg

* Here is the same test as above with CUBIC-SCE in LFQ:

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/sce-s9-bidir-asym-1mbit/sce-s9-bidir-asym-1mbit-ns-cubic-sce-lfq_cobalt-1Mbit-10Mbit-20ms_all_scaled_delivery.svg

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/sce-s9-bidir-asym-1mbit/sce-s9-bidir-asym-1mbit-ns-cubic-sce-lfq_cobalt-1Mbit-10Mbit-20ms_box_totals.svg

* We will also look further into why utilization is closer to capacity when using plain CUBIC and fq_codel:

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/sce-s9-bidir-asym-1mbit/sce-s9-bidir-asym-1mbit-ns-cubic-fq_codel-1Mbit-10Mbit-20ms_box_totals.svg


*** Full results from both runs are here:

http://sce.dnsmgr.net/results/ect1-2020-05-07T143702-s8-s9/

http://sce.dnsmgr.net/results/ect1-2020-05-08T084306-s8-s9/