[EToSat] Picoquic, satellites and BBR
Christian Huitema <huitema@huitema.net> Fri, 03 January 2020 03:40 UTC
Return-Path: <huitema@huitema.net>
X-Original-To: etosat@ietfa.amsl.com
Delivered-To: etosat@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C1F1D12004E for <etosat@ietfa.amsl.com>; Thu, 2 Jan 2020 19:40:21 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.6
X-Spam-Level:
X-Spam-Status: No, score=-2.6 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UbzfZu-JHDyd for <etosat@ietfa.amsl.com>; Thu, 2 Jan 2020 19:40:19 -0800 (PST)
Received: from mx43-out1.antispamcloud.com (mx43-out1.antispamcloud.com [138.201.61.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A82F312004D for <etosat@ietf.org>; Thu, 2 Jan 2020 19:40:19 -0800 (PST)
Received: from xse485.mail2web.com ([66.113.197.231] helo=xse.mail2web.com) by mx105.antispamcloud.com with esmtp (Exim 4.89) (envelope-from <huitema@huitema.net>) id 1inDos-0002E6-26 for etosat@ietf.org; Fri, 03 Jan 2020 04:40:17 +0100
Received: from xsmtp22.mail2web.com (unknown [10.100.68.61]) by xse.mail2web.com (Postfix) with ESMTPS id 47prJW22lRz23kc for <etosat@ietf.org>; Thu, 2 Jan 2020 19:40:11 -0800 (PST)
Received: from [10.5.2.13] (helo=xmail03.myhosting.com) by xsmtp22.mail2web.com with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.92) (envelope-from <huitema@huitema.net>) id 1inDop-00042D-5T for etosat@ietf.org; Thu, 02 Jan 2020 19:40:11 -0800
Received: (qmail 7068 invoked from network); 3 Jan 2020 03:40:10 -0000
Received: from unknown (HELO [192.168.200.66]) (Authenticated-user:_huitema@huitema.net@[72.235.197.82]) (envelope-sender <huitema@huitema.net>) by xmail03.myhosting.com (qmail-ldap-1.03) with ESMTPA for <etosat@ietf.org>; 3 Jan 2020 03:40:10 -0000
To: "etosat@ietf.org" <etosat@ietf.org>
References: <BL0PR11MB3394354FD21C86492999FB8590420@BL0PR11MB3394.namprd11.prod.outlook.com> <CAJ5e+HCjai4Z+Cqddo3FDNDnKO_Lpow=T_JfB0+LJ_6JXEuPwQ@mail.gmail.com> <SN6PR11MB3087D0B3421F93B42A2B6C49CE5D0@SN6PR11MB3087.namprd11.prod.outlook.com>
From: Christian Huitema <huitema@huitema.net>
Autocrypt: addr=huitema@huitema.net; prefer-encrypt=mutual; keydata= mQENBFIRX8gBCAC26usy/Ya38IqaLBSu33vKD6hP5Yw390XsWLaAZTeQR64OJEkoOdXpvcOS HWfMIlD5s5+oHfLe8jjmErFAXYJ8yytPj1fD2OdSKAe1TccUBiOXT8wdVxSr5d0alExVv/LO I/vA2aU1TwOkVHKSapD7j8/HZBrqIWRrXUSj2f5n9tY2nJzG9KRzSG0giaJWBfUFiGb4lvsy IaCaIU0YpfkDDk6PtK5YYzuCeF0B+O7N9LhDu/foUUc4MNq4K3EKDPb2FL1Hrv0XHpkXeMRZ olpH8SUFUJbmi+zYRuUgcXgMZRmZFL1tu6z9h6gY4/KPyF9aYot6zG28Qk/BFQRtj7V1ABEB AAG0J0NocmlzdGlhbiBIdWl0ZW1hIDxodWl0ZW1hQGh1aXRlbWEubmV0PokBOQQTAQIAIwUC UhFfyAIbLwcLCQgHAwIBBhUIAgkKCwQWAgMBAh4BAheAAAoJEJNDCbJVyA1yhbYH/1ud6x6m VqGIp0JcZUfSQO8w+TjugqxCyGNn+w/6Qb5O/xENxNQ4HaMQ5uSRK9n8WKKDDRSzwZ4syKKf wbkfj05vgFxrjCynVbm1zs2X2aGXh+PxPL/WHUaxzEP7KjYbLtCUZDRzOOrm+0LMktngT/k3 6+EZoLEM52hwwpIAzJoscyEz7QfqMOZtFm6xQnlvDQeIrHx0KUvwo/vgDLK3SuruG1CSHcR0 D24kEEUa044AIUKBS3b0b8AR7f6mP2NcnLpdsibtpabi9BzqAidcY/EjTaoea46HXALk/eJd 6OLkLE6UQe1PPzQC4jB7rErX2BxnSkHDw50xMgLRcl5/b1a5AQ0EUhFfyAEIAKp7Cp8lqKTV CC9QiAf6QTIjW+lie5J44Ad++0k8gRgANZVWubQuCQ71gxDWLtxYfFkEXjG4TXV/MUtnOliG 5rc2E+ih6Dg61Y5PQakm9OwPIsOx+2R+iSW325ngln2UQrVPgloO83QiUoi7mBJPbcHlxkhZ bd3+EjFxSLIQogt29sTcg2oSh4oljUpz5niTt69IOfZx21kf29NfDE+Iw56gfrxI2ywZbu5o G+d0ZSp0lsovygpk4jK04fDTq0vxjEU5HjPcsXC4CSZdq5E2DrF4nOh1UHkHzeaXdYR2Bn1Y wTePfaHBFlvQzI+Li/Q6AD/uxbTM0vIcsUxrv3MNHCUAEQEAAYkCPgQYAQIACQUCUhFfyAIb LgEpCRCTQwmyVcgNcsBdIAQZAQIABgUCUhFfyAAKCRC22tOSFDh1UOlBB/94RsCJepNvmi/c YiNmMnm0mKb6vjv43OsHkqrrCqJSfo95KHyl5Up4JEp8tiJMyYT2mp4IsirZHxz/5lqkw9Az tcGAF3GlFsj++xTyD07DXlNeddwTKlqPRi/b8sppjtWur6Pm+wnAHp0mQ7GidhxHccFCl65w uT7S/ocb1MjrTgnAMiz+x87d48n1UJ7yIdI41Wpg2XFZiA9xPBiDuuoPwFj14/nK0elV5Dvq 4/HVgfurb4+fd74PV/CC/dmd7hg0ZRlgnB5rFUcFO7ywb7/TvICIIaLWcI42OJDSZjZ/MAzz BeXm263lHh+kFxkh2LxEHnQGHCHGpTYyi4Z3dv03HtkH/1SI8joQMQq00Bv+RdEbJXfEExrT u4gtdZAihwvy97OPA2nCdTAHm/phkzryMeOaOztI4PS8u2Ce5lUB6P/HcGtK/038KdX5MYST Fn8KUDt4o29bkv0CUXwDzS3oTzPNtGdryBkRMc9b+yn9+AdwFEH4auhiTQXPMnl0+G3nhKr7 jvzVFJCRif3OAhEm4vmBNDE3uuaXFQnbK56GJrnqVN+KX5Z3M7X3fA8UcVCGOEHXRP/aubiw Ngawj0V9x+43kUapFp+nF69R53UI65YtJ95ec4PTO/Edvap8h1UbdEOc4+TiYwY1TBuIKltY 1cnrjgAWUh/Ucvr++/KbD9tD6C8=
Message-ID: <ab05b524-633d-1660-bfc2-1ff8332dea66@huitema.net>
Date: Thu, 02 Jan 2020 17:40:09 -1000
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.3.1
MIME-Version: 1.0
In-Reply-To: <SN6PR11MB3087D0B3421F93B42A2B6C49CE5D0@SN6PR11MB3087.namprd11.prod.outlook.com>
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Content-Language: en-US
X-Originating-IP: 66.113.197.231
X-Spampanel-Domain: xsmtpout.mail2web.com
X-Spampanel-Username: 66.113.197.0/24
Authentication-Results: antispamcloud.com; auth=pass smtp.auth=66.113.197.0/24@xsmtpout.mail2web.com
X-Spampanel-Outgoing-Class: unsure
X-Spampanel-Outgoing-Evidence: Combined (0.15)
X-Recommended-Action: accept
X-Filter-ID: Mvzo4OR0dZXEDF/gcnlw0c2Pj46HODYmpAMVAv0J1pOpSDasLI4SayDByyq9LIhVUZbR67CQ7/vm /hHDJU4RXkTNWdUk1Ol2OGx3IfrIJKywOmJyM1qr8uRnWBrbSAGDYbC1vFNC/qFxauDStI0QV8RX qYbtEQV1z/L435ZRxFQBqAQRVbLrpZ90S7fvEpU6+rYZvu7UEJiU3s27VgKHO7lwS3dBJTnTxDoD vBGGxph9w6EwXICYy0ePXtGEMhqrwBb733ZN4jAbrTI5wHo5JWU6UgOqKJ9sMwhVoOBGSAIboXtx P9OF0EfNs5TqNq2Yhy7LI0kfFnXdPP6btp4oBeJDeKRq5oPj2hFJhLx+qI3HlR3ootg7OlA3N5WN re/oppAGOX5cHTu1yz4pRT/9FGrxEaaKeSxe0Wrx6M4G5/WoLsdfEoJI0BNUQ4KpaNyNCwGqOUcw rXf55E8Tb8bmXq4yH8StrboPphDtmrtUkwkDMc9xayd+oZJo2heFY+g6kVWClPVvbW5lVyQanRxw 5rdY2rW50fd1ekaDpmIWc1Vmt3mnxMTQMQWbvBqEXskTQn6USYs98Imn+lZXe3dwYfgVB1xo6dCf BaU/iegBU8afR67T7N272aMX0YT5C7M3Ewi7c1r2rHa2DGV73o+YsqERNWjF8mkVR+FbsebS4lT2 ie80q3LAG2MiIaIREzT1xNjuO97khcUFBr/guEWv1bdCp3Zd9clP8wSiJZWbJCj+xRrjVmRxpGtS cvUmgj1LhZhPA1QhQiPMoSWmoh8L812jZAOanSBpz6Rja2u/0jJ7soudVSfd2i2QSN7BzyzJFb3C d6tjlyeqezHL3r3RZvysta6u1iHEyuS7GD1uvcq8z5KsNua73EuSdCUwlABhJOYIJd4MvQ0Nf4Ec bvHO1diDanHV9KirFAIIecsyj+YNTo81GR+jDXFsz/ZQnbbTizvwlZsrbltGiZoUh+c+5pFVgpT1 b21uZVckGp0ccOa2XhkGbmsUNPNkere1WheNsVXmhO8BzADiszcWR9bz/SDtF09JpSbuuCeiIDK0 C/0=
X-Report-Abuse-To: spam@quarantine11.antispamcloud.com
Archived-At: <https://mailarchive.ietf.org/arch/msg/etosat/ZPHjg1JWOT89pD4499Mi25asxlQ>
Subject: [EToSat] Picoquic, satellites and BBR
X-BeenThere: etosat@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "The EToSat list is a non-WG mailing list used to discuss performance implications of running encrypted transports such as QUIC over satellite." <etosat.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/etosat>, <mailto:etosat-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/etosat/>
List-Post: <mailto:etosat@ietf.org>
List-Help: <mailto:etosat-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/etosat>, <mailto:etosat-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 03 Jan 2020 03:40:22 -0000
I spent some time updating the Picoquic code so it performs decently on satellite links. I added a set of tests mimicking the 250/3 "clean" and 250/3 lossy scenarios. I did not implement the proposed extension yet, but here is are the first results, running on a simulator: 1) Upload 100MB over 250 Mbps link with 3 Mbps return link, no transmission errors: completes in about 6.4 seconds. 2) Upload 100MB over 250 Mbps link with 3 Mbps return list, 1.6% transmission errors: completes in about 9.8 seconds. The transmission buffers were set to 1 BDP on each side. The tests use upload instead of download because of some limitations of my setup: it was easier for me to instrument the client rather than the server. Download tests would last the same time. In both cases, the congestion control algorithm is set to BBR, with a couple of small modifications: time stamps in ACKs, and Hystart style replacement of start-up. BBR does not reduce data rate on transient losses like those found in the lossy scenario. Picoquic implements ACK coalescing, with 10 packets per ACK when transmitting at high rate. For the "clean" test, the 6.4 seconds include 0.6 second for setting the connection, 0.6 for requesting/confirming the file, and 0.6 for closing the connection. The actual transmission lasts 5.2 seconds, running at 153 Mbps on average, 61% efficiency. The main overhead is the ramp-up time, from the initial window to the nominal data rate. In the tests that lasts about 3.6 seconds. After that, the transmission rate stabilizes close to full utilization. I did a couple of tests downloading 1GB instead of 100MB, and the last 900MB are transmitted at or close to line rate. For the "lossy" test, there is significant additional delay at the end of the transmission. At full speed, the BDP is 18.75 MB, i.e. close to 13000 packets. Out of these, about 200 will need to be resent once, and 3 of those will have to be resent twice. That means adding a couple of RTOs at the end of the transmission. That explains about 90% of the difference with the "clean" test. I suppose that I could work on the retransmission code and reduce these delays a bit. Most of the work I did so far dealt with performance issues caused by long ACK lists and frequent "holes", not on actual efficiency of error correction. But even with frequent losses, the transmission rate stabilizes at or close to line rate. I added time stamps to the ACK frames when testing the asymmetry. I wanted to measure how much of the delays were due to transmit side queues, and how much to ACK queues. Asymmetry actually ceased to be a problem after ACK coalescing was properly debugged. In the latest tests, all the delay variations are due to the transmit side queues, and I would get the same results without the timestamps. Still, time stamps in ACKs are a good idea in general. It is an insurance against ACK coalescing in ACK queueing in the network, enabling proper bandwidth measurements even when the ACK link is congested. It seems very valuable when using BBR or delay-based congestion control like FAST or LEDBAT. I had an issue with the "startup" algorithm of BBR. Early testing showed that BBR startup phase requires several more RTT than the Hystart process used in modern versions of Reno or Cubic. BBR only ramps up the data rate after the first bandwidth measurement is available, 2*RTT after start, while Reno or Cubic start ramping up after just 1 RTT. BBR only exits startup if three consecutive RTT pass without significant BW measurement increase, which not only adds delay but also creates big queues as data is sent at 2.89 times the bottleneck rate for an addition 2 RTT. This is a tradeoff: longer search for bandwidth in slow start is less likely to stop too early because of transient issues, but on high bandwidth and long delay links this translates to long delays and a big batch of packet losses. The BBR implementation in Picoquic addresses these issues by switching to Hystart instead of BBR-startup if the RTT is above 100 ms. This batch of "satellite" fixes is implemented in the latest version of Picoquic. Of course, I am still waiting for the BDP option, which could solve the "ramp up" issue. I just want to make sure that we are not opening a big DOS mechanism. -- Christian Huitema
- [EToSat] What we are looking for from QUIC Border, John
- Re: [EToSat] What we are looking for from QUIC Christian Huitema
- Re: [EToSat] What we are looking for from QUIC Junho Choi
- Re: [EToSat] What we are looking for from QUIC lloyd.wood@yahoo.co.uk
- Re: [EToSat] What we are looking for from QUIC Su, Chi-Jiun
- Re: [EToSat] What we are looking for from QUIC Junho Choi
- Re: [EToSat] What we are looking for from QUIC Kuhn Nicolas
- Re: [EToSat] What we are looking for from QUIC Emmanuel Lochin
- Re: [EToSat] What we are looking for from QUIC Gorry Fairhurst
- Re: [EToSat] What we are looking for from QUIC Kuhn Nicolas
- Re: [EToSat] What we are looking for from QUIC Ted Hardie
- Re: [EToSat] What we are looking for from QUIC Morten V. Pedersen
- Re: [EToSat] What we are looking for from QUIC Su, Chi-Jiun
- Re: [EToSat] What we are looking for from QUIC Border, John
- Re: [EToSat] What we are looking for from QUIC Christian Huitema
- Re: [EToSat] What we are looking for from QUIC Su, Chi-Jiun
- Re: [EToSat] What we are looking for from QUIC Morten V. Pedersen
- Re: [EToSat] What we are looking for from QUIC lloyd.wood@yahoo.co.uk
- Re: [EToSat] What we are looking for from QUIC Su, Chi-Jiun
- Re: [EToSat] What we are looking for from QUIC François Michel
- Re: [EToSat] What we are looking for from QUIC Morten V. Pedersen
- [EToSat] Picoquic, satellites and BBR Christian Huitema
- Re: [EToSat] Picoquic, satellites and BBR Su, Chi-Jiun