Re: [tcpm] ACK aggregation and congestion window growth

Ingemar Johansson S <ingemar.s.johansson@ericsson.com> Tue, 30 April 2019 07:31 UTC

Return-Path: <ingemar.s.johansson@ericsson.com>
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7A6D012021B for <tcpm@ietfa.amsl.com>; Tue, 30 Apr 2019 00:31:13 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2
X-Spam-Level:
X-Spam-Status: No, score=-2 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=ericsson.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PXxxNzGQxHar for <tcpm@ietfa.amsl.com>; Tue, 30 Apr 2019 00:31:08 -0700 (PDT)
Received: from EUR04-VI1-obe.outbound.protection.outlook.com (mail-eopbgr80085.outbound.protection.outlook.com [40.107.8.85]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id BA998120225 for <tcpm@ietf.org>; Tue, 30 Apr 2019 00:31:07 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vvJRJyeTb/TLYHRuifW44m1TzO+Jp3t0eGzqf/tiOBM=; b=LPOKI79SgqDvPPNRZtDDyLegS/GnXh7XFjMoEcNKFHhax/mZblTy1QhdB9UBAtqTM9fL4qoBQ718qjhaVgLMG1X/uL/oOiTqA+3w/x71H5+jYce0NozgEiiHD+djlWdiOjujuPT9f7uYljPXoeONHLW3ESBicsnqGGn9V7JK2yI=
Received: from HE1PR07MB4425.eurprd07.prod.outlook.com (20.176.167.142) by HE1PR07MB4249.eurprd07.prod.outlook.com (20.176.166.146) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1856.7; Tue, 30 Apr 2019 07:31:04 +0000
Received: from HE1PR07MB4425.eurprd07.prod.outlook.com ([fe80::6944:b95e:c64a:6a35]) by HE1PR07MB4425.eurprd07.prod.outlook.com ([fe80::6944:b95e:c64a:6a35%5]) with mapi id 15.20.1856.008; Tue, 30 Apr 2019 07:31:04 +0000
From: Ingemar Johansson S <ingemar.s.johansson@ericsson.com>
To: Neal Cardwell <ncardwell@google.com>
CC: "tcpm@ietf.org" <tcpm@ietf.org>, Yuchung Cheng <ycheng@google.com>, Eric Dumazet <edumazet@google.com>, Rick Jones <perfgeek=40mac.com@dmarc.ietf.org>, Ingemar Johansson S <ingemar.s.johansson@ericsson.com>
Thread-Topic: [tcpm] ACK aggregation and congestion window growth
Thread-Index: AdT8NTrQyrmIo4MtTwOc3MNifKi5sAAA4qIAAADXLAAAt096MA==
Date: Tue, 30 Apr 2019 07:31:04 +0000
Message-ID: <HE1PR07MB44252B4D0228DEC6C67F36BCC23A0@HE1PR07MB4425.eurprd07.prod.outlook.com>
References: <HE1PR07MB4425D7C321FC82CBACED76A5C23E0@HE1PR07MB4425.eurprd07.prod.outlook.com> <AF133C57-C2E0-4383-A7DD-9C4682E4869B@mac.com> <CADVnQy=LhFxYHiHQREzqgBZKX8y-g-EUvH=xjUBuCzeD3grWSw@mail.gmail.com>
In-Reply-To: <CADVnQy=LhFxYHiHQREzqgBZKX8y-g-EUvH=xjUBuCzeD3grWSw@mail.gmail.com>
Accept-Language: sv-SE, en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator:
authentication-results: spf=none (sender IP is ) smtp.mailfrom=ingemar.s.johansson@ericsson.com;
x-originating-ip: [192.176.1.80]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 68c68628-f080-41d5-65eb-08d6cd3dcd8b
x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600141)(711020)(4605104)(2017052603328)(49563074)(7193020); SRVR:HE1PR07MB4249;
x-ms-traffictypediagnostic: HE1PR07MB4249:
x-microsoft-antispam-prvs: <HE1PR07MB4249123FCAA9A695E291676AC23A0@HE1PR07MB4249.eurprd07.prod.outlook.com>
x-ms-oob-tlc-oobclassifiers: OLM:1122;
x-forefront-prvs: 00235A1EEF
x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(366004)(396003)(376002)(346002)(136003)(39860400002)(51914003)(189003)(199004)(186003)(6916009)(99286004)(66066001)(68736007)(2906002)(26005)(606006)(53936002)(6246003)(7696005)(8936002)(81156014)(54896002)(6306002)(76176011)(81166006)(107886003)(53386004)(86362001)(8676002)(9686003)(54906003)(102836004)(236005)(486006)(316002)(476003)(966005)(4326008)(7736002)(256004)(53546011)(71190400001)(66574012)(14444005)(99936001)(6506007)(55016002)(229853002)(52536014)(71200400001)(6116002)(790700001)(74316002)(97736004)(6436002)(25786009)(3846002)(14454004)(73956011)(76116006)(66446008)(66946007)(5660300002)(11346002)(478600001)(66476007)(66616009)(446003)(66556008)(64756008)(33656002); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR07MB4249; H:HE1PR07MB4425.eurprd07.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1;
received-spf: None (protection.outlook.com: ericsson.com does not designate permitted sender hosts)
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam-message-info: uXgUZeTztwRvfk1oTRc8BRVj6ZB6hkICmohBFcShw96VIUR6bAIR865XkQcmkF4wg/j8bwBQtkQdqEQeLE/C7ncr9WZgkjmgw+U1Xxt+tKWLf0P6qT6M6Q96sEXll+pFdizFwJ5bIzn/tPlYa8ZPK7y61TGbikHPRCb/K2CXhr5AJgPSspD2YYjxaXr9fKTtl22cRsbZM098JwHG08v5tzGyqbZQx8NWYN5uqrbALH9APdpnF9Trso+kDRKuYhEmumfhHZUBBqsEFKJ4nSKwFNs2PLicgwzkGrybffMyXsFQK/ULA8rrbjDlgTNQnFiypIxTFBivYwNgkh0KV9rQ0g1Ntk+3Lxmf1ni9J8dSYiX94m966ISyByQbOTqOh8UU4q7MId2VnaH7rwlws+nFElRJdZnmEY/XAKb4kF/wFqw=
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=SHA1; boundary="----=_NextPart_000_004A_01D4FF37.6D4130A0"
MIME-Version: 1.0
X-OriginatorOrg: ericsson.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 68c68628-f080-41d5-65eb-08d6cd3dcd8b
X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Apr 2019 07:31:04.3078 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 92e84ceb-fbfd-47ab-be52-080c6b87953f
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR07MB4249
Archived-At: <https://mailarchive.ietf.org/arch/msg/tcpm/TWzDzwT6tJ9paUZUxUox5sdO5eA>
Subject: Re: [tcpm] ACK aggregation and congestion window growth
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group <tcpm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tcpm>, <mailto:tcpm-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tcpm/>
List-Post: <mailto:tcpm@ietf.org>
List-Help: <mailto:tcpm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tcpm>, <mailto:tcpm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 30 Apr 2019 07:31:14 -0000

Hi

 

Thanks for the hints. I will try and fix some wireshark logs, seems though that this issue is a bit transient and admittedly it works more often than it does not. This is perhaps a corner case ?

 

/Ingemar 

 

From: Neal Cardwell <ncardwell@google.com>; 
Sent: den 26 april 2019 16:27
To: Ingemar Johansson S <ingemar.s.johansson@ericsson.com>;
Cc: tcpm@ietf.org; Yuchung Cheng <ycheng@google.com>;; Eric Dumazet <edumazet@google.com>;; Rick Jones <perfgeek=40mac.com@dmarc.ietf.org>;
Subject: Re: [tcpm] ACK aggregation and congestion window growth

 

Hi Ingemar,

 

Thanks for the report. From the description of the issue, I suspect there is some combination of the following:

 

(1) This may be related to some buggy behavior in the Linux TCP logic for assessing whether a connection is cwnd-limited or application-limited. In the scenario you describe, with those high speeds and long gaps between ACKs, it sounds like the flow may have cwnd that is unused due to to TSO deferral, and this may be causing the logic to erroneously mark the flow as not being cwnd-limited, which would prevent cwnd growth by CUBIC.

 

(2) If there are entire flights of data that are ACKed with a single ACK, then the CUBIC code may assess the long gaps between transmits and ACKs as an idle period, and erroneously push its epoch_start forward to skip any cwnd growth that would have been slated for that period.

 

(I would guess (1) is more likely, since the Reno emulation code path in CUBIC should not be affected by (2), so CUBIC should eventually grow cwnd in a Reno-style fashion whether or not it hits (2).)

 

We can provide some proposed patches for those issues. Would you be able to apply the patches and test them in your workload?  If so, what exact kernel version would the patches need to be generated for?

 

Also, would you be able to post (suitably anonymized, heads-only) tcpdump packet traces so that we can see what the exact scenario is? This would be particularly useful if you are unable to apply the patches to verify the fix.

 

thanks,

neal

 

 

 

 

 

On Fri, Apr 26, 2019 at 10:03 AM Rick Jones <perfgeek= <mailto:40mac.com@dmarc.ietf.org> 40mac.com@dmarc.ietf.org>; wrote:

Does your ACK aggregator also delay the ACKs of SYNchronize segments? If not, perhaps the congestion control sees the increase (?) in round trip time after connection establishment as a signal of congestion and behaves accordingly. 


On Apr 26, 2019, at 06:46, Ingemar Johansson S < <mailto:ingemar.s.johansson@ericsson.com> ingemar.s.johansson@ericsson.com>; wrote:

Hi

 

I am experimenting with a simple test setup with 2 Ubuntu 18.04 PCs (Cubic CC). 

Between these two I have a simple setup that aggregates ACKs so that they are forwarded to the server only every 10ms. The min RTT is 18ms. 

The problem I see is that even though I should reach 1Gbps throughput, I only get around 200Mbps. 

I would believe that the congestion window should eventually increase enough to handle the burstiness given by the ACK aggregation but it seems like this is not the case. 

Is there a limitation in the Linux stack that prevents congestion window growth when ACKs arrive in bursts like this ? 

 

/Ingemar

==================================

Ingemar Johansson  M.Sc. 

Master Researcher

 

Ericsson Research

Network Protocols & E2E Performance

Labratoriegränd 11

971 28, Luleå, Sweden

Phone +46-1071 43042

SMS/MMS +46-73 078 3289

 <mailto:ingemar.s.johansson@ericsson.com> ingemar.s.johansson@ericsson.com

 <http://www.ericsson.com> www.ericsson.com

 

                Nothing to stop this

             being the best day ever

                            U2

==================================

 

_______________________________________________
tcpm mailing list
 <mailto:tcpm@ietf.org> tcpm@ietf.org
 <https://www.ietf.org/mailman/listinfo/tcpm> https://www.ietf.org/mailman/listinfo/tcpm

_______________________________________________
tcpm mailing list
 <mailto:tcpm@ietf.org> tcpm@ietf.org
 <https://www.ietf.org/mailman/listinfo/tcpm> https://www.ietf.org/mailman/listinfo/tcpm