Re: [tcpm] ACK aggregation and congestion window growth

Neal Cardwell <ncardwell@google.com> Fri, 26 April 2019 14:27 UTC

Return-Path: <ncardwell@google.com>
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7B77F12004B for <tcpm@ietfa.amsl.com>; Fri, 26 Apr 2019 07:27:33 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -17.5
X-Spam-Level:
X-Spam-Status: No, score=-17.5 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, ENV_AND_HDR_SPF_MATCH=-0.5, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, USER_IN_DEF_DKIM_WL=-7.5, USER_IN_DEF_SPF_WL=-7.5] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=google.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ljPRcQSkbT0z for <tcpm@ietfa.amsl.com>; Fri, 26 Apr 2019 07:27:30 -0700 (PDT)
Received: from mail-ot1-x334.google.com (mail-ot1-x334.google.com [IPv6:2607:f8b0:4864:20::334]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7E3D812043C for <tcpm@ietf.org>; Fri, 26 Apr 2019 07:27:30 -0700 (PDT)
Received: by mail-ot1-x334.google.com with SMTP id e5so2762456otk.12 for <tcpm@ietf.org>; Fri, 26 Apr 2019 07:27:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=2WTT3RSYkHslW2jPOrLfi5oJpUYB6VhuqwmeIzvoI9Y=; b=tvIw6Quvu7VAh5O5qZTPNrLJELePlGP81xaNvyUFe11PoJO5Q2rq0k0BA7vtgOFiYX G6zgLuMT2Bt43MPMXuHwrq/uOA1r3UT/6NNPlRUwuZkUEyDEv2Y/BBXaueQarGUm6lwC gKRXstA7aHPNpDq3eItSiiuL4ycWneXHXVL/SgKYUyEVNflolW1S3S18be9R0hkrtLty BzYnzyhtQaC4MaCt2yasku9Mdxwu1UBLWaG13qpu0vwPie5+iECQcWeGd+hDhdinupqj nhPUACrDdrjcZOq9MYNKnWwtIQeAnXxLv81n4vTFLtovdAXIPgz5EWB/Yupa0iMBGJ1e +QQA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=2WTT3RSYkHslW2jPOrLfi5oJpUYB6VhuqwmeIzvoI9Y=; b=uam3PoG6LoeaXJJLrk88Ao83Xahka0zuXoEkefO/ErfNFtVlzvI2Tl66z1GAZOi9wJ i+RgmW3k7epk9GNTl63IcEvr+JeKcTmdUakERdh6+V1jjPhM9kxIaKEq+y2LJM/tRNLo 6/3+XJKqohOwhIe6vXunV0ccdExvyn4diocuaMdblPm7NbwwX7YGvlL3Rc3VO0waedpv RV2a1gkrw/VbYDtMb3QjLx5IZuHgHJqS7PUosMgLzvC52qU4xFbKKi1fV7RzQpn+GwT8 KSH6q9KV9EsSPROI4Q6YBuiMmi2L7KyWS4OywGu9efZg3Ndso8NVy3sMiDnHFumoEvm5 wHEw==
X-Gm-Message-State: APjAAAV6xw5OJ4HmD3r0aoPqVH6kMNgK1+41DEP5s8gS5rU8VQecMLaG GwSVhDcxQMH4M6CVT7Ela2DlDkjgM2pkhwV2GslQpA==
X-Google-Smtp-Source: APXvYqwfoLHw4zPhaat1zCNIboDkwKTOWizjwvcLgZo/GVwJe7EOWMq1jvXBwlZVXyUBl8fiEzaGPNOb3eLR17NWV1Y=
X-Received: by 2002:a9d:5882:: with SMTP id x2mr8242973otg.49.1556288848145; Fri, 26 Apr 2019 07:27:28 -0700 (PDT)
MIME-Version: 1.0
References: <HE1PR07MB4425D7C321FC82CBACED76A5C23E0@HE1PR07MB4425.eurprd07.prod.outlook.com> <AF133C57-C2E0-4383-A7DD-9C4682E4869B@mac.com>
In-Reply-To: <AF133C57-C2E0-4383-A7DD-9C4682E4869B@mac.com>
From: Neal Cardwell <ncardwell@google.com>
Date: Fri, 26 Apr 2019 10:27:10 -0400
Message-ID: <CADVnQy=LhFxYHiHQREzqgBZKX8y-g-EUvH=xjUBuCzeD3grWSw@mail.gmail.com>
To: Ingemar Johansson S <ingemar.s.johansson@ericsson.com>
Cc: "tcpm@ietf.org" <tcpm@ietf.org>, Yuchung Cheng <ycheng@google.com>, Eric Dumazet <edumazet@google.com>, Rick Jones <perfgeek=40mac.com@dmarc.ietf.org>
Content-Type: multipart/alternative; boundary="000000000000f6b74905876fbb50"
Archived-At: <https://mailarchive.ietf.org/arch/msg/tcpm/O0veHjSuSV7dQYVWM3XLTpE-O_0>
Subject: Re: [tcpm] ACK aggregation and congestion window growth
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group <tcpm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tcpm>, <mailto:tcpm-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tcpm/>
List-Post: <mailto:tcpm@ietf.org>
List-Help: <mailto:tcpm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tcpm>, <mailto:tcpm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 26 Apr 2019 14:27:44 -0000

Hi Ingemar,

Thanks for the report. From the description of the issue, I suspect there
is some combination of the following:

(1) This may be related to some buggy behavior in the Linux TCP logic for
assessing whether a connection is cwnd-limited or application-limited. In
the scenario you describe, with those high speeds and long gaps between
ACKs, it sounds like the flow may have cwnd that is unused due to to TSO
deferral, and this may be causing the logic to erroneously mark the flow as
not being cwnd-limited, which would prevent cwnd growth by CUBIC.

(2) If there are entire flights of data that are ACKed with a single ACK,
then the CUBIC code may assess the long gaps between transmits and ACKs as
an idle period, and erroneously push its epoch_start forward to skip any
cwnd growth that would have been slated for that period.

(I would guess (1) is more likely, since the Reno emulation code path in
CUBIC should not be affected by (2), so CUBIC should eventually grow cwnd
in a Reno-style fashion whether or not it hits (2).)

We can provide some proposed patches for those issues. Would you be able to
apply the patches and test them in your workload?  If so, what exact kernel
version would the patches need to be generated for?

Also, would you be able to post (suitably anonymized, heads-only) tcpdump
packet traces so that we can see what the exact scenario is? This would be
particularly useful if you are unable to apply the patches to verify the
fix.

thanks,
neal





On Fri, Apr 26, 2019 at 10:03 AM Rick Jones <perfgeek=
40mac.com@dmarc.ietf.org> wrote:

> Does your ACK aggregator also delay the ACKs of SYNchronize segments? If
> not, perhaps the congestion control sees the increase (?) in round trip
> time after connection establishment as a signal of congestion and behaves
> accordingly.
>
> On Apr 26, 2019, at 06:46, Ingemar Johansson S <
> ingemar.s.johansson@ericsson.com> wrote:
>
> Hi
>
>
>
> I am experimenting with a simple test setup with 2 Ubuntu 18.04 PCs (Cubic
> CC).
>
> Between these two I have a simple setup that aggregates ACKs so that they
> are forwarded to the server only every 10ms. The min RTT is 18ms.
>
> The problem I see is that even though I should reach 1Gbps throughput, I
> only get around 200Mbps.
>
> I would believe that the congestion window should eventually increase
> enough to handle the burstiness given by the ACK aggregation but it seems
> like this is not the case.
>
> Is there a limitation in the Linux stack that prevents congestion window
> growth when ACKs arrive in bursts like this ?
>
>
>
> /Ingemar
>
> ==================================
>
> Ingemar Johansson  M.Sc.
>
> Master Researcher
>
>
>
> Ericsson Research
>
> Network Protocols & E2E Performance
>
> Labratoriegränd 11
>
> 971 28, Luleå, Sweden
>
> Phone +46-1071 43042
>
> SMS/MMS +46-73 078 3289
>
> ingemar.s.johansson@ericsson.com
>
> www.ericsson.com
>
>
>
>                 Nothing to stop this
>
>              being the best day ever
>
>                             U2
>
> ==================================
>
>
>
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm
>
> _______________________________________________
> tcpm mailing list
> tcpm@ietf.org
> https://www.ietf.org/mailman/listinfo/tcpm
>