Re: [tcpm] mitigating TCP ACK loop ("ACK storm") DoS attacks

Neal Cardwell <ncardwell@google.com> Wed, 11 February 2015 13:20 UTC

Return-Path: <ncardwell@google.com>
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id F28101A88AD for <tcpm@ietfa.amsl.com>; Wed, 11 Feb 2015 05:20:56 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.389
X-Spam-Level:
X-Spam-Status: No, score=-1.389 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FM_FORGED_GMAIL=0.622, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01] autolearn=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lFvgqgPu3zny for <tcpm@ietfa.amsl.com>; Wed, 11 Feb 2015 05:20:55 -0800 (PST)
Received: from mail-qc0-x235.google.com (mail-qc0-x235.google.com [IPv6:2607:f8b0:400d:c01::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 90E3E1A8899 for <tcpm@ietf.org>; Wed, 11 Feb 2015 05:20:53 -0800 (PST)
Received: by mail-qc0-f181.google.com with SMTP id p6so2617359qcv.12 for <tcpm@ietf.org>; Wed, 11 Feb 2015 05:20:52 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=WixcvJnBvJkN/P4sMCJO3IiVSJYsd+EI1SoaM9J6UFQ=; b=bcJJDBQpxMMLJCdSCIxWl0nCKccSdnn82MsrquadD5wCytGBl9Ca4FSkwguPMI4ovz BlnMoK7zMiuFMY6ngDWDEHfRvJjC9ohtFcVrmUjlNr2ylzgxKupVBhpGZcBxMNJg6g/f DG6vFa61blFnzcUsBRTFF4aGsZrNXo30wmHx83KiLlrBL9zaYpKDNOWyVq0vqZ7FfwGA apZwtTuO0bsJDL5XeiabOB+c2veGi6WIwP7U/vmSrrA8jTOdlK+lZ99uYuhNUHLc9MrY IcnoAmxVreGg6Kdz68yfaLWS+sVJ03OqgDWPcXyMRkoZmdxmgoWRfHaupk9TF5cBMf8K 53KQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=WixcvJnBvJkN/P4sMCJO3IiVSJYsd+EI1SoaM9J6UFQ=; b=CweU9PiG9s8tUvYfBTtu8gfo+R758vfyxorY92pfoQTO9mHtJlgr1BgaqUuvWk2S1P WUO3mFPOUhiV5ZoGBupdNX76fXr4L9a/fbvQscwmh4ASaxloMptXjCWPZC7mcaFHp3fS XyzAX5iJe+4UFhhHMsTKPj3e8CPOmCfqk5IH97CsvznNnnajrffBlDCufxDdCKHgsOAJ FTHSfVpHDc0m3R/M3DEdFnwxHWwJxPP4xO2NWFEe5wAj1FEwejEU4RNuNVBCVLO1dYac cSSUF1N+a6bBlLr7oEOWwVLaOLwkE1tIaK/d/9iKg6CM4b4rYe1LhefYBzYsbAmxFxTZ FsWQ==
X-Gm-Message-State: ALoCoQnn7PmQ+GsvuFL6dTzzrPCwIkqmZin1Zado/VJfSYfsOBgRzHKAsN1vJN6Ei2VCcrVCXf9b
MIME-Version: 1.0
X-Received: by 10.224.55.71 with SMTP id t7mr64659209qag.53.1423660852655; Wed, 11 Feb 2015 05:20:52 -0800 (PST)
Received: by 10.140.137.68 with HTTP; Wed, 11 Feb 2015 05:20:52 -0800 (PST)
In-Reply-To: <54DB316B.3030301@wizmail.org>
References: <CADVnQynQ07-=gzUGbBivua17guztXG7hF4u3gk9m1D+sYyB_Fw@mail.gmail.com> <54DB316B.3030301@wizmail.org>
Date: Wed, 11 Feb 2015 08:20:52 -0500
Message-ID: <CADVnQykE1JirNWbRozqYj00ZJqVjVwwLiA1wkk1oZYeQphabLA@mail.gmail.com>
From: Neal Cardwell <ncardwell@google.com>
To: Jeremy Harris <jgh@wizmail.org>
Content-Type: text/plain; charset="UTF-8"
Archived-At: <http://mailarchive.ietf.org/arch/msg/tcpm/-aA_tqj--ei-IwW2Qtf0uol3rJ8>
Cc: "tcpm@ietf.org" <tcpm@ietf.org>
Subject: Re: [tcpm] mitigating TCP ACK loop ("ACK storm") DoS attacks
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group <tcpm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tcpm>, <mailto:tcpm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tcpm/>
List-Post: <mailto:tcpm@ietf.org>
List-Help: <mailto:tcpm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tcpm>, <mailto:tcpm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 11 Feb 2015 13:20:57 -0000

On Wed, Feb 11, 2015 at 5:39 AM, Jeremy Harris <jgh@wizmail.org> wrote:
> On 10/02/15 02:11, Neal Cardwell wrote:
>> This patch series mitigates such ack loops by rate-limiting outgoing
>> duplicate ACKs sent in response to incoming TCP packets that are for
>> an existing connection but that are invalid due to any of the reasons
>> mentioned above: sequence number (1), ACK field (2), or timestamp
>> value (3). The rate limit for such duplicate ACKs is specified by a
>> new sysctl, tcp_invalid_ratelimit, which specifies the minimal space
>> between such outbound duplicate ACKs, in milliseconds. The default is
>> 500 (500ms), and 0 disables the mechanism.
>
> Would it not scale better over different link speeds to use a
> packet-counting approach rather than a rate limit?   For example,
> output a dupACK only for alternate triggers.

In our case the most common problem was with a middlebox that was not
only (a) desynchronizing the endpoints' notions of the window, but
also (b) duplicating packets to a high degree. So sending a dupack
once per 2 or even once per 32 packets was not enough to break the
loop. If we had responded to one in every 64 incoming out-of-window
packets, that probably would have broken the DoS loop.

However, if we respond to only one in every 64 incoming out-of-window
packets then responses to legitimate traffic become too infrequent.
For example, if the out-of-window packets are zero window probes that
the remote endpoint is sending in an exponentially-backed-off fashion
(very common), then I'd imagine that responding to only 1 in 64
exponentially-backed-off window-probe packets means that the remote
endpoint is often going to give up and close the connection before
they ever get an ACK with a non-zero window.

As for different link speeds, two ACKs per second means we bound
outgoing dupacks to 1Kbit/sec, which is tolerable for most Internet
hosts. If we were to use a packet-counting approach, then with a
middlebox multiplying incoming packets there is no bound on the
outgoing bit rate of our dupacks, and we are more likely to overload
the remote host's link.

neal