Re: [tcpm] mitigating TCP ACK loop ("ACK storm") DoS attacks
Mirja Kühlewind <mirja.kuehlewind@tik.ee.ethz.ch> Tue, 17 February 2015 13:47 UTC
Return-Path: <mirja.kuehlewind@tik.ee.ethz.ch>
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D2BEB1A896F for <tcpm@ietfa.amsl.com>; Tue, 17 Feb 2015 05:47:41 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -3.91
X-Spam-Level:
X-Spam-Status: No, score=-3.91 tagged_above=-999 required=5 tests=[BAYES_05=-0.5, GB_I_LETTER=-2, J_CHICKENPOX_35=0.6, MIME_8BIT_HEADER=0.3, RCVD_IN_DNSWL_MED=-2.3, T_RP_MATCHES_RCVD=-0.01] autolearn=ham
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id C1DVWLeCxg6J for <tcpm@ietfa.amsl.com>; Tue, 17 Feb 2015 05:47:38 -0800 (PST)
Received: from smtp.ee.ethz.ch (smtp.ee.ethz.ch [129.132.2.219]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 252941A899A for <tcpm@ietf.org>; Tue, 17 Feb 2015 05:47:30 -0800 (PST)
Received: from localhost (localhost [127.0.0.1]) by smtp.ee.ethz.ch (Postfix) with ESMTP id DCA75D930D; Tue, 17 Feb 2015 14:47:28 +0100 (MET)
X-Virus-Scanned: by amavisd-new on smtp.ee.ethz.ch
Received: from smtp.ee.ethz.ch ([127.0.0.1]) by localhost (.ee.ethz.ch [127.0.0.1]) (amavisd-new, port 10024) with LMTP id Ru2nqtkon0qB; Tue, 17 Feb 2015 14:47:28 +0100 (MET)
Received: from [82.130.103.143] (nb-10510.ethz.ch [82.130.103.143]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: mirjak) by smtp.ee.ethz.ch (Postfix) with ESMTPSA id A2F39D930C; Tue, 17 Feb 2015 14:47:28 +0100 (MET)
Message-ID: <54E34670.1090305@tik.ee.ethz.ch>
Date: Tue, 17 Feb 2015 14:47:28 +0100
From: Mirja Kühlewind <mirja.kuehlewind@tik.ee.ethz.ch>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0
MIME-Version: 1.0
To: Neal Cardwell <ncardwell@google.com>, "tcpm@ietf.org" <tcpm@ietf.org>
References: <CADVnQynQ07-=gzUGbBivua17guztXG7hF4u3gk9m1D+sYyB_Fw@mail.gmail.com>
In-Reply-To: <CADVnQynQ07-=gzUGbBivua17guztXG7hF4u3gk9m1D+sYyB_Fw@mail.gmail.com>
Content-Type: text/plain; charset="windows-1252"; format="flowed"
Content-Transfer-Encoding: 8bit
Archived-At: <http://mailarchive.ietf.org/arch/msg/tcpm/R1iN872Qvli3HpuGH06wDjcxQHI>
Cc: Eric Dumazet <edumazet@google.com>
Subject: Re: [tcpm] mitigating TCP ACK loop ("ACK storm") DoS attacks
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group <tcpm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tcpm>, <mailto:tcpm-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/tcpm/>
List-Post: <mailto:tcpm@ietf.org>
List-Help: <mailto:tcpm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tcpm>, <mailto:tcpm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 17 Feb 2015 13:47:42 -0000
Hi all, I probably have a stupid question, but I'll ask anyway as rate-limiting the ACK seems to help the problem but also seems to be kind of a hack and might not resolvie the actually cause of the problem. The question is, why does RFC 793 (section 3.9, page 72) say: "If the ACK acknowledges something not yet sent (SEG.ACK > SND.NXT) then send an ACK"? In this situation the two end points seem to be out of syn, so you have two options: 1) reset the connection and start over. 2) try to resyn which probably means you discard/invalidate (?) the data that have been wrongly acknowledged... What do I miss? Mirja On 10.02.2015 03:11, Neal Cardwell wrote: > TCP DoS scenarios involving ACK loops (aka "ACK storms" or "packet > wars") have come up previously on the TCPM list. For example, Anil > Agarwal brought them up in the Nov 2013 TCPM thread "TCP mismatched > sequence numbers issue" > > I wanted to mention that our TCP team at Google has recently submitted > a patch series for Linux that mitigates such attacks by rate-limiting > the dupacks that are sent in response to out-of-window incoming > packets. The code has been in use at Google and was recently merged > into the official Linux "net-next" tree, which means that it should > land in the next official Linux release. > > The patch series summary can be browsed at the following URL: > > http://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git/commit/?id=f06535c599354816cfbc653ce8965804c7385c61 > > Below I'm including the cover letter summarizing the patch series, for > convenience/reference. > > We are interested to hear any feedback folks may have. > > Thanks! > > neal > > ============== > > tcp: mitigate TCP ACK loops due to out-of-window validation dupacks > > This patch series mitigates "ack loop" DoS scenarios by rate-limiting > outgoing duplicate ACKs sent in response to incoming "out of window" > segments. > > Background > ----------- > > There are several cases in which the TCP RFCs specify that a TCP > endpoint should send a pure duplicate ACK in response to a pure > duplicate ACK that appears to be invalid due to being "out of window": > > (1) RFC 793 (section 3.9, page 69) specifies that endpoints should > send a duplicate ACK in response to an ACK when the incoming > sequence number is invalid due to being outside the receive > window: "If an incoming segment is not acceptable, an > acknowledgment should be sent in reply". > > (2) RFC 793 (section 3.9, page 72) says: "If the ACK acknowledges > something not yet sent (SEG.ACK > SND.NXT) then send an ACK". > > (3) RFC 1323 (section 4.2.1, page 18) specifies that endpoints should > send a duplicate ACK in response to an ACK when the PAWS check for > the incoming timestamp value fails: "If .... SEG.TSval < TS.Recent > and if TS.Recent is valid ... Send an acknowledgement in reply" > > The problem > ------------ > > Normally, this is not a problem. However, a buggy middlebox or > malicious man-in-the-middle can inject a few packets into the > conversation that advance each endpoint's notion of the current window > (sequence, ACK, or timestamp), without either side noticing. In this > case, from then on each side can think the other is sending invalid > segments. Thus an infinite feedback loop of duplicate ACKs can ensue, > as each endpoint receives a duplicate ACK, decides that it is invalid > (due to sequence number, ACK number, or timestamp), and then sends a > dupack in reply, which the other side decides is invalid, responding > with a dupack... ad infinitum. This ping-pong feedback loop can happen > at a very high rate. > > This phenomenon can and does happen in practice. It has been seen in > datacenter and Internet contexts at Google, and has been documented by > Anil Agarwal in the Nov 2013 tcpm thread "TCP mismatched sequence > numbers issue", and Avery Fay in the Feb 2015 Linux netdev thread > "Invalid timestamp? causing tight ack loop (hundreds of thousands of > packets / sec)". > > This patch series > ------------------ > > This patch series mitigates such ack loops by rate-limiting outgoing > duplicate ACKs sent in response to incoming TCP packets that are for > an existing connection but that are invalid due to any of the reasons > mentioned above: sequence number (1), ACK field (2), or timestamp > value (3). The rate limit for such duplicate ACKs is specified by a > new sysctl, tcp_invalid_ratelimit, which specifies the minimal space > between such outbound duplicate ACKs, in milliseconds. The default is > 500 (500ms), and 0 disables the mechanism. > > We rate-limit these duplicate ACK responses rather than blocking them > entirely or resetting the connection, because legitimate connections > can rely on dupacks in response to some out-of-window segments. For > example, zero window probes are typically sent with a sequence number > that is below the current window, and ZWPs thus expect to thus elicit > a dupack in response. > > Testing: this approach has been in use at Google for a while. > > _______________________________________________ > tcpm mailing list > tcpm@ietf.org > https://www.ietf.org/mailman/listinfo/tcpm > -- ------------------------------------------ Dipl.-Ing. Mirja Kühlewind Communication Systems Group Institute TIK, ETH Zürich Gloriastrasse 35, 8092 Zürich, Switzerland Room ETZ G93 phone: +41 44 63 26932 email: mirja.kuehlewind@tik.ee.ethz.ch ------------------------------------------
- [tcpm] mitigating TCP ACK loop ("ACK storm") DoS … Neal Cardwell
- Re: [tcpm] mitigating TCP ACK loop ("ACK storm") … Zimmermann, Alexander
- Re: [tcpm] mitigating TCP ACK loop ("ACK storm") … Wesley Eddy
- Re: [tcpm] mitigating TCP ACK loop ("ACK storm") … Jeremy Harris
- Re: [tcpm] mitigating TCP ACK loop ("ACK storm") … Neal Cardwell
- Re: [tcpm] mitigating TCP ACK loop ("ACK storm") … Mirja Kühlewind
- Re: [tcpm] mitigating TCP ACK loop ("ACK storm") … Scharf, Michael (Michael)
- Re: [tcpm] mitigating TCP ACK loop ("ACK storm") … Eric Dumazet