Re: [Ntp] Microbursts of NTP packets

Hal Murray <hmurray@megapathdsl.net> Tue, 18 August 2020 07:08 UTC

Return-Path: <hmurray@megapathdsl.net>
X-Original-To: ntp@ietfa.amsl.com
Delivered-To: ntp@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 197EE3A17E4 for <ntp@ietfa.amsl.com>; Tue, 18 Aug 2020 00:08:41 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: 1.036
X-Spam-Level: *
X-Spam-Status: No, score=1.036 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HELO_DYNAMIC_IPADDR=1.951, RDNS_DYNAMIC=0.982, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id K5pk5ipqLuoK for <ntp@ietfa.amsl.com>; Tue, 18 Aug 2020 00:08:39 -0700 (PDT)
Received: from ip-64-139-1-69.sjc.megapath.net (ip-64-139-1-69.sjc.megapath.net [64.139.1.69]) by ietfa.amsl.com (Postfix) with ESMTP id 77E953A17DD for <ntp@ietf.org>; Tue, 18 Aug 2020 00:08:34 -0700 (PDT)
Received: from shuksan (localhost [127.0.0.1]) by ip-64-139-1-69.sjc.megapath.net (Postfix) with ESMTP id 25CF440605C; Tue, 18 Aug 2020 00:08:32 -0700 (PDT)
X-Mailer: exmh version 2.7.2 01/07/2005 with nmh-1.3
To: Watson Ladd <watsonbladd@gmail.com>
cc: NTP WG <ntp@ietf.org>, hmurray@megapathdsl.net
From: Hal Murray <hmurray@megapathdsl.net>
In-Reply-To: Message from Watson Ladd <watsonbladd@gmail.com> of "Mon, 17 Aug 2020 21:20:03 EDT." <CACsn0cm7PX-NzJBrA6RR_u=1c3PWjga8t+iccd3Am_VFsDWoKQ@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Date: Tue, 18 Aug 2020 00:08:32 -0700
Message-Id: <20200818070832.25CF440605C@ip-64-139-1-69.sjc.megapath.net>
Archived-At: <https://mailarchive.ietf.org/arch/msg/ntp/aMAcFDm070CU1zHnl19CRW6z9rk>
Subject: Re: [Ntp] Microbursts of NTP packets
X-BeenThere: ntp@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: <ntp.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/ntp>, <mailto:ntp-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ntp/>
List-Post: <mailto:ntp@ietf.org>
List-Help: <mailto:ntp-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ntp>, <mailto:ntp-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 18 Aug 2020 07:08:41 -0000

[I've been working on this with Steven Sommars.]

> We're observing short bursts of high numbers of NTP queries at one point of
> presence, exceeding the queue length of the listening ntp socket, and leading
> to drops. The bursts are very short, so the overall qps is nothing special.
> I'm quite mystified as to what the possible causes could be. 

You didn't provide any numbers, either for duration or size of burst.  What do 
you consider to be "very short"?

I've been using a gap of 1 second to separate bursts and ignoring bursts with 
less than 1000 packets.

Using tcpdump to gather data, I've seen 2 types of bursts.

Steve mention the bursts from Fortigate.  They are typically a bit under 10 
seconds long with a wide range of rates.  Some are over 150K packets or 15K 
packets/sec.

The other is from systems using NAT.  These bursts are typically 100-180 
seconds long which matches the TTL of the pool DNS info.  The burst rate is 
typically under 25 packets/second with a few over 100.

I confirmed with one admin that his IP address was a NAT box with a lab full 
of systems using the pool.  But that is only one data point so I may be 
jumping to conclusions.

Graphs here:
  http://users.megapathdsl.net/~hmurray/Burst/


> This is unfortunately leading to packet drops. If any operators have seen
> this, their input on possible causes and solutions is welcome. 

You probably want some sort of rate limiting to prevent your systems being 
used as a Reflector during DDoS attacks.


-- 
These are my opinions.  I hate spam.