Re: [DNSOP] [Ext] "The Forgotten Object Lesson Of The Dyn DDoS Attack"

Warren Kumari <warren@kumari.net> Fri, 18 January 2019 22:17 UTC

Return-Path: <warren@kumari.net>
X-Original-To: dnsop@ietfa.amsl.com
Delivered-To: dnsop@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9614E12D4E8 for <dnsop@ietfa.amsl.com>; Fri, 18 Jan 2019 14:17:38 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.041
X-Spam-Level:
X-Spam-Status: No, score=-2.041 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.142, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=kumari-net.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1ZelQ047D_RG for <dnsop@ietfa.amsl.com>; Fri, 18 Jan 2019 14:17:36 -0800 (PST)
Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7E176124BAA for <dnsop@ietf.org>; Fri, 18 Jan 2019 14:17:35 -0800 (PST)
Received: by mail-wm1-x333.google.com with SMTP id f188so5860549wmf.5 for <dnsop@ietf.org>; Fri, 18 Jan 2019 14:17:35 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kumari-net.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=IBdNhJmDvtObQebZN03UAbbbsbjur0qXkgKQ4DNEb0M=; b=Nx/7BHXNuGG21zocoJHdFOFuITL4tuMbGYO9tjUIT4d+Im6Vhi66Qnvs790DyBdWKh bO4k+6epGGrWnZ1p0OeLEdrKrFRBARnevw9d0qzFF5uNyqzuG8nTdMuCHyYiV+zccUV5 nmIkVpZqC4etP8XMRpXfUHC0wMJyi3XabUPQq7dPF/4YTe7E+cEVjQETT6d7y3kmTc7x pWoMhW7wiBRe858fHZSkPACnDBmhFUfXVIUFOXK0bA9o3QXsiSLxhwsMUD845jMUs1Bt o2X+0251HAbAInWTKdud05FG4xmpOKyA0lXt4E6F3ZyyGLAptA6x//jF/gFIDpB7MUYF kHtg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=IBdNhJmDvtObQebZN03UAbbbsbjur0qXkgKQ4DNEb0M=; b=iLYSRakpqg0rqciNSmHLeGwbF1RTDSscETG15CzmGyOVvlomEuPdfKjThe84Iy3Hot 7ot31D181gOjaXptziTZ+mOTmJ2Oak4NLGJ1bg9IKtjLzafklDvvE2Ro071dd7gEFyAg 9YMitpgCiu4r686zzYeYgOIE+1DNbajNV+s2CdS5m429RxWMlobfCaay02MuFL6SaMB6 2tYWcSn8fjIXrpAcYFPyUzvn4jDMFil+OgRTqpsd5uAToa1RffIfYW4LE/1z7Dm4idWV ArgiID0l04DPaP+9hTUQL/ZL1G6KUfi5qlFLSIo4GpMefgft2BlSc2aFESlnDAW97obP 5/sA==
X-Gm-Message-State: AJcUukc8OLYnpysu+6+QfXjyKwYtnc58PSLp2bv8ZLw0ZGim18ZGEuKG i1iX2w9rxxnb9awk/M1JUwMxn1jjTfN4p4WRii6/YQ==
X-Google-Smtp-Source: ALg8bN7qBNFUxN/sdEKYNbtDm5Ae+TZmtI3iHvtkUXRdT1K6kM1ZT2sDenP37vI8hlxkvftGVyz8+0nGVx2Eg09K3v4=
X-Received: by 2002:a1c:ce0e:: with SMTP id e14mr17895443wmg.53.1547849853406; Fri, 18 Jan 2019 14:17:33 -0800 (PST)
MIME-Version: 1.0
References: <20190103110300.tyi6ji6f3rcxe2kv@nic.fr> <CALZ3u+b6HDVj5ZOjtF45rNmYg2yPWCeOakjxNzFgJ_5abfqN_g@mail.gmail.com> <8721802D-28A7-46D6-B296-A8835EA3FA36@icann.org> <CADyWQ+EuubnOcvroxT8bnS60ng0FqKPt59jVGTJF=JUSHOZfwA@mail.gmail.com>
In-Reply-To: <CADyWQ+EuubnOcvroxT8bnS60ng0FqKPt59jVGTJF=JUSHOZfwA@mail.gmail.com>
From: Warren Kumari <warren@kumari.net>
Date: Fri, 18 Jan 2019 17:16:56 -0500
Message-ID: <CAHw9_iJMj=TaiU_jsqD7iJojj6udkuJ8Lbe7gh5Jaqxc3cFfsw@mail.gmail.com>
To: Tim Wicinski <tjw.ietf@gmail.com>
Cc: Paul Hoffman <paul.hoffman@icann.org>, dnsop <dnsop@ietf.org>
Content-Type: multipart/alternative; boundary="000000000000ada9ab057fc2e0d6"
Archived-At: <https://mailarchive.ietf.org/arch/msg/dnsop/nl7hA1s-oUy4mgE98m6BKZxt-KU>
Subject: Re: [DNSOP] [Ext] "The Forgotten Object Lesson Of The Dyn DDoS Attack"
X-BeenThere: dnsop@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: IETF DNSOP WG mailing list <dnsop.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/dnsop>, <mailto:dnsop-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/dnsop/>
List-Post: <mailto:dnsop@ietf.org>
List-Help: <mailto:dnsop-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/dnsop>, <mailto:dnsop-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 18 Jan 2019 22:17:39 -0000

On Thu, Jan 3, 2019 at 9:33 PM Tim Wicinski <tjw.ietf@gmail.com>; wrote:

> Actually  draft-huque-dnsop-multi-provider-dnssec was adopted, but the
> author (whom I work with and has heard from
> me about this regularly) has failed to push up an updated version.   I'm
> going to force him to turn the authorship
> over to one of the other authors who is more responsive to the needs of
> the chairs.
>

draft-ietf-dnsop-multi-provider-dnssec (nee
draft-huque-dnsop-multi-provider-dnssec) only solves part of the problem,
and for many of the Dyn customers it wouldn't have solved enough of the
problem to allow them to use multiple providers.

Scenario:
I have a service which is served from 8 datacenters on 2 CDNs - I use a
service like Dyn's Traffic Director to provide geo balancing, to only send
traffic to hosts which are up **and to share the load**.
The first two are relatively trivial to support with multiple DNS providers
- they can independently do geo, they can independently do health checks --
but unless the service does a really good job of exposing its current
available capacity and committed capacity at good resolution, having
independant load balancing gets tricky really quickly.
A single DNS provider can know that location B can serve Nqps, can infer
offered load from how many times it has directed load to B the last X
seconds - them means it can overflow to C when B is *approaching* capacity.
Having a second provider who is unaware of the first one's actions creates
a very real risk of synchronization -- both providers choose B, and so B
gets overloaded - so they both choose C. C is now overloaded, and B is
underutilized, so they both choose B, etc.

This is a well known issue in service load-balancing, in routing protocols
which take circuit utilization into account, etc. It *can* be mitigated /
solved with careful dampening of a Markov decision process, much higher
resolution probing, having one of the providers simply lag by 1/2 the
reaction time, etc.
Balazs Csanad Csaji and Laszlo Monostori's "Stochastic Reactive Production
Scheduling by Multi-agent Based Asynchronous Approximate Dynamic Programming"
(http://old.sztaki.hu/~csaji/csaji-ceemas-2005.pdf) and Down and
Lewis' "Dynamic
Load Balancing in Parallel Queueing Systems: Stability and Optimal Control"
(https://people.orie.cornell.edu/melewis/pubs/load.pdf) are both good reads
on the topic.

There was also a fascinating web page where someone wrote up a system to
avoid exceeding a service's capacity / this sort of issue by using a
standard servo PID controller (https://en.wikipedia.org/wiki/PID_controller)
-- he mapped the "sensed position" as the job capacity, and used the output
of the PID to offer load to the service.
It was obviously one of those of those late-night "Oooh, I wonder what
would happen if..."-type ideas (which morphed into "Here, hold my beer"),
but it actually worked surprisingly well (and was really funny).

I spent some time looking for the page, but cannot find it at the moment...
:-(

W


> Tim
>
>
> On Thu, Jan 3, 2019 at 11:26 AM Paul Hoffman <paul.hoffman@icann.org>;
> wrote:
>
>> On Jan 3, 2019, at 5:02 AM, Töma Gavrichenkov <ximaera@gmail.com>; wrote:
>> > If I were to trace that through the recent DNSOP activity, I could
>> > bring up my own draft (draft-gavrichenkov-dnsop-dnssapi), also not
>> > adopted and now expired.  Maybe there were discussions of the same
>> > sort before me that I'm not aware of.
>>
>> Or he was possibly talking about draft-huque-dnsop-multi-provider-dnssec,
>> which expired yesterday, and also has not been adopted.
>>
>> --Paul Hoffman_______________________________________________
>> DNSOP mailing list
>> DNSOP@ietf.org
>> https://www.ietf.org/mailman/listinfo/dnsop
>>
> _______________________________________________
> DNSOP mailing list
> DNSOP@ietf.org
> https://www.ietf.org/mailman/listinfo/dnsop
>


-- 
I don't think the execution is relevant when it was obviously a bad idea in
the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair of
pants.
   ---maf