Re: [DNSOP] [Ext] "The Forgotten Object Lesson Of The Dyn DDoS Attack"

Warren Kumari <> Fri, 18 January 2019 22:17 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 9614E12D4E8 for <>; Fri, 18 Jan 2019 14:17:38 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.041
X-Spam-Status: No, score=-2.041 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.142, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id 1ZelQ047D_RG for <>; Fri, 18 Jan 2019 14:17:36 -0800 (PST)
Received: from ( [IPv6:2a00:1450:4864:20::333]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 7E176124BAA for <>; Fri, 18 Jan 2019 14:17:35 -0800 (PST)
Received: by with SMTP id f188so5860549wmf.5 for <>; Fri, 18 Jan 2019 14:17:35 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=IBdNhJmDvtObQebZN03UAbbbsbjur0qXkgKQ4DNEb0M=; b=Nx/7BHXNuGG21zocoJHdFOFuITL4tuMbGYO9tjUIT4d+Im6Vhi66Qnvs790DyBdWKh bO4k+6epGGrWnZ1p0OeLEdrKrFRBARnevw9d0qzFF5uNyqzuG8nTdMuCHyYiV+zccUV5 nmIkVpZqC4etP8XMRpXfUHC0wMJyi3XabUPQq7dPF/4YTe7E+cEVjQETT6d7y3kmTc7x pWoMhW7wiBRe858fHZSkPACnDBmhFUfXVIUFOXK0bA9o3QXsiSLxhwsMUD845jMUs1Bt o2X+0251HAbAInWTKdud05FG4xmpOKyA0lXt4E6F3ZyyGLAptA6x//jF/gFIDpB7MUYF kHtg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=IBdNhJmDvtObQebZN03UAbbbsbjur0qXkgKQ4DNEb0M=; b=iLYSRakpqg0rqciNSmHLeGwbF1RTDSscETG15CzmGyOVvlomEuPdfKjThe84Iy3Hot 7ot31D181gOjaXptziTZ+mOTmJ2Oak4NLGJ1bg9IKtjLzafklDvvE2Ro071dd7gEFyAg 9YMitpgCiu4r686zzYeYgOIE+1DNbajNV+s2CdS5m429RxWMlobfCaay02MuFL6SaMB6 2tYWcSn8fjIXrpAcYFPyUzvn4jDMFil+OgRTqpsd5uAToa1RffIfYW4LE/1z7Dm4idWV ArgiID0l04DPaP+9hTUQL/ZL1G6KUfi5qlFLSIo4GpMefgft2BlSc2aFESlnDAW97obP 5/sA==
X-Gm-Message-State: AJcUukc8OLYnpysu+6+QfXjyKwYtnc58PSLp2bv8ZLw0ZGim18ZGEuKG i1iX2w9rxxnb9awk/M1JUwMxn1jjTfN4p4WRii6/YQ==
X-Google-Smtp-Source: ALg8bN7qBNFUxN/sdEKYNbtDm5Ae+TZmtI3iHvtkUXRdT1K6kM1ZT2sDenP37vI8hlxkvftGVyz8+0nGVx2Eg09K3v4=
X-Received: by 2002:a1c:ce0e:: with SMTP id e14mr17895443wmg.53.1547849853406; Fri, 18 Jan 2019 14:17:33 -0800 (PST)
MIME-Version: 1.0
References: <> <> <> <>
In-Reply-To: <>
From: Warren Kumari <>
Date: Fri, 18 Jan 2019 17:16:56 -0500
Message-ID: <>
To: Tim Wicinski <>
Cc: Paul Hoffman <>, dnsop <>
Content-Type: multipart/alternative; boundary="000000000000ada9ab057fc2e0d6"
Archived-At: <>
Subject: Re: [DNSOP] [Ext] "The Forgotten Object Lesson Of The Dyn DDoS Attack"
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: IETF DNSOP WG mailing list <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Fri, 18 Jan 2019 22:17:39 -0000

On Thu, Jan 3, 2019 at 9:33 PM Tim Wicinski <> wrote:

> Actually  draft-huque-dnsop-multi-provider-dnssec was adopted, but the
> author (whom I work with and has heard from
> me about this regularly) has failed to push up an updated version.   I'm
> going to force him to turn the authorship
> over to one of the other authors who is more responsive to the needs of
> the chairs.

draft-ietf-dnsop-multi-provider-dnssec (nee
draft-huque-dnsop-multi-provider-dnssec) only solves part of the problem,
and for many of the Dyn customers it wouldn't have solved enough of the
problem to allow them to use multiple providers.

I have a service which is served from 8 datacenters on 2 CDNs - I use a
service like Dyn's Traffic Director to provide geo balancing, to only send
traffic to hosts which are up **and to share the load**.
The first two are relatively trivial to support with multiple DNS providers
- they can independently do geo, they can independently do health checks --
but unless the service does a really good job of exposing its current
available capacity and committed capacity at good resolution, having
independant load balancing gets tricky really quickly.
A single DNS provider can know that location B can serve Nqps, can infer
offered load from how many times it has directed load to B the last X
seconds - them means it can overflow to C when B is *approaching* capacity.
Having a second provider who is unaware of the first one's actions creates
a very real risk of synchronization -- both providers choose B, and so B
gets overloaded - so they both choose C. C is now overloaded, and B is
underutilized, so they both choose B, etc.

This is a well known issue in service load-balancing, in routing protocols
which take circuit utilization into account, etc. It *can* be mitigated /
solved with careful dampening of a Markov decision process, much higher
resolution probing, having one of the providers simply lag by 1/2 the
reaction time, etc.
Balazs Csanad Csaji and Laszlo Monostori's "Stochastic Reactive Production
Scheduling by Multi-agent Based Asynchronous Approximate Dynamic Programming"
( and Down and
Lewis' "Dynamic
Load Balancing in Parallel Queueing Systems: Stability and Optimal Control"
( are both good reads
on the topic.

There was also a fascinating web page where someone wrote up a system to
avoid exceeding a service's capacity / this sort of issue by using a
standard servo PID controller (
-- he mapped the "sensed position" as the job capacity, and used the output
of the PID to offer load to the service.
It was obviously one of those of those late-night "Oooh, I wonder what
would happen if..."-type ideas (which morphed into "Here, hold my beer"),
but it actually worked surprisingly well (and was really funny).

I spent some time looking for the page, but cannot find it at the moment...


> Tim
> On Thu, Jan 3, 2019 at 11:26 AM Paul Hoffman <>
> wrote:
>> On Jan 3, 2019, at 5:02 AM, Töma Gavrichenkov <> wrote:
>> > If I were to trace that through the recent DNSOP activity, I could
>> > bring up my own draft (draft-gavrichenkov-dnsop-dnssapi), also not
>> > adopted and now expired.  Maybe there were discussions of the same
>> > sort before me that I'm not aware of.
>> Or he was possibly talking about draft-huque-dnsop-multi-provider-dnssec,
>> which expired yesterday, and also has not been adopted.
>> --Paul Hoffman_______________________________________________
>> DNSOP mailing list
> _______________________________________________
> DNSOP mailing list

I don't think the execution is relevant when it was obviously a bad idea in
the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair of