Re: [DNSOP] [Ext] "The Forgotten Object Lesson Of The Dyn DDoS Attack"

On Thu, Jan 3, 2019 at 9:33 PM Tim Wicinski <tjw.ietf@gmail.com> wrote:

> Actually  draft-huque-dnsop-multi-provider-dnssec was adopted, but the
> author (whom I work with and has heard from
> me about this regularly) has failed to push up an updated version.   I'm
> going to force him to turn the authorship
> over to one of the other authors who is more responsive to the needs of
> the chairs.
>

draft-ietf-dnsop-multi-provider-dnssec (nee
draft-huque-dnsop-multi-provider-dnssec) only solves part of the problem,
and for many of the Dyn customers it wouldn't have solved enough of the
problem to allow them to use multiple providers.

Scenario:
I have a service which is served from 8 datacenters on 2 CDNs - I use a
service like Dyn's Traffic Director to provide geo balancing, to only send
traffic to hosts which are up **and to share the load**.
The first two are relatively trivial to support with multiple DNS providers
- they can independently do geo, they can independently do health checks --
but unless the service does a really good job of exposing its current
available capacity and committed capacity at good resolution, having
independant load balancing gets tricky really quickly.
A single DNS provider can know that location B can serve Nqps, can infer
offered load from how many times it has directed load to B the last X
seconds - them means it can overflow to C when B is *approaching* capacity.
Having a second provider who is unaware of the first one's actions creates
a very real risk of synchronization -- both providers choose B, and so B
gets overloaded - so they both choose C. C is now overloaded, and B is
underutilized, so they both choose B, etc.

This is a well known issue in service load-balancing, in routing protocols
which take circuit utilization into account, etc. It *can* be mitigated /
solved with careful dampening of a Markov decision process, much higher
resolution probing, having one of the providers simply lag by 1/2 the
reaction time, etc.
Balazs Csanad Csaji and Laszlo Monostori's "Stochastic Reactive Production
Scheduling by Multi-agent Based Asynchronous Approximate Dynamic Programming"
(http://old.sztaki.hu/~csaji/csaji-ceemas-2005.pdf) and Down and
Lewis' "Dynamic
Load Balancing in Parallel Queueing Systems: Stability and Optimal Control"
(https://people.orie.cornell.edu/melewis/pubs/load.pdf) are both good reads
on the topic.

There was also a fascinating web page where someone wrote up a system to
avoid exceeding a service's capacity / this sort of issue by using a
standard servo PID controller (https://en.wikipedia.org/wiki/PID_controller)
-- he mapped the "sensed position" as the job capacity, and used the output
of the PID to offer load to the service.
It was obviously one of those of those late-night "Oooh, I wonder what
would happen if..."-type ideas (which morphed into "Here, hold my beer"),
but it actually worked surprisingly well (and was really funny).

I spent some time looking for the page, but cannot find it at the moment...
:-(

W

> Tim
>
>
> On Thu, Jan 3, 2019 at 11:26 AM Paul Hoffman <paul.hoffman@icann.org>
> wrote:
>
>> On Jan 3, 2019, at 5:02 AM, Töma Gavrichenkov <ximaera@gmail.com> wrote:
>> > If I were to trace that through the recent DNSOP activity, I could
>> > bring up my own draft (draft-gavrichenkov-dnsop-dnssapi), also not
>> > adopted and now expired.  Maybe there were discussions of the same
>> > sort before me that I'm not aware of.
>>
>> Or he was possibly talking about draft-huque-dnsop-multi-provider-dnssec,
>> which expired yesterday, and also has not been adopted.
>>
>> --Paul Hoffman_______________________________________________
>> DNSOP mailing list
>> DNSOP@ietf.org
>> https://www.ietf.org/mailman/listinfo/dnsop
>>
> _______________________________________________
> DNSOP mailing list
> DNSOP@ietf.org
> https://www.ietf.org/mailman/listinfo/dnsop
>

-- 
I don't think the execution is relevant when it was obviously a bad idea in
the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair of
pants.
   ---maf