Return-Path: X-Original-To: dnsop@ietfa.amsl.com Delivered-To: dnsop@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9614E12D4E8 for ; Fri, 18 Jan 2019 14:17:38 -0800 (PST) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -2.041 X-Spam-Level: X-Spam-Status: No, score=-2.041 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.142, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=kumari-net.20150623.gappssmtp.com Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1ZelQ047D_RG for ; Fri, 18 Jan 2019 14:17:36 -0800 (PST) Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7E176124BAA for ; Fri, 18 Jan 2019 14:17:35 -0800 (PST) Received: by mail-wm1-x333.google.com with SMTP id f188so5860549wmf.5 for ; Fri, 18 Jan 2019 14:17:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kumari-net.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=IBdNhJmDvtObQebZN03UAbbbsbjur0qXkgKQ4DNEb0M=; b=Nx/7BHXNuGG21zocoJHdFOFuITL4tuMbGYO9tjUIT4d+Im6Vhi66Qnvs790DyBdWKh bO4k+6epGGrWnZ1p0OeLEdrKrFRBARnevw9d0qzFF5uNyqzuG8nTdMuCHyYiV+zccUV5 nmIkVpZqC4etP8XMRpXfUHC0wMJyi3XabUPQq7dPF/4YTe7E+cEVjQETT6d7y3kmTc7x pWoMhW7wiBRe858fHZSkPACnDBmhFUfXVIUFOXK0bA9o3QXsiSLxhwsMUD845jMUs1Bt o2X+0251HAbAInWTKdud05FG4xmpOKyA0lXt4E6F3ZyyGLAptA6x//jF/gFIDpB7MUYF kHtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=IBdNhJmDvtObQebZN03UAbbbsbjur0qXkgKQ4DNEb0M=; b=iLYSRakpqg0rqciNSmHLeGwbF1RTDSscETG15CzmGyOVvlomEuPdfKjThe84Iy3Hot 7ot31D181gOjaXptziTZ+mOTmJ2Oak4NLGJ1bg9IKtjLzafklDvvE2Ro071dd7gEFyAg 9YMitpgCiu4r686zzYeYgOIE+1DNbajNV+s2CdS5m429RxWMlobfCaay02MuFL6SaMB6 2tYWcSn8fjIXrpAcYFPyUzvn4jDMFil+OgRTqpsd5uAToa1RffIfYW4LE/1z7Dm4idWV ArgiID0l04DPaP+9hTUQL/ZL1G6KUfi5qlFLSIo4GpMefgft2BlSc2aFESlnDAW97obP 5/sA== X-Gm-Message-State: AJcUukc8OLYnpysu+6+QfXjyKwYtnc58PSLp2bv8ZLw0ZGim18ZGEuKG i1iX2w9rxxnb9awk/M1JUwMxn1jjTfN4p4WRii6/YQ== X-Google-Smtp-Source: ALg8bN7qBNFUxN/sdEKYNbtDm5Ae+TZmtI3iHvtkUXRdT1K6kM1ZT2sDenP37vI8hlxkvftGVyz8+0nGVx2Eg09K3v4= X-Received: by 2002:a1c:ce0e:: with SMTP id e14mr17895443wmg.53.1547849853406; Fri, 18 Jan 2019 14:17:33 -0800 (PST) MIME-Version: 1.0 References: <20190103110300.tyi6ji6f3rcxe2kv@nic.fr> <8721802D-28A7-46D6-B296-A8835EA3FA36@icann.org> In-Reply-To: From: Warren Kumari Date: Fri, 18 Jan 2019 17:16:56 -0500 Message-ID: To: Tim Wicinski Cc: Paul Hoffman , dnsop Content-Type: multipart/alternative; boundary="000000000000ada9ab057fc2e0d6" Archived-At: Subject: Re: [DNSOP] [Ext] "The Forgotten Object Lesson Of The Dyn DDoS Attack" X-BeenThere: dnsop@ietf.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: IETF DNSOP WG mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Jan 2019 22:17:39 -0000 --000000000000ada9ab057fc2e0d6 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Jan 3, 2019 at 9:33 PM Tim Wicinski wrote: > Actually draft-huque-dnsop-multi-provider-dnssec was adopted, but the > author (whom I work with and has heard from > me about this regularly) has failed to push up an updated version. I'm > going to force him to turn the authorship > over to one of the other authors who is more responsive to the needs of > the chairs. > draft-ietf-dnsop-multi-provider-dnssec (nee draft-huque-dnsop-multi-provider-dnssec) only solves part of the problem, and for many of the Dyn customers it wouldn't have solved enough of the problem to allow them to use multiple providers. Scenario: I have a service which is served from 8 datacenters on 2 CDNs - I use a service like Dyn's Traffic Director to provide geo balancing, to only send traffic to hosts which are up **and to share the load**. The first two are relatively trivial to support with multiple DNS providers - they can independently do geo, they can independently do health checks -- but unless the service does a really good job of exposing its current available capacity and committed capacity at good resolution, having independant load balancing gets tricky really quickly. A single DNS provider can know that location B can serve Nqps, can infer offered load from how many times it has directed load to B the last X seconds - them means it can overflow to C when B is *approaching* capacity. Having a second provider who is unaware of the first one's actions creates a very real risk of synchronization -- both providers choose B, and so B gets overloaded - so they both choose C. C is now overloaded, and B is underutilized, so they both choose B, etc. This is a well known issue in service load-balancing, in routing protocols which take circuit utilization into account, etc. It *can* be mitigated / solved with careful dampening of a Markov decision process, much higher resolution probing, having one of the providers simply lag by 1/2 the reaction time, etc. Balazs Csanad Csaji and Laszlo Monostori's "Stochastic Reactive Production Scheduling by Multi-agent Based Asynchronous Approximate Dynamic Programmin= g" (http://old.sztaki.hu/~csaji/csaji-ceemas-2005.pdf) and Down and Lewis' "Dynamic Load Balancing in Parallel Queueing Systems: Stability and Optimal Control" (https://people.orie.cornell.edu/melewis/pubs/load.pdf) are both good reads on the topic. There was also a fascinating web page where someone wrote up a system to avoid exceeding a service's capacity / this sort of issue by using a standard servo PID controller (https://en.wikipedia.org/wiki/PID_controller= ) -- he mapped the "sensed position" as the job capacity, and used the output of the PID to offer load to the service. It was obviously one of those of those late-night "Oooh, I wonder what would happen if..."-type ideas (which morphed into "Here, hold my beer"), but it actually worked surprisingly well (and was really funny). I spent some time looking for the page, but cannot find it at the moment... :-( W > Tim > > > On Thu, Jan 3, 2019 at 11:26 AM Paul Hoffman > wrote: > >> On Jan 3, 2019, at 5:02 AM, T=C3=B6ma Gavrichenkov w= rote: >> > If I were to trace that through the recent DNSOP activity, I could >> > bring up my own draft (draft-gavrichenkov-dnsop-dnssapi), also not >> > adopted and now expired. Maybe there were discussions of the same >> > sort before me that I'm not aware of. >> >> Or he was possibly talking about draft-huque-dnsop-multi-provider-dnssec= , >> which expired yesterday, and also has not been adopted. >> >> --Paul Hoffman_______________________________________________ >> DNSOP mailing list >> DNSOP@ietf.org >> https://www.ietf.org/mailman/listinfo/dnsop >> > _______________________________________________ > DNSOP mailing list > DNSOP@ietf.org > https://www.ietf.org/mailman/listinfo/dnsop > --=20 I don't think the execution is relevant when it was obviously a bad idea in the first place. This is like putting rabid weasels in your pants, and later expressing regret at having chosen those particular rabid weasels and that pair of pants. ---maf --000000000000ada9ab057fc2e0d6 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

On Thu, Jan 3, 2019 at 9:33 PM Tim Wicinski <tjw.ietf@gmail.com> wrote:
Actu= ally=C2=A0=C2=A0draft-huque-dnsop-multi-provider-dnssec was adopted, but th= e author (whom I work with and has heard from=C2=A0
me about this= regularly) has failed to push up an updated version. =C2=A0 I'm going = to force him to turn the authorship
over to one of the other auth= ors who is more responsive to the needs of the chairs.

draft-ietf-dnsop-multi-provider-dnssec (nee dr= aft-huque-dnsop-multi-provider-dnssec) only solves part of the problem, and for= many of the Dyn customers it wouldn't have solved enough of the proble= m to allow them to use multiple providers.
=C2=A0
Scenario:
I have a service which is served from 8 datacen= ters on 2 CDNs - I use a service like Dyn's Traffic Director to provide= geo balancing, to only send traffic to hosts which are up **and to share t= he load**.
The first two are relatively trivial to support wit= h multiple DNS providers - they can independently do geo, they can independ= ently do health checks -- but unless the service does a really good job of = exposing its current available capacity and committed capacity at good reso= lution, having independant load balancing gets tricky really quickly.
A single DNS provider can know that location B can serve Nqps, can = infer offered load from how many times it has directed load to B the last X= seconds - them means it can overflow to C when B is *approaching* capacity= .
Having a second provider who is unaware of the first one'= ;s actions creates a very real risk of synchronization -- both providers ch= oose B, and so B gets overloaded - so they both choose C. C is now overload= ed, and B is underutilized, so they both choose B, etc.=C2=A0
<= div>=
This is a well known issue in service load-balancing, in = routing protocols which take circuit utilization into account, etc. It *can= * be mitigated / solved with careful dampening of a Markov decision process= , much higher resolution probing, having one of the providers simply lag by= 1/2 the reaction time, etc.=C2=A0
Balazs Csanad Csaji and Laszlo Mo= nostori's "Stochastic Reactive Production Scheduling by=C2=A0Multi-agent Based A= synchronous Approx= imate Dynamic Programming" (http://old= .sztaki.hu/~csaji/csaji-ceemas-2005.pdf)=C2=A0and=C2=A0Down and Lewis' "Dynamic Load Balancing in Parallel = Queueing Systems: Stability an= d Optimal Control" (h= ttps://people.orie.cornell.edu/melewis/pubs/load.pdf) are both good reads = on the topic.=C2=A0
<= font face=3D"verdana, sans-serif">
There was also a fasc= inating=C2=A0web page where someone wrote up a system to avoid exceeding a = service's capacity / this sort of issue by using a standard servo PID c= ontroller (https://en.wikipedia.org/wiki/PID= _controller) -- he mapped the=C2=A0"sensed position&quo= t; as the job capacity, and used the output of the PID to offer load to the= service.
= It was obviously=C2=A0one of those of th= ose late-night "Oooh, I wonder what would happen if..."-type idea= s (which morphed into "Here, hold my beer"), but it actually work= ed surprisingly=C2=A0well (and was really funny).
<= span class=3D"gmail_default" style=3D"">=
I spent some time looking for the page, but c= annot find it at the moment... :-(

W


Tim
=


On Thu= , Jan 3, 2019 at 11:26 AM Paul Hoffman <paul.hoffman@icann.org> wrote:
=
On Jan 3, 2019, at 5:02 A= M, T=C3=B6ma Gavrichenkov <ximaera@gmail.com> wrote:
> If I were to trace that through the recent DNSOP activity, I could
> bring up my own draft (draft-gavrichenkov-dnsop-dnssapi), also not
> adopted and now expired.=C2=A0 Maybe there were discussions of the sam= e
> sort before me that I'm not aware of.

Or he was possibly talking about draft-huque-dnsop-multi-provider-dnssec, w= hich expired yesterday, and also has not been adopted.

--Paul Hoffman_______________________________________________
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop
_______________________________________________
DNSOP mailing list
DNSOP@ietf.org
https://www.ietf.org/mailman/listinfo/dnsop


--
I don't think the execution is relevant when= it was obviously a bad idea in the first place.
This is like putting ra= bid weasels in your pants, and later expressing regret at having chosen tho= se particular rabid weasels and that pair of pants.
=C2=A0 =C2=A0---maf<= /div>
--000000000000ada9ab057fc2e0d6--