Re: [DNSOP] Terminology: "primary master"

[Joe Abley <[e-mail]>]

Hi Paul,

On 23 Nov 2017, at 13:48, Paul Vixie <[e-mail]> wrote:

> Joe Abley wrote:
>> ...
>> 
>> Feeding a large array of slaves (eg hundreds, including individual
>> members if clusters) with large numbers of zones from a single master
>> doesn't scale very well.
> 
> when i had to do this i fed 100 from one or two, where the two were HA using non-DNS tech such as NFS or sql mirroring. then i've fed 1000 from each of those hundred, choosing one as primary and one as backup.
> 
> if you have a better way, using extant, open technology, to reach 100,000 zxfr/ixfr clients with secure coherent zone content with minimal delay, i'd like to hear more.

I presume you mean existing, not extant. The DNS is a bit new for any of its technology to become extinct or lost to the winds of history, unless that's a naïve comment because I'm insufficiently aware that I'm unaware of the things I'm unaware of. Or perhaps this is a trans-atlantic difference in vocabulary.

The principal challenges I have seen using a non-trivial transfer graph with NOTIFY and [AI]XFR are when a pair of candidate masters for a particular slave each receive a new revision of a particular zone at a different time. The one that receives it first sends NOTIFY messages to its slaves; if slaves don't use the NOTIFY source to select the master subsequently used to send SOA queries and [AI]XFR requests to (and it is normal not to in many/most/all widely-used implementations) there is a chance they will talk to the master that hasn't yet been updated and hence not retrieve the new revision of the zone. Presumably all masters update eventually, however, so normality will eventually be restored.

Whether this is a problem depends on your tolerance for delay in propagation of new zone data to the slaves, and the potential for equivalent masters (equivalent to the slave) to be carrying different revisions of the zone for a long time. It can work as-is; if it doesn't meet the requirements then you have to think a little harder about how you synchronise your masters or what client behaviour can be arranged when they receive NOTIFY messages.

It's nice when it works as-is, because a pure DNS solution avoids the kind ofshared fate and monitoring/troubleshooting overhead that things like load balancers and network filesystems can introduce. Every time I have seen a failure directly caused by a problem in something that was intended to provide high availability, it has made everybody involved very sad.

The state involved in processing NOTIFY messages, especially when there's loss in a path somewhere which triggers timeouts, back-off and retransfer, can be painful to manage at scale. I have seen some abuse of the specification in the form of fire-and-forget NOTIFYs to avoid the state problem (discard NOTIFY responses received and never attempt to retry sending, assuming that if ours didn't get through a NOTIFY from another master probably will).

Davey Song described a scenario that played out in the Yeti project where multiple masters each serving zones that were equivalent but different caused problems to downstream IXFR clients. The zones were equivalent in the sense that they only differed in RRSIGs -- each server used a local ZSK, corresponding properly to an RR in the DNSKEY RRSet, to sign all the non-DNSKEY RRSets -- but different in the sense that the RRSIGs were different, so really they were different zones with the same origin. IXFR clients got confused, which I think in retrospect was probably to be expected. This feels like the kind of scenario that Tony was talking about in his earlier message in this thread.

> when i had to feed hundreds of zones to large populations, i used MZ (see http://family.redbarn.org/~vixie/mz.pdf and perhaps also dotat.at/prog/nsnotifyd/metazone.5.pdf).

I used an earlier version of that back in the day, as you might remember. Pretty much everything I've been involved in since then has used out-of-band provisioning of zone targets though, using devopsy automation tools (ansible, chef, puppet) or provisioning databases that were replicated independently of the DNS. One notable exception used supermasters in PowerDNS, which you could call in-band (but still a different approach from metazones): <https://doc.powerdns.com/md/authoritative/modes-of-operation/#supermaster-automatic-provisioning-of-slaves>.


Joe