Re: DNSSEC in real life

ned+ietf@mauve.mrochek.com Thu, 22 April 2021 15:45 UTC

Return-Path: <ned+ietf@mauve.mrochek.com>
X-Original-To: ietf@ietfa.amsl.com
Delivered-To: ietf@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9C2703A1543 for <ietf@ietfa.amsl.com>; Thu, 22 Apr 2021 08:45:23 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.898
X-Spam-Level:
X-Spam-Status: No, score=-1.898 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sKe35HyBQ65W for <ietf@ietfa.amsl.com>; Thu, 22 Apr 2021 08:45:18 -0700 (PDT)
Received: from plum.mrochek.com (plum.mrochek.com [172.95.64.195]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 304043A154B for <ietf@ietf.org>; Thu, 22 Apr 2021 08:45:18 -0700 (PDT)
Received: from dkim-sign.mauve.mrochek.com by mauve.mrochek.com (PMDF V6.1-1 #35243) id <01RY5YVRG0XS00DRD2@mauve.mrochek.com> for ietf@ietf.org; Thu, 22 Apr 2021 08:40:15 -0700 (PDT)
MIME-version: 1.0
Content-transfer-encoding: 7BIT
Content-type: TEXT/PLAIN; CHARSET=us-ascii
Received: from mauve.mrochek.com by mauve.mrochek.com (PMDF V6.1-1 #35243) id <01RY4VQ9TXR40085YQ@mauve.mrochek.com> (original mail from NED@mauve.mrochek.com) for ietf@ietf.org; Thu, 22 Apr 2021 08:40:12 -0700 (PDT)
From: ned+ietf@mauve.mrochek.com
Cc: ietf@ietf.org
Message-id: <01RY5YVPO5WW0085YQ@mauve.mrochek.com>
Date: Thu, 22 Apr 2021 07:58:12 -0700 (PDT)
Subject: Re: DNSSEC in real life
In-reply-to: "Your message dated Thu, 22 Apr 2021 00:09:48 -0400" <YID3DAuxxpSk6E/e@straasha.imrryr.org>
References: <93fedaa0-5ad0-dcc0-ff01-43b8e1c97989@mtcc.com> <19f2b2e1-6365-480a-86f2-111377cac2de@www.fastmail.com> <7c77e401-4703-3921-d15d-6d69b74df488@mtcc.com> <914f3492-d56b-40ca-b7e0-bbbc65603dfa@dogfood.fastmail.com> <YIC5jFjv/Q7ehujw@straasha.imrryr.org> <efacee7c-bb7d-4861-9037-4c122d3e28ca@dogfood.fastmail.com> <01RY54LEXW560085YQ@mauve.mrochek.com> <YID3DAuxxpSk6E/e@straasha.imrryr.org>
To: Viktor Dukhovni <ietf-dane@dukhovni.org>
Archived-At: <https://mailarchive.ietf.org/arch/msg/ietf/O62TtXcU5uqiyd2me-GkOT9v9uA>
X-BeenThere: ietf@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: IETF-Discussion <ietf.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/ietf>, <mailto:ietf-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ietf/>
List-Post: <mailto:ietf@ietf.org>
List-Help: <mailto:ietf-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ietf>, <mailto:ietf-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 22 Apr 2021 15:45:24 -0000

> On Wed, Apr 21, 2021 at 05:10:16PM -0700, ned+ietf@mauve.mrochek.com wrote:

> > It's simple enough to configure it for basic domains, but I have a split-horizon
> > setup, and that complicated things quite a bit.

> Sure.  It is often simpler to avoid views and just operate separate
> zones, even if some IP addresses appear in both zones.  But views
> do have their uses.  What versio of BIND are using?  Significant
> further improvements in DNSSEC support have been made in 9.16.

dnssec-policy suppport is the main reason I need to upgrade.

> > However, in the process I managed to make a cut-and-paste error and
> > ended up with one valid and one invalid DS record for one of my
> > infrequently-used domains. Which was noticed by exactly nothing,
> > including the DNSSEC testing tool I happened to use to validate my
> > setup.

> Then you need better monitoring scripts, because DS records should
> always match an extant DNSKEY, proper operational procedure is to first
> retire the DS record, and only later the corresponding key.  With fully
> automated signing, and as registries begin to roll out CDS/CDNSKEY
> support, this sort of mistake will become less common.  But it is easy
> to check for.

I had checks in place. The problem is the reporting tool I chose to use
should have caught this, but didn't.

I think it is asking a lot of people to code all of their own checks. Most
people are going to pick a tool and use it like I did. If that tool is deficient
in some way... how was I supposed to know that?

Of course the real problem is that cut-and-paste was required to do this.
It's really really easy to mess this up when you're processing multiple
domains.

> > Some time later I got a problem report from IANA. It seems that IANA has DNSSEC
> > checking enabled on mail server, I assume because they have enabled DANE
> > (although this was never 100% clear), and I happened to have used this domain
> > for an internal list forwarding address.

> I wouldn't expect a DANE mail server to notice a partial DS RRset
> mismatch, validating resolvers continue to work so long as the
> other DS record is sound.

The one IANA used, and probably still uses, does check. I checked the RFC at the
time and I think the check IANA is doing is allowed, but definitely suboptimal
for a mail server.

But the bigger problem was the error message. If it has said "one of your
DS records is messed up" that would have been fine. But it said "invalid
domain" or something similar. So it was me at one end, report in hand
saying everything is fine, and IANA on the other with a mysterious error.

It was only when IANA thought to do a DNSSEC check using a different tool - one
that does check this - that we figured out the problem.

> > And I still have work to do because Bind has added a number of options
> > that I need to set, but before that can happen I need to upgrade BIND,
> > and before that can happen I need to upgrade some other stuff.

> When you're done upgrading, you'll find BIND 9.16 quite easy to
> work with.  Take a look at the new signing policy features.

Ack.

> > All this is for a total of 9 domains - a toy setup by any measure.

> It is often easier to things at scale than for just a handful of
> domains.  The effort to learn the tools is more efficiently amortised.
> This is why we're seeing a rush to move IT to the cloud and make it
> someone else's problem.

> > I know a little (but not a lot) about the DNS infrastructure at
> > Oracle, and the cost and complexity of migrating it to DNSSEC would be
> > staggering.

> I'm rather sceptical about "staggering", sure there'd be a bunch of man
> weeks spent in meetings, and some devops folks may have to put together
> some Nagios monitoring scripts, and spend a bit of time to enable
> automated zone signing.  It is a non-trivial, but not a staggering cost.

You seriously underestimate the difficulty. You might want to consider the fact
that even if you ignore things like Oracle having multiple separately managed
business units spread across many different countries, like most really large
companies, Oracle is composed almost entirely of acquisitions, new and old, which
for timing and political reasons have undergone varying degrees of assimilation. 

Some of those acquisitions have their own entirely separate DNS setups,
with separate management. 

Of course the goal is to get them all into the collective. But as anyone who has
gone through the process can tell you, there's always resistance, even if in the
long run it's futile...

And there's always another acquisition in the pipeline.

> Yes, the business case to expend the modest, but non-trivial resources
> is often not that great at present, and complications arise if one is
> also depending on just short of standards-compliant DNS load-balancers,
> that don't even do regular EDNS at all well.

> The main adoption barrier is IMHO really poor support for key rollover
> at the registrars.  There's a bunch of recent activity to make that
> better, and some new registrars are much better at this than the long
> established ones, who aren't investing in keeping current, the cost
> savings may make sense in the short to medium term, but long-term I
> expect they're not a good strategy.

Agreed. 

I don't know for sure how many registrars Oracle interacts with - in addition
to being one itself - but my estimate is "a lot".

				Ned