Re: [DNSOP] How Slack didn't turn on DNSSEC

Mark Andrews <> Wed, 01 December 2021 07:37 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 162413A0126 for <>; Tue, 30 Nov 2021 23:37:49 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.099
X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (1024-bit key) header.b=emvNYP6/; dkim=pass (1024-bit key) header.b=kt8XzShW
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id IJ_2mkwGuYPP for <>; Tue, 30 Nov 2021 23:37:43 -0800 (PST)
Received: from ( [IPv6:2001:4f8:0:2::2b]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id E769C3A02BB for <>; Tue, 30 Nov 2021 23:37:43 -0800 (PST)
Received: from ( []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by (Postfix) with ESMTPS id 50D34433F01; Wed, 1 Dec 2021 07:37:41 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple;; s=ostpay; t=1638344261; bh=XlI2V73vz4L21P0S/62SvDcMMuGgmMtgIEECG5OkLBQ=; h=From:Subject:Date:References:Cc:In-Reply-To:To; b=emvNYP6/O86v6brJQeJSH7sqd/gqHConnkAmYjVcim73KwqI8IUbx68bllVjgYfcm tQCdY8JPI+TjGHlbG13kuAGbkKYpc7N/qlzfgxTIbigqIAMny1KP882+hHd9566S71 lTMtLIqUtckb3blLvYpqTQjY5pQ8vZswSehb/KpM=
Received: from (localhost.localdomain []) by (Postfix) with ESMTPS id 3C768F25595; Wed, 1 Dec 2021 07:37:41 +0000 (UTC)
Received: from localhost (localhost.localdomain []) by (Postfix) with ESMTP id 12A48F25597; Wed, 1 Dec 2021 07:37:41 +0000 (UTC)
DKIM-Filter: OpenDKIM Filter v2.10.3 12A48F25597
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=05DFB016-56A2-11EB-AEC0-15368D323330; t=1638344261; bh=HPgUHOBn/IIQzJduWyATocPgohmdw7HrSQQh4UXhI3c=; h=From:Mime-Version:Date:Message-Id:To; b=kt8XzShWDiNrpXlZgzTuHxpT5htNy3vL8EGyL8Spb09Zxd8vPck7GtTq7hWS5VDXO rB0Et5JvVBOuRWv9EHsihVekkna/bTVhZqmGuRgaRgYKN+tf3/rHAgKZuhYafHt6ri Wtcwkj3P2PdA4efly9DUa0aEGjqrJv3nljQVY6q4=
Received: from ([]) by localhost ( []) (amavisd-new, port 10026) with ESMTP id GoN_PR9EYoNS; Wed, 1 Dec 2021 07:37:40 +0000 (UTC)
Received: from (unknown []) by (Postfix) with ESMTPSA id 5588EF25595; Wed, 1 Dec 2021 07:37:40 +0000 (UTC)
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
From: Mark Andrews <>
Mime-Version: 1.0 (1.0)
Date: Wed, 01 Dec 2021 18:30:27 +1100
Message-Id: <>
References: <20211130183809.04E8230CA390@ary.qy>
In-Reply-To: <20211130183809.04E8230CA390@ary.qy>
To: John Levine <>
X-Mailer: iPhone Mail (19B74)
Archived-At: <>
Subject: Re: [DNSOP] How Slack didn't turn on DNSSEC
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: IETF DNSOP WG mailing list <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Wed, 01 Dec 2021 07:37:49 -0000

Add tests to DNSVIZ to catch this sort of error. There is an open issue for this. 

Mark Andrews

> On 1 Dec 2021, at 05:38, John Levine <> wrote:
> This blog post has been making the rounds. Since it is about a
> sequence of DNS operational failures, it seems somewhat relevant here.
> tl;dr first try was rolled back due to what turned out to be an unrelated failure at some ISP
> second try was rolled back when they found they had a CNAME at a zone
> apex, which they had never noticed until it caused DNSSEC validation
> errors.
> third try was rolled back when they found random-looking failures that
> they eventually tracked down to bugs in Amazon's Route 53 DNS server.
> They had a wildcard with A but not AAAA records. When someone did an
> AAAA query, the response was wrong and said there were no records at
> all, not just no AAAA records. This caused failures at clients
> since Google does aggressive NSEC, not at because Cloudflare
> doesn't.
> They also got some bad advice, e.g., yes the .COM zone adds and
> deletes records very quickly, but that doesn't mean you can unpublish
> a DS and just turn off DNSSEC because its TTL is a day. Their tooling
> somehow didn't let them republish the DNSKEY at the zone apex that
> matched the DS, only a new one that didn't.
> It is clear from the blog post that this is a fairly sophisticated
> group of ops people, who had a reasonable test plan, a bunch of test
> points set up in dnsviz and so forth.  Neither of these bugs seem
> very exotic, and could have been caught by routine tests.
> Can or should we offer advice on how to do this better, sort of like
> RFC 8901 but one level of DNS expertise down?
> R's,
> John
> _______________________________________________
> DNSOP mailing list