IETF Logo Mail Archive

Re: [DNSOP] How Slack didn't turn on DNSSEC

Mark Andrews <marka@isc.org> Wed, 01 December 2021 07:37 UTC

Return-Path: <marka@isc.org>
X-Original-To: dnsop@ietfa.amsl.com
Delivered-To: dnsop@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 162413A0126 for <dnsop@ietfa.amsl.com>; Tue, 30 Nov 2021 23:37:49 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.099
X-Spam-Level:
X-Spam-Status: No, score=-2.099 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=isc.org header.b=emvNYP6/; dkim=pass (1024-bit key) header.d=isc.org header.b=kt8XzShW
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IJ_2mkwGuYPP for <dnsop@ietfa.amsl.com>; Tue, 30 Nov 2021 23:37:43 -0800 (PST)
Received: from mx.pao1.isc.org (mx.pao1.isc.org [IPv6:2001:4f8:0:2::2b]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id E769C3A02BB for <dnsop@ietf.org>; Tue, 30 Nov 2021 23:37:43 -0800 (PST)
Received: from zimbrang.isc.org (zimbrang.isc.org [149.20.1.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx.pao1.isc.org (Postfix) with ESMTPS id 50D34433F01; Wed, 1 Dec 2021 07:37:41 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=isc.org; s=ostpay; t=1638344261; bh=XlI2V73vz4L21P0S/62SvDcMMuGgmMtgIEECG5OkLBQ=; h=From:Subject:Date:References:Cc:In-Reply-To:To; b=emvNYP6/O86v6brJQeJSH7sqd/gqHConnkAmYjVcim73KwqI8IUbx68bllVjgYfcm tQCdY8JPI+TjGHlbG13kuAGbkKYpc7N/qlzfgxTIbigqIAMny1KP882+hHd9566S71 lTMtLIqUtckb3blLvYpqTQjY5pQ8vZswSehb/KpM=
Received: from zimbrang.isc.org (localhost.localdomain [127.0.0.1]) by zimbrang.isc.org (Postfix) with ESMTPS id 3C768F25595; Wed, 1 Dec 2021 07:37:41 +0000 (UTC)
Received: from localhost (localhost.localdomain [127.0.0.1]) by zimbrang.isc.org (Postfix) with ESMTP id 12A48F25597; Wed, 1 Dec 2021 07:37:41 +0000 (UTC)
DKIM-Filter: OpenDKIM Filter v2.10.3 zimbrang.isc.org 12A48F25597
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=isc.org; s=05DFB016-56A2-11EB-AEC0-15368D323330; t=1638344261; bh=HPgUHOBn/IIQzJduWyATocPgohmdw7HrSQQh4UXhI3c=; h=From:Mime-Version:Date:Message-Id:To; b=kt8XzShWDiNrpXlZgzTuHxpT5htNy3vL8EGyL8Spb09Zxd8vPck7GtTq7hWS5VDXO rB0Et5JvVBOuRWv9EHsihVekkna/bTVhZqmGuRgaRgYKN+tf3/rHAgKZuhYafHt6ri Wtcwkj3P2PdA4efly9DUa0aEGjqrJv3nljQVY6q4=
Received: from zimbrang.isc.org ([127.0.0.1]) by localhost (zimbrang.isc.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id GoN_PR9EYoNS; Wed, 1 Dec 2021 07:37:40 +0000 (UTC)
Received: from smtpclient.apple (unknown [120.18.54.54]) by zimbrang.isc.org (Postfix) with ESMTPSA id 5588EF25595; Wed, 1 Dec 2021 07:37:40 +0000 (UTC)
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
From: Mark Andrews <marka@isc.org>
Mime-Version: 1.0 (1.0)
Date: Wed, 01 Dec 2021 18:30:27 +1100
Message-Id: <D470559D-7506-4F0D-8574-FCED1B7C3531@isc.org>
References: <20211130183809.04E8230CA390@ary.qy>
Cc: dnsop@ietf.org
In-Reply-To: <20211130183809.04E8230CA390@ary.qy>
To: John Levine <johnl@taugh.com>
X-Mailer: iPhone Mail (19B74)
Archived-At: <https://mailarchive.ietf.org/arch/msg/dnsop/Oc2EiG-OK_wSm_6fTpa0nW0AVt4>
Subject: Re: [DNSOP] How Slack didn't turn on DNSSEC
X-BeenThere: dnsop@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: IETF DNSOP WG mailing list <dnsop.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/dnsop>, <mailto:dnsop-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/dnsop/>
List-Post: <mailto:dnsop@ietf.org>
List-Help: <mailto:dnsop-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/dnsop>, <mailto:dnsop-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 01 Dec 2021 07:37:49 -0000

Add tests to DNSVIZ to catch this sort of error. There is an open issue for this. 

-- 
Mark Andrews

> On 1 Dec 2021, at 05:38, John Levine <johnl@taugh.com> wrote:
> 
> This blog post has been making the rounds. Since it is about a
> sequence of DNS operational failures, it seems somewhat relevant here.
> 
> https://slack.engineering/what-happened-during-slacks-dnssec-rollout/
> 
> tl;dr first try was rolled back due to what turned out to be an unrelated failure at some ISP
> 
> second try was rolled back when they found they had a CNAME at a zone
> apex, which they had never noticed until it caused DNSSEC validation
> errors.
> 
> third try was rolled back when they found random-looking failures that
> they eventually tracked down to bugs in Amazon's Route 53 DNS server.
> They had a wildcard with A but not AAAA records. When someone did an
> AAAA query, the response was wrong and said there were no records at
> all, not just no AAAA records. This caused failures at 8.8.8.8 clients
> since Google does aggressive NSEC, not at 1.1.1.1 because Cloudflare
> doesn't.
> 
> They also got some bad advice, e.g., yes the .COM zone adds and
> deletes records very quickly, but that doesn't mean you can unpublish
> a DS and just turn off DNSSEC because its TTL is a day. Their tooling
> somehow didn't let them republish the DNSKEY at the zone apex that
> matched the DS, only a new one that didn't.
> 
> It is clear from the blog post that this is a fairly sophisticated
> group of ops people, who had a reasonable test plan, a bunch of test
> points set up in dnsviz and so forth.  Neither of these bugs seem
> very exotic, and could have been caught by routine tests.
> 
> Can or should we offer advice on how to do this better, sort of like
> RFC 8901 but one level of DNS expertise down?
> 
> R's,
> John
> 
> _______________________________________________
> DNSOP mailing list
> DNSOP@ietf.org
> https://www.ietf.org/mailman/listinfo/dnsop

v2.23.0 | Report a Bug | By Email