Re: snarls in real life

Mark Andrews <marka@isc.org> Thu, 22 April 2021 04:12 UTC

Return-Path: <marka@isc.org>
X-Original-To: ietf@ietfa.amsl.com
Delivered-To: ietf@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 3B6583A08B0 for <ietf@ietfa.amsl.com>; Wed, 21 Apr 2021 21:12:34 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -7.119
X-Spam-Level:
X-Spam-Status: No, score=-7.119 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=isc.org header.b=XcCmmu4G; dkim=pass (1024-bit key) header.d=isc.org header.b=nD+Wf1YJ
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ff3I-L8dpJcr for <ietf@ietfa.amsl.com>; Wed, 21 Apr 2021 21:12:30 -0700 (PDT)
Received: from mx.pao1.isc.org (mx.pao1.isc.org [149.20.64.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 19E7C3A08EA for <ietf@ietf.org>; Wed, 21 Apr 2021 21:12:24 -0700 (PDT)
Received: from zmx1.isc.org (zmx1.isc.org [149.20.0.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx.pao1.isc.org (Postfix) with ESMTPS id BD64F3AB023; Thu, 22 Apr 2021 04:12:22 +0000 (UTC)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=isc.org; s=ostpay; t=1619064742; bh=70/GMQzBq1hOuWzqcILjU+PQ4uzZrmG/aRHKKMQ2nXs=; h=Subject:From:In-Reply-To:Date:Cc:References:To; b=XcCmmu4G493bFsvaxEAMg15gGtprUWDrrUv3NwXVgJqxvmBDlHzq2LSPDXZh+DOP5 rUp5K39WwL2vx9gz2YekXQkD1oGz8eMUVyBzSGm9zQ2DajHURRupFyk3M34KCxQp8w L7LKzKW6Heugc3dhp/sdV1hnl2mZgxuelvI57Kr0=
Received: from zmx1.isc.org (localhost.localdomain [127.0.0.1]) by zmx1.isc.org (Postfix) with ESMTPS id 7F28F16006A; Thu, 22 Apr 2021 04:12:22 +0000 (UTC)
Received: from localhost (localhost.localdomain [127.0.0.1]) by zmx1.isc.org (Postfix) with ESMTP id 29B40160060; Thu, 22 Apr 2021 04:12:22 +0000 (UTC)
DKIM-Filter: OpenDKIM Filter v2.9.2 zmx1.isc.org 29B40160060
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=isc.org; s=05DFB016-56A2-11EB-AEC0-15368D323330; t=1619064742; bh=4lOqhXQG41ItSFyvRYnSpdxJoajf8PJi7pu359xA6tA=; h=Content-Type:Mime-Version:Subject:From:Date: Content-Transfer-Encoding:Message-Id:To; b=nD+Wf1YJm+G9GN2vtfpNabKZG5n98RBD5Z6sZDbBQkl9Axvf2fDOD8AToiX1DMZAK UURYstNIT1bzt/uunC9mq2LGqkhteyBrvO0D6+tSXADWWrwh3nvnnq2hGaIe6/sn95 ROG8tt2cHS4TP7nIeXMiKt8PuFe3Fhnj8SKJQaqc=
Received: from zmx1.isc.org ([127.0.0.1]) by localhost (zmx1.isc.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id sAPnHPmAV81b; Thu, 22 Apr 2021 04:12:22 +0000 (UTC)
Received: from [172.30.42.67] (n49-177-132-25.bla3.nsw.optusnet.com.au [49.177.132.25]) by zmx1.isc.org (Postfix) with ESMTPSA id 12479160050; Thu, 22 Apr 2021 04:12:20 +0000 (UTC)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.7\))
Subject: Re: snarls in real life
From: Mark Andrews <marka@isc.org>
In-Reply-To: <efacee7c-bb7d-4861-9037-4c122d3e28ca@dogfood.fastmail.com>
Date: Thu, 22 Apr 2021 14:12:18 +1000
Cc: ietf@ietf.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <4C52380E-4B18-4D48-8474-7A2B05D54479@isc.org>
References: <93fedaa0-5ad0-dcc0-ff01-43b8e1c97989@mtcc.com> <19f2b2e1-6365-480a-86f2-111377cac2de@www.fastmail.com> <7c77e401-4703-3921-d15d-6d69b74df488@mtcc.com> <914f3492-d56b-40ca-b7e0-bbbc65603dfa@dogfood.fastmail.com> <YIC5jFjv/Q7ehujw@straasha.imrryr.org> <efacee7c-bb7d-4861-9037-4c122d3e28ca@dogfood.fastmail.com>
To: Bron Gondwana <brong@fastmailteam.com>
X-Mailer: Apple Mail (2.3445.9.7)
Archived-At: <https://mailarchive.ietf.org/arch/msg/ietf/BPW6zWOm6vEynrsHr0NLR2FrjPs>
X-BeenThere: ietf@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: IETF-Discussion <ietf.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/ietf>, <mailto:ietf-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ietf/>
List-Post: <mailto:ietf@ietf.org>
List-Help: <mailto:ietf-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ietf>, <mailto:ietf-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 22 Apr 2021 04:12:34 -0000


> On 22 Apr 2021, at 10:00, Bron Gondwana <brong@fastmailteam.com> wrote:
> 
> On Thu, Apr 22, 2021, at 09:47, Viktor Dukhovni wrote:
>> My domain has been signed since 2014 without any disruptions, with just
>> a modest monitoring script that has alerted me to pendign expiration
>> (automated re-signing wasn't kicking in) a couple of times, well before
>> the signatures expired.  The bugs that resulted in resigning not
>> happening have been fixed for some time, and I don't have to expend any
>> energy to keep DNSSEC running, it just works.
> 
> That's you - you're an expert in this field.  Most people aren't.  And yet - as you mention, you had a bug with automated re-signing failing and had to add monitoring.
> 
> Also, I suspect that the content of your zone is managed by... you.
> 
> Extrapolating from that to assume that everyone else in the world will have the same experience... maybe the tooling has become heaps better than when we looked in 2016, but the list of DNSSEC failures hasn't exactly trickled to zero - cdc.gov in the year 2021 being a nice example case:
> 
> https://mailman.nanog.org/pipermail/nanog/2021-January/211507.html

CDC just have plain incompetent DNS administrators.  Serving different (unsigned/bad) content on
ns[123].cdc.gov to that on the delegated server for akam.cdc.gov at the time was just idiotic.
It will cause issues even without DNSSEC.  If they are trying to make ns[123].cdc.gov hidden
primaries for akam.cdc.gov they did an abysmal job of it.  Put the hidden primaries on different
addresses or use TSIG to select a different view with the unsigned content.

They sent what appeared to be spoofed (signatures stripped) responses to every validating resolver
on the planet.  The validating resolvers eventually figure it out but not without exceeding client
timeouts and/or query limits.

Currently they are returning REFUSED for names ending in akam.cdc.gov which means they effectively
only have a single working nameserver for cdc.gov for anyone trying to reach their web site.

% dig dnskey akam.cdc.gov +norec +bufsize=1400 @ns1.cdc.gov

; <<>> DiG 9.15.4 <<>> dnskey akam.cdc.gov +norec +bufsize=1400 @ns1.cdc.gov
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 8482
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 4a0cdd6da4a272451d341bfd6080edd34da99a5353fecd1d (good)
;; QUESTION SECTION:
;akam.cdc.gov.			IN	DNSKEY

;; Query time: 229 msec
;; SERVER: 198.246.96.61#53(198.246.96.61)
;; WHEN: Thu Apr 22 13:30:27 AEST 2021
;; MSG SIZE  rcvd: 69

% dig dnskey akam.cdc.gov +norec +bufsize=1400 @ns2.cdc.gov

; <<>> DiG 9.15.4 <<>> dnskey akam.cdc.gov +norec +bufsize=1400 @ns2.cdc.gov
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 35055
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: b57f84015326a3acecdc0fa36080ede53418789f416c7f42 (good)
;; QUESTION SECTION:
;akam.cdc.gov.			IN	DNSKEY

;; Query time: 394 msec
;; SERVER: 198.246.96.92#53(198.246.96.92)
;; WHEN: Thu Apr 22 13:30:46 AEST 2021
;; MSG SIZE  rcvd: 69

% dig dnskey akam.cdc.gov +norec +bufsize=1400 @ns3.cdc.gov

; <<>> DiG 9.15.4 <<>> dnskey akam.cdc.gov +norec +bufsize=1400 @ns3.cdc.gov
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 9881
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 55043d8473bc59bbfd7e57206080edf744d56968c25be8bc (good)
;; QUESTION SECTION:
;akam.cdc.gov.			IN	DNSKEY

;; Query time: 303 msec
;; SERVER: 198.246.125.10#53(198.246.125.10)
;; WHEN: Thu Apr 22 13:31:04 AEST 2021
;; MSG SIZE  rcvd: 69

% dig dnskey akam.cdc.gov +norec +bufsize=1400 @auth00.ns.uu.net

; <<>> DiG 9.15.4+ <<>> dnskey akam.cdc.gov +norec +bufsize=1400 @auth00.ns.uu.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7246
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 6, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 5ac471018fb08a53284b835c6080ee0d00573362d9cb243e (good)
;; QUESTION SECTION:
;akam.cdc.gov.			IN	DNSKEY

;; AUTHORITY SECTION:
akam.cdc.gov.		86400	IN	NS	a9-64.akam.net.
akam.cdc.gov.		86400	IN	NS	a5-66.akam.net.
akam.cdc.gov.		86400	IN	NS	a28-65.akam.net.
akam.cdc.gov.		86400	IN	NS	a8-67.akam.net.
akam.cdc.gov.		86400	IN	NS	a1-43.akam.net.
akam.cdc.gov.		86400	IN	NS	a2-64.akam.net.

;; Query time: 1266 msec
;; SERVER: 198.6.1.65#53(198.6.1.65)
;; WHEN: Thu Apr 22 13:31:26 AEST 2021
;; MSG SIZE  rcvd: 198

% 

% dig www.cdc.gov @ns1.cdc.gov +norec

; <<>> DiG 9.15.4 <<>> www.cdc.gov @ns1.cdc.gov +norec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58095
;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 0753cc8fb65017d8e84280f26080ef345d618fe913abd0fe (good)
;; QUESTION SECTION:
;www.cdc.gov.			IN	A

;; ANSWER SECTION:
www.cdc.gov.		300	IN	CNAME	www.akam.cdc.gov.

;; Query time: 704 msec
;; SERVER: 198.246.96.61#53(198.246.96.61)
;; WHEN: Thu Apr 22 13:36:21 AEST 2021
;; MSG SIZE  rcvd: 91

% dig www.cdc.gov @ns2.cdc.gov +norec

; <<>> DiG 9.15.4 <<>> www.cdc.gov @ns2.cdc.gov +norec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5508
;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 527eb3a3f9958642ae7a63f76080ef4ff9709b05d69578e1 (good)
;; QUESTION SECTION:
;www.cdc.gov.			IN	A

;; ANSWER SECTION:
www.cdc.gov.		300	IN	CNAME	www.akam.cdc.gov.

;; Query time: 227 msec
;; SERVER: 198.246.96.92#53(198.246.96.92)
;; WHEN: Thu Apr 22 13:36:47 AEST 2021
;; MSG SIZE  rcvd: 91

% dig www.cdc.gov @ns3.cdc.gov +norec

; <<>> DiG 9.15.4 <<>> www.cdc.gov @ns3.cdc.gov +norec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14138
;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: d152743ae9d43d26022fb3856080ef59995338b4117d63fe (good)
;; QUESTION SECTION:
;www.cdc.gov.			IN	A

;; ANSWER SECTION:
www.cdc.gov.		300	IN	CNAME	www.akam.cdc.gov.

;; Query time: 338 msec
;; SERVER: 198.246.125.10#53(198.246.125.10)
;; WHEN: Thu Apr 22 13:36:57 AEST 2021
;; MSG SIZE  rcvd: 91

% dig www.cdc.gov @auth00.ns.uu.net +norec

; <<>> DiG 9.15.4 <<>> www.cdc.gov @auth00.ns.uu.net +norec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11325
;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 6, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 1248394757c2c358b65340b76080ef880260f2463fd1f873 (good)
;; QUESTION SECTION:
;www.cdc.gov.			IN	A

;; ANSWER SECTION:
www.cdc.gov.		300	IN	CNAME	www.akam.cdc.gov.

;; AUTHORITY SECTION:
akam.cdc.gov.		86400	IN	NS	a5-66.akam.net.
akam.cdc.gov.		86400	IN	NS	a1-43.akam.net.
akam.cdc.gov.		86400	IN	NS	a9-64.akam.net.
akam.cdc.gov.		86400	IN	NS	a2-64.akam.net.
akam.cdc.gov.		86400	IN	NS	a8-67.akam.net.
akam.cdc.gov.		86400	IN	NS	a28-65.akam.net.

;; Query time: 223 msec
;; SERVER: 198.6.1.65#53(198.6.1.65)
;; WHEN: Thu Apr 22 13:37:44 AEST 2021
;; MSG SIZE  rcvd: 220

% 


> Bron.
> 
> --
>   Bron Gondwana, CEO, Fastmail Pty Ltd
>   brong@fastmailteam.com

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742              INTERNET: marka@isc.org