Re: [Sidrops] ARIN RPKI Service Impact - 12 August 2020 - manifest issue - resolved

John Curran <jcurran@arin.net> Fri, 14 August 2020 11:32 UTC

Return-Path: <jcurran@arin.net>
X-Original-To: sidrops@ietfa.amsl.com
Delivered-To: sidrops@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 1C6683A0FB3 for <sidrops@ietfa.amsl.com>; Fri, 14 Aug 2020 04:32:33 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.899
X-Spam-Level:
X-Spam-Status: No, score=-1.899 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5M6No4B2tmQF for <sidrops@ietfa.amsl.com>; Fri, 14 Aug 2020 04:32:31 -0700 (PDT)
Received: from smtp2.arin.net (smtp2.arin.net [192.136.136.52]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id BDF073A0FA2 for <sidrops@ietf.org>; Fri, 14 Aug 2020 04:32:31 -0700 (PDT)
Received: from CAS01CHA.corp.arin.net (cas01cha.corp.arin.net [10.1.30.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by smtp2.arin.net (Postfix) with ESMTPS id 2F8B510757BB; Fri, 14 Aug 2020 07:32:28 -0400 (EDT)
Received: from CAS01CHA.corp.arin.net (10.1.30.62) by CAS01CHA.corp.arin.net (10.1.30.62) with Microsoft SMTP Server (TLS) id 15.0.1104.5; Fri, 14 Aug 2020 07:32:27 -0400
Received: from CAS01CHA.corp.arin.net ([fe80::51fb:9cc2:1f9a:288b]) by CAS01CHA.corp.arin.net ([fe80::988:2227:cf44:809%17]) with mapi id 15.00.1104.000; Fri, 14 Aug 2020 07:32:27 -0400
From: John Curran <jcurran@arin.net>
To: Christopher Morrow <christopher.morrow@gmail.com>
CC: "sidrops@ietf.org" <sidrops@ietf.org>
Thread-Topic: [Sidrops] ARIN RPKI Service Impact - 12 August 2020 - manifest issue - resolved
Thread-Index: AQHWceMId69U4v5740qdgSmyMWUcvqk3vHMA
Date: Fri, 14 Aug 2020 11:32:27 +0000
Message-ID: <EEA16680-1733-4532-9081-7520502AC0CC@arin.net>
References: <DE33EFAE-FBD2-478F-92A9-1FBD81CCC43F@arin.net> <CAL9jLaZoFk8qnaZHvXdNqq9vFpWG_ZhRz4f-ufy6HbKQGJ8eoA@mail.gmail.com>
In-Reply-To: <CAL9jLaZoFk8qnaZHvXdNqq9vFpWG_ZhRz4f-ufy6HbKQGJ8eoA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-ms-exchange-messagesentrepresentingtype: 1
x-ms-exchange-transport-fromentityheader: Hosted
x-originating-ip: [192.136.136.37]
Content-Type: text/plain; charset="utf-8"
Content-ID: <C27992139DB1D942AF777DE7DFC502FB@corp.arin.net>
Content-Transfer-Encoding: base64
MIME-Version: 1.0
Archived-At: <https://mailarchive.ietf.org/arch/msg/sidrops/DynKtRtoj3iQihTIC2eHC92lUuc>
Subject: Re: [Sidrops] ARIN RPKI Service Impact - 12 August 2020 - manifest issue - resolved
X-BeenThere: sidrops@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: A list for the SIDR Operations WG <sidrops.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/sidrops>, <mailto:sidrops-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/sidrops/>
List-Post: <mailto:sidrops@ietf.org>
List-Help: <mailto:sidrops-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/sidrops>, <mailto:sidrops-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 14 Aug 2020 11:32:33 -0000

On 13 Aug 2020, at 10:31 PM, Christopher Morrow <christopher.morrow@gmail.com> wrote:
> 
> howdy john!
> ...
> Are there lessons learned here for the other validators and CA folk?
> Are there test cases we can use in other CA deployments? (both RIR and
> delegated)
> ...
>>        I’ll provide a more detailed post-mortem here once available.

Hello Chris!

   Short answer - see above (i.e. “I‘ll provide a more detailed post-mortem here once available.”)
	
   In the meantime, I’ll speculate a bit – warning that this is the view from 10km up by someone with only an offhand knowledge of such things – 

	1) CA operators (e.g. ARIN) should test against a larger portion of the validator ecosystem when doing major changes. 
	2) ARIN needs more diverse and coordinated test environment usage by the RPKI community 
	3) Additional stringency to specs for the more common validators would help in some cases 

  If you’re looking right now for insight of this incident sufficient for writing test cases, I’d look at Job’s OpenBSD writeup - 
  <http://sobornost.net/~job/arin-manifest-issue-2020.08.12.txt>

Best wishes (and stay safe!)
/John

John Curran
President and CEO
American Registry for Internet Numbers