[Sidrops] ARIN RPKI Service Impact - 12 August 2020 - manifest issue - resolved

John Curran <jcurran@arin.net> Thu, 13 August 2020 17:54 UTC

Return-Path: <jcurran@arin.net>
X-Original-To: sidrops@ietfa.amsl.com
Delivered-To: sidrops@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 80D343A07E8 for <sidrops@ietfa.amsl.com>; Thu, 13 Aug 2020 10:54:03 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.899
X-Spam-Level:
X-Spam-Status: No, score=-1.899 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pCGmvOttTYXn for <sidrops@ietfa.amsl.com>; Thu, 13 Aug 2020 10:54:02 -0700 (PDT)
Received: from smtp2.arin.net (smtp2.arin.net [192.136.136.52]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 0761A3A0F3A for <sidrops@ietf.org>; Thu, 13 Aug 2020 10:53:55 -0700 (PDT)
Received: from CAS01CHA.corp.arin.net (cas01cha.corp.arin.net [10.1.30.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by smtp2.arin.net (Postfix) with ESMTPS id 0D27E10757B3 for <sidrops@ietf.org>; Thu, 13 Aug 2020 13:53:55 -0400 (EDT)
Received: from CAS01CHA.corp.arin.net (10.1.30.62) by CAS01CHA.corp.arin.net (10.1.30.62) with Microsoft SMTP Server (TLS) id 15.0.1104.5; Thu, 13 Aug 2020 13:53:55 -0400
Received: from CAS01CHA.corp.arin.net ([fe80::51fb:9cc2:1f9a:288b]) by CAS01CHA.corp.arin.net ([fe80::988:2227:cf44:809%17]) with mapi id 15.00.1104.000; Thu, 13 Aug 2020 13:53:55 -0400
From: John Curran <jcurran@arin.net>
To: "sidrops@ietf.org" <sidrops@ietf.org>
Thread-Topic: ARIN RPKI Service Impact - 12 August 2020 - manifest issue - resolved
Thread-Index: AQHWcZq2hd8cgII59EKc0bG2pPBa2Q==
Date: Thu, 13 Aug 2020 17:53:54 +0000
Message-ID: <DE33EFAE-FBD2-478F-92A9-1FBD81CCC43F@arin.net>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-ms-exchange-messagesentrepresentingtype: 1
x-ms-exchange-transport-fromentityheader: Hosted
x-originating-ip: [192.136.136.37]
Content-Type: text/plain; charset="utf-8"
Content-ID: <203BD54338262D4094D97E7A9032586A@corp.arin.net>
Content-Transfer-Encoding: base64
MIME-Version: 1.0
Archived-At: <https://mailarchive.ietf.org/arch/msg/sidrops/40cOmI4sGL4nYregSABXSztc9_E>
X-Mailman-Approved-At: Thu, 13 Aug 2020 12:45:40 -0700
Subject: [Sidrops] ARIN RPKI Service Impact - 12 August 2020 - manifest issue - resolved
X-BeenThere: sidrops@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: A list for the SIDR Operations WG <sidrops.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/sidrops>, <mailto:sidrops-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/sidrops/>
List-Post: <mailto:sidrops@ietf.org>
List-Help: <mailto:sidrops-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/sidrops>, <mailto:sidrops-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 13 Aug 2020 17:54:04 -0000

RPKI folks - 

	In the process of upgrading our HSM yesterday, ARIN updated its RPKI signing infrastructure incorrectly – the result was an encoding error in our manifest that caused rpki-client and FORT validators no longer consider ARIN’s RPKI data to be valid. (see attached service announcement) 

	This has been since resolved and we’re in the process of reissuing ROAs created during the time.  We are not aware of any delegated repositories impacted during this period. 

	Our thanks to the OpenBSD team - Sebastian Benoit, Theo Buehler, Joel Sing, Job Snijders, and Claudio Jeker - who were instrumental in hunting down this issue. 

	I’ll provide a more detailed post-mortem here once available.

My apologies for the service impact,
/John

John Curran
President and CEO
American Registry for Internet Numbers

===  https://www.arin.net/announcements/20200813/

RPKI Service Notice Update

Posted: Thursday, 13 August 2020 
Service Update 

After upgrading our HSM on Wednesday, August 12, 2020, Job Snijders reported to us that our RPKI repository was no longer validating using rpki-client or fort. Upon investigation, it was discovered that we had an encoding error in our new software. (Specifically, there was a mismatch in the “parameters” field between the “algorithm identifier” of the certificate and the certificate To Be Signed [TBS]. The TBS set the “parameters” as null and the certificate as empty.)

A fix has been made and there will be a pending data clean up in the next few days to fix some of the ROAs created during the interim.

We would like to thank Sebastian Benoit, Theo Buehler, Joel Sing, Job Snijders, and Claudio Jeker, from the OpenBSD project (https://openbsd.org ), as they spent considerable time working with us to identify the root cause of the issue.

Regards,

Richard Jimmerson
Chief Operating Officer
American Registry for Internet Numbers (ARIN)
===