[Sidrops] Reason for Outage report (was: Re: ARIN RPKI Service Impact - 12 August 2020 - manifest issue - resolved)

John Curran <jcurran@arin.net> Wed, 26 August 2020 14:54 UTC

Return-Path: <jcurran@arin.net>
X-Original-To: sidrops@ietfa.amsl.com
Delivered-To: sidrops@ietfa.amsl.com
Received: from localhost (localhost []) by ietfa.amsl.com (Postfix) with ESMTP id 117DF3A1551 for <sidrops@ietfa.amsl.com>; Wed, 26 Aug 2020 07:54:29 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.898
X-Spam-Status: No, score=-1.898 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([]) by localhost (ietfa.amsl.com []) (amavisd-new, port 10024) with ESMTP id 2Sz0h0bq4uIy for <sidrops@ietfa.amsl.com>; Wed, 26 Aug 2020 07:54:25 -0700 (PDT)
Received: from smtp1.arin.net (smtp1.arin.net [IPv6:2001:500:110:201::51]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 1BAD83A1563 for <sidrops@ietf.org>; Wed, 26 Aug 2020 07:54:24 -0700 (PDT)
Received: from CAS02CHA.corp.arin.net (cas02cha.corp.arin.net []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by smtp1.arin.net (Postfix) with ESMTPS id C340910757B1 for <sidrops@ietf.org>; Wed, 26 Aug 2020 10:54:23 -0400 (EDT)
Received: from CAS01CHA.corp.arin.net ( by CAS02CHA.corp.arin.net ( with Microsoft SMTP Server (TLS) id 15.0.1104.5; Wed, 26 Aug 2020 10:54:18 -0400
Received: from CAS01CHA.corp.arin.net ([fe80::51fb:9cc2:1f9a:288b]) by CAS01CHA.corp.arin.net ([fe80::988:2227:cf44:809%17]) with mapi id 15.00.1104.000; Wed, 26 Aug 2020 10:54:18 -0400
From: John Curran <jcurran@arin.net>
To: "sidrops@ietf.org" <sidrops@ietf.org>
Thread-Topic: Reason for Outage report (was: Re: [Sidrops] ARIN RPKI Service Impact - 12 August 2020 - manifest issue - resolved)
Thread-Index: AQHWe7jFr1X00EMwDk+S8tTniWA6vw==
Date: Wed, 26 Aug 2020 14:54:17 +0000
Message-ID: <727F6FBD-F73C-4F58-AE2D-0276B2A183A3@arin.net>
References: <DE33EFAE-FBD2-478F-92A9-1FBD81CCC43F@arin.net>
In-Reply-To: <DE33EFAE-FBD2-478F-92A9-1FBD81CCC43F@arin.net>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-messagesentrepresentingtype: 1
x-ms-exchange-transport-fromentityheader: Hosted
x-originating-ip: []
Content-Type: multipart/alternative; boundary="_000_727F6FBDF73C4F58AE2D0276B2A183A3arinnet_"
MIME-Version: 1.0
Archived-At: <https://mailarchive.ietf.org/arch/msg/sidrops/Czz4MTRDNoroMPvtrxqJehyZ5UE>
Subject: [Sidrops] Reason for Outage report (was: Re: ARIN RPKI Service Impact - 12 August 2020 - manifest issue - resolved)
X-BeenThere: sidrops@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: A list for the SIDR Operations WG <sidrops.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/sidrops>, <mailto:sidrops-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/sidrops/>
List-Post: <mailto:sidrops@ietf.org>
List-Help: <mailto:sidrops-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/sidrops>, <mailto:sidrops-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 26 Aug 2020 14:54:36 -0000

On 13 Aug 2020, at 1:53 PM, John Curran <jcurran@arin.net<mailto:jcurran@arin.net>> wrote:
I’ll provide a more detailed post-mortem here once available.

RPKI Folks -

As promised, please find the "reason for outage" report for ARIN’s 12 August RPKI incident -
https://www.arin.net/announcements/20200826/  (and attached below in plaintext)

As noted, we’ll be expanding our internal RPKI testing strategy - please review and let us know if there’s additional validators/packages we should be testing against.


John Curran
President and CEO
American Registry for Internet Numbers


12-13 August RPKI Outage: Update
Posted: Wednesday, 26 August 2020
Service Update

12 August 2020 at 12:21 PM to 13 August at 11:36 AM

On 12 August at 12:21 PM, ARIN deployed a new version of its RPKI system. ARIN’s repository showed no errors in both RIPE’s Validator and NLNetLab’s Routinator systems. At 12:46 PM on that same day we received a service issue notice that ARIN’s repository was not working with rpki-client. ARIN Engineering worked closely with the OpenBSD software developers to pinpoint the error within the RPKI system. Both ARIN engineering and the OpenBSD developers independently found the error within ARIN’s repository. The fix was developed and deployed on 13 August at 11:36 AM.

Here is a detailed analysis of the error:

During RPKI repository generation, ARIN creates “manifests.” A manifest is cryptographic object specific to the RPKI which is used to help guarantee the integrity of the repository. One manifest is associated with each resource certificate in the repository. The manifest, flagged by the OpenSSL-based validators, had a subtle encoding issue. The manifest in question essentially contains two copies of an AlgorithmIdentifier variable in different locations (and used for different purposes). Per RFC 5280, Section, these two instances must match completely. In ARIN’s manifest, one contained an empty string (“”) as a parameter and the other contained a NULL (pointer to nothing). The empty string parameter was incorrect and the OpenSSL-based validators were flagging this because the two definitions of AlgorithmIdentifier did not match.

Planned Corrective actions:

As a corrective action, ARIN will be broadening its testing strategy. In future releases, we will be validating not only LibreSSL-based validators (RIPE’s Validator and NLNetlab’s Routinator) but also OpenSSL-based validators such as rpki-client and Fort. The list of validators we do test against the ARIN repository will be noted within the RPKI section of ARIN’s website.

ARIN apologizes for any inconvenience that this may have cause those who run OpenSSL-based validators. We take any incompatibility seriously and are working to better service the community on this emerging but yet very important service. We would like to thank Job Snijders and the OpenBSD rpki-client team for helping assist us in solving this issue quickly.


Mark Kosters
Chief Technology Officer
American Registry for Internet Numbers (ARIN)