Re: [Sidrops] Reason for Outage report (was: Re: ARIN RPKI Service Impact - 12 August 2020 - manifest issue - resolved)

[Job Snijders <[e-mail]>]

Dear John,

I'd like to offer some contributions to this document's outline of the
background of some implementations. Mostly just nitpicking.

On Wed, Aug 26, 2020 at 02:54:17PM +0000, John Curran wrote:
> 12-13 August RPKI Outage: Update
> Posted: Wednesday, 26 August 2020
> Service Update
> 
> 12 August 2020 at 12:21 PM to 13 August at 11:36 AM
> 
> On 12 August at 12:21 PM, ARIN deployed a new version of its RPKI
> system. ARIN’s repository showed no errors in both RIPE’s Validator
> and NLNetLab’s Routinator systems. 

The current versions of routinator and ripe ncc's validator have weak
(lacking) support for manifest handling, there are other issues in both
softwares that don't yield errors where they should yield errors related
to manifest handling. Neither implementation handles manifests correctly
at the moment, so neither software currently can be used to confirm the
correct publication of manifest related data. :-(

> At 12:46 PM on that same day we received a service issue notice that
> ARIN’s repository was not working with rpki-client. ARIN Engineering
> worked closely with the OpenBSD software developers to pinpoint the
> error within the RPKI system. Both ARIN engineering and the OpenBSD
> developers independently found the error within ARIN’s repository. The
> fix was developed and deployed on 13 August at 11:36 AM.

It was not just rpki-client (linked against LibreSSL), but the FORT
validator (linked against OpenSSL) also flagged the same issue and
emitted an error. LibreSSL and OpenSSL's RPKI extensions do share some
public APIs, (and have some code in common), but they are distinct
implementations from separate development communities. 

> Here is a detailed analysis of the error:
> 
> During RPKI repository generation, ARIN creates “manifests.” A
> manifest is cryptographic object specific to the RPKI which is used to
> help guarantee the integrity of the repository. One manifest is
> associated with each resource certificate in the repository. The
> manifest, flagged by the OpenSSL-based validators, had a subtle
> encoding issue. 
>
> The manifest in question essentially contains two
> copies of an AlgorithmIdentifier variable in different locations (and
> used for different purposes). Per RFC 5280, Section 4.1.1.2, these two
> instances must match completely. In ARIN’s manifest, one contained an
> empty string (“”) as a parameter and the other contained a NULL
> (pointer to nothing). The empty string parameter was incorrect and the
> OpenSSL-based validators were flagging this because the two
> definitions of AlgorithmIdentifier did not match.

[ off topic note: from what I understand, the technical reason these
identifiers MUST match is to make certain X.509 fingerprinting
operations possible. If both identifiers had the (required) parameter
field omitted the situation would've been different. ]

> Planned Corrective actions:
> 
> As a corrective action, ARIN will be broadening its testing strategy.
> In future releases, we will be validating not only LibreSSL-based
> validators (RIPE’s Validator and NLNetlab’s Routinator) but also
> OpenSSL-based validators such as rpki-client and Fort. The list of
> validators we do test against the ARIN repository will be noted within
> the RPKI section of ARIN’s website.

This is a good and sensible plan.

Note: Neither RIPE's Validator and routinator are not based on LibreSSL
or OpenSSL. RIPE uses a cryptographic library called "Bouncy Castle" and
routinator appears (to in part?) rely on library bindings from the rust
ecosystem (which under the hood may link LibreSSL or OpenSSL), but also
in part seem to attempt to build their own crypto. LibreSSL and OpenSSL
both reject X.509 data with this type of encoding issue, and
consequently any RPKI validator implementations linked against these
libraries will automatically consider the algorithmidentifier mismatch
an error.

It is key to ensure not only a diverse set of validator implementations
is used for testing, but also care is taken to ensure a diverse set of
cryptographic libraries are put through the test. It was not rpki-client
or FORT rejecting the data, it was libressl and openssl rejecting the
data.

FORT and rpki-client have committed to a strict (safe) interpretation of
how cryptographic data should be handled and what those design choices
mean for internet operations in the global routing table. If either of
these implementations are used to confirm the correct operating of the
ARIN Trust Anchor (in cojunction with less strict implementations), one
will not only find the lowest common denominator, but in doing also
adhere to the highest standards.
 
Kind regards,

Job