Re: [Sidrops] ARIN RPKI Service Impact - 12 August 2020 - manifest issue - resolved

[Christopher Morrow <[e-mail]>]

howdy john!
1st: thanks for the pointer/RFO/report... (again, learning from others
mistakes means maybe we'll have less pain going forward? :) )
2nd: thanks for the call out to community members who helped poke the
bear here and get to a solution! :)

On Thu, Aug 13, 2020 at 3:45 PM John Curran <[e-mail]> wrote:
>
> RPKI folks -
>
>         In the process of upgrading our HSM yesterday, ARIN updated its RPKI signing infrastructure incorrectly – the result was an encoding error in our manifest that caused rpki-client and FORT validators no longer consider ARIN’s RPKI data to be valid. (see attached service announcement)
>

Are there lessons learned here for the other validators and CA folk?
Are there test cases we can use in other CA deployments? (both RIR and
delegated)

-chris

>         This has been since resolved and we’re in the process of reissuing ROAs created during the time.  We are not aware of any delegated repositories impacted during this period.
>
>         Our thanks to the OpenBSD team - Sebastian Benoit, Theo Buehler, Joel Sing, Job Snijders, and Claudio Jeker - who were instrumental in hunting down this issue.
>
>         I’ll provide a more detailed post-mortem here once available.
>
> My apologies for the service impact,
> /John
>
> John Curran
> President and CEO
> American Registry for Internet Numbers
>
> ===  https://www.arin.net/announcements/20200813/
>
> RPKI Service Notice Update
>
> Posted: Thursday, 13 August 2020
> Service Update
>
> After upgrading our HSM on Wednesday, August 12, 2020, Job Snijders reported to us that our RPKI repository was no longer validating using rpki-client or fort. Upon investigation, it was discovered that we had an encoding error in our new software. (Specifically, there was a mismatch in the “parameters” field between the “algorithm identifier” of the certificate and the certificate To Be Signed [TBS]. The TBS set the “parameters” as null and the certificate as empty.)
>
> A fix has been made and there will be a pending data clean up in the next few days to fix some of the ROAs created during the interim.
>
> We would like to thank Sebastian Benoit, Theo Buehler, Joel Sing, Job Snijders, and Claudio Jeker, from the OpenBSD project (https://openbsd.org ), as they spent considerable time working with us to identify the root cause of the issue.
>
> Regards,
>
> Richard Jimmerson
> Chief Operating Officer
> American Registry for Internet Numbers (ARIN)
> ===
> _______________________________________________
> Sidrops mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/sidrops