Re: [Sidrops] ARIN RPKI Service Impact - 12 August 2020 - manifest issue - resolved

Christopher Morrow <> Fri, 14 August 2020 02:31 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id B59063A0C27 for <>; Thu, 13 Aug 2020 19:31:34 -0700 (PDT)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.098
X-Spam-Status: No, score=-2.098 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id j1Cs9qow5Hkk for <>; Thu, 13 Aug 2020 19:31:33 -0700 (PDT)
Received: from ( [IPv6:2607:f8b0:4864:20::f44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 524773A0C24 for <>; Thu, 13 Aug 2020 19:31:33 -0700 (PDT)
Received: by with SMTP id s15so3686427qvv.7 for <>; Thu, 13 Aug 2020 19:31:33 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=mA4BzYOtR1328Q/5aHnIuNhfFHz6mmTFnohNLEhZJ6Q=; b=IpvpxmTSvdUHyfISxIjDC6digT+T2STk99XLcqw98Xh/SnL1QGEOEq9PYXpR161VG5 DYwlODy9fTokg6VFbE7NyAInPr4ls17QBgQk+nEaEqFVLdh+Z4a9hxivZZSRt6YlvKG8 F7m6K8PFJkJs1vObf7/0g6qVlvIa73mqCek5HCXr2sLCVaAPbYzUUAwMlvuClQe80xGh mx/8Dq1+SykR8yeQxu4nRdWT5Jx6NF0sIaWA7cezk/Pxl/ujyzeuap3vnBlmq+c7oW1I MgAx/nfeCI7SsF8djLlgNZ9EFEAzha7z2A3Z/JoFJMlPdBYY6wX7bZCX6JjrOm2BzF7/ CHKw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=mA4BzYOtR1328Q/5aHnIuNhfFHz6mmTFnohNLEhZJ6Q=; b=pukM2dSJWO6cxOZS9dPkfB+3Znp22PJZlSpDI7oKbAj8lLsuqCTTgyCTRz63KMaH4P BNSy8akisQgpbRQ9XAn0cW+7BN/znkmCZSe+i7x4g95WqRpf36VgCu5qaDWvhqJgw26K 1IEn4bVRybPZJ+RhEQ7JisxUFE1JSGdfUPgWx37xpCb/REiAYu8GCyBLPVr13xj2KVGe OXm6wI58+Oya38i8G/R9sy4rGqJGxn0Q0QTL2A89B+bLAtoVd+p4SETjApT7kK1q5ZUW i/BnLDAc7h+yhghVLigSBhp20VCw0GABm810jyD1EiEOOtjlEV7PJpIkA4tnv1/XFz55 Qaqg==
X-Gm-Message-State: AOAM533j2ramq5lXXkTAPmEX4VM6ZtEQHldshqc4k2Nd1gG9fVQ/eJtd XsicnT8BK2rV/wGrd1LRMMiMeqGI4y81QRPCOyQ=
X-Google-Smtp-Source: ABdhPJyiY2gAAczL1cdjktjEofzhOtoZhIiGyYlvE+nD7yroZLrVLIJjfixV6yV5R9JmMksecx+R+itzIaIBBO/8LmU=
X-Received: by 2002:ad4:4dc5:: with SMTP id cw5mr812859qvb.238.1597372292276; Thu, 13 Aug 2020 19:31:32 -0700 (PDT)
MIME-Version: 1.0
References: <>
In-Reply-To: <>
From: Christopher Morrow <>
Date: Thu, 13 Aug 2020 22:31:21 -0400
Message-ID: <>
To: John Curran <>
Cc: "" <>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: <>
Subject: Re: [Sidrops] ARIN RPKI Service Impact - 12 August 2020 - manifest issue - resolved
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: A list for the SIDR Operations WG <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Fri, 14 Aug 2020 02:31:35 -0000

howdy john!
1st: thanks for the pointer/RFO/report... (again, learning from others
mistakes means maybe we'll have less pain going forward? :) )
2nd: thanks for the call out to community members who helped poke the
bear here and get to a solution! :)

On Thu, Aug 13, 2020 at 3:45 PM John Curran <> wrote:
> RPKI folks -
>         In the process of upgrading our HSM yesterday, ARIN updated its RPKI signing infrastructure incorrectly – the result was an encoding error in our manifest that caused rpki-client and FORT validators no longer consider ARIN’s RPKI data to be valid. (see attached service announcement)

Are there lessons learned here for the other validators and CA folk?
Are there test cases we can use in other CA deployments? (both RIR and


>         This has been since resolved and we’re in the process of reissuing ROAs created during the time.  We are not aware of any delegated repositories impacted during this period.
>         Our thanks to the OpenBSD team - Sebastian Benoit, Theo Buehler, Joel Sing, Job Snijders, and Claudio Jeker - who were instrumental in hunting down this issue.
>         I’ll provide a more detailed post-mortem here once available.
> My apologies for the service impact,
> /John
> John Curran
> President and CEO
> American Registry for Internet Numbers
> ===
> RPKI Service Notice Update
> Posted: Thursday, 13 August 2020
> Service Update
> After upgrading our HSM on Wednesday, August 12, 2020, Job Snijders reported to us that our RPKI repository was no longer validating using rpki-client or fort. Upon investigation, it was discovered that we had an encoding error in our new software. (Specifically, there was a mismatch in the “parameters” field between the “algorithm identifier” of the certificate and the certificate To Be Signed [TBS]. The TBS set the “parameters” as null and the certificate as empty.)
> A fix has been made and there will be a pending data clean up in the next few days to fix some of the ROAs created during the interim.
> We would like to thank Sebastian Benoit, Theo Buehler, Joel Sing, Job Snijders, and Claudio Jeker, from the OpenBSD project ( ), as they spent considerable time working with us to identify the root cause of the issue.
> Regards,
> Richard Jimmerson
> Chief Operating Officer
> American Registry for Internet Numbers (ARIN)
> ===
> _______________________________________________
> Sidrops mailing list