[Sidrops] Re: draft-li-sidrops-rpki-repository-problem-statement-00

[Job Snijders <[e-mail]>]

Dear Yingying Su, authors, working group,

I find that I disagree with much of this internet-draft's
characterisation of 'problems' in the space of RPKI deployment.

I've processed the internet-draft through 'cat -n' to add line numbers,
and will reply to parts of the internet-draft where I perceive factual
issues which in turn undermine conclusion presented in the I-D.

   173	   We have observed
   174	   that the current RPKI Repository lacks the ability to provide
   175	   reliable services to RPs.  RPKI allows certificates issued by a CA to
   176	   be stored only at the PP operated by that CA, which means that any
   177	   failure of a PP can hinder RPs from obtaining complete RPKI data.

The publication point need not be operated by the CA. For example, all
CAs subordinate to NIC.BR publish into NIC.BR's publication service. The
subordinate SAs do not operate NIC.BR's publication point.

   178	   Then, as delegated RPKI becomes more popular, an increasing number of
   179	   CAs choose to maintain their own repository instances.

I fail to see evidence for this assertion. As far as I understand the
'hosted RPKI' approach by far is the most popular choice with no signs
this approach's popularity is declining.

Today more than half of the Internet's global routing table entries are
RPKI-valid and estimates by Kentik suggest that between 70% and 80% of
Internet traffic is forwarded towards IP destinations covered by RPKI
ROAs.

At the moment of writing there are 44 FQDNs associated with non-RIR
operated publication services and 14 FQDNs associated with NIR or RIR
operated publication services.

The RIR operated publication services contain 222,934 ROA objects; the
non-RIR publication services contain 42,578 ROA objects. ~ 75% of the
ROAs in the non-RIR publication services are issued and served by
National Internet Registries (NIRs). This means that only 10,691 ROAs
(4% of the global total) reside in non-RIR/non-NIR publication services.
Of that 4%, about 84% of ROAs are issued by a single entity (a large
cloud provider). Accounting for all of the above and excluding the one
cloud provider, only 1,620 ROAs (0.6%) reside in non-RIR/non-NIR
publications.

None of these numbers suggest that there is increasing popularity
towards delegated 'self-hosted' publication services. In fact, we can
see that currently only a tiny tiny minority of operators choose to run
their own publication services, with some taking this approach
exclusively for experimentation/development/analytical purposes.

   179     However, some
   180	   of these repositories lack robust, secure, and reliable
   181	   infrastructure.

Sure, but this is a problem for those publication operators and for the
CAs who choose to publish into such unreliable publication services. CAs
ought to apply a vendor selection process when it comes to RPKI
publication services. Failure to select a vendor with a robust secure
reliable publication service is not a problem of the RPKI, it is the CAs
own responsibility.

   181     Furthermore, the proliferation of repositories will
   182	   affect the scalability of RPKI, as RPs need to traverse all PPs to
   183	   refresh their local caches.

I'd argue there is no meaningfull proliferation of publication services
given the overwhelming popularity of RIR/NIR operated publication
services (based on the number of ROA objects available via NIR/RIR
operated publication servers).

   205	2.  Reliability Analysis
   206	
   207	   Although the current RPKI Repository is globally distributed, each CA
   208	   only stores the RPKI objects it issues in the unique PP it runs.  If
   209	   any PP fails (malfunctions or is attacked), RPs will not be able to
   210	   fetch the RPKI objects issued by its CA, and the integrity of the
   211	   global RPKI object view cannot be guaranteed.

RFC 7115 correctly notes in section 6:

    """
    Like the DNS, the global RPKI presents only a loosely consistent
    view, depending on timing, updating, fetching, etc.  Thus, one
    cache or router may have different data about a particular prefix
    than another cache or router.  There is no 'fix' for this, it is
    the nature of distributed data with distributed caches.
    """

There never was any guarantee for "integrity of the global RPKI object
view" to begin with, this simply is part of the RPKI's design. Each
Relying Party can cryptographically verify the objects it managed to
download, and each distinct caRepository contained within a publication
service can accuractly be checked for integrity using RPKI Manifests
(RFC 9286), additionally Relying Parties can compare purported new and
locally cached versions of objects to overcome malfunction or tampering.

Thus, I see no value in a problem statement which boils down to "if the
RP cannot download the file, it cannot cannot access the file's content"
- which would be a truism, and ignores various resilliency techniques
that RPs have implemented.

   213	   As of August, 2024, there are 42,950 RCs and 303,918 ROAs in RPKI
   214	   Repository. 

I checked this August 31st, 2024 tarball produced as part of the rpkiviews.org project:
http://josephine.sobornost.net/josephine.sobornost.net/rpkidata/2024/08/31/rpki-20240831T233656Z.tgz

This archive contains 261,446 ROA objects and 43,507 Resource
Certificates. It is unclear to me how the authors saw an additional
42,472 ROA objects compared to my own datasets. Is there a difference in
methodology? For example, are invalid (expired) ROAs are also included?

I calculated the low, high, median, and mean number of valid ROA objects
as seen by validator instances in Frankfurt and Amsterdam throughout
August 2024:

                Amsterdam (NL)     Frankfurt (DE)
Minimum:           256,819             256,826
Maximum:           261,590             261,592
Median:            259,133             259,171
Mean:              259,139             259,126
Std Deviation:       1,467               1,484

Throughout August 2024 we saw an increase of about 5,000 extra ROA
objects, bringing us to a total of ~ 261,500.

   214     For each RC, we extracts the SIA field that records the
   215	   URI of its HTTPS-based RRDP file (Since all PPs now support RRDP and
   216	   serve RPs through RRDP files, the analysis of the RPKI Repository
   217	   focuses primarily on RRDP Repository).  The result shows that there
   218	   are currently 63 independent repository instances (SIA fields of RCs
   219	   held by the same entity may share the same PP, therefore, the number
   220	   of repositories is much lower than that of RCs).

Thank goodness :-)

   252	   Measurement results show that although RRDP enables the use of
   253	   Content Distribution Networks (CDNs) infrastructure for resilient
   254	   service, only 9 out of 63 repositories are hosted in CDNs, with 8
   255	   being hosted on Cloudflare AS13335 and 1 on Amazon AS16509.  It also
   256	   shows that out of 63 repositories, 60 of them are hosted in a single
   257	   AS.

63 minus 9 minus 1 is 53. How can 60 be hosed in a single AS? Which AS
is this? Is there a calculation error? The above observation does not
match with my own datasets.

   258     It means that the accessibility of these repositories is highly
   258	   dependent on the reachability of a single AS.

If the previous paragraph contained a measurement error, the above
conclusion does not hold true.

   258     Worse still, among the
   259	   repositories whose corresponding ASes have deployed ROAs, there are
   260	   14 of them carry the ROA of the ASes in which they are located.  It
   261	   means that once the repository goes down and the RPs cannot fetch its
   262	   ROAs, the route of the AS that the repository locates in may be
   263	   downgraded by ROV adopters.  Then, even if the repository is
   264	   restored, those ROV adopters still cannot access the repository, in
   265	   which case the access of the repository depends on the reachability
   266	   of the AS it locates in, while the reachability of the AS also
   267	   depends on the access of the repository.

Yes, this was already noted in RFC 6483 section 3:

   """
   Potential circular dependence is a consideration here: if the
   authoritative publication point of the repository of ROAs, or that of
   any certificate used in relation to an address prefix, is located at
   an address that lies within the address prefix described in a ROA,
   then the repository can only be accessed by the RP once a route for
   the prefix has been accepted by the RP's local routing domain.
   """

Publication service operators will need to take care of potential for
circular dependices.

   269	   Real-world incidents of repository breakdowns do occur at times.  On
   270	   6 April 2020, the repository maintained by RIPE NCC suffered a sudden
   271	   increase in connection to service [RIPE-downtime], resulting in it
   272	   appearing as down to many RPs for 7 hours (RIPE's services are
   273	   already hosted on CDNs).  On 15 May 2020, the Japan operated
   274	   repository was out of service for 10 hours due to hardware failure
   275	   [service-outage], and between 26 Jan 2022 and 2 Feb 2022, due to full
   276	   disk space, all ROAs in its repository again became invalid
   285	   [disk-outage].  During 10 July 2024 to 12 July 2024, we also found
   286	   that RPKI data synchronized through Routinator [routinator] once
   287	   again had missing parts from RIPE NCC.

Yes, problems occur in the real world. Then again, temporal inability to
synchronize with a given publication service does not immediately render
previously fetched & cached objects invalid. As per RFC 9286, Relying
Party implementation should use locally cached objects until such time
they become stale. In practise this means there is time budget for
publication point outages.

   289	3.  Scalability Analysis
   290	
   291	   In current RPKI infrastructure, refreshing the local cache of each RP
   292	   involves traversing all repositories to fetch the updated RPKI
   293	   objects.  However, RPKI Repository has grown from 5 repositories run
   294	   by five RIRs to more than 60 repositories and is expected to increase
   295	   dramatically with the further deployment of ROA.

I challenge the assertion there will be a dramatic increase in
non-RIR/non-NIR repositories.

   297	   On the one hand, with a deeper understanding of RPKI, ROA deployers
   298	   prefer to adopt delegated RPKI to flexibly control their RPKI
   299	   objects.  In the survey, 45.3% of the AS administrators using hosted
   300	   RPKI say they have plans to run their own repositories for flexible
   301	   certificate signing and control.

Perhaps the survey inadvertently contained leading questions? Almost
everyone I've spoken about RPKI ROA deployment strongly prefers to
outsource the issuance of ROAs and subsequent hosting of ROA objects to
the RIR they already are paying for those services. Very few companies
want to take on additional responsibility for zero commercial gain. The
numbers from earlier in this email also do not corroborate that 'self
hosting' is popular or going to be popular.

   301     The result shows that delegated
   302	   RPKI will emerge as a trend, and the number of independent
   303	   repositories will inevitably increase.  The work [beyond-limits]
   304	   predicts that when ROA is fully deployed, the number of repositories
   305	   will reach 10K and there will be 140K active RPs, the RPKI snapshot
   306	   download would easily exceed 1 hour (on the premise that each
   307	   repository is well accessible).

I predict the number of publication services will not reach 10K when
fully deployed, I base this on the simple observation that the Internet
operator community now already is above 50% routing table coverage, and
there only are ~ 44 FQDNs associated with non-RIR/non-NIR operations.

   307     Since RPs are required to check the
   308	   updates of all repositories when refreshing their local caches, the
   309	   increasing number of repositories will result in higher refresh
   310	   costs.  It may prevent RPs from obtaining routing information in a
   311	   timely manner, and also prohibits the release of use cases such as
   312	   temporary ROAs to enable ISPs to perform tactical traffic engineering
   313	   to ease congestion.

I challenge the assertion that short-lived ROAs cannot be used for
tactical traffic-engineering. Keep in mind that multiple Prefix entries
can be aggregated into a single ROA, in doing so substantially reducing
the number of bytes that have to be transferred from publication service
to Relying Party.

   315	   On the other hand, RPKI Repository is designed without a strict
   316	   admission mechanism, allowing entities to apply for RC from RIRs or
   317	   other RPKI CA entities, register as delegated CAs, and then operate
   318	   repositories with a small fee and identity verification.  Therefore,
   319	   some repositories for unwanted purposes (measurement, attack
   320	   experiments, etc.) are gradually emerging.  Furthermore, the low
   321	   barriers to running repositories and joining RPKI Repository will
   322	   introduce unforeseen risks to RPs.  For example, a malicious CA can
   323	   create a large number of descendant RCs and operate numerous
   324	   repositories to make RPs endlessly retrieve repositories, thus
   325	   exhausting and paralyzing RPs.

Such CAs would violate RIR terms & conditions and be hampered by various
protection mechanisms embedded in modern day Relying Party
implementations. Furthermore in the RPKI attackers need to take note of
attribution due to the non-repudial nature of the signature chains.
Deterrents against malicious actions exist in the form of reactive legal
action.

   325     Although the most popular RP
   326	   software, Routinator [routinator], has set the default value for the
   327	   maximum searchable RPKI-tree depth to 32 in its latest version
   328	   0.14.1, and the RPKI-client has set the default value to 12, neither
   329	   has provided a value for RPKI-tree width (as it is difficult to set a
   330	   reasonable threshold).

Both Routinator and rpki-client have implicit width limits. Whether
those limits are useful or not is a separate conversation, but the claim
'there are no width limits' does not hold true.

   330     Similarly, it is also challenging for RP
   331	   softwares to clearly limit the maximum number of accessible
   332	   repositories.  In addition, a series of recent academic works have
   341	   emerged that exploit repository vulnerabilities to cause repository
   342	   downtime, thereby preventing it from providing normal services to RPs
   343	   [behind-the-scenes].  There are also works [beyond-limits][stalloris]
   344	   that manipulate malicious repositories to attack RPs, thus
   345	   obstructing the synchronization of RPKI data.

The papers referenced here are not entirely without controversy.

   347	   In addition, with the increasing rate of ROV deployment, more and
   348	   more RPs will connect to RPKI Repository.  It will undoubtedly
   349	   increase the burden on the repositories.  In summary, as RPKI
   350	   deployment progresses further, RPs will need to access more
   351	   repositories, and repositories will need to serve more RPs.  This
   352	   increase in the number of bidirectional connections will threaten the
   353	   scalability of the RPKI system.

Yes, more RPs == more data. Tending to the RPKI is like tending to a
garden, active maintenance is required. Recent examples to improve
future scalability include the implementation & publication of RFC 9589
(optimizing RRDP -> Rsync switchovers), or for example the
implementation of support for HTTP content encoding compression in Fort
and Rpki-client.

   355	4.  Security Analysis
   356	
   357	   In current RPKI infrastructure, RPKI authority (CA and its
   358	   corresponding repository manager) uploads the RCs and ROAs it signs
   359	   for subordinate INR holders to the repository it specified.  It means
   360	   that RPKI authorities control the signing and management of RPKI
   361	   objects in their repositories and have significant unilateral power.
   362	   The disproportionate division of power between RPKI authorities and
   363	   INR holders will poses three issues:

Yes, this is the nature of subordination.

   365	   (1) Since the current RPKI Repository is not tamper resistant,
   366	   authorities can unilaterally undermine any RPKI objects without
   367	   consent from subordinate INR holders.  Malicious or breached
   368	   authorities can perform arbitrary operation on INR holders'
   369	   certificates, such as deletion, corruption, modification, or
   370	   revocation (see [RFC8211] for more details).  These operations often
   371	   involve diminishing the set of INRs associated with the affected INR
   372	   holders.  A malicious RPKI authority can also compromise RPs, for
   373	   example, showing incomplete or inaccurate RPKI object views to RPs.

Manifests protect individual caRepositories by informing Relying Parties
what the state of the caRepository ought to be. Also, CAs themselves
choose which publication service they want to use. RFC 8211 enumerates a
number of detrimental actions in order to inform the broader community
of 'what could go wrong', repeating the keypoints does not make for a
new problem statement.

   375	   (2) INR holders and RPs can only rely on RPKI authorities without the
   376	   ability to verify that their security requirements are being met.
   377	   Specifically, INR holders cannot know if their RPKI objects are
   378	   securely stored in the authorities' repositories and can be publicly
   379	   seen and fetched by all RPs; RPs cannot verify the integrity and
   380	   accuracy of the RPKI objects they synchronize from RPKI Repository.

INR holders are able to monitor the contents of publication services.
Indeed INR holders cannot verify whether all RPs fetched & validated a
given object; then again, website operators also cannot know whether all
people on the planet can access their website. Various monitoring
services exist to resolve these types of issues both for website
operators and INR holders.

   397	   (3) The history of RPKI objects is not easily auditable.  A complete
   398	   longitudinal view of RPKI objects is necessary to defend against
   399	   malicious manipulation of the RPKI Repository and to resolve
   400	   conflicts between authorities and INR holders.  Although RPs can
   401	   periodically fetch RPKI objects to keep track of their historical
   402	   versions, it is costly, without any guarantee of completeness.  Since
   403	   RPKI lacks a trustworthy historical RPKI object record, it is also
   404	   challenging, if not impossible, to hold authorities accountable even
   405	   if attacks are detected after the fact.

The above simply is not true. Already today various projects exist for
longitudinal view of RPKI objects:

https://ftp.ripe.net/rpki/
http://www.rpkiviews.org/
https://delta-rpkilines.nlnetlabs.net/

Using the public keys contained within the RIR TALs and timestamps
provided in the above archives, one can reconstruct with great accuracy
what objects existed at what times. While there is no guarantee of
completeness, the number of data collectors (at least 10 nodes globally
distributed that I know of) makes it challenging to reliably hide
manipulation within a global publication scope.

   407	   In the survey, 44.1% of the AS administrators explicitly expressed
   408	   concerns about malicious RPKI authorities (11.5% of them chose "not
   409	   sure").  Two administrators provided additional feedback, indicating
   410	   that they consider the threat from the RPKI authorities to be the
   411	   most serious problem.  One of them said they had lost all their ROAs
   412	   due to administrative/human reasons.  It can be seen that threats
   413	   from RPKI authorities are also widely recognized in the industry.

Selecting one or more Trust Anchor Locators (as Relying Party) for use
is the RP assuming trust in the Trust Anchor operator, this means that
violation of that trust is considered out of scope. The rest of the
trust arcs are 'derived trust', calculated using subjective logic.
A subset of INR holders are in the position to transfer their INR
holdings from one RIR to another RIR should there be significant
concerns about operational threats. Failure to pay bill for rendered
services might result in revocation of ROAs, such is life when
purchasing services.

   415	5.  Summary: Problems of the current RPKI Repository
   416	
   417	   Based on the measurement and analysis described in the previous
   418	   section, this section summarizes the main problems of the current
   419	   RPKI Repository and the key reason behind these issues: the binding
   420	   of the repository PP with the CA.

The previous section contains a number of potential issues, so I am
hesitant to accept what follows.

   422	5.1.  P1: Every RPKI object is a singleton in RPKI Repository.
   423	
   424	   Although the current RPKI Repository is globally distributed, RPKI
   425	   does not provide distributed storage for RPKI objects.  The binding
   426	   of PP and CA causes each RPKI objects to be stored only in the PP run
   427	   by the CA that issued the object, instead of being stored in multiple
   428	   repository nodes.  Therefore, the current RPKI Repository cannot
   429	   guarantee that RP can obtain complete RPKI data when some repository
   430	   nodes fail.  Even worse, the singleton nature of RPKI objects also
   431	   introduces unwanted interdependence between the accessibility of a
   432	   CA’s PP and the reachability of the AS that the PP locates.  Since
   433	   RPKI ensures the routing security and reachability of ASes, it is
   434	   fair to expect PPs to remain accessible during any incident when
   435	   corresponding ASes become unreachable.

The definition of 'singleton' is missing in this context. In the
existing technology stack it is perfectly possible to store RPKI objects
in multiple places. One example is that objects are accessible via both
RRDP and RSYNC. Publication service operators can use DNS to provide
multiple A & AAAA records to clients which point to multiple nodes
hosting copies of the objects in hand. Publication operators can also
leverage BGP anycasting to introduce multiple copies distributed across
multiple nodes.

Furthermore it is technically possible for RIRs to provide Certification
services to out-of-region INR holders. The fact that at present day none
of the RIR offers such a service is a political problem, not a technical
one.

   437	5.2.  P2: RPKI Repository is costly in RP refreshing.
   438	
   439	   Refreshing the local cache of each RP involves traversing all
   440	   repositories to fetch the updated RPKI objects.  However, the binding
   441	   of the repository PP to the CA causes the number of repository
   442	   instances to increase dramatically with the further deployment of ROA
   443	   (as the number of CAs that hold RCs increase).  Furthermore, as AS
   444	   administrators gain a deeper understanding of RPKI technology,
   453	   delegated RPKI has emerged as a trend, CAs are more willing to
   454	   maintain the repository PP themselves to achieve more flexible
   455	   certificate signing and management.  The growth in the number of
   456	   repository instances increases the cost of RP refreshes, the growth
   457	   in the number of RPs brings burdens to repositories, and both will
   458	   threaten the scalability of RPKI.

Delegated RPKI has not emerged as a trend.

   460	5.3.  P3: Unilateral reliance on authority.
   461	
   462	   The current RPKI authority (CA) not only operates a certificate
   463	   engine but also operates a publication repository to store all RPKI
   464	   objects it signs.  It means that although the RPKI authority issues
   465	   certificates to subordinate INR holders, the management of these
   466	   certificates (especially their storage and distribution) is entirely
   467	   controlled by the RPKI authority.  The current RPKI Repository
   468	   architecture exacerbates the power imbalance between the RPKI CA and
   469	   subordinate INR holders and may also lead to RPs being deceived by
   470	   receiving inaccurate or incomplete RPKI data from malicious RPKI
   471	   authorities.  However, INR holders and RPs have no choice but to rely
   472	   unilaterally on the RPKI authority.  This unilateral reliance
   473	   prevents RPKI from proactively defending against the threats from
   474	   malicious RPKI authorities.  Moreover, it is also challenging to
   475	   provide a complete and trustworthy longitudinal view of RPKI objects
   476	   for post-incident audits, as the RPKI data is entirely controlled by
   477	   the CA that signed it.

This is the nature of subordination.

   497	7.  Security Considerations
   498	
   499	   TBD

While the internet-draft notes that though delegation a chain of CAs
could introduce a substantial number of publication service endpoints;
what's omitted from the internet-draft is that various Relying Party
implementations have developed mitigations to address the risk of
misbehaving publication service endpoints.

For example, rpki-client imposes a size limit and a time limit on
per-endpoint synchronization operations. A global repository count limit
also exists. Repositories are contacted in random order, noting that in
practise the RIR operated publication endpoints are placed towards the
front of the queue (as the TA certificates point via the SIA point to
RIR operated endpoints). If an endpoint ends up not being queued for
synchronization, locally cached objects will be used, and perhaps in the
next process execution it will be synchronized.

All in all, the above means that the further 'away' from the root, the
less reliable global visibility becomes. This decrease in reliable
accessibility applies to both legit and malicious endpoints. This is a
healthy property of the architecture. This property encourages operators
to use the RIR provided services rather than solving things in-house.
This in turn reduces the number of endpoints to synchronize to, and
reduces bandwidth consumption.

The longer a chain becomes, the more fragile it becomes (be it because
of intermediate Certification Authorities or distinct intermediate
publication endpoints). Often times overly long dependency chains are
the result of choices the INR holder themselves made, self-inflicted
risk so to speak. Yes, the RPKI technology stack has a fair bit of rope
for people to hang themselves. This probably is true for most heavy
machinery. The solution resides in outreach and education "how to
optimally configure ROAs, where to host them"; and IMHO less so in
adoption an imperfect problem statement.

If an INR holder wants to maximally increase the chances of all RPs
globally fetching & validating their ROAs, then they should simply use
their RIR's certification & publication services!

Kind regards,

Job