[Sidrops] Re: WG Last Call for draft-ietf-sidrops-publication-server-bcp

[Tim Bruijnzeels <[e-mail]>]

Hi Chris, all,

Thank you for your comments! Late comments are better than no comments. This is at
least partly why we asked for Last Call, becuase this tends to motivate people ;)

In any case, a quick update:

- We incorporated Job's comments and made him co-author (he contributed significant
  text earlier in the doc lifecycle as well)
- We incorporated minor improvements in wording by Mike Hollyman

We currently have a PR for the next document revision here:
https://github.com/timbru/draft-sidrops-publication-server-bcp/pull/33

Of course, that is just transparency. Feel free to comment here on the list. If you
comment on github we will make sure that the list is informed.

In any case, the current edit buffer does not yet include your comments Chris, but
we plan to work on this over the coming days.

My proposal to the chairs would be do a new last call on -03 when it's ready, and
addresses Chris's comments.

Tim




> On 23 May 2025, at 19:35, Christopher Morrow <[e-mail]> wrote:
> 
> owdy sidorops and authors of the subject document... I probably should
> have made 1-2 of these next comments sooner :(
> (incidentally I think the document will be useful!)
> 
> I can also attempt to make a  github PR similar to job's once done
> typing here.. I think there should be a bit more information /
> guidance
> provided to the operators of Publication Services around availability,
> I think there are scenarios today where publication servers:
>   * live behind a single ISP
>       - limited redundancy/reslience to routing anomalies in that
> provider (or their peers)
>   * are deployed in a single geographical region (same town)
>      - same power grid
>      - same natural-disaster zone
>   * live behind smaller than expected links
>      - what is expected? 1 Gbps? 10Gbps?
>      - link size vs utilization?
>      - capacity planning concerns exist here - "Can service N
> concurrent request streams?"
>   * are a single entity
>      - pub service is a single computer/server
>      - inability to scale efficiently as demand rises
>      - in ability to split services across domains (geographical /
> network / power / etc)
>      - prone to hardware enforced outages
> 
> I think the idea that the RPKI system of publication services can be a
> loose group of things sprinkled across the internet without a ton
> of care about resilience of the system, because overall the relying
> parties can always call back later if a particular Publication Service
> is not currently available, has helped growth, experience, and
> experimentation in the system. I don't think the system works well
> long
> term ini this model, because more users and more tight coupling to the
> business of packet forwarding will require closer observation
> of the RPKI system's artifacts. I think this will drive more thought
> and effort into resiliency and redundancy, this document seems like
> a good place to start that conversation. Some has been done in
> RFC6481[0] for sure, but a slightly more structured and comprehensive
> discussion would help Publication Service operators align with network
> operations on requirements/expectations.
> 
> If we look at the guidance in RFC6481 section 3 there is one pointer to:
> " *  The publication repository SHOULD be hosted on a highly
>         available service and high-capacity publication platform."
> 
> This is a nice start, but does not provide guidance and reasoning for
> that guidance, really.
> I think being a bit more explicit about what that means and why it's
> necessary would be good.
> I think it's reasonable to build a simple model and drive the advice
> from that, something like:
> 
> * 20k ASN participating in the RPKI system
> * An hourly update cadence for Relying Parties
> * Average repository size of 1MBytes
> 
> Missing repository information because of outages between the relying
> party and the publication service MAY not be critical
> immediately, but will lead to delays in the system and possibly
> imapcts on the relying parties as they force age out content
> and suffer delays in their workload collecting from the globally
> distributed system. One view is that the delays are expected
> because this is the internet and over time such outages will occur, so
> we should be resilient in the total system to this, I think
> this is the prevailing sentiment today... Some study about lengthy
> outages and impact to the RPKI  and routing systems would
> be super informative, I bet.
> 
> Using wikipedia's 'how much outage is too much' page[1]  and the below
> information might be handy in deciding how much
> time can elapse between a Publication Service going offline for a
> percentage of the Internet and when flags should go up.
> As a starting stake in the ground I'd suggest 99.9 uptime seems like a
> great start?
> 
> When it comes to discussing capabilities and expectations about the
> Publication Service itself, It would be expected for a
> Publication Service to be able to service:
>  * 20000/3600 == 6 full request sets per second
>     - every ASN has 1 RP talking to every Publication Service every hour
>     - of course this will probably be at least 2x larger, so round to
> 10x for growth and ease of math
>     - assume spherical cow request/repository can accomplish all
> tasks in 1 connection (and in that 1 second)
>     - 60 requests * 1MB each second
>     - produces about 500 mbps sustained traffic for this request rate
>  * It's possible that RFC9286[2] expectations on nextUpdate existing
> may cause the relying party clients behave
>    differently, but that remains to be seen.
> 
> I think the above would guide a discussion on the point raised in
> RFC6481, I think adding to that some discussion
> about resilience and redundancy (which RFC6481 attempts to talk about
> a bit in Section 2.1 and point 5 of section 3)
> and how that could improve the ability of the Publication Service to
> be available would be helpful. I expect that guidance such
> as:
>  * maintain at least 2 transit paths to the wider Internet for each deployment
>    - plan for outages on the local and immediate provider network systems
>    - plan for outages to your networking systems as maintenance
> requirements arise
>  * plan more than one Publication Service deployment, in more than
> one geographically distinct region(s)
>    - remove local power/natural-disaster scenarios as much as
> possible from outage causes
>    - provide for disaster recovery  capabilities
>  * move the Publication Service closer to the core of the network if possible
>    - shorten the connections between relying parties and the service
> 
> Along with the last set of points would be some guidance about how to
> (again RFC6481 points a bit at this in section 3) to
> provide a coherent view of the Publication Service data across
> multiple relying party accessible endpoints. This should not
> be hard to accomplish, but certainly some thought and guidance would
> be handy to help folks move toward that world.
> 
> I think a review of historic outages to Publication Services would
> also show where each of the above points would have
> proven useful to the wider community of relying parties and
> Publication Services.
> 
> Sorry, this set of comment(s) got longer than I expected :(
> 
> -chris
> 
> 0: https://www.rfc-editor.org/rfc/rfc6481.html#page-8 - 'Resource
> Certificate Publication Repository Considerations'
> 1: https://en.wikipedia.org/wiki/High_availability - How much downtime
> to tolerate?
> 2: https://www.rfc-editor.org/rfc/rfc9286.html#name-manifest -
> next-update definition in Manifest
> 
> On Wed, May 21, 2025 at 6:07 AM Job Snijders <[e-mail]> wrote:
>> 
>> Dear all,
>> 
>> I am _in favor_ of publishing this publication BCP (no pun intended ;-)
>> but would like to see a few changes in order to help future RPKI
>> operators.
>> 
>> I've submitted the proposed changes through a github pull-request to
>> save the author team a bit of time.
>> https://github.com/timbru/draft-sidrops-publication-server-bcp/pull/31
>> 
>> My comments so far:
>> 
>> 1/ The Glossary can perhaps be expanded a bit: at the moment it makes
>> mention of some entities but not all entities that play a role. Maybe
>> add "Publication Point Operator" and "Relying Party", because
>> "Publisher" already is there?
>> 
>> 2/ As for the word "repository"... we are approaching the point where
>> everything is called a repository. I suggest replacing the phrase "RRDP
>> Repository" with "RRDP Server" and "Rsync Repository" with "Rsync
>> Server".
>> 
>> It is the RRDP _server_ and Rsync _server_ host one or more
>> 'caRepository'.  Calling the RRDP server endpoint a repository (of
>> repositories...) may increase confusion.
>> 
>> 3/ Add to glossary: "rsyncd - Rsync daemon"
>> 
>> 4/ "Because RPs use cached data" ... this is not a given, not all
>> RP implementations have implemented the concept of a 'validated cache'
>> (yet).
>> 
>> I suggested different wording:
>> 
>>        RPs are expected to make use of cached data from a previous,
>>        successful fetch (Section 6 of @!RFC9286). Therefor, short
>>        outages on the server side don't need to be cause for immediate
>>        concern, provided the server operator restores access
>>        availability in a timely fashion (e.g. before objects expire).
>> 
>> 5/ Section 4.1 sorta buried the lede. I'd like it if the first sentence
>> can strongly suggest to NOT RUN YOUR OWN. Add a sentence "In general, it
>> is NOT RECOMMENDED to run your own Publication Server, but to make use
>> of a publication server provided by the Issuer". Or, s/Issuer/RIR/.
>> This is a BCP document after all, no need to beat around the bush.
>> 
>> 6/ "... RPs rejecting CA publication points" I don't think RPs reject
>> the publication point, RPs simply stop considering connecting to a
>> publication point because the referal became invalid. Perhaps the
>> paragraph can be reworded.
>> 
>>        """
>>        If the Publication Server is unavailable for some reason, this
>>        will prevent Publishers from making updated RPKI objects
>>        available. The most immediate impact of this is that the
>>        publisher cannot distribute new issuances or revocations of
>>        ROAs, ASPAs or BGPSec Router Certificates for the duration of
>>        this outage. Thus, in effect, it cannot signal changes in its
>>        routing operations. If the outage persists for an extended
>>        period, then RPKI Manifests, CRLs, and Signed Objects might
>>        became stale, hampering, for example, BGP Origin Validation
>>        ([@!RFC6811].
>>        """
>> 7/ "content regression has occured due to data loss" maybe just needs to
>> be "content regression has occured"
>> 
>> 8/ Section 4.4 "The ROAs may not reflect what the publisher intended."
>>   perhaps reword to:
>>   "The published ROAs no longer reflect the CA's intentions." ?
>> 
>> 9/ The paragraph with "The publisher may not renew their manifest or CRL
>>   in time" contains a duplication (expiry == stale). Suggestion:
>>   """
>>   - The CA might not re-issue their Manifest or CRL in time, because
>>     they operated under the assumption that the currently published
>>     Manifest and CRL have not yet became stale.
>>   """
>> 
>> 10/ Some commas are missing here: "In addition to this the publishing CA
>> MAY perform ..." I've submitted suggested rewording through a pull
>> request.
>> 
>> 11/ ref update is needed, [I-D.ietf-sidrops-rrdp-same-origin]
>> 
>> 12/ "publishing in a repository" -> "publishing through a server" ?
>> 
>> ---------
>> 
>> I'll review the second half of the document soon!
>> 
>> Thank you for your work on this document.
>> 
>> Kind regards,
>> 
>> Job
>> 
>> 
>> On Wed, May 21, 2025 at 10:01:14AM +0200, Luigi Iannone wrote:
>>> Hi All,
>>> 
>>> the authors of draft-ietf-sidrops-publication-server-bcp requested WG Last Call as they consider that the document is ready to be published.
>>> 
>>> Abstract:
>>>  This document describes best current practices for operating an RFC
>>>   8181 RPKI Publication Server and its rsync (RFC 5781) and RRDP (RFC
>>>   8182) public repositories.
>>> 
>>> Details can be found at: https://datatracker.ietf.org/doc/draft-ietf-sidrops-publication-server-bcp/
>>> 
>>> This email formally opens the two weeks WG Last Call, to be ended June 4, 2025.
>>> 
>>> Please review this WG document and let the WG know if you agree that it is ready to be handed over to the AD.
>>> 
>>> If you have objections, please state your reasons why, and explain what it would take to address your concerns.
>>> 
>>> Thanks in adavance for your feedback.
>>> 
>>> Luigi & Russ
>> 
>>> _______________________________________________
>>> Sidrops mailing list -- [e-mail]
>>> To unsubscribe send an email to [e-mail]
>> 
>> _______________________________________________
>> Sidrops mailing list -- [e-mail]
>> To unsubscribe send an email to [e-mail]
> 
> _______________________________________________
> Sidrops mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]