Content-Type: text/plain;
	charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.600.51.1.1\))
From: Tim Bruijnzeels <tbruijnzeels@ripe.net>
In-Reply-To: 
 <CAL9jLaZDOdEcgYD3r6qYFq1yZwfzqioQ0Omik_MQp1e6PLmuOQ@mail.gmail.com>
Date: Tue, 24 Jun 2025 17:06:06 +0200
Content-Transfer-Encoding: quoted-printable
Message-Id: <52E73BD2-5D95-4AB4-AAA8-612D4B190188@ripe.net>
References: <74674002-1C2A-4F7F-90EF-423506421030@gigix.net>
 <aC2ly6SHgMpWYtxg@feather.sobornost.net>
 <CAL9jLaYaPsN4LdY3JFDe1t6z1hH+KGSews9iOvGpqdZMODr85A@mail.gmail.com>
 <F23C0707-5185-413E-8E41-7838E15D678B@ripe.net>
 <CAL9jLaZDOdEcgYD3r6qYFq1yZwfzqioQ0Omik_MQp1e6PLmuOQ@mail.gmail.com>
To: Christopher Morrow <christopher.morrow@gmail.com>
Message-ID-Hash: UVQUSBUFKPQ3IYOLAS42YARQWMCRMRNY
CC: sidrops@ietf.org
Precedence: list
Subject: =?utf-8?q?=5BSidrops=5D_Re=3A_WG_Last_Call_for_draft-ietf-sidrops-publicatio?=
 =?utf-8?q?n-server-bcp?=
Archived-At: 
 <https://mailarchive.ietf.org/arch/msg/sidrops/eIEVEa21yCUoXDBqBYgg86ZYPfY>

Hi Chris, all,

as you may have seen we just uploaded a version that includes the =
comments so far, except for Chris's comments:

=
https://datatracker.ietf.org/doc/draft-ietf-sidrops-publication-server-bcp=
/03/

Chris, could you be more specific as to what you would like to see in =
the document? Either through a PR on the main branch here:
https://github.com/timbru/draft-sidrops-publication-server-bcp

Or through email.

I will respond to your earlier comments below to get things rolling from =
our side.

Tim

>>=20
>>> On 23 May 2025, at 19:35, Christopher Morrow =
<christopher.morrow@gmail.com> wrote:
>>>=20
>>> owdy sidorops and authors of the subject document... I probably =
should
>>> have made 1-2 of these next comments sooner :(
>>> (incidentally I think the document will be useful!)
>>>=20
>>> I can also attempt to make a  github PR similar to job's once done
>>> typing here.. I think there should be a bit more information /
>>> guidance
>>> provided to the operators of Publication Services around =
availability,
>>> I think there are scenarios today where publication servers:
>>>  * live behind a single ISP
>>>      - limited redundancy/reslience to routing anomalies in that
>>> provider (or their peers)
>>>  * are deployed in a single geographical region (same town)
>>>     - same power grid
>>>     - same natural-disaster zone
>>>  * live behind smaller than expected links
>>>     - what is expected? 1 Gbps? 10Gbps?
>>>     - link size vs utilization?
>>>     - capacity planning concerns exist here - "Can service N
>>> concurrent request streams?"
>>>  * are a single entity
>>>     - pub service is a single computer/server
>>>     - inability to scale efficiently as demand rises
>>>     - in ability to split services across domains (geographical /
>>> network / power / etc)
>>>     - prone to hardware enforced outages

Good points in principle. Perhaps some of this could be explicitly
called out in section 4.3 (Publication Server) Availability and section
5.5 RRDP Availability.

I find it non-trivial to strike a balance between talking about the
mandatory requirements for availability using formal key words and =
providing
helpful considerations without requiring one specific way to implement
that meets the overall requirement if you see what I mean.

I am not opposed to naming some of these things. If you have a PR or
suggestion that would be much appreciated.


>>> I think the idea that the RPKI system of publication services can be =
a
>>> loose group of things sprinkled across the internet without a ton
>>> of care about resilience of the system, because overall the relying
>>> parties can always call back later if a particular Publication =
Service
>>> is not currently available, has helped growth, experience, and
>>> experimentation in the system. I don't think the system works well
>>> long
>>> term ini this model, because more users and more tight coupling to =
the
>>> business of packet forwarding will require closer observation
>>> of the RPKI system's artifacts. I think this will drive more thought
>>> and effort into resiliency and redundancy, this document seems like
>>> a good place to start that conversation. Some has been done in
>>> RFC6481[0] for sure, but a slightly more structured and =
comprehensive
>>> discussion would help Publication Service operators align with =
network
>>> operations on requirements/expectations.

Ok. First of all we RECOMMEND against a "Self Hosted Publication Server"
in section 4.1 and a number of other places in the draft.

It is not that centralisation of publication servers is the main reason
for this, but it's that running a server and repository properly is =
harder
than people think. And therefore using a well-maintained server/repo is
better for the RPKI ecosystem then running one yourself and running it
poorly.


>>>=20
>>> If we look at the guidance in RFC6481 section 3 there is one pointer =
to:
>>> " *  The publication repository SHOULD be hosted on a highly
>>>        available service and high-capacity publication platform."
>>>=20
>>> This is a nice start, but does not provide guidance and reasoning =
for
>>> that guidance, really.
>>> I think being a bit more explicit about what that means and why it's
>>> necessary would be good.

I think we talk about why availability is important in a number of =
places,
and also explain how various failures affect actors. E.g. publication =
server
failures affecting CAs (no new ROAs) and content availability affecting =
RPs.

But if you have specific suggestions :D


>>> I think it's reasonable to build a simple model and drive the advice
>>> from that, something like:
>>>=20
>>> * 20k ASN participating in the RPKI system
>>> * An hourly update cadence for Relying Parties
>>> * Average repository size of 1MBytes
>>>=20
>>> Missing repository information because of outages between the =
relying
>>> party and the publication service MAY not be critical
>>> immediately, but will lead to delays in the system and possibly
>>> imapcts on the relying parties as they force age out content
>>> and suffer delays in their workload collecting from the globally
>>> distributed system. One view is that the delays are expected
>>> because this is the internet and over time such outages will occur, =
so
>>> we should be resilient in the total system to this, I think
>>> this is the prevailing sentiment today... Some study about lengthy
>>> outages and impact to the RPKI  and routing systems would
>>> be super informative, I bet.

Whatever we say in a publication server BCP, not all operators
will follow it. In other words, I hope this BCP will help, but it won't
prevent that bad repositories exist. So, RPs should be prepared to deal
with them regardless of what we say here. Perhaps an RP BCP would be a
good idea though!

>>>=20
>>> Using wikipedia's 'how much outage is too much' page[1]  and the =
below
>>> information might be handy in deciding how much
>>> time can elapse between a Publication Service going offline for a
>>> percentage of the Internet and when flags should go up.
>>> As a starting stake in the ground I'd suggest 99.9 uptime seems like =
a
>>> great start?
>>>=20
>>> When it comes to discussing capabilities and expectations about the
>>> Publication Service itself, It would be expected for a
>>> Publication Service to be able to service:
>>> * 20000/3600 =3D=3D 6 full request sets per second
>>>    - every ASN has 1 RP talking to every Publication Service every =
hour
>>>    - of course this will probably be at least 2x larger, so round to
>>> 10x for growth and ease of math
>>>    - assume spherical cow request/repository can accomplish all
>>> tasks in 1 connection (and in that 1 second)
>>>    - 60 requests * 1MB each second
>>>    - produces about 500 mbps sustained traffic for this request rate
>>> * It's possible that RFC9286[2] expectations on nextUpdate existing
>>> may cause the relying party clients behave
>>>   differently, but that remains to be seen.
>>>=20

While numbers may help, I am not sure that we can use them meaningfully
in the BCP. A lot of this depends on the size of the repository and
frequency of change as well. And RP behaviour may change in future.

We kind of implicitly assume that this is all mitigated by using a CDN
for RRDP. We talk about capacity measurement for rsync in section 6.3.
If you can suggest some text about expected RP behaviour that may help.




>>> I think the above would guide a discussion on the point raised in
>>> RFC6481, I think adding to that some discussion
>>> about resilience and redundancy (which RFC6481 attempts to talk =
about
>>> a bit in Section 2.1 and point 5 of section 3)
>>> and how that could improve the ability of the Publication Service to
>>> be available would be helpful. I expect that guidance such
>>> as:
>>> * maintain at least 2 transit paths to the wider Internet for each =
deployment
>>>   - plan for outages on the local and immediate provider network =
systems
>>>   - plan for outages to your networking systems as maintenance
>>> requirements arise
>>> * plan more than one Publication Service deployment, in more than
>>> one geographically distinct region(s)
>>>   - remove local power/natural-disaster scenarios as much as
>>> possible from outage causes
>>>   - provide for disaster recovery  capabilities
>>> * move the Publication Service closer to the core of the network if =
possible
>>>   - shorten the connections between relying parties and the service
>>>=20
>>> Along with the last set of points would be some guidance about how =
to
>>> (again RFC6481 points a bit at this in section 3) to
>>> provide a coherent view of the Publication Service data across
>>> multiple relying party accessible endpoints. This should not
>>> be hard to accomplish, but certainly some thought and guidance would
>>> be handy to help folks move toward that world.

I believe this is addressed in 5.8 for RRDP and 6.1 for rsync.

>>>=20
>>> I think a review of historic outages to Publication Services would
>>> also show where each of the above points would have
>>> proven useful to the wider community of relying parties and
>>> Publication Services.
>>>=20
>>> Sorry, this set of comment(s) got longer than I expected :(

Ok, I probably did not address everything you said. But I (we authors)
are happy to keep the discussion going.

Tim


>>>=20
>>> -chris
>>>=20
>>> 0: https://www.rfc-editor.org/rfc/rfc6481.html#page-8 - 'Resource
>>> Certificate Publication Repository Considerations'
>>> 1: https://en.wikipedia.org/wiki/High_availability - How much =
downtime
>>> to tolerate?
>>> 2: https://www.rfc-editor.org/rfc/rfc9286.html#name-manifest -
>>> next-update definition in Manifest