Re: [sidr] RPKI Repository Distribution Protocol - a proposal for an rsync replacement for the RPKI

[Tim Bruijnzeels <[e-mail]>]

Hi,
 
As some of you might know I have also done some thinking on analysing the problems with the current infrastructure, and also have some ideas about improvements and talked about this at the Vancouver interim. It's way too much for inline email, so I also took the liberty of writing my ideas down in an IETF format, that hopefully helps to discuss this in a more structured way:

 http://www.ietf.org/internet-drafts/draft-tbruijnzeels-sidr-delta-protocol-00.txt

This too is of course a first draft and intended for discussion..


On Nov 20, 2012, at 9:18 AM, Russ Housley <[e-mail]> wrote:
> At the meeting in Atlanta, we heard some very impressive performance numbers for the restructured RIPE repository with rsync.  

Although the performance has improved dramatically for some RPs now that the structure is hierarchical, I don't think today's performance is a good indicator for the future.

We now see:
- Around 50-60 unique IP addresses for RPs connecting every day
- One of these (gw1.antd.nist.gov) is responsible for roughly 9k connections per day
- Ignoring that one for now the remaining are responsible for around 1.8k connections per day: 
   => about 36 fetches / day for each client, or once every 40 minutes

We have around 4.5k objects in our repository.

In a full deployment scenario we can expect many more relying parties, one for each ASN seems likely, so that is 40k clients. And while we don't agree yet what the size of the total repository would be, indications are it's in the order of 1-2M at least, so roughly 1/5th of that places our repository at 200-400k. There are also indications that more frequent updates are actually desirable. Say once every 5 minutes vs 40..

In short that's 10^3 times the clients, 10^2 times the objects, and 10 times the connection frequency. 

With those numbers it's very hard to predict exactly what behaviour we will see, it's very likely non-linear, but if it were, we're talking 10^6 times today's load.

> While your code is not feature complete, do you have any expectations of similar performance?

When measured 1:1 rsync performance is actually pretty good.

As described in my document: One of the major problems I see is that in this space we're not talking 1:1, but something like 1 server : 40k clients.



> On Nov 19, 2012, at 11:02 PM, Bryan Weber wrote:
> 
>> SIDR working group,
>> 
>> I humbly submit a proposal for your consideration. It is for an rsync replacement that for now I am calling the 'RPKI Repository Distribution Protocol (RRDP)'. It is intended to replace the use of rsync by the RPKI validators or at least re-open some previous discussion on possible replacements.
>> 
>> The primary benefit is that RRDP is a combination of a very limited Distributed Version Control System (DVCS) with a transport agnostic communication protocol. This means that repository changes can be snapshotted and retrieved as an atomic change. This solves the problem of in progress transfers when a new repository generation is kicked off. It also means that different protocols (HTTP/SSL/TLS/etc.) can be used for the actual transport. Finally, it can work with the existing rsync URIs and it can work alongside rsync at the same time. Should the protocol be considered for real adoption this should help to ease in gradual adoption.
>> 

I like the idea of the snapshotted atomic changes. But I have problems with a communication protocol where the server needs to chat with up to 40k RPs. The CPU and memory usage stats per client may not be the same as with rsync (see slides interim), but the same fundamental problem remains: you're proposing a smart server that has to spend cycles talking to too many clients.

As described in my proposal I prefer a protocol that lets RPs work out which changes to fetch by themselves, and although I too do not want to restrict to one protocol, I want something that can leverage existing proven http CDN infrastructure to deliver data to clients.

>> A brief paper that describes the protocol can be found at http://www.cobenian.com/documentation/rrdp.pdf. This is very much a first draft, but I would appreciate any feedback/comments you have and any interest you might have in trying to use this protocol with a real validator. I am all ears to suggestions for improvements. I would especially love to make it more aware of the contents of RPKI manifests.
>> 

First of all I want to thank you for thinking about solutions, even if our ideas may differ somewhat.

I had drafts of my proposal on my machine for a while, presented parts of it in Vancouver, and have discussed with various people, but I did not send it to sidr yet. Now that you sent your version, I felt like sharing mine though ;)


>> The beginning of a reference implementation can be found at http://www.github.com/cobenian/rrdp. This is not feature complete and certainly not compliant with the specification at the moment, but it should be in the very near future (a few days to a few weeks).



Tim