[Sidrops] implementation trick: deterministic last file modification timestamps to optimize RRDP <> RSYNC failovers

[Job Snijders <[e-mail]>]

Dear group,

I'd like to share a description of an idea, and some running code, which
demonstrate an optimization in order to reduce bandwidth and CPU cycles
consumed for both RSYNC server and RSYNC client.

In this message I'll elaborate on how the load minimization works, and
share a pointer to running code suitable for Publication Point operators
to take advantage of the idea.

Background
==========

RFC 8182 Section 4.1 notes that a design objective was to allow RSYNC
and RRDP to coexist. To enable co-existence both a hash and a URI are
provided which can be used to represent the set of files on a
filesystem. RFC 8182 doesn't provide much detail _how_ exactly RPs can
query, combine, and validate objects from repositories of different
types. In this email I'll try to fill in some of the blanks.

In the RSYNC synchronization protocol, a file's last modification
timestamp (from here on 'mod-time') and the filesize are used to
determine whether sending a copy the file copy over the wire is needed.
Using mod-time+filesize (the default mode) consumes far less CPU cycles
than calculating and comparing the 128-bit MD4 checksum of the file
(rsync's '-c' command line option) for the purpose of determining the
list of files to be transferred.

During normal operations, RPs will first connect to the RRDP service,
and if and only if there is some kind of issue with the RRDP service
(expired HTTPS TLS certificate, IP unreachable, malformed XML, etc) the
RPs will make an attempt to connect to the RSYNC service.

Most RP implementations serialize the RRDP datastructures to a
filesystem where each Signed Object, Certificate and CRL is represented
as a file. Most implementations (up until now! :-) do not set the newly
created file's mod-time to any specific value; instead files end up on
the filesystem with mod-time 'now' (the moment of serializing RRDP data
to disk).

Following from rsync's default algorithm (using mod-time + filesize),
one can imagine that the mod-time (for a file retrieved via RRDP) will
never ever match the mod-time of the same file observed through the
RSYNC service. This means that following any kind of failure in RRDP, a
full filelist synchronization needs to happen via RSYNC.

The synchronization minimization technique for RP implementations
=================================================================

Assuming the RP previously successfully synchronized via RRDP, and
implemented the notion of a local validated cache, and needs to switch
to RSYNC; the RP can use its local validated cache as a starting point
to start the RSYNC synchronization task.

GPL Rsync and Openrsync offer a command line option '--compare-dest=DIR'.
This option instructs rsync to use DIR as an additional hierarchy to
compare destination files against doing transfers (if the files are
missing in the destination directory). If a file is found in DIR that is
identical to the sender's file, the file will NOT be transferred to the
destination directory. This is useful for creating a sparse hierarchy
(transferring only files that have changed compared to an earlier
synchronization job).

The use of --compare-dest=DIR has as added benefit that it ensures the
'known good' local copy (the validated cache) is not overwritten should
the RSYNC server somehow ends up sending trash, as the difference
between the local validated cache and what the remote server offered
ends up in a 'staging' directory for further examination & validation.
For example, using '--compare-dest=DIR' allows the RP to find newer
manifests while minimizing file transfer and local storage usage.

As mentioned before, a mismatch in mod-time and/or filesize will result
in a transfer of the file. If the RP can store the file (fetched via
RRDP) with a mod-time that likely is the same mod-time as exposed via
the RSYNC service, chances are the file need not be transferred!

How to set 'good' mod-times without connecting to the RSYNC server?

While the RRDP XML documents themselves do not contain any information
about mod-time (only a hash, URI, and the Base64-encoded file are
available via RRDP); each and every RPKI Signed Object, Certificate, and
CRL does have useful timestamps embedded in the objects themselves!

I've found that the following 'internal' timestamps in more than 30% of
cases coincidence with what the RSYNC service lists as the purported
mod-time:

    Signed Objects (ROA, MFT, ASPA, GBR): CMS signing-time
    X.509 Certificates: the X.509 notBefore timestamp
    X.509 CRLs: the CRL lastUpdate timestamp

At the moment of writing, 100% of Signed Objects in the RPKI ecosystem
contain a CMS signing-time field, and this field turns out to be
incredibly useful to minimize RSYNC synchronization. A lot of CAs
immediately serialize to disk after signing, and RSYNC servers often
appear to publish that directory structure.

In other words: if an RP uses a filesystem hierarchy as its local
validated cache, and changes the mod-time of files retrieved via RRDP to
the CMS signing-time, X.509 notBefore or CRL lastUpdate timestamp; an RP
can easily see a 30% - 40% reduction in the list of files to be
transferred if synchronization via RSYNC is needed.

I've implemented the above trick in OpenBSD's rpki-client:
https://github.com/openbsd/src/commit/08ac1330e7955e03b6bba4571e0a9906d7ef5279

With less than a 100 lines of code the debuggability of the local
validated cache is improved: file mod-times suddenly mean something,
this help pinpoint what could be files of interest when using 'ls -al';
and file transfer burden is reduced.

I encourage fellow RP implementers (FORT, OctoRPKI, Routinator) to
investigate implementation of '--compare-dest=DIR' combined with 'fixup'
of the mod-times of files stored on the filesystem after RRDP fetches.
These tricks improves the experience for both RP operators as well as
publication point operators.

The synchronization minimization technique for Publication Points
=================================================================

Some Publication Point operators have in the past (or will in the
future) seek to simplify their operations by moving to a 'single source
of truth' for the origin of their publication point data. Krill-sync is
an example implementation of the concept of synthesizing a RSYNC
publication point based on a RRDP data source, by serializing the RRDP
files into a filesystem hierarchy.

Knowing that now - at least some - RP implementations will 'fixup' the
mod-time of files fetched via RRDP; logically, on the Publication Point
side of the house a similar 'fixup' will help improve/reduce the list of
files to be transferred: deterministic mod-times on both sides of the
synchronization channel improve performance.

To aid Publication Point operators, I've authored a liberally licensed
open-source utility named 'rpkitouch' (as in my mind it bears some
resemblance to the POSIX 'touch' utility).

The rpkitouch utility will extract the CMS signing-time, X.509
notBefore, or CRL lastUpdate timestamp from the contents of a file and
adjust the mod-time of the file on the filesystem accordingly.

    https://github.com/job/rpkitouch

This utility compiles and runs on both Linux and BSDs linked against
either OpenSSL or LibreSSL. The utility is not opinionated on the
validity of objects (e.g. from a validity time window or cryptographic
perspective), it only performs the most minimal sanity checking in order
to be able to reliably extract the relevant internal timestamp.

I believe the utility is performant enough for production purposes.
Below are measurements on how long it takes to 'fix up' the mod-times of
the primary publication point of the RIRs (measured on my T14 laptop
with NVME disk, executing the following command:)

   $ time find . -type f -print0 | xargs -0 rpkitouch

   AfriNIC - rsync://rpki.afrinic.net/repository/
   files: 7,088
   time: 0m01.40s real
   
   APNIC - rsync://rpki.apnic.net/{member_repository,repository}/
   files: 28,769
   time: 0m05.96s real
   
   ARIN - rsync://rpki.arin.net/repository/
   files: 67,324
   time: 0m13.98s real (0m04.05s with xargs -P6)
   
   LACNIC - rsync://repository.lacnic.net/rpki/
   files: 15,324
   time: 0m02.57s real
   
   RIPE - rsync://rpki.ripe.net/repository/
   files: 93,744
   time: 0m18.04s real (0m08.23s with xargs -P6)

Please note that the rpkitouch utility is *not* needed if "RRDP-to-RSYNC"
tools (like krill-sync) implement a similar mod-time fixup strategy! I
offer the rpkitouch utility both for real-world usage and as example
to inspire other implementation efforts. Please note that the practise
of backdating is irrelevant in this context, the trick is in picking
deterministic timestamps for the mod-time.

I encourage all developers of Publication Point software to use the idea
of setting file mod-times to the CMS signing-time, X.509 notBefore, or
CRL lastUpdate.

Conclusion
==========

RP implementations - already today - will benefit from adjusting the
mod-time of a file fetched via RRDP to a timestamp more likely to match
the mod-time on the RSYNC server.

Publication Point operators benefit from adjusting the mod-time of files
presented via their RSYNC service to align with the internal timestamps,
to take advantage of any RP's having implemented a similar mod-time
strategy.

Should Publication Point operators choose not incorporate the rpkitouch
utility (or implement its equivalent in their pipeline), there still is
benefit for RPs because many CAs/PPs immediately serialize to disk after
signing.

My hope is that by sharing this idea eventually the burden of switching
from RRDP to RSYNC (and back again) is reduced for both Relying Parties
as well as RPKI Publication Point operators.

Kind regards,

Job