Re: [sidr] sidrSlides for "RPKI Over BitTorrent" presentation

[Rob Austein <[e-mail]>]

At Thu, 24 May 2012 17:11:55 -0400, Christopher Morrow wrote:
> 
> On Thu, May 24, 2012 at 4:13 PM, John G. Scudder <[e-mail]> wrote:
> > Wes,
> >
> > On May 23, 2012, at 9:22 PM, Wes Hardaker wrote:
> >
> >> Bittorrent works well for sharing the load, but either requires a lot of
> >> bittorrent start files (whatever they're called), which then becomes
> >> hard to syncronize; or a small number of bittorrent files (one per
> >> aggregation point) that indicate you need to refetch and entire .zip
> >> file even if you already have most of it anyway.

The .zip file is just a wrapper around a .torrent file and an
associated text manifest (which is mostly present to provide a
stronger checksum than BitTorrent provides).

.zip files over HTTPS are the control channel.  RPKI data files over
BitTorrent are the data channel.  Confusing the two will lead to brain
trauma.

> > I'm far from an expert on Bittorrent but I believe this is not right. A
> > .torrent file (what you called a "bittorrent start file") can contain
> > contain a list of files to be transferred, there's no mandate to zip them
> > into a single file first. Presumably the BT protocol is smart about not
> > transferring files you already have locally -- that's kind of the whole
> > point. Rob's "operational overview" slide makes it clear he's imagining
> > operating in the collection-of-files mode.
> 
> per publication-point I think?

No.  It's one torrent per gatherer.  More precisely, one torrent at
time per gatherer, as each gatherer generates a new torrent when it
thinks it has a new batch of data that's worth sharing.

> So you have to collect N-thousand of these torrent files, and keep
> N-thousand torrent jobs running and keep state on N-thousand remote
> torrent files changing over time... some of this could be alleviated
> in many ways, worst case though is that.

One torrent per gatherer at a time.  The mechanism outlined in the
slides uses HTTPS polling to distribute the .zip files containing the
.torrent files.  The HTTPS service used for this can be replicated via
the usual techniques.

So the clients poll the HTTPS servers to get the .torrent files, then
use the .torrent files to retrieve the ten zillion little object files
from the BitTorrent cloud.  Note that BitTorrent is clever enough not
to retrieve files it already has, so in effect this produces a similar
effect to rsync (client only pulls what it needs), without imposing
the same kind of server load.