[Wpack] Web Archives, Replaying Web Pages and WPACK

[Ilya Kreymer <[e-mail]>]

Hi,

I wanted to reach out again to the webpack group, as I believe I am working
on solving some of the same problems as wpack, but from the perspective of
web archiving. It would be great if there was some way to collaborate with
this group, though I am struggling to understand how that could be done.

The overarching goal I believe seems to be the same: to replay HTTP network
traffic in a way that recreates an authentic representation of a website,
and to have a way to verify that the traffic was not forged. It seems a
'web archive' and a 'bundled http exchange' are fundamentally describing
the same type of object, with perhaps different storage requirements and
use cases.

I wanted to share a system, called https://replayweb.page/ which can replay
HTTP network traffic stored in a variety of formats directly in the
browser, using existing web standards, particularly Service Workers,
Fetch and IndexedDB (for caching).

Here are a few examples, which replay bundled HTTP traffic, and can even be
embedded in other pages as iframes:
https://webrecorder.net/embed-demo-1.html - replaying smaller
archives/bundles
https://webrecorder.net/embed-demo-2.html - replaying from a 17GB
archive/bundle
https://webrecorder.net/embed-demo-3.html - replaying more complex web
sites, including one with 3d viewer

These examples are all isolated and rendered independent of each other:
through Javascript rewriting and injection, the original Origin of the page
is emulated so that the site behaves as it is running on its initial
origin. This allows for replaying of complex, interactive web pages, though
is not perfect.

As I come from the web archiving community, I've focused mostly on WARC
format, as that is an existing ISO standard and widely in use, and the
system also supports replaying from HAR and the web bundles created via the
WBN tool.

However, the WARC format alone is a bit limiting, and there seems to
be a misunderstanding
about WARC
<https://github.com/WICG/webpackage/blob/fc9b3e75309546c805b5cdb1db74b2d58a8e0b28/explainers/navigation-to-unsigned-bundles.md#warc>:
It provides random access to HTTP traffic, but does not contain a built in
index necessary for random access (it is assumed the index is maintained
separately). To work around this, I've created a new 'bundling format', a
'bespoke zip format', which can contain WARCs (and other types of data,
even .wbn bundles), along with other metadata, and a compressed index.
This ZIP-based format is explained here:
https://github.com/webrecorder/web-archive-collection-format

Since the ZIP format allows for random access (see: ZipInfo
<https://github.com/Rob--W/zipinfo.js/>), it is possible to load all
bundled data on-demand via range requests. This allows the format to scale
to tens and probably hundreds of GBs.

The system also supports referencing URLs via query params in the fragment,
for example:
https://replayweb.page/?source=/examples/netpreserve-twitter.warc#view=replay&url=https%3A%2F%2Ftwitter.com%2Fnetpreserve&ts=20190603053135
loads
the WARC file, then loads the specified URL from the archive/bundle.

I wanted to share all of this to see if there's perhaps some way to align
with the work you're doing here, though I must admit it is not easy to
understand if that is possible or of interest to this group. Again, from my
perspective, it seems like you're working on a very similar problem,
attempting to standardize this at the browser level, but perhaps for
different use cases.

One area I'm especially interested in is verification for Saving a Bundle
in the Browser
<https://github.com/WICG/webpackage/blob/2a78f2930a228ee6872630ecb023fa71151cc164/draft-yasskin-wpack-use-cases.md#save-and-share-a-web-page-snapshot>
.
Unfortunately, it seems that this use case is currently out of scope. I am
especially interested in building tools to solve this problem, so that (to
use this example) Casey can save the page in their browser, share it with
Dakota, and that *Dakota can verify that this is what Casey saw in their
browser*, and it was not forged. I think being able to site a web bundle
from a client's perspective would be extremely useful for archival,
fact-checking, sharing, etc.. use cases and could make the web more
trustworthy.

Please let me know if there is any interest in collaborating, or if these
existing tools could somehow help this spec move forward.

(If anyone is interested, the replayweb.page tool can be found on github
at: https://github.com/webrecorder/replayweb.page (UI frontend) and
https://github.com/webrecorder/wabac.js (service worker backend)

Thank you,
Ilya
webrecorder.net