[Wpack] WPACK and Web Archiving-focused bundle format

[Ilya Kreymer <[e-mail]>]

Hello,

We would like to start the process of standardizing a format that fully
supports all the requirements for bundling web data, http request/response
pairs, page lists, and associated metadata for web archiving use cases, a
'web archive bundle' format. I wanted to inquire if the WPACK working group
would be the right place for introducing such an effort, or if it would be
out of scope.

Looking at the charter, there is definitely overlap with some, but not all
of the goals. In particular, the bullet points #1, 2, 3, 5 from
https://datatracker.ietf.org/doc/charter-ietf-wpack/ are shared goals of
the web archiving bundle format, while some of the other goals are less
important.

The current CBOR-based bundle proposal is not sufficient to address the
full scope of web archiving requirements, and that's totally fine, as key
goals for that format are quite different.

I am wondering if this group would be open to accepting a proposal for a
different standard specifically geared towards addressing all of the web
archiving use cases, or if we should pursue other paths, such as other
working groups/new working group within IETF for such standardization
efforts.

Here is a (very) brief summary of some of the requirements for web
archiving that will be fulfilled in this new format:
- forwards and backwards compatibility with existing ISO WARC format, which
will be used to store the raw http request/response data.
- an index based on URL + timestamp
- index support for multiple request/response pairs of same URL, at same or
different timestamp
- support for storing request body, eg. for POST requests
- random-access based URL+timestamp index that can be partially loaded via
random access.
- support for different size web archive bundles, from a single page to
very large archive bundles consisting of many GBs of data or hundreds of
pages that are loaded entirely via random access.
- support for text index to allow full-text search
- support for combining multiple web archive bundles for a growing web
archive collection.

We of course plan to elaborate on all of these but first want to understand
if this group would be the appropriate place for this work, or not.

Any advice/additional guidance on this would be appreciated, or any next
steps to pursue this effort, if this is the right place for this work.

Thank you,
Ilya
Webrecorder