Re: Dictionary Compression for HTTP (at Facebook)

[Benjamin Kaduk <[e-mail]>]

On Fri, Aug 24, 2018 at 10:24:23AM +0000, Felix Handte wrote:
> Jyrki,
> 
> Glad to hear it! We're excited too!
> 
> We look forward to playing with Brotli's new capabilities when they're available.
> 
> Yes, Zstd can accept unformatted buffers as dictionaries, which it uses as you describe. Additionally though, Zstd describes a format for structured dictionaries[1], which includes metadata in addition to an LZ77 prefix.

One topic that came up during IESG review of draft-kucherawy-dispatch-zstd was
whether/when third-party or standard dictionaries would become available and how
dictionary IDs would be assigned for those cases (since at present, IIUC, the
dictionary IDs would need to be pre-negotiated between the two parties).  No
IANA registry was created at that time, but with a 4-byte dictionary identifier space
to work with, it seems like there might be space to create a registry for dictionary
IDs (including private use space, of course), and just publishing well-known
dictionaries.

-Ben

> [1] https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#dictionary-format <https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#dictionary-format>
> [2] https://tools.ietf.org/html/draft-vandevenne-shared-brotli-format-01#section-9 <https://tools.ietf.org/html/draft-vandevenne-shared-brotli-format-01#section-9>
>