Set-Cookie and Cookie Optimized Binary Encoding

[James M Snell <[e-mail]>]

Just continuing on with my exploration of improvements we could possibly
make for the binary encoding of various headers. Currently I'm looking at
Cookie and Set-Cookie. Here's what I've come up with so far...

Here's an random example of a simple Set-Cookie taken from the har file
samples Mark has collected:

  Set-Cookie: skin=noskin; path=/; domain=.amazon.com; expires=Sat,
03-Nov-2012 13:04:26 GMT

Looking at this, ignoring the actual key and value for a minute, there is
quite a bit of wasted overhead in this format (Set-Cookie2 is even worse,
but I'll get to that in a bit). The main question is, can we do better and
still provide an easy 1.1 <-> 2.0 transform for Cookies. I definitely think
we can. Here's what I have in mind:

+-----------------------------------------+
|H|S|B|P|M|X|X|X|len(key)|key|len(val)|val|
+-----------------------------------------+
|len(path)|path|len(domain)|domain|expires|
+-----------------------------------------+
|num_params|... repeating param key block |
+-----------------------------------------+

  H = Http-Only Bit
  S = Secure-Only Bit
  B = Binary-Value Bit
  P = Optional-Param List Bit (indicates whether the Set-Cookie includes
the optional parameters block (third row in the record above)
  M = Max-Age Bit - If set, the expires field specifies the Maximum-Age of
the Cookie, otherwise expires is a compact encoded date time (as we've
already discussed). Either way, the expires field is always 4 bytes.
  X = Reserved Flag Bit

  len(key)    -> 1 byte
  len(val)    -> 4 bytes
  len(path)   -> 2 bytes
  len(domain) -> 2 bytes
  expires     -> 4 bytes

For typical Set-Cookies, with no optional parameters (I'll explain this
later) this gives us 14-bytes of overhead. For the example cookie above, we
go from 78 bytes down to 36 bytes. In the Http/1 format, adding HttpOnly
and Secure flags adds at least 18 bytes of additional data. In this
optimized form those require no additional space.

When set, the Binary Value bit indicates that the val field contains
arbitrary binary data as opposed to encoded-text. As Mark and others have
pointed out, allowing for Binary cookie values can lead to trouble with
backwards compatibility with 1.1 applications. To address this I propose
the following: When translating an optimized Http/2 Set-Cookie to Http/1,
if the Binary Value Bit is set, the val is Base64 encoded and a new
"Binary" flag is added to the Http/1 Set-Cookie. For example:

  Set-Cookie: data={base64}; path=/; domain=.example.net; expires=...;
HttpOnly; Secure; Binary

When translating from Http/1 to Http/2, if the Set-Cookie includes the
Binary flag, we can attempt to base64 decode the value and include it
directly in the optimized Http/2 record syntax directly, setting the Binary
Value Bit, and hopefully saving significant amount of space.

For the Cookie Header itself, we would also have a binary syntax..

+-----------------------------------+
|B|XXXXXXX|len(key)|key|len(val)|val|
+-----------------------------------+

  B = Binary-Value Bit

When value is simple text, this record format actually comes out longer
than the existing format, but that's OK, I think. For binary cookie values,
however, this format gives us the option of reducing the record size
significantly. So it's a tradeoff. Here, however, the translation to and
from Http/1 gets a bit more difficult because the Cookie header does not
allow for optional flags the way Set-Cookie does, so we have to get a bit
creative. What I propose is that we use a convention similar to that used
in RFC5987 and append an asterisk * to the key name..

For example, when translating a Http/2 Binary Cookie header to Http/1, it
would be encoded as:

  Cookie: key*={base64}

When translating back to Http/2, if we see the * in the key name, we can
attempt to base64 decode the value and set the Binary Value Flag in the
Http/2 Binary Cookie header. It's not perfect, but it ought to work.

Getting back to Set-Cookie, there's the part about the optional parameters.
While the current Set-Cookie format is pretty stable, there have been times
in the past (Set-Cookie2) where additional parameters were specified.
Set-Cookie is one of those things where it may make sense to support some
long term extensibility. For that purpose, I have included 3 additional
reserved flag bits and an optional block of extension key=value pairs that
can be tacked on at the end. If the Optional Parameter Bit is set, we know
to go looking for those additional parameters at the end. The value of
those parameters must be text, keys are text. len(key) = 1 byte, len(val) =
4 bytes. This gives a significant amount of latitude for extension and
translates fine to the existing Http/1 Set-Cookie syntax.

With these changes, I estimate that we can reduce the overall overhead of
transmitting Set-Cookie and Cookie values by around 50% on average, while
maintaining compatibility with Http/1.

Thoughts?