Re: [QUIC] h2 mapping - hpack dynamic table by reference

[Mike Bishop <[e-mail]>]

Jana and I were talking about that at the HTTP Workshop.  The current solution is probably the best that can be achieved without modifying HPACK, because HPACK takes a hard dependency on reliable in-order delivery, which a dedicate stream can provide.  If we were to define HPACKv2 (or QPACK, if you prefer) and make its semantics more out-of-order-friendly, we could do a number of things, as you suggest.

Some approaches we talked about:

  *   Establish a static dictionary in the first flight and never change it.
  *   Newly-added entries have explicit and never-reused indices.  All HPACK frames are on-stream and only block if they reference an entry you don’t yet have.
  *   Split HPACK into two different frame types – one only modifies the header table; one only emits headers.  Each “modify” frame increments a generation number and records the number of “emit” frames in the previous generation.  Emitting frames block until the header table reaches its generation; the old header state can be discarded when you know you’ve seen all emitting frames from that generation.

Your idea is similar to the last of these, but choosing to encode to an old state for better reliability is an interesting wrinkle.  You’d have to add an explicit signal to evict the old state for that to work.  Defining an endpoint’s memory commitment and management gets a little fuzzier in this world, no matter what approach we take.

I think we’ll need to seriously consider something in this space.  It also implies changing HPACK implementations to be more cautious about adding to the table.  I did some prototypes during HTTP/2 and concluding that “index everything” was not substantially worse than the hypothetical ideal given a reasonable table size, so we went with that to save time.  With QPACK, that will be a decidedly sub-optimal strategy, and there will be some art in designing compressors to be as effective as possible in the presence of reordering.

From: QUIC [mailto:[e-mail]] On Behalf Of Patrick McManus
Sent: Thursday, November 3, 2016 2:29 PM
To: [e-mail]
Subject: [QUIC] h2 mapping - hpack dynamic table by reference

Hi Robbie, Mike, et al. Thanks for getting the ball rolling with https://tools.ietf.org/html/draft-shade-quic-http2-mapping-00..
I have a thought on the mapping - Putting all of the header frames onto one in-order stream is a pretty big hammer.

Is there stomach for looking at a strategy where hpack references the dynamic table state via some kind of instruction counter? That way the headers could be on the data (>=5) stream and just the dynamic table manipulation stuff from hpack could be on stream 3.. implementations would be able to choose whether to generate hpack that referenced the current counter or an older one (within some window).. an older one is going to be more robust to reordering happening on stream 3 so the dep between the streams is loosened. Anecdotally the rate of change of the dynamic table slows down (though I have no data - could collect it if we're serious) after an initial burst and at that point not using table updates until they've been acked could be a reasonable thing to do... and this even allows pre-populating the dynamic table at setup time with site-local values rather than in response to a query. Rewinding to any particular instruction seems to be a big burden, but we could probably do some things with checkpoints and acks to bound it all.

this is a fairly significant bit of surgery to h2's formats, but given that one of the real quic opportunities is to perform better under loss it could be worthwhile to sever ordering dependencies even further. Seems like a reasonable thing to investigate to me.
Thanks
-Patrick