Re: Core drafts -02 out

["Charles 'Buck' Krasic" <[e-mail]>]

On Thu, Mar 23, 2017 at 4:12 PM, Mike Bishop <[e-mail]>
wrote:

> With regard to header block versus frame, though, you explicitly said in
> your reply before that the encoder can switch modes between frames in a
> single block.  So I think you mean what you wrote; it’s just that switching
> whether things can be interpreted out of order mid-block feels weird.
>

You're right.  After re-reading Header Compression and Decompression
<https://tools.ietf.org/html/rfc7540#section-4.3> just now, I see that
 I've been messing up the terminology, I was confusing header blocks with
header block fragments.  I will take a pass to correct that, and I think I
need to call out that what I'm proposing diverges from the HTTP/2
description---my thinking is that a header block should be serialized
packet/fragment-wise, as opposed to rfc5740 which says serialize the whole
header block and then divide it into header block fragments.

I agree that switching mid-block would feel weird, but it wouldn't be
common at all.  It should only be necessary if the table becomes full.  I
think experience will bear out that avoiding that


>
> And POSIX or not, I don't think you can unilaterally define a semantic
> about delivery to the remote app layer.  Though you could come close – if
> the call completed when the written data and all preceding data on the
> stream had been ACK’d, you know at least that the data is available to the
> remote app layer – whether it reads promptly is its own problem.
>

I think you can go further than that and reason correctly about ordering in
a coherent implementation.

If HA has been fully acked before HB is encoded, then I think an
implementation can quite reasonably guarantee that HA will be decoded
before HB.   I am confident this is true of ours.

If it helps clarify at all, I'd remind that our implementation is
single-threaded user level code.   I can imagine how it might be more
difficult (but not impossible) to reason about in an implementation that is
multi-threaded and/or divided across a kernel-user boundary.


>
> Sent from my Windows 10 phone
>
>
>
> *From: *Charles 'Buck' Krasic <[e-mail]>
> *Sent: *Thursday, March 23, 2017 2:58 PM
> *To: *Mike Bishop <[e-mail]>
> *Cc: *Stefan Eissing <[e-mail]>; Martin Thomson
> <[e-mail]>; IETF QUIC WG <[e-mail]>
>
> *Subject: *Re: Core drafts -02 out
>
>
>
>
> On Thu, Mar 23, 2017 at 10:52 AM, Mike Bishop <
> [e-mail]> wrote:
>
>> All I see in the links are http:// and the draft name, no host.  I’m
>> guessing you meant https://github.com/krasic/draf
>> t-krasic-quic-hpack/blob/master/draft-krasic-quic-hpack-00.html?  Once
>> the tracker reopens (sometime Sunday, I think) you should probably upload
>> those at datatracker.ietf.org to make it an official submission.  You’ll
>> need to make the draft name in the file match the file name before it will
>> successfully upload.
>>
>>
>>
> Thanks for the prompt feedback!
>
> Apologies, as I mentioned it is my first attempt at an IETF draft, and it
> my first time using the tools as described in SETUP.md
> <https://github.com/martinthomson/i-d-template/blob/master/doc/SETUP.md>.
>
> Until I the tracker reopens and I submit properly, I would suggest
> https://krasic.github.io/draft-krasic-quic-hpack/dra
> ft-krasic-quic-qpack-latest.html
>
>
>> Also, for ease of discussion, you might pick a different name than QPACK,
>> since there’s already a draft with that name.  (Or we both change names,
>> and call the eventually-adopted one QPACK; that’d be fine, too.)
>>
>
> I'm fine with whatever.  If  draft-krasic vs draft-bishop isn't enough to
> differentiate,  how about QPACK-ALT, or maybe QPACK-LITE?
>
>>
>>
>> As to actual feedback:  I see what you’re getting at here because you’ve
>> explained it – the decoder acknowledges the sequence number it’s up to,
>> then the encoder tracks that and uses it on the encode side.  I think I’d
>> have a really hard time parsing the document without that.  I think the
>> strongest advantage of this proposal is that an encoder for this scheme
>> could be used over HTTP/2, enabling shared code; you should bring that out
>> more.
>>
>
> I'll try to rework the text to emphise those points more and clarify.
>
> Nits:  the decoder per-se doesn't ack, but the encoder tracks using
> existing transport level ack mechanisms (more on this below).   Also, I
> think of this scheme as something a shared implementation could offer as an
> option, but I hadn't imagined the option would be ever be enabled in
> HTTP/2.  It might be interesting to consider though.
>
>>
>>
>> The way you’ve described the encoder requires tight integration between
>> the packetization logic and the header compressor in two ways:
>>
>>    - It needs to know the “packet epoch” (encode epoch of the first
>>    frame in the current packet) while compressing, but until you know the size
>>    of the compressed output, you can’t know whether it will fit in the current
>>    packet.  That in turn means you’ll need to roll back any state and
>>    re-encode with a different packet epoch if it turns out not to fit.
>>
>> I don't think roll back is esssential.
>
> The space available in the current packet could be known when invoking the
> hpack encoder, so it would stop emitting representations if the packet
> reaches the space limit, and in that case return a header block that for a
> prefix of the provided header field list, and in later packets it would
> generate a continuation header block(s) for remaining header fields.
>
>
>>    -
>>    - The “commit epoch” is determined, not by data at the compressor
>>    level, but by reaching into the transport and checking which packets
>>    containing the STREAM frames containing the HPACK data in question have
>>    been acknowledged.  Also note that ACK doesn’t signify delivery to the
>>    application layer (HTTP), just presence in the appropriate queue.
>>
>>
>>
> My understanding is that APIs are out of scope of most of the docs.
> However, in our implementation, the stream write interfaces uniformly
> provide a means of to confirm acknowledgement,  so it is possible to do it
> without reaching down.
>
> I agree that ACK doesn't signify delivery to what is above HTTP, but one
> of advantages of not having legacy APIs (POSIX sockets) between QUIC and
> HTTP is that the implemenation can certainly ensure that ACK signifies
> delivery to HTTP.
>
>> In 2.2.1.1, the encoder is informed whether “QPACK is enabled,” but since
>> that can be toggled on/off on a per-header-frame basis (and I hope you
>> really meant per-header-block), it seems like the decoder is just following
>> the encoder’s lead on this.  Who’s actually making that decision?  You say
>> it gets turned off if a drop results, but the encoder is the one who
>> knows/determines that.
>>
> Yes, all decisions are made by the encoder, and as in HPACK the decoder
> strictly follows the encoder's lead.
>
> The "informed" language was simply alluding to the encoder side of the
> HPACK API being extended with QPACK related params.
>
> And yes, I meant "header block".  Will fix that.
>
> This has the same problem as the current HPACK state that RST on a control
>> stream loses critical data and there’s no way to recover.  I’m personally
>> hoping we can make our way out of that requirement.
>>
> My position is that REFUSED should be the only kind of RST needed or
> allowed for control streams, and dealing with that is tractible.
>
>
>>
>> *From:* Charles 'Buck' Krasic [mailto:[e-mail]]
>> *Sent:* Tuesday, March 21, 2017 6:04 PM
>> *To:* Mike Bishop <[e-mail]>
>> *Cc:* Stefan Eissing <[e-mail]>; Martin Thomson <
>> [e-mail]>; IETF QUIC WG <[e-mail]>
>>
>> *Subject:* Re: Core drafts -02 out
>>
>>
>>
>> Hi Folks.
>>
>>
>>
>> I've put together a first attempt at my QPACK proposal:
>>
>>
>>
>> draft-krasic-quic-hpack-00.html
>>
>> draft-krasic-quic-hpack-00.txt
>> <https://krasic.github.io/draft-krasic-quic-hpack/draft-krasic-quic-hpack-00.txt>
>>
>>
>>
>> Apologies in advance, this is my first attempt at an IETF draft.
>>
>>
>>
>>
>>
>> On Wed, Mar 15, 2017 at 10:37 AM, Mike Bishop <
>> [e-mail]> wrote:
>>
>> Thanks for the feedback!
>>
>> Yes, you've run straight into the big quandary with HPACK.  I don't think
>> anyone expects that we will ship this way; I have a proposal in
>> https://tools.ietf.org/html/draft-bishop-quic-http-and-qpack for an
>> HPACK-replacement that would solve many of these.  Buck Krasic has a
>> proposal which he has sketched in e-mail but not submitted as a draft.  The
>> main point of the sequence number was to get us off the "everything on
>> stream 3" model and let us sort out the problems of HPACK later.  #228
>> tracks fixing HPACK, in whatever form that takes.
>>
>> We could solve it in the same way that we did PRIORITY, by adding an
>> "affected stream number" field and moving the HEADERS/PUSH_PROMISE frames
>> to Stream 3 as well.  However, that is just as blocking as the current
>> approach, so not really an improvement.  Worse, the reason we can tolerate
>> large header frames is because they occur on their own streams and don't
>> block arrival of data from other streams.  If all headers occur on a single
>> stream, that's not true -- you *are* blocking muxing of other streams again.
>>
>> I like the comparison to concurrent edits.  We've discussed having
>> rolling deltas in various mechanisms; the problem is that it requires
>> reaching into the transport for ACK state to figure out when you can
>> discard old deltas for good on the receiver side.  Buck's proposal is
>> similar, essentially requiring the receiver to echo back the point up to
>> which it has received all frames, and the sender shouldn't reference state
>> that the receiver hasn't fully assimilated yet.  It feels like adding
>> application-level ACKs to me, which I'd like to avoid.
>>
>> HPACK is also one of the reasons for the two-stream-per-request
>> approach.  There are two reasons for this -- one is that HPACK frames (in
>> the current design) can't be lost when a stream is RST, so the draft
>> forbids resetting control streams.  Since we still need to be able to RST
>> requests, we add the semantic that killing the data stream implies the same
>> thing about the control stream.  It's messy, and hopefully we can remove
>> that once HPACK is fixed.  The second, as you guessed, is not dealing with
>> DATA frames.  #245 notes that, if we fix HPACK, we could go back to a
>> single stream per request; proponents of both "keep framing out of the way"
>> and "fewer streams is easier to manage" have weighed in there.  Please add
>> your voice.
>>
>> As to buffering, it's actually easy enough -- just don't read from data
>> streams until you've seen the headers.  Sure, the sender will fill up their
>> flow control window on that stream -- and then they'll stop and send you
>> QUIC BLOCKED frames, until you start reading the body and generating
>> WINDOW_UPDATE frames.  One of the biggest arguments for two streams in my
>> mind is making sure that a request blocked in this way doesn't impede the
>> flow of control frames on that stream -- for example, should we
>> successfully get certificate auth frames added, a certificate request.
>> I've had two customers this week telling me it's a bug in our server code
>> that their TLS reneg gets stuck in TCP buffers behind a giant request body,
>> because the server won't read an unauthenticated body and the client won't
>> hold off sending the body until authentication succeeds.  I'd like to see
>> blocks like that become less possible in QUIC, at least.
>>
>> Yes, making SETTINGS immutable reduces the flexibility.  The opinion at
>> the interim was that we could live without it, as we know of exactly one
>> implementation of one extension that does mid-stream setting changes, and I
>> own it.  😊  If there's a compelling use case for changing settings
>> mid-stream we weren't aware of, that's new information that could justify
>> the complexity.  (But note that with 0-RTT connection setup, it would be
>> nearly as cheap to open a new connection with the new setting and
>> transition your traffic to it.)
>>
>> Negotiation still works, since each side would simply advertise that they
>> support an extension or not, and consider the extension to be active once
>> you've seen the other side's SETTINGS frame.  See
>> https://tools.ietf.org/html/draft-ietf-quic-http-01#section-5.2.5.3 for
>> the complexity this allowed us to remove.  An extension could add to the
>> list easily -- if you support extension X, you should also remember the
>> value for setting X_VAL.  Clients that don't use the extension don't care.
>> An extension that needed some more complex negotiation would have to use an
>> extension-specific frame, it's true, but we haven't so far seen an
>> extension do that.
>>
>> -----Original Message-----
>> From: QUIC [mailto:[e-mail]] On Behalf Of Stefan Eissing
>> Sent: Wednesday, March 15, 2017 6:22 AM
>> To: Martin Thomson <[e-mail]>; IETF QUIC WG <
>> [e-mail]>
>> Subject: Re: Core drafts -02 out
>>
>>
>> > Am 14.03.2017 um 00:57 schrieb Martin Thomson <[e-mail]
>> >:
>> >
>> > The editors have submitted -02 versions of the base set of QUIC drafts.
>> [...]
>> > https://tools.ietf.org/html/draft-ietf-quic-http-02
>>
>> Thanks, Martin! I assume that was announced to get feedback from the
>> lurkers here. ;-)
>>
>> I'll give this a try. All mistakes and misunderstandings are mine alone.
>>
>> ------------------------------------------------------------
>> -----------------------------
>>
>> Very well written spec. Easy to understand for someone having read rfc
>> 7540 a bit.
>>
>> I have some comments to the proposed HTTP mapping approach. Where the wg
>> has already discussed and exhausted alternatives, please excuse my
>> ignorance and ignore my comments. I had not the time to follow all
>> discussions ongoing on this topic. Feel free to cherry-pick what seems
>> helpful.
>>
>>
>> > 4.  Stream Mapping and Usage
>>
>> +1 to directly using quic stream ids instead of virtual h2 stream
>> +identifier
>>
>>
>> However, using 2 quic streams for a single request/response does not sit
>> well with me. I assume that stems from the wish to get rid of DATA frames.
>> Which sounds nice, but is it worth it? By doubling the # of streams for a
>> client, how much overhead does that introduce (I speak of a server holding
>> >10k quic "connections")?
>>
>> Also, the server needs to buffer data on quic streams 7, 11, 15, etc.
>> because HEADERs might arrive some time in the future on streams 5, 9, 13,
>> etc. or not. There is no way to route this data somewhere, because the meta
>> information is still missing.
>>
>> And there is still Holb on:
>>
>> > 4.2.1.  Header Compression
>> > ...
>> > DISCUSS:  Keep HPACK with HOLB?  Redesign HPACK to be order-
>> >       invariant?  How much do we need to retain compatibility with
>> >       HTTP/2's HPACK?
>>
>> Using a counter in HEADERS is a crutch:
>> - it is a highly specific solution to a common problem in http/quic:
>> synchronicity in connection level state changes. SETTINGS (see below) has
>> the same problem, as does have PRIORITY in HEADERS. It seems that
>> performance wise, all HEADERS could as well be sent on stream 3.
>>
>> Now, solving Hol blocking for HEADERS would be a fine achievement.
>>
>> > 5.  HTTP Framing Layer
>> > Frames are used only on the connection (stream 3) and message
>> >    (streams 5, 9, etc.) control streams.
>>
>> And streams 4, 8, 12, etc. I assume.
>>
>> > 5.2.3.  SETTINGS
>> > ...
>> > SETTINGS frames always apply to a connection, never a single stream.
>> >  A SETTINGS frame MUST be sent as the first frame of the connection
>> > control stream (see Section 4) by each peer, and MUST NOT be sent
>> > subsequently or on any other stream.  If an endpoint receives an
>> > SETTINGS frame on a different stream, the endpoint MUST respond with
>> > a connection error of type HTTP_SETTINGS_ON_WRONG_STREAM.  If an
>> > endpoint receives a second SETTINGS frame, the endpoint MUST respond
>> > with a connection error of type HTTP_MULTIPLE_SETTINGS.
>>
>> What about HPACK state? The connection state change problem again visible.
>>
>> This is a severe restriction on extensions mechanisms that want to affect
>> a connection. Because if the http/quic does not solve this problem, how are
>> they expected to do it? They either announce themselves on the first
>> SETTINGS or remain silent forever, it seems. How would an extension
>> handshake work then? Own stream 3 handshake frames?
>>
>> > 5.2.3.3.  Usage in 0-RTT
>>
>> What about HPACK state? Does it need to be kept or is it reset?
>>
>> > HTTP_PUSH_ALREADY_IN_CACHE (0x03):  The server has attempted to push
>> >      content which the client has cached.
>> > ...
>> > HTTP_REQUEST_CANCELLED (0x04):  The client no longer needs the
>> >     requested data.
>>
>> Nitpick: seems redundant. The first could be replaced by the second.
>> Stream number will suffice.
>>
>> ----------------------------------------------------------
>>
>> Taking two steps back:
>>
>> I think the main difficulty comes from the lack of a "hq connection
>> state" concept and how changes to that state can be managed. Evidence:
>> - The 0-RTT mentions certain SETTINGS that need to be remembered by a
>> client. How would an extension add to this? Will every extension have to
>> come up with its own solution?
>> - The HEADERs sequence number is a highly specific fix for the missing
>> state change
>> - The SETTINGS-ONCE restriction simply avoids the problem by killing a h2
>> mechanism
>>
>> If hq defines stream 3 as the place where connection state changes happen
>> *AND* synchronizes OPEN/CLOSE/RST of other streams on it, client and server
>> can have shareable concept of the connection state.
>>
>>
>> I am no HPACK expert. The basic problem looks like concurrent editing
>> against a repository.
>> Both client and server start with connection state zero (CS-0) and the
>> predefined HPACK dictionary (HP-0). After SETTINGS exchange, client is in
>> CS-1 and server is in CS-2 for its side. Let's call the  CS-1 HPACK state
>> HP-1.
>>
>> Client sends new HEADERS on 5 and keeps the HPACK delta around (HP-1.5).
>> The HEADERS carries the connection state number it is based on (CS-1).
>> Client sends new HEADERS on stream 9, also based on CS-1. Client keeps that
>> delta around (HP-1.9).
>>
>> Client then decides to announce a new connection state (CS-3) by sending
>> how it applied the deltas HP-1.5 and HP-1.9 to come up with HP-3. The
>> description allows the server to update its HP-1 to HP-3 as well.
>>
>> Response HEADERS are also based on a server connection state, explicitly,
>> so the client knows which HP-X to use when decoding them. At some time, the
>> server sends an announcment of CS-4 with HPACK data on stream 3 back to the
>> client.
>>
>> For a 0-RTT, client and server could exchange the connection state id in
>> the initial SETTINGS, to make sure they have at least the same name
>> remembered as the other side. Client: "I was in CS-19 and you were in
>> CS-8." Server: "Yep."
>>
>> By implicitly adding all SETTINGS changes to connection states, the
>> problem is also solved for extensions.
>>
>> If one side receives HEADERS with an unknown connection state:
>> - if the state id is greater than any known one: set a stream timeout and
>> wait for changes on stream 3 to arrive
>> - state id less than max(known conn state): STREAM_RST_UNKNOWN_CONN_STATE
>>
>> New SETTINGS value: MAX_CONN_STATE number of maximum connection state the
>> client/server is willing to keep, exchanged initially. Announcing a new
>> connection state allows the other side to drop the lowest one, if
>> MAX_CONN_STATE are used.
>>
>> To get optimal HPACK size compression, every HEADERs would also announce
>> a new connections state. To have less potential HOLB, connection states do
>> not change during request bursts.
>>
>> Something like that.
>>
>> -Stefan
>>
>>
>>
>>
>>
>> --
>>
>> Charles 'Buck' Krasic | Software Engineer | [e-mail] | +1 (408)
>> 412-1141
>>
>
>
>
> --
> Charles 'Buck' Krasic | Software Engineer | [e-mail] | +1 (408)
> 412-1141 <(408)%20412-1141>
>
>


-- 
Charles 'Buck' Krasic | Software Engineer | [e-mail] | +1 (408)
412-1141