RE: Core drafts -02 out

[Mike Bishop <[e-mail]>]

With regard to header block versus frame, though, you explicitly said in your reply before that the encoder can switch modes between frames in a single block.  So I think you mean what you wrote; it’s just that switching whether things can be interpreted out of order mid-block feels weird.

And POSIX or not, I don't think you can unilaterally define a semantic about delivery to the remote app layer.  Though you could come close – if the call completed when the written data and all preceding data on the stream had been ACK’d, you know at least that the data is available to the remote app layer – whether it reads promptly is its own problem.

Sent from my Windows 10 phone

From: Charles 'Buck' Krasic<mailto:[e-mail]>
Sent: Thursday, March 23, 2017 2:58 PM
To: Mike Bishop<mailto:[e-mail]>
Cc: Stefan Eissing<mailto:[e-mail]>; Martin Thomson<mailto:[e-mail]>; IETF QUIC WG<mailto:[e-mail]>
Subject: Re: Core drafts -02 out



On Thu, Mar 23, 2017 at 10:52 AM, Mike Bishop <[e-mail]<mailto:[e-mail]>> wrote:
All I see in the links are http:// and the draft name, no host.  I’m guessing you meant https://github.com/krasic/draft-krasic-quic-hpack/blob/master/draft-krasic-quic-hpack-00.html?  Once the tracker reopens (sometime Sunday, I think) you should probably upload those at datatracker.ietf.org<http://datatracker.ietf.org> to make it an official submission.  You’ll need to make the draft name in the file match the file name before it will successfully upload.

Thanks for the prompt feedback!

Apologies, as I mentioned it is my first attempt at an IETF draft, and it my first time using the tools as described in SETUP.md<https://github.com/martinthomson/i-d-template/blob/master/doc/SETUP.md>.

Until I the tracker reopens and I submit properly, I would suggest https://krasic.github.io/draft-krasic-quic-hpack/draft-krasic-quic-qpack-latest.html

Also, for ease of discussion, you might pick a different name than QPACK, since there’s already a draft with that name.  (Or we both change names, and call the eventually-adopted one QPACK; that’d be fine, too.)

I'm fine with whatever.  If  draft-krasic vs draft-bishop isn't enough to differentiate,  how about QPACK-ALT, or maybe QPACK-LITE?

As to actual feedback:  I see what you’re getting at here because you’ve explained it – the decoder acknowledges the sequence number it’s up to, then the encoder tracks that and uses it on the encode side.  I think I’d have a really hard time parsing the document without that.  I think the strongest advantage of this proposal is that an encoder for this scheme could be used over HTTP/2, enabling shared code; you should bring that out more.

I'll try to rework the text to emphise those points more and clarify.

Nits:  the decoder per-se doesn't ack, but the encoder tracks using existing transport level ack mechanisms (more on this below).   Also, I think of this scheme as something a shared implementation could offer as an option, but I hadn't imagined the option would be ever be enabled in HTTP/2.  It might be interesting to consider though.

The way you’ve described the encoder requires tight integration between the packetization logic and the header compressor in two ways:

  *   It needs to know the “packet epoch” (encode epoch of the first frame in the current packet) while compressing, but until you know the size of the compressed output, you can’t know whether it will fit in the current packet.  That in turn means you’ll need to roll back any state and re-encode with a different packet epoch if it turns out not to fit.

I don't think roll back is esssential.

The space available in the current packet could be known when invoking the hpack encoder, so it would stop emitting representations if the packet reaches the space limit, and in that case return a header block that for a prefix of the provided header field list, and in later packets it would generate a continuation header block(s) for remaining header fields.


  *
  *   The “commit epoch” is determined, not by data at the compressor level, but by reaching into the transport and checking which packets containing the STREAM frames containing the HPACK data in question have been acknowledged.  Also note that ACK doesn’t signify delivery to the application layer (HTTP), just presence in the appropriate queue.

My understanding is that APIs are out of scope of most of the docs.   However, in our implementation, the stream write interfaces uniformly provide a means of to confirm acknowledgement,  so it is possible to do it without reaching down.

I agree that ACK doesn't signify delivery to what is above HTTP, but one of advantages of not having legacy APIs (POSIX sockets) between QUIC and HTTP is that the implemenation can certainly ensure that ACK signifies delivery to HTTP.
In 2.2.1.1, the encoder is informed whether “QPACK is enabled,” but since that can be toggled on/off on a per-header-frame basis (and I hope you really meant per-header-block), it seems like the decoder is just following the encoder’s lead on this.  Who’s actually making that decision?  You say it gets turned off if a drop results, but the encoder is the one who knows/determines that.
Yes, all decisions are made by the encoder, and as in HPACK the decoder strictly follows the encoder's lead.

The "informed" language was simply alluding to the encoder side of the HPACK API being extended with QPACK related params.

And yes, I meant "header block".  Will fix that.

This has the same problem as the current HPACK state that RST on a control stream loses critical data and there’s no way to recover.  I’m personally hoping we can make our way out of that requirement.
My position is that REFUSED should be the only kind of RST needed or allowed for control streams, and dealing with that is tractible.


From: Charles 'Buck' Krasic [mailto:[e-mail]<mailto:[e-mail]>]
Sent: Tuesday, March 21, 2017 6:04 PM
To: Mike Bishop <[e-mail]<mailto:[e-mail]>>
Cc: Stefan Eissing <[e-mail]<mailto:[e-mail]>>; Martin Thomson <[e-mail]<mailto:[e-mail]>>; IETF QUIC WG <[e-mail]<mailto:[e-mail]>>

Subject: Re: Core drafts -02 out

Hi Folks.

I've put together a first attempt at my QPACK proposal:

draft-krasic-quic-hpack-00.html<http://draft-krasic-quic-hpack-00.html>
draft-krasic-quic-hpack-00.txt<https://krasic.github.io/draft-krasic-quic-hpack/draft-krasic-quic-hpack-00.txt>

Apologies in advance, this is my first attempt at an IETF draft.


On Wed, Mar 15, 2017 at 10:37 AM, Mike Bishop <[e-mail]<mailto:[e-mail]>> wrote:
Thanks for the feedback!

Yes, you've run straight into the big quandary with HPACK.  I don't think anyone expects that we will ship this way; I have a proposal in https://tools.ietf.org/html/draft-bishop-quic-http-and-qpack for an HPACK-replacement that would solve many of these.  Buck Krasic has a proposal which he has sketched in e-mail but not submitted as a draft.  The main point of the sequence number was to get us off the "everything on stream 3" model and let us sort out the problems of HPACK later.  #228 tracks fixing HPACK, in whatever form that takes.

We could solve it in the same way that we did PRIORITY, by adding an "affected stream number" field and moving the HEADERS/PUSH_PROMISE frames to Stream 3 as well.  However, that is just as blocking as the current approach, so not really an improvement.  Worse, the reason we can tolerate large header frames is because they occur on their own streams and don't block arrival of data from other streams.  If all headers occur on a single stream, that's not true -- you *are* blocking muxing of other streams again.

I like the comparison to concurrent edits.  We've discussed having rolling deltas in various mechanisms; the problem is that it requires reaching into the transport for ACK state to figure out when you can discard old deltas for good on the receiver side.  Buck's proposal is similar, essentially requiring the receiver to echo back the point up to which it has received all frames, and the sender shouldn't reference state that the receiver hasn't fully assimilated yet.  It feels like adding application-level ACKs to me, which I'd like to avoid.

HPACK is also one of the reasons for the two-stream-per-request approach.  There are two reasons for this -- one is that HPACK frames (in the current design) can't be lost when a stream is RST, so the draft forbids resetting control streams.  Since we still need to be able to RST requests, we add the semantic that killing the data stream implies the same thing about the control stream.  It's messy, and hopefully we can remove that once HPACK is fixed.  The second, as you guessed, is not dealing with DATA frames.  #245 notes that, if we fix HPACK, we could go back to a single stream per request; proponents of both "keep framing out of the way" and "fewer streams is easier to manage" have weighed in there.  Please add your voice.

As to buffering, it's actually easy enough -- just don't read from data streams until you've seen the headers.  Sure, the sender will fill up their flow control window on that stream -- and then they'll stop and send you QUIC BLOCKED frames, until you start reading the body and generating WINDOW_UPDATE frames.  One of the biggest arguments for two streams in my mind is making sure that a request blocked in this way doesn't impede the flow of control frames on that stream -- for example, should we successfully get certificate auth frames added, a certificate request.  I've had two customers this week telling me it's a bug in our server code that their TLS reneg gets stuck in TCP buffers behind a giant request body, because the server won't read an unauthenticated body and the client won't hold off sending the body until authentication succeeds.  I'd like to see blocks like that become less possible in QUIC, at least.

Yes, making SETTINGS immutable reduces the flexibility.  The opinion at the interim was that we could live without it, as we know of exactly one implementation of one extension that does mid-stream setting changes, and I own it.  😊  If there's a compelling use case for changing settings mid-stream we weren't aware of, that's new information that could justify the complexity.  (But note that with 0-RTT connection setup, it would be nearly as cheap to open a new connection with the new setting and transition your traffic to it.)

Negotiation still works, since each side would simply advertise that they support an extension or not, and consider the extension to be active once you've seen the other side's SETTINGS frame.  See https://tools.ietf.org/html/draft-ietf-quic-http-01#section-5.2.5.3 for the complexity this allowed us to remove.  An extension could add to the list easily -- if you support extension X, you should also remember the value for setting X_VAL.  Clients that don't use the extension don't care.  An extension that needed some more complex negotiation would have to use an extension-specific frame, it's true, but we haven't so far seen an extension do that.

-----Original Message-----
From: QUIC [mailto:[e-mail]<mailto:[e-mail]>] On Behalf Of Stefan Eissing
Sent: Wednesday, March 15, 2017 6:22 AM
To: Martin Thomson <[e-mail]<mailto:[e-mail]>>; IETF QUIC WG <[e-mail]<mailto:[e-mail]>>
Subject: Re: Core drafts -02 out


> Am 14.03.2017 um 00:57 schrieb Martin Thomson <[e-mail]<mailto:[e-mail]>>:
>
> The editors have submitted -02 versions of the base set of QUIC drafts.
[...]
> https://tools.ietf.org/html/draft-ietf-quic-http-02

Thanks, Martin! I assume that was announced to get feedback from the lurkers here. ;-)

I'll give this a try. All mistakes and misunderstandings are mine alone.

-----------------------------------------------------------------------------------------

Very well written spec. Easy to understand for someone having read rfc 7540 a bit.

I have some comments to the proposed HTTP mapping approach. Where the wg has already discussed and exhausted alternatives, please excuse my ignorance and ignore my comments. I had not the time to follow all discussions ongoing on this topic. Feel free to cherry-pick what seems helpful.


> 4.  Stream Mapping and Usage

+1 to directly using quic stream ids instead of virtual h2 stream
+identifier

However, using 2 quic streams for a single request/response does not sit well with me. I assume that stems from the wish to get rid of DATA frames. Which sounds nice, but is it worth it? By doubling the # of streams for a client, how much overhead does that introduce (I speak of a server holding >10k quic "connections")?

Also, the server needs to buffer data on quic streams 7, 11, 15, etc. because HEADERs might arrive some time in the future on streams 5, 9, 13, etc. or not. There is no way to route this data somewhere, because the meta information is still missing.

And there is still Holb on:

> 4.2.1.  Header Compression
> ...
> DISCUSS:  Keep HPACK with HOLB?  Redesign HPACK to be order-
>       invariant?  How much do we need to retain compatibility with
>       HTTP/2's HPACK?

Using a counter in HEADERS is a crutch:
- it is a highly specific solution to a common problem in http/quic: synchronicity in connection level state changes. SETTINGS (see below) has the same problem, as does have PRIORITY in HEADERS. It seems that performance wise, all HEADERS could as well be sent on stream 3.

Now, solving Hol blocking for HEADERS would be a fine achievement.

> 5.  HTTP Framing Layer
> Frames are used only on the connection (stream 3) and message
>    (streams 5, 9, etc.) control streams.

And streams 4, 8, 12, etc. I assume.

> 5.2.3.  SETTINGS
> ...
> SETTINGS frames always apply to a connection, never a single stream.
>  A SETTINGS frame MUST be sent as the first frame of the connection
> control stream (see Section 4) by each peer, and MUST NOT be sent
> subsequently or on any other stream.  If an endpoint receives an
> SETTINGS frame on a different stream, the endpoint MUST respond with
> a connection error of type HTTP_SETTINGS_ON_WRONG_STREAM.  If an
> endpoint receives a second SETTINGS frame, the endpoint MUST respond
> with a connection error of type HTTP_MULTIPLE_SETTINGS.

What about HPACK state? The connection state change problem again visible.

This is a severe restriction on extensions mechanisms that want to affect a connection. Because if the http/quic does not solve this problem, how are they expected to do it? They either announce themselves on the first SETTINGS or remain silent forever, it seems. How would an extension handshake work then? Own stream 3 handshake frames?

> 5.2.3.3.  Usage in 0-RTT

What about HPACK state? Does it need to be kept or is it reset?

> HTTP_PUSH_ALREADY_IN_CACHE (0x03):  The server has attempted to push
>      content which the client has cached.
> ...
> HTTP_REQUEST_CANCELLED (0x04):  The client no longer needs the
>     requested data.

Nitpick: seems redundant. The first could be replaced by the second. Stream number will suffice.

----------------------------------------------------------

Taking two steps back:

I think the main difficulty comes from the lack of a "hq connection state" concept and how changes to that state can be managed. Evidence:
- The 0-RTT mentions certain SETTINGS that need to be remembered by a client. How would an extension add to this? Will every extension have to come up with its own solution?
- The HEADERs sequence number is a highly specific fix for the missing state change
- The SETTINGS-ONCE restriction simply avoids the problem by killing a h2 mechanism

If hq defines stream 3 as the place where connection state changes happen *AND* synchronizes OPEN/CLOSE/RST of other streams on it, client and server can have shareable concept of the connection state.


I am no HPACK expert. The basic problem looks like concurrent editing against a repository.
Both client and server start with connection state zero (CS-0) and the predefined HPACK dictionary (HP-0). After SETTINGS exchange, client is in CS-1 and server is in CS-2 for its side. Let's call the  CS-1 HPACK state HP-1.

Client sends new HEADERS on 5 and keeps the HPACK delta around (HP-1.5). The HEADERS carries the connection state number it is based on (CS-1). Client sends new HEADERS on stream 9, also based on CS-1. Client keeps that delta around (HP-1.9).

Client then decides to announce a new connection state (CS-3) by sending how it applied the deltas HP-1.5 and HP-1.9 to come up with HP-3. The description allows the server to update its HP-1 to HP-3 as well.

Response HEADERS are also based on a server connection state, explicitly, so the client knows which HP-X to use when decoding them. At some time, the server sends an announcment of CS-4 with HPACK data on stream 3 back to the client.

For a 0-RTT, client and server could exchange the connection state id in the initial SETTINGS, to make sure they have at least the same name remembered as the other side. Client: "I was in CS-19 and you were in CS-8." Server: "Yep."

By implicitly adding all SETTINGS changes to connection states, the problem is also solved for extensions.

If one side receives HEADERS with an unknown connection state:
- if the state id is greater than any known one: set a stream timeout and wait for changes on stream 3 to arrive
- state id less than max(known conn state): STREAM_RST_UNKNOWN_CONN_STATE

New SETTINGS value: MAX_CONN_STATE number of maximum connection state the client/server is willing to keep, exchanged initially. Announcing a new connection state allows the other side to drop the lowest one, if MAX_CONN_STATE are used.

To get optimal HPACK size compression, every HEADERs would also announce a new connections state. To have less potential HOLB, connection states do not change during request bursts.

Something like that.

-Stefan




--
Charles 'Buck' Krasic | Software Engineer | [e-mail]<mailto:[e-mail]> | [phone]<tel:(408)%20412-1141>



--
Charles 'Buck' Krasic | Software Engineer | [e-mail]<mailto:[e-mail]> | [phone]<tel:(408)%20412-1141>