Re: How to handle content-encoding

[Cory Benfield <[e-mail]>]

> On 7 Dec 2016, at 12:34, Daurnimator <[e-mail]> wrote:
> 
> On 31 May 2016 at 12:47, Daurnimator <[e-mail]> wrote:
>> 
>> To me it was simple to add support for Transfer-Encoding, without any
>> ambiguities or issues. For HTTP1 in the stream logic:
>>  -  (if zlib is installed) we automatically add `TE: gzip, deflate`.
>>  - On reply, if Transfer-Encoding contains gzip or deflate, we decode it
>> before passing it onto the caller.
>> This is permitted as TE and Transfer-Encoding are hop-by-hop headers.

In my experience, the gzip and deflate options for Transfer-Encoding are essentially unsupported. RFC 7230 doesn’t provide any requirement to support them, and so they are almost always not used.

>> However, HTTP2 does not support transfer-encoding.
>> Furthermore, certain servers **stares at twitter.com** send
>> `Content-Encoding: gzip` even if you *don't* send `Accept-Encoding: gzip`
>> This seems to demand that I support Content-Encoding.

Are you sending Accept-Encoding: identity? RFC 7231 Section 5.3.4 says the following things about Accept-Encoding:

1. If the request contains no Accept-Encoding header then any content-encoding is considered acceptable.
2. If the request does contain an accept-encoding header, but the server cannot satisfy it (for example, the server may not have access to an uncompressed resource), then the server may send a content-coding anyway (this is because the server SHOULD send a response without any content-coding, but is not required to)

>> How should I be adding this? What have other implementations done? (and what
>> do they wish they'd done differently?)
>> The current state seems to be *against* the spec: should the spec be
>> changed? should implementations be updated?
>> HTTP2 has no transfer-encoding equivalent... why not?

Lets tackle these questions in order.

1. My implementations support only Content-Encoding. My clients support all three major compressed content-encodings. This has more or less been just fine: almost all servers also only support Content-Encoding. Generally speaking, servers that support content-encoding for static content do so not by compressing on the fly but by having compressed and uncompressed versions of each resource on disk. For dynamic responses, compression is frequently handled by middlewares that apply the content-encoding on-the-fly. This is generally not an issue: the ETag can be calculated afterwards if that’s required.

2. I don’t see that there’s any spec violations here. Can you clarify about where you believe the spec violation is occurring?

3. HTTP/2 deliberately removed all support for hop-by-hop headers of this form. For more discussion on this specific issue, see this thread from 2014: https://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/1179.html <https://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/1179.html>. This is a substantial thread with a number of links out of it to other discussions. The major counter-arguments were that Transfer-Encoding: gzip required logic to detect already-compressed content to avoid double-compression (that is, to avoid compressing resources that were already compressed) and that it was not widely deployed.

Some further attempts have been made to redefine Transfer-Encoding: gzip using HTTP/2 frames, but again there has been relatively lukewarm interest from implementers to support that draft, so it has stalled.

Essentially, it seems that the bulk of the HTTP/2 community isn’t interested in Transfer-Encoding: gzip: they’ve concluded, rightly or wrongly, that Content-Encoding: gzip is fine.

Cory