Re: How to handle content-encoding

Cory Benfield <> Thu, 08 December 2016 09:52 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id E5402129E7A for <>; Thu, 8 Dec 2016 01:52:34 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -6.201
X-Spam-Status: No, score=-6.201 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-5, RCVD_IN_SORBS_WEB=3.595, RP_MATCHES_RCVD=-2.896, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id txJxF2bp3kFz for <>; Thu, 8 Dec 2016 01:52:32 -0800 (PST)
Received: from ( []) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 50078129CDA for <>; Thu, 8 Dec 2016 01:52:32 -0800 (PST)
Received: from lists by with local (Exim 4.80) (envelope-from <>) id 1cEvKo-0007A0-4E for; Thu, 08 Dec 2016 09:49:50 +0000
Resent-Date: Thu, 08 Dec 2016 09:49:50 +0000
Resent-Message-Id: <>
Received: from ([]) by with esmtps (TLS1.2:RSA_AES_128_CBC_SHA1:128) (Exim 4.80) (envelope-from <>) id 1cEvKb-00078d-5G for; Thu, 08 Dec 2016 09:49:37 +0000
Received: from ([]) by with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84_2) (envelope-from <>) id 1cEvKT-0007Qx-PK for; Thu, 08 Dec 2016 09:49:31 +0000
Received: by with SMTP id a197so208551346wmd.0 for <>; Thu, 08 Dec 2016 01:49:08 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20150623; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=fr2WNysiyUudv3zCMA1uyu17QuMy8PXTCNbY3zKUERI=; b=yvURFbTUttDhZAodkqzB5a9QlJ8ZGxWsadJfjYR2OYvFXj5O1/01xS7BLQTIcyqs1J 1kLKPkgRwXJ1yJdxOb3YwAfk7NfLUYvOq1dIRZYr54UNSI1ttJ1r+d6JAgzqbrMH3ClK 99z+DC8ka6j8g6sh6KVoBSykSTi3LpRrFe+qcUX7A045INUmfcWBzKXuYKIbaTl9tWSr 2/r5J93ula0WjQzTPPl6dYbJPDUSz16NV9IFdG+jaYufe0Mk6X0y3SFml4ZjmXW76PHF ki3Nj5Jkg7ytE87ivz74FnL1lAq3+/e0xd0c+qvX5jmw9yzrzFSlLJDvsvO28Swajo/s e/QA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20130820; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=fr2WNysiyUudv3zCMA1uyu17QuMy8PXTCNbY3zKUERI=; b=G7CPKA0Pze+l6AsoaJepAVhgC+XgH9pqX2q2VzTght6Qdxbb6lu+qAwIn0+mFiBeBC HjM7da0A7XZPpc3GJaiG5uZ9F9gXB+EdZG0D+RNsUOTspXvCt8Bg5ToX4gS6M+C4+tIq Yvrj4fGYP7L6Nrk7X0r1bb8Mpd65HFttRnxhKr8EaTfIZIxRczcnq+OjfXnHF5+hnJGs lRDpD/YaepAIzWI6dqDvs9Iku54BPGR7x8LayLS3mAkD4jeRB93vAooAAEdF3wZMExbV bBy5FSHJedctx2pZNwIPwMIJtKglvi+nqeU66IU8fOHhRgblo1saGzSNdgnB3m5uX7BO M8gw==
X-Gm-Message-State: AKaTC000RbErj88keTDfx0Ttzd8MX4XyxA3NycKhfTf9DToJ9bkNmOHK8gfkNr+pBKkRrw==
X-Received: by with SMTP id n127mr1422967wmg.28.1481190542536; Thu, 08 Dec 2016 01:49:02 -0800 (PST)
Received: from [] ( []) by with ESMTPSA id ba10sm36021489wjb.32.2016. (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 08 Dec 2016 01:49:01 -0800 (PST)
From: Cory Benfield <>
Message-Id: <>
Content-Type: multipart/alternative; boundary="Apple-Mail=_FA7363CE-4157-48A5-A77F-88ED51FFD863"
Mime-Version: 1.0 (Mac OS X Mail 10.2 \(3259\))
Date: Thu, 08 Dec 2016 09:49:00 +0000
In-Reply-To: <>
Cc: HTTP Working Group <>
To: Daurnimator <>
References: <> <>
X-Mailer: Apple Mail (2.3259)
Received-SPF: pass client-ip=;;
X-W3C-Hub-Spam-Status: No, score=-0.8
X-W3C-Hub-Spam-Report: AWL=-0.458, BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_WEB=3.599, SPF_PASS=-0.001, W3C_AA=-1, W3C_WL=-1
X-W3C-Scan-Sig: 1cEvKT-0007Qx-PK babff8e009ed88418bcd986131d204a0
Subject: Re: How to handle content-encoding
Archived-At: <>
X-Mailing-List: <> archive/latest/33134
Precedence: list
List-Id: <>
List-Help: <>
List-Post: <>
List-Unsubscribe: <>

> On 7 Dec 2016, at 12:34, Daurnimator <> wrote:
> On 31 May 2016 at 12:47, Daurnimator <> wrote:
>> To me it was simple to add support for Transfer-Encoding, without any
>> ambiguities or issues. For HTTP1 in the stream logic:
>>  -  (if zlib is installed) we automatically add `TE: gzip, deflate`.
>>  - On reply, if Transfer-Encoding contains gzip or deflate, we decode it
>> before passing it onto the caller.
>> This is permitted as TE and Transfer-Encoding are hop-by-hop headers.

In my experience, the gzip and deflate options for Transfer-Encoding are essentially unsupported. RFC 7230 doesn’t provide any requirement to support them, and so they are almost always not used.

>> However, HTTP2 does not support transfer-encoding.
>> Furthermore, certain servers **stares at** send
>> `Content-Encoding: gzip` even if you *don't* send `Accept-Encoding: gzip`
>> This seems to demand that I support Content-Encoding.

Are you sending Accept-Encoding: identity? RFC 7231 Section 5.3.4 says the following things about Accept-Encoding:

1. If the request contains no Accept-Encoding header then any content-encoding is considered acceptable.
2. If the request does contain an accept-encoding header, but the server cannot satisfy it (for example, the server may not have access to an uncompressed resource), then the server may send a content-coding anyway (this is because the server SHOULD send a response without any content-coding, but is not required to)

>> How should I be adding this? What have other implementations done? (and what
>> do they wish they'd done differently?)
>> The current state seems to be *against* the spec: should the spec be
>> changed? should implementations be updated?
>> HTTP2 has no transfer-encoding equivalent... why not?

Lets tackle these questions in order.

1. My implementations support only Content-Encoding. My clients support all three major compressed content-encodings. This has more or less been just fine: almost all servers also only support Content-Encoding. Generally speaking, servers that support content-encoding for static content do so not by compressing on the fly but by having compressed and uncompressed versions of each resource on disk. For dynamic responses, compression is frequently handled by middlewares that apply the content-encoding on-the-fly. This is generally not an issue: the ETag can be calculated afterwards if that’s required.

2. I don’t see that there’s any spec violations here. Can you clarify about where you believe the spec violation is occurring?

3. HTTP/2 deliberately removed all support for hop-by-hop headers of this form. For more discussion on this specific issue, see this thread from 2014: <>. This is a substantial thread with a number of links out of it to other discussions. The major counter-arguments were that Transfer-Encoding: gzip required logic to detect already-compressed content to avoid double-compression (that is, to avoid compressing resources that were already compressed) and that it was not widely deployed.

Some further attempts have been made to redefine Transfer-Encoding: gzip using HTTP/2 frames, but again there has been relatively lukewarm interest from implementers to support that draft, so it has stalled.

Essentially, it seems that the bulk of the HTTP/2 community isn’t interested in Transfer-Encoding: gzip: they’ve concluded, rightly or wrongly, that Content-Encoding: gzip is fine.