Re: How to handle content-encoding

Daurnimator <quae@daurnimator.com> Wed, 07 December 2016 12:39 UTC

Return-Path: <ietf-http-wg-request+bounce-httpbisa-archive-bis2juki=lists.ie@listhub.w3.org>
X-Original-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Delivered-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7177F129E88 for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Wed, 7 Dec 2016 04:39:31 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -6.302
X-Spam-Level:
X-Spam-Status: No, score=-6.302 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.001, RCVD_IN_DNSWL_HI=-5, RCVD_IN_SORBS_WEB=3.595, RP_MATCHES_RCVD=-2.896, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=daurnimator.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Zq4ipWEMfKJP for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Wed, 7 Dec 2016 04:39:29 -0800 (PST)
Received: from frink.w3.org (frink.w3.org [128.30.52.56]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id D9F16129E8E for <httpbisa-archive-bis2Juki@lists.ietf.org>; Wed, 7 Dec 2016 04:38:23 -0800 (PST)
Received: from lists by frink.w3.org with local (Exim 4.80) (envelope-from <ietf-http-wg-request@listhub.w3.org>) id 1cEbR7-0005No-By for ietf-http-wg-dist@listhub.w3.org; Wed, 07 Dec 2016 12:35:01 +0000
Resent-Date: Wed, 07 Dec 2016 12:35:01 +0000
Resent-Message-Id: <E1cEbR7-0005No-By@frink.w3.org>
Received: from mimas.w3.org ([128.30.52.79]) by frink.w3.org with esmtps (TLS1.2:RSA_AES_128_CBC_SHA1:128) (Exim 4.80) (envelope-from <quae@daurnimator.com>) id 1cEbQu-0005GB-K9 for ietf-http-wg@listhub.w3.org; Wed, 07 Dec 2016 12:34:48 +0000
Received: from mail-wj0-f179.google.com ([209.85.210.179]) by mimas.w3.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84_2) (envelope-from <quae@daurnimator.com>) id 1cEbQn-0007HO-4A for ietf-http-wg@w3.org; Wed, 07 Dec 2016 12:34:43 +0000
Received: by mail-wj0-f179.google.com with SMTP id xy5so359232019wjc.0 for <ietf-http-wg@w3.org>; Wed, 07 Dec 2016 04:34:19 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=daurnimator.com; s=daurnimator; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=SiNHUNXcqRlCz37DYWBqsrgrnhu8Ubdng6FHKRwJeg8=; b=gp2ZLPNv6973Q85yWv42HDEwVG5ktBW/JwI6X3N7WsSQ03vVGJRtIW4HsUirvc89Sc 2ACd7fF9+ZyNY9Vq54vyk+KRZZAxrHDk+Nm1p3hAYCWRjvsSZ2gwMMWRdpMcYw+ZFSBX D2DwtcDZAQozaogeSfMuZ/CAtm6VZwAaM4XZQ=
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=SiNHUNXcqRlCz37DYWBqsrgrnhu8Ubdng6FHKRwJeg8=; b=gQNSXrYVfEaNenqsRuIj31WsyMiBTinRSk8k1xaPfQrVN1haIUTOzv22D8GoxmHYIL syE6SxkQMiqGfUuSzvdwkiDMf9kHFJWPtwNu0wdH+AhL9Gt9q1La0r69w2bv3DCtIkjh beoYY7ltyu0NQ3uWUi5zpmoFqVJzizaHGq7sYIPKHHp3ErmMcFX7HU270gVRcJW6KYBB +vBFZ+4IW2YKzj2OCDewwuX/wSaTdHzHcNZ29vaeC0i/PSuSOREXRKJsTgztduhmm4Kr Fby9URGxgPRZqy29AIpqFCAMErsR0nyL6EqiG1EWaXObtpD9bcIhWR1X2ebwf0tyKKZK avqw==
X-Gm-Message-State: AKaTC03sI4tb/5ZiAAX9AjUT/rDEa4lpVwzEXkqTWGtm6sGSF8YTtQ3sGLGFk1lc/3tufQ==
X-Received: by 10.194.87.103 with SMTP id w7mr35280309wjz.164.1481114044728; Wed, 07 Dec 2016 04:34:04 -0800 (PST)
Received: from mail-wm0-f52.google.com (mail-wm0-f52.google.com. [74.125.82.52]) by smtp.gmail.com with ESMTPSA id vr9sm31143923wjc.35.2016.12.07.04.34.03 for <ietf-http-wg@w3.org> (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 07 Dec 2016 04:34:03 -0800 (PST)
Received: by mail-wm0-f52.google.com with SMTP id a197so165551957wmd.0 for <ietf-http-wg@w3.org>; Wed, 07 Dec 2016 04:34:03 -0800 (PST)
X-Received: by 10.25.27.145 with SMTP id b139mr21108915lfb.114.1481114043161; Wed, 07 Dec 2016 04:34:03 -0800 (PST)
MIME-Version: 1.0
Received: by 10.25.79.76 with HTTP; Wed, 7 Dec 2016 04:34:02 -0800 (PST)
In-Reply-To: <CAEnbY+fW_n4sFrFQSVcMWBoqxEWw3yoKnhCu1seRXj4GBr6wfA@mail.gmail.com>
References: <CAEnbY+fW_n4sFrFQSVcMWBoqxEWw3yoKnhCu1seRXj4GBr6wfA@mail.gmail.com>
From: Daurnimator <quae@daurnimator.com>
Date: Wed, 7 Dec 2016 23:34:02 +1100
X-Gmail-Original-Message-ID: <CAEnbY+cMmPKefxZHW++KT2Rf7F8oL4E-cUP7jDs-6LpR8fBy8g@mail.gmail.com>
Message-ID: <CAEnbY+cMmPKefxZHW++KT2Rf7F8oL4E-cUP7jDs-6LpR8fBy8g@mail.gmail.com>
To: HTTP Working Group <ietf-http-wg@w3.org>
Content-Type: text/plain; charset=UTF-8
Received-SPF: none client-ip=209.85.210.179; envelope-from=quae@daurnimator.com; helo=mail-wj0-f179.google.com
X-W3C-Hub-Spam-Status: No, score=-0.4
X-W3C-Hub-Spam-Report: BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_WEB=3.6, URIBL_BLOCKED=0.001, W3C_AA=-1, W3C_WL=-1
X-W3C-Scan-Sig: mimas.w3.org 1cEbQn-0007HO-4A 23a89b86b6430e4502a4464bc32b6e34
X-Original-To: ietf-http-wg@w3.org
Subject: Re: How to handle content-encoding
Archived-At: <http://www.w3.org/mid/CAEnbY+cMmPKefxZHW++KT2Rf7F8oL4E-cUP7jDs-6LpR8fBy8g@mail.gmail.com>
Resent-From: ietf-http-wg@w3.org
X-Mailing-List: <ietf-http-wg@w3.org> archive/latest/33123
X-Loop: ietf-http-wg@w3.org
Resent-Sender: ietf-http-wg-request@w3.org
Precedence: list
List-Id: <ietf-http-wg.w3.org>
List-Help: <http://www.w3.org/Mail/>
List-Post: <mailto:ietf-http-wg@w3.org>
List-Unsubscribe: <mailto:ietf-http-wg-request@w3.org?subject=unsubscribe>

On 31 May 2016 at 12:47, Daurnimator <quae@daurnimator.com> wrote:
> I'm thinking through how to add support for Content-Encoding to lua-http
> https://github.com/daurnimator/lua-http/issues/22
>
> A brief digression to lua-http structure (library terminology is borrowed
> from http2):
>   - a 'connection' encapsulates a socket, a connection has many streams
>   - a 'stream' is a request/response pair (a request can have multiple
> header blocks, and many data chunks)
>       - The same stream structure is used for both client and server
>       - You can implement a HTTP proxy by forwarding items from one stream
> to another
>   - a 'request' is a pre-prepared object consisting of a request header
> block, a function to obtain body chunks, and a destination.
>       - `request:go()` returns the 'main' response header block and a stream
> (from which you can read the body one chunk at a time)
>
> There is a desire to compress content to save bandwidth, HTTP has had two
> main ways to do this: Transfer-Encoding and Content-Encoding.
>
> To me it was simple to add support for Transfer-Encoding, without any
> ambiguities or issues. For HTTP1 in the stream logic:
>   -  (if zlib is installed) we automatically add `TE: gzip, deflate`.
>   - On reply, if Transfer-Encoding contains gzip or deflate, we decode it
> before passing it onto the caller.
> This is permitted as TE and Transfer-Encoding are hop-by-hop headers.
>
> However, HTTP2 does not support transfer-encoding.
> Furthermore, certain servers **stares at twitter.com** send
> `Content-Encoding: gzip` even if you *don't* send `Accept-Encoding: gzip`
> This seems to demand that I support Content-Encoding.
>
> As far as the specifications go, Content-Encoding is *meant* to be used to
> for end-to-end encoding that intermediate hops do not touch.
>   - Intermediaries should cache Content-Encoded bodies in their encoded form
>   - ETag is dependant on Content-Encoding
>
> This makes it hard to find a place for it in lua-http's structure.
> If I add it transparently in the stream (as done for Transfer-Encoding) then
> it will be hop-by-hop (not end-to-end)
> This seems to demand (at least for client requests) that it is switched
> on/off at the request layer.
> From there though, it seems it would need to add some sort of stream body
> filter?
>
> How should I be adding this? What have other implementations done? (and what
> do they wish they'd done differently?)
> The current state seems to be *against* the spec: should the spec be
> changed? should implementations be updated?
> HTTP2 has no transfer-encoding equivalent... why not?
>
> Regards,
> Daurn.
>
>
> Links:
>   - https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.11
> Original content-encoding spec
>   - https://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.5.1
> Hop-by-hop headers
>   - https://tools.ietf.org/html/rfc7231#section-3.1.2.1 Current spec
>   - https://bugzilla.mozilla.org/show_bug.cgi?id=68517 Mozilla disregards
> Content-Encoding spec
>   -
> https://stackoverflow.com/questions/11641923/transfer-encoding-gzip-vs-content-encoding-gzip
>   - https://daurnimator.github.io/lua-http/ lua-http documentation

*bump*.

Didn't get any replies :(