ETag specification: load balance friendly and merge with Digest header from

Sergey Ponomarev <stokito@gmail.com> Tue, 28 July 2020 08:35 UTC

Return-Path: <ietf-http-wg-request+bounce-httpbisa-archive-bis2juki=lists.ie@listhub.w3.org>
X-Original-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Delivered-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 0BBD33A08C6 for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Tue, 28 Jul 2020 01:35:31 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.717
X-Spam-Level:
X-Spam-Status: No, score=-2.717 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_ADSP_CUSTOM_MED=0.001, DKIM_INVALID=0.1, DKIM_SIGNED=0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=0.001, MAILING_LIST_MULTI=-1, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=fail (2048-bit key) reason="fail (body has been altered)" header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hIs_nLo0Q-WW for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Tue, 28 Jul 2020 01:35:28 -0700 (PDT)
Received: from lyra.w3.org (lyra.w3.org [128.30.52.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 51E683A08C0 for <httpbisa-archive-bis2Juki@lists.ietf.org>; Tue, 28 Jul 2020 01:35:27 -0700 (PDT)
Received: from lists by lyra.w3.org with local (Exim 4.92) (envelope-from <ietf-http-wg-request@listhub.w3.org>) id 1k0L39-00030Z-Ng for ietf-http-wg-dist@listhub.w3.org; Tue, 28 Jul 2020 08:33:27 +0000
Resent-Date: Tue, 28 Jul 2020 08:33:27 +0000
Resent-Message-Id: <E1k0L39-00030Z-Ng@lyra.w3.org>
Received: from www-data by lyra.w3.org with local (Exim 4.92) (envelope-from <stokito@gmail.com>) id 1k0L38-0002xl-JJ for ietf-http-wg@listhub.w3.org; Tue, 28 Jul 2020 08:33:26 +0000
Received: from mimas.w3.org ([128.30.52.79]) by lyra.w3.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from <stokito@gmail.com>) id 1jumL4-0004Ig-65 for ietf-http-wg@listhub.w3.org; Mon, 13 Jul 2020 00:28:58 +0000
Received: from mail-oi1-x243.google.com ([2607:f8b0:4864:20::243]) by mimas.w3.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from <stokito@gmail.com>) id 1jumL2-0002QZ-1J for ietf-http-wg@w3.org; Mon, 13 Jul 2020 00:28:57 +0000
Received: by mail-oi1-x243.google.com with SMTP id y22so9601465oie.8 for <ietf-http-wg@w3.org>; Sun, 12 Jul 2020 17:28:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to:cc; bh=U0ONnWH4ID4aHP/WNPqZREIIiLhd+LycebRovU+C0Bo=; b=Icy/u4+1Jhm2hlJrw8BCbLsCNPfehHmMeco9PuwGKI5k0dykE/cXbldtmCB8ZFnpzv Hr448HyBrcgmfya4+FFdELFRQrkZEObRaKrjmNeSOiz34d+hKeqIWzMSMlIgQcYSQdMY Fg/BldwSSzNYli18fx25C7Sxpw8uDqCfTieyQMkClXd79/0X9eS0KA0lBpm078ipO9l3 mzMwt7MesGXgGJkH7WFH2lZq2PfykT5UvSIGE4G0BrkwuWURjwkQuRzAIG1Bfy3hOpdt Hf/y3oQNx47KZn8JUu1/qBe3/MCP9l7WIjvbpM0HEl1m68w8y4/Nipa2W6gtIMtZaU55 7WBA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=U0ONnWH4ID4aHP/WNPqZREIIiLhd+LycebRovU+C0Bo=; b=rcMYLsxDzeMUHMM+GWNvNenUFf8R8o0wZxT6K0I0Vkbr7aQAAjxGL81n1/P4MxNg8J BlH3lbDz+q9KPpYYUZ8K4SGKSEFNbt2wRHpB22wIT3qZ6MptZRTxRGGPlPhLv3i8SV6q qZTwr0qk0ibZ3ArX9x2d7GyfkeV9NX4ePXq5h2aT6dwBkU3FNO+BTCNrDjrTJdfB81ok VPgKe0ujpOKHUFw/WZDecz6kvHHS5wEmSWp3yO9gmh2eYt5r9IRtsiXF3qN0gLdAdq7a tLl5pARuGLiRLyKRXdzmKbjAmjry1q2zeSzAoiilDFOPnOU7fqNcXD2xUHJJOoxmkQnQ 0/iQ==
X-Gm-Message-State: AOAM530W+fok2KFi0WGV0dRbs+CIyZ0UvX9QsL1W9Er/1jMSfgYYi4MU vMQmjc5FghcdxgEwU/ZUrPKkifrhcF2Prns4P3UzAwneeBA=
X-Google-Smtp-Source: ABdhPJzQ5XgzacCEyXUgDkaUWkEh1JHCXKXoXECfnmWttGCnMHjLeAnvVs2SNinwXSQA2DfKjHugl+R5dSlMSYhbmig=
X-Received: by 2002:aca:7212:: with SMTP id p18mr12152314oic.40.1594600124620; Sun, 12 Jul 2020 17:28:44 -0700 (PDT)
MIME-Version: 1.0
From: Sergey Ponomarev <stokito@gmail.com>
Date: Mon, 13 Jul 2020 03:28:08 +0300
Message-ID: <CADR0UcXFgHrg9Q59AqN0D-PaZuVp96LPhGaQSo3a9cg+zFpU1w@mail.gmail.com>
To: ietf-http-wg@w3.org
Cc: robipolli@gmail.com
Content-Type: multipart/alternative; boundary="000000000000fca35105aa47c52c"
Received-SPF: pass client-ip=2607:f8b0:4864:20::243; envelope-from=stokito@gmail.com; helo=mail-oi1-x243.google.com
X-W3C-Hub-Spam-Status: No, score=-4.1
X-W3C-Hub-Spam-Report: BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, W3C_AA=-1, W3C_WL=-1
X-W3C-Scan-Sig: mimas.w3.org 1jumL2-0002QZ-1J 028755ddbf8145a2aac69fe2b382f6e8
X-caa-id: 1703cc05b2
X-Original-To: ietf-http-wg@w3.org
Subject: ETag specification: load balance friendly and merge with Digest header from
Archived-At: <https://www.w3.org/mid/CADR0UcXFgHrg9Q59AqN0D-PaZuVp96LPhGaQSo3a9cg+zFpU1w@mail.gmail.com>
Resent-From: ietf-http-wg@w3.org
X-Mailing-List: <ietf-http-wg@w3.org> archive/latest/37915
X-Loop: ietf-http-wg@w3.org
Resent-Sender: ietf-http-wg-request@w3.org
Precedence: list
List-Id: <ietf-http-wg.w3.org>
List-Help: <https://www.w3.org/Mail/>
List-Post: <mailto:ietf-http-wg@w3.org>
List-Unsubscribe: <mailto:ietf-http-wg-request@w3.org?subject=unsubscribe>

Hi,

I just implemented ETag caching for BusyBox httpd which is a http server
for embedded devices like WiFi routers.
While implementing I had to choose what exactly should be generated as ETag.
ETag is specified in https://tools.ietf.org/html/rfc2616#section-14.19 as
an opaque value and a server is free to generate it as it needs.
In the https://httpwg.org/specs/rfc7232.html#rfc.section.2.3 Conditional
Requests are better explained strategies to generate and compare ETags.
But even in the upcoming HTTP Caching draft-ietf-httpbis-cache-09 no any
practical details about ETag generation.

I did small research and found out that all web servers do it in their own
way and this causes several problems:
1. ETag may be badly or even wrongly generated.
2. When two different servers e.g. Apache and Nginx are behind load
balancer then their ETags will be always discarded because they are
generated differently. That's why some sysadmins disable ETag on one of the
servers.
These problems can be easily fixed if HTTP specification will provide a
recommended way to generate ETags while keeping freedom of choice.

Typical ETag is based on file's Last Modification Time and Size which can
be easily retrieved from the file system but can be a more strict hash or
checksum and sometimes a semantic version.

Just a quick overview of typical algorithms used in webservers.
Consider  we have a file with
* Size 1047 i.e. 417 in hex.
* MTime i.e. last modification on Mon, 06 Jan 2020 12:54:56 GMT which
is 1578315296 milliseconds in unix time or 1578315296666771000 nanoseconds.
* Inode which is a physical file number 66 i.e. 42 in hex

Different webservers returns ETag like:
Nginx: "5e132e20-417"                         i.e.
"hex(MTime)-hex(Size)". Not configurable.
Apache/2.2: "42-417-59b782a99f493"  i.e.  "hex(INode)-hex(Size)-hex(MTime
in nanoseconds)". Can be configured but MTime anyway will be in nanos
http://httpd.apache.org/docs/2.4/mod/core.html#fileetag
Apache/2.4: "417-59b782a99f493"       i.e.  "hex(Size)-hex(MTime in
nanoseconds)" i.e. without INode which is friendly for load balancing when
identical file have different INode on different servers.
OpenWrt uhttpd: "42-417-5e132e20"    i.e.
"hex(INode)-hex(Size)-hex(MTime)". Not configurable.
Tomcat 9: W/"1047-1578315296666"   i.e.  Weak"Size-MTime in Nanoseconds".
This is incorrect ETag because it should be strong as for a static file
i.e. octal compatibility.
LightHTTPD:  most weird:  "hashcode(42-1047-1578315296666771000)" i.e.
INode-Size-MTime but then reduced to a simple integer by hashcode. Can be
configured but you can only disable one part (etag.use-inode = "disabled")

Hex numbers are used here so often because it's cheap to convert a decimal
number to a shorter hex string.
Inode while adding more guarantees makes load balancing not possible and
very fragile if you simply copied the file during application redeploy.
MTime in nanoseconds is not available on all platforms and we don't need
such granularity. Apache have reported bugs on this like
https://bz.apache.org/bugzilla/show_bug.cgi?id=55573
The order MTime-Size or Size-MTime  is also matters because MTime is more
likely changed so comparing ETag string may be faster for a dozen
CPU cycles.
Even if this is not a full checksum hash but definitely not a weak ETag.
This is enough to show that we expect octal compatibility for Range
requests.
Apache and Nginx shares almost all trafik in Internet but most static files
are shared via Nginx and it is not configurable.

If I am not missing anything then it looks like Nginx uses the most
reasonable schema. And I used it for BusyBox httpd.
The whole ETag generated by printf("\"%" PRIx64 "-%" PRIx64 "\"", last_mod,
file_size)

My proposition is to take Nginx schema and make it as a recommended
ETag algorithm. Or at least just to mention in rfc7232 as an example.
And other servers should have at least possibility to configure such ETag
form.
I'll try to engage other web servers teams into the discussion and 'll try
to create patches for them.

While having the simple MTime-Size ETag algorithm solves a bunch of
problems but some systems wants to have more guarantees and they need hash
based ETags.
Any hash even MD5 or CRC32 is great to use as ETag.

There is a draft of Digest Headers
https://github.com/httpwg/http-extensions/blob/master/draft-ietf-httpbis-digest-headers.md
..
It's idea is similar to Subresource Integration (SRI).
And in fact instead of introducing the new Digest header we can just reuse
ETag header with prefix.

Respectively instead of:

    Digest: sha-256=4REjxQ4yrqUVicfSKYNO/cF9zNj5ANbzgDZt3/h3Qxo=

We can use

    ETag: "sha-256=4REjxQ4yrqUVicfSKYNO/cF9zNj5ANbzgDZt3/h3Qxo="

Client can easily parse ETag header and by prefix determine the way to
validate.
We'll have "structured ETag" and they are already supported by proxies.

For the same file server can send two comma separated ETags: one MTimeSize
and additional digest based. Old clients just resend them via
If-None-Match. If a server like BusyBox can only validate MTimeSize Etag it
will validate it and ignore sha256 based ETag.

BTW the file hashes can be stored ext4 in extended attributes to avoid
recalculating.

Please tell your thoughts and opinions and share best practice for ETags.

See also:
Apache code to generate ETag
https://searchcode.com/codesearch/view/28934406/
LightHTTPD
https://git.lighttpd.net/lighttpd/lighttpd1.4/src/branch/master/src/etag.c

-- 
Sergey Ponomarev <https://linkedin.com/in/stokito>, skype:stokito