Re: Heuristic caching without validators

"Roy T. Fielding" <fielding@gbiv.com> Thu, 17 January 2013 07:46 UTC

Return-Path: <ietf-http-wg-request@listhub.w3.org>
X-Original-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Delivered-To: ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2096C21F8B2E for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Wed, 16 Jan 2013 23:46:43 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -10.599
X-Spam-Level:
X-Spam-Status: No, score=-10.599 tagged_above=-999 required=5 tests=[BAYES_00=-2.599, RCVD_IN_DNSWL_HI=-8]
Received: from mail.ietf.org ([64.170.98.30]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3Yo8iUIz7rrg for <ietfarch-httpbisa-archive-bis2Juki@ietfa.amsl.com>; Wed, 16 Jan 2013 23:46:40 -0800 (PST)
Received: from frink.w3.org (frink.w3.org [128.30.52.56]) by ietfa.amsl.com (Postfix) with ESMTP id BED1D21F8B1E for <httpbisa-archive-bis2Juki@lists.ietf.org>; Wed, 16 Jan 2013 23:46:40 -0800 (PST)
Received: from lists by frink.w3.org with local (Exim 4.72) (envelope-from <ietf-http-wg-request@listhub.w3.org>) id 1Tvk9k-0008Co-LK for ietf-http-wg-dist@listhub.w3.org; Thu, 17 Jan 2013 07:45:00 +0000
Resent-Date: Thu, 17 Jan 2013 07:45:00 +0000
Resent-Message-Id: <E1Tvk9k-0008Co-LK@frink.w3.org>
Received: from lisa.w3.org ([128.30.52.41]) by frink.w3.org with esmtp (Exim 4.72) (envelope-from <fielding@gbiv.com>) id 1Tvk9h-0008C7-O0 for ietf-http-wg@listhub.w3.org; Thu, 17 Jan 2013 07:44:57 +0000
Received: from caiajhbdccah.dreamhost.com ([208.97.132.207] helo=homiemail-a74.g.dreamhost.com) by lisa.w3.org with esmtp (Exim 4.72) (envelope-from <fielding@gbiv.com>) id 1Tvk9g-0002gA-Sh for ietf-http-wg@w3.org; Thu, 17 Jan 2013 07:44:57 +0000
Received: from homiemail-a74.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a74.g.dreamhost.com (Postfix) with ESMTP id 6D2FC67C06B; Wed, 16 Jan 2013 23:44:35 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gbiv.com; h=subject :mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; s=gbiv.com; bh=xoZUfxAEiSLt9162HmAfQGMxwBw=; b=WNDEzIOzIy3Hxs1qDar+BiNmlw2Z plNRHyo6Ub6J6Skgf9kztAZnx1o0pLxjDnt9IhKpgj5NC6f59B81mE+iNaWvEMoE MgowqW1AKkcAmTiyaEpc3K+thi1HBR5ARYrqBHpJmQaB7QzIwgXim0wCHUOQ53Ea 3Dz3T6pZm3i7ymg=
Received: from [192.168.1.84] (99-21-208-82.lightspeed.irvnca.sbcglobal.net [99.21.208.82]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: fielding@gbiv.com) by homiemail-a74.g.dreamhost.com (Postfix) with ESMTPSA id 1D26067C069; Wed, 16 Jan 2013 23:44:35 -0800 (PST)
Mime-Version: 1.0 (Apple Message framework v1283)
Content-Type: text/plain; charset="us-ascii"
From: "Roy T. Fielding" <fielding@gbiv.com>
In-Reply-To: <18D1E3CE-D2B8-4E48-9EC2-8E68962D6F1C@mnot.net>
Date: Wed, 16 Jan 2013 23:44:33 -0800
Cc: "Adrien W. de Croy" <adrien@qbik.com>, HTTP Working Group <ietf-http-wg@w3.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <12DDC8BC-6666-4960-82A9-D8B73A03D8F4@gbiv.com>
References: <emfe6950e6-f177-4326-b1b0-b2c6bd7b1d4a@bombed> <18D1E3CE-D2B8-4E48-9EC2-8E68962D6F1C@mnot.net>
To: Mark Nottingham <mnot@mnot.net>
X-Mailer: Apple Mail (2.1283)
Received-SPF: none client-ip=208.97.132.207; envelope-from=fielding@gbiv.com; helo=homiemail-a74.g.dreamhost.com
X-W3C-Hub-Spam-Status: No, score=-3.5
X-W3C-Hub-Spam-Report: AWL=-3.423, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_NONE=-0.0001
X-W3C-Scan-Sig: lisa.w3.org 1Tvk9g-0002gA-Sh 4090940e351b211fc3b48b753abdf26e
X-Original-To: ietf-http-wg@w3.org
Subject: Re: Heuristic caching without validators
Archived-At: <http://www.w3.org/mid/12DDC8BC-6666-4960-82A9-D8B73A03D8F4@gbiv.com>
Resent-From: ietf-http-wg@w3.org
X-Mailing-List: <ietf-http-wg@w3.org> archive/latest/15951
X-Loop: ietf-http-wg@w3.org
Resent-Sender: ietf-http-wg-request@w3.org
Precedence: list
List-Id: <ietf-http-wg.w3.org>
List-Help: <http://www.w3.org/Mail/>
List-Post: <mailto:ietf-http-wg@w3.org>
List-Unsubscribe: <mailto:ietf-http-wg-request@w3.org?subject=unsubscribe>

On Jan 16, 2013, at 10:38 PM, Mark Nottingham wrote:
> On 17/01/2013, at 12:23 PM, Adrien W. de Croy <adrien@qbik.com> wrote:
>> 
>>> The original intent was to leave this open, AIUI, precisely because most content on the Web doesn't provide any freshness information.
>> There's something a bit disturbing about that.
>> 
>> If most web content doesn't provide freshness information (which actually is what we also see), then heuristic freshness calculations are arguably the most important part of caching.
>> 
>> Having the most important part left unspecified and open seems like a bit of a problem for interop.
>> 
>> Where can a web author or cache implementor go to see what to expect the behaviour would be?  Maybe we need a BCP?
> 
> 
> Well, an algorithm could be suggested in a separate document, but we can't require people to follow it. Lots of effort has gone into tweaking heuristics over the years by various caches, and it's often left up to the administrator (e.g., see refresh_pattern in Squid).

Moreover, heuristics ought to be based on local network needs, such
as the cost/load of network bandwidth, the latency required to
perform a conditional GET, and whether or not the cache is shared
by many users.  The heuristics used by a corporate proxy are going
to be very different from those used on a boat (or a spacecraft, for
that matter).

....Roy