IETF Logo Mail Archive

Re: [Last-Call] Last Call: <draft-koster-rep-06.txt> (Robots Exclusion Protocol) to Informational RFC

John R Levine <johnl@taugh.com> Wed, 09 March 2022 02:28 UTC

Return-Path: <johnl@taugh.com>
X-Original-To: last-call@ietfa.amsl.com
Delivered-To: last-call@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D7C6F3A011B for <last-call@ietfa.amsl.com>; Tue, 8 Mar 2022 18:28:20 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.11
X-Spam-Level:
X-Spam-Status: No, score=-2.11 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=iecc.com header.b=Pe35tN3L; dkim=pass (2048-bit key) header.d=taugh.com header.b=Tr6dU913
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id i9KvFQWILbUt for <last-call@ietfa.amsl.com>; Tue, 8 Mar 2022 18:28:16 -0800 (PST)
Received: from gal.iecc.com (gal.iecc.com [IPv6:2001:470:1f07:1126:0:43:6f73:7461]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id BD0CF3A00E1 for <last-call@ietf.org>; Tue, 8 Mar 2022 18:28:15 -0800 (PST)
Received: (qmail 65159 invoked from network); 9 Mar 2022 02:28:13 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=iecc.com; h=date:message-id:from:to:cc:subject:in-reply-to:references:mime-version:content-type; s=fe85.622810bd.k2203; bh=OVMOxyCExIrRVyePVyNpNWKKk08Khzka1WhhtY4WGCY=; b=Pe35tN3LiU1YNC7EewJxs0lmMQbLm7Sdl0N0ROMZT6oRqVChnKdFIgROrBTKeOghKgtPfwpkhtJaxjlvvkUG8aSKYbPYETQWyNX+7xCTarffP+5EU0PCNLmh++LzXdWpiekLEtb36syOt5O0O7lt3O4RscBMnXlu5LlLNFJBKKjKPYaof/tBnGrDcHeTJSjnYslIIr0zdQ7KfnMGFeKf7r2MPfMD8otD69qpGh47W0IqcFA9nmbRlVVwtW48iQEWC5R5Y+FXFP/ATVPtlU5fnz0HQJy5sPdoPf4xaHY91tcUkY2D3D8tdIE1M/ZP+P/EXUbRow8yD/GFxWMcxboi2w==
DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=taugh.com; h=date:message-id:from:to:cc:subject:in-reply-to:references:mime-version:content-type; s=fe85.622810bd.k2203; bh=OVMOxyCExIrRVyePVyNpNWKKk08Khzka1WhhtY4WGCY=; b=Tr6dU913PVQh6r7aZPIdj3jyFuoXGPvDa1dmCIXDmJtv29QQUpuuh/QSK73yrRwRdHS9inYYhQqj1gum2aMfM/2kYbD7iU324vP4cN1pNwYBiG3yovVdTv++LoONvAtrPuOtAqAQ0gRoK6UJrG3UaTThWg341yiQpZmJ6sTcpxHUnjp5j8DRHZUWyUAqzu/bmsPULKuZCES1mYJwkZSZgdcuS9K40OwvVYRf+e7yG6BlRLWVg4QLadBTnhKEZUYCnRmzjQXShcwRRBZfptfTx/R3gJqk0hrHResZkzrv8ieEAGoo1W+ctwmWsYcvkumNxfftAqa9PcoJ8dHVzBleJw==
Received: from ary.qy ([IPv6:2001:470:1f07:1126::78:696d:6170]) by imap.iecc.com ([IPv6:2001:470:1f07:1126::78:696d:6170]) with ESMTPS (TLS1.2 ECDHE-RSA AES-256-GCM AEAD) via TCP6; 09 Mar 2022 02:28:13 -0000
Received: by ary.qy (Postfix, from userid 501) id E472738B5F53; Tue, 8 Mar 2022 21:28:12 -0500 (EST)
Received: from localhost (localhost [127.0.0.1]) by ary.qy (Postfix) with ESMTP id B714A38B5F35; Tue, 8 Mar 2022 21:28:12 -0500 (EST)
Date: 8 Mar 2022 21:28:12 -0500
Message-ID: <3efab652-be64-e179-b387-0468a2da9f1c@taugh.com>
From: "John R Levine" <johnl@taugh.com>
To: "=?UTF-8?Q?Martin_J=2E_D=C3=BCrst?=" <duerst@it.aoyama.ac.jp>
Cc: "last-call@ietf.org" <last-call@ietf.org>
X-X-Sender: johnl@ary.qy
In-Reply-To: <618c8f70-3d09-fe09-6088-597b2b63655e@it.aoyama.ac.jp>
References: <20220228222932.825F33844270@ary.qy> <245C65D2-EC38-4C49-9CA0-3DD687CB37DA@mnot.net> <CA+9kkMAnmoJ0n3mPscZvc6kbyOZjQU78vb+iA0Pw5Qq=_kKZEw@mail.gmail.com> <ee8c0615-9207-cf7a-b1a0-905f33062e7a@taugh.com> <CA+9kkMBn-jJbwKjOdOpLL3PFS0REVUBUoSa+2MD0NxnvttHCcg@mail.gmail.com> <91329874-9301-40EC-8155-FBFE55DB89E4@akamai.com> <618c8f70-3d09-fe09-6088-597b2b63655e@it.aoyama.ac.jp>
MIME-Version: 1.0
Content-Type: text/plain; format=flowed; charset=US-ASCII
Archived-At: <https://mailarchive.ietf.org/arch/msg/last-call/FhyTa-tmv6CpQ1rcMZCicOKG9UE>
Subject: Re: [Last-Call] Last Call: <draft-koster-rep-06.txt> (Robots Exclusion Protocol) to Informational RFC
X-BeenThere: last-call@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: IETF Last Calls <last-call.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/last-call>, <mailto:last-call-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/last-call/>
List-Post: <mailto:last-call@ietf.org>
List-Help: <mailto:last-call-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/last-call>, <mailto:last-call-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 09 Mar 2022 02:28:21 -0000

> The reason it makes sense to allow derivative works is that sometimes (not 
> necessarily in this case, but we don't know in advance), things can change.

There are in fact some widely used extensions to the format described in 
the draft.  Many search engines recognize "Sitemap:" with the location of 
a site map, and "Crawl-delay:" to slow down crawling.  On the other hand, 
a lot don't recognize # $ * in sec 2.2.3 as special.  I don't think we 
need to fix these to publish the draft, but it wouldn't be absurd to 
update it in light of experience.

> I don't like to guess, but it could be that the authors some experience (or 
> hearsay from friends) with another technology standardized in the IETF where 
> there was a lot of discussion, and incorrectly generalized from that.

Someone already suggested moving the file to a different location, which 
considering that all robots.txt files have been in the same place for 25 
years, seems like a bad idea.

Regards,
John Levine, johnl@taugh.com, Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

v2.8.7 | Report a Bug | By Email