IETF Logo Mail Archive

Re: [Last-Call] Last Call: <draft-koster-rep-06.txt> (Robots Exclusion Protocol) to Informational RFC

John Levine <johnl@taugh.com> Wed, 16 March 2022 02:05 UTC

Return-Path: <johnl@iecc.com>
X-Original-To: last-call@ietfa.amsl.com
Delivered-To: last-call@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 375113A12BD for <last-call@ietfa.amsl.com>; Tue, 15 Mar 2022 19:05:25 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.862
X-Spam-Level:
X-Spam-Status: No, score=-1.862 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.248, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=iecc.com header.b=XFeVe2Jr; dkim=pass (2048-bit key) header.d=taugh.com header.b=f688wDp8
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fN8m3DnBnl0b for <last-call@ietfa.amsl.com>; Tue, 15 Mar 2022 19:05:19 -0700 (PDT)
Received: from gal.iecc.com (gal.iecc.com [IPv6:2001:470:1f07:1126:0:43:6f73:7461]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 334F73A12B8 for <last-call@ietf.org>; Tue, 15 Mar 2022 19:05:18 -0700 (PDT)
Received: (qmail 59851 invoked from network); 16 Mar 2022 02:05:17 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=iecc.com; h=date:message-id:from:to:cc:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:cleverness; s=e9c9.623145dd.k2203; bh=IXt9zKydiRueon2CI4XQgTuO2f5bBpSEvyD+VoqhDbQ=; b=XFeVe2Jr+xct48yZLO1Ib+d3iqGzburz28m/w36nUoS8b1nuBYRH7XxvMCQt20t3Sl4EsrvR+JAsenxOUuHHh0CpCMWpZvF09cKisoUFeaOe/T1wbfF8FSnahvRErt891c6dT1OLVA+BXp8SH8j79X1kvxc23DYNG4rL8A942X81vAVrpG0bffs44DZfA7qDEa+koG5UgW3Z7AGXhGqJnoGS15+J2nE8hnQ1qnKYIWifTCOafUDtcEnblk50vv594cRvsxWv8ll5C1ks/LUrtusSGA2VobIEqdhkPYQRa7pge6Xzxb4z3B/V26s7JunKwlQ0NlgmYPWN3n+CvIyeaw==
DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=taugh.com; h=date:message-id:from:to:cc:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:cleverness; s=e9c9.623145dd.k2203; bh=IXt9zKydiRueon2CI4XQgTuO2f5bBpSEvyD+VoqhDbQ=; b=f688wDp8QrnFk/4mcovjcvK5Pv3ZDdAxR+NlC9mL8YzEyiP2eTpLzE/z7RBzEXgeXdeW98ByMNUk24tuzWPUL539TgrfTzGMPOrx67ETXKpz+bZjd8KrCNDjH2noBcZYnt4zXIIawmcwP4yghRPhu81Q0lxXMBl/FsHvdyf+I7re+7uB0eh0G8HNVjIBSXcMsyG1exIV6bSKG4aOSWn5eeUVbGJNQGZZeMcnoeU83QHNLldcxPJYMAktXmSHCIJg9QMtTn4LR6vPeyTM+yxg1gfZddG67/Dyyro7Bas6bLXFROc4StuhN1n271N7DR0aRvLO4Cmg2lkMDC/n7V52Mw==
Received: from ary.qy ([IPv6:2001:470:1f07:1126::78:696d:6170]) by imap.iecc.com ([IPv6:2001:470:1f07:1126::78:696d:6170]) with ESMTPS (TLS1.2 ECDHE-RSA AES-256-GCM AEAD) via TCP6; 16 Mar 2022 02:05:16 -0000
Received: by ary.qy (Postfix, from userid 501) id D6781393B632; Tue, 15 Mar 2022 22:05:13 -0400 (EDT)
Date: 15 Mar 2022 22:05:13 -0400
Message-Id: <20220316020513.D6781393B632@ary.qy>
From: "John Levine" <johnl@taugh.com>
To: last-call@ietf.org
Cc: mnot@mnot.net
In-Reply-To: <888588E5-D0F2-4BFE-9EBB-68E7C7FC4744@mnot.net>
Organization: Taughannock Networks
X-Headerized: yes
Cleverness: minimal
Mime-Version: 1.0
Content-type: text/plain; charset=utf-8
Content-transfer-encoding: 8bit
Archived-At: <https://mailarchive.ietf.org/arch/msg/last-call/iDcLSaOwTerkDv8bOEcR96ko4XI>
Subject: Re: [Last-Call] Last Call: <draft-koster-rep-06.txt> (Robots Exclusion Protocol) to Informational RFC
X-BeenThere: last-call@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: IETF Last Calls <last-call.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/last-call>, <mailto:last-call-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/last-call/>
List-Post: <mailto:last-call@ietf.org>
List-Help: <mailto:last-call-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/last-call>, <mailto:last-call-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 16 Mar 2022 02:05:26 -0000

It appears that Mark Nottingham  <mnot@mnot.net> said:
>
>
>> On 16 Mar 2022, at 3:52 am, Murray S. Kucherawy <superuser@gmail.com> wrote:
>> 
>> If the authors are prepared to make this change, I'd like to hear why "IETF Standards Track w/consensus" is (or
>should be) off the table, especially if other search engines are going to indicate support.
>
>That's reasonable, if support eventuates. If it doesn't, it would be pretty awkward -- I think we'd want at least
>some level of review or at least acknowledgement by other major implementers before declaring it Standards Track.

A lot of search engines describe how they handle robots.txt. For example,

Yahoo https://help.yahoo.com/kb/SLN22600.html

Bing https://www.bing.com/webmasters/help/how-to-create-a-robots-txt-file-cb7c31ec

Yandex https://yandex.com/support/webmaster/controlling-robot/robots-txt.html

Baidu https://www.baidu.com/search/robots_english.html

DataForSEO https://dataforseo.com/dataforseo-bot

Petal Search https://webmaster.petalsearch.com/site/petalbot

Mojeek https://www.mojeek.com/bot.html

(Those last three I never heard of but their spiders show up in my logs.)

>From what I can see, the current draft is OK, but I would make it clearer that the * and $ metacharacters
aren't widely supported, and I would add Crawl-Delay and Siteindex directives which are.

R's,
John

v2.8.7 | Report a Bug | By Email