RFC 9309 on Robots Exclusion Protocol
rfc-editor@rfc-editor.org Sat, 10 September 2022 00:50 UTC
Return-Path: <wwwrun@rfcpa.amsl.com>
X-Original-To: ietf-announce@ietfa.amsl.com
Delivered-To: ietf-announce@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2E4F1C1522BA for <ietf-announce@ietfa.amsl.com>; Fri, 9 Sep 2022 17:50:48 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -6.658
X-Spam-Level:
X-Spam-Status: No, score=-6.658 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HEADER_FROM_DIFFERENT_DOMAINS=0.249, RCVD_IN_DNSWL_HI=-5, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_BLOCKED=0.001, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gS09rieUCZSo for <ietf-announce@ietfa.amsl.com>; Fri, 9 Sep 2022 17:50:44 -0700 (PDT)
Received: from rfcpa.amsl.com (rfc-editor.org [50.223.129.200]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 821A8C14CE2E for <ietf-announce@ietf.org>; Fri, 9 Sep 2022 17:50:44 -0700 (PDT)
Received: by rfcpa.amsl.com (Postfix, from userid 499) id 393F7121926; Fri, 9 Sep 2022 17:50:44 -0700 (PDT)
To: ietf-announce@ietf.org, rfc-dist@rfc-editor.org
Subject: RFC 9309 on Robots Exclusion Protocol
From: rfc-editor@rfc-editor.org
Cc: rfc-editor@rfc-editor.org, drafts-update-ref@iana.org
Content-type: text/plain; charset="UTF-8"
Message-Id: <20220910005044.393F7121926@rfcpa.amsl.com>
Date: Fri, 09 Sep 2022 17:50:44 -0700
Archived-At: <https://mailarchive.ietf.org/arch/msg/ietf-announce/dII0BICq10GuXdtYWoSCbRDv0eE>
X-BeenThere: ietf-announce@ietf.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: "IETF announcement list. No discussions." <ietf-announce.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/ietf-announce>, <mailto:ietf-announce-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ietf-announce/>
List-Post: <mailto:ietf-announce@ietf.org>
List-Help: <mailto:ietf-announce-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ietf-announce>, <mailto:ietf-announce-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 10 Sep 2022 00:50:48 -0000
A new Request for Comments is now available in online RFC libraries. RFC 9309 Title: Robots Exclusion Protocol Author: M. Koster, G. Illyes, H. Zeller, L. Sassman Status: Standards Track Stream: IETF Date: September 2022 Mailbox: m.koster@greenhills.co.uk, garyillyes@google.com, henner@google.com, lizzi@google.com Pages: 12 Updates/Obsoletes/SeeAlso: None I-D Tag: draft-koster-rep-12.txt URL: https://www.rfc-editor.org/info/rfc9309 DOI: 10.17487/RFC9309 This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching. This is now a Proposed Standard. STANDARDS TRACK: This document specifies an Internet Standards Track protocol for the Internet community, and requests discussion and suggestions for improvements. Please refer to the current edition of the Official Internet Protocol Standards (https://www.rfc-editor.org/standards) for the standardization state and status of this protocol. Distribution of this memo is unlimited. This announcement is sent to the IETF-Announce and rfc-dist lists. To subscribe or unsubscribe, see https://www.ietf.org/mailman/listinfo/ietf-announce https://mailman.rfc-editor.org/mailman/listinfo/rfc-dist For searching the RFC series, see https://www.rfc-editor.org/search For downloading RFCs, see https://www.rfc-editor.org/retrieve/bulk Requests for special distribution should be addressed to either the author of the RFC in question, or to rfc-editor@rfc-editor.org. Unless specifically noted otherwise on the RFC itself, all RFCs are for unlimited distribution. The RFC Editor Team Association Management Solutions, LLC
- RFC 9309 on Robots Exclusion Protocol rfc-editor