[rfc-dist] RFC 9309 on Robots Exclusion Protocol
rfc-editor@rfc-editor.org Sat, 10 September 2022 00:50 UTC
Return-Path: <wwwrun@rfcpa.amsl.com>
X-Original-To: rfc-dist@ietfa.amsl.com
Delivered-To: rfc-dist@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 2E4F1C1522BA for <ietf-announce@ietfa.amsl.com>; Fri, 9 Sep 2022 17:50:48 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -6.658
X-Spam-Level:
X-Spam-Status: No, score=-6.658 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HEADER_FROM_DIFFERENT_DOMAINS=0.249, RCVD_IN_DNSWL_HI=-5, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_BLOCKED=0.001, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gS09rieUCZSo for <ietf-announce@ietfa.amsl.com>; Fri, 9 Sep 2022 17:50:44 -0700 (PDT)
Received: from rfcpa.amsl.com (rfc-editor.org [50.223.129.200]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 821A8C14CE2E for <ietf-announce@ietf.org>; Fri, 9 Sep 2022 17:50:44 -0700 (PDT)
Received: by rfcpa.amsl.com (Postfix, from userid 499) id 393F7121926; Fri, 9 Sep 2022 17:50:44 -0700 (PDT)
To: ietf-announce@ietf.org, rfc-dist@rfc-editor.org
Subject: [rfc-dist] RFC 9309 on Robots Exclusion Protocol
From: rfc-editor@rfc-editor.org
Cc: rfc-editor@rfc-editor.org, drafts-update-ref@iana.org
Content-type: text/plain; charset="UTF-8"
Message-Id: <20220910005044.393F7121926@rfcpa.amsl.com>
Date: Fri, 09 Sep 2022 17:50:44 -0700
Archived-At: <https://mailarchive.ietf.org/arch/msg/rfc-dist/dg0n7W9M7TVlhkUlN6X8XJ8-_lc>
X-BeenThere: rfc-dist@rfc-editor.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: RFC Announcements <rfc-dist.rfc-editor.org>
List-Unsubscribe: <https://mailman.rfc-editor.org/mailman/options/rfc-dist>, <mailto:rfc-dist-request@rfc-editor.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/rfc-dist/>
List-Post: <mailto:rfc-dist@rfc-editor.org>
List-Help: <mailto:rfc-dist-request@rfc-editor.org?subject=help>
List-Subscribe: <https://mailman.rfc-editor.org/mailman/listinfo/rfc-dist>, <mailto:rfc-dist-request@rfc-editor.org?subject=subscribe>
X-List-Received-Date: Sat, 10 Sep 2022 00:50:48 -0000
A new Request for Comments is now available in online RFC libraries.
RFC 9309
Title: Robots Exclusion Protocol
Author: M. Koster,
G. Illyes,
H. Zeller,
L. Sassman
Status: Standards Track
Stream: IETF
Date: September 2022
Mailbox: m.koster@greenhills.co.uk,
garyillyes@google.com,
henner@google.com,
lizzi@google.com
Pages: 12
Updates/Obsoletes/SeeAlso: None
I-D Tag: draft-koster-rep-12.txt
URL: https://www.rfc-editor.org/info/rfc9309
DOI: 10.17487/RFC9309
This document specifies and extends the "Robots Exclusion Protocol"
method originally defined by Martijn Koster in 1994 for service
owners to control how content served by their services may be
accessed, if at all, by automatic clients known as crawlers.
Specifically, it adds definition language for the protocol,
instructions for handling errors, and instructions for caching.
This is now a Proposed Standard.
STANDARDS TRACK: This document specifies an Internet Standards Track
protocol for the Internet community, and requests discussion and suggestions
for improvements. Please refer to the current edition of the Official
Internet Protocol Standards (https://www.rfc-editor.org/standards) for the
standardization state and status of this protocol. Distribution of this
memo is unlimited.
This announcement is sent to the IETF-Announce and rfc-dist lists.
To subscribe or unsubscribe, see
https://www.ietf.org/mailman/listinfo/ietf-announce
https://mailman.rfc-editor.org/mailman/listinfo/rfc-dist
For searching the RFC series, see https://www.rfc-editor.org/search
For downloading RFCs, see https://www.rfc-editor.org/retrieve/bulk
Requests for special distribution should be addressed to either the
author of the RFC in question, or to rfc-editor@rfc-editor.org. Unless
specifically noted otherwise on the RFC itself, all RFCs are for
unlimited distribution.
The RFC Editor Team
Association Management Solutions, LLC