Re: [Last-Call] Last Call: <draft-koster-rep-08.txt> (Robots Exclusion Protocol) to Proposed Standard

Hi John,

Thanks for the encouraging words regarding the standardization of this
protocol.

crawl-delay is a highly useless signal to deal with from the perspective of
a crawler and most crawlers decided to rather use a visual setting (i.e. a
slider) in a tool such as the Yandex.Webmaster tool and Google Search
Console a long time ago. That's why we decided to not even mention it in
the draft. We did explicitly allow implementors to define their own custom
rules, but we brought up the sitemap rule as an example. We can add
crawl-delay as a secondary example for a custom rule if you think that
would be useful

The Sitemap rule is very useful in general, however it cannot be a
directive, which the explicitly defined "disallow" and "allow" are. This is
because different implementors approach sitemap ingestion vastly
differently to protect their own systems; for example, one search engine
might not ingest "anonymous" sitemap submissions such as the robots.txt
submissions at all, or they might not ingest from spam ridden domains at
all. Basically it's merely a hint, so we decided to give a nod to sitemaps
for it's one of the most prominent rules in existing robots.txt files, and
just allow implementors to decide how they're implementing and using this
custom rule.

Happy to chat more about this, however each of these decisions were made
after months of discussions and agreed on by large consumers of robots.txt
files.

On Thu, May 5, 2022 at 10:35 PM John Levine <johnl@taugh.com> wrote:

> It appears that The IESG  <last-call@ietf.org> said:
> >
> >The IESG has received a request from an individual submitter to consider
> the
> >following document: - 'Robots Exclusion Protocol'
> >  <draft-koster-rep-08.txt> as Proposed Standard
> >
> >The IESG plans to make a decision in the next few weeks, and solicits
> final
> >comments on this action. Please send substantive comments to the
> >last-call@ietf.org mailing lists by 2022-06-02. Exceptionally, comments
> may
> >be sent to iesg@ietf.org instead. In either case, please retain the
> beginning
> >of the Subject line to allow automated sorting.
>
> I'm glad to see that the authors agreed to move this to standards
> track where it belongs, and I encourage you to publish it.
>
> Nits: it would be nice if it explicitly described the Crawl-delay: and
> Sitemap: lines which are widely implemented even if not by the
> authors' employer.
>
> R's,
> John
>
> --
> last-call mailing list
> last-call@ietf.org
> https://www.ietf.org/mailman/listinfo/last-call
>