[ai-control] prevent robots.txt entries from becoming law

[Brewster Kahle <[e-mail]>]

Putting publishers online in the early 90’s at WAIS Inc and then running the Internet Archive, I was involved when robots.txt started, and have had to deal with it for 30 years.   Yes, there are issues around AI, but let’s not lose some basic principles of IETF now.  Sorry to chime in late, but it is important.

The proposal to insert AI language hijacks robots.txt and twists it into a tool for monetizing the internet. It also invites people to add it into laws (EU seems moving there). We successfully fought off efforts for decades (yes, there were those wanting to make robots.txt into law in the US).

The only way around this is to not add in the AI components into the standard, and let anyone voluntarily use it as they will (already happening).

To be more specific:

Robots.txt seemed like a great thing in 1994.  Keep robots from inadvertently moving cameras in coffee-pot watching websites, modifying mailing lists, etc.  It worked for a while as a voluntary system, but as time went along it did not even reflect the policies of the websites.   Then we got compound websites (think geocities and facebook etc) and robots.txt gives inappropriate power for the platforms.  

Often robots.txt files are put in place by a technical person in response to a temporary technical issue, then never rescinded.  For instance, I talked repeatedly with a deputy librarian of congress about a robot exclusion on a subsite of public domain materials that had been placed on the LOC.gov <http://loc.gov/> website.  This was not the policy of the Library of Congress. He could not, in practice, get it removed from the LOC website, no matter how he tried.   This is replicated over and over across the web, making robots.txt a less effective tool over time.   Making it a matter of law would not serve website owners well, legitimate crawlers well, and certainly not users of the WWW.

Having anything in robots.txt reflected in law is such a bad idea, that I suggest we, the Internet standards community, do our best to make it not happen.  (I do understand the temptation to think we should make law, but others might disagree :) )   So let's not add any AI features into this standard.

-brewster kahle
Founder, Digital Librarian
Internet Archive