Re: [weirds] [Regops] Search Engines Indexing RDAP Server Content

"John Levine" <> Fri, 29 January 2016 17:24 UTC

Return-Path: <>
Received: from localhost ( []) by (Postfix) with ESMTP id CFA9A1A88BD for <>; Fri, 29 Jan 2016 09:24:03 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: 1.664
X-Spam-Level: *
X-Spam-Status: No, score=1.664 tagged_above=-999 required=5 tests=[BAYES_50=0.8, HELO_MISMATCH_COM=0.553, HOST_MISMATCH_NET=0.311, KHOP_DYNAMIC=0.001, SPF_PASS=-0.001] autolearn=no
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id s0E_-SFJjY7C for <>; Fri, 29 Jan 2016 09:24:02 -0800 (PST)
Received: from ( [IPv6:2001:470:1f06:1126::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by (Postfix) with ESMTPS id F34A81A88B4 for <>; Fri, 29 Jan 2016 09:24:01 -0800 (PST)
Received: (qmail 83775 invoked from network); 29 Jan 2016 17:24:00 -0000
Received: from unknown ( by with QMQP; 29 Jan 2016 17:24:00 -0000
Date: 29 Jan 2016 17:23:38 -0000
Message-ID: <20160129172338.51466.qmail@ary.lan>
From: "John Levine" <>
In-Reply-To: <>
X-Headerized: yes
Mime-Version: 1.0
Content-type: text/plain; charset=utf-8
Content-transfer-encoding: 8bit
Archived-At: <>
Subject: Re: [weirds] [Regops] Search Engines Indexing RDAP Server Content
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: "WHOIS-based Extensible Internet Registration Data Service \(WEIRDS\)" <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Fri, 29 Jan 2016 17:24:04 -0000

>We have a difference of opinion, because I tend to think that it's a *good* idea to restrict access
>to *some* data based on a client's identify and level of authorization. I do not want *my* PII
>appearing in search engine search results because it's accessible via RDAP.

Sorry, but I really don't understand what you want here.  There have
been vertical WHOIS search engines forever that let you search WHOIS
data in various ways.  Most of them charge money, but as often as not
some access is free.  That horse left the barn a decade ago.

The stuff that search engines can see is exactly the same as what
casual users see.  If you're saying that you want to make some stuff
unavailable to anyone who doesn't have a password, OK, but now aren't
you running into the whole ICANN mess of what's in WHOIS and what


PS: On the third hand, if you want to make it slightly harder for
casual searches to find RDAP info, publishing a robots.txt that says
"Disallow: /" should take about 30 seconds. and will keep the entire
site out of all of the legit search engines.