IETF Logo Mail Archive

[weirds] Search Engines Indexing RDAP Server Content

"Hollenbeck, Scott" <shollenbeck@verisign.com> Fri, 29 January 2016 13:34 UTC

Return-Path: <shollenbeck@verisign.com>
X-Original-To: weirds@ietfa.amsl.com
Delivered-To: weirds@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E7B251ACD1F for <weirds@ietfa.amsl.com>; Fri, 29 Jan 2016 05:34:42 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.9
X-Spam-Level:
X-Spam-Status: No, score=-1.9 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1] autolearn=ham
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 60YNMa69M4-y for <weirds@ietfa.amsl.com>; Fri, 29 Jan 2016 05:34:40 -0800 (PST)
Received: from mail-qg0-x264.google.com (mail-qg0-x264.google.com [IPv6:2607:f8b0:400d:c04::264]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 7C3261ACC8D for <weirds@ietf.org>; Fri, 29 Jan 2016 05:34:40 -0800 (PST)
Received: by mail-qg0-x264.google.com with SMTP id b35so4247651qge.0 for <weirds@ietf.org>; Fri, 29 Jan 2016 05:34:40 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=verisign-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:thread-topic:thread-index:date:message-id :accept-language:content-language:content-type :content-transfer-encoding:mime-version; bh=sCffLiYHSitBLsmgYx6H4GfHdXd/DpitP5gPV+LPmt8=; b=mJp5TSkm5iUPvUCHp223D+ItZF8w3H3OenIqEptp15VlIjh/DplQHdHAh3xCvW/aHl 1hZp4DK/VQGqKf0KtVcmlon3v779Gtva9SICNdmKfqR845WIfmD3IWmD8xCU03DBc4qL RdaAp6dwoAtu71tZf3vPewRsdixm2yL55l0qdiZz7MQXgzU2yygdKxlNiq14UypMMeHt w7ZDQjS8SotzWLVQeWJfzg+WyUwZ4+FN5noGV3xyQp55hgjl+KGn9z2iYOubt6uB/gSo fMgXFMuitS9fDxqsN4QrSMmfebkn9nR+sjSdOJuTlcAoSaIc9iWhanIVTWfcpdJG82UT dU1Q==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:subject:thread-topic:thread-index:date :message-id:accept-language:content-language:content-type :content-transfer-encoding:mime-version; bh=sCffLiYHSitBLsmgYx6H4GfHdXd/DpitP5gPV+LPmt8=; b=fmT1Ev2Z1+Mu6N+6aVR5zUdgQH8kQDQ+9vGNwAmHnMtqH/BUOb8LoZkjQgJQ43BZ8T kE5r+VhZ1GKDd//jPvzp8MsaKT373jqSxFge8sMddvwCdk/fyHqfRtWxdrhSum2SkRVs yhlZvkBmopubQ0u9kOZo0Grn5Tltg0669Yu2t3kyZPWyclwTOxFv19TGcQj5PrcUCFaN +I6EQuM5F76xC50prQsc9jH4pJE9nH8Z0dFqSrxpZf4b22IlhJWBs7p7gb5cgGsL4gOD 3+rvZxndyMUabIUKR9rpCZifvydig06wglGoVsm8xjNm74n3Y6bxGtQcJBsPuK+OzFQJ Nseg==
X-Gm-Message-State: AG10YORtE8ur6/+PBXja6B4TZa3vl551MAAAQYgzU+ZlzWuUFUlN2FfVOyw59s05RtVghg1s9tEhnBJAULZqZQwcMhPlaaif
X-Received: by 10.55.26.217 with SMTP id l86mr10183049qkh.103.1454074479621; Fri, 29 Jan 2016 05:34:39 -0800 (PST)
Received: from brn1lxmailout02.verisign.com (brn1lxmailout02.verisign.com. [72.13.63.42]) by smtp-relay.gmail.com with ESMTPS id z142sm1449891qka.11.2016.01.29.05.34.39 (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 29 Jan 2016 05:34:39 -0800 (PST)
X-Relaying-Domain: verisign.com
Received: from BRN1WNEXCHM01.vcorp.ad.vrsn.com (brn1wnexchm01 [10.173.152.255]) by brn1lxmailout02.verisign.com (8.13.8/8.13.8) with ESMTP id u0TDYd1k009430 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=FAIL); Fri, 29 Jan 2016 08:34:39 -0500
Received: from BRN1WNEXMBX01.vcorp.ad.vrsn.com ([::1]) by BRN1WNEXCHM01.vcorp.ad.vrsn.com ([::1]) with mapi id 14.03.0174.001; Fri, 29 Jan 2016 08:34:36 -0500
From: "Hollenbeck, Scott" <shollenbeck@verisign.com>
To: "weirds@ietf.org" <weirds@ietf.org>, "regops@nlnetlabs.nl" <regops@nlnetlabs.nl>, "gtld-tech@icann.org" <gtld-tech@icann.org>
Thread-Topic: Search Engines Indexing RDAP Server Content
Thread-Index: AdFamcnpcIqNwxGiSxmdhJxaDQVVfw==
Date: Fri, 29 Jan 2016 13:34:35 +0000
Message-ID: <831693C2CDA2E849A7D7A712B24E257F4A149BDE@BRN1WNEXMBX01.vcorp.ad.vrsn.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [10.173.152.4]
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Archived-At: <http://mailarchive.ietf.org/arch/msg/weirds/x8nee6xzT1RKyj3or9Nc1QW8H7U>
Subject: [weirds] Search Engines Indexing RDAP Server Content
X-BeenThere: weirds@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: "WHOIS-based Extensible Internet Registration Data Service \(WEIRDS\)" <weirds.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/weirds>, <mailto:weirds-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/weirds/>
List-Post: <mailto:weirds@ietf.org>
List-Help: <mailto:weirds-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/weirds>, <mailto:weirds-request@ietf.org?subject=subscribe>
X-List-Received-Date: Fri, 29 Jan 2016 13:34:43 -0000

(cross-posting to multiple lists - sorry if that's inconvenient)

So I saw a tweet from Gavin Brown (@GavinBrown) that describes how one particular search engine has indexed the RDAP server of a gTLD registry operator:

https://twitter.com/GavinBrown/status/692718904058191872

This is all the more reason to work on a client authentication specification that includes support for varying responses based on client identity and authorization. I've been working on such a specification and welcome feedback on the approach:

https://datatracker.ietf.org/doc/draft-hollenbeck-weirds-rdap-openid/

It also begs the question of the need for a BCP describing operational practices for server operators. There are ways for web servers to influence or restrict crawler behavior, but what's appropriate in this context?

Scott

v2.23.0 | Report a Bug | By Email