Re: [weirds] Search Engines Indexing RDAP Server Content

"Hollenbeck, Scott" <shollenbeck@verisign.com> Wed, 03 February 2016 13:35 UTC

Return-Path: <shollenbeck@verisign.com>
X-Original-To: weirds@ietfa.amsl.com
Delivered-To: weirds@ietfa.amsl.com
Received: from localhost (ietfa.amsl.com [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 281BE1AC3D5 for <weirds@ietfa.amsl.com>; Wed, 3 Feb 2016 05:35:18 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.9
X-Spam-Level:
X-Spam-Status: No, score=-1.9 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1] autolearn=ham
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KCuI6WuaTLzA for <weirds@ietfa.amsl.com>; Wed, 3 Feb 2016 05:35:15 -0800 (PST)
Received: from mail-qg0-x264.google.com (mail-qg0-x264.google.com [IPv6:2607:f8b0:400d:c04::264]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 9D0001A8AB6 for <weirds@ietf.org>; Wed, 3 Feb 2016 05:35:15 -0800 (PST)
Received: by mail-qg0-x264.google.com with SMTP id t74so2246970qgt.2 for <weirds@ietf.org>; Wed, 03 Feb 2016 05:35:15 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=verisign-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:thread-topic:thread-index:date:message-id :references:in-reply-to:accept-language:content-language :content-type:content-transfer-encoding:mime-version; bh=BeizAIe9vByArsZsriT5E1DtG06lpuh9LhlLGZfGkBA=; b=ylqzjleGQEpRdpn+SzSCWeuix7xEBaqfnq2agapIdStfcxk492t7Ts9aY0YgQhuvyr qQwXnhMPUWWA1YHJF2fhBtQLECjc7Ya66c2xa0Mk/EHIQX2RwIzHhwEpe5HSS9PizI2G 3x7Uf9BHUPwFHzGmj0L3DfkCo8bwY7nK8Ag1a3NelTz61HGRuP6Bz+mLyP5jRGQeBjKd gYZUzr18TvjEfGrg6+2jZIh7scmV3kdGxRP7FTufb76wfcZE8sGqIhjOAC7XwSu1I/Pu UHMWAuoFzrOxrgLegnC7Xhfjjz//cs1G59L8fL3lZec4r0fRw0NSWcE3TOWK0ghna+XL y+Lg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:thread-topic:thread-index :date:message-id:references:in-reply-to:accept-language :content-language:content-type:content-transfer-encoding :mime-version; bh=BeizAIe9vByArsZsriT5E1DtG06lpuh9LhlLGZfGkBA=; b=Qn/+aZM+pKIBIvOSEnfeTQvZsf1cXAMnM03c1Wo4+/3cz4hHJAI0DuSj9eMMDowcvq M0KV5nO9ZyEmbPTGtWfG+70za7RrPoy1e9Pp9SUwz+ZH5HV+4g1xtmXNp1p16ufR4TAY AxAcvdGjCdbOZqtQ3Fm51GnVFwPThQ79+nrI89qLndG+Yn1Z9SHAzk35mJH8coWE3xGr S3IPlUxWhBijVByIHlu/tVREWpyLuoB8Bytn2wyuJoc4b2so4EoOY8zptDTNdctAH9kn fxDXRvBNzNESLzmkpd1SlcMWuTtWD2b5s0C+ALVondFrSXeCeGvNiOuN1yZT0UpvM2Ea nzoQ==
X-Gm-Message-State: AG10YORFA0NTked736DJFu08RphZ8MLAeuQ0oiVElH+ix2+SIfhBtpKLh22tgQ7cA3dkm0y6P/med3ddEC4RD99ZVKTubPOP
X-Received: by 10.55.73.6 with SMTP id w6mr1527835qka.82.1454506514846; Wed, 03 Feb 2016 05:35:14 -0800 (PST)
Received: from brn1lxmailout01.verisign.com (brn1lxmailout01.verisign.com. [72.13.63.41]) by smtp-relay.gmail.com with ESMTPS id d123sm869029qka.12.2016.02.03.05.35.14 (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 03 Feb 2016 05:35:14 -0800 (PST)
X-Relaying-Domain: verisign.com
Received: from brn1wnexcas01.vcorp.ad.vrsn.com (brn1wnexcas01 [10.173.152.205]) by brn1lxmailout01.verisign.com (8.13.8/8.13.8) with ESMTP id u13DZE9b032222 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=FAIL); Wed, 3 Feb 2016 08:35:14 -0500
Received: from BRN1WNEXMBX01.vcorp.ad.vrsn.com ([::1]) by brn1wnexcas01.vcorp.ad.vrsn.com ([::1]) with mapi id 14.03.0174.001; Wed, 3 Feb 2016 08:35:13 -0500
From: "Hollenbeck, Scott" <shollenbeck@verisign.com>
To: James Galvin <galvin@elistx.com>
Thread-Topic: [weirds] Search Engines Indexing RDAP Server Content
Thread-Index: AQHRXoMCdNOC6XCuwkSoj8cnxWSTuZ8aUGeQ
Date: Wed, 03 Feb 2016 13:35:13 +0000
Message-ID: <831693C2CDA2E849A7D7A712B24E257F4A150EF0@BRN1WNEXMBX01.vcorp.ad.vrsn.com>
References: <831693C2CDA2E849A7D7A712B24E257F4A149BDE@BRN1WNEXMBX01.vcorp.ad.vrsn.com> <BAE1B437-A834-4E30-9A3D-DAA3B209E35B@elistx.com>
In-Reply-To: <BAE1B437-A834-4E30-9A3D-DAA3B209E35B@elistx.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [10.173.152.4]
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
MIME-Version: 1.0
Archived-At: <http://mailarchive.ietf.org/arch/msg/weirds/BwP6EfqIU9t_UVrw_bTSJLZs4SM>
Cc: "regops@nlnetlabs.nl" <regops@nlnetlabs.nl>, "gtld-tech@icann.org" <gtld-tech@icann.org>, "weirds@ietf.org" <weirds@ietf.org>
Subject: Re: [weirds] Search Engines Indexing RDAP Server Content
X-BeenThere: weirds@ietf.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: "WHOIS-based Extensible Internet Registration Data Service \(WEIRDS\)" <weirds.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/weirds>, <mailto:weirds-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/weirds/>
List-Post: <mailto:weirds@ietf.org>
List-Help: <mailto:weirds-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/weirds>, <mailto:weirds-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 03 Feb 2016 13:35:18 -0000

> -----Original Message-----
> From: James Galvin [mailto:galvin@elistx.com]
> Sent: Wednesday, February 03, 2016 8:01 AM
> To: Hollenbeck, Scott
> Cc: weirds@ietf.org; regops@nlnetlabs.nl; gtld-tech@icann.org
> Subject: Re: [weirds] Search Engines Indexing RDAP Server Content
> 
> This is not a very good example of the need for operational practices.
> 
> It represents Google UK’s view of what you can find when you start at
> “http://rdg.afilias.info/rdap/help”, which includes explicit
> examples of how to use the RDAP server.

Jim, you're probably right about it not being a good example. I'm more concerned with the information that could be returned in response to an entity query being archived.

Side note: the response received for http://rdg.afilias.info/rdap/help is encoded using HTML. Sections 7 and 4.3 of RFC 7483 say it should be JSON-encoded.

"The appropriate response to /help queries as defined by [RFC7482] is to use the notices structure as defined in Section 4.3."

Would it have been indexed at all if it wasn't HTML? There's an operational practice thing in here somewhere... ;)

Scott