What is the right way to do Web Services discovery?

Phillip Hallam-Baker <phill@hallambaker.com> Tue, 22 November 2016 15:04 UTC

Return-Path: <hallam@gmail.com>
X-Original-To: ietf@ietfa.amsl.com
Delivered-To: ietf@ietfa.amsl.com
Received: from localhost (localhost []) by ietfa.amsl.com (Postfix) with ESMTP id 9B62E129661 for <ietf@ietfa.amsl.com>; Tue, 22 Nov 2016 07:04:40 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.399
X-Spam-Status: No, score=-2.399 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_FORGED_FROMDOMAIN=0.199, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([]) by localhost (ietfa.amsl.com []) (amavisd-new, port 10024) with ESMTP id rsf-8UEa4g4N for <ietf@ietfa.amsl.com>; Tue, 22 Nov 2016 07:04:37 -0800 (PST)
Received: from mail-wm0-x236.google.com (mail-wm0-x236.google.com [IPv6:2a00:1450:400c:c09::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 39FD9129565 for <ietf@ietf.org>; Tue, 22 Nov 2016 07:04:37 -0800 (PST)
Received: by mail-wm0-x236.google.com with SMTP id t79so29808350wmt.0 for <ietf@ietf.org>; Tue, 22 Nov 2016 07:04:37 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:from:date:message-id:subject:to; bh=jiYYaVhnW+ikR+pG3FOZpAMhiJJLyRL2ya76hT7cZp8=; b=HWu+E4x2QgDAVz7Faei+pZiVOmWdUjz9DGFIPF7IJdA15kQN3NC+o9oe6NGZ/9gWcl OgzVBkzHgg9cLstSWUdHJKHmnWL4ALlVcfKOhyWsG4T3kzDNAbbqycd2w9QiI/T5Mc3U Y2N0KAwNQgMkDo6Ijhvm1lPPY2P9X/ixmJ5BYaj0Dfu3J+nYcTwq8L4IDwYAeko9HSeQ NqytolonQp0cSgljkLEFuhMf9v6XkeyrovySsW6fVUx5irgt6iX09+BSRh2ZPYFaQIJA v/ScOcbMFn5kC53NYwfxvSz0VbJaAidt6RKl9GP4xlxSiCv8YMY8bw3IXErqXZllaKfh Q41g==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:from:date:message-id:subject :to; bh=jiYYaVhnW+ikR+pG3FOZpAMhiJJLyRL2ya76hT7cZp8=; b=PNzk/PA5hhsEhWfo8oALOf+ymjvgrkbeKGaevE9pVWak7rIps5YZ9Q+UrBZgu1Rg38 irI2PsbhjYfjfcxRTMItNv9cqvHHr8jCiH6tucZk6tQdRmBsb+BVR+6X41t8Z+hrniXf kzM+DnYMC1HaAbW4/39oOpmLbpMsmlcl0Fa5vJzzHYzWxTs8CtXSys6Sq4FeNrS0WATL LiTnlACKAqoEkEFihzU4QU1bfsu/MKHtRA0swAnWRUtZvnHMB+haFbL9FVC5c/Cnvo15 TW5UBm+g0AZRa5nNITR/fOOkJiEICYQxq+Pydr/S0O7FulAn4BXIzJYgRJBt1VZGmavj 2YYg==
X-Gm-Message-State: AKaTC00cepNBJHPaVI4r83oNwm6wPE/r+L/9wLpAiaUjIoSjZcHYw4T+gjcoHxkSGh6/4e1sR2mZbVX4qRFIrg==
X-Received: by with SMTP id k69mr2785287wmg.137.1479827075283; Tue, 22 Nov 2016 07:04:35 -0800 (PST)
MIME-Version: 1.0
Sender: hallam@gmail.com
Received: by with HTTP; Tue, 22 Nov 2016 07:04:34 -0800 (PST)
From: Phillip Hallam-Baker <phill@hallambaker.com>
Date: Tue, 22 Nov 2016 10:04:34 -0500
X-Google-Sender-Auth: ZGpxTwoP5pscCwIAc5Wu0byI_Qw
Message-ID: <CAMm+LwgtJuLdL_RKJNSVNGODGj8D25nfj0jkhnBLFS=aaXG+rA@mail.gmail.com>
Subject: What is the right way to do Web Services discovery?
To: IETF Discussion Mailing List <ietf@ietf.org>
Content-Type: multipart/alternative; boundary="001a1147010626c41e0541e517e6"
Archived-At: <https://mailarchive.ietf.org/arch/msg/ietf/_bpZxdMNF0KXK4zhweWpiGnIIAQ>
X-BeenThere: ietf@ietf.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: IETF-Discussion <ietf.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/ietf>, <mailto:ietf-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ietf/>
List-Post: <mailto:ietf@ietf.org>
List-Help: <mailto:ietf-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ietf>, <mailto:ietf-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 22 Nov 2016 15:04:41 -0000

I am asking here as there seems to be a disagreement in HTTP land and DNS

Here are the constraints as I see them:

0) Foir any discovery mechanism to be viable, it must work in 100% of
cases. That includes IPv4, IPv6 and either with NAT.

1) Attempting to introduce new DNS records is a slow process. For practical
purposes, any discovery mechanism that requires more than SRV + TXT is not
going to be widely used.

2) Apps area seems to have settled on a combination of SRV+TXT as the basis
for discovery. But right now the way these are used is left to individual
protocol designers to decide. Which is another way of saying 'we don't have
a standard'.

3) The DNS query architecture as deployed works best if the server can
anticipate the further requests. So a system that uses only SRV+TXT allows
for a lot more optimization than one using a large number of records.

4) There are not enough TCP ports to support all the services one would
want. Further keeping ports open incurs costs. Pretty much the only
functionality from HTTP that Web Services make use of is the use of the URL
stem to effectively create more ports. A hundred different Web services can
all share port 80.

5) The SRV record does not specify the URL stem though. Which means that
either it has to be specified in some other DNS record (URI or TXT path) or
it has to follow a convention (i.e. .well-known).

6) Sometimes SRV records don't get through and so any robust service has to
have a strategy for dealing with that situation.

7) If we are going to get to a robust defense against traffic analysis, it
has to be possible to secure the initial TLS handshake, i.e. before SNI is
performed. This in turn means that it must be possible to pull information
out of that exchange and into the DNS. Right now we don't know what that
information is but this was not a use case considered by DANE.

8) We are probably going to want to transition Web Services to 'something
like QUIC' in the near future. Web Services really don't need a lot more
than a TCP stream. Most of HTTP just gets in the way. But the multiplexing
features in QUIC could be very useful.

Right now we have different ideas on how this should work in the HTTP space
and DNS space. And this appears to be fine with the two groups as they
don't need to talk to each other. But it really isn't possible to build
real systems unless you offend the purists in at least one camp. I think we
should do better and offend both.

So here is my proposal for discovery of a service with IANA protocol label

First the service description records. This is a TXT record setting policy
for all instances of the fred service and a set of SRV service

_fred._tcp.example.com TXT "minv=1.2 maxv=3"
_fred._tcp.example.com SRV 0 100 80 host1.example.com
_fred._tcp.example.com SRV 0 100 80 host2.example.com

There is also a set of round robin A records for systems behind legacy NAT.
You could do AAAA as well but these probably aren't needed as it is
unlikely that a router blocking SRV will pass AAAA

fred.example.com A
fred.example.com A

And finally, we have the host description entries

host1.example.com A
_fred._tcp.host1.example.com TXT "minv=1.2 maxv=2 tls=1.2 path=/fred12"
host2.example.com A
_fred._tcp.host2.example.com TXT "tls=1.3"

So here we have some host level service description tags which obviously
override the ones specified at the service level. With the proviso that a
client might well abort if the service level description suggests there is
no acceptable host. The path descriptor allows the use of the well known
service to be avoided on host1. It defaults on host2

In the normal run of things, a DNS server would recognize that a request
for _fred._tcp.example.com SRV was likely the start of a request chain and
send all the records describing the service in a single bundle. This should
usually fit in a single UDP response.

This approach gives us two levers allowing us to set policy for the
service. We can define policy for all service instances or granular per
host information.

The bit that I have not got nailed down is what the HTTP URL should be
after the service discovery is performed. My view is that they should be


Which works nicely with the existing code and but not for TLS operations.
We will either need certs for host1.example.com and host2.example.com or
have to override the TLS stack to accept certs for example.com.

The problem becomes even more apparent if the redirects are to
host1.cloudly.com and host2.cloudly.com where cloudly is a cloud service
provider. So the alternative is to do this:


The problem is that it does not work well when trying to use this strategy
with existing http clients built into scripting languages. Instead of just
writing a module that does the SRV lookup and spits out the URLs and
attributes, now we need to rewrite our client so it will hit the right DNS

Given that most libraries seem to have hooks to allow a client to make its
own TLS certificate path math choices, I am very strongly in favor of the
first approach. But I am willing to be persuaded otherwise.