Re: [TLS] Multi-CDN and ESNI

[Erik Nygren <[e-mail]>]

Another important scenario that is closely related to multi-cdn is how to
safely enable and disable ESNI, as well as how to handle cases where not
all CDNs in a multi-CDN setup have ESNI turned on.  As some examples:

* A site is using a CDN that has ESNI enabled.  As part of switching off of
that CDN to their own hosting provider or another CDN (which either has no
ESNI or has a different ESNI key) there is the same sort of inconsistency
in the records, or a missing record.

* A site is on a multi-CDN setup.  There is no way to get all CDNs to turn
on ESNI at the same time (or likely even within a reasonable time window,
and there may be cases where only some support it).

For example, with #3 one way to handle it would be to treat an NXDOMAIN for
the _esni ESNI record as a "disable sending ESNI in this case".

Some general commentary on some of the proposed options:

On #1) Experience with coordinating keys across providers suggests this
likely won't work.  (Plus will open up lots of security issues.)   The ESNI
DNS record, ESNI key, and ESNI server-side config likely need to be managed
by the same entity for any given case in order for this to be operationally
maintainable.

On #2) Given the number of addresses involved in some of these cases this
may not work or scale well.  In some cases the number of address candidates
may be more than possible in an rrset.  For CDNs that change their DNS
results frequently and have low TTLs, this is also likely to have a very
high rate of mismatches.

On #3) As Nick mentions this won't help with the address collapsing case.
It also needs a way to handle the offboarding situation described above.

I think there's also an option #4 that has a slightly higher integration
overhead for
sites but solves other problems as well.  If we take a step back, the issue
here is that
DNS is a database but any two given responses lack some sort of primary key
tying them
together.  One approach would be to introduce an intermediate record/object
that explicitly
does tie them together.  Two examples of this are the ALTSVC-in-DNS
proposal as well as
something like a "Service Binding" record (which I wrote up when we were
talking
about this back in 2014 following a brainstorm, in which case the"tlshp"
parameter
in that draft is effectively the ESNI key):
      https://www.ietf.org/archive/id/draft-nygren-service-bindings-00.txt

More recently with the ALTSVC DNS record, we'd add an ALTSVC attribute
that contains either the ESNI key or the name of the record containing the
ESNI key:
     https://tools.ietf.org/html/draft-schwartz-httpbis-dns-alt-svc-02

One big advantage of using an ALTSVC record is that it also addresses the
"http-srv" problem.
(See a discussion thread titled "Alternative to SRV?" on the http-srv list
from back in August.)

Another big advantage of using an ALTSVC record approach here is that it
means
that ESNI could be generally configured by Alt-Svc as well.  For example,
this means that the ESNI key could also just be included when doing an
Alt-Svc to QUIC.

What this ends up looking like is something like is an rrset along the
lines of:

  _443._https.example.com. IN TYPE???  {priority1}
{transportprotocol1} {servername1} {port1} {extension_fieldset_1}

  _443._https.example.com. IN TYPE???  {priority2}
{transportprotocol2} {servername2} {port2} {extension_fieldset_2}

  _443._https.example.com. IN TYPE???  {priority3}
{transportprotocol3} {servername3} {port3} {extension_fieldset_3}


For example:

  _443._https.example.com. IN ALTSVC  10 quic2
example.com.examplecdn.net 443 ( esni=key124._esni.examplecdn.net )

  _443._https.example.com. IN ALTSVC  20 h2 example.com.examplecdn.net
443 ( esni=key124._esni.examplecdn.net; ipv6-only=true )

  _443._https.example.com. IN ALTSVC  40 http/1.1
legacy.example.com.examplecdn.net 443 (
esni=key124._esni.examplecdn.net )


I suspect that some CDNs would then have customers CNAME over "_443._
https.example.com."
to some name on the CDN.  For example in the example above it might be that
it actually
CNAMEs to _443._https.example.com.examplecdn.net which then means that the
A and AAAA records for "example.com.examplecdn.net" can be returned as
additionals from the same examplecdn.net authority
along with the ALTSVC record.

So in the multi-CDN case, the multi-CDN switcher is switching on the CNAME
to the ALTSVC record,
allowing each CDN or hosting provider target to provide their own ALTSVC
record.
Tying this back up to the above the ALTSVC record then becomes this
intermediate object
that explicitly ties the server (A/AAAA records) and ESNI key together.

One downside is that ALTSVC is primarily defined for HTTPS today,
but nothing prevents it from being more broadly used as an extensible
SRV alternative.

(Side-note: no matter what we'll also want to think and document
how ESNI and cross-hostname Alt-Svc interact.)

        Erik



On Tue, Oct 23, 2018 at 5:28 PM Patrick McManus <[e-mail]>
wrote:

> Definitely agree this is something that needs to be addressed..
>
> As Mike notes, the fundamental issue is that there are 2 pieces of
> information that are statefully related (the key and address) but obtained
> statelessly from each other and can therefore come from un-coordinated
> sources. 2 CDNs are commonly un-coordinated sources.. but I've seen this
> with other kinds of cloud providers too. All of which should be target
> audiences of ESNI (because they terminate a variety of hostnames on a
> smaller number of addresses).
>
> fwiw there is a muddled github issue open on this in the old
> individual-draft repo:
> https://github.com/ekr/draft-rescorla-tls-esni/issues/35 .. which I'll
> re-open when the tls-wg repo starts being used for this draft.
>
> I see three solution spaces.
>
> 1] As Rich suggests, you can coordinate the uncoordinated sources to have
> one keying record to rule them all. I suspect that's actually not good for
> anonymity unless it somehow had a normalized global set .. and given the
> one way nature of a CNAME pointer, I feel like this would be really fragile.
>
> 2] you could scope the keys to only be valid for certain addresses by
> putting the valid addresses in the key record. This wouldn't help you do
> ESNI when they didn't match, but at least you could avoid hard failure. (or
> you could ignore the addressing record but that might be a bridge too far?)
>
> 3] My preferred approach, you could scope the keys to the same
> intermediate name. So if www.example.com -> cname www.cdn-a.com -> ESNI
> record www.cdn-a.com we could make a rule that the ESNI record would only
> apply to address records with a final intermediate of www.cdn-a.com. (I'm
> presuming we'll shift from TXT to ESNI RRTYPE without a prefix, but I don't
> think it matters for this..).. when there is a mismatch you could use the
> addressing intermediate as a place to try a new ESNI query from.. That's a
> perf penalty, but its only paid in the hopefully rare case where these
> things are unsyncd.
>
> #3 is admittedly a bit non traditional, but I spent a bit of time looking
> at the features of DNS APIs that already have the necessary surface to make
> a query for a new RRTYPE... and the cname intermediates are indeed
> generally available.. its true of getdns, res_query, the IOS and MacOS
> interfaces, DnsQuery* on windows, and of course the custom dns stacks in
> cronet/android and firefox/doh could do it.
>
> I hope to work this into a PR.. my first attempt wasn't very readable, but
> I'll try again tomorrow.
>
> -P
>
>
>
>
>
>
> On Tue, Oct 23, 2018 at 1:59 PM Salz, Rich <[e-mail]> wrote:
>
>> I think perhaps we need to take a step back and explain something that
>> might not be well-known outside the community of CDN’s and their
>> customers.  It is not uncommon for (admittedly larger) origins to use
>> multiple CDN’s, and to switch among them. This can be done on a per-request
>> basis, because of things like contractual arrangements that make one
>> preferable, or it can be done globally but switched in a matter of seconds
>> because of a short TTL on the www.example.com entry.
>>
>>
>>
>> The issues that Mike discusses impact on this, somewhat negatively.
>>
>>
>>
>> A quick hack thought is to allow multiple entries in the TXT record,
>> forcing a wee bit more work on the CDN.
>>
>>
>> _______________________________________________
>> TLS mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/tls
>>
> _______________________________________________
> TLS mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/tls
>