Re: [TLS] Multi-CDN and ESNI

[Patrick McManus <[e-mail]>]

Hey Nick,

On Tue, Oct 23, 2018 at 8:45 PM Nick Sullivan <[e-mail]> wrote:

> This line of commentary describes one instance of a more general situation
> that is unrelated to the multi-provider case: what happens when you connect
> to a server that doesn't know the ESNI key you're using? This can even
> happen on a single provider due to DNS caching issues, for example.
>
> The two cases are related and there are two features missing from the
> current ESNI draft that might be useful to mitigate these scenarios:
> 1) Allow multiple simultaneous ESNI keys. The DNS provider could present
> ESNI keys from multiple HTTPS providers (perhaps as different DNS records)
> and the client could choose to send multiple entries in its ESNI extension,
> each encrypted to a different key. This would likely solve the multi-CDN
> scenario at the cost of some clienthello bloat.
>

I would like to see this feature, but I don't think it addresses this
problem..

consider load balancer L, CDN A and CDN B. In a world in which L is aware
of A and B (duh) but not ESNI, and A wants to deploy ESNI but B has not yet
done so. You have the likely situation where at some point you've got an
address record for B but a key for only A and you run into hard failure.

even if A and B both want to do ESNI they need to be aware of each other
and coordinate key rotations somehow to synthesize records - or else you
end up spf like includes (please no). Requiring cross provider coordination
seems like assuming knowledge the providers don't reliably have and
probably could't reliably maintain. You could boot the problem to cusomter
configuration but.. 1] esni is so much better as automatic infrastructure
so CDNs can just turn it on and 2] origin names may not wish to disclose
all of their providers to each other.


> 2) Allow in-band distribution of ESNI keys if the server does not know the
> ESNI key sent by the client. The server could prove ownership over a
> specific long-term "fallback key" advertised in the ESNI record (or maybe a
> fallback domain) when sent a TLS connection for which no ESNI extensions
> can be decrypted, then in-band send back a new ESNI public key that the
> client could use to send the SNI in a follow-up message. It's also possible
> that handshake encryption alone is enough. This might require some TLS
> message gymnastics, but doesn't seem impossible. I've heard more well-baked
> versions of this idea batted around by others. (note that this incurs at
> least one additional round-trip, but considering that a server not having
> the active ESNI key should be rare, it is likely ok)
>
>
My concern would be a legacy TLS stack. The problematic case is when the
lookup for ESNI yields a key, but the host at the end of the A record has
never heard of it (or perhaps even heard of TLS 1.3). These are different
orgs, so its totally possible and inevitable in the sense that there are no
flag days. How does that work with this scheme?

This is tricky because the failure mode is so painful. provider A can
essentially break provider B by deploying ESNI - externalizing some of its
own risk. A and B don't have a super reliable way of knowing about each
other to even make sane risk judgments.


> To comment specifically on your proposals, Patrick, I don't see #3 as a
> workable solution since many authoritative DNS providers hide the fact that
> a CNAME is used when returning A/AAAA records. This obfuscates the
> indirection that would be necessary to do what you propose, and it does it
> for a good reason (saving round trips on DNS resolution). Furthermore,
> intermediate values of CNAME chains seems like a brittle mechanism to rely
> on for policy enforcement.
>
>
Roughly, this is inspired by what krb does with canonical names in
practice. And it makes logical sense - the key applies to the canonical
name, not the alias. That said - this is a pain point I agree.

I'm glad you brought up flattening - I know of a handful of orgs that do
that. But do any of them do it across administrative domains or just as
optimizations to their own 'zone files'?  I understood it to be the latter,
and I don't think that would really be a problem here as long as it was
consistent across rrtypes.

another thought, Kazuho had mentioned reverse lookups at some point.
Definitely too painful for the normal case, but perhaps plausible as a
fallback? Doing a reverse lookup of the address record to obtain the
canonical key for the address (and absence of a reverse-lookup key means
that host doesn't really do ESNI so proceed the old way). it would be
backwards compatible but I worry both that its too fragile itself (IP
addressing can be pretty disjoint from http operations) and perhaps too
much of a time penalty to bear given the 'externalization' problem. But
let's throw it out there.




> Nick
>
> On Tue, Oct 23, 2018 at 2:28 PM Patrick McManus <[e-mail]>
> wrote:
>
>> Definitely agree this is something that needs to be addressed..
>>
>> As Mike notes, the fundamental issue is that there are 2 pieces of
>> information that are statefully related (the key and address) but obtained
>> statelessly from each other and can therefore come from un-coordinated
>> sources. 2 CDNs are commonly un-coordinated sources.. but I've seen this
>> with other kinds of cloud providers too. All of which should be target
>> audiences of ESNI (because they terminate a variety of hostnames on a
>> smaller number of addresses).
>>
>> fwiw there is a muddled github issue open on this in the old
>> individual-draft repo:
>> https://github.com/ekr/draft-rescorla-tls-esni/issues/35 .. which I'll
>> re-open when the tls-wg repo starts being used for this draft.
>>
>> I see three solution spaces.
>>
>> 1] As Rich suggests, you can coordinate the uncoordinated sources to have
>> one keying record to rule them all. I suspect that's actually not good for
>> anonymity unless it somehow had a normalized global set .. and given the
>> one way nature of a CNAME pointer, I feel like this would be really fragile.
>>
>> 2] you could scope the keys to only be valid for certain addresses by
>> putting the valid addresses in the key record. This wouldn't help you do
>> ESNI when they didn't match, but at least you could avoid hard failure. (or
>> you could ignore the addressing record but that might be a bridge too far?)
>>
>> 3] My preferred approach, you could scope the keys to the same
>> intermediate name. So if www.example.com -> cname www.cdn-a.com -> ESNI
>> record www.cdn-a.com we could make a rule that the ESNI record would
>> only apply to address records with a final intermediate of www.cdn-a.com.
>> (I'm presuming we'll shift from TXT to ESNI RRTYPE without a prefix, but I
>> don't think it matters for this..).. when there is a mismatch you could use
>> the addressing intermediate as a place to try a new ESNI query from..
>> That's a perf penalty, but its only paid in the hopefully rare case where
>> these things are unsyncd.
>>
>> #3 is admittedly a bit non traditional, but I spent a bit of time looking
>> at the features of DNS APIs that already have the necessary surface to make
>> a query for a new RRTYPE... and the cname intermediates are indeed
>> generally available.. its true of getdns, res_query, the IOS and MacOS
>> interfaces, DnsQuery* on windows, and of course the custom dns stacks in
>> cronet/android and firefox/doh could do it.
>>
>> I hope to work this into a PR.. my first attempt wasn't very readable,
>> but I'll try again tomorrow.
>>
>> -P
>>
>>
>>
>>
>>
>>
>> On Tue, Oct 23, 2018 at 1:59 PM Salz, Rich <[e-mail]> wrote:
>>
>>> I think perhaps we need to take a step back and explain something that
>>> might not be well-known outside the community of CDN’s and their
>>> customers.  It is not uncommon for (admittedly larger) origins to use
>>> multiple CDN’s, and to switch among them. This can be done on a per-request
>>> basis, because of things like contractual arrangements that make one
>>> preferable, or it can be done globally but switched in a matter of seconds
>>> because of a short TTL on the www.example.com entry.
>>>
>>>
>>>
>>> The issues that Mike discusses impact on this, somewhat negatively.
>>>
>>>
>>>
>>> A quick hack thought is to allow multiple entries in the TXT record,
>>> forcing a wee bit more work on the CDN.
>>>
>>>
>>> _______________________________________________
>>> TLS mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/tls
>>>
>> _______________________________________________
>> TLS mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/tls
>>
>