Re: [radext] Fwd: RE: Mail reguarding draft-ietf-radext-dynamic-discovery

[Stefan Winter <[e-mail]>]

Hi,

>> This would be a whitelist of stuff we want, and everything is else is
>> declared bad. Would that address your concern?
> 
> One case not covered that I know of: NXDOMAIN is not returned when an owner
> has records of type other than the type you query.  What you get is a
> NOERROR response without any of the requested type of RRs, and
> usually an authority SOA record, and maybe some additional section RRs.
> Try a dig for NAPTRs for ieee.org for example.
> 
> I'd suggest defining NXDOMAIN/NSEC/NSEC3 and the above case as a
> "negative result" instead, and just say in step 4 to proceed to
> step 9 on a credible negative result, step 5 if credible NAPTR RRs
> were retrieved, and otherwise O = { empty set } and terminate.

Okay, I've modified the text to be more explicit:

3.2.  Configuration Variables

   The algorithm contains various variables for timeouts.  These
   variables are named here and reasonable default values are provided.
   Implementations wishing to deviate from these defaults should make
   they understand the implications of changes.

      DNS_TIMEOUT: maximum amount of time to wait for the complete set
      of all DNS queries to complete: Default = 3 seconds

      MIN_EFF_TTL: minimum DNS TTL of discovered targets: Default = 60
      seconds

      BACKOFF_TIME: if no conclusive DNS response was retrieved after
      DNS_TIMEOUT, do not attempt dynamic discovery before BACKOFF_TIME
      has elapsed.  Default = 600 seconds


3.3.  Definitions

   Positive DNS response: a response which contains the RR that was
   queried for.

   Negative DNS response: a response which does not contain the RR that
   was queried for, but contains an SOA record along with a TTL
   indicating cache duration for this negative result.

   DNS Error: Where the algorithm states "name resolution returns with
   an error", this shall mean that either the DNS request timed out, or
   a DNS response which is neither a positive nor a negative response
   (e.g. SERVFAIL).

   Effective TTL: The validity period for discovered RADIUS/TLS target
   hosts.  Calculated as: Effective TTL (set of DNS TTL values) = max {
   MIN_EFF_TTL, min { DNS TTL values } }

(the last definition helps streamlining the formulations about the
minimum TTLs later on; stay tuned for the final -07 draft.

> I think this point would benefit from the attention of a jaded^Wexperienced
> DNS developer, which I am not.  There could easily be other modes which I've
> not seen.

Anyone reading the radext ML is invited to comment; in any case, sloppy
formulations on the radext side would be caught in IETF last call or AD
reviews.

> Minor quibble -- it's presented in steps, but 8 is really happening during 7
> and functioning as a break condition.

Yes... formulations aren't totally final yet. I'm happy if the meaning
of things gets across (which I believe does).

>>    9.   O' = (set of {hostname; port; order/preference; min{all TTLs
>>         that led to this result} } for all lookup results).  Keep note
>>         of the remaining TTL of each of the discovered records (e.g. SRV
>>         and AAAA).
> 
>> I.e. you either get a final result for all the records, or the discovery
>> has failed.
> 
>> Would that address your concern?
> 
> It still needs work.  What's a "complete set of hostnames"?  This is the
> area where I'm probably going to try your patience :-).

I'm not sure how to formulate this cleanly... the idea is: in each DNS
query, there may be several RRs. All of these RRs again lead to more
queries which can have multiple RRs; effectively creating a tree of DNS
replies, where hostnames are penultimate to the leafs, and A and AAAA
records are the leafs.

A "final result" here is one where all branches have been explored and
have all yielded hostnames. (the ultimate A and AAAA resolution comes in
a later step, as it's common for both the NAPTR and the fallback SRV
resolution; and it is a host prefence which protocol to try in the end.

Actually, the following text hopefully captures that:

"  9.   If these name resolutions return an error for any of the
        subsequent lookups (e.g. an SRV target which does not exist), or
        for other reasons does not yield a complete set of hostnames
        (i.e. where all branches of NAPTR and possibly their follow-up
        SRV RRs have been evaluated) before DNS_TIMEOUT has elapsed, O =
        { empty set } and terminate."

(it has become step 9 meanwhile)

> The words "perform SRV lookup" could mean to just launch a SRV query,
> as intended, or could be taken as "follow the rules for using SRV records
> in the RFC that defines them."  I don't think not mentioning 2782 gets
> the draft off the hook here, because once SRV is mentioned as something that
> is a known quantity, the reasonable recourse for an un-annotated external
> reference is to consult relevant RFCs.
> 
> As an aside, when dealing with "s" flags, 3958 explicitly states "normal
> SRV processing is applied" (2.2.3).  Were it not for the explicit provision
> in 2.2.4. about declaring failure, I would take this to mean that the
> Usage Rules in 2782 are supposed to be followed.
> 
> It could perhaps be rephrased "look up RRs of type SRV."

I've defined my use of "SRV lookups" now in the new Definitions section.
It reads:

"  SRV lookup: for the purpose of this specification, SRV lookup
   procedures are defined as per [RFC2782], but excluding that RFCs "A"
   fallback as defined in its section "Usage Rules", final "else"
   clause."

>> The ports are merely default ports. If someone runs a RADIUS/TLS server
>> on 1812 instead of 2083, then that's fine and he can announce that fact
>> in DNS if he so likes.
> 
> I'm satisfied with that answer, and it was the one I expected.  Do
> note, the language in 6613 for port 1812 in particular is a bit stronger
> than merely defining a default port, but that could plausibly be a
> shortcoming
> in 6613's phrasing.

That may be as it likes; I don't think it warrants changes to the
dynamic discovery I-D.

>> Regarding 3.1.3, I'll need to think more about it, and need another rev
>> of the document before it manifests.
> 
> Likewise, it will be some time until I can thoroughly investigate this.

Actually, I now have text for that; stay tuned for -07.

> Some of these vectors will exist whether or not the implementation stops at
> three seconds.  Also the 3 second value is for the benefit of RADIUS
> protocol
> fallback needs and should probably not be conflated with a secure value for
> abating DoS attacks.  Implementations could use an arbitrary value here
> that is
> more than 3 seconds but still reasonably finite.  What administrators
> will want
> instinctively is for the resolution to continue for their usual client
> retry interval,
> and continue thereafter if refreshed by a retry.

I've now moved that integer to an explicit configuration variable with a
(sane?) default of 3 seconds.

> In some vectors three seconds is more than enough time to cause
> a lot of hurt, repeated bursts of three seconds moreso.  At the very
> least the DNS caches will be subject to loading under these scenarios.
> Note that RADIUS servers are likely to be exempt from many of the
> safeguards applied to client machines on the DNS server side, and
> they are likely to have nice fast 1G or 10G links direct to the DNS
> servers with a very low RTT.  A lot of queries can happen under those
> conditions in 3 seconds if the terminal records are aimed at NX records
> in a locally authoritative domain.
> 
> Most mature implementations will likely opt to place arbitrary limits
> on the following quantities, per realm, tunable either at compile or
> runtime so that resource use can be matched to reasonable expections
> within the particular federation(s) involved:
> 
> 1) total number of backtracks since the last TTL expiry or
> well-known-rule instigation
> 2) number of NAPTR records for which (summarized) TTL information is
> tracked.
> 
> The width/depth of NAPTR trees will be limited by 2).  The rate of
> queries for bogus domains will be limited by 1).
>
> In addition implementations will likely limit the number of ongoing or
> cached DDDS discoveries, in a FIFO fashion, failing and/or expunging
> the oldest queries when new ones arrive and the cap is exceeded.  As
> long as this cap is kept high enough that legitimate queries have
> time to complete before they get expunged to make room for a bogus
> query, RADIUS service itself is not in jeopardy.  If the rate of
> these queries is large enough to spin that FIFO too fast, then the
> server has bigger problems on its hands that need to be addressed at
> a general DoS mitigation level.
>
> I would suggest generalizing that language to simply advise
> implementations to set sensible limits to prevent over-stacking of
> pending DDDS discoveries.

That is some form of rate-limiting and has also been suggested by others
on the list. I'm sympathetic with adding text towards rate-limiting; I
am against being prescrptive towards implementations how they should do
that. I'm adding this sentence to the corresponding section in Security
Considerations:

"Note that even a timeout of three seconds may wreak havoc
   when being attacked large-scale; implementations are advised to
   consider rate-limiting either their amount of new executions of the
   dynamic discovery algorithm as a whole, or the amount of intermediate
   responses to track, or at least the number of pending DNS queries."

> If discussions have come up with an advised default value that is
> coincidentally
> also 3 seconds, distinguish this value from the limit on individual
> queries and
> present solid rationale as to why 10 to 30 seconds (typical for retries)
> is too
> long whereas 3 seconds is not.

The three seconds are explained earlier in the draft (needs to be below
5 due to RADIUS clients usually giving up after that amount of time).
Now that the value is configurable, implementations might tune that down
to smaller values if they see fit.

> Note that caching DDDS results for later use is required by the
> draft and also a vector for resource DoS, though not for sockets.
> 
>>> 3) The concerns in 2) may also be mitigated substantially if the draft
>>>     were to put normative limits on how small NAPTR and SRV TTL values
>>>     may be in compliant DDNS database entries.  Allowing small TTLs in
>>>     these records leaves open the possibility that "relied upon" RRs
>>>     will expire before negative results return from DNS servers, the
>>>     algorithm will backtrack, and if those negative results also have
>>>     a short TTL or are not well cached, this process could repeat
>>>     itself for as long as the discovery algorithm is allowed to run.
>>>     The draft would do well to permit compliant implementations to
>>>     apply a reasonable minimum TTL value to all NAPTR and SRV records
>>>     received, something on the order of 60 seconds, and to advise
>>>     database creators to keep NAPTR and SRV record TTLs even much
>>>     higher than that.  One source of advice on this matter is an
>>>     expired draft, draft-ietf-speermint-srv-naptr-use.
> 
>> Answering your point 3) first.
> 
>> I believe there is a misconception on the TTLs here. Of course the
>> maintainers of a DNS zone should define their entries with reasonably
>> high TTLs to avoid thrashing. That's a normal consideration for any DNS
>> label and RR though, I don't think we need to engage in the business of
>> handholding or explain "DNS domain administration 101".
> 
>> Also keep in mind that changes in the network infrastructure sometimes
>> REQUIRE low TTLs, e.g. when trying to change a hostname to A mapping
>> in-place.
> 
>> The reason why I wrote misconception though is that these are not the
>> TTLs the algorithm gets (I tried to construct a pun involving "these are
>> not the TTLs you are looking for" but didn't manage :-) ). The name
>> resolution library on the server which does dynamic discovery is very
>> unlikely to get fresh entries from the respective authoritative DNS
>> server with the full TTL of that authoritative server. It will much more
>> likely get a cached reply from its resolver. And resolvers keep track of
>> how long *they* should cache answers received from upstream. I.e. they
>> will take the initial TTL from the authoritative server, and will
>> decrement it on their cached entry. When queried, they will reply with
>> their own copy's (lower) TTL.
> 
>> No matter how large the TTL on the authoritative server is, it can at
>> any time happen that a resolver will reply in the last second of its own
>> cache lifetime and will rightfully state that the entry is TTLed with a
>> 1 second duration.
> 
>> So, no amount of advice towards DNS administrators will make your issue
>> go away.
> 
> This is understood.  Do note that if a DNS server provides a lingering
> record with an extremely low TTL that expires, and the authoritative TTL
> is set reasonably, the re-query is only going to happen once in the
> short term, as opposed to when the authoritative TTL is e.g. zero.
> 
> (Think of what happens when a sports team tour bus or two of 50 or so
> eduroamers from the same institution arrives at your school and their
> NAPTR TTLs are 1, then they proceed to hop repeatedly around
> several autonomous access points without fast-reauth enabled.)
> 
> Advice to DNS administrators does not technically solve the issue, though,
> you are correct.  It just serves (if followed) to reduce the occurence of
> instances where the implementation might need to disobey the TTL -- TTLs
> would more often expire in the interstices between RADIUS requests and
> be refreshed with values that would endure the full DDDS discovery.
> 
> It also sets reasonable expectations on the part of DNS administrators and
> federation policy makers as to how DDDS implementations are likely to
> behave.
> If the guidance suggests a minimum TTL, they'll be duly warned that
> implementation behavior might not be reliable if they elect to do otherwise.

They can read a book about DNS zone administration; TTL thrashing is a
well-understood problem.

> Your point about the scope of the specification is well taken, though.
> 
>>> 2) It is not clear as to how to proceed if TTLs, either positive or
>>>     negative, expire during the execution of the algorithm due to DNS
>>>     delays; Section 2.3.4 only specifies that they apply when considering
>>>     the re-use of results for additional user sessions.  RFC3403 Section 3
>>>     does specify that records that are "relied upon" will cause a restart
>>>     of the algorithm if their TTL has expired during a "backtrack", but
>>>     says nothing of what to do during the descent.
> 
>> Picking up your point from 3), it sounds like an interesting idea to
>> specify that any TTL < 60 seconds (or an arguable other integer) should
>> be considered as TTL = 60 seconds; thereby overriding the DNS caching
>> rules. Then your concerns would probably be mitigated.
> 
> Entirely mitigated, yes.  Another option is to say that results are sticky
> during descent.  From an implementation standpoint the former is easier
> since we have to keep the TTL as state anyway.

I've added the notion of "Effective TTL, which is either the DNS TTL or
the configuration parameter MIN_EFF_TTL, whichever is higher.

>> I like that, and would put that in the draft. But I believe it should be
>> discussed on the mailing list first; I could imagine DNS people to be
>> disenchanted by an override of their data ("Why do you think you know
>> better than the DNS admin himself?!?!").
> 
>>>   Also, it is not
>>>     explicitly stated in RFC3403 Section 3 that a negative result which
>>>     has caused a previous skip of a branch of the NAPTR tree is "relied
>>>     upon", so that may be worth explicit mention.
> 
>> If also all negative results would get at least a 60s survival time,
>> then this becomes a non-issue, right?
> 
> Partially: it solves it for descent, but not for backtrack which can
> happen much later.
> This is more a technicality than an issue -- the right thing to do
> (maintain a summarized
> "negative" TTL for each skipped NAPTR RR which is the minimum of all
> positive/negative
> TTLs involved in the decision to backtrack past the RR in question, and then
> check it when re-using the data) was comprehensible to me, but it isn't
> stated outright and might be unfathomable to some implementers.  I probably
> should not have lumped this point into 2) as it does not just apply to
> initial descent.  Also it is really RFC3403s problem -- whether patching it
> up in the application definition is appropriate I leave to the standards
> gurus :-)

Well, there is one open question to it (but it is so minor that the
answer to it may not matter very much).

Imagine a realm without dynamic discovery targets, neither NAPTR nor SRV
labels.

The NAPTR query will yield a negative TTL X; the subsequent SRV fallback
query will yield its own negative TTL Y.

How long should the caller of the algorithm be told to back off?

I think min { X,Y } is the answer; if any one of the two has elapsed,
either a new NAPTR might show up -> worth re-executing the algorithm; or
it may not, but a new SRV might come up -> again worth re-executing.

I've changed the algorithm to take this into account.

Greetings,

Stefan Winter

> 
> --
> Brian S. Julin
> 
> 
> 
> 
> _______________________________________________
> radext mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/radext
> 


-- 
Stefan WINTER
Ingenieur de Recherche
Fondation RESTENA - Réseau Téléinformatique de l'Education Nationale et
de la Recherche
6, rue Richard Coudenhove-Kalergi
L-1359 Luxembourg

Tel: [phone]
Fax: [phone]