Re: [urn] -14

[Keith Moore <[e-mail]>]

On 11/02/2015 10:37 AM, John C Klensin wrote:
>
> --On Monday, November 02, 2015 08:11 -0500 Keith Moore
> <[e-mail]> wrote:
>
>> ...
>>> I mostly agree with that too, but I think we need to be
>>> careful and reasonable.   Perfection is impossible and not
>>> every detail is important (hugely or not) and I would hope
>>> that any "this needs to be fully specified and/or fixed
>>> before the document(s) can go into WGLC" statements would be
>>> accompanied by an explanation for that particular case as to
>>> why.  In particular, if there are things we are going to
>>> leave to the discretion and needs of individual namespaces (I
>>> think, given the nature of URNs, that is inevitable and
>>> consistent with 2141 and earlier documents), then it is
>>> important to be clear about the boundary but anything we say
>>> beyond that has to be in the form of non-normative examples.
>> It's that kind of language that makes me worry.  I am very
>> skeptical that leaving things to "the discretion and needs of
>> individual namespaces" is at all appropriate.   It is
>> essential that clients be able to process URNs without having
>> to know details about specific namespaces.   Saying we can
>> leave things to namespace discretion seems basically like a
>> failure to do our job in creating a standard.
> This is precisely the kind of disagreement (whether apparent or
> real) that I think needs to be exposed, discussed, resolved, and
> then taken sufficiently off the table that attempts to reopen it
> are strongly resisted by the WG.   If we are to make progress,
> that is true independent of which of us are correct.

Naming is inherently a difficult topic to discuss because (as computer 
scientists like to say) all problems can be solved by adding another 
layer of indirection.  It follows that any time someone proposes a new 
kind of name, particularly something as vaguely defined as "uniform 
resource name", anyone can imagine his particular problem being 
addressed by that naming and indirection scheme.

What seems to have happened for the entire history of URN discussions is 
that different participants each imagined URNs addressing the particular 
problem that he or she wanted to solve. As long as the URN definition 
focused entirely on syntax and "persistence" (vaguely defined), the 
conflicts between those goals were not exposed.   But as soon as we talk 
about resolution, the conflicts rear their ugly heads.   Some would like 
to finesse those conflicts by simply declaring resolution a 
namespace-dependent behavior.   I don't think that accomplishes anything 
because it doesn't facilitate interoperability.   If we can't define URN 
standards with sufficient precision to permit interoperability, what's 
the point of this effort?   I believe that we can do so, otherwise I 
wouldn't be investing time in this.

I should define what I mean by interoperability in this space.   A few 
things that immediately come to mind:

a) interoperability across namespaces - a client that supports 
operations on URNs should be able to support those operations for URNs 
of any namespace, as long as resolution services for the namespace 
adhere to one of a hopefully-small number of standards, and those 
resolution services are advertised and accessible to the client.

b) interoperability across clients - if two standards-conforming clients 
both support URNs, they should behave similarly when given the same URN 
and the same request for use of that URN (whether specified in the 
r-component of that URN, by context, or by some other means).

c) interoperability across resolution services - it should be possible 
to design a resolution service protocol that serves the needs of most if 
not all NIDs, and also possible to implement a server for that protocol 
that serves the needs of most if not all NIDs.

This, to me, implies that namespace-specific behavior should be kept to 
a minimum, or at least, that it should be possible for a client to 
implement common operations without relying on either specific knowledge 
of namespaces or nonstandard features of resolution services.   If there 
are additional operations that are possible by exploiting such 
namespace-specific knowledge or nonstandard features, that's not a 
catastrophe especially if it there is a path that facilitates 
standardization of the necessary features.

> Now, let me be sure that I understand what you are suggesting.
> If a "client" has to be able to "process URNs without having to
> know details about specific namespaces", that means to me that
> one cannot have per-namespace "resolvers" or resolution
> mechanisms, i.e., that, if any URN is resolvable, all URNs must
> be, and that there must be a generic resolution mechanism that
> works for all URNs and that, moreover, that there be no
> namespace-specific resolution machinery to be applied for any
> namespace after that generic resolution mechanism does its job.

I don't believe that.   In particular, I expect to have a DDDS-like 
database, that is distributed by various means (not necessarily or 
specifically DNS), that advertises (namespace-id, protocol, service-URL) 
tuples, where protocol is an IANA-registered URN resolution protocol 
identifier, and service-URL specifies how to contact the resolution 
service (the details can be protocol-specific).    If DDDS is 
insufficient for this, replace it.   But even if DDDS has some specific 
flaws, the core idea of distributing this database over the network is 
sound.

(And maybe we need those tuples to be slightly more elaborate, or to be 
a different data structure, in order to facilitate earlier matching of 
resolution services with the assigned portion of URNs and the requested 
services.  So this is more of a concept than a proposal.  I'm trying to 
avoid having to draft protocol specifications for both the database and 
the resolution services just to answer this email.)

> That, in turn, suggests to me that, from the following
> category-descriptions, one can have valid URNs in only one of
> them, not all three (or others)

I don't believe this either.   A resolution service advertised by the 
above means should be able to return both URLs of resource locations 
(when applicable) and various kinds of metadata about the resource (when 
applicable).

> My impression was that the WG agreed to the validity of all of
> those three casss as URNs long ago, at least for the second and
> third probably about the time we decided that DDDS was _a_ way
> to resolve _some_ URNs but not a necessary element of a generic
> URN system.

Well, of course _some_ URNs aren't resolvable, or even intended to be 
resolvable.   So any time you ask a resolution service about a 
particular URN, one of the potential answers can always be "I don't know 
anything about that URN".   The existence of a resolution service for a 
namespace doesn't assure that all URNs from that namespace are resolvable.

> However, one doesn't even need to look to 2141bis and history
> since 2141 (May 1997) or 2483 (January 1999) to have that
> per-namespace problem.
>
> 	urn:ISSN:1046-8188 (from RFC 3044 Section 3.1) treats the NSS
> at the identifier within the ISSN namespace but
>
>     urn:example:sha-1:29ead03e784b2f636a23ffff95ed12b56e2f2637
> (RFC 6963, Section 5) involves a "sub-namespace" that much be
> interpreted differently at some point rather than viewing the
> NSS as atomic.   Presumably urn:ISSN:foobar:9999-9999 would be
> considered a syntax error.  At least something
> namespace-specific would want to reject it rather than passing
> it on to some resolution machinery.

URNs namespaces have always been able to have structure within the 
NSS.   Also, each namespace has always been able to define its own 
particular syntax for its NSSs, and also particular equivalence rules 
for its NSSs.    I'm not proposing to change those decisions.

But it's not strictly necessary to advertise each namespace's structure, 
syntax, and equivalence rules to every client.  If a client queries a 
resolution service about a URN that happens to be invalid, the 
resolution service can say "I don't know anything about that URN".  Or 
it can say "that URN is malformed" if it knows enough to say that.   Or 
it can say "I can't answer your query, but I can refer your query to 
resolution service X which should have specific information about that 
URN". Similarly, a client could ask a resolution service whether two 
URNs from the same namespace were equivalent according to that 
namespace's rules.

As an optimization, some information about the structure of an NSS or 
how its queries are delegated, or the equivalence rules for a namespace 
could be advertised by some means.   This might reduce load on the 
resolution servers.   In the 1990s, we tried using regular expression 
rewrite rules to advertise those sorts of things; today we might try a 
different approach.  Worst case, the namespace database could advertise 
a separate javascript library for each namespace that could be called to 
pick apart delegation structure and/or determine equivalence.   It's not 
my favorite approach (and might even have more pitfalls than NAPTR 
records) but I'm open to considering different ways of doing it.

> Because of that, I think 2141 itself works iff one uses generic
> URN processing to identify the NID and then passes the NSS off
> to an NID-family-specific processor (I say "family" because I
> don't believe we will have, or should have, separate processors
> for each NID even though I expect the total number of processors
> will be much larger than one).

I hope the total number of resolution protocols will be small, though I 
never expect it to be one.   There can of course be a great many 
resolution services, both for redundancy and to serve multiple NIDs or 
subspaces within an NID.

>
> Now, if you agree with the above, I've very confused about what
> you are talking about or where the boundaries lie.  In
> particular...
>
>> I do accept that different resolution services will probably
>> recognize somewhat different sets of r-parameter facets and
>> offer somewhat different kinds of metadata, if for no other
>> reason than that these need to be able to evolve over time.
>> But I don't think this is a namespace-specific attribute, but
>> rather an attribute of the resolution service.
> Again, that completely confuses me, at least in part because I
> think that a generic URN processor can be expected to do only
> three things:
>
> 	(i) Parse the namestring, identifying the elements of
> 	<namestring> and <assigned-name>.[1]
> 	
> 	(ii) Try to find an appropriate resolution service based
> 	on the NID.  If it cannot find one, do something
> 	reasonable [2].
> 	
> 	(iii) If it can, identify a resolution service that it
> 	understands [3], pass the NSS, and any of q-component,
> 	r-component, and maybe f-component that are present,
>     to the resolution service [4].

I disagree that these are the only things that a generic URN client can 
be expected to do.   I agree that (i) and (ii) are reasonable 
expectations.  Beyond those things, I think a generic URN client should 
be able to

(iv) request URLs that represent locations of the resource named by the 
URN (which will be returned by the resolution service only if the 
request is applicable to that kind of resource, the resolution service 
has the information, etc.)
(v) obtain the content of the resource named by the URN (again, when 
applicable for the resource named by the URN, when a resolution service 
has that information).
(vi) learn about the kinds of metadata that are available for that URN
(vii) obtain any specific kind of metadata that is available for the URN

This implies that the client needs to be able to parse the r-component 
to determine the type of service being requested, as well as any 
constraints that the r-component imposes on resource content.  It's also 
possible to have a conflict between a service requested in an 
r-component and the service that's implied by the context in which the 
URN appears, so clients may need to detect some of those conflicts.

I also disagree significantly about (iii).    Neither the q-component 
nor the f-component should be passed to a resolution service UNLESS, 
perhaps, the resolution service is being requested to return the 
resource content - and even then, passing the f-component is dubious.   
You really don't want the resolution service trying to interpret 
q-component or f-component.   It strikes me as the worst kind of 
layering violation, because it blurs an otherwise clean separation 
between the responsibilities of a URN namespace and the responsibilities 
of the resource owner/maintainer.   (I'm also skeptical that resolution 
services should ever be entrusted to return resource content, though I 
recognize that there are probably corner cases for which this makes sense.)

> Note that the above implies that the semantics and within-string
> syntax of r-component and the handling of q-component are mostly
> the problem of the resolution service.
r-component - mostly but (unfortunately) not entirely.
q-component - it's not for the resolution service at all, but 
exclusively for use by the named resource.

> Because we expect the
> resolution service to digest r-components, we don't need to be
> as specific about its within-string syntax as we do about
> q-component (or f-component), which, at least as I understand
> our definitions so far, we expect to be passed more or less
> intact from the resolution service.

I believe we do need to be somewhat specific about r-components because 
clients will need to digest them to some degree.
q-components and f-components should never be passed to the resolution 
service.

>
>> Can you cite examples of things that need to be left to
>> namespace discretion?
> Whether the namespace is resolved at all.

I think I see your point, though I believe private "resolution" services 
will be very important.   (Call them something other than resolution 
services if you want, but they should support the same protocols and 
operations.)   So the namespace can decide whether or not it wants to 
advertise "authoritative" resolution services, but it can't forbid 
private resolution.

> Whether sub-namespaces are allowed.

Sure, each namespace determines the structure of its NSS (including 
whether there is any).     Though I find myself wondering whether you're 
talking about NSS structure or delegation structure, which don't have to 
be along the same lines.

> Whether q-component and/or f-components are consider valid.

Disagree.   Whether q-component is valid depends on the type of resource 
being named and/or the protocol being used to access it. A namespace can 
elect to not assign names to any resources for which a q-component could 
possibly be valid (e.g. it could only assign names to paper books or 
physical sound recording media), or it could elect to not return any 
URLs which support query syntax.   But the meaning of the q-component is 
namespace independent, and it's important that this be made clear.

Similarly, the meaning of the f-component is entirely dependent on the 
media type of the resource and the content itself, and is not within the 
purview of the namespace to define.   Even if a namespace refuses to 
assign any of its names to any resources that contain fragment 
identifiers, there are fragments that are meaningful even in the absence 
of fragment identifiers in the resource content. For better or worse, 
this is the web architecture, and we aren't going to change that.

> Whether r-components are accepted and what syntax and semantics they have.

Disagree.  R-components need to be standardized, at least in broad 
terms, because clients need to be able to interpret them to some 
degree.  Furthermore, it appears that clients may need to be able to add 
r-components to URNs in order to request the services that they need 
(e.g. to get resource content or URLs or metadata) in the absence of an 
explicit request in the r-component.   And even though there might 
inevitably be a need for some namespace-specific requests that can be 
included in an r-component, resolution services will need to support a 
"common core" of r-component requests and semantics.


> (i) Parse the namestring, identifying the elements of
> 	<namestring> and <assigned-name>.[1]
> [1] Given the way URIs are defined, this is actually a rather
> difficult problem because, whether there are generic URI parsers
> or not, there is really no such thing as a generic URI (or even
> URN) resolver that can encounter and generically report error
> states.  If I put "http:[[]]" into an href, any standardized
> error-handling is part of the HTML spec (very much scheme and
> content-specific) not part of either 3986 or, IIR, the http
> spec.   I hope we don't have a URK in our future named "fail"
> whose hier-part is unspecified but might be an error message or
> code, which it would be one way to deal with putative URNs that
> cannot be parsed.   Especially if we need to be as specific as I
> gather you want (and as my instincts tell me will probably be
> necessary), I don't see how to avoid saying something about this
> in 2141bis but don't, at the moment, know what to say.

I'm not sure what you're saying here.   A URI parser might not be able 
to detect every possible kind of syntax error in a URI, but it can 
certainly detect some kinds of errors.  That parser can do no more (and 
should do no less) than to pass the error indication to its caller.   
What the caller does with that error is up to either the protocol 
specification being implemented by that caller, or to the implementation 
itself.   If some kinds of syntax errors effectively get interpreted by 
the caller as "resource not found", that's perhaps unfortunate, but it 
shouldn't preclude a parser reporting more precise errors just in case 
some caller will handle them intelligently.

Anyway, having written generic URI parsers that seemed to be sufficient 
to the task, I don't really see the problem here.

> 	(ii) Try to find an appropriate resolution service based
> 	on the NID.  If it cannot find one, do something
> 	reasonable [2].
>
> [2] This is obviously a mess.   It is the mess that NAPTR and
> DDDS were intended to solve by creating a DNS-based indirection
> mechanism for binding resolution mechanisms for NIDs, but that
> has failed to achieve wide adoption and the WG (and the DDDS
> specs themselves) have decided that not all URNs are required to
> be bound to DDDS.  Maybe that was a mistake, but the horse has
> left the barn and vanished over the horizon.
Even when DDDS was being hashed out, I don't recall anyone who expected 
all URNs to be required to be bound to DDDS.   It wasn't a mistake, it 
was simply a consequence of other decisions and painfully-acquired 
understanding, e.g. (a) URNs weren't only to be used for 
network-accessible resources; (b) even among network-accessible 
resources, there would always be URNs that you couldn't resolve without 
access rights (and some for which the existence of the resource wouldn't 
even be acknowledged without access rights -- if I recall correctly Ron 
Daniel argued this quite convincingly); (c) some URN namespaces which we 
thought desirable simply didn't lend themselves to centralized 
collection of either locations or metadata.   Also, many of us realized 
that DDDS was more of a proof-of-concept than anything else.  We hoped 
we could get something deployed enough to show what could be done with 
URNs even though we realized that if it were successful, it would have 
to either evolve or be replaced.

Also, even though the DNS realization of DDDS was the most obvious one, 
DDDS is not and was not intended to be specific to DNS.

>   I'm sure, based on
> experience, that "look the NID up in the IANA registry in real
> time" is not going to be acceptable operationally (even if there
> were no other issues).

Well, no, not if we're using the current IANA servers that are only 
provisioned and maintained according to current expectations.   But we 
should all know better than to assume that.   I imagine that IANA would 
maintain the registry but not run the equivalent of the root servers for 
real-time lookup of NID space.   There are other ways of distributing 
that information than having IANA provide the service directly.   For 
example, it could outsource that via a large-scale DNS service provider 
if DNS were still used to advertise that information (which is perhaps 
not the best possible way, but certainly not the worst, and probably can 
be made sufficient). Alternatively, it could provide the top-level 
NID-to-service mapping information to the public on a rate-limited basis 
- anyone could get the information but the response would be slow enough 
that no client implementation would want to use it to do the lookup in 
real time. Publishers of software using URNs could obtain the 
information and distribute the information to their customers via normal 
software update channels.

I'm not saying I have the best answer right off the bat.  I am saying 
that arguments of the form "DDDS failed, therefore no net-based 
distribution of the NID mapping information can work" don't seem to have 
much merit.   At the very least, they need more detail and/or supporting 
evidence to be convincing.

> The result is that I don't see any
> alternative to doing some handwaving around "find a resolver
> based on the NID and do something intelligent if you can't find
> one" and then either assuming that "can't find resolver with a
> reasonable amount of effort" means "this NID does not resolve"
> or "need to return some sort of 'fail' indication".  That makes
> me _very_ unhappy but, given where we have gotten, I'm having
> trouble figuring out alternatives that do not require inventing
> a somewhat more general and flexible successor to DDDS and
> insisting that all URNs be able to use it.

For reasons alluded to above, failure will always be a possible outcome 
where URN lookup is concerned.   I don't know why that should bother us, 
because while we expected that URNs could be used to name (almost) 
anything, we never expected to be able to build an oracle that could 
find every resource that could possibly be named.

So: general and flexible successor to DDDS - yes; insisting that all 
URNs be able to use it - no.  Maybe insisting that it be potentially 
applicable to most namespaces is a reasonable goal, but we can't force 
namespaces to operate or advertise resolution services or even recommend 
some services operated by others.   And even if every applicable 
namespace operated a resolution service, they won't necessarily be able 
to populate their services with information about every URN assigned 
under their namespace (again, because of confidentiality requirements).


>
> [4] If we were modeling a process, rather than syntax and (at
> least some) semantics, we could specify the difference between
>
>    generic-engine CALLS NID-resolver(NSS, r-component)
>      GETS BACK <string>
>      CONSTRUCTS (string, decoded-q-component, f-component)
>      and returns that to wherever it goes
>
> and
>
>    generic-engine CALLS NID-resolver(NSS, r-component,
>         q-component, f-component)
>      GETS BACK a string that includes q-component and
>         f-component as appropriate
>      and returns that to wherever it goes
>
> with "string" in both cases being more or less the hier-part of
> a URL for case (b) (and I don't know how to think about (a) and
> (c), much less others).
>
> But we can't so, if we are are going to define a generic
> resolution service, we are going to need to define what it can
> return and probably the operations it performs to get there.  I
> don't think we can do that for the reasons that make the three
> different URN cases identified above as (a), (b), and (c) (and
> others) important.

I don't see the problem.   I think the NID-resolver gets called with the 
assigned-portion of the URN + the r-component, plus perhaps some 
additional client-supplied parameters indicating what the client wants 
and/or what protocol(s) and result data models the client supports (or 
maybe the client modifies the r-component to include its request).   I 
think the NID-resolver returns either:
- an error indication,
- a redirect,
- one of the types of result documents expected/requested by the client, 
which depending on the request might be a list of URLs, or a set of 
metadata,
- or maybe even the resource content itself (though maybe that's a bad 
idea).

So to me the resolution service is fairly simple in concept and can be 
defined as some HTTP FORM parameters:
?urn=urn:example:foo??some-r-component&request_type=xxx&result_types=(aaa,bbb,ccc)
Of course, working out the details of the result documents and how the 
various kinds of results relate to one another is a huge task, though I 
expect that librarians and archivists are well along this path 
already.   IETF will do well enough to establish a protocol framework, 
some basic request types, an IANA registry for result types, a generic 
syntax for r-components, and an IANA registry for r-component facet names.

Defining client behavior is a bit trickier, because the client will need 
to handle several different scenarios in order to do what the user or 
requesting resource wants.   To get resource content it might need to 
consult multiple resolution services, follow redirects, first get a list 
of URLs and then try to find one of those that satisfies its selection 
criteria (French language, PDF format).    It might find that the 
requested resource isn't available but there is a related resource that 
is available (later version, HTML format instead), and determine whether 
that resource is acceptable.   In some cases it can ask the user; in 
other cases it will have to decide based on predetermined preferences 
and/or heuristics.

Keith