Re: [alto] A unified approach to value schemas and ALTO maps

[Wendy Roome <[e-mail]>]

Jensen,

Thanks for commenting!  My responses are in-line.

> From:  Jensen Zhang <[e-mail]>
> Date:  Tue, July 26, 2016 at 21:18
> Subject:  Re: [alto] A unified approach to value schemas and ALTO maps
> 
> Hi Wendy,
> 
> I want to support this idea. Please see comments inline.
> 
> On Wed, Jul 27, 2016 at 1:38 AM, Wendy Roome <[e-mail]>
> wrote:
>> Recently we talked about using "value schemas" so that we can add new
>> value types without defining new media types and message formats.
>> 
>> Here is another take on that approach. It unifies *everything* into one
>> common message format. Extensions simply require defining new entity
>> domains and (simple) value types.
>> 
>> So here's my idea. Every ALTO resource boils down to a mapping from 1-,
>> 2-, 3- or 4-tuples to values. The tuple elements are names of entities in
>> a domain.
>> 
>> Examples:
>>    Cost Map: (pid, pid)  =>  value
>>    Endpoint Costs:  (addr, addr) =>  value
>>    Endpoint Props:  (addr, prop-name)  =>  value
>>    MultiCost Map:   (pid, pid, cost-type)  =>  value
>>    MultiCost Calendar Map: (pid, pid, date-range, cost-type)  =>  value
>>    Network Map:  (cidr)  =>  pid
>> 
>> Note that I flipped the network map around. I think cidr => pid is cleaner
>> than pid => cidr-array, and it enforces the rule that a cidr is in only
>> one pid.
> 
> Agree. And maybe another benefit from cidr => pid is for incremental update.
> Once a cidr changes its pid, pid => cidr-array will require an update of the
> whole cidr-array. But cidr => pid only requires an update of one pid item.

Exactly! 

>> 
>> With this approach, the meta section of a response would give the domain
>> for each tuple element, and the value type. For example, here is a
>> conventional cost map:
>> 
>>    meta:
>>       tuple-domains: [pid, pid]
>>       value-type:
>>          specification: rfc7285
>>          format: cost-type
>>          parameters:
>>              metric: routingcost, mode: numerical
>>    map:
>>       pid1:
>>          pid1: ###,  pid2: ###, ...
>> 
>> The value field gives document that specifies this value format (or a
>> registered name), a format name that is defined in that document, and any
>> additional parameters necessary to understand this value.
> 
> The specification metadata is very interesting and I like this design. But I
> don't think RFC is the good solution for the specification citation. RFC is
> human-readable, but it is hard for programs to parse RFC documents. Maybe we
> find a schema language (or JSON schema) to specify the value format.

Okay, here is where I respectfully disagree. I do not believe a
program-readable schema will help.

Why not? First, JSON is self-describing. An ALTO client does not need a
schema to parse JSON. JSON libraries do that on their own. The JSON
libraries I¹ve used build a DOM from the JSON and allow client programs to
explore that DOM.

IMHO, formal schemas have two uses. One is to allow a program to verify that
a given JSON tree matches the schema. However, if the server¹s response does
not match the schema, what can a client do about it, other than give up on
that ALTO server?

The other is to document the layout, and (maybe!) facilitate automatically
creating a program, or at least a skeleton of a program, to program with
that data. But no ALTO client is doing to do that on-the-fly. That happens
off-line, when the programmer decides how the client will use an ALTO
server.

I believe the important thing is to specify the semantics. E.g., what do
these values mean? For example, take ordinal vs numerical mode. You cannot
capture the difference in a schema.

As another example, consider duration values. Frequently durations have
suffix with the units: 10s, 1.23ms, 15us, etc. Or they have colons, as in
1:23.4. A grammar can define the legal values, but unless the programmer
knows "ms" means milliseconds, the grammar does not help.

So I believe values will have to be described by some document, which may
have a formal schema. The document will define some unique name to identify
values of this kind, and ALTO clients will switch on that name. The name
could be rfc####, or it could be yet another IANA registry, or whatever.
>  
>> 
>> Here is a multi-cost map:
>>    meta:
>>       tuple-domains: [pid, pid, cost-type-name]
>>       value-types:
>>          3:
> 
> Is this a typo? Or what's the mean of '3:' here?

The 3 means the value-type depends on the value of the third tuple element -
the cost-type. But now that I look at it again, that is overly general. It
is probably sufficient to say that either all values are the same type, or
else the value type depends on the value of the last element in the tuple.
E.g., cost-type, property-name, etc.
>  
>>             cost:
>>                specification: rfc7285
>>                format: cost-type
>>                parameters:
>>                  metric: routingcost, mode: numerical
>>             delay:
>>                specification: rfc####
>>                format: duration
>>                parameters:
>>                   metric: propagation-time
>>    map:
>>       pid1:
>>          pid2:
>>             cost: 123, delay: 2ms
>> 
>> Here the different cost types have different value types. meta.value-types
>> means that the value depends on the 3d element of the tuple - the
>> cost-type-name - and gives the value format for each cost-type-name.
> 
> It is also more flexible for incremental updates.
>  
>> 
>> 
>> So are people interested in pursuing this approach? The good news is that
>> will unify everything. The bad news is it will replace everything in rfc
>> 7285.
>> 
>>     - Wendy Roome
>> 
> 
> I find this approach is also compatible with FCS, since FCS returns a fid =>
> value map currently. I'd like to push it forward.
> 
> One more question: I think (src, dst) => cost map and (dst, src) => cost map
> will have the same "tuple-domains" value, but they are different for clients.
> How to differentiate them?

Because the domains in the tuples are ordered. E.g., in a cost map, the
first domain is always the source and the second is the destination.
> 
> Best,
> Jensen

Thanks! Let's keep the discussion going. Anyone else like to comment?
Richard??

- Wendy Roome
>