Re: [netmod] instance file parsing

[Andy Bierman <[e-mail]>]

On Tue, Dec 4, 2018 at 9:14 AM Robert Wilton <[e-mail]> wrote:

>
> On 03/12/2018 18:13, Juergen Schoenwaelder wrote:
> > On Mon, Dec 03, 2018 at 09:56:48AM -0800, Andy Bierman wrote:
> >> On Mon, Dec 3, 2018 at 3:36 AM Juergen Schoenwaelder <
> >> [e-mail]> wrote:
> >>
> >>> On Mon, Dec 03, 2018 at 10:21:20AM +0000, Robert Wilton wrote:
> >>>> On 30/11/2018 19:28, Juergen Schoenwaelder wrote:
> >>>>> I doubt it makes sense to carry an entire yang library schema with
> the
> >>>>> instance data. The modules are actually known via the namespaces.
> >>>> Knowing the module name or namespace isn't sufficient to be able to
> parse
> >>>> the data:
> >>>>   - E.g. if the contents are in the format of RFC7895bis, but the
> client
> >>>> tries to parse it using the schema from RFC7895, then it will fail.
> >>>>   - Once NBC changes are allowed via YANG versioning it will become
> more
> >>>> necessary to know the revision or version of the modules to be able to
> >>> parse
> >>>> the data.  E.g. not even just the latest module revision will
> necessarily
> >>>> work.   It is entirely plausible that different server revisions may
> be
> >>>> generating instance data using slightly different schema.
> >>> Perhaps this is a problem of the versioning solution then. ;-)
> >>>
> >>>> If the server has deviations then the client may also need to know
> those
> >>>> deviations to correctly parse the file.
> >>>>
> >>>> So, I think that it is pretty clear that knowing the right schema is
> >>>> required to be able to correctly parse and interpret the instance
> data.
> >>>>
> >>>> I think that there are 4 ways that this can be achieved:
> >>>>   1) embedded the necessary schema information into the YANG instance
> >>> data.
> >>>>   2) put the necessary schema information online somewhere and have a
> URI
> >>>> reference to it.
> >>>>   3) some combination of (1) and (2), e.g. packages defined centrally,
> >>> with
> >>>> deviations listed in the file.
> >>>>   4) the schema is determined using some out of bound mechanism, or
> >>> possibly
> >>>> it is hard coded.
> >>> Perhaps we need to define first what "parse" means. I am not sure
> >>> parsing requires to know all schema details. Anyway, if all schema
> >>> details are needed, then the simplest solution seems to be to read the
> >>> schema from another instance file. This then only requires to
> >>> bootstrap the schema model version.
> >>>
> >>>
> >> I agree with Rob.
> >> The solution has to support accurate parsing of the instance data.
> >> This means that the parser has the correct schema tree to use for
> >> validating an instance document against the schema.
> >>
> >> Since the new yang-library can have a different schema tree for each
> >> datastore,
> >> clearly the option of specifying the datastore is needed.
> > We need to get the terminology straight. I can parse XML and JSON
> > files without the need to have schema information available. So I
> > think the word 'parsing' is quite misleading here. And I can extract
> > data out of XML and JSON files that I find interesting without schema
> > information as well. So its a certain type of tools that may take
> > advantage of being schema aware but not all tools need to be schema
> > aware.
> >
> >>>> I don't think that there is a one size fits all answer here.  I think
> >>> that
> >>>> that it depends on the use case.  Certainly, I think that
> facilitating 1
> >>> -3
> >>>> is useful, but they should be optional rather than mandatory.  I.e.
> >>> defining
> >>>> nodes for these doesn't seem to cost much if a server isn't obliged to
> >>>> populate them.
> >>>>
> >>>> I do think that YANG packages (themselves defined in instance data
> >>>> documents) could be very helpful here.  I.e. rather than listing all
> the
> >>>> modules, instead list the packages + any deviations from that.  I'm
> >>>> presuming here that the definitions of the packages are available via
> a
> >>> URI.
> >>>
> >>>
> >> Yes -- this has been my goal for YANG Packages since the start.
> >> By using nested packages and offline caching, the entire YANG library
> for a
> >> device
> >> could be recognized in a few URIs.
> > An instance file storing the schema tree is an offline cache. It falls
> > out of the instance file format for free. Yes, packages are yet
> > something entirely different but so far no work in this direction is
> > chartered so we should get instance file formats defined without
> > solving another bigger problem first.
>
> In the IETF 103 discussion on YANG versioning, it seemed to be the
> consensus in the room that the versioning design team need to consider
> also versioning of sets of modules, rather than just individual
> modules.  I.e. something along the lines of YANG packaging.
>
>

I am not suggesting YANG Packages has to be solved in order to
work on this instance document format. The IETF thinks YANG modules
are the appropriate level of abstraction.

Issues like "the extra metadata takes up too much space" are entirely
subjective.
Obviously a parser does not need any schema to parse anydata.
This is only needed if there is a real schema expected within the anydata.

It is optional for the writer to add the extra metadata.
It is optional for the reader to use the extra metadata.
Not sure why the solution should forbid this functionality.



> Thanks,
> Rob
>


Andy




>
>
> >
> >
> >>>>>    If
> >>>>> you want to capture the schema, dump the relevant yang library into
> >>>>> another instance file.
> >>>> That just means another file to carry around and manage.
> >>> Yes, but compared to solutions that require new and/or much more
> >>> elaborate data formats, this is very cheap and efficient, in
> >>> particular for systems where the schema itself is relatively
> >>> static. Also in terms of engineering time this is a rather cheap
> >>> solution since you do not need to invent a new way to document and
> >>> communicate a schema.
> >>>
> >>>
> >> I agree the cost of an extra instance document is not a problem,
> especially
> >> since
> >> it would be optional and only used if the defaults are not sufficient:
> >>     - latest revision of module will be OK
> >>     - assume all features present will be OK
> >>     - assume no deviations will be OK
> >>
> >> If the defaults are not OK, then the parser will incorrectly accept or
> >> reject certain instance data,
> >> unless the correct schema tree is obtained somehow.
> > Or the tool processing instance file content just does not care about
> > the schema and it simply assumes that a certain (module,path)
> > combination has fixed semantics. I am big fan of being able to use
> > generic tools to filter and process XML or JSON encoded data.
> >
> > /js
> >
>