[T2TRG] Syntax evolution for SDF (OneDM Simple Definition Format)

[Carsten Bormann <[e-mail]>]

The SDF (Simple Definition Format, [1] [2]) developed by the One Data Model liaison group is represented in JSON.

To define the syntax for this format, currently a json-schema.org-style schema is being used [3].  This has been defined in such a way that future extensions to the format are accepted (a.k.a. "ignore unknown rule").
For his SDF file linter [4], Ari Keränen has developed an “alternative schema” that is less loosely defined — after all, you want to catch misspellings in a linter, as opposed to silently accepting them as if they were future extensions.

I think that is a symptom of a problem we very often have with data definitions (“schemas”): we want a good representation of what a current valid instance would be, which could be processed by the majority of the tools and reflects the current consensus of the spec evolution.  But we also want a more loosely defined data definition that outlines the boundaries of where we think extensions might put their data; the latter would be the basis for “validating” an instance, while the former would be used for “linting" it.

For SDF, I’ll call the first one (for linting) the “current detailed” syntax.  The second would be the “framework” syntax, with an intent to keep this invariant until we have extensions that don’t just exercise known extension points, but actually represent larger changes.

The “alt-schema” [5] is trying to be that “current detailed” syntax.
The definition in the language definition repository is more on the side of a “framework” syntax.

Now this is a snapshot.  When we evolve the language, the “current detailed” syntax morphs into the “next detailed” syntax, where additional features have been added, with the intent of that becoming the “current detailed” with the next minor revision of the standard.  The “framework” syntax should be a superset of either specification.  But the framework will also need to change at some point, requiring both a “current framework” syntax and drafts for a “next framework” for a major revision.

Below is some graphics; if you don’t have SVG, here is an ASCII-art version of that.


.-----------------.              +-----------------+
|                 |              |                 |
|   Current       |              |  Next           |
|   Framework     ===============>  Framework      |
|                 |              |                 |
|                 |              |                 |
|_________________|              +-----------------+
         | \
         |  \                ==>  Evolution
         |   \               -->  Subset
         |    \
         |     \
         |      \
         |       \
         |        .------------------+
         |                            \
         v                             v
.--------+--------.              +------+----------+
|                 |              |                 |
|   Current       |              |  Next           |
|   Detailed      ===============>  Detailed       |
|                 |              |                 |
|                 |              |                 |
|_________________|              +-----------------+


One question that Ari brought up is how a maintain these syntax specs, preferably from a single input file can do multiple of these outputs.
Obviously, this can be ifdefed, or one can use m4, etc.
One can also use a program to generate the definition json, which probably is a reasonable thing anyway for quirky, hard to get right definition formats.
Best would, of course, be a data definition language that directly caters toward evolution.

Any suggestions how we should handle this issue?

Grüße, Carsten


[1]: https://github.com/one-data-model/language
[2]: https://www.youtube.com/watch?v=sTrqa5jYVKo — Video of brief Tutorial
[3]: https://github.com/one-data-model/language/blob/master/sdf-schema.json
[4]: https://github.com/EricssonResearch/ipso-odm/blob/master/odmlint/odmlint.js
[5]: https://github.com/EricssonResearch/ipso-odm/blob/master/sdf-alt-schema.json