[Cbor] Re: Consensus call on EDN literals single ABNF

[Carsten Bormann <[e-mail]>]

On 21. Jul 2024, at 18:25, Rohan Mahy <[e-mail]> wrote:
> 
> Hi Ira,
> Could you please give your reasoning in a few words?

I’m also not Ira.

What Christian said (606 words).

I have written up my own view below (716 words), if you care.

Grüße, Carsten


There are strong technical reasons for the two-layer model.

The original focus of the original edn-literal proposal was to have a pluggable syntax for application-specific literals.
This idea is quite successful and has been embraced already to “plug” a number of holes in EDN.
From an architectural point of view, a pluggable literal syntax fits exceedingly well with the whole idea of CBOR tags, which are a pluggable extension mechanism for the data model behind the interchange format.
(Side observation: It took us a lot of energy in 2013 to get the concept of tags accepted, because the concept was rather innovative at the time.)

A pluggable literal syntax can only be done cleanly when it is based on a common base syntax for all those pluggables.
This becomes clear when you look at section 3.1.
The functionality to handle unknown application-extension identifiers is rather important for deployability, to minimize friction when introducing new identifiers.
A lot of experience went into getting this right.

Increasing the coupling between the base ABNF and each application-extension by moving to a one-layer model would damage the proposal by making the base ABNF unstable and by requiring it to change it for each new application-extension.

I don't see a point in mashing up the ABNF grammar in the specification to make it single-level.
The argument that this has always been done this way in production parser implementation is not at all compelling for our case (the intuition here appears to come from SIP, which does require production-quality text-based parsing — exactly what we are trying to get rid of with CBOR.  OBTW, SIP/SDP is two-layer; you wouldn't munch the SDP syntax into the SIP syntax.)
EDN is not about production parsing, it is a tools and documents syntax.

More importantly, an EDN implementation does not have to be built directly from the ABNF the document uses for defining the syntax.

The next little innovation was recognizing that we could treat hex/base64 literals the same way as the pluggables; this was just an obvious simplifying step.
Actually, the various text-based representations for binary data do make great pluggables — there really is no good reason to be limited to base16/32/64, and the clear separation even allows us to cleanly defer defining base32 because we currently have no implementation experience.

Again, this clean model can be broken up and special-cased and be made more complicated in the document, but there would need to be a really good reason to do so.

The objective of including ABNF was being able to explain the syntax, and, if possible, even to generate code from that.
(Parser combinators (nom) were mentioned; I don't see a reason why nom cannot be used for directly implementing the two layer approach, by the way.)
Doing code generation directly from the ABNF is not the single, normative way to do an implementation.
There is nothing wrong with an implementer who comes up with their own grammar, and I wouldn’t mind cultivating a single-level ABNF as a separate project.
Putting this into the main document (and replacing the clean syntax) is just premature optimization, like as if TCP and IP had been described in a single document — this was certainly possible (and was done this way in 1974), but the invention of layers and the separation of functionalities was what made IP so powerful.

EDN-literals as defined today has a clear mental model for the pluggable part.
It is eminently easy to check that the syntax makes sense.
I'd say that swapping out a major part of the grammar is a rather risky late change.
There are a few other CBOR and CDDL drafts that really wait for our attention, and I really don’t want to waste more time on this subject now that we have a stable basis.

If there were any merit to doing this change, I might be more open to it.
But the upside is very limited, and it also would be a big regression.

Of course, what we really want is for ABNF itself to specify how the two-layer approach works (using the first layer to transform, then parse using the second layer).  Stay tuned...