Re: [Cbor] CDDL Codegen in production

[Robert Aftias <[e-mail]>]

Hi Carsten

I'm the dev of our cddl-codegen tool. To address your comments most of the
limitations were simply due to time constraints and priorities. The one
thing that I can think of that wasn't purely a time issue is being able to
define human readable names to types/fields in some contexts: specifically
map keys where the key value is not very informative. I thought about using
comments to do it, but in the cddl-parsing library that Andrew wrote the
comments are stripped out. However, even with comments being preserved,
especially without taking whitespace into account, associating the comments
unambiguously to fields/types would be difficult when for example map keys
are units or other types that don't provide much context.

This I imagine could be a common situation (maps with uint keys) as it
shrinks the resulting CBOR size vs having descriptive key names. Encoding
as an array (so we don't store the keys in cbor and we can have descriptive
names) with optional fields can result in more complicated deserialization
code in some situations due to increased ambiguity during deserialization
(think [?uint, uint, ?text, text] vs { ? 0: uint, 1: uint, ? 2: text, 3:
text}) which is part of the reason why our codegen lib doesn't support
optional fields in deserialization, along with time constraints and it not
being used in our partner's cddl files.

For example to arrive at a meaningful name for key 1 we could comment it as:

foo = {
 ? 0 : bar,
 ; meaningful-name
 ? 1: uint,
 ? 2: baz
}

or

foo = {
 ? 0 : bar,
 ? 1 : uint, ; meaningful-name
 ? 2 : baz
}

but AST-wise those would be in different places - between keys 0 and 1 or 1
and 2 in the two examples, so without taking whitespace into account it's
ambiguous which key the comment is describing. Even with whitespace it's a
little subjective if you formatted the cddl in other ways.

Even if we don't use the comment for the field name it would be very nice
to be able to inject the comment into the resulting rust code.

Our cddl-codgen tool will use the field/key name as the rust field name if
it exists, and use the type otherwise, but in this situation it's not very
helpful to have a bunch of fields named key_0, key_1, key_2, etc.

This also happens when you inline types within other types like
foo = [
    0: [uint, text].
    1: { * uint => text }
]
since we need to define new types here for both due to wasm-bindgen (rust
to wasm auto-conversion) restrictions. However, in this case it's not a big
deal at all since it'd probably be better practice anyway to just define
new types and refer to them in the outer type and this wouldn't impact the
CBOR representation of the CDDL. We did that to get around uninformative
names for variants of group choices too.

Maybe there's another reasonable approach though for the field names where
restructuring without changing the underlying CBOR can't achieve it?

A cddl-to-cddl topological rule sorter would possibly be useful to have for
others. Right now to get around that our lib does a multi-pass over the
cddl for figuring out if some features are unsupported transitively as I
didn't have the time to implement a proper type dependency system. The
multi-pass gets the job done but it's not the most elegant solution. It
does however preserve declaration-order which could be desired in the
resulting codegen. For sockets I just didn't look into how codegen would be
impacted at all since the cddl files we were parsing didn't use them in a
necessary manner so I didn't devote any time to it.

A ccdl-to-cddl generics resolver would get rid of the need for us to
support generics. Andrew's cddl parsing lib does have generic information
that I believe I could use to implement generics in our codegen tool, but I
just had other priorities and didn't have time to try and implement it yet.

Thanks,
Rob

On Fri, Sep 4, 2020 at 4:05 AM Carsten Bormann <[e-mail]> wrote:

> Hi Sebastien,
>
> this is very encouraging news!
>
> Yes, I’m very curious.  We are seeing a lot of pickup of CBOR in the
> Fintech space, and it is nice to see that CDDL is being embraced, too.
>
> I see that you have open-sourced the code generator at
> https://github.com/Emurgo/cddl-codegen (with MIT License).
> I’m impressed to see that you achieved all that in ~2700 lines of code.
>
> > There are still rough edges to our codegen library because some features
> of CDDL don't work well with codegen (hopefully CDDL can improve this going
> forward!),
>
> We are certainly interested in hearing what CDDL features are troublesome
> to implement in a code generator.  The github repo lists a number of
> features that are not implemented, but doesn’t say whether you just didn’t
> need them or they are simply too much work.
>
> Some of these features could be addressed by CDDL-to-CDDL preprocessors.
> E.g., maybe one of the functions I could add to the cddlc tool is a
> Generics resolver, as well as a topological rule sorter (which might then
> also do the necessary work for sockets).
>
> Grüße, Carsten
>
>