[Tools-discuss] Re: Exploring ABNF extracts from RFCs

[Carsten Bormann <[e-mail]>]

Hi Dominique,

> (I hope this list is the appropriate place to bring this project up; let me know if there is a better place)

Not sure, as it both is about tools we use and the specs that make use of these tools, so I have added [e-mail] and a few more lists and will do a full cite in here.

> Inspired by some of the work happening in the W3C community around extracting, curating and publishing reusable data and code from Web specifications [1], I have been exploring what this might look like for RFCs.

This is much-needed work!
I have started a much simpler effort that does similar things for CDDL [c1] [c4], so I think we will be able to compare our approaches and maybe find some common way to deal with “exportable” from specifications.

[c1]: https://www.ietf.org/archive/id/draft-bormann-cbor-rfc-cddl-models-03.html

> I had initially planned to present and continue that exploration during the IETF 120 hackathon [2], but didn't realize the hackathon start was moved to Saturday, so ended up working on the project on my own.
> 
> The current state of my exploration is visible in two github repos:
>  https://github.com/dontcallmedom/rfcref/tree/main/abnf
>  https://github.com/dontcallmedom/ABNFary
> 
> I have in particular focused on extracting ABNF from (some) RFCs (re-using aex on pre-XML RFCs, and a simpler XML-extraction script on XML RFCs), and crucially, identifying the dependencies that exist between ABNF fragments (where e.g. the URI production defined in RFC3986 gets "imported" into e.g. RFC9110):
>  https://github.com/dontcallmedom/rfcref/tree/main/abnf/dependencies

I put up extracted sourcecode from the 8650+ RFCs [c7] and the XML I-Ds [c8], which already has turned out to be useful for a lot of applications but needs some additional care and more frequent updates; I hope something like this becomes rsyncable content in the near future.

[c7]: https://user.informatik.uni-bremen.de/~cabo/rfc/
[c8]: https://user.informatik.uni-bremen.de/~cabo/i-d/

> With these dependencies semi-manually detected and documented, a (node.js) tool I wrote allows to build consolidated ABNF files that are guaranteed to have all their rules defined:
>  https://github.com/dontcallmedom/rfcref/tree/main/abnf/consolidated

This is very much what we need to make ABNF more prominent in building (and not just specifying) our applications.

At some point we might want to have a syntax where a new document’s ABNF can point to another document’s ABNF and explicitly import from that.  CDDL is gaining such a syntax [c2], and a similar syntax for ABNF should be easy to construct (and would actually become part of the CDDL ecosystem as CDDL includes ABNF).

[c2]: https://www.ietf.org/archive/id/draft-ietf-cbor-cddl-modules-02.html

A rough snapshot of a consolidated set of CDDL modules is part of the cddlc tool [c3], you may specifically want to look at [c4].

[c3]: https://github.com/cabo/cddlc
[c4]: https://github.com/cabo/cddlc/tree/master/data

> (I identified a couple of errata in the process of consolidating some of these RFCs [3])

Indeed, not all RFCs are providing good exportable ABNF (or even CDDL); e.g., some reuse the same rule name with different right hand sides…

> Finally, a built a simple Web interface on top of the said consolidated ABNF that allows to see interlinked ABNF rules, with railroad diagrams illustrating the "endemic" rules of a given RFC:
>  https://dontcallmedom.github.io/ABNFary/?num=9110
> 
> The broader motivation for that project is to enable easier and broader usage of ABNF definitions (and more generally, data/code provided in RFCs); the similar effort in W3C I mentioned [1] has found impactful re-use in implementation conformance and interop testing, developer documentation, IDEs, libraries, fuzz-testing, spec authoring tools, spec validation.
> 
> Although the errata I filed show potential, I'm seeking input and feedback on whether this project (and in particular its current focus on ABNF) is likely to provide opportunities for similar re-use.
> 
> And if so, I'm also interested to hear what would be the best way to make this a collaborative effort that would align best with the IETF community expectations and processes.
> 
> (I'm about to disappear for a few weeks, but thought I would start this discussion while the project is still fresh in my mind)

I think that this is a great discussion point for the next interim of the CBOR WG (August 21?  I think we’ll set that date in the CBOR meeting today).  The WG is in the process of completing [c2]; we certainly want to be able to make this as useful as possible with the existing body of specs and probably need similar tooling.

Apart from ABNF and CDDL, the third spec language we probably want to think about is YANG [c5]; this has yangcatalog.org [c6], but also a lot of datatracker and IANA support.  With YANG, there has been much more attention to making models referenceable, so we might learn a bit from them as well.

[c5]: https://www.rfc-editor.org/rfc/rfc7950.html
[c6]: https://www.yangcatalog.org/

Grüße, Carsten

> Dom
> 
> 1. https://github.com/w3c/webref/
> 2. https://wiki.ietf.org/en/meeting/120/hackathon#collecting-consolidated-abnf-from-rfcs
> 3. https://www.rfc-editor.org/errata/eid8040 https://www.rfc-editor.org/errata/eid8039
> 
> -----------------------------------------------
> Tools-discuss mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> https://mailarchive.ietf.org/arch/browse/tools-discuss/