[abnf-discuss] Indentation and parsing (Re: [Cbor] cddl 0.8.17: Add .abnf (draft-ietf-cbor-cddl-control))

The CBOR WG maintains CDDL, the ABNF equivalent for data structured in a JSON or CBOR like manner.

Over at cbor@ietf.org, we are currently discussing the integration of ABNF support into CDDL for describing the content of text and byte strings [1].  

[1]: <https://www.ietf.org/archive/id/draft-ietf-cbor-cddl-control-02.html#name-embedded-abnf>

A prototype for that support already is implemented in the original CDDL tool; support in another CDDL tool is in progress.

The one surprise that I had when I implemented this was that I had completely forgotten that ABNF is indentation sensitive (or, more precisely, rules out indentation except on continuation lines, where indentation is required).  Tools like BAP have been working around this aspect in a sufficiently effective way that I simply lost track of that fact.

I think we have a way forward for dealing with this in CDDL (namely, dealing with indentation before handing over parts to ABNF).
But several interesting side discussions ensued, and Paul suggested below that maybe we should take this to abnf-discuss now.

One question was how the fact that proper ABNF does not accept indentation on a rule can be repaired.  A followup question was, of course, how much the parsing of ABNF depends on the way indentation is handled.  More below; a whole thread is in the archive [2].  (Another question is how a relaxed syntax could be expressed in ABNF itself.)

[2]: <https://mailarchive.ietf.org/arch/msg/cbor/p5D3G6DNy5ZAiurp2HLA4E8VtAM>

Opinions welcome; I have set reply-to to abnf-discuss (but am aware that this setting is ignored for wide-replies by most mail readers).

Grüße, Carsten

> On 2021-02-27, at 16:41, Paul Kyzivat <pkyzivat@alum.mit.edu> wrote:
> 
> On 2/26/21 6:51 PM, Carsten Bormann wrote:
>> Hi Paul,
>>> The obvious revision would be to simply change the syntax of rule to:
>>> 
>>>     rule           =  WSP rulename defined-as elements c-nl
>> Well, a single WSP won’t cut it…
>>     rule           =  *WSP rulename defined-as elements c-nl
>>> 
>>> That seems unambiguous to me. But my formal parsing knowledge isn't sufficient to say if it is LL(1).
>> Once you allow white space before rulenames, it is most definitely not LL(1).
>> When you see…
>> A = B
>>   C
>> …you need to look ahead one more to distinguish, say…
>> A = B
>>   C = D
>> (New rule)
>> …from…
>> A = B
>>   C D
>> (Still in same rule.)
>> I do think it is LL(2), but I didn’t formally check.
> 
> If there is no separate lexer, so that terminal symbols are all single bytes, then isn't the required effectively unbounded?
> 
>> Who else should we draw into this conversation?
> 
> We should restart the discussion on abnf-discuss.
> 
> 	Thanks,
> 	Paul
> 
> _______________________________________________
> CBOR mailing list
> CBOR@ietf.org
> https://www.ietf.org/mailman/listinfo/cbor