Re: [abnf-discuss] Are the Core Rules (Appendix B.1 of RFC 5234) always present in an ABNF spec?

[Paul Kyzivat <[e-mail]>]

Dave,

Comment at end

On 9/20/23 12:26 PM, Dave Crocker wrote:
> On 9/20/2023 8:20 AM, Carsten Bormann wrote:
>> The problem here is that RFC 8259 seems to (implicitly) import “DIGIT" and “HEXDIG" from RFC 5234 Appendix B.1, but also defines a rule “char”, which conflicts with “CHAR” in Appendix B.1 (rule names are case-insensitive per Section 2.1 of RFC 5234).
>>
>> Is this a bug in 8259?
> 
> Probably.  To use a rule that is not defined within the document using 
> it has to mean that it comes from somewhere else.  (This is a tautology, 
> but seems to be needed as a foundation for the concern here.)
> 
> If it comes from somewhere else, how is that importation specified?
> 
> If it is implied, how is a reader to reliably and accurately know what 
> is imported and where it comes from?  Readers vary widely in subject 
> matter knowledge and a specification needs to work for all readers.
> 
> In other words, it must not be 'implied'.  It has to be specified 
> explicitly.
> 
>> And, if yes, is that bug that “char” and “CHAR” conflict (i.e., because B.1 is always implicitly imported), or that 8259 forgets to import DIGIT and HEXDIG (i.e., because B.1 is not implicitly imported)?
> 
> This hits some different issues, IMO.
> 
> One is formal specification requirements.  To that, anything clearly and 
> precisely defined in fine.  So if a rulename is common elsewhere, there 
> is nothing wrong with defining it differently, within the 4 walls of a 
> given specification.
> 
> The other is a human factors concern.  If readers are generally already 
> used to using the rulename and that rulename has a particular semantic, 
> defining it differently within a spec creates a cognitive conflict, 
> called proactive inhibition.  That invites error.
> 
> However sometimes it can be useful to note the history of the name and 
> explicitly redefine it.  As long as the rationale for doing this is 
> clear and documented, it might make sense to do. This would be an 
> explicitly 'derivative' specification of the rulename, rather than 
> purely replacement.
> 
> Lastly, while it might make sense to say that a spec citing the ABNF 
> spec can be assumed to be importing rules from it, that seems to me a 
> pretty risky approach to spec writing.  For example:
> 
>   * there is nothing in the ABNF spec that declares these rules to be
>     automatically imported by users of ABNF
>   * having something in an Appendix means it is extra, and therefore not
>     'inherent'
>   * the language at the start of the Appendix is "This appendix contains
>     some basic rules that are in common use." which is casts this as
>     informative rather than normative.

I agree with all of what you say.

But because there is a practical need to "import"/"copy" rules from one 
spec to another and 5234 doesn't provide one, various ad hoc mechanisms 
for specifying this have evolved across multiple documents. Some are:

1) verbage in the explanatory text of the document explaining that 
certain rules are to be taken from another document

2) an ABNF comment explaining the same thing

3) use of a prose-val, such as:
	CHAR = <as defined in RFC5234>

I generally prefer (3) when only a few rules are needed and their 
definitions don't reference lots of other rules. If the situation is 
more complex then I generally use (2).

This comes up a lot in documents that extend another document, including 
extending the ABNF in the other document. (Preferably the base document 
will have included "extension points" in the grammar.)

It can be hard to verify the ABNF in a document that extends the ABNF of 
another document. It is necessary to actually cut and paste to create 
the composite ABNF grammar and then check it. (Or implement it.)

Its even harder when there are multiple extension documents to a single 
base document. Their rule definitions can potentially conflict. 
Verifying that a new one doesn't conflict requires searching out all 
other extensions.

I would *like* to see ABNF extended to have a more formal mechanism for 
importation of ABNF, and for extension of ABNF grammars. This would 
facilitate automatic verification of ABNF in documents. But I'm dubious 
whether there is sufficient interest to pursue this.

	Thanks,
	Paul