Re: [abnf-discuss] Mail regarding draft-seantek-constrained-abnf

[Paul Kyzivat <[e-mail]>]

On 7/9/16 3:51 AM, Sean Leonard wrote:
>
>> On Jul 8, 2016, at 3:39 PM, Paul Kyzivat <[e-mail]> wrote:
>>
>> [...]
>>
>> "Defined over X" could be a predicate that a verifier could decide for a given ABNF grammar. [...]
>>
>> "Defined over ASCII" and "Defined over Unicode" could also be determined by analysis of the grammar.
>
> Up to this point, that is determined by (human) analysis of the specification text. It’s worked out overall.

IIUC this can be determined mechanically. First it is necessary to 
decide whether we mean "defined over Unicode codepoints" or "defined 
over Unicode characters". I guess defining over characters is a moving 
target, while code points are (relatively) stable.

And I think for ABNF, defining over codepoints is good enough. If you 
know that, then I/O processes that convert to/from UTF-8, etc. are valid 
to use.

> As much as I would like to see some more Unicode action, we already have two proposals on the table that I would like to focus on and get adopted & passed, before working on new stuff. The proposals are abnf-more-core-rules and constrained-abnf.
>
> In abnf-more-core-rules, there already are mechanisms for incorporating Unicode, in the productions:
>  UNICODE
>  BEYONDASCII
>  C1
>  BEYONDC1 (to be included in next draft)
>
> I suppose the following could be added:
>  LATIN1
>  BEYONDLATIN1
>
> I submit that these mechanisms are probably “quite enough” for IETF usage. There is a strong bias in ABNF for ASCII, which basically reflects the institutional bias in the IETF for US-English and the history of the Internet’s development. I have yet to see an IETF standard that calls out a specific character beyond the ASCII range and gives it special semantics (other than NEL, and even then, NEL is not really used).

I'm not convinced. You can *already* do Unicode. All the above do is 
give you some standard symbols to use for ranges of values that are 
interesting in the context of Unicode. They don't standardize any way of 
introducing Unicode characters that don't fall into the ASCII subset. 
People will still need to enter those numerically, or define symbols for 
ones they care about.

Your draft for more core rules hasn't received a lot of support - 
perhaps because it isn't perceived to add sufficient value to be worth 
the trouble.

ISTM that an important part of "supporting" Unicode in ABNF would be 
allowing use of Unicode characters in some form of quoted string. 
(Perhaps yet another new form.) That might add perceived value to those 
who really want to define grammars that actually *use* non-ascii 
characters.

I think in turn that defining that only makes sense if the grammars 
using those work over Uncode scalar values. (The alternative to that 
seems to be to define those literals to expand into UTF-8, so each 
character in the string might represent multiple bytes. And then you 
need to know what encoding the parser is running over.)

That in turn presumes some way to get unicode scalar values into the 
parser. I think maybe that requires some sort of "framework" that 
discusses input/output to the parser.

OTOH, I don't see a groundswell of users clamoring for this sort of 
thing. So for now I think this is ought to just be a background discussion.

	Thanks,
	Paul

	Thanks,
	Paul