Re: [Ltru] [apps-discuss] Fwd: Defining a CBOR tag for RFC 5646 Language Tags

["Doug Ewell" <[e-mail]>]

Dave Cridland <dave at cridland dot net> wrote:

> Of course, an invalid-UTF-8 based proposal simply means that it's no
> longer UTF-8 per-se, and so needs itself to be tagged differently.
> Other than that, I don't see it's a bad idea from a technical
> standpoint. The use of the word "invalid" probably scares people, but
> I note that's really a shorthand for "not backwards compatible by
> existing UTF-8 processors".

The proposal from 1997 ("MLSF") did call it an extra layer on top of
UTF-8, and included lots of health warnings that it was not really
UTF-8. That didn't remove the danger, though, because it looked so much
like UTF-8. John's response about decoders was spot-on.

> Exactly the same caveats apply to Plane 14 tagging, mind, and
> moreover, we could invent our own - indeed, that's what we're doing by
> having these arrays of (tag, string) tuples.

Since CBOR processors are likely to be bespoke, as you said, but the
underlying UTF-8 processor is not -- as you also said -- it would be
much easier to strip out Plane 14 characters for display if necessary
than to implement a whole new UTF-8–like decoding-stream layer that
understands MLSF.

As Mark knows, I never bought into the deprecation argument about how
evil Plane 14 tag characters are. Handling them correctly just isn't
that difficult. For CBOR, you may be better off with the tag/string
tuples; the tags in that case are much easier to see and don't need to
be stripped from the string for display or comparison. But if this
"tagged text" model is too far out of step with the CBOR/JSON way of
thinking, Plane 14 is out there.

--
Doug Ewell | Thornton, CO, USA
http://ewellic.org | @DougEwell