[core] CBOR Tag 38 (in draft-ietf-core-problem-details)

[Carsten Bormann <[e-mail]>]

draft-ietf-core-problem-details is a specification that enters somewhat new territory for CoRE WG specifications:
It is concerned with carrying around data that include human-readable text, where the human is an end-user and not a programmer.

We could address this with something that is specific to CoRE, but it is better to use a mechanism that is common to the CBOR ecosystem.

Internationalization experts insist on having language-tags with human-readable text, as the rendering of the text may differ between different languages.
BCP47 is the way we do language-tags in the IETF.

We actually have a tag in the CBOR tags registry for that: CBOR tag 38, http://peteroupc.github.io/CBOR/langtags.html
This was registered by Peter Occil to be able to ship language-tags with text, and it solves the problem nicely.

Except that language-tags themselves have a problem:
Other than maybe originally thought, they are mostly useless for establishing base direction.
Some writing systems are left-to-right (ltr), others are right-to-left (rtl) (and some are more complex, but not much in use commercially).

For most of us this problem is invisible, as we are firmly rooted in the left-to-right world.
W3C has written up some of this:

https://www.w3.org/International/questions/qa-direction-from-language.html
https://www.w3.org/TR/international-specs/#text_direction

TL;DR: As a language tag does not really indicate a writing system, you need to be able to separately establish a base direction with a language-tag.

So, it seems that CBOR tag 38 has a bug, in that it doesn’t address this essential component of using language-tags with text.

How do we fix this bug?
(1) we could define another CBOR tag (pair of tags) that just establish the base direction for their tag content.
(2) as the most straightforward way of addressing the bug, we could add an (optional) base direction to the array inside tag 38.

The latter could be done in two ways:
(2a) we define a new tag, let’s call it N38 here, that allows the optional element.
(2b) we could fix up 38 to allow the optional element.

So far, we have not done the equivalent of 2b to CBOR tags.  
E.g., for tag 4 and 5 there are analog tags 264 and 265 that simply increase the range of exponents (by allowing big integers).
For tag 4 and 5, this is probably actually the right way to handle this belated extension: Most applications of tag4/5 do not need the extended exponent range.

But tag 38 really is broken the way it is.
Of course, it works fine where you don’t happen to need the indication of a base direction.
But it is not like this is a different area of application (like with 264 and 4): *Any* language tagging really needs this.
Any application that limits itself to what tag 38 can do today as is, is essentially broken from an internationalization point of view.

To me, this looks like a strong incentive to do 2b.
But this (expanding the domain of an existing tag) is a new thing in CBOR land, so I thought I might want to bring this up here.

Specification-technically, the reference document for tag 38 would change from Peter’s document to the appendix in the problem-details specification.

2a would probably need another 1+1 tag for N38, or applications may not migrate to the new tag (cognitive dissonance).
The use of N38 is not forward compatible ever:  
An application that switches to the new tag can’t interoperate with one that expects 38 even if it does not need to establish a base direction.
Experience with such a situation indicates that this often creates a deployment barrier to migrating to the new form.

2b clearly is backward compatible: old data (and old producers) work with new consumers.
It is not forward compatible (new data/new producers do not work with non-updated old consumers) only when the base direction feature is actually used.
This is very similar to a lot of other extensions we do routinely, such as adding new Unicode characters.

Of course, I’m assuming here that applications that employ language tags will want to employ them correctly and therefore will be updated to consume base direction indication.
With that, I believe that 2b is the best way to handle this bug.
We don’t have a feature matrix showing CBOR implementations and tags supported right in the generic decoder, but I would surmise that the number of implementations that actually have to change to accommodate base directions is still quite manageable.

We will discuss problem-details in a couple of hours in the CoRE interim meeting:
See <https://mailarchive.ietf.org/arch/msg/core/t0vAGs8hCGqFgsSKgp9TEXEdzp0> for details.
If we want to delve more deeply into the impacts that updating this tag would have on the CBOR ecosystem, we can do this a week later in the CBOR interim.

As an SDO (3GPP) waits for us to complete problem-details, we’ll need a decision soon.
So please reply to this message (which reaches both WGs), or come to one of the meetings.
(I didn’t include an i18n mailing list, because the i18n discussion is complex, but has essentially converged.)

Oh, and while you read the W3C documents, please note that neither the existing tag 38 nor the various proposals to fix/complement it solve delimiting embedded runs; this would be getting a bit closer to creating a little markup language though.

Grüße, Carsten