Re: [Ietf-languages] Latin Sub tags

[Mark Davis ☕️ <[e-mail]>]

> In January 2010, ISO 639-3/RA reclassified the individual language [lav]
Latvian as a macrolanguage, encompassing the two new individual languages
[lvs] Standard Latvian and [ltg] Latgalian.

If there is no code for a given language X, people will often end up
tagging it with a language code Y that does exist where common names for X
reference the name for Y (even where there is little mutual
comprehensibility between X and Y). But then later on, a code is assigned
for X. The question is, what to do about the relationship between X and Y?

(Below I’m following BCP47 in using the ‘shortest form’, thus ‘de’ (ISO
639-1) for ‘deu’ (ISO 639-3), ‘ar’ = ‘ara’, etc.)

   1.

   *No macrolanguage. *This model is the one used for German, French, and
   many other languages. For German, the original code ‘de’ is left as the
   “standard” form Y, and the X code is just a related language. The ‘de’
   code is not reclassified as a macrolanguage.
   2.

   *Reclassifying as macrolanguage. *Another model is to reclassify the
   original code so that it doesn’t really refer to ‘a’ language, but rather
   encompasses many different languages, with perhaps little mutual
   comprehensibility among them. Under that model, ‘de’ changes to a
   macrolanguage that encompasses {gct, gsw, swg, wae, bar, vmf, ksh, ltz,
   pfl, sxu, nds, …} *and* a separate code (eg, /dea/) is added that would
   mean standard German specifically. This is the model that ISO used with
   ‘ar’, for example: ‘ar’ became a macro language code, and ‘arb’ was added
   as the standard form.
   3.

   *New code for macrolanguage.* There is a third possible model: ‘de’
   remains a non-macro language, but a new macro language (eg, /dey/) gets
   added that encompasses{de, gct, gsw, swg, wae, bar, vmf, ksh, ltz, pfl,
   sxu, nds, …}. (I don’t know whether this has ever happened.)

Although well-intentioned, reclassification of macrolanguages has the
by-product of introducing compatibility complications for a large number of
— if not most — implementations, because in practice implementations need
to interact with data using the original code. (There is little consistency
in the reclassifications, as well: why is de different from ar?)

For example, to combat the ambiguity in CLDR we treat the macrolanguage and
the predominant encompassed language as identical, and favor the
macrolanguage form for backwards compatibility. Why do this? That’s because
when an implementation asks for a set of locale data for ‘ar’, it really
doesn’t expects to get a hodgepodge of possible encompassed languages (some
Algerian Arabic, some Egyptian Arabic, some Gulf Arabic, and some Modern
Standard Arabic) in return; it expects Modern Standard Arabic.

While it is too late to change the macrolanguage structure, I would advise
against further “reclassifications”, since they just make it harder for
people, not easier.
Mark


On Fri, Dec 1, 2023 at 3:50 PM <[e-mail]> wrote:

> In our very first meeting of the MA a couple of weeks ago, he started to
> discuss these matters, so please do not expect any solution let alone
> changes in the coming weeks or months.
>
> The goal is to have ONE database, the data of which can be obtained in
> various formats, visually on webpages and as data files.  CSV should
> certainly continue to be supported; I agree.
>
> As I said, the discussions have just started, but as for versioning, a
> system such as a GIT (as in GitHub) was mentioned as an option.  Which
> includes the option that the database *is* a CSV-file.
>
> Best, Sebastian
>
>
>
> *From:* Ietf-languages <[e-mail]> *On Behalf Of *Hugh
> Paterson III
> *Sent:* Friday, 1 December, 2023 19:03
> *To:* Doug Ewell <[e-mail]>
> *Cc:* [e-mail]; IETF Languages Discussion <[e-mail]>
> *Subject:* Re: [Ietf-languages] Latin Sub tags
>
>
>
> As a matter of practicality, it would be helpful for my processes if the
> format of the data files would remain tab delimited or CSV. Sebastian
> mentions a "database format". I can imagine that as an SQL dump or JSON
> file, which are valid and in some cases useful formats. However, as a
> current data consumer of these standards I do prefer the tab and CSV
> formats that have existed for some time. It would be useful if there was a
> single URI to download from. My classic example has been wordpress.
> https://wordpress.org/latest if one follows the included link, "latest"
> will allow curl or wget type resources to always get the latest "version"
> of wordpress. This URL strategy is really helpful in automation contexts
> where content is fetched. Note that even though wordpress is versioned with
> a semantic versioning system for releases, "latest" in the URL is a generic
> pointer to the latest version.
>
>
>
> I am curious how the MA will approach the matter of data versioning. This
> is something that some of my colleagues and I have often desired from
> downloaded data (from various sources) in the BCP-47 space. Versions allow
> us to quickly look at the state of our systems and determine if we need to
> update data.
>
>
>
> - Hugh
>
>
>
> On Fri, Dec 1, 2023 at 1:04 PM Doug Ewell <[e-mail]> wrote:
>
> [e-mail] wrote:
>
> > I am sorry to read this; I was under the impression that core members
> > of this list were aware of the revision of ISO 639 and the setting up
> > of a Maintenance Agency; this process has been ongoing for many
> > years...
>
> I had heard in very broad, general terms that administration of the parts
> of 639 was to be consolidated, at some point in the future.
>
> I had not heard any details about how the process of updating the code
> lists will change, how frequently the changes will occur, how they will be
> announced, whether and how the format of official data files will change,
> etc.
>
> None of this is meant to suggest that these changes are a bad thing. I
> think they will greatly reduce the confusion people have about ISO 639.
>
> > I am not so sure about that the new setting for ISO 639  really means
> > profound changes for BCP 47.  There will still be a new version of the
> > ISO 639 database (file) in regular points in time (I believe this will
> > certainly not become a continuous process, but rather be bundled to
> > one to a few times per year) which will have to formally be adopted by
> > the BCP 47 registry.
>
> Please do keep this list informed as details become available.
>
> --
> Doug Ewell, CC, ALB | Lakewood, CO, US | ewellic.org
>
> _______________________________________________
> Ietf-languages mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/ietf-languages
>