Re: [Ietf-languages] Latin Sub tags

[Sebastian Drude <[e-mail]>]

Thanks a lot, Mark, 

for sharing this.  
It is certainly highly relevant for our discussions in the MA.

Sebastian

-- 

Museu P.E. Goeldi, CCH, Linguistica � ▪  �Av. Perimetral, 1901

Terra Firme, CEP: 66077-530 � ▪ � Belém do Pará – PA  �▪ � Brazil

 <mailto:[e-mail]> [e-mail]  �▪ � [phone]  �▪ � [phone]

 �

From: Ietf-languages <[e-mail]> On Behalf Of Mark Davis ☕️
Sent: Sunday, December 3, 2023 10:46 PM
To: [e-mail]
Cc: Hugh Paterson III <[e-mail]>; Doug Ewell <[e-mail]>; IETF Languages Discussion <[e-mail]>
Subject: Re: [Ietf-languages] Latin Sub tags

 �

> In January 2010, ISO 639-3/RA reclassified the individual language [lav] Latvian as a macrolanguage, encompassing the two new individual languages [lvs] Standard Latvian and [ltg] Latgalian.

If there is no code for a given language X, people will often end up tagging it with a language code Y that does exist where common names for X reference the name for Y (even where there is little mutual comprehensibility between X and Y). But then later on, a code is assigned for X. The question is, what to do about the relationship between X and Y?

(Below I’m following BCP47 in using the ‘shortest form’, thus ‘de’ (ISO 639-1) for ‘deu’ (ISO 639-3), ‘ar’ = ‘ara’, etc.)

1.	No macrolanguage. This model is the one used for German, French, and many other languages. For German, the original code ‘de’ is left as the “standard” form Y, and the X code is just a related language. The ‘de’ code is not reclassified as a macrolanguage.
2.	Reclassifying as macrolanguage. Another model is to reclassify the original code so that it doesn’t really refer to ‘a’ language, but rather encompasses many different languages, with perhaps little mutual comprehensibility among them. Under that model, ‘de’ changes to a macrolanguage that encompasses {gct, gsw, swg, wae, bar, vmf, ksh, ltz, pfl, sxu, nds, …} and a separate code (eg, /dea/) is added that would mean standard German specifically. This is the model that ISO used with ‘ar’, for example: ‘ar’ became a macro language code, and ‘arb’ was added as the standard form.
3.	New code for macrolanguage. There is a third possible model: ‘de’ remains a non-macro language, but a new macro language (eg, /dey/) gets added that encompasses{de, gct, gsw, swg, wae, bar, vmf, ksh, ltz, pfl, sxu, nds, …}. (I don’t know whether this has ever happened.)

Although well-intentioned, reclassification of macrolanguages has the by-product of introducing compatibility complications for a large number of — if not most — implementations, because in practice implementations need to interact with data using the original code. (There is little consistency in the reclassifications, as well: why is de different from ar?)

For example, to combat the ambiguity in CLDR we treat the macrolanguage and the predominant encompassed language as identical, and favor the macrolanguage form for backwards compatibility. Why do this? That’s because when an implementation asks for a set of locale data for ‘ar’, it really doesn’t expects to get a hodgepodge of possible encompassed languages (some Algerian Arabic, some Egyptian Arabic, some Gulf Arabic, and some Modern Standard Arabic) in return; it expects Modern Standard Arabic.

While it is too late to change the macrolanguage structure, I would advise against further “reclassifications”, since they just make it harder for people, not easier.

Mark

 �

 �

On Fri, Dec 1, 2023 at 3:50 PM <[e-mail] <mailto:[e-mail]> > wrote:

In our very first meeting of the MA a couple of weeks ago, he started to discuss these matters, so please do not expect any solution let alone changes in the coming weeks or months. � 

The goal is to have ONE database, the data of which can be obtained in various formats, visually on webpages and as data files. � CSV should certainly continue to be supported; I agree. � 

As I said, the discussions have just started, but as for versioning, a system such as a GIT (as in GitHub) was mentioned as an option. � Which includes the option that the database is a CSV-file.

Best, Sebastian

 �

From: Ietf-languages <[e-mail] <mailto:[e-mail]> > On Behalf Of Hugh Paterson III
Sent: Friday, 1 December, 2023 19:03
To: Doug Ewell <[e-mail] <mailto:[e-mail]> >
Cc: [e-mail] <mailto:[e-mail]> ; IETF Languages Discussion <[e-mail] <mailto:[e-mail]> >
Subject: Re: [Ietf-languages] Latin Sub tags

 �

As a matter of practicality, it would be helpful for my processes if the format of the data files would remain tab delimited or CSV. Sebastian mentions a "database format". I can imagine that as an SQL dump or JSON file, which are valid and in some cases useful formats. However, as a current data consumer of these standards I do prefer �the tab and CSV formats that have existed for some time. It would be useful if there was a single URI to download from. My classic example has been wordpress. �https://wordpress.org/latest if one follows the included link, "latest" will allow curl or wget type resources to always get the latest "version" of wordpress. This URL strategy is really helpful in automation contexts where content is fetched. Note that even though �wordpress is versioned with a semantic versioning system for releases, "latest" in the URL is a generic pointer to the latest version. �

 �

I am curious how the MA will approach the matter of data versioning. This is something that some of my colleagues �and I have often desired from downloaded data (from various sources) in the BCP-47 space. Versions allow us to quickly �look at the state of our systems and determine if we need to update data.

 �

- Hugh

 �

On Fri, Dec 1, 2023 at 1:04 PM Doug Ewell <[e-mail] <mailto:[e-mail]> > wrote:

[e-mail] <mailto:[e-mail]>  wrote:

> I am sorry to read this; I was under the impression that core members
> of this list were aware of the revision of ISO 639 and the setting up
> of a Maintenance Agency; this process has been ongoing for many
> years...

I had heard in very broad, general terms that administration of the parts of 639 was to be consolidated, at some point in the future.

I had not heard any details about how the process of updating the code lists will change, how frequently the changes will occur, how they will be announced, whether and how the format of official data files will change, etc.

None of this is meant to suggest that these changes are a bad thing. I think they will greatly reduce the confusion people have about ISO 639.

> I am not so sure about that the new setting for ISO 639 � really means
> profound changes for BCP 47. � There will still be a new version of the
> ISO 639 database (file) in regular points in time (I believe this will
> certainly not become a continuous process, but rather be bundled to
> one to a few times per year) which will have to formally be adopted by
> the BCP 47 registry.

Please do keep this list informed as details become available.

--
Doug Ewell, CC, ALB | Lakewood, CO, US | ewellic.org <http://ewellic.org> 

_______________________________________________
Ietf-languages mailing list
[e-mail] <mailto:[e-mail]> 
https://www.ietf.org/mailman/listinfo/ietf-languages