[Ltru] Re: [ExtV] Glottolog source?

[Doug Ewell <[e-mail]>]

On Monday, July 22, 2024 at 6:51, Sebastian Drude wrote:

> For dialects, I believe the basic resource is languoid.csv,
> considering all the lines which have "False" in the column
> "bookkeeping" and "Dialect" in the column "level".  The Glottocode
> sits in the first column.
> [...]
> Retired entries are maintained (with the value in the "bookkeeping"
> column set to "True").

Unfortunately, this turns out to be true only for "language" entries, not for "family" or (crucially) "dialect" entries.

From the 2021 article by Forkel and Hammarström, which Sebastian forwarded to us as a PDF on June 27 (privately, not on this list):

"If a language-level languoid was completely erroneous, it is moved to the Bookkeeping category. [...] In contrast, the family-level and dialect-level glottocodes are not 'protected' and may be removed from the inventory between releases. Since they do not necessarily reflect a real-world entity like an L1 language, it cannot systematically be explained what 'happened' to them, e.g., if they never really existed."

Regardless of whether Glottolog's curators or contributors believe that a previously coded dialect may not actually exist, the fact remains that someone, somewhere, at some point, may have used its Glottocode conformantly to represent it. That is why Section 3.7 of RFC 5646 states clearly with regard to extensions:

"The specification MUST be stable.  That is, extension subtags, once defined by a specification, MUST NOT be retracted or change in meaning in any substantial way."

This requirement exists for a very sound reason: It is absolutely imperative that BCP 47 tags, including those that use extensions, must not lose their meaning because one of its components simply vanishes from the external standard which once defined it. That was one of the key reasons for replacing RFC 3066 in the first place -- to prevent tags that depended directly on ISO 639 and 3166 from losing their meaning when code elements were removed from those standards with no trace.

I had actually suspected, before reading the article, that some Glottocodes might disappear from the list, and have confirmed that, in every version of Glottolog since 3.1 -- when the format of 'languoid.csv'  was changed to more or less its present form -- at least one dialect code marked "False" (i.e. not Bookkeeping) and "dialect" has been quietly removed from the following version. In the table below:

column 1 = last version in which the code appeared
column 2 = following version, in which it was permanently removed
column 3 = Glottocode
column 4 = description

3.1	3.2	bika1249	Bika
3.2	3.3	aiwa1238	Aiwanat
3.3	3.4	bula1258	Bulalakawnon
3.4	4.0	kara1463	Karapapakh
4.0	4.1	awun1244	Awuna
4.1	4.2	aizu1242	Aizuare
4.2	4.3	belu1238	Beludji
4.3	4.4	bedu1242	Beduanda
4.4	4.5	akit1239	Akit
4.5	4.6	serb1266	Serbo-Bosnian Vlax
4.6	4.7	ante1238	Antekerrepenh
4.7	4.8	abaw1238	Abawa
4.8	5.0	bara1408	Baraba Tatar

Most of these codes begin with 'a' or 'b' for a reason: there are actually several codes in each release that are erased in this way, and I simply chose the one for this table that appeared first alphabetically.

The article by Forkel and Hammarström continues:

"However, some tracking possibilities are always guaranteed because both families and dialects are linked to language-level languoids. [... ] if a dialect-level glottocode disappears, it is possible to check which language-level languoid it pertained to and to check which dialects are now associated with it."

This also is not true, at least when considering the reasonably sized file on the Glottolog site. Each of the retired codes listed above was completely removed, with no direct or indirect reference in the following version. Thus it is not possible to check which language-level languoid it pertained to, unless one consults the previous, superseded version of Glottolog.

It is possible that these deleted codes are carried over into the 'superseded.csv' file which is included in 'glottolog-X.X.zip', a comprehensive zip file which also includes the entire Glottolog tree structure (resulting, BTW, in path names which exceed the maximum length supported by many Windows tools) and which totals 99 megabytes as of version 5.0. I would hope that users of Extension V would not be required to download a 99-megabyte file to get the code list needed to support the extension. (Then again, I also hoped that users of Extensions T and U would not have to download the 30-megabyte 'cldr-common-X.X.zip' file to get the code lists for those extensions, but that argument was lost.)

Given the current Glottolog policy of deleting Glottocodes that refer to dialects, rather than marking them "obsolete" or "retired" or "bookkeeping" or some such, it seems the only way to use Glottolog conformantly in Extension V is for us to create our own registry, using codes taken from 'languoids.csv' but with an added policy of preserving and identifying retired codes. This is essentially what we had to do with ISO 639 and 3166, back in 2005.

This would be a large registry; version 5.0 of 'languoids.csv' lists 13,507 dialect codes, which is about twice the number of languages in ISO 639-3. But at least the deleted Glottocodes from versions prior to 5.0 would not need to be included, because they would not have been valid in Extension V. The grandfathering process, if you will, would only have to begin once the extension is approved.

Without a mechanism like this, or unless Harald and Robert change their policy so that dialect codes are guaranteed to be preserved, I do not see any way we can use Glottolog for this extension and stay conformant with BCP 47. Perhaps Sebastian may be able to persuade them.

--
Doug Ewell, CC, ALB | Lakewood, CO, US | ewellic.org