Re: [Ltru] Consensus call: extlang

[Peter Constable <[e-mail]>]

> From: [e-mail] [mailto:[e-mail]] On Behalf Of
> Broome, Karen

> Of course, if you had actually asked for Mandarin using a precise code
> for Mandarin (either "cmn" or "zh-cmn"), that list doesn't look so ugly,
> does it?
>
> But you didn't ask for Mandarin, you asked for Chinese...

I completely agree. I also note some validity in Mark's concern: there are a lot of users that have been and are still using zh rather than cmn/zh-cmn, and probably a large proportion of them are expecting Mandarin content. But as your following comments note, there's no silver bullet for dealing with that.


> This is where I think we are prioritizing generalized fallback over
> precision in identification -- even if what we are precisely
> identifying is intentionally vague. I lose the ability to identify
> "some kind of Chinese, I don't know what kind" if zh is thought to be
> synonymous with Mandarin. (A real need given the history of these tags.)

I think there's general consensus here that zh is *not* thought to be synonymous with Mandarin; there's just a high-ish degree of statistical correlation in existing usage.


> I also will not know what the user intended if they use the tag "zh".

At best, we can only make a statistical inference: based on usage to date, Mandarin is most likely meant.


> When it becomes possible to tag more specifically, we won't be able to
> know whether the decision to use the more general tag ("zh") was a
> conscious choice or not.

In some closed systems, it may be possible to know, but in an open system involving interchange with other parties in other spheres of control, no.


> zh is a bad tag; its semantics have always been muddy. We are going to
> rather great lengths to avoid deprecating it.

Deprecation for something that has been so widely used is a big step, and it may take some time for people to buy into that. But you are right in saying it is problematic: if there are other Chinese choices besides Mandarin, zh just leaves people forced to make assumptions or guess that may not be valid.

I'd point out that all these things are true whether we use extlang or not, and that extlang is helpful in relation to that problem only to the extent that it can help processes take actions that will match user expectations better.

I think initially I thought it might on the basis that anyone asking for zh-xxx might, by right-truncation fallback, still get matches with existing zh content. I was assuming that would generally be a useful response from the user perspective, and while it probably is for Chinese it isn't in the general case, across all macrolanguages -- if for no other reason, because there isn't a bunch of existing content tagged aym or bal or bik etc. that people want to continue getting matches for.

Also, I was thinking of filtering (in RFC4747 terms), and for filtering I got it backwards: right-truncation fallback allows a request for zh to match content tagged zh-xxx, and in that case it *isn't* helpful in the general case: if someone is making a request using a macrolanguage ID, they're still looking for some particular variety, and in most cases returning *any* encompassed variety probably isn't helpful.

Rather than filtering, it's *lookup* for which a request for zh-cmn would match existing zh content. But I think it's fair to say that lookup is most typically used for UI resource matching or similar scenarios in which the options for available content are likely to be fairly limited and controlled, with users being presented with the set of specific options available. In that kind of controlled environment, fallback from (zh-)cmn to zh is less likely to be needed.



Peter
_______________________________________________
Ltru mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/ltru