Re: [Ltru] Mail regarding draft-ietf-ltru-4646bis and draft-ietf-ltru-matching

[John Cowan <[e-mail]>]

Got it. Yes, with extended filtering, you need to canonicalize toward
extlangs.  I thought you were talking about simple equality matching after
canonicalization.

Note that almost all macrolanguages do not have extlang tags; the languages
can be identified by the macrolanguage or by the individual language tag,
but not both.  Indeed, only ar, kok, lv, ms, sw, uz, zh have extlang tags
subordinated to them; sgn does too but is not a macrolanguage.




On Thu, Aug 29, 2019 at 9:14 PM Florian Rivoal <[e-mail]> wrote:

>
>
> > On Aug 30, 2019, at 10:08, Florian Rivoal <[e-mail]> wrote:
> >
> >
> >
> >> On Aug 29, 2019, at 22:44, John Cowan <[e-mail]> wrote:
> >>
> >>
> >>
> >> On Thu, Aug 29, 2019 at 5:00 AM Florian Rivoal <[e-mail]>
> wrote:
> >>
> >> In case that influences what you have to say, note that what I intend
> to do is not to store the canonicalized-to-extlang form anywhere. It would
> only be for internal processing: when performing an extended filtering
> operation, where it is unknown whether the ranges and tags are in extlang
> form or not, canonicalize both to extlang form do the extended filtering
> operation on that.
> >>
> >> In that case you can equally canonicalize away from the extlang form as
> toward it.  I recommend that.
> >
> > Can you?
> >
> > Let's say you want to match (using extended filtering) the zh range
> against documents that may contain the zh-yue or yue tags (and possibly
> other zh-cmn, zh-hakka, zh, zh-HK…). This could be something a typesetter
> wants to do to use a particular font and set of line breaking rules for any
> chunk of Chinese (in the broad sense) text.
> >
> > If we canonicalize to extlang form:
> >  zh -> zh
> >  zh-yue -> zh-yue
> >  yue -> zh-yue
> > Therefore, the zh range will match both the documents that contained
> zh-yue or yue. This is what I want.
> >
> > If we canonicalize away from extlang form:
> >  zh -> zh
> >  zh-yue -> yue
> >  yue -> yue
> > Therefore, the zh range will match neither documents that contained
> zh-yue nor yue. This is not what I want, and is worse than not
> canonicalizing at all.
> >
> > So it seems to me that no, we cannot canonicalize away from the extlang
> form and get the same results.
> >
> > If the extended filtering operation did something smart with
> macrolanguages, then I wouldn't need canonicalization at all, but it
> doesn't, so I feel I need to canonicalize, and as described above, only
> canonicalization to extlang actually seems to help.
> >
> > Am I missing something?
> >
> > —Florian
>
> Sorry for not including that in the previous message, but to give another
> example, if I want to use the no range to match any of: no, no-bok, no-nyn,
> nb, or nn, canonicalization to extlang form works, and canonicalization
> away from it doesn't.
>
> —Florian