Re: #428 Accept-Language ordering for identical qvalues

[Julian Reschke <[e-mail]>]

On 2013-01-17 11:17, Roy T. Fielding wrote:
> On Jan 17, 2013, at 1:14 AM, Julian Reschke wrote:
>
>> On 2013-01-17 09:59, Roy T. Fielding wrote:
>>> The change is to improve interoperability when the preferences sent
>>> result in a tie or contain no qvalues.
>>>
>>> http://www.w3.org/International/questions/qa-lang-priorities.en.php
>>>
>>> Firefox and Chrome have an ordered language UI that takes whatever
>>> list the user comes up with and creates q-values to associate with
>>> each language tag after the first.  The languages are always listed
>>> in order of preference.  I've heard that Opera and MSIE do the same
>>
>> But they have different qvalues, so these UAs do *not* rely on ordering.
>
> They order them *and* they send qvalues, because of broken sites like
>
>     http://wiki.nginx.org/AcceptLanguageModule

...so they do not rely on recipients treating the list as ordered.

> and the change I made has no effect on UAs that send qvalues.

Agreed.

> It does, however, improve the lot for users of other user agents
> that either do not send qvalues or allow the user to specify the
> value by hand, either of which can result in same-valued tags.

Well, it's doing that at the cost of making recipients non-compliant 
that implement RFC 2068/2616. So this is a change that would need to be 
listed in the Changes section.

>>> but haven't tested.  Chrome has a bug with the ordering of tags to
>>> match the UI, but that's orthogonal to this issue.
>>>
>>> ...
>>>
>>> Older browsers did not send qvalues. Hence, server implementations
>>> of language negotiation do use the ordering provided as I described
>>> in the change.
>>
>> Some, apparently. But not all. Servers have been written according to the spec, ignoring ordering. If we make ordering significant, these will not interoperate anymore.
>
> In what way do they interoperate now, and in what way does that
> change?  As far as I can tell, the only effect this change has
> is a suggestion that they not respond randomly.

Right now they interoperate as specified by the spec. If we change the 
spec, they do not anymore (or only some of the time).

>>>> I believe this change should be backed out.
>>>
>>> The change has no impact on user agents that send distinct
>>> qvalues.  At most, it would change the interpretation for those
>>> few requests that still rely on ordered language tags, for which
>>> the prior specs had no interpretation at all.
>>
>> Well, it had an interpretation ("same weight").
>>
>> It's good that there are only few requests on relying on this. Do you want to encourage more, potentially breaking servers that ignore ordering?
>
> It isn't possible to break those servers any more than they
> are already broken by responding in a random fashion.

They are not broken. They do what the spec says.

>>> http://forums.thedailywtf.com/forums/t/15895.aspx
>>
>> Yes, I'm aware that this is a FAQ. But we're not starting from scratch here.
>>
>>> What I added is how Apache httpd has implemented it since before
>>> any of the HTTP specs were RFCs, which was compatible with how
>>> CERN httpd implemented it before that (no qvalues at all).
>>> The change has no impact on conformance because language
>>> negotiation is optional and the change is not expressed as a
>>> requirement.  It also resolves an inconsistency with RFC4647.
>>
>> It does have an impact, because servers that previously implemented the optional feature now do not anymore.
>
> A server is not required to obey Accept-Language.

Yes, but a server that *does* implement it according to the *current* 
spec (literally) won't be conforming to the new spec anymore.

>>> There are far more examples on the Web where applications
>>> incorrectly assume the list is ordered
>>
>> That'll be hard to count. :-)
>>
>>>    http://pic.dhe.ibm.com/infocenter/tivihelp/v2r1/index.jsp?topic=%2Fcom.ibm.itame.doc_6.1%2Fam61_webseal_admin134.htm
>>>
>>>    http://www.developershome.com/wap/detection/detection.asp?page=acceptLanguageHeader
>>>
>>> than there are servers that implement language negotiation and
>>> actually want to resolve ties at random.
>>
>> They do not "want" to resolve at random; they do so because they have implemented what the spec says. There's no reason to create an ordered list structure when the spec says that an unordered list is sufficient.
>
> Yes, which is why we are changing the spec so that they can eventually
> improve their implementation (or not).  An unordered list is not
> sufficient because users don't send unordered lists, not even when
> they are hand-crafted.  Everyone assumes they are ordered or they
> send them with correct qvalues.

If people stick to what 2068 and 2616 says, there is no problem.

>>> As Harald said,
>>>
>>>> it seems still to be fairly normal to give a sequence of
>>>> language-ranges in this header without any q= values, and expect the
>>>> result to be deterministic.
>>
>> I'd like so see evidence of that, not hearsay. Browsers do *not* rely on this.
>
> ENOCARE.

Not helpful.

>>> because that's how it works in practice for the vast majority
>>> of systems that implement content negotiation.  The spec should
>>> reflect what is most interoperable for the users.
>>
>> As far as I can tell, what's interoperable is the exact opposite: not relying on ordering, but sending qvalues.
>
> One has nothing to do with the other.  The added text does not
> change the ordering for UAs that send proper, non-identical
> qvalues on language tags, even if they choose not to sort them
> by order (which the latest browsers do for a good reason).

Yes. But it makes recipients that are currently conforming potentially 
non-conforming. Yes, for an optional feature, but still.

It seems there are two logical approaches:

(1) Leave things as they are, but explain that HTTP is not totally 
consistent with RFC4647, and that there may be broken senders that rely 
on ordering, and also recipients that do support this; thus it might be 
a good idea to mirror that behavior.

OR

(2) Change the spec to what it says no, noting that this is a change 
from 2068/2616, and also inconsistent with other Accept header fields 
(or do you want to change those, too???).

Note that introducing an inconsistency here may be a bad idea; either it 
defeats code re-use, or the "new old" semantics may bleed over 
unintentionally into the other header fields.

My preference is (1). I also don't think we have not even rough 
consensus for a change from 2616 here.

Best regards, Julian