Re: [apps-discuss] URI parsing tests: userinfo handling

[Sam Ruby <[e-mail]>]

On 01/02/2015 08:44 AM, Julian Reschke wrote:
> On 2015-01-02 14:31, Sam Ruby wrote:
>>
>> On 01/02/2015 07:19 AM, Julian Reschke wrote:
>>> On 2015-01-02 11:43, Sam Ruby wrote:
>>>> On 01/02/2015 05:01 AM, Julian Reschke wrote:
>>>>> On 2015-01-01 20:08, Sam Ruby wrote:
>>>>>  > ...
>>>>>  > I have evidence that RFC 3986 doesn't match a variety of user agent
>>>>>> behavior.  Agents that aren't limited to browsers, but also to
>>>>>> libraries
>>>>>> that are used by what you would consider "middleware".
>>>>>>
>>>>>> Here is a filtered list of test results that only considers RFC 3986
>>>>>> valid URI references as inputs:
>>>>>>
>>>>>> https://url.spec.whatwg.org/interop/test-results/?filter=valid
>>>>>> ...
>>>>>
>>>>>
>>>>> So I'm now looking at the first valid entry with UA differences:
>>>>>
>>>>> <https://url.spec.whatwg.org/interop/test-results/19b44e58a2?filter=valid>
>>>>>
>>>>> which tests:
>>>>>
>>>>>    http://user:pass@foo:21/bar;par?b#c
>>>>>
>>>>> This one is valid per the RFC 3986 ABNF and it contains a "userinfo"
>>>>> component.
>>>>>
>>>>> RFC 3986:
>>>>>
>>>>> "...Use of the format "user:password" in the userinfo field is
>>>>> deprecated. Applications should not render as clear text any data
>>>>> after
>>>>> the first colon (":") character found within a userinfo subcomponent
>>>>> unless the data after the colon is the empty string (indicating no
>>>>> password). Applications may choose to ignore or reject such data
>>>>> when it
>>>>> is received as part of a reference and should reject the storage of
>>>>> such
>>>>> data in unencrypted form. The passing of authentication information in
>>>>> clear text has proven to be a security risk in almost every case where
>>>>> it has been used...." --
>>>>> <http://greenbytes.de/tech/webdav/rfc3986.html#rfc.section.3.2.1.p.1>
>>>>>
>>>>> RFC 7230:
>>>>>
>>>>> "The URI generic syntax for authority also includes a deprecated
>>>>> userinfo subcomponent ([RFC3986], Section 3.2.1) for including user
>>>>> authentication information in the URI. Some implementations make
>>>>> use of
>>>>> the userinfo component for internal configuration of authentication
>>>>> information, such as within command invocation options, configuration
>>>>> files, or bookmark lists, even though such usage might expose a user
>>>>> identifier or password. A sender MUST NOT generate the userinfo
>>>>> subcomponent (and its "@" delimiter) when an "http" URI reference is
>>>>> generated within a message as a request target or header field value.
>>>>> Before making use of an "http" URI reference received from an
>>>>> untrusted
>>>>> source, a recipient SHOULD parse for userinfo and treat its
>>>>> presence as
>>>>> an error; it is likely being used to obscure the authority for the
>>>>> sake
>>>>> of phishing attacks." --
>>>>> <http://greenbytes.de/tech/webdav/rfc7230.html#rfc.section.2.7.1.p.8>
>>>>>
>>>>> Looking at the test results suggests that
>>>>>
>>>>> a) only the Python implementation is really broken (it probably
>>>>> follows
>>>>> RFC 2396 which treated params as a separate component; Firefox had a
>>>>> similar problem until a few years ago; but that was fixed by me)
>>>>
>>>> FWIW, there was a problem with my script that captured test results, I
>>>> wasn't capturing username and password, and this is now fixed:
>>>>
>>>> https://github.com/webspecs/url/commit/72b286722209771b4a18a53a3493c20e88d95736
>>>>
>>>> But that is likely separate from what you are commenting on, namely the
>>>> problem with an incomplete pathname.
>>>> ...
>>>
>>> FWIW, it could also be an incorrect use of Python's urllib.parse
>>> (<https://docs.python.org/3.0/library/urllib.parse.html>) which is
>>> documented to extract the last path segment's parameter as a separate
>>> component. So test code will need to properly reconstruct the path with
>>> that information.
>>
>> I'm inclined to think otherwise as Python's implementation treats the
>> ';' as a delimiter and not a part of the path.  As an example, the
>> following two calls produce identical results:
>>
>> urlparse('http://foo/a/b')
>> urlparse('http://foo/a/b;')
>
> Actually I was wrong; special handling of parameters dates back to RFC
> 1808 and was removed in RFC 2396, see
> <https://tools.ietf.org/html/rfc1808#section-2.1>:
>
>> <scheme>://<net_loc>/<path>;<params>?<query>#<fragment>
>
> So this is what these implementations try to do.
>
> To reconstruct the path, given that API, you'll need to re-append the
> params. And yes, the problem with Python's implementation probably is
> that you can't distinguish an empty params component from absence of
> that component. This should be raised as a bug report against the Python
> libraries.
>
> (My point being that this -- when properly used -- affects only the edge
> case of a path ending in a ";")

Given that it isn't documented as such, I would consider that to be more 
of a workaround than proper use, but it isn't worth arguing over:

https://github.com/webspecs/url/commit/020175339c39166201f2a30109ca0b48371b455c

Thanks for opening the issue against Python!

> Best regards, Julian

- Sam Ruby