Re: [ietf-types] The application/www-form-urlencoded format

["Stephen D. Williams" <[e-mail]>]

  On 9/26/10 2:22 AM, Anne van Kesteren wrote:
> On Sat, 25 Sep 2010 23:14:39 +0200, Bjoern Hoehrmann <[e-mail]> wrote:
>>   http://tools.ietf.org/html/draft-hoehrmann-urlencoded -- the draft de-
>> scribes the application/www-form-urlencoded format, a variant of the
>> application/x-www-form-urlencoded format first described in RFC 1866.

What the character encoding errors are at the end of section 5 needs to be explained, at least for 2 cases.
>
> I think it is unfortunate it still allows encoding in various ways. So while things could be more readable as you pointed out in 
> the past user agents are still allowed to obscure most everything.

It should be the case that only what has to be encoded is encoded.  This is required for any signature canonicalization anyway.  It 
could be the case that a more restricted situation exists, although I don't see it for form submission.
> ...
>>
>> The draft probably still has some rough edges in the prose but the
>> format is not going to change. I believe it addresses the feedback I
>> got since the first draft published four years ago; public feedback at
>> http://lists.w3.org/Archives/Public/www-archive/2006Sep/thread.html#msg30
>
> The bug about + seems to be still be there. Escapes are first decoded and then + is replaced with U+0020. Also 
> application/x-www-form-urlencoded is on its way of being standardized as part of HTML5 now.

I was just debugging an Oauth implementation (customized RFC 5849) that has this kind of mistake, plus they used the older set of 
unreserved characters rather than those defined in RFC 3986:
unreserved = ALPHA / DIGIT / "-" / "." / "_" / "~"

Some specs allowed more:
RFC 2396: Section 2.3:
unreserved = alphanum | mark
mark = "-" | "_" | "." | "!" | "~" | "*" | "'" | "(" | ")"

Java's urlencoder seems broken as it is almost RFC 3986 but with '*' instead of '~':
http://download.oracle.com/javase/1.4.2/docs/api/java/net/URLEncoder.html
# The special characters "|.|", "|-|", "|*|", and "|_|" remain the same.
# The space character "| |" is converted into a plus sign "|+|".

For canonicalization purposes, RFC 5849 says in sction 3.6:
This method is different from the encoding scheme used by the
"application/x-www-form-urlencoded" content-type (for example, it
encodes space characters as "%20" and not using the "+" character).
It MAY be different from the percent-encoding functions provided by
web-development frameworks (e.g., encode different characters, use
lowercase hexadecimal characters).

They use uppercase hex and ' '->'%20 rather than the option of ' '->'+'.

If ' '->'+' is to be used, it has to be done after percent encoding so that '+' gets percent encoded before ' '->'+'. At this point 
however, most people can read "%20" as space almost as fast as '+'.  On the other hand, '+' is more compact and is still more 
readable.  A spec should deterministically choose one or the other so that signatures can be done without further agreement.  
Signing a form is a common need.

While these specs have different purposes, it is often the case (as with OAuth), that key/value pairs can show up in HTTP headers, 
POST bodies that may be www-form-urlencoded or something like XML, in the URL as a query string, along with being canonicalized for 
HMAC signature.  Unifying the operations reduces confusion and coding.

>
>
>>       Note: The media type does not have a 'charset' parameter, it
>>       is incorrect specify one and to associate any significance to
>>       it if specified. The character encoding is always UTF-8. The
>>       Unicode encoding form signature is not supported; a leading
>>       U+FEFF character will be considered part of a <name>.
>
> Most other such formats ignore a leading U+FEFF.

U+FEFF, the BOM, is only needed where the encoding can be something other than UTF-8.  If there is no choice, it isn't needed and 
probably shouldn't be present unless the situation might change.  If UTF-16 can be used, then it is needed.

>
>
>> [1] The regular expression to match both names is slightly simpler if
>>     they only differ in the "x-", and it seems fitting to standardize
>>     an "x-" type by removing the "x-", but if there is a good argument
>>     why using similar names is a bad idea, I am also quite open to name
>>     it application/name-value-pairs or something like that instead.
>
> Well, application/x-www-form-urlencoded is not going away anytime soon. It is what you get for <form> by default. That some 
> consider it non-standard is just semantics. Having said that I do not mind dropping the x- for an improved version of that format.
>
>

sdw

-- 
Stephen D. Williams [e-mail] [e-mail] LinkedIn: http://sdw.st/in
V:650-450-UNIX (8649) V:866.SDW.UNIX V:703.371.9362 F:703.995.0407
AIM:sdw Skype:StephenDWilliams Yahoo:sdwlignet Resume: http://sdw.st/gres
Personal: http://sdw.st facebook.com/sdwlig twitter.com/scienteer