Re: draft-ietf-httpbis-header-structure-00, unicode range

[Kari Hurtta <[e-mail]>]

Ilari Liusvaara <[e-mail]>: (Tue Dec 13 19:54:19 2016)
> On Tue, Dec 13, 2016 at 07:33:26PM +0200, Kari Hurtta wrote:
> > 2.  Definition of HTTP Header Common Structure
> > https://tools.ietf.org/html/draft-ietf-httpbis-header-structure-00#section-2
> > 
> > |     unicode_string = * unicode_codepoint
> > |             # XXX: Is there a place to import this from ?
> > |             # Unrestricted unicode, because there is no sane
> > |             # way to restrict or otherwise make unicode "safe".
> > 
> > What is range of unicode_codepoint ?
> > 
> > Next section implies that it is only plane 0 ?
> > ( range 0x0000 - 0xFFFF )
> > 
> > 3.  HTTP/1 Serialization of HTTP Header Common Structure
> > https://tools.ietf.org/html/draft-ietf-httpbis-header-structure-00#section-3
> > 
> > |     h1_unicode_string = DQUOTE *(
> > |                         ( "\" DQUOTE )
> > |                         ( "\" "\" ) /
> > |                         ( "\" "u" 4*HEXDIG ) /
> > |                         0x20-21 /
> > |                         0x23-5B /
> > |                         0x5D-7E /
> > |                         UTF8-2 /
> > |                         UTF8-3 /
> > |                         UTF8-4
> > |                         ) DQUOTE
> > |     # This is UTF8 with HTTP1 unfriendly codepoints
> > |     # (00-1f, 7f) neutered with \uXXXX escapes.
> > 
> > 
> > here seems unicode to limited to plane 0
> > ( 0x0000 - 0xFFFF )
> > 
> > Or is unicode values > 0xFFFF
> > encoded with surrogates  (values 0xd8000 - 0xdffff) ?
> > ( UCS-2 or UTF-16 is used )
> 
> Well, that production lists UTF8-4, which is presumably 4-byte UTF-8
> sequences, and all valid ones are astral plane codepoints.

OK.
 
> Also, if one is to use native unicode (UTF-8), escape sequences are
> only needed for range 0-159. 

    Then that 
	 ( "\" "u" 4*HEXDIG ) /
    is overkill.

>                               Or otherwise you also need escapes for
> astral planes (and I hope the escape system there would be more sane
> than the one JSON has...)

That ( "\" "u" 4*HEXDIG ) produces same than JSON have 
when UTF-16 is used:


https://tools.ietf.org/html/rfc4627#section-2.5

   To escape an extended character that is not in the Basic Multilingual
   Plane, the character is represented as a twelve-character sequence,
   encoding the UTF-16 surrogate pair.  So, for example, a string
   containing only the G clef character (U+1D11E) may be represented as
   "\uD834\uDD1E".

> 
> -Ilari

/ Kari Hurtta