Re: [I18ndir] [art] Fwd: New Version Notification for draft-bray-unichars-06.txt

["Martin J. Dürst" <[e-mail]>]

Hello Tim, others,

On 2023-10-02 02:31, Tim Bray wrote:
> On Sep 30, 2023 at 6:53:28 PM, "Manger, James" <
> [e-mail]> wrote:

>> Explaining the 1,081,344 size and the U+D800-U+DFFF gap would be
>> interesting.
>>
> 
> Yes! That history is mystery to me.

Well, at the start, Unicode was pure 16-bit only, and ISO 10646 was 
officially all 32 bits. They had different encoding principles and 
character allocations initially (that was Unicode 1.0 and some ISO 10646 
draft), but users told both sides that having two different universal 
character standards wasn't really what they wanted (one of the few 
examples where https://xkcd.com/927/ didn't hold :-).

So the encoding principles (e.g. precomposed (ISO) vs. decomposed 
(Unicode), character repertoires, and code point assignments got aligned 
(*), but the 16-bit/32-bit difference stayed.

(*) Well, for precomposed vs. decomposed, it was a compromise (let's 
tolerate both), with NFD to define the correspondence and later NFC for 
a more practical "default representation".

The fact that Unicode tried to keep things within 16 bits was actually 
in many ways beneficial; it forced careful allocation of characters 
where an open 32-bit space could have lead to much more wasteful 
allocations. (Korean Hangul being the most notable exception; with 
something as human as human writing and its encoding, there are always 
exceptions.)

At that point, implementations started to show up, e.g. Windows NT and 
Java and the like, and they used Unicode, i.e. "16-bit characters".

A bit later, it became clear that a 16-bit space was not enough. But it 
also became clear that a 32-bit space was way too much. So people were 
looking for ways to carry around a space somewhat wider than 16 bits, 
but still encodeable in 16-bit code units. The end of the 16-bit space 
had already been taken (among else by the special meanings for U+FFFF 
and U+FFFE), but some contiguous spaces were still available.

I assume that after playing around with various ideas (but I have never 
heard about these), people came up with the current solution: Reserve 
two contiguous blocks of 16-bit code units (now called the high and low 
surrogates) of 1024 code units each to encode a total of 2^20 additional 
code points and call the result UTF-16.



> Also the nonchars in the Arabic-extended region.

Do you mean U+FDDD..FDEF in the Arabic Presentation Forms-A block? If 
not, please say what else.

As the description in the code chart says, these are for 
process-internal use. Let's say you want to encode some internal 
formatting information in an application, or want to implement a clever 
text searching algorithm that only works with some special code points 
that are not used for actual characters. Then you can use these, if you 
make sure they never leak.

I'm not sure exactly why these were introduced, but my guess is that 
they were added with the Object Replacement Character (U+FFFC), which 
was an example of such an "application-internal" use which, as far as I 
remember, was discovered when Microsoft objected to encode something in 
that position. (To be fair, Microsoft was not the only implementer who 
used the trick of hanging off in-line images and the like off a special 
character.)

For more details, please see the Unicode/L2 documents linked from 
https://en.wikipedia.org/wiki/Arabic_Presentation_Forms-A.

Regards,    Martin.