Re: [EAI] UTF32

["Martin J. Dürst" <[e-mail]>]

On 2015/04/22 22:21, John C Klensin wrote:

> Now you may be thinking about something else.  The Unicode code
> space ranges only from 0 to 0x10FFFF.   A 32bit code space would
> be 0 to 0xFFFFFFFF.  If one believed that the Unicode code space
> were too small and that a full 32 bit space were needed, that
> would be a different matter entirely.  The Unicode folks are
> convinced that more than the current space will never be needed
> to represent all symbols of interest but, at one stage, they
> believed that a 16bit code space would be enough.  I haven't
> studied the mechanisms in some years, but, if a larger code
> space were needed, extensions would be needed to both the UTF-8
> and UTF-16 encoding models to accommodate it while anything that
> used a 32 bit space directly would presumably be fairly
> transparent, at least until one got into various Unicode tables
> and algorithms that assume the smaller code space.   But, again,
> that has little to do with the difference between UTF-16 and
> UTF-32 except as a side effect.

More specifically, at one point, Unicode thought that a 16-bit space 
might be enough (for that time being, at least), while on the other hand 
ISO/IEC JTC1/SC2/WG2, the ones responsible for ISO 10646, thought that 
an architecture with a full 31 bits would be better (the 32nd bit was 
always reserved because nobody wanted to repeat the 8-bit "signed char" 
vs. "unsigned char" mess). UCS-4 and UTF-8 were both designed to 
encompass this 31-bit code space. UTF-8 needed up to 6 bytes for a 
character, in a very straightforward way, according to its original 
design (There is still code out there with traces from this period.). 
UCS-2 was of course limited to 16 bits.

With time passing, it became clearer on both sides that 16 bits wasn't 
enough but 31 bits was overkill. The introduction of UTF-16 created an 
upper limit of 0x10FFFF in one of the Unicode encoding forms. Therefore 
both Unicode and SC2/WG2 agreed on this overall limit. UTF-32 was 
introduced as a version of UCS-4 with an explicit upper codepoint limit 
of 0x10FFFF, and the definition of UTF-8 was changed to only go up to 
four bytes.

In the case of an extraterrestrial invasion by a culture with millions 
of characters or a sudden excessive emoji binge, the limits for UCS-32 
and UTF-8 could be changed again, and some further kludge could be 
introduced in UTF-16. But the chance that we'll get there is very low, 
at least for the moment.

Regards,   Martin.