Re: [EAI] Unicode vs. UTF-8 / Encoding vs. Representation

[John C Klensin <[e-mail]>]

--On Tuesday, December 28, 2010 3:23 PM +0900 Yangwoo Ko
<[e-mail]> wrote:

> On Tue, Dec 28, 2010 at 1:17 PM, John C Klensin
> <[e-mail]> wrote:
>>>  If you have
>>> alternative labeling, that would be great to consider.
>> 
>> I hope someone in the WG will have a good suggestion.   If
>> not, we may be back to "ASCII" versus "non-ASCII" as the
>> disjoint pair, "Unicode"  to describe the international CCS
>> (inclusive of ASCII), and "UTF-8" to describe the particular
>> encoding that is used for this protocol and recommended by
>> RFC 2277 (a recommendation that is reinforced by
>> draft-iab-idn-encoding and other more recent work).
> 
> I hope so too. But I don't expect that we are going to have a
> nice answer to this soon enough. If it still significantly
> matters, will it be useful to list up the cases of possible
> confusions when current terminogies are used? With the list,
> we may be able to add some notes to one of our documents
> (possibly framework doc)  to address the known possible
> confusions.

If we try to push this definitional issue very far, it is far
more complicated than has been suggested so far.  For the work
of the EAI WG, I suggest that there are two practical
possibilities:

	(1) The current terminology is not precisely correct
	given other, Standard, usage but, with small
	adjustments, can be used in the documents with some
	explicit local definitions.
	
	(2) In the tradition of John Tukey and others, we invent
	entirely new language that is precisely-defined locally.
	As Andrew points out, that is precisely what IDNA2008
	ended up having to do after struggling with a variation
	on this problem in the DNS space.

Despite Andrew's misgivings, the definitional problem here is
not nearly as hard as that for IDNA.  I think either of the
above would be possible without significantly delaying the WG.
However, the first would be a lot safer in terms of the risk of
introducing new errors as terminology is redefined and then
applied (a problem we certainly got into with IDNA).

Dave suggested in a later note...

>...
> The change in our reference is to stop using ASCII to refer to
> an encoding.  For Internet ASCII, the formal definition
> usually is "7-bit ASCII in an 8-bit field, with the high bit
> off", but "7-bit" should suffice...

Unfortunately, this doesn't work either if one aspires to the
kind of precision for which you seem to be asking.  First of
all, the ANSI definition of ASCII contains both a character
repertoire and an encoding and is, if I recall, fairly explicit
about that.   ANSI also didn't define anything about ASCII in
longer data fields than 7 bits, so the encoding itself is
two-levels: (i) the ASCII repertoire specified as a seven-bit
encoding and (ii) the embedding of [intrinsically 7-bit] ASCII
into an 8-bit field.

While there has never been such a thing as "8-bit ASCII" from a
standards point of view, that term, or "ASCII-8", have often
been used both to describe embedding of ASCII into octets _and_
as the name of an extended, ASCII-based, character repertoire
and encoding that is essentially Latin-1 or ISO 8859-1.   While
the ASCII repertoire is a proper subset of the Unicode
repertoire and (i) the ASCII code points have the same integer
values as the corresponding Unicode code points if the left
octet is stripped from the latter and (ii) when Unicode is
encoded in UTF-8, the octets corresponding to the ASCII
repertoire have the same values as the ASCII code points
embedded in octets with a leading zero bit, there are other CCSs
for which ASCII (both the repertoire and the leading-zero
imbedding) are proper subsets --including the entire ISO 8859
family.   All of the character repertoires of the 8859 standards
are subsets of the Unicode repertoires, but only 8859-1 bears an
obvious relationship to the Unicode code point assignments (and,
in UTF-8, the top half of the 8859-1 repertoire turns into two
octets and the relationship to the original encodings disappears
entirely.

We have usually used "ASCII [embedded] in an 8-bit field, with
the high bit off" on the Internet (indeed, that is part of the
problem that RFC 20 tried to solve), but we have had ASCII
floating around in 7-bit form (not stored in octets) and there
is have been two other octet embeddings that puts the spare bit
somewhere in the middle (one of them as a zero and one as a
parity bit).  There are also 7-bit encodings of other things
floating around, including the encoding described in RFC 2152
(and its predecessor 1642) and the 1642 modification described
in IMAP4rev1 [RFC 2060], a modification not incorporated in the
1642->2152 update.

So, no, "7-bit" doesn't do it either.

Dave's note suggest that a general terminology solution is
needed, a view with which I agree.  But, just as the EAI WG has
been urged to avoid doing work that extends beyond the narrow
scope of what is specifically needed to email address and header
internationalization, an effort to sort out character set
repertoire, definition, encoding, and embedding issues in a
general and appropriate way should not be imposed on the EAI WG.
It might be appropriate to do so by reopening and revising RFC
3536 (which I note uses some terminology that is now considered
obsolete enough to have caused loud protests when used in other
documents) or by initiating a separate effort, but it would be,
IMO, unreasonable and unfair to delay EAI progress for that work
to be initiated and completed.  As Andrew said (and the above
explanation should reinforce), "introducing and rigorously
defining new terms is hard".

    john