Re: [I18ndir] [art] New Version Notification for draft-bray-unichars-06.txt

["Manger, James" <[e-mail]>]

% jshell

|  Welcome to JShell -- Version 20.0.2

|  For an introduction type: /help intro



jshell> import static java.nio.charset.StandardCharsets.*;



jshell> HexFormat.of().toHexDigits(new String(new char[] { 0xDEAD}).codePointAt(0))

$2 ==> "0000dead"



jshell> HexFormat.of().formatHex(new String(new char[] { 0xDEAD}).getBytes(UTF_16BE))

$3 ==> "fffd"



jshell> HexFormat.of().formatHex(new String(new char[] { 0xDEAD}).getBytes(UTF_8))

$4 ==> "3f"

The Java code snippets above show that:
2. You can store an unpaired surrogate in a Java string
3. The easiest way to get a UTF-16 encoding replaces ill-formed code units with U+FFFD
4. The easiest way to get a UTF-8 encoding replaces ill-formed code units with U+3F QUESTION MARK


jshell> var cb = CharBuffer.allocate(1).append((char)0xDEAD);

cb ==>



jshell> Charset.forName("UTF-8").newEncoder().encode(cb.rewind())

|  Exception java.nio.charset.MalformedInputException: Input length = 1

|        at CoderResult.throwException (CoderResult.java:274)

|        at CharsetEncoder.encode (CharsetEncoder.java:820)

|        at (#20:1)

A UTF-8 encoder in Java can also be configured to signal an error for ill-formed code units.

From CharsetEncoder Javadoc<https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/nio/charset/CharsetEncoder.html#:~:text=How%20an%20encoding%20error%20is%20handled%20depends>
How an encoding error is handled depends upon the action requested for that type of error, which is described by an instance of the CodingErrorAction<https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/nio/charset/CodingErrorAction.html> class. The possible error actions are to ignore<https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/nio/charset/CodingErrorAction.html#IGNORE> the erroneous input, report<https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/nio/charset/CodingErrorAction.html#REPORT> the error to the invoker via the returned CoderResult<https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/nio/charset/CoderResult.html> object, or replace<https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/nio/charset/CodingErrorAction.html#REPLACE> the erroneous input with the current value of the replacement byte array. The replacement is initially set to the encoder's default replacement, which often (but not always) has the initial value { (byte)'?' }; its value may be changed via the replaceWith<https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/nio/charset/CharsetEncoder.html#replaceWith(byte%5B%5D)> method.

--
James Manger




General

From: Rob Sayre <[e-mail]>
Date: Saturday, 7 October 2023 at 10:29 am
To: Tim Bray <[e-mail]>
Cc: Manger, James <[e-mail]>, [e-mail] <[e-mail]>, ART Area <[e-mail]>
Subject: Re: [art] New Version Notification for draft-bray-unichars-06.txt
[External Email] This email was sent from outside the organisation – be cautious, particularly with links and attachments.
On Fri, Oct 6, 2023 at 4:10 PM Tim Bray <[e-mail]<mailto:[e-mail]>> wrote:

  1.
  2.  U+FFFD is an obvious choice to replace code units or scalars you don’t want. But Unicode does allow choices. Unicode ch3<https://www.unicode.org/versions/Unicode15.1.0/ch03.pdf> C10 only says “with a marker such as U+FFFD”. Unicode TR36<https://unicode.org/reports/tr36/#Substituting_for_Ill_Formed_Subsequences> says “where U+FFFD is not available, a common alternative is "?"”. Java, for instance, uses “?” is some common circumstances. Unichars does not admit such an option.
Also worth a reference. If you’re writing Java code you should probably do what Java does, no?

Can we write the code here, and see what Java does? I find the other points uncontroversial,

thanks,
Rob