Re: [I18ndir] [art] New Version Notification for draft-bray-unichars-06.txt

[Tim Bray <[e-mail]>]

 Thanks. It’s a long weekend in Canada, which may help me recover from my
depression concerning the world’s most popular programming language.

On Oct 6, 2023 at 5:10:11 PM, "Manger, James" <
[e-mail]> wrote:

> % jshell
>
> |  Welcome to JShell -- Version 20.0.2
>
> |  For an introduction type: /help intro
>
>
>
> jshell> import static java.nio.charset.StandardCharsets.*;
>
>
>
> jshell> HexFormat.of().toHexDigits(new String(new char[] {
> 0xDEAD}).codePointAt(0))
>
> $2 ==> "0000dead"
>
>
>
> jshell> HexFormat.of().formatHex(new String(new char[] {
> 0xDEAD}).getBytes(UTF_16BE))
>
> $3 ==> "fffd"
>
>
>
> jshell> HexFormat.of().formatHex(new String(new char[] {
> 0xDEAD}).getBytes(UTF_8))
>
> $4 ==> "3f"
>
>
>
> The Java code snippets above show that:
>
> 2. You can store an unpaired surrogate in a Java string
>
> 3. The easiest way to get a UTF-16 encoding replaces ill-formed code units
> with U+FFFD
>
> 4. The easiest way to get a UTF-8 encoding replaces ill-formed code units
> with U+3F QUESTION MARK
>
>
>
> jshell> var *cb* = CharBuffer.allocate(1).append((char)0xDEAD);
>
> cb ==>
>
>
>
> jshell> Charset.forName("UTF-8").newEncoder().encode(cb.rewind())
>
> |  Exception java.nio.charset.MalformedInputException: Input length = 1
>
> |        at CoderResult.throwException (CoderResult.java:274)
>
> |        at CharsetEncoder.encode (CharsetEncoder.java:820)
>
> |        at (#20:1)
>
>
>
> A UTF-8 encoder in Java can also be configured to signal an error for
> ill-formed code units.
>
>
>
> From CharsetEncoder Javadoc
> <https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/nio/charset/CharsetEncoder.html#:~:text=How%20an%20encoding%20error%20is%20handled%20depends>
>
> How an encoding error is handled depends upon the action requested for
> that type of error, which is described by an instance of the
> CodingErrorAction
> <https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/nio/charset/CodingErrorAction.html> class.
> The possible error actions are to ignore
> <https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/nio/charset/CodingErrorAction.html#IGNORE> the
> erroneous input, report
> <https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/nio/charset/CodingErrorAction.html#REPORT> the
> error to the invoker via the returned CoderResult
> <https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/nio/charset/CoderResult.html> object,
> or replace
> <https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/nio/charset/CodingErrorAction.html#REPLACE> the
> erroneous input with the current value of the replacement byte array. The
> replacement is initially set to the encoder's default replacement, which
> often (but not always) has the initial value { (byte)'?' }; its value may
> be changed via the replaceWith
> <https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/nio/charset/CharsetEncoder.html#replaceWith(byte%5B%5D)>
>  method.
>
>
>
> --
>
> James Manger
>
>
>
>
>
>
>
> * General From: *Rob Sayre <[e-mail]>
> *Date: *Saturday, 7 October 2023 at 10:29 am
> *To: *Tim Bray <[e-mail]>
> *Cc: *Manger, James <[e-mail]>, [e-mail] <
> [e-mail]>, ART Area <[e-mail]>
> *Subject: *Re: [art] New Version Notification for
> draft-bray-unichars-06.txt
>
> [External Email] This email was sent from outside the organisation – be
> cautious, particularly with links and attachments.
>
> On Fri, Oct 6, 2023 at 4:10 PM Tim Bray <[e-mail]> wrote:
>
>
>    1.
>    2. U+FFFD is an obvious choice to replace code units or scalars you
>    don’t want. But Unicode does allow choices. Unicode ch3
>    <https://www.unicode.org/versions/Unicode15.1.0/ch03.pdf> C10 only
>    says “with a marker such as U+FFFD”. Unicode TR36
>    <https://unicode.org/reports/tr36/#Substituting_for_Ill_Formed_Subsequences>
>    says “where U+FFFD is not available, a common alternative is "?"”. Java,
>    for instance, uses “?” is some common circumstances. Unichars does not
>    admit such an option.
>
> Also worth a reference. If you’re writing Java code you should probably do
> what Java does, no?
>
>
>
> Can we write the code here, and see what Java does? I find the other
> points uncontroversial,
>
>
>
> thanks,
>
> Rob
>
>
>
>
>
>
>