Re: [I18ndir] [art] Modern Network Unicode

[Carsten Bormann <[e-mail]>]

This is a great discussion.

To me, it seems to converge on the following.

(1) Sending sane data is the job of the data originator.  

(2) Do not include gratuitous normalization steps in your processing, once the data have been originated in a sane form.

(2a) If you broke it, you fix it (as far as possible): If your processing steps did involve gratuitous normalization, you have to renormalize to NFC before sending.

Here, “sane” is defined as:

(0) Data SHOULD be originated in NFC, unless that would be inappropriate for the specific script, in which case the community consensus rules for the script govern.

For Latin script, this happens to collapse to what 5198 says.

This set of rules places the onus on the place where the data is generated, which is usually the place that knows most about the specific script and about the intent of the originator.  If you know that place isn’t doing its job, add the rule:

(1a) If the data originator does not do (0), the software placing the data on the network may need to sanitize (normalize towards sane).

1a is similar to 2a in that it doesn’t create perfect results, so both SHOULD be avoided — there is no way to, after the fact, perfectly sanitize data that weren’t originated sane or that were gratuitously normalized on the way.

With these definitions, MNU can direct towards: 
(A) Senders: send sane data
(B) Recipients: break as little as reasonable when data received isn’t sane
(C) B is not a valid excuse not to do A, and specifically: recipients are not expected to clean up after senders (because there is no correct way to do that).

(Rule C is the often forgotten third rule of the Postel principle.
It also means that an entity that is a recipient of MNU and then sends the data on as MNU has no need to gratuitously normalize, but it does not entirely get rid of rule 1a for recipients of data from places known not to be sane.)

Grüße, Carsten