Re: [openpgp] A way to securely define cleartext signature charset

[Jon Callas <[e-mail]>]

> On Sep 11, 2018, at 1:36 AM, Andre Heinecke <[e-mail]> wrote:
> 
> Hi Jon, Neil,
> 
> thanks for your comments!
> 
> On Monday, September 10, 2018 4:53:03 PM CEST Jon Callas wrote:
>>> On Sep 10, 2018, at 11:23 AM, Neil Hunsperger 
> <Neil_Hunsperger=[e-mail]> wrote:
>>> I'll add a data point. Some years back, the PGP Desktop product added an 
> unsigned "Charset" header to its ASCII armor. The result looked like this:
> 
> That would also be an option. I don't prefer it because it would be unsigned 
> but it would already help with usability issues.

[…]

> 
> And that is the problem. E.g. a webmailer in which you paste UTF-8 Text, then 
> the webmailer sees that it can encode that message as latin 1 and sends it as 
> latin 1. Now on the receiving side you have a content-type saying "latin 1" 
> but the message was actually signed in UTF-8. And so you have to "try" 
> multiple charsets if you whish to verify the message (as it could also be 
> signed as latin1).

I think you’re over-thinking this, Andre.

OpenPGP has many things in it from its past, and its past includes the computing environment of 1991, namely DOS, and as a system to post messages on BBSes. There are many things that we’d do differently, if we were starting over. There are many things that are basically layering violations, as well.

For example, the compression stuff there is so that people wouldn’t have to edit something up, then run Zip, and then encrypt that. Another thing, and what is relevant here is the distinction between “Text” and “Binary,” which inherits in some backwards way from FTP.

With RFC 2440, we made a (then) bold step of effectively saying that OpenPGP says that Text is Unicode, coded in UTF-8. By the time of the early 2000s, just about everyone had moved to Unicode but with holdouts, particularly in Japan, where they still liked (and like) Shift-JIS. The amusingly named “ISO Latin-5” is actually Cyrillic, which is not Latin at all, but it was an eight-bit character set for them. I’m sure I am oversimplifying and there are other holdouts as well, because long tails are like that. Nonetheless, every day that tail becomes more tail-like, because Unicode is what just about everyone uses.

> I'm not sure if you say that we should not add standardized way to define the 
> charset for cleartext signatures or that we should?

I’m saying we *have* a standardized way. The charset is Unicode encoded in UTF-8.

And then we have this bag on the side so that the long-tail people can put hints in. Back to Neil’s comments, we found out in the mid-2000s that Unicode wasn’t getting traction in Japan and they were doing all sorts of contortions to deal with crossed character sets. The final compromise within the working group was to put this in so that we could both recognize that the official stance (text is Unicode) and let the dissenters have a better experience.

It’s actually good that the header isn’t signed, because if it were, it would end up causing more issues — is it part of the message or not, to start with. That header is metadata, and it’s metadata that can’t be definitely stated because of layering. It’s good that a component can slide that hint in without having to be integrated with the whole of the crypto system.

> 
> I don't really see the problem of either silly character sets or Latin-5 / JIS 
> messages. As long as It can be converted to the display charset / for passage 
> through the openpgp engine it should be ok.

I concur; the OpenPGP engine should not have to worry about character sets. It’s got too much to worry about as it is.

> 
>> Thus, this section lets an implementation throw its hands up in the air and
>> scream wherever and whenever it wants, while giving a decent way to
>> clearsign Japanese text.
> 
> Yeah, but from a usability standpoint I do not like guessign, screaming and 
> failing if it can be avoided at all :-).

The only time you have to guess is if someone is not using Unicode. That’s the whole point.

	Jon