Re: [jose] Canonical JSON form

Bret Jordan <jordan.ietf@gmail.com> Mon, 22 October 2018 00:16 UTC

Return-Path: <jordan.ietf@gmail.com>
X-Original-To: jose@ietfa.amsl.com
Delivered-To: jose@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 25F85130DFB for <jose@ietfa.amsl.com>; Sun, 21 Oct 2018 17:16:22 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -0.998
X-Spam-Level:
X-Spam-Status: No, score=-0.998 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, FREEMAIL_REPLY=1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id E3peAX-1Mn55 for <jose@ietfa.amsl.com>; Sun, 21 Oct 2018 17:16:19 -0700 (PDT)
Received: from mail-yb1-xb2e.google.com (mail-yb1-xb2e.google.com [IPv6:2607:f8b0:4864:20::b2e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id B2F4A130DF9 for <jose@ietf.org>; Sun, 21 Oct 2018 17:16:19 -0700 (PDT)
Received: by mail-yb1-xb2e.google.com with SMTP id o204-v6so1443934yba.9 for <jose@ietf.org>; Sun, 21 Oct 2018 17:16:19 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:mime-version:subject:date:references:to:in-reply-to:message-id; bh=QDIcZ9JCQt+PK1LifnFcxD+1VbS7J6qkzrYJuca8TW8=; b=N0nrg1yZJg32SAlqTCADPto/XZ9mVGT1EaHrHoNUqoIVmk4Fjpmgk4XgM7/EQgKGbu +EspOPkc6a8m1leutfqV5GQ90ynYIHk1RVR91/eJDebyWLPO9GSkN2qkzx89XoiZ/ci+ pbOMvfEkVXM219CQa7B8CoOLJilMoYsrc467qsSTikeNYhwkm5xC1n2MA1YRSXoex0dx Ga0kXYjJykcR9Ged9VQr4d+OYeWEZd7AUhNK1X+VDngU1eiVzEbAY60ezUS62gGwFVK9 m9R1iT1CVWhILFIjd7xfrZ9Y80Cd2bbSVbFXrk8jzx4OGxzxFHTm0GFXjBM+prm4DBu6 Bp/A==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:mime-version:subject:date:references:to :in-reply-to:message-id; bh=QDIcZ9JCQt+PK1LifnFcxD+1VbS7J6qkzrYJuca8TW8=; b=qEDI8wMfDEGhtLXcRkKeYtMbnAoNrYcn9YsavCO4D+3NJ9eORXO7w9DRrxd8diBV22 E1I1wj726X87fd+1au7brzt/6dpl/+CAXweQP5CFuXPuy1XgJRpOU8ASF/byPXKcMmYK US3OT/ZVq7yRX4z1Bjo+TZ+eMIJvP1pEBH/iFzfuxzVv3C8r4L3/A7R5ICqWhklzO6S2 XCtDHt7yteMH+C2rnSTmFnMwiywpDb2YUNm0URVq9ymf6DQVG7mBIdwF0sV0r6Fv/hk1 7uZ2HOwYvpm+T8gY4znhFDMTNtMuoOI6apP/0lFUzieIgxUOPRnE+PpeBlZFJsDxt+3g 8tXg==
X-Gm-Message-State: ABuFfojCZLwN3R9MKo/NoEVjmVNS6wCKEY9CWOv23AUxz4LckRQSCJgs Aex344RJUZvnM1TmLespK1ttP2fU
X-Google-Smtp-Source: ACcGV61Ft/316bhGr2NaUzq6v+17U3KgGkwXL0tjsYYb2ckXB4i38hf5gKXZ1+VjOTkLvE8s4Q/PJQ==
X-Received: by 2002:a25:c4c5:: with SMTP id u188-v6mr30393094ybf.424.1540167378855; Sun, 21 Oct 2018 17:16:18 -0700 (PDT)
Received: from ?IPv6:2605:a601:3260:266:5803:ffca:df13:4192? ([2605:a601:3260:266:5803:ffca:df13:4192]) by smtp.gmail.com with ESMTPSA id i128-v6sm7835971ywe.42.2018.10.21.17.16.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 21 Oct 2018 17:16:17 -0700 (PDT)
From: Bret Jordan <jordan.ietf@gmail.com>
Content-Type: multipart/alternative; boundary="Apple-Mail=_39BD6B90-7EAE-4082-AB24-ABE0297D5F24"
Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\))
Date: Sun, 21 Oct 2018 18:15:32 -0600
References: <12DD2F97-80C3-4606-9C6B-03F7A4BF19DE@gmail.com> <D21F3A95-0085-4DB7-A882-3496CC091B34@gmail.com> <CAOASepM=hB_k7Syqw4+b7L2vd6E_J0DSAAW0mHYdLExBZ6VBuw@mail.gmail.com> <00ad01d460f4$69ae8a00$3d0b9e00$@augustcellars.com> <8436AEE7-B25A-4538-B8F6-16D558D9A504@gmail.com> <MEAPR01MB35428606C09BF315DE04CC79E5E10@MEAPR01MB3542.ausprd01.prod.outlook.com> <CAHbuEH6DCD7Zc+PK3TnCBkKv1esnROwyCcDb8ZR+TKwgQQ+yXQ@mail.gmail.com> <0E6BD488-74D5-4640-BC31-5E45B0531AFC@gmail.com> <CAHbuEH5oH-Km6uAjrSr0pEHswFBLuDpfVweQ+gpj472yk+8iTQ@mail.gmail.com> <073CB50F-8D91-4EF6-90BE-FC897D557AA6@oracle.com> <A37D69B1-6B77-4E11-8BB9-A0209C77752C@tzi.org> <45bf6c0f-e510-4afc-4277-bdd486a8ce8c@gmail.com> <213796DB-D875-46B0-9F3C-1A56F9E154BA@gmail.com> <ff1dcd4e-2bf4-b85b-dde3-2cc8fe29fb17@gmail.com> <447AB837-7208-4A96-91CC-89D30A2734FA@gmail.com> <24cc6bb7-ea40-1a9c-8847-8d6c74131587@gmail.com> <92B9F9AF-BBCA-472D-9155-935F695CE7CE@gmail.com> <3b6a338b-5588-deb2-9a9c-23e0cc24a2f1@gmail.com>
To: Anders Rundgren <anders.rundgren.net@gmail.com>, jose@ietf.org
In-Reply-To: <3b6a338b-5588-deb2-9a9c-23e0cc24a2f1@gmail.com>
Message-Id: <D7106932-DCD7-4D94-AECD-90FF07121635@gmail.com>
X-Mailer: Apple Mail (2.3445.9.1)
Archived-At: <https://mailarchive.ietf.org/arch/msg/jose/sDYbW83AohxxDStGIxnqDiFNZaE>
Subject: Re: [jose] Canonical JSON form
X-BeenThere: jose@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Javascript Object Signing and Encryption <jose.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/jose>, <mailto:jose-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/jose/>
List-Post: <mailto:jose@ietf.org>
List-Help: <mailto:jose-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/jose>, <mailto:jose-request@ietf.org?subject=subscribe>
X-List-Received-Date: Mon, 22 Oct 2018 00:16:22 -0000

Anders,

It is sad that we have to look at doing work outside of the IETF in order to solve the needs of the market.  But I would love to work with you on this effort, even if it means doing it outside of the IETF.

Your solution that uses 5 layers of signatures, which draft / document / write up is this based on?   I have a very similar need and would like to try and do what ever you are doing. 

Thanks,
Bret
PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050
"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."

> On Oct 19, 2018, at 10:55 PM, Anders Rundgren <anders.rundgren.net@gmail.com> wrote:
> 
> On 2018-10-12 23:31, Bret Jordan wrote:
>> Anders,
>> How do we move forward?  What can we do to make this happen?
> 
> As you can see from the responses, there is considerable resistance and skepticism against this work.
> I guess even fewer have bothered reading the actual I-D.
> 
> Personally I don't care that much since the JSON based systems I work with have up to 5 layers of signatures which doesn't translate well to the current JOSE standards.
> 
> Maybe the other editors of https://tools.ietf.org/html/draft-erdtman-jose-cleartext-jws-01 (which builds on a flavor of Canonical JSON) have an idea?
> 
> There is also a (very active) W3C community group working with a similar concept so clear-text signatures based on JSON canonicalization will probably reach the market regardless of IETF's position on the matter: https://w3c-dvcg.github.io/ld-signatures/
> 
> The 1 million of random and "edge" JSON Numbers used for testing the devised JSON canonicalization scheme, uncovered flaws in other systems as well...
> https://github.com/dotnet/coreclr/issues/17467
> 
> Ciao,
> Anders
> 
> 
>> Thanks,
>> Bret
>> PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050
>> "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."
>>> On Oct 12, 2018, at 10:47 AM, Anders Rundgren <anders.rundgren.net@gmail.com <mailto:anders.rundgren.net@gmail.com>> wrote:
>>> 
>>> On 2018-10-12 17:20, Bret Jordan wrote:
>>>> Please correct me if I am wrong…. The way I understand the problem is as follows:
>>> 
>>> Hi Bret,
>>> Comments in line...
>>>> 1) If you verified the JSON string at consumption time, before it has been unmarshal-ed, then all you need to do is decide how to handle white space and carriage returns.  You could basically regex remove all white space and CR / CRLF and have a workable solution.
>>> 
>>> It depends on what your goal is.  Canonicalization builds on the assumption that there is a unique representation of the data, preferably even after it has passed through a processor like an intermediary.
>>> 
>>>> 2) Where this breaks down is, when a tool unmarshals the data into a map or struct, then you have no guarantee that you would recreate the keys in the same order (a struct may force it to the order of the struct definition). So you have no way of being able to verify the hash after it has been unmarshal-ed.  Further, if you recreate the JSON and send it back out, the next person that gets the data might have a hash that can not be verified in option 1 above.
>>> 
>>> Right, therefore option 1 is not very useful.  Sorting of keys is the cure for this issue.
>>> 
>>> 
>>>> 3) Another problem once you have unmarshal-ed the data is what do you do with JSON numbers.
>>> 
>>> Right, but even string data needs adjustments.  "\u0020" and " " both represents a space character.
>>> 
>>> 
>>>> Some programming languages store them as a float, some as who-knows-what.  So you would need a way to ensure that the number was always stored in the same way, especially for strongly typed systems (is this architecture dependent too?). So the options here are, if the ontology / semantics of the JSON data were well defined in schema (a meaning it was standardized and documented), then the code could know what it should do and interoperability tests could be made to ensure that it worked.
>>> 
>>> This is (IMO) the only part of the puzzle that is non-trivial.  In my take on the matter, I have "prescribed" that the JSON Number type must be coerced into an IEEE 754 double precision number and be serialized according to ECMAScript V6+ rules.
>>> 
>>> If your application needs higher precision or bigger range, you are forced using the quoted string notation which (AFAIK...) is used by every IETF standard of any significance to date defining a JSON structure.
>>> 
>>> 
>>>> What am I not understanding here?  And what am I missing?
>>> 
>>> 
>>> As I wrote earlier, there are (at least) two entirely different and documented approaches.
>>> 
>>> Using a schema based canonicalizer as you mention is also an option but that is a much more ambitious project.
>>> 
>>> Regards,
>>> Anders
>>> 
>>> 
>>>> Thanks,
>>>> Bret
>>>> PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050
>>>> "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."
>>>>> On Oct 12, 2018, at 12:38 AM, Anders Rundgren <anders.rundgren.net@gmail.com <mailto:anders.rundgren.net@gmail.com> <mailto:anders.rundgren.net@gmail.com>> wrote:
>>>>> 
>>>>> On 2018-10-11 22:05, Bret Jordan wrote:
>>>>>> Anders,
>>>>>> I really like what you have done with this.  I am trying to figure out if it will work 100% for my needs, or if it will need some tweaking.  If it does work, then I think we should really try and figure out how we get your work standardized.
>>>>> 
>>>>> Thanx Bret!
>>>>> 
>>>>> The https://tools.ietf.org/html/draft-erdtman-jose-cleartext-jws-01 I-D provides quite a lot of features including an extension option that can be used for adding possibly missing functionality.
>>>>> 
>>>>> There is one thing that is good to know for anyone thinking about standardizing Canonical JSON and that's the fact that canonicalization also can be performed on the text level as described by: https://gibson042.github.io/canonicaljson-spec/
>>>>> 
>>>>> This has the advantage that it is very simple and supports the entire JSON RFC without restrictions.
>>>>> 
>>>>> 
>>>>> So why didn't I took this [superficially obvious] route? There are several reasons for that:
>>>>> 
>>>>> A downside of source level canonicalization is that it doesn't integrate with JSON parsers and serializers. https://tools.ietf.org/html/draft-rundgren-json-canonicalization-scheme-01 was explicitly designed to eventually be an option of a standard JSON serializer as it already is in my Java reference implementation.
>>>>> 
>>>>> Another issue is that it is unclear what the value is with using the JSON "Number" format outside of the IEEE range.  In fact, it excludes parsers like JavaScript's JSON.parse() unless JavaScaript would be updated to always use a "BigNumber" as fundamental numeric type.
>>>>> 
>>>>> Regards,
>>>>> Anders
>>> 
>