Re: [Cbor] [EXTERNAL] Re: Handling duplicate map keys

Mike Jones <Michael.Jones@microsoft.com> Sun, 24 November 2019 22:11 UTC

Return-Path: <Michael.Jones@microsoft.com>
X-Original-To: cbor@ietfa.amsl.com
Delivered-To: cbor@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E77D11200C4 for <cbor@ietfa.amsl.com>; Sun, 24 Nov 2019 14:11:02 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2
X-Spam-Level:
X-Spam-Status: No, score=-2 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=microsoft.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id c-sXLnlLRvs5 for <cbor@ietfa.amsl.com>; Sun, 24 Nov 2019 14:10:59 -0800 (PST)
Received: from NAM06-BL2-obe.outbound.protection.outlook.com (mail-eopbgr650117.outbound.protection.outlook.com [40.107.65.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 9F80712003E for <cbor@ietf.org>; Sun, 24 Nov 2019 14:10:59 -0800 (PST)
ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kDnYUhZP70Wbg+mlaYyOc3zDqjoNTffiqBkL6bHRCvzbIxEodeL8T24cwQUR2ALbjWrHhRvqDVW1UNtPUyJFX9Yv/xCSk0/Jm2O6WkYeXzPr37NuivmdXCk/J0OQ6irejNM+GtdwSRJiXfi0/sPPyZVkClmcl5EATLMkQuNIO7Z4cUXiP7pWWMwPtST9MrEoM3ds61hMezPsV0XXZvYE4fDj9GT423ZHIha611JWar8TvZQJuysd7UibjUEYwmT9Px7I4Ab/Ah3ij2lj9BmQyC8DpNQhHC2e3UaGqt3J1meOjuUdwT+XmUCa6NZuOoY9IjXaToFBlaMq3NBaowipUw==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ha9IWPuJjFb4TxLJNUW3Mf4Rakx/WLgdJ5NMc+UKl4Q=; b=SKx4aO+tkfB/eAiIzWoUN6y1DQ4boTKfBsDaFoD30Eu2IsHe6xa62Jd+y/l/uMN9/ojCnXvf/8/2+RWTW3mujnGUVlJkqqQgWclTPiXf7EMdqnZrGEN1T0bWMwoyB9Oazv/TusXSzH7hNwfuLASTM9H32cCa4tGmzm2JWRbCbSRYkAXHijj+GzfIuR0ytfbwDNjdQorchdB0UvatKGXLKxGeuh5jC35UTAJl43o30fMnYUpQpunvRO5bjbYAkR5hBYpdDsTLdTjD6Q1qzOZLdyJsXtH/VYMeEyDje4X5oSy35uB4KIBCdrx+j7K+EDDwRBKiCdr/16qh3BzlsU1XlA==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ha9IWPuJjFb4TxLJNUW3Mf4Rakx/WLgdJ5NMc+UKl4Q=; b=Uvxl5FnEdJrfyOFPZq1WdFUae+akzLFov4LsRK9xtNdVsFh+EW9VkDIAbThv3GRTiICeA36hJ/q4vz9hTNztvWVrlJeStpueUYt8VUZ4pmDv6IOAu0Qp1PZwSqBCbt9n3YjcjSAp7ecTbhtd4kxO6J1Ngm7g7PkOnIVAUXAWqwA=
Received: from DM6PR00MB0569.namprd00.prod.outlook.com (20.179.51.12) by DM6PR00MB0845.namprd00.prod.outlook.com (20.179.167.151) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2525.0; Sun, 24 Nov 2019 22:10:57 +0000
Received: from DM6PR00MB0569.namprd00.prod.outlook.com ([fe80::2cfe:cf2c:2101:86be]) by DM6PR00MB0569.namprd00.prod.outlook.com ([fe80::2cfe:cf2c:2101:86be%9]) with mapi id 15.20.2526.000; Sun, 24 Nov 2019 22:10:57 +0000
From: Mike Jones <Michael.Jones@microsoft.com>
To: Laurence Lundblade <lgl@island-resort.com>, Jeffrey Yasskin <jyasskin@chromium.org>
CC: Jeffrey Yasskin <jyasskin=40google.com@dmarc.ietf.org>, "cbor@ietf.org" <cbor@ietf.org>
Thread-Topic: [EXTERNAL] Re: [Cbor] Handling duplicate map keys
Thread-Index: AQHVoGqK3tfbfd3j3UaGZbBcN3tqC6eYAo4AgAB4jwCAAGeBgIABRhYAgAB+gwCAAD+nYA==
Date: Sun, 24 Nov 2019 22:10:57 +0000
Message-ID: <DM6PR00MB0569D44CC35DD2AFEA56C621F54B0@DM6PR00MB0569.namprd00.prod.outlook.com>
References: <CANh-dX=EVDa4EwrgWKof4kCD3nkfV3BvH0Cg5ZKOivmXJ_Dm8g@mail.gmail.com> <E5C74E1F-A5F4-4AB8-9787-3999C4697C3B@island-resort.com> <CANh-dX=1gkAtfSG-yzCsVAkk=oLM-=dN_JLCr1kQK3d6Jb0fSw@mail.gmail.com> <F81E1A57-6072-44FA-A148-8F3ED7520791@island-resort.com> <CANh-dXmxzoEjb-yhL-R1p-xvBF8kZwOpzoS_fziPfuxFhbykhA@mail.gmail.com> <B3C04481-1414-4F90-8C99-62AD7E1819DD@island-resort.com>
In-Reply-To: <B3C04481-1414-4F90-8C99-62AD7E1819DD@island-resort.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=dabd4818-01b3-4b86-86ca-000086860efd; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2019-11-24T22:09:33Z; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47;
authentication-results: spf=none (sender IP is ) smtp.mailfrom=Michael.Jones@microsoft.com;
x-originating-ip: [50.47.93.218]
x-ms-publictraffictype: Email
x-ms-office365-filtering-ht: Tenant
x-ms-office365-filtering-correlation-id: 4d39a00f-8ea1-4a35-64cb-08d7712b2ea1
x-ms-traffictypediagnostic: DM6PR00MB0845:
x-microsoft-antispam-prvs: <DM6PR00MB0845F65E1B1E80DD98033FEBF54B0@DM6PR00MB0845.namprd00.prod.outlook.com>
x-ms-oob-tlc-oobclassifiers: OLM:626;
x-forefront-prvs: 02318D10FB
x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(366004)(396003)(346002)(376002)(39860400002)(136003)(199004)(189003)(54906003)(25786009)(110136005)(71200400001)(71190400001)(53546011)(6506007)(7696005)(76176011)(186003)(55016002)(6306002)(54896002)(102836004)(236005)(26005)(316002)(6436002)(3846002)(66066001)(229853002)(9686003)(66446008)(33656002)(14454004)(8936002)(66946007)(478600001)(10290500003)(6116002)(790700001)(22452003)(5660300002)(256004)(7736002)(4326008)(2906002)(8676002)(52536014)(66476007)(99286004)(66556008)(81166006)(81156014)(64756008)(74316002)(10090500001)(446003)(6246003)(8990500004)(86362001)(76116006)(14444005)(11346002); DIR:OUT; SFP:1102; SCL:1; SRVR:DM6PR00MB0845; H:DM6PR00MB0569.namprd00.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1;
received-spf: None (protection.outlook.com: microsoft.com does not designate permitted sender hosts)
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info: yIUxBJss6rTyjLjAxAtKCkxHIU8mBDxKdoGy7qaHwwW+pmgNrVnmnmSIFQAI6NVmeiaFW2QzpMU2uNPg+n587vqKnqXdqYJzf1zDbJ4AV/+nOV0lLaiaFD1mZpLgVAG8ejASEjV8T2XUTCgLTXb+oENvnrAfwYPdv2gYY5HWdeBpzvbCqLSkKT60ScmUv/RKKHoCtKAgsrU2f3miaRiSZuv4OEI2shSp3h9kcxOoqQlFF+xbUNiwQrhOpfUUZKWpwE6fzGHcEf5UhywJpV5wOyEGpU9x/bU1Iuu54yiWMmQhkNpzXZehabVGgIyhJC8vFenO4nAjHZds2CZqRmRDyaDP5GJEhLLpHoT5tKNxSDWxz1CgizphXomJUx5SsUZE7IsmGz/uTanObcB4EfwbC54a7qt/9pW8KjFd7cHx35aXyvAjFb9hjNvcNtTirL/b
x-ms-exchange-transport-forked: True
Content-Type: multipart/alternative; boundary="_000_DM6PR00MB0569D44CC35DD2AFEA56C621F54B0DM6PR00MB0569namp_"
MIME-Version: 1.0
X-OriginatorOrg: microsoft.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4d39a00f-8ea1-4a35-64cb-08d7712b2ea1
X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Nov 2019 22:10:57.5396 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 6sQF0xxmwxQlfMVedreHWoIIcPeNBqNxt7ahnjRCvOjcsXtuLVlnExgMs5iLgclYIqmsThH7wmfkiVR0j6akaw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR00MB0845
Archived-At: <https://mailarchive.ietf.org/arch/msg/cbor/VoNbYwxnQEO6k_buRTOE-euDOPo>
Subject: Re: [Cbor] [EXTERNAL] Re: Handling duplicate map keys
X-BeenThere: cbor@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "Concise Binary Object Representation \(CBOR\)" <cbor.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/cbor>, <mailto:cbor-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/cbor/>
List-Post: <mailto:cbor@ietf.org>
List-Help: <mailto:cbor-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/cbor>, <mailto:cbor-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sun, 24 Nov 2019 22:11:03 -0000

I advocate wording that says you MUST NOT create protocols or specifications that count on any behavior for duplicate map keys other than rejecting the input.  Anything else opens up a Pandora’s Box of non-interoperability.

                                                       -- Mike

From: CBOR <cbor-bounces@ietf.org> On Behalf Of Laurence Lundblade
Sent: Sunday, November 24, 2019 10:22 AM
To: Jeffrey Yasskin <jyasskin@chromium.org>
Cc: Jeffrey Yasskin <jyasskin=40google.com@dmarc.ietf.org>; cbor@ietf.org
Subject: [EXTERNAL] Re: [Cbor] Handling duplicate map keys




On Nov 24, 2019, at 2:48 AM, Jeffrey Yasskin <jyasskin@chromium.org<mailto:jyasskin@chromium.org>> wrote:

I think you make a good point that if a protocol is going to specify "take first", it can probably specify "error on duplicate key" for the same cost. That might imply some much simpler text:

"Protocols based on CBOR SHOULD fail with an error if a map contains a duplicate key, except that if the key isn't used at all, they MAY ignore it instead. Protocols that do not reject duplicate keys MUST (?) document why the cost of rejecting duplicates is too high and why accepting them will not lead to security vulnerabilities. An array might be a better choice for such protocols.”

I think I’d invert it and say that protocols that require duplicate detection for security reasons should describe that requirement in security considerations so the implementor gets a good solid hint that they need to worry about it.

There's an interesting difference of approach here. It's plausible to say that protocol designers should pick the secure design only when they realize their design has security implications, but I prefer to say that they should pick the secure design unless they think about it and realize the design doesn't have security implications. I think the second will get us noticeably more security in a world where designers don't have time to think about every aspect of every piece of their design.

I don’t know how many generic decoders exist that really can’t do or support duplicate detection. I also don’t know how many use cases there are that can’t afford duplicate detection. If there are few of both, then making no duplicate detection the exception that needs to be called out seems right.

I have been thinking about CWT and EAT as examples. Maybe EAT should say that duplicate detection should be required unless a profile of EAT says it is not, the secure choice. For the most part EAT will be processed on servers where there is clearly no resource issue. The only issue would be the implementation  environment (Java, PHP, Python…).

EATs might be decoded on constrained devices sometimes, but that is rare. In that case a profile can relax the requirement.

EATs are always signed so duplicates would only come from bad implementations. There are plenty of ways that am implementation can go wrong other than duplicating keys that are completely undetectable.



There is lot of text implying that duplicates are bad, but I think it would be worth being explicit.

You are an idiot if you design a protocol that considers duplicate input valid or necessary. Duplicate map keys are always an error in CBOR and will not work with many generic decoders.

I wouldn't say "idiot", just "wrong". The spec already says you "MUST NOT" do this, but I'm not opposed to making that statement more direct.

Pardon my wording. Certainly, I do not want that as normative text.



Thanks,
Jeffrey