[Cbor] Concatenation tag (was: Re: Reminder and call for agenda: CBOR WG Virtual Meeting on 2022-06-01)

[Christian Amsüss <[e-mail]>]

Hello Carsten, hello group,

> > who is wondering whether indefinite-length strings / bytestrings really
> > needed to be in the original CBOR spec, or would not have been better
> > served with a "concatenate" tag around an indefinite-length array.
> 
> Knowing after almost a decade how we have learned to use tags, this
> could indeed be done differently…

well we won't change this any time soon, but I'd like to propose
something a bit like tag 65535 (after it occurred to me while writing
the other mail that it works not only for indefinite-length strings but
also for indefinite-length arrays):

Concatenation tag
=================

This registers tag TBDcat to indicate the concatenation of two CBOR
items.

The tag takes a finite length array of largely homogenous types as an
argument, and concatenates them based on the type of the first argument:

* Text and byte strings are plainly concatenated. Subsequent arguments
  each may be of either type (converted through the UTF-8 encoding).
  Byte strings concatenated into text strings need to contain valid
  UTF-8 data. (These rules are consistent with those of Section 2.2 of
  RFC 9165 on CDDL concatenation).

* Arrays are concatenated.

* In dictionaries, the union of the dictionaries is formed. The second
  occurrence of a key takes precedence.

All items in the dictionary may be tags; their behavior under
concatenation is specified in the tag, but needs to be compatible with
any expanded form that tag may have. (For example, concatenation tags
can be nested, and mixed with array tags).

Use case: Implementation aids
-----------------------------

Arrays, strings and maps of indefinite length are sometimes supported
badly by CBOR libraries, especially those for constrained devices --
their consumers might expect strings in contiguous memory, which the
library can not provide.

Rather than implementing additional interfaces for slice-wise access, a
CBOR consuming library can represent indefinite length data structures
by nested tags TBDcat. For example, if it received

```
(_ "hello, ", "world")
```

it may present that to the application as

```
TBDcat(["hello", TBDcat(["world", ""])])
```

and also accept data represented that way for encoding.

Whether a library performs that transformation is up to its
configuration and/or its documentation. It is not recommended to define
CBOR based protocols in which there is a semantic difference between an
indefinite length array / string / map and one constructed by this tag,
but they may specify that only one of them can be used for encoding. [
Or would it be better to recommend never to serialize anything with
TBDcat unless other tags are involved? ].

Use case: A bit inhomogenous arrays
-----------------------------------

Arrays that can largely be expressed as homogenous arrays can use this
as an escape hatch for out-of-range items:

```
TBDcat(64(h'000101020305080d1522375990e9'), [377, 610])
```

Use case: packed CBOR
---------------------

The concatenation tag can be used with packed CBOR, in partular when a
dictionary item does not on its own support midfix expansions.

[ This might be even more useful when used as part of dictionary items,
but how that works is under active discussion. ]


Thoughts, comments?

BR
c

-- 
To use raw power is to make yourself infinitely vulnerable to greater powers.
  -- Bene Gesserit axiom