Re: [apps-discuss] Gen-ART review of draft-bormann-cbor-04

[Carsten Bormann <[e-mail]>]

On Aug 5, 2013, at 19:43, Joe Hildebrand (jhildebr) <[e-mail]> wrote:

> Sorry, my response is also correspondingly long.  


The present message wraps up the more general comments sent by Joe
Hildebrand concerning CBOR.  Lots of juicy comments.  Thanks to all!
For these, we are not using the line-by-line style because many of the
topics were touched on by more than one person.

A separate message will follow with line-by-line replies to the more
specific comments.

# Data model

## Extensibility

It is easy to fall into the trap of optimizing for the current
environment.  Experience says there will be some future requirements
that are genuinely unknown today, which may require extending a format
in unforeseen ways.  CBOR plans for this by leaving ~ 10 % of the
encoding space unused (additional information 28..30).  There is
fundamentally no way to guarantee backward compatibility for this kind
of unpredicted extension process.  Leaving code space unallocated may
allow for forward compatibility, though.

CBOR also will be used in multiple environments.  Instead of forcing
each environment to invent their own format, CBOR allows adapting the
format by providing extensible tagging and extensible primitive
("simple") values.  Both are based on registries.  Tags are more
important in practice (and likely to be more numerous).  The tagging
scheme even allows first-come-first-served registration.

Obviously, there is a trade-off between adaptability and
interoperability.  CBOR tries to maximize interoperability by allowing
a generic parser to easily handle tags and primitives the registration
of which it is not aware of.  This will also enable evolving the
decoders and the applications using them on separate timescales, if
that is desired.

We have the same cognitive dissonance as everybody else about opening
the floodgates to extensibility.  The alternative, making it painful
or impossible to adapt the spec to a specific environment, is not
necessarily more desirable. Many existing data formats make
extensibility impossible; if a protocol or application wants one of
those, that's fine; in fact, some of the other data formats that have
been brought up since we started the discussion of CBOR have also
chosen the "no extensibility" path.

## Tags

CBOR has a number of predefined tags.  This is one of the pain points
for extensibility: most tags will not be of interest to some
application or protocol developers, but having them specified early in
the process will lead to better interoperability for those who want
them. During the early development of the spec, Paul and I disagreed
about which tags should be in the first RFC, but ultimately it doesn't
really matter because the choice of tags is up to the protocol or
application that uses CBOR. For all of these tags, a specific CBOR
application will specify which of them are actually in use.

There is little disagreement that we need a tag for a date/time
format; there is probably no way to agree on a single one, so the
document currently includes both RFC 3339 and epoch-based formats.
Bignums (integer) may be less obvious to some, they are very obvious
in other environments.  Having (extended precision) base2 floating
point as well as a base10 floating point in there also seems obvious
for those environments that need it. These are the typical programming
language types.  YAGNI doesn't apply; leaving them out of the spec
just means that there likely will be several, incompatible versions of
them.

Four tags on strings indicate further likely kinds of data.  Nesting
CBOR encoding into a byte string seems natural to support.  We also
included tags for URIs, regular expressions, and MIME messages (OK,
maybe less obvious).

Then we have two tags that indicate that a text string is encoded in
Base64/Base64url (33/34).  The argument could be made that these could
be decoded and sent as byte strings; however, there may be a need to
keep the exact format (whitespace and padding), e.g. for signature
checking.

Tags 21-23 do the inverse: they tag byte strings with the base
encoding that any JSON converter on the path should be
using. Converter hints may be a new concept in a data format; in a
world where JSON is widely used in processing chains, they could be
eminently useful (Paul and I still disagree on this one.)

Tags can be nested.  Actually, since every tag specifies its possible
contents, none of the tags specified in -04 can be nested.  To cite an
obvious example where nested tagging would be used, tag 1 accepts a
floating point value and might be changed to also accept a base10
floating point value (tag 1 currently does not allow that, though).

Disallowing the nesting of tags just forces a protocol designer to
turn tag1(tag2(x)) into something more complex like
tag1([tag2(x)]). We don't think forcing the introduction of this
complexity reduces complexity.

For an implementation that does know both tags, the processing will be
natural.  Any other implementation can ignore the unknown tags or
(preferably) hand them to the application in a similar way the other
containing data structures are handed to the application (i.e, tagx(y)
is not much different in handling from [x, y] or {x: y}).

A later BCP might want to warn specifiers of future tags that
designing them in such a way you get recursion (as opposed to mere
nesting) is not such a good idea, but you have to come up with very
contrived examples before that becomes a concern.

# Encoding issues

## Numbers

Interchanging numbers is a much more complicated issue than one would
think.  Recent discussions of number precision issues in the JSON WG
show that it is easy to underestimate the problem and get this wrong.

In the area of integer numbers, CBOR is highly opinionated in favor of
unsigned (i.e., non-negative) numbers — this is what is actually
needed in the majority of applications.  Many applications also will
clearly need negative numbers, so they have their own encoding.
Obviously, if you encode N bits for a negative number, you need N+1
bits in a signed representation of that since you need to add the sign
bit.  We chose to keep N at 8, 16, 32, 64, so N+1 has a slightly odd
size, which implementers indeed need to be aware of if that is larger
than their number range (not a problem, e.g., for JavaScript).  CBOR
protocols should be explicit about the number range and precision that
is expected to be supported.

In the area of floating point numbers, CBOR ignores IEEE 754 decimal
numbers (in favor of its own, simpler format) and only supports three
sizes of the IEEE 754 binary floating point numbers.  That is, again,
opinionated (but can be fixed in a pinch by introducing appropriate
tags).  The inclusion of 16-bit floating point is motivated by its
usefulness for low-precision sensor data.  The reason this may not be
obvious to many is likely that programming languages evolve on time
scales where this 2008 addition to IEEE 754 is still very, very
recent.

## Maps

As with most modern programming languages, CBOR has a map type; this
is intended to be a general map and not to be limited to text string
keys as in JSON objects.  Dependent specifications have already been
started that make good use of non-text keys.  In particular, integer
keys are a good match for environments that already maintain some form
of registry based on integers.

CBOR maps are intended to be maps, not multimaps.  Duplicate keys are
an encoding error.  Unfortunately, what should be considered duplicate
is often application dependent (are 0 and 0.0 the same key?  "Å" and
"Å"?), so a generic parser will only be able to do part of the work.

CBOR maps do not ascribe meaning to the order of items in the
map. This is hard to prescribe in the format; it is a requirement on
the applications using it.

# Implementation issues

Complexity/Code size: We will leave an assessment of that to those who
have implemented the spec (and we have heard from a few).  The
pseudo-code in Appendix C should give some impression how a minimal
implementation could look like.

Performance: Memory allocation indeed is a significant contributor to
the CPU use of a format decoder.  CBOR tries to help a bit by
providing right at the start of a data item the size of the object
that needs to be allocated (as opposed to the length of its
representation as in ASN.1 BER).

Of course, that is not possible in indefinite length encoding
("streaming"); here we trade off some additional CPU in the decoder
(needed for reallocation/copying of unknown-length structures) against
less complexity in the encoder, for those cases where the encoder
would need to perform significant buffering to obtain these sizes
before encoding a data item.

# Document issues

We are open to suggestions for replacement words for "well-formed" and
"valid" in Section 3.7.  A definition of "valid" is not actually
needed; that term is only used in its English sense in the discussion
of generic parsers (and the connotation with XML's sense of valid is
unfortunate).  "Well-formed" is defined algorithmically in the
pseudocode in Appendix C.

For a binary format, there is a strong requirement for a diagnostic
notation that everyone understands that is trying to get an
application going.  So defining it in the document is the best way to
get this.  This definition is not needed for interop, so we can do
away with ABNF etc.  I started out by saying something equivalent to
"MUST NOT be parsed", but an RFC 2119 MUST struck me as silly when
diagnostic notation is not meant for interop in the first
place. (Also, there might be tools for encoding test vectors etc. that
do parse diagnostic notation; in -05, we have added a pointer to YAML
as another format that may be useful for text-based specification of
CBOR data items in configuration files.)

Grüße, Carsten