[Cbor] Re: Early allocation for packed CBOR (Re: Reminder: CBOR WG Virtual Meeting on 2024-12-11)

[Carsten Bormann <[e-mail]>]

Hi Vadim,

> On 15. Dec 2024, at 04:24, Vadim Goncharov <[e-mail]> wrote:
> 
> The problem is that cbor-packed does not achieve good compression at all.
> For 2017's CBOR source of 1210 bytes, tag 25/256 method achieves 904
> bytes (note it's forced for 3 bytes per even shortest reference, in
> contrast to cbor-packed cheating with single byte) and cbor-packed
> claims 793 for same method and 564 with prefixes - while even
> limited LZF achieves 425..435 bytes (depends on key order) and less
> limited LZ4 variants get 404..416 bytes.

There is no doubt that classical data compression will give you smaller encoded sizes than packed CBOR.
But you already know where to find deflate, brotli, or zstd; the latter with some nice additions such as dictionaries.
Trying to hyper-optimize Packed CBOR (e.g., by changing everything :-) is out of scope, as you always can go to well-established classic data compression for that.

CBOR packed is just solving a different problem [1]:

>> This specification describes Packed CBOR, a simple transformation of
>> a CBOR data item into another CBOR data item that is almost as easy
>> to consume as the original CBOR data item. A separate decompression
>> step is therefore often not required at the recipient.

[1]: https://www.ietf.org/archive/id/draft-ietf-cbor-packed-13.html#section-abstract-3

Different from classical data compression, there is no need to unpack/decompress Packed CBOR before using the data.  (Many applications will want to do that anyway, but the point is that APIs that make sense for constrained implementations can easily get by without.)
Packed CBOR is valid CBOR, so it can be embedded into other CBOR data and processed by generic decoders/encoders (*).

Obviously, not everyone needs Packed CBOR, so I can understand some haggling about tag space, in particular for the 40 code points the draft acquires for the referencing tags from the 1+1 space (plus three function tags and one basic table setup tag).
I believe this number is appropriate, given that the reference tags are pretty universal/future-proof as they can be used with table setup mechanisms we haven’t even invented yet.

$ tag-report.rb
range  used     %                 free                total
0 1+0    13 54.17                   11                   24
1 1+1    73 31.47                  159                  232
2 1+2  1087  1.67                64193                65280
3 1+4 65539  0.00           4294836221           4294901760
4 1+8     2  0.00 18446744069414584318 18446744069414584320

Grüße, Carsten

(*) And, yes, it would be interesting to come up with a design for instead using the 24 unused bit combinations (ai=28..30), but that, again, cannot be used as part of valid CBOR; it’s more like a CBOR 2.0 project.