[Cbor] Number formats (Re: Reminder and call for agenda: CBOR WG Virtual Meeting on 2023-05-31)

[Carsten Bormann <[e-mail]>]

Hi Emile,

interesting observations.  It seems to me you would like XDR (RFC 4506), except that this requires following a data description for ingesting (and was developed when 32 bits were a canonical size).

I think it is important for the CBOR WG to keep an eye on where binary (as in not text-based) representation formats should be moving.  And that should involve skating where the puck will be, not necessarily Brownian motion on old discussions.

With respect to number representation, right now an important trend is being able to accomodate low-precision numbers.
In 2013, CBOR was one of the first representation formats to embrace fp16 (16-bit se5m10 floating point), while others generally only had the 32-bit and 64-bit floating point formats.
Low-precision numbers typically occur in large amounts, so in 2020 we added tags for typed arrays (RFC 8746), which by the way do provide for direct use of little-endian numbers as well as of two’s complement formats for signed numbers — the conversion overhead into a preferred number format can be significant for arrays of numbers, and a single tag can control the format for the entire array.

The next step here might be to look for popular low-precision formats not yet covered in that tag set, e.g., Google’s bfloat16 or bf16 (se8m7 truncated fp32), the popular fp8 formats (se2m5 to se5m2), unsigned formats, weirder lengths (19-bit Nvidia se8m10 tensorfloat/“tf32", 24-bit AMD se7m16 fp24), shorter floats (fp4, fp2, fp1!), integers (int4, int1), or the various new proposals to leave the mold of IEEE 754 entirely.

The tag system provides CBOR with a way to easily accommodate these formats as they become more popular in the AI/ML applications.

Grüße, Carsten