Re: [saag] ASN.1 vs. DER Encoding

[Nico Williams <[e-mail]>]

I wrote earlier that:

> And then they (rightly!) hate BER/DER/CER, so they propose inventing
> something new, often badly.  In this field, there is nothing new.  I'm
> sure even flatbuffers isn't new.
> 
> I say "rightly" because TLV encodings are just terrible.  We really do
> need non-TLV encodings (see below).

Now to back up that assertion:

1) TLV encodings are bloated by nature due to being highly redundant.

2) That redundancy is a source of errors when manually coding a codec.

   Now, this is not that big a deal because we should all be using code
   generators, and none should be manually writing a codec.  Yet there
   is so much hand-rolled BER/DER codec code out there...

   Also, all encodings will have lengths buried in structures whose
   lengths are also written elsewhere -- this redundancy is not entirely
   avoidable, but TLVs add more of it than is absolutely necessary.

3) DER is a canonical variant of BER, using a) definite length
   encodings of structures (SEQUENCEs) and other things, and b)
   minimal-length variable-length encodings of lengths and values.

   This has a few negative side-effects:

   a) it's not possible to know a length until the value it is the
      length of has been encoded, which means one must encode "from the
      right",

   b) there is no possibility of on-line encoding.

   On the other hand, CER uses indefinite length encoding, which avoids
   those two isses, but then nobody uses CER, not in Internet protocols
   anyways.  IIRC some other choice made in CER's specification turns
   out to be suboptimal, thus the choice of DER or CER is always
   dissatisfying in some sense.

   To be fair, we shouldn't need canonical encodings at all.  And yet
   isn't there a canonical JSON effort?  We never seem to fully stop
   having to re-encode structures...

   Also, it's fair to note that while DER has no possibility of on-line
   encoding, CER's use of indefinite length encodings means that
   decoding is necessarily online, with the recipient not able to know
   the total size of a message as it reads it.

   On the whole, indefinite length encodings are better.

4) BER (and DER and CER) is supposed to be self-describing, which means
   "you don't need to know the schema in order to parse the message",
   but this is only half true, as type information is lost when using
   IMPLICIT tags (you get to know if a value is of structured or scalar
   type, but not the actual type).

   Using EXPLICIT tags, on the other hand, makes the encoding a TLTLV
   encoding, thus adding even more bloat!  We use EXPLICIT tags in
   Kerberos, FYI.

5) If you're doing anything other than dump a structure, you don't need
   it to be self-describing -- you'll know the schema not least because
   we publish them.  As long as there's an indicator of top-level type
   on the outside, you can decode the inside by reference to the schema
   and encoding rules without needing TLV encoding rules.

   Thus there is almost zero benefit to self-describing encodings.
   
   Self-describing encodings are merely a crutch.  Perhaps they had
   their utility before compilers became the norm, but they have provide
   no real benefit now.

I think I could make more arguments in this vein.  I'll stop here.

Nico
--