Re: [apps-discuss] Standardizing protocol buffers

[Phillip Hallam-Baker <[e-mail]>]

+1 and how does all this relate to JSON?

Also we should note that we have another very similar scheme inside TLS.

There are good reasons to prefer binary encoding over JSON and vice versa.
But it would be rather nice if we had a way to abstract the encoding choice
from the protocol.

For example, JSON does not support binary types which makes it rather
inefficient as a wrapper for video streams or the like. Another problem
with JSON is that it uses an escaped encoding for strings and those raise
security issues and performance issues. [And no, telling people not to make
mistakes is not a security solution]

There are also advantages to JSON of course but which one is best is going
to depend on the target application. So I think that is a choice that
should be made in deployment rather than by the protocol designer.


I think we should try to minimize the number of packing options. Most
decisions do not matter at all, some matter a lot but there is a very clear
choice (don't repeat DER) and some depend on circumstances.


As far as the details go, there are only a few useful design choices in a
packing scheme. I am certain this proposal has some of them wrong.


1) Integer encoding

The encoding of negative integers needs to be much better thought out. As
should the fact that an integer might well be larger than 64 bits.

The Varint scheme uses the top bit of each integer to encode the run
length. This means that if we AND each octet of an integer with 0x80 we
will get one of the following

0x00
0x80 0x00
0x80 0x80 0x00
...

The problem with this scheme is that it requires each integer to be packed
into 7 bit form which is kind of inefficient. Lots of rotates and so on. A
more time efficient scheme is to follow the ASN1 approach and pack the run
length into the first byte in some fashion. So if the top byte is:

0xxx xxxx length = 1
10xx xxxx length = 2
110x xxxx length = 3
1110 xxxx length = 4
1111 nnnn length = 4 + nnnn

One of the big advantages of this approach is that it avoids the need for
the 7 bit shifting which is not only inefficient, it requires special case
fixup for negative integers. A twos compliment number can be fitted in very
easily.


2) Tagged or untagged data

JSON and XML data is tagged which means that it is very flexible. Adding an
extra data field does not cause chaos. It was not immediately apparent to
me which approach this proposal takes.

ASN.1 data gives a choice on a per field basis. This does not seem to be a
helpful approach (to say the least).


3) Typed or untyped

This is not the same as tagged or untagged. A data structure {"a" : 1} has
different semantics from {
"b" : 1}. The tags may imply a type but they are not themselves a type
declaration.

JSON data is implicitly typed by the syntax. ASN.1 is not.

Any conversion from a binary format to JSON needs to have the type
information. That may come from either a schema or be explicit in the data
stream as in BER encoding.


4) Can it be implemented as a linear traversal?

The really insane part of ASN.1 is actually two lines buried in the X.500
documentation defining DER. According to the DER encoding rules, an object
is represented using the definite length encoding. So each object is
prefixed by the total length of the data fields.

This is a problem because the included data fields may also be objects and
the length field is encoded using variable length encoding. So emitting DER
encoding requires a totally pointless double recursion down the tree and
back again, keeping track of multiple values. It is just stupid, stupid,
stupid.

Oh and there is absolutely no security reason for DER encoding, none, zip,
nada.

Again, the draft seems pretty vague on this point as well.






On Thu, Oct 11, 2012 at 4:26 PM, Carsten Bormann <[e-mail]> wrote:

> On Oct 11, 2012, at 19:21, "Rex Fernando (rex)" <[e-mail]> wrote:
>
> > http://www.ietf.org/id/draft-rfernando-protocol-buffers-00.txt
>
> Hey, I can do that, too.
>
> http://www.ietf.org/id/draft-bormann-apparea-bpack-00.txt
>
> How many of these do we need?
>
> (I'm actually quite serious with binarypack, as there is a lot of stuff
> out there that uses msgpack, and it probably would be a good idea to do a
> version with IETF change control.  Area of application for me is mostly
> constrained node/network REST, but of course it is a quite universal
> hammer.  Not the impact wrench that protocol buffers is :-)
>
> Grüße, Carsten
>
> _______________________________________________
> apps-discuss mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/apps-discuss
>



-- 
Website: http://hallambaker.com/