Re: [ogpx] type-system : eliminate closing tags for collections in binary serialization?

["Robert G. Jakabosky" <[e-mail]>]

On Sunday 28, Han Sontse wrote:
> My instinct is to have the closing tag AND the object count.
> And apply the rule that count MUST equal the enclosed objects.
>
> Having both is a basic sanity check on the content.
>
> Robert's comments earlier about potential to exploit the
> protocol are found first within the ambiguities of the specification.

The main reason for my first e-mail was about the security issues related to 
being too lenient with malformed LLSD messages.  I am not going to complain 
too much if the current binary format is kept as is, as long as the rules are 
tightend to remove those ambiguities from the spec.

I am not a big fan of the current LLSD binary format, but it is not the worst 
binary format that I have seen.  I just hope that it isn't going to be used 
for the high-frequency low-latency messages needed for scene updates, since 
the cpu/latency cost of doing lookups on the map keys will be to high.

> Being a little pedantic in the object structure is good for
> security and consistency.   Some things should not be shortcut
> to save a few transmitted symbols.   UDP wrappers are a good
> example why it's a bad idea.

The issue with the SecondLife UDP protocol is how it is used to transfer 
reliable non-realtime messages, also it doesn't have support for 
encrypting/authenticating high security messages (i.e. money transfers).  
Which is why atleast some of those messages have been moved over to the 
https "Event queue".

The message encoding format isn't that bad, I like that it doesn't use tags in 
the encoded messages.

Here is an example of a nice data serialization system that doesn't require 
type info or tags to be encoded with every message:
http://hadoop.apache.org/avro/docs/current/spec.html

Avro uses JSON schema's to define records, enums, arrays, maps, unions, and 
fixed (i.e. binary values).  To handle forward/backwards compatibility, they 
have a section in the spec on schema resolution.  Avro even has a RPC 
protocol that allows the client/server to verify that they are both using the 
same schema.

I am not saying that VWRAP needs to switch from LLSD/LLIDL to Avro.  It is 
just a good idea to look at how other similar serialization systems have 
designed there binary protocols.

-- 
Robert G. Jakabosky