Re: [ogpx] type-system : eliminate closing tags for collections in binary serialization?

[Dahlia Trimble <[e-mail]>]

I've used binary LLSD for scene update messages in a prototype viewer and
protocol for OpenSim, and I haven't found deserialization overhead to be an
issue. When you do things like sort 10000 scene nodes by distance from the
camera a couple times a second while you're also constructing 100+
procedural objects and hashing hundreds of material descriptions,
deserializing a few messages is insignificant. I suspect there's probably
other delay causing problems with LLSD in general, or possibly related to
the transport mechanism, but I haven't seen any specifically with binary
serialization/deserialization. Note: this was using the libomv
implementation.

On Mon, Mar 29, 2010 at 12:28 AM, Robert G. Jakabosky <[e-mail]
> wrote:

> On Sunday 28, Han Sontse wrote:
> > My instinct is to have the closing tag AND the object count.
> > And apply the rule that count MUST equal the enclosed objects.
> >
> > Having both is a basic sanity check on the content.
> >
> > Robert's comments earlier about potential to exploit the
> > protocol are found first within the ambiguities of the specification.
>
> The main reason for my first e-mail was about the security issues related
> to
> being too lenient with malformed LLSD messages.  I am not going to complain
> too much if the current binary format is kept as is, as long as the rules
> are
> tightend to remove those ambiguities from the spec.
>
> I am not a big fan of the current LLSD binary format, but it is not the
> worst
> binary format that I have seen.  I just hope that it isn't going to be used
> for the high-frequency low-latency messages needed for scene updates, since
> the cpu/latency cost of doing lookups on the map keys will be to high.
>
> > Being a little pedantic in the object structure is good for
> > security and consistency.   Some things should not be shortcut
> > to save a few transmitted symbols.   UDP wrappers are a good
> > example why it's a bad idea.
>
> The issue with the SecondLife UDP protocol is how it is used to transfer
> reliable non-realtime messages, also it doesn't have support for
> encrypting/authenticating high security messages (i.e. money transfers).
> Which is why atleast some of those messages have been moved over to the
> https "Event queue".
>
> The message encoding format isn't that bad, I like that it doesn't use tags
> in
> the encoded messages.
>
> Here is an example of a nice data serialization system that doesn't require
> type info or tags to be encoded with every message:
> http://hadoop.apache.org/avro/docs/current/spec.html
>
> Avro uses JSON schema's to define records, enums, arrays, maps, unions, and
> fixed (i.e. binary values).  To handle forward/backwards compatibility,
> they
> have a section in the spec on schema resolution.  Avro even has a RPC
> protocol that allows the client/server to verify that they are both using
> the
> same schema.
>
> I am not saying that VWRAP needs to switch from LLSD/LLIDL to Avro.  It is
> just a good idea to look at how other similar serialization systems have
> designed there binary protocols.
>
> --
> Robert G. Jakabosky
> _______________________________________________
> ogpx mailing list (VWRAP working group)
> [e-mail]
> https://www.ietf.org/mailman/listinfo/ogpx
>