[mmox] Compact Binary Serialization

[Catherine Pfeffer <[e-mail]>]

> There needs to be support for a binary serialization that does not
> contain embedded type or key references, but instead use explicit
> external schema. Thus, to describe a "quaternion" you would simply
> specify four fp16 (or fp32) values in sequence, with no specific type
> information. This can extend to describing general entity property
> values: the type information can be carried by the schema for the
> entity, and does not need to be encoded in the actual data. Allowing
> external schema leads to significant bandwidth savings during
transmission.

Wow, this one is violent!

OK, let's examine it in detail:

1) Example of the quaternion.

As I understand it, a quaternion is currently implemented as:
   <array>
        <float>0.5</float>  <!-- x -->
        <float>-0.5</float>  <!-- y -->
        <float>0.5</float>  <!-- z -->
        <float>0.5</float>  <!-- s -->
   </array>

(in the order x, y, z, s and not math's order s, x, y, z, that's one of
Second Life's strangenesses)

bandwidth could be spared with something like:

   <array>
        <float repeat="4">0.5,-0.5,0.5,0.5</float>  <!-- x, y, z, s -->
   </array>

(If you ask me why this is enclosed within <array> tags while we could
remove these tags without losing any information, I think the answer is:
"because it leads to a cleaner C++ code to access the data in that way".
Correct me if I'm wrong here, Infinity)

Such a functioning could spare bandwith (57 bytes versus 87 bytes for this
quaternion), while still remaining compatible with the other serializations
(but not in a 1:1 straightforward compatible way as they are now).

There could even be more bandwidth spared with constructs like:
   <array>
        <floats>0.5,-0.5,0.5,0.5</float>  <!-- x, y, z, s -->
   </array>

because you don't really need to specify in the data that there are four
floats, it's obvious from the number of commas. That would be 47 bytes
versus 87, including the useless <array> tags. Nearly the half.


2) Moving the type information to the schema.

Yes, that's the "ordinary way" of encoding data in XML.

Ordinary way:
     XML schema or DTD says: we have the avatar's nose orientation
quaternion here. That's four fp32.
     XML data over the wire:
<nose-orientation>0.5,-0.5,0.5,0.5</nose-orientation>

LLSD (simplifying):
     LLIDL says: nose orientation is an array of four fp32s
     LLSD data over the wire: <floats>0.5,-0.5,0.5,0.5</floats>

(Example of nose orientation might be idiotic, I had no time to choose it
better.)
(I have used <floats> and not 4 <float>s because that was boring to type)

**** Jon, you are touching something at the very core of LLSD here. ****

Where ordinary XML markup would say "this is nose orientation", XML
serialization of LLSD says "this is four floats".

While such choices seem weird in an XML file, they make a lot of sense for
binary serialization where the data type of the following data can be
specified in a single byte: "the next sequence of four bytes is going to be
a floating point number". It would be impossible to declare "now there is
going to be the nose's orientation. For the number of bytes involved, look
at the schema" in a binary file using a simple byte.

That does not mean it is impossible to put semantics in the binary file sent
over the wire, but you have to do it with maps, i.e. with the binary
serialization equivalent of:
  <map>
       <key>nose-orientation</key>
       <array>
          <floats>0.5,-0.5,0.5,0.5</floats>
       </array>
   </map>

Have I summed up correctly the whole reasoning behind that, Infinity?

Compare the bandwidth difference with:
      <nose-orientation>0.5,-0.5,0.5,0.5</nose-orientation>
No need to comment on that.

I see this as another example of "Yeah, that's strange XML, but it maps 1:1
to a nice and natural binary serialization" :-(.

Now back to your question: "should we move the type information to the
schema, as all people do in XML?". The answer is probably: "no, because we
want to keep a simple 1:1 mapping with the binary serialization".

Now, could we do partial exceptions for common types like vectors or
quaternions? Perharps. I already suggested:
         <integers>127,255,127</integers>
         <floats>100.0,75.3,28.6</floats>
         <floats>0.5,-0.5,0.5,0.5</floats>

but it could also take other forms with more semantics like
         <colour>127,255,127</colour>
         <vector>100.0,75.3,28.6</vector>
         <quat>0.5,-0.5,0.5,0.5</quat>

although I see the second solution as harder to fit in the existing LLIDL,
which works at a lower level of semantics.

(I intentionally did not store the colour in a vector, because that's
another SL-ism).


I hope that helps.

-- 
Cathy