Re: [mmox] XML serialization

[Jon Watte <[e-mail]>]

Meadhbh Hamrick (Infinity) wrote:
> our system works fine with the new serialization. this would require a 
> non-zero amount of resources be spent on deploying the change. ergo, 
> we would need some justification for making the change.
>

Do you agree that any kind of generally suitable interoperability will 
require engineering effort, sometimes significant, on the part of all 
possible implementors, including Linden Lab or the Open Sim developers?

You do not have to update your internal serialization at all just 
because we, collectively, design a serialization scheme with 
differences. You don't need to do any work until and unless such time that
1) there is an actual standard, or at least a concrete proposal
2) you decide that supporting this standard is in the best interests of 
your company or group

>
> but may i ask? why do we need to change something that we know works, 
> is already deployed, and seems to work without flaw?
>

Because it can work better. It can be more compact, it can better suit a 
variety of parser styles, it can be easier read and written by human 
beings, it can be easier expanded or adapted, it can more easily be 
processed using fast hashing-based (order independent) processing methods.

If your argument is that "while I understand that some other people feel 
that direction X is better, we are currently using direction Y in the 
Second Life implementation, so I think Second Life's way should prevail" 
then you're not actually signed on to the generally applicable, vendor 
neutral interoperability standard bandwagon.

> * can we do a bake off?

Once we're done with a serialization format that fulfills all the 
necessary requirements, I believe there won't be anything "else" to 
bake-off against.
See here for some requirements I would put on a standardized format:
http://www.interopworld.com/node/32

I will copy and paste for the convenience of those who don't like 
out-of-line references:

I have looked at the proposal, and see a number of issues that I think 
need to be addressed before the proposal can be used as a general 
interoperability standard.

! Proposal Name

The name probably should change to something like "VWSD" to avoid any 
accusation of vendor bias. I trust we can just get this done without too 
much argument about specific naming.

! Large and Small Integers

The "integer" data type needs to be more flexible. 64-bit integers are 
important, and you can even view UUIDs as 128-bit integers. I propose 
that integers can be specified as a specific bit width, and signed or 
unsigned. If you want to stay byte aligned, the set of allowable sizes 
may be limited to 8, 16, 32 or 64. I additionally propose that a 
Variable Length Integer Encoding be specified.

! Float32 and Float16

Reals should come in at least two forms: 64-bit and 32-bit. The reason 
is that localized position relative to some center usually is best 
described as a 32-bit float. Additionally, mesh data is generally 
described as 32-bit float vertices, rather than 64-bit. There may be 
some additional benefit in supporting 16-bit floats, for things like 
normals, color values or direction vectors.

! Compact Binary Serialization

There needs to be support for a binary serialization that does not 
contain embedded type or key references, but instead use explicit 
external schema. Thus, to describe a "quaternion" you would simply 
specify four fp16 (or fp32) values in sequence, with no specific type 
information. This can extend to describing general entity property 
values: the type information can be carried by the schema for the 
entity, and does not need to be encoded in the actual data. Allowing 
external schema leads to significant bandwidth savings during transmission.

! XPath Friendly XML

The XML encoding needs to be XPath friendly, and friendly to introducing 
or adapting new types in a negotiated higher-version schema. Generally, 
this means that it looks something like:
<map>
<value key="success" type="boolean">true</value>
<value key="cpu_temp" type="float32">67.0</value>
</map>
The currently proposed serialization is poor because it's not easy or 
efficient to find specific values you care about by key; it's also 
inefficient because it requires any new data types to update the XSD 
schema, because each type is a tag name.

! Machine Parsable Descriptions

It is likely that the language describing the serialization structures 
will actually be read by machines at runtime, rather than used to 
generate code. This is definitely true for things like proxy servers and 
systems that treat entities as "generic collections of data" but want to 
have some understanding of the underlying data for display, translation, 
or other processing. While it would be possible to write a parser for 
the current LLSD syntax, introducing a new language syntax into the 
world doesn't seem warranted. I propose that the data structure 
definition language be moved to some XML schema.


Sincerely,

jw