Re: [mmox] Compact Binary Serialization

Dan Olivares <dcolivares@gmail.com> Wed, 25 February 2009 06:27 UTC

Return-Path: <dcolivares@gmail.com>
X-Original-To: mmox@core3.amsl.com
Delivered-To: mmox@core3.amsl.com
Received: from localhost (localhost [127.0.0.1]) by core3.amsl.com (Postfix) with ESMTP id B9A0C3A6919 for <mmox@core3.amsl.com>; Tue, 24 Feb 2009 22:27:04 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.14
X-Spam-Level:
X-Spam-Status: No, score=-1.14 tagged_above=-999 required=5 tests=[AWL=-1.307, BAYES_00=-2.599, FF_IHOPE_YOU_SINK=2.166, J_CHICKENPOX_61=0.6]
Received: from mail.ietf.org ([64.170.98.32]) by localhost (core3.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id iczaw7M5vjEO for <mmox@core3.amsl.com>; Tue, 24 Feb 2009 22:27:03 -0800 (PST)
Received: from mail-fx0-f167.google.com (mail-fx0-f167.google.com [209.85.220.167]) by core3.amsl.com (Postfix) with ESMTP id A950B3A68A5 for <mmox@ietf.org>; Tue, 24 Feb 2009 22:27:02 -0800 (PST)
Received: by fxm11 with SMTP id 11so3461481fxm.13 for <mmox@ietf.org>; Tue, 24 Feb 2009 22:27:21 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=H9RrZJT/J2qRNbttyx7/sEkAqoLU1dMOEcF1K+verLo=; b=ni4ZxpQYguJXw3KKB9umE53gmAxt2doeXthRyFPz/JVKehWcKhpUBbMZWAkhp8S/DU QfZcUTdTEq1LtKTtjcdaCMGIsR2TMMo94sPLfnKKAGyZl9oq5SlmsIJXwqJQ2LLJZ2jL 5P3dTb56BdCdK/JLRiks+TSVY7DMVmGdMsZVY=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=cOujEUZS6kYRiZ094mPz4AsLU0sANZjkU81T1XIsjKnDkQqi72JNFObWtD1ILsmVoZ U5TY4jPWbKkijCQJ9xp6KL+n/xGwlThSV1VPyumQXnCKGx5YWeMQ3tGXznDGBEDyi4NM aqnaQlQS6OanJbWdSUBkUARKfyrCKsgRfBJ6U=
MIME-Version: 1.0
Received: by 10.103.244.4 with SMTP id w4mr160983mur.90.1235543237965; Tue, 24 Feb 2009 22:27:17 -0800 (PST)
In-Reply-To: <49A4563A.5060008@gmail.com>
References: <ebe4d1860902240901oaff2260o65c4337598a14fc5@mail.gmail.com> <49A4563A.5060008@gmail.com>
Date: Wed, 25 Feb 2009 01:27:17 -0500
Message-ID: <a768bcd90902242227i8edbfe4x26442349c99988e5@mail.gmail.com>
From: Dan Olivares <dcolivares@gmail.com>
To: Jon Watte <jwatte@gmail.com>
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: quoted-printable
Cc: mmox@ietf.org
Subject: Re: [mmox] Compact Binary Serialization
X-BeenThere: mmox@ietf.org
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: Massively Multi-participant Online Games and Applications <mmox.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/listinfo/mmox>, <mailto:mmox-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www.ietf.org/mail-archive/web/mmox>
List-Post: <mailto:mmox@ietf.org>
List-Help: <mailto:mmox-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/mmox>, <mailto:mmox-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 25 Feb 2009 06:27:04 -0000

Hmm,

Just one thing to point out here;

<colour>127,255,127</colour>
<vector>100.0,75.3,28.6</vector>
<quat>0.5,-0.5,0.5,0.5</quat>

might seem ok, however, consider what happens when someone in a locale
with a different way of displaying the number tries to figure out what
this means?

In the United States, the following number format applies:  (-)123,456,789.00
In Germany, the following number format applies: 123.456.789,00
In Russia and Finland, the following number format applies: 123 456 789,00

This, in my opinion, is the case for separating them with explicit
separations within the syntax using non-ambiguious terms
Is <colour>127,255,127</colour>, one hundred twenty seven million two
hundred fifty five thousand one hundred twenty seven?
Is <vector>100.0,75.3,28.6</vector> one hundred thousand and seventy
five tenths?

Best Regards

Daniel Olivares


On Tue, Feb 24, 2009 at 3:19 PM, Jon Watte <jwatte@gmail.com> wrote:
> Catherine Pfeffer wrote:
>>
>> (in the order x, y, z, s and not math's order s, x, y, z, that's one of
>> Second Life's strangenesses)
>>
>
> They are in good hands. Direct3D, OLIVE, and a lot of other 3D systems use
> XYZW. It's a bit like the question of row vectors on the left (D3D) or
> column vectors on the right (OpenGL). There are two ways to do it, and the
> world is split about equally.
>
>
>> LLSD (simplifying):
>>     LLIDL says: nose orientation is an array of four fp32s
>>     LLSD data over the wire: <floats>0.5,-0.5,0.5,0.5</floats>
>>
>
> Why do you need to send "floats" at all?
>
>>
>> I see this as another example of "Yeah, that's strange XML, but it maps
>> 1:1 to a nice and natural binary serialization" :-(.
>>
>
> I think the binary serialization in LLSD is too verbose. My suggestion was
> to allow the SD language to express type information to the point where you
> can assume the data types from the context. For example, to update
> properties on an object, you would need to send, in binary, something like:
>
> <object-id> <number-of-updates>
> (<property-id> <data-for-property>) *
>
> "number-of-updates" is a small integer, so will typically fit in a byte.
> "property-id" is a small integer in the context of "object-id" and thus will
> typically take one byte. "data-for-property" for a quaternion stored as four
> fp16 is 8 bytes. Thus, the actual storage for this one property (in the
> context of where it's used) is 9 bytes; the total message size (assuming
> there is only one property value) is 10 bytes plus the size of an object
> identifier.
>
> Note that "data-for-property" is ONLY the raw data, not with any indication
> of what that data is. The schema for the entity already describes what that
> data is, so you don't need to send that with every binary packet you send.
> Thus, in that form of binary serialization, you optimize for transmission
> size, which is important for interactive, real-time transmissions. When you
> want a more "resilient" format, and size doesn't matter, you simply use XML
> instead.
>
> To clarify: object of id <object-id> has some type. That type has some
> schema, that maps small-integer property-id values to string-name and
> data-type. Typically, when you encounter an object, you will understand the
> type/structure of that object once, but you will see changes to properties
> many times. And, in systems which have fewer types than objects (which is
> most systems except for SL, I think), you have additional savings by only
> needing to encounter a type once, and then that work is re-used for each
> object of the same type.
>
> In this context, the schema for the object type would be described in some
> data description language. That language could perhaps be one area where
> something like LLSD would make sense -- but LLSD, as it stands, is not up to
> the task IMO. Hence, my suggestions (one of which is to support external
> schema for transfer encoding efficiency).
>
> Btw: if you know that you will have the schema when you
> serialize/de-serialize objects, then the "more natural" XML encoding would
> also map easily to the internal binary format, because all the data you need
> about whether things are arrays or ints or whatnot is available through the
> schema, so the more natural "<nose-orientation>1,2,3,4</nose-orientation>"
> XML serialization would also be possible (and, IMO, desirable).
>
>> Now back to your question: "should we move the type information to the
>> schema, as all people do in XML?". The answer is probably: "no, because we
>> want to keep a simple 1:1 mapping with the binary serialization".
>>
>
> But my point is that the binary serialization does not serve the needs of
> what a binary serialization should do, which is save transmission space. It
> contains both key and datatype, where the verbose key name is typically
> inlined instead of sent by reference-to-external-schema. If you can describe
> things with external schema, then you can save a LOT of transmission space
> where it matters, which is in the binary serialization.
>
> Sincerely,
>
> jw
>
> _______________________________________________
> mmox mailing list
> mmox@ietf.org
> https://www.ietf.org/mailman/listinfo/mmox
>