[Forces-protocol] Re: Data encoding -- first part

Zsolt Haraszti <zsolt@modularnet.com> Mon, 18 October 2004 22:16 UTC

Received: from ietf-mx.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA29957 for <forces-protocol-web-archive@ietf.org>; Mon, 18 Oct 2004 18:16:54 -0400 (EDT)
Received: from megatron.ietf.org ([132.151.6.71]) by ietf-mx.ietf.org with esmtp (Exim 4.33) id 1CJg02-00036w-Tg for forces-protocol-web-archive@ietf.org; Mon, 18 Oct 2004 18:29:26 -0400
Received: from localhost.localdomain ([127.0.0.1] helo=megatron.ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1CJfW4-0004pp-2S; Mon, 18 Oct 2004 17:58:08 -0400
Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by megatron.ietf.org with esmtp (Exim 4.32) id 1CJf9T-0008Tn-Qi for forces-protocol@megatron.ietf.org; Mon, 18 Oct 2004 17:34:47 -0400
Received: from ietf-mx.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA25547 for <forces-protocol@ietf.org>; Mon, 18 Oct 2004 17:34:40 -0400 (EDT)
Received: from [204.85.2.226] (helo=mail.modnetinc.com) by ietf-mx.ietf.org with smtp (Exim 4.33) id 1CJfLT-000266-Co for forces-protocol@ietf.org; Mon, 18 Oct 2004 17:47:11 -0400
Received: (qmail 28177 invoked by uid 530); 18 Oct 2004 21:34:12 -0000
Received: from zsolt@modularnet.com by proteus-01.proteandevices.com by uid 527 with qmail-scanner-1.16 (clamscan: 0.54. Clear:. Processed in 0.823526 secs); 18 Oct 2004 21:34:12 -0000
Received: from proteus-02.proteandevices.com (HELO ?127.0.0.1?) (192.168.0.202) by 0 with SMTP; 18 Oct 2004 21:34:11 -0000
From: Zsolt Haraszti <zsolt@modularnet.com>
To: "Joel M. Halpern" <jhalpern@megisto.com>
In-Reply-To: <5.1.0.14.0.20041018165715.03523890@mail.megisto.com>
References: <41741D78.4070205@idt.com> <468F3FDA28AA87429AD807992E22D07E025791E5@orsmsx408> <002d01c4b50b$1ecc9c10$020aa8c0@wwm1> <1098102734.1042.134.camel@jzny.localdomain> <1098113089.2364.12.camel@localhost.localdomain> <1098115003.2884.67.camel@localhost.localdomain> <4173FB88.1000008@idt.com> <1098126011.2884.162.camel@localhost.localdomain> <41741D78.4070205@idt.com> <5.1.0.14.0.20041018165715.03523890@mail.megisto.com>
Content-Type: text/plain
Organization: Modular Networks, Inc.
Message-Id: <1098135051.2884.302.camel@localhost.localdomain>
Mime-Version: 1.0
X-Mailer: Ximian Evolution 1.4.6 (1.4.6-2)
Date: Mon, 18 Oct 2004 17:30:51 -0400
Content-Transfer-Encoding: 7bit
X-Spam-Score: 0.0 (/)
X-Scan-Signature: 39bd8f8cbb76cae18b7e23f7cf6b2b9f
Content-Transfer-Encoding: 7bit
Cc: "Khosravi, Hormuzd M" <hormuzd.m.khosravi@intel.com>, ram.gopal@nokia.com, "Yang, Lily L" <lily.l.yang@intel.com>, "Steven Blake (petri-meat)" <slblake@petri-meat.com>, forces-protocol@ietf.org, Jamal Hadi Salim <hadi@znyx.com>, Alan DeKok <alan.dekok@idt.com>, Ellen M Deleganes <ellen.m.deleganes@intel.com>, Weiming Wang <wmwang@mail.hzic.edu.cn>
Subject: [Forces-protocol] Re: Data encoding -- first part
X-BeenThere: forces-protocol@ietf.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: forces-protocol <forces-protocol.ietf.org>
List-Unsubscribe: <https://www1.ietf.org/mailman/listinfo/forces-protocol>, <mailto:forces-protocol-request@ietf.org?subject=unsubscribe>
List-Archive: <http://www1.ietf.org/pipermail/forces-protocol>
List-Post: <mailto:forces-protocol@ietf.org>
List-Help: <mailto:forces-protocol-request@ietf.org?subject=help>
List-Subscribe: <https://www1.ietf.org/mailman/listinfo/forces-protocol>, <mailto:forces-protocol-request@ietf.org?subject=subscribe>
Sender: forces-protocol-bounces@ietf.org
Errors-To: forces-protocol-bounces@ietf.org
X-Spam-Score: 0.0 (/)
X-Scan-Signature: a7d6aff76b15f3f56fcb94490e1052e4
Content-Transfer-Encoding: 7bit

On Mon, 2004-10-18 at 16:58, Joel M. Halpern wrote:
> Why not just declare that structs start with their length.  Then we can 
> declare that an array consists of its element count (or length and element 
> count) followed by the representation of its contents.  [Ignoring for the 
> moment the issue of dense vs sparse packing, subscript representation, etc.]
> 

That's because when the struct is of fixed size, it is really not
needed and completely destroys even the possibility that you can
memcpy() a large array of these structs.

We may have put too much stress on the efficiency design goal.

But as we are talking to customers and partners, they consistent
concern is efficiency.

Therefore we decided that we try to design the data transfer such
that it scales 1:1 with the data size for large data elements, i.e.,
tables.  If your table is stored on X MBs in memory, it should ideally
require X MBs to transfer it over the wire.

/zsolt

> Yours,
> Joel
> 
> At 04:45 PM 10/18/2004 -0400, Zsolt Haraszti wrote:
> >This is another trade-off issue.  You are absolutely right, inserting
> >the size info in front of each struct can be very useful, especially if
> >it is a variable size struct.  We will have this in ARRAYs, not
> >part of the struct itself, but part of the ARRAY encoding.
> >That does not help when the STRUCT is embedded in another struct though.
-- 
Zsolt Haraszti                Phone:  +1 919-765-0027/2017
Modular Networks              Mobile:      +1 919-522-2337
                              Email:  zsolt@modularnet.com


_______________________________________________
Forces-protocol mailing list
Forces-protocol@ietf.org
https://www1.ietf.org/mailman/listinfo/forces-protocol