Re: [vwrap] The <embed> tag... is the group still interested in LLSD or DSD?

[Morgaine <[e-mail]>]

On Fri, May 6, 2011 at 7:46 AM, Vaughn Deluca <[e-mail]>wrote:

> Morgaine,
>
>  Your proposal sounds good, and  a lot more workable than starting from
> scratch.  However you wrote earlier:
>
> > Indeed, in the early days of our IETF effort, we spent months discussing
> the
> > width of integers in LLSD because there was only one width allowed and it
> was
> > hardwired in the spec.
>
> What were the reasons to allow onl a single integer type? There must have
> been a good arguments for that?
>
>
Joshua has replied about the history of it, but I'd like to reply about its
future.

When you create an ADT system and expect it to have multiple language
bindings, then you should not base your ADT's set of primitive types on your
initial choice of languages, because that introduces several very poor
consequences:


   - It reduces your set of primitive types to the lowest common denominator
   of types directly supported by your initial set of languages, and hence
   makes your ADT system a very narrow one, the opposite of "flexible type
   system".


   - It makes those initial language bindings "special" because they have
   been given special consideration, whereas further language bindings won't
   enjoy similar support as design targets.  That's not designing for the
   future.


   - It makes application developers repeatedly pay the penalty for not
   having specific types directly suited to their application, a continuing
   cost.


   - It places the burden of type poverty on all language bindings instead
   of only on the language that is type-inadequate.  Better languages have to
   suffer the penalty of an inferior one in their midst.


   - The lack of the shortest integer types impacts on binary serializations
   especially severely.  While all serializations are rendered less efficient
   when they are forced to use a single generic integer type, it is the binary
   serialization that is impacted most.  This is both unhelpful and ironic,
   since it is the binary serialization that is chosen when you want the most
   efficiency.  It is the worst possible outcome that the binary serialization
   is affected worst.


   - By requiring your most efficient language bindings (C/C++) to use fixed
   length integers instead of those that they employ internally, it reduces
   their efficiency disproportionately every time they cross the binding
   boundary.  Less efficient languages won't notice any slowdown because the
   loss of binding efficiency is masked by their overheads, whereas C/C++ have
   virtually no overheads and so are impacted most.  This is the wrong way
   around, namely designing to benefit the least efficient use case.



So given quite a number of negative consequences, what's the actual benefit
of taking the design approach described by Joshua?  Well allegedly it's so
that the initial set of languages can all represent the primitive data types
of the ADT directly, but I dispute that that goal has any merit in practice.

First of all, tying your design to the type and representational poverty of
Python and ECMAScript misses the point that your most efficient scalable
server systems are not likely to be written in them.  Clients may be written
in less efficient languages, but inefficiencies in clients don't impact on
system scalability because all those inefficiencies occur in parallel.  It's
the efficiency of server side that matters far more when designing efficient
scalable systems, so server side requirements should be considered first.
(Then bandwidth requirements next, and client requirements last.)

Secondly, it makes no sense to limit the primitive types transported by a
protocol to those which slow languages like Python and ECMAScript can
represent, because the overheads inherent in those languages vastly outweigh
any impact from protocol-language mismatch.  What's more, those languages
are not prevented from using their internal representations just because a
protocol provides shorter integers for efficiency, so it is not a valid
argument that supporting Python and ECMAScript precludes using shorter types
in the protocol.  That's just incorrect.

And thirdly, having to use a single size of integer for all purposes
introduces semantic problems which we alluded to quite recently when
discussing Service Level Semantic Interoperability --
http://www.ietf.org/mail-archive/web/vwrap/current/msg00657.html .  It's a
very bad approach at the best of times, in effect a loss of type information
for everything that is integral but is not 32 bits in length.  It results in
additional value bounds checking in the application to eliminate
inappropriate values arriving within the single-length protocol field, since
the protocol provides no direct support for any integer semantic other than
signed 32-bit.  This is bad.

I really can't see any valid reason why a protocol's primitive types would
be chosen on the basis that's been described.  It's just not a good
engineering basis.




> Up til now i saw suggestions for adding more integer types and for adding a
> stream buffer type.
>
I think we should aim for keeping the set of additions as small as possible.
>
You suggest to add "all the usual integer types and maybe others", what
> would those other types be?
>


Mostly just all the integer types, because they are the key to efficiency on
the wire and in language bindings, and they also improve the semantic
clarity.

There is only one other type missing that results in significant and common
inefficiencies, and that is bit arrays.  They are used not only for actual
bit-oriented bulk data but more frequently for carrying packed boolean
fields, like say "Restricted" and other metadata bits that we discussed
recently, and many other uses.  We're very likely to see multiple booleans
transmitted in messages, so lack of bit packing would be unfortunate for the
efficiency of our boolean serializations.  However ...

It's so hard to get any change at all agreed here that I'll waive on the bit
arrays or bit packing and just try to push the full range of integer fields.




> Are you thinking of Dzonatas' his stream buffer type, or something else?
>


"Stream buffer type" made no sense to me if its name accurately described
what it did.  All protocol values are immutable by definition, and hence a
field in a protocol message cannot be a buffer for anything at all.  Maybe
something else was intended, but it was never described to us so I can't
comment on it further.



> The version negotiation system seems to be orthogonal to adding extra
> types to LLSD, i think we should  try and reach consensus on the data types
> first. It indeed looks like this can be done with very little pain or delay.
>


Yes, it's nicely orthogonal, as long as we number our protocol as VWRAP-1 or
something like that.  And yes, it could be done with very little pain or
delay .... but it won't be.  Just watch. :DDDDDDDDDDDDD


Morgaine.




========================

On Fri, May 6, 2011 at 7:46 AM, Vaughn Deluca <[e-mail]>wrote:

> Morgaine, vwrap,
>
>  Your proposal sounds good, and  a lot more workable than starting from
> scratch.  However you wrote earlier:
>
> > Indeed, in the early days of our IETF effort, we spent months discussing
> the
> > width of integers in LLSD because there was only one width allowed and it
> was
> > hardwired in the spec.
>
> What were the reasons to allow onl a single integer type? There must have
> been a good arguments for that?
>
> Up til now i saw suggestions for adding more integer types and for adding a
> stream buffer type.
> I think we should aim for keeping the set of additions as small as
> possible.
> You suggest to add "all the usual integer types and maybe others", what
> would those other types be?
> Are you thinking of Dzonatas' his stream buffer type, or something else?
>
> The version negotiation system seems to be orthogonal to adding extra
> types to LLSD, i think we should  try and reach consensus on the data types
> first. It indeed looks like this can be done with very little pain or delay.
>
> -- Vaughn
>
>
> On Thu, May 5, 2011 at 4:33 PM, Morgaine <[e-mail]>wrote:
>
>> On Thu, May 5, 2011 at 7:32 AM, Vaughn Deluca <[e-mail]>wrote:
>>
>>>
>>> So what do you suggest is the way forward?
>>>
>>
>>
>> I suggest the following as a way forward with very little pain or delay:
>>
>>
>>    - As other serialization systems have done, add those predefined types
>>    that we can easily foresee will be very beneficial to VW applications.  That
>>    certainly includes all the usual integer types, maybe others.  We could
>>    agree on a useful set of them today, it's not hard.  It wouldn't be perfect,
>>    but it would be "our best shot" at supporting what the immediate future
>>    requires.  It would give our types system added *flexibility*, and its
>>    extra *efficiency* would improve the performance of our binary
>>    serializations.
>>
>>
>>    - As Carlo has suggested in the past, add a version negotiation system
>>    so that when VW projects find our 2011 set of base types inadequate, they
>>    can evolve the types system and negotiate use of a newer one dynamically at
>>    protocol initiation time, without needing to come back to the IETF.  This
>>    would give us a measure of *extensibility* without needing to make our
>>    initial elementary types extensible.
>>
>>
>> - Can LLSD itself be made extensible?
>>>
>>>
>> It can be made extensible, yes, but with this group's track record of
>> continually saying "No" even to simple technical advances, I would not want
>> to be an advocate for more complex extensible base types.  Instead, I'll
>> make do with Carlo's approach, ie. extensibility through negotiating
>> alternative versions.
>>
>>
>>>
>>> I am all for rigorous design, but i think i prefer actual implementation
>>> of an imperfect system over sitting and waiting for an ultimate dream.
>>> If we don't get demonstrated commitment for specifying (and
>>> implementing!) an alternative, my vote go's to sticking with LLSD, if only
>>> to show conclusively were it falls short of our needs.
>>>
>>>
>> Which is why I suggest that we do no more than add those types that we can
>> envisage being needed in designs *today*, without speculating about
>> tomorrow.  Such flexibility is the least we can do while pretending to be
>> "designing for the future".  If we go forward with a design that *WE KNOW
>> FULL WELL* is inflexible and inefficient through providing only a single
>> integer type, then we cannot pretend to be designing for the future at all,
>> it's designing for a specific application today.
>>
>>
>>>
>>> I am all for rigorous design, but i think i prefer actual implementation
>>> of an imperfect system over sitting and waiting for an ultimate dream.
>>> If we don't get demonstrated commitment for specifying (and
>>> implementing!) an alternative, my vote go's to sticking with LLSD, if only
>>> to show conclusively were it falls short of our needs.
>>>
>>
>>
>> There is no point going through all this IETF effort just to produce a
>> protocol that we know is designed for today, without even a nod for the
>> future.  It will be obsolete before it's released.
>>
>> If we did so in ignorance, fine, we would deserve to be ignored by future
>> VW designers who will find our design skills so lacking in future vision.
>> But we would not be doing so in ignorance.  Others in this group have
>> commented on the lack of future proofing in LLSD too, not just myself.  We
>> would be rolling out a poor types system with our eyes wide open, despite
>> being made aware of its lack of flexibility and extensibility.  Future VW
>> designers won't have any sympathy at all, we will be laughed at and
>> condemned as knowingly regressive.  And they would be right.
>>
>> I do not believe that this group is so technically incompetent that we
>> can't produce a types system with a measure of flexibility and extensibility
>> (through negotiation at least).  What we do clearly suffer from is extreme
>> inertia and resistance from some participants with a regressive outlook on
>> protocol design, so much so that you're suggesting that we move forward with
>> a known inflexible system just because of our people problems here.
>>
>> Well I can't help that.  I've been doing my bit for pulling the cart into
>> the future, despite intense opposition.  That's how engineering advances are
>> made, by always designing for tomorrow.  I wish a few more people here would
>> help with that.
>>
>>
>> Morgaine.
>>
>>
>>
>> ========================
>>
>>
>> On Thu, May 5, 2011 at 7:32 AM, Vaughn Deluca <[e-mail]>wrote:
>>
>>> Morgaine,
>>>
>>> You observe LLSD is not extendable. Dzonatas argues that the this can be
>>> solved at a higher level using XML.  You object that this  entangles the
>>> serialisation with basic ADT
>>>
>>> So what do you suggest is the way forward?
>>> - Can LLSD itself be made extensible?
>>> -if not, do we have enough commitment in the group to start from scratch?
>>> -if not, do the benefits of an absolutely clean engineering solution
>>> outweighs the costs of disrupting the already slow progress here? We reach
>>> for the sky, but run the risk of over extending our current capabilities.
>>>
>>> I am all for rigorous design, but i think i prefer actual implementation
>>> of an imperfect system over sitting and waiting for an ultimate dream.
>>> If we don't get demonstrated commitment for specifying (and
>>> implementing!) an alternative, my vote go's to sticking with LLSD, if only
>>> to show conclusively were it falls short of our needs.
>>>
>>> -- Vaughn
>>>
>>>
>>> On Thu, May 5, 2011 at 12:59 AM, Morgaine <
>>> [e-mail]> wrote:
>>>
>>>> I have no idea what either of the last list two posts meant.
>>>>
>>>> I tried to interpret the preceding post as best I could and to find room
>>>> for technical discussion, but clearly the attempt has failed.
>>>>
>>>> I'll leave it at that.
>>>>
>>>>
>>>> Morgaine.
>>>>
>>>>
>>>>
>>>> ===========================
>>>>
>>>>
>>>>
>>>> On Wed, May 4, 2011 at 11:41 PM, Dzonatas Sol <[e-mail]>wrote:
>>>>
>>>>> On 05/04/2011 03:21 PM, Dzonatas Sol wrote:
>>>>>
>>>>>> Actually, I invented one tag called <embed> that does all the above
>>>>>> already, so how about the predefined embed'ed data type instead of endless
>>>>>> extensibility?
>>>>>>
>>>>>>
>>>>> On that NOTE: any more needed extensibility is transistioned into
>>>>> capabilities.
>>>>>
>>>>> Whereas, capabilities means more resources (as dynamic extensions
>>>>> grow); therefore, this requires minimization of number of connections
>>>>> instead of one connection per resource. Thus, that proportion leads into the
>>>>> reason for combined/bundled queries that I have already mentioned and
>>>>> implemented, as maximum # of connections are too easily reached without such
>>>>> mechanism.
>>>>>
>>>>>
>>>>> --
>>>>> --- https://twitter.com/Dzonatas_Sol ---
>>>>> Web Development, Software Engineering, Virtual Reality, Consultant
>>>>>
>>>>> _______________________________________________
>>>>> vwrap mailing list
>>>>> [e-mail]
>>>>> https://www.ietf.org/mailman/listinfo/vwrap
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> vwrap mailing list
>>>> [e-mail]
>>>> https://www.ietf.org/mailman/listinfo/vwrap
>>>>
>>>>
>>>
>>
>