Re: [ogpx] type-system : guidance for binary serialization implementers

[Dahlia Trimble <[e-mail]>]

OpenSim in general uses the serializer and deserializer classes in
libOpenmetaverse (1). There may be a few lingering places in the code base
where they are hard coded with other methods, but I suspect they would be
converted to using the libOpenmetaverse implementation rather than be
modified to conform to any spec modifications.


1. http://www.openmetaverse.org/projects/libopenmetaverse

<http://www.openmetaverse.org/projects/libopenmetaverse>
On Sun, Mar 28, 2010 at 10:34 AM, Meadhbh Hamrick <[e-mail]>wrote:

> so this has stuck in my craw for a couple weeks, finally remembering
> to post it to the list.
>
> the JSON and XML serialization schemes use well known grammars that
> have wide adoption. not so for the binary serialization. so if i
> wanted to write an XML parser, i would have a lot of 3rd party
> resources and discussion to guide me. specifically when it comes to
> handling errors.
>
> as i was writing my own implementation of the LLSD binary
> serialization scheme, a couple issues came up.
>
> i think we should *somewhere* address them. either as a section of the
> type-system document, or as an informational RFC. i would like to have
> clear guidance for how implementers would handle the following
> exceptional events.
>
> one of the things we were trying to do was make a type system and
> serialization schemes that were resilient in the face of error. so i
> think there's a strong preference that we recover as best we can from
> an error instead of throwing up our hands and just saying "error!"
>
> i would actually go and look at how LL implemented the binary
> serialization (i'm sure it's in the viewer code somewhere) but i'm in
> the middle of my gnu/bsd cool down period, and i don't want to reset
> the timer. i think there may be other people in the same situation,
> which is why we should have a document describing what we should do in
> the face of parsing errors.
>
> so... if someone out there who's more of a viewer person wants to look
> at the linden implementation and describe it's behavior in a textual
> format, and then someone wants to look at the opensim implementation
> and describe it's behavior in a textual format, we could then document
> how things work and avoid this situation for other people.
>
> here's a list of encoding / decoding errors i think about, along with
> suggestions for how to handle them.
>
> 1. the object count following the opening tag ('{' or '[') is less
> than the actual number of objects in the collection. (that is, the
> closing tag is not found after iterating through "object count" number
> of elements, but is found later in the stream.)
>
> the "simple" thing to do would be to simply say, "the array ends after
> $(OBJECT_COUNT) number of elements, even if there's not a closing
> tag."
>
> the "smart" thing to do would be to look at the LLIDL definition of
> the array you're parsing and try to come up with a "best fit" for
> what's supposed to be there vs. what's actually there. in other words,
> you look at the LLIDL and if it says you're supposed to have three
> elements in the array and you have three elements in the array, then
> you're done. or maybe you're supposed to have one additional element
> in the array.
>
> the "smart" thing to do could get quite complicated while the "simple"
> thing is sure to introduce failures the "smart" thing could recover
> from.
>
> 2. the object count following the opening tag ('{' or '[') is greater
> than the actual number of objects in the collection. (that is, you get
> a closing tag before "object count" number of elements)
>
> the "simple" thing to do is to simply say, "a closing tag ends the
> collection"
>
> combining this with the previous error case, we could say, "a
> collection continues until the closing tag is reached or
> $(OBJECT_COUNT) elements are found in the collection."
>
> 3. there is no closing tag. (that is, after "object count" number of
> elements, you find a tag that is not a '}' or a ']'.)
>
> this is more or less the same as error case number 1 above.
>
> though i wonder if it makes sense to differentiate the two cases. if
> we were, we would have to have the smarts enough to look forward in
> the stream, count the number of opening tags, then count the number of
> closing tags and see if they're unbalanced. ugh.
>
> 4. lack of 'k' elements in a map. (that is, you haven't reached the
> "object count" number of elements and are expecting a 'k' but get
> something else.)
>
> my druthers would be to say, if you encounter a type that can be
> converted to a string, you just do the conversion and use the result
> as the key.
>
> another way to handle it would be to ignore it.
>
> 5. you encounter a bare 'k'. (that is, you encounter a 'k' tag outside
> the context of a map.)
>
> i vote for "convert to string"
>
> 6. duplicate keys in a map. (actually this applies to all serializations)
>
> i vote for "the last key in the map with the same name wins." that is,
> if you have two or more keys with the same name, the one found
> furthest in the stream is the one that's used.
>
> ideas? comments?
> --
> meadhbh hamrick * it's pronounced "maeve"
> @OhMeadhbh * http://meadhbh.org/ * [e-mail]
> _______________________________________________
> ogpx mailing list (VWRAP working group)
> [e-mail]
> https://www.ietf.org/mailman/listinfo/ogpx
>