Re: [core] Designs to resolve streaming issues in SenML

[Michael Koster <[e-mail]>]

Note: A reference implementation (in progress) of the Hypermedia Collection described in https://datatracker.ietf.org/doc/draft-ietf-core-interfaces/ <https://datatracker.ietf.org/doc/draft-ietf-core-interfaces/> uses the collection+senml content type mentioned below and is available at https://github.com/mjkoster/MachineHypermediaToolkit <https://github.com/mjkoster/MachineHypermediaToolkit> for an example of the use case.

> On Jan 14, 2016, at 6:43 PM, Michael Koster <[e-mail]> wrote:
> 
> Hi,
> 
> I have been working with JSON serializations of senml and link data lately and I have some comments on the new SenML draft. I haven’t kept up with the format changes because I have something already working that I like that is based on the senml core wg draft-01. 
> 
> There are two problems with the new formats (-02 and -04) for my use case. The first is that I have added more element types to extend senml to represent collections that may have links, forms, and data items. The second is the issue of mapping vs. streaming as discussed previously.  First, I’ll try to explain the issue as I see it with mapping vs. streaming.
> 
> This is how the “collection” example appears in section 6.1.4 of draft-04:
> 
>    [{"bn": "http://[2001:db8::2]/ <http://[2001:db8::2]/>",
>      "bt": 1320078429,
>      "ver": 3},
>       { "n": "temperature", "v": 27.2, "u": "Cel" },
>       { "n": "humidity", "v": 80, "u": "%RH" }
>    ]
> 
> The block formatting of the first map and addition of whitespace to the second 2 maps gives the visual impression that there is some hierarchy. The reformatted example below using standard json pretty print makes it easier to see how this parses into a data structure using standard json tools. I’d recommend following this format to make the examples consistent and easy to read. I will do the same in my drafts.
> 
> [
>   {
>     "bn": "http://[2001:db8::2]/ <http://[2001:db8::2]/>",
>     "bt": 1320078429,
>     "ver": 3
>   },
>   {
>     "n": "temperature",
>     "u": "Cel",
>     "v": 27.2
>   },
>   {
>     "n": "humidity",
>     "u": "%RH",
>     "v": 80
>   }
> ]
> As pointed out, the streaming format can only be processed in streaming form. Storing this in memory using a JSON parser does not result in a structure that can be easily sorted or indexed by base element. If I were to design a parser for this, it would have a state machine that would consume the top level array elements in sequence and create an in-memory structure that looks like the structures described in draft-02:
> 
> [
>   {
>     "bn": "http://[2001:db8::2]/ <http://[2001:db8::2]/>",
>     "bt": 1320078429,
>     "ver": 1
>   },
>   [
>     {
>       "n": "temperature",
>       "u": "Cel",
>       "v": 27.2
>     },
>     {
>       "n": "humidity",
>       "u": "%RH",
>       "v": 80
>     }
>   ]
> ]
> Or draft-01:
> 
> {
>   "bn": "http://[2001:db8::2]/ <http://[2001:db8::2]/>",
>   "bt": 1320078429,
>   "ver": 1
>   "e": [
>     {
>       "n": "temperature",
>       "u": "Cel",
>       "v": 27.2
>     },
>     {
>       "n": "humidity",
>       "u": "%RH",
>       "v": 80
>     }
>   ],
> }
> With these earlier structures I could use standard JSON parse tools and then selection algorithms that match the “bn” value and select the contents within that item’s scope (for patching, etc.). With the streaming formats I need to add another level of serialize/deserialize between the content format and the actual program data model and process elements in order.
> 
> IOW, I already have a JSON parser and a JSON data structure filter that work together. Now with this format change, I need to add another layer of deserializing to create an index or remap the base elements.
> 
> The other issue is that in draft-01 the element tag “e” maps to the XML serialization <e/> tags nicely. I have tools that process both XML and JSON that understand this. Also, this is easily extensible to add more element classes to the senml data model, which I have done of links and forms.
> 
> {
>   "bn": "/",
>   "e": [
>     {
>       "n": "test",
>       "sv": "testValue"
>     }
>   ],
>   "l": [
>     {
>       "href": "",
>       "rel": [
>         "self"
>       ]
>     },
>     {
>       "href": "test",
>       "rel": "sub"
>     }
>   ]
> }
> 
> I have indicated this with a content format “collection+senml+json”, to indicate the senml data model with a collection extension or subtype.
> 
> Since the recent changes in senml, I have considered forking -01 and creating a new set of media types called hyper-senml, or “hsml”, so “application/hsml+json”, “application/hsml+xml”, “application/collection+hsml+json”, etc.
> 
> I guess my preferred outcome would be a senml optimized for mapping that looks like -01 and can be extended to add element classes, and a separate, streaming optimized format.
> 
> Best regards,
> 
> Michael
> 
> 
>> On Jan 13, 2016, at 5:57 AM, Cullen Jennings (fluffy) <[e-mail] <mailto:[e-mail]>> wrote:
>> 
>> The changes we made to SenML to support streaming in the -02 version raised many issues for lots of people. In the end, one of the biggest problems is it forces all things receiving SenML to support streaming. 
>> 
>> I have an alternative design for streaming that I think addresses the issues raised and I would like the WG to think about two alternative designs for SenML and give me some direction on which one to use. 
>> 
>> The first design is the current -02 design with simple fixes and is in draft draft-jennings-core-senml-03
>> 
>> The second design is the new design I am proposing and is in draft-jennings-core-senml-04. It many way it moves back to be much closer to what was in -01 than what is in -02. 
>> 
>> The drafts can be found at 
>> 
>> https://datatracker.ietf.org/doc/draft-jennings-core-senml/03/ <https://datatracker.ietf.org/doc/draft-jennings-core-senml/03/>
>> 
>> and 
>> 
>> https://datatracker.ietf.org/doc/draft-jennings-core-senml/04/
>> 
>> 
>> Practically speaking the differences are not that much between the two but they do resolve a bunch of the issue. Let me discuss this in terms of the JSON representation and the rest of the representations mirror that.  In the new design measurements are put in JSON object as always. The object are put in array to represent a series of measurements. The objects can also have the base values right in the object and the base values apply to all the things in the object and any future objects in the array. 
>> 
>> Here is an example of something that is not a stream in both old and new design.
>> 
>> OLD 
>> 
>> [{"bn": "urn:dev:mac:0024befffe804ff1/"},
>> [ { "n": "voltage", "t": 0, "u": "V", "v": 120.1 },
>>   { "n": "current", "t": 0, "u": "A", "v": 1.2 } ]
>> ]
>> 
>> NEW
>> 
>> [{"bn": "urn:dev:mac:0024befffe804ff1/"},
>> { "n": "voltage", "t": 0, "u": "V", "v": 120.1 },
>> { "n": "current", "t": 0, "u": "A", "v": 1.2 } 
>> ]
>> 
>> Note the old design has an array with a base object, followed by a second array of measurements. The new design just has an single array and the first object happens to only have base values that apply to future object in the array. 
>> 
>> This could also be sent as
>> 
>> [
>>    {
>> 	"t": 0, 
>>      	"n": "voltage", 
>> 	"u": "V", 
>>     	"bn": "urn:dev:mac:0024befffe804ff1/",
>> 	"v": 120.1
>>   },
>>   { 
>> 	"n": "current", 
>> 	"t": 0, 
>> 	"u": "A", 
>> 	"v": 1.2 
>>   } 
>> ]
>> 
>> Here is an example of streaming with old and new design 
>> 
>> OLD 
>> 
>> [{"bn": "http://[2001:db8::1]",
>>  "bt": 1320067464,
>>  "bu": "%RH"},
>> [ { "v": 21.2, "t": 0 },
>>   { "v": 21.3, "t": 10 },
>>   { "v": 21.4, "t": 20 },
>>   { "v": 21.4, "t": 30 },
>>   { "v": 21.5, "t": 40 },
>>   { "v": 21.5, "t": 50 },
>>   { "v": 21.5, "t": 60 },
>>   { "v": 21.6, "t": 70 },
>>   { "v": 21.7, "t": 80 },
>>   { "v": 21.5, "t": 90 },
>> ...
>> 
>> NEW 
>> 
>> [ {"bn": "http://[2001:db8::1]",
>>  "bt": 1320067464,
>>  "bu": "%RH"},
>>   { "v": 21.2, "t": 0 },
>>   { "v": 21.3, "t": 10 },
>>   { "v": 21.4, "t": 20 },
>>   { "v": 21.4, "t": 30 },
>>   { "v": 21.5, "t": 40 },
>>   { "v": 21.5, "t": 50 },
>>   { "v": 21.5, "t": 60 },
>>   { "v": 21.6, "t": 70 },
>>   { "v": 21.7, "t": 80 },
>>   { "v": 21.5, "t": 90 },
>> ...
>> 
>> 
>> The draft does not have it yet but an applications that supports receiving streaming is written is a substantially different way that one that does not. I think the best way to deal with this is simply use a different mine type for streaming ( calls it senml_stream for now but send me suggestions on name for it). That way it can be clearly indicated and negotiated in existing transport protocols. 
>> 
>> Appreciate any thoughts people have on pro / cons of the -03 design vs -04 design.
>> 
>> 
>> 
>> 
>> _______________________________________________
>> core mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/core
>