Re: [core] Designs to resolve streaming issues in SenML

[Michael Koster <[e-mail]>]

Here is a concrete proposal. 

My goal with the collection+senml content format is to create a format for machine hypermedia representations that is roughly analogous to a HTML web page in the web of documents world. 

The idea was a compact representation that can represent machine hypermedia data, links, and forms and would leverage existing formats and processors; link-format, senml, json. 

When I started, it looked like SenML was a good base due to what appeared to be an extension mechanism: the element tag that was an artifact of the XML serialization. I didn’t expect it to be removed because I have XML/JSON tools that programmatically generate that style as a lossless XML-JSON transform. This is generally useful because EXI is still a very popular format and in many cases more compact than CBOR.

If we can allow the base element to retain this functionality, and put the streaming support in the sequence of top level array elements, it seems like exactly what is needed. It retains backward compatibility with current senml implementations and allows some extensibility at both levels, while supporting sequential processing.

I propose to restore the base senml functionality of draft -01 within the base element definition in the new draft, that is to allow measurement data within the base element using the “e” key. To facilitate sequential processing, a topmost structure of array will allow an arbitrary sequence of base element maps and data element arrays as defined in draft -04.

Are there any issues with this construction? 

Best regards,

Michael


> On Jan 16, 2016, at 7:24 AM, Michael Koster <[e-mail]> wrote:
> 
> Hi Christian,
> 
> 1.
> Thanks for the code example! It seems well optimized indeed.
> 
> It also serves to highlight one of the points. 
> 
> It’s more than just a matter of me liking my own favorite language (actually library) internal representation of JSON. Most modern libraries produce dictionary and array representations for parsed JSON that work more or less the same way.
> 
> The optimizations you recommend require a new string input parser, where I am using the JSON parser that comes with the library. My point was that I will need to either make my own string parser for SenML or make a re-parser with the state machine I described in order to use -02 + but there is another issue anyway:
> 
> One of the big advantages of using JSON is connecting the embedded world to the web world. To do this in a low-friction way, it’s good to be able to use well known tools and patterns. Streaming JSON 
> 2. 
> Thanks also for looking into my code. I could easily use the -02 or -03 in the senml processing as you show. What is missing still from everything after -01 is the element tag that I am re-using to indicate links and forms in the document:
> 
> {
>   "bn":"/3300/1/",
>   "e":[
>     {"n":"5700","v":"31.3"},
>     {"n":"5602","v":"37.1"},
>     {"n":"5601","v":"18.3"},
>     {"n":"5605"},
>     {"n":"5603","v":"0"},
>     {"n":"5604","v":"100"},
>     {"n":"5750","sv":"Inboard Bearing"}
>   ],
>   "l":[
>     {"href":"","rel":"self","rt":"temperature","u":"C"},
>     {"href":"5700","rt":"currentValue","u":"C"},
>     {"href":"5602","rt":"maxValue","u":"C"},
>     {"href":"5601","rt":"minValue","u":"C"},
>     {"href":"5605","rt":"resetMaxMin"},
>     {"href":"5603","rt":"minScale","u":"C"},
>     {"href":"5604","rt":"maxScale","u":"C"},
>     {"href":"5750","rt":"appType"}
>   ]
> }
> 
> I will think about other designs that do not depend on entity tags in the representation; maybe there is a better design for the content format in general. But I would still like to use SenML. Maybe not extend it but embed it in another new format…
> 
> In the meantime, I don’t have a good alternative yet.
> 
> Best regards,
> 
> Michael
> 
> PS Can we succinctly describe the use case for ordered serial processing of senml+json elements being important? From reading the draft, it seems like the important bit is saving the buffer memory for large responses. Is it a bigger deal with CBOR encoding or less important. 
> 
> Is the most important consideration being able to know the base values ahead of the data? 
> 
> Also, where is -03? there doesn’t seem to be a version -03 that you all are talking about on the IETF website:
> https://tools.ietf.org/html/draft-jennings-core-senml-04 <https://tools.ietf.org/html/draft-jennings-core-senml-04>
> 
> 
>> On Jan 16, 2016, at 5:27 AM, Christian Amsüss <[e-mail] <mailto:[e-mail]>> wrote:
>> 
>> Hello Cullen,
>> 
>> On Wed, Jan 13, 2016 at 01:57:02PM +0000, Cullen Jennings (fluffy) wrote:
>>> The objects can also have the base values right in the object and the
>>> base values apply to all the things in the object and any future
>>> objects in the array. 
>>> 
>>> Appreciate any thoughts people have on pro / cons of the -03 design vs -04 design.
>> 
>> The -03 draft is parsable with minimum RAM: I've assembled a crude
>> demo[1] that shows that given a list of known integer-valued resources
>> on a device, it is possible to parse incoming SenML-03 with less than 30
>> bytes of state, with support for arbitrary SenML extensions, all legal
>> JSON number representations and full URL concatenation semantics.
>> 
>> With the -04 draft, one would have to allocate a buffer for the "n"
>> value, because before the object finishes, the "bn" might change later
>> in the same object, and change the context in which it needs to be
>> interpreted. This could be remedied by making base and entry updates
>> mutually exclusive (arguably a less straight-forward design), but as
>> things are, I'd prefer -03.
>> 
>> Best regards
>> Christian
>> 
>> [1] https://gitlab.com/c.amsuess-energyharvesting/senml-03-streaming-parser-demo/tree/master <https://gitlab.com/c.amsuess-energyharvesting/senml-03-streaming-parser-demo/tree/master>
>> 
>> -- 
>> Christian Amsüss                      | Energy Harvesting Solutions GmbH
>> founder, system architect             | headquarter:
>> mailto:[e-mail] <mailto:[e-mail]>  | Arbeitergasse 15, A-4400 Steyr
>> tel:[phone]                | http://www.energyharvesting.at/ <http://www.energyharvesting.at/>
>>                                      | ATU68476614
>> _______________________________________________
>> core mailing list
>> [e-mail] <mailto:[e-mail]>
>> https://www.ietf.org/mailman/listinfo/core
>