Re: [core] Designs to resolve streaming issues in SenML

[Michael Koster <[e-mail]>]

Some more thoughts on data model.

It turns out to be more than just convenient that the senml representation 
itself is an explicit mapping of a data model.

the “e” tag is more than just a semantic glitch in converting XML to JSON. 
To preserve the semantics of the data model, the data items as a set are 
related to the base at the same scope level in the model. The XML form
is “flattened” and JSON is not. 

I used this to extend the data model by adding a links class at the same 
scope level as base name and items.

This is how the collection with links and items can be expressed in both
machine and human comprehensible form as a model:

{
"bn":"/3300/1/",
“i":[
{"n":"5700","v":"31.3"},
{"n":"5602","v":"37.1"},
{"n":"5601","v":"18.3"},
{"n":"5605"},
{"n":"5603","v":"0"},
{"n":"5604","v":"100"},
{"n":"5750","sv":"Inboard Bearing"}
],
"l":[
{"href":"","rel":"self","rt":"temperature","u":"C"},
{"href":"5700","rt":"currentValue","u":"C"},
{"href":"5602","rt":"maxValue","u":"C"},
{"href":"5601","rt":"minValue","u":"C"},
{"href":"5605","rt":"resetMaxMin"},
{"href":"5603","rt":"minScale","u":"C"},
{"href":"5604","rt":"maxScale","u":"C"},
{"href":"5750","rt":"appType"}
]
}

Base name, links, and items all are easily mapped to property names
using a simple JSON-LD context and can be understood and modeled 
by higher level tools as-is. Removing the “i” or “l” tags gives the modeling 
tool nothing to use to identify property instances (like base name, links, 
and items). This is extensible by adding new property  classes (element 
tags).

It is very useful to have the representation match the model so closely.
I think there will be future generalizations of this pattern that don’t need
such explicit representation formats, but for now I find it more than just
convenient.

If the representation wasn’t structured in this way, I would need to 
create an internal representation that is, in order to make use of
the model tools that I’m already using. Doing this would IMO not
be good for the human understanding, which is important in the
phase here where people are learning.

Maybe there is a way to preserve the model semantics in the 
representation and at the same time support efficient sequential 
processing of elements.

Best regards,

Michael


> On Jan 18, 2016, at 3:05 AM, Christian Amsüss <[e-mail]> wrote:
> 
> Hello Michael,
> 
> it appears that there are three concepts both being called or related to
> "streaming" that get mixed up in this thread, I'll try to flesh them out
> fist, describing their purpose, status of drafts, and interaction with
> client and server side.
> 
> A. The draft paragraph about "store or transmit SenML in a stream-like
>  fashion" in the "Multiple Datapoints" section:
> 
>  This is more about transportation, and implies that there is some kind
>  of boundary between elements at which transmission can pause.
> 
>  Such boundaries are present in all drafts so far.
> 
>  It appears to me that this would primarily work with HTTP in a fashion
>  similar to long polling, which is something both communicating parties
>  need to be aware of (all drafts: "MUST specify that they are doing
>  this"), lest they time out the connection.
> 
> B. Buffer-less operation:
> 
>  For very small applications (eg. using uIP and few kB of RAM), it is
>  desirable to use data formats that never require back-seeking, that
>  limit back-seeks to a fixed length or that can do with a fixed-length
>  buffer to hold information for that. The specification does not need
>  to actively describe those operations, requiring some basic structure
>  is sufficient.
> 
>  It has been suggested earlier on this list that -01 with an additional
>  requirement that "bn" and "bt" to come first in the dictionaries would
>  allow this mode of operation too -- truth, but hard to achieve with
>  generic JSON implementations and semantics.
> 
>  Thus, this goal is only achieved by drafts -02 and later. (This is
>  also what my demo is about, which shows that -03 is particularly
>  suitable).
> 
>  That this only an issue for the receiving side (the sender is free
>  to choose element sequences as is most practical for it anyway, as
>  long as it follows the specification). Having this negotiable would
>  defeat its purpose: If receivers may rely on it, we can't allow
>  senders to not support it.
> 
> C. Multiple base elements:
> 
>  The idea to have multiple base elements came up independently of the
>  others, and gives smaller data streams for cases in which different
>  sensors' time series are produced, or when there are larger gaps in a
>  timeline. The proposal worked well with the array-as-root scheme
>  envisioned with exactly 2 elements for B., and was implemented
>  together.
> 
>  It is present in -03 and -04.
> 
>  Senders are free not to use it. In my impression, this is easily to
>  implement in receivers. (Even if they desire to directly access
>  in-memory representations of JSON; the key to an entry might then be
>  (list-number, list-position) instead of just list-position).
> 
> 
>> PS Can we succinctly describe the use case for ordered serial
>> processing of senml+json elements being important? From reading the
>> draft, it seems like the important bit is saving the buffer memory for
>> large responses. 
> 
> The one I'm arguing here is B., the buffer-less operation. The use case
> are devices with limited heap memory (say 4k for incoming and outgoing
> packages). I'm using SenML in core-interfaces batches[1] to get and set
> device state. When those devices announce supporting batch operation,
> they need to accept SenML PUTs to as many resources as are batched
> together, so far without arbitrary constraints. Due to the proposed
> extensibility of SenML, even if I limited the number of resources in a
> batch, I couldn't predict the maximum size of an incoming update.
> 
> Now when the update arrives and a SenML that does not satisfy B. comes
> in, all I can do is to store the whole message body and parse it when
> the last block arrived. If the sender updated too many resources or used
> too large extensions, I'd need to 4.13-Request-Entity-too-large out.
> Still, until then (particularly until the overflowing package arrives),
> memory is clogged with unparsable data.
> 
>> Is it a bigger deal with CBOR encoding or less important. 
> 
> Whether CBOR or JSON is being parsed is almost irrelevant here. The only
> difference that comes to my mind is that with CBOR, we might be able to
> prescribe a serialization where "bn" and "bt" always precede "e" even in
> -01-style, but even that would make it harder to utilize for whomever
> implement this with a native-object-style library.
> 
>> Is the most important consideration being able to know the base values
>> ahead of the data?
> 
> This is about all information that may be in the dictionary and modifies
> how to interpret the elements' data. In core SenML, that information is
> the base attributes, and the version -- that is, all the possible root
> variables. (In a scenario .)
> 
> For extensions, this completely depends on them unless the receiver
> chooses to ignore them anyway. As far as I understand, your link format
> extension could add items to a core-interfaces linked batch, and then
> set a value in one go; for that, the extension data would need to
> precede the elements in parsing. Other extensions (say, metadata about
> when to expect changes) may be useful no matter where in the data stream
> it appears.
> 
> [1] https://tools.ietf.org/html/draft-ietf-core-interfaces-04#section-6.2
> 
>> Also, where is -03? there doesn’t seem to be a version -03 that you all are talking about on the IETF website:
>> https://tools.ietf.org/html/draft-jennings-core-senml-04 <https://tools.ietf.org/html/draft-jennings-core-senml-04>
> 
> The data tracker (or rather tools.ietf.org) seems not to take drafts
> submitted on the same day well. You can view it as [2].
> 
> [2] https://datatracker.ietf.org/doc/draft-jennings-core-senml/03/?include_text=1
> 
> On Sat, Jan 16, 2016 at 07:24:41AM -0800, Michael Koster wrote:
>> The optimizations you recommend require a new string input parser,
>> where I am using the JSON parser that comes with the library. My point
>> was that I will need to either make my own string parser for SenML or
>> make a re-parser with the state machine I described in order to use
>> -02 +[...]
> 
> I don't see yet where the state machine comes in. Surely, much
> state-machingin is going on in the optimized example, but when it comes
> to the core difference from -01 to -02, that is, that
> 
>    {any-key-but-e: any-data, "e": elements-list}
> 
> becomes
> 
>    [{any-key-but-e: any-data}, elements-list]
> 
> , I don't see how this is not changing access from _senml[key] to
> _senml[0][key] or _senml["e"] to _senml[1].
> 
> (If it's about the C. multiple-base-elements, I'd ask to keep these
> issues separate -- if C. breaks too much, although I like it, we could
> have a -02 with only two elements and still satisfy B., but I don't
> think we need to resort to that.)
> 
>> [...] but there is another issue anyway:
>> 
>> One of the big advantages of using JSON is connecting the embedded
>> world to the web world. To do this in a low-friction way, it’s good to
>> be able to use well known tools and patterns.
> 
> I fully agree that easy use of established web tools is essential here:
> but these very threads are where we should come up with mechanisms that
> work both with them and constrained devices.
> 
> 
> There is one point where I think that state-machining over incoming JSON
> objects does make sense, that is when it comes to names -- I like to see
> them as URIs (that's consistent with the examples given, but not fully
> required in the drafts), and when the "n" elements are relative URIs,
> they are not useful until joined with the base name. But that is not a
> change introduced in -01, but a preference of mine that is reflected in
> the demo to show that relative URIs are not that complicated even for
> embedded systems. (Existing web tools usually have their ways of joining
> URIs shipped).
> 
> 
>> Thanks also for looking into my code. I could easily use the -02 or
>> -03 in the senml processing as you show. What is missing still from
>> everything after -01 is the element tag that I am re-using to indicate
>> links and forms in the document:
> 
> I did see that you had extensions in your code, but as Carsten
> suggested, the "l" could stay in the first dictionary alongside the "bn"
> and "ver". Maybe the name "base dictionary" is suboptimal here -- the
> dictionary itself is not something that gets somehow prefixed to the
> following entries, it's just the place where the base attributes reside
> along with other things, like "ver" and extensions.
> 
> 
> On Sat, Jan 16, 2016 at 09:43:55AM -0800, Michael Koster wrote:
>> It would be useful if the order of top level element types could be arbitrary e.g.: [ {},[],[],[],{},[],{},{} ... ]
> 
> That's something I could well see in a -05. It would also make minimal
> SenML files that don't use any base attributes a little smaller, and
> embedded parsing wouldn't suffer from it AFAICT.
> 
> On Sat, Jan 16, 2016 at 10:33:00AM -0800, Michael Koster wrote:
>> PS for correctness my example should look more like this:
>> [ 
>>  { 
>>    "bn": "/light/brightness",
>>    "l": [ ... ],
>>    "f": [ ...],
>>    "e": [
>>      { "n": "tbr", "v": "44.3", },
>>      { "n": "tt", "v": "1.0", }
>>    ]
>>  }
>> ]
> 
> Just let's please not allow the "e" element any more. Anything with a
> top-level array is incompatible with older draft users anyway.
> 
> 
> Best regards
> Christian
> 
> -- 
> Christian Amsüss                      | Energy Harvesting Solutions GmbH
> founder, system architect             | headquarter:
> mailto:[e-mail]  | Arbeitergasse 15, A-4400 Steyr
> tel:[phone]                | http://www.energyharvesting.at/
>                                      | ATU68476614