Re: [core] Designs to resolve streaming issues in SenML

[Christian Amsüss <[e-mail]>]

Hello Michael,

it appears that there are three concepts both being called or related to
"streaming" that get mixed up in this thread, I'll try to flesh them out
fist, describing their purpose, status of drafts, and interaction with
client and server side.

A. The draft paragraph about "store or transmit SenML in a stream-like
  fashion" in the "Multiple Datapoints" section:

  This is more about transportation, and implies that there is some kind
  of boundary between elements at which transmission can pause.
  
  Such boundaries are present in all drafts so far.

  It appears to me that this would primarily work with HTTP in a fashion
  similar to long polling, which is something both communicating parties
  need to be aware of (all drafts: "MUST specify that they are doing
  this"), lest they time out the connection.

B. Buffer-less operation:

  For very small applications (eg. using uIP and few kB of RAM), it is
  desirable to use data formats that never require back-seeking, that
  limit back-seeks to a fixed length or that can do with a fixed-length
  buffer to hold information for that. The specification does not need
  to actively describe those operations, requiring some basic structure
  is sufficient.

  It has been suggested earlier on this list that -01 with an additional
  requirement that "bn" and "bt" to come first in the dictionaries would
  allow this mode of operation too -- truth, but hard to achieve with
  generic JSON implementations and semantics.

  Thus, this goal is only achieved by drafts -02 and later. (This is
  also what my demo is about, which shows that -03 is particularly
  suitable).

  That this only an issue for the receiving side (the sender is free
  to choose element sequences as is most practical for it anyway, as
  long as it follows the specification). Having this negotiable would
  defeat its purpose: If receivers may rely on it, we can't allow
  senders to not support it.

C. Multiple base elements:

  The idea to have multiple base elements came up independently of the
  others, and gives smaller data streams for cases in which different
  sensors' time series are produced, or when there are larger gaps in a
  timeline. The proposal worked well with the array-as-root scheme
  envisioned with exactly 2 elements for B., and was implemented
  together.

  It is present in -03 and -04.

  Senders are free not to use it. In my impression, this is easily to
  implement in receivers. (Even if they desire to directly access
  in-memory representations of JSON; the key to an entry might then be
  (list-number, list-position) instead of just list-position).


> PS Can we succinctly describe the use case for ordered serial
> processing of senml+json elements being important? From reading the
> draft, it seems like the important bit is saving the buffer memory for
> large responses. 

The one I'm arguing here is B., the buffer-less operation. The use case
are devices with limited heap memory (say 4k for incoming and outgoing
packages). I'm using SenML in core-interfaces batches[1] to get and set
device state. When those devices announce supporting batch operation,
they need to accept SenML PUTs to as many resources as are batched
together, so far without arbitrary constraints. Due to the proposed
extensibility of SenML, even if I limited the number of resources in a
batch, I couldn't predict the maximum size of an incoming update.

Now when the update arrives and a SenML that does not satisfy B. comes
in, all I can do is to store the whole message body and parse it when
the last block arrived. If the sender updated too many resources or used
too large extensions, I'd need to 4.13-Request-Entity-too-large out.
Still, until then (particularly until the overflowing package arrives),
memory is clogged with unparsable data.

> Is it a bigger deal with CBOR encoding or less important. 

Whether CBOR or JSON is being parsed is almost irrelevant here. The only
difference that comes to my mind is that with CBOR, we might be able to
prescribe a serialization where "bn" and "bt" always precede "e" even in
-01-style, but even that would make it harder to utilize for whomever
implement this with a native-object-style library.

> Is the most important consideration being able to know the base values
> ahead of the data?

This is about all information that may be in the dictionary and modifies
how to interpret the elements' data. In core SenML, that information is
the base attributes, and the version -- that is, all the possible root
variables. (In a scenario .)

For extensions, this completely depends on them unless the receiver
chooses to ignore them anyway. As far as I understand, your link format
extension could add items to a core-interfaces linked batch, and then
set a value in one go; for that, the extension data would need to
precede the elements in parsing. Other extensions (say, metadata about
when to expect changes) may be useful no matter where in the data stream
it appears.

[1] https://tools.ietf.org/html/draft-ietf-core-interfaces-04#section-6.2

> Also, where is -03? there doesn’t seem to be a version -03 that you all are talking about on the IETF website:
> https://tools.ietf.org/html/draft-jennings-core-senml-04 <https://tools.ietf.org/html/draft-jennings-core-senml-04>

The data tracker (or rather tools.ietf.org) seems not to take drafts
submitted on the same day well. You can view it as [2].

[2] https://datatracker.ietf.org/doc/draft-jennings-core-senml/03/?include_text=1

On Sat, Jan 16, 2016 at 07:24:41AM -0800, Michael Koster wrote:
> The optimizations you recommend require a new string input parser,
> where I am using the JSON parser that comes with the library. My point
> was that I will need to either make my own string parser for SenML or
> make a re-parser with the state machine I described in order to use
> -02 +[...]

I don't see yet where the state machine comes in. Surely, much
state-machingin is going on in the optimized example, but when it comes
to the core difference from -01 to -02, that is, that

    {any-key-but-e: any-data, "e": elements-list}

becomes

    [{any-key-but-e: any-data}, elements-list]

, I don't see how this is not changing access from _senml[key] to
_senml[0][key] or _senml["e"] to _senml[1].

(If it's about the C. multiple-base-elements, I'd ask to keep these
issues separate -- if C. breaks too much, although I like it, we could
have a -02 with only two elements and still satisfy B., but I don't
think we need to resort to that.)

> [...] but there is another issue anyway:
> 
> One of the big advantages of using JSON is connecting the embedded
> world to the web world. To do this in a low-friction way, it’s good to
> be able to use well known tools and patterns.

I fully agree that easy use of established web tools is essential here:
but these very threads are where we should come up with mechanisms that
work both with them and constrained devices.


There is one point where I think that state-machining over incoming JSON
objects does make sense, that is when it comes to names -- I like to see
them as URIs (that's consistent with the examples given, but not fully
required in the drafts), and when the "n" elements are relative URIs,
they are not useful until joined with the base name. But that is not a
change introduced in -01, but a preference of mine that is reflected in
the demo to show that relative URIs are not that complicated even for
embedded systems. (Existing web tools usually have their ways of joining
URIs shipped).


> Thanks also for looking into my code. I could easily use the -02 or
> -03 in the senml processing as you show. What is missing still from
> everything after -01 is the element tag that I am re-using to indicate
> links and forms in the document:

I did see that you had extensions in your code, but as Carsten
suggested, the "l" could stay in the first dictionary alongside the "bn"
and "ver". Maybe the name "base dictionary" is suboptimal here -- the
dictionary itself is not something that gets somehow prefixed to the
following entries, it's just the place where the base attributes reside
along with other things, like "ver" and extensions.


On Sat, Jan 16, 2016 at 09:43:55AM -0800, Michael Koster wrote:
> It would be useful if the order of top level element types could be arbitrary e.g.: [ {},[],[],[],{},[],{},{} ... ]

That's something I could well see in a -05. It would also make minimal
SenML files that don't use any base attributes a little smaller, and
embedded parsing wouldn't suffer from it AFAICT.

On Sat, Jan 16, 2016 at 10:33:00AM -0800, Michael Koster wrote:
> PS for correctness my example should look more like this:
> [ 
>   { 
>     "bn": "/light/brightness",
>     "l": [ ... ],
>     "f": [ ...],
>     "e": [
>       { "n": "tbr", "v": "44.3", },
>       { "n": "tt", "v": "1.0", }
>     ]
>   }
> ]

Just let's please not allow the "e" element any more. Anything with a
top-level array is incompatible with older draft users anyway.


Best regards
Christian

-- 
Christian Amsüss                      | Energy Harvesting Solutions GmbH
founder, system architect             | headquarter:
mailto:[e-mail]  | Arbeitergasse 15, A-4400 Steyr
tel:[phone]                | http://www.energyharvesting.at/
                                      | ATU68476614