[core] privacy issues (was: Re: Designs to resolve streaming issues in SenML)

[Stephen Farrell <[e-mail]>]

I'm willing to take the serious risk of diving at entirely
the wrong place, (and apologies in advance if so;-), but...

[1] seems to be using a MAC address to identify a sensor
reading. Ick. Even if a sensor never moves, that can be
bad if the sensor reading is only emitted as a result of
some privacy-relevant occurrence in the environment. If
a sensor gets carried about then it can be much worse.

I really hope we can provide something better than
such a long-term stable and easily tracked identifier,
even if that requires a bit more work. I don't think
section 13 [2] is really there yet either.

Is there text in some other place that could be
referenced or is this the document that ought define
some way to use identifiers that are less privacy
unfriendly? I do think we (IETF) should somewhere do
the work to define some such identifiers so that the
folks developing sensors and sinks for sensor data
can do a lot better.

Cheers,
S.

[1] https://tools.ietf.org/html/draft-jennings-core-senml-04#section-6.1.2
[2] https://tools.ietf.org/html/draft-jennings-core-senml-04#section-13

On 25/01/16 22:15, Cullen Jennings (fluffy) wrote:
> 
> This message really helps clarify things. Thanks. More inline
> 
> 
> On Jan 18, 2016, at 4:05 AM, Christian Amsüss <[e-mail]<mailto:[e-mail]>> wrote:
> 
> Hello Michael,
> 
> it appears that there are three concepts both being called or related to
> "streaming" that get mixed up in this thread, I'll try to flesh them out
> fist, describing their purpose, status of drafts, and interaction with
> client and server side.
> 
> A. The draft paragraph about "store or transmit SenML in a stream-like
>  fashion" in the "Multiple Datapoints" section:
> 
>  This is more about transportation, and implies that there is some kind
>  of boundary between elements at which transmission can pause.
> 
>  Such boundaries are present in all drafts so far.
> 
>  It appears to me that this would primarily work with HTTP in a fashion
>  similar to long polling, which is something both communicating parties
>  need to be aware of (all drafts: "MUST specify that they are doing
>  this"), lest they time out the connection.
> 
> agree and I think a different mine type can be used for both sides to understand they are doing this. In some cases the stream may never end and the JSON to close the data structures would never be sent so at some level it’s not even valid JSON.
> 
> 
> B. Buffer-less operation:
> 
>  For very small applications (eg. using uIP and few kB of RAM), it is
>  desirable to use data formats that never require back-seeking, that
>  limit back-seeks to a fixed length or that can do with a fixed-length
>  buffer to hold information for that. The specification does not need
>  to actively describe those operations, requiring some basic structure
>  is sufficient.
> 
>  It has been suggested earlier on this list that -01 with an additional
>  requirement that "bn" and "bt" to come first in the dictionaries would
>  allow this mode of operation too -- truth, but hard to achieve with
>  generic JSON implementations and semantics.
> 
>  Thus, this goal is only achieved by drafts -02 and later. (This is
>  also what my demo is about, which shows that -03 is particularly
>  suitable).
> 
>  That this only an issue for the receiving side (the sender is free
>  to choose element sequences as is most practical for it anyway, as
>  long as it follows the specification). Having this negotiable would
>  defeat its purpose: If receivers may rely on it, we can't allow
>  senders to not support it.
> 
> I think supporting this is important. It’s good for small things but it also tends to support server implementations that are large but need to be able to process really a lot of data quickly.
> 
> 
> C. Multiple base elements:
> 
>  The idea to have multiple base elements came up independently of the
>  others, and gives smaller data streams for cases in which different
>  sensors' time series are produced, or when there are larger gaps in a
>  timeline. The proposal worked well with the array-as-root scheme
>  envisioned with exactly 2 elements for B., and was implemented
>  together.
> 
> 
> I’m starting to think that the use case for multiple base elements is very weak and that perhaps we should only allow a single set of base units which are before any of the measurements. If we get this for free or cheap, great, but not something I care a lot about.
> 
> 
>  It is present in -03 and -04.
> 
>  Senders are free not to use it. In my impression, this is easily to
>  implement in receivers. (Even if they desire to directly access
>  in-memory representations of JSON; the key to an entry might then be
>  (list-number, list-position) instead of just list-position).
> 
> 
> PS Can we succinctly describe the use case for ordered serial
> processing of senml+json elements being important? From reading the
> draft, it seems like the important bit is saving the buffer memory for
> large responses.
> 
> The one I'm arguing here is B., the buffer-less operation. The use case
> are devices with limited heap memory (say 4k for incoming and outgoing
> packages). I'm using SenML in core-interfaces batches[1] to get and set
> device state. When those devices announce supporting batch operation,
> they need to accept SenML PUTs to as many resources as are batched
> together, so far without arbitrary constraints. Due to the proposed
> extensibility of SenML, even if I limited the number of resources in a
> batch, I couldn't predict the maximum size of an incoming update.
> 
> Now when the update arrives and a SenML that does not satisfy B. comes
> in, all I can do is to store the whole message body and parse it when
> the last block arrived. If the sender updated too many resources or used
> too large extensions, I'd need to 4.13-Request-Entity-too-large out.
> Still, until then (particularly until the overflowing package arrives),
> memory is clogged with unparsable data.
> 
> Is it a bigger deal with CBOR encoding or less important.
> 
> Whether CBOR or JSON is being parsed is almost irrelevant here. The only
> difference that comes to my mind is that with CBOR, we might be able to
> prescribe a serialization where "bn" and "bt" always precede "e" even in
> -01-style, but even that would make it harder to utilize for whomever
> implement this with a native-object-style library.
> 
> Is the most important consideration being able to know the base values
> ahead of the data?
> 
> This is about all information that may be in the dictionary and modifies
> how to interpret the elements' data. In core SenML, that information is
> the base attributes, and the version -- that is, all the possible root
> variables. (In a scenario .)
> 
> For extensions, this completely depends on them unless the receiver
> chooses to ignore them anyway. As far as I understand, your link format
> extension could add items to a core-interfaces linked batch, and then
> set a value in one go; for that, the extension data would need to
> precede the elements in parsing. Other extensions (say, metadata about
> when to expect changes) may be useful no matter where in the data stream
> it appears.
> 
> [1] https://tools.ietf.org/html/draft-ietf-core-interfaces-04#section-6.2
> 
> Also, where is -03? there doesn’t seem to be a version -03 that you all are talking about on the IETF website:
> https://tools.ietf.org/html/draft-jennings-core-senml-04 <https://tools.ietf.org/html/draft-jennings-core-senml-04>
> 
> The data tracker (or rather tools.ietf.org<http://tools.ietf.org>) seems not to take drafts
> submitted on the same day well. You can view it as [2].
> 
> [2] https://datatracker.ietf.org/doc/draft-jennings-core-senml/03/?include_text=1
> 
> On Sat, Jan 16, 2016 at 07:24:41AM -0800, Michael Koster wrote:
> The optimizations you recommend require a new string input parser,
> where I am using the JSON parser that comes with the library. My point
> was that I will need to either make my own string parser for SenML or
> make a re-parser with the state machine I described in order to use
> -02 +[...]
> 
> I don't see yet where the state machine comes in. Surely, much
> state-machingin is going on in the optimized example, but when it comes
> to the core difference from -01 to -02, that is, that
> 
>    {any-key-but-e: any-data, "e": elements-list}
> 
> becomes
> 
>    [{any-key-but-e: any-data}, elements-list]
> 
> , I don't see how this is not changing access from _senml[key] to
> _senml[0][key] or _senml["e"] to _senml[1].
> 
> (If it's about the C. multiple-base-elements, I'd ask to keep these
> issues separate -- if C. breaks too much, although I like it, we could
> have a -02 with only two elements and still satisfy B., but I don't
> think we need to resort to that.)
> 
> [...] but there is another issue anyway:
> 
> One of the big advantages of using JSON is connecting the embedded
> world to the web world. To do this in a low-friction way, it’s good to
> be able to use well known tools and patterns.
> 
> I fully agree that easy use of established web tools is essential here:
> but these very threads are where we should come up with mechanisms that
> work both with them and constrained devices.
> 
> 
> There is one point where I think that state-machining over incoming JSON
> objects does make sense, that is when it comes to names -- I like to see
> them as URIs (that's consistent with the examples given, but not fully
> required in the drafts), and when the "n" elements are relative URIs,
> they are not useful until joined with the base name. But that is not a
> change introduced in -01, but a preference of mine that is reflected in
> the demo to show that relative URIs are not that complicated even for
> embedded systems. (Existing web tools usually have their ways of joining
> URIs shipped).
> 
> It might be better if specification which used SenML said the combination of bn+n MUST be a URI and perhaps even constrain what type of URI are OK.
> 
> 
> Thanks also for looking into my code. I could easily use the -02 or
> -03 in the senml processing as you show. What is missing still from
> everything after -01 is the element tag that I am re-using to indicate
> links and forms in the document:
> 
> I did see that you had extensions in your code, but as Carsten
> suggested, the "l" could stay in the first dictionary alongside the "bn"
> and "ver". Maybe the name "base dictionary" is suboptimal here -- the
> dictionary itself is not something that gets somehow prefixed to the
> following entries, it's just the place where the base attributes reside
> along with other things, like "ver" and extensions.
> 
> if any of the post -01 version broke the ability to add extensions, that is just an mistake or my part and we can fix it. Allowing JSON-LD style links in the bn makes sense to me.
> 
> 
> 
> On Sat, Jan 16, 2016 at 09:43:55AM -0800, Michael Koster wrote:
> It would be useful if the order of top level element types could be arbitrary e.g.: [ {},[],[],[],{},[],{},{} ... ]
> 
> That's something I could well see in a -05. It would also make minimal
> SenML files that don't use any base attributes a little smaller, and
> embedded parsing wouldn't suffer from it AFAICT.
> 
> I need to send more justification about this to the list but looking at common JSON parsers for languages that do no allow variant arrays, it seems that many of theses libraries don’t easily support JSON with array of different types.
> 
> For example the golang JSON stuff. You can see how this works at
> 
> https://github.com/cisco/senmlCat/blob/master/senmlCat.go
> 
> On a side note, senmlCat can covert SenML to and from various formats and can also act as a HTTP server that receives POST of SenML and then writes them files, or influxdb, or kafka. I would not call it more alpha than done but hey, send a pull request.  I could not talk about some of the SenML project but having some open source code to point at that we can talk about is useful.
> 
> If you poke thought the libraries at http://www.json.org/ for a language like C, you can see they either do mallocs for every element in the list - which gets very slow compared to something like protobuf - or the schema tells the system the type of element in the array.
> 
> Now clearly JSON, XML, CBOR etc can all support variant arrays. And clearly in any language you can write a parser and data structure that allows them (even Fortran). We are just talking about performance and convenience  of using variant arrays in a language like C. I think it would very nice to be able to avoid variant arrays if we don’t need them.
> 
> 
> 
> On Sat, Jan 16, 2016 at 10:33:00AM -0800, Michael Koster wrote:
> PS for correctness my example should look more like this:
> [
>  {
>    "bn": "/light/brightness",
>    "l": [ ... ],
>    "f": [ ...],
>    "e": [
>      { "n": "tbr", "v": "44.3", },
>      { "n": "tt", "v": "1.0", }
>    ]
>  }
> ]
> 
> Just let's please not allow the "e" element any more. Anything with a
> top-level array is incompatible with older draft users anyway.
> 
> 
> Best regards
> Christian
> 
> --
> Christian Amsüss                      | Energy Harvesting Solutions GmbH
> founder, system architect             | headquarter:
> mailto:[e-mail]  | Arbeitergasse 15, A-4400 Steyr
> tel:[phone]                | http://www.energyharvesting.at/
>                                      | ATU68476614
> 
> 
> 
> _______________________________________________
> core mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/core
>