Re: [core] Subscribe/Notify for CoAP

[Zach Shelby <[e-mail]>]

Hi,

This is a really good conversation. One thing we need to do is remember what our minimum requirements are for subscribe/notify and not try to solve something more than we should. From what I know it is sufficient to be able to receive notifications when a URI changes or periodically. You might consider new URIs under this URI path to be changes to a URI though... 

On May 28, 2010, at 3:08 PM, Adriano Pezzuto (apezzuto) wrote:

> Hello,
> with this flavor, the SUBSCRIBE may be replaced by a simple POST. So no needs to have an explicit SUBSCRIBE method.

This is what coap-01 assumes. 

> If we want support a more oriented peer-to-peer transaction on CoAP, it looks to me both SUB and NOTIFY should be treated as messages.

The Request/Response model can be used peer-to-peer, and in CoAP you can even use a Request asynchronously. Additionally, we want a native RESTful subscribe/notify technique, that is, we want to subscribe to resources and be susequently notified about those resources when they change.

In that sense, you can think of a subscription as a request to receive asynchronous notifications about a resource (about a URI). This can be achieved by:

1.  as a SUBSCRIBE Request on the URI with some option indicating the lifetime as in coap-01,
2.  as a Subcsribe message (no methods) on the URI with some option, 
3.  or as a POST Request on the URI with options indicating that this is actually a subscription and the lifetime.  This bends the meaning of POST quite a bit and might be prone to mistakes?

The asynchronous response is a harder one. Can we bend the meaning of a Request for making asynchronous notifications? Also a notification includes a URI about a resource on the requestor, which is inverse from normal Requests in REST. So the options here are:

1. as a Notify message about a URI (no methods) as in coap-01,
2. as a NOTIFY Request about a URI as was in coap-00,
3. hmmm... reusing CRUD methods really doesn't work for this one at all... 

I wonder what the HTTP designers would have done if this would have been a requirement back then? Maybe we should ask them...

Zach

>  
> Adriano
>  
> From: [e-mail] [mailto:[e-mail]] 
> Sent: venerdì 28 maggio 2010 13.52
> To: Adriano Pezzuto (apezzuto)
> Cc: core; [e-mail]; [e-mail]
> Subject: Re: [core] Subscribe/Notify for CoAP
>  
> > So strictly speaking, both NOTIFY and SUBSCRIBE are message types not  methods!
> 
> SUBSCRIBE can be just an asynchronous GET on update then I think it is a method and NOTIFY is an asynchronous response so a message.
> But if we want to have selective notifications when a resource is created, updated or deleted then I think you're right NOTIFY and SUBSCRIBE are messages.
> 
> Does that make sense ?
> 
> Matthieu
> 
> <image001.gif>"Adriano Pezzuto (apezzuto)" <[e-mail]>
> 
> 
> 
> 
> "Adriano Pezzuto (apezzuto)" <[e-mail]> 
> Envoyé par : [e-mail]
> 28/05/2010 12:50
> 
> <image003.png>
> A
> <image004.png>
> <[e-mail]>
> <image003.png>
> cc
> <image004.png>
> core <[e-mail]>
> <image003.png>
> Objet
> <image004.png>
> Re: [core] Subscribe/Notify for CoAP
> <image004.png>
> <image004.png>
> 
> Hi Roberto,
> I understand your point and personally agree on M2M is quite different from web page model. On the architectural RESTful style, the method information tells the server what to do with data kept in the URI information.
> 
> So strictly speaking, both NOTIFY and SUBSCRIBE are message types not  methods!
> 
> Adriano 
> 
> From: Robert Cragie [mailto:[e-mail]] 
> Sent: venerdì 28 maggio 2010 11.53
> To: Adriano Pezzuto (apezzuto)
> Cc: Paul Duffy (paduffy); Zach Shelby; core
> Subject: Re: [core] Subscribe/Notify for CoAP
> 
> I think the point here is that there are two levels we are considering.
> 
> At the lowest (foundation) level, there are the transaction messages as described below. This provides a flexible mechanism for the application with regard to synchronicity and threading. This is important for the types of devices CoAP is being aimed at.
> 
> Above that, the methods can be used according to the architectural style. So in this case, it is RESTful (COnstrained Restful Environments), which will naturally limit the number of methods and how transactions occur.
> 
> The actual architecture using a combination of the methods and messages also depends on what is required at the application layer. Consider a typical client which wants to subscribe to a resource. That client controls the feed of data but needs a component which is capable of handling (possibly buffering) the data it receives through notifications. Is this a separate server? Or would we want to consider it part of an enhanced client model which is able to process feeds of data? These are the sort of models which have led to the myriad of solutions (GENA, Webhooks, long polling, pubsubhubbub, RESTMS etc.) based around HTTP which are all essentially ingenious ways of getting around the limitations imposed by HTTP and how it is processed for anything which deviates from the classic web page access model.
> 
> I think the aim of CoAP should be clean from the word go with regard to supporting these more peer-to-peer transactions, where the client can exist on either entity and both entities can feed data to each other; typical in M2M applications.
> 
> Robert
> 
> Robert Cragie (Pacific Gas & Electric)
> 
> Gridmerge Ltd.
> 89 Greenfield Crescent,
> Wakefield, WF4 4WA, UK
> [phone]
> [phone]
> http://www.gridmerge.com
> 
> On 27/05/2010 7:08 PM, Adriano Pezzuto (apezzuto) wrote: 
> Hi Robert,
> 
> in my personal opinion, the option 1a) brings some sort of ambiguity to CoAP specs.
> 
> My be my understatement of new CoAP specs is not so deep, but now we have 5 methods and 3 message types: request, response and notify. Which methods are allowed with which messages types?
> I suppose you have to use PUT/POST method with notify message for asynch data notification. How to make a subscribe? I suppose you would use a SUBSCRIBE method with request/response message or SUBSCRIBE with notify message? Also what about POST/DELETE methods in a notify message? They not make any sense..
> 
> I think the choice is between: option 1) -> only CRUD methods and option 1b) -> CRUD + SUB/NOTIFY methods, keeping in mind cost/benefits of both solutions.
> 
> Adriano
> 
> From: Robert Cragie [mailto:[e-mail]] 
> Sent: mercoledì 26 maggio 2010 20.09
> To: Adriano Pezzuto (apezzuto)
> Cc: Paul Duffy (paduffy); Zach Shelby; core
> Subject: Re: [core] Subscribe/Notify for CoAP
> 
> Hi Adrian,
> 
> I would also prefer to keep the protocol in CoAP asynchronous. You can always map an asynchronous protocol to a synchronous one but, as we see in HTTP, it always ends up as a kludge to do it the other way round. The efforts which have been gone to to make HTTP quasi-asynchronous via all the schemes mentioned below and many more besides (all non-interoperable of course) is testament to how important this is for M2M communication.
> 
> So, back to Zach's list, I favor 1a) for the following reasons:
> 
> Foundation level of messages:
> 
> 1. request/response can be asynchronous or synchronous messages (as there is a transaction ID in there)
> 2. notify is an asynchronous message
> Derived methods:
> 
> I think it makes sense to add a pub/sub model as a useful mechanism for M2M.
> 
> So, looking at it the other way round: It will be entirely possible to translate whatever is currently built on HTTP to CoAP based on the above, with all its restrictions regarding synchronous and client/server transactions. What may be harder is to translate directly is a CoAP-based application to HTTP. So I guess the question is: Do we want to be hamstrung to synchronous client/server transactions as dictated by HTTP and provide a direct mapping to HTTP, then have to come up with similar kludges for asynchronous peer-to-peer transactions as has been done in numerous ways for HTTP, or do we want to define the protocol cleanly to start with and accept that some sort of transaction relaying/conversion would have to take place at a mapping node?
> 
> Robert
> Robert Cragie (Pacific Gas & Electric)
> 
> Gridmerge Ltd.
> 89 Greenfield Crescent,
> Wakefield, WF4 4WA, UK
> [phone]
> [phone]
> http://www.gridmerge.com
> 
> On 26/05/2010 7:17 AM, Adriano Pezzuto (apezzuto) wrote: 
> Hi,
> it looks to me that CoAP should use an explicit sub/notify mechanism since this is the core of the machine-to-machine interaction model.
> HTTP suffers of this lack and we have seen a plethora of solutions to give an asynch taste to it. Webhooks and websockets are only the lasts of the list.
> As someone has already pointed out on this list, it is theoretically possible to describe sub/notify using only CRUD methods but it looks a little bit tricky and verbose.
> 
> Now we have a chance to build from scratch a new protocol with and I think using explicit sub/notify methods with a clear and well defined semantic is the best option. It is easily understanding from every developer and will prevent to build other fanny solutions on top of the CoAP. HTTP does not have this well defined semantic and (for hundreds of other reasons also) it is not used as wide protocol for machine-to-machine communication.
> 
> CoAP - as binary protocol - and with an explicit asynch model has a chance to be a really wide protocol for M2M communication not only for constrained environments.
> 
> my 2 cents
> 
> - adriano
> 
> -----Original Message-----
> From: [e-mail] [mailto:[e-mail]] On Behalf Of Paul Duffy (paduffy)
> Sent: mercoledì 26 maggio 2010 0.47
> To: Zach Shelby
> Cc: core
> Subject: Re: [core] Subscribe/Notify for CoAP
> 
> On 5/25/2010 6:41 PM, Zach Shelby wrote:
> 
> Hi,
> 
> On May 26, 2010, at 12:23 AM, Charles Palmer wrote:
> 
> 
> Hi folks
> 
> It occurs to me that CoRE should be keeping a close eye on ZigBee SE2.0 work, so that it is as easy as possible for ZigBee SE to use CoRE when ready. That suggests to me that we should align with their subscribe/notify process.
> 
> 
> I am not sure I understand that. I mean, ZigBee SE2.0 is defining an application specific subscribe/notify mechanism for that purpose so far for HTTP. This uses standard HTTP methods and some custom payload and REST interfaces. CoAP Req/Res is already totally compatible with SE2.0 in that respect, so alignment is already OK there. Nothing stopping someone from using SE2.0 over CoAP.
> 
> Specifying a native susbcription/notify into CoAP is another matter. We can't adopt a solution specific to one application as that won't solve the problems of other applications nor general HTTP mapping at all (probably would make it worse). It seems that for the near future there will be a bunch of HTTP push mechanisms in use without any clear standard appearing - or am I wrong there?
> 
> 
> 
> 
> If COAP extends HTTP semantics with new subscription methods, it will 
> not be possible to easily interchange HTTP/COAP, and translation 
> gateways will become more complex to implement.
> 
> 
> 
> Zach
> 
> 
> Regards - Charles
> 
> 
> -----Original Message-----
> From: [e-mail] [mailto:[e-mail]] On Behalf Of Paul Duffy
> Sent: 25 May 2010 03:48
> To: Zach Shelby
> Cc: core
> Subject: Re: [core] Subscribe/Notify for CoAP
> 
> Recommend something like #2, primarily to avoid introducing non HTTP
> method semantics, simplifying HTTP/COAP translation.gateways, etc.
> 
> 
> On 5/24/2010 11:49 AM, Zach Shelby wrote:
> 
> (thread renamed)
> 
> We have two different paths with regard to a subscribe/notify mechanism for CoAP:
> 
> 1. Use specific Subscription and Notify mechanisms for CoAP as HTTP really does not include this concept.
> 1a) Notify as a message and SUBSCRIBE as a method. This is currently used in coap-01.
> 1b) NOTIFY and SUBSCRIBE as methods. This was used in coap-00, but we had a good list discussion about this leading to a. In practice it doesn't make a big difference if notification is a message or method.
> 
> 2. Use an HTTP specific framework such as Webhooks, the ZigBee SE2.0 proposal or GENA.
> 
> So far we have focused on 1 in the WG, and every now and again 2 comes up. At least I am not convinced that we need to suffer the drawbacks of HTTP here. Anyways 2 does not help our mapping to HTTP in reality very much as there is no standard way of doing this over HTTP. Thus a CoAP-HTTP proxy may end up anyways translating between multiple HTTP frameworks depending on the application. So instead of doing a CoAP Notify/Subscribe to Webhooks mapping, you will could end up having to do a CoAP Webhooks to HTTP GENA mapping.
> 
> 
> 
> I don't understand this last para. If CoAP sticks to the semantics of
> the current HTTP methods, would this not offer a fairly straightforward
> translation to/from HTTP?
> 
> 
> 
> From what I have heard so far 1 still seems like a wise choice, although I need to look at Webhooks more deeply. In reality many applications specify their own way of doing a push interface using REST methods and specific payloads (ZigBee SE2.0 is such an example). That is just fine, and might be used instead of a specific CoAP notify/subscribe in that case. So 1 doesn't prevent the application using its own mechanism, it just provides a native way for doing push.
> 
> What do people think?
> 
> Zach
> 
> On May 17, 2010, at 6:44 PM,[e-mail] wrote:
> 
> 
> 
> Hi,
> 
> My comments about the subscribe/notify mechanism of Zigbee IP.
> 
> Pros:
> - Derived from the webhooks concept
> - If used by CORE it will be easier to map to HTTP because it uses only CRUD verbs.
> - The subscription message is extendable and could support advanced options (delays, increment, ...)
> - Only one listening port whatever the transport binding is.
> 
> Cons:
> - No interoperability without well known URIs and a well defined subscription message format (Not sure CoAP draft is the right place to specify this).
> - XML/EXI is too complex to be the required format for the default subscription/notification mechanism.
> - The notification should not require a subsequent GET to retrieve the content.
> - Subresource subscription is redundant.
> 
> Hope this could help,
> Matthieu
> 
> 
> <graycol.gif>"Charles Palmer"<[e-mail]>
> 
> 
> 
> 
> "Charles Palmer"<[e-mail]>
> Envoyé par : [e-mail]
> 15/05/2010 14:06
> 
> <ecblank.gif>
> A
> <ecblank.gif>
> "core"<[e-mail]>
> <ecblank.gif>
> cc
> <ecblank.gif>
> <ecblank.gif>
> Objet
> <ecblank.gif>
> Re: [core] Selecting a WG document for CoAP
> <ecblank.gif> <ecblank.gif>
> 
> Dear all
> 
> Those interested in the subscribe/notify discussion might like to look
> at the draft Smart Energy Profile 2.0 Application Protocol
> Specification. It is available here:
> http://zigbee.org/Markets/ZigBeeSmartEnergy/ZigBeeSmartEnergy20PublicApp
> licationProfile.aspx
> 
> It manages subscribe/notify by using POST. This seems to remove the need
> for SUBSCRIBE and notify.
> 
> "Imagine a host A, which exposes a resource at http{s}://A/resource and
> a second host B, which wishes to learn of changes to this resource. To
> facilitate a subscription/ notification mechanism, A would expose a
> resource http{s}://A/sub and B would expose a resource http{s}://B/ntfy.
> To subscribe to notifications regarding http{s}://A/resource, B would
> send a POST to the address http{s}://A/sub/B containing the URI of the
> resource of interest (http{s}://A/resource) and the URI of B's
> notification resource (http{s}://B/ntfy). Following this subscription
> step, should A wish to notify B of a change to the resource addressed at
> http{s}://A/resource, A would send a POST to the address
> http{s}://B/ntfy containing the URI of the resource changed
> (http{s}://A/resource) and the URI of A's subscription resource
> (http{s}://A/sub/B), should A wish to change its subscription. The host
> B can then query the resource (or not) at its leisure."
> 
> Sleepy nodes are not allowed to subscribe, and must poll.
> 
> Regards - Charles Palmer
> 
> -----Original Message-----
> From: [e-mail] [mailto:[e-mail]] On Behalf Of
> Angelo P. Castellani
> Sent: 14 May 2010 13:14
> To: Zach Shelby
> Cc: core
> Subject: Re: [core] Selecting a WG document for CoAP
> 
> Zach,
> 
> thanks for the comments, but please refer to my most recent e-mail for
> a more specific list of technical issues I'm pointing to.
> 
> I want to do only a little integration to what I've written there.
> 
> In my very personal opinion, to maximize adherence with the REST model
> and minimize implementation effort SUBSCRIBE and NOTIFY should be
> mapped to methods (as discussed many times together...).
> 
> Uniform interface principle (Fielding PhD dissertation Section 5.1.5,
> "The central feature that distinguishes the REST architectural style
> [...]") states that to simplify the software architecture, resource
> interfaces/interfaces should be as general as possible.
> 
> I agree with this principle in this specific case, mainly because
> handling a special message type for notify leads node software design
> to a more complex architecture.
> 
> The reason is that this new message type requires special handling and
> introduces a complexity in the software modularization.
> 
> Best,
> Angelo
> 
> On Fri, May 14, 2010 at 13:06, Zach Shelby<[e-mail]> wrote:
> 
> 
> Hi Angelo,
> 
> On May 13, 2010, at 14:24 , Angelo P. Castellani wrote:
> 
> 
> 
> Dear C. Bormann, all,
> 
> before deciding for the final direction, I've the following
> observations about draft-shelby-core-coap-01
> 
> While I mostly share Zach's view of the protocol approach, and
> appreciate many aspects of the proposal, there are in my opinion
> 
> 
> still
> 
> 
> some open issues that need to be at least discussed before the
> 
> 
> current
> 
> 
> document can be selected.
> 
> 
> Of course there are plenty of open issues. Remember that working group
> 
> 
> documents still undertake as much change and improvement as the WG
> wants, so by no means is coap-01 set in stone. I would expect at least
> 5-10 more revisions still along the way. Already several of your ideas
> have been integrated into coap-01, and several more are under
> consideration, so it is coming along. Patience ;-)
> 
> 
>  
> In particular, I would like to highlight the following:
> 
> a) As it is now, it is not possible to have a straightforward
> translation of HTTP -> CoAP and viceversa: see
> http://www.ietf.org/mail-archive/web/core/current/msg00133.html (this
> impacts the scalability of the web service model too)
> 
> 
> In coap-01 the Method names are now identical to HTTP methods. The
> 
> 
> Req/Res interaction is a direct translation. The URI hierarchy is
> compatible, and the URI binary code format we are still working on
> obviously. The only thing that takes some state to translate is
> Subscribe/Notify. But note, Subscribe/Notify takes some state no matter
> how you do it, even with HTTP-HTTP there is no clean and easy way to
> handle subscriptions.
> 
> 
>  
> b) Moreover, CoAP's implementation is not as simple as it should be.
> I've investigated the implementation and some design choices (as
> Options) are leading to very high program complexity (ROM) on a
> MSP430-based device.
> 
> 
> This we can definitely improve, and already made several optimizations
> 
> 
> from -00 to -01. Here I need some very concrete proposals though. Also
> remember that many things are optional, for example subscribe/notify is
> not required if you don't need it.
> 
> 
>  
> c) Finally when comparing HTTP message size with CoAP message size,
> the resulting compression isn't as good as you may expect.
> 
> Example:
> HTTP: GET /sensor/temp.xml HTTP/1.0 = 32 B
> CoAP: 24 B with options parsing procedure requiring an added
> implementation complexity
> 
> 
> Right, that is not how it will work in practice. Working with a real
> 
> 
> HTTP server that HTTP header will be more complex, and on the CoAP side
> you would chose shorter URLs. The biggest improvement possible here is
> from using binary coded URLs of course. We need to look at a wider range
> of interactions and real HTTP headers as well to check that we are
> efficient enough.
> 
> 
>  
> Addressing all these points potentially leads to major technical
> modifications (especially point a) on the current draft, hence it is
> appropriate in my opinion to discuss these points before making the
> final decision.
> 
> 
> I am not sure what else you have in mind for a). If we just forget
> 
> 
> about Subscribe/Notify then you are good to go. But then you are also
> not fulfilling the charter or the industry needs in that respect.
> 
> 
> Thanks,
> Zach
> 
> 
> 
> Best regards,
> 
> Angelo P. Castellani
> 
> 
> On Mon, May 10, 2010 at 18:51, Carsten Bormann<[e-mail]>wrote:
> 
> 
> The CORE WG has a milestone to select a WG document for CoAP in
> 
> 
> April:
> 
> 
> http://datatracker.ietf.org/wg/core/charter/
> ...
> Apr 2010 Select WG document for basis of the CoAP protocol
> 
> Of the various documents that have been contributed,
> 
> 
> draft-shelby-core-coap has significant discussion, as well as the
> largest number of updates (including a previous version that was still
> called -6lowapp-coap).
> 
> 
> Today, another updated version of that draft was announced. See
> http://www.ietf.org/mail-archive/web/core/current/msg00138.html
> for the announcement and
> http://tools.ietf.org/html/draft-shelby-core-coap-01
> for the draft itself.
> 
> However, as the authors say, there are still significant TODOs.
> 
> Are we in a state yet where we can say whether this is the right
> 
> 
> direction for the WG to take?
> 
> 
> If yes, is it the right direction? Should we adopt it as a WG
> 
> 
> document?
> 
> 
> If you don't think we can say yet, is there a set of technical
> 
> 
> decisions you would like the authors to take with priority?
> 
> 
> Note that once a document has become a WG document, the authors act
> 
> 
> as editors for the working group, making (and usually fleshing out the
> details of) any change that the WG decides it needs.
> 
> 
> If you think we can still improve the draft, this is not an obstacle
> 
> 
> to making it a WG document.
> 
> 
> But of course we shouldn't do that if we intend to reverse its
> 
> 
> fundamental technical direction.
> 
> 
> In order to stay roughly in sync with our milestones, we should
> 
> 
> reach at a decision on how to go forward this week.
> 
> 
> Gruesse, Carsten
> 
> _______________________________________________
> core mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/core
> 
> 
> 
> _______________________________________________
> core mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/core
> 
> 
> --
> Zach Shelby, Chief Nerd, Sensinode Ltd.
> http://zachshelby.org - My blog "On the Internet of Things"
> http://6lowpan.net - My book "6LoWPAN: The Wireless Embedded Internet"
> Mobile: [phone]
> 
> 
> 
> 
> _______________________________________________
> core mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/core
> 
> --------------------------------
> Onzo is a limited company number 06097997 registered in England& Wales. The
> registered office is 6 Great Newport Street, London, WC2H 7JB, United Kingdom.
> 
> This email message may contain confidential and/or privileged information, and
> is intended solely for the addressee(s). If you have received this email in
> error, please notify Onzo immediately. Unauthorised copying, disclosure or
> distribution of the material in this email is forbidden.
> --------------------------------
> 
> _______________________________________________
> core mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/core
> 
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> ______________________________________________________________________
> 
> _______________________________________________
> core mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/core
> 
> 
>  
> _______________________________________________
> core mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/core
> 
> --------------------------------
> Onzo is a limited company number 06097997 registered in England& Wales. The
> registered office is 6 Great Newport Street, London, WC2H 7JB, United Kingdom.
> 
> This email message may contain confidential and/or privileged information, and
> is intended solely for the addressee(s). If you have received this email in
> error, please notify Onzo immediately. Unauthorised copying, disclosure or
> distribution of the material in this email is forbidden.
> --------------------------------
> 
> 
>  
> 
> _______________________________________________
> core mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/core
> _______________________________________________
> core mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/core
> 
> 
> 
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> _____________________________________________________________________________________________________________________
> core mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/core
> 
> _______________________________________________
> core mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/core

-- 
Zach Shelby, Chief Nerd, Sensinode Ltd.
http://zachshelby.org  - My blog "On the Internet of Things"
http://6lowpan.net - My book "6LoWPAN: The Wireless Embedded Internet"
Mobile: [phone]