Re: [Netconf] In an update, when is a delete a delete?

[Martin Bjorklund <[e-mail]>]

Hi,

Thank you Alex for a well-written summary.

Alexander Clemm <[e-mail]> wrote:
> Let me briefly summarize where I think we are. The thread has
> clearly moved on from the initial issue of how to distinguish
> between causes for updates to no longer include a given data node
> (because it was deleted, or because it changed its value to no
> longer match a filter) to other issues.
> 
> For those other issues, I think there are in fact two separate items
> that we are trying to discuss at the same time, which are really
> orthogonal to one another: 
> 
> - The first item concerns the concept of a "filter" vs a "selector".
>   A filter is what gets specified for any subscription to
>   notifications, which defines which notifications a subscriber
>   wants to receive.  The filter is applied to the notification as a
>   whole, i.e. either the notification is delivered or it is not.  It
>   does not apply to subsets of contents within the notification.  A
>   selector, on the other hand, is used to specify updates of which
>   data nodes to include in a YANG-push subscription.  The same
>   update notification could include updates of several data nodes,
>   hence a filter applied to the notification as a whole would be
>   inappropriate here - the semantics is slightly different: as a
>   subscriber, "of which data nodes would you like to receive
>   updates", not "which notifications would you like to receive".
> 
> Filters and selectors can be specified using the same syntax.  We
> are faced with a choice between specifying a single construct as
> part of a subscription, which is treated as a filter (in case of a
> "regular" subscription for notification updates) or as a selector
> (in case of a subscription to datastore updates) , or having
> separate objects, i.e. adding a separate "selector" construct for
> YANG-push.  Using a single object amounts to overloading.  It is
> more compact but with arguably a more complex semantics.  Using
> separate object results in a model that is more verbose, but has
> arguably simpler semantics.
> 
> - The second item concerns the issue of whether the filter/selector
>   construct has a dynamic type or a static type.  This is where the
>   issue of identity vs case statement etc comes in.  In case of a
>   dynamic type, we use two objects: One object of a generic type
>   (anydata) holds the filter/selector construct itself, the second
>   object specifies how to interpret it, i.e. which type it is.  (In
>   case of "overloading, we can also make explicit the distinction
>   whether it is a filter or a selector).  That second object is an
>   identityref, referencing one of the identities that designates the
>   filter/selector type.  In case of a static type, we use a case
>   statement (to distinguish which specific type it is).

If you can make the identity-based solution work, that's fine with
me.  But the current text needs to work out the details.  It's not
clear at all what what the contents of the "anyxml" node is supposed
to be.

That said, even the "choice"-based solution is exensible, since it can
be augmented.  Unless it turns out that the identity-based solution
can be made to work well, I would prefer the "choice"-based solution.

> The argument that Eric is making is that we should have a single
> overloaded object that can serve as a filter or a selector depending
> on the context and whether it is used in a notification subscription
> or a YANG-push subscription, and that we use a dynamic type
> including an identityref that indicates whether the object serves as
> a selector or a filter.  From my perspective, I feel that not
> overloading may be conceptually a bit "cleaner", but at the end of
> the day I am fine either way.  And I am not entirely sure, Martin,
> what you are proposing.

I have a strong preference for clear semantics in separate objects.  I
think overloading the semantics in this particular example would be a
huge mistake.  Just to mention one detail, the context for the
filter and selector are completely different - the filter works on the
contents of a notification and the selector works on the contents of a
datastore.  The filter returns a boolean and the selector a node set.
(that was two details :)



/martin


> Either way, we should document the issue
> and our choice clearly.
> 
> --- Alex
> 
> 
> -----Original Message-----
> From: Martin Bjorklund [mailto:[e-mail]] 
> Sent: Wednesday, May 24, 2017 12:36 AM
> To: [e-mail]
> Cc: Alexander Clemm <[e-mail]>; [e-mail]
> Subject: Re: [Netconf] In an update, when is a delete a delete?
> 
> "Eric Voit (evoit)" <[e-mail]> wrote:
> > > From: Martin Bjorklund, May 23, 2017 1:57 PM
> > > 
> > > "Eric Voit (evoit)" <[e-mail]> wrote:
> > > > > Martin Bjorklund, May 22, 2017 3:30 PM
> > > > >
> > > > > Alexander Clemm <[e-mail]> wrote:
> > > > > > Hi Martin,
> > > > > >
> > > > > > Almost overlooked your question below.  What is meant by the 
> > > > > > filter is specified in section 3.5 of the YANG-Push document.
> > > > > >
> > > > > > "Only a single filter can be applied to a subscription at a time.
> > > > > > The following filter types are included in the yang-push data model:
> > > > > > [subtree] [xpath]"
> > > > >
> > > > > Actually, only "subtree" is defined in yang-push, "xpath" is 
> > > > > defined in subscribed-notifications.
> > > >
> > > > At the top of yang-push page 7, xpath selection is described.  Is 
> > > > there something you feel missing?
> > > 
> > > Yes.  First of all, the YANG module defines an identity called 
> > > "xpath", based on "sn:filter".  So this filter has nothing to do 
> > > with selecting nodes in a datastore; this filter is used to match 
> > > against a generated notification record.
> > 
> > To address this we could split the xpath identity into two types:
> > "xpath-selection" and "xpath-boolean".  These would have different 
> > definitions, but both reference
> > http://www.w3.org/TR/1999/REC-xpath-19991116 .  The difference is that 
> > one results in a xpath node-set expression, and the other an xpath 
> > Boolean expression.  Make sense?
> 
> Not really.  In subscribed-notifications you have defined a generic filter mechanism, that is used to block/allow the sending of generated notifications to a subscriber.  This filter mechanism cannot be used to select nodes to subscribe to for changes in a datastore.  You need to define a separate mechansim for that in yang-push.  (Maybe not even call it "filter", but perhaps "selection").
> 
> Some comments on subscribed-notifications: This generic mechanism allows various filter syntaxes.  This generic mechanism needs to explain what is required by a filter syntax definition (an identity, evaluation rules).  In section 2.2 the document says that two filter syntaxes are supported, but only one is defined (xpath).  It needs to explain that a filter is supposed to return true or false (this part RFC 5277 got right, see section 3.6).  Also, the current module has the filter in an anyxml node; it is not clear how an XPath expression is encoded in anyxml.
> 
> > > Second, the XPath filter is sorely underspecified.  The XPath 
> > > context is not described,
> > 
> > I understand and agree on your comment about the context.  The intent 
> > here is to provide equivalent capabilities of a GET.
> 
> I understand that.
> 
> > As it would be a
> > huge undertaking to try to consolidate an industry-wide view of the 
> > minimal xpath syntax and capabilities in networking
> 
> Ehh... yes?  What does this have to do with specifiying the XPath context?
> 
> > , I am hoping this
> > doesn't fall under the umbrella of YANG subscription.  I would be glad 
> > to support someone who wishes to take this up though.
> > 
> > > the expected result data type is not defined, and it is not 
> > > described how the result is supposed to be used.
> > 
> > As for the result, the anydata output should be provided to the 
> > subscriber (with appropriate security applied).
> 
> I was referring to the output of the filter evaluation.
> 
> > They can determine
> > how to use it.  The preferred embodiment would be to maintain a local 
> > extract of the Publisher's datastore (as defined by the filter).
> > 
> > > > >  [Side note - I think this is wrong, subscribed-notifications 
> > > > > should also define "subtree".]
> > > > >
> > > > > But these filters are used by the server to decide if a certain 
> > > > > notification that has been generated will be sent to the client 
> > > > > or not.
> > > >
> > > > Yes, the filters in subscribed-notification are supposed to give a 
> > > > boolean indication as to whether a specific event should traverse 
> > > > the filter in its entirety. RFC6241 section 6 subtree filters are 
> > > > written to provide a subset of content.  I suppose it would be 
> > > > possible to define an event-based subtree-filter-type where a 
> > > > non-null result of the subtree filter means that a particular 
> > > > event should traverse that filter.  Is this what you are suggesting?
> > > 
> > > Yes.  Note that this is already provided by RFC 5277, and I have 
> > > always assumed that this new work will provide at least the same 
> > > functions as RFC
> > > 5277 (and more).  (But note that the XPath filter is underspecified 
> > > also in RFC
> > > 5277...)
> > 
> > I also want to make sure that a non-null result from a filter allows 
> > the event to pass.  I suspect that an xpath Boolean filter could be 
> > designed to do this, but I will tweak the subscribed-notifications 
> > text so that unnecessary filtering expression complexity is not 
> > artificially required.
> 
> Please make sure you understand how subtree filters and XPath filters work in RFC 5277.  There is nothing wrong with that functionality.
> 
> > > > > If you want to define filters to specifify which nodes to 
> > > > > subscribe to, I think you need to define new filters, not try to 
> > > > > reuese these notification filters.
> > > >
> > > > Filtering syntax is hard, so we have been trying to adopt whatever 
> > > > is available for GET.  This way we don't have to educate users on 
> > > > a new universe of what is possible.  I fully expect that lots of 
> > > > learnings are going to come in the industry here over time, and 
> > > > this will be revisited in the future.
> > > 
> > > ?
> > > 
> > > I am not proposing any new filter syntax.  I am saying that the 
> > > current filter nodes as defined in subscribed-notification cannot be 
> > > used to select nodes to subscribe to for changes.
> > 
> > Understand.  Hopefully with the "xpath-selection" change proposed 
> > above, this will be covered.
> 
> No, see above.
> 
> 
> /martin
> 
> 
> 
> > 
> > Eric
> > 
> > > > > As for your question, I think such a filter should be defined to 
> > > > > return a node- set to which the client subscribe to changes.  If 
> > > > > any node (or subnode
> > > > > to) in
> > > > > this node-set changes, the notif will be sent.  Then the 
> > > > > question about value comparision is not relevant anymore.
> > > >
> > > > Excellent, on-change should only send an update if the results of 
> > > > the subscription filter have changed since the previous push.  It 
> > > > is quite possible that an object has been created and then deleted 
> > > > since the last push.
> > > 
> > > I don't understand what you're trying to say with these sentences 
> > > (but since the first word was "Excellent" maybe it's ok ;)
> > > 
> > > 
> > > /martin
> > > 
> > > 
> > > > Representing this was the genesis of Alex's question.
> > > >
> > > > Eric
> > > >
> > > > > /martin
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > >
> > > > > > In effect, the filter specifies which data nodes to consider 
> > > > > > when sending updates.
> > > > > >
> > > > > > ---Alex
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Martin Bjorklund [mailto:[e-mail]]
> > > > > > Sent: Wednesday, May 17, 2017 11:50 PM
> > > > > > To: Alexander Clemm <[e-mail]>
> > > > > > Cc: [e-mail]
> > > > > > Subject: Re: [Netconf] In an update, when is a delete a delete?
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Alexander Clemm <[e-mail]> wrote:
> > > > > > > Hello all,
> > > > > > >
> > > > > > > In updating the YANG-Push document 
> > > > > > > (draft-ietf-netconf-yang-push), we have come across one 
> > > > > > > issue that we wanted to raise with the working group.
> > > > > > >
> > > > > > > As part of an on-change subscription, update records reflect 
> > > > > > > the type of change (e.g. whether the value of an object has 
> > > > > > > changed, or whether an object was created or deleted); a 
> > > > > > > subscription allows also to specify whether interested only 
> > > > > > > in specific types of changes (for example, only creates and 
> > > > > > > deleted but no value
> > > changes).
> > > > > > >
> > > > > > > At the same time, a subscription filter specifies which 
> > > > > > > objects to include as part of a subscription and which not.
> > > > > >
> > > > > > Hmm, which filter are you talking about?  The only XPath 
> > > > > > filter I find in the current set of documents is the 
> > > > > > "ietf-subscribed-notifications:xpath" filter type (which btw 
> > > > > > is sorely underspecified).  Section 2.2 of
> > > > > > draft-ietf-netconf-subscribed-notifications-02 says:
> > > > > >
> > > > > >    Events which evaluate to "true" as a
> > > > > >    result of the evaluation by the filter must traverse the filter in
> > > > > >    their entirety.
> > > > > >
> > > > > > It's not clear what this means, but my guess is that this is 
> > > > > > supposed to work like the old RFC 5277 filters, where the 
> > > > > > filter expression is evaluated on the notification contents, 
> > > > > > and if the expression returns "true" (for XPath filters this 
> > > > > > means converting the results to a boolean), then the 
> > > > > > notification is sent, otherwise not.
> > > > > >
> > > > > > But it seems you are referring to some other filter which 
> > > > > > would be used to select a node set for which changes are reported?
> > > > > >
> > > > > > I would like to understand which filter mechanism you mean 
> > > > > > before having an opinion in this matter.
> > > > > >
> > > > > >
> > > > > > /martin
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > > (Really, it is not
> > > > > > > so much of a "filter" on a stream that is generated 
> > > > > > > independently of the filter, than it is a policy of which 
> > > > > > > objects to include as part of subscribed update records.) 
> > > > > > > However, a subscription filter (such as
> > > > > > > XPath) can be used to also specify a value filter, which 
> > > > > > > will include or exclude objects based on their current 
> > > > > > > value. This makes it possible to e.g. subscribe to an object 
> > > > > > > "foo" but only if its value is 5.
> > > > > > >
> > > > > > > Now, this means that the same object could be included in 
> > > > > > > one update, but excluded in another update, due to its value 
> > > > > > > no longer meeting the filter criteria.  For example, if 
> > > > > > > foo's value changes from 5 to 3 in one cycle, a periodic 
> > > > > > > subscription will no longer include foo in its next update.  
> > > > > > > The question now concerns how to properly handle this in the 
> > > > > > > case of an on-change
> > > subscription.
> > > > > > >
> > > > > > > One possibility concerns reporting the fact that "foo" no 
> > > > > > > longer meets the subscription criteria and is no longer 
> > > > > > > included in the update record as a "delete" event.  If foo's value again becomes "5"
> > > > > > > at a later point in time, that would be reported as a "create"
> > > > > > > event.  If foo's value changes again from 5 at a later point 
> > > > > > > in time and then changes back to 3 before the time of the 
> > > > > > > update (perhaps because the value changed during the 
> > > > > > > dampening interval), it would be reported as another 
> > > > > > > "delete" event (without ever reporting a create event).  On 
> > > > > > > the other hand, if foo's value changed from 3 to
> > > > > > > 6 and back again, nothing would be reported because it did 
> > > > > > > not meet the filter criteria at any point in time.
> > > > > > >
> > > > > > > >From the perspective of the receiver this may make sense if 
> > > > > > > >it is synching its copy of the state.  However, from the 
> > > > > > > >perspective of the publisher, the object was never created 
> > > > > > > >or deleted - only its value changed, and the case when the 
> > > > > > > >object was truly created or deleted can no longer be 
> > > > > > > >distinguished from the case when its value
> > > > > changed.
> > > > > > > >A "create" simply means "an object now meets a filter 
> > > > > > > >criteria, that was not reported in the previous cycle" 
> > > > > > > >(which does not mean that the object was actually created - 
> > > > > > > >it may have been created, or it may have simply undergone a value change).
> > > > > > >
> > > > > > > An alternative (let's call it alternative 2) is therefore to 
> > > > > > > make a distinction between whether an object was created or 
> > > > > > > deleted, or whether its value fell in or out of a filter range.
> > > > > > > This appears semantically cleaner.  However, it will require 
> > > > > > > modifying the encoding to allow for distinction between 
> > > > > > > those cases (currently, just plain patch encoding is used).
> > > > > > >
> > > > > > > A third alternative is to let filters select only data nodes 
> > > > > > > to subscribe to, and separate out the value filter (or 
> > > > > > > disallow it as a feature altogether).  This alternative has 
> > > > > > > the drawback of being less conceptually powerful, even if it 
> > > > > > > may be easier to
> > > implement.
> > > > > > >
> > > > > > > Thoughts?  Any preferences between 1, 2, and 3?
> > > > > > > --- Alex
> > > > > > >
> > > > > >
> > > > >
> > > > > _______________________________________________
> > > > > Netconf mailing list
> > > > > [e-mail]
> > > > > https://www.ietf.org/mailman/listinfo/netconf
> > > >
> > 
>