Re: [Netconf] In an update, when is a delete a delete?

["Eric Voit (evoit)" <[e-mail]>]

> From: Martin Bjorklund, May 25, 2017 5:10 AM
> 
> Hi,
> 
> Thank you Alex for a well-written summary.
> 
> Alexander Clemm <[e-mail]> wrote:
> > Let me briefly summarize where I think we are. The thread has clearly
> > moved on from the initial issue of how to distinguish between causes
> > for updates to no longer include a given data node (because it was
> > deleted, or because it changed its value to no longer match a filter)
> > to other issues.
> >
> > For those other issues, I think there are in fact two separate items
> > that we are trying to discuss at the same time, which are really
> > orthogonal to one another:
> >
> > - The first item concerns the concept of a "filter" vs a "selector".
> >   A filter is what gets specified for any subscription to
> >   notifications, which defines which notifications a subscriber
> >   wants to receive.  The filter is applied to the notification as a
> >   whole, i.e. either the notification is delivered or it is not.  It
> >   does not apply to subsets of contents within the notification.  A
> >   selector, on the other hand, is used to specify updates of which
> >   data nodes to include in a YANG-push subscription.  The same
> >   update notification could include updates of several data nodes,
> >   hence a filter applied to the notification as a whole would be
> >   inappropriate here - the semantics is slightly different: as a
> >   subscriber, "of which data nodes would you like to receive
> >   updates", not "which notifications would you like to receive".
> >
> > Filters and selectors can be specified using the same syntax.  We are
> > faced with a choice between specifying a single construct as part of a
> > subscription, which is treated as a filter (in case of a "regular"
> > subscription for notification updates) or as a selector (in case of a
> > subscription to datastore updates) , or having separate objects, i.e.
> > adding a separate "selector" construct for YANG-push.  Using a single
> > object amounts to overloading.  It is more compact but with arguably a
> > more complex semantics.  Using separate object results in a model that
> > is more verbose, but has arguably simpler semantics.
> >
> > - The second item concerns the issue of whether the filter/selector
> >   construct has a dynamic type or a static type.  This is where the
> >   issue of identity vs case statement etc comes in.  In case of a
> >   dynamic type, we use two objects: One object of a generic type
> >   (anydata) holds the filter/selector construct itself, the second
> >   object specifies how to interpret it, i.e. which type it is.  (In
> >   case of "overloading, we can also make explicit the distinction
> >   whether it is a filter or a selector).  That second object is an
> >   identityref, referencing one of the identities that designates the
> >   filter/selector type.  In case of a static type, we use a case
> >   statement (to distinguish which specific type it is).
> 
> If you can make the identity-based solution work, that's fine with me.  But the
> current text needs to work out the details.  It's not clear at all what what the
> contents of the "anyxml" node is supposed to be.
> 
> That said, even the "choice"-based solution is exensible, since it can be
> augmented.  Unless it turns out that the identity-based solution can be made
> to work well, I would prefer the "choice"-based solution.
> 
> > The argument that Eric is making is that we should have a single
> > overloaded object that can serve as a filter or a selector depending
> > on the context and whether it is used in a notification subscription
> > or a YANG-push subscription, and that we use a dynamic type including
> > an identityref that indicates whether the object serves as a selector
> > or a filter.  From my perspective, I feel that not overloading may be
> > conceptually a bit "cleaner", but at the end of the day I am fine
> > either way.  And I am not entirely sure, Martin, what you are
> > proposing.
> 
> I have a strong preference for clear semantics in separate objects.  I think
> overloading the semantics in this particular example would be a huge mistake.
> Just to mention one detail, the context for the filter and selector are
> completely different - the filter works on the contents of a notification and the
> selector works on the contents of a datastore.  The filter returns a boolean
> and the selector a node set.
> (that was two details :)

Ok.  Unless anyone objects, let's go with separate objects then.  

We can still retain the identity hierarchy.  The result to would be something like:

In subscribed-notifications...:

Identity filter;
    Identity test-filter {base filter;}
       Identity xpath-test-filter {base  test-filter;}
       Identity  subtree-test-filter {base test-filter;}

"anyxml filter"
Description "Event stream evaluation criteria encoded in a syntax of a supported type of test-filter.  If the filter is applied against an event stream and there is a non-empty or positive result, the event is passed along."

In YANG Push...:

Identity node-selection-criteria {base sn:filter;}
    Identity subtree-node-selection-criteria {base node-selection-criteria;}
    Identity xpath-node-selection-criteria {base node-selection-criteria;}

"anyxml selector"
Description "YANG datastore query encoded in the syntax of a supported type of node-selection-criteria.  If the selector is applied against a datastore for periodic extracts, the resulting node-set result is passed along. If the selector is applied against a datastore looking for changes, deltas from the last update in the form of a patch result are passed along."

Eric


> /martin
> 
> 
> > Either way, we should document the issue and our choice clearly.
> >
> > --- Alex
> >
> >
> > -----Original Message-----
> > From: Martin Bjorklund [mailto:[e-mail]]
> > Sent: Wednesday, May 24, 2017 12:36 AM
> > To: [e-mail]
> > Cc: Alexander Clemm <[e-mail]>; [e-mail]
> > Subject: Re: [Netconf] In an update, when is a delete a delete?
> >
> > "Eric Voit (evoit)" <[e-mail]> wrote:
> > > > From: Martin Bjorklund, May 23, 2017 1:57 PM
> > > >
> > > > "Eric Voit (evoit)" <[e-mail]> wrote:
> > > > > > Martin Bjorklund, May 22, 2017 3:30 PM
> > > > > >
> > > > > > Alexander Clemm <[e-mail]> wrote:
> > > > > > > Hi Martin,
> > > > > > >
> > > > > > > Almost overlooked your question below.  What is meant by the
> > > > > > > filter is specified in section 3.5 of the YANG-Push document.
> > > > > > >
> > > > > > > "Only a single filter can be applied to a subscription at a time.
> > > > > > > The following filter types are included in the yang-push data model:
> > > > > > > [subtree] [xpath]"
> > > > > >
> > > > > > Actually, only "subtree" is defined in yang-push, "xpath" is
> > > > > > defined in subscribed-notifications.
> > > > >
> > > > > At the top of yang-push page 7, xpath selection is described.
> > > > > Is there something you feel missing?
> > > >
> > > > Yes.  First of all, the YANG module defines an identity called
> > > > "xpath", based on "sn:filter".  So this filter has nothing to do
> > > > with selecting nodes in a datastore; this filter is used to match
> > > > against a generated notification record.
> > >
> > > To address this we could split the xpath identity into two types:
> > > "xpath-selection" and "xpath-boolean".  These would have different
> > > definitions, but both reference
> > > http://www.w3.org/TR/1999/REC-xpath-19991116 .  The difference is
> > > that one results in a xpath node-set expression, and the other an
> > > xpath Boolean expression.  Make sense?
> >
> > Not really.  In subscribed-notifications you have defined a generic filter
> mechanism, that is used to block/allow the sending of generated notifications
> to a subscriber.  This filter mechanism cannot be used to select nodes to
> subscribe to for changes in a datastore.  You need to define a separate
> mechansim for that in yang-push.  (Maybe not even call it "filter", but perhaps
> "selection").
> >
> > Some comments on subscribed-notifications: This generic mechanism
> allows various filter syntaxes.  This generic mechanism needs to explain what
> is required by a filter syntax definition (an identity, evaluation rules).  In
> section 2.2 the document says that two filter syntaxes are supported, but only
> one is defined (xpath).  It needs to explain that a filter is supposed to return
> true or false (this part RFC 5277 got right, see section 3.6).  Also, the current
> module has the filter in an anyxml node; it is not clear how an XPath
> expression is encoded in anyxml.
> >
> > > > Second, the XPath filter is sorely underspecified.  The XPath
> > > > context is not described,
> > >
> > > I understand and agree on your comment about the context.  The
> > > intent here is to provide equivalent capabilities of a GET.
> >
> > I understand that.
> >
> > > As it would be a
> > > huge undertaking to try to consolidate an industry-wide view of the
> > > minimal xpath syntax and capabilities in networking
> >
> > Ehh... yes?  What does this have to do with specifiying the XPath context?
> >
> > > , I am hoping this
> > > doesn't fall under the umbrella of YANG subscription.  I would be
> > > glad to support someone who wishes to take this up though.
> > >
> > > > the expected result data type is not defined, and it is not
> > > > described how the result is supposed to be used.
> > >
> > > As for the result, the anydata output should be provided to the
> > > subscriber (with appropriate security applied).
> >
> > I was referring to the output of the filter evaluation.
> >
> > > They can determine
> > > how to use it.  The preferred embodiment would be to maintain a
> > > local extract of the Publisher's datastore (as defined by the filter).
> > >
> > > > > >  [Side note - I think this is wrong, subscribed-notifications
> > > > > > should also define "subtree".]
> > > > > >
> > > > > > But these filters are used by the server to decide if a
> > > > > > certain notification that has been generated will be sent to
> > > > > > the client or not.
> > > > >
> > > > > Yes, the filters in subscribed-notification are supposed to give
> > > > > a boolean indication as to whether a specific event should
> > > > > traverse the filter in its entirety. RFC6241 section 6 subtree
> > > > > filters are written to provide a subset of content.  I suppose
> > > > > it would be possible to define an event-based
> > > > > subtree-filter-type where a non-null result of the subtree
> > > > > filter means that a particular event should traverse that filter.  Is this
> what you are suggesting?
> > > >
> > > > Yes.  Note that this is already provided by RFC 5277, and I have
> > > > always assumed that this new work will provide at least the same
> > > > functions as RFC
> > > > 5277 (and more).  (But note that the XPath filter is
> > > > underspecified also in RFC
> > > > 5277...)
> > >
> > > I also want to make sure that a non-null result from a filter allows
> > > the event to pass.  I suspect that an xpath Boolean filter could be
> > > designed to do this, but I will tweak the subscribed-notifications
> > > text so that unnecessary filtering expression complexity is not
> > > artificially required.
> >
> > Please make sure you understand how subtree filters and XPath filters work
> in RFC 5277.  There is nothing wrong with that functionality.
> >
> > > > > > If you want to define filters to specifify which nodes to
> > > > > > subscribe to, I think you need to define new filters, not try
> > > > > > to reuese these notification filters.
> > > > >
> > > > > Filtering syntax is hard, so we have been trying to adopt
> > > > > whatever is available for GET.  This way we don't have to
> > > > > educate users on a new universe of what is possible.  I fully
> > > > > expect that lots of learnings are going to come in the industry
> > > > > here over time, and this will be revisited in the future.
> > > >
> > > > ?
> > > >
> > > > I am not proposing any new filter syntax.  I am saying that the
> > > > current filter nodes as defined in subscribed-notification cannot
> > > > be used to select nodes to subscribe to for changes.
> > >
> > > Understand.  Hopefully with the "xpath-selection" change proposed
> > > above, this will be covered.
> >
> > No, see above.
> >
> >
> > /martin
> >
> >
> >
> > >
> > > Eric
> > >
> > > > > > As for your question, I think such a filter should be defined
> > > > > > to return a node- set to which the client subscribe to
> > > > > > changes.  If any node (or subnode
> > > > > > to) in
> > > > > > this node-set changes, the notif will be sent.  Then the
> > > > > > question about value comparision is not relevant anymore.
> > > > >
> > > > > Excellent, on-change should only send an update if the results
> > > > > of the subscription filter have changed since the previous push.
> > > > > It is quite possible that an object has been created and then
> > > > > deleted since the last push.
> > > >
> > > > I don't understand what you're trying to say with these sentences
> > > > (but since the first word was "Excellent" maybe it's ok ;)
> > > >
> > > >
> > > > /martin
> > > >
> > > >
> > > > > Representing this was the genesis of Alex's question.
> > > > >
> > > > > Eric
> > > > >
> > > > > > /martin
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > In effect, the filter specifies which data nodes to consider
> > > > > > > when sending updates.
> > > > > > >
> > > > > > > ---Alex
> > > > > > >
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Martin Bjorklund [mailto:[e-mail]]
> > > > > > > Sent: Wednesday, May 17, 2017 11:50 PM
> > > > > > > To: Alexander Clemm <[e-mail]>
> > > > > > > Cc: [e-mail]
> > > > > > > Subject: Re: [Netconf] In an update, when is a delete a delete?
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > Alexander Clemm <[e-mail]> wrote:
> > > > > > > > Hello all,
> > > > > > > >
> > > > > > > > In updating the YANG-Push document
> > > > > > > > (draft-ietf-netconf-yang-push), we have come across one
> > > > > > > > issue that we wanted to raise with the working group.
> > > > > > > >
> > > > > > > > As part of an on-change subscription, update records
> > > > > > > > reflect the type of change (e.g. whether the value of an
> > > > > > > > object has changed, or whether an object was created or
> > > > > > > > deleted); a subscription allows also to specify whether
> > > > > > > > interested only in specific types of changes (for example,
> > > > > > > > only creates and deleted but no value
> > > > changes).
> > > > > > > >
> > > > > > > > At the same time, a subscription filter specifies which
> > > > > > > > objects to include as part of a subscription and which not.
> > > > > > >
> > > > > > > Hmm, which filter are you talking about?  The only XPath
> > > > > > > filter I find in the current set of documents is the
> > > > > > > "ietf-subscribed-notifications:xpath" filter type (which btw
> > > > > > > is sorely underspecified).  Section 2.2 of
> > > > > > > draft-ietf-netconf-subscribed-notifications-02 says:
> > > > > > >
> > > > > > >    Events which evaluate to "true" as a
> > > > > > >    result of the evaluation by the filter must traverse the filter in
> > > > > > >    their entirety.
> > > > > > >
> > > > > > > It's not clear what this means, but my guess is that this is
> > > > > > > supposed to work like the old RFC 5277 filters, where the
> > > > > > > filter expression is evaluated on the notification contents,
> > > > > > > and if the expression returns "true" (for XPath filters this
> > > > > > > means converting the results to a boolean), then the
> > > > > > > notification is sent, otherwise not.
> > > > > > >
> > > > > > > But it seems you are referring to some other filter which
> > > > > > > would be used to select a node set for which changes are reported?
> > > > > > >
> > > > > > > I would like to understand which filter mechanism you mean
> > > > > > > before having an opinion in this matter.
> > > > > > >
> > > > > > >
> > > > > > > /martin
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > (Really, it is not
> > > > > > > > so much of a "filter" on a stream that is generated
> > > > > > > > independently of the filter, than it is a policy of which
> > > > > > > > objects to include as part of subscribed update records.)
> > > > > > > > However, a subscription filter (such as
> > > > > > > > XPath) can be used to also specify a value filter, which
> > > > > > > > will include or exclude objects based on their current
> > > > > > > > value. This makes it possible to e.g. subscribe to an
> > > > > > > > object "foo" but only if its value is 5.
> > > > > > > >
> > > > > > > > Now, this means that the same object could be included in
> > > > > > > > one update, but excluded in another update, due to its
> > > > > > > > value no longer meeting the filter criteria.  For example,
> > > > > > > > if foo's value changes from 5 to 3 in one cycle, a
> > > > > > > > periodic subscription will no longer include foo in its next update.
> > > > > > > > The question now concerns how to properly handle this in
> > > > > > > > the case of an on-change
> > > > subscription.
> > > > > > > >
> > > > > > > > One possibility concerns reporting the fact that "foo" no
> > > > > > > > longer meets the subscription criteria and is no longer
> > > > > > > > included in the update record as a "delete" event.  If foo's value
> again becomes "5"
> > > > > > > > at a later point in time, that would be reported as a "create"
> > > > > > > > event.  If foo's value changes again from 5 at a later
> > > > > > > > point in time and then changes back to 3 before the time
> > > > > > > > of the update (perhaps because the value changed during
> > > > > > > > the dampening interval), it would be reported as another
> > > > > > > > "delete" event (without ever reporting a create event).
> > > > > > > > On the other hand, if foo's value changed from 3 to
> > > > > > > > 6 and back again, nothing would be reported because it did
> > > > > > > > not meet the filter criteria at any point in time.
> > > > > > > >
> > > > > > > > >From the perspective of the receiver this may make sense
> > > > > > > > >if it is synching its copy of the state.  However, from
> > > > > > > > >the perspective of the publisher, the object was never
> > > > > > > > >created or deleted - only its value changed, and the case
> > > > > > > > >when the object was truly created or deleted can no
> > > > > > > > >longer be distinguished from the case when its value
> > > > > > changed.
> > > > > > > > >A "create" simply means "an object now meets a filter
> > > > > > > > >criteria, that was not reported in the previous cycle"
> > > > > > > > >(which does not mean that the object was actually created
> > > > > > > > >- it may have been created, or it may have simply undergone a
> value change).
> > > > > > > >
> > > > > > > > An alternative (let's call it alternative 2) is therefore
> > > > > > > > to make a distinction between whether an object was
> > > > > > > > created or deleted, or whether its value fell in or out of a filter
> range.
> > > > > > > > This appears semantically cleaner.  However, it will
> > > > > > > > require modifying the encoding to allow for distinction
> > > > > > > > between those cases (currently, just plain patch encoding is used).
> > > > > > > >
> > > > > > > > A third alternative is to let filters select only data
> > > > > > > > nodes to subscribe to, and separate out the value filter
> > > > > > > > (or disallow it as a feature altogether).  This
> > > > > > > > alternative has the drawback of being less conceptually
> > > > > > > > powerful, even if it may be easier to
> > > > implement.
> > > > > > > >
> > > > > > > > Thoughts?  Any preferences between 1, 2, and 3?
> > > > > > > > --- Alex
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > Netconf mailing list
> > > > > > [e-mail]
> > > > > > https://www.ietf.org/mailman/listinfo/netconf
> > > > >
> > >
> >