Re: [Netconf] In an update, when is a delete a delete?

[Martin Bjorklund <[e-mail]>]

"Eric Voit (evoit)" <[e-mail]> wrote:
> > From: Martin Bjorklund, May 25, 2017 5:10 AM
> > 
> > Hi,
> > 
> > Thank you Alex for a well-written summary.
> > 
> > Alexander Clemm <[e-mail]> wrote:
> > > Let me briefly summarize where I think we are. The thread has clearly
> > > moved on from the initial issue of how to distinguish between causes
> > > for updates to no longer include a given data node (because it was
> > > deleted, or because it changed its value to no longer match a filter)
> > > to other issues.
> > >
> > > For those other issues, I think there are in fact two separate items
> > > that we are trying to discuss at the same time, which are really
> > > orthogonal to one another:
> > >
> > > - The first item concerns the concept of a "filter" vs a "selector".
> > >   A filter is what gets specified for any subscription to
> > >   notifications, which defines which notifications a subscriber
> > >   wants to receive.  The filter is applied to the notification as a
> > >   whole, i.e. either the notification is delivered or it is not.  It
> > >   does not apply to subsets of contents within the notification.  A
> > >   selector, on the other hand, is used to specify updates of which
> > >   data nodes to include in a YANG-push subscription.  The same
> > >   update notification could include updates of several data nodes,
> > >   hence a filter applied to the notification as a whole would be
> > >   inappropriate here - the semantics is slightly different: as a
> > >   subscriber, "of which data nodes would you like to receive
> > >   updates", not "which notifications would you like to receive".
> > >
> > > Filters and selectors can be specified using the same syntax.  We are
> > > faced with a choice between specifying a single construct as part of a
> > > subscription, which is treated as a filter (in case of a "regular"
> > > subscription for notification updates) or as a selector (in case of a
> > > subscription to datastore updates) , or having separate objects, i.e.
> > > adding a separate "selector" construct for YANG-push.  Using a single
> > > object amounts to overloading.  It is more compact but with arguably a
> > > more complex semantics.  Using separate object results in a model that
> > > is more verbose, but has arguably simpler semantics.
> > >
> > > - The second item concerns the issue of whether the filter/selector
> > >   construct has a dynamic type or a static type.  This is where the
> > >   issue of identity vs case statement etc comes in.  In case of a
> > >   dynamic type, we use two objects: One object of a generic type
> > >   (anydata) holds the filter/selector construct itself, the second
> > >   object specifies how to interpret it, i.e. which type it is.  (In
> > >   case of "overloading, we can also make explicit the distinction
> > >   whether it is a filter or a selector).  That second object is an
> > >   identityref, referencing one of the identities that designates the
> > >   filter/selector type.  In case of a static type, we use a case
> > >   statement (to distinguish which specific type it is).
> > 
> > If you can make the identity-based solution work, that's fine with me.
> > But the
> > current text needs to work out the details.  It's not clear at all
> > what what the
> > contents of the "anyxml" node is supposed to be.
> > 
> > That said, even the "choice"-based solution is exensible, since it can
> > be
> > augmented.  Unless it turns out that the identity-based solution can
> > be made
> > to work well, I would prefer the "choice"-based solution.
> > 
> > > The argument that Eric is making is that we should have a single
> > > overloaded object that can serve as a filter or a selector depending
> > > on the context and whether it is used in a notification subscription
> > > or a YANG-push subscription, and that we use a dynamic type including
> > > an identityref that indicates whether the object serves as a selector
> > > or a filter.  From my perspective, I feel that not overloading may be
> > > conceptually a bit "cleaner", but at the end of the day I am fine
> > > either way.  And I am not entirely sure, Martin, what you are
> > > proposing.
> > 
> > I have a strong preference for clear semantics in separate objects.  I
> > think
> > overloading the semantics in this particular example would be a huge
> > mistake.
> > Just to mention one detail, the context for the filter and selector
> > are
> > completely different - the filter works on the contents of a
> > notification and the
> > selector works on the contents of a datastore.  The filter returns a
> > boolean
> > and the selector a node set.
> > (that was two details :)
> 
> Ok.  Unless anyone objects, let's go with separate objects then.  
> 
> We can still retain the identity hierarchy.  The result to would be
> something like:
> 
> In subscribed-notifications...:
> 
> Identity filter;
>     Identity test-filter {base filter;}
>        Identity xpath-test-filter {base  test-filter;}
>        Identity  subtree-test-filter {base test-filter;}

I'm not sure why you would have this hierarchy.

I would define a base identity "filter", and describe that a filter
works on the notification content, and returns a boolean if the filter
passed or not.  Also explain how the filter input data is supposed to
be defined (i.e., explain your current anyxml approach (I think you
will see that it doesn't quite works, and you'll end up with specific
per-filter-type parameters - see below)).  Then I'd define "xpath" and
"subtree" based on "filter", and use text from RFC 5277 to explain how
such filters are evaluated (+ more text on XPath, as noted earlier in
this thread.)  This would be done in subscribed-notifications.

In addition to this, in yang-push, I'd define similar identities for
"datastore-selector".

> "anyxml filter"
> Description "Event stream evaluation criteria encoded in a syntax of a
> supported type of test-filter.  If the filter is applied against an
> event stream and there is a non-empty or positive result, the event is
> passed along."
> 
> In YANG Push...:
> 
> Identity node-selection-criteria {base sn:filter;}
>     Identity subtree-node-selection-criteria {base
>     node-selection-criteria;}
>     Identity xpath-node-selection-criteria {base node-selection-criteria;}
> 
> "anyxml selector"
> Description "YANG datastore query encoded in the syntax of a supported
> type of node-selection-criteria.  If the selector is applied against a
> datastore for periodic extracts, the resulting node-set result is
> passed along. If the selector is applied against a datastore looking
> for changes, deltas from the last update in the form of a patch result
> are passed along."

So you're suggesting that a filter that might take some parameters
would not specify these parameters in YANG, but instead in plain text
-- Suppose I want to define a new regexp-based filter, which takes two
input parameters "dialect" and "expression".  How would this be done
if it has to go into an "anyxml" node?


/martin


> 
> Eric
> 
> 
> > /martin
> > 
> > 
> > > Either way, we should document the issue and our choice clearly.
> > >
> > > --- Alex
> > >
> > >
> > > -----Original Message-----
> > > From: Martin Bjorklund [mailto:[e-mail]]
> > > Sent: Wednesday, May 24, 2017 12:36 AM
> > > To: [e-mail]
> > > Cc: Alexander Clemm <[e-mail]>; [e-mail]
> > > Subject: Re: [Netconf] In an update, when is a delete a delete?
> > >
> > > "Eric Voit (evoit)" <[e-mail]> wrote:
> > > > > From: Martin Bjorklund, May 23, 2017 1:57 PM
> > > > >
> > > > > "Eric Voit (evoit)" <[e-mail]> wrote:
> > > > > > > Martin Bjorklund, May 22, 2017 3:30 PM
> > > > > > >
> > > > > > > Alexander Clemm <[e-mail]> wrote:
> > > > > > > > Hi Martin,
> > > > > > > >
> > > > > > > > Almost overlooked your question below.  What is meant by the
> > > > > > > > filter is specified in section 3.5 of the YANG-Push document.
> > > > > > > >
> > > > > > > > "Only a single filter can be applied to a subscription at a time.
> > > > > > > > The following filter types are included in the yang-push data
> > > > > > > > model:
> > > > > > > > [subtree] [xpath]"
> > > > > > >
> > > > > > > Actually, only "subtree" is defined in yang-push, "xpath" is
> > > > > > > defined in subscribed-notifications.
> > > > > >
> > > > > > At the top of yang-push page 7, xpath selection is described.
> > > > > > Is there something you feel missing?
> > > > >
> > > > > Yes.  First of all, the YANG module defines an identity called
> > > > > "xpath", based on "sn:filter".  So this filter has nothing to do
> > > > > with selecting nodes in a datastore; this filter is used to match
> > > > > against a generated notification record.
> > > >
> > > > To address this we could split the xpath identity into two types:
> > > > "xpath-selection" and "xpath-boolean".  These would have different
> > > > definitions, but both reference
> > > > http://www.w3.org/TR/1999/REC-xpath-19991116 .  The difference is
> > > > that one results in a xpath node-set expression, and the other an
> > > > xpath Boolean expression.  Make sense?
> > >
> > > Not really.  In subscribed-notifications you have defined a generic
> > > filter
> > mechanism, that is used to block/allow the sending of generated
> > notifications
> > to a subscriber.  This filter mechanism cannot be used to select nodes
> > to
> > subscribe to for changes in a datastore.  You need to define a
> > separate
> > mechansim for that in yang-push.  (Maybe not even call it "filter",
> > but perhaps
> > "selection").
> > >
> > > Some comments on subscribed-notifications: This generic mechanism
> > allows various filter syntaxes.  This generic mechanism needs to
> > explain what
> > is required by a filter syntax definition (an identity, evaluation
> > rules).  In
> > section 2.2 the document says that two filter syntaxes are supported,
> > but only
> > one is defined (xpath).  It needs to explain that a filter is supposed
> > to return
> > true or false (this part RFC 5277 got right, see section 3.6).  Also,
> > the current
> > module has the filter in an anyxml node; it is not clear how an XPath
> > expression is encoded in anyxml.
> > >
> > > > > Second, the XPath filter is sorely underspecified.  The XPath
> > > > > context is not described,
> > > >
> > > > I understand and agree on your comment about the context.  The
> > > > intent here is to provide equivalent capabilities of a GET.
> > >
> > > I understand that.
> > >
> > > > As it would be a
> > > > huge undertaking to try to consolidate an industry-wide view of the
> > > > minimal xpath syntax and capabilities in networking
> > >
> > > Ehh... yes?  What does this have to do with specifiying the XPath
> > > context?
> > >
> > > > , I am hoping this
> > > > doesn't fall under the umbrella of YANG subscription.  I would be
> > > > glad to support someone who wishes to take this up though.
> > > >
> > > > > the expected result data type is not defined, and it is not
> > > > > described how the result is supposed to be used.
> > > >
> > > > As for the result, the anydata output should be provided to the
> > > > subscriber (with appropriate security applied).
> > >
> > > I was referring to the output of the filter evaluation.
> > >
> > > > They can determine
> > > > how to use it.  The preferred embodiment would be to maintain a
> > > > local extract of the Publisher's datastore (as defined by the filter).
> > > >
> > > > > > >  [Side note - I think this is wrong, subscribed-notifications
> > > > > > > should also define "subtree".]
> > > > > > >
> > > > > > > But these filters are used by the server to decide if a
> > > > > > > certain notification that has been generated will be sent to
> > > > > > > the client or not.
> > > > > >
> > > > > > Yes, the filters in subscribed-notification are supposed to give
> > > > > > a boolean indication as to whether a specific event should
> > > > > > traverse the filter in its entirety. RFC6241 section 6 subtree
> > > > > > filters are written to provide a subset of content.  I suppose
> > > > > > it would be possible to define an event-based
> > > > > > subtree-filter-type where a non-null result of the subtree
> > > > > > filter means that a particular event should traverse that filter.  Is
> > > > > > this
> > what you are suggesting?
> > > > >
> > > > > Yes.  Note that this is already provided by RFC 5277, and I have
> > > > > always assumed that this new work will provide at least the same
> > > > > functions as RFC
> > > > > 5277 (and more).  (But note that the XPath filter is
> > > > > underspecified also in RFC
> > > > > 5277...)
> > > >
> > > > I also want to make sure that a non-null result from a filter allows
> > > > the event to pass.  I suspect that an xpath Boolean filter could be
> > > > designed to do this, but I will tweak the subscribed-notifications
> > > > text so that unnecessary filtering expression complexity is not
> > > > artificially required.
> > >
> > > Please make sure you understand how subtree filters and XPath filters
> > > work
> > in RFC 5277.  There is nothing wrong with that functionality.
> > >
> > > > > > > If you want to define filters to specifify which nodes to
> > > > > > > subscribe to, I think you need to define new filters, not try
> > > > > > > to reuese these notification filters.
> > > > > >
> > > > > > Filtering syntax is hard, so we have been trying to adopt
> > > > > > whatever is available for GET.  This way we don't have to
> > > > > > educate users on a new universe of what is possible.  I fully
> > > > > > expect that lots of learnings are going to come in the industry
> > > > > > here over time, and this will be revisited in the future.
> > > > >
> > > > > ?
> > > > >
> > > > > I am not proposing any new filter syntax.  I am saying that the
> > > > > current filter nodes as defined in subscribed-notification cannot
> > > > > be used to select nodes to subscribe to for changes.
> > > >
> > > > Understand.  Hopefully with the "xpath-selection" change proposed
> > > > above, this will be covered.
> > >
> > > No, see above.
> > >
> > >
> > > /martin
> > >
> > >
> > >
> > > >
> > > > Eric
> > > >
> > > > > > > As for your question, I think such a filter should be defined
> > > > > > > to return a node- set to which the client subscribe to
> > > > > > > changes.  If any node (or subnode
> > > > > > > to) in
> > > > > > > this node-set changes, the notif will be sent.  Then the
> > > > > > > question about value comparision is not relevant anymore.
> > > > > >
> > > > > > Excellent, on-change should only send an update if the results
> > > > > > of the subscription filter have changed since the previous push.
> > > > > > It is quite possible that an object has been created and then
> > > > > > deleted since the last push.
> > > > >
> > > > > I don't understand what you're trying to say with these sentences
> > > > > (but since the first word was "Excellent" maybe it's ok ;)
> > > > >
> > > > >
> > > > > /martin
> > > > >
> > > > >
> > > > > > Representing this was the genesis of Alex's question.
> > > > > >
> > > > > > Eric
> > > > > >
> > > > > > > /martin
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > In effect, the filter specifies which data nodes to consider
> > > > > > > > when sending updates.
> > > > > > > >
> > > > > > > > ---Alex
> > > > > > > >
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Martin Bjorklund [mailto:[e-mail]]
> > > > > > > > Sent: Wednesday, May 17, 2017 11:50 PM
> > > > > > > > To: Alexander Clemm <[e-mail]>
> > > > > > > > Cc: [e-mail]
> > > > > > > > Subject: Re: [Netconf] In an update, when is a delete a delete?
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > Alexander Clemm <[e-mail]> wrote:
> > > > > > > > > Hello all,
> > > > > > > > >
> > > > > > > > > In updating the YANG-Push document
> > > > > > > > > (draft-ietf-netconf-yang-push), we have come across one
> > > > > > > > > issue that we wanted to raise with the working group.
> > > > > > > > >
> > > > > > > > > As part of an on-change subscription, update records
> > > > > > > > > reflect the type of change (e.g. whether the value of an
> > > > > > > > > object has changed, or whether an object was created or
> > > > > > > > > deleted); a subscription allows also to specify whether
> > > > > > > > > interested only in specific types of changes (for example,
> > > > > > > > > only creates and deleted but no value
> > > > > changes).
> > > > > > > > >
> > > > > > > > > At the same time, a subscription filter specifies which
> > > > > > > > > objects to include as part of a subscription and which not.
> > > > > > > >
> > > > > > > > Hmm, which filter are you talking about?  The only XPath
> > > > > > > > filter I find in the current set of documents is the
> > > > > > > > "ietf-subscribed-notifications:xpath" filter type (which btw
> > > > > > > > is sorely underspecified).  Section 2.2 of
> > > > > > > > draft-ietf-netconf-subscribed-notifications-02 says:
> > > > > > > >
> > > > > > > >    Events which evaluate to "true" as a
> > > > > > > >    result of the evaluation by the filter must traverse the
> > > > > > > >    filter in
> > > > > > > >    their entirety.
> > > > > > > >
> > > > > > > > It's not clear what this means, but my guess is that this is
> > > > > > > > supposed to work like the old RFC 5277 filters, where the
> > > > > > > > filter expression is evaluated on the notification contents,
> > > > > > > > and if the expression returns "true" (for XPath filters this
> > > > > > > > means converting the results to a boolean), then the
> > > > > > > > notification is sent, otherwise not.
> > > > > > > >
> > > > > > > > But it seems you are referring to some other filter which
> > > > > > > > would be used to select a node set for which changes are
> > > > > > > > reported?
> > > > > > > >
> > > > > > > > I would like to understand which filter mechanism you mean
> > > > > > > > before having an opinion in this matter.
> > > > > > > >
> > > > > > > >
> > > > > > > > /martin
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > > (Really, it is not
> > > > > > > > > so much of a "filter" on a stream that is generated
> > > > > > > > > independently of the filter, than it is a policy of which
> > > > > > > > > objects to include as part of subscribed update records.)
> > > > > > > > > However, a subscription filter (such as
> > > > > > > > > XPath) can be used to also specify a value filter, which
> > > > > > > > > will include or exclude objects based on their current
> > > > > > > > > value. This makes it possible to e.g. subscribe to an
> > > > > > > > > object "foo" but only if its value is 5.
> > > > > > > > >
> > > > > > > > > Now, this means that the same object could be included in
> > > > > > > > > one update, but excluded in another update, due to its
> > > > > > > > > value no longer meeting the filter criteria.  For example,
> > > > > > > > > if foo's value changes from 5 to 3 in one cycle, a
> > > > > > > > > periodic subscription will no longer include foo in its next
> > > > > > > > > update.
> > > > > > > > > The question now concerns how to properly handle this in
> > > > > > > > > the case of an on-change
> > > > > subscription.
> > > > > > > > >
> > > > > > > > > One possibility concerns reporting the fact that "foo" no
> > > > > > > > > longer meets the subscription criteria and is no longer
> > > > > > > > > included in the update record as a "delete" event.  If foo's
> > > > > > > > > value
> > again becomes "5"
> > > > > > > > > at a later point in time, that would be reported as a "create"
> > > > > > > > > event.  If foo's value changes again from 5 at a later
> > > > > > > > > point in time and then changes back to 3 before the time
> > > > > > > > > of the update (perhaps because the value changed during
> > > > > > > > > the dampening interval), it would be reported as another
> > > > > > > > > "delete" event (without ever reporting a create event).
> > > > > > > > > On the other hand, if foo's value changed from 3 to
> > > > > > > > > 6 and back again, nothing would be reported because it did
> > > > > > > > > not meet the filter criteria at any point in time.
> > > > > > > > >
> > > > > > > > > >From the perspective of the receiver this may make sense
> > > > > > > > > >if it is synching its copy of the state.  However, from
> > > > > > > > > >the perspective of the publisher, the object was never
> > > > > > > > > >created or deleted - only its value changed, and the case
> > > > > > > > > >when the object was truly created or deleted can no
> > > > > > > > > >longer be distinguished from the case when its value
> > > > > > > changed.
> > > > > > > > > >A "create" simply means "an object now meets a filter
> > > > > > > > > >criteria, that was not reported in the previous cycle"
> > > > > > > > > >(which does not mean that the object was actually created
> > > > > > > > > >- it may have been created, or it may have simply undergone a
> > value change).
> > > > > > > > >
> > > > > > > > > An alternative (let's call it alternative 2) is therefore
> > > > > > > > > to make a distinction between whether an object was
> > > > > > > > > created or deleted, or whether its value fell in or out of a
> > > > > > > > > filter
> > range.
> > > > > > > > > This appears semantically cleaner.  However, it will
> > > > > > > > > require modifying the encoding to allow for distinction
> > > > > > > > > between those cases (currently, just plain patch encoding is
> > > > > > > > > used).
> > > > > > > > >
> > > > > > > > > A third alternative is to let filters select only data
> > > > > > > > > nodes to subscribe to, and separate out the value filter
> > > > > > > > > (or disallow it as a feature altogether).  This
> > > > > > > > > alternative has the drawback of being less conceptually
> > > > > > > > > powerful, even if it may be easier to
> > > > > implement.
> > > > > > > > >
> > > > > > > > > Thoughts?  Any preferences between 1, 2, and 3?
> > > > > > > > > --- Alex
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > Netconf mailing list
> > > > > > > [e-mail]
> > > > > > > https://www.ietf.org/mailman/listinfo/netconf
> > > > > >
> > > >
> > >
>