Re: [Netconf] In an update, when is a delete a delete?

["Eric Voit (evoit)" <[e-mail]>]

Hi Andy, Alex, & Martin,

From: Andy Bierman, May 24, 2017 6:55 PM

Hi,

Let's compare to what 5277 already has defined.
I agree with Martin that new functionality should be defined with
new terms and new specs.

First, RFC 5277 allows the event-type to be filtered.
This is basic functionality every server should support.
(e.g., select netconf-config-change, ignore netconf-session-start)
This event type test is not separate from the event content filter.
Both are boolean tests.

<eric> Agree that event tests of both boolean and selection are needed (like RFC-5277), and that both may allow events to pass.  I have a proposal for some text to cover this below (see green).

Notification filters for YANG Push need to select the event-types used,
such as push-change-update.  It would be nice if this was handled automatically,
instead of listing them in the <establish-subscription>.

<evoit> Either you provide a stream or a datastore as a target for the filter.  Even then, a filter-type remains needed to decode the syntax of the filter anyxml.

Traditional notification filters would require that the structure of an event such
as push-change-update be known to the developer in full detail.
IMO this is a bad approach.  A new filter type that selects data nodes
would be much better.  The current proposal is easy to under-specify
and (IMO) nearly impossible for operators to use.

<evoit> Agree that a subtree filter for events is needed.

Andy



On Wed, May 24, 2017 at 3:25 PM, Alexander Clemm <[e-mail]<mailto:[e-mail]>> wrote:
Let me briefly summarize where I think we are. The thread has clearly moved on from the initial issue of how to distinguish between causes for updates to no longer include a given data node (because it was deleted, or because it changed its value to no longer match a filter) to other issues.

For those other issues, I think there are in fact two separate items that we are trying to discuss at the same time, which are really orthogonal to one another:

- The first item concerns the concept of a "filter" vs a "selector".  A filter is what gets specified for any subscription to notifications, which defines which notifications a subscriber wants to receive.  The filter is applied to the notification as a whole, i.e. either the notification is delivered or it is not.  It does not apply to subsets of contents within the notification.  A selector, on the other hand, is used to specify updates of which data nodes to include in a YANG-push subscription.  The same update notification could include updates of several data nodes, hence a filter applied to the notification as a whole would be inappropriate here - the semantics is slightly different: as a subscriber, "of  which data nodes would you like to receive updates", not "which notifications would you like to receive".

<evoit> fully agree.

Filters and selectors can be specified using the same syntax.  We are faced with a choice between specifying a single construct as part of a subscription, which is treated as a filter (in case of a "regular" subscription for notification updates) or as a selector (in case of a subscription to datastore updates) , or having separate objects, i.e. adding a separate "selector" construct for YANG-push.  Using a single object amounts to overloading.  It is more compact but with arguably a more complex semantics.  Using separate object results in a model that is more verbose, but has arguably simpler semantics.

<evoit> I don’t think the semantics are simpler.  In all cases, you already need a filter-type to be able to interpret the filter syntax (e.g., xpath vs. subtree). With this as a given, the current draft’s overloading does have a few benefits:

·        Prohibits population of both an event and a selection filter – there is only filter object.

·        Simple: 1:1 mapping from RPC filter to the single yang data node filter object.

·        It turns out that moving to filter identities from a strict explicit-case based hierarchy simplified the model groupings.  This removed a full page from the YANG model from the previous version.  This is goodness.

·        Nicely extensible via identity hierarchies (rather than also adding a parallel case structure which needs independent treatment).

I can live with it if people want to add an explicit subtyping based on ‘selection’ and ‘test’.  But it seems to add complexity and rigidity to me.

- The second item concerns the issue of whether the filter/selector construct has a dynamic type or a static type.  This is where the issue of identity vs case statement etc comes in.  In case of a dynamic type, we use two objects:  One object of a generic type (anydata) holds the filter/selector construct itself, the second object specifies how to interpret it, i.e. which type it is.  (In case of "overloading, we can also make explicit the distinction whether it is a filter or a selector).  That second object is an identityref, referencing one of the identities that designates the filter/selector type.  In case of a static type, we use a case statement (to distinguish which specific type it is).

<evoit> There are so many variations of filter types, having two independent, intersecting constructs doesn’t seem beneficial to me.


The argument that Eric is making is that we should have a single overloaded object that can serve as a filter or a selector depending on the context and whether it is used in a notification subscription or a YANG-push subscription, and that we use a dynamic type including an identityref that indicates whether the object serves as a selector or a filter.  From my perspective, I feel that not overloading may be conceptually a bit "cleaner", but at the end of the day I am fine either way.  And I am not entirely sure, Martin, what you are proposing.  Either way, we should document the issue and our choice clearly.

<eric> Yes.  If we do go with the overloading, here is how I would construct it…:

(a) In subscribed-notifications, update the identity filters so the following are covered:

Identity filter;
    Identity test-filter {base filter;}
       Identity xpath-test-filter {base  test-filter;}
       Identity  subtree-test-filter {base test-filter;}

(b) In YANG Push, update the identity filters so the following are covered:

Identity node-selection-criteria {base sn:filter;}
    Identity subtree-node-selection-criteria {base node-selection-criteria;}
    Identity xpath-node-selection-criteria {base node-selection-criteria;}

(c) in subscribed-notifications, improve the definition of  “anyxml filter” to
"Evaluation criteria encoded in a syntax of a filter-type.  If the filter is applied against an event stream and there is a non-empty or positive result, the event is passed along.  If the filter is applied against a datastore for periodic extracts, the resulting node-set result is passed along. If the filter is applied against a datastore looking for changes, deltas from the last update in the form of a patch result are passed along."


--- Alex


-----Original Message-----
From: Martin Bjorklund [mailto:[e-mail]<mailto:[e-mail]>]
Sent: Wednesday, May 24, 2017 12:36 AM
To: [e-mail]<mailto:[e-mail]>
Cc: Alexander Clemm <[e-mail]<mailto:[e-mail]>>; [e-mail]<mailto:[e-mail]>
Subject: Re: [Netconf] In an update, when is a delete a delete?

"Eric Voit (evoit)" <[e-mail]<mailto:[e-mail]>> wrote:
> > From: Martin Bjorklund, May 23, 2017 1:57 PM
> >
> > "Eric Voit (evoit)" <[e-mail]<mailto:[e-mail]>> wrote:
> > > > Martin Bjorklund, May 22, 2017 3:30 PM
> > > >
> > > > Alexander Clemm <[e-mail]<mailto:[e-mail]>> wrote:
> > > > > Hi Martin,
> > > > >
> > > > > Almost overlooked your question below.  What is meant by the
> > > > > filter is specified in section 3.5 of the YANG-Push document.
> > > > >
> > > > > "Only a single filter can be applied to a subscription at a time.
> > > > > The following filter types are included in the yang-push data model:
> > > > > [subtree] [xpath]"
> > > >
> > > > Actually, only "subtree" is defined in yang-push, "xpath" is
> > > > defined in subscribed-notifications.
> > >
> > > At the top of yang-push page 7, xpath selection is described.  Is
> > > there something you feel missing?
> >
> > Yes.  First of all, the YANG module defines an identity called
> > "xpath", based on "sn:filter".  So this filter has nothing to do
> > with selecting nodes in a datastore; this filter is used to match
> > against a generated notification record.
>
> To address this we could split the xpath identity into two types:
> "xpath-selection" and "xpath-boolean".  These would have different
> definitions, but both reference
> http://www.w3.org/TR/1999/REC-xpath-19991116 .  The difference is that
> one results in a xpath node-set expression, and the other an xpath
> Boolean expression.  Make sense?

Not really.  In subscribed-notifications you have defined a generic filter mechanism, that is used to block/allow the sending of generated notifications to a subscriber.  This filter mechanism cannot be used to select nodes to subscribe to for changes in a datastore.  You need to define a separate mechansim for that in yang-push.  (Maybe not even call it "filter", but perhaps "selection").

<evoit> see above.  Selection is now part of the filter identify hierarchy.

Some comments on subscribed-notifications: This generic mechanism allows various filter syntaxes.  This generic mechanism needs to explain what is required by a filter syntax definition (an identity, evaluation rules).  In section 2.2 the document says that two filter syntaxes are supported, but only one is defined (xpath).  It needs to explain that a filter is supposed to return true or false (this part RFC 5277 got right, see section 3.6).  Also, the current module has the filter in an anyxml node; it is not clear how an XPath expression is encoded in anyxml.

I am hoping to leave xpath expression encoding up to the vendor.  I suspect that attempting to come up with some cross-vendor parseable encoding mechanism is unlikely to make the xpath filter definitions themselves cross-vendor anyway.  We should wait for someone who wants to pick up general filtering as a topic rather than addressing this piecemeal.


> > Second, the XPath filter is sorely underspecified.  The XPath
> > context is not described,
>
> I understand and agree on your comment about the context.  The intent
> here is to provide equivalent capabilities of a GET.

I understand that.

> As it would be a
> huge undertaking to try to consolidate an industry-wide view of the
> minimal xpath syntax and capabilities in networking

Ehh... yes?  What does this have to do with specifiying the XPath context?

> , I am hoping this
> doesn't fall under the umbrella of YANG subscription.  I would be glad
> to support someone who wishes to take this up though.
>
> > the expected result data type is not defined, and it is not
> > described how the result is supposed to be used.
>
> As for the result, the anydata output should be provided to the
> subscriber (with appropriate security applied).

I was referring to the output of the filter evaluation.

> They can determine
> how to use it.  The preferred embodiment would be to maintain a local
> extract of the Publisher's datastore (as defined by the filter).
>
> > > >  [Side note - I think this is wrong, subscribed-notifications
> > > > should also define "subtree".]
> > > >
> > > > But these filters are used by the server to decide if a certain
> > > > notification that has been generated will be sent to the client
> > > > or not.
> > >
> > > Yes, the filters in subscribed-notification are supposed to give a
> > > boolean indication as to whether a specific event should traverse
> > > the filter in its entirety. RFC6241 section 6 subtree filters are
> > > written to provide a subset of content.  I suppose it would be
> > > possible to define an event-based subtree-filter-type where a
> > > non-null result of the subtree filter means that a particular
> > > event should traverse that filter.  Is this what you are suggesting?
> >
> > Yes.  Note that this is already provided by RFC 5277, and I have
> > always assumed that this new work will provide at least the same
> > functions as RFC
> > 5277 (and more).  (But note that the XPath filter is underspecified
> > also in RFC
> > 5277...)
>
> I also want to make sure that a non-null result from a filter allows
> the event to pass.  I suspect that an xpath Boolean filter could be
> designed to do this, but I will tweak the subscribed-notifications
> text so that unnecessary filtering expression complexity is not
> artificially required.

Please make sure you understand how subtree filters and XPath filters work in RFC 5277.  There is nothing wrong with that functionality.

<evoit> 100% agree.  I will make sure the text covers this.

Eric


> > > > If you want to define filters to specifify which nodes to
> > > > subscribe to, I think you need to define new filters, not try to
> > > > reuese these notification filters.
> > >
> > > Filtering syntax is hard, so we have been trying to adopt whatever
> > > is available for GET.  This way we don't have to educate users on
> > > a new universe of what is possible.  I fully expect that lots of
> > > learnings are going to come in the industry here over time, and
> > > this will be revisited in the future.
> >
> > ?
> >
> > I am not proposing any new filter syntax.  I am saying that the
> > current filter nodes as defined in subscribed-notification cannot be
> > used to select nodes to subscribe to for changes.
>
> Understand.  Hopefully with the "xpath-selection" change proposed
> above, this will be covered.

No, see above.


/martin



>
> Eric
>
> > > > As for your question, I think such a filter should be defined to
> > > > return a node- set to which the client subscribe to changes.  If
> > > > any node (or subnode
> > > > to) in
> > > > this node-set changes, the notif will be sent.  Then the
> > > > question about value comparision is not relevant anymore.
> > >
> > > Excellent, on-change should only send an update if the results of
> > > the subscription filter have changed since the previous push.  It
> > > is quite possible that an object has been created and then deleted
> > > since the last push.
> >
> > I don't understand what you're trying to say with these sentences
> > (but since the first word was "Excellent" maybe it's ok ;)
> >
> >
> > /martin
> >
> >
> > > Representing this was the genesis of Alex's question.
> > >
> > > Eric
> > >
> > > > /martin
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > >
> > > > > In effect, the filter specifies which data nodes to consider
> > > > > when sending updates.
> > > > >
> > > > > ---Alex
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Martin Bjorklund [mailto:[e-mail]<mailto:[e-mail]>]
> > > > > Sent: Wednesday, May 17, 2017 11:50 PM
> > > > > To: Alexander Clemm <[e-mail]<mailto:[e-mail]>>
> > > > > Cc: [e-mail]<mailto:[e-mail]>
> > > > > Subject: Re: [Netconf] In an update, when is a delete a delete?
> > > > >
> > > > > Hi,
> > > > >
> > > > > Alexander Clemm <[e-mail]<mailto:[e-mail]>> wrote:
> > > > > > Hello all,
> > > > > >
> > > > > > In updating the YANG-Push document
> > > > > > (draft-ietf-netconf-yang-push), we have come across one
> > > > > > issue that we wanted to raise with the working group.
> > > > > >
> > > > > > As part of an on-change subscription, update records reflect
> > > > > > the type of change (e.g. whether the value of an object has
> > > > > > changed, or whether an object was created or deleted); a
> > > > > > subscription allows also to specify whether interested only
> > > > > > in specific types of changes (for example, only creates and
> > > > > > deleted but no value
> > changes).
> > > > > >
> > > > > > At the same time, a subscription filter specifies which
> > > > > > objects to include as part of a subscription and which not.
> > > > >
> > > > > Hmm, which filter are you talking about?  The only XPath
> > > > > filter I find in the current set of documents is the
> > > > > "ietf-subscribed-notifications:xpath" filter type (which btw
> > > > > is sorely underspecified).  Section 2.2 of
> > > > > draft-ietf-netconf-subscribed-notifications-02 says:
> > > > >
> > > > >    Events which evaluate to "true" as a
> > > > >    result of the evaluation by the filter must traverse the filter in
> > > > >    their entirety.
> > > > >
> > > > > It's not clear what this means, but my guess is that this is
> > > > > supposed to work like the old RFC 5277 filters, where the
> > > > > filter expression is evaluated on the notification contents,
> > > > > and if the expression returns "true" (for XPath filters this
> > > > > means converting the results to a boolean), then the
> > > > > notification is sent, otherwise not.
> > > > >
> > > > > But it seems you are referring to some other filter which
> > > > > would be used to select a node set for which changes are reported?
> > > > >
> > > > > I would like to understand which filter mechanism you mean
> > > > > before having an opinion in this matter.
> > > > >
> > > > >
> > > > > /martin
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > > (Really, it is not
> > > > > > so much of a "filter" on a stream that is generated
> > > > > > independently of the filter, than it is a policy of which
> > > > > > objects to include as part of subscribed update records.)
> > > > > > However, a subscription filter (such as
> > > > > > XPath) can be used to also specify a value filter, which
> > > > > > will include or exclude objects based on their current
> > > > > > value. This makes it possible to e.g. subscribe to an object
> > > > > > "foo" but only if its value is 5.
> > > > > >
> > > > > > Now, this means that the same object could be included in
> > > > > > one update, but excluded in another update, due to its value
> > > > > > no longer meeting the filter criteria.  For example, if
> > > > > > foo's value changes from 5 to 3 in one cycle, a periodic
> > > > > > subscription will no longer include foo in its next update.
> > > > > > The question now concerns how to properly handle this in the
> > > > > > case of an on-change
> > subscription.
> > > > > >
> > > > > > One possibility concerns reporting the fact that "foo" no
> > > > > > longer meets the subscription criteria and is no longer
> > > > > > included in the update record as a "delete" event.  If foo's value again becomes "5"
> > > > > > at a later point in time, that would be reported as a "create"
> > > > > > event.  If foo's value changes again from 5 at a later point
> > > > > > in time and then changes back to 3 before the time of the
> > > > > > update (perhaps because the value changed during the
> > > > > > dampening interval), it would be reported as another
> > > > > > "delete" event (without ever reporting a create event).  On
> > > > > > the other hand, if foo's value changed from 3 to
> > > > > > 6 and back again, nothing would be reported because it did
> > > > > > not meet the filter criteria at any point in time.
> > > > > >
> > > > > > >From the perspective of the receiver this may make sense if
> > > > > > >it is synching its copy of the state.  However, from the
> > > > > > >perspective of the publisher, the object was never created
> > > > > > >or deleted - only its value changed, and the case when the
> > > > > > >object was truly created or deleted can no longer be
> > > > > > >distinguished from the case when its value
> > > > changed.
> > > > > > >A "create" simply means "an object now meets a filter
> > > > > > >criteria, that was not reported in the previous cycle"
> > > > > > >(which does not mean that the object was actually created -
> > > > > > >it may have been created, or it may have simply undergone a value change).
> > > > > >
> > > > > > An alternative (let's call it alternative 2) is therefore to
> > > > > > make a distinction between whether an object was created or
> > > > > > deleted, or whether its value fell in or out of a filter range.
> > > > > > This appears semantically cleaner.  However, it will require
> > > > > > modifying the encoding to allow for distinction between
> > > > > > those cases (currently, just plain patch encoding is used).
> > > > > >
> > > > > > A third alternative is to let filters select only data nodes
> > > > > > to subscribe to, and separate out the value filter (or
> > > > > > disallow it as a feature altogether).  This alternative has
> > > > > > the drawback of being less conceptually powerful, even if it
> > > > > > may be easier to
> > implement.
> > > > > >
> > > > > > Thoughts?  Any preferences between 1, 2, and 3?
> > > > > > --- Alex
> > > > > >
> > > > >
> > > >
> > > > _______________________________________________
> > > > Netconf mailing list
> > > > [e-mail]<mailto:[e-mail]>
> > > > https://www.ietf.org/mailman/listinfo/netconf
> > >
>

_______________________________________________
Netconf mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/netconf