Re: [netmod] yang-instance-file include-defaults leaf

["Rob Wilton (rwilton)" <[e-mail]>]

Balazs,

Would this text be sufficient to alleviate your concerns?

leaf includes-defaults {
   type ncwd:with-defaults-mode;
   default report-all;
   description "
     Indicates how data nodes with default values are
     represented for all data nodes contained in the
     instance-data-set.

     It uses the same definitions as per section 3 of RFC 6243,
     but applied in the context of an instance data file rather
     than a NETCONF request using the <with-defaults>
     parameter.

     For JSON files, the encoding of the 'report-all-tagged'
     option is as defined in section 4.8.9 of RFC 8040.";
   reference “With-defaults Capability for NETCONF, RFC 6243.”
}

I think that having a default value for the leaf is helpful, but I don’t mind if you prefer to set it to trim instead.

Andy, would this also be acceptable to you?

Regards,
Rob


From: Andy Bierman <[e-mail]>
Sent: 14 September 2021 18:09
To: Balázs Lengyel <[e-mail]>
Cc: Rob Wilton (rwilton) <[e-mail]>; NetMod WG <[e-mail]>
Subject: Re: [netmod] yang-instance-file include-defaults leaf

Hi,

I do not have time for a conference on this leaf.
You think this cut-and-paste is good engineering and not confusing to users.
I disagree.

IMO it is less confusing to use the correct typedef and declare in your leaf description-stmt
that the inclusion of defaults is not considered a retrieval, so 6243, sec 3 does not apply.


Andy


On Tue, Sep 14, 2021 at 8:24 AM Balázs Lengyel <[e-mail]<mailto:[e-mail]>> wrote:
Hello,
My view is that reusing the data ncwd:with-defaults-mode datatype, might be acceptable, but it is wrong, because:

  1.  As opposed to what Andy has wrote the data type does reference "RFC 6243<https://datatracker.ietf.org/doc/html/rfc6243>; Section 3<https://datatracker.ietf.org/doc/html/rfc6243#section-3>." (See line 58 in the YANG module). Section 3 contains a lot of concepts that are inappropriate for the instance-data draft (server, data retrieval, capabilities)
  2.  The typedef’s description is "Possible modes to report default data." From the 8 listed use-cases only two UC4 and UC5 is about reporting. In UC1,UC2,UC3, UC7,UC8 the instance-data-set is usually prepared by human SW developers which is not reporting. In UC6 we do not know whether this is about reporting, replying, or just sending the data.
In the instance-data draft we always avoided assumptions about who creates the instance-data and why. There are so many potential use-cases, that they may have very different processes creating and using the data.  The data is just present or absent. The wording in With-default RFC is too specific: “data is reported ...”.

  1.  report-all-tagged description contains “XML attribute” which is incorrect if we use JSON encoding. You can have an attribute in JSON, but here specifically an XML attribute is mentioned.
  2.  Rob, you stated  that it is preferred to use the 2119 terms SHALL, SHALL NOT in the enum descriptions (and I also think it is the good solution). So, I reworded the text, that if the data node is covered by the instance-data-set then the enum’s value MUST be adhered to. The ncwd:with-defaults-mode datatype uses wording like “is reported”, “not reported”.

That said, if this is the cost of getting this document to the IESG, I can accept the data-type. Please decide, so we can move forward.
I am willing to have a conference, would you arrange it?
Regards Balazs

From: Rob Wilton (rwilton) <[e-mail]<mailto:[e-mail]>>
Sent: 2021. szeptember 13., hétfő 12:05
To: Andy Bierman <[e-mail]<mailto:[e-mail]>>; Balázs Lengyel <[e-mail]<mailto:[e-mail]>>
Cc: NetMod WG <[e-mail]<mailto:[e-mail]>>
Subject: RE: [netmod] yang-instance-file include-defaults leaf

Hi Andy, Balazs,

I’m wondering whether it might be helpful to arrange a meeting to get to consensus on this issue (which I believe is important)?

The concern that I raised in my original AD review was that the data set is allowed to be partial with the RECOMMENDATION that default values are removed from the data set.  Hence, I think that those two choices combine to make it impossible for a consumer of an instance data file to know whether a data node has been excluded because it contained the default value vs it has been excluded because it not part of the data set.

However, I think that Andy is potentially coming from the perspective that if these instance data files represent configuration then understanding how RFC 6243 with-defaults handling applies is important.  To that end,  I quite like Andy’s suggestion of directly reusing the ncwd:with-defaults-mode type, because it guarantees that the definitions are entirely consistent, and I believe that it sufficient to address my concern.
     leaf includes-defaults {
         type ncwd:with-defaults-mode;
     }
Balazs, is there some way, perhaps with a tweaked description, or assigned default value for the leaf includes-defaults leaf, that this could be made to work for you?

Thanks,
Rob



From: Andy Bierman <[e-mail]<mailto:[e-mail]>>
Sent: 10 September 2021 18:03
To: Balázs Lengyel <[e-mail]<mailto:[e-mail]>>
Cc: Rob Wilton (rwilton) <[e-mail]<mailto:[e-mail]>>; NetMod WG <[e-mail]<mailto:[e-mail]>>
Subject: Re: [netmod] yang-instance-file include-defaults leaf



On Fri, Sep 10, 2021 at 5:15 AM Balázs Lengyel <[e-mail]<mailto:[e-mail]>> wrote:
See below as BALAZS4

From: Andy Bierman <[e-mail]<mailto:[e-mail]>>
Sent: 2021. szeptember 10., péntek 4:57
To: Balázs Lengyel <[e-mail]<mailto:[e-mail]>>
Cc: Rob Wilton (rwilton) <[e-mail]<mailto:[e-mail]>>; NetMod WG <[e-mail]<mailto:[e-mail]>>
Subject: Re: [netmod] yang-instance-file include-defaults leaf



On Thu, Sep 9, 2021 at 6:19 AM Balázs Lengyel <[e-mail]<mailto:[e-mail]>> wrote:
Hello Andy,
See below as BALAZS3.

From: Andy Bierman <[e-mail]<mailto:[e-mail]>>
Sent: 2021. augusztus 25., szerda 18:29
To: Balázs Lengyel <[e-mail]<mailto:[e-mail]>>
Cc: Rob Wilton (rwilton) <[e-mail]<mailto:[e-mail]>>; NetMod WG <[e-mail]<mailto:[e-mail]>>
Subject: Re: [netmod] yang-instance-file include-defaults leaf

Hi,

Here is the latest text. It is inconsistent with RFC 6243, section 3.
IMO the subsections should be cited instead of the copy-and-change approach.
BALAZS3: The 6243 sections contain parts about “data retrieval” “capabilities” or “conceptual data nodes set by the server”
These parts are not relevant for many of the instance data use cases, so I would like to stick with including text here.



I guess I do not understand why the "with-defaults" leaf or at leaf "with-defaults-mode" typedef
cannot be used. IMO this is bad practice.  Applications that already know how to deal
with defaults according to RFC 6243 should be supported.


     leaf includes-defaults {
         type ncwd:with-defaults-mode;
     }

I do not see any text in this typedef that is server-specific.
Andy
BALAZS4:  While I generally agree that duplication is a bad practice, I avoided using ncwd:with-defaults-mode because:

•        It references RFC 6243<https://datatracker.ietf.org/doc/html/rfc6243>; Section 3<https://datatracker.ietf.org/doc/html/rfc6243#section-3> which is full of inappropriate references (server, data retrieval, capabilities)

•        The typedef’s description is "Possible modes to report default data." However, in a number of use cases (e.g., UC2 Preloading default configuration data, UC7, U8) the data is not reported.

•        report-all-tagged description contains “XML attribute” which is incorrect if we use JSON encoding

•        I prefer to use the 2119 term SHALL, SHALL NOT in the enum descriptions.

Is this acceptable?




The data type I suggested does not have any references to section 3.
It does not have any MUST,SHOULD,MAY text at all.
The JSON encoding in RFC 7951 supports attributes.

IMO this leaf will cause a lot of confusion for users.
includes-defaults looks like with-defaults but it isn't.
It uses the exact same enums as with-defaults, but they mean different things.


Andy



   leaf includes-defaults {

         type enumeration {

           enum report-all {

             value 1;

             description

               "All data nodes SHOULD be included independent of

                 any default values.";



AB: should follow 6243, 3.1

           }

           enum trim {

             value 2;

             description

               "Data nodes that have a default defined and where

                 the actual value is the default value SHOULD

                 NOT be included.";



AB: should follow 6243, section 3.2

           }

           enum explicit {

             value 3;

             description

               "Data nodes that have a default defined and where

                 the actual value is the default value SHOULD NOT be

                 included. However, if the actual value was set by

                 a NETCONF client or other management application

                 by the way of an explicit management operation the

                 data node SHOULD be included.";



AB: should follow 6243, section 3.3

           }

         }

         description

           "As instance-data-sets MAY contain incomplete data sets,

             thus any data node MAY be absent.



             Providing the instance-data-set intends to contain a

             full data set, this leaf specifies whether the data set

             includes data nodes that have a default defined and

             where the actual value is the same as the default value.



             Data nodes that have no default values should always

             be included.

             Data nodes that have a default value, but the actual

             value is not equal to the schema defined default

             should always be included.";





AB: The last paragraph should be removed or changed. Why are these nodes special?

Nodes that are actually present and do not contain the YANG default

are not relevant to this object.

BALAZS3: OK



         reference

           "RFC 6243<https://datatracker.ietf.org/doc/html/rfc6243>: With-defaults Capability for NETCONF";

       }

The best way to indicate a representation of a YANG default in the data set is to include
the "default" attribute in each default node. https://datatracker.ietf.org/doc/html/rfc6243#section-6
This actually works for explicit mode and leaf-lists (unlike the current solution).
BALAZS3: OK. Added report-all-tagged enum to includes-defaults leaf.
Here is the current proposed text:
------------------------------------------------------------
    leaf includes-defaults {
      type enumeration {
        enum report-all {
          value 1;
          description
            "All data nodes SHALL be included independent of
              any default values, if the data node
              is covered by the instance-data-set.";
        }
        enum report-all-tagged  {
          value 2;
          description
            "All data nodes SHALL be included independent of
              any default values if the data node
              is covered by the instance-data-set.
              Any nodes considered to be default data SHALL
              contain a 'default' attribute set to 'true'";
        }
        enum trim {
          value 3;
          description
            "Data nodes that have a default defined and where
              the actual value is equal to the schema default
              value SHALL NOT be included.";
        }
        enum explicit {
          value 4;
          description
            "Data nodes where the actual value is equal to the
              schema default value SHALL NOT be included.
              However, if the actual value was set by a NETCONF
              client or other management application by the way
              of an explicit management operation, the data node
              SHALL be included, if the data node is covered by
              the instance-data-set.";
        }
      }
      description
        "An instance-data-set may contain or exclude default
          data. This leaf indicates whether default data is
          included.

          As instance-data-sets MAY contain incomplete data
          sets: MAY NOT cover all data nodes. A leaf or
          leaf-list MAY be absent because the instance-data-set
          does not intend to include the data node independent
          of default handling.";
      reference
        "RFC 6243: With-defaults Capability for NETCONF
         RFC 8040 :RESTCONF Protocol";
    }
------------------------------------------------------------


On Tue, Aug 24, 2021 at 1:41 AM Balázs Lengyel <[e-mail]<mailto:[e-mail]>> wrote:
Hello Andy,
In the -17 I removed the default value for includes-defaults as you proposed.

I am not sure I understand the rest of the comments as instance-file-format does not use the concept of “basic-mode”. It has a single leaf to indicate what is the situation with defaults in the specific instance-data-set.
As this is not a live server request/reply situation we do not want to specify a basic and additional modes, we just want to specify the handling used for this specific instance data set.

Regards Balazs

From: Andy Bierman <[e-mail]<mailto:[e-mail]>>
Sent: 2021. augusztus 23., hétfő 18:58
To: Balázs Lengyel <[e-mail]<mailto:[e-mail]>>
Cc: Rob Wilton (rwilton) <[e-mail]<mailto:[e-mail]>>; NetMod WG <[e-mail]<mailto:[e-mail]>>
Subject: Re: [netmod] yang-instance-file include-defaults leaf



On Mon, Aug 23, 2021 at 5:17 AM Balázs Lengyel <[e-mail]<mailto:[e-mail]>> wrote:
Hello Rob,
I think this won’t fly.
In sections 1.2 and 2 we state:

“Instance data files MAY contain partial data sets.”

Which is important for many use-cases.  This means you cannot say that a default value will or must be included, as they might be omitted because they are not part of the partial data set.

In a way it is difficult to separate between leaves that are missing because

-        They are not part of the partial data-set

-        They are omitted because they have the default value and one of the trim or explicit options is used

If this becomes important the report-all options shall be used.




I thought we already agreed there cannot be a default or there is no way to
represent "no defaults added".

Note that "report-all" is not useful if basic-mode=explicit, since a leaf reporting the YANG default
could be set by the client.  Only report-all-tagged will clearly identify defaults in this case.

Also note that if basic-mode=report-all then there will be no defaults ever reported.
This mode means the server does not consider any node to be a default and always returns
every node (if with-defaults used or not).


This is the reason I used the SHOULD word.

Regards Balazs

Andy


From: Rob Wilton (rwilton) <[e-mail]<mailto:[e-mail]>>
Sent: 2021. augusztus 23., hétfő 12:27
To: Balázs Lengyel <[e-mail]<mailto:[e-mail]>>; Andy Bierman <[e-mail]<mailto:[e-mail]>>; NetMod WG <[e-mail]<mailto:[e-mail]>>
Subject: RE: [netmod] yang-instance-file include-defaults leaf

Hi Balazs, Andy, Netmod,

Sorry for the delayed response.  I would still like to strength the description of the defaults.  E.g., RFC 6243 uses MUSTs rather than SHOULDs.

Hence, I have generated some proposed alternative descriptions, that are somewhat stricter, but also more generically focussed only on the default values.

With these definitions, I think that we could define the “include-defaults” default value to be “explicit”, since if the leaf if not included, then I think that this effectively what the meaning would be anyway.


In particular, I would propose changing the descriptions as follows:

       leaf includes-defaults {
         type enumeration {
           enum report-all {
             value 1;
             description
               "All data nodes SHOULD be included independent of
                 any default values.";
           }
           enum trim {
             value 2;
             description
               "Data nodes that have a default defined and where
                 the actual value is the default value SHOULD
                 NOT be included.";
           }
           enum explicit {
             value 3;
             description
               "Data nodes that have a default defined and where
                 the actual value is the default value SHOULD NOT be
                 included. However, if the actual value was set by
                 a NETCONF client or other management application
                 by the way of an explicit management operation the
                 data node SHOULD be included.";
           }
         }

Proposed:

       leaf includes-defaults {
         type enumeration {
           enum report-all {
             value 1;
             description
               "The instance data set includes all data nodes,
                including those that contain the schema default.”;
           }
           enum trim {
             value 2;
             description
               "The instance data set excludes all data nodes
                that contain the schema default.";
           }
           enum explicit {
             value 3;
             description
               "The instance data set may include some data nodes
                that match the schema default and may exclude some
                data nodes that match the schema default.”;
           }
         }
         description
           "This leaf provides an indication of how default data
            is presented within an instance data set, modelled on
            RFC 6243.

            Interpretation of the use of defaults depends on the
            context of what the instance data set represents.

            E.g., if the instance data set represents configuration,
            Then include-defaults aligns to the meaning of the
            default-handling basic modes in RFC 6243.  If the
            instance data set represents operational data from the
            operational state datastore [RFC 8342], then
            include-defaults aligns to the definition of that
            datastore in RFC 8342.”;

Would text along these lines work?

Thanks,
Rob


From: Balázs Lengyel <[e-mail]<mailto:[e-mail]>>
Sent: 28 July 2021 23:08
To: Rob Wilton (rwilton) <[e-mail]<mailto:[e-mail]>>; Andy Bierman <[e-mail]<mailto:[e-mail]>>
Cc: NetMod WG <[e-mail]<mailto:[e-mail]>>
Subject: RE: [netmod] yang-instance-file include-defaults leaf

Hello Rob,
Removing the “default trim;” will address Andy’s comment.

Your in-use-values is very specific to one of the use-cases: reading/documenting operational values. It is not useful for the other use-cases. I think the “documenting operational datastore” use-case could be handled by indicating the includes-defaults=report-all. Case (i) would contain the value case (ii) will not.
Regards Balazs

From: Rob Wilton (rwilton) <[e-mail]<mailto:[e-mail]>>
Sent: 2021. július 27., kedd 17:38
To: Andy Bierman <[e-mail]<mailto:[e-mail]>>; Balázs Lengyel <[e-mail]<mailto:[e-mail]>>
Cc: NetMod WG <[e-mail]<mailto:[e-mail]>>
Subject: RE: [netmod] yang-instance-file include-defaults leaf

Hi Andy, Balazs,

So, the reason that I want a flag to indicate whether default values are in use is because of this definition of operational in RFC 8342:

   Requests to retrieve nodes from <operational> always return the value
   in use if the node exists, regardless of any default value specified
   in the YANG module.  If no value is returned for a given node, then
   this implies that the node is not used by the device.

It was written this way because otherwise a consumer of operational data cannot differentiate between:

(i)               This value is not present because it matches the default value specified in the YANG module, and

(ii)              This value is not present because the server has failed to return it for some reason (e.g., perhaps the daemon that would have provided this value is down or not available, or perhaps it is a bug, or perhaps it is not implemented and is a missing deviation).

So, I think that in some cases, the absence of a data node does not necessarily mean that the default value is in effect, and I wanted the instance-data document to be able to contain and correctly report this data.

I think that this behaviour could be captured by a single leaf.  Another way of articulating this would be:

leaf in-use-values {
  type boolean;
  default false;
  description
    “Only if set to true, the absence of a value in the
     instance data for a given data node implies that the
    node is not used rather than implicitly taking the
     default value specified by any corresponding
    ‘default’ statement specified in the YANG schema.”;
}

With this, I’m not sure whether we need the “includes-default” leaf currently specified in the draft, but if we do, then I would think that leaf should be entirely optional, i.e., without the default “trim”.

Regards,
Rob


From: Andy Bierman <[e-mail]<mailto:[e-mail]>>
Sent: 10 July 2021 17:41
To: Rob Wilton (rwilton) <[e-mail]<mailto:[e-mail]>>
Cc: NetMod WG <[e-mail]<mailto:[e-mail]>>; Balázs Lengyel <[e-mail]<mailto:[e-mail]>>
Subject: Re: [netmod] yang-instance-file include-defaults leaf



On Fri, Jul 9, 2021 at 5:23 AM Rob Wilton (rwilton) <[e-mail]<mailto:[e-mail]>> wrote:
Andy,

Yes, when I suggested this, I was thinking that a boolean flag might be sufficient.  My point being that automatically filtering out default values isn’t always the right thing to do.



The solution is simple.
Get rid of the inappropriate "default trim" statement.

If the leaf is present then it identifies the basic-mode that was used to include defaults.
If not then the information is either not known, not applicable, or defaults were not added.

The "default" statement is a bug because there is no default basic-mode.
All of the basic-modes are in use in deployments and no camp has ever
been able to convince the others that theirs is right.


Andy

E.g., something along these lines:

leaf exclude-defaults {
  type boolean;
  default true;
  description
    “Can be used to reduce the size of the content data file.

      When unset or set to true, data nodes that have a default defined and
      where the actual value is the default value are excluded from the content
      data.

      When set to false, data nodes with default value are not filtered, and
      may appear in the content data.”
}

Would this satisfy your concern?

Regards,
Rob


From: netmod <[e-mail]<mailto:[e-mail]>> On Behalf Of Andy Bierman
Sent: 08 July 2021 18:16
To: NetMod WG <[e-mail]<mailto:[e-mail]>>
Subject: [netmod] yang-instance-file include-defaults leaf

Hi,

The module has this object:


    leaf includes-defaults {

       type enumeration {

         enum report-all {

           value 1;

           description

             "All data nodes SHOULD be included independent of

               any default values.";

         }

         enum trim {

           value 2;

           description

             "Data nodes that have a default defined and where

               the actual value is the default value SHOULD

               NOT be included.";

         }

         enum explicit {

           value 3;

           description

             "Data nodes that have a default defined and where

               the actual value is the default value SHOULD NOT be

               included. However, if the actual value was set by

               a NETCONF client or other management application

               by the way of an explicit management operation the

               data node SHOULD be included.";

         }

       }

       default trim;


The draft is extremely server-centric, like most IETF standards, but this
leaf is too server-centric to ignore.

Consider the possibility that the source of the file is NOT a NETCONF server.
This data may not be known so the default of "trim" may not be correct.

IMO this leaf is noise because any tool that knows the schema will also
know the YANG defaults.  The solution is incomplete anyway because
the presence of a leaf that has a YANG default is not enough.
The  "report-all-tagged" mode must be used to identify defaults.
IMO this leaf should be removed, but at least add an enum called "unknown".


Andy