[Cats] Re: draft-ietf-cats-metric-definition-10 early Opsdir review

[Fung Lim <[e-mail]>]

hi Kehan,

Thank you very much for your edits.

I saw the new draft incorporates a dedicated Operational Considerations 
section (Section 7), which addresses the gap identified previously. 
Sections 7.1-7.3 helps consolidate operator facing guidance on 
multi-vendor normalization negotiation, metric level selection, as well 
as metric collection failure handling. These are meaningful improvements 
that significantly improve deployability.

I've identified some remaining gaps which relate mainly to verifying 
correct operation, management interoperability and migration/coexistence 
with TE/ALTO.

Comments 1, 3 and 5 can be closed. Comments 2 and 4 was addressed partially.
Comments 6, 7, 8, 9, and 10 remain open.

Details:

1. Missing Operational Considerations

Disposition: Closed

7.1–7.3 cover intra-domain normalization, advertisement overhead, and 
collection failure. This suffice for the structural comment from -10.


2. Cross-vendor score comparability

Disposition: Partial

Intra-domain offline negotiation is aligned with the 25 Aug list 
discussion. Inter-domain use and comparability warning are not 
explicitly mentioned.

e.g.

- when scores are safe across, not just within a single administrative 
domain
- when Level 1 category metrics should be preferred over Level 2 global 
scores in multi-vendor settings
- operational calibration: who runs benchmarks, how often, what 
deviation triggers re-calibration, etc.


3. Faults / troubleshooting

Disposition: Closed

7.3 adds fallback to traditional network-aware steering and recommended 
alarms on SEC-4 freshness failures. This suffice to close the -10 fault 
comment.


4. Verifying correct operation

Disposition: Partial

Registry calibration (e.g. score deviation less than 0.1) is described 
as an implementation activity. There is no mention of operator procedure 
for production checks.

e.g. How should operators periodically validate that normalized scores 
reflect actual instance capabilities? Correlating score trends with 
observed application QoE metrics, etc.


5. Metric update frequency / overhead

Disposition: Closed

Per-instance advertisement at most once per measurement window, plus 
guidance to use longer windows, Level 2-only advertisement, and 
statistical fields, is enough for the -10 churn comment.


6. Management interoperability

Disposition: Not addressed

Whether a YANG data model for CATS metric configuration, policy, and 
exposed state is needed or planned
How normalized scores and runtime parameters (Measurement_Window, 
aggregation weights) would be configured via standard management interfaces
Prometheus / NETCONF / IPFIX remain Level 0 collection examples. 
Operators may benefit with a short statement of what is configuration 
(normalization bounds, weights, Measurement_Window), what is operational 
state (current scores), and what is statistics (historical trends), even 
if YANG is deferred.


7. Migration / coexistence with TE/ALTO

Disposition: Not addressed

7.3's fallback to "traditional network-aware steering" does not say how 
Level 1 communication metrics relate to IGP TE metrics (RFC 7471), how 
to adopt incrementally without dual-metric steering conflicts, or 
whether there is any mapping to ALTO cost metrics (RFC 9439).


8. Role assignment / registry formatting

Disposition: Not addressed

Section 6.4.3.6 still has "Service contact instace" (likely C-SMA). 
Section 6.1.3.6 still has inline "### Output" after the C-PS role text.


9. Editorial and reference nits

Disposition: Not addressed

Still present from -10, plus new broken 4.4.x pointers after Section 4 
was renumbered to 4.3.x, and the self-reference in 10.1 still points at 
an older version.


10. Accounting / performance management

Disposition: Not addressed

No rollover guidance for Measurement_Window accumulators, no retention 
guidance, no sentence on whether billing/SLA export is in scope.
One sentence that billing/SLA export is out of scope would close that 
sub-item.


I hope the above is useful.

regards,
Lim Fung



On 2026-09-04 16:26, Kehan Yao wrote:
> Hi Fung,
>
> Thanks so much for the review.
> Authors have uploaded a new version with a new section on operational 
> considerations, including several important aspects that you mentioned 
> below. Please further review if this version is okay.
>
> Best regards,
> Kehan
>
> Internet-Draft draft-ietf-cats-metric-definition-11.txt is now 
> available. It
> is a work item of the Computing-Aware Traffic Steering (CATS) WG of 
> the IETF.
>
>    Title:   CATS Metrics Definition
>    Authors: Kehan Yao
>             Cheng Li
>             L. M. Contreras
>             Jordi Ros-Giralt
>             Guanming Zeng
>    Name:    draft-ietf-cats-metric-definition-11.txt
>    Pages:   50
>    Dates:   2026-09-04
>
> Abstract:
>
>    Computing-Aware Traffic Steering (CATS) is a traffic engineering
>    approach that optimizes the steering of traffic to a service instance
>    by considering the dynamic state of computing and network resources.
>    To enable such decisions, CATS components exchange metrics that
>    describe resource conditions affecting service instance selection.
>    This document focuses on compute and communication metrics for CATS
>    and defines a hierarchical abstraction of these metrics to improve
>    interoperability, scalability, and operational simplicity.  It does
>    not aim to standardize raw infrastructure (Level 0) metrics; instead,
>    it specifies higher-level representations that can be derived from
>    raw measurements using aggregation and normalization functions.
>
> The IETF datatracker status page for this Internet-Draft is:
> https://datatracker.ietf.org/doc/draft-ietf-cats-metric-definition/
>
> There is also an HTML version available at:
> https://www.ietf.org/archive/id/draft-ietf-cats-metric-definition-11.html
>
> A diff from the previous version is available at:
> https://author-tools.ietf.org/iddiff?url2=draft-ietf-cats-metric-definition-11
>
> Fung Lim via Datatracker <[e-mail]> 于2026年7月3日周五 12:44写道：
>
>     Document: draft-ietf-cats-metric-definition
>     Title: CATS Metrics Definition
>     Reviewer: Fung Lim
>     Review result: Has Issues
>
>     Hi,
>
>     I have been selected as the Operational Directorate (opsdir)
>     reviewer for this
>     Internet-Draft.
>
>     The Operational Directorate reviews all operational and
>     management-related
>     Internet-Drafts to ensure alignment with operational best
>     practices and that
>     adequate operational considerations are covered.
>
>     A complete set of _"Guidelines for Considering Operations and
>     Management in
>     IETF Specifications"_ can be found at
>     https://datatracker.ietf.org/doc/draft-ietf-opsawg-rfc5706bis/.
>
>     While these comments are primarily for the Operations and
>     Management Area
>     Directors (Ops ADs), the authors should consider them alongside
>     other feedback
>     received.
>
>     - Document: draft-ietf-cats-metric-definition-10
>     - Reviewer: Fung Lim
>     - Review Date: 3rd July 2026
>     - Intended Status: Standards Track
>
>     ---
>
>     ## Summary
>
>     - Has Issues: I have concerns about this document that I think
>     should be
>     resolved before publication.
>
>     This document defines a three-level hierarchical abstraction for
>     Computing-Aware Traffic Steering (CATS) metrics, standardizing
>     Level 1 category
>     metrics (computing, communication, service, composed) and a Level
>     2 global
>     normalized score, along with a metric field framework. The design is
>     well-motivated: it deliberately avoids standardizing fine-grained
>     Level 0
>     metrics while providing interoperable encodings and normative
>     score semantics
>     at higher levels. The metric framework described in this document
>     is sound and
>     several operational concerns are implicitly addressed.
>
>     However, as a Standards Track document that defines inputs to
>     traffic-steering
>     decisions, it requires a dedicated Operational Considerations
>     section (per
>     Section 3.1 of draft-ietf-opsawg-rfc5706bis). Several operational
>     topics that
>     are partially scattered across design principles, registry
>     entries, and
>     security text would benefit from consolidation into
>     operator-facing guidance.
>     In particular, cross-vendor score comparability, metric collection
>     failure
>     handling, control-plane churn from metric updates, and management
>     interface
>     expectations should be addressed.
>
>     1. Missing Operational Considerations section (§3.1 of rfc5706bis)
>
>     Per draft-ietf-opsawg-rfc5706bis, a technical specification
>     advancing as an
>     IETF RFC MUST include an Operational Considerations section
>     immediately before
>     Security Considerations, or explicit boilerplate with rationale if
>     none apply.
>     This document jumps from Section 6 (CATS Metric Registry Entries)
>     directly to
>     Section 7 (Security Considerations).
>
>     For a Standards Track document whose metrics directly influence
>     instance
>     selection and traffic steering, an Operational Considerations
>     section is
>     essential rather than optional. The document already contains
>     operationally
>     relevant material — score semantics (Section 4.3), level selection
>     trade-offs
>     (Section 5), role assignments (C-SMA, C-NMA, C-PS in Section 6),
>     calibration
>     guidance, and security requirements — but it is not consolidated where
>     operators will look for it.
>
>     2. Cross-vendor score comparability needs explicit operational
>     guidance (§4.1,
>     §4.5 of rfc5706bis)
>
>     Section 4.3 warns that normalized scores "may not be directly
>     comparable across
>     different implementations" because aggregation and normalization
>     functions are
>     implementation-specific. Section 4.2.1 and 4.2.2 state these
>     functions "are not
>     standardized" and are "controlled by operator policies."
>
>     This is a fundamental operational constraint that affects
>     multi-vendor and
>     multi-provider CATS deployments. An operator comparing a score of
>     7 from
>     Provider A against a score of 7 from Provider B has no assurance
>     of equivalent
>     capability. The normative 0–3 / 4–7 / 8–10 steering bands (Section
>     4.3) help
>     within a single administrative domain but do not solve
>     cross-domain comparison.
>
>     The Operational Considerations section should address:
>
>     a. When scores are safe to use for steering (intra-domain only vs.
>     inter-domain)
>     b. Whether operators should prefer Level 1 category metrics over
>     Level 2 global
>     scores for multi-vendor environments c. How calibration
>     (referenced in Section
>     6 registry entries) should be applied operationally — who runs
>     benchmarks, how
>     often, and what deviation thresholds trigger re-calibration d.
>     Whether future
>     CATS signaling documents will carry normalization policy
>     identifiers to aid
>     comparison
>
>     3. No fault management or troubleshooting guidance (§5.4 of
>     rfc5706bis)
>
>     CATS metrics drive steering decisions. The document does not
>     discuss how
>     operators detect or respond to metric-related faults:
>
>     Stale or missing metrics from C-SMA or C-NMA (Section 7 addresses
>     freshness at
>     the security level via SEC-4, but not operational fallback
>     behavior) Collection
>     agent failures or telemetry pipeline outages Score anomalies
>     (sudden drops,
>     stuck values, contradictory Level 1 vs. Level 2 scores) Integrity
>     violation
>     handling (SEC-1 says metrics SHOULD trigger alerts, but no alert
>     schema or
>     operational procedure) Per §5.4 of rfc5706bis, essential faults,
>     alarms, and
>     events should be identified. At minimum, the document should discuss:
>
>     a. Fallback steering behavior when metrics are unavailable or fail
>     freshness
>     checks b. Whether last-known-good values are used or instances are
>     de-prioritized c. Recommended health indicators for C-SMA, C-NMA,
>     and C-PS
>     components
>
>     4. No guidance on verifying correct operation (§4.7 of rfc5706bis)
>
>     The registry entries include calibration methods (e.g., "score
>     deviation ...
>     less than 0.1 over repeated test rounds") but the document does
>     not describe
>     how operators verify end-to-end CATS metric operation in production:
>
>     Active vs. passive validation approaches
>     Synthetic workload injection for score validation
>     Correlation between normalized scores and observed application QoE
>     Simple health checks (e.g., confirming C-SMA is publishing
>     expected metric
>     types at expected intervals) Section 6 registry entries reference
>     benchmark
>     calibration but treat it as an implementation activity without
>     operator-facing
>     procedures.
>
>     5. Metric update frequency and control-plane impact not addressed
>     (§4.5 of
>     rfc5706bis)
>
>     Section 3.1 discusses the trade-off between metric update
>     frequency and
>     "signaling overhead, reducing scalability of the metric
>     distribution protocol."
>     Section 6 suggests continuous sampling every 10 seconds with a default
>     Measurement_Window of 10 seconds.
>
>     However, the document does not quantify or guide:
>
>     Recommended maximum metric advertisement rate per service instance
>     Impact of high churn on CATS control-plane protocols (to be defined in
>     companion documents) When coarser sampling or Level 2-only
>     advertisement is
>     preferable to reduce overhead Interaction between Statistics field
>     choices
>     (max, min, mean, cur) and update frequency This is particularly
>     important
>     because CATS may scale to hundreds or thousands of service
>     instances (Section
>     5).
>
>     Nits
>
>     6. No management interoperability discussion (§5.2 of rfc5706bis)
>
>     The document references Prometheus, NETCONF, and IPFIX for Level 0
>     metric
>     collection but does not discuss:
>
>     Whether a YANG data model for CATS metric configuration, policy,
>     and exposed
>     state is needed or planned How normalized scores and runtime
>     parameters
>     (Measurement_Window, aggregation weights) would be configured via
>     standard
>     management interfaces Distinction between configuration data
>     (normalization
>     bounds, weights), operational state (current scores), and statistics
>     (historical trends) For a Standards Track metric definition
>     consumed by network
>     elements, some discussion of management model expectations — even
>     if deferred
>     to a companion document — would help operators plan deployments.
>
>     7. Migration and coexistence with existing metrics (§4.3 of
>     rfc5706bis)
>
>     The Introduction references existing TE metrics (RFC 7471) and network
>     performance registries (RFC 8911, RFC 8912). The document does not
>     discuss:
>
>     How CATS Level 1 communication metrics relate to or subsume
>     existing IGP TE
>     metrics Whether operators can incrementally adopt CATS metrics
>     alongside
>     traditional TE without dual-metric steering conflicts Mapping or
>     equivalence
>     between ALTO performance cost metrics (RFC 9439) and CATS
>     normalized scores A
>     brief migration narrative would help operators who already run TE
>     metric
>     extensions.
>
>     8. Role assignment ambiguity in registry entries
>
>     Section 6.4.3.6 assigns metric calculation to "Service contact
>     instance" rather
>     than C-SMA (used consistently elsewhere). Section 6.1.3.6 contains
>     a formatting
>     error where ### Output appears inline after the C-PS role description,
>     disrupting the registry structure.
>
>     Section 6.4.3.6 <http://6.4.3.6>:
>
>     Service contact instace: Collects Level 0 service raw metrics...
>
>     This should likely reference C-SMA for consistency with the framework
>     terminology.
>
>     9. Editorial and reference nits
>
>     "Section Section 4.4" (Section 4.1, line 605) — duplicate word
>     "To-be-assgined" → "To-be-assigned" (Sections 6.1.4.2, 6.2.5.2,
>     6.3.5.2)
>     "Service contact instace" → "Service contact instance" (Section
>     6.4.3.6)
>     Normative self-reference [I-D.ietf-cats-metric-definition] points
>     to draft-08
>     (Section 9.1) — should reference the current draft version
>     Appendix A examples
>     use inconsistent Source capitalization ("Directly measured" vs.
>     lowercase
>     'directly measured' in Section 4.1) IANA placeholder "XXX" identifiers
>     throughout Section 6 — expected for draft stage, but Operational
>     Considerations
>     should note operators need stable identifiers before deployment
>
>     10. Accounting and performance management partially addressed
>
>     Section 6 registry entries define singleton output values and
>     calibration
>     targets, which is a good start. However:
>
>     No counter rollover behavior for Measurement_Window accumulators
>     No guidance on historical metric retention for capacity planning
>     No discussion of metric export for billing or SLA reporting (may
>     be N/A for
>     some deployments, but worth a sentence)
>
>     Recommendations
>
>     1. Add an Operational Considerations section immediately before
>     Security
>     Considerations. Consolidate scattered operational material and
>     address: metric
>     level selection, runtime parameter defaults, cross-vendor score
>     limitations,
>     metric collection failures and fallback steering, update rate
>     guidance, and
>     coexistence with existing TE metrics.
>
>     2. Document fault management and fallback behavior. Specify what
>     happens when
>     C-SMA/C-NMA stops publishing, metrics fail freshness checks
>     (SEC-4), or scores
>     are contradictory. Define minimum health indicators and
>     recommended alert
>     conditions.
>
>     3. Add verification guidance. Describe how operators validate that
>     normalized
>     scores reflect actual instance capability — referencing the
>     calibration methods
>     already in Section 6 registry entries and extending them into
>     deployable
>     procedures (synthetic workloads, periodic re-calibration, QoE
>     correlation).
>
>     4. Clarify cross-domain and multi-vendor deployment constraints.
>     Expand Section
>     4.3's comparability warning into explicit operational guidance:
>     when scores are
>     trustworthy, when Level 1 categories are preferable to Level 2
>     global scores,
>     and how calibration maintains consistency within an administrative
>     domain.
>
>     5. Discuss management model expectations. Even if a YANG module is
>     deferred,
>     state whether configuration of normalization bounds, aggregation
>     weights, and
>     Measurement_Window is expected via NETCONF/RESTCONF or
>     operator-local policy,
>     and how config vs. operational state vs. statistics should be
>     distinguished.
>
>     6. Address metric advertisement churn. Provide guidance on
>     sampling intervals,
>     maximum update rates, and when to prefer Level 2 over Level 1 to limit
>     control-plane overhead at scale.
>
>     7. Fix editorial issues in Section 6 registry entries (inline ###
>     Output, role
>     naming inconsistency, typos) and update the normative
>     self-reference to the
>     current draft version.
>
>     8. Add a brief migration narrative explaining how CATS metrics
>     complement or
>     replace existing IGP TE metrics and ALTO cost metrics during
>     incremental
>     adoption.
>
>     I hope these comments are useful and constructive. The
>     hierarchical metric
>     framework is well-conceived and addresses a genuine
>     interoperability need for
>     CATS; adding consolidated operational guidance will significantly
>     improve its
>     deployability in multi-vendor environments.
>
>     Lim Fung
>
>
>     -- 
>     Cats mailing list -- [e-mail]
>     To unsubscribe send an email to [e-mail]
>