[Cats] Re: draft-ietf-cats-metric-definition-10 early Opsdir review

[Yao Kehan <[e-mail]>]

Hi Fung,



Thank you so much for these very constructive review comments. We authors will carefully consider your suggestions as well as RFC 5706bis, and add a dedicated Operational Considerations section in the next revision, addressing and resolving the related issues accordingly. 



Best regards,

Kehan



发件人：Fung Lim via Datatracker  <[e-mail]>收件人：ops-dir <[e-mail]>抄　送: cats <[e-mail]>,"draft-ietf-cats-metric-definition.all" <[e-mail]>发送时间：2026-07-03 12:44:50主题：[Cats] draft-ietf-cats-metric-definition-10 early Opsdir reviewDocument: draft-ietf-cats-metric-definitionTitle: CATS Metrics DefinitionReviewer: Fung LimReview result: Has IssuesHi,I have been selected as the Operational Directorate (opsdir) reviewer for thisInternet-Draft.The Operational Directorate reviews all operational and management-relatedInternet-Drafts to ensure alignment with operational best practices and thatadequate operational considerations are covered.A complete set of _"Guidelines for Considering Operations and Management inIETF Specifications"_ can be found athttps://datatracker.ietf.org/doc/draft-ietf-opsawg-rfc5706bis/.While these comments are primarily for the Operations and Management AreaDirectors (Ops ADs), the authors should consider them alongside other feedbackreceived.- Document: draft-ietf-cats-metric-definition-10- Reviewer: Fung Lim- Review Date: 3rd July 2026- Intended Status: Standards Track---## Summary- Has Issues: I have concerns about this document that I think should beresolved before publication.This document defines a three-level hierarchical abstraction forComputing-Aware Traffic Steering (CATS) metrics, standardizing Level 1 categorymetrics (computing, communication, service, composed) and a Level 2 globalnormalized score, along with a metric field framework. The design iswell-motivated: it deliberately avoids standardizing fine-grained Level 0metrics while providing interoperable encodings and normative score semanticsat higher levels. The metric framework described in this document is sound andseveral operational concerns are implicitly addressed.However, as a Standards Track document that defines inputs to traffic-steeringdecisions, it requires a dedicated Operational Considerations section (perSection 3.1 of draft-ietf-opsawg-rfc5706bis). Several operational topics thatare partially scattered across design principles, registry entries, andsecurity text would benefit from consolidation into operator-facing guidance.In particular, cross-vendor score comparability, metric collection failurehandling, control-plane churn from metric updates, and management interfaceexpectations should be addressed.1. Missing Operational Considerations section (§3.1 of rfc5706bis)Per draft-ietf-opsawg-rfc5706bis, a technical specification advancing as anIETF RFC MUST include an Operational Considerations section immediately beforeSecurity Considerations, or explicit boilerplate with rationale if none apply.This document jumps from Section 6 (CATS Metric Registry Entries) directly toSection 7 (Security Considerations).For a Standards Track document whose metrics directly influence instanceselection and traffic steering, an Operational Considerations section isessential rather than optional. The document already contains operationallyrelevant material — score semantics (Section 4.3), level selection trade-offs(Section 5), role assignments (C-SMA, C-NMA, C-PS in Section 6), calibrationguidance, and security requirements — but it is not consolidated whereoperators will look for it.2. Cross-vendor score comparability needs explicit operational guidance (§4.1,§4.5 of rfc5706bis)Section 4.3 warns that normalized scores "may not be directly comparable acrossdifferent implementations" because aggregation and normalization functions areimplementation-specific. Section 4.2.1 and 4.2.2 state these functions "are notstandardized" and are "controlled by operator policies."This is a fundamental operational constraint that affects multi-vendor andmulti-provider CATS deployments. An operator comparing a score of 7 fromProvider A against a score of 7 from Provider B has no assurance of equivalentcapability. The normative 0–3 / 4–7 / 8–10 steering bands (Section 4.3) helpwithin a single administrative domain but do not solve cross-domain comparison.The Operational Considerations section should address:a. When scores are safe to use for steering (intra-domain only vs. inter-domain)b. Whether operators should prefer Level 1 category metrics over Level 2 globalscores for multi-vendor environments c. How calibration (referenced in Section6 registry entries) should be applied operationally — who runs benchmarks, howoften, and what deviation thresholds trigger re-calibration d. Whether futureCATS signaling documents will carry normalization policy identifiers to aidcomparison3. No fault management or troubleshooting guidance (§5.4 of rfc5706bis)CATS metrics drive steering decisions. The document does not discuss howoperators detect or respond to metric-related faults:Stale or missing metrics from C-SMA or C-NMA (Section 7 addresses freshness atthe security level via SEC-4, but not operational fallback behavior) Collectionagent failures or telemetry pipeline outages Score anomalies (sudden drops,stuck values, contradictory Level 1 vs. Level 2 scores) Integrity violationhandling (SEC-1 says metrics SHOULD trigger alerts, but no alert schema oroperational procedure) Per §5.4 of rfc5706bis, essential faults, alarms, andevents should be identified. At minimum, the document should discuss:a. Fallback steering behavior when metrics are unavailable or fail freshnesschecks b. Whether last-known-good values are used or instances arede-prioritized c. Recommended health indicators for C-SMA, C-NMA, and C-PScomponents4. No guidance on verifying correct operation (§4.7 of rfc5706bis)The registry entries include calibration methods (e.g., "score deviation ...less than 0.1 over repeated test rounds") but the document does not describehow operators verify end-to-end CATS metric operation in production:Active vs. passive validation approachesSynthetic workload injection for score validationCorrelation between normalized scores and observed application QoESimple health checks (e.g., confirming C-SMA is publishing expected metrictypes at expected intervals) Section 6 registry entries reference benchmarkcalibration but treat it as an implementation activity without operator-facingprocedures.5. Metric update frequency and control-plane impact not addressed (§4.5 ofrfc5706bis)Section 3.1 discusses the trade-off between metric update frequency and"signaling overhead, reducing scalability of the metric distribution protocol."Section 6 suggests continuous sampling every 10 seconds with a defaultMeasurement_Window of 10 seconds.However, the document does not quantify or guide:Recommended maximum metric advertisement rate per service instanceImpact of high churn on CATS control-plane protocols (to be defined incompanion documents) When coarser sampling or Level 2-only advertisement ispreferable to reduce overhead Interaction between Statistics field choices(max, min, mean, cur) and update frequency This is particularly importantbecause CATS may scale to hundreds or thousands of service instances (Section5).Nits6. No management interoperability discussion (§5.2 of rfc5706bis)The document references Prometheus, NETCONF, and IPFIX for Level 0 metriccollection but does not discuss:Whether a YANG data model for CATS metric configuration, policy, and exposedstate is needed or planned How normalized scores and runtime parameters(Measurement_Window, aggregation weights) would be configured via standardmanagement interfaces Distinction between configuration data (normalizationbounds, weights), operational state (current scores), and statistics(historical trends) For a Standards Track metric definition consumed by networkelements, some discussion of management model expectations — even if deferredto a companion document — would help operators plan deployments.7. Migration and coexistence with existing metrics (§4.3 of rfc5706bis)The Introduction references existing TE metrics (RFC 7471) and networkperformance registries (RFC 8911, RFC 8912). The document does not discuss:How CATS Level 1 communication metrics relate to or subsume existing IGP TEmetrics Whether operators can incrementally adopt CATS metrics alongsidetraditional TE without dual-metric steering conflicts Mapping or equivalencebetween ALTO performance cost metrics (RFC 9439) and CATS normalized scores Abrief migration narrative would help operators who already run TE metricextensions.8. Role assignment ambiguity in registry entriesSection 6.4.3.6 assigns metric calculation to "Service contact instance" ratherthan C-SMA (used consistently elsewhere). Section 6.1.3.6 contains a formattingerror where ### Output appears inline after the C-PS role description,disrupting the registry structure.Section 6.4.3.6:Service contact instace: Collects Level 0 service raw metrics...This should likely reference C-SMA for consistency with the frameworkterminology.9. Editorial and reference nits"Section Section 4.4" (Section 4.1, line 605) — duplicate word"To-be-assgined" → "To-be-assigned" (Sections 6.1.4.2, 6.2.5.2, 6.3.5.2)"Service contact instace" → "Service contact instance" (Section 6.4.3.6)Normative self-reference [I-D.ietf-cats-metric-definition] points to draft-08(Section 9.1) — should reference the current draft version Appendix A examplesuse inconsistent Source capitalization ("Directly measured" vs. lowercase39directly measured39 in Section 4.1) IANA placeholder "XXX" identifiersthroughout Section 6 — expected for draft stage, but Operational Considerationsshould note operators need stable identifiers before deployment10. Accounting and performance management partially addressedSection 6 registry entries define singleton output values and calibrationtargets, which is a good start. However:No counter rollover behavior for Measurement_Window accumulatorsNo guidance on historical metric retention for capacity planningNo discussion of metric export for billing or SLA reporting (may be N/A forsome deployments, but worth a sentence)Recommendations1. Add an Operational Considerations section immediately before SecurityConsiderations. Consolidate scattered operational material and address: metriclevel selection, runtime parameter defaults, cross-vendor score limitations,metric collection failures and fallback steering, update rate guidance, andcoexistence with existing TE metrics.2. Document fault management and fallback behavior. Specify what happens whenC-SMA/C-NMA stops publishing, metrics fail freshness checks (SEC-4), or scoresare contradictory. Define minimum health indicators and recommended alertconditions.3. Add verification guidance. Describe how operators validate that normalizedscores reflect actual instance capability — referencing the calibration methodsalready in Section 6 registry entries and extending them into deployableprocedures (synthetic workloads, periodic re-calibration, QoE correlation).4. Clarify cross-domain and multi-vendor deployment constraints. Expand Section4.339s comparability warning into explicit operational guidance: when scores aretrustworthy, when Level 1 categories are preferable to Level 2 global scores,and how calibration maintains consistency within an administrative domain.5. Discuss management model expectations. Even if a YANG module is deferred,state whether configuration of normalization bounds, aggregation weights, andMeasurement_Window is expected via NETCONF/RESTCONF or operator-local policy,and how config vs. operational state vs. statistics should be distinguished.6. Address metric advertisement churn. Provide guidance on sampling intervals,maximum update rates, and when to prefer Level 2 over Level 1 to limitcontrol-plane overhead at scale.7. Fix editorial issues in Section 6 registry entries (inline ### Output, rolenaming inconsistency, typos) and update the normative self-reference to thecurrent draft version.8. Add a brief migration narrative explaining how CATS metrics complement orreplace existing IGP TE metrics and ALTO cost metrics during incrementaladoption.I hope these comments are useful and constructive. The hierarchical metricframework is well-conceived and addresses a genuine interoperability need forCATS adding consolidated operational guidance will significantly improve itsdeployability in multi-vendor environments.Lim Fung-- Cats mailing list -- [e-mail] unsubscribe send an email to [e-mail]：[Cats] draft-ietf-cats-metric-definition-10 early Opsdir reviewDocument: draft-ietf-cats-metric-definitionTitle: CATS Metrics DefinitionReviewer: Fung LimReview result: Has IssuesHi,I have been selected as the Operational Directorate (opsdir) reviewer for thisInternet-Draft.The Operational Directorate reviews all operational and management-relatedInternet-Drafts to ensure alignment with operational best practices and thatadequate operational considerations are covered.A complete set of _"Guidelines for Considering Operations and Management inIETF Specifications"_ can be found athttps://datatracker.ietf.org/doc/draft-ietf-opsawg-rfc5706bis/.While these comments are primarily for the Operations and Management AreaDirectors (Ops ADs), the authors should consider them alongside other feedbackreceived.- Document: draft-ietf-cats-metric-definition-10- Reviewer: Fung Lim- Review Date: 3rd July 2026- Intended Status: Standards Track---## Summary- Has Issues: I have concerns about this document that I think should beresolved before publication.This document defines a three-level hierarchical abstraction forComputing-Aware Traffic Steering (CATS) metrics, standardizing Level 1 categorymetrics (computing, communication, service, composed) and a Level 2 globalnormalized score, along with a metric field framework. The design iswell-motivated: it deliberately avoids standardizing fine-grained Level 0metrics while providing interoperable encodings and normative score semanticsat higher levels. The metric framework described in this document is sound andseveral operational concerns are implicitly addressed.However, as a Standards Track document that defines inputs to traffic-steeringdecisions, it requires a dedicated Operational Considerations section (perSection 3.1 of draft-ietf-opsawg-rfc5706bis). Several operational topics thatare partially scattered across design principles, registry entries, andsecurity text would benefit from consolidation into operator-facing guidance.In particular, cross-vendor score comparability, metric collection failurehandling, control-plane churn from metric updates, and management interfaceexpectations should be addressed.1. Missing Operational Considerations section (§3.1 of rfc5706bis)Per draft-ietf-opsawg-rfc5706bis, a technical specification advancing as anIETF RFC MUST include an Operational Considerations section immediately beforeSecurity Considerations, or explicit boilerplate with rationale if none apply.This document jumps from Section 6 (CATS Metric Registry Entries) directly toSection 7 (Security Considerations).For a Standards Track document whose metrics directly influence instanceselection and traffic steering, an Operational Considerations section isessential rather than optional. The document already contains operationallyrelevant material — score semantics (Section 4.3), level selection trade-offs(Section 5), role assignments (C-SMA, C-NMA, C-PS in Section 6), calibrationguidance, and security requirements — but it is not consolidated whereoperators will look for it.2. Cross-vendor score comparability needs explicit operational guidance (§4.1,§4.5 of rfc5706bis)Section 4.3 warns that normalized scores "may not be directly comparable acrossdifferent implementations" because aggregation and normalization functions areimplementation-specific. Section 4.2.1 and 4.2.2 state these functions "are notstandardized" and are "controlled by operator policies."This is a fundamental operational constraint that affects multi-vendor andmulti-provider CATS deployments. An operator comparing a score of 7 fromProvider A against a score of 7 from Provider B has no assurance of equivalentcapability. The normative 0–3 / 4–7 / 8–10 steering bands (Section 4.3) helpwithin a single administrative domain but do not solve cross-domain comparison.The Operational Considerations section should address:a. When scores are safe to use for steering (intra-domain only vs. inter-domain)b. Whether operators should prefer Level 1 category metrics over Level 2 globalscores for multi-vendor environments c. How calibration (referenced in Section6 registry entries) should be applied operationally — who runs benchmarks, howoften, and what deviation thresholds trigger re-calibration d. Whether futureCATS signaling documents will carry normalization policy identifiers to aidcomparison3. No fault management or troubleshooting guidance (§5.4 of rfc5706bis)CATS metrics drive steering decisions. The document does not discuss howoperators detect or respond to metric-related faults:Stale or missing metrics from C-SMA or C-NMA (Section 7 addresses freshness atthe security level via SEC-4, but not operational fallback behavior) Collectionagent failures or telemetry pipeline outages Score anomalies (sudden drops,stuck values, contradictory Level 1 vs. Level 2 scores) Integrity violationhandling (SEC-1 says metrics SHOULD trigger alerts, but no alert schema oroperational procedure) Per §5.4 of rfc5706bis, essential faults, alarms, andevents should be identified. At minimum, the document should discuss:a. Fallback steering behavior when metrics are unavailable or fail freshnesschecks b. Whether last-known-good values are used or instances arede-prioritized c. Recommended health indicators for C-SMA, C-NMA, and C-PScomponents4. No guidance on verifying correct operation (§4.7 of rfc5706bis)The registry entries include calibration methods (e.g., "score deviation ...less than 0.1 over repeated test rounds") but the document does not describehow operators verify end-to-end CATS metric operation in production:Active vs. passive validation approachesSynthetic workload injection for score validationCorrelation between normalized scores and observed application QoESimple health checks (e.g., confirming C-SMA is publishing expected metrictypes at expected intervals) Section 6 registry entries reference benchmarkcalibration but treat it as an implementation activity without operator-facingprocedures.5. Metric update frequency and control-plane impact not addressed (§4.5 ofrfc5706bis)Section 3.1 discusses the trade-off between metric update frequency and"signaling overhead, reducing scalability of the metric distribution protocol."Section 6 suggests continuous sampling every 10 seconds with a defaultMeasurement_Window of 10 seconds.However, the document does not quantify or guide:Recommended maximum metric advertisement rate per service instanceImpact of high churn on CATS control-plane protocols (to be defined incompanion documents) When coarser sampling or Level 2-only advertisement ispreferable to reduce overhead Interaction between Statistics field choices(max, min, mean, cur) and update frequency This is particularly importantbecause CATS may scale to hundreds or thousands of service instances (Section5).Nits6. No management interoperability discussion (§5.2 of rfc5706bis)The document references Prometheus, NETCONF, and IPFIX for Level 0 metriccollection but does not discuss:Whether a YANG data model for CATS metric configuration, policy, and exposedstate is needed or planned How normalized scores and runtime parameters(Measurement_Window, aggregation weights) would be configured via standardmanagement interfaces Distinction between configuration data (normalizationbounds, weights), operational state (current scores), and statistics(historical trends) For a Standards Track metric definition consumed by networkelements, some discussion of management model expectations — even if deferredto a companion document — would help operators plan deployments.7. Migration and coexistence with existing metrics (§4.3 of rfc5706bis)The Introduction references existing TE metrics (RFC 7471) and networkperformance registries (RFC 8911, RFC 8912). The document does not discuss:How CATS Level 1 communication metrics relate to or subsume existing IGP TEmetrics Whether operators can incrementally adopt CATS metrics alongsidetraditional TE without dual-metric steering conflicts Mapping or equivalencebetween ALTO performance cost metrics (RFC 9439) and CATS normalized scores Abrief migration narrative would help operators who already run TE metricextensions.8. Role assignment ambiguity in registry entriesSection 6.4.3.6 assigns metric calculation to "Service contact instance" ratherthan C-SMA (used consistently elsewhere). Section 6.1.3.6 contains a formattingerror where ### Output appears inline after the C-PS role description,disrupting the registry structure.Section 6.4.3.6:Service contact instace: Collects Level 0 service raw metrics...This should likely reference C-SMA for consistency with the frameworkterminology.9. Editorial and reference nits"Section Section 4.4" (Section 4.1, line 605) — duplicate word"To-be-assgined" → "To-be-assigned" (Sections 6.1.4.2, 6.2.5.2, 6.3.5.2)"Service contact instace" → "Service contact instance" (Section 6.4.3.6)Normative self-reference [I-D.ietf-cats-metric-definition] points to draft-08(Section 9.1) — should reference the current draft version Appendix A examplesuse inconsistent Source capitalization ("Directly measured" vs. lowercase39directly measured39 in Section 4.1) IANA placeholder "XXX" identifiersthroughout Section 6 — expected for draft stage, but Operational Considerationsshould note operators need stable identifiers before deployment10. Accounting and performance management partially addressedSection 6 registry entries define singleton output values and calibrationtargets, which is a good start. However:No counter rollover behavior for Measurement_Window accumulatorsNo guidance on historical metric retention for capacity planningNo discussion of metric export for billing or SLA reporting (may be N/A forsome deployments, but worth a sentence)Recommendations1. Add an Operational Considerations section immediately before SecurityConsiderations. Consolidate scattered operational material and address: metriclevel selection, runtime parameter defaults, cross-vendor score limitations,metric collection failures and fallback steering, update rate guidance, andcoexistence with existing TE metrics.2. Document fault management and fallback behavior. Specify what happens whenC-SMA/C-NMA stops publishing, metrics fail freshness checks (SEC-4), or scoresare contradictory. Define minimum health indicators and recommended alertconditions.3. Add verification guidance. Describe how operators validate that normalizedscores reflect actual instance capability — referencing the calibration methodsalready in Section 6 registry entries and extending them into deployableprocedures (synthetic workloads, periodic re-calibration, QoE correlation).4. Clarify cross-domain and multi-vendor deployment constraints. Expand Section4.339s comparability warning into explicit operational guidance: when scores aretrustworthy, when Level 1 categories are preferable to Level 2 global scores,and how calibration maintains consistency within an administrative domain.5. Discuss management model expectations. Even if a YANG module is deferred,state whether configuration of normalization bounds, aggregation weights, andMeasurement_Window is expected via NETCONF/RESTCONF or operator-local policy,and how config vs. operational state vs. statistics should be distinguished.6. Address metric advertisement churn. Provide guidance on sampling intervals,maximum update rates, and when to prefer Level 2 over Level 1 to limitcontrol-plane overhead at scale.7. Fix editorial issues in Section 6 registry entries (inline ### Output, rolenaming inconsistency, typos) and update the normative self-reference to thecurrent draft version.8. Add a brief migration narrative explaining how CATS metrics complement orreplace existing IGP TE metrics and ALTO cost metrics during incrementaladoption.I hope these comments are useful and constructive. The hierarchical metricframework is well-conceived and addresses a genuine interoperability need forCATS adding consolidated operational guidance will significantly improve itsdeployability in multi-vendor environments.Lim Fung-- Cats mailing list -- [e-mail] unsubscribe send an email to [e-mail]