Re: [IPFIX] Options Templates, Templates, and related Data Sets sent in separate transport sessions

[Garri Djavadyan <[e-mail]>]

Thank you for your comments, Brian!

As the vendor does not participate in this discussion, I feel I should
be playing devil's advocate for the sake of understanding the reasons
for the misinterpretation that can help future IPFIX implementers.

Brian, I think one can say that what the vendor is doing is what is
written in 7011 §8 ¶4: they assign and maintain Data-Template IDs per
one Transport Session + Observation Domain ID, and the Options-Template
IDs per other Transport Session + the same Observation Domain ID.

Then, it seems they assume that the Collecting process can scope the
(Options) Data Records in one Transport Session to the Data Records in
other Transport Session because the Observation Domain ID is the same.
The Observation Domain ID is still unique per the Exporting Process,
and they can claim that both Transport Sessions are initiated by the
same Exporting Process.

Maybe it can help if the definition for the Observation Domain ID also
mentions that it is unique per Transport Session as the Collecting
Process is unable to assert if Transport Sessions are related to the
same Exporting Process?

Also, the Observation Domain ID section in 7011 §8 says:

      Collecting Processes SHOULD use the Transport Session and the
      Observation Domain ID field to separate different export streams
      originating from the same Exporter.

I think it can also help if there is a definition for an Export Stream
in the Terminology. That definition may emphasise that it is a stream
of related Template, Options-Template, and Data Records carried over
the same Transport Session.

Again, I am sorry if the above may seem too obvious to making no sense.
Just trying to understand the vendor's logic in attempt to improve the
document.

Thank you.

Kind regards,
Garri Djavadyan



On Wed, 2024-04-10 at 10:50 +0200, Brian Trammell (IETF) wrote:
> There are no protocol police, of course, but…
> 
> “[t]he Exporting Process assigns and maintains Template IDs per
> Transport Session and Observation Domain” et seq (7011 §8 ¶4) seems
> to me to pretty clearly preclude the exporter behavior you describe;
> a fairly strong requirement that templates were scoped to a transport
> session is the reason we had to go through so much handwaving to
> attempt to describe transport session semantics in a sessionless
> protocol.
> 
> I’m not sure what update would make this any clearer, unless that
> sentence in 7011 §8 ¶4 could be strengthened with normative language.
> (FWIW I’d consider that an editorial erratum, since the requirement
> to scope templates to a session for interop purposes underpins the
> entire design of template management)
> 
> Cheers,
> 
> Brian
> 
> > On 10 Apr 2024, at 10:18, Garri Djavadyan <[e-mail]>
> > wrote:
> > 
> > Unfortunately, after a few months of discussing and sharing
> > excerpts
> > from RFC5153 and RFC7011 with the vendor, they refused to admit
> > non-
> > compliance by stating that "there is no explicit mandatory
> > requirement
> > for these templates [Templates and Options-Templates] to share a
> > transport session".
> > 
> > I understand that they are fixated on the "explicit mandatory
> > requirements" because what they had designed is already implemented
> > in
> > hardware and not easy to change. I guess there was a confusion over
> > the
> > RFC's UDP and SCTP requirements during the early architecture
> > design
> > phase.
> > 
> > Considering that this happened to one vendor, do you think the
> > RFC7011
> > can be updated a bit to make it more explicit in that all related
> > template and data records must be carried over the same transport
> > session, so that future RFC 7011 implementers do not fall into the
> > same
> > trap again?
> > 
> > Thank you.
> > 
> > Kind regards,
> > Garri Djavadyan
> > 
> > 
> > On Tue, 2024-02-06 at 00:29 +0100, Garri Djavadyan wrote:
> > > Paul,
> > > 
> > > Thank you so much for your reply.
> > > 
> > > Indeed, a source address translator in front of several Exporters
> > > that
> > > use two Transport UDP Sessions to represent one IPFIX export
> > > stream,
> > > have conflicting Observation Domain IDs, and different
> > > samplingInterval
> > > values in the Data Records would be a nightmare for a Collecting
> > > Process that treats multiple Transport Sessions sourced from the
> > > same
> > > IP as the same export stream.
> > > 
> > > It seems I did not pay enough attention to the following sentence
> > > in
> > > the section 3.1 of the RFC7011:
> > > 
> > >    Collecting Processes SHOULD use the Transport Session and the
> > >    Observation Domain ID field to separate different export
> > > streams
> > >    originating from the same Exporter.
> > >    
> > > 
> > > I think a definition of the Export Stream in the Terminology (2)
> > > section of the RFC7011 could also help avoid doubts, saying
> > > something
> > > like: 
> > > 
> > >    A Transport Session carrying related Template Records, Options
> > >    Template Records, and Data Records.
> > > 
> > > 
> > > Now it is clear that we faced a bug that needs to be reported to
> > > the
> > > respective vendor.
> > > 
> > > Thank you.
> > > 
> > > Kind regards,
> > > Garri Djavadyan
> > > 
> > > 
> > > 
> > > On Mon, 2024-02-05 at 10:17 +0000, Aitken, Paul wrote:
> > > > Garri,
> > > > 
> > > >  From the receiver's point of view, the different transport
> > > > sessions
> > > > are 
> > > > independent export streams which could come from different
> > > > Exporters.
> > > > The receiver can't tell whether these come from one exporter or
> > > > two.
> > > > The 
> > > > data in the "options" Export Stream cannot be scoped to data in
> > > > the
> > > > other stream as Template IDs are unique per Transport Session
> > > > and
> > > > the
> > > > Observation Domain is locally unique to the Exporting Process
> > > > and 
> > > > multiple EPs may be in play.
> > > > 
> > > > Only SCTP provides multiple streams; TPC and UDP do not. See
> > > > section
> > > > 6.1 
> > > > of RFC 5153.
> > > > 
> > > > P.
> > > > 
> > > > 
> > > > On 03/02/24 14:38, Garri Djavadyan wrote:
> > > > > Hi IPFIX experts,
> > > > > 
> > > > > 
> > > > > Recently, my colleagues and I faced an IPFIX exporter
> > > > > implementation
> > > > > that sends IPFIX messages with the Template and related Data
> > > > > Sets
> > > > > in
> > > > > one transport UDP session and the Options Template and
> > > > > related
> > > > > Data
> > > > > Sets in another transport UDP session (sourced from a
> > > > > different
> > > > > UDP
> > > > > port) to the same IPFIX collector port.
> > > > > 
> > > > > While trying to understand this behaviour, I checked RFC7011
> > > > > and
> > > > > a
> > > > > few
> > > > > related RFCs but could not find any notes suggesting use
> > > > > cases
> > > > > when
> > > > > it
> > > > > might be necessary. Also, I could not find any notes
> > > > > prohibiting
> > > > > this.
> > > > > 
> > > > > However, we know of at least one use case when sending the
> > > > > aforementioned 4 IPFIX messages over 2 separate transport
> > > > > sessions
> > > > > is
> > > > > undesirable. For example, when multiple self-sufficient IPFIX
> > > > > collectors are listening on the same UDP socket using the
> > > > > socket
> > > > > option
> > > > > SO_REUSEPORT (for horizontal scaling), the kernel-level load-
> > > > > balancing
> > > > > algorithm can distribute 2 related sessions of the same
> > > > > exporter
> > > > > to
> > > > > different processes. As a result, the process receiving the
> > > > > Flow
> > > > > Records, cannot process them properly without the related
> > > > > Options.
> > > > > In
> > > > > theory, it can also happen in the environments where layer-4
> > > > > load-
> > > > > balancers are used to distribute transport UDP sessions
> > > > > between
> > > > > IPFIX
> > > > > collecting hosts.
> > > > > 
> > > > > Considering this, could you please help answer the following
> > > > > question?
> > > > > 
> > > > > Did I interpret the IPFIX RFCs correctly and the mentioned
> > > > > exporter's
> > > > > behaviour is a compliant implementation?
> > > > > 
> > > > > There is a note in the "Template Management" section 8 of
> > > > > RFC7011
> > > > > saying:
> > > > > 
> > > > >     Options Templates and Templates that are related or
> > > > > interdependent
> > > > >     (e.g., by sharing common properties as described in
> > > > > [RFC5473])
> > > > >     SHOULD be sent together in the same IPFIX Message.
> > > > > 
> > > > > but the Templates and the Options Templates in the case that
> > > > > I
> > > > > mentioned are not interdependant by the common properties per
> > > > > RFC5473.
> > > > > 
> > > > > 
> > > > > 
> > > > > Thank you in advance!
> > > > > 
> > > > > 
> > > > > Kind regards,
> > > > > Garri Djavadyan
> > > > >   
> > > > > 
> > > > > _______________________________________________
> > > > > IPFIX mailing list
> > > > > [e-mail]
> > > > > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ipfix__;!!OSsGDw!J3DauxNGB6V13gsv8HRy629ddHwNTtJFeE5njPIdVQCVssXh2upsaiMkWBlmd8llnYXke9dEef0WZvmNag$
> > > > >  [ietf[.]org]
> > > 
> > 
> > _______________________________________________
> > IPFIX mailing list
> > [e-mail]
> > https://www.ietf.org/mailman/listinfo/ipfix
>