Re: [IPFIX] Options Templates, Templates, and related Data Sets sent in separate transport sessions

["Brian Trammell (IETF)" <[e-mail]>]

There are no protocol police, of course, but…

“[t]he Exporting Process assigns and maintains Template IDs per Transport Session and Observation Domain” et seq (7011 §8 ¶4) seems to me to pretty clearly preclude the exporter behavior you describe; a fairly strong requirement that templates were scoped to a transport session is the reason we had to go through so much handwaving to attempt to describe transport session semantics in a sessionless protocol.

I’m not sure what update would make this any clearer, unless that sentence in 7011 §8 ¶4 could be strengthened with normative language. (FWIW I’d consider that an editorial erratum, since the requirement to scope templates to a session for interop purposes underpins the entire design of template management)

Cheers,

Brian

> On 10 Apr 2024, at 10:18, Garri Djavadyan <[e-mail]> wrote:
> 
> Unfortunately, after a few months of discussing and sharing excerpts
> from RFC5153 and RFC7011 with the vendor, they refused to admit non-
> compliance by stating that "there is no explicit mandatory requirement
> for these templates [Templates and Options-Templates] to share a
> transport session".
> 
> I understand that they are fixated on the "explicit mandatory
> requirements" because what they had designed is already implemented in
> hardware and not easy to change. I guess there was a confusion over the
> RFC's UDP and SCTP requirements during the early architecture design
> phase.
> 
> Considering that this happened to one vendor, do you think the RFC7011
> can be updated a bit to make it more explicit in that all related
> template and data records must be carried over the same transport
> session, so that future RFC 7011 implementers do not fall into the same
> trap again?
> 
> Thank you.
> 
> Kind regards,
> Garri Djavadyan
> 
> 
> On Tue, 2024-02-06 at 00:29 +0100, Garri Djavadyan wrote:
>> Paul,
>> 
>> Thank you so much for your reply.
>> 
>> Indeed, a source address translator in front of several Exporters
>> that
>> use two Transport UDP Sessions to represent one IPFIX export stream,
>> have conflicting Observation Domain IDs, and different
>> samplingInterval
>> values in the Data Records would be a nightmare for a Collecting
>> Process that treats multiple Transport Sessions sourced from the same
>> IP as the same export stream.
>> 
>> It seems I did not pay enough attention to the following sentence in
>> the section 3.1 of the RFC7011:
>> 
>>    Collecting Processes SHOULD use the Transport Session and the
>>    Observation Domain ID field to separate different export streams
>>    originating from the same Exporter.
>>    
>> 
>> I think a definition of the Export Stream in the Terminology (2)
>> section of the RFC7011 could also help avoid doubts, saying something
>> like: 
>> 
>>    A Transport Session carrying related Template Records, Options
>>    Template Records, and Data Records.
>> 
>> 
>> Now it is clear that we faced a bug that needs to be reported to the
>> respective vendor.
>> 
>> Thank you.
>> 
>> Kind regards,
>> Garri Djavadyan
>> 
>> 
>> 
>> On Mon, 2024-02-05 at 10:17 +0000, Aitken, Paul wrote:
>>> Garri,
>>> 
>>>  From the receiver's point of view, the different transport
>>> sessions
>>> are 
>>> independent export streams which could come from different
>>> Exporters.
>>> The receiver can't tell whether these come from one exporter or
>>> two.
>>> The 
>>> data in the "options" Export Stream cannot be scoped to data in the
>>> other stream as Template IDs are unique per Transport Session and
>>> the
>>> Observation Domain is locally unique to the Exporting Process and 
>>> multiple EPs may be in play.
>>> 
>>> Only SCTP provides multiple streams; TPC and UDP do not. See
>>> section
>>> 6.1 
>>> of RFC 5153.
>>> 
>>> P.
>>> 
>>> 
>>> On 03/02/24 14:38, Garri Djavadyan wrote:
>>>> Hi IPFIX experts,
>>>> 
>>>> 
>>>> Recently, my colleagues and I faced an IPFIX exporter
>>>> implementation
>>>> that sends IPFIX messages with the Template and related Data Sets
>>>> in
>>>> one transport UDP session and the Options Template and related
>>>> Data
>>>> Sets in another transport UDP session (sourced from a different
>>>> UDP
>>>> port) to the same IPFIX collector port.
>>>> 
>>>> While trying to understand this behaviour, I checked RFC7011 and
>>>> a
>>>> few
>>>> related RFCs but could not find any notes suggesting use cases
>>>> when
>>>> it
>>>> might be necessary. Also, I could not find any notes prohibiting
>>>> this.
>>>> 
>>>> However, we know of at least one use case when sending the
>>>> aforementioned 4 IPFIX messages over 2 separate transport
>>>> sessions
>>>> is
>>>> undesirable. For example, when multiple self-sufficient IPFIX
>>>> collectors are listening on the same UDP socket using the socket
>>>> option
>>>> SO_REUSEPORT (for horizontal scaling), the kernel-level load-
>>>> balancing
>>>> algorithm can distribute 2 related sessions of the same exporter
>>>> to
>>>> different processes. As a result, the process receiving the Flow
>>>> Records, cannot process them properly without the related
>>>> Options.
>>>> In
>>>> theory, it can also happen in the environments where layer-4
>>>> load-
>>>> balancers are used to distribute transport UDP sessions between
>>>> IPFIX
>>>> collecting hosts.
>>>> 
>>>> Considering this, could you please help answer the following
>>>> question?
>>>> 
>>>> Did I interpret the IPFIX RFCs correctly and the mentioned
>>>> exporter's
>>>> behaviour is a compliant implementation?
>>>> 
>>>> There is a note in the "Template Management" section 8 of RFC7011
>>>> saying:
>>>> 
>>>>     Options Templates and Templates that are related or
>>>> interdependent
>>>>     (e.g., by sharing common properties as described in
>>>> [RFC5473])
>>>>     SHOULD be sent together in the same IPFIX Message.
>>>> 
>>>> but the Templates and the Options Templates in the case that I
>>>> mentioned are not interdependant by the common properties per
>>>> RFC5473.
>>>> 
>>>> 
>>>> 
>>>> Thank you in advance!
>>>> 
>>>> 
>>>> Kind regards,
>>>> Garri Djavadyan
>>>>   
>>>> 
>>>> _______________________________________________
>>>> IPFIX mailing list
>>>> [e-mail]
>>>> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ipfix__;!!OSsGDw!J3DauxNGB6V13gsv8HRy629ddHwNTtJFeE5njPIdVQCVssXh2upsaiMkWBlmd8llnYXke9dEef0WZvmNag$
>>>>  [ietf[.]org]
>> 
> 
> _______________________________________________
> IPFIX mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/ipfix