Re: [ipfix] drawing the line between protocol and data model document

[Benoit Claise <[e-mail]>]

Tal, Jeff,

[let me reply to both of you in a single email]

> Benoit,
>  
> I agree with Jeff. I would add that, no more than 2 options should be 
> available for bytes/packet counters (32/64-bit unsigned integers).
>  
> But I would consider offering *only* a 64-bit value for byte counters. 
> From SNMP we learned that 32-bit byte counters cause countless 
> problems. Since IPFIX byte counters may be a result of aggregates, 
> with today's networks it is very feasible to overflow the 32-bit 
> integer container.

How often do you poll the SNMP variables like interface counters? 5 minutes?
This is order of magnitude higher than the flow expiration, in most of 
the cases.

http://www.ietf.org/internet-drafts/draft-claise-netflow-9-02.txt
    -> 3.2 Flow Expiration

   A Flow is considered to be inactive if no packets belonging to the 
   Flow have been observed at the Observation Point for a given timeout 
   interval otherwise it is considered as an active flow.  
   A Flow can be exported under the following conditions: 
    
      1. If the Exporter can detect the end of a Flow, it SHOULD export 
      the Flow Records at the end of the Flow. For example, a Flow 
      generated by TCP [3] type of traffic where the FIN or RST bits 
      indicate the end of the Flow. 
       
      2. If the Flow has been inactive for a certain period of time. 
      This inactivity timeout SHOULD be configurable, with a minimum 
      value of 0 for an immediate expiration. For example, a Flow 
      generated by UDP [2] type of traffic. 
       
      3. For long-lasting Flows, the Exporter SHOULD export the Flow 
      Records on a regular basis. This periodicity SHOULD be 
      configurable. 
       
      4. If the Exporter experiences internal constraints, a Flow MAY 
      be forced to expire prematurely (for example, counters wrapping 
      or low memory). 

So it's only in case 3 (so potentially an aggregation, as you described) that we could potentially needs a higher counter.
And in that case again, a simple solution is to export the flow records just before it reaches the max value of counter32 (in case the length for counter32 is choosen).

But if you don't want that, there is also another solution: when this case 3. occurs, you can define a new template with the length corresponding to counter64.
And then you have 2 templates: one with counter32 (used most of the time) and one with counter64 (used in a few cases).

Another solution, nothing prevents you to export flow data records always with counter64

See below...

>  
> Tal
>
>     -----Original Message-----
>     *From:* majordomo listserver
>     [mailto:[e-mail]]*On Behalf Of *MEYER,JEFFREY D
>     (HP-Cupertino,ex1)
>     *Sent:* Wednesday, June 11, 2003 10:00 AM
>     *To:* 'Benoit Claise'
>     *Cc:* Mark Fullmer; [e-mail]; [e-mail]
>     *Subject:* RE: [ipfix] drawing the line between protocol and data
>     model document
>
>     Hi,
>      
>       >From a practical perspective, simply saying that a field is
>     numeric and may be encoded as 1 or 100 bytes seems like it puts an
>     unnecessary and unjustified burden on the collector, or any other
>     app which might want to handle this data.
>
Am I correct that the burden would only be when you decode the template 
records?
Because when you receive a Data FlowSet, the collecting process must 
read the FlowSet ID and automatically know the data type length.

>      
>       Is there some particular reason why saying a specific
>     information item, e.g. numBytes, is a 32-bit integer or a 64-bit
>     integer is considered undesireable.
>
But then, why to limit to 32-bit and 64-bit integer? What about other 
possibilities? Where to stop? Then we end up with a long information 
model because this principle of 32-bit, 64-bit integer should be then 
applied to all data type that are counters.

>     It certainly seems like the developer of a consuming application
>     would benefit from knowing how to target variables in their
>     application for given information items.  I guess one could assume
>     that everything will fit in a 64-bit integer and code your app
>     that way, but again, this seems like an unnecessary burden for
>     unclear gains.
>
The gain, as I see it, is in the flexibility with a simple information model

Regards, Benoit.

>      
>       I don't think anything needs to change in the protocol.  Simply
>     a more constrained approach to dealing with numeric fields with
>     explicit types.
>      
>     Regards,
>
>       Jeff Meyer
>
>         -----Original Message-----
>         *From:* Benoit Claise [mailto:[e-mail]]
>         *Sent:* Wednesday, June 11, 2003 4:31 AM
>         *To:* Benoit Claise
>         *Cc:* Mark Fullmer; [e-mail];
>         [e-mail]
>         *Subject:* Re: [ipfix] drawing the line between protocol and
>         data model document
>
>         Dear all,
>
>>         Mark,
>>
>>>On Thu, May 01, 2003 at 10:51:49AM -0400, [e-mail] wrote:
>>>
>>><snip>
>>>
>>>  
>>>
>>>>The richness of the protocol encoding specification is another
>>>>matter. The protocol would reference each element in the data
>>>>model and define exaclty how that element is encoded. In the above 
>>>>example, the protocol is free to encode that as an unsigned byte 
>>>>if the value is < 256. 
>>>>    
>>>>
>>>
>>>The ramifications of doing this with a template based system are a bit
>>>ugly.  Consider a flow with just 2 counters -- octets, packets.
>>>Each could have a 1 to 8 byte encoding.  That's a potential of 64
>>>different templates for representing 1 type of flow.
>>>
>>>This (at first glance) looks like it adds considerable cycles to the
>>>encode / decode process to save a few bytes in the transport.
>>>
>>>The current v9 spec appears to allow this, but only for IN_BYTES and
>>>IN_PKTS.
>>>
>>>  
>>>
>>>>                                          counter with length
>>>>   IN_BYTES                 1       N     N x 8 bits for bytes
>>>>                                          associated with an IP Flow
>>>>    
>>>>
>>>
>>>Is this true?  Benoit?
>>>  
>>>
>>         I'm trying to catch up with my IPIFX emails; I know I'm late
>>         on this thread, which seems to have reached consensus as far
>>         as I can tell, but as this is a direct question to me, let me
>>         answer.
>>         Actually, there are:
>>         BYTES            1             4             field type for
>>         the 32 bit bytes counter
>>         BYTES_64      23           8             field type for the
>>         64 bit bytes counter
>
>         Let me reply to my own email because, after speaking to
>         different people, I have to correct what I said.
>         We actually do:
>
>                                          Incoming counter with length
>   IN_BYTES                 1       N     N x 8 bits for bytes
>                                          associated with an IP Flow
>
>                                          Incoming counter with length
>   IN_PKTS		    2       N     N x 8 bits for packets
>                                          associated with an IP Flow
>
>And no BYTES_64 is defined.
>
>Regards, Benoit.
>
>
>>
>>
>>         So the draft that you refer to needs a quick update.
>>         <!--[endif]-->
>>         Regards, Benoit
>>
>>
>>
>>>mark
>>>
>>>--
>>>Help        mailto:[e-mail] and say "help" in message body
>>>Unsubscribe mailto:[e-mail] and say
>>>"unsubscribe ipfix" in message body
>>>Archive     http://ipfix.doit.wisc.edu/archive/
>>>  
>>>
>>
>