Re: [IPFIX] Export of long lived flow information

Hi all

Paul, thanks for the terminology correction, I did mean "total" counters and was confusing people with "updates".

I was thinking that a mechanism that allowed a non-permanent flow to be exported multiple time would be useful.  For example, security applications generally want to know about a new flow ASAP, so they can act on the information, but a short active timeout values lead to using more export bandwidth.  I was thinking we could do something like export a report of the flow after the first packet, and then export the final version of the flow once the normal timeouts had decided it was over.

I had in mind something like using a delta count, followed by a total count.  Reading the below definition of Total counts though, I'm not sure that will work, but I think it depends on how we interpret the definition of "Flow".  If two records have matching key fields but different starting timestamps, are they the same Flow?

I would argue that a single Flow can't have two flowStartTimes, so maybe not.  This would mean that we shouldn't reset the flowStartTimes between sending reports for the same permanent Flow.

Cheers, Andrew

On 24 Oct 2012, at 22:19, Paul Aitken wrote:
> John, Gerhard,
> 
> Looking at the definitions:
> 
> octetTotalCount:
> 
>         The total number of octets in incoming packets
>         for this Flow at the Observation Point since the Metering
>         Process (re-)initialization for this Observation Point.
> 
> packetTotalCount:
> 
>         The total number of incoming packets for this Flow
>         at the Observation Point since the Metering Process
>         (re-)initialization for this Observation Point.
> 
> 
> So even if there's a long gap in the traffic, the intention is that a MP which is reporting totalCount fields remembers that it saw the flow before.
> 
> So +1 to what Gerhard says: "flowStartTime is the time of the very first packet ever observed for this flow." (ie, the "original" first packet).
> 
> 
> Whereas when reporting deltaCount fields, the MP may "forget" about the flow (by purging the cache entry) - so after the traffic gap the flow must start over as if it's entirely new, since the MP retains no history. Therefore the flowStartTime must be the first packet in the new delta - though, the collector might choose to aggregate this with previous deltas, setting the flowStartTime to the earliest reported and the flowEndTime to the latest.
> 
> This is undoubtedly something that we should capture in one of our updated docs.
> 
> Thanks for the great question John!
> 
> P.
> 
> 
> On 24/10/12 21:35, Gerhard Muenz wrote:
>> 
>> Hi,
>> 
>> flowStartTime is the time of the first packet you are reporting on in the given record. So, including totalCount fields implicitly means that flowStartTime is the time of the very first packet ever observed for this flow.
>> 
>> Now, you can continue discussing what happens if both totalCounts and deltaCounts are included in the same record :)
>> 
>> Regards,
>> Gerhard
>> 
>> 
>> On 24.10.2012 22:28, John Court wrote:
>>> Just to be crystal clear on this point of persistent caches.  Even when
>>> sending "totalCount" fields, the flowStartTime is still relative to the
>>> current flow record, it doesn't represent the "original" first packet
>>> ever seen for the flow key in the cache ?  I just want to make sure of
>>> the semantics of flowStartTime in all cases.
>>> 
>>> Thanks again for the comments and clarifications
>>> 
>>> John Court
>>> Software Engineer
>>> IBM Security Systems Division
>>> IBM Australia Development Laboratory
>>> Office:  +61 7 5552 4014
>>> Mobile: +61 430 841328
>>> 
>>> 
>>> 
>>> 
>>> 
>>> From: Paul Aitken <paitken@cisco.com>
>>> To: John Court/Australia/IBM@IBMAU,
>>> Cc: Brian Trammell <trammell@tik.ee.ethz.ch>, ipfix@ietf.org
>>> Date: 24/10/2012 19:49
>>> Subject: Re: [IPFIX] Export of long lived flow information
>>> ------------------------------------------------------------------------
>>> 
>>> 
>>> 
>>> John,
>>> 
>>> I suspect I have been mis-interpreting your concept of "persistent caches".
>>> 
>>> In a normal cache, the entries are eventually removed - because they've
>>> ended, or they've not seen traffic for an amount of time, or they're
>>> just too old, or there's simply not enough room in the cache.
>>> 
>>> Whereas in a permanent cache, the entries are never removed.
>>> 
>>> P.
>>> 
>>> 
>>> _______________________________________________
>>> IPFIX mailing list
>>> IPFIX@ietf.org
>>> https://www.ietf.org/mailman/listinfo/ipfix
>>> 
> 
> _______________________________________________
> IPFIX mailing list
> IPFIX@ietf.org
> https://www.ietf.org/mailman/listinfo/ipfix