Re: [IPFIX] Export of long lived flow information

[Andrew Johnson <[e-mail]>]

Hi folks

I've been thinking for a while now that we should be able to send a mixture of delta and update records for the same flow, but I haven't really figured out how to make the semantics obvious to the collector.

i.e.:    Delta (new record), update, update, update(ends the flow)


Perhaps we could use the flow end reason to make things more explicit.  For example, all but the last packet have a flowEndReason of 0 and then the last one updates the reason.

Would this sort of thing be useful?  Would collectors be able to work with it?


Cheers, Andrew



On 23 Oct 2012, at 08:43, Brian Trammell wrote:
> Hi, John,
> 
> Your assumptions are basically correct. The IPFIX model of a Metering Process (MP) makes an implicit assumption that it will be configured or configurable to export information about long-lived flows every n minutes through active timeout. From the MP side, this allows periodic complete flushing of the flow cache. 
> 
> More importantly (IMO), on the Collecting Process (CP) side, it provides a guarantee that every packet observed and selected for export will be accounted for within Ta + Tde + Tdc, where Ta is the active timeout, Tde is the MP to EP delay (how long it takes for an exported flow to make it from the MP cache, through the Exporting Process (EP), into an IPFIX Message, which is implementation dependent but typically short) and Tde is the export delay (OWD from EP to CP). This is important for streaming process applications, as after this time, the CP and downstream processes can assume that no further information about the past will become available.
> 
> The tradeoff here is that shorter Ta causes more records to be exported about long flows, and a longer Ta causes a longer delay, which some streaming applications can't tolerate.
> 
> In any case, the assumption is that the flow is no longer in the cache after active timeout, so you don't know when the flow really started. Since the flowStartTime is the timestamp of the first observed packet within the record, and flowEndTime is the timestamp of the last observed packet within the record, the timestamps would then be record-local.... i.e. a flow less than twice the active timeout (with at least one packet per idle timeout) would result in two flow records. The first would have a timestamp range between the first packet of the flow and the last packet observed before the active timeout; the second between the first packet observed after the active timeout and the last packet of the flow.
> 
> This does, admittedly, make it rather difficult to stitch records for long flows together -- you can't match timestamps, and essentially need to simulate the flow cache with a longer active timeout. For applications requiring a single record per flow, active timeouts can be set to be practically infinite, with the tradeoff that you never know at the CP when you'll get a flow with a start time far in the past.
> 
> Best regards,
> 
> Brian
> 
> 
> On Oct 23, 2012, at 3:01 AM, John Court wrote:
> 
>> Hi, 
>> 
>> I have been a subscriber to the list for a little over a year, and an implementer of IPFIX export for at least one product.  This WG has done great work overall ! 
>> 
>> One area that still has me a little confused even after researching as many of the RFCs as possible including RFC5472 is how to treat export of long lived flows. 
>> 
>> At the moment I use "DeltaCount" information elements for everything and at specific intervals export long lived flows with the flowEndReason of "flowActiveTimeout".  This of course results in multiple flow records for long lived connections over time.  Since this situation doesn't seem to be covered explicitly I was hoping someone on the list would point me in the right direction or confirm my assumptions.  On thing that is particularly unclear is what to do about flowStart/flowEnd times when sending this type of record. 
>> 
>> Thanks 
>> 
>> John Court
>> Senior Software Engineer
>> IBM Security Systems Division
>> IBM Australia Development Laboratory
>> Office:  [phone]
>> Mobile: [phone]
>> 
>> _______________________________________________
>> IPFIX mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/ipfix
> 
> _______________________________________________
> IPFIX mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/ipfix