Re: [IPFIX] Export of long lived flow information

[Brian Trammell <[e-mail]>]

Hi, John,

Okay, I understand now... I wasn't really responding directly to you on this point, just to the general thread on timestamps in MPs with persistent-cache export, and delta versus total counters.

In your case, if you've _already got_ a persistent cache for other reasons, then by all means use it. :) In that case you just need an extra timestamp, to keep track of the time of the first packet observed after the last export, in addition to the first "real" packet, if you want to export "complete" (delta) flows.

Best regards,

Brian


On Oct 24, 2012, at 8:00 AM, John Court <[e-mail]> wrote:

> Hi Brian, 
> 
> I suspect I have been mis-interpreting your concept of "persistent caches".  From the way you describe it, I would categorise what I export from as a persistent cache.  The reason being that for packet processing efficiency reasons, all the "connection" concepts are kept in one place and that includes the accounting (i.e. packets, octets).  I only get to see these periodically or on connection close. In IPFIX terminology the meter isn't resetting the flowStartTime or doing actual flow termination outside of detection of "connection" termination.  Perhaps it could but that time may be used in other processing, I will have to see.   
> 
> I am not arguing with any of your points.  I see the logic behind the operation, I just need to adjust some outputs for long lived connections.   
> 
> Thanks 
> 
> John Court
> 
> 
> 
> 
> From:        Brian Trammell <[e-mail]> 
> To:        John Court/Australia/IBM@IBMAU, 
> Cc:        [e-mail] 
> Date:        24/10/2012 15:19 
> Subject:        Re: [IPFIX] Export of long lived flow information 
> 
> 
> 
> Hi, John, 
> 
> I'll say here that I've never really understood the arguments for export from persistent caches, and don't recommend the approach unless you have other overriding application requirements. But that's merely my opinion, and other people seem to like persistent caches and running total export (especially router people, who need to keep the cache around for other reasons anyway), and I guess it's just a sort of point of view thing. 
> 
> My problem with export from persistent caches with updating totals is that if you want to do anything with the data other than drive per-flow or per-aggregate displays (a la rrdtool or MRTG), you need to do the same amount of postprocessing as you would with nonpersistent export with Ta equal to your desired export interval in order to get "real flow data", and you lose packet and byte counters per active export interval. When each subpart of the flow has accurate first/last timestamps, you get a time series which can give you some idea of the evolution of the packet and byte rates (is it regular i.e. bulk transfer? is its rate slowly variable i.e. bulk transfer under congestion control? is it a low rate highly variable flow? and so on). With persistent cache export you kind of have the same thing, but (1) you have to subtract to get it and (2) you have no accurate midpoint timestamp; you can only deduce it from the export time and your best guess about Tde and Tdc. 
> 
> I'm not sure what you mean about "how to coordinate resetting (start time) on export"... if you're using non-persistent export, you expire the flow completely out of the cache on active timeout (with flowEndReason activeTimeout) and start a new flow record when you see the first (continuation) packet. If you want to keep ancillary information about the flow (e.g. deduced AS information if you're not a packet-forwarding device, other private labeling information that is expensive to calculate or requires examination of leading payload), you can keep it around in a secondary cache to be resurrected by flow key and associated with the new flow record when the continuation packet comes. 
> 
> As 5101bis is still open, I'll see if I can come up with some suitable language on the meaning of timestamps therefor (without the editorializing on persistent export), and propose to the list. 
> 
> Best regards, 
> 
> Brian 
> 
> 
> On Oct 23, 2012, at 11:59 PM, John Court <[e-mail]> wrote: 
> 
> Thanks for the interest in resolving this, 
> 
> Andrew, I understand your persistent cache argument.  The reason I don't personally use "Total" and DO use "Delta"  is more around what the collector does should it not see some of the updates and suddenly gets one that shows large counter values.  This could mistakenly result in showing huge traffic over a short period incorrectly.  This is particularly true if you do as Brian suggested and are setting the flowStartTime based only on the current record view. 
> 
> Brian, thanks for your detailed explanation.  Everything with the exception of flowStartTime was as I am currently doing.  I had perhaps mistakenly taken the approach that keeping the flowStartTime as the "conceptual" start rather than for this reporting period would make it easier for the collector to understand what was happening.  Not actually sure how I will co-ordinate reseting that time on export yet :-) 
> 
> I would suggest that this sort of situation and Brians explanation be added as an example perhaps in 5101 or even 5472 as it would help with Collector interoperability I am sure :-) 
> 
> Thanks again. 
> 
> 
> John Court
> Software Engineer
> IBM Security Systems Division
> IBM Australia Development Laboratory
> Office:  [phone]
> Mobile: [phone]
> 
> _______________________________________________
> IPFIX mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/ipfix 
>