Re: [IPFIX] Export of long lived flow information

[John Court <[e-mail]>]

Hi Brian,

I suspect I have been mis-interpreting your concept of "persistent 
caches".  From the way you describe it, I would categorise what I export 
from as a persistent cache.  The reason being that for packet processing 
efficiency reasons, all the "connection" concepts are kept in one place 
and that includes the accounting (i.e. packets, octets).  I only get to 
see these periodically or on connection close. In IPFIX terminology the 
meter isn't resetting the flowStartTime or doing actual flow termination 
outside of detection of "connection" termination.  Perhaps it could but 
that time may be used in other processing, I will have to see. 

I am not arguing with any of your points.  I see the logic behind the 
operation, I just need to adjust some outputs for long lived connections. 

Thanks

John Court




From:   Brian Trammell <[e-mail]>
To:     John Court/Australia/IBM@IBMAU, 
Cc:     [e-mail]
Date:   24/10/2012 15:19
Subject:        Re: [IPFIX] Export of long lived flow information



Hi, John,

I'll say here that I've never really understood the arguments for export 
from persistent caches, and don't recommend the approach unless you have 
other overriding application requirements. But that's merely my opinion, 
and other people seem to like persistent caches and running total export 
(especially router people, who need to keep the cache around for other 
reasons anyway), and I guess it's just a sort of point of view thing.

My problem with export from persistent caches with updating totals is that 
if you want to do anything with the data other than drive per-flow or 
per-aggregate displays (a la rrdtool or MRTG), you need to do the same 
amount of postprocessing as you would with nonpersistent export with Ta 
equal to your desired export interval in order to get "real flow data", 
and you lose packet and byte counters per active export interval. When 
each subpart of the flow has accurate first/last timestamps, you get a 
time series which can give you some idea of the evolution of the packet 
and byte rates (is it regular i.e. bulk transfer? is its rate slowly 
variable i.e. bulk transfer under congestion control? is it a low rate 
highly variable flow? and so on). With persistent cache export you kind of 
have the same thing, but (1) you have to subtract to get it and (2) you 
have no accurate midpoint timestamp; you can only deduce it from the 
export time and your best guess about Tde and Tdc.

I'm not sure what you mean about "how to coordinate resetting (start time) 
on export"... if you're using non-persistent export, you expire the flow 
completely out of the cache on active timeout (with flowEndReason 
activeTimeout) and start a new flow record when you see the first 
(continuation) packet. If you want to keep ancillary information about the 
flow (e.g. deduced AS information if you're not a packet-forwarding 
device, other private labeling information that is expensive to calculate 
or requires examination of leading payload), you can keep it around in a 
secondary cache to be resurrected by flow key and associated with the new 
flow record when the continuation packet comes.

As 5101bis is still open, I'll see if I can come up with some suitable 
language on the meaning of timestamps therefor (without the editorializing 
on persistent export), and propose to the list.

Best regards,

Brian


On Oct 23, 2012, at 11:59 PM, John Court <[e-mail]> wrote:

Thanks for the interest in resolving this, 

Andrew, I understand your persistent cache argument.  The reason I don't 
personally use "Total" and DO use "Delta"  is more around what the 
collector does should it not see some of the updates and suddenly gets one 
that shows large counter values.  This could mistakenly result in showing 
huge traffic over a short period incorrectly.  This is particularly true 
if you do as Brian suggested and are setting the flowStartTime based only 
on the current record view. 

Brian, thanks for your detailed explanation.  Everything with the 
exception of flowStartTime was as I am currently doing.  I had perhaps 
mistakenly taken the approach that keeping the flowStartTime as the 
"conceptual" start rather than for this reporting period would make it 
easier for the collector to understand what was happening.  Not actually 
sure how I will co-ordinate reseting that time on export yet :-) 

I would suggest that this sort of situation and Brians explanation be 
added as an example perhaps in 5101 or even 5472 as it would help with 
Collector interoperability I am sure :-) 

Thanks again. 


John Court
Software Engineer
IBM Security Systems Division
IBM Australia Development Laboratory
Office:  [phone]
Mobile: [phone]

_______________________________________________
IPFIX mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/ipfix