Re: [IPFIX] Export of long lived flow information

[John Court <[e-mail]>]

>>OLD:
>>The absolute timestamp of the first|last packet of this Flow.
>>NEW:
>>The absolute timestamp of the first|last packet accounted in this Flow 
>>Record.

For some reason that does make it much clearer to me although that could 
just be because I have had the benefit of all the preceding discussion.

I think that clears up exactly what the statistics represent in a flow 
record.  The only nastiness is that we use terminology of flowEnd* when 
really if you are using totals, conceptually the flow really ISN'T ending 
its just being "Reported".  I am not suggesting changes to anything just 
hopefully making it clear why the mis-interpretations may occur.

For my own purposes its clear I can only use deltas as my work is at the 
application level and only bridges and routers would seem to make sense 
for the "total" counters concept given your definition. For me the 
aggregation into "totals" is up to the collector.

Again thanks for every ones patience it has certainly helped further my 
education and hopefully improve future interoperability of IPFIX devices 
:-)

John Court




From:   Gerhard Muenz <[e-mail]>
To:     John Court/Australia/IBM@IBMAU, 
Cc:     Paul Aitken <[e-mail]>, Andrew Johnson 
<[e-mail]>, [e-mail]
Date:   26/10/2012 04:27
Subject:        Re: [IPFIX] Export of long lived flow information




Hi,

It seems that some Information Element descriptions need clarification. 
For example, I think that the description of the Information Elements 
flowStart* and flowEnd* should be clarified:

OLD:
The absolute timestamp of the first|last packet of this Flow.
NEW:
The absolute timestamp of the first|last packet accounted in this Flow 
Record.

I assume that this is how it is implemented in most existing 
implementations. Also, I assume that this is the intended meaning.

My understanding is that both, deltaCounts and totalCounts contain the 
number of packets or octets observed in the indicated time interval. So, 
for identical flowStart* and flowEnd* timestamps, the values are the same.

However, the description of totalCounts says that you report the number 
of packets or octets observed for this Flow since re-initialization. So, 
you must never reset the counter for this Flow, even after observing a 
FIN or RST.
If you reset flow counters, or if you remove Flows from your Cache, you 
cannot use totalCounts any more unless you re-initialize the Metering 
Process (e.g. after flushing the entire permanent Cache).

Using totalCounts, the flowStart* timestamp is identical in all Flow 
Records of the same Flow. Also, the collector knows that - for all but 
the first Flow Record of a Flow - the totalCount values include packets 
which were already reported in earlier Flow Records for the same Flow. 
Hence, each new Flow Record of the same Flow is an update of the 
previous ones. Summing up totalCount values in these Flow Records 
results in duplicate counts.
On the other hand, with deltaCounts, the Flow Records refer to distinct 
time intervals. So, you can sum up counters without having duplicates.

Although these subtle differences are not very obvious, the Information 
Element descriptions are quite clear. flowEndReason can be used to 
report some extra information but is not needed to understand the 
meaning of the Flow Records.

Thanks,
Gerhard


On 25.10.2012 03:04, John Court wrote:
> Yep I think everyone is starting to see the ambiguity that needs to be
> cleared up :-)
>
> Paul,
>
> *Definitely. If it's a permanent flow and you're exporting totalCount
> fields - which are measured "since the Metering Process
> (re-)initialization for this Observation Point" - then the flowStartTime
> must surely be the time the first ever packet was observed.*
>
> If you take that literally shouldn't that be interpreted to mean that
> the totalCount continues into the next time a connection is up between
> the same flow key ?  Even if a flowEndReason of :
>
>   0x03: end of Flow detected
> The Flow was terminated because the Metering Process
> detected signals indicating the end of the Flow, for
> example, the TCP FIN flag.
>
> That clearly wouldn't be of much use IMO and makes it difficult to see
> what the flowEndReason field semantics mean in that context.  Just
> pointing out that taking that definition literally doesn't give a useful
> answer on its own either :-).  Although maybe that does make sense in a
> router context ?  Can you clarify this some more, perhaps you never
> intend using the flowEndReason IE in your case ?
>
> Thanks
>
>
>
>
>
> From: Paul Aitken <[e-mail]>
> To: Andrew Johnson <[e-mail]>,
> Cc: Gerhard Muenz <[e-mail]>, John Court/Australia/IBM@IBMAU,
> [e-mail]
> Date: 25/10/2012 08:07
> Subject: Re: [IPFIX] Export of long lived flow information
> ------------------------------------------------------------------------
>
>
>
> Andrew,
>
>  > I was thinking that a mechanism that allowed a non-permanent flow to
> be exported multiple time would be useful.  For example, security
> applications generally want to know about a new flow ASAP, so they can
> act on the information, but a short active timeout values lead to using
> more export bandwidth.  I was thinking we could do something like export
> a report of the flow after the first packet, and then export the final
> version of the flow once the normal timeouts had decided it was over.
>
> I have in the past discussed the idea of exporting a "new flow alert"
> using zero-valued counters in order to make the collector aware that
> we've started monitoring it - so I'm claiming prior art on that.
>
>
>  > I had in mind something like using a delta count, followed by a total
> count.  Reading the below definition of Total counts though, I'm not
> sure that will work, but I think it depends on how we interpret the
> definition of "Flow".  If two records have matching key fields but
> different starting timestamps, are they the same Flow?
>
> 5101 defines:
>
>        A Flow is defined as a set of IP packets passing an Observation
>        Point in the network during a certain time interval.
>
>
> - so it's all about the timestamps :-)
>
>
>  > I would argue that a single Flow can't have two flowStartTimes, so
> maybe not.
>
> However, two flows with different flowStartTimes can be merged into one
> flow.
>
>
>  > This would mean that we shouldn't reset the flowStartTimes between
> sending reports for the same permanent Flow.
>
> Definitely. If it's a permanent flow and you're exporting totalCount
> fields - which are measured "since the Metering Process
> (re-)initialization for this Observation Point" - then the flowStartTime
> must surely be the time the first ever packet was observed.
>
> P.
>
>