Re: [IPFIX] Export of long lived flow information

[Andrew Johnson <[e-mail]>]

Hi folks

A flow is defined as a set of packets which all have some common properties, and the original idea of a flow was based on the common 5-tuple of IP address, IP protocol and ports.  A Flow Record is formed from any observation of packets belonging to a Flow (not all packets within the flow are necessarily observed).

If my PC opens an HTTP connection to some server and downloads a simple web page, then we'd expect to see one traditional flows (per direction, but ignore that for now).  If I open a new HTTP connection to the same server, and coincidentally use the same source port, weeks later, is that the same flow?

I had always thought of these as two flows, and the flowEndReason implies it, but that would introduce some sort of time property as one of the common property shared by the packets that make up a flow.  A collector might aggregate the two reports, but removing the time property is much like any other form of aggregation.

It seems to me that we're using the timeout values of the cache as a sort of ill-defined common property, but things get confusing when we export total counts, or because the cache is low on resources, etc.  Ideally, we'd want to be able to send more than one Flow Record for the same Flow and provide enough information for the Collector to reconstruct what the Monitoring Process is using to define a Flow.


Cheers, Andrew



On 25 Oct 2012, at 22:40, John Court wrote:
> >>OLD:
> >>The absolute timestamp of the first|last packet of this Flow.
> >>NEW:
> >>The absolute timestamp of the first|last packet accounted in this Flow 
> >>Record.
> 
> For some reason that does make it much clearer to me although that could just be because I have had the benefit of all the preceding discussion. 
> 
> I think that clears up exactly what the statistics represent in a flow record.  The only nastiness is that we use terminology of flowEnd* when really if you are using totals, conceptually the flow really ISN'T ending its just being "Reported".  I am not suggesting changes to anything just hopefully making it clear why the mis-interpretations may occur. 
> 
> For my own purposes its clear I can only use deltas as my work is at the application level and only bridges and routers would seem to make sense for the "total" counters concept given your definition. For me the aggregation into "totals" is up to the collector. 
> 
> Again thanks for every ones patience it has certainly helped further my education and hopefully improve future interoperability of IPFIX devices :-) 
> 
> John Court 
> 
> 
> 
> 
> From:        Gerhard Muenz <[e-mail]> 
> To:        John Court/Australia/IBM@IBMAU, 
> Cc:        Paul Aitken <[e-mail]>, Andrew Johnson <[e-mail]>, [e-mail] 
> Date:        26/10/2012 04:27 
> Subject:        Re: [IPFIX] Export of long lived flow information 
> 
> 
> 
> 
> Hi,
> 
> It seems that some Information Element descriptions need clarification. 
> For example, I think that the description of the Information Elements 
> flowStart* and flowEnd* should be clarified:
> 
> OLD:
> The absolute timestamp of the first|last packet of this Flow.
> NEW:
> The absolute timestamp of the first|last packet accounted in this Flow 
> Record.
> 
> I assume that this is how it is implemented in most existing 
> implementations. Also, I assume that this is the intended meaning.
> 
> My understanding is that both, deltaCounts and totalCounts contain the 
> number of packets or octets observed in the indicated time interval. So, 
> for identical flowStart* and flowEnd* timestamps, the values are the same.
> 
> However, the description of totalCounts says that you report the number 
> of packets or octets observed for this Flow since re-initialization. So, 
> you must never reset the counter for this Flow, even after observing a 
> FIN or RST.
> If you reset flow counters, or if you remove Flows from your Cache, you 
> cannot use totalCounts any more unless you re-initialize the Metering 
> Process (e.g. after flushing the entire permanent Cache).
> 
> Using totalCounts, the flowStart* timestamp is identical in all Flow 
> Records of the same Flow. Also, the collector knows that - for all but 
> the first Flow Record of a Flow - the totalCount values include packets 
> which were already reported in earlier Flow Records for the same Flow. 
> Hence, each new Flow Record of the same Flow is an update of the 
> previous ones. Summing up totalCount values in these Flow Records 
> results in duplicate counts.
> On the other hand, with deltaCounts, the Flow Records refer to distinct 
> time intervals. So, you can sum up counters without having duplicates.
> 
> Although these subtle differences are not very obvious, the Information 
> Element descriptions are quite clear. flowEndReason can be used to 
> report some extra information but is not needed to understand the 
> meaning of the Flow Records.
> 
> Thanks,
> Gerhard
> 
> 
> On 25.10.2012 03:04, John Court wrote:
> > Yep I think everyone is starting to see the ambiguity that needs to be
> > cleared up :-)
> >
> > Paul,
> >
> > *Definitely. If it's a permanent flow and you're exporting totalCount
> > fields - which are measured "since the Metering Process
> > (re-)initialization for this Observation Point" - then the flowStartTime
> > must surely be the time the first ever packet was observed.*
> >
> > If you take that literally shouldn't that be interpreted to mean that
> > the totalCount continues into the next time a connection is up between
> > the same flow key ?  Even if a flowEndReason of :
> >
> >   0x03: end of Flow detected
> > The Flow was terminated because the Metering Process
> > detected signals indicating the end of the Flow, for
> > example, the TCP FIN flag.
> >
> > That clearly wouldn't be of much use IMO and makes it difficult to see
> > what the flowEndReason field semantics mean in that context.  Just
> > pointing out that taking that definition literally doesn't give a useful
> > answer on its own either :-).  Although maybe that does make sense in a
> > router context ?  Can you clarify this some more, perhaps you never
> > intend using the flowEndReason IE in your case ?
> >
> > Thanks
> >
> >
> >
> >
> >
> > From: Paul Aitken <[e-mail]>
> > To: Andrew Johnson <[e-mail]>,
> > Cc: Gerhard Muenz <[e-mail]>, John Court/Australia/IBM@IBMAU,
> > [e-mail]
> > Date: 25/10/2012 08:07
> > Subject: Re: [IPFIX] Export of long lived flow information
> > ------------------------------------------------------------------------
> >
> >
> >
> > Andrew,
> >
> >  > I was thinking that a mechanism that allowed a non-permanent flow to
> > be exported multiple time would be useful.  For example, security
> > applications generally want to know about a new flow ASAP, so they can
> > act on the information, but a short active timeout values lead to using
> > more export bandwidth.  I was thinking we could do something like export
> > a report of the flow after the first packet, and then export the final
> > version of the flow once the normal timeouts had decided it was over.
> >
> > I have in the past discussed the idea of exporting a "new flow alert"
> > using zero-valued counters in order to make the collector aware that
> > we've started monitoring it - so I'm claiming prior art on that.
> >
> >
> >  > I had in mind something like using a delta count, followed by a total
> > count.  Reading the below definition of Total counts though, I'm not
> > sure that will work, but I think it depends on how we interpret the
> > definition of "Flow".  If two records have matching key fields but
> > different starting timestamps, are they the same Flow?
> >
> > 5101 defines:
> >
> >        A Flow is defined as a set of IP packets passing an Observation
> >        Point in the network during a certain time interval.
> >
> >
> > - so it's all about the timestamps :-)
> >
> >
> >  > I would argue that a single Flow can't have two flowStartTimes, so
> > maybe not.
> >
> > However, two flows with different flowStartTimes can be merged into one
> > flow.
> >
> >
> >  > This would mean that we shouldn't reset the flowStartTimes between
> > sending reports for the same permanent Flow.
> >
> > Definitely. If it's a permanent flow and you're exporting totalCount
> > fields - which are measured "since the Metering Process
> > (re-)initialization for this Observation Point" - then the flowStartTime
> > must surely be the time the first ever packet was observed.
> >
> > P.
> >
> >
> 
>