Re: [IPFIX] [Fwd: questions regarding rfc 5101 implementation]

[Brian Trammell <[e-mail]>]

On Jan 20, 2010, at 12:19 PM, Paul Aitken wrote:

> Gerhard, Allwyn,
> 
>>> So, the first question is regarding the use of templates. Is an
>>> implementation supposed to maintain multiple templates for the
>>> different types of traffic? Or, is it supposed to have a single
>>> template and to use "dummy" values when a field is not applicable?
>>> I realize this could be implementation specific, but is there a
>>> guideline?
>> 
>> That is not specified in the standard, so it is implementation
>> specific. There is no guideline.
>> 
>> Cisco uses dummy values in NetFlow.V9 (e.g. zero port-numbers for
>> ICMP flows).
>> 
>> The problem is that you may have a large number of templates, which
>> can be inefficient. On the other hand, such invalid dummy values do
>> not exist for all IEs.
>> 
>> If we use a single template, the solution will be to have a new IE
>> which defines a bitvector, and 1s in the bit vector indicate "valid"
>> fields in the record, 0s "invalid" fields (similar to
>> flowKeyIndicator).
>> 
>> The idea with this new IE is not mine, Paul Aitken mentioned it to
>> me some time ago.
> 
> I'm planning to write a new draft to express this idea. We may use
> structured data rather than a bitfield to express the invalid
> (unobserved) fields, since there would be issues with mediators that
> don't understand the bitfield. The idea is not mine; Andrew Johnson
> mentioned it to me some time ago.

I remember a conversation with Andrew, too, maybe in San Diego?

Basically, "yes" to this thread, "but".

I'll note that template flexibility is one of the big advantages of IPFIX, and that I don't really buy the multiple template inefficiency argument. In the worst case export, with multiple templates representing the same underlying data model but omitting nulls or substituting reduced- for full-length encoding IEs, the template switch overhead is 4 octets per record (plus, unless you're using UDP (which you shouldn't be) with a ridiculous template refresh time, epsilon octets per record for additional template export). This worst case represents each record in its own data set, and the average case is probably somewhat better than that, especially depending on implementation and deployment specifics such as maximum buffer delay time (which leads to messages being exported without being MTU-sized), path MTU, and per-template buffering. 

A null bitfield introduces for all but the most trivial templates a 2- to 4-octet overhead, and only allows null omission, not concurrent export of full- and reduced-length encoded IEs. There are advantages to the FLE/RLE switch: for a common example, consider octet and packet count IEs, which are by default 8 octets, but in almost all flows (by flow count, not by octet volume) can be represented with 2 or 4. This saves 8 octets per record (16 per biflow) when allowing 4- and 8- octets encodings, which saves 4 (12) net octets even in the pathological case, where every other record in the stream is RLE. Of course, once you're using this multiple-template export trick for RLS, you can apply the same for null omission, with no implementation pain.

In any case, in general we do need something better than simple dummy values. They're okay for counters (where a 0 and a "dummy" value have exactly the same semantics) and identifiers where the number space has an explicit null value, probably okay in cases (such as ports) where additional information (such as a protocolIdentifier value set to a portless transport) allows disambiguation, and only maybe okay in cases where protocol-specific values (think 0.0.0.0 address for DHCP) or nonsensical ones (e.g. 0 UTC 1 January 1970) can be disambiguated with special-casing or human intervention. Unless and until null bitfields are available, multiple-template export is what we have.

Cheers,

Brian