Re: [IPFIX] [Fwd: questions regarding rfc 5101 implementation]

[Gerhard Muenz <[e-mail]>]

Brian,

Brian Trammell wrote:
> On Jan 20, 2010, at 12:19 PM, Paul Aitken wrote:
> 
>> Gerhard, Allwyn,
>> 
>>>> So, the first question is regarding the use of templates. Is an
>>>>  implementation supposed to maintain multiple templates for the
>>>>  different types of traffic? Or, is it supposed to have a
>>>> single template and to use "dummy" values when a field is not
>>>> applicable? I realize this could be implementation specific,
>>>> but is there a guideline?
>>> That is not specified in the standard, so it is implementation 
>>> specific. There is no guideline.
>>> 
>>> Cisco uses dummy values in NetFlow.V9 (e.g. zero port-numbers for
>>>  ICMP flows).
>>> 
>>> The problem is that you may have a large number of templates,
>>> which can be inefficient. On the other hand, such invalid dummy
>>> values do not exist for all IEs.
>>> 
>>> If we use a single template, the solution will be to have a new
>>> IE which defines a bitvector, and 1s in the bit vector indicate
>>> "valid" fields in the record, 0s "invalid" fields (similar to 
>>> flowKeyIndicator).
>>> 
>>> The idea with this new IE is not mine, Paul Aitken mentioned it
>>> to me some time ago.
>> I'm planning to write a new draft to express this idea. We may use 
>> structured data rather than a bitfield to express the invalid 
>> (unobserved) fields, since there would be issues with mediators
>> that don't understand the bitfield. The idea is not mine; Andrew
>> Johnson mentioned it to me some time ago.
> 
> I remember a conversation with Andrew, too, maybe in San Diego?
> 
> Basically, "yes" to this thread, "but".
> 
> I'll note that template flexibility is one of the big advantages of
> IPFIX, and that I don't really buy the multiple template inefficiency
> argument. In the worst case export, with multiple templates
> representing the same underlying data model but omitting nulls or
> substituting reduced- for full-length encoding IEs, the template
> switch overhead is 4 octets per record (plus, unless you're using UDP
> (which you shouldn't be) with a ridiculous template refresh time,
> epsilon octets per record for additional template export). This worst
> case represents each record in its own data set, and the average case
> is probably somewhat better than that, especially depending on
> implementation and deployment specifics such as maximum buffer delay
> time (which leads to messages being exported without being
> MTU-sized), path MTU, and per-template buffering.

I think the argument against using many templates is that the exporting 
process implementation becomes more complex.
With "invalid" fields, it can use a single template per cache/metering 
process. Without, it has to decide among multiple templates.

Note that in general, I'm also not in favor of invalid fields in data 
records.

Regards,
Gerhard


> A null bitfield introduces for all but the most trivial templates a
> 2- to 4-octet overhead, and only allows null omission, not concurrent
> export of full- and reduced-length encoded IEs. There are advantages
> to the FLE/RLE switch: for a common example, consider octet and
> packet count IEs, which are by default 8 octets, but in almost all
> flows (by flow count, not by octet volume) can be represented with 2
> or 4. This saves 8 octets per record (16 per biflow) when allowing 4-
> and 8- octets encodings, which saves 4 (12) net octets even in the
> pathological case, where every other record in the stream is RLE. Of
> course, once you're using this multiple-template export trick for
> RLS, you can apply the same for null omission, with no implementation
> pain.
> 
> In any case, in general we do need something better than simple dummy
> values. They're okay for counters (where a 0 and a "dummy" value have
> exactly the same semantics) and identifiers where the number space
> has an explicit null value, probably okay in cases (such as ports)
> where additional information (such as a protocolIdentifier value set
> to a portless transport) allows disambiguation, and only maybe okay
> in cases where protocol-specific values (think 0.0.0.0 address for
> DHCP) or nonsensical ones (e.g. 0 UTC 1 January 1970) can be
> disambiguated with special-casing or human intervention. Unless and
> until null bitfields are available, multiple-template export is what
> we have.
> 
> Cheers,
> 
> Brian _______________________________________________ IPFIX mailing
> list [e-mail] https://www.ietf.org/mailman/listinfo/ipfix