Re: [IPFIX] unobserved fields

Brian,

> Hi, Paul, Andrew,
>
> I'm not sure I understand the proposed mechanism here...
>
> Observed/unobserved is a per-field/per-record attribute, while extended field specifiers modify IEs per-column: every instance of a given IE in a Template is "decorated" with its extension.

Actually each EFS can contribute additional data within the template 
(ie, applying per column) and/or additional info within data records 
(ie, applying per row).

Look at slide 7 here: 
http://tools.ietf.org/agenda/85/slides/slides-85-ipfix-2.pdf

eg, within the template we'd want to indicate which OID is being 
exported, while we want to indicate indices per data-record.

> As I understand the requirements for -unobserved-fields, we want a way to distinguish, from an MP: "I am capable of observing this attribute, but it was not present in the observation"

Or, I don't (yet) have sufficient data to calculate a derived metric, eg 
average / min / max, count or time based metrics.

> versus "I am not capable of observing this attribute". We have the second already: simply leave the IE out of the template.

No. Leaving the metric out of the template does not actively indicate 
that the MP is not capable of measuring it. A CP cannot make any 
assumptions about metrics which the MP isn't reporting. Rather than this 
*passive* method, we're missing a way for the MP to *actively* inform 
the CP that although it's capable of measuring some metric, it's 
currently not able to provide any further information about the metric.

> And, indeed, you can also use leaving things out of the template to signify the first: the presence of a field in some templates from an EP/MP combination implies that it's observable, its absence in the other implies it's not observed.

No it doesn't. That's the whole point of -unobserved-fields! The MP 
could have observed it and between the MP and EP, decided not to export 
it. eg, certain IP addresses, user names, URLs, could be filtered from 
export stream for privacy or security. The absence of a field in the 
template isn't directly related to the field's observability.

> FWIW, YAF does this for those things that haven't moved behind the 6313 curtain: no reverse elements if it's not a biflow, for instance.
>
> Given that, how would you use a per-IE specifier to signal a per-record attribute? What am I missing?

EFS are also per-record, eg OID index. See the MIB draft.

P.

> On 7 Dec 2012, at 20:01, Paul Aitken <paitken@cisco.com> wrote:
>
>> Andrew,
>>
>>> My earlier email about Extended Field Specifiers reminded me of an other draft (draft-aitken-ipfix-unobserved-fields) that doesn't seem to have gone anywhere, but that I would love to move forward. Having a standard way to know if  field is unobserved/NULL for a given flow would be very valuable.
>> It's not that this draft didn't go anywhere. Rather, I've written all I can for now, yet the WG isn't in a position to adopt the work. However it's a real issue that needs to be solved -in fact, we're already using some of the techniques in this document - so in my mind it's only "on hold" until I think of something to add and/or the WG can adopt it.
>>
>> You're right, Extended Field Specifiers would allow us to tag each field with an "Observed" attribute with advantages:
>>
>>     - we wouldn't need to change fields to varlen just to report length = 0.
>>
>>     - we could have a many-valued reason rather than a boolean.
>>         ie, we can report why the value is not available or not applicable.
>>
>> P.
>> _______________________________________________
>> IPFIX mailing list
>> IPFIX@ietf.org
>> https://www.ietf.org/mailman/listinfo/ipfix