Re: [IPFIX] Semantic and structured data

[Atsushi Kobayashi <[e-mail]>]

Hi Benoit,

On Thu, 18 Mar 2010 09:52:20 +0100
Benoit Claise <[e-mail]> wrote:

> Kobayashi-san,
> > Hi Benoit, and all,
> >
> > I agree solution3 rather than new semantic IE.
> > As Benoit mentioned, semantic IE seems to increase the difficulty to
> > interpret them.
> >
> > As Brian mentioned, I think it should avoid to have unnecessary
> > complexity. But, I am not sure what type, e.g., NONE, OR, AND, NOR,
> > RANGE, and ORDERRED, is needed. I would like to avoid that one data
> > structure is represented in multiple ways.
> So do I.
>  From this, I conclude that a simple enumeration type is better compared 
> to the bitvector encoding.
> If we include a bitvector with one bit for NOT, we end up in this situation.
> 
>     NOT OR = NOR
>     NOT AND = NAND
>     NOT ORDERED = UNORDERED (*)
>     NOT UNORDERED = ORDERED (*)
>     NOT NONE => doesn't make sense  (**)
>     NOT RANGE => doesn't make sense  (**)
>     NOT NOT => doesn't make sense  (**)
> 
> 
> (*) implies that we can represent the same semantic different ways. Now, 
> we can argue that we don't need UNORDERED

If using "UNORDERED", we can use "AND" or "OR" alternatively.
In my conclusion, we can remove it.

> (**) add non possible and confusing semantics
> > To examine it, we needs more
> > practical examples in addition to: Selector Report
> > Interpretation and interface lists on aggregated Flow Records.
> >

That make sense.

> > How about BGP AS Path, Community? Anything else?
> >
> > When I have the following BGP AS Path mixing as-sequence and as-set, how
> > to present it?
> >
> > 10 20 30 40 {50,60}
> >
> > (basicList, ORDERED, (basicList, ORDERED, AS10,AS20,AS30,AS40),
> > (basicList, OR, AS50, AS60))
> >
> > Is it correct?
> >    
> Exactly, which implies that the ORDER semantic is required for this use 
> case.
> 

We needs more other examples.

Regards,
Atsushi


> Regards, Benoit.
> > Regards,
> > Atsushi
> >
> > On Wed, 17 Mar 2010 14:10:35 +0100
> > Benoit Claise<[e-mail]>  wrote:
> >
> >    
> >> Thanks Gerhard for your feedback.
> >> See inline.
> >>      
> >>> Hi all,
> >>>
> >>> Some general thoughts from my side:
> >>>
> >>> - I appreciate that you want to add a basic notion of semantic to the
> >>>    structured data.
> >>>        
> >> Great.
> >>      
> >>> - Up to now, semantic was not in the protocol but in the info model.
> >>>
> >>> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> >>> |0|               Field ID    |       Element Length            |
> >>> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> >>> | Semantic  |             BasicList Content ...                 |
> >>> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> >>> |                           ...                                 |
> >>> +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> >>>
> >>>    Can't we encode Semantic in an IE?
> >>>    Then, we could do without a new IANA registry.
> >>>    And: This IE could be used for other purposes (e.g., in Templates)
> >>>        
> >> We thought about this one. However, one of the IPFIX principle is that
> >> the semantic of one IE can't depend on the value or position of some
> >> other IEs.
> >> One example of this is the (MPLS label position, MPLS label). Way before
> >> structured data, our initial implementation contained: MPLS label
> >> position, MPLS label.
> >> So the collector had to look first at the MPLS label position value in
> >> order to correctly understand the following  MPLS label.
> >> Somehow, with these conventions, we were having an information model on
> >> the top of the IE. This was wrong.
> >> Also it was not obvious for a collector, i.e. without hardcoding the
> >> information.
> >> Finally, what if the IE order changed within the flow record?
> >> Conclusion: we had to change back our implementation mplslabelposition1,
> >> mplslabelposition2, etc... Like we did in IPIFX ;-)
> >>
> >>
> >>      
> >>> - I'm not sure whether the bitvector encoding has an advantage over an
> >>>    enumeration type.
> >>>
> >>> Unfortunately, I do not have the time to participate in a deep
> >>> discussion - I'm busy with IPFIX-CONFIG and other stuff.
> >>>        
> >> Thanks and regards, Benoit.
> >>      
> >>> Regards,
> >>> Gerhard
> >>>
> >>> Benoit Claise wrote:
> >>>        
> >>>> Hi Brian,
> >>>>          
> >>>>> Hi, Benoit,
> >>>>>
> >>>>> Replies inline...
> >>>>>
> >>>>> On Mar 16, 2010, at 2:40 AM, Benoit Claise wrote:
> >>>>>
> >>>>>            
> >>>>>> Hi Brian,
> >>>>>>
> >>>>>> Thanks for your feedback.
> >>>>>>              
> >>>>>>> hi, Benoit, all...
> >>>>>>>
> >>>>>>> Agreed that we should do something about this (i.e., that solution
> >>>>>>> 1 is no solution.); that said, a few comments in no particular order:
> >>>>>>>
> >>>>>>> 1. In considering adding explicit semantics to structured data, we
> >>>>>>> as a WG are taking on the task of defining semantics for IPFIX as
> >>>>>>> a whole. Semantics, as I understand them, are in IPFIX largely
> >>>>>>> contextual and template dependent, but in almost all cases I can
> >>>>>>> think of it seems like these are implicitly "AND" semnantics (this
> >>>>>>> flow has source IP A _AND_ destination IP B _AND_...).
> >>>>>>>
> >>>>>>>                
> >>>>>> Agreed.
> >>>>>>              
> >>>>>>> We will need to make an explicit statement on this.
> >>>>>>>                
> >>>>>> Not sure why.
> >>>>>>              
> >>>>>>> We will need to determine whether these implicit semantics are a
> >>>>>>> property of the protocol (in which case Structured Data is really
> >>>>>>> a protocol-level extension), or a property of each information
> >>>>>>> element (in which case all 5103 IEs have implicit AND semantics;
> >>>>>>> question 2: will we need to add semantics to the IANA registry in
> >>>>>>> this case?). We will need to be quite careful about this. It's not
> >>>>>>> as simple as defining semantics within structured data then
> >>>>>>> calling it done.
> >>>>>>>
> >>>>>>>
> >>>>>>>                
> >>>>>> I'm not sure why each individual IE should have a semantic ... in
> >>>>>> the IANA registry.
> >>>>>>              
> >>>>> I'm not either. On reflection it seems like overkill. I'm just
> >>>>> saying that if we, as a WG, are moving from stating that semantics
> >>>>> are explicitly out of scope to defining them as in-scope, we need to
> >>>>> have a consistent approach, and answers to all the questions that
> >>>>> arise when we consider moving the protocol from a simple framing
> >>>>> mechanism to a framing mechanism with some logic behind it, so we
> >>>>> answer them once, so that all future efforts having to do with
> >>>>> semantics are consistent. This, I acknowledge, is an argument in
> >>>>> favor of solution 3...)
> >>>>>
> >>>>> One very simple new question that arises here, to illustrate my
> >>>>> point: Is it legal to export a record that has sourceIPAddress X AND
> >>>>> NOT sourceIPAddress X?
> >>>>>            
> >>>>    From a protocol point of view, yes
> >>>>   From a semantic point of view, I don't see a use case for that.
> >>>> Now, this question is not different that: with RFC5101,  is it legal
> >>>> to export a record that has two instances of sourceIPAddress?
> >>>>          
> >>>>>> As you wrote, "in almost all cases I can think of it seems like
> >>>>>> these are implicitly "AND" semnantics", so the case we try to solve
> >>>>>> is when there are multiple instances of a single IE. We know that
> >>>>>> RFC 5101 foresaw that case " The Collector MUST support the use of
> >>>>>> Templates containing multiple occurrences of the similar
> >>>>>> Information Elements", but the idea is not to change RFC5101.  If
> >>>>>> we put some semantic in the IPFIX structured data, that would solve
> >>>>>> the vast majority of our cases. Also, we could say: if you want
> >>>>>> some semantic when exporting multiple IEs, then you SHOULD use
> >>>>>> IPFIX structured data.
> >>>>>>
> >>>>>> Also, the default value for the semantic field in the IPFIX
> >>>>>> structured data SHOULD be "NONE", to express that flow record
> >>>>>> doesn't include any semantic.... like in RFC5101. You might draw
> >>>>>> your own conclusion, maybe because you know your network, maybe
> >>>>>> because you have configured the exporter, but then it's your decision.
> >>>>>> The way I see the proposed solution is: in IPFIX structured data,
> >>>>>> you MAY use the semantic field as a way to express the relationship
> >>>>>> between IEs within the structure.
> >>>>>>              
> >>>>>>> 2. I don't consider draft-sommer-ipfix-mediator-ext-01 a valid
> >>>>>>> argument against Solution 2, as it's trying to solve a somewhat
> >>>>>>> different and more limited problem than structured data. Solution
> >>>>>>> 2 _might_ cause a problem for this draft, but certainly not the
> >>>>>>> other way around (unless we as a WG want to subsume that work into
> >>>>>>> this draft, which would probably require rechartering...). Also, I
> >>>>>>> don't think we need semantics for all of the list types, just
> >>>>>>> basicList (Illustrative question: How do I meaningfully interpret
> >>>>>>> two "or" subTemplateMultiLists with disjoint IE sets? Two "and"
> >>>>>>> subTemplateMultiLists? Nestings thereof?)... So, we don't really
> >>>>>>> have an explosion to deal with if we do Solution 2 correctly:
> >>>>>>> andBasicList, orBasicList, xorBasicList, notBasicList; Four IEs,
> >>>>>>> nestable, done. What can't we do with those four IEs? In this
> >>>>>>> case, we could even step back and say that semantics outside these
> >>>>>>> four are within the protocol explicitly _undefined_, and to be
> >>>>>>> interpreted withi
> >>>>>>>                
> >>> n the
> >>>        
> >>>>>>>    context of each Template.
> >>>>>>>
> >>>>>>>
> >>>>>>>                
> >>>>>> If I translate some more what I wrote "While it's solved the router
> >>>>>> and most mediation function needs today", this would be "Me,
> >>>>>> myself, and I don't need more that OR and AND now ;-)".
> >>>>>> However, who am I to tell that others don't need it now... and that
> >>>>>> the logical solution is to use the IPFIX structured data
> >>>>>> Furthermore, if we think a little bit longer term, the next big
> >>>>>> step in IPFIX is the mediation function. In my company, every
> >>>>>> features want to export his own data with NetFlow/IPFIX... up to
> >>>>>> the point where a CPE would not have enough bandwidth across the
> >>>>>> WAN to export all the "management" information. So we'll need more
> >>>>>> and more of aggregated flow records (both in time and space) even
> >>>>>> in the router. Again, the logical solution will be to use the IPFIX
> >>>>>> structured data. At this point, we will most probably need
> >>>>>> something else than OR and AND, i.e. RANGE, ORDERRED, etc...
> >>>>>>
> >>>>>> An example of subTemplateMultiLists with disjoint IE sets, let's
> >>>>>> imagine that you have to export an aggregated observation point,
> >>>>>> composed of multiple template records
> >>>>>>       template 1: exporterIPaddress
> >>>>>>       template 2: exporterIPaddress, basicList of interfaces
> >>>>>>       template 3: exporterIPaddress, LC
> >>>>>>              
> >>>>> and then you'd want to OR these... Okay, makes sense...
> >>>>>
> >>>>>            
> >>>>>>> 3. If we really _do_ want ranges and so on (which, again, we'd
> >>>>>>> need to get WG consensus on; this is explicitly out of scope in my
> >>>>>>> reading of the present charter), then we could do them in the
> >>>>>>> scope of Solution 3.
> >>>>>>>                
> >>>>>> Btw, acknowledging that one day we will have to solve this is good
> >>>>>> enough for solution 3. I mean, we don't have to populate the
> >>>>>> semantic IANA now... even if that would be more efficient.
> >>>>>>              
> >>>>>>> However, this seems a little not-quite-fleshed-out-enough for me
> >>>>>>> to say whether I like it or not. Could you present an example of
> >>>>>>> how you would use the proposed Semantic field to model your
> >>>>>>> example? "(eth1 OR eth2) AND (NOT (eth3 OR eth4)) OR linecard2"
> >>>>>>>
> >>>>>>> (FWIW, I would do this with my proposal to Solution 2 as follows:)
> >>>>>>>
> >>>>>>> (orBasicList (andBasicList (orBasicList ingressInterface eth1
> >>>>>>> eth2) (notBasicList (orBasicList ingressInterface eth3 eth4))
> >>>>>>> (andBasicList lineCardID 2))
> >>>>>>>
> >>>>>>>
> >>>>>>>                
> >>>>>> (BasicList, OR, (basicList, AND, (basicList, OR, eth1, eth2),
> >>>>>> (basicList, NOR, eth3, eth4)), (basicList, NONE, linecard2)))
> >>>>>>              
> >>>>> Hm. Okay. This makes sense. A couple of questions then, about
> >>>>> solution 3:
> >>>>>
> >>>>> 1. Would we want to try and bitfield this, to define the semantics
> >>>>> we _know_ we have, then leave the rest of it reserved? Something like:
> >>>>>
> >>>>> MSb                      LSb
> >>>>> +---+------+---------------+
> >>>>> | ! | multi| reserved      |
> >>>>> +---+------+---------------+
> >>>>>
> >>>>> ! = negate sense of semantics if 1 (this is the NOT flag)
> >>>>> multi (multiplicity) = 00: undefined, 01: or/oneOrMore, 10:
> >>>>> xor/exactlyOne, 11: and/exactlyAll
> >>>>> reserved = place for adding other bells and whistles like RANGE and
> >>>>> so on in the future.
> >>>>>            
> >>>> Maybe we want to start by asking the question: which semantic do we
> >>>> need now?
> >>>> Is a need for NONE, OR, AND, ORDERED for now. Anything else?
> >>>>          
> >>>>> 2. Are we sure we want to do this in one byte? If we do bitfielding
> >>>>> as above this gives us 32 possible extensions, which seems like
> >>>>> _way_ more than enough, but does stick another odd offset in there,
> >>>>> which slows things down on machines that need aligned access.
> >>>>> Probably one byte is okay and we let the implementation use
> >>>>> paddingOctets and set padding to fix this...
> >>>>>            
> >>>> With structured data, is the alignment still important? When I look
> >>>> at the examples throughout the draft... As you wrote, paddingOctets
> >>>> might be the solution in this case.
> >>>>          
> >>>>> 3. Would it make sense maybe to have two sets of structured data
> >>>>> elements, one with the semantics byte, and one without (which is
> >>>>> then explicitly undefined)? Then exporters who don't need it don't
> >>>>> have to bother sticking an extra odd-aligned zero in the stream for
> >>>>> every list.
> >>>>>
> >>>>> I'm sure I'l have more questions, but none come to mind now... But
> >>>>> it seems like we're converging on the least-unnecessarily-complex
> >>>>> solution here, which is good. :)
> >>>>>            
> >>>> Happy about that. ;-)
> >>>> Note: I was envisioning even something more simpler: one byte,
> >>>> containing all the semantic possibilities, administered by IANA.  So
> >>>> no reserved, no !
> >>>>
> >>>> Regards, Benoit.
> >>>>          
> >>>        
> >> _______________________________________________
> >> IPFIX mailing list
> >> [e-mail]
> >> https://www.ietf.org/mailman/listinfo/ipfix
> >>
> >>      
> > ---
> > Atsushi KOBAYASHI<[e-mail]>
> > NTT Information Sharing Platform Lab.
> > tel:[phone] fax:[phone]
> >
> >    
> 

-- 
Atsushi Kobayashi <[e-mail]>