RE: [ipfix-req] proposed changes for requirements document

["Tal Givoly" <[e-mail]>]

Carter,

Quoting from the meeting minutes of the previous meeting:

-- Dave Plonka wrote on 25 November 2002 10:30 -0600:
> The issue of reliability was discussed.  Juergen proposed, and it was
> agreed to by a hum, that the requirements document should specify
> that the IPFIX protocol MAY have "aditional reliability" and that
> it MUST be open to reliability extensions.  Tal Givoly will supply
> clarifying text about this higher level of reliability, describing
> it in an implementation independent way.

Therefore, I do not imply TCP vs. UDP but rather the "additional
reliability" that we agreed the IPFIX protocol MAY have and MUST be
extensible enough to support. Therefore, my understanding was that it was
not tossed out - but rather classified as MAY support but MUST be extensible
enough to support.

In order to demonstrate that it can be supported, I am proposing the wording
to qualify what that means from a requirement standpoint.

If you are referring to "negotiation" - that indeed is not currently a
requirement, nor have I claimed it should be in any of these recent
postings.

I hope this helps clarify.

Tal

-----Original Message-----
From: majordomo listserver [mailto:[e-mail]]On Behalf
Of Carter Bullard
Sent: Wednesday, January 15, 2003 4:51 PM
To: 'Tal Givoly'
Cc: [e-mail]; 'MEYER,JEFFREY D (HP-Cupertino,ex1)';
'Juergen Quittek'
Subject: RE: [ipfix-req] proposed changes for requirements document


Hey Tal,
   Could that mean TCP vs UDP vs RTP vs multicast, or do you
mean IPFIX level reliability?   If its IPFIX level, then
you're going to have to negotiate the optional reliability
features, and I thought that negotiation had been tossed out?

Carter



> -----Original Message-----
> From: Tal Givoly [mailto:[e-mail]]
> Sent: Wednesday, January 15, 2003 7:45 PM
> To: Carter Bullard
> Cc: [e-mail]; 'MEYER,JEFFREY D
> (HP-Cupertino,ex1)'; 'Juergen Quittek'
> Subject: RE: [ipfix-req] proposed changes for requirements document
>
>
> Carter,
>
> Haven't we agreed that the protocol MAY support more reliable
> transports than merely being able to identify the fact that
> loss has occurred? If that is the case, then we must allow
> for mechanisms by which the protocol can support them. Merely
> sequencing the records and allowing identification of the
> amount of records lost supports the "must" requirements, but
> leaves no ability to support the "may" requirement in a
> cost-effective manner (e.g. out of band recovery).
>
> The minimum must requirements of being able to identify how
> much data has been lost in a potentially unreliable delivery
> mechanism or for any other cause of loss will still be there,
> but for those applications requiring higher reliability, I
> believe it is required to provide some additional description
> as part of the requirements.
>
> Tal
>
> -----Original Message-----
> From: Carter Bullard [mailto:[e-mail]]
> Sent: Wednesday, January 15, 2003 1:08 PM
> To: 'MEYER,JEFFREY D (HP-Cupertino,ex1)'; 'Juergen Quittek';
> 'Tal Givoly'
> Cc: [e-mail]
> Subject: RE: [ipfix-req] proposed changes for requirements document
>
>
> Hey Jeff,
>    I agree that enumerating the various conditions is
> the correct thing, so I say great!
>
>    On a different note, there is a huge difference
> between detecting/indicating loss of reliability
> and assuring a level of reliability.  I am of the
> school that assuring reliability should be outside the
> scope of IPFIX.  Even simple retransmission capabilities
> are not a potential solution as the principal target
> device for IPFIX is not going to be able to support
> the feature.
>
>    But by providing the information needed to detect
> loss, such as through per record sequence numbering in
> conjunction with timestamping, an application will have all
> the info needed to enable out-of-band recovery. IPFIX could
> support recovery itself if it could handle selective record
> transfer requests.
>
> Carter
>
>
>
>
> > -----Original Message-----
> > From: majordomo listserver [mailto:[e-mail]] On
> > Behalf Of MEYER,JEFFREY D (HP-Cupertino,ex1)
> > Sent: Wednesday, January 15, 2003 3:12 PM
> > To: 'Juergen Quittek'; Tal Givoly
> > Cc: [e-mail]
> > Subject: RE: [ipfix-req] proposed changes for requirements document
> >
> >
> > Hi,
> >
> >   There seems to be agreement that losing flow information
> implies a
> > lack of reliability.  I think understanding where this information
> > loss might occur and what steps are taken to "recover" the lost
> > information is the essence of the different "levels of reliability".
> >
> >
> >   Today the requirements draft has the following statement
> in section
> > 6.3.2:
> >
> >
> >     "Possible reasons for flow information loss include resource
> >     shortage at the exporter, loss of packets between the
> >     exporting process and collecting process, etc."
> >
> >
> >   Rather than ending with "etc." it would be nice to try to
> enumerate
> > all the basic loss scenarios.  I don't think there are that many in
> > the model defined:
> >
> >
> >     ObsPt -> Metering Proc -> Exporter -> Collector
> >
> >
> >   Moving from left to right across the model, I see
> >   the following "failures" impacting reliability:
> >
> >     1. A metering process may somehow fail to "keep up" with
> >       the packets going past the ObsPt.
> >
> >       This lack of reliability is not recoverable by the
> >       exporter (i.e. The same box won't be able to "see"
> >       these flows again).  It should be reported as covered
> >       in section 5.1.
> >
> >     2. The exporter may be experiencing congestion and
> >       not be able to buffer new flows recorded by the metering
> >       process.
> >
> >       This lack of reliability will again be unrecoverable
> >       from the perspective of the exporting system.  Either
> >       old flows or newly reported flows need to be dropped.
> >       Such loss should be reported.
> >
> >     3. A packet carrying a flow sent by the Exporter to the
> Collector
> >       is dropped by the network.  This is an issue for unreliable
> >       transports such as UDP. For reliable transports, these
> >       drops are recovered by the transport itself.
> >
> >       This lack of reliability is only an issue for UDP based
> >       protocols and MAY be addressed through some means,
> >       such as buffering on the Exporter and an acknowledgement/
> >       timeout scheme.
> >
> >       Having a "failover Collector" greatly increases the
> >       likelihood that the Exporter will not exhaust buffering
> >       before successfully retransmitting flow information.
> >
> >     4. The network connection between the Exporter and the
> >       Collector is lost due to a failure (either in the network
> >       or on the collector).
> >
> >       The disposition of some flow information in transit may be
> >       unknown to the Exporter.  In addition subsequent flow
> >       information is also at risk of being lost.
> >
> >       This lack of reliability MAY be addressed by some
> >       means such as buffering on the Exporter and having
> >       a failover Collector(s) available.
> >
> >       An application layer acknowledgement scheme aids
> >       the exporter in identifying what information sent
> >       has actually been received by the Collector.
> >
> >     5. The network connection between the Exporter and
> >       the Collector is taken down for maintenance or other
> >       administrative purposes.
> >
> >       In this case it is desireable to gracefully transition
> >       the delivery of flows from one Collector to an
> >       auxilliary.
> >
> >       Procedures for graceful handoff MAY be refined from
> >       those for a simple failure (#4), to increase efficiency.
> >
> >     6. Congestion is detected between the Exporter and the
> >       Collector, which jeopardizes the delivery of flow
> >       information.   The slow receiver/link problem.
> >
> >        This may be considered either a basic failure of
> >       the collector as in #4 above, or unavoidable as in
> >       #2 above.
> >
> >
> >   So to me, it seems like reliability boils down to whether
> >   or not the protocol addresses 3, 4 and 5.
> >
> >   I think the protocol MUST address 3, 4 and 5.  However in
> >   a deployment, there MAY only be one Collector set up.  In
> >   this case the reliable delivery is likely to be impacted
> >   by failures 4 and 5 from time to time.
> >
> >   That said, I don't see this as an extension, but really
> >   a capability of the protocol.  Or perhaps it is a property of
> >   the exporter, i.e. are any flows ever buffered for
> >   retransmission in the event of loss of the communication
> >   channel between the exporter and collector.
> >
> >   Perhaps an indication on the part of the exporter as to
> >   whether it makes any attempt to retransmit to a secondary
> >   should be called out.  However I don't see lots of "levels"
> >   of reliability.
> >
> >
> > Regards,
> >
> >   Jeff Meyer
> >
> >
> > > -----Original Message-----
> > > From: Juergen Quittek [mailto:[e-mail]]
> > > Sent: Tuesday, January 14, 2003 3:26 PM
> > > To: Tal Givoly
> > > Cc: [e-mail]
> > > Subject: RE: [ipfix-req] proposed changes for
> requirements document
> > >
> > >
> > > Hi Tal,
> > >
> > > Thank you for the comments. See my replies inline.
> > >
> > >     Juergen
> > >
> > > -- Tal Givoly wrote on 14 January 2003 10:32 -0800:
> > >
> > > > Juergen,
> > > >
> > > > Some recommendation of modification to the proposed changes:
> > > >
> > > > <snip>
> > > > 4. Replace the first paragraph in section "6.3.2. Reliability":
> > > >
> > > >    "Absence of reliability of the data transfer, for
> > > example caused by
> > > >     loss or reordering of packets containing flow
> > > information, MUST be
> > > >     indicated."
> > > >
> > > >    by
> > > >
> > > >    "Different levels of reliability of the data transfer MAY be
> > > >     supported. The collecting process MUST have sufficient
> > > information
> > > >     for making a conservative assumption about the applied
> > > reliability
> > > >     level. This means that the applied level of reliability
> > > MUST be as
> > > >     high as or higher than indicated to the collecting
> process."
> > > > </snip>
> > > >
> > > > <tal>
> > > >
> > > > Regarding item 4, there seem to be a few concerns regarding
> > > the proposed
> > > > wording:
> > > >
> > > >  - It implies that the collecting process must indicate
> the level
> > > >    of reliability it is interested in, hence, there must be a
> > > >    mechanism to indicate this.
> > >
> > > No. It does not. At least, I cannot see this. Why do you think so?
> > >
> > > >  - The words "conservative assumptions" are ambiguous.
> > >
> > > I agree. They should be changed.
> > >
> > > >  - It implies the level of reliability may be higher than
> > > required by
> > > >    collecting process - it is unclear to me why this is
> implied by
> > > >    the consensus to allow the protocol to "MAY" support higher
> > > >    levels of reliability.
> > >
> > > Of course the level may be higher or lower than required by the
> > > collecting process. It just says that the collecting
> process needs
> > > sufficient information on the reliability in order to find out
> > > whether
> > or not the
> > > level is sufficient.
> > >
> > > >
> > > >
> > > > Therefore, may I propose the following alternative as a
> > > replacement for the
> > > > first paragraph in "6.3.2 Reliability":
> > > >
> > > >    "The data exchange between the exporter and the collector MAY
> > >
> > > I'd suggest "data transfer from the exporting process to the
> > > collecting process".
> > >
> > > >     support different levels of reliability.
> > > >
> > > >     For the most basic level of reliability, the
> following should
> > > >     be maintained:
> > > >      - The collecting process MUST be able to detect
> the level of
> > > >        reliability.
> > > >      - The collection process MUST be able to detect any
> > > loss of data.
> > > >
> > > >     For higher levels of reliability, the following should be
> > >
> > > Are these several higher levels or is it a single higher level
> > > including everything below?
> > >
> > > >     maintained:
> > > >      - All reliability properties requirements of lower
> levels of
> > > >        reliability MUST be maintained.
> > >
> > > Above you say "the following should b maintained", within the
> > > following you say "MUST". Is should x MUST = should?
> > >
> > > >      - The exporter process SHOULD be able to communicate
> > with more
> > > >        than one collection process.
> > >
> > > This is already covered as MAY requirement by section 8.3.
> > >
> > > >      - In case of failure the collection process, or in the
> > > >        connectivity between the exporter process and one
> > collection
> > > >        process, transmission of data SHOULD be redirected, in a
> > > >        fail-over fashion, to a secondary collection process.
> > >
> > > This is already covered by the last paragraph of section 6.3.2
> > >
> > > >      - In case of failover, data that has not been acknowledged
> > > >        SHOULD be retransmitted to the secondary
> > collection process.
> > >
> > > Isn't this a solution rather than a requirement?
> > >
> > > >      - The collection process MUST be able to eliminate
> duplicate
> > > >        reporting of the same records emanating from the
> fail over
> > > >        mechanism."
> > >
> > > This is already covered as MAY requirement by section 8.3.
> > >
> > > >
> > > > </tal>
> > > >
> > > > <snip>
> > > > 5. Add new paragraph after second paragraph in section "6.3.2":
> > > >
> > > >    "The chosen method of data transfer MUST be extensible
> > concerning
> > > >     reliability. It MUST be possible to increase reliability by
> > > >     extensions."
> > > > </snip>
> > > >
> > > > <tal>
> > > > Regarding item 5, I agree that a paragraph should be added
> > > after the second
> > > > paragraph in 6.3.2. The main point on which I differ is
> > > that your proposed
> > > > wording implies that there must be an "ability to ... by
> > > extensions" whereas
> > > > I don't believe that is mandatory to meet the essence of
> > > the requirements.
> > > > If a protocol supports adequate levels of reliability, as
> > > several of the
> > > > currently evaluated protocols do, there may not be a need
> > > for an extension
> > >
> > > Weren't among the people teaching us that there cannot be
> > sufficient
> > > reliability? What is sufficient depends on the
> requirements of the
> > > collecting process or the application processing the
> > transfered data.
> > >
> > > > model. If, on the other hand, the protocol selected does
> > > not have a reliable
> > > > delivery mechanism, then a well-defined means to extend it
> > > must be present.
> > >
> > > Some people said that without application level ACKs there is no
> > > real reliability. Consequently, all you ask for above is not
> > > reliable and would need to be
> > > extensible.
> > >
> > > The main lesson I learned is that there are several levels of
> > > reliability and that you cannot say something is reliable or not
> > > without specifying
> > > what you mean with "reliable". The only consequence is that
> > > you have to
> > > ask for every level, that it still is open for further
> > > increasing reliability.
> > >
> > > > Therefore, may I suggest a slightly different wording than
> > > the version you
> > > > proposed:
> > > >
> > > >    "If higher levels of reliability are not supported,
> > there must be
> > >
> > > The term "higher levels" is only implicitly defined by your
> > text above
> > > and still ambiguous. How many levels need to be supported
> such that
> > > you need not to be extensible anymore?
> > >
> > > >     a well-defined mechanism to extend the protocol to allow for
> > > >     increasing the reliability to support higher degrees of
> > >
> > > Is "degree" equivqalent to "level"?
> > >
> > > >     reliability."
> > > >
> > > >
> > > >
> > > > I believe that even though the wording above may seem to
> > hint at an
> > > > implementation, in fact, it merely may state some lowest
> > > common denominator
> > > > features recurring in practically all existing reliable
> > > delivery protocols.
> > > >
> > > > Best regards,
> > > >
> > > > Tal
> > > > </tal>
> > > >
> > >
> > >
> > >
> > > --
> > > Help        mailto:[e-mail] and say "help"
> > > in message body
> > > Unsubscribe mailto:[e-mail] and say
> "unsubscribe
> > > ipfix" in message body
> > > Archive     http://ipfix.doit.wisc.edu/archive/
> > >
> >
> > --
> > Help        mailto:[e-mail] and say "help"
> > in message body
> > Unsubscribe mailto:[e-mail] and say "unsubscribe
> > ipfix" in message body
> > Archive     http://ipfix.doit.wisc.edu/archive/
> >
>
>



--
Help        mailto:[e-mail] and say "help" in message
body
Unsubscribe mailto:[e-mail] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/


--
Help        mailto:[e-mail] and say "help" in message body
Unsubscribe mailto:[e-mail] and say
"unsubscribe ipfix" in message body
Archive     http://ipfix.doit.wisc.edu/archive/