[IPFIX] review: draft-muenz-ipfix-compression-00

[Paul Aitken <[e-mail]>]

Gerhard, Lothar,

Good idea and impressive figures. Please find some feedback inline.


> IP Flow Information Export WG                                   G. Muenz
> Internet-Draft                                   University of Tuebingen
> Expires: January 4, 2009                                        L. Braun
>                                         Technische Universitaet Muenchen
>                                                             July 3, 2008
> 
> 
>       Lossless Compression for IP Flow Information Export (IPFIX)
>                    <draft-muenz-ipfix-compression-00>
> 
> Status of this Memo
> 
>    By submitting this Internet-Draft, each author represents that any
>    applicable patent or other IPR claims of which he or she is aware
>    have been or will be disclosed, and any of which he or she becomes
>    aware will be disclosed, in accordance with Section 6 of BCP 79.
> 
>    Internet-Drafts are working documents of the Internet Engineering
>    Task Force (IETF), its areas, and its working groups.  Note that
>    other groups may also distribute working documents as Internet-
>    Drafts.
> 
>    Internet-Drafts are draft documents valid for a maximum of six months
>    and may be updated, replaced, or obsoleted by other documents at any
>    time.  It is inappropriate to use Internet-Drafts as reference
>    material or to cite them other than as "work in progress."
> 
>    The list of current Internet-Drafts can be accessed at
>    http://www.ietf.org/ietf/1id-abstracts.txt.
> 
>    The list of Internet-Draft Shadow Directories can be accessed at
>    http://www.ietf.org/shadow.html.
> 
>    This Internet-Draft will expire on January 4, 2009.
> 
> Abstract
> 
>    In this document, we discuss the benefits and possible realizations
>    of IPFIX traffic compression.  Experiments with real measurement data
>    show that a significant compression gain can be achieved by applying
>    the DEFLATE compression method to IPFIX data sets.  Compression of
>    IPFIX traffic can be based on underlying transport protocols, such as
>    IPComp and TLS/DTLS, or realized as an extension of the IPFIX
>    protocol.
> 
> 
> 
> 
> 
> 
> 
> Muenz & Braun     draft-muenz-ipfix-compression-00.txt          [Page 1]
> 
> Internet-Draft              IPFIX Compression                  July 2008
> 
> 
> Table of Contents
> 
>    1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  3
> 
>    2.  Terminology  . . . . . . . . . . . . . . . . . . . . . . . . .  3
> 
>    3.  Reduction and Compression of Measurement Data  . . . . . . . .  3
>      3.1.  IPFIX Data Reduction Mechanisms  . . . . . . . . . . . . .  4
>      3.2.  Lossless Compression of Mesurement Data  . . . . . . . . .  4
> 
>    4.  Gain of IPFIX Compression  . . . . . . . . . . . . . . . . . .  5
>      4.1.  University Network Flow Compression  . . . . . . . . . . .  6
>      4.2.  ISP Backbone Flow Compression  . . . . . . . . . . . . . .  7
>      4.3.  Packet Report Compression  . . . . . . . . . . . . . . . .  7
> 
>    5.  Possible Realization of IPFIX Compression  . . . . . . . . . .  8
>      5.1.  Compressed IPsec Tunnel  . . . . . . . . . . . . . . . . .  9
>      5.2.  TLS/DTLS Compression . . . . . . . . . . . . . . . . . . .  9
>      5.3.  Compressed IPFIX Messages  . . . . . . . . . . . . . . . .  9
>      5.4.  Compressed Data Sets . . . . . . . . . . . . . . . . . . . 10
> 
>    6.  Security Considerations  . . . . . . . . . . . . . . . . . . . 10
> 
>    7.  IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 11
> 
>    8.  References . . . . . . . . . . . . . . . . . . . . . . . . . . 11
>      8.1.  Normative References . . . . . . . . . . . . . . . . . . . 11
>      8.2.  Informative References . . . . . . . . . . . . . . . . . . 12
> 
>    Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 13
>    Intellectual Property and Copyright Statements . . . . . . . . . . 15
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Muenz & Braun     draft-muenz-ipfix-compression-00.txt          [Page 2]
> 
> Internet-Draft              IPFIX Compression                  July 2008
> 
> 
> 1.  Introduction
> 
>    Exporting measurement data with low bandwidth utilization was not a
>    primary design goal of the IPFIX protocol.  Indeed, bandwidth
>    efficiency is not mentioned in [RFC3917].  Although the separation of

Even so, IPFIX was based on NetFlow v9 which is arguably more efficient 
that some of the alternatives under consideration.


>    Templates and Data Records and the binary encoding prevents
>    unnecessary inflation of the exported Flow information, the
>    underlying mechanisms were mainly motivated by the objective to
>    simplify the work of the IPFIX Exporter.
> 
>    However, bandwidth efficient export of Flow information is very
>    beneficial in practice.  If it was possible to export the same
>    information with significantly reduced data volume (in terms of
>    number of bytes), we could enjoy the following advantages:
>    o  It would be possible to utilize links with smaller bandwidth
>       between Exporters and Collectors.
>    o  Network congestion could be avoided which reduces the loss ratio
>       if UDP or PR-SCTP are used as transport protocol.
>    o  Less network I/O performance would be required at both, Exporters
>       and Collectors.  Similarly, the sender and receiver buffer sizes
>       of the Exporters and Collectors could be reduced.

These advantages can also be realised in other ways, eg by exporting 
less data by sampling or aggregating.


>    This document discusses different options how the volume of exported
>    Flows can be reduced without information loss.  Particularly, we will
>    show that lossless compression methods enable significant data volume
>    reduction.

Clearly there's a trade-off between data volume reduction and CPU 
resource required for the (de)compression.


>                Section 3 gives an overview on existing approaches and
>    methods to reduce and compress traffic measurement data.
>    Subsequently, Section 4 presents measurement results showing the high
>    compressibility of IPFIX Data Sets containing Flow Records or Packet
>    Records.  Finally, Section 5 describes different ways how compression
>    methods can be applied to the IPFIX protocol.
> 
> 
> 2.  Terminology
> 
>    This document adopts the terminologies used in [RFC5101],
>    [I-D.ietf-ipfix-file], and [I-D.ietf-psamp-protocol].  As in
>    [RFC5101], these specific terms have the first letter of a word
>    capitalized when used in this document.
> 
> 
> 3.  Reduction and Compression of Measurement Data
> 
>    This section summarizes existing approaches of lossless reduction and
>    compression of measurement data.  Section 3.1 starts with a summary
>    of existing IPFIX data reduction mechanisms.  Section 3.2 presents
>    other existing work applying lossless compression methods to
>    measurement data.

"Existing" is a relative term that will be meaningless some years in the 
future. State something absolute, eg "defined in RFCxyz" or at least "at 
the time of writing".


> 
> 
> 
> Muenz & Braun     draft-muenz-ipfix-compression-00.txt          [Page 3]
> 
> Internet-Draft              IPFIX Compression                  July 2008
> 
> 
> 3.1.  IPFIX Data Reduction Mechanisms
> 
>    The IPFIX protocol and its standardized extensions provide three
>    different mechanisms to reduce the amount of exported data without
>    loss of information:
> 
>    Reduced size encoding:  As specified in [RFC5101], integer and float
>       types can be encoded with reduced size if the smaller field size
>       is sufficient to carry the exported values.  The actual encoding
>       size of a field in a Flow Record is defined by the field length in
>       the Template definition.  The application of reduced size encoding
>       requires the definition of an appropriate Template.
> 
>    Data reduction by separation of common properties:  Flows and packets
>       observed in a network often share common properties.  In order to
>       avoid the repeated export of common properties with every Flow or
>       Packet Record, common properties can be exported once for all
>       Flows or packets in a separate Data Record defined by an Option
>       Template.  How this can be realized is described in
>       [I-D.ietf-ipfix-reducing-redundancy].
>       With respect to data reduction, this approach is very limited and
>       difficult to implement.  The definition of common properties
>       requires that Flows or packets share exactly the same values for a
>       given set of Information Elements.  Redundancy among different
>       Information Elements, for example within the same Flow or among
>       different Flows, cannot be reduced.  Redundancy of values which
>       are very similar but not identical cannot be reduced either.
>       Finally, dynamically identifying common properties in a stream of
>       Data Records and estimating the probability of reappearance in the
>       future is difficult.  Therefore, this data reduction approach is
>       mainly useful if the existence of certain common properties in the
>       Flows or packets is known a priori.
> 
>    Export of bidirectional Flows:  The export of bidirectional Flow
>       (Biflow) information as specified in [RFC5153] reduces the number
>       of exported records if bidirectional traffic is observed.  A
>       Biflow Record is larger than a Flow Record because it contains
>       additional fields of Reverse Information Elements, typically
>       carrying measurement data about the reverse Flow direction.  A
>       Biflow Record may also contain the biflowDirection Information
>       Element to indicate the initiator of the Biflow.  In most cases, a
>       small data reduction can be achieved because the Flow Key is
>       exported only once for both directions.
> 
> 3.2.  Lossless Compression of Mesurement Data

"Measurement".


> 
>    We are aware of the following approaches which deployed general
>    purpose compression methods to reduce the amount of measurement data
> 
> 
> 
> Muenz & Braun     draft-muenz-ipfix-compression-00.txt          [Page 4]
> 
> Internet-Draft              IPFIX Compression                  July 2008
> 
> 
>    sent from a traffic monitor to a collector.
> 
>    nFlow:  In 2004, Luca Deri proposed nFlow [nFlow] as an alternative
>       to IPFIX and NetFlow.  One feature was the export of compressed
>       flow records using ZLIB [RFC1950] or GZIP [RFC1952] compressed
>       data format.
> 
>    DiMAPI:  DiMAPI (Distributed Monitoring Application Programming
>       Interface) has been developed in the European IST project LOBSTER
>       [LOBSTER].  It enables the transport of measurement data from
>       distributed traffic sensors to analysis applications running on
>       remote hosts.  In [PMI08], the authors evaluate different
>       compression methods used to reduce the amount of measurement data
>       transferred over the network.
> 
> 
> 4.  Gain of IPFIX Compression
> 
>    In order to circumstantiate the utility of applying compression
>    methods to Flow information, we conducted several experiments using
>    IPFIX Flow Records and PSAMP Packet Records collected in different
>    networks.  Therefore, we chose the well-known compression method
>    DEFLATE which is specified in [RFC1951].  DEFLATE is based on LZ77
>    algorithm and Huffman coding.  It is deployed in various Internet
>    protocols and data formats, e.g.  IMAP [RFC4978], IRIS [RFC4993], SIP
>    and RTSP [RFC3320], and PNG [RFC2083].
> 
>    We applied the DEFLATE [RFC1951] compression method to Data Sets
>    containing different numbers of Data Records and calculated the
>    compression ratios.  Therefore, we determined the compressed size
>    using the ZLIB compressed data format [RFC1950].  ZLIB provides a
>    generic container for compressed data which can be used in
>    combination with different compression methods.  The ZLIB format
>    prepends a header of two bytes specifying compression method and
>    parameters, and appends a checksum of four bytes.  Note that ZLIB is
>    preferred to the alternative GZIP format [RFC1952] since header and
>    trailer are shorter.  We calculated the compression ratios including
>    the ZLIB header and trailer bytes.  In order to get the raw size of
>    the compressed Data Sets, six bytes need to be subtracted from the
>    figures indicated in the tables below, resulting in higher
>    compression ratios.
> 
>    The results presented in the following subsections show that the
>    compression gain is large for Flow Records and much smaller for
>    Packet Records.

Why are they different?


>                    Better compression can be achieved with larger Data
>    Sets.  However, the compressed data volume never exceeds the original
>    size of the Data Sets in our experiments.
> 
> 
> 
> 
> Muenz & Braun     draft-muenz-ipfix-compression-00.txt          [Page 5]
> 
> Internet-Draft              IPFIX Compression                  July 2008
> 
> 
> 4.1.  University Network Flow Compression
> 
>    We applied DEFLATE to Flows measured at the router connecting the
>    class B network of a university to the Internet at a link speed of
>    2.4 Gigabit per second.  Using unsampled Cisco NetFlow, the router
>    generates about 1,800 Flow Records per second in the evening hours
>    (active timeout is 30 minutes).  The Flow Records were re-encoded
>    into IPFIX Data Sets using the Template shown in Table 1 below.  The
>    definition of the Information Elements can be found in [RFC5102].
> 
>              +----------+--------------------------+--------+
>              | Field No | Information Element      | Length |
>              +----------+--------------------------+--------+
>              | 1        | sourceIPv4Address        | 4      |
>              | 2        | destinationIPv4Address   | 4      |
>              | 3        | sourceTransportPort      | 2      |
>              | 4        | destinationTransportPort | 2      |
>              | 5        | protocolIdentifier       | 1      |
>              | 6        | octetDeltaCount (*)      | 4      |
>              | 7        | packetDeltaCount (*)     | 4      |
>              | 8        | flowStartSeconds         | 4      |
>              | 9        | flowEndSeconds           | 4      |
>              +----------+--------------------------+--------+
> 
>                          (*) reduced size encoding
> 
>                           Table 1: Flow Template
> 
>    The total length of one Data Record is 29 bytes.  The Set header adds
>    another four bytes to the Data Set.
> 
>    In every compression run, the same 120,000 Flow Records were written
>    into Data Sets of different length.  The compression was applied to
>    the entire Data Sets (i.e., Set header plus Data Records).  Table 2
>    shows the results.  As can be seen, the Data Set can be compressed to
>    15 percent or less of its original size.  Data Sets with 100 Flow
>    Records could even be compressed to 2.1 percent of the original size.
> 
>    +-----------------+--------------+------------------+---------------+
>    | Number of       | Uncompressed | Compressed size  | Mean          |
>    | Records per     | size         | (mean; min; max) | compression   |
>    | Data Set        |              |                  | ratio         |
>    +-----------------+--------------+------------------+---------------+
>    | 10              | 294          | 43.0; 38; 44     | 6.83          |
>    | 20              | 584          | 45.5; 43; 48     | 12.83         |
>    | 40              | 1164         | 49.2; 45; 51     | 23.65         |
>    | 60              | 1744         | 53.9; 52; 56     | 32.35         |
> 
> 
> 
> 
> Muenz & Braun     draft-muenz-ipfix-compression-00.txt          [Page 6]
> 
> Internet-Draft              IPFIX Compression                  July 2008
> 
> 
>    | 100             | 2904         | 62.7; 61; 64     | 46.31         |
>    +-----------------+--------------+------------------+---------------+
> 
>              Table 2: University Network: Compression Results

Plotting the uncompressed size against the mean compressed size shows a 
fairly linear relationship.

Please also give a column indicating the amount of effort required to 
perform the compression. Does this also scale linearly, or is there a 
sweet point beyond which excessive effort is required for a diminishing 
return?


> 
> 4.2.  ISP Backbone Flow Compression
> 
>    We repeated the experiment of Section 4.1 with Flows measured in the
>    Gigabit backbone network of a regional ISP in Germany.  The observed
>    Flows contain a mixture of web traffic, mail traffic, database
>    queries, VPN tunnels, dial-in traffic etc.  The measurements were
>    performed at a router using unsampled Cisco NetFlow with active and
>    idle flow timeouts set to 150 seconds, resulting in about 300 Flow
>    Records per second.  As before, the Flow Records were converted into
>    IPFIX Data Sets using the Template of Table 1.  Encapsulating again
>    120,000 Flow Records into Data Sets of different length, we achieved
>    the compression results shown in Table 3.  As can be seen, the
>    results are almost identical to those obtained in Section 4.1.

And again, the uncompressed : compressed size is remarkably linear.


> 
>    +-----------------+--------------+------------------+---------------+
>    | Number of       | Uncompressed | Compressed size  | Mean          |
>    | Records per     | size         | (mean; min; max) | compression   |
>    | Data Set        |              |                  | ratio         |
>    +-----------------+--------------+------------------+---------------+
>    | 10              | 294          | 43.1; 39; 46     | 6.82          |
>    | 20              | 584          | 45.6; 41; 48     | 12.80         |
>    | 40              | 1164         | 49.7; 45; 52     | 23.42         |
>    | 60              | 1744         | 54.0; 50; 57     | 32.29         |
>    | 100             | 2904         | 63.4; 59; 66     | 45.80         |
>    +-----------------+--------------+------------------+---------------+
> 
>             Table 3: ISP Backbone Network: Compression Results
> 
> 4.3.  Packet Report Compression
> 
>    As specified in [I-D.ietf-psamp-protocol], the IPFIX protocol can be
>    used to export PSAMP Packet Reports.  In order to evaluate the
>    compression gain that can be achieved with Packet Records, we
>    compressed Data Sets containing IP packet header sections covering
>    the IP and transport layer headers.  The Template definition is shown
>    in Table 4 below.  The definition of the Information Elements can be
>    found in [I-D.ietf-psamp-info].
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Muenz & Braun     draft-muenz-ipfix-compression-00.txt          [Page 7]
> 
> Internet-Draft              IPFIX Compression                  July 2008
> 
> 
>              +----------+------------------------+----------+
>              | Field No | Information Element    | Length   |
>              +----------+------------------------+----------+
>              | 1        | observationTimeSeconds | 4        |
>              | 2        | ipHeaderPacketSection  | variable |
>              +----------+------------------------+----------+
> 
>                       Table 4: Packet Report Template
> 
>    120,000 packets were captured in the network of a research institute.
>    The resulting Packet Records were encapsulated in Data Sets of
>    different length.  Table 5 shows the original and compressed lengths
>    of the Data Sets as well as the compression ratios.  As can be seen,
>    the compression ratio stays below 2 even for a large number of Packet
>    Records per Data Set. Obviously, Packet Records are much less
>    compressible than Flow Records.

I disagree. You're not comparing like with like.

eg, packet records could be exported in the same format as table 1, 
except with packetDeltaCount = 1. Arguably, these should be slightly 
more compressible than the equivalent flow records.


>                                    The huge difference can be explained
>    by the fact that Flow Records usually contain packet header fields
>    where the observed values are very redundant.

Certainly true for the fields chosen for the template in table 1. ie, 
sourceIPv4Address, destinationIPv4Address, sourceTransportPort and 
destinationTransportPort - even protocolIdentifier - may all be expected 
to contain high levels of redundancy across multiple flows.

The experiment only seems to show that ipHeaderPacketSection may contain 
less redundancy than these individual fields do. It doesn't seem to 
prove anything about PSAMP versus IPFIX.


> 
>    +---------------+-------------------+-----------------+-------------+
>    | Number of     | Uncompressed size | Compressed size | Mean        |
>    | Records per   | (mean; min; max)  | (mean; min;     | compression |
>    | Data Set      |                   | max)            | ratio       |
>    +---------------+-------------------+-----------------+-------------+
>    | 10            | 405.5; 302; 574   | 318.3; 172; 437 | 1.27        |
>    | 20            | 807.0; 656; 1144  | 575.0; 341; 744 | 1.40        |
>    | 40            | 1610.1; 1324;     | 1029.9; 615;    | 1.56        |
>    |               | 2260              | 1274            |             |
>    | 60            | 2413.1; 2056;     | 1451.9; 889;    | 1.66        |
>    |               | 3376              | 1771            |             |
>    | 100           | 4019.3; 3512;     | 2275.4; 1769;   | 1.76        |
>    |               | 5004              | 2675            |             |
>    +---------------+-------------------+-----------------+-------------+
> 
>                Table 5: Packet Reports: Compression Results
> 
> 
> 5.  Possible Realization of IPFIX Compression
> 
>    In this section, we discuss different options that allow applying
>    compression methods to IPFIX traffic.  At first, we describe two
>    solutions which are based on compression options provided by IPComp
>    and DTLS.  As third and fourth option, we propose IPFIX-specific
>    solutions which do not depend on the underlying transport protocol,
>    but require extensions of the IPFIX protocol.
> 
> 
> 
> 
> 
> 
> Muenz & Braun     draft-muenz-ipfix-compression-00.txt          [Page 8]
> 
> Internet-Draft              IPFIX Compression                  July 2008
> 
> 
> 5.1.  Compressed IPsec Tunnel
> 
>    IP payload compression (IPComp) [RFC3173] enables the compression of
>    IP datagrams exchanged between a pair of communicating nodes.
>    Therefore, an IPComp association (IPCA) between the two nodes must be
>    established.  DEFLATE [RFC2394] and Lempel-Ziv-Stac (LZS) [RFC2395]
>    are among the standardized compression methods.  IPComp has been
>    conceived for compressing IP datagrams before sending them over an
>    encrypted tunnel.  Dynamic negotiation of IPCAs has been standardized
>    as part of the Internet Key Exchange protocol (IKE) [RFC4306].
> 
>    If IPFIX traffic is transported over an IPsec tunnel, IPComp may be
>    used for compression.  In this case, the compression is transparent
>    to both, Exporting Process and Collecting Process.
> 
> 5.2.  TLS/DTLS Compression
> 
>    TLS [RFC4346] and DTLS [RFC4347] enable the negotiation and
>    deployment of a lossless compression method.  Two compression methods
>    have been standardized for usage with TLS: DEFLATE [RFC3749] and
>    Lempel-Ziv-Stac (LZS) [RFC3943].  For DTLS, no compression methods
>    have been standardized, yet.
> 
>    RFC 3749 [RFC3749] recommends the usage of stateful compression for
>    TLS, which means that a compression history across packets is
>    maintained to achieve higher compression ratios.  DTLS has been
>    conceived for transport protocols that do not guarantee reliable,
>    sequenced packet delivery.  In this case, only stateless per-packet
>    compression is possible.  For example, DEFLATE and LZS could be
>    applied on a per-packet basis.
> 
>    The IPFIX protocol specification [RFC5101] mandates the support of
>    DTLS if SCTP or UDP are used as transport protocol for IPFIX.
>    Similarly, if TCP is used as transport protocol, TLS must be
>    supported.  Although no compression method has been standardized for
>    DTLS, DEFLATE [RFC1951] can be used for stateless compression of
>    individual SCTP messages or UDP datagrams.  The utilization of DTLS
>    or TLS is not mandated by the IPFIX protocol.  If DTLS or TLS are
>    employed, compression methods may be negotiated and deployed between
>    the Exporting Process and the Collecting Process.
> 
> 5.3.  Compressed IPFIX Messages
> 
>    A compressed variant of the IPFIX Message format could be designed as
>    follows.  The IPFIX Message header remains uncompressed and is
>    structured as defined in Section 3.1 of [RFC5101].  The message
>    payload following the IPFIX Message header is compressed using
>    DEFLATE as specified in [RFC1951].

The payload may include templates and options, which you've not 
discussed. Since these may be sent infrequently (compared with data 
records) it may be more efficient _not_ to compress them.

[Later: OK, see section 5.4]


>                                       Optionally, the ZLIB format
> 
> 
> 
> Muenz & Braun     draft-muenz-ipfix-compression-00.txt          [Page 9]
> 
> Internet-Draft              IPFIX Compression                  July 2008
> 
> 
>    [RFC1950] could be used to enable the choice of alternative
>    compression methods and to protect the data integrity by a checksum.
>    In order to distinguish compressed IPFIX Messages from normal IPFIX
>    Messages, a new IPFIX Version Number must be used for compressed
>    IPFIX Messages, for example 0x000b.

Alternatively, the template could be marked as compressed. eg by using a 
new template set ID (eg, Set ID 4 = a set of templates whose data will 
be sent in compressed formats). Or, such templates could contain a 
marker (eg, a new zero-length flag field) indicating that the 
corresponding data records will be sent in compressed format.

This would allow some (even all) data sets to be compressed, while 
allowing templates and options to remain uncompressed.

[Later: OK, see section 5.4]


> 
>    Leaving the IPFIX Message header uncompressed facilitates the work of
>    both, Exporting Processes and Collecting Processes.  The version
>    number and length field are required to decode the message correctly.
>    The Exporting Process does not know in advance how much time the
>    compression will require.  Keeping the export time in the IPFIX
>    Message Header allows the Exporting Process to fill this field after
>    compressing the message payload.

If the EP was feeding a Compressed IPsec Tunnel (per 5.1 above) then the 
export time cannot be filled accurately. Instead, it'll be the time the 
EP handed the data to the tunnel process.

Indeed, if the EP is feeding a regular interface driver, the export time 
will indicate when the EP handed the data to the driver - which may then 
buffer the data pending transmission on the media, and it may be some 
time before the driver is able to transmit. So the export time may often 
be a best-effort.


>                                     Finally, the uncompressed
>    Observation Domain ID allows the Collecting Process to classify IPFIX
>    Messages before decompressing the message payload.

Indeed, the CP may simply want to store compressed records for later 
processing.


> 
> 5.4.  Compressed Data Sets
> 
>    Instead of compressing the entire payload of IPFIX Messages, it is
>    also possible to introduce a new Set type called Deflate Template
>    Set. Just like normal Template Sets, the Deflate Template Set is used
>    to carry Templates, namely Deflate Templates.  The format of Deflate
>    Template Records is the same as the Template Record format specified
>    in Section 3.4.1 of [RFC5101].  However, the corresponding Data Sets
>    differ: starting with the first byte after the Set header, the
>    sequence of Deflate Data Records is compressed using DEFLATE
>    [RFC1951].  Again, the ZLIB format [RFC1950] could be used to enable
>    the choice of alternative compression methods and to protect the data
>    integrity by a checksum.  The Set header remains uncompressed and
>    includes the length of the Set after compression.  In order to
>    distinguish Deflate Template Sets from other Sets, a new Set ID value
>    must be specified (e.g., Set ID = 4).
> 
>    Compression at the level of Data Sets is more flexible than the
>    compression of IPFIX Messages as described in Section 5.3 since it
>    allows exporting compressed and uncompressed Records within a single
>    IPFIX Message.  On the other hand, the size of the compressed data
>    blocks is smaller, which usually results in lower compression ratios.

Some example figures would be interesting.


> 
> 
> 6.  Security Considerations
> 
>    The same security considerations as for the IPFIX protocol [RFC5101]
>    apply.
> 
>    The protocol extensions discussed in Section 5.3 and Section 5.4
>    introduce additional security issues for the Collector.  An attacker
>    may send forged compressed IPFIX Messages or Data Sets to the
> 
> 
> 
> Muenz & Braun     draft-muenz-ipfix-compression-00.txt         [Page 10]
> 
> Internet-Draft              IPFIX Compression                  July 2008
> 
> 
>    Collector which require very large amount of memory after
>    decompression, leading to possible memory exhaustion.  The
>    decompression of such data may also require enormous processing
>    resources, causing a temporary denial of service.  The Collector must
>    implement a protection mechanism which ensures that the decompression
>    is interrupted if it spends large amount of memory or runtime.

However, that could lead to loss of genuine IPFIX data.

P.


> 
> 
> 7.  IANA Considerations
> 
>    The two compression solutions described in Section 5.3 and
>    Section 5.4 require the assignment of a new IPFIX Version Number or
>    IPFIX Set ID respectively.  As specifed in [RFC5101], the
>    standardization of these solutions requires a Standards Action
>    [RFC2434].
> 
> 
> 8.  References
> 
> 8.1.  Normative References
> 
>    [RFC2434]  Narten, T. and H. Alvestrand, "Guidelines for Writing an
>               IANA Considerations Section in RFCs", BCP 26, RFC 2434,
>               October 1998.
> 
>    [RFC5101]  Claise, B., "Specification of the IP Flow Information
>               Export (IPFIX) Protocol for the Exchange of IP Traffic
>               Flow Information", RFC 5101, January 2008.
> 
>    [RFC5102]  Quittek, J., Bryant, S., Claise, B., Aitken, P., and J.
>               Meyer, "Information Model for IP Flow Information Export",
>               RFC 5102, January 2008.
> 
>    [I-D.ietf-ipfix-file]
>               Trammell, B., Boschi, E., Mark, L., Zseby, T., and A.
>               Wagner, "An IPFIX-Based File Format",
>               draft-ietf-ipfix-file-01 (work in progress),
>               February 2008.
> 
>    [I-D.ietf-psamp-protocol]
>               Claise, B., "Packet Sampling (PSAMP) Protocol
>               Specifications", draft-ietf-psamp-protocol-09 (work in
>               progress), December 2007.
> 
>    [I-D.ietf-psamp-info]
>               Dietz, T., Claise, B., Aitken, P., Dressler, F., and G.
>               Carle, "Information Model for Packet Sampling Exports",
>               draft-ietf-psamp-info-08 (work in progress),
> 
> 
> 
> Muenz & Braun     draft-muenz-ipfix-compression-00.txt         [Page 11]
> 
> Internet-Draft              IPFIX Compression                  July 2008
> 
> 
>               February 2008.
> 
> 8.2.  Informative References
> 
>    [I-D.ietf-ipfix-reducing-redundancy]
>               Boschi, E., "Reducing Redundancy in IP Flow Information
>               Export (IPFIX) and Packet  Sampling (PSAMP) Reports",
>               draft-ietf-ipfix-reducing-redundancy-04 (work in
>               progress), May 2007.
> 
>    [RFC5153]  Boschi, E., Mark, L., Quittek, J., Stiemerling, M., and P.
>               Aitken, "IPFIX Implementation Guidelines", RFC 5153,
>               April 2008.
> 
>    [RFC3917]  Quittek, J., Zseby, T., Claise, B., and S. Zander,
>               "Requirements for IP Flow Information Export (IPFIX)",
>               RFC 3917, October 2004.
> 
>    [RFC1950]  Deutsch, L. and J-L. Gailly, "ZLIB Compressed Data Format
>               Specification version 3.3", RFC 1950, May 1996.
> 
>    [RFC1951]  Deutsch, P., "DEFLATE Compressed Data Format Specification
>               version 1.3", RFC 1951, May 1996.
> 
>    [RFC1952]  Deutsch, P., Gailly, J-L., Adler, M., Deutsch, L., and G.
>               Randers-Pehrson, "GZIP file format specification version
>               4.3", RFC 1952, May 1996.
> 
>    [RFC2394]  Pereira, R., "IP Payload Compression Using DEFLATE",
>               RFC 2394, December 1998.
> 
>    [RFC2395]  Friend, R. and R. Monsour, "IP Payload Compression Using
>               LZS", RFC 2395, December 1998.
> 
>    [RFC3173]  Shacham, A., Monsour, B., Pereira, R., and M. Thomas, "IP
>               Payload Compression Protocol (IPComp)", RFC 3173,
>               September 2001.
> 
>    [RFC3749]  Hollenbeck, S., "Transport Layer Security Protocol
>               Compression Methods", RFC 3749, May 2004.
> 
>    [RFC3943]  Friend, R., "Transport Layer Security (TLS) Protocol
>               Compression Using Lempel-Ziv-Stac (LZS)", RFC 3943,
>               November 2004.
> 
>    [RFC4306]  Kaufman, C., "Internet Key Exchange (IKEv2) Protocol",
>               RFC 4306, December 2005.
> 
> 
> 
> 
> Muenz & Braun     draft-muenz-ipfix-compression-00.txt         [Page 12]
> 
> Internet-Draft              IPFIX Compression                  July 2008
> 
> 
>    [RFC4346]  Dierks, T. and E. Rescorla, "The Transport Layer Security
>               (TLS) Protocol Version 1.1", RFC 4346, April 2006.
> 
>    [RFC4347]  Rescorla, E. and N. Modadugu, "Datagram Transport Layer
>               Security", RFC 4347, April 2006.
> 
>    [nFlow]    Deri, L., "nFlow: Monitoring Flows on IPv4/v6 Networks",
>               TERENA Networking Conference 2004, June 2004.
> 
>    [LOBSTER]  "IST LOBSTER Project Homepage",
>               Homepage http://www.ist-lobster.org, 2008.
> 
>    [PMI08]    Politopoulos, P., Markatos, E., and S. Ioannidis,
>               "Evaluation of Compression of Remote Network Monitoring
>               Data Streams", IEEE Workshop on End-to-End Monitoring
>               Techniques and Services E2EMON 2008, April 2008.
> 
>    [RFC4978]  Gulbrandsen, A., "The IMAP COMPRESS Extension", RFC 4978,
>               August 2007.
> 
>    [RFC4993]  Newton, A., "A Lightweight UDP Transfer Protocol for the
>               Internet Registry Information Service", RFC 4993,
>               August 2007.
> 
>    [RFC3320]  Price, R., Bormann, C., Christoffersson, J., Hannu, H.,
>               Liu, Z., and J. Rosenberg, "Signaling Compression
>               (SigComp)", RFC 3320, January 2003.
> 
>    [RFC2083]  Boutell, T., "PNG (Portable Network Graphics)
>               Specification Version 1.0", RFC 2083, March 1997.
> 
> 
> Authors' Addresses
> 
>    Gerhard Muenz
>    University of Tuebingen
>    Computer Networks and Internet
>    Sand 13
>    Tuebingen  D-72076
>    DE
> 
>    Phone: [phone]
>    Email: [e-mail]
>    URI:   http://net.informatik.uni-tuebingen.de/~muenz
> 
> 
> 
> 
> 
> 
> 
> Muenz & Braun     draft-muenz-ipfix-compression-00.txt         [Page 13]
> 
> Internet-Draft              IPFIX Compression                  July 2008
> 
> 
>    Lothar Braun
>    Technische Universitaet Muenchen
>    Network Architectures and Services, Institute for Informatics
>    Boltzmannstrasse 3
>    Garching  D-85748
>    DE
> 
>    Phone: [phone]
>    Email: [e-mail]
>    URI:   http://www.net.in.tum.de/
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Muenz & Braun     draft-muenz-ipfix-compression-00.txt         [Page 14]
> 
> Internet-Draft              IPFIX Compression                  July 2008
> 
> 
> Full Copyright Statement
> 
>    Copyright (C) The IETF Trust (2008).
> 
>    This document is subject to the rights, licenses and restrictions
>    contained in BCP 78, and except as set forth therein, the authors
>    retain all their rights.
> 
>    This document and the information contained herein are provided on an
>    "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
>    OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
>    THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
>    OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
>    THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
>    WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
> 
> 
> Intellectual Property
> 
>    The IETF takes no position regarding the validity or scope of any
>    Intellectual Property Rights or other rights that might be claimed to
>    pertain to the implementation or use of the technology described in
>    this document or the extent to which any license under such rights
>    might or might not be available; nor does it represent that it has
>    made any independent effort to identify any such rights.  Information
>    on the procedures with respect to rights in RFC documents can be
>    found in BCP 78 and BCP 79.
> 
>    Copies of IPR disclosures made to the IETF Secretariat and any
>    assurances of licenses to be made available, or the result of an
>    attempt made to obtain a general license or permission for the use of
>    such proprietary rights by implementers or users of this
>    specification can be obtained from the IETF on-line IPR repository at
>    http://www.ietf.org/ipr.
> 
>    The IETF invites any interested party to bring to its attention any
>    copyrights, patents or patent applications, or other proprietary
>    rights that may cover technology that may be required to implement
>    this standard.  Please address the information to the IETF at
>    [e-mail].
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Muenz & Braun     draft-muenz-ipfix-compression-00.txt         [Page 15]
> 

-- 
Paul Aitken
Cisco Systems Ltd, Edinburgh, Scotland.
_______________________________________________
IPFIX mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/ipfix