Re: [DNSOP] I-D Action: draft-ietf-dnsop-dns-capture-format-03.txt

[Jim Hague <[e-mail]>]

On 04/07/2017 00:22, Richard Gibson wrote:
> I looked over this draft in detail, and found a handful of ambiguous
> points ("Clarifications" and "Potentially Missing Data" below). But more
> importantly, it is very close to defining a format that could replace
> much of my organization's in-house technology. Would you consider some
> generalizations to take it over the finish line ("Extension Fields" and
> "Opt-in Lossyness")? Only the suggestions related to representing time
> and "classtype" items would change the representation of existing data
> in such a way that implementations already supporting the draft
> specification would require changes.
>
> *Clarifications*
> * Items in the "classtype" table (section 7.11) are missing data type
> documentation. Both "type" and "class" should be unsigned numbers.

Thanks. Yes, the type needs adding.

>   * And speaking of 7.11, why are CLASS/TYPE pairs represented as CBOR
> maps instead of more efficient two-item arrays? If it was an intentional
> decision for clarity, then maybe the section 7.7 block preamble
> "earliest-time" field should also be promoted to a map ("time-seconds",
> "time-useconds", "time-pseconds", mirroring Q/R items) for the same
reason.

All the tables in the BlockTables section that are multi-valued are
maps. I did consider making the Class/Type pairs an array, but decided
to go for consistency with the other block tables containing composite
data and make it a map. Yes, two-item arrays would be more
space-efficient, saving two bytes per item. However, in the data we've
observed, the number of entries in the Class/Type table is, as you'd
expect, small, typically 15-20 entries. So we're looking at a saving of
maybe 40 bytes per block. By the time you've run through compression,
that advantage will be further eroded, so I ended up deciding that the
cost of consistency here was worth paying.

We've also considered specifying an implicit Class entry of IN (i.e. if
the Class items isn't present in the map, assume IN), but as, again, the
space saving is negligible prefer to keep the values explicit.

Timestamps, on the other hand, I always regarded as a basic data type,
so naturally a structure. Plus, of course, there's one per
query/response item, so in a block the size savings are in the 10-15k
bytes region, which is rather more significant.

> * In "query-sig" table items (section 7.13) "transport-flags" field, the
> bit corresponding to "trailing bytes" shouldn't be limited to UDP.

Interesting point. We haven't to date observed trailing data over TCP,
but that's not to say that somebody won't try it.

> * In section 7.18, "and an unsigned key" appears to be meaningless and
> should probably be removed.

In most places where we are discussing a map, we've specified the type
of the map key in the text, though I notice we're not 100% consistent
with that.

> *Potentially Missing Data*
> * In "query-sig" table items (section 7.13), "transport-flags" should
> probably be extended to include a TLS bit (cf. RFC 7858).

Agreed. We should also look at indicators for DNS-over-HTTP,
DNS-over-QUIC and any other exotica.

> *Extension Fields*
> Of the many potentially open-ended key-value maps (file preamble, file
> preamble configuration, block preamble, block statistics, query
> signatures, Q/R data), only block statistics allows for
> "implementation-specific fields", and no further guidance is provided. I
> think all maps should allow such fields, with a recommendation that they
> use an implementation-specific prefix to avoid collisions with fields
> added by other implementations or later versions of C-DNS.

You are right that extensibility of the tables is not something we have
considered deeply up to now, and it's definitely something that should
be done. FWIW, my initial inclination is to designate as
implementation-specific all key values above a threshold that allows
plenty of growth space for standardised fields, as long as we can be
sure that generic readers can safely skip over the fields they don't
understand.

This is a topic we need to discuss and flesh out.

 Example use
> cases:
[...]
> * Extend the block preamble (section 7.7) to override file preamble
> fields like "host-id" and "server-addresses", enabling fleet-wide file
> merges.

I don't quite follow why you'd need to put this informational-only stuff
into the block preamble rather than the file preamble/configuration. Can
you expand on that a bit?

> *Opt-in Lossyness*
> The format is generally quite good about allowing for detail without
> requiring it. However, there are some areas where more space savings
> could be had:
> * Communicate aggregation of IP addresses into prefixes (i.e., the
> irrelevance of least-significant bits in ip-address values) with new
> "client-prefix-length-ipv4" and "client-prefix-length-ipv6" and
> "server-prefix-length-ipv4" and "server-prefix-length-ipv6" file
> preamble configuration options.
> * Communicate case-normalizing aggregation of names (e.g., transforming
> "eXaMpLe.com" into "example.com <http://example.com>") with a new
> boolean-valued "name-normalization" file preamble configuration option.

These are items that could be addressed by implementation-specific
fields, though I do see the motivation behind wanting a standardised
representation for interchange.

This raises a question about a tension between the background of C-DNS
to date and the slightly different angle you are coming from. We've been
very much focused on using C-DNS to record traffic in a form where the
packets can be recreated in wire format (i.e. as PCAP). The optional
data items mean that data may be missing from those packets, but the
core query and response will still be present.

So, to take the next item:

> * In "rr" table items (section 7.15), "ttl" should be optional to
> accommodate decrementing in recursive resolver responses.

and (skipping out of order) your final:

> * For truly customizable aggregation, I think all query signature
> (section 7.13) and Q/R (section 7.18) data item fields should be
> optional... but especially Q/R data "client-port" and "transaction-id".

moves the recording to a point where reconstructing wire format means
that the application doing the reconstruction has to not just omit
information not present in C-DNS, but must start generating values to
fill in for the missing items. This feels a bit like a step that needs
discussion; we need to think over the design from your point of view.
Possibly those fields should be optional, but with recommendations for
how to populate them when/if generating PCAP.

> * In Q/R data items (section 7.18) and malformed packet records (section
> 7.20), I'd like "time-useconds" broken out into "time-seconds" and
> optional "time-useconds", both for parity with block-preamble
> "earliest-time" and for space savings in applications that are content
> with second-level resolution.

time-useconds is a time offset, rather than absolute time. Splitting
into seconds and optionally useconds means that people wanting usecond
resolution must pay a size overhead in their collection files as there
will will be an additional field, a byte for each query/response. If
there is a general demand for only having time resolution at the second
level, possibly there should instead be a configuration field indicating
whether the offset is in usec or sec. I see that second offsets would
provide for notable size savings.

I'm interested to hear whether others also have similar use cases to the
above.

We'll be doing a 10 minute slot on C-DNS in Prague, and would welcome
discussion there.
-- 
Jim Hague - [e-mail]          Never trust a computer you can't lift.