Re: OSI-DS 33 and 34 - DUA and DSA Metrics

[Andrew Waugh <[e-mail]>]

I've only had time for a quick glance at these. A couple of points
jumped out at me wrt to DSAs.

1. Conformance to other Gosips

(eg Canadian, Australian...)

2. Distributed vs Centralised DSAs

DSP vs DAP doesn't cover all of the options! It is quite possible
to construct a distributed DIT without DSP; the DSAs can have
knowledge and always return referals.

3. Conformance to Schema

A very important point is the ability to define your own schema.
Points should be given for the ability to define new
	* object classes
	* attribute sets
	* attributes (using existing attribute syntaxes)
	* attribute syntaxes
	* tree structuring rules
	* naming rules
Also you should ask if the abilities claimed require recompilition
of the DSA or are table driven.

5. Performance

Two points: size and depth of the test DSA.

Testing with 5000 entries is way too small. A very interesting test
is to pick a couple of typical searches (I chose a whole subtree
search for a commonName - indexed in default Quipu - and a search for
a telephone - not indexed) and, starting at some small size, just
keep doubling until the DSA won't start. You can measure
	* start up time
	* amount of disk required to store the tree
	* amount of memory required to hold the running program
	* time of the two searches
and see if the increases are linear as the DIT size increases.
(for example, it is interesting to see the sudden increase in 
indexed search time when the running Quipu exceeds the size of core
memory).

You MUST test different depths; broad and flat DITs vs deep and narrow.
The performance of DSAs could vary considerable. (Quipu, for example,
has problems with deep trees - the amount of runtime memory blows
out. On our SparcStation 2 we could run flat trees with 300,000 entries,
but deep trees with only 70,000 entries.)

The performance analysis should indicate whether DAP was used to talk
to the DSA (not some lightweight protocol) and whether the DUA was
on the same host as the DSA.

More points to follow (eventually...)

andrew waugh