Re: [arch-d] Centralization or diversity

[John Day <[e-mail]>]

FWIW, there were also problems with keeping the test suites up-to-date with the errata.  Bugs could be fixed in the standards and incorporated into implementations faster than the people developing the test suites could keep up (or wanted to). Of course the fixes were in the implementations before they were in the standards. And what was worse, when pointed out to them that their test-suited didn’t conform to the spec, they didn’t care. They still insisted they were right.

For some standards, it was possible to develop utilities that did a specific function and used part of a standard to do that function but didn’t need the rest of the standard. But the people running the test suites couldn’t deal with it. If it didn’t implement the entire protocol, it didn’t pass. The utility couldn’t cause any of the rest of the protocol to be generated. It conformed to the standard for everything it did. They were using the standard as it had been intended. 

But the testers were effectively bean-counters and totally stupid.

Conformance fell under the ANSI committee I chaired and I remember writing them (some military base in Arizona whose name I have thankfully forgotten) nasty letters telling them to get their act together or get out. They were doing more harm than good. Wish we had known about the IS-IS issues as well to add to the list.

One needs some way to test implementations before trying them live.  But one has to treat them (and it has to be instilled in the testers) that the purpose is to help:  obey Postel’s rule, find problems, and then determine whether it is the tests, the ’standard’ or the new implementation that is at fault. And above all if it isn’t externally visible, it is none of the tester’s business.  Even if there is ‘formal specification’, one can’t be sure that a bug is in the prose, the formal spec, the tests, or the implementation. They are all suspect. (Often formal specifications are more complex than the code. And we know what that means!)

Years before the standards issues arose, I did a survey of testing methods for our own use. I expected to find that two or three were pretty effective, pick one or two.  What I found were 20 or 25 that were all pretty effective and the advice was pick 2 or 3 and it didn’t really matter which ones as long as they were different. (Not the result I was looking for.)  ;-)

Take care,
John

> On Jan 8, 2020, at 15:58, [e-mail] wrote:
> 
> 
>> Yes, i was thinking of ISO PICS. But i lost track of those ISO standards
>> since the 90th. I guess i would have to see a successfull and still
>> currently used standard and how its PICS do or do not help. Just
>> not alot of those ISO specs left in wide deployments, right.
>> X.500 ? Maybe i can get more insight from the security community.
> 
> 
> You need not go that far.  Remember IS-IS?  Still thriving.  
> 
> And yes, my experience with PICs and test suites comes from that.
> 
> 
>>> What it does do is to encourage people to write ???conformance test suites???.  These then get sold to unsuspecting customers and end-users.  Unfortunately, the quality of such suites is so low that you spend way, way, way more time debugging the test suite and you never find bugs in your implementation.
>> 
>> This is just the worst possible outcome, and i am sure one can learn
>> from those bad experiences. Just think of taking a more organized
>> step towards being able to collect protocol implementation information
>> from the industry. Right now every WG who wants to raise the level of
>> an IETF protocol, e.g.: to(wards) full standard is coming up with a
>> questionaire in an ad-hoc fashion (we're just doing this in Multicast
>> for IGMP/MLD). 
> 
> 
> You can do that if you like, but the one proven, effective method is interoperability testing.
> 
> Running code trumps questionaires, conformance test suites, theory, and mandate.
> 
> 
>> My point was: If the only murphy you have is competitive pressure, it works not
>> it there is only limited competition, such as a total of maybe no
>> more than 3 competing national infrastructures. 
> 
> 
> And countries that stifle competition inflict vendor lock on themselves.
> 
> 
>> See above. The best i think we (IETF) can do is to educate better about
>> this by appropriate documents. Certainly some additional protocol work
>> will result from that. I for once had simple slides 10 years ago how to fixup
>> non-dual plane networks to be dual-plane. And maybe we can come up with
>> questionaires to get better numbers from actual deployments (the
>> PICS discussion).
> 
> 
> The IETF’s job is not education. It does not work as the education is not requested
> and not welcome. The IETF’s job is standards and it needs to focus on that.
> 
> 
>> Figuring out ideas for a complete ecosystem of monetization and
>> deployment between competition and regulation is better left for
>> an ongoing bar-bof with emphasis on bar.
> 
> 
> I concur. Let’s stop fooling ourselves into thinking that anything that the IAB/IETF writes has any impact
> on regulation or economics.
> 
> 
>> Actually one of the interesting conclusions was that dual-plane in
>> many cases is really free but deployments often just don't fully
>> utilize it.
> 
> 
> I’d agree that it’s close to free.  If you accept that you need a dual-plane approach for resiliency, then
> adding a second vendor to the mix is relatively easy. There’s additional cost because of decreased 
> purchasing volume. There’s additional costs in management and operations.
> 
> However, as compared to the costs of an unnecessary outage, it’s still trivial.
> 
> Tony
> 
> _______________________________________________
> Architecture-discuss mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/architecture-discuss