Re: [arch-d] Centralization or diversity

I see that this topic is fragmenting into different threads, which is fine,
so I'm not sure where I should insert this comment, but when we're talking
about "centralized" and "decentralized", it's worth noting that
"decentralized" fault tolerance can trip over implementation errors that
are common across many devices owned or operated by a centralized entity.
One of the largest SS7 outages in the US happened in 1990, when a one-line
error that was present in a large number of switches caused >50 percent
call failures across the entire ATT network.

So, implementation diversity matters, not just decentralization.

There's a nice write-up at
http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collapse.html, that
matches my recollection of reports at the time.

For me, the money quote was

When the destination switch received the second of the two closely timed
messages while it was still busy with the first (buffer not empty, line 7),
the program should have dropped out of the if clause (line 7), processed
the incoming message, and set up the pointers to the database (line 11).
Instead, because of the break statement in the else clause (line 10), the
program dropped out of the case statement entirely and began doing optional
parameter work which overwrote the data (line 13). Error correction
software detected the overwrite and shut the switch down while it could
reset. Because every switch contained the same software, the resets
cascaded down the network, incapacitating the system.

Best,

Spencer

On Wed, Nov 13, 2019 at 12:57 AM Andrew Campling
<andrew.campling@419.consulting> wrote:

> "Martin Thomson" <*mt@lowentropy.net* <mt@lowentropy.net>> wrote on Tue,
> 05 November 2019 22:58:
>
> The draft specifically calls out the notion of a single point of failure
> being a problem.  But my experience with centralized services is that they
> aren't centralized in the fault tolerance sense.  If I look at the big
> services, that scale is only achieved with careful distributed systems
> design.  Name any modern service of even modest scale and you generally
> find excellent fault tolerance.
>
> I thought that the document made it quite clear that it wasn’t
> specifically referring to a single point of failure in a technical, fault
> tolerance sense.  In fact it made this clear by, for example, also
> highlighting issues such as “administrative or governance system can become
> weak through too much power or imagined power concentratred in one place”.
>
> Finally, I don't like the emphasis on DNS in this document.  It only
> serves to sensationalize.
>
> I thought that the reference to DNS was particularly helpful given one of
> the potential side-effects of the push behind DoH could be to centralise
> what is currently a highly decentralised system.  I agree with the comment
> in section 4 that “where such centralised points are created, they will
> eventually fail, or they will be misused through surveillance or legal
> actions regardless of the best efforts of the Internet community.  The best
> defense to data leak is to avoid creating that data store to begin with”.
>
> In addition, noting the references to RFC 1958 and RFC3935, I believe that
> it would be prudent for RFC8484 to be reviewed accordingly.
>
>
> *Andrew*
>
>
> _______________________________________________
> Architecture-discuss mailing list
> Architecture-discuss@ietf.org
> https://www.ietf.org/mailman/listinfo/architecture-discuss
>