Re: [Dime] Question regarding RFC 6733 on transport end disconnections

[Ivan Skytte Jørgensen <[e-mail]>]

On Thursday 18 October 2018 15:56:52 Miguel Rodriguez Caudevilla wrote:
> 
> We have a cluster wide distributed stack. The diameter instances in the cluster can be administratively be locked, which in consequence ends a transport end connection with a peer.
[...]
> The stack implementers have a different argument. The administrative lock of a single diameter instance, they consider that it is performed in a way that all the services on cluster level are available. So, it would be quite strange to indicate the peer disconnect-case REBOOTING for the loss of a single node considering we can still provide diameter service through all the other nodes in the cluster. So, the loss of a node (and connections handled by it) is a transient fault that is easily handled if the peer reconnects. They consider that this way the peer reconnects as quickly as possible in such cases. That the stack can achieve this according to the standard as well if it does not send any DPR with connection loss reasons.
> 
> A given Diameter instance of the peer state machine MUST NOT use more than one transport connection to communicate with a given peer, unless multiple instances exist on the peer, in which, case a separate connection per process is allowed.
> 
> And consider that a diameter disconnect cause (eg: REBOOTING or whatever else) sent by a diameter node to a peer actually reflects the overall state of the sending node. Since we have a cluster wide distributed implementation it would be not correct to send a disconnect case REBOOTING as that indicates the whole node is going to be rebooted. We just perform some node level operations while still providing full service to diameter peers.
> 
> We would like to have your point of view regarding this discussion.
> Is this intentionally left open for the stack implementation?
> Did we interpret something wrong?

The RFCs can be confusing/convoluted in this area. But re-read section 1.2:

"Diameter Node
      A Diameter node is a host process that implements the Diameter
      protocol and acts as either a client, an agent, or a server."

and combine with section 4.3.1:

"If multiple Diameter nodes run on
the same host, each Diameter node MUST be assigned a unique
DiameterIdentity."

combine with section 5.6.1:

"Upon receipt of a CER, the
   identity of the connecting peer can be uniquely determined from the
   Origin-Host."


Don't be fooled by the "host" part of "origin-host". The payload ia a diameter identity. The AVP was named before anyone thought of these problems.


Then you can deduce that a diameter node is an entity (that being a process, thread, task, fiber, machine, vm, ...) that:
 - has an identity, which is unique and used as the host identity (which is actually a node identity)
 - has a state machine

and therefore:
 - has at most one transport connection per peer.


For simple setups those distinctions aren't that important but when you have a cluster of cooperating nodes with shared state they are.

If you have such a cluster of nodes then you cannot have them all have the same host-identity because identities must be unique and if you somehow manage to get two transport connections to a peer using the same host-identity then the peer must close one of them (as per election process).

What you can do is that each node (vm/process/thread/whatever) has a unique identity:
  * node1.example.com
  * node2.example.com
  * node3.example.com
  * node4.example.com
  ...
and the messages (Eg. CCR/EAP/UD... but not CER/DPR) use a common identity in origin-host.
  * system.example.com

So if node1 sends a eg. CCR to a peer it will have:
  * origin-host: system.example.com
  * route-record: node1.example.com

The peer will simply treat node1 as an intermediate proxy. Nothing special. And your problem with node lock / reboots becomes trivial - just send a DPR with whatever-reason and the peer will only treat it as a message from node1 and only concerning node1. 

The only drawback I can think of is that there might be some non-compliant peers that don't support proxies. But back in 2013/2014 when I tested such cluster setups I didn't encounter any peers that were non-compliant in this area.


Regards,
  Ivan