Port Disabled, Network Storm detected, No communication with BackupCPU
Currently at one of installed based, there are showing error of the network storm detected at controller CN1 (primary and secondary). The controller port CN1 is not detected in RNRP but can be Ping as per attached picture.
Any suggestion of the cause and solution?
Look for controller logs rows containing "NSP" (Network Storm Protection). Depending on rate of controller log output, old printouts may be overwritten by new.
Primary CPU (log contain data from 2016-10-23 to 2017-02-23):
Backup CPU (log contain data from 2015-01-31 to 2017-01-24):
I 2015-02-25 19:14:08.695 NSP: Blocking Ethernet port CN1 at limit 800 pkt/s
I 2015-02-25 19:14:17.695 NSP: Unblocking Ethernet port CN1
I 2017-01-24 14:57:28.343 NSP: Blocking Ethernet port CN1 at limit 800 pkt/s
I 2017-01-24 14:57:37.343 NSP: Unblocking Ethernet port CN1
The backup CPU has repeated NSP (storm protection) occurrences in 2015 and one occurrence in 2017.
A network storm may cause a controller to shutdown (this is per design). The NSP will block controller ports receiving too many packets per second. The block is removed after some time.
If a storm affect both Ethernet interfaces at the same time, the controller may keep one port closed until controller reset. This bug is corrected in the latest controller firmware, 5.1.1-3 released with 5.1 FP4 RevE. A workaround if one port is permanently closed is to download a dummy RNRP parameter change (e.g. increase max number of remote areas). This will restart RNRP and in most times also open the permanently blocked port. Be sure to make such download from a LOCALLY connected Control Builder M tool (i.e. connected to the same subnetwork/RNRP area as the controller itself).
The problem is very likely in the network, maybe due to a ring structure (or worse, a Spanning Tree structure) where the media redundancy protocol is acting too slow to prevent a loop when e.g. a ring is closed. ABB recommend using flat networks (no rings), or rings where media redundancy protocl is ultra fast acting (faster than one second).
A simple loop is an Ethernet cable connecting both ends to the same switch. This is less likely though.
The "ping" tool is of less use since RNRP will re-route traffic to secondary network if primary is closed due to loop. To see "true" ping results, first stop the ABB RNRP Service in Windows
RNRP Monitor and RNRP log (see RNRP Wizard) may contain addtional data.
Look at the physical installation of the network. There must be some configuration permitting a loop. Find it. Prevent or remove it.