Backup NIS module did not take over when primary module failed
We have ABB Symphony Harmony installed at plant with 5 PCUs. We faced an issue a few days back whereby we received a PCU 8 power fault and Loop error alarms. On the HMI, all the values turned purple. On visiting the panel, we found both primary and secondary NIS modules in stopped state. We started the module 3 NIS successfully, however, module 5 NIS did not start, therefore, we replaced it.
NIS specification: Network Interface Module, Type: INNIS21 (Make: ABB)(Network Interface Module, Infi-Net to Node Communication Interface SPNIS21
After replacement of NIS module, all the errors and faults normalized. In the past, on the same PCU8, we have had instances when secondary NIS did not take over from primary module.
Therefore, your support is needed regarding following questions:
- Why does PCU8 backup NIS module not take over from primary? Any checks recommended?
- Any steps to rectify this issue?
Answers
This question has not yet been answered.
by Alan Rank: 40 on 10/28/2018 1:49:51 AM | Like (0) | Report
Hi,
What were the error codes of the NIS and NPM modules?
When you say all the values turned purple do you mean all of the PCU 8 values (PCU 8 failure) or all of the values (Loop failure)? From the screen shot it looks like only PCU 8 values.
What do you mean by module 3 NIS and module 5 NIS. NIS can only be attached to NPM (module 0 and 1) or ICT/IIL etc (module 0, 1 or 2).
-Alan
by ahmad.hamid Rank: 3345 on 10/28/2018 5:21:48 AM | Like (0) | Report
Hi Alan,
Thanks for your comment.
- From error code, you mean what was the error code on Engineering station?
- Yes, all the values of respective PCU turned purple
- You are right, pardon my error in description. NIS is attached to NPM module in our case (Also clear in picture). Primary NIS failed and backup did not take over.
Awaiting your clarification, so that I can share further information.
Ahmad
by Alan Rank: 40 on 10/28/2018 5:39:52 PM | Like (1) | Report
Hi Ahmad,
Regarding the error code - The NPM and NIS LEDs will provide a code if the module fails. The EWS problem / status reports will give diagnostic data but only if the module is still running or the primary's backup has failed. In a situation such as yours the module LED error code is important for diagnosing the problem.
The NIS is a slave module to the NPM and not directly responsible for redundancy so the answer might lie with what the secondary NPM was doing at the time and what error codes it might have been displaying.
Did the power fault stop the BRC controllers in the rack as well or was it just the NIS/NPMs that failed?
What firmware are the NIS and NPM modules running?
If you manually test the failover of the NIS/NPM from primary to secondary and back again does this work? There is obviously risk that the NIS/NPM might fail and comms would be lost.
You mentioned that you replaced a NIS to get the PCU working properly, can you put this into a test rack and see if it fails again. Maybe that was the original problem.
Regards,
Alan
by ahmad.hamid Rank: 3345 on 11/1/2018 4:27:04 AM | Like (0) | Report
Hi Alan,
I have been working on the questions you asked and below are the updates:
-Failure LEDs were not noted at the time of failure. However, during plant standby conditions, I inserted the failed NIS module and noted the fault LEDs of NIS and NPM. Picture is attached.
- The fault did not stop the BRC controllers. Only NIS/NPM failed and we got the purple tags on HMI as shown in the picture I earlier shared.
- I manually checked the failover of NIS/NPM and it was successful.
- The answer to last point is in first point. I reinstalled the failed card and NIS/NPM stopped again.
Hope that clarifies. I'm ready to share further information if required.
Regards,
Ahmad Hamid
M:- 00923457443599
by ahmad.hamid Rank: 3345 on 11/1/2018 4:43:17 AM | Like (0) | Report
I'm not sure if a picture can be pasted in comments. However, below is the status of LEDs noted during failure:
- INNIS21: LEDs glowing: 1,2,3,4,5
- INNPM12: LEDs glowing: 2,5
Regards,
Ahmad
by RickW Rank: 51 on 11/1/2018 6:32:54 AM | Like (0) | Report
The NIS21/NPM12 manual has the error codes. For your error codes:
NIS21 LED's 1,2,3,4,5 Error code 1F, Expander bus failure, Corrective action = 1. reset NPM, 2. replace NIS21 or NPM module if error recurs.
NPM12 LED's 2,5 Error code 12, NIS21 module not responding, Corrective action = 1. replace NIS, 2. replace NPM.
To me it looks like you have a hard failure of the NIS21.
by ahmad.hamid Rank: 3345 on 11/1/2018 9:34:54 AM | Like (0) | Report
Hi Rick,
Thanks for explaining the error codes. You are right, NIS failed and we replaced it. But the point is why did the backup NIS module not take over? We have had this problem only with this specific PCU. I'm looking for some help regarding figuring out reasons for backup NIS not taking over and remedial actions for future. Any help will be appreciated. Thanks.
by Alan Rank: 40 on 11/1/2018 2:26:54 PM | Like (0) | Report
Hi Ahmad,
Could the backup NIS/NPM have already been in a failed state or as a result of the power failure when the primary also failed. As you mentioned the problem was only a specific PCU, which could point to an intermittent hardware problem. As you also mentioned, you did identify a faulty NIS module.
How do you monitor the backup module status? The N90STA tag does work but is a bit vague in that the detail is buried inside the status bytes. The MODSTAT (fc95) can be used to decode the status byte information for a node by looking at the primary and redundant NPMs (backup status (BKSTS), power status (NSF) and Loop errors (LR1/LR2 gives a far amount of detail on the health of NIS/NPMs. The MODSTAT fc outputs can then be connected to tags which can then provide the operator with a specific response for each condition, such as call maintenance.
Notes:
Byte numbers are 1-16 in the NIS/NPM doco and 0-15 in the function code doco.
For NPMs address both primary and redundant module address for status byte information (0/1) but for examining controller status bytes only address the primary module.
Good luck,
Alan
Add new comment