Dataset drop-outs between PSD Node 41/42 and other PSC, PSD/ESD and F&G nodes
Before performing above, Node 41 DUAP was taken by started at 19:35 and completed at 20:30 (approx.. 1 hour for 3500 KB DUAP). At 20:45hrs Node 42 was isolated and ECONFIG (P2) command performed. Temporary PC49 with counters to count number of times that the Node Receive Dataset valid pins go to zero was translated and ONB translation to Node 42 was very slow. Then Net.11 Node 42 CI547 card was replaced with new unit, but ONB to Node 42 was still slow after replacement.
Then we pulled-out Net.12 Node 42 CI547 card (checked DIP switch settings – OK), but ONB to Node 42 was still slow with only Net.11 connected. Net.11 transceiver temporarily changed, but there was no improvement in On-Line Builder speed.
Net.12 Node 42 was reconnected, Net.11 and Net.12 returned to their original transceivers, despite of them there was no improvement in ONB speed.
At 22:20 Production shutdown initiated in Node 31/32. (Node 41 P1 and de-isolated, Node 42 P4 (off-line) and isolated). Trip was initiated by timeout of Receive Datasets in Node 31/32 [R413101 and R423201]. As per agreement with onshore management, we disconnected CIMIO server from CSA and CSB (cables disconnected at server in 2D-ICP12041). Done in case this server was requesting excessive information from Node 41/42. We have tested ONB to some other nodes – 11, 35, 37 normal, 41 slow 42 down.
We have restarted Node 42 into P1 mode but still isolated (application software now includes new PC49 for counting Receive Dataset Valid pin drop-outs) at 23:52.
What is more, 400C1 Connectivity Server was restarted at 00:50 and then 400C2 Connectivity Server at 01:05. Then we tested ONB to some other nodes – 21, 35, 36, 37, 38, 77, 78 OK; 41 slow, 42 very slow *.
Net.11 Node 42 disconnected, On-Line Builder to Node 42 slower than *.
Net.11 Node 42 reconnected, Net.12 Node 42 disconnected, On-Line Builder slower than *.
At the end of the day, Node 42 was synchronized and de-isolated. Then inhibits were removed and production restart begins.
** Node 41 CPU average load 75%, Node 42 average CPU load 75%.
**On-Line Builder sessions to Node 41 and Node 42 are slow, particularly to Node 42.
**Multiple nodes on the Masterbus are reporting communications problems via the On- Line Builder messages – see documents of yesterday and today.
**Periodically (from 30minutes to 6 hours) other nodes report loss of comms to Node 42 via the system event list – see attached.
- ANPER Channel Load Analysis (~30 minutes)
- ANPER Task Load Analysis (~30 minutes)
- Record all System Messages
- Traverse all nodes, incl RTA boards and check for system messages emitted in conjunction with DS dropouts
The above is non-intrusive and can be done with process online.
Forward the results (preferably in ASCII format and notepad.exe) from the above to regional ABB support center for analysis.
New problem or old ?
Did you change anything in the system ? Eg adding datasets, nodes etc ?
Loosing Dataset comms sounds like an extended symptom of all the other problems you're having - ie intermittant communications failures. My first guess is that this is a hardware issue.
Did you add / change / remove any MB300 components ?
What kind of media is your MB300 running on ?
Have you physically inspected the cables, hubs, taps, plugs etc ?
Have there been any changes in the plant that could affect the MB300 ? eg new sources of noise ?