Load problem on advant connectivity server after MS Update
Had everyone similar problems? Any idea?
Voted best answer
With AC400 connect it is good practice to reboot the connectivity server and the RTA boards (PU410 if you have) often. We cycle the PU410 at least once a month.
The PU410 driver results in a multitude of WUDFHost processes (one per "communication channel" with the emulated RTA).
Seeing high load on one of the WUDFHost processes may indicate excessive use of that communication channel, as if something is stuck in a loop.
Do you see any System Messages if you open the RTA Board Configuration tool? Remember that you may elect to send messages only to the ONB log which in that case you need to open and view them from the ONB menubar. I recommend selecting To Screen and Log since "no news are good news" = better see those bad news as they are created instead of hiding them in the log. The log does not work if ONB is closed. Keep the RTA Board Configuration tool open for the duration of the check.
I "hope" that you will see (lots) of system messages indicating some flow problem that give secondary effect when data is pushed via WUDFHost to/from the OPC server process. Please return with a sample of the system messages you see (full detail please) and the version of 800xA for Advant Master (including rollups, TCs).
What version of PU410 firmware do you use? The tool below has a Version Info (remember to click Update) that will tell - I believe 126.96.36.199 is the most recent. Suggest upgrading if on old version (you can download 188.8.131.52 from ABB Library using this document number: 3BSE056809R1007)
If there are no system messages, we might need to engage R&D to find out what is going on. Please submit a support case. With the case, extract all events logged by the ManagementTool.exe (on 5.1 its located at C:\Program Files (x86)\ABB Industrial IT\Operate IT\AC 400 Connect\RTADriver\ManagementTool\ManagementTool.exe)
1. Open ManagementTool.exe
2. Click Trace/Debug
3. Click Get trace/dbg button (a popup window with blue text is opened)
4. Repeat step 3 until the text in the popup windows goes "grey" (=no more queued messages to retrieve)
5. Mark all text in the popup window and paste into notepad.exe and save.
6. Close the tool
Submit the text file with the support case - don't worry if its full of nasty output (a lot of harmless messages are still active, but may look "bad" for a person not being aware of it).
Its easy to call the hand of those two, just execute the ListResendStatistics applog operation; if you see large amounts of resending, you are affected and should upgrade to those corrected versions.
There are several known issues related to time synchronization between the controllers, the CS and TTD logging services. Basically if the time synch keeps changing or gets too far out AND you use TTD logs, then the CPU load on the RTA board and Connectivity starts to increase. I encountered this issue with a system that had been working very successfully for several years, but the GPS Time Source was lost and the system began to degrade.
First, check the CPU load in the RTA board, then check the time synch settings and configuration of the system.
I was on site but had to fix the issue quick because the plant is back to full operation now. Time sync was not a problem. A (repair) reinstallation of the Advant connectivity make it even worse, also a reboot of the RTA board. The deinstallation of all the Microsoft patches fix the issuse. That mean that one (or more) patches of the last 14 month generate that problem. Too bad that I had not the time to investigate that more in detail like Stefan suggest.