AC 800PEC PP D513 crash
Last night a controller went down during production.
The visualization gets no more live data. The OPCDA connection to the controller is off. A Reconnect via the OPC Server Configuration (Smiley) fails. A download attempt from the Connectivity Server fails. "No Connection with 172.x.x.x".
But RNRP shows both paths as "up". For RNRP, the Network seems to be available. But the Engineer didn't try to ping.
After the reboot by switching off the power; the controller comes up again and a download from Connectivity Server works correctly.
03:15:32 --> Node_down (Reboot)
03:18:06 --> Node_up (First Download)
03:16:50.818--> Empty Controller came up (no flash used)
03:18:23.965--> Download from ConnectvityServer
E 2016-09-08 02:54:04.471 On Unit= SubDataAccess 172.x.x.x ConnErr OPCServer 5500 Conn error to DA subscr controller
E 2016-09-08 03:15:41.195 On Unit= SubAlarmEvent 172.x.x.x ConnErr OPCServer 6500 Conn error to AE subscr controller
I 2016-09-08 03:16:50.818 OPC Data Access: Contact again with controller: 172.x.x.x
E 2016-09-08 03:16:50.818 Off Unit= SubDataAccess 172.x.x.x ConnErr OPCServer 5500 Conn error to DA subscr controller
E 2016-09-08 03:16:54.305 Off Unit= SubAlarmEvent 172.x.x.x ConnErr OPCServer 6500 Conn error to AE subscr controller
W 2016-09-08 03:16:54.313 Unit= _SWAlarmEvent OPCServer 6002 Failed to subscribe on 172.28.80.168
I 2016-09-08 03:18:23.965 Create application instance: R169872093_ApplicationName
All other 17 Controller were without any Problems. So it seems not do be a Server issue.
Did have anyone a similar issue?
I had the same problem as you a fell years ago, what happens is the Control Builder task is killed by some fail, normally related with some communication card. You can see this using PECView, a software tha came with PECTools package and alow you to check all the tasks directly in the controller. This is the reason you can't conncet using MMS but still see the RNRP connection. The only way to return the controller is turning off the power.
To solve this you have to check in yor controller log what fail happens before the shutdown. In my case was the VIP card. New firmware and better connection management helped to reduce to almost nothing the crashs frequency.
First, Thank you for helping.
The Controller wasn't crashing completly. The machine did move, but there was no Communication to the CS.
I'm not really sure, which log you mean. There is the Controller Log in the Attachment.
In mine opinion there is a Last Download around 2:39:37. Next Entry seems to be the reboot with empty FlashFile. 03:18:09 there is the first Download after the reboot.
May be a Problem is this Warning on every downlaod: Unit= _SWFirmware CT0018 1010 Lost event notif to 172.28.80.21:22. (CT0018--> Controller, IP--> CS Server)
I don't know PecView. We are using PECTool 2.3.xxxx. I Have to ask ABB. Is it also possible to restart Task with this tool? On other Controllers we have some issues with crashing SOAP Services.
It seems to be a Problem with the Controler Load.
I looked in the ControlliT where the load was around 85%. I look also directly at PECTool, where the load is much higher (7-10%). So, if I do a download at 95% load, of course, can bad effects occur and crash individual services.
The reason of this difference between PecTool and ControlIT, neither the PEC guys still ControlIT guys can really explain. Either way, the controller needs to be optimized. We will see if the effects disappear.
PECview is now a part of PECTool and will no longer be separately developed.
Thanks again for the help.
Sorry, Pecview was used with version 4.1 and Pectool is the software for version 5. Is quite common to see more load on Pectool than on ControlIT, some ABB guys say is because of some firmware functions that is not computed by ControlBuilder. What is interesting is why your load is above 80%. By defaut we have a option on Pectool where you change the ControlIT tasks to limit the load to 80% what cause a lot of troubles but whithout crash your controller. As I never worked with Hot mill firmware is a good idea to ask to swedish guys about your problem.
The load is that big because of a realy big Application. We have about 65 IO Clusters via Profibus CI854 (4x). Our Supplier works very carefull with the Controller Resources. But most of the resources disappear as soon as the Hardware is connected. In my Meaning the HW-Communication with ProfibusSlaves takes a lot to much Ressources, but ABB can not give a Reason or a Solution for this "issue".
If you have a Solution for a low ressources PB connection, let me know...
So, now you have my full attention. 150Clients, sounds very interesting. Let's compare a few details:
- How many CI854 do you have on your CEX Bus ( we = 4)
- In Attachment you can see our Configuration of CI854, any differences?
- In Attachment you can see our Configuration of PEC3, any differences?
- In Attachment you can see our Taskload. are you much slower?
Do you know how important the addresses are? Example. One Line has 15 Slaves between Address 2 to Address 73. "Highest station address (HSA) in Configuration is 126. So there are many Gaps between Addresses and a big Gap between last Address 73 and HSA 126. Does it use CPU Power waiting for these Gaps?
Thank you very much for your answer
I take a look on your pictures, and what I see is your profibus configuration is ok. As you asked, the node number don't afect the cpu load at all. I believe your profibus don't use so many resources because you use almost all of it on your application tasks. In your picture I see 75% of cpu load used to run your programs and is quite normal have an significative change based on your network activity. What can help you to lower a little bit the cpu load is fix your tasks offsets to make the setpoint closer to the actual value. If you have some experience with software optimization, take an close look on softwares running with Fast and NormalFast tasks. We usually create more than one task with the same interval (as Fast_A and Fast_B) to garantee a better execution flux and avoid loops what reduce the cpu load.