All Q CPU's goes Error

[MrDippy]

Hi all,

We have multiple plant equipment with Q series PLC's. Some running almost trouble free for years. Most of them have QJ71E71-100 talking to the SCADA  HMI's. 

A plant wide Data logging SCADA system also collects data off these PLC's on the network. About 2 weeks ago, almost all the PLC CPU's started going to Error at the same time. We have tried looking for Viruses on the SCADA Hmi computers but found nothing. The QJ71E71 also locks up when the CPU errors.  I have tried almost everything: Check all CPU's has its own Station number.  Make sure there was no IP conflicts.  Tried UDP/IP & TCP/IP protocols.  Changed Initial setting/Destination existance  conformation times. But the Errors keep happening causing the plants to shut down.

There are a couple of older machines with E910 HMIs connect via Ethernet to its local QJ71E71. These HMI network boards would also lock out.

Unplugging the machines off the network stops the CPU errors and other lock-outs so this is clearly a network issue. But I have never come across anything like this before whereas multiple cpu's would error disabling half the factory simply because of the network. Has anyone out there come across this problem? 

 

[Gambit]

You have created a ring ethernet network structure this should be a bus. You've probably created a broadcasting loop.
Did the CPU's go into WDT???

Than this probably is the cause of the problem.

 

Edited December 22, 2016 by Gambit

[kaare_t]

As @Gambit indicates my guess is also that you have a broadcast loop somewhere in your network. This would affect all connected devices on the designated LAN, but limited to the subnet you are connected to. If it is a loop, then it can be hard to find it without managed switches (or access to the cli of the managed switches), but you should check if any "new" connections have been made in the network just before it stopped working.

You should, as @Gambit suggests, connect a computer via the USB/Serial interface and read out the error messages of a couple of the Q CPU's...

[MrDippy]

Thanks for the suggestions guys. I think we were a bit complacent connecting these machines onto the main network and not thinking more about it because it just worked without problems. What I found was the IT guys looking  after our network at our other site, installed  some security port scanner.  After the scanner was uninstalled, no more lock-outs.  This may be coincidental but good lesson learnt here. We will wire in a separate network with managed switches just for the machines.

 

[kaare_t]

Glad to hear, merry Christmas.