When CPU usage gets to ~20% Ignition starts to log many PLC comm errors
Logging onto the Server (Windows) causes the CPU usage to drop immediately and comm errors to end
In the image below that trends CPU usage, comm errors started at the peak, logging into the server caused the errors to end and the cpu usage to decrease.
Run the above in a gateway timer script, dedicated threading, create a memory tag and write to it. Then you can view from the client.
From your previous thread you have been advised that the best route forward is to upgrade your hardware (server), Ignition Version, and server OS (Windows Server 2012 R2) as all are out of support.
If you don't get an answer on a topic, it is fine to reply to yourself after a few days to bring a bit more attention. Once. Getting an answer that one doesn't like isn't an excuse to post again.
If you implement the code I posted you can then trend it within Ignition, an easyChart with X-trace would help more.
If your purchasing manager is dead set on not going to new hardware, a free (other than time) suggestion is rip out Windows and replace with Ubuntu Server 20.04 LTS etc. Supported until 2025, and may cure that Java issue you have that prevents you from moving to 7.9.18 (/ and eventually 8.1.12+).
Or rip out your purchasing manager and replace with someone who understands the role Ignition plays within your facility
What happens if it turns out to be an Ignition bug? The answer is still going to be “upgrade”. Software on commodity hardware isn’t like machinery. Lifespans in the decades just doesn’t happen.
The Ignition projects have been running with no issues for years. There have been no substantial changes to anything in about 2 years. PLCs and tags have actually been removed from the platform. Up until Friday, 10 Dec, all was well. Starting around 7:30 AM on 13 Dec things started breaking.
Everyone agrees that an upgrade is way past due but silos that exist in global companies make it difficult at best to upgrade. They actually tried to upgrade to 7.9.x over 4 years ago and were not successful due to the software running on most of the client computers. The upgrade worked fine, but none of the clients could access Ignition anymore. I was not personally involved in the upgrade, so I don’t have a good feel for the issue.
My current theory is that one to many applications has been added to the network that is interfering with PLC communication. Right now we know that logging onto the server reduces Ignitions CPU load and heads off any problems. Currently we are logging into the server every 15 minutes or so.
It is a possible a network traffic change triggered the current situation. But I can’t think of any way logging into the server could mitigate any such situation. Which makes me think it is a bug. At the very least, you should upgrade to v7.8.5.
@mcgheeiv Well, I certainly hope the Server OS has had its security patches in the last 2 years, or lives in a tight DMZ if not. Windows updates can break many things.
To me this sounds like an anti-virus situation that thinks it is being very kind to everyone by doing its aggressive scans when there are no interactive sessions active. I’ve definitely seen a lot of Windows systems in the past blasting away, then when I log in everything goes back to normal because now “I’m using it”…
EDIT: Just re-read the details and saw that it is Ignition CPU usage, so my statement above probably doesn’t apply…
Anti virus was disabled for 80 minutes yesterday. During this time there were no CPU spikes or comm failures.
My top working theory is that AV is interfering with the logging process, 7.8.3 uses an older version of log4j. We notice a faster, steady climb in CPU usage when the Ignition console is open.
I will mark Kevin’s comment as as the solution if/when we are able to verify.
We are exploring options to 7.9.x. Any suggestion as to which minor level we should go to first is greatly appreciated.
OS issues on the computers running Vision clients prevented this upgrade around 5 years ago.
My client modified their AV settings and the issue appears to have gone away. @Kevin.Collins, thanks so much for the clue. Thanks to everyone else for all the other advice and recommendations.