Odd CPU Usage Fixed by Logging onto Server

Windows Server 12
Ignition 7.8.3

  • CPU used by Ignition climbs steadily
  • Memory usage is very stable
  • When CPU usage gets to ~20% Ignition starts to log many PLC comm errors
  • Logging onto the Server (Windows) causes the CPU usage to drop immediately and comm errors to end

In the image below that trends CPU usage, comm errors started at the peak, logging into the server caused the errors to end and the cpu usage to decrease.

Annotation 2021-12-15 083903

Hi Mack, firstly, no need to start a new thread:

Secondly, your image has no reference.

from java.lang.management import ManagementFactory
bean = ManagementFactory.getOperatingSystemMXBean()
print bean.getSystemCpuLoad() * 100.0

Run the above in a gateway timer script, dedicated threading, create a memory tag and write to it. Then you can view from the client.

From your previous thread you have been advised that the best route forward is to upgrade your hardware (server), Ignition Version, and server OS (Windows Server 2012 R2) as all are out of support.

Not just no need, but against the forum rules:

https://forum.inductiveautomation.com/faq

If you don't get an answer on a topic, it is fine to reply to yourself after a few days to bring a bit more attention. Once. Getting an answer that one doesn't like isn't an excuse to post again.

I apologize. My intent was to present the information at hand in a cleaner manner.

If you implement the code I posted you can then trend it within Ignition, an easyChart with X-trace would help more.

If your purchasing manager is dead set on not going to new hardware, a free (other than time) suggestion is rip out Windows and replace with Ubuntu Server 20.04 LTS etc. Supported until 2025, and may cure that Java issue you have that prevents you from moving to 7.9.18 (/ and eventually 8.1.12+).

Or rip out your purchasing manager and replace with someone who understands the role Ignition plays within your facility :slight_smile:

What happens if it turns out to be an Ignition bug? The answer is still going to be “upgrade”. Software on commodity hardware isn’t like machinery. Lifespans in the decades just doesn’t happen.

The Ignition projects have been running with no issues for years. There have been no substantial changes to anything in about 2 years. PLCs and tags have actually been removed from the platform. Up until Friday, 10 Dec, all was well. Starting around 7:30 AM on 13 Dec things started breaking.

Everyone agrees that an upgrade is way past due but silos that exist in global companies make it difficult at best to upgrade. They actually tried to upgrade to 7.9.x over 4 years ago and were not successful due to the software running on most of the client computers. The upgrade worked fine, but none of the clients could access Ignition anymore. I was not personally involved in the upgrade, so I don’t have a good feel for the issue.

My current theory is that one to many applications has been added to the network that is interfering with PLC communication. Right now we know that logging onto the server reduces Ignitions CPU load and heads off any problems. Currently we are logging into the server every 15 minutes or so.

It is a possible a network traffic change triggered the current situation. But I can’t think of any way logging into the server could mitigate any such situation. Which makes me think it is a bug. At the very least, you should upgrade to v7.8.5.

@mcgheeiv Well, I certainly hope the Server OS has had its security patches in the last 2 years, or lives in a tight DMZ if not. Windows updates can break many things.

Server is 2012 R2, gets patches once per month, the last was mid November, and lives in a difficult to touch network.

To me this sounds like an anti-virus situation that thinks it is being very kind to everyone by doing its aggressive scans when there are no interactive sessions active. I’ve definitely seen a lot of Windows systems in the past blasting away, then when I log in everything goes back to normal because now “I’m using it”… :expressionless:

EDIT: Just re-read the details and saw that it is Ignition CPU usage, so my statement above probably doesn’t apply…

We are on 7.8.3, which does use an older version of log4j. Any thoughts about virus protection not being smart enough to know the difference?

The issue started around 7:30 AM on Dec 13, only a few days after the log4j vulnerability was found.

We are using Ignition 7.8.3, which uses log4j, albeit not a version with the issue.

Conditions get worse when viewing the Ignition console

Condition is “fixed” when a user logs onto the computer

If anti-virus is intercepting http traffic at the WIndows TCP/IP stack level, then yes, that would appear to increase Ignition CPU usage.

Another reason to not deploy SCADA software on Windows. (Well, a variant of a reason.)

1 Like

Anti virus was disabled for 80 minutes yesterday. During this time there were no CPU spikes or comm failures.

My top working theory is that AV is interfering with the logging process, 7.8.3 uses an older version of log4j. We notice a faster, steady climb in CPU usage when the Ignition console is open.

I will mark Kevin’s comment as as the solution if/when we are able to verify.

We are exploring options to 7.9.x. Any suggestion as to which minor level we should go to first is greatly appreciated.

OS issues on the computers running Vision clients prevented this upgrade around 5 years ago.

Hi again Mack, good to hear you're getting closer to the root cause.

Please view this thread on advice on going from 7.8.x to 7.9.x and beyond;

I did 7.8.4 to 7.9.12 in one step with no issues, however, all systems vary.

Ignition 7.9.10(+) uses an embedded Java 9 JRE for Vision client.

7.9.10 wasn't available in 2016, from memory it was circa 2018.

2 Likes

My client modified their AV settings and the issue appears to have gone away. @Kevin.Collins, thanks so much for the clue. Thanks to everyone else for all the other advice and recommendations.

2 Likes