This high CPU usage on OPC UA reconnect bug was fixed in 7.9.11. It’s probably worth your time to do the upgrade to the latest 7.9 release (7.9.12).
Hello All,
We have been experiencing the exact same problems/behavior too. We run 7.9.10 on a windows server 2016 machine for every site. Same behavior on 7.9.9 too.
One of our integrators recommended much of what was said in this thread. We started with only setting the memory to 2 or 4GB depending the normal consumption and run G1GC.
We had a couple trouble tickets in with Sepasoft, not completely related, and they told us their minimum requirement for memory is actually 8GB even though we had little to no issues at 2 or 4GB until the weird spikes, like you were describing. Now when we have spikes they look like your trends where they try and reach far past what they ever were before. It does make sense that it could do this because we gave it more room to grow. We also let this go for weeks and it got progressively worse until it was affecting everything in the system. We rebooted the gateway and it was like flipping a switch. Everything was completely back to normal only peaking near 2-3 GB again.
We have one side saying to keep memory as low as possible or to only be slightly above normal consumption (what we have been doing for years). The other side is saying crank it up because that is the minimum recommended requirement, even if the system is using nowhere near that amount (converted all sites to this now). The behavior has been the same just more prevalent when we bump resources much higher. What are we supposed to do?
I see the comments about upgrading the Ignition version to 7.9.11, or the latest 7.9.12, because these versions should have fixed this behavior. Sepasoft recommended the same thing, to upgrade to their latest version. I’m not fully on board with always needing to upgrade to the latest version, because it may have fixes for a lot of other things and is most supported. I would rather upgrade when there is proof that the issue I am experiencing is a known bug from the version I am on and a newer release fixes it. Every upgrade requires a fair amount of work and planning outages is not easy for some of the 24/7 operations. We also introduce new bugs from the next release (nature of the software beast) and they may be worse than the one we were hoping to fix. We like to know for a fact that the upgrade is worth it. I would hope that is a fair request but doesn’t always seem to be heard.
I hope this comes through as someone trying to stay in tune with recommendations from the experts but also someone who is explaining the real consequences of doing that.
Thanks for reading, to anyone who made it through this post.
That's known as a "catch-22".
Due to the complexity of software in general, sometimes the proof you need for your specific case is to try the upgrade and see if it is fixed. Be prepared to roll back to the prior version if the new version introduced a regression in your operation.
If you insist on proof for every upgrade, you will need to set up a complete duplicate system that you can use to confirm problems and fixes. For such a long-developing problem, you'd have to license it separately or it wouldn't make a good testbed. If you can't easily confirm the problems with the old version and a test setup, you might also need stand-in devices for the real production devices, so there would be representative OPC traffic. You might want to automate some client sessions, too, for the same reason. Heck, you might just need to duplicate your entire facility for testing. /:
Seriously, though. You have a bug that is suspected to be fixed by a newer version. That's more than enough reason to upgrade. Especially since official support is really only for the latest minor version in any major series.
2 Likes
I appreciate your response and do realize that this is all to uncommon for software in general. Do you know if there is documentation about this issue? Is it specifically in the changelog for 7.9.11? I would like to bring that info back to my team so we can use it as leverage for the next upgrade. We were looking for reasons to make the leap out of 7.9.10 and think this may be enough to get there. Side note, we are not prepared for 8.0 yet. We need more dev time, otherwise that would have been the next leap.
There’s no change log entry for this. You might ask Sepasoft if they have fixed any known memory leaks in their software, though.
Memory leaks are the most difficult class of problem to track down in Ignition. Sometimes it’s a bug in Ignition (there was a problem with the Vision tab strip recently, that I can remember), but more often it’s caused by the project or code built by users of the software instead.
When that happens it’s not really even possible for us to diagnose or fix the problem until it can be reliably reproduced or we get lucky with a memory dump that clearly shows the problem.
4 Likes
Thanks Kevin. For one of our recent issues, that was clearly Sepasoft related, we worked with their support them through the logs. Their only recommendation was to upgrade to their latest module rev with no indication as to the cause. I will keep this in mind during future troubleshooting.
Thanks Kevin, I’ll get to doing that ASAP.