Gateway High CPU usage with Memory maxing out

General Discussion
1

Hi,

running V7.5.4 with Java 7_9, MS Server 2008

After performing a reboot of the gateway we have now found that the gateway server has high CPU usage and the gateway memory slowly increases until maxed out.

All was working perfectly beforehand with latest MS and Java updates.

Under console/threads the TagSubscriptionModel-thread-1 (47) is the culprit using 50%.

Locking Owns monitor java.lang.Object@174b459 Owns monitor java.lang.Object@ec7b4d Owns synchronizerjava.util.concurrent.ThreadPoolExecutor$Worker@3be555 Stack java.util.HashMap.createEntry(Unknown Source)java.util.HashMap.addEntry(Unknown Source)java.util.HashMap.put(Unknown Source)java.util.HashSet.add(Unknown Source)java.util.AbstractCollection.addAll(Unknown Source)java.util.HashSet.(Unknown Source)com.inductiveautomation.ignition.gateway.sqltags.scanclasses.BasicScanClassInstance.removeTags(BasicScanClassInstance.java:186)com.inductiveautomation.ignition.gateway.sqltags.scanclasses.SimpleExecutableScanClass.removeTags(SimpleExecutableScanClass.java:358)com.inductiveautomation.ignition.gateway.sqltags.tagproviders.internal.execution.LeasedExecutableScanClass$LeasedSC.removeTags(LeasedExecutableScanClass.java:556)com.inductiveautomation.ignition.gateway.sqltags.tagproviders.internal.execution.LeasedExecutableScanClass.setLeasedTags(LeasedExecutableScanClass.java:305)com.inductiveautomation.ignition.gateway.sqltags.tagproviders.internal.InternalSQLTagsProvider.evaluateScanClassLeases(InternalSQLTagsProvider.java:154)com.inductiveautomation.ignition.gateway.sqltags.providers.AbstractStoreBasedTagProvider.subscriptionChanged(AbstractStoreBasedTagProvider.java:1976)com.inductiveautomation.ignition.gateway.sqltags.model.TagSubscriptionModel$ChangeNotificationJob.run(TagSubscriptionModel.java:243)com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$ThrowableCatchingRunnable.run(BasicExecutionEngine.java:526)java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)java.util.concurrent.FutureTask.run(Unknown Source)java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)java.lang.Thread.run(Unknown Source)

Once the Heap memory usage has maxed out then the theads CPU usage drops but java still maintains the 50%+ CPU usage.

Even with All devices disabled, All projects disabled, All Transaction groups disabled. Same result.

Can anyone shed any light on this?

Last time a simular issue happened I had to create a fresh install and restore a backup from that install as though something in the servers ignition gateway had become corrupted…

EDIT: I have restored an earlier backup of the gateway and everything seems to be working again as normal… I’m still after so ideas as to why this may happen?

Cheers

Tim…

2

Hi,

This is related to a problem with leased scan classes. In the case that we used to track it down, it had to do with tags referenced in alert messages, and how they affected the leased/unleased state. It may be triggered by other situations as well. It was fixed in 7.5.5, so you can try installing the latest beta. If you don’t want to do that, the temporary work around is to change your scan classes back to “direct” instead of leased (which it sounds like you might have done).

Regards,

3

Many Thanks Colby,

Management will be happy to get an answer.

Cheers