I am testing out 8.1.22 and just upgraded from 8.1.17. After upgrading, our log is just being flooded with change notification error (tags.execution.batch.changeNotification). The basic error just says "java.lang.NullPointerException: null' on every entry. I switched to debug and it appears to not even actually change to debug mode. The little icon still has the red error icon even when switching 'tags.execution.batch.changeNotification' to debug as shown in the image. Any suggestions ideas before elevating to support? This is in our dev environment, but really would like to capture the memory leak error fixed in 8.1.22 in our prod environment soon.
We have this issue as well but in 8.1.21, and need to upgrade to 22 for the remote provider memory leak as well
Would it be possible to expand the error message so we can see a stack trace?
Thanks,
Garth
Same here, we just added another backend gateway for a new acquisition and are having to restart the gateway once a week as a band-aid.
cries in JDK 11
1 Like
If this is a test environment, any chance you can add the following line to your data/ignition.conf file under the # Java Additional Parameters heading (updating the number 6 if necessary)?
wrapper.java.additional.6=-XX:-OmitStackTraceInFastThrow
On restart, in theory, it should start providing a stack trace.
Thanks,
Garth
@nminchin or @justin.huggins, we haven't been able to replicate this internally so any additional information you could provide would be beneficial. Ideally adding the item noted above to the ignition.conf file would be great as it should throw a full error. It isn't something that should impact a running system other than a restart of the Gateway is required for it to take effect (hence the reason I brought up the fact it was a test system).
If that can't be done, if you could make a copy of a Gateway backup available and send me information via a direct message on how to access it would be something that we could also work with.
A call to support would also work if you are so inclined. At this point, we have no reports of this issue that I can find and would like to figure out a solution.
Thanks,
Garth
Can do, on it now. Also, coincidentally, I have a 1600s flight recording on startup which I forgot I enabled, if that would help?
I might reduce it to 300s... 1600 might be MASSIVE
I'll take it. It will give us insight into what is running. As we are flying blind with this, not sure if it will help at all but it can't hurt.
1 Like
Here's the new log (seems a lot more useful!):
java.lang.NullPointerException: null
at com.inductiveautomation.ignition.gateway.tags.runtime.nodes.complex.UdtTypeDefinitionNode.getInvalidTypeBrowseValue(UdtTypeDefinitionNode.java:155)
at com.inductiveautomation.ignition.gateway.tags.runtime.nodes.complex.AbstractComplexTagNode.getDisplayValue(AbstractComplexTagNode.java:143)
at com.inductiveautomation.ignition.gateway.tags.runtime.nodes.BasicTagDefinition.getBrowseInfo(BasicTagDefinition.java:374)
at com.inductiveautomation.ignition.gateway.tags.config.BatchStructureChangeOperation.lambda$createTagStructureEvent$0(BatchStructureChangeOperation.java:64)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
at java.base/java.util.LinkedList$LLSpliterator.forEachRemaining(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
at java.base/java.util.stream.ReferencePipeline.forEach(Unknown Source)
at com.inductiveautomation.ignition.gateway.tags.config.BatchStructureChangeOperation.createTagStructureEvent(BatchStructureChangeOperation.java:64)
at com.inductiveautomation.ignition.gateway.tags.config.BatchStructureChangeOperation.execute(BatchStructureChangeOperation.java:85)
at com.inductiveautomation.ignition.gateway.tags.evaluation.BatchContextImpl$OpController.run(BatchContextImpl.java:240)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Awesome! Thanks for getting that for me!
GW backup sent via PM
@nminchin, thank you for the stuff you provided. What is happening is, due to the history of this Gateway, there was a time when the Default tag provider had a PROVIDERID set to 0. Over time, this wasn't something that became expected, so when moving to 8.x the initial migration of Tag Providers didn't set the correct sequence in the config.idb, which causes any new providers added to get the same PROVIDERID as the SYSTEM internal provider.
Currently, your System and default_fixed provider have the same PROVIDERID and during startup, your tags in default_fixed are attempting to load into System which is a managed tag provider and it causing the errors you are seeing.
At this point it is best to work with support as there is some cleanup that would need to occur. We are looking into ways this can be fixed but the earliest it would be in a release would be 8.1.24. It is unclear why this now becoming an issue when it likely was not prior to this upgrade. It should have always been a problem, but we are currently trying to isolate what we changed to make the behavior change.
Thanks,
Garth
1 Like
I am also getting much more data when adding that additional parameter to the conf file.
java.lang.NullPointerException: null
at com.inductiveautomation.ignition.gateway.tags.runtime.nodes.complex.UdtTypeDefinitionNode.getInvalidTypeBrowseValue(UdtTypeDefinitionNode.java:155)
at com.inductiveautomation.ignition.gateway.tags.runtime.nodes.complex.AbstractComplexTagNode.getDisplayValue(AbstractComplexTagNode.java:143)
at com.inductiveautomation.ignition.gateway.tags.runtime.nodes.BasicTagDefinition.getBrowseInfo(BasicTagDefinition.java:374)
at com.inductiveautomation.ignition.gateway.tags.config.BatchStructureChangeOperation.lambda$createTagStructureEvent$0(BatchStructureChangeOperation.java:64)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
at java.base/java.util.LinkedList$LLSpliterator.forEachRemaining(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
at java.base/java.util.stream.ReferencePipeline.forEach(Unknown Source)
at com.inductiveautomation.ignition.gateway.tags.config.BatchStructureChangeOperation.createTagStructureEvent(BatchStructureChangeOperation.java:64)
at com.inductiveautomation.ignition.gateway.tags.config.BatchStructureChangeOperation.execute(BatchStructureChangeOperation.java:85)
at com.inductiveautomation.ignition.gateway.tags.evaluation.BatchContextImpl$OpController.run(BatchContextImpl.java:240)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Are you seeing a Gateway folder in one of your own tag providers?
does not seem to be one anywhere, not that I can find anyway.
Hello @ggross, I have the same issue. I migrated a project from vision 7.7.10 to perspective 8.1.32, could the problem be related to the import of the tags?
@Fabricio_Colazo I would recommend contacting support to investigate the problem. It looks probable that your Gateway is in a state that I noted above.
Garth
1 Like