I have noticed that my Ignition gateway has been running at very high memory. It jumps from about 4 GB to about 14-15 GB. I have my gateway set at 16 GB so it's getting pretty close there. A couple of questions I have regarding this. I do run a lot of scripts on a 60 second timer loop. Tracking averages over a certain times or doing some type of calculations. I also have a lot of the tags being historized using the history feature on the tags. Are any of these the reasons I have so much memory being used? Also I was checking the logs and noticed this error keeps coming up every second which isnt allowing me to see any other errors:
Unable to send updates to client.
java.io.IOException: Channel is not open.
at com.inductiveautomation.perspective.gateway.model.PageModel.send(PageModel.java:381)
at com.inductiveautomation.perspective.gateway.model.PropertySyncManager$SendUpdates.run(PropertySyncManager.java:224)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at com.inductiveautomation.perspective.gateway.threading.BlockingWork$BlockingWorkRunnable.run(BlockingWork.java:58)
Would this have anything to do with it and what is this error and how do I fix it?
And follow up question where do I check to see what is actually taking up all my memory?
Big jumps in memory usage may be because of inefficient scripting. For example using lots of system.net.httpClient will quickly fill up your memory, etc.
This specific error java.io.IOException can be caused by being unable to connect to your endpoint on a http request, so check your comms or connection to whatever client you're sending updates to.
I'm going to guess you have a one-second timer event that is trying to send messages to Perspective. Which will have such failures for any browser tabs that have been closed but not yet timed out on the gateway side. If that is what you are doing, you should probably rework your architecture to pull data into your UI instead of pushing data from gateway scope.
As for memory usage, it jumping up when the gateway is busy is perfectly normal, as long as it settles back down when more idle. Java is constantly allocating and discarding memory (garbage), which is then reclaimed in bunches. Yielding a memory usage sawtooth. When your gateway takes on a particularly big task, with lots of data processed, the sawtooth will spike close to the limit and java will have to reclaim in a hurry to keep providing memory to your algorithm(s).
If not already, consider performing all of the once-a-minute operations in a single scheduled event script, so the various tasks will be sequential. That allows java to reclaim memory from a finished task to support the following task(s).
There is about 11 clients that it is connected to. How would I know which one is causing the error since nothing in the log tells me which one is causing the issue?
I dont think I have any one-second timer events on there. But where would I check that? Also all my once-a-minute operations are in a single script which runs each one one after another.
Ok. Might be polling bindings (perhaps a now() expression--that polls at one second by default) trying to deliver to closed pages. In which case it can be ignored--just a side effect of browser behavior over which Ignition has no control.
That's good. Your once-a-minute memory spikes are probably not a concern.
I usually ignore it but the problem is when I need to troubleshoot a new script or anything else I can't see the logs becuase this error take over and just popups over and over again
Then you'll have to track it down after all. Show the rest of the log entry--if it comes from a UI binding there will be view and component identifiers in the header of the log entry.
If there are none, start investigating your scripts. One by one, or perhaps using MDC.
Or you could have a script somewhere writing to pages/sessions which are no longer active. Do you have any scripts which manipulate sessions/pages, or which use sendMessage()? If you're targeting specific sessions and/or pages and those pages are no longer open I think you would see this.
Yes, if the script is maintaining out-of-date information about sessions. This is entirely dependent on how you've written your scripts. If you're adding session or page ids to registries without cleaning them up, then that would explain the issue.
Don't look for anything in that View. The exception/error/warning is being raised because that View was present somewhere, and then was removed from existence without properly informing the Gateway it was being removed. Was that View truly in a Popup?
The primary cause of this issue is when users have multiple tabs open for a Session and close one of those tabs. In this scenario, the Page is unable to inform the Gateway that it should be de-registered. As a result, bindings and scripts which were active on that page continue to try to update as information on the Gateway updates, but the Page no longer has an active connection (Channel is not open).
That's not the cause of the issue, but it leads to the exact scenario which causes this:
You open a new window/tab/Page for the Session.
A user eventually closes the extra tabs/windows/Pages they don't need. When this occurs, the Page is not removed from the Gateway registry until the SESSION is terminated, at which all content of that session is removed.
So I had all clients closed and I made it so that all pages open up on the same tab instead of a new window. However, the error is still there. Any other thoughts?
Even if all content opens on a single page, there is still a window of time this could naturally occur: the period between the time a user closes that last window when they're done with their session and when the Gateway determines the session is no longer active. By default, a Session will remain active on the Gateway for 60 seconds after a user loses connection (same thing as when the final tab/window is closed). Up until that 60 second timer has elapsed, any scripts for that page which are writing to properties will raise this same exception/warning.
What if all sessions are closed and been closed for an hour but the error is still there? That's basically what I did. I terminated all clients and none came back on for atleast an hour but that error was still there.
Designer sessions which hang around. Designer sessions are notorious for hanging around because there's no way to terminate them and they don't timeout and die; they hang around forever. Even a Gateway restart is not enough to do away with them I don't think.
Proprietary Session registries. Some users have complicated custom project registries which track pages/sessions, but those registries are fragile because they often miss Page cleanups for the same reason you're encountering this.
If neither of these explains what you're seeing, you might need to reach out to Support because this exhausts the explanations I have for why this could occur.