Hi,
whenever we run queries, the cpu and memory goes up and eventually both cpu and memory percentage goes down after the query is finished on the Ignition Status Page.
Only after I run a query, the memory usage which we see in the task manager for Zulu Platform goes to its maximum allocated memory (16GB) and stays at 16GB. This memory comes down only if the gateway is restarted.
In the Ignition Status Page once the queries are complete, the memory starts going up again and continues to a certain point 10GB to 11GB; drops all the way to around 2 or 3Gb and starts going up again. This is without any queries being run again.
Total Memory available = 63.8 GB
MySql Memory Allocation = 26 GB
Ignition Memory Allocation = 16 GB
Currently I have Ignition and MySQL on the same machine.
Could anyone please point me in the right direction or give me any ideas on what is causing this and how to solve this?
As Kevin says, absolutely normal. However, you should set Ignition’s starting memory and max memory to the same, so Ignition claims it outright. That prevents a later process hogging memory (databases!) from hurting Ignition’s performance.
So When you say it is normal, the memory goes up and ClockDriftDetector comes into play which affects the performance of the perspective sessions and the desginer.
Possibly more cores. You have enough RAM. Before that, consider setting min and max memory to ~5GB, since it bottoms out at 2-3GB. In my experience, giving java too much memory can make GC longer than necessary. A sawtooth pattern that bottoms ~50% and peaks ~80% seems to be ideal.
But Ignition does run best if it has plenty of cores.
Hi,
My saw tooth pattern goes from 2.4GB to 11GB. If I set around 5Gb does it not affect the performance of the sessions?
I tried setting it to 6GB and on the top it changes to a flat line.
Ok, then 5GB would be too low. The idea is that garbage collecting 8.6GB (11GB => 2.4GB) takes time. If the max limit is lower, GC is going to happen earlier and more often, with a smaller amount collected. Hopefully faster. Java mostly garbage collects just the easiest stuff until memory pressure makes it collect more difficult stuff. (That's the peak of the sawtooth.)
If you can't get the a max below 11GB to yield a faster, smaller sawtooth (with shorter pause the world), then you will need better hardware. Especially more cores.
My issue is a little different. I have ignition installed but not in use, nor is a SQL database even set up yet. Is this program needed or can I delete it?
Thank you for your input. This is issue happens randomly. The only way I can solve it, is by just restarting my computer and then it loads right up like normal.
Hello, I've the same problem, the Gateway is not returning the memory to the OS and continues getting more and more until collapses. Reading the explanations in this topic for the sawteeth there is something I don't understand. Regarding to the memory trend, we can see peaks and bottoms with 9 GB of difference between them. My question is, that assuming that if we are in a bottom 9GB of the heap have been removed, afterwards at the peak, from where has been taken another 9GB of memory ? If the memory is not returned to the OS as explained, why don't we just see a flat line at 11GB ?
Java does not return memory to the OS. Ever. That is how it is designed. You configure the RAM Ignition is allowed to use for its heap in ignition.conf. In production, the initial value and maximum value should be the same so that Java gets its allotment immediately.
That allotment should be significantly less than the total in the machine in order to leave RAM for the OS to use, especially for I/O buffers. If installing on a laptop for local development, you must leave a lot of RAM for the OS, your development DB, your graphical desktop, and any designer(s) you might run (which also are java, and use gobs of memory).
If your situation is Ignition using all of its allowed memory, while the OS still has plenty available, then adjust ignition.conf to give it more. You might simply be doing more with Ignition that will fit in its allowance.
For more specific advice, share more specific information. Screen shots of Ignition's performance page and of your OS's memory usage would help. How many tags are defined? How many clients are running? What else is running in the same machine as the gateway?
Include a snippet from ignition.conf to show your setup.
I was referencing the exposed case that is very similar to mine. But let's go, this is my particular case detail.
As you can see in the snapshot, my system, on a daily basis normal status, consumes around 1 GB of heap memory with 14 Perspective and 5 Vision Clients and around 12 Devie connections. My solution is designed event-driven, so, no many things are happening without an user or an external system event. You can see in the snapshot the normal trend (Marked as A) where the peaks are 1GB and valleys are 600MB. Then (marked as B in the snaphot) I open a new Perspective session showing a view designed for diagnostic purposes that retrieves a message log from the database (a lot of records there) and put them in a table in the view. As you can see the memory raises then up to 5GB.
After that, I close the just opened session expecting the memory release, and wait for hours, an days, but the system never returns to the initial situation. You can see sawteeth of 2GB-5GB. Even if I close all the client sessions.
So my suspicion is that the system is not really releasing the allocated memory and this drives the system to collapse sooner or later.
My specific questions attending to your explanations and my conclusions are mainly this:
If the Gateway process is not returning memory to the OS, why is oscillating between 2GB and 5GB? why is taking 3 GB when in normal status uses 1GB as maximum ?
Why the performance trend shows itself as a sawteeth and not as a flat line of 5 GB ?
How is, the not released memory, reused when necessary ? If relly is reused why the system ends collapsing as you can see in the previous snapshot ?
5a) The memory trend shown in the gateway web interface is Ignition's memory--java's heap. Not the OS's memory. Java's garbage collection technology will reclaim the memory of unused objects on a relaxed pace when there's plenty of heap available, and on a more aggressive pace when there's little free heap. The downward slope on your sawtooth is relaxed reclaim. Note that java isn't giving memory back to the OS when it reclaims, it is just marking it for reuse within java.
5b) I would guess (you haven't shared enough) that your diagnostic view starts something in Ignition, either natively or perhaps a long-running thread of your own, that continues grabbing chunks of data at intervals, and then java's GC slowly reclaims it.
5c) Again, guessing, but you likely have a memory leak that starts with that diagnostic view. In my experience, big memory leaks are almost always inappropriate jython programming. Infinite loops that keep threads in memory are the biggest culprit. Sometimes in events, but often in asynchronous background threads that never end. An audit of your code would be necessary. Perhaps also some java flight recordings.
5a) Understood, I agree that the 8GB are reserved for the heap and never returned to the OS.
5b) Notice that the same diagnostic view in Vision instead of Perspective does not make the Gateway hold memory in the heap. I've tested it. Notice also that, the Perspective session has been closed a long time ago, so nothing launched from a closed Perspecive session is expected to continue executing on the Gateway. If this can happen, how could I detect it (a thread dump or something like that) ? It's possible to determine if a thread opened in a session is running even if the session has been closed?
5c) The view is very simple, there is not much programming there, just the code to bind the query result to a table. Also, I created a very simple view, just with the table and the binding query, with no additional scripts there and the behavior was the same. Guessing, I think that after closing the session, the Gateway recycles the memory, but for any reason, something has been kept alive there that gets memory again.
Vision runs all of its scripting and calculations and property bindings in the Vision Client (local to the user). Perspective runs all of its scripting and calculations and property bindings in the Gateway. Perspective only runs its own javascript in the client browser (and javascript provided in 3rd-party add-on modules for Perspective).
Trend traffic for Vision (along with all other DB results) is transferred as datasets, which, as low-level, column-optimized binary objects, tend to be relatively small both in gateway memory and on the wire to the client. Trend traffic for Perspective has to be converted from dataset form into JSON in the gateway to then transfer to the client browser. Ignition's row-oriented JSON for datasets is anything but efficient, both in the gateway and on the wire to the client.
Ergo, Perspective clients load a gateway (and network) much more than Vision clients.
I'm not sure why your Perspective session isn't cleaning up. All of its code and bindings would have to be reviewed. Note that Perspective depends on timeouts to detect when a browser tab has been closed, as browsers aren't very cooperative (from a webserver's viewpoint) when a user close a tab. If you've set your Perspective session timeout very high, you might see stuff like this. You might want to contact support to look more closely.
Thank you Phil for your detailed explanations. I'd be satisfied if at the end of all of this comments, I could have a final opinion about the reported sawteeth. I mean, if after restart, with all clients and devices connected and shownig the landing page, the Gateway is taken less than 1GB from the heap as you can see in the snapshot. Entering in a situation showing sawteeth toggling between 2GB and 5 GB with the same clients showing the landing page after two days is not normal and something is wrong in the code.
Does that situation always start with opening the diagnostics view? If so, then something in the diagnostics view is doing it. It could be your code. It could be an IA bug in some component that you are using. Anyways, you aren't going to get the answer you want from this forum with the level of detail you've provided. I recommend you ask support to help you if you aren't able to audit the code yourself.
