We seem to have a sizable memory leak in our production version. It is admittedly an old version 7.1.8 and I will try to upgrade when I can and if you know of these being explicitly fixed that will only help my case.
Anyway I have a Win2008 X64 server with 4 GB RAM hosting Ignition exclusively, the server is using the 64 bit wrapper with 3584 MB dedicated to max memory and wrapper timeout is the default 2 minutes. Today we started having database connection problems. The status page was indicating usage of 3400-something MB so I figured it was restart time. There were 17 clients connected at the time which is pretty standard load (we sometimes get to 30 or so).
After the restart, Ignition immediately spiked up to 3000 MB and crashed within 5 minutes and then did so a second time. I then started closing clients.
The only think of note I could find in the wrapper log was about 100000 lines of “Client subscription proxy got a tag change event after it was shut down.” This is pretty normal I think based on the fact that we use an external database driver and probably really are sending data after “shut down” since we frequently have subscriptions that we fork to our internal historian and some of those make it to the database for a couple of minutes after screens are closed.
We got this message about 50 times for several minutes before the actual failure.INFO | jvm 1 | 2011/08/04 21:40:08 | WARN [BasicExecutionEngine ] [21:40:08,672]: Tried to unregister non existant unit [sqltags scanclasses [besql_db] default].
The final messages in the log were related to the wrapper terminating the instance:
ERROR | wrapper | 2011/08/04 21:40:42 | JVM appears hung: Timed out waiting for signal from JVM.
ERROR | wrapper | 2011/08/04 21:40:42 | JVM did not exit on request, terminatedSo one question is this a known behavior that has been fixed in 7.2? and should we really be hitting the 3.5 GB limit within minutes of startup? Should I be doing something differently to mitigate this problem? (other than upgrade which I will go to 7.2.6 probably on Monday).
Not sure I want to jump to 7.2.8 since I haven’t had any time with it in testing. I’ve tested up through 7.2.6. If there is a good reason or few differences since them I might be willing to fast track it but I dont want accidentally introduce unknown problems.
On this server, there are no OPC or OPCUA servers and there are only 2 Mysql DB connections. We use a thirdparty database driver for tags which is on another machine. It is a large database with over 485,000 tags and 2000 scan classes. Not all of those are active all of the time though. While I will grant some memory consumption because of that but 3.2 GB is still quite a big number.
Edit: Ok I tried to install 7.2.8 on my test machine and did not get very far. The installer starts then stops. Sometimes it starts the ignition service which I stop prior to running the install. At no point does it replace any files though.
Well, that certainly is a lot of tags, and many more scan classes than we’re used to, but I still don’t think that 3gb makes sense. I’m trying to set up some memory profiling tests to see if I can get any kind of clue, but from what I can tell so far it probaby shouldn’t take more than 1gb.
This number will get much better in 7.3, where we’ve reduced the size of tags quite a bit. Additionally, we can likely improve your situation further by making the system a bit smarter about how scan classes are managed. For example, I believe it’s creating, scheduling, and executing a scan class object for each of your defined scan classes when that’s not really necessary- it should just track which tags are in each, and whether or not they’re stale.
Anyhow, I’m in the middle of trying to get more precise information, so I’ll let you know what I find.
Regards,
Ive noticed what I think is a memory leak on 7.2.5. On a restart, the server usually runs at about 250 mb of memory. it keeps gradually rising. I first noticed that the memory was running around 937 mb about a week ago. I restarted the server on Tuesday, and since then it has risen to 500 mb.
Has anything changed memory wise between 7.2.5 and 7.2.8?
also, I am not using the controllogix opc-ua server.
I’ve actually finished some of the rest of the original post through a support call. Basically, 7.2.8 does not address the issue and is as bad as 7.1.8 for us but at least I get some improvements in the designer as a result. We did increase memory by a couple of GB to get us through the next couple of weeks but are at our limits of the machine.
What I see with some analysis of the Ignition engine is that there is significant number of strings held in memory. For 400000 tags, there are 8,000,000 char arrays and 8,000,000 string objects held open which consume 400 MB and 300 MB respectively. There are also a large number of object and HashMap entries held open. See the attached screenshot for a quick memory dump of the process.
I tried to do leak analysis since I’m sure there is a problem there. I suspect it is related to what happens when configchange is updated in an external database and the data is reloaded from the database. Unfortunately I cannot get any version of jvisualvm to work reliably with ignition to make memory snapshots for comparison.
Yes, a big part of what we’ve managed to do with 7.3 is to reduce string allocations. For example, by “interning” key strings such as the scan class name, we can reduce it from an allocation per tag to a single string. I’m sorry we haven’t been a bit more “interactive” helping you with this, but with such a focus on 7.3, we haven’t had much time. However, I’m hopeful that we can get your type of situation to be really efficient, since these external tags don’t really need most of the stuff that other tags have.
Regards,
