Hi,
I have been having problems with sudden memory growth. I believe that I have traced it to using and processing a large amount of session property memory.
When I start a session, I load entries into a list that contains two serials, and a couple of other attributes and store that list in a session.custom property. As the session goes along, the list gets searched and a single entry updated until all serial numbers have been processed.
Everything was seemingly fine, memory was steady just showing the usual garbage collection sawtooth, when we were loading a couple of thousand serials. We are now processing some 15000 serial jobs, and the memory use starts growing quickly when we process those. It seems like it is still collecting garbage, but not fully, and the memory use starts climbing.
I don’t really know why this is happening, but it does seem correlated with the large jobs.
I have put a memory graph below. It is just showing you the start, once it get up to 4 g it says there, it never comes back down in my experience.
Where is the best place I can allocate memory for a large list in a session? Or may something else be going on?
Thanks
Everything a Perspective session does is in the gateway, so there’s no “different” place to put something. That said, if you are on 8.1.14, I think there’s a known memory leak.
I’m on 8.1.10. Should it normally be ok to store large amounts of data in the perspective session custom properties like that?
I don’t know. I wouldn’t think it a problem, but it probably isn’t very efficient (JSON encodings). Throw more memory at it to see if there’s an inflection point.
I just did that, I was at 4096, set max to 8192. We’ll see what happens. Thanks
At a minimum, make sure these custom properties are marked private.
They should probably not be true session properties, but live in some other more persistent datastore, much as it would be ideal to have session.custom work for this.
That graph doesn’t look like an Ignition graph. Is it an OS graph? If so, that is normal behavior. Java always keeps OS memory once it claims it. Production systems should have min and max memory allowances for the gateway set equal. Use Ignition’s own memory chart to see if you are wasting memory.
It is writing the value of tag [System]Gateway/Performance/Memory Usage once a minute to a database, you are seeing an SSRS report from the database.
Are you saying that wrapper.java.maxmemory and wrapper.java.initmemory should be set to the same value?
There is just this one application running at any one time in a perspective session, so nothing else would be using these custom properties.
Would putting them in a document tag be a better solution? I guess it would have the same json encode/decode needs. Now that I think of it, could the constant JSON encoding/decoding be causing this? I wouldn’t think so, but I’m reaching for whatever here.
Is there a different session-persistent datastore besides tags?
Thanks
This referenced memory leak was actually introduced in 8.1.10, it just wasn't reported and identified until later versions. The use-case here is directly in-line with replication steps, so it's potentially a result of that issue. The fix for the memory leak has been merged into 8.1.16 nightlies and the stable release of 8.1.16 should be in just a couple of weeks.
1 Like
Thanks, I will be watching with bated breath. I am supposed to put this in another facility this month that does 20x the work that the current one does.
You could always spin up a development server with the 8.1.16 nightly build and see if the problem has been resolved there before you decide to adopt 8.1.16 for deployment. This would give you an excellent idea as to whether or not the build/version would be better-suited for your upcoming deployment.
I set them both toe 8192. The server has 16g total.
When I did this my sawtooth got really big, and it has just been idling since I did it, work doesn’t start back up until tomorrow.
Is that what you would expect?
Thanks
Good idea, if I get time I will, but I don’t see much time for that in the near future 
Yes, that sawtooth looks reasonable. The previous max was probably a bit too low.
Would this memory leak bug be bypassed if I used a Document tag to store this instead of a perspective property?
Not necessarily. The memory leak is encountered when a Perspective property changes value. Assuming your new property is bound to the document tag, then the property would update as often as if you’d just written to the property in the first place.