We have dozens of projects of various sizes, and a single Client project as an entry point. On login to the entry point Client project we have a startup script that determines which project and gateway to retarget to. For some time now, we have seen OutOfMemory heap space errors when retargeting with large projects. We also tried a traditional load balancer, with the same results. Eventually we just had to ignore our largest projects from the retargeting to keep things stable. We are now seeing issues after retargeting with smaller projects, but that have more memory intensive tasks. All of our projects are set to the maximum client memory, but it almost seems like that setting is not honored after retargeting since we are getting OutOfMemory errors when the task manager is only showing a few hundred MB of memory usage for the client which is well below the 4096 set. All clients are using the bundled JRE.
Does anyone know if the system.util.retarget preserves project settings like Client Memory or has anyone else run into this issue?
It's definitely supposed to, yes. And each client launch should be a fresh start, basically, so this is pretty weird. I'd recommend you engage with our support department for a proper investigation.
What Ignition version are you on? I had an issue where system.util.retarget appeared to be leaking memory on 8.1.5. Support recommended I update to 8.1.13 and the issue went away. I think 8.1.13 was the newest version at the time, so it may have been fixed somewhere between those two versions.
Some anecdata I collected at that time using two minimal projects that only had one window. Each window contained only one button retargeting to each other project:
It was support ticket #36649 if that's relevant to anyone at IA.
All bets are off, then. I know for a fact we've fixed a bunch of memory leaks since then, and I'm not sure that my statement about retargeting obeying memory settings even applies that far back.
An easy check would be system.util.getProperty("javaws.sr.memory.max"). This won't necessarily be accurate, but will tell you what Ignition is trying to set the value to, if anything.
Thank you, thats very helpful. I'm assuming there is no way to set this with scripting?
In looking at the memory trend in my gateway retargeted client I can see the garbage collection happening right at 256MB, which I believe is the defaut limit.
An easier way is to go to the client list on the gateway. Here is the same project, on the same gateway. The first one has the max memory, and was not retargeted from another gateway
The second was retargeted from another gateway
So to me this points to a bug in ignition, closely related to the designer bug... When retargeting to another gateway, the client memory is reset to the 256MB default.
If IA is not going to fix this in 7 any sort of direction you could point me in to set this in scripting would be helpful. We do have a custom scripting module, so if there is an SDK method that would work.
I realize this is getting a bit deep and if you don't want to spend time with this I will call support next week.
You can't reallocate the heap of an already extant Java process.
I don't really know what's going wrong.
Looking through the history, in 7.9.6 we fixed a bug (treated as critical) where retargeting multiple times would "lose" the memory settings of the initial client launch, but memory setting should have been obeyed before and after that.
Oh... are you using the native client launchers, or JNLP? Can you isolate the 'not enough memory' issue to specific client computers? You might need to clear the Java cache on those computers (if you're using JNLP, and if you can confirm it only happens on certain computers).
Yep, native client launcher with the bundled JDK. Happens regardless of machine, and regardless of clearing cache.
I can easily recreate the memory being reset to 256 by having a client on gatewayA retarget to the same or a different project on gatewayB in a client startup script. Both projects are set to 4096 memory. Retargeting to the same gateway, different project also with 4096 works properly.
We upgraded all gateways to 7.9.21 and still see the same issue. I'll reach out to support and try to see if we can figure out a solution with a custom module or something... Would be great if this could be patched but I realize supporting 7 is not a high priority.
