I have a Vision Client Project that’s experiencing severe performance issues. This typically occurs after both swapping windows and updating tags very rapidly. The windows in question contain a large number of templates, sometimes hundreds in a template repeater. The template parameters are string tag paths to UDT instances. First the client seems to slow, then freeze. If left frozen for several minutes/hours the GUI will eventually flicker, but stay frozen. If I click within the client while it’s frozen, Windows will state that the process isn’t responding and offer to terminate.
- Running the Vision client via the Native Launcher
- Project Client memory is configured for an initial 1024M, max 4096M
- Client CPU utilization is ~27.5% during freeze events
- Client Memory utilization is ~350MB during freeze events
- I also tried supplying jvm-args to the client launcher shortcut for similar Xms and Xms settings, but that had no impact
- There is a running Alarm Horn Client Timer Function that fires system.util.playSoundClip every 3 seconds, which appears multiple times in each stack dump for some reason.
- There are sometimes SwingWorker threads held in a blocking state on TemplateRepeater$TemplateLoader tasks, seems like the smoking gun to me, but no clue why or how to fix.
- Ignition Version - 7.9.11
- Gateway Java Version - 1.8.1_121-b13
- Client Java Version - 1.8.0 u171
- Not using embedded JRE
I took various JStack trace dumps after freeze events. I took multiple after each event, one event on 11/21, one on 11/22 to see how the state might be changing. Big thanks to the brave soul who wants to dive into these!
crash_dump_20191121_1.txt (60.8 KB)
crash_dump_20191121_2.txt (50.5 KB)
crash_dump_20191121_3.txt (48.8 KB)
crash_dump_20191122_1.txt (43.7 KB)
crash_dump_20191122_2.txt (36.0 KB)
crash_dump_20191122_3.txt (39.9 KB)
I would recommend getting in touch with our support department about this issue. They have the adequate resources to look into this and can properly escalate any issues / bugs.
Thanks for volunteering @jcoffman!
What do you mean when you say “crash”?
To me, the only reasonable definition of a crash is that the Vision Client process terminated unexpectedly.
By this definition none of your thread dumps could have been taken after a “crash”, they were taken while the UI was frozen. Viewing the first 3 dumps attached corroborates this: the UI thread is RUNNING window deserialization code.
So what eventually happens? The UI never unfreezes? an actual crash occurs?
That’s correct, the client freezes and eventually windows will offer to terminate. It is not fully crashing, will update post.
You’re probably going to have to do what JJ suggested and talk to support. They’ll probably want a project or window export to try and reproduce this.
It’s not surprising that use of templates and the repeater like this can be slow, but what you’re describing sounds like a memory leak despite your observation otherwise. I can’t think of any reason for the performance to “degrade” like this.
If you ignore the OS’s offer to terminate, does it unfreeze when it’s done?
By multiple times, do you mean a really large amount of times? We have a different issue (with Edge Historian) that will bog down the server with waiting threads multiplying to dozens of pages in the thread viewer. When this happens, the gateway gets slow responding to the point all the tags flash red overlays intermittently in clients.
I filed a support ticket in conjunction with this. All the same info plus a gateway backup.
This is a very high-priority concern with potential to delay our acceptance testing.
I’ve never witnessed the client unfreezing after the event occurs.
I see 12 of these “Direct Clip” threads in the 11-22 dumps, 32 in the 11-21 dumps.
Here’s an example:
"Direct Clip" #22092 daemon prio=6 os_prio=0 tid=0x17e4ec00 nid=0x12bc in Object.wait() [0x1f8bf000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x069efb20> (a java.lang.Object)
at java.lang.Object.wait(Unknown Source)
at com.sun.media.sound.DirectAudioDevice$DirectClip.run(Unknown Source)
- locked <0x069efb20> (a java.lang.Object)
at java.lang.Thread.run(Unknown Source)
I’m playing the alarm horn sound clip in a 3,000 ms, Fixed Rate, Dedicated client timer script:
system.util.playSoundClip(file, 1.0, False)
It’s obviously bogged down with those stacking up waiting, but at that number I’d think it’s more likely a symptom than a cause.
Comment out all your code that plays a sound clip and see if your client still bogs down and crashes.
We were still able to cause a freeze event with this disabled.
What kind of client memory usage are you seeing?