Client freezing in a strange way

Ignition
1

At one of our installations, clients get often stuck.
At first, I thought it was caused by an infinite loop somewhere (in a gui script or binding). But there seems to be no connection to what screen is open, and on top of that, the behaviour is very strange.

When the screen freezes, it’s not visible in CPU usage. And it still seems to react on clicks: I can still log out and back in. I have to do it blindly (know where the buttons are), but it does get registered in the audit log. I can also switch pages, it will not show the new page, but it will subscribe to the new tags.

So it seems like not the entire GUI thread is blocked, but some part of the rendering pipeline stopped working.

As even the diagnostics panel is frozen, I have a very hard time troubleshooting this issue. They apparently used to have it once per week, but it started occuring more and more, and now multiple times per day.

This is on a 7.9.8 installation. It happens on multiple clients, and I alsow tried with OpenJDK vs Oracle JDK. The issue seems to always be present.

Has anyone ever seen something similar to this on java apps?

1 Like
2

Install a JDK on that client and then next time this happens use the jstack tool to get a thread dump from that running client. Maybe we’ll see something there.

3

Also check your garbage collection settings. IIRC, you should see it in the Perfomance area of the gateway status page.

4

Have you changed the default client memory setting from the 256mb ?

5

Is the client running an antivirus? Recent updates to it?
We saw something similar where the antivirus flagged any file double compressed as dangerous.
And we all know Java does have compressed files inside of compressed files.

6

@Kevin.Herron I’ll try the jstack tool when I can get it to freeze again, thanks for the tip

@JordanCClark, It only shows stats from the server as far as I can see

@jpark yes, it’s already at 1024MB max, which should be fine.

I don’t think it’s memory related. As memory issues typically make the app slow down until it stops completely. In this case, the client is working smoothly, and suddenly doesn’t update the screen anymore.

@MMaynard thanks for the hint. I’ll check with our client. What was the antivirus causing the issues? We’ve never seen antivirus issues causing problems on other projects.

7

That is commonly called by a stuck network operation (like system.* calls that need to get their answer from the gateway) called from an event script or runScript binding.

8

@pturmel, wouldn’t that cause the gui thread to stop completely? I can stil do things that get logged in the gateway. It’s like the client is still reactive, but just doesn’t repaint.

9

Any power tables with lots of scripting? Sounds like an uncaught exception is happening somewhere in a paint method. You could try running the client from the command line, piping the logs to a file; that way you’ll have access to the logs without needing anything from the running client.

1 Like
10

Cisco AMP if I remember correctly.

11

How do you actually do this? When starting from the client launcher, it seems to spawn a new process, and kill the old one. So I get no logs in my console.

12

The client launcher’s own log reports the command line used for that spawned process. Run that yourself in a terminal.

13

Doesn’t seem to work, at least not on a Windows. It can’t find the path to launch it.

Normal launch:

PS C:\Users\sd\Downloads> java -jar '.\clientlauncher(2).exe'
INFO  [ClientLauncher                ] [09:37:06,896]: Using C:\Users\sd\.ignition\clientlauncher-data\clientlauncher.log to store logging info. Check there for all future logging events.
INFO  [SplashFrame                   ] [09:37:08,395]: Local file checksum and server checksum do not match, launchclient.jar needs to be downloaded
INFO  [SplashFrame                   ] [09:37:08,404]: Downloading "http://192.168.0.189:8088/main/system/launchbin/launchclient.jar" (timeout=30)...
INFO  [SplashFrame                   ] [09:37:08,449]: Starting download of 494618 bytes
INFO  [ClientLauncherFrame           ] [09:37:13,822]: Using java executable under C:\Program Files\Java\jre1.8.0_241
INFO  [ClientLauncherFrame           ] [09:37:13,822]: Starting Java with the following parameters:java -classpath C:\Users\sd\.ignition\clientlauncher-data\launchclient.jar -Djavaws.sr.gateway.addr.0=192.168.0.189:8088/main -Djavaws.sr.scope=C -Djavaws.sr.launchts=1599809599130 -Djavaws.sr.main=com.inductiveautomation.factorypmi.application.runtime.ClientLaunchHook -Djavaws.sr.platform.edition= -Xms512M -Xmx1024M -XX:MaxPermSize=128m -Dsun.java2d.d3d=false -Djavaws.sr.project=Remo_3 -Djavaws.sr.screen=0 com.inductiveautomation.ignition.client.launch.BootstrapSwingFS

Launching the given command

PS C:\Users\sd\Downloads> java -classpath C:\Users\sd\.ignition\clientlauncher-data\launchclient.jar -Djavaws.sr.gateway.addr.0=192.168.0.189:8088/main -Djavaws.sr.scope=C -Djavaws.sr.launchts=1599809599130 -Djavaws.sr.main=com.inductiveautomation.factorypmi.application.runtime.ClientLaunchHook -Djavaws.sr.platform.edition= -Xms512M -Xmx1024M -XX:MaxPermSize=128m -Dsun.java2d.d3d=false -Djavaws.sr.project=Remo_3 -Djavaws.sr.screen=0 com.inductiveautomation.ignition.client.launch.BootstrapSwingFS
Error: Could not find or load main class .sr.gateway.addr.0=192.168.0.189:8088.main