503: Service Unavailable upon Client Update

Ignition
1

Hello all,

I have recently moved from IGN 7.9 to IGN 8.1, and we have recently started to notice that not all of our clients (around 500+) are being updated upon an update push. It's about 50-50 whether the client gets updated or not. The work-around currently is just to restart the client, but not everyone will know to do so if they are not paying attention. Every client is receiving the same error:

INFO com.inductiveautomation.factorypmi.application.runtime.ClientPanel - Received project change notification: notificationMode=PUSH, updateMessage=com.inductiveautomation.ignition.common.gateway.messages.ProjectUpdate@6e153447
com.inductiveautomation.ignition.client.launch.steps.HttpResponseException: 503: Service Unavailable
at com.inductiveautomation.factorypmi.application.runtime.AbstractProjectStep.downloadDiffs(AbstractProjectStep.java:95)
at com.inductiveautomation.factorypmi.application.runtime.AbstractProjectStep.downloadDiffs(AbstractProjectStep.java:42)
at com.inductiveautomation.factorypmi.application.runtime.UpdateProjectPane$DownloadProject.run(UpdateProjectPane.java:154)
at com.inductiveautomation.ignition.client.launch.AbstractStepRunner.run(AbstractStepRunner.java:33)
at java.base/java.lang.Thread.run(Unknown Source)
10:24:04.838 [AWT-EventQueue-0] ERROR com.inductiveautomation.ignition.client.util.gui.ErrorUtil - null
com.inductiveautomation.ignition.client.launch.steps.HttpResponseException: 503: Service Unavailable
at com.inductiveautomation.factorypmi.application.runtime.AbstractProjectStep.downloadDiffs(AbstractProjectStep.java:95)
at com.inductiveautomation.factorypmi.application.runtime.AbstractProjectStep.downloadDiffs(AbstractProjectStep.java:42)
at com.inductiveautomation.factorypmi.application.runtime.UpdateProjectPane$DownloadProject.run(UpdateProjectPane.java:154)
at com.inductiveautomation.ignition.client.launch.AbstractStepRunner.run(AbstractStepRunner.java:33)
at java.base/java.lang.Thread.run(Unknown Source)

I am unsure on where to start troubleshooting this issue; any help would be appreciated!

2

Check your gateway logs at the same timestamp; there’s an HTTP error happening (but I suspect it’s not coming from Ignition itself, but something in between).
Are there any firewalls, analyzers, or other network middlemen between your clients and the gateway? Is there anything in common between the failing locations (network infrastructure wise)?

3

There is a firewall between the clients and the gateway, but I am not seeing any HTTP errors in the gateway logs for ± 10 minutes of that timeframe. I actually don’t see anything out of the ordinary other than one of our projects has a script running about 200+ times (have not looked into this yet), but it’s not for the main project that is being updated. We have a development server as well with significantly less clients open at a time (~10-20), and these clients NEVER get the HTTP error. I have also watched the RAM/CPU usage when updating the project, and it never spikes above 20-25% usage on both sides.

Is it possible there are too many clients trying to download the same update at once which is causing the HTTP traffic to jam and then maybe the firewall is closing these connections? It’s just strange that this wasn’t happening until moving to IGN 8.1.

4

Maybe one of the routes is hitting the concurrency limit?

I’d expect the error message to reflect that, though…

1 Like
5

You could try setting ignition.routes.defaultConcurrency parameter to something higher than 5; this would be a wrapper.java.additional parameter in the ignition.conf file: Gateway Configuration File Reference - Ignition User Manual 8.1 - Ignition Documentation
This will require a gateway restart to take effect.

Which particular Ignition 8.X version is it?

6

Currently using 8.1.15. We plan on updating our server (if not this weekend, then next weekend), and I will attempt to change the defaultConcurrency when we do this. I will update this post with the results soon; thanks for your assistance!