Clients losing connection at random

Ignition
1

I have noticed in the past few weeks that sometimes my perspective client pages will become "disconnected" from our databases it seems. Basically, the page will still have scripting functionality (can still navigate views and such), however, anything that is linked to tags or queries seems to show as an error or just not load. This will happen on all my client pages that I have open for a few minutes, then it seems to connect again.

The weirdest thing about this, is I have been looking at the gateway and cannot find any logs that seem relevant or any databases that are faulting. The only thing that is suggestive from the gateway is the high memory use, which unfortunately is very common for our Gateway, most likely due to excessive slow queries or just from the amount of clients open at once. I am curious if anyone has experienced something similar and found a way to mitigate this or if it could be a indicator of something else.

2

My first step is to add historian to the [System] tags. It'll let you go back in time to see what may have happened that coincided with your client issues.

Database, performance, sessions might all be things to keep track of for further troubleshooting.

3

  1. High memory usage could be the issue. If you have lots of clients, and they're all continuously doing expensive IO operations (SQL Queries, blocking tag reads/writes, alarm status queries etc.) (especially in transform scripts), it could overload your gateway. I'd read this post for some refactoring inspiration if needed: Building for Performance - #3 by lrose

  2. This could also be an issue with network connectivity. If the network that your gateway lives on is noisy and congested, your gateway could experience issues. I've seen gateways slow down and behave weird when the network used for the connection to the redundant gateway was heavily congested. I'd suggest running a packet capture on the gateway server and looking for big blocks of TCP retransmissions.

  3. Consider contacting support. They can help you diagnose an issue such as this.

1 Like