# Multiple OPC connections faulting at the same time **URL:** \ **Category:** Ignition\ **Tags:** ignition81\ **Created:** [November 1, 2021, 9:54pm UTC](https://forum.inductiveautomation.com/t/multiple-opc-connections-faulting-at-the-same-time/52592 "2021-11-01T21:54:13Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![scotteolson3](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@scotteolson3](https://forum.inductiveautomation.com/u/scotteolson3)\ **Post date:** [November 1, 2021, 9:54pm UTC](https://forum.inductiveautomation.com/t/multiple-opc-connections-faulting-at-the-same-time/52592/1 "2021-11-01T21:54:13Z")
I am having a persistent issue where my OPC connections all drop at the same time. Currently we have 8 connections, 4 of which will drop simultaneously and have to be manually reconnected (3 are intentionally deactivated and 1 is Ignitions default connection). I had some issues relating to CPU/RAM capacity that I thought was causing the problem as the CPU usage would easily hit 90%+ usage, but after upgrading the cloud instance the problem persists (though less frequently). I have logging for some of the OPC related Logger’s turned to DEBUG and capture events like this: ![Screen Shot 2021-11-01 at 2.46.58 PM](https://us1.discourse-cdn.com/inductiveautomation/original/3X/7/a/7a2f0fbcbec951304ad4c517c949dcd9c7c8fd1e.png) I do have the SessionFsm DEBUG info as well: ![Screen Shot 2021-11-01 at 2.50.04 PM](https://us1.discourse-cdn.com/inductiveautomation/original/3X/c/0/c02eda9b09ba4fb7c77b6ca5d2e131dde1bd623b.png) Which Logger/Loggers should I change the level of to better capture that event that is causing this? Any help in diagnosing the issue would be. It is occurring with both a Siemens device and multiple Beckhoff devices. ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [November 1, 2021, 9:57pm UTC](https://forum.inductiveautomation.com/t/multiple-opc-connections-faulting-at-the-same-time/52592/2 "2021-11-01T21:57:52Z")
The SessionFsm logger is already a good start. It’s pretty clear that the disconnect is because the keep alive requests stop getting responded to. A Wireshark capture would be a good supplement if you can disable security for the connections so we can decode the traffic. What versions of Ignition are you using? ---
**Author:** ![scotteolson3](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@scotteolson3](https://forum.inductiveautomation.com/u/scotteolson3)\ **Post date:** [November 1, 2021, 10:00pm UTC](https://forum.inductiveautomation.com/t/multiple-opc-connections-faulting-at-the-same-time/52592/3 "2021-11-01T22:00:47Z")
Currently running on 8.1.7 I will work on getting a wireshark capture, that is a good idea. The complexity of our system (and potential culprit) is we are running our gateway in the cloud and then connecting our local devices via a site-to-site VPN. ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [November 1, 2021, 10:02pm UTC](https://forum.inductiveautomation.com/t/multiple-opc-connections-faulting-at-the-same-time/52592/4 "2021-11-01T22:02:25Z")
The way you’ve described it the obvious first guess is that you have a network problem, not an Ignition problem. The only 4 OPC UA connections that are talking to other hosts on the network all stop responding at the same time? Yeah… ---
**Author:** ![scotteolson3](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@scotteolson3](https://forum.inductiveautomation.com/u/scotteolson3)\ **Post date:** [November 1, 2021, 10:04pm UTC](https://forum.inductiveautomation.com/t/multiple-opc-connections-faulting-at-the-same-time/52592/5 "2021-11-01T22:04:10Z")
My thinking exactly. There are only 2 things that all 4 connections have in common, the gateway and the network which makes them the obvious suspects. I think the gateway connection is a bit easier for me to diagnose/troubleshoot which is why I am starting here. ---
**Author:** ![scotteolson3](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@scotteolson3](https://forum.inductiveautomation.com/u/scotteolson3)\ **Post date:** [November 1, 2021, 10:38pm UTC](https://forum.inductiveautomation.com/t/multiple-opc-connections-faulting-at-the-same-time/52592/6 "2021-11-01T22:38:35Z")
I am capturing the traffic until they fault again. I will report back when they do! Sometimes it’s once per day other times it’s every hour. It is really frustrating to diagnose when there is no consistency. ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [November 1, 2021, 10:45pm UTC](https://forum.inductiveautomation.com/t/multiple-opc-connections-faulting-at-the-same-time/52592/7 "2021-11-01T22:45:55Z")
Wireshark (and it’s command-line counterpart `tshark`) has good support for configuring traffic filters and for capturing into rolling file buffers, which makes it easier to leave a long-running capture running. ---
**Author:** ![scotteolson3](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@scotteolson3](https://forum.inductiveautomation.com/u/scotteolson3)\ **Post date:** [November 11, 2021, 6:38pm UTC](https://forum.inductiveautomation.com/t/multiple-opc-connections-faulting-at-the-same-time/52592/8 "2021-11-11T18:38:49Z")
Good news (or bad since the problem persists) - I was running a Wireshark capture when the connection went down. The timing of the event somewhat corresponded to pushing a project update to the instance that is running this project. Would there be the possibility that the connection is dropping because of that? Kevin, can I send the wireshark file to you in a DM? ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [November 11, 2021, 6:40pm UTC](https://forum.inductiveautomation.com/t/multiple-opc-connections-faulting-at-the-same-time/52592/9 "2021-11-11T18:40:36Z")
Sure, try to DM it. If that doesn’t work I can get you a dropbox upload link. ---
**Author:** ![scotteolson3](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@scotteolson3](https://forum.inductiveautomation.com/u/scotteolson3)\ **Post date:** [November 11, 2021, 6:44pm UTC](https://forum.inductiveautomation.com/t/multiple-opc-connections-faulting-at-the-same-time/52592/10 "2021-11-11T18:44:30Z")
Ah, yeah 4MB limit and the file is 8.3MB. The dropbox link would be great ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [November 11, 2021, 6:54pm UTC](https://forum.inductiveautomation.com/t/multiple-opc-connections-faulting-at-the-same-time/52592/11 "2021-11-11T18:54:05Z")
Can you tell me the IPs and ports involved? I’m not seeing any OPC UA traffic right away. Also not sure what the two different caps are. ---
**Author:** ![scotteolson3](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@scotteolson3](https://forum.inductiveautomation.com/u/scotteolson3)\ **Post date:** [November 11, 2021, 6:57pm UTC](https://forum.inductiveautomation.com/t/multiple-opc-connections-faulting-at-the-same-time/52592/12 "2021-11-11T18:57:28Z")
The 50.x.x.x:8088 is the Ignition instance while the 10.x.x.x is local traffic. Importantly, our Ignition instance is hosted in the cloud and all the traffic is routed through a site-to-site VPN. ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [November 11, 2021, 6:58pm UTC](https://forum.inductiveautomation.com/t/multiple-opc-connections-faulting-at-the-same-time/52592/13 "2021-11-11T18:58:40Z")
Sorry, it doesn’t look like you’ve managed to capture any of the traffic we need. I know with my local VPN connection it presents in Wireshark as its own network adapter that I can capture on. Not sure what your setup will look like. There’s only traffic between 2 IPs in here, a 50.x and a 10.x, and none of it is OPC UA. ---
**Author:** ![scotteolson3](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@scotteolson3](https://forum.inductiveautomation.com/u/scotteolson3)\ **Post date:** [November 11, 2021, 6:59pm UTC](https://forum.inductiveautomation.com/t/multiple-opc-connections-faulting-at-the-same-time/52592/14 "2021-11-11T18:59:18Z")
Gotcha, I will expand the net and try again! Thanks Kevin ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [November 11, 2021, 7:01pm UTC](https://forum.inductiveautomation.com/t/multiple-opc-connections-faulting-at-the-same-time/52592/15 "2021-11-11T19:01:31Z")
Oops, ignore my reference to captures plural, I had you upload into an existing folder that had an old capture in it already. But what I said above still applies to your capture.