# Loss of gateway on UDT addition **URL:** \ **Category:** Ignition\ **Created:** [July 30, 2014, 6:14pm UTC](https://forum.inductiveautomation.com/t/loss-of-gateway-on-udt-addition/8821 "2014-07-30T18:14:42Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![matt.eaton](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/matt.eaton/32/72046_2.png) [@matt.eaton](https://forum.inductiveautomation.com/u/matt.eaton)\ **Post date:** [July 30, 2014, 6:14pm UTC](https://forum.inductiveautomation.com/t/loss-of-gateway-on-udt-addition/8821/1 "2014-07-30T18:14:42Z")
Okay, so here's what I'm noticing happens a lot. I have a well, it has many components, as we develop new components need to be added. I make a UDT with all the appropriate information and add it to the well UDT. There are about 80 instances of the well UDT and maybe 20 individual members in the added UDT. There is a long pause, I lose connection with the gateway. My wrapper has hundreds of entries like this: ( inserted over tag names) > [@](#): > > INFO | jvm 1 | 2014/07/23 01:05:23 | WARN [SubscriptionManager] [01:05:23,417]: Server returned StatusCode[Severity=Bad, Subcode=Bad\_MonitoredItemIdInvalid] for request to remove item on 'ns=2:s='. > INFO | jvm 1 | 2014/07/23 01:05:23 | WARN [SubscriptionManager] [01:05:23,417]: Server returned StatusCode[Severity=Bad, Subcode=Bad\_MonitoredItemIdInvalid] for request to remove item on 'ns=2:s='. > INFO | jvm 1 | 2014/07/23 01:05:23 | WARN [SubscriptionManager] [01:05:23,522]: Server returned StatusCode[Severity=Bad, Subcode=Bad\_MonitoredItemIdInvalid] for request to remove item on 'ns=2:s='. > INFO | jvm 1 | 2014/07/23 01:05:23 | WARN [SubscriptionManager] [01:05:23,527]: Server returned StatusCode[Severity=Bad, Subcode=Bad\_MonitoredItemIdInvalid] for request to remove item on 'ns=2:s='. > INFO | jvm 1 | 2014/07/23 01:05:23 | WARN [SubscriptionManager] [01:05:23,527]: Server returned StatusCode[Severity=Bad, Subcode=Bad\_MonitoredItemIdInvalid] for request to remove item on 'ns=2:s='. > INFO | jvm 1 | 2014/07/23 01:05:23 | WARN [SubscriptionManager] [01:05:23,527]: Server returned StatusCode[Severity=Bad, Subcode=Bad\_MonitoredItemIdInvalid] for request to remove item on 'ns=2:s='. followed by: > [@](#): > > INFO | jvm 1 | 2014/07/23 01:05:32 | WARN [DeleteMonitoredItemsServiceCall] [01:05:32,415]: Server returned StatusCode[Severity=Bad, Subcode=Bad\_MonitoredItemIdInvalid] for item 'UaSubscriptionItem{nodeId=ns=2:s=, itemId=3734, clientHandle=13479747}'. > INFO | jvm 1 | 2014/07/23 01:05:32 | WARN [DeleteMonitoredItemsServiceCall] [01:05:32,415]: Server returned StatusCode[Severity=Bad, Subcode=Bad\_MonitoredItemIdInvalid] for item 'UaSubscriptionItem{nodeId=ns=2:s=, itemId=4302, clientHandle=13480758}'. > INFO | jvm 1 | 2014/07/23 01:05:32 | WARN [DeleteMonitoredItemsServiceCall] [01:05:32,415]: Server returned StatusCode[Severity=Bad, Subcode=Bad\_MonitoredItemIdInvalid] for item 'UaSubscriptionItem{nodeId=ns=2:s=, itemId=4384, clientHandle=13480933}'. > INFO | jvm 1 | 2014/07/23 01:05:32 | WARN [DeleteMonitoredItemsServiceCall] [01:05:32,415]: Server returned StatusCode[Severity=Bad, Subcode=Bad\_MonitoredItemIdInvalid] for item 'UaSubscriptionItem{nodeId=ns=2:s=, itemId=4436, clientHandle=13480997}'. > INFO | jvm 1 | 2014/07/23 01:05:32 | WARN [DeleteMonitoredItemsServiceCall] [01:05:32,508]: Server returned StatusCode[Severity=Bad, Subcode=Bad\_MonitoredItemIdInvalid] for item 'UaSubscriptionItem{nodeId=ns=2:s=, itemId=3311, clientHandle=13479062}'. > INFO | jvm 1 | 2014/07/23 01:05:32 | WARN [DeleteMonitoredItemsServiceCall] [01:05:32,508]: Server returned StatusCode[Severity=Bad, Subcode=Bad\_MonitoredItemIdInvalid] for item 'UaSubscriptionItem{nodeId=ns=2:s=, itemId=3315, clientHandle=13479068}'. Okay, warnings, fine. All these tags function fine normally and resubscribe on changes. Seems like a hiccup in that process. But the real issue looks like this: > [@](#): > > INFO | jvm 1 | 2014/07/23 01:12:18 | WARN [ClockDriftDetector] [01:12:18,368]: Clock drift, degraded performance, or pause-the-world detected. > INFO | jvm 1 | 2014/07/23 01:12:18 | lastTime=Wed Jul 23 01:11:45 UTC 2014 > INFO | jvm 1 | 2014/07/23 01:12:18 | currentTime=Wed Jul 23 01:12:18 UTC 2014 > INFO | jvm 1 | 2014/07/23 01:12:18 | deviation from expected 1000ms delta=32038ms. > INFO | jvm 1 | 2014/07/23 01:12:18 | ERROR [PublishRequestPump] [01:12:18,371]: Received failure notification for request '[PublishRequest requestHandle=4019]'. > INFO | jvm 1 | 2014/07/23 01:12:18 | StatusCode[Severity=Bad, Subcode=Bad\_Timeout]: Request timed out after 74911ms. > INFO | jvm 1 | 2014/07/23 01:12:18 | at com.inductiveautomation.xopc.client.stack.TCPClientChannel$UaRequestTimeout.run(TCPClientChannel.java:419) > INFO | jvm 1 | 2014/07/23 01:12:18 | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > INFO | jvm 1 | 2014/07/23 01:12:18 | at java.util.concurrent.FutureTask.run(FutureTask.java:266) > INFO | jvm 1 | 2014/07/23 01:12:18 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > INFO | jvm 1 | 2014/07/23 01:12:18 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > INFO | jvm 1 | 2014/07/23 01:12:18 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > INFO | jvm 1 | 2014/07/23 01:12:18 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > INFO | jvm 1 | 2014/07/23 01:12:18 | at java.lang.Thread.run(Thread.java:745) > INFO | jvm 1 | 2014/07/23 01:12:18 | ERROR [PublishRequestPump] [01:12:18,372]: Received failure notification for request '[PublishRequest requestHandle=4020]'. > INFO | jvm 1 | 2014/07/23 01:12:18 | StatusCode[Severity=Bad, Subcode=Bad\_Timeout]: Request timed out after 64912ms. > INFO | jvm 1 | 2014/07/23 01:12:18 | at com.inductiveautomation.xopc.client.stack.TCPClientChannel$UaRequestTimeout.run(TCPClientChannel.java:419) > INFO | jvm 1 | 2014/07/23 01:12:18 | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > INFO | jvm 1 | 2014/07/23 01:12:18 | at java.util.concurrent.FutureTask.run(FutureTask.java:266) > INFO | jvm 1 | 2014/07/23 01:12:18 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > INFO | jvm 1 | 2014/07/23 01:12:18 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > INFO | jvm 1 | 2014/07/23 01:12:18 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > INFO | jvm 1 | 2014/07/23 01:12:18 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > INFO | jvm 1 | 2014/07/23 01:12:18 | at java.lang.Thread.run(Thread.java:745) > INFO | jvm 1 | 2014/07/23 01:12:34 | ERROR [OpcUaConnection:KEPServerEX] [01:12:34,238]: Reading server time and state timed out, disconnecting... Then there are lots of these ClockDriftDetector alarms and errors repeated. At this point I've got to restart the gateway since I can't afford a lot of downtime. I'm starting here in the forum but I'll be calling tech support and updating with their answers. Thanks. ---
**Author:** ![Paullys50](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paullys50/32/3324_2.png) [@Paullys50](https://forum.inductiveautomation.com/u/Paullys50)\ **Post date:** [July 30, 2014, 8:12pm UTC](https://forum.inductiveautomation.com/t/loss-of-gateway-on-udt-addition/8821/2 "2014-07-30T20:12:41Z")
I’m noticing the same thing. Please post what you find out from tech support, I’m new with Ignition and wasn’t sure if that was “typical” or not. Seems like it would be a big deal if you need to make changes on a running system. ---
**Author:** ![matt.eaton](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/matt.eaton/32/72046_2.png) [@matt.eaton](https://forum.inductiveautomation.com/u/matt.eaton)\ **Post date:** [July 30, 2014, 9:31pm UTC](https://forum.inductiveautomation.com/t/loss-of-gateway-on-udt-addition/8821/3 "2014-07-30T21:31:36Z")
Sent my logs in to have them examined ---
**Author:** ![Colby.Clegg](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/colby.clegg/32/3282_2.png) [@Colby.Clegg](https://forum.inductiveautomation.com/u/Colby.Clegg)\ **Post date:** [July 31, 2014, 9:49pm UTC](https://forum.inductiveautomation.com/t/loss-of-gateway-on-udt-addition/8821/4 "2014-07-31T21:49:02Z")
Hi, Any modification to a UDT will cause all instances to be reloaded, which will unsubscribe and resubscribe everything. I’m only guessing, but it seems like that set of operations is causing a problem here, perhaps due to the overall size of the UDT. Could you maybe export all of the tags to XML and (zip and) email them to support? I’d like to see how many objects we’re talking about overall. Regards, ---
**Author:** ![IronYeti](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/ironyeti/32/4124_2.png) [@IronYeti](https://forum.inductiveautomation.com/u/IronYeti)\ **Post date:** [August 1, 2014, 5:38pm UTC](https://forum.inductiveautomation.com/t/loss-of-gateway-on-udt-addition/8821/5 "2014-08-01T17:38:14Z")
I am also having this issue I believe. My tag count seems to be ~67k. I am sending my tag xml to tech support as well. This has become a real for me as well, as the server basically halts with every edit to my UDT. It’s really hard to develop like this. And to make matters worse, customer is going live while we’re still developing. I am also experiencing memory getting maxed out, making the gateway unresponsive and usually requiring me to bounce the server, after which is runs fine. Not sure if it’s related, but it seems like it. BTW, I’m running 7.7.1-rc1 ![](https://us1.discourse-cdn.com/inductiveautomation/original/2X/a/a1f7b84f609148df4b7647d85bca39ae41f4a133.png) ---
**Author:** ![Colby.Clegg](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/colby.clegg/32/3282_2.png) [@Colby.Clegg](https://forum.inductiveautomation.com/u/Colby.Clegg)\ **Post date:** [August 1, 2014, 6:56pm UTC](https://forum.inductiveautomation.com/t/loss-of-gateway-on-udt-addition/8821/6 "2014-08-01T18:56:54Z")
Hi, Thanks, I got your tag file, and I’m looking into it now. I’ve been informed that there was an issue found with large numbers of subscribed items and kepware, which has been fixed for 7.7.1, and might be backported to 7.6. The core issue had to do with having many tags in a single scan class, and how that relates to subscriptions in kepware. Therefore, as a work around in the meantime, you might able to create more scan classes and break your tags up evenly across them. I’ll let you know if we find out more… Regards, ---
**Author:** ![matt.eaton](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/matt.eaton/32/72046_2.png) [@matt.eaton](https://forum.inductiveautomation.com/u/matt.eaton)\ **Post date:** [August 1, 2014, 7:33pm UTC](https://forum.inductiveautomation.com/t/loss-of-gateway-on-udt-addition/8821/7 "2014-08-01T19:33:18Z")
Hey Colby, Did you still want a copy of my tags xml as well? It sounds like I’ve got the same issue. ---
**Author:** ![IronYeti](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/ironyeti/32/4124_2.png) [@IronYeti](https://forum.inductiveautomation.com/u/IronYeti)\ **Post date:** [August 1, 2014, 7:37pm UTC](https://forum.inductiveautomation.com/t/loss-of-gateway-on-udt-addition/8821/8 "2014-08-01T19:37:36Z")
I am using the IA OPC-UA built-in server. I currently have everything in one scan class, but I will break these up a bit and see what happens. Colby, I emailed my tags to support already. ---
**Author:** ![Colby.Clegg](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/colby.clegg/32/3282_2.png) [@Colby.Clegg](https://forum.inductiveautomation.com/u/Colby.Clegg)\ **Post date:** [August 1, 2014, 8:27pm UTC](https://forum.inductiveautomation.com/t/loss-of-gateway-on-udt-addition/8821/9 "2014-08-01T20:27:14Z")
@IronYeti, yes, I should have specified, your tag file is the one I’m currently looking at. Although I confused it with the other logs in regards to kepware. On yours, OPC server specifics aside, due to the structure of the UDTs, there seem to be a ton of things happening on each edit. I’m looking at what we might be able to do. @Matt.Eaton, you can send them to support if you want, but I’m fairly certain your issue is the one that has been recently fixed. We should be able to get updates up early next week. ---
**Author:** ![matt.eaton](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/matt.eaton/32/72046_2.png) [@matt.eaton](https://forum.inductiveautomation.com/u/matt.eaton)\ **Post date:** [August 7, 2014, 3:04pm UTC](https://forum.inductiveautomation.com/t/loss-of-gateway-on-udt-addition/8821/10 "2014-08-07T15:04:03Z")
Can we get an update on the progress of this issue? When you say my issue is one that has been recently fixed do you mean there is an update to Ignition I can install? ---
**Author:** ![Colby.Clegg](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/colby.clegg/32/3282_2.png) [@Colby.Clegg](https://forum.inductiveautomation.com/u/Colby.Clegg)\ **Post date:** [August 7, 2014, 5:57pm UTC](https://forum.inductiveautomation.com/t/loss-of-gateway-on-udt-addition/8821/11 "2014-08-07T17:57:04Z")
Sorry that it’s taking longer than expected to get the next 7.7.1 update up. If possible, you could try this build of the OPC-UA module, which should fix the kepware/subscription issue: [OPC-UA-module.modl](https://us1.discourse-cdn.com/inductiveautomation/original/2X/2/2d8dbd1b1afcb5c38c7a8adde458e51a95b67920.modl) (1.52 MB) ---
**Author:** ![cbabb](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/cbabb/32/49029_2.png) [@cbabb](https://forum.inductiveautomation.com/u/cbabb)\ **Post date:** [August 22, 2014, 2:25pm UTC](https://forum.inductiveautomation.com/t/loss-of-gateway-on-udt-addition/8821/12 "2014-08-22T14:25:02Z")
This issue is affecting me also with 7.7.1-rc1… I’ll try rc2 to see if it eliminates the problem. One other item I noticed is that on the Ignition Gateway Overview Map… currently, with all of my UDT’s being unsubscribed and re-subscribed (there are A LOT of them and this process is taking upwards of 45 mins + to process everything), my server CPU usage is pegged at 100%, yet the map is only showing CPU usage at 7-19% with live values enabled… this cannot be correct… It also drops my redundant server completely off the overview map while this is happening… ---
**Author:** ![matt.eaton](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/matt.eaton/32/72046_2.png) [@matt.eaton](https://forum.inductiveautomation.com/u/matt.eaton)\ **Post date:** [October 2, 2014, 2:49am UTC](https://forum.inductiveautomation.com/t/loss-of-gateway-on-udt-addition/8821/13 "2014-10-02T02:49:56Z")
I’ve got the OPC-UA module installed on my gateway as mentioned previously in this thread by Colby and now my gateway becomes unresponsive for around 90 seconds on a large edit but comes back without the need for a restart. I don’t see the same huge list of errors on the console either. Now after ICC 2014 I got the hint that we might have set the memory usage for Ignition too low. (the standard 2GB) It’s up to 6GB and the gateway handles the UDT changes without any any loss of responsiveness. But what I am seeing now is that the memory usage is going up and staying up after each large edit. ---
**Author:** ![Colby.Clegg](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/colby.clegg/32/3282_2.png) [@Colby.Clegg](https://forum.inductiveautomation.com/u/Colby.Clegg)\ **Post date:** [October 14, 2014, 11:18pm UTC](https://forum.inductiveautomation.com/t/loss-of-gateway-on-udt-addition/8821/14 "2014-10-14T23:18:37Z")
Hi, One slightly unfortunate aspect of Java memory usage is that it’s pretty happy to use as much as you say it can. If you’re not getting Out of Memory errors, and the rest of the system is running fine, I wouldn’t be too concerned about the particular memory usage number reported. The large edits are resulting in a lot of data being generated, and it’s probably just not cleaning it up because it doesn’t really think it needs to. Regards,