Loss of gateway on UDT addition

Ignition
1

Okay, so here's what I'm noticing happens a lot.
I have a well, it has many components, as we develop new components need to be added. I make a UDT with all the appropriate information and add it to the well UDT. There are about 80 instances of the well UDT and maybe 20 individual members in the added UDT. There is a long pause, I lose connection with the gateway.
My wrapper has hundreds of entries like this:

( inserted over tag names)

followed by:

Okay, warnings, fine. All these tags function fine normally and resubscribe on changes. Seems like a hiccup in that process.

But the real issue looks like this:

Then there are lots of these ClockDriftDetector alarms and errors repeated. At this point I've got to restart the gateway since I can't afford a lot of downtime.

I'm starting here in the forum but I'll be calling tech support and updating with their answers. Thanks.

2

I’m noticing the same thing. Please post what you find out from tech support, I’m new with Ignition and wasn’t sure if that was “typical” or not.

Seems like it would be a big deal if you need to make changes on a running system.

3

Sent my logs in to have them examined

4

Hi,

Any modification to a UDT will cause all instances to be reloaded, which will unsubscribe and resubscribe everything. I’m only guessing, but it seems like that set of operations is causing a problem here, perhaps due to the overall size of the UDT.

Could you maybe export all of the tags to XML and (zip and) email them to support? I’d like to see how many objects we’re talking about overall.

Regards,

5

I am also having this issue I believe. My tag count seems to be ~67k. I am sending my tag xml to tech support as well.

This has become a real for me as well, as the server basically halts with every edit to my UDT. It’s really hard to develop like this. And to make matters worse, customer is going live while we’re still developing.

I am also experiencing memory getting maxed out, making the gateway unresponsive and usually requiring me to bounce the server, after which is runs fine. Not sure if it’s related, but it seems like it.

BTW, I’m running 7.7.1-rc1

6

Hi,

Thanks, I got your tag file, and I’m looking into it now.

I’ve been informed that there was an issue found with large numbers of subscribed items and kepware, which has been fixed for 7.7.1, and might be backported to 7.6. The core issue had to do with having many tags in a single scan class, and how that relates to subscriptions in kepware. Therefore, as a work around in the meantime, you might able to create more scan classes and break your tags up evenly across them.

I’ll let you know if we find out more…

Regards,

7

Hey Colby,
Did you still want a copy of my tags xml as well? It sounds like I’ve got the same issue.

8

I am using the IA OPC-UA built-in server. I currently have everything in one scan class, but I will break these up a bit and see what happens.

Colby, I emailed my tags to support already.

9

@IronYeti, yes, I should have specified, your tag file is the one I’m currently looking at. Although I confused it with the other logs in regards to kepware. On yours, OPC server specifics aside, due to the structure of the UDTs, there seem to be a ton of things happening on each edit. I’m looking at what we might be able to do.

@Matt.Eaton, you can send them to support if you want, but I’m fairly certain your issue is the one that has been recently fixed. We should be able to get updates up early next week.

10

Can we get an update on the progress of this issue?
When you say my issue is one that has been recently fixed do you mean there is an update to Ignition I can install?

11

Sorry that it’s taking longer than expected to get the next 7.7.1 update up. If possible, you could try this build of the OPC-UA module, which should fix the kepware/subscription issue:
OPC-UA-module.modl (1.52 MB)

12

This issue is affecting me also with 7.7.1-rc1… I’ll try rc2 to see if it eliminates the problem.

One other item I noticed is that on the Ignition Gateway Overview Map… currently, with all of my UDT’s being unsubscribed and re-subscribed (there are A LOT of them and this process is taking upwards of 45 mins + to process everything), my server CPU usage is pegged at 100%, yet the map is only showing CPU usage at 7-19% with live values enabled… this cannot be correct…

It also drops my redundant server completely off the overview map while this is happening…

13

I’ve got the OPC-UA module installed on my gateway as mentioned previously in this thread by Colby and now my gateway becomes unresponsive for around 90 seconds on a large edit but comes back without the need for a restart. I don’t see the same huge list of errors on the console either.

Now after ICC 2014 I got the hint that we might have set the memory usage for Ignition too low. (the standard 2GB) It’s up to 6GB and the gateway handles the UDT changes without any any loss of responsiveness.

But what I am seeing now is that the memory usage is going up and staying up after each large edit.

14

Hi,

One slightly unfortunate aspect of Java memory usage is that it’s pretty happy to use as much as you say it can. If you’re not getting Out of Memory errors, and the rest of the system is running fine, I wouldn’t be too concerned about the particular memory usage number reported. The large edits are resulting in a lot of data being generated, and it’s probably just not cleaning it up because it doesn’t really think it needs to.

Regards,