Slow Gateway and tag problem

Ignition
#1

I had a problem today where Ignition wasn’t working properly.
Some of the problems were:
1. Tag Quality cycling between good and bad every 10 seconds from multiple Scan Classes
2. Lots of errors to do with failing database connections and Timeouts.
2. Gateway web pages very slow to update and get status info
3. Designer not responding and failed to open project.

This is only a small system with about 600 tags. And it has only one Client.

I checked the resources on the Gateway PC and it was using 20% CPU (Processor is a i7-2710QE) and about 80% Memory (of 8GB). Although high memory usage, I thought this was ok.

I though I would have to restart the gateway but tried just restarting the Client and it fixed all the problems. After restarting the Client, CPU usage dropped to 5% and java.exe memory usage dropped from about 3GB to 1GB.

What could cause these problems and how can I prevent it happening again?
Could it be to do with viewing long term trends?
My main concern is that many Transactions and Tag Change Scripts failed and these are used for reporting, so the data for this period will be wrong.
Any help is appreciated.

Below are some of the faults I had for your info:

Gateway Faults.jpg1026×958 337 KB

#2

It’s happened again!
It seems that when I try to open a particular Trend it causes all the problems.
This time I had to restart the gateway PC.

I’m using an easy chart component that displays fine when opened with the default time period of 20 minutes. Although it does take a few seconds to display the data.
But when I change the realtime period to a few hours it doesn’t display the data and then turns red. Soon after this all the tags start flashing with bad data and it doesn’t seem to be able to recover.

The chart only has three pens and I cannot see anything wrong with it.
Has anybody had this problem before?
Any ideas on how to fix it?

#3

Ok, I think I’ve found the problem, but not sure what to do about it.
I had a look in the database at the “sqlth_te” table. This stores a list of each tag that has a history.
For most tags there are 1 to 5 entries. Only one active history and the others retired.
For the trend that is causing the problem there are 278,083 retired entries!
I’m not sure how this happened but what should I do about it?
Is there some query that I can run on the table to clean this up?
Can I just delete the retired entries?

Thanks David

#4

It’s seems I’ve fixed my own problem.
I delete all the records in the sqlth_te table for the problem tag except where there were more than 2 records in the sqlt_data tables.
This left 3 entries and deleted 278,080 entries.
The chart now opens fine and doesn’t cause any other tag problems.

My only concern now is how did this happen and what can I do to prevent it happening again?
Does anybody know how this could happen?

Thanks David

#5

Hi,

I’m glad you got it sorted out. As for why it happened, it’s a bit hard to say. I’m having a hard time tracking down the specific tickets/version, but I know we had a situation that could lead to this that was fixed some time ago. For example, we recently had a customer with a similar problem, with lots of entries in the _te table. However, when we analyzed it, we found that they had been created by a previous version (7.6.0 or something), and that it hadn’t been adding them in the newer version.

Did you observe new entries going in? If not, it’s possible that as long as you’re on close to the latest version, it shouldn’t happen again. After you deleted the entries, what does your history look like? Is most of the data there, under the few ids? Or was the recent data spread out across all of these ids?

Finally, did you notice any different between the rows in the _te table at all? One problem that we’ve seen (and not too uncommonly, unfortunately) is that multiple definitions for the same tag get put into the Ignition internal db, and run in parallel. If the tags have different scan classes or history settings, the parallel entries will conflict and will constantly retire the previous and create a new entry. Version 7.7.3 has a variety of mechanisms to fix this, but if you observe that this seems to be the case and the problem continues, we can advise you on how to fix it yourself.

Regards,

#6

Thanks Colby,

I’m running version 7.7.0 and haven’t had any previous versions installed on the PC.

After I deleted the entries, I had about the last 2 hours of data and then there was a big hole. The deleted entries must have been for this time. This makes sense as when I tried using the chart it would work when showing data for the previous 2 hours but would then fail if a longer time period was selected.

I think you are correct with the different scan classes. I have been changing them around a bit trying to isolate different tag problems I’ve been having. You can see below that there are a couple of differences in the rows. There were two tags that I had to clean up as show here:

sqlth_te table extract.jpg682×857 240 KB

#7

Hi,

Yes, that definitely appears to be what’s going on. The other ways that this could happen would be if you had multiple Ignitions with the same gateway name pointing at the same db (like a dev machine that was restored to a production machine), but since this is just a few tags, I’m guessing somehow those particular tags got duplicated in the internal db.

If possible, I believe the easiest way to stop this would be to install 7.7.3-rc1, which you can get from downloads/early access (7.7.3 final is coming out next week, but as far as I know, it only fixes additional issue, none caused by the rc). There have been many things fixed since 7.7.0, so this isn’t a bad idea in general.

Apart from upgrading, you could try the following:
(step 0: back up your system, and do a tag export of the entire provider from the designer)

  1. Figure out how many and which tags have duplicates in the internal db. Go to Configuration>Console>Advanced in the gateway, click past the warning, and run:
select count(*), name,path, ownerid, providerid, tagtype from sqltag group by providerid,name,path,ownerid,tagtype having count(*)>1 order by ownerid desc

2a) If there are just a few tags, go into the designer, and delete them. Then go to Tags>Realtime, edit the tag provider, and just hit ‘save’. This will restart the provider. Go back to the designer, refresh the tag tree, they should come back in. Repeat until the query shows you’re down to just 1.
2b) If there are many tags, you can just delete everything from the designer, restart the tag provider, delete everything again, etc, until you have nothing, and then import the tags that you exported before you started. The tag export does not contain duplicates.

Beyond that, with a little bit of work, we can probably get all of the data that is spread across many ids down to just one id once the _te table is stable. I can give you some ideas on that later, if you need.

Regards,

#8

Thanks Colby,

The data is not too critical at this stage so I don’t mind losing some of it.
I’ll just wait for the official version of 7.7.3 and then install that.