Hey guys,
I’ve been getting some complaints about some temperature trends dropping to zero a couple of times a day and I’ve been looking into the problem. The trends look like this:
At first I thought it may be a communication issue, but I log all communication loss alarms and I don’t get any at these times. I then tacked down the actual tag history data in the database and I found this:
It’s inserting nulls with a data integrity code of 600. What is code 600? I can’t find any information for that code anywhere. Also, you’ll notice that the dark red pen does not have this null in the data even though it’s on the same controller as the other pen tags. Any ideas on this or what else I can look for? Thanks!
Ignition version 7.5.3 (b1163)
Hi,
600 means “Unknown”, which I suspect indicates that something strange might be going on with the processor at that time. Maybe the device is becoming disconnected? Take a look at the console around the corresponding time.
The fact that the other pen isn’t affected is a little strange, maybe it’s on a different scan class?
Ultimately I’m not sure why it’s dropping to 0, bad quality data should just be ignored by the chart. What line mode are you using for those pens?
Regards,
Hmmm… I went back through in the console and found out that for some reason the gateway just restarted all on it’s own at that time…
I still don’t understand why it only affected those pens though, everything other trend is fine at that time.
I checked and all of the tags are on the same scan class. By line mode I’m assuming you mean the line type in the style section? It is set to “Line” on all pens.
Are all of your temperature trends coming from the same PLC?
Also, you could try setting the pen’s line style to “Line /w Gaps”. That might at least keep the bad data from showing up on the charts.
Hey guys,
This problem went away for a little while for some reason but the last couple of days it’s been back. I double checked the line mode on my charts and it is set for “Line w/ Gaps” but I’m still getting these drops to zero.
No, not all of the temperatures are coming from the same controller. Which is weird because I don’t see this in any other trends much at all. The only thing I can think of is that because the temperatures are so consistent and there is a lot of space in between records that is causing issues.
Is there anything I should be looking for in the console or the logs at these times to try and narrow down what the issue is? I looked the last couple of days and saw no errors at all at the times where these null rows are being inserted.
There are probably two things to look at:
- Are those values going to 600 quality as it seemed like they were before?
- Are rows being inserted into “sqlth_sce”? This table shouldn’t get very many entries. If you run a “select count(*)…” and get a big number, it could be something going on with that.
Also, there were some fixes to trending made in 7.5.4, so that could help as well. However, if you want to (and can) call support and get on GoToMeeting, we can try to narrow down precisely what causing the issue, and see if it’s been fixed.
There’s one other thing you might be able to do to help us figure out what could be causing it:
- Try to make sure no other clients are running, or are at least polling history (however, not the end of the world if you can’t, because I can filter out the results to one client)
- Turn the logger for “History.SQLTags” to Trace
- In the client/designer, go Help>Diagnostics, and click the button that says “Clear Tag History Cache”
- Trigger the query in the chart, but changing the timeframe slightly, to get the results with the bad value.
- Change back the logger, and post your wrapper.log file.
This should show what the raw loaded values were, what the written values were, etc.
Regards,
Alrighty, I scrubbed out all of the records with a quality value of 600 for now because our QA department was complaining about the trends dropping off, but next time that happens I will get you that information.
To answer your questions:
-
Yes, it seems to be the same situation as it was before.
-
No, no new rows have been added for a while, there are 6 rows in that table (I actually cleaned it out a little while ago in an effort to solve a different issue).
Getting all of the clients off will be difficult since there’s always around 20 or so people on the system 24/7. I guess that’s one of the downsides to having unlimited clients? 
No problem, don’t worry about the clients, you’ve still given me plenty to work with. I’ll try to see if I can mock it up. It would seem like the bad quality data is being worked into the result as 0, but that the row is still being marked as “good”. Otherwise, the chart should ignore bad data, and you would see a gap.
As I mentioned, it may be fixed in 7.5.4, but I’ll try to confirm that.
Regards,
Hey Colby,
I got some more 600 quality rows and I got the data you requested (I hope…). I’ve included the wrapper.log file.
wrapper.log (2.05 MB)
Do you happen to know what time those happened? I don’t see anything talking about loading a quality 600 tag. There are some mentions of “Config Error” (300), but maybe if I had a better idea of where to look, I could notice something.
Regards,
Hey,
It’s ok, I’ve tracked down the issue. Will be fixed in the 7.5.5 release.
Regards,
Cool, thanks Colby!
For future reference the 600 quality tags came in at 10:58:51 AM.
I’ve upgraded to 7.5.5 over the weekend and it has seemed to take care of these trending issues. Now it will just show a gap like it is supposed to for bad quality data. Thanks for the fix, much appreciated!
