All the historically logged tags will have this same issue.
Understood, it's just how their existing system is. Not sure if it can be fixed and or if they would be willing to make any changes. I just need to see if we can do anything with the new Ignition system we are setting up for them. I don't want to push them too hard only to find there was an "easy" fix.
Asking around, someone mentioned transaction groups. I believe they also require a module license, but I also didn't see a way to disable logging on a specific value for a specific tag. If it did stop the sample, it seemed like the trigger would be used for the entire group so Iād need a transaction group for every tag. Not knowing which tags are going to have the bad value.
Someone else mentioned the system.tag.storeTagHistory function. Even if it would work I worry you would end up with lots of running script, delays in timestamps and overhead, to check all the tags and see if it's ok to log a sample.
They are only asking for one sample per minute, so maybe it's not too much for a script to read all of the current values of every tag and as long as the value doesn't equal one specific value, go ahead and write historical data to the tag. None of the tags would actually have historical data logging turned on. They also need to be okay with logging the value every minute, whether it's changed or not. Trying to check each value and see if it changed since the last time seems problematic.
You may just want to script the whole thing, skipping the tag historian.
I wish I knew enough about all this to attempt something like that. So many unknows.
I'll look at doing the two tags if nothing else comes up.
May test with the system.tag.storeTagHistory function just to see if that could work. Also, asking support if they think it could work.
I know that certain people here don't like UDTs, but sounds like a perfect case for one. Create a UDT with one OPC tag, pturmel's expression tag, and a high and low value parameter. The OPC tag gets the data, the expression tag filters the data based on the limits and then stores it. That way it is all automated, all you do is create one new tag instance, supply the OPC path and limits and everything else "just works".
The question will be what you store at the beginning and end of the bad data periods, as that will affect aggregates. Last good value? Zero? Max value? Mark as bad quality with last known (so that you know when the data drop periods occur)?
I feel your pain, we have a bunch of equipment that will respond with 65535, 32767, or 1023 when it is not happy, but it responds and the quality is good. We have the luxury of having a data concentrator in the field, so can filter before it gets to Ignition, but just setting that up is a pain.
Thanks for the extra detail, that's kind of been the plan from the beginning but I wasn't aware of many of the commands and options. It just seemed way cleaner if I could get KEPServerEX to send the value along with the bad quality or stop the historian from logging it in the first place. None of that seems possible unless maybe I want to write my own driver or create my own historian database.
Uh-oh, what's wrong with UDTs? The plan has been to use UDTs to group tags to a device.
I think I'm going to end up with the two tag solution but in the last email from support they did do a 180 suggesting that since I knew the value I wanted to remove that somehow made things better or easier. I think multiple tags is uglier but way easier than trying to write code to remove the value from the historian and even if you don't break anything then that script always needs to run and hopefully runs before anybody requests any historical data.
So you're thinking of using an expression tag and then the system.tag.storeTagHistorian function to store the value to that same tag only when the value is okay or always recording it and using the forceQuality function?
That's part of my ignorance of the historian. If you're recording samples on change or periodic, what's the difference if one particular sample is null or bad quality or was simply never recorded?
I thought about that earlier and was liking the idea of bad quality as like you said it would give you a way to see when the main field device decided a value was bad. Are bad quality values not automatically removed from historical data requests or is that something you must ask for in all requests?
What happens to this trend if the third sample (10) exists or was never stored or has bad quality or is null? Wouldn't it be the same? Wouldn't removing the value, marking it bad or never recording it be better than last good, zero or max? Maybe last good if the value doesn't change much and there are no large gaps in data.
So soon?
Some of us love UDTs but hate UDT-typed UI parameters/properties.
In that trend, 10 would only be logged if your 'max time between samples' forced it to be. With analog storage, we're automatically going to 'skip' the intermediate value by, essentially, buffering until a change in trend happens.
The issue is if instead, the 'bad' value is the your first 15, or your only 15. Suddenly you're flattening a slope, or missing a jump, or something else. It's scary.
Just trying to make it simple. So say it really should have been a small spike of 20 not 10, we still do not have the real sample. Seems better to have the trend as is than to have a sample of min, max or maybe last known.
Trying this out but not sure how to get the expression tag to read the OPC value and pass the value to a memory tag with historical logging enabled setting the quality bad (using forceQuality() ) when a certain value is found.
I can use this script in the Script Console to read the value and then write to a memory tags with historical logging on. I guess I can not have this script in the expression tag? I need to use the runScript function to call a defined script?
server = "KEPServerEX"
path = "nsu=KEPServerEX;s=BACnet.device1500402.pri_pqm_aprnt_power_total_9.PresentValue"
qualifiedValue = system.opc.readValue(server, path)
system.tag.writeBlocking({[.]Apparent Power (Primary)}, qualifiedValue.getValue())
That's not what I recommended at all. No memory tag. Just the expression tag, with history on the expression tag. An expression something like this:
if({path/to/source} >= xyz, forceQuality(someConstant, 524), {path/to/source})
Consider trying null for someConstant.
Sorry, it sounded like more than one tag was needed. "make an expression tag that passes the original value" and "Historize the latter tag instead of the original." ???
So the expression tag can do everything or I need an OPC tag to read the value then the expression tag will read it ({path/to/source}) and set it's own quality to bad if needed.
This. The expression tag is in addition to the original OPC tag. The expression tag is "downstream" of the OPC tag, hence my use of "latter". Yeah, I could have been more clear.
Yeah, I was able to go by that assumption and see and get it working. I just dragged the tag onto a table on a temporary screen and could see that when I set it to the value that the expression is checking for it does mark the value as bad quality and it's not in the list of historical values in the table. I'm not sure if it matters which 500 bad code I use. I was wondering if there was a way to keep the value and just mark it bad. I'm thinking that's what it's doing if you include the value in the forceQuality function, but you need to tell it to show values with bad quality in the table or in any script the request the historical value?
Anyway, fun stuff and thanks for the help.
Sure, include the {path/to/source} in the forceQuality function to pass the value unchanged.
In the test table I see the icon when the value is marked bad with forceQuality() and no further samples are added but so far I can't stop the system.tag.queryTagHistory function from returning the bad value.
I did not have the ignoreBadQuality option but after adding it, nothing changed. I ask for the historical value sometime later and it still says it was 999.
history = system.tag.queryTagHistory(paths=folderPathArray, intervalMinutes=1, startDate=startTime,
endDate=endTime, returnSize=POSTDelay, aggregationMode="Maximum", ignoreBadQuality=1)
ignoreBadQuality - A boolean flag indicating that bad quality values should not be used in the query process. If set, any value with a "bad" quality will be completely ignored in calculations and in the result set.
Maybe it only works with certain types of calculations?
Hmmm. Have you tried using null as the constant in forceQuality ?
I did use null first before I tested the system.tag.queryTagHistory script but wanted the have the value so if you did look at all values you would know it was a value that was forced bad. I can try it with null and see if that changes things.
I remember now, using null, in the Tag Browser the value is shown as Uncertain and in the test table it is shown with the bad icon and a value of 0. It was also showing 0 when using the system.tag.queryTagHistory script. Now with the ignoreBadQuality=1 added it is still the same. 
Hmm. I've run out of ideas.