Custom QualifiedValue

I would like to implement a custom QualifiedValue which contains some additional information beyond the base class of BasicQualifiedValue. However, I'm running into an issue where deserialization fails when attempting to read the config: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error cloning object through serialization. Do I have to register the custom type with a deserializer somewhere, or something?

My custom Qualified Value looks like this (MVP):

import com.inductiveautomation.ignition.common.TypeUtilities;
import com.inductiveautomation.ignition.common.model.values.QualifiedValue;
import com.inductiveautomation.ignition.common.model.values.QualityCode;

import java.util.Date;

public class CustomQualifiedValue implements QualifiedValue {

    private final Object value;
    private final QualityCode quality;
    private final Date timestamp;
    private final Object extra;

    public CustomQualifiedValue(Object value, Date timestamp) {
        this.value = value;
        this.quality = QualityCode.Good;
        this.timestamp = timestamp;
        this.extra = null;
    }

    public CustomQualifiedValue(Object value, QualityCode qualityCode, Date timestamp) {
        this.value = value;
        this.quality = qualityCode;
        this.timestamp = timestamp;
        this.extra = null;
    }

    public CustomQualifiedValue(Object value, QualityCode qualityCode, Date timestamp, Object extra) {
        this.value = value;
        this.quality = qualityCode;
        this.timestamp = timestamp;
        this.extra = extra;
    }

    @Override
    public Object getValue() {
        return value;
    }

    @Override
    public QualityCode getQuality() {
        return quality;
    }

    @Override
    public Date getTimestamp() {
        return timestamp;
    }

    public Object getExtra() {
        return extra;
    }

    @Override
    public boolean equals(Object val, boolean includeTimestamp) {
        if (val != null && QualifiedValue.class.isAssignableFrom(val.getClass())) {
            QualifiedValue other = (QualifiedValue)val;
            return TypeUtilities.equals(this.quality, other.getQuality()) &&
                    TypeUtilities.deepEquals(this.value, other.getValue(), true) &&
                    (!includeTimestamp || TypeUtilities.equals(this.timestamp, other.getTimestamp()));
        } else {
            return false;
        }
    }
}

Edit: It looks like there are options for configuring serialization/deserialization via the AbstractGatewayModuleHook (shown below) and I can add my own DeserializationHandler, but the handler class a lot to chew on. Is there a reference implementation for the BasicQualifiedValue or similar?

public class GatewayHook extends AbstractGatewayModuleHook {
    @Override
    public void configureSerializer(XMLSerializer serializer) {
        super.configureSerializer(serializer);
    }

    @Override
    public void configureDeserializer(XMLDeserializer deserializer) {
        super.configureDeserializer(deserializer);
    }
}

Edit: Also, I noticed that trying to create a BasicQualifiedValue with a specified timestamp doesn't actually get created with the specified timestamp. Does anyone know what the cause is of this?

Edit: It looks like I am unable to write an arbitrary timestamp to a tag. Using the test code below, both methods of writing a QualifiedValue to a tag result in the read timestamp being different than the written timestamp. It appears that even though I specify a timestamp to write in both cases, the timestamp is overwritten by whatever the current time is, or the timestamp of the last QualifiedValue is used.

// Create a dummy qualified value with timestamp and value,
//      nowhere near the current time (y2k as example)
Calendar calendar = Calendar.getInstance();
calendar.setTimeZone(TimeZone.getTimeZone("UTC"));
calendar.set(2000, Calendar.JANUARY, 1, 0, 0, 0);
calendar.set(Calendar.MILLISECOND, 0);
Date timestamp = calendar.getTime();
Object value = 42;
QualifiedValue qualifiedValue = new BasicQualifiedValue(value, QualityCode.Good, timestamp);

// Write the qualified value to a tag via saveConfig, read the value, 
//     check whether timestamps match
tagConfig.set(WellKnownTagProps.Value, qualifiedValue);
QualityCode saveResult = _tagProvider
        .saveTagConfigsAsync(Collections.singletonList(tagConfig), CollisionPolicy.Abort)
        .get(30, TimeUnit.SECONDS)
        .get(0);
if (saveResult.isNotGood()) {
    _logger.error("Could not edit tag: " + path.toStringFull());
    throw new Exception("Could not edit tag: " + path.toStringFull());
}
QualifiedValue readValue1 =
        _tagProvider.readAsync(Collections.singletonList(path),
                        SecurityContext.emptyContext())
                .get(30, TimeUnit.SECONDS).get(0);
if (readValue1.getQuality().isNotGood()) {
    _logger.error("Tag value is not good: " + path.toStringFull());
    throw new Exception("Tag value is not good: " + path.toStringFull());
}
var readTs1 = readValue1.getTimestamp();
if (readTs1 != timestamp) {
    _logger.error("save/read timestamps do not match");
}

// Write the qualified value to a tag via writeAsync, read the value, 
//      check whether timestamps match
QualityCode writeResult =
        _tagProvider.writeAsync(Collections.singletonList(path),
                                Collections.singletonList(qualifiedValue),
                                SecurityContext.emptyContext())
                .get(30, TimeUnit.SECONDS)
                .get(0);
if (writeResult.isNotGood()) {
    _logger.error("Could not write value to tag: " + path.toStringFull());
    throw new Exception("Could not write value to tag: " + path.toStringFull());
}
QualifiedValue readValue2 =
        _tagProvider.readAsync(Collections.singletonList(path),
                        SecurityContext.emptyContext())
                .get(30, TimeUnit.SECONDS).get(0);
if (readValue2.getQuality().isNotGood()) {
    _logger.error("Tag value is not good: " + path.toStringFull());
    throw new Exception("Tag value is not good: " + path.toStringFull());
}
var readTs2 = readValue2.getTimestamp();
if (readTs2 != timestamp) {
    _logger.error("write/read timestamps do not match");
}

Edit: The root of what I'm trying to accomplish is to determine the 'source of a change' or 'change reason' for a tag. So, when I read a tag or receive a tag change event from a subscription, I want to determine whether my app/code/module was what changed the tag value, or something else (e.g., another module, or an end user). A custom qualified value would be strongly preferred and the 'source' could be easily determined based on the type of QualifiedValue or a nested value, such as a change source identifier. Alternatively, if I could write arbitrary timestamps to a tag, I could keep track of the timestamp+value pairs and figure out whether my app/code/module was the source of a tag change at the cost of increased CPU + RAM consumption, which would not be preferred, and it looks like I can't write arbitrary timestamps to a tag so this isn't an option. I do not think writing and reading tag configs/properties would work because they can persist across tag value changes.

Any tips/guidance would be much appreciated.

CloneUtil, for better or for worse, uses Java's built in serialization mechanism to attempt to deep-copy any object reference given. So your CustomQualifiedValue would have to implement Serializable. Java's serialization is, in a word, bad, but it's unfortunately deeply entangled with Ignition due to choices made literally over a decade ago. We're actively working to remove these tangles over time, but it's a process. If you just implement Serializable, I expect that everything should work...as long as whatever arbitrary Object you provide can itself also cleanly serialize. Java primitives and arrays generally can, as well as some list and map implementations, but it quickly becomes a minefield. Consider restricting yourself to something narrower than just Object if this is the approach you stick to.

This isn't the path you have to go here; there's no way to contribute your own logic to CloneUtil or avoid its use.

This may or may not be intentional (I'm not an expert on the tag system) - but one question - where are you initiating this write from? If it's gateway scoped code, then yeah, this is likely impossible. If it's client or designer scoped code, then it might still be possible, and it's just that the RPC layer is stripping the timestamps.

Why not, every time your module is writing a new value to a tag, also write that value to an additional property? Then you can compare the two and know that the source of the change was (or wasn't) your module anywhere downstream.

1 Like

Excellent, thank you for the pointers. This is in the gateway context. I implemented Serializable but am still having what appears to be the same issue.

I can reproduce the issue via manually calling this:

var cqv2 = CloneUtil.cloneBySerialization(qualifiedValue);

When stepping into this via the debugger, the function looks like:

public static <T extends Serializable> T cloneBySerialization(T value) {
    ObjectOutputStream oos = null;
    ObjectInputStream ois = null;

    Serializable var5;
    try {
        ByteArrayOutputStream bos = new ByteArrayOutputStream();
        oos = new ObjectOutputStream(bos);
        oos.writeObject(value);
        oos.flush();
        ByteArrayInputStream bin = new ByteArrayInputStream(bos.toByteArray());
        ois = new ObjectInputStream(bin);
        var5 = (Serializable)ois.readObject();
    } catch (Exception var9) {
        Exception e = var9;
        LogManager.getLogger(CloneUtil.class).debug("Error cloning object through serialization.", e);
        throw new RuntimeException("Error cloning object through serialization.", e);
    } finally {
        IOUtils.closeQuietly(oos);
        IOUtils.closeQuietly(ois);
    }

    return var5;
}

The line that throws an exception is:

ois = new ObjectInputStream(bin);

Which throws an exception:

java.lang.ClassNotFoundException: com.inductiveautomation.ignition.examples.modified.CustomQualifiedValue

So, it looks like it can't find my class. This is wrapped in the previously seen exception, which I missed:

java.lang.RuntimeException: Error cloning object through serialization.

The interesting thing is, when I copy the code from the cloneBySerialization(T value) function and paste it into my module, it is able to serialize and deserialize CustomQualifiedValue just fine. So the underlying issue appears to be that Ignition is not able to see or reference classes in my module via CloneUtil. Also, Intellij won't let me step into or set breakpoints in ObjectInputStream, so I can't really dig any deeper.

Any suggestions on where to go from here?

I am new to module development in Ignition so correct me if I'm wrong here, but this is what I'm seeing and my train of thought:

When tag changes occur slowly, say write then handle three changes to a tag, you can handle them progressively, so storing data in TagConfigurationModel (i.e., config) properties works (W#=Write, R#=Read/TagChangeEvent, #=sequence of tag change), so write read, write read, write read:

W1 > R1 > W2 > R2 > W3 > R3

If implementing the TagChangeListener interface function tagChanged(TagChangeEvent tagChangeEvent) to handle tag changes, the TagChangeEvent only contains the QualifiedValue, and not the whole tag config with properties. In tandem, since the environment is asynchronous (and follows CQRS with eventual consistency?), with what appears to be batch processing happening on the back end, tag change events R# can all happen after writes W#. When a tag is changing a lot, quickly, I've seen this happen while debugging, where all the writes go through, then change events lag behind:

W1 > W2 > W3 > R1 > R2 > R3

In my case, I want to make sure I don't miss any changes, so I am explicitly using TagChangeEvents rather than polling reads. Since TagChangeEvents only contain QualifiedValues and not config properties, and I can't async read the config because it's already stale (W2 and W3 have already been committed) by the time I receive R1, I can't utilize config properties to store data specific to a QualifiedValue/TagChangeEvent. So, I think that if a tag is changing at a high frequency, and I want to be able to reliably store, retrieve, and verify data for every tag change, the only place I can store data is in a QualifiedValue.

Right - CloneUtil isn't working for your class because we deliberately segment each module into a different classloader so that they can bring in different versions of possibly conflicting dependencies and don't stomp all over each other.

I don't think there's really any way around that.

But, good news/bad news...
I don't think the custom qualified value or directly manipulating tag properties is what you want to do at all. And there is a mechanism for doing it - the so called 'tag actor' system, where you register a 'factory' via your module hook, and the tag system automatically creates as many instances of your actor (which can intercept writes, manipulate values, log, etc) as are required by the underlying tags.
To my mind, this is the perfect solution to your problem.

The bad news, then:
There's close to zero public facing documentation on this system (and it would be the blind leading the blind if I tried to offer more than very broad spectrum advice).
Some starting points:

1 Like

I understand the rationale for doing this. Is there a procedure for submitting a feature request for adding a class to the SDK, derived from BasicQualifiedValue, which includes a Map<String, String> UserData field along with value, quality and timestamp? Having this defined in the SDK would get around the CloneUtil class not found issue since the derived QualifiedValue would be defined on the SDK side, and using a Map would provide some flexibility beyond a single field while ensuring serializability by constraining types to String, or perhaps a generic T (Map<String,T>) with a constraint such as Serializable or union of allowable types. At face value from the perspective of a naive end user, (I know this is a leap) this doesn't sound like a huge ask, but I'm aware it could snowball.

I've looked over the attached ExampleTagActor.txt in the linked post, and I'm having trouble discerning where the TagActor is utilized in the signal chain, as well as how it fits in to a solution. Is this intercepting tag writes, tag configuration saves, or tag change events? It looks like the tag configuration and qualified value can be modified by a TagActor, but I'm not sure about where in the signal chain this is occurring (writes, saves, change events). Also, I'm having trouble seeing the forest through the trees; could you please expand on how the TagActor and factory fits into a solution? This approach also appears to also be directly manipulating QualifiedValues and tag properties.

Everything in the new tag system is a tag actor.

If it's flexible enough to drive tag values, including OPC, scaling, history, etc... I'm pretty confident it's flexible enough for your purposes. As a bonus, you can get it working today.

We wouldn't add that feature to 8.1, so at minimum you're waiting until 8.3 is released at the end of this year. More likely much longer than that, because it's low priority relative to other customer demands.

So, again, I'm not the expert in this area, but I'll just really try to push that working inside the capabilities that are already exposed is definitely going to be a better experience.

My rough understanding of the whole idea is something like this:

  1. You get the tag manager from your instance of GatewayContext: https://files.inductiveautomation.com/sdk/javadoc/ignition81/8.1.40/com/inductiveautomation/ignition/gateway/model/GatewayContext.html#getTagManager()
  2. You get the tag config manager:
    GatewayTagManager
  3. From there, you register your actor factories: TagConfigurationManager
    • Optionally, you can contribute known property types to the tag system - this is how e.g. the alarm notification module adds the 'pipeline' properties to tags: TagConfigurationManager
  4. Your TagActorFactory defines the "things that it cares about" via getMonitoredProperties:
    TagActorFactory
  5. Your factory is called to create new TagActors by the tag system, whenever your relevant properties change. Your TagActors themselves act as either 'read' or 'write' agents in a constructed pipeline (highly recommend reading the class documentation on TagActor): TagActor
2 Likes