This post is a follow-up to the previous post on “Machine Learning in Ignition using MATLAB”
My goal is to find an appropriate, industrial approach for integrating machine learning into production processes. Previously, I have looked into directly embedding the AI on the equipment or in the SCADA platform. This introduces many technical challenges (see the previous post). I am now working on an alternative approach using OPC-UA and evaluating the algorithm as a service on a separate server.
The below example uses a Free OpcUa Python Server, numpy, and opencv. It seems to be working fine but I have run into a few errors / annoyances that I don’t know how to debug. Hopefully you can help.
This is the OPC-UA Python server code:
from opcua import Server, ua
import numpy as np
import cv2 as cv
import time
#Get Image Classifier labels
rows = open('model/synset_words.txt').read().strip().split("\n")
CLASSES = [r[r.find(" ") + 1:].split(",")[0] for r in rows]
#Initialize Pre-Trained NN
weights = 'model/DenseNet_121.caffemodel'
architecture ='model/DenseNet_121.prototxt.txt'
net = cv.dnn.readNetFromCaffe(architecture,weights)
#Test on example image in filesystem
test = cv.imread('images/cat.png')
blob = cv.dnn.blobFromImage(test, 0.017, (224, 224), (103.94,116.78,123.68))
net.setInput(blob)
Output = net.forward()
Output = Output.reshape(len(Output[0][:]))
expanded = np.exp(Output - np.max(Output))
prob = expanded / expanded.sum()
index = np.argsort(prob)
Output="The Top 5 Classes are " + CLASSES[index[-1]] + ":" + str(prob[index[-1]]) + ", " + CLASSES[index[-2]] + ":" + str(prob[index[-2]])+ ", " + CLASSES[index[-3]] + ":" + str(prob[index[-3]])+ ", " + CLASSES[index[-4]] + ":" + str(prob[index[-4]])+ ", " + CLASSES[index[-5]] + ":" + str(prob[index[-5]])
print(Output)
#create OPCUA server object
server = Server()
address = "Address Goes here"
url = "opc.tcp://" + address + ":4840"
server.set_endpoint(url)
name = "OPENCV_OPCUA_SERVER"
addspace = server.register_namespace(name)
#create tags for application
node = server.get_objects_node()
#Simple Tag logic
#Client puts data in DataIn and sets DataNew to True
#Server waits for DataNew to be True and then takes DataIn and runs algorithm
#Server puts output from algorithm in ResultOut and sets ResultOut to True
#Client wait for ResultOut to be True and then takes ResultOut
DataNew = node.add_variable("ns=1;s=DataNew", "DataNew", False, ua.VariantType.Boolean)
DataIn = node.add_variable("ns=1;s=DataIn", "DataIn", np.ndarray.tolist(np.zeros((224,224,3))))
ResultNew = node.add_variable("ns=1;s=ResultNew", "ResultNew", False, ua.VariantType.Boolean)
ResultOut = node.add_variable("ns=1;s=ResultOut", "ResultOut", "None", ua.VariantType.String)
DataNew.set_writable()
DataIn.set_writable()
ResultNew.set_writable()
ResultOut.set_writable()
#start server
server.start()
print("Server started at {}".format(url))
#Create algorithm Handler
class OpencvHandler(object):
datain=[]
def datachange_notification(self, node, val, data): #This is inherited from opcua.common.subscription.SubHandler
print("DataNew =" + str(val))
if val==True:
datain = DataIn.get_value()
datain = np.asarray(datain)
datain = datain.astype('uint8')
blob = cv.dnn.blobFromImage(datain, 0.017, (224, 224), (103.94, 116.78, 123.68))
net.setInput(blob)
Output = net.forward()
Output = Output.reshape(len(Output[0][:]))
expanded = np.exp(Output - np.max(Output))
prob = expanded / expanded.sum()
index = np.argsort(prob)
Output = "The Top 5 Classes are " + CLASSES[index[-1]] + ":" + str(prob[index[-1]]) + ", " + CLASSES[
index[-2]] + ":" + str(prob[index[-2]]) + ", " + CLASSES[index[-3]] + ":" + str(
prob[index[-3]]) + ", " + CLASSES[index[-4]] + ":" + str(prob[index[-4]]) + ", " + CLASSES[
index[-5]] + ":" + str(prob[index[-5]])
ResultOut.set_value(Output)
ResultNew.set_value(True)
#Create subscription for handler
handler=OpencvHandler()
sub=server.create_subscription(500, handler)
handle=sub.subscribe_data_change(DataNew)
#Interact with tags
#DataIn.set_value(np.ndarray.tolist(test))
#DataNew.set_value(True)
#time.sleep(1)
#print(ResultOut.get_value())
#print(ResultNew.get_value())
This is the Ignition Script:
from java.io import ByteArrayInputStream
from javax.imageio import ImageIO
import time
import jarray
bytes=system.file.readFileAsBytes("C:\cat.bmp")
stream= ByteArrayInputStream(bytes)
image=ImageIO.read(stream)
#IMG=[]
Temp2=[]
for i in range(image.width):
#Temp1=[]
for j in range(image.height):
#Temp=[]
#Temp.append((image.getRGB(j,i) & 0xff))
#Temp.append((image.getRGB(j,i) & 0xff00) >> 8)
#Temp.append((image.getRGB(j,i) & 0xff0000) >> 16)
#Temp1.append(Temp)
Temp2.append((image.getRGB(j,i) & 0xff))
Temp2.append((image.getRGB(j,i) & 0xff00) >> 8)
Temp2.append((image.getRGB(j,i) & 0xff0000) >> 16)
#IMG.append(Temp1)
#system.opc.writeValue("FreeOpcUa Python Server","ns=1;s=DataIn",IMG))
system.opc.writeValue("FreeOpcUa Python Server","ns=1;s=DataIn",jarray.array(Temp2,'L'))
system.opc.writeValue("FreeOpcUa Python Server","ns=1;s=DataNew",True)
time.sleep(1)
ClassString=system.opc.readValue("FreeOpcUa Python Server","ns=1;s=ResultOut")
event.source.parent.getComponent('Text Field').text=ClassString.value
As a proof of concept, I classified this picture of a cat using DenseNet:
The example is very simple. You click the button, it takes that image of a cat, converts it into an integer array, sends it over OPC-UA to the python server and then turns on a tag that is connected to a data change subscription on the server. This subscription takes the data, applied the opencv NN, and writes the result back to another tag, which ignition picks up. As you can see, it gave me back my result in Ignition. It did all of this in less than a second.
Questions:
-
As you can see from the ignition code, I originally formatted the integer array as a 3-D array. This would not write, I had to make it a 1-D array converted to a java long array. My first test of this was with a pure python client and it had no problem writing the array. What am I doing wrong here? How can I format my data so that I can write to the OPC tag in the format in which it is received by the AI service? It also appears as one long array in the Tag browser. I used an open-source OPC-UA viewer and the array appears as the correct size there. When I try to pass the 3D python list of lists from Ignition of I get this:
Exception raised while parsing message from client, closing
Traceback (most recent call last):
File “…\venv\lib\site-packages\opcua\server\binary_server_asyncio.py”, line 75, in _process_data
ret = self.processor.process(hdr, buf)
File “…\venv\lib\site-packages\opcua\server\uaprocessor.py”, line 96, in process
return self.process_message(msg.SequenceHeader(), msg.body())
File “…\venv\lib\site-packages\opcua\server\uaprocessor.py”, line 116, in process_message
return self._process_message(typeid, requesthdr, seqhdr, body)
File “…\venv\lib\site-packages\opcua\server\uaprocessor.py”, line 189, in _process_message
results = self.session.read(params)
AttributeError: ‘NoneType’ object has no attribute ‘read’ -
I received the following warnings/errors occasionally. They only occur when I am using ignition. Not when using a separate python client. What do I need to do to address these? Or should I just ignore them?
Endpoints other than open requested but private key and certificate are not set.
Unknown message received FourByteNodeId(i=841)