Hello,
We have a number of AI/ML applications we are looking to put into Ignition. I have been working on one method of doing so. The process I came up with is as follows:
- Develop the ML algorithm in a dedicated Math/Statistics Software (In our case, MATLAB)
- Use the available code generation tools to export the algorithm to JAVA (from MATLAB)
- Write a simple JAVA class wrapper for the MATLAB algorithm JAVA generated code
- Place the .jar and .jar.pack.gz files in server locations (there was another post on this)
- Install the MATLAB run-time library on the ignition server (This is where I am having a problem)
- Import the java Class and interact via scripting
As a first use-case, I decided to experiment with googlenet.
My first test was to verify that I could pass a python list of lists from ignition to the java class and perform linear algebra. This was successful. See the first Code set below.
My second test was to verify I could export the googlenet network and run the algorithm in eclipse. This was succcessful. See the second Code set below.
Lastly, I tried to incorporate it in Ignition. I ran into the error below. In Eclipse, I had to point to the MATLAB runtime javabuilder.jar file, which is here (C:\Program Files\MATLAB\MATLAB Runtime\v96\toolbox\javabuilder\jar). I installed the runtime library in the same location on the server. How can I make ignition see this?
MATLAB creates a NNpredict.jar file. This is also referenced by my class. I also put this on the server. I ran into a problem packing this. It is 75 MB and Eclipse gave me a heap memory error. I got around this by using: pack200 -Evalue --effort=0 NNpredict.jar.pack.gz NNpredict.jar
Could this be causing the problem? While I mention it, how can I increase the heap memory for pack200? The internet did not yield any solutions.
In the example below, I am classifying an image of a cat, in case you are wondering about cat.png. Also, I basically gave myself a crash course in Java this week. There is a good chance there is something wrong with my code… I read the first half of Schildt, Java, the complete reference twelfth edition. There is probably some key thing in the second half that would have made this work…
Anyway, thanks in advance. If I can get this to work, we will have ML and AI available to our entire SCADA platform!!!
16:23:57.240 [AWT-EventQueue-0] ERROR com.inductiveautomation.factorypmi.application.binding.action.ActionAdapter - <HTML>Error executing script for event: <code><b>actionPerformed</b></code><BR>on component: <code><b>Button 1</b></code>.
com.inductiveautomation.ignition.common.script.JythonExecException: Traceback (most recent call last):
File "<event:actionPerformed>", line 1, in <module>
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Unknown Source)
at org.python.core.Py.loadAndInitClass(Py.java:909)
at org.python.core.Py.findClassInternal(Py.java:844)
at org.python.core.Py.findClassEx(Py.java:895)
at org.python.core.packagecache.SysPackageManager.findClass(SysPackageManager.java:133)
at org.python.core.packagecache.PackageManager.findClass(PackageManager.java:28)
at org.python.core.packagecache.SysPackageManager.findClass(SysPackageManager.java:122)
at org.python.core.PyJavaPackage.__findattr_ex__(PyJavaPackage.java:137)
at org.python.core.PyObject.__findattr__(PyObject.java:863)
at org.python.core.packagecache.PackageManager.lookupName(PackageManager.java:136)
at org.python.core.JavaImporter.find_module(JavaImporter.java:39)
at org.python.core.JavaImporter.find_module(JavaImporter.java:25)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
java.lang.NoClassDefFoundError: java.lang.NoClassDefFoundError: com/mathworks/toolbox/javabuilder/MWArray
at org.python.core.Py.JavaError(Py.java:495)
at org.python.core.Py.findClassEx(Py.java:901)
at org.python.core.packagecache.SysPackageManager.findClass(SysPackageManager.java:133)
at org.python.core.packagecache.PackageManager.findClass(PackageManager.java:28)
at org.python.core.packagecache.SysPackageManager.findClass(SysPackageManager.java:122)
at org.python.core.PyJavaPackage.__findattr_ex__(PyJavaPackage.java:137)
at org.python.core.PyObject.__findattr__(PyObject.java:863)
at org.python.core.packagecache.PackageManager.lookupName(PackageManager.java:136)
at org.python.core.JavaImporter.find_module(JavaImporter.java:39)
at org.python.core.JavaImporter.find_module(JavaImporter.java:25)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:186)
at org.python.core.PyObject.__call__(PyObject.java:345)
at org.python.core.PyMethod.instancemethod___call__(PyMethod.java:220)
at org.python.core.PyMethod.__call__(PyMethod.java:211)
at org.python.core.PyMethod.__call__(PyMethod.java:201)
at org.python.core.PyMethod.__call__(PyMethod.java:196)
at org.python.core.imp.find_module(imp.java:469)
at org.python.core.imp.import_next(imp.java:718)
at org.python.core.imp.import_module_level(imp.java:827)
at org.python.core.imp.importName(imp.java:917)
at org.python.core.ImportFunction.__call__(__builtin__.java:1220)
at org.python.core.PyObject.__call__(PyObject.java:357)
at org.python.core.__builtin__.__import__(__builtin__.java:1173)
at org.python.core.imp.importOne(imp.java:936)
at org.python.pycode._pyx3.f$0(<event:actionPerformed>:1)
at org.python.pycode._pyx3.call_function(<event:actionPerformed>)
at org.python.core.PyTableCode.call(PyTableCode.java:165)
at org.python.core.PyCode.call(PyCode.java:18)
at org.python.core.Py.runCode(Py.java:1275)
at com.inductiveautomation.ignition.common.script.ScriptManager.runCode(ScriptManager.java:636)
at com.inductiveautomation.factorypmi.application.binding.action.ActionAdapter.runActions(ActionAdapter.java:193)
at com.inductiveautomation.factorypmi.application.binding.action.ActionAdapter.invoke(ActionAdapter.java:284)
at com.inductiveautomation.factorypmi.application.binding.action.RelayInvocationHandler.invoke(RelayInvocationHandler.java:57)
at com.sun.proxy.$Proxy36.actionPerformed(Unknown Source)
at java.desktop/javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
at java.desktop/javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source)
at java.desktop/javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
at java.desktop/javax.swing.DefaultButtonModel.setPressed(Unknown Source)
at java.desktop/javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown Source)
at java.desktop/java.awt.Component.processMouseEvent(Unknown Source)
at java.desktop/javax.swing.JComponent.processMouseEvent(Unknown Source)
at java.desktop/java.awt.Component.processEvent(Unknown Source)
at java.desktop/java.awt.Container.processEvent(Unknown Source)
at java.desktop/java.awt.Component.dispatchEventImpl(Unknown Source)
at java.desktop/java.awt.Container.dispatchEventImpl(Unknown Source)
at java.desktop/java.awt.Component.dispatchEvent(Unknown Source)
at java.desktop/java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
at java.desktop/java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
at java.desktop/java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
at java.desktop/java.awt.Container.dispatchEventImpl(Unknown Source)
at java.desktop/java.awt.Window.dispatchEventImpl(Unknown Source)
at java.desktop/java.awt.Component.dispatchEvent(Unknown Source)
at java.desktop/java.awt.EventQueue.dispatchEventImpl(Unknown Source)
at java.desktop/java.awt.EventQueue.access$500(Unknown Source)
at java.desktop/java.awt.EventQueue$3.run(Unknown Source)
at java.desktop/java.awt.EventQueue$3.run(Unknown Source)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
at java.desktop/java.awt.EventQueue$4.run(Unknown Source)
at java.desktop/java.awt.EventQueue$4.run(Unknown Source)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
at java.desktop/java.awt.EventQueue.dispatchEvent(Unknown Source)
at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
at java.desktop/java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.desktop/java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.desktop/java.awt.EventDispatchThread.run(Unknown Source)
Caused by: org.python.core.PyException: Traceback (most recent call last):
File "<event:actionPerformed>", line 1, in <module>
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Unknown Source)
at org.python.core.Py.loadAndInitClass(Py.java:909)
at org.python.core.Py.findClassInternal(Py.java:844)
at org.python.core.Py.findClassEx(Py.java:895)
at org.python.core.packagecache.SysPackageManager.findClass(SysPackageManager.java:133)
at org.python.core.packagecache.PackageManager.findClass(PackageManager.java:28)
at org.python.core.packagecache.SysPackageManager.findClass(SysPackageManager.java:122)
at org.python.core.PyJavaPackage.__findattr_ex__(PyJavaPackage.java:137)
at org.python.core.PyObject.__findattr__(PyObject.java:863)
at org.python.core.packagecache.PackageManager.lookupName(PackageManager.java:136)
at org.python.core.JavaImporter.find_module(JavaImporter.java:39)
at org.python.core.JavaImporter.find_module(JavaImporter.java:25)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
java.lang.NoClassDefFoundError: java.lang.NoClassDefFoundError: com/mathworks/toolbox/javabuilder/MWArray
... 74 common frames omitted
Caused by: java.lang.NoClassDefFoundError: com/mathworks/toolbox/javabuilder/MWArray
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Unknown Source)
at org.python.core.Py.loadAndInitClass(Py.java:909)
at org.python.core.Py.findClassInternal(Py.java:844)
at org.python.core.Py.findClassEx(Py.java:895)
... 72 common frames omitted
Caused by: java.lang.ClassNotFoundException: com.mathworks.toolbox.javabuilder.MWArray
at java.base/java.net.URLClassLoader.findClass(Unknown Source)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
... 77 common frames omitted
The initial linear algebra test code in java is here:
public class MatrixOperations {
private double[][] mat1;
private double[][] mat2;
// Constructor to initialize matrix Class
public MatrixOperations(double[][] mat1, double[][] mat2) {
//Assume the array has consistent row lengths
this.mat1=new double[mat1.length][mat1[0].length];
for (int i=0;i<mat1.length;i++) {
for (int j=0;j<mat1[0].length;j++) {
this.mat1[i][j]=mat1[i][j];
}
}
this.mat2=new double[mat2.length][mat2[0].length];
for (int i=0;i<mat2.length;i++) {
for (int j=0;j<mat2[0].length;j++) {
this.mat2[i][j]=mat2[i][j];
}
}
}
public double[][] add(){
//if they are adding, assume the matrices are the same size
double[][] computation=new double[this.mat1.length][this.mat1[0].length];
for (int i=0;i<this.mat1.length;i++) {
for (int j=0;j<this.mat1[0].length;j++) {
computation[i][j]=this.mat1[i][j]+mat2[i][j];
}
}
return computation;
}
public double[][] multiply() {
//if they are multiplying, assume the matrix sizes are compatible
double[][] computation=new double[this.mat1.length][this.mat2[0].length];
for (int i=0;i<this.mat1.length;i++) {
for (int j=0;j<this.mat2[0].length;j++) {
for (int k=0;k<this.mat1[0].length;k++) {
computation[i][j]+=this.mat1[i][k]*mat2[k][j];
}
}
}
return computation;
}
public static void main(String[] args) {
byte x= -2;
System.out.println(x);
System.out.println("This is a simple program that computes matrix algrebra");
double[][] xIn = {{1.0, 1.0}, {2.0, 2.0}, {3.0, 3.0}, {4.0, 4.0}, {5.0, 5.0}};
double[][] yIn = {{1.0, 1.0, 1.0, 1.0, 1.0}, {2.0, 2.0, 2.0, 2.0, 2.0}};
int size1 = xIn.length;
int size2 = xIn[0].length;
System.out.println("Here are two example matrices of size "+ size1 + " by " + size2 +" and " + size2 + " by " + size1);
int i, j;
for(i=0; i<size1; i++){
for (j=0;j<size2; j++){
{System.out.print(xIn[i][j]+" ");}
}
System.out.println();
}
System.out.println();
for(i=0; i<size2; i++){
for (j=0;j<size1; j++){
{System.out.print(yIn[i][j]+" ");}
}
System.out.println();
}
double[][] zMult = new double[size1][size1];
MatrixOperations MAT = new MatrixOperations(xIn,yIn);
zMult=MAT.multiply();
System.out.println();
System.out.println("There Product is: ");
for(i=0; i<size1; i++){
for (j=0;j<size1; j++){
{System.out.print(zMult[i][j]+" ");}
}
System.out.println();
}
}
}
In ignition, I run the following:
import MatrixOperations
n=5
Matrix1=[]
Matrix2=[]
for i in range(n):
Matrix1.append([1]*n)
Matrix2.append([1]*n)
Compute=MatrixOperations(Matrix1,Matrix2)
#print Compute.main([''])
ComputeMultiply=Compute.multiply()
ComputeAdd=Compute.add()
print ComputeMultiply[1][1]
print ComputeAdd[1][1]
The googlenet code is here:
import java.util.Arrays;
import javax.imageio.ImageIO;
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.IOException;
import java.awt.image.DataBufferByte;
import com.mathworks.toolbox.javabuilder.*;
import NNpredict.Class1;
public class googlenetwrapper {
private static Class1 class1Instance;
private static void setup() throws MWException {
class1Instance = new Class1();
}
public static double[][] runNNpredict(int[][][] inData) {
MWArray In = null;
MWNumericArray Out = null;
Object[] results = null;
double[][] Pvector = null;
try {
In = new MWNumericArray(inData, MWClassID.UINT8);
results = class1Instance.NNpredict(1, In);
if (results[0] instanceof MWNumericArray) {
Out = (MWNumericArray) results[0];
}
Pvector = (double[][]) Out.toDoubleArray();
return Pvector;
} catch (Exception e) {
e.printStackTrace();
return Pvector;
} finally {
// Dispose of native resources
MWArray.disposeArray(In);
MWArray.disposeArray(results);
}
}
public static void main(String[] args) {
// Demonstrate functionality of class. Read in example image, run googlenet on it, and display top 10 probabilities
//Read in image and convert to 3D array
try {
BufferedImage image = ImageIO.read(new File("D:\\MATLAB\\ExportTesting\\Java\\cat.png"));
int width=image.getWidth();
int height=image.getHeight();
int[][][] pixels = new int[height][width][3];
byte[] imagepixels = ((DataBufferByte) image.getRaster().getDataBuffer()).getData();
int n=0;
for (int i = 0; i < height; i++) {
for (int j = 0; j < width; j++) {
for (int k=0;k<3;k++) {
pixels[i][j][k]=imagepixels[n];
n++;
}
}
}
//Initialize MATLABs objects
try {
setup();
} catch (Exception e) {
e.printStackTrace();
System.exit(1);
}
//Pass image array to network
try {
//Run ML
double[][] Pvector=runNNpredict(pixels);
//Include Index and sort on Magnitude
double[][] Svector=new double[Pvector[0].length][2];
for (int i=0; i<Pvector[0].length;i++) {
Svector[i][0]=Pvector[0][i];
Svector[i][1]=i;
}
Arrays.sort(Svector,(a, b) -> Double.compare(-a[0], -b[0]));
//Display results
double[][] TopTen=new double[10][2];
System.out.println("The top ten probabilities and class indexes are:");
for (int i=0; i<10;i++) {
TopTen[i][0]=Svector[i][0];
TopTen[i][1]=Svector[i][1];
System.out.print(TopTen[i][0]+" "+TopTen[i][1]);
System.out.println();
}
} catch (Exception e) {
e.printStackTrace();
System.exit(1);
} finally {
// Dispose of native resources
class1Instance.dispose();
}
} catch(IOException e) {}
}
}