Connect to camera using opencv

[daniel.garcia-redrue]

Hi all,
I’m newbie in module development using Ignition SDK. I tried the examples and I focused on the Vision Component Example. I am trying to develop a new component to connect to an IP camera using OpenCV.
Right now my code generates a .modl file, but maybe I’m doing things wrong when I repaint my component because I’m getting the exception:
Exception in thread “AWT-EventQueue-0” java.lang.NullPointerException

• at imas.client.OCRComponent.paintComponent(OCRComponent.java:116)*
• at java.desktop/javax.swing.JComponent.paint(Unknown Source)*
• at java.desktop/javax.swing.JComponent.paintChildren(Unknown Source)*
• at com.inductiveautomation.vision.api.client.components.model.AbstractVisionPanel.paintChildren(AbstractVisionPanel.java:290)*
• at java.desktop/javax.swing.JComponent.paint(Unknown Source)*
• at java.desktop/javax.swing.JComponent.paintChildren(Unknown Source)*
• at java.desktop/javax.swing.JComponent.paint(Unknown Source)*
• at java.desktop/javax.swing.JLayeredPane.paint(Unknown Source)*
• at java.desktop/javax.swing.JComponent.paintChildren(Unknown Source)*
• at java.desktop/javax.swing.JComponent.paint(Unknown Source)*
• at java.desktop/javax.swing.JComponent.paintChildren(Unknown Source)*
• at java.desktop/javax.swing.JComponent.paint(Unknown Source)*
• at java.desktop/javax.swing.JComponent.paintChildren(Unknown Source)*
• at com.inductiveautomation.ignition.designer.designable.DesignPanel$DesignableContainerLayer.paintChildren(DesignPanel.java:1169)*
• at java.desktop/javax.swing.JComponent.paint(Unknown Source)*
• at java.desktop/javax.swing.JComponent.paintChildren(Unknown Source)*
• at java.desktop/javax.swing.JComponent.paint(Unknown Source)*
• at java.desktop/javax.swing.JComponent.paintToOffscreen(Unknown Source)*
• at java.desktop/javax.swing.RepaintManager$PaintManager.paintDoubleBufferedImpl(Unknown Source)*
• at java.desktop/javax.swing.RepaintManager$PaintManager.paintDoubleBuffered(Unknown Source)*
• at java.desktop/javax.swing.RepaintManager$PaintManager.paint(Unknown Source)*
• at java.desktop/javax.swing.RepaintManager.paint(Unknown Source)*
• at java.desktop/javax.swing.JComponent._paintImmediately(Unknown Source)*
• at java.desktop/javax.swing.JComponent.paintImmediately(Unknown Source)*
• at java.desktop/javax.swing.RepaintManager$4.run(Unknown Source)*
• at java.desktop/javax.swing.RepaintManager$4.run(Unknown Source)*
• at java.base/java.security.AccessController.doPrivileged(Native Method)*
• at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)*
• at java.desktop/javax.swing.RepaintManager.paintDirtyRegions(Unknown Source)*
• at java.desktop/javax.swing.RepaintManager.paintDirtyRegions(Unknown Source)*
• at java.desktop/javax.swing.RepaintManager.prePaintDirtyRegions(Unknown Source)*
• at java.desktop/javax.swing.RepaintManager$ProcessingRunnable.run(Unknown Source)*
• at java.desktop/java.awt.event.InvocationEvent.dispatch(Unknown Source)*
• at java.desktop/java.awt.EventQueue.dispatchEventImpl(Unknown Source)*
• at java.desktop/java.awt.EventQueue$4.run(Unknown Source)*
• at java.desktop/java.awt.EventQueue$4.run(Unknown Source)*
• at java.base/java.security.AccessController.doPrivileged(Native Method)*
• at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)*
• at java.desktop/java.awt.EventQueue.dispatchEvent(Unknown Source)*
• at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)*
• at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)*
• at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)*
• at java.desktop/java.awt.EventDispatchThread.pumpEvents(Unknown Source)*
• at java.desktop/java.awt.EventDispatchThread.pumpEvents(Unknown Source)*
• at java.desktop/java.awt.EventDispatchThread.run(Unknown Source)*

This is my component class:

package imas.client;
import imas.client.VideoCamera;
import java.util.LinkedList;
import java.util.Queue;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import javax.swing.JFrame;
import javax.swing.Timer;
import java.awt.Graphics;
import java.awt.Graphics2D;
import java.awt.geom.AffineTransform;
import java.awt.Font;
import java.awt.Dimension;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import java.awt.image.BufferedImage;

import org.opencv.core.Core;
import org.opencv.core.Mat;
import org.opencv.videoio.VideoCapture;

import com.inductiveautomation.ignition.client.model.LocaleListener;
import com.inductiveautomation.vision.api.client.components.model.AbstractVisionComponent;


public class OCRComponent extends AbstractVisionComponent implements LocaleListener {
    /**
     * PROPERTIES
     */
    private String cameraURL = "";
    private int state = STATE_OFF;
     
    /// ENUM PROPERTIES
    public static final int STATE_OFF = 0;
    public static final int STATE_ON = 1;
    	
    private static VideoCamera cam;
   
   // a timer for acquiring the video stream
    private Timer _timer;
   // a flag to change the button behavior
   private boolean cameraActive = false;
	
	
	 /**
     * The listener for the swing timer
     */
    private ActionListener animationListener = new ActionListener() {
        public void actionPerformed(ActionEvent e) { 
        	if(cam.isOpened())
        	{
        		repaint();
        	}
        	else
        	{
        		if(cameraURL != "")
        		{
        			 cam.startCamera(cameraURL);
        			 repaint();
        		}
        	}
        }
    };
	
	 public OCRComponent() {
        setOpaque(false);
        setPreferredSize(new Dimension(640, 480));
        setFont(new Font("Dialog", Font.PLAIN, 24));       
        try
    	{
        	cam = new VideoCamera();
    	}catch(Exception e)
    	{
    		System.out.println(e.getMessage());
    	}
        System.out.println("OCR component inicializado");
	 }

    public static void main(String[] args) {
        JFrame frame = new JFrame("ocrTest");
        frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        frame.add(new OCRComponent());
        frame.setSize(800,800); 
        frame.setBounds(100, 100, 300, 300);
        frame.setVisible(true);
    }
	    
    	@Override
    	protected void onStartup() {
    		// Seems like a no-op, but actually will trigger logic to re-start the timer if necessary
    		System.out.println("onStartup(URL): " + cameraURL);
    	}

    	@Override
    	protected void onShutdown() {
    		if (_timer != null && _timer.isRunning()) {
    			_timer.stop();
    			System.out.println("onShutdown, timer detenido");
    		}
    	}
    
	    @Override
	    protected void paintComponent(Graphics _g) {
	    	Graphics2D g = (Graphics2D) _g;
	    	// Preserve the original transform to roll back to at the end
	        AffineTransform originalTx = g.getTransform();
	    	g.drawImage(cam.lastFrame,10,10,cam.lastFrame.getWidth(),cam.lastFrame.getHeight(), null);
	        // Reverse any transforms we made
	        g.setTransform(originalTx);
	    }
	    
	    public String getCameraURL() {
	    	System.out.println("getCameraURL(): " + cameraURL);
	        return cameraURL;
	    }

	    public void setCameraURL(String _url) {
	    	String oldValue = this.cameraURL;
	        this.cameraURL = _url;	 
	        this.cam.startCamera(this.cameraURL);
	        repaint();
	    }
	    
	    public int getState() {
	        return state;
	    }

	    public void setState(int _state) {
	        this.state = _state;
	        repaint();
	    }
	    
}

And my VideoCamera class to connect:

package imas.client;

import javax.swing.JPanel;
import org.opencv.core.Core;
import org.opencv.videoio.VideoCapture;
import org.opencv.core.Mat;

import java.awt.Graphics;
import java.awt.image.BufferedImage;
import java.awt.image.DataBufferByte;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;

public class VideoCamera extends JPanel
{
	/// Video stream
	VideoCapture camera;
	/// Camera URL
	private String url;
	// a timer for acquiring the video stream
	private ScheduledExecutorService timer;
	
   BufferedImage lastFrame;

    public VideoCamera() 
    {
    	System.loadLibrary(Core.NATIVE_LIBRARY_NAME);   	
        camera  = new VideoCapture();        
    }
    
    public void startCamera(String _url)
    {
    	this.url = _url;
		this.camera.open(this.url);
		
		// is the video stream available?
		if (isOpened())
		{
			// grab a frame every 33 ms (30 frames/sec)
			Runnable frameGrabber = new Runnable() {
				
				@Override
				public void run()
				{
					// effectively grab and process a single frame
					Mat frame = new Mat();
					camera.read(frame);
					lastFrame = Mat2BufferedImage(frame);
				}
			};
						
			this.timer = Executors.newSingleThreadScheduledExecutor();
			this.timer.scheduleAtFixedRate(frameGrabber, 0, 33, TimeUnit.MILLISECONDS);
		}
		else
		{
			// log the error
			System.err.println("Impossible to open the camera connection...");
		}
    }
    
    public boolean isOpened()
    {
    	return camera.isOpened();
    }
    
    public void resetURL(String _newURL) 
    { 	
    	url = _newURL;
    	if(isOpened())
    	{
    		camera.release();
    	}
    	if(_newURL == "0") 
    	{
    		camera.open(0);
    	}
    	else
    	{
    		camera.open(url);   		
    	}   	
    }
    
    public BufferedImage Mat2BufferedImage(Mat m)
    {
        int type = BufferedImage.TYPE_BYTE_GRAY;
        if (m.channels() > 1)
        {
            type = BufferedImage.TYPE_3BYTE_BGR;
        }
        int bufferSize = m.channels() * m.cols() * m.rows();
        byte[] b = new byte[bufferSize];
        m.get(0, 0, b); // get all the pixels
        BufferedImage img = new BufferedImage(m.cols(), m.rows(), type);
        final byte[] targetPixels = ((DataBufferByte) img.getRaster().getDataBuffer()).getData();
        System.arraycopy(b, 0, targetPixels, 0, b.length);
        return img;
    }
    
}

[Kevin.Herron]

You’re getting a NullPointerException in paintComponent method at OCRComponent.java:116.

You’re dereferencing a null value somewhere in that method. Maybe the cam variable, or maybe cam.lastFrame.

[daniel.garcia-redrue]

Yes, was that. Thanks a lot.
In addition, the System.loadLibrary(Core.NATIVE_LIBRARY_NAME); was not working. I used instead de System.load(“PATH_TO_MY_DLL”) and now I can connect to the camera.

[pturmel]

Thanks for doing this. I have plans to incorporate opencv in a module I’m working on, and this tip will be very useful for multiple platform support.

[daniel.garcia-redrue]

My module connects now to a camera but from the designer or from a client in the same host that the gateway. The problem is that developing for the designer/client scope as usual (or that I think) for a vision component there is no access to the dll (if the client does not have it in the same path). So I’m thinking in calling the System.load("…/opencv_java346.dll") using RPC. Is there any option to only have the dll in the gateway?

Thanks in advance

[pturmel]

Not really, if you need the images in the client. RPC will not make OpenCV itself work in the client, though it can return byte-encoded images for the client to display.

You’ll have to run your video process in the gateway (background task, perhaps) or install OpenCV in all of the affected clients.

[daniel.garcia-redrue]

Thanks for the tip @pturmel. I’ve moved my VideoCamera to the gateway scope (and left my component in the client) but I’m still having problems to launch a client in a different machine than the gateway. For my purpose, installing OpenCV in all the clients is not a feasible option. So I’ll focus on launching OpenCV as a background task in the gateway. How could I do this? My first option is to call the System.load(“File.dll”) in the startup() of the GatewayHook and then, each vision component will use a VideoCamera to get images from the camera, but I’m not sure if this is going to work. Is my runnable frameGrabber doing things correctly?

Thanks in advance again

[This is the last version of my VideoCamera class]

    public class VideoCamera extends JPanel
    {	
        String url;
        VideoCapture camera;
        LinkedList<BufferedImage> frameQueue = new LinkedList<BufferedImage>();
        BufferedImage lastFrame;
        ScheduledExecutorService timer;

        public VideoCamera(int _fps) 
        {   	 	
	        framesPerSecond = _fps;
	        camera  = new VideoCapture(); 
         }


public void startCamera(String _url)
{
	this.url = _url;
	/// Release current connection
	if(isOpened())
	{
		this.camera.release();
	}
	if(_url == "0")
	{
		this.camera.open(0);
	}
	else
	{
		this.camera.open(this.url);
	}  	
	// is the video stream available?
	if (isOpened())
	{
		// grab a frame every 33 ms (30 frames/sec)
		frameGrabber = new Thread() {			
			@Override
			public void run()
			{
				// effectively grab and process a single frame
				Mat frame = new Mat();
				camera.read(frame);
				lastFrame = Mat2BufferedImage(frame);
				frameQueue.add(lastFrame);
			}
		};					
		this.timer = Executors.newSingleThreadScheduledExecutor();
		this.timer.scheduleAtFixedRate(frameGrabber, 0, 1000/framesPerSecond, TimeUnit.MILLISECONDS);
	}
	else
	{
		// log the error
		System.err.println("Impossible to open the camera connection...at " + this.url);
	}
}


public void stopCamera()
{
	if(isOpened())
	{
		this.camera.release();
		if(frameGrabber != null)
		{
			frameGrabber.interrupt();
		}
	}
}

/// Returns if video stream is opened
public boolean isOpened()
{
	return camera.isOpened();
}

/// Returns the last frame read
public BufferedImage getLastFrame()
{
	if(frameQueue.size() > 0)
		return frameQueue.remove();
	else
		return lastFrame;
}

public void setFPS(int _fps)
{
	if(_fps > 0)
	{
		this.framesPerSecond = _fps;
		if(frameGrabber != null)
		{
			frameGrabber.interrupt();
		}
		this.timer.scheduleAtFixedRate(frameGrabber, 0, 1000/framesPerSecond, TimeUnit.MILLISECONDS);
	}
}

public BufferedImage Mat2BufferedImage(Mat m)
{
    int type = BufferedImage.TYPE_BYTE_GRAY;
    if (m.channels() > 1)
    {
        type = BufferedImage.TYPE_3BYTE_BGR;
    }
    int bufferSize = m.channels() * m.cols() * m.rows();
    byte[] b = new byte[bufferSize];
    m.get(0, 0, b); // get all the pixels
    BufferedImage img = new BufferedImage(m.cols(), m.rows(), type);      
    final byte[] targetPixels = ((DataBufferByte) img.getRaster().getDataBuffer()).getData();
    System.arraycopy(b, 0, targetPixels, 0, b.length);
    return img;
}

   
 }

[pturmel]

Your VideoCamera component won’t be able to hold any OpenCV data types (VideoCapture or otherwise), just native java image types. You’ll have to set up RPC for the VideoCamera component to request the latest image from the gateway on a regular pace. I would set up the OpenCV library in the gateway hook’s setup() method, but I wouldn’t start the capture thread until the first RPC call comes in.

FWIW, the module I’m writing delivers the image sequences from the gateway to the client as an MJPEG stream that the IP Camera Viewer component can display. I’m going to conditionally load OpenCV in the client from the module setup() hook if OpenCV is available there, but that won’t be a requirement for image stream display.

[daniel.garcia-redrue]

Hi again,
after a few weeks focused on other stuff, I've reorganized my code:

1. I've moved my VideoCamera class to the GatewayScope (this is the class that uses opencv to connect to the cameras)

2. I've kept my Component in the client scope (the component gets the last frame read by its videoCamera instance and paints it on a JPanel)

3. I load the openCV library in the GatewayHook

However, I have the same problem. When I launch a client in a different host than the gateway, the error says:

Cannot load library "..../opencv346.dll"

I don't understand why the clients need to load the library (that is located in the gateway host) or what different options I have.

Thanks in advance for the help...I really need it

[pturmel]

You must still be trying to use OpenCV classes in the client. You can’t, not even indirectly, unless you load OpenCV everywhere. You must convert the Mat images from your VideoCapture class to a java native image format or compressed bytes. This must happen in the gateway. Your client component has to work strictly with native java image formats.

FWIW, my module has progressed to the point where I can convert RTSP camera streams to MJPEG and send them to IP Camera viewers, with efficient multi-client delivery. It is a CPU hog, unfortunately.

[pturmel]

One other note: OpenCV is intrinsically NOT thread safe. Its java wrapper does not fix this for you. You must not pass OpenCV objects from one thread to another–you will leak memory in the java process outside the java heap. Yeah, figured that out first hand. /:

[daniel.garcia-redrue]

I’m passing the object BufferedImage lastFrame;
Maybe adding the dependency of gateway in the client in the pom file

       <dependency>
            <groupId>ignition.imas</groupId>
            <artifactId>cam-module-gateway</artifactId>
           <version>8.0.0-SNAPSHOT</version>
        </dependency>

tries to use openCV in the client

[pturmel]

I've announced a release candidate for my driver module...

Automation Professionals' Image Streamer Driver Module

Automation Professionals is pleased to announce a Release Candidate for its new driver module, supporting the live Network Image transmission format of Keyence CV-5000 machine vision controllers, plus support for common Network Video streams via OpenCV. Stream metadata is exposed as OPC items, and the output streams are Motion JPEG. The IP Camera Viewer Component in Vision can consume these streams, making it possible to integrate IP Cameras that don't have MJPEG support or have poor support f…

[daniel.garcia-redrue]

Thanks a lot for sharing!!
I´m right now doing some tests with the trial version, but I could´t connect with my Bosch DINION 5000 yet. These are my configurations

Am I doing something wrong?

On the other hand, the .modl file only install one type of device, is this right?

Thanks in advance

[pturmel]

Hmm. The location override isn’t right, and shouldn’t be needed at all if you’ve followed the instructions for installing FFmpeg and OpenCV.

Which you must not have, because it only installed the Keyence driver. You can’t use the Keyence driver to connect to an IP Camera. Please look in the gateway logs for “opencv” to find the error messages.

I also see that I messed up the internationalization properties for some location override fields. The “Machine” field is not for the machine name. It is “Gateway” for the server, or a MacID for a client (not yet implemented).

[pturmel]

An update to the release candidate has been posted to the other topic. Please report any further problems over there.