COGNEX Insight Images and Overlays on Paintable Canvas

There are several posts about displaying Cognex images and overlays on the ignition screen. There are a few people selling modules and there is the popular web module that can display the SVG itself. We have used the web module in the past and it sort of works for what we want. I don’t like it because its another module that must be purchased and we’ve had some flaky stuff happen on Ignition upgrades.

I thought I would share my current approach and maybe it helps some others that might be asking the same question down the road.

First, the application: We have a new machine that has 4 Cognex Insight 7000 series cameras doing something undisclosed for IP reasons. To help support the remote site that will be running this new machine, we would like to store the images for a period of time before removing them, along with key information used to run the part (ID, offsets, angles, and other values from the PLC and Cognex cameras). We would also like to display the images to the operator on the HMI.

Next, the setup: We are running an CompactLogix PLC with a PC running a 2 client license of Ignition (there other reasons why we did not use the sites main Ignition server). On this PC we also have a SQL express instance running (also, other reasons why we chose not to use the sites SQL Cluster). Essentially, we chose to have this machine entirely stand-alone.PC also has IIS FTP setup or Filezilla (both tested and both work, I like IIS because i don’t like any more 3rd party software than I must have).

The operation:

  • As parts enter the machine Cognex Insight cameras aquire an image and pass variables to the PLC via Ethernet/IP.

  • The camera also uploads a copy of the image and overlay data to an FTP.

  • Gateway script runs every 500 ms to check FTP location and if file is present, inserts the bytes into a SQL table. I chose to use a SQL table to hold the binary data because the files are relatively small less than 1MB and I can hold a lot of them before it makes a noticeable impact on the SQL database size. I also feel it is easier to manage the records this way that in a file system. One quick Delete statement and I can clear out all the old records. I chose to do this as a gateway script because whether the client is open or not, the images are being stored.

  • In addition, the SVG file is stored in plain text in a varbinary(max) field in the same record with the image. The DB is also storing date-time info, flags for retaining images (preventing deletion), etc.

  • The paintable canvas has a custom property that has the current picture ID from the DB. The overview page that has the Paintable Canvas has a timer that runs every 500ms to check to see if there is a new record in the database. If a new record exists, the ID is updated and a call to a new extension function called “Draw” on the paintable canvas is called. Yes, I could have the GW script update a gateway tag that would trigger a Draw to remove the timer, and I might still, but its working for now.

  • The Draw function on the PC loads the image into the cache-img and also takes the SVG data in the DB and parses it into a dataset custom property on the PC, called Overlay Data. Another post made reference to how big the SVG standard was and maybe while true, Cognex utilized only a few primitive shapes from the SVG standard and they do it very consistently. The code below will not load any SVG, but should (and is for me) working with Cognex. SVG files are xml-type file that hold the overlay information. I chose to use the elementTree libraries and regular expression libraries that are included in Ignition to parse the SVG files.

  • Note: With Cognex version 4.8 and higher, the overlay graphics cannot be saved in the same file as the image. One major reason for this is that if you want to store the image and re-run it on an emulator offline to diagnose problems, you want the raw image without the pesky overlay messing up the evaluation, hence the reason to have the raw image and all the overlay graphics separate.

  • When the PC has it’s repaint event triggered, it reloads the cached image and the OverlayData custom property and draws those images. It was very important to not have any SQL calls or files being read as the repaint event needs to happen very quickly. Use of the cached image and the overlay dataset minimize the time required to process the repaint event.

I hope this helps to anyone thinking of trying the same thing. I like this approach because it means i do not have to purchase and maintain an extra module or 3rd party module and it uses native functions and Java 2D graphics. It also gives the user a little more control over how and what are displayed, but it does require the developer to understand a little more about how things like SVG graphics and Java 2D graphics work.

Anyone can feel free to roast me if they see an issue with the solution or if you have a potential improvement, please pass it along.

With the pervasive nature of Vision in industrial applications and with Cognex being the largest global manufacturer of machine vision, I think it would be nice to see a component from Ignition to display and interact with these devices. The solutions out in industry to allow the operators and technicians view images in realtime leave a lot to be desired and require having a computer hooked up running their programming software or purchasing an expensive monitor like the Cognex VisionView appliance or the add-on for a Rockwell panelview (where the user has to exit the Rockwell software to windows CE and launch the Cognex software). It creates a muh richer and meaningful user experience if the images and data from the Cognex cameras are able to be displayed in an Ignition window and images stored for troubleshooting.

This method also opens the opportunity to do some crazy stuff in the future, like the vision fails to find a feature, the user could click on the paintable canvas and the X,Y location can be transformed back to camera coordinates and used as an alternate method to find a pattern.

Draw extension function on Paintable Canvas:

def Draw(self):
def Run(s=self):
		s.Refresh = s.Refresh +1
		
	def checkAttrib (attributes,strname):
		val = None
		if strname in attributes.keys():
			val = attributes[strname]
		return val
	
	
	def parseSVG (data, tag, attrib,text):
		tag = tag.lower().strip()
	#	print attrib
		if tag == 'text' or tag == 'circle' or tag == 'line' or  tag == 'rect' or tag == 'path' or tag == 'polyline':
			x = checkAttrib(attrib,'x')
			y = checkAttrib(attrib,'y')
			stroke=checkAttrib(attrib,'stroke')
			x1 = checkAttrib(attrib,'x1')
			y1 = checkAttrib(attrib,'y1')
			x2 = checkAttrib(attrib,'x2')
			y2 = checkAttrib(attrib,'y2')
			h = checkAttrib(attrib,'height')
			w = checkAttrib(attrib,'width')
			fill = checkAttrib(attrib,'fill')
			StrokeW = checkAttrib(attrib,'stroke-width')
			d = checkAttrib(attrib,'d')
			r= checkAttrib(attrib,'r')
			cx= checkAttrib(attrib,'cx')
			cy= checkAttrib(attrib,'cy')
			transform = checkAttrib(attrib,'transform')
			style = checkAttrib(attrib,'style')
			id = checkAttrib(attrib,'id')
			points = checkAttrib(attrib,'points')
			
			if text == None or text == '':
				text = None
			
			
			font = None
			fontsize = None
			
			if tag == 'text':
				params = style.split(";")
				for param in params:
					lstparam = param.split(":")
					if lstparam[0] == "fill":
						stroke = lstparam[1]
					elif lstparam == "font-family":
						font = lstparam[1]
					elif lstparam[0] == "font-size":
						fontsize = int(lstparam[1].replace("pt","").strip())
						
			row = [tag,x,y,x1,y1,x2,y2,h,w,StrokeW,stroke,d,fill,r,cx,cy,transform,style,id,points,text,font,fontsize]
			data.append(row)
		
		return data
					
		
	from java.io import ByteArrayInputStream
	from javax.imageio import ImageIO
	from java.awt import Image
	from java.awt.image import BufferedImage
	from java.awt import Color
	
	print "Draw"
	canvas = self

	Error = 0
	image = system.db.runScalarPrepQuery("select image from images where image_id = ?",[self.ID])

	
	if Error == 0 and image <> None:
		#set canvas properties
		bais = ByteArrayInputStream(image)
		bImageFromConvert = ImageIO.read(bais)
		canvas.putClientProperty("img-cached",bImageFromConvert)
		
		
		ID = self.ID
		overlay = system.db.runScalarPrepQuery("select SVG_Overlay from images where image_id = ?",[ID])
		
		headers = ["tag","x","y","x1","y1","x2","y2","h","w","StrokeW","stroke","d","fill","r","cx","cy","transform","style","id","points","text","font","fontsize"]
		data=[]
		
		
		import xml.etree.ElementTree as ET
		root = ET.fromstring(overlay)
		
		#for elem in root:
		#	print elem
		
		for child in root:
			tag = child.tag[child.tag.find("}")+1:]
			data = parseSVG (data, tag, child.attrib,child.text)
			for x in child:
				tag = x.tag[x.tag.find("}")+1:]
				data= parseSVG (data, tag, x.attrib,x.text)
				
					
		SVGData = system.dataset.toDataSet(headers, data)
		self.OverlayData=SVGData
		system.util.invokeLater(Run,100)
	else:
		image = BufferedImage(300,300,BufferedImage.TYPE_BYTE_GRAY)
		graphics = image.createGraphics()
		graphics.setPaint(Color( 255,255,255))
		graphics.fillRect ( 0, 0, image.getWidth(), image.getHeight() )
		
		canvas.putClientProperty("img-cached",image)
		
		headers = ["tag","x","y","x1","y1","x2","y2","h","w","StrokeW","stroke","d","fill","r","cx","cy","transform","style","id","points","text","font","fontsize"]
		data=[]
		SVGData = system.dataset.toDataSet(headers, data)
		self.OverlayData=SVGData
		system.util.invokeLater(Run,100)
		
		

Repaint event on Paintable Canvas

from java.awt import Color
from java.awt import GradientPaint
from java.awt.geom import GeneralPath
from java.awt.geom import Rectangle2D
from java.awt.geom import Ellipse2D	
from java.io import ByteArrayInputStream
from javax.imageio import ImageIO
from java.awt import Image
from  java.awt import RenderingHints
from java.awt.image import BufferedImage
from java.lang import Math
from java.awt.geom import AffineTransform
from java.awt.geom import Line2D
from java.awt import BasicStroke 
from java.awt import Font
import re

						
def getColor(strColor):
	import system
	try:
		h = strColor.lstrip('#')
		RGBColor = tuple(int(h[i:i+2], 16) for i in (0, 2, 4))
	except:
		RGBColor = (0,0,255)
	return system.gui.color(RGBColor[0],RGBColor[1],RGBColor[2])


image = event.source.getClientProperty("img-cached")

print "repaint"

if image <> None:
	
	g = event.graphics
	#g.setRenderingHint(RenderingHints.KEY_ANTIALIASING,RenderingHints.VALUE_ANTIALIAS_ON)
	
	#Get details of paintable canvas
	cwidth = event.source.getWidth() 
	cheight = event.source.getHeight()
	

	scale = AffineTransform()

	
	#Get details of paintable canvas
	iwidth = image.getWidth()
	iheight = image.getHeight()


	bounds = Rectangle2D.Float(0,0,iwidth,iheight)	
	boundsNew=scale.createTransformedShape(bounds).getBounds2D()

	
	xr = cwidth/boundsNew.getWidth()
	yr = cheight/boundsNew.getHeight()
	
	if xr<yr:
		rr = xr
	else:
		rr=yr
	if rr> 1.00:
		rr = 1.00
	
	scale.scale(rr,rr)

	boundsNew=scale.createTransformedShape(bounds).getBounds2D()

	yo = (cheight -boundsNew.getHeight())/2 
	xo =  (cwidth -boundsNew.getWidth())/2 
	
	print xo,yo
	
	t = AffineTransform()
	t.translate(xo,yo)
	t.concatenate(scale)

	g.drawImage(image,t,event.source)
	
	print t
	############### DRAW SVG DATA
	SVGData = event.source.OverlayData
	g.setTransform(t)
	line = Line2D.Float(0,0,50,50)
	g.setStroke(BasicStroke(10))
	g.draw(line)
	
	for row in range(SVGData.rowCount):
		if SVGData.getValueAt(row,"tag") == 'line':
			g.setColor(getColor(SVGData.getValueAt(row,"stroke")))
			x1 = float(SVGData.getValueAt(row,"x1"))
			y1 = float(SVGData.getValueAt(row,"y1"))
			x2 = float(SVGData.getValueAt(row,"x2"))
			y2 = float(SVGData.getValueAt(row,"y2"))
			line = Line2D.Float(x1,y1,x2,y2)
			g.setStroke(BasicStroke(int(20)))
			g.draw(line)
		if SVGData.getValueAt(row,"tag") == 'circle':
			g.setColor(getColor(SVGData.getValueAt(row,"stroke")))
			x = float(SVGData.getValueAt(row,"cx"))
			y = float(SVGData.getValueAt(row,"cy"))
			r = float(SVGData.getValueAt(row,"r"))
			
			circle = Ellipse2D.Float(x-r,y-r,r+r,r+r)
			g.setStroke(BasicStroke(int(SVGData.getValueAt(row,"StrokeW"))))
			g.draw(circle)
		if SVGData.getValueAt(row,"tag") == 'text':
			g.setColor(getColor(SVGData.getValueAt(row,"stroke")))
			g.setFont(Font("Ariel", Font.PLAIN, int(24)))
			x = float(SVGData.getValueAt(row,"x"))
			y = float(SVGData.getValueAt(row,"y"))
			text = SVGData.getValueAt(row,"text")

			g.drawString(text,x,y)
			
		if SVGData.getValueAt(row,"tag") == 'polyline':
			g.setColor(getColor(SVGData.getValueAt(row,"stroke")))
			g.setStroke(BasicStroke(1))
			
			# Get Transform and Rotate
			
			TR = SVGData.getValueAt(row,"transform").strip()
			loc = TR.find("rotate")
			
			Transform = TR[:loc].strip().replace('translate(','').replace(')','')
			Rotate = float(TR[loc:].strip().replace('rotate(','').replace(')',''))

			Tx = float(Transform.split(' ')[0].strip())
			Ty = float(Transform.split(' ')[1].strip())
			
			t2 = AffineTransform(t)
			
			t2.rotate(Math.toRadians(float(Rotate)),Tx,Ty)
			t2.translate(Tx,Ty)
			
			g.setTransform(t2)
			
			poly = GeneralPath()
			points = SVGData.getValueAt(row,"points")
			i=0
			for j in points.split(' '):
				t4 = j.split(',')
				
				if i == 0:
					poly.moveTo(int(t4[0]),int(t4[1]))
				else:
					poly.lineTo(int(t4[0]),int(t4[1]))
				i+=1

			poly.closePath()
			
			if SVGData.getValueAt(row,"fill") != None:
				g.setColor(getColor(SVGData.getValueAt(row,"fill")))
				g.fill(poly)
				g.setColor(getColor(SVGData.getValueAt(row,"stroke")))
			
			g.draw(poly)
			
			g.setTransform(t)
		if SVGData.getValueAt(row,"tag") == 'path':
			g.setColor(getColor(SVGData.getValueAt(row,"stroke")))
			g.setStroke(BasicStroke(int(SVGData.getValueAt(row,"StrokeW"))))
			
			
			poly = GeneralPath()
			d = SVGData.getValueAt(row,"d").strip()
			
			subd = re.findall(r"[a-zA-Z][ ]*[0-9]*[.]*[0-9]*[,]*[0-9?]*[.]*[0-9]*",d)
			
			
			for cnt in subd:
				cmd = cnt.strip()[0]
				
				if cmd == 'M':	
					points = re.findall(r"[0-9]*[.]*[0-9]",cnt)
					poly.moveTo(float(points[0]),float(points[1]))
					
				if cmd == 'L':	
					points = re.findall(r"[0-9]*[.]*[0-9]",cnt)
					poly.lineTo(float(points[0]),float(points[1]))
					
				
				if cmd == 'z' or cmd == 'Z':
					poly.closePath()
			
			if SVGData.getValueAt(row,"fill") != None:
				g.setColor(getColor(SVGData.getValueAt(row,"fill")))
				g.fill(poly)
				g.setColor(getColor(SVGData.getValueAt(row,"stroke")))
			
			g.draw(poly)
	
	TF = AffineTransform()
	g.setTransform(TF)
	line = Line2D.Float(0,30,50,50)
	g.setColor(system.gui.color(0,0,255))
	g.setStroke(BasicStroke(1))
	g.draw(line)
9 Likes

Thanks for posting. This is encouraging as I am trying to do much the same thing. I’m using Perspective so I have the added issue of running a webserver and putting images where the browser can read them. I’ve tried using IIS but coordinating permissions to have the Cognex drop images in via ftp to an IIS ftp server, and then have an IIS webserver running to serve the images to the browser in Ignition has been tedious. Any suggestions on making this image shuffle easier are welcome. I would also like to connect to a live image on the browser but that’s proving to be even more of a hassle. I’m at the “what you’re doing should work” stage with Cognex support but it’s not.

On some of the newer Cognex cameras (or maybe firmware), they have a web server built in that you can configure. We have switched to this method as you can train patterns and adjust acquisition settings through the web interface. We simply have a new frame on the page and point it to the camera’s web server. It is much cleaner and more flexible.

Here is a link

I think your link may have been removed.
I’m doing the same thing but can’t seem to get it working. I’m getting the following error:

Access to 10.2.12.92 was denied
You don’t have the user rights to view this page
HTTP ERROR 403

I’ve sent Cognex screen shots of my setup including:

Firmware rev (6.01.00 (305))
User Access Settings
Web HMI Settings (I think this is the feature you are referring to)

And so far I’ve only received the frustrating reply, “Yes, that should work, maybe redownload the firmware”.

Did you have any hiccups in setting yours up or did it just work as advertised?
Camera I’m testing with is IS7902C

The 7900 series camera should work just fine out of the box. Its actually one of my staff that did the project recently. I’ll ask him to reply to this thread with more details, he is a vision engineer that works with the cameras all the time. I just know enough to dangerous.

From what I know, it was pretty straight forward. I’ve see the HMI developed with Vision and it works pretty well. We have not tried perspective, but being a web HMI from Cognex, I cannot imagine its a problem to use with perspective