Communicating with external applications over TCP/IP socket /websockets using gateway scripts

How can we communicate Ignition with external applications over TCP/IP sockets or websockets using gateway Jython scripts,? Does the gateway scripts have libraries for TCP sockets or websockets communication? I don’t see it in python gateway script library documentation. Has anyone developed a module for it (either commercial or free module). The existing TCP/IP module in Ignition is specifically designed for bar code reader which is not generic in nature.

Can JAVA/Jython combination be used so that JAVA can be used for external interface using TCP/IP or websocket and Jython for communication within the gateway server?

regards

3 Likes

Are you calling GET / POST services?
I don't know if this addresses your question or not but...
for GET, I am using:

initialize variables

url = system.tag.read('[default]Config/MES/url').value
api = system.tag.read('[default]Config/MES/api').value
endPoint = url + api + 'machinecommunication/equipmentInfo/?equipmentId=%s' %equipmentId
accessToken = system.tag.read('[default]Config/MES/token').value
headerValues = {'Content-Type':'application/json','Authorization': 'Bearer %s'%accessToken}

call the MES API and convert the return value to a JSON document

try:
equipmentString = system.net.httpGet(endPoint, headerValues=headerValues)
equipmentString = equipmentString[0] + '"success":True,' + equipmentString[1:9999]
except IOError:
# manage IO Error

For POST, I found that the built in system.net.httpPost did not work with the service I'm calling so I use the urllib python libraries:

import Python libraries

import urllib
import urllib2
from urllib2 import URLError, HTTPError

initialize variables

accessToken = system.tag.read('Config/MES/token').value
url = system.tag.read('Config/MES/url').value
api = system.tag.read('Config/MES/api').value

construct endpoint and parameters

endPoint= url + api + 'machinecommunication/completeOperation'
postData= {'equipmentId':equipmentId}
header = {'Authorization': 'Bearer %s'%accessToken}
data = urllib.urlencode(postData)

call the MES API and convert the return value to a JSON document

try:
apiRequest = urllib2.Request(endPoint, data, header)
apiResponse = urllib2.urlopen(apiRequest)
mesResponse = apiResponse.read()
apiResponse.close() # optional: actively close the connection vs. wait for garbage collection. JIRA task MES 1603
jsonResponse= system.util.jsonDecode(qctResponse)
except HTTPError, he:
# manage HTTP Errors
jsonResponse = {'success':False,'error':'HTTP Error %s' %str(he)}
except URLError, ue: # next version of Jython recommends using "URLError as "
# manage URL Errors
jsonResponse = {'success':False,'error':URL Error %s' %str(ue)}
except IOError, ioe:
# manage IO Errors
jsonResponse = {'success':False,'error':'IO Error %s' %str(ioe)}
except:
# manage miscellaneous errors
jsonResponse = {'success':False,'error':'Miscellaneous Error: Not HTTPError or URLError or IOError'}

No, the GET/POST script function is REST api for getting value from a specified URL on an external server.

What I want is an external application communicating with Ignition gateway with over a TCP in request/response mode using Jython scripts on ignition server.

I think someone had told ( think @Philtrman ) that its possible to call a java code from a Python script. Then I can use JAVA application to communicate with my external application on TCP socket and use Python for back-end communication with Ignition server.

If a sample code can be uploaded, explaining how Jython and be used to within JAVA or vise/versa I will be grateful.

Sure, we use it all the time where Ignition is the client. Use it to connect to devices, MES, printers.... Acting as a client is easy.... server is a bit harder. Our MES is all XML over TCP and we have load tested with millions of transactions per day with no issues.

Super simple send only in Jython using Java TCP

from java.net import Socket, InetSocketAddress
from java.io import DataOutputStream
def printZPL(zpl, ip):
	
	zpl = zpl.encode('utf-8')
	port = 9100

	try:
		printer = Socket()
		printer.connect(InetSocketAddress(ip, port), 2000)
		outPrinter = DataOutputStream(printer.getOutputStream())
		outPrinter.write(zpl)
		outPrinter.close()
		printer.close()
		
		return True
	except IOError:
		return False

Send/Recieve XML over TCP Socket in Jython

import socket, xml.etree.cElementTree as ET, xmltodict


def connect(server):
	"""
	connect : Create a socket and connect
	"""
	
	# Create a python socket
	sock = socket.socket()
	
	# Set the timeout to something reasonable so our program does not lockup
	sock.settimeout(5)
	
	# Connect
	server_address = (server, 1000)
	sock.connect(server_address)
	
	# Return the socket for further use
	return sock
	

def recv(sock):
	"""
	recv : Recieve the response
	"""
	
	# Setup a var to hold the return data
	total_data=[]
	
	# Recieve the data until no more data is returned
	sock.settimeout(45)
	while True:
		data = sock.recv(4096)
		if not data: break
		total_data.append(data)
		
	# Join all the data recieved into a single string
	resp = ''.join(total_data)
		
	# Return the response
	return resp
	
def submit(xml, ip='127.0.0.1', raw_resp=0):
	"""
	submit : Submit an XML request
	"""
	
	# Connect 
	try:
		ets = connect(ip)
		logger.trace("Connecting to %s" % ip)
	except:
		return formatResp( xmlResp(0, "Failed to create a socket to %s.", ip))
	
	# Convert the XML to a string
	requestXML = ET.tostring(xml, 'UTF-8')
	
	# Send the XML string to the open socket
	ets.sendall(requestXML)
	
	# Recieve the response
	try:
		responseXML = recv(ets)
		logger.trace("Got response from %s, %s" % (ip, responseXML))
	except Exception, e:
		return formatResp( xmlResp(0, "Failed to recieve response from socket. %s" % e, ip))
	
	# Close the connection
	ets.close()

	# Format the response
	response = formatResp(responseXML)
	
	# Return the response data
	return response

4 Likes

Thanks a lot. However i have a few questions:

  1. Is there a limitation on size of the data (the XML in your case) being sent/received over TCP? Generally TCP is not reliable for large packet size. Websockets are more reliable. What’s you opinion?

  2. Your external application (MES in your case) resides on a separate machine or on localhost (you have given ip address as ‘127.0.0.1’).

  3. In the above example, assume Ignition is the TCP server responding to requests from MES which a socket client , requesting XML data from server. Hope the socket client can be written in any language (java or JavaScript etc?).

regards

TCP has been around for decades... websockets is new, integrating to many legacy or simple devices, webscokets is just not going to happen. I don't worry about size limitations, I have sent many megabytes per request and never have issues.

Ignition is the client in these examples, the other end is the server (listener). As I said, client in Ignition is very easy. Were connecting to pre-existing systems, nothing custom on the other end, just whatever is provided by the vendor.

1 Like

So if I understand it correctly, this code is used inside an Ignition client script or gateway script to request data from an external TCP socket server (some third party application such as MES or bar-code reader etc) by calling the submit(…) function , giving appropriate parameters for xml, ip and raw_resp parameters and the application returns an XML response. (the raw_resp parameter is not used in code).

I am not 100% clear about the loop

while True:
	data = sock.recv(4096)
	if not data: break
	total_data.append(data)

Presume you are receiving data in chunks of 4096 byte from server and concatenating them in to a single string. Correct me if I am wrong. Hope this continuous re-initialization of total_data will not cause memory leaks (not sure hoe garbage collection in Python works).

My requirement is slightly different. My Ignition side script should be master (or TCP server) waiting on a thread for connection from a TCP client which is my third party application. My application connects to this server on start and sends requests to the TCP server (which is a script in Ignition on a thread). So the roles of client/server are reversed compared to your example.

What is the difficulty in making Ignition side script a TCP server, the client could be a non Python application. Can you suggest sample code for server side implementation of TCP socket in Jython?

regards
PRAMANJ

This approach is reasonable. However, I strongly suggest you not use the jython sock library. Jython's garbage collection is simply java's garbage collection, but python's byte-string conversions are less than ideal. There will always be confusion and heartache over processing of non-ascii content. Use java's native networking streams and process everything possible as bytes. Use bytearray streams and/or bytebuffer types to hold them. Go straight from streams to XML and back to streams with java's native parser and converter.

Read up on and follow examples for java networking, but implement in jython. Consider using two background threads per socket: One to open/connect the socket, establish an outbound message queue, and then repeatedly read & block on input. As complete messages are received as bytes, convert to some form of message object and dispatch to interested parties (I generally use a dataset tag to hold the latest message of a given type). Use the second thread to dequeue outbound messages and write the bytes to the socket.

The above works for both client and server situations, based on how the socket is opened.

No, I don't have an example I can share at the moment, but I do use this approach in multiple customer sites.

3 Likes

I have this communication between Ignition server and my application written as module in JAVA using TCP sockets, However as far as the API’s for SQLTagManager in JAVA are bit cumbersome whereas Jython tag-libraries are quite easy to use. Hence I am trying to try the Python approach for the connection between Ignition & module as well.

But as you said I should try best of both worlds i.e. JAVA for socket communication and Python for back-end integration with tags etc. Its the interface between JAVA & Python that that I am lost. I am trying to read some documentation at http://www.jython.org/jythonbook/en/1.0/JythonAndJavaIntegration.html correct me if am on right track.

Just call all the java stuff from jython. Use jython’s NetBeans shorthand where you can.
Something like this (ultra-simplified):

from java.net import InetSocketAddress
from java.nio import ByteBuffer
from java.nio.channels import SocketChannel
from java.nio.charset import StandardCharsets

def opener(target):
	parts = target.rsplit(':', 1)
	if len(parts)>1:
		print "Socket Address %s:%s" % (parts[0], parts[1])
		sa = InetSocketAddress(parts[0], int(parts[1]))
	else:
		print "Socket Address %s Default Port 1234" % parts[0]
		sa = InetSocketAddress(parts[0], 1234)
	ch=SocketChannel.open()
	ch.connect(sa)
	# Save channel somewhere for the write thread
	# Set up a message queue and start writer.
	shared.later.callAsync(writer, ch, mq)
	bb=ByteBuffer.allocate(4096)
	while True:
		ch.read(bb)
		# extract and dispatch message(s) or loop for more

def writer(ch, mq):
	while True:
		msg = mq.pop()
		bb = StandardCharsets.UTF_8.encode(msg + "\r\n")
		ch.write(bb)

You’ll have to work out blocking vs. nonblocking as needed to find message boundaries, and possibly have some form of state machine or request tracking map to match up requests and replies. Some things you might want to do will be simpler with Streams instead of ByteBuffers. You might also want to use java.nio instead of java.net.

1 Like

FWIW, I try to use jython to prototype algorithms for which I intend to write a module. It’s slower, and more cumbersome for a variety of interfaces, but it sure is easy to edit and test. One must simply use only java imports and then the jython is trivially rewritable as java.

2 Likes

So in other words, python is suitable for quick prototyping and testing an algorithm or a concept but may not be recommendable for serious long term deployment for the sake of efficiency. In that case I will continue to perfect my JAVA based module which uses the Ignition SDK.

My only worry with that approach was that it becomes Ignition version dependent and also needs a good understanding of the SqlTags API’s of gateway otherwise we run the risk of memory leaks. I am paranoid about memory leaks, as SCADA is a perpetually running application unlike other business applications which run on user inputs only. I wonder how Ignition is able to achieve this feat, it never fails!

Also I wonder how the Ignition clients communicate with gateway server! They say its HTTP but not sure if the entire SQLTags are sent as XML or only the tags used by Client are sent over HTTP per client or how is it? I believe with the advent of perspective , this is significantly going to change for better architecture and efficiency. Hopefully it will allow us to access the SQLTags received by the components on perspective clients so that we can directly consume them in out HTML pages in client.

Until then and for compatibility with classic vision module clients , we have to use our own workaround like TCP sockets etc.

No, that's a bit too harsh a judgement. Jython is perfectly fine if you don't need a module in the end, and the performance hit doesn't matter. Or you can mitigate the performance hit with careful avoidance of runtime code definitions. (So jython functions can can be cached and optimized.)

That's not entirely true. Most Ignition APIs are stable. For example, my NoteChart module is compiled against 7.7.8 and that one .modl file runs on that, 7.8.3+, and 7.9.x. The maven build system is over-constrained for this purpose, though, which is why I've put together my own Ant build.

Jython can leak memory too, particularly on scripting restarts. It's a big part of why I discourage use of shared.* script modules for all but the most standardized and stable scripting classes and functions.
Jython simply uses Java's garbage collection, so there's no significant difference in garbage collection. You simply must remove objects from persistent collections when you are done with them. That's basically it. There's no magic bullet in Jython that prevents memory leaks. Java has debugging tools that help you profile memory usage -- that's a big help finding leaks.

It is polling http(s). Clients subscribe to tags, and the gateway notes changes to those tags. On each poll, the client picks up just the tags that 1) are subscribed, and 2) changed. The tag poll also picks up any queued messages from the sendMessage() function. Other client interactions with the gateway are http(s) request/response. Client payloads are not XML -- they are java serialized binary.
Perspective uses websockets instead of polling http(s), so tag changes and messages can be pushed immediately to subscribers. I understand this is coming to the Vision module as well, eventually. Perspective's payloads, after the initial React page and script load, is encoded json. I presume Vision would continue using java serialization.

Neither the Vision client payloads nor the Perspective client payloads are expected to be usable outside Vision and Perspective, respectively. Perspective is extensible by a module to make new components, but those components are built on perspective resources, not your own web page designs. While perspective api endpoints (json) will be technically accessible outside perspective, expect difficulty authenticating to use them that way. Plan on using the WebDev module or your own module to expose Ignition internal data to generic consumers.

3 Likes

Thanks for the insightful discussion.

Its not to underestimate the potential of what advantages jython can bring to you.

I understand, if we are using some core API's that didn't undergo a change over various versions, I guess they can just work across higher versions without any change.

I think the websocket is a better model than HTTP as HTTP was only designed for pull request from clients, there was no way server could push data to client except for AJAX requests, whereas Websockets can push data from server to clients and is more suited for real time applications..

That's sad, but its understandable but I still hope Perspective will have more flexibility and openness towards third party components.than the vision module. The problem with our own modules accessing the Ignition tags data is that we need to also take care of access protection based on login credentials.

Let wait and watch.

regards

For people that:

  • Want to send a message over TCP/IP and do something with the response
  • No xml or any parsing
  • Use Java libraries

I have some code for you that worked for me:

# Import the necessary Java classes
from java.net import Socket, InetAddress
from java.io import BufferedReader, InputStreamReader, PrintWriter

# Set the target host and port
host = "localhost"
port = 8080

# Create a socket connection
socket = Socket(InetAddress.getByName(host), port)

# Create input and output streams for the socket
input_stream = BufferedReader(InputStreamReader(socket.getInputStream()))
output_stream = PrintWriter(socket.getOutputStream(), True)

# Send the message
message = "Hello, server!"
output_stream.println(message)

# Wait for the response
response = input_stream.readLine()
print("Received response:", response)

# Close the socket
socket.close()
2 Likes

Do note that such blocking code should only be run in an asynchronous thread, never in or called from any event script.

2 Likes

Thanks for you comment Phil, this code does the same with separate threads (With some ChatGPT help):

from java.lang import Thread
from java.net import Socket, InetAddress
from java.io import BufferedReader, InputStreamReader, PrintWriter

# Define a class that extends Thread
class TCPSender(Thread):
	def __init__(self, host, port, message):
		Thread.__init__(self)
		self.host = host
		self.port = port
		self.message = message
		self.response = None
	def run(self):
		try:
            # Create a socket connection
			socket = Socket(InetAddress.getByName(self.host), self.port)

            # Create input and output streams for the socket
			input_stream = BufferedReader(InputStreamReader(socket.getInputStream()))
			output_stream = PrintWriter(socket.getOutputStream(), True)

            # Send the message
			output_stream.println(self.message)

			# Wait for the response
			self.response = input_stream.readLine()
			
            # Close the socket
			socket.close()			
		except Exception as e:
			print("Exception occurred:", str(e))

def sendSynchroneMessage(message, ip, port = 23):
	# Create an instance of the TCPSender thread
	tcp_sender = TCPSender(ip, port, message)
	
	# Start the thread
	tcp_sender.start()
	# Wait for the thread to finish
	tcp_sender.join()
	# Access the response received from the thread
	response = tcp_sender.response
	return response

sendSynchroneMessage("Hello", "localhost", port = 23)
1 Like

No need to subclass Thread. Just use system.util.invokeAsynchronous() to launch the function that performs those (potentially) blocking operations.

Apologies about hijacking the thread,

I am trying to implement an RFID reader directly with ignition for inventory tracking and wondering if this would do the trick!? perhaps I should create my own thread but I will try here first..

The Manual: https://media.ifm.com/CIP/mediadelivery/asset/78a40ca61e68ba4d86e501054576f955/11430783_EN.pdf?contentdisposition=inline

I see tcp/ip as the communication protocol in page 5 of your manual, but I think in your case you will want to add it as a tcp/ip device in Ignition so you can see each scan coming in via tags.