Gateway Script to monitor directory for new files

Ignition
1

Hi, I’m looking for some advice as to the best way to monitor a network directory location for new files. We have an external provider that FTP’s files into a specific directory on our network, and I would like the Ignition Gateway to monitor this directory and read in the new file as a string.

From that point, I have some XML parsing code written and tested, I just can’t figure out how to monitor and get new files read in.

(For completeness - once I have read and parsed the file I will move it to an Archived location)

I have a solution working on the client level using the Instrument Interface module - but since I have a mixture of Windows and Macs on site, that solution is not universal. That’s why I would rather my Gateway (Windows Server 2012) do the monitoring in the background even when the clients are off.

Any help would be much appreciated.

Cheers

Dan

1 Like
2

Hi Dan,

Here is a (severely) shortened script I currently use.

import os, shutil

inputPath = '\\\\192.168.132.11\\dswatch\\DT\\OUT\\'
archivePath = '\\\\192.168.132.11\\dswatch\\DT\\OUTARCHIVE\\'

files = os.listdir(inputPath)

if len(files) > 0:
  for file in files:  
    doSomeStuff
  for file in files:
    shutil.move(inputPath + file, archivePath + file)
3

To explain it a bit, in my full version all the files are processed for a single insertion into a database, then moved to the archive folder. depending on your workflow, you could move the file in the same loop.

4

Really depends on the nature of the file system/network. Jordan’s way is fairly simplistic, but might suit you if you are looking to run the script on a poll/timer and not needing to do anything too complex.

Otherwise, you might want to look at Java’s WatchService api, as it’s built for doing exactly what you are looking for. Benefit there is that you get to update things in a more ‘push’ or ‘reactive’ fashion as you’re notified of changes when the watch service detects them, rather than having to poll and compare results: you simply leave it up to the native filesystem to notify you of changes. You’d have to verify it works with network drives, but if so, seems like it would be a good fit.

1 Like
5

Thanks JordanCClark and PerryAJ, that simple code is exactly what I needed.

Works perfectly, at this stage I’m happy to poll through a timed script and keep it simple. Thanks for the advice regarding WatchService though - I might have another use for that elsewhere in my project.

Cheers
Dan

6

Here is some code that uses WatchService

from java.nio.file import FileSystems,Files
from java.io import File
from java.nio.file import StandardWatchEventKinds as swek
from java.util.concurrent import TimeUnit

class FileWatcher():
	isRunning = False
	def __init__(self):
		self.run()

	def _doAsync(self):
		self.isRunning = True
		watcher = FileSystems.getDefault().newWatchService()
		p = File("\\\\server\\share").toPath()
		key = p.register(watcher,swek.ENTRY_CREATE,swek.ENTRY_DELETE,swek.ENTRY_MODIFY)
		while self.isRunning:
			k = watcher.poll(5,TimeUnit.SECONDS)
			if k:
				for e in k.pollEvents():
					kind = e.kind()
					if kind == swek.ENTRY_DELETE:
						print p.resolve(e.context())
					if kind == swek.ENTRY_MODIFY:
						print p.resolve(e.context())
					elif kind == swek.ENTRY_CREATE:
						print p.resolve(e.context())
						print Files.probeContentType(p.resolve(e.context()))
				k.reset()
	def kill(self):
		self.isRunning = False
		
	def run(self):
		if not self.isRunning:
			system.util.invokeAsynchronous(self._doAsync)
if "fw" not in system.util.getGlobals():
	system.util.getGlobals()["fw"] = FileWatcher()

This will create a single watcher, and store a reference to it in globals. Use system.util.getGlobals()[“fw”].kill() to turn it off, and system.util.getGlobals()[“fw”].run() to turn it back on.

8 Likes
7

Hi - when wanting to use various Java tools, like the api here, or others - where do we have to install those to ensure they work? (I know I could pip install for example if in Python in general, but with Jython buried into Java Ignition I’m not sure where or how to put those other modules we’d like to use.)
Thanks
John

8

Most Java modules you’ll end up using will already be available in the relevant scope. To add third-party python modules, just add them to Ignition/user-lib/pylib in the installation directory on the gateway machine - we’ll automatically distribute the files/folders to clients so they can use the modules as well.

1 Like
9

ok - thank you. Is there a list of the ones that are already available, or is it just whatever is in the Jython 2.5 base library?

10

Hi,
I am new to most of this and so the minor aspects of implementation tend to go over my head. Like the OP, I want to utilize the WatchService to monitor a directory for new .csv files that are automatically generated by equipment. I have written project scripts that will parse and transfer the .csv to SQL but I want the Watcher to listen for new files and trigger that process. My questions are:

  1. For my understanding, this code instantiates a new variable of type FileWatcher in the Global scope. Upon construction, the init function is called automatically, which in turn initiates the Watcher’s listener loop. The listener function is called asynchronously so that it can safely run a continuous loop without preventing other processes. Is that accurate?

  2. If I put this code in a Gateway Startup Event Script, should I set the script to run on a dedicated thread? If so should I still call the Watcher loop asynchronously? In general, would this be a proper place to put this kind of script?

  3. If this script resides on the Gateway, is the Global scope shown here accessible from Clients? I had read that Memory Tags were the preferred method for scoping variables globally. (Globally scoped objects)

  4. Kind of a catch-all question: I need to allow users to register selected directories to the WatchService. After this code runs once in the Gateway, can Client scripts instantiate new FileWatcher variables, and pass directory paths to the constructor?

Thanks!

11
  1. It is in gateway scope, either project or global, depending on what is kicking it off. A gateway startup script would be project scope. The FileWatcher instance is stored in the gateway’s global variables.
  2. Not necessary to use a dedicated thread. It makes its own thread, and the startup event will end right away.
  3. There’s no sharing of memory between clients and gateway, even on the same machine. All communications is via tags or messages.
  4. FileWatcher instances created in a client would watch directories in the client machine. Use network shares to have clients and gateway “see” the same files.
1 Like
12

I placed Kyle's code in my project's Gateway Event Startup Script, but it is yielding the following error:

Traceback (most recent call last):
File "<[CSV_Test] Startup Script>", line 36, in
File "<[CSV_Test] Startup Script>", line 9, in init
File "<[CSV_Test] Startup Script>", line 34, in run
NameError: global name 'system' is not defined

I tried to troubleshoot but I don't feel I have enough knowledge to get anywhere.

13

What version of Ignition are you using?

14

I am using Version 7.9.7

I was able to add this at the top of the script and get logs out:
system.util.getGlobals()["logger"] = system.util.getLogger("myLogger")

15

That is weird that it isn’t working for you. What scope are you running the script from? I just tried in the designer and the gateway, from a tag event, and it worked fine.

For reference, here is the script I ran from a valueChanged event on a tag. It writes the results to a tag called [default]New File. Place this script on a boolean tag, and change the folder path in line 16.

	if currentValue.value:
		from java.nio.file import FileSystems,Files
		from java.io import File
		from java.nio.file import StandardWatchEventKinds as swek
		from java.util.concurrent import TimeUnit
		
		class FileWatcher():
			isRunning = False
			def __init__(self):
				self.run()
		
			def _doAsync(self):
				self.isRunning = True
				watcher = FileSystems.getDefault().newWatchService()
				p = File("C:\\Users\\kylec\\Downloads").toPath()

				key = p.register(watcher,swek.ENTRY_CREATE,swek.ENTRY_DELETE,swek.ENTRY_MODIFY)
				while self.isRunning:
					k = watcher.poll(5,TimeUnit.SECONDS)
					if k:
						for e in k.pollEvents():
							kind = e.kind()
							if kind == swek.ENTRY_DELETE:
								system.tag.write("[default]New File",p.resolve(e.context()).toString())
							if kind == swek.ENTRY_MODIFY:
								system.tag.write("[default]New File",p.resolve(e.context()).toString())
							elif kind == swek.ENTRY_CREATE:
								system.tag.write("[default]New File",p.resolve(e.context()).toString())
								print Files.probeContentType(p.resolve(e.context()))
						k.reset()
						
			def kill(self):
				self.isRunning = False
				
			def run(self):
				if not self.isRunning:
					system.util.invokeAsynchronous(self._doAsync)
					
		if "fw" not in system.util.getGlobals():
			system.util.getGlobals()["fw"] = FileWatcher()
	else:
		if "fw" in system.util.getGlobals():
			system.util.getGlobals()["fw"].kill()
			del system.util.getGlobals()["fw"]
1 Like
16

After troubleshooting I found that placing the script in either Gateway or Event scripts did not solve the problem. I placed a service call into Ignition technical support and received this solution: When creating a Class, ‘system’ must be imported within the function that needs to use it. The following example was placed in a Client Tag Change Event script, which was attached to a boolean tag.

if currentValue.value:
	class FileWatcher():
		def __init__(self):
			import system
			system.util.invokeAsynchronous(self._doAsync)
			
		def _doAsync(self):
			pass
	
	system.util.getGlobals()["fw"] = FileWatcher()
else:
	pass
1 Like
18

Gateway tag change scripts are subject to legacy scoping; you’ll need to either cascade system through to the functions namespace manually or use a tag event script.

1 Like
20

Or put all of the interesting code in a project script module, including the class definition. Then all will be defined with modern scoping. Then the event script just calls the project script.

2 Likes
21

Hi All,

I just wanted to update with the final code I used as my solution, in case it can help anyone. I implemented multiple watchers, modified polling to use WatchService.take for immediate notification of changes, and created Gateway Tag Change scripts to govern each watcher’s start and stop. This way they can be manipulated from clients if necessary by using:

system.tag.write('endWatchA', 1)
system.tag.write('startWatchA', 1)

The Gateway Shutdown script was necessary to prevent multiple watcher threads opening when Saving and Publishing changes from the Designer.

As a final note, the try/except block in the watch method should catch a ClosedWatchServiceException but I could not get Ignition to recognize this exception. I tried importing java.lang.Exception, and java.nio.file.ClosedWatchServiceException

Gateway Startup Event Script

project.watcher

####Project Script ‘Watcher’

from java.nio.file import FileSystems,Files
from java.io import File
from java.nio.file import StandardWatchEventKinds as swek
import system
system.util.getGlobals()['GWLog'] = system.util.getLogger("GatewayStartupEventLog")

class fileWatcher():
	def __init__(self, folderPath, type):
		self.folderPath = folderPath
		self.type = type
		self._run = False
		self.watcher = None
		system.util.getGlobals()['GWLog'].info('INIT %s' %self.type)
		self.run()
	
	def watch(self):	
		self.watcher = FileSystems.getDefault().newWatchService()
		p = File(self.folderPath).toPath()
		key = p.register(self.watcher, swek.ENTRY_CREATE)
		
		kill = False
		while self._run:
			try:
				k = self.watcher.take()
			except:
				kill = True
			if not kill:
				for e in k.pollEvents():
					if e.kind() == swek.OVERFLOW:
						pass
					if e.kind() == swek.ENTRY_CREATE:	
						#Do something with self.type & file path str(p.resolve(e.context()))
				k.reset()
		system.util.getGlobals()['GWLog'].info('END %s' %self.type)
	
	def run(self):
		system.util.getGlobals()['GWLog'].info('START %s' %self.type)
		if not self._run:
			system.util.getGlobals()['GWLog'].info('RUN %s' %self.type)
			self._run = True
			system.util.invokeAsynchronous(self.watch)
	
	def kill(self):
		system.util.getGlobals()['GWLog'].info('KILL %s' %self.type)
		self._run = False
		self.watcher.close()

system.util.getGlobals()['aWatch'] = fileWatcher('C:/Users/pathA', 1)
system.util.getGlobals()['bWatch'] = fileWatcher('C:/Users/pathB', 2)
system.util.getGlobals()['cWatch'] = fileWatcher('C:/Users/pathC', 3)

####Gateway Tag Change Script ‘endWatchA’ w/ Memory Tag ‘endWatchA’ as BOOLEAN

Repeat as necessary for endWatchB, endWatchC, etc.

if currentValue.value:
	system.util.getGlobals()['aWatch'].kill()
	system.tag.write('endWatchA', 0)

####Gateway Tag Change Script ‘startWatchA’ w/ Memory Tag ‘startWatchA’ as BOOLEAN

Repeat as necessary for startWatchB, startWatchC, etc.

if currentValue.value:
	system.util.getGlobals()['aWatch'].run()
	system.tag.write('startWatchA', 0)

####Gateway Shutdown Event Script

system.util.getGlobals()['aWatch'].kill()
system.util.getGlobals()['bWatch'].kill()
system.util.getGlobals()['cWatch'].kill()
2 Likes
22

I wanted to update with a solution for an obstacle I ran into when trying to open files identified by this watcher. I saw the following regardless of using WatchService.take() or WatchService.poll().

  • File paths are reliably collected and always immediately verifiable with os.path(path) returning True
  • Upon immediate inspection after a WatchService notification, some files return False from Files.isReadable(path) and will throw IOError[2] No such file if opened
  • After a short delay (i.e. 3 seconds) these files can be opened

I used this check to ensure every new file is eventually opened:

import os
from java.nio.file import Files, Paths
from time import sleep

pathCheck = os.path.isfile(path)
readCheck = Files.isReadable(Paths.get(path))
target = None

if pathCheck & readCheck:
	target = open(path, 'rb')
else:
	for i in range(5):
		sleep(3)
		try:
			target = open(path, 'rb')
			break
		except IOError, e:
			if e.errno == 2:
				pass
			else:
				raise
2 Likes