Ignition jython, using import for script console compatibility

Hello,

I was developing in Ignition 8.1.38 an object-oriented function in Jython and noticed that importing project modules resolves some seemingly random problems that only occur when testing from the Script Console.

I'd like to ask if you see any issues with leaving these imports enabled in the modules, as it appears to work the same way in the Designer's scripting console and in the Gateway.

Apart from the fact that when Ignition detects changes in the modules, it recompiles and restarts the script instance and everything starts from scratch.

Here's a simple example of what I've observed:
I have a package called modules with the modules modBase, mod1, and mod2.
mod1 and mod2 are identical and access modBase functionalities.

# modBase code
myInternalVar = 0

class myClass():
	pass

def incAndGet():
	modules.modBase.myInternalVar += 1
	return modules.modBase.myInternalVar

logger = system.util.getLogger("TEST")
logger.info("modules.modBase loaded")

# mod1 and mod2 code
#import modules

def incAndGet():
	return modules.modBase.incAndGet()

def getClass():
	return modules.modBase.myClass

# Test code, tested in Script Console and Gateway Event Handler
logger = system.util.getLogger("TEST")


logger.info("mod1.increment and get: " + str(modules.mod1.incAndGet()))
logger.info("mod2.increment and get: " + str(modules.mod2.incAndGet()))


logger.info("myClass id from mod1: " + str(id(modules.mod1.getClass())))
logger.info("myClass id from mod2: " + str(id(modules.mod2.getClass())))
logger.info("myClass id from modBase: " + str(id(modules.modBase.myClass)))

inst1 = modules.mod1.getClass()() # create instance
logger.info(
	"instance of myClass isinstance of myClass: "
	+ str(isinstance(inst1, modules.modBase.myClass))
)

Results After running first time after saving in Designer ("script recompiled and restart")

Without importing

Importing modules

The result does not change at the gateway.

Therefore, to avoid discrepancies between tests performed in Designer and those executed in Gateway, I think it's best to maintain the imports.

An interesting point is that initially, I was importing the exact module I needed:

import modules.modBase

This created a separate entry for each imported module name in sys.modules.

But then I accidentally discovered that importing the package is sufficient to prevent this from happening. This also prevents the fully qualified module names from being loaded into sys.modules.

So let me know if you see any problems with this.

Thank you in advance

That should be

def incAndGet():
	global myInternalVar
	myInternalVar += 1
	return myInternalVar

{Seriously racy, btw. I hope you aren't trying to control parallelism with this.}

But anyways, the designer script console has very custom library module loading, in order to encapsulate the REPL properly, IIUC. import bypasses some of this and can lead to strange behavior.

(The designer script console is not suitable for testing gateway scope scripts for many other reasons.)

3 Likes

This is somewhat of a digression from the main topic at hand, but I'd love to know what leads folks to these OOP-heavy approaches to scripting within Ignition and exactly what they're (trying) to do with them. For better and for worse you usually have a much better time in Ignition if you embrace the state management we provide first party (e.g. tags) instead of trying to wrangle all your state in code...at least, that seems to be what people are ultimately after.

4 Likes

Unless you need locking/mutual exclusion across parallel tasks in an algorithm. That's about it.

Software engineers who are used to reasoning about code state and less used to reasoning about tag state, is my guess?

Yeah, that seems to be the kind of folks who go for this, but I'll be the first to admit: Ignition is a pretty terrible environment to purely write software in.
The whole point is the layers of abstraction we provide that gloss over lots of it. If you're trying to redo it all in code, you're swimming upstream.

A primary part of the value proposition of Ignition (in comparison to just writing your own bespoke software, if you have the capacity to do so) is that if something is broken, it's IA's problem. The more you're reinventing in code, the more you're making it your problem again.

All that aside...
For the OP: You don't want to import someProjectLibrary or anything like that. It's a bad habit to get into for a variety of reasons, the most significant of which is a fairly catastrophic memory leak.
If you're careful, you can store persistent state via system.util.globals (or Phil's Toolkit modules' global var map), but it must be pure data, not arbitrary custom classes, which you can "rehydrate" into stateful classes if you want to encapsulate logic.

I'd also recommend reading through this topic which describes a related known issue that will not be fixed in 8.1 but is already fixed in 8.3: [IGN-6503]Project Library Script Initialization Races

6 Likes

That's right, the mistake is mine. That's why I have a second module initialization in the script console image I posted.

Until now, we were working with stateless modules. Therefore, we couldn't even detect if something was running in different instances of the modules when testing from the script console.

Due to the solution's design (which we inherited), we have some Python objects that coordinate operations. When these objects need to be reused in subsequent executions (when the conditions for continuing the operation are met), they are recreated and populated with the data stored in the database.

On the other hand, a kind of type checking is performed using the value of a constant when a decision needs to be made in some scripts that redirect the flow to the appropriate type of operation.

So we set out to improve our code by using real type checking and caching some objects (to avoid some of the load on the database), and that's when things started to go wrong until we ran the small experiment that I published.

Now I see that our mistake was trying to force the code to work first in the script console, expecting that code that works there would work in all environments.

I retested our code from the gateway and the clients, and it works correctly without all those import lines.

Lessons learned:

  • Don't import any modules or packages (for Ignition script modules).
  • Test the script in the target environment.

Thank you all for your support

3 Likes

If you are disciplined enough to keep all of your code in the project library (you should--events and transforms should be one-liners), you can use my @system.util.runInGateway function, which allows you to call project library functions from the script console and have them run in gateway scope.

3 Likes

Hard agree on this. Definitely something I learned the hard way.
I have only found 2 legit usecases in Ignition for classes:

  1. As a simple data struct. Simple example
class NQStructure(object):
    def __init__(self, folderName):
        self.createNQ = '{}/create'
        self.updateNQ = '{}/update'
        self.deleteNQ = '{}/delete'
        self.readNQ = '{}/read'

Then in a module doing

NQ = NQStructure('myFolder')

makes the rest of my code easier to read, no hardcoded NQ’s, I can extend it with NQ.someSpecialNQ='somePath/to/NQ. And since most things need NQ’s for all of CRUD, I know it will be used in a lot of places and then forces me to keep my NQ’s more or less sanely organized.

  1. As an interface to tags or UDT’s that need similar behavior/functions so I define once and can use in many modules. Again though - let tag’s do the state management as you say.

Every other case I tried to use it seemed to be more trouble than it was worth.

1 Like

Hello,

I couldn't resist continuing with the "software engineering" and testing until I understood what was happening.

I've concluded the following:

  1. Upon startup, Ignition compiles and makes all packages and modules available. (But it doesn't execute them, unlike import.)

  2. Each module has a reference to each root package and module in its globals. (Accessible through ´__dict__, or globals() during initialization.)

  3. When referencing a module for the first time, it is initialized (first execution through the module's logic).

  4. In the designer's script console, modules and packages are loaded using a different Java object (I believe for write protection).

So, if we examine sys.modules, we can see that all the root modules and packages are there.

If we compare the ID reference of a root module in a module's global variables with the ID of sys.modules and the ID of the root module accessed via FQN, we can see that it's the same in every context except for the references in the module's global variables when it's executed in the script's console.

Continuing with my initial example (with 3 modules inside a package called modules):

import sys
# Tests before workaround
root_module = [modules, sys.modules["modules"], modules.mod1.__dict__["modules"]]
inner_module = [modules.mod1, sys.modules["modules"].mod1]
print root_module
print inner_module

In the script's console, I get:

To develop and test stateful modules and to properly support OOP, I ended up developing a function that replaces the root module references with those from sys.modules when the module starts.

import sys

# modules or packages whose references will not be replaced
ignored_root_mod_n_pck = ["system", "fpmi", "__builtins__"]

# replaces ScriptConsole modules and packages root level references to point the same in sys
def replace_root_mod_n_pck_refs(mod_globals, debug_print=False):
	if ".JythonConsole$" not in str(type(system)):
		return  # not in script console
	global ignored_root_mod_n_pck

	# Create a list of root level modules and packages
	root_modules = {}
	for key in sys.modules:
		rep = repr(sys.modules[key])
		if key in ignored_root_mod_n_pck:
			continue
		if ("module " in rep and "at " in rep) \
			or "app package" in rep:
			root_modules[str(key)] = sys.modules[key]

	# replace root references with sys ones
	for key in mod_globals.keys():
		rep = repr(mod_globals[key])
		if ("module " in rep and "at " in rep) \
			or "app package" in rep:
			if key in root_modules.keys():
				if mod_globals[key] != root_modules[key]:
					if debug_print: 
						print (
							mod_globals["__name__"] + ": replacing base for: "
							+ str(key) + " from id " + str(id(mod_globals[key]))
							+ " to id " + str(id(root_modules[key]))
						)
					mod_globals[key] = root_modules[key]

I put this code in a module called scriptConsole and had to call it on the first line of each module. This way, on the first execution of the module, and before any references are used, all the global variables are reassigned.

# always first line, like importing root modules and packages
scriptConsole.replace_root_mod_n_pck_refs(globals(), False)

This accomplishes the useful part that import provided, but only when run from the script console. It might still have memory leaks, but I think it should only be in the designer's JVM.

I also used this function to get a list of modules and packages from sys.modules.

import sys

# List all root modules or packages in sys with a string containing 'name, repr, type'.
def get_sys_root_mod_n_pck():
	modules_list = []
	mod_dict = sys.modules
	for key in mod_dict:
		rep = repr(mod_dict[key])
		if ("module " in rep and "at " in rep) \
			or "app package" in rep:
			modules_list.append(str(key) + ", " + repr(mod_dict[key]) + ", " + str(type(mod_dict[key])))
	
	return modules_list

This might be a terrible and ill-advised workaround. But it explains what's happening. And perhaps the script console has a good reason to reload or create a separate container for modules and packages in each module's global map, but that's what prevents testing stateful modules and OOP.

I don't know, maybe there's another scenario where this also occurs. I did some testing on the gateway, perspective, and client using object-oriented programming with inheritance metaclasses and type checking, and it seems to work fine.

By the way, as everyone here has said, it's best to keep things simple.

This is just for academic purposes.

All feedback are welcome.