Python subprocess not working with message handler

I have created a method which lets me run arbitrary commands in my project repository from the designer script console. I use it to run git commands on the gateway, get the stdout and stderr output back from the gateway and print them in the script console.

I have a message handler in the gateway event script which calls the following code:

def runCommand(commandString):
	"""Invoke a command from the designer script console
	
	Runs a command in the repository path and returns stdout and stderr to the script console
	"""
	
	logger.info("Running command '" + commandString + "'")
	statusMessage = "\n"
	
	# Run command in the repository path
	process = subprocess.Popen(["cmd", "/c", "cd " + REPOSITORY_PATH + " &" + commandString], stdout = subprocess.PIPE, stderr = subprocess.PIPE)
	sts = process.communicate() # [0] = stdout, [1] = stderr
	
	# Log command to gateway scope logger and send log info back to designer scope script console
	statusMessage += "Gateway: Command '" + commandString + "' executed\n"
	logger.info("Command '" + commandString + "' executed")
	statusMessage += formatProcessOutput(sts)
	logger.debug("Command output: " + formatProcessOutput(sts))
	if sts[1] is not None and sts[1] != "":
		logger.error("Git commit script error: " + sts[1])
	
	return statusMessage

I invoke this script from the designer (script console) using system.util.sendRequest().
This works fine with most commands (git status, git log, git commit and so on). However today I tried to run git push and git pull. Both commands timed out.
When I run the above code directly on the machine where the ignition gateway is runing (from the designer script console), it is also working. As soon as I involve gateway message handlers the script blocks at “process.communicate()” and never terminates. The message handler times out. Increasing the timeout does not help.

After extensive testing I could not find any explanation for this. The command takes approx. 2 seconds to finish, with faster commands there is no problem at all.
Is there a way to invoke such long lasting scripts via gateway message handlers (and still get a return message)?

:scream:

How confident are you git for windows isn’t prompting for input (e.g. for authentication) when this happens? I’m 90% sure that’s the problem you’re actually having.

Horrible, I know. However there is a need for a handy devOps workaround as long as we do not have a good solution for Git integration. As long as these functions are only accessible for our developers the benefit it brings us outweighs the potential problems and danger.

Extremely confident for this specific command.
Of course this will not work for commands where Input is needed. The way we intend to use these commands, 90% of the time there will be no need for input.

The problem even persists when I override the stdin with nul values so that the process does not wait for input.

echo nul | git push
or
git push &< /dev/nul

As I have mentioned, when I run a designer on the same machine where the Ignition gateway is running (so that I can access the file system from the designer scope) and run the command directly in the script console, it runs without any problems. There is a delay of a few seconds, but when the command finishes I finally get the return value.

When I invoke the function using a message handler so that it runs in the gateway scope it stops working. It works perfectly for short running commands, but if the commands take too long it never runs beyond communicate(). Somehow it seems like communicate() is blocking and does not return anything if the action takes too long. Again, this is only a problem if message handlers are involved. The timeout setting does not change anything since the code hangs at communicate() forever.
When I do something stupid like time.sleep(10), waiting withing the message handler works fine.

Could this be a threading issue which is related to the message handler system? :thinking:

My guess is this is a permissions issue. The user using Designer on gateway has permissions to run this command directly via script console or however you were testing it, but the user the gateway is logged in as does not, hence the failure via message handling. That’s my intuition at least.

I run some database backup logic via batch files and subprocess.Popen with process.communicate() and don’t have any issue so I don’t think it’s a message handler thing.

No, unfortunately (this would be an easy solution) it is not a permission problem.
As I said, the mechanism is working perfectly for most commands (git status, git log, git commit -m and so on). Only if a command is taking longer, like git push and git pull (they involve network connectivity, this could also be a factor here) it fails.

What I don’t get though is the network connectivity doesn’t seem to be a problem when you are doing directly from designer/script console, but from the very same machine, just run via a gateway message handler, now the network connectivity becomes an issue? I don’t get how that would happen.

Correct. I am not saying network connectivity is a problem, just saying it might have an influence here. Mainly because as soon as we have networking is involved we have a lot more delays.

What account is the Ignition Gateway service running under? If it’s the default, LOCAL_SYSTEM, Windows will prevent it from accessing any network resources.

1 Like

Hm, seems that I need to revise my previous statement. The LOCAL_SYSTEM account has sufficient rights to execute my scripts, I ensured this. But if it does not have access to any network resources this might be the cause of all that trouble.

Are there any (functional) concerns with running the Gateway service as a different user? From other forum post recommending to change the user I would not assume so?

1 Like

Generally speaking, the big concern is that if you switch to e.g. a domain user, suddenly your Ignition service might be subject to a password expiration policy, or something along those lines.

There may be other alternatives?

Thank you, it was the LocalSytem account which caused troubles.
Using a different account works. However I did not find a way to assign the necessary privileges to the LocalSystem account.

1 Like

90% of the time I’ve had an issue where a script I wrote worked in script console, but did not work via message handler/gateway tag change events, the culprit has been roles. The other culprit is usually mapped drives (or lack thereof in the gateway .conf file).