Auto restart gateway timer scripts

I currently have >10 gateway timer scripts with the following configurations: 1000ms, fixed delay, dedicated threading. This was set up this way to ensure each script ran once to completion before beginning another run, and to ensure it does not affect other scripts hence dedicated threading.

The problem I have is when one/many of the scripts hang, it will appear on the gateway/running scripts sections and freezes there. Majority of the time it was just remain there frozen as 1 and other scripts continue to run but there has been times where hundreds/thousands of the same scripts appear here and freezes the system. (Ideally should be 1 per timer)

Currently, when I see this issue I would go to the gateway and cancel the script, then go to the designer and save to restart the timer scripts or restart the entire system. This requires a manual intervention.

I want to set up a way, where when one of the scripts hangs/freezes to restart the script again automatically, send an email notifying it had issues and restarted. Is this achievable in ignition?

How do you evaluate a script has frozen?

Within this section, a script which should be done within a few sections has an elapsed time of hours.

Sometimes, I even see 20 rows and 20 pages of the same script here all freezing or hanging.

I mean programmatically? The script is not going to look at the gateway webpage, it needs to know the difference between a hanging script and a script that is still working.

In theory, you can set up a system that manages your script executions, but you need a foolproof way to evaluate that a script has failed, and a way to end that thread.

The bigger question is why are your scripts hanging?

1 Like

Currently, my script will write to a tag when it starts running and another timer script that will check if the tag value is older than 5 minutes send email alerting the script is hanging. I however, do not have a way to shut down the running thread and start running the script again in a new thread.

What makes the thread hang? It will make the answer different about how to deal with it.

Script/system/tag issues… but these were resolved. I want to take care of future scenarios that were uncaught.

Killing threads is generally a bad idea in systems that use dynamic memory and garbage collectors, so doing everything in your power to catch a problem in the initial script itself and self terminating is the best option. The issue you could face is that eventually you leak a lot of memory by force killing script threads and you crash ignition. This would all happen automatically under the surface if you made this system.

1 Like

Fix the sharp edge of the table, don’t buy a box of bandaids.

4 Likes

I absolutely agree with the general advice in this thread that this is generally inadvisable.

That said, for the sake of it, the ScriptManager class is your entry point to the list of running threads in 8.1/your best source of programmatic access.

In 8.3 the web API has the corresponding functionality that's available in 8.1 but with the ability to authenticate via API tokens - so you could list running scripts and issue terminate requests via system.net.httpClient as another option.