# \[IGN-6503\]Project Library Script Initialization Races

**URL:** <https://forum.inductiveautomation.com/t/ign-6503-project-library-script-initialization-races/98636>\
**Category:** Ignition\
**Tags:** scripting\
**Created:** [November 14, 2024, 8:51pm UTC](https://forum.inductiveautomation.com/t/ign-6503-project-library-script-initialization-races/98636 "2024-11-14T20:51:28Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\
**Post date:** [November 14, 2024, 8:51pm UTC](https://forum.inductiveautomation.com/t/ign-6503-project-library-script-initialization-races/98636/1 "2024-11-14T20:51:28Z")

</div>

Are you annoyed by pseudo-random breakage in Ignition project library scripting? Odd top-level constants being garbled, or operations in the top level that have side effects running twice? Or more?

Not happening much, but always around project edits, and sometimes even on gateway restart?

Yeah, me too. ☹

Poking around under the hood back in the spring, for a module development task, I ran across some problematic IA bytecode and pinged some people. But it went nowhere, as I could not, at that time, make a reliable reproducer.

But I dreamed some code this week, thanks to an exotic consulting task, that led to a reproducer in a [nice, self-contained project](https://www.automation-pros.com/ignition/script-module-load-race_2024-11-13_1104.zip). On decent multi-core hardware, the reproducer runs its script init in parallel, near-simultaneously, in **five** threads. On startup and on every project edit.

Import that into any gateway and immediately see the race in the logs.

I'm sure IA will fix this, but there are many, I suspect, who would benefit from a workaround in the meantime.

If you try the above project and it shows the problem, replace the script module's import section with this alternate:

```python
# Catch racing initialization threads (See IA support ticket #131484)
import sys, threading
globals().setdefault('initLock', threading.Lock())
if not initLock.acquire(False):
	# not first
	with initLock:
		pass
	sys.exit()
# One more leaker possibility. No need to wait on this one.
# The key checked can be any of the top level variables instantiated below.
if 'logger' in globals():
	initLock.release()
	sys.exit()

from java.util.concurrent.atomic import AtomicInteger

```

And add this at the bottom of that script:

```python
# The following must remain at the end. There *must* be error checking
# above to ensure this executes.
initLock.release()

```

And you should see one "Initializing" report per project edit thereafter.

Tweak to suit.

I will report when I'm told this is fixed.

{ You shouldn't expect any proper external python IDE to like that work-around, due to its use of variable names that have never received an assignment. 🤷‍♂️ }

---

<div class="post-metadata">

**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\
**Post date:** [November 16, 2024, 3:58pm UTC](https://forum.inductiveautomation.com/t/ign-6503-project-library-script-initialization-races/98636/2 "2024-11-16T15:58:43Z")

</div>

I should mention that the work-around shown above should **NOT** be placed in **all** of your project library scripts. It solves the multiple execution problem but at the risk of AB-BA deadlock across interdependent scripts. Some thoughts:

- Definitions of simple functions and simple assignments to top-level constants do not need this fix. While multiple executions will repeatedly replace top level objects, the replacements are identical to the first pass--no harm, no foul.

- Definitions of top-level objects to be used as deletable/clearable caches are also effectively unharmed by multiple initialization, though some cache entries established early (while racing) might be blown away.

- Definitions of utility classes that have short lifetimes are similarly unaffected. That early instances might not have the same type as later instances (because the class was redefined multiple times) is meaningless for such cases--the instances don't live long enough for that to matter, and the implementations are identical.

- Establishing top-level dictionaries with my `system.util.globalVarMap()` or the native `system.util.getGlobals()` to obtain JVM-lifetime persistent storage is safe. Values placed in such dictionaries with `.setdefault()` are also unaffected by repeat initialization.

Scenarios that **DO** need this fix:

- Jython classes that establish singleton instances during initialization. Multiple singletons will be created, and those grabbed early will not have the same state as the last one.

- Top level constants that are assembled in multiple steps, using `+=` or `list.append()` operations, will be corrupted in odd ways by the parallel execution. (Encountered this a lot some years ago--I had to stop doing that.)

- Top level objects that form the root of linked object relationships are vulnerable. Early linkages may be chopped off by the repeat initialization and those linked objects' memory leaked.

- Scripts that launch asynchronous threads during initialization will launch multiples, with potentially long-lived clashing behavior.

- Top level constants initialized from database queries, web API requests, or other time-consuming operations (particularly when calling a function just defined locally) will race and race and race the whole time the first execution runs, and the last execution may leave the constant with errors if any resource has rate limits.

The fix I recommended to IA in my bug report does not have the AB-BA deadlock risk, so hopefully this all becomes moot.

---

<div class="post-metadata">

**Author:** ![louagejasper](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/louagejasper/32/86311_2.png) [@louagejasper](https://forum.inductiveautomation.com/u/louagejasper)\
**Post date:** [December 10, 2024, 11:29am UTC](https://forum.inductiveautomation.com/t/ign-6503-project-library-script-initialization-races/98636/3 "2024-12-10T11:29:29Z")

</div>

So the workaround does not always work?  
I think this is quite important to get right for Ignition.

---

<div class="post-metadata">

**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\
**Post date:** [December 10, 2024, 1:38pm UTC](https://forum.inductiveautomation.com/t/ign-6503-project-library-script-initialization-races/98636/4 "2024-12-10T13:38:45Z")

</div>

The work-around always works if it runs to completion (the `.release()`). There are situations where that could be blocked or broken, most notably AB-BA deadlocks.

---

<div class="post-metadata">

**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\
**Post date:** [February 4, 2025, 12:32pm UTC](https://forum.inductiveautomation.com/t/ign-6503-project-library-script-initialization-races/98636/5 "2025-02-04T12:32:52Z")

</div>

Looks like this is ~~now fixed~~ :

> [@Nightly 8.1 Changelogs - 2025](https://forum.inductiveautomation.com/t/nightly-8-1-changelogs-2025/100308/25):
>
> Nightly Changelog: 8.1.46-SNAPSHOT-b20250203 Scripting Scripting IGN-11792: Backport script module load order initialization fix Fixed an issue with script initialization potentially overlapping, causing inconsistent object IDs and other ill effects.

---

<div class="post-metadata">

**Author:** ![mazeyrat](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/mazeyrat/32/10955_2.png) [@mazeyrat](https://forum.inductiveautomation.com/u/mazeyrat)\
**Post date:** [June 14, 2025, 10:15am UTC](https://forum.inductiveautomation.com/t/ign-6503-project-library-script-initialization-races/98636/6 "2025-06-14T10:15:45Z")

</div>

do you thing this bug could be rely to the script initialization race:

> [@\[bug-3893\]Message Handler null in ScriptConfig during project save](https://forum.inductiveautomation.com/t/bug-3893-message-handler-null-in-scriptconfig-during-project-save/49995):
>
> Gateway script MessageHandlerException, project 'Project6', message handler 'threadHandler': com.inductiveautomation.ignition.common.script.message.MessageHandlerException: MessageHandlerManager is null in ScriptConfig! This error occurs when a message is sent at the same that the project containing the destination message handler is saved. I assume because the save process is to recreate the message handler, AKA, delete and create. For message handlers that are processing many requests per sec…

I have it often with 8.1.48 and I search for a workaround...

When it occur some script tag expression are broken until gateway restart....

---

<div class="post-metadata">

**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\
**Post date:** [June 14, 2025, 2:16pm UTC](https://forum.inductiveautomation.com/t/ign-6503-project-library-script-initialization-races/98636/7 "2025-06-14T14:16:42Z")

</div>

Doesn't sound like it. This race happens when a parallel "thundering herd" of events happen right **after** scripting restart. The message handler bug seems to be in the **middle** of scripting restart, entirely in java.

---

<div class="post-metadata">

**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\
**Post date:** [June 14, 2025, 2:20pm UTC](https://forum.inductiveautomation.com/t/ign-6503-project-library-script-initialization-races/98636/8 "2025-06-14T14:20:13Z")

</div>

> [@pturmel](#):
>
> Looks like this is ~~now fixed~~ :

BTW, 8.1.46 was pulled because that fix could trigger AB-BA deadlock with jython's own module import lock in some cases. (Where users were **importing** other project library scripts.)

---

<div class="post-metadata">

**Author:** ![mazeyrat](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/mazeyrat/32/10955_2.png) [@mazeyrat](https://forum.inductiveautomation.com/u/mazeyrat)\
**Post date:** [June 14, 2025, 2:23pm UTC](https://forum.inductiveautomation.com/t/ign-6503-project-library-script-initialization-races/98636/9 "2025-06-14T14:23:29Z")

</div>

> [@pturmel](#):
>
> - Top level constants that are assembled in multiple steps, using `+=` or `list.append()` operations, will be corrupted in odd ways by the parallel execution. (Encountered this a lot some years ago--I had to stop doing that.)
> - Top

As the fix seems no to be targeted in 8.1.49  
I will probably try your workaround in few common script lib to avoid this...

---

<div class="post-metadata">

**Author:** ![mazeyrat](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/mazeyrat/32/10955_2.png) [@mazeyrat](https://forum.inductiveautomation.com/u/mazeyrat)\
**Post date:** [September 30, 2025, 2:25pm UTC](https://forum.inductiveautomation.com/t/ign-6503-project-library-script-initialization-races/98636/10 "2025-09-30T14:25:45Z")

</div>

We have put the workaround. Work nice force script update, but when we try to import a project we have a deadlock detected. Any idea what's wrong ?

 ![image](https://us1.discourse-cdn.com/inductiveautomation/original/3X/1/e/1edd90d4de4be8fe3cd85e943dea9ebda1c4a874.png)

 ![image](https://us1.discourse-cdn.com/inductiveautomation/original/3X/f/7/f729cbf3e225d201bea059b8bfd4f156b87d79a5.png)

---

<div class="post-metadata">

**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\
**Post date:** [September 30, 2025, 2:55pm UTC](https://forum.inductiveautomation.com/t/ign-6503-project-library-script-initialization-races/98636/11 "2025-09-30T14:55:18Z")

</div>

The work-around is not a complete fix. It only catches the races in **one** project library script. It cannot deal with races across multiple scripts, and can deadlock in such cases. You'll have to study the thread dump to determine which of your scripts are deadlocking.

(Make sure you do not use python's `import` statement, in any form, with project library scripts.)

The proper fix is in v8.3.

---

<div class="post-metadata">

**Author:** ![Alex\_Ceithamer](https://avatars.discourse-cdn.com/v4/letter/a/a3d4f5/32.png) [@Alex\_Ceithamer](https://forum.inductiveautomation.com/u/Alex_Ceithamer)\
**Post date:** [September 30, 2025, 6:55pm UTC](https://forum.inductiveautomation.com/t/ign-6503-project-library-script-initialization-races/98636/12 "2025-09-30T18:55:00Z")

</div>

I appreciate this detailed explanation and workaround! Please correct me if I'm wrong, but I think code design is another angle worth considering. I haven't run into this issue myself, likely because of how I structure my project-library scripts.

Coming from an Object-Oriented background, I tend to keep the top level of each script limited to just function, class, and CONSTANT definitions. Any heavier setup/logic lives inside functions, which can be explicitly invoked during startup events (Perspective's "Startup" Session Event or Vision's "Startup" Client event). This way, nothing runs implicitly during script initializations, and all execution is deliberate and context-aware.

Curious if others have found cases where this approach doesn’t hold up.

---

<div class="post-metadata">

**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\
**Post date:** [September 30, 2025, 7:19pm UTC](https://forum.inductiveautomation.com/t/ign-6503-project-library-script-initialization-races/98636/13 "2025-09-30T19:19:39Z")

</div>

> [@Alex\_Ceithamer](#):
>
> where this approach doesn’t hold up.

Doesn't hold up at all. In my experiments, any combination of project startup scripts with gateway tag change event scripts, or multiple of the latter, or various similar startup effects from gateway scripting project calls, will produce the problematic parallel initialization threads.

Re-read comment #2.

---

<div class="post-metadata">

**Author:** ![mazeyrat](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/mazeyrat/32/10955_2.png) [@mazeyrat](https://forum.inductiveautomation.com/u/mazeyrat)\
**Post date:** [October 2, 2025, 3:42pm UTC](https://forum.inductiveautomation.com/t/ign-6503-project-library-script-initialization-races/98636/14 "2025-10-02T15:42:17Z")

</div>

For my deadlock when I import a project, I change the workaround with a try finally, and it seems to avoid deadly deadlock.

```python
import sys, threading
globals().setdefault('initLock', threading.Lock())
if not initLock.acquire(False):
	# not first
	with initLock:
		pass
	sys.exit()
# One more leaker possibility. No need to wait on this one.
# The key checked can be any of the top level variables instantiated below.

try:
	if 'ENV' in globals():
		sys.exit()

   ENV = "DEV"
   ...

finally:	
	initLock.release()

```

---

<div class="post-metadata">

**Author:** ![pturmel](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/pturmel/32/3295_2.png) [@pturmel](https://forum.inductiveautomation.com/u/pturmel)\
**Post date:** [October 2, 2025, 3:46pm UTC](https://forum.inductiveautomation.com/t/ign-6503-project-library-script-initialization-races/98636/15 "2025-10-02T15:46:24Z")

</div>

I don't see how that can possibly work. You are instantiating `initLock_shared_commun_constantescustom`, acquiring `initlock`, and then releasing (without acquiring) `initLock_shared_commun_constantescustom`. 🤷‍♂️

---

<div class="post-metadata">

**Author:** ![mazeyrat](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/mazeyrat/32/10955_2.png) [@mazeyrat](https://forum.inductiveautomation.com/u/mazeyrat)\
**Post date:** [October 2, 2025, 4:02pm UTC](https://forum.inductiveautomation.com/t/ign-6503-project-library-script-initialization-races/98636/16 "2025-10-02T16:02:27Z")

</div>

oops I forgot to change the last name for the example

---

<div class="post-metadata">

**Author:** ![cmaynes](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/cmaynes/32/6852_2.png) [@cmaynes](https://forum.inductiveautomation.com/u/cmaynes)\
**Post date:** [October 9, 2025, 9:41pm UTC](https://forum.inductiveautomation.com/t/ign-6503-project-library-script-initialization-races/98636/17 "2025-10-09T21:41:20Z")

</div>

Just want to pile on, as this has been a long-standing issue for us using Ignition in our company. For better or worse at this point, we have hundreds of custom classes and many thousands of lines of code in custom script modules across the global project (converted over from shared scripts to a project when we moved from v7.9.21 to v8.1.48) and the ~30 vision projects we have. This seemed to be an issue, at least for us, in v7.9 of ignition and this has continued / maybe gotten worse in v8.1.

This issue is causing sporadic issues with clients during startup when shared constants end up corrupted and the client has to be restarted. We also see it on the gateway most times we try to update projects, resulting in some modules seeing one class definition and others seeing another and isinstance/type checks failing all over the place. We have to restart the gateway instead when this happens, disrupting operations.

We have attempted a great many solutions, but it seems like the only really viable one is for the internal ignition APIs/ScriptManager to serialize imports and ensure each module is executed exactly once.

Just wanted to “vote” for this to be fixed sooner than later.

---

<div class="post-metadata">

**Author:** ![paul-griffith](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/paul-griffith/32/37249_2.png) [@paul-griffith](https://forum.inductiveautomation.com/u/paul-griffith)\
**Post date:** [October 9, 2025, 10:19pm UTC](https://forum.inductiveautomation.com/t/ign-6503-project-library-script-initialization-races/98636/18 "2025-10-09T22:19:50Z")

</div>

The fix has already been applied, in 8.3.  
We're not going to backport any change to 8.1 because it's not worth the risk - the previous attempt to fix this issue is the reason 8.1.46 got pulled down. Known behavior that's been this way for years 'wins' in an LTS version that's in maintenance mode over making a change in an extremely regression prone area.

---

<div class="post-metadata">

**Author:** ![cmaynes](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/cmaynes/32/6852_2.png) [@cmaynes](https://forum.inductiveautomation.com/u/cmaynes)\
**Post date:** [October 9, 2025, 11:00pm UTC](https://forum.inductiveautomation.com/t/ign-6503-project-library-script-initialization-races/98636/19 "2025-10-09T23:00:39Z")

</div>

Thank you!

You just made my day!!

---

<div class="post-metadata">

**Author:** ![mazeyrat](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/mazeyrat/32/10955_2.png) [@mazeyrat](https://forum.inductiveautomation.com/u/mazeyrat)\
**Post date:** [October 12, 2025, 4:08pm UTC](https://forum.inductiveautomation.com/t/ign-6503-project-library-script-initialization-races/98636/20 "2025-10-12T16:08:51Z")

</div>

@paul-griffith the drawback with this fix,

```python
import sys, threading
globals().setdefault('initLock', threading.Lock())
if not initLock.acquire(False):
	# not first
	with initLock:
		pass
	sys.exit()
# One more leaker possibility. No need to wait on this one.
# The key checked can be any of the top level variables instantiated below.

try:
	if 'ENV' in globals():
		sys.exit()

   ENV = "DEV"
   ...

finally:	
	initLock.release()

```

all the functions in the try/finally blocks are not detected in designer script and nothing is listed for this script library...(same behaviours with a "with ..." block)

```python
try:
  if 'ENV' in globals():
    sys.exit()

   ENV = "DEV"
   
  def maFunction():
    ...

finally:	
	initLock.release()

```

[Next page](https://forum.inductiveautomation.com/t/ign-6503-project-library-script-initialization-races/98636.md?page=2)