Dedicated gateway timer silently stops being scheduled after several overlapping "Restarting gateway scripts" (8.1.43)

Environment: Ignition 8.1.43 (b2024082010), Azul Java 17.0.11, Linux. About 10 gateway timer scripts in one project, all Dedicated thread, Fixed Delay, delays from 1 s to 1 h.

What happens Once a day an external tool updates script files on disk for three projects. A parent library project is inherited by two child projects. The gateway log shows:

  • Setting LastModification to "external" on ... for 3 script resources
  • Project: Restarting gateway scripts... 5 times within 3 seconds
  • then 3 more Restarting gateway scripts... between 8 and 62 seconds later

On one day after this, one timer (Dedicated, 1,000 ms, Fixed Delay) stopped running. It stayed that way for 2 h 20 min until we disabled and re-enabled the project. The other timers in the same project, created at the same restart with consecutive thread IDs, were fine.

Thread dump taken during the failure Healthy sibling timer:

"gateway-script-dedicated-timer-[ProjectB/TimerX]-1"  TIMED_WAITING
  Unsafe.park / LockSupport.parkNanos
  AbstractQueuedSynchronizer$ConditionObject.awaitNanos
  ScheduledThreadPoolExecutor$DelayedWorkQueue.take
  ThreadPoolExecutor.getTask / runWorker

Affected timer:

"gateway-script-dedicated-timer-[ProjectB/TimerY]-1"  WAITING
  Unsafe.park / LockSupport.park
  ForkJoinPool.managedBlock
  AbstractQueuedSynchronizer$ConditionObject.await
  ScheduledThreadPoolExecutor$DelayedWorkQueue.take
  ThreadPoolExecutor.getTask / runWorker

So the worker thread exists, but the scheduler queue is empty. Nothing is scheduled for it.

What I checked

  • No BLOCKED threads and no lock owner.
  • No Error executing global timer script ... line for that timer. For other timers, that line appears once after each start, as expected.
  • No long pause (the clock drift detector was quiet).
  • A single project save restarts everything and the timer comes back. A separate test timer that raises java.lang.Error is caught, logged once, and keeps running.
  • The same daily restart has been fine on other days.

Questions

  1. Can a timer's scheduled task be cancelled or lost without any log line when several projects restart within a few seconds, with a second wave still arriving during start-up?
  2. Is there a supported way to detect from script that a timer is no longer scheduled?
  3. Is there a logger I can set to DEBUG to see timer scheduling and cancel events?
  4. Is this a known issue in 8.1.43 and fixed in a later 8.1.x?

I can't reproduce it on demand. Any pointers on what to capture the next time it happens would help.

I guess this might be relevant we have about 500 gateways running the same logic. we see failures 2 or 3 times a month always on different instances. This is for scale and how often this happens.

If the specific customer doesn't see or report the issue... the next restart (daily) can fix the issue

I would suspect this:

(Fixed in v8.3. Fix shown in that topic is only a partial solution.)

is there a way to verify this is happening when one is failed? This is my suspect as well. I was told to build a case to increase our speed to 8.3.

the thread dump just shows nothing scheduled. I can't even find the running process anywhere either. It looks like something removed it from the scheduler.