Environment: Ignition 8.1.43 (b2024082010), Azul Java 17.0.11, Linux. About 10 gateway timer scripts in one project, all Dedicated thread, Fixed Delay, delays from 1 s to 1 h.
What happens Once a day an external tool updates script files on disk for three projects. A parent library project is inherited by two child projects. The gateway log shows:
Setting LastModification to "external" on ...for 3 script resourcesProject: Restarting gateway scripts...5 times within 3 seconds- then 3 more
Restarting gateway scripts...between 8 and 62 seconds later
On one day after this, one timer (Dedicated, 1,000 ms, Fixed Delay) stopped running. It stayed that way for 2 h 20 min until we disabled and re-enabled the project. The other timers in the same project, created at the same restart with consecutive thread IDs, were fine.
Thread dump taken during the failure Healthy sibling timer:
"gateway-script-dedicated-timer-[ProjectB/TimerX]-1" TIMED_WAITING
Unsafe.park / LockSupport.parkNanos
AbstractQueuedSynchronizer$ConditionObject.awaitNanos
ScheduledThreadPoolExecutor$DelayedWorkQueue.take
ThreadPoolExecutor.getTask / runWorker
Affected timer:
"gateway-script-dedicated-timer-[ProjectB/TimerY]-1" WAITING
Unsafe.park / LockSupport.park
ForkJoinPool.managedBlock
AbstractQueuedSynchronizer$ConditionObject.await
ScheduledThreadPoolExecutor$DelayedWorkQueue.take
ThreadPoolExecutor.getTask / runWorker
So the worker thread exists, but the scheduler queue is empty. Nothing is scheduled for it.
What I checked
- No BLOCKED threads and no lock owner.
- No
Error executing global timer script ...line for that timer. For other timers, that line appears once after each start, as expected. - No long pause (the clock drift detector was quiet).
- A single project save restarts everything and the timer comes back. A separate test timer that raises
java.lang.Erroris caught, logged once, and keeps running. - The same daily restart has been fine on other days.
Questions
- Can a timer's scheduled task be cancelled or lost without any log line when several projects restart within a few seconds, with a second wave still arriving during start-up?
- Is there a supported way to detect from script that a timer is no longer scheduled?
- Is there a logger I can set to DEBUG to see timer scheduling and cancel events?
- Is this a known issue in 8.1.43 and fixed in a later 8.1.x?
I can't reproduce it on demand. Any pointers on what to capture the next time it happens would help.