Hello everyone,
My module has several execution scripts that are making a series of REST calls. I have noticed that when some of these REST calls repeatedly fail (return a bad response), it has been overloading the Ignition system resources and sometimes leading to full crashes. I am trying to troubleshoot what may be at the root of this problem. Here is some context:
-
Here is a list of my execution scripts that are using the shared execution engine:
-
Here is a sample section of code with how I register these scripts:
public void register() {
if(parentRecord.getEnabled().equals(true)) {
log.info("Launching Agent: {}", execId);
context.getExecutionManager().register(execId, "ClearAllInterlocks", new ClearAllInterlocks(context, record, execId, parentRecord, httpPool), 500);
context.getExecutionManager().register(execId, "CreateMission", new CreateMission(context, record, execId, parentRecord, httpPool), 500);
}
}
- One thing in particular that I have noticed is that there are an increasingly large number of Timed Waiting threads in the system. I cannot determine what processes are creating these threads or what they belong to by using the Ignition Gateway, but I suspect this may be related to the problem. Here is a snapshot of the system performance when experiencing some of these issues:
My main suspicion at the moment is that the execution scripts are being registered with a Fixed Rate instead of a Fixed Delay. When the REST calls fail, it is causing the execution time to last longer than the rate and causes a large buildup of Timed Waiting threads. I don't have a ton of visibility into how the ExecutionManager or threading works so I was wondering does this theory make any sense? If not, any other ideas? If so, would the correct solution be to switch from
context.getExecutionManager().register(execId, "ClearAllInterlocks", new ClearAllInterlocks(context, record, execId, parentRecord, httpPool), 500);```
to something like
context.getExecutionManager().scheduleWithFixedDelay(new ClearAllInterlocks(context, record, execId, parentRecord, httpPool), 500, 500, TimeUnit.MILLISECONDS);
Thanks in advance for the help!
