Hello,
I have a long running gateway timer script that listens to a piece of equipment's 'event stream' and takes various action according to what the 'event steam' posts (e.g. the machine goes out of run mode). If the script fails, I have it throw and exception and post the error message to the logs before trying to start up again 30 seconds later. I have some other functionality in place that sms's me when the script stops and starts running.
The thing has been running fine for a long while now but yesterday (a Sunday) at ~11:30am, I started getting very frequent sms's for the alarm becoming active (that the script failed) and clearing (re-starting).
Error Message:
com.inductiveautomation.ignition.common.script.JythonExecException: Traceback (most recent call last): File "", line 94, in File "", line 94, in at com.inductiveautomation.ignition.gateway.history.stores.MemoryStore.insertInBuffer(MemoryStore.java:186) at com.inductiveautomation.ignition.gateway.history.stores.MemoryStore.syncdStoreData(MemoryStore.java:84) at com.inductiveautomation.ignition.gateway.history.stores.AbstractStore.storeData(AbstractStore.java:233) at com.inductiveautomation.ignition.gateway.history.stores.MultiStageStore.syncdStoreData(MultiStageStore.java:144) at com.inductiveautomation.ignition.gateway.history.stores.AbstractStore.storeData(AbstractStore.java:233) at com.inductiveautomation.ignition.gateway.history.DefaultStoreAndForwardEngine.storeData(DefaultStoreAndForwardEngine.java:145) at com.inductiveautomation.ignition.gateway.history.HistoryManagerImpl.storeHistory(HistoryManagerImpl.java:261) at com.inductiveautomation.ignition.gateway.script.GatewayTagUtilities.storeTagHistoryImpl(GatewayTagUtilities.java:459) at com.inductiveautomation.ignition.common.script.builtin.AbstractTagUtilities.storeTagHistory(AbstractTagUtilities.java:612) at jdk.internal.reflect.GeneratedMethodAccessor211.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) java.lang.Exception: java.lang.Exception: Unable to add data to memory buffer, buffer full.
at org.python.core.Py.JavaError(Py.java:547)
at org.python.core.Py.JavaError(Py.java:538)
at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:192)
at com.inductiveautomation.ignition.common.script.ScriptManager$ReflectedInstanceFunction.__call__(ScriptManager.java:553)
at org.python.core.PyObject.__call__(PyObject.java:512)
at org.python.core.PyObject.__call__(PyObject.java:517)
at org.python.pycode._pyx2218.f$0(:108)
at org.python.pycode._pyx2218.call_function()
at org.python.core.PyTableCode.call(PyTableCode.java:173)
at org.python.core.PyCode.call(PyCode.java:18)
at org.python.core.Py.runCode(Py.java:1687)
at com.inductiveautomation.ignition.common.script.ScriptManager.runCode(ScriptManager.java:804)
at com.inductiveautomation.ignition.gateway.project.ProjectScriptLifecycle$TrackingProjectScriptManager.runCode(ProjectScriptLifecycle.java:843)
at com.inductiveautomation.ignition.common.script.ScriptManager.runCode(ScriptManager.java:752)
at com.inductiveautomation.ignition.gateway.project.ProjectScriptLifecycle$TrackingProjectScriptManager.runCode(ProjectScriptLifecycle.java:824)
at com.inductiveautomation.ignition.common.script.TimerScriptTask.run(TimerScriptTask.java:90)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.python.core.PyException: java.lang.Exception: java.lang.Exception: Unable to add data to memory buffer, buffer full.
... 22 common frames omitted
Caused by: java.lang.Exception: Unable to add data to memory buffer, buffer full.
at com.inductiveautomation.ignition.gateway.history.stores.MemoryStore.insertInBuffer(MemoryStore.java:186)
at com.inductiveautomation.ignition.gateway.history.stores.MemoryStore.syncdStoreData(MemoryStore.java:84)
at com.inductiveautomation.ignition.gateway.history.stores.AbstractStore.storeData(AbstractStore.java:233)
at com.inductiveautomation.ignition.gateway.history.stores.MultiStageStore.syncdStoreData(MultiStageStore.java:144)
at com.inductiveautomation.ignition.gateway.history.stores.AbstractStore.storeData(AbstractStore.java:233)
at com.inductiveautomation.ignition.gateway.history.DefaultStoreAndForwardEngine.storeData(DefaultStoreAndForwardEngine.java:145)
at com.inductiveautomation.ignition.gateway.history.HistoryManagerImpl.storeHistory(HistoryManagerImpl.java:261)
at com.inductiveautomation.ignition.gateway.script.GatewayTagUtilities.storeTagHistoryImpl(GatewayTagUtilities.java:459)
at com.inductiveautomation.ignition.common.script.builtin.AbstractTagUtilities.storeTagHistory(AbstractTagUtilities.java:612)
at jdk.internal.reflect.GeneratedMethodAccessor211.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:190)
... 19 common frames omitted
Line 94 of the script calls the Ignition system function to storeTagHistory:
system.tag.storeTagHistory(histProv, tagProv, paths, values)
If I comment that line out - the next time the script starts - it does not fail.
Interestingly, if I uncomment the line, I have observed that it doesn't fail on the same iteration of that line that is within an endless for loop - but is usually a short duration after the 600th loop (the first ~10000 event/loops happen within the first few seconds (downloading the equipment's current state) and then slows down to 1 event per second.
I searched the forum for issues related to this and took a look at my store and forward status for this database:
I have a support ticket in with our IT guys who report that there was a full system outage yesterday. I am still trying to get more information out of them in this regard.
I also intend on submitting a ticket to IA that will reference this post.