Script difference console vs gateway

I have a script that runs fine in the script console but adding it as a gateway (tag change) script the result is different.

I think this is due to the script console using a single thread whereas the gateway uses multithreading.

I tried using invokeAsync in the gateway but even though I’m using S+F it drops results, log shows “java.lang.Exception: Unable to add data to memory buffer, buffer full.”

My script is split into two parts, first I open a log file looping through 5 machines over LAN, split the CSV and insert into SQL (S+F).

The next part is to compare the timestamps of each record in the files and calculate the delta. Before using Invoke, I think the two for loops were running at the same time in a separate thread.

Each file has around 15K lines per day (I plan to run this script once at 00:30:00 to check yesterday’s data) and currently I’m looking at 5 machines but that will likely increase.

What’s the best way to move forward?

One suggestion I can offer is to double check that the functions you’re using don’t need additional clarification when used on the gateway. On something like the script console, you’re already attached to some things by default. On the gateway, there are more options and a default isn’t used. Some examples of this escape me at the moment but I know this same problem has caused me a few headaches before I realized what was going on.

OK thanks, by functions you mean any system.XX.YY etc?

The only one I’m using in the first half is system.db.runSFPrepUpdate, for which the user manual states “This function is unaffected when run in the Gateway scope.” I already was explicitly naming what DB it inserts to.

It’s working fine in the script console with the invoke, just if I can get round this buffer filling up when executed from the gateway? I get about 3-4K out of 15K inserted the rest are dropped.

Two that come to mind are

  1. Make sure you specify the database connection in the SQL command.
    system.db.runSFPrepUpdate(‘BlahBlah’,‘NameOfDatabaseConnection’)
  2. Make sure you specify the tag source on any tag read and writes.
    system.tag.read("[default]PathToMyTag").value

Yep both OK.

Full error log event:

java.lang.Exception: Unable to add data to memory buffer, buffer full.
at com.inductiveautomation.ignition.gateway.history.stores.MemoryStore.insertInBuffer(MemoryStore.java:170)
at com.inductiveautomation.ignition.gateway.history.stores.MemoryStore.syncdStoreData(MemoryStore.java:77)
at com.inductiveautomation.ignition.gateway.history.stores.AbstractStore.storeData(AbstractStore.java:225)
at com.inductiveautomation.ignition.gateway.history.stores.MultiStageStore.syncdStoreData(MultiStageStore.java:157)
at com.inductiveautomation.ignition.gateway.history.stores.AbstractStore.storeData(AbstractStore.java:225)
at com.inductiveautomation.ignition.gateway.history.DefaultStoreAndForwardEngine.storeData(DefaultStoreAndForwardEngine.java:145)
at com.inductiveautomation.ignition.gateway.history.HistoryManagerImpl.storeHistory(HistoryManagerImpl.java:196)
at com.inductiveautomation.ignition.gateway.script.ialabs.IALabsDatasourceFunctions.runSFPrepUpdateImpl(IALabsDatasourceFunctions.java:174)
at com.inductiveautomation.ignition.common.script.builtin.ialabs.AbstractIALabsDatasourceFunctions.runSFPrepUpdate(AbstractIALabsDatasourceFunctions.java:63)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.python.core.PyReflectedFunction.call(PyReflectedFunction.java:186)
at com.inductiveautomation.ignition.common.script.ScriptManager$ReflectedInstanceFunction.call(ScriptManager.java:431)
at org.python.core.PyObject.call(PyObject.java:422)
at org.python.core.PyObject.call(PyObject.java:426)
at org.python.pycode.pyx2.longProcess$1(:48)
at org.python.pycode.pyx2.call_function()
at org.python.core.PyTableCode.call(PyTableCode.java:165)
at org.python.core.PyBaseCode.call(PyBaseCode.java:301)
at org.python.core.PyFunction.function___call
(PyFunction.java:376)
at org.python.core.PyFunction.call(PyFunction.java:371)
at org.python.core.PyFunction.call(PyFunction.java:361)
at org.python.core.PyFunction.call(PyFunction.java:356)
at com.inductiveautomation.ignition.common.script.ScriptManager.runFunction(ScriptManager.java:649)
at com.inductiveautomation.ignition.gateway.script.GatewaySystemUtilities$1.run(GatewaySystemUtilities.java:69)
at java.lang.Thread.run(Unknown Source)

How large do you have the Store and Forward buffer set to?
I would try bumping it up to like 2500 and see if that handles it.

Bingo, thanks! Didn’t know you could adjust that, so it was at default (250).