Hi everyone,
I have a set of scripts that pull in JSON-based data from three API endpoints (10,000 records max per endpoint). That data is then transmitted over and stored on custom properties on a perspective component before they are joined together and displayed to a table.
During the loading step of this process, Ignition crashes due to the amount of memory allocation. Has someone encountered something similar before? If so, what is the recommended approach to avoid crashes?
Do you have more than 1 client doing this? That would cause a lot more load on the gateway.
Are the datasets the same for each client aka would it be feasible to just do this in a gateway timer script, put the datasets into a gateway dataset tag and have your perspectice clients bound to that? If you do have multiple clients and what I am describing is feasible (or even if the clients just need a specific subset of the main dataset) you could instantly cut down the memory of 10k records x 3 datastes x N clients to just 10k records x 3 datasets in the gateway (constant no matter how many clients).
Hopefully the clients just need a subset of the records such that you can query them once in the gateway and then when you read from your perspective client - that is where you just filter the gateway dataset tag. This is a great use case for @pturmel ‘s integration toolkit. You can filter datsets with expressions. Alternatively you can script it yourself.
I think you'll have to share more information about what your processing/parsing step (your script) looks like, and what the shape/format of the data is.
...and prepare for the possibility the answer is "don't do that", i.e. receive less from the API at once.
Watch your memory usage. If you are still running the 2G default in ignition.conf, and built a busy system, it might be riding on the bleeding edge all the time, and this operation just breaks the camel's back.
If so, you might just need to allow Ignition to use more memory.