What is the best way to include a "rate limit" for scripts that loop through an api to pull data into a dataset in the report module? Or even just a way to limit requests in general? Can a hard request limit be set in the gateway to limit more than x requests to specific / any external urls in a given time frame?
could you provide a bit more context on the script? Is it a gateway script, project library script, scripting on a component?
An example of the code you already have would be helpful too!
Certainly, hopefully this is useful:
Image;
def updateData(data, sample):
dt = system.date.format(system.date.now(), 'yyyy-MM-dd')
url = "#####example#URL##########".format(2025)
result = system.net.httpGet(url, username='ignition', password='ZYcSqz13Yv', useCaches = False)
datasource = system.util.jsonDecode(result)
max_analysis_results = [0]
def sortOrders(data):
product = data['InventoryTransaction']['Product']['Name']
pounds = data['InventoryTransaction']['Pounds']
quantity = data['InventoryTransaction']['Quantity']
base_data = [product, pounds, quantity]
return base_data
return None, None
rawOrders = [sortOrders(dataset) for dataset in datasource]
rawOrders = [item for item in rawOrders if item[0] is not None]
headers = ["Product", "Pounds", "Quantity" ]
order = []
for base_data in rawOrders:
row = base_data
order.append(row)
data['OrderInfo'] = system.dataset.toDataSet(headers,order)
There's nothing in Ignition that will do this for you.
For data you need periodically refreshed consider a gateway timer or scheduled script that calls it periodically and puts the results into a document tag or something.
p.s. your password showing in both your screenshot and script text.
2 Likes
The shared code is one example of the suspected API throttling, it does a data request from a company built and maintained API that serves as a data handler, but I believe the nature of how it updates or runs is causing problems. IT says they are experiencing major throttling issues all coming from ignition requests, and while it is difficult for me to know whether these types of scripts are responsible or not, I am hoping for some kind of solution that can just straight out time lock the number of requests Ignition can push out in a given time frame.
Not even something (as arduous as it would be) that could be added to each request script similar to this to prevent it from actually running/ querying the API too often?
This API poll is only getting called once per report generation, so unless you have multiple reports generated per second, this specific script is likely not the source of your throttling.
Anything you want to invent to solve this can be added to it, but there's nothing ready to go for you.
But @ryan.white is on to something here -- you should probably better understand what your project is doing and where these APIs are being called from and how often.
How are they determining this? Can you get an explicit list of the API endpoints that are being throttled from your IT group? That might be able to help you narrow down what scripts are the cause.
Otherwise you will need to crawl all your library, gateway, and client scripts to see which ones are touching APIs and determine if any are doing so on a tight loop/high rate.
There are only 2 gateway events that are "scheduled" (run more than once a day) and they both only trigger hourly, so I don't see it being that either, there are project library scripts that allow "shortcutting" to access API of specific topics without having to fully key in, but those would also only run when called so...
if you solve some of the other issues that everyone else mentioned, it may also be useful to move some of the logic to a project library script or gateway event where timers are easier to implement.
I've tested around with using invokeLater in a function to call itself with some delay to implement a timer. But I'm not sure if this is the best approach
I'm not so patiently awaiting them to fulfill my request for such a list, and my apologies for being rather uninformed. I was a PLC programmer and HMI designer prior to this job, so everything I know about Python, network, scripting, etc. has been learned on the go.
How much data on average are you pulling per request from this API(rough row count)? Is there a possibility that this API can't handle large amounts of data well so it has a very low threshold for rate limiting?
How many project instances do you have active at a time normally? These could add up if multiple people request something around the same time. (Would still have to be a lot and often)
Everybody starts somewhere. I'm in a similar boat, just with ~5 years more time in it than you. There's still so much I don't know about either.
IT should be able to tell you exactly what endpoints you're hitting, where the requests are coming from (and if there are multiple Ignition Gateways, or just one), how many requests, etc...
If they can't, well... IT should never be trusted in the first place.
2 Likes
The amount of data differs wildly, some calls pull only 5 rows for 3 columns, some pull 450 rows for 10 columns, I'd say a safe average is about 30 rows for 5 columns. It definitely has a bit of a lower to my understanding it was mostly built to "push" data as opposed to ignition "pulling?" on it,
Pardon my arrogance again, but what do you mean when you say project instances? Number of scripts in the library, or number of dashboards or reports?
Number of Perspective Sessions and/or Vision clients active at once with access to the api.
My approach would be:
- Audit your projects, identify everywhere that might be making an API call. Search your projects for "httpGet" (or perhaps "system.net.http").
- If multiple scripts (>1) are making the same API request, move all of them to reference a single function from a common location (Project Library, under the Gateway Scripting Project, for example).
- Add logging (system.util.getLogger) within your script, so that you can temporarily change the logging level and see how many times you are (or, Ignition is) calling the function & API, perhaps also any other useful information.
- This:
- Keep being a squeaky wheel with IT, especially after confidence in findings above. Have them watch as you issue a single request, see if an improperly configured router (or load balancer) is creating issues.