Thank you everyone for the suggestions. This issue was resolved by using
# set client = system.net.httpClient() somewhere outside of the function definition scope below
url = payload["url"]
headers = {
"Accept": "application/json;charset=UTF-8"
}
# Request with custom headers
response = client.get(url, headers=headers)
I was using a BufferedReader and then a system.util.jsonDecode("".join(content)) before.
Both of these were causing significant memory issues instead of receiving the data as true JSON.
I also implemented the following code.