We had some sort of something going on that caused our heap to reach max (2GB) and was causing all sorts of issues with HTTP requests.
We increased the heap max to 4GB but upon restarting the server is operating nominally at a much lower heap size, performing all the same functions it was before.
How can we go about figuring out what was running rampant should it happen again?
We were specifically getting the error below always triggered by an httplib.HTTPSConnection() call
Line 98 referenced above is this line here
The log went crazy, but we donât show any out of memory errors yesterday
Just looking for ideas on how to track down the issue in the future vs just increasing heap or a gateway restart.
Are you making sure to close that HTTPSConnection object in a finally block?
This is my pattern I use for the requests. I suppose I may have missed a close somewhere in other calls but I donât believe that to be the case.
conn = httplib.HTTPSConnection(settings[0].value)
try:
conn.request("POST", "/" + str(settings[1].value).strip() + "/api/v2/odata/WCI/Ice.BO.UD02Svc/UD02s", json.dumps(data), headers)
response = conn.getresponse()
LogRestInfo('RollupToEpicor (' + res + ') Status: ' + str(response.status))
status = (response.status == 201)
if not status:
LogRestWarn('RollupToEpicor (' + res + ') Status: ' + str(response.status))
except Exception, err:
LogRestError('RollupToEpicor (' + res + ') Unable To Connect', err)
status = False
conn.close()
Another thing to try would be to ditch the Jython stdlib and use Javaâs HttpURLConnection or system.net.httpClient if you are 8.x.
1 Like
We are on 8.x if you had a choice would you use system.net.httpClient or HttpUrlConnection? I noted too this morning that our heap appears nominally larger starting to creep at 4am. So we are going to monitor a pattern there as well.
The peaks are peakier and the dips are dippier but overall increased.
Use system.net.httpClient()
. Besides being significantly more ergonomic to use, itâs also capable of doing things like connection pooling and resource sharing âunder the hoodâ, rather than making each HTTP request in a vacuum, as HttpUrlConnection does.
1 Like
I like the sounds of that! Thank you for the input, I'll get my connection handling updated regardless of if they caused the issue or not. Thanks a bunch!
@PGriffith when looking through the forum at examples and pain points for this I was not able to find anyway of closing the connections. Is that not at all necessary with this java wrapper library? If I say opened a connection in a project library to our ERP system could I make all my calls within that same connection?
This is the sort of structure I have going on at the moment (pre switching to the new system.net methods)
Yes, httpClient requests are independent of each other - you donât need to manually close any resources. Itâs a best practice to create a longer-lived httpClient()
instance, so that it can share resources between requests, but even if you create a new instance for each request youâre at exactly the same behavior as an independent HTTP(s) connection per request.
1 Like
Great! Thanks again for the clarity. I will get to work on switching that over. A little off this topic, but the Project Libraries, when the gateway starts are global instances of those created or do they exist as statics. Going this system.net route Iâm considering if there is any value in wrapping my connections in a library that I then use from all my other libraries. It would just contain all my header generation so maybe I would say
myhttplib.PostEpicorRollup("[specific plc tree tagpath]", "[epicor instance to post to production/dev]", [data, to, post])
All my header and auth/url magic would happen in the background of that lib instead of every method like I have today. Love me some good old ecapsulation
True static instances won't be created - anything in your project libraries is subject to being reloaded whenever changes are made to the scripting library. This shouldn't be an issue for your purposes; you could 'cache' an httpClient()
instance in your script and only expose public API methods. If you do genuinely need something static for the lifetime of the gateway, then you can use system.util.globals
:
https://docs.inductiveautomation.com/display/DOC81/system.util.getGlobals
'Soft-caching' the client would look something like this:
1 Like
I would agree with this statement.
Ran some prototype tests using webhook.site and I think this is going to work great! Thanks for all the help!
Sorry another question re: async in this case. Looking at the example in the documentation if my understanding is correct it an async call would be effectively turned into synchronous by waiting for the response from the promise.
Is there a way to get the response or status codes from within my callback function. I started poking at the arguments identified in the documentation but Iâm not seeing a launching off point to get status.
I really only want to log it as I really wonât have anything to handle, just want to see when it goes BOOM and be able to define what âBOOMâ is. Epicorâs implementation of REST can be a bit off the beaten path so I have to handle status codes for REST calls in a ânon standardâ fashion occasionally.
Below is where I am at and the output Iâm seeing in my callback.
Yes, the callback function will get a Response
object, the same that youâd get from calling the blocking version of the function, so you can use the same attributes/methods to inspect it.
Ohhhhh man do I feel dumb now ha ha h! I see why I was confused now. I figured I would post my confusion in case anyone else slips on this in the future.
I was expecting to see my callback output here when testing
However, duh me, itâs a different thread so if I check the console itâs there the way I expect it to be.
I guess I just needed a new day and more coffee.
From a support standpoint an example using a callback vs the .get() and what arguments to expect incoming to the def might be a nice addition.
https://docs.inductiveautomation.com/display/DOC80/system.net.httpClient
Thanks again for the assist great help as always
1 Like
Just to clarify further is the .get() method actually a blocking method like I first thought? It looks like itâs occurring in the single worker thread not the main thread as it will show up in the console under a thread ID but will not show in the main log
Ultimately everything is flowing the way I would like but Iâm trying to figure out how to get the async method info into the main web log. The part outlined below
Context sensitive parts of the script
# Library scope vars
showInfoMessages = True
showWarnMessages = True
printToConsole = False
httpClient = system.net.httpClient()
# Peform async call
def EpicorGetAsync(reqname, res, endpoint, payload={}, getvars={}):
LogRestInfo(reqname + ' (' + res + ') Triggered')
promise = httpClient.getAsync(url = endpoint, data = payload, params = getvars)
promise.whenComplete(AsyncCallback)
LogRestInfo(reqname + ' (' + res + ') Promised ' + str(promise))
return promise
#End EpicorPostAsync()
# Handles asyn callback logging
def AsyncCallback(response, error):
apiinfo = json.loads(response.getHeaders()['apikey'][0])
service = apiinfo['Service']
method = apiinfo['Method']
query = apiinfo['QueryID']
reqname = "Promised " + service + "->" + method + "->" + query
if response.good:
LogRestInfo(reqname + ' (' + res + ') Status: ' + str(response.statusCode))
else:
LogRestInfo(reqname + ' (' + res + ') Status: ' + str(response.statusCode), err)
#End AsyncCallback
# Condensed log handlers
def LogRestInfo(msg):
if showInfoMessages:
logger = system.util.logger("Epicor Rest Logger")
logger.info(msg)
if printToConsole:
print(msg)
#End LogRestInfo()
def LogRestWarn(msg):
if showWarnMessages:
logger = system.util.logger("Epicor Rest Logger")
logger.warn(msg)
if printToConsole:
print(msg)
#End LogRestWarn()
def LogRestError(msg, err):
logger = system.util.logger("Epicor Rest Logger")
logger.error(msg + ': ' + str(err))
if printToConsole:
print(msg + ': ' + str(err))
#End LogRestError()
Promise.get()
blocks execution in whatever script context it gets invoked in. How are you actually triggering these test calls? SwingWorker
is not what I would expect the http clientâs background worker threads to be named.
In those cases I was using the script console
Iâm pretty sure the script console is the source of the SwingWorker threads; we donât run script console execution on the EDT since accidental infinite loops are pretty common. You could try calling get()
on your wrapped PostAsync
function - it will still execute your logging statements on a background thread, but will block execution in the script console until things complete.
Fair point and guilty as charged on the infinite loops lol
End game is to have this EpicorPostAsync wrapped called very minute to roll up and ship tag history info to Epicor in a specific format that I can then consume over in that system.
It does this for every PLC in our tag tree
I wanted to fire off all the POSTs to Epicor as async but still show in the web log debugging messages or errors that might have occured in that process. Sort of like how I have this tracing built.