Tracking memory heap/performance issue in httplib

We had some sort of something going on that caused our heap to reach max (2GB) and was causing all sorts of issues with HTTP requests.
image
We increased the heap max to 4GB but upon restarting the server is operating nominally at a much lower heap size, performing all the same functions it was before.
How can we go about figuring out what was running rampant should it happen again?

We were specifically getting the error below always triggered by an httplib.HTTPSConnection() call


Line 98 referenced above is this line here

The log went crazy, but we don’t show any out of memory errors yesterday

Just looking for ideas on how to track down the issue in the future vs just increasing heap or a gateway restart.

Are you making sure to close that HTTPSConnection object in a finally block?

This is my pattern I use for the requests. I suppose I may have missed a close somewhere in other calls but I don’t believe that to be the case.

conn = httplib.HTTPSConnection(settings[0].value)
try:		
	conn.request("POST", "/" + str(settings[1].value).strip() + "/api/v2/odata/WCI/Ice.BO.UD02Svc/UD02s", json.dumps(data), headers)
	response = conn.getresponse()
				
	LogRestInfo('RollupToEpicor (' + res + ') Status: ' + str(response.status))		
	status = (response.status == 201)
					
	if not status:
		LogRestWarn('RollupToEpicor (' + res + ') Status: ' + str(response.status))
except Exception, err:
	LogRestError('RollupToEpicor (' + res + ') Unable To Connect', err)
	status = False
				
conn.close()

Another thing to try would be to ditch the Jython stdlib and use Java’s HttpURLConnection or system.net.httpClient if you are 8.x.

1 Like

We are on 8.x if you had a choice would you use system.net.httpClient or HttpUrlConnection? I noted too this morning that our heap appears nominally larger starting to creep at 4am. So we are going to monitor a pattern there as well.

image

The peaks are peakier and the dips are dippier but overall increased.

Use system.net.httpClient(). Besides being significantly more ergonomic to use, it’s also capable of doing things like connection pooling and resource sharing ‘under the hood’, rather than making each HTTP request in a vacuum, as HttpUrlConnection does.

1 Like

I like the sounds of that! Thank you for the input, I’ll get my connection handling updated regardless of if they caused the issue or not. Thanks a bunch!

@PGriffith when looking through the forum at examples and pain points for this I was not able to find anyway of closing the connections. Is that not at all necessary with this java wrapper library? If I say opened a connection in a project library to our ERP system could I make all my calls within that same connection?

This is the sort of structure I have going on at the moment (pre switching to the new system.net methods)

Yes, httpClient requests are independent of each other - you don’t need to manually close any resources. It’s a best practice to create a longer-lived httpClient() instance, so that it can share resources between requests, but even if you create a new instance for each request you’re at exactly the same behavior as an independent HTTP(s) connection per request.

1 Like

Great! Thanks again for the clarity. I will get to work on switching that over. A little off this topic, but the Project Libraries, when the gateway starts are global instances of those created or do they exist as statics. Going this system.net route I’m considering if there is any value in wrapping my connections in a library that I then use from all my other libraries. It would just contain all my header generation so maybe I would say

myhttplib.PostEpicorRollup("[specific plc tree tagpath]", "[epicor instance to post to production/dev]", [data, to, post])

All my header and auth/url magic would happen in the background of that lib instead of every method like I have today. Love me some good old ecapsulation :smiley:

True static instances won’t be created - anything in your project libraries is subject to being reloaded whenever changes are made to the scripting library. This shouldn’t be an issue for your purposes; you could ‘cache’ an httpClient() instance in your script and only expose public API methods. If you do genuinely need something static for the lifetime of the gateway, then you can use system.util.globals:
https://docs.inductiveautomation.com/display/DOC81/system.util.getGlobals
‘Soft-caching’ the client would look something like this:

1 Like

I would agree with this statement.

Ran some prototype tests using webhook.site and I think this is going to work great! Thanks for all the help!

Sorry another question re: async in this case. Looking at the example in the documentation if my understanding is correct it an async call would be effectively turned into synchronous by waiting for the response from the promise.

Is there a way to get the response or status codes from within my callback function. I started poking at the arguments identified in the documentation but I’m not seeing a launching off point to get status.

I really only want to log it as I really won’t have anything to handle, just want to see when it goes BOOM and be able to define what “BOOM” is. Epicor’s implementation of REST can be a bit off the beaten path so I have to handle status codes for REST calls in a “non standard” fashion occasionally.

Below is where I am at and the output I’m seeing in my callback.

Yes, the callback function will get a Response object, the same that you’d get from calling the blocking version of the function, so you can use the same attributes/methods to inspect it.

Ohhhhh man do I feel dumb now ha ha h! I see why I was confused now. I figured I would post my confusion in case anyone else slips on this in the future.

I was expecting to see my callback output here when testing

However, duh me, it’s a different thread so if I check the console it’s there the way I expect it to be.

:flushed::flushed::flushed: I guess I just needed a new day and more coffee.

From a support standpoint an example using a callback vs the .get() and what arguments to expect incoming to the def might be a nice addition.
https://docs.inductiveautomation.com/display/DOC80/system.net.httpClient

Thanks again for the assist great help as always :smiley:

1 Like

Just to clarify further is the .get() method actually a blocking method like I first thought? It looks like it’s occurring in the single worker thread not the main thread as it will show up in the console under a thread ID but will not show in the main log


Ultimately everything is flowing the way I would like but I’m trying to figure out how to get the async method info into the main web log. The part outlined below

Context sensitive parts of the script

# Library scope vars
showInfoMessages = True
showWarnMessages = True
printToConsole = False
httpClient = system.net.httpClient()

# Peform async call
def EpicorGetAsync(reqname, res, endpoint, payload={}, getvars={}):
	LogRestInfo(reqname + ' (' + res + ') Triggered')
	promise = httpClient.getAsync(url = endpoint, data = payload, params = getvars)
	promise.whenComplete(AsyncCallback)
	LogRestInfo(reqname + ' (' + res + ') Promised ' + str(promise))
	return promise
#End EpicorPostAsync()

# Handles asyn callback logging
def AsyncCallback(response, error):
	apiinfo = json.loads(response.getHeaders()['apikey'][0])
	service = apiinfo['Service']
	method = apiinfo['Method']
	query = apiinfo['QueryID']
	reqname = "Promised " + service + "->" + method + "->" + query
	if response.good:
		LogRestInfo(reqname + ' (' + res + ') Status: ' + str(response.statusCode))
	else:
		LogRestInfo(reqname + ' (' + res + ') Status: ' + str(response.statusCode), err)
#End AsyncCallback


# Condensed log handlers
def LogRestInfo(msg):
	if showInfoMessages:
		logger = system.util.logger("Epicor Rest Logger")
		logger.info(msg)
		if printToConsole:
			print(msg)
#End LogRestInfo()

def LogRestWarn(msg):
	if showWarnMessages:
		logger = system.util.logger("Epicor Rest Logger")
		logger.warn(msg)
		if printToConsole:
			print(msg)
#End LogRestWarn()
	
def LogRestError(msg, err):
	logger = system.util.logger("Epicor Rest Logger")
	logger.error(msg + ': ' + str(err))
	if printToConsole:
		print(msg + ': ' + str(err))
#End LogRestError()

Promise.get() blocks execution in whatever script context it gets invoked in. How are you actually triggering these test calls? SwingWorker is not what I would expect the http client’s background worker threads to be named.

In those cases I was using the script console
image

I’m pretty sure the script console is the source of the SwingWorker threads; we don’t run script console execution on the EDT since accidental infinite loops are pretty common. You could try calling get() on your wrapped PostAsync function - it will still execute your logging statements on a background thread, but will block execution in the script console until things complete.

Fair point and guilty as charged on the infinite loops lol

End game is to have this EpicorPostAsync wrapped called very minute to roll up and ship tag history info to Epicor in a specific format that I can then consume over in that system.

It does this for every PLC in our tag tree

I wanted to fire off all the POSTs to Epicor as async but still show in the web log debugging messages or errors that might have occured in that process. Sort of like how I have this tracing built.