How do I close an httpClient properly?
I cannot find a way to close an httpClient after an exception occurs.
Here’s my issue:
I have a Gateway Timer script that creates a httpClient and then calls post(). And then I parse the response. During the parsing, an Exception is thrown. This is fine with me and the script just runs again at next timer.
However, after a few hours or so, get the following error in my logs
com.inductiveautomation.ignition.common.script.JythonExecException: java.lang.InternalError: java.lang.InternalError: java.io.IOException: Unable to establish loopback connection
and somewhere in the middle of the trace I see
at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.(Unknown Source)
... 33 common frames omitted
Caused by: java.net.SocketException: No buffer space available (maximum connections reached?): bind
My guess is that the exception is causing the httpClient to not close properly.
I have tried setting the httpClient timeout to shorter period, but it still happens.
I can reproduce the error with this function (called by Gateway Timer event), and letting it run for hours.
def http_force_exception():
client = system.net.httpClient(3000)
body = {'hello': 'world'}
response = client.post('http://blahblahblah', data=body)
fake_str = response.json['fake_key'] #fake_key does not exist and causes an Exception.
What’s the rate on your timer script?
I’m not seeing instances of HttpClient accumulating or anything, so I’m guessing the issue is that by doing this you are opening ephemeral ports faster than your OS/TCP stack is closing them. I’ve seen that Windows is particularly slow when it comes to letting them go.
You can try launching your gateway with a value specified for jdk.internal.httpclient.selectorTimeout (in the wrapper.java.additional.params section of ignition.conf). It’s the time in milliseconds between runs of the HTTPClient’s ‘reaping’ action. The default value is 3000 (3 seconds), but lowering it might help (if the issue is actually a pseudo-socket leak.
Also, as a generally good practice, you should instantiate your httpClient() instance outside of the function. It can just be a top-level variable in your project library and referenced in http_force_exception().
At first, I was calling several of these every few seconds (variations of different posts()). And I noticed the java.io.IOException every few days or so.
In order to reproduce the error, I started calling http_force_exception() every 50ms. This caused the error within 1-3 hours.
I thought it may be that I was opening them faster than it was being closed, so I tried this test:
When the java.io.IOException happened, I stopped the timer script and waited for half an hour, hoping that all opened ports will be closed automatically. Then I triggered one single http_force_exception(). But java.io.IOException still happened on that single call.
The only way I could create any more httpClients is to restart the Ignition Server.
Regarding your suggestion to instantiate httpClient() outside of the function, does it still work if different timer threads call the same function at the same time?
E.g. If I have 3 Timer Scripts (or Tag Change Scripts) that may run simultaneously.
Script 1: calls func_1(data_to_post1)
Script 2: calls func_1(data_to_post2) as well
Script 3: calls func_3()
client1 = system.net.httpClient()
def func_1(data_to_post):
response = client1.post('http://blahblahblah', data=data_to_post)
# handle the response
client3 = system.net.httpClient()
def func_3():
response = client3.post('http://blahblahblah', data='blah')
# handle the response
In the above case, is client1 able to handle two separate calls at the same time?
If not, then what would be the preferred way to handle this?
If yes, then does it mean client3 is not needed, since I can use client1?
Thank you for your help!
Yes, I would expect that to be absolutely fine.
Yes.
Thank you, I will give that a try.
Not sure if you ever solved this or found a workaround, but 8.1.16 contains a fix that might resolve this issue.
The release note was:
Added a version argument to system.net.httpClient() system function to allow specifying either HTTP_2 (for http/2 first with fallback to http/1.1) or HTTP_1_1 (for explicit use of http/1.1) for the HTTP protocol. When omitted, the previous default of HTTP_2 is implied. Also verified during the addition of this version argument was the 8.1.15 update to 11.0.14.1 JDK enabling periodic garbage collection to successfully reap closed connections where they may have been left lingering on previous versions (when http/2 was being used).
It's the JDK update mentioned that may help. I guess technically the JDK update was part of 8.1.15 but not sure if there's a release note that might help make the association with this issue.
Thank you for following up.
Previously, I had
client = system.net.httpClient()
and response = client.post('http://blahblahblah', data=body)
inside a function that was regularly called. And that seemed to have caused problems.
Now, I moved client = system.net.httpClient() outside the function to module level, and still kept response = client.post('http://blahblahblah', data=body) inside the function and the issue disappeared.
I think I’ll keep it like this for now.
Thank you for the update and suggestion though!
Good call… preserving the instance of the client will definitely help workaround the issue (and also benefit from reused connections w/ http/2).