Error httpclient on reset trial

Ignition
1

When the trail expires on our gateway, and we do a reset, we keep getting these errors
when we try to to a httpclient call, polled by a gatewayscript.

I cant even reset the service without force ending it with taskmanager

we just use these for local tests, but its kinda annoying if we gotta reset the server every 2h

Any idea how to fix this?

version 8.1.14

INFO   | jvm 1    | 2022/07/04 15:28:55 | E [g.s.SystemUtilities           ] [13:28:55]: Error running function from system.util.invokeAsynchronous 
INFO   | jvm 1    | 2022/07/04 15:28:55 | com.inductiveautomation.ignition.common.script.JythonExecException: Traceback (most recent call last):
INFO   | jvm 1    | 2022/07/04 15:28:55 |   File "<module:handleResetA>", line 92, in removeBlackBox
INFO   | jvm 1    | 2022/07/04 15:28:55 |   File "<module:handleResetA>", line 55, in getCarriersBlackbox
INFO   | jvm 1    | 2022/07/04 15:28:55 |   File "<module:handleResetA>", line 55, in getCarriersBlackbox
INFO   | jvm 1    | 2022/07/04 15:28:55 | IOError: Address already in use: connect
INFO   | jvm 1    | 2022/07/04 15:28:55 | 
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.Py.fromIOException(Py.java:216)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.Py.IOError(Py.java:183)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at com.inductiveautomation.ignition.common.script.builtin.AbstractNetUtilities.httpGet(AbstractNetUtilities.java:274)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at jdk.internal.reflect.GeneratedMethodAccessor72.invoke(Unknown Source)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:190)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at com.inductiveautomation.ignition.common.script.ScriptManager$ReflectedInstanceFunction.__call__(ScriptManager.java:541)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyObject.__call__(PyObject.java:433)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyObject.__call__(PyObject.java:437)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.pycode._pyx2572.getCarriersBlackbox$4(<module:handleResetA>:61)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.pycode._pyx2572.call_function(<module:handleResetA>)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyTableCode.call(PyTableCode.java:173)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyBaseCode.call(PyBaseCode.java:119)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyFunction.__call__(PyFunction.java:406)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.pycode._pyx2572.removeBlackBox$8(<module:handleResetA>:93)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.pycode._pyx2572.call_function(<module:handleResetA>)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyTableCode.call(PyTableCode.java:173)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyBaseCode.call(PyBaseCode.java:306)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyFunction.function___call__(PyFunction.java:474)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyFunction.__call__(PyFunction.java:469)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyFunction.__call__(PyFunction.java:464)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at com.inductiveautomation.ignition.common.script.ScriptManager.runFunction(ScriptManager.java:831)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at com.inductiveautomation.ignition.gateway.script.GatewaySystemUtilities.lambda$_invokeAsyncImpl$0(GatewaySystemUtilities.java:142)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at java.base/java.lang.Thread.run(Unknown Source)
INFO   | jvm 1    | 2022/07/04 15:28:55 | Caused by: org.python.core.PyException: IOError: Address already in use: connect
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	... 25 common frames omitted
INFO   | jvm 1    | 2022/07/04 15:28:55 | E [g.s.SystemUtilities           ] [13:28:55]: Error running function from system.util.invokeAsynchronous 
INFO   | jvm 1    | 2022/07/04 15:28:55 | com.inductiveautomation.ignition.common.script.JythonExecException: Traceback (most recent call last):
INFO   | jvm 1    | 2022/07/04 15:28:55 |   File "<module:handleResetA>", line 84, in removeWashInOne
INFO   | jvm 1    | 2022/07/04 15:28:55 |   File "C:****\model\location.py", line 35, in getLocation
INFO   | jvm 1    | 2022/07/04 15:28:55 |     response = client.get(url="%s/%s" %(self.base_url, location_id), headers=self.headers)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at com.inductiveautomation.ignition.common.script.builtin.http.JythonHttpClient.send(JythonHttpClient.java:103)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at com.inductiveautomation.ignition.common.script.builtin.http.JythonHttpClient.get(JythonHttpClient.java:306)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at jdk.internal.reflect.GeneratedMethodAccessor86.invoke(Unknown Source)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
INFO   | jvm 1    | 2022/07/04 15:28:55 | java.io.IOException: java.io.IOException: Unable to GET https://****/locations/WASH_IN_1
INFO   | jvm 1    | 2022/07/04 15:28:55 | 
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.Py.JavaError(Py.java:547)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.Py.JavaError(Py.java:538)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:192)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyObject.__call__(PyObject.java:422)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyMethod.instancemethod___call__(PyMethod.java:237)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyMethod.__call__(PyMethod.java:228)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at ****.model.location$py.getLocation$4(C:****/model/location.py:42)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at ****.model.location$py.call_function(C:****/model/location.py)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyTableCode.call(PyTableCode.java:173)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyBaseCode.call(PyBaseCode.java:150)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyFunction.__call__(PyFunction.java:426)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyMethod.__call__(PyMethod.java:141)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.pycode._pyx2572.removeWashInOne$7(<module:handleResetA>:86)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.pycode._pyx2572.call_function(<module:handleResetA>)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyTableCode.call(PyTableCode.java:173)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyBaseCode.call(PyBaseCode.java:306)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyFunction.function___call__(PyFunction.java:474)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyFunction.__call__(PyFunction.java:469)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyFunction.__call__(PyFunction.java:464)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at com.inductiveautomation.ignition.common.script.ScriptManager.runFunction(ScriptManager.java:831)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at com.inductiveautomation.ignition.gateway.script.GatewaySystemUtilities.lambda$_invokeAsyncImpl$0(GatewaySystemUtilities.java:142)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at java.base/java.lang.Thread.run(Unknown Source)
INFO   | jvm 1    | 2022/07/04 15:28:55 | Caused by: org.python.core.PyException: java.io.IOException: java.io.IOException: Unable to GET https://****/locations/WASH_IN_1
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	... 22 common frames omitted
INFO   | jvm 1    | 2022/07/04 15:28:55 | Caused by: java.io.IOException: Unable to GET https://****/locations/WASH_IN_1
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at com.inductiveautomation.ignition.common.script.builtin.http.JythonHttpClient.send(JythonHttpClient.java:103)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at com.inductiveautomation.ignition.common.script.builtin.http.JythonHttpClient.get(JythonHttpClient.java:306)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at jdk.internal.reflect.GeneratedMethodAccessor86.invoke(Unknown Source)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java:190)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	... 19 common frames omitted
INFO   | jvm 1    | 2022/07/04 15:28:55 | Caused by: java.net.ConnectException: Address already in use: no further information
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at java.net.http/jdk.internal.net.http.HttpClientImpl.send(Unknown Source)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at java.net.http/jdk.internal.net.http.HttpClientFacade.send(Unknown Source)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at com.inductiveautomation.ignition.common.script.builtin.http.JythonHttpClient.send(JythonHttpClient.java:101)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	... 24 common frames omitted
INFO   | jvm 1    | 2022/07/04 15:28:55 | Caused by: java.net.ConnectException: Address already in use: no further information
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at java.net.http/jdk.internal.net.http.common.Utils.toConnectException(Unknown Source)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at java.net.http/jdk.internal.net.http.PlainHttpConnection$ConnectEvent.handle(Unknown Source)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.handleEvent(Unknown Source)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.lambda$run$3(Unknown Source)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at java.base/java.util.ArrayList.forEach(Unknown Source)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.run(Unknown Source)
INFO   | jvm 1    | 2022/07/04 15:28:55 | Caused by: java.net.BindException: Address already in use: no further information
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
INFO   | jvm 1    | 2022/07/04 15:28:55 | 	... 5 common frames omitted
2

Sounds like the target of your call doesn’t support multiple connections. Since you are using invokeAsynchronous, that suggests that you are also failing to kill off prior long-running scripts when scripting restarts (which happens on trial reset).

1 Like
3

yeah i can see that, the asycn function is just looping through some calls to simulate some api calls (at high speed)

but ill have to look into this, that would be a quite a problem

4

So its nothing the do with the asycn.

The gateway keeps the tcp sockets open at status CLOSE_WAIT…
just opening a client causes 100’s of ports to get stuck

5

Show your code.

6

im guessing this is the relevant bit… im thinking maybe im not supposed to do client = system.net.httpClient() i na loop?

   for loc in storage_units:
         if loc['row_id'] not in instances:
            lock=0
            try:
            	client = system.net.httpClient()
                response = client.get(url="%s/storage-units/%s/rows/%s" % (base_url,storage_unit,loc['row'] ), headers=headers)				
                if response.good:
                	if "lock_code" in response.json:
                	   lock = response.json["lock_code"]
7

BTW, are you making a new httpClient object for every call? If so, don’t. It is designed to be reused.

8

yup xd
guess thats it…

kinda weird its designed it like that though no?
especialy since there is no way to close the client

9

It is supposed to be thread-safe, too. Consider making one as a top-level object in a project script module, and using that one everywhere in that project.

1 Like
10

Not really. Modern web frameworks use a variety of connection reuse strategies that can't be supported if there is no place to hold state.

Think of "httpClient" as the transport part of a browser, and each .get() as the network part of opening a tab.

11

yea i mean,
i get how it works in java
but for the system function: system.net.httpClient()
Wouldnt it be easier to use if it just like grabbed the one httpclient from the gateway system xd
idk nvm i guess you can now create clients with different settings this way.

I am not sure why there is no way to close off the client tho… no disconnect function for something that hogs sockets?

12

Try upgrading to at least 8.1.15. See the end of this release note:

Added a version argument to system.net.httpClient() system function to allow specifying either HTTP_2 (for http/2 first with fallback to http/1.1) or HTTP_1_1 (for explicit use of http/1.1) for the HTTP protocol. When omitted, the previous default of HTTP_2 is implied. Also verified during the addition of this version argument was the 8.1.15 update to 11.0.14.1 JDK enabling periodic garbage collection to successfully reap closed connections where they may have been left lingering on previous versions (when http/2 was being used).

2 Likes
13

so upgrade to 8.1.15 and set the httpclient version to 1.1?

14

Well, upgrade to the latest, not just 8.1.15. Don’t set the version at all.

15

alright updated to 8.1.18
all see if this fixes the sockets^^

so it seems the sockets might still be getting used, but im not running into errors yet for now,
so i’ll mark it as solved for now for aslong i dont run into trouble again xP