Ignition sudden crash

[andrea.v]

Hello,

i'm having this issue that, after afew days that the system looks healthy, suddenly stops to be responsive, becomes impossible to reach the localhost website (from the same machine that is installed) and becomes impossible to start the client locally.

If i restart the machine the error fixes itself , then reappear after a couple of days..

The screenshots of the errors are here Ignition Issue - Album on Imgur

Edit: also becomes difficult to troubleshoot because i find this issue possibly many hours after the occurance, so the log files becomes huge and i'm not really sure what to look for.

This is the wrapper:

INFO | jvm 1 | 2021/03/07 06:55:46 | I [h.d.H.ENGINE ] [05:55:46]: Checkpoint start
INFO | jvm 1 | 2021/03/07 06:55:46 | I [h.d.H.ENGINE ] [05:55:46]: checkpointClose start
INFO | jvm 1 | 2021/03/07 06:55:46 | I [h.d.H.ENGINE ] [05:55:46]: checkpointClose synched
INFO | jvm 1 | 2021/03/07 06:55:46 | I [h.d.H.ENGINE ] [05:55:46]: checkpointClose script done
INFO | jvm 1 | 2021/03/07 06:55:46 | I [h.d.H.ENGINE ] [05:55:46]: dataFileCache commit start
INFO | jvm 1 | 2021/03/07 06:55:46 | I [h.d.H.ENGINE ] [05:55:46]: dataFileCache commit end
INFO | jvm 1 | 2021/03/07 06:55:46 | I [h.d.H.ENGINE ] [05:55:46]: checkpointClose end
INFO | jvm 1 | 2021/03/07 06:55:46 | I [h.d.H.ENGINE ] [05:55:46]: Checkpoint end - txts: 649634
INFO | jvm 1 | 2021/03/07 07:50:05 | E [c.i.i.c.s.TimerScriptTask ] [06:50:05]: Error executing global timer script: Aps_E_ferry_LowRes/CameraRecord @10,000ms . Repeat errors of this type will be logged as 'debug' messages.
INFO | jvm 1 | 2021/03/07 07:50:05 | com.inductiveautomation.ignition.common.script.JythonExecException: Traceback (most recent call last):
INFO | jvm 1 | 2021/03/07 07:50:05 | File "<TimerScript:Aps_E_ferry_LowRes/CameraRecord @10,000ms >", line 8, in
INFO | jvm 1 | 2021/03/07 07:50:05 | File "user-lib\pylib\urllib2.py", line 124, in urlopen
INFO | jvm 1 | 2021/03/07 07:50:05 | return _opener.open(url, data)
INFO | jvm 1 | 2021/03/07 07:50:05 | File "user-lib\pylib\urllib2.py", line 381, in open
INFO | jvm 1 | 2021/03/07 07:50:05 | response = self._open(req, data)
INFO | jvm 1 | 2021/03/07 07:50:05 | File "user-lib\pylib\urllib2.py", line 398, in _open
INFO | jvm 1 | 2021/03/07 07:50:05 | result = self.call_chain(self.handle_open, protocol, protocol +
INFO | jvm 1 | 2021/03/07 07:50:05 | File "user-lib\pylib\urllib2.py", line 360, in call_chain
INFO | jvm 1 | 2021/03/07 07:50:05 | result = func(*args)
INFO | jvm 1 | 2021/03/07 07:50:05 | File "user-lib\pylib\urllib2.py", line 1118, in http_open
INFO | jvm 1 | 2021/03/07 07:50:05 | return self.do_open(httplib.HTTPConnection, req)
INFO | jvm 1 | 2021/03/07 07:50:05 | File "user-lib\pylib\urllib2.py", line 1093, in do_open
INFO | jvm 1 | 2021/03/07 07:50:05 | raise URLError(err)
INFO | jvm 1 | 2021/03/07 07:50:05 | urllib2.URLError: <urlopen error (-1, 'Unmapped exception: java.net.SocketException: No buffer space available (maximum connections reached?): connect')>
INFO | jvm 1 | 2021/03/07 07:50:05 |
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyException.doRaise(PyException.java:219)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.Py.makeException(Py.java:1239)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.Py.makeException(Py.java:1243)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.Py.makeException(Py.java:1247)
INFO | jvm 1 | 2021/03/07 07:50:05 | at urllib2$py.do_open$93(user-lib\pylib\urllib2.py:1112)
INFO | jvm 1 | 2021/03/07 07:50:05 | at urllib2$py.call_function(user-lib\pylib\urllib2.py)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyTableCode.call(PyTableCode.java:165)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyBaseCode.call(PyBaseCode.java:166)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyFunction.call(PyFunction.java:338)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyMethod.call(PyMethod.java:139)
INFO | jvm 1 | 2021/03/07 07:50:05 | at urllib2$py.http_open$96(user-lib\pylib\urllib2.py:1118)
INFO | jvm 1 | 2021/03/07 07:50:05 | at urllib2$py.call_function(user-lib\pylib\urllib2.py)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyTableCode.call(PyTableCode.java:165)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyBaseCode.call(PyBaseCode.java:301)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyBaseCode.call(PyBaseCode.java:194)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyFunction.call(PyFunction.java:387)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyMethod.instancemethod___call(PyMethod.java:220)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyMethod.call(PyMethod.java:211)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyMethod.call(PyMethod.java:206)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyObject._callextra(PyObject.java:527)
INFO | jvm 1 | 2021/03/07 07:50:05 | at urllib2$py._call_chain$34(user-lib\pylib\urllib2.py:357)
INFO | jvm 1 | 2021/03/07 07:50:05 | at urllib2$py.call_function(user-lib\pylib\urllib2.py)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyTableCode.call(PyTableCode.java:165)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyBaseCode.call(PyBaseCode.java:301)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyBaseCode.call(PyBaseCode.java:194)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyFunction.call(PyFunction.java:387)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyMethod.call(PyMethod.java:170)
INFO | jvm 1 | 2021/03/07 07:50:05 | at urllib2$py._open$36(user-lib\pylib\urllib2.py:403)
INFO | jvm 1 | 2021/03/07 07:50:05 | at urllib2$py.call_function(user-lib\pylib\urllib2.py)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyTableCode.call(PyTableCode.java:165)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyBaseCode.call(PyBaseCode.java:166)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyFunction.call(PyFunction.java:338)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyMethod.call(PyMethod.java:139)
INFO | jvm 1 | 2021/03/07 07:50:05 | at urllib2$py.open$35(user-lib\pylib\urllib2.py:389)
INFO | jvm 1 | 2021/03/07 07:50:05 | at urllib2$py.call_function(user-lib\pylib\urllib2.py)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyTableCode.call(PyTableCode.java:165)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyBaseCode.call(PyBaseCode.java:166)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyFunction.call(PyFunction.java:338)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyMethod.call(PyMethod.java:139)
INFO | jvm 1 | 2021/03/07 07:50:05 | at urllib2$py.urlopen$1(user-lib\pylib\urllib2.py:124)
INFO | jvm 1 | 2021/03/07 07:50:05 | at urllib2$py.call_function(user-lib\pylib\urllib2.py)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyTableCode.call(PyTableCode.java:165)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyBaseCode.call(PyBaseCode.java:301)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyBaseCode.call(PyBaseCode.java:127)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyFunction.call(PyFunction.java:317)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.pycode._pyx23.f$0(<TimerScript:Aps_E_ferry_LowRes/CameraRecord @10,000ms >:8)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.pycode._pyx23.call_function(<TimerScript:Aps_E_ferry_LowRes/CameraRecord @10,000ms >)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyTableCode.call(PyTableCode.java:165)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.PyCode.call(PyCode.java:18)
INFO | jvm 1 | 2021/03/07 07:50:05 | at org.python.core.Py.runCode(Py.java:1275)
INFO | jvm 1 | 2021/03/07 07:50:05 | at com.inductiveautomation.ignition.common.script.ScriptManager.runCode(ScriptManager.java:636)
INFO | jvm 1 | 2021/03/07 07:50:05 | at com.inductiveautomation.ignition.common.script.ScriptManager.runCode(ScriptManager.java:603)
INFO | jvm 1 | 2021/03/07 07:50:05 | at com.inductiveautomation.ignition.common.script.TimerScriptTask.run(TimerScriptTask.java:88)
INFO | jvm 1 | 2021/03/07 07:50:05 | at java.util.TimerThread.mainLoop(Unknown Source)
INFO | jvm 1 | 2021/03/07 07:50:05 | at java.util.TimerThread.run(Unknown Source)
INFO | jvm 1 | 2021/03/07 07:50:05 | Caused by: org.python.core.PyException: Traceback (most recent call last):
INFO | jvm 1 | 2021/03/07 07:50:05 | File "<TimerScript:Aps_E_ferry_LowRes/CameraRecord @10,000ms >", line 8, in
INFO | jvm 1 | 2021/03/07 07:50:05 | File "user-lib\pylib\urllib2.py", line 124, in urlopen
INFO | jvm 1 | 2021/03/07 07:50:05 | return _opener.open(url, data)
INFO | jvm 1 | 2021/03/07 07:50:05 | File "user-lib\pylib\urllib2.py", line 381, in open
INFO | jvm 1 | 2021/03/07 07:50:05 | response = self._open(req, data)
INFO | jvm 1 | 2021/03/07 07:50:05 | File "user-lib\pylib\urllib2.py", line 398, in _open
INFO | jvm 1 | 2021/03/07 07:50:05 | result = self._call_chain(self.handle_open, protocol, protocol +
INFO | jvm 1 | 2021/03/07 07:50:05 | File "user-lib\pylib\urllib2.py", line 360, in _call_chain
INFO | jvm 1 | 2021/03/07 07:50:05 | result = func(*args)
INFO | jvm 1 | 2021/03/07 07:50:05 | File "user-lib\pylib\urllib2.py", line 1118, in http_open
INFO | jvm 1 | 2021/03/07 07:50:05 | return self.do_open(httplib.HTTPConnection, req)
INFO | jvm 1 | 2021/03/07 07:50:05 | File "user-lib\pylib\urllib2.py", line 1093, in do_open
INFO | jvm 1 | 2021/03/07 07:50:05 | raise URLError(err)
INFO | jvm 1 | 2021/03/07 07:50:05 | urllib2.URLError: <urlopen error (-1, 'Unmapped exception: java.net.SocketException: No buffer space available (maximum connections reached?): connect')>
INFO | jvm 1 | 2021/03/07 07:50:05 |
INFO | jvm 1 | 2021/03/07 07:50:05 | ... 55 common frames omitted
INFO | jvm 1 | 2021/03/07 07:51:31 | E [a.J.Journal ] [06:51:31]: Attempted to store alarm information before journal system was initialized. Data will not be stored. alarm-source=default/aps-e-ferry/Alarms/al_db6_w0/AL11 - IP CameraOffline 10.1.0.83, alarm-name=AL11 - IP CameraOffline 10.1.0.83
INFO | jvm 1 | 2021/03/07 10:16:11 | E [a.J.Journal ] [09:16:11]: Attempted to store alarm information before journal system was initialized. Data will not be stored. alarm-source=default/aps-e-ferry/Alarms/al_db6_w10/FLT 63 - APSShore emergency release, alarm-name=FLT 63 - APSShore emergency release
INFO | jvm 1 | 2021/03/07 10:16:11 | E [a.J.Journal ] [09:16:11]: Attempted to store alarm information before journal system was initialized. Data will not be stored. alarm-source=default/aps-e-ferry/Alarms/al_db6_w4/FLT 06 - APSShore Emergency Stop, alarm-name=FLT 06 - APSShore Emergency Stop
INFO | jvm 1 | 2021/03/07 17:56:47 | I [h.d.H.ENGINE ] [16:56:47]: Checkpoint start
INFO | jvm 1 | 2021/03/07 17:56:47 | I [h.d.H.ENGINE ] [16:56:47]: checkpointClose start
INFO | jvm 1 | 2021/03/07 17:56:47 | I [h.d.H.ENGINE ] [16:56:47]: checkpointClose synched
INFO | jvm 1 | 2021/03/07 17:56:47 | I [h.d.H.ENGINE ] [16:56:47]: checkpointClose script done
INFO | jvm 1 | 2021/03/07 17:56:47 | I [h.d.H.ENGINE ] [16:56:47]: dataFileCache commit start
INFO | jvm 1 | 2021/03/07 17:56:47 | I [h.d.H.ENGINE ] [16:56:47]: dataFileCache commit end
INFO | jvm 1 | 2021/03/07 17:56:47 | I [h.d.H.ENGINE ] [16:56:47]: checkpointClose end
INFO | jvm 1 | 2021/03/07 17:56:47 | I [h.d.H.ENGINE ] [16:56:47]: Checkpoint end - txts: 852255
INFO | jvm 1 | 2021/03/08 05:00:22 | I [h.d.H.ENGINE ] [04:00:22]: Checkpoint start
INFO | jvm 1 | 2021/03/08 05:00:22 | I [h.d.H.ENGINE ] [04:00:22]: checkpointClose start
INFO | jvm 1 | 2021/03/08 05:00:22 | I [h.d.H.ENGINE ] [04:00:22]: checkpointClose synched
INFO | jvm 1 | 2021/03/08 05:00:22 | I [h.d.H.ENGINE ] [04:00:22]: checkpointClose script done
INFO | jvm 1 | 2021/03/08 05:00:22 | I [h.d.H.ENGINE ] [04:00:22]: dataFileCache commit start
INFO | jvm 1 | 2021/03/08 05:00:22 | I [h.d.H.ENGINE ] [04:00:22]: dataFileCache commit end
INFO | jvm 1 | 2021/03/08 05:00:22 | I [h.d.H.ENGINE ] [04:00:22]: checkpointClose end
INFO | jvm 1 | 2021/03/08 05:00:22 | I [h.d.H.ENGINE ] [04:00:22]: Checkpoint end - txts: 1056767
INFO | jvm 1 | 2021/03/08 14:55:26 | E [a.J.Journal ] [13:55:26]: Attempted to store alarm information before journal system was initialized. Data will not be stored. alarm-source=default/aps-e-ferry/Alarms/al_db6_w10/FLT 63 - APSShore emergency release, alarm-name=FLT 63 - APSShore emergency release
INFO | jvm 1 | 2021/03/08 14:55:26 | E [a.J.Journal ] [13:55:26]: Attempted to store alarm information before journal system was initialized. Data will not be stored. alarm-source=default/aps-e-ferry/Alarms/al_db6_w4/FLT 06 - APSShore Emergency Stop, alarm-name=FLT 06 - APSShore Emergency Stop
INFO | jvm 1 | 2021/03/08 14:55:37 | E [a.J.Journal ] [13:55:37]: Attempted to store alarm information before journal system was initialized. Data will not be stored. alarm-source=default/aps-e-ferry/Alarms/al_db6_w10/FLT 63 - APSShore emergency release, alarm-name=FLT 63 - APSShore emergency release
INFO | jvm 1 | 2021/03/08 14:55:37 | E [a.J.Journal ] [13:55:37]: Attempted to store alarm information before journal system was initialized. Data will not be stored. alarm-source=default/aps-e-ferry/Alarms/al_db6_w8/FLT 43 - APSShore FESTO pressure reading fault, alarm-name=FLT 43 - APSShore FESTO pressure reading fault
INFO | jvm 1 | 2021/03/08 15:47:36 | E [c.i.i.g.m.MulticastScheduler ] [14:47:36]: IOException thrown when using IP address 231.1.1.1 and send port 4445
INFO | jvm 1 | 2021/03/08 15:47:36 | java.net.SocketException: Network is unreachable: Datagram send failed
INFO | jvm 1 | 2021/03/08 15:47:36 | at java.net.DualStackPlainDatagramSocketImpl.socketSend(Native Method)
INFO | jvm 1 | 2021/03/08 15:47:36 | at java.net.DualStackPlainDatagramSocketImpl.send(Unknown Source)
INFO | jvm 1 | 2021/03/08 15:47:36 | at java.net.DatagramSocket.send(Unknown Source)
INFO | jvm 1 | 2021/03/08 15:47:36 | at com.inductiveautomation.ignition.gateway.multicast.MulticastScheduler.run(MulticastScheduler.java:218)
INFO | jvm 1 | 2021/03/08 15:47:36 | at com.inductiveautomation.ignition.common.execution.impl.BasicExecutionEngine$TrackedTask.run(BasicExecutionEngine.java:565)
INFO | jvm 1 | 2021/03/08 15:47:36 | at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
INFO | jvm 1 | 2021/03/08 15:47:36 | at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
INFO | jvm 1 | 2021/03/08 15:47:36 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
INFO | jvm 1 | 2021/03/08 15:47:36 | at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
INFO | jvm 1 | 2021/03/08 15:47:36 | at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
INFO | jvm 1 | 2021/03/08 15:47:36 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
INFO | jvm 1 | 2021/03/08 15:47:36 | at java.lang.Thread.run(Unknown Source)
INFO | jvm 1 | 2021/03/08 15:47:36 | W [d.S.B.TimeoutDaemon ] [14:47:36]: ScheduledRequest[com.inductiveautomation.xopc.drivers.siemens.requests.ReadRequest@53d5a954] request with key "2683" failed due to timeout. device-name=S71500
INFO | jvm 1 | 2021/03/08 15:47:36 | W [d.S.ReadRequest ] [14:47:36]: Request failed due to TIMEOUT device-name=S71500

[pturmel]

@andrea.v

urlopen error (-1, ‘Unmapped exception: java.net.SocketException: No buffer space available (maximum connections reached?): connect

Looks like a socket leak. Are you using try-finally blocks around your socket code to ensure sockets are closed after use? Do you have to use urllib? (It has a bad reputation in jython.) If at all possible, convert your code to use the new httpClient functionality in v8.1.

[andrea.v]

Thank You Phil !

I will add try-catch and see if it fixes the issue the next time i can test this.

Sadly for this project has been decided to use already ignition 7.9 …

I dont know any other ways to send http commands, i’m new to python and java

Thanks again

Andrea

[pturmel]

I would use java’s socket and URL classes instead of python, since Ignition uses jython.

[andrea.v]

Do you know where i can find any examples?
My implementation now is made in the designer, this is my code at the moment:

1- on a 10 second script in the gateaway:

import urllib2
value=system.tag.read("aps-e-ferry/Alarms/Record")
if value.value > 0 :
intvalue = value.value - 10
system.tag.write("aps-e-ferry/Alarms/Record", intvalue)
else :
try:
req = urllib2.urlopen('http://localhost:8090/command.cgi?oid=1&ot=2&cmd=recordStop')
except Exception , e:
print e
finally:
req.close()

2- in the trigger event of afew tags.

if initialChange==0 and currentValue > 0 :
import urllib2
try:
req = urllib2.urlopen('http://localhost:8090/command.cgi?oid=1&ot=2&cmd=record')
system.tag.write("aps-e-ferry/Alarms/Record", 70)
except Exception , e:
print e
finally:
req.close()

The goal is to send a request to another program locally that is managing video recording of a ip camera for 60ish seconds in case of allarm.

If you feel like there are better way to implement this keeping the code in the designer please share.

Thank you again for your time

Andrea