# Ignition service on primary server hangs **URL:** \ **Category:** Ignition\ **Created:** [July 3, 2012, 3:54pm UTC](https://forum.inductiveautomation.com/t/ignition-service-on-primary-server-hangs/4752 "2012-07-03T15:54:28Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![nrolling](https://avatars.discourse-cdn.com/v4/letter/n/5f8ce5/32.png) [@nrolling](https://forum.inductiveautomation.com/u/nrolling)\ **Post date:** [July 3, 2012, 3:54pm UTC](https://forum.inductiveautomation.com/t/ignition-service-on-primary-server-hangs/4752/1 "2012-07-03T15:54:28Z")
I am using redundant Ignition servers and I am running into a problem where my Ignition on my primary server hangs. When I view the status of the service on the primary server in a command line (service ignition status), it shows that it is running. I can not get to the gateway webpage on the primary, and the gateway webpage on the backup does not show the primary as running. The backup is active and the fail over works. I am using Ignition 7.5, running on SuSE Enterprise Linux 11. The Linux servers are running as VM’s under ESX 5. They are located on separate physical servers. This has been a very stable setup that we have been running for about a year. I have seen the service hang twice in the last week. After the first time I went through the wrapper.log and tried to clear up any errors that I could. This time there is the error: INFO | jvm 1 | 2012/07/03 07:37:16 | Jul 3, 2012 7:37:16 AM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run INFO | jvm 1 | 2012/07/03 07:37:16 | SEVERE: Socket accept failed INFO | jvm 1 | 2012/07/03 07:37:16 | java.net.SocketException: Too many open files [wrapper.zip](https://us1.discourse-cdn.com/inductiveautomation/original/2X/2/2d0189a7b0051532eda23fe86a350656c0e6d30e.zip) (13.1 KB) ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [July 3, 2012, 4:04pm UTC](https://forum.inductiveautomation.com/t/ignition-service-on-primary-server-hangs/4752/2 "2012-07-03T16:04:03Z")
On the server with the problem, find the ignition process ID. In a terminal, do: ```auto ps aux | grep ignition ``` Take the process id, run: ```auto lsof -p 22118 > ~/Desktop/ignition_files.txt ``` Sub your actual process ID in where I used ‘22118’. Then upload/send that file in. (ignition\_files.txt on your desktop) Rebooting the server should get you running again. ---
**Author:** ![nrolling](https://avatars.discourse-cdn.com/v4/letter/n/5f8ce5/32.png) [@nrolling](https://forum.inductiveautomation.com/u/nrolling)\ **Post date:** [July 3, 2012, 4:47pm UTC](https://forum.inductiveautomation.com/t/ignition-service-on-primary-server-hangs/4752/3 "2012-07-03T16:47:29Z")
Yes, when I stopped the ignition service and started it again the redundancy was back to normal. So when I ran lsof to get this file everything was running normally. [ignition\_files.txt](https://us1.discourse-cdn.com/inductiveautomation/original/2X/2/2b5c4c049dc3f974e5c82d621ce89b0b96af529d.txt) (1.3 KB) ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [July 3, 2012, 4:50pm UTC](https://forum.inductiveautomation.com/t/ignition-service-on-primary-server-hangs/4752/4 "2012-07-03T16:50:11Z")
Well if it happens again try and run the lsof thing again before you restart it, might help track down what’s going on. ---
**Author:** ![nrolling](https://avatars.discourse-cdn.com/v4/letter/n/5f8ce5/32.png) [@nrolling](https://forum.inductiveautomation.com/u/nrolling)\ **Post date:** [July 3, 2012, 4:53pm UTC](https://forum.inductiveautomation.com/t/ignition-service-on-primary-server-hangs/4752/5 "2012-07-03T16:53:10Z")
Ok, thanks Kevin. ---
**Author:** ![nrolling](https://avatars.discourse-cdn.com/v4/letter/n/5f8ce5/32.png) [@nrolling](https://forum.inductiveautomation.com/u/nrolling)\ **Post date:** [July 9, 2012, 2:18pm UTC](https://forum.inductiveautomation.com/t/ignition-service-on-primary-server-hangs/4752/6 "2012-07-09T14:18:00Z")
I have this problem again with same error. I ran the lsop command, but the output looks the same to me. I have not restarted the server or the ignition service yet, it is still in a hung state. Is there anything else I can try to narrow the problem down while it is still in this state? [wrapper.zip](https://us1.discourse-cdn.com/inductiveautomation/original/2X/3/37632711909ac41418c9b5247f3672de34276e1b.zip) (60.3 KB) [ignition\_files2.txt](https://us1.discourse-cdn.com/inductiveautomation/original/2X/0/0b8953caa759010c929a89388ea55466b44b78b9.txt) (1.3 KB) ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [July 9, 2012, 2:30pm UTC](https://forum.inductiveautomation.com/t/ignition-service-on-primary-server-hangs/4752/7 "2012-07-09T14:30:50Z")
I think that’s the wrong process - I need the PID of the java process running Ignition. I _think_ you’ve got the process running the service wrapper. ---
**Author:** ![nrolling](https://avatars.discourse-cdn.com/v4/letter/n/5f8ce5/32.png) [@nrolling](https://forum.inductiveautomation.com/u/nrolling)\ **Post date:** [July 9, 2012, 3:08pm UTC](https://forum.inductiveautomation.com/t/ignition-service-on-primary-server-hangs/4752/8 "2012-07-09T15:08:22Z")
Ok. I think I have a better idea of what the problem is now. When I ran the lsop for the Java process it looks like I have TCP connections that are not getting closed. java 17222 root 1212u IPv6 31821398 0t0 TCP x.y.net:41581-\>[mail.y.com](http://mail.y.com):imap (CLOSE\_WAIT) I had written code to use a modified version of the python imap library to send alarm notifications and then hit up a mailbox for the reply to ack the alarm. I am guessing that there is something different with the newer version of Python that is not agreeing with the code I am using. Its my understanding that the imap library is now included with the Python files that come with Ignition, so I will see if I can use that. ---
**Author:** ![Kevin.Herron](https://sea1.discourse-cdn.com/inductiveautomation/user_avatar/forum.inductiveautomation.com/kevin.herron/32/7259_2.png) [@Kevin.Herron](https://forum.inductiveautomation.com/u/Kevin.Herron)\ **Post date:** [July 9, 2012, 3:29pm UTC](https://forum.inductiveautomation.com/t/ignition-service-on-primary-server-hangs/4752/9 "2012-07-09T15:29:24Z")
Included, but untested. 😃 ---
**Author:** ![nrolling](https://avatars.discourse-cdn.com/v4/letter/n/5f8ce5/32.png) [@nrolling](https://forum.inductiveautomation.com/u/nrolling)\ **Post date:** [July 24, 2012, 4:41pm UTC](https://forum.inductiveautomation.com/t/ignition-service-on-primary-server-hangs/4752/10 "2012-07-24T16:41:27Z")
I changed my script so that it references the jython imap library that now comes with ignition. I tested the script in the script playground and it is working. However, I still have problems with the sockets not closing in a timely matter when I make a connection to the mail server. I was wondering if anyone has used this imap method before. I have attached the script. On a side note I have noticed that ignition 7.5 does not play nice with older java versions (1.6 update 26) on our server. CPU utilization was way up to 75% with this combination. Once we updated java to 1.7 everything was happy and running at a much lower CPU utilization. [alarm notification.py](https://us1.discourse-cdn.com/inductiveautomation/original/2X/2/260de0f9c804781c58aa81cbc8ce8686ef9104a4.py) (13.6 KB) ---
**Author:** ![dave.fogle](https://avatars.discourse-cdn.com/v4/letter/d/43a26b/32.png) [@dave.fogle](https://forum.inductiveautomation.com/u/dave.fogle)\ **Post date:** [July 25, 2012, 9:54pm UTC](https://forum.inductiveautomation.com/t/ignition-service-on-primary-server-hangs/4752/11 "2012-07-25T21:54:57Z")
What is the printed message after you call M.logout()? Are you successfully getting the BYE response? ---
**Author:** ![nrolling](https://avatars.discourse-cdn.com/v4/letter/n/5f8ce5/32.png) [@nrolling](https://forum.inductiveautomation.com/u/nrolling)\ **Post date:** [July 25, 2012, 10:30pm UTC](https://forum.inductiveautomation.com/t/ignition-service-on-primary-server-hangs/4752/12 "2012-07-25T22:30:09Z")
\\> ---
**Author:** ![nrolling](https://avatars.discourse-cdn.com/v4/letter/n/5f8ce5/32.png) [@nrolling](https://forum.inductiveautomation.com/u/nrolling)\ **Post date:** [July 25, 2012, 10:58pm UTC](https://forum.inductiveautomation.com/t/ignition-service-on-primary-server-hangs/4752/13 "2012-07-25T22:58:08Z")
I just noticed that I do not have () after close and logout. When I try a condensed version of that script in a command line with Jython I get the error above. It should be M.close() and M.logout(). ---
**Author:** ![dave.fogle](https://avatars.discourse-cdn.com/v4/letter/d/43a26b/32.png) [@dave.fogle](https://forum.inductiveautomation.com/u/dave.fogle)\ **Post date:** [July 26, 2012, 9:03pm UTC](https://forum.inductiveautomation.com/t/ignition-service-on-primary-server-hangs/4752/14 "2012-07-26T21:03:30Z")
So did calling the functions correctly fix the issue? ---
**Author:** ![nrolling](https://avatars.discourse-cdn.com/v4/letter/n/5f8ce5/32.png) [@nrolling](https://forum.inductiveautomation.com/u/nrolling)\ **Post date:** [July 30, 2012, 2:29pm UTC](https://forum.inductiveautomation.com/t/ignition-service-on-primary-server-hangs/4752/15 "2012-07-30T14:29:16Z")
The close and logout return the proper responses now (ok, and bye). I still get a close\_wait for a few of the connections to the mail server but nothing like it was before. ---
**Author:** ![dave.fogle](https://avatars.discourse-cdn.com/v4/letter/d/43a26b/32.png) [@dave.fogle](https://forum.inductiveautomation.com/u/dave.fogle)\ **Post date:** [July 30, 2012, 4:52pm UTC](https://forum.inductiveautomation.com/t/ignition-service-on-primary-server-hangs/4752/16 "2012-07-30T16:52:30Z")
Ok, good to know. Glad it seems to be working for you now. Let us know if you start seeing more connection issues again.