I have been having this issue for some time now and have come to the conclusion that I must be over looking something.
Example:
At 10:30pm yesterday, I stopped logging ALL DATA.... I cannot see any reason why it would have stopped.
And JUST AS STRANGE, the data started logging again at 3:01am this morning. What is strange about this is that at the SAME time, a user logged into a client and started the runtime.
I am thinking that due to INACTIVITY with a client to the server it is going into some sort of SLEEP state. I know that sounds crazy, but I cannot find any CAUSE.
PLEASE help… I’m pulling my fracking hair out.
Any 's??
What OPC server and version are you using? Inactivity shouldn’t cause FSQL to stop working - it’s designed to run for extended periods of time without user intervention.
Can you verify that:
a. FactorySQL is set up as a Windows service (control panel->admin tools->services)
b. The service starts automatically (“startup type” set to “automatic”)
c. Data logs automatically without the frontend/system tray application open (reboot without a user logging into the FactorySQL computer and check the database)
The system tray application can provide the same functionality as the service to support legacy OPC server applications. I wouldn’t be surprised if the user logging on at 3:01 caused the system tray app (not the frontend) to start logging data.
Once you’ve verified that the system starts up and runs as a service (without a user ever logging on), I would dig around the log files (FSQL Frontend->help->Log Viewer) for anomalies.
I am using KEPwar Enhanced OPC/DDE Server V4.300.449.0 - U
Drivers: AB Suite: Allen-Bradley DH+ V4.30.50.0 - U
: Allen-Bradley Ethernet V4.70.79.0 - U
1. Can you verify that:
a. FactorySQL is set up as a Windows service (control panel->admin tools->services)
b. The service starts automatically ("startup type" set to "automatic")
c. Data logs automatically without the frontend/system tray application open (reboot without a user logging into the FactorySQL computer and check the database)
It was STARTED and set to MANUAL. I have changed it to AUTOMATIC. I rebooted and nothing started working until I logged in. Also, we have NOVELL installed on this computer and the login prompt is NOVELL’s not windows. Not sure if this effects anything.
Queston on the SERVICES, when you double click on FPMI or FSQL or MySQL and select the “Log On” tab, it is set to “Local System Account” and the “Allow service to interact with desktop” check box is not checked. Is this correct? And what settings should I use on the “Recovery” tab?
2. The system tray application can provide the same functionality as the service to support legacy OPC server applications. I wouldn't be surprised if the user logging on at 3:01 caused
the system tray app (not the frontend) to start logging data.
So, the “USER” being on ANOTHER seperate computer, connecting to the server, could cause the system tray app to start? Also, I miss spoke. At 3:01 the user clicked on a BUTTON withing the client and accessed a Graph. So, the user was already logged in, but I’m not sure WHEN they logged in, either BEFORE it stopped logging, or WHILE it stopped logging. But in the past, I have come in on a MONDAY and logged in and everything is working fine till I look at a trend and noticed that there is no data for lets say SUNDAY, but it started logging the exact moment I logged into the client.
Just checking the obvious but what kind of power options does the computer running FSQL have set? Is it set to power down the hard drive or to “standby” after a period of “non-use”? User activity may be “waking-up” the computer.
Check the user account that’s set to run the service (admin tools->services->FactorySQL service properties window, click the “log on” tab).
I don’t have much Novell experience, but I suspect that a local account should be fine to run applications, even on a machine that’s a Netware client. You’ll want to do everything you can to isolate/verify the fact that the service is or is not starting on it’s own.
It can’t be power/inactivity settings because you’ve verified that the system doesn’t start automatically on a reboot.
Trying to connect a remote frontend can’t start the service on that distant end. You can start and stop groups remotely, but the service/tray app had to have been running already.
Normally, the Novell Client takes precedence over everything else. It’s why none of the services start until after you log in.
However, there’s a utility in the Novell Cool Solutions pages called Autolog that lets you automatically log in a workstation. One of the options you can set up is to login to the local workstation without logging in to Novell (I have a training station set up this way). This should take care of the services problem without compromising the Novell security.
Now, instead of losing data from KEP or FSQL issues, I am having issues with the gateway.
It will run fine for a week or so, but when I come in at 7am, the gateway will not be running.
So, naturally I start it back up and goto the "System Console" on the gateway interface.
Here is the ONLY thing that is on it ONCE I have restarted it.
And in Window's Event Viewer I have this:
It appears that the TIME at which I got THIS error, is VERY close to the time at which I tried to log in this morning.
Another observation. A user was ABLE to log in at 7:00 am with no problems, BUT when he logged in, all of our SQLTags were stale and stayed stale. Also, every screen he went to, had an xml parsed error.
This is now not a data loss issue, but seems related to IA software. This is an issue just with the GATEWAY. Data is still being collected, just no access to FPMI till I restarted the gateway.
Yes, the messages indicating that the service has stopped unexpectedly are not good- either there’s an error occurring inside the gateway, or some outside entity is force-quitting the gateway process.
I’ve never seen the gateway application crash completely- so I’m inclined to look towards the second possibility.
What was the solution on the FactorySQL side? Is there any chance that the Novell stuff is interacting with the FactoryPMI gateway application?
However, we should probably also start by having a look at the PMI gateway logs. Could you go to “logs” under the FactoryPMI install directory, zip up a few of them that are around the days/times that this problem has occurred, and email them to support (at) inductiveautomation.com?
On the FSQL issue, I was able to set as service, but the AUTOLOGIN for novell didn’t seem to work too well. LOL We have NOT had any other issues with that though.
As far as the FPMI gateway is concerned, here is what I have:
<tr>
<td>Oct 14, 2009 07:26:52 AM</td>
<td title="Level"><font color="#993300"><strong>ERROR</strong></font></td>
<td title="Message">Servlet.service() for servlet Gateway threw exception</td>
<td title="org.apache.catalina.core.ContainerBase.[mainEngine].[localhost].[/gateway].[Gateway] category">org.apache.catalina.core.ContainerBase.[mainEngine].[localhost].[/gateway].[Gateway]</td>
<td title="http-80-Processor10 thread">http-80-Processor10</td>
</tr>
<tr><td bgcolor="#993300" style="color:White; font-size : xx-small;" colspan="6">java.lang.OutOfMemoryError
<br> at java.util.zip.Deflater.init(Native Method)
<br> at java.util.zip.Deflater.<init>(Unknown Source)
<br> at java.util.zip.GZIPOutputStream.<init>(Unknown Source)
<br> at java.util.zip.GZIPOutputStream.<init>(Unknown Source)
<br> at com.inductiveautomation.factorypmi.gateway.sqltags.db.DatasourceTagProvider$Base64GZippedPrintWriter.createXMLPrintWriter(DatasourceTagProvider.java:4715)
<br> at com.inductiveautomation.factorypmi.gateway.sqltags.db.DatasourceTagProvider.processClientPoll(DatasourceTagProvider.java:880)
<br> at com.inductiveautomation.factorypmi.gateway.servlets.Gateway.runSQLTagPoll(Gateway.java:1153)
<br> at com.inductiveautomation.factorypmi.gateway.servlets.Gateway.doPost(Gateway.java:407)
<br> at javax.servlet.http.HttpServlet.service(HttpServlet.java:709)
<br> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
<br> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
<br> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
<br> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
<br> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
<br> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
<br> at com.inductiveautomation.factorypmi.gateway.ErrorReportValve.invoke(ErrorReportValve.java:95)
<br> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
<br> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
<br> at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
<br> at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
<br> at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
<br> at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
<br> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
<br> at java.lang.Thread.run(Unknown Source)
</td></tr>
There are many MANY threads like this. There are anywhere from 4 to 10 of these errors PER/SECOND. It start to give these errors, then about 25-30 minutes later, that is when it crashes.
Occasionally I get this one:
ClientAbortException: java.net.SocketException: Software caused connection abort: socket write error
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:366)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:433)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:348)
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:392)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:381)
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:88)
at com.inductiveautomation.factorypmi.gateway.servlets.Gateway$ContentLengthBufferingOutputStream.write(Gateway.java:313)
at java.io.OutputStream.write(Unknown Source)
at com.inductiveautomation.factorypmi.common.utils.Base64$OutputStream.write(Base64.java:1491)
at com.inductiveautomation.factorypmi.common.utils.Base64$OutputStream.write(Base64.java:1539)
at java.io.ByteArrayOutputStream.writeTo(Unknown Source)
at com.inductiveautomation.factorypmi.gateway.servlets.Gateway$2.run(Gateway.java:2201)
at com.inductiveautomation.factorypmi.gateway.servlets.Gateway.doDBAction(Gateway.java:2364)
at com.inductiveautomation.factorypmi.gateway.servlets.Gateway.runQuery(Gateway.java:1925)
at com.inductiveautomation.factorypmi.gateway.servlets.Gateway.doPost(Gateway.java:404)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:709)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at com.inductiveautomation.factorypmi.gateway.ErrorReportValve.invoke(ErrorReportValve.java:95)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Unknown Source)
Caused by: java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:746)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:433)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:348)
at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:769)
at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:125)
at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:579)
at org.apache.coyote.Response.doWrite(Response.java:559)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:361)
... 30 more
That first error shows that the Gateway has run out of memory. Stack traces and log messages are pretty useless for debugging this.
How is the memory looking for the Gateway now (I assume you’re restarted it)? You can check on the Gateway Status page, and also report what Windows thinks the Gateway is using through the Task Manager (FPMIGateway.exe)
Have you tried zipping it? We’ve found that the log format zips up really well. And then, instead of emailing it, you could go to inductiveautomation.com / upload (without spaces) and send it to us.
This happened again today. Gateway crashed Jan 20 11:02:37 a.m. I stopped and restarted thegateway around 11:14 a.m. Attatched is the log file for today.
[attachment=0]FactoryPMI Gateway Log Jan 20.zip[/attachment]
's??
