Hey all,
I have a problem every 5 or so days where the gateway seems to lock up. I can still connect via the webpage and it still collects data but does not put the data into the database, it only keeps it in the store and forward system. Also, clients can not be launched. When I go to the Overview page it looks like this:
Everything looks fine except the database has a 0 query/second rate. Also, those 25 clients that it says are open all are redundant, there is actually only about 4 people that were just trying to log in multiple times. When I launch a client it launches fine into the login screen. Then when I log in it just sits there for a while:
And eventually I’ll get this error:
The only way I can get it to work correctly is to restart the Master gateway, after that clients can connect correctly and everything in the store and forward store location gets transfered into the database. I’ll include the wrapper.log file as well but I didn’t see any thing in there that seems to be a problem. Has anyone seen this before?
wrapper.log (3.74 MB)
When this happens the most useful piece of data will probably be a thread dump. You can get this from the Console > Threads tab or, if that isn’t accessible, from the GCU.
Ok, next time it happens I’ll try and get that information for you.
It happened again and I was able to get a thread dump for you. It looks like it’s either some kind of timer or the alerting thread but I don’t really know what I’m looking at.
thread-dump 20110910.txt (94.5 KB)
This is a problem with the database pooling system, in particular the “idle evictor”. I would bet that you turned on the “Test while idle” option for the database connection. We’ve seen problems with this before, and intend to remove it. Check to see if it’s turned on, and if so, turn it off.
After turning it off, you’ll need to restart the gateway to clear the block.
Regards,
It looks like the latest version of the pooling library fixes this bug, so we should be able to update it soon. In the mean time though, just turn off the “test while idle” option.
Regards,
Thanks, Colby!
I did turn that on, I was trying to solve another problem I was having in this thread
Apparently I made an even bigger problem! Thanks for getting back to me I will turn it back off and restart and see what happens.
Yeah, that other problem should be fixed in 7.2.9. The idle testing wouldn’t help much there anyone, because it only tests “closed” connections (well, somewhat closed… closed from a use point of view, but open in the pool).
Regards,
