We're finding scheduled backups from our gateways take a long time to complete. For example, we had scheduled hourly backups on our central development server at one point, but found most were turning up as 0 KB files as they were taking longer than an hour to complete. This was mostly resolved by switching to backups every 6 hours, but now and then we still get a 0 KB backup. I assumed this was an issue with our network or backup file server, but tested backing up to a local hard-drive on the server with similar results. It seems something is causing Ignition to take a long time to complete the backup. It immediately creates a 0 KB backup file, but takes over an hour to complete the ~44 MB backup. Any ideas what could cause this?
I would guess you are suffering from internal DB contention. Lots of writes to memory tags (for persistence through restart) and lots of tag reconfiguration operations would do it. (Any write to a UDT parameter or to tag properties other than .value, .timestamp, and .quality produce writes to tag configuration in the IDB.)
1 Like
Thanks, @pturmel. We should have very few (manual) tag reconfigurations, but it's possible a poorly written UDT upgrade script is hanging around. We do have some frequently updated Memory tags. However, it seems there must be another factor as I can take a manual backup from the same gateways to the same locations in seconds (or up to a few minutes for some on slower networks) via the web interface in a browser running on the gateway. However, if gateway already has blocked scheduled backup threads as shown below, I have to either wait for these to clear or restart the gateway to clear these before I can take a quick manual backup.
Some more info: Ignition 8.1.44 gateway is not heavily loaded.
The two blocked threads are scheduled backups.
The scheduled backups generally complete eventually, but take a long time. For example, one gateway just completed a 10.6 MB scheduled backup in 3.5 hours.
Hmm. No further idea.
Yeah, it's puzzling. After my last post, I realized the gateway is running under a different (service) user than the browser on the gateway. Another quick test verified that's not the issue. Downloads from web interface complete in seconds when running the browser as the gateway service user on the gateway.
I'm not sure what else could be different between scheduled versus web interface triggered backups with both processes running as the same user on the same computer and backing up to the same location.
While it doesn't resolve the weird scheduled vs. manual backup issue noted above, I realize what we really want is version control. Beckhoff TwinCAT's full Git integration is a gamechanger for PLC code, and I hope to see something similar in a future Ignition version. Meanwhile, I'll have to read up on what people here have been doing with Git and Ignition--after improvements in 8.3, if not before.
You should get a thread dump with these blocked threads and either post it here or send it in to our support department.
2 Likes
I should've grabbed a thread dump yesterday when those two threads were sitting blocked the whole time I looked at them. I have a couple long backups cycling between blocked and not blocked now--but too fast to catch while blocked.