We received a comment from a Perspective user that there were problems connecting to Ignition via the perspective app on a cell phone. I was able to confirm this issue though accessing perspective via webpage seemed fine.
I decided to reset the Perspective module. After doing this the GW faulted and would not restart. I then did net stop Ignition // net start Ignition to reset everything.
When Ignition restarted, there were no projects listed on the gateway! Even the backup folder was empty!
We've been doing regular weekly backups and fortunately have been offloading them to a separate NAS. I was able to restore everything from one of those backups so we're OK fortunately.
Has anyone else experienced this? What could I have done to prevent this from happening? This would have been world-ending for us if we didn't have the remote backups.
Details about "the gateway faulted" (and what was wrong with Perspective prior to that) would be helpful. Also, what the state of the filesystem was while the gateway reported having no projects. In general, I'd strongly recommend contacting our support department officially. Gather a diagnostics bundle now, and ideally have as much historical information as possible.
Is the host a VM? Managed by IT with associated "security" tooling that may have rolled back files on disk?
When I say faulted I mean that a screen appeared when going to the Ignition home page (this was similar to the screen you get when Ignition is starting or shutting down).
The host server is physical, but IT definitely reaches their hands unannounced into the system. This possibly may be what caused this though it's hard to believe.
Can you provide guidance on how to pull diagnostics? Also, is that even possible now that I've done a GW restore?
It will have retroactive system logs, which will hopefully have information about whatever was going wrong when you were having issues. That's why I'd urge collecting them as soon as possible.
I've learned to not put anything past IT departments
Understood. This usually means some kind of critical server error - the good news is that at least the cause of that should be clear as day in the logs you capture.
Interpreting them usefully is, again, usually an exercise for our support department, but you can open a diagnostic bundle directly in Kindling (or any other .zip tool) and take a look yourself.
Thank you Paul - I've grabbed the diagnostics file and will be reaching out to the support department. We're fine since we had backed up the backup but it will be good for us to understand exactly what happened.
