Hello,
Recently our Engineering team was asked to move a lot of Ignition Infrastructure to the Cloud. We were able to identify some things that could (including databases). So, to protect the data, we started by using the built-in Store & Forward (SF) method for database transactions. This allowed a more fool proof way to avoid data not making it to the DB if there were connection issues. We are running Ignition 8.1.28 (will be upgrading soon to 8.1.35) and we use mostly Vision but have some Perspective projects.
On the surface, this change to SF transactions was working great, but we came across 2 issues:
The first issue is that our SF records seemed to randomly get stuck or not send out for hours at a time. Another Engineer on our team worked with IA Support and the fix was to restart the gateway. Has anyone else had a similar issue or a better way to know what is happening?
The second issue was brought up as a complaint from operations. The complaint is that data was not always updated on the window/view and they had to manually refresh it more often now or go to another page and come back. Normally, after sending data to the DB, we were able to refresh components on the windows/views almost immediately after inserting the data. Now, with SF, we have to delay this update and the timing is not always right. I am struggling with how soon to refresh the windows/views once the data is actually available. Since the SF methods only tell you if it made it to the SF pool and not to the actual DB, I'm struggling to find an efficient way to manage this. I considered having tag scripts or gateway scripts to monitor specific tables to know when new data was available but I'm wondering if there is a better way since they could become pretty taxing on the gateway and DB to have scripting constantly hitting the DB. I'm trying to avoid going down this route in case others have tried it or can offer advice before I dive in.
I am open to all suggestions, and I appreciate your help!
What possessed your Engineering team to think your production databases could be moved to the cloud? Ignition usage of a database for your historian and for general UI data is pretty latency sensitive.
-
Don't move your facility's production database to the cloud, unless you also move Ignition itself to the cloud. Full stop.
-
Don't move Ignition itself to the cloud if you have any native protocol OPC drivers in use.
-
Don't move Ignition itself to the could if any part of the facility that relies on Ignition needs to be able to run when the WAN goes down.
(Moving your SCADA package, its plant-floor HMIs, and its primary database to the cloud also violates the U.S. Department of Homeland Security best practices for industrial networks. These are supposed to reside on the innermost protected network of a facility. If you are involved in water/sewer/electric or other public infrastructure, you may have some obligations here.)
Store and forward methods are totally inappropriate for use with User Interfaces. You cannot make this stuff work.
3 Likes
I fully agree with everything Phil said. I'll also add that the current S+F backing has some pathological failure cases that are leading us to a near complete rewrite in 8.3 which should perform significantly better.
I should have added some more context to the Ignition functions that moved to the Cloud. Zero of our SCADA functionality is in the cloud and never will be. Any of our historian type data flows through mqtt (works great). Lastly, the only stuff we really moved was "paperwork/reporting" functionality. I had worked with a few integrators and IA Support to layout and go through what we could move or what absolutely should not just to make sure we weren't setting ourselves up for failure and to affirm some of our thinking.
Truth be told, our Engineering group did not want to move any Ignition functions to the cloud but we did not have a choice for some of it. Our IT department owns all on premise infrastructure (Networking and Server stacks) so the decision was out of our hands. We had to make some of this work. Anything on the plant floor side all stays local and will stay local because our group owns that equipment and infrastructure and we control it. We run a distributed architecture with central gateways for each area. They do not rely on each for anything and mostly run independent. We move base MES data through MQTT to a central gateway. Our Reporting and Data entry used to be Site specific but is now centrally located for all Sites. That decision was made since our Site Servers were being removed as well.
As I mentioned about SF functions, I specifically mean the scripting functions where the return is T/F if it successfully sends to the SF pool. Whereas some of the other functions can respond with the table ID or logical ID from the insert. I understand this is a limitation but wanted to make it clear that is why I don't know when the data makes it based on the method I am using. I'm not exactly sure what you mean by SF methods being totally inappropriate for User Interfaces. As an example, the operator inputs data into a form and submits it. The submit sends it to a DB and then updates the data on the window to show that the new data is there. This is exactly what is taught in the Advanced Inductive Training course. Maybe I'm just not explaining it correctly or understanding your comment.
Maybe my question is more like this. How can I send data to a database (protected) from a window/view and then properly update that open window/view when the data is successfully posted to the DB? When I say "protected", I mean that the data is remembered so that if it fails to insert into the DB, it doesn't just fail but can be pused through a queue. I assumed the SF method was one of the easiest ways to do this but dealt with these 2 highlighted issues.
Should I potentially avoid S+F for what I am trying to do or should I work on a different way to refresh the data on the window/view?
Right, that last part is simply impossible with S&F, because the data isn't there yet for the redisplay query.
No such guarantee. High likelihood over a long time (hours->days), but no guarantee. That's another reason S&F isn't appropriate for user interfaces--no success feedback to the user.
Just use the regular/prep/NQ update queries and use the return code to report success or failure to your UI. If it takes longer than your users like, have them complain to your (utterly brain-dead) IT group.
{ Were I in your position, with impossible to meet requirements, I'd be polishing up my resume. Seriously. Life is too short to work for incompetents. }
Thanks Phil and Paul for the insight, much appreciated. I will plan to go back to my previous scripting db methods and plan to handle them more methodically in case of failure.