web-API to read/write tags json

So which script is giving the issue, the browseTags or the read implementation. Any browse will take some time, browseConfig might be faster.

We are not having an issue per se, our current architecture works, we have some performance concerns as we continue to add more equipment, there are some internal requirements that will lead us to read all the tags in a “Main folder” and as stated in the email, it currently takes about 45 seconds.

We are trying to explore options to read the data from Ignition.

Some tips might help. Create a test DB and configure it in transaction group to UPDATE the values . All tables will be automatically created.

To quickly test your JSON api server, download the ignition.php file from the folder at Git, change the MySQL query according to your requirement.
Ignition-React-Django-JSON-Realtime/src/backend/apache_php/ignition.php

To configure, rights for your apache server, follow this link, Feel free to ask any clarifications if required. I am also excited to get your feedback.

Travix Cox in one of his webinars on Ignition’s SQL DataBase module did mention about using the SQL database for inter process communication with external applications as well. Other option is to use Jython server side scripts to communicate over TCP sockets. However I took the Java Module approach to communicate over TCP sockets which is very fast inter process communication approach (a few milli seconds to read a few hundred tags, once the tagPaths are read and data structures are initialized) being memory to memory for my application. However I am going to try all other possible approaches as well for my applications eventually. Each approach has its pluses and minuses depending upon your requirements and applications. As I said GCM is not my main module , its a glue between Ignition and my other modules viz AR-SCADA mSCADA, UNISEMS etc.

Is it taking 45s to read all the tags, or 45s to browse the tag tree to get all the tag paths?

Browsing the tag tree and caching the results would probably be the easiest implementation to save some time. Without seeing your code, however, we can’t analyze what you are doing.

You can’t make confusing statements about your own product. Will other modules work without GCM?. Backup your claims with proper performance benchmarks and online demo. I can confidently say that, it’s impossible to beat my humble solution with your GCM.

This old post has some scripting you might find useful. You might have to tweak tag.py for current versions of Ignition, though.

There is nothing confusing, I have always said GCM is not a stand alone module, its bundled with NJSCADA the nodeJS server which serves my other modules like ARSCADA etc and it meets its specific requirements. I will be making the whole NJSCADA with GCM and ARSCADA available for download and trial. No body is challenging your module.

But of course GCM can also spurt JSON files to third party applications like its writing to PubNub dash boards. The packets can be configured that need to be sent out. The demos and documentation will explain it completely. There are some testing issues that I am resolving right now. Have patience!

Beat it how? I highly doubt it would be difficult to beat your solution, in terms of performance. While your demo code works fine, it isn’t really production ready, nor is it a fully featured api.

At the end of the day, this isn’t a transport issue. They are using web dev, so we should be focusing on helping optimize his code there

1 Like

Proof of the pudding is in the eating. Let’s do a hackathon. Propose the game.

Ah, You are comparing to GCM, I was discussing just in general.

1 Like

Mr.Pramanj frequently claims that his “NJSCADA+AR-SCADA+mSCADA+UNISEMS” combo is the cure for all the challenges of automation without giving any solid proof till date. This is the reason, i decided to challenge him. He is a brilliant coder but needs to fine tune his business skills.

I respect all those great veterans and warriors who have put up tremendous amount of time, efforts, hard earned money and energy in building Ignition, standing up against big automation giants. At the same time, we must keep an open mind to welcome the disruptive technology as well.

BTW, i did some deep research and built a solution for Ignition to consume big MQTT JSON payloads. This is my conclusion:

For large scale IoT data collection, MQTT vs FTP, MQTT wins hands down. FTP is the second best option, in case MQTT is not available. Congratulations. You won the debate. :slight_smile:

I never claimed anything like that, I just said new approach based on new technology like nodeJS, SVG etc. may be worth trying due to its advantages. Sorry if it sounded like that to anyone, it was just a discussion to trigger thought process. See discussion J2EE v/s NodeJS for real time applications

I first started using JAVA Apache Tomcat server connected to Ignition to serve my ARSCADA and other clients in the first version of it. However I thought of trying nodeJS instead of it, as it was gaining a lot of importance importance in IT world in past several years and offered some advantages over Apache Tomcat server such as small footprint, support for push technology (Socket.IO), faster response etc, Each technology has its advantages and disadvantages and it takes time and effort to build and try them. In mission critical applications such as SCADA, users often try previous generation proven technology due to the risks involved but that shouldn’t stop us from evaluating new technology. JAVA was also a new technology when Ignition started in 2003, and no one thought of using JAVA J2EE for real time application that time, which was pioneered by Ignition, and now it has become state of the art for even SCADA.

Of course hats off to all Ignition great stalwarts who developed this massive, robust and award winning product. I am just trying to extend its functionality leveraging new developments in technology such as AR, SVG, etc.

Though I haven’t made the download of trial version available yet (it’s under preparation) but I have made demo videos of all modules on my web site to give an idea about them.

If your SQL approach is superior then I am open to trying it or any other approach in stead of my GCM. My main objective is to drive my main modules viz , ARSCADA, mSCADA etc. thru NJSCADA, where my value addition to Ignition exists. GCM as I said is a glue, I am definitely going to try alternative approaches for it if there is a merit at a later point in time.

My initial thinking was the direct connect JAVA module approach will be suitable for me as its memory to memory data transfer which will be very fast (and it is indeed) , rather than SQL approach which involves disk IO which may be slow. I had also thought of trying JYTHON approach but thought, being interpreted language , it may not be that efficient. Also I didn’t have an experience with SQL performance in Ignition and jython gateway scripting, so I took the JAVA approach. Now that you have given the confidence in SQL response times, I will definitely try both the approaches in addition to GCM. That is what this discussion forum is all about, sharing experiences.

Thanks for your open mind and taking it in the right spirit. Evaluate my solution and give your feedback. I am also willing to learn something better from others.

I have great admiration for your coding skills and deep knowledge in Ignition. But, when you are trying to sell a piggy back SCADA on top of Ignition, it sounds to me like you are trying to sell ice to an eskimo. You may need to reinvent your product, if you want to sell.

You claim that you are trying to offer something which Ignition doesn’t. Tell me a couple of key points to support your claim in few lines.

I felt the need for AR in SCADA which is an upcoming technology so my first product was ARSCADA. I have seen some AR solution in ABB SCADA in a video of an Industrial Automation Conference. But that was based on XBOX and required 3D modelling of the whole plant which is expensive. Instead I saw the upcoming technology of 3D virtulal tours (www.krpano.com) which can be easily built using 360degree cameras. Operators can visualize the plant in these 360degree images in addition to the 2D schematic diagrams in traditional SCADA. So I thought of adding this module to Ignition as ARSCADA module which Ignition doesn’t have. I thought it will add value to operators as it gives a physical identity of plant without much cost and effort.

Secondly I saw the PubNub tool which allows us to integrate any two applications over Internet using their libraries supporting 70+languages. I thought this can be used to visualize Ignition parameters on Dashboards on remote devices including mobile phones. So the RM SCADA came. Then ability to host AR SCADA screens to remote devices was added so that even the virtual tours can be seen from remote devices using the RM SCADA module using PubNub.

Thirdly, I felt the need for a Simulation tool so I developed UNISEMS tool based on FORTRAN which is widely used in Fossil and Nuclear power plant simulators. A lot of legacy code and expertise is available in Simulation industry which can be reused in building Simulators to Ignition based plants with this tool.

Fourthly I added an OPC UA module in place of GCM to communicate with Ignition so that I can Integrate my modules thru OPC UA rather than having to install GCM and also it will allow me to integrate with any other SCADA supporting OPC UA.

Fifthly having components written for AR SCADA in SVG/JavaScript language , I thought why not develop a 2D interface like Ignition reusing these components and adding missing features like pipes/ symbols/ etc so that we have a SCADA like tool as well based on new technologies, so the mSCADA came which includes PubNub Dash boards as well. (Of course this doesn’t make sense for Ignition as addon, but may be it may make sense for some other stand alone applications like Groov or some edge devices).

Sixthly wanted to leverage NodeJS instead of Tomcat and J2EE. Hence the NJSCADA product came instead of Tomcat.

In the mean time your FTP idea came which I didn’t still understand so nothing came out of that! But you made some strides in interfacing with Ignition thru SQL+DJ+REACT+PYTHON. Congrats for that excellent work. Just wanted to know if you can do two way communication with Ignition, as GCM can, and its an important requirement for me to write to tags in Ignition. I think you can achieve it with two way binding the DB tags with Ignition. But that’s some thing to explore.

Today, every decent plant is monitored by security cameras in all critical areas. Every control room has got a full fledged 24x7 video monitoring and recording station approved by plant insurance. If required, plant technicians and operators are authorised to record any critical incident with their approved mobile and send the video to the support team. A 3D graphics is a big load on the system and no consultant will approve a 3rd party control screen in a production plant. A sticky bit or a delayed response can cost lives and loss of production. Ignition will NOT take any responsibility for any failure on such systems. ARSCADA looks like an outdated dead technology for me.

PubNub is NOT your technology. It’s a paid service and anybody can decide on this. Nobody needs your service or expertise to use PubNub.

UNISEMS is a just a old fashioned simulation tool. Tons of awesome simulation tools are freely available on the internet. Nobody will pay you for this.

Any OPC UA client can consume data from Ignition OPC UA server. I don’t see any major breakthrough on this front.

SVG/JavaScript/NodeJS – Again, nobody will approve a third party client to run their plant. Bear in mind, React perspective client is coming in a couple of months. The game is over.

Security cameras and IP cameras are for live surveillance of plant, ARSCADA is for viewing offline images with real time values on it for training and visualization of plant including inaccessible areas. Its not a 3D graphics technology and has very low overhead. Let people decide if there is a value in it.

As systems integrators our job is to integrate diverse systems and technologies such as PubNub and Ignition. Did I ever say PubNub is my technology. I am bringing the new possibilities to table with it to the notice. PubNub is there for people to use it to their imagination.

UNISEMS may be of use to some existing simulators which might want to migrate to Ignition and reuse legacy code. Some people may still prefer the older approach. You never know if requirement exists. I just made the know how available to the notice. Please use the tons of free tools.

OPC UA helps me integrate my modules with any SCADA system having OPC server, thats the breakthrough for me. I am not inventing any thing new here. No one is stopping you to consume what ever data you want with OPC UA. Technology exists for you to use it.

SVG/NodeJS etc – OK fine let no one use them . I am happy creating some thing new using the Ignition SDK and the platform that they have offered.

These are your views please don’t bias others if you don’t see any thing positive. I didn’t ask your opinion. I just try replying to your questions.

Thanks!

1 Like

When you have to serve 100’s of hungry analytical /ML clients, “load distribution, IT/security compliance, scalability and high availability” will become big issues. I think, this kind of heavy payload consumers can be handled better by “database+apache/Django JSON api” back end servers. What’s your opinion?.

Serving clients on HTTP protocol has higher overhead compared to socket.io (websocket) in terms of packet bigger packet size (header footer size is high) as well as you have to serve individual clients one by one in HTTPwhere as one packet can be broadcast to all clients in socket.io. Secondly data base IO will definitely be slower than memory to memory by common sense, though it may not be significant if packet size is small. We have to see loading of Ignition server CPU is loaded for a medium to large project.

Implementation wise definitely database approach will be easier and scaleable I guess. Security wise it will be the same if its on HTTPS protocol. It all depends on application. Why would one need ML on 100’s of clients? ML is implemented on server I guess.

My requirement was faster memory to memory transfer of packet, but thru database would also fine for this i guess.

2 Likes

Your GCM module sounds very impressive. I would like to evaluate your GCM module, if you can publish/subscribe JSON payloads via GCM-MQTT client for 100,000 tags. Is it possible?.