Beat it how? I highly doubt it would be difficult to beat your solution, in terms of performance. While your demo code works fine, it isn't really production ready, nor is it a fully featured api.
At the end of the day, this isn't a transport issue. They are using web dev, so we should be focusing on helping optimize his code there
Proof of the pudding is in the eating. Let's do a hackathon. Propose the game.
Ah, You are comparing to GCM, I was discussing just in general.
Mr.Pramanj frequently claims that his āNJSCADA+AR-SCADA+mSCADA+UNISEMSā combo is the cure for all the challenges of automation without giving any solid proof till date. This is the reason, i decided to challenge him. He is a brilliant coder but needs to fine tune his business skills.
I respect all those great veterans and warriors who have put up tremendous amount of time, efforts, hard earned money and energy in building Ignition, standing up against big automation giants. At the same time, we must keep an open mind to welcome the disruptive technology as well.
BTW, i did some deep research and built a solution for Ignition to consume big MQTT JSON payloads. This is my conclusion:
For large scale IoT data collection, MQTT vs FTP, MQTT wins hands down. FTP is the second best option, in case MQTT is not available. Congratulations. You won the debate. 
I never claimed anything like that, I just said new approach based on new technology like nodeJS, SVG etc. may be worth trying due to its advantages. Sorry if it sounded like that to anyone, it was just a discussion to trigger thought process. See discussion J2EE v/s NodeJS for real time applications
I first started using JAVA Apache Tomcat server connected to Ignition to serve my ARSCADA and other clients in the first version of it. However I thought of trying nodeJS instead of it, as it was gaining a lot of importance importance in IT world in past several years and offered some advantages over Apache Tomcat server such as small footprint, support for push technology (Socket.IO), faster response etc, Each technology has its advantages and disadvantages and it takes time and effort to build and try them. In mission critical applications such as SCADA, users often try previous generation proven technology due to the risks involved but that shouldn't stop us from evaluating new technology. JAVA was also a new technology when Ignition started in 2003, and no one thought of using JAVA J2EE for real time application that time, which was pioneered by Ignition, and now it has become state of the art for even SCADA.
Of course hats off to all Ignition great stalwarts who developed this massive, robust and award winning product. I am just trying to extend its functionality leveraging new developments in technology such as AR, SVG, etc.
Though I haven't made the download of trial version available yet (it's under preparation) but I have made demo videos of all modules on my web site to give an idea about them.
If your SQL approach is superior then I am open to trying it or any other approach in stead of my GCM. My main objective is to drive my main modules viz , ARSCADA, mSCADA etc. thru NJSCADA, where my value addition to Ignition exists. GCM as I said is a glue, I am definitely going to try alternative approaches for it if there is a merit at a later point in time.
My initial thinking was the direct connect JAVA module approach will be suitable for me as its memory to memory data transfer which will be very fast (and it is indeed) , rather than SQL approach which involves disk IO which may be slow. I had also thought of trying JYTHON approach but thought, being interpreted language , it may not be that efficient. Also I didn't have an experience with SQL performance in Ignition and jython gateway scripting, so I took the JAVA approach. Now that you have given the confidence in SQL response times, I will definitely try both the approaches in addition to GCM. That is what this discussion forum is all about, sharing experiences.
Thanks for your open mind and taking it in the right spirit. Evaluate my solution and give your feedback. I am also willing to learn something better from others.
I have great admiration for your coding skills and deep knowledge in Ignition. But, when you are trying to sell a piggy back SCADA on top of Ignition, it sounds to me like you are trying to sell ice to an eskimo. You may need to reinvent your product, if you want to sell.
You claim that you are trying to offer something which Ignition doesn't. Tell me a couple of key points to support your claim in few lines.
I felt the need for AR in SCADA which is an upcoming technology so my first product was ARSCADA. I have seen some AR solution in ABB SCADA in a video of an Industrial Automation Conference. But that was based on XBOX and required 3D modelling of the whole plant which is expensive. Instead I saw the upcoming technology of 3D virtulal tours (www.krpano.com) which can be easily built using 360degree cameras. Operators can visualize the plant in these 360degree images in addition to the 2D schematic diagrams in traditional SCADA. So I thought of adding this module to Ignition as ARSCADA module which Ignition doesn't have. I thought it will add value to operators as it gives a physical identity of plant without much cost and effort.
Secondly I saw the PubNub tool which allows us to integrate any two applications over Internet using their libraries supporting 70+languages. I thought this can be used to visualize Ignition parameters on Dashboards on remote devices including mobile phones. So the RM SCADA came. Then ability to host AR SCADA screens to remote devices was added so that even the virtual tours can be seen from remote devices using the RM SCADA module using PubNub.
Thirdly, I felt the need for a Simulation tool so I developed UNISEMS tool based on FORTRAN which is widely used in Fossil and Nuclear power plant simulators. A lot of legacy code and expertise is available in Simulation industry which can be reused in building Simulators to Ignition based plants with this tool.
Fourthly I added an OPC UA module in place of GCM to communicate with Ignition so that I can Integrate my modules thru OPC UA rather than having to install GCM and also it will allow me to integrate with any other SCADA supporting OPC UA.
Fifthly having components written for AR SCADA in SVG/JavaScript language , I thought why not develop a 2D interface like Ignition reusing these components and adding missing features like pipes/ symbols/ etc so that we have a SCADA like tool as well based on new technologies, so the mSCADA came which includes PubNub Dash boards as well. (Of course this doesn't make sense for Ignition as addon, but may be it may make sense for some other stand alone applications like Groov or some edge devices).
Sixthly wanted to leverage NodeJS instead of Tomcat and J2EE. Hence the NJSCADA product came instead of Tomcat.
In the mean time your FTP idea came which I didn't still understand so nothing came out of that! But you made some strides in interfacing with Ignition thru SQL+DJ+REACT+PYTHON. Congrats for that excellent work. Just wanted to know if you can do two way communication with Ignition, as GCM can, and its an important requirement for me to write to tags in Ignition. I think you can achieve it with two way binding the DB tags with Ignition. But that's some thing to explore.
Today, every decent plant is monitored by security cameras in all critical areas. Every control room has got a full fledged 24x7 video monitoring and recording station approved by plant insurance. If required, plant technicians and operators are authorised to record any critical incident with their approved mobile and send the video to the support team. A 3D graphics is a big load on the system and no consultant will approve a 3rd party control screen in a production plant. A sticky bit or a delayed response can cost lives and loss of production. Ignition will NOT take any responsibility for any failure on such systems. ARSCADA looks like an outdated dead technology for me.
PubNub is NOT your technology. It's a paid service and anybody can decide on this. Nobody needs your service or expertise to use PubNub.
UNISEMS is a just a old fashioned simulation tool. Tons of awesome simulation tools are freely available on the internet. Nobody will pay you for this.
Any OPC UA client can consume data from Ignition OPC UA server. I don't see any major breakthrough on this front.
SVG/JavaScript/NodeJS -- Again, nobody will approve a third party client to run their plant. Bear in mind, React perspective client is coming in a couple of months. The game is over.
Security cameras and IP cameras are for live surveillance of plant, ARSCADA is for viewing offline images with real time values on it for training and visualization of plant including inaccessible areas. Its not a 3D graphics technology and has very low overhead. Let people decide if there is a value in it.
As systems integrators our job is to integrate diverse systems and technologies such as PubNub and Ignition. Did I ever say PubNub is my technology. I am bringing the new possibilities to table with it to the notice. PubNub is there for people to use it to their imagination.
UNISEMS may be of use to some existing simulators which might want to migrate to Ignition and reuse legacy code. Some people may still prefer the older approach. You never know if requirement exists. I just made the know how available to the notice. Please use the tons of free tools.
OPC UA helps me integrate my modules with any SCADA system having OPC server, thats the breakthrough for me. I am not inventing any thing new here. No one is stopping you to consume what ever data you want with OPC UA. Technology exists for you to use it.
SVG/NodeJS etc ā OK fine let no one use them . I am happy creating some thing new using the Ignition SDK and the platform that they have offered.
These are your views please donāt bias others if you donāt see any thing positive. I didnāt ask your opinion. I just try replying to your questions.
Thanks!
When you have to serve 100's of hungry analytical /ML clients, "load distribution, IT/security compliance, scalability and high availability" will become big issues. I think, this kind of heavy payload consumers can be handled better by "database+apache/Django JSON api" back end servers. What's your opinion?.
Serving clients on HTTP protocol has higher overhead compared to socket.io (websocket) in terms of packet bigger packet size (header footer size is high) as well as you have to serve individual clients one by one in HTTPwhere as one packet can be broadcast to all clients in socket.io. Secondly data base IO will definitely be slower than memory to memory by common sense, though it may not be significant if packet size is small. We have to see loading of Ignition server CPU is loaded for a medium to large project.
Implementation wise definitely database approach will be easier and scaleable I guess. Security wise it will be the same if its on HTTPS protocol. It all depends on application. Why would one need ML on 100ās of clients? ML is implemented on server I guess.
My requirement was faster memory to memory transfer of packet, but thru database would also fine for this i guess.
Your GCM module sounds very impressive. I would like to evaluate your GCM module, if you can publish/subscribe JSON payloads via GCM-MQTT client for 100,000 tags. Is it possible?.
The purpose of GCM is so far limited to serve my modules (AR-SCADA and RM SCADA, the mSCADA and UNISEMS excluded right now). I am trying to make an evaluation version available on GitHub pretty soon. You can then evaluate its potential and limitations. We can then have a look at the other possibilities, if its achievable or not.
This Ignition project can serve 15,000 dynamic tags from transaction groups as JSON api. You can add 1000's of tags to the transaction group and serve them as JSON payload.
To enable unsigned modules in ignition, just add the last line in your ignition.conf.
Java Additional Parameters
wrapper.java.additional.1=-XX:+UseConcMarkSweepGC
wrapper.java.additional.2=-XX:+CMSClassUnloadingEnabled
wrapper.java.additional.3=-Ddata.dir=/var/lib/ignition/data
wrapper.java.additional.4=-Dorg.apache.catalina.loader.WebappClassLoader.ENABLE_CLEAR_REFERENCES=false
#wrapper.java.additional.5=-Xdebug
#wrapper.java.additional.6=-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8000
wrapper.java.additional.7=-Dignition.allowunsignedmodules=true
Download demo project from Git:
Thatās very impressive. What is the cycle time to read 1000 tags from data base to php? i.e. how many times can you fetch 1000 tags json payload in a second from db to .php server? And how much time does Ignition take to write 1000 tags from transaction group to DB? And waht is the approximate packet size of a typical 1000 tag json payload? Transaction groups is indeed a nice built in feature of Ignition and it can be used in many innovative ways like this.
I am running the complete project on an abandoned home pc (2008 model) on Ubuntu 18.04, 4 GB RAM. To update 5000 tags on MySQL, TG takes around 3 to 4 secs. If i run it on "H2 in-memory db", for 10,000 tags it takes less than 4 secs. The performance is directly proportional to the RAM. I would like to run it on a 32/64GB RAM machine with H2 IMDB and see some lightning performance 
Pls check out this one.
In my opinion its too slow! I tested my TCP socket approach for 1000 tags (about 11KB data) It took <20 milli seconds from Ignition to NJSCADA (installed on same physical machine as Ignition. I am working on implementing on two seperate servers over LAN and see the response. )
That's a very impressive benchmark. In case, you serve it as JSON payload, what will be the performance you think?.
Basically I am sending only values of tags in stead of tag/value pairs (As in JSON) every cycle to optimize response. The Tag names are sent only once in first cycle and subsequently only values as long as the packet it the same. So I would say with JSON (tag/value pairs) the size may approximately be 2.5 to 3 time iā¦e for my example it could be about 30KB max. However the increase in cycle time may not be in same proportion (i.e. 2.5 to 3 times, it may increase from 20 to 30 milliseconds max. (Actually 20 ms is on higher side, on average it takes about 15 milliseconds.)
