web-API to read/write tags json

I felt the need for AR in SCADA which is an upcoming technology so my first product was ARSCADA. I have seen some AR solution in ABB SCADA in a video of an Industrial Automation Conference. But that was based on XBOX and required 3D modelling of the whole plant which is expensive. Instead I saw the upcoming technology of 3D virtulal tours (www.krpano.com) which can be easily built using 360degree cameras. Operators can visualize the plant in these 360degree images in addition to the 2D schematic diagrams in traditional SCADA. So I thought of adding this module to Ignition as ARSCADA module which Ignition doesn’t have. I thought it will add value to operators as it gives a physical identity of plant without much cost and effort.

Secondly I saw the PubNub tool which allows us to integrate any two applications over Internet using their libraries supporting 70+languages. I thought this can be used to visualize Ignition parameters on Dashboards on remote devices including mobile phones. So the RM SCADA came. Then ability to host AR SCADA screens to remote devices was added so that even the virtual tours can be seen from remote devices using the RM SCADA module using PubNub.

Thirdly, I felt the need for a Simulation tool so I developed UNISEMS tool based on FORTRAN which is widely used in Fossil and Nuclear power plant simulators. A lot of legacy code and expertise is available in Simulation industry which can be reused in building Simulators to Ignition based plants with this tool.

Fourthly I added an OPC UA module in place of GCM to communicate with Ignition so that I can Integrate my modules thru OPC UA rather than having to install GCM and also it will allow me to integrate with any other SCADA supporting OPC UA.

Fifthly having components written for AR SCADA in SVG/JavaScript language , I thought why not develop a 2D interface like Ignition reusing these components and adding missing features like pipes/ symbols/ etc so that we have a SCADA like tool as well based on new technologies, so the mSCADA came which includes PubNub Dash boards as well. (Of course this doesn’t make sense for Ignition as addon, but may be it may make sense for some other stand alone applications like Groov or some edge devices).

Sixthly wanted to leverage NodeJS instead of Tomcat and J2EE. Hence the NJSCADA product came instead of Tomcat.

In the mean time your FTP idea came which I didn’t still understand so nothing came out of that! But you made some strides in interfacing with Ignition thru SQL+DJ+REACT+PYTHON. Congrats for that excellent work. Just wanted to know if you can do two way communication with Ignition, as GCM can, and its an important requirement for me to write to tags in Ignition. I think you can achieve it with two way binding the DB tags with Ignition. But that’s some thing to explore.

Today, every decent plant is monitored by security cameras in all critical areas. Every control room has got a full fledged 24x7 video monitoring and recording station approved by plant insurance. If required, plant technicians and operators are authorised to record any critical incident with their approved mobile and send the video to the support team. A 3D graphics is a big load on the system and no consultant will approve a 3rd party control screen in a production plant. A sticky bit or a delayed response can cost lives and loss of production. Ignition will NOT take any responsibility for any failure on such systems. ARSCADA looks like an outdated dead technology for me.

PubNub is NOT your technology. It’s a paid service and anybody can decide on this. Nobody needs your service or expertise to use PubNub.

UNISEMS is a just a old fashioned simulation tool. Tons of awesome simulation tools are freely available on the internet. Nobody will pay you for this.

Any OPC UA client can consume data from Ignition OPC UA server. I don’t see any major breakthrough on this front.

SVG/JavaScript/NodeJS – Again, nobody will approve a third party client to run their plant. Bear in mind, React perspective client is coming in a couple of months. The game is over.

Security cameras and IP cameras are for live surveillance of plant, ARSCADA is for viewing offline images with real time values on it for training and visualization of plant including inaccessible areas. Its not a 3D graphics technology and has very low overhead. Let people decide if there is a value in it.

As systems integrators our job is to integrate diverse systems and technologies such as PubNub and Ignition. Did I ever say PubNub is my technology. I am bringing the new possibilities to table with it to the notice. PubNub is there for people to use it to their imagination.

UNISEMS may be of use to some existing simulators which might want to migrate to Ignition and reuse legacy code. Some people may still prefer the older approach. You never know if requirement exists. I just made the know how available to the notice. Please use the tons of free tools.

OPC UA helps me integrate my modules with any SCADA system having OPC server, thats the breakthrough for me. I am not inventing any thing new here. No one is stopping you to consume what ever data you want with OPC UA. Technology exists for you to use it.

SVG/NodeJS etc – OK fine let no one use them . I am happy creating some thing new using the Ignition SDK and the platform that they have offered.

These are your views please don’t bias others if you don’t see any thing positive. I didn’t ask your opinion. I just try replying to your questions.

Thanks!

1 Like

When you have to serve 100’s of hungry analytical /ML clients, “load distribution, IT/security compliance, scalability and high availability” will become big issues. I think, this kind of heavy payload consumers can be handled better by “database+apache/Django JSON api” back end servers. What’s your opinion?.

Serving clients on HTTP protocol has higher overhead compared to socket.io (websocket) in terms of packet bigger packet size (header footer size is high) as well as you have to serve individual clients one by one in HTTPwhere as one packet can be broadcast to all clients in socket.io. Secondly data base IO will definitely be slower than memory to memory by common sense, though it may not be significant if packet size is small. We have to see loading of Ignition server CPU is loaded for a medium to large project.

Implementation wise definitely database approach will be easier and scaleable I guess. Security wise it will be the same if its on HTTPS protocol. It all depends on application. Why would one need ML on 100’s of clients? ML is implemented on server I guess.

My requirement was faster memory to memory transfer of packet, but thru database would also fine for this i guess.

2 Likes

Your GCM module sounds very impressive. I would like to evaluate your GCM module, if you can publish/subscribe JSON payloads via GCM-MQTT client for 100,000 tags. Is it possible?.

The purpose of GCM is so far limited to serve my modules (AR-SCADA and RM SCADA, the mSCADA and UNISEMS excluded right now). I am trying to make an evaluation version available on GitHub pretty soon. You can then evaluate its potential and limitations. We can then have a look at the other possibilities, if its achievable or not.

This Ignition project can serve 15,000 dynamic tags from transaction groups as JSON api. You can add 1000’s of tags to the transaction group and serve them as JSON payload.

  1. Install SimpleTagProvider-unsigned.modl.
    (This module is built from Ignition SDK example)
  2. This will automatically create 50,000 dynamic tags
  3. ignition_group1.php will serve 1000 dynamic tags as JSON api.

To enable unsigned modules in ignition, just add the last line in your ignition.conf.

Java Additional Parameters

wrapper.java.additional.1=-XX:+UseConcMarkSweepGC
wrapper.java.additional.2=-XX:+CMSClassUnloadingEnabled
wrapper.java.additional.3=-Ddata.dir=/var/lib/ignition/data
wrapper.java.additional.4=-Dorg.apache.catalina.loader.WebappClassLoader.ENABLE_CLEAR_REFERENCES=false
#wrapper.java.additional.5=-Xdebug
#wrapper.java.additional.6=-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8000

wrapper.java.additional.7=-Dignition.allowunsignedmodules=true

Download demo project from Git:

1 Like

That’s very impressive. What is the cycle time to read 1000 tags from data base to php? i.e. how many times can you fetch 1000 tags json payload in a second from db to .php server? And how much time does Ignition take to write 1000 tags from transaction group to DB? And waht is the approximate packet size of a typical 1000 tag json payload? Transaction groups is indeed a nice built in feature of Ignition and it can be used in many innovative ways like this.

1 Like

I am running the complete project on an abandoned home pc (2008 model) on Ubuntu 18.04, 4 GB RAM. To update 5000 tags on MySQL, TG takes around 3 to 4 secs. If i run it on “H2 in-memory db”, for 10,000 tags it takes less than 4 secs. The performance is directly proportional to the RAM. I would like to run it on a 32/64GB RAM machine with H2 IMDB and see some lightning performance :slight_smile:

Pls check out this one.

In my opinion its too slow! I tested my TCP socket approach for 1000 tags (about 11KB data) It took <20 milli seconds from Ignition to NJSCADA (installed on same physical machine as Ignition. I am working on implementing on two seperate servers over LAN and see the response. )

That’s a very impressive benchmark. In case, you serve it as JSON payload, what will be the performance you think?.

Basically I am sending only values of tags in stead of tag/value pairs (As in JSON) every cycle to optimize response. The Tag names are sent only once in first cycle and subsequently only values as long as the packet it the same. So I would say with JSON (tag/value pairs) the size may approximately be 2.5 to 3 time i…e for my example it could be about 30KB max. However the increase in cycle time may not be in same proportion (i.e. 2.5 to 3 times, it may increase from 20 to 30 milliseconds max. (Actually 20 ms is on higher side, on average it takes about 15 milliseconds.)

Awesome. I would like to check the GCM-JSON version. Now, i agree that you have every reason to be passionate about your product. Release the evaluation copy asap.

In fact the local version (NJSCADA on same server as Ignition, but of course its clients ARSCADA, RMSCADA can be anywhere on LAN and PubNub, provided you have Internet connection on Ignition machine) is already ready, but I am working on the LAN version (i.e. NJSCADA on a separate server than Ignition on a LAN). If you want I can send the local version first for evaluation in couple of days. I have to do some documentation and update the documentation. However I thought let me complete the LAN version and send, as it would work in both LAN and Network mode.

1 Like

Performance of TG group on MySQL — 5000 tags.

Performance of TG group on H2-IMDB — 5000 tags.

On “H2-IMDB”, for 5000 tags it takes less than 300 msec. Almost 10 times faster than MySQL disk i/o db.

In memory is obviously understandable (about 300 ms average), but will require huge memory I guess. But not sure if my interpretations is correct:

  1. Do you have only group1 with 5000 tags being written every 1 second?
  2. What are these tag types (float or int or string …?).

In my case I am extracting 5 mata data (like type, access, format, units and definition) about each tag from database and send to NJSCADA in the first cycle , but subsequent cycles I send only values. This is required for my application. It first sort of copies the tags data base from Ignition to NJSCADA for specified tags in first cycle then only values are updated.

Not much. I didn’t see any significant increase in RAM usage when i enable H2-IMDB. Nevertheless, performance is directly proportional to the RAM when you deal with any IM application.

I have configured many groups but i have enabled only group1.

All are floats. This is a sample JSON.

{“group1_ndx”: 4998, “Tag_5000”: 0.7661028504371643, “t_stamp”: “2018-08-13 16:10:46”}, {“group1_ndx”: 4999, “Tag_5000”: 0.029909193515777588, “t_stamp”: “2018-08-13 16:10:46”}, {“group1_ndx”: 5000, “Tag_5000”: 0.732514500617981, “t_stamp”: “2018-08-13 16:10:46”}, {“group1_ndx”: 5001, “Tag_5000”: 0.8658828139305115, “t_stamp”: “2018-08-13 16:10:46”}

my JSON sends only names and values , time stamp and group name is redundant in my opinion, as they are same for each json packet!

1 Like

Thanks for the tutorial like answer…