Apache Kafka is an open-source streaming platform that was initially built by LinkedIn. It was later handed over to Apache foundation and open sourced it in 2011.
According to Wikipedia:
Apache Kafka is an open-source stream-processing software platform developed by the Apache Software Foundation, written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. Its storage layer is essentially a “massively scalable pub/sub message queue architected as a distributed transaction log,”[3] making it highly valuable for enterprise infrastructures to process streaming data. Additionally, Kafka connects to external systems (for data import/export) via Kafka Connect and provides Kafka Streams, a Java stream processing library.
Publish a JSON payload of 5000 dynamic tags from Ignition every 3 secs with Kafka-Producer(Python) and consume it with Kafka-Consumer(JS).
Step 1 --- Setup Kafka Broker:
Follow this link. It's very easy to setup. Quickstart
Step 2 --- Generate and serve dynamic Ignition tags as JSON payload:
Follow this link. Download and run the "Ignition-IOT-Nirvana" demo project from github.
Step 3 --- Download the Kafka-PubSub and php code
Step 4 --- Publish and Subscribe the JSON payload to the Kafka Broker:
1.To Publish JSON payload:
Concurrently you can PubSub multiple topics on the same broker, run multiple brokers and multiple clients on the same server or scale it out across several servers horizontally !!.
Let’s build a Big Data Kafka client module for Ignition – "IgnitionKafkaClient.modl":
Why do we need a “Big Data Kafka client” for Ignition?:
Few years back, most of us would have never imagined a SCADA project generating and logging millions of records per day into a historian database. Today, Big Data has become the hottest topic among SCADA professionals including this forum. Even highly accomplished SCADA veterans are becoming bit nervous and taking a back step when this topic is opened up in a public forum. Yes… SCADA servers can NOT remain as holy cows in isolated control rooms any more. SCADA business has been pushed into the era of Big data.
What’s the big deal about Big Data?:
Investors and stake holders have realized that data is money. Big Data is Big money. They want to make more money from their data silos which is getting rotten in control room archives … as simple as that !!.
SCADA historians are traditionally designed to work with RDBMS sql database. They don’t know anything about NO SQL database. When it comes to logging millions of records and serving real time data to 100’s of clients concurrently, the RDBMS sql databases are getting severely strained and are not able to cope up with the heavy demand-supply load. The reason is simple. They are NOT designed to handle this kind of load. It’s like feeding an insatiable hungry monster. Whatever scaling we do, it will very soon reach it’s limit and will demand more.
Got it… come to the point !!:
This is the challenge which triggered my curiosity to build an Ignition module to handle Big Data. I did some deep research and found Apache Kafka as the best and feasible platform for building an Ignition module. You can read the first writeup in this page and dig deeper to know more about Kafka.
First, i am NOT a Java expert. I have been playing around with it on and off as an enthusiastic programmer, built some solutions but nothing serious. Nevertheless, necessity is the mother of invention.
Kafka is built on Java and Ignition too is built on Java. There’s a great amount of synergy and eco-system existing between the two. I found, all tools and battle tested building blocks are available to build a decent Kafka client for Ignition. This is the reason, i am initiating this community project for Ignition users. Let’s put our heads together and build our own Kafka client for Ignition.
I will explain the project in my next writeup. All are welcome Thanks.
Please read the first post in this page to setup and practice Kafka.
Ignition Kafka client module name ---- IgnitionKafkaClient.modl
Official Apache Kafka broker — available.
Basic Java Kafka Producer — available.
Java Kafka producer which can read Ignition tags and publish on the Kafka broker — Not yet. This is the core of the project.
Basic Java Kafka Consumer — available.
Java Kafka Consumer which can subscribe to the Kafka broker, consume messages and write to Ignition tags. — Not yet.
Official Ignition Demo project to test the IgnitionKafkaClient.modl ---- available.
Ignition SDK to practice and learn module building — available.
…
Basic Java Producer and Consumer to PubSub messages On Kafka broker:
You can also PubSub from the kafka-producer.py and kafka-consumer.js clients available in the “Ignition-Kafka-Nirvana” project mentioned in the first post above. You can check interoperability between python-kafka-client, nodejs-kafka-client and java-kafka-client.
Feel free to share your ideas and code snippets to build our IgnitionKafkaClient.modl. Thanks.
Thanks.