Let’s build a Big Data Kafka client module for Ignition – "IgnitionKafkaClient.modl":
Why do we need a “Big Data Kafka client” for Ignition?:
Few years back, most of us would have never imagined a SCADA project generating and logging millions of records per day into a historian database. Today, Big Data has become the hottest topic among SCADA professionals including this forum. Even highly accomplished SCADA veterans are becoming bit nervous and taking a back step when this topic is opened up in a public forum. Yes… SCADA servers can NOT remain as holy cows in isolated control rooms any more. SCADA business has been pushed into the era of Big data.
What’s the big deal about Big Data?:
Investors and stake holders have realized that data is money. Big Data is Big money. They want to make more money from their data silos which is getting rotten in control room archives … as simple as that !!.
SCADA historians are traditionally designed to work with RDBMS sql database. They don’t know anything about NO SQL database. When it comes to logging millions of records and serving real time data to 100’s of clients concurrently, the RDBMS sql databases are getting severely strained and are not able to cope up with the heavy demand-supply load. The reason is simple. They are NOT designed to handle this kind of load. It’s like feeding an insatiable hungry monster. Whatever scaling we do, it will very soon reach it’s limit and will demand more.
Got it… come to the point !!:
This is the challenge which triggered my curiosity to build an Ignition module to handle Big Data. I did some deep research and found Apache Kafka as the best and feasible platform for building an Ignition module. You can read the first writeup in this page and dig deeper to know more about Kafka.
First, i am NOT a Java expert. I have been playing around with it on and off as an enthusiastic programmer, built some solutions but nothing serious. Nevertheless, necessity is the mother of invention.
Kafka is built on Java and Ignition too is built on Java. There’s a great amount of synergy and eco-system existing between the two. I found, all tools and battle tested building blocks are available to build a decent Kafka client for Ignition. This is the reason, i am initiating this community project for Ignition users. Let’s put our heads together and build our own Kafka client for Ignition.
I will explain the project in my next writeup. All are welcome 
Thanks.
Ignition is a SCADA monster. Let’s unleash it !!.