Before I go down this road, I wanted to check: does anyone have an existing module for KSQL?
Background:
We have a topic that contains sensor data for ~6000 locations and is terabyte size when data is there for just 3 days. This means to get data out of it for a single location, use of Kafka consumer is not feasible because even if we polled for just the most recent record, the return will be huge and 99% of it will be data not relevant to the location which is running Ignition.
That same data also lives in data bricks and we do have a JDBC connection to that, however by the time the data meanders its way to data bricks, it is around 1-2 hours old.
In the web portal for Kafka (Lense) you can issue queries in SQL against a Kafka broker and get a fully relevant response (i.e. include store number and sensor key in the WHERE statement).
So in order to make pulling from humongous data into a single site Ignition possible, I am looking into KSQL because I think it has potential to be exposed as Jython scripting in Ignition.
I simply wanted to check here if anyone has already had this experience or one similar.
Thanks,
Nick