Hello there,
I have never connected Ignition gateway with DataBricks EDL connection. I want to do this now with JDBC Driver.
I’ve added JDBC driver with spark jar. Now I am not getting how to get Connect URL and generate token to connect required schema.
Please suggest.
Ignition version: 8.1
Spark version: 4.2
I don’t really understand what Databricks is, but I don’t think this is going to work like you think.
It looks like Databricks uses JDBC to take in data, but I don’t think you can use JDBC to get data out of it.
@Gaurav.Shitole I happen to work at the same company, and the protocols for token generation and connecting to the company’s Databricks environment, and gaining access to datasets will likely not be similar for others. I can send you some documentation specific to us.
https://docs.databricks.com/dev-tools/api/latest/authentication.html
https://docs.databricks.com/integrations/bi/jdbc-odbc-bi.html
I'm interested in using DataBricks to store history data directly from Ignition, have you had any experience/success leveraging databricks?
I've seen it done a few ways. At a basic level, connect Ignition to your Databricks cluster via JDBC or API. As long as your cluster has the proper permissions, you can perform INSERTs and UPDATEs on tables you've created in Databricks.
That may not be the most robust way or allowed in your company. You may want to consider Databricks' data ingestion for historical data.
I know Databricks has updated its product a bunch since this post was originally written. I just posted some directions to connect Ignition to Databricks in another thread in case it would help someone else.