Has anyone been successful in adding a JDBC driver for Databricks? We are particularly interested in Azure Databricks and only querying data out of the data lake. I have downloaded the Databricks JDBC driver jar file from the Databricks and created a new driver in the Ignition geteway but don't have anything to connect to yet to test. Just wondering if anyone else has ever done this.
1 Like
Yes, our team has connected Ignition to Azure Databricks. We are running Ignition 8.1.45
- Download the latest JDBC driver from Databricks and extract the files from the ZIP: https://www.databricks.com/spark/jdbc-drivers-download
- Login to the web interface of your instance of Ignition.
- Go to Config>>Databases>>Drivers and click on Create new JDBC Driver.
- Give it a name like "Databricks"
- In the classname line type "com.databricks.client.jdbc.Driver"
- In the JAR files click on "choose file" and locate the DatabricksJDBC42.jar file on your hard drive.
- Driver Type = "Generic"
- Default URL values and instructions etc
- Default Validation Query = SELECT 1
- Default Translator: POSTGRES (Note: we could have taken the time to create a new Ignition translator config that would work better with Databricks ANSI SQL but POSTGRES works for most queries.)
- Click on Save Changes.
- Go to Config>>Databases>>Connections and click on Create New Database Connection.
- Give it a name like "Databricks"
- JDBC Driver = "Databricks"
- Connection URL : Login to your Databricks workspace.
A) Go to Compute>>SQL Warehouses
B) Select the SQL Warehouse you want to connect to for compute.
C) Click on Connection Details. Then copy the JDBC URL string to the
connection URL in Ignition. - SUPER IMPORTANT At the very end of the connection string after the final semi-colon add "EnableArrow=0;" This command disables a memory thing that doesn't play nice with the Databricks JDBC JAR and JAVA 17 that Ignition runs on. SEE: THIS URL for details
- Username = token
- Password = [Your Personal Access Token (Developer Access Token) from your Databricks workspace]
- Set Validation Timeout to at least 900000 so it only sends the keep alive every 15 minutes instead of the default 10 seconds.
That's all you should need to connect. Make sure your SQL statements include the full [catalog].[schema].[table_name] reference.
If you do care to use arrow for whatever reason, this can also be solved by adding an additional "--add-opens" directive in ignition.conf like:
wrapper.java.additional.N=--add-opens=java.base/java.nio=ALL-UNNAMED
where N is the next unused number.
3 Likes
Does Ignition have a Databricks Translator already configured? I found documentation for a Snowflake one but no Databricks.
Not that I'm aware of. Support might have something if they've ever helped somebody set one up.