How to integrate data into Databricks with Windsor.ai

Databricks is a cloud-based data analytics platform built on the Apache Spark engine, designed to process large-scale data efficiently. It provides a collaborative environment for data engineers, scientists, and analysts to build complex machine learning models and perform real-time analytics with a fully managed infrastructure.

Databricks’ key advantages include seamless handling of large datasets, cloud scalability, and integration with AWS, Azure, and Google Cloud services. The platform enhances data processing with Delta Lake for reliability and performance, supports multiple programming languages, and offers built-in AI and ML tools. By bringing automated cluster management, cost optimization, and strong security, Databricks levels up data analysis workflows. 

By integrating Datarbicks with the Windsor.ai data movement platform, you can:

• Automatically extract data from multiple sources and connect it to Databricks, enabling advanced big data processing and AI-driven insights.
• Streamline data ingestion, transformation, and analysis, reducing manual work and ensuring real-time updates for informed decision-making.
• Leverage Databricks’ cloud scalability to efficiently process large volumes of data while optimizing costs with automated resource management.

Explore our step-by-step guide to seamlessly integrate your data into Databricks with the Windsor.ai ELT connector.

1. Create a Windsor.ai account and log in.

2. Select the data source which you want to stream data from, e.g., Google Analytics 4 (GA4). Sign in with your associated Google account and select the next step, “Data preview.”

3. You’ll see your Google Analytics 4 data displayed in your Windsor.ai account. 

Now, let’s proceed with setting up the Databricks environment for data integration.

1. First of all, make sure you have an active Databricks Developer account. Go to Databricks and log in to your developer account.

2. In the sidebar, select Catalog, then click the “+” icon and choose “Create Catalog.”

3. Enter the Catalog name (it can be anything you wish, but it should contain only ASCII letters (‘a’ – ‘z,’ ‘A’ – ‘Z’), digits (‘0’ – ‘9’), and underbar (‘_’)) and click “Create.” 

4. Go to your newly created catalog and click “Create Schema.” Enter the Schema Name (anything you want) and click “Create.”

5. Get the required fields from Databricks to create the connection between Databricks and Windsor.ai.

First, get the Access Token:

• On the top right corner, click on your account and select Settings. 

• Find the Developer section in the sidebar and click “Manage” in the Access Tokens row.

• Click “Generate New Token,” enter a Comment (anything you wish) for your token, and finish with the “Generate” button. 

• Copy the created access token.

6. Now get the Server Hostname and HTTP Path.

Find SQL Warehouses in the sidebar, select the Connection Details tab, and copy the Server Hostname and HTTP Path.

7. You also need to get the Catalog and Schema. 

Find Catalog in the sidebar and copy Catalog and Schema names.

Here we go, you’ve set up the Databrick catalog and schema on the Databricks console and have gathered the required credentials. 

Now, let’s import your data from Windsor.ai into the created Databricks catalog table.

1. Go to your Windsor.ai account and move to the Google Analytics 4 data preview page. Scroll down to data destinations, select Databricks, and click “Add Destination Task.”

2. Enter all the required credentials: 

• Task name (you can provide any based on the data integration purpose).
• Access token, server hostname, HTTP path, catalog and schema you got from Databricks developer console.
• Table name (you can provide any based on the data integration purpose, or it will be automatically created in your catalog schema by Windsor.ai). If you already have a table for your Google Analytics data, you can enter that table name.

Click “Test Connection.”

If the connection is set properly, you’ll see a success message at the bottom; otherwise, an error message will appear. When successful, click “Save” in the lower right corner of the form. The data stream to the Databricks table has started.

3. You can now see the task running in the selected data destination section. The green ‘upload’ button with the status ‘ok’ indicates that the task is active and running successfully.

4. Verify that your data is being added to the Databricks table. Go to your Databricks catalog and select the relevant table.

Cheers! Your Google Analytics 4 data is now integrated into the Databricks and ready for detailed analysis.