While working with Microsoft Fabric and Azure Databricks, integration can bring out the best in both platforms. Many organizations aim to integrate their Databricks Unity Catalog with Microsoft Fabric to streamline their data management processes. While documentation (Integrate Databricks Unity Catalog with OneLake — Microsoft Fabric | Microsoft Learn) exists on this topic, locating environment and configuration details can sometimes be challenging, particularly for first-timers. This blog post is a practical guide on how to navigate these challenges.

Understanding the Basics

We want to connect our external Unity Catalog, particularly the external tables from it, with Microsoft Fabric. This integration enables you to manage and query your data within Microsoft Fabric seamlessly.

Key Details Required for Integration

When setting up the integration, you’ll need several key details:

·     Databricks Workspace URL

·     Databricks Workspace Token

·     Unity Catalog Name

·     Schema Details

·     Microsoft Fabric Workspace ID

·     Lakehouse ID

·     Connection ID

Each of these pieces of information is crucial for the proper configuration of your environment.

Step-by-Step Guide to Configuration

Let’s break down the process step by step, focusing on where to find the necessary information and how to use it.

1. Databricks Workspace URL

·     Where to Find It:
Navigate to your Databricks workspace and capture the URL from your browser’s address bar.

·     Important Note:
Ensure the URL does not have a trailing slash (/). It should end with .net

 

2. Generating a Databricks Workspace Token

·     Where to Find It:
In your Databricks workspace, go to the User section under Settings. Then, navigate to Security > Developer > Access Token.

·     Generate New Token: Click on “Generate New Token”.

·     Enter Details:

·     Comment: Add a descriptive comment.

·     Lifetime: Select the token’s duration.

·     Click Generate:

·     Copy the Token: Immediately copy the token, as it can’t be retrieved later.

3. Unity Catalog Name and Schema Details

·     What You Need:
You’ll need the name of the Unity Catalog and the specific schemas you plan to create a shortcut for in Microsoft Fabric.

·     Where to Find It:
This information can typically be found within the Databricks workspace under the Unity Catalog section.

4. Microsoft Fabric Workspace ID, Lakehouse ID, and Connection ID

·     Where to Find It:
These IDs can be found within your Microsoft Fabric environment. They are necessary for establishing the correct connection points between Fabric and Databricks.

·     How to Use Them:
These IDs need to be correctly entered into your Python script

Lakehouse and workspace ID

5. For Connection id details follow

Integrate Databricks Unity Catalog with OneLake — Microsoft Fabric | Microsoft Learn

6. Update details in notebook

Once all the configuration details are noted, update the variables in notebook like — Integrate Databricks Unity Catalog with OneLake — Microsoft Fabric | Microsoft Learn

7. Expected outcome

After running the notebook, expect the short-cut created for your tables in lakehouse

8. Unlocking Managed Delta Tables: How a Simple Code Change Enables Sync

While the official documentation might not explicitly mention it, there’s a handy trick to get managed Delta tables working seamlessly with sync.

You can easily sync both external and managed tables by using a utility script from Microsoft documentation. Just download the util.py file from this GitHub link to your Fabric workspace.

Once copied, change below line of code to include Managed table type

And then simply reference this script in your code to sync, and you’ll be all set to sync your external and managed tables seamlessly.

Conclusion

Integrating Unity Catalog with Microsoft Fabric streamlines data management, enabling seamless access and querying across both platforms. This guide walked you through the key steps and configurations needed for a successful setup. By following these steps, you can efficiently link your Unity Catalog with Microsoft Fabric, enhancing your data workflow.