Microsoft Fabric

Microsoft Fabric is a comprehensive analytics and data integration solution that includes cloud-based data warehousing as part of its extensive suite of capabilities.

Overview

Hightouch lets you pull data stored in Microsoft Fabric and push it to downstream destinations. Connecting Hightouch to Microsoft Fabric requires some setup in both platforms. In Microsoft Fabric, you need to create a service principal and grant it access to your Fabric resource.

Service principal creation

First, we need to create a service principal. These are the credentials Hightouch will use to access your Microsoft Fabric resources.

1. Sign in to the Microsoft Entra admin center.
2. On the sidebar, select Identity > Applications > App registrations.
3. Select New registration.
4. Pick a name for your application and select Accounts in this organizational directory only as the supported account type.
5. Click Register.
6. Note your newly created Application (client) ID and Directory (tenant) ID.
7. Click Add a certificate or secret and then New client secret.
8. Name your client secret and select its expiration date. To maintain access to Fabric, make sure to refresh your client secret prior to its expiration.
9. Note your client secret Value.

For more detailed instructions on service principal creation, you can check out Microsoft's documentation.

Fabric configuration

By default, Microsoft Fabric may not be accessible via service principal. To connect, you'll need to take the following steps.

1. Log in to Microsoft Fabric.
2. Click Synapse Data Engineering.
3. In the top right of the window, select Settings > Governance and insights > Admin portal.
4. On the sidebar, select Tenant settings.
5. Scroll down to Developer settings and select Service principals can use Fabric APIs. Make sure the toggle is set to Enabled.
6. Next go to OneLake settings and select Users can access data stored in OneLake with apps external to Fabric. This toggle should be Enabled as well.

Attaching your service principal to your Fabric warehouse

1. Log in to Microsoft Fabric.
2. Select the workspace that contains your warehouse.
3. Hover over your warehouse and select the three dots menu.
4. Choose Manage permissions from the menu.
5. On the next page, select Add user.
6. Type in the name of your service principal and check the box for Read all data using SQL (ReadData).
7. Select Grant.

Connection configuration

In Hightouch, go to the Sources overview page and click the Add source button. Select Microsoft Fabric and follow the steps below.

Configure your source

Enter the following required fields into Hightouch:

• Host: The hostname or IP address of your Microsoft Fabric instance. If using the hostname, make sure to only insert the hostname and not the full URL (remove the leading http:// and the final /).
• Port: The port number of your Microsoft Fabric instance. The default is 1433, but yours may be different.
• Database: The name of the database in Microsoft Fabric.

Optionally you can enter a Request timeout duration and whether to Trust server certification.

Choose your sync engine

For optimal performance, Hightouch tracks incremental changes in your data model—such as added, changed, or removed rows—and only syncs those records. You can choose between two different sync engines for this work.

The Basic engine requires read-only access to Fabric. Hightouch executes a query in your database, reads all query results, and then determines incremental changes using Hightouch's infrastructure. This engine is easier to set up since it requires read—not write—access to Fabric.

The Lightning engine requires read and write access to Fabric. The engine stores previously synced data in a separate schema in Fabric managed by Hightouch. In other words, the engine uses Fabric to track incremental changes to your data rather than performing these calculations in Hightouch. Therefore, these computations are completed more quickly.

Hightouch recommends using the Lightning sync engine when syncing more than 100 thousand rows of data.

If you select the Basic engine, you can switch to the Lightning engine later. Once you've configured the Lightning engine, you can't move back to the Basic engine without recreating Fabric as a source.

To learn more, including migration steps and tips, check out the Lightning sync engine docs.

Basic versus Lightning engine comparison

The Lightning sync engine requires granting write access to your data warehouse, which makes its setup more involved than the Basic sync engine. However, it is more performant and reliable than the Basic engine. This makes it the ideal choice to guarantee faster syncs, especially with large data models. It also supports more features, such as Warehouse Sync Logs, Match Booster, and Identity Resolution.

Lightning engine setup

To set up the Lightning engine, run the following commands. Make sure to replace your_service_principal with the name of the service principal created earlier.

IF NOT EXISTS (SELECT 1 FROM sys.schemas WHERE name = 'hightouch_audit')
BEGIN
    EXEC('CREATE SCHEMA [hightouch_audit]');
END;
IF NOT EXISTS (SELECT 1 FROM sys.schemas WHERE name = 'hightouch_planner')
BEGIN
    EXEC('CREATE SCHEMA [hightouch_planner]');
END;
GRANT CONTROL ON SCHEMA::hightouch_audit TO [your_service_principal];
GRANT CONTROL ON SCHEMA::hightouch_planner TO [your_service_principal];

Provide credentials

Then you'll enter the credentials from the previous service principal creation step

• Tenant ID (or Directory ID)
• Client ID (or Application ID)
• Client Secret

Test your connection

When setting up Microsoft Fabric as a source for the first time, Hightouch validates your credentials and access to your database. Once the test passes, click Continue to finish setup.

Next steps

Once your source configuration has passed the necessary validation, you've completed setting up Microsoft Fabric as a source. Next, you can set up models to define which data you want to pull from Microsoft Fabric.

The Microsoft Fabric source supports these modeling methods:

• writing a query in the SQL editor
• using the visual table selector

Tips and troubleshooting

Most issues with the Microsoft Fabric source can be resolved by verifying that your Microsoft Fabric configuration is correct. It must allow external connections via service principal.