Fabric - Source Control & Branching - Part 2

Jon Lunn
2 days ago
11 min read

In part 1 [https://www.cloudformations.org/post/fabric-source-control-branching-part-1]

We looked at how to set up Fabric for a multi-developer team, allocating branches and workspaces and:

Dev team each get their own workspace
Branches get allocated to workspaces
DevOps is the prime source of source control
We know what each platform responsibilities are for Fabric & DevOps

Now we're going to look at deploying Fabric items through DevOps using the Microsoft-supported Fabric CI/CD Python library.

Set up DevOps components
- Environments
- Libraries
Creating an DevOps YAML deployment pipeline
Creating a Fabric CI/CD script to deploy the items
Using AI Tools to help speed up development of DevOps

This blog will not go fully in depth of all the sections, but enough to get you started on creating your own Fabric CI/CD pipeline.

Fabric CI/CD

Fabric CI/CD is a Python library developed and now supported by Microsoft, that sits over the Fabric API's to have a friendly experience for deploying items. It now supports most items in Fabric, so it should not be an issue in the most used objects. You can use it in two ways, full repo deployment or you can deploy individual items.

Using the full deployment option works out your dependences, so a notebook that a pipeline relies on will be deployed first, then the pipeline. Doing it some objects at a time may cause issues and fail if you haven't deployed dependant items first. You can also limit items, so if there are items in your repository that are not supported, they will not be deployed and cause an error.

Using the basic example from the Fabric CI/CD website:

from azure.identity import AzureCliCredential from fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items 

token_credential = AzureCliCredential() 

# Initialize the FabricWorkspace object with the required parameters 
target_workspace = FabricWorkspace
		(    workspace_id="your-workspace-id",     
			environment="your-target-environment", 
	   		repository_directory="your-repository-directory", 
			item_type_in_scope= 	
					["Notebook", "DataPipeline", "Environment"], 

token_credential=token_credential,  # or any other TokenCredential )  

# Publish all items defined in item_type_in_scope publish_all_items(target_workspace) 

# Unpublish all items defined in item_type_in_scope not found in 
# repository 

unpublish_all_orphan_items(target_workspace)

You can see it just requires a few parameters:

token_credential - The authentication token to deploy with. It must be added to the workspace you are deploying to
workspace_id - The workspace guid
repository_directory - The path of the repo in DevOps
item_type_in_scope - (Optional) Which Fabric items are in scope for deployment
environment - Which environment you are deploying to

This is that base script, you can adjust it to your own requirements. You may have noticed why an 'environment' setting. This is used to drive the items metadata that needs to be adjusted when deploying from Dev > Test > Prd, and that uses the 'parameters.yaml'

Parameter File

Your Fabric items contain metadata, for example default lake houses, connection id's etc.

Let take a look at a notebooks metadata:

In here is the default lake house settings and a workspace Id. So if you deploy it from your Dev to Test workspace, it will still be pointing to those items. The 'parameters.yaml' is set up to do a look up for the environment parameter, then do a find and replace on those values.

find_replace:
    # Bronze Lakehouse
    - find_value: "6a4db8ac-38e2-4363-a70d-04d2ff8066f3" 
      replace_value:
          tst: "19881bb9-2762-4882-b00f-3ff74ba1c5ea" 
          prd: "19881bb9-2762-4882-b00f-3ff74ba1c5ea" 
      item_type: "Notebook" 

    # Workspace Id
    - find_value: "a3b5788f-685d-43aa-8e0a-8e588d8ea8ff" 
      replace_value:
          tst: "30510e9c-d8ed-4232-8d90-989821d80c5d" 
          prd: "30510e9c-d8ed-4232-8d90-989821d80c5d"

In the python script above you would set the parameter as 'enviroment="tst"'. This would find the value '6a4db8ac-38e2-4363-a70d-04d2ff8066f3' and replace it with '19881bb9-2762-4882-b00f-3ff74ba1c5ea'.

You can name the environment anything you like, you don't have to be restricted with 'tst/prd'.

Note that the '# Bronze Lakehouse' will only update notebooks:

item_type: "Notebook"

You can use an array to define multiple types:

item_type: ["Notebook", "DataPipeline"]

So you can target items specifically or do a global find/replace on values.

I've used it to update connection id's in pipelines, SQL DB connections in semantic models, lakehouse shortcut connection items and loads more. It can also be used to change the 'Active Set' in Variable Libraries, when deploying them to a new workspace, which saves a manual click later.

The parameter.yml file needs to be in the location of the Fabric repo, and not in any other folder. My set up is like this:

repo root
│
├── cicd
│   └── deploy_to_fabric.py
│
└── src
    └── ms.fabric
        └── dev
            ├── parameter.yml
            ├── Notebook_Load_Customers.Notebook
            ├── Pipeline_Daily_Load.DataPipeline
            ├── Sales.SemanticModel
            └── Sales_Report.Report

Now we have the basic process in place, what else do we need to set up in DevOps?

Environments

If you have a Dev/Test/Prod workspace set up, you'll need an environments for each stage you will deploy to, in this case it will be two, Test and Prod.

Environments are just the deployment target, and you can see the deployment history, but more importantly set up approvals and checks.

Approvals - Requires someone to approve before a pipeline deploys to that environment, for example PROD
Checks - Add gates before deployment, such as branch control, business hours, or manual approval.

Approvals should be required before pushing to production, it should all be validated, tested before going anywhere to Prod. So get your manager, senior etc to approve (Then they can take the blame if things go wrong).

Libraries

Not to be confused with Fabric's Variable Libraries, but they do the same function. They are a library for shared DevOps pipelines. I would create one for each environment you are going to deploy to.

In this case we'll be storing the service principal, tenant and workspace Id's, that the DevOps pipeline will pick up and send to the python script. You can link these to Azure Key Vaults (AKV) and have all your sensitive items in AKV. In this case I'm keeping the tech choices down to Fabric and DevOps. I would recommend setting anything sensitive as secure using the padlock icon, and securing the libraries so only the deployment pipeline and release owners can use it.

Service Connections

In DevOps, you can set up service connections that can control authentication to items, however the Fabric CI/CD library uses a Python 'TokenCredential' not the service connections.

You can add in a step in the pipeline to authenticate to the service connection via a AzureCLI task, then the python script can use that AzureCLI login context.

The second option for authentication is to use the direct service principal details from the library, which are loaded and set in the pipeline as parameters. These are then used in calling to the python script in the deployment pipeline stage. This is the approach that will be used in this demo

DevOps Pipelines

Not being a master at DevOps, I turned to ChatGPT (It's not cheating, it's a tool) to help me build the pipeline. I've managed to update some pipelines that had been handed over to me in some projects, but this really was the first time I need to create one from scratch. So I had the base requirements:

Manually run pipeline
Restrict item types to deploy
Deploy to test workspace
Deploy to prod workspace
- With an approval check
Use the library for any parameters needed to drive the pipeline
Send any parameters to the Python script

As mentioned, the Fabric CI/CD pipeline can be used in two way, deploy the repo, or deploy individual items. In the example I'll show here will be deploy selected item. I would recommend two pipelines, one for a big update, and another so smaller focused deployments.

After a few adjustments I got this:


# Manual-only pipeline
trigger: none
pr: none

parameters:
- name: itemTypes
  displayName: Item types to deploy (Fabric CICD item_type_in_scope)
  type: object
  default:
    - Notebook
    - DataPipeline
    - SemanticModel
    - Report
    - Lakehouse
    - VariableLibrary
    - UserDataFunction
    - DataAgent

- name: items
  displayName: Specific items to deploy (names, optional)
  type: object
  default: []   # e.g. ["MyNotebook","MyPipeline"]

pool:
  vmImage: ubuntu-latest
stages:
- stage: Deploy_Test
  displayName: Deploy to TEST
  variables:
    - group: fabric-cicd-test-spn
  jobs:
  - deployment: Deploy
    displayName: Deploy (TEST)
    environment: fabric-test
    strategy:
      runOnce:
        deploy:
          steps:
          - checkout: self

          - task: UsePythonVersion@0
            displayName: Use Python 3.12
            inputs:
              versionSpec: '3.12'
              addToPath: true
              architecture: 'x64'
#
          - script: |
              python --version
              python -m pip install --upgrade pip
              pip install fabric-cicd azure-identity azure-keyvault-secrets
            displayName: Install dependencies

          - script: |
              pwd
              ls -la
              find . -name deploy_to_fabric.py || true
            displayName: "Debug locate deploy_to_fabric.py"

          - script: |
              python cicd/deploy_to_fabric.py
            displayName: Deploy to Fabric (TEST)
            env:
              AZ_TENANT_ID: $(AZ_TENANT_ID)
              AZ_CLIENT_ID: $(AZ_CLIENT_ID)
              AZ_CLIENT_SECRET: $(AZ_CLIENT_SECRET)
              FABRIC_ENV: tst
              FABRIC_WORKSPACE_ID: $(FABRIC_WORKSPACE_ID)

              # Repo / fabric root
              BUILD_SOURCESDIRECTORY: $(Build.SourcesDirectory)
              FABRIC_REPO_SUBDIR: src/fabric/dev_workspace

              # Parameters (arrays -> comma-separated strings)
              ITEM_TYPES: ${{ join(',', parameters.itemTypes) }}
              ITEMS: ${{ join(',', parameters.items) }}

- stage: Deploy_Prod
  displayName: Deploy to PROD
  dependsOn: Deploy_Test
  condition: succeeded()
  variables:
    - group: fabric-cicd-prod-spn

  jobs:
  - deployment: Deploy
    displayName: Deploy (PROD)
    environment: fabric-prod
    strategy:
      runOnce:
        deploy:
          steps:
          - checkout: self   # ✅ REQUIRED for deployment jobs
          - task: UsePythonVersion@0
            displayName: Use Python 3.12
            inputs:
              versionSpec: '3.12'
              addToPath: true
              architecture: 'x64'

          - script: |
              python --version
              python -m pip install --upgrade pip
              pip install fabric-cicd azure-identity azure-keyvault-secrets
            displayName: Install dependencies

          - script: |
              pwd
              ls -la
              find . -name deploy_to_fabric.py || true
            displayName: "Debug locate deploy_to_fabric.py"

          - script: |
              python cicd/deploy_to_fabric.py
            displayName: Deploy to Fabric (PROD)
            env:
              AZ_TENANT_ID: $(AZ_TENANT_ID)
              AZ_CLIENT_ID: $(AZ_CLIENT_ID)
              AZ_CLIENT_SECRET: $(AZ_CLIENT_SECRET)
              FABRIC_ENV: prd
              FABRIC_WORKSPACE_ID: $(FABRIC_WORKSPACE_ID)

              # Repo / fabric root
              BUILD_SOURCESDIRECTORY: $(Build.SourcesDirectory)
              FABRIC_REPO_SUBDIR: src/fabric/dev_workspace

              # Parameters (arrays -> comma-separated strings)
              ITEM_TYPES: ${{ join(',', parameters.itemTypes) }}
              ITEMS: ${{ join(',', parameters.items) }}

So lets break this down a bit

# Manual-only pipeline
trigger: none
pr: none

You do have the option of running the pipeline manually and it being triggered by an event, for example a commit to the branch, which can then be deployed to the test environment.

parameters:
- name: itemTypes
  displayName: Item types to deploy (Fabric CICD item_type_in_scope)
  type: object
  default:
    - Notebook
    - DataPipeline
    - SemanticModel
    - Report
    - Lakehouse
    - VariableLibrary
    - UserDataFunction
    - DataAgent

This list restricts the type of items to be deployed, and is passed as a parameter to the python script.

- name: items
  displayName: Specific items to deploy (names, optional)
  type: object
  default: []   # e.g. ["MyNotebook","MyPipeline"]

This section is for input on what items are for deployment, and if you look in the repo, all the items will have a suffix on the folder of the type, '.Notebook' or '.DataPipeline'. You'll have to put into the array "Notebookname.Notebook" for it to pick up the item for deployment.

pool:
  vmImage: ubuntu-latest

The next item is what sort of virtual machine (VM) will be used to run the process. In this case an VM based on the ubuntu version of Linux will be used to load and deploy the items.

stages:
- stage: Deploy_Test
  displayName: Deploy to TEST
  variables:
    - group: fabric-cicd-test-spn

So next is defined the stage, and in those stages can be jobs and tasks to run. In this case our first stage will be to deploy to test. The library that will be used that contains the required variable for this stage is declared here, so 'fabric-cicd-test-spn' is the library name.

  jobs:
  - deployment: Deploy
    displayName: Deploy (TEST)
    environment: fabric-test
    strategy:
      runOnce:
        deploy:
          steps:
          - checkout: self

          - task: UsePythonVersion@0
            displayName: Use Python 3.12
            inputs:
              versionSpec: '3.12'
              addToPath: true
              architecture: 'x64'
#
          - script: |
              python --version
              python -m pip install --upgrade pip
              pip install fabric-cicd azure-identity azure-keyvault-secrets
            displayName: Install dependencies

          - script: |
              pwd
              ls -la
              find . -name deploy_to_fabric.py || true
            displayName: "Debug locate deploy_to_fabric.py"

Under the stage are the jobs that need to be run to set up the VM.

Sets Python Version as 3.12
Pip installs required Python libraries not in the default set
Get the script 'deploy_to_fabric.py' from the repo

- script: |
              python cicd/deploy_to_fabric.py
            displayName: Deploy to Fabric (TEST)
            env:
              AZ_TENANT_ID: $(AZ_TENANT_ID)
              AZ_CLIENT_ID: $(AZ_CLIENT_ID)
              AZ_CLIENT_SECRET: $(AZ_CLIENT_SECRET)
              FABRIC_ENV: tst
              FABRIC_WORKSPACE_ID: $(FABRIC_WORKSPACE_ID)

              # Repo / fabric root
              BUILD_SOURCESDIRECTORY: $(Build.SourcesDirectory)
              FABRIC_REPO_SUBDIR: src/fabric/dev_workspace

              # Parameters (arrays -> comma-separated strings)
              ITEM_TYPES: ${{ join(',', parameters.itemTypes) }}
              ITEMS: ${{ join(',', parameters.items) }}

This next part runs 'deploy_to_fabric.py' with the parameters set

AZ_TENANT_ID, AZ_CLIENT_ID, AZ_CLIENT_SECRET - Parameters for authentication to the workspace
FABRIC_ENV - The environment to be used that is in the 'parameter.yaml' file, to base its find and replace on
FABRIC_REPO_SUBDIR - The path of the source items in the repo
BUILD_SOURCESDIRECTORY, - The folder repo items get copied to on the VM
ITEM_TYPES - Which item types can be deployed, set in the pipeline parameters earlier in the process
ITEMS - the objects we are going to deploy

The next stage is deploy to prod, and is mostly the same apart from a few items. The library to be used for the variables, and the environment 'prd' to be used in the parameter.yaml lookup.

There are two items only in the Prod stage:

dependsOn: Deploy_Test
condition: succeeded()

So it will process after Deploy to test, and it must has succeeded in its deployment.

There is no section here that defines approval to continue, that is declared in the Environment setup.

  environment: fabric-prod

As long as the 'fabric-prod' one is set up correctly it will display an approve button to continue.

This now needs to be added to your repo, so create a folder to store it. Don't store it in the folder path of your Fabric items. You can then create a new pipeline.

Select Pipelines in DevOps
One where is your code, select Azure Repos Git
Select your repository
The select Existing Azure Pipelines YAML file
Select the branch, and path for the yaml file
Then review and save it

Deploy To Fabric

So the last needed part for this is the "deploy_to_fabric.py" script, which is called to deploy the items.

Like the modern data engineer I am, I again fired up the AI, and it helped me flesh out a basic script to do the deployment:

Use parameters from the DevOps pipeline
Use a service principal to authenticate to the workspace
Allow deployment of selected items
Print out what is being deployed
Exception handling and warning for the process

It is commented a fair bit, so I'll not break it down to much.

One thing to note, it may be best to move from using a service principal, as a few items in Fabric don't support deployment that way. Fabric User Data Functions don't seem to like it, so test items and see how they work. As mentioned you could use AzureCLI to get the authentication, or move to a service account, and use the Azure Identity Interactive authentication method.

########################################################################################################
# Deployment Template V0.2
# Overview: Basic script to deploy items from a repo to a workspace
# Check https://microsoft.github.io/fabric-cicd/latest/
# The version 0.1.31 is the version this script is based on. Newer or 
# older versions may introduce breaking changes
##########################################################################
##############################


# Import libraries
import subprocess
import os
import sys
import traceback
from pathlib import Path

from azure.identity import ClientSecretCredential, AzureCliCredential

from azure.core.exceptions import ClientAuthenticationError

from fabric_cicd import FabricWorkspace, publish_all_items, append_feature_flag

from azure.keyvault.secrets import SecretClient


# Set library features
append_feature_flag("enable_experimental_features")
append_feature_flag("enable_items_to_include")

#########################################################################################################
# Set up variables
#########################################################################################################

# Base items used to deploy to fabric
tenant_id = os.environ.get("AZ_TENANT_ID", "")
workspace_id = os.environ.get("FABRIC_WORKSPACE_ID", "")
client_secret = os.environ.get("AZ_CLIENT_SECRET", "")
client_id = os.environ.get("AZ_CLIENT_ID", "")
environment = os.environ.get("FABRIC_ENV", "")

# This is the platform types, lakehouse, notebooks
item_types = os.environ.get("ITEM_TYPES", "")
item_type_in_scope = [i.strip() for i in item_types.split(",") if i]

# Where the solution is downloaded on the vm
build_folder = os.environ.get("BUILD_SOURCESDIRECTORY","")
repo_folder = os.environ.get("FABRIC_REPO_SUBDIR","")

##########################################################################
##############################
# Items to deploy
##########################################################################
##############################
# This is the acutal objects that we need, the list of notebooks, 
pipelines etc to be deployed

items = os.environ.get("ITEMS", "")
items_to_include = [i.strip() for i in items.split(",") if i]


##########################################################################
##############################
# Deploy from where
##########################################################################
##############################
repo_root = Path(build_folder)
fabric_subdir = repo_folder
repository_directory = str(repo_root / fabric_subdir)
parameter_file_to_find = Path(repository_directory) / 'parameter.yml'


##########################################################################
##############################
# Deploy process starts here
##########################################################################
##############################

# Check if parameter files exists
if parameter_file_to_find.exists():
    print("parameter.yml found:", parameter_file_to_find)
else:
    print("parameter.yml not found...exiting")
    exit()

# Prompt details
print('-------------------------------------------------------------------
---')
print(f'Using repo files: {repository_directory}')
print(f'The following items are in scope for deployment:')
for x in item_type_in_scope:
    print(f' - {x}')

if not items_to_include:
    print("❌ Deployment aborted: no items specified.")
    print("ITEMS environment variable was empty or invalid.")
    print(f"Raw ITEMS value: '{items_raw}'")
    sys.exit(1)

print(f'Items in this deployment:')
for x in items_to_include:
    print(f' - {x}')

print('Publishing the items')
print("ITEMS env:", os.environ.get("ITEMS"))

try:
    token_credential = ClientSecretCredential(client_id=client_id, 
client_secret=client_secret, tenant_id=tenant_id)

        
    ws = FabricWorkspace(
        workspace_id=workspace_id,
        environment=environment,
        repository_directory=repository_directory,
        item_type_in_scope=item_type_in_scope,
        token_credential=token_credential,
    )

    publish_all_items(ws, items_to_include=items_to_include)

except ClientAuthenticationError as e:
    print("❌ Authentication failed (Service Principal)")
    print(f"Tenant ID   : {tenant_id}")
    print(f"Client ID   : {client_id}")
    print(f"Workspace ID: {workspace_id}")
    print(str(e))
    sys.exit(1)
except ValueError as e:
    print("❌ Invalid configuration or parameters")
    print(str(e))
    sys.exit(1)
except Exception as e:
    print("❌ Unexpected error during Fabric deployment")
    traceback.print_exc()
    sys.exit(1)

print('-------------------------------------------------------------------
---')
print('Deployment script complete')
print('-----------------------------------------------------------------------')

Overview

So now we have the set up of:

Python Script
Environment
Libraries
DevOps Pipeline

Next on to actually deploying it!

Outcomes We Deliver

Products

Advisory

Delivery

Training

Powered by Products. Delivered with Expertise.

Fabric - Source Control & Branching - Part 2

Fabric CI/CD

Parameter File

Environments

Libraries

Service Connections

DevOps Pipelines

Deploy To Fabric

Overview

Recent Posts

Subscribe to to get updates on new posts.

Turn insight into action

If something you have read resonates, let’s talk about what it could mean for your data platform or roadmap.

Powered by products. Delivered with expertise.