Source Control & Branching - Part 3

In part 1 [https://www.cloudformations.org/post/fabric-source-control-branching-part-1]

We looked at how to set up Fabric for a multi-developer team, allocating branches and workspaces and:

• Dev team each get their own workspace

• Branches get allocated to workspaces

• DevOps is the prime source of source control

• We know what each platform responsibilities are for Fabric & DevOps

In Part 2 [https://www.cloudformations.org/post/fabric-source-control-branching-part-2]

We looked at the setup of the parts we need to use in the DevOps Pipeline

• Environments

• Libraries

• DevOps Pipeline Yaml

• Python CI/CD setup

Now in Part 3, we're going to deploy the items

Checks to do

There are a few things you need to check first:

1. Does the account your going to use to authenticate to (In this case a Service Principal) been added to the workspace with contributor level access?

2. Has the DevOps Library been updated with the values for the Service Principal and workspace id you are deploying to?

3. Has any DevOps Environment that is production been gated to to allow deployments only after approval?

4. Got the list of items to deploy?

You have done all those, then alright! Lets push that button! Flick that switch! Saddle that horse!

DevOps Pipeline

I've created a new pipeline, by clicking on the 'Pipelines' pane and selecting 'New Pipeline' and selecting the source as 'Azure Repos Git':

Then select the Repo the yaml file is stored in:

Then the option for 'Existing Azure Pipelines YAML file':

Then the branch and path:

Then click save the pipeline, then run the pipeline:

If you are using the base DevOps pipeline, you'll need to specify which item you need to deploy: In this case it is a notebook called 'Variable Library Example.Notebook':

Click next, and it should start running, and show something like this:

When you first run the pipeline, you'll have to click on the 'Deploy to TEST' just to give it permissions on access the Library.

Once that's done, you should see an output like this:

Selecting the 'Deploy to Fabric (TEST)' stages give you an output of the deployment steps.

Typical errors here are normally:

• Authentication errors - The Service Principal, Username hasn't been added to the workspace

• It can't locate the parameter file

• It hasn't deployed an object as the wrong item has been added, or it's not in the listed type.

  
  

If you now look in the environment, you'll now see a pipeline run, so you can track the history of what has been deployed.

Maintaining the process

So you now have a reusable pipeline to use, and mostly you'll be updating the parameter.yml file with the metadata you'll need to find and replace. The other item you may need to update is the DevOps yml, maybe to add in a new item type to deploy. We recently created a Data Agent, and that was added to the list of allowed types and deployed.

Next steps

For my current code, I need to move some items out the DevOps pipeline such as 'FABRIC_REPO_SUBDIR' and 'FABRIC_ENV' and move those parameters to a library, and also parametrise the 'environment: fabric-cicd-prod' item as well.

Other next steps would be to add in stages for pre/post deployment steps, maybe having to run some items like notebooks/stored procedures/pipelines, once the items have been deployed. DevOps is powerful tool and I've only showed a little bit of what it can do in terms of deploying to Fabric.

Fabric CI/CD has streamlined the process for deployments, and I will never go back to using Fabric Deployment Pipelines'. What has been fun is the use of AI to generate the yml, and some of the code for the python script. It has been a great accelerator in getting up and running with DevOps.