top of page
Insights on Data, Platforms and AI
Practical insight on data, platforms and AI from people who deliver them.
The Cloud Formations blog focuses on what actually works in modern data programmes. We share lessons from real delivery, platform design decisions, and applied AI use cases, grounded in hands-on experience across strategy and execution.
Expect clear perspectives, technical depth where it matters, and content designed to help you make better decisions, not just follow trends.
Subscribe to to get updates on new posts.
Fabric User Data Functions
Fabric User Data Functions are a bit of a strange hybrid. They can be used like Azure Functions, to call some code outside and act like a serverless compute function, but also can extend notebooks as a source of reusable data logic. I've been using them since preview, and now they support Azure Key Vault (AKV) and Fabric Variable Libraries, I think they are ready (mostly) for productionable use. What are they all about So in my data solutions and processes, there tends to be
Jon Lunn
21 hours ago
The Modern Data Engineer
We've been chatting internally about the future of what we do, and Paul outlined in his post, how AI is going to be used across processes and platforms. Check it out here Data Platforms – The New Operating Systems For AI This post is a bit more focused on how to use it in my, (and your) role as a data engineer. What has changed, what's good and what's bad. Back in the day A lot has changed over the years of my career, a long time ago I was just writing SQL queries for Access
Jon Lunn
Jul 13
Fabric - Lakehouse Table Health Check
Microsoft have rolled out a new 'sys' stored procedure called 'sys.sp_get_table_health_metrics' for checking the health of your tables in your Lakehouse. And it outputs a lot of metrics looking at the state of your table and anomaly detection in them. What does it check for? Well it checks to see if your Lakehouse Delta tables are physically well-optimised for SQL query performance. It returns file-level health metrics for a specified Lakehouse table, including file-size dist
Jon Lunn
Jul 8
Data Platforms – The New Operating Systems For AI
Instead, we need to think bigger. Think about a conceptual shift to realise this wave of AI powered value from our data solutions. Not just a data platform that delivers AI capabilities and outputs. A data platform that forms the foundations for our AI capabilities. Maybe better described as a complete agentic framework of reasoning engines, plans, skills, and tools. The AI architecture sitting on top of our data architecture. On top, not added at the end.
@mrpaulandrew
Jul 2
Materialized Lake View - Update!
A few months ago, I did a post on Materialized Lake Views, and said that it wasn't quite ready to fit in your normal data pipelines. You could only schedule the refresh with its own in built scheduler or hit the API to kick of a refresh. I had a lovely method to do an API refresh using an Fabric User Data Function, but the API didn't support a Service Principal or Workspace Identity, so I didn't end up using it sadly. However, as part of the June Fabric update, we now have in
Jon Lunn
Jun 16
Fabric Runtime 2.0 - Identity Columns
Check it out on youtube as well Fabric Runtime 2.0 is in public preview, and there is one small change for this that makes my life as a data engineer a little bit easier. Identity columns. I miss the good old T-SQL Identity column "Id INT IDENTITY(1,1)", now we have something like it in Spark/Delta. So with the the old spark/delta table runtime, you couldn't have an default, automatically increasing column. But now we have these updates. Apache Spark 4.1 Delta Lake: 4.1 One o
Jon Lunn
Jun 9
Fabric Capacities - DEV/TST/PROD
If you like watch this on the CF YouTube Channel You know you can have more that one capacity? Most of the clients I've interacted with, even since the Power BI capacity days, they have just purchased one big old capacity, and assigned it to every workspace they needed. There have been a few clients that have had multi-region capacities, spun up across the globe for thing likes, billing to specific cost centres and regions and data ownership and sovereignty issues, but for th
Jon Lunn
Mar 16
How Is The AI Hype Translating Into Real Business Value? + Q&A
CF.Webinar - Free to attend knowledge sharing. Artificial intelligence has had another explosive year- more tools, more experimentation, more hype. But how much of that excitement has actually translated into measurable business value and what can leaders expect as we head into 2026? To answer these questions, Cloud Formations surveyed analysts, engineers, data product managers, and AI practitioners working hands-on with data and emerging AI capabilities every day. Their insi
Christine Andrew
Mar 11
Unlocking the Value of AI for Your Business Series: Ep 5 - Why AI Governance is an enabler, not a blocker
With the final episode, we are going to focus on how we safely utilise AI through AI governance without making it a blocker to advancement.
Branden Millward
Mar 2
Unlocking the Value of AI for Your Business Series: Ep 4 - AI Agents and the Rise of Agentic AI
The next iteration of AI is moving away from hard coded models and towards AI Agents and how they interact and intergrate with humans.
Branden Millward
Feb 27
Turn insight into action
If something you have read resonates, let’s talk about what it could mean for your data platform or roadmap.
bottom of page