From ProcFwk to CF.Cumulus: A Story of Metadata, Momentum, and Meaning

@mrpaulandrew
Jun 11
3 min read

In January 2020, I sat down with a simple idea: could I build a metadata-driven orchestration framework for Azure Data Factory pipelines that was scalable, flexible and reusable? That idea became ProcFwk, a side project and open-source project born out of necessity, curiosity, and a desire to simplify the orchestration of complex data workloads.

I never imagined that four and a half years later, that same framework would evolve into CF.Cumulus, a fully-fledged, open-source cloud data platform product now available on the Microsoft Azure Marketplace.

The Early Days: Metadata as a Superpower

ProcFwk was built on a simple principle: let metadata drive the orchestration. Instead of hardcoding pipeline logic, using an Azure SQL Database as the metadata layer and Azure Functions App to dynamically control execution. It was lightweight, adaptable, and surprisingly powerful when scaled to many concurrent pipelines. The community response was humbling, with dozens of repository forks, production deployments, and feedback from engineers around the world. Including Microsoft's product team.

But as time passed, the tech stack aged, .NET 3.1 Core was deprecated. Azure Data Factory NuGet packages fell behind. And as I focused on building Cloud Formations as a company founded with Dan, ProcFwk quietly waited for its next chapter.

The Rebirth: CF.Cumulus.Control (Momentum)

Over the 2023 holiday break, I finally gave the framework the attention it deserved. We forked the original repository, upgraded the Azure Functions to .NET 8, refactored authentication to use Azure Managed Identities, and stripped out legacy schema code. We even laid the groundwork for Microsoft Fabric pipeline support. But this wasn’t just a technical refresh. It was a reimagining.

ProcFwk became CF.Cumulus.Control, the orchestration engine of a broader vision: a metadata-driven Lakehouse accelerator that could ingest, transform, and govern data at scale.

The Cloud Shift Toolkit: A Bigger Vision (Meaning)

CF.Cumulus is now part of our Cloud Shift Toolkit, alongside:

CF.Nimbus™ - AI-powered legacy code interrogation.
CF.Stratus™ - Platform assurance and operational excellence.

Together, these tools form a cohesive ecosystem designed to accelerate time-to-insight for businesses navigating data transformation. Whether you're modernising legacy systems or building greenfield data platforms, our toolkit provides a structured, open, and extensible foundation.

What CF.Cumulus Is Today

CF.Cumulus is no longer just an orchestration framework. It’s a metadata driven Lakehouse platform that supports:

Declarative ingestion and transformation
Scalable compute with Databricks, Synapse, and soon Microsoft Fabric
Sensitive data handling via a dedicated Govern plugin
Full lifecycle support from dev to production
Open-source transparency with enterprise-grade reliability

It’s been used by multinational organisations across many different Azure regions and it’s still open-source. Mainly because we as a company believe in community, collaboration, and building on open standards.

Available on Azure Marketplace

The latest milestone for us that inspired this blog. CF.Cumulus is now live on the Azure Marketplace. This means any organisation can deploy the framework directly into their Azure environment, backed by our training, support, and roadmap. Or, use the community version to roll your own instance.

It’s truly a proud moment for me. Not just because of the technology, but because of the journey. From a blog post in 2020 to a product that’s helping businesses unlock value from their data faster than ever before. Accelerating time to value.

CF.Cumulus: A Story of Metadata, Momentum, and Meaning

If you’re looking for a way to simplify your data platform delivery, reduce time-to-value, and embrace open standards, CF.Cumulus might just be the product you need. And if you’ve been part of this journey by using my old ProcFwk, contributing feedback, ideas and pull requests, or just cheering from the side lines, thank you my friends!

Also I want to take a moment to recognise the amazing team at Cloud Formations that I've been lucky enough to build around me. What started as a solo open-source project has grown into a product that’s now shaped, supported, and delivered by a group of passionate, talented engineers. Their dedication, creativity, and belief in the vision have been instrumental in turning CF.Cumulus into what it is today. I’m incredibly proud to work alongside them as peers.

To clarify, this is far from the end of the story, we’re still building. Still learning. Still open-source. And still committed to helping organisations unlock the full potential of their data. As you may have heard me say, the problem of getting insight from data has not changed. Just the tooling and technologies, CF.Cumulus we continue to solve this problem and evolve with the industry, embracing new techniques, embracing hype, becoming cloud agnostic, maybe one day!!

To be continued.