Data Platforms – The New Operating Systems For AI

@mrpaulandrew
Jul 2
3 min read

Translating Hype To Support the Industry & Community Thinking

Hi, I'm Paul and I write blogs to help process the thoughts in my head... :-)

AKA, the musings (post series link) of a slightly grumpy, battle hardened data engineer, technology strategist and enterprise architect.

Hey friends! Its been a while since I wrote a blog, so apologies in advance if I’m a little rusty. I can however assure that there is some coherent thought still included. Not just waffle for the sake of waffle. Not AI noise! For further clarity and for information, the following is output of all my own brain power. Although happy to accept it is thinking that has grown from several ideas that in the beginning came from an equal amount of human brain power and agent collaboration. Informed by, definitely not taken and repeated.

That said, the basis for this blog, like lots of talks and training I have delivered as a technical lead, is evolution. Specifically, the next wave of evolution for our data platforms. Having spent many years designing and building them, what next is a common stance in the technology industry. Where automation and abstraction remain king or queen (as you prefer).

Informing this evolution, below could be our starting cloud data platform, still applicable today. How batch vs stream data is handling here we could debate but not the focus of this blog.

Data platform reference architecture starting point

Please also consider with the above that if you ask 20 architects to draw a data platform you will get 20 different diagrams! Product icons optional.

To think about this evolution, we of course need to include a healthy dose of artificial intelligence (AI). There is no doubt that AI has become an excellent productivity tool in the right hands. Just like power tools in the hands of a carpenter that previously had to do everything manually. That said, beyond obvious productivity gains, AI use cases naturally falling out of existing or even new data analytics solutions still feels like an aspiration rather than an easy addition to the technical capabilities delivered. Why is that? Data quality, metadata, data models, governance all assumed as contributors at this point. But not the real reason.

Based on what I know and have experienced to date the fundamental problem and answer the why, is exactly as I approached it above. Trying to deliver AI use cases on top of our current data solution design thinking. AI outputs should not simply be added and planned for in the solution to enhance the consumer's experience. For a specific (frustrating) example from the Microsoft product stack, deploying a so called ‘Data Agent’ and giving in the context of some ‘Lakehouse’ tables to field a handful of business user questions. Helpful, but in my opinion, this is simply short sighted as a goal.

Instead, we need to think bigger. Think about a conceptual shift to realise this wave of AI powered value from our data solutions.

Not just a data platform that delivers AI capabilities and outputs. A data platform that forms the foundations for our AI capabilities. Maybe better described as a complete agentic framework of reasoning engines, plans, skills, and tools. The AI architecture sitting on top of our data architecture. On top, not added at the end.

These are all AI capabilities that must now function on a technology plane above our data platforms. To be clear, not to replace the data platform, but to harness it. In the same way programs/applications are hosted and harness the capabilities of an operating system. In turn the operating system harnesses the capabilities of the firmware, then the hardware and so on.

In essence, or the TLDR, we need to think about our data platforms as the new operating systems for our AI capabilities.

Building on the picture above, this is what I settled on as an architecture diagram. Our new data and AI platform reference architecture, considering the data layer as the facilitator (operating system) to the agentic layer.

Not surprisingly, it took a lot of debate internally and several iterations of this to arrive at a diagram we thought represented this evolution.

For context, here are the others I drew, slide show style.

What do you think? We are an amazing community and I would greatly value your feedback as we grow together in the technology industry. In the next blog, we can walk through the version and design thinking behind each of these diagram and a continuation of the coherent positioning that got us here.

Many thanks for reading.