The Modern Data Engineer

We've been chatting internally about the future of what we do, and Paul outlined in his post, how AI is going to be used across processes and platforms. Check it out here Data Platforms – The New Operating Systems For AI

This post is a bit more focused on how to use it in my, (and your) role as a data engineer. What has changed, what's good and what's bad.

Back in the day

A lot has changed over the years of my career, a long time ago I was just writing SQL queries for Access databases, then started on SQL Server 2000, and writing SQL queries and views for consumption by good old Crystal Reports. Then moved more in the Data Warehouse world, were my career took off. After that, it was stored procedures, and the joy of commit/rollbacks, snapshots, and error handling. Then building on those skills was source control. The things that have changed since then is more operational with DevOps/DataOps. The development process has changed a bit. I've shifted from SQL to Python, SSIS to ADF, MDX to DAX, Reporting Services to Power BI, from SQL Server to Azure and Fabric.

What hasn't changed is the basic approach to problems, what has changed is the technology, in that we have to be flexible. One of my early mentors said to me once:

Chess grandmasters don't play chess. Chess grandmasters play patterns of chess

I think that deserved the quote formatting! The basic pattern of what I'm doing in ETL is the same, but the syntax and technology is different now. The way I'm pumping data has moved from SSIS to ADF, but the main problem I'm trying to solve is the same. SQL or PySpark. There's no new approach to updating data in a 'Type 2' slowly changing dimension. We still do inserts, updates, merges etc, just the code and scale has changed for the most part. Somethings have become easier, real-time streaming for example, but we are mostly dealing with the same challenges, both in terms of the process, but also the way the customers data needs sorting . That process and pattern remains roughly the same.

I've been CI/CD'd

The role of data engineer has changed, morphed into new areas. Now I have to do:

• Data Engineering - Pipelines, transformations, data modelling, quality checks

Ok, no change there, I can do it in SQL Server, Synapse, Databricks and now Fabric.

But also have to do:

• DevOps - Source control, CI/CD, automated deployments, approvals

• Governance - Security, lineage, standards, documentation, cost control

And now I'm doing more of this:

• AI - Automation, intelligent monitoring, data quality insights, assisted development

Yes the AI buzz has made it to me. And to be fair I like it, mostly.

So my skill set has undergone continuous improvement and continuous development. Life is one huge ongoing pull request, bringing in changes to your soul. That is either metaphysical or melodramatic, you choose.

The A to the I

The new phrase 'Vibe Coding' has entered the chat. Vibe Coding has a negative connotation in most conversions and memes I've seen. One comment was:

Vibe Coding is done by project managers, typing in a list of requirements to an agent, then deploying the code

It seems Vibe Coding has some bad vibes around:

• Lack of understanding

• Hidden bugs, poor testing and debugging

• Lack of maintainability

• Security risks

I'm sure it can speed up development, but seems to lack some engineering discipline, mostly around testing, debugging and security.

We now tend to talk about AI accelerators, using AI to create code in the development process. This is now what I do. I've use AI to build yaml for DevOps deployment pipelines, PySpark notebooks, debug SQL & PySpark, build small python applications, Fabric User Data Functions etc, but there has been a condition that I know what's it is doing, and then adjusting it to fit into the data engineering pipelines. In using AI to create code, I'm reminded of the Russian proverb 'Trust, but verify'. We know it will work, but we need to add some accountability.

AI will output what you ask of it, if you don't ask for error handling, re-try's, debugging friendly steps, it normally doesn't output it for you.

Early on in using AI, I had an issue with a complicated bit of PySpark, and ChatGPT gave me a great output. It was a nice tidy, a streamlined work of art, but it wasn't debug friendly.

I normally like a nice set of steps so I can track down what is going on where, rather than have one bit of super-code that does all the transformations at once. It may make the process look inefficient by doing transformation steps broken down into stages, but the use of debugging and tracking down data issues makes up for it. You can step though the stages and see a better picture of what is being done where.

That's the lifesaver in the testing phase. I got into the process of deploying code, then constantly refeeding it back into ChatGPT to find out where the issue was. I stopped and reverted to the way I used to do, explicit stages, and quickly understood the code better and what it was doing. The AI accelerator was going in the wrong direction, but with a little bit of steering, it started to come together.

Basic workflow using AI

'Context is for kings' is the old saying, however it is also very important for prompts.

So I tend to state in my prompts:

• Fabric items used

• The language - PySpark & SQL

• Constraints - Delta tables, SQL Endpoints

• Logging setup - I have a default bit of code that is use for logging and error capture

Then you have all the items that you need next like:

• Parameters

• Data source

• Data target

• Data processing rules

Then for notebooks I normally outline the basic structure:

• Parameters

• Imports

• Configuration loading

• Reusable functions

• Source reading

• Cleaning/transformation logic

• Data quality checks (optional)

• Write to target

• Return output to pipeline if needed

Most of the time, this is not needed as the output is normally structured that way.

Once that is done it's time for validation! It's human in the loop stuff of checking syntax, does it match the code guidelines, performance, results and is it maintainable. The last one is important, always ask yourself the question 'In six months time, would I be able to figure out what is going on?'

Then if needed:

• Refactor

Then the next step is:

• Document

Markup in notebooks and comments in SQL are there for the future you, from the present you. Future you will always appreciate past you for it.

The Modern Data Engineer

So who am I and what do I do? So looking at the list of stuff I've done recently:

• Data solution builder

• Automation specialist

• Data modeller

• DevOps dude

• AI user and enabler

• Tea fuelled snack muncher

The last one is just me though.

So I build integrated data solutions, use DevOps to control and deploy changes, AI to accelerate development, improve monitoring and create smarter data products. Which sounds great, but what's changed since I started out in data engineering?

The technical skills have changed. PySpark, Delta table, Notebooks were not around a few years ago. Generative AI, wasn't around a few years ago, and is helping with the changing technical landscape, that always should be the goal, to help not to do. Much like search engines did, and replaced looking stuff up in a book, or the grey bearded/haired experts that knew the secret sauce recipe. I've been comfortable with SQL for years, Python, not so much, but using AI to build Python stuff has helped me learn Python quicker, as you can use AI to explain its thinking and why is it doing it this way!

For the non-technical skills in data engineering, these haven't changed as much. Communication, business understanding, collaboration, being an obsessive problem solver (Again that last one is just me). But what has evolved is the goal of data engineering.

I used to just move data, query data, and analyse data, but now that goal has shifted since I first fired up SSMS and read 'SAMS Learn SQL in 24 Hours'. Moving data is now the basics of what I do and it may seem at first that it is the core of data engineering. If you think that is all to data engineering, you've been looking at it all wrong.

Modern data engineers now have the goal of creating reliable, governed, reusable and intelligent data solutions that help the business make better decisions... using tools that help with reliable, governed, reusable development and release practices.

Anyone can move data, that's the chess part. Being the grand master is understanding the pattern, the process and the capability of what you do.

Hope that has help sort of figure out what we should be doing. Let's see what the future brings.

Footnote

If you like the Gilbert and Sullivan plays about duty (Fun fact, they are all about duty) I got AI to write me a ditty based on 'I am the very model of a modern Major General', but with a Fabric twist:

I Am the Very Model of a Modern Fabric Engineer

I am the very model of a modern Fabric engineer,

I've workspaces and capacities that everyone should volunteer.

I branch my code in DevOps, then I merge it with a pull request,

And pray my deployment pipeline thinks my latest commit is best.

I populate a Lakehouse with a medallion design in mind,

Where Bronze and Silver cleanse the data, Gold is polished, well refined.

My shortcuts span a dozen clouds, yet live within OneLake's domain,

Until someone moves the source and then I map them all again.

I build semantic models that are lightning fast in Direct Lake,

Though one refreshed dependency is all it really takes to break.

I certify my datasets so the business know which ones are right,

Then answer why "that other report" gives numbers quite a different sight.