Fabric User Data Functions

Fabric User Data Functions are a bit of a strange hybrid. They can be used like Azure Functions, to call some code outside and act like a serverless compute function, but also can extend notebooks as a source of reusable data logic. I've been using them since preview, and now they support Azure Key Vault (AKV) and Fabric Variable Libraries, I think they are ready (mostly) for productionable use.

What are they all about

So in my data solutions and processes, there tends to be a lot of repeated logic, functions across notebooks and pipeline. Fabric User Data Functions (UDF's) are a way to centralise these bits of platform or business logic, with the goal of de-duplication code and making it easier to maintain.

I do normally have a 'Functions' notebook that can be called via my data processing notebooks, for generic functions and stuff like this, but now I can call these UDF's from pipelines.

They are all based on Python, unlike Azure Functions, which can use a variety of languages. Each UDF can contain one or more functions, so you want you can categorise these functions into specific uses. One for API calls, one for data cleaning etc. I have one currently checking if a pipeline is running, and some other pipeline operational checking.

The role in data engineering

UDF's can fit into a number of ways in the data engineering process:

• Configuration Lookup

• Process Control

• Data Quality Checks

• Monitoring and Logging

As much as you may want to shift out data processing tasks, I would only use them for light tasks. A heavy data processing task is not what they are suited for. If you call them from a notebook, your Spark cluster isn't running that process.

Area	Current limit
Function execution timeout	240 seconds
Request payload size	4 MB
Response size	30 MB
Published function Python version	Python 3.11
Invocation log retention	30 days
Private .whl library size	28.6 MB

Anything running longer than 240 seconds will time out and fail. So UDF's should be focused utility, look ups, config values, tiny amounts of data. No writing a shed load of data to a delta table.

When I first read about them, my main idea was for a generic logging/error solution that I can call from a notebook and also a pipeline. All I need to do is hit the UDF and it the data, the UDF will do all the rest. So lets look how this would be set up

Error Logging

For a basic example, I'd like to send details to my Fabric SQL Database, and a table called 'logging.ErrorLog' with the following columns:

• ErrorLogId - Basic identity column

• Source - VARCHAR - The source of the error, for example Notebook name

• MessagePayload - NVARCHAR - to store the JSON I'll send to it

• CreatedAt - DATETIME - Timestamp for the logging, I'll use a default to populated it

Python Decorator

In thre UDF, you'll need to use a decorator, which is a marker on how python should use a the function.

So for example, you have to use '@udf.function()' to declare it as a Fabric Data Function

def some_function_one():
	print("one")

@udf.function()
def some_function_two():
	print("two")

@udf.function()
def some_function_three():
	print("three")

So 'some_function_one' isn't callable from the outside the UDF. But 'some_function_two' and 'some_function_three' are useable ones and can be called, as they have '@udf.function()' to note they are a UDF. It is perfectly acceptable to do this mix of functions you can't call, so you can have generic functions used by declared 'udf' items. May be a connection setting or something reusable.

Creating the function

When you create a function, it adds some default code to it:

You can test and run it if you like, I'm going to delete it, and add my code:

import json
import fabric.functions as fn
from fabric.functions import FabricSqlConnection

udf = fn.UserDataFunctions()

@udf.connection(argName="sqldb", alias="dbmetadata")
@udf.function()
def log_error(source: str, message: dict, sql_db: FabricSqlConnection) -> str:
    if not source:
        raise ValueError("source is required")
    if message is None:
        raise ValueError("message is required")

    message_json = json.dumps(message, default=str)

    insert_sql = """
        INSERT INTO logging.ErrorLog
        (
            Source,
            MessagePayload
        )
        VALUES
        (
            ?,
            ?
        );
    """

    with sqldb.connect() as conn:
        with conn.cursor() as cursor:
            cursor.execute(insert_sql, source, message_json)
        conn.commit()

    return "Error logged"

As I'm hitting a Fabric SQL database, I can use the connections feature to connect to it, so I don't have to mess around with getting connection settings from Variable Libraries or Azure Key Vault.

You select from the list of items you can connect to then it's done. You can only edit the alias name!

In the code it's called here:

@udf.connection(argName="sqldb", alias="dbmetadata")

Note: UDF's don't seem to like '_'. When I created this code I used 'sql_db', but the editor didn't like it, and get throwing an error.

It will prompt you to publish it, once you have done that you can test it. So click on the '...' or the flask icon.

So you can add in values to your parameters and away you go.

Security

Connections has smoothed out the process, however both the owner/creator of the function needs to be able to access the object, in this case a database. In this case you don't need to worry about the access, if however you are calling an API, you may have to get items from Variable Libraries or Key Vault.

Calling them from a notebook

Calling the function from a notebook is straight forward, using the 'notebookutils' library:

# Get the UDF item
my_udf = notebookutils.udf.getFunctions("UDFErrorLog")

# Call a function inside that UDF item
result = my_udf.log_error(
    source="Silver Notebook",
    message={
        "error": "FileLoadDate mismatch",
        "severity": "High",
        "notebook": "nb_silver_load"
    }
)

print(result)

You call the UDF here:

my_udf = notebookutils.udf.getFunctions("UDFErrorLog")

And then the sub-function:

my_udf.log_error

Calling them from a pipeline

In pipelines there is an activity you can call, its called 'Functions' and you can call Fabric and Azure Functions. You can send it parameters like the notebook one. One point, it does require a connection being set up!

Issues

Reliability is an issue. I have to call a function that runs in a pipeline every 20mins from 7am to 7pm and quite frequently, about once or twice a week, I get an error which falls into two types:

• Cannot connect to the Variable Library

• Cannot get the token from the Entra endpoint

So that is about a 1-2% failure rate across the week, I don't find that acceptable. It's not a high priority pipeline, but is this is expected to be a dependable reusable function. I'll have to add in retries on the pipeline, but what about functions I call from notebooks? MS needs to ensure its a bit better in terms of the service, however I think platform wobbles in Fabric and Entra have increased of late.

In the real world

I've been using UDF's to hit Fabric API's to get the status of a job, error and status logging and they are mostly useful as a utility. For some items I've had in my 'Functions' and 'Utility' notebooks, I'm a bit hesitant to pull them out there into UDF's, as I want a single location to store and maintain them. But for some Fabric API items, I'll be moving them to UDF's.

What you can do is call across workspaces, in both Pipelines and Notebooks, you can access UDF's in other workspaces, so you could centralise some items that your users could use.

I've found them useful for some tasks, and as aways they are not a silver bullet for your data problems.