Reading and Writing Secrets in Azure Key Vault Using Microsoft Fabric Notebooks

Sections

• Introduction
• Creating App Registration
• Storing Secret in Azure Key Vault
• Microsoft Fabric Notebook and Key Vault Interaction
• Conclusion

Introduction

Microsoft Fabric Notebooks provide users with a flexible tool that can be valuable for many business problems and processes. With its diverse language selection and freedom of writing their code, users can apply logic to accomplish their tasks without many constraints. In many cases, users tend to deal with various systems which in most cases require some form of authentication. 

Since we will be talking about Microsoft services, Azure Key Vault is a great service for storing artifacts that contain sensitive information for authentication purposes.

In this walkthrough, I’ll go through two methods on how to read and write secrets from within a notebook using the API as well as Fabric MSSPARKUTILS package. 

The Azure Key Vault API has been around for a while and is a very stable way of communicating with Azure Key Vault and is currently the best way to do so. 

MSSPARKUTILS for Fabric makes the process significantly easier, BUT it currently does not have all the necessary functionality to accommodate writing secrets to Key Vault. Although the package lacks those capabilities, I believe it will soon have them as the Azure Synapse version of the package contains those abilities, so it will be just a matter of when and not if.

The walkthrough was written with the following user experience assumptions:

1. Creating App Registrations within Microsoft Entra ID
2. Creating Resource Groups as well as resources within (Azure Key Vault)
3. Comfortable with creating and using Microsoft Fabric Notebooks as well as Python/PySpark

There is a wide range of content available online regarding these three points if one is unfamiliar to you.

• App Registrations
• MSSparkUtils for Fabric

Creating App Registration

An App Registration allows us to create a service principle that we can use to authenticate into Azure Key Vault using a token. This method removes any need for using user credentials making the process very secure.

Within Azure, navigate to Microsoft Entra ID, and select App Registrations. Create an App Registration with the following Settings:

After successfully registering navigate to Certificates & Secrets and create a new client secret. This value will be used for authentication purposes later on, make sure to keep a copy as it will become hidden once you leave this page.

After copying the secret value, your app registration Certificates & Secret page should look like this:

While we are still on the App Registration Page, navigate to Overview and copy the Application (client) ID as it will be needed for an API call.

Storing Secret in Azure Key Vault

With the secret we have created for our App Registration, we can go ahead and add it to a Key Vault.

We will start by creating a Resource Group:

Within the newly created resource group, we will create the Azure Key Vault that we will be using for storing our App Registration Secret. Make sure the correct resource group is selected and provide a name for your Key Vault. No other additional options will be required:

Azure Key Vaults created currently follow the Azure role-based access control permission model. This will require the addition of ourselves and the App Registration we created earlier to be added to the Key Vault Secrets Officer role. Within the Key Vault, navigate to Access Control (IAM) and select Add role assignment. 

Make sure Key Vault Secrets Officer role is selected in the Role section of the assignment. 

In the next step, we will add ourselves as well as the service principal that is associated with our App Registration. Once the appropriate members have been selected, click on review + assign.

Now, when viewing the Role Assignments for the Key Vault, we can see that both the App Registration service principal and we have implicit access to the secrets:

We can now go ahead and add our secret to the Key Vault. The value will be the secret value we established during the creation of the App Registration:

Once created, you will be able to see the Secret in the list of secrets:

Following the above steps, add three additional secrets:

• Microsoft-Fabric-Key-Vault-ID: This is the App Registration Application (client) ID
• Microsoft-Fabric-Tenant-ID: This is the Tenant ID of the tenant that contains your subscription/resource group and Key Vault
• Test-Key-Vault-Secret: This is the secret that we will use when we run the API call to change its value, feel free to input whichever value you would like currently.

After adding the additional three secrets, we should now have four total:

Microsoft Fabric Notebook and Key Vault Interaction

Now that we have the required items set up. We can go ahead and use Microsoft Fabric Notebooks to access the Key Vault Secret as well as update it with a new value.

Within My Workspace, go ahead and create a new Notebook.

To Retrieve a secret within a notebook is very easy when using the MSSparkUtils for Fabric. This package is already built-in so there is nothing that we have to import.

For these particular commands, as long as the user running them has appropriate access to the Key Vault, they will be able to retrieve them.

# App Registration Application (client) ID 

# The base function is as follows: mssparkutils.credentials.getSecret('azure key vault name','secret name') 

# Please change the below values to your Key Vault URI and Secret Names (if different) 

MICROSOFT_FABRIC_KEY_VAULT_ID = mssparkutils.credentials.getSecret('https://fabricnotebooks-kv.vault.azure.net/','Microsoft-Fabric-Key-Vault-ID') 

# App Registration Secret 

MICROSOFT_FABRIC_KEY_VAULT_TOKEN = mssparkutils.credentials.getSecret('https://fabricnotebooks-kv.vault.azure.net/','Microsoft-Fabric-Key-Vault-Token') 

# Azure Tenant ID 

MICROSOFT_FABIRC_TENANT_ID = mssparkutils.credentials.getSecret('https://fabricnotebooks-kv.vault.azure.net/','Microsoft-Fabric-Tenant-ID') 

If you try to print these variables, they will print as [REDACTED], but rest assured that the value is correct, assuming the Key Vault contains the correct value. You can check the logs of the command and confirm that the secrets were retrieved: 

Currently, the MSSparkUtils package for Fabric does not contain a command that will write a secret value to a Key Vault, therefore we will have to resort to using the API. The command .putSecret is available in the MSSparkUtils package for Azure Synapse, so I foresee that it will be available in Fabric soon.

The first step we have to take is to get a bearer token using our App Registration which will then be utilized during the API calls against our Azure Key Vault. To accomplish this, we can run the following API call:

import requests

# Url to obtain bearer token
uri = 'https://login.microsoftonline.com/'+ MICROSOFT_FABIRC_TENANT_ID +'/oauth2/v2.0/token'

# Required parameters to properly obtain the bearer token
data = {'client_id':MICROSOFT_FABRIC_KEY_VAULT_ID,
        'grant_type':'client_credentials',
        'client_secret':MICROSOFT_FABRIC_KEY_VAULT_TOKEN,
        'scope':'https://vault.azure.net/.default'}

# API call to get bearer token
response = requests.post(uri, data = data)
response = response.json()

# Set variables from api call to access_token and token_type
ms_token_type = response.get('token_type', None)
ms_access_token = response.get('access_token', None)

Now that we have the bearer token, we can go ahead and create a variable that we will use when passing in a new secret value:

# Variable for new secret value to be used in API call
NEW_KEY_VAULT_SECRET = 'New Secret Value1'

We can now go ahead and make the API call against our Key Vault and write the value of NEW_KEY_VAULT_SECRET to update Test-Key-Vault-Secret:

# URI pointed directly at the required secret
uri = 'https://fabricnotebooks-kv.vault.azure.net//secrets/Test-Key-Vault-Secret?api-version=7.2'

# Set headers
_headers ={'Authorization': ms_token_type+ ' ' +ms_access_token,
           'Content-Type': 'application/json'}
# Set body containing new value for refresh token
_body = {
  'value' : NEW_KEY_VAULT_SECRET
}
# Make the API call to set a new value for a secret 
APIcall = requests.put(uri, headers = _headers, json = _body)

Now, when we go to our Key Vault and access Test-Key-Vault-Secret, we can see that the secret value has changed and there is a newer CURRENT VERSION.

You can use this API call to not only update an existing secret, but also create it with a pre-defined value if it does not yet exist.  

Conclusion

In most cases, you might come across situations where you’re only required to read in a secret to run a certain process. Some systems and APIs return a new refresh token which is to be used for a future API call, this is when you would want to update your secret value programmatically.

Hopefully the MSSparkUtils package is updated in the future which will contain the .getSecret functionality, which will make the above API calls irrelevant as using the package will make the process significantly easier for processes within a Fabric notebook.