Roles & Permissions (Azure)
In order for Matillion ETL to detect Azure Blob Storage containers, additional credentials may be required. Matillion ETL can either use Instance Credentials or User Defined Credentials, the latter of which will require you to gather credentials from your Azure account and enter them into the Matillion ETL client.
Using Identities (Instance Credentials)
1. To use Instance Credentials, your Matillion ETL VM must already be set up. If you wish to use a User Identity (as opposed to a System Assigned Identity, which is unique to the VM) then you will need to search for the Managed Identities blade on the Azure Portal and set one up, if you have not already done so.
2. From the Azure Portal, browse to Virtual Machines and select the virtual machine containing your instance and select Identity from the menu.
- If you wish to use a System Assigned Identity, select that tab and set the Status to On. Make note of the Object ID.
- If you wish to use a User Assigned Identity, select User assigned and Add a User Identity of your choice. Make note of the User Identity name.
3. Now browse to Storage accounts on the Azure Portal and select the account(s) that contains Blob Storage that you wish for Matillion ETL to have access to.
4. Select Access control (IAM).
Choose the Role of Owner from the dropdown.
- For System Assigned Identities, set the Assign access to dropdown to Virtual Machine and select/search the VM that you turned on System Assigned Identity.
- For User Assigned Identities, set the Assign access to dropdown to Azure AD user, group, or application and select/search for the User Identity you assigned to your VM.
5. Click Save
6. Inside your Matillion ETL instance, ensure that your Azure Credentials are set to Instance Credentials.
Using Apps (User Defined Credentials)
Creating an App and Owning Storage Accounts To add Storage Accounts to Matillion ETL,we must first create an App. This requires a user with the 'Application administrator' directory role.
1. From the Azure Portal navigate to Azure Active Directory → App registrations and click New application registration.
2. Give this new App any name and Sign-On URL. Ensure that Web app / API is selected for the Application type.
3. Click Create.
4. Now return to the start of the Azure Portal and browse to Storage Accounts → Access control (IAM) → Add.
5. Choose Owner as the role and search for the name of your new App.
6. Select your App from the list and click Save. Now this app is an owner of the storage account. This can be repeated with the same app for any number of storage accounts.
7. If you haven't already, add a Blob Storage resource to this storage account.
8. Return to the storage account and browse to Overview → Blobs. To add a new container, click the + Container button, give it a name and click OK.
Now, when you import details from your App into Matillion ETL, your client will be able to discover those buckets the App has ownership of. To use this App in your Matillion ETL client, see the next section.
Gathering Azure credentials
For a Matillion ETL instance to take advantage of Azure resources, you are required to provide credentials in the form of a Tenant ID, which is unique to your Azure account, then a Client ID and Secret Key which are taken from a Registered App.
Tenant IDFrom the Azure Portal, browse to Azure Active Directory → Properties and take the Directory ID as your Tenant ID.
Browse from the Azure Portal to Azure Active Directory → App Registrations → Registered App and select an App that is associated with your desired Storage Accounts. Take the Application ID as your Client ID.
Secret KeyBrowse from the Azure Portal to Azure Active Directory → App Registrations → Registered App. Select the App associated with your desired Storage Accounts, then navigate to Settings → Keys and create a new key for your instance.
The above credentials can now be put into the Manage Credentials menu of your Matillion ETL client. Open Manage Credentials via the Project Menu and click the Azure tab. Credentials can be tested using the Test button.