IAM Roles & Permissions (GCP)

GCP credentials are required for Matillion ETL instance to access various services such as discovering Cloud Storage buckets and using KMS. Appropriate permissions must be given via your GCP admin console and details of your GCP account must be entered into the Matillion ETL instance via Project → Manage Credentials where credentials for other platforms may also be entered.


Each Matillion ETL instance takes a single set of GCP credentials. If you are wanting to set up a new Matillion ETL instance to work with a new GCP Project, it is advised you follow the steps in GCP Account Setup for BigQuery and Storage and Launching Matillion ETL for BigQuery, in that order.

You can grant access to other GCP services using Access Control.

 

GCP & BigQuery Roles


When using Matillion ETL for GCP and BigQuery or even when using BigQuery components on other Matillion ETL for other platforms, it is required that the user has access to a GCP account with the following roles.

At the current time, Matillion ETL uses the admin BigQuery role:

roles/bigquery.admin

The admin BigQuery role includes the following roles:

Role Description
roles/bigquery.user Provides permissions to run jobs, including queries, within the project.
roles/bigquery.dataViewer

When applied to a dataset, dataViewer provides permissions to:

  • Read the dataset's metadata and to list tables in the dataset.
  • Read data and metadata from the dataset's tables.

When applied at the project or organization level, this role can also enumerate all datasets in the project. Additional roles, however, are necessary to allow the running of jobs.

roles/bigquery.dataEditor

When applied to a dataset, dataEditor provides permissions to:

  • Read the dataset's metadata and to list tables in the dataset.
  • Create, update, get, and delete the dataset's tables.

When applied at the project or organization level, this role can also create new datasets.

roles/bigquery.dataOwner

When applied to a dataset, dataOwner provides permissions to:

  • Read, update, and delete the dataset.
  • Create, update, get, and delete the dataset's tables.

When applied at the project or organization level, this role can also create new datasets.


Matillion ETL also requires the Storage admin role:

roles/storage.admin

The Storage admin role includes the following roles:

Role Description
roles/storage.objectCreator Allows users to create objects. Does not give permission to delete or overwrite objects.
roles/storage.objectViewer Grants access to view objects and their metadata, excluding ACLs.
roles/storage.objectAdmin Grants full control of objects.
 

Managing and Testing AWS Credentials

When using Matillion ETL the credentials are attached to your Environment definition.

To Manage Credentials:

  • From the Project menu choose Manage Credentials
  • To check if your Instance Credentials are set and working click Test in the Instance Credentials section
  • To add user defined Credentials:
    • Click the + button
    • Add a NameAccess Key ID and Secret Access Key then click Test.

Note: Its OK to get a warning at this point if you are not adding all access policies The output should indicate which policies you have set up successfully.



To add credentials to an environment:
  • Expand the Environment panel and choose the environment you wish to modify.
  • In the Credentials section choose either "Instance Credentials" (default) or the name of any user defined credentials you have created (in the example below "Manual Credentials")


This section lists the actions required in order to use the KMS option in the Manage Passwords dialog.
Note that in addition to the below, the chosen KMS key must be:
  • In the same Region as Matillion
  • Enabled
kms:ListAliases - Enables Matillion to populate the “Master Key” dropdown by listing all the KMS aliases which are associated with a Key.
kms:Encrypt - Enables Matillion to store an encrypted password.
kms:Decrypt - Enables Matillion to retrieve and use an encrypted password.

 

Managing and Testing GCP Credentials

Begin by launching your Matillion Instance and opting to create a project when prompted if you do not already have one.

The Create Project menu asks for a Project Group and Name, which can be anything. Default Project and Default Dataset, can be set to your Project ID and Dataset ID as noted earlier. The Environment name can be set to anything.

To enable a connection to the GCP services, you must enter your Service Account key into the Matillion Credentials Manager. To do this, select 'Manage' beside the 'GCP Credentials' field.  If you are not creating a new project, you can access the Credentials Manager through Project Manage → Credentials.

This new window will allow you to manage the credentials Matillion ETL will use to access platform-specific services. Ensure the tab is set to 'GCP' and click the + icon to add a new set of credentials. Name it whatever you wish, then select 'Browse...' and find the JSON file downloaded earlier from the GCP Service Account. This file is all that is required for Matillion to use your GCP project.

Clicking 'Test' should acknowledge a success for BigQuery and GoogleCloudStorage. Clicking 'OK' will take you back to the Create Project screen where you can select 'Test' again to ensure all details are correct. If either test should fail, double-check your entered information to ensure each is correct. If this still fails to produce a successful Test, it is recommended that you contact Matillion Support directly.

With a successful test, clicking 'OK' will create your project and you are now free to use Matillion and supported GCP features.