IAM Roles and Permissions (GCP)
    • Dark
      Light

    IAM Roles and Permissions (GCP)

    • Dark
      Light

    Article Summary

    Overview

    Google Cloud Platform (GCP) credentials are required for Matillion ETL instances to access various services such as discovering Cloud Storage buckets, PubSub, and KMS.

    Appropriate permissions must be given via your GCP admin console and details of your GCP account must be entered into the Matillion ETL instance via Project Manage Credentials where credentials for other platforms may also be entered.

    Important Information


    GCP & BigQuery Roles

    When using Matillion ETL for GCP and BigQuery or even when using BigQuery components on other Matillion ETL platforms, it is required that the user has access to a GCP account with the BigQuery roles.

    The required roles while creating a Service Account for GCP are:

    HeadingRole
    ProjectEditor
    Browser
    BigQueryBigQuery Admin
    BigQuery Data Editor
    BigQuery Data Owner
    BigQuery Data Viewer
    BigQuery User
    StorageStorage Admin
    Storage Object Admin
    Storage Object Creator
    Storage Object Viewer
    PubSubPubsub Admin
    Pubsub Editor
    Pubsub Publisher
    Pubsub Subscriber
    KMSkms ListAliases
    kms Encrypt
    kms Decrypt

    Matillion ETL uses admin BigQuery roles as shown below:

    roles/bigquery.admin

    The admin BigQuery role includes the following roles:

    RoleDescription
    roles/bigquery.userProvides permissions to run jobs, including queries, within the project.
    roles/bigquery.dataViewer

    When applied to a dataset, dataViewer provides permissions to:

    • Read the dataset's metadata and to list tables in the dataset.
    • Read data and metadata from the dataset's tables.

    When applied at the project or organization level, this role can also enumerate all datasets in the project. Additional roles, however, are necessary to allow the running of jobs.

    roles/bigquery.dataEditor

    When applied to a dataset, dataEditor provides permissions to:

    • Read the dataset's metadata and to list tables in the dataset.
    • Create, update, get, and delete the dataset's tables.

    When applied at the project or organization level, this role can also create new datasets.

    roles/bigquery.dataOwner

    When applied to a dataset, dataOwner provides permissions to:

    • Read, update, and delete the dataset.
    • Create, update, get, and delete the dataset's tables.

    When applied at the project or organization level, this role can also create new datasets.

    Matillion ETL requires the Storage admin role:

     roles/storage.admin

    The Storage admin role includes the following roles:

    RoleDescription
    roles/storage.objectCreatorAllows users to create objects. Does not give permission to delete or overwrite objects.
    roles/storage.objectViewerGrants access to view objects and their metadata, excluding ACLs.
    roles/storage.objectAdminGrants full control of objects.

    The PubSub includes the following roles in Matillion:

    roles/pubsub.admin
    RolesDescription
    roles/pubsub.adminFull access to the topics, subscriptions, and snapshots.
    roles/pubsub.editorModify topics and subscriptions, publish and consume messages.
    roles/pubsub.publisherPublish messages to a topic
    roles/pubsub.subscriberConsume messages from a subscription, attach subscriptions to a topic, and seek to a snapshot.

    The KMS includes the following roles in Matillion:

    RolesDescription
    kms:ListAliasesEnables Matillion to populate the "Master Key" dropdown by listing all the KMS aliases which are associated with a Key.
    kms:EncryptEnables Matillion to store an encrypted password.
    kms:DecryptEnables Matillion to retrieve and use an encrypted password.

    Managing and Testing GCP Credentials

    When using Matillion ETL the credentials are attached to your Environment definition.

    Manage Credentials

    1. In Matillion ETL, in the top left corner of the screen, click ProjectManage Credentials.

      Project dropdown menu

      Project dropdown menu

    2. Now, in the Manage Credentials window, if the instance credentials are available, you can Test them by using the Test button at the top of the screen.

    3. On the Manage Credentials window, new User Defined Credentials can be added by using the + button. Make sure to select the GCP tab in User Defined Credentials section.

      Manage Credentials Window

      Manage Credentials Window

    4. Next you enter the details required to create a new credential. Then, click Test.

      • Name – Enter the name for the user credential.
      • Service Account – Browse and select the appropriate service account, which you have created while setting up an account for GCP.
      Create GCP Credential

      Create GCP Credential

    5. If further information is needed for the Service Account, please read the GCP Account Setup for BigQuery and Storage guide.

    6. User defined credentials are then listed by name under the GCP tab. You select the User Credential you have created from the list, and click Test at the bottom of the manage credentials window.

      New created User Test

      New Created User Test

    Please Note

    • You can use 🖉 for editing or X icon for any deletion in the each entry listed. When creating or editing credentials, a Test button is made available in the new dialog to check the details before finalising your credentials.
    • This Test will check access to any services that Matillion ETL uses. You may continue even if the tests fail, however some parts of the product may be impaired or non-functional without appropriate credentials.
    • Different environments can use different credentials if required.

    Add Credentials to an Environment

    1. Expand the Environment panel and choose the environment you wish to modify. Right click on the environment and select Add Environment.

      Add Environment

      Add Environment

    2. Enter the details to create Environment and then, click Test.
      • Environment Name – Enter the environment you wish to create.
      • GCP Credentials – Select the GCP credential from the dropdown
      • Default project – Select the project from the dropdown.
      • Default Dataset – Select the dataset from the from the dropdown.

      Once all settings and testing done, click Finish.

      Create Environment

      Create Environment


    Testing GCP Credentials

    1. Begin by launching your Matillion Instance and select Create Project if you do not already have existing project in your instance.

    2. Browser will direct you to Create Project window. Enter the Project Details and, then click Next.

      • Project group – Select the Project group from the dropdown.
      • Project Name – Enter the project name.
      • Project Description – Provide a description for the project.
      Create project

      Create Project

    3. On the next page of Environment, enter details and click Test. Then, click Finish.

      • Environment name – Enter the name for the environment to create.
      • GCP Credentials – Select the GCP credentials, from the dropdown or click Manage to select.
      • Default project – Select the Default project from the dropdown.
      • Default Database – Select the default database name.
      Environment details

      Environment details

    4. Now the browser will take you to the new Project in Matillion instance, go to the Manage Credentials window by selecting Project menu,, select the newly created user credential and click Test, you should acknowledge a success for BigQuery, GoogleCloudStorage, PubSub, and KMS in the new project.

      Successful GCP Credential Test

      Successful GCP Credential Test