Matillion ETL API - v0

Overview


Note: v1 of the Matillion ETL API is now available. Details can be found here

The Matillion ETL API is designed to make it easy to interact with Matillion ETL programmatically. It is primarily designed to support DevOps use cases of Matillion ETL such as Managing transitioning between Dev/Test/Live environment using a third party SCM (source control management) system.

The API uses a standard RESTful approach.

Please see here for Matillion ETL API examples, such as using Matillion ETL with a third party Source Control Management System.

Also see Loading Task Informationand the API Query component.

Note: If using a Bash or Python component to call the API on that same instance, use the instance's Private IP or local host (127.0.0.1:8080 for HTTP or 127.0.0.1:8443 for HTTPS) for the instance address.

 

Base URL

The REST API is available using the following URL pattern.

http(s)://<Matillion Instance URL>/rest/<Version><Endpoint>

Where :

  • Matillion ETL Instance URL is the same as the URL you would normally access your instance on. 
  • Version is the API version (Not versions created in the tool). Available options are v0 (discussed mainly within this article) and v1.
  • Endpoint is the part of the API used for this call. See below for the available endpoints.

Parameters can then be added to the end of the URL in order to refine the API call, such as to narrow the returned data by specifying particular groups or projects. There should always be a '?' after the endpoint to mark the start of a parameter section.

For example, checking projects (endpoint) in a certain group (parameter):

http://matillion.server.com/rest/v0/projects?groupName=TestGroup

Authentication

Authentication is done via basic HTTP authentication (unless you disable authentication in which case none is required).

Endpoint 1: /projects
Method Purpose Parameters
GET Get the structure of the projects. Groups, projects, environments, variables, versions, jobs - all with id, name and (if applicable) description When export=true, include everything from the specified version. export:
  • false (default)  - Returns a list of Groups, Projects,  Jobs, Variables and Environments with names, ID's and descriptions but not Job detail.
  • true includes all Job detail. if true groupName, projectName and versionName must be set.
groupName
  • The exact case sensitive name of the group of the project to export.
projectName
  • The exact case sensitive name of the project to export.
versionName
  • The exact case sensitive name of the version to export within the project. For the default version specify default.
DELETE Delete project. This is distructive be careful with this! groupName
  • The exact case sensitive name of the group of the project to delete from Matillion ETL.
projectName
  • The exact case sensitive name of the project to delete from Matillion ETL.
POST Insert an export obtained by GET back into a Matillion ETL instance. This will not overwrite. Simply post the JSON file back as binary data.
OPTIONS Generate the WADL for this resource

 


Endpoint 2: /tasks
 
Method Purpose Parameters
GET Get all currently executing tasks and their statuses. groupName
  • The exact case sensitive name of the group of the project for which to list tasks.
projectName
  • The exact case sensitive name of the project for which to list tasks.
since
  • The date from which jobs are taken. Requires a string-format date (yyyy-MM-dd). Requires that running=false.
running
  • false - Returns only jobs that are not currently-running.
  • true (default)  Returns only currently-running jobs.
POST Creates a new task and returns the task ID. groupName
  • The exact case sensitive name of the group of the project where the job to launch exists.
projectName
  • The exact case sensitive name of the project where the job to launch exists.
versionName
  • The exact case sensitive name of the version where the job to launch exists. For the default version specify default.
jobName
  • The exact case sensitive name of the orchestration job to run
environmentName
  • The exact case sensitive name of the environment in the project to run the job against
variables
  • Variables to pass to the job in the form of URL encoded variableName1=value1,variableName2=value2
    See the "Variables" subsection below for more information.
GET /tasks/{id} Gets detailed information of a task with the specified ID  
OPTIONS Generate the WADL for this resource.

 

 
Variables
 
Variables can be set when posting a task via the API such that the task run will use those variables in place of its own defaults. For example, if a Transformation job uses a Calculator component that uses values stored in variables as part of its calculation, the default values for those variables will be used unless otherwise stated. However by passing variables (of the same name as ones used in the Calculator component) into the task through the API, your Transformation job will use these values instead.

Variables are set by suffixing the '&variables' keyword, followed by setting the variable name, followed by setting the variable value. For example, setting the variable 'increment' to '5':
curl -X POST "http://xx.xx.x.xxx/rest/v0/tasks?groupName=Matillion&projectName=APIShowcase&versionName=default&environmentName=Example3&jobName=APIOrch&variables=increment=5"
For more API examples, see the API Examples documentation.
 

API Profiles

An API Profile defines a method for interpreting API and XML based APIs, allowing its data to be mapped to rows and columns in a table. A list of API Profiles currently utilized by a given Matillion ETL instance can be found through Project → Manage API Profiles.

Detailed information on writing API profiles can be found through the API Profiles Support documentation.
 

Enabling API for users

Before using Matillion ETL API services such as the basic API Query component, API must be enabled for the user in question. To do this, an administrator must check the API box found by the user's name in the 'User Configuration' section of the Admin Menu.


See here for API Examples.