Manage API Profiles Wizards

Overview

Most endpoints that return a list of entities will need to have pagination. Without pagination, a query could return millions or even billions more rows of data than anticipated, causing unnecessary network traffic.

Paging requires an implied ordering. By default, this may be the item’s unique identifier but can be other ordered fields, such as a created date.

Below are the different types of API pagination strategies available in Matillion ETL:

1. Full Path
2. Relative Path
3. Link Header
4. Offset-based
5. Cursor-based
6. Page-based

The strategies listed above can be enabled and managed in the Response Configuration dialog of the Manage Extract Profiles and Manage Query Profiles dialogs, respectively. The methods for adding pagination in these wizards are identical.

This page will provide information about how to configure pagination and strategies for the Manage Extract Profiles and Manage Query Profiles wizards, referred to in Configuring the Response Configuration, as well as listing all of the strategies and properties in the respective wizards.

Configuring the Response Configuration

Refer to Manage Extract Profiles and Manage Query Profiles to access the respective wizards in your Matillion ETL instance.

The Response Configuration dialog in each of the wizards will display a Select Fields section on the left, where you can view the structure of the data returned from the API call, and attempt to identify the repeat element from that data. Users can also specify this manually.

To populate the Repeating Element field you must right-click on the appropriate element in the tree structure, and select Set Repeating Element, or select Set [Param Name], as appropriate. These are not free text fields.

To remove the value from the Repeating Elements field, right-click in the aforementioned field, and click Remove.

Optionally, users can configure paging by first toggling the Paging button, situated on the right of the dialog, to ENABLED, and then selecting a paging strategy from the drop-down menu provided. Users should complete the fields related to the selected paging strategy. Refer to Paging strategy properties for more information.

Choose the appropriate method to populate these fields. If the tree view selection is required, right-click on an element situated to the left, and click on the name of the corresponding paging field. For more information, refer to Paging strategy properties.

To remove the value from the paging field, right-click in the aforementioned field, and click Remove.

Once you have configured all of the steps in both wizards, read Manage Extract Profiles and Manage Query Profiles for more information.

The following examples show the outcome for each type of API profile, using Full Path pagination:

• Endpoint configuration for an API Extract profile.



• Endpoint response in the RSD file of an API Query profile.

Paging strategy properties

The table below describes all of the pagination strategies, their features, and the implementation methods for the Manage API Profiles wizards. Certain properties are unique to each of the API Profiles wizards.