Loop Iterator

Loop Iterator Component

Loop over values of a simple sequence.

This component is a simple for loop. It enables you to run an attached component multiple times, each time with a unique value of an iteration variable. That variable can be referenced from the attached component.

To attach the iterator to another component, use the blue output connector and link to the desired component. To detach, right click on the attached component and select 'Disconnect from Iterator'.

If you need to iterate more than one component, put them into a separate orchestration or transformation job and use a Run Transformation or Run Orchestration component attached to the iterator. In this way, you can run an entire ETL flow multiple times, once for each value of the iteration variable.

Note: All iterator components are limited to 5000 iterations.

Properties


Property Setting Description
Name Text The descriptive name for the component.
Concurrency Select Sequential - Iterations are done in sequence, waiting for each to complete before starting the next. This is the default.
Concurrent - Iterations are run concurrently. This requires all "Variables to Iterate" to be defined as Copied variables, so that each iteration gets its own copy of the variable isolated from the same variable being used by other concurrent executions.
Note: The maximum concurrency is 8 times the number of processors on your cloud instance. For example: An instance with 2 processors has a maximum concurrency of 16.
Variable to Iterate Select Choose an existing variable to iterate. (You may need to define one first in the Project-->Edit Environment Variables screen.) Only numeric variables are available.
Starting Value Integer The starting value for the variable. Starting and ending values are both inclusive.
Increment Value Integer The increment value for the variable. This may be negative, providing the end value is smaller than the starting value.
End Value Integer The end value for the variable. Starting and ending values are both inclusive..
Break on Failure Select No: Attempt to run the attached component for each iteration, regardless of success or failure.
Yes: If the attached component does not run successfully, fail immediately.
Note: If a failure occurs during any iteration, the failure link is followed. This parameter controls whether it is followed immediately or after all iterations have been attempted.
Note: This is only available in Sequential mode. When running with concurrency, all iterations will be attempted.

Variable Exports

This component makes the following values available to export into variables:

Source Description
Iteration Attempted The number of iterations that this component attempts to reach (Max Iterations parameter).
Iteration Generated The number of iterations that have been initiated. Iterators terminate after failure so this number will be the successful iterations plus any potential failure.
Iteration Successful The number of iterations successfully performed. This is the max iteration number, minus failures and any unattempted iterations (since the component terminates after failure).

Example

In this example, we implement a simple Time-Series table as outlined in the Amazon Redshift Documentation.
In this example, we implement a simple Time-Series table as outlined in the Google BigQuery Documentation.

Create Table is run once for each year from 2010 to the current year, resolving the iteration variable into the table name parameter. This creates a whole set of new tables.

First we must define a variable to iterate, and provide a sensible default value. When running a job or a component without iteration, this default value is used.

The iteration is configured to start at 2010 and iterate until the current year using an expression. (The expression syntax is javascript, and is covered here)

The create table is set to to only create if the existing table does not exist. Therefore, a new table will usually be skipped until the start of a new year.

Nested iteration is also possible, so we could nest two Loop Iterator components together to create tables based on year and quarter. Here we iterate year in the same way, then iterate from 1-4 for the quarters.

Video