Is there a way to pass the table output results to the outer job without using a Global variable? I'm worried there could be conflicts if I use a global variable.
The Transformation has a Row Count variable available to export but that doesn't appear to do anything.
If I write to a local variable 'row_count' that then isn't available to the outer job obviously.
Is Global Variable the only option? High chance another job will overwrite the variable before my IF statment runs in the main job. And Don't want to create a million variables for every table write.
2 Community Answers
Ian Funnell —
Global variables are the way to go, yes. You’ll need to do the export from the component which is doing the table output.
Even though we confusingly called them “global”, Matillion’s global variables actually behave much more like private object member fields. It’s quite possible for multiple jobs to be running concurrently, each with its own opinion on the value of all the variables.
When a job starts running by any method (scheduler, interactive, or from SQS), the following happens:
It takes a copy of all the variables and initializes them to their default values.
The job can update the values while it’s running, but it only affects that job’s copy
If another job is currently running, and has updated a variable, a newly-started job will still start off with the variable’s default value. The new job can’t see any other running job’s environment variables.
Matillion’s “local” variables are a special category, for exclusive use with iterator components in concurrent mode.