Data Transfer Object
    • Dark
      Light

    Data Transfer Object

    • Dark
      Light

    Article Summary

    Data Transfer Component

    The Data Transfer component enables users to transfer files from a chosen source to a chosen target.

    This component can use a number of common network protocols to transfer data to a variety of sources. This component copies, not moves, the target file. Setting up this component requires selecting a source type and a target type. The component's other properties will change to reflect those choices.

    Currently supported data sources include: Azure Blob Storage, Box, Dropbox, FTP, Google Cloud Storage, HDFS, HTTP, HTTPS, Microsoft SharePoint, Amazon S3, SFTP, and Windows Fileshare.

    Currently supported targets: Azure Blob Storage, Google Cloud Storage, HDFS, Amazon S3, SFTP, and Windows Fileshare.

    Please Note

    • To ensure that instance credentials access is managed correctly at all times, we always advise that customers limit scopes (permissions) where applicable.
    • FTPS is not supported through this component. We recommend using SFTP instead, or installing a tool that supports FTPS and calling it from a Bash Script component. You could also do this using cURL, which is available as standard.
    • When reading from or writing from Windows Fileshare, the SMB2 protocol will be preferred. SMB1 is still supported.
    • When the Source Type property is set to FTP, HTTP, HTTPS, SFTP, or Windows Fileshare, users will need to set a Source URL property—this property provides a template URL, with placeholder values inside square brackets []. Please replace the placeholder values with the values of your actual source URL.
    • Below are listed the universal parameters for this component that are independent of Source and Target types. Then, property tables are given for source and target properties, grouped by their source and target Types.

    Properties

    Snowflake Properties

    PropertySettingDescription
    NameStringA human-readable name for the component.
    Source TypeSelectSelect the type of data source.
    Unpack ZIP FileYes/NoSelect if the source data is a ZIP file that you wish to unpack before being transferred.
    Target TypeSelectSelect the target type for the new file.
    Target Object NameStringThe filename of the new file.
    gzip DataYes/NoSelect if you wish to gzip the transferred data when it arrives at the target.

    Redshift Properties

    PropertySettingDescription
    NameStringA human-readable name for the component.
    Source TypeSelectSelect the type of data source.
    Unpack ZIP FileYes/NoSelect if the source data is a ZIP file that you wish to unpack before being transferred.
    Target TypeSelectSelect the target type for the new file.
    Target Object NameStringThe filename of the new file.
    gzip DataYes/NoSelect if you wish to gzip the transferred data when it arrives at the target.

    BigQuery Properties

    PropertySettingDescription
    NameStringA human-readable name for the component.
    Source TypeSelectSelect the type of data source.
    Unpack ZIP FileYes/NoSelect if the source data is a ZIP file that you wish to unpack before being transferred.
    Target TypeSelectSelect the target type for the new file.
    Target Object NameStringThe filename of the new file.
    gzip DataYes/NoSelect if you wish to gzip the transferred data when it arrives at the target.

    Synapse Properties

    PropertySettingDescription
    NameStringA human-readable name for the component.
    Source TypeSelectSelect the type of data source.
    Unpack ZIP FileYes/NoSelect if the source data is a ZIP file that you wish to unpack before being transferred.
    Target TypeSelectSelect the target type for the new file.
    Target Object NameStringThe filename of the new file.
    gzip DataYes/NoSelect if you wish to gzip the transferred data when it arrives at the target.

    Delta Lake Properties

    PropertySettingDescription
    NameStringA human-readable name for the component.
    Source TypeSelectSelect the type of data source.
    Unpack ZIP FileYes/NoSelect if the source data is a ZIP file that you wish to unpack before being transferred.
    Target TypeSelectSelect the target type for the new file.
    gzip DataYes/NoSelect if you wish to gzip the transferred data when it arrives at the target.
    Target Object NameStringThe filename of the new file.

    Source Properties

    SourcePropertySettingDescription
    Azure Blob StorageBlob LocationTextThe URL, including full path and file name, that points to the source file that exists on Azure Blob Storage.
    When a user enters a forward slash character / after a folder name, a validation of the file path is triggered. This works in the same manner as the Go button.
    BoxAuthenticationSelectSelect the OAuth entry. OAuth entries should be set up in advance and can be created by first clicking the Manage button. For help configuring an entry, read our Box Extract Authentication Guide.
    File IdStringThe ID of the Box file.
    DropboxAuthenticationSelectSelect the OAuth entry. OAuth entries should be set up in advance and can be created by first clicking the Manage button. For help configuring an entry, read our Dropbox Extract Authentication Guide.
    File TypeSelectSelect either File or Paper.
    PathURLThe path to the Dropbox file.
    File IDStringThe file ID
    Download As ZipSelectSelect whether or not to download as a zip file.
    Download FormatSelectSelect HTML or Markdown.
    FTPSet Home Directory as RootYes/NoYes: URLs are relative to the user's home directory.
    No: URLs are relative to the server root.
    Source URLTextThe URL, including full path and file name, that points to the source file. The source URL template includes placeholder values inside either square brackets [] or angled brackets < >. Replace the placeholder values with the actual values of your source URL.
    Source UsernameTextThis is your URL connection username. It is optional and will only be used if the data source requests it.
    Source PasswordTextThis is your URL connection password. It is optional and will only be used if the data source requests it.
    Google Cloud StorageSource URLTextThe URL, including full path and file name, that points to the source file.
    When a user enters a forward slash character / after a folder name, a validation of the file path is triggered. This works in the same manner as the Go button.
    HDFSSource URLTextThe URL, including full path and file name, that points to the source file.
    HTTPSource URLTextThe URL, including full path and file name, that points to the source file. The source URL template includes placeholder values inside either square brackets [] or angled brackets < >. Replace the placeholder values with the actual values of your source URL.
    Source UsernameTextThis is your URL connection username. It is optional and will only be used if the data source requests it.
    Source PasswordTextThis is your URL connection password. It is optional and will only be used if the data source requests it.
    HTTPSPerform Certificate ValidationChoiceCheck the SSL certificate for the host is valid before taking data.
    Source URLTextThe URL, including full path and file name, that points to the source file. The source URL template includes placeholder values inside either square brackets [] or angled brackets < >. Replace the placeholder values with the actual values of your source URL.
    Source UsernameTextThis is your URL connection username. It is optional and will only be used if the data source requests it.
    Source PasswordTextThis is your URL connection password. It is optional and will only be used if the data source requests it.
    Microsoft SharepointURLTextThe URL is the web address that you visit to sign into your SharePoint account.
    UserTextA valid SharePoint username to use for authentication.
    PasswordTextA valid SharePoint password. Users have the option to store their password inside the component; however, we highly recommend using the Password Manager feature instead.
    SharePoint EditionSelectSelect your edition of SharePoint.
    Connection OptionsParameter/ValueParameter: A JDBC parameter supported by the Database Driver. The available parameters are determined automatically from the driver, and may change from version to version. They are usually not required, since sensible defaults are assumed.
    Value: A value for the given parameter.
    File TypeSelectChoose whether the file type is a document or attachment.
    LibrarySelectThe location where you can upload, create, update, and collaborate on files. For more information, see here.
    File URLTextThe URL that points to the file.
    S3Source URLTextThe URL, including full path and file name, that points to the source file. The source URL template includes placeholder values inside either square brackets [] or angled brackets < >. Replace the placeholder values with the actual values of your source URL.
    When a user enters a forward slash character / after a folder name, a validation of the file path is triggered. This works in the same manner as the Go button.
    SFTPSet Home Directory as RootYes/NoYes: URLs are relative to the user's home directory.
    No: URLs are relative to the server root.
    Source URLTextThe URL, including full path and file name, that points to the source file. The source URL template includes placeholder values inside either square brackets [] or angled brackets < >. Replace the placeholder values with the actual values of your source URL.
    The Source URL must start with sftp:// or else an authentication failure message will be returned.
    Source UsernameTextThis is your URL connection username. It is optional and will only be used if the data source requests it.
    Source PasswordTextThis is your URL connection password. It is optional and will only be used if the data source requests it.
    Source SFTP KeyTextThis is your SFTP Private Key. It is optional and will only be used if the data source requests it.
    This must be the complete private key, beginning with "-----BEGIN RSA PRIVATE KEY-----" and conforming to the same structure as an RSA private key.
    Windows FileshareSource URLTextThe URL, including full path and file name, that points to the source file. The source URL template includes placeholder values inside either square brackets [] or angled brackets < >. Replace the placeholder values with the actual values of your source URL.
    Source DomainTextThe domain that the source file is located on.
    Source UsernameTextThis is your URL connection username. It is optional and will only be used if the data source requests it.
    Source PasswordTextThis is your URL connection password. It is optional and will only be used if the data source requests it.

    Target Properties

    SourcePropertySettingDescription
    Azure Blob StorageBlob LocationTextThe full URL that points to the target file location on Azure Blob Storage.
    Google Cloud StorageTarget URLTextThe URL (without file name) that points to where the new file will be created. The URL template includes placeholder values inside either square brackets [] or angled brackets < >. Replace the placeholder values with the actual values of your source URL.
    HDFSTarget URLTextThe URL (without file name) that points to where the new file will be created. The URL template includes placeholder values inside either square brackets [] or angled brackets < >. Replace the placeholder values with the actual values of your source URL.
    S3Target URLS3 TreeThe URL of the S3 bucket to get the files from. This follows the format s3://bucket-name/path
    Access Control List OptionsSelectChoose from ACL settings that Amazon provide. Leaving it empty doesn't change the current settings. A full list can be found here.
    EncryptionSelectDecide on how the files are to be encrypted inside the target S3 Bucket.
    None: No encryption.
    SSE Encryption: Encrypt the data according to a key stored on KMS. Read AWS Key Management Service (AWS KMS) to learn more.
    S3 Encryption: Encrypt the data according to a key stored on an S3 bucket. Read Using server-side encryption with Amazon S3-managed encryption keys (SSE-S3) to learn more.
    KMS Key IDSelectThe ID of the KMS encryption key you have chosen to use in the 'Encryption' property.
    SFTPSet Home Directory as RootYes/NoYes: URLs are relative to the user's home directory.
    No: URLs are relative to the server root.
    Target URLTextThe URL (without file name) that points to where the new file will be created. The URL template includes placeholder values inside either square brackets [] or angled brackets < >. Replace the placeholder values with the actual values of your source URL.
    The Target URL must start with sftp:// or else an authentication failure message will be returned.
    Target UsernameTextThis is your URL connection username. It is optional and will only be used if the data source requests it.
    Target PasswordTextThis is your URL connection password. It is optional and will only be used if the data source requests it.
    Target SFTP KeyTextThis is your SFTP Private Key. It is optional and will only be used if the data source requests it.
    This must be the complete private key, beginning with "-----BEGIN RSA PRIVATE KEY-----" and conforming to the same structure as an RSA private key.
    Windows FileshareTarget URLTextThe URL (without file name) that points to where the new file will be created. The URL template includes placeholder values inside either square brackets [] or angled brackets < >. Replace the placeholder values with the actual values of your source URL.
    Target DomainTextThe domain that the newly created file is to be located on.
    Target UsernameTextThis is your URL connection username. It is optional and will only be used if the data source requests it.
    Target PasswordTextThis is your URL connection password. It is optional and will only be used if the data source requests it.

    Copying Files to an Azure Premium Storage blob

    When copying files to an Azure Premium Storage blob, Matillion may provide the following error:

    Self-suppression not permitted.

    This is because, unlike standard Azure Storage, Azure Premium Storage does not support block blobs, append blobs, files, tables, or queues. Premium Storage supports only page blobs that are incrementally sized.

    A page blob is a collection of 512-byte pages that are optimised for random read and write operations. Thus, all writes must be 512-byte aligned and so any file that is not sized to a multiple of 512 will fail to write.

    For additional information about Azure Storage blobs, read Understanding block blobs, append blobs, and page blobs.


    What's Next