Using KMS encrypted passwords in Python

Overview

Typical uses of Python scripts in Matillion ETL often involve the inclusion of credentials. It is always ill-advised to store a password in plaintext in a component. Unfortunately, Matillion ETL does not allow Python scripts to take passwords directly from the Password Manager, as this would allow users to print stored passwords within the client.

The solution is for the user to decrypt their own passwords from within a Python Script.

Step-by-Step

1. Import the modules required. We'll be using boto3 and base64.

import boto3
import base64

2. The password or client secret you want to include in the Matillion ETL Python script should be encrypted beforehand.  This can be done using the AWS CLI Command below.

aws kms encrypt --key-id <keyid> --plaintext <clientsecret>

Where <keyid> is your KMS Key ID and <clientsecret> is the plaintext to be encrypted - in this case, your password or client secret.  More information on this AWS CLI Command can be found here. For example:

aws kms encrypt --key-id 1234abcd-12ab-34cd-56ef-1234567890ab --plaintext "myclientsecrettext"

3. Once encrypted, it is safe to store this credential inside your python script using a variable (here, kms_encrypted_secret).

kms_encrypted_secret = "MyEncryptedSecret"

4. Set a variable (here, client) to the AWS KMS client class.

client = boto3.client('kms')

5. Call AWS to decrypt (a method of the client class we just stored) the encrypted password or secret. This needs to be base64 encoded.

decrypted_key = client.decrypt(CiphertextBlob=base64.b64decode(kms_encrypted_secret))

Note we have used both of our previously set variables, client and kms_encrypted_secret. If you have chosen different names, be sure to reflect those changes here.

6. Set the returned value to a variable. In this case, client_secret.

client_secret = decrypted_key['Plaintext']

Again, be careful to change the decrypted_key variable name if you've changed it previously.


 
Thus, the password or client secret is never stored in plaintext on the client, nor did we have to allow easy access to the secured passwords stored in the Matillion ETL Password Manager. Note, however, that client_secret can still be printed and exposed.

 

Example

import boto3
import base64
###################################################################################

# Set client_id from the API provider to get access token for.
# The client secret below should be KMS encrypted, this can be done via the aws command line.

# Example - aws kms encrypt --key-id <your key id> --plaintext <client secret>

# The results of which should be copied below.

kms_encrypted_secret = "AQECAHg+HylJSUf4LDePe5tdYU5K/SBuMmi5+Ho+Cwqoym7hBgAAAHIwcAYJKoZIhvcNAQcGoGMwYQIBADBcBgkqhkiG9w0BGwEwHgYJYIZIAWUDBAEuUBOSCCigq0ubRf++flIaigIBEIAvhDSvD43P7M1MSq3dcX8yMqzfN4TF6xF+bNu/L6MV6M95Zcdn4Qze5uuH1OYUXEY="
###################################################################################
client = boto3.client('kms')
# Call AWS to decrypt encrypted secret (needs to be base64encoded)

decrypted_key = client.decrypt(CiphertextBlob=base64.b64decode(kms_encrypted_secret))
# Set returned value to client_secret variable

#client_secret = decrypted_key['Plaintext']
Don't print this unless you really need to!

#print client_secret