When working with Ansible you will at some point have to deal with data that is of a more sensitive nature such as passwords, API- & certificate keys etc. Storing secrets in plain text is bad practice, but still quite common.

If possible the best option is to simply not store any secrets at all and instead fetch/inject these during deployment or runtime with tools such as Hashicorp’s Vault. But for smaller projects this can be too expensive, complex and time-consuming to configure. Thankfully Redhat has included a tool called Ansible Vault in the default Ansible installation. Ansible Vault can encrypt secrets inline or separate files and then automatically decrypt during playbook execution.

To perform Ansible Vault encryption/decryption-operations a vault password needs to be supplied via command prompt or file. Having to supply the vault password manually during each playbook execution will become tedious in the long run and fetching the password from a clear-text file bring about as much benefit as hiding the key on top of the safe.

What’s the better alternative? Let someone else manage the vault password for you! That someone can be a managed third-party service such as Azure Keyvault, AWS SSM, AWS Secrets Manager or Google Secret Manager.

Some of the top benefits using a cloud service provider:

  • RBAC (Who has access to the vault?)
  • Auditing/Traceability (Who decrypted which vault, when and where?)
  • Versioning
  • Enables programmatic logic (Easy to implement in existing pipelines via API and custom SDKs)

Table of Contents

A simple Ansible Vault Password client integrated with Google Secret Manager

The Ansible Vault client in this example will be built using Python. There are of course plenty of other languages to chose from to achieve the same result. Python is however a particular good fit as it is guaranteed to be present wherever Ansible is installed.

Prerequisites

  • Ansible, Python 3 and pip3 installed.
  • A Google Cloud Platform account with access to IAM and Secret Manager
  • gcloud CLI installed and configured (authenticated gcloud auth login and project set gcloud config set project my-gcp-project-12345)

Environment setup and testing Ansible Vault

  1. First export some environment variables.

    ## The password Ansible Vault will use for cryptographic operations
    export VAULT_PASSWORD="Winter2020"
    
    ## Tag that associates Ansible Vault with the environment our vault secrets belongs to.
    #  E.g. development, production etc. 
    export VAULT_ID="dev"
    
    ## The Google Cloud Project ID used
    export GCP_PROJECT_ID="my-gcp-project-12345"
    
  2. Create file password that stores $VAULT_PASSWORD in plain text and vars.yml that stores some super secret data in plain text.

    cat << EOF > password
    $VAULT_PASSWORD
    EOF
    
    cat << "EOF" > vars.yml
    credit_card_no: "4539680266571992"
    favorite_color: "blue"
    mothers_maiden_name: "bates"
    EOF
    
  3. Test encrypting vars.yml with Ansible vault using password from file password

    ansible-vault encrypt --vault-id ${VAULT_ID}@password vars.yml
    
  4. vars.yml should now be encrypted.

    cat vars.yml
    
     $ANSIBLE_VAULT;1.2;AES256;dev
     61626463303539626332336233663033323437343365353634656132663236613936313132366533
     3761363737636430633962323462386631356632353238660a643762326139663566306166626264
     31303066613663373866363037383633306561353135356164653338633733343066336435353733
     6430356161306266390a393634633965646331386238663466303464633863393336356439343038
     32363333383831636334383561313464376539306439613063393962393535636232616636346336
     62383737633364326636653931313962356239353361373661623332643764313965663837366136
     31333336306538636562633935343961303835356631373064323232633165303432366632303233
     61656165616233383032343539346434306565643462613566346661653331326339376137306263
     3265
    

Add Ansible Vault password as a Secret in Google Secret Manager

  • Create a new Secret with identifier ansible-vault-dev in Google Secret Manager containing $VAULT_PASSWORD. Flag --location is set to a nearby location & --replication-policy=user-managed is set as replication is not required for this example. The --data-file-flag is set to a dash "-", which specifies that secret value will be fetched from stdout. Also remove file password as it is no longer going to be used.

    export GCP_VAULT_SECRET_ID="ansible-vault-$VAULT_ID"
    
    echo "${VAULT_PASSWORD}" | gcloud secrets create ${GCP_VAULT_SECRET_ID} \
      --locations=europe-north1 \
      --replication-policy=user-managed \
      --data-file=-
    
    rm password
    

Setup GCP Service Account for Authorization and Authentication

  1. Create a new service account that will be named with the prefix sa- used to read the secret.

    export GCP_SERVICE_ACCOUNT="sa-${GCP_VAULT_SECRET_ID}"
    
    gcloud iam service-accounts create ${GCP_SERVICE_ACCOUNT} \
      --description="Service Account for retrieving Ansible vault password in dev env" \
      --display-name="$GCP_SERVICE_ACCOUNT"
    
  2. With a “principle of least privilege” approach set a policy binding to only allow the service account to read from this particular secret.

    gcloud secrets add-iam-policy-binding ${GCP_VAULT_SECRET_ID} \
      --member="serviceAccount:${GCP_SERVICE_ACCOUNT}@${GCP_PROJECT_ID}.iam.gserviceaccount.com" \
      --role='roles/secretmanager.secretAccessor'
    
  3. Create a credentials key for the Service Account. These credentials will be used by the Ansible Vault client script to authenticate to Google Secret Manager.

    Note: Anyone with access to this key will be able to read the value of $GCP_VAULT_SECRET_ID and any resource granted by additional configured roles. Keep it safe.

    gcloud iam service-accounts keys create key.json \
      --iam-account $GCP_SERVICE_ACCOUNT@$GCP_PROJECT_ID.iam.gserviceaccount.com
    

Ansible Vault Password Client

  1. The Ansible Vault password client will utilize the google-cloud-secret-manager python library. pip install if not already installed.

    pip3 install google-cloud-secret-manager
    
  2. Create a new file called ansible-vault-gcp-client.py and paste in the code. To utilize the built-in Ansible function that provides the vault-id to the password script the filename has to end with ...-client.{extension}.

    Replace the placeholder {GCP_PROJECT_ID} with actual GCP project ID.

    #!/usr/bin/env python3
    import argparse
    
    # Replace placeholder ${GCP_PROJECT_ID} with actual project ID
    project_id = "{GCP_PROJECT_ID}"
    secret_id_prefix = "ansible-vault-"
    
    def access_secret_version(project_id, secret_id, version_id):
        # Import the Secret Manager client library.
        from google.cloud import secretmanager
    
        # Create the Secret Manager client.
        client = secretmanager.SecretManagerServiceClient()
    
        # Build the resource name of the secret version.
        name = f"projects/{project_id}/secrets/{secret_id}/versions/{version_id}"
    
        # Access the secret version.
        response = client.access_secret_version(request={"name": name})
    
        # Print the secret payload.
        # WARNING: Do not print the secret in a production environment - this
        # snippet is showing how to access the secret material.
        payload = response.payload.data.decode('UTF-8')
        print(format(payload))
    
    # Build command line parser 
    parser = argparse.ArgumentParser()
    
    # Accept argument flag for Vault ID from Ansible
    parser.add_argument("--vault-id")
    args = parser.parse_args()
    
    if args.vault_id:
        vault_id = args.vault_id
    
    access_secret_version(project_id, secret_id_prefix + vault_id, "latest")   
    

    The code is more or less derived from Google’s sample code provided in their python-secret-manager repository. And a friendly disclaimer that the code here is only an example and should be viewed as NOT PRODUCTION-READY.

  3. Export path to Service Account key.json file to environment. Google Cloud SDK will automatically use these credentials when this environment variable is present.

    export GOOGLE_APPLICATION_CREDENTIALS="$PWD/key.json"
    
  4. Make ansible-vault-gcp-client.py script executable.

    chmod +x ansible-vault-gcp-client.py
    

Using the client with Ansible

  1. To see the script in action try to decrypt file vars.yml which was previously encrypted with Ansible Vault.

    ansible-vault decrypt --vault-id dev@ansible-vault-gcp-client.py vars.yml
    

    Should return "Decryption successful" and the content should now be readable with cat vars.yml:

    credit_card_no: "4539680266571992"
    favorite_color: "blue"
    mothers_maiden_name: "bates"
    

    Success!

  2. To use the client together with a playbook we’ll modify vars.yml to follow best practices by referencing the secrets which will be placed in a new file, vault.yml.

    cp vars.yml vault.yml
    
    ## Remove secret value from vars.yml 
    ## and replace with vault variable references
    sed -i -r 's/^(.*)\:(.*)/\1: "{{ vault_\1 }}"/' vars.yml
    
    ## update variables with vault_ prefix in vault.yml
    sed -i -e 's/^/vault_/' vault.yml
    

    When done cat vars.yml should produce this:

    credit_card_no: "{{ vault_credit_card_no }}"
    favorite_color: "{{ vault_favorite_color }}"
    mothers_maiden_name: "{{ vault_mothers_maiden_name }}"
    

    and cat vault.yml:

    vault_credit_card_no: "4539680266571992"
    vault_favorite_color: "blue"
    vault_mothers_maiden_name: "bates"
    
  3. Encrypt vault.yml with our script.

    ansible-vault encrypt --vault-id ${VAULT_ID}@ansible-vault-gcp-client.py vault.yml
    
  4. Let’s now create an Ansible Playbook, print-secrets.yml, with the only task of including and printing out secrets that has been encrypted with Ansible Vault

    cat << EOF > print-secrets.yml
    - name: Print secrets from Ansible Vault
    hosts: localhost
    gather_facts: false
    
    tasks:
    - include_vars:
        file: "{{ item }}"
        loop:
        - vars.yml
        - vault.yml
    - debug: 
        msg:
            - "Credit Card number is: {{credit_card_no}}"
            - "Favorite color is: {{ favorite_color }}"
            - "Mother's maiden name is:  {{ mothers_maiden_name }}"
    EOF
    
  5. Now execute the playbook:

    ansible-playbook --vault-id ${VAULT_ID}@ansible-vault-gcp-client.py print-secrets.yml
    

    The result should now print out the secrets:

    PLAY [Print secrets from Ansible Vault] ***********************************************
    
    TASK [include_vars] *******************************************************************
    ok: [localhost] => (item=vars.yml)
    ok: [localhost] => (item=vault.yml)
    
    TASK [debug] **************************************************************************
    ok: [localhost] => {
        "msg": [
            "Credit Card number is: 4539680266571992",
            "Favorite color is: blue",
            "Mother's maiden name is:  bates"
        ]
    }
    
    PLAY RECAP ****************************************************************************
    localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0 
    

Auditing

Head on over to the metrics dashboard for Secret Manager API. There it should now display at least a couple of requests towards the method google.cloud.secretmanager.v1. SecretManagerService.AccessSecretVersion. Requests towards this method are performed every time an encrypt/decrypt action with Ansible Vault and the script ansible-vault-gcp-client.py is executed.

If we’d like we could narrow it down even further by using the Credentials Filter and only selecting the SA-ansible-vault-dev credential, which is the service account previously created with permission to read the Ansible Vault password.

For more granular auditing there’s also the option to enable Audit Logs for Secret Manager API. Every read or write operation will then be indexed and possible to query via Cloud Logging.