Importing variables from a secret manager

Various Importing variables from a secret manager

How to import variables and secrets from third party services and use them with Codemagic

Codemagic allows you to add encrypted secrets and variables in the UI which can be used during your workflow. You can find out more about storing sensitive values in Codemagic here

However, it is possible to use third-party secret managers in your pipelines. In order to do this, you will need to override environment variables defined in your codemagic.yaml configuration file during the build as explained here

AWS Secrets Manager

The following steps illustrate how to use AWS Secrets Manager with Codemagic.

Configure AWS Environment access

You will need to configure the environment variables AWS_DEFAULT_REGION , AWS_SECRET_ACCESS_KEY , and AWS_ACCESS_KEY_ID in the Codemagic UI.

Open your Codemagic app settings, and go to the Environment variables tab.
Enter the desired Variable name, e.g. AWS_DEFAULT_REGION.
Enter the variable value as Variable value.
Enter the variable group name, e.g. aws_credentials. Click the button to create the group.
Make sure the Secure option is selected.
Click the Add button to add the variable.
Repeat the steps to also add AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID variables.

Add the variable group to your codemagic.yaml file

  environment:
    groups:
      - aws_credentials

Storing a secret in AWS Secrets Manager

Make sure that any secrets you store are in the same region as your AWS config file.

Log into your AWS account and search for Secrets Manager
Use the drop list at the top right to select the region that corresponds with Default region you set in your environment variables.
Click on the Store a new secret button.
Set the secret type to Other type of secret.
In the Key/value pairs section select Plain text and delete any placeholder text.
Paste in the secret you would like to store, this can be an RSA certificate or simple string value.
Click the Next button.
In the Secret name and description section enter a name for your secret in the Secret name field. For example, if you are storing your App Store Connect API key, you might want to name it using the same naming convention as Codemagic, e.g. APP_STORE_CONNECT_PRIVATE_KEY.
Click the Next button.
The is no need to configure any secret rotation, so click the Next button again.
On the Review screen, click on the Store button to store your secret.
Repeat this process for other values you would like to store.

Retrieving secrets

Secrets can be retrieved using the AWS CLI.

Secrets stored as plain text values can be retrieved as JSON text and parsed using jq command. Note the use of the -r flag to strip the quotes from the JSON property value returned.

The following example shows how to retrieve a secret called APP_STORE_CONNECT_ISSUER_ID that was stored as plain text.

aws secretsmanager get-secret-value --secret-id APP_STORE_CONNECT_ISSUER_ID | jq -r '.SecretString'

If you have stored your secret using a key/pair value you can use the following syntax:

aws secretsmanager get-secret-value \
  --secret-id APP_STORE_CONNECT_ISSUER_ID | \
  jq -r '.SecretString' | \
  jq -r '.APP_STORE_CONNECT_ISSUER_ID'

The environment variable GCLOUD_SERVICE_ACCOUNT_CREDENTIALS must be provided with a valid Service Account JSON key, even if you will overwrite it with a different key later. Add this variable to a group called service_account and then import it into you workflow as follows:

  environment:
    groups:
      - service_account
      - aws_credentials
  vars:
    GCLOUD_SERVICE_ACCOUNT_CREDENTIALS_HOLDER: $GCLOUD_SERVICE_ACCOUNT_CREDENTIALS

In your first script step, retrieve the Service Account JSON from AWS and add it to CM_ENV.

  scripts:
    - name: Set Service Account JSON from AWS
      script: | 
        echo "GCLOUD_SERVICE_ACCOUNT_CREDENTIALS<<DELIMITER" >> $CM_ENV
        echo "$(aws secretsmanager get-secret-value \
          --secret-id GCLOUD_SERVICE_ACCOUNT_CREDENTIALS | jq -r '.SecretString')" >> $CM_ENV
        echo "DELIMITER" >> $CM_ENV

Reference the variable for publishing to Google Play as follows:

  publishing:
    google_play:
      credentials: $GCLOUD_SERVICE_ACCOUNT_CREDENTIALS_HOLDER
      track: $GOOGLE_PLAY_TRACK

Add the following variables in the codemagic.yaml

  environment:
    groups:
      - aws_credentials
    vars:
      APP_STORE_CONNECT_PRIVATE_KEY: $APP_STORE_CONNECT_PRIVATE_KEY
      APP_STORE_CONNECT_KEY_IDENTIFIER: $APP_STORE_CONNECT_KEY_IDENTIFIER
      APP_STORE_CONNECT_ISSUER_ID: $APP_STORE_CONNECT_ISSUER_ID
      CERTIFICATE_PRIVATE_KEY: $CERTIFICATE_PRIVATE_KEY

In your first script step, retrieve the secrets from AWS and add them to CM_ENV.

  scripts:
    - name: Set iOS credentials from AWS secrets
      script: | 
        echo "APP_STORE_CONNECT_PRIVATE_KEY<<DELIMITER" >> $CM_ENV
        echo "$(aws secretsmanager get-secret-value \
          --secret-id ASC_PRIVATE_KEY | jq -r '.SecretString')" >> $CM_ENV
        echo "DELIMITER" >> $CM_ENV
        #
        echo "CERTIFICATE_PRIVATE_KEY<<DELIMITER" >> $CM_ENV
        echo "$(aws secretsmanager get-secret-value \
          --secret-id CERTIFICATE_PRIVATE_KEY | jq -r '.SecretString')" >> $CM_ENV
        echo "DELIMITER" >> $CM_ENV
        #
        echo "APP_STORE_CONNECT_KEY_IDENTIFIER=$(aws secretsmanager get-secret-value \
          --secret-id APP_STORE_CONNECT_KEY_IDENTIFIER | jq -r '.SecretString')" >> $CM_ENV
        #
        echo "APP_STORE_CONNECT_ISSUER_ID=$(aws secretsmanager get-secret-value \
          --secret-id APP_STORE_CONNECT_ISSUER_ID | jq -r '.SecretString')" >> $CM_ENV

Reference the variables for iOS publishing as usual:

  publishing:
    app_store_connect:              
      api_key: $APP_STORE_CONNECT_PRIVATE_KEY      
      key_id: $APP_STORE_CONNECT_KEY_IDENTIFIER     
      issuer_id: $APP_STORE_CONNECT_ISSUER_ID

Doppler

The following steps illustrate how to use Doppler Secrets Manager with Codemagic.

Configure Doppler access

You will need to configure the DOPPLER_TOKEN environment variable in Codemagic UI. To create the token, login to Doppler, navigate to the ACCESS tab in your project click on Generate. Enter a token name, give it read access and click Generate Service Token.

Open your Codemagic app settings, and go to the Environment variables tab.
Enter the desired Variable name, e.g. DOPPLER_TOKEN.
Enter the tokan value as Variable value.
Enter the variable group name, e.g. doppler_credentials. Click the button to create the group.
Make sure the Secure option is selected.
Click the Add button to add the variable.

Add the variable group to your codemagic.yaml file

  environment:
    groups:
      - doppler_credentials

Storing a secret in Doppler

To store a secret in Doppler, do the following:

Log into your Doppler account and go to your workspace then your project.
Navigate to the SECRETS tab,
Click on the Add Secret button.
Paste in the secret you would like to store, this can be an RSA certificate or simple string value.
In the NAME section enter a name for your secret field. For example, if you are storing your App Store Connect API key, you might want to name it using the same naming convention as Codemagic, e.g. APP_STORE_CONNECT_PRIVATE_KEY.
Repeat this process for other values you would like to store.

Install the Doppler CLI

The Codemagic base build image doesn’t have the Doppler CLI by default, so we need to install it first.

The following example shows how to install the Doppler CLI on various build instance types:

  scripts:
    - name: Install Doppler on macOS
      script: | 
        brew install gnupg
        brew install dopplerhq/cli/doppler

  scripts:
    - name: Install Doppler on Linux
      script: | 
        (curl -Ls --tlsv1.2 --proto "=https" \
          --retry 3 https://cli.doppler.com/install.sh || \
          wget -t 3 -qO- https://cli.doppler.com/install.sh) | \
          sudo sh

  scripts:
    - name: Install Doppler on Windows
      script: | 
        Set-ExecutionPolicy RemoteSigned -Scope CurrentUser
        iex "& {$(irm get.scoop.sh)} -RunAsAdmin"
        scoop bucket add doppler https://github.com/DopplerHQ/scoop-doppler.git
        scoop install doppler

For more information, see the official Doppler docs.

Retrieving secrets

The following example shows how to retrieve a secret called APP_STORE_CONNECT_ISSUER_ID as plain text and add it to the System Environment directly.

echo "APP_STORE_CONNECT_ISSUER_ID=$(doppler secrets get APP_STORE_CONNECT_ISSUER_ID --plain)" >> $CM_ENV

If you want to retrieve a secret with multiline variable, like the GCLOUD_SERVICE_ACCOUNT_CREDENTIALS you can do it like this:

echo "GCLOUD_SERVICE_ACCOUNT_CREDENTIALS<<DELIMITER" >> $CM_ENV    
echo "$(doppler secrets get GCLOUD_SERVICE_ACCOUNT_CREDENTIALS --plain)" >> $CM_ENV
echo "DELIMITER" >> $CM_ENV

Note: If you add the GCLOUD_SERVICE_ACCOUNT_CREDENTIALS make sure you choose No when it asks you to replace \n with new lines.

You can download all of your secrets and add them to your environment in a single step.

Make sure to set the $DOPPLER_ENV variable in the vars section.

  environment:
    groups:
      - doppler_credentials # <-- (Includes the $DOPPLER_TOKEN)
    vars:
      DOPPLER_ENV: dev
  
  scripts:
    - name: Import Dopler secrets
      script: | 
        doppler secrets download --no-read-env \
          --no-file \
          --format env-no-quotes \
          --config $DOPPLER_ENV \
          --token $DOPPLER_TOKEN >> $CM_ENV

For Windows Users

You should add the doppler path to the system path at the beginning of each script like this:

$env:Path += ";C:\Users\builder\scoop\shims"

When accessing the environment variables on the Windows, reference them as $env:VAR_NAME. See more here.

Hashicorp Vault

The following steps illustrate how to use Hashicorp Vault with Codemagic.

Configure HAshicorp Vault access

This example uses token authentication. To authenticate with other methods provided by Hashicorp Vault, please consult their documentation.

You will need to configure the VAULT_NAMESPACE, VAULT_ADDRESS, and VAULT_TOKEN environment variables in the Codemagic UI.

Open your Codemagic app settings, and go to the Environment variables tab.
Enter the desired Variable name, e.g. VAULT_NAMESPACE.
Enter the variable value as Variable value.
Enter the variable group name, e.g. vault_credentials. Click the button to create the group.
Make sure the Secure option is selected.
Click the Add button to add the variable.
Repeat the steps to also add VAULT_ADDRESS and VAULT_TOKEN variables.

Add the variable group to your codemagic.yaml file

  environment:
    groups:
      - vault_credentials

Storing a secret in Hashicorp Vault

Install the Hashicorp Vault CLI on your local machine:

brew tap hashicorp/tap
brew install hashicorp/tap/vault

To set a simple string value, you can add a single secret to a group as follows:

vault kv put secret/greetings message=hello

Retrieve the secret as follows:

vault kv get -field=message secret/greetings

To add a secret from file such as an RSA key or JSON key, you can add the contents of a file as follows:

vault kv put secret/gcloud GCLOUD_SERVICE_ACCOUNT_CREDENTIALS=@gcloud.json

To retrieve the secret you would do the following:

vault kv get -field=GCLOUD_SERVICE_ACCOUNT_CREDENTIALS secret/gcloud

Retrieving secrets from Hashicorp Vault

Secrets can be retrieved using the Hashicorp Vault CLI.

To retrieve a secret (e.g. ‘message’) stored as plain text value on a specified path (e.g. ‘secret/greetings’):

vault kv get -field=message secret/greetings

Android
iOS

The environment variable GCLOUD_SERVICE_ACCOUNT_CREDENTIALS must be provided with a valid Service Account JSON key, even if you will overwrite it with a different key later. Add this variable to a group called service_account and then import it into you workflow.
Add the secret for your Google Console Service Account to Hashicorp Vault as follows:

vault kv put secret/google \\
GCLOUD_SERVICE_ACCOUNT_CREDENTIALS=@/path/to/service_account.json

Add the following environment variables and groups in your codemagic.yaml

  environment:
    groups:
      - service_account
      - vault_credentials
    vars:
      GCLOUD_SERVICE_ACCOUNT_CREDENTIALS_HOLDER: $GCLOUD_SERVICE_ACCOUNT_CREDENTIALS

In your first script step, retrieve the Service Account JSON from Hashicorp Vault and add it to CM_ENV.

  scripts:
    - name: Set Service Account JSON from Hashicorp Vault
      script: | 
        echo "GCLOUD_SERVICE_ACCOUNT_CREDENTIALS<<DELIMITER" >> $CM_ENV
        echo "$(vault kv get -field=GCLOUD_SERVICE_ACCOUNT_CREDENTIALS secret/google)" >> $CM_ENV
        echo "DELIMITER" >> $CM_ENV

Reference the variable for publishing to Google Play.

  publishing:
    google_play:
      credentials: $GCLOUD_SERVICE_ACCOUNT_CREDENTIALS_HOLDER
      track: $GOOGLE_PLAY_TRACK

Add your App Store credentials to Hashicorp Vault.

vault kv put secret/appstore \\
APP_STORE_CONNECT_PRIVATE_KEY=@/path/to/app_store_connect_api_key_file
APP_STORE_CONNECT_KEY_IDENTIFIER=XXXXXXXXXX
APP_STORE_CONNECT_ISSUER_ID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
CERTIFICATE_PRIVATE_KEY=@/path/to/ios_distribution_cert_private_key_file

Add the following environment variables and groups in your codemagic.yaml

  environment:
    groups:
      - vault_credentials
    vars:
      APP_STORE_CONNECT_PRIVATE_KEY: $APP_STORE_CONNECT_PRIVATE_KEY
      APP_STORE_CONNECT_KEY_IDENTIFIER: $APP_STORE_CONNECT_KEY_IDENTIFIER
      APP_STORE_CONNECT_ISSUER_ID: $APP_STORE_CONNECT_ISSUER_ID
      CERTIFICATE_PRIVATE_KEY: $CERTIFICATE_PRIVATE_KEY

In your first script step, retrieve the secrets from Hashicorp Vault and add them to CM_ENV.

  scripts:
    - name: Set iOS credentials from Hashicorp Vault
      script: | 
        echo "APP_STORE_CONNECT_PRIVATE_KEY<<DELIMITER" >> $CM_ENV
        echo "$(vault kv get -field=APP_STORE_CONNECT_PRIVATE_KEY secret/appstore)" >> $CM_ENV
        echo "DELIMITER" >> $CM_ENV
        #
        echo "CERTIFICATE_PRIVATE_KEY<<DELIMITER" >> $CM_ENV
        echo "$(vault kv get -field=CERTIFICATE_PRIVATE_KEY secret/appstore)" >> $CM_ENV
        echo "DELIMITER" >> $CM_ENV
        #
        echo "APP_STORE_CONNECT_KEY_IDENTIFIER=$(vault kv get -field=APP_STORE_CONNECT_KEY_IDENTIFIER secret/appstore)" >> $CM_ENV
        #
        echo "APP_STORE_CONNECT_ISSUER_ID=$(vault kv get -field=APP_STORE_CONNECT_ISSUER_ID secret/appstore)" >> $CM_ENV

Reference the variables for iOS publishing as usual.

  publishing:
    app_store_connect:              
      api_key: $APP_STORE_CONNECT_PRIVATE_KEY      
      key_id: $APP_STORE_CONNECT_KEY_IDENTIFIER     
      issuer_id: $APP_STORE_CONNECT_ISSUER_ID

Importing variables from a secret manager

AWS Secrets Manager

Configure AWS Environment access

Storing a secret in AWS Secrets Manager

Retrieving secrets

Use secrets in codemagic.yaml

Doppler

Configure Doppler access

Storing a secret in Doppler

Install the Doppler CLI

Retrieving secrets

For Windows Users

Hashicorp Vault

Configure HAshicorp Vault access

Storing a secret in Hashicorp Vault

Retrieving secrets from Hashicorp Vault

Getting help and support

Use secrets in `codemagic.yaml`