AWS Secrets Manager is an excellent service for managing sensitive information like database credentials, certificates, passwords, and tokens in the cloud. While infrastructure as code (IaC) tools like Terraform can integrate with secret management services like Secrets Manager, KMS, or Parameter Store, Terraform’s native secrets management has some limitations.
This article discusses best practices for secret management in Terraform. We cover how Terraform can be configured to create and retrieve secrets using Secrets Manager and discuss the challenges of Terraform’s native secrets management approach. We also look at how good third-party tools can address Terraform’s limitations and provide an example of how such a tool can streamline secret management for multiple platforms.
Summary of secrets supported in Terraform
Let’s take a quick look at the different secrets that can be used within Terraform and their use cases in an AWS environment.
Type of Secret
Usage
IAM user credentials
Authentication with AWS accounts
Database credentials
Establishing connectivity with database instances
Tokens
Configuring API keys and establishing connectivity with third-party services like GitHub, GitLab, etc.
SSH key pairs
Enabling access to EC2 instances
Environment variables
Configuring services like lambda functions, ECS, EC2, CodeBuilds, etc.
Summary of secret management best practices in Terraform
The table below gives an overview of the best practices to follow while managing secrets in Terraform. We will explain each in detail in the following section.
Best Practice
Description
Avoid storing secrets in plain text
Hardcoding secrets in code can expose them easily from the version control system. Store secrets in external secret management services like AWS Secrets Manager, Parameter Store, or Hashicorp Vault.
Use the Terraform data module to retrieve secrets
The Terraform data module can be used to securely retrieve credentials from services like AWS Secrets Manager without exposing them.
Pass secrets to Terraform as input variables, variable definition files, or environment variables
Terraform supports passing secrets as input variables, variable definition files, and environment variables to eliminate hardcoding secrets.
Store Terraform state in a remote backend instead of a local state file
Storing Terraform state in remote backends like S3 can prevent secrets from being exposed using the local state file.
Use tools that can inject secrets dynamically for team collaboration
Tools like Doppler can help inject secrets dynamically into Terraform configurations, eliminating committing secrets to the version control system.
Best practices for secret management in Terraform
Now that you’ve gotten a brief idea about the best practices, let’s take a deeper dive into each one, with examples.
{{banner-1="/design/banners"}}
Avoid storing secrets in plain text
Storing secrets as plain text inside code exposes the secrets to anyone having access to your version control system when you push the code. This can be dangerous if the repository is public and can be avoided by using secrets management services like AWS Secrets Manager, AWS Parameter Store, or Hashicorp Vault.
Here’s an example of storing secrets using AWS Secrets Manager in Terraform.
Create a main.tf file and add the following code to create your secret.
Now, use terraform plan and pass the secret as a variable to review your changes:
➜ terraform plan -var db_password=securevalue -out tfplan
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with
the following symbols:
+ create
Terraform will perform the following actions:
# aws_secretsmanager_secret.db_password will be created
+ resource "aws_secretsmanager_secret" "db_password" {
+ arn = (known after apply)
+ force_overwrite_replica_secret = false
+ id = (known after apply)
+ name = "dev/mysql/password"
+ name_prefix = (known after apply)
+ policy = (known after apply)
+ recovery_window_in_days = 30
+ rotation_enabled = (known after apply)
+ rotation_lambda_arn = (known after apply)
+ tags_all = (known after apply)
}
# aws_secretsmanager_secret_version.db_password_version will be created
+ resource "aws_secretsmanager_secret_version" "db_password_version" {
+ arn = (known after apply)
+ id = (known after apply)
+ secret_id = (known after apply)
+ secret_string = (sensitive value)
+ version_id = (known after apply)
+ version_stages = (known after apply)
}
Plan: 2 to add, 0 to change, 0 to destroy.
Finally, perform terraform apply to deploy the configuration.
The Terraform data module allows Terraform to use external information in its configuration. This can be done by specifying the appropriate provider, such as an AWS Secrets Manager data source.
The following example demonstrates how you can use the AWS Secrets Manager data source to retrieve secrets into your Terraform configuration from AWS.
Create a main.tf file and add the following code specifying the Secrets Manager data module:
Pass secrets to Terraform as input variables, variable definition files, or environment variables
Secrets can be passed to Terraform as input variables, variable definition files, or environment variables.
Input variables
Users can pass the secrets with the -var command line option while running Terraform commands. The following example shows how users can pass input variables during the Terraform plan state and apply the Terraform configuration to deploy resources.
➜ terraform plan -var db_password=secretvalue -out tfplan
aws_secretsmanager_secret.db_password: Refreshing state... [id=arn:aws:secretsmanager:ap-south-1:123123123123:secret:dev/mysql/password-zGdSnY]
aws_secretsmanager_secret_version.db_password_version: Refreshing state... [id=arn:aws:secretsmanager:ap-south-1:123123123123:secret:dev/mysql/password-zGdSnY|FBBB224C-1C04-4544-A59C-5BFF7089B019]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with
the following symbols:
-/+ destroy and then create replacement
Terraform will perform the following actions:
# aws_secretsmanager_secret_version.db_password_version must be replaced
-/+ resource "aws_secretsmanager_secret_version" "db_password_version" {
~ arn = "arn:aws:secretsmanager:ap-south-1:123123123123:secret:dev/mysql/password-zGdSnY" -> (known after apply)
~ id = "arn:aws:secretsmanager:ap-south-1:123123123123:secret:dev/mysql/password-zGdSnY|FBBB224C-1C04-4544-A59C-5BFF7089B019" -> (known after apply)
~ secret_string = (sensitive value) # forces replacement
~ version_id = "FBBB224C-1C04-4544-A59C-5BFF7089B019" -> (known after apply)
~ version_stages = [
- "AWSCURRENT",
] -> (known after apply)
# (1 unchanged attribute hidden)
}
Plan: 1 to add, 0 to change, 1 to destroy.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Saved the plan to: tfplan
To perform exactly these actions, run the following command to apply:
terraform apply "tfplan"
➜ terraform apply tfplan
aws_secretsmanager_secret_version.db_password_version: Destroying... [id=arn:aws:secretsmanager:ap-south-1:123123123123:secret:dev/mysql/password-zGdSnY|FBBB224C-1C04-4544-A59C-5BFF7089B019]
aws_secretsmanager_secret_version.db_password_version: Destruction complete after 0s
aws_secretsmanager_secret_version.db_password_version: Creating...
aws_secretsmanager_secret_version.db_password_version: Creation complete after 1s [id=arn:aws:secretsmanager:ap-south-1:123123123123:secret:dev/mysql/password-zGdSnY|AF6DCE17-9D9E-41E0-A465-8CA2D5A63BB2]
Apply complete! Resources: 1 added, 0 changed, 1 destroyed.
Variable definition files
Users can enter the secrets inside a .tfvars file and specify this on the command line using the -var-file argument. However, this requires users to be careful to not push the tfvars file to the version control system and maintain it themselves.
Let’s go through an example by creating a secrets.tfvars file with the following content:
db_password="securevalue"
Apply the configuration to deploy the changes using the following command:
terraform plan -var-file=secrets.tfvars -out tfplan
terraform apply tfplan
{{banner-3="/design/banners"}}
Environment variables
Users can export secrets in the command line as TF_VAR_ followed by the variable name before running the Terraform commands. The following example shows how users can use environment variables to pass secrets to Terraform:
➜ export TF_VAR_db_password=securevalue
➜ terraform plan -out tfplan
aws_secretsmanager_secret.db_password: Refreshing state... [id=arn:aws:secretsmanager:ap-south-1:123123123123:secret:dev/mysql/password-zGdSnY]
aws_secretsmanager_secret_version.db_password_version: Refreshing state... [id=arn:aws:secretsmanager:ap-south-1:123123123123:secret:dev/mysql/password-zGdSnY|9F82C9E6-438A-4576-8D5E-117FB563C9E2]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with
the following symbols:
-/+ destroy and then create replacement
Terraform will perform the following actions:
# aws_secretsmanager_secret_version.db_password_version must be replaced
-/+ resource "aws_secretsmanager_secret_version" "db_password_version" {
~ arn = "arn:aws:secretsmanager:ap-south-1:123123123123:secret:dev/mysql/password-zGdSnY" -> (known after apply)
~ id = "arn:aws:secretsmanager:ap-south-1:123123123123:secret:dev/mysql/password-zGdSnY|9F82C9E6-438A-4576-8D5E-117FB563C9E2" -> (known after apply)
~ secret_string = (sensitive value) # forces replacement
~ version_id = "9F82C9E6-438A-4576-8D5E-117FB563C9E2" -> (known after apply)
~ version_stages = [
- "AWSCURRENT",
] -> (known after apply)
# (1 unchanged attribute hidden)
}
Plan: 1 to add, 0 to change, 1 to destroy.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Saved the plan to: tfplan
To perform exactly these actions, run the following command to apply:
terraform apply "tfplan"
➜ secrets manager tf terraform apply tfplan
aws_secretsmanager_secret_version.db_password_version: Destroying... [id=arn:aws:secretsmanager:ap-south-1:123123123123:secret:dev/mysql/password-zGdSnY|9F82C9E6-438A-4576-8D5E-117FB563C9E2]
aws_secretsmanager_secret_version.db_password_version: Destruction complete after 0s
aws_secretsmanager_secret_version.db_password_version: Creating...
aws_secretsmanager_secret_version.db_password_version: Creation complete after 0s [id=arn:aws:secretsmanager:ap-south-1:123123123123:secret:dev/mysql/password-zGdSnY|FBBB224C-1C04-4544-A59C-5BFF7089B019]
Apply complete! Resources: 1 added, 0 changed, 1 destroyed.
Store Terraform state in a remote backend instead of a local state file
Since a Terraform state file contains sensitive information about your resource configurations, it is recommended to store the state files in a remote backend like S3 bucket and avoid pushing them to the version control system.
To create a new S3 backend for your Terraform project, follow these steps:
Add the following code inside your main.tf file after specifying your bucket name:
Next, perform terraform init to initialize your remote backend:
➜ terraform init
Initializing the backend...
Do you want to copy existing state to the new backend?
Pre-existing state was found while migrating the previous "local" backend to the newly configured "s3" backend. No existing state was found in the newly configured "s3" backend. Do you want to copy this state to the new "s3" backend? Enter "yes" to copy and "no" to start with an empty state.
Enter a value: yes
Successfully configured the backend "s3"! Terraform will automatically use this backend unless the backend configuration changes.
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v4.67.0
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work.
If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary.
Note that it is recommended to have versioning enabled on your S3 bucket so that you can recover old state files in case.
{{banner-4="/design/banners"}}
Use tools that can inject secrets dynamically for team collaboration
Integration with tools like Doppler can help inject secrets from multiple providers dynamically into your Terraform code, helping keep a single source of truth for the secrets across your projects.
The following example demonstrates how users can make use of Doppler CLI to dynamically inject secrets into Terraform.
First, make sure you have Doppler CLI installed on your system and configure it by doing doppler login and doppler setup to grant Doppler API access to the required secrets.
Next, integrate Doppler with AWS Secrets Manager. Doppler supports integration with multiple services; check out this article to learn how to integrate Doppler with AWS Secrets Manager.
Once you’ve finished the integration, create your secret from the Doppler dashboard. You could choose to generate a random secret value as follows for added security:
Finally, run the following command to inject the secret inside your Terraform configuration:
doppler run --name-transformer tf-var -- terraform plan -out tfplan
Let’s verify this by running the command from the terminal.
➜ doppler run --name-transformer tf-var -- terraform plan -out tfplan
aws_secretsmanager_secret.db_password: Refreshing state... [id=arn:aws:secretsmanager:ap-south-1:123123123123:secret:dev/mysql/password-zGdSnY]
aws_secretsmanager_secret_version.db_password_version: Refreshing state... [id=arn:aws:secretsmanager:ap-south-1:123123123123:secret:dev/mysql/password-zGdSnY|79D3EC33-FE5C-4F8E-B20E-F822214C4199]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement
Terraform will perform the following actions:
# aws_secretsmanager_secret_version.db_password_version must be replaced
-/+ resource "aws_secretsmanager_secret_version" "db_password_version" {
~ arn = "arn:aws:secretsmanager:ap-south-1:123123123123:secret:dev/mysql/password-zGdSnY" -> (known after apply)
~ id = "arn:aws:secretsmanager:ap-south-1:123123123123:secret:dev/mysql/password-zGdSnY|79D3EC33-FE5C-4F8E-B20E-F822214C4199" -> (known after apply)
~ secret_string = (sensitive value) # forces replacement
~ version_id = "79D3EC33-FE5C-4F8E-B20E-F822214C4199" -> (known after apply)
~ version_stages = [
- "AWSCURRENT",
] -> (known after apply)
# (1 unchanged attribute hidden)
}
Plan: 1 to add, 0 to change, 1 to destroy.
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Saved the plan to: tfplan
To perform exactly these actions, run the following command to apply:
terraform apply "tfplan"
➜ secrets manager tf terraform apply tfplan
aws_secretsmanager_secret_version.db_password_version: Destroying... [id=arn:aws:secretsmanager:ap-south-1:{accountNumber}:secret:dev/mysql/password-zGdSnY|79D3EC33-FE5C-4F8E-B20E-F822214C4199]
aws_secretsmanager_secret_version.db_password_version: Destruction complete after 0s
aws_secretsmanager_secret_version.db_password_version: Creating...
aws_secretsmanager_secret_version.db_password_version: Creation complete after 0s [id=arn:aws:secretsmanager:ap-south-1:123123123123:secret:dev/mysql/password-zGdSnY|9F82C9E6-438A-4576-8D5E-117FB563C9E2]
Apply complete! Resources: 1 added, 0 changed, 1 destroyed.
As you can see, Doppler dynamically injected the secrets into the Terraform configuration while keeping the secret as a sensitive value.
Challenges of Terraform’s native secrets management approach
Terraform’s native secrets management has the following drawbacks:
Insecure by default: By default, secrets are stored as plain text in Terraform. Users have to encrypt manually to create a secure approach.
Limited scope: Terraform’s secrets capability is limited to Terraform configurations only; it cannot be used to store or manage secrets outside Terraform.
Tightly coupled: The secrets created using Terraform are tightly coupled to Terraform configurations. Terraform’s native approach does not provide a centralized way to store secrets across multiple accounts or environments.
Version control issues: Since Terraform stores secrets as plain text by default, they need to be excluded from version control. This causes issues with collaboration and auditing.
How Doppler helps overcome Terraform’s limitations
Doppler can help improve Terraform by providing the following advantages:
Secure by default: All the secrets stored using Doppler are encrypted at rest and in transit. Users do not have to manually encrypt secrets.
Broad scope: Doppler is a dedicated secrets management solution that can help store and manage secrets across all infrastructure, applications, and teams.
Decoupled from configurations: Secrets in Doppler are decoupled from Terraform configurations and can be used across multiple accounts or environments. This makes configurations more robust and reusable.
Better version control: Since Doppler stores secrets externally, Terraform configurations can be stored within version control as usual, helping improve collaboration and auditing.
Support for multiple providers: Doppler integrates with various secret services and cloud providers beyond the limited set supported by Terraform natively.
Advanced features: Doppler provides additional features like secret rotation, auditing, access control, etc., helping achieve a robust secret management solution.
Simple integration: Doppler integrates easily with Terraform using its Terraform provider. Secrets can be fetched directly from Doppler into Terraform configurations.
{{banner-5="/design/banners"}}
Conclusion
This article provides an in-depth look at using AWS Secrets Manager with Terraform. While Terraform can be used to provision and manage secrets in Secrets Manager, Terraform’s native approach has some limitations. We described these limitations in detail and went through best practices and security recommendations that can help improve Terraform’s secret management capability.
The article also highlights how combining Terraform with tools like Doppler can make secret management more efficient and secure. Since Doppler integrates with multiple secret management services, it can be a reliable option for managing secrets across different cloud providers, empowering teams to scale their deployments efficiently while maintaining security and control over sensitive data.
If you’re interested in learning more about Doppler, check out the free team trial for 14 days without a credit card or request a demo to try Doppler for the enterprise.
