The Ops Community ⚙️: Mel Kaulfuß 👩🏻‍🦰💻✌️

Best Practices for Terraform workflows in CI/CD

Mel Kaulfuß 👩🏻‍🦰💻✌️ — Mon, 11 Sep 2023 04:27:45 +0000

Most engineers get started with Terraform using the command-line interface (CLI) to deploy and manage infrastructure directly from their local machine. However, as projects evolve, so does the need for collaborative, scalable, secure, and resilient workflows. Transitioning from a local Terraform workflow to a shared CI/CD pipeline can seem like a daunting task but if you take the leap, you won’t look back.

In this guide, I'll cover some best practices and the things you'll need to consider for your Terraform CI/CD workflows.

Table of Contents:

Version control
Project structure
Managing Terraform configuration for different environments
Managing secrets
Terraform state and state locking
Handling concurrency
Version constraints
Enforcing code quality
Using policies for governance and compliance
Pre-preparing the build environment
Using a shared plugin cache
Approval gates
Managing CI/CD with Terraform

Version Control

Like any other code, Infrastructure code, including Terraform, should be version controlled too. Using a Source Code Management (SCM) system like git with platforms such as GitHub, GitLab, or Bitbucket ensures that all infrastructure changes are auditable, collaborative, and reversible. Being able to revert to a previous configuration to see what changes were made, when they were made, and who made them is critical when managing your infrastructure using code. This also means everyone on the team can view and contribute to the projects consistently, and means you can utilize fully-automated CI/CD by setting up triggers for your pipeline on repository actions like commits and pull requests.


# Local .terraform directories
**/.terraform/*

# .tfstate files
*.tfstate*.tfstate.*

# Crash log files
crash.log
crash.*.log

# Exclude all .tfvars files, which are likely to contain sensitive data, such as
# password, private keys, and other secrets. These should not be part of version 
# control as they are data points which are potentially sensitive and subject 
# to change depending on the environment.
*.tfvars
*.tfvars.json

# Ignore override files as they are usually used to override resources locally and so
# are not checked in
override.tf
override.tf.json
*_override.tf
*_override.tf.json

# Include override files you do wish to add to version control using negated pattern
# !example_override.tf

# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
# example: *tfplan*

# Ignore CLI configuration files
.terraformrc
terraform.rc

🙋‍♀️ When storing your Terraform code in version control, make sure you use a terraform .gitignore file like this one](https://github.com/github/gitignore/blob/main/Terraform.gitignore) to ensure files with sensitive data aren't accidentally committed.

Project structure

There are many different approaches for how to structure and store Terraform code, and there is no one-size-fits-all approach. It all comes down to how you want to manage your code, any contributions, and how it will end up being used in projects and teams.

Terraform files can be stored in:

a project repository (alongside application code)
a separate repository (for the Terraform files)
multiple repositories

The best way to get started is to store your Terraform code in the same repo as the application code. This is also a common approach for smaller, less-distributed teams. It's always best to start simple! Understand how you like to work, really feel any pain, so you can better understand the most suitable abstractions that work for your team moving forward. Abstracting too early on leads to ill-informed decisions.

Separating Terraform code into reusable and composable modules that live in separate repositories is also a great solution, especially for teams with a lot of projects, and complex infrastructure requirements. Besides keeping your Terraform DRY, a benefit of this approach is that you can have a mix of private (restricted) repositories for managing more secure things such as roles and permissions, and engineers are still able to compose their own Terraform configurations by including these pre-defined (and locked down) modules.

The standard module structure is a great guide that provides a recommended file and directory layout structure for reusable modules distributed in separate repositories. Or if you’re interested in further assessing the differences between two more common approaches (Mono repo vs. Multi repo), this blog post from HashiCorp dives into more detail and explains the pros and cons of each approach.

Managing Terraform configuration for different environments

There are a variety of conventions for how to handle different environments (e.g. development, staging and production) in Terraform.

You can:

Create variables prefixed by the environment i.e. dev-s3-bucket, staging-s3-bucket, prod-s3-bucket.
Use a git branch naming convention and long-lived environment branches.
Use Terraform workspaces.
Separate environment specific configurations in into their own folders i.e. development, staging, production.
And/or use supplementary tools like Terragrunt.

A common approach is to separate environment configurations into individual folders within the same project.

my-application/
└── terraform/
  ├── main.tf
  ├── variables.tf
  ├── outputs.tf
  └── environments/
  ├── development/
  │ └── main.tf  
  ├── staging/        
  │ └── main.tf        
  └── production/            
    └── main.tf

When working in a specific environment, scripts automatically locate and execute the Terraform commands in that environment’s directory. This is a great way to ensure each environment’s configuration and state remains isolated.

If you’re just getting started with Terraform, it's best to start simple with something that works for your immediate and short-term needs. As you learn what works and what doesn't work you can always adapt your approach.

Managing secrets

What not to do

When we're working locally we’ll often handle secrets in pretty "simple" ways. We'll commonly use environment variables on our local machines, and sometimes we might even hard-code them during the initial stages of development 😓. These methods might be quick, but they don't scale and they pose significant security risks, configurations can be accidentally pushed to version control with sensitive data included and it happens more often than you think.

resource "aws_db_instance" "my_db_instance" {  
  # ...
  username = "admin"  
  # DO NOT DO THIS!!!  
  password = "myS3cretp@ssw0rd"
}

Use a secrets manager

Secrets management is serious business, especially in shared, collaborative CI/CD workflows. You should use a dedicated centralized secrets manager such as:

These tools store, manage, log, and control access to tokens, passwords, certificates, and other secrets and offer greater control and visibility. When managing access with tools like these, it's important to apply the principle of least privilege – that is to restrict access to the users and systems that absolutely need it.

Tips for working with secrets

Once the required secrets are made available to your build system or build agents:

Use environment variables and Terraform data sources to reference your secrets in your Terraform code.
Make sure state files are not tracked in version control, and that they’re included in your Terraform.gitignore
Use a remote state file along with advanced linting and validation techniques to prevent exposing secret values

Most CI/CD tools allow you to choose and integrate with any secrets management platform in various ways. For example, in Buildkite you can use environment hooks to conditionally allow access to secrets based on the requesting repository, team, pipeline, commit author, or any other logic you choose.

State and state locking

Terraform maintains the "state" of your infrastructure in a state file (it's a snapshot of the current infrastructure resources). It's usually named terraform.tfstate. State files should not be tracked in version control because they can potentially contain sensitive information.

In a local Terraform workflow, this state file is stored on a developer's machine. Using local state in Terraform projects that have multiple collaborators can cause some pretty frustrating problems. Changes can confusingly get mixed up, and that usually results in more errors (that can be very difficult to untangle). Also, to make things worse, without a clear, consistent record of changes to state rolling back to a previous version cleanly can be problematic.

Fortunately Terraform provides an option for storing state remotely by configuring a backend in your Terraform code, usually in the root module.

terraform {  
  backend "s3" {    
    bucket         = "my-tfstate-bucket"    
    key            = "path/to/my/key"    
    region         = "us-west-2"    
    dynamodb_table = "my-lock-table"    
    encrypt        = true  
  }
}

This example configures a backend using Amazon S3 combined with DynamoDB for state locking, so that when Terraform detects a supported backend is in use, it will automatically lock the state.

State locking is especially important in team environments or CI/CD pipelines where there can be multiple attempts to modify infrastructure simultaneously. When Terraform state is locked, it prevents multiple concurrent operations (e.g. terraform apply or terraform plan) from being performed on the same state. This mechanism ensures that state changes are atomic and mitigates the risk of state corruption or inconsistencies due to overlapping operations. If another user or process tries to perform an operation while the state is locked, Terraform will return an error and prevent that action from proceeding until the lock is released.

Restrict access to the remote state file, to highly privileged administrators, and build systems in certain build contexts i.e. main branch builds, and even a set of specific users.

Handling concurrency

We can’t talk about state locking without also looking at how to handle concurrency, a major concern for anyone making changes in a shared environment, be it infrastructure, or application. Most CI/CD platforms will have a way of controlling concurrency to ensure jobs and tasks don't collide with each other, for example, only one job can ever deploy to an environment at a time.

In Buildkite, you can use concurrency groups and/or concurrency gates to control concurrency and still benefit from running CI jobs in parallel.

- command: "deploy.sh"
  label: ":rocket: Deploy production"
  branches: "main"
  agents:    
    deploy: true  
    concurrency: 1  
    concurrency_group: "my-payment-gateway/deploy"

In the Buildkite pipeline step definition above, only one job in the 'my-payment-gateway/deploy' concurrency group will be executed at a time, across an entire Buildkite organization’s pipelines and builds.

Creating strict CI/CD concurrency rules combined with Terraform state locking ensures that infrastructure deployments won’t conflict with each other, and that updates will always happen sequentially.

Version constraints

Making sure that the installed versions of components remain consistent between developer workstations and build environments reduces inconsistencies, errors and confusion. Using strict versioning is one way to ensure consistency, which becomes especially important when working at scale, on a shared codebase, or in a distributed team.

There are three components in Terraform which use of versioning:

Terraform CLI
Terraform providers
Terraform modules

Terraform uses Semantic Versioning's major.minor.patch format, ie. 2.3.11 to specify versions and requirements for these components.

Understanding the trade-offs

Adopting and maintaining strict versioning takes effort, but provides consistency and increased reliability.
A more lenient, low-effort approach makes taking advantage of new features and bug fixes slightly easier without having to update your terraform configuration, but this trade off can result in errors and confusion.

Before you decide to adopt strict versioning, consider the size and experience of your team, along with:

The amount of time you can dedicate to monitoring, reviewing, updating, and testing new versions.
Whether you can live with the technical debt generated if you fall behind new versions.
Whether you can guarantee changes are thoroughly tested and simulated.
How well your development, test, and staging environments reflect your production environment. How critical the infrastructure or service is.
Whether you can rollback if an issue is identified at a later stage, outside of testing.

You might decide to accept the relatively low risk of using a more lenient, less strict versioning standard, or decide the risk is not acceptable, and enforce strict versioning constraints.

HashiCorp recommends using the ~> version constraint to pin only major and minor versions, meaning patch versions are used without needing to update the Terraform configuration.

terraform {  
  required_providers {    
    aws = {      
      version = "~> 2.13.0"    
    }  
  }  
  required_version = "~> 0.12.29"
}

The ~> symbol in the above required_version = "~> 0.12.29" allows the patch version to be greater than 29 but requires the major and minor versions (0.12) to match the version that the configuration specifies.

Enforcing code quality with formatting, linting and validation

Linting Terraform code is incredibly valuable, it enforces a consistent style across Terraform projects, making code easier to read and understand. When integrated into CI/CD, it catches syntactical errors early, minimizing things going wrong with infrastructure deployments. Automated quality checks significantly speed up the code review process by ensuring only high-standard code is ready to review and deploy. Linting can improve code reliability, reduce cognitive load and help developers learn by setting clear coding styles and standards.
There are some great features and tools to help improve Terraform code quality:

Terraform CLI:

terraform fmt and terraform validate are built in methods to help with formatting and validating
Extending Terraform's native linting and validation:
- tflint offers more comprehensive linting, catching resource-specific errors and allowing custom rule checks.
- tfsec scans Terraform code for potential security vulnerabilities, providing in-depth security insights.
- checkov evaluates Terraform configurations against hundreds of security and compliance benchmarks, ensuring robustness beyond basic validations. Together, these tools fortify the Terraform workflow, ensuring code quality, security, and compliance.

Checking code formatting and validation should be automated and happen in CI/CD pipelines. If the checks don't pass – conditionally block or fail builds so that the issues need to be fixed before being merged. To avoid the inevitable “it worked on my machine” moments, consider using a git pre-commit hook to enforce checks being run locally before code is committed and pushed. Incorporating container images into your project development workflow and documenting a process around how they are built and used can also help address other consistency issues.

When it comes to consistency and code quality, what automation can’t solve, open communication, collaboration, good documentation (such as engineering manuals) and README files can.

Using policies for governance and compliance

Limiting and restricting the resources that can be created, and the configuration of those resources is essential. This kind of governance can usually be set in the platform in which the resources are being created, using features like AWS Control Tower. But tools like Open Policy Agent (OPA) offer an even more powerful layer of governance and compliance for Terraform projects.

OPA provides fine-grained, programmable control over Terraform configurations. Being able to codify policies means best practices and any organizational and regulatory standards set are automatically enforced.

Automating governance and compliance policies fosters a secure and compliant infrastructure-as-code practice by:

Preemptively catching violations, reducing the risk of deploying non-compliant resources, and strengthening security posture and operational reliability.
Streamlining the review process, ensuring consistent adherence to policies without manual oversight.
Providing contributors with a guardrail to:
- Ensure their infrastructure changes remain within defined boundaries
- Get feedback early in the development lifecycle before deploying to a target environment and getting a permission error The challenge with having multiple layers of governance is that it can often lead to conflicting policies between those defined within your project, and those enforced by maintainers of the target platform, so it’s important for these teams to collaborate closely and define a clear process for how governance can remain aligned across teams.

Pre-preparing the build environment

For Terraform to run consistently and efficiently in CI/CD, the build environment should be prepared in advance. Ahead of any jobs being run, install any build dependencies such as the Terraform CLI, and other required command-line tools. Additionally, build any machine images or container images used in CI/CD pipelines on a regular basis. Dependencies will be kept up to date, and considerable time saved by not having to do this set up every time a job runs or a container/machine is started. You can also fetch an up-to-date clone of any git repositories used as part of these image builds. And to really speed things up, run a terraform init to pre-fetch the required Terraform providers and plugins, so that the job in CI/CD is already ready to go.

Using a shared plugin cache

To speed things up even more, consider a shared plugin cache. By default, Terraform looks for plugins in a project’s local .terraform directory, which is generally unique to each job. You can override this by leveraging the TF_PLUGIN_CACHE_DIR environment variable to specify a different local directory as a shared plugin cache. This means each distinct plugin binary is only downloaded once, and shared by all terraform workloads on the same machine.

As a bonus, if you’re using container images for this process, these same pre-built container images can be re-used for local development on developer machines, for a consistent and fast development and testing environment.

Approval gates

Branch protection rules are commonly used to disable forced pushes, ensure that pull requests (PR’s) are reviewed, and ensure that required CI tests have passed before allowing a change to a protected branch.

In most scenarios, when a PR is raised, an associated CI/CD pipeline is triggered. On successful execution of the pipeline, and when the required tests and validations have passed, this is often sufficient evidence that the PR can be merged, either automatically or with additional reviews and approvals. This can often be enough, but in some cases you might want to conditionally block a build from proceeding, and require a privileged user to review the status and results of steps within the build, and optionally unblock it.

Requiring approval in a CI/CD pipeline instead of relying solely on branch protection rules can be more adaptable and situationally responsive. There are scenarios where it might be preferable to block a CI/CD pipeline and require manual approval. These include:

Sensitive infrastructure changes: Certain modifications, such as those affecting security groups, databases, or critical services, can have broad implications. For these high-stakes changes, an additional layer of manual oversight in the CI/CD pipeline, regardless of the branch being used, can prevent potential mishaps.
Resource cost implications: If a change might spin up numerous resources or higher-tiered services that could increase costs substantially, having an approval step can help teams manage and be aware of budgetary impacts.
Production deployments: When pushing changes to production environments, extra caution is often warranted. Even if code has passed all branch-related checks, a final approval before production deployment ensures that changes are validated in the context of the production environment.
Stateful operations: Operations that can cause data loss, like database migrations or schema changes, may benefit from a pipeline approval process. This ensures a deliberate confirmation step before proceeding with potentially irreversible actions.
Complex multi-service deployments: For changes spanning multiple services or microservices, coordinating deployments can be challenging. An approval step can allow teams to ensure that interdependencies are correctly managed.
Regulatory and compliance concerns: For organizations subject to regulatory requirements, having a manual approval step can assist in maintaining compliance, ensuring that changes align with legal and industry standards.
Incident response and outages: During active incidents or outages, it might be necessary to halt all deployments temporarily. Even if the code change isn't directly related to the ongoing issue, introducing new changes could complicate mitigation efforts.
Maintenance windows: Some organizations have predetermined maintenance windows where changes are allowed. Outside of these windows, deployments might be blocked, requiring manual approval to proceed.
Validation of external resources: If a change relies on external APIs, datasets, or services, you might want to have an approval gate to ensure these external resources are available, updated, and working as expected.

Branch protection rules are valuable for enforcing coding standards and collaborative practices. However approvals that are embedded as part of the CI/CD workflow provide a more flexible way to address real-time operational, financial, and security concerns.

Buildkite has block steps, these can be used at any point in the middle of a build to conditionally block a build from proceeding using custom logic. For example, a build provisioning new infrastructure can be blocked (or paused) following the terraform plan or terraform apply in Staging, and only be progressed to terraform apply in Production following a review and approval by a particular user. Block steps, when used together with dynamic pipelines, can provide a framework for deciding the next course of action to take, based on user input.

Managing CI/CD with Terraform

Terraform isn't just for orchestrating application resources and infrastructure; it can also be used for managing CI/CD pipelines, and other resources in your CI/CD platform. While managing the CI/CD platform is often out of scope for an individual project, utilizing Terraform in this capacity allows for a unified, version-controlled approach to both infrastructure and CI/CD configuration.
Buildkite has an official provider in the Terraform Registry that can be used to configure and manage pipelines, teams, clusters, and more.

Managing CI/CD resources with Terraform - Demo - YouTube

Discover how to use Terraform to manage a build and deploy stack. This demo show you how to:– Integrate and manage Github and Buildkite– Create and manage Bu...

youtube.com

Conclusion

There are heaps of best practices, tools, and strategies available for working with Terraform, but it's important to remember that it's not always necessary —or feasible— to adopt them all at once.

I've explored a number of practices and configurations that I hope will help enhance your Terraform workflows. Remember, starting simple is perfectly acceptable and honestly, advisable too! Don't abstract or over-engineer too early, let your needs, and your project's requirements guide the evolution of your Terraform CI/CD workflows. Every team and project is unique, so it's vital to choose practices that work for you! Have fun!

Originally published on the Buildkite blog

Using GitHub's pull request merge queue in your CI pipelines

Mel Kaulfuß 👩🏻‍🦰💻✌️ — Tue, 28 Feb 2023 01:32:47 +0000

There's been quite an audible buzz of excitement about GitHub's pull request merge queues from our users recently. The merge queue pattern increases the efficiency and stability of software delivery workflows by automatically handling pull request merges into your busiest target branches. Merge queues ensure that pull requests get merged in the correct order, and significantly reduce CI workloads, and potential delivery blockers.

Prior to merge queue, merging pull requests into main involved:

Merging main into the pull request branch, so changes wouldn’t break main when merged
Kicking off a build in CI (for these newly pulled and merged changes)
Discovering another PR was merged before the build finished
Rinse and repeat 🌀😵‍💫
Finally merging the PR into main

I’d often be rushing to get my PR merged before someone else snuck theirs through, or I’d forget to pull the latest changes from main into my feature branch before pushing to main. Often, my PR merge would fail, because it wasn’t in sync with the main branch. I saw red–quite literally.

Merge queues automatically build pull request branches with the most recent change (or PR) to be merged ahead of it in the queue. 🙌

The merge queue isn’t new – Uber and Shopify have been using them for close to five years in their pipelines in order maintain high throughput and low commit turnaround time on their busy main branches (think thousands of merges to main per day!). If you have lots of contributors, and high volume of pull requests going to main and if you have frequent flaky merges blocking subsequent commits and creating a backlog of builds, you might find pull request merge queue to be a useful tool,as it minimizes merges tripping over each other on the way to main. There’s some good news if you’re a GitHub Enterprise or open source organization, because you can easily play with it now too: merge queue is now available in public beta.

Using merge queue in your Buildkite pipelines

Include the gh-readonly-queue/{base_branch}/* filter pattern in Pipeline Settings → GitHub → Branch Limiting.

Check the newly-released “Skip builds with existing commits” checkbox in your Pipeline Settings → GitHub → GitHub Settings. This will ensure that you will no longer kick off any unnecessary builds (because there's a green build on the commit ahead of your PR in the merge queue). 🤯

The efficiency merge queue offers is exciting, though you should keep in mind:

This is an experimental feature and GitHub has flagged that it’s likely to change
Configuring branch protection and required checks don’t currently work, and are on our radar in the next couple of months

There's more about how to use it in the Buildkite docs, you can obviously also use merge queue in GitHub Actions, and in some other CI/CD providers, so go try merge queues out – I'd love to hear what you think!

Managing Buildkite CI/CD resources with Terraform

Mel Kaulfuß 👩🏻‍🦰💻✌️ — Mon, 21 Nov 2022 23:20:57 +0000

Those responsible for managing infrastructure and associated user accounts & permissions across a large team understand the importance of having a central place to manage these things. They’ll also understand that the ease with which they’re able to manage these things has a direct correlation with potential security risks and scalability. Eliminating manual tasks is a core principle of SRE, especially when it reduces cost, and automation improves how quickly we can adapt to change; in technologies, environments, and integrations with third-party tools, and reduces the chance of human error.

Infrastructure as Code exists as a way to improve efficiency, remain nimble and adaptable to change. It can also be the key to fostering DevOps practices in engineering teams – allowing developers to better understand and be involved in defining configuration as part of their software development processes.

I probably don't need to tell you that defining configuration as code; provides consistency and clearly documents conventions, reduces the chance of introducing errors and allows infrastructure changes to be tested just like application code.

It also offers all the benefits that using a source control manager provides:

Centralised read/write and team permissions
Rollbacks and versioning
A robust commit and review process on PRs
Surfaces configuration details for others to learn from and contribute to.

It’s no surprise that our users also want to manage their CI/CD resources with Infrastructure as Code.

Introducing the Buildkite Terraform provider

I recently caught up with Chloe Hutchison from Elastic to talk about how they use Terraform internally to manage CI/CD.

“The less you have to think about as a software engineer the better. We try to make magic like that happen everywhere as much as possible.” And that’s why we use the Terraform provider…It helps us reinforce best practices and standards. And by defining pipelines via Terraform, it helps the CI team make sure that everybody's using pipelines defined in the repository, and away from the Buildkite UI in terms of making steps and pipelines – which we think is the best pattern because it's next to their code and keeps everything consistent between teams. We tell them: Don't think about that; just use this.” –– Chloe Hutchison, CI Team Lead & Principal SRE, Elastic

Many of our users rely on Terraform to manage their infrastructure and over the years have built and used their own custom Buildkite providers. We now have an official Buildkite provider in the Terraform Registry that is built and fully supported by us. Our users are now better able to manage their Buildkite CI/CD resources as infrastructure as code.

The official Buildkite provider allows users to configure and manage:

Agent tokens
Pipelines
Pipeline schedules
Teams

Chloe says that the team at Elastic “doesn't like to use 'click-ops' at all if we can avoid it…we had things that were provisioned by 'click-ops' before there were good Terraform providers available and it's a nightmare to manage; changes get lost, people get confused, and nothing is consistent.”

Elastic’s CI team uses the Buildkite Terraform provider via the Cloud Development Kit for Terraform (CDKTF). CDKTF allows users to use familiar programming languages like TypeScript, Go, .NET, Python, and Java to define and provision infrastructure, giving users access to the Terraform ecosystem while also leveraging their existing toolchains.

Chloe describes the benefits of CDKTF as being, “a programming layer on top of Terraform which allows us to use multiple providers, and do a bit of fancy code around it…I like doing it that way because it means that if there are any limitations with one of the providers that we're using, we can just code around them.”

“Our teams are using it to set up repos, CI pipelines, team memberships, and to specify Access Control Lists (ACLs). We have a data structure that teams can use to say “Give me a whole new project”. It goes and creates the GitHub repository, the Vault secrets (to be used in CI), Buildkite pipelines, and does a bunch of stuff in the background so that teams get a secure environment for each Buildkite agent they run. It can even set up scheduled builds”, says Chloe.

Check out this demo to learn how to use Terraform to configure remote state, include official providers, create a Buildkite pipeline, trigger builds automatically once a commit is pushed to GitHub, and configure an Elastic Stack of autoscaling Buildkite agents in AWS.

Additional resources

For more information on using Terraform with Buildkite, check out these resources:

Terraform Provider Docs
Managing CI/CD Resources with Terraform talk demo repository
Official Buildkite Provider GitHub repo
Terraform Techniques with Buildkite, Samuel Cochran, 2016
Join us at Unblock (our annual user conference) to see more Buildkite and CI/CD related content!