Managing Buildkite CI/CD resources with Terraform

Those responsible for managing infrastructure and associated user accounts & permissions across a large team understand the importance of having a central place to manage these things. They’ll also understand that the ease with which they’re able to manage these things has a direct correlation with potential security risks and scalability. Eliminating manual tasks is a core principle of SRE, especially when it reduces cost, and automation improves how quickly we can adapt to change; in technologies, environments, and integrations with third-party tools, and reduces the chance of human error.

Infrastructure as Code exists as a way to improve efficiency, remain nimble and adaptable to change. It can also be the key to fostering DevOps practices in engineering teams – allowing developers to better understand and be involved in defining configuration as part of their software development processes.

I probably don't need to tell you that defining configuration as code; provides consistency and clearly documents conventions, reduces the chance of introducing errors and allows infrastructure changes to be tested just like application code.

It also offers all the benefits that using a source control manager provides:

• Centralised read/write and team permissions
• Rollbacks and versioning
• A robust commit and review process on PRs
• Surfaces configuration details for others to learn from and contribute to.

It’s no surprise that our users also want to manage their CI/CD resources with Infrastructure as Code.

Introducing the Buildkite Terraform provider

I recently caught up with Chloe Hutchison from Elastic to talk about how they use Terraform internally to manage CI/CD.

“The less you have to think about as a software engineer the better. We try to make magic like that happen everywhere as much as possible.” And that’s why we use the Terraform provider…It helps us reinforce best practices and standards. And by defining pipelines via Terraform, it helps the CI team make sure that everybody's using pipelines defined in the repository, and away from the Buildkite UI in terms of making steps and pipelines – which we think is the best pattern because it's next to their code and keeps everything consistent between teams. We tell them: Don't think about that; just use this.” –– Chloe Hutchison, CI Team Lead & Principal SRE, Elastic

Many of our users rely on Terraform to manage their infrastructure and over the years have built and used their own custom Buildkite providers. We now have an official Buildkite provider in the Terraform Registry that is built and fully supported by us. Our users are now better able to manage their Buildkite CI/CD resources as infrastructure as code.

The official Buildkite provider allows users to configure and manage:

• Agent tokens
• Pipelines
• Pipeline schedules
• Teams

Chloe says that the team at Elastic “doesn't like to use 'click-ops' at all if we can avoid it…we had things that were provisioned by 'click-ops' before there were good Terraform providers available and it's a nightmare to manage; changes get lost, people get confused, and nothing is consistent.”

Elastic’s CI team uses the Buildkite Terraform provider via the Cloud Development Kit for Terraform (CDKTF). CDKTF allows users to use familiar programming languages like TypeScript, Go, .NET, Python, and Java to define and provision infrastructure, giving users access to the Terraform ecosystem while also leveraging their existing toolchains.

Chloe describes the benefits of CDKTF as being, “a programming layer on top of Terraform which allows us to use multiple providers, and do a bit of fancy code around it…I like doing it that way because it means that if there are any limitations with one of the providers that we're using, we can just code around them.”

“Our teams are using it to set up repos, CI pipelines, team memberships, and to specify Access Control Lists (ACLs). We have a data structure that teams can use to say “Give me a whole new project”. It goes and creates the GitHub repository, the Vault secrets (to be used in CI), Buildkite pipelines, and does a bunch of stuff in the background so that teams get a secure environment for each Buildkite agent they run. It can even set up scheduled builds”, says Chloe.

Check out this demo to learn how to use Terraform to configure remote state, include official providers, create a Buildkite pipeline, trigger builds automatically once a commit is pushed to GitHub, and configure an Elastic Stack of autoscaling Buildkite agents in AWS.

Additional resources

For more information on using Terraform with Buildkite, check out these resources:

• Terraform Provider Docs
• Managing CI/CD Resources with Terraform talk demo repository
• Official Buildkite Provider GitHub repo
• Terraform Techniques with Buildkite, Samuel Cochran, 2016
• Join us at Unblock (our annual user conference) to see more Buildkite and CI/CD related content!