The Ops Community ⚙️: briancaffey

My Infrastructure as Code Rosetta Stone - Deploying the same web application on AWS ECS Fargate with CDK, Terraform and Pulumi

briancaffey — Sat, 07 Jan 2023 20:40:55 +0000

tl;dr

I wrote three infrastructure as code libraries for deploying containerized 3-tier web apps on AWS ECS Fargate using CDK, Terraform and Pulumi. This article will provide an overview of my experience working with these three IaC tools and will show how I use my libraries in automated infrastructure deployment pipelines with GitHub Actions.

CDK Construct Library: github.com/briancaffey/cdk-django
Terraform Modules: github.com/briancaffey/terraform-aws-django
Pulumi Component Library: github.com/briancaffey/pulumi-aws-django
Mono repo with a sample Django micro blogging app (μblog) and frontend app (Vue SPA written with Quasar), GitHub Action workflows for infrastructure and (separate) application deployment pipelines, IaC code that consumes each of the libraries listed above, VuePress documentation site and miscellaneous items (k6 load testing scripts, Cypress tests, docker-compose, etc.): github.com/briancaffey/django-step-by-step

eli5

Pretend we are at the beach building sandcastles. We can build sandcastles using our hands, but this takes a lot of time, and we might bump into each other and accidentally knock over part of our sandcastle. I made some tools for building sandcastles. We have one tool for building a sand castle base that includes the wall around the outside, the moat, the door and different sections inside the walls. And I made another tool for deploying smaller sand castle houses inside the walls of the sandcastle base. We fill the tool with sand and water and then turn it over inside of our base and we can build an entire city of sandcastles. Also, the tool lets us carefully remove parts of our sandcastle without knocking over any of the other parts. We can share the tool with all of our friends and they can make cool sandcastles too, and the tool is free for them to use.

Instead of sandcastles, I'm working with computer systems that can power internet applications, like YouTube for example. I'm building tools that can allow me or anyone else to build really awesome internet applications using computers.

The tools are not physical tools like the ones for building sandcastles, but instead these tools are made with code. The code for websites like YouTube allow you upload videos to YouTube, but the code I'm writing allows you to upload any type of website (even on like YouTube) to the internet. When we run this code, it creates applications on the internet. Also, sand is very expensive and Jeff Bezos owns the beach.

Why I made an Infrastructure as Code Rosetta Stone with CDK, Terraform and Pulumi

To push myself to learn more about AWS, IaC, CI/CD, automation and Platform Engineering

Learn differences between major IaC tools and how to use them to do exactly the same thing (build a web app) on the same Cloud (AWS) in the same way (serverless container technology using ECS Fargate).
Get more experience publishing software packages (npm) and finding the right level of abstraction for IaC libraries that is both dynamic and straightforward

To fail as many times as possible

Every time I fail when I think I have things right, I learn something new
Failed IaC pipelines can sometimes be scary, and every failure I have on these project can teach me about potential failure modes for live projects running in production
You can often times be "stuck" where you have a set of resources that you can't update or delete. Learning to get unstuck from these scenarios is important

To take an application-first approach to DevOps

Application developers are increasingly being tasked with operational duties
While learning about IaC, I had a hard time finding in-depth materials covering application development, CI/CD pipelines and automation and Infrastructure as Code and how these three knowledge domains work together. There are important considerations to make when between a Hello World docker image
You could probably use another framework with these IaC libraries like Flask or Rails, but for now I'm building these projects with Django first-in-mind

To develop a project I can reference when helping myself and others

companies and projects that do IaC and CI/CD for the most part have things in private repos for obvious reasons, there isn't any good reason to share this type of code unless you are sharing it with an auditor
Hopefully the sample application, IaC and CI/CD pipelines aren't overly complex. There are more complex examples of open source companies out there, but their repos have steep learning curves and a lot going on
People often ask about how to split up IaC deployments and application deployments. I want to be able to use this project to show people how it can be done

To encourage others (specifically Developer Advocates / Developer Relations / Solutions Architects in the CDK, Terraform and Pulumi communities) to share complete and non-trivial examples of IaC software **in use with an actual application.**

There are many ways one could create "IaC Rosetta Stone" (public cloud providers x CI/CD providers x IaC tools is a big number)
This takes a lot of effort and time

I have nothing to sell you

So many articles about Cloud/DevOps are trying to sell you a tool. Outside of what I consider to be mainstream vendors like GitHub and AWS, there are no products that I'm promoting here
I'm also not trying to sell anyone on using my IaC packages
Hopefully my IaC packages can serve as helpful reference or starting point

Walk before running

I want to build up confidence with vanilla use cases before getting too fancy
With a solid foundation in these tools, I want to learn about some of the more advanced patterns teams are adopting (Pulumi Automation API, Terragrunt for Terraform, self-mutating CDK Pipelines)

12 Factor App, DevOps and Platform Engineering

12 Factor App is great, and has guided how I approach both Django application development and IaC library development
The platformengineering.org community has some good guiding principles

CDK/Terraform/Pulumi terminology

constructs, modules and components

A CDK construct, Terraform module and Pulumi component generally mean the same thing: an abstract grouping of one or more cloud resources.

In this article I will refer to constructs/modules/components as c/m/c for short, and the term stack can generally be used to refer to either a CloudFormation stack, a Pulumi Stack or a Terraform group of resources that are part of a module that has had apply ran against it.

what is a stack?

AWS has a resource type called CloudFormation Stacks, and Pulumi also has a concept of stacks. Terraform documentation doesn't refer to stacks, and instead in Terraform docs use the words "Terraform configuration" to refer to some group of resources that were built using a module.

CDK Constructs and Pulumi Components are somewhat similar, however CDK Constructs map to CloudFormation and the Pulumi components I'm using from the @pulumi/aws package generally map directly to Terraform resources from the AWS Provider (the Pulumi AWS Provider uses much of the same code that the Terraform AWS Provider uses).

verbs

In CDK you synth CDK code to generate CloudFormation templates. You can also run diff to see what changes would be applied during a stack update.

In Terraform you init to download all providers and modules. This is sort of like running npm install in CDK and Pulumi. You then run terraform plan to see the changes that would result. terraform apply does CRUD operations on your cloud resources.

In Pulumi you run pulumi preview to see what changes would be made to a stack. You can use the --diff flag to see the specifics of what would change.

To summarize:

In CDK you synth CloudFormation and use these templates to deploy stacks made up of constructs. An "app" can contain multiple stacks, and you can deploy one or more stacks in an app at a time
In Terraform you plan a configuration made up of modules, and then run terraform apply to build the configuration/stack (discuss.hashicorp.com/t/what-is-a-terraform-stack/31985)
Pulumi: You preview a Pulumi stack made up of components, and then run pulumi up to build the resources
To tear down a stack in all three tools, you run destroy

Infrastructure as Code library repos

Let's look at the three repos that I wrote for deploying the same type of 3-tier web application to AWS using ECS Fargate.

Language

cdk-django and pulumi-aws-django are both written in TypeScript. terraform-aws-django is written in HCL, a domain specific language created by HashiCorp. The cdk-django is published to both npm and PyPI, so you can use it in JavaScript, TypeScript and Python projects, other languages are supported as well, but you need to write your library in TypeScript so it can be transpiled to other languages using jsii.

My Pulumi library is written in TypeScript and is published to NPM. For now it can only be used in JavaScript and TypeScript projects. There is a way in Pulumi to write in any language and then publish to any other major language, but I haven't done this yet. See this GitHub repo for more information on this.

The HCL is pretty simple when you get used to it. I find that I don't like adding lots of logic in Terraform code because it takes away from the readability of a module. There is a tool called CDKTF which allows you to write HCL Terraform in TypeScript, but I haven't used it yet.

Release management, versioning and publishing

pulumi-aws-django and terraform-aws-django both use release-please for automatically generating a changelog file and bumping versions. release-please is an open source tool from Google that they use to version their Terraform GCP modules. Whenever I push new commits to main, a new PR is created that adds changes to the CHANGELOG.md file, bumps the version of the library in package.json and adds a new git tag (e.g. v1.2.3) based on commit messages.

cdk-django uses projen for maintaining the changelog and bumping versions and publishing to npm. It is popular among developers in the CDK community and is a really awesome tool since it basically uses one file (.projenrc.ts) to configure your entire repo, including files like tsconfig.json, package.json, and even GitHub Action workflows. It has a lot of configuration options, but I'm using it in a pretty simple way. It generates a new release and items to the changelog when I manually trigger a GitHub Action.

These tools are both based on conventional commits to automatically update the Changelog file.

I'm still manually publishing my pulumi-aws-django package from the CLI. I need to add a GitHub Action to do this for me. This and other backlog items are listed at the end of the article!

Makefile, examples and local development

Each repo has a Makefile that includes commands that I frequently use when developing new features or fixing bugs. Each repo has commands for the following:

synthesizing CDK to CloudFormation / running terraform plan / previewing pulumi up for both the base and app stacks
creating/updating an ad hoc base stack called dev
destroying resources in the ad-hoc base stack called dev
creating an ad hoc app stack called alpha that uses resources from dev
destroying an ad hoc app stack called alpha that uses resources from dev
creating/updating a prod base stack called stage
destroying resources in the prod base stack called stage
creating a prod app stack using called stage that uses resources from the stage base stack
destroying resources in the prod app stack called stage

Here's an example of what these commands look like in pulumi-aws-django for prod infrastructure base and app stacks:

prod-base-preview:  build
    pulumi -C examples/prod/base --stack stage --non-interactive preview

prod-base-up:   build
    pulumi -C examples/prod/base --stack stage --non-interactive up --yes

prod-base-destroy:  build
    pulumi -C examples/prod/base --stack stage --non-interactive destroy --yes

prod-app-preview:   build
    pulumi -C examples/prod/app --stack stage --non-interactive preview

prod-app-preview-diff:  build
    pulumi -C examples/prod/app --stack stage --non-interactive preview --diff

prod-app-up:    build
    pulumi -C examples/prod/app --stack stage --non-interactive up --yes

prod-app-destroy:   build
    pulumi -C examples/prod/app --stack stage --non-interactive destroy --yes

I currently don't have tests for all of these libraries, but for now the most effective way of testing that things are working correctly is to use the c/m/cs to create environments and smoke check the environments to make sure everything works correctly.

Adding unit tests is another item for the backlog.

ad-hoc vs prod

the last article I wrote was about ad hoc environments. Also known as "on-demand" environments or "preview" environments.
the motivation for using ad-hoc environments is speed and cost (you can stand up an environment in less time and you share the costs of the base environment, including VPC, ALB, RDS)
you can completely ignore "ad-hoc" environments and use the "prod" infrastructure for any number of environments (such as dev, QA, RC, stage and prod)
prod can be used for a production environment and any number of pre-production environments
multiple environments built with "prod" infrastructure can be configured with a "knobs and dials" (e.g., how big are app and DB instances, how many tasks to run in a service, etc.)
the "prod" infrastructure should be the same for the "production" environment and the "staging" environment

Directory structure

The directory structures for each repo are all similar with some minor differences.

There are two types of environments: ad-hoc and prod. Within ad-hoc and production, there are two directories base and app.

Each repo has a directory called internal which contain building blocks used by the c/m/cs that are exposed. The contents of the internal directories are not intended to be used by anyone who is using the libraries.

CDK construct library repo structure

~/git/github/cdk-django$ tree -L 4 -d src/
src/
├── constructs
│   ├── ad-hoc
│   │   ├── app
│   │   └── base
│   ├── internal
│   │   ├── alb
│   │   ├── bastion
│   │   ├── customResources
│   │   │   └── highestPriorityRule
│   │   ├── ecs
│   │   │   ├── iam
│   │   │   ├── management-command
│   │   │   ├── redis
│   │   │   ├── scheduler
│   │   │   ├── web
│   │   │   └── worker
│   │   ├── rds
│   │   ├── sg
│   │   └── vpc
│   └── prod
│       ├── app
│       └── base
└── examples
    └── ad-hoc
        ├── app
        │   └── config
        └── base
            └── config

Terraform module library repo structure

~/git/github/terraform-aws-django$ tree -L 4 -d modules
modules
├── ad-hoc
│   ├── app
│   └── base
├── internal
│   ├── alb
│   ├── autoscaling
│   ├── bastion
│   ├── ecs
│   │   ├── ad-hoc
│   │   │   ├── celery_beat
│   │   │   ├── celery_worker
│   │   │   ├── cluster
│   │   │   ├── management_command
│   │   │   ├── redis
│   │   │   └── web
│   │   └── prod
│   │       ├── celery_beat
│   │       ├── celery_worker
│   │       ├── cluster
│   │       ├── management_command
│   │       └── web
│   ├── elasticache
│   ├── iam
│   ├── rds
│   ├── route53
│   ├── s3
│   ├── sd
│   └── sg
└── prod
    ├── app
    └── base

Pulumi component library repo structure

~/git/github/pulumi-aws-django$ tree -L 3 src/
src/
├── components
│   ├── ad-hoc
│   │   ├── README.md
│   │   ├── app
│   │   └── base
│   └── internal
│       ├── README.md
│       ├── alb
│       ├── bastion
│       ├── cw
│       ├── ecs
│       ├── iam
│       ├── rds
│       └── sg
└── util
    ├── index.ts
    └── taggable.ts

Pulumi examples directory

~/git/github/pulumi-aws-django$ tree -L 3 examples/
examples/
└── ad-hoc
    ├── app
    │   ├── Pulumi.alpha.yaml
    │   ├── Pulumi.yaml
    │   ├── index.ts
    │   ├── node_modules
    │   ├── package-lock.json
    │   ├── package.json
    │   └── tsconfig.json
    └── base
        ├── Pulumi.yaml
        ├── bin
        ├── index.ts
        ├── package-lock.json
        ├── package.json
        └── tsconfig.json

CLOC

Let's use CLOC (count lines of code) to compare the lines of code used in the c/m/c of CDK/CloudFormation/Terraform/Pulumi.

cdk-django

~/git/github/cdk-django$ cloc src/constructs/
      14 text files.
      14 unique files.
       0 files ignored.

github.com/AlDanial/cloc v 1.94  T=0.04 s (356.1 files/s, 30040.9 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
TypeScript                      13            155             59            908
Python                           1             18              8             33
-------------------------------------------------------------------------------
SUM:                            14            173             67            941
-------------------------------------------------------------------------------

terraform-aws-django

~/git/github/terraform-aws-django$ cloc modules/
      68 text files.
      58 unique files.
      11 files ignored.

github.com/AlDanial/cloc v 1.94  T=0.15 s (385.9 files/s, 20585.1 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
HCL                             55            472            205           2390
Markdown                         3              7              0             20
-------------------------------------------------------------------------------
SUM:                            58            479            205           2410
-------------------------------------------------------------------------------

pulumi-aws-django

~/git/github/pulumi-aws-django$ cloc src/components/
      15 text files.
      15 unique files.
       0 files ignored.

github.com/AlDanial/cloc v 1.94  T=0.11 s (134.5 files/s, 12924.2 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
TypeScript                      13            110            176           1119
Markdown                         2              6              0             30
-------------------------------------------------------------------------------
SUM:                            15            116            176           1149
-------------------------------------------------------------------------------

Communities

The CDK, Terraform and Pulumi communities are all great and a lot of people helped when I got stuck on issues writing these libraries. Thank you!

μblog

μblog is a micro blogging application that I have written using Django and Vue.js. Here's a screenshot of the homepage:

It is a pretty simple app. Users can write posts with text and an optional images. Logged in users can write posts and like posts.

Mono-repo structure

It lives in a GitHub mono repo called django-step-by-step. This mono repo contains a few different things:

backend Django application
frontend Vue.js application
IaC code that uses c/m/c from cdk-django, terraform-aws-django and pulumi-aws-django
GitHub Actions workflows for both Infrastructure deployments and application deployments

μblog is the reference application that I deploy to infrastructure created with CDK, Terraform and Pulumi. μblog is meant to represent a generic 12 Factor application that uses:

gunicorn for a backend API
Vue.js for a client that consumes the backend API
celery for async task processing
celery beat for scheduling tasks
Postgres for relational data
Redis for caching and message brokering
S3 for object storage
Django admin for a simple admin interface

There is a lot more I could add on μblog. For now I'll just mention that it:

has a great local development environment (supports both docker-compose and virtual environments)
demonstrates how to use Django in different ways. It implements the same application using Function Based View and Class Based Views, and implements both a REST API (both with FBV and CBV) and GraphQL.
GitHub Actions for running unit tests
k6 for load testing
contains a documentation site deployed to GitHub pages (made with VuePress) can be found here: https://briancaffey.github.io/django-step-by-step/

Infrastructure Deep Dive

Let's go through each of the c/m/cs used in the three libraries. I'll cover some of the organizational decisions, dependencies and differences between how things are done between CDK, Terraform and Pulumi.

I'll first talk about the two stacks used in ad hoc environments: base and app. Then I'll talk about the prod environments which are also composed of base and app stacks.

Keep in mind that there aren't that many differences between the ad hoc environment base and app stacks and the prod environment app and base stacks. A future optimization could be to use a single base and app stack, but I think there is a trade-off between readability and DRYness of infrastructure code, especially with Terraform. In general I try to use very little conditionals and logic with Terraform code. It is much easier to have dynamic configuration in CDK and Pulumi, and probably also for other tools like CDKTF (that I have not yet tried).

Splitting up the stacks

While it is possible to put all resources in a single stack with both Terraform, CDK and Pulumi, it is not recommended to do so.

Terraform enables this with outputs and terraform_remote_state
Pulumi encourages the use of micro stacks
CDK has an article on how to create an app with multiple stacks

My design decision was to keep things limited to 2 stacks. Later on it would be interesting to try splitting out another stack.

Also, on-demand environments really lends itself to stacks that are split up.

In the section "Passing unique identifiers", the CDK recommends that we keep the two stacks in the same app. In Terraform and Pulumi, each stack environment is in its own app.

There is a balance to be found between single stacks vs micro stacks. Both the base and app c/m/cs could be split out further. For example, the base c/m/cs could be split into networking and rds. The app stack could be split into different ECS services so that their infrastructure can be deployed independently, like cluster, backend and frontend. The more resources that a stack has, the longer it takes to deploy and the more risky it gets, but adding lots of stacks can add to mental overhead, and pipeline complexity. Each tool has ways of dealing with these complexities (CDK Pipelines, Terragrunt, Pulumi Automation API), but I won't be getting into any of these options in this article. I would like to try these out and share in a future article.

My rules of thumbs are:

single stacks are bad because you don't want to put all your eggs in one basket, however your IaC tool should give you confidence about what is going to change when you try to make a change
Lots of small stacks can cause overhead and make things more complex than they need to be

Ad hoc base overview

Here's an overview of the resources used in an ad hoc base environment.

(Inputs)
(Optional environment configs)
VPC and Service Discovery
S3
Security Groups
Load Balancer
RDS
Bastion Host

Visualization

Here's a dependency graph showing all of the resources in ad hoc base stack. This can be found on the Resources tab of the ad hoc base stack in the Pulumi console.

Inputs

There are only two required inputs for the ad hoc base stack

ACM certificate ARN
Domain Name

I store these values in environment variables for the pipelines in CDK, Terraform and Pulumi. When running pipelines from my local environment, they are exported in my shell before running deploy/apply/up or synth/plan/preview.

VPC

The VPC is the first resource that is created as part of the base stack. There official, high-level constructs in each IaC tool for building VPCs and all related networking resources.

awsx has a VPC module
terraform-aws-vpc module
L2 VPC Construct in CDK

The setting in the Terraform VPC module one_nat_gateway_per_az = false doesn't seem to exist on the awsx.ec2.Vpc module. This will add to cost savings since it will use 1 NAT Gateway instead of 2 or 3.

Security Groups

Pulumi and Terraform can be used in a similar way to define security groups. CDK has a much more concise option for defining ingress and egress rules for security groups.

    const albSecurityGroup = new SecurityGroup(scope, 'AlbSecurityGroup', {
      vpc: props.vpc,
    });

    albSecurityGroup.addIngressRule(Peer.anyIpv4(), Port.tcp(443), 'HTTPS');
    albSecurityGroup.addIngressRule(Peer.anyIpv4(), Port.tcp(80), 'HTTP');

Load Balancer Resources

There's not much to comment on here. In each library I have resource group that defines the following:

Application Load Balancer
A default target group
An HTTP listener that redirects to HTTPS
An HTTPS listener with a default "fixed-response" action

Properties from these resources are used in the "app" stack to build listener rules for ECS services that are configured with load balancers, such as the backend and frontend web services.

Ad hoc app environments all share a common load balancer from the base stack that they use.

RDS Resources

All three libraries have the RDS security group and Subnet Group in the same c/m/c as the RDS instance. The SG and DB Subnet group could alternatively be grouped closer to the other network resources.

Currently the RDS resources are part of the "base" stack for each library. A future optimization may be to break the RDS instance out of the "base" stack and put it in its own stack. The "RDS" stack would be dependent on the "base" stack, and then "app" stack would then be dependent on both the "base" stack and the "RDS" stack. More stacks isn't necessarily a bad thing, but for my initial implementation of these libraries I have decided to keep the "micro stacks" approach limited to only 2 stacks for an environment.

The way that database secrets are handled is another difference between CDK and Terraform and Pulumi. I am currently "hardcoding" the RDS password for Terraform and Pulumi, and in CDK I am using a Secrets Manager Secret for the database credential.

    const secret = new Secret(scope, 'dbSecret', {
      secretName: props.dbSecretName,
      description: 'secret for rds',
      generateSecretString: {
        secretStringTemplate: JSON.stringify({ username: 'postgres' }),
        generateStringKey: 'password',
        excludePunctuation: true,
        includeSpace: false,
      },
    });

In the DatabaseInstance props we can then use this secret like so:

    credentials: Credentials.fromSecret(secret),

In the application deployed with CDK, I use a Django settings module package that uses a package called aws_secretsmanager_caching to get and cache the secrets manager secret for the database, whereas in the apps deployed with Terraform and Pulumi I read in the password from an environment variable.

The Terraform and Pulumi database instance arguments simply accept a password field. This will be another item for the backlog for Terraform and Pulumi. The randompassword and secretversion components can be used to do this.

Bastion Host

There are two main use cases for the bastion host in ad-hoc environments.

When creating a new ad hoc app environment, the bastion host is used to create a new database called {ad-hoc-env-name}-db that the new ad hoc environment will use. (There might be another way of doing this, but using a bastion host is working well for now).
If you using a database management tool on you local machine like DBeaver, the bastion host can help you connect to the RDS instance in a private subnet. The bastion host instance is configured to run a service that forwards traffic on port 5432 to the RDS instance. If you port forward from your local machine to the bastion host on port 5432, you can connect RDS by simple connecting to localhost:5432 on your local machine.

You don't need to manage SSH keys since you connect to the instance in a private subnet using SSM:

aws ssm start-session --target $INSTANCE_ID

Outputs

Here are the outputs for the ad hoc base stack used in Terraform and Pulumi:

vpc_id
assets_bucket_name
private_subnet_ids
app_sg_id
alb_sg_id
listener_arn
alb_dns_name
task_role_arn
execution_role_arn
rds_address

In CDK, the stack references in the app stack don't reference the unique identifiers from the base stack (such as the VPC id or bastion host instance id), but instead they reference the properties of the stack which have types like Vpc and RdsInstance. More on this later in the following section Passing data between stacks.

Ad hoc app overview

The ad hoc app is an group of resources that powers an on-demand environment that is meant to be short lived for testing, QA, validation, demos, etc.

This visualization shows all of the resources in the ad hoc app stack. It also comes from the Pulumi console.

ECS Cluster

This is a small component that defines both ECS Cluster and the default capacity providers
It defaults to not using FARGATE_SPOT; ad hoc environments do use FARGATE_SPOT for cost savings

NOTE: defaultCapacityProviderStrategy on cluster not currently supported. (link)

Shared environment variables

The backend containers should all have the same environment variables, so I define them once in the app stack and pass these into the service resource c/m/cs.

I struggled to get this right in pulumi. A lot of Pulumi examples used JSON.stringify for containerDefinitions in task definitions. I was able to get help from the Pulumi Slack channel; someone recommended that I use pulumi.jsonStringify which was added in a relatively recent version of pulumi/pulumi.
CDK allows you to declare environment variables for a containerDefinition like { FOO: "bar" }
Pulumi and Terraform require that values are passed like { name: "FOO", value: "bar"}
You could transform { FOO: "bar" } into the name/value format, but I didn't bother to do this
extra env vars in Terraform to allow for dynamically passing extra environment variables, and I used the concat function to add these to the list of default environment variables.

Here's what the code looks like for joining extra environment variables to the default environment variables:

    // CDK
    if (extraEnvVars) {
      environmentVariables = { ...extraEnvVars, ...environmentVariables };
    }

    # terraform
    env_vars = concat(local.env_vars, var.extra_env_vars)

    // Pulumi
    if (extraEnvVars) {
      envVars = envVars.apply(x => x.concat(extraEnvVars!))
    }

Route53 Record

This is pretty straightforward in each library. Each ad hoc environment gets a Route 53 record, and listener rules for the web services (Django and Vue.js SPA) match on a combination of the host header and path patterns.

This part is pretty opinionated in that it assumes you want to host the frontend and backend services on the same URL. For example, requests matching example.com/api/* are routed to the backend API and all other requests matching example.com/* are routed to the frontend service.

Redis

I go into more depth about why I run a Redis instance in an ECS service in my other article. This is only for the ad hoc environments. Production environments are configured with ElastiCache running Redis.

I decided to not make this service use any persistent storage. It may be a good idea to not use FARGATE_SPOT for this service, since restarts to the redis service could cause issues in ad hoc environments. For example, you may get a lot of celery errors in ad hoc environments if redis is not reachable.

Web Service

The web service is what defines the main Django application as well as the frontend website (JavaScript SPA or SSR site). I designed the Web Service resources group to be able to support both traditional Django apps (powered by templates), or for Django apps that service only a limited number of endpoints. This c/m/c has an input parameter called pathPatterns which determines which paths it serves. For example, the API container may serve traffic for /api/* and /admin/* only, or it may want to serve all traffic (/*).

The way I use these components in ad hoc and prod environments is heavily opinionated in that:

it assumes that the frontend SPA/SSR site should have a lower priority rule than the backend service and should route request paths matching /*, while the backend service routes requests for a specific list of path patterns (/api/*, /admin/*, /graphql/*, etc.).

You may want Django to handle most of your routes and 404 pages, in which case you would want the SPA to only handle requests matching certain paths. This would require some more consideration and careful refactoring.

Celery

The reason for having a celery service is to be able to have potentially multiple workers that scale independently
I use the same Pulumi component for both works and schedulers

The terminology for this resource group could be better. Celery is one of many options for running async task workers, so it should probably be called something like AsyncWorker across the board rather than using the term celery.

Management Command

Defines a task that can be used to run commands like collectstatic and migrate
These tasks are ran both after the initial app stack deployment and before rolling application upgrades

In my Django app I have a single management command that calls migrate and collectstatic and runs them in the same process one after another. This management command could also be used for clearing caches during updates, loading fixtures, etc.

One other thing to note about this c/m/c is that it outputs a complete script that can be used in GitHub Actions (or on your CLI when testing locally) that does the following:

saves the START timestamp
runs the task with the required settings
waits for the task to complete
saves the END timestamp
collects the logs for the task between START and END and prints them to stdout

Here's an example of what the script looks like in Pulumi:

    const executionScript = pulumi.interpolate`#!/bin/bash
START_TIME=$(date +%s000)
TASK_ID=$(aws ecs run-task --cluster ${props.ecsClusterId} --task-definition ${taskDefinition.arn} --launch-type FARGATE --network-configuration "awsvpcConfiguration={subnets=[${props.privateSubnetIds.apply(x => x.join(","))}],securityGroups=[${props.appSgId}],assignPublicIp=ENABLED}" | jq -r '.tasks[0].taskArn')
aws ecs wait tasks-stopped --tasks $TASK_ID --cluster ${props.ecsClusterId}
END_TIME=$(date +%s000)
aws logs get-log-events --log-group-name ${cwLoggingResources.cwLogGroupName} --log-stream-name ${props.name}/${props.name}/\${TASK_ID##*/} --start-time $START_TIME --end-time $END_TIME | jq -r '.events[].message'
`;
    this.executionScript = executionScript;

In GitHub Actions we get this command as a stack output, save it to a file, make it executable and then run it. This is what it looks like with CDK as a CloudFormation stack output:

      - name: "Run backend update command"
        id: run_backend_update
        run: |
          # get the script from the stack output with an output key that contains the string `backendUpdate`
          BACKEND_UPDATE_SCRIPT=$(aws cloudformation describe-stacks \
            --stack-name $AD_HOC_APP_NAME \
            | jq -r '.Stacks[0].Outputs[]|select(.OutputKey | contains("backendUpdate")) | .OutputValue' \
          )

          echo "$BACKEND_UPDATE_SCRIPT" > backend_update_command.sh
          cat backend_update_command.sh
          sudo chmod +x backend_update_command.sh
          ./backend_update_command.sh

Passing data between stacks

Pulumi uses stack references, Terraform uses remote state and CDK uses Stack Outputs or Stack References.

Here's what this looks like in Terraform

data "terraform_remote_state" "this" {
  backend = "local"

  config = {
    path = "../base/terraform.tfstate"
  }
}

module "main" {
  source = "../../../modules/ad-hoc/app"

  vpc_id                         = data.terraform_remote_state.this.outputs.vpc_id
  assets_bucket_name             = data.terraform_remote_state.this.outputs.assets_bucket_name
  private_subnet_ids             = data.terraform_remote_state.this.outputs.private_subnet_ids
  app_sg_id                      = data.terraform_remote_state.this.outputs.app_sg_id
  alb_sg_id                      = data.terraform_remote_state.this.outputs.alb_sg_id
  listener_arn                   = data.terraform_remote_state.this.outputs.listener_arn
  alb_dns_name                   = data.terraform_remote_state.this.outputs.alb_dns_name
  service_discovery_namespace_id = data.terraform_remote_state.this.outputs.service_discovery_namespace_id
  rds_address                    = data.terraform_remote_state.this.outputs.rds_address
  domain_name                    = data.terraform_remote_state.this.outputs.domain_name
  base_stack_name                = data.terraform_remote_state.this.outputs.base_stack_name
  region                         = var.region
}

In CDK:

const baseStack = new Stack(app, 'ExampleAdHocBaseStack', { env, stackName: adHocBaseEnvName });
baseStack.node.setContext('config', adHocBaseEnvConfig);

const appStack = new Stack(app, 'ExampleAdHocAppStack', { env, stackName: adHocAppEnvName });
appStack.node.setContext('config', adHocAppEnvConfig);

const adHocBase = new AdHocBase(baseStack, 'AdHocBase', { certificateArn, domainName });

const addHocApp = new AdHocApp(appStack, 'AdHocApp', {
  baseStackName: adHocBaseEnvName,
  vpc: adHocBase.vpc,
  alb: adHocBase.alb,
  appSecurityGroup: adHocBase.appSecurityGroup,
  serviceDiscoveryNamespace: adHocBase.serviceDiscoveryNamespace,
  rdsInstance: adHocBase.databaseInstance,
  assetsBucket: adHocBase.assetsBucket,
  domainName: adHocBase.domainName,
  listener: adHocBase.listener,
});

and in Pulumi:

const stackReference = new pulumi.StackReference(`${org}/ad-hoc-base/${environment}`)

const vpcId = stackReference.getOutput("vpcId") as pulumi.Output<string>;
const assetsBucketName = stackReference.getOutput("assetsBucketName") as pulumi.Output<string>;
const privateSubnets = stackReference.getOutput("privateSubnetIds") as pulumi.Output<string[]>;
const appSgId = stackReference.getOutput("appSgId") as pulumi.Output<string>;
const albSgId = stackReference.getOutput("albSgId") as pulumi.Output<string>;
const listenerArn = stackReference.getOutput("listenerArn") as pulumi.Output<string>;
const albDnsName = stackReference.getOutput("albDnsName") as pulumi.Output<string>;
const serviceDiscoveryNamespaceId = stackReference.getOutput("serviceDiscoveryNamespaceId") as pulumi.Output<string>;
const rdsAddress = stackReference.getOutput("rdsAddress") as pulumi.Output<string>;
const domainName = stackReference.getOutput("domainName") as pulumi.Output<string>;
const baseStackName = stackReference.getOutput("baseStackName") as pulumi.Output<string>;

// ad hoc app env
const adHocAppComponent = new AdHocAppComponent("AdHocAppComponent", {
  vpcId,
  assetsBucketName,
  privateSubnets,
  appSgId,
  albSgId,
  listenerArn,
  albDnsName,
  serviceDiscoveryNamespaceId,
  rdsAddress,
  domainName,
  baseStackName
});

CLI scaffolding

CDK and Pulumi have some good options for how to scaffold a project.

Pulumi has pulumi new aws-typescript among lots of other options (run pulumi new -l to see over 200 project types). I used this to create the library itself, the examples and the pulumi projects that I use in django-step-by-step that consume the library.
CDK has projen CLI commands which can help set up either library code or project code
The major benefits of these tools is setting up tsconfig.json and package.json correctly
Terraform is so simple that it doesn't really need tooling for scaffolding

Best practices

For terraform-aws-django, I tried to follow the recommendations from terraform-best-practices.com which helped me a lot with things like consistent naming patterns and directory structures. For example:

use the name this for resources in a module where that resource is the only resource of its type

CDK and Pulumi lend themselves to more nesting and abstractions because they can be written in more familiar programming languages with better abstractions, functions, loops, classes, etc., so there are some differences in directory structure of my libraries when comparing Terraform to both CDK and Pulumi.

For Pulumi and CDK, I mostly tried to follow along with recommendations from their documentation and example projects. While working with Pulumi I struggled a bit with the concepts of Inputs, Outputs, pulumi.interpolate, apply(), all() and the differences between getX and getXOutput. There is a little bit of a learning curve here, but the documentation and examples go a long way in showing how to do things the right way.

Environment configuration

Environment configuration allows for either a base or app stack to be configured with non-default values. For example:

you may decide to start a new base environment but you want to provision a powerful database instance class and size. You would change this using environment configuration
You might want to create an ad hoc app environment but you need it to include some special environment variables, you could set these in environment config.

In the examples above, our IaC can optionally take environment configuration values that overwrite default values, or extend default values.

Pulumi defines environment-specific config in files called Pulumi.{env}.yaml (Pulumi article on configuration)
Terraform uses {env}.tfvars for this type of configuration
CDK has several options for this type of configuration (cdk.context.json, extending stack props, etc.)

For CDK I have been using setContext and the tryGetContext method:

setContext needs to be set on the node before any child nodes are added:

const baseStack = new Stack(app, 'ExampleAdHocBaseStack', { env, stackName: adHocBaseEnvName });
baseStack.node.setContext('config', adHocBaseEnvConfig);

const appStack = new Stack(app, 'ExampleAdHocAppStack', { env, stackName: adHocAppEnvName });
appStack.node.setContext('config', adHocAppEnvConfig);

And the config objects are read from JSON files like this:

var adHocBaseEnvConfig = JSON.parse(fs.readFileSync(`src/examples/ad-hoc/base/config/${adHocBaseEnvName}.json`, 'utf8'));
var adHocAppEnvConfig = JSON.parse(fs.readFileSync(`src/examples/ad-hoc/app/config/${adHocAppEnvName}.json`, 'utf8'));

The context can be used in constructs like this:

    const extraEnvVars = this.node.tryGetContext('config').extraEnvVars;

Pulumi has similar functions for getting context values, here's an example of how I get extra environment variables for app environments using Pulumi's config:

    interface EnvVar {
      name: string;
      value: string;
    }

    let config = new pulumi.Config();
    let extraEnvVars = config.getObject<EnvVar[]>("extraEnvVars");

In my Pulumi.alpha.yaml file I have the extraEnvVars set like this:

config:
  aws:region: us-east-1
  extraEnvVars:
    - name: FOO
      value: BAR
    - name: BIZ
      value: BUZ

I haven't done too much with configuration, but it seems like the right place to build out all of the dials and switches for optional settings in stack resources that you want people to be able to change in their ad hoc environments, or that you want to set per "production" environment (QA, stage, prod, etc.)

Local development

Using the Makefile targets in each library repo, my process for developing c/m/cs involves making code changes followed by Makefile targets that preview/plan/diff against my AWS account, then running deploy/apply/up and waiting for things to finish deploying. Once I can validate that things are looking correct in my account, I run the destroy command and make sure that all of the resources are removed successfully. RDS instances can take up to 10 minutes to create, which means that the base stack takes some time to test. The app environment is able to be spun up quickly, but it can sometimes get stuck and take some time to delete services.

Here are some sample times for deploying ad hoc stacks with CDK.

# CDK ad hoc base deployment time

 ✅  ExampleAdHocBaseStack (dev)

✨  Deployment time: 629.64s

# CDK ad hoc app deployment time

 ✅  ExampleAdHocAppStack (alpha)

✨  Deployment time: 126.62s

Here is an example of what the pulumi preview commands shows for the ad-hoc base stack:

# Pulumi preview
~/git/github/pulumi-aws-django$ pulumi -C examples/ad-hoc/base --stack dev preview
Previewing update (dev)

View Live: https://app.pulumi.com/briancaffey/ad-hoc-base/dev/previews/718625b2-48f5-4ef4-8ed4-9b2694fda64a

     Type                                                    Name                        Plan
 +   pulumi:pulumi:Stack                                     ad-hoc-base-dev             create
 +   └─ pulumi-contrib:components:AdHocBaseEnv               myAdHocEnv                  create
 +      ├─ pulumi-contrib:components:AlbResources            AlbResources                create
 +      │  ├─ aws:alb:TargetGroup                            DefaultTg                   create
 +      │  ├─ aws:alb:LoadBalancer                           LoadBalancer                create
 +      │  ├─ aws:alb:Listener                               HttpListener                create
 +      │  └─ aws:alb:Listener                               HttpsListener               create
 +      ├─ pulumi-contrib:components:BastionHostResources    BastionHostResources        create
 +      │  ├─ aws:iam:Role                                   BastionHostRole             create
 +      │  ├─ aws:iam:RolePolicy                             BastionHostPolicy           create
 +      │  ├─ aws:iam:InstanceProfile                        BastionHostInstanceProfile  create
 +      │  └─ aws:ec2:Instance                               BastionHostInstance         create
 +      ├─ pulumi-contrib:components:RdsResources            RdsResources                create
 +      │  ├─ aws:rds:SubnetGroup                            DbSubnetGroup               create
 +      │  ├─ aws:ec2:SecurityGroup                          RdsSecurityGroup            create
 +      │  └─ aws:rds:Instance                               DbInstance                  create
 +      ├─ pulumi-contrib:components:SecurityGroupResources  SecurityGroupResources      create
 +      │  ├─ aws:ec2:SecurityGroup                          AlbSecurityGroup            create
 +      │  └─ aws:ec2:SecurityGroup                          AppSecurityGroup            create
 +      ├─ aws:s3:Bucket                                     assetsBucket                create
 +      ├─ awsx:ec2:Vpc                                      dev                         create
 +      │  └─ aws:ec2:Vpc                                    dev                         create
 +      │     ├─ aws:ec2:InternetGateway                     dev                         create
 +      │     ├─ aws:ec2:Subnet                              dev-private-1               create
 +      │     │  └─ aws:ec2:RouteTable                       dev-private-1               create
 +      │     │     ├─ aws:ec2:RouteTableAssociation         dev-private-1               create
 +      │     │     └─ aws:ec2:Route                         dev-private-1               create
 +      │     ├─ aws:ec2:Subnet                              dev-private-2               create
 +      │     │  └─ aws:ec2:RouteTable                       dev-private-2               create
 +      │     │     ├─ aws:ec2:RouteTableAssociation         dev-private-2               create
 +      │     │     └─ aws:ec2:Route                         dev-private-2               create
 +      │     ├─ aws:ec2:Subnet                              dev-public-1                create
 +      │     │  ├─ aws:ec2:RouteTable                       dev-public-1                create
 +      │     │  │  ├─ aws:ec2:RouteTableAssociation         dev-public-1                create
 +      │     │  │  └─ aws:ec2:Route                         dev-public-1                create
 +      │     │  ├─ aws:ec2:Eip                              dev-1                       create
 +      │     │  └─ aws:ec2:NatGateway                       dev-1                       create
 +      │     └─ aws:ec2:Subnet                              dev-public-2                create
 +      │        ├─ aws:ec2:RouteTable                       dev-public-2                create
 +      │        │  ├─ aws:ec2:RouteTableAssociation         dev-public-2                create
 +      │        │  └─ aws:ec2:Route                         dev-public-2                create
 +      │        ├─ aws:ec2:Eip                              dev-2                       create
 +      │        └─ aws:ec2:NatGateway                       dev-2                       create
 +      └─ aws:servicediscovery:PrivateDnsNamespace          PrivateDnsNamespace         create


Outputs:
    albDnsName                 : output<string>
    albSgId                    : output<string>
    appSgId                    : output<string>
    assetsBucketName           : output<string>
    baseStackName              : "dev"
    bastionHostInstanceId      : output<string>
    domainName                 : "example.com"
    listenerArn                : output<string>
    privateSubnetIds           : output<string>
    rdsAddress                 : output<string>
    serviceDiscoveryNamespaceId: output<string>
    vpcId                      : output<string>

Resources:
    + 44 to create

Running infrastructure pipelines in GitHub Actions

I don't currently have GitHub Actions working for all tools in all environments, this part is still a WIP but is working at a basic level. Another item for the backlog!

In the .github/workflows directory of the django-step-by-step repo, I will have the following 2 * 2 * 2 * 3 = 24 pipelines for running infrastructure as code pipelines:

{ad_hoc,prod}_{base,app}_{create_update,destroy}_{cdk,terraform,pulumi}.yml

For CDK I'm using CDK CLI commands
For Terraform I'm also using terraform CLI commands
For Pulumi I'm using the official Pulumi GitHub Action

Pulumi has a great article about how to use their official GitHub Action. This action calls the Pulumi CLI under the hood with all of the correct flags.

The general pattern that all of these pipelines use is:

Do a synth/plan/preview, and upload the synth/plan/preview file to an artifact
Pause and wait on manual review of the planned changes
download the artifact and run deploy/apply/up against it, or optionally cancel the operation if the changes you see in the GitHub Actions pipeline logs are not what you expected.

I do this by having two jobs in each GitHub Action: one for synth/plan/preview and one for deploy/apply/up.

The job for deploy/apply/up includes an environment that is configured in GitHub to be a protected environment that requires approvals. Even if you are the only approver (which I am on this project), it is the easiest and safest way preview infrastructure changes before they happen. If you see something in the plan and it isn't what you wanted to change, you cancel the job.

Application deployments

There are two GitHub Actions pipelines for deploying the frontend and the backend. Both of these pipelines run bash scripts that call AWS CLI commands to perform rolling updates on all of the services used in the application (frontend, API, workers, scheduler)
The backend deployment script runs database migrations, the collectstatic command and any other commands needed to run before the rolling update starts (clearing the cache, loading fixtures, etc.)
What is important to note here is that application deployments are not dependent on the IaC tool we use. Since we are tagging things consistently across CDK, Terraform and Pulumi, we can look up resources by tag rather than getting "outputs" of the app stacks.

Interacting with AWS via IaC

CDK interacts directly with CloudFormation (and custom resources which allow for running arbitrary SDK calls and lambda functions) and provides L1, L2 and L3 constructs which offer different levels of abstraction over CloudFormation.
Terraform has the AWS Provider and the terraform-aws-modules.
Pulumi has AWS Classic (@pulumi/aws) provider and AWSx (Crosswalk for Pulumi) library and aws_native provider.

aws_native "manages and provisions resources using the AWS Cloud Control API, which typically supports new AWS features on the day of launch."

aws_native looks like a really interesting option, but it is currently in public preview so I have not decided to use it. I am using the AWSx library only for my VPC and associated resources, everything else uses the AWS Classic provider.

For CDK I use mostly L2 constructs and some L1 constructs.

Fot Terraform I use the VPC from the terraform-aws-modules, and everything else uses the AWS Terraform Provider.

What I did not put in IaC

ECR (Elastic Container Registry)
ACM (Amazon Certificate Manager)
(Roles used for deployments)

I created the Elastic Container Registry backend and frontend repos manually in the AWS Console. I also manually requested an ACM certificate for *.mydomain.com for the domain that I use for testing that I purchased through Route53 domains.

I currently am using another less-than best practice of using Administrative Credentials stored in GitHub secrets. The better approach here is to make roles for different pipelines and use OIDC to authenticate instead of storing credentials. This is another good item for the backlog.

Tagging

Terraform and CDK both make it easy to automatically tag all resources in a stack
It is possible to do this in Pulumi, but you need to write a little bit of code.
https://www.pulumi.com/blog/automatically-enforcing-aws-resource-tagging-policies/
https://github.com/joeduffy/aws-tags-example/tree/master/autotag-ts
Tagging is important since I look up resources by tag in GitHub Actions pipelines (for example, the Bastion Host is looked up by tag)
Automatically tagging resources works through stack transformations are unique to Pulumi

Smoke checking application environments

Here's the list of things I check when standing up an application environment:

[x] Run the init/tsc, synth/plan/preview and deploy/apply/up commands successfully
[x] Access the bastion host (make aws-ssm-start-session)
[x] Run ECSExec to access a shell in a backend container (make aws-ecs-exec)
[x] Test database connectivity (python manage.py showmigrations)
[x] Run the migrations (python manage.py migrate)
[x] Run collectstatic (python manage.py collectstatic)
[x] Visit the site (alpha.example.com)
[x] Publish a blog post
[x] Publish a blog post with an image
[x] Check celery worker logs for successfully complete scheduled tasks
[x] Trigger an autoscaling event by running k6 load tests against an environment
[x] Optionally deploy another backend or frontend image tag using the GitHub Actions pipelines for backend and frontend updates
[x] Destroy the app stack
[x] Destroy the base stack

Backlog and next steps

Here are some of the next things I'll be working on in these project, roughly in order of importance:

Introduce manual approvals in GitHub Actions for all deployments and allow for the previewing or "planning" before proceeding with an live operations in infrastructure pipelines
Switch to using OIDC for AWS authentication from GitHub Actions and remove AWS secrets from GitHub
Show how to do account isolation (different accounts for prod vs pre-prod environments)
GitHub Actions deployment pipeline for publishing pulumi-aws-django package
Complete all GitHub Action deployment pipelines for base and app stacks (both ad hoc and prod)
For Pulumi and Terraform, use a Secrets Manager secret for the database instead of hardcoding it. Use the random functions to do this
Refactor GitHub Actions and make them reusable across different projects
Writing tests for Pulumi and CDK. Figure out how to write tests for Terraform modules
Use graviton instances and have the option to select between different architectures
Standardize all resources names across CDK, Terraform and Pulumi
The Pulumi components that define the resources associated with each ECS service are not very dry
Interfaces could be constructed with inheritance (base set of properties that is extended for different types of services)
Fix the CDK issue with priority rule on ALB listeners. I need to used a custom resource for this which is currently a WIP. Terraform and Pulumi look up the next highest listener rule priority under the hood, so you are not required to provide it, but CDK requires it, which means that you can't do ad hoc environments in CDK without a custom resource that looks up what the next available priority number is.
Make all three of the libraries less opinionated. For example, the celery worker and scheduler should be optional and the frontend component should also be optional
experiment with using a frontend with SSR. This is supported by Quasar, the framework I'm currently using to build my frontend SPA site

If you want to get involved or help with any of the above, please let me know!

Conclusion

I first started out with IaC following this project aws-samples/ecs-refarch-cloudformation (which is pretty old at this point) and wrote a lot of CloudFormation by hand. The pain of doing that lead me to explore the CDK with Python. I learned TypeScript by rewriting the Python CDK code I wrote in TypeScript. I later worked with a team that was more experienced in Terraform and learned how to use that. I feel like Pulumi takes the best of the two tools and has a really great developer experience. There is a little bit of a learning curve with Pulumi, and you give up some of the simplicity of Terraform.

Sharing

Thank you for reading to the end! I have posted the article on the following channels, please like and share the article and follow me wherever you are active. Also please share your feedback!

(links pending)

my personal blog (briancaffey.github.io)
DEV.to
Ops Community
Medium
Substack
My MailChimp mailing list
Hacker News
Twitter
LinkedIn
Facebook
HashNode
Hackernoon
Reddit (r/aws, r/Terraform, r/pulumi, r/django, r/devops)
Terraform Forum
Pulumi Slack channel
CDK.dev Slack channel

Setting up ad hoc development environments for Django applications with AWS ECS, Terraform and GitHub Actions

briancaffey — Sat, 11 Jun 2022 21:40:56 +0000

tl;dr

This article will show how software development teams can build on-demand instances of a web application project for dog-food testing, quality review, internal and external demos and other use cases that require short-lived but feature-complete environments. It will focus on the technical implementation of building ad hoc environments using a specific set of tools (including AWS ECS, Terraform and GitHub Actions). I will also be giving context on high-level implementation decisions based on what I think are best practices guided by the 12-Factor Application methodology. If any of this interests you, please have a read and let me know what you think in the comments on the outlets where I'll be sharing this article (links at the end).

GitHub Links

This article references three open-source code repositories on GitHub.

django-step-by-step
- this repo contains an example microblogging application called μblog built with Django
- the same application is implemented as a traditional Model Template View (MTV) site, a decoupled REST API and Javascript web application and a GraphQL API
- it is a monorepo that also includes a frontend Vue.js application, CI/CD pipelines, a VuePress documentation site as well as tooling and instructions for settings up a local development environments (both with and without docker)
- it includes a complete set of GitHub Action examples for automating the processes of creating, updating and destroying ad hoc environments that will be an important part of what is covered in this article
terraform-aws-django
- a collection of modules for running Django applications on AWS using Terraform
- one of the submodules can be used for creating ad hoc environments which will be what we use to create ad hoc environments
- this module has been published to Terraform Registry and is used in the terraform/live/ad-hoc directory of the django-step-by-step repo
terraform-aws-ad-hoc-environments
- a Terraform module that provides shared infrastructure used by ad hoc environments (including VPC, RDS instance, bastion host, security groups and IAM roles, etc.)
- this module has also been published to Terraform Registry and is also used in django-step-by-step
- this module is designed to be used with the terraform-aws-django Terraform module

Assumptions

There are all sorts of applications, and all sort of engineering teams. For some context on what I'm describing in this article, here are some basic assumptions that I'm making about the type of engineering team and software application product that would be a good fit for this type of development workflow.

engineering team is composed of a backend team, a frontend team, a devops team and works closely with a product team
backend team primarily develops a REST API
frontend team develops a JavaScript SPA (frontend website)
SPA consumes backend REST API
product team frequently needs to demo applications to prospective clients
development teams don't have deep expertise in infrastructure, containers, CI/CD or automation
devops team has been tasked with building automation that will allow anyone on the team to quickly spin up a complete environment for testing and demoing purposes within minutes

Here are assumptions about specific tools and technologies used at the company:

backend is a REST API developed with Django and a Postgres database
backend is packaged into a docker container
frontend is also packaged into a docker container using multi-stage builds and NGINX
frontend does not require any build-time configuration (all configuration needed by frontend is fetched from backend)
backend application's configuration is driven by plain-text environment variables at run-time
engineering team uses AWS
automation pipeline exists for building, tagging and pushing backend and frontend container images to an ECR repository
devops team uses AWS ECS for running containerized workloads
devops team uses Terraform for provisioning infrastructure
devops team uses GitHub Actions for building automation pipelines
team is somewhat cost-conscious

What are ad hoc environments?

Ad hoc environments are short-lived environments that are designed to be used for testing a specific set of features or for demoing a specific application configuration in an isolated environment. It is intended to be a functional duplicate of the main production environment. An ad hoc environment is the first cloud environment that the application code will be deployed to after a developer has been working on it in a local development environment.

Trade-offs to make when designing ad hoc environment infrastructure and automation

Now that we have a sense of what we are building and the team we are working with, let's think about the high-level trade-offs that we will face as we build a solution for providing on-demand ad hoc environments. When building infrastructure and workflows for ad hoc environments, there are a few things to solve for:

simplicity of the end-user interface and process for requesting an ad hoc environment
startup speed
total cost of ownership
degree of similarity to production environments
shared vs isolated resources
automation complexity

Let's look at these items by considering how we can set up the main infrastructure components that will be used to run our ad hoc application environments.

Relational Databases

Startup speed can be measured by the time between when an environment is requested and when that environment can be used by whoever requested it. In this period of time, an automation pipeline may do some of the following:

run terraform init, terraform plan and terraform apply to build infrastructure
run scripts to prepare the application such as database migrations
seeding initial sample data with a script or database dump
message the user with information about the environment (URLs, commands for accessing an interactive shell, etc.)

RDS instances can take a long time to create relative to other AWS resources such as S3 buckets and IAM roles. RDS instances are also more costly than other resources. We could use a single, shared RDS instance placed in a private subnet of a shared VPC. Each ad hoc environment could use a different named database in the RDS instance in the form {ad-hoc-env-name}-db. Using one RDS instance per ad hoc environment would be slow to startup and tear down and also costly if there are many developers using ad hoc environments simultaneously.

If we choose to isolate the application's relational database at the database level (and not the RDS instance level), then we will need our automation workflow to create a database per ad hoc environment.

Let's spin up a simple example to illustrate how this would would work.

A developer is working on feature-abc that involves a significant refactor of the data model.
The developer decides to spin up an ad hoc environment called feature-abc.
Our automation will need to create a database in the RDS instance called feature-abc.
We can configure a bastion host with psql installed that has network and security group access to our RDS instance, and we can give our GitHub Actions an SSH key that can be used to run createdb over SSH.
The automation also runs database migrations once the application has started, and we can view the logs of the database migration to check for any irregularities or other issues.
This will give the developer and the rest of team confidence that the promoting feature-abc to the next pre-production environments will not have any errors.
The developer may even choose to load a SQL dump of the next pre-production environment into their feature-abc database get even more confidence that there will be no data integrity errors.
When the developer's PR is merged and approved, the ad hoc environment feature-abc can be destroyed, including the feature-abc database in the shared RDS instance.

With this approach we won't incur the costs of multiple RDS instances. Ad hoc environments will start up faster because an RDS instance per environment is not required. We do have slightly less resource isolation, and we need to introduce a bastion host, but I consider this an acceptable trade-off.

Redis (key-value database)

Redis is another database used in the application and it plays a few different roles:

primarily, it is a caching layer that can cache request responses to reduce load on the database and speed up our application
it is a message broker for our async task workers (celery)
it can be used as a backend for other 3rd party Django apps that our main application may need to use (such as django-constance, cache-ops, django-channels, etc.)

AWS offers a managed Redis service called ElastiCache. Redis running on an ElastiCache instance can do database isolation similar to how Postgres running on RDS can do database isolation as we discussed previously, but there are some key differences:

redis databases are numbered, not named
the backend application uses isolated numbered databases for the different 3rd party apps that I just mentioned (for example: celery can use database 0, API caching layer can use database 1, etc.)

This makes it difficult to use a single ElastiCache instance for our ad hoc environments since we would need to figure out which numbered database to assign to a specific role for each ad hoc environment (e.g. how do we know which numbered database to use for the API caching for the feature-abc ad hoc environment).

So how can we approach providing isolated redis instances for multiple ad hoc environments? Spoiler: my solution is to run redis as a stateful service in ECS. Before we dig into how to do this, we need to talk about another important part of our application: compute.

Compute

Our backend application is composed of a few different services that all share the same code base. In other words, our backend's services uses the same docker image but run different processes for each component:

gunicorn for the core API
celery for the task workers
celerybeat for task scheduling

If our application used websockets, we could have another service that runs an asgi server process (like daphne or uvicorn).

Since our backend application is packaged into a container and we are using AWS as our cloud provider, ECS is a great choice for running our backend services. ECS is a container orchestration tool that I usually describe as a nice middle ground between docker swarm and Kubernetes. Simply put, it is a flexible option for running our containerized services that make up our backend application.

With ECS you can choose to run containers directly on EC2 instances that you manage, or you can run containers using Fargate. Fargate is a serverless compute option that takes care of managing both the underlying "computer" and operating system that run our application's containers. All of our backend dependencies are defined in our Dockerfile, so we do not to maintain or update the underlying operating system that runs our containers -- AWS handles all of this for us. To use Fargate, we simply tell AWS which containers to run and how much CPU and memory to use in the ECS Task that runs the containers. To scale our app horizontally, the ECS service that managed ECS tasks simply increases the number of tasks that run.

Since we are going to use the Fargate launch type for our ECS Tasks, let's talk about the ergonomics of these serverless compute instances compared to running our services directly on an EC2 instances.

We can't SSH into Fargate compute instances. We can instead use AWS Systems Manager and EcsExec to open an interactive shell in a running backend container. This can be useful for developers who might need to run a management command or access an interactive Django shell to verify behavior in their ad hoc environment.
We can't simply change code on the server and restart services. This can sometimes be a useful pattern for debugging something that can only be tested on a cloud environment (e.g. something that can't easily be reproduced on your local machine), so this requires that developers push new images to their backend services for every change they want to see reflected on their ad hoc environment. Later on I'll discuss how we can provide tooling for developers to quickly update the image used in their backend services.

With AWS Fargate, you will pay more than you would for a comparable amount of CPU and memory on EC2 instances. Similar to EC2 spot instances, Fargate offers interruptable instances called Fargate Spot which costs significantly less than regular Fargate instances. Fargate spot is appropriate for our ad hoc environments since ad hoc environments are non-critical workloads. In the event that a Fargate spot instance is interrupted, the ECS service will automatically launch another Fargate task to replace the task that was stopped.

In my opinion, ECS with Fargate is ideal for running the stateless services that make up our backend application. In terms of parity with our production environment, we can keep almost everything the same, except use regular Fargate instances instead of Fargate spot instances.

Redis, revisited

We can run redis as an ECS service instead of using ElastiCache. In order to do this, we will need our backend services (gunicorn, celery and celerybeat) to be able to communicate with a fourth ECS service that will be running redis (using an official redis image from Docker Hub, or a redis image that we define in ECR).

By default, there is no way for our backend services to know how to communicate with any other service in our ECS cluster. If you have used docker-compose, you may know that you can use the service name redis in a backend service to easily communicate with a redis service called redis. This networking convenience is not available to use out of the box with ECS. To achieve this in AWS, we need some way to manage a unique ad hoc environment-specific Route 53 DNS record that points to the private IP of the Fargate task that is running redis in an ECS cluster for a given ad hoc environment. Such a service exists in AWS and it is called Cloud Map. Cloud Map offers service discovery so that our backend services can make network calls to a static DNS address that will reliably point to the correct private IP of the ECS task running the redis container.

We can define a service discovery namespace (which will essentially be a top level domain, or TLD) that all of our ad hoc environments can share. Let's assume this namespace is called ad-hoc. Each ad hoc environment can then define a service discovery service in the shared namespace for redis that is called {ad-hoc-env-name}-redis. This way, we can have a reliable address that we can configure as an environment for our backend that will look like this: redis://{ad-hoc-env-name}-redis.ad-hoc:6379/0. {ad-hoc-env-name-redis}.ad-hoc will be the hostname of the redis service, and Route 53 will create records that point to {ad-hoc-env-name}-redis.ad-hoc to the private IP of the redis Fargate task for each ad hoc environment.

Load Balancing

We now have our backend services (gunicorn, celery and celerybeat) running on Fargate spot instances, and these services can communicate with the redis service in our ad hoc environment's ECS cluster using service discovery that we configured with Cloud Map. We still need to think about a few things:

how will we expose our API service to the public (or private) internet
how will we expose our frontend application to the public (or private) internet
how will we make sure that requests go the correct ECS services

Application load balancers (ALBs) are a great way to expose web app traffic to the internet. We could either have one application load balancer per ad hoc environment, or one application load balancer shared between all ad hoc environments. ALBs are somewhat slow to create and they also incur a significant monthly cost. They are also highly scalable, so using a shared ALB for all ad hoc environments would work.

Individual ad hoc environments can then create target groups and listener rules for a shared ALB for each service that needs to serve requests from the internet (the backend and the frontend). In our case this is the backend API server and the frontend server that serves our static frontend site using NGINX.

ECS services that need to be exposed to the internet can specify the target group, port and container to use for load balancing. A target group is created that defines the health check and other settings, and a load balancer listener rule is created on the shared load balancer that will forward traffic matching certain conditions to the target group for our service.

For a given ad hoc environment, we need to specify that only traffic with certain paths should be sent to the backend service, and all other traffic should be sent to the frontend service. For example, we may only want to send traffic that starts with the path /api or /admin to the backend target group, and all other traffic should be sent to the frontend target group. We can do this by setting conditions on the listener rules that forward traffic do the frontend and backend target groups based on the hostname and path.

We want our listener rule logic to forward /api, /admin and any other backend traffic to the backend target group, and forward all other traffic (/*) to the frontend target group. In order to do this, we need the backend listener rule to have a higher priority than the frontend listener rule for each ad hoc environment. Since we are using the same load balancer for all ad hoc environments, the priority values for each listener rule need to be unique. If we don't set the priority explicitly, then the priority will be set automatically to the next available value in ascending order. In order to make sure that the backend listener rule has a higher priority than the frontend listener rule for each ad hoc environment, we need to tell Terraform that the frontend module depends_on the backend module. This way the backend listener rule will have a higher priority (e.g. priority of 1) because it will be created first, and the frontend listener rule will have a lower priority (e.g. priority of 2).

More on shared resources vs per-environment resources

Up until now we have discussed infrastructure design decisions at a high level, but we have not yet talked about how to organize our infrastructure as code. At a basic level, components of our ad hoc environment either fall into shared infrastructure or infrastructure that is specific to an individual ad hoc environment. Here's a list of the resources that are shared and the resources that are specific to each ad hoc environment.

Shared resources include:

VPC
IAM policies
Security groups
RDS instance
Service Discovery namespace
Application Load Balancer
Bastion host

Ad hoc environment resources include:

ECS Cluster
ECS Tasks and Services (for backend and frontend applications)
ECS Tasks for running management commands (such as migrate)
CloudWatch logging groups for containers defined in ECS Tasks
ALB Target groups
ALB listener rules
Route 53 record that points to the load balancer (e.g. ad-hoc-env-name.example.com)
S3 bucket for static and media assets
Service Discovery Service for redis service in ECS cluster

Shared resources can be defined in one terraform configuration and deployed once. These resources will be long-lived as long as the application is under active development and the team requires on-demand provisioning of ad hoc environments.

Ad hoc environment resources can be defined in another terraform configuration that references outputs from the shared resource configuration using terraform_remote_state. Each ad hoc environment can be defined by a <name>.tfvars file that contains the name of the ad hoc environment (such as brian, brian2, demo-feature-abc, etc.). This <name> value will also be the name of the Terraform workspace and will be used to name and tag AWS resources associated with the corresponding ad hoc environment.

The <name>.tfvars file will allow developers to use a simple, standard file interface for defining application specific values, such as the version of the backend and frontend. This brings developers into the concepts and practices of "infrastructure as code" and "configuration as code" and also helps the entire team keep track of how different environments are configured.

Ad hoc environment <name>.tfvars files are stored in a directory of a special git repository that also defines the ad hoc environment terraform configuration. Currently, the tfvars files are stored here.

Now let's look at the two terraform configurations used for defining shared resources and ad hoc environment resources.

Ad Hoc Environment Diagram

Here's an overview of the resources used for the ad hoc environments. The letters represent shared resources and the numbers represent per-environment resources.

Shared architecture

A. VPC (created using the official AWS VPC Module)

B. Public subnets for bastion host, NAT Gateways and Load Balancer

C. Private subnets for application workloads and RDS

D. Application Load Balancer that is shared between all ad hoc environments. A pre-provisioned wildcard ACM certificate is attached to the load balancer that is used to secure traffic for load-balanced ECS services

E. Service discovery namespace that provides a namespace for application workloads to access the redis service running in ECS

F. IAM roles needed for ECS tasks to access AWS services

G. RDS instance using postgres engine that is shared between all ad hoc environments

H. Bastion host used to access RDS from GitHub Actions (needed for creating per-environment databases)

I. NAT Gateway used to give traffic in private subnets a route to the public internet

Environment-specific architecture

ECS Cluster that groups all ECS tasks for a single ad hoc environment
Listener rules and target groups that direct traffic from the load balancer to the ECS services for an ad hoc environment.
Redis service running in ECS that provides caching and serves as a task broker for celery
Route53 records that point to the load balancer
Frontend service that serves the Vue.js application over NGINX
API service that serves the backend with Gunicorn
Celery worker that process jobs in the default queue
Celery beat that schedules celery tasks
collectstatic task
migrate task
CloudWatch log groups are created for each ECS task in an ad hoc environment
Each ad hoc environment gets a database in the shared RDS instance

Shared resources terraform configuration

Let's have a detailed look at the terraform configuration for shared resources that will support ad hoc environments.

VPC

We can use the AWS VPC module for creating the shared VPC with Terraform. This module provides a high level interface that will provision lots of the components that are needed for a VPC following best practices, and it is less code for the DevOps team to manage compared to defining each component of a VPC (route tables, subnets, internet gateways, etc.).

Cloud Map Service Discovery Namespace

Cloud Map is used in order to allow services in our ECS cluster to communicate with each other. The only reason that Cloud Map is needed is so that the backend services (API, celery workers, beat) can communicate with Redis, which will be an important service for our application, providing caching and also serving as a broker for celery. If we were to use Django Channels for websockets, the Redis service would also function as the backend for Django Channels.

We will only need to specify service_registries on the redis service in our ECS cluster. What this will do is provide an address that our other services can use to communicate with redis. This address is created in the form of a Route 53 record, and it points to the private IP address of the redis service. If the private IP of the redis service is updated, the Route 53 record record for our redis service will be updated as well.

In order for service discovery to work in the VPC that we created, we need to add the following options to the terraform AWS VPC module:

# DNS settings
enable_dns_hostnames = true
enable_dns_support   = true

Security Groups

There are two important security groups that we will set up as part of the shared infrastructure layer to be used by each ad hoc environment: one security group for the load balancer, and one security group where all of our ECS services will run.

The load balancer security group will allow all traffic on port 80 and 443 for HTTP and HTTPS traffic. The ECS security group will only allow inbound traffic from the application load balancer security group. It will also allow for traffic from port 6379 for redis traffic.

IAM Roles

There are two important IAM roles that we will need for our ECS tasks. We need a task execution role that our ECS tasks will use to interact with other AWS services, such as S3, Secrets Manager, etc.

RDS Instance

We will create one RDS instance in one of the private subnets in our VPC. This RDS instance will have one Postgres database per ad hoc environment. This RDS instance has a security group that allows all traffic from our ECS security group.

Load Balancer

We will use one load balancer for all ad hoc environments. This load balancer will have a wildcard ACM certificate attached to it (*.dev.example.com, for example). Each ad hoc environment will create a Route 53 record that will point to this load balancer's public DNS name. For example, brian.dev.example.com will be the address of my ad hoc environment. Requests to this address will then be routed to either the frontend ECS service or the backend ECS service depending on request header values and request path values that will be set on the listener rules.

By default, a load balancer supports up to 50 listener rules, so we can create plenty of ad hoc environments before we need to increase the default quota. There will be a discussion at the end of this article about AWS service quotas.

Bastion Host

The bastion host will be created in one of the VPC's public subnets. This will primarily be used for connecting to RDS to create new databases for new ad hoc environments, or for manually manipulating data in an ad hoc environment for debugging.

Ad hoc environment resources

Now that we have defined a shared set of infrastructure that our ad hoc environments will use, let's have a look at the resources that will be specific to ad hoc environments that will be added on top of the shared resources.

ECS Cluster

The ECS Cluster is a simple grouping of ECS tasks and services.

ECS Tasks and Services

Each environment will have a set of ECS tasks and services that will be used to run the application.

There are four important ECS services in our application that are used to run "long-running" ECS tasks. Long-running tasks are tasks that start processes that run indefinitely, rather than running until completion. The long-running tasks in our application include:

backend web application (gunciron web server)
backend celery worker
backend celery beat
frontend web site (nginx web server)
redis

The infrastructure code also defines some tasks that are not long-running but rather short lived tasks that run until completion and do not start again. These tasks include:

collectstatic
database migrations
any other ad-hoc task that we want to run, usually wrapped in a Django management command

How to setup an ad hoc environment

Now that we have been over the resources that will be created to support our ad hoc environments, let's talk about how we can enable individuals on our team to create and update ad hoc environments.

Design decisions

The devops team will decide on the interface that will be used for creating an ad hoc environment. Since we are using Terraform, this interface will be a Terraform configuration. The minimum amount of information that our ad hoc environment configuration needs is image tags for the frontend and backend images to use. Other configurations will be provided by default values set in variables.tf, and these defaults can easily be overridden by passing values to terraform plan and terraform apply. I'm choosing to use <name>.tfvars as the way to pass configuration values to our ad hoc environments where <name> is the name of the ad hoc environment being created. This will give us the following benefits:

all ad hoc environments will be visible to the entire team in git since each ad hoc environment will have a <name>.tfvars file associated with it
adding additional customization to an ad hoc environment does not add additional complexity to our automation pipeline since all customization is added through a single file that will be referenced by $WORKSPACE.tfvars

The downsides of this approach are:

creating ad hoc environments requires knowledge of git, so non-technical product team members might need help from the engineering team when setting up an ad hoc environment
there is an additional "manual" step of creating a <name>.tfvars file that must be done before running a pipeline to create an ad hoc environment

Provided that a <name>.tfvars file has been created and pushed to the repo, creating or updating an ad hoc environment will be as simple as running a pipeline in GitHub Actions that specifies the <name> of our ad hoc environment. If no such <name>.tfvars file exists, our pipeline will fail.

GitHub Action

Creating ad hoc environments will involve manually triggering a GitHub Action that runs on workflow_dispatch:

on:
  workflow_dispatch:
    inputs:
      workspace:
        description: 'Name of terraform workspace to use'
        required: true
        default: 'dev'
        type: string

We only have to enter the name of the ad hoc environment we want to create or update. The ad hoc environment name is used as the Terraform workspace name. This name is also the name of the <name>.tfvars file that must be created per environment.

This workflow will do terraform init, terraform plan and terraform apply using the <name>.tfvars file. When everything has been created, we will use the AWS CLI to prepare the environment so that it can be used. We will use the aws ecs run-task command to run database migrations needed so that the application code can make database queries.

How to update code in an existing ad hoc environment

Assuming that we have deployed an ad hoc environment called brian with version v1.0.0 of the backend application and v2.0.0 of the frontend application, let's think about the process of updating the application to v1.1.0 of the backend and v2.1.0 of the frontend.

The simplest approach to updating the application would be edit the brian.tfvars file with the new versions:

# brian.tfvars
be_image_tag = "v1.1.0"
fe_image_tag = "v2.1.0"

If we run the same pipeline that we initially used to deploy ad hoc environment (with terraform init, terraform plan and terraform apply) against the updated brian.tfvars file, this will result in a rolling update of the frontend and backend services (more on rolling updates here).

If there are database migrations included in the new version of the code that is going out, we need to run database migrations after the terraform apply completes. We use a top level output from the ad hov environment terraform configuration that is a run-task command with all appropriate arguments that will run database migrations when called from GitHub Actions.

Order of Operations

For ad hoc environments, it is probably fine to update the services and then run the database migrations. Ad hoc environments may only have a single "user" -- the developer, so we don't need to worry about any errors that may occur if requests are made against the new version of code before database migrations have been applied.

Let's consider a simple example to illustrate what can go wrong here. If we add a total_views to our blog post model to track the total number of page views a post has, we would add a field to the model, generate migration file with makemigrations, and then update our views and model serializers to make use of this new field. In the time between updating our service and running the database migrations, any requests to endpoints that access the new database field will fail since the table does not yet exist.

If we first run database migrations and then update application code (ECS services), then we can avoid errors about fields not existing. In our production application, we want to aim for fewer errors, so we should be using this "order of operations": first run new database migrations and then update application code.

GitHub Action for application updates

We need a GitHub Action that can do the following:

fetch the current container definition JSON files for each backend tasks (aws ecs describe-task-definition)
write new container definitions JSON with the new backend image tag (using jq)
register new task definitions with the new container definition JSON files for each task (aws ecs register-task-definition)
call run-task with the newly updated migration ECS task (aws ecs run-task)
wait for the task to exit and display the logs (aws ecs wait tasks-stopped)
update the backend services (gunicorn, celery, celery beat) (aws ecs update-service)
wait for the new backend services to be stable (aws ecs wait services-stable)

Here's a visual representation of the backend update process:

In order have the correct arguments for all of the AWS CLI calls used in the above workflow, we can use the AWS CLI to fetch resource names by tag.

Here is what I'm using for the script. There lots of comments, so please refer to those comments for an explanation of what the script is doing.

#!/bin/bash

# This script will be called to update an ad hoc environment backend
# with a new image tag. It will first run pre-update tasks (such as migrations)
# and then do a rolling update of the backend services.

# It is called from the ad_hock_backend_update.yml GitHub Actions file

# Required environment variables that need to be exported before running this script:

# WORKSPACE - ad hoc environment workspace
# SHARED_RESOURCES_WORKSPACE - shared resources workspace
# BACKEND_IMAGE_TAG - backend image tag to update services to (e.g. v1.2.3)
# AWS_ACCOUNT_ID - AWS account ID is used for the ECR repository URL

echo "Updating backend services..."

# first define a variable containing the new image URI
NEW_BACKEND_IMAGE_URI="$AWS_ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/backend:$BACKEND_IMAGE_TAG"


# register new task definitions
# https://docs.aws.amazon.com/cli/latest/reference/ecs/describe-task-definition.html#description
for TASK in "migrate" "gunicorn" "default" "beat"
do
  echo "Updating $TASK task definition..."

  # in Terraform we name our tasks based on the ad hoc environment name
  # (also the Terraform workspace name) and the name of the task
  # (e.g. migrate, gunicorn, default, beat)
  TASK_FAMILY=$WORKSPACE-$TASK

  # save the task definition JSON to a variable
  TASK_DESCRIPTION=$(aws ecs describe-task-definition \
    --task-definition $TASK_FAMILY \
  )

  # save container definitions to a file for each task
  echo $TASK_DESCRIPTION | jq -r \
    .taskDefinition.containerDefinitions \
    > /tmp/$TASK_FAMILY.json

  # write new container definition JSON with updated image
  echo "Writing new $TASK_FAMILY container definitions JSON..."

  # replace old image URI with new image URI in a new container definitions JSON
  cat /tmp/$TASK_FAMILY.json \
    | jq \
    --arg IMAGE "$NEW_BACKEND_IMAGE_URI" '.[0].image |= $IMAGE' \
    > /tmp/$TASK_FAMILY-new.json

  # Get the existing configuration for the task definition (memory, cpu, etc.)
  # from the variable that we saved the task definition JSON to earlier
  echo "Getting existing configuration for $TASK_FAMILY..."

  MEMORY=$( echo $TASK_DESCRIPTION | jq -r \
    .taskDefinition.memory \
  )

  CPU=$( echo $TASK_DESCRIPTION | jq -r \
    .taskDefinition.cpu \
  )

  ECS_EXECUTION_ROLE_ARN=$( echo $TASK_DESCRIPTION | jq -r \
    .taskDefinition.executionRoleArn \
  )

  ECS_TASK_ROLE_ARN=$( echo $TASK_DESCRIPTION | jq -r \
    .taskDefinition.taskRoleArn \
  )

  # check the content of the new container definition JSON
  cat /tmp/$TASK_FAMILY-new.json

  # register new task definition using the new container definitions
  # and the values that we read off of the existing task definitions
  echo "Registering new $TASK_FAMILY task definition..."

  aws ecs register-task-definition \
    --family $TASK_FAMILY \
    --container-definitions file:///tmp/$TASK_FAMILY-new.json \
    --memory $MEMORY \
    --cpu $CPU \
    --network-mode awsvpc \
    --execution-role-arn $ECS_EXECUTION_ROLE_ARN \
    --task-role-arn $ECS_TASK_ROLE_ARN \
    --requires-compatibilities "FARGATE"

done

# Now we need to run migrate, collectstatic and any other commands that need to be run
# before doing a rolling update of the backend services

# We will use the new task definitions we just created to run these commands

# get the ARN of the most recent revision of the migrate task definition
TASK_DEFINITION=$( \
  aws ecs describe-task-definition \
    --task-definition $WORKSPACE-migrate \
    | jq -r \
    .taskDefinition.taskDefinitionArn \
)

# get private subnets as space separated string from shared resources VPC
SUBNETS=$( \
  aws ec2 describe-subnets \
    --filters "Name=tag:env,Values=$SHARED_RESOURCES_WORKSPACE" "Name=tag:Name,Values=*private*" \
    --query 'Subnets[*].SubnetId' \
    --output text \
)

# replace spaces with commas using tr
SUBNET_IDS=$(echo $SUBNETS | tr ' ' ',')

# https://github.com/aws/aws-cli/issues/5348
# get ecs_sg_id - just a single value
ECS_SG_ID=$( \
  aws ec2 describe-security-groups \
    --filters "Name=tag:Name,Values=$SHARED_RESOURCES_WORKSPACE-ecs-sg" \
    --query 'SecurityGroups[*].GroupId' \
    --output text \
)

echo "Running database migrations..."

# timestamp used for log retrieval (milliseconds after Jan 1, 1970 00:00:00 UTC)
START_TIME=$(date +%s000)

# run the migration task and capture the taskArn into a variable called TASK_ID
TASK_ID=$( \
  aws ecs run-task \
    --cluster $WORKSPACE-cluster \
    --task-definition $TASK_DEFINITION \
    --network-configuration "awsvpcConfiguration={subnets=[$SUBNET_IDS],securityGroups=[$ECS_SG_ID],assignPublicIp=ENABLED}" \
    | jq -r '.tasks[0].taskArn' \
  )

echo "Task ID is $TASK_ID"

# wait for the migrate task to exit
# https://docs.aws.amazon.com/cli/latest/reference/ecs/wait/tasks-stopped.html#description
# > It will poll every 6 seconds until a successful state has been reached.
# > This will exit with a return code of 255 after 100 failed checks.
aws ecs wait tasks-stopped \
  --tasks $TASK_ID \
  --cluster $WORKSPACE-cluster

# timestamp used for log retrieval (milliseconds after Jan 1, 1970 00:00:00 UTC)
END_TIME=$(date +%s000)

# print the CloudWatch log events to STDOUT
aws logs get-log-events \
  --log-group-name "/ecs/$WORKSPACE/migrate" \
  --log-stream-name "migrate/migrate/${TASK_ID##*/}" \
  --start-time $START_TIME \
  --end-time $END_TIME \
  | jq -r '.events[].message'

echo "Migrations complete. Starting rolling update for backend services..."

# update backend services
for TASK in "gunicorn" "default" "beat"
do

  # get taskDefinitionArn for each service to be used in update-service command
  # this will get the most recent revision of each task (the one that was just created)
  # https://docs.aws.amazon.com/cli/latest/reference/ecs/describe-task-definition.html#description
  TASK_DEFINITION=$( \
    aws ecs describe-task-definition \
      --task-definition $WORKSPACE-$TASK \
      | jq -r \
      .taskDefinition.taskDefinitionArn \
  )

  # update each service with new task definintion
  aws ecs update-service \
    --cluster $WORKSPACE-cluster \
    --service $WORKSPACE-$TASK \
    --task-definition $TASK_DEFINITION \
    --no-cli-pager

done

echo "Services updated. Waiting for services to become stable..."

# wait for all service to be stable (runningCount == desiredCount for each service)
aws ecs wait services-stable \
  --cluster $WORKSPACE-cluster \
  --services $WORKSPACE-gunicorn $WORKSPACE-default $WORKSPACE-beat

echo "Services are now stable. Backend services are now up to date with $BACKEND_IMAGE_TAG."

echo "Backend update is now complete!"

With this GitHub Actions workflow, a developer can now easily change the version of backend code that is running in their ad hoc environments without needing to involve Terraform. Using the ad_hoc_backend_update.yml GitHub Actions workflow, a developer only needs to enter the name of the workspace and the version of the backend code they want to use. The workflow will then run the migrate task and update the backend services.

Using `ignore_changes` in the definitions

There is one more important point to make about Terraform before we conclude this discussion on updating backend code for existing ad hoc environments. Consider the scenario where a developer has launched an ad hoc environment with backend version v1.0.0. They make a small change to the backend code and push version v1.0.1. Next, the developer remembers that a backend environment variable needs to be changed. This can be done by updating their *.tfvars file. If they now re-run the ad hoc environment update pipeline without also updating the backend version in their *.tfvars file, then their code will be reverted from v1.0.1 to v1.0.0. We would need to coordinate version changes between updating the backend with the pipelines that use Terraform commands and the pipelines that use the AWS CLI commands.

There is a setting on the aws_ecs_service resource in Terraform we can can use to prevent this from happening. This setting is called ignore_changes and is defined under the resource's lifecycle configuration block. With this setting, when we update the *.tfvars file with our new environment variable value, we will create another recent task definition with the same v1.0.0 image, but the ECS service will not update in response to this change (that's what the ignore_changes is for). Once we make the *.tfvars file update and redeploy using the Terraform pipeline, nothing on our ad hoc changes, but we did get a new task definitions defined in our AWS account for each backend service. When we go to make the backend update with the pipeline that uses AWS CLI commands, the most recent task revision is used to create the new task definition, so it will include the environment variable change that we added earlier.

Frontend updates

The process described above is needed for updating the backend application. Updating the frontend application involves a similar process to the backend update. The main difference is that no task (such as the migrate command on the backend) needs to run before the service is updated. Here's an overview of the frontend update process:

fetch the container definition JSON for the frontend tasks (aws ecs describe-task-definition)
write new container definition JSON with the new frontend image tag (using jq)
register new task definitions with the new container definition JSON file for the frontend task (aws ecs register-task-definition)
update the frontend service (aws ecs update-service)
wait for the new backend services to be stable (aws ecs wait services-stable)

Setting up everything from a new AWS account and GitHub Actions

Here's a quick overview of initial setup steps that are needed in order to use the automation defined in the GitHub Actions for ad hoc environments.

Configure AWS credentials locally

There is one Terraform command that we will run on our local machine to setup a remote backend for storing our Terraform state. In order to run this, we need to configure AWS credentials locally.

Run the `make tf-bootstrap` command

This command will setup an S3 bucket and DynamoDB table for storing Terraform state. Running this command will also require that a bootstrap.tfvars file has been created from the template. This will define the AWS region and name to be used for creating resources.

Build and push a backend and frontend image

The tf-bootstrap command also creates ECR repositories for the backend and frontend images. We can use the ecr_backend.yml and ecr_frontend.yml GitHub Actions workflows to build and push the backend and frontend images to the ECR repositories. These pipelines accept a single parameter which is a git tag that must exist in the repo. This git tag will then be used as the image tag for the backend and frontend images.

Purchase a domain name in Route 53

I use Route 53 to manage DNS records, and have purchased a domain name that I use for testing and debugging in this and other projects.

Create a wildcard ACM certificate

I chose to create this certificate outside of Terraform and import it via ARN. We need a wildcard certificate so that multiple ad hoc environments can be hosted on subdomains of the same domain.

Create a new EC2 key pair

The key pair should be created manually and it needs to be added to GitHub repository secrets so that it can be used in the ad hoc environment pipelines.

Add secrets to GitHub

The following secrets are needed for GitHub Actions to run. Add these as repository secrets:

ACM_CERTIFICATE_ARN - ARN of the wildcard ACM certificate
AWS_ACCESS_KEY_ID - AWS access key ID
AWS_ACCOUNT_ID - AWS account ID
AWS_DEFAULT_REGION - AWS default region (I use us-east-1)
AWS_SECRET_ACCESS_KEY - AWS secret access key
DOMAIN_NAME - domain name for the ad hoc environment (e.g. example.com)
KEY_NAME - name of the EC2 key pair
SSH_PRIVATE_KEY - private key for the EC2 key pair
TF_BACKEND_BUCKET - name of the S3 bucket used for storing Terraform state
TF_BACKEND_DYNAMODB_TABLE - name of the DynamoDB table used for locking the Terraform state file
TF_BACKEND_REGION - AWS region for the S3 bucket and DynamoDB table

These secrets are referenced in the GitHub Actions workflows.

Create shared resources

Now that we have GitHub secrets configured, we can run the shared_resources_create_update.yml GitHub Actions workflow. This will create shared resources environment in which we can build our ad hoc environments. This workflow requires a name (e.g. dev). This require that we create a dev.tfvars file in terraform/live/shared-resources directory. This usually takes between 5 and 7 minutes to complete since it needs to create an RDS instance which takes a few minutes to provision.

Create an ad hoc environment

We can now create an ad hoc environment. This requires the name of the shared resources environment (e.g. dev) and the name of the ad hoc environment (e.g. brian-test). The only thing we need to do before creating the brian-test ad hoc environment is to create the brian-test.tfvars file in the terraform/live/ad-hoc/envs directory. This will define the versions of the application and any other environment configuration that is needed. This pipeline usually takes about 3 minutes to finish.

Update the backend version in an ad hoc environment

Now that the backend is up and running and usable, we can update the backend version used in our ad hoc environment. This can be done by running the ad_hoc_backend_update.yml GitHub Actions workflow. To run this workflow you must specify:

the shared resource workspace (e.g. dev)
the ad hoc environment name (e.g. brian-test)
the new version of the backend to be used (e.g. v1.0.2)

The backend version should already have been built and pushed to the ECR repository using the ecr_backend.yml GitHub Actions workflow.

Use ECS Exec to access a Django shell in a running container

Instead of SSHing into the container, we can use the ecs-exec command to access a shell in the container. This is useful for debugging and testing. One of the outputs of the ad hoc environment terraform configuration contains the script needed to run the ecs-exec command:

TASK_ARN=$(aws ecs list-tasks \
  --cluster alpha-cluster \
  --service-name  alpha-gunicorn | jq -r '.taskArns | .[0]' \
)
aws ecs execute-command --cluster alpha-cluster \
    --task $TASK_ARN \
    --container gunicorn \
    --interactive \
    --command "/bin/bash"

You will then have a shell in the container:

The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.


Starting session with SessionId: ecs-execute-command-073d1947fa71c058c
root@ip-10-0-2-167:/code# python manage.py shell
Python 3.9.9 (main, Dec 21 2021, 10:03:34)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>

Destroy an ad hoc environment

Use the ad_hoc_env_destroy.yml GitHub Actions workflow to destroy an ad hoc environment. To run this workflow you need to specify the shared resource workspace (e.g. dev) and the ad hoc environment name (e.g. brian-test).

Destroy the shared resources environment

Use the shared_resources_destroy.yml GitHub Actions workflow to destroy the shared resources environment. To run this workflow you need to specify the shared resource workspace (e.g. dev).

This defines the full lifecycle of creating and destroying ad hoc environments.

Future improvements, open questions and next steps

Enhanced Security

The low hanging fruit here is to use least privilege (for roles used in automation) and to define all roles with IaC. Currently I am using Admin roles for the credentials I store in GitHub which is a shortcut to using IaC and is not a best practice.

Keeping track of ad hoc environments

We need to think about how we can keep track of our active ad hoc environments. Active environments will incur additional AWS costs, and we do not want developers or the product team to create lots of environments and then leave them running without actively using them.

We may decide to have some long-lived ad hoc environments, but those would be managed primarily by the DevOps team and respective owners (e.g. QA, product team, etc.).

One way to check the active ad hoc environments would be to use the AWS CLI. We could list the ECS clusters in our development account, and this would show the number of ad hoc environments running. We could go farther and list the ad hoc environments by when they were last updated. We could then request developers or team members to remove ad hoc environments that are not in use.

Or we could have a policy that all ad-hoc environments are deleted automatically at the end of each week.

Terraform Tooling

Testing Terraform Code
Testing GitHub Actions
Using advanced Terraform frameworks like Terragrunt
Using Terraform Cloud for more advanced Terraform features

More secure way of defining RDS username and password

Currently the postgres database does not have a secure password. It is both hardcoded in the module as a default value and it will also be saved in plaintext in the Terraform state file.

Backend application update script

The script used for updating the backend application could be improved or broken up into multiple scripts to better handle errors and failures that happen during the pipeline. The script runs several different commands and could potentially fail at any step, so it would be nice to improve the error messages so that both developers and DevOps teams can more quickly diagnose pipeline failures.

Limiting traffic to ad hoc environments to a VPN

Another good next step would be to show how we can limit traffic to ad hoc environments to a VPN.

Figure out how many ad hoc environments we can create with the default quotas

AWS accounts limit the number of resources you can create, but for most of this quota limits you can request an increase per account. I need to figure out how many ad hoc environments I can create with the default quotas.

Code Repo Organization

One minor improvement would be to move the terraform directory out of the django-step-by-step monorepo into a dedicated repo. We may also want to move GitHub Actions for creating, updating and destroying environments to this new repo. For early stage development, using a single repository that stores both application code and Terraform configuration works, but it would be better to keep these separate at the repository level as the project grows.

Multiple AWS Accounts

Everything shown here uses a single AWS account: ECR images, Terraform remote state storage, all shared resource environments and all ad hoc environments. Using one account keeps things simple for a demonstration of this workflow, but in practice it would be beneficial to use multiple AWS accounts for different purposes. This would also involve more carefully planned IAM roles for cross-account resource access.

Modules for stable environments to be used for long-lived pre-production and production environments

This article looked at how to make tradeoffs between costs, speed of deployment and production parity in ad hoc environments. I'm interested in building a new set of modules that can be used to set up environments that:

are more stable and more long-lived
have less resource sharing (dedicated RDS and ElastiCache resources)
implement autoscaling for load-testing (or maybe implement autoscaling for ad hoc environments)
can be used to perform load testing
have enhanced observability tooling

These environments might be used as part of a QA process that does a final sign-off on a new set of features scheduled for deployment to production environments, for example.

Conclusion

This wraps up the tour of my ad hoc environment infrastructure automation. Thank you for having a read through my article. If you have a similar (or different) approach to building ad hoc environments, I would love to hear about it.