The Ops Community ⚙️: Drew Khoury

Optimizing for a Cloud-native Developer Experience

Drew Khoury — Wed, 25 May 2022 19:01:17 +0000

Chris and Drew share their combined knowledge around developer experience, why (micro) feedback loops matter a whole lot, and present a live demo of a Cloud Native application working locally - complete with unit tests, end-to-end tests and smoke tests.

Originally presented by Chris and Drew Optimizing for Developer Experience in a Cloud Native World - AWS Meetup Group
Concepts based on the work from Tim Cochran via Martin Fowler Maximizing Developer Effectiveness
Cloud Native DX miro board for presentation content
Following the 3 Musketeers pattern

This talk is packed with real patterns used by Chris and Drew working with teams to help them increase their effectiveness. The demo was designed so you can follow along on your own workstation - and be up and running within minutes without the need to install and configure the underlying tech (Node, AWS SAM, Python, Playwright, etc).

Meetup Video: https://www.youtube.com/watch?v=6MP1u7O_3bY

Micro feedback loops

"From what I have observed, you have to nail the basics, the things that developers do 10, 100 or 200 times a day. I call them micro-feedback loops. This could be running a unit test while fixing a bug. It could be seeing a code change reflected in your local environment or development environments." - Tim Cochran

Demo

Demo Repo: GSD-AWS-CDK-Serverless-Example

If you have make, docker-compose and docker installed you should be able to have a local env running in a few minutes.

git clone git@github.com:chrishart0/gsd-aws-cdk-serverless-example.git && cd gsd-aws-cdk-serverless-example
make install && make run

Hint: If you run make after cloning the repo, it will show you the help menu and let you know what commands to run to build, test and deploy the application.

Micro feedback loops demonstrated in this repo:

Unit tests (front-end, back-end, infra)
Local environment - available in the browser
End-to-end tests
Direct deployment to AWS from your workstation
CI/CD runs the same as local, with additional security checks

Giving us your feedback (loop)

We'd love to hear what you think about these patterns, and get some feedback on your own developer experience using the demo repo.

Contact Chris
Contact Drew
Raise an issue in the repo

How to implement Good Software Delivery in 30 seconds

Drew Khoury — Wed, 25 May 2022 18:55:43 +0000

Good Software Delivery (GSD) is the term we use for the set of practices that help deliver, well, good software. There’s a focus on short feedback loops, a consistent developer experience, with more time spent delivering and less time spent on chores like workstation setup and debug.

For context, I’ve previously written the following blogs that explain a lot of the “why” you’d want to implement the app I’m about to demo:

Assumptions

Before we dive in I want to share my assumptions and expectations. This blog/demo is aimed at developers, who are also familiar with pipelines, local development environments, and tools like docker.

While you can clone the repo and run the commands quickly (even with zero knowledge or experience in the underlying Golang app), it will take you longer than 30 seconds to read this blog, and even longer to look through the repo’s README.md and code in the repo to see what’s happening when you type each command. I’ve tried to add as much detail and context for those that’d like to deep dive, both in this blog and in the repo itself.

The purpose isn’t to copy and paste what you see here into production, but I do hope I’m able to demonstrate some ways of setting up your repo and local development environment in a way that could save you and your team time for your next product!

What we’ll do in this demo

In this post, I wanted to share a hands-on implementation of some of these concepts that you can try for yourself. We’ll be using a hello-world app written in Golang, and by the end of the demo I’m hoping:

You’ll see how wrapping our pipeline in 3 Musketeers can greatly reduce the amount of toil spent getting a new project started (thus the 30 seconds claim for this simple app)
You’ll see that you can adapt the concepts in this demo to just about any app (Java, .Net etc) — More complex apps may take longer to compile, but no extra effort should be required in terms of desktop setup (toil)
You’ll be able to extend the logic for other pipeline tasks like testing or security

If this demo takes you longer than 30 seconds to run this demo, I suspect:

We need to get you a better Internet connection (most of the 30 seconds for me is spent downloading the initial Golang image on an empty docker cache)
We need to help you with the base requirements that this blog assumes you already have (make,dockerand docker-compose)

In any case, reach out on LinkedIn if you have suggestions to improve this demo, or set up a virtual meeting with me if you’d like a 1:1 workshop/chat!

The demo (build and run a Go app in 30 seconds)

At a minimum, you’ll need _make_, _docker_ and _docker-compose_ on your workstation. See requirements for more info or continue on if you’ve already got these.

Not sure about running this demo on your computer? Never fear! You can use this online environment from the safety of your browser: GSD HelloWorld katacoda (repo will already be cloned in this environment, with required software ready to go)

You can see all of my Katacoda courses at https://www.katacoda.com/drewkhoury/.

Clone the repo: Start by cloning the GSD HelloWorld repo:

git clone git@github.com:contino/gsd-hello-world.git && cd gsd-hello-world

2. Build the app: Once you have the codebase, build the app using:

make build

This step will create a build artifact (in our case, a docker image) by doing the following:

downloads the appropriate docker image onto your workstation
copies the source for our app into the image
builds the application in the image
configures the port/cmd to be used at runtime

3. Run the app: Now that you have your build artifact handy, you can run the application locally with:

make run

This step will run/deploy the artifact on your workstation by doing the following:

uses docker to create a container based on the image from the build step
exposes port 8080 locally
the docker image runs the cmd ./main which starts the application for us

That’s it, you’re running the GSD HelloWorld application written in Golang and containerized so that it can quickly be deployed in Docker.

Bonus: You can run make test to execute the test suite (note this may legitimately fail if it doesn’t meet the testing threshold), andmake down to clean up once you’re done.

The beauty of this model is in its simplicity. We’re using make as the interface and we’re running all the tasks in containers to ensure consistency. It’s easy enough to peak under the hood to see what either of those tools is doing, and this runs just as easily on your workstation as it does on your CI tool of choice.

Digging into the code

To keep this section brief I’m going to assume some prior knowledge around development, docker, and make.

What if I want to make my own changes or do something similar in my own repo?

For the curious: Checkout the Makefile, docker-compose.yml and Dockerfile in the repo. These drive our pipeline and app build etc, and while they abstract the complexity away from day-to-day usage, they’re intended to be developer-friendly in terms of understanding exactly what each step is doing.

Building the app: For example make build is what builds our application, and in the Makefile you’ll see it does the following:

docker build -t ${FULL_TAG} .

Replacing the app: If you wanted to replace our Golang application with your own Java application, you’d drop in your own Dockerfile , replace the /src, and leave everything else as-is. The beauty is that any developer using the repo still runs make build and they don’t need to worry about Go vs Java dependencies (or in theory even know that you completely rewrote the application) — it just works™.

Note: This repo is a testing ground for GSD practices — simple but powerful ways to supercharge your development.

If you’d like to see what else you can do, have a look at the rest of the https://github.com/contino/gsd-hello-world — we have plenty for you to explore.

build, test and run actions are driven by a simple Makefile — with underlying execution via docker containers
Build badge in README.md
Repo Visualization in README.md
Sonar quality, reliability, and security badges in README.md
Golang security in-pipeline
Local tests via make test
Artifact storage via Github Packages
Github Actions
Deployment to k8s in GCP

Further reading:

May your software be good, and your delivery be continuous — drew

Also posted on medium as How to implement Good Software Delivery in 30 seconds.

AWS 2021 Highlights

Drew Khoury — Wed, 25 May 2022 18:53:28 +0000

AWS updates their services so quickly they literally have thousands of updates each year (1,284 the last time I checked): https://aws.amazon.com/about-aws/whats-new/2021/

This blog will highlight some of my favorite AWS updates for 2021.

AWS Network Firewall

https://aws.amazon.com/network-firewall/ — A managed service by AWS that allows fine-grained control over network traffic.

Before Network Firewall was available, customers were left to manage their own squid proxy or similar service if they wanted fine-grained control over their traffic in the Cloud (like many Enterprise customers do). This meant ensuring their service met security and compliance requirements, traffic and scaling demands, and uptime SLA’s, all while adding to the burden for operational teams. Some routed traffic back to on-premise solutions, but this wasn’t always a viable solution for all customers.

AWS Network Firewall became available in Sydney back in January: https://aws.amazon.com/about-aws/whats-new/2021/01/aws-network-firewall-is-now-available-in-the-asia-pacific-sydney-region/

It received a few updates and deployments to other regions. In April, they rolled out to 10 more regions. By June, it was available as part of AWS GovCloud (US), and by the end of July, it was PCI DSS Compliant.

Amazon EBS io2 Block Express Volumes

If you have the need for speed (that is, I/O speed), then this is the update for you. In July, AWS launched a special kind of volume that is perfect for I/O intensive operations. This isn’t intended for your typical web server with a few hundred concurrent connections, but plenty of Enterprise clients may find themselves needing a little bit of extra grunt.

https://aws.amazon.com/about-aws/whats-new/2021/07/aws-announces-general-availability-amazon-ebs-block-express-volumes/

up to 4x higher throughput, IOPS, and capacity than io2 volumes
designed to deliver sub-millisecond latency
99.999% durability
Supports Multi-Attach and Elastic Volumes

Customers can now provision a single io2 volume with up to 256,000 IOPS, 4000 MB/s of throughput, and a storage capacity of 64 TiB.

PrivateLink for Amazon S3

As of February, Amazon S3 supported AWS PrivateLink, providing direct access to S3 via a private endpoint within your virtual private network.

https://aws.amazon.com/about-aws/whats-new/2021/02/amazon-s3-now-supports-aws-privatelink/

Simplify your network architecture by connecting to S3 from on-premises
In AWS, use private IP addresses in your Virtual Private Cloud (VPC) to connect to S3

This eliminates the need to use public IPs, configure firewall rules, or configure an Internet Gateway to connect to S3.

As of July, AWS also lowered the data processing cost for PrivateLink (phew): https://aws.amazon.com/about-aws/whats-new/2021/07/aws-lowers-data-processing-charges-aws-privatelink/.

Summary

Each of these updates has been something I’ve personally been waiting for or had to work around in the past. They also represent a nice cross-section of services that you’re likely to encounter if you dive deep enough into AWS.

AWS continues to roll out updates across its services which reduces the effort required to enter the Cloud. They continue to deliver services that cater to the speed and scale we could only have dreamt of 15 years ago (oh and Happy Birthday AWS/EC2).

Also posted on medium as AWS 2021 Highlights.

3 Musketeers for an epic Developer Experience

Drew Khoury — Wed, 25 May 2022 18:39:02 +0000

We all want to reduce toil and improve the developer experience (when starting new applications or joining the team). Developers have had to invest significant time in creating production-ready pipelines when building new products and configuring their workstations when joining teams. Their workstations may not run all the same tests as their pipeline and thus they’re left with #slowfeedback and an inconsistent experience.

The larger the organization (ie Enterprise with 10’s-100’s of teams and applications) the worse the problem gets.

GSD is built on the principle of reusability through a pattern called 3 Musketeers (test, build, and deploy your apps from anywhere, the same way). It gives consistency, control, and confidence to development teams.

Docker is a key part of this pattern, but the use of docker, docker-compose and make together really unlock the powers of consistency, control and confidence in your application build, test and deploy steps. The 3musketeers Docs have some great examples of these patterns.

Here's what the most simple usage looks like:

While the pattern itself is simple, the power (and benefit) comes from the widespread usage across your organization. Here’s what usage might look like:

Agree on a standard interface for all pipeline actions (much like API’s have an agreed interface, all pipelines in the org use the same interface, ie make build\, make test\, make deploy\)
Keep the interface simple (avoid custom pipeline logic, one simple Makefile calls Compose, the container handles all the logic)
Create docker containers for each pipeline step, make them stateless, reuse them where it makes sense

Wins: Modern pipelines already support running steps in docker containers. This pattern extends portability across almost any pipeline tool or development workstation and runs the same way in every single environment. Pipeline steps can be tested independently and reused. This pattern makes it easy to go one step further and generate pipelines on-demand based on boilerplate starter templates. Oh yeah, and #fastfeedback (we’re talking pre-commit).

Also posted on medium as 3 Musketeers for an epic Developer Experience.

Teaching DevOps in one afternoon

Drew Khoury — Wed, 25 May 2022 18:36:03 +0000

A non-technical friend of mine approached me wanting to know everything they could about DevOps. He asked if he could have a few hours of my to pick my brain and ask me some questions. If only he know how much joy that brings me.

So we used One DevOps Please as our springboard for the conversation. It worked well enough and we got to spider out into other conversations where he had questions. There were many times I had to pause and say “that’s a conversation for another day” so I wanted to capture some of those topics that we didn’t get to delve into. I figured it would be about the same effort to turn it into a blog post and share it with the world.

Existing Blogs

It’s no surprise that I already have a lot of blog posts within this theme. Feel free to explore those, or skip ahead to the new stuff!

What is DevOps
Build your first Python App using Google App Engine
Learn Ansible
Things I learned working for IT Consulting companies (as it relates to DevOps, Agile, Cloud and team building)
The best guide for Agile that you'll ever find
Talking about building a Learning culture

Umbrella Themes

For this post I’d like to break down “DevOps” into three main themes. I believe that the “people” side of things greatly influences how we do the technical parts so in some ways it’s one of the most difficult and most import parts to get right.

People, Culture & Ways of Working

Continuously improve (Growth mindset)
Encourage innovation
Foster collaboration

Measurable

Measuring everything
Make decisions based on data
Make data easily available

Continuous Delivery

Higher quality software through repeatable, predictable delivery
Reduction of defects
Reduce cycle time of ideation (I have an idea) to production (released in prod)

People

Customer Focused — app teams or cross-functional teams with your org may be your customer.

How is your organization structured? https://web.devopstopologies.com/

Agile — no need to be overly prescriptive. Understand where “Agile” came from and what they indended to do with it: https://agilemanifesto.org/principles.html

Extreme Programming: https://www.agilealliance.org/glossary/xp/

The Planning Game
Small Releases
Metaphor
Simple Design
Testing
Refactoring
Pair Programming
Collective Ownership
Continuous Integration
40-hour week
On-site Customer
Coding Standard
Sit Together
Whole Team
Informative Workspace
Energized Work
Pair Programming
Stories
Weekly Cycle
Quarterly Cycle
Slack
Ten-Minute Build
Continuous Integration
Test-First Programming
Incremental Design

Measure Everything

Let’s say it together “measure everything”.

In particular having metrics that are focused on Business Value Drivers are key.

Metrics should be captured and key metrics displayed on dashboards & TVs so they’re easily accessible.

Teams should produce APIs to be consumed internally — https://cachethq.io/.

Site Reliability Engineering.

Particularly for large organizations it can help to understand where each teams is at on their journey. Usually I’ll see teams try and go through some sort of journey of maturity in “DevOps”, “Cloud”, “Agile” or just “Good Software Development Practices”.

Maturity Assessments:

Agile
CI/CD
12 Factor — https://12factor.net/
Data
6R’s Application Treatment Plan — https://aws.amazon.com/blogs/enterprise-strategy/6-strategies-for-migrating-applications-to-the-cloud/

Continuous Delivery

This is really the art of getting code from laptop to production, and yes it can be part art, part science and a little bit of magic sometimes.

https://continuousdelivery.com/

Low risk releases. blue-green deployments
Faster time to market automate the build and deployment, environment provisioning, and regression testing processes
Higher quality automated tools that discover regressions within minutes
Lower costs investing in build, test, deployment and environment automation, we substantially reduce the cost of making and delivering incremental changes to software by eliminating many of the fixed costs associated with the release process
Better products work in small batches. This means we can get feedback from users throughout the delivery lifecycle based on working software
A/B testing & hypothesis-driven approach to product development
Happier teams continuous delivery makes releases less painful and reduces team burnout

https://continuousdelivery.com/implementing/patterns/

Only build packages once
Deploy the same way to every environment
Smoke test your deployments
Keep your environments similar

Trunk Based Development

A branching model to aspire to (or at least understand and learn from): https://trunkbaseddevelopment.com/

short iterations
less than few day story size
short build times
feature flags
humans and CI use same scripts
Compile > Unit Test > Integration > Functional
Quick build notifications
Shift Left — Undesirable to find out about breakages AFTER the commit

Containers

Containers are an amazing innovation that have many every day uses. They can be used locally, in your build servers, for development and production deployments. You can also use them to test new services and keep your laptop free of clutter!

How to build, test and deploy modern code in a repeatable fashion: https://3musketeers.io/

Containers 101…

https://en.m.wikipedia.org/wiki/Kubernetes

https://en.m.wikipedia.org/wiki/Docker_(software)

https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/

https://danlebrero.com/2018/07/09/kubernetes-explained-in-pictures-the-theme-park-analogy/

(Don’t) Build it yourself

The phrase “undifferentiated heavy lifting”: https://aws.amazon.com/blogs/aws/we_build_muck_s/

When it comes to investing time you should consider the implications of “building and managing infrastrcuture yourself” vs “letting specalisits do the heavy lifting”. This might mean different things for different people, but here are some examples:

Configuring AWS Accounts & setting up guardrails: https://aws.amazon.com/controltower/features/
The quickest and best way (imo) to run k8s: https://cloud.google.com/kubernetes-engine/
Do you even need a k8s cluster? Sometimes you just need someone to host your app for you: https://cloud.google.com/appengine/

And that’s (a little more than) one afternoon teaching DevOps.

Also posted on medium as Teaching DevOps in one afternoon.

DevOps in 10 minutes with Google App Engine

Drew Khoury — Wed, 25 May 2022 18:23:40 +0000

Okay, now that I have your attention let’s talk about Google App Engine for a minute and how you can achieve your DevOps dreams with a fully managed serverless application platform.

App Engine history

App Engine launched 10 years ago as a tool to run web applications on Google infrastructure. App Engine only supported Python at the time, but the service offered dynamic webserving, persistent storage (Datastore), and APIs for authenticating users and sending email.

Fast forward to 2019 and the App Engine offering has matured. The standard environment is optimised for rapid scaling (sudden and extreme spikes of traffic) or apps intended to run for free or at very low cost.

Python 2.7, Python 3.7
Java 8
Node.js 8, and Node.js 10 (beta)
PHP 5.5, and PHP 7.2 (beta)
Go 1.9, and Go 1.11 (beta)

If you need something else, the flexible environment offers Python, Java, Node.js, Go, Ruby, PHP, or .NET, or any other software that you can run in a container.

But it’s more than just a code runtime engine. Google markets App Engine as having zero-config deployments and zero server management, allowing you to focus on writing code (that sounds like my kind of DevOps). App Engine offers Monitoring, Logging & Diagnostics, Traffic Splitting and Application Versioning (those features you normally hear about when talking about “DevOps” or CI/CD).

Getting started with App Engine

Assuming you already have a Google Cloud account, creating a deploying your first App Engine App consists three easy steps:

Create a Project to store all your related resources
Create an App
Deploy your App

gcloud projects create project-demo-xxx \

--name="project demo xxx" --labels=type=test

gcloud app create --region=australia-southeast1 \_

_--project project-demo-xxx

gcloud app deploy --project project-demo-xxx

Normally, navigating a new Cloud vendor, a new product, and new concepts and troubleshooting demo code can be time consuming and error prone. I know, I’ve gone through this process myself.

If you’re interested in giving App Engine a go (and I encourage you to do so, because it’s a fantastic example of what an integrated serverless platform should look like) you can head over to https://github.com/drewkhoury/gae-demo and try it yourself. I’ve streamlined the whole process to try and get you up and running as quickly as possible:

How to authenticate with Google via the command-line
Taking advantage of the local environment Google provides (dev_appserver.py)
Sample Python App that’s optimised for App Engine
Docker containers / Docker compose for consistency (local development and deployment)

Let me know what you think of Google App Engine after you give it a go!

Also posted on:

medium as DevOps in 10 minutes with Google App Engine
and LinkedIn as DevOps in 10 minutes with Google App Engine

The cost of failure is education

Drew Khoury — Wed, 25 May 2022 18:22:12 +0000

If you’ve ever been responsible for software running in production, you should already be well aware of failure. But it’s not only traditional “operations” roles that are affected by failure.

Consider some of the changes we’ve seen in IT in the last decade:

2003: SRE - Site Reliability Engineer, a discipline created at Google that incorporates aspects of software engineering and applies them to IT operations problems.
2006: AWS EC2 - The birth of the cloud “Amazon Elastic Compute Cloud (EC2)” which made Infrastructure-as-a-Service available to developers around the globe via APIs. It paved the way for CloudNative, Serverless, PaaS and the hundreds of services we have available to us today.
2009: DevOps - The combination of software development (Dev) with information technology operations (Ops)”.

It’s clear there have been changes in the people and teams who need to be involved in running software for customers. At the same time there have been significant changes to how we build, test and deploy software.

Culture is a common topic that comes up when we talk about things like DevOps, Lean Software Development, and SRE but what do we mean when we say culture? Do we just give developers pizza and beer until they produce good software and hope they don’t leave for another company offering gourmet pizza and craft beer?

Origins of lean in manufacturing

To understand the origins that paved the way for DevOps, SRE, and the “Culture/Transformation” we’re hearing so much about in IT we need to go back, back to 1990 and back to car manufacturing. Consider the following quote from https://en.wikipedia.org/wiki/Lean_manufacturing

Lean manufacturing makes obvious what adds value, by reducing everything else (which is not adding value). This management philosophy is derived mostly from the Toyota Production System (TPS) and identified as “lean” only in the 1990s

Steps to achieve lean systems

The following steps should be implemented to create the ideal lean manufacturing system:

Design a simple manufacturing system
Recognise that there is always room for improvement
Continuously improve the lean manufacturing system design

Sound familiar? These themes are present in Lean, Agile and DevOps.

The andon cord — Game-changing rope

Back in the early days of the Toyota Production System (TPS), Toyota implemented many unique tools and processes. The Andon Cord was born, and while it was just a rope it was the most powerful rope you’ve ever seen.

Anyone had the right to pull the cord at any time and pulling it would instantly stop all work on the assembly line. The technology behind this wasn’t revolutionary, but the thinking, culture and trust behind this system is something we can learn from today. The “Safety Culture” around this process was reinforced when the team leader arrived at the workstation, thanking the team member who pulled the Cord.

The team member did not, and never would be, in a position of feeling fear or retribution for stopping the production line

Toyota has so much to teach us about working together an respecting each other. Even if the team leader knew a better answer, Toyota principally felt that a solution from the team member was a better outcome.

Postmortums & lean software development

Leaving cars behind and getting back to software, Mary and Tom Poppendeick’s work on Lean Software development in the early 2000s paved the way for the sort of thinking coming out of Google (SRE), and one great example of that is postmortem culture.

Writing a postmortem is not punishment — it is a learning opportunity for the entire company.

As an IT professional with a decade of experience I’ve seen how the culture of the company can significantly impact how people, teams and the organisation grow and improve. The great opportunity that exists when failure happens is learning. The companies and teams that truely believe this are the ones that will continuously improve, both their software and their culture.

Also posted on medium as The cost of failure is education.