The Ops Community ⚙️: Jatin Mehrotra

How to use AI for AWS CLI commands?

Jatin Mehrotra — Tue, 21 Nov 2023 04:59:34 +0000

In this blog, I want to show how I was able to set up and use AWS CodeWhisperer for the command line and enhance my productivity.

Just before AWS Re:Invent 2023 there is an amazing update from AWS which allows to use of AI for the command line. Yes it's true that from today we can use AWS CodeWhisperer with AWS CLI with provides CLI completion, inline documentation and AI natural language -> to CLI command generation.

Here is the link to the official announcement.

https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-codewhisperer-command-line-preview/

Note: It's still in preview, so specifications about this blog may be subject to change in future.

CodeWhisperer for command line setup

At this moment codeWhisperer is available only on macOS as per official announcement

CodeWhisperer for command line is available on macOS for all major shells (bash, zsh, and fish) and major terminal emulators such as Terminal, iTerm2, Hyper, and the built-in terminals in Visual Studio Code and JetBrains.

Link to official Docs: https://docs.aws.amazon.com/codewhisperer/latest/userguide/command-line-getting-started-installing.html
Download dmg file for CodeWhisperer for the command line dmg file from the docs.

Download DMG file and then move the file to Applications. (typical drag and drop for MacOS applications)
After installation, start the newly installed CodeWhisperer application

Install the shell integration hooks as the next step in the installation.

As a next step we need to enable accessibility so that CodeWhisperer can write on insert CLI completions and insert commands in the shell. CodeWhisperer will not work if this is not enabled.

Toggle the accessibility for CodeWhisperer.

Next step will ask to sign on. Upon clicking you will redirected to authorise CodeWhisperer for your AWS account.

You will need an AWS Builder ID. If you already have a Builder ID use your existing credentials otherwise create once as shown in image below. In my case I already have a Builder ID

As a last step, if you have setup everything correctly so far, you will see a prompt in which you allow COdewhisperer to access your data so that CodeWhisperer can do the following:
- Enable CodeWhisperer completions
- Enable CodeWhisperer analysis
- Access to IAM Identity centre accounts and permission sets.

Let's see CodeWhisperer in action for AWS CLI commands

Once the setup is complete, you will see the code whisperer default window.

Generating Commands from text

With CodeWhisperer you can write natural language ( many languages are supported) instructions like create s3 bucket or create a lambda function. Codewhisperer will then convert this text to an instantly executable shell command.
You can either use this feature in 2 ways:


  cw ai CodeWhisperer completions
  # CodeWhisperer completions // # is not the comment

I tried to create cloudfront distribution whose origin is s3 bucket using the following prompt


cw ai create cloudfront distribution named test whose origin is s3 bucket name test
-bucket

Once you have found your command you either execute it, or choose to edit it or regenerate, ask another question or cancel it.
You can write the prompt using # your prompt. In the following I tried to create a lambda in tokyo region and it did pretty good job.


# create cloudfront distribution named test whose origin is s3 bucket name test
-bucket

CLI completion

It also does a great job when it comes to CLI completion in the way it adds *typeahead code completions *

Not only AWS CLI commands, code whisperer has support for docker, kubectl　（tested, git commands too.

Bonus: Customization of Codewhisperer for CLI

You can always customise the behaviour of Codewhisperer from its dashboard which allows to change its theme, disable code whisperer for certain commands, instant execution of dangerous commands like bucket delete etc.

My thoughts

As a DevOps engineer who has to run certain repeated commands every day for daily tasks, I think this update will definitely increase my productivity and make my life way easier with its code completion and text-to-command feature.

As I said before it's in the preview stage, and cannot be used for production but it's going to get even better from this stage.

Connect with me over Linkedin or Twitter and share your thoughts about this blog.

Send Custom Notification with AWS Chatbot

Jatin Mehrotra — Wed, 13 Sep 2023 09:28:38 +0000

AWS Chatbot is ChatOps offering from AWS to monitor, troubleshoot, and operate your AWS environments natively from within your chat channels ( Teams, Slack channels etc)

Until now, there was a limitation that notifications from aws chatbot could not be customised, additional information in the notifications

But with this update, the aws chatbot just got new powers !!!

Outcome of the Blog

By the end of this blog, you will know:
How to send custom notifications using chatbot using lambda.
Why this update will help users to understand and respond quickly to events.

About this Update [ 12/09/2023 ]

This update will help users to understand more insights about the notification and which will help them to respond quicker.
You can add specific details, tag team members, and include remediation steps. It's upto your creativity now that how insightful and information you can make your custom notifications.
There are no additional costs to use custom notifications.

Prerequisites for sending custom notifications using chatbot?

Custom notifications must use the following event format as mentioned in the docs.
Take a note about the parameters which are required and which are optional in the given table in Parameters table.


{
    "version": String, 
    "source": String, 
    "id": String,    
    "content": {
        "textType": String, 
        "title": String,  
        "description": String, 
        "nextSteps": [ String, String, ... ], 
        "keywords": [ String, String, ... ] 
        },
        "metadata": {                     
            "threadId": String,
            "summary": String,
            "eventType": String,
            "relatedResources": [ String, String, ... ],
            "additionalContext" : {
                "customerProvidedKey1": String,
                "customerProvidedKey2": String
                ...
            }
        }
}

Use client specific syntax: If messages are being sent to slack then slack markdown format should be used within custom notification and vice-versa for Teams.
From the docs

You can add chat platform compatible emojis in your custom notifications. You can also tag team members using @mentions in your custom notifications for Slack. Tagged team members are notified when the custom notification is delivered to the chat channel. To tag a team member in Slack, use their user ID. For example, @userID. Tagging team members in Microsoft Teams isn't currently supported.

Architecture Pattern for using custom Notifications

There will be a source which will emit the events with all the details.
Either source will generate event (using Lambda) with all the custom information or use EventBridge Input Transformers to add custom information to the original event.
Final notifications will be sent to SNS topic.
Chatbot will continuously monitor that sns topic and send notification to slack channel.

Let's create a custom notification using aws Lambda.

As we saw in architecture pattern we need 3 things to implement and test out custom notification: AWS Lambda, sns topic and chatbot configured for the slack channel

Note: For visual simplicity and understanding, I will be using AWS console for this blog but the same thing can be achieved using AWS CDK.

Create SNS topic

Create SNS topic and note its arn.

Create a Lambda

We will create a AWS Lambda Function with nodejs runtime and we will expose lambda using function Urls
Code for Lambda


import {
  EventBridgeClient,
  PutEventsCommand,
} from "@aws-sdk/client-eventbridge"

import { SNSClient, PublishCommand } from "@aws-sdk/client-sns";

const snsClient = new SNSClient({ region: process.env.AWS_REGION });
const snsTopicARN = process.env.SNS_TOPIC_ARN


export async function handler() {


    await publishNotificationToSlack()

    return {
        body: JSON.stringify({ message: 'Successful lambda invocation' }),
        statusCode: 200,
    };
}

export const publishNotificationToSlack = async (
) => {



 const slackFormatEvent = {
    "version": "1.0",
    "source": "custom",
    "content": {
        "description": ":fire: Your <https://www.linkedin.com/posts/jatinmehrotra_custom-notifications-are-now-available-for-activity-7107541447716814848-dAAv?utm_source=share&utm_medium=member_desktop|post> is gaining traction!!! :fire:",
      "title": ":linkedin: Check Linkedin <@YOURSLACKID|YOURNAME>"
    }
}

    const response = await snsClient.send(
        new PublishCommand({
            Message: JSON.stringify(slackFormatEvent),
            TopicArn: snsTopicARN,
        })
    );
};

I have attached permissions of full access sns to lambda role so that lambda can send notifications to sns topic directly. From a security point of view you should limit access to only required SNS topic.

Add environment variable to lambda configuration as SNS_TOPIC_ARN: arn of the SNS topic noted earlier

Configure AWS Chatbot client and Slack Channel

In order to set up AWS chatbot client for Slack and slack channel, follow the steps mentioned here in docs.

Note: While configuring the Slack channel, add the SNS topic created earlier.

Test out the Lambda function

Visit the function URL of the lambda function

From the DevOps perspective

This update enables DevOps teams to improve their observability.
Allows the team to understand more insights about the notifications as it allows to define and add more information to your notifications.
With the support of custom notifications and native client support like slack mentions, it definitely allows the team to respond even more quickly at the time of critical events.

Connect with me over Linkedin or Twitter and share your thoughts about this blog.

Saving cost with AWS Lambda recursion control !!!

Jatin Mehrotra — Tue, 11 Jul 2023 11:12:21 +0000

AWS lambda has a new feature of recursion control which stops lambda functions execution during infinite or recursive loops.

There is an amazing blog on this from AWS team which test this feature with AWS SQS, Java lambda function and AWS SAM. Please check it out for more details.

However I took a different approach to test this feature using CDK, SNS topic, and typescript lambda function.

Prerequisites

Understanding how aws lambda is invoked and works.
Understanding of event-driven architecture which involves an event source and destination where lambda outputs.

Let's test it first

To test this feature we will be setting up sns topic, lambda (which publish message back to source sns topic) this creates a recursive loop.
The infrastructure for this test will be provisioned using AWS CDK.
Github repo for CDK code. In order to test this code, please enter your email address in the cdk code.

// to deploy 

cdk deploy --all 

// to destroy 

cdk destroy --all

Architecture Diagram

What is a Recursive loop in Lambda function?

AWS services (source) generate events that invoke Lambda functions, and Lambda functions then send messages to other AWS Services (destination).
Usually in event-driven architectures source and destination cannot be the same.
Due to misconfiguration or coding bugs, the lambda function sends the processed event to the same AWS Service (source) that invokes the Lambda function, causing a recursive loop.
When this recursive loop occurs the lambda function is also executed for infinite times as long as the loop occurs thus adding up to the usage bill for the aws account.

What is a Recursive control in Lambda?

According to this new feature[aws docs] it will detect and automatically break this loop or say stop lambda execution if it is invoked for more than 16+ times.
Behind the scenes, Recursion control uses X-Ray. Lambda uses AWS X-Ray tracing headers. When AWS services that support recursive loop detection send events to Lambda, those events are automatically annotated with metadata.
The updated metadata includes a count of the number of times that the event has invoked the function. That is how lambda detect recursion loop.
You don't need to enable X-Ray active tracing for this feature to work.

Note: Some important points about this feature

Same chain of requests: A chain of requests is a sequence of Lambda invocations caused by the same triggering event. for example SNS → Lambda → SNS = recursive loop
If your function is invoked more than 16 times in the same chain of requests, then Lambda automatically stops the next function invocation in that request chain and notifies you.
Lambda stops only invocations that are part of the same chain of requests. For example, if your function is configured with multiple triggers, then invocations from other triggers aren't affected.
This feature is free to use.

Supported AWS Services and SDK

AWS Services

At this moment lambda recursion control is supported between AWS SQS and AWS SNS event.

AWS SDKs

Node.js
2.1147.0 (SDK version 2)
3.105.0 (SDK version 3)

Python
1.24.46 (boto3)
1.27.46 (botocore)

Java 8 and Java 11
1.12.200 (SDK version 1)
2.17.135 (SDK version 2)

Java 17
2.20.81

.NET
3.7.293.0

Ruby
3.134.0

Types of notifications for Recursive loop

Email - An single email (once in 24 hours) will be sent to AWS account's primary account contact and alternate operations contact.
- I am using AWS organisations and iam role so it wasn't sent to my account email adress, it was my org account manager address.

AWS Health Dashboard- This will be inside your aws account. A notification under theother notifications section with details and link to lambda function.

Note: For both email and health dashboard, notifications will take around 3 hours to arrive.

Cloudwatch alarms - In order to set the alarm we need to check the RecursiveInvocationsDropped metric. We can always set an alarm if this metric count is more than 0.
- I haven't set up alarms personally for this test, so feel free to post about this in the comments. Would love to hear.

How to fix this recursion loop?

Remove or disable the trigger which invokes lambda function. In our case sns topic trigger.

topic.addSubscription(new aws_sns_subscriptions.LambdaSubscription(hello)) //remove this trigger

Identify and fix code which may be cause the loop. in our case I am sending the message from lamba to the same sns topic which is the source.

import { SNSClient, PublishCommand } from "@aws-sdk/client-sns";

export async function main() {
  const client = new SNSClient({ region: process.env.AWS_Region });
  try {
    const input = {
      // PublishInput
      TopicArn: process.env.TOPIC_ARN, // fix this
      Message: "This is a message from lambda",
    };
    const command = new PublishCommand(input);
    const response = await client.send(command);
    console.log(response);
  } catch (error) {
    console.log(error)
  }

}

Setting concurrency to 0. It basically acts like disabling lambda or like how aws says off switch for lambda.
If sqs is the source set up DLQ.
If sns is the source set up on failure destination for lambda.

How to turn off this feature?

By default this feature is turned on for supported AWS Services and sdk's. If your usecases uses recursive patterns, then you can request to turn off Lambda recursive loop detection. To request this change, contact AWS Support.

Conclusion

Recursion loop is not so often scenario but if its happens can cost a lot of money for your account and it can happen anyway due to misconfiguration or ignorance when writing lambda code.
Right now this feature is not supported for other services like Dynamodb and s3 events. In case of dynamodb and s3, the only way to detect recursive loop is using cloudwatch alarms. Please make a note of this.
I really hope in the future this feature is supported in DevOps Guru so that is very easy to find out the cause of this loop.

DevOps: Understanding Build and Package Manager Tools

Jatin Mehrotra — Sun, 15 Jan 2023 06:52:53 +0000

This blog talks about why an understanding of build tools, and package management tools is important for DevOps Engineers.

In this blog we are going to build our knowledge of build tools and based on that we will establish how docker helps us with building code.

This is a beginner-friendly blog that assumes you do not know anything about building code, artifacts or package management tools.

Outcome of the Blog

By the end of this blog, you will know:

What is Maven and grade, their difference and how does it help to build code for JAVA-based applications
What is artifact and it's types and what is artifact repository
How build tools and Docker are related
The role of DevOps Engineers in this Building code phase

Motivation

DevOps Bootcamp | Techworld with Nana

6-month program to start your career as DevOps engineer

techworld-with-nana.com

Disclaimer: I am not selling anything to you, just my personal views.

The idea for this blog comes from the Bootcamp I am currently undergoing to understand and acquire the niche skills of a DevOps Engineer by Nana Janashia. This Bootcamp can really help you to learn those skills needed for DevOps engineers.

Prequisites

I did mention earlier that it is a beginner-friendly blog, but there are a few tools which are needed to follow along.
(Don't worry if you don't know I will post relevant links to brush up wherever needed :D)

IntelliJ IDEA
Maven
Gradle
Nodejs
Git

Why build the code?

When you developed an application,(for our understanding,lets stick our discussion to java based application) it needs to be deployed on the production server so that people can consume/use it.
Usually, any application will also have dependencies too, so how to move code from the local machine to the production server with dependencies?(Entire application ( main code + dependencies) will be big in size)
To deploy the file, it is packed into a single movable file also called a package.
This single package file is called an artifact.
The process of packaging multiple files into one ( compressed form) is called building the code, tools which help us to build code are called build tools.
For java based applications the popular ones are maven, gradle.

What kind of file is the artifact?

Artifacts generated from maven or gradle are JAR or WAR file.
Strictly Speaking of maven and Gradle based artifacts includes the application code + dependencies too but in the case of nodejs-based application tools package management tools not build tools ( will be discussed later) only contains the application code, not dependencies.

What is artifact repository?

Artifact repository is place where artifacts are stored
It keeps artifact storage and makes it convenient to deploy it multiple times, have backup etc
Build tools have commands for publishing to artifact repository and then you van download from anywhere on server using wget or curl.
Artifact repository - nexus, jfrog for java-based application artifact.
Later in the blog we will see how docker helps us to use a single repository for artifacts.

Similarities Between Maven and Gradle and other tools.

All are command line tools which have the options to run the code, test the code, build the code/zip, and publish to artifact repository.
You can configure all these build tools like their build target directory, which files to include etc.

How build tool manages development and management dependencies?

We need to use build tools locally when we are developing applications in order to test and run locally.
Build too also helps us to manage dependencies by automatically installing them.
Maven dependencies are stored in pom.xml, for gradle build.gradle. ( Will be discussed later how they manage dependencies)

Let's Build some JAVA Code

For this blog, use this GitLab repository which has a sample JAVA maven application, JAVA Gradle Application and React Node and clone it to your local environment.

I would recommend using IntelliJ IDE as it will simplify the configuration process of JAVA for the project to a great extent.
JAVA Maven application
JAVA Gradle application
React-Node applicatoion

Building JAVA app using maven

maven requires us to configure how build will take place.
In our repository since it is an application which is using Sprint framework we will need to configure maven to build the code for spring boot by adding a plugin in pom.xml ( dependency resolution file for java apps)

 <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <version>2.3.5.RELEASE</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>repackage</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

To build, you can write the commands in the integrated terminal of IntelliJ.

mvc install

After the build is complete, maven places the jar file inside the target folder.

Building JAVA app using gradle

Gradle does not require any kind of configuration. Dependency resolution file for gradle is build.gradle.
To build using gradle

./gradlew build

Gradles places the JAR file under build directory instead opf target directory unlike maven.

How to run the artifact

Now we know we need to build the artifact and move it to server but how to run the artifact?

java -jar <name of the file>

Let's Manage NodeJS application

NodeJs based application usually doe not have artifact type
In fact, nodejs based application have NPM or YARN as their package management tool, they dont build anything.
They don't have a structure or standard way to do it, unlike maven or Gradle.
JS artifact are zip or TAR files.
zip/tar file only contains application and not dependencies.

In order to run JS application on production server these are the steps

1) Copy artifact ( zip file) and package.json file to production server
2) install dependencies
3) unpack zip/tar artifact
4) run the App

npm pack command # create tgz file containing package.json file too along with source code.
npm test. # for testing
npm publish # for publish to artifact repository

How to manage Building for frontend and backend

Usually, frontend and backend of an application are seperate. Common pattern include like using React for front end with backend based out of nodejs or JAVA.

In any case either a separate artifact can be produced or common artifact can also be the option, which means seperate package.json for the former if backend is also based out of nodejs.
Note: React code needs to be bundled, transpilled, and compressed. A tool called webpackwhich produces server.bundle.js
webpack also has a command line interface
To use webpack, use these commands

cd api 
npm install // for downloading webpack dependency
npm run build

Webpack produces a minified version of the entire react code

Relation between Build tools and Docker

Modern architecture involves micro services sometimes inside a single project one microservice can be JAVA based and another can be nodejs or python based. Does that mean we need to handle different artifact types?

With docker, we don't need to build and move different artifact types.
There is just one artifact type - Docker images
Now, we build Docker images from the application and we don't need a repository for each file type, no need to move multiple files to the server like package.json, no need to zip, just copy everything into the docker filesystem and run it from docker image.
Docker image is also an artifact
In order to start the application you don't need npm or java on the server, execute everything ( command to run the JAR or node application) inside the docker image.

Note - we don't have to create zip or tar files anymore because we can copy JS files into a docker image. But we still need to build apps outside the docker image and then copy it inside the docker image. For eg webpack or in the case of JAVA create a JAR file , then copy JAR file or bundle.js will create a docker image.

// for gradle app 

FROM openjdk:8-jre-alpine

EXPOSE 8080

COPY ./build/libs/java-app-1.0-SNAPSHOT.jar /usr/app/
WORKDIR /usr/app

ENTRYPOINT ["java", "-jar", "java-app-1.0-SNAPSHOT.jar"]


// for nodejs ap


FROM node:10 AS ui-build
WORKDIR /usr/src/app
COPY my-app/ ./my-app/
RUN cd my-app && npm install && npm run build

FROM node:10 AS server-build
WORKDIR /root/
COPY --from=ui-build /usr/src/app/my-app/build ./my-app/build
COPY api/package*.json ./api/
RUN cd api && npm install
COPY api/server.js ./api/

EXPOSE 3080

CMD ["node", "./api/server.js"]

Our role as DevOps Engineers

Now it's interesting to know where our knowledge as a DevOps engineers will come into the picture.

Developers uses these tools locally and configure the tools to run application and test. We don't have much to do in this step.
We come into the picture for building the artifact, because we know where and how it will run.
Developers don't build the app locally, building the artifact -> docker image -> push to repo and run on the server is the responsibility of DevOps in build automation tools like Jenkins.
Our job is to configure the CICD pipeline and build automation which includes, install dependencies -> run tests -> build app -> push to repo as a docker image.
execute tests on build servers like mvn test or npm test, build and package into the docker image.

Till then, Happy Learning !!!

Feel Free to post any comments or questions.

Connect with me over Linkedin or Twitter.

AWS CodePipeline: Build & Test with CodeBuild

Jatin Mehrotra — Fri, 06 Jan 2023 11:02:11 +0000

In this series of Build Better CI CD Pipelines, we will understand concepts around CI CD and build pipelines by actually making them.

In this continuous series I will add blogs with different CI CD tools and deploy various AWS services using the pipeline.

This is the second blog in the series. In this blog, we are going to trigger our pipeline with a commit in the CodeCommit repository followed by a building code and running test actions before deploying it to s3 as a static website.

Prerequisites

Basic knowledge of AWS fundamental services like s3, ec2, and cloud formation.
Working with git and CodeCommit.
Basic knowledge of dockers.
Fundamental knowledge of CI-CD concepts. You can read my previous blogs where I have covered CI CD 101 with GitLab, CD pipeline with S3

Motivation

AWS CodePipeline Step by Step | Udemy

Learn AWS CodePipeline with CodeCommit, CodeBuild & CodeDeploy, DevOps CI/CD on AWS from an AWS certified expert!

udemy.com

The idea for this blog comes from one of the amazing udemy courses by Emre Yilmaz. This course is a great course not only for beginners but also for experienced DevOps Engineers.

Emre Yilmaz covers all the important concepts around CI CD AWS services at the most 101 level with advanced concepts along with hands-on which reinforces the concepts.

What is CodeBuild?

In the simplest terms, AWS CodeBuild allows you to run commands on your project. It acts as a command line in your pipelines.
Its serverless, scalable and fully managed service.

How does CodeBuild execute commands?

CodeBuild uses the Docker container for the build environments. Inside these containers, you can execute commands desired for building or testing code.
Commands are mentioned in a YAML file usually called buildspec.yml(default name), if the user defines a custom name they need to mention it so that CodeBuild can find the YAML file.
We can use docker images provided by AWS or public images on the docker hub or custom images uploaded to ECR.

Note:-

We can use a standalone codebuild project and trigger than console, sdk or cli.
We can also schedule codebuild projects using cloudwatch events
Uses IAM service roles to execute codebuild project
Data in transit is encrypted and at rest using CMK managed by AWS KMS.

Let's Build the Pipeline

1. Create a CodeCommit Repository

Create a CodeCommit repository and push the sample angular application to this repository.
You can find the angular code in this repository. All thanks to Emre Yilmaz.

2. Create the Pipeline for build and unit testing

Configuring Source for the Pipeline

Under the same pane select codepipeline and add name and role name for the pipeline.
Moving forward, we need to select the source which will trigger the pipeline, this will be our code commit repository created in the previous step.
For trigger we are choosing the recommended option of using Cloudwatch Events.

Configuring Build Provider for the Pipeline

For the Build Provider we will use AWS CodeBuild.
Here we will create a new CodeBuild Project for building the angular application.

Under the new window, we will configure our codebuild project settings like docker image, OS, IAM Role, and Buildspec.

Here we are choosing Amazon managed image. provided by AWS with ubuntu as the operating system. For the Building project, we are specified to use a buildspec file which will be found by codebuild if the name is buildspec.yml by default. The structure of buildspec file is explained later.

Deploy the build app to s3

Just like the previous blog, create a production s3 bucket, we will create a bucket, enable static website hosting and configure it in the Deploy stage of CodePipeline.

3. Understanding Buildspec structure + unit testing

Before creating the pipeline, it is very important to understand how codebuild will use buildspec.yml file to build the code which means how to write the buildspec.yml file.

buildspec.yml for building angular code

A YAML file containing your commands and runtime env settings.
Can be a separate file in the source code or we can mention commands in the console.
Commands can be defined and executed in phases like install, pre_build, build, post_build
Can define multiple runtimes for commands like nodejs, ruby or python or 2.0 for amazon Linux. If you don't define runtime it will use the default runtime of the image.
Artifact configuration can also be defined in buildspec which tells which files and directories will be included in artifact.
Environment variables can also be defined as key-value pairs, but can also be referenced using the Ec2 parameter store or secrets manager.
Environment variables can be set outside buildspec for the entire project and use in your commands, which helps in reusing buildspec files across different codebuild projects.
[Imp] We can also use different buildspec files in the source code repository for different code build projects

version: 0.2

phases:
  install:
    runtime-versions:
      nodejs: 16
    commands:
      - npm install -g @angular/cli@9.0.6
  pre_build:
    commands:
      - npm install
  build:
    commands:
      - ng build --prod
    finally:
      - echo final block echo

artifacts:
  base-directory: dist/my-angular-project
  files:
    - '**/*'

Buildspec Phases

Install

Commands executed for installing packages in build env
During installation
Best Practice: runtime version are also defined in this phase

Prebuild

Commands run before the build
Use for installing dependencies, signing into ECR, docker hub etc.
If you are building Docker Images, use this phase for signing into ECR.

Build

Use for running builds and tests.

PostBuild

Command after build.
For example: sending SNS notifications using aws cli.
Pushing images to the docker repository after building it.

artifacts

This section tells codebuild where to find the artifact created by build command and use as build artifact.
We are using wildcards to recursively include all directories and their files for the build base directory

Phases are optional but one must be defined in buidspec

Adding Unit Testing to Pipeline Using CodeBuild

why do we need to add tests to pipelines?

Because we can break the logic and bad logic can ship to the product.
So ideally we should run unit tests in our code and then run the build command after them.

How to add tests in the CodePipeline

either add unit test command either in the build phase in the buildpec.yml OR the preferred way by separating unit test with build process by adding another buildspec.yml file as another codeBuild Project.
Since we want testing to take place before building we will add the action just before the build action group in the CodePipeline ( this logic has been explained in the previous blog)

Note: For testing, we are creating a separate CodeBuild Project with a different buildspec.yml file dedicated to testing, hence we need to specify the name of the buildspec file during project creation unit-test-buildspec.yml. Here we are using Managed docker image from aws but the version is 5.0 not 6.0 as earlier.**

Unit Testing Buildspec.yml

version: 0.2

phases:
  install:
    runtime-versions:
      nodejs: 14
    commands:
      - npm install -g @angular/cli@9.0.6
  pre_build:
    commands:
      - npm install
  build:
    commands:
      - ng test --no-watch --no-progress --browsers=ChromeHeadlessCI

Interesting things about buildspec.yml

How to simulate a failure in build

Sometimes we need to simulate a failure in our builds for that we can simply return a non zero value before build command
If the error happens before the build phase the execution switches to the final phase. If execution happens in the build phase it continues to the remaining phase.

 build:
    commands:
      - exit 1
      - ng build --prod

Run command irrespective of whether pipeline fails or passes

 build:
    commands:
      - ng build --prod
    finally:
      - echo final block echo

Testing Pipeline

Upload the code to CodeCommit Repository. The pipeline will run.

Finally, verify Deployment by going to the s3 website URL.

From DevOps Perspective.

In this Blog, we understood how CodeBuild works and uses docker containers to act as a command line for our code when running in CodePipeline.
We understand how we can add testing to Codepipeline
We understood how to write buildspec.yml and its various syntax.

In the next blog, we are going to learn CodeDeploy to deploy the code to ec2 instances and add further automation.

Till then, Happy Learning !!!

Feel Free to post any comments or questions.

Connect with me over Linkedin or Twitter.

Using AWS CodeCatalyst: I Created and Deployed a React SPA #reinvent

Jatin Mehrotra — Fri, 02 Dec 2022 00:30:55 +0000

In this blog I am going to introduce AWS CodeCatalyst a new service that AWS announced at re:Invent 2022.

It is a Unified software development service to quickly build and deliver applications on AWS.

CodeCatalyst project consists of blueprints which setup new software development projects without any human effort, including CI/CD, deployable code, issue tracking, and AWS services (IAM roles and policies) configured according to best practices.

You can know more about CodeCatalyst on their website .

Features of CodeCatalyst

Dev Environments: CodeCatalyst Dev Environments are automatically created with pre-installed dependencies and language-specific packages with integrated IDE.
Built-In Issue Tracking: It allows to create issues, add custom labels, and feedback on issues.
Customize pipelines: Create and customize automated workflows by configuring pre-defined actions using the desired IDE or directly editing the underlying YAML configuration. We can also use any GitHub Action in your workflows.
Built-in Code coverage, test support: built-in support for code coverage, software composition analysis, and unit tests.
Easy Deployment to AWS: Deploy applications to AWS services such as S3 or Cloudfront using CDK across accounts or regions.
Better Collaboration: Track project activity such as deployments, accepted pull requests through CodeCatalyst’s notifications and personalized activity feed.

Prerequisites

You need to create AWS Builder ID ( to know more about it )

At this moment, this service is in preview mode and only available in Oregon Region.

How to access CodeCatalyst?

CodeCatlyst can access from here.
Set up an alias for CodeCatalyst. This is a unique identifier in CodeCatalyst. People will use this alias to mention in comments and pull requests.
Create a Space: Space is where you collaborate with others on your software projects, verify your space and create it.

Let's see CodeCatalyst in action.

Create a project

Can be either with a blueprint or from scratch. For this blog, we will use a Single-page application blueprint.

After selecting the blueprint you can know more about it on the right pane. What does it do, about its architecture, which aws service components are being used and their explanation, about the IAM role, its permissions and trust policy it needs, account connections environment needs, and what resources it will create?

About SPA Blueprint

This particular blueprint This blueprint creates a single-page application (SPA) project of the web application framework chosen. You can choose either the Angular, React, or Vue web application frameworks. The project uses the AWS Cloud Development Kit (CDK) to deploy to the chosen hosting service, either AWS Amplify Hosting or Amazon Cloudfront + Amazon S3.

On the next page, give a name to the project and click on add IAM role. It will redirect to role creation and page and after creation, it will be added to your spaces and then added this role to the project.

Note: If you don't add the role to your project, workflows will not able to run as they need permissions in order to interact with various AWS services.

Under additional configurations, you can configure configurations like what kind of framework, hosting and CDK programming language, region, repository name, and stack name. Of course, these configurations will change from blueprint to blueprint.

You can also view the code and workflow defined in the blueprint.

Code tells about the code generated in the blueprint.

Workflow tells what intermediary actions to be taken for events (like PUSH/PULL PR)

Create Project.

Once the project is created you can view its repository, a README is populated automatically, and You can create a workflow, Create PR, and create Issues or Environments.

In order to deploy the application we need to run the workflow. URL for the application will be in the output of the workflow.
In this blueprint, there are 2 workflows already created:
1. For Building and testing the code on PR.
2. For running deploy pipeline for push to the Main branch.

In order to create a PR, we will create a branch. In this PR we will add the <p> tag Developed and Deployed using CodeCatalyst and then commit it with a commit message.

After PR is created workflow for build and test will run. You can even see the workflow by clicking on its link.

Once the workflow is successful, you can ask for a review and then merge it. Upon Merge it will run the DeployPipeline workflow which will deploy the application.

You can also check the code coverage and unit test details.

Access the application by URL present in workflow.

From Developer and DevOps Perspective

CodeCatalyst took me a maximum of 10 minutes ( including deployment minutes) to create and deploy which is really, really fast. If I had to do this on my own using IaC, I would have to figure out role permissions, check the documentation, set up CI/CD, and integrate all the components and development environment which would have taken a lot of time.
With this all-in-one software development service, it is very easy and efficient to develop applications on AWS.
This is going to save a lot of significant time and resources that software teams spend on various infrastructure components, and development environments, which will enable to deliver to quickly deliver new features or software updates to customers.

Do you think CodeCatalyst will help your teams? Share your thoughts or feedback over Linkedin or Twitter.

Using CodeCatalyst: I Created and Deployed a React SPA.

Jatin Mehrotra — Fri, 02 Dec 2022 00:13:22 +0000

In this blog I am going to introduce AWS CodeCatalyst a new service that AWS announced at re:Invent 2022.

It is a Unified software development service to quickly build and deliver applications on AWS.

CodeCatalyst project consists of blueprints which setup new software development projects without any human effort, including CI/CD, deployable code, issue tracking, and AWS services (IAM roles and policies) configured according to best practices.
You can know more about CodeCatalyst on their website .

Features of CodeCatalyst

Dev Environments: CodeCatalyst Dev Environments are automatically created with pre-installed dependencies and language-specific packages with integrated IDE.
Built-In Issue Tracking: It allows to create issues, add custom labels, and feedback on issues.
Customize pipelines: Create and customize automated workflows by configuring pre-defined actions using the desired IDE or directly editing the underlying YAML configuration. We can also use any GitHub Action in your workflows.
Built-in Code coverage, test support: built-in support for code coverage, software composition analysis, and unit tests.
Easy Deployment to AWS: Deploy applications to AWS services such as S3 or Cloudfront using CDK across accounts or regions.
Better Collaboration: Track project activity such as deployments, accepted pull requests through CodeCatalyst’s notifications and personalized activity feed.

Prerequisites

You need to create AWS Builder ID ( to know more about it )

At this moment, this service is in preview mode and only available in Oregon Region.

How to access CodeCatalyst?

CodeCatlyst can access from here.
Set up an alias for CodeCatalyst. This is a unique identifier in CodeCatalyst. People will use this alias to mention in comments and pull requests.
Create a Space: Space is where you collaborate with others on your software projects, verify your space and create it.

Let's see CodeCatalyst in action.

Create a project

Can be either with a blueprint or from scratch. For this blog, we will use a Single-page application blueprint.

After selecting the blueprint you can know more about it on the right pane. What does it do, about its architecture, which aws service components are being used and their explanation, about the IAM role, its permissions and trust policy it needs, account connections environment needs, and what resources it will create?

About SPA Blueprint

This particular blueprint This blueprint creates a single-page application (SPA) project of the web application framework chosen. You can choose either the Angular, React, or Vue web application frameworks. The project uses the AWS Cloud Development Kit (CDK) to deploy to the chosen hosting service, either AWS Amplify Hosting or Amazon Cloudfront + Amazon S3.

On the next page, give a name to the project and click on add IAM role. It will redirect to role creation and page and after creation, it will be added to your spaces and then added this role to the project.

Note: If you don't add the role to your project, workflows will not able to run as they need permissions in order to interact with various AWS services.

Under additional configurations, you can configure configurations like what kind of framework, hosting and CDK programming language, region, repository name, and stack name. Of course, these configurations will change from blueprint to blueprint.

You can also view the code and workflow defined in the blueprint.

Code tells about the code generated in the blueprint.

Workflow tells what intermediary actions to be taken for events (like PUSH/PULL PR)

Create Project.

Once the project is created you can view its repository, a README is populated automatically, and You can create a workflow, Create PR, and create Issues or Environments.

In order to deploy the application we need to run the workflow. URL for the application will be in the output of the workflow.
In this blueprint, there are 2 workflows already created:
1. For Building and testing the code on PR.
2. For running deploy pipeline for push to the Main branch.

In order to create a PR, we will create a branch. In this PR we will add the <p> tag Developed and Deployed using CodeCatalyst and then commit it with a commit message.

After PR is created workflow for build and test will run. You can even see the workflow by clicking on its link.

Once the workflow is successful, you can ask for a review and then merge it. Upon Merge it will run the DeployPipeline workflow which will deploy the application.

You can also check the code coverage and unit test details.

Access the application by URL present in workflow.

From Developer and DevOps Perspective

CodeCatalyst took me a maximum of 10 minutes ( including deployment minutes) to create and deploy which is really, really fast. If I had to do this on my own using IaC, I would have to figure out role permissions, check the documentation, set up CI/CD, and integrate all the components and development environment which would have taken a lot of time.
With this all-in-one software development service, it is very easy and efficient to develop applications on AWS.
This is going to save a lot of significant time and resources that software teams spend on various infrastructure components, and development environments, which will enable to deliver to quickly deliver new features or software updates to customers.

Do you think CodeCatalyst will help your teams? Share your thoughts or feedback over Linkedin or Twitter.

AWS CodePipeline : CD pipeline with s3!

Jatin Mehrotra — Sun, 27 Nov 2022 06:42:28 +0000

In this series of Build Better CI CD Pipelines we are going to understand concepts around CI CD and build pipelines by actually making it.

This will be a continuous series where I will be adding blogs with different CI CD tools and deploying various services on AWS using the pipeline.

In This blog we are going to create our first CD ( continuous deployment ) pipeline which is triggered by uploading on an s3 bucket ( source location ) and deploying a static website on another s3 bucket ( production bucket ).

Prerequisites

Basic knowledge of AWS fundamental services like s3, ec2, and cloud formation.
Fundamental knowledge of CI-CD concepts. You can read my previous blog where I have covered CI CD 101 with GitLab.

Why do we need deployment automation

Manual process involves human effort, prone to errors.
To get reviews early from the customer.
To achieve the standard, test, and repeatable process

What is CI CD?

Developers changes code frequently, several times a day, and push to the central repository.
In every code submission, automatic builds ( if it is needed) and unit tests are run.
CI starts with the source stage and finishes with the build stage.
Outcome(aim): To merge code as soon as possible so that integration errors are detected earlier and fixed before deployment.
Then CD comes and is deployed to staging( depending on the project ) for review ( here more e2e and other types of tests can occur ) and then it is manually approved.
After approval, the code is deployed to production. This is continuous delivery.
When no manual approval is required, it is known as continuous deployment.

What is CodePipeline?

CD service of AWS
Could be continuos deployment or continuous delivery
Integrates with 3rd party tools like Jenkins or add custom actions with lambda
Basically an orchestrator of CI/CD workflow because it helps in integrating various other services ( CodeBuild, CodeDeploy) within it.

Concepts around CodePipeline

source revision: source change that triggers pipeline.
stages: group of one or more actions. There can be multiple stages like the source stage, and the build stage. At a time only one source revision is processed, even if there are more source revisions.
action: task performed such as running test, building code. could be sequentially or in parallel.
transition: connection b/w 2 stages, describes the flow between them. We can enable or disable it.
artifact: file or set of files produced as a result of action which is consumed by later action in the same stage or different stage.
pipeline execution: each execution has its own id. Triggered by commit or manual release of the latest commit.

Let's Build the pipeline

Note: In this blog, we are going to use AWS Console so that everyone is on the same page.

1. Create a source bucket

Create a bucket called blog-pipeline-source, with versioning enabled and keeping other options as default.
s3 versioning should be enabled as the pipeline will detect source revisions by s3 version IDs.

2. Upload Artifact to Source Bucket

We are simply going to upload index.html and error.html files as zip to the source bucket.
Note: Make sure both the files are NOT under the subfolder, they should be under the root, otherwise in the production bucket they can't be directly accessed.

The command for creating a zip on Linux/Mac

zip -r blog-website.zip error.html index.html

The index.html and error.html are available on this gist.

3. Create a production bucket

Create a bucket blog-pipeline-production.
This bucket can be in a different region or the same region as opposed to the pipeline and source bucket.
ACL should be enabled because CodePipeline will use ACL to make objects public then only the pipeline will succeed.
Since this bucket will be used for bucket hosting, we need to uncheck block public access and enable static website hosting option.

4. Create a CD pipeline

Note: Pipeline and the source bucket should be in the same region.

Pipeline needs a service role. It is needed for permissions; To access the object in the source and deploy objects in the prod bucket.
Artifacts are stored in an s3 bucket in the same region, usually, CodePipeline will create a single bucket for all the pipelines or you can create your own bucket for storing artifacts.

In the next step, select the source for the pipeline as an s3 provider with a source bucket.
s3 object key should have the name of the zip file we are going to upload.
Change detection event helps to trigger the pipeline. Cloudwatch events are in real-time unlike making pipeline checks periodically.
In this blog, we are not going to build anything so we can skip the step, but we have the option to integrate Jenkins or CodeBuild.

In the next step we are going to set deployer options with the deploy provider being s3 and the production bucket as the target.
Extract file before deployment is important to set so that pipeline unzips the artifact before deployment.
We are going to set cannel ACL as public-read so that our file ( index.html) is available for the public to read since we are hosting it as a static website.
Note: CodePipeline deploys all objects to s3 buckets as private.
There is an option to encrypt production bucket objects too for extra security, if no kms-key is provided there are going to be unencrypted.

After creation, the pipeline automatically runs for the first time.

Go to production bucket and under properties, -> static website hosting -> copy the website bucket endpoint

Access the website by the endpoint URL, Wohoo, you just automated your website with CodePipeline.

5. Automatically trigger on uploading bucket.

What is a stage?

Stages hist the action, here for our pipeline we have 2 stages called Source and Deploy

What is an Action?

Action by the name, is a step that performs some operations.

What are Transitions?

They connect stages.
By default, all transitions are enabled.
We can disable transition.
Stages run fast because we use source and deploy buckets in the same region.
Each pipeline has a unique execution ID, A successful transition means execution IDs are passed from one stage to another hence they both have the same execution ID.
Let's say we want to update our website and trigger the pipeline automatically, all we have to do is just upload the updated zip package to s3.
Let's make some changes to index.html. Adding v2 to to h1.

<html>
    <head>
        <title>Static Website for CodePipeline Blog v2</title>
    </head>
    <body style="text-align:center;font-family:Verdana;margin:1%;">
        <h1 style="color:rgb(128, 55, 0)">This is deployed by CD pipeline! V2</h1>
        <hr>
        <p>Automate anything and everything!!!</p>
        <h2>Made by Jatin Mehrotra!</h2>
    </body>
</html>

Run the zip command again.

zip -r blog-website.zip error.html index.html

Upload to s3, and the Pipeline will be triggered again with a different execution ID.

Confirm the changes on the website too.

Add new action and trigger manually.

Note: Adding actions to the existing pipeline does not trigger the pipeline automatically as it was triggered automatically upon creation. Therefore we need to trigger the pipeline manually.

In the new action we will :
- Deploy buckets can be in other AWS regions.
- Create a bucket in another region and deploy the website to it for higher availability. This new bucket will be in Singapore as opposed to the Tokyo region like other buckets.
All the bucket configuration remains like the production bucket except for the name and region.
Don't forget to enable website bucket hosting.

To add the action, we need to edit the pipeline

Edit the Deploy stage since we are adding action to deploy in the production bucket.

As I mentioned earlier, actions can be executed sequentially or in parallel. The position of add action group is critical because the box before the deploy action symbolizes execution sequentially before the deploy action. Action group beside deploy action group tells execute the action in parallel. The box of add action group below deploy action means to add an action that will run sequentially after deploying action.

We are adding action in parallel so we will choose to add action beside the deploy action.

Define action provider as deploy s3.
Define input artifact, in this case, it is coming from sourceArtifact defined by source.

Save pipeline changes.

## How to trigger manually without uploading to an s3 bucket

Click on release change.

We can see the start pipeline execution trigger.

Clearing the environment.

Deleting code pipeline does not delete a source, deployment, and artifact buckets.
Artifact bucket will be used by all pipelines so no need to delete it.
Roles are not deleted, even the policies created by roles.
Different roles for different policies are the best practice.

From DevOps Perspective.

We have successfully achieved the automation of deploying static website using s3 as a source bucket with our first CD pipeline using CodePipeline.
We did achieve automation, however, there were still manual process elements like creating s3 which can be further automated using Terraform.

Till then, Happy Learning !!!

Feel Free to post any comments or questions.

Connect with me over Linkedin or Twitter.

DevOps Guru: A new way (AIOps) to fix your AWS Infrastructure issues

Jatin Mehrotra — Wed, 23 Nov 2022 08:05:27 +0000

An application deployed in production always has some kind of monitoring and observability to track its performance, operational issues, latency, downtime, error rates, outages, service disruptions, infrastructure code, config changes and ...the list can go on.

Often monitoring and observability operations do not lead to an efficient outcome because there is a lot of noise and false negatives due to lack of information, alarm fatigues etc.

For a DevOps or SRE, this results in spending days of time and effort to detect, debug and resolve operational issues, which is definitely not good from a business critical application and customer point of view.

DevOps Guru is an ML-powered service to address the above-mentioned issues and save your precious hours of detecting and resolving operational issues leading to enhanced application availability and reliability.*

Prequisites

NO ML KNOWLEDGE/EXPERTISE IS REQUIRED. (Trust me when I say this, AWS has made it so easy for us )

What is DevOps Guru?

A service that sends you alerts and recommendations whenever there is an anomaly in the operating patterns that can cause application downtime or service disruptions.
DevOps Guru uses ML models learned from more than 20 years of operational expertise in building, scaling, and maintaining universally available applications for Amazon.com.. This leads to Reactive and Proactive insights, so you can identify operational issues long before they impact your customers.
Reactive Insights: Recommendations to address issues that are happening now.
Proactive Insights: Recommendations that address issues that DevOps Guru predicts will occur in the future (power of ML).

Why to use DevOps Guru?

Saves tons of time, reduces application downtime and ultimately leads to happy customers because it pinpoints the problems with its summary about why the issue took place and how to fix it through an actionable recommendation.

How does DevOps Guru work?

Baseline Creation: It established a baseline which is treated as "normal". For this process, it consumes metrics like latency, error rate, and request rates.
Using a pre-trained ML model, when it identifies an anomaly, it generates alerts with its recommendations for its remediation.

Use Cases for DevOps Guru

Amazon DevOps Guru for RDS helps in detecting performance & operation issues within RDS based engines.
Amazon DevOps Guru for Serverless to detect and diagnose performance and operational issues for Serverless Applications built using AWS
Fix your AWS infrastructure misconfigs and limits using proactive insights.

How to Enable and disable DevOps Guru and check estimation?

To enable, Just click Enable. Yes, that's how easy it is and on top of that, you can configure the SNS topic to send alerts and take action :)

To disable, Again Just click None on resources to analyse, you can decide which resources to analyse based on tags, or cFn stacks or all resources :)

To estimate cost, you can again choose, either on cFn stack, tags or all resources.

Demo Time for DevOps Guru

To illustrate the power of DevOps Guru, we will work with RDS and serverless application which is already deployed using this workshop from AWS

Go to cloud 9, to access the env for this blog.

DevOps Guru for RDS

Scenario 1: There is a mission-critical production application which uses Aurora relational database with the PostgreSQL engine. The application has been updated with a new feature and users are complaining about slow performance and time out.
Simulation of Scenario: We will try to increase the read workload on DB.

PGPASSWORD=${PGPASSWORD3} psql -h $PGHOST3 -U $PGUSER3 -p $PGPORT3 -c "CREATE EXTENSION pg_stat_statements;"

PGPASSWORD=${PGPASSWORD3} pgbench -h $PGHOST3 -U $PGUSER3 -p $PGPORT3 -c 10 -T 1800 -j 10 $PGDATABASE3

These commands are using 10 connections with 10 threads.
After a certain time, we can see insights from DevOps Guru.
DevOps Guru Insights: Select the insight and view its details. Here we can an overview about the insight, information about the metrics as well as graphed anomalies.

It will show which metric is affected in this case ( Average active sessions ( AAS) ), DevOps Guru also shows analysis and recommendations which pinpoints the issue.

It also tells the reason and recommendation behind this anomaly to give better insights about the cause.

The query requires aid ( an index ), which is not present in the table (all thanks to DevOps Guru). The takeaway here is that a missing index can ruin application performance and keep the entire system busy.
Scenario 2: Team member took a coffee break but forgot to commit/close the session on RDS which he has opened which leads to an exclusive lock on the table.
Simulation of the Scenario: We will try to use multiple sessions competing for the same "locked" record and of course due to ACID properties of the transaction, they must wait for each other.
Team member work

cat > workload-lock-txn2.sql << ENDOFFILE

begin;
lock table hr.job_history in exclusive mode;


ENDOFFILE

Our workload

cat > workload-lock-txn1.sql << ENDOFFILE

\setrandom counter 2 200
\setrandom sleeptime 1 5

begin;
select start_date,end_date from hr.job_history where employee_id=5550 FOR UPDATE;
SELECT pg_sleep(:sleeptime);
update hr.job_history set start_date=(select current_date - :counter days), end_date=current_date where employee_id=5550 ;

ENDOFFILE

Running the above workload will create a lock condition.

PGPASSWORD=${PGPASSWORD2} pgbench -h $PGHOST2 -U $PGUSER2 -p $PGPORT2 -d $PGDATABASE2 -f workload-lock-txn1.sql -f workload-lock-txn2.sql -c 120 -T 1800 -j 30 -n

Just like the previous insight, DevOps Guru is smart enough to find the cause and suggest a reason behind the anomaly.

DevOps Guru For Serverless

Scenario 1: As business needs are changing, the production application which was running on virtual machines have been moved to AWS serverless. In the production DynamoDB, there is a misconfiguration of not enabling point-in-time recovery.
DevOps Guru not only sends reactive insights but it also provides proactive insights ( takes about 12 hours once enabled) which we can't foresee in the status quo.
Check the proactive insights tab, to see the insight recommendation.

Scenario 2: We will reduce Read Capacity from 5 to 1 of DynamoDB and update via Cloudformation.

cd ~/environment/amazon-devopsguru-samples/generate-devopsguru-insights
aws cloudformation update-stack --stack-name myServerless-Stack \
    --template-body file:///$PWD/cfn-shops-monitoroper-code.yaml \
    --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM

Now we will try to inject an HTTP request to API using 4 instances of the python script. After some time we can observe 502 internal server errors.

python sendAPIRequest.py &amp; python sendAPIRequest.py &amp; python sendAPIRequest.py &amp; python sendAPIRequest.py

DevOps Guru monitors and reports this anomaly with the cause of the anomaly and applicable recommendations.
We can also confirm the relevant event which caused this issue.

From DevOps Perspective

DevOps guru removes the need for manual monitoring of applications and infrastructure on AWS.
With the use of pre-trained ML models, it indeed fulfils the essential requirement of a production environment where it generates insights and recommendations for real-time application and infra-related issues as well as for future operational issues which cannot be anticipated at this time.

I can definitely say it will surely help to achieve the high availability and operational excellence for AWS infrastructure that you promise to your customers.

Till then, Happy Learning!

Would love to know your opinions on this, let's connect! (@imjatinmehrotra, Linkedin )

CI CD 101 with GitLab

Jatin Mehrotra — Thu, 28 Jul 2022 03:30:00 +0000

This blog talks about fundamental DevOps practices to automate continuous integration and continuous deployment using GitLab.

This is a beginner-friendly blog that assumes you do not know anything about GitLab and CI-CD.

Outcome of the blog

By the end of this blog, you will be confident to automate 3 major components of any CI-CD pipeline

Running test cases
Building a docker image
Deploying it to infrastructure.

Bonus:- Pushing docker image to the repository

Motivation

The idea for this blog comes from my favourite DevOps Teacher who makes it so easy to learn and practice DevOps skills and she is none other than Nana Janashia. Subscribe to her youtube channel, to become a better DevOps Engineer and stay with the latest updates on the industry.

Prerequisites

I did mention earlier that it is a beginner-friendly blog, but there are a few basics which can really help you understand these concepts really well. (Don't worry if you don't know I will post relevant links to brush up wherever needed :D)

Git
Docker
Linux
AWS Account

Let's Build the pipeline

For building the pipeline, use this GitLab repository which has a sample python application and clone it to your local environment. You can also fork it and edit in pipeline editor but it's always advisable to work in a local editor like vs-code.

Gitlab Repository which hosts application code and pipeline configuration

1. Test and run application locally

As a DevOps Engineer, you don't need to know all the details of the application code, knowledge of how to run and test the application is important. So depending on the application, you should ask your dev team, how to run and test the application locally.
For this particular example application in order to test we use the following command

make test

In order to run the application locally, we use the following command

make run

Key Points to Note

You need to know from the dev team how to run and test the application.
We are using make command since it's a python application for node-js-based applications we might have to use the npm command. So availability of commands where we will run the application is very important.

2. Gitlab CI CD internals

GitLab is one platform which allows doing everything in DevOps perse, planning, CI-CD, and deployment to any cloud which really makes it efficient as there is a single interface to deal with in SDLC.
Nana Janashia jas given a great explanation in this video about GitLab architecture, though I will try to explain my understanding of the same so that it's easy to follow up in the blog
GitLab has 2 components:-

Gitlab instance/Server (managed & self-managed) -> Hosts app code and pipeline config
Gitlab Runners (managed & self-managed) -> connected to GitLab servers, runners are agents that run your CI-CD jobs.
- Runners run on a separate machine ( Linux, windows) wrt to the GitLab instance.

For this blog, we will use managed components as it is easier to manage and operate.

3. Let's Build our CI-CD pipeline

In order to create a pipeline, GitLab requires us to create a file with the name .gitlab-ci.yml in the root of the git repository.
Inside this file, there will be commands in the form of scripts which are grouped into jobs like running tests, building docker images or deploying could be 3 different jobs.
Till this point, we know there are jobs inside the pipeline which perform a task, within the jobs, there is a script which consists of commands, but the immediate question is where do commands will be executed?

Commands are executed by GitLab runner's execution environment. There are 2 kinds of execution environments.

GitLab Runner is installed on different OS like Linux, and Windows; this is called a shell executor.

Instead of a shell executor, we can use a Docker container running on Linux machines. Each job will be run in a separate and isolated container.

Managed runners use docker container as default execution environment with the ruby image as default for that container. Of course, we can choose the docker images of our choice depending on the job and use cases.

For running our test case in the pipeline we need make, pip and python available in our execution env, for that we will use python docker image (python:3.0-slim-buster).
Here is the job definition

run_tests:
  image: python:3.9-slim-buster
  before_script:
    - apt-get update && apt-get install make
  script:
    - make test

Semantics :

run_tests: name of the job.
image: name of the docker image for executing commands.
script: commands which need to be executed.
before_script: any commands which need to be installed before running script commands in this case we need the make command to be installed to run our test.

Commit your changes and push them to the remote repository, GitLab will automatically trigger the pipeline to be executed. ( quite an ease )

4. Building docker image of our Application

We want to deploy this application to AWS on ec2 as a docker container, for that we need to make a docker image out of this application and push it to the docker registry. For registry we are going to use dockerhub, aws ecr can also be used for this purpose.
Here is the job definition for building a docker image

variables:
  IMAGE_NAME: jjatdevops/gitlab-ci-cd-python-app
  IMAGE_TAG: python-app-1.0

build_image:
  image: docker:20.10.17
  services:
    - docker:20.10.17-dind
  variables:
   DOCKER_TLS_CERTDIR: "/certs"
  before_script:
    - docker login -u $REGISTRY_USER -p $REGISTRY_PASS
  script:
    - docker build -t $IMAGE_NAME:$IMAGE_TAG .
    - docker push $IMAGE_NAME:$IMAGE_TAG

In this job, we are using variables and docker insider docker container (dind).

Variables can be at the job level that is within the job, accessible only inside the job, or can be at the top level like here, which are accessible across all jobs or at the global level ($REGISTRY_USER, $REGISTRY_PASS) accessible across different repositories.

In order to build Docker images we need to configure GitLab runner to support docker containers, for this we need docker inside docker(dind)

docker:20.10.17 provides docker client which will build image and push to the registry but we also need docker daemon which is a process that allows docker client to execute the command, which is provided by docker:20.10.17-dind container.

Semantics :

services - services in GitLab are parallel container attacher to job container ( inside same network) when job container needs another container to carry out its operation. This is much faster than installing the dependency in the job container.
DOCKER_TLS_CERTDIR - This allows client and daemon to talk to each other by authenticating to each other using a common certificate as mentioned in the docs ( find TLS on the page)
docker login & docker push - First it logins to the docker hub registry ( since it's a private repository ) and pushes the image to the repository.

After commit and push, GitLab will automatically trigger the pipeline.

5. Introducing stages to Pipeline

Till this moment whenever we push both run_tests and build_image jobs run in a parallel fashion. In most of the cases, you want sequential execution of jobs which means if running tests cases job is successful then only build the docker image.
Sequential execution can be achieved in GitLab using stages.
Using stages we can group related jobs into stages that run in the defined order.
Here is the job definition after introducing stages to the pipeline

stages:
  - test
  - build


run_tests:
  stage : test
  image: python:3.9-slim-buster
  before_script:
    - apt-get update && apt-get install make
  script:
    - make test

build_image:
  stage: build
  image: docker:20.10.17
  services:
    - docker:20.10.17-dind
  variables:
   DOCKER_TLS_CERTDIR: "/certs"
  before_script:
    - docker login -u $REGISTRY_USER -p $REGISTRY_PASS
  script:
    - docker build -t $IMAGE_NAME:$IMAGE_TAG .
    - docker push $IMAGE_NAME:$IMAGE_TAG

All jobs in the test execute in parallel ( here we have a single job ).
If all jobs in the test succeed, then only build jobs execute in parallel.

6. Deploying to Ec2 instance

In the final step for pipeline configuration we are going to run the docker image pushed to docker hub in ec2 instance running in AWS cloud.
We will manually start an ec2 instance with 5000 ports allowed on the security group with a new ec2 key pair.
After starting the ec2 instance we need to prepare the ec2 instance in order to run the docker container.
ssh into ec2 instance

ssh -i "your-ssh-key.pem" ec2-user@ec-your-ip-adress.ap-southeast-1.compute.amazonaws.com

Install docker


sudo yum install docker

Add group membership for the default ec2-user so you can run all docker commands without using the sudo command

sudo usermod -a -G docker ec2-user

Enable docker service at AMI boot time

sudo systemctl enable docker.service

Start the Docker service

sudo systemctl start docker.service

We will add $SSH_KEY as file variable in the GitLab file variable. Contents of this variable will be the private key used for creating an ec2 instance.
Note :- GitLab create a temporary file from this variable.
Here is the job definition for deploy

deploy:
  stage: deploy
  before_script:
    - chmod 400 $SSH_KEY
  script: ssh -o StrictHostKeyChecking=no -i $SSH_KEY ec2-user@ec2-3-1-100-185.ap-southeast-1.compute.amazonaws.com "
          docker login -u $REGISTRY_USER -p $REGISTRY_PASS &&
          docker ps -aq | xargs docker stop | xargs docker rm || true   &&
          docker run -d -p 5000:5000 $IMAGE_NAME:$IMAGE_TAG"

Semantics:-

ssh -o StrictHostKeyChecking=no -i $SSH_KEY ec2-user@ec2-3-1-100-185.ap-southeast-1.compute.amazonaws.com - connect to ec2 instance and run the following commands.
docker login -u $REGISTRY_USER -p $REGISTRY_PASS && docker ps -aq | xargs docker stop | xargs docker rm || true && docker run -d -p 5000:5000 $IMAGE_NAME:$IMAGE_TAG" - run the docker container on the ec2 instance

From DevOps Perspective

We have successfully achieved the automation of various SDLC processes like running tests, building docker images, pushing them to the container registry and deploying it to the cloud.
With this knowledge, you can successfully implement any CI-CD pipeline, it doesn't matter if it is JAVA based application or a nodejs-based application.
_We did achieve automation, however, there were still manual process elements like creating ec2 which can be further automated using terraform and guess what it will be part 2 of this blog. _

Note:- GitLab platform offers much more features when it comes to CI-CD, feel free to check out the Dive deeper in GitLab course from NANA. and become an expert at GitLab.

Till then, Happy Learning !!!

Feel Free to post any comments or questions.

Let's Terraform IT: Pt 4 -> Count (index), Conditional Expression, Local Values

Jatin Mehrotra — Sat, 18 Jun 2022 08:02:00 +0000

By this time we know how to write a terraform code for simple infrastructure, but in reality, things are never easy, business needs are everchanging leading to complex infrastructure.

In this blog we will try to build our knowledge on features offered by terraform which helps to transform complex code into simple code and remove redundancy.

From now on, in this series, I want to introduce the topic with a use case because I believe that understanding a concept is one thing but knowing how to apply it in the real world makes the actual difference.

Count and Count Index

Use Case

Until now, we have been creating single resources for example an ec2 instance or an IAM user, but in production-grade applications, there are multiple resources, what would be an ideal way to create multiple resources of the same type? for instance multiple ec2 instances!!!

One approach would be to write 3 identical ec2 resource blocks with different local resource names.

provider "aws" {
  region     = "us-west-2"
  access_key = "your-access-key"
  secret_key = "your-secret-key"
}

resource "aws_instance" "instance1" {

  ami           = "ami-0ca285d4c2cda3300"
  instance_type = "t2.micro"
}

resource "aws_instance" "instance2" {

  ami           = "ami-0ca285d4c2cda3300"
  instance_type = "t2.micro"

}

This approach is not scalable if we are needed to create 40 ec2 instances, which will result in 40 ec2 resource blocks, updating such blocks would be prone to errors and difficult.
Terraform provides a count parameter which allows creating many resources using a single resource block, hence easier updation.
The following code allows us to create 5 ec2 instances using the count parameter.

resource "aws_instance" "instance" {

  ami           = "ami-0ca285d4c2cda3300"
  instance_type = "t2.micro"
  count         = 5
}

Count Index

Use Case

Count parameter solved our problem but it also brings its own set of problems with it. We want to create 5 IAM users, an ideal approach to this problem would be to use the count parameter, however using count would result in 5 iam users with the same name, hence it is difficult to identify resources when the count is used.

resource "aws_iam_user" "users" {
  name  = "iam_users"
  count = 3
  path  = "/system/"
}

this result in 3 iam users with the same name iam_users leading to difficulty in the identification of resources

terraform provides a count.index value which behind the scenes, access count parameter (count runs a loop) index value to identify the loop cycle. This helps in generating unique values with count parameters as count.index will always be unique for a particular loop cycle ( for example if the loop is running the first time it will be 1 and so on)

resource "aws_iam_user" "users" {
  name  = "iam_users.${count.index}"
  count = 2
  path  = "/system/"
}

This results in 3 iam users with the name iam_users.1, iam_user.2.

count.index with count can be combined together using array data type.

variable "names" {
  type    = list(any)
  default = ["admin", "dev"]
}

resource "aws_iam_user" "users" {
  name  = var.names[count.index]
  count = 2
  path  = "/system/"
}

This results in 2 iam users with the name admin, dev.

Behind the scenes with Count and Count Index

Count internally runs a loop and the count index maps the count of the particular loop.

Conditional Expression

Use Case

We need to launch an ec2 instance for the production if a particular variable exists else for the development.

Conditional expression ? : helps us to select a value depending on the condition.

 count         = var.istest ? 1 : 0

If is isTest variable is true, the count will be 1 otherwise 0.

resource "aws_instance" "dev" {

  ami           = "ami-0ca285d4c2cda3300"
  instance_type = "t2.micro"
  count         = var.istest ? 1 : 0
}

resource "aws_instance" "prod" {

  ami           = "ami-0ca285d4c2cda3300"
  instance_type = "t2.large"
  count         = var.istest ? 0 : 1
}

variable "istest" {
  type    = bool
  default = true
}

This code block results in the creation of t2.micro instance if istest = true else t2.large instance for production if istest = false ## Locals

Use Case

We need to same name tags for 2 ec2 instances and ebs volume. One approach would be to write tags in 2 resource blocks for ec2 and in one resource block for ebs volume. This will work however, this will lead to writing code which is repetitive and if in the future the name tag needs to be changed it needs to be replaced in 3 different places. Quite inconvenient!!!!

An ideal way would be to define it in one place and reference it in all the places.

Locals assign a name to an expression or value and allow it to be used in multiple places.

provider "aws" {
  region     = "us-west-2"
  access_key = "your-access-key"
  secret_key = "your-secret-key"
}
locals {
  common_tags = {
    Owner   = "DevOps Team"
    service = "backend"
  }
}
resource "aws_instance" "dev" {
  ami           = "ami-082b5a644766e0e6f"
  instance_type = "t2.micro"
  tags          = local.common_tags
}

resource "aws_instance" "db-dev" {
  ami           = "ami-082b5a644766e0e6f"
  instance_type = "t2.small"
  tags          = local.common_tags
}

resource "aws_ebs_volume" "db_ebs" {
  availability_zone = "us-west-2a"
  size              = 8
  tags              = local.common_tags
}

This code allows us to use the same tags across all resources without repeating the same code in all the resources.

Bonus Accessing data from maps and list in the variable

variable "types" {
  type    = list(any)
  default = ["t2.micro", "t2.large"]
}


resource "aws_instance" "myFirstEc2Instance" {

  ami           = "ami-0ca285d4c2cda3300"
  instance_type = var.instance[0]
}

list values start with index 0 which means instance type t2.micro will be selected.

provider "aws" {
  region     = "us-west-2"
  access_key = "your-access-key"
  secret_key = "your-secret-key"
}

resource "aws_instance" "myFirstEc2Instance" {

  ami           = "ami-0ca285d4c2cda3300"
  instance_type = var.instance["delhi"]
}


variable "instance" {
  type = map(any)
  default = {
    "mumbai" = "t2.large"
    "delhi"  = "t2.micro"
  }
}

To access the map data type value of the key is used to access its value within []

From a DevOps perspective

Count, conditional, and local values are the tools which help to write clean and non-repetitive code, which helps to scale easily when the requirements are changing.

Let's Terraform IT: Pt 3 -> Provider Versioning, Output, Attributes, Variables and Data types

Jatin Mehrotra — Sun, 12 Jun 2022 14:41:36 +0000

We are in part 3 of this series (part 1 , 2 ) and now the real fun with terraform begins.

In this blog, our main focus will be on Production level Best practices with terraform? But wait, we are still in 101 level and talking about production level? Yes, because even at the 101 level, this series tries to inculcate, and grow the sense of production-level practices right from the beginning.

Prerequisites

Basic AWS knowledge
Terminal Access
Part 1
Part 2

Provider Versioning

terraform {
  required_providers {
    github = {
      source  = "integrations/github"
      version = "4.25.0-alpha"
    }
  }
}

Terraform has 2 kinds of providers one which is managed by terraform and one which is managed by the provider itself. For terraform managed providers we don't need to write provider block for terraform managed providers like aws.
But from terraform 0.13 and above, terraform recommends using provider block for all kinds of providers.

terraform uses the provider's underlying plugins to interact with the provider and provision infrastructure. Let's say today the version of provider plugins is v1 and some new service came and provider included that service insider new plugin version v2.
In this case, we need to upgrade our provider version too in order to use that service.

How does provider block work?

From the official docs of aws provider this is how terraform recommends using AWS provider

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
      version = "4.18.0"
    }
  }
}

provider "aws" {
  # Configuration options
}

This piece of code will download plugins for version 4.18.0

terraform init

terraform init

this command produces a lock file .terraform.lock.hcl which maintains the version of your provider plugins.
in order to upgrade to the latest version of plugins, we need to update the version in our lock file and use init command with -upgrade flag

terraform init -upgrade

Note:-

if you don't use -upgrade flag lock file won't allow using the upgraded version even though you mentioned it in the code.
You many even see symbols like version = ~>3.0 or >=3.0 or <=3.0
~> simply means download provider plugin in 3.x range
>= simply means download provider plugin ranging = or greater than 3.0 range
**<= **simply means download provider plugin ranging = or lesser than 3.0 range

Production Importance

It is considered to use fix version for example version = "4.18.0", because there may be a chance when you use ~> that terraform installs the latest plugin version which has bugs and your production may break. So it is always advised to use the tested version and then upgrade to newer plugin versions.

Outputs and Attributes

When we are provisioning infra, there is always a need to know the attributes of provisioned infra for example deployed ec2 IP address or EIP public IP.
Terraform allows doing this by writing output block for the particular resource.
code.tf file

provider "aws" {
  region     = "us-west-2"
  access_key = "your-access-key"
  secret_key = "your-secret-key"
}


resource "aws_instance" "myFirstEc2Instance" {

  ami           = "ami-0ca285d4c2cda3300"
  instance_type = "t2.micro"
}


output "ec2_ip" {
  value = aws_eip.lb.public_ip
}

output block's has the format of output name followed by value = resource_name.local_resource_name.atrribute_name.
attributes for each resource can be found in the attribute reference documentation.

output "ec2_ip" {
  value = aws_eip.lb.public_ip
}

the same format of resource_name.local_resource_name.atrribute_name can be used to cross-reference resources for example associating eip to ec2.

resource "aws_eip_association" "eip_assoc" {
  instance_id   = aws_instance.myFirstEc2Instance.id
  allocation_id = aws_eip.lb.id
}

Variables

Why do we even need it?

Just like any other programming language, when you want to use a single value, hardcoding it will be fine, but if the same value is being used in 10 other places then if in future that value changes; it needs to be updated in 10 other places too.
So an ideal practice is to store value in a central place and then reference it in our code. If in future that value changes, just change in source.

How to store and reference variables?

There are 4 ways to store variables:-

Default
Command-line arguments
file/custom file
Environment

Default variables

Going by the name, if command-line or environment or from a file, variables are sources, then default variables will be used.
In order to use default variables, create a separate file variables.tf

variable "instance_type" {
  default = "t2.micro"
}

In order to access a variable in our code, var.name_of_variable

resource "aws_instance" "myFirstEc2Instance" {

  ami           = "ami-0ca285d4c2cda3300"
  instance_type = var.instance_type
}

Note:- *If there is variable defined in variables.tf, and there is no default value, then terraform will show a prompt at the time of terraform apply
*

variable "instance_type"{}

Command-line args

Command-line arguments are used to add values for testing.
To add command line arguments -var="var_name=value"

terrform apply -var="instance_type=t2.large"

Note:- If the default variable is present it will override by the command line argument.

From File

Now default variables are used for default when no value is used, but usually, in production, there is always a value specified.
In order to source variables from a file create a file called terraform.tfvars ( naming convention is important)

instance_type = "t2.large"

Note :-

to use a custom file name you need to use -var-file flag

terraform aply -var-file="custom_file.tfvars"

Environment variables

There is another way to source variables i.e by using them as environment variables.
syntax is as follows

export TF_VAR_instance_type="t2.nano"

Note:- if you are using environment variables then make sure there is no value present terraform.tfvars otherwise, the export variable value will be overwritten by the tfvars value.

Data type of variable

Let's say there is a condition to create a iam user which by employee id, which only contains the number.
To add this constraint we need, data type defined as numbers inside variables.tf file
variables.tf file

variable "user_name" {
  type = number
}

code.tf file

provider "aws" {
  region     = "us-west-2"
  access_key = "your-access-key"
  secret_key = "your-secret-key"
}

resource "aws_iam_user" "user" {
  name = var.user_name
}

terraform.tfvars file

user_name = "jatin123"

This will result in an error because user_name expects a number rather than a string.

However, if we use a number that works.

user_name = "123"

There are 5 kinds of data types we can use string, number, bool, list, and map. reference

From a cloud engineer perspective

In this article, we saw best practices when it comes to provider versioning.
Different ways to input variables.
How to type-safe our variables using data types.

Feel free to post your questions or suggestions in the comments, let's learn together.