The Ops Community ⚙️: Jeiman Jeya

Let's Dive into AWS CloudFront

Jeiman Jeya — Wed, 05 Apr 2023 03:15:00 +0000

What is AWS CloudFront?

AWS CloudFront is a content delivery network (CDN). It securely delivers data, videos, applications, and APIs to customers globally with low latency and high transfer speeds. It offers a simple and cost-effective way to distribute content to end users, with features such as edge caching and SSL/TLS encryption.

How does it work?

When a user or customer requests content, CloudFront routes the request to the edge location closest to the user in that region, delivering the content with low latency and high transfer speeds. CloudFront can also be used to encrypt content, protect against DDoS attacks, and integrate with other AWS services such as S3, Application Load Balancer, Elastic Beanstalk, and more.

Components in AWS CloudFront

Distribution

When you want to deliver or distribute your content to a location, you create what is called as a distribution. With a distribution, you can choose how to deliver your content to your end users based on configuration settings:

Content origin: Your source of information, that is, an S3 bucket, an Elastic Load Balancer, an Elastic Beanstalk server from which CloudFront gets the files to distribute
Origin request: If you require CloudFront to use a specific set of HTTP headers, cookies or query string in the requests that are sent to your origin
Access: Whether you want your content to be accessible by everyone or restrict access to certain users
Security: If you want your users to access your content using HTTPS and the various encryption protocols (TLS v1.1, v1.2)
Logs: Tell CloudFront to create standard logs or real-time logs that show viewer activity
Geographic restrictions: CloudFront can prevent users from selected countries in accessing your content

CloudFront provides flexibility in configuring your CDN for any web application according to your business or engineering needs.

Functions aka Lambda@Edge

Lambda@Edge is a feature of AWS CloudFront that allows you to run code closer to your application's users, improving performance and reducing latency. With Lambda@Edge, you don't need to provision or manage infrastructure in multiple locations worldwide. You only pay for the compute time you consume on that Lambda application, and there is no charge when your code is not running. By using Lambda@Edge, you can enhance your web applications by making them globally distributed and improving their performance, all without any server administration. Simply upload your code to AWS Lambda, which takes care of everything necessary to run and scale your code with high availability at an AWS location closest to your end user.

One use case for this is to introduce an HTTP Basic Authentication layer on your AWS services, such as S3 buckets or ECS Fargate, to provide an additional layer of security for end users accessing your application. When users access the website, they will be prompted with a native Basic auth form. Based on the logic defined in the Lambda function, you can approve or deny access. From there, simply attach the Lambda to your CloudFront distribution and every user will be prompted to enter login credentials to access content on your application.

Invalidation

Invalidation is the process of instructing CloudFront to purge all content from your distribution, refreshing it for your end users in all edge locations. The next time an end user requests a file or page, CloudFront returns to the origin to fetch the latest version of the file.

This is especially useful if you are running web applications on S3 or an HTTP server and want your users to receive the latest updates immediately, without waiting for the original TTL on the file, which could be up to 30 days. By default, CloudFront sets a TTL of 24 hours for all files, meaning that after 24 hours, it fetches new content from your origin. With invalidations, you can define which types of files, pages, or HTTP paths you would like to invalidate. This provides flexibility to meet your engineering needs.

Policies

CloudFront allows you to define the type of policy in your distribution. CloudFront offers 3 distinct policies:

Specify cache and compression settings: You can define which HTTP headers, cookies, and query strings CloudFront includes in the cache key with a CloudFront cache policy. The cache key is used to determine whether a viewer's HTTP request results in a cache hit (i.e., whether the object is served to the viewer from the CloudFront cache). Including fewer values in the cache key increases the likelihood of a cache hit. You can also specify TTL settings for objects in the CloudFront cache, enabling CloudFront to request and compress that object for your end users.
Specify values to include in origin requests: With a CloudFront origin request policy, you can specify the HTTP headers, cookies, and query strings that CloudFront includes in origin requests. These are the requests that CloudFront sends to the origin when there is a cache miss for your content.
Specify HTTP headers to remove or add in viewer responses: Using a CloudFront response headers policy, you can control the HTTP headers included in HTTP responses that CloudFront sends to viewers (web browsers or other clients). You can add or remove headers from the origin's HTTP response without modifying the origin or writing any code. All of this can be handled through CloudFront.

Above is an example of a CloudFront distribution with 2 different configurations of serving WordPress content from an S3 bucket and an Elastic Load Balancer connected to an EC2 instance.

Advantages of using AWS CloudFront

High performance: AWS CloudFront delivers content with low latency and high transfer speeds, improving user experience and reducing load times.
Flexible distribution options: CloudFront offers a range of configuration settings for content delivery, including geographic restrictions, access control, and security features.
Cost-effective: CloudFront offers a pay-as-you-go pricing model, making it a cost-effective solution for content delivery.
Integration with AWS services: CloudFront integrates seamlessly with other AWS services, such as S3, Elastic Beanstalk, and Application Load Balancer, making it easy to distribute content from these services.
Lambda@Edge: The Lambda@Edge feature allows developers to add custom code to CloudFront, improving application performance and reducing latency.

Disadvantages of using AWS CloudFront

Steep learning curve: CloudFront can be complex to set up and configure, requiring a significant amount of time, focus and expertise.
Limited content size to cache: CloudFront has a limit on the size of content that can be cached, which may not be suitable for larger files or applications that is above 30GB in size per file.
Potential cost overruns: While CloudFront is cost-effective, it can be easy to exceed usage limits, leading to unexpected costs. Keep a close eye on your cache requests and usage.
Limited geographic coverage: While CloudFront has a global network of edge locations, there may be regions where it is not available, limiting its availability for some users.
Invalidation process: The invalidation process can be slow and cumbersome, making it difficult to update content quickly when necessary if you are invalidating an application that has a large number of files at the origin.

Conclusion

AWS CloudFront is a powerful content delivery network (CDN) that works by caching content at edge locations around the world. CloudFront routes the request to the edge location closest to the user, delivering the content with low latency and high transfer speeds. CloudFront can also be used to encrypt content, protect against DDoS attacks, and integrate with other AWS services to further enhance and protect your applications.

Resources

What is DevOps?

Jeiman Jeya — Tue, 04 Apr 2023 02:47:25 +0000

DevOps is more than just a software development methodology – it's a culture. The DevOps culture emphasises 3 main pillars: collaboration, communication, and shared responsibility between development and operations teams. In this post, we will discuss the importance of DevOps culture in software development and how developers and operations teams can benefit from it.

DevOps is not a Title

This conundrum needs to be addressed. If you’ve seen career titles such as DevOps Manager, and DevOps Lead, it means those individuals are managing the DevOps culture in that team. The title does not necessarily imply the term literally.

The Importance of DevOps Culture - The 3 Pillars

Collaboration and Communication

One of the key principles of DevOps culture is collaboration and communication. By breaking down silos between teams, DevOps promotes cross-functional collaboration, which leads to better outcomes. Collaboration between development and operations teams can help identify potential problems earlier in the development cycle, saving time and resources down the line.

Effective communication is also essential in DevOps culture. By sharing information and feedback, teams can work together to ensure that everyone is on the same page and working towards the same goals. This helps identify areas for improvement and implement changes more effectively.

Shared Responsibility

Another principle of DevOps culture is shared responsibility. In a DevOps environment, developers and operations teams share the responsibility of delivering software. This means that developers are responsible not only for writing code but also for ensuring that it runs smoothly in production. Similarly, operations teams are responsible not only for deploying and maintaining infrastructure but also for providing feedback to developers on how their code is performing in a production environment. In other words, “you build it, you ship it!”; developers can utilise the DevOps tools available to deploy their code to production.

DevOps teams can create a culture of accountability and ownership by sharing responsibility. When everyone is responsible for the success of the software, it creates a sense of ownership and pride in the work being done.

Continuous Improvement

Continuous improvement is a fundamental value of DevOps culture. DevOps teams continuously seek ways to enhance their processes, tools, and practices. By continuously evaluating and improving their work, DevOps teams can remain competitive in a rapidly changing digital landscape.

Continuous improvement also assists teams in learning from their mistakes. When something goes wrong, DevOps teams utilize the incident as an opportunity to learn and improve. By implementing changes based on what they have learned, teams can avoid making the same mistakes in the future. DevOps teams should always be looking for ways on improving their automation and processes.

Tools

DevOps teams can employ a range of tools and practices to achieve their goals, such as continuous integration and delivery (CI/CD), automated testing, and infrastructure as code (IaC). By automating tasks and using standardized processes, DevOps teams can reduce errors and improve the quality of their software. Some of the tools include:

GitHub Actions, Azure DevOps - CI/CD tool
Terraform, Pulumi, AWS CDK, Azure Resource Manager - IaC tool
SonarCloud, Codacy - Code analysis and security tool
Azure Test Plan - Automated testing tool

Conclusion

DevOps culture is more than just a set of practices; it is a mindset that emphasizes collaboration, communication, shared responsibility, and continuous improvement. By embracing these values, DevOps teams can create a culture of innovation and agility that can help them stay competitive in today's fast-paced digital landscape with the toolsets available.

Deploy a web app to S3 with CloudFront Invalidation via GitHub Actions

Jeiman Jeya — Mon, 03 Apr 2023 05:21:17 +0000

In this guide, we will show you how to set up a GitHub Actions workflow to deploy your web application to S3 and invalidate your cache on CloudFront for your end users. The guide includes pre-requisites, creating an IAM user, creating a custom policy for the IAM user, fetching your CloudFront distribution ID, saving secrets in GitHub Secrets, and a YAML pipeline workflow. Please note that this post assumes that your source code is hosted on GitHub and is running on a Node.js framework.

Pre-requisites

An AWS account with admin privileges
An S3 bucket
A CloudFront distribution
A GitHub account
Basic understanding of GitHub Actions workflows

Steps

The steps involved are straightforward and shouldn't take too long to complete.

Create an IAM user in AWS
Attach a custom policy tailored to your pipeline needs
Create or retrieve your CloudFront distribution ID
Save the Access Key ID, Secret Access Key from IAM, and CloudFront distribution ID in GitHub Secrets
Create the pipeline using a GitHub workflow

Creating the GitHub Actions IAM user

Visit the IAM page on AWS.
Create a new user. Give it a meaningful and distinct name.
Don’t tick the box that says “Provide user access to the AWS Management Console - optional”. This user does not need access to the console to function.
For permissions, choose the option, “Attach policies directly” and click “Create Policy”. Copy the JSON policy mentioned below in the next section.
Review the new user and create.
Next, visit the users page on IAM and click on the newly created user.
Navigate to the Security credentials tab.
Scroll down to Access keys and click Create access key.
Take note of the Access key ID and Secret access key. You will need it later for saving it to GitHub Secrets.

AWS IAM user policy

To set up the pipeline, you will need an IAM user that would authenticate with your AWS account and perform the updates automatically. It is best to follow security principles and provide this user with the least privileged access to safeguard your AWS account from accidental or unwanted malicious activities.

As we basically need to allow the pipeline to be able to communicate with AWS S3 and CloudFront respectively, we need to supply the following permission scope and create a policy for the IAM user.

S3 permissions

List the buckets
Get the objects in the buckets
Create objects in the buckets
Delete objects in the buckets

CloudFront permissions

Create invalidation
Get invalidation
List the invalidation

Custom JSON policy for the IAM user

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::example.com"
            ]
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:DeleteObject"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::example.com/*"
            ]
        },
        {
            "Action": [
                "cloudfront:CreateInvalidation",
                "cloudfront:GetInvalidation",
                "cloudfront:ListInvalidations"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:cloudfront::<aws_account_id>:distribution/*"
            ]
        }
    ]
}

With the custom policy above, you can attach it to the IAM user you have created. Ensure the following is updated:

S3 bucket name - example.com
AWS account ID - for the CloudFront ARN

Fetching your CloudFront distribution ID

Visit your CloudFront page.
Click on Distributions.
Select your Distribution that points to your S3 bucket.
Take note of the Distribution ID in the first column. You will need this ID when you save it in your GitHub Secrets

Saving secrets in GitHub Secrets

You need to save the following secrets in your GitHub repository secrets:

IAM credentials that you created earlier - Access Key ID (AWS_ACCESS_KEY_ID) and Secret Access Key (AWS_SECRET_ACCESS_KEY)
CloudFront Distribution ID that you retrieved earlier (AWS_DISTRIBUTION_PRODUCTION)

GitHub Actions workflow

A workflow is a configurable, automated process that can run one or more jobs. Workflows are defined by a YAML file that is checked into your repository. They run when triggered by an event in your repository, or they can be triggered manually or on a defined schedule. Below is a template of a S3 Sync workflow.

YAML pipeline workflow

name: S3 Sync - Production

on:
  push:
    branches: 
      - 'master'

jobs:
  build:
    name: Build
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - uses: actions/setup-node@v2.5.1
      with:
        node-version: '15'

    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: ap-southeast-1

    - name: Install packages
      run: npm install

    - name: Run build
      run: npm run build

    - name: Generate
      run: npm run generate

    - name: Upload artifact
      uses: actions/upload-artifact@master
      with:
        name: web-app-dist
        path: './dist'

  deploy_to_production:
    name: Deploy to S3 Production
    runs-on: ubuntu-latest
    needs: build
    environment:
      name: production
      url: https://example.com
    steps:
    - name: Download landing page artifact
      uses: actions/download-artifact@v2
      with:
        name: web-app-dist
        path: dist

    - name: Display structure of downloaded files
      run: ls -R
      working-directory: dist

    # S3 sync
    - name: S3 Sync
      uses: jakejarvis/s3-sync-action@v0.5.1
      with:
        args: --acl public-read --follow-symlinks --delete
      env:
        AWS_S3_BUCKET: 'example.com'
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        AWS_REGION: 'ap-southeast-1'   # optional: defaults to us-east-1
        SOURCE_DIR: 'dist'      # optional: defaults to entire repository

    # Invalidate Cloudfront
    - name: Cloudfront Invalidation
      uses: chetan/invalidate-cloudfront-action@master
      env:
        DISTRIBUTION: ${{ secrets.AWS_DISTRIBUTION_PRODUCTION }}
        PATHS: '/*'
        AWS_REGION: 'ap-southeast-1'
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

Using this workflow, you can easily deploy your applications to S3 by syncing the files and invalidating your cache on your CloudFront distribution. This ensures that your end users receive the latest content from your release.

Workflow breakdown

The workflow is divided into two sections:

CI section, which builds your application, generates the static files, and compresses them to be uploaded as artifacts.
CD section, which downloads the compressed artifact, configures the AWS credentials, syncs the static files to S3, and invalidates your CloudFront distribution.

The reason for this approach is to enable the reuse of our artifacts, if necessary, for other GitHub workflows. It also provides a clear separation between continuous integration builds and continuous deployments.

Conclusion

Setting up a GitHub Actions workflow is fairly simple once you get the hang of it. This workflow allows you to deploy your S3 applications with confidence to various environments.

Let’s Dive into AWS Lambda

Jeiman Jeya — Sat, 01 Apr 2023 15:16:26 +0000

What is AWS Lambda?

AWS Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers. With over 200 AWS services and SaaS applications that can trigger Lambda, the possibilities are endless. Some application use-case include:

File processing
Streaming processing
IoT backends
Web applications
Mobile backends

How it works

Lambda automatically runs your function when it's needed and can scale from just a few requests per day to thousands per second. You'll only pay for the compute time you use, so you won't be charged when your code isn't running. Your function can be triggered using various types of events, which is listed below.

Components in AWS Lambda

Function Invocation

Lambda functions can be invoked manually using several methods, including the AWS console, a function URL (HTTPS) endpoint, the Lambda API, the AWS CLI, an AWS SDK, and the AWS toolkit.

Function Triggers

If you’re designing an event-driven application, triggers would be best suited for this use case. There various types of triggers readily available in Lambda, most of which is listed below:

AWS S3 triggers - changes in a bucket or object
Respond to incoming HTTP request - from AWS API Gateway or a third-party system
Consume events from a queue - AWS SQS
Schedule events using - CloudWatch Events
DocumentDB
DynamoDB
AWS ALB
AWS IoT
AWS Kinesis
AWS MQ
AWS MSK
AWS SNS
Apache Kafka
Amazon Alexa

Function Destination

Destinations are used to perform asynchronous and stream invocations in Lambda. This feature provides visibility into Lambda function invocations and routes the execution results (whether it’s a success or a failure) to other AWS services, simplifying event-driven applications and reducing code complexity.

At the time of this writing, there are 4 types of destinations:

SNS topic
SQS queue
Lambda function
EventBridge event bus

With these features, you will have greater visibility and control over the results of function execution. This is helpful in building better event-driven applications, reducing code, and utilizing Lambda's native failure handling controls. An example use-case would be if you have developed a resilient payment transaction system in Lambda and want to keep track of all successes and failures in the function; you can build another function to process those results into a DynamoDB table for auditing purposes.

Lambda console

AWS offers an intuitive, user-friendly web-based interface for managing your Lambda functions, monitoring tools, CloudWatch log streams, and CPU and memory usage. As previously mentioned, you can manually trigger your Lambda functions as needed from the Web console. Furthermore, you may test your functions without saving an event by configuring a JSON event in the Web console.

CloudWatch Logs

By default, for every Lambda function that you create, AWS will create a CloudWatch log group and bind it to your Lambda function. This allows AWS to send all function application logs, including any log methods defined in your code, to CloudWatch. This feature is useful and convenient for debugging and monitoring your application. In addition, you have the option to group your functions and manage all logs using a single CloudWatch log group.

Advantages of using AWS Lambda

AWS Lambda has several advantages that make it a popular choice for developers:

Cost-effectiveness: With AWS Lambda, you only pay for the time your code is running, which means that you won't be charged when your code is not being executed. This is particularly beneficial for applications that have infrequent usage patterns, as you only pay for the computing resources that you use.
Highly scalable: AWS Lambda automatically adjusts and scales to the amount of traffic that your application receives, which means that your application can handle large spikes in traffic without any manual intervention. This is especially important for applications that experience large fluctuations in traffic, such as e-commerce websites during peak shopping seasons. As a result, scaling is no longer an issue and you can easily manage traffic spikes and large workloads with AWS Lambda.
Easy to use: You don't have to worry about infrastructure management or server maintenance. This means that you can focus on writing code and building your application, rather than worrying about the underlying infrastructure. Additionally, AWS Lambda integrates with a wide range of other AWS services, such as Amazon S3 and Amazon DynamoDB, which makes it easy to build complex applications.
Security: AWS has implemented several security measures to ensure that your code and data are protected. For example, AWS Lambda functions are executed within a secure VPC (Virtual Private Cloud), which means that your functions are isolated and protected from external threats. Additionally, AWS Lambda integrates with AWS Identity and Access Management (IAM), which allows you to control access to your functions and resources.
Flexibility: This feature allows you to write functions in your preferred language, although currently only Node.js, Python, Java, Ruby, C#, Go, and PowerShell are supported. This provides developers with the freedom to choose the language they prefer, rather than being restricted to a specific language. In addition, Lambda functions can be used for a wide variety of use cases, such as file processing, streaming processing, IoT backends, web applications, and mobile backends. Lambda integrates with over 200 AWS services and SaaS applications that can trigger Lambda, which makes the possibilities endless.

Disadvantages of using AWS Lambda

Cold start delays: One of the major disadvantages of AWS Lambda is cold start delays. The first execution of a Lambda function can have a delay in response time due to initialization. This delay can be a problem for time-sensitive applications. To mitigate this problem, developers can use AWS Lambda Provisioned Concurrency, which pre-warms the Lambda function to reduce cold start delays.
Limited execution time: AWS Lambda functions are limited to 15 minutes of execution time, which may not be enough for complex applications. This limitation means that developers must plan their applications accordingly. To overcome this limitation, developers can use AWS Step Functions, which allow them to execute multiple Lambda functions in a sequence, enabling long-running applications.
Dependency management: Managing dependencies can be challenging as the functions are deployed as packages. This can be a problem, especially when your functions have many dependencies or require frequent updates. To overcome this problem, developers can use AWS Lambda Layers, which allow them to package and manage their dependencies separately from their Lambda functions.
Debugging: Debugging Lambda functions can be challenging as your code runs in a serverless environment. This means that you must use different tools and techniques to debug your code. To overcome this problem, developers can use AWS X-Ray, which allows them to trace requests through their Lambda functions and identify errors and performance bottlenecks.
Vendor lock-in: AWS Lambda is a proprietary technology, which may lead to vendor lock-in. This means that once you start using AWS Lambda, it can be challenging to switch to another vendor or technology. To overcome this problem, developers can use AWS SAM (Serverless Application Model), which is an open-source framework for building serverless applications. This framework allows developers to define their serverless applications in a standard way, making it easy to migrate to other serverless platforms.

Conclusion

AWS Lambda is a powerful technology that has several advantages for developers. It is cost-effective, highly scalable, and easy to use, which makes it an ideal choice for building modern applications. With the flexibility to use currently supported programming language, AWS Lambda is an excellent choice for developers who want to build applications quickly and efficiently. If you're looking to build a new application or migrate an existing one to the cloud, AWS Lambda is definitely worth considering.

Resources

Terraforming in the Cloud

Jeiman Jeya — Sun, 26 Mar 2023 08:24:10 +0000

This post discusses the benefits of using Terraform for cloud automation. It explains the limitations of manual infrastructure management, the inception of IaC and how Terraform can help mitigate those issues. The document covers topics such as the Terraform configuration language, backends, workspaces, and Terraform Cloud, while using AWS as a cloud provider example.

The Iron Age

In the Iron Age, managing IT infrastructure was done manually. People would physically set up servers and configure them based on required settings by the OS and applications before deploying the application. This manual process often resulted in several problems, such as cost, scalability, availability, monitoring, and performance visibility.

In order to maintain all of this infrastructure, you would need to hire IT professionals and engineers. However, this can create a variety of issues such as inconsistencies, silos among teams when raising tickets to IT teams to provision new resources, miscommunications, and delays in reviewing and processing infrastructure requests.

Furthermore, when engineering teams need new infrastructure quickly, they typically use cloud consoles, such as AWS, Azure, Google, or Proxmox, to provision those resources. This approach is not an issue, and some engineers actually prefer spinning up resources this way to gain a better understanding of the prerequisites of a Cloud service or resource. While documentation can be helpful in this case, many engineers prefer hands-on experience and getting down to the nitty-gritty.

However, there is an issue to consider. What if you need to make changes that are not available through the user interface console? What if you need to make changes that depend on other changes being made first? This could have a widespread impact on the resources you were trying to set up. It can be difficult to keep track of all the resources and you might overlook updating or naming some of them with best practices in mind.

With modern technology, these problems can indeed be mitigated through automated processes, leading to improved efficiency and reduced costs. That's where Infrastructure as Code comes in.

Lo and behold, Terraform

Terraform is a powerful tool that allows you to manage your infrastructure as code. With Terraform, you can define, provision, and manage resources such as virtual machines, cloud instances, and containers across multiple cloud providers using a declarative configuration language known as HashiCorp Configuration Language, or HCL for short.

Imagine an orchestra of musicians. In this case, Terraform acts as the conductor who ensures that the right number of instruments (providers, modules) are playing and the sound (configuration) is correct. If there is an issue, the conductor replaces the broken instrument with a working one.

Wait, what about AWS CloudFormation, Azure RM and the rest?

CloudFormation is one option for provisioning your infrastructure resources, but if you prefer to deploy to a different cloud provider of your choice, it can be a challenge. Terraform, on the other hand, excels at managing infrastructure for multiple cloud platforms. In addition, compared to CloudFormation and other cloud provider configuration languages, Terraform's configuration files are simpler to understand due to their flat format. This makes it easier to manage dependencies and references, which is where Terraform truly shines with its confident approach.

resource "aws_instance" "helloworld" {
  ami           = "ami-12d3df"
  instance_type = "t3.micro"

  tags = {
    Name = "EC2-Instance-HelloWorld"
  }
}

The Terraform configuration language, designed for human readability, allows you to write infrastructure code with ease. As shown in the snippet above, provisioning an EC2 instance on AWS is a straightforward task.

How does Terraform maintain the newly provisioned resources?

Each Terraform configuration has a backend associated with it that determines how operations are carried out and where data, such as the Terraform state, is stored. The state is a necessary requirement for Terraform to function since it is used to compare existing resources with newly added ones specified in the configuration files.

Backends are typically used to store Terraform state snapshots. A Terraform configuration can specify a backend, integrate with Terraform Cloud, or store state locally by default.

By default, Terraform stores its state in a local file directory named terraform.tfstate. However, if you need to, you can choose to use alternative backend repositories, such as the following:

Amazon S3
Kubernetes
Remote
Postgres
AzureRM (via Storage accounts)
Consul
and many others

The above can be configured by using a backend block in the configuration file (typically the main.tf file). This is how an S3 backend would look like in your configuration file:

terraform {
  backend "s3" {
    bucket = "sample"
    key    = "path/to/my/samplekey"
    region = "ap-southeast-1"
  }
}

To store the backend states on S3, simply supply your AWS IAM credentials from you local machine or your CI/CD tools.

In my opinion, it is best to store state files remotely in a cloud storage because this option allows for state locking. This means that two people working on the same project cannot cause conflicting issues when making changes to the resources.

There is a lot that you can achieve with Terraform states and their respective backends, but you should now have a general understanding of how states work in Terraform and why they are important to have.

Ok, so how do I use the same resource configuration to provision different environments - Staging and Production?

In Terraform, workspaces are separate instances of state data that can be used from the same working directory. Workspaces can be used to manage multiple non-overlapping groups of resources with the same configuration (for example, staging, production, uat, etc.).

Almost all directories that have a main configuration file have a default workspace, named default. If you want to configure new workspaces, simply run the following command: terraform workspace new <workspace_name>.

Basically, it creates a new child working directory in your project directory that will contain your state files as such:

terraform.tfstate.d/
    - staging/
        - terraform.tfstate
    - prod
        - terraform.tfstate
    - uat
        - terraform.tfstate
    - ...

With that, you can maintain different state files for all of your environments. All you have to do is switch workspaces accordingly when working for that specific environment , using terraform workspace select <workspace_name>.

Utilize workspaces when your organization operates in multiple environments where your services are deployed.

JARVIS-level Automation

In typical scenarios, engineering teams commonly integrate Terraform with their preferred CI/CD toolset (e.g. Github Actions, Azure DevOps, Gitlabs, etc.) to ensure flexibility and self-governance over their state files and configuration, and to maintain coherence within their tech ecosystem. However, if teams are not familiar with CI/CD tools, they can rely on Terraform Cloud.

Terraform Cloud is a managed service offering by HashiCorp that eliminates the need for excessive tooling. You can provision your infrastructure in a remote environment hosted by HashiCorp that is optimized for Terraform workflows. This means that you can simply link your Cloud provider of choice with Terraform Cloud and let the platform handle all of the heavy lifting for you. All you need to do from your end is build and commit your Terraform (HCL) files to your repository and link your Git repository to your Terraform Cloud organization. Ideally, this follows the GitOps process. Write your infrastructure code, commit the changes, and Terraform Cloud executes your changes in a remote environment.

Here are some of the pros and cons of using CI/CD tools and Terraform Cloud. They are broken down into 4 criteria:

Control: With CI/CD tools, you have complete control over your infrastructure automation workflows, and you can customize your pipeline as required. With Terraform Cloud, you have less control over your infrastructure automation workflows, but you don't have to manage your own infrastructure.
Integration: CI/CD tools provide a wide range of integrations with other tools and services. Terraform Cloud integrates with a limited set of cloud providers, but it is optimized for Terraform workflows.
Maintenance: With CI/CD tools, you need to manage and maintain your own infrastructure for running the toolset if they are mostly on-premise/self-managed. With Terraform Cloud, you don't have to manage and maintain your own infrastructure.
Cost: CI/CD tools can be expensive, especially if you require a lot of customization and integrations. Terraform Cloud has a free tier, which provides basic functionality, and paid tiers that provide additional features.

So there you have it. Terraform is undeniably a powerful tool for managing infrastructure as code. Its declarative language, ability to manage resources across multiple cloud providers, and powerful set of tools make it an unrivaled choice for managing infrastructure and third-party systems. Whether you are managing a small set of resources or a large, complex infrastructure, Terraform can help you to manage it in a unfailing and consistent way. That’s why go-to tool for managing infrastructure and a great solution in DevOps.

By adopting Terraform, you can guarantee that your infrastructure is constantly in a known state, and that any modifications are made in a controlled manner. Regardless of whether you're overseeing a small or large infrastructure, Terraform is a useful tool for increasing efficiency and effectiveness.

Image sources

Automating mobile application deployments using Fastlane and CI/CD tools

Jeiman Jeya — Tue, 31 May 2022 02:57:15 +0000

The Problem

Engineering teams these days find it troublesome to build, test and deploy their mobile application changes locally without having to maintain the tools required for it. There is a lot of maintenance involved as you need to keep track of what versions of these tools have been installed to avoid compatibility issues when you’re building the application bundles, especially hybrid apps built on React Native.

The Cure

fastlane, an automation tool that aids in handling all of the tedious tasks so you don't have to. It's by far, the easiest way to build and release your mobile apps.

fastlane can easily:

Distribute beta builds to your testers
Publish a new release to the app store in seconds
Reliably code sign your application - alleviates all of your headaches
Reliably maintain your provisioning profiles and application certificates in a Git repository

How it works

All actions in fastlane are written into lanes. Defining lanes are easy. Think of them as functions in any programming language of your choosing. You define all of your actions within that lane.

An example of a lane is as follows:

lane :beta do
  increment_build_number
  build_app
  upload_to_testflight
end

So when it comes to executing that lane, all you do is run fastlane beta in your terminal.

Installation & Setup

In this article, we will look at setting up a fastlane script to build, sign and deploy an iOS application to Testflight.

Pre-requisites

As with most projects, you need to perform the initial project setup to support building your application, such as:

Installing your project dependencies
Install XCode and Android Studio
Install Java SDK
Setup git repository for Android and iOS certificates - will be explained later in the article

fastlane folder structure

In ideal cases, you would have an Android application project and an iOS application project, both hosted in the same code repository as your project

projectFolder/
    app/
    scripts/
    ios/
        fastlane/
            ....
  android/
        fastlane/
            ....
    package.json
    .gitignore
  ...

In this case, you will want to initialise the fastlane folder within the respective android and ios project sub-directories.

Setup

Install fastlane
- via Homebrew (brew install fastlane)
- via RubyGems (sudo gem install fastlane)
Navigate your terminal to your respective android and ios project directory and run fastlane init
You should have a folder structure that is similar to the one below:

fastlane/
    Fastfile
    Appfile

The most interesting file is fastlane/Fastfile, which contains all the information that is needed to distribute your app.

Inside your Fastfile, you can start writing lanes:

lane :my_lane do
  # Whatever actions you like go in here.
end

You can start adding in actions into your lanes. fastlane actions can be found here.
Copy the following file structure to your Fastfile

default_platform(:ios)

def beta(arg1, arg2)
  # You may use Ruby functions to write custom actions for your app deployment
end

platform :ios do

    desc "Building the IPA file only"
      lane :build_ios_app_ipa do
        app_identifier = "com.appbundle.id"
        beta("AppSchemeName", app_identifier)
      end

end

default_platform(:ios) - Initialise your Fastfile file with a default platform
platform :ios do - Add all actions under a platform

From 5 to 6, it will inform fastlane that this particular Fastfile is purely for iOS operations. So instead of running the command fastlane <lane_name>, you will actually run fastlane ios <lane_name>. Therefore, anything parked under platform :ios do will be executed when lanes are invoked in your terminal.

If you are well-versed in Ruby, you may write your own Ruby functions to help you write custom actions that you require for further flexibility, especially when it comes to building several apps with different environments across your organisation.

fastlane will identify them as an action regardless due to the fact that fastlane is written in Ruby.

In this article, we will follow writing the actions using a Ruby function. This is so we can promote action re-usability across other lanes deployments.

Action Steps

Before we start writing our functionality in the lanes, we first list down our action steps:

Setup API Key for App Store Connect (app_store_connect_api_key) - This will allow fastlane to connect to your App Store to perform other actions that requires user authentication
Setup CI (setup_ci) - This will setup a temporary keychain to work on your CI pipeline of your choice
Create and sync provisioning profiles and certificates (match) - This will help us maintain our provisioning profiles and certificates across your teams
Update code signing settings (update_code_signing_settings) - This is to update the code signing identities to match your profile name and app bundle identifier
Increment your app build number (increment_build_number) - This will automate your application build number by retrieving the latest Testflight build number and incrementing the value
Build the app (build_app) - This will build the app for us and generate a binary (IPA) file
Upload your binary to Testflight (upload_to_testflight) - This will automate the process of uploading the binary file to Testflight and informing your testers accordingly

Steps

Step 1: Setup your App Store Connect API key

Visit the following page. It will provide you a step-by-step process in generating an API key
Once you have generated a key, take note of:
- Issuer ID
- Key ID
Download the generated API key - A .p8 file
Store it in a secure location in which you can easily access them. Avoid storing them in the same repository as anyone in your organisation will have access to the company's Apple account.
- In this article, we are storing them in another Git repository with limited read and write scopes to specific engineers within our organisation.

Step 2: Setup the fastlane script

Setup your App Store API Key to generate a hash used for JWT authorization

def setup_api_key()
  sh "if [ -d \"appstoreapi\" ]; then echo 'Folder exist, executing next step'; else git clone #{ENV['APPSTORE_API_GIT_URL']} appstoreapi; fi"
  app_store_connect_api_key(
    key_id: ENV['APPSTORE_KEY_ID'],
    issuer_id: ENV['APPSTORE_ISSUER_ID'],
    key_filepath: Dir.pwd + "/appstoreapi/AuthKey_xxx.p8",
  )
end

In the snippet above, I am cloning a Git repository which contains my App Store Connect API key, followed by utilising the app_store_connect_api_key action from fastlane. It takes in several parameters, however, there are 3 vital parameters:

key_id - The Key ID from which you took note when you generated the API key
issuer_id - The Issuer ID from which you took note when you generated the API key
key_filepath - The file path to your .p8 file

This step will generate a hash that will be used to authenticate the App Store using JWT.

I would highly recommend storing sensitive credentials or URLs and access them from an Environment Variable (.env) file in the same project folder - fastlane/.env

Once you have prepared the file, you can easily access any Environment Variable by simply passing in ENV['ENV_NAME'] into the Fastfile.

2. Define a Ruby function with 2 arguments

def beta(scheme, bundle_id)
end

Within this function, we specify the action steps we mentioned above

def beta(scheme, bundle_id)
    setup_api_key() # Import the setup_api_key function
    setup_ci() # Uses fastlane action to create a temporary keychain access on a CI pipeline
end

3. Utilise the match action from fastlane

match is a fastlane action that allows you to easily sync your certificates and provisioning profiles. It takes a new approach to iOS and macOS code signing, where you share one code signing identity across your engineering team to simplify the setup and prevent code signing issues. The foundation of match was built using the implementation of codesigning.guide concept.

You can store your code signing identities in:

Git repository
Google Cloud
Amazon S3 bucket

In this article, we have chosen to store it a Git repository. However, with the case of storing in a Git repository, you would need to provide a form of basic authorization in order for match to access and clone your repository.

Whichever Git provider you choose (GitHub, Bitbucket, Gitlab or Azure DevOps), you would need to setup a Personal Access Token (PAT), which fastlane will use to clone repository and sync your code signing identities.

In your Fastfile, you will need to encode your PAT with a Base64 encryption

authorization_token_str = ENV['GITHUB_TOKEN']
basic_authorization_token = Base64.strict_encode64(authorization_token_str)

The basic_authorization_token variable will be used in setting up the match implementation below.

match(
  git_url: "<git_url>",
  git_basic_authorization: basic_authorization_token,
  readonly: true,
  type: "appstore",
  app_identifier: [
    bundle_id
])

From the snippet above, this is a very simple implementation. Take note on the readonly parameter. This is crucial in creating and syncing your profiles and certs with your Apple Developer account. If you set readonly to false, you will allow match to automatically sync and create new profiles and certs, should it deem your existing certs and profiles expired or corrupted. Once it has provisioned the profiles, you can set readonly to true. This is to avoid egde cases where it might accidentally create new certs and profiles.

4. Update code signing identities

This step is mainly used to update the Xcode settings on a CI pipeline due to its default behaviour of selecting a default cert and profile from the Mac OS agent.

update_code_signing_settings(
  use_automatic_signing: false,
  path: "../ios/AppName.xcodeproj",
  code_sign_identity: "iPhone Distribution",
  profile_name: "match AppStore com.appbundle.id",
  bundle_identifier: "com.appbundle.id"
)

From the above snippet, you can retrieve your profile_name and bundle_identifier from your Apple Developer account or taking note of them from the match step once it has been executed.

5. Increment Application build number

We simply increment the number by accessing your latest testflight build number and incrementing the value with 1.

increment_build_number({
  build_number: latest_testflight_build_number + 1
})

puts "BUILD_NUMBER: #{lane_context[SharedValues::BUILD_NUMBER]}"

The puts command will print out the Build number in the console, which is shared across the lane context.

You may have other app build versioning models than the one mentioned in this article. Increment the build number however you see fit to your project needs.

6. Build the application

This step will involve building your application to generate a binary (APK) file.

build_app(
  workspace: "../ios/AppName.xcworkspace",
  export_xcargs: "-allowProvisioningUpdates",
  scheme: scheme,
  clean: true,
  silent: true,
  sdk: "iphoneos"
)
puts "IPA: #{lane_context[SharedValues::IPA_OUTPUT_PATH]}"

The build_app action is provided by fastlane.

The puts command will print out the file path location of the IPA file in which you can use to manually upload to Testflight.

7. Upload the binary file to Testflight

This step will involve uploading your binary file to your Testflight account.

app_identifier = "com.appbunde.id"
upload_to_testflight(app_identifier: app_identifier)

upload_to_testflight is an action provided by fastlane.

Putting them all together

The final lane should look something like this:

desc "Build and push a new build to TestFlight"
  lane :release_build do
    app_identifier = "com.appbundle.id"
    beta("AppSchemeName", app_identifier) # The custom function we wrote earlier
    upload_to_testflight(app_identifier: app_identifier)
  end

When you execute fastlane ios release_build in your terminal, it will operate based on the aforementioned steps above.

As you can see, we utilised a Ruby function to group all common operations under the same roof so is to ensure:

Code re-usability
Consistency
Clean code

Integration with CI/CD tools

With the fastlane scripts, you can easily integrate it with any CI/CD tools of your choosing:

GitHub Actions
Gitlab CI
Azure Devops
Bitbucket Pipelines

When setting up your pre-requisite steps in your pipeline YAML file, you can simply include a bash script that executes your fastlane script

cd ios/android folder

fastlane ios/android release_build

The command above will execute your fastlane script by following the aforementioned steps above.

Conclusion

fastlane is a powerful tool that helps streamline your build process across your organisation. It can be used along side your existing CI/CD pipelines or as a standalone pipeline script to be used on your local machines or custom pipeline tools such as Buildkite. It would require additional steps such as, cloning the repository and checking out branches, however, all readily available from fastlane as actions.

Spend some time reading through the documentation as it contains a lot of actions out of the box that will prove useful for your engineering teams (match, pilot, cert, sigh, appium, xctool, supply, and many more) . Furthermore, they provide an extensive list of plugins for third-party integrations such as Firebase, App Center, Yarn, Android Versioning, Dropbox, Slack Upload, S3, Bugsnag and many more. You can supercharge your fastlane scripts by coupling it with a CI/CD tool, such as GitHub Actions, Bitbucket Pipelines or Azure DevOps. The sky's the limit!

Part 2: Automating code quality scanning using Sonar Cloud and GitHub Actions

Jeiman Jeya — Tue, 31 May 2022 01:46:11 +0000

Part 1 of our article talks about the fundamentals of code quality with respect to Sonar Cloud. However, these are generic terms that can be applied to any code quality tool. In this article, we will explain how we can use Sonar Cloud to automate our code quality scanning with our GitHub repositories using GitHub Actions. The goal of this article is to ensure that you have a good understanding of how Sonar Cloud performs code analysis, how the integration features on GitHub (PR decorators, inline commenting, code quality overview using widgets) will benefit your engineering teams in the long run.

Some of the neat features that I found Sonar Cloud provides is:

Enhances your workflow with continuous code quality and code security
Supports all major programming languages you can think of
Provides a clear overview of your overall code health in your repository, pull requests and pipelines
Works with all famous Git providers (GitHub, Bitbucket, Azure DevOps and GitLab)
Works with all famous CI/CD tools (GitHub Actions, Bitbucket Pipelines, Azure Pipelines, GitLab CI/CD, CircleCI, TravisCI, etc)
Bug, Vulnerability, and Code Smell detection
Top-notch coding rules
In-ALM pull request feedback
Go/No Go Quality Gate checks
Automatic analysis (Currently limited to GitHub repositories)

Preliminaries

You will need the following:

A GitHub repository with a backend or frontend application
Access to GitHub Actions
A Sonar Cloud account
A Sonar Cloud project
Some basic knowledge on CI/CD

Setup Process

1. Sign up and import repositories

Ensure you have a GitHub repository readily available. For this, we will be choosing one of my NodeJS backend applications.
Signup for a Sonar Cloud account and log in using your GitHub account to have a seamless experience.
Once you have signed up, you will be redirected to this page:
From this page, you can import an organisation from GitHub.
Click on Import an organisation from Github.
You will be redirected to your GitHub account where it will prompt you to choose an organisation to install SonarCloud. In this article, we will be choosing my own personal GitHub repository.
From there, you will be redirected back to Sonar Cloud, where you will create your Sonar Cloud organisation. Please note that at this point, you have already installed and connected the Sonar Cloud App on your GitHub account.
From the image above, you can provide an organisation key that will be used later in GitHub Actions to send your code analysis metrics.
Next, you would need to choose a Pricing plan. There are 2 options: Paid and Free. For this article, we will be choosing the Free plan.
Click Create Organization.
You will now be able to import any Public repository. If you have Paid for Sonar Cloud, you may import your Private repositories.
We have selected our public repository, named nodejs-backend-starter. This repository contains a very basic Node.js backend application with Unit Testing and Code Coverage setup.
- NOTE: There's also an option to set up a monorepo. Based on the image above, you may see a small section at the bottom right of the image that mentions setting up a monorepo.
Once you have selected your repositories, click Set Up.
It will do an Automatic Analysis of your project providing an initial analysis of it.
Although Sonar Cloud offers automatic analysis for GitHub repositories only, we will be disabling this option as we want our CI pipeline to handle the analysis process for us. With the CI pipeline, you may replicate this entire process for any Git provider and any CI/CD tool available.
Navigate to Administration > Analysis Method > SonarCloud Automatic Analysis. Turn off this option.
Navigate back to your project root page. Your project page should look something like the image below:
As you can see, it states that they can't display any Quality Gate without a New Code definition. Based on the Part 1 article, we will be setting up the following:
1. A New Code Definition
2. A Quality Gate

2 Sonar Cloud Configuration

2.1 Set Up New Code definition

There are 2 options available at this moment. You can either set up a New Code definition on:

A project level
An organisation level

For this article, we will set it up on a project level.

On your project page, click on Set New Code definition. If that is not available, you may navigate to Administration > New Code.
If you have read my previous article, you may know that there are a number of options to choose from. In this case, we will be choosing the Number of days that equates to 30 days by default. The reason is we are not going to be maintaining a version for the project on Sonar Cloud for now.

2.2 Setup a Quality Gate

Navigate to your organisation page (https://sonarcloud.io/organizations/{username}/projects).
Select Quality Gates from the sub-navigation.
From the image above, there's already a default Quality Gate set up for you. We're going to be cloning this and creating our own. Click on Copy on the Sonar way Quality Gate*.*
Name your new cloned Quality Gate.
Since there are already New Code conditions, we are simply appending Overall Code conditions. For this article, we have added the following:
1. Coverage is less than 80%
2. Duplicated Lines is greater than 5%
3. Maintainability Rating is worse than A
4. Reliability Rating is worse than A
5. Security Rating is worse than A
Set your new Quality Gate as Default.
Now that we have our project settings in place, we need to set up a project properties file next.

2.3 Setup a project properties file

This properties file will enable the CI pipeline to perform the necessary configuration and filtering in order to perform the code analysis. Furthermore, this will ensure we have the right settings in place when we push our commit to a branch or create a pull request that triggers the code analysis.

In your code repository, create a sonar-project.properties file.

sonar.exclusions=**/*.bin
sonar.organization=jeimanjeya
sonar.projectKey=jeiman_nodejs-backend-starter
sonar.projectName=jeiman_nodejs-backend-starter
sonar.projectVersion=1.0
sonar.sourceEncoding=UTF-8
sonar.sources=src
sonar.exclusions=node_modules/**
sonar.test.inclusions=test/**
sonar.typescript.lcov.reportPaths=coverage/lcov.info

In this file, we're basically narrowing the focus of the scanner to configure the project source path, unit test inclusion path, coverage report path and exclude redundant folder paths (node_modules).

Commit this file to your:
Root directory: If it is a single application repository (microrepo) architecture (the case for us)
Monorepo directory: If you have a monorepo architecture

Now that we have set up our project configuration and settings, we can focus on configuring the CI pipeline on GitHub Actions.

3. GitHub Actions

3.1 Pipeline/Workflow Structure

You have several to choose from options:

Store all of your workflows in a single YAML file - ci.yml
Separate them into branch workflows:
1. ci_develop.yml - for PR triggers and branch triggers on develop branch
2. ci_master.yml - for PR triggers and branch triggers on main branch
Separate them into trigger workflows:
1. PR triggers - ci_pr.yml
2. Branch triggers - ci_branch.yml
Separate them into monorepo workflows:
1. spa-frontend.yml and ms-backend.yml - contains PR and branch triggers for both develop and main branch

Option 1 is difficult the maintain in the long run if you follow a monorepo architecture
Option 2 or 3 is the best way of maintaining your workflows on GitHub
Option 4 is useful for when you have a monorepo architecture

For this article, we are choosing Option 3 for simplification.

Your final GitHub workflow for branch analysis will look like this:

name: Sonar Cloud - Branch Analysis

# Controls when the action will run. Triggers the workflow on push
# events but only for the main and release-* branch
on:
  push:
    branches: 
      - main
      - release-*

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  sonarcloud:
    name: Build (Sonar Cloud)
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
      with:
        # Disabling shallow clone is recommended for improving relevancy of reporting
        fetch-depth: 0
    - uses: actions/setup-node@v2
      with:
        node-version: '15'

    - name: Node install dependencies
      run: npm install

    - name: Run unit tests
      run: npm run test

    - name: SonarCloud Scan
      uses: sonarsource/sonarcloud-github-action@master
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}

We have included an Action from GitHub Actions and that is the Sonar Cloud Scanner.
In order for Sonar Cloud Scanner to authenticate and upload the analysis reports and metrics, you will need to store the SONAR_TOKEN secret in your GitHub repository.
Navigate to your project settings and retrieve the project token, Project > Administration > Analysis Method > Analyze with a GitHub Action. It will present you with a token.
Navigate back to your repository settings on GitHub, Repository Settings > Secrets > New repository secret. Name it as per the workflow secret name.
For the GITHUB_TOKEN, GitHub automatically appends the token for each workflow that runs on GitHub.
Commit your code into the main branch and watch the pipeline run.
The summary for the branch analysis pipeline has indicated that the analysis was successful.

Navigating back to Sonar Cloud, you will notice that your project branch analysis report has been updated.

We'll move on to running code analysis on a long-living branch next.

3.2 Running code analysis on a long-living branch

Simply check out a new branch from main and name it to release-{anyname}
Commit your code and the CI pipeline will pick it up based on the branch trigger logic in the workflow YAML file.
The SC Scanner will upload your analysis reports to your project on Sonar Cloud.
Don't be alarmed if it shows Not computed as Sonar Cloud requires a second analysis to be able to show your Quality Gate on that long-living branch. It mentions Next scan will generate a Quality Gate.
We pushed another commit to the repository to the long-living branch to show the Quality Gate.

Your final Quality Gate for branch analysis (on long-living branch) will produce the following result.

3.3 Running code analysis on pull requests (PR)

Clone the ci_branch.yml. Name it ci_pr.yml
The only section we are changing is the event trigger. Instead of a push event, we are using the pull_request event.

on:
  pull_request:
    branches:
      - main
      - release-*

Commit the code and raise a PR in your code repository.

Your PR page should produce pipeline status checks alongside SonarCloud Code Analysis on your Merge PR section. Furthermore, your PR will contain decorations provided by the SonarCloud Bot.

Your PR status checks details page should give you a better overview of what needs fixing.

If you navigate back to your Sonar Cloud project and choose your PR analysis report from the branch dropdown menu, you will see similar statistics that were displayed on GitHub.

Let's go ahead and fix our code to ensure we don't have any code smells and fine tune our Quality Gate conditions to match with a Passed state.

Now our PR has clean code (however overall code requires fixes), we can now merge our PR. The pipeline will run another branch analysis on the main branch, which is our default branch on Sonar Cloud.

Piecing it all together

We have accomplished having a Sonar Cloud project connected to a GitHub repository that has GitHub Actions workflows in place to automate our code analysis for branch and PR triggers. Once these analyses are completed on the pipeline, you can navigate back to your Sonar Cloud project and view branch and PR analysis on both your main branch and long-living branches.

One of the plus points in having integration between GitHub and Sonar Cloud is that engineering teams can benefit from having decorations forming up in their pull requests and repository. This will speed up the developer's productivity in fixing bugs, code smells, and vulnerabilities and avoid having further technical debts in their projects.

Tips

You can use any CI/CD tools to perform your code analysis. It does not need to be with GitHub Actions even though your repository resides on GitHub, however, for consistency you can keep them all under one umbrella.
To have a complete end-to-end experience for your code quality scans on your pull requests, ensure you connect and import repositories from one unique Git organisation. The aforementioned features mainly work for that reason. Try not to mix and match repositories from different organisations (even though it's possible to do so).
- 1 GitHub organisation = 1 Sonar Cloud subscription
- 1 Azure DevOps organisation = 1 Sonar Cloud subscription
Create long-living branches on Sonar Cloud if you're following Git flow technique of develop and master branch. You may read up on their community post on how to achieve this. This will ensure that you are performing branch analysis on them alongside your pull request analysis.
Fine-tune your Quality Gate to ensure you get the best experience from it that matches your engineering teams needs
Sonar Cloud supports monorepo projects!

I hope this 2-part series has been beneficial and helpful to your engineering needs. Sonar Cloud is a powerful code analysis tool. Please do explore their documentation to find out more or kindly reach out to me for any assistance.

Demo links to this article

References

Part 1: Concepts of Code Quality in Sonar Cloud

Jeiman Jeya — Tue, 31 May 2022 01:37:13 +0000

In engineering teams/tribes, we often find ourselves stuck in a dilemma of choosing a suitable and efficient code analysis tool. A tool that will provide us all of the essential features and metrics to analyse codebases using best design practices in mind. Teams would want to prevent code problems from being merged by detecting code smells, bugs, and vulnerabilities sooner. Furthermore, they would want to view fast and accurate feedback from their respective pull requests and code merges in their repositories.

The primary goals of software quality engineering are:

Process control and oversight
Implementing standards and metrics
Data collection and analysis
Test development
Identification of issues and solutions
Follow-up to ensure corrective actions

Overview

Over the years, I have worked with many code analysis tools to help developers in various teams such as Codacy, Code Climate, DeepScan, and Sonar Cloud. After spending a considerable amount of time experimenting and setting up Sonar Cloud projects, I realised it stands out from the crowd. It has a comprehensive analysis engine that offers features encompassing all of the aforementioned software quality engineering goals. Some of those features which fascinated me and my team was:

Pull Request decorators
- Inline commenting through report annotations
- Pull request widgets: Provides overall code quality health on your Pull requests
Repository widgets: Provides overall code quality health on your project
Scanning Old code vs New code
- Code Coverage
- Code Duplications
- Maintainability Ratings
Defining custom Quality Gate checks for different projects
Defining New Code definitions for different projects

Some of the aforementioned features (Quality Gate, Profile, New Code Definitions) are in fact settings that need to be configured in Sonar Cloud. You will need to give your attention to configuring these settings as you will not reap the benefits of fully utilising Sonar Cloud.

We are going to explore the Sonar Cloud ecosystem and how all of its core features come together to provide you a comprehensive code analysis experience.

Sonar Cloud Ecosystem

Projects

In Sonar Cloud, a single repository corresponds to a single project. It is how they maintain a unique set of code quality data and metrics for each repository you have.

Monorepo Support

Sonar Cloud does support monorepo projects. You can create multiple projects, each corresponding to a separate monorepo project which are all bounded to the same repository. This will allow you to:

Configure one Quality Gate per project
Receive multiple Quality Gate results
Read project-labeled messages from SonarCloud

Each monorepo project must have a unique project key in Sonar Cloud, which will be used to uniquely identify your projects using your CI tool.

The standard practice is to have the following naming convention: {organisationName-project-monorepoName}

Project 1: sampleorg-domain-frontend
Project 2: sampleorg-domain-backend

This is useful if your organisation maintains a big number of monorepo projects across various engineering tribes so the projects can be easily identified in Sonar Cloud. However, you can follow any naming convention you see fit that suits your needs.

New Code Definitions

Sonar Cloud follows the concept of Clean As You Code. The core idea is that you focus your attention and effort on new code. As you work on features and improvements, SonarCloud analyses your code on each new commit and alerts you to any code quality problems and vulnerabilities. This allows you to address the issues right away and ensure that all new code added to the project is always clean. You may read their documentation to find out more information.

Accompanying this are New Code Definitions. Setting up the right New Code definition for your project is vital to getting the most out of SonarCloud by determining which changes to your code are considered recent enough to merit your full focus and allow you to use the Clean as You Code methodology when addressing issues in your code. There are several options to consider when configuring your New Code Definitions:

Previous version: Issues in the code that have appeared since the most recent version increment of the project
Specific version: Issues that have occurred on a specific version of the project
Number of days: Issues that have appeared on your code since the specified number of days (A numerical number)
Specific date: Issues that have appeared on your code since the specified date

When you perform initial code analysis, it does not provide you a "New Code" analysis. Instead, it scans the whole project, providing you with an overall code quality health. As you go along and start analysing new commits, Sonar Cloud will present you with both an Overall and New code quality health analysis (depicted from the picture above).

You can set a New Code Definition either on a project level or an organisation level, with the latter providing you a way of automatically applying the definitions to new projects that are created.

Quality Gates

A quality gate is technically a metric of sorts that informs you whether your code meets a minimum level of quality required for the project. This consists of a set of conditions that are applied to the results of each analysis performed. If the results meet or exceed the quality gate conditions, it will show one of the following statuses accordingly: Passed or Failed. You can define conditions on New Code and Overall Code. Some examples of the conditions are:

Coverage is less than 80.0%
Duplicated Lines is greater than 3.0%
Maintainability Rating is worse than A
Reliability Rating is worse than A
Security Hotspots Reviewed is less than 100%

The quality gates are analysed and calculated on Main Branch (master by default), other Branches, and Pull Requests. You can create any number of Quality Gates for your projects and enable them per project, or create a default Quality Gate that will be applied to all projects that have been or will be created.

An example of a customised Quality Gate is shown below:

Quality Profiles

Quality profiles as you may have guessed it are programming language rules that are applied during code analysis. By default, the programming languages that are supported on Sonar Cloud have a built-in profile, called "Sonar way", using the standard best practices that are currently in the market. Although the "Sonar way" is best suited for most projects, there are cases where engineering tribes would want to customise their profiles that best suit their needs.

Branches

In Sonar Cloud, there are 2 types of branch analysis: Short-lived branches and Long-lived branches.

Short-Living Branches

As the name suggests, these branches are meant to be used to temporarily perform analysis on them, usually via pull requests. Short-lived branches are deleted automatically after 30 days with no analysis.

Long-Living Branches

These branches are useful for when tribes follow Agile methodology, utilising Git flow techniques to maintain a set of upstream branches (sprint, release). These branches will remain in your Sonar Cloud project history until it is deleted. Some companies maintain upstream branches for a very long time, hence this option is extremely useful to conduct code analysis on these branches alongside your main branch (master).

Defining long-living branches

Long-living branches are defined on a project level. Simply navigate to your Project Settings > Administration > Branches & Pull Requests. The long-living branches follow a regular expression pattern.

By default, the pattern is (branch|release)-.*. This means that when the name of the branch starts with branch- or release-, it will be considered a long-living branch.

Project Settings

All of the aforementioned concepts and terminologies are in fact project and organisation settings (Quality Gates, Quality Profiles, Long-living branches, New Code Definitions, Monorepo Support). Accompanied with are general settings such as Code Coverage Exclusions, Test File Inclusions and Exclusions, Duplication Inclusions, Source File Exclusions and Inclusions, and many more. You may refer to their documentation to set these up.

Summary

I hope this Part 1 article has provided you with insights on how Sonar Cloud operates and the software engineering quality it follows. Part 2 of this article will provide you a walkthrough on how to set up a Sonar Cloud project with a GitHub repository and perform code analysis using GitHub Actions.

Image Source: https://alexandrebrisebois.files.wordpress.com/2014/05/2011-09-18_code_reviews.png