The Ops Community ⚙️: David Krohn

Deploying CloudFormation StackSets with AWS CDK

David Krohn — Fri, 21 Jul 2023 07:07:49 +0000

AWS Cloud Development Kit (CDK) is a powerful framework that allows developers to define cloud infrastructure as code using familiar programming languages. With CDK, you can easily provision and manage AWS resources in a consistent and automated manner. In this blog post, we'll walk you through the process of creating a StackSet using AWS CDK.

Before we dive into the details, let's take a quick look at what a StackSet is and how it can help you manage your AWS infrastructure.

StackSets are containers for CloudFormation stacks that enable simultaneous creation, update, and deletion across multiple AWS accounts and regions. With StacksSets, you can ensure that all environments are consistent and compliant with the policies you have in place.

The native support for StackSet in CDK is somewhat rudimentary, due to the fact that it is more common in CDK to use CDK pipelines to roll out stacks to multiple accounts and regions.

Therefore, in order to use StackSet in CDK, a few things need to be considered. In this blogpost, we will show steps on how to deploy StackSets via CDK.

Let’s see how this works.

Prerequisites

To follow this tutorial, make sure you have the following prerequisites

An AWS account with appropriate permissions to create StackSets and associated resources.
AWS Command Line Interface (CLI)
AWS Cloud Development Kit (AWS CDK)
Basic knowledge of TypeScript

Bootstrap AWS Account

Bootstrapping is the process of providing resources for the AWS CDK before you can deploy AWS CDK applications in an AWS environment. Normally you could use the default

cdk bootstrap aws://ACCOUNT-NUMBER-1/REGION-1 command. However, we need to customise the template to make the assets available to the entire AWS Organization, as we want to use this CDK environment to deploy StackSets. To get the latest version of the CDK bootstrap template, do the following:

cdk bootstrap --show-template > bootstrap-template.yaml

ℹ️ The CDK boostrap template contains an S3 bucket for files and an ECR repository for container images. It also creates few IAM roles.

After that, we need to modify two resources in this template. The s3 bucket for Assets and the KMS Key that will be used to encrypt the assets.

Add the following Part to the Parameters Section of the template:

PrincipalOrgID: Description: >- The identifier of your AWS organization. Used in the KMS key policy and S3 bucket to share the key with all accounts under your organization Type: String

We will reference this Parameter in the Resource section.

Add this CodeSnippet to the FileAssetsBucketEncryptionKey Resource in to the Key Policy Section. This will

KeyPolicy: Statement: - Action: .... - Action: - kms:Decrypt - kms:DescribeKey Effect: Allow Principal: AWS: "*" Resource: "*" Condition: StringEquals: kms:ViaService: - Fn::Sub: s3.${AWS::Region}.amazonaws.com ForAnyValue:StringLike: aws:PrincipalOrgID: - !Ref PrincipalOrgID

Extend the PolicyDocument of the StagingBucketPolicy with the following CodeSnippet. This will ensure that all Accounts of the Organization get access to the objects in the Asset Bucket.

` PolicyDocument:
Id: AccessControl
Version: "2012-10-17"
Statement:
...
- Sid: ''
Effect: Allow
Principal: ''
Action:
- s3:Get

            Resource: !Sub '${StagingBucket.Arn}/*'
            Condition:
              ForAnyValue:StringLike:
                aws:PrincipalOrgID:
                - !Ref PrincipalOrgID
          - Sid: ''
            Effect: Allow
            Principal: '*'
            Action: s3:ListBucket
            Resource: !Sub '${StagingBucket.Arn}'
            Condition:
              ForAnyValue:StringLike:
                aws:PrincipalOrgID:
                - !Ref PrincipalOrgID`

After all the adjustments we need to deploy the template.

aws cloudformation create-stack \ --stack-name CDKToolkit \ --template-body file://bootstrap-template.yaml

Set Up Your CDK Project

After bootstrapping our Account we are ready to Initialize a new CDK project. We will do that in a new directory. The initialisation creates a new CDK project structure with a sample lib/stackset-cdk-demo-stack.ts file, which we will modify to create our StackSet.

` #Create new directory
mkdir stackset-cdk-demo
cd stackset-cdk-demo

#Init new CDK Project
cdk init app --language typescript`

Define the StackSet

Open lib/stackset-cdk-demo-stack.ts and remove the example stack definition. We'll define our stackset instead:

` import * as cdk from 'aws-cdk-lib';
import * as servicecatalog from 'aws-cdk-lib/aws-servicecatalog';
import { StackSetTemplate } from "./stackSetTemplate";
import * as s3 from "aws-cdk-lib/aws-s3";

export class StackSetCdkDemoStack extends cdk.Stack {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);


    const stackSetTemplateSandbox = new StackSetTemplate(this, "stacksettemplate", {
      Config: props.stacksetProps,
      assetBucket: s3.Bucket.fromBucketName(this, "assetbucket", "myCDKAssetBucket")
    });

    new cdk.CfnStackSet(this, "TESTSTACKSET", {
      permissionModel: "SELF_MANAGED",
      stackSetName: "TEST-STACKSET",
      description:
        "example of StackSet with CDK",
      capabilities: ["CAPABILITY_NAMED_IAM"],
      templateUrl: servicecatalog.CloudFormationTemplate.fromProductStack(stackSetTemplateSandbox).bind(this).httpUrl,
      operationPreferences: {
        failureToleranceCount: 30,
        maxConcurrentCount: 30,
      }
    });
  }
}`

⚠️ Ensure to adjust the myCDKAssetBucket to your AWS Account Assets Bucket.

🚨 Using the servicecatalog ProductStack construct we get rid of the PseudoParameter for the assets bucket in lambdas in the template.

❌ Without using the servicatalog ProductStack:

!sub cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}

✅ Using the servicatalog ProductStack:

cdk-hnb659fds-assets-123456789012-eu-central-1

Undefined

This workaround will ensure that all AWS accounts can access the assets in our CDK app.

Create StackSet template

In the lib directory create a new file called stackSetTemplate.ts and add the following code to the file:
`
import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import * as servicecatalog from "aws-cdk-lib/aws-servicecatalog";
import { NodejsFunction } from "aws-cdk-lib/aws-lambda-nodejs";
import * as lambda from "aws-cdk-lib/aws-lambda";

//Standardstackdefinition
export class StackSetTemplate extends servicecatalog.ProductStack {
// export class CdkStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props: cdk.StackProps) {
    super(scope, id, props);

    /**
     * Dummy Node JS Lambda Function
     */
      const lambdaFunction = new NodejsFunction(this, "testFunction", {
        memorySize: 128,
        timeout: cdk.Duration.seconds(60),
        runtime: lambda.Runtime.NODEJS_18_X,
        handler: "handler",
        entry: path.join(__dirname, "lambda/index.ts"),
        bundling: {
          minify: true,
          externalModules: ["aws-sdk"]
        }
      });
      lambdaFunction.applyRemovalPolicy(cdk.RemovalPolicy.RETAIN);
    }
  }
}`

Example Lambda Code

Create a new directory lambda in the lib directory. In the new lambda directory create a new file called index.ts and add the following code:

` import { Handler } from 'aws-lambda';

export const handler: Handler = async (event, context) => {
    console.log('EVENT: \n' + JSON.stringify(event, null, 2));
    return context.logStreamName;
};

Deploy the StackSet

Run cdk deploy

in a terminal to deploy the StackSet and associated CloudFormation stacks.

ℹ️ CDK will ask you to confirm that you want to deploy the changes. Type y and press Enter to continue.

Conclusion

In this blog post, we've explored how to create a StackSet using AWS CDK. We also learned how to share the CDK Assets to the whole AWS Organization. StackSets are an essential tool for managing infrastructure at scale across multiple AWS accounts and regions. Using CDK, you can easily define and deploy complex cloud infrastructure as code, and leverage the full power of AWS CloudFormation to ensure consistency, compliance, and efficiency across your organisation's cloud resources.

Enterprise-scaled Self-Healing StackSets

David Krohn — Thu, 06 Jul 2023 11:55:29 +0000

With more than 5 million articles from over 7,000 brands, OTTO is one of the leading German online shopping platforms. In the future, it will open up to even more brands and partners as part of its transformation. OTTO is part of the internationally active Otto Group, with headquarters in Hamburg, and employs 6,100 people throughout Germany. In the 2020/21 financial year, OTTO generated revenues of 4.5 billion euros.

At OTTO, we faced several challenges to operate AWS CloudFormation StackSets at Scale. We must govern several hundred AWS accounts for our product teams, all while balancing the need for agility and control.

At this scale, operations can take a lot of time, because there are multiple operational tasks that we need to do when AWS accounts are leaving the AWS Organization or Teams are nuking the AWS account, StackSets Instances get drifted, because not all required resources for compliance can be secured ( SCP Limitations ), existing AWS accounts are joining the AWS Organization and all mandatory StackSets needs to be deployed, and manual steps should be reduced to a minimum. Furthermore, there is no feature from the Service itself to gain an overview of the status of drifted Instances and the general health of your StackSet health and compliance.

The cloud competence center at OTTO IT, also known as the Governance at Scale (GAS) team, developed a solution for self-healing on StackSets, that is integrated into the OTTO tooling ecosystem with Confluence and Microsoft Teams.

OTTO worked with globaldatanet to set up its Landing Zone. globaldatanet is an award-winning AWS Advanced Consulting Partner and longtime Cloud Solution Provider for OTTO, supporting the team in cloud security and GAS. Their focus on building cloud-native solutions using Serverless supported over 100 companies within 5 years to develop and innovate products and services in the cloud.

In this post, we’ll demonstrate how to implement fully automated enterprise-scaled self-healing on StackSets using AWS StepFunctions and create a Dashboard to get an overview of your StackSet health and compliance and reduce operational time.

The solution workflow includes the following steps:

The tagging concept for StackSets
Automatically create StackSets configuration in SSM Parameter Store
Implementing StepFunction for StackSet Self-Healing

Let’s see how this works.

Prerequisites

The following prerequisites are necessary for following along with the contents of this post:

Two existing AWS Accounts
Few AWS StackSets

Solution overview

The following architecture shows the whole solution of the Self Healing StackSets.

Architecture of fully-automated Self Healing Solution with integration to Confluence.

Tagging concept for StackSets

The solution requires a JSON file in the AWS parameter store, the easiest way is to create it automatically based on the StackSet configurations and the tags assigned there. We'll go into more detail about this in the next section of the Automatically create StackSets configuration Parameter Store article. In the following, we describe which tags we introduced to our StackSet and what we need these tags for.

⚠️ AWS tags do not allow commas in value, so ":" as divider for arrays

Key	Value	Result	Example
antidependson	StackSet Name	antidependson marks stacksets which collide with each other.	MYSTACKSET
dependson	[List of StackSet Names]	List of Stacksets that need to be rolled out before deploying this stackset (e.g. Enable Config before Activate Config Rules). NOTE : Please reduce to only one dependson-stackset for now. Form "chains" for multi-dependencies.	MY-STACKSET1:MYSTACKSET2
mandatory	true or false	The stackset instances must be present on all AWS accounts	true
selfhealing	true or false	StackSet can be healed via Delete & Redeploy (exception e.g. IDP roles) - Parameter Overwrites will be cached.	true
region	[Regions]	List of Regions in which the stackset instances are to be deployed	eu-west-1:eu-central-1:us-east-1

Automatically create StackSets configuration Parameter Store

The automated generation of the Stackset-configuration via JSON inside the ParameterStore is a multi-purpose-utility:

Removing the chore to configure manually a JSON-document
Ensure the Account vending-machines knows what to deploy in which order
Supporting the self-healing StepFunction about the expected setup of the member-accounts

The Lambda responsible for the task is invoked via a Events-Rule:

Every time a Stackset-Operation has been finished with status "succeeded".

This is due the tags on a Stackset are part of the stackset, not Additional items describing a Stackset, therefore a change to the tags always will result in a Stackset-Update-operation.

In terms of computerscience the Lambda is quite interesting, as the primary problem was to build a nonweighted tree based on the "dependson" and "antidependson" tags and then compile an ordered one-dimensional list, like in the good old "travelling salesmen"-problem.

Implementing StepFunction for StackSet Self-Healing

AWS Step Functions is a cloud service that enables you to coordinate the components of distributed applications and microservices using visual workflows. It allows you to build and automate the execution of complex processes and tasks across multiple AWS services, using a visual interface to define and execute your workflows. Since the Self Healing Solutions needs a complex workflow we decided to use Step Functions for this Usecase. Following we will explain you the workflow of the Self Healing.

StepFunction Workflow

Functionality

ƛ Serverless Functions

StackSetInitCleanupLambda: Performs a search to identify StackSet instances of AWS Accounts that are either not present within the AWS Organization or deployed to AWS accounts that are suspended. Once identified, proceed with the deletion of these instances from all associated StackSets.
MandatoryStackSetDeploymentLambda: Search missing StackSets Instances (which are tagged with mandatory = true) and deploy those Instances
StackSetDriftDetectionLambda: Trigger Drift Detection on all StackSets
TriggerDriftStatusLambda: Check if Drift Detection is completed on all StackSets
SearchStackSetInstanceToHealLambda: Searches for drifted StackSet Instances from StackSets which are tagged with Selfhealing = true
StackSetCleanupLambda: Removes unhealthy StackSet Instances and redeploys them. Parameter Overrides will be cached so the new deployed instance will have the same setting as before.
StatusPrepareHTMLLambda: Prepare the HTML output Dashboard for Confluence and Json log file of the current StackSet Healthiness State
TeamsNotificationLambda: Send Teams Notification which summary to notify the GAS Team after each execution

？！Decisions

InitCleanup Complete: Check whether all unnecessary instances have been removed. If not, StepFunction is triggering the StackSetInitCleanupLambda function again.
MandatoryStackSetDeployment Complete: Checks whether all mandatory instances have been deployed. If not, StepFunction is triggering the MandatoryStackSetDeploymentLambda function again.
StackSetDriftDetection Complete: Wait until StackSet Drift Detection has been finished on all StackSets
Healing Complete: Check if all unhealthy Instances are healed otherwise invoke StackSetCleanupLambda again

Limitations

While developing the solution we faced several limitations. Here are our findings and solutions for that.

🚨 StackSets instance operations: Maximum number of stack instances, across all stack sets, that you can run operations on in each Region at the same time, per administrator account is limited to 10.000 operations.

✅ We implemented a counter to count the current StackSets operations which are in progress, in addition we also catching the Exception from CloudFormation and waiting few seconds to try the operation again.
🚨 Parameter Overwrites Caching: Whenever removing a drifted StackSet Instance which has Parameter Overwrite you will lose the individually parameters of the Instance.

✅ Before deleting the drifted StackSet Instance we cache the Parameter Overwrites and deploy the StackSet Instance after successful deletion again with the cached Parameter Overwrites again.
🚨AWS Step Functions Payload size: AWS Step Functions supports payload sizes up to 256KB. For our solution we need more Payloads between the States especially when we want to pass our log or the concurrent Parameter Overwrites per StackSet.

✅ We are storing our states in an S3 bucket to pass the state. At the end of the execution we are deleting the state from S3 to not to influence the next Step Function execution with the wrong state.

Documentation

After each execution of the StackSet Health StepFunction, we aim to notify our GAS team about the actions taken during the previous run. Therefore, we have implemented a Teams notification that includes a status update, a link to the generated dashboard, and a link to the log file.

The following screenshot illustrates an example of a Teams notification. It provides a summary report and directs you to the dashboard and log file for further details.

Dashboard

Our StackSet Health Dashboard is a simple HTML file which will be generated trough a Lambda Function, saved in S3 and will be distributed trough a CloudFrount. You can integrate this Dashboards in your Confluence or any other internal Wiki. This Dashboard is secured via CloudFormation Function - additionally you can also add a Firewall to restrict the access to an specific CIDR or Geographic region and prevent access from third parties. The screenshot below provides an example of the overall StackSet Health status information for an entire AWS Organization.

Conclusion

In this post, we demonstrated a solution to automatically heal AWS CloudFormation StackSets at scale. By implementing this Solution Organisations we reduced manual effort for StackSet cleanup operations by 4 hours per week, improved the overall reliability of our StackSets, increased our compliance in the organisation, and managed to get a daily updated overview for all StackSet Instances using the dashboards. In summary, the self-healing CloudFormation StackSets solution combines automation, monitoring, and self-recovery capabilities to deliver a robust and resilient system for StackSets.

AWS Landing Zone versus AWS Control Tower

David Krohn — Mon, 30 May 2022 07:40:26 +0000

What is the difference between AWS Landing Zones and AWS Control Tower? Customised Solution or Managed Service?!

AWS Landing Zone and AWS Control Tower help set up and govern a new, secure, multi-account AWS environment based on AWS best practices. Both consist of core accounts and resources which will implement a initial security baseline.
The following table compares the managed service (AWS Control Tower) with the solution (AWS Landing Zone).

Update:

🚨 AWS Control Tower allows existing organizations to set up a landing zone.

Feature
Delivery mechanism	CloudFormation or Terraform	AWS managed service
Architectural support	Fully customizable and owned by customer	Customizable via Solution + AWS recommend best practices with managed blueprints and guardrails
Account structure	Complete flexibility for customer-defined account structure	Two non-configurable core accounts, no SS, no Amazon VPC in core
Federated access	AWS SSO, AWS-Managed Microsoft AD or Active Directory Connector	Preconfigured with AWS SSO (AD or SSO Directory?) and integrated with third-party SSO providers
Operations	Extensible capabilities to manage the most complex and advanced environments	Simple setup and management for reduced operational overhead
Automated account creation	✅ Account Vending Machine	✅
Member account region support (VPC)	✅ All regions are supported¹	➖ North-Virginia (us-east-1), Ohio (us-east-2), Oregon (us-west-2), Irland (eu-west-1), Sydney (ap-southeast-2) ²
General region support	✅ All regions are supported	➖ North-Virginia (us-east-1), Ohio (us-east-2), Oregon (us-west-2), Irland (eu-west-1), Sydney (ap-southeast-2)
Use existing AWS Organization	✅	✅
Use existing SSO environment	✅	❌
Use existing AWS Service Catalog environment	✅	❌
New or Existing Security Hub environment	✅ Multiaccount Scripts	✅

References

AWS Landing Zone

AWS Control Tower

📚 User Guide
📚 Pricing
🎓 Labs
📺 Videos
🧰 Solutions
- 🔧 Customizations for AWS Control Tower
- 🔧 Enabling guardrails in new AWS Regions the AWS Control Tower supports

Which one should I choose?

❓Are you new two AWS?
❗️Use AWS Control Tower
❓Do you need a configurable landing zone with full customization and control over every part?
❗️Use AWS Landing Zone

Member accounts could be provisioned in every region no matter where the Account Vending Machine is deployed. ⚠️You just need to take care that your CloudFormation templates & Lambdas are available in the requested region. ↩
AWS Control Tower could provision new Accounts (Network baseline) into the following regions: North-Virginia (us-east-1), Ohio (us-east-2), Oregon (us-west-2), Irland (eu-west-1) and Sydney (ap-southeast-2). ↩

Encryption of SSM session data using KMS

David Krohn — Mon, 30 May 2022 07:39:47 +0000

Connecting to your EC2 Instances using Session Manager is for years already a default. However most of the users are just using the defaultTLS 1.2 encryption that AWS already provides by default. For most of the usecases this is ok, but sometimes it is required to encrypt your sessions with your own KMS Keys. Encrypting session data with your key also enables sessions to handle confidential data interactions, such as password resets, and further improves your security posture when using Systems Manager Session Manager. To use the option to encrypt session data using a key created in AWS KMS, version 2.3.539.0 or later AWS Systems Manager SSM Agent must be installed on the managed instance.

In addition i would also recommend to log the session data either to S3 or to CloudWatch and to encrypt the session data which is logged to the destination using your own KMS Key. In the following post I will explain how to automatically configure the SSM default settings to secure your session data using your own KMS key.

Setup

Set up the required KMS Keys using this template using StackSet or Stack - this template will create the needed KMS Keys for SSM and CloudWatch with permissions.
Create a LambdaLayer with boto3 or use a prebuild layer.
Deploy the following template using StackSet or Stack - the template which I provided will configure your Session Manager.

After successfull deployment:

Optionally secure CloudWatch LogGroup using SCPs. Since the CloudWatch LogGroup from Session Manager will contain sensitve data, I would recommend to secure the LogGroup using SCPs so that only a few people from your audit team can read them.

{
  "Version": "2012-10-17",
  "Statement": [
{
      "Sid": "CloudWatchLogsDeny",
      "Effect": "Deny",
      "Action": [
        "logs:DescribeLogStreams"
      ],
      "Resource": [
        "arn:aws:logs:*:*:log-group:/aws/ssm/SessionManagerLogGroup:*"
      ],
      "Condition": {
        "ArnNotLike": {
          "aws:PrincipalArn": [
            "arn:aws:iam::*:role/AuditRole"
          ]
        }
      }
    }
  ]
}

Usage

Now you can either use your browser or CLI to connect to your instance using the Session Manager and during the session all data will be encrypted using your own KMS Key.

ℹ️ You can not use aws ssm start-session with the --profile flag that needs a MFA because there is an 🚨 issue right now that the command doens't ask for MFA.

Workaround: Using awsume solves this problem.

CloudFront Functions

David Krohn — Mon, 30 May 2022 07:39:12 +0000

A few weeks ago Amazon announced a new feature for Amazon CloudFront to run code in Edge Locations. But where is the difference between Lambda@Edge and CloudFront Functions?
CloudFront Functions are running in Edge locations whereas Lambda@Edge functions are executed in a regional edge cache (eg.: the AWS region closest to the CloudFront edge location reached by the client). Therefore CloudFront Functions are even closer to the client and are at the same time approximately 1/6th the price of Lambda@Edge.

Use Cases

Authorization: Implement authorization for the content delivered through CloudFront using Basic Authentication or by creating and validating user-generated tokens.

Redirects: Redirect users to a different URL - eg.: If you change to a new website structure you can redirect the user to the new URL.

Header Manipulation: Add, modify, or delete any of the request/response headers - eg.: foward the IP of the client using the Header to your origin.

CloudFront Functions versus Lambda@Edge

Features

Most important differences - if you need more information check this docs: Choosing between CloudFront Functions and Lambda@Edge.

	CloudFront Functions	Lambda@Edge
Execution location	CloudFront Edge Locations	CloudFront Regional Edge Caches
Programming languages	JavaScript (ECMAScript 5.1 compliant)	Python, Nodejs
Event sources	Viewer request Viewer response	Viewer request Viewer response Origin request Origin response
Memory	2 MB	128 MB (viewer triggers) – 10 GB (origin triggers)
Max size of Function	10 KB	1 MB (viewer request / response) 50 MB (origin request / response)
Max execution time	1 ms	5 seconds (viewer request / response) 30 seconds (origin request / response)
Access to geolocation and device data	✅	❌ (viewer request) ✅ (viewer response) ✅ (origin request) ✅ (origin response)
Access to the request body	❌	✅

Pricing example

Service	Price per Invocation	Price per Duration (for every GB-second)	Invocations	Duration	Allocated Memory	Total Cost
CloudFront Function	$0.1	-	2 Million	1ms	-	$2.0
Lambda@Edge	$0.6	$0,00005001	2 Million	10ms	128MB	$12.26

The prices were checked on 30.05.2021 from Lambda@Edge pricing and CloudFront Function pricing

Example template for Basic Auth with CloudFront Functions

Following you will find a CloudFront Function for Basic Auth - I am using it as a second layer of security for private CloudFront origins. For example I am generating exports of Jira content to S3 using a Lambda as a Backup. In Front of CloudFront I have a WAF to restrict to spefic IPs plus these CloudFront functions.

AWSTemplateFormatVersion: 2010-09-09
Description: Creates a Base CloudFront Function for Authentification
Metadata:
  Author:
    Description: David Krohn

Parameters:
  CloudFrountUsername:
    Description: Username CloudFront
    Type: String
  CloudFrountPassword:
    Description: Password CloudFront
    Type: String
    NoEcho: true
Ressources:
  CloudFrontFunctionBasicAuth:
    Type: AWS::CloudFront::Function
    Properties: 
      AutoPublish: true
      FunctionCode: !Sub |
        var USERS = {
            Website: [{
                username: '${CloudFrountUsername}',
                password: '${CloudFrountPassword}',
            }],
        };

        //Response when auth is not valid.
        var response401 = { 
            statusCode: 401,
            statusDescription: 'Unauthorized',
            headers: {
                'www-authenticate': {
                    value: 'Basic'
                },
            },
        };

        var b64 = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/=";

        function btoa(input) {
            input = String(input);
            var bitmap, a, b, c,
                result = "",
                i = 0,
                rest = input.length % 3; // To determine the final padding

            for (; i < input.length;) {
                if ((a = input.charCodeAt(i++)) > 255 ||
                    (b = input.charCodeAt(i++)) > 255 ||
                    (c = input.charCodeAt(i++)) > 255)
                    throw new TypeError("Failed to execute 'btoa' on 'Window': The string to be encoded contains characters outside of the Latin1 range.");

                bitmap = (a << 16) | (b << 8) | c;
                result += b64.charAt(bitmap >> 18 & 63) + b64.charAt(bitmap >> 12 & 63) +
                    b64.charAt(bitmap >> 6 & 63) + b64.charAt(bitmap & 63);
            }

            // If there's need of padding, replace the last 'A's with equal signs
            return rest ? result.slice(0, rest - 3) + "===".substring(rest) : result;
        }

        function handler(event) {
            var request = event.request;
            var headers = request.headers;

            var auth = request.headers.authorization && request.headers.authorization.value;


            var users = USERS['Website'];

            if (users) {
                if (!auth || !auth.startsWith('Basic ')) {
                    return response401;
                }
                if(!users.find(function(user) {

                        // Construct the Basic Auth string
                        var authString = 'Basic ' + btoa(user.username + ':' + user.password);

                        return authString === auth;
                    })) {
                    return response401;
                }
            }
            return request;
        }

      FunctionConfig:
        Comment: !Sub 'Basic Auth for S3 Bucket ${MyWebsiteBucket}'
        Runtime: cloudfront-js-1.0

More samples can be found here: Amazon CloudFront Functions Samples.

One week at globaldatanet as AWS Cloud Engineer - David Edition

David Krohn — Mon, 30 May 2022 07:38:38 +0000

Hello, I am David 👋🏻 - Today I'm going to give you an in-depth look at my week as a Senior AWS Cloud Engineer at globaldatanet. Right now I am working on 2 customer projects which means more structuring of my week but also giving me the possibility to work with a diversified range of different tools and tasks. Usually, roughly one day of my week is dedicated to a customer working in the e-commerce industry, the other days are assigned to another customer, a big insurance company. Let’s dive right in! 🤿.

My week 🗓

I usually start the day with a cup of coffee. At the same time, I first check Slack and Teams to see what news there has been in the past few days. I also scan Twitter and AWS Blog Post for exciting new features for social media and our customers.

Monday

On Mondays, before the day really gets going, I take the time to plan the whole week. That means I first get an overview of the upcoming tasks for this week, which user stories were completed last week and which are coming up this week.
Typically the day starts with a daily team meeting with the customer, in which the tasks for the day are briefly defined, and the previous week is compactly reviewed as well. This Monday I primarily did some troubleshooting for Athena Partitions, fixing the lambda to automatically create new partitions for budget report dashboards every month. For the same customer, I also spend some time building a tool to detect the reason for the cost explosion. For example, showing that one Fargate was corrupted and was starting to create/ delete network interfaces 400 times higher than usual each day. As part of my day, I also worked on a cost optimization task, decoding Alexa Skills mp3s (320kbit to 128kbits) on Amazon S3, as well as adjusting the max-age for S3 objects. As a highlight this Monday we also had a call with AWS directly to present our previously implemented solutions for our current customer and discuss potential new opportunities.

Tuesday

Business as usual Tuesday also started with the daily at 9. Followed up by a call with the technical manager to discuss the user stories in more detail. What have I been working on this Tuesday? Mainly doing the automation for the customer internal web application firewall, as well as scanning the whole organization for non-compliant resources, cleaning up & fixing those resources in accordance with Config cleanup rules.

Wednesday

As you already know how my day starts I will jump right into my tasks on Wednesday. Besides the Config cleanup rules which I was still working on, I also fixed the automation for ConfigAggregators as the aggregators were not processed properly when moving from one account to another organizational unit. Currently, I am also preparing an AWS blog post, as well as we are discussing further details about the customer internal web application firewall, which means today was more of a mix between working on my customer projects and some planning meetings. Besides that, I also reached out to the diagrams.net developers to get more information on the API for the open-source orgtool extension.

Thursday

On Thursday I was mainly creating a tool to automatically create IAM users, tags, and groups to JSON and CSV. Additionally, I also added a feature in the internal cost detection explosion tool to export CSV files. To make sure previously encountered issues can be overcome with ease in the future, I also created a Cleanup/ Closing Account Runbook for the customer.

Friday

On Friday mornings we have our weekly globaldatanet meeting. In 30 minutes the team can exchange information about the different tasks within the week. It's always a good opportunity to stay up to date with what my colleagues are currently working on, even if you are not working on the same project. Mainly for the day, I was still working on the creation of the tool for IAM and also did spend some time reviewing the code and features developed by a colleague for a previous customer project. Currently, we are working on the security competency from AWS, so we dedicated some time on Friday to prepare for the requirements and the external audit.
As for every Friday, our team ends the week with our 🦾 Tech & Beer 🍺. This week Anton shared his journey on how to add custom checks to checkov. Always a highlight and a nice way of ending the week. Unfortunately, right now office time is rare otherwise the Tech & Beer usually is a good starting point for some further drinking and spending some time with the team after work.

Highlight of the month 🚀

For me, it was the possibility to show the AWS Team the work we have done for customer A throughout the last year. Wrapping up our developed services like dashboards and an Account Factory.

Most important Tools and Service used this week 🛠

{
      "week": [
            {
                  "Id": "16",
                  "AWS-Services": [
                    "AWS CloudFormation",
                    "AWS CloudTrail",
                    "AWS Config",
                    "Amazon CloudFront",
                    "AWS Cost Explorer",
                    "AWS Lambda",
                    "AWS Key Management Service",
                    "Amazon Athena"
                  ],
                  "Languages": [
                    "Python",
                    "nodejs",
                  ],
                  "Tools": [
                    {
                        "name": "ORGTOOL",
                        "github": "https://github.com/daknhh/aws-orgtool"

                    },
                    {
                        "name": "cfn-python-lint",
                        "github": "https://github.com/aws-cloudformation/cfn-python-lint"

                    },
                    {
                        "name": "cfn-diagram",
                        "github": "https://github.com/mhlabs/cfn-diagram"

                    },
                    {
                        "name": "Taskfile",
                        "github": "https://github.com/Wildhoney/Taskfile"

                    }
                  ],

            }
      ]
}

Bottom line ❤️

At globaldatanet, I can work with colleagues but also spend time with friends. Whether it's the lunch break together, a drink (or a few ;)) after work, or just a quick chat in between. Throughout my different projects, I have the possibility to work with an exciting tech stack in different industries while using the cutting edge technologies, especially related to AWS Services. I like the flexibility and the start-up feeling that globaldatanet retained, meaning flat hierarchies, quick decisions and the possibility to participate in a lot of cool things and internal decisions.

Optimize your workloads for Sustainability

David Krohn — Mon, 30 May 2022 07:37:33 +0000

Sustainability means "meeting the needs of the present without compromising the ability of future generations to meet their needs," according to the Brundtland Commission. For cloud sustainability, it means addressing the long-term environmental, economic, and societal impact of your business activities. Last year at re:invent 2021 AWS introduced the new Sustainability Pillar for AWS Well-Architected Framework and at the beginning of this year AWS introduced the AWS Customer Carbon Footprint Tool. According to the energy transition plan, Amazon is on the path to powering its operations with 100% renewable energy by 2025. But environmental sustainability is a shared responsibility between you and AWS. This means that AWS maintains the sustainability of the cloud by delivering efficient, shared infrastructure, and you should optimize your workload to efficiently use these resources.

In this post, I will use the design principles for sustainability in the cloud from the Sustainability Pillar of the Well-Architected Framework to explain how to design your workloads to maximize sustainability and minimize impact.

Design Principles:

Understand your impact.
Establish sustainability goals.
Maximize utilization.
Anticipate and adopt new, more efficient hardware and software offerings.
Use managed services.
Reduce the downstream impact of your cloud workloads.

Understand your impact

Understand your workload’s environmental impact using metrics to see if you meet your sustainability goals:

Establish sustainability goals

Establishing short and long-term sustainability goals for your workload, such as reducing the computed resources required per transaction or improving the power efficiency of your compute workload, reducing the network data traveled per request. For example, even if you are not able to go serverless with your application in the first step, you can start to decouple parts of your workload and create small functions to get there in the long run. Other options would include containerizing your workloads or improving the energy efficiency of your compute load by moving to Graviton2-based instances and/or functions. There are multiple transformation guides for transitioning workloads to AWS Graviton2 (EC2, Lambda, RDS & Aurora ). Here is also a nice list of AWS services where graviton processors are available.

To reduce the network data traveled per request, I can recommend you to use a CDN. When requested, the static content is cached from the original server and delivers it to the user. This shortens the distance each packet has to travel. If you use CloudFront as your CDN, you should also enable automatic compression - it will compress certain types of files and serve the compressed objects when viewers support them.

Maximize utilization

Use tools like AWS lambda power tuning, and AWS Compute Optimizer to right-size your resources. In addition, you should use Auto Scaling to automatically scale up and down based on demand. To improve the overall resource efficiency and reduce idle capacity in the entire Cloud AWS, use Amazon EC2 Spot Instances. Spot Instances are unused EC2 capacity in AWS, this instance also gives you up to a 90% discount compared to On-Demand prices. If you are using EKS you should follow the EKS Best Practices Guides - especially the Auto Scaling guide to improve the utilization of your nodes. Moreover, you should generally adopt a serverless, event-driven architecture to maximize overall resource utilization.

To improve the utilization of your AWS storage layer, you should analyze data access patterns to move the storage to different storage classes, with efficient long-term storage, you will optimize your storage footprint. To automatically move the data to another storage class or delete unused data, you can use lifecycle configurations. The following services can configure the lifecycle of your data:

Anticipate and adopt new, more efficient hardware and software offerings

As already mentioned in the Establish sustainability goals section, you should always make sure to use the latest hardware generation such as Graviton processors, or when it comes to GPU usage you should use flexible graphics acceleration rather than dedicated GPU instances. The same counts for your software and its dependencies. Not only from the security aspect, you should always stay up to date or maybe change to a newer programming language that uses hardware resources even more efficiently.

Use managed services

Use managed services, such as Amazon MSK Serverless (GA since 28 April 2022), Amazon Aurora Serverless v2, or Amazon SageMaker Serverless Inference, to help distributing workloads while reducing the overall amount of infrastructure required. This also shifts the responsibility of sustainability optimization to AWS.

Reduce the downstream impact of your cloud workloads

To reduce the amount of energy that is required to use your services always stay up to date with the evolution of hardware and software features like:

Use efficient programming languages like Rust
Use efficient processors to run your workloads like Graviton - the newest generation 3 will consume up to 60% less energy than the generation 2
Analyze the performance of your application using CloudWatch Logs Insights to find the performance guzzler

We at globaldatanet also want to contribute to making the world a little greener and better. Therefore, we have decided to work with Eden Reforestation Projects, one of the world's best reforestation partners, and from now on we will plant 1000 new trees every single month. If you want to learn more about that, have a look at our sustainability page.

Introducing the AWS Firewall Factory

David Krohn — Mon, 30 May 2022 07:36:42 +0000

A few days ago we introduced our AWS Firewall Factory. A simple solution that helps you deploy, update, and stage your Web Application Firewalls while managing them centrally via AWS Firewall Manager.

What are Web Application Firewalls?

AWS Web Application Firewalls (WAFs) protect web applications and APIs from typical attacks from the Internet that can compromise security and availability, and put undue strain on servers and resources. The AWS WAF provides prebuilt security rules that help to control bot traffic and block attack patterns. However, with its help, you can also create your own rules based on your specific requirements. In simple scenarios and for smaller applications, this is very easy to implement on an individual basis. However, in larger environments with tens or even hundreds of applications, it is advisable to aim for central governance and automation. If you want to learn more about this solution or Web application firewalls in general feel free to register to this event.

Architecture

Let's take a quick look at the solution architecture. In order to be able to provision WAFs using AWS Firewall Manager to other accounts we first need to set the Firewall Manager administrator account in our AWS Organization. Additionally we need to have a central S3 Bucket which we will use for our WAF logs. The logs we could use to see potential false positives to develop custom WAF rules. The easiest way to do this is to query the S3 bucket via Athena.

But first you should just start with ManagedRuleGroups to have a base set to secure your applications. AWS Managed Rules for AWS WAF is a managed service that provides protection against common application vulnerabilities or other unwanted traffic, without having to write your own rules.

After deploying your first firewall and using ManagedRuleGroups to detect errors in your application, you can use aforementioned s3 log bucket to create queries and define your own custom rules using the false positives. You do this simply by defining the RuleStatements around your statements in your value file. During the deployment process, the solution calculates the required capacity using the CheckCapacity API from AWS and creates RuleGroups that are attached to the WAF. We have also developed an algorithm that pads the RuleGroups to the maximum RuleGroup size and only then creates a new RuleGroup.

All settings where the WAF should be deployed and with which settings (should be made) are controlled via the dedicated value file. Currently we only support a deployment via Taskfile. Later we will also support deployments via Teamcity.

Currently implemented Features

Automated Capacity Calculation via API - CheckCapacity
Algorithm to split Rules into RuleGroups
Automated Update of RuleGroup if Capacity Changed
Add ManagedRuleGroups via configuration file
Automated Generation of draw.io diagram for each WAF
Checking of the softlimit quota for WCU set in the AWS Account (Stop deployment if Calculated WCU is above the quota)
Easy configuration of WAF Rules trough json file.
Deployment Hash to deploy same WAF more than one time for testing and/or blue/green deployments.
Stopping deployment if soft limit will be exceeded: Firewall Manager policies per organization per Region (L-0B28E140) - Maximum number of web ACL capacity units in a web ACL in WAF for regional (L-D9F31E8A)
NEW RegexMatchStatement and IPSetReferenceStatement is working now 🚀
NEW You can now name your Rules. If you define a Name in your RulesArray the Name + a Base36 Timestamp will be used for creation of your Rule - otherwise a name will be generated. This will help you to query your logs in Athena. The same Rulename also apply to the metric just with adding "-metric" to the name.
New Support for Captcha - You can now add Captcha as Action to your WAFs. This help you to block unwanted bot traffic by requiring users to successfully complete challenges before their web request are allowed to reach AWS WAF protected resources. AWS WAF Captcha is available in the US East (N. Virginia), US West (Oregon), Europe (Frankfurt), South America (Sao Paulo), and Asia Pacific (Singapore) AWS Regions and supports Application Load Balancer, Amazon API Gateway, and AWS AppSync resources.

We hope you'll find this solution helpful to secure your environment! If you have any feedback about the solution, please feel free to reach out to us or open a github issue.

CloudFormation vs. Terraform

David Krohn — Mon, 30 May 2022 07:35:52 +0000

What are the differences? Which tool is the best for your needs?

When you want to implement infrastructure as code, you always come to the question if you should use CloudFormation or HashiCorp's open-source tool Terraform. Both have their advantages and disadvantages. I compared both in the following table to help you decide which tool is best suited for your needs.

Update

Since there was a lot of feedback that you would like to have a comparison including the AWS CDK, I expanded the table by one column for the AWS CDK. I hope the table will help you to make the decision for the right tool for you.

⁉️ Some people asked me to add Pulumi as well to the table - here is your update 🚀.

Feature
Supported AWS Resources	500+	500+	400+	400+
Integration of new features after announcement	➖ After announcement of a new feature there is always a delay until it is integrated into CDK, but you can use Escape Hatches to workaround these missing features.	➖ After announcement of a new feature there is always a delay until it is integrated into CloudFormation, but as a workaround you can develop custom resources to cover new features.	➖ After announcement of a new feature there is always a delay until it is integrated into Terraform, but as a workaround you can use terraform-aws-anything to cover new features.	➖ After announcement of a new feature there is always a delay until it is integrated into Pulumi.
License and Support	➖ Open-source with support from AWS.	➖ Free with support from AWS and large community.	➖ Open-source with support from Hashicorp and large community.	➖ Open-source with support from Pulimi and large community.
Third Party	✅ CDK supports third-party resources.	✅ CloudFormation supports third-Party vendors to create resource providers.	✅ Terraform supports third-party providers and provisioners.	✅ Pulumi supports third-party providers.
CLI	✅ CDK CLI enables you to list the stacks defined in your CDK app, synthesize the stacks into CloudFormation templates, determine the differences between running stack instances and the stacks defined in your CDK code and deploy stacks to any public AWS Region.	✅ CloudFormation has a CLI that enables you to develop and test AWS and third-party resources and register them for use in AWS CloudFormation.	✅ Terraform has a single command-line application.	✅ Pulumi has a single command-line application.
Language	✅ Supports multiple programming languages Typescript, Javascript, Python, Java, and C#	➖ JSON / YAML - YAML is easier to read than JSON but it forces you to have multiple nested scopes.	✅ HCL is human readable as well as machine-friendly.	✅ Supports multiple programming languages Typescript, Javascript, Python, Go - in preview, and C# - in preview
Modularization	✅ CDK supports modules to create a reproducible infrastructure.	➖ Nested stacks and cross-stack references can be used to achieve modularization.	✅ Terraform modules help to create a reproducible infrastructure.	✅ Pulumi modules help to create a reproducible infrastructure.
State Management	✅ Deploys CloudFormation Stacks to manage its state.	✅ Uses Stacks to manage its state.	✅ By default saves the state locally but it is possible to save the state in Terraform Cloud, S3 or DynamoDB with the remote state feature.	✅ Pulumi saves the state (called checkpoint) locally but it is possible to save the state in Pulumi service backend or S3.
Configuration	✅ Supports Environment Variables, CloudFormation Parameters or you can get values from SSM.	✅ Supports up to 60 Parameters. Import values from output of another stack or import from SSM.	✅ Data sources allow data to be fetched or computed for use elsewhere in Terraform configuration.	✅ Pulumi offers a Config object with various getters and setters for retrieving values, in addition they have Data sources that allow to grep, for example an AMI or an AZ from AWS.
Change management	✅ cdk diff compares the desired state against the stack but it doesn't look at the deployed resources until you deploy. You only find out about discrepancies between the stack and deployed resources at deployment time.	✅ Change Sets help you to verify changes before you apply them on your stack.	✅ terraform plan creates a detailed execution plan before applying your changes.	✅ pulumi preview previews your changes explicitly before deploying.
Error handling and rollback	✅ AutoRollbackConfig helps you to configure the behavior of automatically rolling back for a given Deployment Group.	✅ CloudFormation automatically rolls back to the last working state.	❌	❌ Currently there is no rollback functionality in Pulumi but there is an open RFC to implement this functionality.
Import Existing Resources	❌ Currently its not possible to import resources into CDK, but there is an open RFC to implement this functionality. Additional information - there are workarounds how to use existing resources in CDK.	➖ It is possible to import resources into CloudFormation but only for a few resources.	➖ It is possible to import resources into terraform but it does not generate configuration.	✅Pulumi offers an import resource option to request that a resource defined in your Pulumi program adopts an existing resource in the cloud provider instead of creating a new one.
Rolling updates for Auto-Scaling Groups	✅	✅	✅ You can implement rolling updates for Auto-Scaling Groups using the create before destroy lifecycle policy.	✅ You can implement rolling updates for Auto-Scaling Groups using the createBeforeDestroy lifecycle policy.
External waiting conditions	✅ You can use CfnWaitCondition to coordinate resource creation with configuration actions that are external to the stack creation.	✅ You can use WaitCondition to coordinate resource creation with configuration actions that are external to the stack creation.	✅ You can use null_resource to coordinate resource creation with configuration actions that are external to native Terraform resources.	✅ You pass a customTimeouts object as part of resource options to coordinate resource creation with configuration actions that are external to the resource.
Drift Detection	✅	✅	✅	✅
Visualization of dependencies	✅ CDK has a plugin for Visual Studio Code which visualizes dependencies or you can use the cli.	✅ You can use the AWS CloudFormation Designer to view CloudFormation templates.	✅ You can use the terraform graph command to generate a visual representation of either a configuration or execution plan.	✅ You can use pulumi stack graph to export a stack’s dependency graph to a file.
Multi-Cloud Management	❌	❌	✅	✅

SSH and SCP with AWS SSM

David Krohn — Mon, 30 May 2022 07:33:54 +0000

Using AWS Session Manager with enhanced SSH and SCP capability to connect to your EC2 without using firewalls and bastion hosts

Amazon Web Services recently announced new capabilities in the AWS Systems Manager Session Manager. Users are now capable of tunneling SSH (Secure Shell) and SCP (Secure Copy) connections directly from a local client without the need for the AWS management console.

For years, users have relied on firewalls and bastion hosts in order to securely access cloud assets, but these options have security and management overhead tradeoffs. The Session Manager allows for secure, audited console access to cloud resources without the need for additional ingress points.

Local Prerequisites

In order to perform SCP and SSH operations from your local host to the remote cloud asset, you will need to perform the following setup steps on your client.

Install the latest AWS CLI

Update to the latest AWS CLI – An updated command line interface is required on your local host in order to use these new Session Manager features. The version of the AWS CLI should be at least 1.16.213.

How to get the version: aws --version

Install the Session Manager Plugin

Install the Session Manager Plugin – This plugin allows the AWS cli to launch Session Manager sessions with your local SSH client. The Version should be should be at least 1.1.26.0.

How to get the version: session-manager-plugin --version

Update local host SSH config

The tricky portion of this setup involves altering your local host SSH configuration in order to proxy commands through the AWS session manager for any aws ec2 instance-id.

Download AWS SSM SSH ProxyCommand
Move this script to ~/.ssh/aws-ssm-ec2-proxy-command.sh
Make it executable chmod +x ~/.ssh/aws-ssm-ec2-proxy-command.sh
Add following entry to your ~/.ssh/config

host i-* mi-*
  ProxyCommand ~/.ssh/aws-ssm-ec2-proxy-command.sh %h %r %p

EC2 Prerequisites

You will need to perform the following setup steps on your target EC2 instance.

Instance Profile

By default, AWS Systems Manager doesn't have permission to perform actions on your instances. You must grant access by using an AWS Identity and Access Management (IAM) instance profile. An instance profile is a container that passes IAM role information to an Amazon Elastic Compute Cloud (Amazon EC2) instance at launch. You need to add SSM permission to your Instance Profile

SSM Agent

Ensure the latest SSM Agent on Target Instance

yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm
service amazon-ssm-agent restart

Firewall Configuration

Ensure the security group allow outbound to System Manager. No inbound ssh port is required.

Usage

Once these steps are complete, you will be ready to initiate SSH and SCP connections to your cloud assets directly from your local machine.

Obtain the instance-id of the cloud asset. This can be done via the AWS management console or with the AWS cli command aws ec2 describe-instances, and will have a format similar to i-0ba3d05e2b6c0fb36

SSH can be performed as normal using the instance-id as the hostname. Most SSH command line switches can be used such as using a key in the following example:

export AWS_PROFILE='default'
ssh ec2-user@i-0ba3d05e2b6c0fb36

These connections are secured by IAM access and generate cloudtrail events for logging and monitoring.

While immutable infrastructure is a desired goal for multiple reasons, many will find themselves with a need to access or alter systems running live. The AWS Systems Manager Session Manager allows this capability without the need for additional firewall ingress or bastion hosts.

Update: Use SSO with AWS CLI v2 to connect to EC2 over SSH using SSM

Prerequistes

Install and configure AWS CLI v2
Install the Session Manager Plugin

Login via SSO - AWS CLI v2 to connect to an EC2 over SSH using SSM

Update local host SSH config

Add following entry to your ~/.ssh/config

# SSH over Session Manager
host i-* mi-*
  IdentityFile ~/.ssh/id_rsa
  ProxyCommand ~/.ssh/aws-ssm-ec2-proxy-command.sh %h %r %p ~/.ssh/id_rsa.pub
  StrictHostKeyChecking no

Usage

Login to AWS SSO aws2 sso login --profile default
Export AWS_PROFILE export AWS_PROFILE='default'
SSH into your instance by using the following command.

ssh -i /path/my-key-pair.pem ec2-user@instance-id

Creating and managing Accounts in AWS

David Krohn — Mon, 30 May 2022 07:32:14 +0000

It's best practice to use multiple accounts to follow the least privilige principle. Each account should have different security & compliance controls and access patterns. Development accounts have security and compliance controls, they are typically less restrictive that a production account. Production accounts should never be accessed from a user or just with Read-Only access - each resource of these accounts should be created automatically using codepipelines. Whereas specific global management accounts like a security or central-logging account are limited to users which are part of a security or compliance team. The following post will show you how to create and manage accounts using best practices.

1. Automate account creation

Automate the process of setting up Accounts that are secure, well-architected, and ready to use. If you use the Control Tower for your Landing Zone it's coming already with an Account Factory - there's also already existent solutions to customize the Account Factory to your needs. Even if you use the official landing zone solution there are some examples of Account vending machines on github.

Followed you can see an example architecture of an Account Factory with examples on what kind of customization you can/should do.

2. Track and inspect your costs

Following FinOps principles is a must these days and will persist for a long time. So be aware and track/inspect your costs to be able to save some in the end.
If you want to learn more about FinOps take a look at this post.

2.1 Budget notification

Define and use budgets. Set up notifications using Slack, Teams or Email to warn you if you're about to exceed your allocated amount for cost or usage budgets. Additionally you should use cost allocation tags. When you tag your AWS resources, it’s much easier to organize, categorize and track your AWS costs. Cost allocation tags are useful for tracking expenditure on exploratory workloads in your accounts.

2.2 Inspect cost and usage

To inspect your cost and usage you should create dashboards or use the Cost Explorer to display your past usage & cost and forecast expected spend.
Here is an example workshop on how to create cost intelligence dashoards. You should always check your usage spending for cost optimization as well - Cost Explorer or OHTRU will help you to save some cost.

3. Implement SCPs

Prevent unwanted actions, save costs and apply appropriate SCPs to your accounts. When you design and implement SCPs be sure to put those accounts in specific organizational units. If you want to learn more about SCPs take a look at this blog post: SCP best practices.

4. Use an auditing tool

It is best practice to use CloudTrail as an auditing tool to continuously monitor API calls in your AWS environment. Those CloudTrail logs should be forwarded to your central SIEM - if you don't use a central SIEM you can use EventBridge to create rules that trigger on the information captured by CloudTrail eg.: if someone use the S3 API - PutBucketPublicAccessBlock. Those rule can trigger a Lambda to do automated remediation or send a notification to Slack, Teams or an Email.

5. Check your resources for compliance

Use automated and continuous checks against rules in a set of security standards to identify non-compliant resources in your account. You can use Config & Config Rules or Security Hub for that.

5.1 Config

Config is a continous management and monitoring service for your whole infrastructure. It captures snapshots of your configuration of resources and is able to check using Config Rules to detect for non-compliant resources.
As a prerequisite we need to ensure that Config is activated in your AWS Region.

Config Aggregators with advanced queries will help you to gain visibility about the compliance status of your whole organization.
Config has already some manged rules which are easy to activate. Here is an example for a managed config rule activation using CloudFormation.

# Author: David Krohn
AWSTemplateFormatVersion: 2010-09-09

#-----------------------------------------------------------------------------
#Resources 
#-----------------------------------------------------------------------------
Resources:
  CheckForS3BucketServerSideEncryption:
    Type: AWS::Config::ConfigRule
    Properties:
        ConfigRuleName: AWS-s3-bucket-server-side-encryption-enabled
        Description: Checks that your Amazon S3 bucket either has S3 default encryption enabled or that the S3 bucket policy explicitly denies put-object requests without server side encryption.
        Source:
            Owner: AWS
            SourceIdentifier: S3_BUCKET_SERVER_SIDE_ENCRYPTION_ENABLED

5.2 Security Hub

Security Hub continuously monitors your environment using automated security checks based on the best practices and industry standards that you want to follow. Security Hub provides controls for the following standards:

When you enable a standard, all of the controls for that standard are enabled by default. To adjust a standard to your needs (own compliancy requirements) you can then disable and enable specific controls within an enabled standard.
To disable a control, you can use this API call:

aws securityhub update-standards-control --standards-control-arn <CONTROL_ARN> --control-status "DISABLED" --disabled-reason <DESCRIPTION>

To enable a control, use the following API call:

aws securityhub update-standards-control--standards-control-arn <CONTROL_ARN> --control-status "ENABLED"

6. Access control

Create individual identities to access AWS. As we all know we should create credentials for all users and resources on the one hand but also avoid creating long-term credentials in AWS on the other hand. In the following subsections I have two ideas how to tackle the problems to use AWS best practices for access control.

6.1 Federated access for users

Use federated access to avoid long-term credentials to access the Management Console, call APIs, and access resources, without the need to create an IAM user for each identity. You can use Active Directory, any other identity provider with SAML2.0 and configure an IDP in each account, or you can use the AWS SSO service.

6.2 Do not use access keys

Long-term access keys, such as those associated with IAM users remain valid until you manually revoke them. In many scenarios you dont need to create access keys that never expire. You can create IAM roles and generate temporary security credentials using Security Token Service (STS) instead. IAM roles can be associated and comsumed by almost every resources such as EC2 Instances, Fargate, Lambdas or a mobile app etc. Even if you need to do a cross-account access, you can use an IAM role to establish trust between accounts. However when there is no way around to create access keys, you should think of creating a lambda which is taking care of keys being automatically rotated. In any case, try to avoid that credentials are not accidentally exposed to the whole world.

6.3 Secure root user credentials

Do not use the root user for daily work. In times of a global pandemic like right now - you may think about using a vault to mange the access to the root user credentials and MFA. Additionally you should use root activity monitor to get notified if someone is using the root user for activity. If you want to learn when you need to use the root user take a look at this blog post: Tasks that require root user

8. Resource lifecycle management

Defining a lifecycle for your resources will help you to control costs, keep your resources up to date and saves it from being undocumented and unsupportable.

8.1 Development accounts

In development accounts you should consider to define lifecycle policies for all resoures eg. delete S3 buckets or delete old instances. If you use CloudFormation to deploy all resources you could think of using automatic deletion of your stacks after a defined time. If you want to learn more about that - take a look at this post: Scheduling automatic deletion of AWS CloudFormation stacks. As we all know that in development accounts you sometimes just do some testing using manuall deployment via Console or CLI. You should take a look at aws-nuke. aws-nuke is a tool that delete AWS resources automatically. In addition the tool is supporting filters which helps you to preserve some baseline resources. One idea would be to implement aws-nuke on a codebuild task which is triggered via Cloudwatch event. 💡 The Codebuild could also be deployed in a central mangement account and just assumes a role in the target account.

9. Data lifecycle management

Defining a lifecycle for your data will help you lower the operational costs without losing and reduce the complexity of managing your backup operations.

9.1 Development accounts

Since development accounts should not contain any production data its very easy to define a lifecycle for all data in development accounts. I would suggest to decide a specific timerange after all data will be deleted regularly.

9.2 Production accounts

The decision for production accounts is a little more difficult than for development accounts, but it is also feasible. Here you only have to decide how long you need, which data to restore the application in the event of an error and which log files are needed for how long. Whether it is data on S3, instance snapshots or log files on CloudWatch, all services support mechanisms that either delete the data or move it to cheaper storage after a certain period of time. In the case of log files, I would prefer to keep the data in their place for a longer time than too short. Especially if there are holidays in between and you want to analyze an error and the data is then no longer there, that's bad 😉. Below you can find some posts regarding backup handling:

10. Tagging

Tagging of resources is one of the most important things when it comes to accounts - it helps you to identity who created resources, who is responsible for them, how costs should be allocated and even with lifecycle policies. Therefore you should tag every resource to simplify administration. If you want to learn more about Tagging read the following whtepaper: Tagging Best Practices.

11. Network connectivity

If you ask me: the topic of network connectivity could be completely eliminated nowadays, as the first motto should be API first and applications should only talk to each other via secure APIs, but there are still applications today that do not have corresponding APIs whicb have large amounts of data to be transferred or certain regulations ensure that communication has to take place via dedicated connections. So how to handle them? In organizations where you need dedicated connections or every other VPN connectivity to on-prem, I would choose the Transit Gateway. Transit Gateway connects VPCs and on-premise networks through a central hub. You can create datadomains to restrict and control your complete dataflow of your organization on one hand and on the other hand you expand globally with inter-region peering. Plus your data is automatically encrypted and never travels over the public internet.

Account Factory - provision AWS Accounts

David Krohn — Mon, 30 May 2022 07:31:20 +0000

Best Practice for creating AWS Accounts

What is an Account Factory?

The Account Factory is a solution to manage account creation and bootstrapping in a scalable and efficient manner so that new accounts are created with a defined baseline and governance guardrails are in place. If you want to get your hands dirty with an Account Factory - you can take a look at this workshop.

Why should I use a Account Factory?

⚙️ Equivalent process for every account
💸 Save time and resources
🔐 Secure and well-architected accounts
📜 Documented process
🔌 Native integration to your ticket system like Jira - ServiceNow; this means that internal approval processes can be used to trigger the Account provisioning
🚀 Ready to use accounts in minutes

What solutions are outside for account provisioning?

If you use the Control Tower for your Landing Zone it's coming already with an Account Factory - there's also already existent solutions to customize the Account Factory to your needs. Even if you use the official landing zone solution there are some examples of Account vending machines on github.

What steps are part of an account factory?

Creation of a new AWS Account - the first part of the Account Factory is to provision a new Account, the solution is using the organizations API for that. 💡 Activate IamUserAccessToBilling - othwerwise only the root user of the new account can access account billing information.

Deploy the baseline template to new AWS Accounts - regardless of whether you want to provide resources via API / CDK, StackSets or CodePipeline, you need a role that you can assume in the new account, so we need an AccountBaseline that includes a role with trust for the management or automation account.

Remove default VPC - A default VPC is suitable for getting started quickly - not for production workloads.

Deploy security baseline and other customized ressources - this step includes eg.: the activation of GuardDuty, Config, CloudTrail - if you dont use the Organizational Trail, creation of own VPC, security notification service etc. You can use StackSets or CodePipeline for that. It would make sense to separate the configuration which stacks /resources you want to activate from the actual Account Factory. 💡 You can for example use SSM Parameter Store for that - You could also think of using different configs for services , which are deployed in to new accounts or not depending on the selection in the Account Factory.

Optional:
- Use support case API to activate enterprise Account in the new Account.
- Activate root MFA automatically.

Move new Account to desired organizional unit. According to best practices, it is recommend to structure your accounts in an organization and assign scps to organizational units. Since SCPs can cause problems when providing a new account, I recommend to move the new accounts at the end of the Account Factory to the corresponding unit.

Send status mail or slack / teams message. In order to get a status overview of the new Account, you should think of collecting information while the Account Factory is running and send them as a message to the new Account owner or governance at scale team.