How are patterns configured?

Let's take a look at how we approach configuring and customizing the patterns contained within the worX catalog.

• Within each pattern, there are some aspects of configuration that allow the user to share common customization across all resources within a single stack.
• There are also customizations and features that will be specific to each pattern, each with a number of supported options...
• ...reverting to a set of internal defaults based on good practices if the user enables a feature but provides no further customization.

The diagram below represents the typical hierarchy present in the YAML file used to configure one or more patterns.

Configuration Structure

The benefit of this approach is that we can assemble and customize an entire workload's worth of resources from a single configuration file, each pattern configured independantly if needed.

Compatible with your own Terraform code and workflows

With all of the patterns being native Terraform packages, we can add patterns alongside existing Terraform code. Patterns can be used without any additional Terrform for a workload that is managed entirely via low-code that is also compatible with all of the awesome Terraform ecosystem tools that are already used by the ecosystem community.

Common Configuration

Within the YAML file used to configure one or more patterns there are several options that are considered common to all patterns and will be applied during management of resources.

Resource Naming

The prefix_tokens section controls how managed resources will be named consistently.

workload-config.yml

common_config:
  ##############################################################################
  # Consistent Naming for Resource Prefixes
  #
  # These values generate IDs using the following convention by default in order
  # to ensure consistent naming of resources where possible;
  # {prefix}-{tenant}-{location}-{lifecycle}-{name}-{attributes}.
  #
  # Example IDs:
  #  - resource_prefix = "HIO-EW1-DEV"  --> HIO-EW1-DEV-HUB-Security-VPC
  #  - resource_prefix = "HIO-UE2-PRD"  --> HIO-UE2-PRD-HUB-Monitoring-Subnet-1
  #
  ##############################################################################
  prefix_tokens:
    # A short (3-4 letters) abbreviation of the company name, to ensure globally
    # unique IDs for things like S3 buckets. In large organizations the business
    # unit may be used instead
    # Default Value: 'DEMO'
    org: "HIO"

    # (Rarely needed) When a company creates a dedicated resource per customer,
    # A short (3-4 letters) abbreviation of the tenant can be used to identify
    # the customer the resource is dedicated to
    # Default value: <empty>
    tenant: null

    # The name or role of the account the resource is for, such as prod or dev
    # Default value: 'DEV'
    lifecycle: "dev"

    # A short abbreviation for the AWS region hosting the resource. 'GBL' will be
    # used for resources like IAM roles that have no region
    # Default value: <AWS_DEFAULT_REGION>
    location: null

• When building a resource prefix all of the token values will be converted to uppercase for consistency.
• The tenant value will be collapsed and not used if no value is set in the common config
• The values provided for the prefix_tokens are not truncated but should be as short as possible due to limitations on resource name length for some AWS resources (Security Groups, S3 Buckets, etc)
• The tokens will be combined using a separator (-) to create two prefixes; one for regional resources (ie. EC2, VPC, RDS) and a second for global resources (ie. IAM)

AWS resources with immutable name attributes

Some AWS resources have a name attribute in addition to supporting a Name tag. For these resources it is common for the value of the name attribute to only be set at time of creation. The impact of this is that changing the values of the prefix_tokens can result in resources being deprovisioned and then reprovisioned in order to apply the change.

Let's look at some examples to understand how this works in practice;

Pattern without name customization

example

workload-config.yml

# Shared configuration
common_config:
  prefix_tokens:
    org: "HIO"
    lifecycle: "dev"
  default_tags:
    Lifecycle: "Development"
    CostCenter: "DEMO-REINVENT23"

# RDS instance
enterprise_database:
  demo-db-01:
    db_engine: "mysql"
    db_instance_type: "t2.large"
    tags:
      Function: 'RDS database for the demo for re:Invent'

The configuration above would result in the following resource prefixes when provisioned into the eu-west-1 AWS region;

• Regional resources: HIO-EW1-DEV
• Global resources: HIO-GBL-DEV

Some of the resources will be provisioned using their prefix and may optionally have a Name tag set. Others will only use the prefix;

• RDS Option Group: HIO-EW1-DEV-OptionGroup
• RDS Parameter Group: HIO-EW1-DEV-ParameterGroup
• RDS DB instance: name set as demo-db-01 and Name tag set as demo-db-01

Pattern with name customization

example

workload-config.yml

# Shared configuration
common_config:
  prefix_tokens:
    org: "HIO"
    tenant: "RI23"
    lifecycle: "dev"
  default_tags:
    Lifecycle: "Development"
    CostCenter: "DEMO-REINVENT23"

# RDS instance
enterprise_database:
  demo-db-01:
    db_engine: "mysql"
    db_instance_type: "t2.large"
    tags:
      Name: "eudemodb01"
      Function: 'RDS database for the demo for re:Invent'

The configuration above would result in the following resource prefixes when provisioned into the eu-west-1 AWS region;

• Regional resources: HIO-RI23-EW1-DEV
• Global resources: HIO-RI23-GBL-DEV

Some of the resources will be provisioned using their prefix and may optionally have a Name tag set. Others will only use the prefix;

• RDS Option Group: HIO-RI23-EW1-DEV-OptionGroup
• RDS Parameter Group: HIO-RI23-EW1-DEV-ParameterGroup
• RDS DB instance: name set as demo-db-01 and Name tag set as eudemodb01

Resource Labelling

The default_tags section controls how managed resources will be labelled consistently

workload-config.yml

common_config:
  ##############################################################################
  # Consistent Tagging
  #
  # These tags are added to everything that supports tags. Additional
  # resource-specific tags may be added by individual resources
  #
  ##############################################################################
  default_tags:
    Solution: "Object Storage Testing"
    BusinessUnit: "IT"
    Lifecycle: "Development"
    SupportContact: "[email protected]"
    CostCenter: "ABCD-EFGH"

• Any valid YAML key-value pair can be used
• Values provided will be used as-is and not be case modified, truncated or encoded
• Setting any key-value pair for default_tags will replace all of the internal default tags applied

Magic Variables

Magic variables are the name that we use for variable values that are determined dynamically when Terraform prepares (terraform plan) or actions (terraform apply) changes to a managed resource.

These dynamic variables are used to allow the user to customize some of the information provided when configuring pattern features and act as placeholders that will be swapped for their real values at run-time.

Currently supported magic variables

The following is a list of currently supported magic variables. Not all magic variables are supported in all patterns. For each pattern pattern, the specific magic variables supported will be listed in the per-pattern docs and often give examples in the appendix.

• AWS_ACCOUNT_NUMBER

  Description	The AWS account number for the currently loaded AWS credentials
  Example value	1234567643342

• AWS_USER_ID

  Description	Unique identifier of the user/bot/role making the changes.
  Example value	AROHGJ3DSFMAL6WQ3MW32U:[email protected]

• AWS_USER_ARN

  Description	ARN of the user/bot/role making the changes.
  Example value	arn:aws:sts::1234567643342:assumed-role/AG-SSO-AWS-DevOpsAccess/[email protected]

• AWS_USER_ARN, AWS_REGION_ID

  Description	The name of the Region.
  Example value	eu-west-1

• AWS_BUCKET_ARN

  Description	The ARN of the S3 bucket being managed
  Example value	arn:aws:s3:::jeff-example-bucket

Example

In order to better illustrate how they can be used, let's look at an example from an existing pattern that creates an S3 Bucket Policy snippet from a template

Pattern Configuration by User

The user makes use of magic variables that are specific to the pattern in order to restrict access to only the AWS root user and the user managing the resource

pattern-config.yml

pattern_payload:
  # Unit Payload defaults. This collection of defaults is used with each unit
  unit_defaults:
    unit_features:
      ...
      custom_policy:
        enabled: true
        content: |
          {
            "Version": "2012-10-17",
            "Id": "DeveloperBucketAccess",
            "Statement": [
              {
                "Sid": "AllowAccessWithinAccount",
                "Effect": "Allow",
                "Principal": {
                  "AWS": [
                    "arn:aws:sts::1234567643342:assumed-role/AG-SSO-AWS-DevOpsAccess/[email protected]",
                    "arn:aws:iam::1234567643342:root"
                  ]
                },
                "Action": "s3:*",
                "Resource": [
                  "${AWS_BUCKET_ARN}",
                  "${AWS_BUCKET_ARN}/*"
                ]
              }
            ]
          }

When rendered by the pattern at run-time, the magic variables are replaced with the actual values

s3bucket-policy.json

{
    "Version": "2012-10-17",
    "Id": "DeveloperBucketAccess",
    "Statement": [
        {
            "Sid": "AllowAccessWithinAccount",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "${AWS_USER_ARN}",
                    "arn:aws:iam::${AWS_ACCOUNT_NUMBER}:root"
                ]
            },
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::jeff-example-bucket",
                "arn:aws:s3:::jeff-example-bucket/*"
            ]
        }
    ]
}

Use Cases

There are lots of places where using magic variables can simplify creation and portability of pattern configurations. Some of the most main benefits of this this approach are;

• Policies can be written in a way that allow for customization by the user but without having to worry about AWS resource IDs.
• Policies can be re-used across different resources and workloads.
• Resources that are created and managed can be labelled by who last updated them if required.
• Access to resources can be conveniently limited to the user that creates and manages them.

Pattern Features

Understanding Impact

Patterns will document each feature, indicating how it can be enabled/disabled, as well as provide useful information for the user to determine if the feature is required or beneficial

Feature Properties

Name	A short name that will be part of the Table of Contents in documentation. Try to avoid long names where possible
Description	What behavior this feature manages?
Disruptive	Does changing the configuration of this feature from "off" to "on" cause disruption?
Allowed Values	What are the "on" and "off" values?
Default value	What is the default state/config of this feature if left unconfigured?

Enabling and Disabling Features

Within each pattern there may be optional features that can be switched on or off and will result in additional AWS resources being managed.

• Features will always be switched on with a value of true or off with a value of false to ensure that values are compatible with both YAML and Terraform supported types
• For new features that have not been widely deployed they should be configured as "off" by default unless they are intentionally for improving security or resolving a defect.

Example

• Encryption of EBS volumes on an EC2 VM by default is "on" and uses the aws/ebs KMS key unless customized
• Changing the customization of this feature to go from aws/ebs to customer/cmk KMS key may be non-disruptive if the correct permissions are already in place on the customer/cmk KMS key

Customizing Features

• When a feature is "on", either by default or by configuration by the user, there may be aspects of the feature that can be customized.
• Each pattern will have internal default values for all of the customization supported so that the user can turn the feature "on" and it will work without further customization

workload-config.yml

enterprise_container_registry:
    pattern_features:
      shared_kms_key:                                          # Feature block
        enabled: false                                         # Feature switch
      shared_container_registry:                               # Feature block
        enabled: true                                          # Feature switch
        registry_images:                                       # Feature customization
          - blox
          - awsh

Customizing Resource Names

When a pattern provisions one or more resources that support customization, the default names for those resources will be set by combining the resource prefix generated using the prefix_tokens values and the key used to configure that instance.

Default naming of resources

example

workload-config.yml

# Shared configuration
common_config:
  prefix_tokens:
    org: "HIO"
    lifecycle: "dev"

# Pattern configuration
enterprise_container_registry:
  registry-example-name:                    # <-- Resource name key
    pattern_features:
      shared_kms_key:                                          # Feature block
        enabled: false                                         # Feature switch
      shared_container_registry:                               # Feature block
        enabled: true                                          # Feature switch
        registry_images:                                       # Feature customization
          - blox
          - awsh

    image_tag_mutability: "MUTABLE"
    scan_on_push: "true"

For the above config, resources provisioned into eu-west-1 region, this would result in;

Regional Prefix	HIO-EW1-DEV
Global Prefix	HIO-GBL-DEV
Resource Name(s)	HIO-EW1-DEV-registry-example-name

Customizing resource names using tags

For resources that support tags, the Name tag can be used to overide the built-in logic used for what is presented as the resource "Name" in the AWS Console and other similar UI tools.

example

workload-config.yml

# Shared configuration
common_config:
  prefix_tokens:
    org: "HIO"
    lifecycle: "dev"

# Pattern configuration
enterprise_stateful_vm:
  VM01:                                   # <-- Resource name key
    instance_ami: "amazonlinux2"
    instance_type: "t2.micro"
  VM02:                                   # <-- Resource name key
    instance_ami: "amazonlinux2"
    instance_type: "t2.micro"
  VM03:                                   # <-- Resource name key
    instance_ami: "amazonlinux2"
    instance_type: "t2.micro"
    tags:
      Name: VERYSPECIAL01

For the above config, resources provisioned into eu-west-1 region, this would result in;

Regional Prefix	HIO-EW1-DEV
Global Prefix	HIO-GBL-DEV
Resource Name(s)	HIO-EW1-DEV-VM01, HIO-EW1-DEV-VM02, VERYSPECIAL01