thirstydeveloper

Terraform Skeleton Part 6: Protecting State

2021-02-25T19:00:00+00:00

Part 5 moved Terraform’s operational infrastructure into a CloudFormation stack so that Terragrunt no longer manages the state bucket, lock table, and log bucket. Doing so offers us the opportunity to protect these resources in ways not supported by Terragrunt. We will capitalize on that opportunity today.

We’ll cover adding a bucket policy to our CloudFormation template to restrict access to Terraform’s state, leveraging the backend role and Terraformer principal tag introduced in Part 4. I hope my example will ease your navigation of IAM policy syntax and save you the pain of decrypting unhelpful IAM error messages.

We’ll also add some basic protections to the log bucket that Terragrunt fails to implement out of the box.

Goals

Grant only authorized principals access to Terraform state
Protect the log bucket in a similar fashion to the state bucket
Remove unnecessary permissions from the backend role

If you prefer to jump to the end, the code implementing this post’s final result is available on branch release/1.5 on GitHub. Additionally, you can view the diffs from part 5, if that’s more your speed.

Access Roles

There are three types of authorized access to Terraform state we will cover today.

First is our backend role, which the Terragrunt configures Terraform to use for all state operations via the root.hcl from Part 4, reprinted here:

remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite"
  }
  config = {
    bucket   = "terraform-skeleton-state"
    region   = "us-east-1"
    encrypt  = true
    role_arn = "arn:aws:iam::${get_aws_account_id()}:role/terraform/TerraformBackend"

    key = "${dirname(local.relative_deployment_path)}/${local.stack}.tfstate"

    dynamodb_table            = "terraform-skeleton-state-locks"
    accesslogging_bucket_name = "terraform-skeleton-state-logs"
  }
}

Second are humans, who may wish to interact with the state for any number of reasons. One example is changing the name of an existing stack (which in our setup will change the name of the state file).

I like to differentiate between developer-level and administrative-level access to Terraform state, the primary difference being that only administrators should be allowed to delete state files permanently. Deleting a state file is not something to take lightly. If you delete one you still need by accident, and you have no backups, your only choice will be to reconstruct it by hand.

The policy we implement today will grant:

Full access to administrative IAM users, including the AWS account’s root user
Limited access to developer IAM users and the backend role
Deny access to anyone else

State Bucket Policy

We will use an S3 bucket policy to implement the above restrictions. An S3 bucket policy is an ideal choice for two reasons. First, it directly applies permissions to the resource we want to protect (the state bucket). Second, developers can likely create it themselves, unlike adding permissions to IAM users, something typically controlled by an enterprise team.

We’ll add the bucket policy as a resource to our init-admin-account.cf.yml CloudFormation template. We’ll break the policy down statement by statement.

The first statement requires TLS encryption for any requests accessing Terraform state. This policy Terragrunt creates for you; we preserve it here.

  TerraformStateBucketPolicy:
    Type: 'AWS::S3::BucketPolicy'
    DeletionPolicy: Retain
    UpdateReplacePolicy: Retain
    Properties:
      Bucket: !Ref TerraformStateBucket
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Sid: 'AllowTLSRequestsOnly'
            Principal: '*'
            Condition:
              Bool:
                'aws:SecureTransport': false
            Effect: Deny
            Action: '*'
            Resource:
              - !GetAtt "TerraformStateBucket.Arn"
              - !Sub
                - "${Bucket}/*"
                - Bucket: !GetAtt "TerraformStateBucket.Arn"

The second statement uses the aws:PrincipalType and aws:PrincipalTag condition keys to deny access to any IAM users lacking the Terraformer principal tag we introduced in Part 4:

          - Sid: DenyNonTerraformerUsers
            Principal: "*"
            Condition:
              StringEquals:
                aws:PrincipalType: User
              StringNotLike:
                'aws:PrincipalTag/Terraformer': '*'
            Effect: Deny
            Action: '*'
            Resource:
              - !GetAtt "TerraformStateBucket.Arn"
              - !Sub
                - "${Bucket}/*"
                - Bucket: !GetAtt "TerraformStateBucket.Arn"

The third statement begins differentiating between administrative and non-administrative access to the state. We do so again using the Terraformer tag, this time inspecting its value.

          - Sid: RestrictTerraformNonAdmins
            Principal: "*"
            Condition:
              StringEquals:
                aws:PrincipalType: User
              StringLike:
                'aws:PrincipalTag/Terraformer': '*'
              StringNotEquals:
                'aws:PrincipalTag/Terraformer': 'Admin'
            Effect: Deny
            NotAction:
              - 's3:List*'
              - 's3:Get*'
              - 's3:Describe*'
              - 's3:PutObject'
              - 's3:DeleteObject'
            Resource:
              - !GetAtt "TerraformStateBucket.Arn"
              - !Sub
                - "${Bucket}/*"
                - Bucket: !GetAtt "TerraformStateBucket.Arn"

If the IAM user has the Terraformer tag, but its value is not Admin, we grant non-administrative access to that user. We use IAM’s NotAction to whitelist the permitted actions.

Notably, non-administrative access permits s3:DeleteObject but not s3:DeleteObjectVersion. Since our state bucket is versioned (see Part 5), granting s3:DeleteObject is not inherently dangerous because all it does is add a delete marker to the object; you can always restore the version before the delete marker. Granting developers the ability to add delete markers aids state migration, so we do so here.

The fourth statement denies access to all IAM roles other than our backend role:

          - Sid: DenyNonBackendRoles
            Principal: "*"
            Condition:
              StringEquals:
                aws:PrincipalType: AssumedRole
              StringNotLike:
                aws:userId:
                  - !Sub
                    - "${TerraformBackendRoleId}:*"
                    - TerraformBackendRoleId: !GetAtt "TerraformBackendRole.RoleId"
            Effect: Deny
            Action: '*'
            Resource:
              - !GetAtt "TerraformStateBucket.Arn"
              - !Sub
                - "${Bucket}/*"
                - Bucket: !GetAtt "TerraformStateBucket.Arn"

I did not find the syntax intuitive for restricting access to a specific IAM role. Specifying the role ARN in the Principal property does not work. This AWS post explains why and demonstrates using the StringNotLike and aws:userId combination I use here.

The fifth statement grants the backend role access:

          - Sid: ResrictBackendRoleToReadWrite
            Principal: "*"
            Condition:
              StringEquals:
                aws:PrincipalType: AssumedRole
              StringLike:
                aws:userId:
                  - !Sub
                    - "${TerraformBackendRoleId}:*"
                    - TerraformBackendRoleId: !GetAtt "TerraformBackendRole.RoleId"
            Effect: Deny
            NotAction:
              - 's3:ListBucket'
              - 's3:GetBucketVersioning'
              - 's3:GetObject'
              - 's3:PutObject'
            Resource:
              - !GetAtt "TerraformStateBucket.Arn"
              - !Sub
                - "${Bucket}/*"
                - Bucket: !GetAtt "TerraformStateBucket.Arn"

And our final statement denies access to any other principal types (e.g., FederatedUsers), as we’re not considering those as part of this skeleton.

          - Sid: DenyAllOtherPrincipals
            Principal: "*"
            Condition:
              StringNotEquals:
                aws:PrincipalType:
                  - AssumedRole
                  - Account
                  - User
            Effect: Deny
            Action: '*'
            Resource:
              - !GetAtt "TerraformStateBucket.Arn"
              - !Sub
                - "${Bucket}/*"
                - Bucket: !GetAtt "TerraformStateBucket.Arn"

A summary of all changes is available by viewing the diffs from part 5.

There are three last items to note about this policy.

First, the bucket policy does not contain any permissions for users who have the Terraformer tag set to Admin. The lack of permissions means such users will have whatever access the IAM policy attached to their IAM user grants, presumably full access to S3.

Second, the bucket policy does not explicitly grant the AWS account’s root user access. AWS always allows the root user access to remove or modify bucket policies of buckets owned by that root user’s account, making it unnecessary to specify here.

Finally, none of the permissions in the policy grant either adding or modifying the bucket policy, which means that aside from the root user, only Admin Terraformers can do so, assuming those admins have the requisite permissions on their IAM user).

Applying the State Bucket Policy

We’re now ready to deploy the bucket policy.

First, change the Terraformer tag on your IAM user to have a value of Admin.

aws iam tag-user \
  --user-name ${IAM_USER} \
  --tags '{
    "Key": "Terraformer",
    "Value": "Admin"
  }'

Second, if Terragrunt created the state bucket for you, it may already have a bucket policy attached to it. Delete it using the following CLI command, replacing the bucket name as appropriate:

aws s3api delete-bucket-policy --bucket terraform-skeleton-state

Deploy the updated CloudFormation template using the init-admin Makefile target we added in Part 4.¹

make init-admin
aws cloudformation deploy \
		--template-file init/admin/init-admin-account.cf.yml \
		--stack-name tf-admin-init \
		--capabilities CAPABILITY_NAMED_IAM \
		--parameter-overrides \
			AdminAccountId=<omitted> \
			StateBucketName=terraform-skeleton-state \
			StateLogBucketName=terraform-skeleton-state-logs \
			LockTableName=terraform-skeleton-state-locks

Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - tf-admin-init
aws cloudformation update-termination-protection \
		--stack-name tf-admin-init \
		--enable-termination-protection
{
    "StackId": "arn:aws:cloudformation:us-east-1:<omitted>:stack/tf-admin-init/8704b070-5f61-11eb-9ff1-0eea077046db"
}

Let’s see if it works as expected.

Testing the State Bucket Policy

First, we can quickly verify the backend role has the access it requires by running terragrunt apply-all.

Verifying user-level access requires manipulating the Terraformer tag. Ideally, we’d have these tests automated and run as part of a CI pipeline. Perhaps we’ll get to that in another post.

Remove the Terraformer tag from your user altogether:

aws iam untag-user --user-name ${IAM_USER} --tag-keys Terraformer

and verify you can’t even list the state bucket now:

aws s3 ls s3://terraform-skeleton-state

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

Next, set the Terraformer tag on your IAM user to something other than Admin. For instance:

aws iam tag-user \
  --user-name ${IAM_USER} \
  --tags '{
    "Key": "Terraformer",
    "Value": "User"
  }'

Then verify you cannot update the state bucket’s policy:

aws s3api put-bucket-policy --bucket terraform-skeleton-state --policy ""

An error occurred (AccessDenied) when calling the PutBucketPolicy operation: Access Denied

or delete it:

aws s3api delete-bucket-policy --bucket terraform-skeleton-state

An error occurred (AccessDenied) when calling the DeleteBucketPolicy operation: Access Denied

Also, verify you can delete a state object:

aws s3 rm s3://terraform-skeleton-state/app/dev/test-stack.tfstate
delete: s3://terraform-skeleton-state/app/dev/test-stack.tfstate

but can’t delete an object version:

DELETE_MARKER_VERSION=$(aws s3api list-object-versions \
  --bucket terraform-skeleton-state \
  --prefix app/dev/test-stack.tfstate \
  --query 'DeleteMarkers[?IsLatest==`true`].VersionId' | jq -r '.[0]' \
)

aws s3api delete-object \
  --bucket terraform-skeleton-state \
  --key app/dev/test-stack.tfstate \
  --version-id ${DELETE_MARKER_VERSION}

An error occurred (AccessDenied) when calling the DeleteObject operation: Access Denied

The inability to remove the delete marker does introduce a hurdle users will have to clear to restore state files, but restoration is still possible. See the footnotes for more information.²

Change the Terraformer tag to Admin again:

aws iam tag-user \
  --user-name ${IAM_USER} \
  --tags '{
    "Key": "Terraformer",
    "Value": "Admin"
  }'

and delete the delete marker again, verifying it works this time.

aws s3api delete-object \
  --bucket terraform-skeleton-state \
  --key app/dev/test-stack.tfstate \
  --version-id ${DELETE_MARKER_VERSION}

{
    "DeleteMarker": true,
    "VersionId": "..."
}

That about covers it. We now have a bucket policy restricting access to Terraform’s state to only authorized principals.

Cleanup

With the bucket policy in place, let’s turn our attention to a couple of other hardening items we can tackle in the CloudFormation template: protecting the logs bucket and removing unnecessary permissions from the backend role.

Protecting the Logs Bucket

As discussed in Part 3, when Terragrunt creates the log bucket for you, it does not enable encryption or explicitly block public access. We can rectify both of those issues now that the bucket is under CloudFormation’s control.

Using our TerraformStateBucket resource as a template, add BucketEncryption and PublicAccessBlockConfiguration properties to the log bucket:

  TerraformStateLogBucket:
    Type: 'AWS::S3::Bucket'
    DeletionPolicy: Retain
    UpdateReplacePolicy: Retain
    Properties:
      BucketName: !Ref StateLogBucketName
      AccessControl: LogDeliveryWrite
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: aws:kms
      PublicAccessBlockConfiguration:
        BlockPublicAcls: True
        BlockPublicPolicy: True
        IgnorePublicAcls: True
        RestrictPublicBuckets: True

Run make init-admin to deploy.

Backend Role Permissions

When we created the backend role in Part 4, we granted it permissions to create S3 buckets and DynamoDB tables because Terragrunt managed our state bucket and lock table. We can remove those permissions now that CloudFormation deploys both:

diff --git a/init/admin/init-admin-account.cf.yml b/init/admin/init-admin-account.cf.yml
index b8dcda0..3aed8e5 100644
--- a/init/admin/init-admin-account.cf.yml
+++ b/init/admin/init-admin-account.cf.yml
@@ -122,27 +122,6 @@ Resources:
               - 'dynamodb:PutItem'
               - 'dynamodb:DeleteItem'
             Resource: !Sub "arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${LockTableName}"
-          - Sid: AllowStateBucketCreation
-            Effect: Allow
-            Action:
-              - 's3:GetBucketAcl'
-              - 's3:GetBucketLogging'
-              - 's3:CreateBucket'
-              - 's3:PutBucketPublicAccessBlock'
-              - 's3:PutBucketTagging'
-              - 's3:PutBucketPolicy'
-              - 's3:PutBucketVersioning'
-              - 's3:PutEncryptionConfiguration'
-              - 's3:PutBucketAcl'
-              - 's3:PutBucketLogging'
-            Resource:
-              - !Sub "arn:aws:s3:::${StateBucketName}"
-              - !Sub "arn:aws:s3:::${StateLogBucketName}"
-          - Sid: AllowLockTableCreation
-            Effect: Allow
-            Action:
-              - 'dynamodb:CreateTable'
-            Resource: !Sub "arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${LockTableName}"

Run a terragrunt apply-all to verify all’s well.

What’s Next?

In this post, we’ve significantly enhanced the protections surrounding the Terraform state. Looking back at these first six entries, we’ve covered a lot of ground, and I think it will be worthwhile to take a step back and summarize what we’ve done so far. After a recap, I’d like to start covering continuous integration with Terraform and Terragrunt. We’ll see where that takes us.

Footnotes

I hope that you don’t encounter any errors deploying the bucket policy. If you do, AWS is often not much help diagnosing what went wrong. Here are some pointers.

If make init-admin fails, it will likely say:

Failed to create/update the stack. Run the following command to fetch the
list of events leading up to the failure
aws cloudformation describe-stack-events --stack-name tf-admin-init

If you do so, you’ll get a JSON dump of events. Since you presumably failed when creating the bucket policy, look for an event with a ResourceStatus field set to CREATE_FAILED. Here’s an abbreviated example:

{  
    "StackName": "tf-admin-init",
    "ResourceStatus": "CREATE_FAILED",
    "ResourceStatusReason": "Invalid policy syntax. (Service: Amazon S3; Status Code: 400; Error Code: MalformedPolicy; Request ID: 9EB1BD50BCF1FCB7; S3 Extended Request ID: hHkqY/snGYyhc4paSxhBT1IzpmgoWjKvz5I/JYiYUKu3PLSn1CWuAceLU7QEckf/omDhF4ZdeGU=; Proxy: null)",
    "ResourceProperties": "{\"Bucket\":\"terraform-skeleton-state\"..."
}

If you have an Invalid policy syntax error, I recommend removing statements until the policy works, then adding them back in one-by-one until you find the problem. ↩

Since users cannot delete object versions, they cannot restore a state file by deleting its delete marker since it is itself an object version.

As discussed in the AWS docs here and here, users can still restore state files by copying a previous version to become the latest. Here’s an example of how:

Step 1: Find version to restore (e.g., using the LastModified field to limit the search area).

aws s3api list-object-versions \
  --bucket terraform-skeleton-state \
  --key app/dev/test-stack.tfstate \
  --query 'Versions[?contains(LastModified, `'"2021-02-20"'`)]' \
  | jq '.[] | { Key, VersionId, LastModified }'

{
  "Key": "app/stage/test-stack.tfstate",
  "VersionId": "lwMes1R1AfkEZ.lQ4U9d217yeU7rWbcj",
  "LastModified": "2021-02-20T13:03:50+00:00"
}

Step 2: Copy the desired object version to make it the new latest version, replacing ${VERSION_ID} as appropriate.

BUCKET=terraform-skeleton-state
PREFIX=app/dev/test-stack.tfstate
VERSION=TgfUNdVuoZKwSOHF1QeGO_nB8iZGzE3f

aws s3api copy-object \
  --copy-source "${BUCKET}/${PREFIX}?versionId=${VERSION}" \
  --key ${PREFIX} \
  --bucket ${BUCKET}

{  
    ...
    "CopyObjectResult": {
        "ETag": "\"...\"",
        "LastModified": "2021-02-20T13:36:47+00:00"
    }
}

↩

Terraform Skeleton Part 5: Remote State with CloudFormation

2021-02-17T12:20:00+00:00

Part 3 showed us Terraform requires some infrastructure itself to store remote state and discussed some limitations of using Terragrunt to manage the creation of that infrastructure. In part 4, we introduced more operational infrastructure for Terraform and began managing that infrastructure with CloudFormation.

Today, we’ll continue down the path of using CloudFormation to store operational Terraform infrastructure. Achieving a clean separation between the infrastructure Terraform needs to run and the infrastructure Terraform manages opens up additional possibilities for locking down the Terraform state.

We’ll make use of CloudFormation’s import feature to bring the state bucket and lock table Terragrunt created for us under CloudFormation’s control for a seamless transition of ownership.

Goals

Control all Terraform operational infrastructure with CloudFormation
Import existing operational infrastructure (buckets, tables) into CloudFormation

If you prefer to jump to the end, the code implementing this post’s final result is available on branch release/1.4 on GitHub. Additionally, you can view the diffs from part 4, if that’s more your speed.

Define Operational Infrastructure with CloudFormation

There are three items we need to import into CloudFormation:

The state S3 bucket
The log S3 bucket
The DynamoDB lock table

The first step is to define these resources in a CloudFormation template. We’ll add resource definitions to the init-admin-account.cf.yml template we created in part 4.

If you prefer, you can view the diffs on the CloudFormation template from part 4.

First up, the state bucket. Add an AWS::S3::Bucket resource to the CloudFormation template for our state bucket under the Resources block:

TerraformStateBucket:
  Type: 'AWS::S3::Bucket'
  DeletionPolicy: Retain
  UpdateReplacePolicy: Retain
  Properties:
    BucketName: !Ref StateBucketName
    BucketEncryption:
      ServerSideEncryptionConfiguration:
        - ServerSideEncryptionByDefault:
            SSEAlgorithm: aws:kms
    LoggingConfiguration:
      DestinationBucketName: !Ref StateLogBucketName
      LogFilePrefix: TFStateLogs/
    PublicAccessBlockConfiguration:
      BlockPublicAcls: True
      BlockPublicPolicy: True
      IgnorePublicAcls: True
      RestrictPublicBuckets: True
    VersioningConfiguration:
      Status: Enabled

The properties above match what Terragrunt used when it created the state bucket in part 3. Note that we’re also using the StateBucketName and StateLogBucketName parameters we added in part 4.

CloudFormation requires the DeletionPolicy attribute for any resources it will import, and it is a good safety measure in general.

UpdateReplacePolicy is also set. Although not required, it is needed to make our CloudFormation linter pre-commit hook happy.¹

Next, we’ll tackle the log bucket. Add the following to the Resources block:

TerraformStateLogBucket:
  Type: 'AWS::S3::Bucket'
  DeletionPolicy: Retain
  UpdateReplacePolicy: Retain
  Properties:
    BucketName: !Ref StateLogBucketName
    AccessControl: LogDeliveryWrite

Finally, we have the lock table:

TerraformStateLockTable:
  Type: 'AWS::DynamoDB::Table'
  DeletionPolicy: Retain
  UpdateReplacePolicy: Retain
  Properties:
    TableName: !Ref LockTableName
    AttributeDefinitions:
      - AttributeName: LockID
        AttributeType: S
    KeySchema:
      - AttributeName: LockID
        KeyType: HASH
    BillingMode: PAY_PER_REQUEST

If you’re starting fresh, and Terragrunt hasn’t already created the state bucket, log bucket, and lock table for you, you can skip the next section on importing and run the make init-admin target we created in part 3 to deploy your stack. Otherwise, continue on to import the resources Terragrunt created for you.

Import Operational Infrastructure into CloudFormation

While you can use the AWS management console to import the resources, I prefer to work from the command line, but the commands to do so aren’t straightforward. We’ll add some targets to our Makefile in an attempt to simplify.

For a clean import, we need the following capabilities in our Makefile:

Use CloudFormation Drift Detection to ensure our stack is up-to-date and our resource definitions match what Terragrunt created
Create a CloudFormation change set to import the resources and show us what CloudFormation intends to do
Execute the change set to import the resources

If you prefer to jump to the end, here’s the resulting Makefile, and here are the diffs from part 4.

Check for Drift

The first step in the import process is to verify that the stack we will be importing into has no drift, i.e., it has no other unapplied changes. The CloudFormation API uses separate calls to start a drift detection job and check the status of a drift detection job. We’ll add some helper functions to our Makefile to wrap these calls:

define wait_cfn_drift_detect_job
	@while [[ \
		"$$($(CFN_STATUS_DRIFT_DETECTION) $(1) | jq -r .DetectionStatus)" == \
		"DETECTION_IN_PROGRESS" \
	]]; do \
		echo "Detection in progress. Waiting 3 seconds..."; \
		sleep 3; \
	done
endef

define show_cfn_drift
	$(eval DRIFT_ID=$(shell $(CFN_START_DRIFT_DETECTION) $(1) \
		| jq -r .StackDriftDetectionId))
	$(call wait_cfn_drift_detect_job,${DRIFT_ID})
	@$(CFN_STATUS_DRIFT_DETECTION) $(DRIFT_ID) | jq '{ \
		DetectionStatus, \
		StackDriftStatus, \
		DriftedStackResourceCount \
	}'
endef

Next, add a make target to perform the drift detection:

.PHONY: check-init-admin-drift
check-init-admin-drift:
	$(call show_cfn_drift,${ADMIN_INIT_STACK_NAME})

Execute drift detection with:

➜ make check-init-admin-drift
Detection in progress. Waiting 3 seconds...
{
  "DetectionStatus": "DETECTION_COMPLETE",
  "StackDriftStatus": "IN_SYNC",
  "DriftedStackResourceCount": 0
}

If you get a StackDriftStatus other than IN_SYNC, adjust your CloudFormation template to resolve, and verify using the drift check. Once you’re IN_SYNC, create the import change set as follows.

Create Import Changeset

Creating the import change set requires a CloudFormation template and information about the resources to import, including:

The resource type
The logical name of the resource in your template
The unique identifier for that resource in AWS

For instance, our CloudFormation template defines the state bucket with:

Resources:
  TerraformStateLogBucket:
    Type: 'AWS::S3::Bucket'
    ...

which means the resource type is AWS::S3::Bucket, and the logical name is TerraformStateLogBucket.

The unique identifier depends on the resource type. For an S3 bucket, it’s the bucket name. For a DynamoDB table, it’s the table name.

Below is a make target for creating the import change set:

import-terragrunt-changeset.json:
	@aws cloudformation create-change-set \
		--stack-name ${ADMIN_INIT_STACK_NAME} \
		--change-set-name ${ADMIN_INIT_STACK_NAME}-import-terragrunt \
		--change-set-type IMPORT \
		--template-body file://init/admin/init-admin-account.cf.yml \
		--capabilities CAPABILITY_NAMED_IAM \
		--parameters \
			ParameterKey=AdminAccountId,UsePreviousValue=True \
			ParameterKey=StateBucketName,UsePreviousValue=True \
			ParameterKey=StateLogBucketName,UsePreviousValue=True \
			ParameterKey=LockTableName,UsePreviousValue=True \
		--resources-to-import "[ \
			{ \
				\"ResourceType\":\"AWS::S3::Bucket\", \
				\"LogicalResourceId\":\"TerraformStateBucket\", \
				\"ResourceIdentifier\": { \
					\"BucketName\": \"${STATE_BUCKET_NAME}\" \
				} \
			}, \
			{ \
				\"ResourceType\":\"AWS::S3::Bucket\", \
				\"LogicalResourceId\":\"TerraformStateLogBucket\", \
				\"ResourceIdentifier\": { \
					\"BucketName\": \"${STATE_LOG_BUCKET_NAME}\" \
				} \
			}, \
			{ \
			\"ResourceType\":\"AWS::DynamoDB::Table\", \
				\"LogicalResourceId\":\"TerraformStateLockTable\", \
				\"ResourceIdentifier\": { \
					\"TableName\": \"${LOCK_TABLE_NAME}\" \
				} \
			} \
		]" | tee import-terragrunt-changeset.json

CloudFormation assigns each change set a unique ID needed for describing, executing, or discarding the change set with subsequent API calls. The import-terragrunt-changeset.json stores the change set identifier in a JSON file for our other make targets to consume.

Next, we want a target for describing the created change set so we can see what modifications it will make:

.PHONY: prepare-cfn-import-terragrunt
prepare-cfn-import-terragrunt: import-terragrunt-changeset.json
	$(eval CHANGE_SET_ID=$(shell jq -r .Id import-terragrunt-changeset.json))
	aws cloudformation wait change-set-create-complete \
		--change-set-name ${CHANGE_SET_ID} \
		--stack-name ${ADMIN_INIT_STACK_NAME}
	@aws cloudformation describe-change-set \
		--change-set-name ${CHANGE_SET_ID} \
		--stack-name ${ADMIN_INIT_STACK_NAME} \
		| jq '{ Changes, Status, StatusReason }'

The prepare-cfn-import-terragrunt target depends on import-terragrunt-changeset.json to create the change set and communicate its unique id, then describes it once the change set finishes creating.

At this point, we’ll add another target for discarding the change set, in case we don’t like what we see with prepare-cfn-import-terragrunt:

.PHONY: discard-cfn-import-terragrunt
discard-cfn-import-terragrunt: import-terragrunt-changeset.json
	$(eval CHANGE_SET_ID=$(shell jq -r .Id import-terragrunt-changeset.json))
	aws cloudformation delete-change-set \
		--change-set-name ${CHANGE_SET_ID} \
		--stack-name ${ADMIN_INIT_STACK_NAME}
	@rm import-terragrunt-changeset.json

Let’s add a conventional clean target too:

.PHONY: clean
clean:
	rm import-terragrunt-changeset.json

Now, let’s use our targets to create and describe the change set:

➜ make prepare-cfn-import-terragrunt
{
    "Id": "arn:aws:cloudformation:us-east-1:<omitted>:changeSet/tf-admin-init-import-terragrunt/179b1efd-2961-48af-95c2-6f45b96a9925",
    "StackId": "arn:aws:cloudformation:us-east-1:<omitted>:stack/tf-admin-init/8704b070-5f61-11eb-9ff1-0eea077046db"
}
aws cloudformation wait change-set-create-complete \
		--change-set-name arn:aws:cloudformation:us-east-1:<omitted>:changeSet/tf-admin-init-import-terragrunt/179b1efd-2961-48af-95c2-6f45b96a9925 \
		--stack-name tf-admin-init
{
  "Changes": [
    {
      "Type": "Resource",
      "ResourceChange": {
        "Action": "Import",
        "LogicalResourceId": "TerraformStateBucket",
        "PhysicalResourceId": "terraform-skeleton-state",
        "ResourceType": "AWS::S3::Bucket",
        "Scope": [],
        "Details": []
      }
    },
    {
      "Type": "Resource",
      "ResourceChange": {
        "Action": "Import",
        "LogicalResourceId": "TerraformStateLockTable",
        "PhysicalResourceId": "terraform-skeleton-state-locks",
        "ResourceType": "AWS::DynamoDB::Table",
        "Scope": [],
        "Details": []
      }
    },
    {
      "Type": "Resource",
      "ResourceChange": {
        "Action": "Import",
        "LogicalResourceId": "TerraformStateLogBucket",
        "PhysicalResourceId": "terraform-skeleton-state-logs",
        "ResourceType": "AWS::S3::Bucket",
        "Scope": [],
        "Details": []
      }
    }
  ],
  "Status": "CREATE_COMPLETE",
  "StatusReason": null
}

Our describe command should show three Import actions occurring in the change list with no other changes. If you see any other actions, it means your template contains unapplied changes, and you’ll want to address those first.

Execute Import

Having created the import change set and verified the actions it staged, we’ll now add a target for executing it:

.PHONY: cfn-import-terragrunt
cfn-import-terragrunt: import-terragrunt-changeset.json
	$(eval CHANGE_SET_ID=$(shell jq -r .Id import-terragrunt-changeset.json))
	aws cloudformation wait change-set-create-complete \
		--change-set-name ${CHANGE_SET_ID} \
		--stack-name ${ADMIN_INIT_STACK_NAME}
	aws cloudformation execute-change-set \
		--change-set-name ${CHANGE_SET_ID} \
		--stack-name ${ADMIN_INIT_STACK_NAME}
	@rm import-terragrunt-changeset.json
	aws cloudformation wait stack-import-complete \
		--stack-name ${ADMIN_INIT_STACK_NAME}
	$(call show_cfn_drift,${ADMIN_INIT_STACK_NAME})

Execute the import with:

➜ make cfn-import-terragrunt
aws cloudformation wait change-set-create-complete \
		--change-set-name arn:aws:cloudformation:us-east-1:<omitted>:changeSet/tf-admin-init-import-terragrunt/179b1efd-2961-48af-95c2-6f45b96a9925 \
		--stack-name tf-admin-init
aws cloudformation execute-change-set \
		--change-set-name arn:aws:cloudformation:us-east-1:<omitted>:changeSet/tf-admin-init-import-terragrunt/179b1efd-2961-48af-95c2-6f45b96a9925 \
		--stack-name tf-admin-init
aws cloudformation wait stack-import-complete \
		--stack-name tf-admin-init
{
  "DetectionStatus": "DETECTION_COMPLETE",
  "StackDriftStatus": "IN_SYNC",
  "DriftedStackResourceCount": 0
}

At the end of the cfn-import-terragrunt target, we call our show_cfn_drift helper function to verify that the properties of the resources we imported match what we specified in our template. You should see StackDriftStatus is IN_SYNC, which means we’ve successfully imported the resources and the definitions for those resources in our template match reality.²

What’s Next

We’ve now fully separated the creation of operational infrastructure required to run Terraform from the infrastructure Terraform manages. Creating the operational infrastructure with CloudFormation has numerous benefits. We can now harden our state bucket, log bucket, and lock table in ways that were not available to us with Terragrunt managing their creation. We’ll tackle such hardening in upcoming posts.

Footnotes

If you omit UpdateReplacePolicy, the linter will report something like:

W3011 Both UpdateReplacePolicy and DeletionPolicy are needed to protect Resources/TerraformStateBucket from deletion

↩
If you see that there is drift after executing the import change set, the resources you imported have a different configuration than what you specified in your template. For instance, maybe your template specified a bucket is KMS encrypted when in reality, the bucket is AES-256 encrypted. Drift detection will report such differences. To resolve the drift:
1. Decide whether the drift is appropriate and should be retained
2. Modify the template to match any appropriate drift
3. Create a change set on the stack with the updated template
4. Verify the change set reports it will discard the inappropriate drift (or have no changes if there was no inappropriate drift)
5. Execute the change set
↩

Terraform Skeleton Part 4: Backend Role

2021-02-10T22:30:00+00:00

The previous entry enhanced the terraform skeleton with remote state storage using AWS S3 and DynamoDB. Access to the state was granted based on whatever AWS credentials were configured in the shell at the time terraform was executed.

Terraform state is a highly sensitive resource. It is likely to contain lots of sensitive information including passwords and access tokens. Additionally, recovering from a lost state file means either recreating all the infrastructure that was in it, or spending some quality time running terraform import commands for resources that support it, and hand modifying state files for those that do not.

On projects I’ve supported, developers have had near-administrative level permissions on their cloud accounts. Typically, these are the credentials they will use when running terraform, which also requires near-administrative level permissions to manage the wide variety of cloud resources a project is likely to need. Unless instructed otherwise, terraform will use those administrative credentials for accessing the state, and that opens up the potential for disaster (e.g., a developer mistakenly deleting a state bucket).

In this post, we’ll create a dedicated IAM role for backend access so that every developer on our team working with terraform will access state with the same permissions, and those permissions are scoped to just what is needed to read and write state.

Using a dedicated backend role for state access will help avoid “works on my machine problems” and allow us, in future entries, to lock down access to terraform state due to its sensitive nature.

Goals

The skeleton uses a dedicated IAM role for accessing terraform state such that everyone on the team accesses state with the same least-privilege permissions
The IAM role and its permissions are controlled using infrastructure-as-code

If you prefer to jump to the end, the code implementing the final result is available on GitHub.

Setup

You will need:

An AWS account to serve as your terraform “admin” account, holding the state resources
The AWS CLI installed on your workstation
Credentials for that account configured in the terminal for running aws CLI commands

The Administrative Account

Before we create our backend IAM role, it is worth discussing cloud account organization. Chances are strong that you’re going to have more than one account.¹ Typically, one per application environment is recommended:

With multiple accounts, the question of where we put our terraform state arises. We could put the state for each tier-environment in its corresponding account (e.g., app-dev state in the app-dev account) but I don’t like doing that for several reasons:

It assumes our tiers line up one-to-one with accounts, which may not always be true
I prefer to restrict administrative access to terraform state to a subset of those who have access to environment accounts.

Instead, I prefer to use a separate admin account to hold all the state, an approach recommended in the documentation for the S3 backend:

With this approach, our admin account contains:

S3 bucket(s) for terraform state
S3 bucket(s) for state logs²
DynamoDB table(s) for terraform locks
IAM roles for accessing the above

The first three were created in the previous entry. Now, we’ll create our backend role.

The Backend Role

We have a bit of a chicken and egg problem with the backend role. We want to control all infrastructure as code, the backend role included, but we can’t use the backend role for terraform until it is created. A similar problem presents itself if you want to self-manage the creation of your S3 state bucket and DynamoDB lock table.

While it’s possible to use terraform to create these resources by having a stack that either stores state locally initially and/or uses user credentials instead of the backend role, I tend not to do that. I prefer to separate the infrastructure needed for terraform to run from the infrastructure terraform creates, and manage the former using CloudFormation. A simple CloudFormation stack is more than capable of managing the few resources needed to bootstrap an AWS account to serve as our admin account and means we offload the state management of that bootstrap infrastructure to AWS’s CloudFormation tool.

Preparation

Before adding CloudFormation templates to our infra repository, I like to add a pre-commit hook that validates those templates. It’s easy to make syntax errors with CloudFormation; using a validation hook helps to shorten the feedback loop. I do the following:

Store bootstrap CloudFormation templates under init/ in the infra repo
Use a file extension of .cf.yml to differentiate CloudFormation templates from regular YAML files
Use the cfn-python-lint pre-commit hook to validate the CloudFormation templates

Let’s add the hook to our .pre-commit-config.yaml:

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index fa080cf..27217e9 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -24,3 +25,9 @@ repos:
   rev: v0.1.10
   hooks:
     - id: terragrunt-hclfmt
+
+- repo: https://github.com/aws-cloudformation/cfn-python-lint
+  rev: v0.44.5
+  hooks:
+  -   id: cfn-python-lint
+      files: init/.*\.cf\.(yml|yaml)$

This configures the hook to scan all templates under the init/ directory. We also need to tell our existing check-yaml hook to ignore the CloudFormation templates because it will throw errors on CloudFormation’s interpolation syntax. We can do that with:

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index fa080cf..27217e9 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -8,6 +8,7 @@ repos:
       args: [ --markdown-linebreak-ext=* ]
     - id: check-yaml
       args: [ --allow-multiple-documents ]
+      exclude: .*\.cf\.(yml|yaml)$
     - id: check-json
     - id: check-merge-conflict
     - id: detect-aws-credentials

Finally, run pre-commit install and pre-commit run -a to make sure all hooks are currently passing.

Admin Account CloudFormation Template

Create a new CloudFormation template at init/admin/init-admin-account.cf.yml. You’ll need parameters for the admin account ID, state bucket, log bucket, and lock table names. I like to use CloudFormation’s metadata property to group the parameters sensibly, should you use the AWS Management Console to deploy the stack:

---
AWSTemplateFormatVersion: '2010-09-09'
Description: Initialize terraform admin account
Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      - Label:
          default: Admin Account Config
        Parameters:
          - AdminAccountId
      - Label:
          default: Terraform State Resources
        Parameters:
          - StateBucketName
          - StateLogBucketName
          - LockTableName
Parameters:
  AdminAccountId:
    Type: String
    Description: Account ID of the admin account to contain the state
  StateBucketName:
    Type: String
    Description: Name of the S3 bucket for terraform state
  StateLogBucketName:
    Type: String
    Description: Name of the S3 bucket for terraform state logs
  LockTableName:
    Type: String
    Description: Name of the terraform DynamoDB lock table

Next, add a CloudFormation rule to restrict deployment of the stack to only the specified account. This is to help prevent someone from deploying the stack to the wrong account.

Rules:
  EnsureDeployingToCorrectAccount:
    Assertions:
      - Assert:
        'Fn::Equals':
          - !Ref AWS::AccountId
          - !Ref AdminAccountId
        AssertDescription: 'Stack can only be deployed into the specified AdminAccountId'

Next, add a resource to create an IAM managed policy for read/write state access:

Resources:
  TerraformStateReadWritePolicy:
    Type: 'AWS::IAM::ManagedPolicy'
    Properties:
      ManagedPolicyName: TerraformStateReadWrite
      Path: /terraform/
      Description: Read/write access to terraform state
      PolicyDocument:
        Version: 2012-10-17
        # Permissions are based on:
        # https://www.terraform.io/docs/backends/types/s3.html#example-configuration
        # https://github.com/gruntwork-io/terragrunt/issues/919
        Statement:
          - Sid: AllowStateBucketList
            Effect: Allow
            Action:
              - 's3:ListBucket'
              - 's3:GetBucketVersioning'
            Resource: !Sub "arn:aws:s3:::${StateBucketName}"
          - Sid: AllowStateReadWrite
            Effect: Allow
            Action:
              - 's3:GetObject'
              - 's3:PutObject'
            Resource: !Sub "arn:aws:s3:::${StateBucketName}/*"
          - Sid: AllowStateLockReadWrite
            Effect: Allow
            Action:
              - 'dynamodb:DescribeTable'
              - 'dynamodb:GetItem'
              - 'dynamodb:PutItem'
              - 'dynamodb:DeleteItem'
            Resource: !Sub "arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${LockTableName}"
          - Sid: AllowStateBucketCreation
            Effect: Allow
            Action:
              - 's3:GetBucketAcl'
              - 's3:GetBucketLogging'
              - 's3:CreateBucket'
              - 's3:PutBucketPublicAccessBlock'
              - 's3:PutBucketTagging'
              - 's3:PutBucketPolicy'
              - 's3:PutBucketVersioning'
              - 's3:PutEncryptionConfiguration'
              - 's3:PutBucketAcl'
              - 's3:PutBucketLogging'
            Resource:
              - !Sub "arn:aws:s3:::${StateBucketName}"
              - !Sub "arn:aws:s3:::${StateLogBucketName}"
          - Sid: AllowLockTableCreation
            Effect: Allow
            Action:
              - 'dynamodb:CreateTable'
            Resource: !Sub "arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${LockTableName}"

This policy grants everything needed by the S3 backend to manage the state. It’s worth noting that terragrunt requires additional permissions beyond what terraform specifies.

In particular, this policy does not grant delete access to the terraform state. As already discussed, losing your state file can create a tremendous amount of pain. Using a versioned S3 bucket helps, but you can still get yourself into trouble if you delete the bucket itself. Since the backend role does not require delete access, it does not get it.

Furthermore, the policy is restricted to creating only the specified bucket and lock table names. Given that terragrunt will auto-create a bucket and lock table if they do not exist, this restriction helps avoid unintended auto-creations due to developer errors.

Next, we’ll create the backend role, TerraformBackend, and attach the policy to it:

  TerraformBackendRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              AWS: !Ref AWS::AccountId
            Action:
              - 'sts:AssumeRole'
            Condition:
              StringEquals:
                aws:PrincipalType: User
              StringLike:
                'aws:PrincipalTag/Terraformer': '*'
      RoleName: TerraformBackend
      Path: /terraform/
      ManagedPolicyArns:
        - !Ref TerraformStateReadWritePolicy

In the AssumeRolePolicyDocument, I’m specifying who can assume the backend role. There are several ways to go about this, depending on what type of principals are doing the assuming. You could use IAM groups if the principals are IAM users, or SAML context keys if dealing with federated users. I’m going to use an attribute-based access control (ABAC) approach that will grant access if the principal has a tag of Terraformer. I’m also going to restrict access down to IAM users since that’s what I’m working with. The same approach should work for other principal types as well.

Finally, let’s add a Makefile to script deployment of our CloudFormation stack:

ADMIN_INIT_STACK_NAME := tf-admin-init
STATE_BUCKET_NAME := terraform-skeleton-state
STATE_LOG_BUCKET_NAME := terraform-skeleton-state-logs
LOCK_TABLE_NAME := terraform-skeleton-state-locks

# Use a known profile to ensure account ID is correct
ADMIN_ACCOUNT_ID := $(shell \
	aws --profile tf-admin-account sts get-caller-identity | jq -r .Account \
)

BACKEND_ROLE_PATH := terraform/TerraformBackend
BACKEND_ROLE_ARN := arn:aws:iam::${ADMIN_ACCOUNT_ID}:role/${BACKEND_ROLE_PATH}

DEPLOYMENT_DIRS := $(shell find deployments -name terragrunt.hcl \
	-not -path */.terragrunt-cache/* -exec dirname {} \; \
)

.PHONY: init-admin
init-admin:
	aws cloudformation deploy \
		--template-file init/admin/init-admin-account.cf.yml \
		--stack-name ${ADMIN_INIT_STACK_NAME} \
		--capabilities CAPABILITY_NAMED_IAM \
		--parameter-overrides \
			AdminAccountId=${ADMIN_ACCOUNT_ID} \
			StateBucketName=${STATE_BUCKET_NAME} \
			StateLogBucketName=${STATE_LOG_BUCKET_NAME} \
			LockTableName=${LOCK_TABLE_NAME}
	aws cloudformation update-termination-protection \
		--stack-name ${ADMIN_INIT_STACK_NAME} \
		--enable-termination-protection

.PHONY: test-backend-assume
test-backend-assume:
	aws sts assume-role \
		--role-arn ${BACKEND_ROLE_ARN} \
		--role-session-name $(shell whoami)

.PHONY: init-all
init-all:
	for d in ${DEPLOYMENT_DIRS}; do \
		pushd $$d; \
		terragrunt init; \
		popd; \
	done

The Makefile:

Uses the aws cloudformation deploy command, which will both create the stack if it does not exist and update it if it does.
Turns on termination protection as a basic safety measure on our stack.
Provides a test command to verify our ability to assume the backend role.
Provides an init-all command, which will be needed when we change out deployments to use the new backend role.

It’s time to deploy the backend role and use it. To do so:

Open a terminal and ensure you have AWS credentials to deploy the stack loaded.
Run make init-admin to deploy the CloudFormation stack.
Add the Terraformer tag to the appropriate IAM users.³
Ensure you can assume the new backend role with make test-backend-assume.

Next, we need to tell terraform to use the backend role for remote state access. Do so by updating the remote_state block inside root.hcl to include the role_arn property:

diff --git a/deployments/root.hcl b/deployments/root.hcl
index eed54e8..33091af 100644
--- a/deployments/root.hcl
+++ b/deployments/root.hcl
@@ -46,9 +46,10 @@ remote_state {
     if_exists = "overwrite"
   }
   config = {
     bucket   = "terraform-skeleton-state"
     region   = "us-east-1"
     encrypt  = true
+    role_arn = "arn:aws:iam::YOUR_ADMIN_ACCOUNT_ID:role/terraform/TerraformBackend"

To finish switching the deployments over to the new backend configuration we must re-initialize them. Do so with make init-all.

Finally, run terragrunt apply-all from the repository root. Everything should work.

What’s Next?

Although this was a fair amount of work, it is important. We now have a single role that gets used for all state manipulation by terraform. That role can be assumed by any of the developers working with terraform and, in future entries, can also be assumed by CI/CD jobs such that all terraform operations use a consistent set of permissions.

We also began creating infrastructure required to run terraform itself outside of terraform, relying on CloudFormation instead. Upcoming entries will continue this trend, which, along with the backend role we’ve created, will allow us to lock down access to our state bucket.

Footnotes

I’m using the AWS term account here because out of the cloud providers, my teams use AWS the most heavily. Equivalents include Google Cloud Platform (GCP) projects and Microsoft Azure resource groups. ↩
Another option is to place the logs bucket in a dedicated information security account. For more on this, I recommend videos from re:invent covering multi-account architectures. Here’s one from 2019. ↩
Although I don’t talk about it here, you would want to have tight control over who can add the Terraformer tag. Similarly, you need tight control over the ability to attach terraform IAM policies. The control mechanisms to use depend on how your organization manages user access to cloud accounts and is beyond the scope of this post. ↩

Terraform Skeleton Part 3: AWS Backend

2021-01-28T11:00:00+00:00

Terraform uses state files to track the resources it creates back to resource definitions in your *.tf files. Each deployment has its own state. State is stored according to the backend configured for the deployment. Terraform uses a local backend for storing state on the local filesystem by default, which is what we’ve been using for part 1 and part 2 of the terraform skeleton series.¹ This works fine for a simple demonstration but is insufficient for production use because:

You can only run terraform commands on your deployments from that one machine
There are no built-in backups on your state files; if they’re lost, you’re going to have a bad day
There’s no built-in versioning of your state files, which can be handy for recovering from more advanced terraform operations gone wrong

While you can commit the state files into your git repository for achieving distribution across team members, backups, and versioning, that isn’t a good idea because:

State files can include sensitive information you might not want in version control
If multiple team members make changes affecting the same state file, you’ll wind up with a mess of multiple incomplete state files needing to be merged

Remote state storage is built in to terraform and solves these problems, making it easy for teams to collaborate on the same deployments. Terragrunt provides further enhancements that make working with remote state even easier. Today, we’ll add remote state to the skeleton using terragrunt, S3, and DynamoDB. This will give us a simple foundation to build on.

Goals

Our skeleton stores state remotely such that multiple teammates can run terraform commands
State files are stored in a directory structure matching our deployments
Locks are in place around state manipulation to avoid corruption due to concurrent terraform command executions
State is stored securely, with encryption, versioning, and logging enabled
Remote state configuration is DRY - defined once and reused across deployments

If you prefer to jump to the end, the code implementing the final result is available on GitHub.

Setup

You will need:

An AWS account
Credentials for that account configured in the terminal used for running terraform and terragrunt commands

The S3 Backend

Remote state can be accomplished in many different ways with terraform. The teams I support tend to have an Amazon Web Services footprint already, and therefore I typically use terraform’s S3 backend.

The S3 backend stores your state files in S3 and retrieves them for stateful terraform commands. This meets the distribution, versioning, and encryption requirements we require. To avoid corruption from concurrent terraform commands, the S3 backend uses a DynamoDB table to manage lock files. Stateful terraform commands first obtain a lock from the DynamoDB table, effectively single-threading commands operating on the same state file.

We’ll focus on using the S3 backend today.

Backend Configuration with Terraform

Terraform backends are configured using a block in the stack’s *.tf files:

terraform {
  backend "s3" {
    bucket = "terraform-skeleton-state"
    key    = "something/unique/to/the/stack"
    region = "us-east-1"
  }
}

This inline configuration is less than ideal. First, you have to configure the backend for every stack definition (i.e., in every directory under our modules/stacks), leading to a lot of duplication. Second, you cannot use interpolation in the backend block to configure it using variables, locals, or data source attributes. A lack of interpolation can be problematic if you want to use different buckets for different environments for example, since we have multiple deployments (e.g., deployments/app/dev/test-stack, deployments/app/test/test-stack) sharing the same modules/stack/app/test-stack files.

Terraform provides several ways to work around these limitations through partial configuration. This allows you to omit properties from the backend block and instead provide them through an out-of-band mechanism such as a configuration file, a command-line option, or interactively on the command-line. With partial configuration, you could, for example, omit the bucket property in the configuration above, and instead pass it through one of the mentioned mechanisms to achieve different buckets for different environments.

Partial configuration, although a step in the right direction, still requires you to figure out how you’re going to manage the backend configuration properties. If using configuration files, where will those be stored and distributed to the team? How will you ensure every terraform command receives the necessary command-line properties with the correct values? This more or less forces you to use a build tool for your terraform commands (e.g., a makefile) to achieve consistency/reproducibility. While a build tool isn’t necessarily a bad thing, it strikes me as unusual to require it for remote state, a seemingly basic and fundamental capability.

Fortunately, terragrunt gives us a way to make remote state usage easy for anyone on the team.

Backend Configuration with Terragrunt

With terragrunt, you can configure your backend within a terragrunt.hcl, or in our case root.hcl. Terragrunt then generates the necessary terraform backend configuration based on what you specify. Configuring remote state inside root.hcl means every deployment will receive the backend configuration by way of terragrunt’s include mechanism. This removes all duplication of remote state configuration from the stacks.

Additionally, the remote state block supports interpolation, allowing each deployment to share common parts of the remote state configuration while having other parts be unique. For instance, every deployment can share the same bucket for their state files but use a different prefix within that bucket.

Using terragrunt’s remote state configuration is well documented by Gruntwork. Here’s how we can add it to our skeleton.

Implementation

Add a remote_state block to our root.hcl to generate an s3 backend configuration:

--- a/deployments/root.hcl
+++ b/deployments/root.hcl
@@ -39,6 +40,24 @@ locals {
 # environment variables
 inputs = local.merged_config

+remote_state {
+  backend = "s3"
+  generate = {
+    path      = "backend.tf"
+    if_exists = "overwrite"
+  }
+  config = {
+    bucket  = "terraform-skeleton-state"
+    region  = "us-east-1"
+    encrypt = true
+
+    key = "${dirname(local.relative_deployment_path)}/${local.stack}.tfstate"
+
+    dynamodb_table            = "terraform-skeleton-state-locks"
+    accesslogging_bucket_name = "terraform-skeleton-state-logs"
+  }
+}
+

This approach uses:

A single bucket, terraform-skeleton-state, for all deployment state files
A key/prefix unique to each deployment based on the relative path from the root.hcl file
A corresponding DynamoDB lock table, terraform-skeleton-state-locks
A logging bucket, terraform-skeleton-state-logs, for logging all S3 access requests to the state bucket

How This Works

Terragrunt executes terraform commands from a .terragrunt-cache directory. Before executing terraform, terragrunt populates the cache with:

Any files in the current deployment directory (location of your terragrunt.hcl file)
The stack files in the terraform.source directory specified in our root.hcl
Files from generate blocks defined in the HCL files

The last step above is what translates the generate block defined in root.hcl to a backend.tf file for each stack containing the appropriate terraform backend configuration.²

Bucket/Table Initialization

By using the remote_state block in our HCL files, we allow terragrunt to manage the creation of the S3 bucket and DynamoDB table for us.

Terragrunt will:

Automatically create both buckets and/or lock table for you if it does not exist
Configure the state bucket (but not the logs bucket) with
- Versioning and encryption enabled
- Public access disabled
- Enforcing TLS-only access
Enable encryption on the lock table

Activating Remote State

Switching from local state to remote state requires running terragrunt init on each deployment.

The first init will prompt you to create the buckets and/or dynamo table if they don’t exist:

Initializing remote state for the s3 backend
Remote state S3 bucket terraform-skeleton-state does not
exist or you don't have permissions to access it.
Would you like Terragrunt to create it? (y/n)

Every init will then prompt you to copy your existing local state to the new remote backend:

[terragrunt] 2020/12/13 10:04:56 Initializing remote state for the s3 backend
[terragrunt] 2020/12/13 10:04:57 Running command: terraform init

Initializing the backend...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value:

Finally, although this shouldn’t be necessary, I’ve had to rm -rf the .terragrunt-cache directory inside each deployment for terragrunt commands to work after the migration.

After activating remote state for all deployments, we can list our state bucket and see the state files nicely organized:

➜ aws s3 ls --recursive terraform-skeleton-state
2020-12-12 07:38:39       1154 app/dev/test-stack.tfstate
2020-12-13 10:22:49       1148 app/prod/test-stack.tfstate
2020-12-13 10:22:34       1154 app/stage/test-stack.tfstate

Limitations

Using terragrunt’s remote_state block has several advantages:

It’s easy
It includes sensible security defaults for the state bucket and lock table

While it is good enough for today’s skeleton, there are some limitations of the terragrunt approach worth covering:

Terragrunt lacks security defaults on the log bucket

If terragrunt creates the log bucket, it will not have encryption enabled and it will not have public access explicitly blocked. For this reason, you should strongly consider self-management of the log bucket. I tend to do this with a CloudFormation stack.
Disabling auto-creation of the state bucket and lock table is broken³

For a single team owning all infrastructure auto-creation is likely safe enough. Once I start distributing ownership of pieces of infrastructure to different teams, I want to control where state files are stored as much as possible, which means avoiding a situation where a team accidentally creates a new state bucket or lock table. Consistent state file storage makes it easier to audit infrastructure for compliance with team and organization standards using tools like terraform-compliance. Easier auditing in turn makes it more palatable to push down infrastructure ownership responsibilities.
Terragrunt doesn’t offer full control over the credentials used to access the terraform state

The remote_state block has a few fields for specifying a role_arn or AWS profile to use for remote state access, but you can’t control things like IAM session tags, transitive tag usage, or assume role policies.
Terragrunt doesn’t offer full control over all fields on the buckets and table

While terragrunt applies sensible security defaults, you can’t control everything using its remote_state block. For example, you’ll need to self-manage if you need to specify specific KMS keys for encryption of the bucket or table. Similarly, if you want to set up replication of your terraform state bucket, self-management is the way to go.

What’s next?

The skeleton now supports multiple developers working on the infrastructure as a team, sharing a state file stored in S3 with contention resolved through a DynamoDB lock table. State is encrypted and versioned, and access to it is logged. Remote state is configured once, in the root.hcl, and reused across our stacks.

Thus far, terragrunt has been running with whatever AWS credentials were configured in the shell at the time of execution.

This isn’t ideal in a team setting because:

It could lead to “works on my machine problems” if developers have different configurations
It encourages running terraform with administrative-level permissions all the time for all developers working with it
Terraform state is sensitive and should be protected from modification except by a well-defined set of roles

The next entry in the terraform skeleton series will take a first step towards addressing these issues.

Footnotes

Terragrunt stores local state in a terraform.tfstate file located underneath the .terragrunt-cache directory it creates within our deployment directory. ↩
The terragrunt remote state approach we cover here uses the newer generate option, which was added in terragrunt v0.22.0. Before that version, terragrunt wouldn’t generate a backend.tf file but would instead pass CLI arguments to the terraform command, making use of terraform’s partial configuration implementation. The generate approach has several advantages over the old method. The most notable is that previously, you had to include an empty backend configuration block in every stack, whereas with the generate approach you do not. ↩
Terragrunt has a disable_init attribute on the remote_state block, which will prevent auto-creation but, as described by this open issue, also completely disables backend initialization. ↩

Terraform Skeleton Part 2: Variables

2021-01-23T10:53:09+00:00

In part 1 of the terraform skeleton series, we set up a terraform repository that allows the team to apply infrastructure at any level: from individual stacks to entire environments. We build on that foundation in this post, adding a variable hierarchy that similarly allows definition and overriding of variables at each level of the infrastructure.

Goals

As a refresher, our infrastructure is organized by:

Tier
Environment
Layer (Optional)
Stack

A deployment is an instantiation of a stack for a tier-environment.

Today’s goals are to:

Allow defining variable values in files at each level of the infrastructure
Have lower-level variable definitions override higher levels

If you prefer to jump to the end, the code implementing the final result is available on GitHub.

Setup

Let’s add to our test-stack some variables to be defined at each level of deployments/:

# modules/stacks/app/test-stack/variables.tf
variable "global_var" {
  default = "unset"
}

variable "tier_var" {
  default = "unset"
}

variable "env_var" {
  default = "unset"
}

variable "layer_var" {
  default = "unset"
}

variable "stack_var" {
  default = "unset"
}

Let’s also add outputs to the stack so we can see what the variables are set to:

# modules/stacks/app/test-stack/outputs.tf
output "pet" {
  value = random_pet.pet.id
}

output "global_var" {
  value = var.global_var
}

output "tier_var" {
  value = var.tier_var
}

output "env_var" {
  value = var.env_var
}

output "layer_var" {
  value = var.layer_var
}

output "stack_var" {
  value = var.stack_var
}

Next, we’ll cover implementing the variable hierarchy in two ways. The first works for terraform pre-0.12. The second slightly modifies our first approach to work for 0.12+.

Variables via tfvars (pre-0.12)

tfvars files are a standard way of defining variable values for terraform. Terragrunt allows you to define required_var_files and optional_var_files within the terraform block of your terragrunt.hcl, as covered by this Gruntwork post. Terragrunt then passes these files to terraform using its -var-file option. Terraform loads the files and sets the variable values, with later files overriding previous files.

We can use this to implement a tfvars hierarchy. The first step is defining a tfvars file in each level of our deployments directory structure:

deployments/
    app/
        dev/
            test-stack/
                terraform.tfvars
                terragrunt.hcl
        stage/
            test-stack/
                terraform.tfvars
                terragrunt.hcl
        prod/
            test-stack/
                terraform.tfvars
                terragrunt.hcl
            terraform.tfvars
        terraform.tfvars
    terraform.tfvars
    root.hcl

Next, define values at each level:

# deployments/terraform.tfvars
global_var = "set in deployments/"

# deployments/app/terraform.tfvars
tier_var = "set in deployments/app/"

# deployments/app/dev/terraform.tfvars
env_var = "set in deployments/app/dev"

# deployments/app/dev/test-stack/terraform.tfvars
stack_var = "set in deployments/app/dev/test-stack/"

and so on for deployments/app/test/** and deployments/app/prod/**.

Next, we modify our root.hcl file to load all the tfvars files. We make use of the deployment_path_components local we defined in part 1 to generate a list of all possible tfvar locations:

# root.hcl
locals {
  relative_deployment_path   = path_relative_to_include()
  deployment_path_components = compact(
    split("/", local.relative_deployment_path)
  )

  ...

  # Get a list of every possible tfvars path between root_deployments_directory
  # and the path of the deployment
  possible_config_locations = [
    for i in range(0, length(local.deployment_path_components) + 1) :
      join("/", concat(
        [local.root_deployments_dir],
        slice(local.deployment_path_components, 0, i),
        ["terraform.tfvars"]
      ))
  ]
}

Finally, instruct terragrunt to pass the possible_config_locations as optional_var_files to terraform under an extra_arguments block:

# root.hcl
...

terraform {
  ...
  extra_arguments "load_config_files" {
    commands = get_terraform_commands_that_need_vars()
    optional_var_files = local.possible_config_locations
  }
}

Having done this, we can run plan and apply and see the variables are loaded. For instance, running terragrunt apply from deployments/app/dev/test-stack produces:

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

env_var = set in deployments/app/dev
global_var = overridden in deployments/app
layer_var = unset
stack_var = set in deployments/app/dev/test-stack/
tier_var = set in deployments/app/

We can also override values in lower levels. For instance, changing the value of global_var in deployments/app/terraform.tfvars:

# deployments/app/terraform.tfvars
global_var = "overridden in deployments/app"
tier_var   = "set in deployments/app/"

and running terragrunt output global_var in deployments/app/terraform.tfvars produces:

overridden in deployments/app

Where tfvars fail

The above worked well until terraform 0.12, which introduced a controversial feature to print warnings when values are specified for undefined variables. For instance, if we add an unused variable to deployments/terraform.tfvars:

# deployments/terraform.tfvars
global_var = "set-in-deployments/"
unused     = true

any stack we plan or apply now prints:

Warning: Value for undeclared variable

The root module does not declare a variable named "unused"
but a value was found in file
"/Users/td/code/td/terraform-skeleton/deployments/terraform.tfvars".
To use this value, add a "variable" block to the configuration.

The warnings are annoying enough in that they clutter the plan output, but worse still the warning states:

Using a variables file to set an undeclared variable is deprecated and will
become an error in a future release.

That means we can’t reliably use tfvar files as our source of a variable hierarchy for the long term, at least if you have any variables defined in tfvars that are loaded and go unused by any stacks.

You might wonder if this is actually a problem. Hashicorp states the change was made “to give better feedback about mistakes”, so maybe we should just avoid having variables defined in tfvars files that are loaded by stacks that do not use them.

Allow me to present an example where unused variables are helpful.

Unused variables: a use case

My team has several global flags for our infrastructure. An example is one used to conditionally enable or disable access to our system for a third-party team. Any stack that must make changes to facilitate that access defines an enable_third_party_access variable in its variables.tf file. If true, that stack makes the necessary changes.

Multiple stacks defining this variable poses a challenge. When third-party access is requested, we don’t want to have to find every deployment of every stack with this variable and set its value to true. We want to define the enable_third_party_access once, either across all environments or for specific environments, so we don’t forget to enable or disable individual flags. Furthermore, not every stack needs to enable third-party access, and therefore not every stack needs this variable.

Leveraging unused variables means we can set the enable_third_party_access value in fewer places, leading to fewer mistakes.

There are plenty more examples of how unused variables are useful in the Hashicorp issue discussion. So how do we support unused variables in 0.12 and beyond?

Variables via YAML (0.12+)

Fortunately, Hashicorp does give you a workaround for defining values for unused variables, also displayed in the warning messages they print:

If you wish to provide certain "global" settings to all configurations in
your organization, use TF_VAR_... environment variables to set these instead.

So instead of using tfvar files, we need to define environment variables. But how can we do that easily and reproducibly across systems? Terragrunt again comes to the rescue.

We can pass an HCL map to an inputs attribute in our root.hcl, which terragrunt converts to TF_VAR environment variables passed to terraform. Now we just need a map of our variables and their values. There’s no easy way to generate such a map from tfvars files, but terragrunt provides access to all of terraform’s functions in the HCL file. This gives us the building blocks we need to use YAML files instead of tfvars to build our variable hierarchy. Specifically, we’ll use:

fileexists - for checking if a file exists before attempting to load it
file - for loading file contents to a string
yamldecode - for converting a YAML string to an HCL map
merge - for merging multiple HCL maps

Start by converting the terraform.tfvars files to config.yml files¹. For instance, convert:

# deployments/terraform.tfvars
global_var = "set-in-deployments/"
unused     = true

to:

# deployments/config.yml
---
global_var: "set-in-deployments/"
unused: true

Next, modify root.hcl to load the YAML files, first by removing the extra_arguments block we used for loading the tfvars:

# root.hcl
...
terraform {
  source = "${local.root_deployments_dir}/../modules/stacks/${local.tier}/${local.stack}"
  # no extra_arguments block here
}

then by changing the locals to load the YAML files and pass them into the terragrunt inputs:

# root.hcl
locals {
  ...

  # Get a list of every path between root_deployments_directory and the path of
  # the deployment
  possible_config_dirs = [
    for i in range(0, length(local.deployment_path_components) + 1) :
      join("/", concat(
        [local.root_deployments_dir],
        slice(local.deployment_path_components, 0, i)
      ))
  ]

  # Generate a list of possible config files at every possible_config_dir
  # (support both .yml and .yaml)
  possible_config_paths = flatten([
    for dir in local.possible_config_dirs : [
      "${dir}/config.yml",
      "${dir}/config.yaml"
    ]
  ])

  # Load every YAML config file that exists into an HCL map
  file_configs = [
    for path in local.possible_config_paths :
      yamldecode(file(path)) if fileexists(path)
  ]

  # Merge the maps together, with deeper configs overriding higher configs
  merged_config = merge(local.file_configs...)
}

# Pass the merged config to terraform as variable values using TF_VAR_
# environment variables
inputs = local.merged_config

...

And that’s it. We have a YAML-based variable hierarchy that supports overriding and unused variables. The final result is available on GitHub

What’s next?

So far, the skeleton has been using the default local backend for state storage. For this project to work in a team setting with multiple developers and/or CI/CD performing terraform commands, we will need to store terraform state remotely with locking to avoid concurrent executions from stepping on each other. Setting up remote state storage will be the subject of the next post.

Footnotes

The YAML loading doesn’t play nice with config.yml files that are empty or contain just a --- start of document marker. You can delete the empty config.yml file and everything will work, due to the fileexists check. If you prefer to keep the empty config.yml, the most minimal contents required are:
```
---
{}
```
↩

Terraform Skeleton Part 1: Organizing Terragrunt

2021-01-17T12:23:00+00:00

Terraform is my go-to infrastructure definition tool. I love that it enables declarative management of different Cloud, PaaS, and SaaS platforms with its unified HCL language and provider model. At times, however, I’ve wished it was easier to start a project off on the right foot.

When I was getting started, it wasn’t obvious to me how to organize a terraform project in a way that makes “the easy things easy and the hard things possible” and would scale well as a team grows. Terragrunt helps a great deal, but there are many configuration options, adding to the organizational complexity.

This blog series takes lessons learned from using terraform and terragrunt in production across multiple teams and, over many posts, condenses those lessons into a walking skeleton repo to serve as a starting point for teams working with terraform and terragrunt. I’ve used this skeleton’s approach for a moderately complex suite of applications spanning multiple clouds and multiple Kubernetes clusters. I’m also using it with a smaller team running a much simpler monolith application. I believe it is a solid starting point, worth sharing with others for consideration.

I’m using semantic versioning to track changes to the skeleton repo. This post covers major-minor version 1.0. The code this post arrives at is available on the release branch for that version here.

Goals

Version control all infrastructure as code
Provide a basic safety net of explicit tool versions and pre-commit checks
Re-use the same infrastructure definition files across multiple environments
Have multiple levels of blast radiuses; allow the engineer to easily apply a small stack of infrastructure or an entire environment
Focus on manual infrastructure deployment for now. Continuous Integration will follow. Continuous Delivery will be case-by-case.¹

Setup

You’ll need the following installed on your workstation:

git
tfenv for managing terraform versions²
tgenv for managing terragrunt versions
pre-commit for running syntax, semantic, and style checks on git commit

The ‘infra’ repo

I currently start teams off with a monorepo structure for terraform infrastructure. I admit that having one or more separate modules repositories, as Terraform Up & Running advises, is a good destination, but my teams have found it easier to get started making changes within a single repository. The organization that follows works both with mono and distributed repositories, however. Teams I support have made the jump from monorepo to distributed, and it was not painful.³

Our team calls our monorepo infra, and that’s the name I will use here.

Step 1: Boilerplate

After installing the tools listed above and creating a new infra repository:

Create a CHANGELOG.md

I like the style of keepachangelog.com.
Add a .gitignore

I use gitignore.io as a starting point.

Set tool versions

For instance, if using terraform 0.13.5 and terragrunt 0.26.4 set the versions with:

 echo "0.13.5" > .terraform-version
 echo "0.26.4" > .terragrunt-version

Then install the versions with:

 tfenv install
 tgenv install

Commit and update your changelog accordingly e.g.,

 ## [Unreleased]

 ### Added
 - terraform version set to 0.13.5
 - terragrunt version set to 0.26.4

Install pre-commit hooks

These hooks provide a safety net that gets executed on each git commit. I’d start with at least the following:⁴
- terraform_fmt hook formats *.tf files using the terraform fmt command
- terraform_validate similarly runs terraform validate
- terragrunt-hclfmt runs terragrunt hclfmt on terragrunt.hcl files
You specify these in a .pre-commit-config.yaml file. Here’s what it looks like.

Next, run pre-commit install to install the hooks.

Step 2: Directory Structure

Covering the directory structure introduces a lot of terms. I start with two top-level directories:

deployments/
modules/

Deployments

deployments are instantiations of infrastructure. Directories under deployments contain terragrunt.hcl files. You run terragrunt commands inside deployment directories.⁵

Deployments are organized in a directory tree. The first level in the tree is the infrastructure tiers. Small projects may just have a single tier of infrastructure. For larger projects, there are often multiple tiers relying or building on each other. Sometimes these tiers are managed by different teams. For instance, my team has a foundation tier containing shared services like our docker image registry and CI/CD tooling, and a service tier containing application infrastructure:

deployments/
    foundation/
    service/

Each tier consists of one or more instantiations called environments. For instance: development, staging, and production.

deployments/
    foundation/
        dev/
        stage/
        prod/
    service/
        dev/
        stage/
        prod/

Underneath each environment, you have stacks and/or layers of stacks that make up that tier-environment. Layers are optional directories grouping stacks together. Stacks are bundles of infrastructure that are managed as a unit using terragrunt commands.

For instance, our foundation tier might consist of a network and k8s stack, and a jenkins stack underneath an apps layer:

deployments/
    foundation/
        dev/
            network/
                terragrunt.hcl
            k8s/
                terragrunt.hcl
            apps/
                jenkins/
                    terragrunt.hcl
    root.hcl

Bringing it all together, each deployment is an instantiation of a stack for a tier-environment. For example, the jenkins stack for the foundation tier’s dev environment, which my team would summarize as foundation-dev-jenkins. To manage foundation-dev-jenkins, you run terragrunt commands from the deployments/foundation/dev/apps/jenkins directory.

Deployment directories do not contain the infrastructure definition (.tf) files themselves, however. Instead, each deployment references a stack underneath modules/stacks/. This allows us to have a single stack definition deployed multiple times - across multiple environments for example.

Modules

Expanding modules I have:

deployments/
modules/
    components/
    stacks/

stacks are root terraform modules and group together infrastructure that is managed as a unit. Stacks are defined as directories under modules/stacks/, organized by tier, and contain the *.tf files defining the stack’s infrastructure. Using our foundation example from above, modules/stacks looks like this:

modules/
    stacks/
        foundation/
            network/
                ...
            k8s/
                ...
            jenkins/
                main.tf
                outputs.tf
                providers.tf
                variables.tf

Stacks may contain terraform provider resources directly (e.g., aws_s3_bucket), but often will contain child terraform modules, which I call components.

Any components specific to this infra repository reside in directories under modules/components, and those directories similarly contain *.tf files defining the infrastructure belonging to the component. For example:

modules/
    components/
        some-component/
            main.tf
            variables.tf
            outputs.tf
        another-component/
            ...

Summarized, each deployment references a stack, and each stack may instantiate any number of components.

You probably noticed the terragrunt.hcl and root.hcl files in the example above. Let’s talk about those next.

Step 3: HCL files

Each deployment contains a terragrunt.hcl file, marking it as a stack to be deployed with terragrunt. They are usually extremely simple, only containing an include to the root.hcl file.⁶

# terragrunt.hcl
include {
  path = find_in_parent_folders("root.hcl")
}

The root.hcl file contains all the terragrunt configuration. A simple starting point is:

# root.hcl
locals {
  root_deployments_dir       = get_parent_terragrunt_dir()
  relative_deployment_path   = path_relative_to_include()
  deployment_path_components = compact(split("/", local.relative_deployment_path))

  tier  = local.deployment_path_components[0]
  stack = reverse(local.deployment_path_components)[0]
}

# Default the stack each deployment deploys based on its directory structure
# Can be overridden by redefining this block in a child terragrunt.hcl
terraform {
  source = "${local.root_deployments_dir}/../modules/stacks/${local.tier}/${local.stack}"
}

This makes it so each deployment directory, by default, deploys the stack under modules/stacks/<tier> with the same name as the deployment directory. For instance, deployments/foundation/dev/apps/jenkins/ deploys modules/stacks/foundation/jenkins.⁷ I haven’t decided if I like making the layer part of that path or not, so thus far have left it out.

With an understanding of the tools, directory structure, and HCL files, we can put together a minimal skeleton repo.

Step 4: The Skeleton

Here’s a minimal skeleton I would start with, not including boilerplate files:

deployments/
    app/
        dev/
            test-stack/
                terragrunt.hcl
        stage/
            test-stack/
                terragrunt.hcl
        prod/
            test-stack/
                terragrunt.hcl
    root.hcl
modules/
    components/
    stacks/
        app/
            test-stack/
                main.tf
                outputs.tf
                providers.tf

Our basic test-stack could use the random provider to create a single resource and output:

# main.tf
resource "random_pet" "pet" {
}

# outputs.tf
output "pet" {
  value = random_pet.pet.id
}

# providers.tf
terraform {
  required_providers {
    random = {
      source  = "hashicorp/random"
      version = "~> 3.0.0"
    }
  }
}

With this in place, we can run terragrunt plan and terragrunt apply from any directory under deployments/. We can also run plan-all and apply-all from:

The root of the repository to affect all tiers and all environments
Any directory within deployments to affect everything underneath that directory

We can also use –terragrunt-exclude-dir and –terragrunt-include-dir to target *-all commands. For example:

terragrunt apply-all --terragrunt-exclude-dir deployments/*/prod/**

would apply all non-production environments across all tiers.

What’s next?

This skeleton is quite minimal. We haven’t covered variables, backends, RBAC, external modules, build tooling, continuous integration, and so on.

The next post will dive deeper into what we can do with root.hcl, including introducing a variable loading hierarchy that makes our deployments easier to work with.

Footnotes

In short, CI/CD with terraform is tricky. The authors of terragrunt discuss some of the challenges here. I’m a proponent of pushing CI/CD for infrastructure, but doing so carefully and expecting times of manual intervention. Some things are difficult to automate out of the box - state migrations (e.g., terraform state mv) being one example. I’ve trended towards starting teams off with CI and manual applies of infrastructure and working towards CD as the team becomes more comfortable. More on CI/CD in future posts. ↩
Terraform stores its state in state files and those state files contain the version of terraform last used. The version specified in the state file is auto-bumped whenever a newer version of terraform touches that state file. Once that happens, you may force anyone working on that infrastructure, or relying on outputs from your state file via terraform_remote_state data sources to upgrade. The second case is the most damaging because it can affect teams outside of your own, depending on who’s using your outputs. terraform 0.14 addresses this, making state files backward and forward compatible across terraform versions as much as possible, but anyone using older versions of terraform should be particularly careful with terraform versioning. ↩
We’ve since split our infrastructure across multiple repositories, but only after demonstrated needs: distributed ownership, differing change/release rates, and re-use across teams. Much like starting with a monolith before evolving to microservices, starting with an infra monorepo avoids bloat slowing your team unnecessarily. ↩
There are lots of other tools/hooks that are worth looking at that I haven’t had time to yet. Some on my list to explore include:
- tfsec - example hook
- terraform docs - example hook
- tflint - example hook
- checkov - official hook
↩
If you’ve read Terraform Up & Running, my deployments/ directory equates to Yevgeniy’s live/ directory. My team just found the name deployments to be more understandable. ↩
I use the name root.hcl instead of terragrunt.hcl because the latter causes errors running plan-all and apply-all from the root or deployments/ directory. Terragrunt seems to treat the parent hcl file as a stack to be deployed and errors out. I tried terragrunt’s skip option but to no avail, at least as of version 0.26.2 ↩
Terragrunt glues the deployment to the stack definition by copying everything in the deployment directory and everything in the referenced stack directory (containing the *.tf files) into a .terragrunt-cache directory and then executing commands from that directory. By default, the .terragrunt-cache directory lives in your deployment directory. ↩