In part 1 of the terraform skeleton series, we set up a terraform repository that allows the team to apply infrastructure at any level: from individual stacks to entire environments. We build on that foundation in this post, adding a variable hierarchy that similarly allows definition and overriding of variables at each level of the infrastructure.

Goals

As a refresher, our infrastructure is organized by:

  • Tier
  • Environment
  • Layer (Optional)
  • Stack

A deployment is an instantiation of a stack for a tier-environment.

Today’s goals are to:

  • Allow defining variable values in files at each level of the infrastructure
  • Have lower-level variable definitions override higher levels

If you prefer to jump to the end, the code implementing the final result is available on GitHub.

Setup

Let’s add to our test-stack some variables to be defined at each level of deployments/:

# modules/stacks/app/test-stack/variables.tf
variable "global_var" {
  default = "unset"
}

variable "tier_var" {
  default = "unset"
}

variable "env_var" {
  default = "unset"
}

variable "layer_var" {
  default = "unset"
}

variable "stack_var" {
  default = "unset"
}

Let’s also add outputs to the stack so we can see what the variables are set to:

# modules/stacks/app/test-stack/outputs.tf
output "pet" {
  value = random_pet.pet.id
}

output "global_var" {
  value = var.global_var
}

output "tier_var" {
  value = var.tier_var
}

output "env_var" {
  value = var.env_var
}

output "layer_var" {
  value = var.layer_var
}

output "stack_var" {
  value = var.stack_var
}

Next, we’ll cover implementing the variable hierarchy in two ways. The first works for terraform pre-0.12. The second slightly modifies our first approach to work for 0.12+.

Variables via tfvars (pre-0.12)

tfvars files are a standard way of defining variable values for terraform. Terragrunt allows you to define required_var_files and optional_var_files within the terraform block of your terragrunt.hcl, as covered by this Gruntwork post. Terragrunt then passes these files to terraform using its -var-file option. Terraform loads the files and sets the variable values, with later files overriding previous files.

We can use this to implement a tfvars hierarchy. The first step is defining a tfvars file in each level of our deployments directory structure:

deployments/
    app/
        dev/
            test-stack/
                terraform.tfvars
                terragrunt.hcl
        stage/
            test-stack/
                terraform.tfvars
                terragrunt.hcl
        prod/
            test-stack/
                terraform.tfvars
                terragrunt.hcl
            terraform.tfvars
        terraform.tfvars
    terraform.tfvars
    root.hcl

Next, define values at each level:

# deployments/terraform.tfvars
global_var = "set in deployments/"

# deployments/app/terraform.tfvars
tier_var = "set in deployments/app/"

# deployments/app/dev/terraform.tfvars
env_var = "set in deployments/app/dev"

# deployments/app/dev/test-stack/terraform.tfvars
stack_var = "set in deployments/app/dev/test-stack/"

and so on for deployments/app/test/** and deployments/app/prod/**.

Next, we modify our root.hcl file to load all the tfvars files. We make use of the deployment_path_components local we defined in part 1 to generate a list of all possible tfvar locations:

# root.hcl
locals {
  relative_deployment_path   = path_relative_to_include()
  deployment_path_components = compact(
    split("/", local.relative_deployment_path)
  )

  ...

  # Get a list of every possible tfvars path between root_deployments_directory
  # and the path of the deployment
  possible_config_locations = [
    for i in range(0, length(local.deployment_path_components) + 1) :
      join("/", concat(
        [local.root_deployments_dir],
        slice(local.deployment_path_components, 0, i),
        ["terraform.tfvars"]
      ))
  ]
}

Finally, instruct terragrunt to pass the possible_config_locations as optional_var_files to terraform under an extra_arguments block:

# root.hcl
...

terraform {
  ...
  extra_arguments "load_config_files" {
    commands = get_terraform_commands_that_need_vars()
    optional_var_files = local.possible_config_locations
  }
}

Having done this, we can run plan and apply and see the variables are loaded. For instance, running terragrunt apply from deployments/app/dev/test-stack produces:

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

env_var = set in deployments/app/dev
global_var = overridden in deployments/app
layer_var = unset
stack_var = set in deployments/app/dev/test-stack/
tier_var = set in deployments/app/

We can also override values in lower levels. For instance, changing the value of global_var in deployments/app/terraform.tfvars:

# deployments/app/terraform.tfvars
global_var = "overridden in deployments/app"
tier_var   = "set in deployments/app/"

and running terragrunt output global_var in deployments/app/terraform.tfvars produces:

overridden in deployments/app

Where tfvars fail

The above worked well until terraform 0.12, which introduced a controversial feature to print warnings when values are specified for undefined variables. For instance, if we add an unused variable to deployments/terraform.tfvars:

# deployments/terraform.tfvars
global_var = "set-in-deployments/"
unused     = true

any stack we plan or apply now prints:

Warning: Value for undeclared variable

The root module does not declare a variable named "unused"
but a value was found in file
"/Users/td/code/td/terraform-skeleton/deployments/terraform.tfvars".
To use this value, add a "variable" block to the configuration.

The warnings are annoying enough in that they clutter the plan output, but worse still the warning states:

Using a variables file to set an undeclared variable is deprecated and will
become an error in a future release.

That means we can’t reliably use tfvar files as our source of a variable hierarchy for the long term, at least if you have any variables defined in tfvars that are loaded and go unused by any stacks.

You might wonder if this is actually a problem. Hashicorp states the change was made “to give better feedback about mistakes”, so maybe we should just avoid having variables defined in tfvars files that are loaded by stacks that do not use them.

Allow me to present an example where unused variables are helpful.

Unused variables: a use case

My team has several global flags for our infrastructure. An example is one used to conditionally enable or disable access to our system for a third-party team. Any stack that must make changes to facilitate that access defines an enable_third_party_access variable in its variables.tf file. If true, that stack makes the necessary changes.

Multiple stacks defining this variable poses a challenge. When third-party access is requested, we don’t want to have to find every deployment of every stack with this variable and set its value to true. We want to define the enable_third_party_access once, either across all environments or for specific environments, so we don’t forget to enable or disable individual flags. Furthermore, not every stack needs to enable third-party access, and therefore not every stack needs this variable.

Leveraging unused variables means we can set the enable_third_party_access value in fewer places, leading to fewer mistakes.

There are plenty more examples of how unused variables are useful in the Hashicorp issue discussion. So how do we support unused variables in 0.12 and beyond?

Variables via YAML (0.12+)

Fortunately, Hashicorp does give you a workaround for defining values for unused variables, also displayed in the warning messages they print:

If you wish to provide certain "global" settings to all configurations in
your organization, use TF_VAR_... environment variables to set these instead.

So instead of using tfvar files, we need to define environment variables. But how can we do that easily and reproducibly across systems? Terragrunt again comes to the rescue.

We can pass an HCL map to an inputs attribute in our root.hcl, which terragrunt converts to TF_VAR environment variables passed to terraform. Now we just need a map of our variables and their values. There’s no easy way to generate such a map from tfvars files, but terragrunt provides access to all of terraform’s functions in the HCL file. This gives us the building blocks we need to use YAML files instead of tfvars to build our variable hierarchy. Specifically, we’ll use:

  • fileexists - for checking if a file exists before attempting to load it
  • file - for loading file contents to a string
  • yamldecode - for converting a YAML string to an HCL map
  • merge - for merging multiple HCL maps

Start by converting the terraform.tfvars files to config.yml files1. For instance, convert:

# deployments/terraform.tfvars
global_var = "set-in-deployments/"
unused     = true

to:

# deployments/config.yml
---
global_var: "set-in-deployments/"
unused: true

Next, modify root.hcl to load the YAML files, first by removing the extra_arguments block we used for loading the tfvars:

# root.hcl
...
terraform {
  source = "${local.root_deployments_dir}/../modules/stacks/${local.tier}/${local.stack}"
  # no extra_arguments block here
}

then by changing the locals to load the YAML files and pass them into the terragrunt inputs:

# root.hcl
locals {
  ...

  # Get a list of every path between root_deployments_directory and the path of
  # the deployment
  possible_config_dirs = [
    for i in range(0, length(local.deployment_path_components) + 1) :
      join("/", concat(
        [local.root_deployments_dir],
        slice(local.deployment_path_components, 0, i)
      ))
  ]

  # Generate a list of possible config files at every possible_config_dir
  # (support both .yml and .yaml)
  possible_config_paths = flatten([
    for dir in local.possible_config_dirs : [
      "${dir}/config.yml",
      "${dir}/config.yaml"
    ]
  ])

  # Load every YAML config file that exists into an HCL map
  file_configs = [
    for path in local.possible_config_paths :
      yamldecode(file(path)) if fileexists(path)
  ]

  # Merge the maps together, with deeper configs overriding higher configs
  merged_config = merge(local.file_configs...)
}

# Pass the merged config to terraform as variable values using TF_VAR_
# environment variables
inputs = local.merged_config

...

And that’s it. We have a YAML-based variable hierarchy that supports overriding and unused variables. The final result is available on GitHub

What’s next?

So far, the skeleton has been using the default local backend for state storage. For this project to work in a team setting with multiple developers and/or CI/CD performing terraform commands, we will need to store terraform state remotely with locking to avoid concurrent executions from stepping on each other. Setting up remote state storage will be the subject of the next post.

Footnotes

  1. The YAML loading doesn’t play nice with config.yml files that are empty or contain just a --- start of document marker. You can delete the empty config.yml file and everything will work, due to the fileexists check. If you prefer to keep the empty config.yml, the most minimal contents required are:

    ---
{}

    ↩