Terraform from day one: infrastructure as code that lasts

Most production setups that nobody understands started the same way: someone created a database and a few services in the cloud console to get a first version running, intending to tidy it up later. Later rarely comes. A year on, nobody is sure which settings matter, staging no longer matches production, and recreating the setup would mean reverse-engineering it.

Writing infrastructure as code from the first environment avoids that. SoulCare, our home-care platform, has its infrastructure in Terraform. This article covers how we would set up a new project the same way: repository layout, environments, state, review, secrets and what a team taking it over needs.

Why start on day one

Infrastructure as code is cheapest when there is little infrastructure. Importing an existing setup into Terraform means finding every resource, matching its settings and accepting whatever choices were made in a hurry. Starting with code means every resource has a reason, a history in version control and a reviewer.

It also makes a second environment straightforward. If production is defined in code, staging is a second set of inputs to the same code, not a weekend of clicking.

Repository layout

Keep the layout simple enough that someone new can find things. A common structure separates reusable modules from the environments that use them:

infra/
  modules/
    network/
    database/
    api/
  environments/
    staging/
      main.tf
      backend.tf
      terraform.tfvars
    production/
      main.tf
      backend.tf
      terraform.tfvars

• Modules describe one part of the system, such as the network or the database, with inputs for anything that differs between environments.
• Environments call those modules with their own values and their own state.
• Start with fewer modules. It is easier to split a module later than to untangle one that was abstracted too early. A module that is used once is often just a folder.

Whether infrastructure lives in the application repository or its own is a judgement call. Keeping it alongside the application makes related changes reviewable together. A separate repository suits organisations where a platform team manages infrastructure for several products.

Environments

Each environment should be a separate state and, ideally, a separate cloud account. Account-level separation means a mistake in staging cannot touch production, and access can be granted per environment.

Keep the differences between environments explicit and small: instance sizes, domain names, scaling limits and retention periods. If staging is structurally different from production, it stops being a useful test of it.

Here is a simplified example of an environment calling a module, not our production code:

module "database" {
  source = "../../modules/database"

  name                = "app-staging"
  instance_class      = var.db_instance_class
  backup_retention    = 7
  deletion_protection = false
}

State

Terraform’s state file records what it has created. Treat it carefully.

• Store it remotely. Use a remote backend, such as an S3 bucket with locking, so two people cannot apply changes at once and the state is not on one laptop.
• One state per environment. A change in staging should never be able to plan changes to production.
• Restrict and protect it. State can contain sensitive values. Encrypt the bucket, turn on versioning and limit who can read it.
• Avoid editing it by hand. Moves and imports should be done with Terraform commands or moved blocks in code, so they are reviewed like any other change.

Reviewing plans in pull requests

The value of infrastructure as code comes from review. A change to a security group or a database setting should be read by a second person before it reaches production, in the same way as application code.

• Run terraform plan in CI on every pull request and post the output where reviewers can see it.
• Read the plan, not just the diff. The code diff shows intent. The plan shows what will actually be created, changed or destroyed. Pay particular attention to anything marked for replacement.
• Apply from the pipeline, not from laptops. Once merged, apply the reviewed plan from CI using credentials scoped to that environment. This keeps a record of every change and who approved it.
• Check formatting and validation automatically. terraform fmt and terraform validate catch simple mistakes before a reviewer has to.

Secrets

Secrets are where infrastructure code most often goes wrong.

• Never commit them. Passwords, API keys and tokens do not belong in .tfvars files or anywhere else in the repository.
• Use a secrets manager. Create secrets in the cloud provider’s secrets service and give applications permission to read them at runtime. Terraform can create the container for a secret without holding its value.
• Prefer roles to keys. Give services and the CI pipeline roles with the permissions they need, rather than long-lived access keys.
• Plan for rotation. Decide how each secret is changed and what restarts when it is. It is easier to design for this than to do it under pressure.

What a team inheriting the system needs

Infrastructure is often handed over: to an in-house team, a new supplier or a future version of the same team. Code alone is not enough. A useful handover includes:

• a short guide to the repository layout and how to run a plan locally;
• how environments map to cloud accounts and who has access to each;
• where state is stored and how locking works;
• the release process, from pull request to applied change;
• where secrets live and how they are rotated;
• anything created outside Terraform, with the reason, so it is not mistaken for drift;
• known gaps and planned improvements.

If someone new can make a small, reviewed change to staging in their first week, the setup is in good shape.

The trade-offs

Terraform adds work at the start. A first environment takes longer to create in code than by hand, and the team needs to learn its model of state and plans. Some managed services change quickly, and their Terraform support can lag behind the console. There will be occasional manual steps, and they need to be written down.

For a throwaway prototype, clicking through the console is reasonable. For anything that will hold customer data or be handed to another team, the extra effort early is much smaller than reconstructing the setup later.

When to bring in help

If you are about to set up a first production environment, or you have one that was built by hand and nobody wants to touch, a review of how it is defined and deployed is a sensible first step. We can help structure the Terraform, set up plan review and applies in your pipeline, and write the handover your team will need to run it.