Multiple Terraform projects in a mono-repo. How to survive a mess?

Published in

Westwing Tech Blog

10 min readJan 22, 2024

Do you have a set of projects sitting in a mono-repo and having various workspaces, file structures, and Terraform versions? A pain of switching the versions and remembering all path/workspace combinations? Uncertainty about the correctness of the workspace, or plan file before applying it?
I feel you! I’d share my experience in managing such projects, an approach to make it much easier, and a simple tool I wrote a few years ago for that. How is it related to Docker Compose? I’ll tell you…

Does your Terraform repository look like that?

TL;DR

In this article, I would like to introduce the Terra Compose tool. It aims to simplify the management of multiple Terraform projects within a single mono-repo. By solving problems with fragile and long maintenance and uncertainty about the correctness changes, caused by low visibility of the workspace.

For that, it follows the approach of the Docker Compose and puts all the needed information into the YAML config. This way it gets rid of human involvement as much as possible, minimizing the risk of human error.

For those, who like to make their hands dirty instead of reading articles, I would suggest checking out the Terra Compose Demo project.

Problem Deep Dive
Exploring existing solutions
Introducing Terra Compose
Demo
Final words

Problem Deep Dive

In this section will be listed problems, that are faced while managing multiple Terraform projects in mono-repos.

Prerequisites are:

set of projects in a mono-repo with various file structures
various workspaces that are almost equal to environments
various Terraform versions in the projects

Terms

A few terms, used below:

Project — is a part of an infrastructure that is deployed separately and has a separate state. For example, one of the platform components (like runners or alerts) or application-specific infrastructure (like a dedicated application Redis cluster, Security Group, or Database).
Environment — is an isolated space of the application infrastructure’s stack. In general, an application has more than one environment, which almost always is equivalent to the Terraform workspace.
Alias — is a combination of the Project / Environment, that can be described like Runners in the production environment. Unique across the repository.

Various file structures

Diving a bit deeper, in such mono-repos there is one of different approaches to structure Terraform projects — workspace-based and environment-based. Or even a mix of them.

Moreover, sometimes a project has a complex file structure because of the internal modules or components, that add an extra layer to the project structure. In the given example below there is a project portal, having 2 components — app and platform.

├── ...
├── common
│   └── *.tf
├── environments
│   ├── docs
│   │   └── *.tf
│   └── images
│       ├── prod
│       │   └── *.tf
│       └── staging
│           └── *.tf
├── modules
│   └── common
│       └── sg
│           └── *.tf
└── projects
    ├── cms
    │   └── *.tf
    ├── portal
    │   ├── app
    │   │   └── *.tf
    │   └── platform
    │       └── *.tf
    └── runners
        └── *.tf

Despite their differences, these projects often share common components or modules, such as VPC or Security Group (see the common project).

Various workspaces

As was mentioned before, each project usually has a few environments (often Terraform workspaces). In legacy repositories or when projects are maintained by different teams, workspaces and file names follow different naming patterns, the same as the file structure.

├── ...
├── common
│   ├── nonprod.tfvars
│   ├── prod.tfvars
│   └── ...
└── projects
    ├── cms
    │   ├── production.tfvars
    │   ├── release.tfvars
    │   ├── staging.tfvars
    │   ├── training.tfvars
    │   └── ...
    ├── portal
    │   ├── app
    │   │   ├── integration.tfvars
    │   │   ├── live.tfvars
    │   │   └── ...
    │   └── platform
    │       ├── integration.tfvars
    │       ├── live.tfvars
    │       └── ...
    └── ...

Various versions

The projects also might have specific Terraform versions. Especially, in big repositories. I saw cases where across different projects in a repository at the same time there were several different Terraform versions from 0.12.x to 1.3.x.

Sometimes, during the Terraform version upgrade process, there might be also different Terraform versions across the project environments.

Pain points

Summarizing it, we have identified several pain points.

The first challenge is simply memorizing different file structures, workspaces, and Terraform versions. This diversity with the lack of visibility doubling effect, especially for newcomers, makes it challenging to maintain and troubleshoot effectively.

The second issue involves the inability to use unified automation due to differences in file structures and workspaces.

Further categories of concern include upgrading and switching Terraform versions. This process is not that hard, but it requires having a few Terraform versions locally at the same time and being able to quickly switch between them. For sure, there are already solutions for this problem, for instance, the tfenv tool can help here. However, it has some limitations and requires a bit of effort to configure it.

Lastly, as-is it breaks the IaaC approach because so far workspaces are not trackable by any version control system. With that, we have all bad outcomes, related to having it not as a code…

In essence, these challenges manifest as specific and fragile maintenance, and extended maintenance times (especially for larger projects).

A low visibility of the workspace escalates the uncertainty about the correctness while planning and applying the changes.

One generic approach to solving the problem partially could be to split mono-repo into separate repositories, referencing common elements using modules. With that, there might be reasons not to do this. One of them is that it increases the count of moving parts and infrastructure maintenance overhead. Another one, it potentially raises security concerns with different ownership and membership.

Exploring existing solutions

What about the market? Are there any available solutions? While there are several options, none of them comprehensively address the challenges at hand.

Terraform Cloud is a native for Terraform SaaS solution. It offers a shared state, secret data protection, a private modules registry, self-controlled policy changes, and many more. While it is an excellent solution, it might limit you or be too expensive, especially if there are numerous team members to be licensed. Besides that, SaaS is not for everyone…

Terragrunt is a powerful Terraform wrapper that helps to follow the DRY principles and code re-usage. It is widely recognized, but it comes with significant overhead. It requires separate configuration creation for each folder, specifying properties that Terragrunt relies on. Moreover, it does not solve all mentioned pain points.

Terramate is yet another wrapper for Terraform that provides code generation, stack change detection, orchestration, and data sharing. While it looks promising, it introduces even more additional complexity due to a new stack concept. It might work well when starting a new project from scratch, but jumping into it with an existing one might be challenging. Also, it seems, the core feature in Terramate is the generation code, thereby using mostly internal or external modules makes this tool useless overall.

Terraspace is a framework with structure conventions, code generation, and tooling. It has the same downsides, as Terramate does, and demands even more extensive adaptation effort, but fails to address all our problems.

Comparing solutions

When comparing these solutions, it becomes evident that none completely solves our problems. Whether it is version or workspace or project path switching, needs to memorize them, or pain of switch, none of the tools provide a comprehensive solution. They lack certain features and will not be free of charge for such scale projects.

Lastly, considering the three-year journey of the Terra Compose, initiated before tools like Terramate or Terraspace came into the picture, it becomes clear that neither then, nor nowadays these problems could not be fully solved with the current market offerings.

Introducing Terra Compose

The idea

Haven’t we faced it before?

I found that the situation is similar to working with Docker, which gets tricky with lengthy command lines, involving lots of arguments. It very quickly becomes a hassle, trying to even print multiple command arguments.

Drawing parallels with Docker, a solution emerged in the form of Docker Compose. This tool simplifies the setup with a straightforward YAML file, describing the services to run and their configurations.

Compose simplifies the control of your entire application stack, making it easy to manage services, networks, and volumes in a single, comprehensible YAML configuration file. Then, with a single command, you create and start all the services from your configuration file.

It significantly enhances the developer experience, making the process of running Docker commands error-free and increasing visibility by implementing the Config-as-a-Code approach.

Recognizing the similarities, I pondered, “Why couldn’t we have something similar for Terraform?”

Terra Compose: Use cases

Just like Docker Compose simplifies Docker commands, Terra Compose handles various Terraform commands efficiently, providing flexibility in using different Terraform versions in Docker and workspace preparation with automation scripts.

Here we need to define yet another term — action. So, in Terra Compose it is a set of chained commands to perform a consistent operation. It has the following stages:

pre-actions, e.g. change directory, select workspace, init Terraform backend, etc.
action, the target command runs.
post-action, e.g. clean-ups

To streamline the process, there are shortcuts for common actions plan and apply, ensuring a consistent and reproducible behavior. These shortcuts wrap sequences of Terraform commands, reducing errors and improving reliability.

For instance, plan action automatically includes Terraform init, workspace selection, format check, and validation. Thus it shifts left possible failures. As a result, it saves the plan to the file, which then will be used in apply action. In this way, it ensures that only verified changes were applied. Apply action also protects the operator from applying stale changes by highlighting the “since-plan-creation” time.

Besides them, there are:

run — a proxy action for ANY command to be run inside the Docker container.
workspaces — a simple shortcut for running terraform workspace ls command.
shell — spins up an interactive shell inside the Docker container. Useful for troubleshooting, debugging, and some complex Terraform tasks.
help — outputs some information about available actions in Terra Compose and, when there is a config file available, a list of available aliases.

Terra Compose: The workflow

Terra Compose reads the config file, finds the project by the path property if alias, and runs the requested Terraform command in the Docker container with the specified Terraform workspace, version, and tfvars file. Within the run execution, it includes automation at the pre- and post-stages, like changing directories or selecting workspace, and hooks if were defined.

This setup simplifies the collaboration and ensures a consistent and reliable Terraform workflow with improved visibility and developer experience.

Terra Compose: Configuration

Maintaining the Terra Compose requires only a single config in a YAML file, which is easy to manage. It contains project details, defining aliases along the relevant properties. It is a part of the codebase with the same visibility.

Here is an example of the YAML file. There are two main sections: default and aliases. In default section there are common properties (so far only tfversion), that could be overwritten by the property in aliases. In the aliases section, there are all the needed definitions, including Terraform workspace, version, and tfvars file name (if used).

default:
  tfversion: "1.0.0"

aliases:
  common:
    path: "common"
    tfvars: "common"
  alerts:
    path: "projects/alerts"
    workspace: "prod"
    tfversion: "1.3.0"
    backend_config: "key=experiment"
    hooks:
        before_tf_init: "rm -rf ./.terraform"

Additional properties, like backend_config and hooks, allow to customize further the workflow. There are 5 hooks supported so far, allowing to run custom scripts before/after some of the stages, for example before_tf_init, after_all or before_container_run.

Demo

The full demo can be found in the Terra Compose Demo project. Below you can find a small example of the simple usage of plan, apply, workspaces and run actions.

Besides that, it shows the debug mode usage. Debug mode skips pre- and post-stage on plan and apply actions. It aims to speed up the development and debugging processes on repetitive actions.

Final words

Thus, the Terra Compose tool composes multiple Terraform commands, such as plan or apply, providing a validated plan, and speeding up the maintenance process.

Pain points revise

Let’s check our pain points, focusing on the Terra Compose:

Version Switching ✔️
Successfully tackled by running Terraform in a Docker container and having Terraform versions as a property in the config YAML file.
Structure: Snippets and Compatibility ✔️
Addressed by abstracting the project root path and adding it to the configuration. Some flexibility improvements are planned for the future.
Workspace Visibility ✔️
Improved it by printing it in several places in the Terra Compose output.
Workspace Low Tracking ✔️
Now it’s part of the codebase and trackable by VCS, the issue is solved.
Routines Automation / Chaining commands ♻
The issue is partially solved, as automation is currently hard-coded but feasible. Hooks mechanism extends it also a lot. With that, some improvements in the chaining mechanism, like customization, are planned for the future.
Structure, Workspace, Version Management ✔️
Successfully addressed by adding it to the config YAML file, which made it solid.
Complexity and Implementation Time ✔️
Terra Compose is an open-source project, thus — free
The setup process is straightforward, includes configuring a simple YAML config file, and is quick to implement (approximately an hour to onboard a new project).

Future Plans

There are also a few tasks planned for the future:

Improve Terra Compose command interface. It was designed 3 years ago as a quick wrapper, so no solid strategy was behind it. Now it’s time to re-think that.
Make commands customizable to offer users more flexibility in their workflows. The flexible way of chaining commands in action could be an example here.

Conclusion

The open-source tool is available for use, and I want to encourage contributions from the community. Feel free to check its repository for more details. From my experience, it consistently protected me and saved me a lot of time. Therefore, I find it very useful and I believe it could be valuable for you too.

P.S. It was my first article here, so please clap it :)

Links:
- Terra Compose project on GitHub
- Terra Compose Demo project on GitHub
- LinkedIn