Terraform Cloud

What is Terraform Cloud?

Recently HashiCorp released Terraform Cloud to General Public. Terraform Cloud is HashiCorp solution for Terraform code executions, running in their hosted cloud. This solution simplifies environment management, code execution, state file management, as well as permissions management.

Full Example

If you want to see a full example of Terraform Cloud code you can check out our tutorial on GitHub: https://github.com/ops-guru/gcp-terraform-cloud. The tutorial will:

1. Create workspaces using our helper scripts
2. Push variables to workspaces using our helper script
3. Create example GCP infrastructure with dev, stg, and prd environments

Please follow instructions in the README to setup your GCP environment using Terraform Cloud.

How it works?

Terraform Cloud is a Terraform Enterprise solution stripped down of some features. In order to use Terraform Cloud you will have to sign up for an account at https://app.terraform.io/. Once you have your account you will need to setup the ~/.terraformrc file with an API key. This is a crucial piece if you intend to run Terraform Cloud code from your local machine.

Terraform Cloud uses organizations and workspaces to organize your code and environments. To quote the official documentation, “Organizations are shared spaces for teams to collaborate on infrastructure”. Each organization can have multiple workspaces that define groups of resources of your infrastructure. Normally, a user will create one organization (common use case for small and medium companies) and have multiple workspaces describing your environments. Terraform Cloud documentation provides a really good set of best practices on how to organize and name your workspaces.

Please read https://www.terraform.io/docs/cloud/workspaces/naming.html for more details.

Example of an organization and workspaces:

Workspaces are the key of how your environment is organized. Each workspace has the following key features:

• Stores the state file and history of state file changes.
• Stores Terraform variable and environmental variables.
• Stores passwords, credentials, and secret keys as sensitive.
• Connects to a GitHub repo and monitors code changes on a specified branch.

State files

Workspaces eliminate the need for S3, GCS, or any other hosted storage service to store your remote Terraform state files. Terraform Cloud will store state files for you and version it at the same time. You will be able to see a detailed trail of historical changes to the state file which is great for auditing purposes.

Example state file versioning:

If you are developing locally on your laptop, you will need to specify a workspace through the terraform init command.

For example terraform init -backend-config=init.tfvars where init.tfvars looks something like this:

If your workspace is connected to your GitHub repo then you don’t have to do anything. You only need to set workspace to listen to your branch and push the code into your branch.

Variables

Variables can be set directly in the workspace. This feature enables one code base to be reused across multiple environments. In that case, each environment will have a different set of values for variables. You can also safely store passwords and keys by setting the Sensitive flag on the variable. This feature allows security teams or admins to safely set sensitive variables.

One good use case for sensitive variables is GCP service account credentials key pair. This file is necessary for GCP terraform provider and it is usually hard to manage (some people store it in services like Vault). In the example below, the credentials.json content of a service account is stored into the credentials variable.

Example of variables:

GitHub connection

By connecting your workspace to your GitHub repository you get CI/CD pipeline for free with Terraform Cloud. You can set up your workspace to monitor changes in branches and folders. For example, you can set your network-stg workspace to monitor only network folder on master branch. Every time a commit is made, Terraform Cloud will run terraform plan. As a precautionary measure, changes will only be applied after you verify the planned changes and manually confirm.

Another cool feature of GitHub integration with Terraform Cloud is automatic run of terraform plan in response to Pull Requests. For example, if your workspace is monitoring master branch and you create a PR on that branch, Terraform Cloud will run terraform plan as part of the PR validation tests. This gives you a small sanity test before doing the PR merge.

Example of PR checks:

Workspace management

Workspaces can be managed via:

• API calls (https://www.terraform.io/docs/cloud/api/workspaces.html)
• Terraform Enterprise provider (https://www.terraform.io/docs/providers/tfe/index.html)

Terraform Enterprise provider will require you to store a state file somewhere externally (for example, S3 or GCS bucket). API calls will require you to script your own solution. In my opinion, storing the state file for workspaces beats the purpose of having Terraform Cloud. Now you will also need to worry about proper credentials to push the state file to your external storage. I also tried creating a workspace that will hold a state file for created workspaces but that didn’t work at all. I kept getting weird errors and eventually I just gave up.

Investing a bit of time to come up with a script that reads a yaml file with workspace configuration is worth a while. That way your yaml config becomes a single source of truth and as long as it doesn't have any secrets in it, it can be committed to your git repo.

You can find an example scripts to create workspaces and push variables in our example repository: https://github.com/ops-guru/gcp-terraform-cloud/tree/master/workspaces

How does it stack up?

Terraform cloud can potentially replace a need for tools like Terragrunt or for CI/CD tools like Jenkins. For example, workspaces and variables remove a need for Terragrunt’s hcl files. Integration with GitHub removes a necessity for Jenkins or Google Cloud Build for example. I’m pretty sure that we will see more features added to Terraform Cloud in the future. One that I would personally really like to see is the ability to build and execute pipelines directly in Terraform Cloud. For example, changes in network-dev workspace will trigger terraform apply in application-app workspace.

Drawbacks

Some features that work with regular Terraform or Terragrunt are not currently possible with Terraform Cloud.

For example:

• Very slow local terraform plan/apply command:

When executing terraform command from your local machine all the code and all the plugins and providers are being uploaded to Terraform Cloud for execution. This can take 5 minutes or longer per command.

• Can’t deploy Git tags:

At the moment I was only able to deploy Git branches but not Git tags. For example, it would be very useful to deploy tags only to production workspaces.

• Can’t do local-exec easily:

Terraform Cloud executes user’s code on Ubuntu OS instances/dockers. Terraform code being executed doesn’t have sudo privileges to install software through apt. This feature is only available in Terraform Enterprise. That means that if your terraform code uses a linux utility to do some pre or post processing it is very likely that that utility won’t be able to be installed.

You can install some software following these instructions: https://www.terraform.io/docs/cloud/run/install-software.html

• Can’t do remote-exec on private instances in your environment:

In case of GCP you would need to create an SSH tunnel to proxy the requests. This would need the gcloud command installed on the machine which is not easy to achieve.

• Can’t really use kubernetes provider on private clusters:

Same reason as local-exec.

• No integration with Vault:

At the moment there is no easy integration between Vault and Terraform Cloud.

Terraform Cloud is still a very new product which means that there are bugs in the code and documentation. You can find a complete list of issues here: https://github.com/hashicorp/terraform/issues?q=is%3Aissue+is%3Aopen+terraform+cloud+label%3Aterraform-cloud.

The list will likely grow as more and more adopt Terraform Cloud.

Conclusion

Terraform Cloud is definitely a game changer in the way companies will use terraform. For the very low price of $20/month/user it can bring a great value to a devops team. As the product starts gaining traction I’m sure that HashiCorp will introduce more and more features. With efficient use of workspaces and integrations with GitHub a lot of companies can stop relying on tools like terragrunt to manage their environments. At the moment I would recommend Terraform Cloud to small startups or small to medium companies that have some Terraform experience.