In Part 1 and Part 2 we have looked at setting up a Ghost blog on Digitalocean and also hardening the HTTPS setup via Cloudflare. In this step, we will look at how we can configure our deployment in a "repeatable" way by using Terraform.

( Photo by Jeremy Bishop )


This article is part of a series


Before we proceed any further in configuring and optimising our blog instance, I thought it would make sense to see if there are ways of making our setup more repeatable. After all, doing all the steps manually is fun and useful - especially if you want to learn and experiment. However, for consistent result it is much better to script and automate as much as possible.

Enter Terraform, an open source tool allowing to create, modify and version infrastructure set up and configuration. If you are coming from an AWS background, this is conceptually very similar to AWS CloudFormation. In short, you define the desired state of your infrastructure, and the tool takes care of executing all the actions required to achieve that state. It all feels a bit like ...


There are of course differences between CloudFormation and Terraform. I would highlight:

  • Terraform is Open Source
  • Whereas AWS CloudFormation is only used in the context of AWS resources, Terraform abstracts away different third parties as "Providers". This means that Terraform is not tied to a specific IaaS/PaaS/SaaS service, but you can deploy your infrastructure as you see fit. For example, there is a provider for Digitalocean, one for Cloudflare (which we will use as an example in this article), as well as AWS , GCP, Azure and many others.

If you are interested in learning more about CloudFormation VS Terraform, I have found this article quite good, although some things have evolved since that has been authored. For example, AWS now offers a (Beta) tool called CloudFormer for importing existing AWS resources into a CloudFormation Template.

In the next section I will show for example how to store the Cloudflare configuration (DNS records and other settings as discussed in Part 2) using Terraform, and automate its management.

Installing Terraform

Installation is fairly simple - just pull the binary / package that is appropriate for your machine. On Windows, it is sufficient to unpack the binary in a location that is then added to your PATH so you can easily refer to it from terminal.

Once that is out of the way, you can verify that your setup is working by issuing the terraform command

terraform
Usage: terraform [-version] [-help] <command> [args]

The available commands for execution are listed below.
The most common, useful commands are shown first, followed by
less common or more advanced commands. If you're just getting
started with Terraform, stick with the common commands. For the
other commands, please read the help and docs before usage.

Common commands:
    apply              Builds or changes infrastructure
    console            Interactive console for Terraform interpolations
    destroy            Destroy Terraform-managed infrastructure
    env                Workspace management
    fmt                Rewrites config files to canonical format
    get                Download and install modules for the configuration
    graph              Create a visual graph of Terraform resources
    import             Import existing infrastructure into Terraform
    init               Initialize a Terraform working directory
    output             Read an output from a state file
    plan               Generate and show an execution plan
    providers          Prints a tree of the providers used in the configuration
    push               Upload this Terraform module to Atlas to run
    refresh            Update local state file against real resources
    show               Inspect Terraform state or plan
    taint              Manually mark a resource for recreation
    untaint            Manually unmark a resource as tainted
    validate           Validates the Terraform files
    version            Prints the Terraform version
    workspace          Workspace management

All other commands:
    debug              Debug output management (experimental)
    force-unlock       Manually unlock the terraform state
    state              Advanced state management

Create a Configuration

The first thing to do is to create a Terraform Configuration (set of files describing the infrastructure). These are essentially regular text files written in HCL (HashiCorp Configuration Language). Let's see a portion of the configuration file for my Cloudflare DNS rules

provider "cloudflare" {
  # Obtained from env vars
  # CLOUDFLARE_EMAIL
  # CLOUDFLARE_TOKEN
}

variable "domain" {
  type = "string"
  default = "paolotagliaferri.com"
}

resource "cloudflare_record" "mx-10-b" {
  domain  = "${var.domain}"
  name    = "${var.domain}"
  value   = "mxb.eu.mailgun.org"
  type    = "MX"
  priority = "10"
}

Here's an explanation of what's going on

provider "cloudflare" {
  # Obtained from env vars
  # CLOUDFLARE_EMAIL
  # CLOUDFLARE_TOKEN
}

The first block is a provider definition. In short, I'm telling Terraform that the contents of this configuration is managed by the "cloudflare" provider. The provider is responsible for knowing how to provision the required infrastructure, in this case it will use the Cloudflare API to set up the required DNS rules and settings.

The comments are merely a reminder to the reader that when we run Terraform with this configuration, the tool will expect two shell variables set with the correct values. These are the Cloudflare e-mail account and API key associated with your Cloudflare account. Because the configuration is meant to be committed to version control, you do NOT want to have these values directly inside the configuration file, as it would represent a major security risk. The API key must be treated as a confidential secret.

variable "domain" {
  type = "string"
  default = "paolotagliaferri.com"
}

Here is a simple variable definition. The Type of the variable is string, although this can be omitted as it's the default, and the value is the name of my domain. I will use this variable below in my resources, to avoid repetitions.

resource "cloudflare_record" "mx-10-b" {
  domain  = "${var.domain}"
  name    = "${var.domain}"
  value   = "mxb.eu.mailgun.org"
  type    = "MX"
  priority = "10"
}

In this block we are specifying a resource. Resources are the building blocks in your infrastructure. In this case "cloudflare_record" is a type of resource supported by the Cloudflare provider, and representing a DNS record. Next to it, there is a name that we can use for reference ("mx-10-b"). Within the resource, we have a full description of the DNS rule (in this case a MX record for 'paolotagliaferri.com' pointing a 'mxb.eu.mailgun.org' for handling mail traffic, with a priority of 10).

Once you have created a similar file, save it in a folder (my-configuration.tf or whatever more descripting filename suits your boat). You can then invoke

terraform init

So that Terraform can read it and initialise the provider and plugins.

Importing existing resources

Next, you can preview what Terraform would do with your configuration - and without performing actual changes yet - by typing

terraform plan

The above step may ask your Cloudflare credentials unless you set them up already as shell variables - see documentation

At this stage, you may notice that Terraform plans to add resources that you know are already existing in your Cloudflare account. For example, I set up the MX rules manually before the tutorial, so I know they are already there. However Terraforms wants to create them again:

terraform plan

...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + cloudflare_record.mx-10-b
      id:                                     <computed>
      created_on:                             <computed>
      domain:                                 "paolotagliaferri.com"
      hostname:                               <computed>
      metadata.%:                             <computed>
      modified_on:                            <computed>
      name:                                   "paolotagliaferri.com"
      priority:                               "10"
      proxiable:                              <computed>
      proxied:                                "false"
      ttl:                                    <computed>
      type:                                   "MX"
      value:                                  "mxb.eu.mailgun.org"
      zone_id:                                <computed>

Plan: 1 to add, 0 to change, 0 to destroy.

This is because Terraform figures out what to do by comparing the configuration with an associated, persistent state that is (in our case) saved on disk and represents the running infrastructure. As far as Terraform is concerned, we have never applied a plan before and there is no saved state: therefore, these are brand new resources that must be created.

Fortunately, there is a way to import existing resources using the import command. This allows us to map named resources that are part of the configuration with actual, existing resources that are already deployed on Cloudflare. In this way, Terraform can then import the existing resource configuration from the provider (via the Cloudflare APIs in this example), and save it in the state. This means it knows about it going forward.

For example, assuming my existing MX record for 'paolotagliaferri.com' was already available in Cloudflare with a Cloudflare resource ID of 'd7feab5ab5218a91e38a9a88216f0b90' (which I just made up...) - then I would issue the following command

terraform import cloudflare_record.mx-10-b paolotagliaferri.com/d7feab5ab5218a91e38a9a88216f0b90

And let Terraform fetch the state of my existing resource and persist it locally. Running a terraform plan command again would most likely remove any addition/changes associated to that MX record, as Terraform now knows about it in its local state. If however that MX record is configured differently than what stated in your configuration, then it would apply such changes to align it with your configuration.

Note: in order to figure out the ID of your Cloudflare resources, you will need to use the Cloudflare APIs, in this case:

This is definitely not immediate / straightforward and requiring some fiddling, but you can get going pretty quickly with Postman and the Cloudflare API Postman collection.

Of course, if you were to define a configuration upfront before deploying anything for real, then you wouldn't need to import. You can see how useful import can be for real projects where some infrastructure is already existing and you still want to bring it under Terraform management.

‼️ UPDATE 15 Feb 2019: ... and literally 3 days after I wrote this article, Cloudflare has introduced Cf-Terraforming - which is an open source utility (github) for importing existing configurations into Terraform compliant configurations. It's even easier now to move your configuration into Terraform ‼️

Setting up other Cloudflare configurations

In Part 2 we reviewed some best practice configurations (such as enabling the experimental use of TLS v1.3, or Strict SSL) with your Cloudflare setup. Now it's time to add these to your newly made Terraform configuration, for versioning purposes and general awesomeness!

resource "cloudflare_zone_settings_override" "paolotagliaferri-com-settings" {
  name = "${var.domain}"

  settings {
    always_use_https = "on"
    tls_1_3 = "on"
    ssl = "strict"
    min_tls_version = "1.2"

    ...
}

The latest addition to our configuration (of which I include an excerpt) is a new resource called cloudflare_zone_settings_override . This is meant to give you a mechanism to tweak the zone settings that you can normally modify via the Cloudflare web interface, or programmatically using the API ).

Once everything looks like what you would expect, you can issue another

terraform plan

command, review any changes one last time, and then

terraform apply

which will use the Cloudflare APIs to modify the setup to match your described configuration. Neat!

Don't forget to commit your working Terraform configuration file in version control, and manage updates in that way going forward.

Final Considerations

The above is just a quick example showing the versatility of this approach.

This makes sense when working on a solo project, and it makes exponentially more sense when thinking about a team project with multiple infrastructure resources, spread across a number of different providers and several people updating it.

Things to further experiment with, perhaps in future articles:

  • Add the Digitalocean provider to manage the provisioning of the Droplet. This would be particularly useful once the infrastructure grows beyond the lone, one-man-band blog droplet and we need to think about multi-regional, load-balanced deployment.
  • Remote state: as mentioned, the state is stored on disk by default. This is already showing limits in a single developer scenario, for example when you switch machine, then checkout your Terraform configuration. Suddenly it thinks that everything must be provisioned again! As a rule, you shouldn't commit the state file in version control, but rather follow best practice and set up backends / Remote State. This is a must have in multiple developer scenarios (so yeah, any project that is not your pet, stealth project developed in the depths of your man cave / she shed).
  • Provisioners: these can be used to execute scripts locally or remotely within the resource life-cycle.  For example using the File provisioner to deploy specific files on a newly created VM.

Thanks for reading!