Featured Image copied from Wikipedia’s Mars Transition.

Okay, so yes we’re still on the HashiCorp stack – I feel I should say that I have no financial attachment, gain or otherwise to HashiCorp. Just a fan of great tooling 🙂

This time it is Terraform.

In using Terraform we have moved up the food chain because it allows us to define the very infrastructure we need to run our applications on; e.g. Servers, Firewalls, SSL Certificates, Load Balancers, in our case DNS Managed Zones plus a whole lot more. It truly is powerful … if not quite at the level of having a version controlled breathable atmosphere for the Red Planet! We’ll get there I’m sure … baby steps.

In a nutshell …

Terraform = Immutable infrastructure defined using an easy, declarative language.

Once defined an entire environment can be spun up from a standing start, run tests against it and then have it destroyed. That’s convenience for you. And obviously saves a lot of money. Using this for side projects is especially economically efficient – I can make changes locally on laptop, test until happy, then spin up a cluster of machines for exploratory testing and once I’m happy ( or tired 😫 ) just destroy all the infra exactly at the moment I’m done. Even just writing that down makes somewhat giddy.

But more than anything there is a much higher level of confidence as the changes permeate each environment; dev => test => uat => prod; because configuration drift is much, MUCH, less likely.

For a proper introduction as to what Terraform can do, reading the official documentation will help for sure, but if you are looking for something a little more hands on then I would strongly recommend Yevgeniy Brikman’s Terraform Up And Running book which actually has almost literally just been printed, previously having been in O’Reilly’s Early Release program. The book adds more semantics and best practices over the dryer online documentation.

But today the focus is narrower. Specifically the assumption is that we own a domain, lets say example.com, and wish to create a dns sub-domain called dev.example.com which will be used for development purposes. This is desired because it makes us feel happy and content; if changes to any attributes are required then we simply go to the same version control system for that as we do for the background colour of our website or business logic for services – it’s all there! For me this is a similar feeling that David Allen refers to when “closing open loops” in his Get Things Done book.

In a later article we’ll add a DNS record set to point to our Kafka server as in previous posts and that particular circle will be complete.

Warning: £££’s, $$$’s, €€€’s and other fiat moneys may be taken from you

Usual warning here. When using cloud infrastructure you may be charged depending on your provider, trial status etc. As ever we’ll clean up after ourselves so typically should only cost a nominal amount but it is important to understand your service providers prices.

And again we’ll be using Google Cloud for demonstrative purposes.

Setup

Download Terraform for your OS and setup.

The tiny amount of code is on the github repo.

As mentioned we’re assuming Google Cloud usage here as per previous posts.

DNS Record Set

If you have your own domain then you current root DNS record set may look something like …

We want all our code for development to be under version control for all the good infrastructure-as-code reasons, but also in this case somewhat more pragmatically, if you have used Terraform to create a DNS Managed Zone then you can reuse those values in other Terraform projects. Specifically, in a later post, we will use the DNS Name ( dev.example.com ) output from today’s work, to build a fully qualified server name within that sub-domain.

The simple steps we will follow are:-

1. Create a new DNS Managed Zone – dev.example.com
2. Manually update our root DNS record (above) to use the sub-domain DNS Servers
3. And done – well for this article at least.

Less than ideal here, this quick solution does mean that anytime we destroy and then recreate the Dev DNS Managed Zone we will have to manually update our root DNS Record Set. A few solutions we could do here. Either put root in version control ( not always ideal or possible ), or you could have an intermediate Managed Zone – e.g. infra.example.com so we would then have [dev|test|uat|prod].infra.example.com which would then mean dropping and recreating the env specific infrastructure would be entirely automatable – good times. This would leave the manual worry at the infra.example.com level which you would have to assume would be extremely rare.

A word on Terraform State

Terraform has to store the infrastructure state somewhere. For more details see the docs. By default this is stored in a file in the local directory called “terraform.tfstate“. You can store the state remotely as mentioned in the docs ( at a cost regardless of option selected ) – you would use a remote option in production. For us, we’ll just specify the file location to store the state rather than have it in current dir. If you get into the habit of at least specifying the state file location then you won’t get confusing answers about your infrastructure if you do the terraform command in different directories … been there 🙂

To apply our Terraform file definitions to our infrastructure provider we use the terraform apply command, and we will specifically use the -state= flag as defined in the apply command.

Setup Terraform Google Credentials

When using terraform you need to specify your provider ( Google in this case ) and the credentials, we will use credentials file. See instructions on how to get this.

Now in your dev directory create a file called main.tf and should reflect where your credentials file lives, your project id and the relevant region. In terms of the documentation this is taken from the Google Provider section;

provider "google" {
  credentials = "${file("/some/path/to/brownian-motion-driven-dev.json")}"
  project     = "brownian-motion-driven-dev"
  region      = "europe-west1"
}

Now lets create the entry for our managed zone – the point of the whole exercise. Create another file called dns-zone.tf and its contents should read as follows, changing example.com to your own domain. This is taken from the Google DNS Managed Zone doc …

resource "google_dns_managed_zone" "dev" {
  name     = "dev-example"
  dns_name = "dev.example.com."
  description = "Example Dev DNS Managed Zone"
}

output "dev_name_servers" {
  value = "${google_dns_managed_zone.dev.name_servers}"
}

output "dev_dns_name" {
  value = "${google_dns_managed_zone.dev.dns_name}"
}

output "dev_dns_zone_name" {
  value = "${google_dns_managed_zone.dev.name}"
}

Cool. This is pretty simple but lets break it down.

The main section is the resource one, which is followed by 3 output sections which are entirely informational. You can read the documentation for more detailed specifics, but basically the dns_name in the resource section is us defining the new DNS Managed Zone we want, in this case dev.example.com.

The output sections echo the values contained within. So the first 2 are just echoing the values we supplied, but the last is interesting because we have no control over which nameservers will be authoritative for our new managed zone, this provides a mechanism to find out any such calculated values.

For a little consistency you can create a new bash script ( or just follow the commands ) called say, createDnsZone.sh, with contents;

#!/usr/bin/env bash

export TERRAFORM_STATE_DIR=${HOME}/.terraform
mkdir -p ${TERRAFORM_STATE_DIR}

terraform apply -state=${TERRAFORM_STATE_DIR}/terraform.tfstate

and while we’re here create the mirror destruction one destroyDnsZone.sh

#!/usr/bin/env bash
export TERRAFORM_STATE_DIR=${HOME}/.terraform
mkdir -p ${TERRAFORM_STATE_DIR}

terraform destroy -force -state=${TERRAFORM_STATE_DIR}/terraform.tfstate

Now lets run createDnsZone.sh
You should see something like,

And you see the outputs, specifically the dev_name_servers are Google’s “c” nameservers in this instance.

Lets run destroyDnsZone.sh; createDnsZone.sh
And this time I see …

… which shows the “b” name servers are now authoritative. The point is there is no control over the nameservers assigned, but need to use these values to join up our infrastructure.

At the moment although the authoritative DNS servers for dev.example.com have now been established – “authoritative” is a slight misnomer at this point since the whole infrastructure is not yet fully configured, or joined up. Specifically if a server was created at this point under this domain, someserver.dev.example.com say, then it will not be reachable yet via DNS. This is because root DNS Record Set is required to delegate responsibility for that specific sub-domain from itself; meaning that if I ping someserver.dev.example.com there is no route to it via DNS because the root Records would try to lookup such a server known by its own nameservers … which of course wouldn’t exist. In short the ping would get to the root DNS Nameservers, but not to the sub-domain’s nameservers which are the ones which know where someserver lives ( IP Address ). Our problem is that we need to tell the root DNS Record to pass any requests to dev.example.com onto the NameServers specifically assigned to our sub-domain.

So in Google Cloud we just navigate to Networking => Cloud DNS, i.e.

The simply add an NS ( NameServer ) entry matching the new sub-domain. So in our example.com that would be similar to …

Once committed it will take some time for the DNS changes to permeate the internet, up to 24 hours perhaps.

It is this manual step above that we can fix by creating a infra.example.com domain as above. If the DNS entry for infra.example.com is in version control and dev.infra.example.com domain hangs off that, then any destroying and recreating of dev.infra.example.com can be done without manual intervention.

Cleanup

Just run destroyDnsZone.sh to clean up any surviving, unwanted zones. This assumes you don’t want to play with your new sub-domain. Zones are not very expensive to leave lying around for a few days anyway. For Google Cloud can see pricing here but looks like around 20 cents per month per zone at the time of publishing.

Up Next

We will link our trusty Kafka server from previous articles and give it a nice DNS name, thus joining up our learnings to create a nice building block for later adventures.