I’ve been working on octohost lately, updating and upgrading some of the components. One of the things I’ve been looking for a chance to play with has been Consul watches - and I think I’ve found a great use for them.
As background, when you git push to an octohost, it builds a Docker container from the source code and the Dockerfile inside the repository. Once the container is built and ready to go, it does a few specific things:
- It grabs the configuration variables stored in Consul to start the proper number of containers.
- It registers the container as providing a Consul Service.
- It updates the nginx configuration files so that nginx can route traffic to the proper container.
Last week I updated octohost to improve those steps for a few reasons:
- If we changed any of the configuration variables, we had to manually restart the container before it would be picked up.
- If a container dies unexpectedly, we weren’t automatically updating the nginx configuration to reflect the actual state of the application.
- Our nginx configuration file was being built by a gem I created and wanted to retire in place of Consul Template. The monolithic file it generated was very inflexible and I wanted to make it easier to update.
For #1, when a site is pushed to octohost, I’m registering a “watch” for a specific location in Consul’s Key Value space - octohost/$container-name. That kind of watch looks like this example:
{
"watches": [
{
"type": "keyprefix",
"prefix": "octohost/html",
"handler": "sudo /usr/bin/octo reload html"
}
]
}
We’re telling Consul to watch the octohost/html keys and anytime they change, to run sudo /usr/bin/octo reload html. As you can imagine, that reloads the container. Let’s watch it in action:
Pretty nice eh? You can add keys or change values and the watch knows to run the handler to stop and start the container.
NOTE: Before version 0.5, deleting a key doesn’t do what you’d expect, but the Consul team knows about this and has posted a fix.
NOTE: This has been disabled because of this issue: 
For #2 and #3, we look at the Consul Service catalog we are populating here and register a different type of watch - a service watch. An example service watch looks like this:
{
"watches": [
{
"type": "service",
"service": "html",
"handler": "sudo consul-template -config /etc/nginx/templates/html.cfg -once"
}
]
}
We’re telling Consul to watch the html service and if the status changes, run the consul-template handler. This handler rebuilds the template we are using to tell nginx where to route container traffic. Let’s watch that handler in action:
All of that was done by the Consul watch - it fires whenever it detects a change in the service catalog - I didn’t have to do anything. I even killed a container at random, and it removed it from the configuration file auto-magically.
Consul watches are pretty cool. If you’re adding one to your Consul cluster, remember a few things:
- I’ve used separate json files for each watch. That can be done because we’re telling Consul to look in an entire directory for configuration files, the -config-dir option.
- When you add a new file to the config-dir, you need to tell Consul to reload so it can read and activate it. If there’s a syntax error or it can’t load the watch, it notes that in the logs - so keep an eye on them when you’re doing this.
- As of this moment and because it’s brand new software, Consul Template can only do a single pass to populate the values - so the templates need to be pretty simple. We have worked around those limitations by doing our own first pass to pre-populate values that are needed. Thanks to @bryanlarsen and @sethvargo who discussed a workaround here.
I think I’ve just scratched the surface with how to use Consul watches effectively and they have helped to simplify octohost. I’m looking forward to finding new and better uses for them.
NOTE: A special shout out to Armon, Seth, Mitchell and the rest of the crew at Hashicorp for some great software that can be twisted to further my plans for world domination.