Release Engineering

I’ve been thinking about how we get changes from a laptop into production without someone having to SSH in and type things by hand. The answer, as it turns out, is a pipeline. Nothing fancy. Just a chain of steps that each do one thing and pass the result to the next.

Here’s what we’re building.

Git

Everything starts in a Git repository. Puppet manifests, config files, deployment scripts — all of it. If it’s not in Git, it doesn’t exist. This is the part that matters most and the part people resist the most, because Git means your changes are visible.

Commit and Push

You make a change locally, commit it, and push. That push is the trigger. Nothing happens until something gets pushed. No manual deploys, no “I’ll just quickly change this one thing on the server.” The push is the only door in.

Jenkins

Jenkins watches the repository. When something lands, it runs the tests. Syntax checks on the manifests, style checks on the configs, the whole lot. If anything fails, the pipeline stops and we know about it.

We send notifications through Jabber and email. If a build breaks, the relevant people find out before they’ve had their second coffee.

Capistrano

Tests pass, so Jenkins hands off to Capistrano. It pulls the new manifests and config files and pushes them to the Puppetmaster. One command, one destination, no ambiguity about what’s on the server and what’s not.

MCollective

With the Puppetmaster updated, MCollective fires off a Puppet run across all the servers. Every node picks up the new state at roughly the same time. We get consistency without having to log into each box individually.

Why This Matters

The point isn’t the tools. It’s that the whole thing is repeatable. Anyone on the team can deploy, and they’ll get the same result. There’s no tribal knowledge about which server needs which file edited in which order. There’s no “I forgot to restart that service on node three.”

It also means failures are cheap. If a deployment breaks something, you revert the Git commit and re-run the pipeline. Five minutes, no panic, no 3am page.

The pipeline is boring. That’s the point.