I’ve been thinking about continuous delivery a lot lately, and the thing that keeps coming back to me is how much of a pain it is when someone logs into a server and makes a change by hand.
You know the type. Production is down, someone jumps in, tweaks a config file, restarts a service, and everything works again. Great. Except now that server has a configuration that exists in no repository, no documentation, and no one else’s head. Next time something breaks, you’ll spend hours chasing a difference that nobody bothered to record.
I’ve seen it happen more times than I care to count. A log rotation gets disabled because someone needed more disk space for a quick debug. A firewall rule gets opened “just for today” and stays open forever. A dependency gets installed directly on the box because the package manager was being difficult. Each one of these seems harmless in the moment. Together they’re what people call “configuration drift”, which is a polite way of saying your environment has become a snowball of undocumented decisions rolling downhill.
The fix isn’t complicated in theory. Everything that touches production should go through the pipeline. Code, configs, scripts, the works. If it can’t be deployed through the build, it shouldn’t exist in production. Period.
In practice, getting there is the hard part. You need people to trust the pipeline enough to stop keeping their favourite tricks in a text file on their desktop. You need the pipeline to be fast enough that it doesn’t feel like a bottleneck. And you need to handle the cases where something genuinely needs to change outside the normal flow — and even then, that change should be captured and reviewed, not just applied and forgotten.
The payoff shows up in the boring moments. The 3am page that turns out to be a known issue you can fix by redeploying a version you already tested. The new developer who can spin up an environment in an hour instead of a week. The deployment on a Friday that doesn’t make everyone nervous.
It’s not glamorous work. But it’s the kind of thing that separates teams who ship confidently from teams who ship cautiously and hope for the best.