Engineering

I used to take on any project that came my way. That was a mistake. Not every problem deserves your time, and not every client is worth working with. Over the years I’ve learned to filter opportunities through a simple set of questions before committing to anything. Can they fix it themselves? If the answer is yes – or if a decent blog post could solve it – then it’s not a consulting engagement. It’s a homework assignment. The value of consulting lies in the gap between what the client knows and what they need to know. Close that gap and you’ve got nothing to sell. ...

I spent three months building a data pipeline last year and another two months fixing the things I got wrong. What follows is the architecture I ended up with, and more importantly the decisions that shaped it. Where the data comes from We pull from three kinds of sources. Internal databases, mostly PostgreSQL instances running on our own infrastructure. Third-party APIs for things we don’t control, like payment providers and analytics platforms. And CSV files that land in an S3 bucket whenever someone exports a report from a legacy system that nobody knows how to update. ...