Data-Engineering

I spent three months building a data pipeline last year and another two months fixing the things I got wrong. What follows is the architecture I ended up with, and more importantly the decisions that shaped it. Where the data comes from We pull from three kinds of sources. Internal databases, mostly PostgreSQL instances running on our own infrastructure. Third-party APIs for things we don’t control, like payment providers and analytics platforms. And CSV files that land in an S3 bucket whenever someone exports a report from a legacy system that nobody knows how to update. ...

I keep coming back to the same problem: everyone talks about big data as if it’s a thing you have, when really it’s a process you go through. There’s no single tool or platform that solves it. What you end up with is a series of messy stages, each with its own set of headaches, and most organisations I’ve spoken to are stuck somewhere in the middle without a clear picture of what comes next. ...