How I set up a data pipeline that actually works

Fri, 01 Mar 2024 08:16:26 +0000

I spent three months building a data pipeline last year and another two months fixing the things I got wrong. What follows is the architecture I ended up with, and more importantly the decisions that shaped it.

Where the data comes from

We pull from three kinds of sources. Internal databases, mostly PostgreSQL instances running on our own infrastructure. Third-party APIs for things we don’t control, like payment providers and analytics platforms. And CSV files that land in an S3 bucket whenever someone exports a report from a legacy system that nobody knows how to update.

Data-Pipeline on Shafiq Alibhai

How I set up a data pipeline that actually works

Where the data comes from