An ETL pipeline on GitHub, no platform bill

In another article I wrote that I advised an SME against Fabric and that we solved it with a hosted database and Python. That stayed too vague. Here is what that chain looks like in concrete terms.

The four pieces

A script that pulls data from the source. With us that is Python, calling the ERP's API to bring in invoices, journal entries and analytical lines.

A database to put it in. We use Neon, a hosted PostgreSQL. You create a project there in minutes, and for the volume of an SME you stay in the cheap tier.

A place where the processing happens. That is SQL in that same database: turning the raw tables into facts and dimensions a report can work with.

And something that runs it on time. For that we use GitHub Actions.

Power BI eventually looks at the prepared tables in the database, in import mode.

Why GitHub Actions does the scheduling

This is the part where people look up in surprise, because they know GitHub as a place for code, not as a scheduler.

But an Action is simply a computer that runs your script at an agreed moment. You write down in a file when it should run and what it should do. Your code is already there, because you store your scripts somewhere anyway. And the secrets, your API keys and your database password, go into the secrets of your repository. Those are encrypted, they never end up in your code, and the script reads them as an environment variable.

That last point matters more than it sounds. The most common way a key leaks is that someone puts it in a file that later gets shared.

For a public repository this is free. For a private one there are minutes included in your subscription, and a fetch script that runs a couple of minutes per go stays comfortably inside them.

What you get back for it

A log of every run. You see when it ran, how long it took and what went wrong. That is more insight than most people have into their current export process.

An alert when it fails. That is the most important function of the lot, because a connection that stops quietly is more dangerous than no connection. You end up looking at last week's numbers assuming they are today's.

And a history of changes. If your fetch script has been modified and something no longer adds up, you can see exactly what changed and when.

Where it is a poor fit

Let me stay honest, because this is no miracle cure.

It is not real time. You run on a schedule, and if you have to be current to the minute, you need something else.

It is not made for very heavy processing. An Action has limited capacity and a time limit per run. For an SME with invoices and journal entries that is plenty, for millions of rows per hour it is not.

And there has to be someone who understands what is written there. That is the real cost. You trade a monthly license for a piece of your own setup that needs maintaining. If nobody in your organization can do that and you do not want to hire anyone for it either, then a platform with an invoice attached is maybe the more honest choice.

A detail you will run into

Do not schedule your run on the hour. Everybody does that, which makes the queue longest at exactly that moment and starts your job later than you think. Put it on a random minute.

And do not forget to switch off your schedule when the source disappears. I once left a chain running against a test environment that was due to disappear on an agreed date, and switching it off has been sitting on my own task list ever since. A pipeline running against nothing sends a neat error message every day until someone has had enough of it.

The claim

The question is not whether you can afford a data platform. The question is whether you need a data platform before you know how much data you actually process. For most SMEs the answer is no, and then the cheap route is also the fastest.