Blog

What "high availability" actually means for a small app

A plain guide to keeping a database up: what happens when a computer dies, how failover works, why three machines vote and why two need extra care.

The CoyoteDB team6 min read

“High availability” sounds like something only big companies need. It is not. If your app stops when one computer stops, and that matters to you, this post is for you. We define each word as we go.

Start with one computer

Most small apps keep their database on one computer. That computer might be a PC in the back office, a mini PC in a cupboard, or a small server you rent.

Now picture it dying. The power supply pops. The disk fails. Windows decides to restart. Someone trips over the cable.

Two things happen:

  1. The app stops. Nothing can read or save data until that computer is back.
  2. Recent data may be gone. If you restore from last night’s backup, everything saved since then is lost.

Backups fix the second problem, at least partly. They do not fix the first. A backup helps you get the data back later. It does not keep the app running now.

High availability means the database keeps answering when one computer stops. Not “it never breaks”. It means “when one computer breaks, the database carries on without it”.

Copies on more than one computer

The first step is a live copy. A second computer holds a full copy of the database and keeps it up to date, change by change. In Postgres this is called replication.

There is a detail that matters a lot here: when does a save count?

If the main computer says “saved” before the copy has the change, a crash at the wrong moment loses that save. The app was told it was saved, but it was not saved anywhere that survived.

CoyoteDB answers that with a simple rule: a save counts once two computers have it. When a second data node is up, which is always the case in normal running, CoyoteDB waits until it has a copy before it says “saved”. So losing one computer does not lose that write.

Failover: someone has to take charge

A copy on its own is not enough. When the main computer dies, one of the others has to become the new main computer and start taking writes. That switch is called failover.

The tricky part is deciding who takes over, and when. Get it wrong and you get the worst outcome in this whole topic: two computers that both think they are in charge, both taking writes. People call that split brain. Your data splits into two versions that cannot simply be merged back.

So failover needs a way to agree.

Why three machines vote

CoyoteDB’s computers (we call each one a node) vote on which one is the main node. To be in charge, a node needs more than half of all the votes.

With three voters, two votes win. That rule is what makes automatic failover safe. There can only ever be one group with more than half the votes, so there can only ever be one main node.

When the main node stops, the two that are left still have two of three votes between them. That is a majority. They vote, one becomes the main node, and writes carry on. In CoyoteDB’s tests, a new main node was in place about 10 seconds after a hard kill (6 to 11 seconds), and writes flowed again in roughly 10 to 25 seconds (measured on one Mac, default settings).

There is one more rule. If a node loses touch with the others, it shuts its database down before another node takes over. A node cut off by a broken network cable cannot keep taking writes on its own. Two nodes never both think they are in charge.

Why two machines need care

Two computers sounds like enough. Each has a full copy. But think about what each one can see.

With two nodes, each has one vote of two. If they lose touch, each side sees exactly half. Never more than half.

From the inside, “the other computer died” and “the cable between us broke” look exactly the same. If each side took over on its own, a broken cable would give you split brain. So with two computers and no witness, the other computer does not take over by itself. Your data is safe on it, but the database is down until the main node is back.

The fix is small. Add a witness: a tiny helper that holds no data and only votes. Now there are three voters. If one data node stops, the other data node and the witness still make two of three, and switch-over happens on its own.

A witness can run on almost anything small that stays on: a Raspberry Pi, a mini PC, an old laptop that stays plugged in and awake, or a cheap cloud server. It can run a different system from the data computers. If you can, put it somewhere else, so one power cut or one router failure cannot take out the majority.

A simple way to remember it: keep the number of voters odd.

Setup Copies of your data Switch-over One computer down
1 computer 1 None The database stops
2 computers 2 None when one dies Your data is safe on the other; the database is down until the main node is back
2 computers + a witness 2 Automatic The other computer takes over
3 computers 3 Automatic Takes over, and saves still land on two computers

What is protected, and what is not

It helps to be exact about what you get.

Protected: a dead computer. A save that CoyoteDB has acknowledged is on two computers, so losing one does not lose it. In CoyoteDB’s chaos tests, which kill, freeze and cut off nodes on purpose, a checker confirmed that no acknowledged write was lost and that there were never two main nodes at once.

A short pause, not a long outage: during failover, connections to the old main node drop, and another node becomes the main node by itself, as described above.

Running on one copy: if only one data computer is left, saves pause for about 30 seconds, then carry on using the one node that is left. Until the other is back and caught up, there is no second copy.

Not protected by copies: mistakes. If someone deletes a table, the delete is copied to every node at once. That is what backups are for. Keep them as well.

Not what it is for: spreading one database across regions or around the world. CoyoteDB is built for a few computers on one private network, such as a LAN, Tailscale or a site-to-site VPN between a few sites.

So what should you run?

For trying things out, 1 computer is fine: it is real Postgres, just not highly available. For most people we recommend 2 computers plus a witness. When the data really matters, use 3 computers, so a save still lands on two computers even with one down. Either way, the data computers should be the same system type, stay switched on, and sit on a network where they can always reach each other.

Where to go next

  • How it works shows sync, voting and cut-off safety as diagrams.
  • Setup compares the recommended setups side by side.
  • Node counts explained goes a little deeper on how many computers you need.

If you have a setup in mind and are not sure which one fits, ask us.