How we test
How we test CoyoteDB.
We try to break it on purpose. Here is how we do it, what we saw, and what we test next.
A few words first
- Acknowledged write
- A write the database told your app was saved. These are the ones that must survive.
- Power cut
- Switching a computer off instantly, with no chance to finish or save anything.
- Virtual machine
- A computer simulated inside another computer, so its power can be cut on purpose, again and again.
- Main node and copies
- The main node takes the writes. The copies hold live copies of the data and can take over.
Method
Three ways we try to break it.
Every test runs real PostgreSQL processes, with real writes going in while things go wrong. Nothing about failover is faked.
- Real PostgreSQL 18
- Not a lookalike. Data checksums are switched on, so damaged pages can be detected.
- Synchronous copies
- A write is acknowledged only once a copy on another node has it.
- Encrypted links
- Everything between nodes travels encrypted, over one port.
1 · Power cuts
We pull the plug, mid-write.
We run a CoyoteDB cluster on Linux virtual machines on one Hyper-V host. While a steady stream of writes goes in, we switch a machine off instantly. Whatever it had not yet saved to its disk is gone, just like a real power cut.
We did this 38 times, cutting:
- The main node
- A copy
- The witness
- All nodes at once
- A second cut during a switch-over
- A pulled network cable
- A frozen node
2 · Fault injection
We attack a live cluster, on purpose.
A second set of tests, our fault-injection (or "chaos") suite, runs on macOS and on a real Linux machine. It starts a cluster, keeps writing to it, and then hurts it in as many ways as we can think of, with random timing.
It runs on clusters of two, three and five nodes, with and without a witness. Every change to how CoyoteDB stores data, switches over or routes connections has to pass it.
- Processes killed outright
- Processes frozen in place
- Network links cut, both ways and one way
- A slow network
- A laptop going to sleep
- A kill while a node copies the data
- A kill while an old main node is brought back in line
- Rolling restarts, one node after another
- A five-node cluster losing nodes
3 · Checking every write
We write down every answer, then compare.
The workload inserts rows and moves money between bank-style accounts. For every write, it notes in a journal what the database said: saved, failed, or no answer because the connection broke mid-write.
When the faults stop, a checker reads every node and compares it with the journal. Every saved write must be there. No failed write may be there. Every node must hold the same data, and the bank total must not change.
In the fault-injection suite it also runs a linearizability checker, Porcupine, in the style made known by Jepsen. Linearizable means every read saw a value that could really have been there at that moment.
Last, PostgreSQL’s own tools look for damage: pg_checksums checks every data page, and amcheck checks the indexes.
What we saw
The 38 power cuts, in plain words.
Every run had writes going in when the power went. Across all of them:
- No run showed a lost acknowledged write.
- At no point were there two main nodes taking writes at once.
- The bank-style transfers added up in every run checked: no money appeared or vanished.
- Where the final checks ran, every node ended with identical data.
- PostgreSQL’s own integrity checks (pg_checksums and amcheck) found no damaged pages or indexes.
These are results from our own test lab. They show what we saw, not a promise about every computer or network.
How long it took
| Test | What happened | Runs | What we saw |
|---|---|---|---|
| 3 nodes | Power cut to the main node | 10 runs | Writes flowing again after about 10 to 15 seconds |
| 3 nodes | Power cut to all three at once | 5 runs | Back by itself, writes flowing again in roughly 20 to 35 seconds |
| Fault injection | One long run with faults, 34 minutes | 1 run | 156,974 acknowledged writes, none lost |
Times run from the moment of the cut. The first two rows are from the power-cut runs on virtual machines; the last is from the fault-injection suite.
Recovery, tested
A node comes back
It rejoins and catches up by itself.
The old main node comes back
It returns as a copy by itself, not as a second main node.
A node was away too long
It copies the data again by itself, then rejoins.
Our tests found real problems before any user could, and we fixed them. That is what they are for.
Next
What we test next.
We keep widening the tests. These are the next ones.
- Windows
- These runs used Linux, and the fault-injection suite also runs on macOS. Windows is next.
- Separate physical computers
- The power-cut runs used virtual machines on one host. Clusters spread over separate computers on a real network come next.
- Longer runs
- Runs so far last minutes to about half an hour. Longer runs on real hardware follow.
A plain note
Everything on this page is what we saw in our own tests. It is not a promise about your computers, your network or your data. Keep backups of anything you care about. The site’s terms of use apply.
See how it stays up.
The ideas behind these tests, from synchronous copies to cut-off safety, in plain words. Questions about how we test? Ask us.