How we test

How we test CoyoteDB.

We try to break it on purpose. Here is how we do it, what we saw, and what we test next.

A few words first

Acknowledged write
A write the database told your app was saved. These are the ones that must survive.
Power cut
Switching a computer off instantly, with no chance to finish or save anything.
Virtual machine
A computer simulated inside another computer, so its power can be cut on purpose, again and again.
Main node and copies
The main node takes the writes. The copies hold live copies of the data and can take over.

Method

Three ways we try to break it.

Every test runs real PostgreSQL processes, with real writes going in while things go wrong. Nothing about failover is faked.

Real PostgreSQL 18
Not a lookalike. Data checksums are switched on, so damaged pages can be detected.
Synchronous copies
A write is acknowledged only once a copy on another node has it.
Encrypted links
Everything between nodes travels encrypted, over one port.

1 · Power cuts

We pull the plug, mid-write.

We run a CoyoteDB cluster on Linux virtual machines on one Hyper-V host. While a steady stream of writes goes in, we switch a machine off instantly. Whatever it had not yet saved to its disk is gone, just like a real power cut.

We did this 38 times, cutting:

  • The main node
  • A copy
  • The witness
  • All nodes at once
  • A second cut during a switch-over
  • A pulled network cable
  • A frozen node
Inside one host computer: a workload writes to node A, the main node. A's power is cut. Nodes B and C still have a majority, B becomes the new main node, and writes flow again.one host computerworkload+ journalApower cutBnew mainCwrites back

2 · Fault injection

We attack a live cluster, on purpose.

A second set of tests, our fault-injection (or "chaos") suite, runs on macOS and on a real Linux machine. It starts a cluster, keeps writing to it, and then hurts it in as many ways as we can think of, with random timing.

It runs on clusters of two, three and five nodes, with and without a witness. Every change to how CoyoteDB stores data, switches over or routes connections has to pass it.

What the suite does to the cluster
  • Processes killed outright
  • Processes frozen in place
  • Network links cut, both ways and one way
  • A slow network
  • A laptop going to sleep
  • A kill while a node copies the data
  • A kill while an old main node is brought back in line
  • Rolling restarts, one node after another
  • A five-node cluster losing nodes

3 · Checking every write

We write down every answer, then compare.

The workload inserts rows and moves money between bank-style accounts. For every write, it notes in a journal what the database said: saved, failed, or no answer because the connection broke mid-write.

When the faults stop, a checker reads every node and compares it with the journal. Every saved write must be there. No failed write may be there. Every node must hold the same data, and the bank total must not change.

In the fault-injection suite it also runs a linearizability checker, Porcupine, in the style made known by Jepsen. Linearizable means every read saw a value that could really have been there at that moment.

Last, PostgreSQL’s own tools look for damage: pg_checksums checks every data page, and amcheck checks the indexes.

The workload writes to the cluster. When the cluster answers "saved", the workload notes it in its journal. After the faults, a checker compares the journal with every node: every saved write must be there, and every node must be the same.workloadABC1 write2 saved3 notedjournal#1041saved#1042saved#1043no answer4 after the faults: compare✓ every saved write there✓ every node the same

What we saw

The 38 power cuts, in plain words.

Every run had writes going in when the power went. Across all of them:

  • No run showed a lost acknowledged write.
  • At no point were there two main nodes taking writes at once.
  • The bank-style transfers added up in every run checked: no money appeared or vanished.
  • Where the final checks ran, every node ended with identical data.
  • PostgreSQL’s own integrity checks (pg_checksums and amcheck) found no damaged pages or indexes.

These are results from our own test lab. They show what we saw, not a promise about every computer or network.

How long it took

Measured timings from the tests
TestWhat happenedRunsWhat we saw
3 nodesPower cut to the main node10 runsWrites flowing again after about 10 to 15 seconds
3 nodesPower cut to all three at once5 runsBack by itself, writes flowing again in roughly 20 to 35 seconds
Fault injectionOne long run with faults, 34 minutes1 run156,974 acknowledged writes, none lost

Times run from the moment of the cut. The first two rows are from the power-cut runs on virtual machines; the last is from the fault-injection suite.

Recovery, tested

A node comes back

It rejoins and catches up by itself.

The old main node comes back

It returns as a copy by itself, not as a second main node.

A node was away too long

It copies the data again by itself, then rejoins.

Our tests found real problems before any user could, and we fixed them. That is what they are for.

Next

What we test next.

We keep widening the tests. These are the next ones.

Windows
These runs used Linux, and the fault-injection suite also runs on macOS. Windows is next.
Separate physical computers
The power-cut runs used virtual machines on one host. Clusters spread over separate computers on a real network come next.
Longer runs
Runs so far last minutes to about half an hour. Longer runs on real hardware follow.

A plain note

Everything on this page is what we saw in our own tests. It is not a promise about your computers, your network or your data. Keep backups of anything you care about. The site’s terms of use apply.

See how it stays up.

The ideas behind these tests, from synchronous copies to cut-off safety, in plain words. Questions about how we test? Ask us.