Why I built a database from scratch

Some evenings you finish work and you want to build something that isn't work.

That was the mood I was in. Not another CRUD service, not a tutorial where someone has already made every decision for you. Something that would take months and leave me understanding a thing I use every day.

I went looking, and I found John Crickett's coding challenges. It's a long list, and somewhere in it is "build your own memcached". I've always been curious about databases, so that's the one I took.

A year later it's called BeeDB. Three machines, my own implementation of Raft, a write-ahead log, snapshots, and a demo page where you can kill the leader and watch the other two carry on. This is the story of how a weekend challenge turned into that.

The first version was easy, and that was the problem

memcached is a cache with a text protocol. You open a socket, you type set greeting 0 0 5, then hello, and it says STORED. You type get greeting and it hands the value back.

Underneath, it's a hashmap. The interesting part is the protocol: small enough to read in a sitting, strict enough that sloppiness shows up immediately.

About a week of actual work and it was done. It worked. I could telnet into it.

And then I sat there thinking: okay, but that's one program on one laptop. If I unplug it, everything's gone. What do real systems do when one machine isn't enough?

What I already had a crush on

At work I use ScyllaDB, and at some point I'd gone down the rabbit hole of reading how it works inside. What got me was the speed, reads and writes that fast, from a database spread across several machines, felt like a magic trick.

Then I looked at what it does when two people write the same field at the same moment. It keeps the one with the later timestamp, and the other write is just... gone. Nobody is told. The client that sent it was told everything was fine. That is last-write-wins doing exactly what it says, on that one field, and it is a deliberate design choice rather than a flaw.

I understand the reasoning. That design is built to stay up, and staying up is worth a lot. But I kept poking at it and couldn't get comfortable, and around then I read about Raft.

Raft gives you one thing: a replicated log where the committed part is the same everywhere, in the same order. Nodes can differ at the tail, where entries are not committed yet, and the protocol cleans that up. What it will not do is let two nodes commit different things at the same position, so there is never a second version of events to reconcile afterwards.

And I wanted to build the kind of system that makes the opposite promise from Scylla's. If my database says STORED, I want that to still be true after someone yanks a power cable, as long as a majority of the nodes survive or come back with their disks intact. If it can't promise that, I want it to say no instead. That is the harder promise, assumptions and all, which is exactly why I wanted to try it.

The months where nothing made sense

I'd love to say I designed it and then typed it in.

What actually happened: I'd understand every word in a sentence about Raft and still have no idea what a node was supposed to do when a message arrived. Term, log index, commit index. I could recite them and I couldn't use them.

What got me through was breaking things. Start three nodes, kill one, watch the logs disagree. Add a print statement. Kill a different one. Watch two nodes both believe they're the leader and realise I'd misread one line of the paper. Then read that paragraph again, and this time it means something, because I've seen what happens when it's wrong.

I asked a lot of questions along the way: of the paper, of my own logs, and of an AI assistant that never got bored of "but why?". I'd still be stuck somewhere around leader election otherwise.

The order I built things tells you what I knew

Nobody handed me a plan. Each piece exists because the one before it stopped being enough.

First, single-node memcached: sockets, commands, a cache.

Then Raft: elections, replication, terms. The long part.

Then snapshots, because a log that only grows eventually eats the disk, so at some point you save the current state and throw the old entries away.

Then saving the term and the vote to disk, because a node that forgets who it voted for can vote twice in the same election, and then you've got two leaders.

Then the write-ahead log: every write hits a file and gets fsynced before the client is told anything.

Read that list again. The write-ahead log came last. For months I was carefully compacting and snapshotting data that a power cut would have erased. That's the kind of thing you only see once you've understood the thing you were missing.

What does it need before anyone will try it?

After every step I asked the same question: what's the minimum this needs before someone other than me could use it?

Kept: consensus, the write-ahead log, snapshots, compaction, and reads and writes that behave like memcached.

Skipped on purpose: protobuf and gRPC, keeping several replication batches in flight at once, and linearizable reads.

That last one deserves honesty, because it's the most interesting thing BeeDB still can't do. Reads are answered by whichever node you asked. So this can happen:

1. You write x = y. The leader says STORED.
2. You immediately read x from a different node.
3. You get nothing.

Nothing is lost here, and the two things are worth keeping apart: the write is committed and durable on a majority of disks, and the node you asked simply hasn't applied it to its cache yet. A stale read is a freshness problem, not a durability one. The value appears a moment later.

But it is still a real limitation, and it has a name. Linearizability is a property of the whole history of operations: every read and write has to look as if it happened at a single instant, in an order that matches the order things really happened in. A read that misses a write which finished before it started breaks that, even though nothing was lost.

I know the fix. Before answering a read, the leader notes its commit index, checks with a majority that it's still the leader, waits until it has applied everything up to that index, and only then answers. It's called ReadIndex. It isn't built yet, and I'd rather say that than pretend.

The part that wasn't in the plan

The plan was: push it to GitHub, done.

Then I thought about it honestly. Who's going to clone a Java project, start three nodes, and telnet into them to find out whether my election code works? Nobody. I wouldn't.

So I built the thing I never intended to build: a gateway in front of the cluster, with a live stream of what every node is doing, and a chaos switch that kills the leader on demand. The demo page that shows all this, I built with Claude. What's underneath it, the cluster, the gateway, the killing, is mine.

Write a key, kill the leader, watch the two survivors elect a new one and keep serving, in about a second.

The rest of this series is that demo, taken apart. Why one machine isn't enough. What Raft is actually doing. What happens to one write from your keyboard to the disk. And what happens when I kill the leader while you're typing.