BeeDB: the architecture

This one is for people who build these things. If you want the story instead, start with why I built a database from scratch.

BeeDB is a memcached-compatible key-value store on a hand-written Raft implementation: three Java nodes, a Spring Boot gateway, and a browser demo where a visitor can kill the leader and watch what happens. No Netty, no jRaft, no framework doing the interesting part.

The system at a glance: three Java nodes on one VM, hand-written Raft, a Spring Boot gateway in front, the memcached text protocol on the wire, a write-ahead log with snapshots underneath, and a browser demo with a chaos button that kills the leader on request.

The design is shaped by one awkward constraint. It has to be a public demo that strangers are invited to break, running on one small VM, while still being honest about durability. Most of what follows falls out of that.

The shape of it

Everything lives on one machine, which is a deliberate limitation rather than a claim. Three nodes on one host survive a process dying, not a machine dying, and the post on what BeeDB doesn't promise says so out loud.

Why a gateway exists at all. The nodes speak the memcached text protocol on a socket. A browser cannot, and should not: that protocol has no auth, no rate limiting, and its error for "wrong leader" is the string Not Leader: node2, which is useless to anyone who doesn't already know where node2 lives. The gateway is the thing that knows the topology, so it can catch that and retry against the real leader.

It also absorbs the demo's hostile parts: per-session write quotas, a fixed-window rate limit, a chaos endpoint that kills a node on request, and one SSE stream carrying the cluster's state.

nginx does three specific jobs, and two of them are about SSE:

• /api/events gets proxy_buffering off and a 3600s read timeout. Without the first, nginx holds your events and the live view stutters; without the second, the stream dies on a timer.
• /api/ gets limit_req with a burst of 20, nodelay. The SSE location is excluded, because a rate limiter that counts requests would count one long-lived stream once and then never again, which is exactly the wrong shape.
• Everything else is static files for the demo page.

systemd carries two decisions worth stealing. The nodes are one templated unit ([email protected]), so node 1, 2 and 3 are the same file with different arguments, each with its own port and WAL directory. It restarts on failure after 2 seconds, but it sets RestartPreventExitStatus=2: exit code 2 means "your configuration is wrong", and restarting cannot fix a typo. And the gateway's unit has After= the three nodes but deliberately not Requires=, because the gateway is built to start against a cluster that isn't up yet and report that honestly, rather than refuse to boot.

Inside one node

Nine kinds of thread, and each one exists because something must not block something else.

The framing rule. TCP gives you bytes, not commands. Each connection owns a growable buffer, and nothing is parsed until the command line and its declared byte count have both arrived. Exactly one command per connection is dispatched at a time, because memcached clients match replies to requests by order: parallel dispatch would corrupt sessions under load in a way that looks like data corruption and isn't.

One lock, and the rule about it. The Raft state (log, term, vote, commit index, per-peer match indexes) sits behind the node's monitor. The rule learned the expensive way: never hold it across an fsync. An earlier version waited on the WAL inside the lock, which cost 5.9x throughput and made the node look dead to its peers for milliseconds at a time, triggering elections nobody needed.

The durability invariant is one line and it is the most important line in the project:

commitIndex = Math.min(majorityMatchIndex, persistedWalIndex);

Both terms need naming, and the first one is easy to describe loosely and get wrong.

majorityMatchIndex is an order statistic over the followers only, because the leader is not in that array. Sort the followers' match indexes ascending and take the one at position F/2, integer division, where F is the number of followers. That picks the highest index that ceil(F/2) followers have reached, and adding the leader's own copy makes a majority of the cluster. With two followers it is simply the more caught-up of the two:

Here majorityMatchIndex is 95, because the leader and follower A both have it, and that is two of three. persistedWalIndex is how far this leader's own WAL has actually been fsynced, so the clamp says the leader may not commit past its own disk.

That line sits inside Raft's rule rather than replacing it, and the interaction is worth spelling out, because it is the kind of thing that is easy to state loosely and get wrong. The surrounding check tests the term at majorityMatchIndex. The value actually committed is the clamped minimum, and those two are different indexes whenever the leader's own fsync is behind its followers' acknowledgements. Raft only allows a leader to advance the commit point by counting copies for an entry from its current term; earlier entries become committed along with it.

In the code as it stands the two can't diverge in a way that breaks that rule, for a reason that lives two methods away. becomeLeader appends the leader's no-op entry and waits for its WAL write before it starts the replicator threads, so persistedWalIndex has already reached the no-op before any follower can acknowledge anything. The check only fires when majorityMatchIndex is a current-term entry, so the clamped minimum lands somewhere between the no-op and that index, and every entry in that range belongs to the current term.

Which is a correct outcome resting on an ordering in another method, and that is not where a safety property should live. Checking the term at the index actually being committed would make the rule local to the rule. It is on the list.

The reason for the clamp is that "committed" has to mean "on a majority of disks" for Raft's election argument to hold. Enforcing that cost 19% of write throughput, from 1,929 to 1,560 writes/sec, and the faster number was measured on a system that could lose an acknowledged write.

The apply thread is a single writer, which removes an entire category of concurrency bug: the log already decided the order, so nothing else is allowed to touch the cache. It also catches deserialisation failures inside the loop. That one is a scar too: an entry that could not be parsed once killed the apply thread on all three nodes simultaneously, because a deterministic state machine replicates its bugs as faithfully as its data.

Determinism extends to time. A TTL is converted to an absolute epoch-millisecond deadline once, by the node that receives the write, and the log carries the deadline. Storing "expires in 2 seconds" means every replay reinterprets it, and a restarted node resurrects keys the others have already dropped. The log should carry facts, not instructions to be interpreted later.

That fixes replay, not clocks. Each node still compares the stored deadline against its own wall clock, so nodes whose clocks differ will disagree about the exact instant a key stops being visible. Closing that would mean expiring entries through the log too, which BeeDB does not do.

Storage. The WAL is CRC-framed with torn-tail recovery, group-committed at about 11 records per fsync under load. Snapshots serialise the cache, compact the log behind them, and are swapped in with an atomic rename followed by a directory fsync. A follower that has fallen behind the compaction point gets an InstallSnapshot instead of entries. A WAL write that fails does not retry: the node fail-stops. The reason is the one PostgreSQL published in 2018, in the episode known as fsyncgate: on Linux a failed writeback can be reported to a single fsync caller and the dirty pages then dropped, so a second fsync returns success over data that no longer exists. PostgreSQL's fix, shipped in the February 2019 releases, was to panic and replay from the WAL rather than retry. The part BeeDB borrows is that conclusion about retrying, not the architecture around it: BeeDB halts the node and lets the other two carry the cluster, which is a much smaller answer than a database recovering alone.

The gateway

The read path has one design constraint: load must be a function of the cluster, not of the audience. A scheduled poller asks each node for stats once a second, builds one immutable snapshot, and every open SSE stream is fanned that same object. Three stats calls per second, whether one person is watching or a thousand. To be exact: what stays constant is the load the gateway puts on the cluster. Each viewer still costs the gateway one held-open connection and one write per second, which is the cheap part and the part nginx is configured for. GET /api/cluster serves the last snapshot without touching the cluster at all.

The write path is three layers with one job each, which is the part I'd defend in a review:

Two consequences worth naming. BeedbClient.get returning null means exactly one thing, the key is absent; every failure arrives as a typed exception, so the controller never guesses between 404 and 503. And the exception handler is scoped to BeedbException, not Exception: catching everything swallowed Spring's own 400, 404 and 405 and turned every malformed request into a 500.

Failures map to status codes deliberately:

• 503 with Retry-After when a node is unreachable or there is no leader right now.
• 504 when a request was sent and no reply came back. That is the honest "I don't know", and it is the one people get wrong: a timeout is not a failure.
• 404 for a miss, which is not a log line.

There is a known wart here, and it is worth separating three questions that are easy to collapse into one. Did the entry commit? Unknown to the gateway once the socket dies. Is retrying safe? Yes for the commands the API exposes, because applying them twice lands the same value. What does the status say? 503, which reads as "this did not happen", and that is the part that is wrong. Idempotency makes the retry safe; it does not make the outcome known, and a client that treats 503 as "definitely not written" will draw the wrong conclusion about the first attempt. append, prepend, add and replace exist on the server and are not reachable from the API, because BeeDB is at-least-once and a retried append would silently double a value.

Chaos is a state machine, not a button. IDLE, ANNOUNCED, KILLED, with compare-and-set on every transition so two clicks cannot produce two kills, a 10-second announcement so a visitor can watch the thing they caused, a cooldown before another round, and a health gate that requires the cluster to have been seen healthy before it will break it again. The service that decides is not the service that kills: NodeSupervisor is the only component that talks to systemd, and it never decides whether. A demo writer keeps a ring of twenty keys moving so a visitor who touches nothing still sees a live cluster.

Numbers

These are laptop numbers. Measured on a three-node cluster on one NVMe laptop, over loopback, with 32-byte values. The deployed demo runs on a 2-vCPU VPS in Singapore and is slower: 1,567 writes/sec at 8 concurrent clients, p50 4.2 ms, p99 20.4 ms, reads at 0.073 ms.

Loopback numbers are not network numbers, and the read latency is only impressive because those reads are served from local memory with no freshness check. Both caveats are the point of the limitations post.

How it behaves when things break

What I would do differently

Three things, if I started again.

Design the seams before the components. The WAL was built as a class that a Raft node constructs, which is why I still cannot write a test double that discards unforced writes, which is why physical durability is argued rather than proven. One constructor parameter, decided a year earlier, is the difference.

Separate "write it" from "make it durable" in the interface. Appending and fsyncing are one call today. Every interesting durability property lives in the gap between them, and an interface that hides the gap makes those properties hard to state and harder to test.

Use error codes at boundaries, exceptions inside. The gateway got this right by accident, and the server did not: the same failure appears as an exception in one place, a null in another, and a log line in a third.

The rest of this series is the same system from the other side, without the jargon. The limitations post is the one to read next if you want to know what it doesn't do yet.