Bellwether: an open-source reliability layer for self-hosted MongoDB

This is the last part. Nine parts built a three-region MongoDB replica set that holds up under drills: x.509 on every connection, majority writes, a hidden backup member, point-in-time recovery. This part builds the layer that watches it.

On a managed database, failure prediction and index advice come with the service. Self-hosted Community Edition has neither. You self-host for control and data sovereignty; the cost is that you own reliability, including the part that means paying attention to what is quietly going wrong. Bellwether is that layer. It is open source, it reasons about MongoDB internals, and it never touches the cluster without a human.

The cluster it runs against is the one from the earlier parts. The identity it uses is a dedicated read-only x.509 client, issued from the same intermediate CA as every other client here; the mechanics of issuing certificates and creating users are in Part 6 and not repeated. It reads mostly from the hidden backup member built in Part 9, to keep diagnostic load off the voting nodes.

The rest of this article is the design.

Reliability is not one problem. It is two, and they need different handling.

Trajectory failures. A line heading toward a threshold at a rate you can measure now. The oplog is the clean case. It is a capped collection with a configured size, so its window in seconds is roughly that size divided by the write rate. More writes, fewer seconds, while the configured size stays put. Fall below the time a secondary needs to resync after routine downtime and the next node you reboot cannot resume from the oplog; it needs a full initial sync. That failure has not happened yet and you can put a clock on it. Bellwether's oplog finding carries a horizon: estimated time to impact, or already crossed.

Pattern failures. A shape in the workload that is wasteful now, with no countdown. An unindexed query scans the whole collection on every run. Nothing breaks at a deadline; it burns cache and CPU and gets linearly worse as the collection grows. There is no horizon, only a fix.

The tool keeps these apart and names them: predictive for trajectory, advisory for pattern. One predicts a future failure. One diagnoses a present waste. Collapsing both into "AI-powered monitoring" is the marketing lie that makes an engineer close the tab.

The whole system is one principle made concrete:

The deterministic layer holds the truth. The AI interprets it. A second deterministic layer verifies the AI did not corrupt it.

An LLM handed raw metrics invents numbers: a plausible threshold it was never given, a docs-examined figure near but not equal to the real one. Confident and slightly wrong is worse than no answer. So the model measures nothing. Every number in a finding is computed by plain code, deterministic, and the model is told in the prompt that those numbers are ground truth it may not exceed. What the model adds is judgement: burst or trend, which remediation fits this topology, whether an index is worth its write cost. That is real work, and it happens after deterministic detection, never instead of it.

This is also the cost control. The expensive model call fires only when a deterministic detector has already crossed a severity threshold. On an idle cluster the model is never called.

Above the boundary: two deterministic stages (collectors, detectors), one AI stage, one deterministic guard. Below it: nothing moves without a person.

Collectors reach MongoDB through a client that exposes read helpers and nothing else. Not writes-disabled-by-flag; no write method exists on the class. The identity holds read roles only:

clusterMonitor is required for $indexStats; read on the application database covers the profiler and $collStats. Reading the profiling level would need enableProfiler, which MongoDB bundles into dbAdmin alongside createIndex, dropIndex, and dropDatabase. Bellwether will not hold write and DDL privileges to power a read feature, so it does not ask for dbAdmin and accepts one blind spot instead: where it cannot read the profiling level, it degrades and reports unknown rather than escalating a write identity. That trade is the point.

Most reads target the hidden backup node. But the profiler and currentOp are per-node: a slow query is recorded on the member it ran on, and it runs on the primary. So those two collectors sweep every member; the rest read the quiet node. Read from where it is quiet by default; look where the work happens only for the signals that live there.

Most of what Bellwether does is propose-only: the model suggests, a human runs it. Two actions can execute after approval, and only two, fixed in code:

kill_op kills one operation a detector identified, on the member it runs on. create_small_index builds one index on a collection under a document threshold, re-checking the live count before it builds. Everything else is display-only. The whitelist is code; config can only narrow it.

The dangerous one is the index. A compound index's field order is not taste: MongoDB's rule is equality fields first, then the sort field, then range fields (ESR). Get it wrong and the index is weak or useless and still taxes every write. This is exactly what an LLM gets subtly wrong under load. So the model does not choose the index. The detector computes it deterministically from the query shape and ships it inside the finding as evidence. The model reasons about whether to build it, never constructs it. Then a guard checks its output:

We proved it before trusting it. On a fabricated finding the model proposed the correct index. We tampered with the proposal, reordering the fields to a plausible wrong order, and the guard rejected it. The AI cannot ship a wrong index. Not because the prompt asks nicely, but because deterministic code compares its output to a deterministically computed answer and refuses a mismatch.

Fixtures pass. Then you point it at the live cluster and it corrects the design in ways fixtures never could. Three corrections, one lesson.

The oplog does not record what you expect. MongoDB commits a batched write such as an insertMany or deleteMany in one unit, and records it as a single applyOps command entry rather than one entry per document, with the real namespace nested inside each operation:

Filter the oplog by namespace the obvious way and you get an empty result, no error, on the query you most need right. Any tool reasoning about the oplog by outer namespace is silently wrong. Hand-rolling oplog tooling is a hazard, not a weekend job.

A model cannot apply a rule you did not give it the constant for. The index detector escalated a real finding on a 300,012-document collection. The rule: build automatically under the auto-build threshold, otherwise propose. The model was told the rule, not the threshold's value. It saw 300,012 documents, had nothing to compare against, guessed, and chose the executable action for a collection far too large. The guard rejected it four times across both providers; analysis failed. The bug was not in the guard. It was in the evidence: we asked the model to apply a threshold we never stated. Fixtures used a small collection under the threshold, so it never showed. The fix was to put the number in the prompt. State your thresholds; a model reasons from constants you give it, not ones you assume it knows.

A model fills gaps you leave. Where Bellwether could not read the profiling level, it rendered an empty placeholder, and the model read empty as "profiling off" and said so, wrongly. Rendering the value as unknown, not readable instead of a blank fixed it. Same lesson: leave no unlabelled holes in the evidence.

The deterministic layer is only as good as the completeness of what it hands the model. Only a real cluster shows you what you left out.

Bellwether ran as an unattended service on a five-minute timer against the three-region set. On the oplog it measured a ~61-day window against a write rate of tens of bytes per second and correctly found nothing. That window is why the secondaries caught up cleanly after two months powered off instead of needing a resync: the 16 GB oplog from Part 9 held two months of history. The tool measured, live, the property that had saved the cluster.

Then a real unindexed query, run against a real 300,000-document collection until it registered in the profiler. Bellwether swept the profiler across all four members, computed the missing index, escalated, and Claude returned this (abridged):

A query examining fifteen million documents to return five thousand, three thousand to one, 4.6 GiB read for nothing, and the exact index to fix it. Over the auto-build threshold, so proposed for a human, not executed. Field order taken verbatim from the deterministic candidate. The model also noted, unprompted, that profiling was on for only one of four members, so the same query was almost certainly running unobserved on the other three. That is judgement on top of deterministic evidence, which is the whole point.

Ten parts, each a layer, each assuming it will fail and proving it does not.

Bellwether sits on top and watches all of it. It does not trust the AI: it grounds the AI in deterministic truth, checks its work with more deterministic truth, and puts a human between the AI and anything that matters.

Not a replacement for the operational suites the ecosystem sells; those do more, across more databases, with support. Bellwether is focused, open, and self-hostable, for Community Edition operators who want this class of foresight without shipping metrics off-site.

It reasons about MongoDB specifically. The pipeline is database-agnostic, but the collectors and detectors know what an oplog is and what a query shape costs, and that knowledge does not port to Postgres for free. A Postgres implementation would be a sibling sharing the engine, not a rename.

The oplog tooling is hand-rolled, and the applyOps batching trap is a standing hazard for anyone extending it. The AI's diagnosis can be wrong, which is why a human decides and why every proposal carries its evidence. And the one powerful capability, building an index, is narrow, approval-gated, and grounding-checked, because a tool that can change a production database earns that trust by the inch.

What ships today is four detectors and the machinery around them. Oplog window, the one predictive signal. Missing index, redundant index, and profiler-disabled, the three advisory ones. Behind them: the two-provider analysis with failover, the grounding guard, the append-only store, the API with its OpenAPI docs, and the executor with its two bounded actions. That is the whole of it. Nothing in this article describes a capability the code does not have.

The pipeline was built to take more, and the roadmap is a list of detectors that plug into it. Replication lag as a trend rather than a spike. WiredTiger cache pressure before it surfaces as slow queries. Connection-pool saturation heading toward the ceiling. A security signal class reading MongoDB's own auth surface for failure patterns and cert expiry. And an index advisor that predicts rather than diagnoses: not "this query is slow now" but "this query is fine at 300,000 documents and turns disk-bound at your growth rate in a matter of weeks."

The engine underneath is database-agnostic. A Postgres implementation is a sibling that shares it, not a rewrite.

The code is Apache 2.0, at https://github.com/OsegoTech/bellwether . If you run self-hosted MongoDB and want a detector that does not exist yet, the collectors and detectors are the place to add it, and contributions are open.

The cluster is deallocated. The code is open. That closes Logs from the Battlefield. Sharding is a separate series, and it starts from scratch.