Acknowledged is not durable: write concern, read concern, and what a transaction means across three regions

A previous build decided who may write to the ledger. This one decides what a write is allowed to mean. On a single-node database those are the same question. There is one copy of the data; a write that returns success is on disk, and a read returns what the last write left. Durability is not a decision because there is nothing to decide — there is only one node, and it either has the data or it does not. The single-node mental model quietly equates three words that a distributed system pulls apart: acknowledged, replicated, and durable. They are not synonyms. On a replica set spanning three regions and two continents, a write can be acknowledged before it is replicated, replicated before it is durable, and a read can return data that is none of the three. The gap between those words is where financial systems are lost, and closing it deliberately is the subject of this build. The cluster already has the topology that makes this a decision rather than a default. Three regions, a primary that holds authority by design, two secondaries one and two network hops away. That topology was built for resilience — to survive the loss of any single region. The same topology means a write now has somewhere to go before it is safe, and a read has more than one node it could come from. Resilience and ambiguity are the same property seen from two sides. The work here is to remove the ambiguity for the one workload that cannot tolerate it: a financial ledger, where a transaction that is acknowledged and then quietly disappears is not a performance problem but a solvency one.

Durability guarantees operate on something, and what they operate on here is a double-entry transaction. The schema is not the subject of this build, but it is the object every guarantee in it acts upon, so it has to be right before the guarantees mean anything.

Double-entry accounting has one inviolable rule: money is never created or destroyed, only moved. Every transaction produces entries that sum to zero — a debit somewhere balanced by a credit somewhere else. A transfer of five thousand from one account to another is not one fact ("money left A, arrived at B") but two entries that must both be true or both be false: a debit of five thousand against the source, a credit of five thousand to the destination. If one entry survives and the other does not, money has been created or destroyed, and the ledger is corrupt.

This rule shapes the single most important modeling decision, and it is a decision where the relational instinct and the MongoDB-native answer diverge. The relational instinct is two rows in an entries table sharing a transaction id, assembled at read time. The MongoDB-native answer — and the correct one here — is one document per transaction, with the entries as an array inside it:

The reason this is the right shape is the reason it matters for the rest of the build: a single-document write in MongoDB is atomic. Both legs of the transaction commit together or neither does. There is no instant at which the debit exists and the credit does not — not under concurrent reads, not under a crash, not under a failover. Modeling the transaction as one document means the both-or-neither guarantee is provided by the storage engine itself, for free, rather than assembled out of multi-document transactions and application care. The double-entry invariant and single-document atomicity reinforce each other: choose the shape that makes the storage engine enforce indivisibility, and the hardest part of ledger correctness is solved by the data model rather than by code.

A financial collection should refuse to store a malformed transaction. Not warn, not log — refuse. The database is the last line of defense, the one barrier a buggy application or a fat-fingered shell command cannot get past, and for a ledger that barrier is worth having.

MongoDB enforces this with $jsonSchema validation attached to the collection. The validator declares the shape every transaction must take, and the collection rejects any write that violates it:

This enforces the structure that can be checked locally on a single document: required fields present, transaction type and status and entry direction confined to fixed sets, currency matching the ISO 4217 three-letter pattern, at least two entries, every entry carrying an account and direction and a positive amount, and — through additionalProperties: false on the entry sub-schema — no stray fields smuggled into a leg. validationAction: "error" makes violation a rejection rather than a warning.

There is one invariant the validator deliberately does not enforce, and the decision not to enforce it is more interesting than the validation itself: the balance. Nothing in that schema checks that the debits sum to the credits. That omission is intentional, and it reflects where in the stack each guarantee correctly lives.

The temptation is to push the balance check into the database with an expr‘validationrulethatsumstheentriesandcomparesdebitstocreditsoneverywrite.Itcanbedone.Itshouldnotbe.Threereasons.First,an‘expr validation rule that sums the entries and compares debits to credits on every write. It can be done. It should not be. Three reasons. First, an expr‘validationrulethatsumstheentriesandcomparesdebitstocreditsoneverywrite.Itcanbedone.Itshouldnotbe.Threereasons.First,an‘expr rule that sums an array on every insert sits on the write hot path and grows brittle as the schema evolves — two-leg transfers are simple, but multi-leg splits, fees, and multi-currency settlement turn an elegant balance expression into an unmaintainable one. Second, whether a transaction balances is a domain rule, and domain rules belong in application code that is tested, versioned, and reviewed, not buried in a collection validator that is invisible to most of the team and awkward to unit-test. Third, and most important, the real protection against an unbalanced transaction is not validation but atomicity: a balanced transaction written as one atomic document can never be half-applied, so there is no partial state for a balance check to catch. The guarantee comes from the data model, not from a rule.

So the division of labor, which is the industry-standard architecture rather than a shortcut: the database enforces structure through $jsonSchema; the application computes and asserts the balance in its posting logic; and atomicity — soon joined by write concern — ensures the balanced result is indivisible and durable. Knowing which layer owns which guarantee is the actual engineering. Pushing everything into the database is not more rigorous; it is a category error that trades maintainability and clarity for the illusion of a single enforcement point.

The validator earns its place the moment it is tested. A write with a single entry and an invalid direction is rejected before it touches the ledger, the database naming both violations precisely — the array too short, the direction outside its enum:

The malformed transaction never exists. The correctly-formed, balanced version is accepted. Structure is guaranteed; the floor is in place. Now the question that the whole topology was built to make answerable: when a well-formed transaction is written, what does its acknowledgment actually promise?

When an insert returns acknowledged: true, it is natural to read that as "the write is safe." On a replica set, what it actually means is a parameter you set, and the default is not the meaning a ledger wants.

Write concern is the contract between the application and the cluster about how much confirmation a write must collect before it is acknowledged. Its principal axis is w, the number of members — or the keyword majority — that must have the write before the primary reports success. A secondary axis, j, governs whether those members must have flushed the write to their on-disk journal (durable through a crash) or merely accepted it into memory (lost if the process dies).

Three settings span the meaningful range, and on a three-region cluster they have measurably different costs. Measured across twenty-five iterations each, on this cluster, with the cross-region latencies the topology imposes:

Read those numbers carefully, because they contain the entire argument.

w:1 acknowledges in two milliseconds because it waits for no one. The primary writes to its own memory and reports success immediately; the secondaries will replicate eventually, but the acknowledgment does not wait for them. This is fast, and for a ledger it is dangerous: if the primary fails in the window between acknowledging the write and replicating it, the write is gone. The application was told the transaction succeeded; after the failover, the transaction never happened. Two milliseconds buys an acknowledgment that is not a promise.

w:majority acknowledges in a hundred and five milliseconds because it waits for a majority of the voting members — two of the three regions — to confirm they have the write. That hundred milliseconds is not overhead in any wasteful sense; it is one cross-region network round trip, the physical cost of getting a second continent to agree that the write exists. It is the price of the topology, made visible. And it is the price of a real guarantee: a write confirmed by a majority cannot be rolled back, because any new primary must be elected from the majority that already holds it. The transaction will survive the loss of any single region.

w:majority + j:true acknowledges in a hundred and six milliseconds — one millisecond more than majority without journaling. This number is worth dwelling on because it contradicts a common intuition. The instinct is that forcing an on-disk journal flush (j:true) will be slow, because disks are slow. The measurement says the journal flush adds a single millisecond, lost in the noise of the cross-region round trip that dominates everything. The expensive part of durable distributed writes is not the disk. It is the speed of light between regions. The journal is nearly free; the geography is the cost.

That last point is the kind of thing that only falls out of measuring rather than assuming, and it reframes the trade-off. The choice is not "fast write versus slow disk." It is "an acknowledgment that might be a lie, in two milliseconds, versus a guarantee that holds across regions, in a hundred." For a ledger, the hundred milliseconds is not a cost to be minimized away; it is the thing being purchased, and it is cheap for what it is.

A note on honesty in that measurement. The first single-sample pass of this benchmark showed w:majority slower than w:majority + j — the supposedly-cheaper option appearing more expensive. That was not a real finding; it was network jitter on a sample size of one, the cross-region round trip varying enough between two lone measurements to invert them. The fix is the fix for all distributed-systems benchmarking: run many iterations, report the median and the tail. The twenty-five-iteration medians above are stable to within a couple of milliseconds and tell the true story, which a single sample could not. Distributed latency has a distribution; one number from it is a guess.

Write concern decides when a write is durable. Read concern decides whether a read returns data that will stay true. It is the subtler of the two, because a wrong read does not raise an error — it quietly returns data that may be revoked, and the application acts on it before discovering the revocation.

The default read concern, local, returns whatever the queried node currently holds, including writes that have not yet been confirmed by a majority. Most of the time this is correct and fast. The failure case is the one that matters: the primary accepts a write, a local read returns it, the application acts on it — authorizes a payment, displays a balance, releases funds — and then the primary crashes before that write reached a majority. The newly elected primary, drawn from the majority that never saw the write, does not have it. The write is rolled back. The application read a transaction, made an irreversible decision on it, and the transaction then ceased to have ever existed. For a ledger, reading data that can be un-written is reading a guess.

majority read concern closes that gap. It returns only data that a majority of nodes have acknowledged — data that, by the same logic that protects a majority-written transaction, cannot be rolled back. The trade is a small amount of freshness: a majority read might not show the very latest write if that write has not yet propagated. In exchange, everything it does show is permanent. For reading financial state — balances, transaction history, anything an action depends on — that is the correct floor. You may act only on money certain to exist.

linearizable is the strongest and most expensive, reserved for the rare read that must be both current and authoritative. Beyond majority durability, it guarantees the read reflects all writes that completed before it began, and that the node answering is still genuinely the primary — not a deposed primary that has not yet learned of its own demotion and would otherwise serve stale data confidently. It pays for this by confirming primacy before answering, which is why it must be given a timeout, and it is the right tool for the narrow case of a balance check immediately before authorizing a large, irreversible withdrawal.

Run against the committed ledger, all three concerns return the same two transactions:

The identical output is the entire point, and it is the point most easily missed. Read concern is not visible in the happy-path result; with majority-durable data, every concern returns the same rows. The difference is not in what you read but in what the read promises about the failure path — whether the data you just acted on can be taken back. local makes no such promise. majority promises permanence. linearizable promises permanence and currency and authority. You cannot tell them apart by looking at the results, which is exactly why the choice has to be made deliberately, per operation, by someone who understands the guarantee rather than the output.

Everything to this point depends on the application specifying the right concern on every operation. That is a fragile foundation. Safety that depends on every developer remembering, on every write and every read, to opt into durability is safety that will eventually be forgotten — and the failure is silent, because a forgotten concern does not error, it just quietly takes the weak default. A financial system should fail safe when someone forgets, not fail fast and dangerous.

MongoDB allows the cluster's default read and write concerns to be set globally, so that an operation specifying nothing inherits the safe behavior rather than the permissive one:

javascript

After this, a write that names no concern is acknowledged only on majority confirmation, and a read that names no concern returns only majority-durable data. The cluster's response carries a small but meaningful piece of evidence: the source field flips from implicit to global. Before this command the cluster used the built-in default, recorded as implicit — a default no one chose. After it, the source reads global, meaning the durability posture is now an intentional, auditable decision rather than an inherited accident. For a regulated system, that distinction — between behavior that happened to be the case and behavior that was deliberately set — is itself part of the record.

The proof that the default is real is a write that asks for nothing. The same insert that took two milliseconds under explicit w:1 earlier, now issued with no write concern at all, takes a hundred and nine milliseconds:

That number is the whole argument compressed into one measurement. A developer who writes a transaction and gives no thought to durability now gets it anyway — the write waited for majority confirmation across regions because the cluster, not the developer, decided that it must. Forgetting the concern now costs a hundred milliseconds and buys a guarantee, where before it cost nothing and silently risked the transaction. The cost moved to the safe side of the ledger. That is what making safe the default actually means: not that durability is free, but that its absence is no longer the path of least resistance.

The ledger now has a durability model. Transactions are structurally validated at the database and balanced at the application, written as atomic double-entry documents. Writes are acknowledged only when a majority of regions hold them durably, a guarantee whose cost — one cross-region round trip, around a hundred milliseconds — has been measured rather than assumed, and whose dominant component is geography, not disk. Reads return only data that cannot be rolled back, with a stronger linearizable option held in reserve for the rare authoritative read. And the safe behavior is the cluster default, so durability is what happens when no one is paying attention rather than what is forgotten. Every guarantee in that paragraph was demonstrated against the running cluster.

What it is not, yet, is proven under failure. Everything above establishes what should happen when a region is lost: a majority-written transaction survives, a w:1 transaction may not, a majority read never returns data that a failover will revoke. Those are guarantees argued from the protocol and measured in the happy path. They are not the same as having stood up in front of a deliberately killed primary and watched a weakly-written transaction vanish while a majority-written one remained. That demonstration — actually inducing a failover and observing which writes survive and which roll back — is the most visceral possible evidence for everything claimed here, and it is deliberately held back, because inducing and surviving failure is a discipline of its own and the entire subject of a later build. Proving durability's cost and its guarantees is this layer's work; staging its failure is the next one's.

What it also is not, yet, is observed. The cluster now makes correct durability decisions, but nothing here watches whether it is keeping them — no metric on replication lag, no alert when a secondary falls behind and majority acknowledgment starts to slow, no visibility into the hundred-millisecond write becoming a five-hundred-millisecond write because a region is struggling. A durability guarantee you cannot observe is a guarantee you are trusting blind. Making the cluster legible — turning its internal state into metrics and signals an operator can see — is the work that comes before failure testing, because you cannot drill a failure you cannot watch.

The access model that governs who may write these transactions was built in the previous part — least privilege is a claim until the database refuses you. The three-region topology whose latency every measurement here reflects was established when the cluster first formed — the cluster learns to say yes. That build decided where authority sits. This one decided what authority's writes are worth: not acknowledged, but durable — across three regions, on purpose.

Building something similar, or want to argue with where the balance check belongs? I run a community for MongoDB builders on Meetadev — MongoDB AI Builders — where this series and the decisions behind it get worked through in the open. Come break things with us. 🔧