What a wiped MongoDB cluster in Bahrain taught me about cloud architecture

I lost a MongoDB Atlas cluster in Bahrain.

It was a free-tier cluster running on AWS nothing important on it, nothing irreplaceable in the data. The lesson was free, in that sense. But I was using a managed MongoDB service, on a top-tier cloud provider, in a region that AWS markets as production-grade with multi-AZ resilience. By every architectural diagram I'd ever seen, I was doing it right. The cluster was supposed to be resilient.

It went away anyway. Not to a hack. Not to a billing mistake. To geopolitics. The US-Iran tensions of that period reached far enough into the regional infrastructure that the cluster simply stopped being available, and when access came back, the data didn't come back with it.

The architectural lesson took a while to settle, and it's the reason this work exists.

Cloud regions are sold as failure-isolated units. The implicit pitch in every "high availability" diagram is that putting your nodes in different availability zones means a problem in one zone doesn't take out the others. That pitch is mostly true for the failure modes the cloud providers test against; power, cooling, network, hardware. It is not true for failure modes where the trigger isn't technical. Sanctions don't respect availability zones. A regional internet shutdown affects every datacenter in the affected jurisdiction. An undersea cable cut between two countries in the same geopolitical zone affects them both. A sovereign action against a cloud operator affects every region that operator runs under that jurisdiction.

What I learned is that availability zone diversity within a single geopolitical zone is not the same as geographic redundancy across geopolitical zones. My Atlas cluster had multi-AZ replication inside Bahrain. It did not have inter-zone diversity, because Atlas couldn't have given that to me from a Bahrain primary even if I'd asked. The fix wasn't switching managed services. It wasn't even switching clouds. The fix was rethinking the geographic topology which is something a managed service in the affected zone cannot structurally solve, no matter how good its operational practices are.

This article is what that rethinking looks like. The architectural answer is the topology , three nodes across two geopolitical zones not the deployment model. I'm self-hosting on Azure VMs because that's what lets me build and demonstrate the topology directly, and because operating the system end-to-end teaches me parts of MongoDB that managed services abstract away. But the topology is what matters. Managed or self-hosted, AWS or Azure or GCP, the same architectural lesson applies: if your replicas all live in one geopolitical zone, you're exposed to a class of failure that no amount of multi-AZ replication inside that zone will fix.

There are a thousand "deploy MongoDB on the cloud" tutorials. There are very few that take the threat model seriously, write down the consistency model, and treat the topology as a deliberate response to a real-world failure rather than a checkbox on an availability slide. We're not touching a single Azure resource yet. We're going to figure out what we're building before we build it.

The natural follow-up question is whether the answer is just "use a different managed provider." I considered that. It isn't, for reasons worth being explicit about.

Atlas is a good product. For most workloads even most production workloads, a managed MongoDB service is the right answer. It handles failover, backups, monitoring, encryption, certificate rotation, and a dozen other operational concerns that we are going to build by hand. If you're choosing between Atlas and self-hosting for a real production system at a normal company, the calculus often favors Atlas, and the engineers who choose it are not making a mistake.

But Atlas and equivalently, MongoDB on a Cosmos-style managed offering, or any managed MongoDB has one constraint that matters specifically for the Bahrain class of failure: the geographic distribution of replicas is bounded by the provider's available regions and by the provider's permitted topologies between them. If the provider doesn't offer a region in the geopolitical zone you want to diversify into, you can't diversify there. If the provider doesn't allow your cluster to span certain region pairs, you can't span them. Multi-region Atlas is real and it works, but it's still inside the design envelope of one vendor's region map.

For a workload where geopolitical-zone diversity is a hard requirement, self-hosting buys you the ability to pick the exact regions, on the exact cloud, in the exact zones, with the exact replica priorities and tags you want. You pay for that flexibility with operational complexity every concern that Atlas handles becomes yours to solve.

There are other legitimate reasons to self-host. Strict data sovereignty requirements where the operator must control the security boundary entirely, many financial regulators are explicit about this. Integration with an existing cloud-committed enterprise estate where the managed service's billing and identity story doesn't fit. And, frankly, depth of understanding: if you've never operated a MongoDB cluster end-to-end, you have a thinner mental model of the system than someone who has.

I'm building this as if I were running it for a regulated financial workload, because designing to that bar produces a system that handles less-demanding workloads as a side effect. The reverse rarely works: a design that meets a casual bar and tries to retrofit financial-grade controls usually fails in subtle ways.

The rest of this article assumes the decision to self-host has been made. If you're choosing between a managed service and self-host for a real production system, the calculus may favor the managed service but the architecture below is what self-host done seriously looks like.

Before any topology, any region selection, any configuration, the requirements.

The hypothetical workload is a financial application, transactions, balances, audit records. I'm using "financial" because financial workloads impose the strictest real-world requirements I know: durability of committed writes is non-negotiable, consistency of authoritative reads is a correctness invariant, audit trail integrity is regulatory, and downtime has direct revenue and reputational cost. A system designed to those requirements will handle a content site or an analytics workload comfortably; the reverse rarely works.

User base. Africa and the Middle East. Not because that's where every financial application's users are, but because that's where the application I'm modeling would serve users and because designing for this geography makes the architectural challenges more interesting, and the writing about them more original, than another US/EU deployment guide.

Data residency. Soft preference for the regional geography, with deliberate cross-geopolitical-zone diversity for disaster recovery. This is the architectural response to the Bahrain experience. Strict residency putting all three nodes in one geopolitical zone would recreate exactly the failure mode I'm designing against.

Recovery objectives. Tiered, because a single RPO/RTO number across all failure classes is fiction.

Tier 1 is what synchronous majority replication and automatic election give us almost for free. Tier 2 is bounded by backup frequency and restore speed. Tier 3 is the Bahrain scenario recoverable specifically because the architecture preserves a replica outside the affected zone. Most "production guides" claim a single RPO/RTO that conflates these tiers; this one will not.

Edition and authentication. MongoDB Community. x.509 certificates for both inter-node cluster authentication and client-to-server authentication. The reasoning for x.509 over keyfile, in a financial context, is in the threat model below.

Encryption at rest. Layered. Azure Managed Disks are encrypted at rest by default with platform-managed keys. On top of that, encryption at host with a customer-managed key in Azure Key Vault gives us control over the key material, audit visibility on every key access, and the ability to revoke disk access by rotating or disabling the key. Native MongoDB encryption-at-rest is an Enterprise feature; we get the equivalent threat coverage on Community by encrypting the layer below the database. This replaces the older Azure Disk Encryption service, which wrapped LUKS and is scheduled for retirement in September 2028.

Topology. Three-node replica set, one node per region, across three regions spanning at least two geopolitical zones.

These requirements drive every decision in the rest of the article.

A threat model is structured thinking about what can go wrong, who might cause it, and what we do about it. The professional framework is STRIDE - Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, Elevation of privilege. I'm using it because it's the standard, and because it forces a discipline that ad-hoc lists of "security best practices" don't: every control exists to mitigate a specific threat. If you can't name the threat, you don't yet know why the control matters.

I'm adding a seventh category - Insider - because STRIDE doesn't cleanly capture authorized misuse, and authorized misuse is one of the most consequential threat classes for financial systems. It's also, given the Bahrain experience, the category most directly relevant to the geopolitical-zone-diversity decision: an admin compelled to provide access in one jurisdiction cannot reach an admin or a replica in another.

Each subsection below: a one-sentence definition, three or four threats specific to our architecture, and the controls that mitigate them.

The threats: an attacker presents themselves as a legitimate replica node and tries to join the cluster, gaining access to replicated data. An attacker steals a client credential and connects with that identity. An attacker performs a man-in-the-middle attack between client and server, presenting a forged certificate to each side. An attacker poisons internal DNS to redirect a node's connections to their endpoint.

The primary controls are x.509 mutual authentication for both inter-node and client-to-server connections, TLS with full certificate chain validation including SAN matching, and Azure Private DNS Zones scoped to our VNets. Each of these forces the attacker to obtain something they can't easily get a certificate signed by our CA, the private key for that certificate, or write access to our private DNS zone rather than allowing identity to be claimed by anyone who can reach the network port.

This is the section that explains why I picked x.509 over keyfile authentication. Keyfile authentication uses a shared symmetric secret that every node has and presents to every other node. It works, but the same secret is on every node compromise of one node compromises the cluster identity, there is no per-node identity for audit purposes, and there is no certificate chain, expiry, or revocation. For a financial workload subject to real auditors, x.509 is the answer. The operational complexity of running a private CA is itself part of why this is a serious project rather than a tutorial.

The threats: an attacker modifies a write request in transit, changing transaction details. An attacker modifies replication traffic between nodes, causing secondaries to diverge from the primary. An attacker with access to the underlying disk modifies database files directly, bypassing MongoDB's access controls. An attacker who has gained shell access modifies mongod.conf to weaken security and waits for the next restart.

TLS provides message authentication on every frame, not just encryption - a modification in transit is detected by the cipher's authentication tag and the connection is dropped. at-rest encryption renders an offline disk or snapshot unreadable without the key. Filesystem permissions and a dedicated mongod system user mean root-level VM access is required to modify configuration; bastion logging and Azure activity logs make that access detectable and attributable. Configuration as code, with reviewed changes, makes drift from expected configuration investigable.

The threats: an admin makes a destructive change and there's no record proving who did it or when. A user performs a fraudulent transaction and later denies having authorized it. A backup is deleted or modified and no log preserves who deleted it. A configuration change weakens security and the timing or actor cannot be determined.

Bastion session logging captures every interactive operator action. Azure activity logs capture every Azure resource operation in the control plane, surviving even if the VMs are destroyed. Certificate-bound user identity ties every database action to a specific certificate subject, which traces to the issuance event. At the application layer, an append-only audit collection with hash-chain integrity makes tampering with historical records detectable on next verification. MongoDB Community's native audit support is limited; we compensate at the application layer where it matters most.

The threats: an attacker reads database traffic on the network. An attacker accesses a backup file and reads the database from it. An authenticated user reads data outside their authorized scope. Sensitive parameters are inadvertently logged. An attacker reads disk contents directly via cloud-resource access.

TLS in transit closes the network read. At rest closes the disk read. Custom RBAC with least-privilege grants closes the over-broad-authenticated-read. Logging hygiene - query parameters at debug level only, debug off in production closes the inadvertent log exposure. NSG rules close the outermost shell: an attacker cannot reach the database to read its traffic in the first place. Backups are themselves encrypted with keys managed separately from the backup files.

The threats: a connection flood exhausts the connection pool. An expensive query, a large unindexed scan, a malformed aggregation consumes CPU and memory until the node is unresponsive. An internet-scale scan finds an exposed port and a botnet floods it. A misbehaving application driver opens connections faster than it closes them.

No public IPs on the database nodes; they're reachable only through the VNet via NSG-permitted paths. Connection limits in mongod.conf cap per-node exposure. Mandatory index coverage on application queries, enforced through code review and verified through query monitoring, removes the unindexed-scan class. Resource isolation, each node runs MongoDB and only MongoDB - prevents noisy-neighbor effects. Disk-utilization alerts at 70% and 85%, with documented response procedures, prevent the most common availability incident in self-hosted databases: disk-full on a primary.

The threats: an application user finds a way to perform admin operations, through a vulnerability or a misconfigured role. A read-only user finds a write path through an aggregation output, a view definition, or a server-side function. A compromised application server, holding the application user's credentials, is used to extract or modify more data than the attacker should be able to before being noticed. An attacker chains lower-privilege access into higher-privilege access.

The primary defense is least-privilege RBAC with custom roles, not built-in ones. Built-in roles silently expand as MongoDB versions add permissions; custom roles with explicit grants do not. Per-purpose users - application, migration, backup, monitoring - mean compromise of one credential doesn't compromise others. Patch discipline addresses known privilege-escalation vulnerabilities. Monitoring for anomalous query patterns , a sudden spike in admin operations, queries against collections the user shouldn't be touching - provides detection where prevention fails.

The threats: a DBA exfiltrates data using legitimate cluster-admin access. A DBA modifies records to commit or conceal fraud. A platform engineer with SSH access exfiltrates the Encryption key alongside a snapshot. An admin under legal compulsion in a hostile jurisdiction is forced to provide access, the variant most directly relevant to the Bahrain experience. An admin resigns and retains access through credentials not promptly revoked.

The controls in this category are different in kind from those above. Separation of duties: the person who can modify data is not the person who can modify the audit log; the person who manages encryption keys is not the person who has SSH. Just-in-time access elevation rather than standing admin. Dual control for the most sensitive operations: backup restore, CA root rotation, audit retention changes. Tamper-evident audit chains that detect retroactive modification. And this is the architectural control, decided in the design phase itself geographic and organizational distribution of authority. The cross-zone West Europe replica is, in part, an insider-threat control: an admin compelled in one jurisdiction cannot reach a replica in another. Comprehensive offboarding when team members leave is procedural but the most common control to fail in practice.

The matrix below summarizes the seven categories with representative threats and their primary controls. This is the table I wish I'd had when I started designing distributed systems, it makes the relationship between threat and control auditable at a glance.

Three regions, given the requirements above.

The constraints: primary user proximity (Africa/Middle East); voting quorum survives any single region loss (automatic with 3 nodes in 3 regions); geopolitical-zone diversity (at least one node outside the primary user-base zone); and Azure regional maturity (full service availability, availability zones, operational track record).

The selection:

UAE North (Dubai), Availability Zone 1 - primary preference. Mature region with availability zones and full service support. Dubai is a regional connectivity hub with deep international links. For users across the Gulf, Levant, North Africa, and Eastern Africa, latency is good. This is where the application's reads and writes should feel fast.

South Africa North (Johannesburg), Availability Zone 1 - secondary. Different country, different sub-region within the Middle East/Africa geography, still on the African continent. UAE-to-Johannesburg latency is roughly 200ms - high enough that you feel it on writes, low enough that majority writes still complete in well under a second. Provides resilience against a UAE-specific issue without abandoning the user-base geography.

West Europe (Amsterdam), Availability Zone 1 - DR / cross-zone. Different geopolitical zone entirely: EU jurisdiction, EU regulatory framework, separate political and infrastructure dependencies. UAE-to-West Europe latency is roughly 130ms; Johannesburg-to-West Europe roughly 170ms. Both are workable for replication. This is the node that makes the architecture survive the failure mode that wiped my Bahrain cluster.

The combination maps onto the failure tiers cleanly. Any single node loss leaves a 2/3 voting majority across two of the three regions; election proceeds, RTO under a minute. Multiple simultaneous node losses move us into the backup-restore tier but the geographic spread means a single physical event cannot cause this. And if the entire Middle East/Africa zone has a bad day for non-technical reasons, the West Europe node has the data and can be promoted manually as the source of truth for a rebuilt cluster.

Cross-region replication is not free. Every committed write incurs cross-region acknowledgment latency. Writes using {w: "majority", j: true} will see baseline latency on the order of 200ms, possibly more under load. This is an unavoidable cost of the consistency guarantees we want. The next section explains why we accept it.

Consistency is the hardest topic in distributed databases, and the one most "production guides" handle worst. I want to handle it head-on.

The basic problem: with three nodes and replication, a write is acknowledged at one point in time and a read happens at another. Between them, what guarantee does the database give about what the reader sees? The naive answer - "the reader sees what was written" is true on a single node, and not automatically true on three. MongoDB lets you configure the answer per operation, through write concern (when a write counts as committed) and read concern (when a read counts as consistent). The interaction of these two settings defines your consistency model, and it's the single most consequential design choice you'll make.

The principle to internalize: consistency is a property of each operation, not of the database. A correctly designed application uses strict consistency for the operations that need it and weaker consistency where it's safe to do so, because strict consistency has real costs and paying them where they aren't needed is a self-inflicted wound.

For our architecture, every committed financial write uses writeConcern: {w: "majority", j: true, wtimeout: 5000}. This means the write isn't acknowledged to the application until it's been journaled to disk on at least two of the three nodes - a majority. Any single-node failure cannot lose the write, because the write exists durably elsewhere before the application is told it succeeded. This is the foundation of our Tier 1 RPO of less than one second. The five-second timeout matters: without it, a degraded cluster can cause writes to hang indefinitely, and we'd rather fail fast and let the application handle it than silently wait.

For reads, the model is more layered. Reads that inform critical decisions, a balance check before a debit happen inside multi-document transactions with readConcern: "snapshot" and the same majority-and-journal write concern, giving us atomicity over the read-and-write pair and preventing race conditions where a stale balance allows an account to go negative. Reads that don't inform critical decisions -showing a user their balance on screen - use readConcern: "majority" with readPreference: "secondaryPreferred", which guarantees the data won't be rolled back but tolerates a few hundred milliseconds of staleness in exchange for offloading work from the primary. Reporting queries route to secondaries strictly. Audit log writes use the same majority-and-journal concern as financial writes, because an audit gap is worse than no audit at all.

The full mapping looks like this:

A few of these decisions are worth surfacing explicitly. The audit log being at the same durability tier as money movement surprises some engineers. The reasoning: an audit record lost during a primary failover creates a gap in the chain, which undermines the integrity of the entire log and is what auditors look for. We pay the same cost for audit durability as we do for the underlying transactions.

readConcern: "linearizable" rarely appears in our model. It has a legitimate use case - strict reads outside transactions - but in practice, when you need a strict read, you usually need a strict read followed by a write, and that pattern is better served by a transaction with snapshot read concern. Linearizable is the most expensive read MongoDB offers; it round-trips to the majority before answering, and it's the wrong default for "I want my read to be authoritative."

We don't use readConcern: "local" anywhere. Local is the MongoDB default, and on Community deployments most "production" systems use it without thinking - which means most production systems have a class of read that can see writes that get rolled back, and they don't know it. For a system with our requirements, that's not acceptable for any read informing a real decision.

Everything above synthesizes into a single diagram.

Three regions across two geopolitical zones. UAE North holds the primary preference, South Africa North is the in-zone secondary, West Europe is the cross-zone DR replica. The replication mesh runs over TLS with x.509 mutual authentication, never over the public internet. The trust boundary - a hard line, not a soft one - separates the cluster from everything else; the only legitimate paths in are the application tier (with x.509 client certificates) and just-in-time operator access through a bastion. The failure tier coverage is annotated on the diagram so the architecture and its behavior under failure are visible together.

Worth noting what's not in this diagram. No subnet structure. No certificate authority topology. No user roles. No backup infrastructure. No monitoring stack. Each of those is its own substantial body of work; trying to compress them into one diagram produces something that explains nothing. This is the system at one level of abstraction, and there's value in stopping there before zooming in.

If at any point you read a configuration choice in something I publish later and ask "why is it that way," the answer should always trace back to this article - the requirements, the threat model, the consistency choices. The architecture isn't decoration; it's a contract.

The design is now committed. From here, the work is to build it.

That work breaks into discrete bodies - the network layer (VNets across regions, mesh peering, Private DNS, NSGs locking down the security boundary, the VMs the database will live on), the certificate authority that issues every node and client identity, the MongoDB installation and OS hardening that turns three Linux boxes into a replica set, the RBAC model and custom roles that make least privilege real, the application-layer code demonstrating each row of the consistency table on a real transactions schema, the observability stack that tells you whether the consistency contract is being honored, the backup and PITR strategy that bounds the worst case, and the failure drills that prove the runbook actually works.

Each of those is a piece of work in its own right, and each will get written up properly when it's done. If you want to follow that as it lands, If this is the kind of work you want to follow as I build it, the Azure foundation, the PKI, the replica set itself, the failure drills - I'm publishing each piece on Meetadev where this series lives, and sharing each post on LinkedIn. The story that started this is there too, if you came in cold.

This work is being implemented as it's written, not described from memory. The configurations, scripts, and certificate-management procedures will be public on GitHub once they exist, so the work is verifiable rather than merely described. If you spot something that should be designed differently, or a threat I've underweighted, or a choice that doesn't hold up under your particular workload, comment or reach out the engineering quality of what gets built will be higher for the feedback than for my own review alone.

If you're deploying something similar in your own environment, or if a specific section sparked a question worth answering at length, that's the most useful kind of feedback. The most interesting writing on any technical project usually comes from a reader's question that the author hadn't thought to ask.

The design is set. Now we build.