The cluster learns to say yes: initiating a three-region MongoDB replica set with x.509 identity

Initiation is the step that joins them, and it is routinely treated as trivial: run rs.initiate(), watch an election, move on. That framing misses what the step actually commits. Initiation is where the cluster's identity model is written into existence — where authority is placed geographically rather than left to chance, where the first principal is created, and where every assumption made about the certificate authority is finally tested against a running quorum. The configuration is a few lines. The decisions behind those lines are the architecture.

Left to itself, a replica set election chooses a primary from whichever electable member is most current and can reach a majority of voters. In a single datacenter that is enough — every node is equivalent, so any primary is as good as another. Across three regions on two continents the nodes are not equivalent, and treating them as equivalent hands an architectural decision to chance.

The three regions were selected for distinct roles. One sits closest to the user base and should serve reads and writes at the lowest latency. One provides in-continent resilience — a different country and failure domain, still near the primary user geography. One sits in an entirely separate jurisdiction as the disaster-recovery anchor: the node whose value is precisely that it is elsewhere, the one that survives a regional event affecting the others.

Member priority is where those roles become enforceable. Priority is the weight a member carries in an election; the electable member with the highest priority that can reach a majority wins. Assigning priorities deliberately makes the runtime election express the design rather than contradict it.

• UAE North — priority 2. The primary-preference region. Under normal operation the primary lives here, where the application's reads and writes are fastest.

• South Africa North — priority 1. The first failover target. If the primary region is lost, authority moves here, staying on-continent and close to the user base.

• West Europe — priority 0.5. The cross-jurisdiction disaster-recovery anchor, deliberately the least preferred to hold the primary. A last-resort replica should not become the system of record during routine churn; its promotion should be a decision, not an accident.

Every member carries a single vote. Three voting members means the loss of any one region still leaves a two-of-three majority, an election completes, and recovery finishes inside a minute — the availability guarantee the topology was designed for, expressed in the one place it can be enforced: the member configuration itself. Priority decides who would prefer to be primary; votes decide whether a primary can be elected at all. Keeping all three votes equal while varying priority is what lets authority sit by preference in UAE without weakening the quorum that protects against losing it.

Each member also carries a region tag. The tags are inert at election time, but they are the addressing scheme later work depends on — read-preference routing toward a region, and zone-aware data placement once the cluster is sharded. Tags cannot be retrofitted into a running set without a reconfiguration, so they go in at initiation, when the cost is nothing.

A secure deployment carries an ordering problem that has to be understood before initiation makes sense.

Every node was started with access control enabled and x.509 cluster authentication required — the posture that closes the audit gap x.509 was chosen for. But it produces a bootstrapping paradox. The replica set cannot be initiated without an authenticated connection, and a user cannot be created without an authenticated connection, yet no user exists to authenticate as. A cluster that requires authentication before it will allow anyone to create the first authenticated identity cannot, on its face, be started at all.

MongoDB's answer is the localhost exception: on a node whose user collection is empty, a connection from the loopback interface may run the commands needed to bootstrap — initiate the set, create the first user — without authenticating. The exception is deliberately narrow. It applies only over localhost, only while no user exists, and it closes permanently the instant the first user is created. Exactly wide enough to seed identity, and no wider.

One distinction is easy to miss and worth stating precisely. The localhost exception bypasses authorization — who you are, what you may do. It does not bypass transport security. With requireTLS and mandatory client certificates, the TLS layer still demands a certificate to complete the handshake, even over loopback. The two gates are independent and both apply at once: the certificate satisfies the transport layer, and the localhost exception carries the connection past the authorization layer. The bootstrap connection therefore presents the operator certificate to complete TLS, and the exception permits the privileged commands the empty cluster would otherwise reject.

Two things are worth confirming before initiating, rather than assuming.

The first is that all three nodes are running and reachable. The second — less obvious, and a cause of intermittent election trouble when neglected — is time. Replica set members exchange heartbeats and stamp every operation with a timestamp; meaningful clock skew between regions disrupts elections and confuses oplog ordering. Time on these nodes is disciplined by the Azure host's precision time source, and chronyc tracking confirms it: a stratum-1 reference clock, a system-time offset measured in single-digit microseconds, and a leap status of normal. Across three continents the worst-case skew between any two nodes is microseconds — orders of magnitude beneath anything an election could notice. Clock skew is a real failure mode for geographically distributed replica sets; here it has been engineered out at the platform layer, and verified rather than presumed.

Initiation runs from the primary-preference node over the localhost exception. The configuration names each member by its fully-qualified domain name from the private DNS zone — never by IP. This is not stylistic. Each node's certificate carries its FQDN in the subject alternative name, and certificate verification during the inter-node TLS handshake matches the hostname a peer is reached by against that peer's certificate. Address a member by IP and the verification has nothing to match; the handshake fails. The FQDNs are the names the certificates were issued for, so the FQDNs are the names the configuration must use.

The response is a single { ok: 1 }. In that instant the three isolated processes become a replica set. The node the command ran on writes the configuration, propagates it to the peers it can now reach over their certificates and the private DNS, and heartbeats begin. The first election is already in motion.

Within a few seconds the set settles. A compact projection of member state shows the shape immediately:

UAE holds the primary, as the priority weighting dictated. South Africa and West Europe are healthy secondaries. The design became runtime reality with no manual intervention.

The mechanism underneath that result is worth following, because it is what makes the outcome reliable rather than fortunate. MongoDB elections run on a protocol version that borrows from Raft: each election happens in a numbered term, and a candidate wins by gathering votes from a majority of voting members within a term. A member only stands as a candidate, or votes for one, after comparing how current each node's oplog is — a node cannot be elected if a more up-to-date electable member exists. Priority does not override currency; it breaks ties and biases which current, electable member puts itself forward. With all three oplogs identical at initiation, currency is a wash, and priority becomes the deciding input: the priority-2 node stands and is elected, while the priority-1 and priority-0.5 members defer and vote for it.

The full rs.status() makes the quorum explicit. It reports a majority vote count of two against three voting members — the arithmetic of single-region-loss survival. Both secondaries name the UAE primary as their sync source, so replication flows from the system of record outward. And the recorded heartbeat round-trips are the cross-region latencies the topology was designed around: 97 milliseconds to one peer, 168 to the other. The cluster is measuring its own geography.

The most instructive evidence sits on the disaster-recovery node. West Europe was configured at priority 0.5 — the lowest in the set — and its electionParticipantMetrics record what it did during the election:

The fields read as a small confession of design intent. votedForCandidate: true — this node cast its vote. electionCandidateMemberId: 0 — it voted for member 0, the UAE node. priorityAtElection: 0.5 — it did so while holding the lowest priority in the set, declining to nominate itself. The lowest-priority member used its vote to place authority exactly where the architecture intended.

This is the difference between a design that is claimed and a design that is demonstrated. The intent — UAE primary, West Europe a deferential last resort — was written into the priority numbers, but the proof that the numbers did their job is in the telemetry of the node that stood to gain the most from ignoring them and did not. Topology-aware election is not an aspiration in a runbook here; it is a vote recorded in the cluster's own metrics.

The cluster is alive but has no users, and the localhost exception is still open — until that changes. This is the window in which the first principal is created.

x.509 authentication stores no passwords. The "user" is the certificate itself — specifically, the subject distinguished name embedded in it. MongoDB authenticates a client by completing mutual TLS, extracting the subject DN from the presented certificate, and looking that DN up as a username in the $external database, the namespace reserved for externally-authenticated identities. There is no server-side secret to store or leak. The identity is the certificate, validated by the same CA chain that secures the transport.

That makes one requirement absolute, and unforgiving: the username created in $external must equal the subject DN exactly as MongoDB derives it. MongoDB normalizes the DN to RFC 2253 form, ordered from the most specific component to the least — leaf to root. The match is byte-for-byte. A username assembled in any other component order is, as far as the lookup is concerned, a different string: the certificate authenticates at the TLS layer, the DN is extracted cleanly, and the connection is still refused with could not find user, because the name on the certificate and the name in the database are not the same name. The order is not cosmetic; it is part of the identity. The operator certificate's subject, in canonical order, is:

User creation is a write, and writes are served only by the primary; issued against a secondary it returns a not-primary error. With the bootstrap connection established to the primary over the localhost exception, the operator identity is created with that exact DN and granted root:

The response is { ok: 1 }, and with it the localhost exception closes. From this point the cluster authenticates every connection it receives. No unauthenticated path remains into it.

The proof of the exercise is the connection that failed at the close of the previous build. Same operator certificate, same CA chain, same subject DN — now it authenticates as a real user instead of being refused for not being one.

This connection differs from the bootstrap in three deliberate ways, each the production-correct posture rather than a bootstrap concession. It names x.509 as the authentication mechanism explicitly. It authenticates against $external, where the identity lives. And it addresses the node by its fully-qualified domain name, matching the certificate's subject alternative name — so hostname verification is enforced normally, without any of the loopback concessions the bootstrap required.

The shell prompts once for the certificate key's passphrase, completes the handshake, and lands on the primary's prompt. No error. connectionStatus confirms what happened:

The certificate subject is the authenticated identity. The role is root. No password crossed the wire, because none exists. The database that closed the previous build by refusing this exact certificate now grants it the highest privilege in the cluster — and the only thing that changed is that the identity the certificate already asserted now has a record to map to.

A final confirmation, from the authenticated session, captures the persisted member configuration the cluster will carry forward:

The output shows the three members with the priorities that shaped the election — 2, 1, and 0.5 — and the region tags that routing and sharding will later rely on. This is the durable record of the topology decision: not a comment in a runbook, but the live configuration the cluster re-reads on every election from here forward.

A point about operator credentials, because it is the kind of detail that separates a real deployment from a demonstration: the operator certificate and its key live on the operator's workstation. Where a certificate must be present on a node transiently — to bootstrap, or to verify — it is placed, used, and securely erased, so no node retains operator key material at rest. The certificates a node needs to be itself stay on the node; the operator's identity does not.

The cluster is now a cluster: three regions, one replica set, a primary placed where the architecture intended, a quorum that survives any single regional loss, and an identity model in which every connection is authenticated by a certificate and authorized by a role. That is a genuine financial-grade foundation, and it is real rather than theoretical — every command above ran against the running set.

It is also, by design, a foundation and not a finished access model. Exactly one identity exists, and it holds root. That is correct for a freshly bootstrapped cluster and entirely wrong as a steady state. A root certificate used for routine work is a root certificate exposed to routine risk; the rule that no human operates as root day to day, and no application ever holds administrative rights, cannot yet be enforced because there is nothing yet to enforce it with. The single all-powerful identity is the seed a least-privilege model grows from, not the model itself.

That model is the next piece of the build: role-based access control with custom roles, mapping distinct certificates to distinct, narrowly-scoped identities — an administrative role separate from a backup role separate from a monitoring role separate from the application's own constrained access. The cluster can now authenticate and authorize. The next step is to ensure it authorizes each identity no more than it actually needs.

The trust chain every one of these connections rode on was built earlier in this series: the two-tier PKI with the root in Azure Key Vault and the intermediate on the wire. The hardened nodes that became this cluster were the subject of the previous piece, the database that says no first — and this build is the answer to the refusal it ended on. The database said no, by name. Now it says yes — to exactly one name, signed and verified. Next, it learns to say yes carefully.