Kubernetes
MongoDB
ReplicaSet
Port Forwarding
Connectivity Issues

MongoDB ReplicaSet in K8S -- can't connect via port forward

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Introduction

When you port-forward to a MongoDB replica set member in Kubernetes, the initial connection succeeds but the client immediately disconnects or hangs. This happens because MongoDB replica sets return their internal hostnames (like mongo-0.mongo-headless.default.svc.cluster.local) in the isMaster response, and your local client cannot resolve these names. The fix is to either add the internal hostnames to your local /etc/hosts pointing to 127.0.0.1, connect with directConnection=true to skip replica set discovery, or use a mongos router instead.

Why It Fails

When a MongoDB client connects to a replica set member, it runs isMaster (or hello) to discover the full topology:

json
1{
2  "hosts": [
3    "mongo-0.mongo-headless.default.svc.cluster.local:27017",
4    "mongo-1.mongo-headless.default.svc.cluster.local:27017",
5    "mongo-2.mongo-headless.default.svc.cluster.local:27017"
6  ],
7  "setName": "rs0",
8  "primary": "mongo-0.mongo-headless.default.svc.cluster.local:27017"
9}

Your local machine cannot resolve mongo-0.mongo-headless.default.svc.cluster.local, so the driver drops the connection or times out trying to reach the other members.

Fix 1: Use directConnection=true

The simplest fix — tell the driver to treat this as a standalone connection:

bash
1# Port forward to one replica set member
2kubectl port-forward pod/mongo-0 27017:27017
3
4# Connect with directConnection=true
5mongosh "mongodb://localhost:27017/mydb?directConnection=true"

In application code:

python
1from pymongo import MongoClient
2
3# Python
4client = MongoClient("mongodb://localhost:27017/mydb?directConnection=true")
5db = client.mydb
6print(db.list_collection_names())
javascript
1// Node.js
2const { MongoClient } = require("mongodb");
3const client = new MongoClient(
4  "mongodb://localhost:27017/mydb?directConnection=true"
5);
6await client.connect();

This bypasses replica set discovery entirely. The client talks only to the forwarded pod.

Fix 2: Add Hosts to /etc/hosts

If you need replica set features (like read preferences), map the internal DNS names to localhost:

bash
1# Port forward all three members on different local ports
2kubectl port-forward pod/mongo-0 27017:27017 &
3kubectl port-forward pod/mongo-1 27018:27017 &
4kubectl port-forward pod/mongo-2 27019:27017 &

Add to /etc/hosts:

 
127.0.0.1 mongo-0.mongo-headless.default.svc.cluster.local
127.0.0.1 mongo-1.mongo-headless.default.svc.cluster.local
127.0.0.1 mongo-2.mongo-headless.default.svc.cluster.local

Then connect with the replica set URI:

bash
mongosh "mongodb://mongo-0.mongo-headless.default.svc.cluster.local:27017,\
mongo-1.mongo-headless.default.svc.cluster.local:27018,\
mongo-2.mongo-headless.default.svc.cluster.local:27019/mydb?replicaSet=rs0"

This approach is fragile but allows full replica set functionality locally.

Fix 3: Use a Temporary Pod

Run a client pod inside the cluster where DNS resolution works:

bash
1# Launch a temporary pod with mongosh
2kubectl run mongo-client --rm -it --image=mongo:7 -- \
3  mongosh "mongodb://mongo-0.mongo-headless:27017,\
4mongo-1.mongo-headless:27017,\
5mongo-2.mongo-headless:27017/mydb?replicaSet=rs0"

Typical StatefulSet Configuration

yaml
1apiVersion: apps/v1
2kind: StatefulSet
3metadata:
4  name: mongo
5spec:
6  serviceName: mongo-headless
7  replicas: 3
8  selector:
9    matchLabels:
10      app: mongo
11  template:
12    metadata:
13      labels:
14        app: mongo
15    spec:
16      containers:
17        - name: mongo
18          image: mongo:7
19          command: ["mongod", "--replSet", "rs0", "--bind_ip_all"]
20          ports:
21            - containerPort: 27017
22          volumeMounts:
23            - name: data
24              mountPath: /data/db
25  volumeClaimTemplates:
26    - metadata:
27        name: data
28      spec:
29        accessModes: ["ReadWriteOnce"]
30        resources:
31          requests:
32            storage: 10Gi
33---
34apiVersion: v1
35kind: Service
36metadata:
37  name: mongo-headless
38spec:
39  clusterIP: None
40  selector:
41    app: mongo
42  ports:
43    - port: 27017

Initialize the Replica Set

bash
1kubectl exec -it mongo-0 -- mongosh --eval '
2rs.initiate({
3  _id: "rs0",
4  members: [
5    { _id: 0, host: "mongo-0.mongo-headless:27017" },
6    { _id: 1, host: "mongo-1.mongo-headless:27017" },
7    { _id: 2, host: "mongo-2.mongo-headless:27017" }
8  ]
9})'

Common Pitfalls

  • Connecting without directConnection=true via port-forward: MongoDB drivers perform replica set discovery by default. The internal hostnames returned are unresolvable from outside the cluster, causing the driver to hang or disconnect after the initial handshake.
  • Port-forwarding to the Service instead of the Pod: kubectl port-forward svc/mongo-headless 27017:27017 routes to a random pod. For predictable connections, port-forward to the specific pod: kubectl port-forward pod/mongo-0 27017:27017.
  • Using bind_ip: localhost in MongoDB config: By default, MongoDB only listens on localhost. In Kubernetes, --bind_ip_all is required so the pod accepts connections from other pods and from the port-forward tunnel.
  • Forgetting that writes only go to the primary: If you port-forward to a secondary member and try to write, MongoDB rejects the operation. Use directConnection=true with the primary pod, or use rs.status() to identify the current primary first.
  • Not initializing the replica set after deploying the StatefulSet: Deploying the StatefulSet starts three independent mongod processes. They do not form a replica set until you run rs.initiate() on one member. Without initialization, each pod acts as a standalone instance.

Summary

  • Port-forwarding to a MongoDB replica set fails because the client receives internal DNS names it cannot resolve
  • Use directConnection=true in the connection string to bypass replica set discovery (simplest fix)
  • Map internal hostnames in /etc/hosts if you need full replica set features locally
  • Use a temporary pod inside the cluster for debugging with full DNS resolution
  • Always use --bind_ip_all when running MongoDB in Kubernetes containers

Course illustration
Course illustration

All Rights Reserved.