MongoDB ReplicaSet in K8S -- can't connect via port forward
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.
Introduction
When you port-forward to a MongoDB replica set member in Kubernetes, the initial connection succeeds but the client immediately disconnects or hangs. This happens because MongoDB replica sets return their internal hostnames (like mongo-0.mongo-headless.default.svc.cluster.local) in the isMaster response, and your local client cannot resolve these names. The fix is to either add the internal hostnames to your local /etc/hosts pointing to 127.0.0.1, connect with directConnection=true to skip replica set discovery, or use a mongos router instead.
Why It Fails
When a MongoDB client connects to a replica set member, it runs isMaster (or hello) to discover the full topology:
Your local machine cannot resolve mongo-0.mongo-headless.default.svc.cluster.local, so the driver drops the connection or times out trying to reach the other members.
Fix 1: Use directConnection=true
The simplest fix — tell the driver to treat this as a standalone connection:
In application code:
This bypasses replica set discovery entirely. The client talks only to the forwarded pod.
Fix 2: Add Hosts to /etc/hosts
If you need replica set features (like read preferences), map the internal DNS names to localhost:
Add to /etc/hosts:
Then connect with the replica set URI:
This approach is fragile but allows full replica set functionality locally.
Fix 3: Use a Temporary Pod
Run a client pod inside the cluster where DNS resolution works:
Typical StatefulSet Configuration
Initialize the Replica Set
Common Pitfalls
- Connecting without
directConnection=truevia port-forward: MongoDB drivers perform replica set discovery by default. The internal hostnames returned are unresolvable from outside the cluster, causing the driver to hang or disconnect after the initial handshake. - Port-forwarding to the Service instead of the Pod:
kubectl port-forward svc/mongo-headless 27017:27017routes to a random pod. For predictable connections, port-forward to the specific pod:kubectl port-forward pod/mongo-0 27017:27017. - Using
bind_ip: localhostin MongoDB config: By default, MongoDB only listens on localhost. In Kubernetes,--bind_ip_allis required so the pod accepts connections from other pods and from the port-forward tunnel. - Forgetting that writes only go to the primary: If you port-forward to a secondary member and try to write, MongoDB rejects the operation. Use
directConnection=truewith the primary pod, or users.status()to identify the current primary first. - Not initializing the replica set after deploying the StatefulSet: Deploying the StatefulSet starts three independent
mongodprocesses. They do not form a replica set until you runrs.initiate()on one member. Without initialization, each pod acts as a standalone instance.
Summary
- Port-forwarding to a MongoDB replica set fails because the client receives internal DNS names it cannot resolve
- Use
directConnection=truein the connection string to bypass replica set discovery (simplest fix) - Map internal hostnames in
/etc/hostsif you need full replica set features locally - Use a temporary pod inside the cluster for debugging with full DNS resolution
- Always use
--bind_ip_allwhen running MongoDB in Kubernetes containers
Related reading
- Monitor Cronjob running on GKE
- Monitor custom kubernetes pod metrics using Prometheus
- Monitoring HTTP Traffic in Kubernetes
- Mount add files to existing directory using configmap volume mount
- mongodb, replicates and error err not master and slaveOkfalse, code 13435
- MongoDB replication error NetworkInterfaceExceededTimeLimit and MaxTimeMSExpired - Causes and Fix
- Mount local directory into pod in minikube
- Mounting kubernetes volume with User permission

System Design Fundamentals
Build a strong foundation in designing scalable, reliable distributed systems.
View the courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.