ECS
health check
troubleshooting
container
AWS

How do I work out why an ECS health-check is failing?

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design

Introduction

An ECS health-check failure is not one thing. It can come from the container health check, the load balancer target group health check, or the task failing so early that neither check ever stabilizes. The fastest way to debug it is to identify which layer is declaring the task unhealthy, then inspect that layer directly.

Step 1: Identify Which Health Check Is Failing

With ECS services, two checks are common:

  • a container health check defined in the task definition
  • a target group health check from an Application Load Balancer or Network Load Balancer

These failures look similar in the console, but they mean different things. Start with ECS service events and task details.

bash
aws ecs describe-services \
  --cluster my-cluster \
  --services my-service

Look for messages about unhealthy targets, task replacement, failed container checks, or tasks never reaching a steady state.

If the service uses a load balancer, inspect target health too:

bash
aws elbv2 describe-target-health \
  --target-group-arn arn:aws:elasticloadbalancing:REGION:ACCOUNT:targetgroup/example/1234567890

That command often gives a clearer reason, such as wrong path, timeout, or connection failure.

Step 2: Check the Application From Inside the Task

If ECS or the load balancer says the health endpoint is failing, verify what the application actually returns.

If ECS Exec is enabled, connect to a running task and test locally:

bash
curl -i http://localhost:8080/health

You want to confirm three things:

  1. the application is listening on the expected port
  2. the health path is correct
  3. the endpoint returns a success status quickly enough

A surprising number of failures come from simple mismatches such as using / in the target group while the app exposes /health, or listening on 3000 while the service expects 8080.

Step 3: Inspect Logs and Exit Behavior

CloudWatch logs are often more useful than the health-check status itself. If the app crashes during startup, a failing health check is just a symptom.

Check for:

  • binding failures on the expected port
  • database connection timeouts during startup
  • migrations that take too long
  • unhandled exceptions before the HTTP server is ready

For container health checks, also inspect the exact command in the task definition. A shell-based check can fail because the binary is missing, the shell syntax is wrong, or the command returns a nonzero exit code even when the app is fine.

A typical container health check looks like this:

json
1{
2  "healthCheck": {
3    "command": ["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"],
4    "interval": 30,
5    "timeout": 5,
6    "retries": 3,
7    "startPeriod": 60
8  }
9}

If curl is not present in the image, this health check fails every time regardless of application state.

Step 4: Validate Networking and Load Balancer Wiring

If the app is healthy locally but the target group still fails, the problem is often network or routing configuration.

Check these items carefully:

  • security groups allow the load balancer to reach the task port
  • the target group points at the correct container port
  • the service is registered to the correct target group
  • the health check path and protocol match the application
  • the task is listening on all interfaces, not only on 127.0.0.1

That last point matters in containers. An app that listens only on loopback can respond to an in-container curl but still fail load balancer checks.

Step 5: Account for Startup Time

Some services are slow to become ready because they warm caches, run migrations, or connect to downstream systems. In those cases, the health-check parameters are too aggressive.

Look at:

  • 'startPeriod for container health checks'
  • target group health-check timeout and interval
  • ECS service healthCheckGracePeriodSeconds

If the app needs sixty seconds to become ready but the load balancer starts probing immediately, ECS may churn tasks forever even though the application would have stabilized with a grace period.

Common Pitfalls

The most common mistake is treating all ECS health-check failures as load balancer failures. Container and load balancer health are separate systems.

Another mistake is testing only from inside the container. That confirms app behavior, but not whether the load balancer can actually reach the task.

A third pitfall is using a health endpoint that depends on slow or fragile downstream services. Health checks should answer the readiness question quickly and predictably.

Summary

  • First determine whether the failure comes from the container health check or the load balancer target group.
  • Use describe-services, describe-target-health, and application logs together.
  • Test the endpoint locally inside the task and verify the port, path, and status code.
  • Validate security groups, target group wiring, and bind address configuration.
  • If the app is slow to start, adjust grace periods and health-check timing rather than letting ECS churn tasks endlessly.

Related reading
Course
Beginner
27 lessons
10 hours
System Design Fundamentals

Build a strong foundation in designing scalable, reliable distributed systems.

View the course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design

All Rights Reserved.