Introduction

CloudWatch Logs is an AWS service that lets you monitor, store, and access log files from AWS resources such as EC2 instances, Lambda functions, and CloudTrail. Log groups serve as containers for log streams, which in turn contain individual log events. When you need to clean up logging data, whether to reduce costs, reset a test environment, or remove stale streams, you may want to delete all log streams within a specific log group. This article shows how to accomplish this efficiently using the AWS Command Line Interface.

Prerequisites

Before running the commands, make sure the following are in place:

• AWS CLI installed: You need version 2 of the AWS CLI. Install it via the official guide.
• IAM permissions: The IAM identity running the commands needs logs:DescribeLogStreams to list log streams and logs:DeleteLogStream to delete them.
• CLI configured: Run aws configure to set your access key, secret key, default region, and output format.

Step 1: List Log Streams in a Log Group

Before deleting anything, verify which log streams exist. Use the describe-log-streams command to list them.

1aws logs describe-log-streams \
2  --log-group-name "/aws/lambda/my-function" \
3  --query "logStreams[*].logStreamName" \
4  --output text

The --query flag uses JMESPath to extract just the stream names. The --output text flag returns them as tab-separated values, which is easier to process in a script than JSON.

Step 2: Delete a Single Log Stream

The delete-log-stream command removes one stream at a time.

aws logs delete-log-stream \
  --log-group-name "/aws/lambda/my-function" \
  --log-stream-name "2024/01/15/[$LATEST]abc123"

This is useful for targeted cleanup, but for bulk deletion you need to automate the process.

Step 3: Delete All Log Streams with a Bash Script

The following script retrieves all log stream names from a log group and deletes them one by one.

1#!/bin/bash
2
3LOG_GROUP="/aws/lambda/my-function"
4
5LOG_STREAMS=$(aws logs describe-log-streams \
6  --log-group-name "$LOG_GROUP" \
7  --query "logStreams[*].logStreamName" \
8  --output text)
9
10if [ -z "$LOG_STREAMS" ]; then
11  echo "No log streams found in $LOG_GROUP"
12  exit 0
13fi
14
15for STREAM in $LOG_STREAMS; do
16  echo "Deleting stream: $STREAM"
17  aws logs delete-log-stream \
18    --log-group-name "$LOG_GROUP" \
19    --log-stream-name "$STREAM"
20done
21
22echo "All log streams deleted from $LOG_GROUP"

Save this as delete_log_streams.sh, make it executable with chmod +x delete_log_streams.sh, and run it. The script first checks if there are any streams to delete, then iterates through each one.

Handling Pagination for Large Log Groups

The describe-log-streams command returns a maximum of 50 streams per call by default. If your log group has more than 50 streams, you need to handle pagination.

1#!/bin/bash
2
3LOG_GROUP="/aws/lambda/my-function"
4NEXT_TOKEN=""
5
6while true; do
7  if [ -z "$NEXT_TOKEN" ]; then
8    RESPONSE=$(aws logs describe-log-streams \
9      --log-group-name "$LOG_GROUP" \
10      --output json)
11  else
12    RESPONSE=$(aws logs describe-log-streams \
13      --log-group-name "$LOG_GROUP" \
14      --next-token "$NEXT_TOKEN" \
15      --output json)
16  fi
17
18  STREAMS=$(echo "$RESPONSE" | jq -r '.logStreams[].logStreamName')
19
20  for STREAM in $STREAMS; do
21    echo "Deleting: $STREAM"
22    aws logs delete-log-stream \
23      --log-group-name "$LOG_GROUP" \
24      --log-stream-name "$STREAM"
25  done
26
27  NEXT_TOKEN=$(echo "$RESPONSE" | jq -r '.nextToken // empty')
28  if [ -z "$NEXT_TOKEN" ]; then
29    break
30  fi
31done
32
33echo "Done."

This version uses jq to parse the JSON response and extract the nextToken for pagination. It continues fetching and deleting until there are no more pages.

Parallel Deletion with xargs

For log groups with hundreds or thousands of streams, deleting one at a time can be slow. You can speed things up using xargs with parallel execution.

1aws logs describe-log-streams \
2  --log-group-name "/aws/lambda/my-function" \
3  --query "logStreams[*].logStreamName" \
4  --output text | tr '\t' '\n' | \
5xargs -P 10 -I {} aws logs delete-log-stream \
6  --log-group-name "/aws/lambda/my-function" \
7  --log-stream-name "{}"

The -P 10 flag runs up to 10 delete operations in parallel. Adjust this number based on your API rate limits.

Alternative: Delete the Entire Log Group

If you want to remove all streams and do not need to keep the log group itself, deleting the entire log group is simpler and faster.

aws logs delete-log-group \
  --log-group-name "/aws/lambda/my-function"

This removes the log group and all of its streams in a single API call. Be aware that you will need to recreate the log group if your service writes to it again (though many AWS services auto-create log groups).

Common Pitfalls

• Irreversibility: Deleting log streams permanently removes the data. Export logs to S3 first if you need a backup. Use aws logs create-export-task for this.
• API rate limits: AWS throttles CloudWatch Logs API calls. If you delete many streams rapidly, you may receive ThrottlingException errors. Add a short delay between calls or use exponential backoff.
• Special characters in stream names: Lambda log stream names contain characters like [$LATEST] and forward slashes. Always quote the stream name variable in your script to prevent shell expansion.
• Wrong region: If you do not see the expected log streams, verify your CLI is configured for the correct region with aws configure get region.
• Pagination oversight: Forgetting to handle pagination means you only delete the first page of results (up to 50 streams), leaving the rest intact.

Summary

Deleting all log streams from a CloudWatch log group requires listing the streams with describe-log-streams and then calling delete-log-stream for each one. For large log groups, handle pagination and consider parallel deletion with xargs. If you do not need to preserve the log group itself, delete-log-group is the simplest option. Always export important logs to S3 before deletion and be mindful of API rate limits when performing bulk operations.