Introduction

AWS Lambda is a serverless computing service that allows you to run code without provisioning or managing servers. It automatically scales your application by running code in response to each trigger. One of the powerful use-cases of AWS Lambda is its ability to log data, which is often sent to AWS CloudWatch Logs. For enhanced searchability and analysis, many organizations choose to forward these logs to Amazon OpenSearch Service (formerly known as Amazon Elasticsearch Service).

In this article, we will demystify how you can subscribe multiple AWS Lambda logs to a single Amazon OpenSearch Service domain. By doing this, you streamline log management ensuring all log data is aggregated into one centralized repository, simplifying analysis, monitoring, and troubleshooting.

Prerequisites

Before proceeding, ensure that you have:

• An AWS account with IAM roles that have appropriate permissions for Lambda, CloudWatch, and OpenSearch.
• A configured Amazon OpenSearch Service domain.
• Basic understanding of AWS Lambda and its logging mechanism.

Architecture Overview

To efficiently subscribe multiple Lambda log groups to a single OpenSearch Service, we adopt the following architecture:

1. CloudWatch Log Groups: These are the starting point where Lambda functions send their logs.
2. AWS Lambda Function for Log Shipping: A custom Lambda function that fetches logs from CloudWatch and sends them to the OpenSearch domain.
3. Amazon OpenSearch Service: Centralized log management and analysis.

Below is the process flow:

• Log Emission: AWS Lambda functions emit logs to their respective CloudWatch log groups.
• Log Subscription: Each log group is assigned a subscription filter that triggers a custom Lambda.
• Log Forwarding: The custom Lambda processes received logs and forwards them to the OpenSearch domain.

Step-by-Step Guide

Step 1: Setting up CloudWatch Log Group Subscriptions

For each Lambda log group, you need to create a subscription filter that triggers a designated log-shipping Lambda function.

• CloudWatch Logs: It needs `logs:FilterLogEvents` permission.
• OpenSearch: It should have `es:ESHttpPost` and `es:ESHttpPut`.
• Trigger different Lambda functions and verify if their logs are being forwarded to the OpenSearch domain.
• Use OpenSearch dashboards or Kibana to visualize the logs.
• Batch Processing: Optimize your Lambda to process logs in batches to reduce the execution time and cost.
• Log Retention: Utilize OpenSearch Service's indices to manage and automate log retention policies.
• Security: Use encryption both in transit and at rest for your log data.