In scenarios where you need to copy an Amazon DynamoDB table from one AWS account to another without using Amazon S3 as an intermediary, the process can be more complex but still achievable through AWS services like AWS Lambda and AWS Data Pipeline. This article explores the steps and technical details required for this approach, emphasizing the nuances and considerations involved.

Overview

Copying the data directly between different AWS accounts without S3 involves orchestrating several services. Here’s a general outline of the steps involved:

1. Prepare IAM Roles and Policies: Ensure both accounts have the necessary permissions.
2. Use DynamoDB Streams: Capture and process the table changes in near real-time.
3. Leverage AWS Lambda: Read from the DynamoDB stream and write to the destination table.
4. Initial Table Copy: Make an initial scan and write of the items to the new table.

Key Steps

1. Prepare IAM Roles and Policies

Source Account:

• Create an IAM role with permissions to read from the source DynamoDB table and invoke the Lambda function.
• Example IAM policy for the source role:

1  {
2    "Version": "2012-10-17",
3    "Statement": [
4      {
5        "Effect": "Allow",
6        "Action": [
7          "dynamodb:Scan",
8          "dynamodb:DescribeStream",
9          "dynamodb:GetRecords",
10          "dynamodb:GetShardIterator",
11          "dynamodb:ListStreams"
12        ],
13        "Resource": "arn:aws:dynamodb:region:account-id:table/SourceTable"
14      },
15      {
16        "Effect": "Allow",
17        "Action": "lambda:InvokeFunction",
18        "Resource": "arn:aws:lambda:region:account-id:function:FunctionName"
19      }
20    ]
21  }

Destination Account:

• Create an IAM role that allows writing to the DynamoDB table.
• Attach a policy to the Lambda function to allow write actions.
• Example IAM policy for the destination role:

1  {
2    "Version": "2012-10-17",
3    "Statement": [
4      {
5        "Effect": "Allow",
6        "Action": "dynamodb:PutItem",
7        "Resource": "arn:aws:dynamodb:region:account-id:table/DestinationTable"
8      }
9    ]
10  }

2. Use DynamoDB Streams

• Enable Streams: Enable DynamoDB Streams on the source table to capture item-level changes. Choose between NEW_AND_OLD_IMAGES or other options based on your need to capture complete data.
• Stream ARNs and Shards: Understand how to use stream ARNs and shard iterators needed for efficient data transfer through AWS Lambda.

3. Leverage AWS Lambda

• Create a Lambda Function: Set up a Lambda function in the source account to process stream records and write to the destination account's table.
• Code Example:

1  import boto3
2  from boto3.dynamodb.types import TypeDeserializer, TypeSerializer
3  
4  dynamodb = boto3.client('dynamodb', region_name='destination-region')
5  
6  def lambda_handler(event, context):
7      for record in event["Records"]:
8          if record["eventName"] in ["INSERT", "MODIFY"]:
9              new_image = record["dynamodb"]["NewImage"]
10              
11              # Transform the data to the format suitable for put_item
12              deserializer = TypeDeserializer()
13              deserialized_data = {k: deserializer.deserialize(v) 
14                                   for k, v in new_image.items()}
15              
16              # Write the item to the destination DynamoDB table
17              dynamodb.put_item(
18                  TableName='DestinationTable',
19                  Item={k: {'S': str(v) if isinstance(v, (int, float)) else v} 
20                        for k, v in deserialized_data.items()}
21              )

4. Initial Table Copy

• Perform an Initial Scan: Execute a one-time scan on the source table to transfer data to the destination table. This handles the pre-existing data before enabling real-time updates.
• Efficient Scanning: Use parallel scan or segmented strategies for handling large tables, ensuring you can manage throughputs effectively.

Considerations

• Data Consistency: Ensure that streamed data applies changes accurately to the destination. Handle errors or retries in Lambda to assure consistency.
• Throughput and Limits: Monitor read/write capacity units on both tables, considering any autoscaling or limit exceeding events.
• Error Handling: Implement robust error handling mechanisms in your Lambda function to log issues and allow retries.

Summary Table

By employing this structured approach, you can effectively replicate DynamoDB tables across AWS accounts without relying on S3 as an intermediary. This method is particularly useful when direct, near real-time synchronization is needed, and when keeping your architecture lightweight and integrated with the AWS ecosystem is a priority.