With the information technology element finding its roots in every financial organization and across all industries, strong storage capacity forms the backbone for availability, durability, and scalability. Among these, Amazon S3 is one of the most popular services to meet these needs. As enterprises expand geographically, the need for data replication between locations starts to be felt significantly. Hence, the need arose to develop Cross-Region Replication in Amazon S3, where data replication between one bucket sourced from one AWS region to another bucket in a different AWS region is allowed.
The article will describe how to configure Cross-Region Replication in Amazon S3 using Terraform, an IaC software. It gives a general overview of how to set up SLAs, discusses why one would implement cross-region replication, and shows how to get a proof of concept running.
What Is Cross-Region Replication?
CRR stands for Cross-Region Replication, an Amazon S3 feature that replicates objects from one bucket to another bucket in a different region. This replication assists with compliance regulations, disaster recovery, and performance optimization by storing data closer to users.
Key Features of CRR
- It automatically replicates objects between buckets.
- Regions are independent; that is, source and destination buckets can be from different regions.
- Rules: This allows granular control over what gets replicated.
- Encryption and Tagging Support: Supports the replication of encrypted data and object tags.
Why Use Terraform for CRR?
Terraform enables cloud resource management through declarative configuration. While AWS Management Console has default replication policies, Terraform allows version-controlled templates.
Benefits of Terraform
- Consistency. The process reduces the chances of set-up variations.
- Automation. Enabling quick implementation, updating, and tearing down.
- Version control. It simplifies rollbacks and history tracking.
Prerequisites
Before setting up CRR with Terraform, the following should be in place:
- An AWS account with S3 and IAM permissions
- Terraform locally installed
- AWS CLI configured to use for authentication
Implementation
Step 1: Define Your S3 Buckets
You should create at least two buckets: source and destination buckets.
resource "aws_s3_bucket" "source_bucket" {
bucket = "my-source-bucket-example"
acl = "private"
versioning {
enabled = true
}
tags = {
Name = "SourceBucket"
Environment = "Production"
}
}
resource "aws_s3_bucket" "destination_bucket" {
bucket = "my-destination-bucket-example"
acl = "private"
versioning {
enabled = true
}
tags = {
Name = "DestinationBucket"
Environment = "Production"
}
}
Step 2: Create an IAM Role for Replication
To use CRR, there must be an IAM role with the privileges that CRR requires. Let’s create one:
resource "aws_iam_role" "replication_role" {
name = "s3-replication-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
Service = "s3.amazonaws.com"
}
}
]
})
}
resource "aws_iam_policy" "replication_policy" {
name = "s3-replication-policy"
description = "Allows S3 to replicate objects"
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = [
"s3:GetObject",
"s3:ListBucket",
"s3:ReplicateObject",
"s3:ReplicateDelete",
"s3:ReplicateTags"
]
Effect = "Allow"
Resource = [
"arn:aws:s3:::my-source-bucket-example/*",
"arn:aws:s3:::my-source-bucket-example"
]
},
{
Action = "s3:PutObject"
Effect = "Allow"
Resource = "arn:aws:s3:::my-destination-bucket-example/*"
}
]
})
}
resource "aws_iam_role_policy_attachment" "replication_role_attachment" {
role = aws_iam_role.replication_role.name
policy_arn = aws_iam_policy.replication_policy.arn
}
Step 3: S3 Replication Configuration
Configure a replication configuration to associate source and destination buckets.
resource "aws_s3_bucket_replication_configuration" "replication_config" {
bucket = aws_s3_bucket.source_bucket.id
role = aws_iam_role.replication_role.arn
rule {
id = "ReplicationRule1"
status = "Enabled"
filter {
prefix = ""
}
destination {
bucket = aws_s3_bucket.destination_bucket.arn
storage_class = "STANDARD"
}
}
}
Step 4: Init and Apply Configuration
Execute the Terraform configuration using these commands:
terraform init
terraform apply
During the execution of terraform apply, just review the plan and confirm its execution.
How to Test CRR Setup
Create an object in the source bucket and see if it gets replicated in the destination bucket.
# Put a file in the source bucket
aws s3 cp test-file.txt s3://my-source-bucket-example/
# Check if file is there in the destination bucket
aws s3 ls s3://my-destination-bucket-example/
Advantages of Cross-Region Replication
- Disaster recovery. Protect data by mirroring it across different geographical regions.
- Compliance. Meet regulatory requirements for data location and backup.
- Improved performance. Reduce latency by storing data closer to users.
- Backup automation. Manage backups and archives without additional tools or scripts.
Troubleshooting Tips
- Permission errors. Verify the IAM role has correct permissions on source and destination buckets.
- Versioning not enabled. Make sure versioning is enabled on both source and destination buckets.
- Replication delays. Replication is eventually consistent. Monitor replication using CloudWatch.
Conclusion
Amazon S3 Cross-Region Replication is a very powerful feature for any business to achieve appropriate data redundancy and compliance. Terraform enables you to automate and streamline the implementation process so the infrastructure setup remains consistent and scalable.
In this tutorial, we have learned how to implement CRR using Terraform, from bucket configuration to replication rule definitions. Based on the method described in this article, you can design your optimal storage architecture to handle the needs of a modern enterprise application.
Source:
https://dzone.com/articles/amazon-s3-cross-region-replication-terraform