How to Replicate a Large Number of S3 Objects from AWS Global to AWS China

Recently, we faced a challenge to replicate around 38,000 objects from a global AWS S3 bucket to an S3 bucket located in AWS China.

The difficulty wasn’t just moving the data, it was figuring out how best to do so efficiently and securely.

Read on to find out how we approached the transfer and how we reduced the runtime from more than two hours to just one.

The starting point

Before getting to work, we considered which solution would be most effective and transfer the data more efficiently. We identified several possible approaches.

One option was to use AWS CLI commands (such as aws s3 cp or aws s3 sync), or to deploy a more permanent solution similar to the one described here.

In our case, we opted for something straightforward and sufficiently performant to handle the task efficiently.

How we optimised code

While building the Python code and transfer process, we considered a few key factors to ensure everything ran optimally:

Object streaming

Instead of downloading objects in full and then uploading them, data is streamed in small chunks. This approach is essential when working with files that are too large to fit into memory, and it significantly reduces memory pressure during transfers.

ThreadPoolExecutor

Rather than manually creating and managing threads, tasks are submitted to a ThreadPoolExecutor. This allows work to be distributed across a controlled pool of threads, helping to manage resource usage effectively and avoid the overhead of excessive thread creation.

Multipart upload and copy

Amazon S3’s multipart upload and multipart copy APIs allow large objects (over 5 GB) to be split into smaller parts that can be uploaded or copied individually. This enables parallel processing, improves performance, and keeps client-side memory usage low.

Resource optimisation

To further optimise resource usage, the code explicitly closes streams and removes large in-memory objects as soon as they are no longer needed. Garbage collection is also triggered periodically to reduce the risk of memory leaks during large batch operations.

Solution overview

For simplicity, the transfer was run through AWS CodeBuild, which pulled the script from the repository and executed it as part of the build.

Our setup spanned three AWS accounts:

• Account A: Source S3 bucket (global).
• Account B: CodeBuild environment executing the Python script.
• Account C: Destination S3 bucket (AWS China).

To handle access and authentication:

• We used role chaining so that CodeBuild could securely read from the source bucket.
• Since AWS role sessions expire after 55 minutes, our script automatically refreshed sessions mid-run.
• We stored the AWS China account credentials in AWS Secrets Manager (Account B) for secure retrieval during the transfer.

Experiment: Optimised vs. unoptimised Code

As a learning exercise, we put our optimisations to the test.

We ran two versions of the script:

• The unoptimised version used basic threading and streaming but lacked proper cleanup and memory handling.
• The optimised version included ThreadPoolExecutor, fine-tuned memory management, and properly closed streams.

Here’s what happened

The unoptimised version performed significantly worse than the optimised one – no surprises there. While we fully expected a performance gap, what caught us off guard was just how large it turned out to be.

The unoptimised implementation ran for 2 hours and 30 minutes before failing due to resource exhaustion (“Can’t start a new thread”), even on a machine with 4 vCPUs and 8 GB of RAM.

The answer, of course, was the optimised code. It completed the transfer in just 1 hour, successfully replicating all 38,000 objects to AWS China, making it roughly 75% faster than the unoptimised approach.

While this improvement might not seem dramatic when dealing with 140 GB of data or 38,000 objects, the impact becomes far more significant at larger scales. Replicating 10 TB of data under the same conditions would result in a vastly different outcome.

In a nutshell

Replicating data into AWS China introduces a few non-obvious challenges – from authentication and session management to performance at scale. As this exercise showed, getting those details right makes a material difference.

Thoughtful optimisation around streaming, concurrency, and memory management can turn a fragile, slow process into one that’s predictable, efficient, and scalable.

We’re here to help

If you’re facing similar challenges moving or replicating large volumes of files across AWS accounts or regions, we can help you design and implement an approach that’s built to scale.

Get in touch to talk through your setup and see what’s possible.