AWS Data Transfer Hub

When you're tasked with migrating a huge number of buckets with metadata included, your heart might skip a beat. That happened to me.
Fortunately, AWS offers a less-known but incredibly useful service—Data Transfer Hub (DTH) — which turned out to be a lifesaver.

The Challenge

The task was to migrate files along with their metadata from a different cloud provider to AWS. Luckily, DTH supported this provider, making the migration process significantly easier in the end.

Deploying Data Transfer Hub

Deploying DTH is straightforward. You simply need to deploy the stack in your preferred AWS region and configure basic settings such as email and credentials. Once that’s done, the rest of the process is smooth sailing.

DTH deploys several AWS services to support its operations, including:

• Lambda Functions (to manage automation and processing)
• Amazon CloudFront (for content delivery)
• Amazon S3 Buckets
• Amazon Cognito (for authentication)
• AWS AppSync (for managing APIs)
• Amazon DynamoDB (for metadata storage)
• Amazon ECS
• IAM Roles
• VPC and ACLs (for network security)
• AWS Step Functions (for orchestrating workflows)
• Amazon SNS (for notifications and alerts)

This extensive list highlights the complexity of the underlying infrastructure that makes DTH such a powerful tool.

When Stack is succesfully deployed, go to Resources and click on the link provided, it will open up the UI for Sign In, as shown in the image below. Sign in with the temporary credentials set in one of previous steps.

When you are in, choose a Start new transfer task

Choose the S3 since, we are migrating the folders and files in this case :)

You can track the status of task in the new portal.

The logs and graphs are also available, if needed.

When task has been finished, or face some issues, SNS notification will be received. An example is below:

More informations and deployment steps can be found here:
https://github.com/aws-solutions/data-transfer-hub
https://aws.amazon.com/solutions/implementations/data-transfer-hub/

Configuring Data Transfers

When transferring files, beside selecting the source and destination buckets, DTH allows you to configure additional parameters for a tailored migration experience.

When transferring files from other cloud storage providers to Amazon S3, you have multiple options:

• Batch File Migration: You can create a list of folders and upload it as a .txt file to a dedicated S3 bucket. DTH will use this list to migrate only the specified folders to S3.
• Prefix-Based Transfer: This option enables migrating individual folders while preserving their structure by specifying a prefix. This can be used in cases when huge folder is being migrated or when only one is needed to be migrated.

Challenges Faced During Migration

While the migration process was generally smooth, I encountered some difficulties. One major issue was handling a 4TB bucket—it wasn’t possible to transfer everything at once. Instead, I had to transfer the data in blocks of folders.

The reason for this limitation appears to be related to DTH's internal processing. Although there was available memory, certain folders were not transferred successfully. My suspicion is that this is due to Finder Memory limitations, but further investigation is needed.

Informations that may be useful when choosing Finder Memory

The table below presents data collected from my tests, which may help you select the appropriate Finder Memory. However, I recommend gaining a deeper understanding of performance-related factors to optimize task performance effectively.

What if I need to move additional data later? How to avoid duplicating files?

For that purpose, please choose option to compare data before transfer to avoid duplicates:

It will require more time, but is helpful in those situations.

Conclusion

AWS Data Transfer Hub is an excellent tool for large-scale migrations, offering flexibility and automation to ease the process. While it comes with some limitations, careful planning and testing can help overcome potential challenges. If you find yourself facing a daunting migration task, DTH might just be the hidden gem you need.

Author Of article : Aleksandra Ljuboje Read full article