When it comes to storage, the last thing you want to worry about is losing valuable data.  AWS S3 makes sure this loss doesn’t happen by making copies of your objects on multiple devices across multiple facilities. Object storage capability also lets you preserve, retrieve and restore every version of each bucket. AWS S3 can replicate data to any AWS region and offers integration with various AWS resources.

AWS S3 provides you with an object storage capability that enables you to store and manage all your data safely and securely. You can use AWS S3 to store and retrieve any amount of data anytime from EC2, and AWS resources.

In this tutorial, I will show you how you can transfer files from one S3 bucket to another without writing code or configuring any script.

Prerequisites

  • AWS Account
  • Knowledge of AWS S3
  • AWS Data Sync

Demo

1. In your S3 console, create your first bucket called the source bucket.

 

2. Create the second bucket, called the destination bucket.

 

3. We want to transfer files from the source bucket and destination bucket. To do this, upload any kind of files or folders to your source bucket.

 

4. Next, to transfer files from the source bucket to the destination bucket, we will be using AWS Data Sync. The AWS DataSync resources help you sync your workloads from one S3 bucket to another. In your AWS console, search for AWS DataSync and select it.

 


6. In the option under Create a data transfer task select Between AWS Storage services.

 

7. Configure your source location. On the dashboard menu, select Amazon S3 as the Location type.

  • Select the region which your source bucket belongs to and the name of your source bucket you created in step 1
  • Select Standard for your S3 storage class, if you want a subset of your data you can specify the folder path in the Folder section. If you want everything in your source bucket you can use a forward slash [/] to specify that. In your IAM role, you can click on Autogenerate and that will create a transfer for you.
 

8. Click on Next, for your destination location. Select Amazon S3 as your location type.

  • Select the region your destination bucket belongs to and the name of your destination bucket.
  • Use a forward slash on your folder structure, or you can also specify a base path if you want to and click on Auto-generate to create an IAM role for you.
 

9. Next, Configure your Task settings. The Task configuration is the process that will run the transfer. Give your Task a name.  

  • On your Task execution configuration select Verify only the data transferred

  • In your Data transfer configuration settings, select Transfer only data that has changed. The transfer only data that has changed lets you copy only data and metadata that differ between the source and destination, while the Transfer all data settings lets you Copy all source content to the destination, without comparing to existing content on the destination.

You also have the option of keeping deleted files and Overwriting files in your destination. You can also schedule the task if you want the jobs to be happening regularly.

 

10. When you are done, click on Next. You will see the summary of all your configuration, check if any configuration is missing. When you are done, click on Create task.

 

11. Back in your Task menu you will see your Data Sync Task available.

 

12. When you click on the Locations settings on the left pane, you will see the Location ID for each Host.

 

13. In your Task menu, you can see all the information required for your tasks.

 

14. Start your tasks by clicking on the Start menu and selecting Start with defaults.

  • In your Task Status, you can see that your tasks are running and executing.
 

15. When it’s completed, back in your S3 dashboard, select your destination bucket, you will see your files are transferred from your source bucket to the destination bucket.

 

16. When you are done with the transfer task, you can delete your DataSync task to avoid additional costs.

Congratulations 🎉 . You can see how fast and easy is it to transfer files between your S3 buckets.

If you have a hard time configuring and keeping up with AWS resources and struggle with the right talent to leverage AWS Services, contact us for our DevOps-as-a-Service offering. We will ensure you have the support to unblock your operations and scale up your software delivery to the next level.