How To Download An Entire S3 Bucket

The Amazon S3 console only allows you to download a single object with each request. To download an entire S3 bucket, you will need other tools. Learn more about AWS CLI, this is the most popular method.

Download An Entire S3 Bucket With AWS CLI

AWS CLI is the official command line interface (CLI) for AWS services. It offers a unified tool for managing, controlling, and implementing automation with them.

The program is open-source and runs on the terminal of your operating system. The most recent version is AWS CLI v2 and supports all major platforms (Linux, macOS, and Windows).

Install

To install AWS CLI on Linux, run this command (you should have unzip installed on your system):

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

To install AWS CLI on macOS:

• Download this PKG file: https://awscli.amazonaws.com/AWSCLIV2.pkg
• Open the downloaded installer and follow its instructions.

To install AWS CLI on Windows:

• Download this MSI file: https://awscli.amazonaws.com/AWSCLIV2.msi
• Run the installer.

To confirm the installation run this command on a terminal of your system:

aws --version

It should print versions of aws-cli, Python, and your operating system like this:

aws-cli/2.7.24 Python/3.8.8 Linux/4.14.133-113.105.amzn2.x86_64 botocore/2.4.5

Syncing With AWS CLI

The sync subcommand can be used to sync an S3 bucket to your local file system.

aws s3 sync s3://yourbucket path

Replace yourbucket with the name of your S3 bucket and path with the path to the folder you want to store its objects.

This is a syncing operation, meaning the command only copies new and updated objects to the destination (your local folder in this case). AWS CLI uses the size and last modified time to decide whether the object on the S3 bucket and your local copy are the same.

Keep in mind that when syncing objects from S3, AWS CLI will change the last modified time of every file to that of the S3 copy.

For example, this command will sync objects from the learnsharit bucket to the current working directory:

aws s3 sync s3://learnshareit .

If your bucket has two files (tutorial1.pdf and tutorial2.pdf), which aren’t available in the current directory, the output will look like this:

download: s3://learnshareit/tutorial1.pdf to tutorial1.pdf
download: s3://learnshareit/tutorial2.pdf to tutorial2.pdf

Advanced Syncing Options

Like other syncing tools, AWS CLI also gives you some options to customize the operation.

For instance, use the –exclude parameter flag when you want to omit files that match specific patterns. This can be used to prevent AWS CLI from downloading certain file types.

This command will ignore every .jpg file on your S3 bucket and won’t download them to your local folder:

aws s3 sync s3://learnshareit .  --exclude "*.jpg"

You can also modify the pattern to match other rules. This will exclude the Python folder and everything in it:

aws s3 sync s3://learnshareit .  --exclude "Python/*"

The –include parameter, on the other hand, prevents AWS CLI from excluding your desired files.

You can add as many –exclude and –include arguments to the same command as you like. When there are multiple filters like that, the rules appearing later always take precedence over the rules before them.

For instance, this command will exclude every object except for .jpg files:

aws s3 sync s3://learnshareit .  --exclude "*" --include "*.jpg"

But AWS CLI will ignore every file if you change the order of the rules:

aws s3 sync s3://learnshareit .  --include "*.jpg" --exclude "*"

Summary

You can download an entire S3 bucket by using AWS CLI. Developed by Amazon, this tool provides additional management capabilities, including syncing the whole bucket to your local folder.

Maybe you are interested:

• Error: non-static variable count cannot be referenced from a static context
• Error “Could not find module ‘@angular-devkit/build-angular’”
• Your project requires a newer version of the kotlin gradle plugin