Uploading many small files

When you want to upload a folder containing many small files, then rclone and s5cmd are suitable tools.

rclone

Rclone uses by default 4 parallel transfers when copying a whole directory with files. For large files this may be fine but if the directory contains many small files (<10MB) it may be worthwhile to use a large number of concurrent transfers. For example, something like this:

rclone copy --transfers=100 -P /path/to/folder/with/small/files S3:mybucket

In this case we have used 100 concurrent transfers but you may play around with this value to see what works best for you. Here we have defined the remote S3 as in rclone.

s5cmd

s5cmd --numworkers 256 --endpoint-url https://proxy.swift.surfsara.nl cp --destination-region NL /path/to/folder/with/small/files s3://mybucket

Here numworkers which is the number of parallel transfers, is by default 256. This is already quite suitable for the many small files use case. But also here, you may play around a bit with this value to see what’s best.