Uploading many small files¶
When you want to upload a folder containing many small files, then rclone and s5cmd are suitable tools.
rclone¶
Rclone uses by default 4 parallel transfers when copying a whole directory with files. For large files this may be fine but if the directory contains many small files (<10MB) it may be worthwhile to use a large number of concurrent transfers. For example, something like this:
rclone copy --transfers=100 -P /path/to/folder/with/small/files S3:mybucket
In this case we have used 100 concurrent transfers but you may play around with this value to see what works best for you. Here we have defined the remote S3 as in rclone.
s5cmd¶
s5cmd --numworkers 256 --endpoint-url https://proxy.swift.surfsara.nl cp --destination-region NL /path/to/folder/with/small/files s3://mybucket
Here numworkers which is the number of parallel transfers, is by default 256. This is already quite suitable for the many small files use case. But also here, you may play around a bit with this value to see what’s best.