Uploading large filesΒΆ

When you want to upload a small number of large files (>1GB) then s5cmd would be a good choice. This tools uses parallelism in its uploads. Not only in terms of many concurrent uploads of a large number of files but it parallelises the upload of a single large file as well resulting in a very good performance. This can be done using the -c or --concurrency flag. This works as follows:

s5cmd --endpoint-url https://proxy.swift.surfsara.nl cp -c 64 --destination-region NL /path/to/my/huge/file s3://mybucket/mybigfile

In the example above the large file is concurrently uploaded in 64 chunks. But you may play around with this value to see what gives you the best performance.