In order to rsync a huge chunk of data (containing considerably large number of smaller files), the best option one can have, is to run multiple instances of
rsyncs in parallel. This seems to be pretty effective, but at the cost of high load average, more I/O oparations and network bandwidth utilization. So as to parallelize multiple
rsync commands, one might use xargs or a series of rsync commands run in the background using &. But, over all of those alternatives, I would prefer GNU Parallel, a utility used to execute jobs in parallel. It is a single command that can replace certain loops in your code or a sequence of commands run in background.Read more at YourOwnLinux
