Parallelize Shell or Bash Scripts Using Xargs
Learn how to speed up your shell scripts by running them in parallel with xargs.
Who doesn’t want to go faster? It is common to have a setup script that downloads various packages or tools. Here’s an example:
#!/bin/sh -ex # hugo static blog site go get github.com/spf13/hugo # https://github.com/client9/misspell ./godownloader-misspell 0.1.2 # shellcheck.net docker pull koalaman/shellcheck
It runs one line after another in serial. That’s slow.
But notice how each line doesn’t depend on any other (and comment and blank lines don’t matter). If you can write a script in such a way, then you really speed things up by running in parallel using xargs:
cat setup.sh | xargs -P4 -ICMD /bin/sh -exc CMD
That’s it. Your script could run 4x faster.
-P4 is number of processes to run in parallel. Since most of the time it is network calls, there is no harm in making this 8 or even 16.
-ICMD is … is the magic of mixing xargs and shell. You can use something else other than
CMD if you want. It’s the
-I that’s important and using the same word as the last arg.
You can run the setup in serial (normal) for debugging and one-time setups, and in parallel to speed up runs on CI/CD systems such as travis-ci.
There are other ways to parallelize scripts, but this is easiest.