Comments on How to run a command on a list of files?
Parent
How to run a command on a list of files?
Suppose I have a list of files on standard input. These may be the output of find, cat filelist.txt or something else.
How can I run a command on each file in turn?
Post
There are several options, like xargs and for. I'll leave those for other answers and only describe my favorite, GNU Parallel. You will have to install it separately, because unlike the inferior xargs it usually does not come pre-installed.
Parallel can take a list of files on STDIN and run a command on them.
# Count lines in each file under current directory using "wc FILENAME"
find | parallel wc {}
Actually, it can also take the list from other sources, like CLI arguments.
# Apply wc to only txt files in current dir (not subdirs)
parallel wc {} ::: *.txt
It supports various ways of manipulating the filename:
# Rename foo.JPG files to foo.jpg
find | grep '.JPG$' | parallel mv {} {.}.jpg
As you can see, you can do any kind of manipulation you want on the file list before you pipe it to Parallel.
Parallel also supports many other features like parallelization, grouping output with color, a dry run mode and so on. See man parallel for these.
Note that normally, these tools assume 1 file per line (delimited by \n). If you have exotic filenames, such as those with \n in the name, this won't work right. There are various workarounds for this, for example GNU find -print0, fd --print0 or parallel --null. You can find these in the man pages of the tools you use.
I personally prefer a workaround called "don't put newlines in filenames, and if you find any, delete them on sight, uninstall the program that made them, and yell at the developer in the issue tracker".
0 comment threads