How to run a command on a list of files?
Suppose I have a list of files on standard input. These may be the output of find
, cat filelist.txt
or something else.
How can I run a command on each file in turn?
There are several options, like `xargs` and `for`. I'll leave those for other answers and only describe my favorite, GNU …
11mo ago
If I just used `find` to generate a list of files, then find's `-exec` argument is usually the way to run some other pr …
10mo ago
In some cases, when you want to apply a pipeline or a complex command to each file, I find it useful to use `while read` …
6mo ago
To run a command on a list of files, use a combination of commands and file glob patterns. For example, on Unix-based sy …
28d ago
To run a command on a list of files, you can use a command-line interface and a loop statement. For example, in Linux, y …
1mo ago
5 answers
The following users marked this post as Works for me:
User | Comment | Date |
---|---|---|
LAFK | (no comment) | Jun 21, 2023 at 05:59 |
-
If I just used
find
to generate a list of files, then find's-exec
argument is usually the way to run some other program on each file found.If you pipe the command to
xargs
, note that-P n
will run up to n commands in parallel. The best value of n will depend on the relative usage of your CPU and your storage system. -
If I have a program (say,
generate_lists
) that generates a list of files,for filename in $(generate_lists); do some_program "$filename" ; done
is usually helpful. Make sure you quote your use of
$filename
-- more of them have spaces than you'd think.
There are several options, like xargs
and for
. I'll leave those for other answers and only describe my favorite, GNU Parallel. You will have to install it separately, because unlike the inferior xargs
it usually does not come pre-installed.
Parallel can take a list of files on STDIN and run a command on them.
# Count lines in each file under current directory using "wc FILENAME"
find | parallel wc {}
Actually, it can also take the list from other sources, like CLI arguments.
# Apply wc to only txt files in current dir (not subdirs)
parallel wc {} ::: *.txt
It supports various ways of manipulating the filename:
# Rename foo.JPG files to foo.jpg
find | grep '.JPG$' | parallel mv {} {.}.jpg
As you can see, you can do any kind of manipulation you want on the file list before you pipe it to Parallel.
Parallel also supports many other features like parallelization, grouping output with color, a dry run mode and so on. See man parallel
for these.
Note that normally, these tools assume 1 file per line (delimited by \n
). If you have exotic filenames, such as those with \n
in the name, this won't work right. There are various workarounds for this, for example GNU find -print0
, fd --print0
or parallel --null
. You can find these in the man pages of the tools you use.
I personally prefer a workaround called "don't put newlines in filenames, and if you find any, delete them on sight, uninstall the program that made them, and yell at the developer in the issue tracker".
In some cases, when you want to apply a pipeline or a complex command to each file, I find it useful to use while read
:
find . -type d \
| while read d; do
find $d -type f -maxdepth 1 \
| head -n3;
done;
Hardened version:
find . -type d -print0 \
| while IFS= read -r -d '' d; do
find "$d" -type f -maxdepth 1 -print0 \
| head -z -n3;
done;
Portable filenames should only use characters from the portable filename character set https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_282. But if you need to run your script in filesystems that may be compromised, or use non-portable file names for other reasons, use the hardened version.
Read the corresponding documentation of the features:
- find(1):
- sh(1):
- read(1):
- head(1):
(The above shows the first 3 files in each directory.)
However, this forces an invocation of the command or pipeline for each path name. In general, xargs(1) is the simplest (and usually fastest) way:
find . -type f \
| xargs mv -t /tmp;
Hardened version:
find . -type f -print0 \
| xargs -0 mv -t /tmp;
(The above moves all files in .
to /tmp
.)
0 comment threads
To run a command on a list of files, use a combination of commands and file glob patterns. For example, on Unix-based systems, you can use a command like ``for file in *.txt;''. Run the $file command. Use "done" to loop through each file that matches the pattern and execute the desired command.
0 comment threads
To run a command on a list of files, you can use a command-line interface and a loop statement. For example, in Linux, you can use the "for" loop to apply a command to each file in a directory. You can also use wildcard characters like "*" to apply the command to a group of files that match a certain pattern.
0 comment threads