Communities

The Great Outdoors

Photography & Video

Scientific Speculation

Electrical Engineering

Languages & Linguistics

Software Development

Community Proposals

tag:snake search within a tag

answers:0 unanswered questions

user:xxxx search by author id

score:0.5 posts with 0.5+ score

"snake oil" exact phrase

votes:4 posts with 4+ votes

created:<1w created < 1 week ago

post_type:xxxx type of post

Notifications

Mark all as read See all your notifications »

Q&A

Posts Tags Edits

Comments on How to run a command on a list of files?

Parent

How to run a command on a list of files?

+4

−0

Suppose I have a list of files on standard input. These may be the output of find, cat filelist.txt or something else.

How can I run a command on each file in turn?

shell-scripting

posted 11 months ago

CC BY-SA 4.0

matthewsnyder‭

1096 reputation 79 53 177 73

Raw

Markdown

is a duplicate

This question has been asked before and has already been answered. It should be marked as a duplicate.

Please enter the URL of the proposed duplicate in the details field below.

not constructive

This question cannot be answered in a way that is helpful to anyone. It's not possible to learn something from possible answers, except for the solution for the specific problem of the asker.

There are several options, like `xargs` and `for`. I'll leave those for other answers and only describe my favorite, GNU …

11mo ago

If I just used `find` to generate a list of files, then find's `-exec` argument is usually the way to run some other pr …

11mo ago

In some cases, when you want to apply a pipeline or a complex command to each file, I find it useful to use `while read` …

7mo ago

To run a command on a list of files, use a combination of commands and file glob patterns. For example, on Unix-based sy …

1mo ago

To run a command on a list of files, you can use a command-line interface and a loop statement. For example, in Linux, y …

1mo ago

0 comment threads

Post

+4

−0

Worked for LAFK‭

The following users marked this post as Works for me:

User	Comment	Date
LAFK‭	(no comment)	Jun 21, 2023 at 05:59

If I just used find to generate a list of files, then find's -exec argument is usually the way to run some other program on each file found.

If you pipe the command to xargs, note that -P n will run up to n commands in parallel. The best value of n will depend on the relative usage of your CPU and your storage system.
If I have a program (say, generate_lists) that generates a list of files,
```
for filename in $(generate_lists); do some_program "$filename" ; done
```
is usually helpful. Make sure you quote your use of $filename -- more of them have spaces than you'd think.

posted 11 months ago

CC BY-SA 4.0

11mo ago by AdminBee‭

259 reputation 1 6 27 1

Copy Link

Raw

Markdown

3 comment threads

xargs(1) is usually better than looping (1 comment)

for loops on command substitutions have quoting issues (1 comment)

Usually "number of cores" is a good starting value for `n`. (2 comments)

dsr‭ wrote 11 months ago

Sometimes? It really depends. Here are some reasons why it wouldn't be:

Huge number of cores, low I/O: you have a 192 core box but the filesystem is mounted over a transPacific link. You might want n=1 or 2 or maybe 3.
Simple task, great I/O: you're going to stat each file and report on metadata, and they are all stored in a tmpfs. Even if you only have 2 cores, you might want to try n=4.

Those are extremes, of course. Most cases will be somewhere in between, and fairly often this is going to be a one-off where there is no point in re-running for performance gain -- you have the answer!