Communities

Writing
Writing
Codidact Meta
Codidact Meta
The Great Outdoors
The Great Outdoors
Photography & Video
Photography & Video
Scientific Speculation
Scientific Speculation
Cooking
Cooking
Electrical Engineering
Electrical Engineering
Judaism
Judaism
Languages & Linguistics
Languages & Linguistics
Software Development
Software Development
Mathematics
Mathematics
Christianity
Christianity
Code Golf
Code Golf
Music
Music
Physics
Physics
Linux Systems
Linux Systems
Power Users
Power Users
Tabletop RPGs
Tabletop RPGs
Community Proposals
Community Proposals
tag:snake search within a tag
answers:0 unanswered questions
user:xxxx search by author id
score:0.5 posts with 0.5+ score
"snake oil" exact phrase
votes:4 posts with 4+ votes
created:<1w created < 1 week ago
post_type:xxxx type of post
Search help
Notifications
Mark all as read See all your notifications »
Q&A

Comments on Capture separate _and_ combined stdout/stderr

Post

Capture separate _and_ combined stdout/stderr

+5
−0

Using BASH, how can I redirect stdout and stderr each, to two separate files, simultaneously?

That's a mouthful, let me clarify a little:

I have a hypothetical script that may/will generate output on both stdout and stderr. I want to run the script and wind up with three output files:

  1. The contents of stdout only (>stdout.txt).
  2. The contents of stderr only (2>stderr.txt).
  3. The combined contents of stdout and stderr, interleaved as they would have been had they been printed to the terminal in real time (>both.txt 2>both.txt).

With Zsh you can do something like this with MULTIOS enabled:

script.txt >stdout.txt >>both.txt 2>stderr.txt 2>>both.txt

...but I need something that'll work on systems that don't have Zsh installed.

History
Why does this post require moderator attention?
You might want to add some details to your flag.
Why should this post be closed?

1 comment thread

"interleaved as they would have been had they been printed to the terminal" (4 comments)
"interleaved as they would have been had they been printed to the terminal"
Quasímodo‭ wrote almost 2 years ago · edited almost 2 years ago

This seems to be hard enough that even Debian's annotate-output doesn't do it.

Contrarily to what you claim, Zsh also fails to preserve the order in both.txt, as my tests confirmed. From https://zsh.sourceforge.io/Doc/Release/Redirection.html, "the shell opens the file descriptor as a pipe to a process that copies its input to all the specified outputs, similar to tee, provided the MULTIOS option is set, as it is by default". So the race condition is present.

I think you are out of luck here.

jaybeers‭ wrote almost 2 years ago

Aww, I didn't notice the order issue with Zsh; I just did a quick test to see if I had the syntax correct so I could copy/paste the same thing and try it in BASH and see if it worked; it wasn't until I had your solution that I got hopeful enough to actually write a little wrapper around it to really test the output. :)

Dang; well, again, thanks for looking into it!

jaybeers‭ wrote almost 2 years ago

Wait; your answer does give me an idea, though. Our systems are mostly CentOS so annotate-output might not be an option per se, but maybe I can rig something up based on what you've provided that pipes all the different outputs through a filter (awk, or something) that adds timestamps.

The goal is simply to be able to reconstruct the order of output lines, regardless of whether they came from stdout or stderr; if each line has a prepended timestamp, then even if the original output file has them out of order I can just sort the file. I just assumed it was a given that the shell would have to take care of that interleaving but perhaps, as The Dude might say, my thinking has gotten too uptight.

Of course, this solution might get a bit unfortunate for scripts that generate truly copious output, but for the common case I bet it'd be fine and even in those cases it should still work, even if managing the output files is a little more time-consuming.

Quasímodo‭ wrote almost 2 years ago

I'm afraid even that is not reliable either.

Something like cmd 2> >(timestamp_err) > >(timestamp_out) will inevitably cause a race condition since there are still two concurrent processes.

It is not impossible that, in practice, the order of magnitude of the "average delay" between input received and output written by the time stamping programs is much smaller than the average rate of output of cmd. But it is not something to be relied upon anyway...