Logging to files

Logging the output of a shell script is easy enough:

./script.sh 1> log.out 2> log.err

But if it the output is long and you find an error, you might want to know which output corresponds to it. Logging errors to one file, and output+errors to another is a little more tricky. And there are other potentially useful features:

  • Truncate the log file if it gets too long. Keep the last part, and ideally whole lines.
  • Add timestamps to each line in the log file.
  • Write output, error and the combination of them to files, and also show it to the terminal.

It's not rocket science, but why reinvent the wheel? Check out my the file in my new repository instead!

The functions have several options, which should be put at the start and not grouped (not "-ab" but "-a -b"):

  • -a Append to logs instead of overwriting.
  • -b Attempts to prevent mixed order (because of buffering), but doesn't
  • # always help, and can't be used with functions or groups of commands.
  • -c Write the command before any output (most useful when appending).
  • -t Truncate logs to 5k lines (before starting).
  • -u Do not add timestamps.
  • For log_err_all_named the first positional argument is the path to the logfile (without extension); for log_err_all the name is guessed from the command and should be omitted.

For example, log_err_all_named -a -t -c "logfilename" ./script arg1 arg2.

It has two problems, both of which I'm not sure are solvable. First, output is often buffered, so stdout and stderr may get mixed up a bit if they appear in very quick succession (the code tries to turn this off, but it doesn't help completely in my bash). Second, after this script, what goes to the terminal is all in stdout, but it's already logged at this point, so perhaps it doesn't matter too much. Ten points for you if you leave a (link to a) good solution to either one in the comments!

If you prefer to build your own version or you're curious, this is the essence:

content="$(tail -n $cnt logfile.out)"
printf "$content\n" > "logfile.out"
    stdbuf -oL -eL ./script.sh | ts '%Y-%m-%d,%H:%M:%.S ' | tee --append "logfile.out"
  } 3>&1 1>&2 2>&3 | ts '%Y-%m-%d,%H:%M:%.S ' | tee --append "logfile.err"
} |& tee --append "logfile.all" | sed 's/^.\{28\}//'


No comments yet

You need to be logged in to comment.