Log rotating with a Bash script

First of all, you really should not reinvent the square wheel here. Your peers are probably against rotating the logs on daily schedule which automatically applies to all scripts in /etc/logrotate.d/ - this can be avoided by placing the script elsewhere.

Now, the standard approach to log rotation (that is implemented in logrotate) can be implemented by any other facility just as well. E.g. here's a sample implementation in bash:

MAXLOG=<maximum index of a log copy>
for i in `seq $((MAXLOG-1)) -1 1`; do
    mv "log."{$i,$((i+1))}    #will need to ignore file not found errors here
done 
mv log log.1    # since a file descriptor is linked to an inode rather than path,
                #if you move (or even remove) an open file, the program will continue
                #to write into it as if nothing happened
                #see https://stackoverflow.com/questions/5219896/how-do-the-unix-commands-mv-and-rm-work-with-open-files
<make the daemon reopen the log file with the old path>

The last item is done by sending SIGHUP or (less often) SIGUSR1 and having a signal handler in the daemon that replaces the corresponding file descriptor or variable. This way, the switch is atomic, so there's no interruption in logging availability. In bash, this would look like:

trap { exec &>"$LOGFILE"; } HUP

The other approach is to make the writing program itself keep track of the log size each time it writes to it and do the rotation. This limits your options in where you can write to and what rotation logic is to what the program itself supports. But it has the benefit of being a self-contained solution and checking the log size at each write rather than on schedule. Many languages' standard libraries have such a facility. As a drop-in solution, this is implemented in Apache's rotatelogs:

<your_program> 2>&1 | rotatelogs <opts> <logfile> <rotation_criteria>

You can also pipe your output thru Apache rotatelogs utility. Or following script:

#!/bin/ksh
#rotatelogs.sh -n numberOfFiles pathToLog fileSize[B|K|M|G]

numberOfFiles=10

while getopts "n:fltvecp:L:" opt; do
    case $opt in
  n) numberOfFiles="$OPTARG"
    if ! printf '%s\n' "$numberOfFiles" | grep '^[0-9][0-9]*$' >/dev/null; then
      printf 'Numeric numberOfFiles required %s. rotatelogs.sh -n numberOfFiles pathToLog fileSize[B|K|M|G]\n' "$numberOfFiles" 1>&2
      exit 1
    elif [ $numberOfFiles -lt 3 ]; then
      printf 'numberOfFiles < 3 %s. rotatelogs.sh -n numberOfFiles pathToLog fileSize[B|K|M|G]\n' "$numberOfFiles" 1>&2
    fi
  ;;
  *) printf '-%s ignored. rotatelogs.sh -n numberOfFiles pathToLog fileSize[B|K|M|G]\n' "$opt" 1>&2
  ;;
  esac
done
shift $(( $OPTIND - 1 ))

pathToLog="$1"
fileSize="$2"

if ! printf '%s\n' "$fileSize" | grep '^[0-9][0-9]*[BKMG]$' >/dev/null; then
  printf 'Numeric fileSize followed by B|K|M|G required %s. rotatelogs.sh -n numberOfFiles pathToLog fileSize[B|K|M|G]\n' "$fileSize" 1>&2
  exit 1
fi

sizeQualifier=`printf "%s\n" "$fileSize" | sed "s%^[0-9][0-9]*\([BKMG]\)$%\1%"`

multip=1
case $sizeQualifier in
B) multip=1 ;;
K) multip=1024 ;;
M) multip=1048576 ;;
G) multip=1073741824 ;;
esac

fileSize=`printf "%s\n" "$fileSize" | sed "s%^\([0-9][0-9]*\)[BKMG]$%\1%"`
fileSize=$(( $fileSize * $multip ))
fileSize=$(( $fileSize / 1024 ))

if [ $fileSize -le 10 ]; then
  printf 'fileSize %sKB < 10KB. rotatelogs.sh -n numberOfFiles pathToLog fileSize[B|K|M|G]\n' "$fileSize" 1>&2
  exit 1
fi

if ! touch "$pathToLog"; then
  printf 'Could not write to log file %s. rotatelogs.sh -n numberOfFiles pathToLog fileSize[B|K|M|G]\n' "$pathToLog" 1>&2
  exit 1
fi

lineCnt=0
while read line
do
  printf "%s\n" "$line" >>"$pathToLog"
  lineCnt=$(( $lineCnt + 1 ))
  if [ $lineCnt -gt 200 ]; then
    lineCnt=0
    curFileSize=`du -k "$pathToLog" | sed -e 's/^[  ][  ]*//' -e 's%[   ][  ]*$%%' -e 's/[  ][  ]*/[    ]/g' | cut -f1 -d" "`
    if [ $curFileSize -gt $fileSize ]; then
      DATE=`date +%Y%m%d_%H%M%S`
      cat "$pathToLog" | gzip -c >"${pathToLog}.${DATE}".gz && cat /dev/null >"$pathToLog"
      curNumberOfFiles=`ls "$pathToLog".[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]_[0-9][0-9][0-9][0-9][0-9][0-9].gz | wc -l | sed -e 's/^[   ][  ]*//' -e 's%[   ][  ]*$%%' -e 's/[  ][  ]*/[    ]/g'`
      while [ $curNumberOfFiles -ge $numberOfFiles ]; do
        fileToRemove=`ls "$pathToLog".[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]_[0-9][0-9][0-9][0-9][0-9][0-9].gz | head -1`
        if [ -f "$fileToRemove" ]; then
          rm -f "$fileToRemove"
          curNumberOfFiles=`ls "$pathToLog".[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]_[0-9][0-9][0-9][0-9][0-9][0-9].gz | wc -l | sed -e 's/^[   ][  ]*//' -e 's%[   ][  ]*$%%' -e 's/[  ][  ]*/[    ]/g'`
        else
          break
        fi
      done
    fi
  fi
done

I wrote a logrotee this weekend. I probably wouldn't if I've read @JdeBP's great answer about multilog before.

I focused on it being lightweight and being able to bzip2 its output chunks like:

verbosecommand | logrotee  \
  --compress "bzip2 {}" --compress-suffix .bz2 \
  /var/log/verbosecommand.log

There's a lot of to be done and tested yet, though.

You can leverage rotatelogs (docs here). This utility will decouple your script's stdout from the log file, managing the rotation in a transparent way. For example:

your_script.sh | rotatelogs /var/log/your.log 100M

will automatically rotate the output file when it reaches 100M (can be configured to rotate based on a time interval).

A simplest script could be something like this rotatelog.sh:

#! /bin/bash

#DATE_FMT="%Y%m%d-%H%M" # for testing
#DATE_FMT="%Y%m%d-%H"   # rotate each hour
DATE_FMT="%Y%m%d"       # rotate each day

if [ "$1" != "" ]
then
        f=$1
else
        f="rotatelog"
fi

p=$(date +${DATE_FMT})

r=$f-$p.log
exec 2>&1 > $r

while read l
do
        d=$(date +${DATE_FMT})
        if [ $p != $d ]
        then
                x=$r
                p=$d
                r=$f-$p.log
                exec 2>&1 > $r
                gzip $x
        fi
        echo $l
done

You can use like:

your_process | rotatelog.sh yout_log_path_pattern