Modify bash script to exclude text when reading file

Question

The following bash script sends an email with the PHP error logs for each day.

#!/bin/bash
 
# phperrlog v1.0
# by vladimir prelovac http://www.prelovac.com/vladimir/
#
# parse error logs on your server and send you daily updates to email
  
# configure options
  EMAIL="[email protected]"
   
  WORKDIR="/var/scripts"
  TAIL=50  # number of entries to send
 # IGNORE="/backup" # path to ignore
    
# script starts 'ere
     
 cd $WORKDIR
 rm phperrlog.txt 2>/dev/null
      
 LIST=$(ls /var/log/apache2/*-error.log)
 today=$(date +%Y-%m-%d)
       
 for i in $LIST
  do
    if [ -f $i ]; then
       time=$(date -r $i +%F)
       if [ "$time" == "$today" ]; then
         echo $i >>phperrlog.txt
         echo "---------------------------------" >>phperrlog.txt
         tail -n $TAIL $i >>phperrlog.txt
         echo -e "\n\n\n\n" >>phperrlog.txt
       fi
    fi
 done
                                              
  if [ -f  phperrlog.txt ]; then
    mail -s "PHPERRORLOG SCRIPT: server error logs - $today" $EMAIL  < phperrlog.txt
  fi

How can I modify this script so that it excludes all errors similar to this:

[Thu Aug 02 10:54:33 2012] [error] [client 12.345.67.89] Directory index forbidden by Options directive: /var/www/domain/public/templates/img/

[Thu Aug 02 11:25:35 2012] [error] [client 12.345.67.89] client denied by server configuration: /var/www/domain/public/templates/sidebar.tpl

I am more interested in the:

PHP Notice/Warning/Fatal Errors
File does not exist

Side remark: You can just use for i in /var/log/apache2/*-error.log instead of $LIST which avoid a lot of issues with whitespace etc. For your situation it's not that important but it's good to get in the habit of handling whitespace correctly. This also means that you should use "$i" instead of plain $i inside the for loop. — nohillside
– nohillside, Commented Aug 4, 2012 at 9:24

nohillside · Accepted Answer · 2012-08-04 10:04:00Z

grep can read patterns from a file

 -f file, --file=file
         Read one or more newline separated patterns from file.  Empty
         pattern lines match every input line.  Newlines are not considered
         part of a pattern.  If file is empty, nothing is matched.

In your case you have to decide whether you want to use a whitelist (list of patterns you want to see in your report) or blacklist (list of pattern you don't want to see). Once you have collected the relevant patterns replace tail -n $TAIL $i >>phperrlog.txt with

grep -f /path/to/whitelist.txt "$i" | tail -n ${TAIL:-50} >> phperrlog.txt

or

grep -v -f /path/to/blacklist.txt "$i" | tail -n ${TAIL:-50} >> phperrlog.txt

I would probably start with a blacklist and add additional patterns to it over time when I notice lines I don't want to see anymore. An initial blacklist could contain

Directory index forbidden by Options directive
client denied by server configuration

to mach your samples.

Thanks for this worked perfectly, just out of interest what does the: ${TAIL:-50} part do? How come we are minus 50?
If TAIL is undefined or null, the value 50 is substituted. I do this out of habit to make sure that the script doesn't fail in case TAIL is not set.

Nahuel Fouilleul · Accepted Answer · 2012-08-04 12:27:08Z

0

try replace redirection

mail -s "PHPERRORLOG SCRIPT: server error logs - $today" $EMAIL  < phperrlog.txt

with a process and a pipe, for example

grep -v '\[error\]' < phperrlog.txt | mail -s "PHPERRORLOG SCRIPT: server error logs - $today" $EMAIL

or

grep -v '\[error\]' phperrlog.txt | mail -s "PHPERRORLOG SCRIPT: server error logs - $today" $EMAIL

answered Aug 4, 2012 at 12:27

Nahuel Fouilleul

19.5k2 gold badges36 silver badges37 bronze badges

Collectives™ on Stack Overflow

Modify bash script to exclude text when reading file

2 Answers 2

2 Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Related