How to build a long command string?

Question

I've a sequence of commands to be used along with lot of pipings, something like this:

awk '{ if ($8 ~ "isad") {print $2, $5, "SE"} else {print $2, $5, "ANT"} }' "/var/log/apache2/other_vhosts_access.log" | grep -v '127.0.0.1' | tr '[' '\0' | tr [:lower:] [:upper:] | sort -t' ' -s -k3

This basically filters the Apache log and prints three cols of info. First col is IP address, second is time, third is a string. The output could be sorted based on any column so, I need to use '-m' with sort for the time field. Sorting order could also be reversed.

I want a string to store the arguments to sort, and let the combined strings get executed. Something like this:

$AWK_CMD | $SORT_CMD $SORT_KEY $SORT_REV

where

SORT_CMD="sort -t' '"
SORT_KEY="-k$1"      # $1 from cmd line arg
SORT_REV="$2"        # Either -r or blank

Am able to build such strings; when I echo it, it looks fine. Problem is, how to execute it? I get some errors like:

awk '{ if ($8 ~ "isad") {print $2, $5, "SE"} else {print $2, $5, "ANT"} }' "/var/log/apache2/other_vhosts_access.log" | grep -v '127.0.0.1' | tr '[' '\0' | tr [:lower:] [:upper:] | sort -t' ' -s -k3
Running ...
awk: '{
awk: ^ invalid char ''' in expression

Please ignore if 'sort' doesn't sort properly, that can be fixed. I wish to know how I can build the final command string in steps. The ways I tried to execute this command in the script is by:

$final_cmd
`$final_cmd`

Edit: The script that I'm trying to use

KEY=$1      # Values {1, 2, 3}
REVERSE=$2  # Values {0, 1} 

SORT_CMD="sort -t' ' -s"
SORT_KEY="-k$KEY"
if [[ $KEY -eq 2 ]]
then
    SORT_KEY="-m -k2"
fi

if [[ $REVERSE -eq 1 ]]
then
    SORT_REV="-r"
else
    SORT_REV=
fi

final_cmd="awk '{ if (\$8 ~ \"isad\") {print \$2, \$5, \"SE\"} else {print \$2, \$5, \"ANT\"} }' $LOG_FILE '|' grep -v '127.0.0.1' '|' tr '[' '\0' '|' tr [:lower:] [:upper:] '|' $SORT_CMD $SORT_KEY $SORT_REV"

echo $final_cmd
echo "Running ..."
$final_cmd

Answer 1

The different parts can be put in shell functions :

awkfilter() {
    awk '{ if ($8 ~ "isad") {print $2, $5, "SE"} else {print $2, $5, "ANT"} }'
}

toupper() {
    tr '[:lower:]' '[:upper:]'
}

dosort() {
    sort -t' ' -s -k3
}

awkfilter < /var/log/apache2/other_vhosts_access.log |
  grep -vF 127.0.0.1 |
  tr '[' '\0' |
  toupper |
  dosort

then you could make things optional more easily :

dosort() {
    rev=
    if [ "$2" = "reverse" ]
    then
        rev=-r
    fi
    sort -t' ' -s -k"$1" ${rev+"$rev"}
}

When your command-line is starting to be really long, writing it in a script and breaking it in parts (with functions) is usually really helpful.

Answer 2

When you run awk '{ ... }' on a shell prompt or from a shell script, shell parses the quotes and passes the argument to awk w/o the quotes. What happens is that you somehow run it with the quotes in parameter.

Edit: with your update to question, what you need is sh -c "$final_cmd".

Answer 3

In all cases where you are looking at wanting to execute commands with options kept in variables, use a shell that supports arrays.

Your example code, translated into bash, for example (note also that expansions of variables etc. need to be double quoted),

#!/bin/bash

key=$1      # Values {1, 2, 3}
reverse=$2  # Values {0, 1} 

sort_key=()
if [ "$key" -eq 2 ]; then
    sort_key=( -m  )
fi

if [ "$reverse" -eq 1 ]; then
    key+='r'
fi

sort_key+=( -k "$key" )

set -x

awk '$8 ~ "isad" { print $2, $5, ($8 ~ "isad" ? "SE" : "ANT") }' "$log_file" |
grep -Fv '127.0.0.1'          |
tr '[:lower:][' '[:upper:]\0' |
sort -t ' ' -s "${sort_key[@]}" 

In this particular case, it may make more sense to use functions, but arrays would be useful for keeping e.g. indeterminate number of pathnames, or command line arguments created through modification of the script's own command line arguments.

Related:

• How can we run a command stored in a variable?
• Why does my shell script choke on whitespace or other special characters?
• When is double-quoting necessary?