BASH variable inside AWK that is itself an AWK command

Question

I want to use a BASH variable inside AWK that is an AWK command. Here is my test code that I plan to make more complicated if I can find a solution to this problem.

exptransform=(
    'print \"test\"'
)
awk -v awk_exptransform="${exptransform}" '                                                                                                                                                  
    BEGIN{                                                                                                                                                                                   
        printf ("Test.\n")                                                                                                                                                                   
        printf ("This is another test: %s\n",awk_exptransform)                                                                                                                               
        awk_exptransform                                                                                                                                                                     
    }                                                                                                                                                                                        
'

This gives successful output for the first two lines in the begin statement. The third line returns nothing. How do I get AWK to execute the contents of the variable as a command?

I have done many searches on the Internet and on this site with a variety of search terms. I have read the relevant section of sed & awk (Dougherty, & Robbins, 1997). What am I missing? Is this possible?

Here is what I am trying to do:

cat test.txt                                                                                                                                                                                 
exptransform=(                                                                                                                                                                               
    'print $9 - $8'                                                                                                                                                                          
    'print 10^($9 - $8)'                                                                                                                                                                     
)                                                                                                                                                                                            
for (( h=0 ; h<${#exptransform[@]} ; ++h )) ; do                                                                                                                                             
    awk -v awk_exptransform="${exptransform[h]}" 'BEGIN{FS = "\t"} {if ( $4~/1/ && $5~/2/ && $6~/1/ && $7~/1/ && $2!~/20160630_0609/ && $2!~/20170208_2324/ && $2!~/20160804_0635/) awk_exptransform}' test.txt                                                                                                                                                                          
done                                                                                                                                                                                         

Desired output:

subject run     day_1_date      include night   tone_delivered  sleep_arousal   tone_1  tone_n                                                                                               
00105   20170413_0406   20170411        1       2       1       1       70      105                                                                                                          
00105   20170413_0535   20170411        1       2       1       1       70      70                                                                                                           
35                                                                                                                                                                                           
0                                                                                                                                                                                            
99999999999999996863366107917975552                                                                                                                                                          
1                                                                                                                                                                                            

Actual output:

subject run     day_1_date      include night   tone_delivered  sleep_arousal   tone_1  tone_n                                                                                               
00105   20170413_0406   20170411        1       2       1       1       70      105                                                                                                          
00105   20170413_0535   20170411        1       2       1       1       70      70                                                                                                           

Sincerely,

Dante Picchioni

Answer 1

One option would be to:

• convert your code snippets into functions
• place those functions in their own files
• place the main code in its own file, too
• make an awk call with multiple -f flags

Consider:

$ cat exptransform.awk
function awk_exptransform() {
    print "test (inside function)"
}

$ cat main.awk
BEGIN { printf ("Test.\n")
        printf ("This is another test: %s\n","awk_exptransform")
        awk_exptransform()
      } 

OP's current awk call will now become:

$ awk -f exptransform.awk -f main.awk
Test.
This is another test: awk_exptransform
test (inside function)

---

If using GNU awk you can make use of 'include' files which works in a comparable fashion but with a builtin method for referencing your own library of awk scripts.

There are a few ways to slice-n-dice this ...

Using the -i command line flag:

$ awk -i exptransform.awk -f main.awk
Test.
This is another test: awk_exptransform
test (inside function)

Using the @include command within the main script:

$ cat main.include.awk
@include "exptransform.awk"
BEGIN { printf ("Test.\n")
        printf ("This is another test: %s\n","awk_exptransform")
        awk_exptransform()
      } 

$ awk -f main.include.awk
Test.
This is another test: awk_exptransform
test (inside function)

Or calling directly from an inline awk script:

$ awk '
@include "exptransform.awk"
BEGIN { printf ("Test.\n")
        printf ("This is another test: %s\n","awk_exptransform")
        awk_exptransform()
      } 
'

Test.
This is another test: awk_exptransform
test (inside function)

---

Using the above ideas we can come up with a more 'dynamic' solution whereby the OP designates an appropriate function file, insuring all such files define a function of the same name, eg:

$ cat exptransform.1.awk
function awk_exptransform() {                   # same function name
    print "test #1 (inside function)"           # different output
}

$ cat exptransform.2.awk
function awk_exptransform() {                   # same function name
    print "test #2 (inside function)"           # different output
}

Using the -f option to designate the desired function file:

$ awk -f exptransform.1.awk -f main.awk
Test.
This is another test: awk_exptransform
test #1 (inside function)

Using the GNU awk / include file approach:

$ awk -i exptransform.2.awk -f main.awk
Test.
This is another test: awk_exptransform
test #2 (inside function)

---

See GNU awk: Include files for a brief intro to using @include.

See GNU awk: AWKPATH variable for an intro to using AWKPATH to reference a directory(s) of custom include files.

Answer 2

You can always do:

export exptransform='print "test"'
awk '
  BEGIN{
    print "This is another test: " ENVIRON["exptransform"]
    '"$exptransform"'
  }'

That is embed the expansion of the shell variable in the awk code, and also export it as an environment variable so awk can retrieve it and print it.

BTW, you want to avoid -v to pass arbitrary text to awk as it mangles backslashes as the values are subject to \n, \b, \123... expansions.

For an array of such awk code snippets, you may be better of generating the code dynamically:

tests=(
  'print "test"'
  'print 12 * 5'
)

n=0 code=$'BEGIN {
'
for test in "${tests[@]}"; do
  export "V$n=$test"
  code+='printf "Running this test: %s\nResult: ", ENVIRON["V'"$n"$'"]
'"$test
"
  (( n++ ))
done
code+='}'
awk "$code"

Which gives:

Running this test: print "test"
Result: test
Running this test: print 12 * 5
Result: 60

Answer 3

Regarding "How do I get AWK to execute the contents of the variable as a command?" - you cannot. Well, not directly anyway as awk doesn't have an eval command like shell does. You can implement your own version of eval, e.g. see how-to-coerce-awk-to-evaluate-string-as-math-expression, but it sounds like this is what you're really trying to do, using any awk:

$ cat tst.sh
#!/usr/bin/env bash

exptransform='
    print "test"
    print 27 / 9
'

awk '
    BEGIN {
        '"$exptransform"'
    }
'

$ ./tst.sh
test
3

or maybe this:

$ cat ./tst.sh
#!/usr/bin/env bash

exptransform() {
    cat <<-'    !'
        print "test"
        print 27 / 9
    !
}

awk '
    BEGIN {
        '"$(exptransform)"'
    }
'

$ ./tst.sh
test
3

but whether you implement eval or otherwise let shell variables/functions expand to become part of the body of an awk script before awk sees it as above, this is almost certainly a terrible idea. Please post a new question about what you want to do rather than how you're trying to do it so we can help you.

Answer 4

Showing OP how the latest update can be implemented with the answers already provided ...

Modifying OP's current bash / while loop to cut-n-paste bash array entries into the middle of an awk script:

cat test.txt

exptransform=(
    'print $9 - $8'
    'print 10^($9 - $8)'
)

for (( h=0 ; h<${#exptransform[@]} ; ++h )) ; do
    awk '
    BEGIN { FS = "\t" }
          { if ( $4~/1/ && $5~/2/ && $6~/1/ && $7~/1/ && $2!~/20160630_0609/ && $2!~/20170208_2324/ && $2!~/20160804_0635/) 
               '"${exptransform[h]}"'}
    ' test.txt
done

NOTE: no need for a -v variable=value clause

This generates:

subject run day_1_date  include night   tone_delivered  sleep_arousal   tone_1  tone_n
00105   20170413_0406   20170411    1   2   1   1   70  105
00105   20170413_0535   20170411    1   2   1   1   70  70
35
0
99999999999999996863366107917975552
1

Using an awk function to incorporate the desired operations:

$ cat exptransform.awk
function exptransform(pass, a1, a2) {
         if (pass == 1) print a1 - a2
    else if (pass == 2) print 10^(a1 - a2)
}

$ cat main.awk
    { # print                         ## uncomment to have awk print each input line to stdout
      col8[NR] = $8
      col9[NR] = $9
      maxrow++
    }
END { for (i=1; i<=2; i++)
          for (j=2; j<=maxrow; j++)
              exptransform(i, col9[j], col8[j])
    }

$ { cat test.txt; awk -f exptransform.awk -f main.awk test.txt; }
subject run day_1_date  include night   tone_delivered  sleep_arousal   tone_1  tone_n
00105   20170413_0406   20170411    1   2   1   1   70  105
00105   20170413_0535   20170411    1   2   1   1   70  70
35
0
99999999999999996863366107917975552
1

If we uncomment the first entry in main.awk the call becomes:

$ awk -f exptransform.awk -f main.awk test.txt
subject run day_1_date  include night   tone_delivered  sleep_arousal   tone_1  tone_n
00105   20170413_0406   20170411    1   2   1   1   70  105
00105   20170413_0535   20170411    1   2   1   1   70  70
35
0
99999999999999996863366107917975552
1

NOTE: if using GNU awk (compiled with GNU MPFR/MP libraries) we can add the -M/--bignum flag to provide increased arithmetic precision, eg:

$ awk -M -f exptransform.awk -f main.awk test.txt
subject run day_1_date  include night   tone_delivered  sleep_arousal   tone_1  tone_n
00105   20170413_0406   20170411    1   2   1   1   70  105
00105   20170413_0535   20170411    1   2   1   1   70  70
35
0
100000000000000000000000000000000000
1