standard deviation using awk [closed]

Question

I am using below command to get standard deviation of file A names

   1         2         3        avg 
23.3107  20.0372    21.7236   21.6905

awk '{x[NR]=$0 ;} END{a=$4; for (i in x){ss += (x[i]-a)^2} sd = sqrt(ss/n); print $5 = sd}'

getting fatal: division by zero attempted

modified above command as

awk '{x[NR]=$0 ;} END{a=$4; for (i in x){if (a == 0) $6 ="N/A"; else ss += (x[i]-a)^2} sd = sqrt(ss/n); print $5 = sd}'

but still error persisted? thank you cas for understanding my quest?

Answer 1

Where is "n"?

You write:

sd = sqrt(ss/n)

but where in your code did you assign the variable "n"? The way awk sees it, "n" is zero.

Also, where is column 5 in a=$5 (and, third issue, why is this assignment in the END section)? Your example contains only 4 columns.

Answer 2

Did you mean to do something like this? It's the only way i can think of to make sense of your script.

awk -v OFS=$'\t' '
FNR == 1 { $5 = "sdev" ; print }

FNR > 1  { a = $4    # field 4 is 'avg'
           n = NF-1  # exclude the 'avg' field from the ss calculations.

           for (i=1; i <= n; i++) { ss += ($i - a)^2 } 

           $5 = sqrt(ss/n)
           print
         }' inputfile

Note: $i on the for line refers NOT to the value of i, but to the input field numbered i - i.e. it loops through $1, $2, and $3. This may not be obvious to shell or perl users where (scalar) variables are normally prefixed by $.

NF is the number of fields on a line, and FNR is the record (line) number of the current input file (so this awk script supports multiple input files, each with their own header line. If there's only ever going to be one input file at a time, you could use NR instead of FNR).

Sample output:

1       2       3       avg     sdev
23.3107 20.0372 21.7236 21.6905 1.33661

Here's another version which works with any number of fields per line. It assumes that the last field of a line contains the average of all the previous fields on that line.

$NF refers to the value of the last field (i.e. the 'avg') and $new refers to the (last field + 1), i.e. assigning a value to it adds a new field to the end of the line.

awk -v OFS=$'\t' '
FNR == 1 { new = NF+1   # number of new field to add
           $new = "sdev"
           print 
         }

FNR > 1  { a = $NF   # last field is 'avg'
           n = NF-1  # exclude the 'avg' field from the ss calculations.

           for (i=1; i <= n; i++) { ss += ($i - a)^2 } 

           $new = sqrt(ss/n)
           print
         }' inputfile

Sample ouput with 5 values plus an average on each input line:

1       2       3       4       5       avg     sdev
23.3107 20.0372 21.7236 20.5328 21.2016 21.3611 1.13107