3

I'm new to awk programming and I'm learning how arrays work. I found this code:

awk 'BEGIN{OFS=FS=","}NR==FNR{a[$1]=$2;next}$3 in a && $3 = a[$3]' filez filex

They put $1 as index in array and $2 as value then if $3 is equal to an index and for this one $3 = a[$3]. I can't understand what the meaning is, because the first file has just 2 columns -- where did they come up with $3 in comparison to $3 from 2nd file?!?

Input file filez:

111,111
112,114
113,113

Input file filex:

A,bb,111,xxx,nnn
A,cc,112,yyy,nnn
A,dd,113,zzz,ppp
Improve this question
0

2 Answers 2

Reset to default
7

The aim of this script is to replace the values in the third column of the second file (filex) with the corresponding values stored in the second column of the first file (filez).

NR is the number of the current line relative to the first line of the first processed file. It is a "global" line counter. FNR is the number of the current line relative to the beginning of the currently processed file.
NR==FNR is a condition that only evaluates to true for the first file. The corresponding action ({a[$1]=$2;next}) slurps the whole first file, line by line, into the a dictionary, an associative array whose aim is to look up the values of the first file's second column based on the corresponding values in the first column. next makes awk skip the remaining conditions and restart a cycle, reading the next line.

$3 in a && $3 = a[$3] is a condition with a potential side effect (the assignment to $3). It is only evaluated for the second file (when NR==FNR is false; remember that, when NR==FNR was true, $3 in a && $3 = a[$3] was skipped). For each line, if the value of the third field is found (as an index) in the a dictionary, it is replaced with the corresponding value from the dictionary. Then, if $3 = a[$3] evaluates to true, the line is printed (because in AWK programs, which are basically composed of condition (or "pattern")-action pairs, either the condition or the action may be omitted, and an omitted action is equivalent to print).

Assuming this shortened filez:

111,111
112,114

and filex:

A,bb,111,xxx,nnn
A,cc,112,yyy,nnn

what happens, step by step, is:

  1. the first line of filez is read; NR==FNR evaluates to 1==1, true; thus, {a[$1]=$2;next} is executed; a[111] is set to 111; next means that the rest of the script ($3 in a && $3 = a[$3]) is skipped for this line (specifically, $3 is not used for now); nothing is printed, because the performed action doesn't include any printing command;

  2. the second and last line of filez is read; NR==FNR evaluates to 2==2, true; {a[$1]=$2;next} is executed; a[112] is set to 114; $3 in a && $3 = a[$3] is again skipped because of next; and again, nothing is printed;

  3. the first line of filex is read; NR==FNR evaluates to 3==1, false; thus {a[$1]=$2;next} is not executed; the next condition is evaluated: $3 is 111 and is an index value of a, hence $3 in a is true and $3 = a[$3] is evaluated; it causes the assignment of a[111], which is 111, to $3; since 111 is not 0 nor the empty string, the condition-assignment also evaluates to true and the current line is printed;

    A,bb,111,xxx,nnn

  4. the second and last line of filex is read; NR==FNR evaluates to 4==2, false; thus {a[$1]=$2;next} is not executed; the next condition is evaluated: $3 is 112 and is an index value of a, hence $3 in a is true and $3 = a[$3] is evaluated; it causes the assignment of a[112], which is 114, to $3; since 114 is not 0 nor the empty string, the condition-assignment also evaluates to true and the current line is printed.

    A,cc,114,yyy,nnn
Improve this answer
6
  • can you please explain what a[$3] refere to , because if we print it it's the same as column 2 of first file so how could that possible and file 1 has no third column !!!!!! Commented Aug 3, 2020 at 14:37
  • 1
    @DjabriJosef Please, have a look at my edited answer. Commented Aug 3, 2020 at 14:59
  • now , it's clear how $3 = a[$3] works but i thought that this is a condition not an action contrary it replace the value of 3rd field and we are not in in an action environment (i.e., not inside {}) !!! Commented Aug 3, 2020 at 15:08
  • it can be more clear if it was like this $3 in a { $3 = a[$3]} Commented Aug 3, 2020 at 15:11
  • 2
    @DjabriJosef Possibly. But note: 1) $3 in a {$3=a[$3]} and $3 in a && $3=a[$3] are not equivalent: the first would require also an explicit print to print the line. And $3=a[$3] is not used as a condition in the former expression (i.e. it doesn't determine whether the line is printed), while it is in the latter. 2) In AWK, the same set of expressions (including assignments) can be used in both patterns (what I called "conditions") and actions. It is not a requirement for patterns not to alter the environment, by design. Commented Aug 3, 2020 at 15:23
5
  • BEGIN{OFS=FS=","} sets input and output field separators to ,.
  • NR==FNR is only true for the first file. So, for the first file:
    • a[$1]=$2 put $1 as index and $2 as value in array a.
    • next skip the remainder of the script for this line.

Again: next means that the remainder of the script is only executed for the second file (and third, fourth, ... files if they were present).

  • $3 in a && $3 = a[$3] If the third field is a key of the array a, replace the third field with a[$3]. Now, since this is not in an action environment (i.e., not inside {}), it means that if both conditions are true, the current record should be printed. Otherwise, nothing is to be done.

Generally, since the condition on the right is an assignment, that will result in true if the left side is true. However, the assignment is evaluated as false if it assigns an empty string or the number 0. This may be intended, but it is usually an edge-case that the programmer failed to notice. To see it, try the awk code with

filez:

111,111
112,114
113,113
000,000

filex:

A,bb,111,xxx,nnn
A,cc,112,yyy,nnn
A,dd,113,zzz,ppp
A,ee,000,uuu,aaa

Notice that, for the last line, although $3 in a is true, $3 = a[$3] assigns to 0 and thus the line is not printed!

$ awk 'BEGIN{OFS=FS=","}NR==FNR{a[$1]=$2;next}$3 in a && $3 = a[$3]' filez filex
A,bb,111,xxx,nnn
A,cc,114,yyy,nnn
A,dd,113,zzz,ppp

On OP request for clarification:

Remember: The array is populated with a[$1]=$2 in the first file, filez.
$3 = a[$3] is executed for the second file, filex.
For example: In the second line of filez, a[$1]=$2 -> a[112] = 114. In the second line of filex, $3 = a[$3] -> 112=a[112]=114.

Improve this answer
5
  • can you please explain what a[$3] refere to , because if we print it it's the same as column 2 of first file so how could that possible and file 1 has no third column !!!!!! Commented Aug 3, 2020 at 14:33
  • 1
    @DjabriJosef As Quasimodo explained, the entire code section involving $3 is only executed while processing the second file due to the next statement in the first rule block { ... } (which is only executed while processing the first file). Commented Aug 3, 2020 at 14:37
  • @DjabriJosef I have addressed that in detail in the edit. I hope now it will be clear. Commented Aug 3, 2020 at 14:44
  • now , it's clear how $3 = a[$3] works but i thought that this is a condition not an action contrary it replace the value of 3rd field and we r not in in an action environment (i.e., not inside {}) as u said before !!! Commented Aug 3, 2020 at 14:59
  • You can do (almost) anything in the condition part: the only criterion is that it should return a boolean value. For complex conditions, I will often call a user-defined awk function in there, and pass it fields as arguments. Commented Aug 3, 2020 at 15:34

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.