4 
#!/bin/awk
BEGIN {
        while(getline var < compareTo > 0)
        {
                orderIds[var]=var;
        }
}
{
        if(orderIds[$0] == "")
        {
                print $0;
        }
}

Running as 

awk -v compareTo="ids.log.remote" -f sample.awk ids.log.local

This is working, but instead of using associative arrays ( like HashMap ), is there anything like a HashSet in awk?

I got the timings

bash-3.2$ time grep -xFvf ids.log.local ids.log.remote > /dev/null

real    0m0.130s
user    0m0.127s
sys     0m0.002s
bash-3.2$ time grep -xFvf ids.log.local ids.log.remote > /dev/null

real    0m0.126s
user    0m0.125s
sys     0m0.000s
bash-3.2$ time grep -xFvf ids.log.local ids.log.remote > /dev/null

real    0m0.131s
user    0m0.128s
sys     0m0.002s
bash-3.2$ time awk 'NR == FNR {
  orderIds[$0]; next
  }
!($0 in orderIds)
  ' ids.log.local ids.log.remote > /dev/null

real    0m0.053s
user    0m0.051s
sys     0m0.003s
bash-3.2$ time awk 'NR == FNR {
  orderIds[$0]; next
  }
!($0 in orderIds)
  ' ids.log.local ids.log.remote > /dev/null

real    0m0.052s
user    0m0.051s
sys     0m0.001s
bash-3.2$ time awk 'NR == FNR {
  orderIds[$0]; next
  }
!($0 in orderIds)
  ' ids.log.local ids.log.remote > /dev/null

real    0m0.053s
user    0m0.051s
sys     0m0.002s
bash-3.2$ time awk -v compareTo="ids.log.local" -f checkids.awk ids.log.remote > /dev/null

real    0m0.066s
user    0m0.060s
sys     0m0.006s
bash-3.2$ time awk -v compareTo="ids.log.local" -f checkids.awk ids.log.remote > /dev/null

real    0m0.065s
user    0m0.058s
sys     0m0.008s
bash-3.2$ time awk -v compareTo="ids.log.local" -f checkids.awk ids.log.remote > /dev/null

real    0m0.061s
user    0m0.053s
sys     0m0.007s

@Dimitre Radoulov Looks like your awk is faster. Thanks.

Improve this question
4
  • What exactly are you trying to do? Commented Feb 2, 2012 at 20:50
  • In your example, orderIds[var]=var is storing var twice; once as the the index, and then again as the value held by that array element. This is not necessary and inhibits the detection of blank lines in the main section, ie.when var == "". In the BEGIN section, you can set orderIds[var]=1.. The 1 is just a flag to indicate that this particular index (var) has been encountered in "ids.log.remote" Commented Feb 2, 2012 at 21:09
  • @Peter.O yes, I thought about storing 1 and I thought both are integer values and it requires same amount of storage to store ints. So, I left it as var, instead of a flag like 1. Commented Feb 2, 2012 at 21:12
  • @tech_learner, it's not true that both are integer values, rather var here has a string value. But that string isn't going to be freed anyway, since you're using it as the array index, so you're only using up the space of an active pointer. Probably the same length as an integer value, but it's better hygiene not to think of storing a pointer as "storing an int". I also find it more natural to use orderIds[var]=1 here; that's a familiar pattern for implementing sets in terms of assoc arrays that others will more immediately recognize when reading your code. Commented Apr 19, 2012 at 19:34

1 Answer 1

Reset to default
2

I believe this is the most efficient way to do this in awk:

awk 'NR == FNR {
  orderIds[$0]; next
  }
!($0 in orderIds)
  ' ids.log.remote ids.log.local

You may try with grep too:

grep -xFVf ids.log.remote ids.log.local
Improve this answer
3
  • I changed the script similar to yours. Now it looks like modified script is faster. No idea why #!/bin/awk BEGIN { while(getline var < compareTo > 0) { orderIds[var]; } } { if(!($0 in orderIds)) { print $0; } } bash-3.2$ time awk 'NR == FNR { orderIds[$0]; next } !($0 in orderIds) ' ids.log.local ids.log > /dev/null real 0m0.052s user 0m0.050s sys 0m0.001s bash-3.2$ time awk -v compareTo="ids.log.local" -f checkids.awk ids.log > /dev/null real 0m0.036s user 0m0.035s sys 0m0.001s Commented Feb 2, 2012 at 21:35
  • 1
    I suppose it's mostly because every time you're referencing a non-existing key (i.e. orderIds[$0] when $0 is not currently in orderIds) you're actually creating a new entry in the array (with a null string value). Try to test the number of elements in the orderIds array after all input has been read (in an END block) with your code and compare it with my script - with your script you'll end up with more entries (one for every printed line). Out of curiosity, you may try the script with mawk (if you have it), it could be even faster. Commented Feb 2, 2012 at 21:37
  • 1
    Another contributing factor to the time difference between the two awk versions could be that accessing the data component of the array element simply takes longer than just checking for the existence of the key.. Commented Feb 2, 2012 at 23:05

You must log in to answer this question.