Two ssh output as awk input

Question

I have two remote servers that I am trying to ssh and cat some files.. I want to input the output of the ssh to a awk command.

This is what I have got

ssh username@host1 “cat /tmp/test/*” ssh username@host2 “cat /tmp/test/*” | awk ‘ command ‘

But this is not working, if the host2 directory is empty then the output of the host1 is repeated twice.

What am I doing wrong here?

Answer 1

Beware of those curly quotes (“ and ”) some Windows text editors will use, use straight quotes instead (' or "). You should also be using 's, unless you have some reason to use "s, e.g. to let a variable expand, see https://mywiki.wooledge.org/Quotes.

You should be doing this if you want a single pipe to awk:

{ ssh username@host1 'cat /tmp/test/*'; ssh username@host2 'cat /tmp/test/*'; } | awk ' command '

but consider doing this instead so awk can distinguish which command the input came from in case you need that distinction in future:

awk ' command ' <(ssh username@host1 'cat /tmp/test/*') <(ssh username@host2 'cat /tmp/test/*')

Here's the difference:

$ seq 3 > file1
$ seq 2 > file2

Wrong output (because FILENAME always contains - and ARGV[1..2] are empty):

$ { cat file1; cat file2; } | awk '
    FILENAME == ARGV[1] { host="host1" }
    FILENAME == ARGV[2] { host="host2" }
    { print host, $0 }
'
 1
 2
 3
 1
 2

Wrong output (because FILENAME and ARGV[1] contain the same temp file descriptor but ARGV[2] is empty):

$ awk '
    FILENAME == ARGV[1] { host="host1" }
    FILENAME == ARGV[2] { host="host2" }
    { print host, $0 }
' <(cat file1; cat file2)
host1 1
host1 2
host1 3
host1 1
host1 2

Correct output:

$ awk '
    FILENAME == ARGV[1] { host="host1" }
    FILENAME == ARGV[2] { host="host2" }
    { print host, $0 }
' <(cat file1) <(cat file2)
host1 1
host1 2
host1 3
host2 1
host2 2

Other possible way to get the correct output with this approach of providing the input:

$ awk '{print host, $0}' host='host1' <(cat file1) host='host2' <(cat file2)
host1 1
host1 2
host1 3
host2 1
host2 2

Answer 2

You need to group the commands. The simplest way is to use curly braces which group them without creating a subshell:

{ ssh username@host1 'cat /tmp/test/*'; ssh username@host2 'cat /tmp/test/*'; } | 
   awk ' command '

You can also use parentheses, but this is slightly inelegant since it is creating a subshell where you don't need one. However, the end result is the same, and it really shouldn't make a difference in this context. Plus, you don't need to add the final ; as you do when using { }:

( ssh username@host1 'cat /tmp/test/*'; ssh username@host2 'cat /tmp/test/*' ) | 
   awk ' command '

---

The reason you are seeing duplicated output is because both cat commands are being run on the same host. Check the output when using localhost as the host for both commands, and after running set -x:

$ ls /tmp/test
file1

$ ssh localhost "cat /tmp/test/*" ssh localhost "cat /tmp/test/*"
+ ssh localhost 'cat /tmp/test/*' ssh localhost 'cat /tmp/test/*'
/tmp/test/file1
/tmp/test/file1
cat: ssh: Is a directory
cat: localhost: No such file or directory
cat: cat: No such file or directory

As you can see above, this is interpreted as "ssh localhost cat arg1 arg2 arg3 arg4", so the cat is run on localhost ( host1 in your case), and given as arguments /tmp/test/*, ssh, localhost (host2 in your case), cat and then /tmp/test/* again. So the contents of host1's /tmp/test are catted twice.

Answer 3

One possible way is to exec ssh command in subshell and pipe the result on awk:

(ssh username@host1 "cat /tmp/test/*"; ssh username@host2 "cat /tmp/test/*" ) | awk ‘ command ‘