Parsing multiple arguments into a bash script

Question

I've tried a few solutions posted here, but to no avail. I'm hoping it's something silly and obvious that I've missed, that you can spot easily, and I can facepalm myself.

I have a bash script containing multiple commands for data analysis. If I were to run this script on the command line, it would look something like this (simplified):

./script.sh /path/to/file1.gz /path/to/file2.gz output_name

My first two arguments are paths to files needed in the commands in the script.sh for data analysis. The output_name, is, as you'd expect, the name of the file that's output at the end.

This works fine, but I want to instead have a file containing the paths to multiple files, each on a new line, and loop over each line in the file and run the script sequentially. So the arguments file would look like:

/path/to/file1.gz /path/to/file2.gz output_name1

On each line.

And then feed each line into the script in a while loop (I figure this is best anyway).

I tried:

while read -r line; do; ./script.sh; done < arg_list.txt

Where arg_list.txt is the file containing my list of input arguments, I got "failed to open file `'."

Feel free to tell me I'm stupid and missing something!

Answer 1

Using your own structure, read in three whitespace-separated values per line and use them as the arguments to the script

while read -r in1 in2 out
do
    ./script.sh "$in1" "$in2" "$out"
done < arg_list.txt

Answer 2

Assuming the contents of your input file is properly quoted (in the case where any of the arguments contain spaces, tabs, or newlines), then you may use xargs like so:

xargs -n 3 ./script.sh <arg_list.txt

This reads three arguments from the input at a time and calls your script with them.

A group of three arguments don't need to be listed on the same line in the input file, xargs will read three arguments and call your script with them, regardless of whether they are on the same line or spread across two or three lines.

Example (using a printf command in place of a script):

$ cat file
/path/to/file1.gz /path/to/file2.gz output_name1
/path/to/file1.gz /path/to/file2.gz output_name1
'/path/to/file 1.gz' /path/to/file2.gz output_name1
/path/to/file1.gz /path/to/file2.gz output_name1
/path/to/file1.gz
/path/to/file2.gz output_name1
/path/to/file1.gz /path/to/file2.gz output_name1

$ xargs -n 3 printf 'Arg1: "%s", Arg2: "%s", Arg3: "%s"\n' <file
Arg1: "/path/to/file1.gz", Arg2: "/path/to/file2.gz", Arg3: "output_name1"
Arg1: "/path/to/file1.gz", Arg2: "/path/to/file2.gz", Arg3: "output_name1"
Arg1: "/path/to/file 1.gz", Arg2: "/path/to/file2.gz", Arg3: "output_name1"
Arg1: "/path/to/file1.gz", Arg2: "/path/to/file2.gz", Arg3: "output_name1"
Arg1: "/path/to/file1.gz", Arg2: "/path/to/file2.gz", Arg3: "output_name1"
Arg1: "/path/to/file1.gz", Arg2: "/path/to/file2.gz", Arg3: "output_name1"

Answer 3

Actually, if your file contains the exact list of parameters, then the loop become extremely simple:

while read
do
   ./script.sh $REPLY
done

The REPLY variable is an automatic variable used if you do not supply a new name to the read. The REPLY also does not split the line into fields and therefore can be used as is. Please note absence of quotes in this case. If the line read from the file contains spaces and quotes - they would transfer to the command exactly as they are in the file. So if you have a line in the file a b "c d" you would get a call to the script as script.sh a b "c d" - three parameters with a space in the last one. So you have to ensure proper quoting inside the file.

If the parameters you read from file need to be updated somehow, then you can do a more elaborate read:

while read f1 f2 f3
do
   ./script.sh "$f2" "$f1" "$f3.txt"
done

Please note quoting in the file in this case too. For the same line a b "c d", this code will produce script.sh b a "c d".txt. It could be not exactly the desired outcome.

If your file with parameters contain fields separated by something unusual, then you can modify a special variable IFS before read - then the read will split the line by the value of IFS.

If you need something even more elaborate, you can use xargs, awk, or even go to a full scale languages like perl or python. There are tons of solutions.