Concatenating a list of files using For loop

Question

I'm new to unix/bash/code in general, and I'm trying to merge multiple sample files with the same name in different directories into a new file in its own directory (the outputs of multiple data collections).

To do this, I'm trying to create a script which uses two arrays - listA, the name of each sample file, and list B, the names for the newly merged sample files.

My code looks a little like this:

#!/bin/sh

listA=( old1 old2 old3 etc.)
listB=( new1 new2 new3 etc.)

i=0

for i in $listA $listB

do

cp ./folder1/$listA ./merged/$listB
cat ./folder2/$listA >> ./merged/$listB
cat ./folder3/$listA >> ./merged/$listB
cat ./folder4/$listA >> ./merged/$listB

((i=+1))

done
echo "Done stitching"

As is, it seems to merge the files for the first entry in listA into the first file in listB perfectly, but it won't repeat the process for the subsequent entries on the list.

Any advice to make this work as intended? Apologies for my ignorance, I'm very new to all of this and enjoying the learning process immensely - just a bit stumped.

Answer 1

Short answer

while IFS=$'\t' read -r old new; do cat folder*/"$old" > merged/"$new"; done <list.tsv

This assumes that list.tsv contains a tab-delimited list of source (old) and destination (new) files.

Explanation

• while IFS=$'\t' read -r old new; FOO; done <list.tsv: this is a common shell contruct to read each line of list.tsv, then do FOO on each line. In this case, the input file separator IFS is a tab $'\t', i.e. the first field of the line is saved to $old, and the second field (after the tab) is saved to $new.
• cat folder*/"$old" > merged/"$new": concatenate the files in folder*/"$old", and write this to merged/"$new". This assumes that the source files are all in folder*, e.g. named folder1, folder2, etc.
• As with any script, I would certainly test this before running it. You can do that by copying a subset of your files to another directory, then using a short version of your input list list.tsv. Check the results, and if okay, then run it for real (but keep a backup!).

Notes on your suggested solution

I just thought I'd write a few comments on your proposed solution, in order to clarify how the various commands work. Hopefully this is helpful to your understanding!

• listA=( old1 old2 old3 etc.). This creates an array. The way to access parts of it is shown here. Using for i in $listA… will just access the first entry, as you have (kinda) observed.
• for i in a b c. This iterates over a, b, and c, putting each into $i. For example, try running for i in a b c; do echo $i; done. N.B. it makes no sense to refer to $listA within the do…done block. Similarly, i=0 and ((i=+1)) are inconsistent with for usage.
• ./folder1/$listA. ./ is unnecessary. . is the current folder, so it doesn't do anything here.
• cat FOO >>. cat will concatenate by itself (hence the name). There is no need to write separate cat statements that append >> to a file. You can just list multiple files after cat, e.g. cat file1 file2 file3 > output_file.