Processing array values in bash

Question

I am trying to create an array based upon filenames, and get in trouble with whitespaces. This seems common. But - as far as I can see - the quotes are set correctly, I guess it must be the way the array is built.

to_dump="$(find . -maxdepth 1 -print0 )"
to_dump_array=($to_dump)

read -p " ->  " final
case "$final" in
   a) for drop in "${to_dump_array[@]}" ;
      do cp "$drop" --recursive --force Destination_Folder && \
      echo "dropped \"$drop\" ;
      done ;;
   b) echo "Won't drop anything" ;;
esac

I guess there should be a nicer way to build an array from a find query. Also, where else am I wrong?

Answer 1

-print0 should not be used in a $(...) substitution, because strings in bash variables are null-terminated.

I asked a question whose answer was similar to what this question requires: https://stackoverflow.com/a/30469553/1091693

Adapting that answer to your question:

to_dump=()
while IFS= read -r -d ''; do
  to_dump+=( "$REPLY" )
done < <(find . -maxdepth 1 -print0)

This creates an array called to_dump and uses the read command to read NULL-delimited elements from find. The reason < <(...) is being used here rather than a pipe is to avoid an implicit subshell which would prevent the array from being modified.

It's worth noting that your original find command probably wants a -mindepth 1, or it will pick . (the current directory) and you'll end up doing a recursive copy on that.

---

I've noticed you use -maxdepth 1 as an argument to find, so perhaps this will be more useful:

shopt -s nullglob
to_dump=( * .[!.]* ..?* )

Avoiding find, this uses bash builtins only, doesn't fork, and is for the most part quite clean.

The first line, shopt -s nullglob, is a bash(-only) command which turns on the nullglob option. This option is described in man 1 bash:

If set, bash allows patterns which match no files (see Pathname Expansion above) to expand to a null string, rather than themselves.

In simpler terms, if you type * but it doesn't match files, it will remove the *. The default behaviour is to put the * in there anyway.

The second line adds 3 globs to the array:

• *: All files not beginning with .
• .[!.]*: All files beginning with one . and one non-. character. This is to avoid matching the . and .. directories.
• ..?*: All files beginning with .. and at least one more character. Added for the same reason as the previous glob, covering the cases it missed.

Bash expands the globs into the definition of the array, and it expands them correctly -- no splitting on whitespace or anything like that.

A caveat on the usage of nullglob: If you have nullglob turned on, curl google.com/search?q=test will result in curl complaining at you for not passing it arguments, and ls /var/fasdfasafs* will give you a listing of the current directory. This is one of the reasons it's not turned on by default.

Answer 2

Try building the array like this:

read -d $'\0' -r -a to_dump <<< $(find . -maxdepth 1 -print0)

Answer 3

 find . -maxdepth 1

...would appear to me to indicate you want:

a=()
for f in ./..?* ./.[!.]* ./*
do  [ -e "$f" ] && a+=$f
done

Answer 4

I'm not sure the script you have is doing what you think it's doing. I don't think that this converts the null terminated output of the find into an bash array of file names:

to_dump_array=($to_dump)

Have you checked the output of the for loop to see what you're getting?

for drop in "${to_dump_array[@]}"
do
    echo -e "$drop\n"
done

There's a stack overflow with some suggestions on filling an array from find -print0.

https://stackoverflow.com/questions/1116992/capturing-output-of-find-print0-into-a-bash-array

You also may be better using the array index to process the array items rather than trying to assign to the variable directly within the for loop, it avoids relying on the shell splitting correctly:

for drop in $(seq 0 $((${#to_dump_array[@]} - 1)))
do
    cp "${to_dump_array[$drop]}"

This may not be very popular, but if you need to use bash arrays and multiple escapes to manage white space (among other even more unusual characters) in file names then you might be going beyond what the shell was designed for.

You may find something like python, perl, ruby to be faster, more reliable and easier to manage and debug the code base.

I have a number of "red flags" which I use to tell me when I might be going beyond what is wise when shell scripting.

• Multiple levels of escaping or null termination to handle all file name cases.
• Several levels of shell functions calling one another.
• Complex data structures. I.e. going beyond strings, numbers.
• I find myself doing "performance optimisation".
• "Line noise" in expressions.

Yes, you certainly can do all of the above, but should you? Seriously... Python, perl, ruby etc. In python it's this simple, and you don't need to worry about escaping or file names with white space or binary characters.

import os

dirListing = os.listdir("somedirectory")

for eachEntry in dirlisting:
    doSomething