6

I'm trying to set several environment variables with the results from command substitution. I want to run the commands in parallel with & and wait. What I've got currently looks something like

export foo=`somecommand bar` &
export fizz=`somecommand baz` &
export rick=`somecommand morty` &
wait

But apparently when using & variable assignments don't stick. So after the wait, all those variables are unassigned.

How can I assign these variables in parallel?

UPDATE: Here's what I ended up using based off the accepted answer

declare -a data
declare -a output
declare -a processes

var_names=(
    foo
    fizz
    rick
)

for name in "${var_names[@]}"
do
    processes+=("./get_me_a_value_for $name")
done

index=0
for process in "${processes[@]}"; do
    output+=("$(mktemp)")
    ${process} > ${output[$index]} &
    index=$((index+1))
done
wait

index=0
for out in "${output[@]}"; do
    val="$(<"${out}")"
    rm -f "${out}"

    export ${var_names[index]}="$val"

    index=$((index+1))
done

unset data
unset output
unset processes
Improve this question
3
  • 2
    What exactly would the benefit of this complication be, assuming it somehow actually worked? Commented Apr 5, 2017 at 22:11
  • 2
    If bar, baz, and morty take (say) ten seconds each to run, assigning the variables in parallel would take ten seconds, rather than thirty if run in series. Commented Apr 5, 2017 at 22:12
  • I've even tried such constructs as foo=$( ( sleep 3, echo bar ) & ) to work around the problem, but that just makes the assignment not return until the subshell has completed (which in retrospect makes sense). Commented Apr 5, 2017 at 22:15

2 Answers 2

Reset to default
6

After some ruminations, I came up with an ugly workaround:

#!/bin/bash
proc1=$(mktemp)
proc2=$(mktemp)
proc3=$(mktemp)

/path/to/longprocess1 > "$proc1" &
pid1=$!
/path/to/longprocess2 > "$proc2" &
pid2=$!
/path/to/longprocess3 > "$proc3" &
pid3=$!

wait "$pid1" "$pid2" "$pid3"
export var1="<("$proc1")"
export var2="<("$proc2")"
export var3="<("$proc3")"
rm -f "$proc1" "$proc2" "$proc3"

As requested in a comment, here is how to make this more extensible for an arbitrarily large list:

#!/bin/bash
declare -a pids
declare -a data
declare -a output
declare -a processes

# Generate the list of processes for demonstrative purposes
processes+=("/path/to/longprocess1")
processes+=("/path/to/longprocess2")
processes+=("/path/to/longprocess3")

index=0
for process in "${processes[@]}"; do
    output+=("$(mktemp")
    $process > ${output[$index]} &
    pids+=("$!")
    index=$((index+1))
done
wait ${pids[@]}
index=0
for process in "${processes[@]}"; do
    data+="$(<"${output[index]}")"
    rm -f "${output[index]}"
    index=$((index+1))
done
export data

The resultant output will be in the data array.

Improve this answer
6
  • I think you mean export var1=$(<"$proc1") etc? I don't think you need to save the PID's of the children, doesn't wait wait for all of them if none are listed Commented Apr 5, 2017 at 22:47
  • I tracked the PIDs and wait on them explicitly because it's possible there are other backgrounded tasks we don't need to wait for the completion of. Commented Apr 6, 2017 at 0:11
  • That worked with the suggestion from @ilkkachu. To add another constraint, is it possible to make this work for an arbitrary number of variables (such as a list being passed in and iterated over by a for loop)? Commented Apr 6, 2017 at 18:52
  • Certainly, I will modify the answer to show that alternative. Commented Apr 6, 2017 at 19:28
  • @DopeGhoti awesome. This should be thread safe right? Also, I just had to make one change, ("<${output[index]}") had to be $(<"${output[index]}"). Commented Apr 7, 2017 at 22:47
3

If you have more jobs than can safely be run in parallel at the same time, you can use parset from GNU Parallel:

parset foo,fizz,rick somecommand ::: bar baz morty
export foo
export fizz
export rick

See details: https://www.gnu.org/software/parallel/parset.html

Improve this answer
0

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.