Why do I get different values for $x from the snippets below?
#!/bin/bash
x=1
echo fred > junk ; while read var ; do x=55 ; done < junk
echo x=$x
# x=55 .. I'd expect this result
x=1
cat junk | while read var ; do x=55 ; done
echo x=$x
# x=1 .. but why?
x=1
echo fred | while read var ; do x=55 ; done
echo x=$x
# x=1 .. but why?
The right explanation has already been given by jsbillings and geekosaur, but let me expand on that a bit.
In most shells, including bash, each side of a pipeline runs in a subshell, so any change in the shell's internal state (such as setting variables) remains confined to that segment of a pipeline. The only information you can get from a subshell is what it outputs (to standard output and other file descriptors) and its exit code (which is a number between 0 and 255). For example, the following snippet prints 0:
a=0; a=1 | a=2; echo $a
In ksh (the variants derived from the AT&T code, not pdksh/mksh variants) and zsh, the last item in a pipeline is executed in the parent shell. (POSIX allows both behaviors.) So the snippet above prints 2. You can get this behavior in modern bash by setting the lastpipe option with shopt -s lastpipe (in an interactive shell, you also need to disable job control with set +m).
A useful idiom is to include the continuation of the while loop (or whatever you have on the right-hand side of the pipeline, but a while loop is actually common here) in the pipeline:
cat junk | {
while read var ; do x=55 ; done
echo x=$x
}
< <(locate -ber ^\.tag$), thanks to the original slightly unclear answer and geekosaur and glenn jackman's comemnts.. I was initially in a dilemma about accepting the answer, but the nett result was pretty clear, especially with jsbillings follow-up comment :)
... | while read item pipes the stdout of whatever is run before while but with this syntax the body of while is always run in a subshell. I prefer to use syntax for item in $(...) instead, because that keeps the body of the loop in current shell.
Commented
Apr 26, 2020 at 11:43
for item in $(...) syntax does not nicely support all features. For example ... | while read col1 col2 col3 rest cannot be nicely implemented with for .. in syntax.
Commented
Apr 26, 2020 at 11:44
for item in $(…) doesn't do the same thing. It expands wildcards unless you disable that with set -f. It splits at all whitespace, not just newlines, unless you change IFS. It strips trailing blank lines. Don't use for item in $(…) unless you've thought of all of these things.
Commented
Apr 26, 2020 at 11:46
You're running into a variable scope issue. The variables defined in the while loop that is on the right side of the pipe have their own local scope context, and changes to the variable will not be seen outside of the loop. The while loop is essentially a subshell which gets a COPY of the shell environment, and any changes to the environment are lost at the end of the shell. See this StackOverflow question.
UPDATED: I neglected to point out the important fact that the while loop with it's own subshell was due to it being the endpoint of a pipe, I've updated that in the answer.
while loop as the tail end of a pipeline that throws it into a subshell.
Commented
Mar 23, 2011 at 14:44
blah|blah|while read ..., you can have while read ...; done < <(blah|blah)
Commented
Mar 23, 2011 at 15:01
As mentioned in other answers, the parts of a pipeline run in subshells, so modifications made there aren't visible to the main shell.
If we consider just Bash, there are two other workarounds in addition to the cmd | { stuff; more stuff; } structure:
Redirect the input from process substitution:
while read var ; do x=55 ; done < <(echo fred)
echo "$x"
The output from the command in <(...) is made to appear as if it were a named pipe.
The lastpipe option, which makes Bash work like ksh, and runs the last part of the pipeline in the main shell process. Though it only works if job control is disabled, i.e. not in an interactive shell:
bash -c '
shopt -s lastpipe
echo fred | while read var ; do x=55 ; done;
echo "$x"
'
or
bash -O lastpipe -c '
echo fred | while read var ; do x=55 ; done;
echo "$x"
'
Process substitution is of course supported in ksh and zsh too. But since they run the last part of the pipeline in the main shell anyway, using it as a workaround isn't really necessary.
