Return to Answer

• <example.txt sed – sed reads example.txt and does the following:

  • s/[$\\`]/\&/g – escapes every $, \ and `, otherwise they would be special in our here document;
  • s/<\([^<>]*\)>/$(\n$$\1\n)/g – converts every string (not containing < or >, so non-greedily) between < and > to

    $(
    $$string
    )
    

  • 1 i cat <<EOF$ – inserts cat <<EOF$ before the first line;
  • $ a EOF$ – appends EOF$ after the last line.

• | sed – the second sed reads from the first and

  • /^\$\$/ – identifies lines starting with $$ (note they must come from the first sed because every $ from the original file is now preceded by a backslash) and there:
    • s/[^$\\`]/\&/g – every character but $, \ or ` gets escaped by a backslash (the excluded characters are already escaped where appropriate)
    • s/^\$\$/cat -- / – and the leading $$ gets replaced by cat -- .

• | sh >example_processed.txt – POSIX shell interprets the resulting script and writes to example_processed.txt

• EOF$ is used instead of traditional EOF, so nothing in your original file can interfere. Even if there was EOF$ in the original file, in the script it would be EOF\$.
• Newlines in pathnames are not supported, < and the corresponding > must be in the same line of input for our code to work.
• Other characters are supported. The pathname (../another.txt in the example) in the script is fully escaped (character by character), so even if you used a pathname with spaces, asterisks or whatever, it would be safe.
• $(…) strips trailing newline characters, this is usually fine.
• -- is explained here: What does -- (double-dash) mean?
• Relative paths in <…> will be resolved with respect to the working directory of sh, not to the directory containing the input file. In our example it's the same directory, but in general the directories may differ. If you want to resolve relative paths with respect to the directory of the input file then you must run sh in this exact directory, just like we did.

• <example.txt sed – sed reads example.txt and does the following:

  • s/[$\\`]/\&/g – escapes every $, \ and `, otherwise they would be special in our here document;
  • s/<\([^<>]*\)>/$(\n$$\1\n)/g – converts every string between and including < and > (where the inside does not contain < or >, so non-greedily) to

    $(
    $$string
    )
    

  • 1 i cat <<EOF$ – inserts cat <<EOF$ before the first line;
  • $ a EOF$ – appends EOF$ after the last line.

• | sed – the second sed reads from the first and

  • /^\$\$/ – identifies lines starting with $$ (note they must come from the first sed because every $ from the original file is now preceded by a backslash) and there:
    • s/[^$\\`]/\&/g – every character but $, \ or ` gets escaped by a backslash (the excluded characters are already escaped where appropriate)
    • s/^\$\$/cat -- / – and the leading $$ gets replaced by cat -- .

• | sh >example_processed.txt – POSIX shell interprets the resulting script and writes to example_processed.txt

• EOF$ is used instead of traditional EOF, so nothing in your original file can interfere. Even if there was EOF$ in the original file, in the script it would be EOF\$.
• Newlines in pathnames are not supported, < and the corresponding > must be in the same line of input for our code to work.
• Other characters are supported. The pathname (../another.txt in the example) in the script is fully escaped (character by character), so even if you used a pathname with spaces, asterisks or whatever, it would be safe.
• $(…) strips trailing newline characters, this is usually fine.
• -- is explained here: What does -- (double-dash) mean?
• Relative paths in <…> will be resolved with respect to the working directory of sh, not to the directory containing the input file. In our example it's the same directory, but in general the directories may differ. If you want to resolve relative paths with respect to the directory of the input file then you must run sh in this exact directory, just like we did.
• The output goes to example_processed.txt which is deliberately a different name than example.txt. Do not redirect output to the file you're reading.

• EOF$ is used instead of traditional EOF, so nothing in your original file can interfere. Even if there was EOF$ in the original file, in the script it would be EOF\$.
• The pathname (../another.txt) in the script is fully escaped (character by character), so even if you used a pathname with spaces, asterisks or whatever, it would be safe.
• $(…) strips trailing newline characters, this is usually fine.
• -- is explained here: What does -- (double-dash) mean?
• Relative paths in <…> will be resolved with respect to the working directory of sh, not to the directory containing the input file. In our example it's the same directory, but in general the directories may differ. If you want to resolve relative paths with respect to the directory of the input file then you must run sh in this exact directory, just like we did.

• EOF$ is used instead of traditional EOF, so nothing in your original file can interfere. Even if there was EOF$ in the original file, in the script it would be EOF\$.
• Newlines in pathnames are not supported, < and the corresponding > must be in the same line of input for our code to work.
• Other characters are supported. The pathname (../another.txt in the example) in the script is fully escaped (character by character), so even if you used a pathname with spaces, asterisks or whatever, it would be safe.
• $(…) strips trailing newline characters, this is usually fine.
• -- is explained here: What does -- (double-dash) mean?
• Relative paths in <…> will be resolved with respect to the working directory of sh, not to the directory containing the input file. In our example it's the same directory, but in general the directories may differ. If you want to resolve relative paths with respect to the directory of the input file then you must run sh in this exact directory, just like we did.

<example.txt sed '
   s/[$\\`]/\\&/g
   s/<\([^<>]*\)>/$(\n$$\1\n)/g
   1 i cat <<EOF$
   $ a EOF$
' | sed '
   /^\$\$/ {
      s/[^$\\`]/\\&/g
      s/^\$\$/cat /
      }
' | sh >example_processed.txt

• <example.txt sed – sed reads example.txt and does the following:

  • s/[$\\`]/\&/g – escapes every $, \ and `, otherwise they would be special in our here document;
  • s/<\([^<>]*\)>/$(\n$$\1\n)/g – converts every string (not containing < or >, so non-greedily) between < and > to

    $(
    $$string
    )
    

  • 1 i cat <<EOF$ – inserts cat <<EOF$ before the first line;
  • $ a EOF$ – appends EOF$ after the last line.

• | sed – the second sed reads from the first and

  • /^\$\$/ – identifies lines starting with $$ (note they must come from the first sed because every $ from the original file is now preceded by a backslash) and there:
    • s/[^$\\`]/\&/g – every character but $, \ or ` gets escaped by a backslash (the excluded characters are already escaped where appropriate)
    • s/^\$\$/cat / – and the leading $$ gets replaced by cat .

• | sh >example_processed.txt – POSIX shell interprets the resulting script and writes to example_processed.txt

cat <<EOF$
Some content containing $(
cat \.\.\/\a\n\o\t\h\e\r\.\t\x\t
) file
EOF$

• EOF$ is used instead of traditional EOF, so nothing in your original file can interfere. Even if there was EOF$ in the original file, in the script it would be EOF\$.
• The pathname (../another.txt) in the script is fully escaped (character by character), so even if you used a pathname with spaces, asterisks or whatever, it would be safe.
• $(…) strips trailing newline characters, this is usually fine.
• Relative paths in <…> will be resolved with respect to the working directory of sh, not to the directory containing the input file. In our example it's the same directory, but in general the directories may differ. If you want to resolve relative paths with respect to the directory of the input file then you must run sh in this exact directory, just like we did.

<example.txt sed '
   s/[$\\`]/\\&/g
   s/<\([^<>]*\)>/$(\n$$\1\n)/g
   1 i cat <<EOF$
   $ a EOF$
' | sed '
   /^\$\$/ {
      s/[^$\\`]/\\&/g
      s/^\$\$/cat -- /
      }
' | sh >example_processed.txt

• <example.txt sed – sed reads example.txt and does the following:

  • s/[$\\`]/\&/g – escapes every $, \ and `, otherwise they would be special in our here document;
  • s/<\([^<>]*\)>/$(\n$$\1\n)/g – converts every string (not containing < or >, so non-greedily) between < and > to

    $(
    $$string
    )
    

  • 1 i cat <<EOF$ – inserts cat <<EOF$ before the first line;
  • $ a EOF$ – appends EOF$ after the last line.

• | sed – the second sed reads from the first and

  • /^\$\$/ – identifies lines starting with $$ (note they must come from the first sed because every $ from the original file is now preceded by a backslash) and there:
    • s/[^$\\`]/\&/g – every character but $, \ or ` gets escaped by a backslash (the excluded characters are already escaped where appropriate)
    • s/^\$\$/cat -- / – and the leading $$ gets replaced by cat -- .

• | sh >example_processed.txt – POSIX shell interprets the resulting script and writes to example_processed.txt

cat <<EOF$
Some content containing $(
cat -- \.\.\/\a\n\o\t\h\e\r\.\t\x\t
) file
EOF$

• EOF$ is used instead of traditional EOF, so nothing in your original file can interfere. Even if there was EOF$ in the original file, in the script it would be EOF\$.
• The pathname (../another.txt) in the script is fully escaped (character by character), so even if you used a pathname with spaces, asterisks or whatever, it would be safe.
• $(…) strips trailing newline characters, this is usually fine.
• -- is explained here: What does -- (double-dash) mean?
• Relative paths in <…> will be resolved with respect to the working directory of sh, not to the directory containing the input file. In our example it's the same directory, but in general the directories may differ. If you want to resolve relative paths with respect to the directory of the input file then you must run sh in this exact directory, just like we did.