Python script executed in bash script. Errors not outputting to log file

Question

Basically I have a python script that i run against a web page and it outputs a phone number contact if it finds one.

What i'm trying to code is a bash script which can automate its use on a list of urls. Which saves the results to a file so i can see which urls produced a phone number contact and which urls didn't. I want this in the same txt file with each urls result on a different line so that i can deduce which urls produced which results. EG

 $cat log.txt
 HTTPError: 404
 224 265 899
 HTTPError: 404
 847 718 9300, + 1, 662 538 6500

The problem is that when i run my bash script, the http errors print to screen and the phone numbers print to the log file. Meaning i cant deduce which urls produced which results.

So at the moment my bash script reads a URLS.txt file into an array. Then adds the script command to the beginning of each url in the array. Then using a for loop, executes each command..

I've tried to search for a way to add a string to each line of the log.txt everytime the python command is executed but have had no luck. If i could do this I wouldnt need the errors as i could deduce which urls didnt work because there would just be the string on each line that didnt produce a result.

So if i set the string to "none?" the log.txt would look like this

 none?
 none? 224 265 899
 none?
 none? 847 718 9300, + 1, 662 538 6500

This is the code i have after i have put the URLs into an array.

 scr1="python3 extract.py " 
 scr1arr1=( "${array1[@]/#/$scr1}" )
 for each1 in "${scr1arr1[@]}"
 do
 $each1 >> log.txt
 done

So this is what happens on screen when i execute the script.

 HTTPError: 404
 HTTPError: 404

and the log.txt looks like this

 224 265 8990, 
 847 718 9300, + 1, 662 538 6500

Am i doing this all wrong? / Approaching the problem the wrong way?? I've literally spent days trying to work this out, I'm loosing sleep over it! I am very new to coding, so sorry if this has been covered already. I've searched many forums far and wide and not been able to find a solution.

Would python be better for this? I don't know python as well as bash, is it definitely not possible in bash? — Joshua Court
– Joshua Court, Commented Jun 2, 2019 at 12:07
Python would be better for this because inside Python, you can easily check whether there's an error or not, and write to the file only when there is no error. By wrapping your Python script into a bash loop, you have to parse the output of the Python script for every call in bash. That setup is much harder to maintain when changes are needed. — Roland Weber
– Roland Weber, Commented Jun 2, 2019 at 20:08
@DennisWilliamson After reading BashFAQ/050 I'm now aware i have to use a function to save a command. However i'm confused about whether i need to include "eval" in this case. I'm also confused with regards to why the below code isn't working. Is the way i am adding the function to the begining of each array element not correct? — Joshua Court
– Joshua Court, Commented Jun 3, 2019 at 10:25

Ed Morton · Accepted Answer · 2019-06-02 12:28:55Z

2

Your script is apparently printing errors to stderr, not stdout, so you need to add 2>&1 to redirect stderr to the same file as stdout. Try this:

 for each1 in "${scr1arr1[@]}"
 do
    "$each1"
 done > log.txt 2>&1

answered Jun 2, 2019 at 12:28

Ed Morton

209k18 gold badges90 silver badges212 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python script executed in bash script. Errors not outputting to log file

1 Answer 1

Comments

Hot Network Questions