(Shell Script) Variable not pass correctly when fetching value from file?

Question

I have this section of curl command inside my shell script which caused a error 500 repsonse from the server. However, this code works when I manually run the command.

curl -qgsSkH "Content-Type: multipart/form-data" --no-progress-bar --header "Token: $authKey" -F "filename=@$compressedFileName" -F "options={\"application\":\"2\",\"timeout\":\"5000\"}" https://www.domain.com > results.txt

Anyone have any idea why?

Edit: I did some manual debugging and found out that the issue is caused by the variable $authKey. This variable is fetched by the following code in my script.

# Get authentication key from FireEye AX response
while read line
do
    if [[ $line =~ ^SOME-PATTERN:[[:space:]](.*) ]]
    then
        authKey="${BASH_REMATCH[1]}"
    fi
done <auth.txt

If I redefine this variable by using something like the following then the script will work.

authKey="TheAuthenticationKey"

But why is this happening? The earlier part where I pass the string from the file when I read it line by line is already passing the correct value (I double checked by echoing the output to a txt file).

Edit I managed to drill down somemore. When I try to echo the value of the token to a file and add " to the start and the end of the string I got this result.

"X-FeApi-Token: IDR+jxU1kB5ZpXsRNAgqfjTxkjZ0L2atH+l0H/NeatWDAJs=
"

I am not sure how to remove that new line thingy. Anyone can advise?

Answer 1

The problem seems to be caused by a newline or carriage-return and newline in $authKey, as supplied by "FireEye AX"

You can strip carriage returns and line feeds by piping it through tr -d '\r\n'

e.g.

# Get authentication key from FireEye AX response
while read line
do
    if [[ $line =~ ^SOME-PATTERN:[[:space:]](.*) ]]
    then
        authKey="${BASH_REMATCH[1]}"
        authKey=$(printf "%s" "$authKey" | tr -d '\r\n')
    fi
done <auth.txt

I can't remember if $BASH_REMATCH will be passed to a subshell...you could try it in one line instead, maybe it will work:

authKey=$(printf "%s" "${BASH_REMATCH[1]}" | tr -d '\r\n')

Answer 2

I'm guessing that the data file you have been provided has Windows-style line endings (CR/LF) rather than Unix/Linux line endings (LF).

This alternative pattern match will exclude any possible trailing CR:

[[ $line =~ ^SOME-PATTERN:[[:space:]](([^\015]*) ]]
authKey="${BASH_REMATCH[1]}"