sed bash variable to delete a line in file

Question

I'm trying to do specific operations in a file, defining the index on 1 and doing operations line-by-line, deleting the contents of line with sed -i -e "{a}d" file as they go the problem is it only seems to be downloading the line 1 by one, for example, on a file with

the result is 2 3 5, so only downloading 1 and 4. What am I doing wrong?

#!/usr/bin/bash
a=0
filename="$1"
while read -r line; do
    echo $a
    name="$line"
    Operation
    sed -i -e "${a}d" $1
    let a+=1
done < "$filename"

From comment: What I intent to do is:

Echo the current line it's on
Performs an operation on the file (in this case wget)
Delete the line it just downloaded
Increase the current line
Until there are no more lines

It's unclear what you mean by "downloading". Note that you are changing the file in each iteration. Therefore, in iteration n, you will not delete line that was n in the original data. — Kusalananda
– Kusalananda ♦, Commented Dec 1, 2018 at 20:21
Please help us understand what you intend to do. Otherwise we can only guess, and not really help you. — sudodus
– sudodus, Commented Dec 1, 2018 at 20:35
This echoes the current line it's on, then perfoms an operation on the file (in this case wget), then deletes the line it just downloaded and then increases the current line until there are no more lines. I know I could use another wget flag, but this is for a specific need. — j.doe
– j.doe, Commented Dec 1, 2018 at 20:38

score 3 · Accepted Answer · 2018-12-01 21:03:57Z

The procedure you selected is incorrect.

Assuming the value of a is started at 1: a=1

When the script reads the filename on the done<"$filename" line the contents of the file (as it is at that point in time) are read in. Then, you modify the same file in-place by removing one line (lets assume the first), then the next read will take the second original line (now the first on the file) but will remove line index 2 (which is the third now) on the present file. So on and so forth.

To see what happens, run this script:

#!/usr/bin/bash
a=1
filename=infile
seq 5 >"$filename"

while read -r line; do
    echo "Read line $line for index $a"
    echo "Will remove line $(sed "${a}!d" "$filename")"
    sed -ie "${a}d" "$filename"
    echo;echo next
    let a+=1
done < "$filename"

That will do this:

$ ./script
Read line 1 for index 1
Will remove line 1

next
Read line 2 for index 2
Will remove line 3

next
Read line 3 for index 3
Will remove line 5

next
Read line 4 for index 4
Will remove line 

next
Read line 5 for index 5
Will remove line 

next

As you can see, lines 1,3,5 got removed.

To do it correctly: either

start at the end
store an index on an additional file
or re-start the process only for the first line of the file until the file is empty.

Implementing third option with tools you already know:

#!/usr/bin/bash

filename=infile
seq 5 >"$filename"

while [[ -s "$filename" ]]; do
    name=$(head -n1 "$filename")
    echo wget "$name"
    sed -i '1d' "$filename"
done

thanks for the help, really insightful and solved my issue. :) — j.doe
– j.doe, Commented Dec 1, 2018 at 21:20

xenoid · Accepted Answer · 2018-12-01 21:31:44Z

0

IMHO editing a file that you are reading is fraught with peril. Let me suggest different solution:

read the input list (urls.lst) line by line
wget the URL
if successful append the URL to a second file (received.lst)

If the process is interrupted and you want to restart with a list of URLs you haven't got yet, you can obtain that list with:

diff --new-line-format="" --unchanged-line-format=""  urls.lst received.lst

answered Dec 1, 2018 at 21:31

xenoid

9,3484 gold badges29 silver badges52 bronze badges

Add a comment |

Stack Exchange Network

sed bash variable to delete a line in file

2 Answers 2

You must log in to answer this question.

Hot Network Questions

sed bash variable to delete a line in file

2 Answers 2

You must log in to answer this question.

Related

Hot Network Questions