2

I have a file which has many links. As an example here is my file:

link1
link2
link3
link4
link5
link6
link7
link8
link9
link10
link11
link12
link13
link14
link15
link16

I want to run a particular command on a single terminal which runs 1st 4 links simultaneously. For example the command I want to run is

wget link1
wget link2

and etc.. but I want to run first 4 links simultaneously. Then if one of the link is finished downloading then the next link(which is link5) should be automatically submitted for download and so on.

I am looking for a way where I am not downloading the links one by one(uses more time), neither I have to open multiple terminals to submit multiple individual links as separate commands. Any help would be really appreciated.

Improve this question
1
  • Your should not have to use separate terminals. Adding a & to the end of a command will give you back the prompt. See shell job control. However using gnu parallel is better (automated). Commented Dec 12, 2019 at 22:18

3 Answers 3

Reset to default
5

You could use GNU parallel:

parallel --retries 5 -j4 -a file.txt wget {}

This will execute 4 jobs at a time and read input from file.txt. If a job fails it will be retried up to 20 times before moving on to the next job.

Improve this answer
6
  • You are missing the part Then if one of the link is finished downloading then the next link(which is link5) should be automatically submitted for download and so on Commented Dec 12, 2019 at 22:07
  • @guillermochamorro: That is what -j4 does, although it doesn't actually check that it finished successfully Commented Dec 12, 2019 at 22:10
  • 1
    Add --retries 20 to parallel to retry if a download fails. Commented Dec 14, 2019 at 10:34
  • Thanks @OleTange Commented Dec 14, 2019 at 13:56
  • wget retries 20 times by default. Commented Dec 14, 2019 at 16:11
2

With an xargs that implements the -P option for keeping multiple jobs running in parallel (most common implementations do):

xargs -I {} -P 4 wget --quiet {} <file.txt

Doing it without xargs or GNU parallel, but using bash:

#!/bin/bash

jobs=0
while read -r url; do
    if [ "$jobs" -ge 4 ]; then
        wait -n
        jobs=$(( jobs - 1 ))
    fi

    wget --quiet "$url" &
    jobs=$(( jobs + 1 ))
done <file.txt

wait

This starts wget background tasks as fast as possible until there has been four such jobs started. Then it waits for any of these to end with wait -n before starting the next one. The jobs variable holds the number of currently running wget jobs.

At the end, the single wait call will block until all jobs have finished.

It's the wait -n that makes this a bash script rather than a plain /bin/sh script.

Improve this answer
0

While all the answers using parallel and xargs work fine, allow me to introduce you to GNU Wget2. It's the next version of Wget, and while it is still in alpha-mode, it is a drop-in replacement for most usages of Wget.

Wget2 supports multithreaded downloads, so you can simply give it the file and how many downloads you want in parallel, let Wget2 handle the rest for you.

Source: Am maintainer for both GNU Wget and GNU Wget2

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.