4
$\begingroup$

I have a few thousand image urls that I want to download asynchronously, how can I do that while monitoring progress? I'm really asking for 5 things:

  • Show dynamic progress in bar
  • Downloads must be asynchronous
  • Avoid any filename collisions
  • Save with correct file extensions (even if not present in url)
  • Show list of failed download tasks and why

Here's an example to get started:

Monitor[
    URLDownload[
         WebImageSearch["dog", "ImageHyperlinks", MaxItems -> 10], 
         "~/Downloads/"
    ]
]

Updated response to comment

I don't believe URLDownloadSubmit takes a directory, this is the behavior I see (no progress indication):

dogs = WebImageSearch["dog", "ImageHyperlinks", MaxItems -> 10]
URLDownloadSubmit[dogs, "~/Downloads", 
 HandlerFunctions -> <|"TaskProgress" -> Print, 
   "TaskComplete" -> Print|>, 
 HandlerFunctionsKeys -> {"FractionComplete", "ByteCountDownloaded"}]

enter image description here

And the filenames are wrong:

enter image description here

Related but not duplicate:

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ You can use HandlerFunctions like so: URLDownloadSubmit[dogs, "~/Desktop/test", HandlerFunctions -> <|"TaskProgress" -> Print, "TaskComplete" -> Print|>, HandlerFunctionsKeys -> {"FractionComplete", "ByteCountDownloaded"}] to print the FractionComplete and ByteCountDownloaded. It's probably a documentation bug that there's no good example of how to do it, but it is mentioned in the usage of URLDownload and URLDownloadSubmit. You can use ProgressIndicator[Dynamic[frac]] and frac = #FractionDownloaded instead of Print. $\endgroup$ Commented Aug 12, 2020 at 21:26
  • $\begingroup$ @CarlLange thanks, see my update $\endgroup$ Commented Aug 12, 2020 at 21:38
  • 1
    $\begingroup$ I think the message is because "~/Desktop/test" is a file and you are trying to download multiple files to it. Try just "~/Desktop". $\endgroup$ Commented Aug 12, 2020 at 22:11
  • $\begingroup$ @CarlLange Sorry for the typo, I fixed that but it doesn't look like it's doing anything still. If you could your snippet with a progress bar I'll accept! $\endgroup$ Commented Aug 12, 2020 at 23:06

2 Answers 2

Reset to default
4
+75
$\begingroup$

What I would do is run URLDownloadSubmit once on each URL that you want to download. That will make all those files start to download asynchronously in the background as a result.

When downloading that many images, it's not worth it to keep track of how many bytes have been downloaded of each image. It's better to just count the number of files that have been downloaded so far. That should be granular enough.

Start by getting the URLs:

urls = WebImageSearch["dog", "ImageHyperlinks", MaxItems -> 10];

Create a progress indicator:

i = 0;
ProgressIndicator[Dynamic[i], {0, Length[urls]}]

Let outputDir be your output directory. This will start downloading the files:

submitURL[url_] := URLDownloadSubmit[
  url,
  FileNameJoin[{outputDir, Last@Information[url, "Path"]}],
  HandlerFunctions -> <|
    "TaskFinished" -> (i++ &)
    |>]

submitURL /@ urls;

I used FileNameJoin[{outputDir, Last@Information[url, "Path"]}] to keep the original names as requested. However, note that the names may collide with each other. The advantage with UUIDs, as in your screenshot, is that they are always unique.

Improve this answer
$\endgroup$
13
  • $\begingroup$ When there are missing urls, the progress indicator will never be 100%. How can you add an error handler to count and show the few bad ones? $\endgroup$ Commented Aug 24, 2020 at 22:25
  • $\begingroup$ If you can add error handler I will accept $\endgroup$ Commented Aug 25, 2020 at 1:29
  • $\begingroup$ Actually, that collision fix isn't perfect as FileExtension returns "" on some urls, is there a better way to inject the uuid into the filename? $\endgroup$ Commented Aug 25, 2020 at 5:55
  • $\begingroup$ It’s nice that this is abortable but how to handle failures in a nice way? $\endgroup$ Commented Aug 25, 2020 at 7:23
  • $\begingroup$ @M.R. It depends. Perhaps it would work to just make a list of all the files in that have been downloaded and compare it with the URLs that you tried to download. That way you can make a list of all the files that you failed to download. It would be nice to have some examples of filenames where FileExtension fails, perhaps these are especially tricky cases that need special treatmeant not matter how it is done. $\endgroup$ Commented Aug 25, 2020 at 8:03
3
$\begingroup$

Does something like this work for you?

With[
    {
    links = WebImageSearch["dog", "ImageHyperlinks", MaxItems->10],
    dir = CreateDirectory[]
    },

    Monitor[
        Table[URLDownload[links[[i]], dir], {i, Length[links]}],
        ProgressIndicator[i / Length[links]]
    ]
]

Your original question wasn't clear to me that the downloads were to be asynchronous. Perhaps you can use LocalSubmit for this purpose, e.g.:

With[
    {
    links=WebImageSearch["dog","ImageHyperlinks",MaxItems->10],
    dir=CreateDirectory[]
    },

    p=0;
    LocalSubmit[
        Table[
            URLDownload[links[[i]],dir];
            Print[i],
            {i, Length[links]}
        ],
        HandlerFunctions-><|"PrintOutputGenerated" -> Function[p = First @ #PrintOutput]|>,
        HandlerFunctionsKeys->{"PrintOutput"}
    ];
    Dynamic @ ProgressIndicator[p/Length[links]]
]

Response:

Thanks for updating, here are 100 image urls to test on:

urls = CloudGet["https://www.wolframcloud.com/obj/f691b40d-b12a-41b5-9017-9130bc797fd0"];

It seems like it is still downloading synchronously because it takes ~40 seconds, but @C.E.'s only takes ~10 seconds (and speed is the most important factor to me). On the plus side, it seems all 100 images download correctly and the filenames avoid collisions, but the With block won't Abort.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Well not really, because it's not async, and so will be waaay to slow for 10k images. $\endgroup$ Commented Aug 17, 2020 at 22:06
  • $\begingroup$ Do you see what I mean? $\endgroup$ Commented Aug 19, 2020 at 3:36
  • $\begingroup$ You solution is too slow because it is sychronous, so do see a way around this? $\endgroup$ Commented Aug 22, 2020 at 2:49
  • $\begingroup$ @C.E. Your comment doesn't make sense. LocalSubmit uses one kernel for its code, not one kernel per file. $\endgroup$ Commented Aug 22, 2020 at 15:43
  • 1
    $\begingroup$ @CarlWoll No, that was what I first thought that LocalSubmit did. That would have given terrible performance. I assure you that URLDownloadSubmit does not use multiple kernels. $\endgroup$ Commented Aug 25, 2020 at 19:05

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.