10
$\begingroup$

I have a list of download jobs (e.g. {{url, filename} ...}) to do:

jobs = Table[{"https://picsum.photos/200/300/?random", 
              "~/Downloads/" <> ToString[i] <> ".jpg"}, {i, 5}]

Clearly the ProgressIndicator here does not show the percent completion of the asynchronous tasks:

Monitor[Table[URLDownloadSubmit @@ jobs[[i]], {i, 5}], 
        ProgressIndicator[Dynamic[i], {0, 5}]]

So I'm looking for a nice way to monitor the total progress of all tasks created in a call like this:

Scan[URLDownloadSubmit@@#&, jobs]

without resorting to the obsolete symbols URLFetchAsynchronous and URLSaveAsynchronous.

Improve this question
$\endgroup$

4 Answers 4

Reset to default
10
$\begingroup$

Here is an example of how to build a download manager.

We start by defining a function that uses URLDownloadSubmit to initiate a download:

manifest = <||>;

SetAttributes[taskProgress, HoldFirst]
taskProgress[manifest_][event_] := 
 manifest = <|manifest, event["Task"] -> event|>

SetAttributes[taskFinished, HoldFirst]
taskFinished[manifest_][event_] := 
 manifest = <|manifest, event["Task"] -> event|>

SetAttributes[startJob, HoldFirst]
startJob[manifest_][src_, dest_] := URLDownloadSubmit[
   src, dest,
   HandlerFunctions -> <|
     "ConnectionFailed" -> connectionFailed[manifest],
     "CookiesReceived" -> cookiesReceived[manifest],
     "HeadersReceived" -> headersReceived[manifest],
     "TaskFinished" -> taskFinished[manifest],
     "TaskProgress" -> taskProgress[manifest],
     "TaskStatusChanged" -> taskStatusChanged[manifest]
     |>,
   HandlerFunctionsKeys -> {
     "Task", "TaskStatus", "File",
     "ByteCountTotal", "ByteCountDownloaded", "FractionComplete"
     }
   ];

We have defined a variable, manifest, that will hold information about the files being downloaded. It is up to the user to define the event handler functions that they want to use; in my download manager, I will only use TaskProgress and TaskFinished. Whenever any of those events are called, I update manifest with the latest information. The latest information includes the variables specified under HandleFunctionsKeys.

This is all we need, really. Now we can build an interface to visualize manifest.

SetAttributes[abortDownload, HoldFirst]
abortDownload[manifest_, task_] := (
  TaskRemove /@ Select[Tasks[], #["TaskUUID"] === task["TaskUUID"] &];
  manifest = <|
    manifest,
    task -> <|manifest[task], "TaskStatus" -> "Aborted"|>
    |>)

SetAttributes[visualizeManifest, HoldFirst]
visualizeManifest[manifest_] := TableForm[Join[
   {{"File", "Size (MB)", "Downloaded (MB)", "Fraction complete", 
     "Status", ""}}, {
      #File
      , Floor[#ByteCountTotal/10^6]
      , Floor[#ByteCountDownloaded/10^6]
      , ProgressIndicator[#FractionComplete]
      , #TaskStatus
      , Button["Abort", abortDownload[manifest, #Task], 
       Enabled -> (#TaskStatus =!= "Aborted")]
      } & /@ Values[manifest]
   ]]

I will also add a button to begin downloading an Anaconda installer. Anaconda is a software for Python programmers that I picked because the installer is large enough in size that the download won't finish in a blip.

i = 0;
Button["Download", startJob[manifest][
  "https://repo.anaconda.com/archive/Anaconda3-5.2.0-MacOSX-x86_64.pkg",
  "~/Downloads/anaconda" <> ToString[i++] <> ".pkg"
  ]]

Dynamic@visualizeManifest[manifest]

The final result looks like this:

enter image description here

You can easily compute other statistics such as how many of the files have finished downloading by going through the values in the manifest association.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I would like to use this for monitoring multiple thousand image downloads without having to hard code the ToString[i++] <> ".ext"... is there some way of doing this? $\endgroup$ Commented Aug 18, 2020 at 0:05
  • $\begingroup$ @M.R. I'm afraid I don't understand what you mean. There is nothing in the code that I can see that requires hard-coding... $\endgroup$ Commented Aug 18, 2020 at 9:23
6
$\begingroup$

Here's another way to do this, based on extracting the "File" handler key

parallelDownload[things_] :=
 DynamicModule[
  {jobs, results},
  Dynamic[
   Internal`LoadingPanel@
    Grid@
     If[AllTrue[Values@results, # =!= None &],
       Append[
        {
         Button[
          "Get Result",
          NotebookWrite[
           Nest[ParentBox, EvaluationBox[], 5],
           ToBoxes@Values@results
           ]
          ],
         SpanFromLeft
         }
        ],
       Identity
       ]@
      KeyValueMap[
       {
         Row@{#[[1]], ":"},
          If[#2 =!= None, "Complete", "Waiting..."]
         } &,
       results
       ]
   ],
  Initialization :>
   {
    results = <||>,
    jobs = <||>,
    Map[
     With[{job = Flatten[{#}]},
       results[job] = None;
       jobs[job] =
        URLDownloadSubmit[
         Sequence @@ job, 
         HandlerFunctions ->
          <|"TaskFinished" :> Function[results[job] = #File]|>,
         HandlerFunctionsKeys -> {"File"}
         ]
       ] &,
     things
     ]
    }
  ]

If we run that on the jobs it pops a little watching interface:

enter image description here

After that's done it shows a button to replace the box with the result:

enter image description here

And pressing the button gives us the desired result:

{
 "~/Downloads/1.jpg", "~/Downloads/2.jpg", 
 "~/Downloads/3.jpg", "~/Downloads/4.jpg", 
 "~/Downloads/5.jpg"
 }

With ProgressIndicator

Here's an alternate interface for many files

parallelDownloadDynamic[
  things_, 
  var:Dynamic[_]|None:None,
  e:OptionsPattern[Dynamic]
  ] :=
 DynamicModule[
  {jobs, results},
  Dynamic[
   Replace[var,
     Verbatim[Dynamic][s_]:>(Set[s, results])
     ];
   If[AllTrue[Values@results, # =!= None &],
     Button[
        "Get Result",
        NotebookWrite[
           Nest[ParentBox, EvaluationBox[], 1],
           ToBoxes@Values@results
           ]
        ],
     Internal`LoadingPanel@ProgressIndicator[
       Count[Values[results], Except[None]],
       {0, Length[results]}
       ]
     ],
   e
   ],
  Initialization :>
   {
    results = <||>,
    jobs = <||>,
    Map[
     With[{job = Flatten[{#}]},
       results[job] = None;
       jobs[job] =
        URLDownloadSubmit[
         Sequence @@ job, 
         HandlerFunctions ->
          <|"TaskFinished" :> Function[results[job] = Replace[#["File"], Except[_String]->$Failed]]|>,
         HandlerFunctionsKeys -> {"File"}
         ]
       ] &,
     things
     ]
    }
  ]
Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ I always wonder why functions like Internal`LoadingPanel are not in the supported language. (+1) $\endgroup$ Commented Jul 14, 2018 at 2:58
  • 1
    $\begingroup$ @Edmund I'd almost rather they not be... the "System`" context is bloated enough as is. Why they have to be in "Internal`" though is beyond me. Some type of supported "Formatting`" context would be perfect for it. Or even better, some context that makes it clear that there's a front-end kernel separation. Unfortunately we're too far past the point of no-return for that. $\endgroup$ Commented Jul 16, 2018 at 17:50
  • $\begingroup$ @b3m2a1 THis looks like it could work, but not thousands of URLs... Can you try changing into a progress bar and letting the File name come from the url/header? $\endgroup$ Commented Aug 18, 2020 at 0:04
  • $\begingroup$ @M.R. try the edit $\endgroup$ Commented Aug 18, 2020 at 0:23
5
$\begingroup$

You may use the HandlerFunctions option of URLDownloadSubmit to track a counter updated in "TaskFinished" event.

With jobs as defined in OP.

completedCount = 0;
tasks =
  URLDownloadSubmit[#1, #2,
     HandlerFunctions -> <|"TaskFinished" -> (completedCount++ &)|>
     ] & @@@ jobs;
ProgressIndicator[Dynamic[completedCount], {0, Length@tasks}]

This outputs a ProgressIndicator that tracks the completion of the tasks.

Hope this helps.

Improve this answer
$\endgroup$
2
$\begingroup$

Here I create a different list of asynchronous tasks , but should be the same with your downloads.

tasklist = 
  Table[SessionSubmit[ScheduledTask[counter += 1, {m}]], {m, 1, 20}];

Dynamic[
 ProgressIndicator[
  Count[Through@tasklist["TaskStatus"], "Removed"]
  , {0, Length[tasklist]}
  ], UpdateInterval -> 1]

enter image description here

Improve this answer
$\endgroup$
2
  • $\begingroup$ this has nothing to do with parallel downloading $\endgroup$ Commented Jul 12, 2018 at 23:43
  • $\begingroup$ @user5601 Easy with the down votes, the request was for a "way to monitor the total progress of all tasks created", except for CE's answer that went further measuring file sizes, a general solutions for arbitrary asynchronous tasks applies in particular for asynchronous download tasks. $\endgroup$ Commented Jul 15, 2018 at 8:40

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.