When dealing with a process that takes too much time, the tool you need from your toolbox is profiling: find out how mustmuch time is spent in each portion of the process.
As we are are talking about long time scales (in the order of hours), I would use a logging system to support me in collecting the profiling information. In concrete steps
- Configure your logging system to include timestamps, if that is not already the case.
- Break down the slow process in concrete steps at a granularity that you think will give you good information without having to wade through tons of logging. Preferably, each step either performs a set of calculations or an interaction over the network or an interaction with the storage device but not multiple of those. If that makes your steps too small for your comfort, you can also do those smaller steps in a second or third iteration.
- Add logging to each step to identify when it starts and when it finishes
- Run the process and collect the logging
- Identify which steps take the most time and check for each of them if the time taken is reasonable and to be expected or if it is too long and needs to be optimized. If a step is repeated multiple times, take all repetitions into account for determining how long that step takes within the overall process.
- If you identified steps that take too long, but they are still too granular to suggest optimization opportunities, then identify sub-steps within that step and repeat the profiling process.
- If the target time is not a hard requirement, but based on a gut feeling of "it should take about that long", then also consider the possibility that the requirement is wrong. In this case, you can use the measurements you collected and your analysis of them as supporting evidence for why the process takes as long as it does.