How to append items to a list in a parallel process (python)?

Question

this is a really simplistic version of a more complicated code I am working with. The problem is that I would like to append items calculated in the method() function to some list that I can display later. However, when this code is run, the list object is empty, while the results array is full.

    import multiprocessing as mp
    global list
    list = []
    def add(thing):
        list.append(thing)
    def method():
        global list
        add(8) #doesn't work as wanted
        return 7
    def logResult(result):
        results.append(result)

    if (__name__ == '__main__'):
        results = []
        cpu = mp.cpu_count()
        pool = mp.Pool(processes=cpu)
        for x in range(0, 2000):
            pool.apply_async(method,callback=logResult)
        pool.close()
        pool.join()
        print list
        print results

Output:

    []
    [7,7,7,7,7,7,7,7,7....] and so on.

I know that the add method seems redundant, but a simple list.append() inside the method() function doesn't work either. The add method is meant to mirror the logResult method(). I can see why it doesn't work, but I don't know how to fix this. Without the parallelization, the program works as desired, but parallelization is needed for my project as the calculation done is much more tedious than in the method() function. The desired output would be

    [8,8,8,8,8,8,8,8,8,8,8,8,...]
    [7,7,7,7,7,7,7,7,7,7,7,7,...] and so on.

Thanks in advance.

Answer 1

The list should reside in the shared memory in order to be accessible from the worker subprocesses. Consider multiprocessing.Manager().list().

Answer 2

It seems you have set global list in method() which is empty. it does not require in method() if you have set before.

Answer 3

pool.apply_async(method,callback=logResult)

1. Your method functions are missing ().
2. logResult should have something to pass on (according to your code)

simply changing to this:

pool.apply_async(method(),callback=logResult(7))

get result [8,8,8,8,8...], [7,7,7,7,7...]