Calling multiple instances of python scripts in matlab using java.lang.Runtime.getRuntime not working

Question

I am running Matlab2017 on windows 10. I call a python script that runs some Speech Recognition task on cloud with something like this:

 userAuthCode=1;% authentication code for user account to be run on cloud
 cmd = ['C:\Python27\python.exe runASR.py userAuthCode];  
 system(cmd);

When the above command is called, the python script runs the input audio file on the ASR cloud engine, and as it runs, I can see Speech Recognition scores for the audio file from Python in the Matlab console. I want to do the following:

(1) Execute multiple such commands in parallel. Lets say, I have 2 input audio files (each having different audio segments), and I would like to run the above command 2 times, but in parallel, using separate processes. I was able to create a code snippet that should be able to do this:

 for i=1: 2
     userAuthCode=i;
     cmd = ['C:\Python27\python.exe runASR.py userAuthCode];  
     runtime = java.lang.Runtime.getRuntime();        
     pid(i) = runtime.exec(cmd);
 end

 for i=1:2
    pid(i).waitFor();
    % get exit status
    rc(i) = pid(i).exitValue();       
 end

Now, when the above code is executed, I can see ASRE scores for data1 ,but not for data 2.
The exit status in variable rc, is 0,1, which confirms this. The problem is I do not know the cause of the error, as nothing is printed in Matlab. How can I get error message from Python captured in a java/Matlab variable so that i could take a look?

The issue could be that multiple Calls to ASRE in parallel (with different user accounts of course) may not be supported but I won't know unless I can see the error.

(2) When I run a single command standalone, as mentioned at the start of the post, I am able to see Score messages for each audio segment being printed in the Matlab console, as they are obtained from Python. However, with multi-processing using java.lang.Runtime.getRuntime() and the associated code, no messages appears in the Matlab console. Is there a way to display those messages (I am assuming display might be asynchronous?)

Thanks
sedy

Answer 1

One approach is to use multiprocessing in Python. The results and any error messages can be returned in a list.

Example:

Assuming you have three audio files, your_function will run 3 times in parallel with error messages returned.

import subprocess
from multiprocessing import Pool, cpu_count

def multi_processor(function_name):

    # Use a regex to make a list of full paths for audio files in /some/directory
    # You could also just pass in a list of audio files as a parameter to this function
    file_list = []
    file_list = str(subprocess.check_output("find ./some/directory -type f -iname \"*a_string_in_your_aud_file_name*\" ",shell=True)).split('\\n')
    file_list = sorted(file_list)

    # Test, comment out two lines above and put 3 strings in the list so your_function should run three times with 3 processors in parallel
    file_list.append("test1")
    file_list.append("test2")
    file_list.append("test3")

    # Use max number of system processors - 1
    pool = Pool(processes=cpu_count()-1)
    pool.daemon = True

    results = {}
    # for every audio file in the file list, start a new process
    for aud_file in file_list:
        results[aud_file] = pool.apply_async(your_function, args=("arg1", "arg2"))

    # Wait for all processes to finish before proceeding
    pool.close()
    pool.join()

    # Results and any errors are returned
    return {your_function: result.get() for your_function, result in results.items()}


def your_function(arg1, arg2):
    try:
        print("put your stuff in this function")
        your_results = ""
        return your_results
    except Exception as e:
        return str(e)


if __name__ == "__main__":
    multi_processor("your_function")