I have a problem, the solution to which is not on the Internet. After painstakingly recalculating all combinations at a certain point this error pops up regardless of n _jobs. I am attaching the code below. Please advise, experts. This code is in a for loop.
print(f"\nTrying combinations with {group_size} features:")
feature_groups = combinations(all_feature_names, group_size)
gc.collect()
results.extend(
Parallel(n_jobs=-1)(delayed(process_feature_group)(fg) for fg in feature_groups)
The error contains the following text. "A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault calling the function or bu an excessive memory usage causing the Operating System to kill the worker." I tried to change the number of jobs in the range : 3;-1.
If there is not enough input data, then write - I will attach. I will be grateful for answers.
itertool
? Why did you preferjoblib
when there ismultiprocessing
which is more efficient and a part of standard python library?