Parallel computing in joblib

The goal of the project is to improve the Python multiprocessing backend of joblib, used extensively by scikit- learn. The technical challenge is that, to avoid locks, the parallel-computing strategy of the multiprocessing module is to spawn multiple processes. Error management and nested parallelism are difficult in such setting. The project is still ramping up, but we could already identify and fix many failure modes of the Python multiprocessing module when computation crashed in workers. Fixes will be first integrated in joblib, and later contributed upstream in the Python standard library.

Comments are closed.