Multiprocessing In Python: How To Implement A Loop Over "apply_async" As "map_async" Using A Callback Function

I would like to integrate a system of differential equations for several parameter combinations using Python's multiprocessing module. So, the system should get integrated and the

Solution 1:

If I understood you correctly, it stems from something that confuses people quite often. apply_async's callback is called after the single op, but so does map's - it does not call the callback on each element, but rather once on the entire result.

You are correct in noting that map is faster than apply_asyncs. If you want something to happen after each result, there are a few ways to go:

1. You can effectively add the callback to the operation you want to be performed on each element, and map using that.

2. You could use imap (or imap_unordered) in a loop, and do the callback within the loop body. Of course, this means that all will be performed in the parent process, but the nature of stuff written as callbacks means that's usually not a problem (it tends to be cheap functions). YMMV.

---

For example, suppose you have the functions f and cb, and you'd like to mapf on es with cb for each op. Then you could either do:

def look_ma_no_cb(e):
    r = f(e)
    cb(r)
    return r

p = multiprocessing.Pool()
p.map(look_ma_no_cb, es)

or

for r in p.imap(f, es):
    cb(r)