Threadline

multiprocessing.Pool: real parallelism with processes

Read the explanation
What if

Two worker processes crunch at the same time, each with its own interpreter and its own GIL.

012345678Pool(2)pool.map()main: print resultsprintworker 1: start: import this filestartcrunch("a")worker 2: start: import this filestartcrunch("b")

Tap any block to see what that thread is doing.

  • Running
  • Waiting / parked
Step 1/4

main creates a pool of two worker processes.

1def crunch(name):2    total = 03    for i in range(20_000_000):  # pure Python CPU work4        total += i5    return name, os.getpid()6 7def main():8    print(f"main process {os.getpid()}")9    with multiprocessing.Pool(2) as pool:  # two worker processes, each with its own GIL10        for name, pid in pool.map(crunch, ["a", "b"]):11            print(f"{name} ran in process {pid}")12 13if __name__ == "__main__":  # needed: each worker process imports this file14    main()

Explanation

What just happened

multiprocessing.Pool(2) started two worker processes. Each is a separate Python interpreter with its own GIL. pool.map pickled one name to each worker, both crunched at the same time on two cores, and the results came back pickled. main printed two different process IDs, neither of them its own.

Our tests measure it: the two workers use two CPUs’ worth of time at once, which threads with the GIL never do.

Try the what-if switch

Why the __main__ guard

With the default start methods (spawn on macOS, forkserver on Linux since Python 3.14), a new worker process starts by importing your main script. If creating the pool isn’t behind if __name__ == "__main__":, the import tries to start a pool inside the worker. Python refuses, every worker crashes, and map never returns.

Watch out

Every argument and result is pickled and copied between processes. For many small tasks that copying can cost more than it saves.

Common questions

Why do I need if __name__ == '__main__' with multiprocessing?

With the default start methods (spawn on macOS, forkserver on Linux), each worker process imports your main script. Without the guard, every worker tries to start a pool too and crashes with a RuntimeError about the bootstrapping phase, and map never returns.

Why doesn't my global variable change with multiprocessing?

Each worker is a separate process with its own copy of every global. A worker changes its copy; the main process's copy stays the same. Send data back as return values.

Why can't I use a lambda with Pool.map?

Everything sent to a worker is pickled, and a lambda can't be pickled: pool.map(lambda x: x, ...) raises PicklingError. Use a function defined at the top level of a module.

Verified: this timeline matches a real program, checked in CI on Python 3.14, 3.15 and free-threaded 3.15t. How we verify. Unclear or wrong? Tell us