How to Run Multiple Processes in Python

In this tutorial, we will learn how to run multiple processes in Python using the multiprocessing module.

Python’s multiprocessing module allows us to create multiple processes, each running independently of the others. This can be useful in a variety of scenarios, particularly when dealing with tasks that are time-consuming or can be broken down into smaller, parallel tasks.

Step 1: Import the multiprocessing module

Before you can begin using multiple processes, you will first need to import the multiprocessing module into your Python script. This can be done with the following line of code:

1
import multiprocessing

Step 2: Define the function to be run in parallel

Next, you need to define the function that you want to run in parallel. This function should include the work that each process will perform. Here’s an example:

1
2
def worker(number):
    print(f'Worker: {number}')

Step 3: Create and start multiprocessing.Process objects

Once the function is defined, we can then create and start multiple instances of the function running in parallel. We do this by creating multiprocessing.Process objects and calling their start() methods:

1
2
3
4
if __name__ == "__main__":
    for i in range(5):
        process = multiprocessing.Process(target=worker, args=(i,))
        process.start()

Running the script

After following the steps above, you should now have a Python script that runs the specified function in parallel across multiple processes. This script creates 5 worker processes, each of which prints out its worker number when it starts.

Full code:

1
2
3
4
5
6
7
8
9
import multiprocessing
 
def worker(number):
    print(f'Worker: {number}')
 
if __name__ == "__main__":
    for i in range(5):
        process = multiprocessing.Process(target=worker, args=(i,))
        process.start()

Sample Output:

Worker: 0
Worker: 1
Worker: 2
Worker: 3
Worker: 4

This output might vary since the multiprocessing module runs processes at the same time and the output sequence can depend on when each process gets scheduled to run. Refer to Python’s official multiprocessing documentation to learn more about how multiprocessing works and other options available.

Conclusion

By using the multiprocessing module in Python, you can effectively manage and run multiple processes simultaneously, allowing you to improve the efficiency and performance of your applications.

However, it’s important to remember that multithreading and multiprocessing come with their own set of challenges, such as synchronization and data safety issues. Therefore, they should be used judiciously and in appropriate scenarios.