How To Monitor Python Script

In this tutorial, we will learn how to monitor a Python script for understanding its performance and finding out potential bottlenecks. Monitoring can be useful in analyzing code execution, detecting memory leaks and CPU usage, or even tracking errors. We’ll be presenting different techniques for monitoring a Python script, which can help you identify issues and optimize its efficiency.

Before diving into the monitoring process, it is important to have a clear understanding of your Python script. Make sure your code is well-written, structured, and easy to follow. By using comments and proper indentation, it becomes easier to monitor and understand the various sections of your script.

Step 1: Use the timeit module

Python provides a built-in module called timeit that allows you to measure the execution time of your code. You can use this module to find the time taken by various functions, loops, or any block of code.

The simplest way to use the timeit module is to import it and call its default_timer() function before and after the code block you want to measure. The difference between the starting and ending times will give you the execution time of the code block.

Here’s an example:

1
2
3
4
5
6
7
8
import timeit
 
start_time = timeit.default_timer()
# Your code here
end_time = timeit.default_timer()
 
execution_time = end_time - start_time
print("Execution time:", execution_time)

Output:

Execution time: 1.2999807950109243e-06

Step 2: Use the cProfile module

Python provides another built-in module called cProfile that helps you analyze the performance of your script in terms of function calls. This module creates a report that shows the number of calls, the time spent in each function, and other related information.

To use the cProfile module, simply import it and run your script with the cProfile.run() function:

1
2
3
4
5
6
7
import cProfile
 
def my_function():
    # Your code here
    pass
 
cProfile.run('my_function()')

The output will show you the performance details of your script, helping you identify any bottlenecks.

Output:

         4 function calls in 0.000 seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.000    0.000 <string>:1(<module>)
        1    0.000    0.000    0.000    0.000 temp.py:3(my_function)
        1    0.000    0.000    0.000    0.000 {built-in method builtins.exec}
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

Step 3: Use the psutil library

Another useful tool for monitoring Python scripts is the psutil library. It allows you to monitor system-related information such as CPU usage, memory usage, and disk usage during the execution of your script. You can use the psutil library to better understand the resource usage of your script.

First, install the psutil library using the following command:

1
pip install psutil

Next, import the psutil library in your script and monitor the CPU and memory usage as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
import psutil
import os
import time
 
pid = os.getpid()
py_process = psutil.Process(pid)
 
while True:
    cpu_usage = py_process.cpu_percent()
    memory_usage = py_process.memory_info().rss
    print("CPU %:", cpu_usage)
    print("Memory (bytes):", memory_usage)
    time.sleep(1)

This script will display the CPU and memory usage of your Python script every second.

Output:

CPU %: 0.0
Memory (bytes): 15736832
CPU %: 0.0
Memory (bytes): 15749120
CPU %: 0.0
Memory (bytes): 15749120

Step 4: Monitor logs and exceptions

A crucial aspect of monitoring Python scripts is to keep track of logs and exception handling. You can use the built-in logging module to log errors, warnings, or informational messages.

Here’s an example of using the logging module to capture exceptions and log them:

1
2
3
4
5
6
7
8
9
import logging
 
logging.basicConfig(filename='app.log', level=logging.INFO, format='%(asctime)s - %(message)s')
 
try:
    # Your code here
    pass
except Exception as e:
    logging.exception("An error occurred:")

Now, any exceptions will be logged in a file named app.log, making it easier for you to monitor them.

Conclusion

In summary, monitoring a Python script requires a combination of various techniques and tools. By using built-in modules such as timeit, cProfile, and logging, as well as external libraries like psutil, you can effectively monitor the performance and efficiency of your Python scripts. This knowledge will help you identify potential issues, optimize your code, and improve overall code quality.