How To Read Multiple Files From Multiple Folders In Python

In this tutorial, we will learn how to read multiple files from multiple folders in Python using a few simple steps. Reading multiple files is a common task when working with data that is split across several files or directories, such as log files or datasets.

Python provides several ways to handle this, and we will see how to accomplish it using the os and glob modules, which allow us to navigate the file system and match pathnames.

Step 1: Import Necessary Libraries

First things first, you need to import the necessary libraries. In this case, you will be using the os and glob libraries. The os library provides several methods for interacting with the operating system, whereas the glob library can be used to find all the pathnames matching a specified pattern.

1
2
import os
import glob

Step 2: Identify the Folders

Before reading the files, you need to identify the folders which contain the files you want to read. Suppose you have the following directory structure:

Each file contains a simple text:

Contents of file <file_number>

You can specify and store the folder paths in a list as follows:

1
folder_paths = ['data/folder1', 'data/folder2']

Step 3: Read the Files

Now you can loop through the folder paths, use glob to find all text files in each folder and read the contents of the files. Here’s the code to do this:

1
2
3
4
5
6
7
8
9
10
11
# Loop through the folder_paths list
for folder_path in folder_paths:
    # Use glob to find all text files in the folder
    file_paths = glob.glob(os.path.join(folder_path, '*.txt'))
    # Loop through the file_paths list
    for file_path in file_paths:
        # Read the contents of the file
        with open(file_path, 'r') as file:
            file_content = file.read()
            # Print the file content or do further processing
            print(file_content)

This code snippet uses os.path.join() to create the file path, which ensures that the correct path separator is used across different operating systems. It then reads the contents of each file and prints them to the console. You can modify this code as needed to process the contents according to your requirements.

Step 4: Handling Exceptions

In some cases, you might encounter errors while reading the files. To handle such errors gracefully, you can use Python’s try-except block as follows:

1
2
3
4
5
6
7
8
9
for folder_path in folder_paths:
    file_paths = glob.glob(os.path.join(folder_path, '*.txt'))
    for file_path in file_paths:
        try:
            with open(file_path, 'r') as file:
                file_content = file.read()
                print(file_content)
        except Exception as e:
            print(f"Error reading file {file_path}: {e}")

This code snippet will catch any exceptions that occur while reading the files and print an error message with the file path and exception details.

Full Code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
import os
import glob
 
folder_paths = ['data/folder1', 'data/folder2']
 
for folder_path in folder_paths:
    file_paths = glob.glob(os.path.join(folder_path, '*.txt'))
    for file_path in file_paths:
        try:
            with open(file_path, 'r') as file:
                file_content = file.read()
                print(file_content)
        except Exception as e:
            print(f"Error reading file {file_path}: {e}")

Output

Contents of file 1
Contents of file 2
Contents of file 3
Contents of file 4
Contents of file 5
Contents of file 6

Conclusion

In this tutorial, we have learned how to read multiple files from multiple folders using Python’s os and glob libraries. By following these simple steps, you can easily read and process multiple files in different folders.