How To Handle More Than 1,048,576 Rows In Excel Python

Handling a large volume of data in Excel can be challenging, especially when you exceed the row limit of 1,048,576. Fortunately, Python’s data manipulation libraries, specifically pandas, can help us handle large datasets effectively.

In this tutorial, you will learn how to manage more than 1,048,576 rows in an Excel sheet with Python’s pandas library.

Step 1: Install Pandas Library

First, make sure you have Python 3 installed on your system. You can check this by running the following command in your terminal or command prompt:

Next, you need to install the pandas library. Run the following command:

Step 2: Read the Large Dataset

In this step, we will read the large dataset in chunks, which helps manage memory usage. For this tutorial, let’s assume you have a CSV file named “large_dataset.csv” with more than 1,048,576 rows.

Step 3: Process the Data in Chunks

Once you have the dataset in chunks, you can process each chunk separately. For example, suppose you want to filter only the rows that contain a specific value. You could do this in the following way:

Step 4: Write the Result to a New Excel File

Finally, write the filtered data to a new Excel file. First, we need to install the openpyxl library, which allows writing data to Excel files:

Now, write the filtered data to a new Excel file:

Full Code:


In this tutorial, you learned how to handle more than 1,048,576 rows in Excel using Python’s pandas library. By reading the dataset in chunks and processing it chunk by chunk, you can effectively manage large-scale datasets and perform complex data manipulations.