How to Remove Stop Words from a Text File in Python

In this tutorial, you are going to learn how to remove stop words from a text file in Python. Stop words are the most common words in a language that do not usually contain essential meaning, and are typically removed during preprocessing in natural language processing (NLP) tasks.

Words such as “the”, “is”, “in”, “for”, etc., are considered stop words. Removing these words can greatly reduce the dimensionality of your dataset and improve the performance of your machine-learning models.

Step 1: Import Necessary Libraries

To remove stop words from a text, we will be using the nltk library, which stands for Natural Language Toolkit.

NLTK is a leading platform for building Python programs to work with human language data and provides easy-to-use interfaces to many corpora and lexical resources. You need to have nltk installed in your environment.

You can install it using pip:

1
pip install nltk

Step 2: Download the Stopwords Package

NLTK comes with many corpora, toy grammars, trained models, etc. A one-time download of these packages is required which can be done by:

1
2
3
import nltk
 
nltk.download('stopwords')

Step 3: Load the Stopwords Package

Load the English stopwords list which will be used to filter out from the text file:

1
2
3
from nltk.corpus import stopwords
 
stop_words = set(stopwords.words('english'))

Step 4: Reading the Text File

Assume we have a text file named ‘sample.txt’ with the following content:

The quick brown fox jumps over the lazy dog

You can open the file and read each line using the following code:

1
2
with open('sample.txt', 'r') as file:
    lines = file.readlines()

Step 5: Removing Stop Words

For each line in the file, we split the text into words and then check if the word is a stop word, then we ignore it. Otherwise, we include it in the result:

1
2
3
result = []
for line in lines:
    result.append(' '.join([word for word in line.split() if word not in stop_words]))

This will remove stop words from every line in the text file and store the results in the ‘result’ variable.

Here is the full code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
import nltk
from nltk.corpus import stopwords
 
# Downloading stopwords package
nltk.download('stopwords')
 
# Loading stopwords
stop_words = set(stopwords.words('english'))
 
# Reading the text file
with open('sample.txt', 'r') as file:
    lines = file.readlines()
 
# Removing Stop Words
result = []
for line in lines:
    result.append(' '.join([word for word in line.split() if word not in stop_words]))
 
print(result)
 
['The quick brown fox jumps lazy dog']

Conclusion

In this tutorial, you learned how to remove stop words from a text using Python and the nltk library. This is an essential step in pre-processing text data for NLP tasks. Remember, it’s not always necessary to remove stop words. It depends on the task at hand.