How to Remove Stop Words from a Text File using Python NLTK

In the field of Natural Language Processing (NLP), it’s often necessary to remove stop words from text. What are stop words? These are words that do not contain critical meaning and are often removed from texts such as ‘a’, ‘an’, ‘the’, and ‘in’.

Python’s Natural Language Toolkit, or NLTK, provides a user-friendly way to remove these stop words from a text file. In this tutorial, we will walk you through how to accomplish this task step by step.

Step 1: Install and Import the Necessary Libraries

To start with, we need to install the NLTK library in Python. If you haven’t installed it yet, you can use the pip install command to get it in your environment.

1	pip install nltk

Once NLTK is installed, import it along with the os library. We will use the os library to read our text file in Python.

1 2	import nltk import os

Step 2: Import the List of Stop Words from NLTK

Next, we will import the stop words from NLTK. NLTK has a predefined list of stop words that we can utilize. Let’s use the nltk.corpus module to get these stop words.

1 2	from nltk.corpus import stopwords stop_words = set(stopwords.words('english'))

Step 3: Open and Read Your Text File

Now that we have our list of stop words, let’s us read the text file. We will use Python’s built-in open() function to read our file. In this example, the file we will be reading is named ‘example.txt’.

1 2	with open('example.txt', 'r') as file: file_data = file.read()

example.txt:

The quick brown fox jumps over the lazy dog. In a nearby forest, there are tall trees and green leaves. Birds sing melodiously among the branches, while squirrels play in the shadows.

Step 4: Tokenize the Text

After reading the text file, we need to tokenize the text. Tokenizing the text simply means that we are splitting the text into individual words. We use the nltk.word_tokenize() method for this.

1	tokens = nltk.word_tokenize(file_data)

Step 5: Remove Stop Words

Next, we will remove the stop words from the text. We will iterate over our list of tokens and only keep the words that are not in our list of stop words.

1	filtered_text = [word for word in tokens if not word in stop_words]

['The', 'quick', 'brown', 'fox', 'jumps', 'lazy', 'dog', '.', 'In', 'nearby', 'forest', ',', 'tall', 'trees', 'green', 'leaves', '.', 'Birds', 'sing', 'melodiously', 'among', 'branches', ',', 'squirrels', 'play', 'shadows', '.']

Step 6: Convert the List into Text

You can combine the elements of the filtered_text list into a text without spaces between punctuation marks using Python by iterating through the list and checking whether an element is a punctuation mark. If it is, then you can concatenate it with the previous word without adding a space.

# Initialize an empty string to store the combined text

combined_text = ''

# Iterate through the filtered_text list

for word in filtered_text:

# Check if the word is a punctuation mark

if word in ['.', ',', '!', '?', ':', ';']:

# Concatenate the punctuation mark to the combined_text without a space

combined_text = combined_text.rstrip() + word

else:

# Concatenate the word to the combined_text with a space

combined_text += ' ' + word

Full Python Code

import os

import nltk

from nltk.corpus import stopwords

# Stop Words

stop_words = set(stopwords.words('english'))

# Read Text File

with open('example.txt', 'r') as file:

file_data = file.read()

# Tokenize

tokens = nltk.word_tokenize(file_data)

# Remove Stop Words

filtered_text = [word for word in tokens if not word in stop_words]