How to Replace Outliers with the 5th and 95th Percentile Values in Python

Outliers are observations that lie an abnormal distance from other values in a random sample from a population. They’re usually harmful to most data analysis procedures because they can skew your dataset and give misleading results.

In Python, you can identify and replace these outliers with the 5th and 95th percentile values. This tutorial will guide you on how to go about it.

Step 1: Importing Necessary Libraries

First and foremost, we need to import the necessary Python libraries. We’re going to need Pandas for data management, NumPy for numerical operations, and Matplotlib for data visualization. Use the following commands to import these libraries:

1
2
3
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

Step 2: Creating or Loading Your Data

Create a DataFrame and add some randomly generated data to it as shown below:

1
2
3
np.random.seed(0)
data = pd.DataFrame(np.random.randn(1000), columns=['Value'])
data['Value'].iloc[[1, 2, 5]] = [12, 15, 10]

You can also read data from a CSV file or any other source using the pd.read_csv() function or similar.

Step 3: Identifying and Replacing Outliers

We wish to replace values below the 5th percentile and above the 95th percentile. Call the quantile() function to find these two values and use np.where to replace the outliers.

1
2
3
4
q_low = data['Value'].quantile(0.05)
q_hi  = data['Value'].quantile(0.95)
 
data = data[(data['Value'] < q_hi) & (data['Value'] > q_low)]

Congratulations, you have completed the tutorial!

Complete Code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
 
# Step 2:
np.random.seed(0)
data = pd.DataFrame(np.random.randn(1000), columns=['Value'])
data['Value'].iloc[[1, 2, 5]] = [12, 15, 10]
 
# Step 3:
q_low = data['Value'].quantile(0.05)
q_hi  = data['Value'].quantile(0.95)
 
data = data[(data['Value'] < q_hi) & (data['Value'] > q_low)]

Conclusion

In conclusion, being able to identify and handle outliers is an essential skill while working with datasets in Python. This tutorial has outlined the basics of replacing outliers using percentile values, an effective and yet simple method.

There are several other ways to handle outliers, each method unique to the kind of data you’re working with. Therefore, always choose the method that best fits your data context.