How To Change Encoding In Python

In this tutorial, we will learn how to change encoding in Python by exploring different approaches, such as the encode() and decode() methods, as well as read and write functions for files using specific encodings.

This will provide a better understanding of how to work with different text encodings in Python, ensuring your data is processed correctly.

Step 1: Understanding Default Encoding in Python

By default, Python uses UTF-8 encoding for strings. You can check this using the sys module:

1
2
import sys
print(sys.getdefaultencoding())

The output will be:

utf-8

Step 2: Using Encode() and Decode() Methods

To change the encoding of a string in Python, you can use the encode() and decode() methods. Let’s look at an example:

1
2
3
4
5
6
7
8
9
original_string = "This is a sample string."
utf8_encoded = original_string.encode('utf-8')
utf16_encoded = original_string.encode('utf-16')
 
print("UTF-8 Encoded:", utf8_encoded)
print("UTF-16 Encoded:", utf16_encoded)
 
decoded_string = utf16_encoded.decode('utf-16')
print("Decoded String:", decoded_string)

This would give the following output:

UTF-8 Encoded: b'This is a sample string.'
UTF-16 Encoded: b'\xff\xfeT\x00h\x00i\x00s\x00 \x00i\x00s\x00 \x00a\x00 \x00s\x00a\x00m\x00p\x00l\x00e\x00 \x00s\x00t\x00r\x00i\x00n\x00g\x00.'
Decoded String: This is a sample string.

In this example, we first encoded the original_string to UTF-8 and UTF-16 formats using the encode() method, and printed the encoded results. We then used the decode() method to convert the UTF-16 encoded string back to its original form.

Step 3: Reading and Writing Files with Different Encodings

When working with files, you can specify the encoding to use for reading and writing by using the encoding parameter in the open() function. Let’s look at an example:

First, let’s create a sample text file with UTF-8 encoding:

1
2
with open('sample_utf8.txt', 'w', encoding='utf-8') as file:
    file.write("This is a sample text file encoded in UTF-8 format.")

Now, let’s read this file with UTF-16 encoding:

1
2
3
4
5
6
7
# Reading the file with incorrect encoding (UTF-16 instead of UTF-8)
try:
    with open('sample_utf8.txt', 'r', encoding='utf-16') as file:
        content = file.read()
        print(content)
except Exception as e:
    print("Error:", e)

The output will be:

Error: 'utf-16-le' codec can't decode byte 0x78 in position 50: truncated data

As we can see, Python raises an error because we are trying to read the file using the UTF-16 encoding, which is incorrect. To fix this, we need to read the file using the correct encoding (UTF-8):

1
2
3
4
# Reading the file with correct encoding (UTF-8)
with open('sample_utf8.txt', 'r', encoding='utf-8') as file:
    content = file.read()
    print(content)

The output will be:

This is a sample text file encoded in UTF-8 format.

Now we have read the file correctly using the UTF-8 encoding.

Full Code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
import sys
print(sys.getdefaultencoding())
 
original_string = "This is a sample string."
utf8_encoded = original_string.encode('utf-8')
utf16_encoded = original_string.encode('utf-16')
 
print("UTF-8 Encoded:", utf8_encoded)
print("UTF-16 Encoded:", utf16_encoded)
 
decoded_string = utf16_encoded.decode('utf-16')
print("Decoded String:", decoded_string)
 
with open('sample_utf8.txt', 'w', encoding='utf-8') as file:
    file.write("This is a sample text file encoded in UTF-8 format.")
    # Reading the file with incorrect encoding (UTF-16 instead of UTF-8)
    try:
        with open('sample_utf8.txt', 'r', encoding='utf-16') as file:
            content = file.read()
            print(content)
    except Exception as e:
        print("Error:", e)
 
# Reading the file with correct encoding (UTF-8)
with open('sample_utf8.txt', 'r', encoding='utf-8') as file:
    content = file.read()
    print(content)
 

Conclusion

In this tutorial, we learned how to change encoding in Python using the encode() and decode() methods, as well as how to read and write files with different encodings. Properly handling text encodings is essential for working with various data formats and ensuring that your data is processed accurately.

Remember to always verify that you are using the correct encoding when working with text data, either by checking the documentation or by examining the data itself when possible.