How To Get Data From Url In Python

In this tutorial, we will learn how to get data from a URL in Python. This is a common task when working with web services, APIs or simply trying to scrape data from web pages.

We will make use of the popular Python library requests to download the web page and BeautifulSoup to parse the HTML content.

Step 1: Install Required Libraries

First, we need to ensure we have the necessary libraries installed. This can be done using the pip command. Open your terminal or command prompt and run the following commands:

1
2
3
pip install requests
pip install beautifulsoup4
pip install lxml

Step 2: Import the Libraries

Now that we have the necessary libraries installed, let’s import them into our Python script:

1
2
import requests
from bs4 import BeautifulSoup

Step 3: Download the Web Page Content

In this step, we will write a function that downloads the content of a given URL. The requests library makes it very easy to download web pages as it handles all the necessary HTTP handling in the background. Our function will return the downloaded content as a string.

1
2
3
4
5
6
7
def get_page_content(url):
    response = requests.get(url)
 
    if response.status_code == 200:
        return response.text
    else:
        return None

Step 4: Parse the HTML Content

Now that we have a function to download the web page content, we can use BeautifulSoup to parse the HTML and extract any interesting data on the page. Let’s write a function that accepts the HTML content as a string and returns a BeautifulSoup object.

1
2
3
def parse_html(content):
    soup = BeautifulSoup(content, 'lxml')
    return soup

Step 5: Extract Data from the HTML

In this step, we will write a custom function to extract the data we are interested in from the BeautifulSoup object. This function will be specific to the web page we are working with, so you will need to modify it to suit your use case. For this tutorial, let’s assume we want to extract all the headings from an HTML page:

1
2
3
4
5
6
7
def get_headings(soup):
    headings = []
 
    for heading in soup.find_all(['h1', 'h2', 'h3', 'h4', 'h5', 'h6']):
        headings.append(heading.text.strip())
 
    return headings

Step 6: Putting It All Together

Now that we have all the necessary functions, we can put them together to download and parse a web page and extract the data we want. Here’s an example script that demonstrates how to use these functions:

1
2
3
4
5
6
7
8
9
url = 'https://example.com'
content = get_page_content(url)
if content:
    soup = parse_html(content)
    headings = get_headings(soup)
    for heading in headings:
        print(heading)
else:
    print("Failed to download the web page.")

If everything is working correctly, the script should download the web page at the specified URL, parse the HTML content, and print out all the headings on the page.

Full Code

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
import requests
from bs4 import BeautifulSoup
 
def get_page_content(url):
    response = requests.get(url)
 
    if response.status_code == 200:
        return response.text
    else:
        return None
 
def parse_html(content):
    soup = BeautifulSoup(content, 'lxml')
    return soup
 
def get_headings(soup):
    headings = []
 
    for heading in soup.find_all(['h1', 'h2', 'h3', 'h4', 'h5', 'h6']):
        headings.append(heading.text.strip())
 
    return headings
 
url = 'https://example.com'
content = get_page_content(url)
if content:
    soup = parse_html(content)
    headings = get_headings(soup)
    for heading in headings:
        print(heading)
else:
    print("Failed to download the web page.")

Output

Example Domain

Conclusion

In this tutorial, we learned how to download and parse HTML data from a URL in Python using the requests and BeautifulSoup libraries. These tools make it easy to extract and manipulate data from web pages and can be customized to suit your specific needs. By following these steps, you can easily get the data you need from any URL.