Processing XML Data from Web Scraping in Python

import requests
import xml.etree.ElementTree as ET

response = requests.get("https://rss.nytimes.com/services/xml/rss/nyt/Technology.xml")
root = ET.fromstring(response.content)

for item in root.findall('./channel/item'):
    print(item.find('title').text)

Use Python's requests and xml.etree.ElementTree to fetch and parse remote XML/RSS feeds instantly:

Web scraping often involves extracting structured XML data from RSS feeds, syndication channels, and web APIs.

Python streamlines this process by combining network requests with built-in parsing libraries like ElementTree or persistent storage engines like SQLite.

1. Fetching and Parsing XML Data from an RSS Feed

Many websites and news organizations provide information in XML format via RSS feeds. Below is a complete implementation covering SSL handling, status checks, and robust parsing:

Code Example

import requests
import xml.etree.ElementTree as ET

rss_url = "https://rss.nytimes.com/services/xml/rss/nyt/Technology.xml"

try:
    response = requests.get(rss_url)
    if response.status_code == 200:
        xml_content = response.text
    else:
        print(f"Failed to fetch RSS feed. Status code: {response.status_code}")
        exit()
except requests.exceptions.SSLError as e:
    print("SSL Error encountered:", str(e))
    exit()

try:
    root = ET.fromstring(xml_content)
except ET.ParseError as e:
    print("Error parsing XML content:", str(e))
    exit()

print("\nTechnology News Headlines:\n")
for item in root.findall('./channel/item'):
    title = item.find('title').text
    link = item.find('link').text
    pub_date = item.find('pubDate').text
    print(f"Title: {title}\nLink: {link}\nPublished on: {pub_date}\n")

Explanation:

requests.get(): Downloads the remote XML content.
ET.fromstring(): Converts the raw response text into an in-memory XML element tree.
root.findall(): Locates repeating syndication nodes (`./channel/item`).

2. Efficiently Storing RSS Feed Data in SQLite

To prevent duplicate entries and maintain historical records, scraped RSS items can be upserted into an SQLite database:

Code Example

import requests
import xml.etree.ElementTree as ET
import sqlite3
from datetime import datetime

rss_url = "http://feeds.bbci.co.uk/news/world/rss.xml"
db_file = "rss_feed_data.db"

conn = sqlite3.connect(db_file)
cursor = conn.cursor()

cursor.execute('''
CREATE TABLE IF NOT EXISTS rss_feed (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    title TEXT NOT NULL,
    link TEXT NOT NULL UNIQUE,
    pub_date TEXT NOT NULL
)
''')
conn.commit()

response = requests.get(rss_url)
root = ET.fromstring(response.content)

cursor.execute("SELECT MAX(pub_date) FROM rss_feed")
last_pub_date = cursor.fetchone()[0]

new_records = 0
for item in root.findall('./channel/item'):
    title = item.find('title').text
    link = item.find('link').text
    pub_date_text = item.find('pubDate').text

    try:
        cursor.execute("INSERT INTO rss_feed (title, link, pub_date) VALUES (?, ?, ?)",
                       (title, link, pub_date_text))
        new_records += 1
    except sqlite3.IntegrityError:
        # Duplicate link skipped due to UNIQUE constraint
        pass

conn.commit()
conn.close()
print(f"Script completed. {new_records} new record(s) added.")

3. Integrating Google News RSS Feed

You can target specific keyword queries using customized Google News RSS endpoints:

import requests
import xml.etree.ElementTree as ET

rss_url = "https://news.google.com/rss/search?q=Stock+Market&hl=en-IN&gl=IN&ceid=IN:en"

response = requests.get(rss_url)
root = ET.fromstring(response.content)

channel = root.find("channel")
for item in channel.findall("item")[:5]:
    print("-", item.find("title").text)
    print("  Link:", item.find("link").text)

4. Frequently Asked Questions (FAQ) & Common Errors

Web servers sometimes return HTML error pages (like 403 Forbidden or 500 Server Error) with an HTTP 200 status or unescaped ampersands in RSS titles. Always wrap `ET.fromstring()` in a try-except block and verify `response.headers.get('content-type')`.

Many RSS feeds include XML namespaces (e.g., `xmlns:dc="http://purl.org/dc/elements/1.1/"`). Use dictionary prefix mappings when calling `.find()` or `.findall()` (e.g., `root.findall('.//dc:creator', namespaces={'dc': 'http://...'})`).

Conclusion

Scraping and processing XML data from web feeds allows you to aggregate real-time content efficiently using Python's requests and ElementTree libraries.

Interactive Notebook & Practice Exercises

Practice processing XML data from web scraping in our companion Jupyter Notebook:

Interactive Companion Jupyter Notebook

Explore interactive code cells and test fetching and parsing RSS feeds using Python.

View on GitHub

XML Overview Create XML Read XML Modify XML Python Tutorials

Subhendu Mohapatra

Author & Instructor at plus2net

I write and maintain practical tutorials on Python, PHP, SQL, JavaScript, HTML, jQuery, and web development at plus2net. The tutorials focus on clear explanations, working examples, and code that readers can test and adapt while learning.

About the author Watch on YouTube

Connect: