BeautifulSoup CSS Selectors: select() and select_one() Guide

Interactive Google Colab Practice Notebook

Practice selecting HTML elements by CSS classes, IDs, nested elements, pseudo-classes, and XML tags interactively inside Google Colab.

View on GitHub

The select() method in BeautifulSoup allows developers to query HTML and XML documents using standard CSS selectors. It provides a powerful, flexible alternative to traditional find() and find_all() methods when extracting complex nested nodes.

Basic CSS Selection Examples 🔝

You can query web pages fetched via the requests library by passing CSS selectors directly into soup.select().

import requests
from bs4 import BeautifulSoup

link = "https://www.plus2net.com/html_tutorial/html_form.php"
content = requests.get(link)
soup = BeautifulSoup(content.text, 'html.parser')

# Select title tag
print(soup.select("title"))

# All meta tags inside head
print(soup.select("head meta"))

# Tags with class='table-striped'
print(soup.select('.table-striped'))

# All anchor tags inside table-striped class
print(soup.select(".table-striped a"))

# Even and odd link selectors
print(soup.select("a:nth-of-type(even)"))
print(soup.select("a:nth-of-type(odd)"))
print(soup.select("a:nth-of-type(2n)"))

Selecting by Class, ID, and Attributes 🔝

You can target specific HTML elements by combining tag names, CSS classes (.classname), IDs (#id), and attributes ([attribute='value']).

content = """<h2>List of web programming languages</h2>
<div class="my_list">
  <p>My Pages one </p>
  <p class="my_pages">My Pages </p>
  <p id="ck1">My ck1 page</p>
  <a href='https://www.plus2net.com' class='home_link'>Home page</a>
</div>"""

from bs4 import BeautifulSoup
soup = BeautifulSoup(content, 'html.parser')

# Select all div elements
print(soup.select("div"))

# Select element by class name
print(soup.select('.my_list'))

# Select paragraph with specific class
print(soup.select('.my_pages'))

# Select element by ID
print(soup.select('#ck1'))

# Select anchor tag inside a specific class
print(soup.select('a.home_link'))

# Select all p tags with a class attribute
print(soup.select('p[class]'))

# Select all p tags with an id attribute
print(soup.select('p[id]'))

Extracting Table Cell Values by Attribute

To target table data cells within a table that has specific attributes:

str1 = soup.select('table[width="170"] td')
print(str1[1].string)
print(str1[2].string)

Using select_one() for Single Elements 🔝

While select() returns a list of all matching elements, select_one() returns only the first matching element (or None if no match is found).

# Get the first matching paragraph element
first_p = soup.select_one('p')
print(first_p.string)  # Output: My Pages one

Using CSS Selectors with XML 🔝

CSS selectors can also be used to query XML documents by utilizing the lxml XML parser option.

import requests
from bs4 import BeautifulSoup

link = "https://www.plus2net.com/php_tutorial/file-xml-demo.xml"
content = requests.get(link)

# Parse XML with BeautifulSoup
soup = BeautifulSoup(content.text, "xml")

# Select all <name> tags
names = soup.select("name")
print(names)

Practical Use Cases 🔝

• Targeting Precise Layout Blocks: Extract data from specific sidebar widgets or main content containers without scraping site headers and footers.
• Scraping Data Tables: Read rows from styled web tables using attribute selectors like table.data-table tr.
• XML Feed Processing: Query RSS feeds or XML documents using tag-level CSS selectors.

Common Errors & Troubleshooting 🔝

• Attribute Error on Empty Match: Calling .string or .text directly on select_one() without checking if it returned None causes an AttributeError. Always check if the result exists.
• Unsupported Pseudo-Classes: Standard CSS pseudo-classes like :hover or :focus are not supported in static BeautifulSoup parsing. Stick to structural selectors like :nth-of-type() or class/ID selectors.
• XML Parsing Failures: Make sure `lxml` is installed via `pip install lxml` when parsing XML feeds with `BeautifulSoup(content, "xml")`.

Frequently Asked Questions (FAQ) 🔝

Q: What is the difference between find_all() and select()?
A: find_all() relies on Python arguments (e.g. find_all('div', class_='my_class')), whereas select() uses standard CSS syntax (e.g. select('div.my_class')).

Q: How do I select an element with multiple CSS classes?
A: Chain the class names together with dots, such as soup.select('.class1.class2').

Subhendu Mohapatra

Author & Instructor at plus2net

I write and maintain practical tutorials on Python, PHP, SQL, JavaScript, HTML, jQuery, and web development at plus2net. The tutorials focus on clear explanations, working examples, and code that readers can test and adapt while learning.

About the author   Watch on YouTube

Connect: