BeautifulSoup find_all(): Extracting Tags & Regex Filtering Guide

Interactive Google Colab Practice Notebook

Practice searching tags, extracting hyperlinks, stripping text, and filtering attributes using Regular Expressions interactively inside Google Colab.

View on GitHub

The find_all() method in BeautifulSoup is one of the most essential search functions in web scraping. It scans an HTML/XML document for all matching instances of a given tag, attribute, or string, returning them as a Python list (ResultSet).

Finding Specific Tags (e.g. <h2>) 🔝

To extract all occurrences of a specific header tag like <h2>, pass the tag name directly into soup.find_all().

import requests
from bs4 import BeautifulSoup

link = "https://www.plus2net.com/html_tutorial/html_form.php"
content = requests.get(link)
soup = BeautifulSoup(content.text, 'html.parser')

# Extract all <h2> tags as a list
h2_tags = soup.find_all("h2")
print(h2_tags)

# Clean extraction: Loop through and print text strings only
for tag in h2_tags:
    print(tag.get_text(strip=True))

Sample Output

How to select a form component
Form tag
Method attribute of the html form
Action attribute
Applications and uses of html form elements

Collecting All Links & Anchor Text 🔝

A frequent requirement in web scraping is extracting all hyperlink addresses (href) alongside their visible anchor text strings.

import requests
from bs4 import BeautifulSoup

link = "https://www.plus2net.com/html_tutorial/html_form.php"
content = requests.get(link)
soup = BeautifulSoup(content.text, 'html.parser')

# Find all anchor tags that contain an href attribute
my_list = soup.find_all("a", href=True)

for tag in my_list:
    url = tag['href']
    anchor_text = tag.get_text(strip=True)
    print(f"Text: {anchor_text} --- URL: {url}")

Using Regular Expressions with find_all() 🔝

By pairing find_all() with Python's re (regular expression) module, you can search for multiple tag patterns or partial attribute values dynamic matching.

import re
import requests
from bs4 import BeautifulSoup

link = "https://www.plus2net.com/html_tutorial/html_form.php"
content = requests.get(link)
soup = BeautifulSoup(content.text, 'html.parser')

# Match both <h1> and <h2> tags using Regex
headers = soup.find_all(re.compile("^(h1|h2)$", re.I))
print(headers)

# Find all <a> or <div> tags
a_or_div = soup.find_all(re.compile("^(a|div)$", re.I))
print(f"Found {len(a_or_div)} 'a' or 'div' tags.")

Sample Regex Output

[<h1 itemprop="headline">Web Form tag & HTML elements</h1>, 
 <h2>How to select a form component</h2>, 
 <h2>Form tag</h2>, 
 <h2>Method attribute of the html form</h2>, 
 <h2>Action attribute</h2>, 
 <h2>Applications and uses of html form elements</h2>]

Practical Use Cases 🔝

• Internal Sitemap & Audit Crawling: Extract all <a href> tags from a page to build a site crawler or audit internal link integrity.
• Heading Hierarchy Validation: Extract h1 through h6 tags using Regex to verify proper outline structure for SEO compliance.
• Batch Media Harvesting: Collect all <img src> elements across target pages for bulk asset downloading.

Common Errors & Troubleshooting 🔝

• KeyError on Missing Attributes: Accessing tag['href'] directly on an anchor tag without an href attribute will raise a KeyError. Use tag.get('href') or filter with soup.find_all('a', href=True).
• None Type Errors with .string: If a tag contains nested child elements, tag.string returns None. Use tag.get_text(strip=True) to reliably extract all text.
• find() vs find_all(): find() returns a single Tag object (the first match), whereas find_all() returns a ResultSet list containing all matches.

Frequently Asked Questions (FAQ) 🔝

Q: How do I pass multiple tag names to find_all() without Regex?
A: Pass a Python list of tag names: soup.find_all(['h1', 'h2', 'h3']).

Q: Can I limit the maximum number of tags returned by find_all()?
A: Yes. Use the limit parameter: soup.find_all('a', limit=5) returns only the first 5 matching tags.

Subhendu Mohapatra

Author & Instructor at plus2net

I write and maintain practical tutorials on Python, PHP, SQL, JavaScript, HTML, jQuery, and web development at plus2net. The tutorials focus on clear explanations, working examples, and code that readers can test and adapt while learning.

About the author   Watch on YouTube

Connect: