deltagradient: Handling Dynamic Content with Selenium

⚡ Handling Dynamic Content with Selenium in Python

Modern websites often rely on JavaScript to load content dynamically after the initial page load. This can make it tricky to scrape data using traditional methods like BeautifulSoup, which only sees the raw HTML. That’s where Selenium comes in — a powerful tool for simulating real user interaction with web pages, including handling dynamically loaded content.

In this post, you’ll learn how to use Selenium with Python to wait for and extract content that loads dynamically.

🧰 What You’ll Need

Python 3.x
selenium library (pip install selenium)
A WebDriver (e.g., ChromeDriver)

🔧 Installing Selenium

pip install selenium

Download the appropriate WebDriver for your browser version and make sure it’s in your system PATH.

🚀 Basic Setup

Here’s a simple Selenium setup using Chrome:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

# Setup WebDriver
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))

# Open a web page
driver.get("https://example.com")

# Print the page title
print(driver.title)

driver.quit()

⏳ Waiting for Dynamic Elements

To handle dynamic content, explicit waits are key. They allow Selenium to wait until a certain condition is met before proceeding.

Example: Waiting for an Element to Appear

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver.get("https://quotes.toscrape.com/js/")

# Wait for quotes to load
WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CLASS_NAME, "quote"))
)

quotes = driver.find_elements(By.CLASS_NAME, "quote")
for quote in quotes:
    print(quote.text)

Explanation:

WebDriverWait waits up to 10 seconds.
presence_of_element_located checks that at least one element with the class quote is present.
This ensures your script doesn't fail if the content is loaded via JavaScript.

📑 Common Expected Conditions

You can wait for various conditions:

presence_of_element_located
element_to_be_clickable
visibility_of_element_located
text_to_be_present_in_element

These are found in:

from selenium.webdriver.support import expected_conditions as EC

🖱️ Interacting with Dynamic Elements

Once elements are loaded, you can interact with them:

Click a button:

button = WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.ID, "loadMore"))
)
button.click()

Fill a form:

search_box = driver.find_element(By.NAME, "q")
search_box.send_keys("Selenium Python")
search_box.submit()

🧩 Example: Scraping Dynamic Data

Let’s say you want to scrape live news headlines from a JS-powered site:

driver.get("https://example-news-site.com")

# Wait until headlines are loaded
headlines = WebDriverWait(driver, 10).until(
    EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".headline"))
)

for h in headlines:
    print(h.text)

This ensures you're scraping only after the content has appeared on the page.

📸 Capturing Screenshots

To debug or visually confirm that your bot sees the same page you do:

driver.save_screenshot("page.png")

📤 Headless Mode (Optional)

To run the browser in the background (useful for automation):

from selenium.webdriver.chrome.options import Options

options = Options()
options.headless = True
driver = webdriver.Chrome(options=options)

🧠 Final Thoughts

Web scraping or automating with dynamic websites can be tricky, but Selenium gives you the power to interact with the page like a real user. With proper use of explicit waits, you can make sure your script runs reliably even when content loads asynchronously.

✅ TL;DR

Use Selenium when pages load content with JavaScript.
Wait for elements using WebDriverWait and expected_conditions.
Interact with page elements just like a user would.
Go headless for efficiency and deployment.

Search This Blog

Handling Dynamic Content with Selenium