Automating Web Browsing with Python

 

Automating Web Browsing with Python

Web automation allows users to interact with websites programmatically, performing tasks like form filling, clicking buttons, and downloading files. Python’s Selenium library is commonly used for automating web interactions.

---

Installing Required Libraries

pip install selenium webdriver-manager

• Selenium: Automates web browser interactions.
• webdriver-manager: Automatically installs and manages the ChromeDriver.

---

Setting Up Selenium WebDriver

Launching a Browser

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

# Setup WebDriver
service = Service(ChromeDriverManager().install())
driver = webdriver.Chrome(service=service)

# Open a website
driver.get("https://example.com")

# Print title
print(driver.title)

# Close browser
driver.quit()

---

Interacting with Web Elements

Finding Elements

from selenium.webdriver.common.by import By

driver.get("https://example.com")

# Find element by ID
element = driver.find_element(By.ID, "search-box")

# Find element by name
element = driver.find_element(By.NAME, "q")

# Find element by class name
element = driver.find_element(By.CLASS_NAME, "button")

# Find element by tag name
element = driver.find_element(By.TAG_NAME, "h1")

# Find element by XPath
element = driver.find_element(By.XPATH, "//input[@type='text']")

print("Element found:", element)

---

Entering Text into Input Fields

# Locate input field
search_box = driver.find_element(By.NAME, "q")

# Enter text
search_box.send_keys("Python automation")

# Submit the form
search_box.submit()

---

Clicking Buttons

# Locate and click a button
button = driver.find_element(By.CLASS_NAME, "search-btn")
button.click()

---

Handling Dropdowns and Checkboxes

Selecting an Option from a Dropdown

from selenium.webdriver.support.ui import Select

# Locate dropdown
dropdown = Select(driver.find_element(By.ID, "category"))

# Select by visible text
dropdown.select_by_visible_text("Technology")

# Select by index
dropdown.select_by_index(2)

# Select by value
dropdown.select_by_value("tech")

---

Checking and Unchecking a Checkbox

# Locate checkbox
checkbox = driver.find_element(By.ID, "subscribe")

# Check if already selected
if not checkbox.is_selected():
    checkbox.click()

---

Handling Pop-ups and Alerts

Accepting an Alert

from selenium.webdriver.common.alert import Alert

# Wait and switch to alert
alert = Alert(driver)
alert.accept()  # Click OK

---

Dismissing an Alert

alert.dismiss()  # Click Cancel

---

Handling Authentication Pop-ups

# Pass username and password in URL
driver.get("https://username:password@example.com")

---

Handling File Uploads and Downloads

Uploading a File

# Locate file input field
file_input = driver.find_element(By.NAME, "file_upload")

# Provide file path
file_input.send_keys("C:\\Users\\User\\Documents\\report.pdf")

---

Downloading a File Automatically

from selenium.webdriver.chrome.options import Options

# Set download preferences
chrome_options = Options()
chrome_options.add_experimental_option("prefs", {
    "download.default_directory": "C:\\Users\\User\\Downloads",
    "download.prompt_for_download": False,
    "download.directory_upgrade": True,
    "safebrowsing.enabled": True
})

# Launch browser with options
driver = webdriver.Chrome(service=service, options=chrome_options)
driver.get("https://example.com/download")

---

Scrolling and Navigating Pages

Scrolling Down the Page

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

---

Navigating Between Pages

# Go to a new URL
driver.get("https://example.com")

# Go back
driver.back()

# Go forward
driver.forward()

# Refresh page
driver.refresh()

---

Waiting for Elements to Load

Implicit Wait

driver.implicitly_wait(10)  # Waits for up to 10 seconds for elements to appear

---

Explicit Wait

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Wait for an element to be visible
element = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CLASS_NAME, "dynamic-content"))
)

---

Automating Login to a Website

driver.get("https://example.com/login")

# Enter username
driver.find_element(By.NAME, "username").send_keys("myusername")

# Enter password
driver.find_element(By.NAME, "password").send_keys("mypassword")

# Click login button
driver.find_element(By.CLASS_NAME, "login-btn").click()

print("Login successful.")

---

Automating Data Extraction and Emailing Reports

import yagmail

# Extract page content
content = driver.find_element(By.TAG_NAME, "body").text

# Send extracted data via email
yag = yagmail.SMTP("your_email@gmail.com", "your_password")
yag.send(
    to="recipient@example.com",
    subject="Automated Web Data Report",
    contents=content
)

print("Report emailed successfully.")

---

Conclusion

This section covered automating web browsing with Python using Selenium, including form filling, button clicks, dropdown selections, pop-up handling, file uploads/downloads, scrolling, and login automation. These techniques are useful for automating repetitive web tasks.

Would you like additional examples or modifications?