Implemented robust web scraping with error handling

• Day: 2023-04-16
• Time: 20:40 to 21:05
• Project: Dev
• Workspace: WP 2: Operational
• Status: Completed
• Priority: MEDIUM
• Assignee: Matías Nehuen Iglesias
• Tags: Web Scraping, Python, Beautifulsoup, Error Handling, Selenium

Description

Session Goal:

The session aimed to develop a robust web scraping script using Python, BeautifulSoup, and Pandas, focusing on error handling and dynamic content extraction.

Key Activities:

• Implemented a Python script using BeautifulSoup to scrape news articles from a specified website and store the data in a Pandas DataFrame.
• Enhanced the script with error handling to manage missing data for article titles, sections, authors, and dates by assigning None when values are not available.
• Utilized try-except blocks to handle cases where article titles or dates may be missing.
• Added conditional checks to avoid errors when accessing missing elements, specifically setting the href value to ‘No href’ when the anchor tag is absent.
• Integrated DataFrame to include an ‘href’ column in the resulting data.
• Explored the use of Selenium for scraping dynamically loaded content, acknowledging the limitations of BeautifulSoup for such tasks and emphasizing legal and ethical considerations.
• Provided guidance on resolving the gaierror encountered when connecting to non-existent URLs.

Achievements:

• Successfully developed a web scraping script with comprehensive error handling mechanisms.
• Identified and addressed limitations in scraping dynamically loaded content.

Pending Tasks:

• Explore further optimization of the scraping process for performance improvements.
• Investigate additional legal and ethical guidelines for web scraping.

Evidence

• source_file=2023-04-16.sessions.jsonl, line_number=4, event_count=0, session_id=e122ae6c9bd7f500442aeaa7a6a0587ecb6b782a969e158174f27da255da35b3
• event_ids: []