Developed web scraper for Buenos Aires norms

2 min read

Developed web scraper for Buenos Aires norms

  • Day: 2024-08-16
  • Time: 02:05 to 02:35
  • Project: Dev
  • Workspace: WP 2: Operational
  • Status: In Progress
  • Priority: MEDIUM
  • Assignee: Matías Nehuen Iglesias
  • Tags: Web Scraping, Python, Automation, Buenos Aires, Data Extraction

Description

Session Goal: The goal of this session was to develop a software tool to automatically check and download daily government norms published by Buenos Aires Province, focusing on resolutions from the current year.

Key Activities:

  • Developed a Python script utilizing Requests, BeautifulSoup, and Pandas to parse URLs and extract relevant data into a structured format.
  • Analyzed HTML structure to design a software architecture for data extraction and organization.
  • Implemented web scraping steps, including pagination handling and data storage in a Pandas DataFrame.
  • Enhanced error handling in the Python script to safely access list elements and prevent errors.
  • Designed a function to generate URLs for searching Buenos Aires norms, using wildcard parameters and specific filters.
  • Built a Python function to construct search URLs, filtering out empty values for clean query strings.
  • Created a Python script for daily data scraping, appending results to a CSV file with error handling and logging.

Achievements:

  • Successfully developed a web scraper for Buenos Aires government norms, capable of handling pagination and storing data efficiently.
  • Improved error handling mechanisms in the scripts to ensure robustness.

Pending Tasks:

  • Further testing and optimization of the web scraper for different types of norms and date ranges.
  • Integration of the URL generation function with the main scraping workflow.

Evidence

  • source_file=2024-08-16.sessions.jsonl, line_number=1, event_count=0, session_id=df2c3d1300e3cc5fc4915981a46f97b4ddfd0d4dda643e1896a8a2ca8f0abdb6
  • event_ids: []

Graph View

Backlinks