📅 2025-04-29 — Session: Debugged and Automated Upwork Job Data Extraction
🕒 22:25–22:55
🏷️ Labels: Graphql, API, Debugging, Automation, Upwork
📂 Project: Dev
⭐ Priority: MEDIUM
Session Goal
The primary goal of this session was to debug and automate the process of extracting job data from Upwork’s GraphQL API.
Key Activities
- Debugging GraphQL API Access: Explored methods for accessing a GraphQL API in a browser extension, focusing on extracting data from the network tab and building userscripts.
- Manual Job Scraping: Used JavaScript in the browser DevTools Console to manually reproduce job scraping calls, including crafting a sample GraphQL query and handling authentication.
- API Error Debugging: Addressed and resolved a TypeError in the Upwork API, providing troubleshooting steps and a simplified code snippet.
- Data Retrieval and Download: Successfully accessed and retrieved job data from Upwork’s GraphQL API, including job titles and descriptions, and downloaded the job feed as raw JSON data.
- Error Handling in GraphQL Queries: Fixed errors in GraphQL queries by ensuring only defined fields in the schema were queried.
- Automation Techniques: Outlined methods for automating job scraping using a Chrome extension, a Tampermonkey script, and a Python script with Selenium.
Achievements
- Successfully debugged and resolved multiple issues related to Upwork’s API and GraphQL queries.
- Extracted and structured job data from Upwork’s API for further analysis and automation.
Pending Tasks
- Implement and test the outlined automation methods to streamline job scraping from Upwork.
- Further refine error handling mechanisms to ensure robustness in data extraction processes.