Revamped ML Project Structure and Data Pipeline

2 min read

Revamped ML Project Structure and Data Pipeline

  • Day: 2025-10-05
  • Time: 01:25 to 02:45
  • Project: Dev
  • Workspace: WP 2: Operational
  • Status: In Progress
  • Priority: MEDIUM
  • Assignee: Matías Nehuen Iglesias
  • Tags: Machine Learning, Project Management, Refactoring, Data Pipeline

Description

Session Goal

The primary objective of this session was to revamp the structure of a machine learning project, focusing on modularization, data pipeline improvements, and configuration management.

Key Activities

  • Structured Revamp Plan: Developed a comprehensive plan to revamp the ML project, including repository layout, environment setup, and migration steps.
  • Module Refactoring: Detailed refactoring of the EPH project scripts into a modular structure to prevent target leakage and ensure proper cross-validation.
  • Production-Ready IO Module: Created a robust Python module for data handling tasks, focusing on directory management and file operations.
  • YAML Configuration Loader: Developed a minimal script for loading YAML configurations, ensuring safe handling of paths and defaults.
  • Data Pipeline Structure: Outlined responsibilities and conceptual layers for preprocessing data, separating universal alignment from project-specific transformations.
  • Clarification on CPython Artifacts: Provided guidance on handling CPython internal artifacts and future imports to avoid coding pitfalls.
  • Training Loop Analysis: Critiqued a training loop for a classifier and regressor, identifying issues such as data leakage and recommending best practices.

Achievements

  • Successfully outlined a modular structure for the ML project and data pipeline.
  • Created efficient and minimal scripts for configuration and data handling.
  • Addressed and clarified common pitfalls in Python coding practices, particularly with future imports.
  • Provided actionable recommendations for improving training loops and production pipelines.

Pending Tasks

  • Implement the proposed refactoring and modularization in the EPH project.
  • Apply the recommended training loop fixes and production pipeline improvements.

Evidence

  • source_file=2025-10-05.sessions.jsonl, line_number=0, event_count=0, session_id=334f641c1d0a9efad3fa5e98e2e0d51508f4a4982c49cf3908acc81770180fff
  • event_ids: []

Graph View

Backlinks