Refactored RAG pipeline for modular embedding

2 min read

Refactored RAG pipeline for modular embedding

  • Day: 2025-02-20
  • Time: 19:40 to 20:15
  • Project: Dev
  • Workspace: WP 2: Operational
  • Status: In Progress
  • Priority: MEDIUM
  • Assignee: Matías Nehuen Iglesias
  • Tags: Refactoring, Embedding, Rag Pipeline, Modular Design, CLI

Description

Session Goal

The session aimed to refactor the retrieval-augmented generation (RAG) pipeline to separate and modularize the embedder and retriever components, ensuring scalability and integration.

Key Activities

  • Developed a refactoring plan to separate the embedder and retriever components of the RAG pipeline, detailing design decisions and the proposed file structure.
  • Outlined the architecture for a modular embedding orchestrator to handle dynamic data embedding strategies using Langchain.
  • Designed a unified embedding service capable of handling multiple embedding scripts, with storage solutions in FAISS and Parquet, and future API integration.
  • Refined a command-line interface (CLI) parser to improve structure and validation, including error handling and default modes.
  • Provided solutions for managing CLI arguments in Jupyter Notebooks using argparse.

Achievements

  • Completed the design of a flexible, modular embedder and retriever system for the RAG pipeline.
  • Implemented improvements in CLI parsing and validation for enhanced automation and error handling.

Pending Tasks

  • Implement the proposed modular embedding orchestrator and integrate it with existing systems.
  • Develop API endpoints for the unified embedding service to facilitate external access and integration.

Evidence

  • source_file=2025-02-20.sessions.jsonl, line_number=2, event_count=0, session_id=9fae46782a77982de51d8f46b018fc078236ad7a0396bff2a26ca7be3481acf5
  • event_ids: []

Graph View

Backlinks