Validated KB-paper-kb modular handoff and pipeline repair

• Day: 2026-05-19
• Time: 11:10 to 11:25
• Project: Dev
• Workspace: WP 2: Operational
• Status: In Progress
• Priority: HIGH
• Assignee: Matías Nehuen Iglesias
• Tags: Migration, Chunk-Set, Tei_Runner, Contracts, Debugging, Integration-Tests

Description

Session Goal

Validate the post-migration behavior of the KB ↔ paper-kb system and ensure the pipeline can operate independently with reliable handoffs under small workloads. The broader aim was to move from refactoring to proving contract-driven modularity across KB, paper-kb, openalex-gui, and abstract-scroller.

Key Activities

• Reviewed the architecture direction for a four-system modular setup with explicit artifact contracts, boundary rules, and staged integration tests.
• Diagnosed the TEI runner / chunk-set pipeline failures and traced them to local packaging and fixture hygiene rather than a fundamental architecture flaw.
• Identified that empty API responses were downstream symptoms of tei_runner failing before artifacts/chunk_sets was created.
• Narrowed the failure to parse-only execution being coupled too early to Chroma / embedding imports, which prevented chunk-set artifact generation.
• Distinguished a late-stage .done marker bookkeeping bug from the core producer path, indicating the pipeline had already advanced past parsing and artifact emission.
• Captured the architectural intent to keep KB as the contract owner / processing substrate, paper-kb as the content provider, and abstract-scroller as the review surface.

Achievements

• Clarified that the immediate priority is not more refactoring, but validating independent execution and handoff behavior after the chunk-set migration.
• Established a staged repair plan: fix packaging / dev-path issues, decouple parse-only TEI execution from Chroma, validate chunk-set emission, then smoke-test the API and clean up bookkeeping.
• Confirmed the modularization strategy around explicit contracts, repository boundaries, and battle-testing small workloads before broader integration.

Pending Tasks

• Patch tei_runner so non-embedding parsing does not import KB embedding code too early.
• Verify that artifacts/chunk_sets is generated correctly and that the API serves non-empty results.
• Fix .done marker directory creation / idempotency bookkeeping.
• Investigate and remove duplicate chunk records in /chunks if they persist after the pipeline repair.
• Continue the contract-first PR roadmap and integration tests across KB, paper-kb, openalex-gui, and abstract-scroller.

Evidence

• source_file=2026-05-19.sessions.jsonl, line_number=1, event_count=0, session_id=4423f2ddd829f364604ee6a6d157b97ff35634400724f00e82041d75730b7501
• event_ids: []