📅 2024-10-04 — Session: Implemented AI Workflow for Evaluation System
🕒 22:50–23:55
🏷️ Labels: Ai Evaluation, Openai Api, Json Schema, Error Handling
📂 Project: Dev
⭐ Priority: MEDIUM
Session Goal
The session aimed to design and implement an AI workflow for evaluating notebooks using predefined rubrics, integrating OpenAI API, and handling errors effectively.
Key Activities
- AI Workflow Design: Outlined the design and implementation plan for an AI workflow to evaluate notebooks with traffic light evaluations and structured data storage.
- AIEvaluator Class Update: Updated the
AIEvaluatorclass to integrate with the latest OpenAI API, including improved error handling. - JSON Schema Development: Created a JSON schema for rubric evaluations, categorizing them as ‘green’, ‘yellow’, or ‘red’.
- Schema Extraction Function: Developed a Python function to extract specific consigna schemas from a rubric evaluation schema.
- Error Resolution: Addressed JSON schema errors in OpenAI API calls, ensuring valid schema structuring and error handling.
Achievements
- Successfully designed an AI workflow for notebook evaluation.
- Updated and improved the
AIEvaluatorclass for better API integration. - Developed a robust JSON schema for rubric evaluations.
- Implemented error handling strategies for OpenAI API calls.
Pending Tasks
- Further testing of the AI evaluation system with real data to ensure robustness.
- Optimization of the error handling mechanisms for different edge cases.