π 2024-10-04 β Session: AI Evaluation Workflow Implementation
π 22:45β23:55
π·οΈ Labels: AI, Evaluation, Openai, Workflow, Automation, Schema
π Project: Dev
β Priority: MEDIUM
Session Goal
The goal of this session was to design and implement an AI workflow for evaluating notebooks using predefined rubrics. This involved integrating the OpenAI API and ensuring proper error handling and schema validation.
Key Activities
- Designed a framework for an AI evaluation workflow using traffic light evaluations and structured data storage.
- Updated the
AIEvaluator
class to integrate with the OpenAI API, improving error handling and JSON result storage. - Developed a JSON schema for rubric evaluations, categorizing results as βgreenβ, βyellowβ, or βredβ.
- Implemented a Python function to extract specific consigna schemas from a rubric evaluation schema.
- Resolved errors related to invalid JSON schema in OpenAI API calls, ensuring proper structure and validation.
Achievements
- Successfully designed and partially implemented an AI evaluation workflow.
- Improved the
AIEvaluator
class for better API integration and error handling. - Created a robust JSON schema for rubric evaluations.