Integrated MLflow and DVC for MLOps Enhancement

• Day: 2024-04-15
• Time: 15:45 to 17:20
• Project: Dev
• Workspace: WP 2: Operational
• Status: Completed
• Priority: MEDIUM
• Assignee: Matías Nehuen Iglesias
• Tags: Mlflow, DVC, Mlops, Flask, Docker

Description

Session Goal

The session aimed to enhance machine learning operations (MLOps) by integrating MLflow and DVC for better model and data versioning, experiment tracking, and deployment.

Key Activities

• Explored various MLOps tools like MLflow, DVC, and Kubeflow, focusing on their capabilities in model and data versioning.
• Created a workflow diagram using Graphviz to visualize the integration of MLflow and DVC.
• Configured MLflow tracking URI and set up MLflow in Docker for experiment tracking.
• Enhanced a Flask endpoint with MLflow for improved model retraining and experiment tracking.
• Resolved MLflow-related issues such as PermissionError, tracking URI path issues, and experiment not found errors.
• Implemented a naming convention for MLflow experiments to ensure meaningful and unique experiment identifiers.
• Addressed file path discrepancies in Python for model saving and loading, and improved model loading in Flask applications.
• Updated model retrieval and dropdown selection in the application.

Achievements

• Successfully integrated MLflow and DVC into the workflow, enhancing model and data management.
• Resolved several configuration and error handling issues related to MLflow and Flask integration.
• Established a robust framework for experiment tracking and model management in MLOps.

Pending Tasks

• Further testing of the integrated system to ensure stability and performance.
• Documentation of the setup and integration process for future reference.

Evidence

• source_file=2024-04-15.sessions.jsonl, line_number=3, event_count=0, session_id=efc4f625737451e9ea03b18957063130f9219c4acd14d235c67709e1b3949a38
• event_ids: []