📅 2023-08-05 — Session: Developed DBML schema and data processing scripts
🕒 21:00–22:15
🏷️ Labels: DBML, Python, Data Transformation, Data Modeling, Pandas
📂 Project: Dev
⭐ Priority: MEDIUM
Session Goal
The session aimed to develop a comprehensive DBML schema for data transformation processes and enhance data processing capabilities using Python.
Key Activities
- Created a DBML schema to outline the structure for input and transformed data tables from CSV files.
- Defined 1:1 relationships between columns in tables using DBML, visualized with dbdiagram.io.
- Updated the DBML representation to include new tables (RFC1, RFC2, RFC3) and their relationships.
- Provided Python code snippets for loading CSV columns, updating DataFrame sources from GitHub, and standardizing region names.
- Demonstrated concatenation of multiple CSV files and merging them using an ‘ID’ column.
Achievements
- Successfully developed a detailed DBML schema for enhanced data modeling.
- Implemented Python scripts to automate data loading, cleaning, and transformation tasks.
Pending Tasks
- Further refinement of DBML schemas to incorporate additional data tables and relationships.
- Optimization of Python scripts for performance improvements.