data-doc & data-sim Project

Agentic data cleansing and normalization system

Project maintained by realSergiy Hosted on GitHub Pages — Theme by mattgraham

data-sim Application

Overview

The data-sim is a standalone application that programmatically generates realistic manufacturing datasets with embedded data quality challenges. It creates controlled, faulty environments where the data-doc can demonstrate its diagnostic and repair capabilities.

Technology Stack

Backend: Python with LangGraph for agent orchestration
Frontend: React Vite web interface
Database: Supabase (PostgreSQL) via psycopg2
File Storage: Google Drive API
Data Generation: pandas, Faker

Objective

The data-sim will programmatically generate a realistic manufacturing dataset that embodies specific data quality challenges. Its primary function is to create a controlled, faulty environment where the data-doc can demonstrate its diagnostic and repair capabilities.

Data Schema

The simulator will generate the following assets:

Supabase (PostgreSQL) Tables

Table Name	Columns	Description	Related Pain(s)
parts	part_id (TEXT), description (TEXT), revision (TEXT), source_system (TEXT), created_at (TIMESTAMP)	Canonical list of parts, simulating data from two different ERPs	Disparate data, Inconsistent BOM/Part data
bill_of_materials	assembly_part_id (TEXT), component_part_id (TEXT), quantity (INT), version (TEXT)	Defines product structures	Inconsistent BOM/Part data
suppliers	supplier_id (TEXT), supplier_name (TEXT), part_id (TEXT), lead_time_days (INT)	Supplier and lead time information for parts	Makeshift capacity planning
work_in_progress	wo_id (TEXT), part_id (TEXT), status (TEXT), last_scan_location (TEXT), last_scan_time (TIMESTAMP)	Snapshot of current work orders on the shop floor	WIP and quality issues
engineering_changes	eco_id (TEXT), product_affected (TEXT), change_description (TEXT), status (TEXT)	A database record of Engineering Change Orders	Divergent ECOs

Google Drive Files

File Name Pattern	Description	Related Pain(s)
eco_‹eco_id›.pdf	Unstructured PDF documents detailing an engineering change	Divergent ECOs, Reporting and trust deficits
quality_reports/‹wo_id›.csv	CSV exports from a hypothetical quality system with inspection results	WIP and quality issues

Fault Injection

The simulator will inject the following types of faults:

1. Disparate Data Faults

Create duplicate part_ids in the parts table with a different source_system (‘Oracle Cloud’ vs. ‘Oracle Legacy’)
Introduce slight variations in the description for the same part across the two systems

2. Inconsistent BOM/Part Data Faults

Create records in bill_of_materials where a component_part_id does not exist in the parts table (orphaned records)
Create BOMs with conflicting version numbers for the same assembly

3. Divergent ECO Faults

Generate an eco_‹eco_id›.pdf in Google Drive but create no corresponding record in the engineering_changes table
Create a record in engineering_changes with a status of ‘Implemented’ but where the related BOM in bill_of_materials has not been updated

4. WIP & Quality Faults

Insert work_in_progress records with a last_scan_time older than 30 days but a status of ‘Active’
Create work_in_progress records for part_ids that are marked as ‘Obsolete’ in the parts table

Agent Architecture

The data-sim uses LangGraph to orchestrate agents that generate and inject faulty data:

Schema Agent: Creates database schemas and table structures
Data Generation Agent: Uses Faker and pandas to generate realistic manufacturing data
Fault Injection Agent: Systematically introduces data quality issues based on predefined patterns
File Generation Agent: Creates PDF and CSV files in Google Drive with corresponding faults

Web Interface Features

The React Vite frontend provides:

Configuration Panel: Select fault types and severity levels
Data Preview: Review generated data before insertion
Execution Dashboard: Monitor data generation and fault injection progress
Export Options: Download generated datasets for testing