DQM Agent: Keeping Full Datasets in the Browser
A data-quality workflow that uses an LLM to choose and adjudicate checks while executing full-row inspection and file splitting locally in the browser.
- JavaScript
- React
- LangGraph.js
- Express
- Vitest
- Groq
Separates semantic planning from deterministic execution: the LLM selects checks, pure JavaScript evaluates all rows locally, and only aggregate findings plus row indices reach the judging step.
Contents
The Problem
Data-quality rules depend on context, but sending an entire uploaded dataset to an LLM creates unnecessary privacy and cost exposure. The engineering question is where probabilistic judgment helps and where deterministic local code is safer.
Why This Approach?
The workflow assigns the LLM bounded decisions—selecting relevant checks and interpreting aggregate failures—while ordinary JavaScript performs exhaustive row-level computation. This keeps the orchestration explainable and makes core checks independently testable.
Architecture
The four stages are schema analysis, local check execution, row judging, and file splitting. The browser parses CSV, XLSX, JSON, or TSV input. Six implemented checks cover nulls, duplicates, type mismatches, outliers, schema constraints, and referential integrity.
Implementation
computeChecks is a pure utility with Vitest coverage for each check and flagged-row aggregation. The graph runner streams named node updates so the interface can display progress without coupling UI components to check logic. The backend keeps the Groq key off the client.
The privacy boundary is narrower than “nothing leaves the browser”: the schema step sends column metadata and five sample rows, while the row-judge step sends aggregate findings and indices without raw cell values. That distinction is important when evaluating the design.
Experiments and Results
The repository provides deterministic utility tests rather than benchmark metrics. Tests cover positive and negative cases for the six checks and verify aggregation of failing row indices. No production-scale throughput, precision, or false-positive rate is claimed.
What Failed?
The current boundary still exposes five sample rows to the schema analyst. Referential-integrity inference also depends on receiving an appropriate pair of columns, and IQR-based outlier detection can flag valid heavy-tailed observations. The row judge introduces model variability after deterministic checks.
What I Learned
The architecture demonstrates a useful agent pattern: use language models to narrow or interpret a task, then keep bulk computation deterministic. It also shows that privacy claims should name the exact payload at every boundary.
Next Steps
- Replace raw sample rows with configurable redaction or local profiling.
- Add fixtures measuring false positives across representative datasets.
- Add database connectors and repeatable scheduled runs.
- Version prompts and record judge decisions for auditability.