Michael
438bc0f84d
Six targeted changes that drop the user-visible analyzer scan time from
"go for coffee" to sub-second on 1 GB inputs and reduce peak RSS by ~10×.
src/core/io.py
- detect_encoding: open + read sample bytes instead of read_bytes()[:N].
Was allocating the full file in memory just to slice the head; on a
1 GB input this saves a 1 GB intermediate allocation.
- repair_bytes: byte-level smart-quote fold via bytes.replace when the
input is UTF-8. The probe (b"\\xe2\\x80" / b"\\xc2\\xab" / b"\\xc2\\xbb")
is a single C-implemented contains check that skips the entire fold
stage on files with no smart quotes — most of them.
- repair_bytes: skip the per-row csv.reader walk unless a cheap byte
scan finds a currency sigil ($/€/£), the delimiter is non-comma, the
decoder substituted U+FFFD, or _has_field_count_mismatch detects an
unquoted-delimiter row. csv.reader was the dominant cost in
repair_bytes on big files (materializes a list of every row).
- _has_field_count_mismatch: hand-rolled quote-state walker; one pass,
no allocation, returns True at first mismatch. False positives just
fall through to the slower _repair_rows pass.
src/core/analyze.py
- _load_for_analysis: read only ~max(4KB, sample_rows × 256B × 2) head
bytes for the analyzer's sample-mode scan. Drops analyze(sample_rows
=1000) from "read + repair full file" to "read + repair 500KB" —
150× faster on a 1.25 GB file. Falls back to a single full-file
retry if pandas reports fewer rows than the cap.
- Compiled regex character classes for hot-path detectors and a
_vec_match_count helper that runs Series.str.contains in C instead
of Python per-cell loops. Detectors converted: smart_punctuation,
invisible_chars (NBSP + zero-width), whitespace_padding,
null_like_sentinels, mojibake, encoding_uncertainty,
mixed_case_email, leading_zero_ids.
src/core/fixes.py
- _vectorized_translate / _vectorized_regex_sub: pandas-native string
transforms for the fixes that are pure character maps (strip_nbsp,
fold_smart_punctuation, strip_zero_width). Series.str.translate
runs in C — 10-50× faster than per-cell Python.
- _apply_to_strings: replaced inner per-cell loops with Series.map +
boolean-mask diff for the count.
- All fix entry points read an "inplace" flag from payload and thread
it through the helpers.
src/core/normalize.py
- apply_decisions: takes a single working copy at the top, then sets
payload["inplace"] = True so each chained fix mutates that copy.
Previously every fix did df.copy(); N fixes × 6 GB DataFrame =
30+ GB peak. Now: one 6 GB allocation.
Validation: 765 passed, 17 xfailed (no regressions). 100 MB benchmark:
stage before after
------------------------------ ------- --------
detect_encoding 0.97s+1.3GB ~0s + 0 MB
analyze (sample_rows=1000) 235.76s 0.08s
_load_for_analysis (1000 rows) 148.17s 0.01s
repair_bytes (full file) 150s/1.25GB 2.91s/100MB
The user-visible analyzer scan dropped from minutes to sub-second on
1 GB-class files. Full-DataFrame analyze + auto_fix improvements are
more modest (~25%) because trim_whitespace and replace_null_sentinels
still need per-cell Python for the structural-shape checks, but the
hot path through these is now bounded by pandas' .map rather than a
manual for loop.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
DataTools
A bundle of Python data-cleaning tools for CSV and Excel files. Two scripts ship today; more are in build.
| # | Tool | What it does |
|---|---|---|
| 01 | Deduplicator | Find and remove duplicate rows with exact + fuzzy matching, smart normalization, and interactive review. |
| 02 | Text Cleaner | Trim whitespace, fold smart quotes, strip invisible / control characters, normalize Unicode, normalize line endings, optional case conversion. |
Deduplicator
Features
- Zero-config start — auto-detects encoding, delimiters, headers, and match columns
- Fuzzy matching — Jaro-Winkler, Levenshtein, and token set ratio algorithms
- 5 built-in normalizers — email (Gmail dot/plus), phone (E.164), name (titles/suffixes), address (USPS), string (whitespace/case)
- Merge mode — fill missing fields in the surviving row from removed duplicates
- 4 survivor rules — keep first, last, most complete, or most recent row per group
- Interactive review — inspect match groups with inline checkboxes and column dropdowns, cherry-pick values, preview surviving rows live
- Config profiles — save and reload your settings as JSON for repeatable runs
- Dual interface — full CLI for automation, Streamlit GUI for visual review
- Dry-run by default — preview what would change before writing anything
- Audit trail — every run produces a match groups report and timestamped log
Quick Start
Install
pip install -r requirements.txt
CLI
# Preview duplicates (dry run — no files written)
python -m src.cli customers.csv
# Remove duplicates and save the result
python -m src.cli customers.csv --apply
# Fuzzy-match names at 80% similarity, merge missing fields
python -m src.cli customers.csv --fuzzy name --threshold 80 --merge --apply
# Interactively review each match group
python -m src.cli customers.csv --review --apply
GUI
streamlit run src/gui/app.py
Upload a file, click Find Duplicates, review match groups side-by-side, then download the cleaned result.
CLI Usage Summary
python -m src.cli INPUT_FILE [OPTIONS]
Options:
--apply Write output files (default: preview only)
--output, -o PATH Output file path
--subset, -s COLS Columns to match on (comma-separated)
--key, -k COLS Strong-key columns for exact matching
--fuzzy COLS Columns to fuzzy-match
--algorithm, -a ALG levenshtein | jaro_winkler | token_set_ratio
--threshold, -t N Similarity threshold 0-100 (default: 85)
--normalize COL:TYPE Per-column normalizers (e.g., email:email,phone:phone)
--survivor RULE first | last | most-complete | most-recent
--merge Fill missing fields from removed duplicates
--review Interactively review each match group
--config PATH Load settings from a JSON config file
--save-config PATH Save current settings to JSON
--sheet NAME Excel sheet name or 0-based index
--encoding ENC Override auto-detected encoding
--header-row N 0-based header row index
--help Show full help
Sample Output
$ python -m src.cli samples/messy_sales.csv
Reading messy_sales.csv...
50 rows, 8 columns
Finding duplicates...
──────────────────────────────────────────────────
File: messy_sales.csv
Rows in: 50
Rows out: 28
Removed: 22
Groups: 22
──────────────────────────────────────────────────
Match groups:
Group 1: rows [1, 2] → keep row 1 (confidence: 100.0%, matched on: email)
Group 2: rows [3, 4] → keep row 3 (confidence: 92.3%, matched on: name, phone)
...
This was a preview. Add --apply to write the output files.
Output Files
When --apply is used, three files are produced:
| File | Contents |
|---|---|
{input}_deduplicated.csv |
Cleaned data with duplicates removed |
{input}_removed.csv |
Rows that were removed |
{input}_match_groups.csv |
Audit trail: group ID, confidence, matched columns, survivor flag |
Text Cleaner
Character-level hygiene for messy CSV / Excel input. Solves the dirty-data failure modes that silently break VLOOKUPs, dedup runs, and downstream imports:
- Trailing / leading whitespace and tabs in cells
- Non-breaking spaces (
U+00A0) hiding inside text where regular spaces should be - Smart quotes pasted from Word (
""''→""'') - Em / en dashes, ellipsis, other typographic Unicode
- Zero-width and bidi-mark characters (
U+200B,U+200C,U+200D, etc.) - BOMs from Excel "Save As CSV UTF-8"
- Mixed line endings (
\r\n, bare\r) inside multi-line cells - Control characters (
U+0000-U+001Fminus\t \n \r) - Optional Unicode NFC / NFKC normalization
- Optional per-column case conversion (UPPER / lower / smart Title / Sentence)
# Preview what would change (dry-run)
python -m src.cli_text_clean samples/messy_text.csv
# Apply the safe defaults
python -m src.cli_text_clean samples/messy_text.csv --apply
# Title-case the name column, upper-case the SKU column
python -m src.cli_text_clean products.csv --case title:name,upper:sku --apply
# Just trim and collapse — nothing fancy
python -m src.cli_text_clean messy.csv --preset minimal --apply
Three presets: minimal (trim + collapse only), excel-hygiene (default; everything safe ON), paranoid (adds lossy NFKC fold).
Outputs {input}_cleaned.csv plus a per-cell {input}_changes.csv audit (row, column, old, new, ops applied).
See docs/CLI-REFERENCE.md for every flag.
Review & Normalize gate
Every uploaded file passes through a CSV-normalization gate before any tool page sees it. The analyzer scans for ~15 issue types — whitespace pollution, NBSP / zero-width chars, mixed line endings, BOM artifacts, encoding misdetections, smart punctuation, dirty headers, null sentinels, mojibake, and more — and tags each finding by confidence (high / medium / low) and fix action (the algorithm in src/core/fixes.py that resolves it).
In the GUI, the Review & Normalize page renders one expandable card per finding with a decision control (Auto-fix / Skip / Customize), a live before-and-after preview, an encoding-override picker for misdetected codepages, and an Advanced output options block (encoding, delimiter, line terminator) for the download. Tool pages refuse to load until the gate passes.
See docs/USER-GUIDE.md §3.3 for the user-facing walkthrough and docs/TECHNICAL.md §10.2.1–10.2.4 for the developer-facing API.
Documentation
- User Guide — installation, GUI workflow, the Review & Normalize gate
- CLI Reference — every flag with examples and recipe sections
- Technical — architecture, gate internals, finding schema, fix registry
- Developer Guide — extending the bundle, adding fixes / detectors
Requirements
- Python 3.10+
- Dependencies: pandas, openpyxl, rapidfuzz, typer, phonenumbers, loguru, tqdm, charset-normalizer
License
Proprietary. All rights reserved.