Michael
966af8ef94
Tools shipped this batch (4 → 6 of 9 Ready):
04 Missing Value Handler src/core/missing.py + cli_missing.py + GUI
05 Column Mapper src/core/column_mapper.py + cli_column_map.py + GUI
09 Pipeline Runner src/core/pipeline.py + cli_pipeline.py + GUI
with soft tool-dependency graph (recommended,
not enforced) and JSON save/load for repeatable
weekly cleanups.
Format Standardizer reworked for 1 GB international files:
• Vectorised dispatch + LRU cache over phone/date/currency/boolean/email
• Per-row country / address columns drive parsing
• Audit cap (default 10 k rows, ~50 MB RAM)
• standardize_file(): chunked streaming entry point (~165 k rows/sec)
• currency_decimal="auto" for EU comma-decimal locales
• R$ / kr / zł multi-char currency prefixes
• cli_format.py with auto-stream above 100 MB inputs
Encoding detection arbiter + language-aware probe:
Closes the last 4 xfails (cp1250 / mac_iceland / shift_jis_2004 / lying-BOM)
via tied-confidence arbiter + Cyrillic / EE-Latin coverage probes.
Distribution-readiness assets:
• streamlit_app.py — Streamlit Community Cloud entry shim
• src/gui/app_demo.py — single-page demo, ?p=<persona> routing,
100-row cap + watermark, free-vs-paid boundary enforced at surface
• samples/demo/ — 3 niche datasets + pre-tuned pipeline JSONs
• landing/ — 4 static HTML pages (apex chooser + 3 niche),
shared CSS, deploy.py URL-substitution script,
auto-generated robots.txt + sitemap.xml + 404.html + favicon
• docs/PLAN.md, DEMO-PLAN.md, DEPLOYMENT.md, POST-LAUNCH.md, NEXT-STEPS.md
— full strategy + measurement + deployment + master checklist
Test counts:
before: 1,520 passed · 4 skipped · 17 xfailed
after: 1,729 passed · 0 skipped · 0 xfailed
Tier-1 corpora added:
• missing-corpus 3 use cases + 16 edge cases
• column-mapper-corpus 3 use cases + 5 edge cases
• format-cleaner intl 20-row 13-country stress fixture
Engine hardening flushed out by the corpora:
• interpolate guards against object-dtype columns
• mean/median skip all-NaN columns (silences numpy warning)
• fillna runs under future.no_silent_downcasting (silences pandas warning)
• mojibake test no longer skips when ftfy installed (monkeypatch path)
• drop-row threshold semantics: strict-greater (consistent across rows / cols)
• currency_decimal validator allow-set updated for "auto"
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
76 lines
3.0 KiB
Markdown
76 lines
3.0 KiB
Markdown
# DataTools
|
|
|
|
Local CSV / Excel cleaning. CLI + browser GUI, no cloud, no install ceremony.
|
|
|
|
## Tools
|
|
|
|
| # | Tool | Status |
|
|
|---|------|--------|
|
|
| 01 | **Deduplicator** — exact + fuzzy match, 5 normalizers, survivor rules, audit | Ready |
|
|
| 02 | **Text Cleaner** — whitespace, smart chars, BOM, line endings, case ops | Ready |
|
|
| 03 | **Format Standardizer** — dates, phones, emails, addresses, names, currencies, booleans | Ready |
|
|
| 04 | **Missing Value Handler** — disguised-null detection, profile, mean/median/mode/ffill/bfill/interpolate, drop strategies | Ready |
|
|
| 05 | **Column Mapper** — fuzzy auto-rename, target schema with type coercion, required fields with defaults, drop/reorder | Ready |
|
|
| 06 | Outlier Detector | Coming Soon |
|
|
| 07 | Multi-File Merger | Coming Soon |
|
|
| 08 | Validator & Reporter | Coming Soon |
|
|
| 09 | **Pipeline Runner** — chain tools with recommended (not forced) order, save/load JSON, automate weekly cleanups | Ready |
|
|
|
|
## Install
|
|
|
|
```bash
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
Python 3.10+ required.
|
|
|
|
## Run
|
|
|
|
**GUI** (recommended):
|
|
```bash
|
|
streamlit run src/gui/app.py
|
|
```
|
|
|
|
**CLI** — seven entry points:
|
|
```bash
|
|
python -m src.cli customers.csv [--apply] # dedup
|
|
python -m src.cli_text_clean messy.csv [--apply] # text clean
|
|
python -m src.cli_format intl.csv [--apply] # format standardize (auto-streams >100 MB)
|
|
python -m src.cli_missing holes.csv [--apply] # missing values
|
|
python -m src.cli_column_map vendor.csv [--apply] # column mapper
|
|
python -m src.cli_pipeline any_file.csv [--apply] # chain tools end-to-end
|
|
python -m src.cli_analyze any_file.csv [--json] # scan only
|
|
```
|
|
|
|
Every CLI runs preview-only by default; add `--apply` to write output.
|
|
|
|
## Review & Normalize gate
|
|
|
|
Every uploaded file passes through a CSV-normalization gate before any tool sees it. The analyzer flags ~15 issue types (whitespace, NBSP / zero-width chars, BOM, encoding, smart punct, dirty headers, null sentinels, mojibake, …) tagged by **confidence** (high / medium / low) and **fix action**. The GUI shows each finding with Auto-fix / Skip / Customize, a live before/after preview, and an encoding-override picker. Tool pages refuse to load until the gate passes.
|
|
|
|
## Output
|
|
|
|
Every run writes:
|
|
|
|
- `{input}_<tool>.csv` — the cleaned data
|
|
- `{input}_changes.csv` (text cleaner) or `{input}_match_groups.csv` (dedup) — audit trail
|
|
- `logs/<tool>_YYYYMMDD_HHMMSS.log` — debug-level run log
|
|
|
|
Original input file is never modified.
|
|
|
|
## Docs
|
|
|
|
- [User Guide](docs/USER-GUIDE.md) — install, GUI workflow, gate
|
|
- [CLI Reference](docs/CLI-REFERENCE.md) — every flag with recipes
|
|
- [Requirements](docs/REQUIREMENTS.md) — file sizes, encodings, detectors, perf targets
|
|
- [Technical](docs/TECHNICAL.md) — architecture, gate internals, fix registry
|
|
- [Developer Guide](docs/DEVELOPER.md) — adding fixes / detectors / standardizers
|
|
|
|
## Dependencies
|
|
|
|
`pandas`, `openpyxl`, `rapidfuzz`, `phonenumbers`, `typer`, `loguru`, `charset-normalizer`, `streamlit`. Optional: `ftfy` for mojibake repair.
|
|
|
|
## License
|
|
|
|
Proprietary.
|