Michael
966af8ef94
Tools shipped this batch (4 → 6 of 9 Ready):
04 Missing Value Handler src/core/missing.py + cli_missing.py + GUI
05 Column Mapper src/core/column_mapper.py + cli_column_map.py + GUI
09 Pipeline Runner src/core/pipeline.py + cli_pipeline.py + GUI
with soft tool-dependency graph (recommended,
not enforced) and JSON save/load for repeatable
weekly cleanups.
Format Standardizer reworked for 1 GB international files:
• Vectorised dispatch + LRU cache over phone/date/currency/boolean/email
• Per-row country / address columns drive parsing
• Audit cap (default 10 k rows, ~50 MB RAM)
• standardize_file(): chunked streaming entry point (~165 k rows/sec)
• currency_decimal="auto" for EU comma-decimal locales
• R$ / kr / zł multi-char currency prefixes
• cli_format.py with auto-stream above 100 MB inputs
Encoding detection arbiter + language-aware probe:
Closes the last 4 xfails (cp1250 / mac_iceland / shift_jis_2004 / lying-BOM)
via tied-confidence arbiter + Cyrillic / EE-Latin coverage probes.
Distribution-readiness assets:
• streamlit_app.py — Streamlit Community Cloud entry shim
• src/gui/app_demo.py — single-page demo, ?p=<persona> routing,
100-row cap + watermark, free-vs-paid boundary enforced at surface
• samples/demo/ — 3 niche datasets + pre-tuned pipeline JSONs
• landing/ — 4 static HTML pages (apex chooser + 3 niche),
shared CSS, deploy.py URL-substitution script,
auto-generated robots.txt + sitemap.xml + 404.html + favicon
• docs/PLAN.md, DEMO-PLAN.md, DEPLOYMENT.md, POST-LAUNCH.md, NEXT-STEPS.md
— full strategy + measurement + deployment + master checklist
Test counts:
before: 1,520 passed · 4 skipped · 17 xfailed
after: 1,729 passed · 0 skipped · 0 xfailed
Tier-1 corpora added:
• missing-corpus 3 use cases + 16 edge cases
• column-mapper-corpus 3 use cases + 5 edge cases
• format-cleaner intl 20-row 13-country stress fixture
Engine hardening flushed out by the corpora:
• interpolate guards against object-dtype columns
• mean/median skip all-NaN columns (silences numpy warning)
• fillna runs under future.no_silent_downcasting (silences pandas warning)
• mojibake test no longer skips when ftfy installed (monkeypatch path)
• drop-row threshold semantics: strict-greater (consistent across rows / cols)
• currency_decimal validator allow-set updated for "auto"
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
36 lines
2.2 KiB
Markdown
36 lines
2.2 KiB
Markdown
# Missing Value Handler — corpus
|
|
|
|
Acceptance fixtures for `src/core/missing.py`. Each `.csv` under
|
|
`test_data/` is paired with assertions in `tests/test_missing_corpus.py`.
|
|
Add a new case by dropping a CSV here and adding a parametrize entry to
|
|
the runner.
|
|
|
|
## Use cases (target client profiles)
|
|
|
|
| File | Buyer profile | Strategy under test |
|
|
|------|---------------|---------------------|
|
|
| `uc01_shopify_export.csv` | SMB / Shopify operator | `detect-only` |
|
|
| `uc02_marketing_audience.csv` | Marketing / RevOps analyst| `safe-fill` |
|
|
| `uc03_consultant_intake.csv` | Analyst / consultant | `drop-incomplete` + threshold |
|
|
|
|
## Edge cases
|
|
|
|
| File | What it stresses |
|
|
|------|------------------|
|
|
| `ec01_all_nan_column.csv` | column 100 % missing — fill must skip, drop_col must catch at threshold |
|
|
| `ec02_no_missing.csv` | clean file — must be a no-op |
|
|
| `ec03_zero_is_not_missing.csv` | numeric `0`, boolean `false`, `"0"` must NOT be treated as missing |
|
|
| `ec04_excel_errors.csv` | `#N/A`, `#NULL!`, `#VALUE!` Excel error sentinels |
|
|
| `ec05_unicode_whitespace.csv` | NBSP, tab-only, ideographic-space cells treated as whitespace |
|
|
| `ec06_mixed_dtypes.csv` | mixed numeric/string in same column — graceful degrade to mode |
|
|
| `ec07_real_data_with_padding.csv` | leading/trailing whitespace around real data must NOT be dropped |
|
|
| `ec08_single_row.csv` | one-row file — every operation must still work |
|
|
| `ec09_single_column.csv` | one-column file with header-only line + sentinels |
|
|
| `ec10_all_sentinel_variants.csv` | every `DEFAULT_SENTINELS` entry exercised in one file |
|
|
| `ec11_constant_per_column.csv` | `column_fill_values` differs per column |
|
|
| `ec12_drop_threshold_boundary.csv`| boundary values for `row_drop_threshold` (0.5, 0.99, 1.0) |
|
|
| `ec13_ffill_leading_nan.csv` | leading-NaN run survives ffill (no fabrication) |
|
|
| `ec14_interpolate_fallback.csv` | numeric-only strategy on string column triggers fallback |
|
|
| `ec15_headers_only.csv` | empty body — must not crash |
|
|
| `ec16_idempotent_apply.csv` | running `handle_missing` twice yields the same DataFrame |
|