Files

Michael 966af8ef94 feat: 3 new tools, format streaming, distribution-ready demo + landing pages

Tools shipped this batch (4 → 6 of 9 Ready):
  04 Missing Value Handler   src/core/missing.py + cli_missing.py + GUI
  05 Column Mapper           src/core/column_mapper.py + cli_column_map.py + GUI
  09 Pipeline Runner         src/core/pipeline.py + cli_pipeline.py + GUI
                             with soft tool-dependency graph (recommended,
                             not enforced) and JSON save/load for repeatable
                             weekly cleanups.

Format Standardizer reworked for 1 GB international files:
  • Vectorised dispatch + LRU cache over phone/date/currency/boolean/email
  • Per-row country / address columns drive parsing
  • Audit cap (default 10 k rows, ~50 MB RAM)
  • standardize_file(): chunked streaming entry point (~165 k rows/sec)
  • currency_decimal="auto" for EU comma-decimal locales
  • R$ / kr / zł multi-char currency prefixes
  • cli_format.py with auto-stream above 100 MB inputs

Encoding detection arbiter + language-aware probe:
  Closes the last 4 xfails (cp1250 / mac_iceland / shift_jis_2004 / lying-BOM)
  via tied-confidence arbiter + Cyrillic / EE-Latin coverage probes.

Distribution-readiness assets:
  • streamlit_app.py — Streamlit Community Cloud entry shim
  • src/gui/app_demo.py — single-page demo, ?p=<persona> routing,
    100-row cap + watermark, free-vs-paid boundary enforced at surface
  • samples/demo/ — 3 niche datasets + pre-tuned pipeline JSONs
  • landing/ — 4 static HTML pages (apex chooser + 3 niche),
    shared CSS, deploy.py URL-substitution script,
    auto-generated robots.txt + sitemap.xml + 404.html + favicon
  • docs/PLAN.md, DEMO-PLAN.md, DEPLOYMENT.md, POST-LAUNCH.md, NEXT-STEPS.md
    — full strategy + measurement + deployment + master checklist

Test counts:
  before: 1,520 passed · 4 skipped · 17 xfailed
  after:  1,729 passed · 0 skipped · 0  xfailed

Tier-1 corpora added:
  • missing-corpus           3 use cases + 16 edge cases
  • column-mapper-corpus     3 use cases + 5 edge cases
  • format-cleaner intl      20-row 13-country stress fixture

Engine hardening flushed out by the corpora:
  • interpolate guards against object-dtype columns
  • mean/median skip all-NaN columns (silences numpy warning)
  • fillna runs under future.no_silent_downcasting (silences pandas warning)
  • mojibake test no longer skips when ftfy installed (monkeypatch path)
  • drop-row threshold semantics: strict-greater (consistent across rows / cols)
  • currency_decimal validator allow-set updated for "auto"

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-01 22:31:26 +00:00

2.2 KiB

Raw Blame History

Missing Value Handler — corpus

Acceptance fixtures for src/core/missing.py. Each .csv under test_data/ is paired with assertions in tests/test_missing_corpus.py. Add a new case by dropping a CSV here and adding a parametrize entry to the runner.

Use cases (target client profiles)

File	Buyer profile	Strategy under test
`uc01_shopify_export.csv`	SMB / Shopify operator	`detect-only`
`uc02_marketing_audience.csv`	Marketing / RevOps analyst	`safe-fill`
`uc03_consultant_intake.csv`	Analyst / consultant	`drop-incomplete` + threshold

Edge cases

File	What it stresses
`ec01_all_nan_column.csv`	column 100 % missing — fill must skip, drop_col must catch at threshold
`ec02_no_missing.csv`	clean file — must be a no-op
`ec03_zero_is_not_missing.csv`	numeric `0`, boolean `false`, `"0"` must NOT be treated as missing
`ec04_excel_errors.csv`	`#N/A`, `#NULL!`, `#VALUE!` Excel error sentinels
`ec05_unicode_whitespace.csv`	NBSP, tab-only, ideographic-space cells treated as whitespace
`ec06_mixed_dtypes.csv`	mixed numeric/string in same column — graceful degrade to mode
`ec07_real_data_with_padding.csv`	leading/trailing whitespace around real data must NOT be dropped
`ec08_single_row.csv`	one-row file — every operation must still work
`ec09_single_column.csv`	one-column file with header-only line + sentinels
`ec10_all_sentinel_variants.csv`	every `DEFAULT_SENTINELS` entry exercised in one file
`ec11_constant_per_column.csv`	`column_fill_values` differs per column
`ec12_drop_threshold_boundary.csv`	boundary values for `row_drop_threshold` (0.5, 0.99, 1.0)
`ec13_ffill_leading_nan.csv`	leading-NaN run survives ffill (no fabrication)
`ec14_interpolate_fallback.csv`	numeric-only strategy on string column triggers fallback
`ec15_headers_only.csv`	empty body — must not crash
`ec16_idempotent_apply.csv`	running `handle_missing` twice yields the same DataFrame

2.2 KiB Raw Blame History

Missing Value Handler — corpus

Use cases (target client profiles)

Edge cases

2.2 KiB

Raw Blame History