Michael
966af8ef94
Tools shipped this batch (4 → 6 of 9 Ready):
04 Missing Value Handler src/core/missing.py + cli_missing.py + GUI
05 Column Mapper src/core/column_mapper.py + cli_column_map.py + GUI
09 Pipeline Runner src/core/pipeline.py + cli_pipeline.py + GUI
with soft tool-dependency graph (recommended,
not enforced) and JSON save/load for repeatable
weekly cleanups.
Format Standardizer reworked for 1 GB international files:
• Vectorised dispatch + LRU cache over phone/date/currency/boolean/email
• Per-row country / address columns drive parsing
• Audit cap (default 10 k rows, ~50 MB RAM)
• standardize_file(): chunked streaming entry point (~165 k rows/sec)
• currency_decimal="auto" for EU comma-decimal locales
• R$ / kr / zł multi-char currency prefixes
• cli_format.py with auto-stream above 100 MB inputs
Encoding detection arbiter + language-aware probe:
Closes the last 4 xfails (cp1250 / mac_iceland / shift_jis_2004 / lying-BOM)
via tied-confidence arbiter + Cyrillic / EE-Latin coverage probes.
Distribution-readiness assets:
• streamlit_app.py — Streamlit Community Cloud entry shim
• src/gui/app_demo.py — single-page demo, ?p=<persona> routing,
100-row cap + watermark, free-vs-paid boundary enforced at surface
• samples/demo/ — 3 niche datasets + pre-tuned pipeline JSONs
• landing/ — 4 static HTML pages (apex chooser + 3 niche),
shared CSS, deploy.py URL-substitution script,
auto-generated robots.txt + sitemap.xml + 404.html + favicon
• docs/PLAN.md, DEMO-PLAN.md, DEPLOYMENT.md, POST-LAUNCH.md, NEXT-STEPS.md
— full strategy + measurement + deployment + master checklist
Test counts:
before: 1,520 passed · 4 skipped · 17 xfailed
after: 1,729 passed · 0 skipped · 0 xfailed
Tier-1 corpora added:
• missing-corpus 3 use cases + 16 edge cases
• column-mapper-corpus 3 use cases + 5 edge cases
• format-cleaner intl 20-row 13-country stress fixture
Engine hardening flushed out by the corpora:
• interpolate guards against object-dtype columns
• mean/median skip all-NaN columns (silences numpy warning)
• fillna runs under future.no_silent_downcasting (silences pandas warning)
• mojibake test no longer skips when ftfy installed (monkeypatch path)
• drop-row threshold semantics: strict-greater (consistent across rows / cols)
• currency_decimal validator allow-set updated for "auto"
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
75 lines
2.2 KiB
JSON
75 lines
2.2 KiB
JSON
{
|
|
"steps": [
|
|
{
|
|
"tool": "text_clean",
|
|
"options": {},
|
|
"enabled": true,
|
|
"name": "1. Clean text (whitespace + smart quotes from copy-paste)"
|
|
},
|
|
{
|
|
"tool": "format_standardize",
|
|
"options": {
|
|
"column_types": {
|
|
"First Name": "name",
|
|
"Last Name": "name",
|
|
"Company": "name",
|
|
"Email": "email",
|
|
"Phone": "phone"
|
|
},
|
|
"phone_country_column": "Country",
|
|
"phone_format": "E164",
|
|
"email_gmail_canonical": true
|
|
},
|
|
"enabled": true,
|
|
"name": "2. E.164 phones (per-row country) · canonical emails · name casing"
|
|
},
|
|
{
|
|
"tool": "missing",
|
|
"options": {
|
|
"strategy": "none",
|
|
"standardize_sentinels": true,
|
|
"sentinels": ["N/A", "n/a", "—", "?", "(unknown)", "unknown", "(blank)", "(none)", "TBD", "#N/A"]
|
|
},
|
|
"enabled": true,
|
|
"name": "3. Standardize sentinels across vendor exports"
|
|
},
|
|
{
|
|
"tool": "column_map",
|
|
"options": {
|
|
"schema": {
|
|
"fields": [
|
|
{"name": "Lead ID", "dtype": "string", "required": true},
|
|
{"name": "First Name", "dtype": "string"},
|
|
{"name": "Last Name", "dtype": "string"},
|
|
{"name": "Company", "dtype": "string"},
|
|
{"name": "Title", "dtype": "string"},
|
|
{"name": "Email", "dtype": "string"},
|
|
{"name": "Phone", "dtype": "string"},
|
|
{"name": "Country", "dtype": "string"},
|
|
{"name": "Source", "dtype": "string"},
|
|
{"name": "Score", "dtype": "integer"},
|
|
{"name": "Last Activity", "dtype": "date"},
|
|
{"name": "Tags", "dtype": "string"}
|
|
]
|
|
},
|
|
"auto_infer": true,
|
|
"unmapped": "keep",
|
|
"coerce_types": true,
|
|
"reorder_to_schema": true,
|
|
"enforce_required": false
|
|
},
|
|
"enabled": true,
|
|
"name": "4. Coerce types · reorder to canonical schema"
|
|
},
|
|
{
|
|
"tool": "dedup",
|
|
"options": {
|
|
"survivor_rule": "most_complete",
|
|
"merge": true
|
|
},
|
|
"enabled": true,
|
|
"name": "5. Dedup leads across HubSpot / LinkedIn / Manual Scrape (fuzzy + merge)"
|
|
}
|
|
]
|
|
}
|