feat: 3 new tools, format streaming, distribution-ready demo + landing pages
Tools shipped this batch (4 → 6 of 9 Ready):
04 Missing Value Handler src/core/missing.py + cli_missing.py + GUI
05 Column Mapper src/core/column_mapper.py + cli_column_map.py + GUI
09 Pipeline Runner src/core/pipeline.py + cli_pipeline.py + GUI
with soft tool-dependency graph (recommended,
not enforced) and JSON save/load for repeatable
weekly cleanups.
Format Standardizer reworked for 1 GB international files:
• Vectorised dispatch + LRU cache over phone/date/currency/boolean/email
• Per-row country / address columns drive parsing
• Audit cap (default 10 k rows, ~50 MB RAM)
• standardize_file(): chunked streaming entry point (~165 k rows/sec)
• currency_decimal="auto" for EU comma-decimal locales
• R$ / kr / zł multi-char currency prefixes
• cli_format.py with auto-stream above 100 MB inputs
Encoding detection arbiter + language-aware probe:
Closes the last 4 xfails (cp1250 / mac_iceland / shift_jis_2004 / lying-BOM)
via tied-confidence arbiter + Cyrillic / EE-Latin coverage probes.
Distribution-readiness assets:
• streamlit_app.py — Streamlit Community Cloud entry shim
• src/gui/app_demo.py — single-page demo, ?p=<persona> routing,
100-row cap + watermark, free-vs-paid boundary enforced at surface
• samples/demo/ — 3 niche datasets + pre-tuned pipeline JSONs
• landing/ — 4 static HTML pages (apex chooser + 3 niche),
shared CSS, deploy.py URL-substitution script,
auto-generated robots.txt + sitemap.xml + 404.html + favicon
• docs/PLAN.md, DEMO-PLAN.md, DEPLOYMENT.md, POST-LAUNCH.md, NEXT-STEPS.md
— full strategy + measurement + deployment + master checklist
Test counts:
before: 1,520 passed · 4 skipped · 17 xfailed
after: 1,729 passed · 0 skipped · 0 xfailed
Tier-1 corpora added:
• missing-corpus 3 use cases + 16 edge cases
• column-mapper-corpus 3 use cases + 5 edge cases
• format-cleaner intl 20-row 13-country stress fixture
Engine hardening flushed out by the corpora:
• interpolate guards against object-dtype columns
• mean/median skip all-NaN columns (silences numpy warning)
• fillna runs under future.no_silent_downcasting (silences pandas warning)
• mojibake test no longer skips when ftfy installed (monkeypatch path)
• drop-row threshold semantics: strict-greater (consistent across rows / cols)
• currency_decimal validator allow-set updated for "auto"
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
23
test-cases/column-mapper-corpus/README.md
Normal file
23
test-cases/column-mapper-corpus/README.md
Normal file
@@ -0,0 +1,23 @@
|
||||
# Column Mapper — corpus
|
||||
|
||||
Acceptance fixtures for `src/core/column_mapper.py`. Each `.csv` under
|
||||
`test_data/` is paired with assertions in
|
||||
`tests/test_column_mapper_corpus.py`.
|
||||
|
||||
## Use cases (target client profiles)
|
||||
|
||||
| File | Buyer profile | Tested behaviour |
|
||||
|------|---------------|------------------|
|
||||
| `uc01_crm_import.csv` + `schemas/uc01_crm_target.json` | Sales ops admin importing leads into Salesforce / HubSpot | Schema enforcement: rename via aliases, coerce types, drop extras, add `owner` default. |
|
||||
| `uc02_vendor_{a,b,c}.csv` + `schemas/uc02_canonical.json` | Operator unifying vendor exports | Multi-source unification: each vendor uses different headers; auto-inference resolves them all. |
|
||||
| `uc03_type_coercion.csv` + `schemas/uc03_types.json` | Analyst quick-fixing a mistyped CSV | Mixed-type coercion with documented per-column failure counts (bad rows survive as NaN). |
|
||||
|
||||
## Edge cases
|
||||
|
||||
| File | Stresses |
|
||||
|------|----------|
|
||||
| `ec01_duplicate_target.csv` | Mapping two source columns to the same target → InputValidationError. |
|
||||
| `ec02_unicode_columns.csv` | Non-ASCII column names (Japanese) survive rename and coerce. |
|
||||
| `ec03_whitespace_headers.csv` | Leading/trailing whitespace in headers still fuzzy-matches the schema. |
|
||||
| `ec04_no_match.csv` | No source column scores above threshold → empty mapping, fallback unmapped strategy fires. |
|
||||
| `ec05_required_missing.csv` | Required target field has no source column → InputValidationError unless `enforce_required=False`. |
|
||||
13
test-cases/column-mapper-corpus/schemas/uc01_crm_target.json
Normal file
13
test-cases/column-mapper-corpus/schemas/uc01_crm_target.json
Normal file
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"fields": [
|
||||
{"name": "first_name", "dtype": "string", "required": true, "aliases": ["First Name", "fname"]},
|
||||
{"name": "last_name", "dtype": "string", "required": true, "aliases": ["Last Name", "lname"]},
|
||||
{"name": "email", "dtype": "string", "required": true, "aliases": ["EmailAddr", "Email", "email_address"]},
|
||||
{"name": "phone", "dtype": "string", "aliases": ["Phone", "phone_number"]},
|
||||
{"name": "account_name", "dtype": "string", "aliases": ["Company", "Account"]},
|
||||
{"name": "annual_rev", "dtype": "integer", "aliases": ["Annual Revenue", "revenue"]},
|
||||
{"name": "lead_source", "dtype": "category","aliases": ["Lead Source", "source"]},
|
||||
{"name": "created_date", "dtype": "date", "aliases": ["Created", "create_date"]},
|
||||
{"name": "owner", "dtype": "string", "default": "unassigned"}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"fields": [
|
||||
{"name": "first_name", "dtype": "string", "required": true, "aliases": ["FirstName", "FName", "First Name"]},
|
||||
{"name": "last_name", "dtype": "string", "required": true, "aliases": ["LastName", "Surname", "Last Name"]},
|
||||
{"name": "email", "dtype": "string", "required": true, "aliases": ["Email", "E-mail", "email_addr", "EmailAddr"]},
|
||||
{"name": "phone", "dtype": "string", "aliases": ["Phone Number", "Tel", "phone_number"]},
|
||||
{"name": "country", "dtype": "string", "aliases": ["Country", "country_code", "Region"]}
|
||||
]
|
||||
}
|
||||
10
test-cases/column-mapper-corpus/schemas/uc03_types.json
Normal file
10
test-cases/column-mapper-corpus/schemas/uc03_types.json
Normal file
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"fields": [
|
||||
{"name": "id", "dtype": "integer", "required": true},
|
||||
{"name": "age", "dtype": "integer"},
|
||||
{"name": "active", "dtype": "boolean"},
|
||||
{"name": "joined", "dtype": "date"},
|
||||
{"name": "score", "dtype": "float"},
|
||||
{"name": "notes", "dtype": "string"}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
a,b,c
|
||||
1,2,3
|
||||
4,5,6
|
||||
|
@@ -0,0 +1,3 @@
|
||||
名前,Email,価格
|
||||
Alice,a@x.com,100
|
||||
Bob,b@x.com,200
|
||||
|
@@ -0,0 +1,3 @@
|
||||
First Name , Last Name ,EmailAddr
|
||||
Alice,Johnson,alice@x.com
|
||||
Bob,Smith,bob@x.com
|
||||
|
@@ -0,0 +1,3 @@
|
||||
xyz,abc,foobar
|
||||
1,2,3
|
||||
4,5,6
|
||||
|
@@ -0,0 +1,3 @@
|
||||
first_name,age
|
||||
Alice,30
|
||||
Bob,25
|
||||
|
@@ -0,0 +1,4 @@
|
||||
First Name,Last Name,EmailAddr,Phone,Company,Annual Revenue,Lead Source,Created
|
||||
Alice,Johnson,alice@acme.com,555-1234,Acme Corp,1500000,LinkedIn,2025-12-04
|
||||
Bob,Smith,bob@beta.com,555-5678,Beta LLC,250000,Webinar,2025-11-22
|
||||
Carlos,Garcia,carlos@gamma.io,555-9012,Gamma Inc,4200000,Referral,2025-10-30
|
||||
|
@@ -0,0 +1,3 @@
|
||||
FirstName,LastName,Email,Phone Number,Country
|
||||
Alice,Johnson,alice@vendor-a.com,555-1234,USA
|
||||
Bob,Smith,bob@vendor-a.com,555-5678,USA
|
||||
|
@@ -0,0 +1,3 @@
|
||||
first_name,surname,email_addr,phone,country_code
|
||||
Carlos,Garcia,carlos@vendor-b.com,555-9012,USA
|
||||
Diana,Lee,diana@vendor-b.com,555-7777,UK
|
||||
|
@@ -0,0 +1,3 @@
|
||||
FName,Surname,E-mail,Tel,Region
|
||||
Eve,Martinez,eve@vendor-c.com,555-9988,Bronx
|
||||
Frank,Brown,frank@vendor-c.com,555-1111,Queens
|
||||
|
@@ -0,0 +1,6 @@
|
||||
id,age,active,joined,score,notes
|
||||
1,30,true,2025-01-15,87.5,first
|
||||
2,25,false,2025-02-22,not_a_number,second
|
||||
3,not_a_number,yes,2025-03-08,76.0,third
|
||||
4,40,no,bad_date,91.2,fourth
|
||||
5,55,1,2025-05-01,82.0,fifth
|
||||
|
Reference in New Issue
Block a user