giteadmin/datatools-dev
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: 3 new tools, format streaming, distribution-ready demo + landing pages

Browse Source 
Tools shipped this batch (4 → 6 of 9 Ready):
  04 Missing Value Handler   src/core/missing.py + cli_missing.py + GUI
  05 Column Mapper           src/core/column_mapper.py + cli_column_map.py + GUI
  09 Pipeline Runner         src/core/pipeline.py + cli_pipeline.py + GUI
                             with soft tool-dependency graph (recommended,
                             not enforced) and JSON save/load for repeatable
                             weekly cleanups.

Format Standardizer reworked for 1 GB international files:
  • Vectorised dispatch + LRU cache over phone/date/currency/boolean/email
  • Per-row country / address columns drive parsing
  • Audit cap (default 10 k rows, ~50 MB RAM)
  • standardize_file(): chunked streaming entry point (~165 k rows/sec)
  • currency_decimal="auto" for EU comma-decimal locales
  • R$ / kr / zł multi-char currency prefixes
  • cli_format.py with auto-stream above 100 MB inputs

Encoding detection arbiter + language-aware probe:
  Closes the last 4 xfails (cp1250 / mac_iceland / shift_jis_2004 / lying-BOM)
  via tied-confidence arbiter + Cyrillic / EE-Latin coverage probes.

Distribution-readiness assets:
  • streamlit_app.py — Streamlit Community Cloud entry shim
  • src/gui/app_demo.py — single-page demo, ?p=<persona> routing,
    100-row cap + watermark, free-vs-paid boundary enforced at surface
  • samples/demo/ — 3 niche datasets + pre-tuned pipeline JSONs
  • landing/ — 4 static HTML pages (apex chooser + 3 niche),
    shared CSS, deploy.py URL-substitution script,
    auto-generated robots.txt + sitemap.xml + 404.html + favicon
  • docs/PLAN.md, DEMO-PLAN.md, DEPLOYMENT.md, POST-LAUNCH.md, NEXT-STEPS.md
    — full strategy + measurement + deployment + master checklist

Test counts:
  before: 1,520 passed · 4 skipped · 17 xfailed
  after:  1,729 passed · 0 skipped · 0  xfailed

Tier-1 corpora added:
  • missing-corpus           3 use cases + 16 edge cases
  • column-mapper-corpus     3 use cases + 5 edge cases
  • format-cleaner intl      20-row 13-country stress fixture

Engine hardening flushed out by the corpora:
  • interpolate guards against object-dtype columns
  • mean/median skip all-NaN columns (silences numpy warning)
  • fillna runs under future.no_silent_downcasting (silences pandas warning)
  • mojibake test no longer skips when ftfy installed (monkeypatch path)
  • drop-row threshold semantics: strict-greater (consistent across rows / cols)
  • currency_decimal validator allow-set updated for "auto"

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Michael
2026-05-01 22:31:26 +00:00
parent d18b95880d
commit 966af8ef94
89 changed files with 12039 additions and 284 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
test-cases/column-mapper-corpus/test_data/ec01_duplicate_target.csv Normal file
View File

@@ -0,0 +1,3 @@
a,b,c
1,2,3
4,5,6
1 a b c
2 1 2 3
3 4 5 6

3
test-cases/column-mapper-corpus/test_data/ec02_unicode_columns.csv Normal file
View File

@@ -0,0 +1,3 @@
名前,Email,価格
Alice,a@x.com,100
Bob,b@x.com,200
1 名前 Email 価格
2 Alice a@x.com 100
3 Bob b@x.com 200

3
test-cases/column-mapper-corpus/test_data/ec03_whitespace_headers.csv Normal file
View File

@@ -0,0 +1,3 @@
First Name , Last Name ,EmailAddr
Alice,Johnson,alice@x.com
Bob,Smith,bob@x.com
1 First Name Last Name EmailAddr
2 Alice Johnson alice@x.com
3 Bob Smith bob@x.com

3
test-cases/column-mapper-corpus/test_data/ec04_no_match.csv Normal file
View File

@@ -0,0 +1,3 @@
xyz,abc,foobar
1,2,3
4,5,6
1 xyz abc foobar
2 1 2 3
3 4 5 6

3
test-cases/column-mapper-corpus/test_data/ec05_required_missing.csv Normal file
View File

@@ -0,0 +1,3 @@
first_name,age
Alice,30
Bob,25
1 first_name age
2 Alice 30
3 Bob 25

4
test-cases/column-mapper-corpus/test_data/uc01_crm_import.csv Normal file
View File

@@ -0,0 +1,4 @@
First Name,Last Name,EmailAddr,Phone,Company,Annual Revenue,Lead Source,Created
Alice,Johnson,alice@acme.com,555-1234,Acme Corp,1500000,LinkedIn,2025-12-04
Bob,Smith,bob@beta.com,555-5678,Beta LLC,250000,Webinar,2025-11-22
Carlos,Garcia,carlos@gamma.io,555-9012,Gamma Inc,4200000,Referral,2025-10-30
1 First Name Last Name EmailAddr Phone Company Annual Revenue Lead Source Created
2 Alice Johnson alice@acme.com 555-1234 Acme Corp 1500000 LinkedIn 2025-12-04
3 Bob Smith bob@beta.com 555-5678 Beta LLC 250000 Webinar 2025-11-22
4 Carlos Garcia carlos@gamma.io 555-9012 Gamma Inc 4200000 Referral 2025-10-30

3
test-cases/column-mapper-corpus/test_data/uc02_vendor_a.csv Normal file
View File

@@ -0,0 +1,3 @@
FirstName,LastName,Email,Phone Number,Country
Alice,Johnson,alice@vendor-a.com,555-1234,USA
Bob,Smith,bob@vendor-a.com,555-5678,USA
1 FirstName LastName Email Phone Number Country
2 Alice Johnson alice@vendor-a.com 555-1234 USA
3 Bob Smith bob@vendor-a.com 555-5678 USA

3
test-cases/column-mapper-corpus/test_data/uc02_vendor_b.csv Normal file
View File

@@ -0,0 +1,3 @@
first_name,surname,email_addr,phone,country_code
Carlos,Garcia,carlos@vendor-b.com,555-9012,USA
Diana,Lee,diana@vendor-b.com,555-7777,UK
1 first_name surname email_addr phone country_code
2 Carlos Garcia carlos@vendor-b.com 555-9012 USA
3 Diana Lee diana@vendor-b.com 555-7777 UK

3
test-cases/column-mapper-corpus/test_data/uc02_vendor_c.csv Normal file
View File

@@ -0,0 +1,3 @@
FName,Surname,E-mail,Tel,Region
Eve,Martinez,eve@vendor-c.com,555-9988,Bronx
Frank,Brown,frank@vendor-c.com,555-1111,Queens
1 FName Surname E-mail Tel Region
2 Eve Martinez eve@vendor-c.com 555-9988 Bronx
3 Frank Brown frank@vendor-c.com 555-1111 Queens

6
test-cases/column-mapper-corpus/test_data/uc03_type_coercion.csv Normal file
View File

@@ -0,0 +1,6 @@
id,age,active,joined,score,notes
1,30,true,2025-01-15,87.5,first
2,25,false,2025-02-22,not_a_number,second
3,not_a_number,yes,2025-03-08,76.0,third
4,40,no,bad_date,91.2,fourth
5,55,1,2025-05-01,82.0,fifth
1 id age active joined score notes
2 1 30 true 2025-01-15 87.5 first
3 2 25 false 2025-02-22 not_a_number second
4 3 not_a_number yes 2025-03-08 76.0 third
5 4 40 no bad_date 91.2 fourth
6 5 55 1 2025-05-01 82.0 fifth