giteadmin/datatools-dev
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: 3 new tools, format streaming, distribution-ready demo + landing pages

Browse Source 
Tools shipped this batch (4 → 6 of 9 Ready):
  04 Missing Value Handler   src/core/missing.py + cli_missing.py + GUI
  05 Column Mapper           src/core/column_mapper.py + cli_column_map.py + GUI
  09 Pipeline Runner         src/core/pipeline.py + cli_pipeline.py + GUI
                             with soft tool-dependency graph (recommended,
                             not enforced) and JSON save/load for repeatable
                             weekly cleanups.

Format Standardizer reworked for 1 GB international files:
  • Vectorised dispatch + LRU cache over phone/date/currency/boolean/email
  • Per-row country / address columns drive parsing
  • Audit cap (default 10 k rows, ~50 MB RAM)
  • standardize_file(): chunked streaming entry point (~165 k rows/sec)
  • currency_decimal="auto" for EU comma-decimal locales
  • R$ / kr / zł multi-char currency prefixes
  • cli_format.py with auto-stream above 100 MB inputs

Encoding detection arbiter + language-aware probe:
  Closes the last 4 xfails (cp1250 / mac_iceland / shift_jis_2004 / lying-BOM)
  via tied-confidence arbiter + Cyrillic / EE-Latin coverage probes.

Distribution-readiness assets:
  • streamlit_app.py — Streamlit Community Cloud entry shim
  • src/gui/app_demo.py — single-page demo, ?p=<persona> routing,
    100-row cap + watermark, free-vs-paid boundary enforced at surface
  • samples/demo/ — 3 niche datasets + pre-tuned pipeline JSONs
  • landing/ — 4 static HTML pages (apex chooser + 3 niche),
    shared CSS, deploy.py URL-substitution script,
    auto-generated robots.txt + sitemap.xml + 404.html + favicon
  • docs/PLAN.md, DEMO-PLAN.md, DEPLOYMENT.md, POST-LAUNCH.md, NEXT-STEPS.md
    — full strategy + measurement + deployment + master checklist

Test counts:
  before: 1,520 passed · 4 skipped · 17 xfailed
  after:  1,729 passed · 0 skipped · 0  xfailed

Tier-1 corpora added:
  • missing-corpus           3 use cases + 16 edge cases
  • column-mapper-corpus     3 use cases + 5 edge cases
  • format-cleaner intl      20-row 13-country stress fixture

Engine hardening flushed out by the corpora:
  • interpolate guards against object-dtype columns
  • mean/median skip all-NaN columns (silences numpy warning)
  • fillna runs under future.no_silent_downcasting (silences pandas warning)
  • mojibake test no longer skips when ftfy installed (monkeypatch path)
  • drop-row threshold semantics: strict-greater (consistent across rows / cols)
  • currency_decimal validator allow-set updated for "auto"

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Michael
2026-05-01 22:31:26 +00:00
parent d18b95880d
commit 966af8ef94
89 changed files with 12039 additions and 284 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
test-cases/format-cleaner-corpus/international/intl_phones_addresses.csv Normal file
View File

@@ -0,0 +1,21 @@
customer_id,name,phone,country,address,price
INT-001,Alice Johnson,(415) 555-1234,US,"1 Apple Park Way, Cupertino CA 95014",$1499.99
INT-002,Boris Petrov,+7 495 123 4567,RU,"Ulitsa Tverskaya 13, Moscow 125009",₽89500
INT-003,carlos garcia,+34 91 411 1111,ES,"Calle Gran Via 28, Madrid 28013","€1.299,00"
INT-004,JOHN BROWN,020 7946 0958,GB,"10 Downing Street, London SW1A 2AA","£950.00"
INT-005,marie dubois,01 42 86 82 00,FR,"Avenue des Champs-Elysees 100, Paris 75008","€2.499,50"
INT-006,Yuki Tanaka,03-3210-7000,JP,"Marunouchi 2-7-3, Chiyoda-ku Tokyo 100-0005",¥150000
INT-007,Anna Schmidt,030 12345678,DE,"Unter den Linden 5, Berlin 10117","€899,99"
INT-008,giovanni rossi,+39 06 6982,IT,"Via del Corso 320, Roma 00186","€1.450,00"
INT-009,Mei Wang,+86 10 1234 5678,CN,"东长安街 1号, 北京 100006",¥10000
INT-010,Priya Sharma,+91 11 2345 6789,IN,"Connaught Place, New Delhi 110001",₹85000
INT-011,Ahmed Hassan,+20 2 2735 0000,EG,"Tahrir Square, Cairo 11511",E£3500
INT-012,emily smith,+61 2 9374 4000,AU,"Sydney Opera House, Sydney NSW 2000","$2,199.00"
INT-013,Joao Silva,11 3071 0000,BR,"Avenida Paulista 1000, Sao Paulo 01310","R$ 1.299,90"
INT-014,Sofia Lopez,+52 55 5555 0000,MX,"Paseo de la Reforma 222, Ciudad de Mexico 06600","$1,500 MXN"
INT-015,Min-jun Kim,+82 2 2287 0114,KR,"Seoul Plaza, Seoul 04518",₩1500000
INT-016,Mehmet Yilmaz,+90 212 252 0000,TR,"Sultanahmet, Istanbul 34122","₺1.250"
INT-017,david cohen,+972 3 6957 0000,IL,"Dizengoff 50, Tel Aviv 6433222",₪450
INT-018,Hanna Kowalska,+48 22 658 4500,PL,"Marszalkowska 1, Warszawa 00-624","zł 350,00"
INT-019,Lars Nielsen,+45 33 12 88 88,DK,"Vesterbrogade 1, Copenhagen 1620","kr 950"
INT-020,Sven Eriksson,+46 8 506 600 00,SE,"Drottninggatan 1, Stockholm 11151","kr 1.250,50"
1 customer_id name phone country address price
2 INT-001 Alice Johnson (415) 555-1234 US 1 Apple Park Way, Cupertino CA 95014 $1499.99
3 INT-002 Boris Petrov +7 495 123 4567 RU Ulitsa Tverskaya 13, Moscow 125009 ₽89500
4 INT-003 carlos garcia +34 91 411 1111 ES Calle Gran Via 28, Madrid 28013 €1.299,00
5 INT-004 JOHN BROWN 020 7946 0958 GB 10 Downing Street, London SW1A 2AA £950.00
6 INT-005 marie dubois 01 42 86 82 00 FR Avenue des Champs-Elysees 100, Paris 75008 €2.499,50
7 INT-006 Yuki Tanaka 03-3210-7000 JP Marunouchi 2-7-3, Chiyoda-ku Tokyo 100-0005 ¥150000
8 INT-007 Anna Schmidt 030 12345678 DE Unter den Linden 5, Berlin 10117 €899,99
9 INT-008 giovanni rossi +39 06 6982 IT Via del Corso 320, Roma 00186 €1.450,00
10 INT-009 Mei Wang +86 10 1234 5678 CN 东长安街 1号, 北京 100006 ¥10000
11 INT-010 Priya Sharma +91 11 2345 6789 IN Connaught Place, New Delhi 110001 ₹85000
12 INT-011 Ahmed Hassan +20 2 2735 0000 EG Tahrir Square, Cairo 11511 E£3500
13 INT-012 emily smith +61 2 9374 4000 AU Sydney Opera House, Sydney NSW 2000 $2,199.00
14 INT-013 Joao Silva 11 3071 0000 BR Avenida Paulista 1000, Sao Paulo 01310 R$ 1.299,90
15 INT-014 Sofia Lopez +52 55 5555 0000 MX Paseo de la Reforma 222, Ciudad de Mexico 06600 $1,500 MXN
16 INT-015 Min-jun Kim +82 2 2287 0114 KR Seoul Plaza, Seoul 04518 ₩1500000
17 INT-016 Mehmet Yilmaz +90 212 252 0000 TR Sultanahmet, Istanbul 34122 ₺1.250
18 INT-017 david cohen +972 3 6957 0000 IL Dizengoff 50, Tel Aviv 6433222 ₪450
19 INT-018 Hanna Kowalska +48 22 658 4500 PL Marszalkowska 1, Warszawa 00-624 zł 350,00
20 INT-019 Lars Nielsen +45 33 12 88 88 DK Vesterbrogade 1, Copenhagen 1620 kr 950
21 INT-020 Sven Eriksson +46 8 506 600 00 SE Drottninggatan 1, Stockholm 11151 kr 1.250,50