feat: implement text cleaner (script 02) with CLI, GUI, and tests
Builds 02_text_cleaner.py from stub to working: character-level hygiene for CSV/Excel inputs covering trim, whitespace collapse, smart-character folding, Unicode NFC/NFKC, BOM strip, zero-width strip, control-char strip, line-ending normalization, and per-column case conversion. Three presets (minimal/excel-hygiene/paranoid) keep the buyer surface small. - src/core/text_clean.py: pure helpers + CleanOptions/CleanResult + clean_dataframe with dtype-safe column selection - src/cli_text_clean.py: Typer CLI mirroring the dedup CLI shape (dry-run by default, --apply writes cleaned + changes audit, JSON config save/load) - src/gui/pages/2_Text_Cleaner.py: real Streamlit page with preset picker, advanced toggles, preview, before/after metrics, and three download buttons - tests/test_text_clean.py + test_cli_text_clean.py: 92 new tests covering edge cases E1-E50 from the spec - samples/messy_text.csv: demo dataset surfacing UC1, UC3, UC6, UC10 in 10 rows - test-cases/uc16-uc26 + ec05-ec09: per-use-case and per-edge-case fixtures Docs: TECHNICAL.md §10.2 (full Tier 1/2/3 spec), DECISIONS.md v1.7 entry locking the spec, CLI-REFERENCE.md gains the text cleaner section, README.md gains a top-level Text Cleaner block, USER-GUIDE.md status row 02 promoted Skeleton -> Working. 200/200 tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
13
samples/messy_text.csv
Normal file
13
samples/messy_text.csv
Normal file
@@ -0,0 +1,13 @@
|
||||
customer_name,email,vendor,memo
|
||||
Alice Johnson,alice@example.com,ACME Corp ,Welcome aboard
|
||||
Bob Smith,bob@example.com,ACME Corp,Returning customer
|
||||
Charlie Brown,charlie@example.com,Globex,Net 30
|
||||
Diana Prince,diana@example.com,Globex,VIP
|
||||
Edward Norton,ed@example.com,“Best Pet Supplies”,Order#42 - rush
|
||||
Frank Castle,frank@example.com,Stark—Industries,"Line 1
|
||||
Line 2
|
||||
Line 3"
|
||||
grace HOPPER ,grace@example.com,Globex,Loves long memos…
|
||||
Henry Ford,henry@example.com,Ford Motor,Industrial
|
||||
Iris West,iris@example.com,S.T.A.R. Labs,Notewith-bell
|
||||
Jane Doe,jane@example.com,Acme,Standard
|
||||
|
Reference in New Issue
Block a user