datatools-dev/samples/messy_text.csv at 38616d69e24fc0dcfdadbccb320e22c48bc594f8

Files

Michael 54f92ae47e feat: implement text cleaner (script 02) with CLI, GUI, and tests

Builds 02_text_cleaner.py from stub to working: character-level hygiene
for CSV/Excel inputs covering trim, whitespace collapse, smart-character
folding, Unicode NFC/NFKC, BOM strip, zero-width strip, control-char
strip, line-ending normalization, and per-column case conversion. Three
presets (minimal/excel-hygiene/paranoid) keep the buyer surface small.

- src/core/text_clean.py: pure helpers + CleanOptions/CleanResult +
  clean_dataframe with dtype-safe column selection
- src/cli_text_clean.py: Typer CLI mirroring the dedup CLI shape
  (dry-run by default, --apply writes cleaned + changes audit, JSON
  config save/load)
- src/gui/pages/2_Text_Cleaner.py: real Streamlit page with preset
  picker, advanced toggles, preview, before/after metrics, and three
  download buttons
- tests/test_text_clean.py + test_cli_text_clean.py: 92 new tests
  covering edge cases E1-E50 from the spec
- samples/messy_text.csv: demo dataset surfacing UC1, UC3, UC6, UC10
  in 10 rows
- test-cases/uc16-uc26 + ec05-ec09: per-use-case and per-edge-case
  fixtures

Docs: TECHNICAL.md §10.2 (full Tier 1/2/3 spec), DECISIONS.md v1.7
entry locking the spec, CLI-REFERENCE.md gains the text cleaner
section, README.md gains a top-level Text Cleaner block, USER-GUIDE.md
status row 02 promoted Skeleton -> Working.

200/200 tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-04-29 15:14:15 +00:00

599 B

Raw Blame History

1	customer_name	email	vendor	memo
2	Alice Johnson	alice@example.com	ACME Corp	Welcome aboard
3	Bob Smith	bob@example.com	ACME Corp	Returning customer
4	Charlie Brown	charlie@example.com	Globex	Net 30
5	Diana Prince	diana@example.com	Globex	VIP
6	Edward Norton	ed@example.com	“Best Pet Supplies”	Order#42 - rush
7	Frank Castle	frank@example.com	Stark—Industries	Line 1 Line 2 Line 3
8	grace HOPPER	grace@example.com	Globex	Loves long memos…
9	Henry Ford	henry@example.com	Ford Motor	Industrial
10	Iris West	iris@example.com	S.T.A.R. Labs	Notewith-bell
11	Jane Doe	jane@example.com	Acme	Standard

599 B Raw Blame History

599 B

Raw Blame History