Michael
abb720997e
Refactors all 10 docs (README, USER-GUIDE, CLI-REFERENCE, REQUIREMENTS, TECHNICAL, DEVELOPER, BUSINESS, DECISIONS, RECOVERY, docs/README) from prose-heavy to bullet-heavy + table-heavy. Same information density, significantly less reading load. Net: 2600 → 1652 lines (~37% reduction) WHILE adding the new content that landed since v1.6: - Format Standardizer (3rd Ready tool) - 199-row buyer corpus - src/core/errors.py structured hierarchy + ensure_dataframe / ensure_choice / wrap_file_read|write / format_for_user helpers - src/core/_constants.py shared USPS/state lookup tables - Cross-tool audit fixes (NaN matching, removed_df schema, validation, enum-bounds checks, forward-compat config) - Per-domain error_policy across format standardizers - Inconsistent-date-format detector - Excel header-row auto-detection + write_file delimiter param Per-doc changes: - README.md (175 → 71): 9-tool table at top, status column, 3 CLI entry points listed, dropped repeated marketing prose. - docs/README.md (38 → 27): pure index — buyer-facing vs creator-only split + version footer. - USER-GUIDE.md (208 → 118): tool table replaces script descriptions, troubleshooting compressed to bullets, gate explanation tightened. - CLI-REFERENCE.md (451 → 235): collapsed flag tables, removed redundant intro text, kept full recipes section. - REQUIREMENTS.md (146 → 129): 18 numbered sections (was 17), added §18 Error Handling, formatting tightened to single-line entries. - TECHNICAL.md (570 → 350): collapsed §3 build pipeline tables, merged redundant §3.5-3.7 OS sections, added §7 (Error handling) + §11.3 (Format Standardizer spec) + §11.4-11.7 (analyzer / gate / Review page / repair_bytes promoted from §10.2.x sub-numbering). - DEVELOPER.md (285 → 161): module map table replaces per-file prose, extension recipes condensed, new §Errors covers when to use each hierarchy class. - BUSINESS.md (278 → 225): collapsed prose to tables (use cases, competitive landscape, costs, risks); honest-status updated. - DECISIONS.md (269 → 189): scoring rubric + GUI matrix preserved, decision log compressed to single-line entries, added v1.6 entries (Format Standardizer Ready, errors module). - RECOVERY.md (180 → 147): rebuild steps as numbered + tabular, external dependencies as one table, recovery priorities tightened. No information removed; redundancy compressed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
119 lines
4.9 KiB
Markdown
119 lines
4.9 KiB
Markdown
# User Guide
|
||
|
||
**Version**: 1.6 · **Updated**: 2026-05-01
|
||
|
||
## 1. Install
|
||
|
||
You don't need Python — the bundle is self-contained.
|
||
|
||
| OS | File | How |
|
||
|----|------|-----|
|
||
| Windows | `BundleName-Setup-1.0.exe` | Double-click installer → desktop shortcut. |
|
||
| macOS | `BundleName-1.0.dmg` | Mount, drag to Applications. Signed + notarized. |
|
||
| Linux | `BundleName-1.0.AppImage` | `chmod +x`, double-click. (`.tar.gz` fallback available.) |
|
||
|
||
Launching opens your default browser to a local page (`http://localhost:8501`).
|
||
|
||
### How the GUI works
|
||
|
||
- Runs locally on your machine. **No internet, no upload.**
|
||
- Browser is just the display surface. Closing it stops the underlying program.
|
||
- Prefer the terminal? Every tool ships with a CLI too (Section 3).
|
||
|
||
### System requirements
|
||
|
||
- Windows 10/11 (64-bit), macOS 11+, modern Linux (2020+).
|
||
- Modern browser (Chrome, Edge, Firefox, Safari, last 3 years).
|
||
- ~400-500 MB free disk space.
|
||
|
||
Full numbered support matrix: [REQUIREMENTS.md](REQUIREMENTS.md).
|
||
|
||
## 2. What's included
|
||
|
||
| # | Tool | Purpose | Status |
|
||
|---|------|---------|--------|
|
||
| 01 | Deduplicator | Exact + fuzzy match, 5 normalizers, audit | Ready |
|
||
| 02 | Text Cleaner | Whitespace, smart chars, BOM, line endings, case ops | Ready |
|
||
| 03 | Format Standardizer | Dates / phones / emails / addresses / names / currencies / booleans | Ready |
|
||
| 04 | Missing Value Handler | Disguised nulls, imputation, drop-by-threshold | Coming Soon |
|
||
| 05 | Column Mapper | Rename + enforce schema | Coming Soon |
|
||
| 06 | Outlier Detector | z-score, IQR, multivariate | Coming Soon |
|
||
| 07 | Multi-File Merger | Combine multiple files | Coming Soon |
|
||
| 08 | Validator & Reporter | Rules + PDF/Excel report | Coming Soon |
|
||
| 09 | Pipeline Runner | One-click multi-tool launcher | Coming Soon |
|
||
|
||
**Sample data** (`samples/`): `messy_sales.csv`, `bank_export.xlsx`.
|
||
|
||
## 3. Usage
|
||
|
||
### 3.1 GUI (recommended)
|
||
|
||
1. Launch the bundle.
|
||
2. Pick a tool from the sidebar.
|
||
3. Drop your file (or select a sample).
|
||
4. Defaults are pre-filled — click **Run** to preview.
|
||
5. Click **Save Output** to write the cleaned file.
|
||
|
||
Advanced options are tucked in expander panes. The original file is never modified.
|
||
|
||
### 3.2 CLI
|
||
|
||
```bash
|
||
deduplicator customers.csv [--apply]
|
||
text-cleaner messy.csv [--apply]
|
||
format-standardize feed.csv [--apply]
|
||
```
|
||
|
||
Get help: `deduplicator --help`. Full reference: [CLI-REFERENCE.md](CLI-REFERENCE.md).
|
||
|
||
### 3.3 Run order (when running tools manually)
|
||
|
||
If you skip the Pipeline Runner, follow this order:
|
||
|
||
1. **02 Text Cleaner** first — normalizes whitespace + special chars.
|
||
2. **03 Format Standardizer** — dates, phones, etc. need cleaned text.
|
||
3. **04 Missing Value Handler** — sentinel codes hide as numbers.
|
||
4. **05 Column Mapper** — schema before outlier stats.
|
||
5. **06 Outlier Detector** — needs clean numerics. Stats on data with `NaN` or `-999` are mathematically poisoned.
|
||
6. **07 Multi-File Merger**, **08 Validator** as needed.
|
||
7. **01 Deduplicator** is order-flexible (normalizes internally for matching).
|
||
|
||
The Pipeline Runner enforces this automatically.
|
||
|
||
## 4. Review & Normalize gate
|
||
|
||
Every uploaded file is scanned before any tool sees it.
|
||
|
||
**Confidence tiers**:
|
||
- **High** — round-trip safe. One-click "Auto-fix high-confidence" applies them all.
|
||
- **Medium** — usually right, occasional false positives. Preview first.
|
||
- **Low** — heuristic. Off by default; opt in per finding.
|
||
- **Error** — blocks the gate (empty file, U+FFFD, unrepairable rows).
|
||
|
||
**Encoding override**: when the picker reports `encoding_uncertain` or you spot mojibake (`é`) or `<60>` chars, choose the right codepage at the top of the page (cp1252 for Western Excel, KOI8-R for older Russian, Big5 for traditional Chinese, …) → **Re-analyze**.
|
||
|
||
**Advanced output**: an `⚙️` expander on the download lets you tune encoding, delimiter, and line terminator. The download filename auto-adjusts (`.tsv` for tab, `.csv` otherwise).
|
||
|
||
## 5. Output
|
||
|
||
Every run writes:
|
||
- **Cleaned file** next to the input (or wherever you specify).
|
||
- **Audit file** (per-cell changes for text/format tools, match groups for dedup).
|
||
- **Timestamped log** in `logs/`.
|
||
|
||
Original input is never modified.
|
||
|
||
## 6. Troubleshooting
|
||
|
||
- **GUI won't launch / browser doesn't open** — wait 10-15 s; manually visit `http://localhost:8501`. Port-in-use error → close other instances.
|
||
- **Why does my browser open?** — local web app pattern (same as Jupyter, RStudio). Nothing leaves your machine.
|
||
- **Windows SmartScreen** — click "More info" → "Run anyway". Standard for non-EV-signed software.
|
||
- **macOS "App is damaged"** — re-download (file likely corrupted in transit).
|
||
- **Linux AppImage won't run** — `chmod +x file.AppImage`. Missing FUSE → `sudo apt install libfuse2` or use `.tar.gz`.
|
||
- **Slow on large file** — over ~100k rows takes longer; progress bar shows. Multi-million rows → use the CLI directly.
|
||
- **Need help** — email the address on your purchase receipt.
|
||
|
||
## 7. License
|
||
|
||
Single-user. See `LICENSE.txt`.
|