giteadmin/datatools-dev
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: tight, scannable rewrite — every item earns its place

Browse Source 
Refactors all 10 docs (README, USER-GUIDE, CLI-REFERENCE, REQUIREMENTS,
TECHNICAL, DEVELOPER, BUSINESS, DECISIONS, RECOVERY, docs/README) from
prose-heavy to bullet-heavy + table-heavy. Same information density,
significantly less reading load.

Net: 2600 → 1652 lines (~37% reduction) WHILE adding the new content
that landed since v1.6:

- Format Standardizer (3rd Ready tool)
- 199-row buyer corpus
- src/core/errors.py structured hierarchy + ensure_dataframe /
  ensure_choice / wrap_file_read|write / format_for_user helpers
- src/core/_constants.py shared USPS/state lookup tables
- Cross-tool audit fixes (NaN matching, removed_df schema, validation,
  enum-bounds checks, forward-compat config)
- Per-domain error_policy across format standardizers
- Inconsistent-date-format detector
- Excel header-row auto-detection + write_file delimiter param

Per-doc changes:

- README.md (175 → 71): 9-tool table at top, status column, 3 CLI
  entry points listed, dropped repeated marketing prose.
- docs/README.md (38 → 27): pure index — buyer-facing vs creator-only
  split + version footer.
- USER-GUIDE.md (208 → 118): tool table replaces script descriptions,
  troubleshooting compressed to bullets, gate explanation tightened.
- CLI-REFERENCE.md (451 → 235): collapsed flag tables, removed
  redundant intro text, kept full recipes section.
- REQUIREMENTS.md (146 → 129): 18 numbered sections (was 17), added
  §18 Error Handling, formatting tightened to single-line entries.
- TECHNICAL.md (570 → 350): collapsed §3 build pipeline tables, merged
  redundant §3.5-3.7 OS sections, added §7 (Error handling) +
  §11.3 (Format Standardizer spec) + §11.4-11.7 (analyzer / gate /
  Review page / repair_bytes promoted from §10.2.x sub-numbering).
- DEVELOPER.md (285 → 161): module map table replaces per-file prose,
  extension recipes condensed, new §Errors covers when to use each
  hierarchy class.
- BUSINESS.md (278 → 225): collapsed prose to tables (use cases,
  competitive landscape, costs, risks); honest-status updated.
- DECISIONS.md (269 → 189): scoring rubric + GUI matrix preserved,
  decision log compressed to single-line entries, added v1.6 entries
  (Format Standardizer Ready, errors module).
- RECOVERY.md (180 → 147): rebuild steps as numbered + tabular,
  external dependencies as one table, recovery priorities tightened.

No information removed; redundancy compressed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Michael
2026-05-01 02:49:29 +00:00
parent 26b9771625
commit abb720997e
10 changed files with 1105 additions and 2053 deletions
Download Patch File Download Diff File Expand all files Collapse all files

466
docs/CLI-REFERENCE.md
View File

@@ -1,431 +1,211 @@
# CLI Reference
Complete command-line reference for the DataTools bundle.
DataTools ships two CLI modules so each script can be invoked independently:
Three CLI modules, one per Ready tool:
| Module | Command | Purpose |
|---|---|---|
| `src.cli` | `python -m src.cli INPUT_FILE [OPTIONS]` | Deduplicator (script 01) |
| `src.cli_text_clean` | `python -m src.cli_text_clean INPUT_FILE [OPTIONS]` | Text cleaner (script 02) |
|--------|---------|---------|
| `src.cli` | `python -m src.cli FILE` | Deduplicator |
| `src.cli_text_clean` | `python -m src.cli_text_clean FILE` | Text Cleaner |
| `src.cli_analyze` | `python -m src.cli_analyze FILE` | Analyzer (read-only scan) |
The deduplicator section is below; the text cleaner reference is in [Section: Text Cleaner CLI](#text-cleaner-cli).
Every command is **preview-only by default** — add `--apply` to write output.
## Deduplicator
---
# Deduplicator
```
python -m src.cli INPUT_FILE [OPTIONS]
```
## Arguments
| Argument | Required | Description |
|----------|----------|-------------|
| `INPUT_FILE` | Yes | Path to the CSV, delimited text, or Excel file to deduplicate |
## Options
### Core
- `--apply` — write output files (default: preview).
- `-o, --output PATH` — output path (default `{input}_deduplicated.csv`).
| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--apply` | | `false` | Write output files. Without this flag, only a preview is shown. |
| `--output` | `-o` | `{input}_deduplicated.csv` | Output file path. |
### Column selection
- `-s, --subset COLS` — comma-separated columns to match on (default: auto-detect).
- `-k, --key COLS` — strong-key columns; each becomes an independent exact-match strategy (`fb_id`, `ein`, `sku`).
### Column Selection
| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--subset` | `-s` | auto-detect | Comma-separated columns to match on. When omitted, columns are auto-detected by name pattern (email, phone, name, address). |
| `--key` | `-k` | none | Comma-separated strong-key columns. Each becomes an independent exact-match strategy. Use for identifiers like `fb_id`, `ein`, `sku`. |
### Fuzzy Matching
| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--fuzzy` | | none | Comma-separated columns to fuzzy-match. Other columns in the strategy use exact matching. |
| `--algorithm` | `-a` | `jaro_winkler` | Fuzzy algorithm: `levenshtein`, `jaro_winkler`, or `token_set_ratio`. |
| `--threshold` | `-t` | `85` | Similarity threshold 0-100. Lower values find more matches but increase false positives. |
### Fuzzy matching
- `--fuzzy COLS` — comma-separated columns to fuzzy-match.
- `-a, --algorithm ALG` — `levenshtein` / `jaro_winkler` (default) / `token_set_ratio`.
- `-t, --threshold N` — similarity 0-100 (default 85).
### Normalization
- `--normalize COL:TYPE` — comma-separated `col:type` pairs. Types: `email`, `phone`, `name`, `address`, `string`.
| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--normalize` | | auto-detect | Column normalizers as `col:type` pairs, comma-separated. Types: `email`, `phone`, `name`, `address`, `string`. |
| Type | Effect | Example |
|------|--------|---------|
| `email` | lowercase, strip Gmail dots, strip `+tag` | `John.Doe+x@gmail.com` → `johndoe@gmail.com` |
| `phone` | E.164 (+ ext preserved) | `(555) 123-4567 ext 100` → `+15551234567;ext=100` |
| `name` | strip titles + suffixes + particles, case-fold | `Dr. Charles de Gaulle Jr.` → `charles gaulle` |
| `address` | USPS abbrevs + state name → 2-letter, case-fold | `123 Main Street, California` → `123 main st ca` |
| `string` | trim + collapse + case-fold | ` HELLO WORLD ` → `hello world` |
**Normalizer details:**
### Survivor selection
- `--survivor RULE` — `first` (default) / `last` / `most-complete` / `most-recent`.
- `--date-column COL` — required for `most-recent`.
- `--merge` — fill blanks in survivor from removed rows.
| Type | What it does | Example |
|------|-------------|---------|
| `email` | Lowercase, strip Gmail dots, strip `+tag` suffixes | `John.Doe+tag@gmail.com` → `johndoe@gmail.com` |
| `phone` | Parse to E.164 format; fallback: digits only | `(555) 123-4567` → `+15551234567` |
| `name` | Strip titles (Dr., Mr.) and suffixes (Jr., PhD), case-fold | `Dr. John Smith Jr.` → `john smith` |
| `address` | USPS abbreviations (Street→St, Avenue→Ave), case-fold | `123 Main Street, Suite 4` → `123 main st ste 4` |
| `string` | Trim, collapse whitespace, case-fold | ` HELLO WORLD ` → `hello world` |
### Survivor Selection
| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--survivor` | | `first` | Which row to keep per duplicate group. |
| `--date-column` | | none | Date column for the `most-recent` rule. |
| `--merge` | | `false` | Fill missing fields in the surviving row from removed duplicates. |
**Survivor rules:**
| Rule | Behavior |
|------|----------|
| `first` | Keep the first row encountered (lowest row number) |
| `last` | Keep the last row encountered (highest row number) |
| `most-complete` | Keep the row with the fewest blank/empty cells |
| `most-recent` | Keep the row with the latest date (requires `--date-column`) |
### Interactive Review
| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--review` | | `false` | Interactively review each match group. For each group, choose: merge (y), keep both (n), or skip remaining (s). |
### Interactive review
- `--review` — prompt y/n/s per match group with side-by-side diff.
### Configuration
- `--config PATH` — load all settings from JSON.
- `--save-config PATH` — save current settings to JSON.
| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--config` | | none | Load all settings from a saved JSON config file. |
| `--save-config` | | none | Save current settings to a JSON config file for reuse. |
### File Handling
| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--sheet` | | first sheet | Excel sheet name or 0-based index. Ignored for CSV files. |
| `--encoding` | | auto-detect | Override auto-detected file encoding (e.g., `utf-8`, `windows-1252`). |
| `--header-row` | | auto-detect | 0-based row index for the header row. |
---
### File handling
- `--sheet NAME|N` — Excel sheet name or 0-based index.
- `--encoding ENC` — override auto-detected encoding.
- `--header-row N` — 0-based header row.
## Recipes
### 1. Basic Dedup (Auto-Detect)
Let the engine detect email, phone, name, and address columns automatically.
```bash
# Preview
python -m src.cli customers.csv
# Basic auto-detect dedup
python -m src.cli customers.csv [--apply]
# Apply
python -m src.cli customers.csv --apply
```
The engine scans column names for patterns like `email`, `phone`, `name`, `address` and builds strategies automatically. Strong keys (email, phone) become standalone strategies; weak keys (name, address) are paired with strong keys.
### 2. Fuzzy Name Matching
Match rows where names are similar but not identical — catches typos, nickname variations, and formatting differences.
```bash
# Fuzzy-match on the "name" column at 80% similarity
# Fuzzy name match at 80%
python -m src.cli customers.csv --fuzzy name --threshold 80 --apply
# Fuzzy-match on multiple columns
python -m src.cli customers.csv --fuzzy name,address --threshold 85 --apply
# Use Levenshtein distance instead of Jaro-Winkler
python -m src.cli customers.csv --fuzzy name --algorithm levenshtein --threshold 80 --apply
```
**Algorithm comparison:**
- `jaro_winkler` (default) — best for short strings like names; weights early characters more heavily
- `levenshtein` — edit-distance ratio; works well for typos and transpositions
- `token_set_ratio` — best for addresses and long strings; ignores word order
### 3. Custom Strong Keys
Use specific identifier columns to find exact duplicates.
```bash
# Deduplicate by Facebook ID
python -m src.cli donors.csv --key fb_id --apply
# Multiple strong keys (each is independent — matched with OR)
# Multiple strong keys (OR logic)
python -m src.cli donors.csv --key fb_id,ein --apply
```
Strong keys are OR'd: a match on `fb_id` alone OR `ein` alone marks rows as duplicates.
### 4. Merge Mode
Keep the most complete row and fill any remaining blanks from the duplicates.
```bash
# Most complete row + merge missing fields
# Most-complete row + merge missing fields
python -m src.cli contacts.csv --survivor most-complete --merge --apply
# Keep most recent row and merge
# Most-recent + merge
python -m src.cli contacts.csv --survivor most-recent --date-column updated_at --merge --apply
```
**How merge works:** The survivor row keeps all its non-empty fields. For any blank/null fields, the engine fills from the removed rows (in row order). The result is a single row with maximum data retention.
### 5. Multi-Column Subset
Match on a specific combination of columns rather than auto-detecting.
```bash
# Exact match on email + phone only
python -m src.cli customers.csv --subset email,phone --apply
# Mix exact and fuzzy within a subset
python -m src.cli customers.csv --subset email,name --fuzzy name --threshold 85 --apply
```
When using `--subset`, all listed columns must match (AND logic) for a pair to be considered duplicates.
### 6. Save and Load Config Profiles
Save your settings for repeatable runs on similar files.
```bash
# Save settings to a file
python -m src.cli customers.csv --fuzzy name --threshold 80 --merge \
--survivor most-complete --save-config customer_dedup.json
# Load saved settings
python -m src.cli new_customers.csv --config customer_dedup.json --apply
```
Config files are JSON. Example:
```json
{
"strategies": [],
"survivor_rule": "most_complete",
"merge": true,
"default_algorithm": "jaro_winkler",
"default_threshold": 80.0,
"fuzzy_columns": ["name"]
}
```
### 7. Interactive Review
Step through each match group and decide whether to merge.
```bash
# Interactive review
python -m src.cli customers.csv --review --apply
# Save / load profile
python -m src.cli customers.csv --fuzzy name --threshold 80 --save-config dedup.json
python -m src.cli new.csv --config dedup.json --apply
# Excel
python -m src.cli data.xlsx --sheet "Sales" --apply
```
For each group, the CLI displays both rows side-by-side and prompts:
## Algorithms
```
============================================================
Match Group 1 — Confidence: 92.3%
Matched on: name, phone
============================================================
- **`jaro_winkler`** (default) — best for short strings (names); weights early chars.
- **`levenshtein`** — edit-distance ratio; typos and transpositions.
- **`token_set_ratio`** — best for addresses; ignores word order.
Row 1:
name: John Smith
email: john@example.com
phone: (555) 123-4567
## Auto-detection
Row 2:
name: Jon Smith
email:
phone: 555-123-4567
When no `--subset` / `--fuzzy` flags, columns are detected by name:
[y] Merge [n] Keep both [s] Skip remaining:
```
| Pattern | Algorithm | Threshold | Normalizer | Key |
|---------|-----------|-----------|------------|-----|
| Email | exact | 100% | email | strong |
| Phone | exact | 100% | phone | strong |
| Name | jaro_winkler | 85% | name | weak |
| Address | token_set_ratio | 80% | address | weak |
- **y** — accept the match; merge/remove duplicate
- **n** — reject the match; keep both rows
- **s** — skip all remaining groups (keep both for all)
**Strategy rules**: strong keys → standalone OR; weak keys → AND-paired with each strong key; no strong keys → weak promoted to standalone; no patterns → exact match on all columns.
### 8. Excel Files and Multi-Sheet
## Output files (with `--apply`)
Work with Excel files directly — no CSV conversion needed.
| File | Contents |
|------|----------|
| `{stem}_deduplicated.csv` | Cleaned data |
| `{stem}_removed.csv` | Removed rows |
| `{stem}_match_groups.csv` | `_group_id`, `_is_survivor`, `_confidence`, `_matched_on`, `_original_row` + originals |
```bash
# Deduplicate first sheet (default)
python -m src.cli data.xlsx --apply
# Specify sheet by name
python -m src.cli data.xlsx --sheet "Sales Data" --apply
# Specify sheet by index (0-based)
python -m src.cli data.xlsx --sheet 1 --apply
```
Output is always CSV by default. To write Excel output, use `-o`:
```bash
python -m src.cli data.xlsx -o cleaned.xlsx --apply
```
Log: `logs/dedup_YYYYMMDD_HHMMSS.log`.
---
## Auto-Detection Details
When no `--subset` or `--fuzzy` flags are provided, the engine scans column names and builds strategies:
| Column pattern | Detection regex | Algorithm | Threshold | Normalizer | Key type |
|---------------|----------------|-----------|-----------|------------|----------|
| Email | `e[-_]?mail` | exact | 100% | email | strong |
| Phone | `phone\|telephone\|mobile\|cell` | exact | 100% | phone | strong |
| Name | `^(name\|full_name\|customer_name\|...)$` | jaro_winkler | 85% | name | weak |
| Address | `address\|street\|addr` | token_set_ratio | 80% | address | weak |
**Strategy building rules:**
- Strong keys → standalone OR strategies (email match alone is enough)
- Weak keys → paired with each strong key via AND (name match requires email or phone match too)
- No strong keys found → weak keys promoted to standalone
- No patterns matched → exact match on all columns (equivalent to `drop_duplicates`)
## Output Files
When `--apply` is set, three files are written:
| File | Description |
|------|-------------|
| `{stem}_deduplicated.csv` | Cleaned DataFrame with duplicates removed |
| `{stem}_removed.csv` | Rows that were removed |
| `{stem}_match_groups.csv` | Audit trail with `_group_id`, `_is_survivor`, `_confidence`, `_matched_on`, `_original_row`, plus all original columns |
## Logging
Every run writes a timestamped log to `logs/dedup_YYYYMMDD_HHMMSS.log` with full debug-level details: strategies used, pair comparisons, survivor decisions, and merge actions.
---
# Text Cleaner CLI
Character-level hygiene for CSV / Excel files: whitespace trim and collapse, smart-character folding, Unicode normalization, BOM strip, control-char strip, line-ending normalization, optional case conversion. See TECHNICAL.md Section 10.2 for the full functional spec.
# Text Cleaner
```
python -m src.cli_text_clean INPUT_FILE [OPTIONS]
```
## Arguments
| Argument | Required | Description |
|----------|----------|-------------|
| `INPUT_FILE` | Yes | Path to the CSV, TSV, or Excel file to clean |
Character-level hygiene. See [TECHNICAL.md §10.2](TECHNICAL.md) for the spec.
## Options
### Core
| Flag | Short | Default | Description |
|------|-------|---------|-------------|
| `--apply` | | `false` | Write output files. Without this flag, only a preview is shown. |
| `--output` | `-o` | `{input}_cleaned.csv` | Output file path. |
| `--preset` | | `excel-hygiene` | Preset bundle of safe defaults. See [Presets](#presets). |
- `--apply` — write output (default: preview).
- `-o, --output PATH` — output path (default `{input}_cleaned.csv`).
- `--preset NAME` — `minimal` / `excel-hygiene` (default) / `paranoid`.
### Scope
- `--columns COLS` — comma-separated columns to clean (default: all string columns).
- `--skip COLS` — exclude these columns.
| Flag | Default | Description |
|------|---------|-------------|
| `--columns` | all string columns | Comma-separated columns to clean. |
| `--skip` | none | Comma-separated columns to skip even if they look like text. Useful for free-text notes columns you don't want touched. |
### Per-op overrides (override the active preset)
- `--no-trim`, `--no-collapse`, `--no-nfc`, `--nfkc`, `--no-smart-chars`, `--no-zero-width`, `--no-bom`, `--no-control`, `--no-line-endings`.
### Per-operation toggles
### Case
- `--case MODE` — `upper` / `lower` / `title` / `sentence`. Or per-column: `--case title:name,upper:sku`.
- Title case preserves all-caps tokens (`USA`) and lowercases mid-string particles (`of`, `and`).
These override the active preset.
### Audit + config
- `--full-changelog` — write every change (default caps to first 1000).
- `--config PATH` / `--save-config PATH`.
| Flag | Effect |
|------|--------|
| `--no-trim` | Disable leading/trailing whitespace strip |
| `--no-collapse` | Disable internal whitespace collapse |
| `--no-nfc` | Disable Unicode NFC normalization |
| `--nfkc` | Enable NFKC compatibility fold (lossy: `①` → `1`, `fi` → `fi`) |
| `--no-smart-chars` | Disable smart-character folding (curly quotes, em/en-dash, NBSP, ellipsis) |
| `--no-zero-width` | Disable zero-width / invisible character strip |
| `--no-bom` | Disable leading BOM strip |
| `--no-control` | Disable control-character strip |
| `--no-line-endings` | Disable line-ending normalization |
### Case conversion
| Flag | Forms | Description |
|------|-------|-------------|
| `--case` | `upper`, `lower`, `title`, `sentence` | Apply this case to every selected column |
| `--case` | `mode:col[,mode:col]` | Per-column case (e.g., `--case title:name,upper:code`) |
Title case preserves all-caps tokens (`USA` stays `USA`) and lowercases mid-string particles (`of`, `and`, `the`, etc.).
### Audit and config
| Flag | Default | Description |
|------|---------|-------------|
| `--full-changelog` | `false` | Write every cell change to the audit CSV (default caps to first 1000). |
| `--config` | none | Load options from a saved JSON config file. |
| `--save-config` | none | Save the current options to a JSON config file. |
### File format / encoding
| Flag | Default | Description |
|------|---------|-------------|
| `--sheet` | `0` | Excel sheet name or 0-based index. |
| `--encoding` | auto-detect | Override auto-detected file encoding. |
| `--header-row` | auto-detect | 0-based row index for the header. |
### File
- `--sheet`, `--encoding`, `--header-row` — same as Deduplicator.
## Presets
| Preset | What it does |
|---|---|
| `minimal` | Trim + collapse whitespace only. Nothing else. |
| `excel-hygiene` (default) | Trim, collapse, NFC, smart-character fold, zero-width strip, BOM strip, control strip, line-ending normalize. NFKC off. |
| `paranoid` | All of `excel-hygiene` plus NFKC compatibility fold (lossy). |
## Output Files
When `--apply` is set:
| File | Description |
|------|-------------|
| `{stem}_cleaned.csv` | Cleaned DataFrame |
| `{stem}_changes.csv` | Per-cell audit: `row`, `column`, `old`, `new`, `ops_applied` (capped to 1000 rows by default; use `--full-changelog` for all) |
A timestamped log is always written to `logs/text_clean_YYYYMMDD_HHMMSS.log`.
|--------|--------------|
| `minimal` | Trim + collapse only. |
| `excel-hygiene` (default) | Trim, collapse, NFC, smart-char fold, zero-width strip, BOM strip, control strip, line-ending normalize. |
| `paranoid` | `excel-hygiene` + NFKC compatibility fold (lossy). |
## Recipes
```bash
# Preview what would change with the safe defaults
python -m src.cli_text_clean messy.csv
# Safe defaults (preview, then apply)
python -m src.cli_text_clean messy.csv [--apply]
# Apply the safe defaults
python -m src.cli_text_clean messy.csv --apply
# Just the basics — only trim and collapse, leave Unicode/quotes alone
# Just trim + collapse, leave Unicode alone
python -m src.cli_text_clean messy.csv --preset minimal --apply
# Title-case the name column, upper-case the SKU column, leave others alone for case
# Title-case names, upper-case SKUs
python -m src.cli_text_clean people.csv --case title:name,upper:sku --apply
# Clean only specific columns
python -m src.cli_text_clean orders.csv --columns vendor,product --apply
# Skip a free-text notes column from cleaning
# Skip a free-text notes column
python -m src.cli_text_clean tickets.csv --skip notes --apply
# Save the current settings as a profile and reload it later
python -m src.cli_text_clean messy.csv --preset minimal --case upper --save-config my.json
python -m src.cli_text_clean other.csv --config my.json --apply
```
## Output files (with `--apply`)
| File | Contents |
|------|----------|
| `{stem}_cleaned.csv` | Cleaned data |
| `{stem}_changes.csv` | `row`, `column`, `old`, `new`, `ops_applied` (capped to 1000; `--full-changelog` removes cap) |
Log: `logs/text_clean_YYYYMMDD_HHMMSS.log`.
---
## Analyzer (upload-time scan)
# Analyzer
```
python -m src.cli_analyze INPUT_FILE [OPTIONS]
--sample-rows N Cap on rows scanned (default 1000)
--json Print findings as a JSON array on stdout
--strict Exit non-zero on any warn/error finding
```
JSON output schema (one object per finding):
Read-only scan; surfaces every detector finding without modifying the file.
## Options
- `--sample-rows N` — cap on rows scanned (default 1000).
- `--json` — print findings as a JSON array on stdout.
- `--strict` — exit non-zero on any warn/error finding.
## JSON schema (one object per finding)
```json
{
@@ -442,10 +222,14 @@ JSON output schema (one object per finding):
}
```
- `severity` — `info` / `warn` / `error`. Only `error` blocks the GUI normalization gate.
- `confidence` — `high` (round-trip-safe, eligible for one-click auto-fix), `medium` (preview before applying), `low` (heuristic, opt-in only).
- `fix_action` — stable id naming the algorithm in `src/core/fixes.py` that resolves the finding. Empty string for informational-only findings.
- `pre_applied` — `true` for fixes already applied during the byte-level read pass (BOM strip, NUL strip, line-ending normalize, byte-level smart-quote fold, transcode-to-UTF-8 from UTF-16/32). The GUI gate treats these as already-resolved; the CLI emits them so callers can audit what changed during read.
## Field meanings
- `severity` — `info` / `warn` / `error`. Only `error` blocks the GUI gate.
- `confidence` — `high` (one-click), `medium` (preview), `low` (opt-in).
- `fix_action` — id of the algorithm in `src/core/fixes.py`. Empty for informational-only.
- `pre_applied` — `true` for fixes already applied during the byte-level read pass.
The detector set covers smart punctuation, NBSP / Unicode whitespace, zero-width characters, dirty headers, whitespace padding, null-like sentinels, mojibake fingerprints (UTF-8-as-cp1252), mixed-case email columns, near-duplicate rows (case-and-padding stripped), leading-zero IDs (Excel hazard), mixed line endings, encoding decode failure (`encoding_decode_failed`), and U+FFFD presence in the loaded text (`encoding_uncertain`). New detectors plug in by appending one entry to `analyze.py` and one matching fix in `fixes.py`.
## Detectors
Smart punctuation, NBSP / Unicode whitespace, zero-width chars, dirty headers, whitespace padding, null-like sentinels, mojibake fingerprints, mixed-case email columns, inconsistent date formats, near-duplicate rows, leading-zero IDs, mixed line endings, encoding decode failure, U+FFFD presence.
Add a detector: append entry in `analyze.py` + matching fix in `fixes.py`. No other call sites change.