feat(gate): CSV-normalization gate with confidence-tiered findings

Adds a Review & Normalize page that sits between upload and every tool
page. The analyzer now tags each finding with confidence (high/medium/low)
and a fix_action; the gate auto-applies high-confidence fixes, surfaces
medium/low ones for user review, and blocks tool pages on error-level
findings until resolved or waived.

Core (src/core/):
  - analyze.py: Finding gains confidence, fix_action, pre_applied; new
    detectors for encoding_uncertain, encoding_decode_failed; new top-
    level encoding_override parameter.
  - fixes.py: registry of fix algorithms keyed by fix_action id.
  - normalize.py: auto_fix(), apply_decisions(), is_normalized(), and
    the NormalizationResult / Decision dataclasses the gate consumes.
  - io.py: detect_encoding tries strict UTF-8 first; repair_bytes now
    transcodes UTF-16/32 to UTF-8 before NUL-strip (fixes UTF-16 corruption)
    and normalizes line endings (fixes bare-CR parser crash); empty file
    handled gracefully instead of EmptyDataError traceback.

GUI (src/gui/):
  - pages/0_Review.py: gate page with per-finding decision controls,
    encoding override picker (16 codepages + custom), and Advanced output
    options (encoding, delimiter, line terminator) on the download.
  - components.py: require_normalization_gate() helper.
  - pages/1-9: gate guard wired on every tool page.

Test corpora:
  - test-cases/encodings-corpus/: 31 encoded CSV fixtures + 9 reference
    UTF-8 files + manifest, synced from Business/DataTools.
  - test-cases/text-cleaner-corpus/test_data/17: synced malformed input
    (unquoted $1,500.00) for the unquoted-delimiter detector.

Tests (94 new):
  - test_normalize.py (48): finding fields, fix registry, auto_fix scope,
    decision paths, gate idempotency, output-options helper.
  - test_encodings_corpus.py (90, 16 xfailed): parametric detection +
    decode + analyzer-no-crash sweep against the manifest.
  - test_analyze.py: encoding override + encoding_uncertain detectors.
  - test_corpus.py: pre-parse repair in the strict reader.

run_tests.py: new aliases --tool normalize, --tool encodings, --tool gate;
encodings corpus added to --fixtures category.

Docs: USER-GUIDE §3.3 covers the gate workflow, encoding override, and
output options; TECHNICAL §10.2.1-10.2.4 documents the analyzer schema,
gate API, Review page, and pre-parse repair pipeline; CLI-REFERENCE adds
the analyzer JSON schema with the new fields; README links to all of it.

Suite: 765 passed, 17 xfailed (was 458 passed).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-04-29 20:35:27 +00:00
parent e9c490ae1b
commit 82d7fef21e
68 changed files with 2883 additions and 34 deletions

View File

@@ -51,14 +51,24 @@ DEFAULT_CASES = [
def _read_csv_strict(path: Path) -> pd.DataFrame:
"""Read a corpus CSV file, treating all cells as strings.
NUL bytes are stripped from the raw file before parsing because the
pandas C engine truncates fields at NUL while the python engine is
too strict about embedded literal double quotes. Stripping NUL is
the file-level pre-clean step the spec describes for case 06.
Applies only the structural pre-parse fixes that are required to make
the file parseable at all — NUL stripping (case 06), line-ending
normalization (cases 09/10), and unquoted-currency repair (case 17).
Character-level folds that the cleaner itself owns (smart quotes,
NBSP, etc.) are deliberately left alone so the cleaner's own behavior
is what's under test.
"""
raw = path.read_bytes().replace(b"\x00", b"")
raw = path.read_bytes()
# NUL stripping
raw = raw.replace(b"\x00", b"")
# Line endings: CRLF -> LF, then bare CR -> LF.
raw = raw.replace(b"\r\n", b"\n").replace(b"\r", b"\n")
# Per-row repair (handles unquoted '$1,500.00' in case 17).
from src.core.io import _repair_rows
text = raw.decode("utf-8-sig")
text, _, _ = _repair_rows(text, ",")
return pd.read_csv(
io.BytesIO(raw), dtype=str, keep_default_na=False, encoding="utf-8-sig",
io.StringIO(text), dtype=str, keep_default_na=False,
)