fix: cross-tool audit findings + alignment with format standardizer

Closes 12 bugs and 8 gaps surfaced by parallel audits across all core
modules, plus aligns the dedup-side normalizers with the new
format_standardize behavior where they had silently diverged.

Bugs (data integrity / correctness):
- dedup: NaN/None values matched as duplicates because str(None)='None'.
  Two rows with missing email silently merged.
- dedup: removed_df had 0 columns when nothing was removed; downstream
  code expecting matching schema broke. Now preserves column shape.
- dedup: ColumnMatchStrategy threshold accepted any value; out-of-range
  silently broke matching. Validated to [0, 100] in __post_init__.
- dedup: strategy referencing a missing column was silently skipped.
  Now raises ValueError listing available columns.
- fixes: replace_null_sentinels crashed on non-string sentinels (int/None
  from JSON payload). Coerced to str.
- fixes: _vectorized_regex_sub raised raw re.error on bad patterns. Now
  wraps as ValueError with clear message.
- io: detect_header_row mis-identified all-empty and metadata-only rows
  as headers (all([]) is True). Now requires ≥2 non-empty cells.
- config: from_dict crashed when JSON had unknown fields, breaking
  forward compat. Now filters to known fields.
- analyze: mixed-case email detector flagged all-None columns because
  str(None)='None' contains both N and one. Now drops NaN before stringify.

New features and gap closures:
- io: _detect_excel_header_row mirrors detect_header_row for Excel via
  openpyxl read-only; _read_excel uses it when header_row=None.
- io: write_file gains delimiter + encoding params; .tsv extension
  defaults to tab.
- normalizers: normalize_phone preserves extensions as ;ext=N suffix.
- normalizers: normalize_address folds spelled-out US state names to
  2-letter codes (California ≡ CA).
- normalizers: normalize_name drops surname particles (van, de, von)
  so "Charles de Gaulle" ≡ "Charles Gaulle" for matching.
- analyze: new _detect_inconsistent_date_format detector flags columns
  with mixed ISO/US/EU date shapes; routes to format standardizer.
- analyze: _NULL_LIKE recognizes "<na>" (pd.NA repr).
- analyze: duplicate-row finding renamed count → n_extra (rows that
  would actually be removed) with clarified description.
- dedup: group_confidence no longer falsely 100.0 when transitive group
  members lack a recorded direct pair; falls back to 100.0 only when
  truly no pairs were observed.
- dedup: MatchResult / DeduplicationResult docstrings clarify that
  row_indices refer to the input frame's positional index (output index
  is reset).
- text_clean: visualize_hidden_html(None) now returns None (matches
  visualize_hidden_text); strip_bom strips at most one BOM per call;
  sentence_case dead elif branch removed.

Tests:
- tests/test_audit_fixes.py — 28 regression tests, one or more per
  numbered finding, named after BUG/GAP/NIT tags so future readers
  can trace each test back to its audit.
- tests/test_fixes_unit.py — 26 isolated unit tests for previously
  integration-only fix functions (trim_whitespace, strip_nbsp,
  strip_zero_width, normalize_line_endings, clean_headers,
  repair_mojibake — last skipped if ftfy unavailable).
- tests/test_io.py — adds CSV / TSV / semicolon / UTF-8-BOM round-trip
  tests + Excel auto-header-detection tests.
- tests/test_normalizers.py — adds 8 tests for the alignment work
  above (phone extension, state names, particles).

Adds .claude/ to .gitignore (agent worktrees + local settings).

Full project suite: 1197 passed, 4 skipped, 17 xfailed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

tests/test_audit_fixes.py

@@ -0,0 +1,303 @@
+"""Regression tests for bugs surfaced by the cross-tool audit.
+
+Each test pins a specific behavioral bug or gap that an audit
+identified. Test names match the BUG-N / GAP-N tags in the audit
+notes so a future reader can trace why each test exists.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+import numpy as np
+import pandas as pd
+import pytest
+
+from src.core.analyze import _NULL_LIKE, _detect_mixed_case_email
+import src.core.fixes as f
+from src.core.config import (
+    ColumnStrategyConfig,
+    DeduplicationConfig,
+    StrategyConfig,
+)
+from src.core.dedup import (
+    Algorithm,
+    ColumnMatchStrategy,
+    MatchStrategy,
+    deduplicate,
+)
+from src.core.io import detect_header_row
+from src.core.text_clean import sentence_case, smart_title_case, strip_bom
+
+
+# ---------------------------------------------------------------------------
+# BUG-1: dedup NaN values must not match as duplicates
+# ---------------------------------------------------------------------------
+
+class TestDedupNaNHandling:
+    def test_two_nan_emails_do_not_match(self):
+        # Both rows have NaN for email; no other matching column. Without
+        # the fix, str(NaN) == "nan" would match exactly and the rows
+        # would silently merge.
+        df = pd.DataFrame({
+            "id": [1, 2],
+            "email": [np.nan, np.nan],
+        })
+        strategies = [MatchStrategy(column_strategies=[
+            ColumnMatchStrategy(column="email", algorithm=Algorithm.EXACT,
+                                threshold=100.0),
+        ])]
+        result = deduplicate(df, strategies=strategies)
+        assert len(result.deduplicated_df) == 2
+        assert len(result.match_groups) == 0
+
+    def test_one_nan_one_real_does_not_match(self):
+        df = pd.DataFrame({
+            "email": [np.nan, "alice@example.com"],
+        })
+        strategies = [MatchStrategy(column_strategies=[
+            ColumnMatchStrategy(column="email", algorithm=Algorithm.EXACT),
+        ])]
+        result = deduplicate(df, strategies=strategies)
+        assert len(result.deduplicated_df) == 2
+
+    def test_none_does_not_match_string_none(self):
+        df = pd.DataFrame({
+            "name": [None, "None"],
+        })
+        strategies = [MatchStrategy(column_strategies=[
+            ColumnMatchStrategy(column="name", algorithm=Algorithm.EXACT),
+        ])]
+        result = deduplicate(df, strategies=strategies)
+        assert len(result.deduplicated_df) == 2
+
+
+# ---------------------------------------------------------------------------
+# BUG-2: removed_df must preserve column schema even when empty
+# ---------------------------------------------------------------------------
+
+class TestDedupRemovedDfSchema:
+    def test_empty_removed_df_has_same_columns(self):
+        df = pd.DataFrame({
+            "name": ["alice", "bob", "carol"],
+            "email": ["a@x.com", "b@x.com", "c@x.com"],
+        })
+        strategies = [MatchStrategy(column_strategies=[
+            ColumnMatchStrategy(column="email", algorithm=Algorithm.EXACT),
+        ])]
+        result = deduplicate(df, strategies=strategies)
+        # No duplicates → empty removed_df, but columns must match.
+        assert len(result.removed_df) == 0
+        assert list(result.removed_df.columns) == list(result.deduplicated_df.columns)
+
+
+# ---------------------------------------------------------------------------
+# GAP-3: missing column reference should raise
+# ---------------------------------------------------------------------------
+
+class TestDedupMissingColumn:
+    def test_missing_column_raises(self):
+        df = pd.DataFrame({"email": ["a@x.com"]})
+        strategies = [MatchStrategy(column_strategies=[
+            ColumnMatchStrategy(column="e_mail", algorithm=Algorithm.EXACT),
+        ])]
+        with pytest.raises(ValueError, match="not present in the input"):
+            deduplicate(df, strategies=strategies)
+
+
+# ---------------------------------------------------------------------------
+# GAP-4: threshold must be in [0, 100]
+# ---------------------------------------------------------------------------
+
+class TestThresholdValidation:
+    def test_negative_threshold_rejected(self):
+        with pytest.raises(ValueError, match=r"\[0, 100\]"):
+            ColumnMatchStrategy(column="x", threshold=-1)
+
+    def test_over_hundred_rejected(self):
+        with pytest.raises(ValueError, match=r"\[0, 100\]"):
+            ColumnMatchStrategy(column="x", threshold=101)
+
+    def test_zero_and_hundred_allowed(self):
+        ColumnMatchStrategy(column="x", threshold=0)
+        ColumnMatchStrategy(column="x", threshold=100)
+
+    def test_non_numeric_rejected(self):
+        with pytest.raises(TypeError):
+            ColumnMatchStrategy(column="x", threshold="high")  # type: ignore[arg-type]
+
+
+# ---------------------------------------------------------------------------
+# BUG-9: replace_null_sentinels must coerce non-string sentinels
+# ---------------------------------------------------------------------------
+
+class TestReplaceNullSentinelsTypes:
+    def test_int_sentinels_do_not_crash(self):
+        df = pd.DataFrame({"x": ["0", "5", ""]})
+        out, _ = f.replace_null_sentinels(df, {"sentinels": [0, "5"]})
+        assert out.loc[0, "x"] == ""   # "0" matched int 0 stringified
+        assert out.loc[1, "x"] == ""   # "5" matched
+        assert out.loc[2, "x"] == ""   # already empty
+
+    def test_none_sentinel_skipped(self):
+        df = pd.DataFrame({"x": ["a", "b"]})
+        # Should not crash on None entry in the sentinel list.
+        out, _ = f.replace_null_sentinels(df, {"sentinels": ["a", None]})
+        assert out.loc[0, "x"] == ""
+        assert out.loc[1, "x"] == "b"
+
+
+# ---------------------------------------------------------------------------
+# BUG-10: malformed regex should raise ValueError, not re.error
+# ---------------------------------------------------------------------------
+
+class TestVectorizedRegexErrorHandling:
+    def test_malformed_pattern_raises_valueerror(self):
+        df = pd.DataFrame({"x": ["abc"]})
+        with pytest.raises(ValueError, match="Invalid regex pattern"):
+            f._vectorized_regex_sub(df, "[invalid", "")
+
+
+# ---------------------------------------------------------------------------
+# NIT-12: strip_bom strips at most one BOM
+# ---------------------------------------------------------------------------
+
+class TestStripBomSingleChar:
+    def test_strips_one_leading_bom(self):
+        assert strip_bom("hello") == "hello"
+
+    def test_does_not_strip_multiple_consecutive_boms(self):
+        # Per docstring: "at most one BOM". Second BOM stays so the
+        # caller can see something odd happened.
+        assert strip_bom("hello") == "hello"
+
+    def test_no_bom_unchanged(self):
+        assert strip_bom("hello") == "hello"
+
+    def test_non_string_passthrough(self):
+        assert strip_bom(None) is None  # type: ignore[arg-type]
+
+
+# ---------------------------------------------------------------------------
+# Smart title case — particle behavior at boundaries (regression / docs)
+# ---------------------------------------------------------------------------
+
+class TestSmartTitleCaseBoundaries:
+    def test_first_word_particle_capitalized(self):
+        # "a" at index 0 is a particle but must capitalize as the first
+        # word of a title.
+        assert smart_title_case("a story") == "A Story"
+
+    def test_last_word_particle_capitalized(self):
+        # "to" at the end is the last word; must capitalize.
+        assert smart_title_case("things to") == "Things To"
+
+    def test_mid_string_particles_lowercase(self):
+        assert smart_title_case("the cat in the hat") == "The Cat in the Hat"
+
+
+# ---------------------------------------------------------------------------
+# NIT-14: sentence_case dead branch removed — regression guard
+# ---------------------------------------------------------------------------
+
+class TestSentenceCaseUnchanged:
+    def test_basic(self):
+        assert sentence_case("hello. world.") == "Hello. World."
+
+    def test_open_paren_does_not_consume_trigger(self):
+        # The dead-branch removal didn't change behavior; this is a
+        # regression guard that opening punctuation still doesn't
+        # capitalize itself but doesn't reset the trigger either.
+        assert sentence_case('hello. "world"') == 'Hello. "World"'
+
+
+# ---------------------------------------------------------------------------
+# BUG-18: detect_header_row must not pick all-empty rows
+# ---------------------------------------------------------------------------
+
+class TestDetectHeaderRowEmptyRows:
+    def test_all_empty_first_row_skipped(self, tmp_path: Path):
+        # First row is all-empty — the header is on row 1.
+        p = tmp_path / "blank_first.csv"
+        p.write_text(",,\nname,email,phone\nalice,a@x.com,555\n")
+        assert detect_header_row(p) == 1
+
+    def test_pure_header_at_row_zero(self, tmp_path: Path):
+        p = tmp_path / "normal.csv"
+        p.write_text("name,email,phone\nalice,a@x.com,555\n")
+        assert detect_header_row(p) == 0
+
+
+# ---------------------------------------------------------------------------
+# BUG-20: config.from_dict must accept unknown fields (forward compat)
+# ---------------------------------------------------------------------------
+
+class TestConfigForwardCompat:
+    def test_extra_field_in_column_config_ignored(self, tmp_path: Path):
+        # Simulate a config file written by a future version with an
+        # extra ``priority`` field.
+        config_dict = {
+            "strategies": [{
+                "columns": [{
+                    "column": "email",
+                    "algorithm": "exact",
+                    "threshold": 100.0,
+                    "normalizer": None,
+                    "priority": 5,  # future field — must not crash
+                }],
+            }],
+            "survivor_rule": "first",
+            "merge": False,
+        }
+        loaded = DeduplicationConfig.from_dict(config_dict)
+        assert len(loaded.strategies) == 1
+        assert loaded.strategies[0].columns[0].column == "email"
+
+    def test_roundtrip_then_reload_with_extra(self, tmp_path: Path):
+        cfg = DeduplicationConfig(
+            strategies=[StrategyConfig(columns=[
+                ColumnStrategyConfig(column="email"),
+            ])],
+        )
+        path = tmp_path / "cfg.json"
+        cfg.to_file(path)
+        # Manually inject an unknown field to simulate forward-compat.
+        data = json.loads(path.read_text())
+        data["strategies"][0]["columns"][0]["future_thing"] = "abc"
+        path.write_text(json.dumps(data))
+        loaded = DeduplicationConfig.from_file(path)
+        assert loaded.strategies[0].columns[0].column == "email"
+
+
+# ---------------------------------------------------------------------------
+# BUG-22: mixed-case email detector must not flag all-None columns
+# ---------------------------------------------------------------------------
+
+class TestMixedCaseEmailFalsePositive:
+    def test_all_none_email_column_no_finding(self):
+        df = pd.DataFrame({
+            "email": [None, None, None],
+        })
+        findings = _detect_mixed_case_email(df)
+        assert findings == []
+
+    def test_real_mixed_case_still_flagged(self):
+        df = pd.DataFrame({
+            "email": ["Alice@X.com", "bob@y.com"],
+        })
+        findings = _detect_mixed_case_email(df)
+        assert len(findings) == 1
+        assert findings[0].column == "email"
+
+
+# ---------------------------------------------------------------------------
+# NIT-24: <NA> recognized as a null-like sentinel
+# ---------------------------------------------------------------------------
+
+class TestNullLikeIncludesPandasNA:
+    def test_pd_na_string_repr_recognized(self):
+        # str(pd.NA) → "<NA>" — when a DataFrame is loaded with
+        # keep_default_na=False, pandas NA values appear as the literal
+        # string "<NA>" and the analyzer should flag them.
+        assert "<na>" in _NULL_LIKE

tests/test_fixes_unit.py

@@ -0,0 +1,238 @@
+"""Isolated unit tests for individual fix functions in src.core.fixes.
+
+The integration tests at tests/test_normalize.py exercise these
+functions through the full analyze→fix pipeline. These tests pin each
+function's behavior in isolation so a regression surfaces close to the
+broken function rather than at the pipeline output.
+"""
+
+from __future__ import annotations
+
+import pandas as pd
+import pytest
+
+from src.core.fixes import (
+    clean_headers,
+    normalize_line_endings,
+    repair_mojibake,
+    strip_nbsp,
+    strip_zero_width,
+    trim_whitespace,
+)
+
+
+# ---------------------------------------------------------------------------
+# trim_whitespace
+# ---------------------------------------------------------------------------
+
+class TestTrimWhitespace:
+    def test_strips_leading_trailing(self):
+        df = pd.DataFrame({"x": ["  hello  ", " world "]})
+        out, changed = trim_whitespace(df)
+        assert list(out["x"]) == ["hello", "world"]
+        assert changed == 2
+
+    def test_collapses_internal_runs(self):
+        df = pd.DataFrame({"x": ["a   b   c"]})
+        out, _ = trim_whitespace(df)
+        assert out.loc[0, "x"] == "a b c"
+
+    def test_preserves_internal_in_structured(self):
+        # Phone-shaped strings keep internal spacing (often semantic).
+        df = pd.DataFrame({"x": ["(555) 123-4567"]})
+        out, changed = trim_whitespace(df)
+        assert out.loc[0, "x"] == "(555) 123-4567"
+        assert changed == 0
+
+    def test_empty_df(self):
+        df = pd.DataFrame({"x": []})
+        out, changed = trim_whitespace(df)
+        assert len(out) == 0
+        assert changed == 0
+
+    def test_no_string_columns(self):
+        df = pd.DataFrame({"n": [1, 2, 3]})
+        out, changed = trim_whitespace(df)
+        assert changed == 0
+        assert list(out["n"]) == [1, 2, 3]
+
+    def test_nan_preserved(self):
+        df = pd.DataFrame({"x": ["  ok  ", None]})
+        out, _ = trim_whitespace(df)
+        assert out.loc[0, "x"] == "ok"
+        # NaN/None passes through (becomes empty string after strip OR stays)
+        assert out.loc[1, "x"] is None or out.loc[1, "x"] == ""
+
+    def test_idempotent(self):
+        df = pd.DataFrame({"x": ["  hello  world  "]})
+        out1, _ = trim_whitespace(df)
+        out2, changed2 = trim_whitespace(out1)
+        assert changed2 == 0
+        assert list(out2["x"]) == list(out1["x"])
+
+
+# ---------------------------------------------------------------------------
+# strip_nbsp
+# ---------------------------------------------------------------------------
+
+class TestStripNbsp:
+    def test_replaces_nbsp_with_ascii_space(self):
+        df = pd.DataFrame({"x": ["a b"]})
+        out, changed = strip_nbsp(df)
+        assert out.loc[0, "x"] == "a b"
+        assert changed == 1
+
+    def test_no_change_when_clean(self):
+        df = pd.DataFrame({"x": ["a b c"]})
+        out, changed = strip_nbsp(df)
+        assert changed == 0
+
+    def test_other_unicode_spaces(self):
+        # Em space (U+2003), thin space (U+2009)
+        df = pd.DataFrame({"x": ["a b c"]})
+        out, _ = strip_nbsp(df)
+        assert out.loc[0, "x"] == "a b c"
+
+    def test_idempotent(self):
+        df = pd.DataFrame({"x": ["a  b"]})
+        out1, _ = strip_nbsp(df)
+        out2, changed2 = strip_nbsp(out1)
+        assert changed2 == 0
+
+
+# ---------------------------------------------------------------------------
+# strip_zero_width
+# ---------------------------------------------------------------------------
+
+class TestStripZeroWidth:
+    def test_removes_zero_width_space(self):
+        df = pd.DataFrame({"x": ["a​b"]})
+        out, changed = strip_zero_width(df)
+        assert out.loc[0, "x"] == "ab"
+        assert changed == 1
+
+    def test_removes_zero_width_joiner(self):
+        df = pd.DataFrame({"x": ["a‍b"]})
+        out, _ = strip_zero_width(df)
+        assert out.loc[0, "x"] == "ab"
+
+    def test_clean_passthrough(self):
+        df = pd.DataFrame({"x": ["clean"]})
+        out, changed = strip_zero_width(df)
+        assert changed == 0
+
+    def test_idempotent(self):
+        df = pd.DataFrame({"x": ["a​b‌c"]})
+        out1, _ = strip_zero_width(df)
+        out2, changed2 = strip_zero_width(out1)
+        assert changed2 == 0
+
+
+# ---------------------------------------------------------------------------
+# normalize_line_endings
+# ---------------------------------------------------------------------------
+
+class TestNormalizeLineEndings:
+    def test_crlf_to_lf(self):
+        df = pd.DataFrame({"x": ["line1\r\nline2"]})
+        out, changed = normalize_line_endings(df)
+        assert out.loc[0, "x"] == "line1\nline2"
+        assert changed == 1
+
+    def test_bare_cr_to_lf(self):
+        df = pd.DataFrame({"x": ["line1\rline2"]})
+        out, _ = normalize_line_endings(df)
+        assert out.loc[0, "x"] == "line1\nline2"
+
+    def test_already_lf_unchanged(self):
+        df = pd.DataFrame({"x": ["line1\nline2"]})
+        out, changed = normalize_line_endings(df)
+        assert changed == 0
+
+    def test_idempotent(self):
+        df = pd.DataFrame({"x": ["a\r\nb\rc"]})
+        out1, _ = normalize_line_endings(df)
+        out2, changed2 = normalize_line_endings(out1)
+        assert changed2 == 0
+
+
+# ---------------------------------------------------------------------------
+# clean_headers
+# ---------------------------------------------------------------------------
+
+class TestCleanHeaders:
+    def test_strips_bom_from_header(self):
+        df = pd.DataFrame({"name": [1], "email": [2]})
+        out, changed = clean_headers(df)
+        assert "name" in out.columns
+        assert "name" not in out.columns
+        assert changed >= 1
+
+    def test_strips_nbsp_from_header(self):
+        df = pd.DataFrame({"first name": [1]})
+        out, _ = clean_headers(df)
+        assert "first name" in out.columns
+
+    def test_strips_trailing_whitespace_from_header(self):
+        df = pd.DataFrame({"Email ": [1]})
+        out, _ = clean_headers(df)
+        assert "Email" in out.columns
+        assert "Email " not in out.columns
+
+    def test_non_string_label_preserved(self):
+        df = pd.DataFrame({0: [1], 1: [2]})
+        out, changed = clean_headers(df)
+        assert list(out.columns) == [0, 1]
+        assert changed == 0
+
+    def test_clean_headers_idempotent(self):
+        df = pd.DataFrame({"name": [1]})
+        out1, _ = clean_headers(df)
+        out2, changed2 = clean_headers(out1)
+        assert changed2 == 0
+        assert list(out2.columns) == list(out1.columns)
+
+
+# ---------------------------------------------------------------------------
+# repair_mojibake
+# ---------------------------------------------------------------------------
+
+_HAS_FTFY = True
+try:
+    import ftfy  # noqa: F401
+except ImportError:
+    _HAS_FTFY = False
+
+
+@pytest.mark.skipif(not _HAS_FTFY, reason="ftfy library not installed — fix is a no-op")
+class TestRepairMojibake:
+    def test_classic_cafe_repair(self):
+        df = pd.DataFrame({"x": ["café"]})  # café miscoded
+        out, changed = repair_mojibake(df)
+        assert out.loc[0, "x"] == "café"
+        assert changed == 1
+
+    def test_clean_text_unchanged(self):
+        df = pd.DataFrame({"x": ["café"]})
+        out, changed = repair_mojibake(df)
+        assert changed == 0
+
+    def test_no_string_columns(self):
+        df = pd.DataFrame({"n": [1, 2]})
+        out, changed = repair_mojibake(df)
+        assert changed == 0
+
+    def test_idempotent(self):
+        df = pd.DataFrame({"x": ["café"]})
+        out1, _ = repair_mojibake(df)
+        out2, changed2 = repair_mojibake(out1)
+        assert changed2 == 0
+
+
+class TestRepairMojibakeNoFtfy:
+    @pytest.mark.skipif(_HAS_FTFY, reason="ftfy installed — exercises the no-op path")
+    def test_returns_input_unchanged_without_ftfy(self):
+        df = pd.DataFrame({"x": ["café"]})
+        out, changed = repair_mojibake(df)
+        assert changed == 0
+        assert out.loc[0, "x"] == "café"

tests/test_io.py

@@ -261,3 +261,78 @@ class TestReadCsvRepaired:
         df, repair = read_csv_repaired(f)
         assert len(df) == 2
         assert repair.changed is False
+
+
+# ---------------------------------------------------------------------------
+# Round-trip integrity (audit GAP-19, GAP-21)
+# ---------------------------------------------------------------------------
+
+class TestRoundTrip:
+    def test_csv_roundtrip_preserves_values(self, tmp_path):
+        df = pd.DataFrame({
+            "id": ["1", "2", "3"],
+            "name": ["Alice", "Bob", "Carol"],
+            "amount": ["10.50", "20.25", "30.00"],
+        })
+        path = tmp_path / "rt.csv"
+        write_file(df, path)
+        loaded = read_file(path)
+        assert list(loaded.columns) == list(df.columns)
+        assert len(loaded) == len(df)
+        for col in df.columns:
+            assert list(loaded[col]) == list(df[col])
+
+    def test_tsv_roundtrip_via_extension(self, tmp_path):
+        df = pd.DataFrame({"a": ["1", "2"], "b": ["x", "y, z"]})
+        path = tmp_path / "rt.tsv"
+        write_file(df, path)
+        # Confirm tab is used and embedded comma in 'b' survives.
+        loaded = read_file(path)
+        assert list(loaded.columns) == ["a", "b"]
+        assert loaded.iloc[1]["b"] == "y, z"
+
+    def test_semicolon_roundtrip_via_explicit_delimiter(self, tmp_path):
+        df = pd.DataFrame({"a": ["1", "2"], "b": ["x", "y"]})
+        path = tmp_path / "rt.csv"
+        write_file(df, path, delimiter=";")
+        loaded = read_file(path)
+        assert list(loaded.columns) == ["a", "b"]
+        assert loaded.iloc[0]["a"] == "1"
+
+    def test_utf8_bom_non_ascii_roundtrip(self, tmp_path):
+        df = pd.DataFrame({"name": ["café", "naïve", "résumé"]})
+        path = tmp_path / "utf8.csv"
+        write_file(df, path)
+        loaded = read_file(path)
+        assert list(loaded["name"]) == ["café", "naïve", "résumé"]
+
+
+class TestExcelHeaderDetection:
+    def test_excel_with_metadata_rows(self, tmp_path):
+        from openpyxl import Workbook
+        wb = Workbook()
+        ws = wb.active
+        # Two leading blank rows + header + data.
+        ws.append(["Report generated 2024-01-15", None, None])
+        ws.append([None, None, None])
+        ws.append(["name", "email", "phone"])
+        ws.append(["alice", "a@x.com", "555-1234"])
+        ws.append(["bob", "b@x.com", "555-5678"])
+        path = tmp_path / "report.xlsx"
+        wb.save(path)
+        df = read_file(path)
+        # Auto-detected header row 2 → columns are name/email/phone
+        assert list(df.columns) == ["name", "email", "phone"]
+        assert len(df) == 2
+
+    def test_excel_normal_header_row_zero(self, tmp_path):
+        from openpyxl import Workbook
+        wb = Workbook()
+        ws = wb.active
+        ws.append(["name", "email"])
+        ws.append(["alice", "a@x.com"])
+        path = tmp_path / "normal.xlsx"
+        wb.save(path)
+        df = read_file(path)
+        assert list(df.columns) == ["name", "email"]
+        assert len(df) == 1

tests/test_normalizers.py

@@ -156,3 +156,51 @@ class TestGetNormalizer:
     def test_unknown_raises(self):
         with pytest.raises(ValueError):
             get_normalizer("unknown_type")
+
+
+# ---------------------------------------------------------------------------
+# Alignment with format_standardize: extension preservation, state codes,
+# particle handling. See audit GAPs 15/16/17.
+# ---------------------------------------------------------------------------
+
+class TestNormalizerAudit:
+    def test_phone_extension_preserved(self):
+        # Two records with different extensions must NOT normalize to
+        # the same key — they're different people at the same business.
+        a = normalize_phone("+15551234567 ext 100")
+        b = normalize_phone("+15551234567 ext 200")
+        assert a != b
+        assert a == "+15551234567;ext=100"
+
+    def test_phone_no_extension_unchanged(self):
+        assert normalize_phone("+15551234567") == "+15551234567"
+
+    def test_address_state_name_to_code(self):
+        # "California" and "CA" produce the same matching key.
+        a = normalize_address("123 Main St, Los Angeles, California 90001")
+        b = normalize_address("123 Main St, Los Angeles, CA 90001")
+        assert a == b
+
+    def test_address_multiword_state_name(self):
+        a = normalize_address("100 Beacon St, Boston, Massachusetts 02101")
+        b = normalize_address("100 Beacon St, Boston, MA 02101")
+        assert a == b
+
+    def test_address_does_not_butcher_city_named_after_state(self):
+        # "New York" appearing as a city should still fold to "ny" —
+        # this is intentional for matching keys (we want ``New York, NY``
+        # and ``NY, NY`` to be the same record) even though the
+        # standardizer (display) would preserve the city name.
+        out = normalize_address("123 Main St, New York, NY 10001")
+        assert "ny" in out
+
+    def test_name_particle_dropped(self):
+        # "Charles de Gaulle" and "Charles Gaulle" produce the same key.
+        assert normalize_name("Charles de Gaulle") == normalize_name("Charles Gaulle")
+
+    def test_name_van_dropped(self):
+        assert normalize_name("Vincent van Gogh") == normalize_name("Vincent Gogh")
+
+    def test_name_particle_idempotent(self):
+        out = normalize_name("Vincent van Gogh")
+        assert normalize_name(out) == out

tests/test_text_clean.py

@@ -537,8 +537,10 @@ class TestVisualizeHidden:
 
     def test_non_string_passthrough(self):
         from src.core.text_clean import visualize_hidden_text, visualize_hidden_html
+        # Both functions now consistently pass non-strings through
+        # unchanged (audit NIT-13).
         assert visualize_hidden_text(None) is None  # type: ignore[arg-type]
-        assert visualize_hidden_html(None) == ""
+        assert visualize_hidden_html(None) is None  # type: ignore[arg-type]
     def test_html_marks_leading_trailing_ascii_space(self):
         from src.core.text_clean import visualize_hidden_html
         out = visualize_hidden_html("  Alice  ", mark_outer_whitespace=True)