fix: cross-tool audit findings + alignment with format standardizer
Closes 12 bugs and 8 gaps surfaced by parallel audits across all core modules, plus aligns the dedup-side normalizers with the new format_standardize behavior where they had silently diverged. Bugs (data integrity / correctness): - dedup: NaN/None values matched as duplicates because str(None)='None'. Two rows with missing email silently merged. - dedup: removed_df had 0 columns when nothing was removed; downstream code expecting matching schema broke. Now preserves column shape. - dedup: ColumnMatchStrategy threshold accepted any value; out-of-range silently broke matching. Validated to [0, 100] in __post_init__. - dedup: strategy referencing a missing column was silently skipped. Now raises ValueError listing available columns. - fixes: replace_null_sentinels crashed on non-string sentinels (int/None from JSON payload). Coerced to str. - fixes: _vectorized_regex_sub raised raw re.error on bad patterns. Now wraps as ValueError with clear message. - io: detect_header_row mis-identified all-empty and metadata-only rows as headers (all([]) is True). Now requires ≥2 non-empty cells. - config: from_dict crashed when JSON had unknown fields, breaking forward compat. Now filters to known fields. - analyze: mixed-case email detector flagged all-None columns because str(None)='None' contains both N and one. Now drops NaN before stringify. New features and gap closures: - io: _detect_excel_header_row mirrors detect_header_row for Excel via openpyxl read-only; _read_excel uses it when header_row=None. - io: write_file gains delimiter + encoding params; .tsv extension defaults to tab. - normalizers: normalize_phone preserves extensions as ;ext=N suffix. - normalizers: normalize_address folds spelled-out US state names to 2-letter codes (California ≡ CA). - normalizers: normalize_name drops surname particles (van, de, von) so "Charles de Gaulle" ≡ "Charles Gaulle" for matching. - analyze: new _detect_inconsistent_date_format detector flags columns with mixed ISO/US/EU date shapes; routes to format standardizer. - analyze: _NULL_LIKE recognizes "<na>" (pd.NA repr). - analyze: duplicate-row finding renamed count → n_extra (rows that would actually be removed) with clarified description. - dedup: group_confidence no longer falsely 100.0 when transitive group members lack a recorded direct pair; falls back to 100.0 only when truly no pairs were observed. - dedup: MatchResult / DeduplicationResult docstrings clarify that row_indices refer to the input frame's positional index (output index is reset). - text_clean: visualize_hidden_html(None) now returns None (matches visualize_hidden_text); strip_bom strips at most one BOM per call; sentence_case dead elif branch removed. Tests: - tests/test_audit_fixes.py — 28 regression tests, one or more per numbered finding, named after BUG/GAP/NIT tags so future readers can trace each test back to its audit. - tests/test_fixes_unit.py — 26 isolated unit tests for previously integration-only fix functions (trim_whitespace, strip_nbsp, strip_zero_width, normalize_line_endings, clean_headers, repair_mojibake — last skipped if ftfy unavailable). - tests/test_io.py — adds CSV / TSV / semicolon / UTF-8-BOM round-trip tests + Excel auto-header-detection tests. - tests/test_normalizers.py — adds 8 tests for the alignment work above (phone extension, state names, particles). Adds .claude/ to .gitignore (agent worktrees + local settings). Full project suite: 1197 passed, 4 skipped, 17 xfailed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -261,3 +261,78 @@ class TestReadCsvRepaired:
|
||||
df, repair = read_csv_repaired(f)
|
||||
assert len(df) == 2
|
||||
assert repair.changed is False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Round-trip integrity (audit GAP-19, GAP-21)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestRoundTrip:
|
||||
def test_csv_roundtrip_preserves_values(self, tmp_path):
|
||||
df = pd.DataFrame({
|
||||
"id": ["1", "2", "3"],
|
||||
"name": ["Alice", "Bob", "Carol"],
|
||||
"amount": ["10.50", "20.25", "30.00"],
|
||||
})
|
||||
path = tmp_path / "rt.csv"
|
||||
write_file(df, path)
|
||||
loaded = read_file(path)
|
||||
assert list(loaded.columns) == list(df.columns)
|
||||
assert len(loaded) == len(df)
|
||||
for col in df.columns:
|
||||
assert list(loaded[col]) == list(df[col])
|
||||
|
||||
def test_tsv_roundtrip_via_extension(self, tmp_path):
|
||||
df = pd.DataFrame({"a": ["1", "2"], "b": ["x", "y, z"]})
|
||||
path = tmp_path / "rt.tsv"
|
||||
write_file(df, path)
|
||||
# Confirm tab is used and embedded comma in 'b' survives.
|
||||
loaded = read_file(path)
|
||||
assert list(loaded.columns) == ["a", "b"]
|
||||
assert loaded.iloc[1]["b"] == "y, z"
|
||||
|
||||
def test_semicolon_roundtrip_via_explicit_delimiter(self, tmp_path):
|
||||
df = pd.DataFrame({"a": ["1", "2"], "b": ["x", "y"]})
|
||||
path = tmp_path / "rt.csv"
|
||||
write_file(df, path, delimiter=";")
|
||||
loaded = read_file(path)
|
||||
assert list(loaded.columns) == ["a", "b"]
|
||||
assert loaded.iloc[0]["a"] == "1"
|
||||
|
||||
def test_utf8_bom_non_ascii_roundtrip(self, tmp_path):
|
||||
df = pd.DataFrame({"name": ["café", "naïve", "résumé"]})
|
||||
path = tmp_path / "utf8.csv"
|
||||
write_file(df, path)
|
||||
loaded = read_file(path)
|
||||
assert list(loaded["name"]) == ["café", "naïve", "résumé"]
|
||||
|
||||
|
||||
class TestExcelHeaderDetection:
|
||||
def test_excel_with_metadata_rows(self, tmp_path):
|
||||
from openpyxl import Workbook
|
||||
wb = Workbook()
|
||||
ws = wb.active
|
||||
# Two leading blank rows + header + data.
|
||||
ws.append(["Report generated 2024-01-15", None, None])
|
||||
ws.append([None, None, None])
|
||||
ws.append(["name", "email", "phone"])
|
||||
ws.append(["alice", "a@x.com", "555-1234"])
|
||||
ws.append(["bob", "b@x.com", "555-5678"])
|
||||
path = tmp_path / "report.xlsx"
|
||||
wb.save(path)
|
||||
df = read_file(path)
|
||||
# Auto-detected header row 2 → columns are name/email/phone
|
||||
assert list(df.columns) == ["name", "email", "phone"]
|
||||
assert len(df) == 2
|
||||
|
||||
def test_excel_normal_header_row_zero(self, tmp_path):
|
||||
from openpyxl import Workbook
|
||||
wb = Workbook()
|
||||
ws = wb.active
|
||||
ws.append(["name", "email"])
|
||||
ws.append(["alice", "a@x.com"])
|
||||
path = tmp_path / "normal.xlsx"
|
||||
wb.save(path)
|
||||
df = read_file(path)
|
||||
assert list(df.columns) == ["name", "email"]
|
||||
assert len(df) == 1
|
||||
|
||||
Reference in New Issue
Block a user