refactor: dedup, consolidate, harden public APIs across core modules

Closes 16 high-value findings from a parallel cross-module review.

Refactors:
- New src/core/_constants.py centralizes USPS street-suffix
  abbreviations, US state names, and 2-letter postal codes — one source
  of truth for both normalize_address (matching keys) and
  standardize_address (display formatting). Eliminates ~80 lines of
  duplicated dicts across normalizers.py and format_standardize.py.
- format_standardize.py: collapse 4 identical nested _err() helpers
  into one shared _err_or_passthrough() module function; drop a dead
  duplicate `return _err("not a phone number")` branch in
  standardize_phone.
- format_standardize.py: precompile per-locale month-name regexes
  (_MONTH_LOCALE_PATTERNS) and per-state-name regexes
  (_STATE_NAME_PATTERNS) at import time — they were rebuilt on every
  cell, a measurable hot path on million-row inputs.
- dedup.py: extract _is_missing(value) helper; one definition of
  "this cell is None / NaN / pd.NA" instead of two.
- fixes.py: extract _is_string_column(ser) helper; one dtype check
  instead of three duplicates across _apply_to_strings,
  _vectorized_translate, _vectorized_regex_sub.

Production-readiness:
- format_standardize.standardize_dataframe now logs a warning when
  more than 10% of typed cells are unparseable — surfaces the
  silently-broken-pipeline failure mode.
- StandardizeOptions.from_dict validates date_order / phone_format /
  currency_decimal / name_case / boolean_style / *_error_policy
  enum values up front, with a clear error message instead of a deep
  crash inside the per-cell function.
- StandardizeOptions.from_file and DeduplicationConfig.from_file wrap
  read + json.loads with descriptive OSError / ValueError messages
  including the file path.
- standardize_date(month_locales=...) validates locale codes against
  the available set instead of silently passing through unknown ones.
- io.read_file rejects chunk_size <= 0 (was silently failing inside
  pandas) and logs the resolved suffix + chunk_size at info level so
  data-pipeline runs are debuggable.
- io.read_file's FileNotFoundError gains explanatory context.
- io.write_file, text_clean.clean_dataframe, and dedup.deduplicate
  now reject non-DataFrame inputs with clear TypeError instead of
  cryptic pandas tracebacks downstream.
- dedup.deduplicate validates that survivor_rule=KEEP_MOST_RECENT has
  a usable date_column up front; the helper _select_survivor now
  raises (instead of silently falling back to keep_first) when called
  directly with bad arguments.
- dedup.deduplicate gains a structured no-op return when strategies
  is empty after auto-detection — preserves schema instead of crashing.
- analyze._detect_inconsistent_date_format narrows its bare except to
  (TypeError, ValueError) and logs a debug line so genuine bugs don't
  hide behind silent skip.

Tests:
- tests/test_audit_fixes.py grows by 11 cases covering the new
  validation paths (chunk_size, DataFrame guards, KEEP_MOST_RECENT
  date_column, enum validation, locale validation, JSON error wrapping).

Full project suite: 1208 passed, 4 skipped, 17 xfailed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-01 02:23:09 +00:00
parent b23a27d4e3
commit 2eece6467d
10 changed files with 457 additions and 231 deletions

View File

@@ -301,3 +301,76 @@ class TestNullLikeIncludesPandasNA:
# keep_default_na=False, pandas NA values appear as the literal
# string "<NA>" and the analyzer should flag them.
assert "<na>" in _NULL_LIKE
# ---------------------------------------------------------------------------
# Production-readiness review (refactor pass)
# ---------------------------------------------------------------------------
class TestProductionReadyValidation:
def test_chunk_size_must_be_positive(self, tmp_path: Path):
from src.core.io import read_file
f = tmp_path / "tiny.csv"
f.write_text("a,b\n1,2\n")
with pytest.raises(ValueError, match="chunk_size must be positive"):
list(read_file(f, chunk_size=0))
with pytest.raises(ValueError, match="chunk_size must be positive"):
list(read_file(f, chunk_size=-5))
def test_write_file_rejects_non_dataframe(self, tmp_path: Path):
from src.core.io import write_file
with pytest.raises(TypeError, match="requires a pandas DataFrame"):
write_file({"a": [1]}, tmp_path / "out.csv") # type: ignore[arg-type]
def test_clean_dataframe_rejects_non_dataframe(self):
from src.core.text_clean import clean_dataframe
with pytest.raises(TypeError, match="requires a pandas DataFrame"):
clean_dataframe([{"a": 1}]) # type: ignore[arg-type]
def test_deduplicate_rejects_non_dataframe(self):
from src.core.dedup import deduplicate
with pytest.raises(TypeError, match="requires a pandas DataFrame"):
deduplicate({"x": [1]}) # type: ignore[arg-type]
def test_keep_most_recent_requires_date_column(self):
from src.core.dedup import deduplicate, SurvivorRule
df = pd.DataFrame({"name": ["a", "b"]})
with pytest.raises(ValueError, match="date_column"):
deduplicate(df, survivor_rule=SurvivorRule.KEEP_MOST_RECENT)
def test_keep_most_recent_with_unknown_date_column(self):
from src.core.dedup import deduplicate, SurvivorRule
df = pd.DataFrame({"name": ["a", "b"]})
with pytest.raises(ValueError, match="not found in input"):
deduplicate(
df,
survivor_rule=SurvivorRule.KEEP_MOST_RECENT,
date_column="created_at",
)
def test_standardize_options_invalid_enum(self):
from src.core.format_standardize import StandardizeOptions
with pytest.raises(ValueError, match="Invalid date_order"):
StandardizeOptions.from_dict({"date_order": "XYZ"})
def test_standardize_options_invalid_field_type(self):
from src.core.format_standardize import StandardizeOptions
with pytest.raises(ValueError, match="Invalid field type"):
StandardizeOptions.from_dict({"column_types": {"x": "made_up"}})
def test_month_locale_validation(self):
from src.core.format_standardize import standardize_date
with pytest.raises(ValueError, match="Unknown month locale"):
standardize_date("15 januar 2024", month_locales=["DE"]) # uppercase typo
def test_config_from_file_missing(self, tmp_path: Path):
from src.core.config import DeduplicationConfig
with pytest.raises(OSError, match="Could not read"):
DeduplicationConfig.from_file(tmp_path / "missing.json")
def test_config_from_file_bad_json(self, tmp_path: Path):
from src.core.config import DeduplicationConfig
path = tmp_path / "bad.json"
path.write_text("{not: valid json")
with pytest.raises(ValueError, match="Invalid JSON"):
DeduplicationConfig.from_file(path)