datatools-dev/test-cases/format-cleaner-corpus/FORMATS-CASES.md

FORMATS-CASES.md - 03_format_standardizer.py Test Corpus

Version: 1.0 Last updated: April 30, 2026 Companion to: TEST-CASES.md (cleaning rules), QUOTE-CASES.md (parser robustness), ENCODINGS-CASES.md (I/O layer).

This corpus tests 03_format_standardizer.py, which owns "what's there but in the wrong format." Six domains: dates, phones, emails, addresses, names, currencies. Plus a cross-domain integration fixture.

---

0. Scope clarifications you should read first

Three issues to surface before the per-domain sections, because they affect what tests are valid in the first place.

0.1 Email scope conflict with TECHNICAL.md

USER-GUIDE.md Section 2 lists 03's purpose as "dates, currencies, names, phone numbers, addresses." TECHNICAL.md Section 10.1 item 8 puts email normalization inside 01_deduplicator's Tier 1 spec. Email appears in neither place as part of 03.

This corpus tests email normalization as if it lives in 03. The reasoning: 03 is "format standardizer" and email is a format like any other. Putting it in 01 means there's no public API for the buyer to normalize emails outside of running dedup, which is a weird ergonomic for the GUI ("To clean my emails I have to run the deduplicator?"). Better factoring: 03 owns email normalization as a public operation; 01 calls into the same core/ function for matching.

If you disagree, fixture 26_format_emails.csv and its expected output drop out cleanly without affecting the other five domains. If you agree, update USER-GUIDE.md Section 2 and TECHNICAL.md Section 7's per-bundle technical notes.

0.2 Schema preservation rule (TECHNICAL.md Section 9 invariant)

03 changes cell content, never schema. Row count, column count, column order all unchanged. This rules out a few tempting designs:

• Currency normalization that splits $1,234.56 into separate amount and currency columns — rejected. Output stays in one cell.
• Address normalization that splits a single-line address into structured street/city/state/zip columns — rejected. Output stays in one cell.
• Phone normalization that splits phone + extension into two columns — rejected. Extension goes inline as ;ext=123 (RFC 3966 syntax).

If you want structured output, that's a different script (a parser, not a standardizer).

0.3 Boundary with neighboring scripts

If the cell is...	Owner	03's behavior
Empty string	04 (missing values)	Pass through unchanged. Don't decide if it means "missing."
Whitespace-only	02 (text cleaner)	Should already be empty by the time 03 sees it. If not (CLI user skipped 02), trim defensively.
Statistically extreme but format-valid (date in year 1700, phone with 10 zeros)	06 (outliers)	Format-normalize anyway. Don't flag unusual values.
Format-invalid (Feb 30, missing @, letters in numeric)	03	Emit error sentinel <error: <reason>>.
Already correctly formatted	03	Pass through. Idempotency required.

---

1. Default configuration

Tests assume the defaults below. Per-flag deviations are called out per case.

Setting	Default	Notes
--date-format	ISO 8601	YYYY-MM-DD for dates, YYYY-MM-DDTHH:MM:SS[+ZZ:ZZ] for datetimes
--locale	auto-detect	Per-column. Falls back to error if column has no disambiguating value
--two-digit-year-cutoff	69	Python default: years 00-68 → 2000-2068, 69-99 → 1969-1999
--phone-format	E.164	+<country><digits>, extensions via ;ext=
--default-country	US	Used for phones with no country code
--gmail-canonical	off	Strip Gmail dots and +tags. Destructive, opt-in
--expand-abbrev	off	Expand St → Street etc. USPS abbreviation is the default
--name-conservative	on	Title-case only ALL CAPS or all-lowercase input
--currency-locale	auto-detect	Per-column. Same fallback as date locale
--error-policy	sentinel	Errors written as <error: reason>. Alternative: raise, skip-row
--columns	all	All text columns processed; --columns date,phone restricts

Idempotency requirement: format(format(x)) == format(x) for every cell. Already-clean input passes through unchanged.

---

2. Test corpus index

File	Domain	Cases	Expected outputs
24_format_dates.csv	Dates	45	Single column
25_format_phones.csv	Phones	31	Single column
26_format_emails.csv	Emails	31	Two columns (default + gmail-canonical)
27_format_addresses.csv	Addresses	31	Two columns (default + expand-abbrev)
28_format_names.csv	Names	34	Single column
29_format_currencies.csv	Currencies	27	Single column
30_format_integration.csv	Cross-domain	5	Multi-column (full row)

All input fixtures share the schema case_id, category, description, input (except integration, which has the full multi-column shape). Expected output files key by case_id for diff-by-join testing.

---

3. DATES (24_format_dates.csv)

3.1 Use cases by buyer persona

• Shopify: Order export dates joined against manual entries that used a different format. Bookkeeping reports needing consistent date format for sorting.
• Bookkeeper: Bank export reconciliation across multiple banks, each using its own date convention. Tax reports requiring consistent year-month grouping.
• Freelancer: Client data dumps where the date column is in whatever format the client's locale or software produces.
• Marketing agency: Campaign performance data joined across platforms (Google Ads, Facebook Ads, Mailchimp) that all use different date formats.

3.2 Test categories

Category	Cases	What it tests
iso	FD01-FD06	ISO 8601 baseline. Already-clean and minor variants (Z vs offset, T vs space)
us	FD07-FD10	M/D/Y format with 2-digit and 4-digit years. Includes one unambiguous case (day > 12)
eu	FD11-FD15	D/M/Y format with various separators. Includes one unambiguous case
longform	FD16-FD21	Month-name formats (full, abbreviated, with weekday, all caps)
excel	FD22-FD23	Excel serial numbers (45306 = 2024-01-15). Critical: Excel CSV exports often have date columns leak through as numbers
unix	FD24-FD25	Unix timestamps in seconds and milliseconds
partial	FD26-FD29	Year-month, quarter, year-only. Coarser-than-day precision
edge	FD30-FD40	Two-digit year ambiguity, leap day validity, Excel 1900 leap year bug, invalid dates, dates buried in other text
locale	FD41-FD42	French and German month names
timezone	FD43-FD44	Named time zones, DST transitions
padding	FD45	Already-clean idempotency check

3.3 Critical policy decisions

Locale ambiguity (M/D/Y vs D/M/Y): Per-column inspection. The cleaner scans all values in the column; if any value has day > 12, locale is unambiguously D/M/Y; if any has month > 12 (impossible in M/D/Y), locale is unambiguously D/M/Y. If nothing disambiguates, error out and require --locale us|eu. Do not silently guess. Fixture row FD13 (15/01/2024) is ambiguous in isolation; FD14 (30/05/2024) makes the column unambiguously D/M/Y; in a real column containing both, FD13 resolves to 2024-01-15.

Two-digit year cutoff: Python's default of 69 (years 00-68 → 2000s, 69-99 → 1969-1999). FD30 is 1/15/69 and resolves to 1969-01-15. This is opinionated and frequently wrong for birth-year columns. Document the flag clearly; the buyer cleaning customer DOB data needs to override.

Excel serial dates (FD22, FD23): Detection heuristic — column header contains "date", or all values are integers/floats in range 25569–73050 (Jan 1 1970 to Jan 1 2099 in Excel serial). Outside that heuristic the cleaner can't distinguish a date serial from any other number.

Excel 1900 leap year bug (FD33): Excel claims 1900-02-29 exists; it doesn't. Detect and emit error. Don't silently accept and roll over to March 1.

Localized month names (FD41, FD42): Default cleaner ships with English month names. French/German/Spanish/etc. require a locale dictionary. Either ship one (adds size) or document the limitation. Recommendation: ship English + opt-in --month-locale=fr|de|es for the others. This corpus tests as if French and German are supported.

Time zones (FD43, FD44): Named zones (EST, PST) resolve to fixed offsets, NOT dynamically interpreted with DST rules. EST → -05:00 always. If buyers need DST-aware handling, that's a 04-bundle (out of scope) or an opt-in pyzoneinfo flag.

3.4 Edge case: dates buried in text (FD36, FD37)

Date: 2024-01-15 and 2024-01-15 (verified) extract to 2024-01-15. The cleaner uses regex extraction for date-shaped substrings before parsing. Risk: false positives from random number sequences. Mitigation: require an unambiguous date pattern (4-digit year + valid month + valid day with explicit separator).

3.5 What's not tested

• Calendar systems other than Gregorian (Hijri, Hebrew, Japanese era). Out of scope.
• Recurring date strings (every 1st of month). Not a date.
• Date ranges (2024-01-01 to 2024-01-15). Out of scope; would require a different cell semantic.
• Sub-millisecond precision. Pandas/datetime tolerate but aren't tested here.

---

4. PHONES (25_format_phones.csv)

4.1 Use cases by buyer persona

• Shopify: Customer phone list normalization before Klaviyo/Mailchimp import. SMS campaigns require E.164.
• Bookkeeper: Vendor phone deduplication where same vendor has multiple format variants in QuickBooks vs. spreadsheets.
• Freelancer: Lead lists from clients in arbitrary formats.
• Marketing agency: Multi-platform audience reconciliation; ad platforms increasingly require E.164 for matching.

4.2 Test categories

Category	Cases	What it tests
us	FP01-FP08	Common US format variants — plain digits, parens-dash, dots, spaces, country code prefixes
ext	FP09-FP11	Extensions in three syntactic forms (ext, x, #)
vanity	FP12-FP13	Letter-to-digit conversion (1-800-FLOWERS)
intl	FP14-FP19	UK, Germany, France, Japan, Australia
e164	FP20	Already-E.164 idempotency
edge	FP21-FP31	Insufficient/excess digits, placeholders, multiple numbers per cell, NBSP, smart-quote contamination

4.3 Critical policy decisions

Default output: E.164 (+<country><digits>). Universal storage format. Reverses cleanly to any presentation format if the buyer wants display formatting later.

Default country: US, configurable via --default-country=GB|DE|.... For mixed-country columns, cleaner needs explicit country detection per-row, which is hard without context. Real-world advice for the buyer: split phone columns by country before normalizing.

Vanity numbers (FP12, FP13): Letters convert via standard phone keypad: 2=ABC, 3=DEF, ..., 9=WXYZ. FLOWERS → 3569377. Loses some information (you can't reverse 3569377 to FLOWERS). Acceptable tradeoff for storage normalization.

Trunk prefix dropping: UK domestic format 020 7946 0958 (FP15) has a leading 0 that's a domestic trunk prefix, not part of the actual number. E.164 strips it: +442079460958. Same logic for other countries with trunk prefixes.

Placeholders (FP23, FP24): All-zeros 000-000-0000 and all-nines 999-999-9999 are conventional "no phone" sentinels in some CRMs. Emit error rather than silently producing a syntactically valid E.164 that's semantically meaningless. Tradeoff: a real number that happens to be 999-999-9999 (which doesn't exist in NANP, by the way; 999 is reserved) would error too. Acceptable.

Multiple numbers (FP25): Cell containing 555-123-4567 / 555-987-6543. Don't silently pick one; emit error and tell the user to split first. Splitting is a structural change, not a format change, so it belongs upstream of 03.

NBSP and smart-quote contamination (FP27, FP31): Should not reach 03 if 02 ran first. Defensive cleanup is fine; emit a debug log noting the upstream pollution.

4.4 What's not tested

• SMS-vs-voice number distinction.
• Carrier lookup. Out of scope; would require a paid service.
• Number portability validation.
• Toll-free number recognition (888, 877, 866, 855, 844, 833) beyond accepting them as valid digits.

---

5. EMAILS (26_format_emails.csv) — see Section 0.1 for scope caveat

5.1 Use cases by buyer persona

• Shopify: Customer list cleanup before email-marketing platform import (every duplicate costs money on per-contact pricing). Pre-flight check on order export before re-engagement campaigns.
• Bookkeeper: Vendor email list consolidation.
• Freelancer: Client communication list normalization.
• Marketing agency: List hygiene across multiple lead sources before campaign send.

5.2 Test categories

Category	Cases	What it tests
basic	FE01-FE04	Plain ASCII, mixed case, whitespace
displayname	FE05-FE07	RFC display-name forms Name <email>, with and without quotes
prefix	FE08-FE09	mailto: prefix
gmail	FE10-FE14	Gmail-specific dot-equivalence and +tag handling. Includes negative cases (non-Gmail domains) that must NOT be touched
idn	FE15-FE16	Internationalized domain names; Unicode in local part
trailing	FE17-FE20	Punctuation contamination from copy-paste contexts
smartquote	FE21	Word-paste damage
invalid	FE22-FE26	Missing @, double @, multiple @, internal whitespace, no TLD
multiple	FE27-FE28	Multiple emails in one cell
edge	FE29-FE31	Empty, whitespace-only, already-perfect

5.3 Critical policy decisions

Default behavior: lowercase, trim, strip mailto:, strip wrapping <>, extract from Display Name <email> form. Does NOT strip Gmail dots or +tags by default. Those normalizations are destructive (alice and a.l.i.c.e aren't the same email per RFC; only Gmail's specific provider policy treats them as equivalent).

Aggressive mode (--gmail-canonical): Strip dots and +tags for @gmail.com only. Preserve them for all other domains, even if those domains have similar policies (some custom Google Workspace domains, some other providers). Don't second-guess provider policy.

FE13 and FE14 are critical negative tests: a non-Gmail domain with dots or +tag must NOT be touched even in --gmail-canonical mode. Many cleaners get this wrong — they apply Gmail's policy to all domains, which corrupts data.

IDN handling (FE15, FE16): Don't punycode-convert by default. Buyers who need ASCII-only output for legacy systems can opt in via --punycode. Default is to preserve Unicode in domain and local parts.

Display-name extraction (FE05, FE06): Drop the display name. The cleaner extracts the email and discards Alice Smith. Tradeoff: information loss. Alternative would be to preserve display name in a separate column, but that violates schema preservation (Section 0.2). Buyers who want to keep display names should split the column upstream.

Multiple emails per cell (FE27, FE28): Error, don't pick one. Same rationale as multiple phones.

5.4 What's not tested

• Email syntax validation per full RFC 5321/5322 (which permits all sorts of legitimately weird inputs like quoted-string locals). The cleaner uses a "good enough for 99% of real data" regex, not a full RFC parser.
• Disposable-email-domain detection. Out of scope for format cleaning; that's data quality.
• DNS / MX validation. Out of scope; requires network access.
• Email-address-as-username (where domain is a hostname not an internet domain). Errors as TLD-less.

---

6. ADDRESSES (27_format_addresses.csv)

6.1 Use cases by buyer persona

• Shopify: Customer address normalization for shipping label generation; reduces failed deliveries.
• Bookkeeper: Vendor master record cleanup; consistent format for bookkeeping software import.
• Freelancer: Client address book consolidation.
• Marketing agency: Direct mail audience cleanup.

6.2 Test categories

Category	Cases	What it tests
clean	FA01	Already-USPS-formatted idempotency
case	FA02-FA04	All-caps, all-lowercase, mixed-case (preserve)
abbrev	FA05-FA08	Street type expansion/abbreviation, periods after abbreviations
directional	FA09-FA11	North/N, NORTH/N, NE compounds
unit	FA12-FA14	Apartment/Apt, # / Apt, Suite/Ste
state	FA15-FA16	State name → 2-letter code
zip	FA17-FA18	ZIP+4, leading-zero ZIPs (Massachusetts 02xxx)
multiline	FA19	\n-separated address fields
pobox	FA20-FA22	Post Office Box variants
housenum	FA23-FA25	Letter suffix, hyphen, half-number
non_us	FA26-FA28	UK, Canada, Japan (minimal handling)
edge	FA29-FA31	Empty, partial, trailing comma

6.3 Critical policy decisions

US-first scope: USPS abbreviations and state codes are the default. International addresses get whitespace + capitalization only. Document this clearly; buyers with significant non-US data should expect format drift.

USPS abbreviations as the default (St, Ave, Blvd) rather than spelled-out forms. Reasoning: USPS recommends abbreviations; most CRMs expect them; they save space in tabular display. The --expand-abbrev flag inverts this for buyers whose downstream system requires full forms.

Multi-line collapse (FA19): 123 Main St\nApt 4B\nNew York, NY 10001 becomes 123 Main St, Apt 4B, New York, NY 10001. Consistent comma-separated single-line format. Reverse direction not supported — the cleaner doesn't take a single-line address and split into multi-line (that's structural).

State expansion vs abbreviation (FA15, FA16): Default is 2-letter code (NY). The --expand-abbrev flag expands to full state name. Note: this is the OPPOSITE direction from street type abbreviations. State codes are universally expected in tabular data; full state names are only preferred in some downstream systems' "pretty" formats.

ZIP leading zeros (FA18): If the column is already a ZIP-shaped string with leading zeros, preserve them. Cannot restore lost leading zeros — Excel-stripped 2101 (Massachusetts) cannot be confidently recovered to 02101 because 2101 could legitimately be 2101 (Idaho). Mention this as a known limitation; recommend the buyer fix at the source.

Canada handling (FA27): Canadian addresses use the same street-type conventions as US, so St → St works. Postal code format is preserved as-is.

Japan / non-Western (FA28): Field order is reversed (postal code first, then large-to-small geography). Default cleaner doesn't try to restructure; minimal handling only.

6.4 What's not tested

• Address verification against USPS database. Out of scope; would require a paid service or local USPS data.
• Geocoding to lat/long. Out of scope.
• Unit number parsing for buildings with non-standard nomenclatures.
• Military addresses (APO, FPO, DPO) beyond accepting them.
• Rural Route, Highway Contract, General Delivery formats.

---

7. NAMES (28_format_names.csv)

7.1 Use cases by buyer persona

• Shopify: Customer list display normalization. ALL-CAPS imports from older systems become readable.
• Bookkeeper: Vendor name consistency across QuickBooks and spreadsheets.
• Freelancer: Client list capitalization cleanup.
• Marketing agency: First-name personalization in email campaigns (Hi alice vs Hi Alice).

7.2 Test categories

Category	Cases	What it tests
case	FN01-FN04	All-caps, all-lowercase, already-correct, random-case
scots	FN05-FN08	Mc and Mac prefixes
irish	FN09-FN11	O' prefix
hyphen	FN12-FN13	Hyphenated names
particle	FN14-FN17	von, van, de, da (Germanic, Dutch, French, Italian)
title	FN18-FN20	Mr, Dr, Prof
suffix	FN21-FN23	Jr, III, PhD
comma	FN24-FN26	"Last, First" reversal to "First Last"
initial	FN27-FN28	Middle initial, multi-initial
nonlatin	FN29-FN31	Korean, Japanese, Russian (preserve)
edge	FN32-FN34	Single name, empty, whitespace-only

7.3 Critical policy decisions

Conservative by default: Title-case ONLY when input is ALL CAPS or all lowercase. Mixed-case input is preserved as-is (FN04: aLiCe SmItH → aLiCe SmItH). Reasoning: people have idiosyncratic spellings (danah boyd, bell hooks) that the cleaner should never overwrite. If the buyer wants aggressive title-casing, that's --name-aggressive.

Mc vs Mac (FN05-FN08): Default convention is McDonald (cap after Mc) and MacDonald (cap after Mac). Some Mac-prefixed names should be Macdonald (cap only on Mac). Without a names dictionary, the cleaner can't distinguish. Default to capitalizing — produces MacDonald for ambiguous cases. Buyers with significant Scottish/Irish customer bases may need a custom override list.

Particles (FN14-FN17): Particles like von, van, de, da stay lowercase. This is the convention for people with surnames containing these words (Vincent van Gogh, Charles de Gaulle). Note: at the start of a sentence or in last-name-first contexts (De Gaulle, Charles), capitalization rules invert. This corpus tests the natural-order case only.

Comma format reversal (FN24-FN26): Smith, John → John Smith. Tradeoff: irreversibly destroys the comma-format. If the buyer's downstream system expects "Last, First" format, they need --name-format=last-first. Default is natural reading order.

Titles and suffixes:

• Title period stripping: Mr. → Mr. Some style guides keep the period; this corpus drops it for consistency. --keep-title-periods flag if buyers prefer.
• Roman numerals (II, III, IV) stay all-caps. They aren't names; they're numerals.
• PhD, MD, Esq keep their conventional case. Don't lower-case them.

Non-Latin scripts (FN29-FN31): Pass through unchanged. Title-casing rules don't apply to scripts without case (Korean, Japanese, Chinese, Arabic, Hebrew, etc.). Cyrillic does have case but the conservative-by-default rule applies — only ALL CAPS gets title-cased.

Single names (FN32): Madonna, Cher, Pelé. Pass through unchanged when input is already title-case.

7.4 What's not tested

• Honorific stacking (Dr. Mr. Jane Smith — pathological, rare, hard).
• Cultural name-order detection (East Asian family-first vs Western given-first). Without a column-level signal the cleaner can't guess.
• Nickname expansion (Bob → Robert). Out of scope; that's data enrichment, not standardization.
• Name part identification (which token is given, family, middle). Belongs to a parser, not a standardizer.

---

8. CURRENCIES (29_format_currencies.csv)

8.1 Use cases by buyer persona

• Shopify: Order amount normalization across multi-currency stores.
• Bookkeeper: Bank export reconciliation; mixed bank formats produce different currency representations.
• Freelancer: Invoice data normalization.
• Marketing agency: Campaign spend normalization across ad platforms.

8.2 Test categories

Category	Cases	What it tests
us	FC01-FC07	$ prefix/suffix, comma thousands, dot decimal, USD code prefix/suffix
eu	FC08-FC11	€ prefix, dot thousands and comma decimal, space thousands, Swiss apostrophe
intl	FC12-FC14	£, ¥ (no decimal), ₹ (lakhs grouping)
negative	FC15-FC17	Leading minus, accounting parens, sign after symbol
edge	FC18-FC25	Zero, scientific, percentage, range, word values, empty, idempotency
ambig	FC26-FC27	Locale-ambiguous separator (1,234 could be 1234 or 1.234)

8.3 Critical policy decisions

Output format: <symbol_or_code><normalized_number>. Number uses dot decimal, no thousand separators, leading minus for negative. Currency symbol or code preserved if present in input; if no currency indicator, output is just the number.

Locale ambiguity (FC26, FC27): 1,234 is 1234 in US English and 1.234 in German. 1.234 is 1.234 in US English and 1234 in German. Per-column inspection: any value with both , and . (like 1,234.56) locks the locale unambiguously; otherwise the cleaner errors and demands --currency-locale=us|eu. Do not silently guess.

Accounting parens (FC16): ($100.00) → -$100.00. Standard accounting convention. The leading minus is more universally readable than the parens.

Currency symbol position: Preserved. $100 stays prefix-symbol; 100$ (rare but seen) stays suffix-symbol; 100 USD keeps the suffix-code form. Reasoning: changing position is destructive and the buyer can do it themselves with a simple find-replace if they want.

Indian lakhs grouping (FC14): ₹1,23,456.78 flattens to ₹123456.78. Lakhs grouping (groups of 2 after the first 3) is unusual outside India and breaks downstream tools that expect Western thousand-grouping.

JPY no decimal (FC13): Japanese yen conventionally has no fractional part. ¥1,234 → ¥1234. The cleaner doesn't add a decimal that wasn't there.

Scientific notation (FC19): 1.5e6 → 1500000. Expand to plain notation for spreadsheet compatibility. Loses the "this was scientific" information; acceptable tradeoff.

Percentages (FC20): Error. Percentage and currency are different domains. If the column is meant for percentages, that's not currency.

Ranges (FC21): Error. Same reasoning as multi-emails; structural split needed.

Word values (FC22, FC23): Free, TBD, N/A. Error. The buyer might want these mapped to 0 (Free) or empty (TBD/N/A), but those are domain decisions the cleaner can't make safely.

8.4 What's not tested

• Cross-currency conversion (USD to EUR via exchange rate). Massively out of scope.
• Cryptocurrency formats (BTC, ETH amounts with high decimal precision). Out of scope.
• Historical currency notation (pre-decimalization £.s.d). Out of scope.
• Currency code standardization (USD vs US$ vs $US). Default: pass through whatever's there.

---

9. INTEGRATION (30_format_integration.csv)

9.1 Purpose

Five rows, each a complete record with one or more format issues across multiple columns. Tests that running 03 across multiple columns in one pass produces consistent output and doesn't drop or scramble fields.

9.2 Per-row test goals

Row	What it tests
FI01	Standard messy-but-cleanable record. All six format types in one row. Tests that no domain's normalizer interferes with another's.
FI02	International record (UK address, EUR currency, German-format date, mailto-prefixed Gmail address, comma-format Mc-name). Tests cross-domain locale handling.
FI03	Errors (insufficient phone digits) and complex name (DR + JANE DOE + PHD title+name+suffix). Tests error handling and complex name parsing.
FI04	All empty. Tests that empty cells pass through without errors.
FI05	Already-clean record. Idempotency check — the entire row should round-trip unchanged.

9.3 What this fixture catches that single-domain fixtures don't

• Cross-column interference: a name normalizer that reaches into the email column, or vice versa.
• Schema drift: a normalizer that adds, removes, or reorders columns.
• Error-handling consistency: when one column errors (FI03's phone), other columns in the same row still process correctly.
• Idempotency at the row level: FI05 must produce byte-identical output.

---

10. Suggested test workflow

import csv
from pathlib import Path
from src.core.format_standardizer import standardize  # your impl

FORMATS = Path("test_data/formats")
EXPECTED = Path("expected/formats")

def test_single_column_domain(domain):
    """Test FD/FP/FE/FA/FN/FC fixtures with single-column expected output."""
    inp = FORMATS / f"{domain}.csv"
    exp = EXPECTED / f"{domain}_expected.csv"

    with inp.open() as f:
        cases = {r["case_id"]: r for r in csv.DictReader(f)}
    with exp.open() as f:
        expected = {r["case_id"]: r for r in csv.DictReader(f)}

    failures = []
    for case_id, case in cases.items():
        got = standardize(case["input"], domain=domain.split("_")[1])
        want = expected[case_id]["output"]
        if got != want:
            failures.append((case_id, case["input"], got, want))
    return failures

# Test each domain
for domain in ["24_format_dates", "25_format_phones", "28_format_names",
               "29_format_currencies"]:
    failures = test_single_column_domain(domain)
    print(f"{domain}: {len(failures)} failures")

# Email and address have two-policy expected output
def test_two_policy(domain, policy_columns):
    inp = FORMATS / f"{domain}.csv"
    exp = EXPECTED / f"{domain}_expected.csv"
    with inp.open() as f:
        cases = {r["case_id"]: r for r in csv.DictReader(f)}
    with exp.open() as f:
        expected = {r["case_id"]: r for r in csv.DictReader(f)}

    for policy in policy_columns:
        failures = []
        for case_id, case in cases.items():
            got = standardize(case["input"], domain=domain.split("_")[1],
                              mode=policy)
            want = expected[case_id][f"output_{policy}"]
            if got != want:
                failures.append((case_id, case["input"], got, want))
        print(f"{domain} ({policy}): {len(failures)} failures")

test_two_policy("26_format_emails", ["default", "gmail_canonical"])
test_two_policy("27_format_addresses", ["default", "expand_abbrev"])

# Idempotency property test
import random
all_inputs = []
for domain in ["24_format_dates", "25_format_phones", "26_format_emails",
               "27_format_addresses", "28_format_names", "29_format_currencies"]:
    with (FORMATS / f"{domain}.csv").open() as f:
        all_inputs.extend((domain, r["input"]) for r in csv.DictReader(f))

for domain, inp in all_inputs:
    once = standardize(inp, domain=domain.split("_")[1])
    twice = standardize(once, domain=domain.split("_")[1])
    assert once == twice, f"non-idempotent: {domain} {inp!r} -> {once!r} -> {twice!r}"

---

11. What this corpus does NOT cover

Listed so the gaps are explicit:

1. Performance. All fixtures are small. Format standardization on a 500MB customer file may have memory or speed issues; benchmark separately.
2. Cross-script integration with 02 and 04. This corpus tests 03 in isolation. Running 02 → 03 → 04 in pipeline is a separate integration concern.
3. GUI behavior. Single-cell preview, per-row preview, domain auto-detection from column headers. Each is a Streamlit-layer test, not a transformation test.
4. Custom locale dictionaries. The fixtures assume the cleaner ships with English month names and US-default phone country. Customers who buy this product and then complain that German months aren't recognized are flagging a feature request, not a bug.
5. URLs. Listed in BUSINESS.md's adjacent territory but not in 03's scope. If you want URL standardization, that's a feature request.
6. Booleans / yes-no normalization. Y / Yes / 1 / True → true. Borderline 03 territory but explicitly excluded; can be added as a 7th domain if buyers ask for it.
7. Postal codes outside US/UK/Canada. ZIP-style validation only for US.
8. Identifiers (SKU, SSN, EIN). Out of scope; too domain-specific.

---

12. How to extend the corpus

Add a new test case in an existing domain:

1. Edit the relevant fixture's row list in generate_format_test_files.py.
2. Add the corresponding expected output entry.
3. Re-run the generator.
4. If the new case is a category not yet listed, update the per-domain category table in this document.

Add a new domain (e.g., URLs):

1. Define use cases by persona.
2. Define policy decisions and which require a flag vs. being default.
3. Build the input fixture as 31_format_<domain>.csv and the expected output as 31_format_<domain>_expected.csv.
4. Add a Section 13 to this document covering the domain.
5. Update the index table in Section 2.

Add a new policy variant to an existing domain:

1. Add a new column to the expected output file (e.g., output_strict).
2. Document the new policy and what triggers it (which flag) in the domain's Section 5.3 (or equivalent).
3. The two-policy test in Section 10's workflow generalizes to N-policy.