datatools-dev/docs/REQUIREMENTS.md

Requirements

Numbered support matrix. Updated with every shipped capability.

1. File handling

1.1 Size: ≤ 1.5 GB target (larger works, slower). 1.2 Read: CSV, TSV, XLSX, XLS. 1.3 Write: CSV, TSV. 1.4 Excel: multi-sheet picker. 1.5 Empty file: blocked with empty_input error finding.

2. Input encodings (auto-detected)

2.1 Unicode: UTF-8, UTF-8-BOM, UTF-16 LE/BE BOM, UTF-16 LE no-BOM. 2.2 Western: cp1252, ISO-8859-1, ISO-8859-15, Mac Roman. 2.3 Eastern European: cp1250, ISO-8859-2. 2.4 Cyrillic: cp1251, KOI8-R. 2.5 CJK: Shift_JIS / cp932, GB18030, Big5, EUC-KR / cp949. 2.6 ASCII → detected as UTF-8. 2.7 User override: any Python codec name. 2.8 BOM: stripped on read, never written. 2.9 Decode failure → encoding_decode_failed (error). 2.10 U+FFFD in output → encoding_uncertain (error).

3. Output encodings

3.1 UTF-8 (default), UTF-8-BOM (Excel-friendly). 3.2 cp1252, ISO-8859-1/15, cp1250, ISO-8859-2, cp1251. 3.3 Shift_JIS, GB18030, Big5, EUC-KR, UTF-16 LE. 3.4 Lossy fallback: ? + warning when codec can't represent a char.

4. Delimiters

4.1 Input auto-detect: ,, \t, ;, |. 4.2 Output: , (default), \t, ;, |. 4.3 Extension: .tsv for tab, .csv otherwise.

5. Line endings

5.1 Read: LF / CRLF / bare CR — all normalized to LF. 5.2 Embedded in quoted cells: also normalized to LF. 5.3 Write: LF (default), CRLF, CR. 5.4 Mixed → mixed_line_endings finding.

6. Analyzer detectors

File-level (read-time fixes, audit-logged):

• csv_bom_stripped, csv_nul_stripped, csv_smart_quotes_folded, csv_line_endings_normalized, csv_transcoded_to_utf8, csv_unquoted_delimiters_repaired, csv_unrepairable_rows.

Cell-level:

• smart_punctuation_in_data, nbsp_or_unicode_whitespace, zero_width_or_invisible, dirty_column_headers, whitespace_padding, null_like_sentinels, suspected_mojibake, mixed_case_email_column, inconsistent_date_format, near_duplicate_rows, leading_zero_ids.

Encoding integrity: encoding_uncertain, encoding_decode_failed, encoding_lying_bom, empty_input.

Sample size: 1,000 rows (configurable).

7. Finding fields

id, severity (info/warn/error), confidence (high/medium/low), fix_action, pre_applied, tool, count, description, column, samples (≤5).

8. Confidence tiers

• high — round-trip safe, one-click auto-fix.
• medium — preview before applying.
• low — opt-in only, can corrupt if wrong.
• error — must resolve or waive before tool pages unlock.

9. Decision actions

• auto — apply registered fix.
• skip — waive (audit-logged).
• modified — apply with custom payload.

10. Performance (1.5 GB input)

• Initial scan (sample): < 2 s · peak RSS ~110 MB.
• Full-file repair_bytes: 30–40 s (UTF-8); non-UTF-8 fold path now uses str.count instead of a Python char-by-char zip walk — formerly ~100 s on a 1 GB cp1252 file with smart quotes, now <1 s.
• Full-DataFrame analyze: ~4 min (~25 µs/cell). Near-duplicate detector no longer allocates a full-frame copy — peak RSS during the near-duplicate pass drops to roughly the size of the string columns alone (~50% memory cut on text-heavy 1 GB inputs).
• Full-DataFrame auto_fix: ~5 min (~30 µs/cell).
• Output write: ~10 s.
• Recommended RAM: 3–4× input size for the full-Apply path.
• Standardize Formats (standardize_dataframe): ~2.7M rows/sec on cache-warm repetition-heavy columns (synthetic 1M-row in-memory benchmark, 2 typed columns); the fused single-pass loop replaced a 3-pass .tolist() cycle, so per-call overhead is now dominated by the underlying parsers (phonenumbers, dateutil) rather than Python list materialisation. A 1.5 GB CSV with mixed phone+currency+address columns finishes in ~1.5–6 minutes depending on column count. StandardizeOptions.parallel_columns (default 1, serial) lands the thread-pool scaffolding; on CPython 3.12 with the GIL it's roughly neutral, but the API is ready for the free-threaded (PEP 703) Python 3.13+ build where it will help.
• Clean Text (clean_dataframe): ~1M rows/sec on repetition-heavy columns (per-call string cache: the pipeline runs once per unique cell value, not once per row).
• Fix Missing Values (handle_missing): lazy-copy — when sentinel standardization runs but finds nothing, AND no drops AND no fills apply, the input frame is returned as-is. On a clean 1 GB file this saves the 1 GB allocation that the unconditional upfront copy used to take.
• Map Columns (map_columns): rename + drop both already return fresh frames; the explicit upfront df.copy() is now removed and downstream mutating steps (schema-add, coerce) copy on demand via _ensure_owned(). Rename-only and identity-mapping paths run with zero explicit copies.
• Find Duplicates:
  • Exact-only strategies (every column uses Algorithm.EXACT at threshold 100 — covers strong-key dedup like email/phone, the fallback drop-duplicates path, and explicit "match on this exact column" calls) now run in O(n) via groupby. Measured: 10k rows on an email-exact strategy → 73 ms (was ~30 minutes via the old O(n²) pair compare).
  • Fuzzy strategies still pair-compare. Opt in to prefix blocking via deduplicate(..., blocking_columns=['name'], blocking_prefix_len=1) to partition pairs by a cheap key. Measured: 5k rows fuzzy-name dedup → 25.6s with blocking vs. 179s without (7× faster). Trade-off: cross-block matches are missed; lower blocking_prefix_len widens blocks.
  • Normalisation pass remains LRU-cached per call so repeat values (the common dedup workload) skip re-parsing.

11. Tools

1. Find Duplicates — Ready
2. Clean Text — Ready
3. Standardize Formats — Ready
4. Fix Missing Values — Ready
5. Map Columns — Ready
6. Find Unusual Values — Coming Soon
7. Combine Files — Coming Soon
8. Quality Check — Coming Soon
9. Automated Workflows — Ready

Future / not in v1. Tool ideas captured for after-launch consideration live in docs/FUTURE-TOOLS.md — entries there are gated by the new-tool freeze in PLAN.md §2.1 and don't ship without a paying-customer + repeated-demand signal. Currently parked there:

• #10. PDF → CSV extractor (bank statements + similar). No PDF dependency exists in the repo today; this tool would need pdfplumber, streamlit-drawable-canvas, and a templates store. Estimated 3–4 weeks for a text-only MVP, 6–10 weeks for the polished version with multi-page template recall.

11.a Recommended pipeline order (soft, not enforced)

Automated Workflows ships with a SOFT_DEPENDENCIES table; the following ordering is the default and the basis of the warning surface. Re-ordering is allowed; the runner emits a warning string and proceeds.

#	Tool	Why this slot
1	column_map (optional, for header alignment)	Multi-vendor unification — rename early so downstream tools see canonical headers
2	text_clean	NBSP / smart quotes / zero-width pollution silently breaks downstream parsers
3	format_standardize	Phones / dates / currencies → canonical form before missing detection and dedup
4	missing	Sentinel detection, imputation, drop strategies — needs canonical types
5	column_map (optional, for schema enforcement)	Project to target schema, coerce, drop extras AFTER cleaning
6	dedup	Fuzzy matching is most accurate on canonicalised, sentinel-laundered data

12. Gate (Review & Normalize)

• Gates every tool page.
• Auto-fix button: applies all confidence=high findings in one click.
• Per-finding controls: Auto / Skip / Customize.
• Live before/after preview (≤5 sample rows).
• Audit log per fix (id, decision, cells changed).
• Encoding-override picker (16 codepages + custom).
• Advanced output expander: encoding + delimiter + line terminator.
• Result keyed by upload SHA-256; survives reload, invalidated on re-upload.

13. Interfaces

• GUI: Streamlit, browser-based, local, no internet. Sidebar language picker (English, Español).
• CLI: python -m src.cli (dedup) · src.cli_text_clean · src.cli_format · src.cli_missing · src.cli_column_map · src.cli_pipeline · src.cli_analyze. (CLI output is English-only.)
• Python API: from src.core import … (analyze, repair_bytes, clean_dataframe, deduplicate, standardize_dataframe, …).
• JSON output: --json on cli_analyze.
• Language packs: from src.i18n import t, LANGUAGES. Add <code>.json to src/i18n/packs/ + entry in LANGUAGES to add a language.

14. Platforms

• Python ≥ 3.10.
• OS: Linux, macOS, Windows.
• Browser: any modern browser.
• Network: not required at runtime.

15. Dependencies

• Core: pandas, openpyxl, charset-normalizer, typer, loguru.
• Dedup: rapidfuzz, phonenumbers.
• GUI: streamlit.
• Optional: ftfy (mojibake repair).
• Dev: pytest, tox.

16. Test coverage

• 2,033 tests passing, 0 skipped, 0 xfailed.
  • 1,868 core + CLI tests (run with pytest -m 'not gui' for a quick loop). Includes 49 license-layer unit tests (Ed25519 sign/verify, dev-key derivation, production-safe tripwire, schema), 25 license-CLI tests, and 17 Lite-tier feature-map + guard tests.
  • 165 GUI tests under tests/gui/ driving Streamlit pages via AppTest (smoke + EN/ES localization, chrome, gate, workflows, dedup review, advanced panels, error paths, findings panel, activation + license gate, Lite-tier per-page lock behaviour). Marked gui.
  • Includes 15 perf-shape regression tests.
• Fixture corpora: text-cleaner (21), encodings (31), reference UTF-8 (9), format-cleaner (199 buyer cases + 20-row international stress fixture), missing-handler (3 use cases + 16 edge cases), column-mapper (3 use cases + 5 edge cases).
• Run: python run_tests.py [--tool …] [--fixtures] [--coverage].

17. Privacy / data handling

• All processing local; no network calls in the data path.
• No telemetry.
• Original input never modified.
• Audit logs: logs/ next to each run (timestamped).

17a. Licensing

• Storage: ~/.datatools/license.json (or $DATATOOLS_LICENSE_PATH override). Signed with Ed25519 (asymmetric).
• Crypto: Ed25519. The seller holds the private key; every shipped binary embeds only the public key. A motivated reverse engineer who pulls everything out of the binary still can't sign new licenses. Keys are 32 bytes raw, exposed as hex via DATATOOLS_LICENSE_PRIVKEY (seller-side) and DATATOOLS_LICENSE_PUBKEY (build-time bake-in).
• Activation: buyer pastes a base64-encoded license blob (DTLIC1:...) on first launch; app verifies the signature offline + matches the buyer-entered name/email to the embedded values.
• No free trial: every license requires a paid blob from the seller. The user-facing trial flow (button + license_cli trial subcommand) was removed in v1.6 to keep paid-tier economics clean.
• Lifetime: every license is 1 year by default. Renewal applies a fresh blob without losing the embedded buyer identity. Tier may change during renewal (Lite → Core upgrade path).
• Tiers:
  • lite — Find Duplicates + Clean Text + Standardize Formats. Buyer pays once, gets the three universally-useful tools.
  • core — every Ready tool (all 9 in v1.6).
  • pro, enterprise — scaffolded for future SKUs; currently mirror Core. Add per-SKU restrictions by editing FEATURES_BY_TIER in src/license/features.py.
  • trial — kept in the enum for backwards compat with any field-tested trial licenses but no longer issuable.
• Feature flags: every tool has a stable feature id matching its tool_id in :mod:src.gui.tools_registry. Adding a future per- tool SKU is a one-line change to FEATURES_BY_TIER — no consumer code edits.
• Per-tool gating: each tool page (GUI) and tool CLI calls require_feature(FeatureFlag.<TOOL>) at entry. GUI shows an upgrade prompt + button to the Activate page; CLI prints a message naming the locked feature and exits with code 2.
• Lock badge: the home grid shows a red 🔒 Locked pill on tool cards the current tier doesn't unlock.
• Dev bypass: DATATOOLS_DEV_MODE=1 skips every check (used by the test suite and during development). Refused in shipped builds by the production-safe tripwire.
• Production-safe tripwire: assert_production_safe() runs at startup in every frozen build. Refuses to boot when DEV_MODE is set or the verification key is still the embedded dev key (i.e., the build pipeline forgot to override DATATOOLS_LICENSE_PUBKEY). No-op in source / pytest runs.
• No internet: signature verification is fully offline. The shipped binary embeds only the public key; the private key never leaves the seller. See docs/DECISIONS.md for the threat-model discussion.

18. Error handling

• Structured hierarchy: DataToolsError → InputValidationError, ConfigError, FileFormatError, FileAccessError.
• Subclasses extend stdlib ValueError / OSError so existing handlers still catch them.
• Every error carries: message, file path, column, operation, suggestion, underlying cause.