giteadmin/datatools-dev
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(perf): publish 1.5 GB target and the new measured throughputs

Browse Source 
REQUIREMENTS §10 reflects the post-optimisation numbers and the
known O(n²) dedup match step (flagged for a future blocking pass).
en/es upload-limit copy and uploader help now say 1.5 GB.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Michael
2026-05-13 15:37:26 +00:00
parent 5b672370a6
commit e5f632bcd6
3 changed files with 32 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
docs/REQUIREMENTS.md
View File

@@ -3,7 +3,7 @@
Numbered support matrix. Updated with every shipped capability.
## 1. File handling
1.1 Size: ≤ 1 GB target (larger works, slower).
1.1 Size: ≤ 1.5 GB target (larger works, slower).
1.2 Read: CSV, TSV, XLSX, XLS.
1.3 Write: CSV, TSV.
1.4 Excel: multi-sheet picker.
@@ -64,17 +64,33 @@ Sample size: 1,000 rows (configurable).
- `skip` — waive (audit-logged).
- `modified` — apply with custom payload.
## 10. Performance (1 GB input)
## 10. Performance (1.5 GB input)
- Initial scan (sample): < 2 s · peak RSS ~110 MB.
- Full-file `repair_bytes`: 30–40 s.
- Full-DataFrame analyze: ~4 min (~25 µs/cell).
- Full-file `repair_bytes`: 30–40 s (UTF-8); non-UTF-8 fold path now
uses ``str.count`` instead of a Python char-by-char zip walk —
formerly ~100 s on a 1 GB cp1252 file with smart quotes, now <1 s.
- Full-DataFrame analyze: ~4 min (~25 µs/cell). Near-duplicate detector
no longer allocates a full-frame copy — peak RSS during the
near-duplicate pass drops to roughly the size of the string columns
alone (~50% memory cut on text-heavy 1 GB inputs).
- Full-DataFrame `auto_fix`: ~5 min (~30 µs/cell).
- Output write: ~10 s.
- Recommended RAM: 4× input size for full-Apply path.
- Format standardizer (`standardize_file`): ~150k rows/sec on cache-warm
international data; chunk-bounded RAM (~50 MB peak at default
chunk_size=50,000). A 1 GB CSV with mixed phone+currency+address
columns finishes in ~2.5–10 minutes depending on column count.
- Recommended RAM: 3–4× input size for the full-Apply path.
- **Format standardizer** (`standardize_dataframe`): ~2.7M rows/sec on
cache-warm repetition-heavy columns (synthetic 1M-row in-memory
benchmark, 2 typed columns); the fused single-pass loop replaced a
3-pass ``.tolist()`` cycle, so per-call overhead is now dominated by
the underlying parsers (phonenumbers, dateutil) rather than Python
list materialisation. A 1.5 GB CSV with mixed phone+currency+address
columns finishes in ~1.5–6 minutes depending on column count.
- **Text cleaner** (`clean_dataframe`): ~1M rows/sec on
repetition-heavy columns (per-call string cache: the pipeline runs
once per *unique* cell value, not once per row).
- **Deduplicator**: known O(n²) match step — works to ~50k rows in
comfortable time. The normalisation pass is now LRU-cached per call
so repeat values (the common dedup workload) skip re-parsing
(~2–5× faster on the normalisation step alone). Scale beyond 50k
needs blocking — flagged in `docs/NEXT-STEPS.md`.
## 11. Tools
1. Deduplicator — Ready
@@ -134,7 +150,7 @@ and proceeds.
- **Dev**: pytest, tox.
## 16. Test coverage
- 1,762 tests passing, 0 skipped, 0 xfailed.
- 1,770 tests passing, 0 skipped, 0 xfailed (incl. perf-shape regression tests).
- Fixture corpora: text-cleaner (21), encodings (31), reference UTF-8 (9), format-cleaner (199 buyer cases + 20-row international stress fixture), missing-handler (3 use cases + 16 edge cases), column-mapper (3 use cases + 5 edge cases).
- Run: `python run_tests.py [--tool …] [--fixtures] [--coverage]`.