Home now appears in the sidebar as "File Analysis" under a labeled
"Analysis" section together with Reconcile Two Files — both pages
are data-analysis workflows (importing/profiling files vs. matching
across files), so grouping them clarifies the sidebar's mental model.
- tools_registry: new ``analysis`` Section; reconcile moves out of
automations into it.
- i18n: ``nav.section_analysis`` + ``nav.file_analysis_title`` added
to en.json and es.json.
- app.py: home dropped from the unlabeled section and surfaced at the
top of the Analysis group; ``default=True`` preserved so first-visit
routing is unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a Streamlit page at ``/logs`` listing every
``datatools-*.jsonl`` file in ``audit_log_dir()`` (7-day window
per the retention sweep in b3ae913). Each entry shows filename,
mtime, byte size, and a ``st.download_button``. Today's file
gets its own section at the top.
The page also surfaces both paths as copyable monospace text:
the active log path (so users can grep/cat it directly on their
machine) and the folder path (so they can paste into Explorer /
Finder).
Wired into navigation via ``st.Page("pages/_Logs.py", ...)`` with
``url_path="logs"``. The sidebar entry is hidden by the same
``hide_streamlit_chrome`` CSS rule that hides ``/activate`` and
``/close`` — same pattern, same ``:has()`` + plain-fallback
selectors so the LinkContainer collapses cleanly in modern
browsers and the anchor is at least un-clickable in older ones.
License gate is OFF for this page (``gate_license=False``) — if a
user's license expires they may need logs to file a support
request; locking them out of their own audit history would be
hostile.
Next commit will wire the popover link.
Rollback: ``git revert HEAD`` removes the page and its nav entry;
the audit log itself keeps working.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four issues batched together since they all touch the GUI shell:
- ``stMainBlockContainer``'s ``padding-bottom`` bumped from 0.75rem
→ 4rem (~one button-height of free space above the fixed Help/Close
footer). The last line of content on a page that fills the viewport
was previously sitting flush against the footer's top border.
- Farewell overlay's "Close this window" button removed per UX
request. The auto-dismiss path is now the only flow: try
programmatic close (works in Chrome/Edge ``--app`` windows);
failing that, surface the hint and redirect the parent window to
``about:blank`` after a short timeout. Previously the user had to
click the button to get the same fallback. The
``quit.close_window_button`` i18n key is retained as a no-op for
now in case the button comes back; nothing references it.
- Sidebar collapse → expand was broken: clicking « collapsed the
sidebar but the » expand-back affordance was invisible. Two causes
pulled apart:
1. ``.dt-brand { flex: 1 }`` was eating the entire
``stSidebarHeader`` width, squeezing Streamlit's
``stSidebarCollapseButton`` off the right edge. Changed to
``margin: 0 auto 0 0`` so the brand keeps its natural width
and the chevron has room to live next to it.
2. The "hide Streamlit chrome" toolbar block was listing
``stToolbar`` and ``stToolbarActions`` for ``display: none``
— but the post-collapse re-open button
(``stExpandSidebarButton``) lives inside ``stToolbar``, so
hiding the container killed the button too. Dropped both
container testids from the hide list and kept the per-icon
rules for ``stMainMenu`` / ``stAppDeployButton`` /
``stStatusWidget`` / ``stDecoration``.
- Loguru's stderr sink quieted in GUI mode. ``src/gui/app.py`` now
runs ``logger.remove()`` + ``logger.add(sys.stderr, level="ERROR",
…)`` at the top so internal ``logger.debug`` / ``logger.warning``
breadcrumbs (e.g.
``standardize_dataframe: 7/31 cells were unparseable``) no longer
print to the terminal when the user runs ``python -m src.gui``.
CLI entry points already do the same configuration per-script.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two pieces of the mockup 2 layout that hadn't landed yet:
1. Sidebar nav icons — emoji glyphs (🧹✂️🔍 …) swapped for
Streamlit's ``:material/<name>:`` syntax, picking the outline
Material Symbol that best matches each mockup SVG:
Home → :material/home:
Fix Missing Values → :material/help_outline:
Find Unusual Vals → :material/insights:
Clean Text → :material/text_format:
Standardize Fmts → :material/format_list_bulleted:
Find Duplicates → :material/search:
Quality Check → :material/check_circle:
Map Columns → :material/view_column:
Combine Files → :material/account_tree:
Auto Workflows → :material/auto_awesome:
Activate → :material/key:
Close → :material/close:
Streamlit injects the icon name as a literal ligature inside a
first-child ``<span>`` of the nav anchor, expected to render
through the Material Symbols font. theme.py's base rule was
forcing Geist on every span under ``stSidebarNav``, turning the
ligatures back into plain text labels — added a structural
exception that targets ``[data-testid="stSidebarNavLink"] >
span:first-child`` (and any descendant), restoring the Material
font family, neutralizing the inherited ``ss01/cv01/cv11``
feature settings, and sizing to 18px.
Also stripped the leading emojis from every page title in the
en/es i18n packs (``home.title``, ``close_page.title``,
``activation.title``, ``tools.*.page_title``) — the icons live
in the sidebar now, the page H1 no longer needs to carry one.
2. Stats overview on home — new ``_render_stats_overview`` in
_home.py emits a 4-card grid above the per-file findings panels:
Files analyzed, Total findings, Warnings (severity ``warn`` ∪
``error``), Info (severity ``info``). Card layout follows the
mockup §stats verbatim — Geist 28px / 600 / -0.03em for the
numeric value (the "Display number" row in spec §4), tiny
uppercase tracked label, paper-surface card with the standard
warm border + faint shadow. The Warnings / Info cards tint the
number with ``--warn`` / ``--info`` when the count is non-zero.
CSS for ``.dt-stats / .dt-stat / .dt-stat-label / .dt-stat-value /
.dt-stat-unit`` added to ``_DESIGN_TOKENS_CSS``; falls to a
2-column grid below 900px viewport, matching the mockup's media
query.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two follow-ups to the prior sidebar/footer cleanup:
- The "_hidden" section header was still visible in the sidebar
because Streamlit renders ``stNavSectionHeader`` as a sibling of
``stNavSection``, not a child — so the ``:has()`` rule on the
section was hiding the items list but leaving the header
(and its collapse/drilldown marker) behind. Move Activate +
Close into the unlabeled section (key ``""``) alongside Home so
there is no header to leak in the first place, then hide just
the two links via ``stSidebarNavLinkContainer:has(...)`` (with
a defensive ``a[href$=...]`` fallback for browsers without
``:has()`` support).
- The sticky footer was missing on ``pages/_Activate.py`` because
the page never called ``render_sticky_footer`` — added the
call so the Help / Close bar persists when the user follows
the popover's Activate / Manage link.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Collapse the Account section: Activate now lives in the same
hidden sidebar section as Close (single ``_hidden`` group). Both
pages stay registered with ``st.navigation`` so /activate and
/close remain URL-routable for the Help-popover / Close-button
links — only the sidebar entries + their section header are
hidden via CSS.
- Help popover always exposes a license-management link now:
``Activate now →`` when the license is inactive, ``Manage
license →`` when it is active and valid. Both point at
``./activate``.
- Extend the sidebar-hide CSS to also match ``a[href$="/activate"]``
and the section that contains it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three small follow-ups to the sticky-footer rework:
- Left-justify the footer buttons (and reposition the Help popover
to anchor at the left edge so it lines up with its trigger).
- Remove the per-page ``st.divider() + st.caption("Runs locally…")``
trailing block from all 9 tool pages. The new sticky footer
covers that text, so it was rendering as an empty white bar at
the bottom of each tool page.
- Hide the Close entry from the sidebar nav via CSS. The page stays
registered with st.navigation so /close is still routable for the
sticky-footer Close button — only the sidebar link + its section
header are hidden (via :has() on stNavSection).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two issues, same fix surface.
(1) Reported crash on Back-to-Home:
StreamlitAPIException: Could not find page: app.py.
``st.switch_page("app.py")`` doesn't work under ``st.navigation`` —
the entry script is the nav manager itself and is not a registered
page. The fix needs to pass an ``st.Page`` object whose script
identity matches one registered in the nav.
First-pass attempt (``from src.gui.app import _home_page``) hit a
worse failure: importing ``app.py`` from inside a tool-page render
re-executes the nav setup with the WRONG "main script" context, so
every ``st.Page("pages/N_foo.py", ...)`` call in ``_build_navigation``
fails with "file could not be found".
Extract the home renderer into its own module ``src/gui/_home.py``
which has no top-level Streamlit side effects. Both the nav manager
and the back-link helper import ``_home_page`` from there. The Page
object built at click time has the same callable identity as the one
registered, so ``st.switch_page`` resolves it.
(2) Reported UX: the back button scrolled out of view on long pages.
Add a second ``back_to_home_link(key="_back_to_home_link_bottom")``
call near the footer of every tool page (1-9). The unique key avoids
widget-id collision with the top instance. Coming-Soon stubs get it
unconditionally; Ready tools render it only after a result exists
because the page short-circuits with ``st.stop()`` before then —
when no result is on screen the page is short enough that the top
link is sufficient.
2220 tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Restoring ``render_findings_panel`` on the home page. Previous commit
(c575efd) inlined a flat renderer that dropped the per-tool grouping
and the "Open <Tool>" jump links — that was an over-correction. The
user only wanted the bottom tool-card grid gone (already removed in
ff2eaeb). The grouping inside the findings panel is what lets a user
land on a specific finding and one-click into the cleaner that fixes
it; without it they'd have to guess which sidebar entry to open.
Tool-card grid stays removed. Sidebar nav is unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The home page was calling ``render_findings_panel``, which groups
findings by tool into expanders and renders an "Open <Tool>" page
link under each. After uploading a file, the user still saw a tool
list (just under a different shape) — defeating the earlier cleanup
that removed the tool-cards grid.
Inline a flat renderer in ``_home_page``: per uploaded file, render
the filename header + severity summary + a flat list of findings via
``_render_one_finding`` directly. No expanders, no tool names as
section headers, no per-tool page-link buttons. Tool discovery
happens in the sidebar.
``render_findings_panel`` itself is unchanged — it still groups by
tool and remains tested via the findings-panel harness, but is no
longer used on the home page.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The sidebar nav was passing ``tool.name`` (the registry's English
field) to ``st.Page``, so the tool entries stayed in English even
after the user picked Spanish from the language selector. Section
headers were already i18n-driven; tool entries were not.
Switch to ``tool_name(tool_id)`` which routes through ``t(...)`` and
picks up the active language from session state. Verified: with
``ui_lang=es`` the sidebar renders Buscar duplicados / Limpiar texto /
Mapear columnas / etc. instead of the English fallbacks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Close is now a direct shutdown trigger: visiting the Close page (the
sidebar entry) fires shutdown_app() immediately — no confirm step, no
intermediate body. The farewell overlay paints and os._exit(0) lands
~1s later from a daemon thread.
Layout: Close moved into its own bottom-of-sidebar section so the
destructive action is visually separated from Account/Activate.
- New shutdown_app() in components/_legacy.py replaces quit_button.
os._exit thread is skipped when "pytest" is in sys.modules so the
test suite doesn't suicide on rendering 99_Close.
- pages/99_Close.py shrinks to set_page_config + chrome + shutdown_app.
- app.py nav grows a new "Close" section header (new
nav.section_close key in en/es packs) pinned at the bottom of the
navigation dict.
Tests updated:
- TestQuitButtonRenders → TestClosePageShutsDownImmediately.
Assert the shutdown caption renders + no confirm button exists.
- test_smoke EXPECTED_SUBSTRINGS["99_Close"] now pins
"Shutting down" / "Cerrando" (the visible page body) instead of
the removed page title.
2008 tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Home is now upload + analysis only. The page accepts multiple files in
one go, analyzes each independently, and renders findings grouped by
filename in bordered containers. The 3-section tool-card grid is gone —
discovery happens via the sidebar now.
Mechanics:
- file_uploader uses accept_multiple_files=True. Each file's findings
cache in session_state["home_findings_by_file"] keyed by filename so
removing a file via Streamlit's "x" button drops its findings too,
and re-clicking Run only re-analyzes pending files.
- The first uploaded file is mirrored into the singular
home_uploaded_{name,bytes,size} keys so tool pages continue to pick
up an "active" upload through pickup_or_upload — no tool-page changes.
- New i18n keys: upload.intro_multi, upload.uploader_label_multi,
upload.clear_results, upload.empty_state. upload.heading text is
updated to "Upload one or more files to start" (EN + ES).
Dropped tests pinning the tool grid:
- TestHomeToolGridLocalization (test_chrome.py)
- test_home_tool_card_uses_es_name (test_smoke.py)
- TestLiteHomeGridBadges (test_lite_tier.py — locked-card lock-badge
assertions; locking is still enforced per-tool-page via
require_feature_or_render_upgrade)
2009 tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Home is now the only entry point: the "Run analysis" button on the
upload section IS the review step (findings render inline via
render_findings_panel). Tool pages no longer gate on a passed
normalization — running the analyzer is sufficient context.
Removed:
- src/gui/pages/0_Review.py
- src/gui/components/gate.py (re-export seam)
- require_normalization_gate() in src/gui/components/_legacy.py
- "review" section enum in tools_registry.py
- Data Review entry in app.py navigation
- require_normalization_gate() calls + imports in all nine tool pages
- tests/gui/test_gate.py (whole file)
- TestReviewWorkflow in tests/gui/test_workflows.py
- 0_Review entry in tests/gui/test_smoke.py PAGE_SLUGS
- stash_upload's normalization_result+normalization_for stashing
- stash_upload_without_gate (was the gate's negative-path helper)
2017 tests pass (16 retired with the gate flow).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sidebar nav now groups tools under Data Review / Data Cleaners /
Transformations / Automations via st.navigation, replacing the flat
auto-discovered list. Tool display names switch to action-first
phrasing (Find Duplicates, Fix Missing Values, Find Unusual Values,
Standardize Formats, Clean Text, Quality Check, Map Columns, Combine
Files, Automated Workflows) in EN + ES packs and on each page's H1.
The Data Cleaners section follows the requested order: Missing
Values → Outliers → Text Cleaner → Format Standardizer → Deduplicator
→ Quality Check. (Text Cleaner kept inside cleaners since the request
didn't list it but the tool still ships.) Registry now carries a
section field; helpers added: tools_in_section(), section_label().
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two coupled changes:
1. Lite tier
- New Tier.LITE in src/license/schema.py.
- FEATURES_BY_TIER[Tier.LITE] = {Deduplicator, Text Cleaner,
Format Standardizer}. The three universally-useful tools that
cover the most common bookkeeping / RevOps / Klaviyo prep
workflows. Other six tools require Core.
- i18n: license.tier_lite, license.feature_locked_title,
license.feature_locked_body, license.upgrade_link,
license.status_locked (en + es).
- Per-tool feature gate at every GUI tool page
(require_feature_or_render_upgrade) and every tool CLI
(guard(feature=...)). A locked tool renders an upgrade
prompt + Manage-license button (GUI) or exits with code 2
(CLI).
- Home grid: tool cards the user's tier doesn't unlock get a
red 🔒 Locked badge in place of green Ready.
2. Trial removed
- Activation form's "Start 1-year trial" button removed.
- license_cli's `trial` subcommand removed.
- activation.trial_button / activation.trial_help i18n keys
dropped (pack parity test stays green).
- Tier.TRIAL stays in the enum (back-compat with any field-
tested trial licenses); LicenseManager._mint stays internal
for tests and the seller's key generator.
- Decision logged in DECISIONS §9b: a 1-year all-features
trial undercuts paid Lite; paid-only keeps tier economics
clean.
Tests (+29 net): +17 Lite-tier unit/guard tests + 13 Lite-tier
GUI tests + 1 trial-absent assertion - 2 trial CLI tests - 1
trial GUI button test. Total: 1995 → 2024.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduces ``src/i18n`` with a tiny JSON-backed t() lookup, an in-session
language preference, and a sidebar selector wired through
``hide_streamlit_chrome`` so every page picks up the same picker. Covers
home, tool cards, findings panel, gate, shutdown, and pickup banner
strings. Tests pin pack parity and the farewell-overlay JS escape so
future packs can't silently regress.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The footer placement was easy to miss (below all tool cards) and only
rendered on the home page. Hook the button into hide_streamlit_chrome()
so every page that hides default chrome — home + all 9 tool pages — gets
the Quit button at the bottom of the sidebar without per-page edits.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The chrome-hiding CSS was removing the Streamlit header wholesale,
which also took the sidebar's expand chevron with it — a collapsed
sidebar became unreopenable. Make the header transparent instead and
explicitly preserve the sidebar collapsed-control.
Also add a Quit button in the app footer that signals the Streamlit
server (SIGTERM, falling back to SIGINT) so closing the GUI returns
the shell prompt cleanly instead of leaving Python hung.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two low-risk seam moves to enable selling per-tool subsets without
breaking the existing all-in-one bundle. Behaviour identical; every
existing import still resolves; full pytest suite + every page returns
HTTP 200.
1. **Tool registry** (src/gui/tools_registry.py) — replaces the
inline dict-of-dicts in app.py with a Tool dataclass and a TOOLS
list. Adds a tier field ("core" today, "pro" / "enterprise" later)
and tools_for_tier() / tool_by_id() / display_name() helpers. A
per-tool build slices TOOLS at import time without code changes.
2. **components package** (src/gui/components/) — converts the former
single components.py into a package with:
_legacy.py — original file, unchanged.
__init__.py — re-exports the legacy surface; existing
"from src.gui.components import …" calls
continue to work.
shared.py — hide_streamlit_chrome, pickup_or_upload
(every build needs these).
gate.py — require_normalization_gate (Pro / Suite SKUs).
findings.py — analyzer-finding widgets (drops out of a
standalone-Dedup build).
dedup_review.py — match-group cards + apply pipeline (drops out
of a non-dedup build).
The seam modules are narrow re-exports today. As code migrates out
of _legacy.py into the focused modules, the public import path
stays stable via the shim.
E2E: 765 passed, 17 xfailed (unchanged); home page + all 9 tool pages
+ Review page render HTTP 200; full pipeline (analyze → auto_fix →
apply_decisions → output bytes) round-trips on the kitchen-sink
fixture with zero high-confidence findings remaining post-fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Home page (src/gui/app.py) gains an upload + analyze section above the tool
grid: file uploader, "Run analysis" / "Skip" buttons, and a findings panel
grouped by destination tool. Tool cards now carry a "N findings" badge
when the active session's findings reference that tool, so the user sees
at a glance which tools their just-uploaded file would benefit from.
src/gui/components.py adds the shared GUI surface:
- TOOL_DISPLAY_NAMES + tool_display_name() — single source of truth for
GUI labels, keeping detector tool ids decoupled from the UI.
- render_findings_panel(findings) — severity icons, expander per tool,
open-tool page link, sample-cells dataframe.
- upload_and_analyze_section() — the home-page widget; stashes file
bytes and findings in session_state so future tool pages can pick up
the existing upload instead of re-prompting.
- findings_count_for_tool(tool_id) — used by app.py to badge cards.
CSV/TSV uploads run through repair_bytes() before analysis, so the user
also sees csv_bom_stripped / csv_smart_quotes_folded findings synthesized
from the pre-parse repair pass. Excel uploads skip that step.
The Text Cleaner tool card flips from "Coming Soon" to "Ready" — that has
been true since the v3.0 implementation and the home page just hadn't been
updated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add shared hide_streamlit_chrome() helper that removes header bar,
hamburger menu, footer, and deploy button via CSS injection. Called
on every page. Add .streamlit/config.toml with minimal toolbar mode.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Convert single-page deduplicator into a multi-page suite. Home page shows
tool card grid. Deduplicator extracted to its own page (fully working).
8 stub pages added for Text Cleaner, Format Standardizer, Missing Values,
Column Mapper, Outlier Detector, Multi-File Merger, Validator & Reporter,
and Pipeline Runner — each with functional file upload and coming-soon UI.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Delimiter dropdown now includes "Other" option with a text input for
custom delimiter characters. Subtitle updated to mention delimited text.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Auto-detects delimiter on upload and shows a selectbox with comma, tab,
semicolon, and pipe options. Changing re-reads the file immediately.
Line terminators (Windows/Unix/Mac) already handled by universal newlines.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace separate checkbox row and "Customize columns" toggle with a
unified st.data_editor grid — Keep checkboxes at the start of each row,
differing columns render as inline selectbox dropdowns.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace radio + Merge/Keep Both buttons with per-row checkboxes
and a single Confirm button. Users can now:
- Keep all rows (not duplicates) — check all, confirm
- Merge to one row — uncheck all but one, optionally customize columns
- Split a group — keep some rows, remove others (new capability)
Decision format changed from {action, survivor_idx, overrides} to
{keep_indices, overrides}. apply_review_decisions() updated to handle
all three modes. Batch actions updated accordingly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Each match group card now has:
- Radio button to pick which row to keep as the base survivor
- "Customize columns" toggle showing only columns that differ
- Per-column selectbox to pick values from any row in the group
- Decisions stored as {action, survivor_idx, overrides} dicts
Added apply_review_decisions() that builds the final DataFrame by
applying survivor selection + column overrides without re-running
the dedup engine. Batch actions also use the new dict format.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace st.rerun() with on_click callbacks so decisions write to
session state before the natural rerun. Decided groups auto-collapse
with status in the label; undecided groups stay expanded. Added undo
button on decided groups.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Rewrite README.md with project overview, quick-start, and CLI summary
- Add docs/CLI-REFERENCE.md with full flag reference and 8 recipe sections
- Add docs/DEVELOPER.md with architecture, data flow, and extension guides
- Rewrite src/core/__init__.py with public API exports and module docstring
- Add Streamlit GUI (src/gui/) with file upload, advanced options, interactive
match group review with side-by-side diff, and download buttons
- Add .gitignore, requirements.txt, all source code, tests, and sample data
- Add streamlit to requirements.txt
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
