feat(audit): async writer thread — safe to re-enable

Reported earlier: synchronous file writes in ``log_event`` blocked
the GUI render thread on hostile filesystems (Windows antivirus on
``~/.datatools/logs/`` is the prime suspect). A blocking ``open``
call doesn't raise — try/except can't recover from it — so the
only safe re-enable is to take file I/O off the render path.

Refactor:

- ``log_event`` and friends push events onto a ``deque(maxlen=5000)``
  via ``put_nowait`` and return in microseconds.
- A single daemon thread (``datatools-audit-writer``) drains the
  queue and writes batches. Holds the queue lock only long enough to
  snapshot + clear, then does I/O outside the lock so producers can
  keep enqueueing.
- ``audit_log_path()`` is now pure path arithmetic — no ``mkdir``
  no ``open``. The writer thread does the directory creation off
  the request path, so any hang there only affects the writer.
- Bounded queue means an unwritable disk doesn't unbounded-grow
  memory; the queue caps at 5000 and overflow drops OLDEST events
  so the most-recent (most-diagnostic) ones survive.
- First write failure prints once to stderr; subsequent failures
  are silent so logs don't drown the launcher terminal.
- ``flush_audit_log(timeout_s=0.5)`` drains the queue and signals
  the writer to exit; bounded so a stuck disk can't delay shutdown.

Other changes in this commit:

- ``shutdown_app`` now emits a "Session ending" event and calls
  ``flush_audit_log`` before kicking the os._exit timer, so the
  closing session's events make it to disk.
- The Diagnostics sidebar in ``hide_streamlit_chrome`` is
  re-enabled (the ``if False:`` gate is removed). Wrapped in
  try/except defensively — render errors print to stderr, never
  blank the page.
- ``_DISABLED`` kill-switch is gone. The async design IS the
  safety mechanism now.

Tests in ``tests/test_audit.py``:

- log_event burst of 1000 events completes in well under 1s
  (proves non-blocking).
- Events queued before flush land on disk with the expected JSON
  shape; session_start renders; idempotent.
- Pointing the audit dir at a file (so mkdir fails) doesn't hang
  or crash the producer.
- Non-JSON extras are str()-coerced rather than dropped.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-17 02:39:48 +00:00
parent 7cb1bc922d
commit d9e32e578b
3 changed files with 367 additions and 149 deletions

View File

@@ -158,21 +158,17 @@ def hide_streamlit_chrome(*, gate_license: bool = True) -> None:
require_license_or_render_activation,
)
render_license_status_sidebar()
# Diagnostics sidebar is temporarily disabled while bisecting the
# "blank pages" report. The chrome runs on every page, so anything
# that fails here drops every page's body. Wrapping it caught
# exceptions but a silent partial-render (e.g. an open ``with
# st.sidebar:`` context that never closes cleanly because of an
# internal Streamlit hiccup) could still poison subsequent
# rendering. Toggle this back on once the user confirms the bare
# chrome restores page rendering.
if False:
try:
_render_diagnostics_sidebar()
except Exception:
import traceback, sys
print("DataTools: diagnostics sidebar render failed:", file=sys.stderr)
traceback.print_exc()
# Diagnostics sidebar re-enabled now that the audit log is async
# and ``audit_log_path()`` is a pure path computation (no mkdir
# on the request path). Still wrapped in try/except defensively;
# a render error here prints to stderr instead of taking down
# the page body.
try:
_render_diagnostics_sidebar()
except Exception:
import traceback, sys
print("DataTools: diagnostics sidebar render failed:", file=sys.stderr)
traceback.print_exc()
if gate_license:
require_license_or_render_activation()
@@ -723,6 +719,15 @@ def shutdown_app() -> None:
"""
if not st.session_state.get("_app_shutting_down"):
st.session_state["_app_shutting_down"] = True
# Drain the audit log queue to disk before the process dies.
# Bounded by a 500ms timeout so a stuck disk can't delay
# shutdown beyond the daemon-thread's own 1s grace period.
try:
from src.audit import flush_audit_log, log_event
log_event("session", "Session ending")
flush_audit_log(timeout_s=0.5)
except Exception:
pass
if "pytest" not in sys.modules:
def _hard_exit() -> None:
time.sleep(1.0)