feat(audit): async writer thread — safe to re-enable
Reported earlier: synchronous file writes in ``log_event`` blocked the GUI render thread on hostile filesystems (Windows antivirus on ``~/.datatools/logs/`` is the prime suspect). A blocking ``open`` call doesn't raise — try/except can't recover from it — so the only safe re-enable is to take file I/O off the render path. Refactor: - ``log_event`` and friends push events onto a ``deque(maxlen=5000)`` via ``put_nowait`` and return in microseconds. - A single daemon thread (``datatools-audit-writer``) drains the queue and writes batches. Holds the queue lock only long enough to snapshot + clear, then does I/O outside the lock so producers can keep enqueueing. - ``audit_log_path()`` is now pure path arithmetic — no ``mkdir`` no ``open``. The writer thread does the directory creation off the request path, so any hang there only affects the writer. - Bounded queue means an unwritable disk doesn't unbounded-grow memory; the queue caps at 5000 and overflow drops OLDEST events so the most-recent (most-diagnostic) ones survive. - First write failure prints once to stderr; subsequent failures are silent so logs don't drown the launcher terminal. - ``flush_audit_log(timeout_s=0.5)`` drains the queue and signals the writer to exit; bounded so a stuck disk can't delay shutdown. Other changes in this commit: - ``shutdown_app`` now emits a "Session ending" event and calls ``flush_audit_log`` before kicking the os._exit timer, so the closing session's events make it to disk. - The Diagnostics sidebar in ``hide_streamlit_chrome`` is re-enabled (the ``if False:`` gate is removed). Wrapped in try/except defensively — render errors print to stderr, never blank the page. - ``_DISABLED`` kill-switch is gone. The async design IS the safety mechanism now. Tests in ``tests/test_audit.py``: - log_event burst of 1000 events completes in well under 1s (proves non-blocking). - Events queued before flush land on disk with the expected JSON shape; session_start renders; idempotent. - Pointing the audit dir at a file (so mkdir fails) doesn't hang or crash the producer. - Non-JSON extras are str()-coerced rather than dropped. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -158,21 +158,17 @@ def hide_streamlit_chrome(*, gate_license: bool = True) -> None:
|
||||
require_license_or_render_activation,
|
||||
)
|
||||
render_license_status_sidebar()
|
||||
# Diagnostics sidebar is temporarily disabled while bisecting the
|
||||
# "blank pages" report. The chrome runs on every page, so anything
|
||||
# that fails here drops every page's body. Wrapping it caught
|
||||
# exceptions but a silent partial-render (e.g. an open ``with
|
||||
# st.sidebar:`` context that never closes cleanly because of an
|
||||
# internal Streamlit hiccup) could still poison subsequent
|
||||
# rendering. Toggle this back on once the user confirms the bare
|
||||
# chrome restores page rendering.
|
||||
if False:
|
||||
try:
|
||||
_render_diagnostics_sidebar()
|
||||
except Exception:
|
||||
import traceback, sys
|
||||
print("DataTools: diagnostics sidebar render failed:", file=sys.stderr)
|
||||
traceback.print_exc()
|
||||
# Diagnostics sidebar re-enabled now that the audit log is async
|
||||
# and ``audit_log_path()`` is a pure path computation (no mkdir
|
||||
# on the request path). Still wrapped in try/except defensively;
|
||||
# a render error here prints to stderr instead of taking down
|
||||
# the page body.
|
||||
try:
|
||||
_render_diagnostics_sidebar()
|
||||
except Exception:
|
||||
import traceback, sys
|
||||
print("DataTools: diagnostics sidebar render failed:", file=sys.stderr)
|
||||
traceback.print_exc()
|
||||
if gate_license:
|
||||
require_license_or_render_activation()
|
||||
|
||||
@@ -723,6 +719,15 @@ def shutdown_app() -> None:
|
||||
"""
|
||||
if not st.session_state.get("_app_shutting_down"):
|
||||
st.session_state["_app_shutting_down"] = True
|
||||
# Drain the audit log queue to disk before the process dies.
|
||||
# Bounded by a 500ms timeout so a stuck disk can't delay
|
||||
# shutdown beyond the daemon-thread's own 1s grace period.
|
||||
try:
|
||||
from src.audit import flush_audit_log, log_event
|
||||
log_event("session", "Session ending")
|
||||
flush_audit_log(timeout_s=0.5)
|
||||
except Exception:
|
||||
pass
|
||||
if "pytest" not in sys.modules:
|
||||
def _hard_exit() -> None:
|
||||
time.sleep(1.0)
|
||||
|
||||
Reference in New Issue
Block a user