Loresis now sends labelled warn/error events to Chief
Initial ask
Cover LiveCEOEmailCalendar (Loresis, connect.loresis.ai) with structured logging so every labelled warn/error from any user reaches Chief:
- audit the entrypoints and failure paths
- add a small batching logger with a disk spool that drains on recovery
- instrument failures and key actions
- audit the existing error logs: downgrade the system-working cases, move ids out of messages
- never log secrets or personal data
- commit in chunks and hand off to Chief's pipeline (no merge)
No later owner feedback.
Timeline and branch
Branch chief/212c6587f7, cut from main at d90eeeafb4d43e7433edfb077ecb6a3c790d5c2c, pushed to origin. All times UTC, 2026-10-28.
- 01:06:
b8b747850e0d3ba08501979cdc67866385c782f9. Logger plus central wiring. - 01:17:
018f99f358620f39a32365bf493922924d0089b4. Subsystem instrumentation, browser reporter, existing-log audit. - 01:30:
e450f35222c7e219d6a19360a33769dec33ba067. Build-warning cleanup,/healthzfailures logged. - 01:30:
8f5d29416b092873de99663846a545c75d7927c6.run.shrequires the Chief settings; docs. - About 01:25: added
CHIEF_LOG_URLandCHIEF_LOG_TOKENto/opt/liveceo-secrets/loresis.envon loresis-prod. Nothing was restarted.
What was done
- Logger (
src/lib/log.ts). Builds events to the Chief contract, with the workspace id asuser, the image tag asreleaseand server/container ashost. Batches go out every 3 s, at 256 KB or at 500 events. On network failure or a 5xx, batches spool to$DATA_DIR/chief-log-spool.ndjsonand drain oldest-first on recovery. A 400/403 is dropped with a loud stderr line. Emails and credential-shaped text are masked. Every event is also printed fordocker logs. - Central wiring.
- The API route wrapper logs every failure. Server failures and runtime bugs are
error. Refusals, 404s, usage limits and signed-out requests (expected) arewarn. - Next's
onRequestErrorcovers page renders and the routes outside the wrapper. - Uncaught exceptions and rejections are reported.
- Every background job, start-up recovery step and
/healthzfailure is logged. - Subsystems. Mail runs, assistant chat/dashboard requests, tool calls, the Claude subscription credential, knowledge indexing, data connections, GitHub sync/reviews, approved email/calendar/Telegram sends, Recall webhooks and calls, Browser Use, dashboard refresh, website reads, Telegram, voice and auth. Sparse
infoevents mark key actions. - Browser errors. Uncaught errors and React render crashes now reach Chief through
/api/client-errors. - Existing-log audit. All 34 existing
console.error/console.warncalls were replaced: - 9 downgraded from
error: 8 towarnor level-by-cause, 1 toinfo. - 2 sources marked
expected. - Variable parts moved into
ctx/user/stackin all 34. In 3 of them an error text, a count or a workspace id had been part of the message itself. - Production wiring.
deploy/run.shrefuses to deploy withoutCHIEF_LOG_URL/CHIEF_LOG_TOKEN, and the production env file now has both. Rules and the label list are indocs/implementation/chief-logging.md.
Not achieved
- Not deployed. Production still runs
loresis:656e936, which doesn't have this code. Production events start after Chief merges and the next deploy. That deploy needs care, because prod has run local merges of open PRs before. - No production events observed yet. I checked from this machine only:
- one
logging.smoke_testinfo event (Chief answered202 accepted:1) startup.readyfrom a production-mode local server- The four Loresis forks are not covered. They have their own repos and env files and need their own tokens.
- Early browser errors can be lost. A browser error raised before the page's first API call has no workspace header, so the report is refused (409). It then shows up only as
api.request_refused. - Two choices that make some events less precise:
- Mail-run progress lines stay stdout-only.
job.source_dashboardsstill reports aggregated provider read failures aserror.- Untested: event volume under real traffic, and how Chief groups these issues.
- Untouched: 4 Turbopack warnings in
env.tsandwiki/index.ts, files this branch doesn't change. - Chief and the owner: Chief publishes the PR and runs the test gate. I did not merge or deploy.
Before → After
- API failures: only 5xx reached stderr → every failure is a labelled event with route, status, stack and workspace id.
- Background, webhook and browser-side failures: UI state or silence →
warn/errorevents in Chief. - Existing error logs: free text with ids and error text in the message → fixed templates with the variable parts in
ctx. - Deploy without Chief settings: would start without coverage →
run.shstops before touching the running container, and the app refuses to start.
Tradeoffs
- Token in the env file, not in code. The forks share this code and would otherwise log into this project.
- Plain
Errorsentences arewarn, runtime errors (TypeErrorand similar) areerror. The codebase uses plainErrorfor refusals and provider outages. This keeps provider outages from dispatching fixers. Each throw site is still its own issue through its stack. - Logged where a failure is handled, not where it's thrown. One failure is one event. The exception is the subscription credential, which also logs one root-cause
error. - Production refuses to start without the Chief settings. That fails loud, as CLAUDE.md requires, rather than silently losing coverage.
Verification
npm test: 317/317 pass. This includes the newtest/log.test.ts(8 tests: contract shape, scrubbing, caps, ≤500 per POST, spool on outage/5xx, drain on recovery, 403 drop, spool replay at start, production config, API level rules). One Telegram assertion was updated to the new line format; it still checks that no secrets appear.tsc --noEmitandeslint srcare clean.npm run build: exit 0, 4 warnings, none in changed files.- Standalone server with
NODE_ENV=production: - without the token: "Failed to prepare server … CHIEF_LOG_URL is not set", requests answer 500
- with the token:
[startup.ready]delivered, no spool file left - Live ingest:
202 {"accepted":1,"dropped":0}.