d384d3c5954c85fcdaf3d42e2aa800b7d4784844
3 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Alex Newman
|
d384d3c595 |
fix: bug-batch — 17 issues + 4 foundations (chroma, opencode, parser, OAuth, paths, uptime, classification) (#2282)
* feat: foundations F1-F4 + simple bug fixes Foundations (no consumer adoption yet): - F1 spawnHidden wrapper at src/shared/spawn.ts - F2 paths namespace with 18 accessors + invariant test (tests/shared/paths.test.ts) - F3 getUptimeSeconds at src/shared/uptime.ts - F4 ClassifiedProviderError at src/services/worker/provider-errors.ts + 6 tests Issue fixes (file-isolated, parallel-safe): - #2231: SECURITY.md at repo root for GitHub Security tab - #2240: dedupe observationIds before Chroma sync (ResponseProcessor.ts) - #2247: add task_complete to Codex session-end events - #2243: rsync excludes scripts/package.json + scripts/node_modules Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: validate Claude executable with --version and detect desktop app Extract findClaudeExecutable() into shared utility used by both SDKAgent and KnowledgeAgent (deduplication). Every candidate is now validated with --version (3s timeout). Desktop app executables in AppData/Program Files get an actionable error message directing users to install the CLI via npm. Closes #2222 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: use Zod schemas in OpenCode plugin to fix _zod.def crash OpenCode 1.14.x walks arg._zod.def at plugin registration, which crashes on plain JSON Schema objects like {type: "string"}. Replace with z.string().describe() so the Zod internals are present. Closes #2226, #2225, #2154 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: neutralize chroma-mcp CPU storm at the root Two surgical fixes to the chroma backfill path that together cause the sustained 60–80% CPU + orphan accumulation pattern reported across 1. ChromaMcpManager.getSpawnEnv: cap embedding-thread fanout ONNX Runtime / OpenBLAS / MKL all default to cpu_count(), so a 12-core machine spins 12 threads burning embeddings concurrently. The user's getSpawnEnv only handled SSL certs — no thread limits at all. Inject OMP_NUM_THREADS / ONNX_NUM_THREADS / OPENBLAS_NUM_THREADS / MKL_NUM_THREADS defaults of 2 (only if user hasn't pinned them), and ANONYMIZED_TELEMETRY=false to stop background HTTP from the embedding subprocess. Closes the storm at the source. 2. ChromaSync.backfill{Observations,Summaries,Prompts}: per-batch watermark The bump was in a trailing finally block. SIGKILL / OOM / power loss mid-flight skips finally entirely, so the watermark stayed at 0 and the next worker boot re-embedded the entire history (16K obs in #2220's case), which then pegged CPU forever in combination with (1). Move the bump inside the loop so progress is durable per batch. Closes #2214. Verification: - 26/26 chroma tests pass (tests/services/sync, tests/integration/chroma-vector-sync) - Bundle confirms thread caps and per-batch bumps are present - Full suite: 1429 pass / 20 fail — pre-existing failures only, no regression vs v12.4.9 baseline (1429 pass / 27 fail) Closes #2214. Substantially de-amplifies #2220 (the structural Job-Object cleanup is still tracked separately at #2216). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: kill chroma-mcp process tree and limit backfill concurrency Three fixes for orphan chroma-mcp processes and resource exhaustion: 1. killProcessTree() in ChromaMcpManager.stop() tears down the full uvx->uv->python->chroma-mcp spawn chain (pkill -P on POSIX, taskkill /T on Windows) before MCP client.close(). 2. Register chroma process with pgid for supervisor shutdown cascade. 3. backfillAllProjects() now processes max 3 projects concurrently with a re-entrancy guard to prevent overlapping fire-and-forget runs. Closes #2216, advances #2220, #2213 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * build: regenerate plugin artifacts after cherry-picks Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: foundation consumers + Cursor/stdin/queue/docs fixes F1 spawnHidden adoption (#2236): - 8 spawn → spawnHidden conversions across worker-utils, ProcessManager, npx-cli (install/runtime), supervisor/process-registry F3 getUptimeSeconds adoption (#2250): - Server.ts:165 (THE BUG: returned ms) - Server.ts:270, SessionRoutes.ts:326 (4th ms-bug consumer found), DataRoutes.ts:225 (refactor for consistency) #2188 stdin '{}' fallback removal: - Diagnostic logging to <DATA_DIR>/logs/runner-errors.log + CAPTURE_BROKEN marker; exit 0 to preserve Windows Terminal exit-code strategy #2196 ANTHROPIC_BASE_URL docs: - New docs/public/configuration/custom-anthropic-backends.mdx - Note: issue may need separate auto-detect feature; docs document existing plumbing only #2242 check-pending-queue endpoints: - Point at /api/processing-status + /api/processing per DataRoutes.ts; honor CLAUDE_MEM_WORKER_PORT env #2248 Cursor sessions never summarized: - Pulled reporter wbingli's tested fix (commit 46eaba44) - Bug A: cursor adapter now derives transcriptPath from cwd+sessionId - Bug B: parser accepts both line.type and line.role - Bug C: walk backward, prefer non-empty text, fallback to empty - Tests: 10-case regression suite + tests/fixtures/cursor-session.jsonl Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: F2 paths namespace adoption (#2237 + #2238) Replaced 24 hardcoded homedir() + '.claude-mem' sites across 18 source files with paths.<accessor>() calls from src/shared/paths.ts. Accessors used: dataDir, workerPid, settings, database, chroma, combinedCerts, transcriptsConfig, transcriptsState, corpora, supervisorRegistry, envFile, logsDir. Sites converted (file:area): - src/cli/claude-md-commands.ts (database) - src/services/context/ContextConfigLoader.ts (settings) - src/services/infrastructure/ProcessManager.ts (workerPid) - src/services/infrastructure/WorktreeAdoption.ts (settings) - src/services/integrations/CodexCliInstaller.ts (settings) - src/services/sync/ChromaMcpManager.ts (chroma + combinedCerts) - src/services/transcripts/config.ts (transcriptsConfig + transcriptsState) - src/services/worker/ClaudeProvider.ts (envFile) - src/services/worker/GeminiProvider.ts (envFile + 2 more) - src/services/worker/http/routes/DataRoutes.ts (dataDir) - src/services/worker/http/routes/SettingsRoutes.ts (settings + envFile) - src/services/worker/knowledge/CorpusStore.ts (corpora) - src/shared/EnvManager.ts (envFile) - src/supervisor/index.ts (supervisorRegistry) - src/supervisor/process-registry.ts (supervisorRegistry) - src/supervisor/shutdown.ts (supervisorRegistry) - src/utils/claude-md-utils.ts (database) - src/utils/logger.ts (logsDir + settings, lazy to avoid cycle) CLAUDE_MEM_DATA_DIR override now flows through 100% of the worker runtime; no per-file env reads needed. Verification: - Grep guard: zero homedir+'.claude-mem' sites remain in src/ (excluding paths.ts itself and SettingsDefaultsManager.ts) - F2 invariant test: 3/3 pass (60 expects) - Foundation tests: 19/19 pass Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: F4 provider classification + parser fence + OAuth keychain F4 adoption (#2244 + #2254): - Per-provider classifiers: classifyClaudeError, classifyGeminiError, classifyOpenRouterError. Each lives in the provider file. - New retry helper at src/services/worker/retry.ts: withRetry() honors ClassifiedProviderError.kind; retriable=transient/rate_limit (with retryAfterMs); not retriable=unrecoverable/auth_invalid/quota_exhausted. maxRetries=2, perAttemptTimeout=30s, exponential backoff with jitter. - GeminiProvider + OpenRouterProvider fetch calls wrapped with retry. Best-effort request-id capture (x-goog-request-id, x-request-id, x-openrouter-request-id) for dedup logging. - Deleted unrecoverablePatterns allowlist at worker-service.ts:540 area; worker dispatches on err.kind instead. - 28 new classifier tests at tests/worker/provider-classifiers.test.ts: 429-no-Retry-After, 500-with-quota-exceeded, OverloadedError, per-provider auth_invalid signals. #2233 Part A — parser fence handling: - src/sdk/prompts.ts: removed 4 fence markers from XML example blocks. Model now sees plain XML, eliminating the failure-mode that drained quota via repeated retries. - src/sdk/parser.ts: stripCodeFences() at top, called before parseAgentXml. Fence-tolerant regardless of model behavior. - TODO comment references #2233 Part B (tool-use migration as separate scope). - 4 fence-tolerance tests added to tests/sdk/parser.test.ts. #2215 OAuth token keychain: - New src/shared/oauth-token.ts (~360 LOC): readClaudeOAuthToken() reads from platform-native credential stores at worker spawn-time. - macOS: security find-generic-password -s "Claude Code-credentials" - Windows: PowerShell wrapper around CredRead (Win32 Advapi32.dll) - Linux: secret-tool lookup - Fallback: env CLAUDE_CODE_OAUTH_TOKEN with JWT exp claim or sidecar expiresAt validation; refuses stale-token injection. - EnvManager.buildIsolatedEnvWithFreshOAuth() (async) replaces silent process.env copy. Empty injection on absent; marker write on expired. - <DATA_DIR>/oauth-stale.marker surfaces "re-login via Claude Desktop" via existing SessionStart additionalContext mechanism (context.ts). - ClaudeProvider.startSession + KnowledgeAgent.prime/executeQuery now await the async env builder. - 17 oauth-token tests covering decodeJwtExpMs, marker round-trip, env-fallback expiry detection. Verification: - npx tsc --noEmit: only pre-existing bun-types error - bun test (foundations + new): 70 pass, 0 new fails (8 fails are pre-existing parser.test.ts cases unrelated to fence work) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: #2234 quota-aware wall-clock guard New src/services/worker/RateLimitStore.ts (207 LOC) — vendor pattern from meridian/rateLimitStore.ts (MIT, copied not depended). API: - class RateLimitStore: set/get/getAll/getMostRecentByWindow/size/clear, in-memory last-write-wins keyed by rateLimitType. - globalRateLimitStore singleton. - shouldAbortForQuota(authMethod, store, now?) → {abort, reason?, window?} - isApiKeyAuth(authMethod): matches both verbose getAuthMethodDescription strings and concise "api_key". Thresholds (auth-type gated): - api_key: never aborts (user authorized per-call spend). - cli/oauth/subscription: - five_hour utilization >= 0.95 OR resetsAt within 15min (with 0.85 utilization floor to avoid false trip on freshly-reset windows) - seven_day_opus >= 0.93 - seven_day_sonnet >= 0.92 - seven_day >= 0.93 - overage >= 0.95 ClaudeProvider integration (line 198, for-await loop): - Detects message.type === 'system' && subtype === 'rate_limit' - Records rate_limit_info via globalRateLimitStore.set - Calls shouldAbortForQuota(authMethod, globalRateLimitStore) - On abort: session.abortReason = 'quota:<window>', abortController.abort, break out of loop. Worker continues other sessions. Health endpoint (Server.ts:174): - New rateLimits field on /api/health from getMostRecentByWindow(). - Field shape: {five_hour?, seven_day?, seven_day_opus?, seven_day_sonnet?, overage?} each carrying utilization, status, resetsAt, observedAt. Tests (tests/worker/rate-limit-store.test.ts): - 22 cases covering store CRUD, isApiKeyAuth, abort decision matrix. - api_key never aborts at any utilization. - cli aborts at threshold breaches per window. - Reset-grace buffer with utilization floor. Verification: - npx tsc --noEmit: only pre-existing bun error - bun test tests/worker/rate-limit-store.test.ts: 22/22 pass - bun test tests/claude-provider-resume.test.ts: 9/9 pass - bun test tests/server/: 44/44 pass Plugin artifacts regenerated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * build: regenerate worker-service.cjs after final build-and-sync Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: align test assertions with F4 classification + timeout Two test fixes for branch-introduced regressions vs main: 1. tests/gemini_provider.test.ts "should throw on other errors": F4's classifyGeminiError replaced upstream Error message with ClassifiedProviderError. Test was pinned to pre-F4 string. Updated assertion to match new "Gemini bad request (status 400)". 2. tests/infrastructure/graceful-shutdown.test.ts: Test pokes real ~/.claude-mem/supervisor.json registry which on a developer machine contains live worker + chroma-mcp PIDs. SIGTERM → wait → SIGKILL cascade takes ~6s end-to-end. Bumped per-test timeout to 15000ms. Underlying shutdown code unchanged. Future cleanup should mock getSupervisor() here. Result: branch failure count == main (77 pre-existing failures). No new regressions from this branch's work. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review: address 4 Greptile P1/P2 findings on PR #2282 P1 (real bug): clearStaleMarker silently broken in ESM - src/shared/oauth-token.ts:14: add unlinkSync to top-level fs import - src/shared/oauth-token.ts:342: drop inline require('fs'), call unlinkSync directly. ESM has no require, so the previous code threw ReferenceError swallowed by try/catch — making clearStaleMarker a permanent no-op. Stale oauth marker would persist indefinitely after Claude Desktop refreshed the token. P2 (security): execSync shell-string interpolation - src/shared/find-claude-executable.ts:39: execSync(`"${candidate}" --version`) → execFileSync(candidate, ['--version']). Path containing ", ;, & — reachable on Windows via crafted CLAUDE_CODE_PATH in settings.json — would otherwise produce a malformed/exploitable command. P2 (security): PowerShell username injection - src/shared/oauth-token.ts:119: userInfo().username escaped with PS single-quote convention (' → '') before interpolation into `'Claude Code-credentials:${user}'`. Defensive against future Windows versions or domain-joined machines that may permit ' in usernames. P2 (style): Unreachable throw lastError post-loop - src/services/worker/retry.ts:109: explained as the safety net for opts.maxRetries < 0 (pathological input where the loop never executes and lastError is undefined). Annotated with comment + descriptive fallback Error so the dead-looking code is now self-documenting. Verification: - npx tsc --noEmit: clean (only pre-existing bun-types error) - bun test tests/shared/oauth-token.test.ts tests/worker/provider-classifiers.test.ts tests/worker/provider-errors.test.ts: 50 pass / 0 fail Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review: tighten SECURITY.md data-flow and audit dates Fixes CodeRabbit comments #3178957249 (Data Storage section overstated "no external transmission" — softened to call out Claude Agent SDK, alternate provider, Chroma MCP, OAuth keychain, and registry fetches) and #3178957250 (Next Scheduled Audit was earlier than Last Updated; bumped Last Updated to 2026-05-03 and audit to 2026-09-16) on PR #2282. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review: drop inline require('fs') in paths.ts Fixes CodeRabbit outside-diff comment on src/shared/paths.ts:25-29 from PR #2282 review. resolveDataDir() ran require('fs') inside an ESM module (this file uses import.meta.url and .js imports), which can break in strict ESM environments. readFileSync now imports at the top alongside existsSync/mkdirSync. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review: block CLAUDE_CODE_OAUTH_TOKEN from parent env (issue #2215) Fixes CodeRabbit outside-diff comment on src/shared/EnvManager.ts:14-17 from PR #2282 review. The OAuth-token leak fix was bypassed because buildIsolatedEnv() copied every parent env var that wasn't in BLOCKED_ENV_VARS, but CLAUDE_CODE_OAUTH_TOKEN was not blocked. A stale parent token therefore still reached isolatedEnv even when the fresh keychain read returned expired/absent — defeating the fix documented inline at lines 178-183. Adds CLAUDE_CODE_OAUTH_TOKEN to BLOCKED_ENV_VARS and defensively deletes it again at the top of buildIsolatedEnvWithFreshOAuth() so the fresh-spawn-time read is the only path that can populate it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review: validate cursor sessionId against path traversal Fixes CodeRabbit comment #3178957252 on PR #2282. The Cursor adapter took sessionId straight from stdin and concatenated it into a join(homedir(), '.cursor', 'projects', ..., sessionId, ...) path. A crafted value containing path separators or '..' segments could escape ~/.cursor/projects, and the later transcript read would then probe arbitrary local files. deriveCursorTranscriptPath() now rejects any sessionId that doesn't match /^[A-Za-z0-9_-]+$/ — Cursor's real session ids are UUID-style identifiers, so the safe whitelist is non-disruptive. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review: scope stripCodeFences() to full-wrapper payloads only Fixes CodeRabbit comment #3178957253 on PR #2282. The previous regex greedily removed the first opening and last closing triple-backticks anywhere in the input, which could mangle valid content with internal fenced examples or surrounding prose — and ran before XML parsing so it created false negatives. stripCodeFences() now only strips when the entire payload is a single fenced block (start-to-end, with optional language tag and surrounding whitespace), capturing the inner content. Adds a regression test that feeds prose with internal triple-backtick markers around a real <observation> block and asserts the inner ``` are preserved. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review: honor abortSignal during retry backoff sleep Fixes CodeRabbit comment #3178957263 on PR #2282. The retry helper used an unconditional `setTimeout` Promise for backoff between attempts, so an external abort that fired during the wait was delayed until the timer completed. The backoff now races setTimeout against opts.abortSignal: if the signal flips, the timer is cleared and the Promise rejects with 'Aborted' immediately. The abort listener is registered with { once: true } and removed when the timer fires to avoid leaks. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review: abort immediately on provider-side rejected status Fixes CodeRabbit comment #3178957261 on PR #2282. shouldAbortForQuota() only checked utilization thresholds and reset-grace heuristics; a snapshot with status='rejected' (or overageStatus='rejected' on the overage window) but no utilization number could still return { abort: false }, letting the worker keep consuming after the provider had already declared the bucket exhausted. Provider-side rejection is now checked before utilization. When either rejection signal is present the guard returns abort=true with reason "quota:<window> rejected by provider". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review: only bump Chroma watermark on confirmed batch writes Fixes CodeRabbit comments #3178957259 (watermark advances on swallowed batch failures) and #3178957260 (backfillInProgress can stick true if init throws) on PR #2282. addDocuments() previously logged and swallowed per-batch failures with a void return type, so all three backfill loops (observations, summaries, prompts) bumped the watermark unconditionally after the call — turning a transient Chroma failure into permanently-skipped records. addDocuments() now returns the count of documents that actually landed (including delete+add reconcile retries), and each loop only advances the watermark when the batch wrote successfully. Failed batches log a debug message and continue so the loop still gets through the rest. backfillAllProjects() now constructs SessionStore and ChromaSync inside a try block so a constructor throw can't leave the static backfillInProgress guard stuck true and silently skip every later backfill. The finally always clears the guard and best-effort closes each resource. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review: fall back to pid kill when process group is gone Fixes CodeRabbit outside-diff comment on src/supervisor/shutdown.ts:118-134 from PR #2282 review. signalProcess() returned silently when a pgid was present and process.kill(-pgid, signal) threw ESRCH, never attempting the per-pid signal. With the new chroma registration path that records a pgid alongside the pid, an already-collapsed group could turn shutdown into a no-op even though the root pid was still alive. The POSIX branch now tries -pgid first when present, and on ESRCH falls through to process.kill(pid, signal). Non-ESRCH errors still propagate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review: settings path, uptime clamp, fetch timeouts Fixes three smaller CodeRabbit issues on PR #2282: - SettingsRoutes (outside-diff #2282 review on lines 65-79): the parse-error response told users to delete ~/.claude-mem/settings.json even when paths.settings() resolved elsewhere. Now uses the resolved settingsPath variable in the message. - uptime.ts (#3178957264 / lines 2-3): getUptimeSeconds() could return a negative value if startedAtMs was in the future or the system clock moved backward. Clamps with Math.max(0, ...) so health endpoints never see negative seconds. - check-pending-queue.ts (#3178957248 / lines 27-45): checkWorkerHealth, getProcessingStatus and triggerProcessing all called fetch with no timeout, so the script could block forever if the worker accepted the TCP connection but never responded. Wraps each fetch with an AbortController + 10s timeout that throws a clear timeout message. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review: walk descendants recursively when killing chroma-mcp tree Fixes CodeRabbit comment #3178957258 on PR #2282. The POSIX teardown in ChromaMcpManager.killProcessTree() relied on `pkill -P <pid>`, which only signals direct children. Under uv, chroma-mcp spawns python as a grandchild — when uv exits and python re-parents to init, pkill -P never reaches it and the descendant survives the "tree kill". killProcessTree() now collects the full descendant set via a recursive `pgrep -P` walk before each signal phase. The walk returns leaves first so signals propagate bottom-up (SIGTERM children before their parents, then again for SIGKILL after the 500ms grace window so any layer that re-parented during teardown still gets cleaned up). pgrep failures (no children, missing binary) return [] so this stays best-effort and falls back to the existing per-pid signal. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review: tolerate malformed JSONL lines in transcript-parser Fixes Greptile P1 comment 3178964456 on PR #2282. extractLastMessageFromJsonl previously called JSON.parse(rawLine) with no guard. A truncated/malformed JSONL line — common when a transcript was crashed mid-write or partially flushed — would throw SyntaxError, crash the summarization pipeline for that session, and silently lose all prior valid messages. Fix: wrap JSON.parse in try/catch and skip bad lines. The empty-line guard only catches truly empty strings, not malformed fragments. Regression tests added for two cases: - Mixed valid + truncated lines: returns last valid match. - All lines malformed: returns empty string (no throw). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review: classify FK constraint failures BEFORE provider classifier Fixes Greptile P1 comment 3178979583 on PR #2282. The F4 #2244 work introduced a regression: reclassifyAtDispatch always returns a non-null ClassifiedProviderError for known agent types (Claude/Gemini/OpenRouter), so the isFkConstraintFailure branch was dead code. Per-provider classifiers don't recognize "FOREIGN KEY constraint failed", so SQLite FK failures fell through to the default 'transient' kind and would retry indefinitely — restart loop on corrupted session DB state. Old unrecoverablePatterns explicitly listed FK constraint as unrecoverable; restoring that semantic by checking FK FIRST and only deferring to the classifier when not an FK error. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review: validate CLAUDE_MEM_WORKER_PORT in check-pending-queue Parse the env var, range-check (1-65535), and fall back to 37777 with a console.warn on invalid input instead of letting a malformed value flow into the URL builder unchecked (CodeRabbit Minor on PR #2282). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review: SIGKILL union of pre-TERM and post-wait descendant sets When the chroma-mcp root exits during the SIGTERM grace window, its descendants get re-parented to init and drop out of the post-wait pgrep -P scan. Without including the pre-TERM snapshot, those re-parented PIDs would never receive SIGKILL even though they were definitely children before SIGTERM and may still be alive (CodeRabbit Major on PR #2282). Compute Array.from(new Set([...descendantsBeforeTerm, ...descendantsBeforeKill])) and SIGKILL the union. The two sets typically overlap, so dedupe is required. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review: enforce addDocuments return-count in direct sync paths syncObservation/syncSummary/syncUserPrompt now capture the written count from addDocuments() and only bump the watermark when every requested document landed in Chroma. addDocuments() tolerates per-batch failures (returns the actual written count), so the previous unconditional bump was silently marking unsynced rows as synced on transient errors — preventing the next backfill from retrying them (CodeRabbit Major on PR #2282). A partial write now logs a warn with the (requested, written) pair and preserves retryability on the next pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review: guard backfill watermark against non-contiguous failures The backfill watermark is a single monotonic id, so it cannot represent sparse success: "synced through 200, gap at 201–250, then 251 onward" would, on restart, skip 201–250 forever because the watermark sat at either 200 or 251 — both lose data (CodeRabbit Major on PR #2282). Add a per-loop hadGap flag to backfillObservations / backfillSummaries / backfillPrompts. Once any batch under-writes, every subsequent batch must also skip the bump, regardless of whether it itself succeeded. Also tighten the failure check from `writtenInBatch <= 0` to `writtenInBatch < batch.length` so partial-batch writes are caught. The watermark stays at the last contiguously-synced position; the next backfill pass retries from there, eventually closing the gap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review: clear oauth-stale marker when token is absent When an OAuth token disappears entirely (user logs out, keychain cleared), buildIsolatedEnvWithFreshOAuth's absent branch was leaving any prior stale-marker file in place. The session-start hook would then keep surfacing an "expired token, re-login" warning even though the token is no longer expired — it's gone, and re-login was already done elsewhere or not applicable (CodeRabbit Minor on PR #2282). Call clearStaleMarker() in the absent branch the same way the present branch already does. Add a regression test exercising the full buildIsolatedEnvWithFreshOAuth path: pre-write a marker, force absent via spoofed unsupported platform, assert the marker is gone after. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review: skip unknown message.content shapes instead of throwing extractLastMessageFromJsonl already tolerates malformed JSONL lines (JSON.parse failure -> continue), but a valid JSON line whose message.content is an unexpected type (null, number, plain object) was still throwing — contradicting the new tolerance and crashing the entire summary pipeline on a single weird line (CodeRabbit Major + Greptile P1 on PR #2282). Replace the `throw new Error(...)` with `continue` so a single bad content shape skips that line instead of failing the whole transcript read. Forward compat: future content schemas land harmlessly. Add regression tests covering null, number, and plain-object content; each must not throw and must fall back to the most recent valid line. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review: guard null/primitive entries in message.content array Fixes CodeRabbit comment 3179004190 on PR #2282. The Array.isArray branch previously did `c.type === 'text'` directly, which throws if `c` is null or a primitive — possible in malformed logs. Tightened the filter with a type guard: requires c to be a non-null object with type === 'text' and a string text field. Same defensive class as the malformed-line and unknown-content-shape tolerances. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Alex Newman
|
9e2973059a |
UX redesign: installer + provider rename + /learn-codebase + welcome card + SessionStart hint (#2255)
* feat(ux): claude-mem UX improvements with installer enhancements
Squashed PR #2156 commits for clean rebase onto main:
- feat(installer): add provider selection, model prompt, worker auto-start
- refactor: rename *Agent provider classes to *Provider
- feat: add /learn-codebase skill and viewer welcome card
- feat(worker): inject welcome hint when project has zero observations
- fix(pr-2156): address greptile review comments
- fix(pr-2156): address coderabbit review comments
- fix(pr-2156): persist CLAUDE_MEM_PROVIDER for non-claude in non-TTY mode
- fix(pr-2156): file-backed settings reads in installer + env-first SKILL doc
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* build: rebuild plugin artifacts after rebase onto v12.4.7
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(skills): strip claude-mem internals from learn-codebase
The learn-codebase skill, install next-step copy, WelcomeCard, and
welcome-hint previously walked the primary agent through worker endpoints
and synthetic observation payloads. The PostToolUse hook already captures
every Read/Edit the agent makes — the agent should have no awareness that
the memory layer exists. Collapse the skill to one instruction ("read every
source file in full") and rephrase touchpoints to describe only what the
user observes (Claude reading files), not what happens behind the scenes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sync): preflight version mismatch + settings-aware port resolution
Two related fixes for build-and-sync's worker restart step:
1. Read CLAUDE_MEM_WORKER_PORT from ~/.claude-mem/settings.json the same
way the worker does, instead of computing the default port from the
uid alone. Previously, users with a custom port saw a misleading
"Worker not running" message because the restart POST hit the wrong
port and got ECONNREFUSED.
2. Add a preflight check that aborts the sync when the running worker's
reported version does not match the version we are about to build.
Claude Code's plugin loader pins the worker to a specific cache
version per session, so syncing into a newer cache directory has no
effect until the user runs `claude plugin update thedotmack/claude-mem`
to bump the pin. The preflight surfaces this explicitly with the exact
command to run; --force bypasses it for intentional cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(learn-codebase): note sed for partial reads of large files
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: strip comments codebase-wide
Removed prose comments from all tracked source. Preserved directives
(@ts-ignore, eslint-disable, biome-ignore, prettier-ignore, triple-slash
references, webpack magic, shebangs). Deleted two tests that asserted
on comment text rather than runtime behavior.
Net: 401 files, -14,587 / +389 lines, -10.4% bytes.
Verified: typecheck passes, build passes, test count unchanged from
baseline (22 pre-existing fails, all unrelated).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(installer): move runtime setup into npx, eliminate hook dead air
Smart-install ran 3 times during a fresh install — the worst run was silent,
fired by Claude Code's Setup hook after `claude plugin install`, producing
~30s of dead air that looked like the plugin was hung.
This change makes `npx claude-mem install` the single place heavy work
happens, with a visible spinner. Hooks become runtime-only.
- New `src/npx-cli/install/setup-runtime.ts` module: ensureBun, ensureUv,
installPluginDependencies, read/writeInstallMarker, isInstallCurrent.
Marker schema preserved exactly ({version, bun, uv, installedAt}) so
ContextBuilder and BranchManager readers keep working.
- `npx claude-mem install`: ungated copy/register/enable for every IDE,
inserts a "Setting up runtime" task with honest "first install can take
~30s" spinner. The claude-code shell-out to `claude plugin install` is
removed — npx already populated everything Claude reads.
- New `npx claude-mem repair` command for post-`claude plugin update`
recovery, force-reinstalls runtime.
- Setup hook now runs `plugin/scripts/version-check.js` (29ms wall) instead
of smart-install. Mismatch prints "run: npx claude-mem repair" on stderr.
Always exits 0 (non-blocking, per CLAUDE.md exit-code strategy).
- SessionStart loses the smart-install entry; 2 hooks remain (worker start,
context fetch).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(installer): delete smart-install sources, retarget tests
- Delete scripts/smart-install.js + plugin/scripts/smart-install.js (both
are source files kept in sync manually; both must go).
- Delete tests/smart-install.test.ts (covered surface is gone).
- tests/plugin-scripts-line-endings: drop smart-install.js entry.
- tests/infrastructure/plugin-distribution: retarget two assertions at
version-check.js (the new Setup hook script).
- New tests/setup-runtime.test.ts: 9 tests covering marker read/write,
isInstallCurrent semantics. Marker schema invariant verified.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(installer): describe npx-driven setup + version-check Setup hook
Sweep public docs and architecture notes to reflect the new flow:
npx installer does Bun/uv setup with a visible spinner; Setup hook runs
sub-100ms version-check.js; users hit `npx claude-mem repair` after a
`claude plugin update`.
- docs/architecture-overview.md: hook lifecycle table + npx flow paragraph
- docs/public/configuration.mdx: tree + hook config example
- docs/public/development.mdx: build output line
- docs/public/hooks-architecture.mdx: full rewrite of pre-hook section,
timing table, performance table
- docs/public/architecture/{overview,hooks,worker-service}.mdx: tree
comments, JSON config example, Bun requirement section
docs/reports/* untouched (historical incident reports).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(install): mergeSettings writes via USER_SETTINGS_PATH
Greptile P1 (#2156): `settingsFilePath()` only resolved
`process.env.CLAUDE_MEM_DATA_DIR`, while `getSetting()` reads via
`USER_SETTINGS_PATH` which `resolveDataDir()` populates from BOTH the env
var AND a `CLAUDE_MEM_DATA_DIR` entry persisted in
`~/.claude-mem/settings.json`. Result: a user with the data dir saved in
settings.json but not exported in their shell would have provider/model
settings silently written to `~/.claude-mem/settings.json` while
`getSetting()` read from `/custom/path/settings.json` — read/write split.
Drop `settingsFilePath()` and the now-unused `homedir` import; reuse the
already-imported `USER_SETTINGS_PATH` constant.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cli): parse --provider, --model, --no-auto-start install flags
Greptile P1 (#2156): InstallOptions has fields `provider`, `model`,
`noAutoStart`, but the install case in the npx-cli switch only parsed
`--ide`. The other three flags were silently dropped — `npx claude-mem
install --provider gemini` was a no-op.
Extract a `parseInstallOptions(argv)` helper, share it between the bare
`npx claude-mem` and `npx claude-mem install` paths, and validate
`--provider` against the allowed set. Update help text accordingly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(install): pipe runtime-setup output, always show IDE multiselect
Two issues caught in a docker test of the installer:
1. The bun.sh installer, uv installer, and `bun install` were using
stdio: 'inherit', dumping their stdout/stderr through clack's spinner
region — visible as raw "downloading uv 0.11.8…" / "Checked 58
installs across 38 packages…" text streaming under the spinner. Switch
to stdio: 'pipe' and surface captured stderr only on failure (via a
shared describeExecError() helper that includes stdout when stderr is
empty). Spinner stays clean on the happy path.
2. promptForIDESelection() silently picked claude-code when no IDEs were
detected, never showing the user the multiselect. On a fresh machine
with no IDEs present yet (e.g. our docker test container), the user
never got to choose. Now: always show the full IDE list when
interactive; mark detected ones with [detected] hints and pre-select
them; show a warn line if zero are detected explaining they should pick
what they plan to use. Non-TTY callers still get the silent
claude-code default at the call site (unchanged).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(install): skip marketplace work for claude-code-only, offer to install Claude Code
Two related UX fixes from a docker test:
**Delay between "Saved Claude model=…" and "Plugin files copied OK"**
After dropping the needsManualInstall gate, every install was unconditionally
running `copyPluginToMarketplace` (which copied the entire root node_modules
tree — thousands of files, dozens of seconds) and `runNpmInstallInMarketplace`
(npm install --production) even when only claude-code was selected. Neither
is needed for claude-code: that path uses the plugin cache dir + the
installed_plugins.json + enabledPlugins flag, all of which we already write.
- Drop `node_modules` from `copyPluginToMarketplace`'s allowed-entries list;
the dependency-install task populates it on the destination side anyway.
- Re-introduce `needsMarketplace = selectedIDEs.some(id => id !== 'claude-code')`
scoped *only* to `copyPluginToMarketplace`, `runNpmInstallInMarketplace`,
and the pre-install `shutdownWorkerAndWait` (also pointless for claude-code-
only flows since we're not overwriting the worker's running cache dir
source). All other tasks (cache copy, register, enable, runtime setup) stay
unconditional.
**Claude Code missing → silent install of an IDE that isn't there**
When the user picked claude-code on a machine without it (e.g. a fresh
container), the install completed but `claude` was unavailable and the only
hint was a generic warn line. Replace with an explicit pre-flight prompt:
Claude Code is not installed. Claude-mem works best in Claude Code, but
also works with the IDEs below.
? Install Claude Code now?
◆ Yes — install Claude Code (recommended)
◯ No — pick another IDE below
◯ Cancel installation
If the user picks "Yes", run `curl -fsSL https://claude.ai/install.sh | bash`
(or the PowerShell equivalent on Windows), then re-detect IDEs and proceed
with claude-code pre-selected. If the install fails or the user picks "No",
the multiselect still appears with claude-code visible (just unmarked
[detected]), so they can opt in or pick another IDE.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(install): detect Claude Code via `claude` CLI, not ~/.claude dir
The directory `~/.claude` can exist (e.g. mounted in Docker, or created
by tooling) without Claude Code actually being installed. Detect the
`claude` command in PATH instead so the installer correctly offers to
install Claude Code when missing.
* docs(learn-codebase): add reviewer note explaining the cost tradeoff
The skill intentionally reads every file in full to build a cognitive
cache that pays off across the rest of the project. Add a brief note
so reviewers (human or bot) understand the tradeoff before flagging
the unbounded read as a cost issue.
* fix: address Greptile P1 feedback on welcome hint and learn-codebase
- SearchRoutes: skip welcome hint when caller passes ?full=true so
explicit full-context requests aren't intercepted by the hint.
- learn-codebase: replace `sed` instruction with the Read tool's
offset/limit parameters, since Bash is gated in Claude Code by
default.
* feat(install): ASCII-animated logo splash on interactive install
Plays a ~1s bloom animation of the claude-mem sunburst logomark when
the installer starts in an interactive terminal — geometrically rendered
via 12 ray curves around a center disc, in the brand orange. The
wordmark and tagline type on alongside the final frame.
Auto-skipped on non-TTY, in CI, when NO_COLOR or CLAUDE_MEM_NO_BANNER
is set, or when the terminal is too narrow.
Inspired by ghostty +boo.
* feat(banner): replace rotation frames with angular-sector bloom generator
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(banner): replace rotation frames with angular-sector bloom generator
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(banner): three-act choreography renderer with radial gradient and diff redraw
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(banner): update preview script to support small/medium/hero tier selection
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(docker): add COLORTERM=truecolor to test-installer sandbox
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(install): auto-apply PATH for Claude Code with spinner UX
The Claude Code install.sh prints a Setup notes block telling users to
manually edit "your shell config file" to add ~/.local/bin to PATH —
which left fresh installs unable to launch claude from the command line.
After a successful install, detect ~/.local/bin/claude on disk and, if
the dir is missing from PATH, append the right export line to .zshrc /
.bash_profile / .bashrc / fish config (idempotent, marked with a
comment). Also updates process.env.PATH for the current install run.
Wraps the curl|bash install in a clack spinner (interactive only) so the
~4 minute native-build download doesn't look frozen — output is captured
silently and dumped on failure for debuggability. Non-interactive mode
keeps inherited stdio for CI logs.
Verified end-to-end in the test-installer docker sandbox: spinner
animates, .bashrc gets the export, fresh login shell resolves claude.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(banner): video-frame ASCII renderer with three-act choreography
Generator switched from a single Jimp-rendered logo to pre-extracted
video frames concatenated with \x01 separators and gzip-deflated, ported
from ghostty's boo wire format. Renderer rewritten around three acts
(ignite → stagger bloom → text reveal + breathe) with adaptive sizing,
radial gradient, and diff-based redraw.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(onboarding): unify install / SessionStart / viewer around one first-success moment
Three surfaces now point at the same north-star moment — open the viewer, do
anything in Claude Code, watch an observation appear within seconds — with the
same verbatim timing and privacy lines, and a single canonical "how it works"
explainer instead of three diverging copies.
- Canonical explainer at src/services/worker/onboarding-explainer.md served via
GET /api/onboarding/explainer; mirrored into plugin/skills/how-it-works/SKILL.md
- SessionStart welcome hint rewritten as third-person status (no imperatives
Claude tries to execute), pinned with a default-value regression test
- Post-install Next Steps reframed as "two paths": passive default + optional
/learn-codebase front-load; drops /mem-search and /knowledge-agent from this
surface; adds verbatim timing + privacy lines and /how-it-works link
- /api/stats response gains firstObservationAt for the viewer stat row
- Viewer WelcomeCard branches on observationCount === 0: empty state shows live
worker-connection dot + "waiting for activity"; has-data state shows
observations · projects · since [date] and two example prompts. v2 dismiss key
- jimp added to package.json to fix pre-existing banner-frame build break
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(banner): play unconditionally; only honor CLAUDE_MEM_NO_BANNER
The 128-col / TTY / CI / NO_COLOR gates silently swallowed the banner in
narrower terminals, CI logs, and any non-TTY pipe — including Docker runs
where -it should preserve the experience but column width was the wrong
gate. Remove the implicit gates; keep the explicit opt-out only.
If a frame wraps in a narrow terminal, that's better than the banner
not playing at all.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* revert(banner): restore 15:33 gating logic per user request
Reverts eb6fc157. Restores isBannerEnabled to the state at commit
8e448015 (2026-04-30 15:33): TTY check, !CI, !NO_COLOR, !CLAUDE_MEM_NO_BANNER,
and cols >= BANNER.width.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(install): wrap remaining slow steps with spinners
Each IDE installer (Cursor, Gemini CLI, OpenCode, Windsurf, OpenClaw,
Codex CLI, MCP integrations) now runs inside a clack task spinner with
per-step progress messages instead of silent dynamic-import + cpSync.
Pre-overwrite worker shutdown (up to 10s) and the post-install health
probe (up to 3s) also get spinners.
Internal console.log/error/warn from each IDE installer is buffered
during the spinner; if the install fails, captured output is replayed
afterward via log.warn so users can see what broke.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(review): observation count + IDE pre-selection regressions
WelcomeCard's "no observations yet" empty state was triggered when a
project filter narrowed the feed to zero rows, even with thousands of
observations elsewhere. Source the count from global stats.database
to match firstObservationAt's scope.
Restore initialValues: [] in the IDE multiselect — pre-selecting every
detected IDE was the exact regression #2106 was filed for.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(install): trichotomy worker state + cache fallback for script path
ensureWorkerStarted now returns 'ready' | 'warming' | 'dead' instead of
boolean. The spawned-but-still-warming case (common in Docker cold
starts and slow first-time inits) was being misreported as 'did not
start', which contradicted the next-steps panel saying 'still starting
up'. Install task message and Next Steps headline now agree on the
actual state.
Also fixes the actual root cause of 'Worker did not start' on
claude-code-only installs: the worker script path was hardcoded to the
marketplace dir, which is left empty when no non-claude-code IDE is
selected. Now falls back to pluginCacheDirectory(version) when the
marketplace copy isn't present.
Verified end-to-end in docker/claude-mem with --ide claude-code,
--ide cursor, and a fresh container — install task and headline
agree on 'Worker ready at http://localhost:<port>' in all cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: align CLAUDE.md and public docs with current code
Sweep across CLAUDE.md and 10 high-traffic docs/public/ MDX files to
remove point-in-time references and align with the actual current
shape of the codebase. Highlights:
- Hardcoded port 37777 → per-user formula (37700 + uid % 100) on the
front-door pages (introduction, installation, configuration,
architecture/overview, architecture/worker-service, troubleshooting,
hooks-architecture, platform-integration).
- Default model 'sonnet' → 'claude-haiku-4-5-20251001' (matches
SettingsDefaultsManager).
- Node 18 → 20 (matches package.json engines).
- Lifecycle hook count corrected (5 events).
- Removed the nonexistent 'Smart Install' component and pre-built
directory tree referencing files that no longer exist
(context-hook.ts, save-hook.ts, cleanup-hook.ts, etc.); replaced
with the real worker dispatcher shape.
- Removed CLAUDE.md '#2101' issue tag (kept the design rationale).
- Replaced obsolete hooks.json example with a description of the real
bun-runner.js / worker-service.cjs hook event shape.
Lower-traffic doc pages still hardcode 37777 — left for a separate
global pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(scripts): land strip-comments around real parsers (postcss, remark, parse5)
Each language gets a real parser to locate comments, then we splice ranges
out of the original source. The library never serializes — that's how
remark-stringify produced 243 reformat-noise diffs in the first attempt
versus the 21 real strip targets here.
JS/TS/JSX -> ts.createSourceFile + getLeadingCommentRanges
CSS/SCSS -> postcss.parse + walkComments + node.source offsets
MD/MDX -> remark-parse (+ remark-mdx) + AST html / mdx-expression nodes
HTML -> parse5 with sourceCodeLocationInfo
shell/py -> kept hand-rolled hash stripper (no library worth the dep)
Preserves: shebangs, @ts-* directives, eslint-disable, biome-ignore,
prettier-ignore, triple-slash refs, webpack magic, /*! license keep,
@strip-comments-keep file marker. JS/TS handler runs a parse-roundtrip
check and refuses to write if syntax errors increased (catches the
worker-utils.ts breakage class from the 2026-04-29 attempt).
npm scripts:
strip-comments (apply)
strip-comments:check (CI-style, exits non-zero if changes needed)
strip-comments:dry-run (list, no writes)
Verified --check on this repo: 21 changes, -4.0% bytes, no parse-error
regressions, no reformat-suspect false positives.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: strip comments codebase-wide via parser-backed tool
21 files changed, -17,550 bytes (-4.0%) of narrative comments removed
across .ts / .tsx / .js / .mjs and the .gitignore. JS/TS comments stripped
via ts.createSourceFile + getLeadingCommentRanges — same canonical lexer,
same behavior as the 2026-04-29 strip, no reformat noise.
Preexisting baseline (unchanged):
typecheck: 16 errors at HEAD, 16 errors after strip (line numbers shift,
no new error classes — verified via diff of sorted error lists)
build: fails at HEAD with CrushHooksInstaller.js unresolved import
(preexisting, unrelated to this strip)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(install): drop Crush integration references after extract
The Crush integration was extracted to its own branch on May 1, but the
import at install.ts:280 (and the case block + ide-detection entry +
McpIntegrations config + npx-cli help text) still referenced the now-
removed CrushHooksInstaller.js, breaking the build.
Removes:
- case 'crush' block in install.ts
- crush entry in ide-detection.ts
- CRUSH_CONFIG and registration in McpIntegrations.ts
- 'crush' from the IDE Identifiers help line in index.ts
Rebuilds worker-service.cjs to match.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(banner): mark generated banner-frames.ts with @strip-comments-keep
Without this, every build/strip cycle ping-pongs five lines of doc
comments in and out of the auto-generated output. The keep-marker tells
strip-comments.ts to skip the file entirely.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(build): drop banner-frame regen from build script
generate-banner-frames.mjs requires PNG frames in /tmp/cmem-banner-frames
that only exist after the maintainer runs ffmpeg locally on the source
video. CI has neither the video nor the frames, so the build broke on
Windows. The output (src/npx-cli/banner-frames.ts) is committed, so the
regen is a one-shot dev step — not a build step. Run the script directly
when the video changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(worker): unstick the spinner — kill claim-self-lock, wake on fail, auto-broadcast
Three surgical changes that cure the stuck-spinner bug at the source.
Phase 1.1 (L9): claimNextMessage no longer self-excludes its own worker_pid.
A single UPDATE-RETURNING grabs the oldest pending row by id. Removes the
LiveWorkerPidsProvider plumbing that was never injected — Supervisor enforces
single-worker via PID file, so the multi-worker SQL was defending against a
configuration the project does not support.
Phase 1.2 (L19): SessionManager.markMessageFailed wraps PendingMessageStore.markFailed
and emits 'message' on the per-session EventEmitter. The iterator's waitForMessage
now wakes immediately on re-pend instead of parking for 3 minutes. ResponseProcessor
and SessionRoutes routed through the new wrapper.
Phase 1.3 (L24): PendingMessageStore takes an optional onMutate callback fired
from every mutator (enqueue, claimNextMessage, confirmProcessed, markFailed,
transitionMessagesTo, clearFailedOlderThan). SessionManager wires it; WorkerService
passes broadcastProcessingStatus. Ten manual broadcast calls deleted across
SessionCleanupHelper, SessionEventBroadcaster, SessionRoutes, DataRoutes, and
worker-service. Caller discipline becomes structurally impossible to forget.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(worker): delete dead code — legacy routes, processPendingQueues, decorative guards
Pure deletions. Phase 2 of kill-the-asshole-gates.
- Legacy /sessions/:sessionDbId/* routes (handleSessionInit, handleObservations,
handleSummarize, handleSessionStatus, handleSessionDelete, handleSessionComplete)
bypassed all five ingest gates and were a parallel write path. Folded the
initializeSession + broadcastNewPrompt + syncUserPrompt + ensureGeneratorRunning
+ broadcastSessionStarted work into the canonical /api/sessions/init handler so
the hook makes one round trip instead of two.
- processPendingQueues (~104 lines, zero callers) — replaced in Phase 6 by a
one-statement startup sweep.
- spawnInProgress Map and crashRecoveryScheduled Set — decorative dedupe over
generatorPromise and stillExists checks that already provide the real safety.
- STALE_GENERATOR_THRESHOLD_MS — pre-empted live generators and raced with the
finally block; the 3min idle timeout already kills zombies.
- MAX_SESSION_WALL_CLOCK_MS — ran a SELECT on every observation to enforce 24h.
Runaway-spend protection lives in the API key, not in claude-mem.
- Missing-id 400 in shared.ts ingestObservation — Zod already enforces min(1)
on contentSessionId and toolName at the route schema.
- SessionCompletionHandler import + completionHandler field on SessionRoutes
(orphaned after handler deletions).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(worker): SQL-backed getTotalQueueDepth — single source of truth
Was: iterate this.sessions.values() and sum getPendingCount per session.
Now: SELECT COUNT(*) FROM pending_messages WHERE status IN ('pending','processing').
The in-memory sessions Map drifted from the DB rows whenever a generator exited
without confirm/fail, leading to false-positive isProcessing in the UI. Phase 1.3's
auto-broadcast fires on every mutation, but it broadcast a stale Map count.
Reading from the DB makes the UI's spinner state match what the queue actually holds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(worker): typed abortReason replaces wasAborted boolean
Was: a boolean wasAborted that lumped every abort together. The finally block
branched on !wasAborted, so any abort skipped restart — including idle aborts
with pending work, which is exactly the case where we DO want to restart.
Now: ActiveSession.abortReason is a typed enum 'idle' | 'shutdown' | 'overflow'
| 'restart-guard'. The finally block consumes the reason and only skips restart
for 'shutdown' and 'restart-guard'. Idle and overflow aborts fall through, so
if pending work exists they trigger restart correctly.
Dropped 'stale' and 'wall-clock' from the union — Phase 2 deleted those paths.
Natural-completion abort (post-success) intentionally has no reason; it's not
gating restart logic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(worker): unify the two generator-exit finally blocks
Was: worker-service.ts:startSessionProcessor and SessionRoutes:ensureGeneratorRunning
each had their own ~70-line finally block with divergent restart-guard handling.
The worker-service path called terminateSession on RestartGuard trip and orphaned
pending rows (the L16 bug); the SessionRoutes path drained them. Two places to
update when rules changed.
Now: handleGeneratorExit in src/services/worker/session/GeneratorExitHandler.ts
owns the contract:
1. Always kill the SDK subprocess if alive.
2. Always drain processingMessageIds via sessionManager.markMessageFailed
(which wakes the iterator — Phase 1.2).
3. shutdown / restart-guard reasons: drain pending rows via
transitionMessagesTo('failed'), finalize, remove from Map. Fixes L16.
4. pendingCount=0: finalize normally and remove from Map.
5. pendingCount>0: backoff respawn via per-session respawnTimer (no global Set;
Phase 2.4 deleted that). RestartGuard trip drains to 'abandoned'.
Both finally blocks are now ~10-line wrappers that translate local state into the
canonical abortReason and delegate. Restored completionHandler injection into
SessionRoutes (was dropped in Phase 2 cleanup; needed by the unified helper for
finalizeSession).
Behavior change: SessionRoutes' previous "keep idle session in memory" was
deliberately replaced by the plan's "remove from Map on natural completion" —
next observation reinitializes via getMessageIterator → initializeSession.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(worker): startup orphan sweep — reset 'processing' rows at boot
When the worker dies (crash, kill, restart), any pending_messages rows it left
in 'processing' state are by definition orphans (the only worker is dead).
Single SQL UPDATE at boot resets them to 'pending' so the iterator can claim
them again. Replaces the deleted processPendingQueues function (Phase 2.2).
Runs in initializeBackground after dbManager.initialize() and before the
initializationComplete middleware releases blocked HTTP requests, so no
in-flight request can race the sweep. NOT on a periodic timer — after boot,
every 'processing' row has a live consumer and a periodic sweep would race.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(worker): simplify enqueue catch, replace memorySessionId throw with re-pend
7.1: queueObservation's catch was logging two ERROR-level messages and rethrowing.
The rethrow is correct (FK violations / disk full / schema drift should crash
loudly), but the verbose ERROR logging pretended the error was recoverable.
Reduced to one INFO line + rethrow.
7.2: ResponseProcessor's memorySessionId guard was throwing if the SDK hadn't
included session_id on the first user-yield, terminal-failing the entire batch.
Now warns and re-pends in-flight messages via sessionManager.markMessageFailed
(which wakes the iterator — Phase 1.2). The next iteration tries again with
memorySessionId hopefully captured.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sync): mirror builds to installed-version cache for hot reload
When package.json bumps past Claude Code's installed pin, sync-marketplace
wrote new code to cache/<buildVersion>/ but the worker loaded from
cache/<installedVersion>/, so worker:restart reloaded the same old code.
Replace the exit-on-mismatch preflight with a mirror step: when versions
differ, also rsync plugin/ into cache/<installedVersion>/ so worker:restart
hot-reloads new code without a Claude Code session restart. The
build-version cache still gets written for the eventual
`claude plugin update`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: delete dead barrel files and orphan utilities
- src/sdk/index.ts (re-exports parser+prompts; nothing imported the barrel)
- src/services/Context.ts (re-exports ./context/index.js; no importers)
- src/services/integrations/index.ts (no importers)
- src/services/worker/Search.ts (3-line barrel of ./search/index.js)
- src/services/infrastructure/index.ts: drop CleanupV12_4_3 re-export
- src/utils/error-messages.ts (getWorkerRestartInstructions never imported)
- src/types/transcript.ts (170 LoC of types, zero importers)
- src/npx-cli/_preview.ts (banner dev preview, no script wires it)
Build + tests still pass; observations still flowing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(parser): drop unused detectLanguage
Only the user-grammar-aware variant detectLanguageWithUserGrammars()
is actually called.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(types): drop unused SdkSessionRecord + ObservationWithContext
Both interfaces in src/types/database.ts had zero importers anywhere
in src or tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(npx-cli): drop unused getDetectedIDEs + claudeMemDataDirectory
getDetectedIDEs has no callers — install.ts uses detectInstalledIDEs
directly. claudeMemDataDirectory has no callers either.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(ProcessManager): drop dead orphan-reaper + signal-handler helpers
Each had zero callers in src/ or tests/:
- cleanupOrphanedProcesses + enumerateOrphanedProcesses
- ORPHAN_PROCESS_PATTERNS + ORPHAN_MAX_AGE_MINUTES
- forceKillProcess
- waitForProcessesExit
- createSignalHandler
- resetWorkerRuntimePathCache
The orphan reaper was retired in PATHFINDER Plan 02 ("OS process groups
replace hand-rolled reapers", commit
|
||
|
Alex Newman
|
266c746d50 |
feat: Fix observation timestamps, refactor session management, and enhance worker reliability (#437)
* Refactor worker version checks and increase timeout settings - Updated the default hook timeout from 5000ms to 120000ms for improved stability. - Modified the worker version check to log a warning instead of restarting the worker on version mismatch. - Removed legacy PM2 cleanup and worker start logic, simplifying the ensureWorkerRunning function. - Enhanced polling mechanism for worker readiness with increased retries and reduced interval. * feat: implement worker queue polling to ensure processing completion before proceeding * refactor: change worker command from start to restart in hooks configuration * refactor: remove session management complexity - Simplify createSDKSession to pure INSERT OR IGNORE - Remove auto-create logic from storeObservation/storeSummary - Delete 11 unused session management methods - Derive prompt_number from user_prompts count - Keep sdk_sessions table schema unchanged for compatibility * refactor: simplify session management by removing unused methods and auto-creation logic * Refactor session prompt number retrieval in SessionRoutes - Updated the method of obtaining the prompt number from the session. - Replaced `store.getPromptCounter(sessionDbId)` with `store.getPromptNumberFromUserPrompts(claudeSessionId)` for better clarity and accuracy. - Adjusted the logic for incrementing the prompt number to derive it from the user prompts count instead of directly incrementing a counter. * refactor: replace getPromptCounter with getPromptNumberFromUserPrompts in SessionManager Phase 7 of session management simplification. Updates SessionManager to derive prompt numbers from user_prompts table count instead of using the deprecated prompt_counter column. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: simplify SessionCompletionHandler to use direct SQL query Phase 8: Remove call to findActiveSDKSession() and replace with direct database query in SessionCompletionHandler.completeByClaudeId(). This removes dependency on the deleted findActiveSDKSession() method and simplifies the code by using a straightforward SELECT query. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: remove markSessionCompleted call from SDKAgent - Delete call to markSessionCompleted() in SDKAgent.ts - Session status is no longer tracked or updated - Part of phase 9: simplifying session management 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: remove markSessionComplete method (Phase 10) - Deleted markSessionComplete() method from DatabaseManager - Removed markSessionComplete call from SessionCompletionHandler - Session completion status no longer tracked in database - Part of session management simplification effort 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: replace deleted updateSDKSessionId calls in import script (Phase 11) - Replace updateSDKSessionId() calls with direct SQL UPDATE statements - Method was deleted in Phase 3 as part of session management simplification - Import script now uses direct database access consistently 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * test: add validation for SQL updates in sdk_sessions table * refactor: enhance worker-cli to support manual and automated runs * Remove cleanup hook and associated session completion logic - Deleted the cleanup-hook implementation from the hooks directory. - Removed the session completion endpoint that was used by the cleanup hook. - Updated the SessionCompletionHandler to eliminate the completeByClaudeId method and its dependencies. - Adjusted the SessionRoutes to reflect the removal of the session completion route. * fix: update worker-cli command to use bun for consistency * feat: Implement timestamp fix for observations and enhance processing logic - Added `earliestPendingTimestamp` to `ActiveSession` to track the original timestamp of the earliest pending message. - Updated `SDKAgent` to capture and utilize the earliest pending timestamp during response processing. - Modified `SessionManager` to track the earliest timestamp when yielding messages. - Created scripts for fixing corrupted timestamps, validating fixes, and investigating timestamp issues. - Verified that all corrupted observations have been repaired and logic for future processing is sound. - Ensured orphan processing can be safely re-enabled after validation. * feat: Enhance SessionStore to support custom database paths and add timestamp fields for observations and summaries * Refactor pending queue processing and add management endpoints - Disabled automatic recovery of orphaned queues on startup; users must now use the new /api/pending-queue/process endpoint. - Updated processOrphanedQueues method to processPendingQueues with improved session handling and return detailed results. - Added new API endpoints for managing pending queues: GET /api/pending-queue and POST /api/pending-queue/process. - Introduced a new script (check-pending-queue.ts) for checking and processing pending observation queues interactively or automatically. - Enhanced logging and error handling for better monitoring of session processing. * updated agent sdk * feat: Add manual recovery guide and queue management endpoints to documentation --------- Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com> |