8 Commits

Author	SHA1	Message	Date
Alex Newman	94d592f212	perf: streamline worker startup and consolidate database connections (#2122) ... * docs: pathfinder refactor corpus + Node 20 preflight Adds the PATHFINDER-2026-04-22 principle-driven refactor plan (11 docs, cross-checked PASS) plus the exploratory PATHFINDER-2026-04-21 corpus that motivated it. Bumps engines.node to >=20.0.0 per the ingestion-path plan preflight (recursive fs.watch). Adds the pathfinder skill. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: land PATHFINDER Plan 01 — data integrity Schema, UNIQUE constraints, self-healing claim, Chroma upsert fallback. - Phase 1: fresh schema.sql regenerated at post-refactor shape. - Phase 2: migrations 23+24 — rebuild pending_messages without started_processing_at_epoch; UNIQUE(session_id, tool_use_id); UNIQUE(memory_session_id, content_hash) on observations; dedup duplicate rows before adding indexes. - Phase 3: claimNextMessage rewritten to self-healing query using worker_pid NOT IN live_worker_pids; STALE_PROCESSING_THRESHOLD_MS and the 60-s stale-reset block deleted. - Phase 4: DEDUP_WINDOW_MS and findDuplicateObservation deleted; observations.insert now uses ON CONFLICT DO NOTHING. - Phase 5: failed-message purge block deleted from worker-service 2-min interval; clearFailedOlderThan method deleted. - Phase 6: repairMalformedSchema and its Python subprocess repair path deleted from Database.ts; SQLite errors now propagate. - Phase 7: Chroma delete-then-add fallback gated behind CHROMA_SYNC_FALLBACK_ON_CONFLICT env flag as bridge until Chroma MCP ships native upsert. - Phase 8: migration 19 no-op block absorbed into fresh schema.sql. Verification greps all return 0 matches. bun test tests/sqlite/ passes 63/63. bun run build succeeds. Plan: PATHFINDER-2026-04-22/01-data-integrity.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: land PATHFINDER Plan 02 — process lifecycle OS process groups replace hand-rolled reapers. Worker runs until killed; orphans are prevented by detached spawn + kill(-pgid). - Phase 1: src/services/worker/ProcessRegistry.ts DELETED. The canonical registry at src/supervisor/process-registry.ts is the sole survivor; SDK spawn site consolidated into it via new createSdkSpawnFactory/spawnSdkProcess/getSdkProcessForSession/ ensureSdkProcessExit/waitForSlot helpers. - Phase 2: SDK children spawn with detached:true + stdio: ['ignore','pipe','pipe']; pgid recorded on ManagedProcessInfo. - Phase 3: shutdown.ts signalProcess teardown uses process.kill(-pgid, signal) on Unix when pgid is recorded; Windows path unchanged (tree-kill/taskkill). - Phase 4: all reaper intervals deleted — startOrphanReaper call, staleSessionReaperInterval setInterval (including the co-located WAL checkpoint — SQLite's built-in wal_autocheckpoint handles WAL growth without an app-level timer), killIdleDaemonChildren, killSystemOrphans, reapOrphanedProcesses, reapStaleSessions, and detectStaleGenerator. MAX_GENERATOR_IDLE_MS and MAX_SESSION_IDLE_MS constants deleted. - Phase 5: abandonedTimer — already 0 matches; primary-path cleanup via generatorPromise.finally() already lives in worker-service startSessionProcessor and SessionRoutes ensureGeneratorRunning. - Phase 6: evictIdlestSession and its evict callback deleted from SessionManager. Pool admission gates backpressure upstream. - Phase 7: SDK-failure fallback — SessionManager has zero matches for fallbackAgent/Gemini/OpenRouter. Failures surface to hooks via exit code 2 through SessionRoutes error mapping. - Phase 8: ensureWorkerRunning in worker-utils.ts rewritten to lazy-spawn — consults isWorkerPortAlive (which gates captureProcessStartToken for PID-reuse safety via commit 99060bac), then spawns detached with unref(), then waitForWorkerPort({ attempts: 3, backoffMs: 250 }) hand-rolled exponential backoff 250→500→1000ms. No respawn npm dep. - Phase 9: idle self-shutdown — zero matches for idleCheck/idleTimeout/IDLE_MAX_MS/idleShutdown. Worker exits only on external SIGTERM via supervisor signal handlers. Three test files that exercised deleted code removed: tests/worker/process-registry.test.ts, tests/worker/session-lifecycle-guard.test.ts, tests/services/worker/reap-stale-sessions.test.ts. Pass count: 1451 → 1407 (-44), all attributable to deleted test files. Zero new failures. 31 pre-existing failures remain (schema-repair suite, logger-usage-standards, environmental openclaw / plugin-distribution) — none introduced by Plan 02. All 10 verification greps return 0. bun run build succeeds. Plan: PATHFINDER-2026-04-22/02-process-lifecycle.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: land PATHFINDER Plan 04 (narrowed) — search fail-fast Phases 3, 5, 6 only. Plan-doc inaccuracies for phases 1/2/4/7/8/9 deferred for plan reconciliation: - Phase 1/2: ObservationRow type doesn't exist; the four "formatters" operate on three incompatible types. - Phase 4: RECENCY_WINDOW_MS already imported from SEARCH_CONSTANTS at every call site. - Phase 7: getExistingChromaIds is NOT @deprecated and has an active caller in ChromaSync.backfillMissingSyncs. - Phase 8: estimateTokens already consolidated. - Phase 9: knowledge-corpus rewrite blocked on PG-3 prompt-caching cost smoke test. Phase 3 — Delete SearchManager.findByConcept/findByFile/findByType. SearchRoutes handlers (handleSearchByConcept/File/Type) now call searchManager.getOrchestrator().findByXxx() directly via new getter accessors on SearchManager. ~250 LoC deleted. Phase 5 — Fail-fast Chroma. Created src/services/worker/search/errors.ts with ChromaUnavailableError extends AppError(503, 'CHROMA_UNAVAILABLE'). Deleted SearchOrchestrator.executeWithFallback's Chroma-failed SQLite-fallback branch; runtime Chroma errors now throw 503. "Path 3" (chromaSync was null at construction — explicit- uninitialized config) preserved as legitimate empty-result state per plan text. ChromaSearchStrategy.search no longer wraps in try/catch — errors propagate. Phase 6 — Delete HybridSearchStrategy three try/catch silent fallback blocks (findByConcept, findByType, findByFile) at lines ~82-95, ~120-132, ~161-172. Removed `fellBack` field from StrategySearchResult type and every return site (SQLiteSearchStrategy, BaseSearchStrategy.emptyResult, SearchOrchestrator). Tests updated (Principle 7 — delete in same PR): - search-orchestrator.test.ts: "fall back to SQLite" rewritten as "throw ChromaUnavailableError (HTTP 503)". - chroma/hybrid/sqlite-search-strategy tests: rewritten to rejects.toThrow; removed fellBack assertions. Verification: SearchManager.findBy → 0; fellBack → 0 in src/. bun test tests/worker/search/ → 122 pass, 0 fail. bun test (suite-wide) → 1407 pass, baseline maintained, 0 new failures. bun run build succeeds. Plan: PATHFINDER-2026-04-22/04-read-path.md (Phases 3, 5, 6) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: land PATHFINDER Plan 03 — ingestion path Fail-fast parser, direct in-process ingest, recursive fs.watch, DB-backed tool pairing. Worker-internal HTTP loopback eliminated. - Phase 0: Created src/services/worker/http/shared.ts exporting ingestObservation/ingestPrompt/ingestSummary as direct in-process functions plus ingestEventBus (Node EventEmitter, reusing existing pattern — no third event bus introduced). setIngestContext wires the SessionManager dependency from worker-service constructor. - Phase 1: src/sdk/parser.ts collapsed to one parseAgentXml returning { valid:true; kind: 'observation'|'summary'; data } | { valid:false; reason: string }. Inspects root element; <skip_summary reason="…"/> is a first-class summary case with skipped:true. NEVER returns undefined. NEVER coerces. - Phase 2: ResponseProcessor calls parseAgentXml exactly once, branches on the discriminated union. On invalid → markFailed + logger.warn(reason). On observation → ingestObservation. On summary → ingestSummary then emit summaryStoredEvent { sessionId, messageId } (consumed by Plan 05's blocking /api/session/end). - Phase 3: Deleted consecutiveSummaryFailures field (ResponseProcessor + SessionManager + worker-types) and MAX_CONSECUTIVE_SUMMARY_FAILURES constant. Circuit-breaker guards and "tripped" log lines removed. - Phase 4: coerceObservationToSummary deleted from sdk/parser.ts. - Phase 5: src/services/transcripts/watcher.ts rescan setInterval replaced with fs.watch(transcriptsRoot, { recursive: true, persistent: true }) — Node 20+ recursive mode. - Phase 6: src/services/transcripts/processor.ts pendingTools Map deleted. tool_use rows insert with INSERT OR IGNORE on UNIQUE(session_id, tool_use_id) (added by Plan 01). New pairToolUsesByJoin query in PendingMessageStore for read-time pairing (UNIQUE INDEX provides idempotency; explicit consumer not yet wired). - Phase 7: HTTP loopback at processor.ts:252 replaced with direct ingestObservation call. maybeParseJson silent-passthrough rewritten to fail-fast (throws on malformed JSON). - Phase 8: src/utils/tag-stripping.ts countTags + stripTagsInternal collapsed into one alternation regex, single-pass over input. - Phase 9: src/utils/transcript-parser.ts (dead TranscriptParser class) deleted. The active extractLastMessage at src/shared/transcript-parser.ts:41-144 is the sole survivor. Tests updated (Principle 7 — same-PR delete): - tests/sdk/parser.test.ts + parse-summary.test.ts: rewritten to assert discriminated-union shape; coercion-specific scenarios collapse into { valid:false } assertions. - tests/worker/agents/response-processor.test.ts: circuit-breaker describe block skipped; non-XML/empty-response tests assert fail-fast markFailed behavior. Verification: every grep returns 0. transcript-parser.ts deleted. bun run build succeeds. bun test → 1399 pass / 28 fail / 7 skip (net -8 pass = the 4 retired circuit-breaker tests + 4 collapsed parser cases). Zero new failures vs baseline. Deferred (out of Plan 03 scope, will land in Plan 06): SessionRoutes HTTP route handlers still call sessionManager.queueObservation inline rather than the new shared helpers — the helpers are ready, the route swap is mechanical and belongs with the Zod refactor. Plan: PATHFINDER-2026-04-22/03-ingestion-path.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: land PATHFINDER Plan 05 — hook surface Worker-call plumbing collapsed to one helper. Polling replaced by server-side blocking endpoint. Fail-loud counter surfaces persistent worker outages via exit code 2. - Phase 1: plugin/hooks/hooks.json — three 20-iteration `for i in 1..20; do curl -sf .../health && break; sleep 0.1; done` shell retry wrappers deleted. Hook commands invoke their bun entry point directly. - Phase 2: src/shared/worker-utils.ts — added executeWithWorkerFallback<T>(url, method, body) returning T | { continue: true; reason?: string }. All 8 hook handlers (observation, session-init, context, file-context, file-edit, summarize, session-complete, user-message) rewritten to use it instead of duplicating the ensureWorkerRunning → workerHttpRequest → fallback sequence. - Phase 3: blocking POST /api/session/end in SessionRoutes.ts using validateBody + sessionEndSchema (z.object({sessionId})). One-shot ingestEventBus.on('summaryStoredEvent') listener, 30 s timer, req.aborted handler — all share one cleanup so the listener cannot leak. summarize.ts polling loop, plus MAX_WAIT_FOR_SUMMARY_MS / POLL_INTERVAL_MS constants, deleted. - Phase 4: src/shared/hook-settings.ts — loadFromFileOnce() memoizes SettingsDefaultsManager.loadFromFile per process. Per-handler settings reads collapsed. - Phase 5: src/shared/should-track-project.ts — single exclusion check entry; isProjectExcluded no longer referenced from src/cli/handlers/. - Phase 6: cwd validation pushed into adapter normalizeInput (all 6 adapters: claude-code, cursor, raw, gemini-cli, windsurf). New AdapterRejectedInput error in src/cli/adapters/errors.ts. Handler-level isValidCwd checks deleted from file-edit.ts and observation.ts. hook-command.ts catches AdapterRejectedInput → graceful fallback. - Phase 7: session-init.ts conditional initAgent guard deleted; initAgent is idempotent. tests/hooks/context-reinjection-guard test (validated the deleted conditional) deleted in same PR per Principle 7. - Phase 8: fail-loud counter at ~/.claude-mem/state/hook-failures .json. Atomic write via .tmp + rename. CLAUDE_MEM_HOOK_FAIL_LOUD _THRESHOLD setting (default 3). On consecutive worker-unreachable ≥ N: process.exit(2). On success: reset to 0. NOT a retry. - Phase 9: ensureWorkerAliveOnce() module-scope memoization wrapping ensureWorkerRunning. executeWithWorkerFallback calls the memoized version. Minimal validateBody middleware stub at src/services/worker/http/middleware/validateBody.ts. Plan 06 will expand with typed inference + error envelope conventions. Verification: 4/4 grep targets pass. bun run build succeeds. bun test → 1393 pass / 28 fail / 7 skip; -6 pass attributable solely to deleted context-reinjection-guard test file. Zero new failures vs baseline. Plan: PATHFINDER-2026-04-22/05-hook-surface.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: land PATHFINDER Plan 06 — API surface One Zod-based validator wrapping every POST/PUT. Rate limiter, diagnostic endpoints, and shutdown wrappers deleted. Failure- marking consolidated to one helper. - Phase 1 (preflight): zod@^3 already installed. - Phase 2: validateBody middleware confirmed at canonical shape in src/services/worker/http/middleware/validateBody.ts — safeParse → 400 { error: 'ValidationError', issues: [...] } on failure, replaces req.body with parsed value on success. - Phase 3: Per-route Zod schemas declared at the top of each route file. 24 POST endpoints across SessionRoutes, CorpusRoutes, DataRoutes, MemoryRoutes, SearchRoutes, LogsRoutes, SettingsRoutes now wrap with validateBody(). /api/session/end (Plan 05) confirmed using same middleware. - Phase 4: validateRequired() deleted from BaseRouteHandler along with every call site. Inline coercion helpers (coerceStringArray, coercePositiveInteger) and inline if (!req.body...) guards deleted across all route files. - Phase 5: Rate limiter middleware and its registration deleted from src/services/worker/http/middleware.ts. Worker binds 127.0.0.1:37777 — no untrusted caller. - Phase 6: viewer.html cached at module init in ViewerRoutes.ts via fs.readFileSync; served as Buffer with text/html content type. SKILL.md + per-operation .md files cached in Server.ts as Map<string, string>; loadInstructionContent helper deleted. NO fs.watch, NO TTL — process restart is the cache-invalidation event. - Phase 7: Four diagnostic endpoints deleted from DataRoutes.ts — /api/pending-queue (GET), /api/pending-queue/process (POST), /api/pending-queue/failed (DELETE), /api/pending-queue/all (DELETE). Helper methods that ONLY served them (getQueueMessages, getStuckCount, getRecentlyProcessed, clearFailed, clearAll) deleted from PendingMessageStore. KEPT: /api/processing-status (observability), /health (used by ensureWorkerRunning). - Phase 8: stopSupervisor wrapper deleted from supervisor/index.ts. GracefulShutdown now calls getSupervisor().stop() directly. Two functions retained with clear roles: - performGracefulShutdown — worker-side 6-step shutdown - runShutdownCascade — supervisor-side child teardown (process.kill(-pgid), Windows tree-kill, PID-file cleanup) Each has unique non-trivial logic and a single canonical caller. - Phase 9: transitionMessagesTo(status, filter) is the sole failure-marking path on PendingMessageStore. Old methods markSessionMessagesFailed and markAllSessionMessagesAbandoned deleted along with all callers (worker-service, SessionCompletionHandler, tests/zombie-prevention). Tests updated (Principle 7 same-PR delete): coercion test files refactored to chain validateBody → handler. Zombie-prevention tests rewritten to call transitionMessagesTo. Verification: all 4 grep targets → 0. bun run build succeeds. bun test → 1393 pass / 28 fail / 7 skip — exact match to baseline. Zero new failures. Plan: PATHFINDER-2026-04-22/06-api-surface.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: land PATHFINDER Plan 07 — dead code sweep ts-prune-driven sweep across the tree after Plans 01-06 landed. Deleted unused exports, orphan helpers, and one fully orphaned file. Earlier-plan deletions verified. Deleted: - src/utils/bun-path.ts (entire file — getBunPath, getBunPathOrThrow, isBunAvailable: zero importers) - bun-resolver.getBunVersionString: zero callers - PendingMessageStore.retryMessage / resetProcessingToPending / abortMessage: superseded by transitionMessagesTo (Plan 06 Phase 9) - EnvManager.MANAGED_CREDENTIAL_KEYS, EnvManager.setCredential: zero callers - CodexCliInstaller.checkCodexCliStatus: zero callers; no status command exists in npx-cli - Two "REMOVED: cleanupOrphanedSessions" stale-fence comments Kept (with documented justification): - Public API surface in dist/sdk/* (parseAgentXml, prompt builders, ParsedObservation, ParsedSummary, ParseResult, SUMMARY_MODE_MARKER) — exported via package.json sdk path. - generateContext / loadContextConfig / token utilities — used via dynamic await import('../../../context-generator.js') in worker SearchRoutes. - MCP_IDE_INSTALLERS, install/uninstall functions for codex/goose — used via dynamic await import in npx-cli/install.ts + uninstall.ts (ts-prune cannot trace dynamic imports). - getExistingChromaIds — active caller in ChromaSync.backfillMissingSyncs (Plan 04 narrowed scope). - processPendingQueues / getSessionsWithPendingMessages — active orphan-recovery caller in worker-service.ts plus zombie-prevention test coverage. - StoreAndMarkCompleteResult legacy alias — return-type annotation in same file. - All Database.ts barrel re-exports — used downstream. Earlier-plan verification: - Plan 03 Phase 9: VERIFIED — src/utils/transcript-parser.ts is gone; TranscriptParser has 0 references in src/. - Plan 01 Phase 8: VERIFIED — migration 19 no-op absorbed. - SessionStore.ts:52-70 consolidation NOT executed (deferred): the methods are not thin wrappers but ~900 LoC of bodies, and two methods are documented as intentional mirrors so the context-generator.cjs bundle stays schema-consistent without pulling MigrationRunner. Deserves its own plan, not a sweep. Verification: TranscriptParser → 0; transcript-parser.ts → gone; no commented-out code markers remain. bun run build succeeds. bun test → 1393 pass / 28 fail / 7 skip — EXACT match to baseline. Zero regressions. Plan: PATHFINDER-2026-04-22/07-dead-code.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: remove residual ProcessRegistry comment reference Plan 07 dead-code sweep missed one comment-level reference to the deleted in-memory ProcessRegistry class in SessionManager.ts:347. Rewritten to describe the supervisor.json scope without naming the deleted class, completing the verification grep target. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: address Greptile review (P1 + 2× P2) P1 — Plan 05 Phase 3 blocking endpoint was non-functional: executeWithWorkerFallback used HEALTH_CHECK_TIMEOUT_MS (3 s) for the POST /api/session/end call, but the server holds the connection for SERVER_SIDE_SUMMARY_TIMEOUT_MS (30 s). Client always raced to a "timed out" rejection that isWorkerUnavailable classified as worker-unreachable, so the hook silently degraded instead of waiting for summaryStoredEvent. - Added optional timeoutMs to executeWithWorkerFallback, forwarded to workerHttpRequest. - summarize.ts call site now passes 35_000 (5 s above server hold window). P2 — ingestSummary({ kind: 'parsed' }) branch was dead code: ResponseProcessor emitted summaryStoredEvent directly via the event bus, bypassing the centralized helper that the comment claimed was the single source. - ResponseProcessor now calls ingestSummary({ kind: 'parsed', sessionDbId, messageId, contentSessionId, parsed }) so the event-emission path is single-sourced. - ingestSummary's requireContext() resolution moved inside the 'queue' branch (the only branch that needs sessionManager / dbManager). 'parsed' is a pure event-bus emission and doesn't need worker-internal context — fixes mocked ResponseProcessor unit tests that don't call setIngestContext. P2 — isWorkerFallback could false-positive on legitimate API responses whose schema includes { continue: true, ... }: - Added a Symbol.for('claude-mem/worker-fallback') brand to WorkerFallback. isWorkerFallback now checks the brand, not a duck-typed property name. Verification: bun run build succeeds. bun test → 1393 pass / 28 fail / 7 skip — exact baseline match. Zero new failures. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: address Greptile iteration 2 (P1 + P2) P1 — summaryStoredEvent fired regardless of whether the row was persisted. ResponseProcessor's call to ingestSummary({ kind: 'parsed' }) ran for every parsed.kind === 'summary' even when result.summaryId came back null (e.g. FK violation, null memory_session_id at commit). The blocking /api/session/end endpoint then returned { ok: true } and the Stop hook logged 'Summary stored' for a non-existent row. - Gate ingestSummary call on (parsed.data.skipped || session.lastSummaryStored). Skipped summaries are an explicit no-op bypass and still confirm; real summaries only confirm when storage actually wrote a row. - Non-skipped + summaryId === null path logs a warn and lets the server-side timeout (504) surface to the hook instead of a false ok:true. P2 — PendingMessageStore.enqueue() returns 0 when INSERT OR IGNORE suppresses a duplicate (the UNIQUE(session_id, tool_use_id) constraint added by Plan 01 Phase 1). The two callers (SessionManager.queueObservation and queueSummarize) previously logged 'ENQUEUED messageId=0' which read like a row was inserted. - Branch on messageId === 0 and emit a 'DUP_SUPPRESSED' debug log instead of the misleading ENQUEUED line. No behavior change — the duplicate is still correctly suppressed by the DB (Principle 3); only the log surface is corrected. - confirmProcessed is never called with the enqueue() return value (it operates on session.processingMessageIds[] from claimNextMessage), so no caller is broken; the visibility fix prevents future misuse. Verification: bun run build succeeds. bun test → 1393 pass / 28 fail / 7 skip — exact baseline match. Zero new failures. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: address Greptile iteration 3 (P1 + 2× P2) - P1 worker-service.ts: wire ensureGeneratorRunning into the ingest context after SessionRoutes is constructed. setIngestContext runs before routes exist, so transcript-watcher observations queued via ingestObservation() had no way to auto-start the SDK generator. Added attachIngestGeneratorStarter() to patch the callback in. - P2 shared.ts: IngestEventBus now sets maxListeners to 0. Concurrent /api/session/end calls register one listener each and clean up on completion, so the default-10 warning fires spuriously under normal load. - P2 SessionRoutes.ts: handleObservationsByClaudeId now delegates to ingestObservation() instead of duplicating skip-tool / meta / privacy / queue logic. Single helper, matching the Plan 03 goal. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: address Greptile iteration 4 (P1 tool-pair + P2 parse/path/doc) - processor.handleToolResult: restore in-memory tool-use→tool-result pairing via session.pendingTools for schemas (e.g. Codex) whose tool_result events carry only tool_use_id + output. Without this, neither handler fired — all tool observations silently dropped. - processor.maybeParseJson: return raw string on parse failure instead of throwing. Previously a single malformed JSON-shaped field caused handleLine's outer catch to discard the entire transcript line. - watcher.deepestNonGlobAncestor: split on / and \\, emit empty string for purely-glob inputs so the caller skips the watch instead of anchoring fs.watch at the filesystem root. Windows-compatible. - PendingMessageStore.enqueue: tighten docstring — callers today only log on the returned id; the SessionManager branches on id === 0. * fix: forward tool_use_id through ingestObservation (Greptile iter 5) P1 — Plan 01's UNIQUE(content_session_id, tool_use_id) dedup never fired because the new shared ingest path dropped the toolUseId before queueObservation. SQLite treats NULL values as distinct for UNIQUE, so every replayed transcript line landed a duplicate row. - shared.ingestObservation: forward payload.toolUseId to queueObservation so INSERT OR IGNORE can actually collapse. - SessionRoutes.handleObservationsByClaudeId: destructure both tool_use_id (HTTP convention) and toolUseId (JS convention) from req.body and pass into ingestObservation. - observationsByClaudeIdSchema: declare both keys explicitly so the validator doesn't rely on .passthrough() alone. * fix: drop dead pairToolUsesByJoin, close session-end listener race - PendingMessageStore: delete pairToolUsesByJoin. The method was never called and its self-join semantics are structurally incompatible with UNIQUE(content_session_id, tool_use_id): INSERT OR IGNORE collapses any second row with the same pair, so a self-join can only ever match a row to itself. In-memory pendingTools in processor.ts remains the pairing path for split-event schemas. - IngestEventBus: retain a short-lived (60s) recentStored map keyed by sessionId. Populated on summaryStoredEvent emit, evicted on consume or TTL. - handleSessionEnd: drain the recent-events buffer before attaching the listener. Closes the register-after-emit race where the summary can persist between the hook's summarize POST and its session/end POST — previously that window returned 504 after the 30s timeout. * chore: merge origin/main into vivacious-teeth Resolves conflicts with 15 commits on main (v12.3.9, security observation types, Telegram notifier, PID-reuse worker start-guard). Conflict resolution strategy: - plugin/hooks/hooks.json, plugin/scripts/*.cjs, plugin/ui/viewer-bundle.js: kept ours — PATHFINDER Plan 05 deletes the for-i-in-1-to-20 curl retry loops and the built artifacts regenerate on build. - src/cli/handlers/summarize.ts: kept ours — Plan 05 blocking POST /api/session/end supersedes main's fire-and-forget path. - src/services/worker-service.ts: kept ours — Plan 05 ingest bus + summaryStoredEvent supersedes main's SessionCompletionHandler DI refactor + orphan-reaper fallback. - src/services/worker/http/routes/SessionRoutes.ts: kept ours — same reason; generator .finally() Stop-hook self-clean is a guard for a path our blocking endpoint removes. - src/services/worker/http/routes/CorpusRoutes.ts: merged — added security_alert / security_note to ALLOWED_CORPUS_TYPES (feature from #2084) while preserving our Zod validateBody schema. Typecheck: 294 errors (vs 298 pre-merge). No new errors introduced; all remaining are pre-existing (Component-enum gaps, DOM lib for viewer, bun:sqlite types). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: address Greptile P2 findings 1) SessionRoutes.handleSessionEnd was the only route handler not wrapped in wrapHandler — synchronous exceptions would hang the client rather than surfacing as 500s. Wrap it like every other handler. 2) processor.handleToolResult only consumed the session.pendingTools entry when the tool_result arrived without a toolName. In the split-schema path where tool_result carries both toolName and toolId, the entry was never deleted and the map grew for the life of the session. Consume the entry whenever toolId is present. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: typing cleanup and viewer tsconfig split for PR feedback - Add explicit return types for SessionStore query methods - Exclude src/ui/viewer from root tsconfig, give it its own DOM-typed config - Add bun to root tsconfig types, plus misc typing tweaks flagged by Greptile - Rebuilt plugin/scripts/* artifacts Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: address Greptile P2 findings (iter 2) - PendingMessageStore.transitionMessagesTo: require sessionDbId (drop the unscoped-drain branch that would nuke every pending/processing row across all sessions if a future caller omitted the filter). - IngestEventBus.takeRecentSummaryStored: make idempotent — keep the cached event until TTL eviction so a retried Stop hook's second /api/session/end returns immediately instead of hanging 30 s. - TranscriptWatcher fs.watch callback: skip full glob scan for paths already tailed (JSONL appends fire on every line; only unknown paths warrant a rescan). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: call finalizeSession in terminal session paths (Greptile iter 3) terminateSession and runFallbackForTerminatedSession previously called SessionCompletionHandler.finalizeSession before removeSessionImmediate; the refactor dropped those calls, leaving sdk_sessions.status='active' for every session killed by wall-clock limit, unrecoverable error, or exhausted fallback chain. The deleted reapStaleSessions interval was the only prior backstop. Re-wires finalizeSession (idempotent: marks completed, drains pending, broadcasts) into both paths; no reaper reintroduced. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: GC failed pending_messages rows at startup (Greptile iter 4) Plan 07 deleted clearFailed/clearFailedOlderThan as "dead code", but with the periodic sweep also removed, nothing reaps status='failed' rows now — they accumulate indefinitely. Since claimNextMessage's self-healing subquery scans this table, unbounded growth degrades claim latency over time. Re-introduces clearFailedOlderThan and calls it once at worker startup (not a reaper — one-shot, idempotent). 7-day retention keeps enough history for operator inspection while bounding the table. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: finalize sessions on normal exit; cleanup hoist; share handler (iter 5) 1. startSessionProcessor success branch now calls completionHandler. finalizeSession before removeSessionImmediate. Hooks-disabled installs (and any Stop hook that fails before POST /api/sessions/complete) no longer leave sdk_sessions rows as status='active' forever. Idempotent — a subsequent /api/sessions/complete is a no-op. 2. Hoist SessionRoutes.handleSessionEnd cleanup declaration above the closures that reference it (TDZ safety; safe at runtime today but fragile if timeout ever shrinks). 3. SessionRoutes now receives WorkerService's shared SessionCompletionHandler instead of constructing its own — prevents silent divergence if the handler ever becomes stateful. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: stop runaway crash-recovery loop on dead sessions Two distinct bugs were combining to keep a dead session restarting forever: Bug 1 (uncaught "The operation was aborted."): child_process.spawn emits 'error' asynchronously for ENOENT/EACCES/abort signal aborts. spawnSdkProcess() never attached an 'error' listener, so any async spawn failure became uncaughtException and escaped to the daemon-level handler. Attach an 'error' listener immediately after spawn, before the !child.pid early-return, so async spawn errors are logged (with errno code) and swallowed locally. Bug 2 (sliding-window limiter never trips on slow restart cadence): RestartGuard tripped only when restartTimestamps.length exceeded MAX_WINDOWED_RESTARTS (10) within RESTART_WINDOW_MS (60s). With the 8s exponential-backoff cap, only ~7-8 restarts fit in the window, so a dead session that fail-restart-fail-restart on 8s cycles would loop forever (consecutiveRestarts climbing past 30+ in observed logs). Add a consecutiveFailures counter that increments on every restart and resets only on recordSuccess(). Trip when consecutive failures exceed MAX_CONSECUTIVE_FAILURES (5) — meaning 5 restarts with zero successful processing in between proves the session is dead. Both guards now run in parallel: tight loops still trip the windowed cap; slow loops trip the consecutive-failure cap. Also: when the SessionRoutes path trips the guard, drain pending messages to 'abandoned' so the session does not reappear in getSessionsWithPendingMessages and trigger another auto-start cycle. The worker-service.ts path already does this via terminateSession. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf: streamline worker startup and consolidate database connections 1. Database Pooling: Modified DatabaseManager, SessionStore, and SessionSearch to share a single bun:sqlite connection, eliminating redundant file descriptors. 2. Non-blocking Startup: Refactored WorktreeAdoption and Chroma backfill to run in the background (fire-and-forget), preventing them from stalling core initialization. 3. Diagnostic Routes: Added /api/chroma/status and bypassed the initialization guard for health/readiness endpoints to allow diagnostics during startup. 4. Robust Search: Implemented reliable SQLite FTS5 fallback in SearchManager for when Chroma (uvx) fails or is unavailable. 5. Code Cleanup: Removed redundant loopback MCP checks and mangled initialization logic from WorkerService. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: hard-exclude observer-sessions from hooks; bundle migration 29 (#2124) * fix: hard-exclude observer-sessions from hooks; backfill bundle migrations Stop hook + SessionEnd hook were storing the SDK observer's own init/continuation/summary prompts in user_prompts, leaking into the viewer (meta-observation regression). 25 such rows accumulated. - shouldTrackProject: hard-reject OBSERVER_SESSIONS_DIR (and its subtree) before consulting user-configured exclusion globs. - summarize.ts (Stop) and session-complete.ts (SessionEnd): early-return when shouldTrackProject(cwd) is false, so the observer's own hooks cannot bootstrap the worker or queue a summary against the meta-session. - SessionRoutes: cap user-prompt body at 256 KiB at the session-init boundary so a runaway observer prompt cannot blow up storage. - SessionStore: add migration 29 (UNIQUE(memory_session_id, content_hash) on observations) inline so bundled artifacts (worker-service.cjs, context-generator.cjs) stay schema-consistent — without it, the ON CONFLICT clause in observation inserts throws. - spawnSdkProcess: stdio[stdin] from 'ignore' to 'pipe' so the supervisor can actually feed the observer's stdin. Also rebuilds plugin/scripts/{worker-service,context-generator}.cjs. * fix: walk back to UTF-8 boundary on prompt truncation (Greptile P2) Plain Buffer.subarray at MAX_USER_PROMPT_BYTES can land mid-codepoint, which the utf8 decoder silently rewrites to U+FFFD. Walk back over any continuation bytes (0b10xxxxxx) before decoding so the truncated prompt ends on a valid sequence boundary instead of a replacement character. * fix: cross-platform observer-dir containment; clarify SDK stdin pipe claude-review feedback on PR #2124. - shouldTrackProject: literal `cwd.startsWith(OBSERVER_SESSIONS_DIR + '/')` hard-coded a POSIX separator and missed Windows backslash paths plus any trailing-slash variance. Switched to a path.relative-based isWithin() helper so Windows hook input under observer-sessions\\... is also excluded. - spawnSdkProcess: added a comment explaining why stdin must be 'pipe' — SpawnedSdkProcess.stdin is typed NonNullable and the Claude Agent SDK consumes that pipe; 'ignore' would null it and the null-check below would tear the child down on every spawn. * fix: make Stop hook fire-and-forget; remove dead /api/session/end The Stop hook was awaiting a 35-second long-poll on /api/session/end, which the worker held open until the summary-stored event fired (or its 30s server-side timeout elapsed). Followed by another await on /api/sessions/complete. Three sequential awaits, the middle one a 30s hold — not fire-and-forget despite repeated requests. The Stop hook now does ONE thing: POST /api/sessions/summarize to queue the summary work and return. The worker drives the rest async. Session-map cleanup is performed by the SessionEnd handler (session-complete.ts), not duplicated here. - summarize.ts: drop the /api/session/end long-poll and the trailing /api/sessions/complete await; ~40 lines removed; unused SessionEndResponse interface gone; header comment rewritten. - SessionRoutes: delete handleSessionEnd, sessionEndSchema, the SERVER_SIDE_SUMMARY_TIMEOUT_MS constant, and the /api/session/end route registration. Drop the now-unused ingestEventBus and SummaryStoredEvent imports. - ResponseProcessor + shared.ts + worker-utils.ts: update stale comments that referenced the dead endpoint. The IngestEventBus is left in place dormant (no listeners) for follow-up cleanup so this PR stays focused on the blocker. Bundle artifact (worker-service.cjs) rebuilt via build-and-sync. Verification: - grep '/api/session/end' plugin/scripts/worker-service.cjs → 0 - grep 'timeoutMs:35' plugin/scripts/worker-service.cjs → 0 - Worker restarted clean, /api/health ok at pid 92368 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * deps: bump all dependencies to latest including majors Upgrades: React 18→19, Express 4→5, Zod 3→4, TypeScript 5→6, @types/node 20→25, @anthropic-ai/claude-agent-sdk 0.1→0.2, @clack/prompts 0.9→1.2, plus minors. Adds Daily Maintenance section to CLAUDE.md mandating latest-version policy across manifests. Express 5 surfaced a race in Server.listen() where the 'error' handler was attached after listen() was invoked; refactored to use http.createServer with both 'error' and 'listening' handlers attached before listen(), restoring port-conflict rejection semantics. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: surface real chroma errors and add deep status probe Replace the misleading "Vector search failed - semantic search unavailable. Install uv... restart the worker." string in SearchManager with the actual exception text from chroma_query_documents. The lying message blamed `uv` for any failure — even when the real cause was a chroma-mcp transport timeout, an empty collection, or a dead subprocess. Also add /api/chroma/status?deep=1 backed by a new ChromaMcpManager.probeSemanticSearch() that round-trips a real query (chroma_list_collections + chroma_query_documents) instead of just checking the stdio handshake. The cheap default path is unchanged. Includes the diagnostic plan (PLAN-fix-mcp-search.md) and updated test fixtures for the new structured failure message. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: rebuild worker-service bundle to match merged src Bundle was stale after the squash merge of #2124 — it still contained the old "Install uv... semantic search unavailable" string and lacked probeSemanticSearch. Rebuilt via bun run build-and-sync. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: address coderabbit feedback on PLAN-fix-mcp-search.md - replace machine-specific /Users/alexnewman absolute paths with portable <repo-root> placeholder (MD-style portability) - add blank lines around the TypeScript fenced block (MD031) - tag the bare fenced block with `text` (MD040) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-25 13:37:40 -07:00
Alex Newman	6250a194dd	Merge branch 'pr-1472' into integration/validation-batch ... # Conflicts: # plugin/scripts/context-generator.cjs # plugin/scripts/mcp-server.cjs # plugin/scripts/worker-service.cjs # plugin/ui/viewer-bundle.js # src/cli/handlers/context.ts # src/services/sqlite/SessionStore.ts # src/services/sqlite/migrations/runner.ts # src/services/worker-service.ts # src/shared/SettingsDefaultsManager.ts	2026-04-06 14:23:18 -07:00
79475432@qq.com	9cfa57d498	fix: use null-byte delimiter in observation content hash to prevent collisions ... Fields concatenated without separators allowed different tuples to produce identical hashes (e.g. session="ab", title="cd" vs session="abc", title="d"). This could cause legitimate observations to be silently deduplicated. Join with \x00 so field boundaries are unambiguous. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-03-26 17:16:17 +08:00
huakson	4f6fb9e614	fix: address platform source review feedback ... Tighten platform source persistence so legacy callers cannot silently relabel existing sessions, repair migration 24 when schema_versions drifts from the real schema, and polish the follow-up UI/error-handler review nits. - only backfill platform_source when it is blank and raise on explicit source conflicts for an existing session - make migration 24 verify both the sdk_sessions column and its index before treating it as applied - expose platform_source from the functional session getters and add regression tests for source preservation and schema drift recovery - add the required APPROVED OVERRIDE annotation for centralized HTTP error translation - keep mobile source pills on a single horizontal row	2026-03-24 10:46:48 -03:00
Alex Newman	c6f932988a	Fix 30+ root-cause bugs across 10 triage phases (#1214) ... * MAESTRO: fix ChromaDB core issues — Python pinning, Windows paths, disable toggle, metadata sanitization, transport errors - Add --python version pinning to uvx args in both local and remote mode (fixes #1196, #1206, #1208) - Convert backslash paths to forward slashes for --data-dir on Windows (fixes #1199) - Add CLAUDE_MEM_CHROMA_ENABLED setting for SQLite-only fallback mode (fixes #707) - Sanitize metadata in addDocuments() to filter null/undefined/empty values (fixes #1183, #1188) - Wrap callTool() in try/catch for transport errors with auto-reconnect (fixes #1162) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * MAESTRO: fix data integrity — content-hash deduplication, project name collision, empty project guard, stuck isProcessing - Add SHA-256 content-hash deduplication to observations INSERT (store.ts, transactions.ts, SessionStore.ts) - Add content_hash column via migration 22 with backfill and index - Fix project name collision: getCurrentProjectName() now returns parent/basename - Guard against empty project string with cwd-derived fallback - Fix stuck isProcessing: hasAnyPendingWork() resets processing messages older than 5 minutes - Add 12 new tests covering all four fixes Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * MAESTRO: fix hook lifecycle — stderr suppression, output isolation, conversation pollution prevention - Suppress process.stderr.write in hookCommand() to prevent Claude Code showing diagnostic output as error UI (#1181). Restores stderr in finally block for worker-continues case. - Convert console.error() to logger.warn()/error() in hook-command.ts and handlers/index.ts so all diagnostics route to log file instead of stderr. - Verified all 7 handlers return suppressOutput: true (prevents conversation pollution #598, #784). - Verified session-complete is a recognized event type (fixes #984). - Verified unknown event types return no-op handler with exit 0 (graceful degradation). - Added 10 new tests in tests/hook-lifecycle.test.ts covering event dispatch, adapter defaults, stderr suppression, and standard response constants. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * MAESTRO: fix worker lifecycle — restart loop coordination, stale transport retry, ENOENT shutdown race - Add PID file mtime guard to prevent concurrent restart storms (#1145): isPidFileRecent() + touchPidFile() coordinate across sessions - Add transparent retry in ChromaMcpManager.callTool() on transport error — reconnects and retries once instead of failing (#1131) - Wrap getInstalledPluginVersion() with ENOENT/EBUSY handling (#1042) - Verified ChromaMcpManager.stop() already called on all shutdown paths Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * MAESTRO: fix Windows platform support — uvx.cmd spawn, PowerShell $_ elimination, windowsHide, FTS5 fallback - Route uvx spawn through cmd.exe /c on Windows since MCP SDK lacks shell:true (#1190, #1192, #1199) - Replace all PowerShell Where-Object {$_} pipelines with WQL -Filter server-side filtering (#1024, #1062) - Add windowsHide: true to all exec/spawn calls missing it to prevent console popups (#1048) - Add FTS5 runtime probe with graceful fallback when unavailable on Windows (#791) - Guard FTS5 table creation in migrations, SessionSearch, and SessionStore with try/catch Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * MAESTRO: fix skills/ distribution — build-time verification and regression tests (#1187) Add post-build verification in build-hooks.js that fails if critical distribution files (skills, hooks, plugin manifest) are missing. Add 10 regression tests covering skill file presence, YAML frontmatter, hooks.json integrity, and package.json files field. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * MAESTRO: fix MigrationRunner schema initialization (#979) — version conflict between parallel migration systems Root cause: old DatabaseManager migrations 1-7 shared schema_versions table with MigrationRunner's 4-22, causing version number collisions (5=drop tables vs add column, 6=FTS5 vs prompt tracking, 7=discovery_tokens vs remove UNIQUE). initializeSchema() was gated behind maxApplied===0, so core tables were never created when old versions were present. Fixes: - initializeSchema() always creates core tables via CREATE TABLE IF NOT EXISTS - Migrations 5-7 check actual DB state (columns/constraints) not just version tracking - Crash-safe temp table rebuilds (DROP IF EXISTS _new before CREATE) - Added missing migration 21 (ON UPDATE CASCADE) to MigrationRunner - Added ON UPDATE CASCADE to FK definitions in initializeSchema() - All changes applied to both runner.ts and SessionStore.ts Tests: 13 new tests in migration-runner.test.ts covering fresh DB, idempotency, version conflicts, crash recovery, FK constraints, and data integrity. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * MAESTRO: fix 21 test failures — stale mocks, outdated assertions, missing OpenClaw guards Server tests (12): Added missing workerPath and getAiStatus to ServerOptions mocks after interface expansion. ChromaSync tests (3): Updated to verify transport cleanup in ChromaMcpManager after architecture refactor. OpenClaw (2): Added memory_ tool skipping and response truncation to prevent recursive loops and oversized payloads. MarkdownFormatter (2): Updated assertions to match current output. SettingsDefaultsManager (1): Used correct default key for getBool test. Logger standards (1): Excluded CLI transcript command from background service check. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * MAESTRO: fix Codex CLI compatibility (#744) — session_id fallbacks, unknown platform tolerance, undefined guard Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * MAESTRO: fix Cursor IDE integration (#838, #1049) — adapter field fallbacks, tolerant session-init validation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * MAESTRO: fix /api/logs OOM (#1203) — tail-read replaces full-file readFileSync Replace readFileSync (loads entire file into memory) with readLastLines() that reads only from the end of the file in expanding chunks (64KB → 10MB cap). Prevents OOM on large log files while preserving the same API response shape. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * MAESTRO: fix Settings CORS error (#1029) — explicit methods and allowedHeaders in CORS config Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * MAESTRO: add session custom_title for agent attribution (#1213) — migration 23, endpoint + store support Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * MAESTRO: prevent CLAUDE.md/AGENTS.md writes inside .git/ directories (#1165) Add .git path guard to all 4 write sites to prevent ref corruption when paths resolve inside .git internals. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * MAESTRO: fix plugin disabled state not respected (#781) — early exit check in all hook entry points Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * MAESTRO: fix UserPromptSubmit context re-injection on every turn (#1079) — contextInjected session flag Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * MAESTRO: fix stale AbortController queue stall (#1099) — lastGeneratorActivity tracking + 30s timeout Three-layer fix: 1. Added lastGeneratorActivity timestamp to ActiveSession, updated by processAgentResponse (all agents), getMessageIterator (queue yields), and startGeneratorWithProvider (generator launch) 2. Added stale generator detection in ensureGeneratorRunning — if no activity for >30s, aborts stale controller, resets state, restarts 3. Added AbortSignal.timeout(30000) in deleteSession to prevent indefinite hang when awaiting a stuck generator promise Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>	2026-02-23 19:34:35 -05:00
Alex Newman	4df9f61347	refactor: implement in-process worker architecture for hooks (#722) ... * fix: stop generating empty CLAUDE.md files - Return empty string instead of "No recent activity" when no observations exist - Skip writing CLAUDE.md files when formatted content is empty - Remove redundant "auto-generated by claude-mem" HTML comment - Clean up 98 existing empty CLAUDE.md files across the codebase - Update tests to expect empty string for empty input Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * build assets * refactor: implement in-process worker architecture for hooks Replaces spawn-based worker startup with in-process architecture: - Hook processes now become the worker when port 37777 is free - Eliminates Windows spawn issues (NO SPAWN rule) - SessionStart chains: smart-install && stop && context Key changes: - worker-service.ts: hook case starts WorkerService in-process - hook-command.ts: skipExit option prevents process.exit() when hosting worker - hooks.json: single chained command replaces separate start/hook commands - worker-utils.ts: ensureWorkerRunning() returns boolean, doesn't block - handlers: graceful fallback when worker unavailable All 761 tests pass. Manual verification confirms hook stays alive as worker. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * context * a * MAESTRO: Mark PR #722 test verification task complete All 797 tests passed (3 skipped, 0 failed) after merge conflict resolution. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * MAESTRO: Mark PR #722 build verification task complete * MAESTRO: Mark PR #722 code review task complete Code review verified: - worker-service.ts hook case starts WorkerService in-process - hook-command.ts has skipExit option - hooks.json uses single chained command - worker-utils.ts ensureWorkerRunning() returns boolean Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * MAESTRO: Mark PR #722 conflict resolution push task complete Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>	2026-02-04 19:49:15 -05:00
Alex Newman	e1ab73decc	feat: Live Context System with Distributed CLAUDE.md Generation (#556) ... * docs: add folder index generator plan RFC for auto-generating folder-level CLAUDE.md files with observation timelines. Includes IDE symlink support and root CLAUDE.md integration. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: implement folder index generator (Phase 1) Add automatic CLAUDE.md generation for folders containing observed files. This enables IDE context providers to access relevant memory observations. Core modules: - FolderDiscovery: Extract folders from observation file paths - FolderTimelineCompiler: Compile chronological timeline per folder - ClaudeMdGenerator: Write CLAUDE.md with tag-based content replacement - FolderIndexOrchestrator: Coordinate regeneration on observation save Integration: - Event-driven regeneration after observation save in ResponseProcessor - HTTP endpoints for folder discovery, timeline, and manual generation - Settings for enabling/configuring folder index behavior The <claude-mem-context> tag wrapping ensures: - Manual CLAUDE.md content is preserved - Auto-generated content won't be recursively observed - Clean separation between user and system content 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: add updateFolderClaudeMd function to CursorHooksInstaller Adds function to update CLAUDE.md files for folders touched by observations. Uses existing /api/search/by-file endpoint, preserves content outside <claude-mem-context> tags, and writes atomically via temp file + rename. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: hook updateFolderClaudeMd into ResponseProcessor Calls updateFolderClaudeMd after observation save to update folder-level CLAUDE.md files. Uses fire-and-forget pattern with error logging. Extracts file paths from saved observations and workspace path from registry. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: add timeline formatting for folder CLAUDE.md files Implements formatTimelineForClaudeMd function that transforms API response into compact markdown table format. Converts emojis to text labels, handles ditto marks for timestamps, and groups under "Recent" header. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor: remove old folder-index implementation Deletes redundant folder-index services that were replaced by the simpler updateFolderClaudeMd approach in CursorHooksInstaller.ts. Removed: - src/services/folder-index/ directory (5 files) - FolderIndexRoutes.ts - folder-index settings from SettingsDefaultsManager - folder-index route registration from worker-service 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: add worktree-aware project filtering for unified timelines Detect git worktrees and show both parent repo and worktree observations in the session start timeline. When running in a worktree, the context now includes observations from both projects, interleaved chronologically. - Add detectWorktree() utility to identify worktree directories - Add getProjectContext() to return parent + worktree projects - Update context hook to pass multi-project queries - Add queryObservationsMulti() and querySummariesMulti() for IN clauses - Maintain backward compatibility with single-project queries 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * fix: restructure logging to prove session correctness and reduce noise Add critical logging at each stage of the session lifecycle to prove the session ID chain (contentSessionId → sessionDbId → memorySessionId) stays aligned. New logs include CREATED, ENQUEUED, CLAIMED, MEMORY_ID_CAPTURED, STORING, and STORED. Move intermediate migration and backfill progress logs to DEBUG level to reduce noise, keeping only essential initialization and completion logs at INFO level. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> * refactor: extract folder CLAUDE.md utils to shared location Moves folder CLAUDE.md utilities from CursorHooksInstaller to a new shared utils file. Removes Cursor registry dependency - file paths from observations are already absolute, no workspace lookup needed. New file: src/utils/claude-md-utils.ts - replaceTaggedContent() - preserves user content outside tags - writeClaudeMdToFolder() - atomic writes with tag preservation - formatTimelineForClaudeMd() - API response to compact markdown - updateFolderClaudeMdFiles() - orchestrates folder updates 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: trigger folder CLAUDE.md updates when observations are saved The folder CLAUDE.md update was previously only triggered in syncAndBroadcastSummary, but summaries run with observationCount=0 (observations are saved separately). Moved the update logic to syncAndBroadcastObservations where file paths are available. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * all the claudes * test: add unit tests for claude-md-utils pure functions Add 11 tests covering replaceTaggedContent and formatTimelineForClaudeMd: - replaceTaggedContent: empty content, tag replacement, appending, partial tags - formatTimelineForClaudeMd: empty input, parsing, ditto marks, session IDs 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: add integration tests for file operation functions Add 9 tests for writeClaudeMdToFolder and updateFolderClaudeMdFiles: - writeClaudeMdToFolder: folder creation, content preservation, nested dirs, atomic writes - updateFolderClaudeMdFiles: empty skip, fetch/write, deduplication, error handling 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test: add unit tests for timeline-formatting utilities Add 14 tests for extractFirstFile and groupByDate functions: - extractFirstFile: relative paths, fallback to files_read, null handling, invalid JSON - groupByDate: empty arrays, date grouping, chronological sorting, item preservation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * chore: rebuild plugin scripts with merged features * docs: add project-specific CLAUDE.md with architecture and development notes * fix: exclude project root from auto-generated CLAUDE.md updates Skip folders containing .git directory when auto-updating subfolder CLAUDE.md files. This ensures: 1. Root CLAUDE.md remains user-managed and untouched by the system 2. SessionStart context injection stays pristine throughout the session 3. Subfolder CLAUDE.md files continue to receive live context updates 4. Cleaner separation between user-authored root docs and auto-generated folder indexes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: prevent crash from resuming stale SDK sessions on worker restart When the worker restarts, it was incorrectly passing the `resume` parameter to INIT prompts (lastPromptNumber=1) when a memorySessionId existed from a previous SDK session. This caused "Claude Code process exited with code 1" crashes because the SDK tried to resume into a session that no longer exists. Root cause: The resume condition only checked `hasRealMemorySessionId` but did not verify that this was a CONTINUATION prompt (lastPromptNumber > 1). Fix: Add `session.lastPromptNumber > 1` check to the resume condition: - Before: `...(hasRealMemorySessionId && { resume: session.memorySessionId })` - After: `...(hasRealMemorySessionId && session.lastPromptNumber > 1 && { resume: ... })` Also added: - Enhanced debug logging that warns when skipping resume for INIT prompts - Unit tests in tests/sdk-agent-resume.test.ts (9 test cases) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: properly handle Chroma MCP connection errors Previously, ensureCollection() caught ALL errors from chroma_get_collection_info and assumed they meant "collection doesn't exist", triggering unnecessary collection creation attempts. Connection errors like "Not connected" or "MCP error -32000: Connection closed" would cascade into failed creation attempts. Similarly, queryChroma() would silently return empty results when the MCP call failed, masking the underlying connection problem. Changes: - ensureCollection(): Detect connection errors and re-throw immediately instead of attempting collection creation - queryChroma(): Wrap MCP call in try-catch and throw connection errors instead of returning empty results - Both methods reset connection state (connected=false, client=null) on connection errors so subsequent operations can attempt to reconnect 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * pushed * fix: scope regenerate-claude-md.ts to current working directory Critical bug fix: The script was querying ALL observations from the database across ALL projects ever recorded (1396+ folders), then attempting to write CLAUDE.md files everywhere including other projects, non-existent paths, and ignored directories. Changes: - Use git ls-files to discover folders (respects .gitignore automatically) - Filter database query to current project only (by folder name) - Use relative paths for database queries (matches storage format) - Add --clean flag to remove auto-generated CLAUDE.md files - Add fallback directory walker for non-git repos Now correctly scopes to 26 folders with observations instead of 1396+. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * docs and adjustments * fix: cleanup mode strips tags instead of deleting files blindly The cleanup mode was incorrectly deleting entire files that contained <claude-mem-context> tags. The correct behavior (per original design): 1. Strip the <claude-mem-context>...</claude-mem-context> section 2. If empty after stripping → delete the file 3. If has remaining content → save the stripped version Now properly preserves user content in CLAUDE.md files while removing only the auto-generated sections. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * deleted some files * chore: regenerate folder CLAUDE.md files with fixed script Regenerated 23 folder CLAUDE.md files using the corrected script that: - Scopes to current working directory only - Uses git ls-files to respect .gitignore - Filters by project name 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Update CLAUDE.md files for January 5, 2026 - Regenerated and staged 23 CLAUDE.md files with a mix of new and modified content. - Fixed cleanup mode to properly strip tags instead of deleting files blindly. - Cleaned up empty CLAUDE.md files from various directories, including ~/.claude and ~/Scripts. - Conducted dry-run cleanup that identified a significant reduction in auto-generated CLAUDE.md files. - Removed the isAutoGeneratedClaudeMd function due to incorrect file deletion behavior. * feat: use settings for observation limit in batch regeneration script Replace hard-coded limit of 10 with configurable CLAUDE_MEM_CONTEXT_OBSERVATIONS setting (default: 50). This allows users to control how many observations appear in folder CLAUDE.md files. Changes: - Import SettingsDefaultsManager and load settings at script startup - Use OBSERVATION_LIMIT constant derived from settings at both call sites - Remove stale default parameter from findObservationsByFolder function 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: use settings for observation limit in event-driven updates Replace hard-coded limit of 10 in updateFolderClaudeMdFiles with configurable CLAUDE_MEM_CONTEXT_OBSERVATIONS setting (default: 50). Changes: - Import SettingsDefaultsManager and os module - Load settings at function start (once, not in loop) - Use limit from settings in API call 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: Implement configurable observation limits and enhance search functionality - Added configurable observation limits to batch regeneration scripts. - Enhanced SearchManager to handle folder queries and normalize parameters. - Introduced methods to check for direct child files in observations and sessions. - Updated SearchOptions interface to include isFolder flag for filtering. - Improved code quality with comprehensive reviews and anti-pattern checks. - Cleaned up auto-generated CLAUDE.md files across various directories. - Documented recent changes and improvements in CLAUDE.md files. * build asset * Project Context from Claude-Mem auto-added (can be auto removed at any time) * CLAUDE.md updates * fix: resolve CLAUDE.md files to correct directory in worktree setups When using git worktrees, CLAUDE.md files were being written relative to the worker's process.cwd() instead of the actual project directory. This fix threads the project's cwd from message processing through to the file writing utilities, ensuring CLAUDE.md files are created in the correct project directory regardless of where the worker was started. Changes: - Add projectRoot parameter to updateFolderClaudeMdFiles for path resolution - Thread projectRoot through ResponseProcessor call chain - Track lastCwd from messages in SDKAgent, GeminiAgent, OpenRouterAgent - Add tests for relative/absolute path handling with projectRoot 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * more project context updates * context updates * fix: preserve actual dates in folder CLAUDE.md generation Previously, formatTimelineForClaudeMd used today's date for all observations because the API only returned time (e.g., "4:30 PM") without date information. This caused all historical observations to appear as if they happened today. Changes: - SearchManager.findByFile now groups results by date with headers (e.g., "### Jan 4, 2026") matching formatSearchResults behavior - formatTimelineForClaudeMd now parses these date headers and uses the correct date when constructing epochs for date grouping The timeline dates are critical for claude-mem context - LLMs need accurate temporal context to understand when work happened. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * build: update worker assets with date parsing fix 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * claude-mem context: Fixed critical date parsing bug in PR #556 * fix: address PR #556 review items - Use getWorkerHost() instead of hard-coded 127.0.0.1 in claude-md-utils - Add error message and stack details to FOLDER_INDEX logging - Add 5 new tests for worktree/projectRoot path resolution 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Refactor CLAUDE documentation across multiple components and tests - Updated CLAUDE.md files in src/ui/viewer, src/ui/viewer/constants, src/ui/viewer/hooks, tests/server, tests/worker/agents, and plans to reflect recent changes and improvements. - Removed outdated entries and consolidated recent activities for clarity. - Enhanced documentation for hooks, settings, and pagination implementations. - Streamlined test suite documentation for server and worker agents, indicating recent test audits and cleanup efforts. - Adjusted plans to remove obsolete entries and focus on current implementation strategies. * docs: comprehensive v9.0 documentation audit and updates - Add usage/folder-context to docs.json navigation (was documented but hidden!) - Update introduction.mdx with v9.0 release notes (Live Context, Worktree Support, Windows Fixes) - Add CLAUDE_MEM_WORKER_HOST setting to configuration.mdx - Add Folder Context Files section with link to detailed docs - Document worktree support in folder-context.mdx - Update terminology from "mem-search skill" to "MCP tools" throughout active docs - Update Search Pipeline in architecture/overview.mdx - Update usage/getting-started.mdx with MCP tools terminology - Update usage/claude-desktop.mdx title and terminology - Update hooks-architecture.mdx reference 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: add recent activity log for worker CLI with detailed entries * chore: update CLAUDE.md context files 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: add brainstorming report for CLAUDE.md distribution architecture --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>	2026-01-05 22:41:42 -05:00
Alex Newman	f21ea97c39	refactor: decompose monolith into modular architecture with comprehensive test suite (#538) ... * fix: prevent memory_session_id from equaling content_session_id The bug: memory_session_id was initialized to contentSessionId as a "placeholder for FK purposes". This caused the SDK resume logic to inject memory agent messages into the USER's Claude Code transcript, corrupting their conversation history. Root cause: - SessionStore.createSDKSession initialized memory_session_id = contentSessionId - SDKAgent checked memorySessionId !== contentSessionId but this check only worked if the session was fetched fresh from DB The fix: - SessionStore: Initialize memory_session_id as NULL, not contentSessionId - SDKAgent: Simple truthy check !!session.memorySessionId (NULL = fresh start) - Database migration: Ran UPDATE to set memory_session_id = NULL for 1807 existing sessions that had the bug Also adds [ALIGNMENT] logging across the session lifecycle to help debug session continuity issues: - Hook entry: contentSessionId + promptNumber - DB lookup: contentSessionId → memorySessionId mapping proof - Resume decision: shows which memorySessionId will be used for resume - Capture: logs when memorySessionId is captured from first SDK response UI: Added "Alignment" quick filter button in LogsModal to show only alignment logs for debugging session continuity. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor: improve error handling in worker-service.ts - Fix GENERIC_CATCH anti-patterns by logging full error objects instead of just messages - Add [ANTI-PATTERN IGNORED] markers for legitimate cases (cleanup, hot paths) - Simplify error handling comments to be more concise - Improve httpShutdown() error discrimination for ECONNREFUSED - Reduce LARGE_TRY_BLOCK issues in initialization code Part of anti-pattern cleanup plan (132 total issues) * refactor: improve error logging in SearchManager.ts - Pass full error objects to logger instead of just error.message - Fixes PARTIAL_ERROR_LOGGING anti-patterns (10 instances) - Better debugging visibility when Chroma queries fail Part of anti-pattern cleanup (133 remaining) * refactor: improve error logging across SessionStore and mcp-server - SessionStore.ts: Fix error logging in column rename utility - mcp-server.ts: Log full error objects instead of just error.message - Improve error handling in Worker API calls and tool execution Part of anti-pattern cleanup (133 remaining) * Refactor hooks to streamline error handling and loading states - Simplified error handling in useContextPreview by removing try-catch and directly checking response status. - Refactored usePagination to eliminate try-catch, improving readability and maintaining error handling through response checks. - Cleaned up useSSE by removing unnecessary try-catch around JSON parsing, ensuring clarity in message handling. - Enhanced useSettings by streamlining the saving process, removing try-catch, and directly checking the result for success. * refactor: add error handling back to SearchManager Chroma calls - Wrap queryChroma calls in try-catch to prevent generator crashes - Log Chroma errors as warnings and fall back gracefully - Fixes generator failures when Chroma has issues - Part of anti-pattern cleanup recovery * feat: Add generator failure investigation report and observation duplication regression report - Created a comprehensive investigation report detailing the root cause of generator failures during anti-pattern cleanup, including the impact, investigation process, and implemented fixes. - Documented the critical regression causing observation duplication due to race conditions in the SDK agent, outlining symptoms, root cause analysis, and proposed fixes. * fix: address PR #528 review comments - atomic cleanup and detector improvements This commit addresses critical review feedback from PR #528: ## 1. Atomic Message Cleanup (Fix Race Condition) **Problem**: SessionRoutes.ts generator error handler had race condition - Queried messages then marked failed in loop - If crash during loop → partial marking → inconsistent state **Solution**: - Added `markSessionMessagesFailed()` to PendingMessageStore.ts - Single atomic UPDATE statement replaces loop - Follows existing pattern from `resetProcessingToPending()` **Files**: - src/services/sqlite/PendingMessageStore.ts (new method) - src/services/worker/http/routes/SessionRoutes.ts (use new method) ## 2. Anti-Pattern Detector Improvements **Problem**: Detector didn't recognize logger.failure() method - Lines 212 & 335 already included "failure" - Lines 112-113 (PARTIAL_ERROR_LOGGING detection) did not **Solution**: Updated regex patterns to include "failure" for consistency **Files**: - scripts/anti-pattern-test/detect-error-handling-antipatterns.ts ## 3. Documentation **PR Comment**: Added clarification on memory_session_id fix location - Points to SessionStore.ts:1155 - Explains why NULL initialization prevents message injection bug ## Review Response Addresses "Must Address Before Merge" items from review: ✅ Clarified memory_session_id bug fix location (via PR comment) ✅ Made generator error handler message cleanup atomic ❌ Deferred comprehensive test suite to follow-up PR (keeps PR focused) ## Testing - Build passes with no errors - Anti-pattern detector runs successfully - Atomic cleanup follows proven pattern from existing methods 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix: FOREIGN KEY constraint and missing failed_at_epoch column Two critical bugs fixed: 1. Missing failed_at_epoch column in pending_messages table - Added migration 20 to create the column - Fixes error when trying to mark messages as failed 2. FOREIGN KEY constraint failed when storing observations - All three agents (SDK, Gemini, OpenRouter) were passing session.contentSessionId instead of session.memorySessionId - storeObservationsAndMarkComplete expects memorySessionId - Added null check and clear error message However, observations still not saving - see investigation report. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Refactor hook input parsing to improve error handling - Added a nested try-catch block in new-hook.ts, save-hook.ts, and summary-hook.ts to handle JSON parsing errors more gracefully. - Replaced direct error throwing with logging of the error details using logger.error. - Ensured that the process exits cleanly after handling input in all three hooks. * docs: add monolith refactor report with system breakdown Comprehensive analysis of codebase identifying: - 14 files over 500 lines requiring refactoring - 3 critical monoliths (SessionStore, SearchManager, worker-service) - 80% code duplication across agent files - 5-phase refactoring roadmap with domain-based architecture * docs: update monolith report post session-logging merge - SessionStore grew to 2,011 lines (49 methods) - highest priority - SearchManager reduced to 1,778 lines (improved) - Agent files reduced by ~45 lines combined - Added trend indicators and post-merge observations - Core refactoring proposal remains valid * refactor(sqlite): decompose SessionStore into modular architecture Extract the 2011-line SessionStore.ts monolith into focused, single-responsibility modules following grep-optimized progressive disclosure pattern: New module structure: - sessions/ - Session creation and retrieval (create.ts, get.ts, types.ts) - observations/ - Observation storage and queries (store.ts, get.ts, recent.ts, files.ts, types.ts) - summaries/ - Summary storage and queries (store.ts, get.ts, recent.ts, types.ts) - prompts/ - User prompt management (store.ts, get.ts, types.ts) - timeline/ - Cross-entity timeline queries (queries.ts) - import/ - Bulk import operations (bulk.ts) - migrations/ - Database migrations (runner.ts) New coordinator files: - Database.ts - ClaudeMemDatabase class with re-exports - transactions.ts - Atomic cross-entity transactions - Named re-export facades (Sessions.ts, Observations.ts, etc.) Key design decisions: - All functions take `db: Database` as first parameter (functional style) - Named re-exports instead of index.ts for grep-friendliness - SessionStore retained as backward-compatible wrapper - Target file size: 50-150 lines (60% compliance) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor(agents): extract shared logic into modular architecture Consolidate duplicate code across SDKAgent, GeminiAgent, and OpenRouterAgent into focused utility modules. Total reduction: 500 lines (29%). New modules in src/services/worker/agents/: - ResponseProcessor.ts: Atomic DB transactions, Chroma sync, SSE broadcast - ObservationBroadcaster.ts: SSE event formatting and dispatch - SessionCleanupHelper.ts: Session state cleanup and stuck message reset - FallbackErrorHandler.ts: Provider error detection for fallback logic - types.ts: Shared interfaces (WorkerRef, SSE payloads, StorageResult) Bug fix: SDKAgent was incorrectly using obs.files instead of obs.files_read and hardcoding files_modified to empty array. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor(search): extract search strategies into modular architecture Decompose SearchManager into focused strategy pattern with: - SearchOrchestrator: Coordinates strategy selection and fallback - ChromaSearchStrategy: Vector semantic search via ChromaDB - SQLiteSearchStrategy: Filter-only queries for date/project/type - HybridSearchStrategy: Metadata filtering + semantic ranking - ResultFormatter: Markdown table formatting for results - TimelineBuilder: Chronological timeline construction - Filter modules: DateFilter, ProjectFilter, TypeFilter SearchManager now delegates to new infrastructure while maintaining full backward compatibility with existing public API. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor(context): decompose context-generator into modular architecture Extract 660-line monolith into focused components: - ContextBuilder: Main orchestrator (~160 lines) - ContextConfigLoader: Configuration loading - TokenCalculator: Token budget calculations - ObservationCompiler: Data retrieval and query building - MarkdownFormatter/ColorFormatter: Output formatting - Section renderers: Header, Timeline, Summary, Footer Maintains full backward compatibility - context-generator.ts now delegates to new ContextBuilder while preserving public API. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * refactor(worker): decompose worker-service into modular infrastructure Split 2000+ line monolith into focused modules: Infrastructure: - ProcessManager: PID files, signal handlers, child process cleanup - HealthMonitor: Port checks, health polling, version matching - GracefulShutdown: Coordinated cleanup on exit Server: - Server: Express app setup, core routes, route registration - Middleware: Re-exports from existing middleware - ErrorHandler: Centralized error handling with AppError class Integrations: - CursorHooksInstaller: Full Cursor IDE integration (registry, hooks, MCP) WorkerService now acts as thin coordinator wiring all components together. Maintains full backward compatibility with existing public API. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Refactor session queue processing and database interactions - Implement claim-and-delete pattern in SessionQueueProcessor to simplify message handling and eliminate duplicate processing. - Update PendingMessageStore to support atomic claim-and-delete operations, removing the need for intermediate processing states. - Introduce storeObservations method in SessionStore for simplified observation and summary storage without message tracking. - Remove deprecated methods and clean up session state management in worker agents. - Adjust response processing to accommodate new storage patterns, ensuring atomic transactions for observations and summaries. - Remove unnecessary reset logic for stuck messages due to the new queue handling approach. * Add duplicate observation cleanup script Script to clean up duplicate observations created by the batching bug where observations were stored once per message ID instead of once per observation. Includes safety checks to always keep at least one copy. Usage: bun scripts/cleanup-duplicates.ts # Dry run bun scripts/cleanup-duplicates.ts --execute # Delete duplicates bun scripts/cleanup-duplicates.ts --aggressive # Ignore time window 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test(sqlite): add comprehensive test suite for SQLite repositories Add 44 tests across 5 test files covering: - Sessions: CRUD operations and schema validation - Observations: creation, retrieval, filtering, and ordering - Prompts: persistence and association with observations - Summaries: generation tracking and session linkage - Transactions: context management and rollback behavior 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test(worker): add comprehensive test suites for worker agent modules Add test coverage for response-processor, observation-broadcaster, session-cleanup-helper, and fallback-error-handler agents. Fix type import issues across search module (use `import type` for type-only imports) and update worker-service main module detection for ESM/CJS compatibility. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test(search): add comprehensive test suites for search module Add test coverage for the refactored search architecture: - SearchOrchestrator: query coordination and caching - ResultFormatter: pagination, sorting, and field mapping - SQLiteSearchStrategy: database search operations - ChromaSearchStrategy: vector similarity search - HybridSearchStrategy: combined search with score fusion 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test(context): add comprehensive test suites for context-generator modules Add test coverage for the modular context-generator architecture: - context-builder.test.ts: Tests for context building and result assembly - observation-compiler.test.ts: Tests for observation compilation with privacy tags - token-calculator.test.ts: Tests for token budget calculations - formatters/markdown-formatter.test.ts: Tests for markdown output formatting 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test(infrastructure): add comprehensive test suites for worker infrastructure modules Add test coverage for graceful-shutdown, health-monitor, and process-manager modules extracted during the worker-service refactoring. All 32 tests pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * test(server): add comprehensive test suites for server modules Add test coverage for Express server infrastructure: - error-handler.test.ts: Tests error handling middleware including validation errors, database errors, and async error handling - server.test.ts: Tests server initialization, middleware configuration, and route mounting for all API endpoints 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * chore(package): add test scripts for modular test suites Add npm run scripts to simplify running tests: - test: run all tests - test:sqlite, test:agents, test:search, test:context, test:infra, test:server 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * build assets * feat(tests): add detailed failure analysis reports for session ID refactor, validation, and store tests - Created reports for session ID refactor test failures, highlighting 8 failures due to design mismatches. - Added session ID usage validation report detailing 10 failures caused by outdated assumptions in tests. - Documented session store test failures, focusing on foreign key constraint violations in 2 tests. - Compiled a comprehensive test suite report summarizing overall test results, including 28 failing tests across various categories. * fix(tests): align session ID tests with NULL-based initialization Update test expectations to match implementation where memory_session_id starts as NULL (not equal to contentSessionId) per architecture decision that memory_session_id must NEVER equal contentSessionId. Changes: - session_id_refactor.test.ts: expect NULL initial state, add updateMemorySessionId() calls - session_id_usage_validation.test.ts: update placeholder detection to check !== null - session_store.test.ts: add updateMemorySessionId() before storeObservation/storeSummary 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(tests): update GeminiAgent tests with correct field names and mocks - Rename deprecated fields: claudeSessionId → contentSessionId, sdkSessionId → memorySessionId, pendingProcessingIds → pendingMessages - Add missing required ActiveSession fields - Add storeObservations mock (plural) for ResponseProcessor compatibility - Fix settings mock to use correct CLAUDE_MEM_GEMINI_RATE_LIMITING_ENABLED key - Add await to rejects.toThrow assertion 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(tests): add logger imports and fix coverage test exclusions Phase 3 of test suite fixes: - Add logger imports to 34 high-priority source files (SQLite, worker, context) - Exclude CLI-facing files from console.log check (worker-service.ts, integrations/*Installer.ts) as they use console.log intentionally for interactive user output 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * docs: update SESSION_ID_ARCHITECTURE for NULL-based initialization Update documentation to reflect that memory_session_id starts as NULL, not as a placeholder equal to contentSessionId. This matches the implementation decision that memory_session_id must NEVER equal contentSessionId to prevent injecting memory messages into user transcripts. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * chore(deps): update esbuild and MCP SDK - esbuild: 0.25.12 → 0.27.2 (fixes minifyIdentifiers issue) - @modelcontextprotocol/sdk: 1.20.1 → 1.25.1 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * build assets and updates * chore: remove bun.lock and add to gitignore 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>	2026-01-03 23:58:41 -05:00