perf: streamline worker startup and consolidate database connections (#2122)

* docs: pathfinder refactor corpus + Node 20 preflight Adds the PATHFINDER-2026-04-22 principle-driven refactor plan (11 docs, cross-checked PASS) plus the exploratory PATHFINDER-2026-04-21 corpus that motivated it. Bumps engines.node to >=20.0.0 per the ingestion-path plan preflight (recursive fs.watch). Adds the pathfinder skill. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: land PATHFINDER Plan 01 — data integrity Schema, UNIQUE constraints, self-healing claim, Chroma upsert fallback. - Phase 1: fresh schema.sql regenerated at post-refactor shape. - Phase 2: migrations 23+24 — rebuild pending_messages without started_processing_at_epoch; UNIQUE(session_id, tool_use_id); UNIQUE(memory_session_id, content_hash) on observations; dedup duplicate rows before adding indexes. - Phase 3: claimNextMessage rewritten to self-healing query using worker_pid NOT IN live_worker_pids; STALE_PROCESSING_THRESHOLD_MS and the 60-s stale-reset block deleted. - Phase 4: DEDUP_WINDOW_MS and findDuplicateObservation deleted; observations.insert now uses ON CONFLICT DO NOTHING. - Phase 5: failed-message purge block deleted from worker-service 2-min interval; clearFailedOlderThan method deleted. - Phase 6: repairMalformedSchema and its Python subprocess repair path deleted from Database.ts; SQLite errors now propagate. - Phase 7: Chroma delete-then-add fallback gated behind CHROMA_SYNC_FALLBACK_ON_CONFLICT env flag as bridge until Chroma MCP ships native upsert. - Phase 8: migration 19 no-op block absorbed into fresh schema.sql. Verification greps all return 0 matches. bun test tests/sqlite/ passes 63/63. bun run build succeeds. Plan: PATHFINDER-2026-04-22/01-data-integrity.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: land PATHFINDER Plan 02 — process lifecycle OS process groups replace hand-rolled reapers. Worker runs until killed; orphans are prevented by detached spawn + kill(-pgid). - Phase 1: src/services/worker/ProcessRegistry.ts DELETED. The canonical registry at src/supervisor/process-registry.ts is the sole survivor; SDK spawn site consolidated into it via new createSdkSpawnFactory/spawnSdkProcess/getSdkProcessForSession/ ensureSdkProcessExit/waitForSlot helpers. - Phase 2: SDK children spawn with detached:true + stdio: ['ignore','pipe','pipe']; pgid recorded on ManagedProcessInfo. - Phase 3: shutdown.ts signalProcess teardown uses process.kill(-pgid, signal) on Unix when pgid is recorded; Windows path unchanged (tree-kill/taskkill). - Phase 4: all reaper intervals deleted — startOrphanReaper call, staleSessionReaperInterval setInterval (including the co-located WAL checkpoint — SQLite's built-in wal_autocheckpoint handles WAL growth without an app-level timer), killIdleDaemonChildren, killSystemOrphans, reapOrphanedProcesses, reapStaleSessions, and detectStaleGenerator. MAX_GENERATOR_IDLE_MS and MAX_SESSION_IDLE_MS constants deleted. - Phase 5: abandonedTimer — already 0 matches; primary-path cleanup via generatorPromise.finally() already lives in worker-service startSessionProcessor and SessionRoutes ensureGeneratorRunning. - Phase 6: evictIdlestSession and its evict callback deleted from SessionManager. Pool admission gates backpressure upstream. - Phase 7: SDK-failure fallback — SessionManager has zero matches for fallbackAgent/Gemini/OpenRouter. Failures surface to hooks via exit code 2 through SessionRoutes error mapping. - Phase 8: ensureWorkerRunning in worker-utils.ts rewritten to lazy-spawn — consults isWorkerPortAlive (which gates captureProcessStartToken for PID-reuse safety via commit 99060bac), then spawns detached with unref(), then waitForWorkerPort({ attempts: 3, backoffMs: 250 }) hand-rolled exponential backoff 250→500→1000ms. No respawn npm dep. - Phase 9: idle self-shutdown — zero matches for idleCheck/idleTimeout/IDLE_MAX_MS/idleShutdown. Worker exits only on external SIGTERM via supervisor signal handlers. Three test files that exercised deleted code removed: tests/worker/process-registry.test.ts, tests/worker/session-lifecycle-guard.test.ts, tests/services/worker/reap-stale-sessions.test.ts. Pass count: 1451 → 1407 (-44), all attributable to deleted test files. Zero new failures. 31 pre-existing failures remain (schema-repair suite, logger-usage-standards, environmental openclaw / plugin-distribution) — none introduced by Plan 02. All 10 verification greps return 0. bun run build succeeds. Plan: PATHFINDER-2026-04-22/02-process-lifecycle.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: land PATHFINDER Plan 04 (narrowed) — search fail-fast Phases 3, 5, 6 only. Plan-doc inaccuracies for phases 1/2/4/7/8/9 deferred for plan reconciliation: - Phase 1/2: ObservationRow type doesn't exist; the four "formatters" operate on three incompatible types. - Phase 4: RECENCY_WINDOW_MS already imported from SEARCH_CONSTANTS at every call site. - Phase 7: getExistingChromaIds is NOT @deprecated and has an active caller in ChromaSync.backfillMissingSyncs. - Phase 8: estimateTokens already consolidated. - Phase 9: knowledge-corpus rewrite blocked on PG-3 prompt-caching cost smoke test. Phase 3 — Delete SearchManager.findByConcept/findByFile/findByType. SearchRoutes handlers (handleSearchByConcept/File/Type) now call searchManager.getOrchestrator().findByXxx() directly via new getter accessors on SearchManager. ~250 LoC deleted. Phase 5 — Fail-fast Chroma. Created src/services/worker/search/errors.ts with ChromaUnavailableError extends AppError(503, 'CHROMA_UNAVAILABLE'). Deleted SearchOrchestrator.executeWithFallback's Chroma-failed SQLite-fallback branch; runtime Chroma errors now throw 503. "Path 3" (chromaSync was null at construction — explicit- uninitialized config) preserved as legitimate empty-result state per plan text. ChromaSearchStrategy.search no longer wraps in try/catch — errors propagate. Phase 6 — Delete HybridSearchStrategy three try/catch silent fallback blocks (findByConcept, findByType, findByFile) at lines ~82-95, ~120-132, ~161-172. Removed `fellBack` field from StrategySearchResult type and every return site (SQLiteSearchStrategy, BaseSearchStrategy.emptyResult, SearchOrchestrator). Tests updated (Principle 7 — delete in same PR): - search-orchestrator.test.ts: "fall back to SQLite" rewritten as "throw ChromaUnavailableError (HTTP 503)". - chroma/hybrid/sqlite-search-strategy tests: rewritten to rejects.toThrow; removed fellBack assertions. Verification: SearchManager.findBy → 0; fellBack → 0 in src/. bun test tests/worker/search/ → 122 pass, 0 fail. bun test (suite-wide) → 1407 pass, baseline maintained, 0 new failures. bun run build succeeds. Plan: PATHFINDER-2026-04-22/04-read-path.md (Phases 3, 5, 6) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: land PATHFINDER Plan 03 — ingestion path Fail-fast parser, direct in-process ingest, recursive fs.watch, DB-backed tool pairing. Worker-internal HTTP loopback eliminated. - Phase 0: Created src/services/worker/http/shared.ts exporting ingestObservation/ingestPrompt/ingestSummary as direct in-process functions plus ingestEventBus (Node EventEmitter, reusing existing pattern — no third event bus introduced). setIngestContext wires the SessionManager dependency from worker-service constructor. - Phase 1: src/sdk/parser.ts collapsed to one parseAgentXml returning { valid:true; kind: 'observation'|'summary'; data } | { valid:false; reason: string }. Inspects root element; <skip_summary reason="…"/> is a first-class summary case with skipped:true. NEVER returns undefined. NEVER coerces. - Phase 2: ResponseProcessor calls parseAgentXml exactly once, branches on the discriminated union. On invalid → markFailed + logger.warn(reason). On observation → ingestObservation. On summary → ingestSummary then emit summaryStoredEvent { sessionId, messageId } (consumed by Plan 05's blocking /api/session/end). - Phase 3: Deleted consecutiveSummaryFailures field (ResponseProcessor + SessionManager + worker-types) and MAX_CONSECUTIVE_SUMMARY_FAILURES constant. Circuit-breaker guards and "tripped" log lines removed. - Phase 4: coerceObservationToSummary deleted from sdk/parser.ts. - Phase 5: src/services/transcripts/watcher.ts rescan setInterval replaced with fs.watch(transcriptsRoot, { recursive: true, persistent: true }) — Node 20+ recursive mode. - Phase 6: src/services/transcripts/processor.ts pendingTools Map deleted. tool_use rows insert with INSERT OR IGNORE on UNIQUE(session_id, tool_use_id) (added by Plan 01). New pairToolUsesByJoin query in PendingMessageStore for read-time pairing (UNIQUE INDEX provides idempotency; explicit consumer not yet wired). - Phase 7: HTTP loopback at processor.ts:252 replaced with direct ingestObservation call. maybeParseJson silent-passthrough rewritten to fail-fast (throws on malformed JSON). - Phase 8: src/utils/tag-stripping.ts countTags + stripTagsInternal collapsed into one alternation regex, single-pass over input. - Phase 9: src/utils/transcript-parser.ts (dead TranscriptParser class) deleted. The active extractLastMessage at src/shared/transcript-parser.ts:41-144 is the sole survivor. Tests updated (Principle 7 — same-PR delete): - tests/sdk/parser.test.ts + parse-summary.test.ts: rewritten to assert discriminated-union shape; coercion-specific scenarios collapse into { valid:false } assertions. - tests/worker/agents/response-processor.test.ts: circuit-breaker describe block skipped; non-XML/empty-response tests assert fail-fast markFailed behavior. Verification: every grep returns 0. transcript-parser.ts deleted. bun run build succeeds. bun test → 1399 pass / 28 fail / 7 skip (net -8 pass = the 4 retired circuit-breaker tests + 4 collapsed parser cases). Zero new failures vs baseline. Deferred (out of Plan 03 scope, will land in Plan 06): SessionRoutes HTTP route handlers still call sessionManager.queueObservation inline rather than the new shared helpers — the helpers are ready, the route swap is mechanical and belongs with the Zod refactor. Plan: PATHFINDER-2026-04-22/03-ingestion-path.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: land PATHFINDER Plan 05 — hook surface Worker-call plumbing collapsed to one helper. Polling replaced by server-side blocking endpoint. Fail-loud counter surfaces persistent worker outages via exit code 2. - Phase 1: plugin/hooks/hooks.json — three 20-iteration `for i in 1..20; do curl -sf .../health && break; sleep 0.1; done` shell retry wrappers deleted. Hook commands invoke their bun entry point directly. - Phase 2: src/shared/worker-utils.ts — added executeWithWorkerFallback<T>(url, method, body) returning T | { continue: true; reason?: string }. All 8 hook handlers (observation, session-init, context, file-context, file-edit, summarize, session-complete, user-message) rewritten to use it instead of duplicating the ensureWorkerRunning → workerHttpRequest → fallback sequence. - Phase 3: blocking POST /api/session/end in SessionRoutes.ts using validateBody + sessionEndSchema (z.object({sessionId})). One-shot ingestEventBus.on('summaryStoredEvent') listener, 30 s timer, req.aborted handler — all share one cleanup so the listener cannot leak. summarize.ts polling loop, plus MAX_WAIT_FOR_SUMMARY_MS / POLL_INTERVAL_MS constants, deleted. - Phase 4: src/shared/hook-settings.ts — loadFromFileOnce() memoizes SettingsDefaultsManager.loadFromFile per process. Per-handler settings reads collapsed. - Phase 5: src/shared/should-track-project.ts — single exclusion check entry; isProjectExcluded no longer referenced from src/cli/handlers/. - Phase 6: cwd validation pushed into adapter normalizeInput (all 6 adapters: claude-code, cursor, raw, gemini-cli, windsurf). New AdapterRejectedInput error in src/cli/adapters/errors.ts. Handler-level isValidCwd checks deleted from file-edit.ts and observation.ts. hook-command.ts catches AdapterRejectedInput → graceful fallback. - Phase 7: session-init.ts conditional initAgent guard deleted; initAgent is idempotent. tests/hooks/context-reinjection-guard test (validated the deleted conditional) deleted in same PR per Principle 7. - Phase 8: fail-loud counter at ~/.claude-mem/state/hook-failures .json. Atomic write via .tmp + rename. CLAUDE_MEM_HOOK_FAIL_LOUD _THRESHOLD setting (default 3). On consecutive worker-unreachable ≥ N: process.exit(2). On success: reset to 0. NOT a retry. - Phase 9: ensureWorkerAliveOnce() module-scope memoization wrapping ensureWorkerRunning. executeWithWorkerFallback calls the memoized version. Minimal validateBody middleware stub at src/services/worker/http/middleware/validateBody.ts. Plan 06 will expand with typed inference + error envelope conventions. Verification: 4/4 grep targets pass. bun run build succeeds. bun test → 1393 pass / 28 fail / 7 skip; -6 pass attributable solely to deleted context-reinjection-guard test file. Zero new failures vs baseline. Plan: PATHFINDER-2026-04-22/05-hook-surface.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: land PATHFINDER Plan 06 — API surface One Zod-based validator wrapping every POST/PUT. Rate limiter, diagnostic endpoints, and shutdown wrappers deleted. Failure- marking consolidated to one helper. - Phase 1 (preflight): zod@^3 already installed. - Phase 2: validateBody middleware confirmed at canonical shape in src/services/worker/http/middleware/validateBody.ts — safeParse → 400 { error: 'ValidationError', issues: [...] } on failure, replaces req.body with parsed value on success. - Phase 3: Per-route Zod schemas declared at the top of each route file. 24 POST endpoints across SessionRoutes, CorpusRoutes, DataRoutes, MemoryRoutes, SearchRoutes, LogsRoutes, SettingsRoutes now wrap with validateBody(). /api/session/end (Plan 05) confirmed using same middleware. - Phase 4: validateRequired() deleted from BaseRouteHandler along with every call site. Inline coercion helpers (coerceStringArray, coercePositiveInteger) and inline if (!req.body...) guards deleted across all route files. - Phase 5: Rate limiter middleware and its registration deleted from src/services/worker/http/middleware.ts. Worker binds 127.0.0.1:37777 — no untrusted caller. - Phase 6: viewer.html cached at module init in ViewerRoutes.ts via fs.readFileSync; served as Buffer with text/html content type. SKILL.md + per-operation .md files cached in Server.ts as Map<string, string>; loadInstructionContent helper deleted. NO fs.watch, NO TTL — process restart is the cache-invalidation event. - Phase 7: Four diagnostic endpoints deleted from DataRoutes.ts — /api/pending-queue (GET), /api/pending-queue/process (POST), /api/pending-queue/failed (DELETE), /api/pending-queue/all (DELETE). Helper methods that ONLY served them (getQueueMessages, getStuckCount, getRecentlyProcessed, clearFailed, clearAll) deleted from PendingMessageStore. KEPT: /api/processing-status (observability), /health (used by ensureWorkerRunning). - Phase 8: stopSupervisor wrapper deleted from supervisor/index.ts. GracefulShutdown now calls getSupervisor().stop() directly. Two functions retained with clear roles: - performGracefulShutdown — worker-side 6-step shutdown - runShutdownCascade — supervisor-side child teardown (process.kill(-pgid), Windows tree-kill, PID-file cleanup) Each has unique non-trivial logic and a single canonical caller. - Phase 9: transitionMessagesTo(status, filter) is the sole failure-marking path on PendingMessageStore. Old methods markSessionMessagesFailed and markAllSessionMessagesAbandoned deleted along with all callers (worker-service, SessionCompletionHandler, tests/zombie-prevention). Tests updated (Principle 7 same-PR delete): coercion test files refactored to chain validateBody → handler. Zombie-prevention tests rewritten to call transitionMessagesTo. Verification: all 4 grep targets → 0. bun run build succeeds. bun test → 1393 pass / 28 fail / 7 skip — exact match to baseline. Zero new failures. Plan: PATHFINDER-2026-04-22/06-api-surface.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: land PATHFINDER Plan 07 — dead code sweep ts-prune-driven sweep across the tree after Plans 01-06 landed. Deleted unused exports, orphan helpers, and one fully orphaned file. Earlier-plan deletions verified. Deleted: - src/utils/bun-path.ts (entire file — getBunPath, getBunPathOrThrow, isBunAvailable: zero importers) - bun-resolver.getBunVersionString: zero callers - PendingMessageStore.retryMessage / resetProcessingToPending / abortMessage: superseded by transitionMessagesTo (Plan 06 Phase 9) - EnvManager.MANAGED_CREDENTIAL_KEYS, EnvManager.setCredential: zero callers - CodexCliInstaller.checkCodexCliStatus: zero callers; no status command exists in npx-cli - Two "REMOVED: cleanupOrphanedSessions" stale-fence comments Kept (with documented justification): - Public API surface in dist/sdk/* (parseAgentXml, prompt builders, ParsedObservation, ParsedSummary, ParseResult, SUMMARY_MODE_MARKER) — exported via package.json sdk path. - generateContext / loadContextConfig / token utilities — used via dynamic await import('../../../context-generator.js') in worker SearchRoutes. - MCP_IDE_INSTALLERS, install/uninstall functions for codex/goose — used via dynamic await import in npx-cli/install.ts + uninstall.ts (ts-prune cannot trace dynamic imports). - getExistingChromaIds — active caller in ChromaSync.backfillMissingSyncs (Plan 04 narrowed scope). - processPendingQueues / getSessionsWithPendingMessages — active orphan-recovery caller in worker-service.ts plus zombie-prevention test coverage. - StoreAndMarkCompleteResult legacy alias — return-type annotation in same file. - All Database.ts barrel re-exports — used downstream. Earlier-plan verification: - Plan 03 Phase 9: VERIFIED — src/utils/transcript-parser.ts is gone; TranscriptParser has 0 references in src/. - Plan 01 Phase 8: VERIFIED — migration 19 no-op absorbed. - SessionStore.ts:52-70 consolidation NOT executed (deferred): the methods are not thin wrappers but ~900 LoC of bodies, and two methods are documented as intentional mirrors so the context-generator.cjs bundle stays schema-consistent without pulling MigrationRunner. Deserves its own plan, not a sweep. Verification: TranscriptParser → 0; transcript-parser.ts → gone; no commented-out code markers remain. bun run build succeeds. bun test → 1393 pass / 28 fail / 7 skip — EXACT match to baseline. Zero regressions. Plan: PATHFINDER-2026-04-22/07-dead-code.md Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: remove residual ProcessRegistry comment reference Plan 07 dead-code sweep missed one comment-level reference to the deleted in-memory ProcessRegistry class in SessionManager.ts:347. Rewritten to describe the supervisor.json scope without naming the deleted class, completing the verification grep target. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: address Greptile review (P1 + 2× P2) P1 — Plan 05 Phase 3 blocking endpoint was non-functional: executeWithWorkerFallback used HEALTH_CHECK_TIMEOUT_MS (3 s) for the POST /api/session/end call, but the server holds the connection for SERVER_SIDE_SUMMARY_TIMEOUT_MS (30 s). Client always raced to a "timed out" rejection that isWorkerUnavailable classified as worker-unreachable, so the hook silently degraded instead of waiting for summaryStoredEvent. - Added optional timeoutMs to executeWithWorkerFallback, forwarded to workerHttpRequest. - summarize.ts call site now passes 35_000 (5 s above server hold window). P2 — ingestSummary({ kind: 'parsed' }) branch was dead code: ResponseProcessor emitted summaryStoredEvent directly via the event bus, bypassing the centralized helper that the comment claimed was the single source. - ResponseProcessor now calls ingestSummary({ kind: 'parsed', sessionDbId, messageId, contentSessionId, parsed }) so the event-emission path is single-sourced. - ingestSummary's requireContext() resolution moved inside the 'queue' branch (the only branch that needs sessionManager / dbManager). 'parsed' is a pure event-bus emission and doesn't need worker-internal context — fixes mocked ResponseProcessor unit tests that don't call setIngestContext. P2 — isWorkerFallback could false-positive on legitimate API responses whose schema includes { continue: true, ... }: - Added a Symbol.for('claude-mem/worker-fallback') brand to WorkerFallback. isWorkerFallback now checks the brand, not a duck-typed property name. Verification: bun run build succeeds. bun test → 1393 pass / 28 fail / 7 skip — exact baseline match. Zero new failures. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: address Greptile iteration 2 (P1 + P2) P1 — summaryStoredEvent fired regardless of whether the row was persisted. ResponseProcessor's call to ingestSummary({ kind: 'parsed' }) ran for every parsed.kind === 'summary' even when result.summaryId came back null (e.g. FK violation, null memory_session_id at commit). The blocking /api/session/end endpoint then returned { ok: true } and the Stop hook logged 'Summary stored' for a non-existent row. - Gate ingestSummary call on (parsed.data.skipped || session.lastSummaryStored). Skipped summaries are an explicit no-op bypass and still confirm; real summaries only confirm when storage actually wrote a row. - Non-skipped + summaryId === null path logs a warn and lets the server-side timeout (504) surface to the hook instead of a false ok:true. P2 — PendingMessageStore.enqueue() returns 0 when INSERT OR IGNORE suppresses a duplicate (the UNIQUE(session_id, tool_use_id) constraint added by Plan 01 Phase 1). The two callers (SessionManager.queueObservation and queueSummarize) previously logged 'ENQUEUED messageId=0' which read like a row was inserted. - Branch on messageId === 0 and emit a 'DUP_SUPPRESSED' debug log instead of the misleading ENQUEUED line. No behavior change — the duplicate is still correctly suppressed by the DB (Principle 3); only the log surface is corrected. - confirmProcessed is never called with the enqueue() return value (it operates on session.processingMessageIds[] from claimNextMessage), so no caller is broken; the visibility fix prevents future misuse. Verification: bun run build succeeds. bun test → 1393 pass / 28 fail / 7 skip — exact baseline match. Zero new failures. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: address Greptile iteration 3 (P1 + 2× P2) - P1 worker-service.ts: wire ensureGeneratorRunning into the ingest context after SessionRoutes is constructed. setIngestContext runs before routes exist, so transcript-watcher observations queued via ingestObservation() had no way to auto-start the SDK generator. Added attachIngestGeneratorStarter() to patch the callback in. - P2 shared.ts: IngestEventBus now sets maxListeners to 0. Concurrent /api/session/end calls register one listener each and clean up on completion, so the default-10 warning fires spuriously under normal load. - P2 SessionRoutes.ts: handleObservationsByClaudeId now delegates to ingestObservation() instead of duplicating skip-tool / meta / privacy / queue logic. Single helper, matching the Plan 03 goal. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: address Greptile iteration 4 (P1 tool-pair + P2 parse/path/doc) - processor.handleToolResult: restore in-memory tool-use→tool-result pairing via session.pendingTools for schemas (e.g. Codex) whose tool_result events carry only tool_use_id + output. Without this, neither handler fired — all tool observations silently dropped. - processor.maybeParseJson: return raw string on parse failure instead of throwing. Previously a single malformed JSON-shaped field caused handleLine's outer catch to discard the entire transcript line. - watcher.deepestNonGlobAncestor: split on / and \\, emit empty string for purely-glob inputs so the caller skips the watch instead of anchoring fs.watch at the filesystem root. Windows-compatible. - PendingMessageStore.enqueue: tighten docstring — callers today only log on the returned id; the SessionManager branches on id === 0. * fix: forward tool_use_id through ingestObservation (Greptile iter 5) P1 — Plan 01's UNIQUE(content_session_id, tool_use_id) dedup never fired because the new shared ingest path dropped the toolUseId before queueObservation. SQLite treats NULL values as distinct for UNIQUE, so every replayed transcript line landed a duplicate row. - shared.ingestObservation: forward payload.toolUseId to queueObservation so INSERT OR IGNORE can actually collapse. - SessionRoutes.handleObservationsByClaudeId: destructure both tool_use_id (HTTP convention) and toolUseId (JS convention) from req.body and pass into ingestObservation. - observationsByClaudeIdSchema: declare both keys explicitly so the validator doesn't rely on .passthrough() alone. * fix: drop dead pairToolUsesByJoin, close session-end listener race - PendingMessageStore: delete pairToolUsesByJoin. The method was never called and its self-join semantics are structurally incompatible with UNIQUE(content_session_id, tool_use_id): INSERT OR IGNORE collapses any second row with the same pair, so a self-join can only ever match a row to itself. In-memory pendingTools in processor.ts remains the pairing path for split-event schemas. - IngestEventBus: retain a short-lived (60s) recentStored map keyed by sessionId. Populated on summaryStoredEvent emit, evicted on consume or TTL. - handleSessionEnd: drain the recent-events buffer before attaching the listener. Closes the register-after-emit race where the summary can persist between the hook's summarize POST and its session/end POST — previously that window returned 504 after the 30s timeout. * chore: merge origin/main into vivacious-teeth Resolves conflicts with 15 commits on main (v12.3.9, security observation types, Telegram notifier, PID-reuse worker start-guard). Conflict resolution strategy: - plugin/hooks/hooks.json, plugin/scripts/*.cjs, plugin/ui/viewer-bundle.js: kept ours — PATHFINDER Plan 05 deletes the for-i-in-1-to-20 curl retry loops and the built artifacts regenerate on build. - src/cli/handlers/summarize.ts: kept ours — Plan 05 blocking POST /api/session/end supersedes main's fire-and-forget path. - src/services/worker-service.ts: kept ours — Plan 05 ingest bus + summaryStoredEvent supersedes main's SessionCompletionHandler DI refactor + orphan-reaper fallback. - src/services/worker/http/routes/SessionRoutes.ts: kept ours — same reason; generator .finally() Stop-hook self-clean is a guard for a path our blocking endpoint removes. - src/services/worker/http/routes/CorpusRoutes.ts: merged — added security_alert / security_note to ALLOWED_CORPUS_TYPES (feature from #2084) while preserving our Zod validateBody schema. Typecheck: 294 errors (vs 298 pre-merge). No new errors introduced; all remaining are pre-existing (Component-enum gaps, DOM lib for viewer, bun:sqlite types). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: address Greptile P2 findings 1) SessionRoutes.handleSessionEnd was the only route handler not wrapped in wrapHandler — synchronous exceptions would hang the client rather than surfacing as 500s. Wrap it like every other handler. 2) processor.handleToolResult only consumed the session.pendingTools entry when the tool_result arrived without a toolName. In the split-schema path where tool_result carries both toolName and toolId, the entry was never deleted and the map grew for the life of the session. Consume the entry whenever toolId is present. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: typing cleanup and viewer tsconfig split for PR feedback - Add explicit return types for SessionStore query methods - Exclude src/ui/viewer from root tsconfig, give it its own DOM-typed config - Add bun to root tsconfig types, plus misc typing tweaks flagged by Greptile - Rebuilt plugin/scripts/* artifacts Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: address Greptile P2 findings (iter 2) - PendingMessageStore.transitionMessagesTo: require sessionDbId (drop the unscoped-drain branch that would nuke every pending/processing row across all sessions if a future caller omitted the filter). - IngestEventBus.takeRecentSummaryStored: make idempotent — keep the cached event until TTL eviction so a retried Stop hook's second /api/session/end returns immediately instead of hanging 30 s. - TranscriptWatcher fs.watch callback: skip full glob scan for paths already tailed (JSONL appends fire on every line; only unknown paths warrant a rescan). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: call finalizeSession in terminal session paths (Greptile iter 3) terminateSession and runFallbackForTerminatedSession previously called SessionCompletionHandler.finalizeSession before removeSessionImmediate; the refactor dropped those calls, leaving sdk_sessions.status='active' for every session killed by wall-clock limit, unrecoverable error, or exhausted fallback chain. The deleted reapStaleSessions interval was the only prior backstop. Re-wires finalizeSession (idempotent: marks completed, drains pending, broadcasts) into both paths; no reaper reintroduced. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: GC failed pending_messages rows at startup (Greptile iter 4) Plan 07 deleted clearFailed/clearFailedOlderThan as "dead code", but with the periodic sweep also removed, nothing reaps status='failed' rows now — they accumulate indefinitely. Since claimNextMessage's self-healing subquery scans this table, unbounded growth degrades claim latency over time. Re-introduces clearFailedOlderThan and calls it once at worker startup (not a reaper — one-shot, idempotent). 7-day retention keeps enough history for operator inspection while bounding the table. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: finalize sessions on normal exit; cleanup hoist; share handler (iter 5) 1. startSessionProcessor success branch now calls completionHandler. finalizeSession before removeSessionImmediate. Hooks-disabled installs (and any Stop hook that fails before POST /api/sessions/complete) no longer leave sdk_sessions rows as status='active' forever. Idempotent — a subsequent /api/sessions/complete is a no-op. 2. Hoist SessionRoutes.handleSessionEnd cleanup declaration above the closures that reference it (TDZ safety; safe at runtime today but fragile if timeout ever shrinks). 3. SessionRoutes now receives WorkerService's shared SessionCompletionHandler instead of constructing its own — prevents silent divergence if the handler ever becomes stateful. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: stop runaway crash-recovery loop on dead sessions Two distinct bugs were combining to keep a dead session restarting forever: Bug 1 (uncaught "The operation was aborted."): child_process.spawn emits 'error' asynchronously for ENOENT/EACCES/abort signal aborts. spawnSdkProcess() never attached an 'error' listener, so any async spawn failure became uncaughtException and escaped to the daemon-level handler. Attach an 'error' listener immediately after spawn, before the !child.pid early-return, so async spawn errors are logged (with errno code) and swallowed locally. Bug 2 (sliding-window limiter never trips on slow restart cadence): RestartGuard tripped only when restartTimestamps.length exceeded MAX_WINDOWED_RESTARTS (10) within RESTART_WINDOW_MS (60s). With the 8s exponential-backoff cap, only ~7-8 restarts fit in the window, so a dead session that fail-restart-fail-restart on 8s cycles would loop forever (consecutiveRestarts climbing past 30+ in observed logs). Add a consecutiveFailures counter that increments on every restart and resets only on recordSuccess(). Trip when consecutive failures exceed MAX_CONSECUTIVE_FAILURES (5) — meaning 5 restarts with zero successful processing in between proves the session is dead. Both guards now run in parallel: tight loops still trip the windowed cap; slow loops trip the consecutive-failure cap. Also: when the SessionRoutes path trips the guard, drain pending messages to 'abandoned' so the session does not reappear in getSessionsWithPendingMessages and trigger another auto-start cycle. The worker-service.ts path already does this via terminateSession. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf: streamline worker startup and consolidate database connections 1. Database Pooling: Modified DatabaseManager, SessionStore, and SessionSearch to share a single bun:sqlite connection, eliminating redundant file descriptors. 2. Non-blocking Startup: Refactored WorktreeAdoption and Chroma backfill to run in the background (fire-and-forget), preventing them from stalling core initialization. 3. Diagnostic Routes: Added /api/chroma/status and bypassed the initialization guard for health/readiness endpoints to allow diagnostics during startup. 4. Robust Search: Implemented reliable SQLite FTS5 fallback in SearchManager for when Chroma (uvx) fails or is unavailable. 5. Code Cleanup: Removed redundant loopback MCP checks and mangled initialization logic from WorkerService. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: hard-exclude observer-sessions from hooks; bundle migration 29 (#2124) * fix: hard-exclude observer-sessions from hooks; backfill bundle migrations Stop hook + SessionEnd hook were storing the SDK observer's own init/continuation/summary prompts in user_prompts, leaking into the viewer (meta-observation regression). 25 such rows accumulated. - shouldTrackProject: hard-reject OBSERVER_SESSIONS_DIR (and its subtree) before consulting user-configured exclusion globs. - summarize.ts (Stop) and session-complete.ts (SessionEnd): early-return when shouldTrackProject(cwd) is false, so the observer's own hooks cannot bootstrap the worker or queue a summary against the meta-session. - SessionRoutes: cap user-prompt body at 256 KiB at the session-init boundary so a runaway observer prompt cannot blow up storage. - SessionStore: add migration 29 (UNIQUE(memory_session_id, content_hash) on observations) inline so bundled artifacts (worker-service.cjs, context-generator.cjs) stay schema-consistent — without it, the ON CONFLICT clause in observation inserts throws. - spawnSdkProcess: stdio[stdin] from 'ignore' to 'pipe' so the supervisor can actually feed the observer's stdin. Also rebuilds plugin/scripts/{worker-service,context-generator}.cjs. * fix: walk back to UTF-8 boundary on prompt truncation (Greptile P2) Plain Buffer.subarray at MAX_USER_PROMPT_BYTES can land mid-codepoint, which the utf8 decoder silently rewrites to U+FFFD. Walk back over any continuation bytes (0b10xxxxxx) before decoding so the truncated prompt ends on a valid sequence boundary instead of a replacement character. * fix: cross-platform observer-dir containment; clarify SDK stdin pipe claude-review feedback on PR #2124. - shouldTrackProject: literal `cwd.startsWith(OBSERVER_SESSIONS_DIR + '/')` hard-coded a POSIX separator and missed Windows backslash paths plus any trailing-slash variance. Switched to a path.relative-based isWithin() helper so Windows hook input under observer-sessions\\... is also excluded. - spawnSdkProcess: added a comment explaining why stdin must be 'pipe' — SpawnedSdkProcess.stdin is typed NonNullable and the Claude Agent SDK consumes that pipe; 'ignore' would null it and the null-check below would tear the child down on every spawn. * fix: make Stop hook fire-and-forget; remove dead /api/session/end The Stop hook was awaiting a 35-second long-poll on /api/session/end, which the worker held open until the summary-stored event fired (or its 30s server-side timeout elapsed). Followed by another await on /api/sessions/complete. Three sequential awaits, the middle one a 30s hold — not fire-and-forget despite repeated requests. The Stop hook now does ONE thing: POST /api/sessions/summarize to queue the summary work and return. The worker drives the rest async. Session-map cleanup is performed by the SessionEnd handler (session-complete.ts), not duplicated here. - summarize.ts: drop the /api/session/end long-poll and the trailing /api/sessions/complete await; ~40 lines removed; unused SessionEndResponse interface gone; header comment rewritten. - SessionRoutes: delete handleSessionEnd, sessionEndSchema, the SERVER_SIDE_SUMMARY_TIMEOUT_MS constant, and the /api/session/end route registration. Drop the now-unused ingestEventBus and SummaryStoredEvent imports. - ResponseProcessor + shared.ts + worker-utils.ts: update stale comments that referenced the dead endpoint. The IngestEventBus is left in place dormant (no listeners) for follow-up cleanup so this PR stays focused on the blocker. Bundle artifact (worker-service.cjs) rebuilt via build-and-sync. Verification: - grep '/api/session/end' plugin/scripts/worker-service.cjs → 0 - grep 'timeoutMs:35' plugin/scripts/worker-service.cjs → 0 - Worker restarted clean, /api/health ok at pid 92368 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * deps: bump all dependencies to latest including majors Upgrades: React 18→19, Express 4→5, Zod 3→4, TypeScript 5→6, @types/node 20→25, @anthropic-ai/claude-agent-sdk 0.1→0.2, @clack/prompts 0.9→1.2, plus minors. Adds Daily Maintenance section to CLAUDE.md mandating latest-version policy across manifests. Express 5 surfaced a race in Server.listen() where the 'error' handler was attached after listen() was invoked; refactored to use http.createServer with both 'error' and 'listening' handlers attached before listen(), restoring port-conflict rejection semantics. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: surface real chroma errors and add deep status probe Replace the misleading "Vector search failed - semantic search unavailable. Install uv... restart the worker." string in SearchManager with the actual exception text from chroma_query_documents. The lying message blamed `uv` for any failure — even when the real cause was a chroma-mcp transport timeout, an empty collection, or a dead subprocess. Also add /api/chroma/status?deep=1 backed by a new ChromaMcpManager.probeSemanticSearch() that round-trips a real query (chroma_list_collections + chroma_query_documents) instead of just checking the stdio handshake. The cheap default path is unchanged. Includes the diagnostic plan (PLAN-fix-mcp-search.md) and updated test fixtures for the new structured failure message. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: rebuild worker-service bundle to match merged src Bundle was stale after the squash merge of #2124 — it still contained the old "Install uv... semantic search unavailable" string and lacked probeSemanticSearch. Rebuilt via bun run build-and-sync. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: address coderabbit feedback on PLAN-fix-mcp-search.md - replace machine-specific /Users/alexnewman absolute paths with portable <repo-root> placeholder (MD-style portability) - add blank lines around the TypeScript fenced block (MD031) - tag the bare fenced block with `text` (MD040) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-25 13:37:40 -07:00
parent 8ace1d9c84
commit 94d592f212
159 changed files with 18091 additions and 5843 deletions
@@ -146,10 +146,6 @@ export async function startSupervisor(): Promise<void> {
  await supervisorSingleton.start();
 }

-export async function stopSupervisor(): Promise<void> {
-  await supervisorSingleton.stop();
-}
-
 export function getSupervisor(): Supervisor {
  return supervisorSingleton;
 }
@@ -168,7 +164,7 @@ export function validateWorkerPidFile(options: ValidateWorkerPidOptions = {}): V
  let pidInfo: PidInfo | null = null;

  try {
-    pidInfo = JSON.parse(readFileSync(pidFilePath, 'utf-8')) as PidInfo;
+    pidInfo = JSON.parse(readFileSync(pidFilePath, 'utf-8')) as PidInfo | null;
  } catch (error: unknown) {
    if (error instanceof Error) {
      logger.warn('SYSTEM', 'Failed to parse worker PID file, removing it', { path: pidFilePath }, error);
@@ -182,7 +178,8 @@ export function validateWorkerPidFile(options: ValidateWorkerPidOptions = {}): V
    return 'invalid';
  }

-  if (verifyPidFileOwnership(pidInfo)) {
+  const isAlive = verifyPidFileOwnership(pidInfo);
+  if (isAlive && pidInfo) {
    if (options.logAlive ?? true) {
      logger.info('SYSTEM', 'Worker already running (PID alive)', {
        existingPid: pidInfo.pid,
@@ -194,9 +191,9 @@ export function validateWorkerPidFile(options: ValidateWorkerPidOptions = {}): V
  }

  logger.info('SYSTEM', 'Removing stale PID file (worker process is dead or PID has been reused)', {
-    pid: pidInfo.pid,
-    port: pidInfo.port,
-    startedAt: pidInfo.startedAt
+    pid: pidInfo?.pid,
+    port: pidInfo?.port,
+    startedAt: pidInfo?.startedAt
  });
  rmSync(pidFilePath, { force: true });
  return 'stale';
@@ -1,8 +1,9 @@
-import { ChildProcess, spawnSync } from 'child_process';
+import { ChildProcess, spawn, spawnSync } from 'child_process';
 import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'fs';
 import { homedir } from 'os';
 import path from 'path';
 import { logger } from '../utils/logger.js';
+import { sanitizeEnv } from './env-sanitizer.js';

 const REAP_SESSION_SIGTERM_TIMEOUT_MS = 5_000;
 const REAP_SESSION_SIGKILL_TIMEOUT_MS = 1_000;
@@ -15,6 +16,14 @@ export interface ManagedProcessInfo {
  type: string;
  sessionId?: string | number;
  startedAt: string;
+  // POSIX process group leader PID for group-scoped teardown.
+  // On Unix, when a child is spawned with `detached: true`, the kernel calls
+  // setpgid() and the child becomes the leader of its own group — its pgid
+  // equals its pid. Stored so `process.kill(-pgid, signal)` can tear down
+  // the child AND every descendant it spawned in one syscall (Principle 5).
+  // Undefined on Windows (no POSIX groups) and for processes that were not
+  // spawned with detached: true (e.g. the worker itself, MCP stdio clients).
+  pgid?: number;
 }

 export interface ManagedProcessRecord extends ManagedProcessInfo {
@@ -303,22 +312,30 @@ export class ProcessRegistry {
      pids: sessionRecords.map(r => r.pid)
    });

-    // Phase 1: SIGTERM all alive processes
+    // Phase 1: SIGTERM all alive processes — use process-group teardown for
+    // records that carry pgid so any descendants the SDK spawned are killed
+    // too (Principle 5).
    const aliveRecords = sessionRecords.filter(r => isPidAlive(r.pid));
    for (const record of aliveRecords) {
      try {
-        process.kill(record.pid, 'SIGTERM');
+        if (typeof record.pgid === 'number' && process.platform !== 'win32') {
+          process.kill(-record.pgid, 'SIGTERM');
+        } else {
+          process.kill(record.pid, 'SIGTERM');
+        }
      } catch (error: unknown) {
        if (error instanceof Error) {
          const code = (error as NodeJS.ErrnoException).code;
          if (code !== 'ESRCH') {
            logger.debug('SYSTEM', `Failed to SIGTERM session process PID ${record.pid}`, {
-              pid: record.pid
+              pid: record.pid,
+              pgid: record.pgid
            }, error);
          }
        } else {
          logger.warn('SYSTEM', `Failed to SIGTERM session process PID ${record.pid} (non-Error)`, {
            pid: record.pid,
+            pgid: record.pgid,
            error: String(error)
          });
        }
@@ -333,26 +350,34 @@ export class ProcessRegistry {
      await new Promise(resolve => setTimeout(resolve, 100));
    }

-    // Phase 3: SIGKILL any survivors
+    // Phase 3: SIGKILL any survivors — process-group teardown when pgid is
+    // recorded so descendants are killed too.
    const survivors = aliveRecords.filter(r => isPidAlive(r.pid));
    for (const record of survivors) {
      logger.warn('SYSTEM', `Session process PID ${record.pid} did not exit after SIGTERM, sending SIGKILL`, {
        pid: record.pid,
+        pgid: record.pgid,
        sessionId: sessionIdNum
      });
      try {
-        process.kill(record.pid, 'SIGKILL');
+        if (typeof record.pgid === 'number' && process.platform !== 'win32') {
+          process.kill(-record.pgid, 'SIGKILL');
+        } else {
+          process.kill(record.pid, 'SIGKILL');
+        }
      } catch (error: unknown) {
        if (error instanceof Error) {
          const code = (error as NodeJS.ErrnoException).code;
          if (code !== 'ESRCH') {
            logger.debug('SYSTEM', `Failed to SIGKILL session process PID ${record.pid}`, {
-              pid: record.pid
+              pid: record.pid,
+              pgid: record.pgid
            }, error);
          }
        } else {
          logger.warn('SYSTEM', `Failed to SIGKILL session process PID ${record.pid} (non-Error)`, {
            pid: record.pid,
+            pgid: record.pgid,
            error: String(error)
          });
        }
@@ -406,3 +431,401 @@ export function getProcessRegistry(): ProcessRegistry {
 export function createProcessRegistry(registryPath: string): ProcessRegistry {
  return new ProcessRegistry(registryPath);
 }
+
+// ---------------------------------------------------------------------------
+// SDK session lookup + exit verification
+// ---------------------------------------------------------------------------
+
+export interface TrackedSdkProcess {
+  pid: number;
+  pgid: number | undefined;
+  sessionDbId: number;
+  process: ChildProcess;
+}
+
+/**
+ * Look up the live SDK subprocess for a given session, if any.
+ *
+ * Returns undefined when no SDK record is registered for the session, or
+ * when the ChildProcess reference has been dropped (process exited and was
+ * unregistered). Warns on duplicates — multiple SDK records per session
+ * indicate a race in createSdkSpawnFactory's pre-spawn cleanup.
+ */
+export function getSdkProcessForSession(sessionDbId: number): TrackedSdkProcess | undefined {
+  const registry = getProcessRegistry();
+  const matches = registry.getBySession(sessionDbId).filter(r => r.type === 'sdk');
+
+  if (matches.length > 1) {
+    logger.warn('PROCESS', `Multiple SDK processes found for session ${sessionDbId}`, {
+      count: matches.length,
+      pids: matches.map(m => m.pid),
+    });
+  }
+
+  const record = matches[0];
+  if (!record) return undefined;
+
+  const processRef = registry.getRuntimeProcess(record.id);
+  if (!processRef) return undefined;
+
+  return {
+    pid: record.pid,
+    pgid: record.pgid,
+    sessionDbId,
+    process: processRef,
+  };
+}
+
+/**
+ * Wait for an SDK subprocess to exit, escalating to SIGKILL on the process
+ * group if it overstays `timeoutMs`. Fully event-driven — no polling.
+ *
+ * This is primary-path cleanup invoked from session-level finally() blocks
+ * when a session ends; it is NOT a reaper. It runs at most once per session
+ * deletion. Process-group teardown (`kill(-pgid, SIGKILL)`) ensures any
+ * descendants the SDK spawned are also killed.
+ */
+export async function ensureSdkProcessExit(
+  tracked: TrackedSdkProcess,
+  timeoutMs: number = 5000
+): Promise<void> {
+  const { pid, pgid, process: proc } = tracked;
+
+  // Already exited? Trust exitCode, not proc.killed — proc.killed only means
+  // Node sent a signal; the process may still be running.
+  if (proc.exitCode !== null) return;
+
+  const exitPromise = new Promise<void>((resolve) => {
+    proc.once('exit', () => resolve());
+  });
+
+  const timeoutPromise = new Promise<void>((resolve) => {
+    setTimeout(resolve, timeoutMs);
+  });
+
+  await Promise.race([exitPromise, timeoutPromise]);
+
+  if (proc.exitCode !== null) return;
+
+  // Timeout: escalate to SIGKILL on the whole process group so any
+  // descendants the SDK spawned are killed too (Principle 5).
+  logger.warn('PROCESS', `PID ${pid} did not exit after ${timeoutMs}ms, sending SIGKILL to process group`, {
+    pid, pgid, timeoutMs,
+  });
+  try {
+    if (typeof pgid === 'number' && process.platform !== 'win32') {
+      process.kill(-pgid, 'SIGKILL');
+    } else {
+      proc.kill('SIGKILL');
+    }
+  } catch {
+    // Already dead — fine.
+  }
+
+  // Wait up to 1s for SIGKILL to take effect (event-driven, not blind sleep).
+  const sigkillExit = new Promise<void>((resolve) => {
+    proc.once('exit', () => resolve());
+  });
+  const sigkillTimeout = new Promise<void>((resolve) => {
+    setTimeout(resolve, 1000);
+  });
+  await Promise.race([sigkillExit, sigkillTimeout]);
+}
+
+// ---------------------------------------------------------------------------
+// Pool slot waiters — backpressure without eviction
+// ---------------------------------------------------------------------------
+//
+// waitForSlot is used by SDKAgent to avoid starting more concurrent SDK
+// subprocesses than configured. It is event-driven: when a process exits and
+// is unregistered, notifySlotAvailable() wakes exactly one waiter. There is
+// no polling. There is no idle-session eviction (Principle 1 — do not kick
+// live sessions to make room; a full pool must apply backpressure upstream).
+
+const TOTAL_PROCESS_HARD_CAP = 10;
+const slotWaiters: Array<() => void> = [];
+
+function getActiveSdkCount(): number {
+  return getProcessRegistry().getAll().filter(record => record.type === 'sdk').length;
+}
+
+function notifySlotAvailable(): void {
+  const waiter = slotWaiters.shift();
+  if (waiter) waiter();
+}
+
+/**
+ * Wait until a pool slot is available to spawn another SDK subprocess.
+ *
+ * Resolves immediately when active SDK process count is below `maxConcurrent`.
+ * Otherwise enqueues a waiter that is woken by a subsequent exit handler.
+ * Rejects with a timeout error if no slot opens within `timeoutMs`.
+ * Rejects immediately if the registry is already at the hard cap.
+ */
+export async function waitForSlot(maxConcurrent: number, timeoutMs: number = 60_000): Promise<void> {
+  const activeCount = getActiveSdkCount();
+  if (activeCount >= TOTAL_PROCESS_HARD_CAP) {
+    throw new Error(`Hard cap exceeded: ${activeCount} processes in registry (cap=${TOTAL_PROCESS_HARD_CAP}). Refusing to spawn more.`);
+  }
+
+  if (activeCount < maxConcurrent) return;
+
+  logger.info('PROCESS', `Pool limit reached (${activeCount}/${maxConcurrent}), waiting for slot...`);
+
+  return new Promise<void>((resolve, reject) => {
+    const timeout = setTimeout(() => {
+      const idx = slotWaiters.indexOf(onSlot);
+      if (idx >= 0) slotWaiters.splice(idx, 1);
+      reject(new Error(`Timed out waiting for agent pool slot after ${timeoutMs}ms`));
+    }, timeoutMs);
+
+    const onSlot = () => {
+      clearTimeout(timeout);
+      if (getActiveSdkCount() < maxConcurrent) {
+        resolve();
+      } else {
+        slotWaiters.push(onSlot);
+      }
+    };
+
+    slotWaiters.push(onSlot);
+  });
+}
+
+// ---------------------------------------------------------------------------
+// SDK subprocess spawn
+// ---------------------------------------------------------------------------
+
+export interface SpawnedSdkProcess {
+  stdin: NonNullable<ChildProcess['stdin']>;
+  stdout: NonNullable<ChildProcess['stdout']>;
+  stderr: NonNullable<ChildProcess['stderr']>;
+  readonly killed: boolean;
+  readonly exitCode: number | null;
+  kill: ChildProcess['kill'];
+  on: ChildProcess['on'];
+  once: ChildProcess['once'];
+  off: ChildProcess['off'];
+}
+
+export interface SpawnSdkOptions {
+  command: string;
+  args: string[];
+  cwd?: string;
+  env?: NodeJS.ProcessEnv;
+  signal?: AbortSignal;
+}
+
+/**
+ * Spawn a Claude SDK subprocess in its own POSIX process group.
+ *
+ * The spawn uses `detached: true` so the child becomes the leader of a new
+ * process group (setpgid). The leader's PID equals its pgid on Unix, so we
+ * store `child.pid` as both pid and pgid on the managed process record.
+ * Shutdown then signals the group via `process.kill(-pgid, signal)`, tearing
+ * down the SDK child AND every descendant in one syscall (Principle 5).
+ *
+ * Windows caveat: `detached: true` does not create a POSIX group. The
+ * recorded pgid is still the child PID so Windows teardown at least kills
+ * the direct child; full subtree teardown on Windows requires Job Objects
+ * or `taskkill /T /F` (see shutdown.ts).
+ *
+ * Node's child_process.spawn is used intentionally — Bun.spawn does NOT
+ * support `detached: true` (see PATHFINDER-2026-04-22/_reference.md Part 2
+ * row 3), and this module must work under Bun as well as Node.
+ */
+export function spawnSdkProcess(
+  sessionDbId: number,
+  options: SpawnSdkOptions
+): { process: SpawnedSdkProcess; pid: number; pgid: number } | null {
+  const registry = getProcessRegistry();
+
+  // On Windows, use cmd.exe wrapper for .cmd files to properly handle paths with spaces.
+  const useCmdWrapper = process.platform === 'win32' && options.command.endsWith('.cmd');
+  const env = sanitizeEnv(options.env ?? process.env);
+
+  // Filter empty string args AND their preceding flag (Issue #2049).
+  // The Agent SDK emits ["--setting-sources", ""] when settingSources defaults to [].
+  // Simply dropping "" leaves an orphan --setting-sources that consumes the next
+  // flag as its value, crashing Claude Code 2.1.109+ with
+  // "Invalid setting source: --permission-mode". Drop the flag too so the SDK
+  // default (no setting sources) is preserved by omission.
+  const filteredArgs: string[] = [];
+  for (const arg of options.args) {
+    if (arg === '') {
+      if (filteredArgs.length > 0 && filteredArgs[filteredArgs.length - 1].startsWith('--')) {
+        filteredArgs.pop();
+      }
+      continue;
+    }
+    filteredArgs.push(arg);
+  }
+
+  // Unix: detached:true causes the kernel to setpgid() on the child so the
+  // child becomes leader of a new process group whose pgid equals its pid.
+  // Windows: detached:true decouples the child from the parent console; there
+  // is no POSIX group, but the flag is still safe to pass.
+  //
+  // stdin must be 'pipe' (not 'ignore') because SpawnedSdkProcess.stdin is
+  // typed NonNullable<...> and the Claude Agent SDK consumes that pipe to
+  // stream prompts in. With 'ignore', child.stdin would be null and the
+  // null-check below (line ~737) would tear the child down immediately.
+  const child = useCmdWrapper
+    ? spawn('cmd.exe', ['/d', '/c', options.command, ...filteredArgs], {
+        cwd: options.cwd,
+        env,
+        detached: true,
+        stdio: ['pipe', 'pipe', 'pipe'],
+        signal: options.signal,
+        windowsHide: true,
+      })
+    : spawn(options.command, filteredArgs, {
+        cwd: options.cwd,
+        env,
+        detached: true,
+        stdio: ['pipe', 'pipe', 'pipe'],
+        signal: options.signal,
+        windowsHide: true,
+      });
+
+  // ALWAYS attach an 'error' listener BEFORE any other code runs, regardless of
+  // whether the child has a PID. child_process.spawn emits 'error' asynchronously
+  // for ENOENT, EACCES, AbortSignal-driven aborts, etc. Without a listener these
+  // become uncaughtException — the cause of "The operation was aborted." escaping
+  // to the daemon during crash-recovery loops.
+  child.on('error', (err: Error) => {
+    logger.warn('SDK_SPAWN', `[session-${sessionDbId}] child emitted error event`, {
+      sessionDbId,
+      pid: child.pid,
+      errorName: err.name,
+      errorCode: (err as NodeJS.ErrnoException).code,
+    }, err);
+  });
+
+  if (!child.pid) {
+    logger.error('PROCESS', 'Spawn succeeded but produced no PID', { sessionDbId });
+    return null;
+  }
+
+  const pid = child.pid;
+  const pgid = pid; // On Unix with detached:true, pgid === pid. On Windows, this is an alias.
+
+  // Capture stderr for debugging spawn failures.
+  if (child.stderr) {
+    child.stderr.on('data', (data: Buffer) => {
+      logger.debug('SDK_SPAWN', `[session-${sessionDbId}] stderr: ${data.toString().trim()}`);
+    });
+  }
+
+  // Register the process in the supervisor registry with pgid recorded so
+  // the shutdown cascade can signal the whole group.
+  const recordId = `sdk:${sessionDbId}:${pid}`;
+  registry.register(recordId, {
+    pid,
+    type: 'sdk',
+    sessionId: sessionDbId,
+    startedAt: new Date().toISOString(),
+    pgid,
+  }, child);
+
+  // Auto-unregister on exit. child.on('exit') is the authoritative event-driven
+  // signal that a process has left — no polling, no sweeper needed (Principle 4).
+  child.on('exit', (code: number | null, signal: string | null) => {
+    if (code !== 0) {
+      logger.warn('SDK_SPAWN', `[session-${sessionDbId}] Claude process exited`, { code, signal, pid });
+    }
+    registry.unregister(recordId);
+    // Wake one pool-slot waiter since a slot just freed up.
+    notifySlotAvailable();
+  });
+
+  if (!child.stdin || !child.stdout || !child.stderr) {
+    logger.error('PROCESS', 'Spawned SDK child missing required stdio streams', {
+      sessionDbId,
+      pid,
+      hasStdin: Boolean(child.stdin),
+      hasStdout: Boolean(child.stdout),
+      hasStderr: Boolean(child.stderr),
+    });
+    try { child.kill('SIGKILL'); } catch { /* already dead */ }
+    return null;
+  }
+
+  const spawned: SpawnedSdkProcess = {
+    stdin: child.stdin,
+    stdout: child.stdout,
+    stderr: child.stderr,
+    get killed() { return child.killed; },
+    get exitCode() { return child.exitCode; },
+    kill: child.kill.bind(child),
+    on: child.on.bind(child),
+    once: child.once.bind(child),
+    off: child.off.bind(child),
+  };
+
+  return { process: spawned, pid, pgid };
+}
+
+/**
+ * SDK-compatible spawn factory.
+ *
+ * The Claude Agent SDK's `spawnClaudeCodeProcess` option calls our factory
+ * with its own spawn arguments; we forward them into `spawnSdkProcess` which
+ * creates the child in its own process group and records it in the supervisor
+ * registry. The returned shape is the minimal subset of ChildProcess that the
+ * SDK consumes — stdin/stdout/stderr pipes, killed/exitCode getters, and
+ * kill/on/once/off.
+ *
+ * Pre-spawn cleanup: if a previous process for this session is still alive
+ * (e.g. a crash-recovery attempt that collided with a still-running SDK),
+ * SIGTERM it. Multiple processes sharing the same --resume UUID waste API
+ * credits and can conflict with each other (Issue #1590).
+ */
+export function createSdkSpawnFactory(sessionDbId: number) {
+  return (spawnOptions: SpawnSdkOptions): SpawnedSdkProcess => {
+    const registry = getProcessRegistry();
+
+    // Kill any existing process for this session before spawning a new one.
+    const existing = registry.getBySession(sessionDbId).filter(r => r.type === 'sdk');
+    for (const record of existing) {
+      if (!isPidAlive(record.pid)) continue;
+      try {
+        if (typeof record.pgid === 'number') {
+          // Signal the whole group — kill the SDK child and any descendants.
+          if (process.platform !== 'win32') {
+            process.kill(-record.pgid, 'SIGTERM');
+          } else {
+            process.kill(record.pid, 'SIGTERM');
+          }
+        } else {
+          process.kill(record.pid, 'SIGTERM');
+        }
+        logger.warn('PROCESS', `Killing duplicate SDK process PID ${record.pid} before spawning new one for session ${sessionDbId}`, {
+          existingPid: record.pid,
+          sessionDbId,
+        });
+      } catch (error: unknown) {
+        const code = error instanceof Error ? (error as NodeJS.ErrnoException).code : undefined;
+        if (code !== 'ESRCH') {
+          if (error instanceof Error) {
+            logger.warn('PROCESS', `Failed to SIGTERM duplicate SDK process PID ${record.pid}`, { sessionDbId }, error);
+          } else {
+            logger.warn('PROCESS', `Failed to SIGTERM duplicate SDK process PID ${record.pid} (non-Error)`, {
+              sessionDbId, error: String(error),
+            });
+          }
+        }
+      }
+    }
+
+    const result = spawnSdkProcess(sessionDbId, spawnOptions);
+    if (!result) {
+      // Match the legacy failure mode: the SDK needs a process-like object
+      // even on spawn failure; throwing here surfaces via exit code 2 to the
+      // hook layer (Principle 2 — fail-fast).
+      throw new Error(`Failed to spawn SDK subprocess for session ${sessionDbId}`);
+    }
+
+    return result.process;
+  };
+}
@@ -34,16 +34,18 @@ export async function runShutdownCascade(options: ShutdownCascadeOptions): Promi
    }

    try {
-      await signalProcess(record.pid, 'SIGTERM');
+      await signalProcess(record, 'SIGTERM');
    } catch (error: unknown) {
      if (error instanceof Error) {
        logger.debug('SYSTEM', 'Failed to send SIGTERM to child process', {
          pid: record.pid,
+          pgid: record.pgid,
          type: record.type
        }, error);
      } else {
        logger.warn('SYSTEM', 'Failed to send SIGTERM to child process (non-Error)', {
          pid: record.pid,
+          pgid: record.pgid,
          type: record.type,
          error: String(error)
        });
@@ -56,16 +58,18 @@ export async function runShutdownCascade(options: ShutdownCascadeOptions): Promi
  const survivors = childRecords.filter(record => isPidAlive(record.pid));
  for (const record of survivors) {
    try {
-      await signalProcess(record.pid, 'SIGKILL');
+      await signalProcess(record, 'SIGKILL');
    } catch (error: unknown) {
      if (error instanceof Error) {
        logger.debug('SYSTEM', 'Failed to force kill child process', {
          pid: record.pid,
+          pgid: record.pgid,
          type: record.type
        }, error);
      } else {
        logger.warn('SYSTEM', 'Failed to force kill child process (non-Error)', {
          pid: record.pid,
+          pgid: record.pgid,
          type: record.type,
          error: String(error)
        });
@@ -110,7 +114,38 @@ async function waitForExit(records: ManagedProcessRecord[], timeoutMs: number):
  }
 }

-async function signalProcess(pid: number, signal: 'SIGTERM' | 'SIGKILL'): Promise<void> {
+async function signalProcess(record: ManagedProcessRecord, signal: 'SIGTERM' | 'SIGKILL'): Promise<void> {
+  const { pid, pgid } = record;
+
+  // Unix path: when the record carries a pgid (set when the child was spawned
+  // with detached:true so it became its own group leader), signal the negative
+  // PID to tear down the whole process group in one syscall — the SDK child
+  // AND every descendant it spawned. This replaces hand-rolled orphan sweeps
+  // (Principle 5: OS-supervised process groups over hand-rolled reapers).
+  //
+  // Falls back to single-PID kill when pgid is absent (the worker itself,
+  // MCP stdio clients, anything not spawned with detached:true).
+  if (process.platform !== 'win32') {
+    try {
+      if (typeof pgid === 'number') {
+        process.kill(-pgid, signal);
+      } else {
+        process.kill(pid, signal);
+      }
+    } catch (error: unknown) {
+      if (error instanceof Error) {
+        const errno = (error as NodeJS.ErrnoException).code;
+        if (errno === 'ESRCH') {
+          return;
+        }
+      }
+      throw error;
+    }
+    return;
+  }
+
+  // Windows: no POSIX process groups. SIGTERM uses single-PID kill; SIGKILL
+  // uses tree-kill or taskkill /T to walk the descendant tree.
  if (signal === 'SIGTERM') {
    try {
      process.kill(pid, signal);
@@ -126,50 +161,35 @@ async function signalProcess(pid: number, signal: 'SIGTERM' | 'SIGKILL'): Promis
    return;
  }

-  if (process.platform === 'win32') {
-    const treeKill = await loadTreeKill();
-    if (treeKill) {
-      await new Promise<void>((resolve, reject) => {
-        treeKill(pid, signal, (error) => {
-          if (!error) {
-            resolve();
-            return;
-          }
+  const treeKill = await loadTreeKill();
+  if (treeKill) {
+    await new Promise<void>((resolve, reject) => {
+      treeKill(pid, signal, (error) => {
+        if (!error) {
+          resolve();
+          return;
+        }

-          const errno = (error as NodeJS.ErrnoException).code;
-          if (errno === 'ESRCH') {
-            resolve();
-            return;
-          }
-          reject(error);
-        });
+        const errno = (error as NodeJS.ErrnoException).code;
+        if (errno === 'ESRCH') {
+          resolve();
+          return;
+        }
+        reject(error);
      });
-      return;
-    }
-
-    const args = ['/PID', String(pid), '/T'];
-    if (signal === 'SIGKILL') {
-      args.push('/F');
-    }
-
-    await execFileAsync('taskkill', args, {
-      timeout: HOOK_TIMEOUTS.POWERSHELL_COMMAND,
-      windowsHide: true
    });
    return;
  }

-  try {
-    process.kill(pid, signal);
-  } catch (error: unknown) {
-    if (error instanceof Error) {
-      const errno = (error as NodeJS.ErrnoException).code;
-      if (errno === 'ESRCH') {
-        return;
-      }
-    }
-    throw error;
+  const args = ['/PID', String(pid), '/T'];
+  if (signal === 'SIGKILL') {
+    args.push('/F');
  }
+
+  await execFileAsync('taskkill', args, {
+    timeout: HOOK_TIMEOUTS.POWERSHELL_COMMAND,
+    windowsHide: true
+  });
 }

 async function loadTreeKill(): Promise<TreeKillFn | null> {