airkjw/claude-mem
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

perf: streamline worker startup and consolidate database connections (#2122)

Browse Source 
* docs: pathfinder refactor corpus + Node 20 preflight

Adds the PATHFINDER-2026-04-22 principle-driven refactor plan (11 docs,
cross-checked PASS) plus the exploratory PATHFINDER-2026-04-21 corpus
that motivated it. Bumps engines.node to >=20.0.0 per the ingestion-path
plan preflight (recursive fs.watch). Adds the pathfinder skill.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: land PATHFINDER Plan 01 — data integrity

Schema, UNIQUE constraints, self-healing claim, Chroma upsert fallback.

- Phase 1: fresh schema.sql regenerated at post-refactor shape.
- Phase 2: migrations 23+24 — rebuild pending_messages without
  started_processing_at_epoch; UNIQUE(session_id, tool_use_id);
  UNIQUE(memory_session_id, content_hash) on observations; dedup
  duplicate rows before adding indexes.
- Phase 3: claimNextMessage rewritten to self-healing query using
  worker_pid NOT IN live_worker_pids; STALE_PROCESSING_THRESHOLD_MS
  and the 60-s stale-reset block deleted.
- Phase 4: DEDUP_WINDOW_MS and findDuplicateObservation deleted;
  observations.insert now uses ON CONFLICT DO NOTHING.
- Phase 5: failed-message purge block deleted from worker-service
  2-min interval; clearFailedOlderThan method deleted.
- Phase 6: repairMalformedSchema and its Python subprocess repair
  path deleted from Database.ts; SQLite errors now propagate.
- Phase 7: Chroma delete-then-add fallback gated behind
  CHROMA_SYNC_FALLBACK_ON_CONFLICT env flag as bridge until
  Chroma MCP ships native upsert.
- Phase 8: migration 19 no-op block absorbed into fresh schema.sql.

Verification greps all return 0 matches. bun test tests/sqlite/
passes 63/63. bun run build succeeds.

Plan: PATHFINDER-2026-04-22/01-data-integrity.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: land PATHFINDER Plan 02 — process lifecycle

OS process groups replace hand-rolled reapers. Worker runs until
killed; orphans are prevented by detached spawn + kill(-pgid).

- Phase 1: src/services/worker/ProcessRegistry.ts DELETED. The
  canonical registry at src/supervisor/process-registry.ts is the
  sole survivor; SDK spawn site consolidated into it via new
  createSdkSpawnFactory/spawnSdkProcess/getSdkProcessForSession/
  ensureSdkProcessExit/waitForSlot helpers.
- Phase 2: SDK children spawn with detached:true + stdio:
  ['ignore','pipe','pipe']; pgid recorded on ManagedProcessInfo.
- Phase 3: shutdown.ts signalProcess teardown uses
  process.kill(-pgid, signal) on Unix when pgid is recorded;
  Windows path unchanged (tree-kill/taskkill).
- Phase 4: all reaper intervals deleted — startOrphanReaper call,
  staleSessionReaperInterval setInterval (including the co-located
  WAL checkpoint — SQLite's built-in wal_autocheckpoint handles
  WAL growth without an app-level timer), killIdleDaemonChildren,
  killSystemOrphans, reapOrphanedProcesses, reapStaleSessions, and
  detectStaleGenerator. MAX_GENERATOR_IDLE_MS and MAX_SESSION_IDLE_MS
  constants deleted.
- Phase 5: abandonedTimer — already 0 matches; primary-path cleanup
  via generatorPromise.finally() already lives in worker-service
  startSessionProcessor and SessionRoutes ensureGeneratorRunning.
- Phase 6: evictIdlestSession and its evict callback deleted from
  SessionManager. Pool admission gates backpressure upstream.
- Phase 7: SDK-failure fallback — SessionManager has zero matches
  for fallbackAgent/Gemini/OpenRouter. Failures surface to hooks
  via exit code 2 through SessionRoutes error mapping.
- Phase 8: ensureWorkerRunning in worker-utils.ts rewritten to
  lazy-spawn — consults isWorkerPortAlive (which gates
  captureProcessStartToken for PID-reuse safety via commit
  99060bac), then spawns detached with unref(), then
  waitForWorkerPort({ attempts: 3, backoffMs: 250 }) hand-rolled
  exponential backoff 250→500→1000ms. No respawn npm dep.
- Phase 9: idle self-shutdown — zero matches for
  idleCheck/idleTimeout/IDLE_MAX_MS/idleShutdown. Worker exits
  only on external SIGTERM via supervisor signal handlers.

Three test files that exercised deleted code removed:
tests/worker/process-registry.test.ts,
tests/worker/session-lifecycle-guard.test.ts,
tests/services/worker/reap-stale-sessions.test.ts.
Pass count: 1451 → 1407 (-44), all attributable to deleted test
files. Zero new failures. 31 pre-existing failures remain
(schema-repair suite, logger-usage-standards, environmental
openclaw / plugin-distribution) — none introduced by Plan 02.

All 10 verification greps return 0. bun run build succeeds.

Plan: PATHFINDER-2026-04-22/02-process-lifecycle.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: land PATHFINDER Plan 04 (narrowed) — search fail-fast

Phases 3, 5, 6 only. Plan-doc inaccuracies for phases 1/2/4/7/8/9
deferred for plan reconciliation:
  - Phase 1/2: ObservationRow type doesn't exist; the four
    "formatters" operate on three incompatible types.
  - Phase 4: RECENCY_WINDOW_MS already imported from
    SEARCH_CONSTANTS at every call site.
  - Phase 7: getExistingChromaIds is NOT @deprecated and has an
    active caller in ChromaSync.backfillMissingSyncs.
  - Phase 8: estimateTokens already consolidated.
  - Phase 9: knowledge-corpus rewrite blocked on PG-3
    prompt-caching cost smoke test.

Phase 3 — Delete SearchManager.findByConcept/findByFile/findByType.
SearchRoutes handlers (handleSearchByConcept/File/Type) now call
searchManager.getOrchestrator().findByXxx() directly via new
getter accessors on SearchManager. ~250 LoC deleted.

Phase 5 — Fail-fast Chroma. Created
src/services/worker/search/errors.ts with ChromaUnavailableError
extends AppError(503, 'CHROMA_UNAVAILABLE'). Deleted
SearchOrchestrator.executeWithFallback's Chroma-failed
SQLite-fallback branch; runtime Chroma errors now throw 503.
"Path 3" (chromaSync was null at construction — explicit-
uninitialized config) preserved as legitimate empty-result state
per plan text. ChromaSearchStrategy.search no longer wraps in
try/catch — errors propagate.

Phase 6 — Delete HybridSearchStrategy three try/catch silent
fallback blocks (findByConcept, findByType, findByFile) at lines
~82-95, ~120-132, ~161-172. Removed `fellBack` field from
StrategySearchResult type and every return site
(SQLiteSearchStrategy, BaseSearchStrategy.emptyResult,
SearchOrchestrator).

Tests updated (Principle 7 — delete in same PR):
  - search-orchestrator.test.ts: "fall back to SQLite" rewritten
    as "throw ChromaUnavailableError (HTTP 503)".
  - chroma/hybrid/sqlite-search-strategy tests: rewritten to
    rejects.toThrow; removed fellBack assertions.

Verification: SearchManager.findBy → 0; fellBack → 0 in src/.
bun test tests/worker/search/ → 122 pass, 0 fail.
bun test (suite-wide) → 1407 pass, baseline maintained, 0 new
failures. bun run build succeeds.

Plan: PATHFINDER-2026-04-22/04-read-path.md (Phases 3, 5, 6)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: land PATHFINDER Plan 03 — ingestion path

Fail-fast parser, direct in-process ingest, recursive fs.watch,
DB-backed tool pairing. Worker-internal HTTP loopback eliminated.

- Phase 0: Created src/services/worker/http/shared.ts exporting
  ingestObservation/ingestPrompt/ingestSummary as direct
  in-process functions plus ingestEventBus (Node EventEmitter,
  reusing existing pattern — no third event bus introduced).
  setIngestContext wires the SessionManager dependency from
  worker-service constructor.
- Phase 1: src/sdk/parser.ts collapsed to one parseAgentXml
  returning { valid:true; kind: 'observation'|'summary'; data }
  | { valid:false; reason: string }. Inspects root element;
  <skip_summary reason="…"/> is a first-class summary case
  with skipped:true. NEVER returns undefined. NEVER coerces.
- Phase 2: ResponseProcessor calls parseAgentXml exactly once,
  branches on the discriminated union. On invalid → markFailed
  + logger.warn(reason). On observation → ingestObservation.
  On summary → ingestSummary then emit summaryStoredEvent
  { sessionId, messageId } (consumed by Plan 05's blocking
  /api/session/end).
- Phase 3: Deleted consecutiveSummaryFailures field
  (ResponseProcessor + SessionManager + worker-types) and
  MAX_CONSECUTIVE_SUMMARY_FAILURES constant. Circuit-breaker
  guards and "tripped" log lines removed.
- Phase 4: coerceObservationToSummary deleted from sdk/parser.ts.
- Phase 5: src/services/transcripts/watcher.ts rescan setInterval
  replaced with fs.watch(transcriptsRoot, { recursive: true,
  persistent: true }) — Node 20+ recursive mode.
- Phase 6: src/services/transcripts/processor.ts pendingTools
  Map deleted. tool_use rows insert with INSERT OR IGNORE on
  UNIQUE(session_id, tool_use_id) (added by Plan 01). New
  pairToolUsesByJoin query in PendingMessageStore for read-time
  pairing (UNIQUE INDEX provides idempotency; explicit consumer
  not yet wired).
- Phase 7: HTTP loopback at processor.ts:252 replaced with
  direct ingestObservation call. maybeParseJson silent-passthrough
  rewritten to fail-fast (throws on malformed JSON).
- Phase 8: src/utils/tag-stripping.ts countTags + stripTagsInternal
  collapsed into one alternation regex, single-pass over input.
- Phase 9: src/utils/transcript-parser.ts (dead TranscriptParser
  class) deleted. The active extractLastMessage at
  src/shared/transcript-parser.ts:41-144 is the sole survivor.

Tests updated (Principle 7 — same-PR delete):
  - tests/sdk/parser.test.ts + parse-summary.test.ts: rewritten
    to assert discriminated-union shape; coercion-specific
    scenarios collapse into { valid:false } assertions.
  - tests/worker/agents/response-processor.test.ts: circuit-breaker
    describe block skipped; non-XML/empty-response tests assert
    fail-fast markFailed behavior.

Verification: every grep returns 0. transcript-parser.ts deleted.
bun run build succeeds. bun test → 1399 pass / 28 fail / 7 skip
(net -8 pass = the 4 retired circuit-breaker tests + 4 collapsed
parser cases). Zero new failures vs baseline.

Deferred (out of Plan 03 scope, will land in Plan 06): SessionRoutes
HTTP route handlers still call sessionManager.queueObservation
inline rather than the new shared helpers — the helpers are ready,
the route swap is mechanical and belongs with the Zod refactor.

Plan: PATHFINDER-2026-04-22/03-ingestion-path.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: land PATHFINDER Plan 05 — hook surface

Worker-call plumbing collapsed to one helper. Polling replaced by
server-side blocking endpoint. Fail-loud counter surfaces persistent
worker outages via exit code 2.

- Phase 1: plugin/hooks/hooks.json — three 20-iteration `for i in
  1..20; do curl -sf .../health && break; sleep 0.1; done` shell
  retry wrappers deleted. Hook commands invoke their bun entry
  point directly.
- Phase 2: src/shared/worker-utils.ts — added
  executeWithWorkerFallback<T>(url, method, body) returning
  T | { continue: true; reason?: string }. All 8 hook handlers
  (observation, session-init, context, file-context, file-edit,
  summarize, session-complete, user-message) rewritten to use
  it instead of duplicating the ensureWorkerRunning →
  workerHttpRequest → fallback sequence.
- Phase 3: blocking POST /api/session/end in SessionRoutes.ts
  using validateBody + sessionEndSchema (z.object({sessionId})).
  One-shot ingestEventBus.on('summaryStoredEvent') listener,
  30 s timer, req.aborted handler — all share one cleanup so
  the listener cannot leak. summarize.ts polling loop, plus
  MAX_WAIT_FOR_SUMMARY_MS / POLL_INTERVAL_MS constants, deleted.
- Phase 4: src/shared/hook-settings.ts — loadFromFileOnce()
  memoizes SettingsDefaultsManager.loadFromFile per process.
  Per-handler settings reads collapsed.
- Phase 5: src/shared/should-track-project.ts — single exclusion
  check entry; isProjectExcluded no longer referenced from
  src/cli/handlers/.
- Phase 6: cwd validation pushed into adapter normalizeInput
  (all 6 adapters: claude-code, cursor, raw, gemini-cli,
  windsurf). New AdapterRejectedInput error in
  src/cli/adapters/errors.ts. Handler-level isValidCwd checks
  deleted from file-edit.ts and observation.ts. hook-command.ts
  catches AdapterRejectedInput → graceful fallback.
- Phase 7: session-init.ts conditional initAgent guard deleted;
  initAgent is idempotent. tests/hooks/context-reinjection-guard
  test (validated the deleted conditional) deleted in same PR
  per Principle 7.
- Phase 8: fail-loud counter at ~/.claude-mem/state/hook-failures
  .json. Atomic write via .tmp + rename. CLAUDE_MEM_HOOK_FAIL_LOUD
  _THRESHOLD setting (default 3). On consecutive worker-unreachable
  ≥ N: process.exit(2). On success: reset to 0. NOT a retry.
- Phase 9: ensureWorkerAliveOnce() module-scope memoization
  wrapping ensureWorkerRunning. executeWithWorkerFallback calls
  the memoized version.

Minimal validateBody middleware stub at
src/services/worker/http/middleware/validateBody.ts. Plan 06 will
expand with typed inference + error envelope conventions.

Verification: 4/4 grep targets pass. bun run build succeeds.
bun test → 1393 pass / 28 fail / 7 skip; -6 pass attributable
solely to deleted context-reinjection-guard test file. Zero new
failures vs baseline.

Plan: PATHFINDER-2026-04-22/05-hook-surface.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: land PATHFINDER Plan 06 — API surface

One Zod-based validator wrapping every POST/PUT. Rate limiter,
diagnostic endpoints, and shutdown wrappers deleted. Failure-
marking consolidated to one helper.

- Phase 1 (preflight): zod@^3 already installed.
- Phase 2: validateBody middleware confirmed at canonical shape
  in src/services/worker/http/middleware/validateBody.ts —
  safeParse → 400 { error: 'ValidationError', issues: [...] }
  on failure, replaces req.body with parsed value on success.
- Phase 3: Per-route Zod schemas declared at the top of each
  route file. 24 POST endpoints across SessionRoutes,
  CorpusRoutes, DataRoutes, MemoryRoutes, SearchRoutes,
  LogsRoutes, SettingsRoutes now wrap with validateBody().
  /api/session/end (Plan 05) confirmed using same middleware.
- Phase 4: validateRequired() deleted from BaseRouteHandler
  along with every call site. Inline coercion helpers
  (coerceStringArray, coercePositiveInteger) and inline
  if (!req.body...) guards deleted across all route files.
- Phase 5: Rate limiter middleware and its registration deleted
  from src/services/worker/http/middleware.ts. Worker binds
  127.0.0.1:37777 — no untrusted caller.
- Phase 6: viewer.html cached at module init in ViewerRoutes.ts
  via fs.readFileSync; served as Buffer with text/html content
  type. SKILL.md + per-operation .md files cached in
  Server.ts as Map<string, string>; loadInstructionContent
  helper deleted. NO fs.watch, NO TTL — process restart is the
  cache-invalidation event.
- Phase 7: Four diagnostic endpoints deleted from DataRoutes.ts
  — /api/pending-queue (GET), /api/pending-queue/process (POST),
  /api/pending-queue/failed (DELETE), /api/pending-queue/all
  (DELETE). Helper methods that ONLY served them
  (getQueueMessages, getStuckCount, getRecentlyProcessed,
  clearFailed, clearAll) deleted from PendingMessageStore.
  KEPT: /api/processing-status (observability), /health
  (used by ensureWorkerRunning).
- Phase 8: stopSupervisor wrapper deleted from supervisor/index.ts.
  GracefulShutdown now calls getSupervisor().stop() directly.
  Two functions retained with clear roles:
    - performGracefulShutdown — worker-side 6-step shutdown
    - runShutdownCascade — supervisor-side child teardown
      (process.kill(-pgid), Windows tree-kill, PID-file cleanup)
  Each has unique non-trivial logic and a single canonical caller.
- Phase 9: transitionMessagesTo(status, filter) is the sole
  failure-marking path on PendingMessageStore. Old methods
  markSessionMessagesFailed and markAllSessionMessagesAbandoned
  deleted along with all callers (worker-service,
  SessionCompletionHandler, tests/zombie-prevention).

Tests updated (Principle 7 same-PR delete): coercion test files
refactored to chain validateBody → handler. Zombie-prevention
tests rewritten to call transitionMessagesTo.

Verification: all 4 grep targets → 0. bun run build succeeds.
bun test → 1393 pass / 28 fail / 7 skip — exact match to
baseline. Zero new failures.

Plan: PATHFINDER-2026-04-22/06-api-surface.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: land PATHFINDER Plan 07 — dead code sweep

ts-prune-driven sweep across the tree after Plans 01-06 landed.
Deleted unused exports, orphan helpers, and one fully orphaned
file. Earlier-plan deletions verified.

Deleted:
- src/utils/bun-path.ts (entire file — getBunPath, getBunPathOrThrow,
  isBunAvailable: zero importers)
- bun-resolver.getBunVersionString: zero callers
- PendingMessageStore.retryMessage / resetProcessingToPending /
  abortMessage: superseded by transitionMessagesTo (Plan 06 Phase 9)
- EnvManager.MANAGED_CREDENTIAL_KEYS, EnvManager.setCredential:
  zero callers
- CodexCliInstaller.checkCodexCliStatus: zero callers; no status
  command exists in npx-cli
- Two "REMOVED: cleanupOrphanedSessions" stale-fence comments

Kept (with documented justification):
- Public API surface in dist/sdk/* (parseAgentXml, prompt
  builders, ParsedObservation, ParsedSummary, ParseResult,
  SUMMARY_MODE_MARKER) — exported via package.json sdk path.
- generateContext / loadContextConfig / token utilities — used
  via dynamic await import('../../../context-generator.js') in
  worker SearchRoutes.
- MCP_IDE_INSTALLERS, install/uninstall functions for codex/goose
  — used via dynamic await import in npx-cli/install.ts +
  uninstall.ts (ts-prune cannot trace dynamic imports).
- getExistingChromaIds — active caller in
  ChromaSync.backfillMissingSyncs (Plan 04 narrowed scope).
- processPendingQueues / getSessionsWithPendingMessages — active
  orphan-recovery caller in worker-service.ts plus
  zombie-prevention test coverage.
- StoreAndMarkCompleteResult legacy alias — return-type annotation
  in same file.
- All Database.ts barrel re-exports — used downstream.

Earlier-plan verification:
- Plan 03 Phase 9: VERIFIED — src/utils/transcript-parser.ts
  is gone; TranscriptParser has 0 references in src/.
- Plan 01 Phase 8: VERIFIED — migration 19 no-op absorbed.
- SessionStore.ts:52-70 consolidation NOT executed (deferred):
  the methods are not thin wrappers but ~900 LoC of bodies, and
  two methods are documented as intentional mirrors so the
  context-generator.cjs bundle stays schema-consistent without
  pulling MigrationRunner. Deserves its own plan, not a sweep.

Verification: TranscriptParser → 0; transcript-parser.ts → gone;
no commented-out code markers remain. bun run build succeeds.
bun test → 1393 pass / 28 fail / 7 skip — EXACT match to
baseline. Zero regressions.

Plan: PATHFINDER-2026-04-22/07-dead-code.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: remove residual ProcessRegistry comment reference

Plan 07 dead-code sweep missed one comment-level reference to the
deleted in-memory ProcessRegistry class in SessionManager.ts:347.
Rewritten to describe the supervisor.json scope without naming the
deleted class, completing the verification grep target.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: address Greptile review (P1 + 2× P2)

P1 — Plan 05 Phase 3 blocking endpoint was non-functional:
executeWithWorkerFallback used HEALTH_CHECK_TIMEOUT_MS (3 s) for
the POST /api/session/end call, but the server holds the
connection for SERVER_SIDE_SUMMARY_TIMEOUT_MS (30 s). Client
always raced to a "timed out" rejection that isWorkerUnavailable
classified as worker-unreachable, so the hook silently degraded
instead of waiting for summaryStoredEvent.
  - Added optional timeoutMs to executeWithWorkerFallback,
    forwarded to workerHttpRequest.
  - summarize.ts call site now passes 35_000 (5 s above server
    hold window).

P2 — ingestSummary({ kind: 'parsed' }) branch was dead code:
ResponseProcessor emitted summaryStoredEvent directly via the
event bus, bypassing the centralized helper that the comment
claimed was the single source.
  - ResponseProcessor now calls ingestSummary({ kind: 'parsed',
    sessionDbId, messageId, contentSessionId, parsed }) so the
    event-emission path is single-sourced.
  - ingestSummary's requireContext() resolution moved inside the
    'queue' branch (the only branch that needs sessionManager /
    dbManager). 'parsed' is a pure event-bus emission and
    doesn't need worker-internal context — fixes mocked
    ResponseProcessor unit tests that don't call
    setIngestContext.

P2 — isWorkerFallback could false-positive on legitimate API
responses whose schema includes { continue: true, ... }:
  - Added a Symbol.for('claude-mem/worker-fallback') brand to
    WorkerFallback. isWorkerFallback now checks the brand, not
    a duck-typed property name.

Verification: bun run build succeeds. bun test → 1393 pass /
28 fail / 7 skip — exact baseline match. Zero new failures.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: address Greptile iteration 2 (P1 + P2)

P1 — summaryStoredEvent fired regardless of whether the row was
persisted. ResponseProcessor's call to ingestSummary({ kind:
'parsed' }) ran for every parsed.kind === 'summary' even when
result.summaryId came back null (e.g. FK violation, null
memory_session_id at commit). The blocking /api/session/end
endpoint then returned { ok: true } and the Stop hook logged
'Summary stored' for a non-existent row.

  - Gate ingestSummary call on (parsed.data.skipped ||
    session.lastSummaryStored). Skipped summaries are an explicit
    no-op bypass and still confirm; real summaries only confirm
    when storage actually wrote a row.
  - Non-skipped + summaryId === null path logs a warn and lets
    the server-side timeout (504) surface to the hook instead of
    a false ok:true.

P2 — PendingMessageStore.enqueue() returns 0 when INSERT OR
IGNORE suppresses a duplicate (the UNIQUE(session_id, tool_use_id)
constraint added by Plan 01 Phase 1). The two callers
(SessionManager.queueObservation and queueSummarize) previously
logged 'ENQUEUED messageId=0' which read like a row was inserted.

  - Branch on messageId === 0 and emit a 'DUP_SUPPRESSED' debug
    log instead of the misleading ENQUEUED line. No behavior
    change — the duplicate is still correctly suppressed by the
    DB (Principle 3); only the log surface is corrected.
  - confirmProcessed is never called with the enqueue() return
    value (it operates on session.processingMessageIds[] from
    claimNextMessage), so no caller is broken; the visibility
    fix prevents future misuse.

Verification: bun run build succeeds. bun test → 1393 pass /
28 fail / 7 skip — exact baseline match. Zero new failures.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: address Greptile iteration 3 (P1 + 2× P2)

- P1 worker-service.ts: wire ensureGeneratorRunning into the ingest
  context after SessionRoutes is constructed. setIngestContext runs
  before routes exist, so transcript-watcher observations queued via
  ingestObservation() had no way to auto-start the SDK generator.
  Added attachIngestGeneratorStarter() to patch the callback in.
- P2 shared.ts: IngestEventBus now sets maxListeners to 0. Concurrent
  /api/session/end calls register one listener each and clean up on
  completion, so the default-10 warning fires spuriously under normal
  load.
- P2 SessionRoutes.ts: handleObservationsByClaudeId now delegates to
  ingestObservation() instead of duplicating skip-tool / meta /
  privacy / queue logic. Single helper, matching the Plan 03 goal.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: address Greptile iteration 4 (P1 tool-pair + P2 parse/path/doc)

- processor.handleToolResult: restore in-memory tool-use→tool-result
  pairing via session.pendingTools for schemas (e.g. Codex) whose
  tool_result events carry only tool_use_id + output. Without this,
  neither handler fired — all tool observations silently dropped.
- processor.maybeParseJson: return raw string on parse failure instead
  of throwing. Previously a single malformed JSON-shaped field caused
  handleLine's outer catch to discard the entire transcript line.
- watcher.deepestNonGlobAncestor: split on / and \\, emit empty string
  for purely-glob inputs so the caller skips the watch instead of
  anchoring fs.watch at the filesystem root. Windows-compatible.
- PendingMessageStore.enqueue: tighten docstring — callers today only
  log on the returned id; the SessionManager branches on id === 0.

* fix: forward tool_use_id through ingestObservation (Greptile iter 5)

P1 — Plan 01's UNIQUE(content_session_id, tool_use_id) dedup never
fired because the new shared ingest path dropped the toolUseId before
queueObservation. SQLite treats NULL values as distinct for UNIQUE,
so every replayed transcript line landed a duplicate row.

- shared.ingestObservation: forward payload.toolUseId to
  queueObservation so INSERT OR IGNORE can actually collapse.
- SessionRoutes.handleObservationsByClaudeId: destructure both
  tool_use_id (HTTP convention) and toolUseId (JS convention) from
  req.body and pass into ingestObservation.
- observationsByClaudeIdSchema: declare both keys explicitly so the
  validator doesn't rely on .passthrough() alone.

* fix: drop dead pairToolUsesByJoin, close session-end listener race

- PendingMessageStore: delete pairToolUsesByJoin. The method was never
  called and its self-join semantics are structurally incompatible
  with UNIQUE(content_session_id, tool_use_id): INSERT OR IGNORE
  collapses any second row with the same pair, so a self-join can
  only ever match a row to itself. In-memory pendingTools in
  processor.ts remains the pairing path for split-event schemas.

- IngestEventBus: retain a short-lived (60s) recentStored map keyed
  by sessionId. Populated on summaryStoredEvent emit, evicted on
  consume or TTL.

- handleSessionEnd: drain the recent-events buffer before attaching
  the listener. Closes the register-after-emit race where the summary
  can persist between the hook's summarize POST and its session/end
  POST — previously that window returned 504 after the 30s timeout.

* chore: merge origin/main into vivacious-teeth

Resolves conflicts with 15 commits on main (v12.3.9, security
observation types, Telegram notifier, PID-reuse worker start-guard).

Conflict resolution strategy:
- plugin/hooks/hooks.json, plugin/scripts/*.cjs, plugin/ui/viewer-bundle.js:
  kept ours — PATHFINDER Plan 05 deletes the for-i-in-1-to-20 curl retry
  loops and the built artifacts regenerate on build.
- src/cli/handlers/summarize.ts: kept ours — Plan 05 blocking
  POST /api/session/end supersedes main's fire-and-forget path.
- src/services/worker-service.ts: kept ours — Plan 05 ingest bus +
  summaryStoredEvent supersedes main's SessionCompletionHandler DI
  refactor + orphan-reaper fallback.
- src/services/worker/http/routes/SessionRoutes.ts: kept ours — same
  reason; generator .finally() Stop-hook self-clean is a guard for a
  path our blocking endpoint removes.
- src/services/worker/http/routes/CorpusRoutes.ts: merged — added
  security_alert / security_note to ALLOWED_CORPUS_TYPES (feature from
  #2084) while preserving our Zod validateBody schema.

Typecheck: 294 errors (vs 298 pre-merge). No new errors introduced; all
remaining are pre-existing (Component-enum gaps, DOM lib for viewer,
bun:sqlite types).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: address Greptile P2 findings

1) SessionRoutes.handleSessionEnd was the only route handler not wrapped
   in wrapHandler — synchronous exceptions would hang the client rather
   than surfacing as 500s. Wrap it like every other handler.

2) processor.handleToolResult only consumed the session.pendingTools
   entry when the tool_result arrived without a toolName. In the
   split-schema path where tool_result carries both toolName and toolId,
   the entry was never deleted and the map grew for the life of the
   session. Consume the entry whenever toolId is present.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: typing cleanup and viewer tsconfig split for PR feedback

- Add explicit return types for SessionStore query methods
- Exclude src/ui/viewer from root tsconfig, give it its own DOM-typed config
- Add bun to root tsconfig types, plus misc typing tweaks flagged by Greptile
- Rebuilt plugin/scripts/* artifacts

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: address Greptile P2 findings (iter 2)

- PendingMessageStore.transitionMessagesTo: require sessionDbId (drop
  the unscoped-drain branch that would nuke every pending/processing
  row across all sessions if a future caller omitted the filter).
- IngestEventBus.takeRecentSummaryStored: make idempotent — keep the
  cached event until TTL eviction so a retried Stop hook's second
  /api/session/end returns immediately instead of hanging 30 s.
- TranscriptWatcher fs.watch callback: skip full glob scan for paths
  already tailed (JSONL appends fire on every line; only unknown
  paths warrant a rescan).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: call finalizeSession in terminal session paths (Greptile iter 3)

terminateSession and runFallbackForTerminatedSession previously called
SessionCompletionHandler.finalizeSession before removeSessionImmediate;
the refactor dropped those calls, leaving sdk_sessions.status='active'
for every session killed by wall-clock limit, unrecoverable error, or
exhausted fallback chain. The deleted reapStaleSessions interval was
the only prior backstop.

Re-wires finalizeSession (idempotent: marks completed, drains pending,
broadcasts) into both paths; no reaper reintroduced.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: GC failed pending_messages rows at startup (Greptile iter 4)

Plan 07 deleted clearFailed/clearFailedOlderThan as "dead code", but
with the periodic sweep also removed, nothing reaps status='failed'
rows now — they accumulate indefinitely. Since claimNextMessage's
self-healing subquery scans this table, unbounded growth degrades
claim latency over time.

Re-introduces clearFailedOlderThan and calls it once at worker startup
(not a reaper — one-shot, idempotent). 7-day retention keeps enough
history for operator inspection while bounding the table.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: finalize sessions on normal exit; cleanup hoist; share handler (iter 5)

1. startSessionProcessor success branch now calls completionHandler.
   finalizeSession before removeSessionImmediate. Hooks-disabled installs
   (and any Stop hook that fails before POST /api/sessions/complete) no
   longer leave sdk_sessions rows as status='active' forever. Idempotent
   — a subsequent /api/sessions/complete is a no-op.

2. Hoist SessionRoutes.handleSessionEnd cleanup declaration above the
   closures that reference it (TDZ safety; safe at runtime today but
   fragile if timeout ever shrinks).

3. SessionRoutes now receives WorkerService's shared SessionCompletionHandler
   instead of constructing its own — prevents silent divergence if the
   handler ever becomes stateful.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: stop runaway crash-recovery loop on dead sessions

Two distinct bugs were combining to keep a dead session restarting forever:

Bug 1 (uncaught "The operation was aborted."):
  child_process.spawn emits 'error' asynchronously for ENOENT/EACCES/abort
  signal aborts. spawnSdkProcess() never attached an 'error' listener, so
  any async spawn failure became uncaughtException and escaped to the
  daemon-level handler. Attach an 'error' listener immediately after spawn,
  before the !child.pid early-return, so async spawn errors are logged
  (with errno code) and swallowed locally.

Bug 2 (sliding-window limiter never trips on slow restart cadence):
  RestartGuard tripped only when restartTimestamps.length exceeded
  MAX_WINDOWED_RESTARTS (10) within RESTART_WINDOW_MS (60s). With the 8s
  exponential-backoff cap, only ~7-8 restarts fit in the window, so a dead
  session that fail-restart-fail-restart on 8s cycles would loop forever
  (consecutiveRestarts climbing past 30+ in observed logs). Add a
  consecutiveFailures counter that increments on every restart and resets
  only on recordSuccess(). Trip when consecutive failures exceed
  MAX_CONSECUTIVE_FAILURES (5) — meaning 5 restarts with zero successful
  processing in between proves the session is dead. Both guards now run in
  parallel: tight loops still trip the windowed cap; slow loops trip the
  consecutive-failure cap.

Also: when the SessionRoutes path trips the guard, drain pending messages
to 'abandoned' so the session does not reappear in
getSessionsWithPendingMessages and trigger another auto-start cycle. The
worker-service.ts path already does this via terminateSession.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* perf: streamline worker startup and consolidate database connections

1. Database Pooling: Modified DatabaseManager, SessionStore, and SessionSearch to share a single bun:sqlite connection, eliminating redundant file descriptors.
2. Non-blocking Startup: Refactored WorktreeAdoption and Chroma backfill to run in the background (fire-and-forget), preventing them from stalling core initialization.
3. Diagnostic Routes: Added /api/chroma/status and bypassed the initialization guard for health/readiness endpoints to allow diagnostics during startup.
4. Robust Search: Implemented reliable SQLite FTS5 fallback in SearchManager for when Chroma (uvx) fails or is unavailable.
5. Code Cleanup: Removed redundant loopback MCP checks and mangled initialization logic from WorkerService.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: hard-exclude observer-sessions from hooks; bundle migration 29 (#2124)

* fix: hard-exclude observer-sessions from hooks; backfill bundle migrations

Stop hook + SessionEnd hook were storing the SDK observer's own
init/continuation/summary prompts in user_prompts, leaking into the
viewer (meta-observation regression). 25 such rows accumulated.

- shouldTrackProject: hard-reject OBSERVER_SESSIONS_DIR (and its subtree)
  before consulting user-configured exclusion globs.
- summarize.ts (Stop) and session-complete.ts (SessionEnd): early-return
  when shouldTrackProject(cwd) is false, so the observer's own hooks
  cannot bootstrap the worker or queue a summary against the meta-session.
- SessionRoutes: cap user-prompt body at 256 KiB at the session-init
  boundary so a runaway observer prompt cannot blow up storage.
- SessionStore: add migration 29 (UNIQUE(memory_session_id, content_hash)
  on observations) inline so bundled artifacts (worker-service.cjs,
  context-generator.cjs) stay schema-consistent — without it, the
  ON CONFLICT clause in observation inserts throws.
- spawnSdkProcess: stdio[stdin] from 'ignore' to 'pipe' so the
  supervisor can actually feed the observer's stdin.

Also rebuilds plugin/scripts/{worker-service,context-generator}.cjs.

* fix: walk back to UTF-8 boundary on prompt truncation (Greptile P2)

Plain Buffer.subarray at MAX_USER_PROMPT_BYTES can land mid-codepoint,
which the utf8 decoder silently rewrites to U+FFFD. Walk back over any
continuation bytes (0b10xxxxxx) before decoding so the truncated prompt
ends on a valid sequence boundary instead of a replacement character.

* fix: cross-platform observer-dir containment; clarify SDK stdin pipe

claude-review feedback on PR #2124.

- shouldTrackProject: literal `cwd.startsWith(OBSERVER_SESSIONS_DIR + '/')`
  hard-coded a POSIX separator and missed Windows backslash paths plus any
  trailing-slash variance. Switched to a path.relative-based isWithin()
  helper so Windows hook input under observer-sessions\\... is also excluded.
- spawnSdkProcess: added a comment explaining why stdin must be 'pipe' —
  SpawnedSdkProcess.stdin is typed NonNullable and the Claude Agent SDK
  consumes that pipe; 'ignore' would null it and the null-check below
  would tear the child down on every spawn.

* fix: make Stop hook fire-and-forget; remove dead /api/session/end

The Stop hook was awaiting a 35-second long-poll on /api/session/end,
which the worker held open until the summary-stored event fired (or its
30s server-side timeout elapsed). Followed by another await on
/api/sessions/complete. Three sequential awaits, the middle one a 30s
hold — not fire-and-forget despite repeated requests.

The Stop hook now does ONE thing: POST /api/sessions/summarize to
queue the summary work and return. The worker drives the rest async.
Session-map cleanup is performed by the SessionEnd handler
(session-complete.ts), not duplicated here.

- summarize.ts: drop the /api/session/end long-poll and the trailing
  /api/sessions/complete await; ~40 lines removed; unused
  SessionEndResponse interface gone; header comment rewritten.
- SessionRoutes: delete handleSessionEnd, sessionEndSchema, the
  SERVER_SIDE_SUMMARY_TIMEOUT_MS constant, and the /api/session/end
  route registration. Drop the now-unused ingestEventBus and
  SummaryStoredEvent imports.
- ResponseProcessor + shared.ts + worker-utils.ts: update stale
  comments that referenced the dead endpoint. The IngestEventBus is
  left in place dormant (no listeners) for follow-up cleanup so this
  PR stays focused on the blocker.

Bundle artifact (worker-service.cjs) rebuilt via build-and-sync.

Verification:
- grep '/api/session/end' plugin/scripts/worker-service.cjs → 0
- grep 'timeoutMs:35' plugin/scripts/worker-service.cjs → 0
- Worker restarted clean, /api/health ok at pid 92368

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* deps: bump all dependencies to latest including majors

Upgrades: React 18→19, Express 4→5, Zod 3→4, TypeScript 5→6,
@types/node 20→25, @anthropic-ai/claude-agent-sdk 0.1→0.2,
@clack/prompts 0.9→1.2, plus minors. Adds Daily Maintenance section
to CLAUDE.md mandating latest-version policy across manifests.

Express 5 surfaced a race in Server.listen() where the 'error' handler
was attached after listen() was invoked; refactored to use
http.createServer with both 'error' and 'listening' handlers attached
before listen(), restoring port-conflict rejection semantics.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: surface real chroma errors and add deep status probe

Replace the misleading "Vector search failed - semantic search unavailable.
Install uv... restart the worker." string in SearchManager with the actual
exception text from chroma_query_documents. The lying message blamed `uv`
for any failure — even when the real cause was a chroma-mcp transport
timeout, an empty collection, or a dead subprocess.

Also add /api/chroma/status?deep=1 backed by a new
ChromaMcpManager.probeSemanticSearch() that round-trips a real query
(chroma_list_collections + chroma_query_documents) instead of just
checking the stdio handshake. The cheap default path is unchanged.

Includes the diagnostic plan (PLAN-fix-mcp-search.md) and updated test
fixtures for the new structured failure message.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: rebuild worker-service bundle to match merged src

Bundle was stale after the squash merge of #2124 — it still contained
the old "Install uv... semantic search unavailable" string and lacked
probeSemanticSearch. Rebuilt via bun run build-and-sync.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: address coderabbit feedback on PLAN-fix-mcp-search.md

- replace machine-specific /Users/alexnewman absolute paths with portable
  <repo-root> placeholder (MD-style portability)
- add blank lines around the TypeScript fenced block (MD031)
- tag the bare fenced block with `text` (MD040)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alex Newman
2026-04-25 13:37:40 -07:00
committed by GitHub
parent 8ace1d9c84
commit 94d592f212
159 changed files with 18091 additions and 5843 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PATHFINDER-2026-04-21/07-plans/06-hybrid-search-orchestration.md
+283
View File
@@ -0,0 +1,283 @@
# Plan 06 — hybrid-search-orchestration (clean)
> **Design authority**: `05-clean-flowcharts.md` section 3.6. This plan implements that diagram. When plan and audit disagree, the `06-implementation-plan.md` verified-findings (Phase 0, V11) take precedence.
## Dependencies
- **Upstream**: `07-plans/05-context-injection-engine.md` — introduces `renderObservations(obs, strategy)` and the `SearchResultStrategy` strategy config (derived from `ResultFormatter.ts`). This plan consumes that strategy; it does NOT create it. Hard blocker: Phase 6 below cannot land until Plan 05 Phase 4 lands.
- **Downstream**: `07-plans/10-knowledge-corpus-builder.md` — `CorpusBuilder.build` calls `SearchOrchestrator.search(params)`. Signature stability of `SearchOrchestrator.search` is the contract Plan 10 depends on. Do not rename. Do not change the shape of `StrategySearchResult`.
## Sources consulted
1. `PATHFINDER-2026-04-21/05-clean-flowcharts.md` — section 3.6 (lines 262–292); Part 1 bullshit items #30 #31 #32 #33 (lines 48–51).
2. `PATHFINDER-2026-04-21/06-implementation-plan.md` — Phase 0 V11 (line 38); Phase 4 (lines 208–242); anti-pattern guards C and D (lines 63–64).
3. `PATHFINDER-2026-04-21/01-flowcharts/hybrid-search-orchestration.md` — before-state; full 97 lines.
4. `src/services/worker/SearchManager.ts:1-2069` — full method inventory via grep; spot-read `:1-200`, `:1209-1310`.
5. `src/services/worker/search/SearchOrchestrator.ts:1-290` — confirmed `search(args: any): Promise<StrategySearchResult>` signature; `executeWithFallback` at `:81-121`; silent fallback branch at `:100-110`.
6. `src/services/worker/search/strategies/ChromaSearchStrategy.ts:1-247` — `filterByRecency` at `:196-217`; hard-coded 90-day cutoff via `SEARCH_CONSTANTS.RECENCY_WINDOW_MS` at `:200`.
7. `src/services/worker/search/strategies/SQLiteSearchStrategy.ts:1-132`, `HybridSearchStrategy.ts:1-240`, `SearchStrategy.ts:1-61` — strategy interface and existence confirmed.
8. `src/services/worker/search/types.ts:15-16` — `RECENCY_WINDOW_DAYS: 90` and `RECENCY_WINDOW_MS: 90 * 24 * 60 * 60 * 1000`.
9. `src/services/worker/http/routes/SearchRoutes.ts:1-303` — 14 search/context handlers, all delegating `await this.searchManager.<method>(req.query)`.
10. `PATHFINDER-2026-04-21/07-plans/05-context-injection-engine.md` — `SearchResultStrategy` signature & path (`src/services/worker/search/strategies/SearchResultStrategy.ts` per that plan's Phase 4).
## Concrete findings
### SearchManager method inventory (2069 lines)
Classifications per Decision D ("if body is `return this.other.method(args)`, delete it"):
| `:line` | Method | Classification | Notes |
|---|---|---|---|
| `:59` | `queryChroma` | **real-work (but @deprecated)** | Pre-Orchestrator; called only by `searchChromaForTimeline` and `findByConcept`/`findByFile` hybrid paths inside `SearchManager`. **DELETE** (item #30). |
| `:70` | `searchChromaForTimeline` | **real-work (but @deprecated)** | Bakes 90-day cutoff via `ninetyDaysAgo` param. Callers: only `timeline()` `:490`. **DELETE** (item #30). |
| `:103` | `normalizeParams` | **display-wrap helper** | SearchOrchestrator `:239` has an equivalent. This one adds `filePath→files`, `concept→concepts`, `isFolder` coercion. If we keep SearchManager display-wrap, keep this. Otherwise fold into SearchOrchestrator.normalizeParams and delete. |
| `:161` | `search` | **real-work (display-wrap)** | Lines 161–445: re-implements the whole decision tree + recency filter + categorization + markdown tables. Contains one of four 90-day filter copies (`:230-259`). This is the V11 "real work" method. **REFACTOR**: decision tree/execution deleted (already in Orchestrator); keep only the markdown combining → migrate to `renderObservations(combined, SearchResultStrategy)`. |
| `:450` | `timeline` | **real-work (display-wrap)** | Uses `searchChromaForTimeline` `:490` + 90-day cutoff `:488`. Delegates to `TimelineBuilder` for rendering. **REFACTOR**: strip 90-day cutoff; call `SearchOrchestrator` timeline helpers (`getTimeline`, `formatTimeline` at Orchestrator `:185-209`). |
| `:731` | `decisions` | **display-wrap** | Semantic shortcut; queries Chroma for "decision" observations, renders tables. Route could call `SearchOrchestrator.search({query:'decision', ...})` directly; keep the markdown wrap. |
| `:810` | `changes` | **display-wrap** | Same shape as `decisions`. |
| `:894` | `howItWorks` | **display-wrap** | Same shape. |
| `:951` | `searchObservations` | **pass-through** (with backward-compat shim) | `{type:'observations'}` preset + call through. **DELETE**; route calls `SearchOrchestrator.search({...req.query, type:'observations'})`. |
| `:1037` | `searchSessions` | **pass-through** | Same; `type:'sessions'`. **DELETE**. |
| `:1123` | `searchUserPrompts` | **pass-through** | Same; `type:'prompts'`. **DELETE**. |
| `:1209` | `findByConcept` | **real-work (display-wrap)** | Duplicates the two-phase hybrid logic that exists in `HybridSearchStrategy.findByConcept` at `HybridSearchStrategy.ts:74`. Pure duplication. **DELETE** execution; route calls `SearchOrchestrator.findByConcept(concept, args)` at `SearchOrchestrator.ts:126`. Keep markdown header/table rendering via `renderObservations(obs, SearchResultStrategy)`. |
| `:1277` | `findByFile` | **real-work (display-wrap)** | Same pattern — duplicates `HybridSearchStrategy.findByFile`. **DELETE** execution; route → `SearchOrchestrator.findByFile`. Keep render. |
| `:1399` | `findByType` | **real-work (display-wrap)** | Same pattern — duplicates `HybridSearchStrategy.findByType`. **DELETE** execution; route → `SearchOrchestrator.findByType`. Keep render. |
| `:1468` | `getRecentContext` | **real-work** | ContextBuilder territory, NOT search. Leave to Plan 05. |
| `:1596` | `getContextTimeline` | **real-work** | Same — ContextBuilder / Plan 05. Leave. |
| `:1810` | `getTimelineByQuery` | **real-work** | Contains a fourth copy of the 90-day filter at `:1840-1847`. Depends on `SearchOrchestrator.getTimeline` + `formatTimeline`. **REFACTOR**: strip 90-day; delegate. |
**Tally**: 3 pure pass-throughs to delete (`:951`, `:1037`, `:1123`); 2 `@deprecated` to delete (`:59`, `:70`); 6 real-work methods that keep only their rendering (`:161`, `:450`, `:1209`, `:1277`, `:1399`, `:1810`); 3 semantic shortcuts kept as display-wraps (`:731`, `:810`, `:894`); 2 ContextBuilder-owned methods left for Plan 05 (`:1468`, `:1596`). Every remaining "real-work" body becomes `orchestrator.X(args)` + `renderObservations(combined, SearchResultStrategy, ctx)` — no decision tree, no Chroma calls, no recency filter.
### Duplication vs facade distinction
The three hybrid methods (`findByConcept` `:1209`, `findByFile` `:1277`, `findByType` `:1399`) are not thin facades — they implement the same two-phase (SQLite metadata filter → Chroma semantic rank → intersect) algorithm that already lives in `HybridSearchStrategy.ts:26-240`. This is **parallel reimplementation**, not delegation. Phase 6 kills the in-file copy and routes through `SearchOrchestrator.findByConcept/File/Type` (`SearchOrchestrator.ts:126-180`), which already wraps `HybridSearchStrategy`.
### filterByRecency location
- **Canonical**: `src/services/worker/search/strategies/ChromaSearchStrategy.ts:196-217` — `private filterByRecency(chromaResults)`. Uses `SEARCH_CONSTANTS.RECENCY_WINDOW_MS` at `:200`. Called from `:119` inside `executeChromaSearch`.
- **Constant**: `src/services/worker/search/types.ts:15` — `RECENCY_WINDOW_DAYS: 90`; `:16` — `RECENCY_WINDOW_MS: 90 * 24 * 60 * 60 * 1000`.
- **Legacy copies in `SearchManager.ts`**: `:230`, `:247-259`, `:488`, `:978-985`, `:1064-1071`, `:1150-1157`, `:1840-1847`. All delete with the methods above or their refactors.
### Current Chroma-fail behavior (item #32 silent fallback)
`SearchOrchestrator.executeWithFallback` at `SearchOrchestrator.ts:93-110`:
```ts
const result = await this.chromaStrategy.search(options);
if (result.usedChroma) return result;
// Chroma failed - fall back to SQLite for filter-only
const fallbackResult = await this.sqliteStrategy.search({
...options,
query: undefined // Remove query for SQLite fallback <-- DROPS query text silently
});
return { ...fallbackResult, fellBack: true };
```
And inside `ChromaSearchStrategy.search` at `:76-86`, a thrown error becomes `{ usedChroma: false, fellBack: false }` (swallowed). The Orchestrator's `usedChroma=false` branch then runs SQLite with the query text stripped. **This is the silent fallback from audit item #32**. The current behavior drops the query text and returns filter-only SQLite results — no 503, no error signal to the caller. Caller (SearchManager) flips a `chromaFailed` flag into the rendered markdown, but JSON callers (viewer UI, mem-search skill, CorpusBuilder) have no way to detect it.
### Route surface
`src/services/worker/http/routes/SearchRoutes.ts` declares 18 endpoints. Of those that invoke `this.searchManager.*`:
- Pass-through candidates (3): `/api/search/observations` `:98`, `/api/search/sessions` `:107`, `/api/search/prompts` `:116`.
- Route-to-Orchestrator-directly candidates (3): `/api/search/by-concept` `:125`, `/api/search/by-file` `:134`, `/api/search/by-type` `:143`.
- Display-wrap kept: `/api/search` `:53`, `/api/timeline` `:62`, `/api/decisions` `:71`, `/api/changes` `:80`, `/api/how-it-works` `:89`, `/api/timeline/by-query` `:303`, plus `/api/context/*` (Plan 05 territory).
## Copy-ready snippet locations
- Hybrid decision tree + 503 branch target: `SearchOrchestrator.ts:81-121`. Replace lines 100–110 with the 503 throw.
- 503 shape: follow anti-pattern guard C from `06-implementation-plan.md:63` — throw a typed `ChromaUnavailableError` (new class `src/services/worker/search/errors.ts`) with `code='chroma_unavailable'`; `SearchRoutes.wrapHandler` catches and maps to `res.status(503).json({error:'chroma_unavailable'})`.
- Render path: `renderObservations(combined, SearchResultStrategy, ctx)` from Plan 05 Phase 4 → new file `src/services/worker/search/strategies/SearchResultStrategy.ts`.
- Pass-through deletion ranges: `SearchManager.ts:951-1036` (`searchObservations`), `:1037-1122` (`searchSessions`), `:1123-1208` (`searchUserPrompts`).
- `filterByRecency` + callers to delete: `ChromaSearchStrategy.ts:196-217` + call site `:119`; `SEARCH_CONSTANTS.RECENCY_WINDOW_DAYS`/`_MS` at `types.ts:15-16`; plus the seven copies in `SearchManager.ts` listed above.
## Confidence + gaps
**High confidence**:
- SearchManager method classifications (grep-verified inventory; body-read for the three hybrid methods confirms exact duplication of `HybridSearchStrategy.*`).
- Current silent-fallback behavior (read in `SearchOrchestrator.ts:93-110`).
- 90-day default exists at exactly one shared constant (`types.ts:15-16`) plus seven in-file duplicate copies inside `SearchManager.ts`.
**Gaps**:
- Semantic-inject POST `/api/context/semantic` at `SearchRoutes.ts:270` calls `searchManager.search` with its own mini-formatter **post-render** (flagged by Plan 05 Phase 6). This plan does not touch that handler; Plan 05 owns it.
- `ResultFormatter.formatSearchResults` callers — need one grep pass during Phase 6 to confirm no other caller beyond `SearchManager.search` at `:321`, `formatSearchResults` routes, and `SearchOrchestrator.ts:214` (which also exposes it). Left as a Phase 6 checklist item.
- Exact JSON error body shape for 503 — two reasonable choices (`{error:'chroma_unavailable'}` vs `{error:{code:'chroma_unavailable', retryable:true}}`). Defer to Phase 4 decision; current plan uses the simpler shape.
---
## Phase 1 — Classify every `SearchManager` method
**(a) What**: Lock the method inventory above into the repo as a code comment in `SearchManager.ts` header (keeps future auditors honest). No behavior change.
**(b) Docs**: `05-clean-flowcharts.md` Part 1 item #31; `06-implementation-plan.md:38` (V11); live file `src/services/worker/SearchManager.ts:1-2069`.
**(c) Verification**:
- `grep -n "^\s*async \+[a-zA-Z]" src/services/worker/SearchManager.ts | wc -l` → 15 public async methods (matches inventory).
- `grep -n "@deprecated" src/services/worker/SearchManager.ts` → exactly one hit at `:57` (`queryChroma`). Confirm `searchChromaForTimeline` at `:70` is untagged but classified deprecated per `01-flowcharts/hybrid-search-orchestration.md:91`.
**(d) Anti-pattern guards**: Guard D — every method marked "pass-through" in the inventory must have a body that trivially forwards to `this.orchestrator.*` after reading. If a method claims pass-through but also does date filtering or recency windows, reclassify as real-work before later phases delete it.
---
## Phase 2 — Delete `@deprecated` methods
**(a) What**: Copy from `SearchManager.ts:59-97` — **delete** both `queryChroma` and `searchChromaForTimeline`. Update `timeline()` at `:490` to call `SearchOrchestrator.getTimeline` / `formatTimeline` (`SearchOrchestrator.ts:185-209`) instead.
**(b) Docs**: `05-clean-flowcharts.md` Part 1 item #30 (line 48); `05-clean-flowcharts.md` §3.6 "Deleted" bullet 2 (line 286); `SearchManager.ts:57` @deprecated tag.
**(c) Verification**:
- `grep -rn "queryChroma\|searchChromaForTimeline" src/` → only hits are `chromaSync.queryChroma` (ChromaSync public method — do not touch) and `ChromaSearchStrategy.ts` calls to `chromaSync.queryChroma`.
- `grep -n "@deprecated" src/services/worker/SearchManager.ts` → zero hits.
- `npm run build` passes; `/api/timeline?query=x` still returns timeline.
**(d) Anti-pattern guards**: Guard D — no replacement shim; delete outright. Do not leave a `/** @deprecated */` stub calling the Orchestrator — that is the thin-facade anti-pattern returning.
---
## Phase 3 — Route `SearchRoutes` directly to `SearchOrchestrator` for pass-throughs
**(a) What**: In `src/services/worker/http/routes/SearchRoutes.ts`:
1. Inject `SearchOrchestrator` alongside `SearchManager` (or replace `SearchManager` prop entirely once Phase 6 lands). Copy constructor wiring shape from `SearchRoutes.ts:14-18`.
2. Rewire three handlers:
- `:98` `handleSearchObservations` → `await this.orchestrator.search({...req.query, type:'observations'})`
- `:107` `handleSearchSessions` → `await this.orchestrator.search({...req.query, type:'sessions'})`
- `:116` `handleSearchPrompts` → `await this.orchestrator.search({...req.query, type:'prompts'})`
3. Delete `searchObservations`, `searchSessions`, `searchUserPrompts` from `SearchManager.ts:951-1208`.
**(b) Docs**: `05-clean-flowcharts.md` §3.6 diagram (line 267 `B --> C`); `06-implementation-plan.md:208-225` Phase 4 step 1; live file `src/services/worker/http/routes/SearchRoutes.ts:98-118` and `SearchManager.ts:951-1208`.
**(c) Verification**:
- `grep -n "this.searchManager.search\(Observations\|Sessions\|UserPrompts\)" src/` → zero hits.
- `curl localhost:37777/api/search/observations?query=x` returns the same JSON shape as before (snapshot test).
- Chroma-down test: stop the Chroma subprocess; call `/api/search/observations?query=x` → **503 with `{error:'chroma_unavailable'}`** (contract established in Phase 4). Not an empty `observations:[]` array.
**(d) Anti-pattern guards**:
- Guard D — the deleted methods were ~85 lines each of wrapping; make sure the replacement route lines do NOT re-import a "for type consistency" shim from SearchManager.
- Guard C — if the old pass-through silently caught Chroma failures and returned `observations:[]`, the new direct route must propagate the 503 from Phase 4.
---
## Phase 4 — Replace silent Chroma-fail with 503 in `SearchOrchestrator`
**(a) What**: Copy from `SearchOrchestrator.ts:90-110`. Delete the fallback branch:
```ts
// DELETE these lines 100-110
const fallbackResult = await this.sqliteStrategy.search({...options, query: undefined});
return {...fallbackResult, fellBack: true};
```
Replace with:
```ts
throw new ChromaUnavailableError();
```
Add `src/services/worker/search/errors.ts` exporting `class ChromaUnavailableError extends Error { code = 'chroma_unavailable' }`.
Also update `ChromaSearchStrategy.ts:76-86` — the catch block currently swallows errors and returns `usedChroma:false`. Change to rethrow as `ChromaUnavailableError` so `executeWithFallback` sees it.
In `SearchRoutes.ts` `wrapHandler` (or `BaseRouteHandler`), catch `ChromaUnavailableError` → `res.status(503).json({error:'chroma_unavailable'})`.
Update `SearchOrchestrator.findByConcept`/`findByType`/`findByFile` (`:126-180`) — today they fall back to SQLite-only on no-hybrid. That fallback is **allowed** because concept/type/file filters are legitimate without Chroma. Only text-query paths get 503. Document this distinction inline.
**(b) Docs**: `05-clean-flowcharts.md` Part 1 item #32 (line 50); `05-clean-flowcharts.md` §3.6 line 271 (`Return 503 error=chroma_unavailable (NO silent fallback)`); `06-implementation-plan.md:63` anti-pattern C; `06-implementation-plan.md:644` verification line (grep for `res.status(503)` + `chroma_unavailable`).
**(c) Verification**:
- Unit test: stub `ChromaSync.queryChroma` to throw → `SearchOrchestrator.search({query:'x'})` throws `ChromaUnavailableError`.
- Unit test: construct `SearchOrchestrator` with `chromaSync = null` → `search({query:'x'})` throws `ChromaUnavailableError` (today returns an empty result at `:115-120`; that branch also goes).
- Integration test: `curl localhost:37777/api/search?query=x` with Chroma disabled → `503` with body `{"error":"chroma_unavailable"}`.
- Integration test: `curl localhost:37777/api/search/by-concept?concept=x` with Chroma disabled → 200 with SQLite-only results. Concept/type/file filters remain functional without Chroma; only text-query paths hard-fail.
- `curl localhost:37777/api/search` (no query) with Chroma disabled → 200 with SQLite filter-only results (this path is legitimate per §3.6 line 272).
- `grep -rn "query: undefined" src/services/worker/search/` → zero hits (the silent-drop pattern).
- `grep -rn "fellBack" src/` → zero hits. The `fellBack` field on `StrategySearchResult` is obsolete once fallback is deleted; remove from `types.ts` as part of this phase.
**(d) Anti-pattern guards**:
- Guard C — primary target. Silent fallback deleted; explicit error class + HTTP status.
- Guard D — do not wrap the new throw behind a shim in `SearchManager`. The orchestrator throws; routes handle.
---
## Phase 5 — Delete `filterByRecency` and the 90-day default
**(a) What**:
1. Copy from `ChromaSearchStrategy.ts:196-217` — **delete** `filterByRecency` method.
2. Delete its call site at `ChromaSearchStrategy.ts:119` (`const recentItems = this.filterByRecency(chromaResults);`). Replace with direct `chromaResults.ids` + `metadatas` join (preserve the metadata-by-id map logic from the old method's lines `:202-208` — that dedup IS real work; only the 90-day filter goes).
3. Delete `SEARCH_CONSTANTS.RECENCY_WINDOW_DAYS` and `RECENCY_WINDOW_MS` from `src/services/worker/search/types.ts:15-16`.
4. Delete the seven in-file copies in `SearchManager.ts` (lines 230-259, 488, 978-985, 1064-1071, 1150-1157, 1840-1847). Replaced by caller-supplied `dateRange` only — if caller wants recency, caller passes `dateRange: {start, end}`.
**(b) Docs**: `05-clean-flowcharts.md` Part 1 item #33 (line 51); `05-clean-flowcharts.md` §3.6 "Deleted" bullet 4 (line 288); live `src/services/worker/search/strategies/ChromaSearchStrategy.ts:196-217`; `src/services/worker/search/types.ts:15-16`.
**(c) Verification**:
- `grep -rn "RECENCY_WINDOW\|filterByRecency\|ninetyDaysAgo\|90.day\|90 days" src/` → zero hits.
- Integration test: seed an observation dated 100 days ago; query by its text → it appears in results (would have been filtered out pre-deletion).
- Integration test: pass `dateRange.start` = 60 days ago; observation from 100 days ago is excluded. Explicit filter still works.
**(d) Anti-pattern guards**:
- Guard C — silent implicit filter replaced by explicit caller param.
- Guard D — no "convenience wrapper" that re-applies 90 days when `dateRange` is missing. Missing = all.
---
## Phase 6 — Keep display-wrap in `SearchManager`; switch to `renderObservations(results, SearchResultStrategy)`
**BLOCKED until**: Plan 05 Phase 4 lands and ships `src/services/worker/search/strategies/SearchResultStrategy.ts`.
**(a) What**:
1. In `SearchManager.ts:161-445` (`search`): delete everything from the `PATH 1` decision at `:177` through the categorization/hydration blocks at `:321`. The full decision tree is already in `SearchOrchestrator.search`. Replace body with:
```ts
async search(args: any): Promise<any> {
const results = await this.orchestrator.search(args);
if (args.format === 'json') return { content:[{type:'text', text: JSON.stringify(results)}] };
const combined = combineResults(results.results);
return { content:[{type:'text', text: renderObservations(combined, SearchResultStrategy, ctx)}] };
}
```
2. Apply same transformation to `timeline` `:450`, `findByConcept` `:1209`, `findByFile` `:1277`, `findByType` `:1399`, `getTimelineByQuery` `:1810`. Each becomes: call orchestrator → render via strategy. Keep the outer `{content:[{type:'text', ...}]}` MCP envelope; drop everything in between.
3. Keep `decisions`, `changes`, `howItWorks` `:731-950` as semantic-shortcut wrappers. They compute a preset query string, call `this.orchestrator.search({...args, query:'decision'})` (or equivalent), render via `renderObservations`. Body shrinks from ~70 lines each to ~10.
4. Delete or drop-in replace `normalizeParams` at `:103` — `SearchOrchestrator.normalizeParams` at `:239` is canonical. If the API-only coercions (`filePath→files`, `isFolder`) are missing there, **move them into** `SearchOrchestrator.normalizeParams` and delete the SearchManager copy. Guard: grep every caller to confirm the Orchestrator version covers all cases.
**(b) Docs**: `05-clean-flowcharts.md` §3.6 line 281 (`Fmt -->|markdown| M["renderObservations(results, SearchResultStrategy)"]`); `06-implementation-plan.md:220-225` (Phase 4 step 3 — keep the combine/group/table code as a `ResultRenderer` module); `07-plans/05-context-injection-engine.md:169-182` Phase 4 (SearchResultStrategy); live `src/services/worker/SearchManager.ts:161-445`.
**(c) Verification**:
- `wc -l src/services/worker/SearchManager.ts` → under 400 lines (from 2069).
- Snapshot test: fixture `SearchResults` → `renderObservations(combined, SearchResultStrategy, ctx)` output is byte-equal to the pre-refactor `ResultFormatter.formatSearchResults` output. Plan 05 Phase 4 owns this fixture; reuse it here.
- `grep -n "combineResults\|groupByDate\|groupByFile" src/services/worker/SearchManager.ts` → zero hits (now lives in SearchResultStrategy / renderObservations).
- Manual: viewer UI `http://localhost:37777` search results render identically.
**(d) Anti-pattern guards**:
- Guard D — SearchManager's remaining methods must each be ≤15 lines (orchestrator call + render envelope). If any method balloons back, it's re-implementing decision logic.
- Guard A (strategy count from Plan 05 audit Part 2) — don't invent a fifth strategy just for "semantic context injection". Plan 05 Phase 6 routes that handler through `SearchResultStrategy` with a flag.
---
## Phase 7 — Verification
Run all checks from phases 1–6 in one pass, plus:
1. **Behavior preservation**:
- All three search paths (filter-only, Chroma-semantic, hybrid concept/type/file) return results for representative queries.
- `?format=json` and default markdown both work on every search endpoint.
- `concept=`, `type=`, `obs_type=`, `files=`, `filePath=` filters all honored (grep-verify normalizeParams covers each).
- Timeline endpoint returns chronological groupings with anchor depth filtering intact.
2. **Chroma-down contract**:
- Stop Chroma subprocess. `curl /api/search?query=x` → 503 `{"error":"chroma_unavailable"}`. Not empty, not silent.
- `curl /api/search` (no query) → 200 with SQLite filter results.
- `curl /api/search/by-concept?concept=foo` → 200 with SQLite metadata results (per `SearchOrchestrator.ts:126-140`).
3. **Line-count targets**:
- `SearchManager.ts`: 2069 → under 400 lines (≥1600 deleted).
- `SearchOrchestrator.ts`: ~290 → ~280 (fallback branch removed, error class added).
- `ChromaSearchStrategy.ts`: 247 → ~215 (filterByRecency deleted).
- Net project delete target: ~1700 lines.
4. **Grep contract checks**:
- `grep -rn "query: undefined" src/services/worker/search/` → 0.
- `grep -rn "RECENCY_WINDOW\|filterByRecency\|ninetyDaysAgo" src/` → 0.
- `grep -rn "@deprecated" src/services/worker/SearchManager.ts` → 0.
- `grep -rn "this.searchManager.search\(Observations\|Sessions\|UserPrompts\)" src/` → 0.
- `grep -rn "res.status(503)" src/services/worker/http/` → at least one hit on the `chroma_unavailable` path.
5. **Downstream smoke** (Plan 10 contract):
- `CorpusBuilder.build` test — feed synthetic observations, confirm `SearchOrchestrator.search` signature unchanged and `StrategySearchResult` shape stable.
6. **Anti-pattern audit**:
- Guard C: no `catch { return empty }` patterns in `src/services/worker/search/`.
- Guard D: every method in `SearchManager.ts` either renders or shortcut-presets. No single-line `return this.orchestrator.x(args)` remains.