Files
claude-mem/PATHFINDER-2026-04-21/07-plans/05-context-injection-engine.md
T
Alex Newman 94d592f212 perf: streamline worker startup and consolidate database connections (#2122)
* docs: pathfinder refactor corpus + Node 20 preflight

Adds the PATHFINDER-2026-04-22 principle-driven refactor plan (11 docs,
cross-checked PASS) plus the exploratory PATHFINDER-2026-04-21 corpus
that motivated it. Bumps engines.node to >=20.0.0 per the ingestion-path
plan preflight (recursive fs.watch). Adds the pathfinder skill.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: land PATHFINDER Plan 01 — data integrity

Schema, UNIQUE constraints, self-healing claim, Chroma upsert fallback.

- Phase 1: fresh schema.sql regenerated at post-refactor shape.
- Phase 2: migrations 23+24 — rebuild pending_messages without
  started_processing_at_epoch; UNIQUE(session_id, tool_use_id);
  UNIQUE(memory_session_id, content_hash) on observations; dedup
  duplicate rows before adding indexes.
- Phase 3: claimNextMessage rewritten to self-healing query using
  worker_pid NOT IN live_worker_pids; STALE_PROCESSING_THRESHOLD_MS
  and the 60-s stale-reset block deleted.
- Phase 4: DEDUP_WINDOW_MS and findDuplicateObservation deleted;
  observations.insert now uses ON CONFLICT DO NOTHING.
- Phase 5: failed-message purge block deleted from worker-service
  2-min interval; clearFailedOlderThan method deleted.
- Phase 6: repairMalformedSchema and its Python subprocess repair
  path deleted from Database.ts; SQLite errors now propagate.
- Phase 7: Chroma delete-then-add fallback gated behind
  CHROMA_SYNC_FALLBACK_ON_CONFLICT env flag as bridge until
  Chroma MCP ships native upsert.
- Phase 8: migration 19 no-op block absorbed into fresh schema.sql.

Verification greps all return 0 matches. bun test tests/sqlite/
passes 63/63. bun run build succeeds.

Plan: PATHFINDER-2026-04-22/01-data-integrity.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: land PATHFINDER Plan 02 — process lifecycle

OS process groups replace hand-rolled reapers. Worker runs until
killed; orphans are prevented by detached spawn + kill(-pgid).

- Phase 1: src/services/worker/ProcessRegistry.ts DELETED. The
  canonical registry at src/supervisor/process-registry.ts is the
  sole survivor; SDK spawn site consolidated into it via new
  createSdkSpawnFactory/spawnSdkProcess/getSdkProcessForSession/
  ensureSdkProcessExit/waitForSlot helpers.
- Phase 2: SDK children spawn with detached:true + stdio:
  ['ignore','pipe','pipe']; pgid recorded on ManagedProcessInfo.
- Phase 3: shutdown.ts signalProcess teardown uses
  process.kill(-pgid, signal) on Unix when pgid is recorded;
  Windows path unchanged (tree-kill/taskkill).
- Phase 4: all reaper intervals deleted — startOrphanReaper call,
  staleSessionReaperInterval setInterval (including the co-located
  WAL checkpoint — SQLite's built-in wal_autocheckpoint handles
  WAL growth without an app-level timer), killIdleDaemonChildren,
  killSystemOrphans, reapOrphanedProcesses, reapStaleSessions, and
  detectStaleGenerator. MAX_GENERATOR_IDLE_MS and MAX_SESSION_IDLE_MS
  constants deleted.
- Phase 5: abandonedTimer — already 0 matches; primary-path cleanup
  via generatorPromise.finally() already lives in worker-service
  startSessionProcessor and SessionRoutes ensureGeneratorRunning.
- Phase 6: evictIdlestSession and its evict callback deleted from
  SessionManager. Pool admission gates backpressure upstream.
- Phase 7: SDK-failure fallback — SessionManager has zero matches
  for fallbackAgent/Gemini/OpenRouter. Failures surface to hooks
  via exit code 2 through SessionRoutes error mapping.
- Phase 8: ensureWorkerRunning in worker-utils.ts rewritten to
  lazy-spawn — consults isWorkerPortAlive (which gates
  captureProcessStartToken for PID-reuse safety via commit
  99060bac), then spawns detached with unref(), then
  waitForWorkerPort({ attempts: 3, backoffMs: 250 }) hand-rolled
  exponential backoff 250→500→1000ms. No respawn npm dep.
- Phase 9: idle self-shutdown — zero matches for
  idleCheck/idleTimeout/IDLE_MAX_MS/idleShutdown. Worker exits
  only on external SIGTERM via supervisor signal handlers.

Three test files that exercised deleted code removed:
tests/worker/process-registry.test.ts,
tests/worker/session-lifecycle-guard.test.ts,
tests/services/worker/reap-stale-sessions.test.ts.
Pass count: 1451 → 1407 (-44), all attributable to deleted test
files. Zero new failures. 31 pre-existing failures remain
(schema-repair suite, logger-usage-standards, environmental
openclaw / plugin-distribution) — none introduced by Plan 02.

All 10 verification greps return 0. bun run build succeeds.

Plan: PATHFINDER-2026-04-22/02-process-lifecycle.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: land PATHFINDER Plan 04 (narrowed) — search fail-fast

Phases 3, 5, 6 only. Plan-doc inaccuracies for phases 1/2/4/7/8/9
deferred for plan reconciliation:
  - Phase 1/2: ObservationRow type doesn't exist; the four
    "formatters" operate on three incompatible types.
  - Phase 4: RECENCY_WINDOW_MS already imported from
    SEARCH_CONSTANTS at every call site.
  - Phase 7: getExistingChromaIds is NOT @deprecated and has an
    active caller in ChromaSync.backfillMissingSyncs.
  - Phase 8: estimateTokens already consolidated.
  - Phase 9: knowledge-corpus rewrite blocked on PG-3
    prompt-caching cost smoke test.

Phase 3 — Delete SearchManager.findByConcept/findByFile/findByType.
SearchRoutes handlers (handleSearchByConcept/File/Type) now call
searchManager.getOrchestrator().findByXxx() directly via new
getter accessors on SearchManager. ~250 LoC deleted.

Phase 5 — Fail-fast Chroma. Created
src/services/worker/search/errors.ts with ChromaUnavailableError
extends AppError(503, 'CHROMA_UNAVAILABLE'). Deleted
SearchOrchestrator.executeWithFallback's Chroma-failed
SQLite-fallback branch; runtime Chroma errors now throw 503.
"Path 3" (chromaSync was null at construction — explicit-
uninitialized config) preserved as legitimate empty-result state
per plan text. ChromaSearchStrategy.search no longer wraps in
try/catch — errors propagate.

Phase 6 — Delete HybridSearchStrategy three try/catch silent
fallback blocks (findByConcept, findByType, findByFile) at lines
~82-95, ~120-132, ~161-172. Removed `fellBack` field from
StrategySearchResult type and every return site
(SQLiteSearchStrategy, BaseSearchStrategy.emptyResult,
SearchOrchestrator).

Tests updated (Principle 7 — delete in same PR):
  - search-orchestrator.test.ts: "fall back to SQLite" rewritten
    as "throw ChromaUnavailableError (HTTP 503)".
  - chroma/hybrid/sqlite-search-strategy tests: rewritten to
    rejects.toThrow; removed fellBack assertions.

Verification: SearchManager.findBy → 0; fellBack → 0 in src/.
bun test tests/worker/search/ → 122 pass, 0 fail.
bun test (suite-wide) → 1407 pass, baseline maintained, 0 new
failures. bun run build succeeds.

Plan: PATHFINDER-2026-04-22/04-read-path.md (Phases 3, 5, 6)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: land PATHFINDER Plan 03 — ingestion path

Fail-fast parser, direct in-process ingest, recursive fs.watch,
DB-backed tool pairing. Worker-internal HTTP loopback eliminated.

- Phase 0: Created src/services/worker/http/shared.ts exporting
  ingestObservation/ingestPrompt/ingestSummary as direct
  in-process functions plus ingestEventBus (Node EventEmitter,
  reusing existing pattern — no third event bus introduced).
  setIngestContext wires the SessionManager dependency from
  worker-service constructor.
- Phase 1: src/sdk/parser.ts collapsed to one parseAgentXml
  returning { valid:true; kind: 'observation'|'summary'; data }
  | { valid:false; reason: string }. Inspects root element;
  <skip_summary reason="…"/> is a first-class summary case
  with skipped:true. NEVER returns undefined. NEVER coerces.
- Phase 2: ResponseProcessor calls parseAgentXml exactly once,
  branches on the discriminated union. On invalid → markFailed
  + logger.warn(reason). On observation → ingestObservation.
  On summary → ingestSummary then emit summaryStoredEvent
  { sessionId, messageId } (consumed by Plan 05's blocking
  /api/session/end).
- Phase 3: Deleted consecutiveSummaryFailures field
  (ResponseProcessor + SessionManager + worker-types) and
  MAX_CONSECUTIVE_SUMMARY_FAILURES constant. Circuit-breaker
  guards and "tripped" log lines removed.
- Phase 4: coerceObservationToSummary deleted from sdk/parser.ts.
- Phase 5: src/services/transcripts/watcher.ts rescan setInterval
  replaced with fs.watch(transcriptsRoot, { recursive: true,
  persistent: true }) — Node 20+ recursive mode.
- Phase 6: src/services/transcripts/processor.ts pendingTools
  Map deleted. tool_use rows insert with INSERT OR IGNORE on
  UNIQUE(session_id, tool_use_id) (added by Plan 01). New
  pairToolUsesByJoin query in PendingMessageStore for read-time
  pairing (UNIQUE INDEX provides idempotency; explicit consumer
  not yet wired).
- Phase 7: HTTP loopback at processor.ts:252 replaced with
  direct ingestObservation call. maybeParseJson silent-passthrough
  rewritten to fail-fast (throws on malformed JSON).
- Phase 8: src/utils/tag-stripping.ts countTags + stripTagsInternal
  collapsed into one alternation regex, single-pass over input.
- Phase 9: src/utils/transcript-parser.ts (dead TranscriptParser
  class) deleted. The active extractLastMessage at
  src/shared/transcript-parser.ts:41-144 is the sole survivor.

Tests updated (Principle 7 — same-PR delete):
  - tests/sdk/parser.test.ts + parse-summary.test.ts: rewritten
    to assert discriminated-union shape; coercion-specific
    scenarios collapse into { valid:false } assertions.
  - tests/worker/agents/response-processor.test.ts: circuit-breaker
    describe block skipped; non-XML/empty-response tests assert
    fail-fast markFailed behavior.

Verification: every grep returns 0. transcript-parser.ts deleted.
bun run build succeeds. bun test → 1399 pass / 28 fail / 7 skip
(net -8 pass = the 4 retired circuit-breaker tests + 4 collapsed
parser cases). Zero new failures vs baseline.

Deferred (out of Plan 03 scope, will land in Plan 06): SessionRoutes
HTTP route handlers still call sessionManager.queueObservation
inline rather than the new shared helpers — the helpers are ready,
the route swap is mechanical and belongs with the Zod refactor.

Plan: PATHFINDER-2026-04-22/03-ingestion-path.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: land PATHFINDER Plan 05 — hook surface

Worker-call plumbing collapsed to one helper. Polling replaced by
server-side blocking endpoint. Fail-loud counter surfaces persistent
worker outages via exit code 2.

- Phase 1: plugin/hooks/hooks.json — three 20-iteration `for i in
  1..20; do curl -sf .../health && break; sleep 0.1; done` shell
  retry wrappers deleted. Hook commands invoke their bun entry
  point directly.
- Phase 2: src/shared/worker-utils.ts — added
  executeWithWorkerFallback<T>(url, method, body) returning
  T | { continue: true; reason?: string }. All 8 hook handlers
  (observation, session-init, context, file-context, file-edit,
  summarize, session-complete, user-message) rewritten to use
  it instead of duplicating the ensureWorkerRunning →
  workerHttpRequest → fallback sequence.
- Phase 3: blocking POST /api/session/end in SessionRoutes.ts
  using validateBody + sessionEndSchema (z.object({sessionId})).
  One-shot ingestEventBus.on('summaryStoredEvent') listener,
  30 s timer, req.aborted handler — all share one cleanup so
  the listener cannot leak. summarize.ts polling loop, plus
  MAX_WAIT_FOR_SUMMARY_MS / POLL_INTERVAL_MS constants, deleted.
- Phase 4: src/shared/hook-settings.ts — loadFromFileOnce()
  memoizes SettingsDefaultsManager.loadFromFile per process.
  Per-handler settings reads collapsed.
- Phase 5: src/shared/should-track-project.ts — single exclusion
  check entry; isProjectExcluded no longer referenced from
  src/cli/handlers/.
- Phase 6: cwd validation pushed into adapter normalizeInput
  (all 6 adapters: claude-code, cursor, raw, gemini-cli,
  windsurf). New AdapterRejectedInput error in
  src/cli/adapters/errors.ts. Handler-level isValidCwd checks
  deleted from file-edit.ts and observation.ts. hook-command.ts
  catches AdapterRejectedInput → graceful fallback.
- Phase 7: session-init.ts conditional initAgent guard deleted;
  initAgent is idempotent. tests/hooks/context-reinjection-guard
  test (validated the deleted conditional) deleted in same PR
  per Principle 7.
- Phase 8: fail-loud counter at ~/.claude-mem/state/hook-failures
  .json. Atomic write via .tmp + rename. CLAUDE_MEM_HOOK_FAIL_LOUD
  _THRESHOLD setting (default 3). On consecutive worker-unreachable
  ≥ N: process.exit(2). On success: reset to 0. NOT a retry.
- Phase 9: ensureWorkerAliveOnce() module-scope memoization
  wrapping ensureWorkerRunning. executeWithWorkerFallback calls
  the memoized version.

Minimal validateBody middleware stub at
src/services/worker/http/middleware/validateBody.ts. Plan 06 will
expand with typed inference + error envelope conventions.

Verification: 4/4 grep targets pass. bun run build succeeds.
bun test → 1393 pass / 28 fail / 7 skip; -6 pass attributable
solely to deleted context-reinjection-guard test file. Zero new
failures vs baseline.

Plan: PATHFINDER-2026-04-22/05-hook-surface.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: land PATHFINDER Plan 06 — API surface

One Zod-based validator wrapping every POST/PUT. Rate limiter,
diagnostic endpoints, and shutdown wrappers deleted. Failure-
marking consolidated to one helper.

- Phase 1 (preflight): zod@^3 already installed.
- Phase 2: validateBody middleware confirmed at canonical shape
  in src/services/worker/http/middleware/validateBody.ts —
  safeParse → 400 { error: 'ValidationError', issues: [...] }
  on failure, replaces req.body with parsed value on success.
- Phase 3: Per-route Zod schemas declared at the top of each
  route file. 24 POST endpoints across SessionRoutes,
  CorpusRoutes, DataRoutes, MemoryRoutes, SearchRoutes,
  LogsRoutes, SettingsRoutes now wrap with validateBody().
  /api/session/end (Plan 05) confirmed using same middleware.
- Phase 4: validateRequired() deleted from BaseRouteHandler
  along with every call site. Inline coercion helpers
  (coerceStringArray, coercePositiveInteger) and inline
  if (!req.body...) guards deleted across all route files.
- Phase 5: Rate limiter middleware and its registration deleted
  from src/services/worker/http/middleware.ts. Worker binds
  127.0.0.1:37777 — no untrusted caller.
- Phase 6: viewer.html cached at module init in ViewerRoutes.ts
  via fs.readFileSync; served as Buffer with text/html content
  type. SKILL.md + per-operation .md files cached in
  Server.ts as Map<string, string>; loadInstructionContent
  helper deleted. NO fs.watch, NO TTL — process restart is the
  cache-invalidation event.
- Phase 7: Four diagnostic endpoints deleted from DataRoutes.ts
  — /api/pending-queue (GET), /api/pending-queue/process (POST),
  /api/pending-queue/failed (DELETE), /api/pending-queue/all
  (DELETE). Helper methods that ONLY served them
  (getQueueMessages, getStuckCount, getRecentlyProcessed,
  clearFailed, clearAll) deleted from PendingMessageStore.
  KEPT: /api/processing-status (observability), /health
  (used by ensureWorkerRunning).
- Phase 8: stopSupervisor wrapper deleted from supervisor/index.ts.
  GracefulShutdown now calls getSupervisor().stop() directly.
  Two functions retained with clear roles:
    - performGracefulShutdown — worker-side 6-step shutdown
    - runShutdownCascade — supervisor-side child teardown
      (process.kill(-pgid), Windows tree-kill, PID-file cleanup)
  Each has unique non-trivial logic and a single canonical caller.
- Phase 9: transitionMessagesTo(status, filter) is the sole
  failure-marking path on PendingMessageStore. Old methods
  markSessionMessagesFailed and markAllSessionMessagesAbandoned
  deleted along with all callers (worker-service,
  SessionCompletionHandler, tests/zombie-prevention).

Tests updated (Principle 7 same-PR delete): coercion test files
refactored to chain validateBody → handler. Zombie-prevention
tests rewritten to call transitionMessagesTo.

Verification: all 4 grep targets → 0. bun run build succeeds.
bun test → 1393 pass / 28 fail / 7 skip — exact match to
baseline. Zero new failures.

Plan: PATHFINDER-2026-04-22/06-api-surface.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: land PATHFINDER Plan 07 — dead code sweep

ts-prune-driven sweep across the tree after Plans 01-06 landed.
Deleted unused exports, orphan helpers, and one fully orphaned
file. Earlier-plan deletions verified.

Deleted:
- src/utils/bun-path.ts (entire file — getBunPath, getBunPathOrThrow,
  isBunAvailable: zero importers)
- bun-resolver.getBunVersionString: zero callers
- PendingMessageStore.retryMessage / resetProcessingToPending /
  abortMessage: superseded by transitionMessagesTo (Plan 06 Phase 9)
- EnvManager.MANAGED_CREDENTIAL_KEYS, EnvManager.setCredential:
  zero callers
- CodexCliInstaller.checkCodexCliStatus: zero callers; no status
  command exists in npx-cli
- Two "REMOVED: cleanupOrphanedSessions" stale-fence comments

Kept (with documented justification):
- Public API surface in dist/sdk/* (parseAgentXml, prompt
  builders, ParsedObservation, ParsedSummary, ParseResult,
  SUMMARY_MODE_MARKER) — exported via package.json sdk path.
- generateContext / loadContextConfig / token utilities — used
  via dynamic await import('../../../context-generator.js') in
  worker SearchRoutes.
- MCP_IDE_INSTALLERS, install/uninstall functions for codex/goose
  — used via dynamic await import in npx-cli/install.ts +
  uninstall.ts (ts-prune cannot trace dynamic imports).
- getExistingChromaIds — active caller in
  ChromaSync.backfillMissingSyncs (Plan 04 narrowed scope).
- processPendingQueues / getSessionsWithPendingMessages — active
  orphan-recovery caller in worker-service.ts plus
  zombie-prevention test coverage.
- StoreAndMarkCompleteResult legacy alias — return-type annotation
  in same file.
- All Database.ts barrel re-exports — used downstream.

Earlier-plan verification:
- Plan 03 Phase 9: VERIFIED — src/utils/transcript-parser.ts
  is gone; TranscriptParser has 0 references in src/.
- Plan 01 Phase 8: VERIFIED — migration 19 no-op absorbed.
- SessionStore.ts:52-70 consolidation NOT executed (deferred):
  the methods are not thin wrappers but ~900 LoC of bodies, and
  two methods are documented as intentional mirrors so the
  context-generator.cjs bundle stays schema-consistent without
  pulling MigrationRunner. Deserves its own plan, not a sweep.

Verification: TranscriptParser → 0; transcript-parser.ts → gone;
no commented-out code markers remain. bun run build succeeds.
bun test → 1393 pass / 28 fail / 7 skip — EXACT match to
baseline. Zero regressions.

Plan: PATHFINDER-2026-04-22/07-dead-code.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: remove residual ProcessRegistry comment reference

Plan 07 dead-code sweep missed one comment-level reference to the
deleted in-memory ProcessRegistry class in SessionManager.ts:347.
Rewritten to describe the supervisor.json scope without naming the
deleted class, completing the verification grep target.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: address Greptile review (P1 + 2× P2)

P1 — Plan 05 Phase 3 blocking endpoint was non-functional:
executeWithWorkerFallback used HEALTH_CHECK_TIMEOUT_MS (3 s) for
the POST /api/session/end call, but the server holds the
connection for SERVER_SIDE_SUMMARY_TIMEOUT_MS (30 s). Client
always raced to a "timed out" rejection that isWorkerUnavailable
classified as worker-unreachable, so the hook silently degraded
instead of waiting for summaryStoredEvent.
  - Added optional timeoutMs to executeWithWorkerFallback,
    forwarded to workerHttpRequest.
  - summarize.ts call site now passes 35_000 (5 s above server
    hold window).

P2 — ingestSummary({ kind: 'parsed' }) branch was dead code:
ResponseProcessor emitted summaryStoredEvent directly via the
event bus, bypassing the centralized helper that the comment
claimed was the single source.
  - ResponseProcessor now calls ingestSummary({ kind: 'parsed',
    sessionDbId, messageId, contentSessionId, parsed }) so the
    event-emission path is single-sourced.
  - ingestSummary's requireContext() resolution moved inside the
    'queue' branch (the only branch that needs sessionManager /
    dbManager). 'parsed' is a pure event-bus emission and
    doesn't need worker-internal context — fixes mocked
    ResponseProcessor unit tests that don't call
    setIngestContext.

P2 — isWorkerFallback could false-positive on legitimate API
responses whose schema includes { continue: true, ... }:
  - Added a Symbol.for('claude-mem/worker-fallback') brand to
    WorkerFallback. isWorkerFallback now checks the brand, not
    a duck-typed property name.

Verification: bun run build succeeds. bun test → 1393 pass /
28 fail / 7 skip — exact baseline match. Zero new failures.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: address Greptile iteration 2 (P1 + P2)

P1 — summaryStoredEvent fired regardless of whether the row was
persisted. ResponseProcessor's call to ingestSummary({ kind:
'parsed' }) ran for every parsed.kind === 'summary' even when
result.summaryId came back null (e.g. FK violation, null
memory_session_id at commit). The blocking /api/session/end
endpoint then returned { ok: true } and the Stop hook logged
'Summary stored' for a non-existent row.

  - Gate ingestSummary call on (parsed.data.skipped ||
    session.lastSummaryStored). Skipped summaries are an explicit
    no-op bypass and still confirm; real summaries only confirm
    when storage actually wrote a row.
  - Non-skipped + summaryId === null path logs a warn and lets
    the server-side timeout (504) surface to the hook instead of
    a false ok:true.

P2 — PendingMessageStore.enqueue() returns 0 when INSERT OR
IGNORE suppresses a duplicate (the UNIQUE(session_id, tool_use_id)
constraint added by Plan 01 Phase 1). The two callers
(SessionManager.queueObservation and queueSummarize) previously
logged 'ENQUEUED messageId=0' which read like a row was inserted.

  - Branch on messageId === 0 and emit a 'DUP_SUPPRESSED' debug
    log instead of the misleading ENQUEUED line. No behavior
    change — the duplicate is still correctly suppressed by the
    DB (Principle 3); only the log surface is corrected.
  - confirmProcessed is never called with the enqueue() return
    value (it operates on session.processingMessageIds[] from
    claimNextMessage), so no caller is broken; the visibility
    fix prevents future misuse.

Verification: bun run build succeeds. bun test → 1393 pass /
28 fail / 7 skip — exact baseline match. Zero new failures.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: address Greptile iteration 3 (P1 + 2× P2)

- P1 worker-service.ts: wire ensureGeneratorRunning into the ingest
  context after SessionRoutes is constructed. setIngestContext runs
  before routes exist, so transcript-watcher observations queued via
  ingestObservation() had no way to auto-start the SDK generator.
  Added attachIngestGeneratorStarter() to patch the callback in.
- P2 shared.ts: IngestEventBus now sets maxListeners to 0. Concurrent
  /api/session/end calls register one listener each and clean up on
  completion, so the default-10 warning fires spuriously under normal
  load.
- P2 SessionRoutes.ts: handleObservationsByClaudeId now delegates to
  ingestObservation() instead of duplicating skip-tool / meta /
  privacy / queue logic. Single helper, matching the Plan 03 goal.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: address Greptile iteration 4 (P1 tool-pair + P2 parse/path/doc)

- processor.handleToolResult: restore in-memory tool-use→tool-result
  pairing via session.pendingTools for schemas (e.g. Codex) whose
  tool_result events carry only tool_use_id + output. Without this,
  neither handler fired — all tool observations silently dropped.
- processor.maybeParseJson: return raw string on parse failure instead
  of throwing. Previously a single malformed JSON-shaped field caused
  handleLine's outer catch to discard the entire transcript line.
- watcher.deepestNonGlobAncestor: split on / and \\, emit empty string
  for purely-glob inputs so the caller skips the watch instead of
  anchoring fs.watch at the filesystem root. Windows-compatible.
- PendingMessageStore.enqueue: tighten docstring — callers today only
  log on the returned id; the SessionManager branches on id === 0.

* fix: forward tool_use_id through ingestObservation (Greptile iter 5)

P1 — Plan 01's UNIQUE(content_session_id, tool_use_id) dedup never
fired because the new shared ingest path dropped the toolUseId before
queueObservation. SQLite treats NULL values as distinct for UNIQUE,
so every replayed transcript line landed a duplicate row.

- shared.ingestObservation: forward payload.toolUseId to
  queueObservation so INSERT OR IGNORE can actually collapse.
- SessionRoutes.handleObservationsByClaudeId: destructure both
  tool_use_id (HTTP convention) and toolUseId (JS convention) from
  req.body and pass into ingestObservation.
- observationsByClaudeIdSchema: declare both keys explicitly so the
  validator doesn't rely on .passthrough() alone.

* fix: drop dead pairToolUsesByJoin, close session-end listener race

- PendingMessageStore: delete pairToolUsesByJoin. The method was never
  called and its self-join semantics are structurally incompatible
  with UNIQUE(content_session_id, tool_use_id): INSERT OR IGNORE
  collapses any second row with the same pair, so a self-join can
  only ever match a row to itself. In-memory pendingTools in
  processor.ts remains the pairing path for split-event schemas.

- IngestEventBus: retain a short-lived (60s) recentStored map keyed
  by sessionId. Populated on summaryStoredEvent emit, evicted on
  consume or TTL.

- handleSessionEnd: drain the recent-events buffer before attaching
  the listener. Closes the register-after-emit race where the summary
  can persist between the hook's summarize POST and its session/end
  POST — previously that window returned 504 after the 30s timeout.

* chore: merge origin/main into vivacious-teeth

Resolves conflicts with 15 commits on main (v12.3.9, security
observation types, Telegram notifier, PID-reuse worker start-guard).

Conflict resolution strategy:
- plugin/hooks/hooks.json, plugin/scripts/*.cjs, plugin/ui/viewer-bundle.js:
  kept ours — PATHFINDER Plan 05 deletes the for-i-in-1-to-20 curl retry
  loops and the built artifacts regenerate on build.
- src/cli/handlers/summarize.ts: kept ours — Plan 05 blocking
  POST /api/session/end supersedes main's fire-and-forget path.
- src/services/worker-service.ts: kept ours — Plan 05 ingest bus +
  summaryStoredEvent supersedes main's SessionCompletionHandler DI
  refactor + orphan-reaper fallback.
- src/services/worker/http/routes/SessionRoutes.ts: kept ours — same
  reason; generator .finally() Stop-hook self-clean is a guard for a
  path our blocking endpoint removes.
- src/services/worker/http/routes/CorpusRoutes.ts: merged — added
  security_alert / security_note to ALLOWED_CORPUS_TYPES (feature from
  #2084) while preserving our Zod validateBody schema.

Typecheck: 294 errors (vs 298 pre-merge). No new errors introduced; all
remaining are pre-existing (Component-enum gaps, DOM lib for viewer,
bun:sqlite types).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: address Greptile P2 findings

1) SessionRoutes.handleSessionEnd was the only route handler not wrapped
   in wrapHandler — synchronous exceptions would hang the client rather
   than surfacing as 500s. Wrap it like every other handler.

2) processor.handleToolResult only consumed the session.pendingTools
   entry when the tool_result arrived without a toolName. In the
   split-schema path where tool_result carries both toolName and toolId,
   the entry was never deleted and the map grew for the life of the
   session. Consume the entry whenever toolId is present.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: typing cleanup and viewer tsconfig split for PR feedback

- Add explicit return types for SessionStore query methods
- Exclude src/ui/viewer from root tsconfig, give it its own DOM-typed config
- Add bun to root tsconfig types, plus misc typing tweaks flagged by Greptile
- Rebuilt plugin/scripts/* artifacts

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: address Greptile P2 findings (iter 2)

- PendingMessageStore.transitionMessagesTo: require sessionDbId (drop
  the unscoped-drain branch that would nuke every pending/processing
  row across all sessions if a future caller omitted the filter).
- IngestEventBus.takeRecentSummaryStored: make idempotent — keep the
  cached event until TTL eviction so a retried Stop hook's second
  /api/session/end returns immediately instead of hanging 30 s.
- TranscriptWatcher fs.watch callback: skip full glob scan for paths
  already tailed (JSONL appends fire on every line; only unknown
  paths warrant a rescan).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: call finalizeSession in terminal session paths (Greptile iter 3)

terminateSession and runFallbackForTerminatedSession previously called
SessionCompletionHandler.finalizeSession before removeSessionImmediate;
the refactor dropped those calls, leaving sdk_sessions.status='active'
for every session killed by wall-clock limit, unrecoverable error, or
exhausted fallback chain. The deleted reapStaleSessions interval was
the only prior backstop.

Re-wires finalizeSession (idempotent: marks completed, drains pending,
broadcasts) into both paths; no reaper reintroduced.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: GC failed pending_messages rows at startup (Greptile iter 4)

Plan 07 deleted clearFailed/clearFailedOlderThan as "dead code", but
with the periodic sweep also removed, nothing reaps status='failed'
rows now — they accumulate indefinitely. Since claimNextMessage's
self-healing subquery scans this table, unbounded growth degrades
claim latency over time.

Re-introduces clearFailedOlderThan and calls it once at worker startup
(not a reaper — one-shot, idempotent). 7-day retention keeps enough
history for operator inspection while bounding the table.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: finalize sessions on normal exit; cleanup hoist; share handler (iter 5)

1. startSessionProcessor success branch now calls completionHandler.
   finalizeSession before removeSessionImmediate. Hooks-disabled installs
   (and any Stop hook that fails before POST /api/sessions/complete) no
   longer leave sdk_sessions rows as status='active' forever. Idempotent
   — a subsequent /api/sessions/complete is a no-op.

2. Hoist SessionRoutes.handleSessionEnd cleanup declaration above the
   closures that reference it (TDZ safety; safe at runtime today but
   fragile if timeout ever shrinks).

3. SessionRoutes now receives WorkerService's shared SessionCompletionHandler
   instead of constructing its own — prevents silent divergence if the
   handler ever becomes stateful.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: stop runaway crash-recovery loop on dead sessions

Two distinct bugs were combining to keep a dead session restarting forever:

Bug 1 (uncaught "The operation was aborted."):
  child_process.spawn emits 'error' asynchronously for ENOENT/EACCES/abort
  signal aborts. spawnSdkProcess() never attached an 'error' listener, so
  any async spawn failure became uncaughtException and escaped to the
  daemon-level handler. Attach an 'error' listener immediately after spawn,
  before the !child.pid early-return, so async spawn errors are logged
  (with errno code) and swallowed locally.

Bug 2 (sliding-window limiter never trips on slow restart cadence):
  RestartGuard tripped only when restartTimestamps.length exceeded
  MAX_WINDOWED_RESTARTS (10) within RESTART_WINDOW_MS (60s). With the 8s
  exponential-backoff cap, only ~7-8 restarts fit in the window, so a dead
  session that fail-restart-fail-restart on 8s cycles would loop forever
  (consecutiveRestarts climbing past 30+ in observed logs). Add a
  consecutiveFailures counter that increments on every restart and resets
  only on recordSuccess(). Trip when consecutive failures exceed
  MAX_CONSECUTIVE_FAILURES (5) — meaning 5 restarts with zero successful
  processing in between proves the session is dead. Both guards now run in
  parallel: tight loops still trip the windowed cap; slow loops trip the
  consecutive-failure cap.

Also: when the SessionRoutes path trips the guard, drain pending messages
to 'abandoned' so the session does not reappear in
getSessionsWithPendingMessages and trigger another auto-start cycle. The
worker-service.ts path already does this via terminateSession.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* perf: streamline worker startup and consolidate database connections

1. Database Pooling: Modified DatabaseManager, SessionStore, and SessionSearch to share a single bun:sqlite connection, eliminating redundant file descriptors.
2. Non-blocking Startup: Refactored WorktreeAdoption and Chroma backfill to run in the background (fire-and-forget), preventing them from stalling core initialization.
3. Diagnostic Routes: Added /api/chroma/status and bypassed the initialization guard for health/readiness endpoints to allow diagnostics during startup.
4. Robust Search: Implemented reliable SQLite FTS5 fallback in SearchManager for when Chroma (uvx) fails or is unavailable.
5. Code Cleanup: Removed redundant loopback MCP checks and mangled initialization logic from WorkerService.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: hard-exclude observer-sessions from hooks; bundle migration 29 (#2124)

* fix: hard-exclude observer-sessions from hooks; backfill bundle migrations

Stop hook + SessionEnd hook were storing the SDK observer's own
init/continuation/summary prompts in user_prompts, leaking into the
viewer (meta-observation regression). 25 such rows accumulated.

- shouldTrackProject: hard-reject OBSERVER_SESSIONS_DIR (and its subtree)
  before consulting user-configured exclusion globs.
- summarize.ts (Stop) and session-complete.ts (SessionEnd): early-return
  when shouldTrackProject(cwd) is false, so the observer's own hooks
  cannot bootstrap the worker or queue a summary against the meta-session.
- SessionRoutes: cap user-prompt body at 256 KiB at the session-init
  boundary so a runaway observer prompt cannot blow up storage.
- SessionStore: add migration 29 (UNIQUE(memory_session_id, content_hash)
  on observations) inline so bundled artifacts (worker-service.cjs,
  context-generator.cjs) stay schema-consistent — without it, the
  ON CONFLICT clause in observation inserts throws.
- spawnSdkProcess: stdio[stdin] from 'ignore' to 'pipe' so the
  supervisor can actually feed the observer's stdin.

Also rebuilds plugin/scripts/{worker-service,context-generator}.cjs.

* fix: walk back to UTF-8 boundary on prompt truncation (Greptile P2)

Plain Buffer.subarray at MAX_USER_PROMPT_BYTES can land mid-codepoint,
which the utf8 decoder silently rewrites to U+FFFD. Walk back over any
continuation bytes (0b10xxxxxx) before decoding so the truncated prompt
ends on a valid sequence boundary instead of a replacement character.

* fix: cross-platform observer-dir containment; clarify SDK stdin pipe

claude-review feedback on PR #2124.

- shouldTrackProject: literal `cwd.startsWith(OBSERVER_SESSIONS_DIR + '/')`
  hard-coded a POSIX separator and missed Windows backslash paths plus any
  trailing-slash variance. Switched to a path.relative-based isWithin()
  helper so Windows hook input under observer-sessions\\... is also excluded.
- spawnSdkProcess: added a comment explaining why stdin must be 'pipe' —
  SpawnedSdkProcess.stdin is typed NonNullable and the Claude Agent SDK
  consumes that pipe; 'ignore' would null it and the null-check below
  would tear the child down on every spawn.

* fix: make Stop hook fire-and-forget; remove dead /api/session/end

The Stop hook was awaiting a 35-second long-poll on /api/session/end,
which the worker held open until the summary-stored event fired (or its
30s server-side timeout elapsed). Followed by another await on
/api/sessions/complete. Three sequential awaits, the middle one a 30s
hold — not fire-and-forget despite repeated requests.

The Stop hook now does ONE thing: POST /api/sessions/summarize to
queue the summary work and return. The worker drives the rest async.
Session-map cleanup is performed by the SessionEnd handler
(session-complete.ts), not duplicated here.

- summarize.ts: drop the /api/session/end long-poll and the trailing
  /api/sessions/complete await; ~40 lines removed; unused
  SessionEndResponse interface gone; header comment rewritten.
- SessionRoutes: delete handleSessionEnd, sessionEndSchema, the
  SERVER_SIDE_SUMMARY_TIMEOUT_MS constant, and the /api/session/end
  route registration. Drop the now-unused ingestEventBus and
  SummaryStoredEvent imports.
- ResponseProcessor + shared.ts + worker-utils.ts: update stale
  comments that referenced the dead endpoint. The IngestEventBus is
  left in place dormant (no listeners) for follow-up cleanup so this
  PR stays focused on the blocker.

Bundle artifact (worker-service.cjs) rebuilt via build-and-sync.

Verification:
- grep '/api/session/end' plugin/scripts/worker-service.cjs → 0
- grep 'timeoutMs:35' plugin/scripts/worker-service.cjs → 0
- Worker restarted clean, /api/health ok at pid 92368

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* deps: bump all dependencies to latest including majors

Upgrades: React 18→19, Express 4→5, Zod 3→4, TypeScript 5→6,
@types/node 20→25, @anthropic-ai/claude-agent-sdk 0.1→0.2,
@clack/prompts 0.9→1.2, plus minors. Adds Daily Maintenance section
to CLAUDE.md mandating latest-version policy across manifests.

Express 5 surfaced a race in Server.listen() where the 'error' handler
was attached after listen() was invoked; refactored to use
http.createServer with both 'error' and 'listening' handlers attached
before listen(), restoring port-conflict rejection semantics.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: surface real chroma errors and add deep status probe

Replace the misleading "Vector search failed - semantic search unavailable.
Install uv... restart the worker." string in SearchManager with the actual
exception text from chroma_query_documents. The lying message blamed `uv`
for any failure — even when the real cause was a chroma-mcp transport
timeout, an empty collection, or a dead subprocess.

Also add /api/chroma/status?deep=1 backed by a new
ChromaMcpManager.probeSemanticSearch() that round-trips a real query
(chroma_list_collections + chroma_query_documents) instead of just
checking the stdio handshake. The cheap default path is unchanged.

Includes the diagnostic plan (PLAN-fix-mcp-search.md) and updated test
fixtures for the new structured failure message.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: rebuild worker-service bundle to match merged src

Bundle was stale after the squash merge of #2124 — it still contained
the old "Install uv... semantic search unavailable" string and lacked
probeSemanticSearch. Rebuilt via bun run build-and-sync.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: address coderabbit feedback on PLAN-fix-mcp-search.md

- replace machine-specific /Users/alexnewman absolute paths with portable
  <repo-root> placeholder (MD-style portability)
- add blank lines around the TypeScript fenced block (MD031)
- tag the bare fenced block with `text` (MD040)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-25 13:37:40 -07:00

29 KiB
Raw Blame History

Plan 05 — context-injection-engine (U2 unified renderObservations)

Date: 2026-04-22 Flowchart: PATHFINDER-2026-04-21/05-clean-flowcharts.md section 3.5 (context-injection-engine clean) Before-state: PATHFINDER-2026-04-21/01-flowcharts/context-injection-engine.md Design authority: 05-clean-flowcharts.md Part 1 item #34, Part 2 Decision D4, Part 3 section 3.5.


Dependencies

Upstream: none direct. This plan introduces U2 renderObservations(obs, strategy) — the single traversal that all four existing formatters become strategy configs for.

Downstream:

  • 06-hybrid-search-orchestrationSearchResultStrategy is a renderObservations strategy (05 section 3.6 arrow Fmt -->|markdown| M["renderObservations(results, SearchResultStrategy)"]).
  • 10-knowledge-corpus-builderCorpusDetailStrategy is a renderObservations strategy (05 section 3.11 arrow D --> E["renderObservations(obs, CorpusDetailStrategy)"]).
  • 09-lifecycle-hooks — consumes the single GET /api/session/start endpoint introduced in 05 section 3.1; that endpoint returns {sessionDbId, contextMarkdown, semanticMarkdown} in one payload (Phase 6 below).

Note on 06-implementation-plan.md: Phase 8 of the implementation plan covers the same renderer unification and owns the verification-findings list (V1V20). There is no V-number for renderObservations itself — the audit's item #34 is the sole design reference. Cited here explicitly so downstream agents don't look for a V-number that doesn't exist.


Sources consulted

  1. PATHFINDER-2026-04-21/05-clean-flowcharts.md — full file (607 lines). Section 3.5 at lines 232258; Part 1 item #34 at line 52; Decision D4 at line 75; deletion ledger row for this refactor at line 543 (600 lines formatters → +320 renderer + 4 strategies = 280 net).
  2. PATHFINDER-2026-04-21/06-implementation-plan.md — Phase 8 at lines 368408. No V-number for renderObservations.
  3. PATHFINDER-2026-04-21/01-flowcharts/context-injection-engine.md — before diagram; documents the existing two-path surface (/api/context/inject GET for SQLite context + /api/context/semantic POST for Chroma injection) and the HeaderRenderer/TimelineRenderer/SummaryRenderer/FooterRenderer fan-out.
  4. Live codebase — file:line table below.
  5. Existing 07-plans/ — directory empty at planning time; this is the first plan file.

Live file:line inventory (the four formatters + orchestration)

Concern File Lines Key symbols
AgentFormatter (LLM markdown) src/services/context/formatters/AgentFormatter.ts 227 renderAgentHeader :36, renderAgentLegend :46, renderAgentContextEconomics :75, renderAgentDayHeader :103, renderAgentTableRow :127, renderAgentFullObservation :142, renderAgentSummaryItem :177, renderAgentSummaryField :189, renderAgentPreviouslySection :197, renderAgentFooter :214, renderAgentEmptyState :225, private compactTime :120, private formatHeaderDateTime :21
HumanFormatter (ANSI terminal) src/services/context/formatters/HumanFormatter.ts 238 renderHumanHeader :35, renderHumanLegend :47, renderHumanColumnKey :60, renderHumanContextIndex :72, renderHumanContextEconomics :87, renderHumanDayHeader :116, renderHumanFileHeader :126, renderHumanTableRow :135, renderHumanFullObservation :155, renderHumanSummaryItem :186, renderHumanSummaryField :200, renderHumanPreviouslySection :208, renderHumanFooter :225, renderHumanEmptyState :236, private formatHeaderDateTime :20
ResultFormatter (search markdown, class) src/services/worker/search/ResultFormatter.ts 301 class ResultFormatter :21, formatSearchResults :25 (the top-level walker), combineResults :115, formatSearchTableHeader :141, formatTableHeader :149, formatObservationSearchRow :157, formatSessionSearchRow :178, formatPromptSearchRow :199, formatObservationIndex :221, formatSessionIndex :237, formatPromptIndex :250, estimateReadTokens :264, formatChromaFailureMessage :275, formatSearchTips :288
CorpusRenderer (corpus detail, class) src/services/worker/knowledge/CorpusRenderer.ts 133 class CorpusRenderer :10, renderCorpus :14 (the top-level walker), renderObservation :39 (private, the per-obs detail renderer), estimateTokens :90, generateSystemPrompt :97
Orchestrator src/services/context/ContextBuilder.ts 186 generateContext :130, buildContextOutput :80, initializeDatabase :49, renderEmptyState :73 (calls both empty-state functions)
Day-grouping walker (shared today) src/services/context/sections/TimelineRenderer.ts 183 groupTimelineByDay :21, renderTimeline :168, renderDayTimeline :151 (forHuman branch :159), renderDayTimelineAgent :56, renderDayTimelineHuman :97, private getDetailField :46
Section dispatch (forHuman branching) src/services/context/sections/HeaderRenderer.ts 61 renderHeader :15 (branches forHuman for 5 sub-sections)
Section dispatch src/services/context/sections/SummaryRenderer.ts 65 shouldShowSummary :15, renderSummaryFields :46 (branches forHuman)
Section dispatch src/services/context/sections/FooterRenderer.ts 42 renderPreviouslySection :15 (branches forHuman), renderFooter :28 (branches forHuman)
Token economics (KEEP) src/services/context/TokenCalculator.ts 78 calculateTokenEconomics, formatObservationTokenDisplay, shouldShowContextEconomics
Mode filtering (KEEP) src/services/domain/ModeManager.ts 266 ModeManager.getInstance(), getActiveMode, getTypeIcon, getWorkEmoji
HTTP caller (today) src/services/worker/http/routes/SearchRoutes.ts handleContextInject :209 (GET, dynamically imports context-generator.generateContext), handleSemanticContext :258 (POST, inlines its own formatter at :286293)

Top-level LoC of the four formatters: 227 + 238 + 301 + 133 = 899 lines. Section dispatch files (Header/Summary/Footer/Timeline) add another 61 + 65 + 42 + 183 = 351 lines of forHuman branching that collapse once strategies own the shape.

Copy-ready: the shared "walk" all four formatters share

Every formatter does some subset of the same four-step traversal. The invariants below become the body of renderObservations:

  1. Optional header: project/title/date line + legend + economics. Today: HeaderRenderer.renderHeader (HeaderRenderer.ts:15) + ResultFormatter.formatSearchResults :53 + CorpusRenderer.renderCorpus :17. → Strategy flag: header: 'context' | 'search' | 'corpus' | 'none'.
  2. Group and iterate — the core walk. Today: groupTimelineByDay (TimelineRenderer.ts:21) for agent/human paths; groupByDate (shared/timeline-formatting.ts) + file-bucketing at ResultFormatter.ts:5672 for search; flat iteration for corpus at CorpusRenderer.ts:2831. → Strategy flag: grouping: 'by-day' | 'by-day-then-file' | 'none'.
  3. Per-observation row — either compact line or full-detail block. Today: renderAgentTableRow/renderAgentFullObservation, renderHumanTableRow/renderHumanFullObservation, formatObservationSearchRow/formatObservationIndex, CorpusRenderer.renderObservation. → Strategy flag: density: 'compact' | 'table' | 'full-detail' + colorize: boolean + columns: [...] + showTokens: {read, work}.
  4. Optional tail: summary fields + previously section + footer tips. Today: SummaryRenderer.renderSummaryFields, FooterRenderer.renderPreviouslySection, FooterRenderer.renderFooter, ResultFormatter.formatSearchTips. → Strategy flag: tail: 'context' | 'search-tips' | 'corpus-stats' | 'none'.

The five constants all four share: ModeManager.getTypeIcon(type) for the type emoji, formatTime(epoch) / formatDate / formatDateTime from shared/timeline-formatting.ts, extractFirstFile for file extraction, parseJsonArray for facts parsing, and the title-fallback rule obs.title || 'Untitled'. These move unchanged into the renderer.

Confidence + gaps

High confidence:

  • File inventory, LoC, and symbol-level API of the four formatters.
  • That all four read the same shape (Observation with id/title/narrative/facts/type/created_at_epoch/files_modified/files_read).
  • Decision D4's four-strategy ceiling: Agent, Human, SearchResult, CorpusDetail — no others.

Gaps / risks:

  • ANSI-color preservation in HumanContextStrategy is a regression surface. HumanFormatter.ts uses colors.bright, colors.cyan, colors.gray, colors.dim, colors.yellow, colors.magenta, colors.green, colors.blue imported from ../types.js. Any divergence — including trailing spaces around ANSI wrappers, padding in renderHumanTableRow at :145 (' '.repeat(time.length) when showTime=false), and the ×60 separator at :39 and :237 — is a user-visible regression. Phase 8 fixtures must assert byte equality including escape sequences.
  • ResultFormatter has two row formats (formatSearchTableHeader without Work column + formatTableHeader with Work column). SearchResultStrategy must support both, gated by a columns array — otherwise index-rendering callers (formatObservationIndex used elsewhere) regress silently. Grep during Phase 4 to enumerate callers before choosing defaults.
  • Semantic-injection POST handler at SearchRoutes.ts:286293 implements its own mini-formatter (## Relevant Past Work (semantic match) header + ### title (date) + narrative). Anti-pattern E forbids this post-refactor. Phase 6 folds it into a SearchResultStrategy variant or a narrow SemanticInjectStrategy (still counts as a SearchResult strategy per Decision D4's four-total rule — treat this as a strategy flag, not a fifth strategy).

Phase contract (applies to every phase)

Every phase below carries:

  • (a) What: "Copy from …" instructions. The four existing formatters become four strategy configs feeding ONE renderObservations.
  • (b) Docs: 05-clean-flowcharts.md section 3.5 + Decision D4 + live file:line for each of the four formatters (table above).
  • (c) Verification: unit tests per strategy against a fixed Observation[] fixture; byte-for-byte match against the old formatter's output for identical inputs.
  • (d) Anti-pattern guards:
    • Guard A (audit Part 2): only four strategies — AgentContextStrategy, HumanContextStrategy, SearchResultStrategy, CorpusDetailStrategy. Any fifth strategy fails review.
    • Guard E (audit Part 2): single renderer path. No caller may implement its own walker. Grep check (Phase 8) enforces.

Phase 1 — Extract common traversal into renderObservations(obs, strategy)

(a) What: Create a new module src/services/rendering/renderObservations.ts (new folder src/services/rendering/ so no caller is forced to import across feature boundaries). Copy the walk from the three existing walkers:

  • Day grouping: from TimelineRenderer.groupTimelineByDay (src/services/context/sections/TimelineRenderer.ts:21).
  • Day-then-file grouping: from ResultFormatter.formatSearchResults (src/services/worker/search/ResultFormatter.ts:5672).
  • Flat iteration: from CorpusRenderer.renderCorpus (src/services/worker/knowledge/CorpusRenderer.ts:2831).

Signature:

export interface RenderStrategy {
  name: 'agent-context' | 'human-context' | 'search-result' | 'corpus-detail';
  header?: (ctx: HeaderCtx) => string[];
  grouping: 'by-day' | 'by-day-then-file' | 'none';
  renderGroupHeader?: (key: string) => string[];
  renderSubgroupHeader?: (key: string) => string[]; // e.g., file within day
  renderSummaryItem?: (s: SummaryItem, time: string) => string[];
  renderRow: (obs: Observation, ctx: RowCtx) => string;
  renderFullObservation?: (obs: Observation, ctx: RowCtx) => string[];
  tail?: (ctx: TailCtx) => string[];
  emptyState?: (ctx: HeaderCtx) => string;
}
export function renderObservations(
  items: Array<Observation | SummaryItem>,
  strategy: RenderStrategy,
  ctx: RenderContext,
): string;

The orchestrator owns: (1) token budget enforcement (from calculateTokenEconomics, TokenCalculator.ts:25), (2) mode filtering (from ModeManager.getActiveMode(), ModeManager.ts:15), (3) full-vs-compact selection (from getFullObservationIds in ObservationCompiler.ts). Strategies do not re-implement any of this.

(b) Docs: 05 section 3.5 lines 234251; Decision D4 line 75. File:line for all four formatters per inventory table.

(c) Verification:

  • Unit tests: tests/services/rendering/renderObservations.test.ts — three tests, one per grouping mode, with a synthetic Observation[] of 5 items across 2 days and 3 files.
  • Build check: npm run build-and-sync passes after new module is in place (not yet wired).

(d) Anti-pattern guards: A — stop at four strategy names (compile-time name union enforces). E — module is the single renderer; callers will switch to it in Phase 6, Phase 7 deletes the old paths.


Phase 2 — AgentContextStrategy from AgentFormatter

(a) What: Create src/services/context/strategies/AgentContextStrategy.ts and copy the output-shape bytes from AgentFormatter.ts into strategy callbacks:

  • headerrenderAgentHeader (:36) + renderAgentLegend (:46) + renderAgentColumnKey (:61, no-op) + renderAgentContextIndex (:68, no-op) + renderAgentContextEconomics (:75) composed in order per HeaderRenderer.renderHeader :15.
  • grouping: 'by-day'; renderGroupHeaderrenderAgentDayHeader (:103).
  • renderSummaryItemrenderAgentSummaryItem (:177).
  • renderRowrenderAgentTableRow (:127); renderFullObservationrenderAgentFullObservation (:142).
  • tailrenderAgentSummaryField (:189) for each of the four fields + renderAgentPreviouslySection (:197) + renderAgentFooter (:214).
  • emptyStaterenderAgentEmptyState (:225).

The shared formatHeaderDateTime (:21) and compactTime (:120) move into src/services/rendering/render-helpers.ts or stay inline in the strategy (two callers — no DRY pressure yet).

(b) Docs: 05 section 3.5 arrow Strategy -->|AgentContextStrategy| AgentOut["Compact markdown for LLM"] (line 244); inventory row for AgentFormatter.ts above.

(c) Verification: snapshot test — feed the same Observation[] fixture to (i) the old buildContextOutput(..., forHuman=false) and (ii) renderObservations(items, AgentContextStrategy, ctx); assert string equality. Zero-tolerance: LLM context is consumed by models — any whitespace change shifts KV-cache and can surface as behavioral regressions.

(d) Anti-pattern guards: A — strategy file defines the config object only, no walker. E — no custom grouping code; reuse Phase 1's by-day grouping.


Phase 3 — HumanContextStrategy from HumanFormatter (preserves ANSI)

(a) What: Create src/services/context/strategies/HumanContextStrategy.ts. Copy output-shape bytes from HumanFormatter.ts:

  • headerrenderHumanHeader (:35) + renderHumanLegend (:47) + renderHumanColumnKey (:60) + renderHumanContextIndex (:72) + renderHumanContextEconomics (:87).
  • grouping: 'by-day-then-file'; renderGroupHeaderrenderHumanDayHeader (:116); renderSubgroupHeaderrenderHumanFileHeader (:126).
  • renderSummaryItemrenderHumanSummaryItem (:186).
  • renderRowrenderHumanTableRow (:135) — preserves colors.dim, colors.cyan, colors.bright, colors.reset escapes and the '.repeat(time.length) padding for showTime=false (see HumanFormatter.ts:145).
  • renderFullObservationrenderHumanFullObservation (:155).
  • tailrenderHumanSummaryField (:200) per field (with its per-field ANSI color from SummaryRenderer.ts:5256blue/yellow/green/magenta) + renderHumanPreviouslySection (:208) + renderHumanFooter (:225).
  • emptyStaterenderHumanEmptyState (:236) — note the literal ×60 separator and the \n layout.

ANSI colors import from src/services/context/types.js stays inside this strategy only. The renderer core is ANSI-agnostic.

(b) Docs: 05 section 3.5 arrow Strategy -->|HumanContextStrategy| HumanOut["ANSI-colored terminal"] (line 245); inventory row for HumanFormatter.ts; D4 explicit about "columns/density/grouping" plus colorize per Phase 8 sketch in 06-implementation-plan.md line 385.

(c) Verification: snapshot test with explicit ANSI-escape comparison. Fixture MUST include: a no-time continuation row (to exercise the ' '.repeat(time.length) padding at :145), a full-observation row with facts (exercises :167177), and the empty-state path (exercises :237). Assert raw buffer equality — not stripped-ANSI equality. Confidence gap: this is the highest regression risk in the plan (see Gaps above).

(d) Anti-pattern guards: A — one human strategy. E — no duplicate ANSI wrapping helper; colors constants travel with the strategy.


Phase 4 — SearchResultStrategy from ResultFormatter

(a) What: Create src/services/worker/search/strategies/SearchResultStrategy.ts. Copy from ResultFormatter.ts:

  • header ← the Found N result(s) matching "…" line at :53 (parameterized on query + counts).
  • grouping: 'by-day-then-file'; renderGroupHeader ← day label ### ${day} (:57); renderSubgroupHeader**${file}** + formatSearchTableHeader :141 (the | ID | Time | T | Title | Read | header).
  • renderRow dispatches on item kind: formatObservationSearchRow (:157), formatSessionSearchRow (:178), formatPromptSearchRow (:199). The lastTime threading for " continuation stays in the renderer's RowCtx (from Phase 1).
  • tailformatSearchTips (:288) appended when not empty.
  • emptyStateNo results found matching "${query}" (:38) / formatChromaFailureMessage (:275) gated by a new ctx.chromaFailed flag.

The index-column variant (formatObservationIndex :221 etc., with the Work column) becomes a strategy option columns: ['id','time','type','title','read'] | ['id','time','type','title','read','work']. Before choosing a default, grep Phase 4 callers to enumerate usages — confidence gap noted above.

(b) Docs: 05 section 3.6 line 281 (renderObservations(results, SearchResultStrategy)); inventory row for ResultFormatter.ts. Cross-reference: 06-hybrid-search-orchestration plan (downstream) will consume this strategy.

(c) Verification: feed the same SearchResults fixture to ResultFormatter.formatSearchResults and to renderObservations(combined, SearchResultStrategy, ctx); assert byte equality including the date-group headers, file headers, table pipe characters, and trailing blank lines.

(d) Anti-pattern guards: A — single SearchResultStrategy; if semantic-injection handler at SearchRoutes.ts:286293 needs a different shape, it becomes a flag on this strategy (variant: 'table' | 'injection'), not a fifth strategy. E — delete any caller that still walks results.observations.map(...) by hand (Phase 7 grep).


Phase 5 — CorpusDetailStrategy from CorpusRenderer

(a) What: Create src/services/worker/knowledge/strategies/CorpusDetailStrategy.ts. Copy from CorpusRenderer.ts:

  • headerCorpusRenderer.renderCorpus :1426 (the # Knowledge Corpus: …, description, stats block, --- divider). Parameterized on CorpusFile.name/description/stats.
  • grouping: 'none' — corpus walks flat (:2831).
  • renderFullObservationCorpusRenderer.renderObservation (:39) — full narrative, facts list, concepts, files_read, files_modified. No compact row form; every observation renders at full detail (per CorpusRenderer.ts:5).
  • tail: undefined — corpus has no tail beyond the trailing ---.

generateSystemPrompt (:97) is not part of the strategy — it's a separate function on the corpus feature that stays where it is. estimateTokens (:90) already moves to shared/timeline-formatting.ts as estimateTokens (it's already there per ResultFormatter.ts:17 import); delete the duplicate at CorpusRenderer.ts:90.

(b) Docs: 05 section 3.11 line 457 (renderObservations(obs, CorpusDetailStrategy)); inventory row for CorpusRenderer.ts. Cross-reference: 10-knowledge-corpus-builder plan (downstream) consumes this strategy.

(c) Verification: feed the same CorpusFile to CorpusRenderer.renderCorpus and to renderObservations(corpus.observations, CorpusDetailStrategy, {corpus}); assert byte equality. Important: corpus output is a prompt — whitespace divergence changes prompt-cache hit rate on the SDK side (see 05 section 3.11 cost note, line 476).

(d) Anti-pattern guards: A — single CorpusDetailStrategy. E — KnowledgeAgent and CorpusBuilder both route through it; no direct CorpusRenderer instantiation post-Phase 7.


Phase 6 — Switch ContextBuilder.generateContext + /api/session/start handler to renderObservations

(a) What:

  1. Rewrite src/services/context/ContextBuilder.ts:
    • buildContextOutput :80 collapses to: resolve strategy = forHuman ? HumanContextStrategy : AgentContextStrategy, build RenderContext (economics, fullObservationIds, priorMessages, mostRecentSummary), call renderObservations(timeline, strategy, ctx). The explicit renderHeader/renderTimeline/renderSummaryFields/renderPreviouslySection/renderFooter fan-out at :95119 deletes in favor of strategy-owned header/renderGroupHeader/renderRow/tail.
    • renderEmptyState :73 collapses to strategy.emptyState?.(ctx).
    • generateContext :130 signature is unchanged — external callers see identical input/output.
  2. Add the new /api/session/start handler (per 05 section 3.1 line 95 GET /api/session/start?project=…). Owned by lifecycle-hooks plan (09); this plan lands the renderer-facing side: one call into generateContext(forHuman:false) for contextMarkdown, one call into SearchOrchestrator.search(query, limit=5) + renderObservations(results, SearchResultStrategy, {variant:'injection'}) for semanticMarkdown. Both served from a single response body.
  3. Delete the inline mini-formatter at SearchRoutes.ts:286293 (the ## Relevant Past Work … block); route through SearchResultStrategy.

(b) Docs: 05 section 3.5 entry arrows lines 236242; 05 section 3.1 lines 95 + 100 (one /api/session/start returns ctx + semantic); 06 plan Phase 8 lines 391394.

(c) Verification:

  • End-to-end byte-identity: capture the pre-refactor output of GET /api/context/inject?projects=X&colors=true and …&colors=false for a seeded DB; after the switch, curl the same and diff. Zero diff.
  • New /api/session/start returns {sessionDbId, contextMarkdown, semanticMarkdown} (per 05 section 3.1 line 100) with the two markdown fields byte-matching the previous two-endpoint responses.
  • npm run build-and-sync passes.

(d) Anti-pattern guards: A — no new strategies introduced. E — SearchRoutes.handleSemanticContext either deleted (covered by /api/session/start) or its body becomes a single renderObservations(…, SearchResultStrategy, {variant:'injection'}) call — no more inline lines.push('### …').


Phase 7 — Delete the four old formatter files; update imports

(a) What:

  1. rm src/services/context/formatters/AgentFormatter.ts (227 lines).
  2. rm src/services/context/formatters/HumanFormatter.ts (238 lines).
  3. rm src/services/worker/search/ResultFormatter.ts (301 lines).
  4. rm src/services/worker/knowledge/CorpusRenderer.ts (133 lines).
  5. Delete src/services/context/sections/{HeaderRenderer,TimelineRenderer,SummaryRenderer,FooterRenderer}.ts — their forHuman branching is now owned by strategies. ObservationCompiler.ts keeps the data-loading helpers (queryObservations, buildTimeline, getFullObservationIds — these feed the renderer, not part of the deletion).
  6. Update imports at: ContextBuilder.ts (switch to renderObservations + strategies), SearchManager.ts / SearchRoutes.ts (switch to SearchResultStrategy), KnowledgeAgent.ts / CorpusBuilder.ts (switch to CorpusDetailStrategy). Grep for every import … from '.*AgentFormatter|HumanFormatter|ResultFormatter|CorpusRenderer' — expect zero after this phase.

Net line impact: deletes 227 + 238 + 301 + 133 + 61 + 183 + 65 + 42 = 1,250 lines. Adds ~320 for renderObservations + 4 strategies + shared helpers. Net ≈ 930 lines — beats the audit's estimate at 05 line 543 (280 net) because the forHuman branching in the section renderers was not counted there.

(b) Docs: 05 section 3.5 "Deleted" list lines 253256; 06 plan Phase 8 verification line 397.

(c) Verification:

  • grep -rn "AgentFormatter\|HumanFormatter\|ResultFormatter\|CorpusRenderer" src/ tests/ → zero hits.
  • grep -rn "renderHeader\|renderTimeline\|renderSummaryFields\|renderPreviouslySection\|renderFooter" src/services/context/sections/ → zero hits (directory removed).
  • npx tsc --noEmit passes.
  • npm run build-and-sync passes.

(d) Anti-pattern guards: D — no compatibility shim re-exports old names. E — single walker; grep for (const .* of .*observations) in src/services/worker/ and src/services/context/ should only match inside renderObservations.ts (and test fixtures).


Phase 8 — Verification: byte-identical output for all four paths

(a) What: Add four golden-file fixtures under tests/fixtures/rendering/:

  • agent-context.txt — output of old generateContext(input, forHuman=false) captured before Phase 6.
  • human-context.ansi — raw bytes including ANSI escapes from old generateContext(input, forHuman=true).
  • search-result.md — output of old ResultFormatter.formatSearchResults(results, "test query").
  • corpus-detail.md — output of old CorpusRenderer.renderCorpus(corpus).

Capture on the branch tip before Phase 1 so the baseline is pre-refactor. Each phase's unit test (Phases 25) diffs against its golden file.

A final integration test runs the four renderers end-to-end against a seeded DB and diffs all four outputs simultaneously.

(b) Docs: 06 plan Phase 8 verification lines 396399 ("Snapshot tests: for each strategy, feed the same fixture Observation[] and assert output is byte-equal to the old formatter's output").

(c) Verification:

  • All four snapshot tests green.
  • Grep audit: grep -rn "setInterval\|formatObservation\|renderObservation" src/ | grep -v renderObservations.ts | grep -v test — zero hits outside the one renderer.
  • SessionStart end-to-end: trigger a real Claude Code session with npm run build-and-sync; Agent context in the session + ANSI context in terminal both diff-clean against pre-refactor capture.
  • Chroma corpus query test: build a corpus, query it 3× within 5 minutes, assert cache_read_input_tokens > 0 on SDK response (proves corpus prompt bytes are stable, per 05 section 3.11 cost note).

(d) Anti-pattern guards: A — tests enforce the four-strategy ceiling by unioned name type. E — the grep audit above is the single-walker check.


Constraints summary

  • Zero behavior change for LLM (Agent) output bytes and human terminal ANSI bytes. Enforced by Phase 8 golden files.
  • Token-budget logic stays in the orchestrator (calculateTokenEconomics at TokenCalculator.ts:25; getFullObservationIds at ObservationCompiler.ts). Strategies receive computed RowCtx.isFull, never re-decide.
  • Mode filtering stays in the orchestrator (ModeManager.getActiveMode() at ModeManager.ts:15). Strategies receive filtered Observation[].
  • ANSI color codes preserved: all colors.* literals from src/services/context/types.js travel into HumanContextStrategy only. The renderer core is ANSI-agnostic.
  • Four strategies, no more: AgentContextStrategy, HumanContextStrategy, SearchResultStrategy, CorpusDetailStrategy. Variants live as strategy config flags.

Phase count

8 phases.

  • Phase 1: extract renderer.
  • Phase 2: AgentContextStrategy.
  • Phase 3: HumanContextStrategy (ANSI).
  • Phase 4: SearchResultStrategy.
  • Phase 5: CorpusDetailStrategy.
  • Phase 6: wire ContextBuilder.generateContext + /api/session/start.
  • Phase 7: delete old formatters + section renderers.
  • Phase 8: byte-identical verification.

Blast radius + estimated LoC

  • Files deleted: 8 (four formatters + four section renderers).
  • Files created: ~6 (renderObservations.ts + 4 strategy files + shared helpers).
  • Lines deleted: ~1,250 (AgentFormatter 227 + HumanFormatter 238 + ResultFormatter 301 + CorpusRenderer 133 + HeaderRenderer 61 + TimelineRenderer 183 + SummaryRenderer 65 + FooterRenderer 42).
  • Lines added: ~320 (renderer + four strategies, per audit estimate at 05 line 543).
  • Net: 930 lines, ~3.3× the audit's row-level estimate of 280, once the forHuman branching in *Renderer.ts section files is counted.

Risk: lowest of the cleanup plan (pure reorganization, no behavior change). Snapshot tests are the safety net.