94d592f212
* docs: pathfinder refactor corpus + Node 20 preflight
Adds the PATHFINDER-2026-04-22 principle-driven refactor plan (11 docs,
cross-checked PASS) plus the exploratory PATHFINDER-2026-04-21 corpus
that motivated it. Bumps engines.node to >=20.0.0 per the ingestion-path
plan preflight (recursive fs.watch). Adds the pathfinder skill.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: land PATHFINDER Plan 01 — data integrity
Schema, UNIQUE constraints, self-healing claim, Chroma upsert fallback.
- Phase 1: fresh schema.sql regenerated at post-refactor shape.
- Phase 2: migrations 23+24 — rebuild pending_messages without
started_processing_at_epoch; UNIQUE(session_id, tool_use_id);
UNIQUE(memory_session_id, content_hash) on observations; dedup
duplicate rows before adding indexes.
- Phase 3: claimNextMessage rewritten to self-healing query using
worker_pid NOT IN live_worker_pids; STALE_PROCESSING_THRESHOLD_MS
and the 60-s stale-reset block deleted.
- Phase 4: DEDUP_WINDOW_MS and findDuplicateObservation deleted;
observations.insert now uses ON CONFLICT DO NOTHING.
- Phase 5: failed-message purge block deleted from worker-service
2-min interval; clearFailedOlderThan method deleted.
- Phase 6: repairMalformedSchema and its Python subprocess repair
path deleted from Database.ts; SQLite errors now propagate.
- Phase 7: Chroma delete-then-add fallback gated behind
CHROMA_SYNC_FALLBACK_ON_CONFLICT env flag as bridge until
Chroma MCP ships native upsert.
- Phase 8: migration 19 no-op block absorbed into fresh schema.sql.
Verification greps all return 0 matches. bun test tests/sqlite/
passes 63/63. bun run build succeeds.
Plan: PATHFINDER-2026-04-22/01-data-integrity.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: land PATHFINDER Plan 02 — process lifecycle
OS process groups replace hand-rolled reapers. Worker runs until
killed; orphans are prevented by detached spawn + kill(-pgid).
- Phase 1: src/services/worker/ProcessRegistry.ts DELETED. The
canonical registry at src/supervisor/process-registry.ts is the
sole survivor; SDK spawn site consolidated into it via new
createSdkSpawnFactory/spawnSdkProcess/getSdkProcessForSession/
ensureSdkProcessExit/waitForSlot helpers.
- Phase 2: SDK children spawn with detached:true + stdio:
['ignore','pipe','pipe']; pgid recorded on ManagedProcessInfo.
- Phase 3: shutdown.ts signalProcess teardown uses
process.kill(-pgid, signal) on Unix when pgid is recorded;
Windows path unchanged (tree-kill/taskkill).
- Phase 4: all reaper intervals deleted — startOrphanReaper call,
staleSessionReaperInterval setInterval (including the co-located
WAL checkpoint — SQLite's built-in wal_autocheckpoint handles
WAL growth without an app-level timer), killIdleDaemonChildren,
killSystemOrphans, reapOrphanedProcesses, reapStaleSessions, and
detectStaleGenerator. MAX_GENERATOR_IDLE_MS and MAX_SESSION_IDLE_MS
constants deleted.
- Phase 5: abandonedTimer — already 0 matches; primary-path cleanup
via generatorPromise.finally() already lives in worker-service
startSessionProcessor and SessionRoutes ensureGeneratorRunning.
- Phase 6: evictIdlestSession and its evict callback deleted from
SessionManager. Pool admission gates backpressure upstream.
- Phase 7: SDK-failure fallback — SessionManager has zero matches
for fallbackAgent/Gemini/OpenRouter. Failures surface to hooks
via exit code 2 through SessionRoutes error mapping.
- Phase 8: ensureWorkerRunning in worker-utils.ts rewritten to
lazy-spawn — consults isWorkerPortAlive (which gates
captureProcessStartToken for PID-reuse safety via commit
99060bac), then spawns detached with unref(), then
waitForWorkerPort({ attempts: 3, backoffMs: 250 }) hand-rolled
exponential backoff 250→500→1000ms. No respawn npm dep.
- Phase 9: idle self-shutdown — zero matches for
idleCheck/idleTimeout/IDLE_MAX_MS/idleShutdown. Worker exits
only on external SIGTERM via supervisor signal handlers.
Three test files that exercised deleted code removed:
tests/worker/process-registry.test.ts,
tests/worker/session-lifecycle-guard.test.ts,
tests/services/worker/reap-stale-sessions.test.ts.
Pass count: 1451 → 1407 (-44), all attributable to deleted test
files. Zero new failures. 31 pre-existing failures remain
(schema-repair suite, logger-usage-standards, environmental
openclaw / plugin-distribution) — none introduced by Plan 02.
All 10 verification greps return 0. bun run build succeeds.
Plan: PATHFINDER-2026-04-22/02-process-lifecycle.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: land PATHFINDER Plan 04 (narrowed) — search fail-fast
Phases 3, 5, 6 only. Plan-doc inaccuracies for phases 1/2/4/7/8/9
deferred for plan reconciliation:
- Phase 1/2: ObservationRow type doesn't exist; the four
"formatters" operate on three incompatible types.
- Phase 4: RECENCY_WINDOW_MS already imported from
SEARCH_CONSTANTS at every call site.
- Phase 7: getExistingChromaIds is NOT @deprecated and has an
active caller in ChromaSync.backfillMissingSyncs.
- Phase 8: estimateTokens already consolidated.
- Phase 9: knowledge-corpus rewrite blocked on PG-3
prompt-caching cost smoke test.
Phase 3 — Delete SearchManager.findByConcept/findByFile/findByType.
SearchRoutes handlers (handleSearchByConcept/File/Type) now call
searchManager.getOrchestrator().findByXxx() directly via new
getter accessors on SearchManager. ~250 LoC deleted.
Phase 5 — Fail-fast Chroma. Created
src/services/worker/search/errors.ts with ChromaUnavailableError
extends AppError(503, 'CHROMA_UNAVAILABLE'). Deleted
SearchOrchestrator.executeWithFallback's Chroma-failed
SQLite-fallback branch; runtime Chroma errors now throw 503.
"Path 3" (chromaSync was null at construction — explicit-
uninitialized config) preserved as legitimate empty-result state
per plan text. ChromaSearchStrategy.search no longer wraps in
try/catch — errors propagate.
Phase 6 — Delete HybridSearchStrategy three try/catch silent
fallback blocks (findByConcept, findByType, findByFile) at lines
~82-95, ~120-132, ~161-172. Removed `fellBack` field from
StrategySearchResult type and every return site
(SQLiteSearchStrategy, BaseSearchStrategy.emptyResult,
SearchOrchestrator).
Tests updated (Principle 7 — delete in same PR):
- search-orchestrator.test.ts: "fall back to SQLite" rewritten
as "throw ChromaUnavailableError (HTTP 503)".
- chroma/hybrid/sqlite-search-strategy tests: rewritten to
rejects.toThrow; removed fellBack assertions.
Verification: SearchManager.findBy → 0; fellBack → 0 in src/.
bun test tests/worker/search/ → 122 pass, 0 fail.
bun test (suite-wide) → 1407 pass, baseline maintained, 0 new
failures. bun run build succeeds.
Plan: PATHFINDER-2026-04-22/04-read-path.md (Phases 3, 5, 6)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: land PATHFINDER Plan 03 — ingestion path
Fail-fast parser, direct in-process ingest, recursive fs.watch,
DB-backed tool pairing. Worker-internal HTTP loopback eliminated.
- Phase 0: Created src/services/worker/http/shared.ts exporting
ingestObservation/ingestPrompt/ingestSummary as direct
in-process functions plus ingestEventBus (Node EventEmitter,
reusing existing pattern — no third event bus introduced).
setIngestContext wires the SessionManager dependency from
worker-service constructor.
- Phase 1: src/sdk/parser.ts collapsed to one parseAgentXml
returning { valid:true; kind: 'observation'|'summary'; data }
| { valid:false; reason: string }. Inspects root element;
<skip_summary reason="…"/> is a first-class summary case
with skipped:true. NEVER returns undefined. NEVER coerces.
- Phase 2: ResponseProcessor calls parseAgentXml exactly once,
branches on the discriminated union. On invalid → markFailed
+ logger.warn(reason). On observation → ingestObservation.
On summary → ingestSummary then emit summaryStoredEvent
{ sessionId, messageId } (consumed by Plan 05's blocking
/api/session/end).
- Phase 3: Deleted consecutiveSummaryFailures field
(ResponseProcessor + SessionManager + worker-types) and
MAX_CONSECUTIVE_SUMMARY_FAILURES constant. Circuit-breaker
guards and "tripped" log lines removed.
- Phase 4: coerceObservationToSummary deleted from sdk/parser.ts.
- Phase 5: src/services/transcripts/watcher.ts rescan setInterval
replaced with fs.watch(transcriptsRoot, { recursive: true,
persistent: true }) — Node 20+ recursive mode.
- Phase 6: src/services/transcripts/processor.ts pendingTools
Map deleted. tool_use rows insert with INSERT OR IGNORE on
UNIQUE(session_id, tool_use_id) (added by Plan 01). New
pairToolUsesByJoin query in PendingMessageStore for read-time
pairing (UNIQUE INDEX provides idempotency; explicit consumer
not yet wired).
- Phase 7: HTTP loopback at processor.ts:252 replaced with
direct ingestObservation call. maybeParseJson silent-passthrough
rewritten to fail-fast (throws on malformed JSON).
- Phase 8: src/utils/tag-stripping.ts countTags + stripTagsInternal
collapsed into one alternation regex, single-pass over input.
- Phase 9: src/utils/transcript-parser.ts (dead TranscriptParser
class) deleted. The active extractLastMessage at
src/shared/transcript-parser.ts:41-144 is the sole survivor.
Tests updated (Principle 7 — same-PR delete):
- tests/sdk/parser.test.ts + parse-summary.test.ts: rewritten
to assert discriminated-union shape; coercion-specific
scenarios collapse into { valid:false } assertions.
- tests/worker/agents/response-processor.test.ts: circuit-breaker
describe block skipped; non-XML/empty-response tests assert
fail-fast markFailed behavior.
Verification: every grep returns 0. transcript-parser.ts deleted.
bun run build succeeds. bun test → 1399 pass / 28 fail / 7 skip
(net -8 pass = the 4 retired circuit-breaker tests + 4 collapsed
parser cases). Zero new failures vs baseline.
Deferred (out of Plan 03 scope, will land in Plan 06): SessionRoutes
HTTP route handlers still call sessionManager.queueObservation
inline rather than the new shared helpers — the helpers are ready,
the route swap is mechanical and belongs with the Zod refactor.
Plan: PATHFINDER-2026-04-22/03-ingestion-path.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: land PATHFINDER Plan 05 — hook surface
Worker-call plumbing collapsed to one helper. Polling replaced by
server-side blocking endpoint. Fail-loud counter surfaces persistent
worker outages via exit code 2.
- Phase 1: plugin/hooks/hooks.json — three 20-iteration `for i in
1..20; do curl -sf .../health && break; sleep 0.1; done` shell
retry wrappers deleted. Hook commands invoke their bun entry
point directly.
- Phase 2: src/shared/worker-utils.ts — added
executeWithWorkerFallback<T>(url, method, body) returning
T | { continue: true; reason?: string }. All 8 hook handlers
(observation, session-init, context, file-context, file-edit,
summarize, session-complete, user-message) rewritten to use
it instead of duplicating the ensureWorkerRunning →
workerHttpRequest → fallback sequence.
- Phase 3: blocking POST /api/session/end in SessionRoutes.ts
using validateBody + sessionEndSchema (z.object({sessionId})).
One-shot ingestEventBus.on('summaryStoredEvent') listener,
30 s timer, req.aborted handler — all share one cleanup so
the listener cannot leak. summarize.ts polling loop, plus
MAX_WAIT_FOR_SUMMARY_MS / POLL_INTERVAL_MS constants, deleted.
- Phase 4: src/shared/hook-settings.ts — loadFromFileOnce()
memoizes SettingsDefaultsManager.loadFromFile per process.
Per-handler settings reads collapsed.
- Phase 5: src/shared/should-track-project.ts — single exclusion
check entry; isProjectExcluded no longer referenced from
src/cli/handlers/.
- Phase 6: cwd validation pushed into adapter normalizeInput
(all 6 adapters: claude-code, cursor, raw, gemini-cli,
windsurf). New AdapterRejectedInput error in
src/cli/adapters/errors.ts. Handler-level isValidCwd checks
deleted from file-edit.ts and observation.ts. hook-command.ts
catches AdapterRejectedInput → graceful fallback.
- Phase 7: session-init.ts conditional initAgent guard deleted;
initAgent is idempotent. tests/hooks/context-reinjection-guard
test (validated the deleted conditional) deleted in same PR
per Principle 7.
- Phase 8: fail-loud counter at ~/.claude-mem/state/hook-failures
.json. Atomic write via .tmp + rename. CLAUDE_MEM_HOOK_FAIL_LOUD
_THRESHOLD setting (default 3). On consecutive worker-unreachable
≥ N: process.exit(2). On success: reset to 0. NOT a retry.
- Phase 9: ensureWorkerAliveOnce() module-scope memoization
wrapping ensureWorkerRunning. executeWithWorkerFallback calls
the memoized version.
Minimal validateBody middleware stub at
src/services/worker/http/middleware/validateBody.ts. Plan 06 will
expand with typed inference + error envelope conventions.
Verification: 4/4 grep targets pass. bun run build succeeds.
bun test → 1393 pass / 28 fail / 7 skip; -6 pass attributable
solely to deleted context-reinjection-guard test file. Zero new
failures vs baseline.
Plan: PATHFINDER-2026-04-22/05-hook-surface.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: land PATHFINDER Plan 06 — API surface
One Zod-based validator wrapping every POST/PUT. Rate limiter,
diagnostic endpoints, and shutdown wrappers deleted. Failure-
marking consolidated to one helper.
- Phase 1 (preflight): zod@^3 already installed.
- Phase 2: validateBody middleware confirmed at canonical shape
in src/services/worker/http/middleware/validateBody.ts —
safeParse → 400 { error: 'ValidationError', issues: [...] }
on failure, replaces req.body with parsed value on success.
- Phase 3: Per-route Zod schemas declared at the top of each
route file. 24 POST endpoints across SessionRoutes,
CorpusRoutes, DataRoutes, MemoryRoutes, SearchRoutes,
LogsRoutes, SettingsRoutes now wrap with validateBody().
/api/session/end (Plan 05) confirmed using same middleware.
- Phase 4: validateRequired() deleted from BaseRouteHandler
along with every call site. Inline coercion helpers
(coerceStringArray, coercePositiveInteger) and inline
if (!req.body...) guards deleted across all route files.
- Phase 5: Rate limiter middleware and its registration deleted
from src/services/worker/http/middleware.ts. Worker binds
127.0.0.1:37777 — no untrusted caller.
- Phase 6: viewer.html cached at module init in ViewerRoutes.ts
via fs.readFileSync; served as Buffer with text/html content
type. SKILL.md + per-operation .md files cached in
Server.ts as Map<string, string>; loadInstructionContent
helper deleted. NO fs.watch, NO TTL — process restart is the
cache-invalidation event.
- Phase 7: Four diagnostic endpoints deleted from DataRoutes.ts
— /api/pending-queue (GET), /api/pending-queue/process (POST),
/api/pending-queue/failed (DELETE), /api/pending-queue/all
(DELETE). Helper methods that ONLY served them
(getQueueMessages, getStuckCount, getRecentlyProcessed,
clearFailed, clearAll) deleted from PendingMessageStore.
KEPT: /api/processing-status (observability), /health
(used by ensureWorkerRunning).
- Phase 8: stopSupervisor wrapper deleted from supervisor/index.ts.
GracefulShutdown now calls getSupervisor().stop() directly.
Two functions retained with clear roles:
- performGracefulShutdown — worker-side 6-step shutdown
- runShutdownCascade — supervisor-side child teardown
(process.kill(-pgid), Windows tree-kill, PID-file cleanup)
Each has unique non-trivial logic and a single canonical caller.
- Phase 9: transitionMessagesTo(status, filter) is the sole
failure-marking path on PendingMessageStore. Old methods
markSessionMessagesFailed and markAllSessionMessagesAbandoned
deleted along with all callers (worker-service,
SessionCompletionHandler, tests/zombie-prevention).
Tests updated (Principle 7 same-PR delete): coercion test files
refactored to chain validateBody → handler. Zombie-prevention
tests rewritten to call transitionMessagesTo.
Verification: all 4 grep targets → 0. bun run build succeeds.
bun test → 1393 pass / 28 fail / 7 skip — exact match to
baseline. Zero new failures.
Plan: PATHFINDER-2026-04-22/06-api-surface.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: land PATHFINDER Plan 07 — dead code sweep
ts-prune-driven sweep across the tree after Plans 01-06 landed.
Deleted unused exports, orphan helpers, and one fully orphaned
file. Earlier-plan deletions verified.
Deleted:
- src/utils/bun-path.ts (entire file — getBunPath, getBunPathOrThrow,
isBunAvailable: zero importers)
- bun-resolver.getBunVersionString: zero callers
- PendingMessageStore.retryMessage / resetProcessingToPending /
abortMessage: superseded by transitionMessagesTo (Plan 06 Phase 9)
- EnvManager.MANAGED_CREDENTIAL_KEYS, EnvManager.setCredential:
zero callers
- CodexCliInstaller.checkCodexCliStatus: zero callers; no status
command exists in npx-cli
- Two "REMOVED: cleanupOrphanedSessions" stale-fence comments
Kept (with documented justification):
- Public API surface in dist/sdk/* (parseAgentXml, prompt
builders, ParsedObservation, ParsedSummary, ParseResult,
SUMMARY_MODE_MARKER) — exported via package.json sdk path.
- generateContext / loadContextConfig / token utilities — used
via dynamic await import('../../../context-generator.js') in
worker SearchRoutes.
- MCP_IDE_INSTALLERS, install/uninstall functions for codex/goose
— used via dynamic await import in npx-cli/install.ts +
uninstall.ts (ts-prune cannot trace dynamic imports).
- getExistingChromaIds — active caller in
ChromaSync.backfillMissingSyncs (Plan 04 narrowed scope).
- processPendingQueues / getSessionsWithPendingMessages — active
orphan-recovery caller in worker-service.ts plus
zombie-prevention test coverage.
- StoreAndMarkCompleteResult legacy alias — return-type annotation
in same file.
- All Database.ts barrel re-exports — used downstream.
Earlier-plan verification:
- Plan 03 Phase 9: VERIFIED — src/utils/transcript-parser.ts
is gone; TranscriptParser has 0 references in src/.
- Plan 01 Phase 8: VERIFIED — migration 19 no-op absorbed.
- SessionStore.ts:52-70 consolidation NOT executed (deferred):
the methods are not thin wrappers but ~900 LoC of bodies, and
two methods are documented as intentional mirrors so the
context-generator.cjs bundle stays schema-consistent without
pulling MigrationRunner. Deserves its own plan, not a sweep.
Verification: TranscriptParser → 0; transcript-parser.ts → gone;
no commented-out code markers remain. bun run build succeeds.
bun test → 1393 pass / 28 fail / 7 skip — EXACT match to
baseline. Zero regressions.
Plan: PATHFINDER-2026-04-22/07-dead-code.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: remove residual ProcessRegistry comment reference
Plan 07 dead-code sweep missed one comment-level reference to the
deleted in-memory ProcessRegistry class in SessionManager.ts:347.
Rewritten to describe the supervisor.json scope without naming the
deleted class, completing the verification grep target.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: address Greptile review (P1 + 2× P2)
P1 — Plan 05 Phase 3 blocking endpoint was non-functional:
executeWithWorkerFallback used HEALTH_CHECK_TIMEOUT_MS (3 s) for
the POST /api/session/end call, but the server holds the
connection for SERVER_SIDE_SUMMARY_TIMEOUT_MS (30 s). Client
always raced to a "timed out" rejection that isWorkerUnavailable
classified as worker-unreachable, so the hook silently degraded
instead of waiting for summaryStoredEvent.
- Added optional timeoutMs to executeWithWorkerFallback,
forwarded to workerHttpRequest.
- summarize.ts call site now passes 35_000 (5 s above server
hold window).
P2 — ingestSummary({ kind: 'parsed' }) branch was dead code:
ResponseProcessor emitted summaryStoredEvent directly via the
event bus, bypassing the centralized helper that the comment
claimed was the single source.
- ResponseProcessor now calls ingestSummary({ kind: 'parsed',
sessionDbId, messageId, contentSessionId, parsed }) so the
event-emission path is single-sourced.
- ingestSummary's requireContext() resolution moved inside the
'queue' branch (the only branch that needs sessionManager /
dbManager). 'parsed' is a pure event-bus emission and
doesn't need worker-internal context — fixes mocked
ResponseProcessor unit tests that don't call
setIngestContext.
P2 — isWorkerFallback could false-positive on legitimate API
responses whose schema includes { continue: true, ... }:
- Added a Symbol.for('claude-mem/worker-fallback') brand to
WorkerFallback. isWorkerFallback now checks the brand, not
a duck-typed property name.
Verification: bun run build succeeds. bun test → 1393 pass /
28 fail / 7 skip — exact baseline match. Zero new failures.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: address Greptile iteration 2 (P1 + P2)
P1 — summaryStoredEvent fired regardless of whether the row was
persisted. ResponseProcessor's call to ingestSummary({ kind:
'parsed' }) ran for every parsed.kind === 'summary' even when
result.summaryId came back null (e.g. FK violation, null
memory_session_id at commit). The blocking /api/session/end
endpoint then returned { ok: true } and the Stop hook logged
'Summary stored' for a non-existent row.
- Gate ingestSummary call on (parsed.data.skipped ||
session.lastSummaryStored). Skipped summaries are an explicit
no-op bypass and still confirm; real summaries only confirm
when storage actually wrote a row.
- Non-skipped + summaryId === null path logs a warn and lets
the server-side timeout (504) surface to the hook instead of
a false ok:true.
P2 — PendingMessageStore.enqueue() returns 0 when INSERT OR
IGNORE suppresses a duplicate (the UNIQUE(session_id, tool_use_id)
constraint added by Plan 01 Phase 1). The two callers
(SessionManager.queueObservation and queueSummarize) previously
logged 'ENQUEUED messageId=0' which read like a row was inserted.
- Branch on messageId === 0 and emit a 'DUP_SUPPRESSED' debug
log instead of the misleading ENQUEUED line. No behavior
change — the duplicate is still correctly suppressed by the
DB (Principle 3); only the log surface is corrected.
- confirmProcessed is never called with the enqueue() return
value (it operates on session.processingMessageIds[] from
claimNextMessage), so no caller is broken; the visibility
fix prevents future misuse.
Verification: bun run build succeeds. bun test → 1393 pass /
28 fail / 7 skip — exact baseline match. Zero new failures.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: address Greptile iteration 3 (P1 + 2× P2)
- P1 worker-service.ts: wire ensureGeneratorRunning into the ingest
context after SessionRoutes is constructed. setIngestContext runs
before routes exist, so transcript-watcher observations queued via
ingestObservation() had no way to auto-start the SDK generator.
Added attachIngestGeneratorStarter() to patch the callback in.
- P2 shared.ts: IngestEventBus now sets maxListeners to 0. Concurrent
/api/session/end calls register one listener each and clean up on
completion, so the default-10 warning fires spuriously under normal
load.
- P2 SessionRoutes.ts: handleObservationsByClaudeId now delegates to
ingestObservation() instead of duplicating skip-tool / meta /
privacy / queue logic. Single helper, matching the Plan 03 goal.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: address Greptile iteration 4 (P1 tool-pair + P2 parse/path/doc)
- processor.handleToolResult: restore in-memory tool-use→tool-result
pairing via session.pendingTools for schemas (e.g. Codex) whose
tool_result events carry only tool_use_id + output. Without this,
neither handler fired — all tool observations silently dropped.
- processor.maybeParseJson: return raw string on parse failure instead
of throwing. Previously a single malformed JSON-shaped field caused
handleLine's outer catch to discard the entire transcript line.
- watcher.deepestNonGlobAncestor: split on / and \\, emit empty string
for purely-glob inputs so the caller skips the watch instead of
anchoring fs.watch at the filesystem root. Windows-compatible.
- PendingMessageStore.enqueue: tighten docstring — callers today only
log on the returned id; the SessionManager branches on id === 0.
* fix: forward tool_use_id through ingestObservation (Greptile iter 5)
P1 — Plan 01's UNIQUE(content_session_id, tool_use_id) dedup never
fired because the new shared ingest path dropped the toolUseId before
queueObservation. SQLite treats NULL values as distinct for UNIQUE,
so every replayed transcript line landed a duplicate row.
- shared.ingestObservation: forward payload.toolUseId to
queueObservation so INSERT OR IGNORE can actually collapse.
- SessionRoutes.handleObservationsByClaudeId: destructure both
tool_use_id (HTTP convention) and toolUseId (JS convention) from
req.body and pass into ingestObservation.
- observationsByClaudeIdSchema: declare both keys explicitly so the
validator doesn't rely on .passthrough() alone.
* fix: drop dead pairToolUsesByJoin, close session-end listener race
- PendingMessageStore: delete pairToolUsesByJoin. The method was never
called and its self-join semantics are structurally incompatible
with UNIQUE(content_session_id, tool_use_id): INSERT OR IGNORE
collapses any second row with the same pair, so a self-join can
only ever match a row to itself. In-memory pendingTools in
processor.ts remains the pairing path for split-event schemas.
- IngestEventBus: retain a short-lived (60s) recentStored map keyed
by sessionId. Populated on summaryStoredEvent emit, evicted on
consume or TTL.
- handleSessionEnd: drain the recent-events buffer before attaching
the listener. Closes the register-after-emit race where the summary
can persist between the hook's summarize POST and its session/end
POST — previously that window returned 504 after the 30s timeout.
* chore: merge origin/main into vivacious-teeth
Resolves conflicts with 15 commits on main (v12.3.9, security
observation types, Telegram notifier, PID-reuse worker start-guard).
Conflict resolution strategy:
- plugin/hooks/hooks.json, plugin/scripts/*.cjs, plugin/ui/viewer-bundle.js:
kept ours — PATHFINDER Plan 05 deletes the for-i-in-1-to-20 curl retry
loops and the built artifacts regenerate on build.
- src/cli/handlers/summarize.ts: kept ours — Plan 05 blocking
POST /api/session/end supersedes main's fire-and-forget path.
- src/services/worker-service.ts: kept ours — Plan 05 ingest bus +
summaryStoredEvent supersedes main's SessionCompletionHandler DI
refactor + orphan-reaper fallback.
- src/services/worker/http/routes/SessionRoutes.ts: kept ours — same
reason; generator .finally() Stop-hook self-clean is a guard for a
path our blocking endpoint removes.
- src/services/worker/http/routes/CorpusRoutes.ts: merged — added
security_alert / security_note to ALLOWED_CORPUS_TYPES (feature from
#2084) while preserving our Zod validateBody schema.
Typecheck: 294 errors (vs 298 pre-merge). No new errors introduced; all
remaining are pre-existing (Component-enum gaps, DOM lib for viewer,
bun:sqlite types).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: address Greptile P2 findings
1) SessionRoutes.handleSessionEnd was the only route handler not wrapped
in wrapHandler — synchronous exceptions would hang the client rather
than surfacing as 500s. Wrap it like every other handler.
2) processor.handleToolResult only consumed the session.pendingTools
entry when the tool_result arrived without a toolName. In the
split-schema path where tool_result carries both toolName and toolId,
the entry was never deleted and the map grew for the life of the
session. Consume the entry whenever toolId is present.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: typing cleanup and viewer tsconfig split for PR feedback
- Add explicit return types for SessionStore query methods
- Exclude src/ui/viewer from root tsconfig, give it its own DOM-typed config
- Add bun to root tsconfig types, plus misc typing tweaks flagged by Greptile
- Rebuilt plugin/scripts/* artifacts
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: address Greptile P2 findings (iter 2)
- PendingMessageStore.transitionMessagesTo: require sessionDbId (drop
the unscoped-drain branch that would nuke every pending/processing
row across all sessions if a future caller omitted the filter).
- IngestEventBus.takeRecentSummaryStored: make idempotent — keep the
cached event until TTL eviction so a retried Stop hook's second
/api/session/end returns immediately instead of hanging 30 s.
- TranscriptWatcher fs.watch callback: skip full glob scan for paths
already tailed (JSONL appends fire on every line; only unknown
paths warrant a rescan).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: call finalizeSession in terminal session paths (Greptile iter 3)
terminateSession and runFallbackForTerminatedSession previously called
SessionCompletionHandler.finalizeSession before removeSessionImmediate;
the refactor dropped those calls, leaving sdk_sessions.status='active'
for every session killed by wall-clock limit, unrecoverable error, or
exhausted fallback chain. The deleted reapStaleSessions interval was
the only prior backstop.
Re-wires finalizeSession (idempotent: marks completed, drains pending,
broadcasts) into both paths; no reaper reintroduced.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: GC failed pending_messages rows at startup (Greptile iter 4)
Plan 07 deleted clearFailed/clearFailedOlderThan as "dead code", but
with the periodic sweep also removed, nothing reaps status='failed'
rows now — they accumulate indefinitely. Since claimNextMessage's
self-healing subquery scans this table, unbounded growth degrades
claim latency over time.
Re-introduces clearFailedOlderThan and calls it once at worker startup
(not a reaper — one-shot, idempotent). 7-day retention keeps enough
history for operator inspection while bounding the table.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: finalize sessions on normal exit; cleanup hoist; share handler (iter 5)
1. startSessionProcessor success branch now calls completionHandler.
finalizeSession before removeSessionImmediate. Hooks-disabled installs
(and any Stop hook that fails before POST /api/sessions/complete) no
longer leave sdk_sessions rows as status='active' forever. Idempotent
— a subsequent /api/sessions/complete is a no-op.
2. Hoist SessionRoutes.handleSessionEnd cleanup declaration above the
closures that reference it (TDZ safety; safe at runtime today but
fragile if timeout ever shrinks).
3. SessionRoutes now receives WorkerService's shared SessionCompletionHandler
instead of constructing its own — prevents silent divergence if the
handler ever becomes stateful.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: stop runaway crash-recovery loop on dead sessions
Two distinct bugs were combining to keep a dead session restarting forever:
Bug 1 (uncaught "The operation was aborted."):
child_process.spawn emits 'error' asynchronously for ENOENT/EACCES/abort
signal aborts. spawnSdkProcess() never attached an 'error' listener, so
any async spawn failure became uncaughtException and escaped to the
daemon-level handler. Attach an 'error' listener immediately after spawn,
before the !child.pid early-return, so async spawn errors are logged
(with errno code) and swallowed locally.
Bug 2 (sliding-window limiter never trips on slow restart cadence):
RestartGuard tripped only when restartTimestamps.length exceeded
MAX_WINDOWED_RESTARTS (10) within RESTART_WINDOW_MS (60s). With the 8s
exponential-backoff cap, only ~7-8 restarts fit in the window, so a dead
session that fail-restart-fail-restart on 8s cycles would loop forever
(consecutiveRestarts climbing past 30+ in observed logs). Add a
consecutiveFailures counter that increments on every restart and resets
only on recordSuccess(). Trip when consecutive failures exceed
MAX_CONSECUTIVE_FAILURES (5) — meaning 5 restarts with zero successful
processing in between proves the session is dead. Both guards now run in
parallel: tight loops still trip the windowed cap; slow loops trip the
consecutive-failure cap.
Also: when the SessionRoutes path trips the guard, drain pending messages
to 'abandoned' so the session does not reappear in
getSessionsWithPendingMessages and trigger another auto-start cycle. The
worker-service.ts path already does this via terminateSession.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* perf: streamline worker startup and consolidate database connections
1. Database Pooling: Modified DatabaseManager, SessionStore, and SessionSearch to share a single bun:sqlite connection, eliminating redundant file descriptors.
2. Non-blocking Startup: Refactored WorktreeAdoption and Chroma backfill to run in the background (fire-and-forget), preventing them from stalling core initialization.
3. Diagnostic Routes: Added /api/chroma/status and bypassed the initialization guard for health/readiness endpoints to allow diagnostics during startup.
4. Robust Search: Implemented reliable SQLite FTS5 fallback in SearchManager for when Chroma (uvx) fails or is unavailable.
5. Code Cleanup: Removed redundant loopback MCP checks and mangled initialization logic from WorkerService.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: hard-exclude observer-sessions from hooks; bundle migration 29 (#2124)
* fix: hard-exclude observer-sessions from hooks; backfill bundle migrations
Stop hook + SessionEnd hook were storing the SDK observer's own
init/continuation/summary prompts in user_prompts, leaking into the
viewer (meta-observation regression). 25 such rows accumulated.
- shouldTrackProject: hard-reject OBSERVER_SESSIONS_DIR (and its subtree)
before consulting user-configured exclusion globs.
- summarize.ts (Stop) and session-complete.ts (SessionEnd): early-return
when shouldTrackProject(cwd) is false, so the observer's own hooks
cannot bootstrap the worker or queue a summary against the meta-session.
- SessionRoutes: cap user-prompt body at 256 KiB at the session-init
boundary so a runaway observer prompt cannot blow up storage.
- SessionStore: add migration 29 (UNIQUE(memory_session_id, content_hash)
on observations) inline so bundled artifacts (worker-service.cjs,
context-generator.cjs) stay schema-consistent — without it, the
ON CONFLICT clause in observation inserts throws.
- spawnSdkProcess: stdio[stdin] from 'ignore' to 'pipe' so the
supervisor can actually feed the observer's stdin.
Also rebuilds plugin/scripts/{worker-service,context-generator}.cjs.
* fix: walk back to UTF-8 boundary on prompt truncation (Greptile P2)
Plain Buffer.subarray at MAX_USER_PROMPT_BYTES can land mid-codepoint,
which the utf8 decoder silently rewrites to U+FFFD. Walk back over any
continuation bytes (0b10xxxxxx) before decoding so the truncated prompt
ends on a valid sequence boundary instead of a replacement character.
* fix: cross-platform observer-dir containment; clarify SDK stdin pipe
claude-review feedback on PR #2124.
- shouldTrackProject: literal `cwd.startsWith(OBSERVER_SESSIONS_DIR + '/')`
hard-coded a POSIX separator and missed Windows backslash paths plus any
trailing-slash variance. Switched to a path.relative-based isWithin()
helper so Windows hook input under observer-sessions\\... is also excluded.
- spawnSdkProcess: added a comment explaining why stdin must be 'pipe' —
SpawnedSdkProcess.stdin is typed NonNullable and the Claude Agent SDK
consumes that pipe; 'ignore' would null it and the null-check below
would tear the child down on every spawn.
* fix: make Stop hook fire-and-forget; remove dead /api/session/end
The Stop hook was awaiting a 35-second long-poll on /api/session/end,
which the worker held open until the summary-stored event fired (or its
30s server-side timeout elapsed). Followed by another await on
/api/sessions/complete. Three sequential awaits, the middle one a 30s
hold — not fire-and-forget despite repeated requests.
The Stop hook now does ONE thing: POST /api/sessions/summarize to
queue the summary work and return. The worker drives the rest async.
Session-map cleanup is performed by the SessionEnd handler
(session-complete.ts), not duplicated here.
- summarize.ts: drop the /api/session/end long-poll and the trailing
/api/sessions/complete await; ~40 lines removed; unused
SessionEndResponse interface gone; header comment rewritten.
- SessionRoutes: delete handleSessionEnd, sessionEndSchema, the
SERVER_SIDE_SUMMARY_TIMEOUT_MS constant, and the /api/session/end
route registration. Drop the now-unused ingestEventBus and
SummaryStoredEvent imports.
- ResponseProcessor + shared.ts + worker-utils.ts: update stale
comments that referenced the dead endpoint. The IngestEventBus is
left in place dormant (no listeners) for follow-up cleanup so this
PR stays focused on the blocker.
Bundle artifact (worker-service.cjs) rebuilt via build-and-sync.
Verification:
- grep '/api/session/end' plugin/scripts/worker-service.cjs → 0
- grep 'timeoutMs:35' plugin/scripts/worker-service.cjs → 0
- Worker restarted clean, /api/health ok at pid 92368
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* deps: bump all dependencies to latest including majors
Upgrades: React 18→19, Express 4→5, Zod 3→4, TypeScript 5→6,
@types/node 20→25, @anthropic-ai/claude-agent-sdk 0.1→0.2,
@clack/prompts 0.9→1.2, plus minors. Adds Daily Maintenance section
to CLAUDE.md mandating latest-version policy across manifests.
Express 5 surfaced a race in Server.listen() where the 'error' handler
was attached after listen() was invoked; refactored to use
http.createServer with both 'error' and 'listening' handlers attached
before listen(), restoring port-conflict rejection semantics.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: surface real chroma errors and add deep status probe
Replace the misleading "Vector search failed - semantic search unavailable.
Install uv... restart the worker." string in SearchManager with the actual
exception text from chroma_query_documents. The lying message blamed `uv`
for any failure — even when the real cause was a chroma-mcp transport
timeout, an empty collection, or a dead subprocess.
Also add /api/chroma/status?deep=1 backed by a new
ChromaMcpManager.probeSemanticSearch() that round-trips a real query
(chroma_list_collections + chroma_query_documents) instead of just
checking the stdio handshake. The cheap default path is unchanged.
Includes the diagnostic plan (PLAN-fix-mcp-search.md) and updated test
fixtures for the new structured failure message.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: rebuild worker-service bundle to match merged src
Bundle was stale after the squash merge of #2124 — it still contained
the old "Install uv... semantic search unavailable" string and lacked
probeSemanticSearch. Rebuilt via bun run build-and-sync.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: address coderabbit feedback on PLAN-fix-mcp-search.md
- replace machine-specific /Users/alexnewman absolute paths with portable
<repo-root> placeholder (MD-style portability)
- add blank lines around the TypeScript fenced block (MD031)
- tag the bare fenced block with `text` (MD040)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
392 lines
28 KiB
Markdown
392 lines
28 KiB
Markdown
# Plan 10 — knowledge-corpus-builder (clean)
|
||
|
||
**Target section**: `PATHFINDER-2026-04-21/05-clean-flowcharts.md` § 3.11 (lines 450–476), Part 1 items #35 (line 53) and #36 (line 54).
|
||
**Before-state**: `PATHFINDER-2026-04-21/01-flowcharts/knowledge-corpus-builder.md` (lines 1–87).
|
||
**Implementation-plan correspondence**: `PATHFINDER-2026-04-21/06-implementation-plan.md` Phase 13 — "KnowledgeAgent simplification" (lines 567–597). **Direct V-number: NONE** — the verified-findings matrix (V1–V20, lines 22–47) does not include a corpus-specific entry. No upstream discrepancy was registered for this area; treat 05 § 3.11 + Phase 13 as the canonical pair.
|
||
|
||
## Dependencies
|
||
|
||
- **Upstream**:
|
||
- Plan 05-context-injection-engine — defines `CorpusDetailStrategy` (one of the four strategy configs in 05 § 3.5 lines 232–259 and Part 2 decision D4 line 75). This plan calls `renderObservations(obs, CorpusDetailStrategy)` from CorpusBuilder.
|
||
- Plan 06-hybrid-search-orchestration — defines the clean `SearchOrchestrator.search` signature (05 § 3.6 lines 262–292). CorpusBuilder is a *consumer* — the live call is `SearchOrchestrator.search(args)` at `src/services/worker/search/SearchOrchestrator.ts:71`.
|
||
- **Downstream**: none.
|
||
|
||
## Phase 0 — Documentation Discovery (already done)
|
||
|
||
### Sources consulted
|
||
1. `PATHFINDER-2026-04-21/05-clean-flowcharts.md` — full file (607 lines). Section 3.11 (lines 450–476) is canonical; Part 1 items #35–36 (lines 53–54) set the kill rationale; Part 5 ledger row (line 556) promises ~110 net lines deleted in this area.
|
||
2. `PATHFINDER-2026-04-21/06-implementation-plan.md` — full file (691 lines). Phase 13 (lines 567–597). **No V-number in 06's verified-findings table (V1–V20) covers the corpus.** Stated explicitly: Phase 13 cites 05 § 3.11 directly without a V-correction, because the audit's claims matched the live code.
|
||
3. `PATHFINDER-2026-04-21/01-flowcharts/knowledge-corpus-builder.md` — full file (87 lines). "Before" flowchart + the Confidence+Gaps section pinpoints the regex at `KnowledgeAgent.ts:179`.
|
||
4. Live codebase (confirmed paths, line counts, and specific anchors):
|
||
- `src/services/worker/knowledge/KnowledgeAgent.ts` (284 lines)
|
||
- `src/services/worker/knowledge/CorpusStore.ts` (127 lines)
|
||
- `src/services/worker/knowledge/CorpusBuilder.ts` (174 lines)
|
||
- `src/services/worker/knowledge/CorpusRenderer.ts` (133 lines)
|
||
- `src/services/worker/knowledge/types.ts` (56 lines)
|
||
- `src/services/worker/knowledge/index.ts` (14 lines)
|
||
- `src/services/worker/http/routes/CorpusRoutes.ts` (283 lines)
|
||
- `src/services/worker-service.ts:455-456` — constructor wiring
|
||
- `src/servers/mcp-server.ts:499,517,551` — MCP tool surface that mirrors HTTP
|
||
5. Dependency plans (cross-refs only, not re-planned here):
|
||
- 05 § 3.5 (CorpusDetailStrategy) — renderer contract at 05 lines 379–389
|
||
- 05 § 3.6 (SearchOrchestrator.search) — live signature at `src/services/worker/search/SearchOrchestrator.ts:71`.
|
||
|
||
### Allowed APIs (copy from; do not invent)
|
||
|
||
- **Claude Agent SDK** — `query({ prompt, options })` already used at `KnowledgeAgent.ts:75` and `:190`. Per 05 § 3.11 (line 461 node "S"): call as `SDK.query(systemPrompt=corpus, userPrompt=question)` — a fresh query every call. The existing SDK usage patterns (cwd, disallowedTools, pathToClaudeCodeExecutable, env) at `KnowledgeAgent.ts:77-84` stay.
|
||
- **Prompt caching** — the SDK supplies it automatically when the same system prompt is sent within the 5-min TTL. 05 § 3.11 "Cost note" (lines 476): "cached system prompt TTL is 5 min. Cost approximately equal to session-resume path without the session-expiration brittleness." The refactor does not add any caching code — it relies on the SDK's own behavior.
|
||
- **CorpusDetailStrategy** — comes from Plan 05 (renderer contract at 05 lines 379–389). This plan consumes it; it does not define it.
|
||
- **`bun:sqlite` / file I/O** — `CorpusStore` already uses `fs.writeFileSync/readFileSync`. No new storage primitives.
|
||
|
||
### Anti-patterns to prohibit (cited in every phase)
|
||
|
||
- **A — Invent SDK methods for session resume.** The SDK has no documented session-expiry ping or refresh endpoint. Don't add one.
|
||
- **B — Polling.** The regex test `/session|resume|expired|invalid.*session|not found/i` at `KnowledgeAgent.ts:179` is a polling heuristic in disguise — try, match on error text, retry. Delete.
|
||
- **C — Silent fallback.** The current "session expired → silently reprime → retry" path at `KnowledgeAgent.ts:146–160` hides a contract violation. Replacement contract: every `/query` runs a **fresh** SDK query; there is no expiration state to recover from.
|
||
- **D — Facades that pass through.** `KnowledgeAgent.reprime` at `KnowledgeAgent.ts:168–171` is a two-line call to `prime`. Both die together.
|
||
- **E — Two code paths for the same data.** After the refactor, there is exactly one path that sends a corpus to the SDK: inside the `/query` handler.
|
||
|
||
### Corpus.json schema change (from `types.ts:40–51`)
|
||
|
||
Before:
|
||
```ts
|
||
interface CorpusFile {
|
||
version: 1;
|
||
name: string;
|
||
description: string;
|
||
created_at: string;
|
||
updated_at: string;
|
||
filter: CorpusFilter;
|
||
stats: CorpusStats;
|
||
system_prompt: string;
|
||
session_id: string | null; // <-- DROP
|
||
observations: CorpusObservation[];
|
||
}
|
||
```
|
||
|
||
After (per 06 Phase 13 task 2, line 579 — with this plan's note that observations stay because `/query` still needs them to build the system prompt):
|
||
```ts
|
||
interface CorpusFile {
|
||
version: 2; // bump so older files with session_id are recognized
|
||
name: string;
|
||
description: string;
|
||
created_at: string;
|
||
updated_at: string;
|
||
filter: CorpusFilter;
|
||
stats: CorpusStats;
|
||
system_prompt: string;
|
||
observations: CorpusObservation[];
|
||
}
|
||
```
|
||
|
||
> 06 Phase 13 line 579 suggests trimming further to `{name, filters, renderedCorpus, generatedAt}`. This plan keeps the richer shape so `/query` can recompute `renderObservations(obs, CorpusDetailStrategy)` on demand without re-hitting SQLite. If the stored `system_prompt` + observations combined are too large, switch to storing `renderedCorpus` directly; decision flagged in "Gaps" below.
|
||
|
||
### HTTP surface (constraint from prompt)
|
||
|
||
Keep:
|
||
- `POST /api/corpus` (build)
|
||
- `POST /api/corpus/:name/query`
|
||
- `POST /api/corpus/:name/rebuild`
|
||
- `DELETE /api/corpus/:name`
|
||
- `GET /api/corpus` (list) and `GET /api/corpus/:name` (get) — present today at `CorpusRoutes.ts:29-30`; 05 § 3.11 doesn't mention them but they are user-facing read endpoints. Keep.
|
||
|
||
Delete (per 05 § 3.11 lines 468–474):
|
||
- `POST /api/corpus/:name/prime` (handler at `CorpusRoutes.ts:33` / `:213-228`)
|
||
- `POST /api/corpus/:name/reprime` (handler at `CorpusRoutes.ts:35` / `:267-282`)
|
||
|
||
---
|
||
|
||
## Phase 1 — Remove `session_id` from the corpus schema and `CorpusStore`
|
||
|
||
### (a) What to implement — Copy from …
|
||
|
||
- Copy from **05 § 3.11 line 470**: "`session_id` persisted in corpus.json" is in the deleted list. Also **06 Phase 13 task 2** (line 579): "Simplify `CorpusStore`… No `session_id`."
|
||
|
||
### (b) Docs
|
||
|
||
- 05 § 3.11 (lines 450–474) — sets the "no session_id" rule.
|
||
- 06 Phase 13 task 2 (line 579) — task text.
|
||
- Live file:line targets:
|
||
- `src/services/worker/knowledge/types.ts:49` — `session_id: string | null;` inside `CorpusFile`. Remove.
|
||
- `src/services/worker/knowledge/types.ts:40` — bump `version: 1` → `version: 2`.
|
||
- `src/services/worker/knowledge/types.ts:53-56` — `QueryResult { answer, session_id }`. Remove `session_id` from `QueryResult` (new shape: `{ answer }`).
|
||
- `src/services/worker/knowledge/CorpusStore.ts:61, :67, :77` — `list()` return type drops `session_id`; payload builder at `:74-78` drops the field.
|
||
- `src/services/worker/knowledge/CorpusBuilder.ts:104` — literal `session_id: null` inside the built corpus. Delete the line.
|
||
|
||
### (c) Verification
|
||
|
||
- `grep -n "session_id" src/services/worker/knowledge/` → zero lines. (Today: 18 matches across KnowledgeAgent.ts, CorpusStore.ts, CorpusBuilder.ts, types.ts.)
|
||
- Compile clean: `npx tsc --noEmit`.
|
||
- Unit test: `CorpusStore.read` on a legacy corpus file that still has `session_id` returns a valid `CorpusFile` (extra field ignored by the structural cast, or migrated — see "Blast radius" note below).
|
||
- `corpus.json` schema assertion (new integration test): build a corpus; read the file back with `JSON.parse`; assert `!("session_id" in parsed)`.
|
||
|
||
### (d) Anti-pattern guards
|
||
|
||
- **A**: Don't add a "migration helper" that re-writes old `session_id: "..."` fields into some new shape. Ignore the field on read; the worker never re-emits it.
|
||
- **C**: Don't default `session_id` to `null` "for backward compat" — drop the field outright.
|
||
|
||
---
|
||
|
||
## Phase 2 — Delete `KnowledgeAgent.prime` as a distinct operation
|
||
|
||
### (a) What to implement — Copy from …
|
||
|
||
- Copy from **05 § 3.11 deleted list, line 469**: "`KnowledgeAgent.prime` as a distinct operation — build IS prime (corpus.json is the prime artifact)."
|
||
- 06 Phase 13 task 1 (line 578).
|
||
|
||
### (b) Docs
|
||
|
||
- 05 § 3.11 (lines 450–474) — deleted-nodes rationale.
|
||
- Live file:line targets:
|
||
- `src/services/worker/knowledge/KnowledgeAgent.ts:52-117` — entire `prime()` method (66 lines). Delete.
|
||
- `src/services/worker/knowledge/KnowledgeAgent.ts:163-171` — entire `reprime()` method (9 lines). Delete (see Phase 4 for endpoint). `reprime` just calls `prime`, so it dies with it (anti-pattern **D**).
|
||
- `src/services/worker/knowledge/KnowledgeAgent.ts:12-41` — imports `OBSERVER_SESSIONS_DIR`, `ensureDir`, `buildIsolatedEnv`, `sanitizeEnv`, `KNOWLEDGE_AGENT_DISALLOWED_TOOLS`. Some still used by the rewritten `query()` in Phase 5; reassess after Phase 5 lands. The disallowedTools list at `:28-41` stays (still applied per call per 05 § 3.11 — Q&A only).
|
||
|
||
### (c) Verification
|
||
|
||
- `grep -n "^\s*async prime\|\.prime(" src/services/worker/knowledge/` → zero.
|
||
- `grep -n "async reprime\|\.reprime(" src/services/worker/knowledge/` → zero.
|
||
- Corpus still builds end-to-end: `curl -X POST /api/corpus -d '{"name":"t","limit":5}'` returns metadata; the resulting `~/.claude-mem/corpora/t.corpus.json` has observations + system_prompt but no SDK session was spawned during build.
|
||
- `wc -l src/services/worker/knowledge/KnowledgeAgent.ts` drops by roughly 75 lines (prime 66 + reprime 9). Tracked against the 110-line net-delete target in 05 Part 5.
|
||
|
||
### (d) Anti-pattern guards
|
||
|
||
- **A**: Don't add `buildAndPrime(corpus)` as a "unified" helper. Build *is* prime; the SDK is not touched at build time anymore.
|
||
- **D**: `reprime` is a pass-through; delete the method, don't keep a stub.
|
||
|
||
---
|
||
|
||
## Phase 3 — Delete the auto-reprime regex and the session-expiration retry path
|
||
|
||
### (a) What to implement — Copy from …
|
||
|
||
- Copy from **05 Part 1 line 53** (item #35): "KnowledgeAgent auto-reprime on session-expiration regex match … just always prime on query — or store corpus content in a file the SDK loads fresh. No session_id persistence."
|
||
- Copy from **05 § 3.11 deleted list, line 471**: "Auto-reprime on regex-matched expiration (~40 lines)."
|
||
|
||
### (b) Docs
|
||
|
||
- 05 Part 1 #35 (line 53) — kill rationale.
|
||
- 05 § 3.11 (lines 450–474) — replacement flow ("SDK.query(systemPrompt=corpus, userPrompt=question) — fresh query — no session resume").
|
||
- Live file:line targets:
|
||
- `src/services/worker/knowledge/KnowledgeAgent.ts:119-161` — `query()` method with its try/catch auto-reprime branch. Delete the entire body; Phase 5 rewrites it.
|
||
- `src/services/worker/knowledge/KnowledgeAgent.ts:173-180` — `isSessionResumeError()`. **Exact regex to delete** (captured at `:179`):
|
||
```
|
||
/session|resume|expired|invalid.*session|not found/i
|
||
```
|
||
Delete the whole method.
|
||
- `src/services/worker/knowledge/KnowledgeAgent.ts:183-230` — `executeQuery()` (the resume path). Delete; Phase 5 replaces it.
|
||
|
||
### (c) Verification
|
||
|
||
- `grep -n "isSessionResumeError\|auto.?reprime\|session.*expired" src/services/worker/knowledge/` → zero.
|
||
- `grep -nE "session\|resume\|expired\|invalid.*session\|not found" src/services/worker/knowledge/` → zero (the raw regex string is gone).
|
||
- No retry-on-error logic anywhere in `KnowledgeAgent`. A failed `/query` call propagates to the route handler as a thrown error, returned to the client as `{error: '…'}`.
|
||
|
||
### (d) Anti-pattern guards
|
||
|
||
- **B**: Do not replace the regex with a different error-string match. The whole "detect expiry → retry" pattern goes.
|
||
- **C**: If `SDK.query` throws, do **not** silently reprime and retry. Propagate. The caller decides.
|
||
- **A**: The SDK does not expose a `refreshSession` or `isSessionValid` method — confirmed by the existing usage in `SDKAgent.ts` (not imported for our code path). Don't invent one.
|
||
|
||
---
|
||
|
||
## Phase 4 — Delete `/prime` and `/reprime` endpoints
|
||
|
||
### (a) What to implement — Copy from …
|
||
|
||
- Copy from **05 § 3.11 deleted list, lines 472–474**: "`reprime` endpoint (rebuild covers it)" and (by implication) `prime` endpoint (since `prime` as an operation is gone).
|
||
- 06 Phase 13 task 1 (line 578): "Delete `KnowledgeAgent.prime` and the `reprime` endpoint."
|
||
|
||
### (b) Docs
|
||
|
||
- Constraint from the request: keep `POST /api/corpus`, `POST /api/corpus/:name/query`, `POST /api/corpus/:name/rebuild`, `DELETE /api/corpus/:name`. Drop `/prime` and `/reprime`.
|
||
- Live file:line targets:
|
||
- `src/services/worker/http/routes/CorpusRoutes.ts:33` — `app.post('/api/corpus/:name/prime', …)` registration. Delete.
|
||
- `src/services/worker/http/routes/CorpusRoutes.ts:35` — `app.post('/api/corpus/:name/reprime', …)` registration. Delete.
|
||
- `src/services/worker/http/routes/CorpusRoutes.ts:209-228` — `handlePrimeCorpus` handler (20 lines). Delete.
|
||
- `src/services/worker/http/routes/CorpusRoutes.ts:263-282` — `handleReprimeCorpus` handler (20 lines). Delete.
|
||
- `src/servers/mcp-server.ts:499` — MCP tool `prime_corpus`. Delete (tool registration + handler). The deferred-tool namespace exposes it today as `mcp__plugin_claude-mem_mcp-search__prime_corpus`.
|
||
- `src/servers/mcp-server.ts:551` — MCP tool `reprime_corpus`. Delete.
|
||
- `src/servers/mcp-server.ts:517` — `query_corpus` description mentions "The corpus must be primed first"; update to "Ask a question about the corpus; the corpus content is loaded fresh per query."
|
||
|
||
### (c) Verification
|
||
|
||
- `curl -X POST http://localhost:37777/api/corpus/foo/prime` → HTTP 404 (route no longer registered; Express default 404).
|
||
- `curl -X POST http://localhost:37777/api/corpus/foo/reprime` → HTTP 404.
|
||
- `grep -n "prime_corpus\|reprime_corpus" src/` → zero.
|
||
- `grep -n "handlePrimeCorpus\|handleReprimeCorpus" src/` → zero.
|
||
- MCP client listing no longer shows `prime_corpus` or `reprime_corpus` tools.
|
||
|
||
### (d) Anti-pattern guards
|
||
|
||
- **D**: Don't leave thin `/prime` and `/reprime` handlers that just return 410 Gone. Delete the routes; 404 is the correct response.
|
||
- **A**: Don't add a compatibility-shim tool `prime_corpus_deprecated`.
|
||
|
||
---
|
||
|
||
## Phase 5 — Rewrite `/query` to issue a fresh SDK query with corpus content as system prompt
|
||
|
||
### (a) What to implement — Copy from …
|
||
|
||
- Copy from **05 § 3.11 lines 460–463** (the clean flowchart):
|
||
```
|
||
Q["POST /api/corpus/:name/query {question}"] --> R["CorpusStore.read(name)"]
|
||
R --> S["SDK.query(systemPrompt=corpus, userPrompt=question) (fresh query — no session resume)"]
|
||
S --> T["Return answer"]
|
||
```
|
||
- Copy from **06 Phase 13 task 3** (line 580): "Rewrite `KnowledgeAgent.query` to always pass `systemPrompt = renderedCorpus` to the SDK. Claude prompt-caching reduces cost when the same corpus is queried repeatedly within the 5-min TTL."
|
||
|
||
### (b) Docs
|
||
|
||
- 05 § 3.11 (lines 450–476), especially the Cost note (line 476).
|
||
- Live file:line targets:
|
||
- `src/services/worker/knowledge/KnowledgeAgent.ts` — new `query(corpus, question)` body. Copy the SDK-invocation pattern from the current `executeQuery` at `:185-230`, but with:
|
||
- `prompt: question` (user prompt)
|
||
- `options.systemPrompt: renderedCorpus` (new — load the corpus as system prompt)
|
||
- **Remove** `options.resume: corpus.session_id` (line 194)
|
||
- Keep `options.model`, `options.cwd`, `options.disallowedTools`, `options.pathToClaudeCodeExecutable`, `options.env` (lines 193, 195–198).
|
||
- `src/services/worker/knowledge/KnowledgeAgent.ts:14` — `import { CorpusRenderer }` already exists. Use it. The corpus-rendering call is the combination of `corpus.system_prompt` + `renderer.renderCorpus(corpus)`. Exact shape (copy from the current `prime` prompt at `KnowledgeAgent.ts:61-69`, minus the "Acknowledge" ending):
|
||
```
|
||
const systemPrompt = [
|
||
corpus.system_prompt,
|
||
'',
|
||
'Here is your complete knowledge base:',
|
||
'',
|
||
renderer.renderCorpus(corpus),
|
||
].join('\n');
|
||
```
|
||
- **Note for Phase 6**: `renderer.renderCorpus(corpus)` is the migration target for `renderObservations(obs, CorpusDetailStrategy)`. In this phase, call the existing renderer; Phase 6 swaps the internals.
|
||
- `src/services/worker/http/routes/CorpusRoutes.ts:235-261` — `handleQueryCorpus`. Keep the handler; change the response shape from `{answer, session_id}` (line 260) to `{answer}` only.
|
||
- `src/services/worker/knowledge/types.ts:53-56` — `QueryResult` narrowed to `{ answer: string }`.
|
||
|
||
### (c) Verification
|
||
|
||
- Send three queries against the same corpus within 5 min. Inspect SDK response usage (cache fields). Expected: call 1 writes full system prompt to the cache; calls 2 and 3 report `cache_read_input_tokens > 0`.
|
||
- `grep -n "resume:" src/services/worker/knowledge/KnowledgeAgent.ts` → zero.
|
||
- `grep -n "systemPrompt" src/services/worker/knowledge/KnowledgeAgent.ts` → exactly one occurrence (inside new `query`).
|
||
- Every `/query` call produces a subprocess with no `--resume` flag. Verify with `lsof` or SDK logs.
|
||
- End-to-end: `curl -X POST /api/corpus/foo/query -d '{"question":"What did we learn about Chroma?"}'` returns `{answer: "..."}` with no `session_id` field.
|
||
|
||
### (d) Anti-pattern guards
|
||
|
||
- **A**: The SDK option is `systemPrompt`; do not invent `systemMessage`, `initialContext`, or `primePrompt`. Verify the exact SDK option name in `@anthropic-ai/claude-agent-sdk` types before shipping.
|
||
- **C**: If `SDK.query` throws, propagate the error. No silent retry. No fallback to "cached answer".
|
||
- **E**: There is exactly one SDK-call site in the knowledge module after this phase — inside `KnowledgeAgent.query`. Anyone adding a second SDK call elsewhere in the module is introducing duplication.
|
||
|
||
---
|
||
|
||
## Phase 6 — Switch `CorpusBuilder` rendering to `renderObservations(obs, CorpusDetailStrategy)`
|
||
|
||
### (a) What to implement — Copy from …
|
||
|
||
- Copy from **05 § 3.11 line 457** (the clean flowchart node E): `E["renderObservations(obs, CorpusDetailStrategy)<br/>(U2 unified renderer)"]`.
|
||
- Copy from **05 Part 2 Decision D4** (line 75): "One renderer. `renderObservations(obs[], strategy)` where `strategy` selects columns, density, and grouping. The four existing formatters become four small strategy configs."
|
||
- Copy the `RenderStrategy` contract from **05 § 3.5 / 06 Phase 8** (06 lines 379–389).
|
||
|
||
### (b) Docs
|
||
|
||
- 05 § 3.11 (lines 450–476), 05 § 3.5, 05 Part 2 D4.
|
||
- **This plan depends on Plan 05-context-injection-engine** to have defined `CorpusDetailStrategy` at `src/services/rendering/renderObservations.ts` (path per 06 Phase 8 task 1, line 379). If Plan 05 has not shipped, this phase BLOCKS on it.
|
||
- Live file:line targets:
|
||
- `src/services/worker/knowledge/CorpusBuilder.ts:44` — `this.renderer = new CorpusRenderer();` constructor line. Replace with import of `renderObservations` and `CorpusDetailStrategy`.
|
||
- `src/services/worker/knowledge/CorpusBuilder.ts:109` — `corpus.system_prompt = this.renderer.generateSystemPrompt(corpus)`. Keep (the system-prompt *preamble* is distinct from the observation rendering). Or migrate to a separate strategy if 05 specifies one; 05 does not, so keep.
|
||
- `src/services/worker/knowledge/CorpusBuilder.ts:112` — `const renderedText = this.renderer.renderCorpus(corpus)`. Replace with `const renderedText = renderObservations(corpus.observations, CorpusDetailStrategy);`.
|
||
- `src/services/worker/knowledge/CorpusBuilder.ts:113` — `corpus.stats.token_estimate = this.renderer.estimateTokens(renderedText)`. Keep (token estimator is independent); if Plan 05 moves `estimateTokens` into the unified renderer's output, update.
|
||
- `src/services/worker/knowledge/KnowledgeAgent.ts` (Phase 5 rewrite) — swap `renderer.renderCorpus(corpus)` inside the query-time systemPrompt builder for `renderObservations(corpus.observations, CorpusDetailStrategy)`.
|
||
- `src/services/worker/knowledge/CorpusRenderer.ts` — after both call-sites migrate, delete `renderCorpus()` (lines 14–34) and `renderObservation()` (lines 39–85). Keep `generateSystemPrompt()` (lines 97–132) and `estimateTokens()` (lines 90–92) unless Plan 05 absorbs them. If nothing remains, delete the file; otherwise trim.
|
||
|
||
### (c) Verification
|
||
|
||
- `grep -n "renderCorpus\|renderObservation(" src/services/worker/knowledge/CorpusBuilder.ts` → zero.
|
||
- `grep -n "renderObservations" src/services/worker/knowledge/` → exactly two call-sites (CorpusBuilder and KnowledgeAgent).
|
||
- Snapshot test: feed the same fixture `CorpusObservation[]` to the old `CorpusRenderer.renderCorpus` and the new `renderObservations(obs, CorpusDetailStrategy)` call; assert byte-equal output (or diff in a controlled way documented in Plan 05's snapshot contract).
|
||
- `wc -l src/services/worker/knowledge/CorpusRenderer.ts` drops from 133 to roughly 40 (only `generateSystemPrompt` + `estimateTokens` remain, if they remain at all).
|
||
|
||
### (d) Anti-pattern guards
|
||
|
||
- **A**: The function name is `renderObservations` (plural), per 05 D4 and 06 Phase 8. Don't invent `renderCorpusObservations` or `renderForAgent`.
|
||
- **E**: After this phase, there is one traversal of `observations` in the knowledge module — inside `renderObservations`. Don't leave `renderObservation` (singular) as a helper in CorpusRenderer; Plan 05 owns it.
|
||
|
||
---
|
||
|
||
## Phase 7 — Verification (final)
|
||
|
||
### (a) What to implement — Copy from …
|
||
|
||
- Copy the verification pattern from **06 Phase 13 task 4 / verification block** (lines 581–588).
|
||
- Copy the cost-check from **05 § 3.11 Cost note** (line 476).
|
||
|
||
### (b) Docs
|
||
|
||
- 05 § 3.11 (lines 450–476).
|
||
- 06 Phase 13 (lines 567–597).
|
||
|
||
### (c) Verification
|
||
|
||
1. **Grep gauntlet** (exact commands):
|
||
- `grep -rn "session_id" src/services/worker/knowledge/` → **zero**.
|
||
- `grep -rn "session_id" src/services/worker/http/routes/CorpusRoutes.ts src/servers/mcp-server.ts` → zero for corpus/knowledge paths.
|
||
- `grep -rn "isSessionResumeError\|auto.?reprime\|session.*expired" src/services/worker/knowledge/` → zero.
|
||
- `grep -rn "/session|resume|expired|invalid.*session|not found/" src/services/worker/knowledge/` → zero (the exact regex string must be gone).
|
||
- `grep -rn "\.prime(\|\.reprime(" src/services/worker/knowledge/ src/servers/mcp-server.ts` → zero.
|
||
- `grep -rn "prime_corpus\|reprime_corpus" src/` → zero.
|
||
- `grep -rn "handlePrimeCorpus\|handleReprimeCorpus" src/` → zero.
|
||
2. **HTTP endpoints**:
|
||
- `POST /api/corpus` → 200, returns metadata.
|
||
- `POST /api/corpus/:name/rebuild` → 200.
|
||
- `POST /api/corpus/:name/query` → 200, `{answer: "..."}` only (no `session_id`).
|
||
- `DELETE /api/corpus/:name` → 200.
|
||
- `POST /api/corpus/:name/prime` → **404**.
|
||
- `POST /api/corpus/:name/reprime` → **404**.
|
||
3. **Cost smoke test** (per 05 line 476, "cached system prompt TTL is 5 min"):
|
||
- Build a 20-observation corpus.
|
||
- Run `POST /api/corpus/test/query` three times within 90 seconds, each with a different question.
|
||
- Record SDK response usage counters for each call. Expect: call 1 `cache_read_input_tokens == 0`; calls 2 and 3 `cache_read_input_tokens > 0` (approximately equal to the rendered corpus length in tokens).
|
||
- If no cache hits on calls 2–3, escalate to "Gaps" below — cost model is broken and the refactor must be revisited.
|
||
4. **corpus.json on disk**:
|
||
- `cat ~/.claude-mem/corpora/test.corpus.json | jq 'has("session_id")'` → `false`.
|
||
- `jq '.version'` → `2`.
|
||
5. **Line-count delta** (target from 05 Part 5 line 556: net -110 LOC for this area):
|
||
- Before: KnowledgeAgent 284 + CorpusStore 127 + CorpusBuilder 174 + CorpusRenderer 133 + CorpusRoutes 283 = **1001 lines** in the five files.
|
||
- After: roughly -75 (prime+reprime) -10 (CorpusStore `session_id` fields) -40 (auto-reprime + regex + executeQuery body) -40 (prime+reprime HTTP handlers) -93 (CorpusRenderer renderCorpus+renderObservation shift to shared renderer) +30 (new slim query() using systemPrompt). Net ≈ **-228**.
|
||
- 05 Part 5 promised -110; actual deletion is larger because the audit underweighted the CorpusRenderer migration credit (it's also double-counted in Plan 08/unified-renderer).
|
||
6. **Full `npm run build-and-sync`** passes.
|
||
7. **MCP tool listing** no longer exposes `prime_corpus` or `reprime_corpus`.
|
||
|
||
### (d) Anti-pattern guards
|
||
|
||
- **A**: Every grep that returns a non-zero match is a failed phase. No "we'll clean it up later" waivers.
|
||
- **B**: If the cost smoke test fails (no cache hits on call 2/3), do not "fix" by reintroducing session-resume. Investigate the SDK's prompt-caching behavior and file the bug.
|
||
- **C**: Any handler that silently returns a cached answer without calling the SDK is a regression. Every `/query` must invoke the SDK.
|
||
|
||
---
|
||
|
||
## Blast radius + migration
|
||
|
||
- **corpus.json schema**: `version: 1` → `version: 2`. Old files with `session_id` still parse because TypeScript structural casting is permissive on reads; extra field is ignored, never re-emitted. No explicit migration script — corpus files are rebuilt on `/rebuild` anyway.
|
||
- **MCP surface shrinks**: downstream users of the MCP search plugin lose `prime_corpus` and `reprime_corpus` tool names. Coordinate with plugin release notes.
|
||
- **Cost profile**: depends on SDK prompt-caching TTL (5 min). See Gap 1 below.
|
||
|
||
## Confidence + Gaps
|
||
|
||
**Confidence — High**:
|
||
- All deletion targets have exact file:line references verified against live code.
|
||
- The 06 Phase 13 verification steps align 1:1 with 05 § 3.11 deletion list.
|
||
- Every HTTP and MCP endpoint has been mapped to a specific line in `CorpusRoutes.ts` or `mcp-server.ts`.
|
||
|
||
**Gap 1 (flagged per prompt — prompt-caching TTL)**: 05 line 476 asserts "cached system prompt TTL is 5 min" → cost roughly equal to session-resume. **This is an assumption**, not a measured fact. If the Claude Agent SDK's caching hits on `systemPrompt` behave differently than expected (e.g., cache key sensitive to small whitespace changes in the rendered corpus; cache disabled when `options.cwd` varies; TTL shorter than 5 min), every `/query` becomes a full prompt-ingest — per-call cost jumps ~20×. **Required**: Phase 7 step 3 (the cost smoke test) must run and the cache-hit ratio must be logged before declaring the phase shipped. If cache miss rate > 10% on repeat queries within 5 min, escalate.
|
||
|
||
**Gap 2 — corpus.json storage shape**: 06 Phase 13 task 2 (line 579) suggests `{name, filters, renderedCorpus, generatedAt}` — storing the fully-rendered string instead of observations. This plan keeps observations because `renderObservations(obs, CorpusDetailStrategy)` is recomputed per query (Phase 5). Tradeoff: storing `renderedCorpus` saves one render per query (small) but loses the ability to change strategies without a rebuild. **Decision deferred**: ship Phase 1–7 with observations preserved; reopen if Plan 05 lands and stores `renderedCorpus` directly.
|
||
|
||
---
|
||
|
||
## Phase Count
|
||
|
||
**7 phases**: schema cleanup → `prime` deletion → auto-reprime deletion → endpoint deletion → `/query` rewrite → renderer unification → verification.
|
||
|
||
## Anticipated LOC Impact
|
||
|
||
- 05 Part 5 row 19 (line 556): `-140 / +30 / net -110`.
|
||
- This plan's line-by-line trace (see Phase 7 step 5): actual net deletion closer to **-228** once the `CorpusRenderer` shrink lands.
|
||
- Five files touched: `KnowledgeAgent.ts`, `CorpusStore.ts`, `CorpusBuilder.ts`, `CorpusRenderer.ts`, `CorpusRoutes.ts`, plus `mcp-server.ts` and `types.ts` edits.
|