* docs: pathfinder refactor corpus + Node 20 preflight
Adds the PATHFINDER-2026-04-22 principle-driven refactor plan (11 docs,
cross-checked PASS) plus the exploratory PATHFINDER-2026-04-21 corpus
that motivated it. Bumps engines.node to >=20.0.0 per the ingestion-path
plan preflight (recursive fs.watch). Adds the pathfinder skill.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: land PATHFINDER Plan 01 — data integrity
Schema, UNIQUE constraints, self-healing claim, Chroma upsert fallback.
- Phase 1: fresh schema.sql regenerated at post-refactor shape.
- Phase 2: migrations 23+24 — rebuild pending_messages without
started_processing_at_epoch; UNIQUE(session_id, tool_use_id);
UNIQUE(memory_session_id, content_hash) on observations; dedup
duplicate rows before adding indexes.
- Phase 3: claimNextMessage rewritten to self-healing query using
worker_pid NOT IN live_worker_pids; STALE_PROCESSING_THRESHOLD_MS
and the 60-s stale-reset block deleted.
- Phase 4: DEDUP_WINDOW_MS and findDuplicateObservation deleted;
observations.insert now uses ON CONFLICT DO NOTHING.
- Phase 5: failed-message purge block deleted from worker-service
2-min interval; clearFailedOlderThan method deleted.
- Phase 6: repairMalformedSchema and its Python subprocess repair
path deleted from Database.ts; SQLite errors now propagate.
- Phase 7: Chroma delete-then-add fallback gated behind
CHROMA_SYNC_FALLBACK_ON_CONFLICT env flag as bridge until
Chroma MCP ships native upsert.
- Phase 8: migration 19 no-op block absorbed into fresh schema.sql.
Verification greps all return 0 matches. bun test tests/sqlite/
passes 63/63. bun run build succeeds.
Plan: PATHFINDER-2026-04-22/01-data-integrity.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: land PATHFINDER Plan 02 — process lifecycle
OS process groups replace hand-rolled reapers. Worker runs until
killed; orphans are prevented by detached spawn + kill(-pgid).
- Phase 1: src/services/worker/ProcessRegistry.ts DELETED. The
canonical registry at src/supervisor/process-registry.ts is the
sole survivor; SDK spawn site consolidated into it via new
createSdkSpawnFactory/spawnSdkProcess/getSdkProcessForSession/
ensureSdkProcessExit/waitForSlot helpers.
- Phase 2: SDK children spawn with detached:true + stdio:
['ignore','pipe','pipe']; pgid recorded on ManagedProcessInfo.
- Phase 3: shutdown.ts signalProcess teardown uses
process.kill(-pgid, signal) on Unix when pgid is recorded;
Windows path unchanged (tree-kill/taskkill).
- Phase 4: all reaper intervals deleted — startOrphanReaper call,
staleSessionReaperInterval setInterval (including the co-located
WAL checkpoint — SQLite's built-in wal_autocheckpoint handles
WAL growth without an app-level timer), killIdleDaemonChildren,
killSystemOrphans, reapOrphanedProcesses, reapStaleSessions, and
detectStaleGenerator. MAX_GENERATOR_IDLE_MS and MAX_SESSION_IDLE_MS
constants deleted.
- Phase 5: abandonedTimer — already 0 matches; primary-path cleanup
via generatorPromise.finally() already lives in worker-service
startSessionProcessor and SessionRoutes ensureGeneratorRunning.
- Phase 6: evictIdlestSession and its evict callback deleted from
SessionManager. Pool admission gates backpressure upstream.
- Phase 7: SDK-failure fallback — SessionManager has zero matches
for fallbackAgent/Gemini/OpenRouter. Failures surface to hooks
via exit code 2 through SessionRoutes error mapping.
- Phase 8: ensureWorkerRunning in worker-utils.ts rewritten to
lazy-spawn — consults isWorkerPortAlive (which gates
captureProcessStartToken for PID-reuse safety via commit
99060bac), then spawns detached with unref(), then
waitForWorkerPort({ attempts: 3, backoffMs: 250 }) hand-rolled
exponential backoff 250→500→1000ms. No respawn npm dep.
- Phase 9: idle self-shutdown — zero matches for
idleCheck/idleTimeout/IDLE_MAX_MS/idleShutdown. Worker exits
only on external SIGTERM via supervisor signal handlers.
Three test files that exercised deleted code removed:
tests/worker/process-registry.test.ts,
tests/worker/session-lifecycle-guard.test.ts,
tests/services/worker/reap-stale-sessions.test.ts.
Pass count: 1451 → 1407 (-44), all attributable to deleted test
files. Zero new failures. 31 pre-existing failures remain
(schema-repair suite, logger-usage-standards, environmental
openclaw / plugin-distribution) — none introduced by Plan 02.
All 10 verification greps return 0. bun run build succeeds.
Plan: PATHFINDER-2026-04-22/02-process-lifecycle.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: land PATHFINDER Plan 04 (narrowed) — search fail-fast
Phases 3, 5, 6 only. Plan-doc inaccuracies for phases 1/2/4/7/8/9
deferred for plan reconciliation:
- Phase 1/2: ObservationRow type doesn't exist; the four
"formatters" operate on three incompatible types.
- Phase 4: RECENCY_WINDOW_MS already imported from
SEARCH_CONSTANTS at every call site.
- Phase 7: getExistingChromaIds is NOT @deprecated and has an
active caller in ChromaSync.backfillMissingSyncs.
- Phase 8: estimateTokens already consolidated.
- Phase 9: knowledge-corpus rewrite blocked on PG-3
prompt-caching cost smoke test.
Phase 3 — Delete SearchManager.findByConcept/findByFile/findByType.
SearchRoutes handlers (handleSearchByConcept/File/Type) now call
searchManager.getOrchestrator().findByXxx() directly via new
getter accessors on SearchManager. ~250 LoC deleted.
Phase 5 — Fail-fast Chroma. Created
src/services/worker/search/errors.ts with ChromaUnavailableError
extends AppError(503, 'CHROMA_UNAVAILABLE'). Deleted
SearchOrchestrator.executeWithFallback's Chroma-failed
SQLite-fallback branch; runtime Chroma errors now throw 503.
"Path 3" (chromaSync was null at construction — explicit-
uninitialized config) preserved as legitimate empty-result state
per plan text. ChromaSearchStrategy.search no longer wraps in
try/catch — errors propagate.
Phase 6 — Delete HybridSearchStrategy three try/catch silent
fallback blocks (findByConcept, findByType, findByFile) at lines
~82-95, ~120-132, ~161-172. Removed `fellBack` field from
StrategySearchResult type and every return site
(SQLiteSearchStrategy, BaseSearchStrategy.emptyResult,
SearchOrchestrator).
Tests updated (Principle 7 — delete in same PR):
- search-orchestrator.test.ts: "fall back to SQLite" rewritten
as "throw ChromaUnavailableError (HTTP 503)".
- chroma/hybrid/sqlite-search-strategy tests: rewritten to
rejects.toThrow; removed fellBack assertions.
Verification: SearchManager.findBy → 0; fellBack → 0 in src/.
bun test tests/worker/search/ → 122 pass, 0 fail.
bun test (suite-wide) → 1407 pass, baseline maintained, 0 new
failures. bun run build succeeds.
Plan: PATHFINDER-2026-04-22/04-read-path.md (Phases 3, 5, 6)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: land PATHFINDER Plan 03 — ingestion path
Fail-fast parser, direct in-process ingest, recursive fs.watch,
DB-backed tool pairing. Worker-internal HTTP loopback eliminated.
- Phase 0: Created src/services/worker/http/shared.ts exporting
ingestObservation/ingestPrompt/ingestSummary as direct
in-process functions plus ingestEventBus (Node EventEmitter,
reusing existing pattern — no third event bus introduced).
setIngestContext wires the SessionManager dependency from
worker-service constructor.
- Phase 1: src/sdk/parser.ts collapsed to one parseAgentXml
returning { valid:true; kind: 'observation'|'summary'; data }
| { valid:false; reason: string }. Inspects root element;
<skip_summary reason="…"/> is a first-class summary case
with skipped:true. NEVER returns undefined. NEVER coerces.
- Phase 2: ResponseProcessor calls parseAgentXml exactly once,
branches on the discriminated union. On invalid → markFailed
+ logger.warn(reason). On observation → ingestObservation.
On summary → ingestSummary then emit summaryStoredEvent
{ sessionId, messageId } (consumed by Plan 05's blocking
/api/session/end).
- Phase 3: Deleted consecutiveSummaryFailures field
(ResponseProcessor + SessionManager + worker-types) and
MAX_CONSECUTIVE_SUMMARY_FAILURES constant. Circuit-breaker
guards and "tripped" log lines removed.
- Phase 4: coerceObservationToSummary deleted from sdk/parser.ts.
- Phase 5: src/services/transcripts/watcher.ts rescan setInterval
replaced with fs.watch(transcriptsRoot, { recursive: true,
persistent: true }) — Node 20+ recursive mode.
- Phase 6: src/services/transcripts/processor.ts pendingTools
Map deleted. tool_use rows insert with INSERT OR IGNORE on
UNIQUE(session_id, tool_use_id) (added by Plan 01). New
pairToolUsesByJoin query in PendingMessageStore for read-time
pairing (UNIQUE INDEX provides idempotency; explicit consumer
not yet wired).
- Phase 7: HTTP loopback at processor.ts:252 replaced with
direct ingestObservation call. maybeParseJson silent-passthrough
rewritten to fail-fast (throws on malformed JSON).
- Phase 8: src/utils/tag-stripping.ts countTags + stripTagsInternal
collapsed into one alternation regex, single-pass over input.
- Phase 9: src/utils/transcript-parser.ts (dead TranscriptParser
class) deleted. The active extractLastMessage at
src/shared/transcript-parser.ts:41-144 is the sole survivor.
Tests updated (Principle 7 — same-PR delete):
- tests/sdk/parser.test.ts + parse-summary.test.ts: rewritten
to assert discriminated-union shape; coercion-specific
scenarios collapse into { valid:false } assertions.
- tests/worker/agents/response-processor.test.ts: circuit-breaker
describe block skipped; non-XML/empty-response tests assert
fail-fast markFailed behavior.
Verification: every grep returns 0. transcript-parser.ts deleted.
bun run build succeeds. bun test → 1399 pass / 28 fail / 7 skip
(net -8 pass = the 4 retired circuit-breaker tests + 4 collapsed
parser cases). Zero new failures vs baseline.
Deferred (out of Plan 03 scope, will land in Plan 06): SessionRoutes
HTTP route handlers still call sessionManager.queueObservation
inline rather than the new shared helpers — the helpers are ready,
the route swap is mechanical and belongs with the Zod refactor.
Plan: PATHFINDER-2026-04-22/03-ingestion-path.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: land PATHFINDER Plan 05 — hook surface
Worker-call plumbing collapsed to one helper. Polling replaced by
server-side blocking endpoint. Fail-loud counter surfaces persistent
worker outages via exit code 2.
- Phase 1: plugin/hooks/hooks.json — three 20-iteration `for i in
1..20; do curl -sf .../health && break; sleep 0.1; done` shell
retry wrappers deleted. Hook commands invoke their bun entry
point directly.
- Phase 2: src/shared/worker-utils.ts — added
executeWithWorkerFallback<T>(url, method, body) returning
T | { continue: true; reason?: string }. All 8 hook handlers
(observation, session-init, context, file-context, file-edit,
summarize, session-complete, user-message) rewritten to use
it instead of duplicating the ensureWorkerRunning →
workerHttpRequest → fallback sequence.
- Phase 3: blocking POST /api/session/end in SessionRoutes.ts
using validateBody + sessionEndSchema (z.object({sessionId})).
One-shot ingestEventBus.on('summaryStoredEvent') listener,
30 s timer, req.aborted handler — all share one cleanup so
the listener cannot leak. summarize.ts polling loop, plus
MAX_WAIT_FOR_SUMMARY_MS / POLL_INTERVAL_MS constants, deleted.
- Phase 4: src/shared/hook-settings.ts — loadFromFileOnce()
memoizes SettingsDefaultsManager.loadFromFile per process.
Per-handler settings reads collapsed.
- Phase 5: src/shared/should-track-project.ts — single exclusion
check entry; isProjectExcluded no longer referenced from
src/cli/handlers/.
- Phase 6: cwd validation pushed into adapter normalizeInput
(all 6 adapters: claude-code, cursor, raw, gemini-cli,
windsurf). New AdapterRejectedInput error in
src/cli/adapters/errors.ts. Handler-level isValidCwd checks
deleted from file-edit.ts and observation.ts. hook-command.ts
catches AdapterRejectedInput → graceful fallback.
- Phase 7: session-init.ts conditional initAgent guard deleted;
initAgent is idempotent. tests/hooks/context-reinjection-guard
test (validated the deleted conditional) deleted in same PR
per Principle 7.
- Phase 8: fail-loud counter at ~/.claude-mem/state/hook-failures
.json. Atomic write via .tmp + rename. CLAUDE_MEM_HOOK_FAIL_LOUD
_THRESHOLD setting (default 3). On consecutive worker-unreachable
≥ N: process.exit(2). On success: reset to 0. NOT a retry.
- Phase 9: ensureWorkerAliveOnce() module-scope memoization
wrapping ensureWorkerRunning. executeWithWorkerFallback calls
the memoized version.
Minimal validateBody middleware stub at
src/services/worker/http/middleware/validateBody.ts. Plan 06 will
expand with typed inference + error envelope conventions.
Verification: 4/4 grep targets pass. bun run build succeeds.
bun test → 1393 pass / 28 fail / 7 skip; -6 pass attributable
solely to deleted context-reinjection-guard test file. Zero new
failures vs baseline.
Plan: PATHFINDER-2026-04-22/05-hook-surface.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: land PATHFINDER Plan 06 — API surface
One Zod-based validator wrapping every POST/PUT. Rate limiter,
diagnostic endpoints, and shutdown wrappers deleted. Failure-
marking consolidated to one helper.
- Phase 1 (preflight): zod@^3 already installed.
- Phase 2: validateBody middleware confirmed at canonical shape
in src/services/worker/http/middleware/validateBody.ts —
safeParse → 400 { error: 'ValidationError', issues: [...] }
on failure, replaces req.body with parsed value on success.
- Phase 3: Per-route Zod schemas declared at the top of each
route file. 24 POST endpoints across SessionRoutes,
CorpusRoutes, DataRoutes, MemoryRoutes, SearchRoutes,
LogsRoutes, SettingsRoutes now wrap with validateBody().
/api/session/end (Plan 05) confirmed using same middleware.
- Phase 4: validateRequired() deleted from BaseRouteHandler
along with every call site. Inline coercion helpers
(coerceStringArray, coercePositiveInteger) and inline
if (!req.body...) guards deleted across all route files.
- Phase 5: Rate limiter middleware and its registration deleted
from src/services/worker/http/middleware.ts. Worker binds
127.0.0.1:37777 — no untrusted caller.
- Phase 6: viewer.html cached at module init in ViewerRoutes.ts
via fs.readFileSync; served as Buffer with text/html content
type. SKILL.md + per-operation .md files cached in
Server.ts as Map<string, string>; loadInstructionContent
helper deleted. NO fs.watch, NO TTL — process restart is the
cache-invalidation event.
- Phase 7: Four diagnostic endpoints deleted from DataRoutes.ts
— /api/pending-queue (GET), /api/pending-queue/process (POST),
/api/pending-queue/failed (DELETE), /api/pending-queue/all
(DELETE). Helper methods that ONLY served them
(getQueueMessages, getStuckCount, getRecentlyProcessed,
clearFailed, clearAll) deleted from PendingMessageStore.
KEPT: /api/processing-status (observability), /health
(used by ensureWorkerRunning).
- Phase 8: stopSupervisor wrapper deleted from supervisor/index.ts.
GracefulShutdown now calls getSupervisor().stop() directly.
Two functions retained with clear roles:
- performGracefulShutdown — worker-side 6-step shutdown
- runShutdownCascade — supervisor-side child teardown
(process.kill(-pgid), Windows tree-kill, PID-file cleanup)
Each has unique non-trivial logic and a single canonical caller.
- Phase 9: transitionMessagesTo(status, filter) is the sole
failure-marking path on PendingMessageStore. Old methods
markSessionMessagesFailed and markAllSessionMessagesAbandoned
deleted along with all callers (worker-service,
SessionCompletionHandler, tests/zombie-prevention).
Tests updated (Principle 7 same-PR delete): coercion test files
refactored to chain validateBody → handler. Zombie-prevention
tests rewritten to call transitionMessagesTo.
Verification: all 4 grep targets → 0. bun run build succeeds.
bun test → 1393 pass / 28 fail / 7 skip — exact match to
baseline. Zero new failures.
Plan: PATHFINDER-2026-04-22/06-api-surface.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: land PATHFINDER Plan 07 — dead code sweep
ts-prune-driven sweep across the tree after Plans 01-06 landed.
Deleted unused exports, orphan helpers, and one fully orphaned
file. Earlier-plan deletions verified.
Deleted:
- src/utils/bun-path.ts (entire file — getBunPath, getBunPathOrThrow,
isBunAvailable: zero importers)
- bun-resolver.getBunVersionString: zero callers
- PendingMessageStore.retryMessage / resetProcessingToPending /
abortMessage: superseded by transitionMessagesTo (Plan 06 Phase 9)
- EnvManager.MANAGED_CREDENTIAL_KEYS, EnvManager.setCredential:
zero callers
- CodexCliInstaller.checkCodexCliStatus: zero callers; no status
command exists in npx-cli
- Two "REMOVED: cleanupOrphanedSessions" stale-fence comments
Kept (with documented justification):
- Public API surface in dist/sdk/* (parseAgentXml, prompt
builders, ParsedObservation, ParsedSummary, ParseResult,
SUMMARY_MODE_MARKER) — exported via package.json sdk path.
- generateContext / loadContextConfig / token utilities — used
via dynamic await import('../../../context-generator.js') in
worker SearchRoutes.
- MCP_IDE_INSTALLERS, install/uninstall functions for codex/goose
— used via dynamic await import in npx-cli/install.ts +
uninstall.ts (ts-prune cannot trace dynamic imports).
- getExistingChromaIds — active caller in
ChromaSync.backfillMissingSyncs (Plan 04 narrowed scope).
- processPendingQueues / getSessionsWithPendingMessages — active
orphan-recovery caller in worker-service.ts plus
zombie-prevention test coverage.
- StoreAndMarkCompleteResult legacy alias — return-type annotation
in same file.
- All Database.ts barrel re-exports — used downstream.
Earlier-plan verification:
- Plan 03 Phase 9: VERIFIED — src/utils/transcript-parser.ts
is gone; TranscriptParser has 0 references in src/.
- Plan 01 Phase 8: VERIFIED — migration 19 no-op absorbed.
- SessionStore.ts:52-70 consolidation NOT executed (deferred):
the methods are not thin wrappers but ~900 LoC of bodies, and
two methods are documented as intentional mirrors so the
context-generator.cjs bundle stays schema-consistent without
pulling MigrationRunner. Deserves its own plan, not a sweep.
Verification: TranscriptParser → 0; transcript-parser.ts → gone;
no commented-out code markers remain. bun run build succeeds.
bun test → 1393 pass / 28 fail / 7 skip — EXACT match to
baseline. Zero regressions.
Plan: PATHFINDER-2026-04-22/07-dead-code.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: remove residual ProcessRegistry comment reference
Plan 07 dead-code sweep missed one comment-level reference to the
deleted in-memory ProcessRegistry class in SessionManager.ts:347.
Rewritten to describe the supervisor.json scope without naming the
deleted class, completing the verification grep target.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: address Greptile review (P1 + 2× P2)
P1 — Plan 05 Phase 3 blocking endpoint was non-functional:
executeWithWorkerFallback used HEALTH_CHECK_TIMEOUT_MS (3 s) for
the POST /api/session/end call, but the server holds the
connection for SERVER_SIDE_SUMMARY_TIMEOUT_MS (30 s). Client
always raced to a "timed out" rejection that isWorkerUnavailable
classified as worker-unreachable, so the hook silently degraded
instead of waiting for summaryStoredEvent.
- Added optional timeoutMs to executeWithWorkerFallback,
forwarded to workerHttpRequest.
- summarize.ts call site now passes 35_000 (5 s above server
hold window).
P2 — ingestSummary({ kind: 'parsed' }) branch was dead code:
ResponseProcessor emitted summaryStoredEvent directly via the
event bus, bypassing the centralized helper that the comment
claimed was the single source.
- ResponseProcessor now calls ingestSummary({ kind: 'parsed',
sessionDbId, messageId, contentSessionId, parsed }) so the
event-emission path is single-sourced.
- ingestSummary's requireContext() resolution moved inside the
'queue' branch (the only branch that needs sessionManager /
dbManager). 'parsed' is a pure event-bus emission and
doesn't need worker-internal context — fixes mocked
ResponseProcessor unit tests that don't call
setIngestContext.
P2 — isWorkerFallback could false-positive on legitimate API
responses whose schema includes { continue: true, ... }:
- Added a Symbol.for('claude-mem/worker-fallback') brand to
WorkerFallback. isWorkerFallback now checks the brand, not
a duck-typed property name.
Verification: bun run build succeeds. bun test → 1393 pass /
28 fail / 7 skip — exact baseline match. Zero new failures.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: address Greptile iteration 2 (P1 + P2)
P1 — summaryStoredEvent fired regardless of whether the row was
persisted. ResponseProcessor's call to ingestSummary({ kind:
'parsed' }) ran for every parsed.kind === 'summary' even when
result.summaryId came back null (e.g. FK violation, null
memory_session_id at commit). The blocking /api/session/end
endpoint then returned { ok: true } and the Stop hook logged
'Summary stored' for a non-existent row.
- Gate ingestSummary call on (parsed.data.skipped ||
session.lastSummaryStored). Skipped summaries are an explicit
no-op bypass and still confirm; real summaries only confirm
when storage actually wrote a row.
- Non-skipped + summaryId === null path logs a warn and lets
the server-side timeout (504) surface to the hook instead of
a false ok:true.
P2 — PendingMessageStore.enqueue() returns 0 when INSERT OR
IGNORE suppresses a duplicate (the UNIQUE(session_id, tool_use_id)
constraint added by Plan 01 Phase 1). The two callers
(SessionManager.queueObservation and queueSummarize) previously
logged 'ENQUEUED messageId=0' which read like a row was inserted.
- Branch on messageId === 0 and emit a 'DUP_SUPPRESSED' debug
log instead of the misleading ENQUEUED line. No behavior
change — the duplicate is still correctly suppressed by the
DB (Principle 3); only the log surface is corrected.
- confirmProcessed is never called with the enqueue() return
value (it operates on session.processingMessageIds[] from
claimNextMessage), so no caller is broken; the visibility
fix prevents future misuse.
Verification: bun run build succeeds. bun test → 1393 pass /
28 fail / 7 skip — exact baseline match. Zero new failures.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: address Greptile iteration 3 (P1 + 2× P2)
- P1 worker-service.ts: wire ensureGeneratorRunning into the ingest
context after SessionRoutes is constructed. setIngestContext runs
before routes exist, so transcript-watcher observations queued via
ingestObservation() had no way to auto-start the SDK generator.
Added attachIngestGeneratorStarter() to patch the callback in.
- P2 shared.ts: IngestEventBus now sets maxListeners to 0. Concurrent
/api/session/end calls register one listener each and clean up on
completion, so the default-10 warning fires spuriously under normal
load.
- P2 SessionRoutes.ts: handleObservationsByClaudeId now delegates to
ingestObservation() instead of duplicating skip-tool / meta /
privacy / queue logic. Single helper, matching the Plan 03 goal.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: address Greptile iteration 4 (P1 tool-pair + P2 parse/path/doc)
- processor.handleToolResult: restore in-memory tool-use→tool-result
pairing via session.pendingTools for schemas (e.g. Codex) whose
tool_result events carry only tool_use_id + output. Without this,
neither handler fired — all tool observations silently dropped.
- processor.maybeParseJson: return raw string on parse failure instead
of throwing. Previously a single malformed JSON-shaped field caused
handleLine's outer catch to discard the entire transcript line.
- watcher.deepestNonGlobAncestor: split on / and \\, emit empty string
for purely-glob inputs so the caller skips the watch instead of
anchoring fs.watch at the filesystem root. Windows-compatible.
- PendingMessageStore.enqueue: tighten docstring — callers today only
log on the returned id; the SessionManager branches on id === 0.
* fix: forward tool_use_id through ingestObservation (Greptile iter 5)
P1 — Plan 01's UNIQUE(content_session_id, tool_use_id) dedup never
fired because the new shared ingest path dropped the toolUseId before
queueObservation. SQLite treats NULL values as distinct for UNIQUE,
so every replayed transcript line landed a duplicate row.
- shared.ingestObservation: forward payload.toolUseId to
queueObservation so INSERT OR IGNORE can actually collapse.
- SessionRoutes.handleObservationsByClaudeId: destructure both
tool_use_id (HTTP convention) and toolUseId (JS convention) from
req.body and pass into ingestObservation.
- observationsByClaudeIdSchema: declare both keys explicitly so the
validator doesn't rely on .passthrough() alone.
* fix: drop dead pairToolUsesByJoin, close session-end listener race
- PendingMessageStore: delete pairToolUsesByJoin. The method was never
called and its self-join semantics are structurally incompatible
with UNIQUE(content_session_id, tool_use_id): INSERT OR IGNORE
collapses any second row with the same pair, so a self-join can
only ever match a row to itself. In-memory pendingTools in
processor.ts remains the pairing path for split-event schemas.
- IngestEventBus: retain a short-lived (60s) recentStored map keyed
by sessionId. Populated on summaryStoredEvent emit, evicted on
consume or TTL.
- handleSessionEnd: drain the recent-events buffer before attaching
the listener. Closes the register-after-emit race where the summary
can persist between the hook's summarize POST and its session/end
POST — previously that window returned 504 after the 30s timeout.
* chore: merge origin/main into vivacious-teeth
Resolves conflicts with 15 commits on main (v12.3.9, security
observation types, Telegram notifier, PID-reuse worker start-guard).
Conflict resolution strategy:
- plugin/hooks/hooks.json, plugin/scripts/*.cjs, plugin/ui/viewer-bundle.js:
kept ours — PATHFINDER Plan 05 deletes the for-i-in-1-to-20 curl retry
loops and the built artifacts regenerate on build.
- src/cli/handlers/summarize.ts: kept ours — Plan 05 blocking
POST /api/session/end supersedes main's fire-and-forget path.
- src/services/worker-service.ts: kept ours — Plan 05 ingest bus +
summaryStoredEvent supersedes main's SessionCompletionHandler DI
refactor + orphan-reaper fallback.
- src/services/worker/http/routes/SessionRoutes.ts: kept ours — same
reason; generator .finally() Stop-hook self-clean is a guard for a
path our blocking endpoint removes.
- src/services/worker/http/routes/CorpusRoutes.ts: merged — added
security_alert / security_note to ALLOWED_CORPUS_TYPES (feature from
#2084) while preserving our Zod validateBody schema.
Typecheck: 294 errors (vs 298 pre-merge). No new errors introduced; all
remaining are pre-existing (Component-enum gaps, DOM lib for viewer,
bun:sqlite types).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: address Greptile P2 findings
1) SessionRoutes.handleSessionEnd was the only route handler not wrapped
in wrapHandler — synchronous exceptions would hang the client rather
than surfacing as 500s. Wrap it like every other handler.
2) processor.handleToolResult only consumed the session.pendingTools
entry when the tool_result arrived without a toolName. In the
split-schema path where tool_result carries both toolName and toolId,
the entry was never deleted and the map grew for the life of the
session. Consume the entry whenever toolId is present.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: typing cleanup and viewer tsconfig split for PR feedback
- Add explicit return types for SessionStore query methods
- Exclude src/ui/viewer from root tsconfig, give it its own DOM-typed config
- Add bun to root tsconfig types, plus misc typing tweaks flagged by Greptile
- Rebuilt plugin/scripts/* artifacts
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: address Greptile P2 findings (iter 2)
- PendingMessageStore.transitionMessagesTo: require sessionDbId (drop
the unscoped-drain branch that would nuke every pending/processing
row across all sessions if a future caller omitted the filter).
- IngestEventBus.takeRecentSummaryStored: make idempotent — keep the
cached event until TTL eviction so a retried Stop hook's second
/api/session/end returns immediately instead of hanging 30 s.
- TranscriptWatcher fs.watch callback: skip full glob scan for paths
already tailed (JSONL appends fire on every line; only unknown
paths warrant a rescan).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: call finalizeSession in terminal session paths (Greptile iter 3)
terminateSession and runFallbackForTerminatedSession previously called
SessionCompletionHandler.finalizeSession before removeSessionImmediate;
the refactor dropped those calls, leaving sdk_sessions.status='active'
for every session killed by wall-clock limit, unrecoverable error, or
exhausted fallback chain. The deleted reapStaleSessions interval was
the only prior backstop.
Re-wires finalizeSession (idempotent: marks completed, drains pending,
broadcasts) into both paths; no reaper reintroduced.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: GC failed pending_messages rows at startup (Greptile iter 4)
Plan 07 deleted clearFailed/clearFailedOlderThan as "dead code", but
with the periodic sweep also removed, nothing reaps status='failed'
rows now — they accumulate indefinitely. Since claimNextMessage's
self-healing subquery scans this table, unbounded growth degrades
claim latency over time.
Re-introduces clearFailedOlderThan and calls it once at worker startup
(not a reaper — one-shot, idempotent). 7-day retention keeps enough
history for operator inspection while bounding the table.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: finalize sessions on normal exit; cleanup hoist; share handler (iter 5)
1. startSessionProcessor success branch now calls completionHandler.
finalizeSession before removeSessionImmediate. Hooks-disabled installs
(and any Stop hook that fails before POST /api/sessions/complete) no
longer leave sdk_sessions rows as status='active' forever. Idempotent
— a subsequent /api/sessions/complete is a no-op.
2. Hoist SessionRoutes.handleSessionEnd cleanup declaration above the
closures that reference it (TDZ safety; safe at runtime today but
fragile if timeout ever shrinks).
3. SessionRoutes now receives WorkerService's shared SessionCompletionHandler
instead of constructing its own — prevents silent divergence if the
handler ever becomes stateful.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: stop runaway crash-recovery loop on dead sessions
Two distinct bugs were combining to keep a dead session restarting forever:
Bug 1 (uncaught "The operation was aborted."):
child_process.spawn emits 'error' asynchronously for ENOENT/EACCES/abort
signal aborts. spawnSdkProcess() never attached an 'error' listener, so
any async spawn failure became uncaughtException and escaped to the
daemon-level handler. Attach an 'error' listener immediately after spawn,
before the !child.pid early-return, so async spawn errors are logged
(with errno code) and swallowed locally.
Bug 2 (sliding-window limiter never trips on slow restart cadence):
RestartGuard tripped only when restartTimestamps.length exceeded
MAX_WINDOWED_RESTARTS (10) within RESTART_WINDOW_MS (60s). With the 8s
exponential-backoff cap, only ~7-8 restarts fit in the window, so a dead
session that fail-restart-fail-restart on 8s cycles would loop forever
(consecutiveRestarts climbing past 30+ in observed logs). Add a
consecutiveFailures counter that increments on every restart and resets
only on recordSuccess(). Trip when consecutive failures exceed
MAX_CONSECUTIVE_FAILURES (5) — meaning 5 restarts with zero successful
processing in between proves the session is dead. Both guards now run in
parallel: tight loops still trip the windowed cap; slow loops trip the
consecutive-failure cap.
Also: when the SessionRoutes path trips the guard, drain pending messages
to 'abandoned' so the session does not reappear in
getSessionsWithPendingMessages and trigger another auto-start cycle. The
worker-service.ts path already does this via terminateSession.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* perf: streamline worker startup and consolidate database connections
1. Database Pooling: Modified DatabaseManager, SessionStore, and SessionSearch to share a single bun:sqlite connection, eliminating redundant file descriptors.
2. Non-blocking Startup: Refactored WorktreeAdoption and Chroma backfill to run in the background (fire-and-forget), preventing them from stalling core initialization.
3. Diagnostic Routes: Added /api/chroma/status and bypassed the initialization guard for health/readiness endpoints to allow diagnostics during startup.
4. Robust Search: Implemented reliable SQLite FTS5 fallback in SearchManager for when Chroma (uvx) fails or is unavailable.
5. Code Cleanup: Removed redundant loopback MCP checks and mangled initialization logic from WorkerService.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: hard-exclude observer-sessions from hooks; bundle migration 29 (#2124)
* fix: hard-exclude observer-sessions from hooks; backfill bundle migrations
Stop hook + SessionEnd hook were storing the SDK observer's own
init/continuation/summary prompts in user_prompts, leaking into the
viewer (meta-observation regression). 25 such rows accumulated.
- shouldTrackProject: hard-reject OBSERVER_SESSIONS_DIR (and its subtree)
before consulting user-configured exclusion globs.
- summarize.ts (Stop) and session-complete.ts (SessionEnd): early-return
when shouldTrackProject(cwd) is false, so the observer's own hooks
cannot bootstrap the worker or queue a summary against the meta-session.
- SessionRoutes: cap user-prompt body at 256 KiB at the session-init
boundary so a runaway observer prompt cannot blow up storage.
- SessionStore: add migration 29 (UNIQUE(memory_session_id, content_hash)
on observations) inline so bundled artifacts (worker-service.cjs,
context-generator.cjs) stay schema-consistent — without it, the
ON CONFLICT clause in observation inserts throws.
- spawnSdkProcess: stdio[stdin] from 'ignore' to 'pipe' so the
supervisor can actually feed the observer's stdin.
Also rebuilds plugin/scripts/{worker-service,context-generator}.cjs.
* fix: walk back to UTF-8 boundary on prompt truncation (Greptile P2)
Plain Buffer.subarray at MAX_USER_PROMPT_BYTES can land mid-codepoint,
which the utf8 decoder silently rewrites to U+FFFD. Walk back over any
continuation bytes (0b10xxxxxx) before decoding so the truncated prompt
ends on a valid sequence boundary instead of a replacement character.
* fix: cross-platform observer-dir containment; clarify SDK stdin pipe
claude-review feedback on PR #2124.
- shouldTrackProject: literal `cwd.startsWith(OBSERVER_SESSIONS_DIR + '/')`
hard-coded a POSIX separator and missed Windows backslash paths plus any
trailing-slash variance. Switched to a path.relative-based isWithin()
helper so Windows hook input under observer-sessions\\... is also excluded.
- spawnSdkProcess: added a comment explaining why stdin must be 'pipe' —
SpawnedSdkProcess.stdin is typed NonNullable and the Claude Agent SDK
consumes that pipe; 'ignore' would null it and the null-check below
would tear the child down on every spawn.
* fix: make Stop hook fire-and-forget; remove dead /api/session/end
The Stop hook was awaiting a 35-second long-poll on /api/session/end,
which the worker held open until the summary-stored event fired (or its
30s server-side timeout elapsed). Followed by another await on
/api/sessions/complete. Three sequential awaits, the middle one a 30s
hold — not fire-and-forget despite repeated requests.
The Stop hook now does ONE thing: POST /api/sessions/summarize to
queue the summary work and return. The worker drives the rest async.
Session-map cleanup is performed by the SessionEnd handler
(session-complete.ts), not duplicated here.
- summarize.ts: drop the /api/session/end long-poll and the trailing
/api/sessions/complete await; ~40 lines removed; unused
SessionEndResponse interface gone; header comment rewritten.
- SessionRoutes: delete handleSessionEnd, sessionEndSchema, the
SERVER_SIDE_SUMMARY_TIMEOUT_MS constant, and the /api/session/end
route registration. Drop the now-unused ingestEventBus and
SummaryStoredEvent imports.
- ResponseProcessor + shared.ts + worker-utils.ts: update stale
comments that referenced the dead endpoint. The IngestEventBus is
left in place dormant (no listeners) for follow-up cleanup so this
PR stays focused on the blocker.
Bundle artifact (worker-service.cjs) rebuilt via build-and-sync.
Verification:
- grep '/api/session/end' plugin/scripts/worker-service.cjs → 0
- grep 'timeoutMs:35' plugin/scripts/worker-service.cjs → 0
- Worker restarted clean, /api/health ok at pid 92368
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* deps: bump all dependencies to latest including majors
Upgrades: React 18→19, Express 4→5, Zod 3→4, TypeScript 5→6,
@types/node 20→25, @anthropic-ai/claude-agent-sdk 0.1→0.2,
@clack/prompts 0.9→1.2, plus minors. Adds Daily Maintenance section
to CLAUDE.md mandating latest-version policy across manifests.
Express 5 surfaced a race in Server.listen() where the 'error' handler
was attached after listen() was invoked; refactored to use
http.createServer with both 'error' and 'listening' handlers attached
before listen(), restoring port-conflict rejection semantics.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: surface real chroma errors and add deep status probe
Replace the misleading "Vector search failed - semantic search unavailable.
Install uv... restart the worker." string in SearchManager with the actual
exception text from chroma_query_documents. The lying message blamed `uv`
for any failure — even when the real cause was a chroma-mcp transport
timeout, an empty collection, or a dead subprocess.
Also add /api/chroma/status?deep=1 backed by a new
ChromaMcpManager.probeSemanticSearch() that round-trips a real query
(chroma_list_collections + chroma_query_documents) instead of just
checking the stdio handshake. The cheap default path is unchanged.
Includes the diagnostic plan (PLAN-fix-mcp-search.md) and updated test
fixtures for the new structured failure message.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: rebuild worker-service bundle to match merged src
Bundle was stale after the squash merge of #2124 — it still contained
the old "Install uv... semantic search unavailable" string and lacked
probeSemanticSearch. Rebuilt via bun run build-and-sync.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: address coderabbit feedback on PLAN-fix-mcp-search.md
- replace machine-specific /Users/alexnewman absolute paths with portable
<repo-root> placeholder (MD-style portability)
- add blank lines around the TypeScript fenced block (MD031)
- tag the bare fenced block with `text` (MD040)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
29 KiB
Plan 08 — transcript-watcher-integration (clean)
Feature scope: src/services/transcripts/* + src/cli/handlers/observation.ts HTTP loopback.
Source of truth (design): PATHFINDER-2026-04-21/05-clean-flowcharts.md § 3.12; Part 1 items #17, #18, #19.
Phase-7 counterpart in 06: PATHFINDER-2026-04-21/06-implementation-plan.md Phase 7 (Transcript watcher cleanup).
Before-state: PATHFINDER-2026-04-21/01-flowcharts/transcript-watcher-integration.md.
Dependencies (must land first)
| Plan | Dependency | What this plan consumes |
|---|---|---|
07-plans/01-privacy-tag-filtering.md |
stripMemoryTags(text) (06 Phase 1) |
Single call used inside ingestObservation. We never strip in the watcher. |
07-plans/07-session-lifecycle-management.md |
ingestObservation(payload) helper (06 Phase 2) + SessionManager.initializeSession / endSession direct API (06 § 3.8) |
Watcher calls the helper directly (no workerHttpRequest, no observationHandler.execute). Session lifecycle routes session_init / session_end to SessionManager without HTTP. |
Downstream dependents: none.
Dependency-verified facts (live-code citations)
- V18 confirmed (
06-implementation-plan.md:45). All three artifacts still present:- 5-s rescan timer —
src/services/transcripts/watcher.ts:124(rescanIntervalMs ?? 5000) +setInterval(...)at:125. pendingToolsmap —src/services/transcripts/processor.ts:23(inSessionStateinterface) +.setat:202,.get/.deleteat:232-236,.clearat:317.- HTTP loopback —
src/cli/handlers/observation.ts:17loops throughworkerHttpRequest('/api/sessions/observations', ...). Chain: watcher.ts:221 → processor.ts:252observationHandler.execute→ observation.ts:17workerHttpRequestback to the same worker. This is the "call the CLI handler from inside the worker, which HTTP-loops back to the worker" anti-pattern.
- 5-s rescan timer —
- Schema list (exhaustive): only one JSONL transcript schema ships today: Codex, defined in
src/services/transcripts/config.ts:9asCODEX_SAMPLE_SCHEMA(confirming63472 — CODEX_SAMPLE_SCHEMA in config.ts is the source of truth). The live config file istranscript-watch.example.json(line 1-95) which registers onlycodexunderschemas.codex. TheCodexCliInstaller.tsis the only installer that merges JSONL schemas into~/.claude-mem/transcript-watch.json(src/services/integrations/CodexCliInstaller.ts:97-99).CursorHooksInstaller.ts,OpenCodeInstaller.ts,GeminiCliHooksInstaller.tsdo not register JSONL transcript schemas — they install PostToolUse hooks that feed the CLI observation handler directly (same path as Claude Code's own hooks). They do not touch the transcript watcher.- The audit's "Cursor, OpenCode, Gemini-CLI" for transcript ingestion is accurate only at the user-facing-feature level (these agents' activity is captured), but the capture path for those three is the hook handler chain, not the JSONL watcher. The watcher's only current JSONL client is Codex.
- tool_use_id availability in Codex schema (
src/services/transcripts/config.ts:47-77):tool-useevent:toolId: 'payload.call_id'— present onfunction_call,custom_tool_call,web_search_call,exec_command.tool-resultevent:toolId: 'payload.call_id'— present onfunction_call_output,custom_tool_call_output,exec_command_output.- Both sides always carry
call_idin the Codex schema. No fallback needed for Codex. - Schema-driven, not hard-coded: the
toolIdfield is part of theSchemaEvent.fieldscontract (src/services/transcripts/types.ts:34). Any future schema that wants to use the transcript watcher must setfields.toolIdon both its tool_use and tool_result events, or pair them some other way. Phase 2 below documents this contract explicitly.
- Watched parent dir per schema:
~/.codex/sessions/**/*.jsonl(config.ts:95,transcript-watch.example.json:83). The glob matches files recursively under~/.codex/sessions/. The parent dir to pass tofs.watch(..., { recursive: true })is the glob-root:expandHomePath('~/.codex/sessions')(everything before the first glob metachar).resolveWatchFiles()atwatcher.ts:143-163already understands glob vs plain-dir vs plain-file — the new watch code will derive the root the same way. - fs.watch recursive support: supported on macOS, Linux (kernel >= 2.6.36 via
inotify, but Node's recursive option landed with macOS + Windows in 0.x and Linux in Node 20 via libuv). CI target:package.json:58declares"node": ">=18.0.0". Recursive fs.watch on Linux requires Node 20+; we must bump the engines floor (see Gaps). Bun supportsfs.watchrecursive on all three platforms. - FileTailer location:
src/services/transcripts/watcher.ts:15-81(unchanged by this plan — lines already do the byte-offset-tail correctly; only the file-discovery layer changes).
Phase contract (applies to every phase below)
- (a) Copy from
05-clean-flowcharts.md§ 3.12 (canonical flowchart). - (b) Docs at the top of each phase: 05 section ref + 06 verified finding (V-number) + live file:line.
- (c) Verification is mechanical: a
grepcount, a runtime test, or a file existence check. - (d) Anti-pattern guards — every phase cites (from
06:59-66):- A — no invented APIs. Grep for the method before using it.
- B — no polling;
fs.watchevents only (no rescansetInterval). - E — one code path for observation ingest; watcher + CLI hook both call
ingestObservation, never a second path.
Phase 1 — Parent-directory recursive watch replaces per-file fs.watch + 5 s rescan
Goal: fs.watch(parentDir, { recursive: true }, onFileEvent) supplants both the per-file fsWatch(filePath, ...) in FileTailer and the setInterval(..., rescanIntervalMs) rescan in TranscriptWatcher.
(a) What to implement — Copy from § 3.12
From the clean flowchart (05-clean-flowcharts.md:484-500):
Boot["Worker startup"] --> LoadCfg["loadTranscriptWatchConfig"]
LoadCfg --> ParentWatch["fs.watch(parent_dir, {recursive})
watches existing files AND new files"]
ParentWatch --> OnChange([File event])
OnChange --> ReadDelta["FileTailer.readNewBytes"]
Code change (watcher.ts):
-
Delete the per-file watcher inside
FileTailer(src/services/transcripts/watcher.ts:16,:28-33,:35-38).FileTailerbecomes a pure byte-offset reader — no internalfs.watchsubscription. Rename itsstart()toreadAvailable()(one-shot tail) and drop theclose()method (nothing to close now). -
In
TranscriptWatcher.setupWatch(:110), deriveglob-rootfromwatch.path:- If
watch.pathhas no glob metachars and is a file: watchdirname(resolved)non-recursively. - Otherwise: walk the path tokens, stop at the first token containing a glob metachar, join the prefix — that's the root dir (e.g.
~/.codex/sessions/**/*.jsonl→~/.codex/sessions). Use the new helpergetGlobRoot(inputPath): string.
- If
-
Replace
setInterval(async () => { ... }, rescanIntervalMs)(:124-132) with:fs.watch(globRoot, { recursive: true, persistent: true }, (eventType, filename) => { if (!filename) return; const absPath = path.resolve(globRoot, filename); if (!globMatches(absPath, resolvedPath)) return; // rename event fires when a new file is created (or renamed/deleted) if (!this.tailers.has(absPath) && existsSync(absPath)) { this.addTailer(absPath, watch, schema, false).catch(err => logger.warn('TRANSCRIPT', 'addTailer failed on fs.watch event', { file: absPath, error: err instanceof Error ? err.message : String(err) })); } const tailer = this.tailers.get(absPath); tailer?.readAvailable().catch(() => undefined); }); -
Update
TranscriptWatcher.stop()(:99-108) to close the single parent watcher per target instead of iterating per-tailer.close()+clearIntervalon the timer array. Delete therescanTimers: NodeJS.Timeout[]field (:87). -
Delete the
rescanIntervalMs?: numberfield fromWatchTarget(src/services/transcripts/types.ts:61). UpdateCodexCliInstaller.tsandtranscript-watch.example.jsonif either still sets it (grep).
(b) Docs cited
- 05 § 3.12 lines 482-500 (clean flowchart).
- Part 1 item #19 (
05-clean-flowcharts.md:37) — "5-s rescan timer for new transcript files". - V18 (
06-implementation-plan.md:45) —rescanIntervalMs ?? 5000atwatcher.ts:124. - Live:
src/services/transcripts/watcher.ts:28(per-filefsWatch),:124-133(rescan interval +setInterval).
(c) Verification
grep -n "setInterval" src/services/transcripts/→ zero matches.grep -n "rescanIntervalMs" src/ transcript-watch.example.json→ zero matches.- Runtime test: start worker against an empty temp dir
T; wait 1 s;touch T/new-session.jsonlthenecho '{"type":"session_meta","payload":{"id":"test","cwd":"/tmp"}}' >> T/new-session.jsonl; assert aTRANSCRIPT Watching transcript filelog line appears within 100 ms of the write (not within the old 5 s window). Follow up with a tool_use line and assertpending_messagesrow appears within another 100 ms. grep -n "new FileTailer.*filePath.*offset.*onLine" src/services/transcripts/→ still exactly one call site inaddTailer(signature preserved for byte-offset state).
(d) Anti-pattern guards
- A: do not invent a "glob walker" class. A single
getGlobRoot(path: string): stringtop-level function is enough. - B: no fallback
setInterval"in case fs.watch misses events". The parent-recursive watch is the contract; missed-event scenarios fall under the Gaps section (Node-version requirement).
Blast radius
Single file rewrite: src/services/transcripts/watcher.ts. Small touch: types.ts (drop rescanIntervalMs). One touch to CodexCliInstaller.ts or transcript-watch.example.json only if they reference that deleted option.
Phase 2 — Delete pendingTools map; match tool_use + tool_result by tool_use_id at parse time
Goal: SessionState.pendingTools: Map<string, …> is gone. Tool pairing happens locally inside each log file's tail buffer keyed by tool_use_id; the per-session map disappears.
(a) What to implement — Copy from § 3.12
Route -->|tool_use + tool_result paired by tool_use_id| Ingest["ingestObservation({sessionDbId, tool_use_id, name, input, output})"]
Code change (processor.ts):
- Remove
pendingTools: Map<string, {name?, input?}>fromSessionState(src/services/transcripts/processor.ts:23). - Remove
pendingTools: new Map()fromgetOrCreateSession(:59). - Rewrite
handleToolUse(:193-222):- Move the per-file pairing buffer out of the session and into
TranscriptWatcheras a per-file map:private pendingToolUses = new Map<string /* filePath */, Map<string /* tool_use_id */, { name: string; input: unknown; ts: number }>>(). Inject it as a callback arg, or move the pairing into the processor keyed by file. - Simpler option (preferred): keep the short-lived pairing in the processor keyed by
${watch.name}:${sessionId}:${tool_use_id}— it still clears ontool_result, but it's keyed by ID, not by session-state entry. Upper bound size with an LRU (max=10_000, drop-oldest) to avoid unbounded growth if a tool_use has no matching tool_result.
- Move the per-file pairing buffer out of the session and into
- Rewrite
handleToolResult(:224-246) to read from that keyed map; on hit, emit oneingestObservation({sessionDbId, tool_use_id, name, input, output})call (Phase 3 wires the helper). On miss, log debug + drop (don't synthesize). - Drop the
apply_patchauto-file-edit branch at:205-213only if Codex stops sendingtool_usewithtoolResponseinline — inspectinghandleToolUsetoday, there's a legacy branch at:215-221that firessendObservationfrom insidehandleToolUsewhentoolResponse !== undefined. That branch is the first half of the duplicated ingest and must be deleted in Phase 3. Keep theapply_patchfile-edit branch (:205-213); file edits are a separate path not in scope here. - Session state retains
lastUserMessage,lastAssistantMessage,cwd,project— untouched.
(b) Docs cited
- 05 § 3.12 line 494 ("paired by tool_use_id").
- Part 1 item #17 (
05-clean-flowcharts.md:35) — "pendingTools map in TranscriptEventProcessor ... match by ID, no state map." - V18 — pendingTools presence confirmed.
- Live:
src/services/transcripts/processor.ts:23(interface field),:59(init),:202(.set),:232-236(lookup/delete),:317(clear on session_end). - Contract source: Codex schema in
src/services/transcripts/config.ts:47-77—toolId: 'payload.call_id'on both tool_use and tool_result.
(c) Verification
grep -rn "pendingTools" src/→ zero matches (interface field, initializer, and three call sites all gone).grep -n "SessionState" src/services/transcripts/processor.ts— interface still exists, but withpendingToolsfield removed (assert via a small diff check in a test).- Runtime: replay a recorded Codex JSONL (fixture). Assert the stream of
pending_messagesrows matches byte-for-byte with the pre-refactor run for the same fixture (the pairing semantics are unchanged; we only moved where the map lives). - Memory test: feed 50 sessions with 1000 tool_use each but no tool_result. The LRU bounds at 10k — not unbounded.
(d) Anti-pattern guards
- A: the pairing map is a private field of
TranscriptEventProcessor, not a newToolPairingServiceclass. - E: only one observation ingest call per paired event — delete the
handleToolUse-inlinesendObservationbranch at:215-221in Phase 3.
Blast radius
src/services/transcripts/processor.ts only. No schema contract change (Codex already populates call_id on both sides).
Phase 3 — Replace observationHandler.execute() HTTP loopback with direct ingestObservation(payload)
Goal: sendObservation no longer calls the CLI handler, which no longer does workerHttpRequest. The worker process calls its own helper in-memory.
(a) What to implement — Copy from § 3.12 + D1
From 05 Part 2 Decision D1 (:69-70):
D1. One observation ingest path. Hook, transcript-watcher, and manual-save all call
ingestObservation(payload). That function does: strip tags → validate privacy → INSERTpending_messages. No HTTP loopback inside the worker process.
From § 3.12 line 494 — ingestObservation({sessionDbId, tool_use_id, name, input, output}).
Code change:
- In
src/services/transcripts/processor.ts:- Replace
sendObservationbody (:248-260) so it builds theIngestObservationPayload(matching the shape owned by07-plans/07-session-lifecycle-management.md) and callsawait ingestObservation(payload)directly. NoobservationHandlerimport. - Remove the import of
observationHandler(:3). - Remove the import of
workerHttpRequestandensureWorkerRunningfrom../../shared/worker-utils.js(:6) from the observation path only —queueSummarystill hits/api/sessions/summarizetoday andupdateContextstill hits/api/context/inject; those two are untouched by Phase 3. Phase 4 deletes both.
- Replace
- In
src/services/transcripts/watcher.ts: no change — the watcher already delegates toprocessor.processEntry; the processor is what imports the helper. IngestObservationPayloadshape reused from Plan 07 (definition lives insrc/services/worker/ingest/index.ts):Plan 07 additionally adds{ contentSessionId, platformSource, cwd, tool_name, tool_use_id, tool_input, tool_response, agentId?, agentType? }tool_use_idas a required field when the caller is the transcript watcher (already present in hook-path flows via the UNIQUE constraint added in Phase 9 of06-implementation-plan.md). Synthesizetool_use_id = payload.call_idfrom the schema'stoolIdfield.
(b) Docs cited
- 05 § 3.12 line 494, Part 2 D1 lines 69-70.
- Part 1 item #18 (
05-clean-flowcharts.md:36) — "observationHandler.execute() HTTP loopback from transcript-watcher ... Extract ingestObservation helper; both call it directly." - V18 —
observation.ts:17HTTP loopback confirmed. - Live:
src/cli/handlers/observation.ts:17(workerHttpRequest('/api/sessions/observations', …)),src/services/transcripts/processor.ts:252(observationHandler.executecall site). - Dependency contract:
07-plans/07-session-lifecycle-management.mdexportsingestObservationatsrc/services/worker/ingest/index.tsper06-implementation-plan.md:126-132.
(c) Verification
grep -n "observationHandler" src/services/transcripts/→ zero matches.grep -n "workerHttpRequest.*observations" src/services/transcripts/→ zero matches.grep -n "workerHttpRequest" src/services/transcripts/→ count ≤ 2 (temporarily:queueSummary+updateContext, deleted in Phase 4).grep -n "workerHttpRequest" src/cli/handlers/observation.ts→ still exactly one (CLI hook path still uses HTTP when the CLI is a separate process from the worker; that's not a loopback, it's the hook-to-worker boundary).- Unit test: seed a single Codex JSONL line with a tool_use + tool_result pair; assert (1) exactly one
pending_messagesINSERT, (2) zero outbound HTTP requests recorded against the worker's own/api/sessions/observationsendpoint (use an HTTP spy).
(d) Anti-pattern guards
- B: no polling — direct function call, not an event bus, not a retry loop.
- E: the hook path and the transcript path both call
ingestObservation(payload). Only ingress shape conversion differs; the helper is the single code path (matches06-implementation-plan.md:146— "One helper, both handlers call it.").
Blast radius
src/services/transcripts/processor.ts only. The watcher chain inside the worker process no longer crosses the HTTP boundary. The CLI hook (observation.ts) remains unchanged for this phase — it runs in the hook subprocess and must HTTP the worker.
Phase 4 — Route session_init / session_end directly to SessionManager (drop /api/sessions/summarize + /api/context/inject loopbacks)
Goal: handleSessionInit calls SessionManager.initializeSession directly. handleSessionEnd calls SessionManager.endSession (which internally queues the summary the same way the hook-side does). The last two in-process HTTP loopbacks disappear from the transcript path.
(a) What to implement — Copy from § 3.12
Route -->|session_init| Init["sessionManager.initializeSession(sessionDbId)
(direct, no HTTP loopback)"]
Route -->|session_end| EndFlow["sessionManager.endSession(sessionDbId)
→ queueSummarize (same as hook path)"]
EndFlow --> WriteCtx["Optional: writeAgentsMd (Cursor flag)"]
Code change (processor.ts):
- Replace
handleSessionInit(:178-191) with a direct call toSessionManager.initializeSession(sessionDbId, userPrompt=fields.prompt, promptNumber). The worker-processSessionManagerinstance is injected via constructor (plan 07 already plumbs this; the watcher receives it inTranscriptWatcherconstructor). - Replace
queueSummary(:322-344): call the same helper that07-plans/07-session-lifecycle-management.mdexposes asendSession({contentSessionId, platformSource, last_assistant_message})→ internally it callsingestSummary(payload)(from06-implementation-plan.md:130). NoworkerHttpRequest('/api/sessions/summarize', …). - Replace
updateContext(:346-392): keep the path-traversal guard (:363-373— real security check, not patch cruft), but replace the HTTP call at:377with a directgenerateContext(allProjects)call fromContextBuilder(the same function/api/context/injecthandler wraps).writeAgentsMdunchanged. - Remove import of
ensureWorkerRunningandworkerHttpRequest(both already freed by this point). sessionCompleteHandler.executeatprocessor.ts:311-315— delete;endSessionsubsumes it.
(b) Docs cited
- 05 § 3.12 lines 493, 495, 497 — direct
initializeSession/endSession,writeAgentsMdkept. - 05 Part 2 D1 line 70 — "no HTTP loopback inside the worker process."
- Dependency: plan 07
06-implementation-plan.md:114-152(Phase 2 helpers:ingestObservation,ingestPrompt,ingestSummary) and:321-326(§ 3.8endSessionblocks until summary). - Live:
src/services/transcripts/processor.ts:185(sessionInitHandler.execute),:334(workerHttpRequest('/api/sessions/summarize', …)),:377(workerHttpRequest(contextUrl)),:363-373(security guard — preserve).
(c) Verification
grep -n "workerHttpRequest\|ensureWorkerRunning" src/services/transcripts/→ zero matches.grep -n "sessionInitHandler\|sessionCompleteHandler\|observationHandler" src/services/transcripts/→ zero matches.grep -n "writeAgentsMd\|isPathSafe" src/services/transcripts/processor.ts→ still present (security guard kept).- Integration: drive a full Codex JSONL run through the watcher; assert the AGENTS.md file is written with the same content as the pre-refactor path.
(d) Anti-pattern guards
- D: no facade — the processor talks to
SessionManagerdirectly, not via aTranscriptSessionBridge. - E:
ingestSummaryis the one code path — transcriptsession_endand hookStopboth call it.
Blast radius
src/services/transcripts/processor.ts — large internal rewrite. No external shape changes: the eventual pending_messages rows are byte-identical to today's hook-path output.
Phase 5 — Remove isProjectExcluded re-check in the processor (moved into ingestObservation)
Goal: The transcript processor does not re-run project-exclusion. ingestObservation (and its siblings) run the check once, centrally (per Plan 07).
(a) What to implement — Copy from § 3.12
From 05 § 3.12 Deleted list (:502-506):
isProjectExcludedre-check inside transcript processor (done once iningestObservation)
Code change:
grep -n "isProjectExcluded" src/services/transcripts/— if any call site exists (it is currently checked insideobservationHandler.execute,src/cli/handlers/observation.ts:59, which the watcher path no longer uses after Phase 3), delete it.- Assert
ingestObservationperforms the exclusion check (Plan 07 requirement, per06-implementation-plan.md:132— "(b) runs privacy / project-exclusion validation").
(b) Docs cited
- 05 § 3.12 deleted-list (
:506). - Dependency:
06-implementation-plan.md:132. - Live:
src/cli/handlers/observation.ts:57-62— current exclusion check (removed from the transcript path by Phase 3's loopback kill; this phase confirms no second copy exists in the watcher).
(c) Verification
grep -rn "isProjectExcluded" src/services/transcripts/→ zero matches.grep -n "isProjectExcluded" src/services/worker/ingest/→ exactly one call (insideingestObservation/ shared privacy-validate path).
(d) Anti-pattern guards
- E: one exclusion check, one code path —
ingestObservationis authoritative.
Blast radius
Essentially a grep-and-delete pass; most likely zero lines to change (the check never lived in the processor, only in the CLI handler we've already unlinked).
Phase 6 — Verification gate
Goal: Prove the four deletions and the single new mechanism by mechanical checks.
Checks
- Parent-dir watch drop test (from Phase 1's ©): write a brand-new JSONL file into a mock watched dir; within 100 ms observe a
Watching transcript filelog line AND apending_messagesINSERT after the first tool_use+tool_result pair. Without the 5-s rescan, this must succeed on a sub-second timeline. pendingToolsgone:grep -rn "pendingTools" src/→0.- HTTP loopback gone:
grep -rn "workerHttpRequest\|ensureWorkerRunning" src/services/transcripts/→0.grep -rn "observationHandler\|sessionInitHandler\|sessionCompleteHandler" src/services/transcripts/→0. - Timer gone:
grep -rn "setInterval" src/services/transcripts/→0. - Single-path ingest:
grep -rn "ingestObservation(" src/— ≥ 2 call sites (transcript processor + hook-path route handler from Plan 07); zero in CLI handler (still uses HTTP to reach the worker). - Schema-contract fuzz: drop a crafted JSONL where
tool_useomitscall_id. Assert: debug log "tool_use without toolId", no crash, no paired observation emitted. Drop atool_resultwith acall_idwe never saw. Assert: debug log "orphan tool_result", no crash. - Cursor / OpenCode / Gemini-CLI unaffected: those paths go through
src/cli/handlers/observation.ts(hook PostToolUse). Run the standard hook-round-trip smoke test (npm run build-and-sync+ trigger a PostToolUse from each); assertpending_messagesrows still appear. This is the non-regression guard for the prompt's "preserve Cursor/OpenCode/Gemini-CLI" constraint — they never depended on the transcript JSONL watcher, so Phases 1-5 cannot break them; this check exists to prove it. - End-to-end: full Codex JSONL fixture → expected SQLite state identical to pre-refactor.
Anti-pattern guards (final sweep)
- A: every new identifier (
getGlobRoot,pendingToolUsesmap,readAvailable) traces to a concrete live function or the plan's invented, single-use helper. No new classes. - B: one
fs.watchsubscription per target, no timers, no polling, no "retry-rescan on SIGCHLD". - E: transcript processor and hook route both import
ingestObservationfrom the same module (src/services/worker/ingest/index.ts), with no privately duplicated strip / privacy / exclusion logic.
Summary of line deletions
Against current live code:
| File | Lines removed | Lines added | Net |
|---|---|---|---|
src/services/transcripts/watcher.ts |
~40 (per-file fsWatch + rescan interval + timer-cleanup scaffolding) | ~25 (parent-dir recursive watch + getGlobRoot) |
-15 |
src/services/transcripts/processor.ts |
~120 (pendingTools state, handleToolUse inline ingest, HTTP queueSummary, HTTP updateContext, handler imports) |
~50 (LRU tool-pairing map, direct ingestObservation/endSession calls, direct generateContext import) |
-70 |
src/services/transcripts/types.ts |
1 (rescanIntervalMs field) |
0 | -1 |
src/cli/handlers/observation.ts |
0 (preserved; hook path still HTTPs the worker) | 0 | 0 |
| Total | ~161 | ~75 | ~-86 |
Plan-level estimate aligns with 05-clean-flowcharts.md:554 row "Transcript 5-s rescan + pendingTools map + HTTP loopback: -150 / +40 / -110" — consistent with our per-file count.
Phase count
6 phases (5 implementation + 1 verification gate), matching the minimum set specified in the prompt.
Gaps and open questions
- Node-version floor must bump.
package.json:58currently pins"node": ">=18.0.0".fs.watch(dir, { recursive: true })on Linux became stable in Node 20 (earlier versions throwERR_FEATURE_UNAVAILABLE_ON_PLATFORM). macOS + Windows + Bun have supported it all along. Action before merging Phase 1: bumpengines.nodeto>=20.0.0(coordinate with infra/CI matrix) and verify the plugin's install path (Bun-managed) satisfies it. If bumping is blocked, a Linux-only fallback (chokidar or a polling Map of child dirs) is needed — but that re-introduces anti-pattern B, so the Node-20 bump is the right move. - Single schema in the live codebase, audit phrasing diverges from implementation. The audit text (and this prompt) references "Cursor, OpenCode, Gemini-CLI transcript ingestion" as preserved. In this codebase those three agents ingest through the PostToolUse hook chain (
CursorHooksInstaller.ts,OpenCodeInstaller.ts,GeminiCliHooksInstaller.ts— none of which register a JSONL schema). The only JSONL schema is Codex (src/services/transcripts/config.ts:9+transcript-watch.example.json). Phases 1-5 therefore only affect the Codex capture path. The preservation claim for Cursor/OpenCode/Gemini-CLI is satisfied trivially — their path doesn't touch this feature. This is worth calling out in the PR description to avoid reviewer confusion.
Sources consulted
PATHFINDER-2026-04-21/05-clean-flowcharts.md— full file, § 3.12 canonical, Part 1 #17/18/19, Part 2 D1, Part 4 timer census, Part 5 deletion row.PATHFINDER-2026-04-21/06-implementation-plan.md— full file, Phase 0 V18, Phase 7 scope, Phase 2 ingest-helper contract.PATHFINDER-2026-04-21/01-flowcharts/transcript-watcher-integration.md— full before-state.src/services/transcripts/watcher.ts(lines 1-242).src/services/transcripts/processor.ts(lines 1-393).src/services/transcripts/config.ts(lines 1-138).src/services/transcripts/types.ts(lines 1-70).src/services/transcripts/field-utils.ts(lines 1-153).src/cli/handlers/observation.ts(lines 1-86).src/services/worker/http/routes/SessionRoutes.ts(lines 560-659 forhandleObservationsByClaudeIdshape).src/services/worker-service.ts(watcher lifecycle at :90, :164, :466, :614-640, :1095-1097).src/services/integrations/{CursorHooksInstaller,OpenCodeInstaller,GeminiCliHooksInstaller,CodexCliInstaller}.ts— confirming only Codex registers a JSONL schema.transcript-watch.example.json— confirming onlycodexschema in the live config template.package.json:57-60— Node engine floor.