UX redesign: installer + provider rename + /learn-codebase + welcome card + SessionStart hint (#2255)
* feat(ux): claude-mem UX improvements with installer enhancements
Squashed PR #2156 commits for clean rebase onto main:
- feat(installer): add provider selection, model prompt, worker auto-start
- refactor: rename *Agent provider classes to *Provider
- feat: add /learn-codebase skill and viewer welcome card
- feat(worker): inject welcome hint when project has zero observations
- fix(pr-2156): address greptile review comments
- fix(pr-2156): address coderabbit review comments
- fix(pr-2156): persist CLAUDE_MEM_PROVIDER for non-claude in non-TTY mode
- fix(pr-2156): file-backed settings reads in installer + env-first SKILL doc
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* build: rebuild plugin artifacts after rebase onto v12.4.7
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(skills): strip claude-mem internals from learn-codebase
The learn-codebase skill, install next-step copy, WelcomeCard, and
welcome-hint previously walked the primary agent through worker endpoints
and synthetic observation payloads. The PostToolUse hook already captures
every Read/Edit the agent makes — the agent should have no awareness that
the memory layer exists. Collapse the skill to one instruction ("read every
source file in full") and rephrase touchpoints to describe only what the
user observes (Claude reading files), not what happens behind the scenes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sync): preflight version mismatch + settings-aware port resolution
Two related fixes for build-and-sync's worker restart step:
1. Read CLAUDE_MEM_WORKER_PORT from ~/.claude-mem/settings.json the same
way the worker does, instead of computing the default port from the
uid alone. Previously, users with a custom port saw a misleading
"Worker not running" message because the restart POST hit the wrong
port and got ECONNREFUSED.
2. Add a preflight check that aborts the sync when the running worker's
reported version does not match the version we are about to build.
Claude Code's plugin loader pins the worker to a specific cache
version per session, so syncing into a newer cache directory has no
effect until the user runs `claude plugin update thedotmack/claude-mem`
to bump the pin. The preflight surfaces this explicitly with the exact
command to run; --force bypasses it for intentional cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(learn-codebase): note sed for partial reads of large files
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: strip comments codebase-wide
Removed prose comments from all tracked source. Preserved directives
(@ts-ignore, eslint-disable, biome-ignore, prettier-ignore, triple-slash
references, webpack magic, shebangs). Deleted two tests that asserted
on comment text rather than runtime behavior.
Net: 401 files, -14,587 / +389 lines, -10.4% bytes.
Verified: typecheck passes, build passes, test count unchanged from
baseline (22 pre-existing fails, all unrelated).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(installer): move runtime setup into npx, eliminate hook dead air
Smart-install ran 3 times during a fresh install — the worst run was silent,
fired by Claude Code's Setup hook after `claude plugin install`, producing
~30s of dead air that looked like the plugin was hung.
This change makes `npx claude-mem install` the single place heavy work
happens, with a visible spinner. Hooks become runtime-only.
- New `src/npx-cli/install/setup-runtime.ts` module: ensureBun, ensureUv,
installPluginDependencies, read/writeInstallMarker, isInstallCurrent.
Marker schema preserved exactly ({version, bun, uv, installedAt}) so
ContextBuilder and BranchManager readers keep working.
- `npx claude-mem install`: ungated copy/register/enable for every IDE,
inserts a "Setting up runtime" task with honest "first install can take
~30s" spinner. The claude-code shell-out to `claude plugin install` is
removed — npx already populated everything Claude reads.
- New `npx claude-mem repair` command for post-`claude plugin update`
recovery, force-reinstalls runtime.
- Setup hook now runs `plugin/scripts/version-check.js` (29ms wall) instead
of smart-install. Mismatch prints "run: npx claude-mem repair" on stderr.
Always exits 0 (non-blocking, per CLAUDE.md exit-code strategy).
- SessionStart loses the smart-install entry; 2 hooks remain (worker start,
context fetch).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(installer): delete smart-install sources, retarget tests
- Delete scripts/smart-install.js + plugin/scripts/smart-install.js (both
are source files kept in sync manually; both must go).
- Delete tests/smart-install.test.ts (covered surface is gone).
- tests/plugin-scripts-line-endings: drop smart-install.js entry.
- tests/infrastructure/plugin-distribution: retarget two assertions at
version-check.js (the new Setup hook script).
- New tests/setup-runtime.test.ts: 9 tests covering marker read/write,
isInstallCurrent semantics. Marker schema invariant verified.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(installer): describe npx-driven setup + version-check Setup hook
Sweep public docs and architecture notes to reflect the new flow:
npx installer does Bun/uv setup with a visible spinner; Setup hook runs
sub-100ms version-check.js; users hit `npx claude-mem repair` after a
`claude plugin update`.
- docs/architecture-overview.md: hook lifecycle table + npx flow paragraph
- docs/public/configuration.mdx: tree + hook config example
- docs/public/development.mdx: build output line
- docs/public/hooks-architecture.mdx: full rewrite of pre-hook section,
timing table, performance table
- docs/public/architecture/{overview,hooks,worker-service}.mdx: tree
comments, JSON config example, Bun requirement section
docs/reports/* untouched (historical incident reports).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(install): mergeSettings writes via USER_SETTINGS_PATH
Greptile P1 (#2156): `settingsFilePath()` only resolved
`process.env.CLAUDE_MEM_DATA_DIR`, while `getSetting()` reads via
`USER_SETTINGS_PATH` which `resolveDataDir()` populates from BOTH the env
var AND a `CLAUDE_MEM_DATA_DIR` entry persisted in
`~/.claude-mem/settings.json`. Result: a user with the data dir saved in
settings.json but not exported in their shell would have provider/model
settings silently written to `~/.claude-mem/settings.json` while
`getSetting()` read from `/custom/path/settings.json` — read/write split.
Drop `settingsFilePath()` and the now-unused `homedir` import; reuse the
already-imported `USER_SETTINGS_PATH` constant.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cli): parse --provider, --model, --no-auto-start install flags
Greptile P1 (#2156): InstallOptions has fields `provider`, `model`,
`noAutoStart`, but the install case in the npx-cli switch only parsed
`--ide`. The other three flags were silently dropped — `npx claude-mem
install --provider gemini` was a no-op.
Extract a `parseInstallOptions(argv)` helper, share it between the bare
`npx claude-mem` and `npx claude-mem install` paths, and validate
`--provider` against the allowed set. Update help text accordingly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(install): pipe runtime-setup output, always show IDE multiselect
Two issues caught in a docker test of the installer:
1. The bun.sh installer, uv installer, and `bun install` were using
stdio: 'inherit', dumping their stdout/stderr through clack's spinner
region — visible as raw "downloading uv 0.11.8…" / "Checked 58
installs across 38 packages…" text streaming under the spinner. Switch
to stdio: 'pipe' and surface captured stderr only on failure (via a
shared describeExecError() helper that includes stdout when stderr is
empty). Spinner stays clean on the happy path.
2. promptForIDESelection() silently picked claude-code when no IDEs were
detected, never showing the user the multiselect. On a fresh machine
with no IDEs present yet (e.g. our docker test container), the user
never got to choose. Now: always show the full IDE list when
interactive; mark detected ones with [detected] hints and pre-select
them; show a warn line if zero are detected explaining they should pick
what they plan to use. Non-TTY callers still get the silent
claude-code default at the call site (unchanged).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(install): skip marketplace work for claude-code-only, offer to install Claude Code
Two related UX fixes from a docker test:
**Delay between "Saved Claude model=…" and "Plugin files copied OK"**
After dropping the needsManualInstall gate, every install was unconditionally
running `copyPluginToMarketplace` (which copied the entire root node_modules
tree — thousands of files, dozens of seconds) and `runNpmInstallInMarketplace`
(npm install --production) even when only claude-code was selected. Neither
is needed for claude-code: that path uses the plugin cache dir + the
installed_plugins.json + enabledPlugins flag, all of which we already write.
- Drop `node_modules` from `copyPluginToMarketplace`'s allowed-entries list;
the dependency-install task populates it on the destination side anyway.
- Re-introduce `needsMarketplace = selectedIDEs.some(id => id !== 'claude-code')`
scoped *only* to `copyPluginToMarketplace`, `runNpmInstallInMarketplace`,
and the pre-install `shutdownWorkerAndWait` (also pointless for claude-code-
only flows since we're not overwriting the worker's running cache dir
source). All other tasks (cache copy, register, enable, runtime setup) stay
unconditional.
**Claude Code missing → silent install of an IDE that isn't there**
When the user picked claude-code on a machine without it (e.g. a fresh
container), the install completed but `claude` was unavailable and the only
hint was a generic warn line. Replace with an explicit pre-flight prompt:
Claude Code is not installed. Claude-mem works best in Claude Code, but
also works with the IDEs below.
? Install Claude Code now?
◆ Yes — install Claude Code (recommended)
◯ No — pick another IDE below
◯ Cancel installation
If the user picks "Yes", run `curl -fsSL https://claude.ai/install.sh | bash`
(or the PowerShell equivalent on Windows), then re-detect IDEs and proceed
with claude-code pre-selected. If the install fails or the user picks "No",
the multiselect still appears with claude-code visible (just unmarked
[detected]), so they can opt in or pick another IDE.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(install): detect Claude Code via `claude` CLI, not ~/.claude dir
The directory `~/.claude` can exist (e.g. mounted in Docker, or created
by tooling) without Claude Code actually being installed. Detect the
`claude` command in PATH instead so the installer correctly offers to
install Claude Code when missing.
* docs(learn-codebase): add reviewer note explaining the cost tradeoff
The skill intentionally reads every file in full to build a cognitive
cache that pays off across the rest of the project. Add a brief note
so reviewers (human or bot) understand the tradeoff before flagging
the unbounded read as a cost issue.
* fix: address Greptile P1 feedback on welcome hint and learn-codebase
- SearchRoutes: skip welcome hint when caller passes ?full=true so
explicit full-context requests aren't intercepted by the hint.
- learn-codebase: replace `sed` instruction with the Read tool's
offset/limit parameters, since Bash is gated in Claude Code by
default.
* feat(install): ASCII-animated logo splash on interactive install
Plays a ~1s bloom animation of the claude-mem sunburst logomark when
the installer starts in an interactive terminal — geometrically rendered
via 12 ray curves around a center disc, in the brand orange. The
wordmark and tagline type on alongside the final frame.
Auto-skipped on non-TTY, in CI, when NO_COLOR or CLAUDE_MEM_NO_BANNER
is set, or when the terminal is too narrow.
Inspired by ghostty +boo.
* feat(banner): replace rotation frames with angular-sector bloom generator
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(banner): replace rotation frames with angular-sector bloom generator
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(banner): three-act choreography renderer with radial gradient and diff redraw
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(banner): update preview script to support small/medium/hero tier selection
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(docker): add COLORTERM=truecolor to test-installer sandbox
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(install): auto-apply PATH for Claude Code with spinner UX
The Claude Code install.sh prints a Setup notes block telling users to
manually edit "your shell config file" to add ~/.local/bin to PATH —
which left fresh installs unable to launch claude from the command line.
After a successful install, detect ~/.local/bin/claude on disk and, if
the dir is missing from PATH, append the right export line to .zshrc /
.bash_profile / .bashrc / fish config (idempotent, marked with a
comment). Also updates process.env.PATH for the current install run.
Wraps the curl|bash install in a clack spinner (interactive only) so the
~4 minute native-build download doesn't look frozen — output is captured
silently and dumped on failure for debuggability. Non-interactive mode
keeps inherited stdio for CI logs.
Verified end-to-end in the test-installer docker sandbox: spinner
animates, .bashrc gets the export, fresh login shell resolves claude.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(banner): video-frame ASCII renderer with three-act choreography
Generator switched from a single Jimp-rendered logo to pre-extracted
video frames concatenated with \x01 separators and gzip-deflated, ported
from ghostty's boo wire format. Renderer rewritten around three acts
(ignite → stagger bloom → text reveal + breathe) with adaptive sizing,
radial gradient, and diff-based redraw.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(onboarding): unify install / SessionStart / viewer around one first-success moment
Three surfaces now point at the same north-star moment — open the viewer, do
anything in Claude Code, watch an observation appear within seconds — with the
same verbatim timing and privacy lines, and a single canonical "how it works"
explainer instead of three diverging copies.
- Canonical explainer at src/services/worker/onboarding-explainer.md served via
GET /api/onboarding/explainer; mirrored into plugin/skills/how-it-works/SKILL.md
- SessionStart welcome hint rewritten as third-person status (no imperatives
Claude tries to execute), pinned with a default-value regression test
- Post-install Next Steps reframed as "two paths": passive default + optional
/learn-codebase front-load; drops /mem-search and /knowledge-agent from this
surface; adds verbatim timing + privacy lines and /how-it-works link
- /api/stats response gains firstObservationAt for the viewer stat row
- Viewer WelcomeCard branches on observationCount === 0: empty state shows live
worker-connection dot + "waiting for activity"; has-data state shows
observations · projects · since [date] and two example prompts. v2 dismiss key
- jimp added to package.json to fix pre-existing banner-frame build break
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(banner): play unconditionally; only honor CLAUDE_MEM_NO_BANNER
The 128-col / TTY / CI / NO_COLOR gates silently swallowed the banner in
narrower terminals, CI logs, and any non-TTY pipe — including Docker runs
where -it should preserve the experience but column width was the wrong
gate. Remove the implicit gates; keep the explicit opt-out only.
If a frame wraps in a narrow terminal, that's better than the banner
not playing at all.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* revert(banner): restore 15:33 gating logic per user request
Reverts eb6fc157. Restores isBannerEnabled to the state at commit
8e448015 (2026-04-30 15:33): TTY check, !CI, !NO_COLOR, !CLAUDE_MEM_NO_BANNER,
and cols >= BANNER.width.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(install): wrap remaining slow steps with spinners
Each IDE installer (Cursor, Gemini CLI, OpenCode, Windsurf, OpenClaw,
Codex CLI, MCP integrations) now runs inside a clack task spinner with
per-step progress messages instead of silent dynamic-import + cpSync.
Pre-overwrite worker shutdown (up to 10s) and the post-install health
probe (up to 3s) also get spinners.
Internal console.log/error/warn from each IDE installer is buffered
during the spinner; if the install fails, captured output is replayed
afterward via log.warn so users can see what broke.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(review): observation count + IDE pre-selection regressions
WelcomeCard's "no observations yet" empty state was triggered when a
project filter narrowed the feed to zero rows, even with thousands of
observations elsewhere. Source the count from global stats.database
to match firstObservationAt's scope.
Restore initialValues: [] in the IDE multiselect — pre-selecting every
detected IDE was the exact regression #2106 was filed for.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(install): trichotomy worker state + cache fallback for script path
ensureWorkerStarted now returns 'ready' | 'warming' | 'dead' instead of
boolean. The spawned-but-still-warming case (common in Docker cold
starts and slow first-time inits) was being misreported as 'did not
start', which contradicted the next-steps panel saying 'still starting
up'. Install task message and Next Steps headline now agree on the
actual state.
Also fixes the actual root cause of 'Worker did not start' on
claude-code-only installs: the worker script path was hardcoded to the
marketplace dir, which is left empty when no non-claude-code IDE is
selected. Now falls back to pluginCacheDirectory(version) when the
marketplace copy isn't present.
Verified end-to-end in docker/claude-mem with --ide claude-code,
--ide cursor, and a fresh container — install task and headline
agree on 'Worker ready at http://localhost:<port>' in all cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: align CLAUDE.md and public docs with current code
Sweep across CLAUDE.md and 10 high-traffic docs/public/ MDX files to
remove point-in-time references and align with the actual current
shape of the codebase. Highlights:
- Hardcoded port 37777 → per-user formula (37700 + uid % 100) on the
front-door pages (introduction, installation, configuration,
architecture/overview, architecture/worker-service, troubleshooting,
hooks-architecture, platform-integration).
- Default model 'sonnet' → 'claude-haiku-4-5-20251001' (matches
SettingsDefaultsManager).
- Node 18 → 20 (matches package.json engines).
- Lifecycle hook count corrected (5 events).
- Removed the nonexistent 'Smart Install' component and pre-built
directory tree referencing files that no longer exist
(context-hook.ts, save-hook.ts, cleanup-hook.ts, etc.); replaced
with the real worker dispatcher shape.
- Removed CLAUDE.md '#2101' issue tag (kept the design rationale).
- Replaced obsolete hooks.json example with a description of the real
bun-runner.js / worker-service.cjs hook event shape.
Lower-traffic doc pages still hardcode 37777 — left for a separate
global pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(scripts): land strip-comments around real parsers (postcss, remark, parse5)
Each language gets a real parser to locate comments, then we splice ranges
out of the original source. The library never serializes — that's how
remark-stringify produced 243 reformat-noise diffs in the first attempt
versus the 21 real strip targets here.
JS/TS/JSX -> ts.createSourceFile + getLeadingCommentRanges
CSS/SCSS -> postcss.parse + walkComments + node.source offsets
MD/MDX -> remark-parse (+ remark-mdx) + AST html / mdx-expression nodes
HTML -> parse5 with sourceCodeLocationInfo
shell/py -> kept hand-rolled hash stripper (no library worth the dep)
Preserves: shebangs, @ts-* directives, eslint-disable, biome-ignore,
prettier-ignore, triple-slash refs, webpack magic, /*! license keep,
@strip-comments-keep file marker. JS/TS handler runs a parse-roundtrip
check and refuses to write if syntax errors increased (catches the
worker-utils.ts breakage class from the 2026-04-29 attempt).
npm scripts:
strip-comments (apply)
strip-comments:check (CI-style, exits non-zero if changes needed)
strip-comments:dry-run (list, no writes)
Verified --check on this repo: 21 changes, -4.0% bytes, no parse-error
regressions, no reformat-suspect false positives.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: strip comments codebase-wide via parser-backed tool
21 files changed, -17,550 bytes (-4.0%) of narrative comments removed
across .ts / .tsx / .js / .mjs and the .gitignore. JS/TS comments stripped
via ts.createSourceFile + getLeadingCommentRanges — same canonical lexer,
same behavior as the 2026-04-29 strip, no reformat noise.
Preexisting baseline (unchanged):
typecheck: 16 errors at HEAD, 16 errors after strip (line numbers shift,
no new error classes — verified via diff of sorted error lists)
build: fails at HEAD with CrushHooksInstaller.js unresolved import
(preexisting, unrelated to this strip)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(install): drop Crush integration references after extract
The Crush integration was extracted to its own branch on May 1, but the
import at install.ts:280 (and the case block + ide-detection entry +
McpIntegrations config + npx-cli help text) still referenced the now-
removed CrushHooksInstaller.js, breaking the build.
Removes:
- case 'crush' block in install.ts
- crush entry in ide-detection.ts
- CRUSH_CONFIG and registration in McpIntegrations.ts
- 'crush' from the IDE Identifiers help line in index.ts
Rebuilds worker-service.cjs to match.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(banner): mark generated banner-frames.ts with @strip-comments-keep
Without this, every build/strip cycle ping-pongs five lines of doc
comments in and out of the auto-generated output. The keep-marker tells
strip-comments.ts to skip the file entirely.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(build): drop banner-frame regen from build script
generate-banner-frames.mjs requires PNG frames in /tmp/cmem-banner-frames
that only exist after the maintainer runs ffmpeg locally on the source
video. CI has neither the video nor the frames, so the build broke on
Windows. The output (src/npx-cli/banner-frames.ts) is committed, so the
regen is a one-shot dev step — not a build step. Run the script directly
when the video changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(worker): unstick the spinner — kill claim-self-lock, wake on fail, auto-broadcast
Three surgical changes that cure the stuck-spinner bug at the source.
Phase 1.1 (L9): claimNextMessage no longer self-excludes its own worker_pid.
A single UPDATE-RETURNING grabs the oldest pending row by id. Removes the
LiveWorkerPidsProvider plumbing that was never injected — Supervisor enforces
single-worker via PID file, so the multi-worker SQL was defending against a
configuration the project does not support.
Phase 1.2 (L19): SessionManager.markMessageFailed wraps PendingMessageStore.markFailed
and emits 'message' on the per-session EventEmitter. The iterator's waitForMessage
now wakes immediately on re-pend instead of parking for 3 minutes. ResponseProcessor
and SessionRoutes routed through the new wrapper.
Phase 1.3 (L24): PendingMessageStore takes an optional onMutate callback fired
from every mutator (enqueue, claimNextMessage, confirmProcessed, markFailed,
transitionMessagesTo, clearFailedOlderThan). SessionManager wires it; WorkerService
passes broadcastProcessingStatus. Ten manual broadcast calls deleted across
SessionCleanupHelper, SessionEventBroadcaster, SessionRoutes, DataRoutes, and
worker-service. Caller discipline becomes structurally impossible to forget.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(worker): delete dead code — legacy routes, processPendingQueues, decorative guards
Pure deletions. Phase 2 of kill-the-asshole-gates.
- Legacy /sessions/:sessionDbId/* routes (handleSessionInit, handleObservations,
handleSummarize, handleSessionStatus, handleSessionDelete, handleSessionComplete)
bypassed all five ingest gates and were a parallel write path. Folded the
initializeSession + broadcastNewPrompt + syncUserPrompt + ensureGeneratorRunning
+ broadcastSessionStarted work into the canonical /api/sessions/init handler so
the hook makes one round trip instead of two.
- processPendingQueues (~104 lines, zero callers) — replaced in Phase 6 by a
one-statement startup sweep.
- spawnInProgress Map and crashRecoveryScheduled Set — decorative dedupe over
generatorPromise and stillExists checks that already provide the real safety.
- STALE_GENERATOR_THRESHOLD_MS — pre-empted live generators and raced with the
finally block; the 3min idle timeout already kills zombies.
- MAX_SESSION_WALL_CLOCK_MS — ran a SELECT on every observation to enforce 24h.
Runaway-spend protection lives in the API key, not in claude-mem.
- Missing-id 400 in shared.ts ingestObservation — Zod already enforces min(1)
on contentSessionId and toolName at the route schema.
- SessionCompletionHandler import + completionHandler field on SessionRoutes
(orphaned after handler deletions).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(worker): SQL-backed getTotalQueueDepth — single source of truth
Was: iterate this.sessions.values() and sum getPendingCount per session.
Now: SELECT COUNT(*) FROM pending_messages WHERE status IN ('pending','processing').
The in-memory sessions Map drifted from the DB rows whenever a generator exited
without confirm/fail, leading to false-positive isProcessing in the UI. Phase 1.3's
auto-broadcast fires on every mutation, but it broadcast a stale Map count.
Reading from the DB makes the UI's spinner state match what the queue actually holds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(worker): typed abortReason replaces wasAborted boolean
Was: a boolean wasAborted that lumped every abort together. The finally block
branched on !wasAborted, so any abort skipped restart — including idle aborts
with pending work, which is exactly the case where we DO want to restart.
Now: ActiveSession.abortReason is a typed enum 'idle' | 'shutdown' | 'overflow'
| 'restart-guard'. The finally block consumes the reason and only skips restart
for 'shutdown' and 'restart-guard'. Idle and overflow aborts fall through, so
if pending work exists they trigger restart correctly.
Dropped 'stale' and 'wall-clock' from the union — Phase 2 deleted those paths.
Natural-completion abort (post-success) intentionally has no reason; it's not
gating restart logic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(worker): unify the two generator-exit finally blocks
Was: worker-service.ts:startSessionProcessor and SessionRoutes:ensureGeneratorRunning
each had their own ~70-line finally block with divergent restart-guard handling.
The worker-service path called terminateSession on RestartGuard trip and orphaned
pending rows (the L16 bug); the SessionRoutes path drained them. Two places to
update when rules changed.
Now: handleGeneratorExit in src/services/worker/session/GeneratorExitHandler.ts
owns the contract:
1. Always kill the SDK subprocess if alive.
2. Always drain processingMessageIds via sessionManager.markMessageFailed
(which wakes the iterator — Phase 1.2).
3. shutdown / restart-guard reasons: drain pending rows via
transitionMessagesTo('failed'), finalize, remove from Map. Fixes L16.
4. pendingCount=0: finalize normally and remove from Map.
5. pendingCount>0: backoff respawn via per-session respawnTimer (no global Set;
Phase 2.4 deleted that). RestartGuard trip drains to 'abandoned'.
Both finally blocks are now ~10-line wrappers that translate local state into the
canonical abortReason and delegate. Restored completionHandler injection into
SessionRoutes (was dropped in Phase 2 cleanup; needed by the unified helper for
finalizeSession).
Behavior change: SessionRoutes' previous "keep idle session in memory" was
deliberately replaced by the plan's "remove from Map on natural completion" —
next observation reinitializes via getMessageIterator → initializeSession.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(worker): startup orphan sweep — reset 'processing' rows at boot
When the worker dies (crash, kill, restart), any pending_messages rows it left
in 'processing' state are by definition orphans (the only worker is dead).
Single SQL UPDATE at boot resets them to 'pending' so the iterator can claim
them again. Replaces the deleted processPendingQueues function (Phase 2.2).
Runs in initializeBackground after dbManager.initialize() and before the
initializationComplete middleware releases blocked HTTP requests, so no
in-flight request can race the sweep. NOT on a periodic timer — after boot,
every 'processing' row has a live consumer and a periodic sweep would race.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(worker): simplify enqueue catch, replace memorySessionId throw with re-pend
7.1: queueObservation's catch was logging two ERROR-level messages and rethrowing.
The rethrow is correct (FK violations / disk full / schema drift should crash
loudly), but the verbose ERROR logging pretended the error was recoverable.
Reduced to one INFO line + rethrow.
7.2: ResponseProcessor's memorySessionId guard was throwing if the SDK hadn't
included session_id on the first user-yield, terminal-failing the entire batch.
Now warns and re-pends in-flight messages via sessionManager.markMessageFailed
(which wakes the iterator — Phase 1.2). The next iteration tries again with
memorySessionId hopefully captured.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(sync): mirror builds to installed-version cache for hot reload
When package.json bumps past Claude Code's installed pin, sync-marketplace
wrote new code to cache/<buildVersion>/ but the worker loaded from
cache/<installedVersion>/, so worker:restart reloaded the same old code.
Replace the exit-on-mismatch preflight with a mirror step: when versions
differ, also rsync plugin/ into cache/<installedVersion>/ so worker:restart
hot-reloads new code without a Claude Code session restart. The
build-version cache still gets written for the eventual
`claude plugin update`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: delete dead barrel files and orphan utilities
- src/sdk/index.ts (re-exports parser+prompts; nothing imported the barrel)
- src/services/Context.ts (re-exports ./context/index.js; no importers)
- src/services/integrations/index.ts (no importers)
- src/services/worker/Search.ts (3-line barrel of ./search/index.js)
- src/services/infrastructure/index.ts: drop CleanupV12_4_3 re-export
- src/utils/error-messages.ts (getWorkerRestartInstructions never imported)
- src/types/transcript.ts (170 LoC of types, zero importers)
- src/npx-cli/_preview.ts (banner dev preview, no script wires it)
Build + tests still pass; observations still flowing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(parser): drop unused detectLanguage
Only the user-grammar-aware variant detectLanguageWithUserGrammars()
is actually called.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(types): drop unused SdkSessionRecord + ObservationWithContext
Both interfaces in src/types/database.ts had zero importers anywhere
in src or tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(npx-cli): drop unused getDetectedIDEs + claudeMemDataDirectory
getDetectedIDEs has no callers — install.ts uses detectInstalledIDEs
directly. claudeMemDataDirectory has no callers either.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(ProcessManager): drop dead orphan-reaper + signal-handler helpers
Each had zero callers in src/ or tests/:
- cleanupOrphanedProcesses + enumerateOrphanedProcesses
- ORPHAN_PROCESS_PATTERNS + ORPHAN_MAX_AGE_MINUTES
- forceKillProcess
- waitForProcessesExit
- createSignalHandler
- resetWorkerRuntimePathCache
The orphan reaper was retired in PATHFINDER Plan 02 ("OS process groups
replace hand-rolled reapers", commit 94d592f2) — these were the leftover
pieces. shutdown.ts uses the supervisor's own kill-pgid path instead.
parseElapsedTime kept (covered by tests/infrastructure/process-manager.test.ts).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(scripts): delete 11 unreferenced DX/forensic scripts
None of these are referenced by package.json npm scripts or docs/.
All last touched on Apr 29 only as part of the comment-stripping
pass — the feature code itself is older and orphaned:
analyze-transformations-smart.js
debug-transcript-structure.ts
dump-transcript-readable.ts
endless-mode-token-calculator.js
extract-prompts-to-yaml.cjs
extract-rich-context-examples.ts
find-silent-failures.sh
fix-all-timestamps.ts
format-transcript-context.ts
test-transcript-parser.ts
transcript-to-markdown.ts
These are standalone tools — runtime behavior unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(scripts): delete unused extraction/ and types/ subdirs
- scripts/extraction/{extract-all-xml.py, filter-actual-xml.py, README.md}
point at ~/Scripts/claude-mem/ — the user's pre-relocation path that no
longer exists. Zero references in package.json, src/, or tests/.
- scripts/types/export.ts duplicates ObservationRecord etc. and has no
importers (CodexCliInstaller imports transcripts/types, not this).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(BranchManager): drop dead getInstalledPluginPath
OpenCodeInstaller has its own (used) getInstalledPluginPath; the
BranchManager copy never had any external callers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(ChromaSyncState): unexport DocKind (used internally only)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(gemini): drop stale earliestPendingTimestamp / processingMessageIds
Both fields were removed from ActiveSession in earlier queue-engine
cleanup. Tests had been silently keeping them because the mock sessions
use 'as any' to bypass strict typing, so the dead fields rode along
without complaint.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: drop 3 unused module-level constants
- src/npx-cli/banner.ts: CURSOR_HOME, CLEAR_DOWN (banner uses
CLEAR_SCREEN which combines clear-down + cursor-home into a single
CSI sequence; the standalone constants were leftovers).
- src/services/worker/BranchManager.ts: DEFAULT_SHELL_TIMEOUT_MS
(BranchManager only uses GIT_COMMAND_TIMEOUT_MS / NPM_INSTALL_TIMEOUT_MS).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(opencode-plugin): drop dead workerPost helper
Only the fire-and-forget variant (workerPostFireAndForget) is actually
called. workerPost was the await-result version with no remaining caller.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: drop 8 truly-unused interface fields
Verified each by grepping for `.field`, `"field"`, `'field'`, and
`field:` patterns across src/ + tests/ + plugin/scripts. Where the
only remaining usage was the assignment site, removed the assignments too.
- GitHubStarsData: watchers_count, forks_count (only stargazers_count read)
- TableColumnInfo: dflt_value (PRAGMA returns it but no caller reads it)
- IndexInfo: seq (PRAGMA returns it but no caller reads it)
- ObservationRecord: source_files (legacy field, no readers)
- HookResult.hookSpecificOutput: permissionDecisionReason
- WatchTarget: rescanIntervalMs (set in config, never read)
- ShutdownResult: confirmedStopped (write-only — assigned but no
reader; updated all 3 return sites to drop it)
- ModePrompts: language_instruction (multilingual support never wired)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(npx-cli): reuse InstallOptions type instead of inline duplicate
parseInstallOptions had its return type written out inline as an
anonymous duplicate of InstallOptions. Use the canonical type
(import type — zero bundle cost).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(integrations): drop unused Platform type alias
The detectPlatform() function that returned this type was deleted earlier
in the branch (along with getScriptExtension that consumed it). The type
itself outlived its consumer; only string literals "Platform:" survive in
console.log diagnostics, which don't reference the alias.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(worker): broadcast processing_status when summarize is queued
broadcastSummarizeQueued was an empty no-op even though
handleSummarizeByClaudeId calls it after enqueueing. The PendingMessageStore
onMutate callback already fires broadcastProcessingStatus on enqueue, but
calling it explicitly from broadcastSummarizeQueued ensures the spinner
ticks on the moment a summary is requested even if the onMutate chain has
any timing race.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(worker): keep spinner on while summary generates
ClaudeProvider's SDK can pull multiple synthetic prompts (e.g.
observation + summarize) before producing responses. Each pull pushed
an ID to session.processingMessageIds. When the SDK's first
observation response came back, ResponseProcessor.confirmProcessed
deleted ALL pending message rows — including the still-in-flight
summary — so getTotalQueueDepth dropped to 0 and the spinner turned
off, even though the summary took another ~22s to actually generate.
Tag each in-flight message with its type ({id, type}) so the response
processor can pop only the FIFO message of the matching type
(observation vs summarize). The summary row stays in 'processing'
until its own response arrives, keeping the spinner lit through the
entire summary window.
Also updates Gemini/OpenRouter providers and GeneratorExitHandler for
the new shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(worker): clear summary from queue on any SDK response
Switch ResponseProcessor from type-aware FIFO matching to strict FIFO
popping (each SDK response → 1 in-flight message consumed). This way
the summary always clears when the SDK responds, even when the
response is unparseable or the summary doesn't actually generate
content — preventing stuck spinner / queue-depth-stuck-at-1.
Spinner behavior is preserved: messages enqueued after the summary
keep the queue depth elevated, and only when the SDK has responded
to every prompt does the queue drain to zero.
Also: when the consumed message is a 'summarize' and parsing fails,
treat it as best-effort and confirmProcessed (no retry) — summaries
that can't be parsed shouldn't keep retrying.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(viewer): redesign welcome card and remove source filters
The first-start welcome card now explains the three feed card types
(observation/summary/prompt) with color-coded badges, points users at
the gear icon for settings and the project dropdown for filtering, and
plugs /mem-search for recall — replacing the old two-line "ask:" prompts.
Source filter tabs (Claude/Codex/etc.) are removed from the header.
Filtering by AI provider was nonsense from a user POV; the project
dropdown is the only header filter now. Source tracking is also
stripped from useSSE, usePagination, App state, and CSS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(viewer): keep welcome card in feed column, swap rows for 3 squares
Two visible problems in the previous design: the card stretched
edge-to-edge while feed cards sit in a centered 650px column, and
the body was a stack of long horizontal rows that scanned line-by-line.
Both fixed: Feed now accepts a pinnedTop slot so the welcome card
renders inside the same .feed-content column as observation cards.
Body is now a 3-column grid of square feature blocks — Live feed,
Tune it, Recall it — each with a custom inline SVG illustration
(stacked cards with color-coded stripes, gear+sliders, magnifier
over cards). Old text-row sections (welcome-card-types,
welcome-card-tips, welcome-card-section, welcome-card-tip-icon)
are removed. Squares stack to one column under 600px.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(viewer): convert welcome card to glassy modal with stylized logo
Card now opens as a centered modal with a frosted/glass backdrop
(blur + saturate) so it doubles as a proper help dialog when reopened
from the header's question-mark button. Removed the observation count,
project count, and "since" date — those don't make sense for a
first-launch surface and felt out of place in a help context.
Header art swapped from the small webp logomark to the new
high-resolution sun/sunburst PNG (claude-mem-logo-stylized.png),
shipped as a checked-in asset in src/ui and plugin/ui.
Bigger throughout: 28px h2, 16px tagline, 88px illustrations,
26px feature padding, 1:1 aspect-ratio squares. Backdrop click and
Esc both close. Mobile collapses the grid to one column and drops
the aspect-ratio constraint.
Reverted the unused pinnedTop slot on Feed.tsx since the welcome
card is now a true overlay rather than an in-feed pinned card.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(viewer): make welcome modal actually glassy
Previous version had a 55%-opacity black backdrop that almost fully
blocked the underlying UI — the "glass" was just a dark plate.
Now the backdrop is fully transparent (no darkening at all), the
panel itself drops to 55% bg-card opacity with its existing
backdrop-filter blur(28px) saturate(170%), and the feature squares
drop to 35% bg-tertiary so they layer as glass-on-glass over the
already-blurred panel. The header and feed below now read clearly
through the modal's frosted blur.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(viewer): bulletproof square features via padding-bottom + clamp() fluid type
Squares were rendering taller than wide because aspect-ratio is treated
as a minimum — content can push the box past 1:1. Switched to the
classic padding-bottom: 100% trick: percentage padding resolves against
the parent's width, so the box is ALWAYS W × W regardless of content.
Inner content sits in an absolutely-positioned flex column that can't
push the shell taller.
Whole modal is now desktop-first and fluid via clamp() — no media-query
stair-steps for type, padding, gaps, border-radius, illustration size,
or modal width. Single mobile breakpoint at <600px collapses the grid
to one column and reverts the padding-bottom trick so each feature can
grow to natural content height.
Tightened the three feature descriptions so they fit comfortably inside
the square at the desktop size.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* style(viewer): 15% black overlay + heavier modal shadow for elevation
Backdrop goes from transparent to rgba(0,0,0,0.15) — just enough
darkening to push the modal visually forward without burying the
underlying UI. Modal shadow stacked: 40px/120px ambient + 16px/48px
contact, both deeper, plus the existing inset 1px highlight.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(build): clear pending_messages queue on build-and-sync
Rewrites scripts/clear-failed-queue.ts to talk directly to SQLite via
bun:sqlite — the previous HTTP endpoints (/api/pending-queue/*) were
removed during the queue engine rewrite, so the script was orphaned.
Wires `npm run queue:clear` into `build-and-sync` so each rebuild
starts with a clean queue.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(worker): collapse parser to binary valid/invalid + clearPendingForSession model
- Parser: { valid: true, observations, summary } | { valid: false } — drops kind/skipped enum dispatch
- ResponseProcessor: two branches only (parseable → store + clearPendingForSession; else → no-op)
- Drop processingMessageIds + per-message claim/confirm/markFailed lifecycle across 3 providers
- PendingMessageStore: 226 → 140 lines; remove markFailed/transitionMessagesTo/confirmProcessed/clearFailedOlderThan/getAllPending/peekPendingTypes... wait keep peekPendingTypes
- Schema migration v31+v32: drop retry_count, failed_at_epoch, completed_at_epoch, worker_pid columns
- SessionQueueProcessor: delete two 1s recovery sleeps (let iterator end on error)
- Server.ts/SettingsRoutes.ts: replace four magic-number setTimeout exit-flush patterns with flushResponseThen helper
- GeneratorExitHandler: 183 → 117 lines (drain in-flight loop gone)
Net: -181 lines. No more silent data loss via maxRetries=3.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(pr-2255): address review comments batch 1
- install.ts: needsMarketplace true when claude-code selected (P1, was no-op)
- install.ts: throw on invalid --model so CLI exits non-zero
- install.ts: skip worker health checks + adapt next-step copy when --no-auto-start
- install.ts: repair regenerates plugin cache when missing
- index.ts: readFlag rejects missing/flag-shaped values
- index.ts: route flag-first invocations (e.g. `--provider claude`) to install
- banner.ts: fail-open if frame payload decode throws
- SearchRoutes.ts: 5s TTL cache for settings reads on hot hook path (P2)
- detect-error-handling-antipatterns.ts: trailing-brace strip whitespace-tolerant
- investigate-timestamps.ts: compute Dec 2025 epochs at runtime (was Dec 2024)
- regenerate-claude-md.ts: include workingDir in fallback walker so root is covered
- sync-marketplace.cjs: parseWorkerPort validates 1..65535 before http.request
- sync-to-marketplace.sh: resolve SOURCE_DIR from script location, not cwd
- Dockerfile.test-installer: bash --login sources .bashrc via .bash_profile
- docs/configuration.mdx: drop nonexistent .worker.port file refs, use settings.json
- docs/architecture-overview.md: dynamic port + queue model after parser collapse
- docs/architecture/worker-service.mdx: dynamic port example + drop port-file claim
- docs/platform-integration.mdx: WORKER_BASE_URL pattern, drop hardcoded 37777
- install/public/install.sh: Node 20 floor (was 18) to match docs
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(pr-2255): reset claimed messages to pending on early-return paths
ResponseProcessor returns early in two cases:
- parser invalid (unparseable response)
- memorySessionId not yet captured
Both paths previously left the just-claimed message in `status='processing'`,
which counts toward `getPendingCount`. The generator-exit handler then sees
`pendingCount > 0` and respawns the generator, looping until the restart
guard trips and `clearPendingForSession` deletes the message — silent data
loss.
Calling `resetProcessingToPending` on these paths lets the next generator
pass re-claim the message and try again, instead of burning the restart
budget on no-op respawns.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(pr-2255): swebench fallback row + troubleshooting port path
- evals/swebench/run-batch.py: append fallback prediction row when
orchestrator future raises, preserving "never drop an instance" guarantee
- docs/troubleshooting.mdx: drop nonexistent .worker.port / worker.port file
references; use settings.json + /api/health for port discovery
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(pr-2255): memoize per-project observation count for welcome-hint hot path
handleContextInject runs on every PostToolUse hook (after every Read/Edit).
The welcome-hint block ran a COUNT(*) on observations for every call once
CLAUDE_MEM_WELCOME_HINT_ENABLED was true. Observation counts are
monotonically increasing — once a project has any observations it always
will — so cache the positive result in a Set and skip the COUNT(*) on
subsequent requests.
Combined with the 5s settings TTL added earlier, the steady-state cost on
the hook hot path drops to a Set lookup.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(pr-2255): use clearProcessingForSession on AI-success path
clearPendingForSession deletes ALL rows for the session. On the success
path of processAgentResponse, that's wrong: messages that arrived as
'pending' during the (1-5s) AI response latency get deleted along with
the 'processing' row we just consumed. In a hook burst (three quick
PostToolUse hooks), B and C land while A is in flight; A's success then
nukes B and C — silent data loss.
Add a status-scoped clearProcessingForSession to PendingMessageStore +
SessionManager, and use it in ResponseProcessor's success path. The
unconditional clearPendingForSession remains correct in
GeneratorExitHandler for hard-stop / restart-guard-trip paths.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Revert "fix(pr-2255): use clearProcessingForSession on AI-success path"
This reverts commit a08995299c30cbad36bddc3e5bddda7af8604b35.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alex Newman
@@ -1,16 +1,3 @@
|
||||
/**
|
||||
* ChromaMcpManager - Singleton managing a persistent MCP connection to chroma-mcp via uvx
|
||||
*
|
||||
* Replaces ChromaServerManager (which spawned `npx chroma run`) with a stdio-based
|
||||
* MCP client that communicates with chroma-mcp as a subprocess. The chroma-mcp server
|
||||
* handles its own embedding and persistent storage, eliminating the need for a separate
|
||||
* HTTP server, chromadb npm package, and ONNX/WASM embedding dependencies.
|
||||
*
|
||||
* Lifecycle: lazy-connects on first callTool() use, maintains a single persistent
|
||||
* connection per worker lifetime, and auto-reconnects if the subprocess dies.
|
||||
*
|
||||
* Cross-platform: Linux, macOS, Windows
|
||||
*/
|
||||
|
||||
import { Client } from '@modelcontextprotocol/sdk/client/index.js';
|
||||
import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js';
|
||||
@@ -27,26 +14,10 @@ import { getSupervisor } from '../../supervisor/index.js';
|
||||
const CHROMA_MCP_CLIENT_NAME = 'claude-mem-chroma';
|
||||
const CHROMA_MCP_CLIENT_VERSION = '1.0.0';
|
||||
const MCP_CONNECTION_TIMEOUT_MS = 30_000;
|
||||
const RECONNECT_BACKOFF_MS = 10_000; // Don't retry connections faster than this after failure
|
||||
const RECONNECT_BACKOFF_MS = 10_000;
|
||||
const DEFAULT_CHROMA_DATA_DIR = path.join(os.homedir(), '.claude-mem', 'chroma');
|
||||
const CHROMA_SUPERVISOR_ID = 'chroma-mcp';
|
||||
|
||||
/**
|
||||
* Pinned chroma-mcp version for deterministic installs.
|
||||
*
|
||||
* Why pin: `uvx chroma-mcp` (unpinned) resolves whatever version PyPI happens
|
||||
* to serve at install time. That has bitten us multiple ways:
|
||||
* - #2046: transient missing httpcore/httpx after dependency resolver shifts
|
||||
* - #2085: surprise breaking changes between point releases
|
||||
* - #2102: subprocess spawn storms triggered by version drift in chromadb deps
|
||||
*
|
||||
* Pinning to a specific known-good version makes installs reproducible across
|
||||
* machines and across time. Bump deliberately, not accidentally.
|
||||
*
|
||||
* Verified 2026-04-25 with `uvx --python 3.13 chroma-mcp==0.2.6 --help` in a
|
||||
* clean uv cache: starts cleanly, no httpcore/httpx ImportError, no `--with`
|
||||
* flags required. If that changes on a future bump, re-add the flags here.
|
||||
*/
|
||||
const CHROMA_MCP_PINNED_VERSION = '0.2.6';
|
||||
|
||||
export class ChromaMcpManager {
|
||||
@@ -59,9 +30,6 @@ export class ChromaMcpManager {
|
||||
|
||||
private constructor() {}
|
||||
|
||||
/**
|
||||
* Get or create the singleton instance
|
||||
*/
|
||||
static getInstance(): ChromaMcpManager {
|
||||
if (!ChromaMcpManager.instance) {
|
||||
ChromaMcpManager.instance = new ChromaMcpManager();
|
||||
@@ -69,23 +37,16 @@ export class ChromaMcpManager {
|
||||
return ChromaMcpManager.instance;
|
||||
}
|
||||
|
||||
/**
|
||||
* Ensure the MCP client is connected to chroma-mcp.
|
||||
* Uses a connection lock to prevent concurrent connection attempts.
|
||||
* If the subprocess has died since the last use, reconnects transparently.
|
||||
*/
|
||||
private async ensureConnected(): Promise<void> {
|
||||
if (this.connected && this.client) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Backoff: don't retry connections too fast after a failure
|
||||
const timeSinceLastFailure = Date.now() - this.lastConnectionFailureTimestamp;
|
||||
if (this.lastConnectionFailureTimestamp > 0 && timeSinceLastFailure < RECONNECT_BACKOFF_MS) {
|
||||
throw new Error(`chroma-mcp connection in backoff (${Math.ceil((RECONNECT_BACKOFF_MS - timeSinceLastFailure) / 1000)}s remaining)`);
|
||||
}
|
||||
|
||||
// If another caller is already connecting, wait for that attempt
|
||||
if (this.connecting) {
|
||||
await this.connecting;
|
||||
return;
|
||||
@@ -107,14 +68,7 @@ export class ChromaMcpManager {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Internal connection logic - spawns uvx chroma-mcp and performs MCP handshake.
|
||||
* Called behind the connection lock to ensure only one connection attempt at a time.
|
||||
*/
|
||||
private async connectInternal(): Promise<void> {
|
||||
// Clean up any stale client/transport from a dead subprocess.
|
||||
// Close transport first (kills subprocess via SIGTERM) before client
|
||||
// to avoid hanging on a stuck process.
|
||||
if (this.transport) {
|
||||
try { await this.transport.close(); } catch { /* already dead */ }
|
||||
}
|
||||
@@ -129,11 +83,6 @@ export class ChromaMcpManager {
|
||||
const spawnEnvironment = this.getSpawnEnv();
|
||||
getSupervisor().assertCanSpawn('chroma mcp');
|
||||
|
||||
// On Windows, .cmd files require shell resolution. Since MCP SDK's
|
||||
// StdioClientTransport doesn't support `shell: true`, route through
|
||||
// cmd.exe which resolves .cmd/.bat extensions and PATH automatically.
|
||||
// This also fixes Git Bash compatibility (#1062) since cmd.exe handles
|
||||
// Windows-native command resolution regardless of the calling shell.
|
||||
const isWindows = process.platform === 'win32';
|
||||
const uvxSpawnCommand = isWindows ? (process.env.ComSpec || 'cmd.exe') : 'uvx';
|
||||
const uvxSpawnArgs = isWindows ? ['/c', 'uvx', ...commandArgs] : commandArgs;
|
||||
@@ -143,11 +92,6 @@ export class ChromaMcpManager {
|
||||
args: uvxSpawnArgs.join(' ')
|
||||
});
|
||||
|
||||
// Run chroma-mcp from the home directory so that pydantic-settings (used
|
||||
// by chroma-mcp internally) does not pick up .env / .env.local files from
|
||||
// the project directory. Those files often contain project-specific vars
|
||||
// that pydantic rejects with "Extra inputs are not permitted", crashing the
|
||||
// subprocess immediately. Fixes #1297.
|
||||
this.transport = new StdioClientTransport({
|
||||
command: uvxSpawnCommand,
|
||||
args: uvxSpawnArgs,
|
||||
@@ -173,7 +117,6 @@ export class ChromaMcpManager {
|
||||
try {
|
||||
await Promise.race([mcpConnectionPromise, timeoutPromise]);
|
||||
} catch (connectionError) {
|
||||
// Connection failed or timed out - kill the subprocess to prevent zombies
|
||||
clearTimeout(timeoutId!);
|
||||
logger.warn('CHROMA_MCP', 'Connection failed, killing subprocess to prevent zombie', {
|
||||
error: connectionError instanceof Error ? connectionError.message : String(connectionError)
|
||||
@@ -192,9 +135,6 @@ export class ChromaMcpManager {
|
||||
|
||||
logger.info('CHROMA_MCP', 'Connected to chroma-mcp successfully');
|
||||
|
||||
// Listen for transport close to mark connection as dead and apply backoff.
|
||||
// CRITICAL: Guard with reference check to prevent stale onclose handlers from
|
||||
// previous transports overwriting the current connection (race condition).
|
||||
const currentTransport = this.transport;
|
||||
this.transport.onclose = () => {
|
||||
if (this.transport !== currentTransport) {
|
||||
@@ -210,11 +150,6 @@ export class ChromaMcpManager {
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the uvx command arguments based on current settings.
|
||||
* In local mode: uses persistent client with local data directory.
|
||||
* In remote mode: uses http client with configured host/port/auth.
|
||||
*/
|
||||
private buildCommandArgs(): string[] {
|
||||
const settings = SettingsDefaultsManager.loadFromFile(USER_SETTINGS_PATH);
|
||||
const chromaMode = settings.CLAUDE_MEM_CHROMA_MODE || 'local';
|
||||
@@ -253,7 +188,6 @@ export class ChromaMcpManager {
|
||||
return args;
|
||||
}
|
||||
|
||||
// Local mode: persistent client with data directory
|
||||
return [
|
||||
'--python', pythonVersion,
|
||||
`chroma-mcp==${CHROMA_MCP_PINNED_VERSION}`,
|
||||
@@ -262,14 +196,6 @@ export class ChromaMcpManager {
|
||||
];
|
||||
}
|
||||
|
||||
/**
|
||||
* Call a chroma-mcp tool by name with the given arguments.
|
||||
* Lazily connects on first call. Reconnects if the subprocess has died.
|
||||
*
|
||||
* @param toolName - The chroma-mcp tool name (e.g. 'chroma_query_documents')
|
||||
* @param toolArguments - The tool arguments as a plain object
|
||||
* @returns The parsed JSON result from the tool's text output
|
||||
*/
|
||||
async callTool(toolName: string, toolArguments: Record<string, unknown>): Promise<unknown> {
|
||||
await this.ensureConnected();
|
||||
|
||||
@@ -284,9 +210,6 @@ export class ChromaMcpManager {
|
||||
arguments: toolArguments
|
||||
});
|
||||
} catch (transportError) {
|
||||
// Transport error: chroma-mcp subprocess likely died (e.g., killed by orphan reaper,
|
||||
// HNSW index corruption). Mark connection dead and retry once after reconnect (#1131).
|
||||
// Without this retry, callers see a one-shot error even though reconnect would succeed.
|
||||
this.connected = false;
|
||||
this.client = null;
|
||||
this.transport = null;
|
||||
@@ -307,14 +230,12 @@ export class ChromaMcpManager {
|
||||
}
|
||||
}
|
||||
|
||||
// MCP tools signal errors via isError flag on the CallToolResult
|
||||
if (result.isError) {
|
||||
const errorText = (result.content as Array<{ type: string; text?: string }>)
|
||||
?.find(item => item.type === 'text')?.text || 'Unknown chroma-mcp error';
|
||||
throw new Error(`chroma-mcp tool "${toolName}" returned error: ${errorText}`);
|
||||
}
|
||||
|
||||
// Extract text from MCP CallToolResult: { content: Array<{ type, text? }> }
|
||||
const contentArray = result.content as Array<{ type: string; text?: string }>;
|
||||
if (!contentArray || contentArray.length === 0) {
|
||||
return null;
|
||||
@@ -325,14 +246,9 @@ export class ChromaMcpManager {
|
||||
return null;
|
||||
}
|
||||
|
||||
// chroma-mcp returns JSON for query/get results, but plain text for
|
||||
// mutating operations (e.g. "Successfully created collection ...").
|
||||
// Try JSON parse first; if it fails, return the raw text for non-error responses.
|
||||
try {
|
||||
return JSON.parse(firstTextContent.text);
|
||||
} catch (parseError: unknown) {
|
||||
// Plain text response (e.g. "Successfully created collection cm__foo")
|
||||
// Return null for void-like success messages, callers don't need the text
|
||||
if (parseError instanceof Error) {
|
||||
logger.debug('CHROMA_MCP', 'Non-JSON response from tool, returning null', {
|
||||
toolName,
|
||||
@@ -343,10 +259,6 @@ export class ChromaMcpManager {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if the MCP connection is alive by calling chroma_list_collections.
|
||||
* Returns true if the connection is healthy, false otherwise.
|
||||
*/
|
||||
async isHealthy(): Promise<boolean> {
|
||||
try {
|
||||
await this.callTool('chroma_list_collections', { limit: 1 });
|
||||
@@ -359,17 +271,6 @@ export class ChromaMcpManager {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Deep semantic-search probe — verifies the actual query path works,
|
||||
* not just that the subprocess responds to one tool. Each stage is wrapped
|
||||
* in its own try/catch so the returned `stage` reflects where it failed.
|
||||
*
|
||||
* Stages:
|
||||
* - 'list' → chroma_list_collections (also counts collections)
|
||||
* - 'query' → chroma_query_documents against cm__claude-mem with a trivial
|
||||
* query and n_results: 1 (measures latency)
|
||||
* - 'done' → both stages succeeded
|
||||
*/
|
||||
async probeSemanticSearch(): Promise<{
|
||||
ok: boolean;
|
||||
stage: 'connect' | 'list' | 'query' | 'done';
|
||||
@@ -379,7 +280,6 @@ export class ChromaMcpManager {
|
||||
}> {
|
||||
let collections: number | undefined;
|
||||
|
||||
// Stage: list — also lazy-connects via callTool
|
||||
try {
|
||||
const listResult: any = await this.callTool('chroma_list_collections', { limit: 100 });
|
||||
if (Array.isArray(listResult)) {
|
||||
@@ -395,7 +295,6 @@ export class ChromaMcpManager {
|
||||
return { ok: false, stage: 'list', error: message };
|
||||
}
|
||||
|
||||
// Stage: query — round-trip through the embedding/vector path
|
||||
const queryStartedAt = Date.now();
|
||||
try {
|
||||
await this.callTool('chroma_query_documents', {
|
||||
@@ -426,10 +325,6 @@ export class ChromaMcpManager {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Gracefully stop the MCP connection and kill the chroma-mcp subprocess.
|
||||
* client.close() sends stdin close -> SIGTERM -> SIGKILL to the subprocess.
|
||||
*/
|
||||
async stop(): Promise<void> {
|
||||
if (!this.client) {
|
||||
logger.debug('CHROMA_MCP', 'No active MCP connection to stop');
|
||||
@@ -457,10 +352,6 @@ export class ChromaMcpManager {
|
||||
logger.info('CHROMA_MCP', 'chroma-mcp MCP connection stopped');
|
||||
}
|
||||
|
||||
/**
|
||||
* Reset the singleton instance (for testing).
|
||||
* Awaits stop() to prevent dual subprocesses.
|
||||
*/
|
||||
static async reset(): Promise<void> {
|
||||
if (ChromaMcpManager.instance) {
|
||||
await ChromaMcpManager.instance.stop();
|
||||
@@ -468,13 +359,6 @@ export class ChromaMcpManager {
|
||||
ChromaMcpManager.instance = null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Get or create a combined SSL certificate bundle for Zscaler/corporate proxy environments.
|
||||
* On macOS, combines the Python certifi CA bundle with any Zscaler certificates from
|
||||
* the system keychain. Caches the result for 24 hours at ~/.claude-mem/combined_certs.pem.
|
||||
*
|
||||
* Returns the path to the combined cert file, or undefined if not needed/available.
|
||||
*/
|
||||
private getCombinedCertPath(): string | undefined {
|
||||
const combinedCertPath = path.join(os.homedir(), '.claude-mem', 'combined_certs.pem');
|
||||
|
||||
@@ -543,11 +427,6 @@ export class ChromaMcpManager {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Build subprocess environment with SSL certificate overrides for enterprise proxy compatibility.
|
||||
* If a combined cert bundle exists (Zscaler), injects SSL_CERT_FILE, REQUESTS_CA_BUNDLE, etc.
|
||||
* Otherwise returns a plain string-keyed copy of process.env.
|
||||
*/
|
||||
private getSpawnEnv(): Record<string, string> {
|
||||
const baseEnv: Record<string, string> = {};
|
||||
for (const [key, value] of Object.entries(sanitizeEnv(process.env))) {
|
||||
|
||||
+14
-169
@@ -1,16 +1,3 @@
|
||||
/**
|
||||
* ChromaSync Service
|
||||
*
|
||||
* Automatically syncs observations and session summaries to ChromaDB via MCP.
|
||||
* This service provides real-time semantic search capabilities by maintaining
|
||||
* a vector database synchronized with SQLite.
|
||||
*
|
||||
* Uses ChromaMcpManager to communicate with chroma-mcp over stdio MCP protocol.
|
||||
* The chroma-mcp server handles its own embedding and persistent storage,
|
||||
* eliminating the need for chromadb npm package and ONNX/WASM dependencies.
|
||||
*
|
||||
* Design: Fail-fast with no fallbacks - if Chroma is unavailable, syncing fails.
|
||||
*/
|
||||
|
||||
import { ChromaMcpManager } from './ChromaMcpManager.js';
|
||||
import { ChromaSyncState, ProjectWatermarks } from './ChromaSyncState.js';
|
||||
@@ -34,13 +21,13 @@ interface StoredObservation {
|
||||
type: string;
|
||||
title: string | null;
|
||||
subtitle: string | null;
|
||||
facts: string | null; // JSON
|
||||
facts: string | null;
|
||||
narrative: string | null;
|
||||
concepts: string | null; // JSON
|
||||
files_read: string | null; // JSON
|
||||
files_modified: string | null; // JSON
|
||||
concepts: string | null;
|
||||
files_read: string | null;
|
||||
files_modified: string | null;
|
||||
prompt_number: number;
|
||||
discovery_tokens: number; // ROI metrics
|
||||
discovery_tokens: number;
|
||||
created_at: string;
|
||||
created_at_epoch: number;
|
||||
}
|
||||
@@ -57,7 +44,7 @@ interface StoredSummary {
|
||||
next_steps: string | null;
|
||||
notes: string | null;
|
||||
prompt_number: number;
|
||||
discovery_tokens: number; // ROI metrics
|
||||
discovery_tokens: number;
|
||||
created_at: string;
|
||||
created_at_epoch: number;
|
||||
}
|
||||
@@ -81,19 +68,12 @@ export class ChromaSync {
|
||||
|
||||
constructor(project: string) {
|
||||
this.project = project;
|
||||
// Chroma collection names only allow [a-zA-Z0-9._-], 3-512 chars,
|
||||
// must start/end with [a-zA-Z0-9]
|
||||
const sanitized = project
|
||||
.replace(/[^a-zA-Z0-9._-]/g, '_')
|
||||
.replace(/[^a-zA-Z0-9]+$/, ''); // strip trailing non-alphanumeric
|
||||
.replace(/[^a-zA-Z0-9]+$/, '');
|
||||
this.collectionName = `cm__${sanitized || 'unknown'}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Ensure collection exists in Chroma via MCP.
|
||||
* chroma_create_collection is idempotent - safe to call multiple times.
|
||||
* Uses collectionCreated flag to avoid redundant calls within a session.
|
||||
*/
|
||||
private async ensureCollectionExists(): Promise<void> {
|
||||
if (this.collectionCreated) {
|
||||
return;
|
||||
@@ -119,14 +99,9 @@ export class ChromaSync {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Format observation into Chroma documents (granular approach)
|
||||
* Each semantic field becomes a separate vector document
|
||||
*/
|
||||
private formatObservationDocs(obs: StoredObservation): ChromaDocument[] {
|
||||
const documents: ChromaDocument[] = [];
|
||||
|
||||
// Parse JSON fields
|
||||
const facts = obs.facts ? JSON.parse(obs.facts) : [];
|
||||
const concepts = obs.concepts ? JSON.parse(obs.concepts) : [];
|
||||
const files_read = parseFileList(obs.files_read);
|
||||
@@ -143,7 +118,6 @@ export class ChromaSync {
|
||||
title: obs.title || 'Untitled'
|
||||
};
|
||||
|
||||
// Add optional metadata fields
|
||||
if (obs.subtitle) {
|
||||
baseMetadata.subtitle = obs.subtitle;
|
||||
}
|
||||
@@ -157,7 +131,6 @@ export class ChromaSync {
|
||||
baseMetadata.files_modified = files_modified.join(',');
|
||||
}
|
||||
|
||||
// Narrative as separate document
|
||||
if (obs.narrative) {
|
||||
documents.push({
|
||||
id: `obs_${obs.id}_narrative`,
|
||||
@@ -166,7 +139,6 @@ export class ChromaSync {
|
||||
});
|
||||
}
|
||||
|
||||
// Text as separate document (legacy field)
|
||||
if (obs.text) {
|
||||
documents.push({
|
||||
id: `obs_${obs.id}_text`,
|
||||
@@ -175,7 +147,6 @@ export class ChromaSync {
|
||||
});
|
||||
}
|
||||
|
||||
// Each fact as separate document
|
||||
facts.forEach((fact: string, index: number) => {
|
||||
documents.push({
|
||||
id: `obs_${obs.id}_fact_${index}`,
|
||||
@@ -187,10 +158,6 @@ export class ChromaSync {
|
||||
return documents;
|
||||
}
|
||||
|
||||
/**
|
||||
* Format summary into Chroma documents (granular approach)
|
||||
* Each summary field becomes a separate vector document
|
||||
*/
|
||||
private formatSummaryDocs(summary: StoredSummary): ChromaDocument[] {
|
||||
const documents: ChromaDocument[] = [];
|
||||
|
||||
@@ -204,7 +171,6 @@ export class ChromaSync {
|
||||
prompt_number: summary.prompt_number || 0
|
||||
};
|
||||
|
||||
// Each field becomes a separate document
|
||||
if (summary.request) {
|
||||
documents.push({
|
||||
id: `summary_${summary.id}_request`,
|
||||
@@ -256,10 +222,6 @@ export class ChromaSync {
|
||||
return documents;
|
||||
}
|
||||
|
||||
/**
|
||||
* Add documents to Chroma in batch via MCP
|
||||
* Throws error if batch add fails
|
||||
*/
|
||||
private async addDocuments(documents: ChromaDocument[]): Promise<void> {
|
||||
if (documents.length === 0) {
|
||||
return;
|
||||
@@ -269,12 +231,9 @@ export class ChromaSync {
|
||||
|
||||
const chromaMcp = ChromaMcpManager.getInstance();
|
||||
|
||||
// Add in batches
|
||||
for (let i = 0; i < documents.length; i += this.BATCH_SIZE) {
|
||||
const batch = documents.slice(i, i + this.BATCH_SIZE);
|
||||
|
||||
// Sanitize metadata: filter out null, undefined, and empty string values
|
||||
// that chroma-mcp may reject (e.g., null subtitle from raw SQLite rows)
|
||||
const cleanMetadatas = batch.map(d =>
|
||||
Object.fromEntries(
|
||||
Object.entries(d.metadata).filter(([_, v]) => v !== null && v !== undefined && v !== '')
|
||||
@@ -290,9 +249,6 @@ export class ChromaSync {
|
||||
});
|
||||
} catch (error) {
|
||||
const errMsg = error instanceof Error ? error.message : String(error);
|
||||
// APPROVED OVERRIDE: Duplicate IDs from partial write before timeout/crash.
|
||||
// chroma_update_documents only updates *existing* IDs — it silently ignores
|
||||
// missing ones. So we delete-then-add to guarantee all IDs are written.
|
||||
if (errMsg.includes('already exist')) {
|
||||
try {
|
||||
await chromaMcp.callTool('chroma_delete_documents', {
|
||||
@@ -333,10 +289,6 @@ export class ChromaSync {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Sync a single observation to Chroma
|
||||
* Blocks until sync completes, throws on error
|
||||
*/
|
||||
async syncObservation(
|
||||
observationId: number,
|
||||
memorySessionId: string,
|
||||
@@ -346,7 +298,6 @@ export class ChromaSync {
|
||||
createdAtEpoch: number,
|
||||
discoveryTokens: number = 0
|
||||
): Promise<void> {
|
||||
// Convert ParsedObservation to StoredObservation format
|
||||
const stored: StoredObservation = {
|
||||
id: observationId,
|
||||
memory_session_id: memorySessionId,
|
||||
@@ -379,10 +330,6 @@ export class ChromaSync {
|
||||
ChromaSyncState.bump(project, 'observations', observationId);
|
||||
}
|
||||
|
||||
/**
|
||||
* Sync a single summary to Chroma
|
||||
* Blocks until sync completes, throws on error
|
||||
*/
|
||||
async syncSummary(
|
||||
summaryId: number,
|
||||
memorySessionId: string,
|
||||
@@ -392,7 +339,6 @@ export class ChromaSync {
|
||||
createdAtEpoch: number,
|
||||
discoveryTokens: number = 0
|
||||
): Promise<void> {
|
||||
// Convert ParsedSummary to StoredSummary format
|
||||
const stored: StoredSummary = {
|
||||
id: summaryId,
|
||||
memory_session_id: memorySessionId,
|
||||
@@ -422,10 +368,6 @@ export class ChromaSync {
|
||||
ChromaSyncState.bump(project, 'summaries', summaryId);
|
||||
}
|
||||
|
||||
/**
|
||||
* Format user prompt into Chroma document
|
||||
* Each prompt becomes a single document (unlike observations/summaries which split by field)
|
||||
*/
|
||||
private formatUserPromptDoc(prompt: StoredUserPrompt): ChromaDocument {
|
||||
return {
|
||||
id: `prompt_${prompt.id}`,
|
||||
@@ -441,10 +383,6 @@ export class ChromaSync {
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Sync a single user prompt to Chroma
|
||||
* Blocks until sync completes, throws on error
|
||||
*/
|
||||
async syncUserPrompt(
|
||||
promptId: number,
|
||||
memorySessionId: string,
|
||||
@@ -453,7 +391,6 @@ export class ChromaSync {
|
||||
promptNumber: number,
|
||||
createdAtEpoch: number
|
||||
): Promise<void> {
|
||||
// Create StoredUserPrompt format
|
||||
const stored: StoredUserPrompt = {
|
||||
id: promptId,
|
||||
content_session_id: '', // Not needed for Chroma sync
|
||||
@@ -476,10 +413,6 @@ export class ChromaSync {
|
||||
ChromaSyncState.bump(project, 'prompts', promptId);
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch all existing document IDs from Chroma collection via MCP
|
||||
* Returns Sets of SQLite IDs for observations, summaries, and prompts
|
||||
*/
|
||||
private async getExistingChromaIds(projectOverride?: string): Promise<{
|
||||
observations: Set<number>;
|
||||
summaries: Set<number>;
|
||||
@@ -495,7 +428,7 @@ export class ChromaSync {
|
||||
const promptIds = new Set<number>();
|
||||
|
||||
let offset = 0;
|
||||
const limit = 1000; // Large batches, metadata only = fast
|
||||
const limit = 1000;
|
||||
|
||||
logger.info('CHROMA_SYNC', 'Fetching existing Chroma document IDs...', { project: targetProject });
|
||||
|
||||
@@ -508,14 +441,12 @@ export class ChromaSync {
|
||||
include: ['metadatas']
|
||||
}) as any;
|
||||
|
||||
// chroma_get_documents returns flat arrays: { ids, metadatas, documents }
|
||||
const metadatas = result?.metadatas || [];
|
||||
|
||||
if (metadatas.length === 0) {
|
||||
break; // No more documents
|
||||
break;
|
||||
}
|
||||
|
||||
// Extract SQLite IDs from metadata
|
||||
for (const meta of metadatas) {
|
||||
if (meta && meta.sqlite_id) {
|
||||
const sqliteId = meta.sqlite_id as number;
|
||||
@@ -549,11 +480,6 @@ export class ChromaSync {
|
||||
return { observations: observationIds, summaries: summaryIds, prompts: promptIds };
|
||||
}
|
||||
|
||||
/**
|
||||
* One-time bootstrap: scan Chroma for a project, derive the highest sqlite_id
|
||||
* per doc_type, and persist as watermarks. After this runs once at install,
|
||||
* the watermark file owns the truth and Chroma is never scanned again.
|
||||
*/
|
||||
async bootstrapWatermarksFromChroma(project: string): Promise<void> {
|
||||
const existing = await this.getExistingChromaIds(project);
|
||||
const max = (set: Set<number>): number => {
|
||||
@@ -572,14 +498,6 @@ export class ChromaSync {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Backfill: Sync all observations missing from Chroma
|
||||
* Reads from SQLite and syncs in batches
|
||||
* @param projectOverride - If provided, backfill this project instead of this.project.
|
||||
* Used by backfillAllProjects() to iterate projects without mutating instance state.
|
||||
* @param storeOverride - If provided, use this SessionStore instead of creating a new one.
|
||||
* Throws error if backfill fails
|
||||
*/
|
||||
async ensureBackfilled(projectOverride?: string, storeOverride?: SessionStore): Promise<void> {
|
||||
const backfillProject = projectOverride ?? this.project;
|
||||
logger.info('CHROMA_SYNC', 'Starting smart backfill', { project: backfillProject });
|
||||
@@ -596,7 +514,6 @@ export class ChromaSync {
|
||||
logger.error('CHROMA_SYNC', 'Backfill failed', { project: backfillProject }, error instanceof Error ? error : new Error(String(error)));
|
||||
throw new Error(`Backfill failed: ${error instanceof Error ? error.message : String(error)}`);
|
||||
} finally {
|
||||
// Only close if we created it
|
||||
if (!storeOverride) {
|
||||
db.close();
|
||||
}
|
||||
@@ -623,10 +540,6 @@ export class ChromaSync {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Backfill observations missing from Chroma for a given project.
|
||||
* Returns the formatted documents that were synced.
|
||||
*/
|
||||
private async backfillObservations(
|
||||
db: SessionStore,
|
||||
backfillProject: string,
|
||||
@@ -661,9 +574,6 @@ export class ChromaSync {
|
||||
obsByDocCount.push({ obs, docs });
|
||||
}
|
||||
|
||||
// Track how many docs we've successfully written so we can bump the
|
||||
// watermark to the highest fully-synced observation id, even if a later
|
||||
// batch fails.
|
||||
let writtenDocs = 0;
|
||||
let lastSyncedObsIdx = -1;
|
||||
try {
|
||||
@@ -672,7 +582,6 @@ export class ChromaSync {
|
||||
await this.addDocuments(batch);
|
||||
writtenDocs += batch.length;
|
||||
|
||||
// Find which observation the last fully-written doc belongs to.
|
||||
let cursor = 0;
|
||||
for (let j = 0; j < obsByDocCount.length; j++) {
|
||||
cursor += obsByDocCount[j].docs.length;
|
||||
@@ -698,10 +607,6 @@ export class ChromaSync {
|
||||
return allDocs;
|
||||
}
|
||||
|
||||
/**
|
||||
* Backfill summaries missing from Chroma for a given project.
|
||||
* Returns the formatted documents that were synced.
|
||||
*/
|
||||
private async backfillSummaries(
|
||||
db: SessionStore,
|
||||
backfillProject: string,
|
||||
@@ -765,10 +670,6 @@ export class ChromaSync {
|
||||
return summaryDocs;
|
||||
}
|
||||
|
||||
/**
|
||||
* Backfill user prompts missing from Chroma for a given project.
|
||||
* Returns the formatted documents that were synced.
|
||||
*/
|
||||
private async backfillPrompts(
|
||||
db: SessionStore,
|
||||
backfillProject: string,
|
||||
@@ -808,8 +709,6 @@ export class ChromaSync {
|
||||
promptDocs.push(this.formatUserPromptDoc(prompt));
|
||||
}
|
||||
|
||||
// Prompts are 1 doc each, so the highest fully-synced prompt id moves
|
||||
// forward in lockstep with each batch.
|
||||
let lastSyncedPromptId = 0;
|
||||
try {
|
||||
for (let i = 0; i < promptDocs.length; i += this.BATCH_SIZE) {
|
||||
@@ -832,10 +731,6 @@ export class ChromaSync {
|
||||
return promptDocs;
|
||||
}
|
||||
|
||||
/**
|
||||
* Query Chroma collection for semantic search via MCP
|
||||
* Used by SearchManager for vector-based search
|
||||
*/
|
||||
async queryChroma(
|
||||
query: string,
|
||||
limit: number,
|
||||
@@ -856,18 +751,14 @@ export class ChromaSync {
|
||||
} catch (error) {
|
||||
const errorMessage = error instanceof Error ? error.message : String(error);
|
||||
|
||||
// chroma-mcp surfaces connection failures as Error messages with no structured
|
||||
// error codes or typed error classes. String matching is the only way to distinguish
|
||||
// transient connection errors (which need collection state reset) from semantic query errors.
|
||||
const isConnectionError =
|
||||
errorMessage.includes('ECONNREFUSED') || // [ANTI-PATTERN IGNORED]: chroma-mcp has no typed error classes, string matching is the only option
|
||||
errorMessage.includes('ENOTFOUND') || // [ANTI-PATTERN IGNORED]: chroma-mcp has no typed error classes, string matching is the only option
|
||||
errorMessage.includes('fetch failed') || // [ANTI-PATTERN IGNORED]: chroma-mcp has no typed error classes, string matching is the only option
|
||||
errorMessage.includes('subprocess closed') || // [ANTI-PATTERN IGNORED]: chroma-mcp has no typed error classes, string matching is the only option
|
||||
errorMessage.includes('timed out'); // [ANTI-PATTERN IGNORED]: chroma-mcp has no typed error classes, string matching is the only option
|
||||
errorMessage.includes('ECONNREFUSED') ||
|
||||
errorMessage.includes('ENOTFOUND') ||
|
||||
errorMessage.includes('fetch failed') ||
|
||||
errorMessage.includes('subprocess closed') ||
|
||||
errorMessage.includes('timed out');
|
||||
|
||||
if (isConnectionError) {
|
||||
// Reset collection state so next call attempts reconnect
|
||||
this.collectionCreated = false;
|
||||
logger.error('CHROMA_SYNC', 'Connection lost during query',
|
||||
{ project: this.project, query }, error as Error);
|
||||
@@ -881,14 +772,7 @@ export class ChromaSync {
|
||||
return this.deduplicateQueryResults(results);
|
||||
}
|
||||
|
||||
/**
|
||||
* Deduplicate Chroma query results by SQLite ID.
|
||||
* Multiple Chroma docs map to the same SQLite ID (one per field).
|
||||
* Keeps the first (best-ranked) distance and metadata per SQLite ID.
|
||||
*/
|
||||
private deduplicateQueryResults(results: any): { ids: number[]; distances: number[]; metadatas: any[] } {
|
||||
// chroma_query_documents returns nested arrays (one per query text)
|
||||
// We always pass a single query text, so we access [0]
|
||||
const ids: number[] = [];
|
||||
const seen = new Set<string>();
|
||||
const docIds = results?.ids?.[0] || [];
|
||||
@@ -900,10 +784,6 @@ export class ChromaSync {
|
||||
|
||||
for (let i = 0; i < docIds.length; i++) {
|
||||
const docId = docIds[i];
|
||||
// Extract sqlite_id from document ID (supports three formats):
|
||||
// - obs_{id}_narrative, obs_{id}_fact_0, etc (observations)
|
||||
// - summary_{id}_request, summary_{id}_learned, etc (session summaries)
|
||||
// - prompt_{id} (user prompts)
|
||||
const obsMatch = docId.match(/obs_(\d+)_/);
|
||||
const summaryMatch = docId.match(/summary_(\d+)_/);
|
||||
const promptMatch = docId.match(/prompt_(\d+)/);
|
||||
@@ -934,14 +814,6 @@ export class ChromaSync {
|
||||
return { ids, distances, metadatas };
|
||||
}
|
||||
|
||||
/**
|
||||
* Backfill all projects that have observations in SQLite but may be missing from Chroma.
|
||||
* Uses a single shared ChromaSync('claude-mem') instance and Chroma connection.
|
||||
* Per-project scoping is passed as a parameter to ensureBackfilled(), avoiding
|
||||
* instance state mutation. All documents land in the cm__claude-mem collection
|
||||
* with project scoped via metadata, matching how DatabaseManager and SearchManager operate.
|
||||
* Designed to be called fire-and-forget on worker startup.
|
||||
*/
|
||||
static async backfillAllProjects(storeOverride?: SessionStore): Promise<void> {
|
||||
const db = storeOverride ?? new SessionStore();
|
||||
const sync = new ChromaSync('claude-mem');
|
||||
@@ -952,10 +824,6 @@ export class ChromaSync {
|
||||
|
||||
logger.info('CHROMA_SYNC', `Backfill check for ${projects.length} projects`);
|
||||
|
||||
// Cold-start bootstrap: if no watermark file exists, derive watermarks
|
||||
// from one Chroma scan per project. This is the slow operation we are
|
||||
// permanently replacing — after this runs once, subsequent worker starts
|
||||
// skip the scan entirely.
|
||||
if (!ChromaSyncState.exists()) {
|
||||
logger.info('CHROMA_SYNC', 'Watermark cache missing — bootstrapping from Chroma (one-time)');
|
||||
for (const { project } of projects) {
|
||||
@@ -983,24 +851,12 @@ export class ChromaSync {
|
||||
}
|
||||
} finally {
|
||||
await sync.close();
|
||||
// Only close if we created it
|
||||
if (!storeOverride) {
|
||||
db.close();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Stamp `merged_into_project` on every Chroma document whose metadata
|
||||
* `sqlite_id` is in the provided set. Used by the worktree adoption engine
|
||||
* to keep Chroma's metadata in lockstep with SQLite after a parent branch
|
||||
* absorbs a worktree branch via merge.
|
||||
*
|
||||
* Batched: fetches docs by `sqlite_id IN sqliteIds`, rewrites metadata with
|
||||
* the new field, and calls `chroma_update_documents` once per page of up to
|
||||
* BATCH_SIZE ids. Idempotent — re-running with the same value is a no-op
|
||||
* because the write doesn't depend on the prior value.
|
||||
*/
|
||||
async updateMergedIntoProject(
|
||||
sqliteIds: number[],
|
||||
mergedIntoProject: string
|
||||
@@ -1012,7 +868,6 @@ export class ChromaSync {
|
||||
|
||||
let totalPatched = 0;
|
||||
|
||||
// Chunk the sqlite_id set to keep each Chroma call bounded.
|
||||
for (let i = 0; i < sqliteIds.length; i += this.BATCH_SIZE) {
|
||||
const idBatch = sqliteIds.slice(i, i + this.BATCH_SIZE);
|
||||
|
||||
@@ -1026,9 +881,6 @@ export class ChromaSync {
|
||||
if (docIds.length === 0) continue;
|
||||
|
||||
const metadatas = (existing?.metadatas ?? []).map(m => {
|
||||
// Merge old metadata with the new field, then filter out null/undefined/''
|
||||
// to match the sanitization other callTool sites apply (chroma-mcp
|
||||
// rejects null values in metadata).
|
||||
const merged: Record<string, any> = {
|
||||
...(m ?? {}),
|
||||
merged_into_project: mergedIntoProject
|
||||
@@ -1056,14 +908,7 @@ export class ChromaSync {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Close the ChromaSync instance
|
||||
* ChromaMcpManager is a singleton and manages its own lifecycle
|
||||
* We don't close it here - it's closed during graceful shutdown
|
||||
*/
|
||||
async close(): Promise<void> {
|
||||
// ChromaMcpManager is a singleton and manages its own lifecycle
|
||||
// We don't close it here - it's closed during graceful shutdown
|
||||
logger.info('CHROMA_SYNC', 'ChromaSync closed', { project: this.project });
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,22 +1,9 @@
|
||||
/**
|
||||
* ChromaSyncState — per-project watermark cache for Chroma backfill.
|
||||
*
|
||||
* Replaces full Chroma metadata scans on every worker start with a tiny JSON file
|
||||
* tracking the highest sqlite_id synced to Chroma for each (project, doc_type).
|
||||
*
|
||||
* File: $CLAUDE_MEM_DATA_DIR/chroma-sync-state.json
|
||||
* Schema: { [project]: { observations: number, summaries: number, prompts: number } }
|
||||
*
|
||||
* Reads/writes are synchronous — the file is small and only touched at startup
|
||||
* and after batched adds. An in-memory cache mirrors the file; writes are
|
||||
* atomic via .tmp + rename.
|
||||
*/
|
||||
import { readFileSync, writeFileSync, renameSync, mkdirSync, existsSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { SettingsDefaultsManager } from '../../shared/SettingsDefaultsManager.js';
|
||||
import { logger } from '../../utils/logger.js';
|
||||
|
||||
export type DocKind = 'observations' | 'summaries' | 'prompts';
|
||||
type DocKind = 'observations' | 'summaries' | 'prompts';
|
||||
|
||||
export interface ProjectWatermarks {
|
||||
observations: number;
|
||||
@@ -67,18 +54,15 @@ function persist(): void {
|
||||
}
|
||||
|
||||
export const ChromaSyncState = {
|
||||
/** Whether the state file exists on disk. Used by callers to detect cold-start. */
|
||||
exists(): boolean {
|
||||
return existsSync(statePath());
|
||||
},
|
||||
|
||||
/** Read current watermarks for a project. Returns zeros if unknown. */
|
||||
get(project: string): ProjectWatermarks {
|
||||
const all = load();
|
||||
return { ...(all[project] ?? ZERO) };
|
||||
},
|
||||
|
||||
/** Bump a single watermark to max(current, id). No-op if id is not greater. */
|
||||
bump(project: string, kind: DocKind, id: number): void {
|
||||
if (!Number.isInteger(id) || id <= 0) return;
|
||||
const all = load();
|
||||
@@ -90,10 +74,6 @@ export const ChromaSyncState = {
|
||||
persist();
|
||||
},
|
||||
|
||||
/**
|
||||
* Replace watermarks for a project wholesale. Used by the bootstrap path
|
||||
* after a one-time Chroma scan derives the initial highest IDs.
|
||||
*/
|
||||
replace(project: string, marks: ProjectWatermarks): void {
|
||||
const all = load();
|
||||
all[project] = { ...marks };
|
||||
@@ -101,12 +81,10 @@ export const ChromaSyncState = {
|
||||
persist();
|
||||
},
|
||||
|
||||
/** Persist any pending writes. Defensive — bump/replace flush already. */
|
||||
flush(): void {
|
||||
if (dirty) persist();
|
||||
},
|
||||
|
||||
/** Test/diagnostic helper: drop in-memory cache so the next read re-reads disk. */
|
||||
resetCache(): void {
|
||||
cache = null;
|
||||
dirty = false;
|
||||
|
||||
Reference in New Issue
Block a user