* fix: prevent duplicate worker daemons and zombie processes
Three root causes of chroma-mcp timeouts:
1. HTTP shutdown (POST /api/admin/shutdown) closed resources but never
called process.exit(). Zombie workers stayed alive, background tasks
reconnected to chroma-mcp, spawning duplicate subprocesses that all
contended for the same persistent data directory.
2. No guard against concurrent daemon startup. When hooks fired
simultaneously, multiple daemons started before either wrote a PID
file. The loser got EADDRINUSE but stayed alive because signal
handlers registered in the constructor prevented exit.
3. Corrupt 147GB HNSW index file caused all chroma queries to timeout
(MCP error -32001). Data fix: deleted corrupt collection, backfill
rebuilds from SQLite.
Code fixes:
- Add PID-based guard in daemon startup: exit if PID file process alive
- Add port-based guard in daemon startup: exit if port already bound
(runs before WorkerService constructor registers keepalive handlers)
- Add process.exit(0) after HTTP shutdown/restart completes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: aggressive startup cleanup and one-time chroma wipe for upgrade
Kill orphaned worker-service.cjs and chroma-mcp processes immediately
at startup (no age gate) while keeping 30-min threshold for mcp-server.
Wipe corrupt chroma data once on upgrade from pre-v10.3 versions —
backfill rebuilds from SQLite automatically.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: wrap shutdown handlers in try/finally to guarantee process.exit
If onShutdown() or onRestart() threw, process.exit(0) was never reached,
leaving the daemon alive as a zombie. Also removed redundant require('fs')
calls in process-manager tests where ESM imports already existed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat: replace WASM embeddings with persistent chroma-mcp MCP connection
Replace ChromaServerManager (npx chroma run + chromadb npm + ONNX/WASM)
with ChromaMcpManager, a singleton stdio MCP client that communicates with
chroma-mcp via uvx. This eliminates native binary issues, segfaults, and
WASM embedding failures that plagued cross-platform installs.
Key changes:
- Add ChromaMcpManager: singleton MCP client with lazy connect, auto-reconnect,
connection lock, and Zscaler SSL cert support
- Rewrite ChromaSync to use MCP tool calls instead of chromadb npm client
- Handle chroma-mcp's non-JSON responses (plain text success/error messages)
- Treat "collection already exists" as idempotent success
- Wire ChromaMcpManager into GracefulShutdown for clean subprocess teardown
- Delete ChromaServerManager (no longer needed)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: address PR review — connection guard leak, timer leak, async reset
- Clear connecting guard in finally block to prevent permanent reconnection block
- Clear timeout after successful connection to prevent timer leak
- Make reset() async to await stop() before nullifying instance
- Delete obsolete chroma-server-manager test (imports deleted class)
- Update graceful-shutdown test to use chromaMcpManager property name
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: prevent chroma-mcp spawn storm — zombie cleanup, stale onclose guard, reconnect backoff
Three bugs caused chroma-mcp processes to accumulate (92+ observed):
1. Zombie on timeout: failed connections left subprocess alive because
only the timer was cleared, not the transport. Now catch block
explicitly closes transport+client before rethrowing.
2. Stale onclose race: old transport's onclose handler captured `this`
and overwrote the current connection reference after reconnect,
orphaning the new subprocess. Now guarded with reference check.
3. No backoff: every failure triggered immediate reconnect. With
backfill doing hundreds of MCP calls, this created rapid-fire
spawning. Added 10s backoff on both connection failure and
unexpected process death.
Also includes ChromaSync fixes from PR review:
- queryChroma deduplication now preserves index-aligned arrays
- SQL injection guard on backfill ID exclusion lists
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Two changes fix the observer process resource leak:
1. Add ensureProcessExit to generator finally blocks in SessionRoutes and
worker-service, matching the pattern already working in SDKAgent.
2. Add stale session reaper (every 2m) that removes sessions with no active
generator and no pending work after 15m idle. This unblocks the orphan
reaper which previously skipped processes for "active" sessions.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix: self-healing claimNextMessage prevents stuck processing messages
claimAndDelete → claimNextMessage with atomic self-healing: resets stale
processing messages (>60s) back to pending before claiming. Eliminates
stuck messages from generator crashes without external timers. Removes
redundant idle-timeout reset in worker-service.ts. Adds QUEUE to logger
Component type.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: update stale comments in SessionQueueProcessor to reflect claim-confirm pattern
Comments still referenced the old claim-and-delete pattern after the
claimNextMessage rename. Updated to accurately describe the current
lifecycle where messages are marked as processing and stay in DB until
confirmProcessed() is called.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: move Date.now() inside transaction and extract stale threshold constant
- Move Date.now() inside claimNextMessage transaction closure so timestamp
is fresh if WAL contention causes retry
- Extract STALE_PROCESSING_THRESHOLD_MS to module-level constant
- Add comment clarifying strict < boundary semantics
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix: backfill all Chroma projects on worker startup
ChromaSync.ensureBackfilled() existed but was never called. After
v10.2.2's bun cache clear destroyed the ONNX model cache, Chroma only
had ~2 days of embeddings while SQLite had 49k+ observations.
- Add static backfillAllProjects() to ChromaSync — iterates all projects
in SQLite, creates temporary ChromaSync per project, runs smart diff
- Call backfillAllProjects() fire-and-forget on worker startup
- Add 'CHROMA_SYNC' to logger Component type (pre-existing gap)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: sanitize project names for Chroma collection naming
Replace characters outside [a-zA-Z0-9._-] with underscores so projects
like "YC Stuff" map to collection "cm__YC_Stuff" instead of failing
Chroma's collection name validation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: route backfill to shared cm__claude-mem collection, harden sanitization
- Use single ChromaSync('claude-mem') in backfillAllProjects() instead of
per-project instances, matching how DatabaseManager and SearchManager
operate — fixes critical bug where backfilled data landed in orphaned
collections that no search path reads from
- Strip trailing non-alphanumeric chars from sanitized collection names
to satisfy Chroma's end-character constraint
- Guard backfill behind Chroma server readiness to avoid N spurious error
logs when Chroma failed to start
- Use CHROMA_SYNC log component consistently for backfill messages
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: pass project as parameter to ensureBackfilled instead of mutating instance state
Eliminates shared mutable state in backfillAllProjects() loop. Project
scoping is now passed explicitly via parameter to both ensureBackfilled()
and getExistingChromaIds(), keeping a single Chroma connection while
avoiding fragile instance property mutation across iterations.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Remove nuclear `bun pm cache rm` from smart-install.js and
sync-marketplace.cjs (only needed for removed sharp dependency).
Add `bun install` in cache version directory after sync so worker
can resolve dependencies. Move HuggingFace model cache to
~/.claude-mem/models/ so reinstalls don't corrupt it. Add self-healing
retry for Protobuf parsing failures.
Fixes recurring issues #1104, #1105, #1110.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: use bun install in sync, add node-addon-api for sharp, consolidate PendingMessageStore
- Switch sync-marketplace from npm to bun install
- Add node-addon-api as dev dep so sharp builds under bun
- Consolidate duplicate PendingMessageStore instantiation in worker-service finally block
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* build assets
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
On Linux, Bun's libuv calls fstat() on inherited pipe file descriptors
and crashes with EINVAL when the pipe originates from Claude Code's hook
system. This causes all PostToolUse hooks to fail silently, preventing
observations from being recorded.
The fix reads stdin entirely in the Node.js parent process (bun-runner.js)
before spawning Bun, then writes the buffered data to a fresh pipe created
by Node's child_process.spawn(). Bun receives a standard pipe that it can
fstat() without errors.
Changes:
- Add collectStdin() to buffer piped input in Node.js with 5s safety timeout
- Change stdio from 'inherit' to ['pipe'|'ignore', 'inherit', 'inherit']
- Write buffered stdin to child.stdin then close for proper EOF signaling
- Handle edge cases: TTY stdin, no stdin, read errors
Fixes#646
Co-authored-by: yczc3999 <zxfgds@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Address PR #1125 review feedback - both fetches now start simultaneously
via Promise.all instead of sequential-then-parallel.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add systemMessage field to HookResult so SessionStart can display a
colored timeline directly to the user in the CLI. The handler now
parallel-fetches both markdown (for Claude context) and ANSI-colored
(for user display) timelines, appending a viewer URL link.
Also update default settings to hide verbose token columns (read/work
tokens, savings amount) and disable full observation expansion, keeping
the cleaner index-only view by default.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wrap SDK query loop in try/finally so subprocess cleanup runs on error paths.
Swap Chroma binary check order to try project-level .bin first (common case).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Chroma requires client-side embeddings — the server is storage only.
The previous commit incorrectly removed @chroma-core/default-embed.
Uses DefaultEmbeddingFunction({ wasm: true }) which forces the WASM
backend instead of native ONNX binaries. Same model (all-MiniLM-L6-v2),
same embeddings, but works on all platforms without segfaults or
ENOENT errors (#1104, #1105, #1110).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Strip CLAUDECODE env var from SDK subprocesses to prevent "cannot be
launched inside another Claude Code session" error (Claude Code 2.1.42+)
- Lazy-load @chroma-core/default-embed to avoid eagerly pulling in
sharp native binaries at bundle startup (fixes ERR_DLOPEN_FAILED)
- Add stderr capture to SDK spawn for diagnosing future process failures
- Exclude lockfiles from marketplace rsync and delete stale lockfiles
before npm install to prevent native dep version mismatches
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Resolve conflicts between Chroma HTTP server PR and main branch changes
(folder CLAUDE.md, exclusion settings, Zscaler SSL, transport cleanup).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace hardcoded TEST-008 build ID with real package version. Add worker
filesystem path, uptime counter, and AI provider status (including last
interaction success/failure tracking) to the health endpoint response.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Missing return statement and closing brace in the programming errors
check caused a build failure after merging main.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes to make OpenClaw agent observations work end-to-end:
1. Session init in before_agent_start — the worker's privacy check
requires a stored user prompt; without calling /api/sessions/init,
all observations were skipped as "private"
2. Race condition fix in agent_end — await summarize before sending
complete, preventing session deletion before in-flight observation
POSTs arrive
3. OAuth token pass-through in buildIsolatedEnv — spawned Claude CLI
processes now receive CLAUDE_CODE_OAUTH_TOKEN from the worker's
env when no explicit API key is configured
Also adds agent-specific emoji mapping and dynamic project naming
for the Telegram observation feed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Reduce timeouts to eliminate 10-30s startup delay when worker is dead
(common on WSL2 after hibernate). Add stale PID detection, graceful
error handling across all handlers, and error classification that
distinguishes worker unavailability from handler bugs.
- HEALTH_CHECK 30s→3s, new POST_SPAWN_WAIT (5s), PORT_IN_USE_WAIT (3s)
- isProcessAlive() with EPERM handling, cleanStalePidFile()
- getPluginVersion() try-catch for shutdown race (#1042)
- isWorkerUnavailableError: transport+5xx+429→exit 0, 4xx→exit 2
- No-op handler for unknown event types (#984)
- Wrap all handler fetch calls in try-catch for graceful degradation
- CLAUDE_MEM_HEALTH_TIMEOUT_MS env var override with validation
- Check CLAUDE_MEM_DATA_DIR env var before settings.json (Greptile)
- Derive project before DB check for consistent output (Greptile)
- Include project in error fallback output (Greptile)
- Set executable permission for shebang compatibility (Greptile)
Lightweight script for Claude Code statusLineCommand integration.
Returns per-project observation and prompt counts via direct SQLite
read (~15ms, no HTTP, no worker dependency).
Counts are scoped with WHERE project = ? to prevent inflated totals
from cross-project observations. Supports CLAUDE_MEM_DATA_DIR from
settings.json for custom data directory configurations.
Cherry-picked source changes from PR #657 (224 commits behind main).
Adds `claude-mem generate` and `claude-mem clean` CLI commands:
- New src/cli/claude-md-commands.ts with generateClaudeMd() and cleanClaudeMd()
- Worker service generate/clean case handlers with --dry-run support
- CLAUDE_MD logger component type
- Uses shared isDirectChild from path-utils.ts (DRY improvement over PR original)
Skipped from PR: 91 CLAUDE.md file deletions (stale), build artifacts,
.claude/plans/ dev artifact, smart-install.js shell alias auto-injection
(aggressive profile modification without consent).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Cherry-picked from PR #844 by @thusdigital. Sessions stayed in active
sessions map forever after summarize, causing the orphan reaper to think
all processes were still active. Adds session-complete as Stop phase 2
hook that calls POST /api/sessions/complete to remove sessions from the
active map, allowing the reaper to correctly identify and clean up
orphaned worker processes. Fixes#842.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bun 1.1.14+ is required because:
- `.changes` property on SQLite statements (added in 1.1.14)
- Multi-statement SQL support in db.run() (fixed in 1.0.26)
Older versions cause silent failures in database migrations.
Fixes#519🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Adds automatic detection and handling of Zscaler enterprise security certificates
on macOS. Combines standard certifi CA certificates with Zscaler certificates into
a single bundle, passed via SSL_CERT_FILE/REQUESTS_CA_BUNDLE/CURL_CA_BUNDLE env vars
to the chroma-mcp subprocess. Certificate bundle is cached for 24 hours. Falls back
gracefully when Zscaler is not present, with no impact on non-Zscaler environments.
Co-Authored-By: RClark4958 <rickdclark48@gmail.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Alex Newman