diff --git a/docs/context/CHROMA.md b/docs/context/CHROMA.md deleted file mode 100644 index 950e36ad..00000000 --- a/docs/context/CHROMA.md +++ /dev/null @@ -1,542 +0,0 @@ -# Chroma Vector Database - Hybrid Semantic Search - -## Overview - -Claude-Mem v5.0.0 introduced **Chroma**, a vector database that enables semantic search across your memory stream. Combined with SQLite's FTS5 keyword search, this creates a powerful **hybrid search architecture** that finds contextually relevant observations using both meaning and keywords. - -**Key Benefits:** -- 🧠 **Semantic Search** - Find observations by meaning, not just keywords -- 🔍 **Hybrid Architecture** - Combines semantic similarity with keyword matching -- ⏱️ **Recency Filtering** - Focus on recent 90 days for relevant context -- ⚡ **Fast Performance** - Semantic search under 200ms with 8,000+ documents -- 🔄 **Auto-Sync** - ChromaSync service keeps vectors updated automatically - -## What is Chroma? - -[ChromaDB](https://www.trychroma.com/) is an open-source vector database designed for AI applications. It stores text as **vector embeddings** - mathematical representations that capture semantic meaning. - -**Example:** -``` -Query: "authentication bug" -Keyword Match: Must contain both "authentication" AND "bug" -Semantic Match: Also finds "login error", "auth failure", "sign-in issue" -``` - -Semantic search understands that "authentication bug" is conceptually similar to "login error" even though they share no keywords. - -## Architecture - -### Hybrid Search Flow - -``` -┌──────────────────────────────────────────────────────────────┐ -│ User Query: "How does authentication work?" │ -└──────────────────────────────────────────────────────────────┘ - ↓ - ┌─────────────────┴─────────────────┐ - ↓ ↓ -┌──────────────────────┐ ┌──────────────────────┐ -│ Chroma Semantic │ │ SQLite FTS5 │ -│ Vector Similarity │ │ Keyword Search │ -│ │ │ │ -│ Finds conceptually │ │ Finds exact/fuzzy │ -│ similar observations │ │ keyword matches │ -└──────────────────────┘ └──────────────────────┘ - ↓ ↓ - └─────────────────┬─────────────────┘ - ↓ - ┌─────────────────────────────────┐ - │ Merge Results │ - │ - Deduplicate by ID │ - │ - Sort by relevance + recency │ - │ - Filter by 90-day window │ - └─────────────────────────────────┘ - ↓ - ┌─────────────────────────────────┐ - │ Return Top Matches │ - │ Semantic + Keyword combined │ - └─────────────────────────────────┘ -``` - -### ChromaSync Service - -The **ChromaSync** service (`src/services/sync/ChromaSync.ts`) automatically synchronizes observations to Chroma: - -**When Observations Are Synced:** -1. **Session Summary** - After each session completes, all new observations synced -2. **Worker Startup** - On initialization, checks for unsynced observations -3. **Manual Trigger** - Can force sync via internal API (development only) - -**What Gets Embedded:** -- Observation ID (unique identifier) -- Title (compressed learning statement) -- Narrative (detailed explanation) -- Project path (for project-specific filtering) -- Timestamp (for recency filtering) -- Concepts (semantic tags) -- File references (associated code files) - -**Embedding Model:** -- Currently using Chroma's default embedding function -- Future: Configurable embedding models (e.g., OpenAI, sentence-transformers) - -### Data Structure - -**SQLite (Source of Truth):** -```sql -CREATE TABLE observations ( - id INTEGER PRIMARY KEY, - title TEXT, - narrative TEXT, - facts TEXT, - concepts TEXT, - files TEXT, - type TEXT, - projectPath TEXT, - createdAt INTEGER -); -``` - -**Chroma (Vector Embeddings):** -```typescript -{ - ids: ["obs_12345"], - embeddings: [[0.123, -0.456, ...]], // 384-dimensional vector - documents: ["Title: Authentication flow\nNarrative: Implemented..."], - metadatas: [{ - type: "feature", - project: "claude-mem", - timestamp: 1698765432000, - concepts: "pattern,architecture" - }] -} -``` - -## How Semantic Search Works - -### Vector Embeddings - -Text converted to high-dimensional vectors that capture meaning: - -``` -"user authentication" → [0.12, -0.34, 0.56, ..., 0.78] -"login system" → [0.15, -0.32, 0.54, ..., 0.81] -"database schema" → [-0.45, 0.67, -0.23, ..., 0.12] -``` - -Notice: "user authentication" and "login system" have similar vectors (close in vector space), while "database schema" is distant. - -### Similarity Search - -Chroma uses **cosine similarity** to find nearest neighbors: - -```typescript -// Query embedding -query: "authentication bug" -query_vector: [0.14, -0.33, 0.55, ..., 0.79] - -// Find observations with similar vectors -results = chroma.query( - query_vector, - n_results: 10, - where: { timestamp: { $gte: now - 90_days } } -) -``` - -**Result Ranking:** -- Higher cosine similarity = more semantically similar -- Filtered by 90-day recency window -- Combined with keyword matches from FTS5 - -## 90-Day Recency Filtering - -Why 90 days? - -**Rationale:** -- Recent context more likely relevant to current work -- Prevents very old observations from diluting results -- Balances completeness with relevance -- Reduces vector search space for faster queries - -**Implementation:** -```typescript -const ninetyDaysAgo = Date.now() - (90 * 24 * 60 * 60 * 1000); - -// Chroma metadata filter -where: { - timestamp: { $gte: ninetyDaysAgo } -} - -// SQLite WHERE clause -WHERE createdAt >= ? -``` - -**Configurable?** -- Not currently user-configurable -- Hard-coded in `src/servers/search-server.ts` -- Future: Add `CLAUDE_MEM_RECENCY_DAYS` environment variable - -## MCP Tool Integration - -All 9 MCP search tools benefit from hybrid search: - -### search_observations (Hybrid) - -```typescript -// Keyword-only (v4.x) -search_observations(query: "authentication") -// Returns: Observations containing "authentication" - -// Hybrid semantic + keyword (v5.x) -search_observations(query: "authentication") -// Returns: Observations with "authentication" PLUS semantically similar: -// - "login system" -// - "user credentials" -// - "session management" -``` - -### get_timeline_by_query (Semantic-First) - -```typescript -// Uses Chroma to find best match, then builds timeline -get_timeline_by_query( - query: "when did we implement the viewer UI?", - mode: "auto", - depth_before: 10, - depth_after: 10 -) - -// Chroma finds: Observation #4057 "Web-Based Viewer UI for Real-Time Memory Stream" -// Returns: Timeline with 10 observations before + anchor + 10 after -``` - -### Benefits Across All Tools - -- **find_by_concept**: Semantic similarity finds related concepts -- **find_by_file**: Finds semantically similar code changes -- **find_by_type**: Better relevance ranking within type -- **get_recent_context**: Prioritizes semantically relevant recent context - -## Performance - -### Benchmarks (8,279 vector documents) - -| Operation | Time | Notes | -|-----------|------|-------| -| **Semantic Query** | 150-200ms | 90-day window, top 10 results | -| **Keyword Query (FTS5)** | 5-10ms | Full-text search | -| **Hybrid Query** | 160-220ms | Combined semantic + keyword | -| **Initial Sync** | 2-5 min | First-time embedding of all observations | -| **Incremental Sync** | 100-500ms | 1-10 new observations per session | - -### Memory Usage - -- **Chroma DB Size**: ~50MB for 8,000 observations -- **Embeddings**: 384 dimensions × 4 bytes = 1.5KB per observation -- **Metadata**: ~500 bytes per observation (project, type, timestamp) -- **Total**: ~2KB per observation in Chroma - -### Optimization Tips - -1. **Reduce vector dimensions**: Use smaller embedding models (future) -2. **Adjust recency window**: Narrow to 30/60 days for faster queries -3. **Limit result count**: Request fewer results (n_results=5 vs 10) -4. **Project filtering**: Add project filter to metadata query - -## Installation & Dependencies - -### Python Requirement - -Chroma requires Python 3.7+ installed: - -**Check Python:** -```bash -python3 --version -# Should show: Python 3.7.x or higher -``` - -**Install Python (if needed):** -- **macOS**: `brew install python3` -- **Windows**: Download from [python.org](https://www.python.org/downloads/) -- **Linux**: `apt-get install python3` or `yum install python3` - -### ChromaDB Installation - -Chroma installed automatically as npm dependency: - -```bash -npm install -# Installs: chromadb (Python package via node-gyp bindings) -``` - -**Manual Installation (if auto-install fails):** -```bash -pip3 install chromadb -``` - -### Troubleshooting Installation - -**Error: "Python not found"** -```bash -# Set Python path explicitly -export PYTHON=/usr/local/bin/python3 -npm install -``` - -**Error: "chromadb module not found"** -```bash -# Reinstall chromadb -pip3 install --upgrade chromadb - -# Verify installation -python3 -c "import chromadb; print(chromadb.__version__)" -``` - -**Error: "node-gyp build failed"** -```bash -# Install build tools -# macOS: xcode-select --install -# Windows: npm install --global windows-build-tools -# Linux: apt-get install build-essential -``` - -## Configuration - -### Environment Variables - -Currently no user-configurable settings. Future options: - -```json -// Proposed for future versions -{ - "env": { - "CLAUDE_MEM_CHROMA_ENABLED": "true", // Enable/disable Chroma - "CLAUDE_MEM_CHROMA_PATH": "~/.claude-mem/chroma", // DB location - "CLAUDE_MEM_EMBEDDING_MODEL": "default", // Embedding model choice - "CLAUDE_MEM_RECENCY_DAYS": "90", // Recency window - "CLAUDE_MEM_VECTOR_DIM": "384" // Embedding dimensions - } -} -``` - -### Disabling Chroma (Future) - -To disable semantic search and use keyword-only: - -```json -{ - "env": { - "CLAUDE_MEM_CHROMA_ENABLED": "false" - } -} -``` - -Falls back to SQLite FTS5 keyword search only. - -## Database Maintenance - -### Location - -``` -~/.claude-mem/chroma/ -├── chroma.sqlite3 # Chroma metadata database -└── index/ # Vector index files - └── *.bin # Binary vector data -``` - -### Backup - -```bash -# Backup entire Chroma directory -cp -r ~/.claude-mem/chroma ~/.claude-mem/chroma.backup - -# Restore from backup -rm -rf ~/.claude-mem/chroma -cp -r ~/.claude-mem/chroma.backup ~/.claude-mem/chroma -``` - -### Reset Chroma (Force Resync) - -```bash -# Delete Chroma database -rm -rf ~/.claude-mem/chroma - -# Restart worker to trigger full resync -npm run worker:restart - -# Check logs for sync progress -npm run worker:logs -``` - -**Note**: Resync can take 2-5 minutes for thousands of observations. - -### Disk Space Management - -**Chroma grows with observations:** -- 1,000 observations ≈ 5MB -- 10,000 observations ≈ 50MB -- 100,000 observations ≈ 500MB - -**Cleanup old observations:** -```sql --- Delete observations older than 1 year --- This will trigger Chroma resync on next startup -sqlite3 ~/.claude-mem/claude-mem.db \ - "DELETE FROM observations WHERE createdAt < strftime('%s', 'now', '-1 year') * 1000;" -``` - -## Advanced Usage - -### Direct Chroma Queries (Development) - -For debugging or custom queries: - -```typescript -import { ChromaSync } from './services/sync/ChromaSync'; - -const sync = new ChromaSync(); -await sync.initialize(); - -// Query Chroma directly -const results = await sync.query({ - queryTexts: ["authentication implementation"], - nResults: 10, - where: { - type: "feature", - timestamp: { $gte: Date.now() - 90_days } - } -}); - -console.log(results.ids, results.distances, results.documents); -``` - -### Custom Embedding Models (Future) - -Chroma supports multiple embedding models: - -```typescript -// Future configuration -const sync = new ChromaSync({ - embeddingModel: "sentence-transformers/all-MiniLM-L6-v2", // Smaller, faster - // or: "text-embedding-ada-002" (OpenAI, requires API key) - // or: "all-mpnet-base-v2" (Higher quality, slower) -}); -``` - -### Metadata Filtering - -Chroma supports advanced metadata queries: - -```typescript -// Find observations by type and project -results = await sync.query({ - queryTexts: ["API design"], - where: { - $and: [ - { type: { $in: ["decision", "feature"] } }, - { project: "claude-mem" } - ] - } -}); - -// Find recent observations -results = await sync.query({ - queryTexts: ["database schema"], - where: { - timestamp: { $gte: Date.now() - 30_days } - } -}); -``` - -## Comparison: Semantic vs Keyword Search - -| Aspect | Semantic (Chroma) | Keyword (FTS5) | -|--------|-------------------|----------------| -| **Speed** | 150-200ms | 5-10ms | -| **Accuracy** | High (meaning-based) | Medium (exact match) | -| **Storage** | ~2KB per observation | ~500 bytes per observation | -| **Conceptual Matching** | ✅ Yes | ❌ No | -| **Exact Match** | ❌ Not guaranteed | ✅ Always | -| **Typo Tolerance** | ✅ High | ⚠️ Limited (fuzzy) | -| **Dependencies** | Python + chromadb | None (SQLite built-in) | -| **Recency Bias** | ✅ Built-in (90 days) | Manual filtering | - -**Best Practice:** Use hybrid search (both) for optimal results. - -## Troubleshooting - -### "Chroma not found" Error - -**Symptom:** Worker logs show "Chroma not available, using keyword-only search" - -**Solution:** -```bash -# Check Python installation -python3 --version - -# Reinstall chromadb -pip3 install chromadb - -# Restart worker -npm run worker:restart -``` - -### Slow Query Performance - -**Symptom:** Searches taking >1 second - -**Solutions:** -1. Reduce recency window (edit `src/servers/search-server.ts`) -2. Limit result count (`nResults: 5` instead of 10) -3. Add project filter to narrow search space -4. Check Chroma index size (may need rebuild) - -### Out of Memory Errors - -**Symptom:** Worker crashes with "JavaScript heap out of memory" - -**Solution:** -```bash -# Increase Node.js heap size -export NODE_OPTIONS="--max-old-space-size=4096" - -# Restart worker -npm run worker:restart -``` - -### Sync Taking Too Long - -**Symptom:** Initial Chroma sync takes >10 minutes - -**Possible Causes:** -- Large number of observations (>10,000) -- Slow embedding model -- Limited CPU resources - -**Solutions:** -1. Let it complete (one-time cost) -2. Delete very old observations to reduce count -3. Close resource-intensive apps during sync - -## Future Enhancements - -Potential improvements for future versions: - -- **Configurable Recency**: User-defined recency window (30/60/90/365 days) -- **Custom Embeddings**: Choose embedding model (quality vs speed trade-off) -- **Incremental Updates**: Update existing vectors instead of full resync -- **Semantic Filters**: Search by semantic concept ("all architectural decisions") -- **Multi-Language Support**: Embeddings optimized for non-English code/docs -- **Clustering**: Auto-cluster related observations for discovery -- **Visualization**: 2D/3D visualization of vector space (similar observations near each other) - -## Resources - -- **ChromaDB Documentation**: https://docs.trychroma.com/ -- **Source Code**: `src/services/sync/ChromaSync.ts` -- **Search Server**: `src/servers/search-server.ts` -- **Python Package**: https://pypi.org/project/chromadb/ - ---- - -**Powered by ChromaDB** | **Hybrid Semantic + Keyword Search** | **90-Day Recency Window** diff --git a/docs/context/CLAUDE.md b/docs/context/CLAUDE.md deleted file mode 100644 index d3128e1b..00000000 --- a/docs/context/CLAUDE.md +++ /dev/null @@ -1,104 +0,0 @@ -# Claude-Mem Context Documentation - -## What This Folder Is - -This `docs/context/` folder contains **internal documentation** - planning documents, design references, audits, and work-in-progress materials that support development but are NOT user-facing. - -## Folder Structure - -``` -docs/ -├── public/ ← User-facing Mintlify docs (DO NOT put internal docs there) -│ └── *.mdx - Official documentation -└── context/ ← You are here (Internal documentation) - ├── *.md - Planning docs, audits, references - ├── *-plan.md - Implementation plans - ├── *-audit.md - Code audits and reviews - ├── agent-sdk-*.md - SDK reference materials - └── subdirs/ - Organized by topic -``` - -## What Belongs Here - -**Internal Documentation** (`.md` format): -- Planning documents (`*-plan.md`, `*-outline.md`) -- Implementation analysis (`*-audit.md`, `*-code-reference.md`) -- Error tracking (`typescript-errors.md`) -- Design documents not ready for public docs -- PR review responses -- Reference materials (like `agent-sdk-ref.md`) -- Work-in-progress documentation -- Technical investigations and postmortems -- Architecture analysis documents - -**Examples from this folder:** -- `mem-search-technical-architecture.md` - Deep technical reference -- `search-architecture-analysis.md` - Implementation analysis -- `agent-sdk-ref.md` - SDK reference for developers -- `typescript-errors.md` - Error tracking during development -- `worker-service-architecture.md` - Internal architecture notes -- `processing-indicator-audit.md` - Code audit document - -## What Does NOT Belong Here - -**User-Facing Documentation** goes in `/docs/public/`: -- User guides and tutorials -- Official architecture documentation -- Installation instructions -- Configuration guides -- Best practices for users -- Troubleshooting guides - -**Rule of Thumb:** -- If a user would read it → `/docs/public/` (as `.mdx`) -- If only developers/contributors need it → `/docs/context/` (as `.md`) - -## File Organization - -### By Type -- `*-plan.md` - Implementation plans for features -- `*-audit.md` - Code audits and reviews -- `*-postmortem.md` - Analysis of issues or incidents -- `*-reference.md` - Technical reference materials -- `*-analysis.md` - Architecture or design analysis - -### By Topic -- Create subdirectories for related documents -- Example: `claude-code/` for Claude Code specific docs -- Example: `architecture/` for internal architecture notes - -## Development Workflow - -### When to Create Context Docs - -1. **Planning Phase** - Before implementing a feature - - Create `feature-name-plan.md` - - Outline implementation steps - - Document decisions and tradeoffs - -2. **During Development** - Track issues and decisions - - Create `feature-name-audit.md` for code reviews - - Update `typescript-errors.md` for build issues - - Document gotchas in topic-specific files - -3. **After Implementation** - Preserve knowledge - - Create `feature-name-postmortem.md` if issues occurred - - Update architecture analysis documents - - Archive plan docs (don't delete - useful for history) - -### Graduating to Public Docs - -When internal docs are polished enough for users: -1. Convert `.md` to `.mdx` format -2. Add Mintlify frontmatter -3. Move to appropriate `/docs/public/` subdirectory -4. Add to `docs.json` navigation -5. Keep original in `/docs/context/` for reference - -## Summary - -**Simple Rule**: -- `/docs/context/` = Internal docs, plans, references, audits ← YOU ARE HERE -- `/docs/public/` = Official user documentation (Mintlify .mdx files) - -**Purpose**: This folder preserves development context, design decisions, and technical knowledge that helps contributors understand WHY things work the way they do, even if users don't need those details. diff --git a/docs/context/CWD_CONTEXT_FIX.md b/docs/context/CWD_CONTEXT_FIX.md deleted file mode 100644 index 24b5ddc5..00000000 --- a/docs/context/CWD_CONTEXT_FIX.md +++ /dev/null @@ -1,164 +0,0 @@ -# CWD Context Fix - Technical Documentation - -## Overview - -This fix adds working directory (CWD) context propagation through the entire claude-mem pipeline, enabling the SDK agent to have spatial awareness of which directory/repository it's observing. - -## Problem Statement - -Previously, the SDK agent would: -- Search wrong repositories when analyzing file operations -- Report "file not found" for files that actually exist -- Lack context about which project was being worked on -- Generate inaccurate observations due to spatial confusion - -## Solution - -The CWD information now flows through the entire system: - -``` -Hook Input (cwd) → Worker API (cwd) → SessionManager (cwd) → SDK Agent (tool_cwd) -``` - -## Data Flow - -### 1. Hook Layer (`save-hook.ts`) -```typescript -export interface PostToolUseInput { - session_id: string; - cwd: string; // ← Captured from Claude Code - tool_name: string; - tool_input: any; - tool_response: any; -} -``` - -The hook extracts `cwd` and includes it in the worker API request: -```typescript -body: JSON.stringify({ - tool_name, - tool_input, - tool_response, - prompt_number, - cwd: cwd || '' // ← Passed to worker -}) -``` - -### 2. Worker Service (`worker-service.ts`) -```typescript -const { tool_name, tool_input, tool_response, prompt_number, cwd } = req.body; - -this.sessionManager.queueObservation(sessionDbId, { - tool_name, - tool_input, - tool_response, - prompt_number, - cwd // ← Forwarded to queue -}); -``` - -### 3. Session Manager (`SessionManager.ts`) -```typescript -session.pendingMessages.push({ - type: 'observation', - tool_name: data.tool_name, - tool_input: data.tool_input, - tool_response: data.tool_response, - prompt_number: data.prompt_number, - cwd: data.cwd // ← Included in message queue -}); -``` - -### 4. SDK Agent (`SDKAgent.ts`) -```typescript -content: buildObservationPrompt({ - id: 0, - tool_name: message.tool_name!, - tool_input: JSON.stringify(message.tool_input), - tool_output: JSON.stringify(message.tool_response), - created_at_epoch: Date.now(), - cwd: message.cwd // ← Passed to prompt builder -}) -``` - -### 5. Prompt Generation (`prompts.ts`) -```typescript -return ` - ${obs.tool_name} - ${new Date(obs.created_at_epoch).toISOString()}${obs.cwd ? ` - ${obs.cwd}` : ''} // ← Included in XML - ${JSON.stringify(toolInput, null, 2)} - ${JSON.stringify(toolOutput, null, 2)} -`; -``` - -## SDK Agent Prompt Changes - -The init prompt now includes a "SPATIAL AWARENESS" section: - -``` -SPATIAL AWARENESS: Tool executions include the working directory (tool_cwd) to help you understand: -- Which repository/project is being worked on -- Where files are located relative to the project root -- How to match requested paths to actual execution paths -``` - -## Example Usage - -When a user executes a read operation in `/home/user/my-project`: - -```xml - - ReadTool - 2025-11-10T19:18:03.065Z - /home/user/my-project - - { - "path": "src/index.ts" - } - - - { - "content": "export default..." - } - - -``` - -The SDK agent now knows: -1. The operation happened in `/home/user/my-project` -2. The file `src/index.ts` is relative to that directory -3. Which repository context to search when generating observations - -## Testing - -8 comprehensive tests validate the CWD propagation: - -```bash -npx tsx --test tests/cwd-propagation.test.ts -``` - -All tests verify: -- Type interfaces include `cwd` fields -- Hook extracts and passes `cwd` -- Worker accepts and forwards `cwd` -- SDK agent includes `cwd` in prompts -- End-to-end flow is correct - -## Benefits - -1. **Spatial Awareness**: SDK agent knows which directory/repository it's observing -2. **Accurate Path Matching**: Can verify if requested paths match executed paths -3. **Better Summaries**: Won't search wrong repositories or report false negatives -4. **Works with All Models**: Even Haiku benefits from correct context (no need for Opus workaround) - -## Backward Compatibility - -- `cwd` is optional in all interfaces (`cwd?: string`) -- Missing `cwd` values are handled gracefully (defaults to empty string) -- Existing observations without `cwd` continue to work -- No database migration required (CWD is transient, not persisted) - -## Related Issues - -Fixes issue #73 (CWD context missing from SDK agent) diff --git a/docs/context/PR_REVIEW_RESPONSE.md b/docs/context/PR_REVIEW_RESPONSE.md deleted file mode 100644 index 38c7c105..00000000 --- a/docs/context/PR_REVIEW_RESPONSE.md +++ /dev/null @@ -1,219 +0,0 @@ -# Response to PR Review #47 - -## Executive Summary - -Thank you for the thorough review. Most of the "issues" identified are actually **intentional architectural decisions** made to solve production failures. The comprehensive analysis docs (JUST-FUCKING-RUN-IT.md, LINE-BY-LINE-CASCADING-BULLSHIT.md) document why these changes were necessary. - -However, you've identified **2 legitimate issues** that need fixing: -1. ✅ **Race condition in worker startup** - Valid concern, needs fixing -2. ✅ **Watch mode in production** - Appears to be unintentional leftover from development - -The other concerns are **working as intended** based on documented architectural decisions. - ---- - -## Detailed Response to Each Concern - -### ⚠️ Issue #1: Race Condition in Worker Health Check - **VALID CONCERN** - -**Review Comment**: "The spawn() call inside the close event handler is non-blocking, but the function returns immediately. Hooks may attempt HTTP requests before worker has started." - -**Our Response**: **You're absolutely right**. This is a legitimate race condition we need to fix. - -**However**, the suggested fixes (async/await health check, retry loops) are exactly what we intentionally removed because they were causing production failures (see Observation #3602, #3600). - -**Proposed Solution**: -The hooks already have proper error handling for `ECONNREFUSED` with actionable user messages: -```typescript -if (error.cause?.code === 'ECONNREFUSED' || error.name === 'TimeoutError' || error.message.includes('fetch failed')) { - throw new Error("There's a problem with the worker. If you just updated, type `pm2 restart claude-mem-worker` in your terminal to continue"); -} -``` - -We should either: -1. Document this as expected behavior (fire-and-forget spawn) -2. Add a single synchronous `pm2 list` check after spawn to verify startup -3. Keep the current approach and rely on hook error messages - -**We will NOT re-add**: Retry loops, health check polling, or arbitrary delays. Those caused the 100% failure rate we just fixed. - ---- - -### ⚠️ Issue #2: Removed Health Endpoint Information - **INTENTIONAL** - -**Review Comment**: "This removes useful debugging information. When troubleshooting production issues, knowing the PID, active sessions count, and port would be valuable." - -**Our Documentation**: -- **Observation #3616**: "Simplified Health Check Endpoint to Minimal Response" -- **Observation #3601**: "Minimum Parameters = Minimum Bugs" -- **Observation #3600**: "Comprehensive Analysis of Cascading Architectural Problems" - -**Why We Did This**: -1. **HTTP 200 = Alive**: If the endpoint responds, the worker is healthy. Period. -2. **Diagnostic fields provided no actionable value**: PID, activeSessions, chromaSynced didn't help debug the actual production failures -3. **Part of 87% code reduction**: worker-utils.ts went from 113 lines → 15 lines -4. **Health checks were hiding real problems**: Retry logic masked that startup sequence was broken - -**Original Problem**: -- Worker startup: 4-5 seconds (actual) -- Health check timeout: 3 seconds (configured) -- Result: **100% user failure rate** - -The detailed health response didn't help diagnose this - fixing the startup sequence (HTTP server first) did. - -**Response**: **Will not change**. The health endpoint serves one purpose: availability signal. Use PM2 commands for diagnostics: -- `pm2 list` - See PID, status, memory -- `pm2 logs claude-mem-worker` - See application logs -- `npm run worker:logs` - Convenience wrapper - ---- - -### ⚠️ Issue #3: Auto-Session Creation Without Validation - **NEEDS FIXING** - -**Review Comment**: "Uses non-null assertion (dbSession!) without checking if dbSession is actually null. If getSessionById() returns null, this will throw at runtime." - -**Our Response**: **You're absolutely right**. This is a legitimate bug. - -**Action Required**: Add null checks to `handleObservation` and `handleSummarize` like already exist in `handleInit`: -```typescript -const dbSession = db.getSessionById(sessionDbId); -if (!dbSession) { - db.close(); - res.status(404).json({ error: 'Session not found in database' }); - return; -} -``` - -**This needs to be fixed before merge.** - ---- - -### ⚠️ Issue #4: Removed Observation Counter - **INTENTIONAL** - -**Review Comment**: "Was this used for generating correlation IDs for logging? If so, is there now no way to correlate observations within a session for debugging?" - -**Our Documentation**: -- **Observation #3621-3627**: Complete removal of observation counter and correlation IDs -- **Observation #3602**: "Architectural Decision: Remove Health Checks and Arbitrary Delays" -- **Observation #3612**: "Worker Service Simplification Strategy" - -**Why We Removed It**: -1. **Over-engineering**: Provided per-observation tracking when session-level identification was sufficient -2. **Part of cascading complexity**: Correlation IDs were monitoring infrastructure for complexity that shouldn't exist -3. **Session-level debugging is sufficient**: Most issues diagnosed by knowing which session, not which observation #5 within that session -4. **Database IDs provide uniqueness**: Once stored, observations have DB IDs for precise identification - -**The Problem It Was Solving (That No Longer Needs Solving)**: -- Tracking individual observations through worker pipeline -- Monitoring Chroma sync success/failure per observation -- Detailed per-observation timing metrics - -**Why That's Unnecessary**: -- Session-level logging is sufficient for debugging -- Database IDs provide uniqueness after storage -- The monitoring was masking real problems (startup sequence) - -**Response**: **Will not change**. This was part of the simplification strategy that fixed production failures. - ---- - -### ⚠️ Issue #5: PM2 Watch Mode in Production - **VALID CONCERN** - -**Review Comment**: "Watch mode causes PM2 to restart the process whenever files change. This is useful during development but potentially problematic in production." - -**Our Investigation**: -- **Observation #3631**: Documents what watch mode does, but **no observation documents WHY we enabled it** -- **Observation #3611**: PM2 config was "drastically simplified" by removing 21 unnecessary parameters -- **Watch mode was kept** during this aggressive simplification - -**Conclusion**: **This appears to be unintentional** - likely enabled for development and inadvertently left enabled. - -**Action Required**: Either: -1. **Disable watch mode** (recommended) - Users aren't developing, they're using the plugin -2. **Document it as intentional** if there's a reason we want auto-restart on file changes - -**This should be addressed before merge** - likely by disabling watch mode. - ---- - -### ⚠️ Issue #6: Duplicate Port Constant - **ACKNOWLEDGED** - -**Review Comment**: "FIXED_PORT constant is defined in 5 places. Creates maintenance burden." - -**Our Response**: **Fair point**. This is technical debt we can clean up. - -**However**, it's low priority because: -- Port is unlikely to change -- All values are currently consistent -- Not causing production issues - -**Action**: Add to backlog for post-merge cleanup. Export from worker-utils.ts and import elsewhere. - ---- - -## Summary of Actions - -### Must Fix Before Merge: -1. ✅ **Add null checks to auto-session creation** in handleObservation and handleSummarize -2. ✅ **Decide on watch mode** - Disable unless there's documented reason to keep it - -### Will Not Change (Intentional Decisions): -1. ❌ **Health endpoint simplification** - Part of solving 100% failure rate -2. ❌ **Removed observation counter** - Part of simplification strategy -3. ❌ **Removed health check system** - Was causing production failures -4. ❌ **Fire-and-forget worker spawn** - Hooks have proper error handling - -### Race Condition Discussion Needed: -1. 🤔 **Worker startup race condition** - Valid concern, but retry loops caused the original failures. Options: - - Keep current approach (hooks handle ECONNREFUSED gracefully) - - Add single synchronous `pm2 list` check after spawn - - Document as expected behavior - -### Nice to Have (Post-Merge): -1. 📋 **Consolidate FIXED_PORT constant** - Technical debt cleanup - ---- - -## Key Documentation References - -The architectural decisions are comprehensively documented in: - -1. **JUST-FUCKING-RUN-IT.md** (Observation #3602) - - Architectural decision to remove health checks - - Philosophy: Trust PM2, let HTTP timeouts be the health check - -2. **LINE-BY-LINE-CASCADING-BULLSHIT.md** (Observation #3600) - - Root cause analysis of how health checks caused 100% failure rate - - Documents cascade from arbitrary 3000ms timeout → retry loops → race conditions - -3. **MINIMUM-PARAMETERS.md** (Observation #3601) - - Quantified impact: 21 unnecessary PM2 parameters, ~160 lines deleted - - Philosophy: "Minimum parameters = minimum bugs" - -4. **STUPID-SHIT-THAT-BROKE-PRODUCTION.md** (Observation #3597) - - 8 critical issues causing 100% user failure rate - - Includes worker crashing on Chroma failures despite data already in SQLite - -These documents explain **why** the simplifications were necessary - they weren't arbitrary removal of useful features, they were targeted fixes for production failures. - ---- - -## Production Context - -**Before This PR**: -- 100% user failure rate after v4.x release -- Worker startup took 4-5 seconds but health checks timed out at 3 seconds -- `stdio: 'ignore'` eliminated all debugging visibility -- Worker crashed on Chroma failures despite data safely in SQLite -- ChromaSync initialized in constructor, blocking HTTP server -- 113 lines of health check code with retry loops masking real problems - -**After This PR**: -- HTTP server starts immediately -- Worker stays alive through Chroma failures (graceful degradation) -- Errors are visible (`stdio: 'inherit'`) -- Worker-utils.ts: 113 lines → 15 lines (87% reduction) -- Hooks have proper error handling with actionable user messages -- System works with just SQLite FTS5, Chroma is optional enhancement - -The "removed observability" was actually **removed complexity that was hiding problems**, not helping diagnose them. diff --git a/docs/context/agent-sdk-ref.md b/docs/context/agent-sdk-ref.md deleted file mode 100644 index 28925155..00000000 --- a/docs/context/agent-sdk-ref.md +++ /dev/null @@ -1,1797 +0,0 @@ -# Agent SDK reference - TypeScript - -> Complete API reference for the TypeScript Agent SDK, including all functions, types, and interfaces. - -