diff --git a/docs/context/platform-integration-guide.md b/docs/context/platform-integration-guide.md new file mode 100644 index 00000000..b9a94556 --- /dev/null +++ b/docs/context/platform-integration-guide.md @@ -0,0 +1,1354 @@ +# Platform Integration Guide - Claude-Mem Worker Service + +**Version:** 7.0.0 (December 2025) +**Target Audience:** Developers building claude-mem integrations (VSCode extensions, IDE plugins, CLI tools) +**Purpose:** Complete reference for integrating with the claude-mem worker service without requiring access to the knowledge base + +--- + +## Table of Contents + +1. [Quick Reference](#quick-reference) +2. [Worker Architecture](#worker-architecture) +3. [API Reference](#api-reference) +4. [Data Models](#data-models) +5. [Integration Patterns](#integration-patterns) +6. [Error Handling & Resilience](#error-handling--resilience) +7. [Development Workflow](#development-workflow) +8. [Testing Strategy](#testing-strategy) +9. [Code Examples](#code-examples) + +--- + +## Quick Reference + +### Worker Service Basics + +```typescript +const WORKER_BASE_URL = 'http://localhost:37777'; +const DEFAULT_PORT = 37777; // Override with CLAUDE_MEM_WORKER_PORT +``` + +### Most Common Operations + +```typescript +// Health check +GET /api/health + +// Create/get session and queue observation +POST /api/sessions/observations +Body: { claudeSessionId, tool_name, tool_input, tool_response, cwd } + +// Queue summary +POST /api/sessions/summarize +Body: { claudeSessionId, last_user_message, last_assistant_message } + +// Complete session +POST /api/sessions/complete +Body: { claudeSessionId } + +// Search observations +GET /api/search?query=authentication&type=observations&format=index&limit=20 + +// Get recent context for project +GET /api/context/recent?project=my-project&limit=3 +``` + +### Environment Variables + +```bash +CLAUDE_MEM_MODEL=claude-haiku-4-5 # Model for observations/summaries +CLAUDE_MEM_CONTEXT_OBSERVATIONS=50 # Observations injected at SessionStart +CLAUDE_MEM_WORKER_PORT=37777 # Worker service port +CLAUDE_MEM_PYTHON_VERSION=3.13 # Python version for chroma-mcp +``` + +### Build Commands (Local Development) + +```bash +npm run build # Compile TypeScript (hooks + worker) +npm run sync-marketplace # Copy to ~/.claude/plugins +npm run worker:restart # Restart PM2 worker +npm run worker:logs # View worker logs +pm2 list # Check worker status +``` + +--- + +## Worker Architecture + +### Request Flow + +``` +Platform Hook/Extension + → HTTP Request to Worker (localhost:37777) + → Route Handler (SessionRoutes/DataRoutes/SearchRoutes/etc.) + → Domain Service (SessionManager/SearchManager/DatabaseManager) + → Database (SQLite3 + Chroma vector DB) + → SSE Broadcast (real-time UI updates) +``` + +### Domain Services + +**DatabaseManager** - SQLite connection management, initialization +**SessionManager** - Event-driven session lifecycle, message queues +**SearchManager** - Search orchestration (FTS5 + Chroma) +**SSEBroadcaster** - Server-Sent Events for real-time updates +**SDKAgent** - Claude Agent SDK for generating observations/summaries +**PaginationHelper** - Query pagination utilities +**SettingsManager** - User settings CRUD +**FormattingService** - Result formatting (index vs full) +**TimelineService** - Unified timeline generation + +### Route Organization + +**ViewerRoutes** - Health check, viewer UI, SSE stream +**SessionRoutes** - Session lifecycle (init, observations, summarize, complete) +**DataRoutes** - Data retrieval (observations, summaries, prompts, stats) +**SearchRoutes** - All search operations (unified search, timeline, semantic shortcuts) +**SettingsRoutes** - User settings, MCP toggle, branch switching + +--- + +## API Reference + +### Session Lifecycle (SessionRoutes) + +#### Create/Get Session + Queue Observation (New API) +```http +POST /api/sessions/observations +Content-Type: application/json + +{ + "claudeSessionId": "abc123", // Claude session identifier (string) + "tool_name": "Bash", + "tool_input": { "command": "ls" }, + "tool_response": { "stdout": "..." }, + "cwd": "/path/to/project" +} + +Response: { "status": "queued" } | { "status": "skipped", "reason": "private" } +``` + +**Privacy Check:** Skips if user prompt was entirely wrapped in `` tags. +**Tag Stripping:** Removes `` and `` tags before storage. +**Auto-Start:** Ensures SDK agent generator is running to process the queue. + +#### Queue Summary (New API) +```http +POST /api/sessions/summarize +Content-Type: application/json + +{ + "claudeSessionId": "abc123", + "last_user_message": "User's message", + "last_assistant_message": "Assistant's response" +} + +Response: { "status": "queued" } | { "status": "skipped", "reason": "private" } +``` + +#### Complete Session (New API) +```http +POST /api/sessions/complete +Content-Type: application/json + +{ + "claudeSessionId": "abc123" +} + +Response: { "success": true } | { "success": true, "message": "No active session found" } +``` + +**Effect:** Stops SDK agent, marks session complete, broadcasts status change. + +#### Legacy Endpoints (Still Supported) + +```http +# Initialize session (legacy, uses sessionDbId) +POST /sessions/:sessionDbId/init +Body: { userPrompt, promptNumber } + +# Queue observations (legacy) +POST /sessions/:sessionDbId/observations +Body: { tool_name, tool_input, tool_response, prompt_number, cwd } + +# Queue summary (legacy) +POST /sessions/:sessionDbId/summarize +Body: { last_user_message, last_assistant_message } + +# Complete session (legacy) +POST /sessions/:sessionDbId/complete +``` + +**Note:** New integrations should use `/api/sessions/*` endpoints with `claudeSessionId`. + +--- + +### Data Retrieval (DataRoutes) + +#### Get Paginated Observations +```http +GET /api/observations?offset=0&limit=20&project=my-project + +Response: { + "items": [...], + "hasMore": boolean, + "offset": number, + "limit": number +} +``` + +#### Get Paginated Summaries +```http +GET /api/summaries?offset=0&limit=20&project=my-project +``` + +#### Get Paginated User Prompts +```http +GET /api/prompts?offset=0&limit=20&project=my-project +``` + +#### Get by ID +```http +GET /api/observation/:id +GET /api/session/:id +GET /api/prompt/:id + +Response: {...entity...} | 404 Not Found +``` + +#### Get Database Stats +```http +GET /api/stats + +Response: { + "worker": { + "version": "7.0.0", + "uptime": 12345, + "activeSessions": 2, + "sseClients": 1, + "port": 37777 + }, + "database": { + "path": "~/.claude-mem/claude-mem.db", + "size": 1048576, + "observations": 500, + "sessions": 50, + "summaries": 25 + } +} +``` + +#### Get Projects List +```http +GET /api/projects + +Response: { "projects": ["claude-mem", "other-project", ...] } +``` + +#### Get Processing Status +```http +GET /api/processing-status + +Response: { "isProcessing": boolean, "queueDepth": number } +``` + +--- + +### Search Operations (SearchRoutes) + +#### Unified Search +```http +GET /api/search?query=authentication&type=observations&format=index&limit=20 + +Parameters: +- query: Search query text (optional, omit for filter-only) +- type: "observations" | "sessions" | "prompts" (default: all) +- format: "index" | "full" (default: "index") +- limit: Number of results (default: 20) +- project: Filter by project name +- obs_type: Filter by observation type (discovery, decision, bugfix, feature, refactor) +- concepts: Filter by concepts (comma-separated) +- files: Filter by file paths (comma-separated) +- dateStart: ISO timestamp (filter start) +- dateEnd: ISO timestamp (filter end) + +Response: { + "observations": [...], + "sessions": [...], + "prompts": [...] +} +``` + +**Format Options:** +- `index`: Minimal fields for list display (id, title, preview) +- `full`: Complete entity with all fields + +#### Unified Timeline +```http +GET /api/timeline?anchor=123&depth_before=10&depth_after=10&project=my-project + +Parameters: +- anchor: Anchor point (observation ID, "S123" for session, or ISO timestamp) +- depth_before: Records before anchor (default: 10) +- depth_after: Records after anchor (default: 10) +- project: Filter by project + +Response: [ + { "type": "observation", "id": 120, "created_at_epoch": ..., ... }, + { "type": "session", "id": 5, "created_at_epoch": ..., ... }, + { "type": "observation", "id": 123, "created_at_epoch": ..., ... }, + ... +] +``` + +#### Semantic Shortcuts +```http +# Find decision observations +GET /api/decisions?format=index&limit=20 + +# Find change-related observations +GET /api/changes?format=index&limit=20 + +# Find "how it works" explanations +GET /api/how-it-works?format=index&limit=20 +``` + +#### Search by Concept +```http +GET /api/search/by-concept?concept=discovery&format=index&limit=10&project=my-project +``` + +#### Search by File Path +```http +GET /api/search/by-file?filePath=src/services/worker-service.ts&format=index&limit=10 +``` + +#### Search by Type +```http +GET /api/search/by-type?type=bugfix&format=index&limit=10 +``` + +#### Get Recent Context +```http +GET /api/context/recent?project=my-project&limit=3 + +Response: { + "summaries": [...], + "observations": [...] +} +``` + +#### Context Preview (for Settings UI) +```http +GET /api/context/preview?project=my-project + +Response: Plain text with ANSI colors (for terminal display) +``` + +#### Context Injection (for Hooks) +```http +GET /api/context/inject?project=my-project&colors=true + +Response: Pre-formatted context string ready for display +``` + +--- + +### Settings & Configuration (SettingsRoutes) + +#### Get/Update User Settings +```http +GET /api/settings +Response: { "sidebarOpen": boolean, "selectedProject": string | null } + +POST /api/settings +Body: { "sidebarOpen": true, "selectedProject": "my-project" } +Response: { "success": true } +``` + +#### MCP Server Status/Toggle +```http +GET /api/mcp/status +Response: { "enabled": boolean } + +POST /api/mcp/toggle +Body: { "enabled": true } +Response: { "success": true, "enabled": boolean } +``` + +#### Git Branch Operations +```http +GET /api/branch/status +Response: { "current": "main", "remote": "origin/main", "ahead": 0, "behind": 0 } + +POST /api/branch/switch +Body: { "branch": "feature/new-feature" } +Response: { "success": true } + +POST /api/branch/update +Response: { "success": true, "updated": boolean } +``` + +--- + +### Viewer & Real-Time Updates (ViewerRoutes) + +#### Health Check +```http +GET /api/health + +Response: { "status": "ok" } +``` + +#### Viewer UI +```http +GET / + +Response: HTML (React app) +``` + +#### SSE Stream +```http +GET /stream + +Response: Server-Sent Events stream + +Event Types: +- processing_status: { type, isProcessing, queueDepth } +- session_started: { type, sessionDbId, project } +- observation_queued: { type, sessionDbId } +- summarize_queued: { type } +- observation_created: { type, observation } +- summary_created: { type, summary } +- new_prompt: { type, id, claude_session_id, project, prompt_number, prompt_text, created_at_epoch } +``` + +--- + +## Data Models + +### Active Session (In-Memory) + +```typescript +interface ActiveSession { + sessionDbId: number; // Database ID (numeric) + claudeSessionId: string; // Claude session identifier (string) + sdkSessionId: string | null; // SDK session ID + project: string; // Project name + userPrompt: string; // Current user prompt text + pendingMessages: PendingMessage[]; // Queue of pending operations + abortController: AbortController; // For cancellation + generatorPromise: Promise | null; // SDK agent promise + lastPromptNumber: number; // Last processed prompt number + startTime: number; // Session start timestamp + cumulativeInputTokens: number; // Total input tokens + cumulativeOutputTokens: number; // Total output tokens +} + +interface PendingMessage { + type: 'observation' | 'summarize'; + tool_name?: string; + tool_input?: any; + tool_response?: any; + prompt_number?: number; + cwd?: string; + last_user_message?: string; + last_assistant_message?: string; +} +``` + +### Database Entities + +```typescript +// SDK Session (stored in sdk_sessions table) +interface SDKSessionRow { + id: number; + claude_session_id: string; + sdk_session_id: string; + project: string; + user_prompt: string; + created_at_epoch: number; + completed_at_epoch?: number; +} + +// Observation (stored in observations table) +interface ObservationRow { + id: number; + sdk_session_id: string; + title: string; + subtitle?: string; + summary: string; + facts: string; // JSON array of fact strings + concepts: string; // JSON array of concept strings + files_touched: string; // JSON array of file paths + obs_type: string; // discovery, decision, bugfix, feature, refactor + project: string; + created_at_epoch: number; + prompt_number: number; +} + +// Session Summary (stored in session_summaries table) +interface SessionSummaryRow { + id: number; + sdk_session_id: string; + summary_text: string; + facts: string; // JSON array + concepts: string; // JSON array + files_touched: string; // JSON array + project: string; + created_at_epoch: number; +} + +// User Prompt (stored in user_prompts table) +interface UserPromptRow { + id: number; + claude_session_id: string; + sdk_session_id: string; + project: string; + prompt_number: number; + prompt_text: string; + created_at_epoch: number; +} +``` + +### Search Results + +```typescript +interface ObservationSearchResult { + id: number; + title: string; + subtitle?: string; + summary: string; + facts: string[]; // Parsed from JSON + concepts: string[]; // Parsed from JSON + files_touched: string[]; // Parsed from JSON + obs_type: string; + project: string; + created_at_epoch: number; + prompt_number: number; + rank?: number; // FTS5 rank score +} + +interface SessionSummarySearchResult { + id: number; + summary_text: string; + facts: string[]; + concepts: string[]; + files_touched: string[]; + project: string; + created_at_epoch: number; + rank?: number; +} + +interface UserPromptSearchResult { + id: number; + claude_session_id: string; + project: string; + prompt_number: number; + prompt_text: string; + created_at_epoch: number; + rank?: number; +} +``` + +### Timeline Item + +```typescript +interface TimelineItem { + type: 'observation' | 'session' | 'prompt'; + id: number; + created_at_epoch: number; + // Entity-specific fields based on type +} +``` + +--- + +## Integration Patterns + +### Mapping Claude Code Hooks to Worker API + +#### SessionStart Hook +```typescript +// Not needed for new API - sessions are auto-created on first observation +``` + +#### UserPromptSubmit Hook +```typescript +// No API call needed - user_prompt is captured by first observation in the prompt +``` + +#### PostToolUse Hook +```typescript +async function onPostToolUse(context: HookContext) { + const { session_id, tool_name, tool_input, tool_result, cwd } = context; + + const response = await fetch('http://localhost:37777/api/sessions/observations', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + claudeSessionId: session_id, + tool_name, + tool_input, + tool_response: tool_result, + cwd + }) + }); + + const result = await response.json(); + // result.status === 'queued' | 'skipped' +} +``` + +#### Summary Hook +```typescript +async function onSummary(context: HookContext) { + const { session_id, last_user_message, last_assistant_message } = context; + + await fetch('http://localhost:37777/api/sessions/summarize', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + claudeSessionId: session_id, + last_user_message, + last_assistant_message + }) + }); +} +``` + +#### SessionEnd Hook +```typescript +async function onSessionEnd(context: HookContext) { + const { session_id } = context; + + await fetch('http://localhost:37777/api/sessions/complete', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + claudeSessionId: session_id + }) + }); +} +``` + +### VSCode Extension Integration + +#### Language Model Tool Registration + +```typescript +import * as vscode from 'vscode'; + +interface SearchTool extends vscode.LanguageModelChatTool { + invoke( + options: vscode.LanguageModelToolInvocationOptions<{ query: string }>, + token: vscode.CancellationToken + ): vscode.ProviderResult; +} + +const searchTool: SearchTool = { + invoke: async (options, token) => { + const { query } = options.input; + + try { + const response = await fetch( + `http://localhost:37777/api/search?query=${encodeURIComponent(query)}&format=index&limit=10` + ); + + if (!response.ok) { + throw new Error(`Search failed: ${response.statusText}`); + } + + const results = await response.json(); + + // Format results for language model + return new vscode.LanguageModelToolResult([ + new vscode.LanguageModelTextPart(JSON.stringify(results, null, 2)) + ]); + } catch (error) { + return new vscode.LanguageModelToolResult([ + new vscode.LanguageModelTextPart(`Error: ${error.message}`) + ]); + } + } +}; + +// Register tool +vscode.lm.registerTool('claude-mem-search', searchTool); +``` + +#### Chat Participant Implementation + +```typescript +const participant = vscode.chat.createChatParticipant('claude-mem', async (request, context, stream, token) => { + const claudeSessionId = context.session.id; + + // First message in conversation - no initialization needed + // Session is auto-created on first observation + + // Process user message + stream.markdown(`Searching memory for: ${request.prompt}\n\n`); + + const response = await fetch( + `http://localhost:37777/api/search?query=${encodeURIComponent(request.prompt)}&format=index&limit=5` + ); + + const results = await response.json(); + + if (results.observations?.length > 0) { + stream.markdown('**Found observations:**\n'); + for (const obs of results.observations) { + stream.markdown(`- ${obs.title} (${obs.project})\n`); + } + } + + return { metadata: { command: 'search' } }; +}); +``` + +--- + +## Error Handling & Resilience + +### Connection Failures + +```typescript +async function callWorkerWithFallback( + endpoint: string, + options?: RequestInit +): Promise { + try { + const response = await fetch(`http://localhost:37777${endpoint}`, { + ...options, + signal: AbortSignal.timeout(5000) // 5s timeout + }); + + if (!response.ok) { + throw new Error(`HTTP ${response.status}: ${response.statusText}`); + } + + return await response.json(); + } catch (error) { + console.error(`Worker unavailable (${endpoint}):`, error); + return null; // Graceful degradation + } +} +``` + +### Retry Logic with Exponential Backoff + +```typescript +async function retryWithBackoff( + fn: () => Promise, + maxRetries = 3, + baseDelay = 100 +): Promise { + for (let attempt = 0; attempt < maxRetries; attempt++) { + try { + return await fn(); + } catch (error) { + if (attempt === maxRetries - 1) throw error; + + const delay = baseDelay * Math.pow(2, attempt); + await new Promise(resolve => setTimeout(resolve, delay)); + } + } + throw new Error('Max retries exceeded'); +} +``` + +### Worker Health Check + +```typescript +async function isWorkerHealthy(): Promise { + try { + const response = await fetch('http://localhost:37777/api/health', { + signal: AbortSignal.timeout(2000) + }); + return response.ok; + } catch { + return false; + } +} +``` + +### Privacy Tag Handling + +The worker automatically strips privacy tags before storage: +- `content` - User-level privacy control +- `content` - System-level tag (prevents recursive storage) + +**Privacy Check:** Observations/summaries are skipped if the entire user prompt was wrapped in `` tags. + +### Custom Error Classes + +```typescript +class WorkerUnavailableError extends Error { + constructor() { + super('Claude-mem worker is not running or unreachable'); + this.name = 'WorkerUnavailableError'; + } +} + +class WorkerTimeoutError extends Error { + constructor(endpoint: string) { + super(`Worker request timed out: ${endpoint}`); + this.name = 'WorkerTimeoutError'; + } +} +``` + +### SSE Stream Error Handling + +```typescript +function connectToSSE(onEvent: (event: any) => void) { + const eventSource = new EventSource('http://localhost:37777/stream'); + + eventSource.onmessage = (event) => { + try { + const data = JSON.parse(event.data); + onEvent(data); + } catch (error) { + console.error('SSE parse error:', error); + } + }; + + eventSource.onerror = (error) => { + console.error('SSE connection error:', error); + eventSource.close(); + + // Reconnect after 5 seconds + setTimeout(() => connectToSSE(onEvent), 5000); + }; + + return eventSource; +} +``` + +--- + +## Development Workflow + +### Project Structure (Recommended) + +``` +vscode-extension/ +├── src/ +│ ├── extension.ts # Extension entry point +│ ├── services/ +│ │ ├── WorkerClient.ts # HTTP client for worker +│ │ └── MemoryManager.ts # High-level memory operations +│ ├── chat/ +│ │ └── participant.ts # Chat participant implementation +│ └── tools/ +│ ├── search.ts # Search language model tool +│ └── context.ts # Context injection tool +├── package.json +├── tsconfig.json +└── README.md +``` + +### Build Configuration (esbuild) + +```javascript +// build.js +const esbuild = require('esbuild'); + +esbuild.build({ + entryPoints: ['src/extension.ts'], + bundle: true, + outfile: 'dist/extension.js', + external: ['vscode'], + format: 'cjs', + platform: 'node', + target: 'node18', + sourcemap: true +}).catch(() => process.exit(1)); +``` + +### package.json (VSCode Extension) + +```json +{ + "name": "claude-mem-vscode", + "displayName": "Claude-Mem", + "version": "1.0.0", + "engines": { + "vscode": "^1.95.0" + }, + "activationEvents": [ + "onStartupFinished" + ], + "main": "./dist/extension.js", + "contributes": { + "chatParticipants": [ + { + "id": "claude-mem", + "name": "memory", + "description": "Search your persistent memory" + } + ], + "languageModelTools": [ + { + "name": "claude-mem-search", + "displayName": "Search Memory", + "description": "Search persistent memory for observations, sessions, and prompts" + } + ] + }, + "scripts": { + "build": "node build.js", + "watch": "node build.js --watch", + "package": "vsce package" + }, + "devDependencies": { + "@types/vscode": "^1.95.0", + "esbuild": "^0.19.0", + "typescript": "^5.3.0" + } +} +``` + +### Local Testing Loop + +```bash +# Terminal 1: Watch build +npm run watch + +# Terminal 2: Check worker status +pm2 list +pm2 logs claude-mem-worker + +# Terminal 3: Test API manually +curl http://localhost:37777/api/health +curl "http://localhost:37777/api/search?query=test&limit=5" + +# VSCode: Press F5 to launch extension host +``` + +### Debug Configuration (.vscode/launch.json) + +```json +{ + "version": "0.2.0", + "configurations": [ + { + "name": "Run Extension", + "type": "extensionHost", + "request": "launch", + "args": [ + "--extensionDevelopmentPath=${workspaceFolder}" + ], + "outFiles": [ + "${workspaceFolder}/dist/**/*.js" + ], + "preLaunchTask": "npm: build" + } + ] +} +``` + +--- + +## Testing Strategy + +### Unit Tests (Worker Client) + +```typescript +import { describe, it, expect } from 'vitest'; +import { WorkerClient } from '../src/services/WorkerClient'; + +describe('WorkerClient', () => { + it('should check worker health', async () => { + const client = new WorkerClient(); + const healthy = await client.isHealthy(); + expect(healthy).toBe(true); + }); + + it('should queue observation', async () => { + const client = new WorkerClient(); + const result = await client.queueObservation({ + claudeSessionId: 'test-123', + tool_name: 'Bash', + tool_input: { command: 'ls' }, + tool_response: { stdout: 'file1.txt' }, + cwd: '/tmp' + }); + expect(result.status).toBe('queued'); + }); + + it('should search observations', async () => { + const client = new WorkerClient(); + const results = await client.search({ query: 'test', limit: 5 }); + expect(results).toHaveProperty('observations'); + }); +}); +``` + +### Integration Tests (With Worker Spawning) + +```typescript +import { spawn } from 'child_process'; +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; + +describe('Worker Integration', () => { + let workerProcess: ReturnType; + + beforeAll(async () => { + // Start worker process + workerProcess = spawn('node', ['dist/worker-service.js'], { + env: { ...process.env, CLAUDE_MEM_WORKER_PORT: '37778' } + }); + + // Wait for worker to be ready + await new Promise(resolve => setTimeout(resolve, 2000)); + }); + + afterAll(() => { + workerProcess.kill(); + }); + + it('should respond to health check', async () => { + const response = await fetch('http://localhost:37778/api/health'); + expect(response.ok).toBe(true); + }); +}); +``` + +### Manual Testing Checklist + +**Phase 1: Connection & Health** +- [ ] Worker starts successfully (`pm2 list`) +- [ ] Health endpoint responds (`curl http://localhost:37777/api/health`) +- [ ] SSE stream connects (`curl http://localhost:37777/stream`) + +**Phase 2: Session Lifecycle** +- [ ] Queue observation creates session +- [ ] Observation appears in database +- [ ] Privacy tags are stripped +- [ ] Private prompts are skipped +- [ ] Queue summary creates summary +- [ ] Complete session stops processing + +**Phase 3: Search & Retrieval** +- [ ] Search observations by query +- [ ] Search sessions by query +- [ ] Search prompts by query +- [ ] Get recent context for project +- [ ] Get timeline around observation +- [ ] Semantic shortcuts (decisions, changes, how-it-works) + +**Phase 4: Real-Time Updates** +- [ ] SSE broadcasts processing status +- [ ] SSE broadcasts new observations +- [ ] SSE broadcasts new summaries +- [ ] SSE broadcasts new prompts + +**Phase 5: Error Handling** +- [ ] Graceful degradation when worker unavailable +- [ ] Timeout handling for slow requests +- [ ] Retry logic for transient failures + +--- + +## Code Examples + +### Complete WorkerClient Implementation + +```typescript +export class WorkerClient { + private baseUrl: string; + + constructor(port: number = 37777) { + this.baseUrl = `http://localhost:${port}`; + } + + async isHealthy(): Promise { + try { + const response = await fetch(`${this.baseUrl}/api/health`, { + signal: AbortSignal.timeout(2000) + }); + return response.ok; + } catch { + return false; + } + } + + async queueObservation(data: { + claudeSessionId: string; + tool_name: string; + tool_input: any; + tool_response: any; + cwd?: string; + }): Promise<{ status: string; reason?: string }> { + const response = await fetch(`${this.baseUrl}/api/sessions/observations`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(data), + signal: AbortSignal.timeout(5000) + }); + + if (!response.ok) { + throw new Error(`Failed to queue observation: ${response.statusText}`); + } + + return await response.json(); + } + + async queueSummarize(data: { + claudeSessionId: string; + last_user_message?: string; + last_assistant_message?: string; + }): Promise<{ status: string; reason?: string }> { + const response = await fetch(`${this.baseUrl}/api/sessions/summarize`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(data), + signal: AbortSignal.timeout(5000) + }); + + if (!response.ok) { + throw new Error(`Failed to queue summary: ${response.statusText}`); + } + + return await response.json(); + } + + async completeSession(claudeSessionId: string): Promise { + const response = await fetch(`${this.baseUrl}/api/sessions/complete`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ claudeSessionId }), + signal: AbortSignal.timeout(5000) + }); + + if (!response.ok) { + throw new Error(`Failed to complete session: ${response.statusText}`); + } + } + + async search(params: { + query?: string; + type?: 'observations' | 'sessions' | 'prompts'; + format?: 'index' | 'full'; + limit?: number; + project?: string; + obs_type?: string | string[]; + concepts?: string | string[]; + files?: string | string[]; + dateStart?: string; + dateEnd?: string; + }): Promise { + const queryString = new URLSearchParams( + Object.entries(params) + .filter(([_, v]) => v !== undefined) + .map(([k, v]) => [k, Array.isArray(v) ? v.join(',') : String(v)]) + ).toString(); + + const response = await fetch( + `${this.baseUrl}/api/search?${queryString}`, + { signal: AbortSignal.timeout(10000) } + ); + + if (!response.ok) { + throw new Error(`Search failed: ${response.statusText}`); + } + + return await response.json(); + } + + async getRecentContext(project: string, limit: number = 3): Promise { + const response = await fetch( + `${this.baseUrl}/api/context/recent?project=${encodeURIComponent(project)}&limit=${limit}`, + { signal: AbortSignal.timeout(10000) } + ); + + if (!response.ok) { + throw new Error(`Get recent context failed: ${response.statusText}`); + } + + return await response.json(); + } + + async getTimeline(params: { + anchor: number | string; + depth_before?: number; + depth_after?: number; + project?: string; + }): Promise { + const queryString = new URLSearchParams( + Object.entries(params) + .filter(([_, v]) => v !== undefined) + .map(([k, v]) => [k, String(v)]) + ).toString(); + + const response = await fetch( + `${this.baseUrl}/api/timeline?${queryString}`, + { signal: AbortSignal.timeout(10000) } + ); + + if (!response.ok) { + throw new Error(`Get timeline failed: ${response.statusText}`); + } + + return await response.json(); + } + + connectSSE(onEvent: (event: any) => void): EventSource { + const eventSource = new EventSource(`${this.baseUrl}/stream`); + + eventSource.onmessage = (event) => { + try { + const data = JSON.parse(event.data); + onEvent(data); + } catch (error) { + console.error('SSE parse error:', error); + } + }; + + eventSource.onerror = (error) => { + console.error('SSE connection error:', error); + }; + + return eventSource; + } +} +``` + +### Search Language Model Tool + +```typescript +import * as vscode from 'vscode'; +import { WorkerClient } from './WorkerClient'; + +export function registerSearchTool(context: vscode.ExtensionContext) { + const client = new WorkerClient(); + + const searchTool = vscode.lm.registerTool('claude-mem-search', { + description: 'Search persistent memory for observations, sessions, and prompts', + inputSchema: { + type: 'object', + properties: { + query: { + type: 'string', + description: 'Search query text' + }, + type: { + type: 'string', + enum: ['observations', 'sessions', 'prompts'], + description: 'Type of results to return' + }, + limit: { + type: 'number', + description: 'Maximum number of results', + default: 10 + } + }, + required: ['query'] + }, + invoke: async (options, token) => { + const { query, type, limit = 10 } = options.input; + + try { + const results = await client.search({ + query, + type, + format: 'index', + limit + }); + + // Format results for language model + let formatted = ''; + + if (results.observations?.length > 0) { + formatted += '## Observations\n\n'; + for (const obs of results.observations) { + formatted += `- **${obs.title}** (${obs.project})\n`; + formatted += ` ${obs.summary}\n`; + if (obs.concepts?.length > 0) { + formatted += ` Concepts: ${obs.concepts.join(', ')}\n`; + } + formatted += '\n'; + } + } + + if (results.sessions?.length > 0) { + formatted += '## Sessions\n\n'; + for (const session of results.sessions) { + formatted += `- ${session.summary_text.substring(0, 100)}...\n\n`; + } + } + + return new vscode.LanguageModelToolResult([ + new vscode.LanguageModelTextPart(formatted) + ]); + } catch (error) { + return new vscode.LanguageModelToolResult([ + new vscode.LanguageModelTextPart(`Error: ${error.message}`) + ]); + } + } + }); + + context.subscriptions.push(searchTool); +} +``` + +--- + +## Critical Implementation Notes + +### sessionDbId vs claudeSessionId + +**IMPORTANT:** Use `claudeSessionId` (string) for new API endpoints, not `sessionDbId` (number). + +- `sessionDbId` - Numeric database ID (legacy endpoints only) +- `claudeSessionId` - String identifier from Claude platform (new endpoints) + +### JSON String Fields + +Fields like `facts`, `concepts`, and `files_touched` are stored as JSON strings and require parsing: + +```typescript +const observation = await client.getObservationById(123); +const facts = JSON.parse(observation.facts); // string[] array +const concepts = JSON.parse(observation.concepts); // string[] array +``` + +### Timestamps + +All `created_at_epoch` fields are in **milliseconds**, not seconds: + +```typescript +const date = new Date(observation.created_at_epoch); // ✅ Correct +const date = new Date(observation.created_at_epoch * 1000); // ❌ Wrong (already in ms) +``` + +### Asynchronous Processing + +Workers process observations/summaries asynchronously. Results appear in the database 1-2 seconds after queuing. Use SSE events for real-time notifications. + +### Privacy Tags + +Always wrap sensitive content in `` tags to prevent storage: + +```typescript +const userMessage = 'API key: sk-1234567890'; +// This observation will be skipped (entire prompt is private) +``` + +--- + +## Additional Resources + +- **Claude-Mem Documentation:** https://claude-mem.ai +- **GitHub Repository:** https://github.com/thedotmack/claude-mem +- **Worker Service README:** `src/services/worker/README.md` +- **API Endpoints:** `src/services/worker/http/routes/*.ts` +- **Domain Services:** `src/services/worker/*.ts` + +--- + +**End of Platform Integration Guide** diff --git a/docs/public/architecture/hooks.mdx b/docs/public/architecture/hooks.mdx index b8f389ab..65695ca9 100644 --- a/docs/public/architecture/hooks.mdx +++ b/docs/public/architecture/hooks.mdx @@ -9,29 +9,170 @@ Claude-Mem implements a **5-stage hook system** that captures development work a ## Architecture Overview +### System Architecture + +This two-process architecture works in both Claude Code and VS Code: + ```mermaid -flowchart TB - subgraph IDE["Claude Code IDE"] - SS[SessionStart] --> UPS[UserPromptSubmit] --> PTU[PostToolUse] --> ST[Stop] --> SE[SessionEnd] - SS --> ctx["context"] - UPS --> new["new"] - PTU --> save["save"] - ST --> sum["summary"] - SE --> clean["cleanup"] +graph TB + subgraph EXT["Extension Process (runs in IDE)"] + direction TB + ACT[Extension Activation] + HOOKS[Hook Event Handlers] + ACT --> HOOKS + + subgraph HOOK_HANDLERS["5 Lifecycle Hooks"] + H1[SessionStart
activate function] + H2[UserPromptSubmit
command handler] + H3[PostToolUse
middleware] + H4[Stop
idle timeout] + H5[SessionEnd
deactivate function] + end + + HOOKS --> HOOK_HANDLERS end - ctx & new & save & sum & clean --> HTTP["HTTP (fire-and-forget)"] + HOOK_HANDLERS -->|"HTTP
(fire-and-forget
2s timeout)"| HTTP[Worker HTTP API
Port 37777] - subgraph Worker["Worker Service (PM2)"] - SM[SessionMgr] ~~~ SA[SDK Agent] ~~~ DM[DatabaseMgr] - SA --> SDK["Claude Agent SDK"] + subgraph WORKER["Worker Process (separate Node.js)"] + direction TB + HTTP --> API[Express Server] + API --> SESS[Session Manager] + API --> AGENT[SDK Agent] + API --> DB[Database Manager] + + AGENT -->|Event-Driven| CLAUDE[Claude Agent SDK] + CLAUDE --> SQLITE[(SQLite + FTS5)] + CLAUDE --> CHROMA[(Chroma Vectors)] end - HTTP --> Worker - SDK --> SQLite[(SQLite DB)] - SDK --> Chroma[(Chroma Vector DB)] + style EXT fill:#e1f5ff + style WORKER fill:#fff4e1 + style HOOK_HANDLERS fill:#f0f0f0 ``` +**Key Principles:** +- Extension process never blocks (fire-and-forget HTTP) +- Worker processes observations asynchronously +- Session state persists across IDE restarts + +### VS Code Extension API Integration Points + +For developers porting to VS Code, here's where to hook into the VS Code Extension API: + +```mermaid +graph LR + subgraph VSCODE["VS Code Extension API"] + direction TB + A["activate(context)"] + B["commands.registerCommand()"] + C["chat.createChatParticipant()"] + D["workspace.onDidSaveTextDocument()"] + E["window.onDidChangeActiveTextEditor()"] + F["deactivate()"] + end + + subgraph HOOKS["Hook Equivalents"] + direction TB + G[SessionStart] + H[UserPromptSubmit] + I[PostToolUse] + J[Stop/Summary] + K[SessionEnd] + end + + subgraph WORKER_API["Worker HTTP Endpoints"] + direction TB + L[GET /api/context/inject] + M[POST /sessions/init] + N[POST /sessions/observations] + O[POST /sessions/summarize] + P[POST /sessions/complete] + end + + A --> G + B --> H + C --> H + D --> I + E --> I + F --> K + + G --> L + H --> M + I --> N + J --> O + K --> P + + style VSCODE fill:#007acc,color:#fff + style HOOKS fill:#f0f0f0 + style WORKER_API fill:#4caf50,color:#fff +``` + +**Implementation Examples:** + +```typescript +// VS Code Extension - SessionStart Hook +export async function activate(context: vscode.ExtensionContext) { + const sessionId = generateSessionId() + const project = workspace.name || 'default' + + // Fetch context from worker + const response = await fetch(`http://localhost:37777/api/context/inject?project=${project}`) + const context = await response.text() + + // Inject into chat or UI panel + injectContextToChat(context) +} + +// VS Code Extension - UserPromptSubmit Hook +const command = vscode.commands.registerCommand('extension.command', async (prompt) => { + await fetch('http://localhost:37777/sessions/init', { + method: 'POST', + body: JSON.stringify({ sessionId, project, userPrompt: prompt }) + }) +}) + +// VS Code Extension - PostToolUse Hook (middleware pattern) +workspace.onDidSaveTextDocument(async (document) => { + await fetch('http://localhost:37777/api/sessions/observations', { + method: 'POST', + body: JSON.stringify({ + claudeSessionId: sessionId, + tool_name: 'FileSave', + tool_input: { path: document.uri.path }, + tool_response: 'File saved successfully' + }) + }) +}) +``` + +### Async Processing Pipeline + +How observations flow from extension to database without blocking the IDE: + +```mermaid +graph TB + A["Extension: Tool Use Event"] --> B{"Skip List?
(TodoWrite, AskUserQuestion, etc.)"} + B -->|"Skip"| X["Discard"] + B -->|"Keep"| C["Strip Privacy Tags
<private>...</private>"] + C --> D["HTTP POST to Worker
Port 37777"] + D --> E["2s timeout
fire-and-forget"] + E --> F["Extension continues
(non-blocking)"] + + D -.Async Path.-> G["Worker: Queue Observation"] + G --> H["SDK Agent picks up
(event-driven)"] + H --> I["Call Claude API
(compress observation)"] + I --> J["Parse XML response"] + J --> K["Save to SQLite
(sdk_sessions table)"] + K --> L["Sync to Chroma
(vector embeddings)"] + + style F fill:#90EE90,stroke:#2d6b2d,stroke-width:3px + style L fill:#87CEEB,stroke:#2d5f8d,stroke-width:3px + style E fill:#ffeb3b,stroke:#c6a700,stroke-width:2px +``` + +**Critical Pattern:** The extension's HTTP call has a 2-second timeout and doesn't wait for AI processing. The worker handles compression asynchronously using an event-driven queue. + ## The 5 Lifecycle Stages | Stage | Hook | Trigger | Purpose | @@ -104,6 +245,37 @@ Hooks are configured in `plugin/hooks/hooks.json`: 1. `context-hook.js` - Fetches and injects prior session context 2. `user-message-hook.js` - Displays context info to user via stderr +### Sequence Diagram + +```mermaid +sequenceDiagram + participant User + participant IDE as IDE/Extension + participant ContextHook as context-hook.js + participant Worker as Worker Service + participant DB as SQLite Database + + User->>IDE: Opens workspace / resumes session + IDE->>ContextHook: Trigger SessionStart hook + ContextHook->>ContextHook: Generate/reuse session_id + ContextHook->>Worker: Health check (max 10s retry) + + alt Worker Ready + ContextHook->>Worker: GET /api/context/inject?project=X + Worker->>DB: SELECT * FROM observations
WHERE project=X
ORDER BY created_at DESC
LIMIT 50 + DB-->>Worker: Last 50 observations + Worker-->>ContextHook: Context markdown + ContextHook-->>IDE: hookSpecificOutput.additionalContext + IDE->>IDE: Inject context to Claude's prompt + IDE-->>User: Session ready with context + else Worker Not Ready + ContextHook-->>IDE: Empty context (graceful degradation) + IDE-->>User: Session ready without context + end + + Note over User,DB: Total time: <300ms (with health check) +``` + ### Context Hook (`context-hook.js`) **Purpose**: Inject context from previous sessions into Claude's initial context. @@ -155,6 +327,45 @@ Hooks are configured in `plugin/hooks/hooks.json`: **Hook**: `new-hook.js` +### Sequence Diagram + +```mermaid +sequenceDiagram + participant User + participant IDE as IDE/Extension + participant NewHook as new-hook.js + participant DB as Direct SQLite Access + participant Worker as Worker Service + + User->>IDE: Submits prompt: "Add login feature" + IDE->>NewHook: Trigger UserPromptSubmit
{ session_id, cwd, prompt } + + NewHook->>NewHook: Extract project = basename(cwd) + NewHook->>NewHook: Strip privacy tags
<private>...</private> + + alt Prompt fully private (empty after stripping) + NewHook-->>IDE: Skip (don't save) + else Prompt has content + NewHook->>DB: INSERT OR IGNORE INTO sdk_sessions
(claude_session_id, project, first_user_prompt) + DB-->>NewHook: sessionDbId (new or existing) + + NewHook->>DB: UPDATE sdk_sessions
SET prompt_counter = prompt_counter + 1
WHERE id = sessionDbId + DB-->>NewHook: promptNumber (e.g., 1 for first, 2 for continuation) + + NewHook->>DB: INSERT INTO user_prompts
(session_id, prompt_number, prompt) + + NewHook->>Worker: POST /sessions/{sessionDbId}/init
{ project, userPrompt, promptNumber }
(fire-and-forget, 2s timeout) + Worker-->>NewHook: 200 OK (or timeout) + + NewHook-->>IDE: { continue: true, suppressOutput: true } + IDE-->>User: Prompt accepted + end + + Note over NewHook,DB: Idempotent: Same session_id → same sessionDbId +``` + +**Key Pattern:** The `INSERT OR IGNORE` ensures the same `session_id` always maps to the same `sessionDbId`, enabling conversation continuations. + **Input** (via stdin): ```json { @@ -214,6 +425,49 @@ The same `session_id` flows through ALL hooks in a conversation. The `createSDKS **Hook**: `save-hook.js` +### Sequence Diagram + +```mermaid +sequenceDiagram + participant Claude as Claude AI + participant IDE as IDE/Extension + participant SaveHook as save-hook.js + participant Worker as Worker Service + participant Agent as SDK Agent + participant DB as SQLite + Chroma + + Claude->>IDE: Uses tool: Read("/src/auth.ts") + IDE->>SaveHook: PostToolUse hook triggered
{ session_id, tool_name, tool_input, tool_response } + + SaveHook->>SaveHook: Check skip list
(TodoWrite, AskUserQuestion, etc.) + + alt Tool in skip list + SaveHook-->>IDE: Discard (low-value tool) + else Tool allowed + SaveHook->>SaveHook: Strip privacy tags from input/response + + SaveHook->>SaveHook: Ensure worker running
(PM2 health check) + + SaveHook->>Worker: POST /api/sessions/observations
{ claudeSessionId, tool_name, tool_input, tool_response, cwd }
(fire-and-forget, 2s timeout) + + SaveHook-->>IDE: { continue: true, suppressOutput: true } + IDE-->>Claude: Tool execution complete + + Note over Worker,DB: Async path (doesn't block IDE) + + Worker->>Worker: createSDKSession(claudeSessionId)
→ returns sessionDbId + Worker->>Worker: Check if prompt was private
(skip if fully private) + Worker->>Agent: Queue observation for processing + Agent->>Agent: Call Claude SDK to compress
observation into structured format + Agent->>DB: Save compressed observation
to sdk_sessions table + Agent->>DB: Sync to Chroma vector DB + end + + Note over SaveHook,DB: Total sync time: ~2ms
AI processing: 1-3s (async) +``` + +**Key Pattern:** The hook returns immediately after HTTP POST. AI compression happens asynchronously in the worker without blocking Claude's tool execution. + **Input** (via stdin): ```json { @@ -278,6 +532,45 @@ Timeout: 2000ms **Hook**: `summary-hook.js` +### Sequence Diagram + +```mermaid +sequenceDiagram + participant User + participant IDE as IDE/Extension + participant SummaryHook as summary-hook.js + participant Worker as Worker Service + participant Agent as SDK Agent + participant DB as SQLite Database + + User->>IDE: Stops asking questions
(pause, idle, or explicit stop) + IDE->>SummaryHook: Stop hook triggered
{ session_id, cwd, transcript_path } + + SummaryHook->>SummaryHook: Read transcript JSONL file + SummaryHook->>SummaryHook: Extract last user message
(type: "user") + SummaryHook->>SummaryHook: Extract last assistant message
(type: "assistant", filter <system-reminder>) + + SummaryHook->>Worker: POST /api/sessions/summarize
{ claudeSessionId, last_user_message, last_assistant_message }
(fire-and-forget, 2s timeout) + + SummaryHook->>Worker: POST /api/processing
{ isProcessing: false }
(stop spinner) + + SummaryHook-->>IDE: { continue: true, suppressOutput: true } + IDE-->>User: Session paused/stopped + + Note over Worker,DB: Async path + + Worker->>Worker: Lookup sessionDbId from claudeSessionId + Worker->>Agent: Queue summarization request + Agent->>Agent: Call Claude SDK with prompt:
"Summarize: request, investigated, learned, completed, next_steps" + Agent->>Agent: Parse XML response + Agent->>DB: INSERT INTO session_summaries
{ session_id, request, investigated, learned, completed, next_steps } + Agent->>DB: Sync to Chroma (for semantic search) + + Note over SummaryHook,DB: Total sync time: ~2ms
AI summarization: 2-5s (async) +``` + +**Key Pattern:** The summary is generated asynchronously and doesn't block the user from resuming work or closing the session. + **Input** (via stdin): ```json { @@ -332,6 +625,38 @@ Body: { isProcessing: false } **Hook**: `cleanup-hook.js` +### Sequence Diagram + +```mermaid +sequenceDiagram + participant User + participant IDE as IDE/Extension + participant CleanupHook as cleanup-hook.js + participant Worker as Worker Service + participant DB as SQLite Database + participant SSE as SSE Clients (Viewer UI) + + User->>IDE: Closes session
(exit, clear, logout) + IDE->>CleanupHook: SessionEnd hook triggered
{ session_id, cwd, transcript_path, reason } + + CleanupHook->>Worker: POST /api/sessions/complete
{ claudeSessionId, reason }
(fire-and-forget, 2s timeout) + + CleanupHook-->>IDE: { continue: true, suppressOutput: true } + IDE-->>User: Session closed + + Note over Worker,SSE: Async path + + Worker->>Worker: Lookup sessionDbId from claudeSessionId + Worker->>DB: UPDATE sdk_sessions
SET status = 'completed', completed_at = NOW()
WHERE claude_session_id = claudeSessionId + Worker->>SSE: Broadcast session completion event
(for live viewer UI updates) + + SSE-->>SSE: Update UI to show session as completed + + Note over CleanupHook,SSE: Total sync time: ~2ms +``` + +**Key Pattern:** Session completion is tracked for analytics and UI updates, but doesn't prevent the user from closing the IDE. + **Input** (via stdin): ```json { @@ -368,107 +693,125 @@ Timeout: 2000ms --- -## Data Flow Diagram +## Session State Machine + +Understanding session lifecycle and state transitions: ```mermaid -flowchart TD - subgraph step1["1. USER SUBMITS PROMPT"] - CC1[Claude Code] --> SS1[SessionStart hook] - SS1 --> CH[context-hook.js] - SS1 --> UMH[user-message-hook.js] - CH --> |"GET /api/context/inject"| CTX[returns context markdown] - UMH --> DISP[displays context info to user] +stateDiagram-v2 + [*] --> Initialized: SessionStart hook
(generate session_id) - CC1 --> UPS1[UserPromptSubmit hook] - UPS1 --> NH[new-hook.js] - NH --> |1| CREATE["db.createSDKSession()"] - NH --> |2| INC["db.incrementPromptCounter()"] - NH --> |3| STRIP["stripMemoryTagsFromPrompt()"] - NH --> |4| SAVE["db.saveUserPrompt()"] - NH --> |5| INIT["POST /sessions/{id}/init"] - INIT --> W1[Worker] - W1 --> SM1[SessionManager] - SM1 --> SA1[SDK Agent] - end + Initialized --> Active: UserPromptSubmit
(first prompt) - subgraph step2["2. CLAUDE USES A TOOL"] - CC2[Claude Code] --> PTU1[PostToolUse hook] - PTU1 --> SH[save-hook.js] - SH --> |"Skip if in SKIP_TOOLS"| CHECK{Check tool} - CHECK --> |allowed| OBS["POST /api/sessions/observations"] - OBS --> W2[Worker] - W2 --> SA2["SDK Agent → Claude compresses"] - SA2 --> STORE1["Store in SQLite + Chroma"] - STORE1 --> SSE[Broadcast to SSE clients] - end + Active --> Active: UserPromptSubmit
(continuation prompts)
promptNumber++ - subgraph step3["3. USER STOPS ASKING QUESTIONS"] - CC3[Claude Code] --> STOP1[Stop hook] - STOP1 --> SUMH[summary-hook.js] - SUMH --> EXT[Extract last messages from transcript] - EXT --> SUM["POST /api/sessions/summarize"] - SUM --> W3[Worker] - W3 --> SA3["SDK Agent → Claude generates summary"] - SA3 --> STORE2["Store in SQLite + Chroma"] - end + Active --> ObservationQueued: PostToolUse hook
(tool execution captured) - subgraph step4["4. SESSION CLOSES"] - CC4[Claude Code] --> SE1[SessionEnd hook] - SE1 --> CLN[cleanup-hook.js] - CLN --> COMP["POST /api/sessions/complete"] - COMP --> W4[Worker] - W4 --> MARK["Mark session as 'completed'"] - end + ObservationQueued --> Active: Observation processed
(async, non-blocking) - step1 --> step2 - step2 --> step3 - step3 --> step4 + Active --> Summarizing: Stop hook
(user pauses/stops) + + Summarizing --> Active: User resumes
(new prompt submitted) + + Summarizing --> Completed: SessionEnd hook
(session closes) + + Active --> Completed: SessionEnd hook
(session closes) + + Completed --> [*] + + note right of Active + session_id: constant (e.g., "claude-session-abc123") + sessionDbId: constant (e.g., 42) + promptNumber: increments (1, 2, 3, ...) + All operations use same sessionDbId + end note + + note right of ObservationQueued + Fire-and-forget HTTP + AI compression happens async + IDE never blocks + end note ``` +**Key Insights:** +- `session_id` never changes during a conversation +- `sessionDbId` is the database primary key for the session +- `promptNumber` increments with each user prompt +- State transitions are non-blocking (fire-and-forget pattern) + --- -## Session ID Threading +## Database Schema -The same `session_id` flows through ALL hooks in a conversation: +The session-centric data model that enables cross-session memory: ```mermaid -flowchart TD - SID["session_id (from Claude Code)"] +erDiagram + SDK_SESSIONS ||--o{ USER_PROMPTS : "has many" + SDK_SESSIONS ||--o{ OBSERVATIONS : "has many" + SDK_SESSIONS ||--o{ SESSION_SUMMARIES : "has many" - subgraph SS["SessionStart"] - SS_ID["session_id"] - end + SDK_SESSIONS { + integer id PK "Auto-increment primary key" + text claude_session_id UK "From IDE (e.g., 'claude-session-123')" + text project "Project name from cwd basename" + text first_user_prompt "Initial prompt that started session" + integer prompt_counter "Increments with each UserPromptSubmit" + text status "initialized | active | completed" + datetime created_at + datetime completed_at + } - subgraph UPS["UserPromptSubmit"] - UPS_ID["session_id (same)"] - UPS_CREATE["new-hook creates:
sdk_sessions.claude_session_id = session_id"] - UPS_RET["returns: sessionDbId (primary key)"] - UPS_ALL["All subsequent operations use sessionDbId"] - UPS_ID --> UPS_CREATE --> UPS_RET --> UPS_ALL - end + USER_PROMPTS { + integer id PK + integer session_id FK "References SDK_SESSIONS.id" + integer prompt_number "1, 2, 3, ... matches prompt_counter" + text prompt "User's actual prompt (tags stripped)" + datetime created_at + } - subgraph PTU["PostToolUse"] - PTU_ID["session_id (same)"] - PTU_GET["createSDKSession() returns sessionDbId"] - PTU_OBS["All observations tagged with sessionDbId"] - PTU_ID --> PTU_GET --> PTU_OBS - end + OBSERVATIONS { + integer id PK + integer session_id FK "References SDK_SESSIONS.id" + integer prompt_number "Which prompt this observation belongs to" + text tool_name "Read, Bash, Grep, Write, etc." + text tool_input_json "Stripped of privacy tags" + text tool_response_text "Stripped of privacy tags" + text compressed_observation "AI-generated structured observation" + datetime created_at + } - subgraph STOP["Stop"] - STOP_ID["session_id (same)"] - STOP_SUM["Summary tagged with sessionDbId"] - STOP_ID --> STOP_SUM - end - - subgraph SEND["SessionEnd"] - SEND_ID["session_id (same)"] - SEND_MARK["Mark sessionDbId as completed"] - SEND_ID --> SEND_MARK - end - - SID --> SS --> UPS --> PTU --> STOP --> SEND + SESSION_SUMMARIES { + integer id PK + integer session_id FK "References SDK_SESSIONS.id" + text request "What user requested" + text investigated "What was explored" + text learned "What was discovered" + text completed "What was accomplished" + text next_steps "What remains to be done" + datetime created_at + } ``` +**Idempotency Pattern:** + +```sql +-- This ensures same session_id always maps to same sessionDbId +INSERT OR IGNORE INTO sdk_sessions (claude_session_id, project, first_user_prompt) +VALUES (?, ?, ?) +RETURNING id; + +-- If already exists, returns existing row +-- If new, creates and returns new row +``` + +**Foreign Key Cascade:** + +All child tables (user_prompts, observations, session_summaries) use `session_id` foreign key referencing `SDK_SESSIONS.id`. This ensures: +- All data for a session is queryable by sessionDbId +- Session deletions cascade to child tables +- Efficient joins for context injection + Never generate your own session IDs. Always use the `session_id` provided by the IDE - this is the source of truth for linking all data together. diff --git a/docs/public/docs.json b/docs/public/docs.json index 8a739ebc..96ad53c1 100644 --- a/docs/public/docs.json +++ b/docs/public/docs.json @@ -55,7 +55,8 @@ "pages": [ "configuration", "development", - "troubleshooting" + "troubleshooting", + "platform-integration" ] }, { diff --git a/docs/public/platform-integration.mdx b/docs/public/platform-integration.mdx new file mode 100644 index 00000000..04c889a8 --- /dev/null +++ b/docs/public/platform-integration.mdx @@ -0,0 +1,1540 @@ +--- +title: Platform Integration Guide +description: Complete reference for integrating claude-mem worker service into VSCode extensions, IDE plugins, and CLI tools +icon: plug +--- + + +**Version:** 7.0.0 (December 2025) +**Target Audience:** Developers building claude-mem integrations (VSCode extensions, IDE plugins, CLI tools) + + +## Quick Reference + +### Worker Service Basics + +```typescript +const WORKER_BASE_URL = 'http://localhost:37777'; +const DEFAULT_PORT = 37777; // Override with CLAUDE_MEM_WORKER_PORT +``` + +### Most Common Operations + +```typescript +// Health check +GET /api/health + +// Create/get session and queue observation +POST /api/sessions/observations +Body: { claudeSessionId, tool_name, tool_input, tool_response, cwd } + +// Queue summary +POST /api/sessions/summarize +Body: { claudeSessionId, last_user_message, last_assistant_message } + +// Complete session +POST /api/sessions/complete +Body: { claudeSessionId } + +// Search observations +GET /api/search?query=authentication&type=observations&format=index&limit=20 + +// Get recent context for project +GET /api/context/recent?project=my-project&limit=3 +``` + +### Environment Variables + +```bash +CLAUDE_MEM_MODEL=claude-haiku-4-5 # Model for observations/summaries +CLAUDE_MEM_CONTEXT_OBSERVATIONS=50 # Observations injected at SessionStart +CLAUDE_MEM_WORKER_PORT=37777 # Worker service port +CLAUDE_MEM_PYTHON_VERSION=3.13 # Python version for chroma-mcp +``` + +### Build Commands (Local Development) + +```bash +npm run build # Compile TypeScript (hooks + worker) +npm run sync-marketplace # Copy to ~/.claude/plugins +npm run worker:restart # Restart PM2 worker +npm run worker:logs # View worker logs +pm2 list # Check worker status +``` + +## Worker Architecture + +### Request Flow + +```plaintext +Platform Hook/Extension + → HTTP Request to Worker (localhost:37777) + → Route Handler (SessionRoutes/DataRoutes/SearchRoutes/etc.) + → Domain Service (SessionManager/SearchManager/DatabaseManager) + → Database (SQLite3 + Chroma vector DB) + → SSE Broadcast (real-time UI updates) +``` + +### Domain Services + + + + SQLite connection management, initialization + + + Event-driven session lifecycle, message queues + + + Search orchestration (FTS5 + Chroma) + + + Server-Sent Events for real-time updates + + + Claude Agent SDK for generating observations/summaries + + + Query pagination utilities + + + User settings CRUD + + + Result formatting (index vs full) + + + Unified timeline generation + + + +### Route Organization + + + + - Health check endpoint + - Viewer UI (React app) + - SSE stream for real-time updates + + + - Session lifecycle (init, observations, summarize, complete) + - Privacy checks and tag stripping + - Auto-start SDK agent generators + + + - Data retrieval (observations, summaries, prompts, stats) + - Pagination support + - Processing status + + + - All search operations + - Unified search API + - Timeline context + - Semantic shortcuts + + + - User settings + - MCP toggle + - Git branch switching + + + +## API Reference + +### Session Lifecycle (SessionRoutes) + +#### Create/Get Session + Queue Observation (New API) + + +```http POST /api/sessions/observations +POST /api/sessions/observations +Content-Type: application/json + +{ + "claudeSessionId": "abc123", // Claude session identifier (string) + "tool_name": "Bash", + "tool_input": { "command": "ls" }, + "tool_response": { "stdout": "..." }, + "cwd": "/path/to/project" +} +``` + +```json Response +{ "status": "queued" } +// or +{ "status": "skipped", "reason": "private" } +``` + + + +**Privacy Check:** Skips if user prompt was entirely wrapped in `` tags. +**Tag Stripping:** Removes `` and `` tags before storage. +**Auto-Start:** Ensures SDK agent generator is running to process the queue. + + +#### Queue Summary (New API) + + +```http POST /api/sessions/summarize +POST /api/sessions/summarize +Content-Type: application/json + +{ + "claudeSessionId": "abc123", + "last_user_message": "User's message", + "last_assistant_message": "Assistant's response" +} +``` + +```json Response +{ "status": "queued" } +// or +{ "status": "skipped", "reason": "private" } +``` + + +#### Complete Session (New API) + + +```http POST /api/sessions/complete +POST /api/sessions/complete +Content-Type: application/json + +{ + "claudeSessionId": "abc123" +} +``` + +```json Response +{ "success": true } +// or +{ "success": true, "message": "No active session found" } +``` + + + +**Effect:** Stops SDK agent, marks session complete, broadcasts status change. + + +#### Legacy Endpoints (Still Supported) + + + + ```http + POST /sessions/:sessionDbId/init + Body: { userPrompt, promptNumber } + ``` + + + ```http + POST /sessions/:sessionDbId/observations + Body: { tool_name, tool_input, tool_response, prompt_number, cwd } + ``` + + + ```http + POST /sessions/:sessionDbId/summarize + Body: { last_user_message, last_assistant_message } + ``` + + + ```http + POST /sessions/:sessionDbId/complete + ``` + + + + +New integrations should use `/api/sessions/*` endpoints with `claudeSessionId`. + + +### Data Retrieval (DataRoutes) + +#### Get Paginated Data + + + + ```http + GET /api/observations?offset=0&limit=20&project=my-project + ``` + + + ```http + GET /api/summaries?offset=0&limit=20&project=my-project + ``` + + + ```http + GET /api/prompts?offset=0&limit=20&project=my-project + ``` + + + +```json Response Format +{ + "items": [...], + "hasMore": boolean, + "offset": number, + "limit": number +} +``` + +#### Get by ID + + + + ```http + GET /api/observation/:id + ``` + + + ```http + GET /api/session/:id + ``` + + + ```http + GET /api/prompt/:id + ``` + + + +#### Get Database Stats + +```http +GET /api/stats +``` + +```json Response +{ + "worker": { + "version": "7.0.0", + "uptime": 12345, + "activeSessions": 2, + "sseClients": 1, + "port": 37777 + }, + "database": { + "path": "~/.claude-mem/claude-mem.db", + "size": 1048576, + "observations": 500, + "sessions": 50, + "summaries": 25 + } +} +``` + +#### Get Projects List + +```http +GET /api/projects +``` + +```json Response +{ + "projects": ["claude-mem", "other-project", ...] +} +``` + +#### Get Processing Status + +```http +GET /api/processing-status +``` + +```json Response +{ + "isProcessing": boolean, + "queueDepth": number +} +``` + +### Search Operations (SearchRoutes) + +#### Unified Search + +```http +GET /api/search?query=authentication&type=observations&format=index&limit=20 +``` + + + Search query text (optional, omit for filter-only) + + + + "observations" | "sessions" | "prompts" + + + + "index" | "full" + + + + Number of results + + + + Filter by project name + + + + Filter by observation type (discovery, decision, bugfix, feature, refactor) + + + + Filter by concepts (comma-separated) + + + + Filter by file paths (comma-separated) + + + + ISO timestamp (filter start) + + + + ISO timestamp (filter end) + + +```json Response +{ + "observations": [...], + "sessions": [...], + "prompts": [...] +} +``` + + +**Format Options:** +- `index`: Minimal fields for list display (id, title, preview) +- `full`: Complete entity with all fields + + +#### Unified Timeline + +```http +GET /api/timeline?anchor=123&depth_before=10&depth_after=10&project=my-project +``` + + + Anchor point (observation ID, "S123" for session, or ISO timestamp) + + + + Records before anchor + + + + Records after anchor + + + + Filter by project + + +```json Response +[ + { "type": "observation", "id": 120, "created_at_epoch": ..., ... }, + { "type": "session", "id": 5, "created_at_epoch": ..., ... }, + { "type": "observation", "id": 123, "created_at_epoch": ..., ... } +] +``` + +#### Semantic Shortcuts + + + + ```http + GET /api/decisions?format=index&limit=20 + ``` + + + ```http + GET /api/changes?format=index&limit=20 + ``` + + + ```http + GET /api/how-it-works?format=index&limit=20 + ``` + + + +#### Search by Concept + +```http +GET /api/search/by-concept?concept=discovery&format=index&limit=10&project=my-project +``` + +#### Search by File Path + +```http +GET /api/search/by-file?filePath=src/services/worker-service.ts&format=index&limit=10 +``` + +#### Search by Type + +```http +GET /api/search/by-type?type=bugfix&format=index&limit=10 +``` + +#### Get Recent Context + +```http +GET /api/context/recent?project=my-project&limit=3 +``` + +```json Response +{ + "summaries": [...], + "observations": [...] +} +``` + +#### Context Preview (for Settings UI) + +```http +GET /api/context/preview?project=my-project +``` + + +Returns plain text with ANSI colors for terminal display + + +#### Context Injection (for Hooks) + +```http +GET /api/context/inject?project=my-project&colors=true +``` + + +Returns pre-formatted context string ready for display + + +### Settings & Configuration (SettingsRoutes) + +#### Get/Update User Settings + + +```http GET +GET /api/settings +``` + +```json GET Response +{ + "sidebarOpen": boolean, + "selectedProject": string | null +} +``` + +```http POST +POST /api/settings +Body: { "sidebarOpen": true, "selectedProject": "my-project" } +``` + +```json POST Response +{ "success": true } +``` + + +#### MCP Server Status/Toggle + + +```http GET Status +GET /api/mcp/status +``` + +```json GET Response +{ "enabled": boolean } +``` + +```http POST Toggle +POST /api/mcp/toggle +Body: { "enabled": true } +``` + +```json POST Response +{ "success": true, "enabled": boolean } +``` + + +#### Git Branch Operations + + + + ```http + GET /api/branch/status + ``` + ```json + { + "current": "main", + "remote": "origin/main", + "ahead": 0, + "behind": 0 + } + ``` + + + ```http + POST /api/branch/switch + Body: { "branch": "feature/new-feature" } + ``` + ```json + { "success": true } + ``` + + + ```http + POST /api/branch/update + ``` + ```json + { "success": true, "updated": boolean } + ``` + + + +### Viewer & Real-Time Updates (ViewerRoutes) + +#### Health Check + +```http +GET /api/health +``` + +```json Response +{ "status": "ok" } +``` + +#### Viewer UI + +```http +GET / +``` + + +Returns HTML for React app + + +#### SSE Stream + +```http +GET /stream +``` + + +**Server-Sent Events stream** + +Event Types: +- `processing_status`: { type, isProcessing, queueDepth } +- `session_started`: { type, sessionDbId, project } +- `observation_queued`: { type, sessionDbId } +- `summarize_queued`: { type } +- `observation_created`: { type, observation } +- `summary_created`: { type, summary } +- `new_prompt`: { type, id, claude_session_id, project, prompt_number, prompt_text, created_at_epoch } + + +## Data Models + +### Active Session (In-Memory) + +```typescript +interface ActiveSession { + sessionDbId: number; // Database ID (numeric) + claudeSessionId: string; // Claude session identifier (string) + sdkSessionId: string | null; // SDK session ID + project: string; // Project name + userPrompt: string; // Current user prompt text + pendingMessages: PendingMessage[]; // Queue of pending operations + abortController: AbortController; // For cancellation + generatorPromise: Promise | null; // SDK agent promise + lastPromptNumber: number; // Last processed prompt number + startTime: number; // Session start timestamp + cumulativeInputTokens: number; // Total input tokens + cumulativeOutputTokens: number; // Total output tokens +} + +interface PendingMessage { + type: 'observation' | 'summarize'; + tool_name?: string; + tool_input?: any; + tool_response?: any; + prompt_number?: number; + cwd?: string; + last_user_message?: string; + last_assistant_message?: string; +} +``` + +### Database Entities + + + + ```typescript + interface SDKSessionRow { + id: number; + claude_session_id: string; + sdk_session_id: string; + project: string; + user_prompt: string; + created_at_epoch: number; + completed_at_epoch?: number; + } + ``` + + + ```typescript + interface ObservationRow { + id: number; + sdk_session_id: string; + title: string; + subtitle?: string; + summary: string; + facts: string; // JSON array of fact strings + concepts: string; // JSON array of concept strings + files_touched: string; // JSON array of file paths + obs_type: string; // discovery, decision, bugfix, feature, refactor + project: string; + created_at_epoch: number; + prompt_number: number; + } + ``` + + + ```typescript + interface SessionSummaryRow { + id: number; + sdk_session_id: string; + summary_text: string; + facts: string; // JSON array + concepts: string; // JSON array + files_touched: string; // JSON array + project: string; + created_at_epoch: number; + } + ``` + + + ```typescript + interface UserPromptRow { + id: number; + claude_session_id: string; + sdk_session_id: string; + project: string; + prompt_number: number; + prompt_text: string; + created_at_epoch: number; + } + ``` + + + +### Search Results + +```typescript +interface ObservationSearchResult { + id: number; + title: string; + subtitle?: string; + summary: string; + facts: string[]; // Parsed from JSON + concepts: string[]; // Parsed from JSON + files_touched: string[]; // Parsed from JSON + obs_type: string; + project: string; + created_at_epoch: number; + prompt_number: number; + rank?: number; // FTS5 rank score +} + +interface SessionSummarySearchResult { + id: number; + summary_text: string; + facts: string[]; + concepts: string[]; + files_touched: string[]; + project: string; + created_at_epoch: number; + rank?: number; +} + +interface UserPromptSearchResult { + id: number; + claude_session_id: string; + project: string; + prompt_number: number; + prompt_text: string; + created_at_epoch: number; + rank?: number; +} +``` + +### Timeline Item + +```typescript +interface TimelineItem { + type: 'observation' | 'session' | 'prompt'; + id: number; + created_at_epoch: number; + // Entity-specific fields based on type +} +``` + +## Integration Patterns + +### Mapping Claude Code Hooks to Worker API + + + + Not needed for new API - sessions are auto-created on first observation + + + No API call needed - user_prompt is captured by first observation in the prompt + + + ```typescript + async function onPostToolUse(context: HookContext) { + const { session_id, tool_name, tool_input, tool_result, cwd } = context; + + const response = await fetch('http://localhost:37777/api/sessions/observations', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + claudeSessionId: session_id, + tool_name, + tool_input, + tool_response: tool_result, + cwd + }) + }); + + const result = await response.json(); + // result.status === 'queued' | 'skipped' + } + ``` + + + ```typescript + async function onSummary(context: HookContext) { + const { session_id, last_user_message, last_assistant_message } = context; + + await fetch('http://localhost:37777/api/sessions/summarize', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + claudeSessionId: session_id, + last_user_message, + last_assistant_message + }) + }); + } + ``` + + + ```typescript + async function onSessionEnd(context: HookContext) { + const { session_id } = context; + + await fetch('http://localhost:37777/api/sessions/complete', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + claudeSessionId: session_id + }) + }); + } + ``` + + + +### VSCode Extension Integration + +#### Language Model Tool Registration + +```typescript +import * as vscode from 'vscode'; + +interface SearchTool extends vscode.LanguageModelChatTool { + invoke( + options: vscode.LanguageModelToolInvocationOptions<{ query: string }>, + token: vscode.CancellationToken + ): vscode.ProviderResult; +} + +const searchTool: SearchTool = { + invoke: async (options, token) => { + const { query } = options.input; + + try { + const response = await fetch( + `http://localhost:37777/api/search?query=${encodeURIComponent(query)}&format=index&limit=10` + ); + + if (!response.ok) { + throw new Error(`Search failed: ${response.statusText}`); + } + + const results = await response.json(); + + // Format results for language model + return new vscode.LanguageModelToolResult([ + new vscode.LanguageModelTextPart(JSON.stringify(results, null, 2)) + ]); + } catch (error) { + return new vscode.LanguageModelToolResult([ + new vscode.LanguageModelTextPart(`Error: ${error.message}`) + ]); + } + } +}; + +// Register tool +vscode.lm.registerTool('claude-mem-search', searchTool); +``` + +#### Chat Participant Implementation + +```typescript +const participant = vscode.chat.createChatParticipant('claude-mem', async (request, context, stream, token) => { + const claudeSessionId = context.session.id; + + // First message in conversation - no initialization needed + // Session is auto-created on first observation + + // Process user message + stream.markdown(`Searching memory for: ${request.prompt}\n\n`); + + const response = await fetch( + `http://localhost:37777/api/search?query=${encodeURIComponent(request.prompt)}&format=index&limit=5` + ); + + const results = await response.json(); + + if (results.observations?.length > 0) { + stream.markdown('**Found observations:**\n'); + for (const obs of results.observations) { + stream.markdown(`- ${obs.title} (${obs.project})\n`); + } + } + + return { metadata: { command: 'search' } }; +}); +``` + +## Error Handling & Resilience + +### Connection Failures + +```typescript +async function callWorkerWithFallback( + endpoint: string, + options?: RequestInit +): Promise { + try { + const response = await fetch(`http://localhost:37777${endpoint}`, { + ...options, + signal: AbortSignal.timeout(5000) // 5s timeout + }); + + if (!response.ok) { + throw new Error(`HTTP ${response.status}: ${response.statusText}`); + } + + return await response.json(); + } catch (error) { + console.error(`Worker unavailable (${endpoint}):`, error); + return null; // Graceful degradation + } +} +``` + +### Retry Logic with Exponential Backoff + +```typescript +async function retryWithBackoff( + fn: () => Promise, + maxRetries = 3, + baseDelay = 100 +): Promise { + for (let attempt = 0; attempt < maxRetries; attempt++) { + try { + return await fn(); + } catch (error) { + if (attempt === maxRetries - 1) throw error; + + const delay = baseDelay * Math.pow(2, attempt); + await new Promise(resolve => setTimeout(resolve, delay)); + } + } + throw new Error('Max retries exceeded'); +} +``` + +### Worker Health Check + +```typescript +async function isWorkerHealthy(): Promise { + try { + const response = await fetch('http://localhost:37777/api/health', { + signal: AbortSignal.timeout(2000) + }); + return response.ok; + } catch { + return false; + } +} +``` + +### Privacy Tag Handling + + +The worker automatically strips privacy tags before storage: +- `content` - User-level privacy control +- `content` - System-level tag (prevents recursive storage) + +**Privacy Check:** Observations/summaries are skipped if the entire user prompt was wrapped in `` tags. + + +### Custom Error Classes + +```typescript +class WorkerUnavailableError extends Error { + constructor() { + super('Claude-mem worker is not running or unreachable'); + this.name = 'WorkerUnavailableError'; + } +} + +class WorkerTimeoutError extends Error { + constructor(endpoint: string) { + super(`Worker request timed out: ${endpoint}`); + this.name = 'WorkerTimeoutError'; + } +} +``` + +### SSE Stream Error Handling + +```typescript +function connectToSSE(onEvent: (event: any) => void) { + const eventSource = new EventSource('http://localhost:37777/stream'); + + eventSource.onmessage = (event) => { + try { + const data = JSON.parse(event.data); + onEvent(data); + } catch (error) { + console.error('SSE parse error:', error); + } + }; + + eventSource.onerror = (error) => { + console.error('SSE connection error:', error); + eventSource.close(); + + // Reconnect after 5 seconds + setTimeout(() => connectToSSE(onEvent), 5000); + }; + + return eventSource; +} +``` + +## Development Workflow + +### Project Structure (Recommended) + +```plaintext +vscode-extension/ +├── src/ +│ ├── extension.ts # Extension entry point +│ ├── services/ +│ │ ├── WorkerClient.ts # HTTP client for worker +│ │ └── MemoryManager.ts # High-level memory operations +│ ├── chat/ +│ │ └── participant.ts # Chat participant implementation +│ └── tools/ +│ ├── search.ts # Search language model tool +│ └── context.ts # Context injection tool +├── package.json +├── tsconfig.json +└── README.md +``` + +### Build Configuration (esbuild) + +```javascript +// build.js +const esbuild = require('esbuild'); + +esbuild.build({ + entryPoints: ['src/extension.ts'], + bundle: true, + outfile: 'dist/extension.js', + external: ['vscode'], + format: 'cjs', + platform: 'node', + target: 'node18', + sourcemap: true +}).catch(() => process.exit(1)); +``` + +### package.json (VSCode Extension) + +```json +{ + "name": "claude-mem-vscode", + "displayName": "Claude-Mem", + "version": "1.0.0", + "engines": { + "vscode": "^1.95.0" + }, + "activationEvents": [ + "onStartupFinished" + ], + "main": "./dist/extension.js", + "contributes": { + "chatParticipants": [ + { + "id": "claude-mem", + "name": "memory", + "description": "Search your persistent memory" + } + ], + "languageModelTools": [ + { + "name": "claude-mem-search", + "displayName": "Search Memory", + "description": "Search persistent memory for observations, sessions, and prompts" + } + ] + }, + "scripts": { + "build": "node build.js", + "watch": "node build.js --watch", + "package": "vsce package" + }, + "devDependencies": { + "@types/vscode": "^1.95.0", + "esbuild": "^0.19.0", + "typescript": "^5.3.0" + } +} +``` + +### Local Testing Loop + + + + ```bash + npm run watch + ``` + + + ```bash + pm2 list + pm2 logs claude-mem-worker + ``` + + + ```bash + curl http://localhost:37777/api/health + curl "http://localhost:37777/api/search?query=test&limit=5" + ``` + + + Press F5 to launch extension host + + + +### Debug Configuration (.vscode/launch.json) + +```json +{ + "version": "0.2.0", + "configurations": [ + { + "name": "Run Extension", + "type": "extensionHost", + "request": "launch", + "args": [ + "--extensionDevelopmentPath=${workspaceFolder}" + ], + "outFiles": [ + "${workspaceFolder}/dist/**/*.js" + ], + "preLaunchTask": "npm: build" + } + ] +} +``` + +## Testing Strategy + +### Unit Tests (Worker Client) + +```typescript +import { describe, it, expect } from 'vitest'; +import { WorkerClient } from '../src/services/WorkerClient'; + +describe('WorkerClient', () => { + it('should check worker health', async () => { + const client = new WorkerClient(); + const healthy = await client.isHealthy(); + expect(healthy).toBe(true); + }); + + it('should queue observation', async () => { + const client = new WorkerClient(); + const result = await client.queueObservation({ + claudeSessionId: 'test-123', + tool_name: 'Bash', + tool_input: { command: 'ls' }, + tool_response: { stdout: 'file1.txt' }, + cwd: '/tmp' + }); + expect(result.status).toBe('queued'); + }); + + it('should search observations', async () => { + const client = new WorkerClient(); + const results = await client.search({ query: 'test', limit: 5 }); + expect(results).toHaveProperty('observations'); + }); +}); +``` + +### Integration Tests (With Worker Spawning) + +```typescript +import { spawn } from 'child_process'; +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; + +describe('Worker Integration', () => { + let workerProcess: ReturnType; + + beforeAll(async () => { + // Start worker process + workerProcess = spawn('node', ['dist/worker-service.js'], { + env: { ...process.env, CLAUDE_MEM_WORKER_PORT: '37778' } + }); + + // Wait for worker to be ready + await new Promise(resolve => setTimeout(resolve, 2000)); + }); + + afterAll(() => { + workerProcess.kill(); + }); + + it('should respond to health check', async () => { + const response = await fetch('http://localhost:37778/api/health'); + expect(response.ok).toBe(true); + }); +}); +``` + +### Manual Testing Checklist + + + + - [ ] Worker starts successfully (`pm2 list`) + - [ ] Health endpoint responds (`curl http://localhost:37777/api/health`) + - [ ] SSE stream connects (`curl http://localhost:37777/stream`) + + + + - [ ] Queue observation creates session + - [ ] Observation appears in database + - [ ] Privacy tags are stripped + - [ ] Private prompts are skipped + - [ ] Queue summary creates summary + - [ ] Complete session stops processing + + + + - [ ] Search observations by query + - [ ] Search sessions by query + - [ ] Search prompts by query + - [ ] Get recent context for project + - [ ] Get timeline around observation + - [ ] Semantic shortcuts (decisions, changes, how-it-works) + + + + - [ ] SSE broadcasts processing status + - [ ] SSE broadcasts new observations + - [ ] SSE broadcasts new summaries + - [ ] SSE broadcasts new prompts + + + + - [ ] Graceful degradation when worker unavailable + - [ ] Timeout handling for slow requests + - [ ] Retry logic for transient failures + + + +## Code Examples + +### Complete WorkerClient Implementation + + +```typescript WorkerClient.ts +export class WorkerClient { + private baseUrl: string; + + constructor(port: number = 37777) { + this.baseUrl = `http://localhost:${port}`; + } + + async isHealthy(): Promise { + try { + const response = await fetch(`${this.baseUrl}/api/health`, { + signal: AbortSignal.timeout(2000) + }); + return response.ok; + } catch { + return false; + } + } + + async queueObservation(data: { + claudeSessionId: string; + tool_name: string; + tool_input: any; + tool_response: any; + cwd?: string; + }): Promise<{ status: string; reason?: string }> { + const response = await fetch(`${this.baseUrl}/api/sessions/observations`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(data), + signal: AbortSignal.timeout(5000) + }); + + if (!response.ok) { + throw new Error(`Failed to queue observation: ${response.statusText}`); + } + + return await response.json(); + } + + async queueSummarize(data: { + claudeSessionId: string; + last_user_message?: string; + last_assistant_message?: string; + }): Promise<{ status: string; reason?: string }> { + const response = await fetch(`${this.baseUrl}/api/sessions/summarize`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(data), + signal: AbortSignal.timeout(5000) + }); + + if (!response.ok) { + throw new Error(`Failed to queue summary: ${response.statusText}`); + } + + return await response.json(); + } + + async completeSession(claudeSessionId: string): Promise { + const response = await fetch(`${this.baseUrl}/api/sessions/complete`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ claudeSessionId }), + signal: AbortSignal.timeout(5000) + }); + + if (!response.ok) { + throw new Error(`Failed to complete session: ${response.statusText}`); + } + } + + async search(params: { + query?: string; + type?: 'observations' | 'sessions' | 'prompts'; + format?: 'index' | 'full'; + limit?: number; + project?: string; + }): Promise { + const queryString = new URLSearchParams( + Object.entries(params) + .filter(([_, v]) => v !== undefined) + .map(([k, v]) => [k, String(v)]) + ).toString(); + + const response = await fetch( + `${this.baseUrl}/api/search?${queryString}`, + { signal: AbortSignal.timeout(10000) } + ); + + if (!response.ok) { + throw new Error(`Search failed: ${response.statusText}`); + } + + return await response.json(); + } + + connectSSE(onEvent: (event: any) => void): EventSource { + const eventSource = new EventSource(`${this.baseUrl}/stream`); + + eventSource.onmessage = (event) => { + try { + const data = JSON.parse(event.data); + onEvent(data); + } catch (error) { + console.error('SSE parse error:', error); + } + }; + + eventSource.onerror = (error) => { + console.error('SSE connection error:', error); + }; + + return eventSource; + } +} +``` + + +### Search Language Model Tool + +```typescript +import * as vscode from 'vscode'; +import { WorkerClient } from './WorkerClient'; + +export function registerSearchTool(context: vscode.ExtensionContext) { + const client = new WorkerClient(); + + const searchTool = vscode.lm.registerTool('claude-mem-search', { + description: 'Search persistent memory for observations, sessions, and prompts', + inputSchema: { + type: 'object', + properties: { + query: { + type: 'string', + description: 'Search query text' + }, + type: { + type: 'string', + enum: ['observations', 'sessions', 'prompts'], + description: 'Type of results to return' + }, + limit: { + type: 'number', + description: 'Maximum number of results', + default: 10 + } + }, + required: ['query'] + }, + invoke: async (options, token) => { + const { query, type, limit = 10 } = options.input; + + try { + const results = await client.search({ + query, + type, + format: 'index', + limit + }); + + // Format results for language model + let formatted = ''; + + if (results.observations?.length > 0) { + formatted += '## Observations\n\n'; + for (const obs of results.observations) { + formatted += `- **${obs.title}** (${obs.project})\n`; + formatted += ` ${obs.summary}\n`; + if (obs.concepts?.length > 0) { + formatted += ` Concepts: ${obs.concepts.join(', ')}\n`; + } + formatted += '\n'; + } + } + + return new vscode.LanguageModelToolResult([ + new vscode.LanguageModelTextPart(formatted) + ]); + } catch (error) { + return new vscode.LanguageModelToolResult([ + new vscode.LanguageModelTextPart(`Error: ${error.message}`) + ]); + } + } + }); + + context.subscriptions.push(searchTool); +} +``` + +## Critical Implementation Notes + + +### sessionDbId vs claudeSessionId + +**IMPORTANT:** Use `claudeSessionId` (string) for new API endpoints, not `sessionDbId` (number). + +- `sessionDbId` - Numeric database ID (legacy endpoints only) +- `claudeSessionId` - String identifier from Claude platform (new endpoints) + + + +### JSON String Fields + +Fields like `facts`, `concepts`, and `files_touched` are stored as JSON strings and require parsing: + +```typescript +const observation = await client.getObservationById(123); +const facts = JSON.parse(observation.facts); // string[] array +const concepts = JSON.parse(observation.concepts); // string[] array +``` + + + +### Timestamps + +All `created_at_epoch` fields are in **milliseconds**, not seconds: + +```typescript +const date = new Date(observation.created_at_epoch); // ✅ Correct +const date = new Date(observation.created_at_epoch * 1000); // ❌ Wrong (already in ms) +``` + + + +### Asynchronous Processing + +Workers process observations/summaries asynchronously. Results appear in the database 1-2 seconds after queuing. Use SSE events for real-time notifications. + + + +### Privacy Tags + +Always wrap sensitive content in `` tags to prevent storage: + +```typescript +const userMessage = 'API key: sk-1234567890'; +// This observation will be skipped (entire prompt is private) +``` + + +## Additional Resources + + + + Complete claude-mem documentation + + + Source code and issue tracker + + + Worker architecture details + + + Database structure and queries + +