Replace search skill with mem-search (#91)
* feat: add mem-search skill with progressive disclosure architecture Add comprehensive mem-search skill for accessing claude-mem's persistent cross-session memory database. Implements progressive disclosure workflow and token-efficient search patterns. Features: - 12 search operations (observations, sessions, prompts, by-type, by-concept, by-file, timelines, etc.) - Progressive disclosure principles to minimize token usage - Anti-patterns documentation to guide LLM behavior - HTTP API integration for all search functionality - Common workflows with composition examples Structure: - SKILL.md: Entry point with temporal trigger patterns - principles/: Progressive disclosure + anti-patterns - operations/: 12 search operation files 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * docs: add CHANGELOG entry for mem-search skill Document mem-search skill addition in Unreleased section with: - 100% effectiveness compliance metrics - Comparison to previous search skill implementation - Progressive disclosure architecture details - Reference to audit report documentation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * docs: add mem-search skill audit report Add comprehensive audit report validating mem-search skill against Anthropic's official skill-creator documentation. Report includes: - Effectiveness metrics comparison (search vs mem-search) - Critical issues analysis for production readiness - Compliance validation across 6 key dimensions - Reference implementation guidance Result: mem-search achieves 100% compliance vs search's 67% 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: Add comprehensive search architecture analysis document - Document current state of dual search architectures (HTTP API and MCP) - Analyze HTTP endpoints and MCP search server architectures - Identify DRY violations across search implementations - Evaluate the use of curl as the optimal approach for search - Provide architectural recommendations for immediate and long-term improvements - Outline action plan for cleanup, feature parity, DRY refactoring * refactor: Remove deprecated search skill documentation and operations * refactor: Reorganize documentation into public and context directories Changes: - Created docs/public/ for Mintlify documentation (.mdx files) - Created docs/context/ for internal planning and implementation docs - Moved all .mdx files and assets to docs/public/ - Moved all internal .md files to docs/context/ - Added CLAUDE.md to both directories explaining their purpose - Updated docs.json paths to work with new structure Benefits: - Clear separation between user-facing and internal documentation - Easier to maintain Mintlify docs in dedicated directory - Internal context files organized separately 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Enhance session management and continuity in hooks - Updated new-hook.ts to clarify session_id threading and idempotent session creation. - Modified prompts.ts to require claudeSessionId for continuation prompts, ensuring session context is maintained. - Improved SessionStore.ts documentation on createSDKSession to emphasize idempotent behavior and session connection. - Refined SDKAgent.ts to detail continuation prompt logic and its reliance on session.claudeSessionId for unified session handling. --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Alex Newman <thedotmack@gmail.com>
This commit is contained in:
basher83
@@ -0,0 +1,309 @@
|
||||
---
|
||||
title: "Database Architecture"
|
||||
description: "SQLite schema, FTS5 search, and data storage"
|
||||
---
|
||||
|
||||
# Database Architecture
|
||||
|
||||
Claude-Mem uses SQLite 3 with the better-sqlite3 native module for persistent storage and FTS5 for full-text search.
|
||||
|
||||
## Database Location
|
||||
|
||||
- **Current**: `~/.claude-mem/claude-mem.db`
|
||||
|
||||
**Note**: Despite the README claiming v4.0.0+ moved the database to `${CLAUDE_PLUGIN_ROOT}/data/`, the actual implementation still uses `~/.claude-mem/`.
|
||||
|
||||
## Database Implementation
|
||||
|
||||
**Primary Implementation**: better-sqlite3 (native SQLite module)
|
||||
- Used by: SessionStore and SessionSearch
|
||||
- Format: Synchronous API with better performance
|
||||
- **Note**: Database.ts (using bun:sqlite) is legacy code
|
||||
|
||||
## Core Tables
|
||||
|
||||
### 1. sdk_sessions
|
||||
|
||||
Tracks active and completed sessions.
|
||||
|
||||
```sql
|
||||
CREATE TABLE sdk_sessions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
sdk_session_id TEXT UNIQUE NOT NULL,
|
||||
claude_session_id TEXT,
|
||||
project TEXT NOT NULL,
|
||||
prompt_counter INTEGER DEFAULT 0,
|
||||
status TEXT NOT NULL DEFAULT 'active',
|
||||
created_at TEXT NOT NULL,
|
||||
created_at_epoch INTEGER NOT NULL,
|
||||
completed_at TEXT,
|
||||
completed_at_epoch INTEGER,
|
||||
last_activity_at TEXT,
|
||||
last_activity_epoch INTEGER
|
||||
);
|
||||
```
|
||||
|
||||
**Indexes**:
|
||||
- `idx_sdk_sessions_claude_session` on `claude_session_id`
|
||||
- `idx_sdk_sessions_project` on `project`
|
||||
- `idx_sdk_sessions_status` on `status`
|
||||
- `idx_sdk_sessions_created_at` on `created_at_epoch DESC`
|
||||
|
||||
### 2. observations
|
||||
|
||||
Individual tool executions with hierarchical structure.
|
||||
|
||||
```sql
|
||||
CREATE TABLE observations (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
session_id TEXT NOT NULL,
|
||||
sdk_session_id TEXT NOT NULL,
|
||||
claude_session_id TEXT,
|
||||
project TEXT NOT NULL,
|
||||
prompt_number INTEGER,
|
||||
tool_name TEXT NOT NULL,
|
||||
correlation_id TEXT,
|
||||
|
||||
-- Hierarchical fields
|
||||
title TEXT,
|
||||
subtitle TEXT,
|
||||
narrative TEXT,
|
||||
text TEXT,
|
||||
facts TEXT,
|
||||
concepts TEXT,
|
||||
type TEXT,
|
||||
files_read TEXT,
|
||||
files_modified TEXT,
|
||||
|
||||
created_at TEXT NOT NULL,
|
||||
created_at_epoch INTEGER NOT NULL,
|
||||
|
||||
FOREIGN KEY (sdk_session_id) REFERENCES sdk_sessions(sdk_session_id)
|
||||
);
|
||||
```
|
||||
|
||||
**Observation Types**:
|
||||
- `decision` - Architectural or design decisions
|
||||
- `bugfix` - Bug fixes and corrections
|
||||
- `feature` - New features or capabilities
|
||||
- `refactor` - Code refactoring and cleanup
|
||||
- `discovery` - Learnings about the codebase
|
||||
- `change` - General changes and modifications
|
||||
|
||||
**Indexes**:
|
||||
- `idx_observations_session` on `session_id`
|
||||
- `idx_observations_sdk_session` on `sdk_session_id`
|
||||
- `idx_observations_project` on `project`
|
||||
- `idx_observations_tool_name` on `tool_name`
|
||||
- `idx_observations_created_at` on `created_at_epoch DESC`
|
||||
- `idx_observations_type` on `type`
|
||||
|
||||
### 3. session_summaries
|
||||
|
||||
AI-generated session summaries (multiple per session).
|
||||
|
||||
```sql
|
||||
CREATE TABLE session_summaries (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
sdk_session_id TEXT NOT NULL,
|
||||
claude_session_id TEXT,
|
||||
project TEXT NOT NULL,
|
||||
prompt_number INTEGER,
|
||||
|
||||
-- Summary fields
|
||||
request TEXT,
|
||||
investigated TEXT,
|
||||
learned TEXT,
|
||||
completed TEXT,
|
||||
next_steps TEXT,
|
||||
notes TEXT,
|
||||
|
||||
created_at TEXT NOT NULL,
|
||||
created_at_epoch INTEGER NOT NULL,
|
||||
|
||||
FOREIGN KEY (sdk_session_id) REFERENCES sdk_sessions(sdk_session_id)
|
||||
);
|
||||
```
|
||||
|
||||
**Indexes**:
|
||||
- `idx_session_summaries_sdk_session` on `sdk_session_id`
|
||||
- `idx_session_summaries_project` on `project`
|
||||
- `idx_session_summaries_created_at` on `created_at_epoch DESC`
|
||||
|
||||
### 4. user_prompts
|
||||
|
||||
Raw user prompts with FTS5 search (as of v4.2.0).
|
||||
|
||||
```sql
|
||||
CREATE TABLE user_prompts (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
sdk_session_id TEXT NOT NULL,
|
||||
claude_session_id TEXT,
|
||||
project TEXT NOT NULL,
|
||||
prompt_number INTEGER,
|
||||
prompt_text TEXT NOT NULL,
|
||||
created_at TEXT NOT NULL,
|
||||
created_at_epoch INTEGER NOT NULL,
|
||||
|
||||
FOREIGN KEY (sdk_session_id) REFERENCES sdk_sessions(sdk_session_id)
|
||||
);
|
||||
```
|
||||
|
||||
**Indexes**:
|
||||
- `idx_user_prompts_sdk_session` on `sdk_session_id`
|
||||
- `idx_user_prompts_project` on `project`
|
||||
- `idx_user_prompts_created_at` on `created_at_epoch DESC`
|
||||
|
||||
### Legacy Tables
|
||||
|
||||
- **sessions**: Legacy session tracking (v3.x)
|
||||
- **memories**: Legacy compressed memory chunks (v3.x)
|
||||
- **overviews**: Legacy session summaries (v3.x)
|
||||
|
||||
## FTS5 Full-Text Search
|
||||
|
||||
SQLite FTS5 (Full-Text Search) virtual tables enable fast full-text search across observations, summaries, and user prompts.
|
||||
|
||||
### FTS5 Virtual Tables
|
||||
|
||||
#### observations_fts
|
||||
|
||||
```sql
|
||||
CREATE VIRTUAL TABLE observations_fts USING fts5(
|
||||
title,
|
||||
subtitle,
|
||||
narrative,
|
||||
text,
|
||||
facts,
|
||||
concepts,
|
||||
content='observations',
|
||||
content_rowid='id'
|
||||
);
|
||||
```
|
||||
|
||||
#### session_summaries_fts
|
||||
|
||||
```sql
|
||||
CREATE VIRTUAL TABLE session_summaries_fts USING fts5(
|
||||
request,
|
||||
investigated,
|
||||
learned,
|
||||
completed,
|
||||
next_steps,
|
||||
notes,
|
||||
content='session_summaries',
|
||||
content_rowid='id'
|
||||
);
|
||||
```
|
||||
|
||||
#### user_prompts_fts
|
||||
|
||||
```sql
|
||||
CREATE VIRTUAL TABLE user_prompts_fts USING fts5(
|
||||
prompt_text,
|
||||
content='user_prompts',
|
||||
content_rowid='id'
|
||||
);
|
||||
```
|
||||
|
||||
### Automatic Synchronization
|
||||
|
||||
FTS5 tables stay in sync via triggers:
|
||||
|
||||
```sql
|
||||
-- Insert trigger example
|
||||
CREATE TRIGGER observations_ai AFTER INSERT ON observations BEGIN
|
||||
INSERT INTO observations_fts(rowid, title, subtitle, narrative, text, facts, concepts)
|
||||
VALUES (new.id, new.title, new.subtitle, new.narrative, new.text, new.facts, new.concepts);
|
||||
END;
|
||||
|
||||
-- Update trigger example
|
||||
CREATE TRIGGER observations_au AFTER UPDATE ON observations BEGIN
|
||||
INSERT INTO observations_fts(observations_fts, rowid, title, subtitle, narrative, text, facts, concepts)
|
||||
VALUES('delete', old.id, old.title, old.subtitle, old.narrative, old.text, old.facts, old.concepts);
|
||||
INSERT INTO observations_fts(rowid, title, subtitle, narrative, text, facts, concepts)
|
||||
VALUES (new.id, new.title, new.subtitle, new.narrative, new.text, new.facts, new.concepts);
|
||||
END;
|
||||
|
||||
-- Delete trigger example
|
||||
CREATE TRIGGER observations_ad AFTER DELETE ON observations BEGIN
|
||||
INSERT INTO observations_fts(observations_fts, rowid, title, subtitle, narrative, text, facts, concepts)
|
||||
VALUES('delete', old.id, old.title, old.subtitle, old.narrative, old.text, old.facts, old.concepts);
|
||||
END;
|
||||
```
|
||||
|
||||
### FTS5 Query Syntax
|
||||
|
||||
FTS5 supports rich query syntax:
|
||||
|
||||
- **Simple**: `"error handling"`
|
||||
- **AND**: `"error" AND "handling"`
|
||||
- **OR**: `"bug" OR "fix"`
|
||||
- **NOT**: `"bug" NOT "feature"`
|
||||
- **Phrase**: `"'exact phrase'"`
|
||||
- **Column**: `title:"authentication"`
|
||||
|
||||
### Security
|
||||
|
||||
As of v4.2.3, all FTS5 queries are properly escaped to prevent SQL injection:
|
||||
- Double quotes are escaped: `query.replace(/"/g, '""')`
|
||||
- Comprehensive test suite with 332 injection attack tests
|
||||
|
||||
## Database Classes
|
||||
|
||||
### SessionStore
|
||||
|
||||
CRUD operations for sessions, observations, summaries, and user prompts.
|
||||
|
||||
**Location**: `src/services/sqlite/SessionStore.ts`
|
||||
|
||||
**Methods**:
|
||||
- `createSession()`
|
||||
- `getSession()`
|
||||
- `updateSession()`
|
||||
- `createObservation()`
|
||||
- `getObservations()`
|
||||
- `createSummary()`
|
||||
- `getSummaries()`
|
||||
- `createUserPrompt()`
|
||||
|
||||
### SessionSearch
|
||||
|
||||
FTS5 full-text search with 8 specialized search methods.
|
||||
|
||||
**Location**: `src/services/sqlite/SessionSearch.ts`
|
||||
|
||||
**Methods**:
|
||||
- `searchObservations()` - Full-text search across observations
|
||||
- `searchSessions()` - Full-text search across summaries
|
||||
- `searchUserPrompts()` - Full-text search across user prompts
|
||||
- `findByConcept()` - Find by concept tags
|
||||
- `findByFile()` - Find by file references
|
||||
- `findByType()` - Find by observation type
|
||||
- `getRecentContext()` - Get recent session context
|
||||
- `advancedSearch()` - Combined filters
|
||||
|
||||
## Migrations
|
||||
|
||||
Database schema is managed via migrations in `src/services/sqlite/migrations.ts`.
|
||||
|
||||
**Migration History**:
|
||||
- Migration 001: Initial schema (sessions, memories, overviews, diagnostics, transcript_events)
|
||||
- Migration 002: Hierarchical memory fields (title, subtitle, facts, concepts, files_touched)
|
||||
- Migration 003: SDK sessions and observations
|
||||
- Migration 004: Session summaries
|
||||
- Migration 005: Multi-prompt sessions (prompt_counter, prompt_number)
|
||||
- Migration 006: FTS5 virtual tables and triggers
|
||||
- Migration 007-010: Various improvements and user prompts table
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
- **Indexes**: All foreign keys and frequently queried columns are indexed
|
||||
- **FTS5**: Full-text search is significantly faster than LIKE queries
|
||||
- **Triggers**: Automatic synchronization has minimal overhead
|
||||
- **Connection Pooling**: better-sqlite3 reuses connections efficiently
|
||||
- **Synchronous API**: better-sqlite3 uses synchronous API for better performance
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
See [Troubleshooting - Database Issues](../troubleshooting.md#database-issues) for common problems and solutions.
|
||||
@@ -0,0 +1,269 @@
|
||||
---
|
||||
title: "Plugin Hooks"
|
||||
description: "7 hook scripts that power Claude-Mem"
|
||||
---
|
||||
|
||||
# Plugin Hooks
|
||||
|
||||
Claude-Mem integrates with Claude Code through 7 hook scripts across 5 lifecycle events that capture events and inject context.
|
||||
|
||||
## Hook Overview
|
||||
|
||||
| Hook Name | Purpose | Timeout | Script |
|
||||
|---------------------|--------------------------------------|---------|-------------------------|
|
||||
| SessionStart | Smart dependency installation | 300s | smart-install.js |
|
||||
| SessionStart | Inject context from previous sessions| 300s | context-hook.js |
|
||||
| SessionStart | Display first-time setup message | 10s | user-message-hook.js |
|
||||
| UserPromptSubmit | Create/track new sessions | 120s | new-hook.js |
|
||||
| PostToolUse | Capture tool execution observations | 120s | save-hook.js |
|
||||
| Stop | Generate session summaries | 120s | summary-hook.js |
|
||||
| SessionEnd | Mark sessions complete | 120s | cleanup-hook.js |
|
||||
|
||||
## Hook Configuration
|
||||
|
||||
Hooks are configured in `plugin/hooks/hooks.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"description": "Claude-mem memory system hooks",
|
||||
"hooks": {
|
||||
"SessionStart": [{
|
||||
"matcher": "startup|clear|compact",
|
||||
"hooks": [{
|
||||
"type": "command",
|
||||
"command": "node \"${CLAUDE_PLUGIN_ROOT}/../scripts/smart-install.js\" && node ${CLAUDE_PLUGIN_ROOT}/scripts/context-hook.js",
|
||||
"timeout": 300
|
||||
}, {
|
||||
"type": "command",
|
||||
"command": "node ${CLAUDE_PLUGIN_ROOT}/scripts/user-message-hook.js",
|
||||
"timeout": 10
|
||||
}]
|
||||
}],
|
||||
"UserPromptSubmit": [{
|
||||
"hooks": [{
|
||||
"type": "command",
|
||||
"command": "node ${CLAUDE_PLUGIN_ROOT}/scripts/new-hook.js",
|
||||
"timeout": 120
|
||||
}]
|
||||
}],
|
||||
"PostToolUse": [{
|
||||
"matcher": "*",
|
||||
"hooks": [{
|
||||
"type": "command",
|
||||
"command": "node ${CLAUDE_PLUGIN_ROOT}/scripts/save-hook.js",
|
||||
"timeout": 120
|
||||
}]
|
||||
}],
|
||||
"Stop": [{
|
||||
"hooks": [{
|
||||
"type": "command",
|
||||
"command": "node ${CLAUDE_PLUGIN_ROOT}/scripts/summary-hook.js",
|
||||
"timeout": 120
|
||||
}]
|
||||
}],
|
||||
"SessionEnd": [{
|
||||
"hooks": [{
|
||||
"type": "command",
|
||||
"command": "node ${CLAUDE_PLUGIN_ROOT}/scripts/cleanup-hook.js",
|
||||
"timeout": 120
|
||||
}]
|
||||
}]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 1. SessionStart Hook - Smart Install (`smart-install.js`)
|
||||
|
||||
**Purpose**: Intelligently manage dependencies and ensure worker service is running.
|
||||
|
||||
**Behavior**:
|
||||
- Checks if dependencies need installation using version marker (`.install-version`)
|
||||
- Only runs npm install when:
|
||||
- First-time installation (no node_modules)
|
||||
- Version changed in package.json
|
||||
- Critical dependency missing (e.g., better-sqlite3)
|
||||
- Provides Windows-specific error messages for build tool issues
|
||||
- Auto-starts PM2 worker service after installation
|
||||
- Fast when already installed (~10ms vs 2-5 seconds)
|
||||
|
||||
**Input** (via stdin):
|
||||
```json
|
||||
{
|
||||
"session_id": "claude-session-123",
|
||||
"cwd": "/path/to/project",
|
||||
"source": "startup"
|
||||
}
|
||||
```
|
||||
|
||||
**Implementation**: `scripts/smart-install.js`
|
||||
|
||||
**Key Features**:
|
||||
- Version caching prevents redundant installs
|
||||
- Cross-platform compatible (Windows, macOS, Linux)
|
||||
- Helpful error messages with troubleshooting steps
|
||||
- Non-blocking worker startup
|
||||
|
||||
**v5.0.3 Enhancement**: Smart caching eliminates 2-5 second npm install on every SessionStart, reducing to ~10ms for already-installed dependencies.
|
||||
|
||||
## 2. SessionStart Hook - Context Injection (`context-hook.js`)
|
||||
|
||||
**Purpose**: Inject context from previous sessions into Claude's initial context.
|
||||
|
||||
**Behavior**:
|
||||
- Retrieves last 10 session summaries with three-tier verbosity (v4.2.0)
|
||||
- Retrieves last 50 observations (configurable via `CLAUDE_MEM_CONTEXT_OBSERVATIONS`)
|
||||
- Returns context via `hookSpecificOutput` in JSON format (fixed in v4.1.0)
|
||||
- Formats results as progressive disclosure index
|
||||
|
||||
**Input** (via stdin):
|
||||
```json
|
||||
{
|
||||
"session_id": "claude-session-123",
|
||||
"cwd": "/path/to/project",
|
||||
"source": "startup"
|
||||
}
|
||||
```
|
||||
|
||||
**Output** (via stdout):
|
||||
```json
|
||||
{
|
||||
"hookSpecificOutput": "# Recent Sessions\n\n## Session 1...\n"
|
||||
}
|
||||
```
|
||||
|
||||
**Implementation**: `src/hooks/context-hook.ts`
|
||||
|
||||
## 3. SessionStart Hook - User Message (`user-message-hook.js`)
|
||||
|
||||
**Purpose**: Display helpful user messages during first-time setup or when viewing context.
|
||||
|
||||
**Behavior**:
|
||||
- Shows first-time setup message when node_modules is missing
|
||||
- Displays formatted context information with colors
|
||||
- Provides tips for using claude-mem effectively
|
||||
- Shows link to web viewer UI (http://localhost:37777)
|
||||
- Exits with code 3 (informational, not error)
|
||||
|
||||
**Output Example**:
|
||||
```
|
||||
📝 Claude-Mem Context Loaded
|
||||
ℹ️ Note: This appears as stderr but is informational only
|
||||
|
||||
[Context details...]
|
||||
|
||||
📺 Watch live in browser http://localhost:37777/ (New! v5.1)
|
||||
```
|
||||
|
||||
**Implementation**: `plugin/scripts/user-message-hook.js` (minified)
|
||||
|
||||
**Key Features**:
|
||||
- User-friendly first-time setup experience
|
||||
- Visual context display with colors
|
||||
- Links to new features (viewer UI)
|
||||
- Non-intrusive messaging
|
||||
|
||||
## 4. UserPromptSubmit Hook (`new-hook.js`)
|
||||
|
||||
**Purpose**: Create new session records and initialize session tracking.
|
||||
|
||||
**Behavior**:
|
||||
- Creates new session in database
|
||||
- Initializes session tracking
|
||||
- Saves raw user prompts for full-text search (as of v4.2.0)
|
||||
- Sends init signal to worker service
|
||||
|
||||
**Input** (via stdin):
|
||||
```json
|
||||
{
|
||||
"session_id": "claude-session-123",
|
||||
"cwd": "/path/to/project",
|
||||
"prompt": "User's actual prompt text"
|
||||
}
|
||||
```
|
||||
|
||||
**Implementation**: `src/hooks/new-hook.ts`
|
||||
|
||||
## 5. PostToolUse Hook (`save-hook.js`)
|
||||
|
||||
**Purpose**: Capture tool execution observations.
|
||||
|
||||
**Behavior**:
|
||||
- Fires after EVERY tool execution (Read, Write, Edit, Bash, etc.)
|
||||
- Sends observations to worker service for processing
|
||||
- Includes correlation IDs for tracing
|
||||
- Filters low-value observations
|
||||
|
||||
**Input** (via stdin):
|
||||
```json
|
||||
{
|
||||
"session_id": "claude-session-123",
|
||||
"cwd": "/path/to/project",
|
||||
"tool_name": "Read",
|
||||
"tool_input": {...},
|
||||
"tool_result": "...",
|
||||
"correlation_id": "abc-123"
|
||||
}
|
||||
```
|
||||
|
||||
**Implementation**: `src/hooks/save-hook.ts`
|
||||
|
||||
## 6. Stop Hook (`summary-hook.js`)
|
||||
|
||||
**Purpose**: Generate session summaries when Claude stops.
|
||||
|
||||
**Behavior**:
|
||||
- Triggers final summary generation
|
||||
- Sends summarize request to worker service
|
||||
- Summary includes: request, completed, learned, next_steps
|
||||
|
||||
**Input** (via stdin):
|
||||
```json
|
||||
{
|
||||
"session_id": "claude-session-123",
|
||||
"cwd": "/path/to/project",
|
||||
"source": "user_stop"
|
||||
}
|
||||
```
|
||||
|
||||
**Implementation**: `src/hooks/summary-hook.ts`
|
||||
|
||||
## 7. SessionEnd Hook (`cleanup-hook.js`)
|
||||
|
||||
**Purpose**: Mark sessions as completed (graceful cleanup as of v4.1.0).
|
||||
|
||||
**Behavior**:
|
||||
- Marks sessions as completed
|
||||
- Skips cleanup on `/clear` commands to preserve ongoing sessions
|
||||
- Allows workers to finish pending operations naturally
|
||||
- Previously sent DELETE requests; now uses graceful completion
|
||||
|
||||
**Input** (via stdin):
|
||||
```json
|
||||
{
|
||||
"session_id": "claude-session-123",
|
||||
"cwd": "/path/to/project",
|
||||
"source": "normal"
|
||||
}
|
||||
```
|
||||
|
||||
**Implementation**: `src/hooks/cleanup-hook.ts`
|
||||
|
||||
## Hook Development
|
||||
|
||||
### Adding a New Hook
|
||||
|
||||
1. Create hook implementation in `src/hooks/your-hook.ts`
|
||||
2. Add to `plugin/hooks/hooks.json`
|
||||
3. Rebuild with `npm run build`
|
||||
|
||||
### Hook Best Practices
|
||||
|
||||
- **Fast execution**: Hooks should complete quickly (< 1s ideal)
|
||||
- **Graceful degradation**: Don't block Claude if worker is down
|
||||
- **Structured logging**: Use logger for debugging
|
||||
- **Error handling**: Catch and log errors, don't crash
|
||||
- **JSON output**: Use `hookSpecificOutput` for context injection
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
See [Troubleshooting - Hook Issues](../troubleshooting.md#hook-issues) for common problems and solutions.
|
||||
@@ -0,0 +1,230 @@
|
||||
---
|
||||
title: "Architecture Overview"
|
||||
description: "System components and data flow in Claude-Mem"
|
||||
---
|
||||
|
||||
# Architecture Overview
|
||||
|
||||
## System Components
|
||||
|
||||
Claude-Mem operates as a Claude Code plugin with five core components:
|
||||
|
||||
1. **Plugin Hooks** - Capture lifecycle events (7 hook files)
|
||||
2. **Worker Service** - Process observations via Claude Agent SDK + HTTP API (10 search endpoints)
|
||||
3. **Database Layer** - Store sessions and observations (SQLite + FTS5 + ChromaDB)
|
||||
4. **Search Skill** - Skill-based search with progressive disclosure (v5.4.0+)
|
||||
5. **Viewer UI** - Web-based real-time memory stream visualization
|
||||
|
||||
## Technology Stack
|
||||
|
||||
| Layer | Technology |
|
||||
|------------------------|-------------------------------------------|
|
||||
| **Language** | TypeScript (ES2022, ESNext modules) |
|
||||
| **Runtime** | Node.js 18+ |
|
||||
| **Database** | SQLite 3 with better-sqlite3 driver |
|
||||
| **Vector Store** | ChromaDB (optional, for semantic search) |
|
||||
| **HTTP Server** | Express.js 4.18 |
|
||||
| **Real-time** | Server-Sent Events (SSE) |
|
||||
| **UI Framework** | React + TypeScript |
|
||||
| **AI SDK** | @anthropic-ai/claude-agent-sdk |
|
||||
| **Build Tool** | esbuild (bundles TypeScript) |
|
||||
| **Process Manager** | PM2 |
|
||||
| **Testing** | Node.js built-in test runner |
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Memory Pipeline
|
||||
```
|
||||
Hook (stdin) → Database → Worker Service → SDK Processor → Database → Next Session Hook
|
||||
```
|
||||
|
||||
1. **Input**: Claude Code sends tool execution data via stdin to hooks
|
||||
2. **Storage**: Hooks write observations to SQLite database
|
||||
3. **Processing**: Worker service reads observations, processes via SDK
|
||||
4. **Output**: Processed summaries written back to database
|
||||
5. **Retrieval**: Next session's context hook reads summaries from database
|
||||
|
||||
### Search Pipeline (v5.4.0+)
|
||||
```
|
||||
User Query → Skill Invoked → HTTP API → SessionSearch Service → FTS5 Database → Search Results → Claude
|
||||
```
|
||||
|
||||
1. **User Query**: User asks naturally: "What bugs did we fix?"
|
||||
2. **Skill Invoked**: Claude recognizes intent and invokes search skill
|
||||
3. **HTTP API**: Skill uses curl to call HTTP endpoint (e.g., `/api/search/observations`)
|
||||
4. **SessionSearch**: Worker service queries FTS5 virtual tables
|
||||
5. **Format**: Results formatted and returned to skill
|
||||
6. **Return**: Claude presents formatted results to user
|
||||
|
||||
**Token Savings**: ~2,250 tokens per session vs MCP approach through progressive disclosure
|
||||
|
||||
## Session Lifecycle
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ 0. Smart Install Hook Fires │
|
||||
│ Checks dependencies (cached), only runs on version changes │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ 1. Session Starts → Context Hook Fires │
|
||||
│ Starts PM2 worker if needed, injects context from previous │
|
||||
│ sessions (configurable observation count) │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ 2. User Types Prompt → UserPromptSubmit Hook Fires │
|
||||
│ Creates session in database, saves raw user prompt for FTS5 │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ 3. Claude Uses Tools → PostToolUse Hook Fires (100+ times) │
|
||||
│ Captures tool executions, sends to worker for AI compression │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ 4. Worker Processes → Claude Agent SDK Analyzes │
|
||||
│ Extracts structured learnings via iterative AI processing │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ 5. Claude Stops → Summary Hook Fires │
|
||||
│ Generates final summary with request, completions, learnings │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
↓
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ 6. Session Ends → Cleanup Hook Fires │
|
||||
│ Marks session complete (graceful, not DELETE), ready for │
|
||||
│ next session context. Skips on /clear to preserve ongoing │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
claude-mem/
|
||||
├── src/
|
||||
│ ├── hooks/ # Hook implementations (7 hooks)
|
||||
│ │ ├── smart-install.ts # Dependency check (cached)
|
||||
│ │ ├── context-hook.ts # SessionStart
|
||||
│ │ ├── user-message-hook.ts # UserMessage (for debugging)
|
||||
│ │ ├── new-hook.ts # UserPromptSubmit
|
||||
│ │ ├── save-hook.ts # PostToolUse
|
||||
│ │ ├── summary-hook.ts # Stop
|
||||
│ │ └── cleanup-hook.ts # SessionEnd
|
||||
│ │
|
||||
│ ├── sdk/ # Claude Agent SDK integration
|
||||
│ │ ├── prompts.ts # XML prompt builders
|
||||
│ │ ├── parser.ts # XML response parser
|
||||
│ │ └── worker.ts # Main SDK agent loop
|
||||
│ │
|
||||
│ ├── services/
|
||||
│ │ ├── worker-service.ts # Express HTTP + SSE service
|
||||
│ │ └── sqlite/ # Database layer
|
||||
│ │ ├── SessionStore.ts # CRUD operations
|
||||
│ │ ├── SessionSearch.ts # FTS5 search service
|
||||
│ │ ├── migrations.ts
|
||||
│ │ └── types.ts
|
||||
│ │
|
||||
│ ├── ui/ # Viewer UI
|
||||
│ │ └── viewer/ # React + TypeScript web interface
|
||||
│ │ ├── components/ # UI components
|
||||
│ │ ├── hooks/ # React hooks
|
||||
│ │ ├── utils/ # Utilities
|
||||
│ │ └── assets/ # Fonts, logos
|
||||
│ │
|
||||
│ ├── shared/ # Shared utilities
|
||||
│ │ ├── config.ts
|
||||
│ │ ├── paths.ts
|
||||
│ │ └── storage.ts
|
||||
│ │
|
||||
│ └── utils/
|
||||
│ ├── logger.ts
|
||||
│ ├── platform.ts
|
||||
│ └── port-allocator.ts
|
||||
│
|
||||
├── plugin/ # Plugin distribution
|
||||
│ ├── .claude-plugin/
|
||||
│ │ └── plugin.json
|
||||
│ ├── hooks/
|
||||
│ │ └── hooks.json
|
||||
│ ├── scripts/ # Built executables
|
||||
│ │ ├── smart-install.js
|
||||
│ │ ├── context-hook.js
|
||||
│ │ ├── user-message-hook.js
|
||||
│ │ ├── new-hook.js
|
||||
│ │ ├── save-hook.js
|
||||
│ │ ├── summary-hook.js
|
||||
│ │ ├── cleanup-hook.js
|
||||
│ │ └── worker-service.cjs # Background worker + HTTP API
|
||||
│ │
|
||||
│ ├── skills/ # Agent skills (v5.4.0+)
|
||||
│ │ ├── search/ # Search skill with progressive disclosure
|
||||
│ │ │ ├── SKILL.md # Skill frontmatter (~250 tokens)
|
||||
│ │ │ └── operations/ # Detailed operation docs
|
||||
│ │ ├── troubleshoot/ # Troubleshooting skill
|
||||
│ │ └── version-bump/ # Version management skill
|
||||
│ │
|
||||
│ └── ui/ # Built viewer UI
|
||||
│ └── viewer.html # Self-contained bundle
|
||||
│
|
||||
├── tests/ # Test suite
|
||||
├── docs/ # Documentation
|
||||
└── ecosystem.config.cjs # PM2 configuration
|
||||
```
|
||||
|
||||
## Component Details
|
||||
|
||||
### 1. Plugin Hooks (7 Hooks)
|
||||
- **smart-install.js** - Cached dependency checker (only runs on version changes)
|
||||
- **context-hook.js** - SessionStart: Starts PM2 worker, injects context
|
||||
- **user-message-hook.js** - UserMessage: Debugging hook
|
||||
- **new-hook.js** - UserPromptSubmit: Creates session, saves prompt
|
||||
- **save-hook.js** - PostToolUse: Captures tool executions
|
||||
- **summary-hook.js** - Stop: Generates session summary
|
||||
- **cleanup-hook.js** - SessionEnd: Marks session complete
|
||||
|
||||
See [Plugin Hooks](/architecture/hooks) for detailed hook documentation.
|
||||
|
||||
### 2. Worker Service
|
||||
Express.js HTTP server on port 37777 (configurable) with:
|
||||
- 10 search HTTP API endpoints (v5.4.0+)
|
||||
- 8 viewer UI HTTP/SSE endpoints
|
||||
- Async observation processing via Claude Agent SDK
|
||||
- Real-time updates via Server-Sent Events
|
||||
- Auto-managed by PM2 process manager
|
||||
|
||||
See [Worker Service](/architecture/worker-service) for HTTP API and endpoints.
|
||||
|
||||
### 3. Database Layer
|
||||
SQLite3 with better-sqlite3 driver featuring:
|
||||
- FTS5 virtual tables for full-text search
|
||||
- SessionStore for CRUD operations
|
||||
- SessionSearch for FTS5 queries
|
||||
- Location: `~/.claude-mem/claude-mem.db`
|
||||
|
||||
See [Database Architecture](/architecture/database) for schema and FTS5 search.
|
||||
|
||||
### 4. Search Skill (v5.4.0+)
|
||||
Skill-based search with progressive disclosure providing 10 search operations:
|
||||
- Search observations, sessions, prompts (full-text FTS5)
|
||||
- Filter by type, concept, file
|
||||
- Get recent context, timeline, timeline by query
|
||||
- API help documentation
|
||||
|
||||
**Token Savings**: ~2,250 tokens per session vs MCP approach
|
||||
- Skill frontmatter: ~250 tokens (loaded at session start)
|
||||
- Full instructions: ~2,500 tokens (loaded on-demand when invoked)
|
||||
- HTTP API endpoints instead of MCP tools
|
||||
|
||||
See [Search Architecture](/architecture/search-architecture) for technical details and examples.
|
||||
|
||||
### 5. Viewer UI
|
||||
React + TypeScript web interface at http://localhost:37777 featuring:
|
||||
- Real-time memory stream via Server-Sent Events
|
||||
- Infinite scroll pagination with automatic deduplication
|
||||
- Project filtering and settings persistence
|
||||
- GPU-accelerated animations
|
||||
- Self-contained HTML bundle (viewer.html)
|
||||
|
||||
Built with esbuild into a single file deployment.
|
||||
@@ -0,0 +1,440 @@
|
||||
---
|
||||
title: "Search Architecture"
|
||||
description: "Skill-based search with HTTP API and progressive disclosure"
|
||||
---
|
||||
|
||||
# Search Architecture
|
||||
|
||||
Claude-Mem uses a skill-based search architecture that provides intelligent memory retrieval through natural language queries. This replaced the MCP-based approach in v5.4.0, saving ~2,250 tokens per session start.
|
||||
|
||||
## Overview
|
||||
|
||||
**Architecture**: Skill-Based Search + HTTP API + Progressive Disclosure
|
||||
|
||||
**Key Components**:
|
||||
1. **Search Skill** (`plugin/skills/search/SKILL.md`) - Auto-invoked when users ask about past work
|
||||
2. **HTTP API Endpoints** (10 routes) - Fast, efficient search operations on port 37777
|
||||
3. **Worker Service** - Express.js server with FTS5 full-text search
|
||||
4. **SQLite Database** - Persistent storage with FTS5 virtual tables
|
||||
5. **Chroma Vector DB** - Semantic search with hybrid retrieval
|
||||
|
||||
## How It Works
|
||||
|
||||
### 1. User Query (Natural Language)
|
||||
|
||||
```
|
||||
User: "What bugs did we fix last session?"
|
||||
```
|
||||
|
||||
### 2. Skill Invocation
|
||||
|
||||
Claude recognizes the intent and invokes the search skill:
|
||||
- Skill frontmatter (~250 tokens) loaded at session start
|
||||
- Full skill instructions loaded on-demand when skill is invoked
|
||||
- Progressive disclosure pattern minimizes context overhead
|
||||
|
||||
### 3. HTTP API Call
|
||||
|
||||
The skill uses `curl` to call the HTTP API:
|
||||
|
||||
```bash
|
||||
curl "http://localhost:37777/api/search/observations?query=bugs&type=bugfix&limit=5"
|
||||
```
|
||||
|
||||
### 4. FTS5 Search
|
||||
|
||||
Worker service queries SQLite FTS5 virtual tables:
|
||||
|
||||
```sql
|
||||
SELECT * FROM observations_fts
|
||||
WHERE observations_fts MATCH ?
|
||||
AND type = 'bugfix'
|
||||
ORDER BY rank
|
||||
LIMIT 5
|
||||
```
|
||||
|
||||
### 5. Results Formatted
|
||||
|
||||
Skill formats results and returns to Claude:
|
||||
|
||||
```
|
||||
## Recent Bugfixes
|
||||
|
||||
1. [bugfix] Fixed authentication token expiry
|
||||
Date: 2025-11-08 14:23:45
|
||||
Files: src/auth/jwt.ts
|
||||
|
||||
2. [bugfix] Resolved database connection leak
|
||||
Date: 2025-11-08 13:15:22
|
||||
Files: src/services/database.ts
|
||||
```
|
||||
|
||||
### 6. User Sees Answer
|
||||
|
||||
Claude presents the formatted results naturally in conversation.
|
||||
|
||||
## Architecture Change (v5.4.0)
|
||||
|
||||
### Before: MCP-Based Search
|
||||
|
||||
**Approach**: 9 MCP tools registered at session start
|
||||
|
||||
**Token Cost**: ~2,500 tokens in tool definitions per session
|
||||
- Each tool's schema, parameters, descriptions loaded
|
||||
- All 9 tools available whether needed or not
|
||||
- No progressive disclosure
|
||||
|
||||
**Example MCP Tool**:
|
||||
```json
|
||||
{
|
||||
"name": "search_observations",
|
||||
"description": "Full-text search across observations...",
|
||||
"inputSchema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"query": { "type": "string", "description": "..." },
|
||||
"type": { "type": "array", "items": { "enum": [...] } },
|
||||
"format": { "enum": ["index", "full"] },
|
||||
// ... many more parameters
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### After: Skill-Based Search
|
||||
|
||||
**Approach**: 1 search skill with progressive disclosure
|
||||
|
||||
**Token Cost**: ~250 tokens in skill frontmatter per session
|
||||
- Only skill description loaded at session start
|
||||
- Full instructions loaded on-demand when skill is invoked
|
||||
- HTTP API endpoints instead of MCP protocol
|
||||
|
||||
**Example Skill Frontmatter**:
|
||||
```markdown
|
||||
# Claude-Mem Search Skill
|
||||
|
||||
Access claude-mem's persistent memory through a comprehensive HTTP API.
|
||||
Search for past work, understand context, and learn from previous decisions.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
Invoke this skill when users ask about:
|
||||
- Past work: "What did we do last session?"
|
||||
- Bug fixes: "Did we fix this before?"
|
||||
- Features: "How did we implement authentication?"
|
||||
...
|
||||
```
|
||||
|
||||
**Token Savings**: ~2,250 tokens per session start (90% reduction)
|
||||
|
||||
## HTTP API Endpoints
|
||||
|
||||
The worker service exposes 10 search endpoints:
|
||||
|
||||
### Full-Text Search
|
||||
|
||||
```
|
||||
GET /api/search/observations
|
||||
GET /api/search/sessions
|
||||
GET /api/search/prompts
|
||||
```
|
||||
|
||||
**Parameters**:
|
||||
- `query` - FTS5 search query (required)
|
||||
- `type` - Filter by type (bugfix, feature, refactor, etc.)
|
||||
- `project` - Filter by project name
|
||||
- `limit` - Maximum results (default: 20)
|
||||
- `offset` - Pagination offset
|
||||
- `format` - Response format (index or full)
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
curl "http://localhost:37777/api/search/observations?query=authentication&type=decision&limit=5"
|
||||
```
|
||||
|
||||
### Filtered Search
|
||||
|
||||
```
|
||||
GET /api/search/by-type
|
||||
GET /api/search/by-concept
|
||||
GET /api/search/by-file
|
||||
```
|
||||
|
||||
**Parameters**:
|
||||
- `type` / `concept` / `filePath` - Filter criteria (required)
|
||||
- `project` - Filter by project
|
||||
- `limit` - Maximum results
|
||||
- `format` - Response format
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
curl "http://localhost:37777/api/search/by-file?filePath=worker-service.ts&limit=10"
|
||||
```
|
||||
|
||||
### Context Retrieval
|
||||
|
||||
```
|
||||
GET /api/context/recent
|
||||
GET /api/context/timeline
|
||||
GET /api/timeline/by-query
|
||||
```
|
||||
|
||||
**Parameters**:
|
||||
- `project` - Filter by project
|
||||
- `limit` - Number of sessions/records
|
||||
- `anchor` - Timeline anchor point (ID or timestamp)
|
||||
- `depth_before` - Records before anchor
|
||||
- `depth_after` - Records after anchor
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
curl "http://localhost:37777/api/context/recent?project=claude-mem&limit=5"
|
||||
```
|
||||
|
||||
### Documentation
|
||||
|
||||
```
|
||||
GET /api/search/help
|
||||
```
|
||||
|
||||
Returns API documentation in JSON format.
|
||||
|
||||
## Progressive Disclosure Pattern
|
||||
|
||||
The search skill uses progressive disclosure to minimize token usage:
|
||||
|
||||
### Layer 1: Skill Frontmatter (Session Start)
|
||||
|
||||
**What's Loaded**: Skill description and when to use it (~250 tokens)
|
||||
|
||||
**Purpose**: Claude can recognize when to invoke the skill
|
||||
|
||||
**Example**:
|
||||
```markdown
|
||||
# Claude-Mem Search Skill
|
||||
|
||||
Access claude-mem's persistent memory through a comprehensive HTTP API.
|
||||
|
||||
## When to Use This Skill
|
||||
Invoke this skill when users ask about:
|
||||
- Past work: "What did we do last session?"
|
||||
- Bug fixes: "Did we fix this before?"
|
||||
...
|
||||
```
|
||||
|
||||
### Layer 2: Full Skill Instructions (On-Demand)
|
||||
|
||||
**What's Loaded**: Complete operation documentation (~2,500 tokens)
|
||||
|
||||
**Purpose**: Detailed instructions for each search operation
|
||||
|
||||
**When Loaded**: Only when Claude invokes the skill
|
||||
|
||||
**Example Structure**:
|
||||
```
|
||||
/skills/search/
|
||||
├── SKILL.md (main frontmatter)
|
||||
├── operations/
|
||||
│ ├── observations.md (detailed instructions)
|
||||
│ ├── sessions.md
|
||||
│ ├── prompts.md
|
||||
│ ├── by-type.md
|
||||
│ ├── by-concept.md
|
||||
│ ├── by-file.md
|
||||
│ ├── recent-context.md
|
||||
│ ├── timeline.md
|
||||
│ ├── timeline-by-query.md
|
||||
│ ├── help.md
|
||||
│ ├── formatting.md
|
||||
│ └── common-workflows.md
|
||||
```
|
||||
|
||||
### Layer 3: API Response
|
||||
|
||||
**What's Returned**: Search results in requested format
|
||||
|
||||
**Format Options**:
|
||||
- `index` - Titles, dates, IDs only (~50-100 tokens per result)
|
||||
- `full` - Complete details (~500-1000 tokens per result)
|
||||
|
||||
**Progressive Usage**: Start with `index`, drill down with `full` as needed
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### Search Skill Structure
|
||||
|
||||
```
|
||||
plugin/skills/search/
|
||||
├── SKILL.md # Main frontmatter (~250 tokens)
|
||||
├── operations/
|
||||
│ ├── observations.md # Search observations
|
||||
│ ├── sessions.md # Search sessions
|
||||
│ ├── prompts.md # Search prompts
|
||||
│ ├── by-type.md # Filter by type
|
||||
│ ├── by-concept.md # Filter by concept
|
||||
│ ├── by-file.md # Filter by file
|
||||
│ ├── recent-context.md # Get recent context
|
||||
│ ├── timeline.md # Timeline around point
|
||||
│ ├── timeline-by-query.md # Search + timeline
|
||||
│ ├── help.md # API documentation
|
||||
│ ├── formatting.md # Result formatting guide
|
||||
│ └── common-workflows.md # Usage patterns
|
||||
```
|
||||
|
||||
### Worker Service Integration
|
||||
|
||||
**File**: `src/services/worker-service.ts`
|
||||
|
||||
**Search Routes**:
|
||||
```typescript
|
||||
// Full-text search
|
||||
app.get('/api/search/observations', handleSearchObservations);
|
||||
app.get('/api/search/sessions', handleSearchSessions);
|
||||
app.get('/api/search/prompts', handleSearchPrompts);
|
||||
|
||||
// Filtered search
|
||||
app.get('/api/search/by-type', handleSearchByType);
|
||||
app.get('/api/search/by-concept', handleSearchByConcept);
|
||||
app.get('/api/search/by-file', handleSearchByFile);
|
||||
|
||||
// Context retrieval
|
||||
app.get('/api/context/recent', handleRecentContext);
|
||||
app.get('/api/context/timeline', handleTimeline);
|
||||
app.get('/api/timeline/by-query', handleTimelineByQuery);
|
||||
|
||||
// Documentation
|
||||
app.get('/api/search/help', handleHelp);
|
||||
```
|
||||
|
||||
**Database Access**:
|
||||
- Uses `SessionSearch` service for FTS5 queries
|
||||
- Uses `SessionStore` for structured queries
|
||||
- Hybrid search with ChromaDB for semantic similarity
|
||||
|
||||
### Security
|
||||
|
||||
**FTS5 Injection Prevention** (v4.2.3):
|
||||
```typescript
|
||||
function escapeFTS5Query(query: string): string {
|
||||
return query.replace(/"/g, '""');
|
||||
}
|
||||
```
|
||||
|
||||
All user-provided search queries are properly escaped to prevent SQL injection.
|
||||
|
||||
**Comprehensive Testing**: 332 injection attack tests covering:
|
||||
- Special characters
|
||||
- SQL keywords
|
||||
- Quote escaping
|
||||
- Boolean operators
|
||||
|
||||
## Benefits
|
||||
|
||||
### 1. Token Efficiency
|
||||
|
||||
**Before (MCP)**:
|
||||
- Session start: ~2,500 tokens for tool definitions
|
||||
- Every session pays this cost
|
||||
- No progressive disclosure
|
||||
|
||||
**After (Skill)**:
|
||||
- Session start: ~250 tokens for skill frontmatter
|
||||
- Full instructions: ~2,500 tokens (only when invoked)
|
||||
- Net savings: ~2,250 tokens per session (~90% reduction)
|
||||
|
||||
### 2. Natural Language Interface
|
||||
|
||||
**Before**: Users needed to learn MCP tool syntax
|
||||
```
|
||||
search_observations with query="authentication" and type="decision"
|
||||
```
|
||||
|
||||
**After**: Users ask naturally
|
||||
```
|
||||
"What decisions did we make about authentication?"
|
||||
```
|
||||
|
||||
Claude translates to appropriate API call.
|
||||
|
||||
### 3. Flexibility
|
||||
|
||||
**HTTP API Benefits**:
|
||||
- Can be called from skills, MCP tools, or other clients
|
||||
- Easy to test with curl
|
||||
- Standard REST conventions
|
||||
- JSON responses
|
||||
|
||||
**Progressive Disclosure**:
|
||||
- Loads only what's needed
|
||||
- Can add more operations without increasing base cost
|
||||
- Documentation co-located with operations
|
||||
|
||||
### 4. Performance
|
||||
|
||||
**Fast Queries**: FTS5 full-text search <10ms for typical queries
|
||||
|
||||
**Caching**: HTTP layer allows response caching
|
||||
|
||||
**Pagination**: Efficient result pagination with offset/limit
|
||||
|
||||
## Migration Notes
|
||||
|
||||
### For Users
|
||||
|
||||
**No Action Required**: The migration from MCP to skill-based search is transparent.
|
||||
|
||||
**Same Questions Work**: Natural language queries work exactly the same way.
|
||||
|
||||
**Invisible Change**: Users won't notice any difference except better performance.
|
||||
|
||||
### For Developers
|
||||
|
||||
**Deprecated**: MCP search server (`src/servers/search-server.ts`)
|
||||
- Source file kept for reference
|
||||
- No longer built or registered
|
||||
- MCP configuration removed from `plugin/.mcp.json`
|
||||
|
||||
**New Implementation**: Skill-based search
|
||||
- Skill files: `plugin/skills/search/`
|
||||
- HTTP endpoints: `src/services/worker-service.ts` (lines 200-400)
|
||||
- Build script: `npm run build` includes skill files
|
||||
- Sync script: `npm run sync-marketplace` copies to plugin directory
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Worker Service Not Running
|
||||
|
||||
If searches fail, check worker service:
|
||||
|
||||
```bash
|
||||
pm2 list # Check status
|
||||
npm run worker:restart # Restart worker
|
||||
npm run worker:logs # View logs
|
||||
```
|
||||
|
||||
### HTTP Endpoints Not Responding
|
||||
|
||||
Test endpoints directly:
|
||||
|
||||
```bash
|
||||
# Health check
|
||||
curl http://localhost:37777/health
|
||||
|
||||
# Search test
|
||||
curl "http://localhost:37777/api/search/observations?query=test&limit=1"
|
||||
```
|
||||
|
||||
### Skill Not Invoking
|
||||
|
||||
If Claude doesn't invoke the skill:
|
||||
|
||||
1. Check skill files exist: `ls ~/.claude/plugins/marketplaces/thedotmack/plugin/skills/search/`
|
||||
2. Restart Claude Code session
|
||||
3. Try explicit skill invocation: `/skill search`
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [Search Tools Usage](/usage/search-tools) - User guide with examples
|
||||
- [Worker Service Architecture](/architecture/worker-service) - HTTP API details
|
||||
- [Database Schema](/architecture/database) - FTS5 tables and indexes
|
||||
@@ -0,0 +1,443 @@
|
||||
---
|
||||
title: "Worker Service"
|
||||
description: "HTTP API and PM2 process management"
|
||||
---
|
||||
|
||||
# Worker Service
|
||||
|
||||
The worker service is a long-running HTTP API built with Express.js and managed by PM2. It processes observations through the Claude Agent SDK separately from hook execution to prevent timeout issues.
|
||||
|
||||
## Overview
|
||||
|
||||
- **Technology**: Express.js HTTP server
|
||||
- **Process Manager**: PM2
|
||||
- **Port**: Fixed port 37777 (configurable via `CLAUDE_MEM_WORKER_PORT`)
|
||||
- **Location**: `src/services/worker-service.ts`
|
||||
- **Built Output**: `plugin/scripts/worker-service.cjs`
|
||||
- **Model**: Configurable via `CLAUDE_MEM_MODEL` environment variable (default: claude-sonnet-4-5)
|
||||
|
||||
## REST API Endpoints
|
||||
|
||||
The worker service exposes 14 HTTP endpoints organized into four categories:
|
||||
|
||||
### Viewer & Health Endpoints
|
||||
|
||||
#### 1. Viewer UI
|
||||
```
|
||||
GET /
|
||||
```
|
||||
|
||||
**Purpose**: Serves the web-based viewer UI (v5.1.0+)
|
||||
|
||||
**Response**: HTML page with embedded React application
|
||||
|
||||
**Features**:
|
||||
- Real-time memory stream visualization
|
||||
- Infinite scroll pagination
|
||||
- Project filtering
|
||||
- SSE-based live updates
|
||||
- Theme toggle (light/dark mode) as of v5.1.2
|
||||
|
||||
#### 2. Health Check
|
||||
```
|
||||
GET /health
|
||||
```
|
||||
|
||||
**Purpose**: Worker health status check
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"uptime": 12345,
|
||||
"port": 37777
|
||||
}
|
||||
```
|
||||
|
||||
#### 3. Server-Sent Events Stream
|
||||
```
|
||||
GET /stream
|
||||
```
|
||||
|
||||
**Purpose**: Real-time updates for viewer UI
|
||||
|
||||
**Response**: SSE stream with events:
|
||||
- `observation-created`: New observation added
|
||||
- `session-summary-created`: New summary generated
|
||||
- `user-prompt-created`: New prompt recorded
|
||||
|
||||
**Event Format**:
|
||||
```
|
||||
event: observation-created
|
||||
data: {"id": 123, "title": "...", ...}
|
||||
```
|
||||
|
||||
### Data Retrieval Endpoints
|
||||
|
||||
#### 4. Get Prompts
|
||||
```
|
||||
GET /api/prompts?project=my-project&limit=20&offset=0
|
||||
```
|
||||
|
||||
**Purpose**: Retrieve paginated user prompts
|
||||
|
||||
**Query Parameters**:
|
||||
- `project` (optional): Filter by project name
|
||||
- `limit` (default: 20): Number of results
|
||||
- `offset` (default: 0): Pagination offset
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"prompts": [{
|
||||
"id": 1,
|
||||
"session_id": "abc123",
|
||||
"prompt": "User's prompt text",
|
||||
"prompt_number": 1,
|
||||
"created_at": "2025-11-06T10:30:00Z"
|
||||
}],
|
||||
"total": 150,
|
||||
"hasMore": true
|
||||
}
|
||||
```
|
||||
|
||||
#### 5. Get Observations
|
||||
```
|
||||
GET /api/observations?project=my-project&limit=20&offset=0
|
||||
```
|
||||
|
||||
**Purpose**: Retrieve paginated observations
|
||||
|
||||
**Query Parameters**:
|
||||
- `project` (optional): Filter by project name
|
||||
- `limit` (default: 20): Number of results
|
||||
- `offset` (default: 0): Pagination offset
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"observations": [{
|
||||
"id": 123,
|
||||
"title": "Fix authentication bug",
|
||||
"type": "bugfix",
|
||||
"narrative": "...",
|
||||
"created_at": "2025-11-06T10:30:00Z"
|
||||
}],
|
||||
"total": 500,
|
||||
"hasMore": true
|
||||
}
|
||||
```
|
||||
|
||||
#### 6. Get Summaries
|
||||
```
|
||||
GET /api/summaries?project=my-project&limit=20&offset=0
|
||||
```
|
||||
|
||||
**Purpose**: Retrieve paginated session summaries
|
||||
|
||||
**Query Parameters**:
|
||||
- `project` (optional): Filter by project name
|
||||
- `limit` (default: 20): Number of results
|
||||
- `offset` (default: 0): Pagination offset
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"summaries": [{
|
||||
"id": 456,
|
||||
"session_id": "abc123",
|
||||
"request": "User's original request",
|
||||
"completed": "Work finished",
|
||||
"created_at": "2025-11-06T10:30:00Z"
|
||||
}],
|
||||
"total": 100,
|
||||
"hasMore": true
|
||||
}
|
||||
```
|
||||
|
||||
#### 7. Get Stats
|
||||
```
|
||||
GET /api/stats
|
||||
```
|
||||
|
||||
**Purpose**: Get database statistics by project
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"byProject": {
|
||||
"my-project": {
|
||||
"observations": 245,
|
||||
"summaries": 12,
|
||||
"prompts": 48
|
||||
},
|
||||
"other-project": {
|
||||
"observations": 156,
|
||||
"summaries": 8,
|
||||
"prompts": 32
|
||||
}
|
||||
},
|
||||
"total": {
|
||||
"observations": 401,
|
||||
"summaries": 20,
|
||||
"prompts": 80,
|
||||
"sessions": 20
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Settings Endpoints
|
||||
|
||||
#### 8. Get Settings
|
||||
```
|
||||
GET /api/settings
|
||||
```
|
||||
|
||||
**Purpose**: Retrieve user settings
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"sidebarOpen": true,
|
||||
"selectedProject": "my-project",
|
||||
"theme": "dark"
|
||||
}
|
||||
```
|
||||
|
||||
#### 9. Save Settings
|
||||
```
|
||||
POST /api/settings
|
||||
```
|
||||
|
||||
**Purpose**: Persist user settings
|
||||
|
||||
**Request Body**:
|
||||
```json
|
||||
{
|
||||
"sidebarOpen": false,
|
||||
"selectedProject": "other-project",
|
||||
"theme": "light"
|
||||
}
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"success": true
|
||||
}
|
||||
```
|
||||
|
||||
### Session Management Endpoints
|
||||
|
||||
#### 10. Initialize Session
|
||||
```
|
||||
POST /sessions/:sessionDbId/init
|
||||
```
|
||||
|
||||
**Request Body**:
|
||||
```json
|
||||
{
|
||||
"sdk_session_id": "abc-123",
|
||||
"project": "my-project"
|
||||
}
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"session_id": "abc-123"
|
||||
}
|
||||
```
|
||||
|
||||
#### 11. Add Observation
|
||||
```
|
||||
POST /sessions/:sessionDbId/observations
|
||||
```
|
||||
|
||||
**Request Body**:
|
||||
```json
|
||||
{
|
||||
"tool_name": "Read",
|
||||
"tool_input": {...},
|
||||
"tool_result": "...",
|
||||
"correlation_id": "xyz-789"
|
||||
}
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"observation_id": 123
|
||||
}
|
||||
```
|
||||
|
||||
#### 12. Generate Summary
|
||||
```
|
||||
POST /sessions/:sessionDbId/summarize
|
||||
```
|
||||
|
||||
**Request Body**:
|
||||
```json
|
||||
{
|
||||
"trigger": "stop"
|
||||
}
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"summary_id": 456
|
||||
}
|
||||
```
|
||||
|
||||
#### 13. Session Status
|
||||
```
|
||||
GET /sessions/:sessionDbId/status
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"session_id": "abc-123",
|
||||
"status": "active",
|
||||
"observation_count": 42,
|
||||
"summary_count": 1
|
||||
}
|
||||
```
|
||||
|
||||
#### 14. Delete Session
|
||||
```
|
||||
DELETE /sessions/:sessionDbId
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"success": true
|
||||
}
|
||||
```
|
||||
|
||||
**Note**: As of v4.1.0, the cleanup hook no longer calls this endpoint. Sessions are marked complete instead of deleted to allow graceful worker shutdown.
|
||||
|
||||
## PM2 Management
|
||||
|
||||
### Configuration
|
||||
|
||||
The worker is configured via `ecosystem.config.cjs`:
|
||||
|
||||
```javascript
|
||||
module.exports = {
|
||||
apps: [{
|
||||
name: 'claude-mem-worker',
|
||||
script: './plugin/scripts/worker-service.cjs',
|
||||
instances: 1,
|
||||
autorestart: true,
|
||||
watch: false,
|
||||
max_memory_restart: '1G',
|
||||
env: {
|
||||
NODE_ENV: 'production',
|
||||
FORCE_COLOR: '1'
|
||||
}
|
||||
}]
|
||||
};
|
||||
```
|
||||
|
||||
### Commands
|
||||
|
||||
```bash
|
||||
# Start worker (auto-starts on first session)
|
||||
npm run worker:start
|
||||
|
||||
# Stop worker
|
||||
npm run worker:stop
|
||||
|
||||
# Restart worker
|
||||
npm run worker:restart
|
||||
|
||||
# View logs
|
||||
npm run worker:logs
|
||||
|
||||
# Check status
|
||||
npm run worker:status
|
||||
```
|
||||
|
||||
### Auto-Start Behavior
|
||||
|
||||
As of v4.0.0, the worker service auto-starts when the SessionStart hook fires. Manual start is optional.
|
||||
|
||||
## Claude Agent SDK Integration
|
||||
|
||||
The worker service routes observations to the Claude Agent SDK for AI-powered processing:
|
||||
|
||||
### Processing Flow
|
||||
|
||||
1. **Observation Queue**: Observations accumulate in memory
|
||||
2. **SDK Processing**: Observations sent to Claude via Agent SDK
|
||||
3. **XML Parsing**: Responses parsed for structured data
|
||||
4. **Database Storage**: Processed observations stored in SQLite
|
||||
|
||||
### SDK Components
|
||||
|
||||
- **Prompts** (`src/sdk/prompts.ts`): Builds XML-structured prompts
|
||||
- **Parser** (`src/sdk/parser.ts`): Parses Claude's XML responses
|
||||
- **Worker** (`src/sdk/worker.ts`): Main SDK agent loop
|
||||
|
||||
### Model Configuration
|
||||
|
||||
Set the AI model used for processing via environment variable:
|
||||
|
||||
```bash
|
||||
export CLAUDE_MEM_MODEL=claude-sonnet-4-5
|
||||
```
|
||||
|
||||
Available models:
|
||||
- `claude-haiku-4-5` - Fast, cost-efficient
|
||||
- `claude-sonnet-4-5` - Balanced (default)
|
||||
- `claude-opus-4` - Most capable
|
||||
- `claude-3-7-sonnet` - Alternative version
|
||||
|
||||
## Port Allocation
|
||||
|
||||
The worker uses a fixed port (37777 by default) for consistent communication:
|
||||
|
||||
- **Default**: Port 37777
|
||||
- **Override**: Set `CLAUDE_MEM_WORKER_PORT` environment variable
|
||||
- **Port File**: `${CLAUDE_PLUGIN_ROOT}/data/worker.port` tracks current port
|
||||
|
||||
If port 37777 is in use, the worker will fail to start. Set a custom port via environment variable.
|
||||
|
||||
## Data Storage
|
||||
|
||||
The worker service stores data in the plugin data directory:
|
||||
|
||||
```
|
||||
${CLAUDE_PLUGIN_ROOT}/data/
|
||||
├── claude-mem.db # SQLite database
|
||||
├── worker.port # Current worker port file
|
||||
└── logs/
|
||||
├── worker-out.log # Worker stdout logs
|
||||
└── worker-error.log # Worker stderr logs
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
The worker implements graceful degradation:
|
||||
|
||||
- **Database Errors**: Logged but don't crash the service
|
||||
- **SDK Errors**: Retried with exponential backoff
|
||||
- **Network Errors**: Logged and skipped
|
||||
- **Invalid Input**: Validated and rejected with error response
|
||||
|
||||
## Performance
|
||||
|
||||
- **Async Processing**: Observations processed asynchronously
|
||||
- **In-Memory Queue**: Fast observation accumulation
|
||||
- **Batch Processing**: Multiple observations processed together
|
||||
- **Connection Pooling**: SQLite connections reused
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
See [Troubleshooting - Worker Issues](../troubleshooting.md#worker-service-issues) for common problems and solutions.
|
||||
Reference in New Issue
Block a user