docs: Update all documentation to reflect v5.4.0 skill-based search
Documentation Updates: - README.md: Updated version badge, What's New, and search section - docs/usage/search-tools.mdx: Rewrote for skill-based natural language approach - docs/architecture/mcp-search.mdx → search-architecture.mdx: Complete rewrite - docs/architecture/overview.mdx: Updated components and search pipeline - docs/usage/getting-started.mdx: Added skill-based search section - docs/configuration.mdx: Updated search configuration for v5.4.0 - docs/introduction.mdx: Updated key features - docs/docs.json: Updated navigation to search-architecture Key Changes: - Emphasized ~2,250 token savings per session start - Converted all examples to natural language queries - Documented 10 HTTP API endpoints - Explained progressive disclosure pattern - Added migration notes (transparent, no user action required) - Removed outdated MCP references 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Alex Newman
@@ -1,447 +0,0 @@
|
||||
---
|
||||
title: "MCP Search Server"
|
||||
description: "9 search tools with examples and usage patterns"
|
||||
---
|
||||
|
||||
# MCP Search Server
|
||||
|
||||
Claude-Mem includes a Model Context Protocol (MCP) server that exposes 9 specialized search tools for querying stored observations and sessions.
|
||||
|
||||
## Overview
|
||||
|
||||
- **Location**: `src/servers/search-server.ts`
|
||||
- **Built Output**: `plugin/scripts/search-server.mjs`
|
||||
- **Configuration**: `plugin/.mcp.json`
|
||||
- **Transport**: stdio
|
||||
- **Tools**: 9 specialized search functions
|
||||
- **Citations**: All results use `claude-mem://` URI scheme
|
||||
|
||||
## Configuration
|
||||
|
||||
The MCP server is automatically registered via `plugin/.mcp.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"claude-mem-search": {
|
||||
"type": "stdio",
|
||||
"command": "${CLAUDE_PLUGIN_ROOT}/scripts/search-server.mjs"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This registers the `claude-mem-search` server with Claude Code, making the 9 search tools available in all sessions. The server is automatically started when Claude Code launches and communicates via stdio transport.
|
||||
|
||||
## Search Tools
|
||||
|
||||
### 1. search_observations
|
||||
|
||||
Full-text search across observation titles, narratives, facts, and concepts.
|
||||
|
||||
**Parameters**:
|
||||
- `query` (required): Search query for FTS5 full-text search
|
||||
- `type`: Filter by observation type(s) (decision, bugfix, feature, refactor, discovery, change)
|
||||
- `concepts`: Filter by concept tags
|
||||
- `files`: Filter by file paths (partial match)
|
||||
- `project`: Filter by project name
|
||||
- `dateRange`: Filter by date range (`{start, end}`)
|
||||
- `orderBy`: Sort order (relevance, date_desc, date_asc)
|
||||
- `limit`: Maximum results (default: 20, max: 100)
|
||||
- `offset`: Number of results to skip
|
||||
- `format`: Output format ("index" for titles/dates only, "full" for complete details)
|
||||
|
||||
**Example**:
|
||||
```
|
||||
search_observations with query="build system" and type="decision"
|
||||
```
|
||||
|
||||
### 2. search_sessions
|
||||
|
||||
Full-text search across session summaries, requests, and learnings.
|
||||
|
||||
**Parameters**:
|
||||
- `query` (required): Search query for FTS5 full-text search
|
||||
- `project`: Filter by project name
|
||||
- `dateRange`: Filter by date range
|
||||
- `orderBy`: Sort order (relevance, date_desc, date_asc)
|
||||
- `limit`: Maximum results (default: 20, max: 100)
|
||||
- `offset`: Number of results to skip
|
||||
- `format`: Output format ("index" or "full")
|
||||
|
||||
**Example**:
|
||||
```
|
||||
search_sessions with query="hooks implementation"
|
||||
```
|
||||
|
||||
### 3. search_user_prompts
|
||||
|
||||
Search raw user prompts with full-text search. Use this to find what the user actually said/requested across all sessions.
|
||||
|
||||
**Parameters**:
|
||||
- `query` (required): Search query for FTS5 full-text search
|
||||
- `project`: Filter by project name
|
||||
- `dateRange`: Filter by date range
|
||||
- `orderBy`: Sort order (relevance, date_desc, date_asc)
|
||||
- `limit`: Maximum results (default: 20, max: 100)
|
||||
- `offset`: Number of results to skip
|
||||
- `format`: Output format ("index" for truncated prompts/dates, "full" for complete prompt text)
|
||||
|
||||
**Example**:
|
||||
```
|
||||
search_user_prompts with query="authentication feature"
|
||||
```
|
||||
|
||||
**Benefits**:
|
||||
- Full context reconstruction from user intent → Claude actions → outcomes
|
||||
- Pattern detection for repeated requests
|
||||
- Improved debugging by tracing from original user words to final implementation
|
||||
|
||||
### 4. find_by_concept
|
||||
|
||||
Find observations tagged with specific concepts.
|
||||
|
||||
**Parameters**:
|
||||
- `concept` (required): Concept tag to search for
|
||||
- `project`: Filter by project name
|
||||
- `dateRange`: Filter by date range
|
||||
- `orderBy`: Sort order (relevance, date_desc, date_asc)
|
||||
- `limit`: Maximum results (default: 20, max: 100)
|
||||
- `offset`: Number of results to skip
|
||||
- `format`: Output format ("index" or "full")
|
||||
|
||||
**Example**:
|
||||
```
|
||||
find_by_concept with concept="architecture"
|
||||
```
|
||||
|
||||
### 5. find_by_file
|
||||
|
||||
Find observations and sessions that reference specific file paths.
|
||||
|
||||
**Parameters**:
|
||||
- `filePath` (required): File path to search for (supports partial matching)
|
||||
- `project`: Filter by project name
|
||||
- `dateRange`: Filter by date range
|
||||
- `orderBy`: Sort order (relevance, date_desc, date_asc)
|
||||
- `limit`: Maximum results (default: 20, max: 100)
|
||||
- `offset`: Number of results to skip
|
||||
- `format`: Output format ("index" or "full")
|
||||
|
||||
**Example**:
|
||||
```
|
||||
find_by_file with filePath="worker-service.ts"
|
||||
```
|
||||
|
||||
### 6. find_by_type
|
||||
|
||||
Find observations by type (decision, bugfix, feature, refactor, discovery, change).
|
||||
|
||||
**Parameters**:
|
||||
- `type` (required): Observation type(s) to filter by (single type or array)
|
||||
- `project`: Filter by project name
|
||||
- `dateRange`: Filter by date range
|
||||
- `orderBy`: Sort order (relevance, date_desc, date_asc)
|
||||
- `limit`: Maximum results (default: 20, max: 100)
|
||||
- `offset`: Number of results to skip
|
||||
- `format`: Output format ("index" or "full")
|
||||
|
||||
**Example**:
|
||||
```
|
||||
find_by_type with type=["decision", "feature"]
|
||||
```
|
||||
|
||||
### 7. get_recent_context
|
||||
|
||||
Get recent session context including summaries and observations for a project.
|
||||
|
||||
**Parameters**:
|
||||
- `project`: Project name (defaults to current working directory basename)
|
||||
- `limit`: Number of recent sessions to retrieve (default: 3, max: 10)
|
||||
|
||||
**Example**:
|
||||
```
|
||||
get_recent_context with limit=5
|
||||
```
|
||||
|
||||
### 8. get_context_timeline
|
||||
|
||||
Get a unified timeline of context (observations, sessions, and prompts) around a specific point in time. All record types are interleaved chronologically.
|
||||
|
||||
**Parameters**:
|
||||
- `anchor` (required): Anchor point - observation ID, session ID (e.g., "S123"), or ISO timestamp
|
||||
- `depth_before` (default: 10): Number of records to retrieve before anchor (max: 50)
|
||||
- `depth_after` (default: 10): Number of records to retrieve after anchor (max: 50)
|
||||
- `project`: Filter by project name
|
||||
|
||||
**Return Format**:
|
||||
Returns `depth_before` records + anchor + `depth_after` records, all interleaved chronologically. Total records: `depth_before + 1 + depth_after`.
|
||||
|
||||
**Use Case**: Understanding "what was happening when X occurred"
|
||||
|
||||
**Example**:
|
||||
```
|
||||
# Timeline around observation #123
|
||||
get_context_timeline with anchor=123 and depth_before=5 and depth_after=5
|
||||
|
||||
# Timeline around a session
|
||||
get_context_timeline with anchor="S456" and depth_before=10 and depth_after=10
|
||||
|
||||
# Timeline around a timestamp
|
||||
get_context_timeline with anchor="2025-11-06T10:30:00Z" and depth_before=15 and depth_after=5
|
||||
```
|
||||
|
||||
**Response Structure**:
|
||||
```json
|
||||
{
|
||||
"timeline": [
|
||||
{
|
||||
"type": "observation",
|
||||
"id": 120,
|
||||
"title": "Context before",
|
||||
"created_at": "2025-11-06T10:25:00Z"
|
||||
},
|
||||
{
|
||||
"type": "user-prompt",
|
||||
"id": 45,
|
||||
"prompt": "User request",
|
||||
"created_at": "2025-11-06T10:28:00Z"
|
||||
},
|
||||
{
|
||||
"type": "observation",
|
||||
"id": 123,
|
||||
"title": "Anchor observation",
|
||||
"created_at": "2025-11-06T10:30:00Z",
|
||||
"isAnchor": true
|
||||
},
|
||||
{
|
||||
"type": "session",
|
||||
"id": "S456",
|
||||
"request": "Session summary",
|
||||
"created_at": "2025-11-06T10:32:00Z"
|
||||
}
|
||||
],
|
||||
"anchor": {
|
||||
"type": "observation",
|
||||
"id": 123
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 9. get_timeline_by_query
|
||||
|
||||
Search for observations using natural language and get timeline context around the best match. Combines search + timeline into a single operation.
|
||||
|
||||
**Parameters**:
|
||||
- `query` (required): Natural language search query to find relevant observations
|
||||
- `mode` (default: "auto"): Operation mode
|
||||
- `"auto"`: Automatically use top search result as timeline anchor
|
||||
- `"interactive"`: Return top N search results for manual anchor selection
|
||||
- `depth_before` (default: 10): Number of timeline records before anchor (max: 50)
|
||||
- `depth_after` (default: 10): Number of timeline records after anchor (max: 50)
|
||||
- `limit` (default: 5): For interactive mode - number of top search results to display (max: 20)
|
||||
- `project`: Filter by project name
|
||||
|
||||
**Use Case**: Faster context discovery - "show me what happened around when we fixed the authentication bug"
|
||||
|
||||
**Example - Auto Mode**:
|
||||
```
|
||||
# Automatically find and show timeline for "authentication bug"
|
||||
get_timeline_by_query with query="authentication bug" and mode="auto" and depth_before=10 and depth_after=10
|
||||
```
|
||||
|
||||
**Example - Interactive Mode**:
|
||||
```
|
||||
# Show top 5 matches, let user choose anchor
|
||||
get_timeline_by_query with query="authentication bug" and mode="interactive" and limit=5
|
||||
```
|
||||
|
||||
**Auto Mode Response**:
|
||||
```json
|
||||
{
|
||||
"search_result": {
|
||||
"id": 123,
|
||||
"title": "Fix authentication bug",
|
||||
"relevance": 0.95
|
||||
},
|
||||
"timeline": [
|
||||
/* timeline records before and after observation 123 */
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Interactive Mode Response**:
|
||||
```json
|
||||
{
|
||||
"top_results": [
|
||||
{
|
||||
"id": 123,
|
||||
"title": "Fix authentication bug",
|
||||
"relevance": 0.95,
|
||||
"created_at": "2025-11-06T10:30:00Z"
|
||||
},
|
||||
{
|
||||
"id": 98,
|
||||
"title": "Authentication refactor",
|
||||
"relevance": 0.82,
|
||||
"created_at": "2025-11-05T14:20:00Z"
|
||||
}
|
||||
],
|
||||
"message": "Select an observation ID to view its timeline context"
|
||||
}
|
||||
```
|
||||
|
||||
## Output Formats
|
||||
|
||||
All search tools support two output formats:
|
||||
|
||||
### Index Format (Default)
|
||||
|
||||
Returns titles, dates, and source URIs only. Uses ~10x fewer tokens than full format.
|
||||
|
||||
**Always use index format first** to get an overview and identify relevant results.
|
||||
|
||||
**Example Output**:
|
||||
```
|
||||
1. [decision] Implement graceful session cleanup
|
||||
Date: 2025-10-21 14:23:45
|
||||
Source: claude-mem://observation/123
|
||||
|
||||
2. [feature] Add FTS5 full-text search
|
||||
Date: 2025-10-21 13:15:22
|
||||
Source: claude-mem://observation/124
|
||||
```
|
||||
|
||||
### Full Format
|
||||
|
||||
Returns complete observation/summary details including narrative, facts, concepts, files, etc.
|
||||
|
||||
**Only use after reviewing index results** to dive deep into specific items of interest.
|
||||
|
||||
## Search Strategy
|
||||
|
||||
**Recommended Workflow**:
|
||||
|
||||
1. **Initial search**: Use default (index) format to see titles, dates, and sources
|
||||
2. **Review results**: Identify which items are most relevant to your needs
|
||||
3. **Deep dive**: Only then use `format: "full"` on specific items of interest
|
||||
4. **Narrow down**: Use filters (type, dateRange, concepts, files) to refine results
|
||||
|
||||
**Token Efficiency**:
|
||||
- Index format: ~50-100 tokens per result
|
||||
- Full format: ~500-1000 tokens per result
|
||||
- Start with 3-5 results to avoid MCP token limits
|
||||
|
||||
## Citations
|
||||
|
||||
All search results use the `claude-mem://` URI scheme for citations:
|
||||
|
||||
- `claude-mem://observation/{id}` - References specific observations
|
||||
- `claude-mem://session/{id}` - References specific sessions
|
||||
- `claude-mem://user-prompt/{id}` - References specific user prompts
|
||||
|
||||
These citations allow Claude to reference specific historical context in responses.
|
||||
|
||||
## FTS5 Query Syntax
|
||||
|
||||
The `query` parameter supports SQLite FTS5 full-text search syntax:
|
||||
|
||||
- **Simple**: `"error handling"`
|
||||
- **AND**: `"error" AND "handling"`
|
||||
- **OR**: `"bug" OR "fix"`
|
||||
- **NOT**: `"bug" NOT "feature"`
|
||||
- **Phrase**: `"'exact phrase'"`
|
||||
- **Column**: `title:"authentication"`
|
||||
|
||||
## Security
|
||||
|
||||
As of v4.2.3, all FTS5 queries are properly escaped to prevent SQL injection attacks:
|
||||
- Double quotes are escaped: `query.replace(/"/g, '""')`
|
||||
- Comprehensive test suite with 332 injection attack tests
|
||||
- Affects: `search_observations`, `search_sessions`, `search_user_prompts`
|
||||
|
||||
## Example Queries
|
||||
|
||||
```
|
||||
# Find all decisions about build system
|
||||
search_observations with query="build system" and type="decision"
|
||||
|
||||
# Show everything related to worker-service.ts
|
||||
find_by_file with filePath="worker-service.ts"
|
||||
|
||||
# Search what we learned about hooks
|
||||
search_sessions with query="hooks"
|
||||
|
||||
# Show observations tagged with 'architecture'
|
||||
find_by_concept with concept="architecture"
|
||||
|
||||
# Find what user asked about authentication
|
||||
search_user_prompts with query="authentication"
|
||||
|
||||
# Get recent context for debugging
|
||||
get_recent_context with limit=5
|
||||
|
||||
# Timeline around a specific observation
|
||||
get_context_timeline with anchor=123 and depth_before=10 and depth_after=10
|
||||
|
||||
# Quick timeline search for authentication work
|
||||
get_timeline_by_query with query="authentication bug" and mode="auto"
|
||||
```
|
||||
|
||||
## Implementation
|
||||
|
||||
The MCP search server is implemented using:
|
||||
- `@modelcontextprotocol/sdk` (v1.20.1)
|
||||
- `SessionSearch` service for FTS5 queries
|
||||
- `SessionStore` for database access
|
||||
- `zod-to-json-schema` for parameter validation
|
||||
|
||||
**Source Code**: `src/servers/search-server.ts`
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Tool Not Available
|
||||
|
||||
If search tools are not available in Claude Code sessions:
|
||||
|
||||
1. Check MCP configuration:
|
||||
```bash
|
||||
cat plugin/.mcp.json
|
||||
```
|
||||
|
||||
2. Verify search server is built:
|
||||
```bash
|
||||
ls -l plugin/scripts/search-server.mjs
|
||||
```
|
||||
|
||||
3. Rebuild if needed:
|
||||
```bash
|
||||
npm run build
|
||||
```
|
||||
|
||||
### Search Returns No Results
|
||||
|
||||
1. Check database has data:
|
||||
```bash
|
||||
sqlite3 ~/.claude-mem/claude-mem.db "SELECT COUNT(*) FROM observations;"
|
||||
```
|
||||
|
||||
2. Verify FTS5 tables exist:
|
||||
```bash
|
||||
sqlite3 ~/.claude-mem/claude-mem.db "SELECT name FROM sqlite_master WHERE type='table' AND name LIKE '%_fts';"
|
||||
```
|
||||
|
||||
3. Test query syntax:
|
||||
```bash
|
||||
# Simple query should work
|
||||
search_observations with query="test"
|
||||
```
|
||||
|
||||
### Token Limit Errors
|
||||
|
||||
If you hit MCP token limits:
|
||||
|
||||
1. Use `format: "index"` instead of `format: "full"`
|
||||
2. Reduce `limit` parameter (try 3-5 instead of 20)
|
||||
3. Use more specific filters to narrow results
|
||||
4. Use `offset` to paginate through results
|
||||
@@ -10,9 +10,9 @@ description: "System components and data flow in Claude-Mem"
|
||||
Claude-Mem operates as a Claude Code plugin with five core components:
|
||||
|
||||
1. **Plugin Hooks** - Capture lifecycle events (7 hook files)
|
||||
2. **Worker Service** - Process observations via Claude Agent SDK
|
||||
3. **Database Layer** - Store sessions and observations (SQLite + FTS5)
|
||||
4. **MCP Search Server** - Query historical context (9 search tools)
|
||||
2. **Worker Service** - Process observations via Claude Agent SDK + HTTP API (10 search endpoints)
|
||||
3. **Database Layer** - Store sessions and observations (SQLite + FTS5 + ChromaDB)
|
||||
4. **Search Skill** - Skill-based search with progressive disclosure (v5.4.0+)
|
||||
5. **Viewer UI** - Web-based real-time memory stream visualization
|
||||
|
||||
## Technology Stack
|
||||
@@ -44,16 +44,19 @@ Hook (stdin) → Database → Worker Service → SDK Processor → Database →
|
||||
4. **Output**: Processed summaries written back to database
|
||||
5. **Retrieval**: Next session's context hook reads summaries from database
|
||||
|
||||
### Search Pipeline
|
||||
### Search Pipeline (v5.4.0+)
|
||||
```
|
||||
Claude Request → MCP Server → SessionSearch Service → FTS5 Database → Search Results → Claude
|
||||
User Query → Skill Invoked → HTTP API → SessionSearch Service → FTS5 Database → Search Results → Claude
|
||||
```
|
||||
|
||||
1. **Query**: Claude uses MCP search tools (e.g., `search_observations`)
|
||||
2. **Search**: MCP server calls SessionSearch service with query parameters
|
||||
3. **FTS5**: Full-text search executes against FTS5 virtual tables
|
||||
4. **Format**: Results formatted as `search_result` blocks with citations
|
||||
5. **Return**: Claude receives citable search results for analysis
|
||||
1. **User Query**: User asks naturally: "What bugs did we fix?"
|
||||
2. **Skill Invoked**: Claude recognizes intent and invokes search skill
|
||||
3. **HTTP API**: Skill uses curl to call HTTP endpoint (e.g., `/api/search/observations`)
|
||||
4. **SessionSearch**: Worker service queries FTS5 virtual tables
|
||||
5. **Format**: Results formatted and returned to skill
|
||||
6. **Return**: Claude presents formatted results to user
|
||||
|
||||
**Token Savings**: ~2,250 tokens per session vs MCP approach through progressive disclosure
|
||||
|
||||
## Session Lifecycle
|
||||
|
||||
@@ -110,9 +113,6 @@ claude-mem/
|
||||
│ │ ├── summary-hook.ts # Stop
|
||||
│ │ └── cleanup-hook.ts # SessionEnd
|
||||
│ │
|
||||
│ ├── servers/ # MCP servers
|
||||
│ │ └── search-server.ts # MCP search tools server (9 tools)
|
||||
│ │
|
||||
│ ├── sdk/ # Claude Agent SDK integration
|
||||
│ │ ├── prompts.ts # XML prompt builders
|
||||
│ │ ├── parser.ts # XML response parser
|
||||
@@ -146,7 +146,6 @@ claude-mem/
|
||||
├── plugin/ # Plugin distribution
|
||||
│ ├── .claude-plugin/
|
||||
│ │ └── plugin.json
|
||||
│ ├── .mcp.json # MCP server configuration
|
||||
│ ├── hooks/
|
||||
│ │ └── hooks.json
|
||||
│ ├── scripts/ # Built executables
|
||||
@@ -157,8 +156,14 @@ claude-mem/
|
||||
│ │ ├── save-hook.js
|
||||
│ │ ├── summary-hook.js
|
||||
│ │ ├── cleanup-hook.js
|
||||
│ │ ├── worker-service.cjs # Background worker
|
||||
│ │ └── search-server.mjs # MCP search server
|
||||
│ │ └── worker-service.cjs # Background worker + HTTP API
|
||||
│ │
|
||||
│ ├── skills/ # Agent skills (v5.4.0+)
|
||||
│ │ ├── search/ # Search skill with progressive disclosure
|
||||
│ │ │ ├── SKILL.md # Skill frontmatter (~250 tokens)
|
||||
│ │ │ └── operations/ # Detailed operation docs
|
||||
│ │ ├── troubleshoot/ # Troubleshooting skill
|
||||
│ │ └── version-bump/ # Version management skill
|
||||
│ │
|
||||
│ └── ui/ # Built viewer UI
|
||||
│ └── viewer.html # Self-contained bundle
|
||||
@@ -183,7 +188,8 @@ See [Plugin Hooks](/architecture/hooks) for detailed hook documentation.
|
||||
|
||||
### 2. Worker Service
|
||||
Express.js HTTP server on port 37777 (configurable) with:
|
||||
- 8 HTTP/SSE endpoints for viewer UI
|
||||
- 10 search HTTP API endpoints (v5.4.0+)
|
||||
- 8 viewer UI HTTP/SSE endpoints
|
||||
- Async observation processing via Claude Agent SDK
|
||||
- Real-time updates via Server-Sent Events
|
||||
- Auto-managed by PM2 process manager
|
||||
@@ -199,13 +205,19 @@ SQLite3 with better-sqlite3 driver featuring:
|
||||
|
||||
See [Database Architecture](/architecture/database) for schema and FTS5 search.
|
||||
|
||||
### 4. MCP Search Server (9 Tools)
|
||||
Provides 9 specialized search tools:
|
||||
- search_observations, search_sessions, search_user_prompts
|
||||
- find_by_concept, find_by_file, find_by_type
|
||||
- get_recent_context, get_context_timeline, get_timeline_by_query
|
||||
### 4. Search Skill (v5.4.0+)
|
||||
Skill-based search with progressive disclosure providing 10 search operations:
|
||||
- Search observations, sessions, prompts (full-text FTS5)
|
||||
- Filter by type, concept, file
|
||||
- Get recent context, timeline, timeline by query
|
||||
- API help documentation
|
||||
|
||||
See [MCP Search Server](/architecture/mcp-search) for search tools and examples.
|
||||
**Token Savings**: ~2,250 tokens per session vs MCP approach
|
||||
- Skill frontmatter: ~250 tokens (loaded at session start)
|
||||
- Full instructions: ~2,500 tokens (loaded on-demand when invoked)
|
||||
- HTTP API endpoints instead of MCP tools
|
||||
|
||||
See [Search Architecture](/architecture/search-architecture) for technical details and examples.
|
||||
|
||||
### 5. Viewer UI
|
||||
React + TypeScript web interface at http://localhost:37777 featuring:
|
||||
|
||||
@@ -0,0 +1,440 @@
|
||||
---
|
||||
title: "Search Architecture"
|
||||
description: "Skill-based search with HTTP API and progressive disclosure"
|
||||
---
|
||||
|
||||
# Search Architecture
|
||||
|
||||
Claude-Mem uses a skill-based search architecture that provides intelligent memory retrieval through natural language queries. This replaced the MCP-based approach in v5.4.0, saving ~2,250 tokens per session start.
|
||||
|
||||
## Overview
|
||||
|
||||
**Architecture**: Skill-Based Search + HTTP API + Progressive Disclosure
|
||||
|
||||
**Key Components**:
|
||||
1. **Search Skill** (`plugin/skills/search/SKILL.md`) - Auto-invoked when users ask about past work
|
||||
2. **HTTP API Endpoints** (10 routes) - Fast, efficient search operations on port 37777
|
||||
3. **Worker Service** - Express.js server with FTS5 full-text search
|
||||
4. **SQLite Database** - Persistent storage with FTS5 virtual tables
|
||||
5. **Chroma Vector DB** - Semantic search with hybrid retrieval
|
||||
|
||||
## How It Works
|
||||
|
||||
### 1. User Query (Natural Language)
|
||||
|
||||
```
|
||||
User: "What bugs did we fix last session?"
|
||||
```
|
||||
|
||||
### 2. Skill Invocation
|
||||
|
||||
Claude recognizes the intent and invokes the search skill:
|
||||
- Skill frontmatter (~250 tokens) loaded at session start
|
||||
- Full skill instructions loaded on-demand when skill is invoked
|
||||
- Progressive disclosure pattern minimizes context overhead
|
||||
|
||||
### 3. HTTP API Call
|
||||
|
||||
The skill uses `curl` to call the HTTP API:
|
||||
|
||||
```bash
|
||||
curl "http://localhost:37777/api/search/observations?query=bugs&type=bugfix&limit=5"
|
||||
```
|
||||
|
||||
### 4. FTS5 Search
|
||||
|
||||
Worker service queries SQLite FTS5 virtual tables:
|
||||
|
||||
```sql
|
||||
SELECT * FROM observations_fts
|
||||
WHERE observations_fts MATCH ?
|
||||
AND type = 'bugfix'
|
||||
ORDER BY rank
|
||||
LIMIT 5
|
||||
```
|
||||
|
||||
### 5. Results Formatted
|
||||
|
||||
Skill formats results and returns to Claude:
|
||||
|
||||
```
|
||||
## Recent Bugfixes
|
||||
|
||||
1. [bugfix] Fixed authentication token expiry
|
||||
Date: 2025-11-08 14:23:45
|
||||
Files: src/auth/jwt.ts
|
||||
|
||||
2. [bugfix] Resolved database connection leak
|
||||
Date: 2025-11-08 13:15:22
|
||||
Files: src/services/database.ts
|
||||
```
|
||||
|
||||
### 6. User Sees Answer
|
||||
|
||||
Claude presents the formatted results naturally in conversation.
|
||||
|
||||
## Architecture Change (v5.4.0)
|
||||
|
||||
### Before: MCP-Based Search
|
||||
|
||||
**Approach**: 9 MCP tools registered at session start
|
||||
|
||||
**Token Cost**: ~2,500 tokens in tool definitions per session
|
||||
- Each tool's schema, parameters, descriptions loaded
|
||||
- All 9 tools available whether needed or not
|
||||
- No progressive disclosure
|
||||
|
||||
**Example MCP Tool**:
|
||||
```json
|
||||
{
|
||||
"name": "search_observations",
|
||||
"description": "Full-text search across observations...",
|
||||
"inputSchema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"query": { "type": "string", "description": "..." },
|
||||
"type": { "type": "array", "items": { "enum": [...] } },
|
||||
"format": { "enum": ["index", "full"] },
|
||||
// ... many more parameters
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### After: Skill-Based Search
|
||||
|
||||
**Approach**: 1 search skill with progressive disclosure
|
||||
|
||||
**Token Cost**: ~250 tokens in skill frontmatter per session
|
||||
- Only skill description loaded at session start
|
||||
- Full instructions loaded on-demand when skill is invoked
|
||||
- HTTP API endpoints instead of MCP protocol
|
||||
|
||||
**Example Skill Frontmatter**:
|
||||
```markdown
|
||||
# Claude-Mem Search Skill
|
||||
|
||||
Access claude-mem's persistent memory through a comprehensive HTTP API.
|
||||
Search for past work, understand context, and learn from previous decisions.
|
||||
|
||||
## When to Use This Skill
|
||||
|
||||
Invoke this skill when users ask about:
|
||||
- Past work: "What did we do last session?"
|
||||
- Bug fixes: "Did we fix this before?"
|
||||
- Features: "How did we implement authentication?"
|
||||
...
|
||||
```
|
||||
|
||||
**Token Savings**: ~2,250 tokens per session start (90% reduction)
|
||||
|
||||
## HTTP API Endpoints
|
||||
|
||||
The worker service exposes 10 search endpoints:
|
||||
|
||||
### Full-Text Search
|
||||
|
||||
```
|
||||
GET /api/search/observations
|
||||
GET /api/search/sessions
|
||||
GET /api/search/prompts
|
||||
```
|
||||
|
||||
**Parameters**:
|
||||
- `query` - FTS5 search query (required)
|
||||
- `type` - Filter by type (bugfix, feature, refactor, etc.)
|
||||
- `project` - Filter by project name
|
||||
- `limit` - Maximum results (default: 20)
|
||||
- `offset` - Pagination offset
|
||||
- `format` - Response format (index or full)
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
curl "http://localhost:37777/api/search/observations?query=authentication&type=decision&limit=5"
|
||||
```
|
||||
|
||||
### Filtered Search
|
||||
|
||||
```
|
||||
GET /api/search/by-type
|
||||
GET /api/search/by-concept
|
||||
GET /api/search/by-file
|
||||
```
|
||||
|
||||
**Parameters**:
|
||||
- `type` / `concept` / `filePath` - Filter criteria (required)
|
||||
- `project` - Filter by project
|
||||
- `limit` - Maximum results
|
||||
- `format` - Response format
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
curl "http://localhost:37777/api/search/by-file?filePath=worker-service.ts&limit=10"
|
||||
```
|
||||
|
||||
### Context Retrieval
|
||||
|
||||
```
|
||||
GET /api/context/recent
|
||||
GET /api/context/timeline
|
||||
GET /api/timeline/by-query
|
||||
```
|
||||
|
||||
**Parameters**:
|
||||
- `project` - Filter by project
|
||||
- `limit` - Number of sessions/records
|
||||
- `anchor` - Timeline anchor point (ID or timestamp)
|
||||
- `depth_before` - Records before anchor
|
||||
- `depth_after` - Records after anchor
|
||||
|
||||
**Example**:
|
||||
```bash
|
||||
curl "http://localhost:37777/api/context/recent?project=claude-mem&limit=5"
|
||||
```
|
||||
|
||||
### Documentation
|
||||
|
||||
```
|
||||
GET /api/search/help
|
||||
```
|
||||
|
||||
Returns API documentation in JSON format.
|
||||
|
||||
## Progressive Disclosure Pattern
|
||||
|
||||
The search skill uses progressive disclosure to minimize token usage:
|
||||
|
||||
### Layer 1: Skill Frontmatter (Session Start)
|
||||
|
||||
**What's Loaded**: Skill description and when to use it (~250 tokens)
|
||||
|
||||
**Purpose**: Claude can recognize when to invoke the skill
|
||||
|
||||
**Example**:
|
||||
```markdown
|
||||
# Claude-Mem Search Skill
|
||||
|
||||
Access claude-mem's persistent memory through a comprehensive HTTP API.
|
||||
|
||||
## When to Use This Skill
|
||||
Invoke this skill when users ask about:
|
||||
- Past work: "What did we do last session?"
|
||||
- Bug fixes: "Did we fix this before?"
|
||||
...
|
||||
```
|
||||
|
||||
### Layer 2: Full Skill Instructions (On-Demand)
|
||||
|
||||
**What's Loaded**: Complete operation documentation (~2,500 tokens)
|
||||
|
||||
**Purpose**: Detailed instructions for each search operation
|
||||
|
||||
**When Loaded**: Only when Claude invokes the skill
|
||||
|
||||
**Example Structure**:
|
||||
```
|
||||
/skills/search/
|
||||
├── SKILL.md (main frontmatter)
|
||||
├── operations/
|
||||
│ ├── observations.md (detailed instructions)
|
||||
│ ├── sessions.md
|
||||
│ ├── prompts.md
|
||||
│ ├── by-type.md
|
||||
│ ├── by-concept.md
|
||||
│ ├── by-file.md
|
||||
│ ├── recent-context.md
|
||||
│ ├── timeline.md
|
||||
│ ├── timeline-by-query.md
|
||||
│ ├── help.md
|
||||
│ ├── formatting.md
|
||||
│ └── common-workflows.md
|
||||
```
|
||||
|
||||
### Layer 3: API Response
|
||||
|
||||
**What's Returned**: Search results in requested format
|
||||
|
||||
**Format Options**:
|
||||
- `index` - Titles, dates, IDs only (~50-100 tokens per result)
|
||||
- `full` - Complete details (~500-1000 tokens per result)
|
||||
|
||||
**Progressive Usage**: Start with `index`, drill down with `full` as needed
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### Search Skill Structure
|
||||
|
||||
```
|
||||
plugin/skills/search/
|
||||
├── SKILL.md # Main frontmatter (~250 tokens)
|
||||
├── operations/
|
||||
│ ├── observations.md # Search observations
|
||||
│ ├── sessions.md # Search sessions
|
||||
│ ├── prompts.md # Search prompts
|
||||
│ ├── by-type.md # Filter by type
|
||||
│ ├── by-concept.md # Filter by concept
|
||||
│ ├── by-file.md # Filter by file
|
||||
│ ├── recent-context.md # Get recent context
|
||||
│ ├── timeline.md # Timeline around point
|
||||
│ ├── timeline-by-query.md # Search + timeline
|
||||
│ ├── help.md # API documentation
|
||||
│ ├── formatting.md # Result formatting guide
|
||||
│ └── common-workflows.md # Usage patterns
|
||||
```
|
||||
|
||||
### Worker Service Integration
|
||||
|
||||
**File**: `src/services/worker-service.ts`
|
||||
|
||||
**Search Routes**:
|
||||
```typescript
|
||||
// Full-text search
|
||||
app.get('/api/search/observations', handleSearchObservations);
|
||||
app.get('/api/search/sessions', handleSearchSessions);
|
||||
app.get('/api/search/prompts', handleSearchPrompts);
|
||||
|
||||
// Filtered search
|
||||
app.get('/api/search/by-type', handleSearchByType);
|
||||
app.get('/api/search/by-concept', handleSearchByConcept);
|
||||
app.get('/api/search/by-file', handleSearchByFile);
|
||||
|
||||
// Context retrieval
|
||||
app.get('/api/context/recent', handleRecentContext);
|
||||
app.get('/api/context/timeline', handleTimeline);
|
||||
app.get('/api/timeline/by-query', handleTimelineByQuery);
|
||||
|
||||
// Documentation
|
||||
app.get('/api/search/help', handleHelp);
|
||||
```
|
||||
|
||||
**Database Access**:
|
||||
- Uses `SessionSearch` service for FTS5 queries
|
||||
- Uses `SessionStore` for structured queries
|
||||
- Hybrid search with ChromaDB for semantic similarity
|
||||
|
||||
### Security
|
||||
|
||||
**FTS5 Injection Prevention** (v4.2.3):
|
||||
```typescript
|
||||
function escapeFTS5Query(query: string): string {
|
||||
return query.replace(/"/g, '""');
|
||||
}
|
||||
```
|
||||
|
||||
All user-provided search queries are properly escaped to prevent SQL injection.
|
||||
|
||||
**Comprehensive Testing**: 332 injection attack tests covering:
|
||||
- Special characters
|
||||
- SQL keywords
|
||||
- Quote escaping
|
||||
- Boolean operators
|
||||
|
||||
## Benefits
|
||||
|
||||
### 1. Token Efficiency
|
||||
|
||||
**Before (MCP)**:
|
||||
- Session start: ~2,500 tokens for tool definitions
|
||||
- Every session pays this cost
|
||||
- No progressive disclosure
|
||||
|
||||
**After (Skill)**:
|
||||
- Session start: ~250 tokens for skill frontmatter
|
||||
- Full instructions: ~2,500 tokens (only when invoked)
|
||||
- Net savings: ~2,250 tokens per session (~90% reduction)
|
||||
|
||||
### 2. Natural Language Interface
|
||||
|
||||
**Before**: Users needed to learn MCP tool syntax
|
||||
```
|
||||
search_observations with query="authentication" and type="decision"
|
||||
```
|
||||
|
||||
**After**: Users ask naturally
|
||||
```
|
||||
"What decisions did we make about authentication?"
|
||||
```
|
||||
|
||||
Claude translates to appropriate API call.
|
||||
|
||||
### 3. Flexibility
|
||||
|
||||
**HTTP API Benefits**:
|
||||
- Can be called from skills, MCP tools, or other clients
|
||||
- Easy to test with curl
|
||||
- Standard REST conventions
|
||||
- JSON responses
|
||||
|
||||
**Progressive Disclosure**:
|
||||
- Loads only what's needed
|
||||
- Can add more operations without increasing base cost
|
||||
- Documentation co-located with operations
|
||||
|
||||
### 4. Performance
|
||||
|
||||
**Fast Queries**: FTS5 full-text search <10ms for typical queries
|
||||
|
||||
**Caching**: HTTP layer allows response caching
|
||||
|
||||
**Pagination**: Efficient result pagination with offset/limit
|
||||
|
||||
## Migration Notes
|
||||
|
||||
### For Users
|
||||
|
||||
**No Action Required**: The migration from MCP to skill-based search is transparent.
|
||||
|
||||
**Same Questions Work**: Natural language queries work exactly the same way.
|
||||
|
||||
**Invisible Change**: Users won't notice any difference except better performance.
|
||||
|
||||
### For Developers
|
||||
|
||||
**Deprecated**: MCP search server (`src/servers/search-server.ts`)
|
||||
- Source file kept for reference
|
||||
- No longer built or registered
|
||||
- MCP configuration removed from `plugin/.mcp.json`
|
||||
|
||||
**New Implementation**: Skill-based search
|
||||
- Skill files: `plugin/skills/search/`
|
||||
- HTTP endpoints: `src/services/worker-service.ts` (lines 200-400)
|
||||
- Build script: `npm run build` includes skill files
|
||||
- Sync script: `npm run sync-marketplace` copies to plugin directory
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Worker Service Not Running
|
||||
|
||||
If searches fail, check worker service:
|
||||
|
||||
```bash
|
||||
pm2 list # Check status
|
||||
npm run worker:restart # Restart worker
|
||||
npm run worker:logs # View logs
|
||||
```
|
||||
|
||||
### HTTP Endpoints Not Responding
|
||||
|
||||
Test endpoints directly:
|
||||
|
||||
```bash
|
||||
# Health check
|
||||
curl http://localhost:37777/health
|
||||
|
||||
# Search test
|
||||
curl "http://localhost:37777/api/search/observations?query=test&limit=1"
|
||||
```
|
||||
|
||||
### Skill Not Invoking
|
||||
|
||||
If Claude doesn't invoke the skill:
|
||||
|
||||
1. Check skill files exist: `ls ~/.claude/plugins/marketplaces/thedotmack/plugin/skills/search/`
|
||||
2. Restart Claude Code session
|
||||
3. Try explicit skill invocation: `/skill search`
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [Search Tools Usage](/usage/search-tools) - User guide with examples
|
||||
- [Worker Service Architecture](/architecture/worker-service) - HTTP API details
|
||||
- [Database Schema](/architecture/database) - FTS5 tables and indexes
|
||||
Reference in New Issue
Block a user