Replace search skill with mem-search (#91)
* feat: add mem-search skill with progressive disclosure architecture Add comprehensive mem-search skill for accessing claude-mem's persistent cross-session memory database. Implements progressive disclosure workflow and token-efficient search patterns. Features: - 12 search operations (observations, sessions, prompts, by-type, by-concept, by-file, timelines, etc.) - Progressive disclosure principles to minimize token usage - Anti-patterns documentation to guide LLM behavior - HTTP API integration for all search functionality - Common workflows with composition examples Structure: - SKILL.md: Entry point with temporal trigger patterns - principles/: Progressive disclosure + anti-patterns - operations/: 12 search operation files 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * docs: add CHANGELOG entry for mem-search skill Document mem-search skill addition in Unreleased section with: - 100% effectiveness compliance metrics - Comparison to previous search skill implementation - Progressive disclosure architecture details - Reference to audit report documentation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * docs: add mem-search skill audit report Add comprehensive audit report validating mem-search skill against Anthropic's official skill-creator documentation. Report includes: - Effectiveness metrics comparison (search vs mem-search) - Critical issues analysis for production readiness - Compliance validation across 6 key dimensions - Reference implementation guidance Result: mem-search achieves 100% compliance vs search's 67% 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: Add comprehensive search architecture analysis document - Document current state of dual search architectures (HTTP API and MCP) - Analyze HTTP endpoints and MCP search server architectures - Identify DRY violations across search implementations - Evaluate the use of curl as the optimal approach for search - Provide architectural recommendations for immediate and long-term improvements - Outline action plan for cleanup, feature parity, DRY refactoring * refactor: Remove deprecated search skill documentation and operations * refactor: Reorganize documentation into public and context directories Changes: - Created docs/public/ for Mintlify documentation (.mdx files) - Created docs/context/ for internal planning and implementation docs - Moved all .mdx files and assets to docs/public/ - Moved all internal .md files to docs/context/ - Added CLAUDE.md to both directories explaining their purpose - Updated docs.json paths to work with new structure Benefits: - Clear separation between user-facing and internal documentation - Easier to maintain Mintlify docs in dedicated directory - Internal context files organized separately 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Enhance session management and continuity in hooks - Updated new-hook.ts to clarify session_id threading and idempotent session creation. - Modified prompts.ts to require claudeSessionId for continuation prompts, ensuring session context is maintained. - Improved SessionStore.ts documentation on createSDKSession to emphasize idempotent behavior and session connection. - Refined SDKAgent.ts to detail continuation prompt logic and its reliance on session.claudeSessionId for unified session handling. --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Alex Newman <thedotmack@gmail.com>
This commit is contained in:
basher83
@@ -0,0 +1,443 @@
|
||||
---
|
||||
title: "Worker Service"
|
||||
description: "HTTP API and PM2 process management"
|
||||
---
|
||||
|
||||
# Worker Service
|
||||
|
||||
The worker service is a long-running HTTP API built with Express.js and managed by PM2. It processes observations through the Claude Agent SDK separately from hook execution to prevent timeout issues.
|
||||
|
||||
## Overview
|
||||
|
||||
- **Technology**: Express.js HTTP server
|
||||
- **Process Manager**: PM2
|
||||
- **Port**: Fixed port 37777 (configurable via `CLAUDE_MEM_WORKER_PORT`)
|
||||
- **Location**: `src/services/worker-service.ts`
|
||||
- **Built Output**: `plugin/scripts/worker-service.cjs`
|
||||
- **Model**: Configurable via `CLAUDE_MEM_MODEL` environment variable (default: claude-sonnet-4-5)
|
||||
|
||||
## REST API Endpoints
|
||||
|
||||
The worker service exposes 14 HTTP endpoints organized into four categories:
|
||||
|
||||
### Viewer & Health Endpoints
|
||||
|
||||
#### 1. Viewer UI
|
||||
```
|
||||
GET /
|
||||
```
|
||||
|
||||
**Purpose**: Serves the web-based viewer UI (v5.1.0+)
|
||||
|
||||
**Response**: HTML page with embedded React application
|
||||
|
||||
**Features**:
|
||||
- Real-time memory stream visualization
|
||||
- Infinite scroll pagination
|
||||
- Project filtering
|
||||
- SSE-based live updates
|
||||
- Theme toggle (light/dark mode) as of v5.1.2
|
||||
|
||||
#### 2. Health Check
|
||||
```
|
||||
GET /health
|
||||
```
|
||||
|
||||
**Purpose**: Worker health status check
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"uptime": 12345,
|
||||
"port": 37777
|
||||
}
|
||||
```
|
||||
|
||||
#### 3. Server-Sent Events Stream
|
||||
```
|
||||
GET /stream
|
||||
```
|
||||
|
||||
**Purpose**: Real-time updates for viewer UI
|
||||
|
||||
**Response**: SSE stream with events:
|
||||
- `observation-created`: New observation added
|
||||
- `session-summary-created`: New summary generated
|
||||
- `user-prompt-created`: New prompt recorded
|
||||
|
||||
**Event Format**:
|
||||
```
|
||||
event: observation-created
|
||||
data: {"id": 123, "title": "...", ...}
|
||||
```
|
||||
|
||||
### Data Retrieval Endpoints
|
||||
|
||||
#### 4. Get Prompts
|
||||
```
|
||||
GET /api/prompts?project=my-project&limit=20&offset=0
|
||||
```
|
||||
|
||||
**Purpose**: Retrieve paginated user prompts
|
||||
|
||||
**Query Parameters**:
|
||||
- `project` (optional): Filter by project name
|
||||
- `limit` (default: 20): Number of results
|
||||
- `offset` (default: 0): Pagination offset
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"prompts": [{
|
||||
"id": 1,
|
||||
"session_id": "abc123",
|
||||
"prompt": "User's prompt text",
|
||||
"prompt_number": 1,
|
||||
"created_at": "2025-11-06T10:30:00Z"
|
||||
}],
|
||||
"total": 150,
|
||||
"hasMore": true
|
||||
}
|
||||
```
|
||||
|
||||
#### 5. Get Observations
|
||||
```
|
||||
GET /api/observations?project=my-project&limit=20&offset=0
|
||||
```
|
||||
|
||||
**Purpose**: Retrieve paginated observations
|
||||
|
||||
**Query Parameters**:
|
||||
- `project` (optional): Filter by project name
|
||||
- `limit` (default: 20): Number of results
|
||||
- `offset` (default: 0): Pagination offset
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"observations": [{
|
||||
"id": 123,
|
||||
"title": "Fix authentication bug",
|
||||
"type": "bugfix",
|
||||
"narrative": "...",
|
||||
"created_at": "2025-11-06T10:30:00Z"
|
||||
}],
|
||||
"total": 500,
|
||||
"hasMore": true
|
||||
}
|
||||
```
|
||||
|
||||
#### 6. Get Summaries
|
||||
```
|
||||
GET /api/summaries?project=my-project&limit=20&offset=0
|
||||
```
|
||||
|
||||
**Purpose**: Retrieve paginated session summaries
|
||||
|
||||
**Query Parameters**:
|
||||
- `project` (optional): Filter by project name
|
||||
- `limit` (default: 20): Number of results
|
||||
- `offset` (default: 0): Pagination offset
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"summaries": [{
|
||||
"id": 456,
|
||||
"session_id": "abc123",
|
||||
"request": "User's original request",
|
||||
"completed": "Work finished",
|
||||
"created_at": "2025-11-06T10:30:00Z"
|
||||
}],
|
||||
"total": 100,
|
||||
"hasMore": true
|
||||
}
|
||||
```
|
||||
|
||||
#### 7. Get Stats
|
||||
```
|
||||
GET /api/stats
|
||||
```
|
||||
|
||||
**Purpose**: Get database statistics by project
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"byProject": {
|
||||
"my-project": {
|
||||
"observations": 245,
|
||||
"summaries": 12,
|
||||
"prompts": 48
|
||||
},
|
||||
"other-project": {
|
||||
"observations": 156,
|
||||
"summaries": 8,
|
||||
"prompts": 32
|
||||
}
|
||||
},
|
||||
"total": {
|
||||
"observations": 401,
|
||||
"summaries": 20,
|
||||
"prompts": 80,
|
||||
"sessions": 20
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Settings Endpoints
|
||||
|
||||
#### 8. Get Settings
|
||||
```
|
||||
GET /api/settings
|
||||
```
|
||||
|
||||
**Purpose**: Retrieve user settings
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"sidebarOpen": true,
|
||||
"selectedProject": "my-project",
|
||||
"theme": "dark"
|
||||
}
|
||||
```
|
||||
|
||||
#### 9. Save Settings
|
||||
```
|
||||
POST /api/settings
|
||||
```
|
||||
|
||||
**Purpose**: Persist user settings
|
||||
|
||||
**Request Body**:
|
||||
```json
|
||||
{
|
||||
"sidebarOpen": false,
|
||||
"selectedProject": "other-project",
|
||||
"theme": "light"
|
||||
}
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"success": true
|
||||
}
|
||||
```
|
||||
|
||||
### Session Management Endpoints
|
||||
|
||||
#### 10. Initialize Session
|
||||
```
|
||||
POST /sessions/:sessionDbId/init
|
||||
```
|
||||
|
||||
**Request Body**:
|
||||
```json
|
||||
{
|
||||
"sdk_session_id": "abc-123",
|
||||
"project": "my-project"
|
||||
}
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"session_id": "abc-123"
|
||||
}
|
||||
```
|
||||
|
||||
#### 11. Add Observation
|
||||
```
|
||||
POST /sessions/:sessionDbId/observations
|
||||
```
|
||||
|
||||
**Request Body**:
|
||||
```json
|
||||
{
|
||||
"tool_name": "Read",
|
||||
"tool_input": {...},
|
||||
"tool_result": "...",
|
||||
"correlation_id": "xyz-789"
|
||||
}
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"observation_id": 123
|
||||
}
|
||||
```
|
||||
|
||||
#### 12. Generate Summary
|
||||
```
|
||||
POST /sessions/:sessionDbId/summarize
|
||||
```
|
||||
|
||||
**Request Body**:
|
||||
```json
|
||||
{
|
||||
"trigger": "stop"
|
||||
}
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"summary_id": 456
|
||||
}
|
||||
```
|
||||
|
||||
#### 13. Session Status
|
||||
```
|
||||
GET /sessions/:sessionDbId/status
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"session_id": "abc-123",
|
||||
"status": "active",
|
||||
"observation_count": 42,
|
||||
"summary_count": 1
|
||||
}
|
||||
```
|
||||
|
||||
#### 14. Delete Session
|
||||
```
|
||||
DELETE /sessions/:sessionDbId
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"success": true
|
||||
}
|
||||
```
|
||||
|
||||
**Note**: As of v4.1.0, the cleanup hook no longer calls this endpoint. Sessions are marked complete instead of deleted to allow graceful worker shutdown.
|
||||
|
||||
## PM2 Management
|
||||
|
||||
### Configuration
|
||||
|
||||
The worker is configured via `ecosystem.config.cjs`:
|
||||
|
||||
```javascript
|
||||
module.exports = {
|
||||
apps: [{
|
||||
name: 'claude-mem-worker',
|
||||
script: './plugin/scripts/worker-service.cjs',
|
||||
instances: 1,
|
||||
autorestart: true,
|
||||
watch: false,
|
||||
max_memory_restart: '1G',
|
||||
env: {
|
||||
NODE_ENV: 'production',
|
||||
FORCE_COLOR: '1'
|
||||
}
|
||||
}]
|
||||
};
|
||||
```
|
||||
|
||||
### Commands
|
||||
|
||||
```bash
|
||||
# Start worker (auto-starts on first session)
|
||||
npm run worker:start
|
||||
|
||||
# Stop worker
|
||||
npm run worker:stop
|
||||
|
||||
# Restart worker
|
||||
npm run worker:restart
|
||||
|
||||
# View logs
|
||||
npm run worker:logs
|
||||
|
||||
# Check status
|
||||
npm run worker:status
|
||||
```
|
||||
|
||||
### Auto-Start Behavior
|
||||
|
||||
As of v4.0.0, the worker service auto-starts when the SessionStart hook fires. Manual start is optional.
|
||||
|
||||
## Claude Agent SDK Integration
|
||||
|
||||
The worker service routes observations to the Claude Agent SDK for AI-powered processing:
|
||||
|
||||
### Processing Flow
|
||||
|
||||
1. **Observation Queue**: Observations accumulate in memory
|
||||
2. **SDK Processing**: Observations sent to Claude via Agent SDK
|
||||
3. **XML Parsing**: Responses parsed for structured data
|
||||
4. **Database Storage**: Processed observations stored in SQLite
|
||||
|
||||
### SDK Components
|
||||
|
||||
- **Prompts** (`src/sdk/prompts.ts`): Builds XML-structured prompts
|
||||
- **Parser** (`src/sdk/parser.ts`): Parses Claude's XML responses
|
||||
- **Worker** (`src/sdk/worker.ts`): Main SDK agent loop
|
||||
|
||||
### Model Configuration
|
||||
|
||||
Set the AI model used for processing via environment variable:
|
||||
|
||||
```bash
|
||||
export CLAUDE_MEM_MODEL=claude-sonnet-4-5
|
||||
```
|
||||
|
||||
Available models:
|
||||
- `claude-haiku-4-5` - Fast, cost-efficient
|
||||
- `claude-sonnet-4-5` - Balanced (default)
|
||||
- `claude-opus-4` - Most capable
|
||||
- `claude-3-7-sonnet` - Alternative version
|
||||
|
||||
## Port Allocation
|
||||
|
||||
The worker uses a fixed port (37777 by default) for consistent communication:
|
||||
|
||||
- **Default**: Port 37777
|
||||
- **Override**: Set `CLAUDE_MEM_WORKER_PORT` environment variable
|
||||
- **Port File**: `${CLAUDE_PLUGIN_ROOT}/data/worker.port` tracks current port
|
||||
|
||||
If port 37777 is in use, the worker will fail to start. Set a custom port via environment variable.
|
||||
|
||||
## Data Storage
|
||||
|
||||
The worker service stores data in the plugin data directory:
|
||||
|
||||
```
|
||||
${CLAUDE_PLUGIN_ROOT}/data/
|
||||
├── claude-mem.db # SQLite database
|
||||
├── worker.port # Current worker port file
|
||||
└── logs/
|
||||
├── worker-out.log # Worker stdout logs
|
||||
└── worker-error.log # Worker stderr logs
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
The worker implements graceful degradation:
|
||||
|
||||
- **Database Errors**: Logged but don't crash the service
|
||||
- **SDK Errors**: Retried with exponential backoff
|
||||
- **Network Errors**: Logged and skipped
|
||||
- **Invalid Input**: Validated and rejected with error response
|
||||
|
||||
## Performance
|
||||
|
||||
- **Async Processing**: Observations processed asynchronously
|
||||
- **In-Memory Queue**: Fast observation accumulation
|
||||
- **Batch Processing**: Multiple observations processed together
|
||||
- **Connection Pooling**: SQLite connections reused
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
See [Troubleshooting - Worker Issues](../troubleshooting.md#worker-service-issues) for common problems and solutions.
|
||||
Reference in New Issue
Block a user