airkjw/claude-mem
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add installation, troubleshooting, and usage documentation for Claude-Mem plugin

Browse Source 
- Created comprehensive installation guide detailing quick start, system requirements, and advanced installation steps.
- Developed troubleshooting guide addressing common issues with worker service, hooks, database, and search tools.
- Introduced getting started documentation explaining automatic operation, session summaries, and context injection.
- Added detailed usage instructions for MCP search tools, including query examples and advanced filtering techniques.
This commit is contained in:
Alex Newman
2025-10-23 23:40:42 -04:00
parent fd4cd0444c
commit 12149470a2
26 changed files with 3569 additions and 2722 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+134 -687
View File
@@ -10,707 +10,202 @@ Claude-Mem seamlessly preserves context across sessions by automatically capturi
---
## Installation
## Quick Start
Install Claude-Mem directly from the plugin marketplace:
Start a new Claude Code session in the terminal and enter the following commands:
```bash
/plugin marketplace add thedotmack/claude-mem
/plugin install claude-mem
```
> /plugin marketplace add thedotmack/claude-mem
> /plugin install claude-mem
```
That's it! The plugin will automatically:
- Download prebuilt binaries (no compilation needed)
- Install all dependencies (including PM2 and SQLite binaries)
- Configure hooks for session lifecycle management
- Set up the MCP search server
- Auto-start the worker service on first session
Restart Claude Code. Context from previous sessions will automatically appear in new sessions.
Start a new Claude Code session and you'll see context from previous sessions automatically loaded.
**Key Features:**
- 🧠 **Persistent Memory** - Context survives across sessions
- 🔍 **7 Search Tools** - Query your project history via MCP
- 🤖 **Automatic Operation** - No manual intervention required
- 📊 **FTS5 Search** - Fast full-text search across observations
- 🔗 **Citations** - Reference past decisions with `claude-mem://` URIs
---
## What's New in v4.0.0
## Documentation
**BREAKING CHANGES - Please Read:**
### Getting Started
- **[Installation Guide](docs/installation.md)** - Quick start & advanced installation
- **[Usage Guide](docs/usage/getting-started.md)** - How Claude-Mem works automatically
- **[MCP Search Tools](docs/usage/search-tools.md)** - Query your project history
- **Data Location Changed**: Database moved from `~/.claude-mem/` to `${CLAUDE_PLUGIN_ROOT}/data/` (inside plugin directory)
- **Fresh Start Required**: No automatic migration from v3.x. Users must start with a clean database
- **Worker Auto-Starts**: Worker service now starts automatically - no manual `npm run worker:start` needed
- **MCP Search Server**: 6 new search tools with full-text search and citations
- **Enhanced Architecture**: Improved plugin integration and data organization
### Architecture
- **[Overview](docs/architecture/overview.md)** - System components & data flow
- **[Hooks](docs/architecture/hooks.md)** - 5 lifecycle hooks explained
- **[Worker Service](docs/architecture/worker-service.md)** - HTTP API & PM2 management
- **[Database](docs/architecture/database.md)** - SQLite schema & FTS5 search
- **[MCP Search](docs/architecture/mcp-search.md)** - 7 search tools & examples
See [CHANGELOG.md](CHANGELOG.md) for complete details.
---
## Table of Contents
- [Installation](#installation)
- [Overview](#overview)
- [How It Works](#how-it-works)
- [Usage](#usage)
- [MCP Search Tools](#mcp-search-tools)
- [Architecture](#architecture)
- [Advanced Installation](#advanced-installation)
- [Configuration](#configuration)
- [Development](#development)
- [Troubleshooting](#troubleshooting)
- [License](#license)
---
## Overview
### What is Claude-Mem?
Claude-Mem is a **Claude Code plugin** that provides persistent memory across sessions. When you work with Claude Code on a project, claude-mem:
1. **Captures** every tool execution (Read, Write, Bash, Edit, etc.)
2. **Processes** observations through Claude Agent SDK to extract learnings
3. **Summarizes** what was accomplished, learned, and what's next
4. **Restores** context automatically when you start new sessions
### Key Features
- **Session Continuity**: Knowledge persists across Claude Code sessions
- **Automatic Context Injection**: Recent session summaries appear when Claude starts
- **Auto-Starting Worker**: Worker service starts automatically on first session (v4.0+)
- **MCP Search Server**: Search and retrieve observations and sessions via 6 specialized tools
- **Structured Observations**: XML-formatted extraction of learnings
- **Smart Filtering**: Skips low-value tool observations
- **Multi-Prompt Sessions**: Tracks multiple prompts within a single session
- **HTTP API**: Modern REST interface for hook communication
- **Process Management**: PM2-managed long-running service
- **Graceful Degradation**: Doesn't block Claude even if worker is down
### System Requirements
- **Node.js**: 18.0.0 or higher
- **Claude Code**: Latest version with plugin support
- **PM2**: Process manager (bundled with plugin - no global install required)
- **SQLite 3**: For persistent storage (bundled)
### Configuration & Development
- **[Configuration](docs/configuration.md)** - Environment variables & settings
- **[Development](docs/development.md)** - Building, testing, contributing
- **[Troubleshooting](docs/troubleshooting.md)** - Common issues & solutions
---
## How It Works
### The Full Lifecycle
```
┌─────────────────────────────────────────────────────────────────┐
│ 1. Session Starts → Context Hook Fires │
│ Injects summaries from last 3 sessions into Claude's context │
└─────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────┐
│ 2. User Types Prompt → UserPromptSubmit Hook Fires │
│ Creates SDK session in database, notifies worker service │
└─────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────┐
│ 3. Claude Uses Tools → PostToolUse Hook Fires (100+ times) │
│ Sends observations to worker service for processing │
└─────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────┐
│ 4. Worker Processes → Claude Agent SDK Analyzes │
│ Extracts structured learnings via iterative AI processing │
└─────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────┐
│ 5. Claude Stops → Stop Hook Fires │
│ Generates final summary with request, status, next steps │
└─────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────┐
│ 6. Session Ends → Cleanup Hook Fires │
│ Marks session complete, ready for next session context │
└─────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ Session Start → Inject context from last 10 sessions │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ User Prompts → Create session, save user prompts │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ Tool Executions → Capture observations (Read, Write, etc.) │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ Worker Processes → Extract learnings via Claude Agent SDK │
└─────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────┐
│ Session Ends → Generate summary, ready for next session │
└─────────────────────────────────────────────────────────────┘
```
### Core Components
**Core Components:**
1. **5 Lifecycle Hooks** - SessionStart, UserPromptSubmit, PostToolUse, Stop, SessionEnd
2. **Worker Service** - HTTP API on port 37777 managed by PM2
3. **SQLite Database** - Stores sessions, observations, summaries with FTS5 search
4. **7 MCP Search Tools** - Query historical context with citations
#### 1. Plugin Hooks (5 Lifecycle Hooks)
- **SessionStart Hook** (`context-hook.js`): Queries database for last 3 sessions and injects context
- **UserPromptSubmit Hook** (`new-hook.js`): Creates/reuses SDK session, sends init signal
- **PostToolUse Hook** (`save-hook.js`): Sends tool observations to worker service
- **Stop Hook** (`summary-hook.js`): Triggers final summary generation
- **SessionEnd Hook** (`cleanup-hook.js`): Marks session as completed/failed
#### 2. Worker Service
Long-running HTTP service (managed by PM2) that:
- Listens on dynamic port 37000-37999
- Provides REST API for hook communication
- Maintains active session state in memory
- Routes observations to Claude Agent SDK
- Writes processed summaries back to database
**Endpoints:**
- `POST /sessions/:id/init` - Initialize session
- `POST /sessions/:id/observations` - Queue tool observations
- `POST /sessions/:id/summarize` - Generate summary
- `GET /sessions/:id/status` - Check status
- `DELETE /sessions/:id` - Clean up session
- `GET /health` - Health check
#### 3. SDK Memory Processor
Uses Claude Agent SDK (`@anthropic-ai/claude-agent-sdk`) to:
- Build specialized XML-structured prompts
- Feed observations through iterative cycles
- Parse Claude's XML responses for structured data
- Accumulate learnings about modifications, discoveries, decisions
- Generate final summaries with lessons learned and next steps
#### 4. MCP Search Server
Claude-Mem includes a Model Context Protocol (MCP) server that exposes 6 specialized search tools for querying stored observations and sessions:
**Search Tools:**
- `search_observations` - Full-text search across observation titles, narratives, facts, and concepts
- `search_sessions` - Full-text search across session summaries, requests, and learnings
- `find_by_concept` - Find observations tagged with specific concepts
- `find_by_file` - Find observations and sessions that reference specific file paths
- `find_by_type` - Find observations by type (decision, bugfix, feature, refactor, discovery, change)
- `advanced_search` - Combined search with filters across observations and sessions
All search results are returned in `search_result` format with **citations enabled**, allowing Claude to reference specific observations and sessions from your project history using the `claude-mem://` URI scheme.
**Configuration:** The MCP server is automatically registered via `plugin/.mcp.json` and runs when Claude Code starts.
#### 5. Database Layer
SQLite database (`${CLAUDE_PLUGIN_ROOT}/data/claude-mem.db`) with tables:
- **sdk_sessions**: Active/completed session tracking
- **observations**: Individual tool executions with FTS5 full-text search
- **session_summaries**: Processed semantic summaries with FTS5 full-text search
- **sessions**, **memories**, **overviews**: Legacy tables
**Full-Text Search:** The database includes FTS5 (Full-Text Search) virtual tables for fast searching across observations and summaries, powering the MCP search tools.
See [Architecture Overview](docs/architecture/overview.md) for details.
---
## Advanced Installation
## MCP Search Tools
For development or testing, you can clone and build from source:
Claude-Mem provides 7 specialized search tools:
```bash
# Clone the repository
git clone https://github.com/thedotmack/claude-mem.git
cd claude-mem
# Install dependencies
npm install
# Build hooks and worker service
npm run build
# Worker service will auto-start on first Claude Code session
# Or manually start with:
npm run worker:start
# Verify worker is running
npm run worker:status
```
### Post-Installation Notes
1. **Automatic Dependency Installation**
Dependencies are installed automatically during plugin installation. The SessionStart hook also ensures dependencies are up-to-date on each session start (this is fast and idempotent). Works cross-platform on Windows, macOS, and Linux.
2. **Verify Plugin Installation**
Check that hooks are configured in Claude Code:
```bash
cat plugin/hooks/hooks.json
```
3. **Data Directory Location**
v4.0.0+ stores data in `${CLAUDE_PLUGIN_ROOT}/data/`:
- Database: `${CLAUDE_PLUGIN_ROOT}/data/claude-mem.db`
- Worker port file: `${CLAUDE_PLUGIN_ROOT}/data/worker.port`
- Logs: `${CLAUDE_PLUGIN_ROOT}/data/logs/`
For development/testing, you can override:
```bash
export CLAUDE_MEM_DATA_DIR=/custom/path
```
4. **Check Worker Logs**
```bash
npm run worker:logs
```
5. **Test Context Retrieval**
```bash
npm run test:context
```
---
## Usage
### Automatic Operation
Claude-Mem works automatically once installed. No manual intervention required!
1. **Start Claude Code** - Context from last 3 sessions appears automatically
2. **Work normally** - Every tool execution is captured
3. **Stop Claude** - Summary is generated and saved
4. **Next session** - Previous work appears in context
### MCP Search Tools
Once claude-mem is installed as a plugin, six search tools become available in your Claude Code sessions:
1. **search_observations** - Full-text search across observations
2. **search_sessions** - Full-text search across session summaries
3. **search_user_prompts** - Search raw user requests
4. **find_by_concept** - Find by concept tags
5. **find_by_file** - Find by file references
6. **find_by_type** - Find by type (decision, bugfix, feature, etc.)
7. **get_recent_context** - Get recent session context
**Example Queries:**
```
"Use search_observations to find all decisions about the build system"
→ Searches for observations with type="decision" and content matching "build system"
"Use find_by_file to show me everything related to worker-service.ts"
→ Returns all observations and sessions that read/modified worker-service.ts
"Use search_sessions to find what we learned about hooks"
→ Searches session summaries for mentions of "hooks" in learnings
"Use find_by_concept to show observations tagged with 'architecture'"
→ Returns observations tagged with the concept "architecture"
search_observations with query="authentication" and type="decision"
find_by_file with filePath="worker-service.ts"
search_user_prompts with query="add dark mode"
get_recent_context with limit=5
```
All results include:
- **Citations**: Results use `claude-mem://observation/{id}` or `claude-mem://session/{id}` URIs
- **Metadata**: Type, concepts, files, facts, and dates
- **Rich Context**: Full observation narratives and session summaries
- **Filtering**: Project, date ranges, types, concepts, files
### Manual Commands
#### Worker Management
**Note**: v4.0+ auto-starts the worker on first session. Manual commands below are optional.
```bash
# Start worker service (optional - auto-starts automatically)
npm run worker:start
# Stop worker service
npm run worker:stop
# Restart worker service
npm run worker:restart
# View worker logs
npm run worker:logs
# Check worker status
npm run worker:status
```
#### Testing
```bash
# Run all tests
npm test
# Test context injection
npm run test:context
# Verbose context test
npm run test:context:verbose
```
#### Development
```bash
# Build hooks and worker
npm run build
# Build only hooks
npm run build:hooks
# Publish to NPM (maintainers only)
npm run publish:npm
```
### Viewing Stored Context
Context is stored in SQLite database. Location varies by version:
- v4.0+: `${CLAUDE_PLUGIN_ROOT}/data/claude-mem.db` (inside plugin)
- v3.x: `~/.claude-mem/claude-mem.db` (legacy)
Query the database directly:
```bash
# v4.0+ uses ~/.claude-mem directory
sqlite3 ~/.claude-mem/claude-mem.db
# View recent sessions
SELECT session_id, project, created_at, status FROM sdk_sessions ORDER BY created_at DESC LIMIT 10;
# View session summaries
SELECT session_id, request, completed, learned FROM session_summaries ORDER BY created_at DESC LIMIT 5;
# View observations for a session
SELECT tool_name, created_at FROM observations WHERE session_id = 'YOUR_SESSION_ID';
```
See [MCP Search Tools Guide](docs/usage/search-tools.md) for detailed examples.
---
## Architecture
## What's New in v4.2.3
### Technology Stack
**Security:**
- Fixed FTS5 injection vulnerability in search functions
- Added comprehensive test suite with 332 injection attack tests
| Layer | Technology |
|------------------------|-------------------------------------------|
| **Language** | TypeScript (ES2022, ESNext modules) |
| **Runtime** | Node.js 18+ |
| **Database** | SQLite 3 with better-sqlite3 driver |
| **HTTP Server** | Express.js 4.18 |
| **AI SDK** | @anthropic-ai/claude-agent-sdk |
| **Build Tool** | esbuild (bundles TypeScript) |
| **Process Manager** | PM2 |
| **Testing** | Node.js built-in test runner |
**Fixes:**
- Fixed ESM/CJS compatibility for getDirname function
- Fixed Windows PowerShell compatibility in SessionStart hook
- Cross-platform dependency installation now works on Windows, macOS, and Linux
### Directory Structure
See [CHANGELOG.md](CHANGELOG.md) for complete version history.
```
claude-mem/
├── src/
│ ├── bin/hooks/ # Entry point scripts for 5 hooks
│ │ ├── context-hook.ts # SessionStart
│ │ ├── new-hook.ts # UserPromptSubmit
│ │ ├── save-hook.ts # PostToolUse
│ │ ├── summary-hook.ts # Stop
│ │ └── cleanup-hook.ts # SessionEnd
│ │
│ ├── hooks/ # Hook implementation logic
│ │ ├── context.ts
│ │ ├── new.ts
│ │ ├── save.ts
│ │ ├── summary.ts
│ │ └── cleanup.ts
│ │
│ ├── servers/ # MCP servers
│ │ └── search-server.ts # MCP search tools server
│ │
│ ├── sdk/ # Claude Agent SDK integration
│ │ ├── prompts.ts # XML prompt builders
│ │ ├── parser.ts # XML response parser
│ │ └── worker.ts # Main SDK agent loop
│ │
│ ├── services/
│ │ ├── worker-service.ts # Express HTTP service
│ │ └── sqlite/ # Database layer
│ │ ├── Database.ts
│ │ ├── HooksDatabase.ts
│ │ ├── SessionSearch.ts # FTS5 search service
│ │ ├── migrations.ts
│ │ └── types.ts
│ │
│ ├── shared/ # Shared utilities
│ │ ├── config.ts
│ │ ├── paths.ts
│ │ └── storage.ts
│ │
│ └── utils/
│ ├── logger.ts
│ ├── platform.ts
│ └── port-allocator.ts
│
├── plugin/ # Plugin distribution
│ ├── .claude-plugin/
│ │ └── plugin.json
│ ├── .mcp.json # MCP server configuration
│ ├── hooks/
│ │ └── hooks.json
│ └── scripts/ # Built executables
│ ├── context-hook.js
│ ├── new-hook.js
│ ├── save-hook.js
│ ├── summary-hook.js
│ ├── cleanup-hook.js
│ ├── worker-service.cjs # Background worker
│ └── search-server.js # MCP search server
│
├── tests/ # Test suite
├── context/ # Architecture docs
└── ecosystem.config.cjs # PM2 configuration
```
---
### Data Flow
## System Requirements
#### Memory Pipeline
```
Hook (stdin) → Database → Worker Service → SDK Processor → Database → Next Session Hook
```
- **Node.js**: 18.0.0 or higher
- **Claude Code**: Latest version with plugin support
- **PM2**: Process manager (bundled - no global install required)
- **SQLite 3**: For persistent storage (bundled)
1. **Input**: Claude Code sends tool execution data via stdin to hooks
2. **Storage**: Hooks write observations to SQLite database
3. **Processing**: Worker service reads observations, processes via SDK
4. **Output**: Processed summaries written back to database
5. **Retrieval**: Next session's context hook reads summaries from database
---
#### Search Pipeline
```
Claude Request → MCP Server → SessionSearch Service → FTS5 Database → Search Results → Claude
```
## Key Benefits
1. **Query**: Claude uses MCP search tools (e.g., `search_observations`)
2. **Search**: MCP server calls SessionSearch service with query parameters
3. **FTS5**: Full-text search executes against FTS5 virtual tables
4. **Format**: Results formatted as `search_result` blocks with citations
5. **Return**: Claude receives citable search results for analysis
### Automatic Memory
- Context automatically injected when Claude starts
- No manual commands or configuration needed
- Works transparently in the background
### Full History Search
- Search across all sessions and observations
- FTS5 full-text search for fast queries
- Citations link back to specific observations
### Structured Observations
- AI-powered extraction of learnings
- Categorized by type (decision, bugfix, feature, etc.)
- Tagged with concepts and file references
### Multi-Prompt Sessions
- Sessions span multiple user prompts
- Context preserved across `/clear` commands
- Track entire conversation threads
---
## Configuration
### Environment Variables
| Variable | Default | Description |
|-------------------------|---------------------------------|---------------------------------------|
| `CLAUDE_PLUGIN_ROOT` | Set by Claude Code | Plugin installation directory |
| `CLAUDE_MEM_DATA_DIR` | `${CLAUDE_PLUGIN_ROOT}/data/` | Data directory override (dev only) |
| `CLAUDE_MEM_WORKER_PORT`| `0` (dynamic) | Worker service port (37000-37999) |
| `NODE_ENV` | `production` | Environment mode |
| `FORCE_COLOR` | `1` | Enable colored logs |
### Files and Directories
**v4.0.0+ Structure:**
```
${CLAUDE_PLUGIN_ROOT}/data/
├── claude-mem.db # SQLite database
├── worker.port # Current worker port file
└── logs/
├── worker-out.log # Worker stdout logs
└── worker-error.log # Worker stderr logs
**Model Selection:**
```bash
./claude-mem-settings.sh
```
**Legacy (v3.x):**
**Environment Variables:**
- `CLAUDE_MEM_MODEL` - AI model for processing (default: claude-sonnet-4-5)
- `CLAUDE_MEM_WORKER_PORT` - Worker port (default: 37777)
- `CLAUDE_MEM_DATA_DIR` - Data directory override (dev only)
```
~/.claude-mem/ # Old location (no longer used)
```
### Plugin Configuration
#### Hooks Configuration
Hooks are configured in `plugin/hooks/hooks.json`:
```json
{
"SessionStart": {
"command": "${CLAUDE_PLUGIN_ROOT}/scripts/context-hook.js",
"timeout": 180000
},
"UserPromptSubmit": {
"command": "${CLAUDE_PLUGIN_ROOT}/scripts/new-hook.js",
"timeout": 60000
},
"PostToolUse": {
"command": "${CLAUDE_PLUGIN_ROOT}/scripts/save-hook.js",
"timeout": 180000
},
"Stop": {
"command": "${CLAUDE_PLUGIN_ROOT}/scripts/summary-hook.js",
"timeout": 60000
},
"SessionEnd": {
"command": "${CLAUDE_PLUGIN_ROOT}/scripts/cleanup-hook.js",
"timeout": 60000
}
}
```
#### MCP Server Configuration
The MCP search server is configured in `plugin/.mcp.json`:
```json
{
"mcpServers": {
"claude-mem-search": {
"type": "stdio",
"command": "${CLAUDE_PLUGIN_ROOT}/scripts/search-server.js"
}
}
}
```
This registers the `claude-mem-search` server with Claude Code, making the 6 search tools available in all sessions. The server is automatically started when Claude Code launches and communicates via stdio transport.
See [Configuration Guide](docs/configuration.md) for details.
---
## Development
### Building from Source
```bash
# Clone repository
# Clone and build
git clone https://github.com/thedotmack/claude-mem.git
cd claude-mem
# Install dependencies
npm install
# Build all components
npm run build
```
The build process:
1. Compiles TypeScript to JavaScript using esbuild
2. Creates standalone executables for each hook in `plugin/scripts/`
3. Bundles MCP search server to `plugin/scripts/search-server.js`
4. Bundles worker service to `plugin/scripts/worker-service.cjs`
### Running Tests
```bash
# Run all tests
# Run tests
npm test
# Run specific test file
node --test tests/session-lifecycle.test.ts
# Start worker
npm run worker:start
# View logs
npm run worker:logs
```
### Development Workflow
1. Make changes to TypeScript source files
2. Run `npm run build` to compile
3. Test with `npm run test:context` or start Claude Code
4. Check worker logs with `npm run worker:logs`
### Adding New Features
#### Adding a New Hook
1. Create hook implementation in `src/hooks/your-hook.ts`
2. Create entry point in `src/bin/hooks/your-hook.ts`
3. Add to `plugin/hooks/hooks.json`
4. Rebuild with `npm run build`
#### Modifying Database Schema
1. Add migration to `src/services/sqlite/migrations.ts`
2. Update types in `src/services/sqlite/types.ts`
3. Update database methods in `src/services/sqlite/HooksDatabase.ts`
#### Extending SDK Prompts
1. Modify prompts in `src/sdk/prompts.ts`
2. Update parser in `src/sdk/parser.ts` to handle new XML structure
3. Test with `npm test`
See [Development Guide](docs/development.md) for detailed instructions.
---
## Troubleshooting
### Worker Service Issues
**Common Issues:**
- Worker not starting → `npm run worker:restart`
- No context appearing → `npm run test:context`
- Database issues → `sqlite3 ~/.claude-mem/claude-mem.db "PRAGMA integrity_check;"`
- Search not working → Check FTS5 tables exist
**Problem**: Worker service not starting
```bash
# Check if PM2 is running
pm2 status
# Check worker logs
npm run worker:logs
# Restart worker
npm run worker:restart
# Full reset
pm2 delete claude-mem-worker
npm run worker:start
```
**Problem**: Port allocation failed
```bash
# Check if port file exists (v4.0+)
cat ${CLAUDE_PLUGIN_ROOT}/data/worker.port
# Manually specify port
CLAUDE_MEM_WORKER_PORT=37500 npm run worker:start
```
### Hook Issues
**Problem**: Hooks not firing
```bash
# Test context hook manually
echo '{"session_id":"test-123","cwd":"'$(pwd)'","source":"startup"}' | node plugin/scripts/context-hook.js
# Check hook permissions
ls -la plugin/scripts/*.js
# Verify hooks.json is valid
cat plugin/hooks/hooks.json | jq .
```
**Problem**: Context not appearing
```bash
# Check if summaries exist
sqlite3 ~/.claude-mem/claude-mem.db "SELECT COUNT(*) FROM session_summaries;"
# View recent sessions
npm run test:context:verbose
# Check database integrity
sqlite3 ~/.claude-mem/claude-mem.db "PRAGMA integrity_check;"
```
### Database Issues
**Problem**: Database locked
```bash
# Close all connections
pm2 stop claude-mem-worker
# Check for stale locks
lsof ~/.claude-mem/claude-mem.db
# Backup and recreate (nuclear option)
cp ~/.claude-mem/claude-mem.db ~/.claude-mem/claude-mem.db.backup
rm ~/.claude-mem/claude-mem.db
npm run worker:start # Will recreate schema
```
### Debugging
Enable verbose logging:
```bash
# Set debug mode
export DEBUG=claude-mem:*
# View all logs
npm run worker:logs
```
Check correlation IDs to trace observations through the pipeline:
```bash
sqlite3 ~/.claude-mem/claude-mem.db "SELECT correlation_id, tool_name, created_at FROM observations WHERE session_id = 'YOUR_SESSION_ID' ORDER BY created_at;"
```
See [Troubleshooting Guide](docs/troubleshooting.md) for complete solutions.
---
@@ -719,17 +214,12 @@ sqlite3 ~/.claude-mem/claude-mem.db "SELECT correlation_id, tool_name, created_a
Contributions are welcome! Please:
1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request
2. Create a feature branch
3. Make your changes with tests
4. Update documentation
5. Submit a Pull Request
### Code Style
- Use TypeScript strict mode
- Follow existing code formatting
- Add tests for new features
- Update documentation as needed
See [Development Guide](docs/development.md) for contribution workflow.
---
@@ -741,64 +231,21 @@ Copyright (C) 2025 Alex Newman (@thedotmack). All rights reserved.
See the [LICENSE](LICENSE) file for full details.
### What This Means
**What This Means:**
- You can use, modify, and distribute this software freely
- If you modify and deploy this software on a network server, you must make your source code available
- Any derivative works must also be licensed under AGPL-3.0
- If you modify and deploy on a network server, you must make your source code available
- Derivative works must also be licensed under AGPL-3.0
- There is NO WARRANTY for this software
For more information about AGPL-3.0, see: https://www.gnu.org/licenses/agpl-3.0.html
---
## Support
- **Documentation**: [docs/](docs/)
- **Issues**: [GitHub Issues](https://github.com/thedotmack/claude-mem/issues)
- **Repository**: [github.com/thedotmack/claude-mem](https://github.com/thedotmack/claude-mem)
- **Author**: Alex Newman ([@thedotmack](https://github.com/thedotmack))
---
## Changelog
### v4.2.3 (Current)
- **SECURITY**: Fixed FTS5 injection vulnerability with proper escaping
- **FIX**: ESM/CJS compatibility for getDirname function
- **FIX**: Windows PowerShell compatibility in SessionStart hook
- Added comprehensive test suite with 332 injection attack tests
- Cross-platform dependency installation now works on Windows, macOS, and Linux
### v4.2.1
- **NEW**: Summary skip logic to prevent duplicate summaries
- **FIX**: Observation type validation for all 6 types
- Enhanced chronological summary guidance
### v4.1.1
- **REMOVED**: Redundant advanced_search MCP tool
- **FIX**: MCP search limit bugs and type contamination
- Improved search prompts and token limit guidance
### v4.1.0
- **NEW**: Graceful session cleanup
- **RESTORED**: MCP search server from backup
- Natural worker shutdown for pending operations
### v4.0.0
- **NEW**: MCP Search Server with 6 specialized search tools
- **NEW**: FTS5 full-text search across observations and session summaries
- **BREAKING**: Data directory moved to `${CLAUDE_PLUGIN_ROOT}/data/`
- **NEW**: Auto-starting worker service
- **NEW**: Session continuity with automatic context injection
- Refactored summary and context handling in hooks
- Implemented structured logging across the application
- Improved error handling and graceful degradation
---
**Built with Claude Agent SDK** | **Powered by Claude Code** | **Made with TypeScript**
docs/architecture/database.md
+304
View File
@@ -0,0 +1,304 @@
# Database Architecture
Claude-Mem uses SQLite 3 with the better-sqlite3 native module for persistent storage and FTS5 for full-text search.
## Database Location
- **Current**: `~/.claude-mem/claude-mem.db`
**Note**: Despite the README claiming v4.0.0+ moved the database to `${CLAUDE_PLUGIN_ROOT}/data/`, the actual implementation still uses `~/.claude-mem/`.
## Database Implementation
**Primary Implementation**: better-sqlite3 (native SQLite module)
- Used by: SessionStore and SessionSearch
- Format: Synchronous API with better performance
- **Note**: Database.ts (using bun:sqlite) is legacy code
## Core Tables
### 1. sdk_sessions
Tracks active and completed sessions.
```sql
CREATE TABLE sdk_sessions (
id INTEGER PRIMARY KEY AUTOINCREMENT,
sdk_session_id TEXT UNIQUE NOT NULL,
claude_session_id TEXT,
project TEXT NOT NULL,
prompt_counter INTEGER DEFAULT 0,
status TEXT NOT NULL DEFAULT 'active',
created_at TEXT NOT NULL,
created_at_epoch INTEGER NOT NULL,
completed_at TEXT,
completed_at_epoch INTEGER,
last_activity_at TEXT,
last_activity_epoch INTEGER
);
```
**Indexes**:
- `idx_sdk_sessions_claude_session` on `claude_session_id`
- `idx_sdk_sessions_project` on `project`
- `idx_sdk_sessions_status` on `status`
- `idx_sdk_sessions_created_at` on `created_at_epoch DESC`
### 2. observations
Individual tool executions with hierarchical structure.
```sql
CREATE TABLE observations (
id INTEGER PRIMARY KEY AUTOINCREMENT,
session_id TEXT NOT NULL,
sdk_session_id TEXT NOT NULL,
claude_session_id TEXT,
project TEXT NOT NULL,
prompt_number INTEGER,
tool_name TEXT NOT NULL,
correlation_id TEXT,
-- Hierarchical fields
title TEXT,
subtitle TEXT,
narrative TEXT,
text TEXT,
facts TEXT,
concepts TEXT,
type TEXT,
files_read TEXT,
files_modified TEXT,
created_at TEXT NOT NULL,
created_at_epoch INTEGER NOT NULL,
FOREIGN KEY (sdk_session_id) REFERENCES sdk_sessions(sdk_session_id)
);
```
**Observation Types**:
- `decision` - Architectural or design decisions
- `bugfix` - Bug fixes and corrections
- `feature` - New features or capabilities
- `refactor` - Code refactoring and cleanup
- `discovery` - Learnings about the codebase
- `change` - General changes and modifications
**Indexes**:
- `idx_observations_session` on `session_id`
- `idx_observations_sdk_session` on `sdk_session_id`
- `idx_observations_project` on `project`
- `idx_observations_tool_name` on `tool_name`
- `idx_observations_created_at` on `created_at_epoch DESC`
- `idx_observations_type` on `type`
### 3. session_summaries
AI-generated session summaries (multiple per session).
```sql
CREATE TABLE session_summaries (
id INTEGER PRIMARY KEY AUTOINCREMENT,
sdk_session_id TEXT NOT NULL,
claude_session_id TEXT,
project TEXT NOT NULL,
prompt_number INTEGER,
-- Summary fields
request TEXT,
investigated TEXT,
learned TEXT,
completed TEXT,
next_steps TEXT,
notes TEXT,
created_at TEXT NOT NULL,
created_at_epoch INTEGER NOT NULL,
FOREIGN KEY (sdk_session_id) REFERENCES sdk_sessions(sdk_session_id)
);
```
**Indexes**:
- `idx_session_summaries_sdk_session` on `sdk_session_id`
- `idx_session_summaries_project` on `project`
- `idx_session_summaries_created_at` on `created_at_epoch DESC`
### 4. user_prompts
Raw user prompts with FTS5 search (as of v4.2.0).
```sql
CREATE TABLE user_prompts (
id INTEGER PRIMARY KEY AUTOINCREMENT,
sdk_session_id TEXT NOT NULL,
claude_session_id TEXT,
project TEXT NOT NULL,
prompt_number INTEGER,
prompt_text TEXT NOT NULL,
created_at TEXT NOT NULL,
created_at_epoch INTEGER NOT NULL,
FOREIGN KEY (sdk_session_id) REFERENCES sdk_sessions(sdk_session_id)
);
```
**Indexes**:
- `idx_user_prompts_sdk_session` on `sdk_session_id`
- `idx_user_prompts_project` on `project`
- `idx_user_prompts_created_at` on `created_at_epoch DESC`
### Legacy Tables
- **sessions**: Legacy session tracking (v3.x)
- **memories**: Legacy compressed memory chunks (v3.x)
- **overviews**: Legacy session summaries (v3.x)
## FTS5 Full-Text Search
SQLite FTS5 (Full-Text Search) virtual tables enable fast full-text search across observations, summaries, and user prompts.
### FTS5 Virtual Tables
#### observations_fts
```sql
CREATE VIRTUAL TABLE observations_fts USING fts5(
title,
subtitle,
narrative,
text,
facts,
concepts,
content='observations',
content_rowid='id'
);
```
#### session_summaries_fts
```sql
CREATE VIRTUAL TABLE session_summaries_fts USING fts5(
request,
investigated,
learned,
completed,
next_steps,
notes,
content='session_summaries',
content_rowid='id'
);
```
#### user_prompts_fts
```sql
CREATE VIRTUAL TABLE user_prompts_fts USING fts5(
prompt_text,
content='user_prompts',
content_rowid='id'
);
```
### Automatic Synchronization
FTS5 tables stay in sync via triggers:
```sql
-- Insert trigger example
CREATE TRIGGER observations_ai AFTER INSERT ON observations BEGIN
INSERT INTO observations_fts(rowid, title, subtitle, narrative, text, facts, concepts)
VALUES (new.id, new.title, new.subtitle, new.narrative, new.text, new.facts, new.concepts);
END;
-- Update trigger example
CREATE TRIGGER observations_au AFTER UPDATE ON observations BEGIN
INSERT INTO observations_fts(observations_fts, rowid, title, subtitle, narrative, text, facts, concepts)
VALUES('delete', old.id, old.title, old.subtitle, old.narrative, old.text, old.facts, old.concepts);
INSERT INTO observations_fts(rowid, title, subtitle, narrative, text, facts, concepts)
VALUES (new.id, new.title, new.subtitle, new.narrative, new.text, new.facts, new.concepts);
END;
-- Delete trigger example
CREATE TRIGGER observations_ad AFTER DELETE ON observations BEGIN
INSERT INTO observations_fts(observations_fts, rowid, title, subtitle, narrative, text, facts, concepts)
VALUES('delete', old.id, old.title, old.subtitle, old.narrative, old.text, old.facts, old.concepts);
END;
```
### FTS5 Query Syntax
FTS5 supports rich query syntax:
- **Simple**: `"error handling"`
- **AND**: `"error" AND "handling"`
- **OR**: `"bug" OR "fix"`
- **NOT**: `"bug" NOT "feature"`
- **Phrase**: `"'exact phrase'"`
- **Column**: `title:"authentication"`
### Security
As of v4.2.3, all FTS5 queries are properly escaped to prevent SQL injection:
- Double quotes are escaped: `query.replace(/"/g, '""')`
- Comprehensive test suite with 332 injection attack tests
## Database Classes
### SessionStore
CRUD operations for sessions, observations, summaries, and user prompts.
**Location**: `src/services/sqlite/SessionStore.ts`
**Methods**:
- `createSession()`
- `getSession()`
- `updateSession()`
- `createObservation()`
- `getObservations()`
- `createSummary()`
- `getSummaries()`
- `createUserPrompt()`
### SessionSearch
FTS5 full-text search with 8 specialized search methods.
**Location**: `src/services/sqlite/SessionSearch.ts`
**Methods**:
- `searchObservations()` - Full-text search across observations
- `searchSessions()` - Full-text search across summaries
- `searchUserPrompts()` - Full-text search across user prompts
- `findByConcept()` - Find by concept tags
- `findByFile()` - Find by file references
- `findByType()` - Find by observation type
- `getRecentContext()` - Get recent session context
- `advancedSearch()` - Combined filters
## Migrations
Database schema is managed via migrations in `src/services/sqlite/migrations.ts`.
**Migration History**:
- Migration 001: Initial schema (sessions, memories, overviews, diagnostics, transcript_events)
- Migration 002: Hierarchical memory fields (title, subtitle, facts, concepts, files_touched)
- Migration 003: SDK sessions and observations
- Migration 004: Session summaries
- Migration 005: Multi-prompt sessions (prompt_counter, prompt_number)
- Migration 006: FTS5 virtual tables and triggers
- Migration 007-010: Various improvements and user prompts table
## Performance Considerations
- **Indexes**: All foreign keys and frequently queried columns are indexed
- **FTS5**: Full-text search is significantly faster than LIKE queries
- **Triggers**: Automatic synchronization has minimal overhead
- **Connection Pooling**: better-sqlite3 reuses connections efficiently
- **Synchronous API**: better-sqlite3 uses synchronous API for better performance
## Troubleshooting
See [Troubleshooting - Database Issues](../troubleshooting.md#database-issues) for common problems and solutions.
docs/architecture/hooks.md
+196
View File
@@ -0,0 +1,196 @@
# Plugin Hooks
Claude-Mem integrates with Claude Code through 5 lifecycle hooks that capture events and inject context.
## Hook Overview
| Hook Name | Purpose | Timeout | Script |
|---------------------|--------------------------------------|---------|-------------------------|
| SessionStart | Inject context from previous sessions| 120s | context-hook.js |
| UserPromptSubmit | Create/track new sessions | 120s | new-hook.js |
| PostToolUse | Capture tool execution observations | 120s | save-hook.js |
| Stop | Generate session summaries | 120s | summary-hook.js |
| SessionEnd | Mark sessions complete | 120s | cleanup-hook.js |
## Hook Configuration
Hooks are configured in `plugin/hooks/hooks.json`:
```json
{
"description": "Claude-mem memory system hooks",
"hooks": {
"SessionStart": [{
"hooks": [{
"type": "command",
"command": "cd \"${CLAUDE_PLUGIN_ROOT}/..\" && npm install --prefer-offline --no-audit --no-fund --loglevel=error && node ${CLAUDE_PLUGIN_ROOT}/scripts/context-hook.js",
"timeout": 120
}]
}],
"UserPromptSubmit": [{
"hooks": [{
"type": "command",
"command": "node ${CLAUDE_PLUGIN_ROOT}/scripts/new-hook.js",
"timeout": 120
}]
}],
"PostToolUse": [{
"matcher": "*",
"hooks": [{
"type": "command",
"command": "node ${CLAUDE_PLUGIN_ROOT}/scripts/save-hook.js",
"timeout": 120
}]
}],
"Stop": [{
"hooks": [{
"type": "command",
"command": "node ${CLAUDE_PLUGIN_ROOT}/scripts/summary-hook.js",
"timeout": 120
}]
}],
"SessionEnd": [{
"hooks": [{
"type": "command",
"command": "node ${CLAUDE_PLUGIN_ROOT}/scripts/cleanup-hook.js",
"timeout": 120
}]
}]
}
}
```
## 1. SessionStart Hook (`context-hook.js`)
**Purpose**: Inject context from previous sessions into Claude's initial context.
**Behavior**:
- Ensures dependencies are installed (runs fast idempotent npm install)
- Auto-starts PM2 worker service if not running
- Retrieves last 10 session summaries with three-tier verbosity (v4.2.0)
- Returns context via `hookSpecificOutput` in JSON format (fixed in v4.1.0)
**Input** (via stdin):
```json
{
"session_id": "claude-session-123",
"cwd": "/path/to/project",
"source": "startup"
}
```
**Output** (via stdout):
```json
{
"hookSpecificOutput": "# Recent Sessions\n\n## Session 1...\n"
}
```
**Implementation**: `src/hooks/context.ts` and `src/bin/hooks/context-hook.ts`
## 2. UserPromptSubmit Hook (`new-hook.js`)
**Purpose**: Create new session records and initialize session tracking.
**Behavior**:
- Creates new session in database
- Initializes session tracking
- Saves raw user prompts for full-text search (as of v4.2.0)
- Sends init signal to worker service
**Input** (via stdin):
```json
{
"session_id": "claude-session-123",
"cwd": "/path/to/project",
"prompt": "User's actual prompt text"
}
```
**Implementation**: `src/hooks/new.ts` and `src/bin/hooks/new-hook.ts`
## 3. PostToolUse Hook (`save-hook.js`)
**Purpose**: Capture tool execution observations.
**Behavior**:
- Fires after EVERY tool execution (Read, Write, Edit, Bash, etc.)
- Sends observations to worker service for processing
- Includes correlation IDs for tracing
- Filters low-value observations
**Input** (via stdin):
```json
{
"session_id": "claude-session-123",
"cwd": "/path/to/project",
"tool_name": "Read",
"tool_input": {...},
"tool_result": "...",
"correlation_id": "abc-123"
}
```
**Implementation**: `src/hooks/save.ts` and `src/bin/hooks/save-hook.ts`
## 4. Stop Hook (`summary-hook.js`)
**Purpose**: Generate session summaries when Claude stops.
**Behavior**:
- Triggers final summary generation
- Sends summarize request to worker service
- Summary includes: request, completed, learned, next_steps
**Input** (via stdin):
```json
{
"session_id": "claude-session-123",
"cwd": "/path/to/project",
"source": "user_stop"
}
```
**Implementation**: `src/hooks/summary.ts` and `src/bin/hooks/summary-hook.ts`
## 5. SessionEnd Hook (`cleanup-hook.js`)
**Purpose**: Mark sessions as completed (graceful cleanup as of v4.1.0).
**Behavior**:
- Marks sessions as completed
- Skips cleanup on `/clear` commands to preserve ongoing sessions
- Allows workers to finish pending operations naturally
- Previously sent DELETE requests; now uses graceful completion
**Input** (via stdin):
```json
{
"session_id": "claude-session-123",
"cwd": "/path/to/project",
"source": "normal"
}
```
**Implementation**: `src/hooks/cleanup.ts` and `src/bin/hooks/cleanup-hook.ts`
## Hook Development
### Adding a New Hook
1. Create hook implementation in `src/hooks/your-hook.ts`
2. Create entry point in `src/bin/hooks/your-hook.ts`
3. Add to `plugin/hooks/hooks.json`
4. Rebuild with `npm run build`
### Hook Best Practices
- **Fast execution**: Hooks should complete quickly (< 1s ideal)
- **Graceful degradation**: Don't block Claude if worker is down
- **Structured logging**: Use logger for debugging
- **Error handling**: Catch and log errors, don't crash
- **JSON output**: Use `hookSpecificOutput` for context injection
## Troubleshooting
See [Troubleshooting - Hook Issues](../troubleshooting.md#hook-issues) for common problems and solutions.
docs/architecture/mcp-search.md
+308
View File
@@ -0,0 +1,308 @@
# MCP Search Server
Claude-Mem includes a Model Context Protocol (MCP) server that exposes 7 specialized search tools for querying stored observations and sessions.
## Overview
- **Location**: `src/servers/search-server.ts`
- **Configuration**: `plugin/.mcp.json`
- **Transport**: stdio
- **Tools**: 7 specialized search functions
- **Citations**: All results use `claude-mem://` URI scheme
## Configuration
The MCP server is automatically registered via `plugin/.mcp.json`:
```json
{
"mcpServers": {
"claude-mem-search": {
"type": "stdio",
"command": "${CLAUDE_PLUGIN_ROOT}/scripts/search-server.js"
}
}
}
```
This registers the `claude-mem-search` server with Claude Code, making the 7 search tools available in all sessions. The server is automatically started when Claude Code launches and communicates via stdio transport.
## Search Tools
### 1. search_observations
Full-text search across observation titles, narratives, facts, and concepts.
**Parameters**:
- `query` (required): Search query for FTS5 full-text search
- `type`: Filter by observation type(s) (decision, bugfix, feature, refactor, discovery, change)
- `concepts`: Filter by concept tags
- `files`: Filter by file paths (partial match)
- `project`: Filter by project name
- `dateRange`: Filter by date range (`{start, end}`)
- `orderBy`: Sort order (relevance, date_desc, date_asc)
- `limit`: Maximum results (default: 20, max: 100)
- `offset`: Number of results to skip
- `format`: Output format ("index" for titles/dates only, "full" for complete details)
**Example**:
```
search_observations with query="build system" and type="decision"
```
### 2. search_sessions
Full-text search across session summaries, requests, and learnings.
**Parameters**:
- `query` (required): Search query for FTS5 full-text search
- `project`: Filter by project name
- `dateRange`: Filter by date range
- `orderBy`: Sort order (relevance, date_desc, date_asc)
- `limit`: Maximum results (default: 20, max: 100)
- `offset`: Number of results to skip
- `format`: Output format ("index" or "full")
**Example**:
```
search_sessions with query="hooks implementation"
```
### 3. search_user_prompts
Search raw user prompts with full-text search. Use this to find what the user actually said/requested across all sessions.
**Parameters**:
- `query` (required): Search query for FTS5 full-text search
- `project`: Filter by project name
- `dateRange`: Filter by date range
- `orderBy`: Sort order (relevance, date_desc, date_asc)
- `limit`: Maximum results (default: 20, max: 100)
- `offset`: Number of results to skip
- `format`: Output format ("index" for truncated prompts/dates, "full" for complete prompt text)
**Example**:
```
search_user_prompts with query="authentication feature"
```
**Benefits**:
- Full context reconstruction from user intent → Claude actions → outcomes
- Pattern detection for repeated requests
- Improved debugging by tracing from original user words to final implementation
### 4. find_by_concept
Find observations tagged with specific concepts.
**Parameters**:
- `concept` (required): Concept tag to search for
- `project`: Filter by project name
- `dateRange`: Filter by date range
- `orderBy`: Sort order (relevance, date_desc, date_asc)
- `limit`: Maximum results (default: 20, max: 100)
- `offset`: Number of results to skip
- `format`: Output format ("index" or "full")
**Example**:
```
find_by_concept with concept="architecture"
```
### 5. find_by_file
Find observations and sessions that reference specific file paths.
**Parameters**:
- `filePath` (required): File path to search for (supports partial matching)
- `project`: Filter by project name
- `dateRange`: Filter by date range
- `orderBy`: Sort order (relevance, date_desc, date_asc)
- `limit`: Maximum results (default: 20, max: 100)
- `offset`: Number of results to skip
- `format`: Output format ("index" or "full")
**Example**:
```
find_by_file with filePath="worker-service.ts"
```
### 6. find_by_type
Find observations by type (decision, bugfix, feature, refactor, discovery, change).
**Parameters**:
- `type` (required): Observation type(s) to filter by (single type or array)
- `project`: Filter by project name
- `dateRange`: Filter by date range
- `orderBy`: Sort order (relevance, date_desc, date_asc)
- `limit`: Maximum results (default: 20, max: 100)
- `offset`: Number of results to skip
- `format`: Output format ("index" or "full")
**Example**:
```
find_by_type with type=["decision", "feature"]
```
### 7. get_recent_context
Get recent session context including summaries and observations for a project.
**Parameters**:
- `project`: Project name (defaults to current working directory basename)
- `limit`: Number of recent sessions to retrieve (default: 3, max: 10)
**Example**:
```
get_recent_context with limit=5
```
## Output Formats
All search tools support two output formats:
### Index Format (Default)
Returns titles, dates, and source URIs only. Uses ~10x fewer tokens than full format.
**Always use index format first** to get an overview and identify relevant results.
**Example Output**:
```
1. [decision] Implement graceful session cleanup
Date: 2025-10-21 14:23:45
Source: claude-mem://observation/123
2. [feature] Add FTS5 full-text search
Date: 2025-10-21 13:15:22
Source: claude-mem://observation/124
```
### Full Format
Returns complete observation/summary details including narrative, facts, concepts, files, etc.
**Only use after reviewing index results** to dive deep into specific items of interest.
## Search Strategy
**Recommended Workflow**:
1. **Initial search**: Use default (index) format to see titles, dates, and sources
2. **Review results**: Identify which items are most relevant to your needs
3. **Deep dive**: Only then use `format: "full"` on specific items of interest
4. **Narrow down**: Use filters (type, dateRange, concepts, files) to refine results
**Token Efficiency**:
- Index format: ~50-100 tokens per result
- Full format: ~500-1000 tokens per result
- Start with 3-5 results to avoid MCP token limits
## Citations
All search results use the `claude-mem://` URI scheme for citations:
- `claude-mem://observation/{id}` - References specific observations
- `claude-mem://session/{id}` - References specific sessions
- `claude-mem://user-prompt/{id}` - References specific user prompts
These citations allow Claude to reference specific historical context in responses.
## FTS5 Query Syntax
The `query` parameter supports SQLite FTS5 full-text search syntax:
- **Simple**: `"error handling"`
- **AND**: `"error" AND "handling"`
- **OR**: `"bug" OR "fix"`
- **NOT**: `"bug" NOT "feature"`
- **Phrase**: `"'exact phrase'"`
- **Column**: `title:"authentication"`
## Security
As of v4.2.3, all FTS5 queries are properly escaped to prevent SQL injection attacks:
- Double quotes are escaped: `query.replace(/"/g, '""')`
- Comprehensive test suite with 332 injection attack tests
- Affects: `search_observations`, `search_sessions`, `search_user_prompts`
## Example Queries
```
# Find all decisions about build system
search_observations with query="build system" and type="decision"
# Show everything related to worker-service.ts
find_by_file with filePath="worker-service.ts"
# Search what we learned about hooks
search_sessions with query="hooks"
# Show observations tagged with 'architecture'
find_by_concept with concept="architecture"
# Find what user asked about authentication
search_user_prompts with query="authentication"
# Get recent context for debugging
get_recent_context with limit=5
```
## Implementation
The MCP search server is implemented using:
- `@modelcontextprotocol/sdk` (v1.20.1)
- `SessionSearch` service for FTS5 queries
- `SessionStore` for database access
- `zod-to-json-schema` for parameter validation
**Source Code**: `src/servers/search-server.ts`
## Troubleshooting
### Tool Not Available
If search tools are not available in Claude Code sessions:
1. Check MCP configuration:
```bash
cat plugin/.mcp.json
```
2. Verify search server is built:
```bash
ls -l plugin/scripts/search-server.js
```
3. Rebuild if needed:
```bash
npm run build
```
### Search Returns No Results
1. Check database has data:
```bash
sqlite3 ~/.claude-mem/claude-mem.db "SELECT COUNT(*) FROM observations;"
```
2. Verify FTS5 tables exist:
```bash
sqlite3 ~/.claude-mem/claude-mem.db "SELECT name FROM sqlite_master WHERE type='table' AND name LIKE '%_fts';"
```
3. Test query syntax:
```bash
# Simple query should work
search_observations with query="test"
```
### Token Limit Errors
If you hit MCP token limits:
1. Use `format: "index"` instead of `format: "full"`
2. Reduce `limit` parameter (try 3-5 instead of 20)
3. Use more specific filters to narrow results
4. Use `offset` to paginate through results
docs/architecture/overview.md
+160
View File
@@ -0,0 +1,160 @@
# Architecture Overview
## System Components
Claude-Mem operates as a Claude Code plugin with four core components:
1. **Plugin Hooks** - Capture lifecycle events
2. **Worker Service** - Process observations via Claude Agent SDK
3. **Database Layer** - Store sessions and observations (SQLite + FTS5)
4. **MCP Search Server** - Query historical context
## Technology Stack
| Layer | Technology |
|------------------------|-------------------------------------------|
| **Language** | TypeScript (ES2022, ESNext modules) |
| **Runtime** | Node.js 18+ |
| **Database** | SQLite 3 with better-sqlite3 driver |
| **HTTP Server** | Express.js 4.18 |
| **AI SDK** | @anthropic-ai/claude-agent-sdk |
| **Build Tool** | esbuild (bundles TypeScript) |
| **Process Manager** | PM2 |
| **Testing** | Node.js built-in test runner |
## Data Flow
### Memory Pipeline
```
Hook (stdin) → Database → Worker Service → SDK Processor → Database → Next Session Hook
```
1. **Input**: Claude Code sends tool execution data via stdin to hooks
2. **Storage**: Hooks write observations to SQLite database
3. **Processing**: Worker service reads observations, processes via SDK
4. **Output**: Processed summaries written back to database
5. **Retrieval**: Next session's context hook reads summaries from database
### Search Pipeline
```
Claude Request → MCP Server → SessionSearch Service → FTS5 Database → Search Results → Claude
```
1. **Query**: Claude uses MCP search tools (e.g., `search_observations`)
2. **Search**: MCP server calls SessionSearch service with query parameters
3. **FTS5**: Full-text search executes against FTS5 virtual tables
4. **Format**: Results formatted as `search_result` blocks with citations
5. **Return**: Claude receives citable search results for analysis
## Session Lifecycle
```
┌─────────────────────────────────────────────────────────────────┐
│ 1. Session Starts → Context Hook Fires │
│ Injects summaries from last 3 sessions into Claude's context │
└─────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────┐
│ 2. User Types Prompt → UserPromptSubmit Hook Fires │
│ Creates SDK session in database, notifies worker service │
└─────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────┐
│ 3. Claude Uses Tools → PostToolUse Hook Fires (100+ times) │
│ Sends observations to worker service for processing │
└─────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────┐
│ 4. Worker Processes → Claude Agent SDK Analyzes │
│ Extracts structured learnings via iterative AI processing │
└─────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────┐
│ 5. Claude Stops → Stop Hook Fires │
│ Generates final summary with request, status, next steps │
└─────────────────────────────────────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────────────┐
│ 6. Session Ends → Cleanup Hook Fires │
│ Marks session complete, ready for next session context │
└─────────────────────────────────────────────────────────────────┘
```
## Directory Structure
```
claude-mem/
├── src/
│ ├── bin/hooks/ # Entry point scripts for 5 hooks
│ │ ├── context-hook.ts # SessionStart
│ │ ├── new-hook.ts # UserPromptSubmit
│ │ ├── save-hook.ts # PostToolUse
│ │ ├── summary-hook.ts # Stop
│ │ └── cleanup-hook.ts # SessionEnd
│ │
│ ├── hooks/ # Hook implementation logic
│ │ ├── context.ts
│ │ ├── new.ts
│ │ ├── save.ts
│ │ ├── summary.ts
│ │ └── cleanup.ts
│ │
│ ├── servers/ # MCP servers
│ │ └── search-server.ts # MCP search tools server
│ │
│ ├── sdk/ # Claude Agent SDK integration
│ │ ├── prompts.ts # XML prompt builders
│ │ ├── parser.ts # XML response parser
│ │ └── worker.ts # Main SDK agent loop
│ │
│ ├── services/
│ │ ├── worker-service.ts # Express HTTP service
│ │ └── sqlite/ # Database layer
│ │ ├── SessionStore.ts # CRUD operations
│ │ ├── SessionSearch.ts # FTS5 search service
│ │ ├── migrations.ts
│ │ └── types.ts
│ │
│ ├── shared/ # Shared utilities
│ │ ├── config.ts
│ │ ├── paths.ts
│ │ └── storage.ts
│ │
│ └── utils/
│ ├── logger.ts
│ ├── platform.ts
│ └── port-allocator.ts
│
├── plugin/ # Plugin distribution
│ ├── .claude-plugin/
│ │ └── plugin.json
│ ├── .mcp.json # MCP server configuration
│ ├── hooks/
│ │ └── hooks.json
│ └── scripts/ # Built executables
│ ├── context-hook.js
│ ├── new-hook.js
│ ├── save-hook.js
│ ├── summary-hook.js
│ ├── cleanup-hook.js
│ ├── worker-service.cjs # Background worker
│ └── search-server.js # MCP search server
│
├── tests/ # Test suite
├── docs/ # Documentation
└── ecosystem.config.cjs # PM2 configuration
```
## Component Details
### 1. Plugin Hooks
See [hooks.md](hooks.md) for detailed hook documentation.
### 2. Worker Service
See [worker-service.md](worker-service.md) for HTTP API and endpoints.
### 3. Database Layer
See [database.md](database.md) for schema and FTS5 search.
### 4. MCP Search Server
See [mcp-search.md](mcp-search.md) for search tools and examples.
docs/architecture/worker-service.md
+243
View File
@@ -0,0 +1,243 @@
# Worker Service
The worker service is a long-running HTTP API built with Express.js and managed by PM2. It processes observations through the Claude Agent SDK separately from hook execution to prevent timeout issues.
## Overview
- **Technology**: Express.js HTTP server
- **Process Manager**: PM2
- **Port**: Fixed port 37777 (configurable via `CLAUDE_MEM_WORKER_PORT`)
- **Location**: `src/services/worker-service.ts`
- **Built Output**: `plugin/scripts/worker-service.cjs`
- **Model**: Configurable via `CLAUDE_MEM_MODEL` environment variable (default: claude-sonnet-4-5)
## REST API Endpoints
The worker service exposes 6 HTTP endpoints:
### 1. Health Check
```
GET /health
```
**Response**:
```json
{
"status": "ok",
"uptime": 12345,
"port": 37777
}
```
### 2. Initialize Session
```
POST /sessions/:sessionDbId/init
```
**Request Body**:
```json
{
"sdk_session_id": "abc-123",
"project": "my-project"
}
```
**Response**:
```json
{
"success": true,
"session_id": "abc-123"
}
```
### 3. Add Observation
```
POST /sessions/:sessionDbId/observations
```
**Request Body**:
```json
{
"tool_name": "Read",
"tool_input": {...},
"tool_result": "...",
"correlation_id": "xyz-789"
}
```
**Response**:
```json
{
"success": true,
"observation_id": 123
}
```
### 4. Generate Summary
```
POST /sessions/:sessionDbId/summarize
```
**Request Body**:
```json
{
"trigger": "stop"
}
```
**Response**:
```json
{
"success": true,
"summary_id": 456
}
```
### 5. Session Status
```
GET /sessions/:sessionDbId/status
```
**Response**:
```json
{
"session_id": "abc-123",
"status": "active",
"observation_count": 42,
"summary_count": 1
}
```
### 6. Delete Session
```
DELETE /sessions/:sessionDbId
```
**Response**:
```json
{
"success": true
}
```
**Note**: As of v4.1.0, the cleanup hook no longer calls this endpoint. Sessions are marked complete instead of deleted to allow graceful worker shutdown.
## PM2 Management
### Configuration
The worker is configured via `ecosystem.config.cjs`:
```javascript
module.exports = {
apps: [{
name: 'claude-mem-worker',
script: './plugin/scripts/worker-service.cjs',
instances: 1,
autorestart: true,
watch: false,
max_memory_restart: '1G',
env: {
NODE_ENV: 'production',
FORCE_COLOR: '1'
}
}]
};
```
### Commands
```bash
# Start worker (auto-starts on first session)
npm run worker:start
# Stop worker
npm run worker:stop
# Restart worker
npm run worker:restart
# View logs
npm run worker:logs
# Check status
npm run worker:status
```
### Auto-Start Behavior
As of v4.0.0, the worker service auto-starts when the SessionStart hook fires. Manual start is optional.
## Claude Agent SDK Integration
The worker service routes observations to the Claude Agent SDK for AI-powered processing:
### Processing Flow
1. **Observation Queue**: Observations accumulate in memory
2. **SDK Processing**: Observations sent to Claude via Agent SDK
3. **XML Parsing**: Responses parsed for structured data
4. **Database Storage**: Processed observations stored in SQLite
### SDK Components
- **Prompts** (`src/sdk/prompts.ts`): Builds XML-structured prompts
- **Parser** (`src/sdk/parser.ts`): Parses Claude's XML responses
- **Worker** (`src/sdk/worker.ts`): Main SDK agent loop
### Model Configuration
Set the AI model used for processing via environment variable:
```bash
export CLAUDE_MEM_MODEL=claude-sonnet-4-5
```
Available models:
- `claude-haiku-4-5` - Fast, cost-efficient
- `claude-sonnet-4-5` - Balanced (default)
- `claude-opus-4` - Most capable
- `claude-3-7-sonnet` - Alternative version
## Port Allocation
The worker uses a fixed port (37777 by default) for consistent communication:
- **Default**: Port 37777
- **Override**: Set `CLAUDE_MEM_WORKER_PORT` environment variable
- **Port File**: `${CLAUDE_PLUGIN_ROOT}/data/worker.port` tracks current port
If port 37777 is in use, the worker will fail to start. Set a custom port via environment variable.
## Data Storage
The worker service stores data in the plugin data directory:
```
${CLAUDE_PLUGIN_ROOT}/data/
├── claude-mem.db # SQLite database
├── worker.port # Current worker port file
└── logs/
├── worker-out.log # Worker stdout logs
└── worker-error.log # Worker stderr logs
```
## Error Handling
The worker implements graceful degradation:
- **Database Errors**: Logged but don't crash the service
- **SDK Errors**: Retried with exponential backoff
- **Network Errors**: Logged and skipped
- **Invalid Input**: Validated and rejected with error response
## Performance
- **Async Processing**: Observations processed asynchronously
- **In-Memory Queue**: Fast observation accumulation
- **Batch Processing**: Multiple observations processed together
- **Connection Pooling**: SQLite connections reused
## Troubleshooting
See [Troubleshooting - Worker Issues](../troubleshooting.md#worker-service-issues) for common problems and solutions.
docs/configuration.md
+298
View File
@@ -0,0 +1,298 @@
# Configuration
## Environment Variables
| Variable | Default | Description |
|-------------------------|---------------------------------|---------------------------------------|
| `CLAUDE_PLUGIN_ROOT` | Set by Claude Code | Plugin installation directory |
| `CLAUDE_MEM_DATA_DIR` | `~/.claude-mem/` | Data directory (dev override) |
| `CLAUDE_MEM_WORKER_PORT`| `37777` | Worker service port |
| `CLAUDE_MEM_MODEL` | `claude-sonnet-4-5` | AI model for processing observations |
| `NODE_ENV` | `production` | Environment mode |
| `FORCE_COLOR` | `1` | Enable colored logs |
## Model Configuration
Configure which AI model processes your observations.
### Available Models
- `claude-haiku-4-5` - Fast, cost-efficient
- `claude-sonnet-4-5` - Balanced (default)
- `claude-opus-4` - Most capable
- `claude-3-7-sonnet` - Alternative version
### Using the Interactive Script
```bash
./claude-mem-settings.sh
```
This script manages `CLAUDE_MEM_MODEL` in `~/.claude/settings.json`.
### Manual Configuration
Edit `~/.claude/settings.json`:
```json
{
"CLAUDE_MEM_MODEL": "claude-sonnet-4-5"
}
```
## Files and Directories
### Data Directory Structure
```
~/.claude-mem/
├── claude-mem.db # SQLite database
├── worker.port # Current worker port file
└── logs/
├── worker-out.log # Worker stdout logs
└── worker-error.log # Worker stderr logs
```
### Plugin Directory Structure
```
${CLAUDE_PLUGIN_ROOT}/
├── .claude-plugin/
│ └── plugin.json # Plugin metadata
├── .mcp.json # MCP server configuration
├── hooks/
│ └── hooks.json # Hook configuration
└── scripts/ # Built executables
├── context-hook.js
├── new-hook.js
├── save-hook.js
├── summary-hook.js
├── cleanup-hook.js
├── worker-service.cjs
└── search-server.js
```
## Plugin Configuration
### Hooks Configuration
Hooks are configured in `plugin/hooks/hooks.json`:
```json
{
"description": "Claude-mem memory system hooks",
"hooks": {
"SessionStart": [{
"hooks": [{
"type": "command",
"command": "cd \"${CLAUDE_PLUGIN_ROOT}/..\" && npm install --prefer-offline --no-audit --no-fund --loglevel=error && node ${CLAUDE_PLUGIN_ROOT}/scripts/context-hook.js",
"timeout": 120
}]
}],
"UserPromptSubmit": [{
"hooks": [{
"type": "command",
"command": "node ${CLAUDE_PLUGIN_ROOT}/scripts/new-hook.js",
"timeout": 120
}]
}],
"PostToolUse": [{
"matcher": "*",
"hooks": [{
"type": "command",
"command": "node ${CLAUDE_PLUGIN_ROOT}/scripts/save-hook.js",
"timeout": 120
}]
}],
"Stop": [{
"hooks": [{
"type": "command",
"command": "node ${CLAUDE_PLUGIN_ROOT}/scripts/summary-hook.js",
"timeout": 120
}]
}],
"SessionEnd": [{
"hooks": [{
"type": "command",
"command": "node ${CLAUDE_PLUGIN_ROOT}/scripts/cleanup-hook.js",
"timeout": 120
}]
}]
}
}
```
### MCP Server Configuration
The MCP search server is configured in `plugin/.mcp.json`:
```json
{
"mcpServers": {
"claude-mem-search": {
"type": "stdio",
"command": "${CLAUDE_PLUGIN_ROOT}/scripts/search-server.js"
}
}
}
```
This registers the `claude-mem-search` server with Claude Code, making the 7 search tools available in all sessions.
## PM2 Configuration
Worker service is managed by PM2 via `ecosystem.config.cjs`:
```javascript
module.exports = {
apps: [{
name: 'claude-mem-worker',
script: './plugin/scripts/worker-service.cjs',
instances: 1,
autorestart: true,
watch: false,
max_memory_restart: '1G',
env: {
NODE_ENV: 'production',
FORCE_COLOR: '1'
}
}]
};
```
### PM2 Settings
- **instances**: 1 (single instance)
- **autorestart**: true (auto-restart on crash)
- **watch**: false (no file watching)
- **max_memory_restart**: 1G (restart if memory exceeds 1GB)
## Customization
### Custom Data Directory
For development or testing, override the data directory:
```bash
export CLAUDE_MEM_DATA_DIR=/custom/path
```
### Custom Worker Port
If port 37777 is in use:
```bash
export CLAUDE_MEM_WORKER_PORT=38000
npm run worker:restart
```
### Custom Model
Use a different AI model:
```bash
export CLAUDE_MEM_MODEL=claude-opus-4
npm run worker:restart
```
## Advanced Configuration
### Hook Timeouts
Modify timeouts in `plugin/hooks/hooks.json`:
```json
{
"timeout": 120 // Default: 120 seconds
}
```
Recommended values:
- SessionStart: 120s (needs time for npm install and context retrieval)
- UserPromptSubmit: 60s
- PostToolUse: 120s (can process many observations)
- Stop: 60s
- SessionEnd: 60s
### Worker Memory Limit
Modify PM2 memory limit in `ecosystem.config.cjs`:
```javascript
{
max_memory_restart: '2G' // Increase if needed
}
```
### Logging Verbosity
Enable debug logging:
```bash
export DEBUG=claude-mem:*
npm run worker:restart
npm run worker:logs
```
## Configuration Best Practices
1. **Use defaults**: Default configuration works for most use cases
2. **Override selectively**: Only change what you need
3. **Document changes**: Keep track of custom configurations
4. **Test after changes**: Verify worker restarts successfully
5. **Monitor logs**: Check worker logs after configuration changes
## Troubleshooting Configuration
### Configuration Not Applied
1. Restart worker after changes:
```bash
npm run worker:restart
```
2. Verify environment variables:
```bash
echo $CLAUDE_MEM_MODEL
echo $CLAUDE_MEM_WORKER_PORT
```
3. Check worker logs:
```bash
npm run worker:logs
```
### Invalid Model Name
If you specify an invalid model name, the worker will fall back to `claude-sonnet-4-5` and log a warning.
Valid models:
- claude-haiku-4-5
- claude-sonnet-4-5
- claude-opus-4
- claude-3-7-sonnet
### Port Already in Use
If port 37777 is already in use:
1. Set custom port:
```bash
export CLAUDE_MEM_WORKER_PORT=38000
```
2. Restart worker:
```bash
npm run worker:restart
```
3. Verify new port:
```bash
cat ~/.claude-mem/worker.port
```
## Next Steps
- [Architecture Overview](architecture/overview.md) - Understand the system
- [Troubleshooting](troubleshooting.md) - Common issues
- [Development](development.md) - Building from source
docs/development.md
+557
View File
@@ -0,0 +1,557 @@
# Development Guide
## Building from Source
### Prerequisites
- Node.js 18.0.0 or higher
- npm (comes with Node.js)
- Git
### Clone and Build
```bash
# Clone repository
git clone https://github.com/thedotmack/claude-mem.git
cd claude-mem
# Install dependencies
npm install
# Build all components
npm run build
```
### Build Process
The build process uses esbuild to compile TypeScript:
1. Compiles TypeScript to JavaScript
2. Creates standalone executables for each hook in `plugin/scripts/`
3. Bundles MCP search server to `plugin/scripts/search-server.js`
4. Bundles worker service to `plugin/scripts/worker-service.cjs`
**Build Output**:
- Hook executables: `*-hook.js` (ESM format)
- Worker service: `worker-service.cjs` (CJS format)
- Search server: `search-server.js` (ESM format)
### Build Scripts
```bash
# Build everything
npm run build
# Build only hooks
npm run build:hooks
# The build script is defined in scripts/build-hooks.js
```
## Development Workflow
### 1. Make Changes
Edit TypeScript source files in `src/`:
```
src/
├── bin/hooks/ # Hook entry points
├── hooks/ # Hook implementations
├── services/ # Worker service and database
├── servers/ # MCP search server
├── sdk/ # Claude Agent SDK integration
├── shared/ # Shared utilities
└── utils/ # General utilities
```
### 2. Build
```bash
npm run build
```
### 3. Test
```bash
# Run all tests
npm test
# Test specific file
node --test tests/session-lifecycle.test.ts
# Test context injection
npm run test:context
# Verbose context test
npm run test:context:verbose
```
### 4. Manual Testing
```bash
# Start worker manually
npm run worker:start
# Check worker status
npm run worker:status
# View logs
npm run worker:logs
# Test hooks manually
echo '{"session_id":"test-123","cwd":"'$(pwd)'","source":"startup"}' | node plugin/scripts/context-hook.js
```
### 5. Iterate
Repeat steps 1-4 until your changes work as expected.
## Adding New Features
### Adding a New Hook
1. Create hook implementation in `src/hooks/your-hook.ts`:
```typescript
import { HookInput } from './types';
export async function yourHook(input: HookInput) {
// Hook implementation
return {
hookSpecificOutput: 'Optional output'
};
}
```
2. Create entry point in `src/bin/hooks/your-hook.ts`:
```typescript
#!/usr/bin/env node
import { readStdin } from '../../shared/stdin';
import { yourHook } from '../../hooks/your-hook';
async function main() {
const input = await readStdin();
const result = await yourHook(input);
console.log(JSON.stringify(result));
}
main().catch(console.error);
```
3. Add to `plugin/hooks/hooks.json`:
```json
{
"YourHook": [{
"hooks": [{
"type": "command",
"command": "node ${CLAUDE_PLUGIN_ROOT}/scripts/your-hook.js",
"timeout": 120
}]
}]
}
```
4. Rebuild:
```bash
npm run build
```
### Modifying Database Schema
1. Add migration to `src/services/sqlite/migrations.ts`:
```typescript
export const migration011: Migration = {
version: 11,
up: (db: Database) => {
db.run(`
ALTER TABLE observations ADD COLUMN new_field TEXT;
`);
},
down: (db: Database) => {
// Optional: define rollback
}
};
```
2. Update types in `src/services/sqlite/types.ts`:
```typescript
export interface Observation {
// ... existing fields
new_field?: string;
}
```
3. Update database methods in `src/services/sqlite/SessionStore.ts`:
```typescript
createObservation(obs: Observation) {
// Include new_field in INSERT
}
```
4. Test migration:
```bash
# Backup database first!
cp ~/.claude-mem/claude-mem.db ~/.claude-mem/claude-mem.db.backup
# Run tests
npm test
```
### Extending SDK Prompts
1. Modify prompts in `src/sdk/prompts.ts`:
```typescript
export function buildObservationPrompt(observation: Observation): string {
return `
<observation>
<!-- Add new XML structure -->
</observation>
`;
}
```
2. Update parser in `src/sdk/parser.ts`:
```typescript
export function parseObservation(xml: string): ParsedObservation {
// Parse new XML fields
}
```
3. Test:
```bash
npm test
```
### Adding MCP Search Tools
1. Add tool definition in `src/servers/search-server.ts`:
```typescript
server.setRequestHandler(CallToolRequestSchema, async (request) => {
if (request.params.name === 'your_new_tool') {
// Implement tool logic
const results = await search.yourNewSearch(params);
return formatResults(results);
}
});
```
2. Add search method in `src/services/sqlite/SessionSearch.ts`:
```typescript
yourNewSearch(params: YourParams): SearchResult[] {
// Implement FTS5 search
}
```
3. Rebuild and test:
```bash
npm run build
npm test
```
## Testing
### Running Tests
```bash
# All tests
npm test
# Specific test file
node --test tests/your-test.test.ts
# With coverage (if configured)
npm test -- --coverage
```
### Writing Tests
Create test files in `tests/`:
```typescript
import { describe, it } from 'node:test';
import assert from 'node:assert';
describe('YourFeature', () => {
it('should do something', () => {
// Test implementation
assert.strictEqual(result, expected);
});
});
```
### Test Database
Use a separate test database:
```typescript
import { SessionStore } from '../src/services/sqlite/SessionStore';
const store = new SessionStore(':memory:'); // In-memory database
```
## Code Style
### TypeScript Guidelines
- Use TypeScript strict mode
- Define interfaces for all data structures
- Use async/await for asynchronous code
- Handle errors explicitly
- Add JSDoc comments for public APIs
### Formatting
- Follow existing code formatting
- Use 2-space indentation
- Use single quotes for strings
- Add trailing commas in objects/arrays
### Example
```typescript
/**
* Create a new observation in the database
*/
export async function createObservation(
obs: Observation
): Promise<number> {
try {
const result = await db.insert('observations', {
session_id: obs.session_id,
tool_name: obs.tool_name,
// ...
});
return result.id;
} catch (error) {
logger.error('Failed to create observation', error);
throw error;
}
}
```
## Debugging
### Enable Debug Logging
```bash
export DEBUG=claude-mem:*
npm run worker:restart
npm run worker:logs
```
### Inspect Database
```bash
sqlite3 ~/.claude-mem/claude-mem.db
# View schema
.schema observations
# Query data
SELECT * FROM observations LIMIT 10;
```
### Trace Observations
Use correlation IDs to trace observations through the pipeline:
```bash
sqlite3 ~/.claude-mem/claude-mem.db
SELECT correlation_id, tool_name, created_at
FROM observations
WHERE session_id = 'YOUR_SESSION_ID'
ORDER BY created_at;
```
### Debug Hooks
Run hooks manually with test input:
```bash
# Test context hook
echo '{"session_id":"test-123","cwd":"'$(pwd)'","source":"startup"}' | node plugin/scripts/context-hook.js
# Test new hook
echo '{"session_id":"test-123","cwd":"'$(pwd)'","prompt":"test"}' | node plugin/scripts/new-hook.js
```
## Publishing
### NPM Publishing
```bash
# Update version in package.json
npm version patch # or minor, or major
# Build
npm run build
# Publish to NPM
npm run release
```
The `release` script:
1. Runs tests
2. Builds all components
3. Publishes to NPM registry
### Creating a Release
1. Update version in `package.json`
2. Update `CHANGELOG.md`
3. Commit changes
4. Create git tag
5. Push to GitHub
6. Publish to NPM
```bash
# Update version
npm version 4.2.4
# Update changelog
# Edit CHANGELOG.md manually
# Commit
git add .
git commit -m "chore: Release v4.2.4"
# Tag
git tag v4.2.4
# Push
git push origin main --tags
# Publish to NPM
npm run release
```
## Contributing
### Contribution Workflow
1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Make your changes
4. Write tests
5. Update documentation
6. Commit your changes (`git commit -m 'Add amazing feature'`)
7. Push to the branch (`git push origin feature/amazing-feature`)
8. Open a Pull Request
### Pull Request Guidelines
- **Clear title**: Describe what the PR does
- **Description**: Explain why the change is needed
- **Tests**: Include tests for new features
- **Documentation**: Update docs as needed
- **Changelog**: Add entry to CHANGELOG.md
- **Commits**: Use clear, descriptive commit messages
### Code Review Process
1. Automated tests must pass
2. Code review by maintainer
3. Address feedback
4. Final approval
5. Merge to main
## Development Tools
### Recommended VSCode Extensions
- TypeScript
- ESLint
- Prettier
- SQLite Viewer
### Useful Commands
```bash
# Check TypeScript types
npx tsc --noEmit
# Lint code (if configured)
npm run lint
# Format code (if configured)
npm run format
# Clean build artifacts
rm -rf plugin/scripts/*.js plugin/scripts/*.cjs
```
## Troubleshooting Development
### Build Fails
1. Clean node_modules:
```bash
rm -rf node_modules
npm install
```
2. Check Node.js version:
```bash
node --version # Should be >= 18.0.0
```
3. Check for syntax errors:
```bash
npx tsc --noEmit
```
### Tests Fail
1. Check database:
```bash
rm ~/.claude-mem/claude-mem.db
npm test
```
2. Check worker status:
```bash
npm run worker:status
```
3. View logs:
```bash
npm run worker:logs
```
### Worker Won't Start
1. Kill existing process:
```bash
pm2 delete claude-mem-worker
```
2. Check port:
```bash
lsof -i :37777
```
3. Try custom port:
```bash
export CLAUDE_MEM_WORKER_PORT=38000
npm run worker:start
```
## Next Steps
- [Architecture Overview](architecture/overview.md) - Understand the system
- [Configuration](configuration.md) - Customize Claude-Mem
- [Troubleshooting](troubleshooting.md) - Common issues
docs/installation.md
+108
View File
@@ -0,0 +1,108 @@
# Installation Guide
## Quick Start
Install Claude-Mem directly from the plugin marketplace:
```bash
/plugin marketplace add thedotmack/claude-mem
/plugin install claude-mem
```
That's it! The plugin will automatically:
- Download prebuilt binaries (no compilation needed)
- Install all dependencies (including PM2 and SQLite binaries)
- Configure hooks for session lifecycle management
- Set up the MCP search server
- Auto-start the worker service on first session
Start a new Claude Code session and you'll see context from previous sessions automatically loaded.
## System Requirements
- **Node.js**: 18.0.0 or higher
- **Claude Code**: Latest version with plugin support
- **PM2**: Process manager (bundled with plugin - no global install required)
- **SQLite 3**: For persistent storage (bundled)
## Advanced Installation
For development or testing, you can clone and build from source:
### Clone and Build
```bash
# Clone the repository
git clone https://github.com/thedotmack/claude-mem.git
cd claude-mem
# Install dependencies
npm install
# Build hooks and worker service
npm run build
# Worker service will auto-start on first Claude Code session
# Or manually start with:
npm run worker:start
# Verify worker is running
npm run worker:status
```
### Post-Installation Verification
#### 1. Automatic Dependency Installation
Dependencies are installed automatically during plugin installation. The SessionStart hook also ensures dependencies are up-to-date on each session start (this is fast and idempotent). Works cross-platform on Windows, macOS, and Linux.
#### 2. Verify Plugin Installation
Check that hooks are configured in Claude Code:
```bash
cat plugin/hooks/hooks.json
```
#### 3. Data Directory Location
v4.0.0+ stores data in `${CLAUDE_PLUGIN_ROOT}/data/`:
- Database: `${CLAUDE_PLUGIN_ROOT}/data/claude-mem.db`
- Worker port file: `${CLAUDE_PLUGIN_ROOT}/data/worker.port`
- Logs: `${CLAUDE_PLUGIN_ROOT}/data/logs/`
For development/testing, you can override:
```bash
export CLAUDE_MEM_DATA_DIR=/custom/path
```
#### 4. Check Worker Logs
```bash
npm run worker:logs
```
#### 5. Test Context Retrieval
```bash
npm run test:context
```
## Upgrading from v3.x
**BREAKING CHANGES - Please Read:**
v4.0.0 introduces breaking changes:
- **Data Location Changed**: Database moved from `~/.claude-mem/` to `${CLAUDE_PLUGIN_ROOT}/data/` (inside plugin directory)
- **Fresh Start Required**: No automatic migration from v3.x. You must start with a clean database
- **Worker Auto-Starts**: Worker service now starts automatically - no manual `npm run worker:start` needed
- **MCP Search Server**: 7 new search tools with full-text search and citations
- **Enhanced Architecture**: Improved plugin integration and data organization
See [CHANGELOG.md](../CHANGELOG.md) for complete details.
## Next Steps
- [Getting Started Guide](usage/getting-started.md) - Learn how Claude-Mem works automatically
- [MCP Search Tools](usage/search-tools.md) - Query your project history
- [Configuration](configuration.md) - Customize Claude-Mem behavior
docs/prompts/drafts/_prompt-analysis.md
-444
View File
@@ -1,444 +0,0 @@
# Prompt Flow Analysis & Rankings
## Rating System
- ✅ **Smart**: Well-designed, clear purpose, effective
- ⚠️ **Problematic**: Has issues but salvageable
- ❌ **Stupid**: Poorly designed, confusing, or counterproductive
- 🧠 **Context Poison**: Will confuse the AI or create inconsistent behavior
- 🔍 **No Clear Purpose**: Exists but unclear why
- 🎯 **Clarity Score**: 1-10 (10 = crystal clear, 1 = incomprehensible)
---
## Element-by-Element Comparison
### INIT PROMPTS (Session Start)
#### CURRENT: "You are a memory processor"
```
You will PROCESS tool executions during this Claude Code session. Your job is to:
1. ANALYZE each tool response for meaningful content
2. DECIDE whether it contains something worth storing
3. EXTRACT the key insight
4. STORE it as an observation in the XML format below
For MOST meaningful tool outputs, you should generate an observation. Only skip truly routine operations.
```
**Rating**: ❌ **Stupid** + 🧠 **Context Poison**
**Clarity**: 3/10
**Issues**:
1. "For MOST" is ambiguous - does that mean 51%? 80%? 95%?
2. Creates bias toward over-storage (fear of missing things)
3. Contradicts "Only skip truly routine operations" later in prompt
4. No clear guidance on what "meaningful" actually means
5. "Only skip truly routine" implies almost everything should be stored
**Why Context Poison**:
- Agent will second-guess every decision
- Creates inconsistent thresholds across sessions
- User gets frustrated with noise
---
#### OLD: "You are a semantic memory compressor"
```
## FIRST: Generate Session Title
IMMEDIATELY generate a title and subtitle for this session based on the user request.
## THEN: Process Tool Responses
You will receive a stream of tool responses. For each one:
1. ANALYZE: Does this contain information worth remembering?
2. DECIDE: Should I store this or skip it?
3. EXTRACT: What are the key semantic concepts?
4. DECOMPOSE: Break into title + subtitle + atomic facts + narrative
5. STORE: Use bash to save the hierarchical memory
6. TRACK: Keep count of stored memories (001, 002, 003...)
# IMPORTANT REMINDERS
- Be selective - quality over quantity
```
**Rating**: ⚠️ **Problematic** but contains ✅ **Smart** elements
**Clarity**: 6/10
**Issues**:
1. "IMMEDIATELY" vs "THEN" creates ordering confusion
2. Session title generation is unclear when it should happen
3. Bash tool dependency is fragile
4. Manual counter tracking is error-prone
**Smart Elements**:
1. "Quality over quantity" is clear directive
2. Hierarchical decomposition gives structure
3. Explicit state tracking (counter)
4. "Be selective" is unambiguous
**Verdict**: The philosophy is better (selective, quality-focused), execution is messier (bash commands, ordering confusion)
---
### OBSERVATION PROMPTS
#### CURRENT: buildObservationPrompt
```
ANALYSIS TASK
-------------
ANALYZE this tool response and DECIDE: Does it contain something worth storing?
Most Read, Edit, Grep, Bash, and Write operations contain meaningful content.
If this contains something worth remembering, output the observation...
```
**Rating**: ❌ **Stupid** + 🧠 **Context Poison**
**Clarity**: 4/10
**Issues**:
1. "Most Read, Edit, Grep, Bash, and Write operations contain meaningful content" - Why repeat this per observation?
2. Contradicts init prompt's "be selective" guidance
3. Creates bias: agent thinks "well, this is a Read, so I should probably store it"
4. No guidance on WHAT to extract from the content
5. One-sentence observations lose context
**Why Context Poison**:
- Tool-type bias ("It's a Read, so store it") instead of content-based evaluation
- Encourages lazy pattern matching instead of semantic analysis
- Results in garbage like "Read package.json file" with no insight
---
#### OLD: Tool Message
```
# Tool Response {timeFormatted}
Tool: {toolName}
User Context: "{userPrompt}"
```
{toolResponse}
```
Analyze and store if meaningful.
```
**Rating**: ✅ **Smart** (minimal, non-leading)
**Clarity**: 8/10
**Smart Elements**:
1. Doesn't tell agent what to think about tool types
2. Trusts agent's judgment from system prompt
3. Short and clear
4. Includes user context for relevance filtering
**Issue**:
1. Maybe TOO minimal - no reminder of format requirements
---
### STORAGE FORMATS
#### CURRENT: XML Observations
```xml
<observation>
<type>feature</type>
<text>Implemented JWT token refresh flow with 7-day expiry</text>
</observation>
```
**Rating**: ⚠️ **Problematic**
**Clarity**: 7/10
**Issues**:
1. One sentence only - loses narrative context
2. Five types (decision, bugfix, feature, refactor, discovery) - are these actually useful categories?
3. No file associations
4. No concept tagging
5. Flat structure - all observations equal weight
**Smart Elements**:
1. Simple to parse
2. Structured typing
3. Regex-parseable
---
#### OLD: Hierarchical Memory (4 levels)
```bash
--title "SDK Transcript Auto-Cleanup"
--subtitle "Automatic deletion of SDK transcripts after completion prevents memory conversations from appearing in UI history"
--facts '["stop-streaming.js: Deletes SDK transcript after overview generation", "Path: ~/.claude/projects/{sanitized-cwd}/{sessionId}.jsonl"]'
--concepts '["cleanup", "SDK-lifecycle", "UX"]'
--files '["hooks/stop-streaming.js"]'
```
**Rating**: ✅ **Smart** (structure) but ❌ **Stupid** (execution via bash)
**Clarity**: 8/10 (concept), 3/10 (implementation)
**Smart Elements**:
1. Multiple levels of granularity (title → subtitle → facts → narrative)
2. Atomic facts enable precise retrieval
3. File associations explicit
4. Concept tags for categorization
5. Subtitle gives the "why it matters"
**Stupid Elements**:
1. Bash command execution is fragile
2. Quote escaping nightmare
3. Manual counter tracking
4. JSON in bash arguments is error-prone
**Verdict**: Great data model, terrible implementation
---
### SUMMARY/FINALIZE PROMPTS
#### CURRENT: buildFinalizePrompt (per prompt)
```xml
<summary>
<request>Implement JWT authentication system</request>
<investigated>Existing auth middleware, session management</investigated>
<learned>Current system uses session cookies; no JWT support</learned>
<completed>Implemented JWT token + refresh flow</completed>
<next_steps>Add token revocation API endpoint</next_steps>
<files_read><file>src/auth.ts</file></files_read>
<files_edited><file>src/auth.ts</file></files_edited>
<notes>Token secret stored in .env</notes>
</summary>
```
**Rating**: ✅ **Smart** (structure) but 🔍 **No Clear Purpose** (frequency)
**Clarity**: 9/10
**Smart Elements**:
1. Structured format with clear fields
2. Tracks what was learned (semantic value)
3. Files read/edited tracked explicitly
4. Next steps captured
**Issues**:
1. Generated PER PROMPT - is this too granular?
2. Will create many summaries per session
3. Unclear how these summaries are used
4. No aggregation across prompts
**Question**: Should this be per-session instead of per-prompt?
---
#### OLD: Session Overview (per session)
```bash
claude-mem store-overview --project "{project}" --session "{sessionId}" --content "2-3 sentence overview"
```
**Rating**: ⚠️ **Problematic**
**Clarity**: 5/10
**Issues**:
1. Only 2-3 sentences - very lossy
2. No structured fields
3. Happens once at end - loses per-prompt context
4. Relies on agent's memory of entire session
**Smart Element**:
1. One overview per session (not noisy)
---
### DECISION GUIDANCE
#### CURRENT: What to Store/Skip
```
Store these:
✓ File contents with logic, algorithms, or patterns
✓ Search results revealing project structure
✓ Build errors or test failures with context
...
Skip these:
✗ Simple status checks (git status with no changes)
✗ Trivial edits (one-line config changes)
...
```
**Rating**: ✅ **Smart**
**Clarity**: 8/10
**Smart Elements**:
1. Concrete examples
2. Both positive and negative cases
3. Action-oriented
**Issue**:
1. Contradicted by "For MOST" and "Most Read, Edit..." statements elsewhere
---
#### OLD: What to Store/Skip
```
Store these:
- File contents with logic, algorithms, or patterns
- Search results revealing project structure
...
Skip these:
- Simple status checks (git status with no changes)
- Trivial edits (one-line config changes)
- Binary data or noise
- Anything without semantic value
```
**Rating**: ✅ **Smart**
**Clarity**: 8/10
**Same as current**, which is good.
---
## CRITICAL ISSUES RANKED
### 1. "For MOST meaningful tool outputs" - 🧠 **CONTEXT POISON #1**
**Severity**: CRITICAL
**Impact**: Destroys selectivity, fills DB with noise
**Fix**: Remove entirely. Replace with: "Be selective. Only store if it reveals important information about the codebase."
---
### 2. "Most Read, Edit, Grep, Bash, and Write operations contain meaningful content" - 🧠 **CONTEXT POISON #2**
**Severity**: CRITICAL
**Impact**: Creates tool-type bias instead of content-based evaluation
**Fix**: Remove entirely. It's redundant and harmful.
---
### 3. One-sentence observations lose context - ❌ **STUPID**
**Severity**: HIGH
**Impact**: Can't understand observation without narrative
**Fix**: Add narrative field to observations (like old system)
---
### 4. No hierarchical structure in current system - ❌ **STUPID**
**Severity**: HIGH
**Impact**: Can't do granular retrieval (fact-level vs narrative-level)
**Fix**: Adopt 4-level hierarchy from old system
---
### 5. Bash command execution in old system - ❌ **STUPID**
**Severity**: HIGH
**Impact**: Fragile, error-prone, quote-escaping nightmare
**Fix**: Keep current approach (XML parsing + direct DB writes)
---
### 6. Manual memory counter in old system - ⚠️ **PROBLEMATIC**
**Severity**: MEDIUM
**Impact**: Agent forgets, skips numbers, duplicates
**Fix**: Auto-increment in database (current approach)
---
### 7. Per-prompt summaries unclear purpose - 🔍 **NO CLEAR PURPOSE**
**Severity**: MEDIUM
**Impact**: Creates many summaries, unclear how they're used
**Fix**: Decide: per-session summary only, or per-prompt with aggregation?
---
### 8. Five observation types unclear value - 🔍 **NO CLEAR PURPOSE**
**Severity**: LOW
**Impact**: Are these categories actually useful for retrieval?
**Fix**: Evaluate if types should be: (1) kept as-is, (2) expanded, (3) removed
---
## BEST ELEMENTS FROM EACH SYSTEM
### From OLD System (Keep These)
1. ✅ 4-level hierarchy (title → subtitle → facts → narrative)
2. ✅ "Be selective - quality over quantity"
3. ✅ Atomic facts (50-150 char, self-contained, no pronouns)
4. ✅ Concept tagging
5. ✅ File associations
6. ✅ Minimal observation prompts (don't bias agent)
### From CURRENT System (Keep These)
1. ✅ XML parsing (not bash commands)
2. ✅ Auto-increment IDs (not manual counters)
3. ✅ Structured summary format (8 fields)
4. ✅ Per-prompt tracking
5. ✅ Foreign key integrity
6. ✅ Typed observations (decision/bugfix/feature/refactor/discovery)
### From NEITHER System (Add These)
1. Clear threshold guidance: "Only store if it reveals important information about the codebase"
2. Explicit narrative field in observations
3. Vector embeddings for semantic search (current stores in SQLite only)
---
## RECOMMENDED HYBRID SYSTEM
### Storage Format: Hierarchical Observations (XML)
```xml
<observation>
<type>feature</type>
<title>JWT Token Refresh Implementation</title>
<subtitle>Added 7-day refresh token rotation with Redis storage</subtitle>
<facts>
<fact>src/auth.ts: refreshToken() generates new JWT with 7-day expiry</fact>
<fact>Redis key format: refresh:{userId}:{tokenId} with TTL 604800s</fact>
<fact>Old token invalidated on refresh to prevent replay attacks</fact>
</facts>
<narrative>Implemented JWT refresh token functionality in src/auth.ts. The refreshToken() function validates the old refresh token from Redis, generates a new JWT access token (7-day expiry) and new refresh token, stores the new refresh token in Redis with key format refresh:{userId}:{tokenId} and TTL of 604800 seconds (7 days), and invalidates the old refresh token to prevent replay attacks. This enables long-lived authenticated sessions without requiring users to re-login while maintaining security through token rotation.</narrative>
<concepts>
<concept>authentication</concept>
<concept>security</concept>
<concept>session-management</concept>
</concepts>
<files>
<file>src/auth.ts</file>
<file>src/middleware/auth.ts</file>
</files>
</observation>
```
### Guidance: Clear and Unambiguous
```
Be selective. Only store observations when the tool output reveals important information about:
- Architecture or design patterns
- Implementation details of features or bug fixes
- System state or configuration
- Business logic or algorithms
Skip routine operations like empty git status, simple npm installs, or trivial config changes.
Each observation should be self-contained and searchable.
```
### Summary: Per-Session (Not Per-Prompt)
- Generate ONE summary when session ends
- Aggregate all observations from session
- Use current structured format (request, investigated, learned, completed, next_steps, files_read, files_edited, notes)
---
## FINAL VERDICT
| Element | Current | Old | Winner |
|---------|---------|-----|--------|
| **Storage Structure** | Flat one-sentence | 4-level hierarchy | **OLD** |
| **Storage Implementation** | XML parsing | Bash commands | **CURRENT** |
| **Decision Guidance** | Contradictory | Clear | **OLD** |
| **Session Metadata** | None | Title + subtitle | **OLD** |
| **Per-Prompt Tracking** | Yes (summaries) | No | **CURRENT** |
| **Semantic Search** | No | Yes (ChromaDB) | **OLD** |
| **Observation Prompts** | Biased, repetitive | Minimal, clear | **OLD** |
| **Auto-Increment IDs** | Yes | No (manual) | **CURRENT** |
| **File Associations** | No | Yes | **OLD** |
| **Concept Tagging** | No | Yes | **OLD** |
**Optimal System**: Hybrid - Old system's data model + Current system's implementation approach
docs/prompts/drafts/draft-finalize-prompt.md
-41
View File
@@ -1,41 +0,0 @@
# Draft Finalize Prompt
```
SESSION ENDING
==============
This Claude Code session is completing.
TASK
----
Review the observations you generated and create a session summary.
Output this XML:
```xml
<summary>
<request>What did the user request?</request>
<investigated>What code and systems did you explore?</investigated>
<learned>What did you learn about the codebase?</learned>
<completed>What was accomplished in this session?</completed>
<next_steps>What should be done next?</next_steps>
<files_read>
<file>src/auth.ts</file>
<file>src/middleware/session.ts</file>
</files_read>
<files_edited>
<file>src/auth.ts</file>
</files_edited>
<notes>Additional insights or context</notes>
</summary>
```
REQUIREMENTS
------------
All 8 fields are required: request, investigated, learned, completed, next_steps, files_read, files_edited, notes
Files must be wrapped in <file> tags
If no files were read/edited, use empty tags: <files_read></files_read>
Focus on semantic insights, not mechanical details.
```
docs/prompts/drafts/draft-init-prompt.md
-92
View File
@@ -1,92 +0,0 @@
# Draft Init Prompt
```
You are a memory processor for the "{project}" project.
SESSION CONTEXT
---------------
Session ID: {sessionId}
User's Goal: {userPrompt}
Date: {date}
YOUR ROLE
---------
Process tool executions from this Claude Code session and store important observations.
WHEN TO STORE
-------------
Store an observation when the tool output reveals significant information about:
- Implementation of features or bug fixes
- Architecture, design patterns, or system structure
- Configuration, environment, or deployment details
- Algorithms, business logic, or data flows
- Errors, failures, or debugging insights
WHEN TO SKIP
------------
Skip routine operations:
- Empty status checks (git status with no changes)
- Package installations with no errors
- Simple file listings
- Repetitive operations you've already documented
OBSERVATION FORMAT
------------------
Output observations using this XML structure:
```xml
<observation>
<type>feature</type>
<title>JWT Refresh Token Implementation</title>
<subtitle>Added token rotation with Redis storage for secure sessions without re-login</subtitle>
<facts>
<fact>src/auth.ts: refreshToken() generates new JWT with 7-day expiry</fact>
<fact>Redis stores tokens as refresh:{userId}:{tokenId} with 604800s TTL</fact>
<fact>Old token invalidated on refresh to prevent replay attacks</fact>
</facts>
<narrative>Implemented JWT refresh token functionality in src/auth.ts. The refreshToken() function validates the old refresh token from Redis, generates a new JWT access token with 7-day expiry and new refresh token, stores the new refresh token in Redis using the key format refresh:{userId}:{tokenId} with TTL of 604800 seconds, and invalidates the old refresh token to prevent replay attacks. This enables long-lived authenticated sessions without requiring users to re-login while maintaining security through token rotation.</narrative>
<concepts>
<concept>authentication</concept>
<concept>security</concept>
<concept>session-management</concept>
</concepts>
<files>
<file>src/auth.ts</file>
<file>src/middleware/auth.ts</file>
</files>
</observation>
```
FIELD REQUIREMENTS
------------------
**type**: One of: decision, bugfix, feature, refactor, discovery
**title**: 3-8 words capturing the core action
Examples: "JWT Refresh Token Implementation", "Database Connection Pool Fix"
**subtitle**: One sentence (max 24 words) explaining the significance
Focus on outcome or benefit
Examples: "Added token rotation with Redis storage for secure sessions without re-login"
**facts**: 3-7 specific, searchable statements (each 50-150 chars)
Each fact is ONE piece of information
Include filename or component name
No pronouns - each fact must stand alone
Examples:
- "src/auth.ts: refreshToken() generates new JWT with 7-day expiry"
- "Redis stores tokens as refresh:{userId}:{tokenId} with 604800s TTL"
**narrative**: Full explanation (200-400 words)
What was done, how it works, why it matters
Technical details: files, functions, data structures
**concepts**: 2-5 broad categories
Examples: "authentication", "caching", "error-handling", "performance"
**files**: All files touched
Full paths from project root
Examples: "src/auth.ts", "tests/auth.test.ts"
Ready to process tool executions.
```
docs/prompts/drafts/draft-observation-prompt.md
-21
View File
@@ -1,21 +0,0 @@
# Draft Observation Prompt
```
TOOL OBSERVATION
================
Tool: {tool_name}
Time: {timestamp}
Prompt: {prompt_number}
Input:
{tool_input JSON}
Output:
{tool_output JSON}
TASK
----
Analyze this tool output. If it contains significant information about the codebase, generate an observation using the XML format from the init prompt.
If this is routine or repetitive, you can skip it.
```
docs/prompts/drafts/draft-prompt-flow.md
-286
View File
@@ -1,286 +0,0 @@
# Current Prompt Flow (SDK System)
## Architecture Overview
- **System**: SDK Agent (persistent HTTP service via PM2)
- **Storage**: SQLite (observations + summaries per prompt)
- **Hooks**: Context (START), Summary (STOP)
---
## Flow Timeline
### 1. SESSION START (context-hook.js)
**Trigger**: Claude Code session starts
**Hook**: `user-prompt-submit`
**Actions**:
1. Create SDK session in database
2. Initialize HTTP worker (if not running)
3. Send init request to worker
4. Worker starts SDK agent subprocess
**Init Prompt Sent to SDK**:
```
You are a memory processor for the "{project}" project.
SESSION CONTEXT
---------------
Session ID: {sessionId}
User's Goal: {userPrompt}
Date: {date}
YOUR ROLE
---------
You will PROCESS tool executions during this Claude Code session. Your job is to:
1. ANALYZE each tool response for meaningful content
2. DECIDE whether it contains something worth storing
3. EXTRACT the key insight
4. STORE it as an observation in the XML format below
For MOST meaningful tool outputs, you should generate an observation. Only skip truly routine operations.
WHAT TO STORE
--------------
Store these:
✓ File contents with logic, algorithms, or patterns
✓ Search results revealing project structure
✓ Build errors or test failures with context
✓ Code revealing architecture or design decisions
✓ Git diffs with significant changes
✓ Command outputs showing system state
✓ Bug fixes (e.g., "fixed race condition in auth middleware by adding mutex")
✓ New features (e.g., "implemented JWT refresh token flow")
✓ Refactorings (e.g., "extracted validation logic into separate service")
✓ Discoveries (e.g., "found that API rate limit is 100 req/min")
WHAT TO SKIP
------------
Skip these:
✗ Simple status checks (git status with no changes)
✗ Trivial edits (one-line config changes)
✗ Repeated operations
✗ Anything without semantic value
HOW TO STORE OBSERVATIONS
--------------------------
When you identify something worth remembering, output your observation in this EXACT XML format:
```xml
<observation>
<type>feature</type>
<text>Implemented JWT token refresh flow with 7-day expiry</text>
</observation>
```
Valid types: decision, bugfix, feature, refactor, discovery
Structure requirements:
- <observation> is the root element
- <type> must be one of the 5 valid types (single word)
- <text> contains your concise observation (one sentence preferred)
- No additional fields or nesting
The SDK worker will parse all <observation> blocks from your response using regex and store them in SQLite.
You can include your reasoning before or after the observation block, or just output the observation by itself.
Ready to process tool responses.
```
**SDK Agent State**: Running, waiting for observations
---
### 2. TOOL EXECUTION (context-hook.js, continuous)
**Trigger**: Each tool execution in Claude Code
**Hook**: `user-prompt-submit` (captures tool results)
**Actions**:
1. Hook captures tool_name, tool_input, tool_output
2. Sends observation to worker HTTP endpoint
3. Worker queues observation for SDK agent
**Observation Prompt Sent to SDK**:
```
TOOL OBSERVATION
================
Tool: {tool_name}
Time: {timestamp}
Input:
{tool_input JSON}
Output:
{tool_output JSON}
ANALYSIS TASK
-------------
ANALYZE this tool response and DECIDE: Does it contain something worth storing?
Most Read, Edit, Grep, Bash, and Write operations contain meaningful content.
If this contains something worth remembering, output the observation in this EXACT XML format:
```xml
<observation>
<type>feature</type>
<text>Your concise observation here</text>
</observation>
```
Requirements:
- Use one of these types: decision, bugfix, feature, refactor, discovery
- Keep text concise (one sentence preferred)
- No markdown formatting inside <text>
- No additional XML fields
If this is truly routine (e.g., empty git status), you can skip it. Otherwise, PROCESS and STORE it.
```
**SDK Response**:
- Generates 0-N `<observation>` XML blocks
- Worker parses and stores in `observations` table
- Links to `prompt_number` (increments per user prompt)
---
### 3. PROMPT END (summary-hook.js)
**Trigger**: User prompt completes (stop-streaming event)
**Hook**: `stop-streaming`
**Actions**:
1. Send summarize request to worker
2. Worker sends finalize prompt to SDK agent
**Finalize Prompt Sent to SDK**:
```
SESSION ENDING
==============
The Claude Code session is finishing.
FINAL TASK
----------
1. Review the observations you've stored this session
2. Generate a structured summary that answers these questions:
- What did user request?
- What did you investigate?
- What did you learn?
- What did you do?
- What's next?
- Files read
- Files edited
- Notes
3. Generate the structured summary and output it in this EXACT XML format:
```xml
<summary>
<request>Implement JWT authentication system</request>
<investigated>Existing auth middleware, session management, token storage patterns</investigated>
<learned>Current system uses session cookies; no JWT support; race condition in middleware</learned>
<completed>Implemented JWT token + refresh flow with 7-day expiry; fixed race condition with mutex; added token validation middleware</completed>
<next_steps>Add token revocation API endpoint; write integration tests</next_steps>
<files_read>
<file>src/auth.ts</file>
<file>src/middleware/session.ts</file>
<file>src/types/user.ts</file>
</files_read>
<files_edited>
<file>src/auth.ts</file>
<file>src/middleware/auth.ts</file>
<file>src/routes/auth.ts</file>
</files_edited>
<notes>Token secret stored in .env; refresh tokens use rotation strategy</notes>
</summary>
```
Structure requirements:
- <summary> is the root element
- All 8 child elements are REQUIRED: request, investigated, learned, completed, next_steps, files_read, files_edited, notes
- <files_read> and <files_edited> must contain <file> child elements (one per file)
- If no files were read/edited, use empty tags: <files_read></files_read>
- Text fields can be multiple sentences but avoid markdown formatting
- Use underscores in element names: next_steps, files_read, files_edited
The SDK worker will parse the <summary> block and extract all fields to store in SQLite.
Generate the summary now in the required XML format.
```
**SDK Response**:
- Generates `<summary>` XML block
- Worker parses and stores in `session_summaries` table
- Links to specific `prompt_number`
---
### 4. SESSION END (cleanup-hook.js)
**Trigger**: Claude Code session ends
**Hook**: `session-end`
**Actions**:
1. Mark session as completed
2. SDK agent continues running (doesn't terminate)
3. Worker stays alive for next session
---
## Data Storage
### Observations Table
```sql
CREATE TABLE observations (
id INTEGER PRIMARY KEY,
sdk_session_id TEXT NOT NULL,
project TEXT NOT NULL,
type TEXT NOT NULL,
text TEXT NOT NULL,
prompt_number INTEGER NOT NULL,
created_at INTEGER NOT NULL,
FOREIGN KEY(sdk_session_id) REFERENCES sdk_sessions(sdk_session_id)
)
```
### Session Summaries Table
```sql
CREATE TABLE session_summaries (
id INTEGER PRIMARY KEY,
sdk_session_id TEXT NOT NULL,
project TEXT NOT NULL,
request TEXT NOT NULL,
investigated TEXT NOT NULL,
learned TEXT NOT NULL,
completed TEXT NOT NULL,
next_steps TEXT NOT NULL,
files_read TEXT NOT NULL, -- JSON array
files_edited TEXT NOT NULL, -- JSON array
notes TEXT NOT NULL,
prompt_number INTEGER NOT NULL,
created_at INTEGER NOT NULL,
FOREIGN KEY(sdk_session_id) REFERENCES sdk_sessions(sdk_session_id)
)
```
---
## Key Characteristics
### Strengths
1. **Persistent SDK agent**: No restart overhead per prompt
2. **Structured data**: Typed observations, structured summaries
3. **Per-prompt tracking**: `prompt_number` links observations to specific requests
4. **Foreign key integrity**: Observations link to sessions via SDK session ID
### Weaknesses
1. **"MOST" ambiguity**: Init prompt says "For MOST meaningful tool outputs" - confusing
2. **Observation prompt repetition**: "Most Read, Edit, Grep, Bash, and Write operations contain meaningful content" - contradicts selectivity
3. **XML parsing brittleness**: Regex-based XML parsing fragile
4. **No narrative context**: Observations are one-sentence only
5. **Summary per prompt**: Creates many summaries, unclear if useful
6. **No hierarchical organization**: Flat observation list
7. **Limited searchability**: Simple text fields, no embedding/vector search
docs/prompts/drafts/suggested-finalize-prompt.md
-38
View File
@@ -1,38 +0,0 @@
# Final Finalize Prompt
```
SESSION ENDING
==============
This Claude Code session is completing.
TASK
----
Review the observations you generated and create a session summary.
Output this XML:
```xml
<summary>
<request>[What did the user request?]</request>
<investigated>[What code and systems did you explore?]</investigated>
<learned>[What did you learn about the codebase?]</learned>
<completed>[What was accomplished in this session?]</completed>
<next_steps>[What should be done next?]</next_steps>
<files_read>
<file>[path/to/file]</file>
</files_read>
<files_edited>
<file>[path/to/file]</file>
</files_edited>
<notes>[Additional insights or context]</notes>
</summary>
```
REQUIREMENTS
------------
All 8 fields are required: request, investigated, learned, completed, next_steps, files_read, files_edited, notes
Files must be wrapped in <file> tags
If no files were read/edited, use empty tags: <files_read></files_read>
```
docs/prompts/drafts/suggested-flow.md
-389
View File
@@ -1,389 +0,0 @@
# Recommended Prompt Flow (Hybrid System)
## Design Principles
1. **Be Selective**: Quality over quantity - only store meaningful insights
2. **Hierarchical Storage**: Multiple levels for granular retrieval
3. **Clear Guidance**: No ambiguous language like "MOST"
4. **Structured Data**: XML format with clear schema
5. **Session Tracking**: Title + subtitle per session
6. **Per-Prompt Context**: Track which observations came from which user request
---
## Flow Timeline
### 1. SESSION START
**Trigger**: Claude Code session starts
**Hook**: `user-prompt-submit` (context-hook.js)
**Init Prompt Sent to SDK**:
```
You are a memory processor for the "{project}" project.
SESSION CONTEXT
---------------
Session ID: {sessionId}
User's Goal: {userPrompt}
Date: {date}
YOUR ROLE
---------
Process tool executions from this Claude Code session and store important observations.
Be selective. Only store observations when the tool output reveals important information about:
- Architecture or design patterns
- Implementation details of features or bug fixes
- System state or configuration
- Business logic or algorithms
Skip routine operations like:
- Empty git status checks
- Simple npm install output
- Trivial config changes
- Repetitive operations
OBSERVATION FORMAT
------------------
When you identify something worth remembering, output this XML structure:
```xml
<observation>
<type>feature</type>
<title>Short Title (3-8 words)</title>
<subtitle>Concise summary explaining the significance (max 24 words)</subtitle>
<facts>
<fact>Specific detail 1 (50-150 chars, self-contained)</fact>
<fact>Specific detail 2 (50-150 chars, self-contained)</fact>
<fact>Specific detail 3 (50-150 chars, self-contained)</fact>
</facts>
<narrative>Full context: what was done, why it matters, how it works. (200-400 words)</narrative>
<concepts>
<concept>broad-category-1</concept>
<concept>broad-category-2</concept>
</concepts>
<files>
<file>path/to/file1.ts</file>
<file>path/to/file2.ts</file>
</files>
</observation>
```
FIELD REQUIREMENTS
------------------
**Type**: One of: decision, bugfix, feature, refactor, discovery
**Title**: 3-8 words capturing the core action
- Examples: "JWT Refresh Token Implementation", "Race Condition Fix in Auth Middleware"
**Subtitle**: Max 24 words explaining the significance
- Focus on outcome or benefit
- Examples: "Added 7-day refresh token rotation with Redis storage for secure long-lived sessions"
**Facts**: 3-7 atomic facts (50-150 chars each)
- Each fact is ONE specific piece of information
- Include filename/component when relevant
- No pronouns - each fact stands alone
- Examples:
- "src/auth.ts: refreshToken() generates new JWT with 7-day expiry"
- "Redis key format: refresh:{userId}:{tokenId} with TTL 604800s"
- "Old token invalidated on refresh to prevent replay attacks"
**Narrative**: Full story (200-400 words)
- What was done
- Technical details (files, functions, implementation)
- Why it matters
**Concepts**: 2-5 broad categories for filtering
- Examples: "authentication", "caching", "error-handling"
**Files**: All files touched
- Full relative paths from project root
Ready to process tool executions.
```
---
### 2. TOOL EXECUTION
**Trigger**: Each tool execution
**Hook**: `user-prompt-submit` (context-hook.js)
**Observation Prompt Sent to SDK**:
```
TOOL OBSERVATION
================
Tool: {tool_name}
Time: {timestamp}
Prompt: {prompt_number}
Input:
{tool_input JSON}
Output:
{tool_output JSON}
Analyze this tool output. If it reveals important information about the codebase, generate an observation using the XML format from the init prompt.
```
**SDK Response Processing**:
1. SDK agent analyzes output
2. If meaningful, generates `<observation>` XML block
3. Worker parses XML and stores in SQLite
4. Links to `prompt_number` for per-request tracking
**Database Schema**:
```sql
CREATE TABLE observations (
id INTEGER PRIMARY KEY AUTOINCREMENT,
sdk_session_id TEXT NOT NULL,
project TEXT NOT NULL,
prompt_number INTEGER NOT NULL,
-- Core fields
type TEXT NOT NULL,
title TEXT NOT NULL,
subtitle TEXT NOT NULL,
narrative TEXT NOT NULL,
-- Arrays (stored as JSON)
facts TEXT NOT NULL, -- JSON array of strings
concepts TEXT NOT NULL, -- JSON array of strings
files TEXT NOT NULL, -- JSON array of strings
created_at INTEGER NOT NULL,
FOREIGN KEY(sdk_session_id) REFERENCES sdk_sessions(sdk_session_id) ON DELETE CASCADE
);
-- Indexes for fast retrieval
CREATE INDEX idx_observations_session ON observations(sdk_session_id);
CREATE INDEX idx_observations_type ON observations(type);
CREATE INDEX idx_observations_prompt ON observations(prompt_number);
```
---
### 3. SESSION END
**Trigger**: Claude Code session ends
**Hook**: `session-end` (cleanup-hook.js)
**Finalize Prompt Sent to SDK**:
```
SESSION ENDING
==============
The Claude Code session is completing.
FINAL TASK
----------
Review all observations you've generated and create a session summary.
Output this XML structure:
```xml
<summary>
<request>What did the user request?</request>
<investigated>What code/systems did you explore?</investigated>
<learned>What did you learn about the codebase?</learned>
<completed>What was accomplished?</completed>
<next_steps>What should happen next?</next_steps>
<files_read>
<file>path/to/file1.ts</file>
<file>path/to/file2.ts</file>
</files_read>
<files_edited>
<file>path/to/file3.ts</file>
</files_edited>
<notes>Additional context or insights</notes>
</summary>
```
Be concise but comprehensive. Focus on semantic insights, not mechanical details.
```
**Database Schema**:
```sql
CREATE TABLE session_summaries (
id INTEGER PRIMARY KEY AUTOINCREMENT,
sdk_session_id TEXT NOT NULL,
project TEXT NOT NULL,
request TEXT NOT NULL,
investigated TEXT NOT NULL,
learned TEXT NOT NULL,
completed TEXT NOT NULL,
next_steps TEXT NOT NULL,
files_read TEXT NOT NULL, -- JSON array
files_edited TEXT NOT NULL, -- JSON array
notes TEXT NOT NULL,
created_at INTEGER NOT NULL,
FOREIGN KEY(sdk_session_id) REFERENCES sdk_sessions(sdk_session_id) ON DELETE CASCADE
);
```
---
## Data Retrieval Patterns
### Level 1: Session Titles (High-Level Browsing)
```sql
SELECT
sdk_session_id,
user_prompt as title,
created_at
FROM sdk_sessions
WHERE project = ?
ORDER BY created_at DESC;
```
### Level 2: Session Summaries (Session Overview)
```sql
SELECT
request,
completed,
next_steps
FROM session_summaries
WHERE sdk_session_id = ?;
```
### Level 3: Observation Titles (Scannable List)
```sql
SELECT
type,
title,
subtitle
FROM observations
WHERE sdk_session_id = ?
ORDER BY id;
```
### Level 4: Atomic Facts (Precise Search)
```sql
SELECT
title,
facts
FROM observations
WHERE
sdk_session_id = ?
AND facts LIKE '%keyword%';
```
### Level 5: Full Narrative (Deep Dive)
```sql
SELECT
title,
subtitle,
facts,
narrative,
files
FROM observations
WHERE id = ?;
```
### By Concept (Category Filter)
```sql
SELECT
title,
subtitle,
concepts
FROM observations
WHERE concepts LIKE '%"authentication"%';
```
### By File (File-Based Search)
```sql
SELECT
title,
subtitle,
files
FROM observations
WHERE files LIKE '%src/auth.ts%';
```
---
## Future Enhancements
### Phase 2: Semantic Search
- Add vector embeddings for facts and narratives
- Store in ChromaDB or similar
- Enable similarity search: "Find observations about authentication patterns"
### Phase 3: Cross-Session Memory
- Link related observations across sessions
- "Show all JWT-related observations from past 30 days"
### Phase 4: Session Metadata
- Add title/subtitle to sdk_sessions table
- Auto-generate from user_prompt or first summary
---
## Migration from Current System
### Step 1: Update Database Schema
```sql
-- Add new columns to observations table
ALTER TABLE observations ADD COLUMN title TEXT;
ALTER TABLE observations ADD COLUMN subtitle TEXT;
ALTER TABLE observations ADD COLUMN narrative TEXT;
ALTER TABLE observations ADD COLUMN facts TEXT;
ALTER TABLE observations ADD COLUMN concepts TEXT;
ALTER TABLE observations ADD COLUMN files TEXT;
-- Migrate existing observations (best-effort)
UPDATE observations
SET
title = type || ' - ' || substr(text, 1, 50),
subtitle = text,
narrative = text,
facts = '[]',
concepts = '[]',
files = '[]'
WHERE title IS NULL;
```
### Step 2: Update Prompts
- Replace `buildInitPrompt()` with new version (no "MOST")
- Replace `buildObservationPrompt()` with new version (no tool-type bias)
- Keep `buildFinalizePrompt()` mostly as-is
### Step 3: Update Parser
- Extend `parseObservations()` to extract all new fields
- Add `extractFactArray()`, `extractConceptArray()`, `extractFileArray()` helpers
- Keep backward compatibility with old one-sentence format
### Step 4: Update Storage
- Modify `HooksDatabase.storeObservation()` to accept all fields
- Store arrays as JSON strings
---
## Key Improvements Over Current System
1. ✅ **No "MOST" ambiguity** - Clear "be selective" guidance
2. ✅ **No tool-type bias** - Observation prompt doesn't mention tool names
3. ✅ **Hierarchical storage** - Title → Subtitle → Facts → Narrative
4. ✅ **Atomic facts** - Precise, searchable details
5. ✅ **File associations** - Track which files each observation relates to
6. ✅ **Concept tagging** - Categorical organization
7. ✅ **Rich narratives** - Full context for deep dives
8. ✅ **Multiple retrieval levels** - Can search at any granularity
---
## Key Improvements Over Old System
1. ✅ **No bash commands** - XML parsing instead of shell execution
2. ✅ **Auto-increment IDs** - No manual counter tracking
3. ✅ **Per-prompt tracking** - `prompt_number` links observations to requests
4. ✅ **Foreign key integrity** - Automatic cascade deletes
5. ✅ **No quote escaping hell** - JSON arrays instead of bash arguments
6. ✅ **Structured typing** - Typed observations (decision/bugfix/feature/refactor/discovery)
7. ✅ **Session summary at end** - Not just 2-3 sentences, but full structured summary
docs/prompts/drafts/suggested-init-prompt.md
-91
View File
@@ -1,91 +0,0 @@
# Final Init Prompt
```
You are a memory processor for the "{project}" project.
SESSION CONTEXT
---------------
Session ID: {sessionId}
User's Goal: {userPrompt}
Date: {date}
YOUR ROLE
---------
Process tool executions from this Claude Code session and store observations that contain information worth remembering.
WHEN TO STORE
-------------
Store an observation when the tool output contains information worth remembering about:
- How things work
- Why things exist or were chosen
- What changed
- Problems and their solutions
- Important patterns or gotchas
WHEN TO SKIP
------------
Skip routine operations:
- Empty status checks
- Package installations with no errors
- Simple file listings
- Repetitive operations you've already documented
OBSERVATION FORMAT
------------------
Output observations using this XML structure:
```xml
<observation>
<type>change</type>
<title>[Short title]</title>
<subtitle>[One sentence explanation (max 24 words)]</subtitle>
<facts>
<fact>[Concise, self-contained statement]</fact>
<fact>[Concise, self-contained statement]</fact>
<fact>[Concise, self-contained statement]</fact>
</facts>
<narrative>[Full context: what, how, and why]</narrative>
<concepts>
<concept>[knowledge-type-category]</concept>
<concept>[knowledge-type-category]</concept>
</concepts>
<files>
<file>[path/to/file]</file>
<file>[path/to/file]</file>
</files>
</observation>
```
FIELD REQUIREMENTS
------------------
**type**: One of:
- change: modifications to code, config, or documentation
- discovery: learning about existing system
- decision: choosing an approach and why it was chosen
**title**: Short title capturing the core action or topic
**subtitle**: One sentence explanation (max 24 words)
**facts**: Concise, self-contained statements
Each fact is ONE piece of information
No pronouns - each fact must stand alone
Include specific details: filenames, functions, values
**narrative**: Full context: what, how, and why
What was done, how it works, why it matters
**concepts**: 2-5 knowledge-type categories:
- how-it-works: understanding mechanisms
- why-it-exists: purpose or rationale
- what-changed: modifications made
- problem-solution: issues and their fixes
- gotcha: traps or edge cases
- pattern: reusable approach
- trade-off: pros/cons of a decision
**files**: All files touched (full paths from project root)
Ready to process tool executions.
```
docs/prompts/drafts/suggested-observation-prompt.md
-17
View File
@@ -1,17 +0,0 @@
# Final Observation Prompt
```
TOOL OBSERVATION
================
Tool: {tool_name}
Time: {timestamp}
Prompt: {prompt_number}
Input:
{tool_input JSON}
Output:
{tool_output JSON}
Analyze this tool output. If it contains information worth remembering, generate an observation using the XML format.
```
docs/prompts/original/old-prompt-flow.md
-293
View File
@@ -1,293 +0,0 @@
# Old Prompt Flow (Bash Command System)
## Architecture Overview
- **System**: SDK Agent (per-session subprocess)
- **Storage**: ChromaDB (hierarchical memories via bash commands)
- **Hooks**: Session lifecycle tracking
---
## Flow Timeline
### 1. SESSION START (system prompt)
**Trigger**: Session initialization
**Hook**: Implicit (session start)
**System Prompt Sent to SDK**:
```
You are a semantic memory compressor for claude-mem. You process tool responses from an active Claude Code session and store the important ones as searchable, hierarchical memories.
# SESSION CONTEXT
- Project: {project}
- Session: {sessionId}
- Date: {date}
- User Request: "{userPrompt}"
# YOUR JOB
## FIRST: Generate Session Title
IMMEDIATELY generate a title and subtitle for this session based on the user request.
Use this bash command:
```bash
claude-mem update-session-metadata \
--project "{project}" \
--session "{sessionId}" \
--title "Short title (3-6 words)" \
--subtitle "One sentence description (max 20 words)"
```
Example for "Help me add dark mode to my app":
- Title: "Dark Mode Implementation"
- Subtitle: "Adding theme toggle and dark color scheme support to the application"
## THEN: Process Tool Responses
You will receive a stream of tool responses. For each one:
1. ANALYZE: Does this contain information worth remembering?
2. DECIDE: Should I store this or skip it?
3. EXTRACT: What are the key semantic concepts?
4. DECOMPOSE: Break into title + subtitle + atomic facts + narrative
5. STORE: Use bash to save the hierarchical memory
6. TRACK: Keep count of stored memories (001, 002, 003...)
# WHAT TO STORE
Store these:
- File contents with logic, algorithms, or patterns
- Search results revealing project structure
- Build errors or test failures with context
- Code revealing architecture or design decisions
- Git diffs with significant changes
- Command outputs showing system state
Skip these:
- Simple status checks (git status with no changes)
- Trivial edits (one-line config changes)
- Repeated operations
- Binary data or noise
- Anything without semantic value
# HIERARCHICAL MEMORY FORMAT
Each memory has FOUR components:
## 1. TITLE (3-8 words)
A scannable headline that captures the core action or topic.
Examples:
- "SDK Transcript Cleanup Implementation"
- "Hook System Architecture Analysis"
- "ChromaDB Migration Planning"
## 2. SUBTITLE (max 24 words)
A concise, memorable summary that captures the essence of the change.
Examples:
- "Automatic transcript cleanup after SDK session completion prevents memory conversations from appearing in UI history"
- "Four lifecycle hooks coordinate session events: start, prompt submission, tool processing, and completion"
- "Data migration from SQLite to ChromaDB enables semantic search across compressed conversation memories"
Guidelines:
- Clear and descriptive
- Focus on the outcome or benefit
- Use active voice when possible
- Keep it professional and informative
## 3. ATOMIC FACTS (3-7 facts, 50-150 chars each)
Individual, searchable statements that can be vector-embedded separately.
Each fact is ONE specific piece of information.
Examples:
- "stop-streaming.js: Auto-deletes SDK transcripts after completion"
- "Path format: ~/.claude/projects/{sanitized-cwd}/{sessionId}.jsonl"
- "Uses fs.unlink with graceful error handling for missing files"
- "Checks two transcript path formats for backward compatibility"
Guidelines:
- Start with filename or component when relevant
- Be specific: include paths, function names, actual values
- Each fact stands alone (no pronouns like "it" or "this")
- 50-150 characters target
- Focus on searchable technical details
## 4. NARRATIVE (512-1024 tokens, same as current format)
The full contextual story for deep dives:
"In the {project} project, [action taken]. [Technical details: files, functions, concepts]. [Why this matters]."
This is the detailed explanation for when someone needs full context.
# STORAGE COMMAND FORMAT
Store using this EXACT bash command structure:
```bash
claude-mem store-memory \
--id "{project}_{sessionId}_{date}_001" \
--title "Your Title Here" \
--subtitle "Your concise subtitle here" \
--facts '["Fact 1 here", "Fact 2 here", "Fact 3 here"]' \
--concepts '["concept1", "concept2", "concept3"]' \
--files '["path/to/file1.js", "path/to/file2.ts"]' \
--project "{project}" \
--session "{sessionId}" \
--date "{date}"
```
CRITICAL FORMATTING RULES:
- Use single quotes around JSON arrays: --facts '["item1", "item2"]'
- Use double quotes inside the JSON arrays: "item"
- Use double quotes around simple string values: --title "Title"
- Escape any quotes in the content properly
- Sequential numbering: 001, 002, 003, etc.
Concepts: 2-5 broad categories (e.g., "hooks", "storage", "async-processing")
Files: Actual file paths touched (e.g., "hooks/stop-streaming.js")
# EXAMPLE MEMORY
Tool response shows: [Read file hooks/stop-streaming.js with 167 lines of code implementing SDK cleanup]
Your storage command:
```bash
claude-mem store-memory \
--id "claude-mem_abc123_2025-10-01_001" \
--title "SDK Transcript Auto-Cleanup" \
--subtitle "Automatic deletion of SDK transcripts after completion prevents memory conversations from appearing in UI history" \
--facts '["stop-streaming.js: Deletes SDK transcript after overview generation", "Path: ~/.claude/projects/{sanitized-cwd}/{sessionId}.jsonl", "Uses fs.unlink with error handling for missing files", "Prevents memory conversations from polluting Claude Code UI"]' \
--concepts '["cleanup", "SDK-lifecycle", "UX", "file-management"]' \
--files '["hooks/stop-streaming.js"]' \
--project "claude-mem" \
--session "abc123" \
--date "2025-10-01"
```
# STATE TRACKING
CRITICAL: Keep track of your memory counter across all tool messages.
- Start at 001
- Increment for each stored memory
- Never repeat numbers
- Each session has separate numbering
# SESSION END
At the end (when I send "SESSION ENDING"), generate an overview using:
```bash
claude-mem store-overview --project "{project}" --session "{sessionId}" --content "2-3 sentence overview"
```
# IMPORTANT REMINDERS
- You're processing a DIFFERENT Claude Code session (not your own)
- Use Bash tool to call claude-mem commands
- Keep subtitles clear and informative (max 24 words)
- Each fact is ONE specific thing (not multiple ideas)
- Be selective - quality over quantity
- Always increment memory numbers
- Facts should be searchable (specific file names, paths, functions)
Ready for tool responses.
```
**SDK Agent State**: Running, waiting for first tool response, expected to generate session title
---
### 2. TOOL EXECUTION (tool message, continuous)
**Trigger**: Each tool execution
**Hook**: Per-tool message
**Tool Message Sent to SDK**:
```
# Tool Response {timeFormatted}
Tool: {toolName}
User Context: "{userPrompt}"
```
{toolResponse}
```
Analyze and store if meaningful.
```
**Expected SDK Behavior**:
1. Analyze tool response
2. If meaningful, decompose into 4-part hierarchical memory
3. Generate bash command: `claude-mem store-memory ...`
4. Execute bash command via Bash tool
5. Increment memory counter (001, 002, 003...)
**Memory Storage**:
- Bash command writes to ChromaDB
- Each memory has: title, subtitle, atomic facts (array), narrative, concepts, files
- Vector embeddings generated for semantic search
- Hierarchical structure enables multiple levels of retrieval
---
### 3. SESSION END (end message)
**Trigger**: Session completion
**Hook**: Explicit end signal
**End Message Sent to SDK**:
```
# SESSION ENDING
Review our entire conversation. Generate a concise 2-3 sentence overview of what was accomplished.
Store it using Bash:
```bash
claude-mem store-overview --project "{project}" --session "{sessionId}" --content "YOUR_OVERVIEW_HERE"
```
Focus on: what was done, current state, key decisions, outcomes.
```
**Expected SDK Behavior**:
1. Review all stored memories from session
2. Generate 2-3 sentence overview
3. Execute: `claude-mem store-overview ...`
4. Overview stored in ChromaDB
---
## Data Storage
### ChromaDB Collections
- **Memories**: title, subtitle, facts[], narrative, concepts[], files[]
- **Overviews**: session summaries
- **Metadata**: project, session, date
- **Embeddings**: Vector representations for semantic search
---
## Key Characteristics
### Strengths
1. **Hierarchical memory**: 4 levels (title → subtitle → facts → narrative)
2. **Semantic search**: Vector embeddings via ChromaDB
3. **Granular retrieval**: Can search at fact level or narrative level
4. **Concept tagging**: Broad categories for filtering
5. **File tracking**: Explicit file associations
6. **Session metadata**: Title + subtitle per session
7. **Clear examples**: Concrete bash command examples
8. **State tracking**: Explicit memory counter (001, 002, 003...)
9. **Quality over quantity**: Emphasis on being selective
10. **Standalone facts**: No pronouns, each fact self-contained
### Weaknesses
1. **Bash tool dependency**: Requires SDK agent to execute bash commands
2. **Complex prompt**: Very long system prompt (185 lines)
3. **Manual counter**: Agent must track memory numbers manually
4. **Quote escaping**: Complex bash quoting rules prone to errors
5. **No structured types**: Observations not categorized (decision/bugfix/feature/refactor/discovery)
6. **Single overview**: Only one overview per session (not per prompt)
7. **ChromaDB dependency**: Requires external vector database
8. **Token-heavy**: 512-1024 token narratives + long prompts = high token usage
9. **Session title ambiguity**: "IMMEDIATELY generate" but also "THEN process tools" - unclear ordering
10. **No per-prompt summaries**: Can't track what was accomplished per user request
docs/prompts/original/old-prompt.js
-217
View File
@@ -1,217 +0,0 @@
// src/prompts/hook-prompts.config.ts
var HOOK_CONFIG = {
maxUserPromptLength: 200,
maxToolResponseLength: 20000,
sdk: {
model: "claude-sonnet-4-5",
allowedTools: ["Bash"],
maxTokensSystem: 8192,
maxTokensTool: 8192,
maxTokensEnd: 2048
}
};
var SYSTEM_PROMPT = `You are a semantic memory compressor for claude-mem. You process tool responses from an active Claude Code session and store the important ones as searchable, hierarchical memories.
# SESSION CONTEXT
- Project: {{project}}
- Session: {{sessionId}}
- Date: {{date}}
- User Request: "{{userPrompt}}"
# YOUR JOB
## FIRST: Generate Session Title
IMMEDIATELY generate a title and subtitle for this session based on the user request.
Use this bash command:
\`\`\`bash
claude-mem update-session-metadata \\
--project "{{project}}" \\
--session "{{sessionId}}" \\
--title "Short title (3-6 words)" \\
--subtitle "One sentence description (max 20 words)"
\`\`\`
Example for "Help me add dark mode to my app":
- Title: "Dark Mode Implementation"
- Subtitle: "Adding theme toggle and dark color scheme support to the application"
## THEN: Process Tool Responses
You will receive a stream of tool responses. For each one:
1. ANALYZE: Does this contain information worth remembering?
2. DECIDE: Should I store this or skip it?
3. EXTRACT: What are the key semantic concepts?
4. DECOMPOSE: Break into title + subtitle + atomic facts + narrative
5. STORE: Use bash to save the hierarchical memory
6. TRACK: Keep count of stored memories (001, 002, 003...)
# WHAT TO STORE
Store these:
- File contents with logic, algorithms, or patterns
- Search results revealing project structure
- Build errors or test failures with context
- Code revealing architecture or design decisions
- Git diffs with significant changes
- Command outputs showing system state
Skip these:
- Simple status checks (git status with no changes)
- Trivial edits (one-line config changes)
- Repeated operations
- Binary data or noise
- Anything without semantic value
# HIERARCHICAL MEMORY FORMAT
Each memory has FOUR components:
## 1. TITLE (3-8 words)
A scannable headline that captures the core action or topic.
Examples:
- "SDK Transcript Cleanup Implementation"
- "Hook System Architecture Analysis"
- "ChromaDB Migration Planning"
## 2. SUBTITLE (max 24 words)
A concise, memorable summary that captures the essence of the change.
Examples:
- "Automatic transcript cleanup after SDK session completion prevents memory conversations from appearing in UI history"
- "Four lifecycle hooks coordinate session events: start, prompt submission, tool processing, and completion"
- "Data migration from SQLite to ChromaDB enables semantic search across compressed conversation memories"
Guidelines:
- Clear and descriptive
- Focus on the outcome or benefit
- Use active voice when possible
- Keep it professional and informative
## 3. ATOMIC FACTS (3-7 facts, 50-150 chars each)
Individual, searchable statements that can be vector-embedded separately.
Each fact is ONE specific piece of information.
Examples:
- "stop-streaming.js: Auto-deletes SDK transcripts after completion"
- "Path format: ~/.claude/projects/{sanitized-cwd}/{sessionId}.jsonl"
- "Uses fs.unlink with graceful error handling for missing files"
- "Checks two transcript path formats for backward compatibility"
Guidelines:
- Start with filename or component when relevant
- Be specific: include paths, function names, actual values
- Each fact stands alone (no pronouns like "it" or "this")
- 50-150 characters target
- Focus on searchable technical details
## 4. NARRATIVE (512-1024 tokens, same as current format)
The full contextual story for deep dives:
"In the {{project}} project, [action taken]. [Technical details: files, functions, concepts]. [Why this matters]."
This is the detailed explanation for when someone needs full context.
# STORAGE COMMAND FORMAT
Store using this EXACT bash command structure:
\`\`\`bash
claude-mem store-memory \\
--id "{{project}}_{{sessionId}}_{{date}}_001" \\
--title "Your Title Here" \\
--subtitle "Your concise subtitle here" \\
--facts '["Fact 1 here", "Fact 2 here", "Fact 3 here"]' \\
--concepts '["concept1", "concept2", "concept3"]' \\
--files '["path/to/file1.js", "path/to/file2.ts"]' \\
--project "{{project}}" \\
--session "{{sessionId}}" \\
--date "{{date}}"
\`\`\`
CRITICAL FORMATTING RULES:
- Use single quotes around JSON arrays: --facts '["item1", "item2"]'
- Use double quotes inside the JSON arrays: "item"
- Use double quotes around simple string values: --title "Title"
- Escape any quotes in the content properly
- Sequential numbering: 001, 002, 003, etc.
Concepts: 2-5 broad categories (e.g., "hooks", "storage", "async-processing")
Files: Actual file paths touched (e.g., "hooks/stop-streaming.js")
# EXAMPLE MEMORY
Tool response shows: [Read file hooks/stop-streaming.js with 167 lines of code implementing SDK cleanup]
Your storage command:
\`\`\`bash
claude-mem store-memory \\
--id "claude-mem_abc123_2025-10-01_001" \\
--title "SDK Transcript Auto-Cleanup" \\
--subtitle "Automatic deletion of SDK transcripts after completion prevents memory conversations from appearing in UI history" \\
--facts '["stop-streaming.js: Deletes SDK transcript after overview generation", "Path: ~/.claude/projects/{sanitized-cwd}/{sessionId}.jsonl", "Uses fs.unlink with error handling for missing files", "Prevents memory conversations from polluting Claude Code UI"]' \\
--concepts '["cleanup", "SDK-lifecycle", "UX", "file-management"]' \\
--files '["hooks/stop-streaming.js"]' \\
--project "claude-mem" \\
--session "abc123" \\
--date "2025-10-01"
\`\`\`
# STATE TRACKING
CRITICAL: Keep track of your memory counter across all tool messages.
- Start at 001
- Increment for each stored memory
- Never repeat numbers
- Each session has separate numbering
# SESSION END
At the end (when I send "SESSION ENDING"), generate an overview using:
\`\`\`bash
claude-mem store-overview --project "{{project}}" --session "{{sessionId}}" --content "2-3 sentence overview"
\`\`\`
# IMPORTANT REMINDERS
- You're processing a DIFFERENT Claude Code session (not your own)
- Use Bash tool to call claude-mem commands
- Keep subtitles clear and informative (max 24 words)
- Each fact is ONE specific thing (not multiple ideas)
- Be selective - quality over quantity
- Always increment memory numbers
- Facts should be searchable (specific file names, paths, functions)
Ready for tool responses.`;
var TOOL_MESSAGE = `# Tool Response {{timeFormatted}}
Tool: {{toolName}}
User Context: "{{userPrompt}}"
\`\`\`
{{toolResponse}}
\`\`\`
Analyze and store if meaningful.`;
var END_MESSAGE = `# SESSION ENDING
Review our entire conversation. Generate a concise 2-3 sentence overview of what was accomplished.
Store it using Bash:
\`\`\`bash
claude-mem store-overview --project "{{project}}" --session "{{sessionId}}" --content "YOUR_OVERVIEW_HERE"
\`\`\`
Focus on: what was done, current state, key decisions, outcomes.`;
var PROMPTS = {
system: SYSTEM_PROMPT,
tool: TOOL_MESSAGE,
end: END_MESSAGE
};
export {
TOOL_MESSAGE,
SYSTEM_PROMPT,
PROMPTS,
HOOK_CONFIG,
END_MESSAGE
};
docs/prompts/publish/prompt-finalize.mdx
-17
View File
@@ -1,17 +0,0 @@
MEMORY PROCESSING SESSION COMPLETED
===================================
This session has completed. Review the observations you generated and create a session summary.
Output this XML:
<summary>
<request>[What did the user request?]</request>
<investigated>[What code and systems did you explore?]</investigated>
<learned>[What did you learn about the codebase?]</learned>
<completed>[What was accomplished in this session?]</completed>
<next_steps>[What should be done next?]</next_steps>
<notes>[Additional insights or context]</notes>
</summary>
**Required fields**: request, investigated, learned, completed, next_steps
**Optional fields**: notes
docs/prompts/publish/prompt-init.mdx
-83
View File
@@ -1,83 +0,0 @@
You are a memory processor for a Claude Code session. Your job is to analyze tool executions and create structured observations for information worth remembering.
You are processing tool executions from a Claude Code session with the following context:
User's Goal: {userPrompt}
Date: {date}
WHEN TO STORE
-------------
Store observations when the tool output contains information worth remembering about:
- How things work
- Why things exist or were chosen
- What changed
- Problems and their solutions
- Important patterns or gotchas
WHEN TO SKIP
------------
Skip routine operations:
- Empty status checks
- Package installations with no errors
- Simple file listings
- Repetitive operations you've already documented
OUTPUT FORMAT
-------------
Output observations using this XML structure:
```xml
<observation>
<type>[ change | discovery | decision ]</type>
<!--
**type**: One of:
- change: modifications to code, config, or documentation
- discovery: learning about existing system
- decision: choosing an approach and why it was chosen
-->
<title>[**title**: Short title capturing the core action or topic]</title>
<subtitle>[**subtitle**: One sentence explanation (max 24 words)]</subtitle>
<facts>
<fact>[Concise, self-contained statement]</fact>
<fact>[Concise, self-contained statement]</fact>
<fact>[Concise, self-contained statement]</fact>
</facts>
<!--
**facts**: Concise, self-contained statements
Each fact is ONE piece of information
No pronouns - each fact must stand alone
Include specific details: filenames, functions, values
-->
<narrative>[**narrative**: Full context: What was done, how it works, why it matters]</narrative>
<concepts>
<concept>[knowledge-type-category]</concept>
<concept>[knowledge-type-category]</concept>
</concepts>
<!--
**concepts**: 2-5 knowledge-type categories:
- how-it-works: understanding mechanisms
- why-it-exists: purpose or rationale
- what-changed: modifications made
- problem-solution: issues and their fixes
- gotcha: traps or edge cases
- pattern: reusable approach
- trade-off: pros/cons of a decision
-->
<files_read>
<file>[path/to/file]</file>
<file>[path/to/file]</file>
</files_read>
<files_modified>
<file>[path/to/file]</file>
<file>[path/to/file]</file>
</files_modified>
<!--
**files**: All files touched (full paths from project root)
-->
</observation>
```
Process the following tool executions.
MEMORY PROCESSING SESSION START
===============================
docs/prompts/publish/prompt-observation.mdx
-6
View File
@@ -1,6 +0,0 @@
<tool_used>
<tool_name>[tool_name]</tool_name>
<tool_time>[time_formatted]</tool_time>
<tool_input>[tool_input JSON]</tool_input>
<tool_output>[tool_output JSON]</tool_output>
</tool_used>
docs/troubleshooting.md
+669
View File
@@ -0,0 +1,669 @@
# Troubleshooting Guide
## Worker Service Issues
### Worker Service Not Starting
**Symptoms**: Worker doesn't start, or `pm2 status` shows no processes.
**Solutions**:
1. Check if PM2 is running:
```bash
pm2 status
```
2. Try starting manually:
```bash
npm run worker:start
```
3. Check worker logs for errors:
```bash
npm run worker:logs
```
4. Full reset:
```bash
pm2 delete claude-mem-worker
npm run worker:start
```
5. Verify PM2 is installed:
```bash
which pm2
npm list pm2
```
### Port Allocation Failed
**Symptoms**: Worker fails to start with "port already in use" error.
**Solutions**:
1. Check if port 37777 is in use:
```bash
lsof -i :37777
```
2. Kill process using the port:
```bash
kill -9 $(lsof -t -i:37777)
```
3. Or use a different port:
```bash
export CLAUDE_MEM_WORKER_PORT=38000
npm run worker:restart
```
4. Verify new port:
```bash
cat ~/.claude-mem/worker.port
```
### Worker Keeps Crashing
**Symptoms**: Worker restarts repeatedly, PM2 shows high restart count.
**Solutions**:
1. Check error logs:
```bash
npm run worker:logs
```
2. Check memory usage:
```bash
pm2 status
```
3. Increase memory limit in `ecosystem.config.cjs`:
```javascript
{
max_memory_restart: '2G' // Increase if needed
}
```
4. Check database for corruption:
```bash
sqlite3 ~/.claude-mem/claude-mem.db "PRAGMA integrity_check;"
```
### Worker Not Processing Observations
**Symptoms**: Observations saved but not processed, no summaries generated.
**Solutions**:
1. Check worker is running:
```bash
npm run worker:status
```
2. Check worker logs:
```bash
npm run worker:logs
```
3. Verify database has observations:
```bash
sqlite3 ~/.claude-mem/claude-mem.db "SELECT COUNT(*) FROM observations;"
```
4. Restart worker:
```bash
npm run worker:restart
```
## Hook Issues
### Hooks Not Firing
**Symptoms**: No context appears, observations not saved.
**Solutions**:
1. Verify hooks are configured:
```bash
cat plugin/hooks/hooks.json
```
2. Test hooks manually:
```bash
# Test context hook
echo '{"session_id":"test-123","cwd":"'$(pwd)'","source":"startup"}' | node plugin/scripts/context-hook.js
```
3. Check hook permissions:
```bash
ls -la plugin/scripts/*.js
```
4. Verify hooks.json is valid JSON:
```bash
cat plugin/hooks/hooks.json | jq .
```
### Context Not Appearing
**Symptoms**: No session context when Claude starts.
**Solutions**:
1. Check if summaries exist:
```bash
sqlite3 ~/.claude-mem/claude-mem.db "SELECT COUNT(*) FROM session_summaries;"
```
2. View recent sessions:
```bash
npm run test:context:verbose
```
3. Check database integrity:
```bash
sqlite3 ~/.claude-mem/claude-mem.db "PRAGMA integrity_check;"
```
4. Manually test context hook:
```bash
npm run test:context
```
### Hooks Timeout
**Symptoms**: Hook execution times out, errors in Claude Code.
**Solutions**:
1. Increase timeout in `plugin/hooks/hooks.json`:
```json
{
"timeout": 180 // Increase from 120
}
```
2. Check worker is running (prevents timeout waiting for worker):
```bash
npm run worker:status
```
3. Check database size (large database = slow queries):
```bash
ls -lh ~/.claude-mem/claude-mem.db
```
4. Optimize database:
```bash
sqlite3 ~/.claude-mem/claude-mem.db "VACUUM;"
```
### Dependencies Not Installing
**Symptoms**: SessionStart hook fails with "module not found" errors.
**Solutions**:
1. Manually install dependencies:
```bash
cd ~/.claude/plugins/marketplaces/thedotmack
npm install
```
2. Check npm is available:
```bash
which npm
npm --version
```
3. Check package.json exists:
```bash
ls -la ~/.claude/plugins/marketplaces/thedotmack/package.json
```
## Database Issues
### Database Locked
**Symptoms**: "database is locked" errors in logs.
**Solutions**:
1. Close all connections:
```bash
pm2 stop claude-mem-worker
```
2. Check for stale locks:
```bash
lsof ~/.claude-mem/claude-mem.db
```
3. Kill processes holding locks:
```bash
kill -9 <PID>
```
4. Restart worker:
```bash
npm run worker:start
```
### Database Corruption
**Symptoms**: Integrity check fails, weird errors.
**Solutions**:
1. Check database integrity:
```bash
sqlite3 ~/.claude-mem/claude-mem.db "PRAGMA integrity_check;"
```
2. Backup database:
```bash
cp ~/.claude-mem/claude-mem.db ~/.claude-mem/claude-mem.db.backup
```
3. Try to repair:
```bash
sqlite3 ~/.claude-mem/claude-mem.db "VACUUM;"
```
4. Nuclear option - recreate database:
```bash
rm ~/.claude-mem/claude-mem.db
npm run worker:start # Will recreate schema
```
### FTS5 Search Not Working
**Symptoms**: Search returns no results, FTS5 errors.
**Solutions**:
1. Check FTS5 tables exist:
```bash
sqlite3 ~/.claude-mem/claude-mem.db "SELECT name FROM sqlite_master WHERE type='table' AND name LIKE '%_fts';"
```
2. Rebuild FTS5 tables:
```bash
sqlite3 ~/.claude-mem/claude-mem.db "
INSERT INTO observations_fts(observations_fts) VALUES('rebuild');
INSERT INTO session_summaries_fts(session_summaries_fts) VALUES('rebuild');
INSERT INTO user_prompts_fts(user_prompts_fts) VALUES('rebuild');
"
```
3. Check triggers exist:
```bash
sqlite3 ~/.claude-mem/claude-mem.db "SELECT name FROM sqlite_master WHERE type='trigger';"
```
### Database Too Large
**Symptoms**: Slow performance, large database file.
**Solutions**:
1. Check database size:
```bash
ls -lh ~/.claude-mem/claude-mem.db
```
2. Vacuum database:
```bash
sqlite3 ~/.claude-mem/claude-mem.db "VACUUM;"
```
3. Delete old sessions:
```bash
sqlite3 ~/.claude-mem/claude-mem.db "
DELETE FROM observations WHERE created_at_epoch < $(date -v-30d +%s);
DELETE FROM session_summaries WHERE created_at_epoch < $(date -v-30d +%s);
DELETE FROM sdk_sessions WHERE created_at_epoch < $(date -v-30d +%s);
"
```
4. Rebuild FTS5 after deletion:
```bash
sqlite3 ~/.claude-mem/claude-mem.db "
INSERT INTO observations_fts(observations_fts) VALUES('rebuild');
INSERT INTO session_summaries_fts(session_summaries_fts) VALUES('rebuild');
"
```
## MCP Search Issues
### Search Tools Not Available
**Symptoms**: MCP search tools not visible in Claude Code.
**Solutions**:
1. Check MCP configuration:
```bash
cat plugin/.mcp.json
```
2. Verify search server is built:
```bash
ls -l plugin/scripts/search-server.js
```
3. Rebuild if needed:
```bash
npm run build
```
4. Restart Claude Code
### Search Returns No Results
**Symptoms**: Valid queries return empty results.
**Solutions**:
1. Check database has data:
```bash
sqlite3 ~/.claude-mem/claude-mem.db "SELECT COUNT(*) FROM observations;"
```
2. Verify FTS5 tables populated:
```bash
sqlite3 ~/.claude-mem/claude-mem.db "SELECT COUNT(*) FROM observations_fts;"
```
3. Test simple query:
```bash
# In Claude Code
search_observations with query="test"
```
4. Check query syntax:
```bash
# Bad: Special characters
search_observations with query="[test]"
# Good: Simple words
search_observations with query="test"
```
### Token Limit Errors
**Symptoms**: "exceeded token limit" errors from MCP.
**Solutions**:
1. Use index format:
```bash
search_observations with query="..." and format="index"
```
2. Reduce limit:
```bash
search_observations with query="..." and limit=3
```
3. Use filters to narrow results:
```bash
search_observations with query="..." and type="decision" and limit=5
```
4. Paginate results:
```bash
# First page
search_observations with query="..." and limit=5 and offset=0
# Second page
search_observations with query="..." and limit=5 and offset=5
```
## Performance Issues
### Slow Context Injection
**Symptoms**: SessionStart hook takes too long.
**Solutions**:
1. Reduce context sessions:
```typescript
// In src/hooks/context.ts
const CONTEXT_SESSIONS = 5; // Reduce from 10
```
2. Optimize database:
```bash
sqlite3 ~/.claude-mem/claude-mem.db "
ANALYZE;
VACUUM;
"
```
3. Add indexes (if missing):
```bash
sqlite3 ~/.claude-mem/claude-mem.db "
CREATE INDEX IF NOT EXISTS idx_sessions_project_created ON sdk_sessions(project, created_at_epoch DESC);
"
```
### Slow Search Queries
**Symptoms**: MCP search tools take too long.
**Solutions**:
1. Use more specific queries
2. Add date range filters
3. Add type/concept filters
4. Reduce result limit
5. Use index format instead of full format
### High Memory Usage
**Symptoms**: Worker uses too much memory, frequent restarts.
**Solutions**:
1. Check current usage:
```bash
pm2 status
```
2. Increase memory limit:
```javascript
// In ecosystem.config.cjs
{
max_memory_restart: '2G'
}
```
3. Restart worker:
```bash
npm run worker:restart
```
4. Clean up old data (see "Database Too Large" above)
## Installation Issues
### Plugin Not Found
**Symptoms**: `/plugin install claude-mem` fails.
**Solutions**:
1. Add marketplace first:
```bash
/plugin marketplace add thedotmack/claude-mem
```
2. Then install:
```bash
/plugin install claude-mem
```
3. Verify installation:
```bash
ls -la ~/.claude/plugins/marketplaces/thedotmack/
```
### Build Failures
**Symptoms**: `npm run build` fails.
**Solutions**:
1. Clean and reinstall:
```bash
rm -rf node_modules package-lock.json
npm install
```
2. Check Node.js version:
```bash
node --version # Should be >= 18.0.0
```
3. Check for TypeScript errors:
```bash
npx tsc --noEmit
```
### Missing Dependencies
**Symptoms**: "Cannot find module" errors.
**Solutions**:
1. Install dependencies:
```bash
npm install
```
2. Check package.json:
```bash
cat package.json
```
3. Verify node_modules exists:
```bash
ls -la node_modules/
```
## Debugging
### Enable Verbose Logging
```bash
export DEBUG=claude-mem:*
npm run worker:restart
npm run worker:logs
```
### Check Correlation IDs
Trace observations through the pipeline:
```bash
sqlite3 ~/.claude-mem/claude-mem.db "
SELECT correlation_id, tool_name, created_at
FROM observations
WHERE session_id = 'YOUR_SESSION_ID'
ORDER BY created_at;
"
```
### Inspect Worker State
```bash
# Check if worker is running
pm2 status
# View logs
pm2 logs claude-mem-worker
# Check port file
cat ~/.claude-mem/worker.port
# Test worker health
curl http://localhost:37777/health
```
### Database Inspection
```bash
sqlite3 ~/.claude-mem/claude-mem.db
# View schema
.schema
# Check table counts
SELECT 'sessions', COUNT(*) FROM sdk_sessions
UNION ALL
SELECT 'observations', COUNT(*) FROM observations
UNION ALL
SELECT 'summaries', COUNT(*) FROM session_summaries
UNION ALL
SELECT 'prompts', COUNT(*) FROM user_prompts;
# View recent activity
SELECT created_at, tool_name FROM observations ORDER BY created_at DESC LIMIT 10;
```
## Common Error Messages
### "Worker service not responding"
**Cause**: Worker not running or port mismatch.
**Solution**: Restart worker with `npm run worker:restart`.
### "Database is locked"
**Cause**: Multiple processes accessing database.
**Solution**: Stop worker, kill stale processes, restart.
### "FTS5: syntax error"
**Cause**: Invalid search query syntax.
**Solution**: Use simpler query, avoid special characters.
### "SQLITE_CANTOPEN"
**Cause**: Database file permissions or missing directory.
**Solution**: Check `~/.claude-mem/` exists and is writable.
### "Module not found"
**Cause**: Missing dependencies.
**Solution**: Run `npm install`.
## Getting Help
If none of these solutions work:
1. **Check logs**:
```bash
npm run worker:logs
```
2. **Create issue**: [GitHub Issues](https://github.com/thedotmack/claude-mem/issues)
- Include error messages
- Include relevant logs
- Include steps to reproduce
3. **Check existing issues**: Someone may have already solved your problem
## Next Steps
- [Configuration](configuration.md) - Customize Claude-Mem
- [Development](development.md) - Build from source
- [Architecture](architecture/overview.md) - Understand the system
docs/usage/getting-started.md
+171
View File
@@ -0,0 +1,171 @@
# Getting Started with Claude-Mem
## Automatic Operation
Claude-Mem works automatically once installed. No manual intervention required!
### The Full Cycle
1. **Start Claude Code** - Context from last 3 sessions appears automatically
2. **Work normally** - Every tool execution is captured
3. **Stop Claude** - Summary is generated and saved
4. **Next session** - Previous work appears in context
### What Gets Captured
Every time Claude uses a tool, claude-mem captures it:
- **Read** - File reads and content access
- **Write** - New file creation
- **Edit** - File modifications
- **Bash** - Command executions
- **Glob** - File pattern searches
- **Grep** - Content searches
- And all other Claude Code tools
### What Gets Processed
The worker service processes tool observations and extracts:
- **Title** - Brief description of what happened
- **Subtitle** - Additional context
- **Narrative** - Detailed explanation
- **Facts** - Key learnings as bullet points
- **Concepts** - Relevant tags and categories
- **Type** - Classification (decision, bugfix, feature, etc.)
- **Files** - Which files were read or modified
### Session Summaries
When you stop Claude (or a session ends), a summary is generated with:
- **Request** - What you asked for
- **Investigated** - What Claude explored
- **Learned** - Key discoveries and insights
- **Completed** - What was accomplished
- **Next Steps** - What to do next
### Context Injection
When you start a new Claude Code session, the SessionStart hook:
1. Queries the database for recent sessions in your project
2. Retrieves the last 10 session summaries
3. Formats them with three-tier verbosity (most recent = most detail)
4. Injects them into Claude's initial context
This means Claude "remembers" what happened in previous sessions!
## Manual Commands (Optional)
### Worker Management
v4.0+ auto-starts the worker on first session. Manual commands below are optional.
```bash
# Start worker service (optional - auto-starts automatically)
npm run worker:start
# Stop worker service
npm run worker:stop
# Restart worker service
npm run worker:restart
# View worker logs
npm run worker:logs
# Check worker status
npm run worker:status
```
### Testing
```bash
# Run all tests
npm test
# Test context injection
npm run test:context
# Verbose context test
npm run test:context:verbose
```
### Development
```bash
# Build hooks and worker
npm run build
# Build only hooks
npm run build:hooks
# Publish to NPM (maintainers only)
npm run publish:npm
```
## Viewing Stored Context
Context is stored in SQLite database at `~/.claude-mem/claude-mem.db`.
Query the database directly:
```bash
# Open database
sqlite3 ~/.claude-mem/claude-mem.db
# View recent sessions
SELECT session_id, project, created_at, status
FROM sdk_sessions
ORDER BY created_at DESC
LIMIT 10;
# View session summaries
SELECT session_id, request, completed, learned
FROM session_summaries
ORDER BY created_at DESC
LIMIT 5;
# View observations for a session
SELECT tool_name, created_at
FROM observations
WHERE session_id = 'YOUR_SESSION_ID';
```
## Understanding Verbosity Levels
Context injection uses three-tier verbosity for efficient token usage:
### Tier 1 (Most Recent Session)
- Full summary with all details
- Request, investigated, learned, completed, next_steps, notes
- ~500-1000 tokens
### Tier 2 (Sessions 2-5)
- Medium detail
- Request, learned, completed
- ~200-400 tokens
### Tier 3 (Sessions 6-10)
- Brief summary
- Request and completed only
- ~100-200 tokens
This ensures you get maximum detail for recent work while still having context from older sessions.
## Multi-Prompt Sessions
Claude-Mem supports sessions that span multiple user prompts:
- **prompt_counter**: Tracks total prompts in a session
- **prompt_number**: Identifies specific prompt within session
- **Session continuity**: Observations and summaries link across prompts
When you use `/clear`, the session doesn't end - it continues with a new prompt number. This preserves context across conversation restarts.
## Next Steps
- [MCP Search Tools](search-tools.md) - Learn how to search your project history
- [Architecture Overview](../architecture/overview.md) - Understand how it works
- [Troubleshooting](../troubleshooting.md) - Common issues and solutions
docs/usage/search-tools.md
+421
View File
@@ -0,0 +1,421 @@
# MCP Search Tools Usage
Once claude-mem is installed as a plugin, 7 search tools become available in your Claude Code sessions for querying project history.
## Quick Reference
| Tool | Purpose |
|-------------------------|----------------------------------------------|
| search_observations | Full-text search across observations |
| search_sessions | Full-text search across session summaries |
| search_user_prompts | Full-text search across raw user prompts |
| find_by_concept | Find observations tagged with concepts |
| find_by_file | Find observations referencing files |
| find_by_type | Find observations by type |
| get_recent_context | Get recent session context |
## Example Queries
### search_observations
Find all decisions about the build system:
```
Use search_observations to find all decisions about the build system
```
Find bugs related to authentication:
```
search_observations with query="authentication" and type="bugfix"
```
Search for refactoring work:
```
search_observations with query="refactor database" and type="refactor"
```
### search_sessions
Find what we learned about hooks:
```
Use search_sessions to find what we learned about hooks
```
Search for completed work on the API:
```
search_sessions with query="API implementation"
```
### search_user_prompts
Find when user asked about authentication:
```
search_user_prompts with query="authentication feature"
```
Trace user requests for a specific feature:
```
search_user_prompts with query="dark mode"
```
**Benefits**:
- See exactly what the user asked for (vs what was implemented)
- Detect patterns in repeated requests
- Debug miscommunications between user intent and implementation
### find_by_file
Show everything related to worker-service.ts:
```
Use find_by_file to show me everything related to worker-service.ts
```
Find all work on the database migration file:
```
find_by_file with filePath="migrations.ts"
```
### find_by_concept
Show observations tagged with 'architecture':
```
Use find_by_concept to show observations tagged with 'architecture'
```
Find all 'security' related observations:
```
find_by_concept with concept="security"
```
### find_by_type
Find all feature implementations:
```
find_by_type with type="feature"
```
Find all decisions and discoveries:
```
find_by_type with type=["decision", "discovery"]
```
### get_recent_context
Get the last 5 sessions for context:
```
get_recent_context with limit=5
```
Get recent context for debugging:
```
Use get_recent_context to show me what we've been working on
```
## Search Strategy
### 1. Start with Index Format
**Always use index format first** to get an overview:
```
search_observations with query="authentication" and format="index"
```
**Why?**
- Index format uses ~10x fewer tokens than full format
- See titles, dates, and sources to identify relevant results
- Avoid hitting MCP token limits
### 2. Review Results
Look at the index results to identify items of interest:
```
1. [decision] Implement JWT authentication
Date: 2025-10-21 14:23:45
Source: claude-mem://observation/123
2. [feature] Add user authentication endpoints
Date: 2025-10-21 13:15:22
Source: claude-mem://observation/124
3. [bugfix] Fix authentication token expiry
Date: 2025-10-20 16:45:30
Source: claude-mem://observation/125
```
### 3. Deep Dive with Full Format
Only use full format for specific items:
```
search_observations with query="JWT authentication" and format="full" and limit=3
```
### 4. Use Filters to Narrow Results
Combine filters for precise searches:
```
search_observations with query="authentication" and type="decision" and dateRange={start: "2025-10-20", end: "2025-10-21"}
```
## Advanced Filtering
### Date Ranges
Search within specific time periods:
```json
{
"dateRange": {
"start": "2025-10-01",
"end": "2025-10-31"
}
}
```
Or use epoch timestamps:
```json
{
"dateRange": {
"start": 1729449600,
"end": 1732128000
}
}
```
### Multiple Types
Search across multiple observation types:
```
find_by_type with type=["decision", "feature", "refactor"]
```
### Multiple Concepts
Search observations with specific concepts:
```
search_observations with query="database" and concepts=["architecture", "performance"]
```
### File Filtering
Search observations that touched specific files:
```
search_observations with query="refactor" and files="worker-service.ts"
```
### Project Filtering
Search within specific projects:
```
search_observations with query="authentication" and project="my-app"
```
## FTS5 Query Syntax
The `query` parameter supports SQLite FTS5 full-text search syntax:
### Simple Queries
```
"authentication" # Single word
"error handling" # Multiple words (OR)
```
### Boolean Operators
```
"error" AND "handling" # Both terms required
"bug" OR "fix" # Either term
"bug" NOT "feature" # First term, not second
```
### Phrase Searches
```
"'exact phrase'" # Exact phrase match
```
### Column Searches
```
title:"authentication" # Search specific column
narrative:"bug fix" # Search narrative field
```
## Result Metadata
All results include rich metadata:
```
## JWT authentication decision
**Type**: decision
**Date**: 2025-10-21 14:23:45
**Concepts**: authentication, security, architecture
**Files Read**: src/auth/middleware.ts, src/utils/jwt.ts
**Files Modified**: src/auth/jwt-strategy.ts
**Narrative**:
Decided to implement JWT-based authentication instead of session-based
authentication for better scalability and stateless design...
**Facts**:
• JWT tokens expire after 1 hour
• Refresh tokens stored in httpOnly cookies
• Token signing uses RS256 algorithm
• Public keys rotated every 30 days
```
## Citations
All search results include citations using the `claude-mem://` URI scheme:
- `claude-mem://observation/123` - Specific observation
- `claude-mem://session/abc-456` - Specific session
- `claude-mem://user-prompt/789` - Specific user prompt
These citations enable referencing specific historical context in your work.
## Token Management
### Token Efficiency Tips
1. **Start with index format**: ~50-100 tokens per result
2. **Use small limits**: Start with 3-5 results
3. **Apply filters**: Narrow results before searching
4. **Paginate**: Use offset to browse results in batches
### Token Estimates
| Format | Tokens per Result |
|--------|-------------------|
| Index | 50-100 |
| Full | 500-1000 |
**Example**:
- 20 results in index format: ~1,000-2,000 tokens
- 20 results in full format: ~10,000-20,000 tokens
## Common Use Cases
### 1. Debugging Issues
Find what went wrong:
```
search_observations with query="error database connection" and type="bugfix"
```
### 2. Understanding Decisions
Review architectural choices:
```
find_by_type with type="decision" and format="index"
```
Then deep dive on specific decisions:
```
search_observations with query="[DECISION TITLE]" and format="full"
```
### 3. Code Archaeology
Find when a file was modified:
```
find_by_file with filePath="worker-service.ts"
```
### 4. Feature History
Track feature development:
```
search_sessions with query="authentication feature"
search_user_prompts with query="add authentication"
```
### 5. Learning from Past Work
Review refactoring patterns:
```
find_by_type with type="refactor" and limit=10
```
### 6. Context Recovery
Restore context after time away:
```
get_recent_context with limit=5
search_sessions with query="[YOUR PROJECT NAME]" and orderBy="date_desc"
```
## Best Practices
1. **Index first, full later**: Always start with index format
2. **Small limits**: Start with 3-5 results to avoid token limits
3. **Use filters**: Narrow results before searching
4. **Specific queries**: More specific = better results
5. **Review citations**: Use citations to reference past decisions
6. **Date filtering**: Use date ranges for time-based searches
7. **Type filtering**: Use types to categorize searches
8. **Concept tags**: Use concepts for thematic searches
## Troubleshooting
### No Results Found
1. Check database has data:
```bash
sqlite3 ~/.claude-mem/claude-mem.db "SELECT COUNT(*) FROM observations;"
```
2. Try broader query:
```
search_observations with query="authentication" # Good
vs
search_observations with query="'exact JWT authentication implementation'" # Too specific
```
3. Remove filters:
```
# Start broad
search_observations with query="auth"
# Then add filters
search_observations with query="auth" and type="decision"
```
### Token Limit Errors
1. Use index format:
```
search_observations with query="..." and format="index"
```
2. Reduce limit:
```
search_observations with query="..." and limit=3
```
3. Use pagination:
```
# First page
search_observations with query="..." and limit=5 and offset=0
# Second page
search_observations with query="..." and limit=5 and offset=5
```
### Search Too Slow
1. Use more specific queries
2. Add date range filters
3. Add type/concept filters
4. Reduce result limit
## Next Steps
- [MCP Search Architecture](../architecture/mcp-search.md) - Technical details
- [Database Schema](../architecture/database.md) - Understanding the data
- [Getting Started](getting-started.md) - Automatic operation