airkjw/claude-mem
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Replace search skill with mem-search (#91)

Browse Source 
* feat: add mem-search skill with progressive disclosure architecture

Add comprehensive mem-search skill for accessing claude-mem's persistent
cross-session memory database. Implements progressive disclosure workflow
and token-efficient search patterns.

Features:
- 12 search operations (observations, sessions, prompts, by-type, by-concept, by-file, timelines, etc.)
- Progressive disclosure principles to minimize token usage
- Anti-patterns documentation to guide LLM behavior
- HTTP API integration for all search functionality
- Common workflows with composition examples

Structure:
- SKILL.md: Entry point with temporal trigger patterns
- principles/: Progressive disclosure + anti-patterns
- operations/: 12 search operation files

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* docs: add CHANGELOG entry for mem-search skill

Document mem-search skill addition in Unreleased section with:
- 100% effectiveness compliance metrics
- Comparison to previous search skill implementation
- Progressive disclosure architecture details
- Reference to audit report documentation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* docs: add mem-search skill audit report

Add comprehensive audit report validating mem-search skill against
Anthropic's official skill-creator documentation.

Report includes:
- Effectiveness metrics comparison (search vs mem-search)
- Critical issues analysis for production readiness
- Compliance validation across 6 key dimensions
- Reference implementation guidance

Result: mem-search achieves 100% compliance vs search's 67%

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat: Add comprehensive search architecture analysis document

- Document current state of dual search architectures (HTTP API and MCP)
- Analyze HTTP endpoints and MCP search server architectures
- Identify DRY violations across search implementations
- Evaluate the use of curl as the optimal approach for search
- Provide architectural recommendations for immediate and long-term improvements
- Outline action plan for cleanup, feature parity, DRY refactoring

* refactor: Remove deprecated search skill documentation and operations

* refactor: Reorganize documentation into public and context directories

Changes:
- Created docs/public/ for Mintlify documentation (.mdx files)
- Created docs/context/ for internal planning and implementation docs
- Moved all .mdx files and assets to docs/public/
- Moved all internal .md files to docs/context/
- Added CLAUDE.md to both directories explaining their purpose
- Updated docs.json paths to work with new structure

Benefits:
- Clear separation between user-facing and internal documentation
- Easier to maintain Mintlify docs in dedicated directory
- Internal context files organized separately

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Enhance session management and continuity in hooks

- Updated new-hook.ts to clarify session_id threading and idempotent session creation.
- Modified prompts.ts to require claudeSessionId for continuation prompts, ensuring session context is maintained.
- Improved SessionStore.ts documentation on createSDKSession to emphasize idempotent behavior and session connection.
- Refined SDKAgent.ts to detail continuation prompt logic and its reliance on session.claudeSessionId for unified session handling.

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Alex Newman <thedotmack@gmail.com>
This commit is contained in:
basher83
2025-11-11 16:15:07 -05:00
committed by GitHub
parent eafdd6a7be
commit 97d565e3cd
92 changed files with 5038 additions and 1812 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/public/usage/getting-started.mdx
+215
View File
@@ -0,0 +1,215 @@
---
title: "Getting Started"
description: "Learn how Claude-Mem works automatically in the background"
---
# Getting Started with Claude-Mem
## Automatic Operation
Claude-Mem works automatically once installed. No manual intervention required!
### The Full Cycle
1. **Start Claude Code** - Context from last 10 sessions appears automatically
2. **Work normally** - Every tool execution is captured
3. **Claude finishes responding** - Stop hook automatically generates and saves a summary
4. **Next session** - Previous work appears in context
### What Gets Captured
Every time Claude uses a tool, claude-mem captures it:
- **Read** - File reads and content access
- **Write** - New file creation
- **Edit** - File modifications
- **Bash** - Command executions
- **Glob** - File pattern searches
- **Grep** - Content searches
- And all other Claude Code tools
### What Gets Processed
The worker service processes tool observations and extracts:
- **Title** - Brief description of what happened
- **Subtitle** - Additional context
- **Narrative** - Detailed explanation
- **Facts** - Key learnings as bullet points
- **Concepts** - Relevant tags and categories
- **Type** - Classification (decision, bugfix, feature, etc.)
- **Files** - Which files were read or modified
### Session Summaries
When Claude finishes responding (triggering the Stop hook), a summary is automatically generated with:
- **Request** - What you asked for
- **Investigated** - What Claude explored
- **Learned** - Key discoveries and insights
- **Completed** - What was accomplished
- **Next Steps** - What to do next
### Context Injection
When you start a new Claude Code session, the SessionStart hook:
1. Queries the database for recent observations in your project (default: 50)
2. Retrieves recent session summaries for context
3. Displays observations in a chronological timeline with session markers
4. Shows full summary details (Investigated, Learned, Completed, Next Steps) **only if the summary was generated after the last observation**
5. Injects formatted context into Claude's initial context
**Summary Display Logic:**
The most recent summary's full details appear at the end of the context display **only when** the summary was generated after the most recent observation. This ensures you see summary details when they represent the latest state of your project, but not when new observations have been captured since the last summary.
For example:
- ✅ **Shows summary**: Last observation at 2:00 PM, summary generated at 2:05 PM → Summary details appear
- ❌ **Hides summary**: Summary generated at 2:00 PM, new observation at 2:05 PM → Summary details hidden (outdated)
This prevents showing stale summaries when new work has been captured but not yet summarized.
This means Claude "remembers" what happened in previous sessions!
## Manual Commands (Optional)
### Worker Management
v4.0+ auto-starts the worker on first session. Manual commands below are optional.
```bash
# Start worker service (optional - auto-starts automatically)
npm run worker:start
# Stop worker service
npm run worker:stop
# Restart worker service
npm run worker:restart
# View worker logs
npm run worker:logs
# Check worker status
npm run worker:status
```
### Testing
```bash
# Run all tests
npm test
# Test context injection
npm run test:context
# Verbose context test
npm run test:context:verbose
```
### Development
```bash
# Build hooks and worker
npm run build
# Build only hooks
npm run build:hooks
# Publish to NPM (maintainers only)
npm run publish:npm
```
## Viewing Stored Context
Context is stored in SQLite database at `~/.claude-mem/claude-mem.db`.
Query the database directly:
```bash
# Open database
sqlite3 ~/.claude-mem/claude-mem.db
# View recent sessions
SELECT session_id, project, created_at, status
FROM sdk_sessions
ORDER BY created_at DESC
LIMIT 10;
# View session summaries
SELECT session_id, request, completed, learned
FROM session_summaries
ORDER BY created_at DESC
LIMIT 5;
# View observations for a session
SELECT tool_name, created_at
FROM observations
WHERE session_id = 'YOUR_SESSION_ID';
```
## Understanding Progressive Disclosure
Context injection uses progressive disclosure for efficient token usage:
### Layer 1: Index Display (Session Start)
- Shows observation titles with token cost estimates
- Displays session markers in chronological timeline
- Groups observations by file for visual clarity
- Shows full summary details **only if** generated after last observation
- Token cost: ~50-200 tokens for index view
### Layer 2: On-Demand Details (Skill-Based Search)
- Ask naturally: "What bugs did we fix?" or "How did we implement X?"
- Claude auto-invokes search skill to fetch full details
- Search by concept, file, type, or keyword
- Timeline context around specific observations
- Token cost: ~100-500 tokens per observation fetched
- Skill uses HTTP API (v5.4.0+) for efficient retrieval
### Layer 3: Perfect Recall (Code Access)
- Read source files directly when needed
- Access original transcripts and raw data
- Full context available on-demand
This ensures efficient token usage while maintaining access to complete history when needed.
## Multi-Prompt Sessions & `/clear` Behavior
Claude-Mem supports sessions that span multiple user prompts:
- **prompt_counter**: Tracks total prompts in a session
- **prompt_number**: Identifies specific prompt within session
- **Session continuity**: Observations and summaries link across prompts
### Important Note About `/clear`
When you use `/clear`, the session doesn't end - it continues with a new prompt number. This means:
- ✅ **Context is re-injected** from recent sessions (SessionStart hook fires with `source: "clear"`)
- ✅ **Observations are still being captured** and added to the current session
- ✅ **A summary will be generated** when Claude finishes responding (Stop hook fires)
The `/clear` command clears the conversation context visible to Claude AND re-injects fresh context from recent sessions, while the underlying session continues tracking observations.
## Searching Your History (v5.4.0+)
Claude-Mem now uses skill-based search for querying your project history. Simply ask naturally:
```
"What bugs did we fix last session?"
"How did we implement authentication?"
"What changes were made to worker-service.ts?"
"Show me recent work on this project"
```
Claude automatically recognizes your intent and invokes the search skill, which uses HTTP API endpoints to query your memory efficiently.
**Token Savings**: ~2,250 tokens per session start vs previous MCP approach
## Next Steps
- [Skill-Based Search](/usage/search-tools) - Learn how to search your project history
- [Architecture Overview](/architecture/overview) - Understand how it works
- [Troubleshooting](/troubleshooting) - Common issues and solutions
docs/public/usage/search-tools.mdx
+402
View File
@@ -0,0 +1,402 @@
---
title: "Skill-Based Search"
description: "Query your project history with natural language"
---
# Skill-Based Search Usage
Once claude-mem is installed as a plugin, you can search your project history using natural language. Claude automatically invokes the search skill when you ask about past work.
## How It Works
**v5.4.0 Migration**: Claude-Mem now uses a skill-based search architecture instead of MCP tools, saving ~2,250 tokens per session start through progressive disclosure.
**Simple Usage:**
- Just ask naturally: *"What did we do last session?"*
- Claude recognizes the intent and invokes the search skill
- The skill uses HTTP API endpoints to query your memory
- Results are formatted and presented to you
**Benefits:**
- **Token Efficient**: ~250 tokens (skill frontmatter) vs ~2,500 tokens (MCP tool definitions)
- **Natural Language**: No need to learn specific tool syntax
- **Progressive Disclosure**: Only loads detailed instructions when needed
- **Auto-Invoked**: Claude knows when to search based on your questions
## Quick Reference
| Operation | Purpose |
|-------------------------|----------------------------------------------|
| Search Observations | Full-text search across observations |
| Search Sessions | Full-text search across session summaries |
| Search Prompts | Full-text search across raw user prompts |
| By Concept | Find observations tagged with concepts |
| By File | Find observations referencing files |
| By Type | Find observations by type |
| Recent Context | Get recent session context |
| Timeline | Get unified timeline around a specific point |
| Timeline by Query | Search and get timeline context in one step |
| API Help | Get search API documentation |
## Example Queries
### Natural Language Queries
**Search Observations:**
```
"What bugs did we fix related to authentication?"
"Show me all decisions about the build system"
"Find refactoring work on the database"
```
**Search Sessions:**
```
"What did we learn about hooks?"
"What was accomplished in the API implementation?"
"Show me recent work on this project"
```
**Search Prompts:**
```
"When did I ask about authentication features?"
"Find all my requests about dark mode"
```
**Note**: Claude automatically translates your natural language queries into the appropriate search operations.
### Search by File
```
"Show me everything related to worker-service.ts"
"What changes were made to migrations.ts?"
"Find all work on the database file"
```
### Search by Concept
```
"Show observations tagged with architecture"
"Find all security-related observations"
"What patterns have we used?"
```
### Search by Type
```
"Find all feature implementations"
"Show me all decisions and discoveries"
"What bugs have we fixed?"
```
### Recent Context
```
"Show me what we've been working on"
"Get context from the last 5 sessions"
"What happened recently on this project?"
```
### Timeline Queries
**Get timeline around a specific point:**
```
"What was happening when we implemented authentication?"
"Show me the context around that bug fix"
"What led to the decision to refactor the database?"
```
**Timeline by query:**
```
"Find when we added the viewer UI and show what happened around that time"
"Search for authentication work and show the timeline"
```
**Benefits:**
- See the complete narrative arc around key events
- All record types (observations, sessions, prompts) in chronological view
- Understand what was happening before and after important changes
## Search Strategy
The search skill uses a progressive disclosure pattern to efficiently retrieve information:
### 1. Ask Naturally
Start with a natural language question:
```
"What bugs did we fix related to authentication?"
```
### 2. Claude Invokes Search Skill
Claude recognizes your intent and loads the search skill (~250 tokens for skill frontmatter).
### 3. Skill Uses HTTP API
The skill calls the appropriate HTTP endpoint (e.g., `/api/search/observations`) with the query.
### 4. Results Formatted
Results are formatted and presented to you, usually starting with an index/summary format.
### 5. Deep Dive if Needed
If you need more details, ask follow-up questions:
```
"Tell me more about observation #123"
"Show me the full details of that decision"
```
**Benefits of This Approach:**
- **Token Efficient**: Only loads what you need, when you need it
- **Natural**: No syntax to learn
- **Progressive**: Start with overview, drill down as needed
- **Automatic**: Claude handles the search invocation
## Advanced Filtering
You can refine searches using natural language filters:
### Date Ranges
```
"What bugs did we fix in October?"
"Show me work from last week"
"Find decisions made between October 1-31"
```
### Multiple Types
```
"Show me all decisions and features"
"Find bugfixes and refactorings"
```
### Concepts
```
"Find database work related to architecture and performance"
"Show security observations"
```
### File-Specific
```
"Show refactoring work that touched worker-service.ts"
"Find changes to auth files"
```
### Project Filtering
```
"Show authentication work on my-app project"
"What have we done on this codebase?"
```
**Note**: Claude translates your natural language into the appropriate API filters automatically.
## Under the Hood: HTTP API
The search skill uses HTTP endpoints on the worker service (port 37777):
- `GET /api/search/observations` - Full-text search observations
- `GET /api/search/sessions` - Full-text search session summaries
- `GET /api/search/prompts` - Full-text search user prompts
- `GET /api/search/by-concept` - Find observations by concept tag
- `GET /api/search/by-file` - Find work related to specific files
- `GET /api/search/by-type` - Find observations by type
- `GET /api/context/recent` - Get recent session context
- `GET /api/context/timeline` - Get timeline around specific point
- `GET /api/timeline/by-query` - Search + timeline in one call
- `GET /api/search/help` - API documentation
These endpoints use FTS5 full-text search with support for:
- Boolean operators (AND, OR, NOT)
- Phrase searches
- Column-specific searches
- Date range filtering
- Project filtering
## Result Metadata
All results include rich metadata:
```
## JWT authentication decision
**Type**: decision
**Date**: 2025-10-21 14:23:45
**Concepts**: authentication, security, architecture
**Files Read**: src/auth/middleware.ts, src/utils/jwt.ts
**Files Modified**: src/auth/jwt-strategy.ts
**Narrative**:
Decided to implement JWT-based authentication instead of session-based
authentication for better scalability and stateless design...
**Facts**:
• JWT tokens expire after 1 hour
• Refresh tokens stored in httpOnly cookies
• Token signing uses RS256 algorithm
• Public keys rotated every 30 days
```
## Citations
All search results include citations using the `claude-mem://` URI scheme:
- `claude-mem://observation/123` - Specific observation
- `claude-mem://session/abc-456` - Specific session
- `claude-mem://user-prompt/789` - Specific user prompt
These citations enable referencing specific historical context in your work.
## Token Management
### Token Efficiency Tips
1. **Start with index format**: ~50-100 tokens per result
2. **Use small limits**: Start with 3-5 results
3. **Apply filters**: Narrow results before searching
4. **Paginate**: Use offset to browse results in batches
### Token Estimates
| Format | Tokens per Result |
|--------|-------------------|
| Index | 50-100 |
| Full | 500-1000 |
**Example**:
- 20 results in index format: ~1,000-2,000 tokens
- 20 results in full format: ~10,000-20,000 tokens
## Common Use Cases
### 1. Debugging Issues
Find what went wrong:
```
search_observations with query="error database connection" and type="bugfix"
```
### 2. Understanding Decisions
Review architectural choices:
```
find_by_type with type="decision" and format="index"
```
Then deep dive on specific decisions:
```
search_observations with query="[DECISION TITLE]" and format="full"
```
### 3. Code Archaeology
Find when a file was modified:
```
find_by_file with filePath="worker-service.ts"
```
### 4. Feature History
Track feature development:
```
search_sessions with query="authentication feature"
search_user_prompts with query="add authentication"
```
### 5. Learning from Past Work
Review refactoring patterns:
```
find_by_type with type="refactor" and limit=10
```
### 6. Context Recovery
Restore context after time away:
```
get_recent_context with limit=5
search_sessions with query="[YOUR PROJECT NAME]" and orderBy="date_desc"
```
## Best Practices
1. **Index first, full later**: Always start with index format
2. **Small limits**: Start with 3-5 results to avoid token limits
3. **Use filters**: Narrow results before searching
4. **Specific queries**: More specific = better results
5. **Review citations**: Use citations to reference past decisions
6. **Date filtering**: Use date ranges for time-based searches
7. **Type filtering**: Use types to categorize searches
8. **Concept tags**: Use concepts for thematic searches
## Troubleshooting
### No Results Found
1. Check database has data:
```bash
sqlite3 ~/.claude-mem/claude-mem.db "SELECT COUNT(*) FROM observations;"
```
2. Try broader natural language query:
```
"Show me anything about authentication" # Broader
vs
"Find exact JWT authentication implementation" # Too specific
```
3. Ask without filters first:
```
"What do we have about auth?"
# Then narrow down
"Show me auth-related decisions"
```
### Worker Service Not Running
If search isn't working, check the worker service:
```bash
pm2 list # Check worker status
npm run worker:restart # Restart if needed
npm run worker:logs # View logs
```
Or use the troubleshooting skill:
```
/skill troubleshoot
```
### Performance Issues
If searches seem slow:
1. Be more specific in your queries
2. Ask for recent work (naturally filters by date)
3. Specify the project you're interested in
4. Ask for fewer results initially
## Technical Details
**Architecture Change (v5.4.0)**:
- **Before**: 9 MCP tools (~2,500 tokens in tool definitions per session start)
- **After**: 1 search skill (~250 tokens in frontmatter, full instructions loaded on-demand)
- **Savings**: ~2,250 tokens per session start
- **Migration**: Transparent - users don't need to change how they ask questions
**How the Skill Works:**
1. User asks a question about past work
2. Claude recognizes the intent matches the search skill description
3. Skill loads full instructions from `plugin/skills/search/SKILL.md`
4. Skill uses `curl` to call HTTP API endpoints
5. Results formatted and returned to Claude
6. Claude presents results to user
## Next Steps
- [Architecture Overview](/architecture/overview) - System components
- [Database Schema](/architecture/database) - Understanding the data
- [Getting Started](/usage/getting-started) - Automatic operation