airkjw/claude-mem
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add smart-explore AST-based code navigation (#1244)

Browse Source 
* feat: add smart-file-read module for token-optimized semantic code search

- Created package.json for the smart-file-read module with dependencies and scripts.
- Implemented parser.ts for code structure parsing using tree-sitter, supporting multiple languages.
- Developed search.ts for searching code files and symbols with grep-style and structural matching.
- Added test-run.mjs for testing search and outline functionalities.
- Configured TypeScript with tsconfig.json for strict type checking and module resolution.

* fix: update .gitignore to include _tree-sitter and remove unused subproject

* feat: add preliminary results and skill recommendation for smart-explore module

* chore: remove outdated plan.md file detailing session start hook issues

* feat: update Smart File Read integration plan and skill documentation for smart-explore

* feat: migrate Smart File Read to web-tree-sitter WASM for cross-platform compatibility

* refactor: switch to tree-sitter CLI for parsing and enhance search functionality

- Updated `parser.ts` to utilize the tree-sitter CLI for AST extraction instead of native bindings, improving compatibility and performance.
- Removed grammar loading logic and replaced it with a path resolution for grammar packages.
- Implemented batch parsing in `parseFilesBatch` to handle multiple files in a single CLI call, enhancing search speed.
- Refactored `searchCodebase` to collect files and parse them in batches, streamlining the search process.
- Adjusted symbol extraction logic to accommodate the new parsing method and ensure accurate symbol matching.

* feat: update Smart File Read integration plan to utilize tree-sitter CLI for improved performance and cross-platform compatibility

* feat: add smart-file-read parser and search to src/services

Copy validated tree-sitter CLI-based parser and search modules from
smart-file-read prototype into the claude-mem source tree for MCP
tool integration.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: register smart_search, smart_unfold, smart_outline MCP tools

Add 3 tree-sitter AST-based code exploration tools to the MCP server.
Direct execution (no HTTP delegation) — they call parser/search
functions directly for sub-second response times.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add tree-sitter CLI deps to build system and plugin runtime

Externalize tree-sitter packages in esbuild MCP server build. Add
10 grammar packages + CLI to plugin package.json for runtime install.
Remove unused @chroma-core/default-embed from plugin deps.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: create smart-explore skill with 3-layer workflow docs

Progressive disclosure workflow: search -> outline -> unfold.
Documents all 3 MCP tools with parameters and token economics.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add comprehensive documentation for the smart-explore feature

- Introduced a detailed technical reference covering the architecture, parser, search engine, and tool registration for the smart-explore feature in claude-mem.
- Documented the three-layer workflow: search, outline, and unfold, along with their respective MCP tools.
- Explained the parsing process using tree-sitter, including language support, query patterns, and symbol extraction.
- Outlined the search module's functionality, including file discovery, batch parsing, and relevance scoring.
- Provided insights into build system integration and token economics for efficient code exploration.

* chore: remove experiment artifacts, prototypes, and plan files

Remove A/B test docs, prototype smart-file-read directory, and
implementation plans. Keep only production code.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: simplify hooks configuration and remove setup script

* fix: use execFileSync to prevent command injection in tree-sitter parser

Replaces execSync shell string with execFileSync + argument array,
eliminating shell interpretation of file paths. Also corrects
file_pattern description from "Glob pattern" to "Substring filter".

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Alex Newman
2026-02-25 21:00:26 -05:00
committed by GitHub
parent 9ab119932a
commit 0e502dbd21
17 changed files with 1634 additions and 594 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plugin/hooks/hooks.json
+7 -30
View File
@@ -1,40 +1,18 @@
{
"description": "Claude-mem memory system hooks",
"hooks": {
"Setup": [
{
"matcher": "*",
"hooks": [
{
"type": "command",
"command": "_R=\"${CLAUDE_PLUGIN_ROOT}\"; [ -z \"$_R\" ] && _R=\"$HOME/.claude/plugins/marketplaces/thedotmack/plugin\"; \"$_R/scripts/setup.sh\"",
"timeout": 300
}
]
}
],
"SessionStart": [
{
"matcher": "startup|clear|compact",
"hooks": [
{
"type": "command",
"command": "_R=\"${CLAUDE_PLUGIN_ROOT}\"; [ -z \"$_R\" ] && _R=\"$HOME/.claude/plugins/marketplaces/thedotmack/plugin\"; node \"$_R/scripts/smart-install.js\"",
"command": "_R=\"${CLAUDE_PLUGIN_ROOT:-$HOME/.claude/plugins/marketplaces/thedotmack/plugin}\"; node \"$_R/scripts/smart-install.js\"",
"timeout": 300
}
]
},
{
"matcher": "startup|clear|compact",
"hooks": [
{
"type": "command",
"command": "_R=\"${CLAUDE_PLUGIN_ROOT}\"; [ -z \"$_R\" ] && _R=\"$HOME/.claude/plugins/marketplaces/thedotmack/plugin\"; node \"$_R/scripts/bun-runner.js\" \"$_R/scripts/worker-service.cjs\" start",
"timeout": 60
},
{
"type": "command",
"command": "_R=\"${CLAUDE_PLUGIN_ROOT}\"; [ -z \"$_R\" ] && _R=\"$HOME/.claude/plugins/marketplaces/thedotmack/plugin\"; node \"$_R/scripts/bun-runner.js\" \"$_R/scripts/worker-service.cjs\" hook claude-code context",
"command": "_R=\"${CLAUDE_PLUGIN_ROOT:-$HOME/.claude/plugins/marketplaces/thedotmack/plugin}\"; node \"$_R/scripts/bun-runner.js\" \"$_R/scripts/worker-service.cjs\" hook claude-code context",
"timeout": 60
}
]
@@ -45,7 +23,7 @@
"hooks": [
{
"type": "command",
"command": "_R=\"${CLAUDE_PLUGIN_ROOT}\"; [ -z \"$_R\" ] && _R=\"$HOME/.claude/plugins/marketplaces/thedotmack/plugin\"; node \"$_R/scripts/bun-runner.js\" \"$_R/scripts/worker-service.cjs\" hook claude-code session-init",
"command": "_R=\"${CLAUDE_PLUGIN_ROOT:-$HOME/.claude/plugins/marketplaces/thedotmack/plugin}\"; node \"$_R/scripts/bun-runner.js\" \"$_R/scripts/worker-service.cjs\" hook claude-code session-init",
"timeout": 60
}
]
@@ -53,11 +31,10 @@
],
"PostToolUse": [
{
"matcher": "*",
"hooks": [
{
"type": "command",
"command": "_R=\"${CLAUDE_PLUGIN_ROOT}\"; [ -z \"$_R\" ] && _R=\"$HOME/.claude/plugins/marketplaces/thedotmack/plugin\"; node \"$_R/scripts/bun-runner.js\" \"$_R/scripts/worker-service.cjs\" hook claude-code observation",
"command": "_R=\"${CLAUDE_PLUGIN_ROOT:-$HOME/.claude/plugins/marketplaces/thedotmack/plugin}\"; node \"$_R/scripts/bun-runner.js\" \"$_R/scripts/worker-service.cjs\" hook claude-code observation",
"timeout": 120
}
]
@@ -68,13 +45,13 @@
"hooks": [
{
"type": "command",
"command": "_R=\"${CLAUDE_PLUGIN_ROOT}\"; [ -z \"$_R\" ] && _R=\"$HOME/.claude/plugins/marketplaces/thedotmack/plugin\"; node \"$_R/scripts/bun-runner.js\" \"$_R/scripts/worker-service.cjs\" hook claude-code summarize",
"command": "_R=\"${CLAUDE_PLUGIN_ROOT:-$HOME/.claude/plugins/marketplaces/thedotmack/plugin}\"; node \"$_R/scripts/bun-runner.js\" \"$_R/scripts/worker-service.cjs\" hook claude-code summarize",
"timeout": 120
},
{
"type": "command",
"command": "_R=\"${CLAUDE_PLUGIN_ROOT}\"; [ -z \"$_R\" ] && _R=\"$HOME/.claude/plugins/marketplaces/thedotmack/plugin\"; node \"$_R/scripts/bun-runner.js\" \"$_R/scripts/worker-service.cjs\" hook claude-code session-complete",
"timeout": 30
"command": "_R=\"${CLAUDE_PLUGIN_ROOT:-$HOME/.claude/plugins/marketplaces/thedotmack/plugin}\"; node \"$_R/scripts/bun-runner.js\" \"$_R/scripts/worker-service.cjs\" hook claude-code session-complete",
"timeout": 120
}
]
}
plugin/package.json
+10 -1
View File
@@ -5,7 +5,16 @@
"description": "Runtime dependencies for claude-mem bundled hooks",
"type": "module",
"dependencies": {
"@chroma-core/default-embed": "^0.1.9"
"tree-sitter-cli": "^0.26.5",
"tree-sitter-c": "^0.24.1",
"tree-sitter-cpp": "^0.23.4",
"tree-sitter-go": "^0.25.0",
"tree-sitter-java": "^0.23.5",
"tree-sitter-javascript": "^0.25.0",
"tree-sitter-python": "^0.25.0",
"tree-sitter-ruby": "^0.23.1",
"tree-sitter-rust": "^0.24.0",
"tree-sitter-typescript": "^0.23.2"
},
"engines": {
"node": ">=18.0.0",
plugin/scripts/CLAUDE.md
+2
View File
@@ -1,3 +1,5 @@
Never read built source files in this directory. These are compiled outputs — read the source files in `src/` instead.
<claude-mem-context>
# Recent Activity
plugin/scripts/mcp-server.cjs
+88 -24
View File
File diff suppressed because one or more lines are too long
plugin/scripts/setup.sh
-228
View File
@@ -1,228 +0,0 @@
#!/usr/bin/env bash
#
# claude-mem Setup Hook
# Ensures dependencies are installed before plugin runs
#
set -euo pipefail
# Use CLAUDE_PLUGIN_ROOT if available, otherwise detect from script location
if [[ -z "${CLAUDE_PLUGIN_ROOT:-}" ]]; then
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(dirname "$SCRIPT_DIR")"
else
ROOT="$CLAUDE_PLUGIN_ROOT"
fi
MARKER="$ROOT/.install-version"
PKG_JSON="$ROOT/package.json"
# Colors (when terminal supports it)
if [[ -t 2 ]]; then
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
else
RED='' GREEN='' YELLOW='' BLUE='' NC=''
fi
log_info() { echo -e "${BLUE}ℹ${NC} $*" >&2; }
log_ok() { echo -e "${GREEN}✓${NC} $*" >&2; }
log_warn() { echo -e "${YELLOW}⚠${NC} $*" >&2; }
log_error() { echo -e "${RED}✗${NC} $*" >&2; }
#
# Detect Bun - check PATH and common locations
#
find_bun() {
# Try PATH first
if command -v bun &>/dev/null; then
echo "bun"
return 0
fi
# Check common install locations
local paths=(
"$HOME/.bun/bin/bun"
"/usr/local/bin/bun"
"/opt/homebrew/bin/bun"
)
for p in "${paths[@]}"; do
if [[ -x "$p" ]]; then
echo "$p"
return 0
fi
done
return 1
}
#
# Detect uv - check PATH and common locations
#
find_uv() {
# Try PATH first
if command -v uv &>/dev/null; then
echo "uv"
return 0
fi
# Check common install locations
local paths=(
"$HOME/.local/bin/uv"
"$HOME/.cargo/bin/uv"
"/usr/local/bin/uv"
"/opt/homebrew/bin/uv"
)
for p in "${paths[@]}"; do
if [[ -x "$p" ]]; then
echo "$p"
return 0
fi
done
return 1
}
#
# Get package.json version
#
get_pkg_version() {
if [[ -f "$PKG_JSON" ]]; then
# Simple grep-based extraction (no jq dependency)
grep -o '"version"[[:space:]]*:[[:space:]]*"[^"]*"' "$PKG_JSON" | head -1 | sed 's/.*"\([^"]*\)"$/\1/'
fi
}
#
# Get marker version (if exists)
#
get_marker_version() {
if [[ -f "$MARKER" ]]; then
grep -o '"version"[[:space:]]*:[[:space:]]*"[^"]*"' "$MARKER" | head -1 | sed 's/.*"\([^"]*\)"$/\1/'
fi
}
#
# Get marker's recorded bun version
#
get_marker_bun() {
if [[ -f "$MARKER" ]]; then
grep -o '"bun"[[:space:]]*:[[:space:]]*"[^"]*"' "$MARKER" | head -1 | sed 's/.*"\([^"]*\)"$/\1/'
fi
}
#
# Check if install is needed
#
needs_install() {
# No node_modules? Definitely need install
if [[ ! -d "$ROOT/node_modules" ]]; then
return 0
fi
# No marker? Need install
if [[ ! -f "$MARKER" ]]; then
return 0
fi
local pkg_ver marker_ver bun_ver marker_bun
pkg_ver=$(get_pkg_version)
marker_ver=$(get_marker_version)
# Version mismatch? Need install
if [[ "$pkg_ver" != "$marker_ver" ]]; then
return 0
fi
# Bun version changed? Need install
if BUN_PATH=$(find_bun); then
bun_ver=$("$BUN_PATH" --version 2>/dev/null || echo "")
marker_bun=$(get_marker_bun)
if [[ -n "$bun_ver" && "$bun_ver" != "$marker_bun" ]]; then
return 0
fi
fi
# All good, no install needed
return 1
}
#
# Write version marker after successful install
#
write_marker() {
local bun_ver uv_ver pkg_ver
pkg_ver=$(get_pkg_version)
bun_ver=$("$BUN_PATH" --version 2>/dev/null || echo "unknown")
if UV_PATH=$(find_uv); then
uv_ver=$("$UV_PATH" --version 2>/dev/null | head -1 || echo "unknown")
else
uv_ver="not-installed"
fi
cat > "$MARKER" <<EOF
{
"version": "$pkg_ver",
"bun": "$bun_ver",
"uv": "$uv_ver",
"installedAt": "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
}
EOF
}
#
# Main
#
# 1. Check for Bun
BUN_PATH=$(find_bun) || true
if [[ -z "$BUN_PATH" ]]; then
log_error "Bun runtime not found!"
echo "" >&2
echo "claude-mem requires Bun to run. Please install it:" >&2
echo "" >&2
echo " curl -fsSL https://bun.sh/install | bash" >&2
echo "" >&2
echo "Or on macOS with Homebrew:" >&2
echo "" >&2
echo " brew install oven-sh/bun/bun" >&2
echo "" >&2
echo "Then restart your terminal and try again." >&2
exit 1
fi
BUN_VERSION=$("$BUN_PATH" --version 2>/dev/null || echo "unknown")
log_ok "Bun $BUN_VERSION found at $BUN_PATH"
# 2. Check for uv (optional - for Python/Chroma support)
UV_PATH=$(find_uv) || true
if [[ -z "$UV_PATH" ]]; then
log_warn "uv not found (optional - needed for Python/Chroma vector search)"
echo " To install: curl -LsSf https://astral.sh/uv/install.sh | sh" >&2
else
UV_VERSION=$("$UV_PATH" --version 2>/dev/null | head -1 || echo "unknown")
log_ok "uv $UV_VERSION found"
fi
# 3. Install dependencies if needed
if needs_install; then
log_info "Installing dependencies with Bun..."
if ! "$BUN_PATH" install --cwd "$ROOT"; then
log_error "Failed to install dependencies"
exit 1
fi
write_marker
log_ok "Dependencies installed ($(get_pkg_version))"
else
log_ok "Dependencies up to date ($(get_marker_version))"
fi
exit 0
plugin/scripts/worker-service.cjs
+248 -255
View File
File diff suppressed because one or more lines are too long
plugin/skills/smart-explore/SKILL.md
+141
View File
@@ -0,0 +1,141 @@
---
name: smart-explore
description: Token-optimized structural code search using tree-sitter AST parsing. Use instead of reading full files when you need to understand code structure, find functions, or explore a codebase efficiently.
---
# Smart Explore
Structural code exploration using AST parsing. **This skill overrides your default exploration behavior.** While this skill is active, use smart_search/smart_outline/smart_unfold as your primary tools instead of Read, Grep, and Glob.
## Your Next Tool Call
This skill only loads instructions. You must call the MCP tools yourself. Your next action should be one of:
```
smart_search(query="<topic>", path="./src") -- discover files + symbols across a directory
smart_outline(file_path="<file>") -- structural skeleton of one file
smart_unfold(file_path="<file>", symbol_name="<name>") -- full source of one symbol
```
Do NOT run Grep, Glob, Read, or find to discover files first. `smart_search` walks directories, parses all code files, and returns ranked symbols in one call. It replaces the Glob → Grep → Read discovery cycle.
## 3-Layer Workflow
### Step 1: Search -- Discover Files and Symbols
```
smart_search(query="shutdown", path="./src", max_results=15)
```
**Returns:** Ranked symbols with signatures, line numbers, match reasons, plus folded file views (~2-6k tokens)
```
-- Matching Symbols --
function performGracefulShutdown (services/infrastructure/GracefulShutdown.ts:56)
function httpShutdown (services/infrastructure/HealthMonitor.ts:92)
method WorkerService.shutdown (services/worker-service.ts:846)
-- Folded File Views --
services/infrastructure/GracefulShutdown.ts (7 symbols)
services/worker-service.ts (12 symbols)
```
This is your discovery tool. It finds relevant files AND shows their structure. No Glob/find pre-scan needed.
**Parameters:**
- `query` (string, required) -- What to search for (function name, concept, class name)
- `path` (string) -- Root directory to search (defaults to cwd)
- `max_results` (number) -- Max matching symbols, default 20, max 50
- `file_pattern` (string, optional) -- Filter to specific files/paths
### Step 2: Outline -- Get File Structure
```
smart_outline(file_path="services/worker-service.ts")
```
**Returns:** Complete structural skeleton -- all functions, classes, methods, properties, imports (~1-2k tokens per file)
**Skip this step** when Step 1's folded file views already provide enough structure. Most useful for files not covered by the search results.
**Parameters:**
- `file_path` (string, required) -- Path to the file
### Step 3: Unfold -- See Implementation
Review symbols from Steps 1-2. Pick the ones you need. Unfold only those:
```
smart_unfold(file_path="services/worker-service.ts", symbol_name="shutdown")
```
**Returns:** Full source code of the specified symbol including JSDoc, decorators, and complete implementation (~1-7k tokens depending on symbol size)
**Parameters:**
- `file_path` (string, required) -- Path to the file (as returned by search/outline)
- `symbol_name` (string, required) -- Name of the function/class/method to expand
## When to Use Standard Tools Instead
Use these only when smart_* tools are the wrong fit:
- **Grep:** Exact string/regex search ("find all TODO comments", "where is `ensureWorkerStarted` defined?")
- **Read:** Small files under ~100 lines, non-code files (JSON, markdown, config)
- **Glob:** File path patterns ("find all test files")
For code files over ~100 lines, prefer smart_outline + smart_unfold over Read.
## Workflow Examples
**Discover how a feature works (cross-cutting):**
```
1. smart_search(query="shutdown", path="./src")
-> 14 symbols across 7 files, full picture in one call
2. smart_unfold(file_path="services/infrastructure/GracefulShutdown.ts", symbol_name="performGracefulShutdown")
-> See the core implementation
```
**Navigate a large file:**
```
1. smart_outline(file_path="services/worker-service.ts")
-> 1,466 tokens: 12 functions, WorkerService class with 24 members
2. smart_unfold(file_path="services/worker-service.ts", symbol_name="startSessionProcessor")
-> 1,610 tokens: the specific method you need
Total: ~3,076 tokens vs ~12,000 to Read the full file
```
**Write documentation about code (hybrid workflow):**
```
1. smart_search(query="feature name", path="./src") -- discover all relevant files and symbols
2. smart_outline on key files -- understand structure
3. smart_unfold on important functions -- get implementation details
4. Read on small config/markdown/plan files -- get non-code context
```
Use smart_* tools for code exploration, Read for non-code files. Mix freely.
**Exploration then precision:**
```
1. smart_search(query="session", path="./src", max_results=10)
-> 10 ranked symbols: SessionMetadata, SessionQueueProcessor, SessionSummary...
2. Pick the relevant one, unfold it
```
## Token Economics
| Approach | Tokens | Use Case |
|----------|--------|----------|
| smart_outline | ~1,500 | "What's in this file?" |
| smart_unfold | ~1,600 | "Show me this function" |
| smart_search | ~2,000-6,000 | "How does X work?" |
| Read (full file) | ~12,000+ | When you truly need everything |
| Explore agent | ~20,000-40,000 | Same as smart_search, 6-12x more expensive |
**8x savings** on file understanding (outline + unfold vs Read). **6-12x savings** on exploration vs Explore agent.