refactor: Clean up search architecture, remove experimental contextualize endpoint (#133)
* Refactor code structure for improved readability and maintainability * Add test results for search API and related functionalities - Created test result files for various search-related functionalities, including: - test-11-search-server-changes.json - test-12-context-hook-changes.json - test-13-worker-service-changes.json - test-14-patterns.json - test-15-gotchas.json - test-16-discoveries.json - test-17-all-bugfixes.json - test-18-all-features.json - test-19-all-decisions.json - test-20-session-search.json - test-21-prompt-search.json - test-22-decisions-endpoint.json - test-23-changes-endpoint.json - test-24-how-it-works-endpoint.json - test-25-contextualize-endpoint.json - test-26-timeline-around-observation.json - test-27-multi-param-combo.json - test-28-file-type-combo.json - Each test result file captures specific search failures or outcomes, including issues with undefined properties and successful execution of search queries. - Enhanced documentation of search architecture and testing strategies, ensuring compliance with established guidelines and improving overall search functionality. * feat: Enhance unified search API with catch-all parameters and backward compatibility - Implemented a unified search API at /api/search that accepts catch-all parameters for filtering by type, observation type, concepts, and files. - Maintained backward compatibility by keeping granular endpoints functional while routing through the same infrastructure. - Completed comprehensive testing of search capabilities with real-world query scenarios. fix: Address missing debug output in search API query tests - Flushed PM2 logs and executed search queries to verify functionality. - Diagnosed absence of "Raw Chroma" debug messages in worker logs, indicating potential issues with logging or query processing. refactor: Improve build and deployment pipeline for claude-mem plugin - Successfully built and synced all hooks and services to the marketplace directory. - Ensured all dependencies are installed and up-to-date in the deployment location. feat: Implement hybrid search filters with 90-day recency window - Enhanced search server to apply a 90-day recency filter to Chroma results before categorizing by document type. fix: Correct parameter handling in searchUserPrompts method - Added support for filter-only queries and improved dual-path logic for clarity. refactor: Rename FTS5 method to clarify fallback status - Renamed escapeFTS5 to escapeFTS5_fallback_when_chroma_unavailable to indicate its temporary usage. feat: Introduce contextualize tool for comprehensive project overview - Added a new tool to fetch recent observations, sessions, and user prompts, providing a quick project overview. feat: Add semantic shortcut tools for common search patterns - Implemented 'decisions', 'changes', and 'how_it_works' tools for convenient access to frequently searched observation categories. feat: Unified timeline tool supports anchor and query modes - Combined get_context_timeline and get_timeline_by_query into a single interface for timeline exploration. feat: Unified search tool added to MCP server - New tool queries all memory types simultaneously, providing combined chronological results for improved search efficiency. * Refactor search functionality to clarify FTS5 fallback usage - Updated `worker-service.cjs` to replace FTS5 fallback function with a more descriptive name and improved error handling. - Enhanced documentation in `SKILL.md` to specify the unified API endpoint and clarify the behavior of the search engine, including the conditions under which FTS5 is used. - Modified `search-server.ts` to provide clearer logging and descriptions regarding the fallback to FTS5 when UVX/Python is unavailable. - Renamed and updated the `SessionSearch.ts` methods to reflect the conditions for using FTS5, emphasizing the lack of semantic understanding in fallback scenarios. * feat: Add ID-based fetch endpoints and simplify mem-search skill **Problem:** - Search returns IDs but no way to fetch by ID - Skill documentation was bloated with too many options - Claude wasn't using IDs because we didn't tell it how **Solution:** 1. Added three new HTTP endpoints: - GET /api/observation/:id - GET /api/session/:id - GET /api/prompt/:id 2. Completely rewrote SKILL.md: - Stripped complexity down to essentials - Clear 3-step prescriptive workflow: Search → Review IDs → Fetch by ID - Emphasized ID usage: "The IDs are there for a reason - USE THEM" - Removed confusing multi-endpoint documentation - Kept only unified search with filters **Impact:** - Token efficiency: Claude can now fetch full details only for relevant IDs - Clarity: One clear workflow instead of 10+ options to choose from - Usability: IDs are no longer wasted context - they're actionable 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * chore: Move internal docs to private directory Moved POSTMORTEM and planning docs to ./private to exclude from PR reviews. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Remove experimental contextualize endpoint - Removed contextualize MCP tool from search-server (saves ~4KB) - Disabled FTS5 fallback paths in SessionSearch (now vector-first) - Cleaned up CLAUDE.md documentation - Removed contextualize-rewrite-plan.md doc Rationale: - Contextualize is better suited as a skill (LLM-powered) than an endpoint - Search API already provides vector search with configurable limits - Created issue #132 to track future contextualize skill implementation Changes: - src/servers/search-server.ts: Removed contextualize tool definition - src/services/sqlite/SessionSearch.ts: Disabled FTS5 fallback, added deprecation warnings - CLAUDE.md: Cleaned up outdated skill documentation - docs/: Removed contextualize plan document 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Complete FTS5 cleanup - remove all deprecated search code This completes the FTS5 cleanup work by removing all commented-out FTS5 search code while preserving database tables for backward compatibility. Changes: - Removed 200+ lines of commented FTS5 search code from SessionSearch.ts - Removed deprecated degraded_search_query__when_uvx_unavailable method - Updated all method documentation to clarify vector-first architecture - Updated class documentation to reflect filter-only query support - Updated CLAUDE.md to remove FTS5 search references - Clarified that FTS5 tables exist for backward compatibility only - Updated "Why SQLite FTS5" section to "Why Vector-First Search" Database impact: NONE - FTS5 tables remain intact for existing installations Search architecture: - ChromaDB: All text-based vector search queries - SQLite: Filter-only queries (date ranges, metadata, no query text) - FTS5 tables: Maintained but unused (backward compatibility) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Remove all FTS5 fallback execution code from search-server Completes the FTS5 cleanup by removing all fallback execution paths that attempted to use FTS5 when ChromaDB was unavailable. Changes: - Removed all FTS5 fallback code execution paths - When ChromaDB fails or is unavailable, return empty results with helpful error messages - Updated all deprecated tool descriptions (search_observations, search_sessions, search_user_prompts) - Changed error messages to indicate FTS5 fallback has been removed - Added installation instructions for UVX/Python when vector search is unavailable - Updated comments from "hybrid search" to "vector-first search" - Removed ~100 lines of dead FTS5 fallback code Database impact: NONE - FTS5 tables remain intact (backward compatibility) Search behavior when ChromaDB unavailable: - Text queries: Return empty results with error explaining ChromaDB is required - Filter-only queries (no text): Continue to work via direct SQLite 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Address PR 133 review feedback Critical fixes: - Remove contextualize endpoint from worker-service (route + handler) - Fix build script logging to show correct .cjs extension (was .mjs) Documentation improvements: - Add comprehensive FTS5 retention rationale documentation - Include v7.0.0 removal TODO for future cleanup Testing: - Build succeeds with correct output logging - Worker restarts successfully (30th restart) - Contextualize endpoint properly removed (404 response) - Search endpoint verified working This addresses all critical review feedback from PR 133. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
Alex Newman
@@ -155,6 +155,12 @@ export class WorkerService {
|
||||
this.app.get('/api/observations', this.handleGetObservations.bind(this));
|
||||
this.app.get('/api/summaries', this.handleGetSummaries.bind(this));
|
||||
this.app.get('/api/prompts', this.handleGetPrompts.bind(this));
|
||||
|
||||
// Fetch by ID
|
||||
this.app.get('/api/observation/:id', this.handleGetObservationById.bind(this));
|
||||
this.app.get('/api/session/:id', this.handleGetSessionById.bind(this));
|
||||
this.app.get('/api/prompt/:id', this.handleGetPromptById.bind(this));
|
||||
|
||||
this.app.get('/api/stats', this.handleGetStats.bind(this));
|
||||
this.app.get('/api/processing-status', this.handleGetProcessingStatus.bind(this));
|
||||
this.app.post('/api/processing', this.handleSetProcessing.bind(this));
|
||||
@@ -168,6 +174,14 @@ export class WorkerService {
|
||||
this.app.post('/api/mcp/toggle', this.handleToggleMcp.bind(this));
|
||||
|
||||
// Search API endpoints (for skill-based search)
|
||||
// Unified endpoints (new consolidated API)
|
||||
this.app.get('/api/search', this.handleUnifiedSearch.bind(this));
|
||||
this.app.get('/api/timeline', this.handleUnifiedTimeline.bind(this));
|
||||
this.app.get('/api/decisions', this.handleDecisions.bind(this));
|
||||
this.app.get('/api/changes', this.handleChanges.bind(this));
|
||||
this.app.get('/api/how-it-works', this.handleHowItWorks.bind(this));
|
||||
|
||||
// Backward compatibility endpoints (use /api/search with type param instead)
|
||||
this.app.get('/api/search/observations', this.handleSearchObservations.bind(this));
|
||||
this.app.get('/api/search/sessions', this.handleSearchSessions.bind(this));
|
||||
this.app.get('/api/search/prompts', this.handleSearchPrompts.bind(this));
|
||||
@@ -223,7 +237,7 @@ export class WorkerService {
|
||||
await this.dbManager.initialize();
|
||||
|
||||
// Connect to MCP search server
|
||||
const searchServerPath = path.join(__dirname, '..', '..', 'plugin', 'scripts', 'search-server.mjs');
|
||||
const searchServerPath = path.join(__dirname, '..', '..', 'plugin', 'scripts', 'search-server.cjs');
|
||||
const transport = new StdioClientTransport({
|
||||
command: 'node',
|
||||
args: [searchServerPath],
|
||||
@@ -659,6 +673,87 @@ export class WorkerService {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Get observation by ID
|
||||
* GET /api/observation/:id
|
||||
*/
|
||||
private handleGetObservationById(req: Request, res: Response): void {
|
||||
try {
|
||||
const id = parseInt(req.params.id, 10);
|
||||
if (isNaN(id)) {
|
||||
res.status(400).json({ error: 'Invalid observation ID' });
|
||||
return;
|
||||
}
|
||||
|
||||
const store = this.dbManager.getSessionStore();
|
||||
const observation = store.getObservationById(id);
|
||||
|
||||
if (!observation) {
|
||||
res.status(404).json({ error: `Observation #${id} not found` });
|
||||
return;
|
||||
}
|
||||
|
||||
res.json(observation);
|
||||
} catch (error) {
|
||||
logger.failure('WORKER', 'Get observation by ID failed', {}, error as Error);
|
||||
res.status(500).json({ error: (error as Error).message });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Get session by ID
|
||||
* GET /api/session/:id
|
||||
*/
|
||||
private handleGetSessionById(req: Request, res: Response): void {
|
||||
try {
|
||||
const id = parseInt(req.params.id, 10);
|
||||
if (isNaN(id)) {
|
||||
res.status(400).json({ error: 'Invalid session ID' });
|
||||
return;
|
||||
}
|
||||
|
||||
const store = this.dbManager.getSessionStore();
|
||||
const sessions = store.getSessionSummariesByIds([id]);
|
||||
|
||||
if (sessions.length === 0) {
|
||||
res.status(404).json({ error: `Session #${id} not found` });
|
||||
return;
|
||||
}
|
||||
|
||||
res.json(sessions[0]);
|
||||
} catch (error) {
|
||||
logger.failure('WORKER', 'Get session by ID failed', {}, error as Error);
|
||||
res.status(500).json({ error: (error as Error).message });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Get user prompt by ID
|
||||
* GET /api/prompt/:id
|
||||
*/
|
||||
private handleGetPromptById(req: Request, res: Response): void {
|
||||
try {
|
||||
const id = parseInt(req.params.id, 10);
|
||||
if (isNaN(id)) {
|
||||
res.status(400).json({ error: 'Invalid prompt ID' });
|
||||
return;
|
||||
}
|
||||
|
||||
const store = this.dbManager.getSessionStore();
|
||||
const prompts = store.getUserPromptsByIds([id]);
|
||||
|
||||
if (prompts.length === 0) {
|
||||
res.status(404).json({ error: `Prompt #${id} not found` });
|
||||
return;
|
||||
}
|
||||
|
||||
res.json(prompts[0]);
|
||||
} catch (error) {
|
||||
logger.failure('WORKER', 'Get prompt by ID failed', {}, error as Error);
|
||||
res.status(500).json({ error: (error as Error).message });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Get database statistics (with worker metadata)
|
||||
*/
|
||||
@@ -944,8 +1039,102 @@ export class WorkerService {
|
||||
// Search API Handlers (for skill-based search)
|
||||
// ============================================================================
|
||||
|
||||
// ============================================================================
|
||||
// Unified Search API Handlers (New Consolidated API)
|
||||
// ============================================================================
|
||||
|
||||
/**
|
||||
* Search observations
|
||||
* Unified search across all memory types (observations, sessions, prompts)
|
||||
* GET /api/search?query=...&format=index&limit=20
|
||||
*/
|
||||
private async handleUnifiedSearch(req: Request, res: Response): Promise<void> {
|
||||
try {
|
||||
const result = await this.mcpClient.callTool({
|
||||
name: 'search',
|
||||
arguments: req.query
|
||||
});
|
||||
res.json(result.content);
|
||||
} catch (error) {
|
||||
logger.failure('WORKER', 'Unified search failed', {}, error as Error);
|
||||
res.status(500).json({ error: (error as Error).message });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Unified timeline (anchor or query-based)
|
||||
* GET /api/timeline?anchor=123 OR GET /api/timeline?query=...
|
||||
*/
|
||||
private async handleUnifiedTimeline(req: Request, res: Response): Promise<void> {
|
||||
try {
|
||||
const result = await this.mcpClient.callTool({
|
||||
name: 'timeline',
|
||||
arguments: req.query
|
||||
});
|
||||
res.json(result.content);
|
||||
} catch (error) {
|
||||
logger.failure('WORKER', 'Unified timeline failed', {}, error as Error);
|
||||
res.status(500).json({ error: (error as Error).message });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Semantic shortcut for finding decision observations
|
||||
* GET /api/decisions?format=index&limit=20
|
||||
*/
|
||||
private async handleDecisions(req: Request, res: Response): Promise<void> {
|
||||
try {
|
||||
const result = await this.mcpClient.callTool({
|
||||
name: 'decisions',
|
||||
arguments: req.query
|
||||
});
|
||||
res.json(result.content);
|
||||
} catch (error) {
|
||||
logger.failure('WORKER', 'Decisions search failed', {}, error as Error);
|
||||
res.status(500).json({ error: (error as Error).message });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Semantic shortcut for finding change-related observations
|
||||
* GET /api/changes?format=index&limit=20
|
||||
*/
|
||||
private async handleChanges(req: Request, res: Response): Promise<void> {
|
||||
try {
|
||||
const result = await this.mcpClient.callTool({
|
||||
name: 'changes',
|
||||
arguments: req.query
|
||||
});
|
||||
res.json(result.content);
|
||||
} catch (error) {
|
||||
logger.failure('WORKER', 'Changes search failed', {}, error as Error);
|
||||
res.status(500).json({ error: (error as Error).message });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Semantic shortcut for finding "how it works" explanations
|
||||
* GET /api/how-it-works?format=index&limit=20
|
||||
*/
|
||||
private async handleHowItWorks(req: Request, res: Response): Promise<void> {
|
||||
try {
|
||||
const result = await this.mcpClient.callTool({
|
||||
name: 'how_it_works',
|
||||
arguments: req.query
|
||||
});
|
||||
res.json(result.content);
|
||||
} catch (error) {
|
||||
logger.failure('WORKER', 'How it works search failed', {}, error as Error);
|
||||
res.status(500).json({ error: (error as Error).message });
|
||||
}
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Backward Compatibility API Handlers
|
||||
// All functionality available via /api/search with type/obs_type/concepts/files params
|
||||
// ============================================================================
|
||||
|
||||
/**
|
||||
* Search observations (use /api/search?type=observations instead)
|
||||
* GET /api/search/observations?query=...&format=index&limit=20&project=...
|
||||
*/
|
||||
private async handleSearchObservations(req: Request, res: Response): Promise<void> {
|
||||
|
||||
Reference in New Issue
Block a user