Alex Newman
c5e68a17c8
* Refactor code structure for improved readability and maintainability * Add test results for search API and related functionalities - Created test result files for various search-related functionalities, including: - test-11-search-server-changes.json - test-12-context-hook-changes.json - test-13-worker-service-changes.json - test-14-patterns.json - test-15-gotchas.json - test-16-discoveries.json - test-17-all-bugfixes.json - test-18-all-features.json - test-19-all-decisions.json - test-20-session-search.json - test-21-prompt-search.json - test-22-decisions-endpoint.json - test-23-changes-endpoint.json - test-24-how-it-works-endpoint.json - test-25-contextualize-endpoint.json - test-26-timeline-around-observation.json - test-27-multi-param-combo.json - test-28-file-type-combo.json - Each test result file captures specific search failures or outcomes, including issues with undefined properties and successful execution of search queries. - Enhanced documentation of search architecture and testing strategies, ensuring compliance with established guidelines and improving overall search functionality. * feat: Enhance unified search API with catch-all parameters and backward compatibility - Implemented a unified search API at /api/search that accepts catch-all parameters for filtering by type, observation type, concepts, and files. - Maintained backward compatibility by keeping granular endpoints functional while routing through the same infrastructure. - Completed comprehensive testing of search capabilities with real-world query scenarios. fix: Address missing debug output in search API query tests - Flushed PM2 logs and executed search queries to verify functionality. - Diagnosed absence of "Raw Chroma" debug messages in worker logs, indicating potential issues with logging or query processing. refactor: Improve build and deployment pipeline for claude-mem plugin - Successfully built and synced all hooks and services to the marketplace directory. - Ensured all dependencies are installed and up-to-date in the deployment location. feat: Implement hybrid search filters with 90-day recency window - Enhanced search server to apply a 90-day recency filter to Chroma results before categorizing by document type. fix: Correct parameter handling in searchUserPrompts method - Added support for filter-only queries and improved dual-path logic for clarity. refactor: Rename FTS5 method to clarify fallback status - Renamed escapeFTS5 to escapeFTS5_fallback_when_chroma_unavailable to indicate its temporary usage. feat: Introduce contextualize tool for comprehensive project overview - Added a new tool to fetch recent observations, sessions, and user prompts, providing a quick project overview. feat: Add semantic shortcut tools for common search patterns - Implemented 'decisions', 'changes', and 'how_it_works' tools for convenient access to frequently searched observation categories. feat: Unified timeline tool supports anchor and query modes - Combined get_context_timeline and get_timeline_by_query into a single interface for timeline exploration. feat: Unified search tool added to MCP server - New tool queries all memory types simultaneously, providing combined chronological results for improved search efficiency. * Refactor search functionality to clarify FTS5 fallback usage - Updated `worker-service.cjs` to replace FTS5 fallback function with a more descriptive name and improved error handling. - Enhanced documentation in `SKILL.md` to specify the unified API endpoint and clarify the behavior of the search engine, including the conditions under which FTS5 is used. - Modified `search-server.ts` to provide clearer logging and descriptions regarding the fallback to FTS5 when UVX/Python is unavailable. - Renamed and updated the `SessionSearch.ts` methods to reflect the conditions for using FTS5, emphasizing the lack of semantic understanding in fallback scenarios. * feat: Add ID-based fetch endpoints and simplify mem-search skill **Problem:** - Search returns IDs but no way to fetch by ID - Skill documentation was bloated with too many options - Claude wasn't using IDs because we didn't tell it how **Solution:** 1. Added three new HTTP endpoints: - GET /api/observation/:id - GET /api/session/:id - GET /api/prompt/:id 2. Completely rewrote SKILL.md: - Stripped complexity down to essentials - Clear 3-step prescriptive workflow: Search → Review IDs → Fetch by ID - Emphasized ID usage: "The IDs are there for a reason - USE THEM" - Removed confusing multi-endpoint documentation - Kept only unified search with filters **Impact:** - Token efficiency: Claude can now fetch full details only for relevant IDs - Clarity: One clear workflow instead of 10+ options to choose from - Usability: IDs are no longer wasted context - they're actionable 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * chore: Move internal docs to private directory Moved POSTMORTEM and planning docs to ./private to exclude from PR reviews. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Remove experimental contextualize endpoint - Removed contextualize MCP tool from search-server (saves ~4KB) - Disabled FTS5 fallback paths in SessionSearch (now vector-first) - Cleaned up CLAUDE.md documentation - Removed contextualize-rewrite-plan.md doc Rationale: - Contextualize is better suited as a skill (LLM-powered) than an endpoint - Search API already provides vector search with configurable limits - Created issue #132 to track future contextualize skill implementation Changes: - src/servers/search-server.ts: Removed contextualize tool definition - src/services/sqlite/SessionSearch.ts: Disabled FTS5 fallback, added deprecation warnings - CLAUDE.md: Cleaned up outdated skill documentation - docs/: Removed contextualize plan document 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Complete FTS5 cleanup - remove all deprecated search code This completes the FTS5 cleanup work by removing all commented-out FTS5 search code while preserving database tables for backward compatibility. Changes: - Removed 200+ lines of commented FTS5 search code from SessionSearch.ts - Removed deprecated degraded_search_query__when_uvx_unavailable method - Updated all method documentation to clarify vector-first architecture - Updated class documentation to reflect filter-only query support - Updated CLAUDE.md to remove FTS5 search references - Clarified that FTS5 tables exist for backward compatibility only - Updated "Why SQLite FTS5" section to "Why Vector-First Search" Database impact: NONE - FTS5 tables remain intact for existing installations Search architecture: - ChromaDB: All text-based vector search queries - SQLite: Filter-only queries (date ranges, metadata, no query text) - FTS5 tables: Maintained but unused (backward compatibility) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Remove all FTS5 fallback execution code from search-server Completes the FTS5 cleanup by removing all fallback execution paths that attempted to use FTS5 when ChromaDB was unavailable. Changes: - Removed all FTS5 fallback code execution paths - When ChromaDB fails or is unavailable, return empty results with helpful error messages - Updated all deprecated tool descriptions (search_observations, search_sessions, search_user_prompts) - Changed error messages to indicate FTS5 fallback has been removed - Added installation instructions for UVX/Python when vector search is unavailable - Updated comments from "hybrid search" to "vector-first search" - Removed ~100 lines of dead FTS5 fallback code Database impact: NONE - FTS5 tables remain intact (backward compatibility) Search behavior when ChromaDB unavailable: - Text queries: Return empty results with error explaining ChromaDB is required - Filter-only queries (no text): Continue to work via direct SQLite 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Address PR 133 review feedback Critical fixes: - Remove contextualize endpoint from worker-service (route + handler) - Fix build script logging to show correct .cjs extension (was .mjs) Documentation improvements: - Add comprehensive FTS5 retention rationale documentation - Include v7.0.0 removal TODO for future cleanup Testing: - Build succeeds with correct output logging - Worker restarts successfully (30th restart) - Contextualize endpoint properly removed (404 response) - Search endpoint verified working This addresses all critical review feedback from PR 133. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
4.2 KiB
4.2 KiB
Unified Search API Consolidation Plan
Overview
Consolidate 10 search endpoints into 6 powerful, semantic endpoints with intelligent aliasing for backward compatibility.
New Endpoint Structure
- /search - Unified Cross-Type Search
- Searches all record types (observations + sessions + prompts) via Chroma multi-collection search
- Optional type filter to narrow down
- Replaces: search_observations, search_sessions, search_user_prompts
- Params: query, response=[index|full], type, project, dateRange, limit, offset, orderBy
- /timeline - Unified Timeline Tool
- Supports both anchor-based and query-based modes via params
- If anchor → direct timeline lookup (like get_context_timeline)
- If query → search-first then timeline (like get_timeline_by_query)
- Params: anchor OR query, depth_before, depth_after, response, mode, project
- /decisions - Decision Observations
- Metadata-first search for type=decision observations
- Uses specialized search logic for precision
- Params: response, project, dateRange, limit, offset, orderBy
- /changes - Change Observations
- Metadata-first search for type=change observations
- Same pattern as /decisions
- /how-it-works - How-It-Works Concept
- Metadata-first search for concept=how-it-works observations
- Same pattern as concept endpoints
- /contextualize - Intelligent Context Builder
- Complex hybrid endpoint: a. Get 7 latest decisions + 7 latest changes + 3 latest how-it-works b. Find newest date across all results c. Get timeline (7 before + 7 after) around that date d. Merge & re-sort into single timeline (newest → oldest) e. Return timeline + narratives of each concept's latest result
- Params: query (for contextualization), project
Implementation Phases
Phase 1: Core Unified Search (search-server.ts)
- Create search tool with Chroma multi-collection query
- Add type filtering support
- Alias old tools: search_observations → search(type=['observations'])
Phase 2: Unified Timeline (search-server.ts)
- Merge get_context_timeline + get_timeline_by_query logic
- Support both anchor and query params (mutually exclusive)
- Alias old timeline tools to new unified implementation
Phase 3: Specialized Concept Endpoints (search-server.ts)
- Create decisions, changes, how_it_works tools
- Use metadata-first search strategy
- Update find_by_type and find_by_concept to call these internally
Phase 4: Contextualize Endpoint (search-server.ts)
- Implement parallel fetching (7 decisions, 7 changes, 3 how-it-works)
- Find newest date, get timeline around it
- Merge, re-sort, extract narratives
- Return structured response with timeline + narratives
Phase 5: HTTP API Routes (worker-service.ts)
- Add 6 new routes: /api/search, /api/timeline, /api/decisions, /api/changes, /api/how-it-works, /api/contextualize
- Update old routes to alias new implementations
- Maintain backward compatibility
Phase 6: Chroma Multi-Collection Search (ChromaSync.ts)
- Add searchAll() method to query all collections in parallel
- Include source collection metadata in results
- Merge and rank by similarity score
Phase 7: SQLite Fallback (SessionSearch.ts)
- Add searchAll() for FTS5 fallback when Chroma unavailable
- Merge results from all three FTS5 tables
Phase 8: Documentation & Skill Updates
- Update mem-search skill with new endpoints
- Update CLAUDE.md and README.md
- Add examples and migration guide
Phase 9: Testing & Deployment
- Unit tests for all new tools
- Integration tests for aliasing
- Manual testing via mem-search skill
- Build → sync → worker restart
Key Design Decisions
✅ Aliasing Strategy: Old endpoints call new implementations internally (zero breaking changes) ✅ Unified Search: Chroma multi-collection search for true cross-type queries ✅ Flexible Timeline: Single tool supports both direct and query-based modes ✅ Specialized Shortcuts: /decisions, /changes, /how-it-works for common queries ✅ Intelligent Context: /contextualize auto-builds rich context with narratives
Migration Impact
- Users: Zero breaking changes, old endpoints work via aliasing
- Codebase: Simplified from 10 conceptual endpoints to 6
- Performance: Improved via Chroma multi-collection search
- Developer UX: Cleaner, more semantic API