Alex Newman
c5e68a17c8
* Refactor code structure for improved readability and maintainability * Add test results for search API and related functionalities - Created test result files for various search-related functionalities, including: - test-11-search-server-changes.json - test-12-context-hook-changes.json - test-13-worker-service-changes.json - test-14-patterns.json - test-15-gotchas.json - test-16-discoveries.json - test-17-all-bugfixes.json - test-18-all-features.json - test-19-all-decisions.json - test-20-session-search.json - test-21-prompt-search.json - test-22-decisions-endpoint.json - test-23-changes-endpoint.json - test-24-how-it-works-endpoint.json - test-25-contextualize-endpoint.json - test-26-timeline-around-observation.json - test-27-multi-param-combo.json - test-28-file-type-combo.json - Each test result file captures specific search failures or outcomes, including issues with undefined properties and successful execution of search queries. - Enhanced documentation of search architecture and testing strategies, ensuring compliance with established guidelines and improving overall search functionality. * feat: Enhance unified search API with catch-all parameters and backward compatibility - Implemented a unified search API at /api/search that accepts catch-all parameters for filtering by type, observation type, concepts, and files. - Maintained backward compatibility by keeping granular endpoints functional while routing through the same infrastructure. - Completed comprehensive testing of search capabilities with real-world query scenarios. fix: Address missing debug output in search API query tests - Flushed PM2 logs and executed search queries to verify functionality. - Diagnosed absence of "Raw Chroma" debug messages in worker logs, indicating potential issues with logging or query processing. refactor: Improve build and deployment pipeline for claude-mem plugin - Successfully built and synced all hooks and services to the marketplace directory. - Ensured all dependencies are installed and up-to-date in the deployment location. feat: Implement hybrid search filters with 90-day recency window - Enhanced search server to apply a 90-day recency filter to Chroma results before categorizing by document type. fix: Correct parameter handling in searchUserPrompts method - Added support for filter-only queries and improved dual-path logic for clarity. refactor: Rename FTS5 method to clarify fallback status - Renamed escapeFTS5 to escapeFTS5_fallback_when_chroma_unavailable to indicate its temporary usage. feat: Introduce contextualize tool for comprehensive project overview - Added a new tool to fetch recent observations, sessions, and user prompts, providing a quick project overview. feat: Add semantic shortcut tools for common search patterns - Implemented 'decisions', 'changes', and 'how_it_works' tools for convenient access to frequently searched observation categories. feat: Unified timeline tool supports anchor and query modes - Combined get_context_timeline and get_timeline_by_query into a single interface for timeline exploration. feat: Unified search tool added to MCP server - New tool queries all memory types simultaneously, providing combined chronological results for improved search efficiency. * Refactor search functionality to clarify FTS5 fallback usage - Updated `worker-service.cjs` to replace FTS5 fallback function with a more descriptive name and improved error handling. - Enhanced documentation in `SKILL.md` to specify the unified API endpoint and clarify the behavior of the search engine, including the conditions under which FTS5 is used. - Modified `search-server.ts` to provide clearer logging and descriptions regarding the fallback to FTS5 when UVX/Python is unavailable. - Renamed and updated the `SessionSearch.ts` methods to reflect the conditions for using FTS5, emphasizing the lack of semantic understanding in fallback scenarios. * feat: Add ID-based fetch endpoints and simplify mem-search skill **Problem:** - Search returns IDs but no way to fetch by ID - Skill documentation was bloated with too many options - Claude wasn't using IDs because we didn't tell it how **Solution:** 1. Added three new HTTP endpoints: - GET /api/observation/:id - GET /api/session/:id - GET /api/prompt/:id 2. Completely rewrote SKILL.md: - Stripped complexity down to essentials - Clear 3-step prescriptive workflow: Search → Review IDs → Fetch by ID - Emphasized ID usage: "The IDs are there for a reason - USE THEM" - Removed confusing multi-endpoint documentation - Kept only unified search with filters **Impact:** - Token efficiency: Claude can now fetch full details only for relevant IDs - Clarity: One clear workflow instead of 10+ options to choose from - Usability: IDs are no longer wasted context - they're actionable 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * chore: Move internal docs to private directory Moved POSTMORTEM and planning docs to ./private to exclude from PR reviews. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Remove experimental contextualize endpoint - Removed contextualize MCP tool from search-server (saves ~4KB) - Disabled FTS5 fallback paths in SessionSearch (now vector-first) - Cleaned up CLAUDE.md documentation - Removed contextualize-rewrite-plan.md doc Rationale: - Contextualize is better suited as a skill (LLM-powered) than an endpoint - Search API already provides vector search with configurable limits - Created issue #132 to track future contextualize skill implementation Changes: - src/servers/search-server.ts: Removed contextualize tool definition - src/services/sqlite/SessionSearch.ts: Disabled FTS5 fallback, added deprecation warnings - CLAUDE.md: Cleaned up outdated skill documentation - docs/: Removed contextualize plan document 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Complete FTS5 cleanup - remove all deprecated search code This completes the FTS5 cleanup work by removing all commented-out FTS5 search code while preserving database tables for backward compatibility. Changes: - Removed 200+ lines of commented FTS5 search code from SessionSearch.ts - Removed deprecated degraded_search_query__when_uvx_unavailable method - Updated all method documentation to clarify vector-first architecture - Updated class documentation to reflect filter-only query support - Updated CLAUDE.md to remove FTS5 search references - Clarified that FTS5 tables exist for backward compatibility only - Updated "Why SQLite FTS5" section to "Why Vector-First Search" Database impact: NONE - FTS5 tables remain intact for existing installations Search architecture: - ChromaDB: All text-based vector search queries - SQLite: Filter-only queries (date ranges, metadata, no query text) - FTS5 tables: Maintained but unused (backward compatibility) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Remove all FTS5 fallback execution code from search-server Completes the FTS5 cleanup by removing all fallback execution paths that attempted to use FTS5 when ChromaDB was unavailable. Changes: - Removed all FTS5 fallback code execution paths - When ChromaDB fails or is unavailable, return empty results with helpful error messages - Updated all deprecated tool descriptions (search_observations, search_sessions, search_user_prompts) - Changed error messages to indicate FTS5 fallback has been removed - Added installation instructions for UVX/Python when vector search is unavailable - Updated comments from "hybrid search" to "vector-first search" - Removed ~100 lines of dead FTS5 fallback code Database impact: NONE - FTS5 tables remain intact (backward compatibility) Search behavior when ChromaDB unavailable: - Text queries: Return empty results with error explaining ChromaDB is required - Filter-only queries (no text): Continue to work via direct SQLite 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Address PR 133 review feedback Critical fixes: - Remove contextualize endpoint from worker-service (route + handler) - Fix build script logging to show correct .cjs extension (was .mjs) Documentation improvements: - Add comprehensive FTS5 retention rationale documentation - Include v7.0.0 removal TODO for future cleanup Testing: - Build succeeds with correct output logging - Worker restarts successfully (30th restart) - Contextualize endpoint properly removed (404 response) - Search endpoint verified working This addresses all critical review feedback from PR 133. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
1 line
10 KiB
JSON
1 line
10 KiB
JSON
[{"type":"text","text":"## Enhanced MCP Search Tool with Document Type and Metadata Filters\n*Source: claude-mem://observation/10631*\n\n**Added type-specific filtering to unified search tool supporting observations, sessions, prompts with metadata filters.**\n\nThe unified search tool in the MCP search server was enhanced with granular filtering capabilities. Four new optional parameters were added to the search tool's input schema: a 'type' parameter to filter by document type (observations/sessions/prompts), an 'obs_type' parameter to filter observations by their observation type, a 'concepts' parameter to filter by concept tags, and a 'files' parameter to filter by file paths. The obs_type, concepts, and files filters are observation-specific and only take effect when the type parameter is set to \"observations\". This enhancement allows clients to perform more targeted searches through the MCP interface while maintaining backward compatibility since all new parameters are optional.\n\n---\nType: feature | Facts: File src/servers/search-server.ts search tool inputSchema extended with new filter parameters; New type parameter filters by document type: observations, sessions, or prompts; New obs_type parameter filters observations by type (decision, bugfix, feature, refactor, discovery, change); New concepts parameter filters observations by concept tags, accepts string or array; New files parameter filters observations by file paths with partial matching, accepts string or array; Type-specific filters (obs_type, concepts, files) only apply when type equals observations | Concepts: what-changed, how-it-works, pattern | Files: src/servers/search-server.ts\n\n---\nDate: 11/17/2025, 9:21:49 PM\n\n---\n\n## Contextualize Tool for Comprehensive Project Overview\n*Source: claude-mem://observation/10597*\n\n**New tool provides one-stop project context with recent activity summaries and timeline visualization.**\n\nA comprehensive 'contextualize' tool was added to provide a quick project overview in a single request. The tool fetches recent observations (default 7), sessions (default 7), and user prompts (default 3), then identifies the newest activity across all types to use as a timeline anchor. The output is structured in sections: header with overview statistics and latest activity timestamp, followed by summaries of recent observations (top 3 with type emoji icons), recent sessions (top 3), and recent user prompts (all, truncated to 60 chars). The timeline section shows the last 3 days of activity around the newest item, displaying up to 5 entries per day with timestamps and type-specific icons. All activity is pulled using configurable limits, and timeline depth defaults to 5 records before/after the newest item. The tool is designed as a starting point for exploration, with footer text suggesting more specialized tools for deeper investigation. This addresses the \"what's been happening\" use case with minimal token overhead.\n\n---\nType: feature | Facts: Added 'contextualize' tool that fetches recent observations (7), sessions (7), and user prompts (3) in configurable quantities; Tool identifies newest activity across all memory types and builds timeline centered around it; Output includes three summary sections showing top 3 items from each category with type icons; Timeline section displays last 3 days of activity with up to 5 items per day in reverse chronological order; All limits configurable via parameters: observations_limit, sessions_limit, prompts_limit, timeline_depth; Defaults to current working directory basename for project filtering; Footer suggests other tools for deeper exploration (search, timeline, decisions, changes, how_it_works) | Concepts: what-changed, how-it-works, pattern | Files: src/servers/search-server.ts\n\n---\nDate: 11/17/2025, 8:36:23 PM\n\n---\n\n## Semantic Shortcut Tools for Common Search Patterns\n*Source: claude-mem://observation/10596*\n\n**Added three convenience tools for frequently accessed observation categories: decisions, changes, and how-it-works.**\n\nThree semantic shortcut tools were added to the MCP search server to provide convenient access to commonly searched observation patterns. The 'decisions' tool is equivalent to find_by_type with type=\"decision\", filtering for observations where important architectural, technical, or process decisions were made. The 'changes' tool provides a broader search by combining results from type=\"change\", concept=\"change\", and concept=\"what-changed\", deduplicating IDs before applying semantic ranking with a \"what changed\" query. The 'how_it_works' tool searches for concept=\"how-it-works\" observations and applies semantic ranking with \"how it works architecture\" to surface the most relevant architectural documentation. All three tools use the metadata-first, semantic-enhanced search pattern where SQLite filters by type/concept first, then ChromaDB ranks results by semantic relevance. They support the same index/full format options as other search tools, defaulting to index format for token efficiency. When ChromaDB is unavailable, they fall back to SQLite-only searches with chronological ordering.\n\n---\nType: feature | Facts: Added 'decisions' tool as shortcut for finding decision-type observations with semantic ranking; Added 'changes' tool that searches type=change OR concept=change OR concept=what-changed with deduplication; Added 'how_it_works' tool for finding how-it-works concept observations with architecture-focused semantic queries; All three tools support index and full output formats with same token-efficiency guidelines as other search tools; Semantic shortcuts use metadata-first filtering followed by ChromaDB semantic ranking for relevance; Changes tool combines results from multiple filters (type and concepts) before semantic ranking; Tools fall back to SQLite-only search when ChromaDB unavailable | Concepts: what-changed, how-it-works, pattern | Files: src/servers/search-server.ts\n\n---\nDate: 11/17/2025, 8:35:32 PM\n\n---\n\n## Unified Timeline Tool Combining Anchor and Query Modes\n*Source: claude-mem://observation/10595*\n\n**New timeline tool supports both anchor-based and query-based timeline exploration in single interface.**\n\nA new unified 'timeline' tool was added to the MCP search server that combines the capabilities of get_context_timeline and get_timeline_by_query into a single interface. The tool accepts either an anchor parameter for anchor-based timeline exploration or a query parameter for search-based timeline discovery, but not both. Input validation ensures exactly one parameter is provided with helpful error messages guiding proper usage. In query mode, the tool performs a hybrid semantic search (ChromaDB with FTS5 fallback) to find the most relevant observation matching the query, then uses that observation as the timeline anchor. In anchor mode, it supports three anchor types: numeric observation IDs, session ID strings (format \"S123\" or \"#S123\"), and ISO timestamp strings. Both modes use the same timeline rendering logic with observations, sessions, and prompts interleaved chronologically, grouped by day, and formatted with tables for observations and standalone entries for sessions/prompts. The header adapts to the mode, showing the query and matched anchor for query-based searches or just the anchor identifier for anchor-based searches.\n\n---\nType: feature | Facts: Added new 'timeline' tool that consolidates get_context_timeline and get_timeline_by_query functionality; Timeline tool accepts either anchor parameter (observation ID, session ID, or ISO timestamp) or query parameter (natural language search); Tool validates that exactly one of anchor or query is provided with clear error messages; Query mode performs hybrid semantic search to find best matching observation and uses it as timeline anchor; Anchor mode supports three anchor types: numeric observation ID, session ID string (S123), or ISO timestamp string; Timeline rendering identical for both modes with chronologically sorted observations, sessions, and prompts grouped by day; Header changes based on mode showing either \"Timeline for query\" or \"Timeline around anchor\" | Concepts: what-changed, how-it-works, pattern | Files: src/servers/search-server.ts\n\n---\nDate: 11/17/2025, 8:34:35 PM\n\n---\n\n## Unified Search Tool Added to MCP Server\n*Source: claude-mem://observation/10594*\n\n**New unified search tool queries all memory types simultaneously with combined chronological results.**\n\nA new unified 'search' tool was added to the MCP search server that queries all memory types (observations, sessions, and user prompts) in a single operation. Unlike the existing type-specific tools (search_observations, search_sessions, search_user_prompts), this unified tool performs a single ChromaDB query without doc_type filtering, then categorizes results based on metadata and hydrates each type from SQLite. All results are combined into a single array with type discriminators, sorted chronologically according to the orderBy parameter, and then limited. The output format distinguishes between types when rendering results, using the appropriate formatter (formatObservationIndex, formatSessionIndex, formatUserPromptIndex) based on the item type. This provides a more efficient search workflow when users want to see all relevant context across different memory types without running multiple separate searches.\n\n---\nType: feature | Facts: Added new 'search' tool that queries observations, sessions, and user prompts in a single operation; Unified search categorizes ChromaDB results by doc_type metadata field (observation, session_summary, user_prompt); Results from all document types combined and sorted chronologically by epoch timestamp; Limit applies across all document types after merging results; Index format header shows breakdown of result counts by type (e.g., \"5 obs, 3 sessions, 2 prompts\"); Unified search uses same hybrid ChromaDB + FTS5 fallback strategy as type-specific search tools | Concepts: what-changed, how-it-works, pattern | Files: src/servers/search-server.ts\n\n---\nDate: 11/17/2025, 8:33:18 PM"}] |