c5e68a17c8
* Refactor code structure for improved readability and maintainability * Add test results for search API and related functionalities - Created test result files for various search-related functionalities, including: - test-11-search-server-changes.json - test-12-context-hook-changes.json - test-13-worker-service-changes.json - test-14-patterns.json - test-15-gotchas.json - test-16-discoveries.json - test-17-all-bugfixes.json - test-18-all-features.json - test-19-all-decisions.json - test-20-session-search.json - test-21-prompt-search.json - test-22-decisions-endpoint.json - test-23-changes-endpoint.json - test-24-how-it-works-endpoint.json - test-25-contextualize-endpoint.json - test-26-timeline-around-observation.json - test-27-multi-param-combo.json - test-28-file-type-combo.json - Each test result file captures specific search failures or outcomes, including issues with undefined properties and successful execution of search queries. - Enhanced documentation of search architecture and testing strategies, ensuring compliance with established guidelines and improving overall search functionality. * feat: Enhance unified search API with catch-all parameters and backward compatibility - Implemented a unified search API at /api/search that accepts catch-all parameters for filtering by type, observation type, concepts, and files. - Maintained backward compatibility by keeping granular endpoints functional while routing through the same infrastructure. - Completed comprehensive testing of search capabilities with real-world query scenarios. fix: Address missing debug output in search API query tests - Flushed PM2 logs and executed search queries to verify functionality. - Diagnosed absence of "Raw Chroma" debug messages in worker logs, indicating potential issues with logging or query processing. refactor: Improve build and deployment pipeline for claude-mem plugin - Successfully built and synced all hooks and services to the marketplace directory. - Ensured all dependencies are installed and up-to-date in the deployment location. feat: Implement hybrid search filters with 90-day recency window - Enhanced search server to apply a 90-day recency filter to Chroma results before categorizing by document type. fix: Correct parameter handling in searchUserPrompts method - Added support for filter-only queries and improved dual-path logic for clarity. refactor: Rename FTS5 method to clarify fallback status - Renamed escapeFTS5 to escapeFTS5_fallback_when_chroma_unavailable to indicate its temporary usage. feat: Introduce contextualize tool for comprehensive project overview - Added a new tool to fetch recent observations, sessions, and user prompts, providing a quick project overview. feat: Add semantic shortcut tools for common search patterns - Implemented 'decisions', 'changes', and 'how_it_works' tools for convenient access to frequently searched observation categories. feat: Unified timeline tool supports anchor and query modes - Combined get_context_timeline and get_timeline_by_query into a single interface for timeline exploration. feat: Unified search tool added to MCP server - New tool queries all memory types simultaneously, providing combined chronological results for improved search efficiency. * Refactor search functionality to clarify FTS5 fallback usage - Updated `worker-service.cjs` to replace FTS5 fallback function with a more descriptive name and improved error handling. - Enhanced documentation in `SKILL.md` to specify the unified API endpoint and clarify the behavior of the search engine, including the conditions under which FTS5 is used. - Modified `search-server.ts` to provide clearer logging and descriptions regarding the fallback to FTS5 when UVX/Python is unavailable. - Renamed and updated the `SessionSearch.ts` methods to reflect the conditions for using FTS5, emphasizing the lack of semantic understanding in fallback scenarios. * feat: Add ID-based fetch endpoints and simplify mem-search skill **Problem:** - Search returns IDs but no way to fetch by ID - Skill documentation was bloated with too many options - Claude wasn't using IDs because we didn't tell it how **Solution:** 1. Added three new HTTP endpoints: - GET /api/observation/:id - GET /api/session/:id - GET /api/prompt/:id 2. Completely rewrote SKILL.md: - Stripped complexity down to essentials - Clear 3-step prescriptive workflow: Search → Review IDs → Fetch by ID - Emphasized ID usage: "The IDs are there for a reason - USE THEM" - Removed confusing multi-endpoint documentation - Kept only unified search with filters **Impact:** - Token efficiency: Claude can now fetch full details only for relevant IDs - Clarity: One clear workflow instead of 10+ options to choose from - Usability: IDs are no longer wasted context - they're actionable 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * chore: Move internal docs to private directory Moved POSTMORTEM and planning docs to ./private to exclude from PR reviews. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Remove experimental contextualize endpoint - Removed contextualize MCP tool from search-server (saves ~4KB) - Disabled FTS5 fallback paths in SessionSearch (now vector-first) - Cleaned up CLAUDE.md documentation - Removed contextualize-rewrite-plan.md doc Rationale: - Contextualize is better suited as a skill (LLM-powered) than an endpoint - Search API already provides vector search with configurable limits - Created issue #132 to track future contextualize skill implementation Changes: - src/servers/search-server.ts: Removed contextualize tool definition - src/services/sqlite/SessionSearch.ts: Disabled FTS5 fallback, added deprecation warnings - CLAUDE.md: Cleaned up outdated skill documentation - docs/: Removed contextualize plan document 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Complete FTS5 cleanup - remove all deprecated search code This completes the FTS5 cleanup work by removing all commented-out FTS5 search code while preserving database tables for backward compatibility. Changes: - Removed 200+ lines of commented FTS5 search code from SessionSearch.ts - Removed deprecated degraded_search_query__when_uvx_unavailable method - Updated all method documentation to clarify vector-first architecture - Updated class documentation to reflect filter-only query support - Updated CLAUDE.md to remove FTS5 search references - Clarified that FTS5 tables exist for backward compatibility only - Updated "Why SQLite FTS5" section to "Why Vector-First Search" Database impact: NONE - FTS5 tables remain intact for existing installations Search architecture: - ChromaDB: All text-based vector search queries - SQLite: Filter-only queries (date ranges, metadata, no query text) - FTS5 tables: Maintained but unused (backward compatibility) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Remove all FTS5 fallback execution code from search-server Completes the FTS5 cleanup by removing all fallback execution paths that attempted to use FTS5 when ChromaDB was unavailable. Changes: - Removed all FTS5 fallback code execution paths - When ChromaDB fails or is unavailable, return empty results with helpful error messages - Updated all deprecated tool descriptions (search_observations, search_sessions, search_user_prompts) - Changed error messages to indicate FTS5 fallback has been removed - Added installation instructions for UVX/Python when vector search is unavailable - Updated comments from "hybrid search" to "vector-first search" - Removed ~100 lines of dead FTS5 fallback code Database impact: NONE - FTS5 tables remain intact (backward compatibility) Search behavior when ChromaDB unavailable: - Text queries: Return empty results with error explaining ChromaDB is required - Filter-only queries (no text): Continue to work via direct SQLite 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Address PR 133 review feedback Critical fixes: - Remove contextualize endpoint from worker-service (route + handler) - Fix build script logging to show correct .cjs extension (was .mjs) Documentation improvements: - Add comprehensive FTS5 retention rationale documentation - Include v7.0.0 removal TODO for future cleanup Testing: - Build succeeds with correct output logging - Worker restarts successfully (30th restart) - Contextualize endpoint properly removed (404 response) - Search endpoint verified working This addresses all critical review feedback from PR 133. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
1 line
8.1 KiB
JSON
1 line
8.1 KiB
JSON
[{"type":"text","text":"## Apply semantic naming principle to clarify FTS5 as degraded fallback\n*Source: claude-mem://observation/10742*\n\n**Rename FTS5 functions to explicitly communicate fallback status per CLAUDE.md semantic naming guidelines.**\n\nThe fix requires applying the semantic naming principle from CLAUDE.md to all FTS5-related code. Instead of ambiguous names like `escapeFTS5()` that hide architectural intent, functions should use verbose descriptive names like `escapeFTS5_fallback_for_missing_chroma()` that explicitly communicate FTS5's role as a degraded fallback mode for users without Python/uvx/Chroma dependencies. This naming strategy extends to all search functions - they should reveal whether they use Chroma semantic search, FTS5 keyword fallback, or direct SQLite filtering. The semantic naming approach ensures that anyone reading the code immediately understands FTS5 is not a primary feature but exists only as a backup when the preferred Chroma system is unavailable. This prevents future developers from treating FTS5 as an equal alternative to Chroma.\n\n---\nType: decision | Facts: CLAUDE.md semantic naming principle requires verbose names that reveal intent; Current name `escapeFTS5()` hides that FTS5 is fallback for missing Chroma; Proposed naming like `escapeFTS5_fallback_for_missing_chroma()` makes degraded mode explicit; All FTS5-related functions need renaming to state \"degraded fallback for users without Python\"; Search function names should reveal which mechanism they use: Chroma, FTS5, or direct filtering; Semantic naming prevents future confusion about FTS5 architectural role | Concepts: pattern, why-it-exists, how-it-works, trade-off | Files: CLAUDE.md\n\n---\nDate: 11/17/2025, 11:47:00 PM\n\n---\n\n## Semantic Naming Strategy for FTS5 Functions Due to UVX Dependency\n*Source: claude-mem://observation/10741*\n\n**FTS5-related functions, variables renamed to indicate temporary usage pending UVX availability for semantic search.**\n\nThe team has decided to implement a semantic naming convention for all FTS5-related code to clearly communicate that FTS5 is a temporary workaround. The naming pattern \"fts5_only_because_uvx_is_disabled\" will be applied to functions and variables that currently implement full-text search using FTS5. This decision stems from the understanding that UVX (which would enable proper semantic search) is currently disabled, forcing reliance on SQLite's FTS5 for text search capabilities. By embedding this context directly in function names, the codebase self-documents the temporary nature of the implementation and the intended migration path. This naming strategy was recently documented in CLAUDE.md and represents a deliberate choice to maintain clarity about technical debt and future refactoring needs despite the increased verbosity.\n\n---\nType: decision | Facts: FTS5 is being used as a temporary solution because UVX is currently disabled; Functions related to FTS5 will be renamed with semantic suffix \"_only_because_uvx_is_disabled\"; The naming convention makes the temporary nature of FTS5 usage explicit in the codebase; This decision was recently added to CLAUDE.md documentation; The renaming applies to all FTS5 functions and related variables throughout the codebase | Concepts: why-it-exists, pattern, trade-off, what-changed\n\n---\nDate: 11/17/2025, 11:46:49 PM\n\n---\n\n## Unified search handler fix requires five-step query routing logic\n*Source: claude-mem://observation/10739*\n\n**Route by query presence, error-based FTS5 fallback only, remove zero-results fallback check.**\n\nThe fix for the unified search handler requires restructuring the query routing logic into five distinct steps. First, check whether the query parameter exists at the top of the handler. If query is undefined, bypass Chroma entirely and invoke a new filter-only path in SessionSearch that performs direct SQLite filtering. If query exists, proceed with Chroma semantic search. The FTS5 fallback logic must change fundamentally - it should only trigger when Chroma throws an error (caught at line 465), not when Chroma successfully returns zero results. When Chroma returns zero results, that is the correct answer since FTS5 contains identical data and would also return zero. This requires removing the zero-results check from line 472's fallback condition, fixing the current incorrect fallback behavior.\n\n---\nType: decision | Facts: Step 1: Check if query parameter exists at beginning of unified search handler; Step 2: If query is undefined, skip Chroma and call new filter-only path in SessionSearch; Step 3: If query exists, attempt Chroma semantic search; Step 4: If Chroma throws error (line 465 catch block), fall back to FTS5; Step 5: If Chroma succeeds with zero results, accept zero as correct answer without FTS5 fallback; Line 472 condition must be modified to remove zero-results check | Concepts: problem-solution, how-it-works, pattern, trade-off | Files: src/servers/search-server.ts\n\n---\nDate: 11/17/2025, 11:42:18 PM\n\n---\n\n## Query Routing Architecture: ChromaDB, SQLite, and FTS5 Fallback Strategy\n*Source: claude-mem://observation/10736*\n\n**Defined three distinct query paths based on query presence and ChromaDB availability to prevent incorrect fallback logic.**\n\nThe proper query routing architecture requires three distinct paths based on query presence and ChromaDB availability. Path 1: When a semantic query exists and ChromaDB is available, route to queryChroma with both query and filters. If ChromaDB errors (service failure), fall back to FTS5. However, if ChromaDB successfully returns zero results, accept that as the final answer without falling back - empty results indicate no semantic match, which is more accurate than FTS5. Path 2: When the query parameter is undefined (filter-only operations), skip ChromaDB entirely and route directly to SQLite filtering using SessionStore, not SessionSearch. This path handles metadata queries like retrieving last decisions. Path 3: When a query exists but ChromaDB is unavailable, use FTS5 as a degraded fallback. The current implementation incorrectly mixes these paths, leading to inappropriate fallback behavior and inefficient routing of filter-only queries through vector search systems.\n\n---\nType: decision | Facts: When query exists and ChromaDB is available, call queryChroma with query and filters; ChromaDB errors should trigger FTS5 fallback, but ChromaDB returning zero results is final and should not trigger fallback; Filter-only queries (undefined query parameter) should skip ChromaDB entirely and use direct SQLite filtering via SessionStore; Filter-only queries should use SessionStore for direct SQLite access, not SessionSearch which involves vector operations; When query exists but ChromaDB is unavailable, FTS5 serves as degraded fallback mode; The current code incorrectly mixes these three query routing paths | Concepts: pattern, how-it-works, problem-solution, trade-off\n\n---\nDate: 11/17/2025, 11:41:54 PM\n\n---\n\n## FTS5 demoted to fallback-only search mechanism\n*Source: claude-mem://observation/10729*\n\n**FTS5 search recognized as redundant when Chroma vector database contains 1:1 SQLite copy.**\n\nThe architectural decision clarifies that FTS5 full-text search, while technically implemented, is essentially redundant in normal operation. Since Chroma vector database maintains a complete 1:1 copy of the SQLite data and provides superior search results, FTS5 exists only as a degraded fallback mode for users who cannot install Python dependencies. In practical deployment, the application requires Chroma to function properly, making FTS5 a legacy accommodation rather than a primary feature. This understanding positions FTS5 as \"Chroma search with shitty results\" - functional but not the intended search mechanism.\n\n---\nType: decision | Facts: FTS5 full-text search produces inferior results compared to Chroma vector search; Chroma database maintains a 1:1 copy of all SQLite data; FTS5 serves only as fallback for users without Python dependencies; Application functionality depends primarily on Chroma, making FTS5 nearly vestigial | Concepts: why-it-exists, trade-off, pattern, how-it-works\n\n---\nDate: 11/17/2025, 11:37:41 PM"}] |