claude-mem/test-results/test-11-search-server-changes.json

[{"type":"text","text":"## Unified search handler implements Chroma-first with FTS5 fallback on zero results\n*Source: claude-mem://observation/10757*\n\n**Lines 403-489 show current implementation attempts Chroma first, falls back to FTS5 when empty.**\n\nThe unified search handler in search-server.ts (lines 390-489) reveals the current implementation architecture. The handler first attempts Chroma semantic search when chromaClient is available, calling queryChroma with the query parameter at line 419. If Chroma throws an error, it catches at line 465 and falls through to FTS5. However, the problematic FTS5 fallback logic at line 470 triggers when all result arrays are empty, not just on errors. This contradicts the insight that FTS5 contains identical data to Chroma, making zero-result fallback pointless. Additionally, lines 476-483 call search methods (searchObservations, searchSessions, searchUserPrompts) with the query parameter without checking if it's undefined, which will cause failures when query is not provided. The handler needs modification to skip Chroma when query is undefined and only fall back to FTS5 on Chroma errors.\n\n---\nType: discovery | Facts: Line 403 checks `if (chromaClient)` to attempt Chroma semantic search first; Line 419 calls `queryChroma(query, 100, whereFilter)` with query parameter as required; Lines 465-467 catch Chroma errors and fall through to FTS5 fallback; Line 470 triggers FTS5 fallback when all result arrays have zero length; Lines 476-483 call SessionSearch methods with query parameter without checking if undefined; Hybrid search applies 90-day recency filter at lines 424-431 | Concepts: how-it-works, problem-solution, gotcha | Files: src/servers/search-server.ts\n\n---\nDate: 11/17/2025, 11:50:29 PM\n\n---\n\n## Search Query Parameter Made Optional for Filter-Only Queries\n*Source: claude-mem://observation/10744*\n\n**Unified search tool now supports filter-only queries by making query parameter optional.**\n\nThe unified search tool was updated to support filter-only query patterns by making the 'query' parameter optional. Previously, the search tool required a natural language query string, but now users can perform searches using only metadata filters such as observation type, date ranges, concepts, or file paths. The parameter description was enhanced to explicitly indicate that the query is optional and should be omitted for filter-only queries. Additionally, the description was clarified to state that ChromaDB is used for semantic ranking when available, with FTS5 serving as a fallback when ChromaDB is unavailable, making the system's fallback behavior more transparent. This change increases the flexibility of the search interface and better aligns with use cases where users want to browse or filter by specific criteria without performing text-based search.\n\n---\nType: change | Facts: The 'query' parameter in the search tool's inputSchema changed from required to optional; File modified: /Users/alexnewman/Scripts/claude-mem/src/servers/search-server.ts; Updated description clarifies query is optional and can be omitted for filter-only queries; Description now explicitly states ChromaDB is used when available, FTS5 as fallback when unavailable; Change enables searching by filters (type, dateRange, concepts, files) without providing search text | Concepts: what-changed, how-it-works, pattern | Files: /Users/alexnewman/Scripts/claude-mem/src/servers/search-server.ts\n\n---\nDate: 11/17/2025, 11:47:49 PM\n\n---\n\n## Search Tool Requires Query Parameter as Non-Optional String\n*Source: claude-mem://observation/10743*\n\n**The search tool schema defines query as mandatory string parameter, preventing filter-only queries through current interface.**\n\nThe search tool definition in src/servers/search-server.ts reveals that the query parameter is currently defined as a mandatory string using z.string(), which prevents filter-only queries from being submitted through the MCP interface. The tool is described as providing unified search across observations, sessions, and user prompts using hybrid semantic and full-text search, with ChromaDB as primary and SQLite FTS5 as fallback. While the schema includes optional filters like type (for document type filtering), the query parameter itself cannot be omitted. To support the newly identified filter-only query pattern where queries go directly to SQLite instead of ChromaDB, this schema will need modification to make the query parameter optional using z.string().optional(), allowing users to submit filter-only requests.\n\n---\nType: discovery | Facts: The search tool is defined at line 363 in src/servers/search-server.ts; The query parameter is defined as z.string() without optional modifier, making it mandatory; The query parameter description states it enables semantic ranking via ChromaDB with FTS5 fallback; The inputSchema includes optional type filter for observations, sessions, or prompts; The search tool provides index and full output format options with index as default | Concepts: how-it-works, problem-solution | Files: src/servers/search-server.ts\n\n---\nDate: 11/17/2025, 11:47:33 PM\n\n---\n\n## Unified search handler fix requires five-step query routing logic\n*Source: claude-mem://observation/10739*\n\n**Route by query presence, error-based FTS5 fallback only, remove zero-results fallback check.**\n\nThe fix for the unified search handler requires restructuring the query routing logic into five distinct steps. First, check whether the query parameter exists at the top of the handler. If query is undefined, bypass Chroma entirely and invoke a new filter-only path in SessionSearch that performs direct SQLite filtering. If query exists, proceed with Chroma semantic search. The FTS5 fallback logic must change fundamentally - it should only trigger when Chroma throws an error (caught at line 465), not when Chroma successfully returns zero results. When Chroma returns zero results, that is the correct answer since FTS5 contains identical data and would also return zero. This requires removing the zero-results check from line 472's fallback condition, fixing the current incorrect fallback behavior.\n\n---\nType: decision | Facts: Step 1: Check if query parameter exists at beginning of unified search handler; Step 2: If query is undefined, skip Chroma and call new filter-only path in SessionSearch; Step 3: If query exists, attempt Chroma semantic search; Step 4: If Chroma throws error (line 465 catch block), fall back to FTS5; Step 5: If Chroma succeeds with zero results, accept zero as correct answer without FTS5 fallback; Line 472 condition must be modified to remove zero-results check | Concepts: problem-solution, how-it-works, pattern, trade-off | Files: src/servers/search-server.ts\n\n---\nDate: 11/17/2025, 11:42:18 PM\n\n---\n\n## Chroma requires query text; FTS5 fallback logic is incorrect\n*Source: claude-mem://observation/10735*\n\n**Chroma cannot do filter-only queries, and FTS5 fallback on zero results is pointless.**\n\nTwo critical insights emerge about the search architecture. First, Chroma cannot perform filter-only queries without query text because it fundamentally operates on semantic search via vector embeddings. When query is undefined, the system must bypass Chroma entirely and proceed directly to SQLite structured filtering. Second, the current FTS5 fallback logic at line 472 is fundamentally flawed - it triggers when Chroma returns zero results. However, since FTS5 maintains a 1:1 copy of the SQLite data that Chroma also indexes, if Chroma returns zero results, FTS5 will also return zero results. FTS5 fallback should only activate when Chroma is unavailable or encounters an error, not when it successfully returns an empty result set.\n\n---\nType: discovery | Facts: Chroma vector database requires query text for semantic search operations; Filter-only queries must skip Chroma and use SQLite structured filtering directly; FTS5 fallback at line 472 triggers on zero results, not on Chroma errors; FTS5 contains 1:1 copy of SQLite data, so zero Chroma results means zero FTS5 results; FTS5 fallback should only activate when Chroma is unavailable or errors, not on empty results | Concepts: problem-solution, gotcha, how-it-works, why-it-exists | Files: src/servers/search-server.ts\n\n---\nDate: 11/17/2025, 11:41:43 PM"}]