claude-mem/test-results/test-06-search-architecture-decision.json

[{"type":"text","text":"## FTS5 Method Renamed to Indicate Fallback Status\n*Source: claude-mem://observation/10747*\n\n**Renamed escapeFTS5 to escapeFTS5_fallback_when_chroma_unavailable across all search methods to clarify temporary usage.**\n\nThe FTS5 query escaping method was renamed from escapeFTS5 to escapeFTS5_fallback_when_chroma_unavailable to align with the newly established naming convention for FTS5-related code. This change was applied across all three search methods in SessionSearch: searchObservations, searchSessions, and searchUserPrompts. The rename makes explicit that FTS5 is only being used as a temporary fallback solution when ChromaDB (which requires UVX) is unavailable, rather than being a permanent part of the architecture. The method's existing documentation already stated that FTS5 provides degraded search capability with no semantic understanding and is only used when uvx/Python/ChromaDB is disabled. This refactoring follows the pattern established in CLAUDE.md where FTS5-related functions are given semantic names indicating their temporary nature pending UVX availability.\n\n---\nType: refactor | Facts: File modified: /Users/alexnewman/Scripts/claude-mem/src/services/sqlite/SessionSearch.ts; Method escapeFTS5 renamed to escapeFTS5_fallback_when_chroma_unavailable throughout the file; Three search methods updated: searchObservations, searchSessions, and searchUserPrompts; All calls to escapeFTS5 replaced with escapeFTS5_fallback_when_chroma_unavailable using replace_all; Method already had documentation indicating FTS5 is degraded fallback when ChromaDB unavailable | Concepts: what-changed, why-it-exists, pattern | Files: /Users/alexnewman/Scripts/claude-mem/src/services/sqlite/SessionSearch.ts\n\n---\nDate: 11/17/2025, 11:48:30 PM\n\n---\n\n## Three-tier search architecture hierarchy revealed\n*Source: claude-mem://observation/10730*\n\n**System uses Chroma vector search primarily, FTS5 as fallback, direct SQLite for structured filtering.**\n\nThe complete search architecture operates as a three-tier hierarchy based on capability and availability. Chroma vector database provides the primary semantic search mechanism, storing vector embeddings that represent a 1:1 copy of all SQLite data. When Python dependencies are unavailable, the system falls back to FTS5 full-text search, which provides keyword matching but with inferior results compared to semantic search. Direct SQLite queries handle structured filtering operations when no text query is involved. This reveals that FTS5 was previously misunderstood as a core feature when it actually exists only as a degraded fallback mode for environments lacking Python support.\n\n---\nType: discovery | Facts: Chroma provides primary semantic search using vector embeddings of SQLite data; FTS5 serves as degraded keyword search fallback when Python dependencies unavailable; Direct SQLite queries handle structured filtering when no text query present; Search architecture forms hierarchy: Chroma (best) → FTS5 (fallback) → SQLite (filtering only); Chroma maintains complete 1:1 vector embedding copy of SQLite database | Concepts: how-it-works, why-it-exists, pattern, trade-off\n\n---\nDate: 11/17/2025, 11:37:53 PM\n\n---\n\n## Final solution synthesis: separate text search from structured filtering\n*Source: claude-mem://observation/10728*\n\n**Root cause is architectural: FTS5 text search and structured filtering conflated; fix separates them into independent query paths.**\n\nThe sequential thinking process concluded with a clear architectural understanding and implementation plan. The root cause is a design assumption that text search and structured filtering always occur together, leading to FTS5 being required even when users only want to filter by type, concepts, or files. The proper fix has two implementation parts: make the query parameter optional in the schema, and add conditional logic to the search methods that branches based on query presence. When query exists, use the full-text search path; when absent, use direct table querying with filters. This architectural separation allows text search and structured filtering to work independently or in combination, fixing the immediate bug while enabling more flexible query patterns. The sequential thinking reached thought 12 of 12 with complete problem analysis and solution design.\n\n---\nType: decision | Facts: Bug exists because system forces FTS5 search even for filter-only queries; Fix part 1: Change line 366 schema from query: z.string() to query: z.string().optional(); Fix part 2: Add conditional branching in searchObservations, searchSessions, searchUserPrompts; When query exists: use FTS5 path with JOIN observations_fts and MATCH clause; When query undefined: skip FTS5, query base table directly with filter WHERE clause only; Architecture separates concerns: text search (FTS5) independent from structured filtering (SQL WHERE); Original error occurred because code assumed text search and filtering always work together | Concepts: problem-solution, how-it-works, why-it-exists, pattern, trade-off\n\n---\nDate: 11/17/2025, 10:09:25 PM\n\n---\n\n## Architectural insight: use SessionStore for filter-only, SessionSearch for text search\n*Source: claude-mem://observation/10722*\n\n**SessionSearch class handles FTS5 text search; SessionStore class should handle structured filter queries without text.**\n\nA key architectural insight emerged: the system already has appropriate separation of concerns between SessionSearch and SessionStore classes. SessionSearch is designed for full-text search using FTS5, which inherently requires search text. SessionStore should handle structured queries based purely on filters (type, concepts, files, dates) without requiring full-text search. The real fix isn't to patch escapeFTS5 or make FTS5 work without a query - it's to route filter-only queries to SessionStore methods instead of SessionSearch methods. This preserves the architectural integrity where each class serves its intended purpose.\n\n---\nType: decision | Facts: SessionSearch.searchObservations always uses FTS5 with WHERE observations_fts MATCH ?; FTS5 fundamentally requires search term, cannot MATCH against nothing; SessionStore class likely contains methods for structured queries without FTS5; Proper architecture: SessionSearch for full-text search, SessionStore for filter-based retrieval; Sequential thinking at thought 8 of 12 | Concepts: pattern, how-it-works, trade-off, why-it-exists\n\n---\nDate: 11/17/2025, 10:08:29 PM\n\n---\n\n## Root cause identified: required query parameter vs filter-only queries\n*Source: claude-mem://observation/10717*\n\n**Schema defines query as required but filter-only queries need query to be optional.**\n\nThe root cause has been identified: a mismatch between the schema definition and actual usage patterns. The Zod schema defines query as z.string(), marking it as required, yet the system appears to receive requests without a query parameter for filter-only searches. This creates two possibilities: either (1) the schema should allow optional queries to support filtering by type/concepts/files without search text, or (2) query should truly be required and filter-only requests should be rejected. The investigation now shifts to understanding whether the search functions can meaningfully operate without a query parameter, which will determine the correct fix approach.\n\n---\nType: discovery | Facts: Line 366 defines query: z.string() as required, not optional; Runtime errors show requests without query are passing through validation; Filter-only queries (using type, obs_type, concepts, files) should be valid use cases; Schema should likely be z.string().optional() to support filter-only retrieval; Sequential thinking extended from 10 to 12 total thoughts; Next investigation examines search.searchObservations to determine if query is truly required | Concepts: problem-solution, gotcha, trade-off\n\n---\nDate: 11/17/2025, 10:07:36 PM"}]