claude-mem/test-results/test-02-sqlite-fts5-implementation.json

[{"type":"text","text":"## Query Routing Architecture: ChromaDB, SQLite, and FTS5 Fallback Strategy\n*Source: claude-mem://observation/10736*\n\n**Defined three distinct query paths based on query presence and ChromaDB availability to prevent incorrect fallback logic.**\n\nThe proper query routing architecture requires three distinct paths based on query presence and ChromaDB availability. Path 1: When a semantic query exists and ChromaDB is available, route to queryChroma with both query and filters. If ChromaDB errors (service failure), fall back to FTS5. However, if ChromaDB successfully returns zero results, accept that as the final answer without falling back - empty results indicate no semantic match, which is more accurate than FTS5. Path 2: When the query parameter is undefined (filter-only operations), skip ChromaDB entirely and route directly to SQLite filtering using SessionStore, not SessionSearch. This path handles metadata queries like retrieving last decisions. Path 3: When a query exists but ChromaDB is unavailable, use FTS5 as a degraded fallback. The current implementation incorrectly mixes these paths, leading to inappropriate fallback behavior and inefficient routing of filter-only queries through vector search systems.\n\n---\nType: decision | Facts: When query exists and ChromaDB is available, call queryChroma with query and filters; ChromaDB errors should trigger FTS5 fallback, but ChromaDB returning zero results is final and should not trigger fallback; Filter-only queries (undefined query parameter) should skip ChromaDB entirely and use direct SQLite filtering via SessionStore; Filter-only queries should use SessionStore for direct SQLite access, not SessionSearch which involves vector operations; When query exists but ChromaDB is unavailable, FTS5 serves as degraded fallback mode; The current code incorrectly mixes these three query routing paths | Concepts: pattern, how-it-works, problem-solution, trade-off\n\n---\nDate: 11/17/2025, 11:41:54 PM\n\n---\n\n## Chroma requires query text; FTS5 fallback logic is incorrect\n*Source: claude-mem://observation/10735*\n\n**Chroma cannot do filter-only queries, and FTS5 fallback on zero results is pointless.**\n\nTwo critical insights emerge about the search architecture. First, Chroma cannot perform filter-only queries without query text because it fundamentally operates on semantic search via vector embeddings. When query is undefined, the system must bypass Chroma entirely and proceed directly to SQLite structured filtering. Second, the current FTS5 fallback logic at line 472 is fundamentally flawed - it triggers when Chroma returns zero results. However, since FTS5 maintains a 1:1 copy of the SQLite data that Chroma also indexes, if Chroma returns zero results, FTS5 will also return zero results. FTS5 fallback should only activate when Chroma is unavailable or encounters an error, not when it successfully returns an empty result set.\n\n---\nType: discovery | Facts: Chroma vector database requires query text for semantic search operations; Filter-only queries must skip Chroma and use SQLite structured filtering directly; FTS5 fallback at line 472 triggers on zero results, not on Chroma errors; FTS5 contains 1:1 copy of SQLite data, so zero Chroma results means zero FTS5 results; FTS5 fallback should only activate when Chroma is unavailable or errors, not on empty results | Concepts: problem-solution, gotcha, how-it-works, why-it-exists | Files: src/servers/search-server.ts\n\n---\nDate: 11/17/2025, 11:41:43 PM\n\n---\n\n## ChromaDB Cannot Perform Filter-Only Queries Without Vector Search\n*Source: claude-mem://observation/10734*\n\n**ChromaDB requires vector search; filter-only queries return zero results, necessitating the hybrid store architecture.**\n\nThe hybrid store architecture using both ChromaDB and SQLite exists because of a fundamental limitation in ChromaDB: it cannot perform filter-only queries without vector search. This revelation clarifies the intended query routing strategy. Metadata queries (like retrieving last decisions) should go directly to SQLite, not ChromaDB. When ChromaDB is running and returns no results, this represents an accurate answer based on vector similarity - the data simply doesn't exist or doesn't match the semantic query. The FTS5 fallback mechanism should only activate when the ChromaDB service itself fails, not when it returns empty results. If ChromaDB returns no results, SQLite will also return no results because the stores maintain a 1:1 relationship. Using FTS5 when ChromaDB returns empty results would provide less accurate results than accepting ChromaDB's empty response, since FTS5 would also return nothing but through a less sophisticated search mechanism.\n\n---\nType: discovery | Facts: ChromaDB cannot execute filter-only queries without vector search components; The hybrid store architecture exists specifically because ChromaDB lacks filter-only query capability; Queries for last decisions or metadata should query SQLite directly instead of ChromaDB; When ChromaDB returns no results, SQLite will also have no results due to 1:1 relationship between stores; FTS5 fallback should only activate if ChromaDB service fails, not when ChromaDB returns empty results; Empty results from ChromaDB are more accurate than FTS5 results when ChromaDB is operational | Concepts: how-it-works, why-it-exists, gotcha, pattern\n\n---\nDate: 11/17/2025, 11:41:32 PM\n\n---\n\n## Three-tier search architecture hierarchy revealed\n*Source: claude-mem://observation/10730*\n\n**System uses Chroma vector search primarily, FTS5 as fallback, direct SQLite for structured filtering.**\n\nThe complete search architecture operates as a three-tier hierarchy based on capability and availability. Chroma vector database provides the primary semantic search mechanism, storing vector embeddings that represent a 1:1 copy of all SQLite data. When Python dependencies are unavailable, the system falls back to FTS5 full-text search, which provides keyword matching but with inferior results compared to semantic search. Direct SQLite queries handle structured filtering operations when no text query is involved. This reveals that FTS5 was previously misunderstood as a core feature when it actually exists only as a degraded fallback mode for environments lacking Python support.\n\n---\nType: discovery | Facts: Chroma provides primary semantic search using vector embeddings of SQLite data; FTS5 serves as degraded keyword search fallback when Python dependencies unavailable; Direct SQLite queries handle structured filtering when no text query present; Search architecture forms hierarchy: Chroma (best) → FTS5 (fallback) → SQLite (filtering only); Chroma maintains complete 1:1 vector embedding copy of SQLite database | Concepts: how-it-works, why-it-exists, pattern, trade-off\n\n---\nDate: 11/17/2025, 11:37:53 PM\n\n---\n\n## Found SessionStore class structure and initialization\n*Source: claude-mem://observation/10723*\n\n**SessionStore manages database and schema, likely contains methods for direct observation retrieval without FTS5.**\n\nThe SessionStore class was located, confirming its role as the database management layer. This class initializes and maintains the SQLite database schema including observations, session summaries, and user prompts tables. Unlike SessionSearch which uses FTS5 for full-text search, SessionStore should provide methods for direct table access using structured WHERE clauses based on filters like type, concepts, files, and dates. The next step is to find existing methods in SessionStore that can retrieve observations using filters alone, without requiring FTS5 text search.\n\n---\nType: discovery | Facts: SessionStore class defined at line 9 in src/services/sqlite/SessionStore.ts; SessionStore manages SQLite database with WAL mode and foreign keys enabled; Class handles schema initialization and migrations; Database contains sdk_sessions, observations, and session_summaries tables; SessionStore provides direct access to these tables for structured queries | Concepts: how-it-works, pattern | Files: src/services/sqlite/SessionStore.ts\n\n---\nDate: 11/17/2025, 10:08:37 PM"}]