Alex Newman
c5e68a17c8
* Refactor code structure for improved readability and maintainability * Add test results for search API and related functionalities - Created test result files for various search-related functionalities, including: - test-11-search-server-changes.json - test-12-context-hook-changes.json - test-13-worker-service-changes.json - test-14-patterns.json - test-15-gotchas.json - test-16-discoveries.json - test-17-all-bugfixes.json - test-18-all-features.json - test-19-all-decisions.json - test-20-session-search.json - test-21-prompt-search.json - test-22-decisions-endpoint.json - test-23-changes-endpoint.json - test-24-how-it-works-endpoint.json - test-25-contextualize-endpoint.json - test-26-timeline-around-observation.json - test-27-multi-param-combo.json - test-28-file-type-combo.json - Each test result file captures specific search failures or outcomes, including issues with undefined properties and successful execution of search queries. - Enhanced documentation of search architecture and testing strategies, ensuring compliance with established guidelines and improving overall search functionality. * feat: Enhance unified search API with catch-all parameters and backward compatibility - Implemented a unified search API at /api/search that accepts catch-all parameters for filtering by type, observation type, concepts, and files. - Maintained backward compatibility by keeping granular endpoints functional while routing through the same infrastructure. - Completed comprehensive testing of search capabilities with real-world query scenarios. fix: Address missing debug output in search API query tests - Flushed PM2 logs and executed search queries to verify functionality. - Diagnosed absence of "Raw Chroma" debug messages in worker logs, indicating potential issues with logging or query processing. refactor: Improve build and deployment pipeline for claude-mem plugin - Successfully built and synced all hooks and services to the marketplace directory. - Ensured all dependencies are installed and up-to-date in the deployment location. feat: Implement hybrid search filters with 90-day recency window - Enhanced search server to apply a 90-day recency filter to Chroma results before categorizing by document type. fix: Correct parameter handling in searchUserPrompts method - Added support for filter-only queries and improved dual-path logic for clarity. refactor: Rename FTS5 method to clarify fallback status - Renamed escapeFTS5 to escapeFTS5_fallback_when_chroma_unavailable to indicate its temporary usage. feat: Introduce contextualize tool for comprehensive project overview - Added a new tool to fetch recent observations, sessions, and user prompts, providing a quick project overview. feat: Add semantic shortcut tools for common search patterns - Implemented 'decisions', 'changes', and 'how_it_works' tools for convenient access to frequently searched observation categories. feat: Unified timeline tool supports anchor and query modes - Combined get_context_timeline and get_timeline_by_query into a single interface for timeline exploration. feat: Unified search tool added to MCP server - New tool queries all memory types simultaneously, providing combined chronological results for improved search efficiency. * Refactor search functionality to clarify FTS5 fallback usage - Updated `worker-service.cjs` to replace FTS5 fallback function with a more descriptive name and improved error handling. - Enhanced documentation in `SKILL.md` to specify the unified API endpoint and clarify the behavior of the search engine, including the conditions under which FTS5 is used. - Modified `search-server.ts` to provide clearer logging and descriptions regarding the fallback to FTS5 when UVX/Python is unavailable. - Renamed and updated the `SessionSearch.ts` methods to reflect the conditions for using FTS5, emphasizing the lack of semantic understanding in fallback scenarios. * feat: Add ID-based fetch endpoints and simplify mem-search skill **Problem:** - Search returns IDs but no way to fetch by ID - Skill documentation was bloated with too many options - Claude wasn't using IDs because we didn't tell it how **Solution:** 1. Added three new HTTP endpoints: - GET /api/observation/:id - GET /api/session/:id - GET /api/prompt/:id 2. Completely rewrote SKILL.md: - Stripped complexity down to essentials - Clear 3-step prescriptive workflow: Search → Review IDs → Fetch by ID - Emphasized ID usage: "The IDs are there for a reason - USE THEM" - Removed confusing multi-endpoint documentation - Kept only unified search with filters **Impact:** - Token efficiency: Claude can now fetch full details only for relevant IDs - Clarity: One clear workflow instead of 10+ options to choose from - Usability: IDs are no longer wasted context - they're actionable 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * chore: Move internal docs to private directory Moved POSTMORTEM and planning docs to ./private to exclude from PR reviews. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Remove experimental contextualize endpoint - Removed contextualize MCP tool from search-server (saves ~4KB) - Disabled FTS5 fallback paths in SessionSearch (now vector-first) - Cleaned up CLAUDE.md documentation - Removed contextualize-rewrite-plan.md doc Rationale: - Contextualize is better suited as a skill (LLM-powered) than an endpoint - Search API already provides vector search with configurable limits - Created issue #132 to track future contextualize skill implementation Changes: - src/servers/search-server.ts: Removed contextualize tool definition - src/services/sqlite/SessionSearch.ts: Disabled FTS5 fallback, added deprecation warnings - CLAUDE.md: Cleaned up outdated skill documentation - docs/: Removed contextualize plan document 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Complete FTS5 cleanup - remove all deprecated search code This completes the FTS5 cleanup work by removing all commented-out FTS5 search code while preserving database tables for backward compatibility. Changes: - Removed 200+ lines of commented FTS5 search code from SessionSearch.ts - Removed deprecated degraded_search_query__when_uvx_unavailable method - Updated all method documentation to clarify vector-first architecture - Updated class documentation to reflect filter-only query support - Updated CLAUDE.md to remove FTS5 search references - Clarified that FTS5 tables exist for backward compatibility only - Updated "Why SQLite FTS5" section to "Why Vector-First Search" Database impact: NONE - FTS5 tables remain intact for existing installations Search architecture: - ChromaDB: All text-based vector search queries - SQLite: Filter-only queries (date ranges, metadata, no query text) - FTS5 tables: Maintained but unused (backward compatibility) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Remove all FTS5 fallback execution code from search-server Completes the FTS5 cleanup by removing all fallback execution paths that attempted to use FTS5 when ChromaDB was unavailable. Changes: - Removed all FTS5 fallback code execution paths - When ChromaDB fails or is unavailable, return empty results with helpful error messages - Updated all deprecated tool descriptions (search_observations, search_sessions, search_user_prompts) - Changed error messages to indicate FTS5 fallback has been removed - Added installation instructions for UVX/Python when vector search is unavailable - Updated comments from "hybrid search" to "vector-first search" - Removed ~100 lines of dead FTS5 fallback code Database impact: NONE - FTS5 tables remain intact (backward compatibility) Search behavior when ChromaDB unavailable: - Text queries: Return empty results with error explaining ChromaDB is required - Filter-only queries (no text): Continue to work via direct SQLite 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Address PR 133 review feedback Critical fixes: - Remove contextualize endpoint from worker-service (route + handler) - Fix build script logging to show correct .cjs extension (was .mjs) Documentation improvements: - Add comprehensive FTS5 retention rationale documentation - Include v7.0.0 removal TODO for future cleanup Testing: - Build succeeds with correct output logging - Worker restarts successfully (30th restart) - Contextualize endpoint properly removed (404 response) - Search endpoint verified working This addresses all critical review feedback from PR 133. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
1 line
8.7 KiB
JSON
1 line
8.7 KiB
JSON
[{"type":"text","text":"## FTS5 Method Renamed to Indicate Fallback Status\n*Source: claude-mem://observation/10747*\n\n**Renamed escapeFTS5 to escapeFTS5_fallback_when_chroma_unavailable across all search methods to clarify temporary usage.**\n\nThe FTS5 query escaping method was renamed from escapeFTS5 to escapeFTS5_fallback_when_chroma_unavailable to align with the newly established naming convention for FTS5-related code. This change was applied across all three search methods in SessionSearch: searchObservations, searchSessions, and searchUserPrompts. The rename makes explicit that FTS5 is only being used as a temporary fallback solution when ChromaDB (which requires UVX) is unavailable, rather than being a permanent part of the architecture. The method's existing documentation already stated that FTS5 provides degraded search capability with no semantic understanding and is only used when uvx/Python/ChromaDB is disabled. This refactoring follows the pattern established in CLAUDE.md where FTS5-related functions are given semantic names indicating their temporary nature pending UVX availability.\n\n---\nType: refactor | Facts: File modified: /Users/alexnewman/Scripts/claude-mem/src/services/sqlite/SessionSearch.ts; Method escapeFTS5 renamed to escapeFTS5_fallback_when_chroma_unavailable throughout the file; Three search methods updated: searchObservations, searchSessions, and searchUserPrompts; All calls to escapeFTS5 replaced with escapeFTS5_fallback_when_chroma_unavailable using replace_all; Method already had documentation indicating FTS5 is degraded fallback when ChromaDB unavailable | Concepts: what-changed, why-it-exists, pattern | Files: /Users/alexnewman/Scripts/claude-mem/src/services/sqlite/SessionSearch.ts\n\n---\nDate: 11/17/2025, 11:48:30 PM\n\n---\n\n## Architectural insight: use SessionStore for filter-only, SessionSearch for text search\n*Source: claude-mem://observation/10722*\n\n**SessionSearch class handles FTS5 text search; SessionStore class should handle structured filter queries without text.**\n\nA key architectural insight emerged: the system already has appropriate separation of concerns between SessionSearch and SessionStore classes. SessionSearch is designed for full-text search using FTS5, which inherently requires search text. SessionStore should handle structured queries based purely on filters (type, concepts, files, dates) without requiring full-text search. The real fix isn't to patch escapeFTS5 or make FTS5 work without a query - it's to route filter-only queries to SessionStore methods instead of SessionSearch methods. This preserves the architectural integrity where each class serves its intended purpose.\n\n---\nType: decision | Facts: SessionSearch.searchObservations always uses FTS5 with WHERE observations_fts MATCH ?; FTS5 fundamentally requires search term, cannot MATCH against nothing; SessionStore class likely contains methods for structured queries without FTS5; Proper architecture: SessionSearch for full-text search, SessionStore for filter-based retrieval; Sequential thinking at thought 8 of 12 | Concepts: pattern, how-it-works, trade-off, why-it-exists\n\n---\nDate: 11/17/2025, 10:08:29 PM\n\n---\n\n## Root cause identified: required query parameter vs filter-only queries\n*Source: claude-mem://observation/10717*\n\n**Schema defines query as required but filter-only queries need query to be optional.**\n\nThe root cause has been identified: a mismatch between the schema definition and actual usage patterns. The Zod schema defines query as z.string(), marking it as required, yet the system appears to receive requests without a query parameter for filter-only searches. This creates two possibilities: either (1) the schema should allow optional queries to support filtering by type/concepts/files without search text, or (2) query should truly be required and filter-only requests should be rejected. The investigation now shifts to understanding whether the search functions can meaningfully operate without a query parameter, which will determine the correct fix approach.\n\n---\nType: discovery | Facts: Line 366 defines query: z.string() as required, not optional; Runtime errors show requests without query are passing through validation; Filter-only queries (using type, obs_type, concepts, files) should be valid use cases; Schema should likely be z.string().optional() to support filter-only retrieval; Sequential thinking extended from 10 to 12 total thoughts; Next investigation examines search.searchObservations to determine if query is truly required | Concepts: problem-solution, gotcha, trade-off\n\n---\nDate: 11/17/2025, 10:07:36 PM\n\n---\n\n## Session Search Returns Multi-Session Summaries with Structured Format\n*Source: claude-mem://observation/10696*\n\n**Search retrieved five session summaries about hybrid search architecture work with completed tasks and learnings**\n\nExamination of test-20-session-search.json reveals how session-level search provides comprehensive work history context. The query retrieved five sessions related to hybrid search architecture work, demonstrating the value of session summaries for understanding project evolution. Each session result follows a structured format with five sections: Completed (what was accomplished), Learned (technical insights gained), Investigated (what was examined), Next Steps (planned follow-up), and Notes (additional context). Sessions are tagged with claude-mem://session/UUID source links for reference. The retrieved sessions trace the complete lifecycle of the hybrid search architecture documentation fix: initial creation of the fix document, relocation between projects, implementation of documentation updates, PR creation and merging, and final comment corrections for architectural accuracy. This demonstrates how session-level search complements observation search by providing narrative flow and project history context rather than atomic technical facts.\n\n---\nType: discovery | Facts: Session search results return summaries with Completed, Learned, Investigated, Next Steps, and Notes sections; Session summaries tagged with source links using claude-mem://session/UUID format; Search query about hybrid search architecture retrieved five relevant sessions from November 15-17, 2025; Session results include session date and comprehensive context about what was accomplished in each session; Retrieved sessions document the evolution of hybrid search architecture fixes across multiple work sessions | Concepts: how-it-works, pattern, why-it-exists | Files: test-results/test-20-session-search.json\n\n---\nDate: 11/17/2025, 9:52:57 PM\n\n---\n\n## Context Retrieval System Architecture Analysis\n*Source: claude-mem://observation/10694*\n\n**Three-layer hybrid semantic search enables precision context injection for SessionStart**\n\nAnalysis of the 28 passing test queries reveals a production-grade context retrieval system designed for precision over bulk context injection. The architecture operates on three distinct layers: observations (AI-compressed insights), sessions (overall summaries), and prompts (exact user language), allowing retrieval strategies tailored to different needs. The hybrid Chroma + FTS5 search enables both semantic matching (finding conceptually related content) and keyword matching (exact term searches). Multi-parameter filter composability transforms broad queries like \"authentication\" into precise requests like \"bugfix observations about authentication in auth.ts from last week\". Timeline functionality provides chronological context to understand why decisions were made and how features evolved. The dual endpoint strategy separates ad-hoc session queries from optimized SessionStart injection patterns. This design philosophy prioritizes targeted, relevant context delivery over simply dumping recent observations.\n\n---\nType: discovery | Facts: Search system implements three-layer architecture: AI-compressed observations, session-level summaries, and verbatim user prompts; Hybrid search combines Chroma semantic embeddings with SQLite FTS5 keyword search for both conceptual and exact matching; Multi-parameter filtering allows stacking filters (type+query+file+dateRange+concept+project) for precision retrieval; Timeline queries (get_context_timeline and get_timeline_by_query) provide chronological context around specific events; Dual endpoint approach: unified /api/search for ad-hoc queries and dedicated endpoints (by-type, by-file, by-concept) for optimized SessionStart context injection; Test categories validate real-world context retrieval patterns: semantic (conceptual similarity), decision (architectural choices), troubleshooting (past bug fixes), file-specific (code history) | Concepts: how-it-works, pattern, why-it-exists\n\n---\nDate: 11/17/2025, 9:52:24 PM"}] |