Alex Newman
c5e68a17c8
* Refactor code structure for improved readability and maintainability * Add test results for search API and related functionalities - Created test result files for various search-related functionalities, including: - test-11-search-server-changes.json - test-12-context-hook-changes.json - test-13-worker-service-changes.json - test-14-patterns.json - test-15-gotchas.json - test-16-discoveries.json - test-17-all-bugfixes.json - test-18-all-features.json - test-19-all-decisions.json - test-20-session-search.json - test-21-prompt-search.json - test-22-decisions-endpoint.json - test-23-changes-endpoint.json - test-24-how-it-works-endpoint.json - test-25-contextualize-endpoint.json - test-26-timeline-around-observation.json - test-27-multi-param-combo.json - test-28-file-type-combo.json - Each test result file captures specific search failures or outcomes, including issues with undefined properties and successful execution of search queries. - Enhanced documentation of search architecture and testing strategies, ensuring compliance with established guidelines and improving overall search functionality. * feat: Enhance unified search API with catch-all parameters and backward compatibility - Implemented a unified search API at /api/search that accepts catch-all parameters for filtering by type, observation type, concepts, and files. - Maintained backward compatibility by keeping granular endpoints functional while routing through the same infrastructure. - Completed comprehensive testing of search capabilities with real-world query scenarios. fix: Address missing debug output in search API query tests - Flushed PM2 logs and executed search queries to verify functionality. - Diagnosed absence of "Raw Chroma" debug messages in worker logs, indicating potential issues with logging or query processing. refactor: Improve build and deployment pipeline for claude-mem plugin - Successfully built and synced all hooks and services to the marketplace directory. - Ensured all dependencies are installed and up-to-date in the deployment location. feat: Implement hybrid search filters with 90-day recency window - Enhanced search server to apply a 90-day recency filter to Chroma results before categorizing by document type. fix: Correct parameter handling in searchUserPrompts method - Added support for filter-only queries and improved dual-path logic for clarity. refactor: Rename FTS5 method to clarify fallback status - Renamed escapeFTS5 to escapeFTS5_fallback_when_chroma_unavailable to indicate its temporary usage. feat: Introduce contextualize tool for comprehensive project overview - Added a new tool to fetch recent observations, sessions, and user prompts, providing a quick project overview. feat: Add semantic shortcut tools for common search patterns - Implemented 'decisions', 'changes', and 'how_it_works' tools for convenient access to frequently searched observation categories. feat: Unified timeline tool supports anchor and query modes - Combined get_context_timeline and get_timeline_by_query into a single interface for timeline exploration. feat: Unified search tool added to MCP server - New tool queries all memory types simultaneously, providing combined chronological results for improved search efficiency. * Refactor search functionality to clarify FTS5 fallback usage - Updated `worker-service.cjs` to replace FTS5 fallback function with a more descriptive name and improved error handling. - Enhanced documentation in `SKILL.md` to specify the unified API endpoint and clarify the behavior of the search engine, including the conditions under which FTS5 is used. - Modified `search-server.ts` to provide clearer logging and descriptions regarding the fallback to FTS5 when UVX/Python is unavailable. - Renamed and updated the `SessionSearch.ts` methods to reflect the conditions for using FTS5, emphasizing the lack of semantic understanding in fallback scenarios. * feat: Add ID-based fetch endpoints and simplify mem-search skill **Problem:** - Search returns IDs but no way to fetch by ID - Skill documentation was bloated with too many options - Claude wasn't using IDs because we didn't tell it how **Solution:** 1. Added three new HTTP endpoints: - GET /api/observation/:id - GET /api/session/:id - GET /api/prompt/:id 2. Completely rewrote SKILL.md: - Stripped complexity down to essentials - Clear 3-step prescriptive workflow: Search → Review IDs → Fetch by ID - Emphasized ID usage: "The IDs are there for a reason - USE THEM" - Removed confusing multi-endpoint documentation - Kept only unified search with filters **Impact:** - Token efficiency: Claude can now fetch full details only for relevant IDs - Clarity: One clear workflow instead of 10+ options to choose from - Usability: IDs are no longer wasted context - they're actionable 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * chore: Move internal docs to private directory Moved POSTMORTEM and planning docs to ./private to exclude from PR reviews. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Remove experimental contextualize endpoint - Removed contextualize MCP tool from search-server (saves ~4KB) - Disabled FTS5 fallback paths in SessionSearch (now vector-first) - Cleaned up CLAUDE.md documentation - Removed contextualize-rewrite-plan.md doc Rationale: - Contextualize is better suited as a skill (LLM-powered) than an endpoint - Search API already provides vector search with configurable limits - Created issue #132 to track future contextualize skill implementation Changes: - src/servers/search-server.ts: Removed contextualize tool definition - src/services/sqlite/SessionSearch.ts: Disabled FTS5 fallback, added deprecation warnings - CLAUDE.md: Cleaned up outdated skill documentation - docs/: Removed contextualize plan document 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Complete FTS5 cleanup - remove all deprecated search code This completes the FTS5 cleanup work by removing all commented-out FTS5 search code while preserving database tables for backward compatibility. Changes: - Removed 200+ lines of commented FTS5 search code from SessionSearch.ts - Removed deprecated degraded_search_query__when_uvx_unavailable method - Updated all method documentation to clarify vector-first architecture - Updated class documentation to reflect filter-only query support - Updated CLAUDE.md to remove FTS5 search references - Clarified that FTS5 tables exist for backward compatibility only - Updated "Why SQLite FTS5" section to "Why Vector-First Search" Database impact: NONE - FTS5 tables remain intact for existing installations Search architecture: - ChromaDB: All text-based vector search queries - SQLite: Filter-only queries (date ranges, metadata, no query text) - FTS5 tables: Maintained but unused (backward compatibility) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Remove all FTS5 fallback execution code from search-server Completes the FTS5 cleanup by removing all fallback execution paths that attempted to use FTS5 when ChromaDB was unavailable. Changes: - Removed all FTS5 fallback code execution paths - When ChromaDB fails or is unavailable, return empty results with helpful error messages - Updated all deprecated tool descriptions (search_observations, search_sessions, search_user_prompts) - Changed error messages to indicate FTS5 fallback has been removed - Added installation instructions for UVX/Python when vector search is unavailable - Updated comments from "hybrid search" to "vector-first search" - Removed ~100 lines of dead FTS5 fallback code Database impact: NONE - FTS5 tables remain intact (backward compatibility) Search behavior when ChromaDB unavailable: - Text queries: Return empty results with error explaining ChromaDB is required - Filter-only queries (no text): Continue to work via direct SQLite 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Address PR 133 review feedback Critical fixes: - Remove contextualize endpoint from worker-service (route + handler) - Fix build script logging to show correct .cjs extension (was .mjs) Documentation improvements: - Add comprehensive FTS5 retention rationale documentation - Include v7.0.0 removal TODO for future cleanup Testing: - Build succeeds with correct output logging - Worker restarts successfully (30th restart) - Contextualize endpoint properly removed (404 response) - Search endpoint verified working This addresses all critical review feedback from PR 133. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
1 line
10 KiB
JSON
1 line
10 KiB
JSON
[{"type":"text","text":"## Build and Deployment of Unified Search API\n*Source: claude-mem://observation/10682*\n\n**Successfully built all hooks, worker service, and search server, then synced to marketplace and restarted worker.**\n\nThe unified search API implementation was successfully built and deployed. The build process generated all necessary components including the worker service, search server, and various hooks, with the search server weighing in at 332.97 KB. The React viewer UI was also compiled with all required assets. After building, the entire project (11,781 files) was synced to the Claude marketplace plugins directory using rsync, followed by npm install to ensure all dependencies were properly installed. Finally, the worker service was restarted to activate the new unified search endpoints. This deployment marks the completion of the unified search API architecture with its six public-facing endpoints and backward-compatible granular routing.\n\n---\nType: change | Facts: Build process compiled 8 components: worker-service (1337.16 KB), search-server (332.97 KB), context-hook (38.29 KB), new-hook (32.14 KB), save-hook (32.49 KB), summary-hook (34.61 KB), cleanup-hook (31.38 KB), and user-message-hook (2.24 KB); React viewer built successfully with viewer-bundle.js, viewer.html, font assets, and 4 icon SVG files; Project synced to ~/.claude/plugins/marketplaces/thedotmack/ using rsync with 11,781 files transferred; npm install executed in marketplace directory to ensure dependencies are current; Worker service restarted to activate the newly deployed unified search API endpoints | Concepts: what-changed, how-it-works | Files: plugin/scripts/worker-service.cjs, plugin/scripts/search-server.mjs, plugin/scripts/context-hook.js, plugin/scripts/new-hook.js, plugin/scripts/save-hook.js, plugin/scripts/summary-hook.js, plugin/scripts/cleanup-hook.js, plugin/scripts/user-message-hook.js, plugin/ui/viewer-bundle.js, plugin/ui/viewer.html\n\n---\nDate: 11/17/2025, 9:43:18 PM\n\n---\n\n## Build and Deployment Pipeline Execution\n*Source: claude-mem://observation/10663*\n\n**Successfully built all hooks, worker service, search server, synced to marketplace, and restarted Cloudflare worker.**\n\nThe complete build and deployment pipeline executed successfully for claude-mem version 6.0.9. The build process first prepared output directories, then compiled the React viewer bundle along with all necessary UI assets including fonts and icons. Next, it built the worker service and search server components followed by all six hook modules (context, new, save, summary, cleanup, and user-message). After successful compilation, the rsync operation transferred all 11,781 project files to the marketplace plugin directory, removing any obsolete files with the --delete flag. Finally, npm install ensured all dependencies were current in the deployment location. This deployment follows the implementation of the unified search API architecture and prepares the system for comprehensive testing of the new search capabilities.\n\n---\nType: change | Facts: Build process compiled 8 components: worker-service (1337.16 KB), search-server (332.08 KB), context-hook (38.29 KB), new-hook (32.14 KB), save-hook (32.49 KB), summary-hook (34.61 KB), cleanup-hook (31.38 KB), and user-message-hook (2.24 KB); React viewer was built successfully with viewer-bundle.js, viewer.html template, font assets, and 4 icon SVG files in plugin/ui directory; Project synced 11,781 files from /Users/alexnewman/Scripts/claude-mem to ~/.claude/plugins/marketplaces/thedotmack/ using rsync with --delete flag; npm install ran in marketplace directory after sync to update dependencies; Build outputs placed in plugin/scripts/ directory for hooks and worker service, plugin/skills/ for skills components | Concepts: how-it-works, what-changed | Files: plugin/ui/viewer-bundle.js, plugin/ui/viewer.html, plugin/scripts/worker-service.cjs, plugin/scripts/search-server.mjs, plugin/scripts/context-hook.js, plugin/scripts/new-hook.js, plugin/scripts/save-hook.js, plugin/scripts/summary-hook.js, plugin/scripts/cleanup-hook.js, plugin/scripts/user-message-hook.js\n\n---\nDate: 11/17/2025, 9:36:22 PM\n\n---\n\n## Twenty Specific Test Queries for Search API Coverage\n*Source: claude-mem://observation/10636*\n\n**Designed concrete test cases covering worker services, architecture decisions, troubleshooting, file tracking, and concept filtering**\n\nTwenty specific test queries were designed to comprehensively validate the unified search API functionality. The queries span all major categories: semantic queries focus on understanding system components (worker startup, SQLite FTS5, hooks, build pipeline), decision queries target architectural rationale (PM2 choice, search architecture, MCP principles), troubleshooting queries seek solutions (worker debugging, hook timeouts, migrations), file-specific queries track evolution (search-server.ts, context-hook, worker-service), concept-based queries filter by knowledge type (patterns, gotchas, discoveries), type-filtered queries segment by observation type (bugfixes, features, decisions), and timeline queries provide contextual history. Each query maps to specific API capabilities including unified search parameters (type, obs_type, concepts, files) and specialized endpoints (/api/decisions, /api/timeline, /api/how-it-works).\n\n---\nType: decision | Facts: Semantic queries test understanding of worker service startup, SQLite FTS5 implementation, hook lifecycle flow, and build pipeline process; Decision queries test retrieval of PM2 architectural choice, search architecture guidelines, and MCP as DRY source principle; Troubleshooting queries test finding bugfixes for worker service debugging, hook timeout problems, and database migration issues; File-specific queries test tracking changes to search-server.ts, context-hook modifications, and worker-service updates; Concept-based queries test filtering by pattern, gotcha, and discovery observation types; Type-filtered queries test filtering all bugfixes, features, and decisions independently; Timeline queries test contextual history retrieval around search architecture work | Concepts: pattern, how-it-works\n\n---\nDate: 11/17/2025, 9:26:45 PM\n\n---\n\n## claude-mem plugin built and deployed to marketplace\n*Source: claude-mem://observation/10605*\n\n**Complete build pipeline executed: React viewer, worker service, search server, hooks compiled and synced to Claude marketplace directory**\n\nThe claude-mem plugin underwent a complete build and deployment cycle. The build system compiled all components including the React-based viewer UI, a CommonJS worker service, an ES module search server, and six specialized hooks for different lifecycle events (context, new session, save, summary, cleanup, and user messages). All compiled outputs were placed in the plugin/scripts/ directory. The build process then used rsync to synchronize the entire project to the Claude marketplace directory at ~/.claude/plugins/marketplaces/thedotmack/, transferring thousands of files including all node_modules dependencies. This deployment makes version 6.0.9 of the plugin available through the Claude marketplace with all required dependencies installed.\n\n---\nType: change | Facts: Build process compiled 6 hooks: context-hook, new-hook, save-hook, summary-hook, cleanup-hook, user-message-hook; Worker service built as worker-service.cjs (1337.16 KB) and search server as search-server.mjs (331.02 KB); React viewer built with bundled JavaScript, HTML template, font assets, and 4 SVG icon files; Plugin synced via rsync to ~/.claude/plugins/marketplaces/thedotmack/ with 11,751 files transferred; Package version 6.0.9 deployed with dependencies auto-installed in marketplace directory | Concepts: what-changed, how-it-works | Files: scripts/build-hooks.js, package.json, plugin/scripts/worker-service.cjs, plugin/scripts/search-server.mjs, plugin/scripts/context-hook.js, plugin/scripts/new-hook.js, plugin/scripts/save-hook.js, plugin/scripts/summary-hook.js, plugin/scripts/cleanup-hook.js, plugin/scripts/user-message-hook.js, plugin/ui/viewer-bundle.js, plugin/ui/viewer.html\n\n---\nDate: 11/17/2025, 9:03:34 PM\n\n---\n\n## Postmortem documentation for failed worker service debug attempt\n*Source: claude-mem://observation/10581*\n\n**Documented debugging anti-patterns and root cause analysis failures from worker crash loop investigation.**\n\nCreated comprehensive postmortem analyzing why worker service debugging failed. The investigation identified five critical mistakes: jumping to symptoms instead of root cause, ignoring the build pipeline, attempting to fix by disabling functionality, not comparing working vs broken state early, and overcomplicating the investigation. The core issue was \"symptom chasing\" - following MCP error messages instead of asking why code that previously worked was suddenly broken. Git diff revealed worker code was unchanged from main branch, strongly suggesting a build or environment issue rather than code bug. The attempted fix violated KISS/YAGNI by adding defensive complexity (commenting out features) instead of finding the actual problem. Postmortem establishes correct debug sequence: compare working/broken states, verify build artifacts, test simplest hypothesis first, and never disable features as a \"fix\". Action items for next attempt emphasize verifying build pipeline health before modifying any source code.\n\n---\nType: decision | Facts: Worker service was in crash loop with 225 restarts failing with \"MCP error -32000: Connection closed\"; Debug attempt chased symptoms (MCP connection errors) instead of investigating root cause (why suddenly broken); Build pipeline was never investigated despite worker file location mismatch and corrupted search-server.mjs output; Git diff showed core worker code was UNCHANGED from main branch, only search-everything additions present; Attempted fix by disabling Chroma and search server features violated KISS/YAGNI principles; Likely root causes identified as corrupted build artifacts, missing node_modules, or ESM/CJS bundling issues; Correct debug sequence should start with comparing main vs current branch and verifying build artifacts | Concepts: problem-solution, gotcha, pattern, why-it-exists | Files: POSTMORTEM-worker-debug-failure.md\n\n---\nDate: 11/17/2025, 8:09:26 PM"}] |