Files
claude-mem/test-results/test-05-pm2-decision.json
T
Alex Newman c5e68a17c8 refactor: Clean up search architecture, remove experimental contextualize endpoint (#133)
* Refactor code structure for improved readability and maintainability

* Add test results for search API and related functionalities

- Created test result files for various search-related functionalities, including:
  - test-11-search-server-changes.json
  - test-12-context-hook-changes.json
  - test-13-worker-service-changes.json
  - test-14-patterns.json
  - test-15-gotchas.json
  - test-16-discoveries.json
  - test-17-all-bugfixes.json
  - test-18-all-features.json
  - test-19-all-decisions.json
  - test-20-session-search.json
  - test-21-prompt-search.json
  - test-22-decisions-endpoint.json
  - test-23-changes-endpoint.json
  - test-24-how-it-works-endpoint.json
  - test-25-contextualize-endpoint.json
  - test-26-timeline-around-observation.json
  - test-27-multi-param-combo.json
  - test-28-file-type-combo.json

- Each test result file captures specific search failures or outcomes, including issues with undefined properties and successful execution of search queries.
- Enhanced documentation of search architecture and testing strategies, ensuring compliance with established guidelines and improving overall search functionality.

* feat: Enhance unified search API with catch-all parameters and backward compatibility

- Implemented a unified search API at /api/search that accepts catch-all parameters for filtering by type, observation type, concepts, and files.
- Maintained backward compatibility by keeping granular endpoints functional while routing through the same infrastructure.
- Completed comprehensive testing of search capabilities with real-world query scenarios.

fix: Address missing debug output in search API query tests

- Flushed PM2 logs and executed search queries to verify functionality.
- Diagnosed absence of "Raw Chroma" debug messages in worker logs, indicating potential issues with logging or query processing.

refactor: Improve build and deployment pipeline for claude-mem plugin

- Successfully built and synced all hooks and services to the marketplace directory.
- Ensured all dependencies are installed and up-to-date in the deployment location.

feat: Implement hybrid search filters with 90-day recency window

- Enhanced search server to apply a 90-day recency filter to Chroma results before categorizing by document type.

fix: Correct parameter handling in searchUserPrompts method

- Added support for filter-only queries and improved dual-path logic for clarity.

refactor: Rename FTS5 method to clarify fallback status

- Renamed escapeFTS5 to escapeFTS5_fallback_when_chroma_unavailable to indicate its temporary usage.

feat: Introduce contextualize tool for comprehensive project overview

- Added a new tool to fetch recent observations, sessions, and user prompts, providing a quick project overview.

feat: Add semantic shortcut tools for common search patterns

- Implemented 'decisions', 'changes', and 'how_it_works' tools for convenient access to frequently searched observation categories.

feat: Unified timeline tool supports anchor and query modes

- Combined get_context_timeline and get_timeline_by_query into a single interface for timeline exploration.

feat: Unified search tool added to MCP server

- New tool queries all memory types simultaneously, providing combined chronological results for improved search efficiency.

* Refactor search functionality to clarify FTS5 fallback usage

- Updated `worker-service.cjs` to replace FTS5 fallback function with a more descriptive name and improved error handling.
- Enhanced documentation in `SKILL.md` to specify the unified API endpoint and clarify the behavior of the search engine, including the conditions under which FTS5 is used.
- Modified `search-server.ts` to provide clearer logging and descriptions regarding the fallback to FTS5 when UVX/Python is unavailable.
- Renamed and updated the `SessionSearch.ts` methods to reflect the conditions for using FTS5, emphasizing the lack of semantic understanding in fallback scenarios.

* feat: Add ID-based fetch endpoints and simplify mem-search skill

**Problem:**
- Search returns IDs but no way to fetch by ID
- Skill documentation was bloated with too many options
- Claude wasn't using IDs because we didn't tell it how

**Solution:**
1. Added three new HTTP endpoints:
   - GET /api/observation/:id
   - GET /api/session/:id
   - GET /api/prompt/:id

2. Completely rewrote SKILL.md:
   - Stripped complexity down to essentials
   - Clear 3-step prescriptive workflow: Search → Review IDs → Fetch by ID
   - Emphasized ID usage: "The IDs are there for a reason - USE THEM"
   - Removed confusing multi-endpoint documentation
   - Kept only unified search with filters

**Impact:**
- Token efficiency: Claude can now fetch full details only for relevant IDs
- Clarity: One clear workflow instead of 10+ options to choose from
- Usability: IDs are no longer wasted context - they're actionable

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* chore: Move internal docs to private directory

Moved POSTMORTEM and planning docs to ./private to exclude from PR reviews.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: Remove experimental contextualize endpoint

- Removed contextualize MCP tool from search-server (saves ~4KB)
- Disabled FTS5 fallback paths in SessionSearch (now vector-first)
- Cleaned up CLAUDE.md documentation
- Removed contextualize-rewrite-plan.md doc

Rationale:
- Contextualize is better suited as a skill (LLM-powered) than an endpoint
- Search API already provides vector search with configurable limits
- Created issue #132 to track future contextualize skill implementation

Changes:
- src/servers/search-server.ts: Removed contextualize tool definition
- src/services/sqlite/SessionSearch.ts: Disabled FTS5 fallback, added deprecation warnings
- CLAUDE.md: Cleaned up outdated skill documentation
- docs/: Removed contextualize plan document

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: Complete FTS5 cleanup - remove all deprecated search code

This completes the FTS5 cleanup work by removing all commented-out
FTS5 search code while preserving database tables for backward compatibility.

Changes:
- Removed 200+ lines of commented FTS5 search code from SessionSearch.ts
- Removed deprecated degraded_search_query__when_uvx_unavailable method
- Updated all method documentation to clarify vector-first architecture
- Updated class documentation to reflect filter-only query support
- Updated CLAUDE.md to remove FTS5 search references
- Clarified that FTS5 tables exist for backward compatibility only
- Updated "Why SQLite FTS5" section to "Why Vector-First Search"

Database impact: NONE - FTS5 tables remain intact for existing installations

Search architecture:
- ChromaDB: All text-based vector search queries
- SQLite: Filter-only queries (date ranges, metadata, no query text)
- FTS5 tables: Maintained but unused (backward compatibility)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: Remove all FTS5 fallback execution code from search-server

Completes the FTS5 cleanup by removing all fallback execution paths
that attempted to use FTS5 when ChromaDB was unavailable.

Changes:
- Removed all FTS5 fallback code execution paths
- When ChromaDB fails or is unavailable, return empty results with helpful error messages
- Updated all deprecated tool descriptions (search_observations, search_sessions, search_user_prompts)
- Changed error messages to indicate FTS5 fallback has been removed
- Added installation instructions for UVX/Python when vector search is unavailable
- Updated comments from "hybrid search" to "vector-first search"
- Removed ~100 lines of dead FTS5 fallback code

Database impact: NONE - FTS5 tables remain intact (backward compatibility)

Search behavior when ChromaDB unavailable:
- Text queries: Return empty results with error explaining ChromaDB is required
- Filter-only queries (no text): Continue to work via direct SQLite

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: Address PR 133 review feedback

Critical fixes:
- Remove contextualize endpoint from worker-service (route + handler)
- Fix build script logging to show correct .cjs extension (was .mjs)

Documentation improvements:
- Add comprehensive FTS5 retention rationale documentation
- Include v7.0.0 removal TODO for future cleanup

Testing:
- Build succeeds with correct output logging
- Worker restarts successfully (30th restart)
- Contextualize endpoint properly removed (404 response)
- Search endpoint verified working

This addresses all critical review feedback from PR 133.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-21 18:59:23 -05:00

1 line
5.2 KiB
JSON

[{"type":"text","text":"## Package script worker:logs includes automatic log flushing\n*Source: claude-mem://observation/10658*\n\n**The worker:logs npm script now flushes PM2 logs before displaying them automatically.**\n\nThe worker:logs npm script has been implemented with automatic log flushing as the default behavior. This addresses the recurring issue where logs were frequently forgotten to be flushed before viewing. The script now chains two PM2 commands: first flushing the claude-mem-worker logs, then displaying the last 100 lines without streaming. This automation ensures users always see fresh logs without needing to remember the manual flush step, improving the debugging workflow reliability.\n\n---\nType: feature | Facts: The worker:logs script runs 'pm2 flush claude-mem-worker' before showing logs; After flushing, the script displays the last 100 lines using 'pm2 logs claude-mem-worker --lines 100 --nostream'; PM2 flushes both stdout and stderr log files located at /Users/alexnewman/.pm2/logs/ | Concepts: how-it-works, problem-solution, pattern\n\n---\nDate: 11/17/2025, 9:34:27 PM\n\n---\n\n## PM2 Decision Query Returns No Results\n*Source: claude-mem://observation/10651*\n\n**Search for \"PM2 instead of direct process\" found no matching records in memory.**\n\nA test query searching for architectural decision documentation about \"PM2 instead of direct process\" returned no results from the memory system. The test file test-05-pm2-decision.json captured this search result, indicating that while the system may be using PM2 for process management, no observation has been recorded explaining the rationale for choosing PM2 over direct process management. This suggests either the decision predates the memory system, wasn't documented as an observation, or uses different terminology that doesn't match this search query.\n\n---\nType: discovery | Facts: Test file test-05-pm2-decision.json contains search results for query \"PM2 instead of direct process\"; Search returned \"No results found\" indicating no observations or records document this architectural decision; Query appears to target a decision about using PM2 for process management versus direct process execution | Concepts: problem-solution, gotcha | Files: test-results/test-05-pm2-decision.json\n\n---\nDate: 11/17/2025, 9:33:24 PM\n\n---\n\n## PM2 Worker Service Restarted\n*Source: claude-mem://observation/10598*\n\n**Restarted claude-mem-worker process managed by PM2 process manager**\n\nThe claude-mem-worker background service was restarted using PM2 process manager via npm script. The worker process (version 6.0.9) is now running with a fresh instance (PID 86354) and shows online status. The restart counter indicates this service has been restarted 136 times, suggesting active development or iterative deployment. The worker is running in fork mode with file watching disabled.\n\n---\nType: change | Facts: PM2 process claude-mem-worker (ID: 1) restarted successfully; Worker process running as version 6.0.9 with PID 86354; Process has been restarted 136 times total; Worker process status is online with 912.0kb memory usage | Concepts: how-it-works, what-changed\n\n---\nDate: 11/17/2025, 8:38:22 PM\n\n---\n\n## PM2 Worker Logs Flushed\n*Source: claude-mem://observation/10532*\n\n**Cleared claude-mem-worker stdout and error logs to prepare for clean debugging session.**\n\nThe PM2 logs for the claude-mem-worker process were flushed to clear accumulated debugging output. This removes old log entries from both the stdout log (claude-mem-worker-out.log) and error log (claude-mem-worker-error.log) in the ~/.pm2/logs directory. Flushing logs before starting fresh debugging ensures that subsequent testing will produce clean, unambiguous output that clearly shows whether the synchronous endpoint is being called with the wait_until_obs_is_saved parameter and whether UNIQUE constraint errors still occur with the updated implementation.\n\n---\nType: change | Facts: PM2 logs flushed for claude-mem-worker process; Flushed log files: ~/.pm2/logs/claude-mem-worker-out.log and claude-mem-worker-error.log; Log flush prepares clean slate for testing synchronous endpoint and UNIQUE constraint issues | Concepts: pattern, why-it-exists\n\n---\nDate: 11/17/2025, 7:05:00 PM\n\n---\n\n## Worker Process Restarted for Phase 3 Testing\n*Source: claude-mem://observation/10492*\n\n**claude-mem-worker restarted via PM2 to apply changes for testing phase 3**\n\nThe claude-mem-worker background process was restarted via PM2 to apply configuration or code changes for phase 3 testing. The worker is running version 6.0.9 of the claude-mem application and is managed by PM2 in fork mode with file watching enabled. The process successfully came online with a new process ID of 17916, indicating a clean restart. This restart was performed as part of initiating phase 3 testing, ensuring the worker is running with the latest changes.\n\n---\nType: change | Facts: claude-mem-worker process restarted using PM2 process manager; Worker running as version 6.0.9 with process ID 17916; Worker status is online with watching enabled in fork mode; Process has restarted 2 times total since initial deployment | Concepts: how-it-works, what-changed\n\n---\nDate: 11/17/2025, 6:44:21 PM"}]