Files
claude-mem/test-results/test-03-hook-lifecycle-flow.json
T
Alex Newman c5e68a17c8 refactor: Clean up search architecture, remove experimental contextualize endpoint (#133)
* Refactor code structure for improved readability and maintainability

* Add test results for search API and related functionalities

- Created test result files for various search-related functionalities, including:
  - test-11-search-server-changes.json
  - test-12-context-hook-changes.json
  - test-13-worker-service-changes.json
  - test-14-patterns.json
  - test-15-gotchas.json
  - test-16-discoveries.json
  - test-17-all-bugfixes.json
  - test-18-all-features.json
  - test-19-all-decisions.json
  - test-20-session-search.json
  - test-21-prompt-search.json
  - test-22-decisions-endpoint.json
  - test-23-changes-endpoint.json
  - test-24-how-it-works-endpoint.json
  - test-25-contextualize-endpoint.json
  - test-26-timeline-around-observation.json
  - test-27-multi-param-combo.json
  - test-28-file-type-combo.json

- Each test result file captures specific search failures or outcomes, including issues with undefined properties and successful execution of search queries.
- Enhanced documentation of search architecture and testing strategies, ensuring compliance with established guidelines and improving overall search functionality.

* feat: Enhance unified search API with catch-all parameters and backward compatibility

- Implemented a unified search API at /api/search that accepts catch-all parameters for filtering by type, observation type, concepts, and files.
- Maintained backward compatibility by keeping granular endpoints functional while routing through the same infrastructure.
- Completed comprehensive testing of search capabilities with real-world query scenarios.

fix: Address missing debug output in search API query tests

- Flushed PM2 logs and executed search queries to verify functionality.
- Diagnosed absence of "Raw Chroma" debug messages in worker logs, indicating potential issues with logging or query processing.

refactor: Improve build and deployment pipeline for claude-mem plugin

- Successfully built and synced all hooks and services to the marketplace directory.
- Ensured all dependencies are installed and up-to-date in the deployment location.

feat: Implement hybrid search filters with 90-day recency window

- Enhanced search server to apply a 90-day recency filter to Chroma results before categorizing by document type.

fix: Correct parameter handling in searchUserPrompts method

- Added support for filter-only queries and improved dual-path logic for clarity.

refactor: Rename FTS5 method to clarify fallback status

- Renamed escapeFTS5 to escapeFTS5_fallback_when_chroma_unavailable to indicate its temporary usage.

feat: Introduce contextualize tool for comprehensive project overview

- Added a new tool to fetch recent observations, sessions, and user prompts, providing a quick project overview.

feat: Add semantic shortcut tools for common search patterns

- Implemented 'decisions', 'changes', and 'how_it_works' tools for convenient access to frequently searched observation categories.

feat: Unified timeline tool supports anchor and query modes

- Combined get_context_timeline and get_timeline_by_query into a single interface for timeline exploration.

feat: Unified search tool added to MCP server

- New tool queries all memory types simultaneously, providing combined chronological results for improved search efficiency.

* Refactor search functionality to clarify FTS5 fallback usage

- Updated `worker-service.cjs` to replace FTS5 fallback function with a more descriptive name and improved error handling.
- Enhanced documentation in `SKILL.md` to specify the unified API endpoint and clarify the behavior of the search engine, including the conditions under which FTS5 is used.
- Modified `search-server.ts` to provide clearer logging and descriptions regarding the fallback to FTS5 when UVX/Python is unavailable.
- Renamed and updated the `SessionSearch.ts` methods to reflect the conditions for using FTS5, emphasizing the lack of semantic understanding in fallback scenarios.

* feat: Add ID-based fetch endpoints and simplify mem-search skill

**Problem:**
- Search returns IDs but no way to fetch by ID
- Skill documentation was bloated with too many options
- Claude wasn't using IDs because we didn't tell it how

**Solution:**
1. Added three new HTTP endpoints:
   - GET /api/observation/:id
   - GET /api/session/:id
   - GET /api/prompt/:id

2. Completely rewrote SKILL.md:
   - Stripped complexity down to essentials
   - Clear 3-step prescriptive workflow: Search → Review IDs → Fetch by ID
   - Emphasized ID usage: "The IDs are there for a reason - USE THEM"
   - Removed confusing multi-endpoint documentation
   - Kept only unified search with filters

**Impact:**
- Token efficiency: Claude can now fetch full details only for relevant IDs
- Clarity: One clear workflow instead of 10+ options to choose from
- Usability: IDs are no longer wasted context - they're actionable

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* chore: Move internal docs to private directory

Moved POSTMORTEM and planning docs to ./private to exclude from PR reviews.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: Remove experimental contextualize endpoint

- Removed contextualize MCP tool from search-server (saves ~4KB)
- Disabled FTS5 fallback paths in SessionSearch (now vector-first)
- Cleaned up CLAUDE.md documentation
- Removed contextualize-rewrite-plan.md doc

Rationale:
- Contextualize is better suited as a skill (LLM-powered) than an endpoint
- Search API already provides vector search with configurable limits
- Created issue #132 to track future contextualize skill implementation

Changes:
- src/servers/search-server.ts: Removed contextualize tool definition
- src/services/sqlite/SessionSearch.ts: Disabled FTS5 fallback, added deprecation warnings
- CLAUDE.md: Cleaned up outdated skill documentation
- docs/: Removed contextualize plan document

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: Complete FTS5 cleanup - remove all deprecated search code

This completes the FTS5 cleanup work by removing all commented-out
FTS5 search code while preserving database tables for backward compatibility.

Changes:
- Removed 200+ lines of commented FTS5 search code from SessionSearch.ts
- Removed deprecated degraded_search_query__when_uvx_unavailable method
- Updated all method documentation to clarify vector-first architecture
- Updated class documentation to reflect filter-only query support
- Updated CLAUDE.md to remove FTS5 search references
- Clarified that FTS5 tables exist for backward compatibility only
- Updated "Why SQLite FTS5" section to "Why Vector-First Search"

Database impact: NONE - FTS5 tables remain intact for existing installations

Search architecture:
- ChromaDB: All text-based vector search queries
- SQLite: Filter-only queries (date ranges, metadata, no query text)
- FTS5 tables: Maintained but unused (backward compatibility)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: Remove all FTS5 fallback execution code from search-server

Completes the FTS5 cleanup by removing all fallback execution paths
that attempted to use FTS5 when ChromaDB was unavailable.

Changes:
- Removed all FTS5 fallback code execution paths
- When ChromaDB fails or is unavailable, return empty results with helpful error messages
- Updated all deprecated tool descriptions (search_observations, search_sessions, search_user_prompts)
- Changed error messages to indicate FTS5 fallback has been removed
- Added installation instructions for UVX/Python when vector search is unavailable
- Updated comments from "hybrid search" to "vector-first search"
- Removed ~100 lines of dead FTS5 fallback code

Database impact: NONE - FTS5 tables remain intact (backward compatibility)

Search behavior when ChromaDB unavailable:
- Text queries: Return empty results with error explaining ChromaDB is required
- Filter-only queries (no text): Continue to work via direct SQLite

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: Address PR 133 review feedback

Critical fixes:
- Remove contextualize endpoint from worker-service (route + handler)
- Fix build script logging to show correct .cjs extension (was .mjs)

Documentation improvements:
- Add comprehensive FTS5 retention rationale documentation
- Include v7.0.0 removal TODO for future cleanup

Testing:
- Build succeeds with correct output logging
- Worker restarts successfully (30th restart)
- Contextualize endpoint properly removed (404 response)
- Search endpoint verified working

This addresses all critical review feedback from PR 133.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-21 18:59:23 -05:00

1 line
7.4 KiB
JSON

[{"type":"text","text":"## claude-mem plugin built and deployed to marketplace\n*Source: claude-mem://observation/10605*\n\n**Complete build pipeline executed: React viewer, worker service, search server, hooks compiled and synced to Claude marketplace directory**\n\nThe claude-mem plugin underwent a complete build and deployment cycle. The build system compiled all components including the React-based viewer UI, a CommonJS worker service, an ES module search server, and six specialized hooks for different lifecycle events (context, new session, save, summary, cleanup, and user messages). All compiled outputs were placed in the plugin/scripts/ directory. The build process then used rsync to synchronize the entire project to the Claude marketplace directory at ~/.claude/plugins/marketplaces/thedotmack/, transferring thousands of files including all node_modules dependencies. This deployment makes version 6.0.9 of the plugin available through the Claude marketplace with all required dependencies installed.\n\n---\nType: change | Facts: Build process compiled 6 hooks: context-hook, new-hook, save-hook, summary-hook, cleanup-hook, user-message-hook; Worker service built as worker-service.cjs (1337.16 KB) and search server as search-server.mjs (331.02 KB); React viewer built with bundled JavaScript, HTML template, font assets, and 4 SVG icon files; Plugin synced via rsync to ~/.claude/plugins/marketplaces/thedotmack/ with 11,751 files transferred; Package version 6.0.9 deployed with dependencies auto-installed in marketplace directory | Concepts: what-changed, how-it-works | Files: scripts/build-hooks.js, package.json, plugin/scripts/worker-service.cjs, plugin/scripts/search-server.mjs, plugin/scripts/context-hook.js, plugin/scripts/new-hook.js, plugin/scripts/save-hook.js, plugin/scripts/summary-hook.js, plugin/scripts/cleanup-hook.js, plugin/scripts/user-message-hook.js, plugin/ui/viewer-bundle.js, plugin/ui/viewer.html\n\n---\nDate: 11/17/2025, 9:03:34 PM\n\n---\n\n## Session Context Review for Endless Mode Phase 3\n*Source: claude-mem://observation/10554*\n\n**Reviewing progress on Phase 3 endless mode implementation with PreToolUse hook and SKIP_TOOLS optimization.**\n\nA context review was initiated to understand the current state of the endless mode implementation. Phase 3 work includes a PreToolUse hook with SKIP_TOOLS optimization that was created in a previous session (#S1937) but remains unbuilt and untested. This discovery phase aims to gather full observations from prior work to determine the next steps needed to complete the endless mode feature.\n\n---\nType: discovery | Facts: Session #S1937 previously created a PreToolUse hook with SKIP_TOOLS optimization; The PreToolUse hook has not yet been built or tested; Work is currently in Phase 3 of an endless mode implementation; Project located at /Users/alexnewman/Scripts/claude-mem | Concepts: how-it-works, what-changed\n\n---\nDate: 11/17/2025, 7:30:21 PM\n\n---\n\n## Added PreToolUse hook to lifecycle configuration\n*Source: claude-mem://observation/10541*\n\n**New PreToolUse hook executes pre-tool-use-hook.js before any tool is invoked by Claude.**\n\nA new PreToolUse lifecycle hook was added to the claude-mem plugin configuration to enable capturing state before tools are executed. This complements the existing PostToolUse hook that runs save-hook.js after tool execution. By adding pre-tool-use-hook.js with the same wildcard matcher and timeout settings as PostToolUse, the system can now monitor and potentially intercept or prepare for tool invocations before they occur. This creates a complete before/after hook pattern for tool usage tracking.\n\n---\nType: feature | Facts: Added PreToolUse hook section to plugin/hooks/hooks.json before the existing PostToolUse hook; PreToolUse hook runs pre-tool-use-hook.js script with 120s timeout; PreToolUse hook uses matcher \"*\" to trigger on all tool invocations; The hook system now captures both before and after states of tool usage | Concepts: what-changed, pattern, why-it-exists | Files: plugin/hooks/hooks.json\n\n---\nDate: 11/17/2025, 7:16:47 PM\n\n---\n\n## PreToolUse Hook Implementation for Transcript Transformation\n*Source: claude-mem://observation/10540*\n\n**Created hook that transforms previous tool results into observation references before each tool execution.**\n\nA new PreToolUse hook was implemented to handle the transformation of tool results into observation references in the transcript. The hook executes before each tool use, ensuring that the previous tool's result has already been written to the transcript file. It searches the transcript backwards to find the most recent tool_result that hasn't been transformed yet (identified by not containing \"[Observation #\"). When found, it sends a synchronous POST request to the worker's transform endpoint with the tool_use_id. If the worker successfully creates an observation, the hook modifies the transcript file in-place, replacing the full tool result content with a compact reference like \"[Observation #123: Title]\". The implementation elegantly handles edge cases: first tool use (no previous tool), already-transformed results, disabled endless mode, and worker failures—all by simply returning success and continuing. This design fulfills the architectural decision to consolidate logic in PreToolUse while avoiding YAGNI violations.\n\n---\nType: feature | Facts: Pre-tool-use hook processes the PREVIOUS tool's result, guaranteeing it's already in transcript; Hook searches transcript backwards to find most recent untransformed tool_result entry; Transformed tool_result content becomes compact reference format: \"[Observation #ID: Title]\"; Hook communicates with worker via POST to /sessions/{id}/observations/transform endpoint; SKIP_TOOLS set filters out low-value tools: ListMcpResourcesTool, SlashCommand, Skill, TodoWrite, AskUserQuestion; Hook gracefully handles first tool use case by checking if previousTool is null; Transformation only occurs when EndlessModeConfig.enabled is true and transcript_path exists; Worker returns ObservationEndpointResponse with status: queued, completed, or timeout | Concepts: how-it-works, pattern, problem-solution | Files: src/hooks/pre-tool-use-hook.ts\n\n---\nDate: 11/17/2025, 7:16:07 PM\n\n---\n\n## Moving All Logic to PreToolUse Hook\n*Source: claude-mem://observation/10538*\n\n**Decision to implement all functionality in PreToolUse hook to guarantee previous tool use is available.**\n\nA new architectural decision was made to consolidate all functionality into the PreToolUse hook. This approach was chosen because PreToolUse guarantees that the previous tool use will be available in the context, which is essential for the intended functionality. The decision explicitly avoids over-engineering (YAGNI - You Aren't Gonna Need It) while acknowledging that the first tool use scenario may require smart handling since there would be no previous tool use at that point. This represents a simplification of the architecture by using a single hook point rather than distributing logic across multiple hooks.\n\n---\nType: decision | Facts: All logic will be moved to the PreToolUse hook instead of other lifecycle hooks; PreToolUse hook guarantees that the previous tool use is in context; The approach avoids YAGNI (You Aren't Gonna Need It) principle violations; Special handling may be needed for the first tool use edge case | Concepts: pattern, why-it-exists, trade-off\n\n---\nDate: 11/17/2025, 7:13:35 PM"}]