Commit Graph

22 Commits

Author SHA1 Message Date
Alex Newman 32be34505a refactor: consolidate mem-search skill, remove desktop-skill duplication
- Delete separate desktop-skill/ directory (was outdated)
- Generate mem-search.zip during build from plugin/skills/mem-search/
- Update docs with correct MCP tool list and new download path
- Single source of truth for Claude Desktop skill

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-14 22:55:09 -05:00
Alex Newman 807d1d6100 feat: migrate scripts to Bun runtime
- Updated shebangs in user-message-hook.js, worker-cli.js, and worker-service.cjs to use Bun instead of Node.
- Modified build-hooks.js to generate Bun-compatible shebangs in built scripts.
- Enhanced sync-marketplace.cjs to trigger a worker restart after syncing files via an HTTP request.
- Improved worker-cli.ts to exit with appropriate status codes after executing commands.
- Added build-worker-binary.js to create a Windows executable for the worker service using Bun's compile feature.
2025-12-11 14:59:44 -05:00
Alex Newman 8bf22b3dc5 feat: implement worker CLI and process management for bun integration 2025-12-10 23:14:56 -05:00
Alex Newman e4bd0ae461 refactor: migrate from better-sqlite3 to bun:sqlite
- Updated build-hooks.js to remove better-sqlite3 dependency and use bun:sqlite.
- Modified smart-install.js to eliminate checks and installations related to better-sqlite3.
- Refactored Database.ts, SessionSearch.ts, SessionStore.ts, and migrations.ts to import and utilize bun:sqlite.
- Replaced exec and pragma calls with appropriate run methods for bun:sqlite compatibility.
- Removed unnecessary native module verification and installation logic for better-sqlite3.
2025-12-10 22:11:55 -05:00
kat-bell d7dc29498c fix(cache): Add package.json to plugin directory for cache dependency resolution
The bundled hook scripts use `external: ['better-sqlite3']` during esbuild,
meaning the dependency must be resolved at runtime. When hooks run from the
cache directory (~/.claude/plugins/cache/thedotmack/claude-mem/X.X.X/),
they couldn't find better-sqlite3 because:

1. Cache directory had no package.json
2. smart-install.js was hardcoded to install in marketplace directory only

This fix:
- Adds plugin/package.json declaring runtime dependencies (better-sqlite3)
- Updates build-hooks.js to auto-generate plugin/package.json from main package.json
- Updates smart-install.js to detect execution context (cache vs marketplace)
  and install dependencies in the correct location

The script now detects if it's running from cache (via path pattern matching)
and installs dependencies there, where the hooks actually execute.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-09 05:27:26 -06:00
Alex Newman c415ff5120 feat(timeline): implement TimelineService for building and formatting timeline items
- Extracted timeline-related functionality from mcp-server.ts to a dedicated TimelineService class.
- Added methods to build, filter, and format timeline items based on observations, sessions, and prompts.
- Introduced interfaces for TimelineItem and TimelineData to standardize data structures.
- Implemented sorting and grouping of timeline items by date, with markdown formatting for output.
- Included utility methods for date and time formatting, as well as token estimation.
2025-12-07 19:14:18 -05:00
Alex Newman 3aaee6f13a refactor: Organize worker into clean route-based HTTP architecture
Major architectural improvements to the worker service:

- Extracted monolithic WorkerService (~1900 lines) into organized route classes
- New HTTP layer with dedicated route handlers:
  - SessionRoutes: Session lifecycle operations
  - DataRoutes: Data retrieval endpoints
  - SearchRoutes: Search/MCP proxy operations
  - SettingsRoutes: Settings and configuration
  - ViewerRoutes: Health, UI, and SSE streaming
- Added comprehensive README documenting worker architecture
- Improved build script to handle worker service compilation
- Added context-generator for hook context operations

This is Phase 1 of worker refactoring - pure code reorganization with zero
functional changes. All existing behavior preserved while improving
maintainability and code organization.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-12-05 20:27:49 -05:00
Alex Newman c5e68a17c8 refactor: Clean up search architecture, remove experimental contextualize endpoint (#133)
* Refactor code structure for improved readability and maintainability

* Add test results for search API and related functionalities

- Created test result files for various search-related functionalities, including:
  - test-11-search-server-changes.json
  - test-12-context-hook-changes.json
  - test-13-worker-service-changes.json
  - test-14-patterns.json
  - test-15-gotchas.json
  - test-16-discoveries.json
  - test-17-all-bugfixes.json
  - test-18-all-features.json
  - test-19-all-decisions.json
  - test-20-session-search.json
  - test-21-prompt-search.json
  - test-22-decisions-endpoint.json
  - test-23-changes-endpoint.json
  - test-24-how-it-works-endpoint.json
  - test-25-contextualize-endpoint.json
  - test-26-timeline-around-observation.json
  - test-27-multi-param-combo.json
  - test-28-file-type-combo.json

- Each test result file captures specific search failures or outcomes, including issues with undefined properties and successful execution of search queries.
- Enhanced documentation of search architecture and testing strategies, ensuring compliance with established guidelines and improving overall search functionality.

* feat: Enhance unified search API with catch-all parameters and backward compatibility

- Implemented a unified search API at /api/search that accepts catch-all parameters for filtering by type, observation type, concepts, and files.
- Maintained backward compatibility by keeping granular endpoints functional while routing through the same infrastructure.
- Completed comprehensive testing of search capabilities with real-world query scenarios.

fix: Address missing debug output in search API query tests

- Flushed PM2 logs and executed search queries to verify functionality.
- Diagnosed absence of "Raw Chroma" debug messages in worker logs, indicating potential issues with logging or query processing.

refactor: Improve build and deployment pipeline for claude-mem plugin

- Successfully built and synced all hooks and services to the marketplace directory.
- Ensured all dependencies are installed and up-to-date in the deployment location.

feat: Implement hybrid search filters with 90-day recency window

- Enhanced search server to apply a 90-day recency filter to Chroma results before categorizing by document type.

fix: Correct parameter handling in searchUserPrompts method

- Added support for filter-only queries and improved dual-path logic for clarity.

refactor: Rename FTS5 method to clarify fallback status

- Renamed escapeFTS5 to escapeFTS5_fallback_when_chroma_unavailable to indicate its temporary usage.

feat: Introduce contextualize tool for comprehensive project overview

- Added a new tool to fetch recent observations, sessions, and user prompts, providing a quick project overview.

feat: Add semantic shortcut tools for common search patterns

- Implemented 'decisions', 'changes', and 'how_it_works' tools for convenient access to frequently searched observation categories.

feat: Unified timeline tool supports anchor and query modes

- Combined get_context_timeline and get_timeline_by_query into a single interface for timeline exploration.

feat: Unified search tool added to MCP server

- New tool queries all memory types simultaneously, providing combined chronological results for improved search efficiency.

* Refactor search functionality to clarify FTS5 fallback usage

- Updated `worker-service.cjs` to replace FTS5 fallback function with a more descriptive name and improved error handling.
- Enhanced documentation in `SKILL.md` to specify the unified API endpoint and clarify the behavior of the search engine, including the conditions under which FTS5 is used.
- Modified `search-server.ts` to provide clearer logging and descriptions regarding the fallback to FTS5 when UVX/Python is unavailable.
- Renamed and updated the `SessionSearch.ts` methods to reflect the conditions for using FTS5, emphasizing the lack of semantic understanding in fallback scenarios.

* feat: Add ID-based fetch endpoints and simplify mem-search skill

**Problem:**
- Search returns IDs but no way to fetch by ID
- Skill documentation was bloated with too many options
- Claude wasn't using IDs because we didn't tell it how

**Solution:**
1. Added three new HTTP endpoints:
   - GET /api/observation/:id
   - GET /api/session/:id
   - GET /api/prompt/:id

2. Completely rewrote SKILL.md:
   - Stripped complexity down to essentials
   - Clear 3-step prescriptive workflow: Search → Review IDs → Fetch by ID
   - Emphasized ID usage: "The IDs are there for a reason - USE THEM"
   - Removed confusing multi-endpoint documentation
   - Kept only unified search with filters

**Impact:**
- Token efficiency: Claude can now fetch full details only for relevant IDs
- Clarity: One clear workflow instead of 10+ options to choose from
- Usability: IDs are no longer wasted context - they're actionable

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* chore: Move internal docs to private directory

Moved POSTMORTEM and planning docs to ./private to exclude from PR reviews.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: Remove experimental contextualize endpoint

- Removed contextualize MCP tool from search-server (saves ~4KB)
- Disabled FTS5 fallback paths in SessionSearch (now vector-first)
- Cleaned up CLAUDE.md documentation
- Removed contextualize-rewrite-plan.md doc

Rationale:
- Contextualize is better suited as a skill (LLM-powered) than an endpoint
- Search API already provides vector search with configurable limits
- Created issue #132 to track future contextualize skill implementation

Changes:
- src/servers/search-server.ts: Removed contextualize tool definition
- src/services/sqlite/SessionSearch.ts: Disabled FTS5 fallback, added deprecation warnings
- CLAUDE.md: Cleaned up outdated skill documentation
- docs/: Removed contextualize plan document

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: Complete FTS5 cleanup - remove all deprecated search code

This completes the FTS5 cleanup work by removing all commented-out
FTS5 search code while preserving database tables for backward compatibility.

Changes:
- Removed 200+ lines of commented FTS5 search code from SessionSearch.ts
- Removed deprecated degraded_search_query__when_uvx_unavailable method
- Updated all method documentation to clarify vector-first architecture
- Updated class documentation to reflect filter-only query support
- Updated CLAUDE.md to remove FTS5 search references
- Clarified that FTS5 tables exist for backward compatibility only
- Updated "Why SQLite FTS5" section to "Why Vector-First Search"

Database impact: NONE - FTS5 tables remain intact for existing installations

Search architecture:
- ChromaDB: All text-based vector search queries
- SQLite: Filter-only queries (date ranges, metadata, no query text)
- FTS5 tables: Maintained but unused (backward compatibility)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: Remove all FTS5 fallback execution code from search-server

Completes the FTS5 cleanup by removing all fallback execution paths
that attempted to use FTS5 when ChromaDB was unavailable.

Changes:
- Removed all FTS5 fallback code execution paths
- When ChromaDB fails or is unavailable, return empty results with helpful error messages
- Updated all deprecated tool descriptions (search_observations, search_sessions, search_user_prompts)
- Changed error messages to indicate FTS5 fallback has been removed
- Added installation instructions for UVX/Python when vector search is unavailable
- Updated comments from "hybrid search" to "vector-first search"
- Removed ~100 lines of dead FTS5 fallback code

Database impact: NONE - FTS5 tables remain intact (backward compatibility)

Search behavior when ChromaDB unavailable:
- Text queries: Return empty results with error explaining ChromaDB is required
- Filter-only queries (no text): Continue to work via direct SQLite

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: Address PR 133 review feedback

Critical fixes:
- Remove contextualize endpoint from worker-service (route + handler)
- Fix build script logging to show correct .cjs extension (was .mjs)

Documentation improvements:
- Add comprehensive FTS5 retention rationale documentation
- Include v7.0.0 removal TODO for future cleanup

Testing:
- Build succeeds with correct output logging
- Worker restarts successfully (30th restart)
- Contextualize endpoint properly removed (404 response)
- Search endpoint verified working

This addresses all critical review feedback from PR 133.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-21 18:59:23 -05:00
Alex Newman ca4f046777 feat: Add search skill with progressive disclosure and refactor existing skills
Enhancements:
- Added search skill with 10 HTTP API endpoints for memory queries
- Refactored version-bump and troubleshoot skills using progressive disclosure pattern
- Added operations/ subdirectories for detailed skill documentation
- Updated CLAUDE.md with skill-based search architecture
- Enhanced worker service with search API endpoints
- Updated CHANGELOG.md with v5.4.0 migration details

Technical changes:
- New plugin/skills/search/ directory with SKILL.md
- New .claude/skills/version-bump/operations/ (workflow.md, scenarios.md)
- New plugin/skills/troubleshoot/operations/ (common-issues.md, worker.md)
- Modified src/services/worker-service.ts (added search endpoints)
- Modified plugin/scripts/worker-service.cjs (rebuilt with search API)
- Reduced main skill files by 89% using progressive disclosure
- Token savings: ~2,250 tokens per session start

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-09 18:41:53 -05:00
Alex Newman 4bc467f7ed feat: Implement Worker Service for long-running HTTP service with PM2 management
- Introduced WorkerService class to handle HTTP requests and manage sessions.
- Added endpoints for health check, session management, and data retrieval.
- Integrated ChromaSync for background data synchronization.
- Implemented SSE for real-time updates to connected clients.
- Added error handling and logging throughout the service.
- Cached Claude executable path for improved performance.
- Included settings management for user configuration.
- Established database interactions for session and observation management.
2025-11-07 13:26:13 -05:00
Alex Newman 79ff1849f0 feat: Add web-based viewer UI for real-time memory stream (#58)
* Add viewer HTML for claude-mem with live stream and settings interface

- Implemented a responsive layout with left and right columns for observations and settings.
- Added status indicators for connection state.
- Integrated server-sent events (SSE) for real-time updates on observations and summaries.
- Created dynamic project filter dropdown based on available observations.
- Developed settings section for environment variables and worker stats.
- Included functionality to save settings and load current stats from the server.
- Enhanced UI with custom styles for better user experience.

* Remove draft implementation plan for v5.1 web UI

* feat: Implement viewer UI with sidebar, feed, and settings management

- Add main viewer template (HTML) with styling for dark mode.
- Create App component to manage state and render Header, Feed, and Sidebar.
- Implement Feed component to display observations and summaries with filtering.
- Develop Header component for project selection and connection status.
- Create ObservationCard and SummaryCard components for displaying individual items.
- Implement Sidebar for settings management and displaying worker/database stats.
- Add hooks for managing SSE connections, settings, and stats fetching.
- Define types for observations, summaries, settings, and stats.

* Enhance UI components and improve layout

- Updated padding and layout for the feed and card components in viewer.html, viewer-template.html, and viewer.html to improve visual spacing and alignment.
- Increased card margins and padding for better readability and aesthetics.
- Adjusted font sizes, weights, and line heights for card titles and subtitles to enhance text clarity and hierarchy.
- Added a new feed-content class to center the feed items and limit their maximum width.
- Modified the Header component to improve the settings icon's SVG structure for better rendering.
- Enhanced the Sidebar component by adding a close button with an SVG icon, improving user experience for closing settings.
- Updated the Sidebar component's props to include an onClose function for handling sidebar closure.

* feat: Add user prompts feature with UI integration

- Implemented a new method in SessionStore to retrieve recent user prompts.
- Updated WorkerService to fetch and broadcast user prompts to clients.
- Enhanced the Feed component to display user prompts alongside observations and summaries.
- Created a new PromptCard component for rendering individual user prompts.
- Modified useSSE hook to handle new prompt events and processing status.
- Updated viewer templates and styles to accommodate the new prompts feature.

* feat: Add project filtering and pagination for observations

- Implemented `getAllProjects` method in `SessionStore` to retrieve unique projects from the database.
- Added `/api/observations` endpoint in `WorkerService` for paginated observations fetching.
- Enhanced `App` component to manage paginated observations and integrate with the new API.
- Updated `Feed` component to support infinite scrolling and loading more observations.
- Modified `Header` to display processing status.
- Refactored `PromptCard` to remove unnecessary processing indicator.
- Introduced `usePagination` hook to handle pagination logic for observations.
- Updated `useSSE` hook to include projects in the state.
- Adjusted types to accommodate new project data.

* Refactor viewer build process and remove deprecated HTML template

- Updated build-viewer.js to copy HTML template to build output with improved logging.
- Removed src/ui/viewer.html as it is no longer needed.
- Enhanced App component to merge observations while removing duplicates using useMemo.
- Improved Feed component to utilize a ref for onLoadMore callback and adjusted infinite scroll logic.
- Updated Sidebar component to use default settings from constants and removed redundant formatting functions.
- Refactored usePagination hook to streamline loading logic and prevent concurrent requests.
- Updated useSSE hook to use centralized API endpoints and improved reconnection logic.
- Refactored useSettings and useStats hooks to utilize constants for API endpoints and timing.
- Introduced ErrorBoundary component for better error handling in the viewer.
- Centralized API endpoint paths, default settings, timing constants, and UI-related constants into dedicated files.
- Added utility functions for formatting uptime and bytes for consistent display across components.

* feat: Enhance session management and pagination for user prompts, summaries, and observations

- Added project field to user prompts in the database and API responses.
- Implemented new API endpoints for fetching summaries and prompts with pagination.
- Updated WorkerService to handle new endpoints and filter results by project.
- Modified App component to manage paginated data for prompts and summaries.
- Refactored Feed component to remove unnecessary filtering and handle combined data.
- Improved usePagination hook to support multiple data types and project filtering.
- Adjusted useSSE hook to only load projects initially, with data fetched via pagination.
- Updated types to include project information for user prompts.

* feat: add SummarySkeleton component and data utility for merging items

- Introduced SummarySkeleton component for displaying loading state in the UI.
- Implemented mergeAndDeduplicateByProject utility function to merge real-time and paginated data while removing duplicates based on project filtering.

* Enhance UI and functionality of the viewer component

- Updated sidebar transition effects to use translate3d for improved performance.
- Added a sidebar header with title and connection status indicators.
- Modified the PromptCard to display project name instead of prompt number.
- Introduced a GitHub and X (Twitter) link in the header for easy access.
- Improved styling for setting descriptions and card hover effects.
- Enhanced Sidebar component to include connection status and updated layout.

* fix: reduce timeout for worker health checks and ensure proper responsiveness
2025-11-05 22:54:38 -05:00
Alex Newman 9a9b00c6d8 Implement hybrid search server with Chroma + SQLite
- Built search-server.mjs successfully (55KB)
- Configured with packages: 'external' to use node_modules dependencies
- MCP config points to ${CLAUDE_PLUGIN_ROOT}/scripts/search-server.mjs
- Ready for deployment to plugin directory

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-31 23:35:44 -04:00
Alex Newman 15c55a57a3 Rename stderr-test-hook to user-message-hook for production
Changes:
- Renamed src/hooks/stderr-test-hook.ts to user-message-hook.ts
- Updated user-message-hook with production-ready messaging
- Updated scripts/build-hooks.js to build user-message-hook
- Updated plugin/hooks/hooks.json to reference user-message-hook.js
- Cleaned up old stderr-test-hook.js files
- Built and deployed user-message-hook.js to plugin directory

This hook displays context information to users via stderr, which is
currently the only way to show messages in Claude Code UI. It runs in
parallel with context-hook during SessionStart.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-26 23:14:19 -04:00
Alex Newman b0fae0cfd4 Add stderr test hook for UI experiment 2025-10-26 22:29:43 -04:00
Alex Newman e9c0ec45db Consolidate hooks: merge bin/hooks and hooks into single hooks/ directory
- Removed bin/hooks/ wrapper layer
- Moved all hook logic into consolidated hooks/*-hook.ts files
- Each hook now handles its own stdin/stdout/JSON wrapping
- Removed ALL try-catch blocks from context-hook (let errors surface)
- Updated build script to reference new src/hooks/ paths
- Reduced from 12+ files to 6 hook files

This simplifies the architecture and makes debugging actually possible.
2025-10-25 23:59:43 -04:00
Alex Newman 6fb9570291 feat: Release v4.0.5 - Auto-install dependencies via bootstrap hooks
Fixed better-sqlite3 distribution by implementing self-bootstrapping hooks
that auto-install dependencies on first run. This eliminates the need for
users to have native compilation tools or manually install dependencies.

## Solution

Instead of bundling 25MB of better-sqlite3 binaries in git or requiring
manual npm install, hooks now bootstrap themselves on first execution:

1. Created `src/shared/bootstrap.ts` with `ensureDependencies()` function
2. Added bootstrap calls to all hook entry points (context, new, save, summary, cleanup)
3. Created `plugin/scripts/package.json` declaring better-sqlite3 dependency
4. Bootstrap checks if `node_modules` exists, runs `npm install` if missing
5. npm automatically downloads prebuilt better-sqlite3 binary for user's platform

## Changes

**Core Bootstrap System:**
- Added src/shared/bootstrap.ts: Auto-install dependencies using npm
- Modified all hooks (context, new, save, summary, cleanup) to call ensureDependencies()
- Created plugin/scripts/package.json with better-sqlite3 dependency

**Build & Distribution:**
- Removed node_modules copying logic from build script
- Build output is now compact (hooks + package.json, no binaries)
- Updated marketplace.json to point to GitHub for direct installation

**Documentation:**
- Updated README: GitHub Marketplace installation is now recommended method
- Installation instructions emphasize no compilation needed
- Version bumped to 4.0.5 throughout

## Benefits

-  No git bloat (repo stays small, no 25MB binaries committed)
-  No compilation needed (npm downloads prebuilt binaries)
-  Works on all platforms (npm handles platform-specific binaries)
-  Zero manual steps (hooks bootstrap themselves automatically)
-  Idempotent (skips install if dependencies already exist)

Installation now works via simple:
```
/plugin marketplace add https://github.com/thedotmack/claude-mem
/plugin install claude-mem
```

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-20 16:44:26 -04:00
Alex Newman daf368e343 Refactor code structure for improved readability and maintainability 2025-10-19 01:45:51 -04:00
Alex Newman 56167c47a2 feat: Implement Claude-mem MCP Search Server with session and observation search capabilities
- Added search functionality for observations and sessions using full-text search.
- Implemented formatting functions for search results with citations.
- Created multiple tools for searching by various criteria including concept, file, type, and advanced search.
- Integrated structured filters and pagination options for search queries.
- Established error handling for search operations and server initialization.
2025-10-18 20:45:41 -04:00
Alex Newman 372854948c feat: Implement Worker Service with session management and SDK integration
- Added WorkerService to handle long-running HTTP service with session management.
- Implemented endpoints for initializing, observing, finalizing, checking status, and deleting sessions.
- Integrated with Claude SDK for processing observations and generating responses.
- Added port allocator utility to dynamically find available ports for the service.
- Configured TypeScript settings for the project.
2025-10-17 15:59:36 -04:00
Alex Newman ec79e085b2 Refactor code structure for improved readability and maintainability 2025-10-16 21:02:56 -04:00
Alex Newman 3e617a8b1e Refactor code structure for improved readability and maintainability 2025-10-16 19:50:24 -04:00
Alex Newman 834cf4095e Add standalone hook entry points for context, new, save, summary, and worker
- Implemented context-hook.ts for handling session start events.
- Created new-hook.ts for user prompt submission events.
- Developed save-hook.ts for post tool use events.
- Added summary-hook.ts for handling stop events.
- Introduced worker.ts as a standalone background process for the SDK agent.
- Each hook reads input from stdin, processes it, and handles errors gracefully.
2025-10-16 15:39:30 -04:00