Files
claude-mem/plugin/skills/mem-search/operations/observations.md
T
Alex Newman c5e68a17c8 refactor: Clean up search architecture, remove experimental contextualize endpoint (#133)
* Refactor code structure for improved readability and maintainability

* Add test results for search API and related functionalities

- Created test result files for various search-related functionalities, including:
  - test-11-search-server-changes.json
  - test-12-context-hook-changes.json
  - test-13-worker-service-changes.json
  - test-14-patterns.json
  - test-15-gotchas.json
  - test-16-discoveries.json
  - test-17-all-bugfixes.json
  - test-18-all-features.json
  - test-19-all-decisions.json
  - test-20-session-search.json
  - test-21-prompt-search.json
  - test-22-decisions-endpoint.json
  - test-23-changes-endpoint.json
  - test-24-how-it-works-endpoint.json
  - test-25-contextualize-endpoint.json
  - test-26-timeline-around-observation.json
  - test-27-multi-param-combo.json
  - test-28-file-type-combo.json

- Each test result file captures specific search failures or outcomes, including issues with undefined properties and successful execution of search queries.
- Enhanced documentation of search architecture and testing strategies, ensuring compliance with established guidelines and improving overall search functionality.

* feat: Enhance unified search API with catch-all parameters and backward compatibility

- Implemented a unified search API at /api/search that accepts catch-all parameters for filtering by type, observation type, concepts, and files.
- Maintained backward compatibility by keeping granular endpoints functional while routing through the same infrastructure.
- Completed comprehensive testing of search capabilities with real-world query scenarios.

fix: Address missing debug output in search API query tests

- Flushed PM2 logs and executed search queries to verify functionality.
- Diagnosed absence of "Raw Chroma" debug messages in worker logs, indicating potential issues with logging or query processing.

refactor: Improve build and deployment pipeline for claude-mem plugin

- Successfully built and synced all hooks and services to the marketplace directory.
- Ensured all dependencies are installed and up-to-date in the deployment location.

feat: Implement hybrid search filters with 90-day recency window

- Enhanced search server to apply a 90-day recency filter to Chroma results before categorizing by document type.

fix: Correct parameter handling in searchUserPrompts method

- Added support for filter-only queries and improved dual-path logic for clarity.

refactor: Rename FTS5 method to clarify fallback status

- Renamed escapeFTS5 to escapeFTS5_fallback_when_chroma_unavailable to indicate its temporary usage.

feat: Introduce contextualize tool for comprehensive project overview

- Added a new tool to fetch recent observations, sessions, and user prompts, providing a quick project overview.

feat: Add semantic shortcut tools for common search patterns

- Implemented 'decisions', 'changes', and 'how_it_works' tools for convenient access to frequently searched observation categories.

feat: Unified timeline tool supports anchor and query modes

- Combined get_context_timeline and get_timeline_by_query into a single interface for timeline exploration.

feat: Unified search tool added to MCP server

- New tool queries all memory types simultaneously, providing combined chronological results for improved search efficiency.

* Refactor search functionality to clarify FTS5 fallback usage

- Updated `worker-service.cjs` to replace FTS5 fallback function with a more descriptive name and improved error handling.
- Enhanced documentation in `SKILL.md` to specify the unified API endpoint and clarify the behavior of the search engine, including the conditions under which FTS5 is used.
- Modified `search-server.ts` to provide clearer logging and descriptions regarding the fallback to FTS5 when UVX/Python is unavailable.
- Renamed and updated the `SessionSearch.ts` methods to reflect the conditions for using FTS5, emphasizing the lack of semantic understanding in fallback scenarios.

* feat: Add ID-based fetch endpoints and simplify mem-search skill

**Problem:**
- Search returns IDs but no way to fetch by ID
- Skill documentation was bloated with too many options
- Claude wasn't using IDs because we didn't tell it how

**Solution:**
1. Added three new HTTP endpoints:
   - GET /api/observation/:id
   - GET /api/session/:id
   - GET /api/prompt/:id

2. Completely rewrote SKILL.md:
   - Stripped complexity down to essentials
   - Clear 3-step prescriptive workflow: Search → Review IDs → Fetch by ID
   - Emphasized ID usage: "The IDs are there for a reason - USE THEM"
   - Removed confusing multi-endpoint documentation
   - Kept only unified search with filters

**Impact:**
- Token efficiency: Claude can now fetch full details only for relevant IDs
- Clarity: One clear workflow instead of 10+ options to choose from
- Usability: IDs are no longer wasted context - they're actionable

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* chore: Move internal docs to private directory

Moved POSTMORTEM and planning docs to ./private to exclude from PR reviews.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: Remove experimental contextualize endpoint

- Removed contextualize MCP tool from search-server (saves ~4KB)
- Disabled FTS5 fallback paths in SessionSearch (now vector-first)
- Cleaned up CLAUDE.md documentation
- Removed contextualize-rewrite-plan.md doc

Rationale:
- Contextualize is better suited as a skill (LLM-powered) than an endpoint
- Search API already provides vector search with configurable limits
- Created issue #132 to track future contextualize skill implementation

Changes:
- src/servers/search-server.ts: Removed contextualize tool definition
- src/services/sqlite/SessionSearch.ts: Disabled FTS5 fallback, added deprecation warnings
- CLAUDE.md: Cleaned up outdated skill documentation
- docs/: Removed contextualize plan document

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: Complete FTS5 cleanup - remove all deprecated search code

This completes the FTS5 cleanup work by removing all commented-out
FTS5 search code while preserving database tables for backward compatibility.

Changes:
- Removed 200+ lines of commented FTS5 search code from SessionSearch.ts
- Removed deprecated degraded_search_query__when_uvx_unavailable method
- Updated all method documentation to clarify vector-first architecture
- Updated class documentation to reflect filter-only query support
- Updated CLAUDE.md to remove FTS5 search references
- Clarified that FTS5 tables exist for backward compatibility only
- Updated "Why SQLite FTS5" section to "Why Vector-First Search"

Database impact: NONE - FTS5 tables remain intact for existing installations

Search architecture:
- ChromaDB: All text-based vector search queries
- SQLite: Filter-only queries (date ranges, metadata, no query text)
- FTS5 tables: Maintained but unused (backward compatibility)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: Remove all FTS5 fallback execution code from search-server

Completes the FTS5 cleanup by removing all fallback execution paths
that attempted to use FTS5 when ChromaDB was unavailable.

Changes:
- Removed all FTS5 fallback code execution paths
- When ChromaDB fails or is unavailable, return empty results with helpful error messages
- Updated all deprecated tool descriptions (search_observations, search_sessions, search_user_prompts)
- Changed error messages to indicate FTS5 fallback has been removed
- Added installation instructions for UVX/Python when vector search is unavailable
- Updated comments from "hybrid search" to "vector-first search"
- Removed ~100 lines of dead FTS5 fallback code

Database impact: NONE - FTS5 tables remain intact (backward compatibility)

Search behavior when ChromaDB unavailable:
- Text queries: Return empty results with error explaining ChromaDB is required
- Filter-only queries (no text): Continue to work via direct SQLite

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: Address PR 133 review feedback

Critical fixes:
- Remove contextualize endpoint from worker-service (route + handler)
- Fix build script logging to show correct .cjs extension (was .mjs)

Documentation improvements:
- Add comprehensive FTS5 retention rationale documentation
- Include v7.0.0 removal TODO for future cleanup

Testing:
- Build succeeds with correct output logging
- Worker restarts successfully (30th restart)
- Contextualize endpoint properly removed (404 response)
- Search endpoint verified working

This addresses all critical review feedback from PR 133.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-21 18:59:23 -05:00

3.9 KiB

Search Observations (Semantic + Full-Text Hybrid)

Search all observations using natural language queries.

When to Use

  • User asks: "How did we implement authentication?"
  • User asks: "What bugs did we fix?"
  • User asks: "What features did we add?"
  • Looking for past work by keyword or topic

Command

curl -s "http://localhost:37777/api/search/observations?query=authentication&format=index&limit=5"

Parameters

  • query (optional): Natural language search query - uses semantic search (ChromaDB) for ranking with SQLite FTS5 fallback (e.g., "authentication", "bug fix", "database migration"). Can be omitted for filter-only searches.
  • format: "index" (summary) or "full" (complete details). Default: "full"
  • limit: Number of results (default: 20, max: 100)
  • project: Filter by project name (optional)
  • dateRange: Filter by date range (optional) - dateRange[start] and/or dateRange[end]
  • obs_type: Filter by observation type: bugfix, feature, refactor, decision, discovery, change (optional)
  • concepts: Filter by concept tags (optional)
  • files: Filter by file paths (optional)

Important: When omitting query, you MUST provide at least one filter (project, dateRange, obs_type, concepts, or files)

When to Use Each Format

Use format=index for:

  • Quick overviews
  • Finding IDs for deeper investigation
  • Listing multiple results
  • Token cost: ~50-100 per result

Use format=full for:

  • Complete details including narrative, facts, files, concepts
  • Understanding the full context of specific observations
  • Token cost: ~500-1000 per result

Example Response (format=index)

{
  "query": "authentication",
  "count": 5,
  "format": "index",
  "results": [
    {
      "id": 1234,
      "type": "feature",
      "title": "Implemented JWT authentication",
      "subtitle": "Added token-based auth with refresh tokens",
      "created_at_epoch": 1699564800000,
      "project": "api-server",
      "score": 0.95
    }
  ]
}

How to Present Results

For format=index, present as a compact list:

Found 5 results for "authentication":

1. **#1234** [feature] Implemented JWT authentication
   > Added token-based auth with refresh tokens
   > Nov 9, 2024 • api-server

2. **#1235** [bugfix] Fixed token expiration edge case
   > Handled race condition in refresh flow
   > Nov 9, 2024 • api-server

Include: ID (for follow-up), type emoji (🔴 bugfix, 🟣 feature, 🔄 refactor, 🔵 discovery, 🧠 decision, change), title, subtitle, date, project.

For complete formatting guidelines, see formatting.md (documentation coming soon).

Filter-Only Examples

Search without query text (direct SQLite filtering):

# Get all observations from November 2025
curl -s "http://localhost:37777/api/search?type=observations&dateRange[start]=2025-11-01&format=index"

# Get all bug fixes from a specific project
curl -s "http://localhost:37777/api/search?type=observations&obs_type=bugfix&project=api-server&format=index"

# Get all observations from last 7 days
curl -s "http://localhost:37777/api/search?type=observations&dateRange[start]=2025-11-11&format=index"

Error Handling

Missing query and filters:

{"error": "Either query or filters required for search"}

Fix: Provide either a query parameter OR at least one filter (project, dateRange, obs_type, concepts, files)

No results found:

{"query": "foobar", "count": 0, "results": []}

Response: "No results found for 'foobar'. Try different search terms."

Tips

  1. Be specific: "authentication JWT" > "auth"
  2. Start with format=index and limit=5-10
  3. Use project filtering when working on one codebase
  4. If no results, try broader terms or check spelling

Token Efficiency: