c5e68a17c8
* Refactor code structure for improved readability and maintainability * Add test results for search API and related functionalities - Created test result files for various search-related functionalities, including: - test-11-search-server-changes.json - test-12-context-hook-changes.json - test-13-worker-service-changes.json - test-14-patterns.json - test-15-gotchas.json - test-16-discoveries.json - test-17-all-bugfixes.json - test-18-all-features.json - test-19-all-decisions.json - test-20-session-search.json - test-21-prompt-search.json - test-22-decisions-endpoint.json - test-23-changes-endpoint.json - test-24-how-it-works-endpoint.json - test-25-contextualize-endpoint.json - test-26-timeline-around-observation.json - test-27-multi-param-combo.json - test-28-file-type-combo.json - Each test result file captures specific search failures or outcomes, including issues with undefined properties and successful execution of search queries. - Enhanced documentation of search architecture and testing strategies, ensuring compliance with established guidelines and improving overall search functionality. * feat: Enhance unified search API with catch-all parameters and backward compatibility - Implemented a unified search API at /api/search that accepts catch-all parameters for filtering by type, observation type, concepts, and files. - Maintained backward compatibility by keeping granular endpoints functional while routing through the same infrastructure. - Completed comprehensive testing of search capabilities with real-world query scenarios. fix: Address missing debug output in search API query tests - Flushed PM2 logs and executed search queries to verify functionality. - Diagnosed absence of "Raw Chroma" debug messages in worker logs, indicating potential issues with logging or query processing. refactor: Improve build and deployment pipeline for claude-mem plugin - Successfully built and synced all hooks and services to the marketplace directory. - Ensured all dependencies are installed and up-to-date in the deployment location. feat: Implement hybrid search filters with 90-day recency window - Enhanced search server to apply a 90-day recency filter to Chroma results before categorizing by document type. fix: Correct parameter handling in searchUserPrompts method - Added support for filter-only queries and improved dual-path logic for clarity. refactor: Rename FTS5 method to clarify fallback status - Renamed escapeFTS5 to escapeFTS5_fallback_when_chroma_unavailable to indicate its temporary usage. feat: Introduce contextualize tool for comprehensive project overview - Added a new tool to fetch recent observations, sessions, and user prompts, providing a quick project overview. feat: Add semantic shortcut tools for common search patterns - Implemented 'decisions', 'changes', and 'how_it_works' tools for convenient access to frequently searched observation categories. feat: Unified timeline tool supports anchor and query modes - Combined get_context_timeline and get_timeline_by_query into a single interface for timeline exploration. feat: Unified search tool added to MCP server - New tool queries all memory types simultaneously, providing combined chronological results for improved search efficiency. * Refactor search functionality to clarify FTS5 fallback usage - Updated `worker-service.cjs` to replace FTS5 fallback function with a more descriptive name and improved error handling. - Enhanced documentation in `SKILL.md` to specify the unified API endpoint and clarify the behavior of the search engine, including the conditions under which FTS5 is used. - Modified `search-server.ts` to provide clearer logging and descriptions regarding the fallback to FTS5 when UVX/Python is unavailable. - Renamed and updated the `SessionSearch.ts` methods to reflect the conditions for using FTS5, emphasizing the lack of semantic understanding in fallback scenarios. * feat: Add ID-based fetch endpoints and simplify mem-search skill **Problem:** - Search returns IDs but no way to fetch by ID - Skill documentation was bloated with too many options - Claude wasn't using IDs because we didn't tell it how **Solution:** 1. Added three new HTTP endpoints: - GET /api/observation/:id - GET /api/session/:id - GET /api/prompt/:id 2. Completely rewrote SKILL.md: - Stripped complexity down to essentials - Clear 3-step prescriptive workflow: Search → Review IDs → Fetch by ID - Emphasized ID usage: "The IDs are there for a reason - USE THEM" - Removed confusing multi-endpoint documentation - Kept only unified search with filters **Impact:** - Token efficiency: Claude can now fetch full details only for relevant IDs - Clarity: One clear workflow instead of 10+ options to choose from - Usability: IDs are no longer wasted context - they're actionable 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * chore: Move internal docs to private directory Moved POSTMORTEM and planning docs to ./private to exclude from PR reviews. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Remove experimental contextualize endpoint - Removed contextualize MCP tool from search-server (saves ~4KB) - Disabled FTS5 fallback paths in SessionSearch (now vector-first) - Cleaned up CLAUDE.md documentation - Removed contextualize-rewrite-plan.md doc Rationale: - Contextualize is better suited as a skill (LLM-powered) than an endpoint - Search API already provides vector search with configurable limits - Created issue #132 to track future contextualize skill implementation Changes: - src/servers/search-server.ts: Removed contextualize tool definition - src/services/sqlite/SessionSearch.ts: Disabled FTS5 fallback, added deprecation warnings - CLAUDE.md: Cleaned up outdated skill documentation - docs/: Removed contextualize plan document 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Complete FTS5 cleanup - remove all deprecated search code This completes the FTS5 cleanup work by removing all commented-out FTS5 search code while preserving database tables for backward compatibility. Changes: - Removed 200+ lines of commented FTS5 search code from SessionSearch.ts - Removed deprecated degraded_search_query__when_uvx_unavailable method - Updated all method documentation to clarify vector-first architecture - Updated class documentation to reflect filter-only query support - Updated CLAUDE.md to remove FTS5 search references - Clarified that FTS5 tables exist for backward compatibility only - Updated "Why SQLite FTS5" section to "Why Vector-First Search" Database impact: NONE - FTS5 tables remain intact for existing installations Search architecture: - ChromaDB: All text-based vector search queries - SQLite: Filter-only queries (date ranges, metadata, no query text) - FTS5 tables: Maintained but unused (backward compatibility) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Remove all FTS5 fallback execution code from search-server Completes the FTS5 cleanup by removing all fallback execution paths that attempted to use FTS5 when ChromaDB was unavailable. Changes: - Removed all FTS5 fallback code execution paths - When ChromaDB fails or is unavailable, return empty results with helpful error messages - Updated all deprecated tool descriptions (search_observations, search_sessions, search_user_prompts) - Changed error messages to indicate FTS5 fallback has been removed - Added installation instructions for UVX/Python when vector search is unavailable - Updated comments from "hybrid search" to "vector-first search" - Removed ~100 lines of dead FTS5 fallback code Database impact: NONE - FTS5 tables remain intact (backward compatibility) Search behavior when ChromaDB unavailable: - Text queries: Return empty results with error explaining ChromaDB is required - Filter-only queries (no text): Continue to work via direct SQLite 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Address PR 133 review feedback Critical fixes: - Remove contextualize endpoint from worker-service (route + handler) - Fix build script logging to show correct .cjs extension (was .mjs) Documentation improvements: - Add comprehensive FTS5 retention rationale documentation - Include v7.0.0 removal TODO for future cleanup Testing: - Build succeeds with correct output logging - Worker restarts successfully (30th restart) - Contextualize endpoint properly removed (404 response) - Search endpoint verified working This addresses all critical review feedback from PR 133. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
125 lines
3.9 KiB
Markdown
125 lines
3.9 KiB
Markdown
# Search Observations (Semantic + Full-Text Hybrid)
|
|
|
|
Search all observations using natural language queries.
|
|
|
|
## When to Use
|
|
|
|
- User asks: "How did we implement authentication?"
|
|
- User asks: "What bugs did we fix?"
|
|
- User asks: "What features did we add?"
|
|
- Looking for past work by keyword or topic
|
|
|
|
## Command
|
|
|
|
```bash
|
|
curl -s "http://localhost:37777/api/search/observations?query=authentication&format=index&limit=5"
|
|
```
|
|
|
|
## Parameters
|
|
|
|
- **query** (optional): Natural language search query - uses semantic search (ChromaDB) for ranking with SQLite FTS5 fallback (e.g., "authentication", "bug fix", "database migration"). Can be omitted for filter-only searches.
|
|
- **format**: "index" (summary) or "full" (complete details). Default: "full"
|
|
- **limit**: Number of results (default: 20, max: 100)
|
|
- **project**: Filter by project name (optional)
|
|
- **dateRange**: Filter by date range (optional) - `dateRange[start]` and/or `dateRange[end]`
|
|
- **obs_type**: Filter by observation type: bugfix, feature, refactor, decision, discovery, change (optional)
|
|
- **concepts**: Filter by concept tags (optional)
|
|
- **files**: Filter by file paths (optional)
|
|
|
|
**Important**: When omitting `query`, you MUST provide at least one filter (project, dateRange, obs_type, concepts, or files)
|
|
|
|
## When to Use Each Format
|
|
|
|
**Use format=index for:**
|
|
- Quick overviews
|
|
- Finding IDs for deeper investigation
|
|
- Listing multiple results
|
|
- **Token cost: ~50-100 per result**
|
|
|
|
**Use format=full for:**
|
|
- Complete details including narrative, facts, files, concepts
|
|
- Understanding the full context of specific observations
|
|
- **Token cost: ~500-1000 per result**
|
|
|
|
## Example Response (format=index)
|
|
|
|
```json
|
|
{
|
|
"query": "authentication",
|
|
"count": 5,
|
|
"format": "index",
|
|
"results": [
|
|
{
|
|
"id": 1234,
|
|
"type": "feature",
|
|
"title": "Implemented JWT authentication",
|
|
"subtitle": "Added token-based auth with refresh tokens",
|
|
"created_at_epoch": 1699564800000,
|
|
"project": "api-server",
|
|
"score": 0.95
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
## How to Present Results
|
|
|
|
For format=index, present as a compact list:
|
|
|
|
```markdown
|
|
Found 5 results for "authentication":
|
|
|
|
1. **#1234** [feature] Implemented JWT authentication
|
|
> Added token-based auth with refresh tokens
|
|
> Nov 9, 2024 • api-server
|
|
|
|
2. **#1235** [bugfix] Fixed token expiration edge case
|
|
> Handled race condition in refresh flow
|
|
> Nov 9, 2024 • api-server
|
|
```
|
|
|
|
**Include:** ID (for follow-up), type emoji (🔴 bugfix, 🟣 feature, 🔄 refactor, 🔵 discovery, 🧠 decision, ✅ change), title, subtitle, date, project.
|
|
|
|
For complete formatting guidelines, see formatting.md (documentation coming soon).
|
|
|
|
## Filter-Only Examples
|
|
|
|
Search without query text (direct SQLite filtering):
|
|
|
|
```bash
|
|
# Get all observations from November 2025
|
|
curl -s "http://localhost:37777/api/search?type=observations&dateRange[start]=2025-11-01&format=index"
|
|
|
|
# Get all bug fixes from a specific project
|
|
curl -s "http://localhost:37777/api/search?type=observations&obs_type=bugfix&project=api-server&format=index"
|
|
|
|
# Get all observations from last 7 days
|
|
curl -s "http://localhost:37777/api/search?type=observations&dateRange[start]=2025-11-11&format=index"
|
|
```
|
|
|
|
## Error Handling
|
|
|
|
**Missing query and filters:**
|
|
```json
|
|
{"error": "Either query or filters required for search"}
|
|
```
|
|
Fix: Provide either a query parameter OR at least one filter (project, dateRange, obs_type, concepts, files)
|
|
|
|
**No results found:**
|
|
```json
|
|
{"query": "foobar", "count": 0, "results": []}
|
|
```
|
|
Response: "No results found for 'foobar'. Try different search terms."
|
|
|
|
## Tips
|
|
|
|
1. Be specific: "authentication JWT" > "auth"
|
|
2. Start with format=index and limit=5-10
|
|
3. Use project filtering when working on one codebase
|
|
4. If no results, try broader terms or check spelling
|
|
|
|
**Token Efficiency:**
|
|
- Start with format=index (~50-100 tokens per result)
|
|
- Use format=full only for relevant items (~500-1000 tokens per result)
|
|
- See [../principles/progressive-disclosure.md](../principles/progressive-disclosure.md)
|