Alex Newman
c0778bef00
* feat: Add discovery_tokens for ROI tracking in observations and session summaries - Introduced `discovery_tokens` column in `observations` and `session_summaries` tables to track token costs associated with discovering and creating each observation and summary. - Updated relevant services and hooks to calculate and display ROI metrics based on discovery tokens. - Enhanced context economics reporting to include savings from reusing previous observations. - Implemented migration to ensure the new column is added to existing tables. - Adjusted data models and sync processes to accommodate the new `discovery_tokens` field. * refactor: streamline context hook by removing unused functions and updating terminology - Removed the estimateTokens and getObservations helper functions as they were not utilized. - Updated the legend and output messages to replace "discovery" with "work" for clarity. - Changed the emoji representation for different observation types to better reflect their purpose. - Enhanced output formatting for improved readability and understanding of token usage. * Refactor user-message-hook and context-hook for improved clarity and functionality - Updated user-message-hook.js to enhance error messaging and improve variable naming for clarity. - Modified context-hook.ts to include a new column key section, improved context index instructions, and added emoji icons for observation types. - Adjusted footer messages in context-hook.ts to emphasize token savings and access to past research. - Changed user-message-hook.ts to update the feedback and support message for clarity. * fix: Critical ROI tracking fixes from PR review Addresses critical findings from PR #111 review: 1. **Fixed incorrect discovery token calculation** (src/services/worker/SDKAgent.ts) - Changed from passing cumulative total to per-response delta - Now correctly tracks token cost for each observation/summary - Captures token state before/after response processing - Prevents all observations getting inflated cumulative values 2. **Fixed schema version mismatch** (src/services/sqlite/SessionStore.ts) - Changed ensureDiscoveryTokensColumn() from version 11 to version 7 - Now matches migration007 definition in migrations.ts - Ensures consistent version tracking across migration system These fixes ensure ROI metrics accurately reflect token costs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Update search documentation to reflect hybrid ChromaDB architecture The backend correctly implements ChromaDB-first semantic search with SQLite temporal ordering and FTS5 fallback, but documentation incorrectly described it as "FTS5 full-text search". This fix aligns all skill guides and tool descriptions with the actual implementation. Changes: - Update SKILL.md to describe hybrid architecture with ChromaDB primary - Update observations.md title and query parameter descriptions - Update all three search tool descriptions in search-server.ts: * search_observations * search_sessions * search_user_prompts All tools now correctly document: - ChromaDB semantic search (primary ranking) - 90-day recency filter - SQLite temporal ordering - FTS5 fallback (when ChromaDB unavailable) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Add discovery_tokens column to observations and session_summaries tables --------- Co-authored-by: Claude <noreply@anthropic.com>
2.9 KiB
2.9 KiB
Search Observations (Semantic + Full-Text Hybrid)
Search all observations using natural language queries.
When to Use
- User asks: "How did we implement authentication?"
- User asks: "What bugs did we fix?"
- User asks: "What features did we add?"
- Looking for past work by keyword or topic
Command
curl -s "http://localhost:37777/api/search/observations?query=authentication&format=index&limit=5"
Parameters
- query (required): Natural language search query - uses semantic search (ChromaDB) for ranking with SQLite FTS5 fallback (e.g., "authentication", "bug fix", "database migration")
- format: "index" (summary) or "full" (complete details). Default: "full"
- limit: Number of results (default: 20, max: 100)
- project: Filter by project name (optional)
When to Use Each Format
Use format=index for:
- Quick overviews
- Finding IDs for deeper investigation
- Listing multiple results
- Token cost: ~50-100 per result
Use format=full for:
- Complete details including narrative, facts, files, concepts
- Understanding the full context of specific observations
- Token cost: ~500-1000 per result
Example Response (format=index)
{
"query": "authentication",
"count": 5,
"format": "index",
"results": [
{
"id": 1234,
"type": "feature",
"title": "Implemented JWT authentication",
"subtitle": "Added token-based auth with refresh tokens",
"created_at_epoch": 1699564800000,
"project": "api-server",
"score": 0.95
}
]
}
How to Present Results
For format=index, present as a compact list:
Found 5 results for "authentication":
1. **#1234** [feature] Implemented JWT authentication
> Added token-based auth with refresh tokens
> Nov 9, 2024 • api-server
2. **#1235** [bugfix] Fixed token expiration edge case
> Handled race condition in refresh flow
> Nov 9, 2024 • api-server
Include: ID (for follow-up), type emoji (🔴 bugfix, 🟣 feature, 🔄 refactor, 🔵 discovery, 🧠 decision, ✅ change), title, subtitle, date, project.
For complete formatting guidelines, see formatting.md (documentation coming soon).
Error Handling
Missing query parameter:
{"error": "Missing required parameter: query"}
Fix: Add the query parameter
No results found:
{"query": "foobar", "count": 0, "results": []}
Response: "No results found for 'foobar'. Try different search terms."
Tips
- Be specific: "authentication JWT" > "auth"
- Start with format=index and limit=5-10
- Use project filtering when working on one codebase
- If no results, try broader terms or check spelling
Token Efficiency:
- Start with format=index (~50-100 tokens per result)
- Use format=full only for relevant items (~500-1000 tokens per result)
- See ../principles/progressive-disclosure.md