Alex Newman
c0778bef00
* feat: Add discovery_tokens for ROI tracking in observations and session summaries - Introduced `discovery_tokens` column in `observations` and `session_summaries` tables to track token costs associated with discovering and creating each observation and summary. - Updated relevant services and hooks to calculate and display ROI metrics based on discovery tokens. - Enhanced context economics reporting to include savings from reusing previous observations. - Implemented migration to ensure the new column is added to existing tables. - Adjusted data models and sync processes to accommodate the new `discovery_tokens` field. * refactor: streamline context hook by removing unused functions and updating terminology - Removed the estimateTokens and getObservations helper functions as they were not utilized. - Updated the legend and output messages to replace "discovery" with "work" for clarity. - Changed the emoji representation for different observation types to better reflect their purpose. - Enhanced output formatting for improved readability and understanding of token usage. * Refactor user-message-hook and context-hook for improved clarity and functionality - Updated user-message-hook.js to enhance error messaging and improve variable naming for clarity. - Modified context-hook.ts to include a new column key section, improved context index instructions, and added emoji icons for observation types. - Adjusted footer messages in context-hook.ts to emphasize token savings and access to past research. - Changed user-message-hook.ts to update the feedback and support message for clarity. * fix: Critical ROI tracking fixes from PR review Addresses critical findings from PR #111 review: 1. **Fixed incorrect discovery token calculation** (src/services/worker/SDKAgent.ts) - Changed from passing cumulative total to per-response delta - Now correctly tracks token cost for each observation/summary - Captures token state before/after response processing - Prevents all observations getting inflated cumulative values 2. **Fixed schema version mismatch** (src/services/sqlite/SessionStore.ts) - Changed ensureDiscoveryTokensColumn() from version 11 to version 7 - Now matches migration007 definition in migrations.ts - Ensures consistent version tracking across migration system These fixes ensure ROI metrics accurately reflect token costs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Update search documentation to reflect hybrid ChromaDB architecture The backend correctly implements ChromaDB-first semantic search with SQLite temporal ordering and FTS5 fallback, but documentation incorrectly described it as "FTS5 full-text search". This fix aligns all skill guides and tool descriptions with the actual implementation. Changes: - Update SKILL.md to describe hybrid architecture with ChromaDB primary - Update observations.md title and query parameter descriptions - Update all three search tool descriptions in search-server.ts: * search_observations * search_sessions * search_user_prompts All tools now correctly document: - ChromaDB semantic search (primary ranking) - 90-day recency filter - SQLite temporal ordering - FTS5 fallback (when ChromaDB unavailable) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: Add discovery_tokens column to observations and session_summaries tables --------- Co-authored-by: Claude <noreply@anthropic.com>
104 lines
2.9 KiB
Markdown
104 lines
2.9 KiB
Markdown
# Search Observations (Semantic + Full-Text Hybrid)
|
|
|
|
Search all observations using natural language queries.
|
|
|
|
## When to Use
|
|
|
|
- User asks: "How did we implement authentication?"
|
|
- User asks: "What bugs did we fix?"
|
|
- User asks: "What features did we add?"
|
|
- Looking for past work by keyword or topic
|
|
|
|
## Command
|
|
|
|
```bash
|
|
curl -s "http://localhost:37777/api/search/observations?query=authentication&format=index&limit=5"
|
|
```
|
|
|
|
## Parameters
|
|
|
|
- **query** (required): Natural language search query - uses semantic search (ChromaDB) for ranking with SQLite FTS5 fallback (e.g., "authentication", "bug fix", "database migration")
|
|
- **format**: "index" (summary) or "full" (complete details). Default: "full"
|
|
- **limit**: Number of results (default: 20, max: 100)
|
|
- **project**: Filter by project name (optional)
|
|
|
|
## When to Use Each Format
|
|
|
|
**Use format=index for:**
|
|
- Quick overviews
|
|
- Finding IDs for deeper investigation
|
|
- Listing multiple results
|
|
- **Token cost: ~50-100 per result**
|
|
|
|
**Use format=full for:**
|
|
- Complete details including narrative, facts, files, concepts
|
|
- Understanding the full context of specific observations
|
|
- **Token cost: ~500-1000 per result**
|
|
|
|
## Example Response (format=index)
|
|
|
|
```json
|
|
{
|
|
"query": "authentication",
|
|
"count": 5,
|
|
"format": "index",
|
|
"results": [
|
|
{
|
|
"id": 1234,
|
|
"type": "feature",
|
|
"title": "Implemented JWT authentication",
|
|
"subtitle": "Added token-based auth with refresh tokens",
|
|
"created_at_epoch": 1699564800000,
|
|
"project": "api-server",
|
|
"score": 0.95
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
## How to Present Results
|
|
|
|
For format=index, present as a compact list:
|
|
|
|
```markdown
|
|
Found 5 results for "authentication":
|
|
|
|
1. **#1234** [feature] Implemented JWT authentication
|
|
> Added token-based auth with refresh tokens
|
|
> Nov 9, 2024 • api-server
|
|
|
|
2. **#1235** [bugfix] Fixed token expiration edge case
|
|
> Handled race condition in refresh flow
|
|
> Nov 9, 2024 • api-server
|
|
```
|
|
|
|
**Include:** ID (for follow-up), type emoji (🔴 bugfix, 🟣 feature, 🔄 refactor, 🔵 discovery, 🧠 decision, ✅ change), title, subtitle, date, project.
|
|
|
|
For complete formatting guidelines, see formatting.md (documentation coming soon).
|
|
|
|
## Error Handling
|
|
|
|
**Missing query parameter:**
|
|
```json
|
|
{"error": "Missing required parameter: query"}
|
|
```
|
|
Fix: Add the query parameter
|
|
|
|
**No results found:**
|
|
```json
|
|
{"query": "foobar", "count": 0, "results": []}
|
|
```
|
|
Response: "No results found for 'foobar'. Try different search terms."
|
|
|
|
## Tips
|
|
|
|
1. Be specific: "authentication JWT" > "auth"
|
|
2. Start with format=index and limit=5-10
|
|
3. Use project filtering when working on one codebase
|
|
4. If no results, try broader terms or check spelling
|
|
|
|
**Token Efficiency:**
|
|
- Start with format=index (~50-100 tokens per result)
|
|
- Use format=full only for relevant items (~500-1000 tokens per result)
|
|
- See [../principles/progressive-disclosure.md](../principles/progressive-disclosure.md)
|