* feat: Add discovery_tokens for ROI tracking in observations and session summaries
- Introduced `discovery_tokens` column in `observations` and `session_summaries` tables to track token costs associated with discovering and creating each observation and summary.
- Updated relevant services and hooks to calculate and display ROI metrics based on discovery tokens.
- Enhanced context economics reporting to include savings from reusing previous observations.
- Implemented migration to ensure the new column is added to existing tables.
- Adjusted data models and sync processes to accommodate the new `discovery_tokens` field.
* refactor: streamline context hook by removing unused functions and updating terminology
- Removed the estimateTokens and getObservations helper functions as they were not utilized.
- Updated the legend and output messages to replace "discovery" with "work" for clarity.
- Changed the emoji representation for different observation types to better reflect their purpose.
- Enhanced output formatting for improved readability and understanding of token usage.
* Refactor user-message-hook and context-hook for improved clarity and functionality
- Updated user-message-hook.js to enhance error messaging and improve variable naming for clarity.
- Modified context-hook.ts to include a new column key section, improved context index instructions, and added emoji icons for observation types.
- Adjusted footer messages in context-hook.ts to emphasize token savings and access to past research.
- Changed user-message-hook.ts to update the feedback and support message for clarity.
* fix: Critical ROI tracking fixes from PR review
Addresses critical findings from PR #111 review:
1. **Fixed incorrect discovery token calculation** (src/services/worker/SDKAgent.ts)
- Changed from passing cumulative total to per-response delta
- Now correctly tracks token cost for each observation/summary
- Captures token state before/after response processing
- Prevents all observations getting inflated cumulative values
2. **Fixed schema version mismatch** (src/services/sqlite/SessionStore.ts)
- Changed ensureDiscoveryTokensColumn() from version 11 to version 7
- Now matches migration007 definition in migrations.ts
- Ensures consistent version tracking across migration system
These fixes ensure ROI metrics accurately reflect token costs.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: Update search documentation to reflect hybrid ChromaDB architecture
The backend correctly implements ChromaDB-first semantic search with SQLite
temporal ordering and FTS5 fallback, but documentation incorrectly described
it as "FTS5 full-text search". This fix aligns all skill guides and tool
descriptions with the actual implementation.
Changes:
- Update SKILL.md to describe hybrid architecture with ChromaDB primary
- Update observations.md title and query parameter descriptions
- Update all three search tool descriptions in search-server.ts:
* search_observations
* search_sessions
* search_user_prompts
All tools now correctly document:
- ChromaDB semantic search (primary ranking)
- 90-day recency filter
- SQLite temporal ordering
- FTS5 fallback (when ChromaDB unavailable)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: Add discovery_tokens column to observations and session_summaries tables
---------
Co-authored-by: Claude <noreply@anthropic.com>
* refactor: Reduce continuation prompt token usage by 95 lines
Removed redundant instructions from continuation prompt that were originally
added to mitigate a session continuity issue. That issue has since been
resolved, making these detailed instructions unnecessary on every continuation.
Changes:
- Reduced continuation prompt from ~106 lines to ~11 lines (~95 line reduction)
- Changed "User's Goal:" to "Next Prompt in Session:" (more accurate framing)
- Removed redundant WHAT TO RECORD, WHEN TO SKIP, and OUTPUT FORMAT sections
- Kept concise reminder: "Continue generating observations and progress summaries..."
- Initial prompt still contains all detailed instructions
Impact:
- Significant token savings on every continuation prompt
- Faster context injection with no loss of functionality
- Instructions remain comprehensive in initial prompt
Files modified:
- src/sdk/prompts.ts (buildContinuationPrompt function)
- plugin/scripts/worker-service.cjs (compiled output)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor: Enhance observation and summary prompts for clarity and token efficiency
* Enhance prompt clarity and instructions in prompts.ts
- Added a reminder to think about instructions before starting work.
- Simplified the continuation prompt instruction by removing "for this ongoing session."
* feat: Enhance settings.json with permissions and deny access to sensitive files
refactor: Remove PLAN-full-observation-display.md and PR_SUMMARY.md as they are no longer needed
chore: Delete SECURITY_SUMMARY.md since it is redundant after recent changes
fix: Update worker-service.cjs to streamline observation generation instructions
cleanup: Remove src-analysis.md and src-tree.md for a cleaner codebase
refactor: Modify prompts.ts to clarify instructions for memory processing
* refactor: Remove legacy worker service implementation
* feat: Enhance summary hook to extract last assistant message and improve logging
- Added function to extract the last assistant message from the transcript.
- Updated summary hook to include last assistant message in the summary request.
- Modified SDKSession interface to store last assistant message.
- Adjusted buildSummaryPrompt to utilize last assistant message for generating summaries.
- Updated worker service and session manager to handle last assistant message in summarize requests.
- Introduced silentDebug utility for improved logging and diagnostics throughout the summary process.
* docs: Add comprehensive implementation plan for ROI metrics feature
Added detailed implementation plan covering:
- Token usage capture from Agent SDK
- Database schema changes (migration #8)
- Discovery cost tracking per observation
- Context hook display with ROI metrics
- Testing and rollout strategy
Timeline: ~20 hours over 4 days
Goal: Empirical data for YC application amendment
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* feat: Add transcript processing scripts for analysis and formatting
- Implemented `dump-transcript-readable.ts` to generate a readable markdown dump of transcripts, excluding certain entry types.
- Created `extract-rich-context-examples.ts` to extract and showcase rich context examples from transcripts, highlighting user requests and assistant reasoning.
- Developed `format-transcript-context.ts` to format transcript context into a structured markdown format for improved observation generation.
- Added `test-transcript-parser.ts` for validating data extraction from transcript JSONL files, including statistics and error reporting.
- Introduced `transcript-to-markdown.ts` for a complete representation of transcript data in markdown format, showing all context data.
- Enhanced type definitions in `transcript.ts` to support new features and ensure type safety.
- Built `transcript-parser.ts` to handle parsing of transcript JSONL files, including error handling and data extraction methods.
* Refactor hooks and SDKAgent for improved observation handling
- Updated `new-hook.ts` to clean user prompts by stripping leading slashes for better semantic clarity.
- Enhanced `save-hook.ts` to include additional tools in the SKIP_TOOLS set, preventing unnecessary observations from certain command invocations.
- Modified `prompts.ts` to change the structure of observation prompts, emphasizing the observational role and providing a detailed XML output format for observations.
- Adjusted `SDKAgent.ts` to enforce stricter tool usage restrictions, ensuring the memory agent operates solely as an observer without any tool access.
* feat: Enhance session initialization to accept user prompts and prompt numbers
- Updated `handleSessionInit` in `worker-service.ts` to extract `userPrompt` and `promptNumber` from the request body and pass them to `initializeSession`.
- Modified `initializeSession` in `SessionManager.ts` to handle optional `currentUserPrompt` and `promptNumber` parameters.
- Added logic to update the existing session's `userPrompt` and `lastPromptNumber` if a `currentUserPrompt` is provided.
- Implemented debug logging for session initialization and updates to track user prompts and prompt numbers.
---------
Co-authored-by: Claude <noreply@anthropic.com>
- Changed user prompt formatting to use full text instead of truncated version.
- Updated date filtering logic to use milliseconds instead of seconds for 90-day cutoff.
- Renamed doc_type values in ChromaSync to ensure consistency and prevent deduplication issues.
- Improved documentation for concept tags in input schema.
- Updated the StdioClientTransport configuration in search-server.ts to include 'stderr: ignore' for the Chroma client.
- Modified the ChromaSync class in ChromaSync.ts to also set 'stderr: ignore' when initializing the Chroma client.
- Implemented a new tool to search for observations using natural language and retrieve timeline context around the best match.
- Introduced two modes: "auto" for automatic timeline anchor selection and "interactive" for user selection of top matches.
- Added input schema validation using zod for query parameters including depth before/after, limit, and project filtering.
- Integrated hybrid semantic search with fallback to FTS5 for observation retrieval.
- Enhanced response formatting for both modes, including detailed timeline context and observation summaries.
- Added `getObservationById` method to retrieve observations by ID in SessionStore.
- Introduced `getSessionSummariesByIds` and `getUserPromptsByIds` methods for fetching session summaries and user prompts by IDs.
- Developed `getTimelineAroundTimestamp` and `getTimelineAroundObservation` methods to provide a unified timeline of observations, sessions, and prompts around a specified anchor point.
- Enhanced ChromaSync to format and sync user prompts, including a new `syncUserPrompt` method.
- Updated WorkerService to sync the latest user prompt to Chroma after updating the worker port.
- Created tests for timeline querying and MCP handler logic to ensure functionality.
- Documented the implementation plan for user prompts and timeline context tool in the Chroma search completion plan.
- Built search-server.mjs successfully (55KB)
- Configured with packages: 'external' to use node_modules dependencies
- MCP config points to ${CLAUDE_PLUGIN_ROOT}/scripts/search-server.mjs
- Ready for deployment to plugin directory
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Added functionality to save raw user prompts for full-text search in the newHook function.
- Introduced new search endpoint 'search_user_prompts' to retrieve user prompts using FTS5.
- Created UserPromptRow and UserPromptSearchResult types for handling user prompt data.
- Implemented searchUserPrompts method in SessionSearch class to perform FTS5 queries.
- Created user_prompts table with FTS5 support and necessary triggers for data synchronization.
- Updated SessionStore to include methods for saving user prompts and managing the new table.
- Deleted the advanced_search tool from the search-server.ts file, which combined full-text search with structured filters for observations and sessions.
- Removed the advancedSearch method from the SessionSearch class, which handled the logic for performing advanced searches using FTS5 and structured filters.
- Filter out observation type from concepts array in parseObservations function to ensure types and concepts are treated as separate dimensions. Added logging for removed types.
- Update prompts documentation to clarify that the observation type must not be included in the concepts array.
- Modify search-server to provide clearer guidance on result limits, emphasizing starting with smaller limits to avoid exceeding token limits.
- Refactor SessionSearch methods to accept options for limit, offset, and orderBy parameters, improving flexibility in querying observations by concept and type.
- Updated the formatSearchTips function to emphasize the importance of using index format first for token efficiency.
- Revised descriptions for various search tools to include reminders about starting with index format and using full format only for specific items of interest.
- Clarified output format descriptions to recommend index format for initial searches.
- Introduced `formatSearchTips` function to provide helpful search tips in the output.
- Added formatting options for search results, allowing users to choose between 'index' (titles/dates only) and 'full' (complete details).
- Updated handlers for observation and session search tools to accommodate the new format option.
- Enhanced result formatting to include date information and improved output structure.
- Replaced `createSdkMcpServer` with `Server` from `@modelcontextprotocol/sdk/server/index.js`.
- Updated tool definitions to use structured input schemas with Zod.
- Enhanced response formatting for search results, combining multiple results into a single text response.
- Added new tools for advanced search and recent session context retrieval.
- Improved error handling and logging throughout the server.
- Added search functionality for observations and sessions using full-text search.
- Implemented formatting functions for search results with citations.
- Created multiple tools for searching by various criteria including concept, file, type, and advanced search.
- Integrated structured filters and pagination options for search queries.
- Established error handling for search operations and server initialization.
Copilot