3cbc041c8bfaab315006c4c25ded2488777f915f
12 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Alex Newman
|
3cbc041c8b |
feat: Add ROI tracking with discovery_tokens for observations and session summaries (#111)
* feat: Add discovery_tokens for ROI tracking in observations and session summaries - Introduced `discovery_tokens` column in `observations` and `session_summaries` tables to track token costs associated with discovering and creating each observation and summary. - Updated relevant services and hooks to calculate and display ROI metrics based on discovery tokens. - Enhanced context economics reporting to include savings from reusing previous observations. - Implemented migration to ensure the new column is added to existing tables. - Adjusted data models and sync processes to accommodate the new `discovery_tokens` field. * refactor: streamline context hook by removing unused functions and updating terminology - Removed the estimateTokens and getObservations helper functions as they were not utilized. - Updated the legend and output messages to replace "discovery" with "work" for clarity. - Changed the emoji representation for different observation types to better reflect their purpose. - Enhanced output formatting for improved readability and understanding of token usage. * Refactor user-message-hook and context-hook for improved clarity and functionality - Updated user-message-hook.js to enhance error messaging and improve variable naming for clarity. - Modified context-hook.ts to include a new column key section, improved context index instructions, and added emoji icons for observation types. - Adjusted footer messages in context-hook.ts to emphasize token savings and access to past research. - Changed user-message-hook.ts to update the feedback and support message for clarity. * fix: Critical ROI tracking fixes from PR review Addresses critical findings from PR #111 review: 1. **Fixed incorrect discovery token calculation** (src/services/worker/SDKAgent.ts) - Changed from passing cumulative total to per-response delta - Now correctly tracks token cost for each observation/summary - Captures token state before/after response processing - Prevents all observations getting inflated cumulative values 2. **Fixed schema version mismatch** (src/services/sqlite/SessionStore.ts) - Changed ensureDiscoveryTokensColumn() from version 11 to version 7 - Now matches migration007 definition in migrations.ts - Ensures consistent version tracking across migration system These fixes ensure ROI metrics accurately reflect token costs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Alex Newman
|
68290a9121 |
Performance improvements: Token reduction and enhanced summaries (#101)
* refactor: Reduce continuation prompt token usage by 95 lines Removed redundant instructions from continuation prompt that were originally added to mitigate a session continuity issue. That issue has since been resolved, making these detailed instructions unnecessary on every continuation. Changes: - Reduced continuation prompt from ~106 lines to ~11 lines (~95 line reduction) - Changed "User's Goal:" to "Next Prompt in Session:" (more accurate framing) - Removed redundant WHAT TO RECORD, WHEN TO SKIP, and OUTPUT FORMAT sections - Kept concise reminder: "Continue generating observations and progress summaries..." - Initial prompt still contains all detailed instructions Impact: - Significant token savings on every continuation prompt - Faster context injection with no loss of functionality - Instructions remain comprehensive in initial prompt Files modified: - src/sdk/prompts.ts (buildContinuationPrompt function) - plugin/scripts/worker-service.cjs (compiled output) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * refactor: Enhance observation and summary prompts for clarity and token efficiency * Enhance prompt clarity and instructions in prompts.ts - Added a reminder to think about instructions before starting work. - Simplified the continuation prompt instruction by removing "for this ongoing session." * feat: Enhance settings.json with permissions and deny access to sensitive files refactor: Remove PLAN-full-observation-display.md and PR_SUMMARY.md as they are no longer needed chore: Delete SECURITY_SUMMARY.md since it is redundant after recent changes fix: Update worker-service.cjs to streamline observation generation instructions cleanup: Remove src-analysis.md and src-tree.md for a cleaner codebase refactor: Modify prompts.ts to clarify instructions for memory processing * refactor: Remove legacy worker service implementation * feat: Enhance summary hook to extract last assistant message and improve logging - Added function to extract the last assistant message from the transcript. - Updated summary hook to include last assistant message in the summary request. - Modified SDKSession interface to store last assistant message. - Adjusted buildSummaryPrompt to utilize last assistant message for generating summaries. - Updated worker service and session manager to handle last assistant message in summarize requests. - Introduced silentDebug utility for improved logging and diagnostics throughout the summary process. * docs: Add comprehensive implementation plan for ROI metrics feature Added detailed implementation plan covering: - Token usage capture from Agent SDK - Database schema changes (migration #8) - Discovery cost tracking per observation - Context hook display with ROI metrics - Testing and rollout strategy Timeline: ~20 hours over 4 days Goal: Empirical data for YC application amendment 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: Add transcript processing scripts for analysis and formatting - Implemented `dump-transcript-readable.ts` to generate a readable markdown dump of transcripts, excluding certain entry types. - Created `extract-rich-context-examples.ts` to extract and showcase rich context examples from transcripts, highlighting user requests and assistant reasoning. - Developed `format-transcript-context.ts` to format transcript context into a structured markdown format for improved observation generation. - Added `test-transcript-parser.ts` for validating data extraction from transcript JSONL files, including statistics and error reporting. - Introduced `transcript-to-markdown.ts` for a complete representation of transcript data in markdown format, showing all context data. - Enhanced type definitions in `transcript.ts` to support new features and ensure type safety. - Built `transcript-parser.ts` to handle parsing of transcript JSONL files, including error handling and data extraction methods. * Refactor hooks and SDKAgent for improved observation handling - Updated `new-hook.ts` to clean user prompts by stripping leading slashes for better semantic clarity. - Enhanced `save-hook.ts` to include additional tools in the SKIP_TOOLS set, preventing unnecessary observations from certain command invocations. - Modified `prompts.ts` to change the structure of observation prompts, emphasizing the observational role and providing a detailed XML output format for observations. - Adjusted `SDKAgent.ts` to enforce stricter tool usage restrictions, ensuring the memory agent operates solely as an observer without any tool access. * feat: Enhance session initialization to accept user prompts and prompt numbers - Updated `handleSessionInit` in `worker-service.ts` to extract `userPrompt` and `promptNumber` from the request body and pass them to `initializeSession`. - Modified `initializeSession` in `SessionManager.ts` to handle optional `currentUserPrompt` and `promptNumber` parameters. - Added logic to update the existing session's `userPrompt` and `lastPromptNumber` if a `currentUserPrompt` is provided. - Implemented debug logging for session initialization and updates to track user prompts and prompt numbers. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Alex Newman
|
39fedfc5fc |
feat: Enhance summary hook to include last user message from transcript (#95)
* feat: Enhance summary hook to include last user message from transcript - Added functionality to extract the last user message from a JSONL transcript file in the summary hook. - Updated the summary hook to send the last user message along with the summary request. - Modified the SDKSession interface to include an optional last_user_message field. - Updated the summary prompt to incorporate the last user message in the output format. - Refactored worker service to handle the last user message in the summarize queue. - Enhanced session manager to track and broadcast processing status based on active sessions and queue depth. - Improved error handling and logging for better traceability during transcript reading and processing. * feat(worker): enhance processing status broadcasting and session management - Added immediate broadcasting of processing status when a prompt is received. - Implemented logging for generator completion in multiple locations. - Updated `broadcastProcessingStatus` to include queue depth and active session count in logs. - Modified session iterator to stop yielding messages after a summary is yielded, with appropriate logging. |
||
|
basher83
|
97d565e3cd |
Replace search skill with mem-search (#91)
* feat: add mem-search skill with progressive disclosure architecture Add comprehensive mem-search skill for accessing claude-mem's persistent cross-session memory database. Implements progressive disclosure workflow and token-efficient search patterns. Features: - 12 search operations (observations, sessions, prompts, by-type, by-concept, by-file, timelines, etc.) - Progressive disclosure principles to minimize token usage - Anti-patterns documentation to guide LLM behavior - HTTP API integration for all search functionality - Common workflows with composition examples Structure: - SKILL.md: Entry point with temporal trigger patterns - principles/: Progressive disclosure + anti-patterns - operations/: 12 search operation files 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * docs: add CHANGELOG entry for mem-search skill Document mem-search skill addition in Unreleased section with: - 100% effectiveness compliance metrics - Comparison to previous search skill implementation - Progressive disclosure architecture details - Reference to audit report documentation 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * docs: add mem-search skill audit report Add comprehensive audit report validating mem-search skill against Anthropic's official skill-creator documentation. Report includes: - Effectiveness metrics comparison (search vs mem-search) - Critical issues analysis for production readiness - Compliance validation across 6 key dimensions - Reference implementation guidance Result: mem-search achieves 100% compliance vs search's 67% 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: Add comprehensive search architecture analysis document - Document current state of dual search architectures (HTTP API and MCP) - Analyze HTTP endpoints and MCP search server architectures - Identify DRY violations across search implementations - Evaluate the use of curl as the optimal approach for search - Provide architectural recommendations for immediate and long-term improvements - Outline action plan for cleanup, feature parity, DRY refactoring * refactor: Remove deprecated search skill documentation and operations * refactor: Reorganize documentation into public and context directories Changes: - Created docs/public/ for Mintlify documentation (.mdx files) - Created docs/context/ for internal planning and implementation docs - Moved all .mdx files and assets to docs/public/ - Moved all internal .md files to docs/context/ - Added CLAUDE.md to both directories explaining their purpose - Updated docs.json paths to work with new structure Benefits: - Clear separation between user-facing and internal documentation - Easier to maintain Mintlify docs in dedicated directory - Internal context files organized separately 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Enhance session management and continuity in hooks - Updated new-hook.ts to clarify session_id threading and idempotent session creation. - Modified prompts.ts to require claudeSessionId for continuation prompts, ensuring session context is maintained. - Improved SessionStore.ts documentation on createSDKSession to emphasize idempotent behavior and session connection. - Refined SDKAgent.ts to detail continuation prompt logic and its reliance on session.claudeSessionId for unified session handling. --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Alex Newman <thedotmack@gmail.com> |
||
|
Alex Newman
|
3529f9274b |
feat: Enhanced logging and SDK prompt improvements (#94)
* Initial plan * Initial analysis: Found root cause of double entries bug Co-authored-by: thedotmack <683968+thedotmack@users.noreply.github.com> * Fix double entries by assigning generatorPromise in handleSessionInit Co-authored-by: thedotmack <683968+thedotmack@users.noreply.github.com> * feat(logging): Enhance HTTP request logging and session management - Added middleware for logging HTTP requests and responses, excluding static assets and health checks. - Introduced a method to summarize request bodies for specific endpoints. - Improved logging for user prompt synchronization with Chroma, including duration tracking. - Enhanced session initialization logging to include additional session details. - Updated observation and summary logging to provide more context and error handling during Chroma synchronization. - Refactored tool name formatting for logging in the SessionManager. - Expanded logger component types to include 'HTTP', 'SESSION', and 'CHROMA'. * Refactor SDK prompts and logging for improved clarity and functionality - Updated buildInitPrompt to clarify the observer's role and what to record. - Enhanced buildSummaryPrompt with clearer instructions for summarizing ongoing sessions. - Improved buildContinuationPrompt to emphasize the focus on deliverables and capabilities. - Refactored WorkerService to utilize a centralized tool formatting function for logging. - Added truncation for logged responses and observations to improve readability. - Updated SessionManager to log the queuing of summarize actions with session details. - Enhanced App and Sidebar components to support refreshing stats on sidebar open. - Refactored useStats hook to allow manual refreshing of stats while maintaining automatic loading on mount. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: thedotmack <683968+thedotmack@users.noreply.github.com> |
||
|
copilot-swe-agent[bot]
|
730c420a13 |
Add CWD propagation through hook, worker, and SDK agent
Co-authored-by: thedotmack <683968+thedotmack@users.noreply.github.com> |
||
|
Alex Newman
|
fda069bb07 |
Fix GitHub issues #76, #74, #75 + session lifecycle improvements (#77)
* Refactor context-hook: Fix anti-patterns and improve maintainability This refactor addresses all anti-patterns documented in CLAUDE.md, improving code quality without changing behavior. **Dead Code Removed:** - Eliminated unused useIndexView parameter throughout - Cleaned up entry point logic **Magic Numbers → Named Constants:** - CHARS_PER_TOKEN_ESTIMATE = 4 (token estimation) - SUMMARY_LOOKAHEAD = 1 (explains +1 in query) - Added clarifying comment for DISPLAY_SESSION_COUNT **Code Duplication Eliminated:** - Reduced 34 lines to 4 lines with renderSummaryField() helper - Replaced 4 identical summary field rendering blocks **Error Handling Added:** - parseJsonArray() now catches JSON.parse exceptions - Prevents session crashes from malformed data **Type Safety Improved:** - Added SessionSummary interface (replaced inline type cast) - Added SummaryTimelineItem for timeline items - Proper Map typing: Map<string, TimelineItem[]> **Variable Naming Clarity:** - summariesWithOffset → summariesForTimeline - isMostRecent → shouldShowLink (explains purpose) - dayTimelines → itemsByDay - nextSummary → olderSummary (correct chronology) **Better Documentation:** - Explained confusing timeline offset logic - Removed apologetic comments, added clarifying ones **Impact:** - 28 lines saved from duplication elimination - Zero behavioral changes (output identical) - Improved maintainability and type safety 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix context-hook to respect settings.json contextDepth The context-hook was ignoring the user's contextDepth setting from the web UI (stored in ~/.claude-mem/settings.json) and always using the default of 50 observations. **Problem:** - Web UI sets contextDepth in ~/.claude-mem/settings.json - Context-hook only read from process.env.CLAUDE_MEM_CONTEXT_OBSERVATIONS - User's preference of 7 observations was ignored, always showing 50 **Solution:** - Added getContextDepth() function following same pattern as getWorkerPort() - Priority: settings.json > env var > default (50) - Validates contextDepth is a positive number **Testing:** - Verified with contextDepth: 7 → shows 7 observations ✓ - Verified with contextDepth: 3 → shows 3 observations ✓ - Settings properly respected on every session start **Files Changed:** - src/hooks/context-hook.ts: Added getContextDepth() + imports - plugin/scripts/context-hook.js: Built output 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix context-hook to read from correct settings file location **Critical Bug Fix:** Previous commit read from completely wrong location with wrong field name. **What was wrong:** - Reading from: ~/.claude-mem/settings.json (doesn't exist) - Looking for: contextDepth (wrong field) - Result: Always falling back to default of 50 **What's fixed:** - Reading from: ~/.claude/settings.json (correct location) - Looking for: env.CLAUDE_MEM_CONTEXT_OBSERVATIONS (correct field) - Matches pattern used in worker-service.ts **Testing:** - With CLAUDE_MEM_CONTEXT_OBSERVATIONS: "15" → shows 15 observations ✓ - With CLAUDE_MEM_CONTEXT_OBSERVATIONS: "5" → shows 5 observations ✓ - Web UI settings now properly respected **Files Changed:** - src/hooks/context-hook.ts: Fixed path and field name in getContextDepth() - plugin/scripts/context-hook.js: Built output 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Fix GitHub issues #76, #74, #75 + session lifecycle improvements Bug Fixes: - Fix PM2 'Process 0 not found' error (#76): Changed pm2 restart to pm2 start (idempotent) - Fix troubleshooting skill distribution (#74, #75): Moved from .claude/skills/ to plugin/skills/ Session Lifecycle Improvements: - Added session lifecycle context to SDK agent prompt - Changed summary framing from "final report" to "progress checkpoint" - Updated summary prompts to use progressive tense ("so far", "actively working on") - Added buildContinuationPrompt() for prompt #2+ to avoid re-initialization - SessionManager now restores prompt counter from database - SDKAgent conditionally uses init vs continuation prompt based on prompt number These changes improve context-loading task handling and reduce incorrect "file not found" reports in summaries (partial fix for #73 - awaiting user feedback). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> * Release v5.3.0: Session lifecycle improvements and bug fixes Improvements: - Session prompt counter now restored from DB on worker restart - Continuation prompts for prompt #2+ (lightweight, avoids re-init) - Summary framing changed from "final report" to "progress checkpoint" - PM2 start command (idempotent, fixes "Process 0 not found" error) - Troubleshooting skill moved to plugin/skills/ for proper distribution Technical changes: - SessionManager loads prompt_counter from DB on initialization - SDKAgent uses buildContinuationPrompt() for requests #2+ - Updated summary prompt to clarify mid-session checkpoints - Fixed worker-utils.ts to use pm2 start instead of pm2 restart - Moved .claude/skills/troubleshoot → plugin/skills/troubleshoot Fixes: - GitHub issue #76: PM2 "Process 0 not found" error - GitHub issue #74, #75: Troubleshooting skill not distributed - GitHub issue #73 (partial): Context-loading tasks reported as failed 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Alex Newman
|
e22edaddf4 | fix: update narrative assignment in SDKAgent to use obs.narrative | ||
|
Alex Newman
|
30a42036aa |
feat: add scroll-to-top button and improve pagination handling
- Implemented a scroll-to-top button in the viewer UI for better navigation. - Added styles for the scroll-to-top button in viewer.html and viewer-template.html. - Created a new ScrollToTop component to manage visibility and scrolling behavior. - Updated Feed component to include the ScrollToTop component. - Enhanced pagination logic in usePagination hook to prevent stale closures and improve performance. - Modified SDKAgent to include additional observation fields for better data handling. |
||
|
Alex Newman
|
4bc467f7ed |
feat: Implement Worker Service for long-running HTTP service with PM2 management
- Introduced WorkerService class to handle HTTP requests and manage sessions. - Added endpoints for health check, session management, and data retrieval. - Integrated ChromaSync for background data synchronization. - Implemented SSE for real-time updates to connected clients. - Added error handling and logging throughout the service. - Cached Claude executable path for improved performance. - Included settings management for user configuration. - Established database interactions for session and observation management. |
||
|
Alex Newman
|
13643a5b18 |
Release v5.1.3: Fix PostToolUse hook field name bug
Bug Fix: - Changed tool_output to tool_response throughout PostToolUse hook chain - PostToolUse events provide tool_response field, not tool_output - Updated save-hook.ts to send tool_response - Updated worker-service.ts endpoint to accept tool_response - Updated worker-service-v2.ts for consistency - Updated worker-types.ts interfaces (PendingMessage, ObservationData) - Updated SessionManager.ts message queue - Updated SDKAgent.ts observation prompt builder Impact: Fixes observation capture for PostToolUse events. Previous version was sending undefined for tool responses, causing incomplete observations. Files changed: - src/hooks/save-hook.ts (interface + destructuring + fetch body) - src/services/worker-service.ts (ObservationMessage interface + handler + SDK prompt) - src/services/worker-service-v2.ts (handler) - src/services/worker-types.ts (PendingMessage + ObservationData interfaces) - src/services/worker/SessionManager.ts (queue push) - src/services/worker/SDKAgent.ts (observation prompt) - plugin/scripts/*.js (rebuilt) - Version bumped to 5.1.3 (patch) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> |
||
|
Alex Newman
|
980151a50e |
feat: Implement Worker Service v2 with improved architecture
- Complete rewrite of the Worker Service following object-oriented principles. - Introduced a single long-lived database connection to reduce overhead. - Implemented event-driven queues to eliminate polling. - Added DRY utilities for pagination and settings management. - Reduced code size significantly from 1173 lines to approximately 600-700 lines. - Created various composed services including DatabaseManager, SessionManager, and SDKAgent. - Enhanced SSE broadcasting capabilities for real-time client updates. - Established a structured approach for session lifecycle management and event handling. - Introduced type safety with shared TypeScript interfaces for better maintainability. |