Commit Graph

39 Commits

Author SHA1 Message Date
Alex Newman f20bb5bced Add SDK usage tracking to JSONL logs
Features:
- New UsageLogger utility that writes usage metrics to daily JSONL files
- Captures token counts, costs, timing, and cache metrics from SDK result messages
- Usage logs stored in ~/.claude-mem/usage-logs/ (one file per day)
- Added analyze-usage.js script for analyzing usage patterns

Usage data captured:
- Token counts (input, output, cache creation, cache read)
- Total cost in USD per API call
- Duration metrics (total and API time)
- Number of turns per session
- Session and project attribution

Analysis script features:
- Aggregates totals by project and model
- Shows cache hit rates and savings
- Displays cost breakdowns and averages
- npm scripts: usage:analyze and usage:today

Files:
- src/utils/usage-logger.ts (new)
- src/services/worker-service.ts (modified - captures SDK result messages)
- scripts/analyze-usage.js (new)
- package.json (added usage:* npm scripts)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-11-02 21:53:42 -05:00
Alex Newman a3b0b70a98 feat: Add functionality to locate Claude Code executable
- Implemented `findClaudePath` function to determine the path of the Claude executable using environment variables or system commands (`which` for Unix/Mac and `where` for Windows).
- Integrated the `findClaudePath` function into the SDK worker and worker service to ensure the correct executable path is used when running the SDK agent.
- Enhanced error handling and logging for better debugging and user feedback regarding the executable path.
2025-10-25 14:53:59 -04:00
Alex Newman 81f8aa7eef worker 2025-10-25 14:45:31 -04:00
Alex Newman 8f8649c2a8 worker 2025-10-25 14:41:16 -04:00
Alex Newman d8d4d2464f worker 2025-10-25 14:38:06 -04:00
Alex Newman 9d32629f9d worker 2025-10-25 14:35:56 -04:00
Alex Newman 6a007a26fe Remove hardcoded path to Claude executable in WorkerService.runSDKAgent method 2025-10-25 14:33:16 -04:00
Alex Newman 22777bdcfe Update default path for Claude Code executable in SDKWorker and WorkerService 2025-10-25 14:28:02 -04:00
Alex Newman d7946522e9 Add configurable path for Claude Code executable in SDKWorker and WorkerService 2025-10-25 14:24:17 -04:00
copilot-swe-agent[bot] b88ce840fa Remove hardcoded Claude executable path and defensive checks
Co-authored-by: thedotmack <683968+thedotmack@users.noreply.github.com>
2025-10-25 06:19:30 +00:00
Alex Newman 81fdf28347 Fix critical bug: getSessionById missing claude_session_id
Critical bugfix for NOT NULL constraint violation.

Problem:
- Worker service calls getSessionById(sessionDbId) to fetch session data
- Worker then uses dbSession.claude_session_id to create ActiveSession
- But getSessionById was NOT selecting claude_session_id from database
- Result: claudeSessionId = undefined in worker
- Caused: "NOT NULL constraint failed: sdk_sessions.claude_session_id" errors
- Impact: Observations and summaries couldn't be stored

Root cause:
- SessionStore.getSessionById() SQL query missing claude_session_id column
- Line 710-713: "SELECT id, sdk_session_id, project, user_prompt"
- Should be: "SELECT id, claude_session_id, sdk_session_id, project, user_prompt"

Fix:
- Added claude_session_id to SELECT query in getSessionById
- Updated return type to include claude_session_id: string
- Now worker correctly receives claude_session_id from database
- Session ID from hook flows properly through entire system

Files changed:
- src/services/sqlite/SessionStore.ts (getSessionById method)

Testing:
- Build succeeded
- Ready for PM2 restart and live testing

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-24 22:18:03 -04:00
Alex Newman b3a565c448 Refactor buildSummaryPrompt to clarify summary instructions and improve user guidance 2025-10-24 22:05:23 -04:00
Alex Newman 5b28c23b20 Refactor WorkerService to use claude_session_id directly from the database
- Updated session initialization to retrieve claude_session_id instead of sdk_session_id.
- Removed redundant comments and code related to sdk_session_id handling.
- Simplified session creation logic by directly using values from the database.
- Cleaned up message handling logic to focus on assistant messages and removed unnecessary checks for system init messages.
2025-10-24 21:54:07 -04:00
Alex Newman f4217cb2b9 Enhance logging in WorkerService for better debugging and summary tracking
- Added logging of received content length and a preview for debugging purposes.
- Introduced detailed logging for summary parsing, including flags for summary components.
- Improved warning logging when no summary tags are found, including a content sample.
- Updated success message for stored summaries to be more descriptive.
2025-10-24 21:48:11 -04:00
Alex Newman e7252c8999 Refactor WorkerService to always store observations and summaries using claudeSessionId 2025-10-24 21:45:52 -04:00
Alex Newman 74637705d7 Release v4.2.7: Enhanced data quality and comprehensive testing
Improvements:
- Enhanced null handling for empty/whitespace fields
- Ensures clean null values in database instead of empty strings
- Improves query efficiency and data consistency

Testing:
- Added comprehensive regression test suite (49 tests)
- Tests v4.2.5 summary fixes and v4.2.6 observation fixes
- Tests edge cases: missing fields, empty fields, whitespace
- New test script: npm run test:parser
- All tests passing with 100% coverage

Code Quality:
- Removed unused extractFileArray() function
- Improved function documentation
- TypeScript diagnostics clean

Technical Details:
- Updated src/sdk/parser.ts extractField function
- Created src/sdk/parser.test.ts regression test suite
- Updated package.json to v4.2.7
- Updated CLAUDE.md with version history
- All changes backward compatible

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-24 21:38:05 -04:00
Alex Newman 322cb94c43 Release v4.2.6: Critical bugfix for observation validation
Critical Bugfix:
- Fixed overly defensive observation validation blocking observations from being saved
- Parser now NEVER skips observations - always saves them
- Invalid or missing type defaults to "change" (generic catch-all type)
- Removed validation requiring title, subtitle, and narrative fields
- Prevents critical data loss - partial observations better than no observations

Impact:
- Before: Missing title, subtitle, OR narrative caused entire observation to be discarded
- After: ALL observations preserved regardless of field completeness
- Even partial observations contain valuable data: concepts, files_read, files_modified, facts
- LLMs make mistakes - system must be resilient and save everything
- Consistent with v4.2.5 summary fix

Technical changes:
- Updated src/sdk/parser.ts:52-67 to never skip observations
- Uses "change" as fallback type for invalid/missing types (no schema change)
- Updated ParsedObservation interface to allow null for title, subtitle, narrative
- Updated SessionStore.storeObservation signature to accept nullable fields
- Updated built worker-service.cjs
- Bumped version to 4.2.6 in all metadata files

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-24 21:25:44 -04:00
Alex Newman 3846d66ccc Refactor parseSummary to always save summary regardless of missing fields
- Removed validation for required fields in parseSummary function.
- Added a note emphasizing the importance of saving the summary even if some fields are missing.
2025-10-24 21:13:43 -04:00
Alex Newman d7b9f68d80 Release v4.2.4: Enhanced summary prompt clarity
Improvements:
- Removed optional skip_summary functionality (summaries now always generated)
- Clarified that summaries are mid-session checkpoints, not session endings
- Improved request field instructions to better form descriptive titles
- Changed wording from "discovered" to "learned" for consistency

Technical changes:
- Updated src/sdk/prompts.ts summary prompt
- Removed "WHEN NOT TO SUMMARIZE" section
- Added clarifying footer text about ongoing sessions
- Updated built worker-service.cjs
- Bumped version to 4.2.4 in all metadata files

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-24 20:50:31 -04:00
Alex Newman 0adbf38c39 fix: update getDirname function to support both ESM and CJS contexts 2025-10-23 23:03:48 -04:00
Alex Newman 15362d1aed Refactor getDirname function to only return __dirname for CJS context 2025-10-23 23:00:34 -04:00
Alex Newman 37cd8b8328 fix: Improve summary prompt clarity and consistency in language 2025-10-23 14:21:31 -04:00
Alex Newman 33d2422faa Refactor summary prompt for clarity and emphasis on deliverables
- Changed the title of the summary prompt to "THIS REQUEST'S SUMMARY" for better context.
- Revised instructions to focus on summarizing what was built/fixed/deployed/configured, rather than the observation process.
- Clarified when not to summarize, emphasizing conversational requests and trivial inquiries.
- Updated examples to better illustrate good summary practices.
2025-10-23 14:08:42 -04:00
Alex Newman 2952b7fb8d feat: Add summary skip logic to prevent duplicate and trivial summaries
Added "WHEN NOT TO SUMMARIZE" section to buildSummaryPrompt that instructs the SDK to skip creating summaries for:
- Work already covered in previous prompts (prevents duplicates)
- Conversational banter with no deliverables
- Trivial requests (questions, status checks)
- Meta-discussions about memory system without shipped changes

Implementation:
- src/sdk/prompts.ts: Added WHEN NOT TO SUMMARIZE section with <skip_summary> output format
- src/sdk/parser.ts: Added skip_summary detection before parsing full summary XML
- src/sdk/parser.ts: Fixed observation type validation to include all 6 types (bugfix, feature, refactor, change, discovery, decision)

This should eliminate the duplicate summaries like the three "restore 6 types" summaries we saw for session d9137878.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-22 00:07:30 -04:00
Alex Newman 3cb003f4e4 fix(prompts): restore all 6 observation types for better search granularity
Restored full type system: bugfix, feature, refactor, change, discovery, decision.
This enables more precise search queries like "show all bugfixes in auth" vs generic "show all changes".

Also updated README to reflect current behavior (10 summaries with three-tier verbosity).

Changes:
- prompts.ts: Expanded type field from 3 to 6 types with clear definitions
- CLAUDE.md: Fixed context hook description (3 → 10 summaries)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-21 23:47:01 -04:00
Alex Newman 8b5e5efeb5 fix(prompts): prevent meta-referential descriptions in observations and summaries
Updated SDK prompts to distinguish between deliverables (what was built/shipped) vs meta-operations (what the memory system is doing). This prevents self-referential pollution like "Process tool executions" instead of actual coding tasks like "Fix authentication bug".

Changes:
- buildInitPrompt: Added deliverable-focused framing with contrastive examples
- buildSummaryPrompt: Injected user's original prompt + explicit examples
- Added verb guidance (implemented/fixed/deployed vs analyzed/tracked/stored)
- Added "NOW DOES" present-tense capability framing

Works across all project types: dev, DevOps, docs, infrastructure, research, config.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-21 23:42:12 -04:00
Alex Newman 2314362028 feat(WorkerService): set sdkSessionId from database during session initialization 2025-10-21 22:42:59 -04:00
Alex Newman 3ca5e33b4a feat(worker-service): integrate real Claude Code session ID retrieval
- Added `claudeSessionId` to the `ActiveSession` interface to store the real Claude Code session ID.
- Updated session initialization to fetch the real Claude Code session ID from the database.
- Modified session creation logic to ensure the real session ID is used consistently throughout the worker service.
- Adjusted message generation to utilize the real session ID instead of a placeholder.
2025-10-21 22:20:31 -04:00
Alex Newman 3042de1093 feat(SessionStore): update SDK session creation to include sdk_session_id 2025-10-21 22:16:06 -04:00
Alex Newman f7f62ef3f3 feat(SessionStore): auto-create session records if missing
- Added functionality to automatically create a session record in the database if it does not exist when storing observations or session summaries.
- Updated `storeObservation` and `storeSummary` methods to include checks for existing session records and insert new records as needed.
- Added logging for auto-created session records for better traceability.
2025-10-21 22:13:13 -04:00
Alex Newman 726f167ebf feat: Implement full-text search for user prompts
- Added functionality to save raw user prompts for full-text search in the newHook function.
- Introduced new search endpoint 'search_user_prompts' to retrieve user prompts using FTS5.
- Created UserPromptRow and UserPromptSearchResult types for handling user prompt data.
- Implemented searchUserPrompts method in SessionSearch class to perform FTS5 queries.
- Created user_prompts table with FTS5 support and necessary triggers for data synchronization.
- Updated SessionStore to include methods for saving user prompts and managing the new table.
2025-10-21 22:02:06 -04:00
Alex Newman 635cc87462 fix: Auto-create sessions in worker when observations/summaries arrive
The worker was still returning 404 "Session not found" when trying to add
observations or summaries. This happened if:
- Worker restarted (sessions lost from memory)
- Race condition where observations arrive before /init
- Session continued after /exit

Changes:
- handleObservation: Auto-create session if not in memory map
- handleSummarize: Auto-create session if not in memory map

This completes the "no validation" philosophy: worker accepts any session_id
and auto-creates the session state if needed. Saves ALL data without blockage.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-21 21:46:50 -04:00
Alex Newman c27f07023c fix: Remove all session validation to prevent continuation errors after /exit
Root cause: Hooks provide session_id as the source of truth. We were adding
unnecessary validation (checking if sessions exist, checking status, etc.)
which caused 409 conflicts when continuing sessions after /exit.

Changes:
1. worker-service.ts: Removed 409 "Session already exists" check in handleInit
2. SessionStore.ts: Made createSDKSession idempotent using INSERT OR IGNORE
3. new-hook.ts: Simplified to just call createSDKSession - no findActiveSDKSession,
   no reactivateSession logic, no status management
4. save-hook.ts: Removed session validation, use fixed port instead of session.worker_port
5. summary-hook.ts: Removed session validation, use fixed port instead of session.worker_port

Philosophy: Hooks manage lifecycle, we just save data with whatever session_id
they give us. No validation, no status checks, no guessing.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-21 21:44:58 -04:00
Alex Newman 02bce8a6e6 Enhance observation parsing and querying functionality
- Filter out observation type from concepts array in parseObservations function to ensure types and concepts are treated as separate dimensions. Added logging for removed types.
- Update prompts documentation to clarify that the observation type must not be included in the concepts array.
- Modify search-server to provide clearer guidance on result limits, emphasizing starting with smaller limits to avoid exceeding token limits.
- Refactor SessionSearch methods to accept options for limit, offset, and orderBy parameters, improving flexibility in querying observations by concept and type.
2025-10-21 19:01:11 -04:00
Alex Newman ef572ec032 Refactor summary generation from session-level to request-level
Changed summary prompts to generate discrete per-request summaries instead of cumulative session summaries. This provides better chronological memory where each summary is a clean unit representing one request/response cycle.

Changes:
- Renamed buildFinalizePrompt() to buildSummaryPrompt() in src/sdk/prompts.ts
- Updated prompt text to focus on "THIS REQUEST" rather than "this session"
- Updated all import and function call sites in worker-service.ts and worker.ts
- Added IMPORTANT warning to emphasize request-level scope

Expected behavior: Each summary will now describe only what happened during that specific request, eliminating cumulative recaps.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-21 17:27:14 -04:00
Alex Newman df0f3fd96c Enhance context hook output formatting and add file aggregation for sessions
- Updated contextHook to support colorized output for terminal and JSON format for hooks.
- Introduced ANSI color codes for improved readability in terminal output.
- Modified the output structure to include session details with color formatting.
- Added a new method in SessionStore to aggregate files read and modified from observations for a session.
- Improved error handling for JSON parsing of file data in the new method.
2025-10-21 16:37:16 -04:00
Alex Newman 1ea692e0b0 chore: Update license information and add user settings manager script 2025-10-21 15:12:34 -04:00
Alex Newman c42444e06c Refactor cleanupHook to remove HTTP session deletion and mark session as completed in the database 2025-10-21 00:59:06 -04:00
Alex Newman daf368e343 Refactor code structure for improved readability and maintainability 2025-10-19 01:45:51 -04:00