Performance improvements: Token reduction and enhanced summaries (#101)

* refactor: Reduce continuation prompt token usage by 95 lines

Removed redundant instructions from continuation prompt that were originally
added to mitigate a session continuity issue. That issue has since been
resolved, making these detailed instructions unnecessary on every continuation.

Changes:
- Reduced continuation prompt from ~106 lines to ~11 lines (~95 line reduction)
- Changed "User's Goal:" to "Next Prompt in Session:" (more accurate framing)
- Removed redundant WHAT TO RECORD, WHEN TO SKIP, and OUTPUT FORMAT sections
- Kept concise reminder: "Continue generating observations and progress summaries..."
- Initial prompt still contains all detailed instructions

Impact:
- Significant token savings on every continuation prompt
- Faster context injection with no loss of functionality
- Instructions remain comprehensive in initial prompt

Files modified:
- src/sdk/prompts.ts (buildContinuationPrompt function)
- plugin/scripts/worker-service.cjs (compiled output)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: Enhance observation and summary prompts for clarity and token efficiency

* Enhance prompt clarity and instructions in prompts.ts

- Added a reminder to think about instructions before starting work.
- Simplified the continuation prompt instruction by removing "for this ongoing session."

* feat: Enhance settings.json with permissions and deny access to sensitive files

refactor: Remove PLAN-full-observation-display.md and PR_SUMMARY.md as they are no longer needed

chore: Delete SECURITY_SUMMARY.md since it is redundant after recent changes

fix: Update worker-service.cjs to streamline observation generation instructions

cleanup: Remove src-analysis.md and src-tree.md for a cleaner codebase

refactor: Modify prompts.ts to clarify instructions for memory processing

* refactor: Remove legacy worker service implementation

* feat: Enhance summary hook to extract last assistant message and improve logging

- Added function to extract the last assistant message from the transcript.
- Updated summary hook to include last assistant message in the summary request.
- Modified SDKSession interface to store last assistant message.
- Adjusted buildSummaryPrompt to utilize last assistant message for generating summaries.
- Updated worker service and session manager to handle last assistant message in summarize requests.
- Introduced silentDebug utility for improved logging and diagnostics throughout the summary process.

* docs: Add comprehensive implementation plan for ROI metrics feature

Added detailed implementation plan covering:
- Token usage capture from Agent SDK
- Database schema changes (migration #8)
- Discovery cost tracking per observation
- Context hook display with ROI metrics
- Testing and rollout strategy

Timeline: ~20 hours over 4 days
Goal: Empirical data for YC application amendment

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat: Add transcript processing scripts for analysis and formatting

- Implemented `dump-transcript-readable.ts` to generate a readable markdown dump of transcripts, excluding certain entry types.
- Created `extract-rich-context-examples.ts` to extract and showcase rich context examples from transcripts, highlighting user requests and assistant reasoning.
- Developed `format-transcript-context.ts` to format transcript context into a structured markdown format for improved observation generation.
- Added `test-transcript-parser.ts` for validating data extraction from transcript JSONL files, including statistics and error reporting.
- Introduced `transcript-to-markdown.ts` for a complete representation of transcript data in markdown format, showing all context data.
- Enhanced type definitions in `transcript.ts` to support new features and ensure type safety.
- Built `transcript-parser.ts` to handle parsing of transcript JSONL files, including error handling and data extraction methods.

* Refactor hooks and SDKAgent for improved observation handling

- Updated `new-hook.ts` to clean user prompts by stripping leading slashes for better semantic clarity.
- Enhanced `save-hook.ts` to include additional tools in the SKIP_TOOLS set, preventing unnecessary observations from certain command invocations.
- Modified `prompts.ts` to change the structure of observation prompts, emphasizing the observational role and providing a detailed XML output format for observations.
- Adjusted `SDKAgent.ts` to enforce stricter tool usage restrictions, ensuring the memory agent operates solely as an observer without any tool access.

* feat: Enhance session initialization to accept user prompts and prompt numbers

- Updated `handleSessionInit` in `worker-service.ts` to extract `userPrompt` and `promptNumber` from the request body and pass them to `initializeSession`.
- Modified `initializeSession` in `SessionManager.ts` to handle optional `currentUserPrompt` and `promptNumber` parameters.
- Added logic to update the existing session's `userPrompt` and `lastPromptNumber` if a `currentUserPrompt` is provided.
- Implemented debug logging for session initialization and updates to track user prompts and prompt numbers.

---------

Co-authored-by: Claude <noreply@anthropic.com>

src/utils/silent-debug.ts

@@ -0,0 +1,86 @@
+/**
+ * Silent Debug Logger
+ *
+ * NOTE: This utility is to be used like Frank's Red Hot, we put that shit on everything.
+ *
+ * USE THIS INSTEAD OF SILENT FAILURES!
+ * Stop doing this: `const value = something || '';`
+ * Start doing this: `const value = something || silentDebug('something was undefined');`
+ *
+ * Logs to ~/.claude-mem/silent.log and returns a fallback value.
+ * Check logs with `npm run logs:silent`
+ *
+ * Usage:
+ *   import { silentDebug } from '../utils/silent-debug.js';
+ *
+ *   const title = obs.title || silentDebug('obs.title missing', { obs });
+ *   const name = user.name || silentDebug('user.name missing', { user }, 'Anonymous');
+ *
+ *   try {
+ *     doSomething();
+ *   } catch (error) {
+ *     silentDebug('doSomething failed', { error });
+ *   }
+ */
+
+import { appendFileSync } from 'fs';
+import { homedir } from 'os';
+import { join } from 'path';
+
+const LOG_FILE = join(homedir(), '.claude-mem', 'silent.log');
+
+/**
+ * Write a debug message to silent.log and return fallback value
+ * @param message - The message to log
+ * @param data - Optional data to include (will be JSON stringified)
+ * @param fallback - Value to return (defaults to empty string)
+ * @returns The fallback value (for use in || fallbacks)
+ */
+export function silentDebug(message: string, data?: any, fallback: string = ''): string {
+  const timestamp = new Date().toISOString();
+
+  // Capture stack trace to get caller location
+  const stack = new Error().stack || '';
+  const stackLines = stack.split('\n');
+  // Line 0: "Error"
+  // Line 1: "at silentDebug ..."
+  // Line 2: "at <CALLER> ..." <- We want this one
+  const callerLine = stackLines[2] || '';
+  const callerMatch = callerLine.match(/at\s+(?:.*\s+)?\(?([^:]+):(\d+):(\d+)\)?/);
+  const location = callerMatch
+    ? `${callerMatch[1].split('/').pop()}:${callerMatch[2]}`
+    : 'unknown';
+
+  let logLine = `[${timestamp}] [${location}] ${message}`;
+
+  if (data !== undefined) {
+    try {
+      logLine += ` ${JSON.stringify(data)}`;
+    } catch (error) {
+      logLine += ` [stringify error: ${error}]`;
+    }
+  }
+
+  logLine += '\n';
+
+  try {
+    appendFileSync(LOG_FILE, logLine);
+  } catch (error) {
+    // If we can't write to the log file, fail silently (it's a debug utility after all)
+    // Only write to stderr as a last resort
+    console.error('[silent-debug] Failed to write to log:', error);
+  }
+
+  return fallback;
+}
+
+/**
+ * Clear the silent log file
+ */
+export function clearSilentLog(): void {
+  try {
+    appendFileSync(LOG_FILE, `\n${'='.repeat(80)}\n[${new Date().toISOString()}] Log cleared\n${'='.repeat(80)}\n\n`);
+  } catch (error) {
+    // Ignore errors
+  }
+}