Performance improvements: Token reduction and enhanced summaries (#101)

* refactor: Reduce continuation prompt token usage by 95 lines

Removed redundant instructions from continuation prompt that were originally
added to mitigate a session continuity issue. That issue has since been
resolved, making these detailed instructions unnecessary on every continuation.

Changes:
- Reduced continuation prompt from ~106 lines to ~11 lines (~95 line reduction)
- Changed "User's Goal:" to "Next Prompt in Session:" (more accurate framing)
- Removed redundant WHAT TO RECORD, WHEN TO SKIP, and OUTPUT FORMAT sections
- Kept concise reminder: "Continue generating observations and progress summaries..."
- Initial prompt still contains all detailed instructions

Impact:
- Significant token savings on every continuation prompt
- Faster context injection with no loss of functionality
- Instructions remain comprehensive in initial prompt

Files modified:
- src/sdk/prompts.ts (buildContinuationPrompt function)
- plugin/scripts/worker-service.cjs (compiled output)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: Enhance observation and summary prompts for clarity and token efficiency

* Enhance prompt clarity and instructions in prompts.ts

- Added a reminder to think about instructions before starting work.
- Simplified the continuation prompt instruction by removing "for this ongoing session."

* feat: Enhance settings.json with permissions and deny access to sensitive files

refactor: Remove PLAN-full-observation-display.md and PR_SUMMARY.md as they are no longer needed

chore: Delete SECURITY_SUMMARY.md since it is redundant after recent changes

fix: Update worker-service.cjs to streamline observation generation instructions

cleanup: Remove src-analysis.md and src-tree.md for a cleaner codebase

refactor: Modify prompts.ts to clarify instructions for memory processing

* refactor: Remove legacy worker service implementation

* feat: Enhance summary hook to extract last assistant message and improve logging

- Added function to extract the last assistant message from the transcript.
- Updated summary hook to include last assistant message in the summary request.
- Modified SDKSession interface to store last assistant message.
- Adjusted buildSummaryPrompt to utilize last assistant message for generating summaries.
- Updated worker service and session manager to handle last assistant message in summarize requests.
- Introduced silentDebug utility for improved logging and diagnostics throughout the summary process.

* docs: Add comprehensive implementation plan for ROI metrics feature

Added detailed implementation plan covering:
- Token usage capture from Agent SDK
- Database schema changes (migration #8)
- Discovery cost tracking per observation
- Context hook display with ROI metrics
- Testing and rollout strategy

Timeline: ~20 hours over 4 days
Goal: Empirical data for YC application amendment

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* feat: Add transcript processing scripts for analysis and formatting

- Implemented `dump-transcript-readable.ts` to generate a readable markdown dump of transcripts, excluding certain entry types.
- Created `extract-rich-context-examples.ts` to extract and showcase rich context examples from transcripts, highlighting user requests and assistant reasoning.
- Developed `format-transcript-context.ts` to format transcript context into a structured markdown format for improved observation generation.
- Added `test-transcript-parser.ts` for validating data extraction from transcript JSONL files, including statistics and error reporting.
- Introduced `transcript-to-markdown.ts` for a complete representation of transcript data in markdown format, showing all context data.
- Enhanced type definitions in `transcript.ts` to support new features and ensure type safety.
- Built `transcript-parser.ts` to handle parsing of transcript JSONL files, including error handling and data extraction methods.

* Refactor hooks and SDKAgent for improved observation handling

- Updated `new-hook.ts` to clean user prompts by stripping leading slashes for better semantic clarity.
- Enhanced `save-hook.ts` to include additional tools in the SKIP_TOOLS set, preventing unnecessary observations from certain command invocations.
- Modified `prompts.ts` to change the structure of observation prompts, emphasizing the observational role and providing a detailed XML output format for observations.
- Adjusted `SDKAgent.ts` to enforce stricter tool usage restrictions, ensuring the memory agent operates solely as an observer without any tool access.

* feat: Enhance session initialization to accept user prompts and prompt numbers

- Updated `handleSessionInit` in `worker-service.ts` to extract `userPrompt` and `promptNumber` from the request body and pass them to `initializeSession`.
- Modified `initializeSession` in `SessionManager.ts` to handle optional `currentUserPrompt` and `promptNumber` parameters.
- Added logic to update the existing session's `userPrompt` and `lastPromptNumber` if a `currentUserPrompt` is provided.
- Implemented debug logging for session initialization and updates to track user prompts and prompt numbers.

---------

Co-authored-by: Claude <noreply@anthropic.com>

src/hooks/new-hook.ts

@@ -77,12 +77,16 @@ async function newHook(input?: UserPromptSubmitInput): Promise<void> {
 
   const port = getWorkerPort();
 
+  // Strip leading slash from commands for memory agent
+  // /review 101 → review 101 (more semantic for observations)
+  const cleanedPrompt = prompt.startsWith('/') ? prompt.substring(1) : prompt;
+
   try {
     // Initialize session via HTTP
     const response = await fetch(`http://127.0.0.1:${port}/sessions/${sessionDbId}/init`, {
       method: 'POST',
       headers: { 'Content-Type': 'application/json' },
-      body: JSON.stringify({ project, userPrompt: prompt, promptNumber }),
+      body: JSON.stringify({ project, userPrompt: cleanedPrompt, promptNumber }),
       signal: AbortSignal.timeout(5000)
     });
 

src/hooks/save-hook.ts

@@ -20,7 +20,11 @@ export interface PostToolUseInput {
 
 // Tools to skip (low value or too frequent)
 const SKIP_TOOLS = new Set([
-  'ListMcpResourcesTool'
+  'ListMcpResourcesTool',  // MCP infrastructure
+  'SlashCommand',          // Command invocation (observe what it produces, not the call)
+  'Skill',                 // Skill invocation (observe what it produces, not the call)
+  'TodoWrite',             // Task management meta-tool
+  'AskUserQuestion'        // User interaction, not substantive work
 ]);
 
 /**

src/hooks/summary-hook.ts

@@ -9,6 +9,7 @@ import { SessionStore } from '../services/sqlite/SessionStore.js';
 import { createHookResponse } from './hook-response.js';
 import { logger } from '../utils/logger.js';
 import { ensureWorkerRunning, getWorkerPort } from '../shared/worker-utils.js';
+import { silentDebug } from '../utils/silent-debug.js';
 
 export interface StopInput {
   session_id: string;
@@ -37,12 +38,16 @@ function extractLastUserMessage(transcriptPath: string): string {
     for (let i = lines.length - 1; i >= 0; i--) {
       try {
         const line = JSON.parse(lines[i]);
-        if (line.role === 'user' && line.content) {
+
+        // Claude Code transcript format: {type: "user", message: {role: "user", content: [...]}}
+        if (line.type === 'user' && line.message?.content) {
+          const content = line.message.content;
+
           // Extract text content (handle both string and array formats)
-          if (typeof line.content === 'string') {
-            return line.content;
-          } else if (Array.isArray(line.content)) {
-            const textParts = line.content
+          if (typeof content === 'string') {
+            return content;
+          } else if (Array.isArray(content)) {
+            const textParts = content
               .filter((c: any) => c.type === 'text')
               .map((c: any) => c.text);
             return textParts.join('\n');
@@ -60,6 +65,63 @@ function extractLastUserMessage(transcriptPath: string): string {
   return '';
 }
 
+/**
+ * Extract last assistant message from transcript JSONL file
+ * Filters out system-reminder tags to avoid polluting summaries
+ */
+function extractLastAssistantMessage(transcriptPath: string): string {
+  if (!transcriptPath || !existsSync(transcriptPath)) {
+    return '';
+  }
+
+  try {
+    const content = readFileSync(transcriptPath, 'utf-8').trim();
+    if (!content) {
+      return '';
+    }
+
+    const lines = content.split('\n');
+
+    // Parse JSONL and find last assistant message
+    for (let i = lines.length - 1; i >= 0; i--) {
+      try {
+        const line = JSON.parse(lines[i]);
+
+        // Claude Code transcript format: {type: "assistant", message: {role: "assistant", content: [...]}}
+        if (line.type === 'assistant' && line.message?.content) {
+          let text = '';
+          const content = line.message.content;
+
+          // Extract text content (handle both string and array formats)
+          if (typeof content === 'string') {
+            text = content;
+          } else if (Array.isArray(content)) {
+            const textParts = content
+              .filter((c: any) => c.type === 'text')
+              .map((c: any) => c.text);
+            text = textParts.join('\n');
+          }
+
+          // Filter out system-reminder tags and their content
+          text = text.replace(/<system-reminder>[\s\S]*?<\/system-reminder>/g, '');
+
+          // Clean up excessive whitespace
+          text = text.replace(/\n{3,}/g, '\n\n').trim();
+
+          return text;
+        }
+      } catch (parseError) {
+        // Skip malformed lines
+        continue;
+      }
+    }
+  } catch (error) {
+    logger.error('HOOK', 'Failed to read transcript', { transcriptPath }, error as Error);
+  }
+
+  return '';
+}
+
 /**
  * Summary Hook Main Logic
  */
@@ -78,18 +140,50 @@ async function summaryHook(input?: StopInput): Promise<void> {
   // Get or create session
   const sessionDbId = db.createSDKSession(session_id, '', '');
   const promptNumber = db.getPromptCounter(sessionDbId);
+
+  // DIAGNOSTIC: Check session and observations
+  const sessionInfo = db.db.prepare(`
+    SELECT id, claude_session_id, sdk_session_id, project
+    FROM sdk_sessions WHERE id = ?
+  `).get(sessionDbId) as any;
+
+  const obsCount = db.db.prepare(`
+    SELECT COUNT(*) as count
+    FROM observations
+    WHERE sdk_session_id = ?
+  `).get(sessionInfo?.sdk_session_id) as { count: number };
+
+  silentDebug('[summary-hook] Session diagnostics', {
+    claudeSessionId: session_id,
+    sessionDbId,
+    sdkSessionId: sessionInfo?.sdk_session_id,
+    project: sessionInfo?.project,
+    promptNumber,
+    observationCount: obsCount?.count || 0,
+    transcriptPath: input.transcript_path
+  });
+
   db.close();
 
   const port = getWorkerPort();
 
-  // Extract last user message from transcript
+  // Extract last user AND assistant messages from transcript
   const lastUserMessage = extractLastUserMessage(input.transcript_path || '');
+  const lastAssistantMessage = extractLastAssistantMessage(input.transcript_path || '');
+
+  silentDebug('[summary-hook] Extracted messages', {
+    hasLastUserMessage: !!lastUserMessage,
+    hasLastAssistantMessage: !!lastAssistantMessage,
+    lastAssistantPreview: lastAssistantMessage.substring(0, 200),
+    lastAssistantLength: lastAssistantMessage.length
+  });
 
   logger.dataIn('HOOK', 'Stop: Requesting summary', {
     sessionId: sessionDbId,
     workerPort: port,
     promptNumber,
-    hasLastUserMessage: !!lastUserMessage
+    hasLastUserMessage: !!lastUserMessage,
+    hasLastAssistantMessage: !!lastAssistantMessage
   });
 
   try {
@@ -98,7 +192,8 @@ async function summaryHook(input?: StopInput): Promise<void> {
       headers: { 'Content-Type': 'application/json' },
       body: JSON.stringify({
         prompt_number: promptNumber,
-        last_user_message: lastUserMessage
+        last_user_message: lastUserMessage,
+        last_assistant_message: lastAssistantMessage
       }),
       signal: AbortSignal.timeout(2000)
     });

src/sdk/parser.ts

@@ -17,11 +17,11 @@ export interface ParsedObservation {
 }
 
 export interface ParsedSummary {
-  request: string;
-  investigated: string;
-  learned: string;
-  completed: string;
-  next_steps: string;
+  request: string | null;
+  investigated: string | null;
+  learned: string | null;
+  completed: string | null;
+  next_steps: string | null;
   notes: string | null;
 }
 

src/sdk/prompts.ts

@@ -18,6 +18,7 @@ export interface SDKSession {
   project: string;
   user_prompt: string;
   last_user_message?: string;
+  last_assistant_message?: string;
 }
 
 /**
@@ -28,8 +29,12 @@ export function buildInitPrompt(project: string, sessionId: string, userPrompt:
 
 CRITICAL: Record what was LEARNED/BUILT/FIXED/DEPLOYED/CONFIGURED, not what you (the observer) are doing.
 
-User's Goal: ${userPrompt}
-Date: ${new Date().toISOString().split('T')[0]}
+You do not have access to tools. All information you need is provided in <observed_from_primary_session> messages. Create observations from what you observe - no investigation needed.
+
+<observed_from_primary_session>
+  <user_request>${userPrompt}</user_request>
+  <requested_at>${new Date().toISOString().split('T')[0]}</requested_at>
+</observed_from_primary_session>
 
 Your job is to monitor a different Claude Code session happening RIGHT NOW, with the goal of creating observations and progress summaries as the work is being done LIVE by the user. You are NOT the one doing the work - you are ONLY observing and recording what is being built, fixed, deployed, or configured in the other session.
 
@@ -128,7 +133,11 @@ Output observations using this XML structure:
 </observation>
 \`\`\`
 
-IMPORTANT! DO NOT do any work other than generate the OBSERVATIONS or PROGRESS SUMMARIES - and remember that you are a memory agent designed to summarize a DIFFERENT claude code session, not this one. Never reference yourself or your own actions. Never output anything other than the XML structures defined for observations and summaries. All other output is ignored and would be better left unsaid.
+IMPORTANT! DO NOT do any work right now other than generating this OBSERVATIONS from tool use messages - and remember that you are a memory agent designed to summarize a DIFFERENT claude code session, not this one. 
+
+Never reference yourself or your own actions. Do not output anything other than the observation content formatted in the XML structure above. All other output is ignored by the system, and the system has been designed to be smart about token usage. Please spend your tokens wisely on useful observations. 
+
+Remember that we record these observations as a way of helping us stay on track with our progress, and to help us keep important decisions and changes at the forefront of our minds! :) Thank you so much for your help!
 
 MEMORY PROCESSING START
 =======================`;
@@ -154,30 +163,30 @@ export function buildObservationPrompt(obs: Observation): string {
     toolOutput = obs.tool_output;  // If parse fails, use raw value
   }
 
-  return `<tool_used>
-  <tool_name>${obs.tool_name}</tool_name>
-  <tool_time>${new Date(obs.created_at_epoch).toISOString()}</tool_time>${obs.cwd ? `\n  <tool_cwd>${obs.cwd}</tool_cwd>` : ''}
-  <tool_input>${JSON.stringify(toolInput, null, 2)}</tool_input>
-  <tool_output>${JSON.stringify(toolOutput, null, 2)}</tool_output>
-</tool_used>`;
+  return `<observed_from_primary_session>
+  <what_happened>${obs.tool_name}</what_happened>
+  <occurred_at>${new Date(obs.created_at_epoch).toISOString()}</occurred_at>${obs.cwd ? `\n  <working_directory>${obs.cwd}</working_directory>` : ''}
+  <parameters>${JSON.stringify(toolInput, null, 2)}</parameters>
+  <outcome>${JSON.stringify(toolOutput, null, 2)}</outcome>
+</observed_from_primary_session>`;
 }
 
 /**
  * Build prompt to generate progress summary
  */
 export function buildSummaryPrompt(session: SDKSession): string {
-  const lastUserMessage = session.last_user_message || '';
+  const lastAssistantMessage = session.last_assistant_message || '';
 
   return `PROGRESS SUMMARY CHECKPOINT
 ===========================
 Write progress notes of what was done, what was learned, and what's next. This is a checkpoint to capture progress so far. The session is ongoing - you may receive more requests and tool executions after this summary. Write "next_steps" as the current trajectory of work (what's actively being worked on or coming up next), not as post-session future work. Always write at least a minimal summary explaining current progress, even if work is still in early stages, so that users see a summary output tied to each request.
 
-Last User Message:
-${lastUserMessage}
+Claude's Full Response to User:
+${lastAssistantMessage}
 
 Respond in this XML format:
 <summary>
-  <request>[Short title related to the last user message above]</request>
+  <request>[Short title capturing the user's request AND the substance of what was discussed/done]</request>
   <investigated>[What has been explored so far? What was examined?]</investigated>
   <learned>[What have you learned about how things work?]</learned>
   <completed>[What work has been completed so far? What has shipped or changed?]</completed>
@@ -185,7 +194,11 @@ Respond in this XML format:
   <notes>[Additional insights or observations about the current progress]</notes>
 </summary>
 
-IMPORTANT! DO NOT do any work other than generate the PROGRESS SUMMARY  - and remember that you are a memory agent designed to summarize a DIFFERENT claude code session, not this one. Never reference yourself or your own actions. Never output anything other than the XML structures defined for observations and summaries. All other output is ignored and would be better left unsaid.`;
+IMPORTANT! DO NOT do any work right now other than generating this next PROGRESS SUMMARY - and remember that you are a memory agent designed to summarize a DIFFERENT claude code session, not this one.
+
+Never reference yourself or your own actions. Do not output anything other than the summary content formatted in the XML structure above. All other output is ignored by the system, and the system has been designed to be smart about token usage. Please spend your tokens wisely on useful summary content.
+
+Thank you, this summary will be very useful for keeping track of our progress!`;
 }
 
 /**
@@ -210,43 +223,17 @@ IMPORTANT! DO NOT do any work other than generate the PROGRESS SUMMARY  - and re
  * First prompt: Uses buildInitPrompt instead (promptNumber === 1)
  */
 export function buildContinuationPrompt(userPrompt: string, promptNumber: number, claudeSessionId: string): string {
-  return `This is continuation prompt #${promptNumber} for session ${claudeSessionId} that you're observing. 
+  return `
+Hello memory agent, you are continuing to observe the primary Claude session.
 
-CRITICAL: Record what was LEARNED/BUILT/FIXED/DEPLOYED/CONFIGURED, not what you (the observer) are doing.
+<observed_from_primary_session>
+  <user_request>${userPrompt}</user_request>
+  <requested_at>${new Date().toISOString().split('T')[0]}</requested_at>
+</observed_from_primary_session>
 
-User's Goal: ${userPrompt}
-Date: ${new Date().toISOString().split('T')[0]}
+You do not have access to tools. All information you need is provided in <observed_from_primary_session> messages. Create observations from what you observe - no investigation needed.
 
-Your job is to continue monitoring the different Claude Code session happening RIGHT NOW, with the goal of creating observations and a progress summary as the work is being done LIVE by the user. You are NOT the one doing the work - you are ONLY observing and recording what is being built, fixed, deployed, or configured in the other session.
-
-WHAT TO RECORD
---------------
-Focus on deliverables and capabilities:
-- What the system NOW DOES differently (new capabilities)
-- What shipped to users/production (features, fixes, configs, docs)
-- Changes in technical domains (auth, data, UI, infra, DevOps, docs)
-
-Use verbs like: implemented, fixed, deployed, configured, migrated, optimized, added, refactored
-
-✅ GOOD EXAMPLES (describes what was built):
-- "Authentication now supports OAuth2 with PKCE flow"
-- "Deployment pipeline runs canary releases with auto-rollback"
-- "Database indexes optimized for common query patterns"
-
-❌ BAD EXAMPLES (describes observation process - DO NOT DO THIS):
-- "Analyzed authentication implementation and stored findings"
-- "Tracked deployment steps and logged outcomes"
-- "Monitored database performance and recorded metrics"
-
-WHEN TO SKIP
-------------
-Skip routine operations:
-- Empty status checks
-- Package installations with no errors
-- Simple file listings
-- Repetitive operations you've already documented
-- If file related research comes back as empty or not found
-- **No output necessary if skipping.**
+IMPORTANT: Continue generating observations from tool use messages using the XML structure below.
 
 OUTPUT FORMAT
 -------------
@@ -309,9 +296,10 @@ Output observations using this XML structure:
 </observation>
 \`\`\`
 
-IMPORTANT! DO NOT do any work other than generate the OBSERVATIONS or PROGRESS SUMMARIES - and remember that you are a memory agent designed to summarize a DIFFERENT claude code session, not this one. Never reference yourself or your own actions. Never output anything other than the XML structures defined for observations and summaries. All other output is ignored and would be better left unsaid.
+Never reference yourself or your own actions. Do not output anything other than the observation content formatted in the XML structure above. All other output is ignored by the system, and the system has been designed to be smart about token usage. Please spend your tokens wisely on useful observations.
 
-MEMORY PROCESSING START
-=======================`;
+Remember that we record these observations as a way of helping us stay on track with our progress, and to help us keep important decisions and changes at the forefront of our minds! :) Thank you so much for your continued help!
 
-}
+MEMORY PROCESSING CONTINUED
+===========================`;
+} 

src/servers/search-server.ts

@@ -137,7 +137,7 @@ function formatObservationIndex(obs: ObservationSearchResult, index: number): st
  * Format session summary as index entry (title, date, ID only)
  */
 function formatSessionIndex(session: SessionSummarySearchResult, index: number): string {
-  const title = session.request || `Session ${session.sdk_session_id.substring(0, 8)}`;
+  const title = session.request || `Session ${session.sdk_session_id?.substring(0, 8) || 'unknown'}`;
   const date = new Date(session.created_at_epoch).toLocaleString();
 
   return `${index + 1}. ${title}
@@ -229,7 +229,7 @@ function formatObservationResult(obs: ObservationSearchResult): string {
  * Format session summary as text content with metadata
  */
 function formatSessionResult(session: SessionSummarySearchResult): string {
-  const title = session.request || `Session ${session.sdk_session_id.substring(0, 8)}`;
+  const title = session.request || `Session ${session.sdk_session_id?.substring(0, 8) || 'unknown'}`;
 
   // Build content from available fields
   const contentParts: string[] = [];

src/services/worker-service.ts

@@ -289,7 +289,8 @@ export class WorkerService {
   private handleSessionInit(req: Request, res: Response): void {
     try {
       const sessionDbId = parseInt(req.params.sessionDbId, 10);
-      const session = this.sessionManager.initializeSession(sessionDbId);
+      const { userPrompt, promptNumber } = req.body;
+      const session = this.sessionManager.initializeSession(sessionDbId, userPrompt, promptNumber);
 
       // Get the latest user_prompt for this session to sync to Chroma
       const db = this.dbManager.getSessionStore().db;
@@ -450,9 +451,9 @@ export class WorkerService {
   private handleSummarize(req: Request, res: Response): void {
     try {
       const sessionDbId = parseInt(req.params.sessionDbId, 10);
-      const { last_user_message } = req.body;
+      const { last_user_message, last_assistant_message } = req.body;
 
-      this.sessionManager.queueSummarize(sessionDbId, last_user_message);
+      this.sessionManager.queueSummarize(sessionDbId, last_user_message, last_assistant_message);
 
       // CRITICAL: Ensure SDK agent is running to consume the queue
       const session = this.sessionManager.getSession(sessionDbId);

src/services/worker-types.ts

@@ -29,6 +29,7 @@ export interface PendingMessage {
   prompt_number?: number;
   cwd?: string;
   last_user_message?: string;
+  last_assistant_message?: string;
 }
 
 export interface ObservationData {

src/services/worker/SDKAgent.ts

@@ -15,6 +15,7 @@ import { existsSync, readFileSync } from 'fs';
 import { DatabaseManager } from './DatabaseManager.js';
 import { SessionManager } from './SessionManager.js';
 import { logger } from '../../utils/logger.js';
+import { silentDebug } from '../../utils/silent-debug.js';
 import { parseObservations, parseSummary } from '../../sdk/parser.js';
 import { buildInitPrompt, buildObservationPrompt, buildSummaryPrompt, buildContinuationPrompt } from '../../sdk/prompts.js';
 import type { ActiveSession, SDKUserMessage, PendingMessage } from '../worker-types.js';
@@ -43,7 +44,21 @@ export class SDKAgent {
 
       // Get model ID and disallowed tools
       const modelId = this.getModelId();
-      const disallowedTools = ['Bash']; // Prevent infinite loops
+      // Memory agent is OBSERVER ONLY - no tools allowed
+      const disallowedTools = [
+        'Bash',           // Prevent infinite loops
+        'Read',           // No file reading
+        'Write',          // No file writing
+        'Edit',           // No file editing
+        'Grep',           // No code searching
+        'Glob',           // No file pattern matching
+        'WebFetch',       // No web fetching
+        'WebSearch',      // No web searching
+        'Task',           // No spawning sub-agents
+        'NotebookEdit',   // No notebook editing
+        'AskUserQuestion',// No asking questions
+        'TodoWrite'       // No todo management
+      ];
 
       // Create message generator (event-driven)
       const messageGenerator = this.createMessageGenerator(session);
@@ -189,7 +204,8 @@ export class SDKAgent {
               sdk_session_id: session.sdkSessionId,
               project: session.project,
               user_prompt: session.userPrompt,
-              last_user_message: message.last_user_message || ''
+              last_user_message: message.last_user_message || '',
+              last_assistant_message: message.last_assistant_message || ''
             })
           },
           session_id: session.claudeSessionId,
@@ -221,15 +237,16 @@ export class SDKAgent {
         sessionId: session.sessionDbId,
         obsId,
         type: obs.type,
-        title: obs.title.substring(0, 60) + (obs.title.length > 60 ? '...' : ''),
-        files: obs.files?.length || 0,
-        concepts: obs.concepts?.length || 0
+        title: obs.title || silentDebug('obs.title is null', { obsId, type: obs.type }, '(untitled)'),
+        filesRead: obs.files_read?.length ?? (silentDebug('obs.files_read is null/undefined', { obsId }), 0),
+        filesModified: obs.files_modified?.length ?? (silentDebug('obs.files_modified is null/undefined', { obsId }), 0),
+        concepts: obs.concepts?.length ?? (silentDebug('obs.concepts is null/undefined', { obsId }), 0)
       });
 
       // Sync to Chroma with error logging
       const chromaStart = Date.now();
       const obsType = obs.type;
-      const obsTitle = obs.title;
+      const obsTitle = obs.title || silentDebug('obs.title is null for Chroma sync', { obsId, type: obs.type }, '(untitled)');
       this.dbManager.getChromaSync().syncObservation(
         obsId,
         session.claudeSessionId,
@@ -239,21 +256,18 @@ export class SDKAgent {
         createdAtEpoch
       ).then(() => {
         const chromaDuration = Date.now() - chromaStart;
-        const truncatedTitle = obsTitle.length > 50
-          ? obsTitle.substring(0, 50) + '...'
-          : obsTitle;
         logger.debug('CHROMA', 'Observation synced', {
           obsId,
           duration: `${chromaDuration}ms`,
           type: obsType,
-          title: truncatedTitle
+          title: obsTitle
         });
       }).catch(err => {
         logger.error('CHROMA', 'Failed to sync observation', {
           obsId,
           sessionId: session.sessionDbId,
           type: obsType,
-          title: obsTitle.substring(0, 50)
+          title: obsTitle
         }, err);
       });
 
@@ -298,14 +312,14 @@ export class SDKAgent {
       logger.info('SDK', 'Summary saved', {
         sessionId: session.sessionDbId,
         summaryId,
-        request: summary.request.substring(0, 60) + (summary.request.length > 60 ? '...' : ''),
+        request: summary.request || silentDebug('summary.request is null', { summaryId }, '(no request)'),
         hasCompleted: !!summary.completed,
         hasNextSteps: !!summary.next_steps
       });
 
       // Sync to Chroma with error logging
       const chromaStart = Date.now();
-      const summaryRequest = summary.request;
+      const summaryRequest = summary.request || silentDebug('summary.request is null for Chroma sync', { summaryId }, '(no request)');
       this.dbManager.getChromaSync().syncSummary(
         summaryId,
         session.claudeSessionId,
@@ -315,19 +329,16 @@ export class SDKAgent {
         createdAtEpoch
       ).then(() => {
         const chromaDuration = Date.now() - chromaStart;
-        const truncatedRequest = summaryRequest.length > 50
-          ? summaryRequest.substring(0, 50) + '...'
-          : summaryRequest;
         logger.debug('CHROMA', 'Summary synced', {
           summaryId,
           duration: `${chromaDuration}ms`,
-          request: truncatedRequest
+          request: summaryRequest
         });
       }).catch(err => {
         logger.error('CHROMA', 'Failed to sync summary', {
           summaryId,
           sessionId: session.sessionDbId,
-          request: summaryRequest.substring(0, 50)
+          request: summaryRequest
         }, err);
       });
 

src/services/worker/SessionManager.ts

@@ -11,6 +11,7 @@
 import { EventEmitter } from 'events';
 import { DatabaseManager } from './DatabaseManager.js';
 import { logger } from '../../utils/logger.js';
+import { silentDebug } from '../../utils/silent-debug.js';
 import type { ActiveSession, PendingMessage, ObservationData } from '../worker-types.js';
 
 export class SessionManager {
@@ -33,27 +34,61 @@ export class SessionManager {
   /**
    * Initialize a new session or return existing one
    */
-  initializeSession(sessionDbId: number): ActiveSession {
+  initializeSession(sessionDbId: number, currentUserPrompt?: string, promptNumber?: number): ActiveSession {
     // Check if already active
     let session = this.sessions.get(sessionDbId);
     if (session) {
+      // Update userPrompt for continuation prompts
+      if (currentUserPrompt) {
+        silentDebug('[SessionManager] Updating userPrompt for continuation', {
+          sessionDbId,
+          promptNumber,
+          oldPrompt: session.userPrompt.substring(0, 80),
+          newPrompt: currentUserPrompt.substring(0, 80)
+        });
+        session.userPrompt = currentUserPrompt;
+        session.lastPromptNumber = promptNumber || session.lastPromptNumber;
+      } else {
+        silentDebug('[SessionManager] No currentUserPrompt provided for existing session', {
+          sessionDbId,
+          promptNumber,
+          usingCachedPrompt: session.userPrompt.substring(0, 80)
+        });
+      }
       return session;
     }
 
     // Fetch from database
     const dbSession = this.dbManager.getSessionById(sessionDbId);
 
+    // Use currentUserPrompt if provided, otherwise fall back to database (first prompt)
+    const userPrompt = currentUserPrompt || dbSession.user_prompt;
+
+    if (!currentUserPrompt) {
+      silentDebug('[SessionManager] No currentUserPrompt provided for new session, using database', {
+        sessionDbId,
+        promptNumber,
+        dbPrompt: dbSession.user_prompt.substring(0, 80)
+      });
+    } else {
+      silentDebug('[SessionManager] Initializing session with fresh userPrompt', {
+        sessionDbId,
+        promptNumber,
+        userPrompt: currentUserPrompt.substring(0, 80)
+      });
+    }
+
     // Create active session
     session = {
       sessionDbId,
       claudeSessionId: dbSession.claude_session_id,
       sdkSessionId: null,
       project: dbSession.project,
-      userPrompt: dbSession.user_prompt,
+      userPrompt,
       pendingMessages: [],
       abortController: new AbortController(),
       generatorPromise: null,
-      lastPromptNumber: this.dbManager.getSessionStore().getPromptCounter(sessionDbId),
+      lastPromptNumber: promptNumber || this.dbManager.getSessionStore().getPromptCounter(sessionDbId),
       startTime: Date.now()
     };
 
@@ -123,7 +158,7 @@ export class SessionManager {
    * Queue a summarize request (zero-latency notification)
    * Auto-initializes session if not in memory but exists in database
    */
-  queueSummarize(sessionDbId: number, lastUserMessage: string): void {
+  queueSummarize(sessionDbId: number, lastUserMessage: string, lastAssistantMessage?: string): void {
     // Auto-initialize from database if needed (handles worker restarts)
     let session = this.sessions.get(sessionDbId);
     if (!session) {
@@ -134,7 +169,8 @@ export class SessionManager {
 
     session.pendingMessages.push({
       type: 'summarize',
-      last_user_message: lastUserMessage
+      last_user_message: lastUserMessage,
+      last_assistant_message: lastAssistantMessage
     });
 
     const afterDepth = session.pendingMessages.length;

src/types/transcript.ts

@@ -0,0 +1,174 @@
+/**
+ * TypeScript types for Claude Code transcript JSONL structure
+ * Based on Python Pydantic models from docs/context/cc-transcript-model-example.py
+ */
+
+export interface TodoItem {
+  id: string;
+  content: string;
+  status: 'pending' | 'in_progress' | 'completed';
+  priority: 'high' | 'medium' | 'low';
+}
+
+export interface UsageInfo {
+  input_tokens?: number;
+  cache_creation_input_tokens?: number;
+  cache_read_input_tokens?: number;
+  output_tokens?: number;
+  service_tier?: string;
+  server_tool_use?: any;
+}
+
+export interface TextContent {
+  type: 'text';
+  text: string;
+}
+
+export interface ToolUseContent {
+  type: 'tool_use';
+  id: string;
+  name: string;
+  input: Record<string, any>;
+}
+
+export interface ToolResultContent {
+  type: 'tool_result';
+  tool_use_id: string;
+  content: string | Array<Record<string, any>>;
+  is_error?: boolean;
+}
+
+export interface ThinkingContent {
+  type: 'thinking';
+  thinking: string;
+  signature?: string;
+}
+
+export interface ImageSource {
+  type: 'base64';
+  media_type: string;
+  data: string;
+}
+
+export interface ImageContent {
+  type: 'image';
+  source: ImageSource;
+}
+
+export type ContentItem =
+  | TextContent
+  | ToolUseContent
+  | ToolResultContent
+  | ThinkingContent
+  | ImageContent;
+
+export interface UserMessage {
+  role: 'user';
+  content: string | ContentItem[];
+}
+
+export interface AssistantMessage {
+  id: string;
+  type: 'message';
+  role: 'assistant';
+  model: string;
+  content: ContentItem[];
+  stop_reason?: string;
+  stop_sequence?: string;
+  usage?: UsageInfo;
+}
+
+export interface FileInfo {
+  filePath: string;
+  content: string;
+  numLines: number;
+  startLine: number;
+  totalLines: number;
+}
+
+export interface FileReadResult {
+  type: 'text';
+  file: FileInfo;
+}
+
+export interface CommandResult {
+  stdout: string;
+  stderr: string;
+  interrupted: boolean;
+  isImage: boolean;
+}
+
+export interface TodoResult {
+  oldTodos: TodoItem[];
+  newTodos: TodoItem[];
+}
+
+export interface EditResult {
+  oldString?: string;
+  newString?: string;
+  replaceAll?: boolean;
+  originalFile?: string;
+  structuredPatch?: any;
+  userModified?: boolean;
+}
+
+export type ToolUseResult =
+  | string
+  | TodoItem[]
+  | FileReadResult
+  | CommandResult
+  | TodoResult
+  | EditResult
+  | ContentItem[];
+
+export interface BaseTranscriptEntry {
+  parentUuid?: string;
+  isSidechain: boolean;
+  userType: string;
+  cwd: string;
+  sessionId: string;
+  version: string;
+  uuid: string;
+  timestamp: string;
+  isMeta?: boolean;
+}
+
+export interface UserTranscriptEntry extends BaseTranscriptEntry {
+  type: 'user';
+  message: UserMessage;
+  toolUseResult?: ToolUseResult;
+}
+
+export interface AssistantTranscriptEntry extends BaseTranscriptEntry {
+  type: 'assistant';
+  message: AssistantMessage;
+  requestId?: string;
+}
+
+export interface SummaryTranscriptEntry {
+  type: 'summary';
+  summary: string;
+  leafUuid: string;
+  cwd?: string;
+}
+
+export interface SystemTranscriptEntry extends BaseTranscriptEntry {
+  type: 'system';
+  content: string;
+  level?: string; // 'warning', 'info', 'error'
+}
+
+export interface QueueOperationTranscriptEntry {
+  type: 'queue-operation';
+  operation: 'enqueue' | 'dequeue';
+  timestamp: string;
+  sessionId: string;
+  content?: ContentItem[]; // Only present for enqueue operations
+}
+
+export type TranscriptEntry =
+  | UserTranscriptEntry
+  | AssistantTranscriptEntry
+  | SummaryTranscriptEntry
+  | SystemTranscriptEntry
+  | QueueOperationTranscriptEntry;

src/utils/silent-debug.ts

@@ -0,0 +1,86 @@
+/**
+ * Silent Debug Logger
+ *
+ * NOTE: This utility is to be used like Frank's Red Hot, we put that shit on everything.
+ *
+ * USE THIS INSTEAD OF SILENT FAILURES!
+ * Stop doing this: `const value = something || '';`
+ * Start doing this: `const value = something || silentDebug('something was undefined');`
+ *
+ * Logs to ~/.claude-mem/silent.log and returns a fallback value.
+ * Check logs with `npm run logs:silent`
+ *
+ * Usage:
+ *   import { silentDebug } from '../utils/silent-debug.js';
+ *
+ *   const title = obs.title || silentDebug('obs.title missing', { obs });
+ *   const name = user.name || silentDebug('user.name missing', { user }, 'Anonymous');
+ *
+ *   try {
+ *     doSomething();
+ *   } catch (error) {
+ *     silentDebug('doSomething failed', { error });
+ *   }
+ */
+
+import { appendFileSync } from 'fs';
+import { homedir } from 'os';
+import { join } from 'path';
+
+const LOG_FILE = join(homedir(), '.claude-mem', 'silent.log');
+
+/**
+ * Write a debug message to silent.log and return fallback value
+ * @param message - The message to log
+ * @param data - Optional data to include (will be JSON stringified)
+ * @param fallback - Value to return (defaults to empty string)
+ * @returns The fallback value (for use in || fallbacks)
+ */
+export function silentDebug(message: string, data?: any, fallback: string = ''): string {
+  const timestamp = new Date().toISOString();
+
+  // Capture stack trace to get caller location
+  const stack = new Error().stack || '';
+  const stackLines = stack.split('\n');
+  // Line 0: "Error"
+  // Line 1: "at silentDebug ..."
+  // Line 2: "at <CALLER> ..." <- We want this one
+  const callerLine = stackLines[2] || '';
+  const callerMatch = callerLine.match(/at\s+(?:.*\s+)?\(?([^:]+):(\d+):(\d+)\)?/);
+  const location = callerMatch
+    ? `${callerMatch[1].split('/').pop()}:${callerMatch[2]}`
+    : 'unknown';
+
+  let logLine = `[${timestamp}] [${location}] ${message}`;
+
+  if (data !== undefined) {
+    try {
+      logLine += ` ${JSON.stringify(data)}`;
+    } catch (error) {
+      logLine += ` [stringify error: ${error}]`;
+    }
+  }
+
+  logLine += '\n';
+
+  try {
+    appendFileSync(LOG_FILE, logLine);
+  } catch (error) {
+    // If we can't write to the log file, fail silently (it's a debug utility after all)
+    // Only write to stderr as a last resort
+    console.error('[silent-debug] Failed to write to log:', error);
+  }
+
+  return fallback;
+}
+
+/**
+ * Clear the silent log file
+ */
+export function clearSilentLog(): void {
+  try {
+    appendFileSync(LOG_FILE, `\n${'='.repeat(80)}\n[${new Date().toISOString()}] Log cleared\n${'='.repeat(80)}\n\n`);
+  } catch (error) {
+    // Ignore errors
+  }
+}

src/utils/transcript-parser.ts

@@ -0,0 +1,254 @@
+/**
+ * TranscriptParser - Properly parse Claude Code transcript JSONL files
+ * Handles all transcript entry types based on validated model
+ */
+
+import { readFileSync } from 'fs';
+import type {
+  TranscriptEntry,
+  UserTranscriptEntry,
+  AssistantTranscriptEntry,
+  SummaryTranscriptEntry,
+  SystemTranscriptEntry,
+  QueueOperationTranscriptEntry,
+  ContentItem,
+  TextContent,
+} from '../types/transcript.js';
+
+export interface ParseStats {
+  totalLines: number;
+  parsedEntries: number;
+  failedLines: number;
+  entriesByType: Record<string, number>;
+  failureRate: number;
+}
+
+export class TranscriptParser {
+  private entries: TranscriptEntry[] = [];
+  private parseErrors: Array<{ lineNumber: number; error: string }> = [];
+
+  constructor(transcriptPath: string) {
+    this.parseTranscript(transcriptPath);
+  }
+
+  private parseTranscript(transcriptPath: string): void {
+    const content = readFileSync(transcriptPath, 'utf-8').trim();
+    if (!content) return;
+
+    const lines = content.split('\n');
+
+    lines.forEach((line, index) => {
+      try {
+        const entry = JSON.parse(line) as TranscriptEntry;
+        this.entries.push(entry);
+      } catch (error) {
+        this.parseErrors.push({
+          lineNumber: index + 1,
+          error: error instanceof Error ? error.message : String(error),
+        });
+      }
+    });
+  }
+
+  /**
+   * Get all entries of a specific type
+   */
+  getEntriesByType<T extends TranscriptEntry>(type: T['type']): T[] {
+    return this.entries.filter((e) => e.type === type) as T[];
+  }
+
+  /**
+   * Get all user entries
+   */
+  getUserEntries(): UserTranscriptEntry[] {
+    return this.getEntriesByType<UserTranscriptEntry>('user');
+  }
+
+  /**
+   * Get all assistant entries
+   */
+  getAssistantEntries(): AssistantTranscriptEntry[] {
+    return this.getEntriesByType<AssistantTranscriptEntry>('assistant');
+  }
+
+  /**
+   * Get all summary entries
+   */
+  getSummaryEntries(): SummaryTranscriptEntry[] {
+    return this.getEntriesByType<SummaryTranscriptEntry>('summary');
+  }
+
+  /**
+   * Get all system entries
+   */
+  getSystemEntries(): SystemTranscriptEntry[] {
+    return this.getEntriesByType<SystemTranscriptEntry>('system');
+  }
+
+  /**
+   * Get all queue operation entries
+   */
+  getQueueOperationEntries(): QueueOperationTranscriptEntry[] {
+    return this.getEntriesByType<QueueOperationTranscriptEntry>('queue-operation');
+  }
+
+  /**
+   * Get last entry of a specific type
+   */
+  getLastEntryByType<T extends TranscriptEntry>(type: T['type']): T | null {
+    const entries = this.getEntriesByType<T>(type);
+    return entries.length > 0 ? entries[entries.length - 1] : null;
+  }
+
+  /**
+   * Extract text content from content items
+   */
+  private extractTextFromContent(content: string | ContentItem[]): string {
+    if (typeof content === 'string') {
+      return content;
+    }
+
+    if (Array.isArray(content)) {
+      return content
+        .filter((item): item is TextContent => item.type === 'text')
+        .map((item) => item.text)
+        .join('\n');
+    }
+
+    return '';
+  }
+
+  /**
+   * Get last user message text (finds last entry with actual text content)
+   */
+  getLastUserMessage(): string {
+    const userEntries = this.getUserEntries();
+
+    // Iterate backward to find the last user message with text content
+    for (let i = userEntries.length - 1; i >= 0; i--) {
+      const entry = userEntries[i];
+      if (!entry?.message?.content) continue;
+
+      const text = this.extractTextFromContent(entry.message.content);
+      if (text) return text;
+    }
+
+    return '';
+  }
+
+  /**
+   * Get last assistant message text (finds last entry with text content, with optional system-reminder filtering)
+   */
+  getLastAssistantMessage(filterSystemReminders = true): string {
+    const assistantEntries = this.getAssistantEntries();
+
+    // Iterate backward to find the last assistant message with text content
+    for (let i = assistantEntries.length - 1; i >= 0; i--) {
+      const entry = assistantEntries[i];
+      if (!entry?.message?.content) continue;
+
+      let text = this.extractTextFromContent(entry.message.content);
+      if (!text) continue;
+
+      if (filterSystemReminders) {
+        // Filter out system-reminder tags and their content
+        text = text.replace(/<system-reminder>[\s\S]*?<\/system-reminder>/g, '');
+        // Clean up excessive whitespace
+        text = text.replace(/\n{3,}/g, '\n\n').trim();
+      }
+
+      if (text) return text;
+    }
+
+    return '';
+  }
+
+  /**
+   * Get all tool use operations from assistant entries
+   */
+  getToolUseHistory(): Array<{ name: string; timestamp: string; input: any }> {
+    const toolUses: Array<{ name: string; timestamp: string; input: any }> = [];
+
+    for (const entry of this.getAssistantEntries()) {
+      if (Array.isArray(entry.message.content)) {
+        for (const item of entry.message.content) {
+          if (item.type === 'tool_use') {
+            toolUses.push({
+              name: item.name,
+              timestamp: entry.timestamp,
+              input: item.input,
+            });
+          }
+        }
+      }
+    }
+
+    return toolUses;
+  }
+
+  /**
+   * Get total token usage across all assistant messages
+   */
+  getTotalTokenUsage(): {
+    inputTokens: number;
+    outputTokens: number;
+    cacheCreationTokens: number;
+    cacheReadTokens: number;
+  } {
+    const assistantEntries = this.getAssistantEntries();
+
+    return assistantEntries.reduce(
+      (acc, entry) => {
+        const usage = entry.message.usage;
+        if (usage) {
+          acc.inputTokens += usage.input_tokens || 0;
+          acc.outputTokens += usage.output_tokens || 0;
+          acc.cacheCreationTokens += usage.cache_creation_input_tokens || 0;
+          acc.cacheReadTokens += usage.cache_read_input_tokens || 0;
+        }
+        return acc;
+      },
+      {
+        inputTokens: 0,
+        outputTokens: 0,
+        cacheCreationTokens: 0,
+        cacheReadTokens: 0,
+      }
+    );
+  }
+
+  /**
+   * Get parse statistics
+   */
+  getParseStats(): ParseStats {
+    const entriesByType: Record<string, number> = {};
+
+    for (const entry of this.entries) {
+      entriesByType[entry.type] = (entriesByType[entry.type] || 0) + 1;
+    }
+
+    const totalLines = this.entries.length + this.parseErrors.length;
+
+    return {
+      totalLines,
+      parsedEntries: this.entries.length,
+      failedLines: this.parseErrors.length,
+      entriesByType,
+      failureRate: totalLines > 0 ? this.parseErrors.length / totalLines : 0,
+    };
+  }
+
+  /**
+   * Get parse errors
+   */
+  getParseErrors(): Array<{ lineNumber: number; error: string }> {
+    return this.parseErrors;
+  }
+
+  /**
+   * Get all entries (raw)
+   */
+  getAllEntries(): TranscriptEntry[] {
+    return this.entries;
+  }
+}