Add a basic Unix socket server using Bun

- Implemented a simple server using the net module.
- The server listens on a specified socket path.
- Added error handling for server errors.
- Included checks to verify the existence of the socket file.

SOCKET_DEBUG_HYPOTHESES.md

@@ -0,0 +1,232 @@
+# Socket File Not Created - Debug Hypotheses
+
+## Problem Statement
+Worker process logs "Socket server listening: /Users/alexnewman/.claude-mem/worker-28.sock" but the socket file never appears on the filesystem. All connection attempts fail with `ENOENT`.
+
+## Hypotheses (Ordered by Likelihood)
+
+### H1: Worker Process Exits Immediately After Socket Creation
+**Theory:** Worker creates socket, logs message, then crashes/exits before we poll for the file.
+
+**Evidence:**
+- We see the log message
+- Socket never appears
+- No other worker output after "listening" message
+
+**Tests:**
+- Check if worker process is running: `ps aux | grep worker`
+- Add worker exit handlers to see exit code
+- Check if worker.ts crashes after startSocketServer()
+
+**Root Cause Possibilities:**
+- Database query fails in loadSession() (worker.ts:75)
+- SDK agent initialization crashes
+- Unhandled promise rejection in run()
+
+---
+
+### H2: detached=false Kills Worker Prematurely
+**Theory:** `detached: false` causes worker to die when replay script continues execution or when replay script changes process state.
+
+**Evidence:**
+- Production uses `detached: true, stdio: 'ignore'`
+- Replay uses `detached: false, stdio: ['ignore', 'pipe', 'pipe']`
+- Worker might be getting killed by parent process lifecycle
+
+**Tests:**
+- Change to `detached: true, stdio: 'ignore', worker.unref()`
+- Check worker persists: `ps aux | grep worker` after spawn
+
+**Expected Fix:**
+- Worker should persist independently
+- Socket should remain available
+
+---
+
+### H3: stdio Piping Interferes with Socket Creation
+**Theory:** Piping stdout/stderr (`stdio: ['ignore', 'pipe', 'pipe']`) prevents proper socket file creation or causes worker to hang.
+
+**Evidence:**
+- Production uses `stdio: 'ignore'`
+- We're trying to capture output with pipes
+- This might interfere with Unix domain socket operations
+
+**Tests:**
+- Change to `stdio: 'ignore'` (no piping)
+- Worker won't output to our console but should work
+
+---
+
+### H4: Socket Path Mismatch
+**Theory:** Worker creates socket at different path than replay script expects.
+
+**Evidence:**
+- getWorkerSocketPath(sessionId) used in both places
+- Both should resolve to ~/.claude-mem/worker-<id>.sock
+- But maybe DATA_DIR differs between environments
+
+**Tests:**
+- Log actual socketPath in worker: `console.error('Creating socket at:', this.socketPath)`
+- List all sockets: `ls -la ~/.claude-mem/*.sock`
+- Check if socket appears elsewhere: `find /tmp -name "worker-*.sock"`
+
+**Root Cause Possibilities:**
+- CLAUDE_MEM_DATA_DIR environment variable difference
+- Worker started with different env
+
+---
+
+### H5: Permissions Issue
+**Theory:** Worker can't create socket file due to directory permissions.
+
+**Evidence:**
+- Socket creation might fail silently
+- Worker logs "listening" before checking if socket file was created
+
+**Tests:**
+- Check ~/.claude-mem permissions: `ls -ld ~/.claude-mem`
+- Try creating socket manually: `nc -U ~/.claude-mem/test.sock`
+- Check worker user vs replay script user
+
+**Expected Error:**
+- Worker should throw EACCES or EPERM but we might not see it
+
+---
+
+### H6: Socket Listen Callback Fires Before File Creation
+**Theory:** The server.listen() callback fires and logs "listening" before the socket file actually appears on filesystem.
+
+**Evidence:**
+- Node.js/Bun might call callback before filesystem sync
+- We see log but no file
+
+**Tests:**
+- Add additional wait time after seeing log
+- Add fs.existsSync check inside worker after listen()
+- Increase poll duration/frequency in replay script
+
+---
+
+### H7: CLI Worker Command Routing Broken
+**Theory:** `dist/claude-mem.min.js worker <sessionId>` doesn't properly route to worker.ts main().
+
+**Evidence:**
+- cli.ts has .command('worker') handler
+- Handler imports and calls main() from sdk/worker.ts
+- But bundling might break this
+
+**Tests:**
+- Run directly: `dist/claude-mem.min.js worker 28`
+- Check if worker main() is actually called
+- Add console.error at top of worker.ts main()
+
+**Root Cause Possibilities:**
+- Bundle doesn't include worker code
+- Import path broken in minified CLI
+- Commander routing fails
+
+---
+
+### H8: Database Session Not Found by Worker
+**Theory:** Worker can't find session in database, exits early.
+
+**Evidence:**
+- loadSession() query might return null
+- Code checks `if (!session) { exit(1) }` (worker.ts:76-79)
+- But we'd expect to see error log
+
+**Tests:**
+- Verify session exists before spawn: `SELECT * FROM sdk_sessions WHERE id = ?`
+- Add debug log in loadSession() before query
+- Check DB file path matches
+
+---
+
+### H9: Socket File Created Then Immediately Deleted
+**Theory:** Socket is created but something deletes it (cleanup from previous run, OS, etc).
+
+**Evidence:**
+- Old socket file might exist and get unlinked (worker.ts:110-112)
+- Maybe multiple workers spawning
+
+**Tests:**
+- Check for multiple worker processes: `ps aux | grep worker`
+- Watch filesystem in real-time: `watch ls -la ~/.claude-mem/`
+- Add delay before cleanup code runs
+
+---
+
+### H10: Bun vs Node Runtime Issue
+**Theory:** Worker runs under different runtime than expected, causing socket issues.
+
+**Evidence:**
+- Replay script uses bun: `#!/usr/bin/env bun`
+- Worker spawned via CLI which uses node: `#!/usr/bin/env node`
+- Runtime difference might affect socket creation
+
+**Tests:**
+- Spawn with explicit bun: `bun dist/claude-mem.min.js worker 28`
+- Or spawn with explicit node
+- Check if runtime matters for Unix sockets
+
+---
+
+### H11: Race Condition in Socket Server Startup
+**Theory:** server.listen() completes but socket isn't ready for connections yet.
+
+**Evidence:**
+- We poll for 15 seconds
+- Maybe socket file appears but isn't ready
+- Connection attempts might be too early
+
+**Tests:**
+- Increase wait time after socket found
+- Try connecting with retry logic
+- Check socket file permissions/readiness
+
+---
+
+### H12: Worker Logs to Wrong Stream
+**Theory:** Worker logs "listening" to stdout/stderr but then crashes, and we only see initial log.
+
+**Evidence:**
+- console.error used in worker (worker.ts:86)
+- With stdio: ['ignore', 'pipe', 'pipe'], stderr is piped
+- Maybe crash happens but we don't see it
+
+**Tests:**
+- Check full worker output captured
+- Look for crash stack traces
+- Add more logging throughout worker.run()
+
+---
+
+## Recommended Debug Sequence
+
+1. **Change spawn config to match production exactly**
+   - `detached: true`
+   - `stdio: 'ignore'`
+   - `worker.unref()`
+   - This eliminates H2, H3
+
+2. **Check worker process persistence**
+   - `ps aux | grep worker` immediately after spawn
+   - If not running → H1, H7, H8
+   - If running → H4, H5, H6
+
+3. **Check socket file location**
+   - `ls -la ~/.claude-mem/*.sock`
+   - `find /tmp -name "worker-*.sock"`
+   - If found elsewhere → H4
+   - If not found → H1, H5, H6
+
+4. **Run worker directly for debugging**
+   - `dist/claude-mem.min.js worker 28` manually
+   - See full output
+   - Check if socket appears
+
+5. **Add more worker logging**
+   - Log at start of main()
+   - Log after loadSession()
+   - Log after startSocketServer() promise resolves
+   - Log socket path being used

scripts/replay-transcript.ts

@@ -0,0 +1,361 @@
+#!/usr/bin/env bun
+/**
+ * Transcript Replay Tool
+ *
+ * Plays back a Claude Code transcript through the memory system to test:
+ * 1. Tool observation capture
+ * 2. SDK worker processing
+ * 3. SQLite storage
+ * 4. Session summary generation
+ */
+
+import { readFileSync } from 'fs';
+import { join } from 'path';
+import * as net from 'net';
+import { HooksDatabase } from '../src/services/sqlite/HooksDatabase';
+import { getWorkerSocketPath } from '../src/shared/paths';
+import { spawn } from 'child_process';
+
+interface TranscriptLine {
+  type: string;
+  message?: {
+    role?: string;
+    content?: Array<{
+      type: string;
+      name?: string;
+      input?: any;
+      output?: string;
+      id?: string;
+    }>;
+  };
+  uuid?: string;
+  sessionId?: string;
+  timestamp?: string;
+}
+
+interface ToolUse {
+  id: string;
+  name: string;
+  input: any;
+  output?: string;
+  timestamp: string;
+}
+
+/**
+ * Parse transcript JSONL file and extract tool uses with their results
+ */
+function parseTranscript(filePath: string): ToolUse[] {
+  const content = readFileSync(filePath, 'utf-8');
+  const lines = content.trim().split('\n');
+
+  const toolUses: Map<string, ToolUse> = new Map();
+
+  for (const line of lines) {
+    try {
+      const event: TranscriptLine = JSON.parse(line);
+
+      // Capture tool_use from assistant messages
+      if (event.type === 'assistant' && event.message?.content) {
+        for (const item of event.message.content) {
+          if (item.type === 'tool_use' && item.name && item.id) {
+            toolUses.set(item.id, {
+              id: item.id,
+              name: item.name,
+              input: item.input,
+              timestamp: event.timestamp || new Date().toISOString(),
+            });
+          }
+        }
+      }
+
+      // Capture tool_result from user messages
+      // Tool results come in user messages with tool_use_id
+      if (event.type === 'user' && event.message?.content) {
+        const content = event.message.content;
+
+        // Content can be array or single object
+        const items = Array.isArray(content) ? content : [content];
+
+        for (const item of items) {
+          if (item && typeof item === 'object' && 'type' in item && item.type === 'tool_result') {
+            const toolUseId = (item as any).tool_use_id;
+            const toolContent = (item as any).content;
+
+            if (toolUseId) {
+              const toolUse = toolUses.get(toolUseId);
+              if (toolUse) {
+                toolUse.output = toolContent || '';
+              }
+            }
+          }
+        }
+      }
+    } catch (err) {
+      // Skip invalid lines
+      continue;
+    }
+  }
+
+  return Array.from(toolUses.values()).filter(t => t.output !== undefined);
+}
+
+/**
+ * Send observation to SDK worker via Unix socket
+ */
+async function sendObservation(
+  socketPath: string,
+  toolName: string,
+  toolInput: any,
+  toolOutput: string
+): Promise<void> {
+  return new Promise((resolve, reject) => {
+    const client = net.createConnection(socketPath, () => {
+      const message = JSON.stringify({
+        type: 'observation',
+        tool_name: toolName,
+        tool_input: toolInput,
+        tool_output: toolOutput,
+      });
+
+      client.write(message + '\n');
+      client.end();
+      resolve();
+    });
+
+    client.on('error', reject);
+    client.setTimeout(5000);
+    client.on('timeout', () => {
+      client.destroy();
+      reject(new Error('Socket timeout'));
+    });
+  });
+}
+
+/**
+ * Send finalize message to SDK worker
+ */
+async function sendFinalize(socketPath: string): Promise<void> {
+  return new Promise((resolve, reject) => {
+    const client = net.createConnection(socketPath, () => {
+      const message = JSON.stringify({ type: 'finalize' });
+      client.write(message + '\n');
+      client.end();
+      resolve();
+    });
+
+    client.on('error', reject);
+    client.setTimeout(5000);
+    client.on('timeout', () => {
+      client.destroy();
+      reject(new Error('Socket timeout'));
+    });
+  });
+}
+
+/**
+ * Main replay function
+ */
+async function replayTranscript(transcriptPath: string, projectName: string = 'claude-mem-test') {
+  console.log('🎬 Starting transcript replay...\n');
+
+  // Parse transcript
+  console.log(`📖 Parsing transcript: ${transcriptPath}`);
+  const toolUses = parseTranscript(transcriptPath);
+  console.log(`   Found ${toolUses.length} tool uses\n`);
+
+  // Initialize database
+  const hooksDb = new HooksDatabase();
+
+  // Create SDK session
+  console.log('🔧 Creating SDK session...');
+  const claudeSessionId = `replay-${Date.now()}`;
+  const userPrompt = 'Replaying transcript for testing';
+
+  const sessionId = await hooksDb.createSDKSession(
+    claudeSessionId,
+    projectName,
+    userPrompt
+  );
+  console.log(`   Session ID: ${sessionId}`);
+
+  // Verify session was created
+  const verifyQuery = (hooksDb as any).db.query(`
+    SELECT id, claude_session_id, project FROM sdk_sessions WHERE id = ?
+  `);
+  const session = verifyQuery.get(sessionId);
+
+  if (!session) {
+    console.error('   ❌ Session not found in database after creation!');
+    process.exit(1);
+  }
+
+  console.log(`   ✅ Session verified in database\n`);
+
+  // Spawn SDK worker
+  console.log('🚀 Spawning SDK worker...');
+  const socketPath = getWorkerSocketPath(sessionId);
+
+  // Spawn worker exactly as production hooks do
+  const workerPath = join(process.cwd(), 'scripts/hooks/worker.js');
+  const worker = spawn('bun', [workerPath, String(sessionId)], {
+    detached: false, // Keep attached to see errors
+    stdio: ['ignore', 'pipe', 'pipe'] // Pipe output to see what's happening
+  });
+
+  worker.stdout?.on('data', (data) => {
+    console.log(`   [worker stdout] ${data}`);
+  });
+
+  worker.stderr?.on('data', (data) => {
+    console.error(`   [worker stderr] ${data}`);
+  });
+
+  worker.on('exit', (code, signal) => {
+    console.error(`   [worker] Exited with code ${code}, signal ${signal}`);
+  });
+
+  worker.on('error', (err) => {
+    console.error(`\n   [worker] Process error:`, err.message);
+  });
+
+  // Wait for socket to be ready
+  console.log(`   Waiting for socket: ${socketPath}`);
+
+  // Poll for socket existence
+  let socketReady = false;
+  for (let i = 0; i < 30; i++) {
+    await new Promise(resolve => setTimeout(resolve, 500));
+    try {
+      const fs = await import('fs');
+      if (fs.existsSync(socketPath)) {
+        socketReady = true;
+        console.log(`   ✅ Socket ready after ${(i + 1) * 500}ms`);
+        break;
+      }
+    } catch (err) {
+      // Continue waiting
+    }
+  }
+
+  if (!socketReady) {
+    console.log(`   ⚠️  Socket not found after 15s, attempting to connect anyway...`);
+  }
+
+  // Additional wait for worker to be fully initialized
+  await new Promise(resolve => setTimeout(resolve, 2000));
+
+  // Send observations
+  console.log(`\n📤 Sending ${toolUses.length} observations...`);
+  let sent = 0;
+  let failed = 0;
+
+  for (const toolUse of toolUses) {
+    try {
+      await sendObservation(
+        socketPath,
+        toolUse.name,
+        toolUse.input,
+        toolUse.output || ''
+      );
+      sent++;
+      process.stdout.write(`\r   Sent: ${sent}/${toolUses.length}`);
+
+      // Small delay between observations
+      await new Promise(resolve => setTimeout(resolve, 100));
+    } catch (err) {
+      failed++;
+      console.error(`\n   ❌ Failed to send observation: ${err.message}`);
+    }
+  }
+
+  console.log(`\n   ✅ Successfully sent ${sent} observations`);
+  if (failed > 0) {
+    console.log(`   ⚠️  Failed to send ${failed} observations`);
+  }
+
+  // Wait for processing
+  console.log('\n⏳ Waiting for SDK to process observations...');
+  await new Promise(resolve => setTimeout(resolve, 5000));
+
+  // Send finalize
+  console.log('\n🏁 Sending finalize message...');
+  try {
+    await sendFinalize(socketPath);
+    console.log('   ✅ Finalize message sent');
+  } catch (err) {
+    console.error(`   ❌ Failed to send finalize: ${err.message}`);
+  }
+
+  // Wait for summary generation
+  console.log('\n⏳ Waiting for summary generation...');
+  await new Promise(resolve => setTimeout(resolve, 10000));
+
+  // Verify results
+  console.log('\n🔍 Verifying results...\n');
+
+  // Check observations using direct DB query
+  const observations = (hooksDb as any).db.query(`
+    SELECT sdk_session_id, project, text, type, created_at
+    FROM observations
+    WHERE sdk_session_id = (
+      SELECT sdk_session_id FROM sdk_sessions WHERE id = ?
+    )
+    ORDER BY created_at_epoch ASC
+  `).all(sessionId);
+
+  console.log(`   📝 Observations stored: ${observations.length}`);
+
+  if (observations.length > 0) {
+    console.log('   Sample observations:');
+    observations.slice(0, 3).forEach((obs: any, i: number) => {
+      console.log(`      ${i + 1}. [${obs.type}] ${obs.text.substring(0, 60)}...`);
+    });
+  }
+
+  // Check summary using direct DB query
+  const summary = (hooksDb as any).db.query(`
+    SELECT request, investigated, learned, completed, next_steps,
+           files_read, files_edited, notes, created_at
+    FROM session_summaries
+    WHERE sdk_session_id = (
+      SELECT sdk_session_id FROM sdk_sessions WHERE id = ?
+    )
+    LIMIT 1
+  `).get(sessionId);
+
+  if (summary) {
+    console.log(`\n   📋 Summary generated:`);
+    console.log(`      Request: ${(summary as any).request?.substring(0, 60)}...`);
+    console.log(`      Completed: ${(summary as any).completed?.substring(0, 60)}...`);
+    const filesRead = JSON.parse((summary as any).files_read || '[]');
+    const filesEdited = JSON.parse((summary as any).files_edited || '[]');
+    console.log(`      Files read: ${filesRead.length}`);
+    console.log(`      Files edited: ${filesEdited.length}`);
+  } else {
+    console.log(`\n   ⚠️  No summary generated`);
+  }
+
+  // Cleanup (worker is detached and will exit on its own)
+  console.log('\n✅ Replay complete!\n');
+
+  return {
+    sessionId,
+    observationsCount: observations.length,
+    hasSummary: !!summary,
+  };
+}
+
+// CLI interface
+const args = process.argv.slice(2);
+const transcriptPath = args[0] || join(process.cwd(), 'test-data/sample-transcript.jsonl');
+const projectName = args[1] || 'claude-mem-test';
+
+replayTranscript(transcriptPath, projectName)
+  .then((result) => {
+    console.log('Results:', result);
+    process.exit(0);
+  })
+  .catch((err) => {
+    console.error('❌ Replay failed:', err);
+    process.exit(1);
+  });

src/sdk/worker.ts

@@ -33,7 +33,9 @@ type WorkerMessage = ObservationMessage | FinalizeMessage;
  * Main worker process entry point
  */
 export async function main() {
+  console.error('[SDK Worker DEBUG] main() called');
   const sessionDbId = parseInt(process.argv[2], 10);
+  console.error(`[SDK Worker DEBUG] Session DB ID: ${sessionDbId}`);
 
   if (!sessionDbId) {
     console.error('[SDK Worker] Missing session ID argument');
@@ -41,6 +43,7 @@ export async function main() {
   }
 
   const worker = new SDKWorker(sessionDbId);
+  console.error('[SDK Worker DEBUG] SDKWorker instance created');
   await worker.run();
 }
 
@@ -106,12 +109,17 @@ class SDKWorker {
    * Start Unix socket server to receive messages from hooks
    */
   private async startSocketServer(): Promise<void> {
+    console.error(`[SDK Worker DEBUG] Starting socket server...`);
+    console.error(`[SDK Worker DEBUG] Socket path: ${this.socketPath}`);
+
     // Clean up old socket if it exists
     if (existsSync(this.socketPath)) {
+      console.error(`[SDK Worker DEBUG] Removing existing socket`);
       unlinkSync(this.socketPath);
     }
 
     return new Promise((resolve, reject) => {
+      console.error(`[SDK Worker DEBUG] Creating net server...`);
       this.server = net.createServer((socket) => {
         let buffer = '';
 
@@ -147,6 +155,8 @@ class SDKWorker {
       });
 
       this.server.listen(this.socketPath, () => {
+        console.error(`[SDK Worker DEBUG] listen() callback fired`);
+        console.error(`[SDK Worker DEBUG] Checking if socket exists: ${existsSync(this.socketPath)}`);
         resolve();
       });
     });
@@ -183,21 +193,28 @@ class SDKWorker {
    * Run SDK agent with streaming input mode
    */
   private async runSDKAgent(): Promise<void> {
+    // Find Claude Code executable
+    const claudePath = process.env.CLAUDE_CODE_PATH || '/Users/alexnewman/.nvm/versions/node/v24.5.0/bin/claude';
+    console.error(`[SDK Worker DEBUG] About to call query with claudePath: ${claudePath}`);
+
     await query({
-      model: MODEL,
-      messages: () => this.createMessageGenerator(),
-      disallowedTools: DISALLOWED_TOOLS,
-      signal: this.abortController.signal,
-      onSystemInitMessage: (msg) => {
-        // Capture SDK session ID from init message
-        if (msg.session_id) {
-          this.sdkSessionId = msg.session_id;
-          this.db.updateSDKSessionId(this.sessionDbId, msg.session_id);
+      prompt: this.createMessageGenerator(),
+      options: {
+        model: MODEL,
+        disallowedTools: DISALLOWED_TOOLS,
+        abortController: this.abortController,
+        pathToClaudeCodeExecutable: claudePath,
+        onSystemInitMessage: (msg) => {
+          // Capture SDK session ID from init message
+          if (msg.session_id) {
+            this.sdkSessionId = msg.session_id;
+            this.db.updateSDKSessionId(this.sessionDbId, msg.session_id);
+          }
+        },
+        onAgentMessage: (msg) => {
+          // Parse and store observations from agent response
+          this.handleAgentMessage(msg.content);
         }
-      },
-      onAgentMessage: (msg) => {
-        // Parse and store observations from agent response
-        this.handleAgentMessage(msg.content);
       }
     });
   }
@@ -206,11 +223,17 @@ class SDKWorker {
    * Create async message generator for SDK streaming input
    * Now pulls from socket messages instead of polling database
    */
-  private async* createMessageGenerator(): AsyncIterable<{ role: 'user'; content: string }> {
+  private async* createMessageGenerator(): AsyncIterable<{ type: 'user'; message: { role: 'user'; content: string } }> {
     // Yield initial prompt
     const claudeSessionId = `session-${this.sessionDbId}`;
     const initPrompt = buildInitPrompt(this.project, claudeSessionId, this.userPrompt);
-    yield { role: 'user', content: initPrompt };
+    yield {
+      type: 'user',
+      message: {
+        role: 'user',
+        content: initPrompt
+      }
+    };
 
     // Process messages as they arrive via socket
     while (!this.isFinalized) {
@@ -229,7 +252,13 @@ class SDKWorker {
           const session = await this.loadSession();
           if (session) {
             const finalizePrompt = buildFinalizePrompt(session);
-            yield { role: 'user', content: finalizePrompt };
+            yield {
+              type: 'user',
+              message: {
+                role: 'user',
+                content: finalizePrompt
+              }
+            };
           }
           break;
         }
@@ -241,7 +270,13 @@ class SDKWorker {
             tool_input: message.tool_input,
             tool_output: message.tool_output
           });
-          yield { role: 'user', content: observationPrompt };
+          yield {
+            type: 'user',
+            message: {
+              role: 'user',
+              content: observationPrompt
+            }
+          };
         }
       }
     }

test-socket.js

@@ -0,0 +1,18 @@
+#!/usr/bin/env bun
+import net from 'net';
+import { existsSync } from 'fs';
+
+const socketPath = '/Users/alexnewman/.claude-mem/test-bun.sock';
+
+const server = net.createServer(() => {});
+
+server.listen(socketPath, () => {
+  console.log('Server listening');
+  console.log('existsSync says:', existsSync(socketPath));
+  console.log('Checking with ls...');
+});
+
+server.on('error', (err) => {
+  console.error('Error:', err);
+  process.exit(1);
+});