feat: Fix observation timestamps, refactor session management, and enhance worker reliability (#437)

* Refactor worker version checks and increase timeout settings

- Updated the default hook timeout from 5000ms to 120000ms for improved stability.
- Modified the worker version check to log a warning instead of restarting the worker on version mismatch.
- Removed legacy PM2 cleanup and worker start logic, simplifying the ensureWorkerRunning function.
- Enhanced polling mechanism for worker readiness with increased retries and reduced interval.

* feat: implement worker queue polling to ensure processing completion before proceeding

* refactor: change worker command from start to restart in hooks configuration

* refactor: remove session management complexity

- Simplify createSDKSession to pure INSERT OR IGNORE
- Remove auto-create logic from storeObservation/storeSummary
- Delete 11 unused session management methods
- Derive prompt_number from user_prompts count
- Keep sdk_sessions table schema unchanged for compatibility

* refactor: simplify session management by removing unused methods and auto-creation logic

* Refactor session prompt number retrieval in SessionRoutes

- Updated the method of obtaining the prompt number from the session.
- Replaced `store.getPromptCounter(sessionDbId)` with `store.getPromptNumberFromUserPrompts(claudeSessionId)` for better clarity and accuracy.
- Adjusted the logic for incrementing the prompt number to derive it from the user prompts count instead of directly incrementing a counter.

* refactor: replace getPromptCounter with getPromptNumberFromUserPrompts in SessionManager

Phase 7 of session management simplification. Updates SessionManager to derive
prompt numbers from user_prompts table count instead of using the deprecated
prompt_counter column.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* refactor: simplify SessionCompletionHandler to use direct SQL query

Phase 8: Remove call to findActiveSDKSession() and replace with direct
database query in SessionCompletionHandler.completeByClaudeId().

This removes dependency on the deleted findActiveSDKSession() method
and simplifies the code by using a straightforward SELECT query.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* refactor: remove markSessionCompleted call from SDKAgent

- Delete call to markSessionCompleted() in SDKAgent.ts
- Session status is no longer tracked or updated
- Part of phase 9: simplifying session management

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* refactor: remove markSessionComplete method (Phase 10)

- Deleted markSessionComplete() method from DatabaseManager
- Removed markSessionComplete call from SessionCompletionHandler
- Session completion status no longer tracked in database
- Part of session management simplification effort

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* refactor: replace deleted updateSDKSessionId calls in import script (Phase 11)

- Replace updateSDKSessionId() calls with direct SQL UPDATE statements
- Method was deleted in Phase 3 as part of session management simplification
- Import script now uses direct database access consistently

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* test: add validation for SQL updates in sdk_sessions table

* refactor: enhance worker-cli to support manual and automated runs

* Remove cleanup hook and associated session completion logic

- Deleted the cleanup-hook implementation from the hooks directory.
- Removed the session completion endpoint that was used by the cleanup hook.
- Updated the SessionCompletionHandler to eliminate the completeByClaudeId method and its dependencies.
- Adjusted the SessionRoutes to reflect the removal of the session completion route.

* fix: update worker-cli command to use bun for consistency

* feat: Implement timestamp fix for observations and enhance processing logic

- Added `earliestPendingTimestamp` to `ActiveSession` to track the original timestamp of the earliest pending message.
- Updated `SDKAgent` to capture and utilize the earliest pending timestamp during response processing.
- Modified `SessionManager` to track the earliest timestamp when yielding messages.
- Created scripts for fixing corrupted timestamps, validating fixes, and investigating timestamp issues.
- Verified that all corrupted observations have been repaired and logic for future processing is sound.
- Ensured orphan processing can be safely re-enabled after validation.

* feat: Enhance SessionStore to support custom database paths and add timestamp fields for observations and summaries

* Refactor pending queue processing and add management endpoints

- Disabled automatic recovery of orphaned queues on startup; users must now use the new /api/pending-queue/process endpoint.
- Updated processOrphanedQueues method to processPendingQueues with improved session handling and return detailed results.
- Added new API endpoints for managing pending queues: GET /api/pending-queue and POST /api/pending-queue/process.
- Introduced a new script (check-pending-queue.ts) for checking and processing pending observation queues interactively or automatically.
- Enhanced logging and error handling for better monitoring of session processing.

* updated agent sdk

* feat: Add manual recovery guide and queue management endpoints to documentation

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>

ragtime/ragtime.ts

@@ -4,13 +4,62 @@ import * as path from "path";
 
 const pathToFolder = "/Users/alexnewman/Scripts/claude-mem/datasets/epstein-mode/";
 const pathToPlugin = "/Users/alexnewman/Scripts/claude-mem/plugin/";
+const WORKER_PORT = 37777;
 
 // Or read from a directory
 const filesToProcess = fs
   .readdirSync(pathToFolder)
   .filter((f) => f.endsWith(".md"))
+  .sort((a, b) => {
+    // Extract numeric part from filename (e.g., "0001.md" -> 1)
+    const numA = parseInt(a.match(/\d+/)?.[0] || "0", 10);
+    const numB = parseInt(b.match(/\d+/)?.[0] || "0", 10);
+    return numA - numB;
+  })
   .map((f) => path.join(pathToFolder, f));
 
+/**
+ * Poll the worker's processing status endpoint until the queue is empty
+ */
+async function waitForQueueToEmpty(): Promise<void> {
+  const maxWaitTimeMs = 5 * 60 * 1000; // 5 minutes maximum
+  const pollIntervalMs = 500; // Poll every 500ms
+  const startTime = Date.now();
+
+  while (true) {
+    try {
+      const response = await fetch(`http://localhost:${WORKER_PORT}/api/processing-status`);
+      if (!response.ok) {
+        console.error(`Failed to get processing status: ${response.status}`);
+        break;
+      }
+
+      const status = await response.json();
+      console.log(`Queue status - Processing: ${status.isProcessing}, Queue depth: ${status.queueDepth}`);
+
+      // Exit when queue is empty
+      if (status.queueDepth === 0 && !status.isProcessing) {
+        console.log("Queue is empty, continuing to next prompt");
+        break;
+      }
+
+      // Check timeout
+      if (Date.now() - startTime > maxWaitTimeMs) {
+        console.warn("Warning: Queue did not empty within timeout, continuing anyway");
+        break;
+      }
+
+      // Wait before polling again
+      await new Promise(resolve => setTimeout(resolve, pollIntervalMs));
+    } catch (error) {
+      console.error("Error polling worker status:", error);
+      // On error, wait a bit and continue to avoid infinite loop
+      await new Promise(resolve => setTimeout(resolve, 1000));
+      break;
+    }
+  }
+}
+
 // var i = 0;
 
 for (const file of filesToProcess) {
@@ -35,5 +84,10 @@ for (const file of filesToProcess) {
     if (message.type === "assistant") {
       console.log("Assistant:", message.message.content);
     }
+    console.log("Raw:",  JSON.stringify(message, null, 2));
   }
+
+  // Wait for the worker queue to be empty before continuing to the next file
+  console.log("\n=== Waiting for worker queue to empty ===\n");
+  await waitForQueueToEmpty();
 }