feat: Fix observation timestamps, refactor session management, and enhance worker reliability (#437)

* Refactor worker version checks and increase timeout settings

- Updated the default hook timeout from 5000ms to 120000ms for improved stability.
- Modified the worker version check to log a warning instead of restarting the worker on version mismatch.
- Removed legacy PM2 cleanup and worker start logic, simplifying the ensureWorkerRunning function.
- Enhanced polling mechanism for worker readiness with increased retries and reduced interval.

* feat: implement worker queue polling to ensure processing completion before proceeding

* refactor: change worker command from start to restart in hooks configuration

* refactor: remove session management complexity

- Simplify createSDKSession to pure INSERT OR IGNORE
- Remove auto-create logic from storeObservation/storeSummary
- Delete 11 unused session management methods
- Derive prompt_number from user_prompts count
- Keep sdk_sessions table schema unchanged for compatibility

* refactor: simplify session management by removing unused methods and auto-creation logic

* Refactor session prompt number retrieval in SessionRoutes

- Updated the method of obtaining the prompt number from the session.
- Replaced `store.getPromptCounter(sessionDbId)` with `store.getPromptNumberFromUserPrompts(claudeSessionId)` for better clarity and accuracy.
- Adjusted the logic for incrementing the prompt number to derive it from the user prompts count instead of directly incrementing a counter.

* refactor: replace getPromptCounter with getPromptNumberFromUserPrompts in SessionManager

Phase 7 of session management simplification. Updates SessionManager to derive
prompt numbers from user_prompts table count instead of using the deprecated
prompt_counter column.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* refactor: simplify SessionCompletionHandler to use direct SQL query

Phase 8: Remove call to findActiveSDKSession() and replace with direct
database query in SessionCompletionHandler.completeByClaudeId().

This removes dependency on the deleted findActiveSDKSession() method
and simplifies the code by using a straightforward SELECT query.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* refactor: remove markSessionCompleted call from SDKAgent

- Delete call to markSessionCompleted() in SDKAgent.ts
- Session status is no longer tracked or updated
- Part of phase 9: simplifying session management

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* refactor: remove markSessionComplete method (Phase 10)

- Deleted markSessionComplete() method from DatabaseManager
- Removed markSessionComplete call from SessionCompletionHandler
- Session completion status no longer tracked in database
- Part of session management simplification effort

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* refactor: replace deleted updateSDKSessionId calls in import script (Phase 11)

- Replace updateSDKSessionId() calls with direct SQL UPDATE statements
- Method was deleted in Phase 3 as part of session management simplification
- Import script now uses direct database access consistently

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* test: add validation for SQL updates in sdk_sessions table

* refactor: enhance worker-cli to support manual and automated runs

* Remove cleanup hook and associated session completion logic

- Deleted the cleanup-hook implementation from the hooks directory.
- Removed the session completion endpoint that was used by the cleanup hook.
- Updated the SessionCompletionHandler to eliminate the completeByClaudeId method and its dependencies.
- Adjusted the SessionRoutes to reflect the removal of the session completion route.

* fix: update worker-cli command to use bun for consistency

* feat: Implement timestamp fix for observations and enhance processing logic

- Added `earliestPendingTimestamp` to `ActiveSession` to track the original timestamp of the earliest pending message.
- Updated `SDKAgent` to capture and utilize the earliest pending timestamp during response processing.
- Modified `SessionManager` to track the earliest timestamp when yielding messages.
- Created scripts for fixing corrupted timestamps, validating fixes, and investigating timestamp issues.
- Verified that all corrupted observations have been repaired and logic for future processing is sound.
- Ensured orphan processing can be safely re-enabled after validation.

* feat: Enhance SessionStore to support custom database paths and add timestamp fields for observations and summaries

* Refactor pending queue processing and add management endpoints

- Disabled automatic recovery of orphaned queues on startup; users must now use the new /api/pending-queue/process endpoint.
- Updated processOrphanedQueues method to processPendingQueues with improved session handling and return detailed results.
- Added new API endpoints for managing pending queues: GET /api/pending-queue and POST /api/pending-queue/process.
- Introduced a new script (check-pending-queue.ts) for checking and processing pending observation queues interactively or automatically.
- Enhanced logging and error handling for better monitoring of session processing.

* updated agent sdk

* feat: Add manual recovery guide and queue management endpoints to documentation

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>

src/services/worker/DatabaseManager.ts

@@ -110,10 +110,4 @@ export class DatabaseManager {
     return session;
   }
 
-  /**
-   * Mark session as completed
-   */
-  markSessionComplete(sessionDbId: number): void {
-    this.getSessionStore().markSessionCompleted(sessionDbId);
-  }
 }

src/services/worker/SDKAgent.ts

@@ -115,6 +115,9 @@ export class SDKAgent {
           const discoveryTokens = (session.cumulativeInputTokens + session.cumulativeOutputTokens) - tokensBeforeResponse;
 
           // Process response (empty or not) and mark messages as processed
+          // Capture earliest timestamp BEFORE processing (will be cleared after)
+          const originalTimestamp = session.earliestPendingTimestamp;
+
           if (responseSize > 0) {
             const truncatedResponse = responseSize > 100
               ? textContent.substring(0, 100) + '...'
@@ -124,8 +127,8 @@ export class SDKAgent {
               promptNumber: session.lastPromptNumber
             }, truncatedResponse);
 
-            // Parse and process response with discovery token delta
-            await this.processSDKResponse(session, textContent, worker, discoveryTokens);
+            // Parse and process response with discovery token delta and original timestamp
+            await this.processSDKResponse(session, textContent, worker, discoveryTokens, originalTimestamp);
           } else {
             // Empty response - still need to mark pending messages as processed
             await this.markMessagesProcessed(session, worker);
@@ -145,8 +148,6 @@ export class SDKAgent {
         duration: `${(sessionDuration / 1000).toFixed(1)}s`
       });
 
-      this.dbManager.getSessionStore().markSessionCompleted(session.sessionDbId);
-
     } catch (error: any) {
       if (error.name === 'AbortError') {
         logger.warn('SDK', 'Agent aborted', { sessionId: session.sessionDbId });
@@ -254,19 +255,21 @@ export class SDKAgent {
   /**
    * Process SDK response text (parse XML, save to database, sync to Chroma)
    * @param discoveryTokens - Token cost for discovering this response (delta, not cumulative)
+   * @param originalTimestamp - Original epoch when message was queued (for backlog processing accuracy)
    */
-  private async processSDKResponse(session: ActiveSession, text: string, worker: any | undefined, discoveryTokens: number): Promise<void> {
+  private async processSDKResponse(session: ActiveSession, text: string, worker: any | undefined, discoveryTokens: number, originalTimestamp: number | null): Promise<void> {
     // Parse observations
     const observations = parseObservations(text, session.claudeSessionId);
 
-    // Store observations
+    // Store observations with original timestamp (if processing backlog) or current time
     for (const obs of observations) {
       const { id: obsId, createdAtEpoch } = this.dbManager.getSessionStore().storeObservation(
         session.claudeSessionId,
         session.project,
         obs,
         session.lastPromptNumber,
-        discoveryTokens
+        discoveryTokens,
+        originalTimestamp ?? undefined
       );
 
       // Log observation details
@@ -336,14 +339,15 @@ export class SDKAgent {
     // Parse summary
     const summary = parseSummary(text, session.sessionDbId);
 
-    // Store summary
+    // Store summary with original timestamp (if processing backlog) or current time
     if (summary) {
       const { id: summaryId, createdAtEpoch } = this.dbManager.getSessionStore().storeSummary(
         session.claudeSessionId,
         session.project,
         summary,
         session.lastPromptNumber,
-        discoveryTokens
+        discoveryTokens,
+        originalTimestamp ?? undefined
       );
 
       // Log summary details
@@ -422,6 +426,9 @@ export class SDKAgent {
       });
       session.pendingProcessingIds.clear();
 
+      // Clear timestamp for next batch (will be set fresh from next message)
+      session.earliestPendingTimestamp = null;
+
       // Clean up old processed messages (keep last 100 for UI display)
       const deletedCount = pendingMessageStore.cleanupProcessed(100);
       if (deletedCount > 0) {

src/services/worker/SessionManager.ts

@@ -113,11 +113,12 @@ export class SessionManager {
       pendingMessages: [],
       abortController: new AbortController(),
       generatorPromise: null,
-      lastPromptNumber: promptNumber || this.dbManager.getSessionStore().getPromptCounter(sessionDbId),
+      lastPromptNumber: promptNumber || this.dbManager.getSessionStore().getPromptNumberFromUserPrompts(dbSession.claude_session_id),
       startTime: Date.now(),
       cumulativeInputTokens: 0,
       cumulativeOutputTokens: 0,
-      pendingProcessingIds: new Set()
+      pendingProcessingIds: new Set(),
+      earliestPendingTimestamp: null
     };
 
     this.sessions.set(sessionDbId, session);
@@ -445,6 +446,14 @@ export class SessionManager {
       // Track this message ID for completion marking
       session.pendingProcessingIds.add(persistentMessage.id);
 
+      // Track earliest timestamp for accurate observation timestamps
+      // This ensures backlog messages get their original timestamps, not current time
+      if (session.earliestPendingTimestamp === null) {
+        session.earliestPendingTimestamp = persistentMessage.created_at_epoch;
+      } else {
+        session.earliestPendingTimestamp = Math.min(session.earliestPendingTimestamp, persistentMessage.created_at_epoch);
+      }
+
       // Convert to PendingMessageWithId and yield
       // Include original timestamp for accurate observation timestamps (survives stuck processing)
       const message: PendingMessageWithId = {

src/services/worker/http/routes/DataRoutes.ts

@@ -51,6 +51,10 @@ export class DataRoutes extends BaseRouteHandler {
     app.get('/api/processing-status', this.handleGetProcessingStatus.bind(this));
     app.post('/api/processing', this.handleSetProcessing.bind(this));
 
+    // Pending queue management endpoints
+    app.get('/api/pending-queue', this.handleGetPendingQueue.bind(this));
+    app.post('/api/pending-queue/process', this.handleProcessPendingQueue.bind(this));
+
     // Import endpoint
     app.post('/api/import', this.handleImport.bind(this));
   }
@@ -364,4 +368,58 @@ export class DataRoutes extends BaseRouteHandler {
       stats
     });
   });
+
+  /**
+   * Get pending queue contents
+   * GET /api/pending-queue
+   * Returns all pending, processing, and failed messages with optional recently processed
+   */
+  private handleGetPendingQueue = this.wrapHandler((req: Request, res: Response): void => {
+    const { PendingMessageStore } = require('../../../sqlite/PendingMessageStore.js');
+    const pendingStore = new PendingMessageStore(this.dbManager.getSessionStore().db, 3);
+
+    // Get queue contents (pending, processing, failed)
+    const queueMessages = pendingStore.getQueueMessages();
+
+    // Get recently processed (last 30 min, up to 20)
+    const recentlyProcessed = pendingStore.getRecentlyProcessed(20, 30);
+
+    // Get stuck message count (processing > 5 min)
+    const stuckCount = pendingStore.getStuckCount(5 * 60 * 1000);
+
+    // Get sessions with pending work
+    const sessionsWithPending = pendingStore.getSessionsWithPendingMessages();
+
+    res.json({
+      queue: {
+        messages: queueMessages,
+        totalPending: queueMessages.filter((m: { status: string }) => m.status === 'pending').length,
+        totalProcessing: queueMessages.filter((m: { status: string }) => m.status === 'processing').length,
+        totalFailed: queueMessages.filter((m: { status: string }) => m.status === 'failed').length,
+        stuckCount
+      },
+      recentlyProcessed,
+      sessionsWithPendingWork: sessionsWithPending
+    });
+  });
+
+  /**
+   * Process pending queue
+   * POST /api/pending-queue/process
+   * Body: { sessionLimit?: number } - defaults to 10
+   * Starts SDK agents for sessions with pending messages
+   */
+  private handleProcessPendingQueue = this.wrapHandler(async (req: Request, res: Response): Promise<void> => {
+    const sessionLimit = Math.min(
+      Math.max(parseInt(req.body.sessionLimit, 10) || 10, 1),
+      100 // Max 100 sessions at once
+    );
+
+    const result = await this.workerService.processPendingQueues(sessionLimit);
+
+    res.json({
+      success: true,
+      ...result
+    });
+  });
 }

src/services/worker/http/routes/SessionRoutes.ts

@@ -33,7 +33,6 @@ export class SessionRoutes extends BaseRouteHandler {
     super();
     this.completionHandler = new SessionCompletionHandler(
       sessionManager,
-      dbManager,
       eventBroadcaster
     );
   }
@@ -72,7 +71,6 @@ export class SessionRoutes extends BaseRouteHandler {
     app.post('/api/sessions/init', this.handleSessionInitByClaudeId.bind(this));
     app.post('/api/sessions/observations', this.handleObservationsByClaudeId.bind(this));
     app.post('/api/sessions/summarize', this.handleSummarizeByClaudeId.bind(this));
-    app.post('/api/sessions/complete', this.handleSessionCompleteByClaudeId.bind(this));
   }
 
   /**
@@ -286,7 +284,7 @@ export class SessionRoutes extends BaseRouteHandler {
 
     // Get or create session
     const sessionDbId = store.createSDKSession(claudeSessionId, '', '');
-    const promptNumber = store.getPromptCounter(sessionDbId);
+    const promptNumber = store.getPromptNumberFromUserPrompts(claudeSessionId);
 
     // Privacy check: skip if user prompt was entirely private
     const userPrompt = PrivacyCheckValidator.checkUserPromptPrivacy(
@@ -353,7 +351,7 @@ export class SessionRoutes extends BaseRouteHandler {
 
     // Get or create session
     const sessionDbId = store.createSDKSession(claudeSessionId, '', '');
-    const promptNumber = store.getPromptCounter(sessionDbId);
+    const promptNumber = store.getPromptNumberFromUserPrompts(claudeSessionId);
 
     // Privacy check: skip if user prompt was entirely private
     const userPrompt = PrivacyCheckValidator.checkUserPromptPrivacy(
@@ -390,31 +388,6 @@ export class SessionRoutes extends BaseRouteHandler {
     res.json({ status: 'queued' });
   });
 
-  /**
-   * Complete session by claudeSessionId (cleanup-hook uses this)
-   * POST /api/sessions/complete
-   * Body: { claudeSessionId }
-   *
-   * Marks session complete, stops SDK agent, broadcasts status
-   */
-  private handleSessionCompleteByClaudeId = this.wrapHandler(async (req: Request, res: Response): Promise<void> => {
-    const { claudeSessionId } = req.body;
-
-    if (!claudeSessionId) {
-      return this.badRequest(res, 'Missing claudeSessionId');
-    }
-
-    const found = await this.completionHandler.completeByClaudeId(claudeSessionId);
-
-    if (!found) {
-      // No active session - nothing to clean up (may have already been completed)
-      res.json({ success: true, message: 'No active session found' });
-      return;
-    }
-
-    res.json({ success: true });
-  });
-
   /**
    * Initialize session by claudeSessionId (new-hook uses this)
    * POST /api/sessions/init
@@ -440,8 +413,9 @@ export class SessionRoutes extends BaseRouteHandler {
     // Step 1: Create/get SDK session (idempotent INSERT OR IGNORE)
     const sessionDbId = store.createSDKSession(claudeSessionId, project, prompt);
 
-    // Step 2: Increment prompt counter
-    const promptNumber = store.incrementPromptCounter(sessionDbId);
+    // Step 2: Get next prompt number from user_prompts count
+    const currentCount = store.getPromptNumberFromUserPrompts(claudeSessionId);
+    const promptNumber = currentCount + 1;
 
     // Step 3: Strip privacy tags from prompt
     const cleanedPrompt = stripMemoryTagsFromPrompt(prompt);

src/services/worker/session/SessionCompletionHandler.ts

@@ -1,23 +1,20 @@
 /**
  * Session Completion Handler
  *
- * Consolidates session completion logic to eliminate duplication across
- * three different completion endpoints (DELETE, POST by DB ID, POST by Claude ID).
+ * Consolidates session completion logic for manual session deletion/completion.
+ * Used by DELETE /api/sessions/:id and POST /api/sessions/:id/complete endpoints.
  *
- * All completion flows follow the same pattern:
- * 1. Delete session from SessionManager (aborts SDK agent)
- * 2. Mark session complete in database
- * 3. Broadcast session completed event
+ * Completion flow:
+ * 1. Delete session from SessionManager (aborts SDK agent, cleans up in-memory state)
+ * 2. Broadcast session completed event (updates UI spinner)
  */
 
 import { SessionManager } from '../SessionManager.js';
-import { DatabaseManager } from '../DatabaseManager.js';
 import { SessionEventBroadcaster } from '../events/SessionEventBroadcaster.js';
 
 export class SessionCompletionHandler {
   constructor(
     private sessionManager: SessionManager,
-    private dbManager: DatabaseManager,
     private eventBroadcaster: SessionEventBroadcaster
   ) {}
 
@@ -29,34 +26,7 @@ export class SessionCompletionHandler {
     // Delete from session manager (aborts SDK agent)
     await this.sessionManager.deleteSession(sessionDbId);
 
-    // Mark session complete in database
-    this.dbManager.markSessionComplete(sessionDbId);
-
     // Broadcast session completed event
     this.eventBroadcaster.broadcastSessionCompleted(sessionDbId);
   }
-
-  /**
-   * Complete session by Claude session ID
-   * Used by POST /api/sessions/complete (cleanup-hook endpoint)
-   *
-   * @returns true if session was found and completed, false if no active session found
-   */
-  async completeByClaudeId(claudeSessionId: string): Promise<boolean> {
-    const store = this.dbManager.getSessionStore();
-
-    // Find session by claudeSessionId
-    const session = store.findActiveSDKSession(claudeSessionId);
-    if (!session) {
-      // No active session - nothing to clean up (may have already been completed)
-      return false;
-    }
-
-    const sessionDbId = session.id;
-
-    // Complete using standard flow
-    await this.completeByDbId(sessionDbId);
-
-    return true;
-  }
 }