feat: Fix observation timestamps, refactor session management, and enhance worker reliability (#437)
* Refactor worker version checks and increase timeout settings - Updated the default hook timeout from 5000ms to 120000ms for improved stability. - Modified the worker version check to log a warning instead of restarting the worker on version mismatch. - Removed legacy PM2 cleanup and worker start logic, simplifying the ensureWorkerRunning function. - Enhanced polling mechanism for worker readiness with increased retries and reduced interval. * feat: implement worker queue polling to ensure processing completion before proceeding * refactor: change worker command from start to restart in hooks configuration * refactor: remove session management complexity - Simplify createSDKSession to pure INSERT OR IGNORE - Remove auto-create logic from storeObservation/storeSummary - Delete 11 unused session management methods - Derive prompt_number from user_prompts count - Keep sdk_sessions table schema unchanged for compatibility * refactor: simplify session management by removing unused methods and auto-creation logic * Refactor session prompt number retrieval in SessionRoutes - Updated the method of obtaining the prompt number from the session. - Replaced `store.getPromptCounter(sessionDbId)` with `store.getPromptNumberFromUserPrompts(claudeSessionId)` for better clarity and accuracy. - Adjusted the logic for incrementing the prompt number to derive it from the user prompts count instead of directly incrementing a counter. * refactor: replace getPromptCounter with getPromptNumberFromUserPrompts in SessionManager Phase 7 of session management simplification. Updates SessionManager to derive prompt numbers from user_prompts table count instead of using the deprecated prompt_counter column. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: simplify SessionCompletionHandler to use direct SQL query Phase 8: Remove call to findActiveSDKSession() and replace with direct database query in SessionCompletionHandler.completeByClaudeId(). This removes dependency on the deleted findActiveSDKSession() method and simplifies the code by using a straightforward SELECT query. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: remove markSessionCompleted call from SDKAgent - Delete call to markSessionCompleted() in SDKAgent.ts - Session status is no longer tracked or updated - Part of phase 9: simplifying session management 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: remove markSessionComplete method (Phase 10) - Deleted markSessionComplete() method from DatabaseManager - Removed markSessionComplete call from SessionCompletionHandler - Session completion status no longer tracked in database - Part of session management simplification effort 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * refactor: replace deleted updateSDKSessionId calls in import script (Phase 11) - Replace updateSDKSessionId() calls with direct SQL UPDATE statements - Method was deleted in Phase 3 as part of session management simplification - Import script now uses direct database access consistently 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * test: add validation for SQL updates in sdk_sessions table * refactor: enhance worker-cli to support manual and automated runs * Remove cleanup hook and associated session completion logic - Deleted the cleanup-hook implementation from the hooks directory. - Removed the session completion endpoint that was used by the cleanup hook. - Updated the SessionCompletionHandler to eliminate the completeByClaudeId method and its dependencies. - Adjusted the SessionRoutes to reflect the removal of the session completion route. * fix: update worker-cli command to use bun for consistency * feat: Implement timestamp fix for observations and enhance processing logic - Added `earliestPendingTimestamp` to `ActiveSession` to track the original timestamp of the earliest pending message. - Updated `SDKAgent` to capture and utilize the earliest pending timestamp during response processing. - Modified `SessionManager` to track the earliest timestamp when yielding messages. - Created scripts for fixing corrupted timestamps, validating fixes, and investigating timestamp issues. - Verified that all corrupted observations have been repaired and logic for future processing is sound. - Ensured orphan processing can be safely re-enabled after validation. * feat: Enhance SessionStore to support custom database paths and add timestamp fields for observations and summaries * Refactor pending queue processing and add management endpoints - Disabled automatic recovery of orphaned queues on startup; users must now use the new /api/pending-queue/process endpoint. - Updated processOrphanedQueues method to processPendingQueues with improved session handling and return detailed results. - Added new API endpoints for managing pending queues: GET /api/pending-queue and POST /api/pending-queue/process. - Introduced a new script (check-pending-queue.ts) for checking and processing pending observation queues interactively or automatically. - Enhanced logging and error handling for better monitoring of session processing. * updated agent sdk * feat: Add manual recovery guide and queue management endpoints to documentation --------- Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
Alex Newman
@@ -0,0 +1,106 @@
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'bun:test';
|
||||
import { SessionStore } from '../src/services/sqlite/SessionStore.js';
|
||||
|
||||
describe('SessionStore', () => {
|
||||
let store: SessionStore;
|
||||
|
||||
beforeEach(() => {
|
||||
store = new SessionStore(':memory:');
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
store.close();
|
||||
});
|
||||
|
||||
it('should correctly count user prompts', () => {
|
||||
const claudeId = 'claude-session-1';
|
||||
store.createSDKSession(claudeId, 'test-project', 'initial prompt');
|
||||
|
||||
// Should be 0 initially
|
||||
expect(store.getPromptNumberFromUserPrompts(claudeId)).toBe(0);
|
||||
|
||||
// Save prompt 1
|
||||
store.saveUserPrompt(claudeId, 1, 'First prompt');
|
||||
expect(store.getPromptNumberFromUserPrompts(claudeId)).toBe(1);
|
||||
|
||||
// Save prompt 2
|
||||
store.saveUserPrompt(claudeId, 2, 'Second prompt');
|
||||
expect(store.getPromptNumberFromUserPrompts(claudeId)).toBe(2);
|
||||
|
||||
// Save prompt for another session
|
||||
store.createSDKSession('claude-session-2', 'test-project', 'initial prompt');
|
||||
store.saveUserPrompt('claude-session-2', 1, 'Other prompt');
|
||||
expect(store.getPromptNumberFromUserPrompts(claudeId)).toBe(2);
|
||||
});
|
||||
|
||||
it('should store observation with timestamp override', () => {
|
||||
const claudeId = 'claude-sess-obs';
|
||||
const sdkId = store.createSDKSession(claudeId, 'test-project', 'initial prompt');
|
||||
|
||||
// Get the sdk_session_id string (createSDKSession returns number ID, need string for FK)
|
||||
// Wait, createSDKSession inserts using sdk_session_id = claude_session_id in the current implementation
|
||||
// "VALUES (?, ?, ?, ?, ?, ?, 'active')" -> claudeSessionId, claudeSessionId, ...
|
||||
|
||||
const obs = {
|
||||
type: 'discovery',
|
||||
title: 'Test Obs',
|
||||
subtitle: null,
|
||||
facts: [],
|
||||
narrative: 'Testing',
|
||||
concepts: [],
|
||||
files_read: [],
|
||||
files_modified: []
|
||||
};
|
||||
|
||||
const pastTimestamp = 1600000000000; // Some time in the past
|
||||
|
||||
const result = store.storeObservation(
|
||||
claudeId, // sdkSessionId is same as claudeSessionId in createSDKSession
|
||||
'test-project',
|
||||
obs,
|
||||
1,
|
||||
0,
|
||||
pastTimestamp
|
||||
);
|
||||
|
||||
expect(result.createdAtEpoch).toBe(pastTimestamp);
|
||||
|
||||
const stored = store.getObservationById(result.id);
|
||||
expect(stored).not.toBeNull();
|
||||
expect(stored?.created_at_epoch).toBe(pastTimestamp);
|
||||
|
||||
// Verify ISO string matches
|
||||
expect(new Date(stored!.created_at).getTime()).toBe(pastTimestamp);
|
||||
});
|
||||
|
||||
it('should store summary with timestamp override', () => {
|
||||
const claudeId = 'claude-sess-sum';
|
||||
store.createSDKSession(claudeId, 'test-project', 'initial prompt');
|
||||
|
||||
const summary = {
|
||||
request: 'Do something',
|
||||
investigated: 'Stuff',
|
||||
learned: 'Things',
|
||||
completed: 'Done',
|
||||
next_steps: 'More',
|
||||
notes: null
|
||||
};
|
||||
|
||||
const pastTimestamp = 1650000000000;
|
||||
|
||||
const result = store.storeSummary(
|
||||
claudeId,
|
||||
'test-project',
|
||||
summary,
|
||||
1,
|
||||
0,
|
||||
pastTimestamp
|
||||
);
|
||||
|
||||
expect(result.createdAtEpoch).toBe(pastTimestamp);
|
||||
|
||||
const stored = store.getSummaryForSession(claudeId);
|
||||
expect(stored).not.toBeNull();
|
||||
expect(stored?.created_at_epoch).toBe(pastTimestamp);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,54 @@
|
||||
import { Database } from 'bun:sqlite';
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'bun:test';
|
||||
|
||||
describe('Refactor Validation: SQL Updates', () => {
|
||||
let db: Database;
|
||||
|
||||
beforeEach(() => {
|
||||
db = new Database(':memory:');
|
||||
// Minimal schema for sdk_sessions based on SessionStore.ts migration004
|
||||
db.run(`
|
||||
CREATE TABLE sdk_sessions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
claude_session_id TEXT UNIQUE NOT NULL,
|
||||
sdk_session_id TEXT UNIQUE,
|
||||
project TEXT NOT NULL,
|
||||
user_prompt TEXT,
|
||||
started_at TEXT,
|
||||
started_at_epoch INTEGER,
|
||||
completed_at TEXT,
|
||||
completed_at_epoch INTEGER,
|
||||
status TEXT DEFAULT 'active'
|
||||
);
|
||||
`);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('should update sdk_session_id using direct SQL (replacing updateSDKSessionId)', () => {
|
||||
// Setup initial state: A session without an sdk_session_id
|
||||
const claudeId = 'claude-session-123';
|
||||
const syntheticId = 'sdk-session-456';
|
||||
|
||||
db.prepare(`
|
||||
INSERT INTO sdk_sessions (claude_session_id, project, started_at, started_at_epoch)
|
||||
VALUES (?, ?, ?, ?)
|
||||
`).run(claudeId, 'test-project', '2025-01-01T00:00:00Z', 1735689600000);
|
||||
|
||||
// Verify initial state
|
||||
const before = db.prepare('SELECT sdk_session_id FROM sdk_sessions WHERE claude_session_id = ?').get(claudeId) as any;
|
||||
expect(before.sdk_session_id).toBeNull();
|
||||
|
||||
// EXECUTE: The exact SQL statement from the refactor in import-xml-observations.ts
|
||||
// Original code: db['db'].prepare('UPDATE sdk_sessions SET sdk_session_id = ? WHERE claude_session_id = ?').run(syntheticSdkSessionId, sessionMeta.sessionId);
|
||||
|
||||
const stmt = db.prepare('UPDATE sdk_sessions SET sdk_session_id = ? WHERE claude_session_id = ?');
|
||||
stmt.run(syntheticId, claudeId);
|
||||
|
||||
// VERIFY: The update happened
|
||||
const after = db.prepare('SELECT sdk_session_id FROM sdk_sessions WHERE claude_session_id = ?').get(claudeId) as any;
|
||||
expect(after.sdk_session_id).toBe(syntheticId);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user