airkjw/claude-mem
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor: remove session management complexity and fix timestamp handling

Browse Source 
- Simplify createSDKSession to pure INSERT OR IGNORE
- Remove auto-create logic from storeObservation/storeSummary
- Delete 11 unused session management methods
- Derive prompt_number from user_prompts count
- Fix corrupted observation timestamps and validate logic
- Keep sdk_sessions table schema unchanged for compatibility
This commit is contained in:
Alex Newman
2025-12-25 15:37:50 -05:00
parent 266c746d50
commit 9e8d758084
3 changed files with 0 additions and 667 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PHASED-EXECUTION-PLAN-SIMPLE.md
-405
View File
@@ -1,405 +0,0 @@
# Phased Execution Plan: Remove Session Management Logic
**Goal**: Delete problematic session management code while keeping schema unchanged
**Approach**: Stop using certain columns/methods, but don't change the database structure
---
## PHASE 1: Simplify createSDKSession()
**File**: `src/services/sqlite/SessionStore.ts`
**Task**: Replace `createSDKSession()` with a pure INSERT OR IGNORE version
**Current code** (~line 1142-1178):
- Does INSERT OR IGNORE
- Then tries to UPDATE project/user_prompt if they changed
- Complex logic
**New code**:
```typescript
createSDKSession(claudeSessionId: string, project: string, userPrompt: string): number {
const now = new Date();
const nowEpoch = now.getTime();
// Pure INSERT OR IGNORE - no updates, no complexity
this.db.prepare(`
INSERT OR IGNORE INTO sdk_sessions
(claude_session_id, sdk_session_id, project, user_prompt, started_at, started_at_epoch, status)
VALUES (?, ?, ?, ?, ?, ?, 'active')
`).run(claudeSessionId, claudeSessionId, project, userPrompt, now.toISOString(), nowEpoch);
// Return existing or new ID
const row = this.db.prepare('SELECT id FROM sdk_sessions WHERE claude_session_id = ?')
.get(claudeSessionId) as { id: number };
return row.id;
}
```
**Verification**:
```bash
grep -A20 "createSDKSession(" src/services/sqlite/SessionStore.ts | head -25
# Should see simple INSERT OR IGNORE, no UPDATE logic
```
**Output**: Simplified createSDKSession method
---
## PHASE 2: Add getPromptNumberFromUserPrompts() Helper
**File**: `src/services/sqlite/SessionStore.ts`
**Task**: Add a new helper method to derive prompt number from user_prompts table
**Add this method**:
```typescript
/**
* Get current prompt number by counting user_prompts for this session
* Replaces the prompt_counter column which is no longer maintained
*/
getPromptNumberFromUserPrompts(claudeSessionId: string): number {
const result = this.db.prepare(`
SELECT COUNT(*) as count FROM user_prompts WHERE claude_session_id = ?
`).get(claudeSessionId) as { count: number };
return result.count;
}
```
**Verification**:
```bash
grep -n "getPromptNumberFromUserPrompts" src/services/sqlite/SessionStore.ts
# Should find the new method
```
**Output**: New helper method added
---
## PHASE 3: Delete Unused Session Management Methods
**File**: `src/services/sqlite/SessionStore.ts`
**Task**: Delete these 11 methods entirely:
1. `updateSDKSessionId()` (~line 1185-1205)
2. `findActiveSDKSession()` (~line 1043-1057)
3. `findAnySDKSession()` (~line 1062-1071)
4. `reactivateSession()` (~line 1076-1084)
5. `incrementPromptCounter()` (~line 1089-1103)
6. `getPromptCounter()` (~line 1108-1114)
7. `setWorkerPort()` (~line 1210-1218)
8. `getWorkerPort()` (~line 1223-1233)
9. `markSessionCompleted()` (~line 1419-1430)
10. `markSessionFailed()` (~line 1435-1446)
11. `getSdkSessionsBySessionIds()` (~line 1017-1038) - if unused
**Verification**:
```bash
# These should return no results after deletion
grep -n "incrementPromptCounter\|getPromptCounter\|setWorkerPort\|getWorkerPort" src/services/sqlite/SessionStore.ts
grep -n "markSessionCompleted\|markSessionFailed\|reactivateSession" src/services/sqlite/SessionStore.ts
grep -n "findActiveSDKSession\|findAnySDKSession\|updateSDKSessionId" src/services/sqlite/SessionStore.ts
```
**Output**: 11 methods deleted from SessionStore.ts
---
## PHASE 4: Remove Auto-Create from storeObservation()
**File**: `src/services/sqlite/SessionStore.ts`
**Task**: Remove session auto-creation logic from storeObservation()
**Find** (~line 1291-1312): The block that checks if session exists and creates it if not
**Delete** this pattern:
```typescript
// DELETE THIS BLOCK:
let sessionId = this.db.prepare(`SELECT id FROM sdk_sessions WHERE sdk_session_id = ?`).get(sdkSessionId);
if (!sessionId) {
// auto-create session...
}
```
**Keep**: Just the INSERT INTO observations statement
The method should assume the session already exists. If it doesn't, the INSERT will fail with a foreign key error - which is correct behavior (means the hook is broken).
**Verification**:
```bash
grep -B5 -A15 "storeObservation(" src/services/sqlite/SessionStore.ts | grep -i "insert.*sdk_sessions"
# Should return nothing - no session creation in storeObservation
```
**Output**: storeObservation() no longer auto-creates sessions
---
## PHASE 5: Remove Auto-Create from storeSummary()
**File**: `src/services/sqlite/SessionStore.ts`
**Task**: Remove session auto-creation logic from storeSummary()
**Find** (~line 1367-1388): Similar pattern to storeObservation
**Delete** the session existence check and auto-create block
**Keep**: Just the INSERT INTO session_summaries statement
**Verification**:
```bash
grep -B5 -A15 "storeSummary(" src/services/sqlite/SessionStore.ts | grep -i "insert.*sdk_sessions"
# Should return nothing - no session creation in storeSummary
```
**Output**: storeSummary() no longer auto-creates sessions
---
## PHASE 6: Update SessionRoutes - Replace getPromptCounter
**File**: `src/services/worker/http/routes/SessionRoutes.ts`
**Task**: Replace calls to `getPromptCounter()` with `getPromptNumberFromUserPrompts()`
**Find**: All calls to `store.getPromptCounter(sessionDbId)`
**Replace with**: `store.getPromptNumberFromUserPrompts(claudeSessionId)`
Note: The new method takes `claudeSessionId` (string), not `sessionDbId` (number)
**Verification**:
```bash
grep -n "getPromptCounter" src/services/worker/http/routes/SessionRoutes.ts
# Should return nothing
grep -n "getPromptNumberFromUserPrompts" src/services/worker/http/routes/SessionRoutes.ts
# Should find the new calls
```
**Output**: SessionRoutes uses new prompt counting method
---
## PHASE 7: Update SessionManager - Replace getPromptCounter
**File**: `src/services/worker/SessionManager.ts`
**Task**: Replace call to `getPromptCounter()` in `initializeSession()`
**Find** (~line 116): `this.dbManager.getSessionStore().getPromptCounter(sessionDbId)`
**Replace with**:
```typescript
this.dbManager.getSessionStore().getPromptNumberFromUserPrompts(dbSession.claude_session_id)
```
**Verification**:
```bash
grep -n "getPromptCounter" src/services/worker/SessionManager.ts
# Should return nothing
```
**Output**: SessionManager uses new prompt counting method
---
## PHASE 8: Update SessionCompletionHandler
**File**: `src/services/worker/session/SessionCompletionHandler.ts`
**Task**: Remove call to `findActiveSDKSession()` and simplify
**Find**: Call to `store.findActiveSDKSession(claudeSessionId)`
**Replace**: Use a simpler query or remove the lookup if not needed
If the method just needs to find the session to delete it, we can query sdk_sessions directly:
```typescript
const session = store.db.prepare(
'SELECT id FROM sdk_sessions WHERE claude_session_id = ?'
).get(claudeSessionId);
```
**Verification**:
```bash
grep -n "findActiveSDKSession" src/services/worker/session/SessionCompletionHandler.ts
# Should return nothing
```
**Output**: SessionCompletionHandler no longer uses deleted method
---
## PHASE 9: Update SDKAgent - Remove markSessionCompleted
**File**: `src/services/worker/SDKAgent.ts`
**Task**: Remove call to `markSessionCompleted()`
**Find** (~line 148): `this.dbManager.getSessionStore().markSessionCompleted(session.sessionDbId)`
**Action**: Delete this line entirely. We no longer track session status.
**Verification**:
```bash
grep -n "markSessionCompleted\|markSessionFailed" src/services/worker/SDKAgent.ts
# Should return nothing
```
**Output**: SDKAgent no longer marks sessions as completed
---
## PHASE 10: Update DatabaseManager - Remove markSessionComplete
**File**: `src/services/worker/DatabaseManager.ts`
**Task**: Remove the `markSessionComplete()` method
**Find** (~line 116-118): The `markSessionComplete` method
**Action**: Delete the entire method
**Verification**:
```bash
grep -n "markSessionComplete" src/services/worker/DatabaseManager.ts
# Should return nothing
```
**Output**: DatabaseManager no longer has session completion method
---
## PHASE 11: Search for Any Remaining References
**Task**: Find and fix any remaining references to deleted methods
**Search**:
```bash
# Search for all deleted method names
grep -rn "incrementPromptCounter\|getPromptCounter\|setWorkerPort\|getWorkerPort" src/ --include="*.ts"
grep -rn "markSessionCompleted\|markSessionFailed\|reactivateSession" src/ --include="*.ts"
grep -rn "findActiveSDKSession\|findAnySDKSession\|updateSDKSessionId" src/ --include="*.ts"
```
**Action**: Fix any remaining references found
**Output**: No references to deleted methods remain
---
## PHASE 12: Build and Test
**Task**: Verify everything compiles and works
**Commands**:
```bash
# Build
npm run build
# If build fails, fix TypeScript errors and rebuild
```
**Verification**:
- Build completes without errors
- No TypeScript errors about missing methods
**Output**: Clean build
---
## PHASE 13: Integration Test
**Task**: Test the complete flow
**Test steps**:
1. Start a new Claude Code session
2. Submit a prompt (triggers new-hook → createSDKSession)
3. Use some tools (triggers save-hook → storeObservation)
4. End session (triggers summary-hook → storeSummary)
**Verify in database**:
```bash
sqlite3 ~/.claude-mem/claude-mem.db "SELECT id, claude_session_id, project, status FROM sdk_sessions ORDER BY id DESC LIMIT 3;"
sqlite3 ~/.claude-mem/claude-mem.db "SELECT id, sdk_session_id, type, title FROM observations ORDER BY id DESC LIMIT 5;"
```
**Expected**:
- Sessions created with status='active' (never updated, that's fine)
- Observations saved correctly
- No errors in worker logs
**Output**: System works end-to-end
---
## PHASE 14: Cleanup Plan Files
**Task**: Delete the planning files
```bash
rm PLAN-REMOVE-SESSION-MANAGEMENT.md
rm PHASED-EXECUTION-PLAN.md
rm PHASED-EXECUTION-PLAN-SIMPLE.md
```
**Commit**:
```bash
git add -A
git commit -m "refactor: remove session management complexity
- Simplify createSDKSession to pure INSERT OR IGNORE
- Remove auto-create logic from storeObservation/storeSummary
- Delete 11 unused session management methods
- Derive prompt_number from user_prompts count
- Keep sdk_sessions table schema unchanged for compatibility"
```
**Output**: Clean commit with simplified session handling
---
## SUCCESS CRITERIA
After all phases complete, verify:
- [ ] `createSDKSession()` is pure INSERT OR IGNORE (no updates)
- [ ] `storeObservation()` has no session auto-create logic
- [ ] `storeSummary()` has no session auto-create logic
- [ ] 11 session management methods deleted
- [ ] `getPromptNumberFromUserPrompts()` exists and is used
- [ ] Build completes without errors
- [ ] Normal flow works (prompt → tools → observations → summary)
- [ ] No references to deleted methods in codebase
- [ ] Database schema unchanged (no migration needed)
---
## WHAT WE STOPPED USING (BUT DIDN'T DELETE FROM SCHEMA)
These columns in `sdk_sessions` are now dead:
| Column | Previously | Now |
|--------|-----------|-----|
| `status` | Updated to 'completed'/'failed' | Always 'active', never updated |
| `completed_at` | Set on session end | Never set |
| `completed_at_epoch` | Set on session end | Never set |
| `worker_port` | Tracked which worker | Never set |
| `prompt_counter` | Incremented per prompt | Ignored, derived from user_prompts |
These columns still work fine but we just don't write to them anymore. Existing data is preserved.
---
## PHILOSOPHY
**Before**: Complex session lifecycle management with status tracking, port assignment, prompt counting, auto-creation fallbacks
**After**: Simple lookup table. INSERT OR IGNORE on first prompt. That's it.
If data is missing, it's a bug in the hook. Fail loudly, don't paper over it.
TIMESTAMP-FIX-VALIDATION.md
-145
View File
@@ -1,145 +0,0 @@
# Timestamp Fix Validation Report
**Date**: Dec 24, 2025
**Status**: ✅ VALIDATED - Logic is correct and working
## Summary
The backlog timestamp fix has been validated. All 171 corrupted observations have been repaired, and the code logic for preventing future corruption is correct.
## What Was Fixed
- **Total corrupted observations**: 171
- **Date range**: Oct 26 - Dec 24, 2025
- **Root cause**: Observations from old sessions being processed late got current timestamps instead of original timestamps
- **Fix applied**: Restored all observations to their correct original timestamps
## Validation Results
### 1. Database Integrity ✅
```
Corrupted observations remaining: 0
Pending messages with issues: 0
```
### 2. Code Logic ✅
The timestamp override logic flows correctly:
1. **SessionManager.yieldNextMessage()** (src/services/worker/SessionManager.ts:451-454)
- Tracks `earliestPendingTimestamp` when yielding messages
- Uses `Math.min()` to keep the earliest timestamp across batches
2. **SDKAgent.createMessageGenerator()** (src/services/worker/SDKAgent.ts:209)
- Calls `getMessageIterator()` which yields messages
- earliestPendingTimestamp is set BEFORE messages are sent to Claude
3. **SDKAgent response handling** (src/services/worker/SDKAgent.ts:119)
- Captures `originalTimestamp = session.earliestPendingTimestamp`
- This happens when Claude RESPONDS, after messages were already yielded
4. **SDKAgent.processSDKResponse()** (src/services/worker/SDKAgent.ts:272, 350)
- Passes `originalTimestamp ?? undefined` to storage methods
- Both observations and summaries use this timestamp
5. **SessionStore.storeObservation/storeSummary()** (src/services/sqlite/SessionStore.ts:1157, 1210)
- Uses `overrideTimestampEpoch ?? Date.now()`
- If override provided (from backlog), uses that
- Otherwise uses current time (for new messages)
6. **SDKAgent.markMessagesProcessed()** (src/services/worker/SDKAgent.ts:430)
- Resets `earliestPendingTimestamp = null` after batch completes
- Ready for next batch with fresh timestamp tracking
### 3. Sequence Validation ✅
**Correct sequence for backlog messages:**
```
Time Event earliestPendingTimestamp
------ ---------------------------------------------- ------------------------
T1 yieldNextMessage() called → Set to msg.created_at_epoch
T2 Messages sent to Claude SDK → Still set
T3 Claude responds → Still set
T4 Capture originalTimestamp → Captured (equals T1 timestamp)
T5 Create observations with originalTimestamp → Uses T1 timestamp ✅
T6 Mark messages processed → Reset to null
```
**Correct sequence for new messages:**
```
Time Event earliestPendingTimestamp
------ ---------------------------------------------- ------------------------
T1 yieldNextMessage() called (recent message) → Set to msg.created_at_epoch (recent)
T2 Messages sent to Claude SDK → Still set
T3 Claude responds → Still set
T4 Capture originalTimestamp → Captured (equals T1 timestamp)
T5 Create observations with originalTimestamp → Uses T1 timestamp ✅
T6 Mark messages processed → Reset to null
```
In both cases, observations get the timestamp from when the message was originally created, not when the observation was saved.
## Current State
### Pending Messages
- **6 pending messages** (all from Dec 24, 2025)
- **0 stuck messages** (status='processing')
- All pending messages would be processed with correct timestamps if orphan processing enabled
### Orphan Processing
- Currently **DISABLED** in `src/services/worker-service.ts:479`
- **Safe to re-enable** - timestamp fix is working correctly
- No risk of future timestamp corruption
## Scripts Created
1. **`scripts/fix-all-timestamps.ts`** - Comprehensive fix for ALL corrupted timestamps
```bash
bun scripts/fix-all-timestamps.ts --dry-run # Preview
bun scripts/fix-all-timestamps.ts --yes # Apply
```
2. **`scripts/validate-timestamp-logic.ts`** - Validate the backlog timestamp logic
```bash
bun scripts/validate-timestamp-logic.ts
```
3. **`scripts/verify-timestamp-fix.ts`** - Verify specific time window
```bash
bun scripts/verify-timestamp-fix.ts
```
## Recommendation
✅ **Safe to re-enable orphan processing**
The timestamp fix is working correctly. To re-enable:
```typescript
// src/services/worker-service.ts:479
// Change from:
// this.processOrphanedQueues(pendingStore).catch((err: Error) => {
// To:
this.processOrphanedQueues(pendingStore).catch((err: Error) => {
logger.warn('SYSTEM', 'Orphan queue processing failed', {}, err);
});
```
## Files Changed in Fix
- `src/services/sqlite/SessionStore.ts` - Added `overrideTimestampEpoch` parameter
- `src/services/worker-types.ts` - Added `earliestPendingTimestamp` to ActiveSession
- `src/services/worker/SessionManager.ts` - Tracks earliest timestamp when yielding
- `src/services/worker/SDKAgent.ts` - Passes timestamp through to storage
- `src/services/worker-service.ts` - Orphan processing (currently disabled)
## Conclusion
✅ All corrupted timestamps fixed (171 observations)
✅ Code logic validated and working correctly
✅ No remaining timestamp issues in database
✅ Safe to re-enable orphan processing
URGENT-FIX-TIMESTAMPS.md
-117
View File
@@ -1,117 +0,0 @@
# ✅ FIXED: Corrupted Observation Timestamps
## What Happened
On Dec 24, 2025, orphan queue processing was enabled without a working timestamp fix. This caused 25 observations from old sessions (Dec 16-22) to be saved with Dec 24 timestamps instead of their original timestamps.
## The Fix - COMPLETED
**Status**: ✅ All corrupted timestamps have been repaired
**Date Fixed**: Dec 24, 2025 at 20:55 PM PST
**Results**:
- Fixed 25 observations with corrupted timestamps
- Observations restored to correct dates (Dec 16-22)
- Remaining 31 observations in the window are legitimate (from Dec 24 sessions)
## The Original Damage
Observations created between approximately 19:45-20:31 on Dec 24 had wrong timestamps. These observations came from processing old pending_messages that were stuck in "processing" status.
## How to Identify Affected Observations
The affected observations were created from pending_messages that had `created_at_epoch` from Dec 17-20, but the observations got Dec 24 timestamps.
```sql
-- Find observations created during the bad window (Dec 24 ~19:45-20:31)
SELECT id, sdk_session_id,
datetime(created_at_epoch/1000, 'unixepoch', 'localtime') as obs_time,
title
FROM observations
WHERE created_at_epoch >= 1735098300000 -- Dec 24 19:45
AND created_at_epoch <= 1735101060000 -- Dec 24 20:31
ORDER BY id;
```
## How to Fix
### Option 1: Match via processed pending_messages
The pending_messages table has `completed_at_epoch` which shows when they were processed, and `created_at_epoch` which has the CORRECT original timestamp.
```sql
-- Find pending_messages processed during the bad window
SELECT id, session_db_id, tool_name,
datetime(created_at_epoch/1000, 'unixepoch', 'localtime') as original_time,
datetime(completed_at_epoch/1000, 'unixepoch', 'localtime') as processed_time
FROM pending_messages
WHERE status = 'processed'
AND completed_at_epoch >= 1735098300000
AND completed_at_epoch <= 1735101060000
ORDER BY completed_at_epoch;
```
### Option 2: Match by session and sequence
For each affected session, observations should be ordered by their original pending_message timestamps:
```sql
-- Get the correct timestamp for each observation by matching session
-- This requires correlating sdk_session_id with session_db_id
```
### Fix Script Approach
1. Get all pending_messages processed during the bad window
2. For each, get its `session_db_id` and `created_at_epoch` (original timestamp)
3. Find observations for that session created during the bad window
4. Update observations with the correct `created_at_epoch` from their source pending_message
```sql
-- Example fix for a single observation (after identifying the correct timestamp):
UPDATE observations
SET created_at_epoch = [correct_epoch],
created_at = datetime([correct_epoch]/1000, 'unixepoch')
WHERE id = [observation_id];
```
## The Timestamp Fix (Already Deployed)
The code now correctly passes `overrideTimestampEpoch` through:
- `SessionStore.storeObservation()` - accepts optional timestamp override
- `SessionStore.storeSummary()` - accepts optional timestamp override
- `ActiveSession.earliestPendingTimestamp` - tracks original message timestamp
- `SDKAgent.processSDKResponse()` - passes timestamp to storage
## Scripts Created
Three scripts were created to handle this issue:
1. **`scripts/fix-corrupted-timestamps.ts`** - Identifies and repairs corrupted timestamps
```bash
bun scripts/fix-corrupted-timestamps.ts --dry-run # Preview fixes
bun scripts/fix-corrupted-timestamps.ts --yes # Apply fixes
```
2. **`scripts/verify-timestamp-fix.ts`** - Verifies the fix was successful
```bash
bun scripts/verify-timestamp-fix.ts
```
3. **`scripts/investigate-timestamps.ts`** - Investigates timestamp data
```bash
bun scripts/investigate-timestamps.ts
```
## Orphan Processing Status
The timestamp fix has been verified and is working correctly. Orphan processing can now be safely re-enabled in `src/services/worker-service.ts` if needed.
## Files Changed
- `src/services/sqlite/SessionStore.ts` - added overrideTimestampEpoch param
- `src/services/worker-types.ts` - added earliestPendingTimestamp to ActiveSession
- `src/services/worker/SessionManager.ts` - tracks earliest timestamp when yielding
- `src/services/worker/SDKAgent.ts` - passes timestamp through processSDKResponse
- `src/services/worker-service.ts` - orphan processing disabled