claude-mem/docs/reports/2026-01-02--observation-duplication-regression.md

Observation Duplication Regression - 2026-01-02

Executive Summary

A critical regression is causing the same observation to be created multiple times (2-11 duplicates per observation). This occurred after recent error handling refactoring work that removed try-catch blocks. The root cause is a race condition between observation persistence and message completion marking in the SDK agent, exacerbated by crash recovery logic.

Symptoms

• 11 observations about "session generator failure" created between 10:01-10:09 PM (same content, different timestamps)
• 8 observations about "fixed missing closing brace" created between 9:32 PM-9:55 PM
• 2 observations about "remove large try-catch blocks" created at 9:33 PM
• Multiple other duplicates across different sessions

Example from database:

-- Same observation created 8 times over 23 minutes
id     | title                                          | created_at
-------|------------------------------------------------|-------------------
36050  | Fixed Missing Closing Brace in SearchManager  | 2026-01-02 21:32:43
36040  | Fixed Missing Closing Brace in SearchManager  | 2026-01-02 21:33:34
36047  | Fixed missing closing brace...                | 2026-01-02 21:33:38
36041  | Fixed missing closing brace...                | 2026-01-02 21:34:33
36060  | Fixed Missing Closing Brace...                | 2026-01-02 21:41:23
36062  | Fixed Missing Closing Brace...                | 2026-01-02 21:53:02
36063  | Fixed Missing Closing Brace...                | 2026-01-02 21:53:33
36065  | Fixed missing closing brace...                | 2026-01-02 21:55:06

Root Cause Analysis

The Critical Race Condition

The SDK agent has a fatal ordering issue in message processing:

File: /Users/alexnewman/Scripts/claude-mem/src/services/worker/SDKAgent.ts

// Line 328-410: processSDKResponse()
private async processSDKResponse(...): Promise<void> {
  // Parse observations from SDK response
  const observations = parseObservations(text, session.contentSessionId);

  // Store observations IMMEDIATELY
  for (const obs of observations) {
    const { id: obsId } = this.dbManager.getSessionStore().storeObservation(...);
    // ⚠️ OBSERVATION IS NOW IN DATABASE
  }

  // Parse and store summary
  const summary = parseSummary(text, session.sessionDbId);
  if (summary) {
    this.dbManager.getSessionStore().storeSummary(...);
    // ⚠️ SUMMARY IS NOW IN DATABASE
  }

  // ONLY NOW mark the message as processed
  await this.markMessagesProcessed(session, worker);  // ⚠️ LINE 487
}

// Line 494-502: markMessagesProcessed()
private async markMessagesProcessed(...): Promise<void> {
  const pendingMessageStore = this.sessionManager.getPendingMessageStore();
  if (session.pendingProcessingIds.size > 0) {
    for (const messageId of session.pendingProcessingIds) {
      pendingMessageStore.markProcessed(messageId);  // ⚠️ TOO LATE!
    }
  }
}

The Window of Vulnerability

Between storing observations (line ~340) and marking the message as processed (line 498), there is a critical window where:

1. Observations exist in database ✅
2. Message is still in 'processing' status ⚠️
3. If SDK crashes/exits → Message remains stuck in 'processing'

How Crash Recovery Makes It Worse

File: /Users/alexnewman/Scripts/claude-mem/src/services/worker/http/routes/SessionRoutes.ts

// Line 183-205: Generator .finally() block
.finally(() => {
  // Crash recovery: If not aborted and still has work, restart
  if (!wasAborted) {
    const pendingStore = this.sessionManager.getPendingMessageStore();
    const pendingCount = pendingStore.getPendingCount(sessionDbId);

    if (pendingCount > 0) {  // ⚠️ Counts 'processing' messages too!
      logger.info('SESSION', `Restarting generator after crash/exit`);

      // Restart generator
      setTimeout(() => {
        this.startGeneratorWithProvider(stillExists, ...);
      }, 1000);
    }
  }
});

File: /Users/alexnewman/Scripts/claude-mem/src/services/sqlite/PendingMessageStore.ts

// Line 319-326: getPendingCount()
getPendingCount(sessionDbId: number): number {
  const stmt = this.db.prepare(`
    SELECT COUNT(*) as count FROM pending_messages
    WHERE session_db_id = ? AND status IN ('pending', 'processing')  // ⚠️
  `);
  return result.count;
}

// Line 299-314: resetStuckMessages()
resetStuckMessages(thresholdMs: number): number {
  const stmt = this.db.prepare(`
    UPDATE pending_messages
    SET status = 'pending', started_processing_at_epoch = NULL
    WHERE status = 'processing' AND started_processing_at_epoch < ?  // ⚠️
  `);
  return result.changes;
}

The Duplication Sequence

1. SDK processes message #1 (e.g., "Read tool on SearchManager.ts")

   • Marks message as 'processing' in database
   • Sends observation prompt to SDK agent

2. SDK returns response with observation

   • parseObservations() extracts: "Fixed missing closing brace..."
   • storeObservation() saves observation #1 to database ✅
   • CRASH or ERROR occurs (e.g., from recent error handling changes)
   • markMessagesProcessed() NEVER CALLED ⚠️
   • Message remains in 'processing' status

3. Crash recovery triggers (line 184-204)

   • getPendingCount() finds message still in 'processing'
   • Generator restarts with 1-second delay

4. Worker restart or stuck message recovery

   • resetStuckMessages() resets message to 'pending'
   • Generator processes the SAME message again

5. SDK processes message #1 AGAIN

   • Same observation prompt sent to SDK
   • SDK returns SAME observation (deterministic from same file state)
   • storeObservation() saves observation #2 ✅ (DUPLICATE!)
   • Process may crash again, creating observation #3, #4, etc.

Why No Database Deduplication?

File: /Users/alexnewman/Scripts/claude-mem/src/services/sqlite/SessionStore.ts

// Line 1224-1229: storeObservation() - NO deduplication!
const stmt = this.db.prepare(`
  INSERT INTO observations
  (memory_session_id, project, type, title, subtitle, ...)
  VALUES (?, ?, ?, ?, ?, ...)  // ⚠️ No INSERT OR IGNORE, no uniqueness check
`);

The database table has:

• ❌ No UNIQUE constraint on (memory_session_id, title, subtitle, type)
• ❌ No INSERT OR IGNORE logic
• ❌ No deduplication check before insertion

Compare to the IMPORT logic which DOES have deduplication:

// Line ~1440: importObservation() HAS deduplication
const existing = this.checkObservationExists(
  obs.memory_session_id,
  obs.title,
  obs.subtitle,
  obs.type
);

if (existing) {
  return { imported: false, id: existing.id };  // ✅ Prevents duplicates
}

Connection to Anti-Pattern Cleanup Work

What Changed

Recent commits removed try-catch blocks as part of anti-pattern mitigation:

0123b15 refactor: add error handling back to SearchManager Chroma calls
776f4ea Refactor hooks to streamline error handling and loading states
0ea82bd refactor: improve error logging across SessionStore and mcp-server
379b0c1 refactor: improve error logging in SearchManager.ts
4c0cdec refactor: improve error handling in worker-service.ts

Commit 776f4ea made significant changes:

• Removed try-catch blocks from hooks (useContextPreview, usePagination, useSSE, useSettings)
• Modified SessionStore.ts error handling
• Modified SearchManager.ts error handling (3000+ lines changed)

How This Triggered the Bug

The duplication regression was latent - the race condition always existed. However:

1. Before: Large try-catch blocks suppressed errors

   • SDK errors were caught and logged
   • Generator continued running
   • Messages got marked as processed (eventually)

2. After: Error handling removed/streamlined

   • SDK errors now crash the generator
   • Generator exits before marking messages processed
   • Crash recovery restarts generator repeatedly
   • Same message processed multiple times

Evidence from Database

Session 75894 (content_session_id: 56f94e5d-2514-4d44-aa43-f5e31d9b4c38):

• 26 pending messages queued (all unique)
• Only 7 observations should have been created
• But 8+ duplicates of "Fixed missing closing brace" were created
• Created over 23-minute window (9:32 PM - 9:55 PM)
• Indicates repeated crashes and recoveries

Fix Strategy

Short-term Fix (Critical)

Option 1: Transaction-based atomic completion (RECOMMENDED)

Wrap observation storage and message completion in a single transaction:

// In SDKAgent.ts processSDKResponse()
private async processSDKResponse(...): Promise<void> {
  const pendingStore = this.sessionManager.getPendingMessageStore();

  // Start transaction
  const db = this.dbManager.getSessionStore().db;
  const saveTransaction = db.transaction(() => {
    // Parse and store observations
    const observations = parseObservations(text, session.contentSessionId);
    const observationIds = [];

    for (const obs of observations) {
      const { id } = this.dbManager.getSessionStore().storeObservation(...);
      observationIds.push(id);
    }

    // Parse and store summary
    const summary = parseSummary(text, session.sessionDbId);
    if (summary) {
      this.dbManager.getSessionStore().storeSummary(...);
    }

    // CRITICAL: Mark messages as processed IN SAME TRANSACTION
    for (const messageId of session.pendingProcessingIds) {
      pendingStore.markProcessed(messageId);
    }

    return observationIds;
  });

  // Execute transaction atomically
  const observationIds = saveTransaction();

  // Broadcast to SSE AFTER transaction commits
  for (const obsId of observationIds) {
    worker?.sseBroadcaster.broadcast(...);
  }
}

Option 2: Mark processed BEFORE storing (SIMPLER)

// In SDKAgent.ts processSDKResponse()
private async processSDKResponse(...): Promise<void> {
  // Mark messages as processed FIRST
  await this.markMessagesProcessed(session, worker);

  // Then store observations (idempotent)
  const observations = parseObservations(text, session.contentSessionId);
  for (const obs of observations) {
    this.dbManager.getSessionStore().storeObservation(...);
  }
}

Risk: If storage fails, message is marked complete but observation is lost. However, this is better than duplicates.

Medium-term Fix (Important)

Add database-level deduplication:

-- Add unique constraint
CREATE UNIQUE INDEX idx_observations_unique
ON observations(memory_session_id, title, subtitle, type);

-- Modify storeObservation() to use INSERT OR IGNORE
INSERT OR IGNORE INTO observations (...) VALUES (...);

Or use the existing checkObservationExists() logic:

// In SessionStore.ts storeObservation()
storeObservation(...): { id: number; createdAtEpoch: number } {
  // Check for existing observation
  const existing = this.checkObservationExists(
    memorySessionId,
    observation.title,
    observation.subtitle,
    observation.type
  );

  if (existing) {
    logger.debug('DB', 'Observation already exists, skipping', {
      obsId: existing.id,
      title: observation.title
    });
    return { id: existing.id, createdAtEpoch: existing.created_at_epoch };
  }

  // Insert new observation...
}

Long-term Fix (Architectural)

Redesign crash recovery to be idempotent:

1. Message status flow should be:

   • pending → processing → processed (one-way, no resets)

2. Stuck message recovery should:

   • Create NEW message for retry (with retry_count)
   • Mark old message as 'failed' or 'abandoned'
   • Never reset 'processing' → 'pending'

3. SDK agent should:

   • Track which observations were created for each message
   • Skip observation creation if message was already processed
   • Use message ID as idempotency key

Testing Plan

1. Reproduce the regression:

   • Create session with multiple tool uses
   • Force SDK crash during observation processing
   • Verify duplicates are NOT created with fix

2. Edge cases:

   • Test worker restart during observation storage
   • Test network failure during Chroma sync
   • Test database write failure scenarios

3. Performance:

   • Verify transaction doesn't slow down processing
   • Test with high observation volume (100+ per session)

Cleanup Required

Run the existing cleanup script to remove current duplicates:

cd /Users/alexnewman/Scripts/claude-mem
npm run cleanup-duplicates

This script identifies duplicates by (memory_session_id, title, subtitle, type) and keeps the earliest (MIN(id)).

Files Requiring Changes

1. src/services/worker/SDKAgent.ts - Add transaction or reorder completion
2. src/services/sqlite/SessionStore.ts - Add deduplication check
3. src/services/sqlite/migrations.ts - Add unique index (optional)
4. src/services/worker/http/routes/SessionRoutes.ts - Improve crash recovery logging

Estimated Impact

• Severity: Critical (data integrity)
• Scope: All sessions since 2026-01-02 ~9:30 PM
• User impact: Confusing duplicate memories, inflated token counts
• Database impact: ~50-100+ duplicate rows

References

• Original issue: Generator failure observations (11 duplicates)
• Related commit: 776f4ea "Refactor hooks to streamline error handling"
• Cleanup script: /Users/alexnewman/Scripts/claude-mem/src/bin/cleanup-duplicates.ts
• Related report: docs/reports/2026-01-02--stuck-observations.md