claude-mem/docs/chroma-search-completion-plan.md

Chroma Search Completion Plan

Current State Analysis

What's Working ✅

1. Hybrid Search Implementation

   • Chroma semantic search + SQLite temporal filtering is working
   • Evidence: Queries like "AI embeddings" find "hybrid search" through semantic similarity
   • All metadata-first tools use Chroma ranking

2. Tools Using Chroma Correctly

   • search_observations - Semantic-first workflow (Chroma top 100 → 90-day filter → SQLite hydrate)
   • find_by_concept - Metadata-first + Chroma semantic ranking
   • find_by_file - Metadata-first + Chroma semantic ranking
   • find_by_type - Metadata-first + Chroma semantic ranking

3. Data Synced to Chroma

   • ✅ Observations (all fields: narrative, facts, text as separate docs)
   • ✅ Session summaries (all fields: request, investigated, learned, completed, next_steps, notes as separate docs)
   • ❌ User prompts (NOT synced yet)

What's Missing ❌

1. search_sessions tool - Only uses SQLite FTS5, not leveraging Chroma semantic search
2. search_user_prompts tool - Only uses SQLite FTS5, not leveraging Chroma semantic search
3. User prompts not synced to Chroma - Need to add to sync experiment and worker process

Why User Prompts Need Semantic Search

Benefits:

• Users often search for "what I asked about X" but phrase it differently than original prompt
• Semantic search finds related requests even with different wording
• Example: Search "authentication setup" finds prompts about "login system", "user auth", "sign-in flow"
• Completes the triad: What was done (observations) + What was learned (summaries) + What was requested (prompts)

Storage pattern:

• Each user prompt becomes ONE document in Chroma (unlike observations/summaries which split by field)
• Metadata: sqlite_id, doc_type: 'user_prompt', sdk_session_id, project, created_at_epoch, prompt_number
• Document ID format: prompt_{id} (simpler than observations since no field splitting)

Implementation Plan

Phase 1: Sync User Prompts to Chroma

Files to modify:

1. experiment/chroma-sync-experiment.ts - Add user_prompts sync section
2. Future: Worker service incremental sync (not in this phase)

Implementation:

// In chroma-sync-experiment.ts after session summaries sync

// Fetch user prompts
console.log('📖 Reading user prompts from SQLite...');
const prompts = store.db.prepare(`
  SELECT * FROM user_prompts WHERE project = ? ORDER BY created_at_epoch DESC LIMIT 1000
`).all(project) as any[];
console.log(`Found ${prompts.length} user prompts`);

// Prepare prompt documents - one document per prompt
const promptDocs: ChromaDocument[] = [];

for (const prompt of prompts) {
  promptDocs.push({
    id: `prompt_${prompt.id}`,
    document: prompt.prompt_text,
    metadata: {
      sqlite_id: prompt.id,
      doc_type: 'user_prompt',
      sdk_session_id: prompt.sdk_session_id,
      project: prompt.project,
      created_at_epoch: prompt.created_at_epoch,
      prompt_number: prompt.prompt_number || 0
    }
  });
}

console.log(`Created ${promptDocs.length} user prompt documents\n`);

// Sync prompts in batches (same pattern as observations/sessions)

Testing:

npm run experiment:sync
# Verify prompts appear in Chroma collection

Phase 2: Update search_sessions to Use Chroma

File: src/servers/search-server.ts (lines ~441-481)

Current implementation:

const results = search.searchSessions(query, options);

New implementation (semantic-first hybrid):

let results: SessionSummarySearchResult[] = [];

// Hybrid search: Try Chroma semantic search first, fall back to FTS5
if (chromaClient) {
  try {
    console.error('[search-server] Using hybrid semantic search for sessions');

    // Step 1: Chroma semantic search (top 100)
    const chromaResults = await queryChroma(query, 100, { doc_type: 'session_summary' });
    console.error(`[search-server] Chroma returned ${chromaResults.ids.length} semantic matches`);

    if (chromaResults.ids.length > 0) {
      // Step 2: Filter by recency (90 days)
      const ninetyDaysAgo = Math.floor(Date.now() / 1000) - (90 * 24 * 60 * 60);
      const recentIds = chromaResults.ids.filter((id, idx) => {
        const meta = chromaResults.metadatas[idx];
        return meta && meta.created_at_epoch > ninetyDaysAgo;
      });

      console.error(`[search-server] ${recentIds.length} results within 90-day window`);

      // Step 3: Hydrate from SQLite in temporal order
      if (recentIds.length > 0) {
        const limit = options.limit || 20;
        results = store.getSessionSummariesByIds(recentIds, { orderBy: 'date_desc', limit });
        console.error(`[search-server] Hydrated ${results.length} sessions from SQLite`);
      }
    }
  } catch (chromaError: any) {
    console.error('[search-server] Chroma query failed, falling back to FTS5:', chromaError.message);
  }
}

// Fall back to FTS5 if Chroma unavailable or returned no results
if (results.length === 0) {
  console.error('[search-server] Using FTS5 keyword search');
  results = search.searchSessions(query, options);
}

Helper needed in queryChroma: Update queryChroma function to extract summary IDs from document IDs:

// Extract unique summary IDs from document IDs
for (const docId of docIds) {
  // Handle both obs_{id}_* and summary_{id}_* formats
  const obsMatch = docId.match(/obs_(\d+)_/);
  const summaryMatch = docId.match(/summary_(\d+)_/);

  if (obsMatch) {
    const sqliteId = parseInt(obsMatch[1], 10);
    if (!ids.includes(sqliteId)) ids.push(sqliteId);
  } else if (summaryMatch) {
    const sqliteId = parseInt(summaryMatch[1], 10);
    if (!ids.includes(sqliteId)) ids.push(sqliteId);
  }
}

Database helper needed: Add to SessionStore.ts:

getSessionSummariesByIds(
  ids: number[],
  options: { orderBy?: 'date_desc' | 'date_asc'; limit?: number } = {}
): SessionSummarySearchResult[] {
  if (ids.length === 0) return [];

  const { orderBy = 'date_desc', limit } = options;
  const orderClause = orderBy === 'date_asc' ? 'ASC' : 'DESC';
  const limitClause = limit ? `LIMIT ${limit}` : '';
  const placeholders = ids.map(() => '?').join(',');

  const stmt = this.db.prepare(`
    SELECT * FROM session_summaries
    WHERE id IN (${placeholders})
    ORDER BY created_at_epoch ${orderClause}
    ${limitClause}
  `);

  return stmt.all(...ids) as SessionSummarySearchResult[];
}

Phase 3: Update search_user_prompts to Use Chroma

File: src/servers/search-server.ts (lines ~956-1010)

Current implementation:

const results = search.searchUserPrompts(query, options);

New implementation (semantic-first hybrid):

let results: UserPromptSearchResult[] = [];

// Hybrid search: Try Chroma semantic search first, fall back to FTS5
if (chromaClient) {
  try {
    console.error('[search-server] Using hybrid semantic search for user prompts');

    // Step 1: Chroma semantic search (top 100)
    const chromaResults = await queryChroma(query, 100, { doc_type: 'user_prompt' });
    console.error(`[search-server] Chroma returned ${chromaResults.ids.length} semantic matches`);

    if (chromaResults.ids.length > 0) {
      // Step 2: Filter by recency (90 days)
      const ninetyDaysAgo = Math.floor(Date.now() / 1000) - (90 * 24 * 60 * 60);
      const recentIds = chromaResults.ids.filter((id, idx) => {
        const meta = chromaResults.metadatas[idx];
        return meta && meta.created_at_epoch > ninetyDaysAgo;
      });

      console.error(`[search-server] ${recentIds.length} results within 90-day window`);

      // Step 3: Hydrate from SQLite in temporal order
      if (recentIds.length > 0) {
        const limit = options.limit || 20;
        results = store.getUserPromptsByIds(recentIds, { orderBy: 'date_desc', limit });
        console.error(`[search-server] Hydrated ${results.length} user prompts from SQLite`);
      }
    }
  } catch (chromaError: any) {
    console.error('[search-server] Chroma query failed, falling back to FTS5:', chromaError.message);
  }
}

// Fall back to FTS5 if Chroma unavailable or returned no results
if (results.length === 0) {
  console.error('[search-server] Using FTS5 keyword search');
  results = search.searchUserPrompts(query, options);
}

Helper needed in queryChroma: Update to handle prompt_{id} format:

// Extract unique prompt IDs from document IDs
for (const docId of docIds) {
  const obsMatch = docId.match(/obs_(\d+)_/);
  const summaryMatch = docId.match(/summary_(\d+)_/);
  const promptMatch = docId.match(/prompt_(\d+)/);

  if (obsMatch) {
    const sqliteId = parseInt(obsMatch[1], 10);
    if (!ids.includes(sqliteId)) ids.push(sqliteId);
  } else if (summaryMatch) {
    const sqliteId = parseInt(summaryMatch[1], 10);
    if (!ids.includes(sqliteId)) ids.push(sqliteId);
  } else if (promptMatch) {
    const sqliteId = parseInt(promptMatch[1], 10);
    if (!ids.includes(sqliteId)) ids.push(sqliteId);
  }
}

Database helper needed: Add to SessionStore.ts:

getUserPromptsByIds(
  ids: number[],
  options: { orderBy?: 'date_desc' | 'date_asc'; limit?: number } = {}
): UserPromptSearchResult[] {
  if (ids.length === 0) return [];

  const { orderBy = 'date_desc', limit } = options;
  const orderClause = orderBy === 'date_asc' ? 'ASC' : 'DESC';
  const limitClause = limit ? `LIMIT ${limit}` : '';
  const placeholders = ids.map(() => '?').join(',');

  const stmt = this.db.prepare(`
    SELECT * FROM user_prompts
    WHERE id IN (${placeholders})
    ORDER BY created_at_epoch ${orderClause}
    ${limitClause}
  `);

  return stmt.all(...ids) as UserPromptSearchResult[];
}

Phase 4: Timeline Context Tool

New tool: get_context_timeline

Purpose: Show observations/sessions/prompts around a specific point in time

API:

{
  name: 'get_context_timeline',
  description: 'Get a timeline of context around a specific observation, session, or timestamp',
  inputSchema: z.object({
    anchor: z.union([
      z.number(), // observation ID
      z.string()  // ISO timestamp or session ID
    ]).describe('Anchor point: observation ID, session ID, or ISO timestamp'),
    depth_before: z.number().min(0).max(50).default(10).describe('Number of records to show before anchor'),
    depth_after: z.number().min(0).max(50).default(10).describe('Number of records to show after anchor'),
    format: z.enum(['index', 'full']).default('index'),
    project: z.string().optional()
  })
}

Implementation approach:

1. Resolve anchor to a timestamp (observation.created_at_epoch, session.created_at_epoch, or parse ISO)
2. Query observations within [anchor_time - depth_before_duration, anchor_time + depth_after_duration]
3. Return chronologically ordered results with anchor highlighted
4. Support mixing observations, sessions, and prompts in single timeline

Database helper:

getTimelineAroundTimestamp(
  anchorEpoch: number,
  depthBefore: number,
  depthAfter: number,
  project?: string
): { observations: any[], sessions: any[], prompts: any[] } {
  // Calculate time windows based on depth
  // For now: each depth = 1 hour (configurable)
  const hourInSeconds = 3600;
  const startEpoch = anchorEpoch - (depthBefore * hourInSeconds);
  const endEpoch = anchorEpoch + (depthAfter * hourInSeconds);

  // Query all three tables
  const observations = this.db.prepare(`...`).all(...);
  const sessions = this.db.prepare(`...`).all(...);
  const prompts = this.db.prepare(`...`).all(...);

  return { observations, sessions, prompts };
}

Testing Plan

Phase 1 Testing

# Run sync experiment
npm run experiment:sync

# Check Chroma collection for prompts
# Should see prompt_* documents with doc_type: 'user_prompt'

Phase 2 Testing

# Test semantic search for sessions
# Example: "authentication system" should find sessions about "login", "user auth", etc.

Phase 3 Testing

# Test semantic search for user prompts
# Example: "fix bug" should find prompts with "error", "issue", "problem", etc.

Phase 4 Testing

# Test timeline around specific observation
# Should show before/after context

Files to Modify

1. experiment/chroma-sync-experiment.ts - Add user_prompts sync
2. src/servers/search-server.ts - Update search_sessions and search_user_prompts, add get_context_timeline
3. src/services/sqlite/SessionStore.ts - Add getSessionSummariesByIds, getUserPromptsByIds, getTimelineAroundTimestamp
4. src/services/sqlite/types.ts - Ensure all return types are exported

Success Criteria

• ✅ All 8 search tools use Chroma semantic search with SQLite temporal fallback
• ✅ User prompts are synced to Chroma and searchable
• ✅ Timeline tool provides chronological context around any point
• ✅ Semantic search works across observations, sessions, and prompts
• ✅ All searches maintain 90-day temporal filtering for relevance

Future Enhancements

1. Incremental sync in worker service - Currently only batch sync via experiment
2. Configurable temporal windows - Make 90-day filter configurable
3. Cross-collection search - Search across observations + sessions + prompts in one query
4. Timeline view improvements - Group by session, highlight anchor, show relationships