Files
claude-mem/src/services/worker/http/routes/DataRoutes.ts
T
Alex Newman 61488042d8 Mem-search enhancements: table output, simplified API, Sonnet default, and removed fake URIs (#317)
* feat: Add batch fetching for observations and update documentation

- Implemented a new endpoint for fetching multiple observations by IDs in a single request.
- Updated the DataRoutes to include a POST /api/observations/batch endpoint.
- Enhanced SKILL.md documentation to reflect changes in the search process and batch fetching capabilities.
- Increased the default limit for search results from 5 to 40 for better usability.

* feat!: Fix timeline parameter passing with SearchManager alignment

BREAKING CHANGE: Timeline MCP tools now use standardized parameter names
- anchor_id → anchor
- before → depth_before
- after → depth_after
- obs_type → type (timeline tool only)

Fixes timeline endpoint failures caused by parameter name mismatch between
MCP layer and SearchManager. Adds new SessionStore methods for fetching
prompts and session summaries by ID.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* docs: reframe timeline parameter fix as bug fix, not breaking change

The timeline tools were completely broken due to parameter name mismatch.
There's nothing to migrate from since the old parameters never worked.

Co-authored-by: Alex Newman <thedotmack@users.noreply.github.com>

* Refactor mem-search documentation and optimize API tool definitions

- Updated SKILL.md to emphasize batch fetching for observations, clarifying usage and efficiency.
- Removed deprecated tools from mcp-server.ts and streamlined tool definitions for clarity.
- Enhanced formatting in FormattingService.ts for better output readability.
- Adjusted SearchManager.ts to improve result headers and removed unnecessary search tips from combined text.

* Refactor FormattingService and SearchManager for table-based output

- Updated FormattingService to format search results as tables, including methods for formatting observations, sessions, and user prompts.
- Removed JSON format handling from SearchManager and streamlined result formatting to consistently use table format.
- Enhanced readability and consistency in search tips and formatting logic.
- Introduced token estimation for observations and improved time formatting.

* refactor: update documentation and API references for version bump and search functionalities

* Refactor code structure for improved readability and maintainability

* chore: change default model from haiku to sonnet

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* feat: unify timeline formatting across search and context services

Extract shared timeline formatting utilities into reusable module to align
MCP search output format with context-generator's date/file-grouped format.

Changes:
- Create src/shared/timeline-formatting.ts with reusable utilities
  (parseJsonArray, formatDateTime, formatTime, formatDate, toRelativePath,
  extractFirstFile, groupByDate)
- Refactor context-generator.ts to use shared utilities
- Update SearchManager.search() to use date/file grouping
- Add search-specific row formatters to FormattingService
- Fix timeline methods to extract actual file paths from metadata
  instead of hardcoding 'General'
- Remove Work column from search output (kept in context output)

Result: Consistent date/file-grouped markdown formatting across both
systems while maintaining their different column requirements.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* refactor: remove redundant legend from search output

Remove legend from search/timeline results since it's already shown
in SessionStart context. Saves ~30 tokens per search result.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* Refactor session summary rendering to remove links

- Removed link generation for session summaries in context generation and search manager.
- Updated output formatting to exclude links while maintaining the session summary structure.
- Adjusted related components in TimelineService to ensure consistency across the application.

* fix: move skillPath declaration outside try block to fix scoping bug

The skillPath variable was declared inside the try block but referenced
in the catch block for error logging. Since const is block-scoped, this
would cause a ReferenceError when the error handler executes.

Moved skillPath declaration before the try block so it's accessible in
both try and catch scopes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* fix: address PR #317 code review feedback

**Critical Fixes:**
- Replace happy_path_error__with_fallback debug calls with proper logger methods in mcp-server.ts
- All HTTP API calls now use logger.debug/error for consistent logging

**Code Quality Improvements:**
- Extract 90-day recency window magic numbers to named constants
- Added RECENCY_WINDOW_DAYS and RECENCY_WINDOW_MS constants in SearchManager

**Documentation:**
- Document model cost implications of Haiku → Sonnet upgrade in CHANGELOG
- Provide clear migration path for users who want to revert to Haiku

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* refactor: simplify CHANGELOG - remove cost documentation

Removed model cost comparison documentation per user feedback.
Kept only the technical code quality improvements.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Alex Newman <thedotmack@users.noreply.github.com>
2025-12-14 21:58:11 -05:00

271 lines
8.8 KiB
TypeScript

/**
* Data Routes
*
* Handles data retrieval operations: observations, summaries, prompts, stats, processing status.
* All endpoints use direct database access via domain services.
*/
import express, { Request, Response } from 'express';
import path from 'path';
import { readFileSync, statSync, existsSync } from 'fs';
import { homedir } from 'os';
import { getPackageRoot } from '../../../../shared/paths.js';
import { getWorkerPort } from '../../../../shared/worker-utils.js';
import { PaginationHelper } from '../../PaginationHelper.js';
import { DatabaseManager } from '../../DatabaseManager.js';
import { SessionManager } from '../../SessionManager.js';
import { SSEBroadcaster } from '../../SSEBroadcaster.js';
import type { WorkerService } from '../../../worker-service.js';
import { BaseRouteHandler } from '../BaseRouteHandler.js';
export class DataRoutes extends BaseRouteHandler {
constructor(
private paginationHelper: PaginationHelper,
private dbManager: DatabaseManager,
private sessionManager: SessionManager,
private sseBroadcaster: SSEBroadcaster,
private workerService: WorkerService,
private startTime: number
) {
super();
}
setupRoutes(app: express.Application): void {
// Pagination endpoints
app.get('/api/observations', this.handleGetObservations.bind(this));
app.get('/api/summaries', this.handleGetSummaries.bind(this));
app.get('/api/prompts', this.handleGetPrompts.bind(this));
// Fetch by ID endpoints
app.get('/api/observation/:id', this.handleGetObservationById.bind(this));
app.post('/api/observations/batch', this.handleGetObservationsByIds.bind(this));
app.get('/api/session/:id', this.handleGetSessionById.bind(this));
app.get('/api/prompt/:id', this.handleGetPromptById.bind(this));
// Metadata endpoints
app.get('/api/stats', this.handleGetStats.bind(this));
app.get('/api/projects', this.handleGetProjects.bind(this));
// Processing status endpoints
app.get('/api/processing-status', this.handleGetProcessingStatus.bind(this));
app.post('/api/processing', this.handleSetProcessing.bind(this));
}
/**
* Get paginated observations
*/
private handleGetObservations = this.wrapHandler((req: Request, res: Response): void => {
const { offset, limit, project } = this.parsePaginationParams(req);
const result = this.paginationHelper.getObservations(offset, limit, project);
res.json(result);
});
/**
* Get paginated summaries
*/
private handleGetSummaries = this.wrapHandler((req: Request, res: Response): void => {
const { offset, limit, project } = this.parsePaginationParams(req);
const result = this.paginationHelper.getSummaries(offset, limit, project);
res.json(result);
});
/**
* Get paginated user prompts
*/
private handleGetPrompts = this.wrapHandler((req: Request, res: Response): void => {
const { offset, limit, project } = this.parsePaginationParams(req);
const result = this.paginationHelper.getPrompts(offset, limit, project);
res.json(result);
});
/**
* Get observation by ID
* GET /api/observation/:id
*/
private handleGetObservationById = this.wrapHandler((req: Request, res: Response): void => {
const id = this.parseIntParam(req, res, 'id');
if (id === null) return;
const store = this.dbManager.getSessionStore();
const observation = store.getObservationById(id);
if (!observation) {
this.notFound(res, `Observation #${id} not found`);
return;
}
res.json(observation);
});
/**
* Get observations by array of IDs
* POST /api/observations/batch
* Body: { ids: number[], orderBy?: 'date_desc' | 'date_asc', limit?: number, project?: string }
*/
private handleGetObservationsByIds = this.wrapHandler((req: Request, res: Response): void => {
const { ids, orderBy, limit, project } = req.body;
if (!ids || !Array.isArray(ids)) {
this.badRequest(res, 'ids must be an array of numbers');
return;
}
if (ids.length === 0) {
res.json([]);
return;
}
// Validate all IDs are numbers
if (!ids.every(id => typeof id === 'number' && Number.isInteger(id))) {
this.badRequest(res, 'All ids must be integers');
return;
}
const store = this.dbManager.getSessionStore();
const observations = store.getObservationsByIds(ids, { orderBy, limit, project });
res.json(observations);
});
/**
* Get session by ID
* GET /api/session/:id
*/
private handleGetSessionById = this.wrapHandler((req: Request, res: Response): void => {
const id = this.parseIntParam(req, res, 'id');
if (id === null) return;
const store = this.dbManager.getSessionStore();
const sessions = store.getSessionSummariesByIds([id]);
if (sessions.length === 0) {
this.notFound(res, `Session #${id} not found`);
return;
}
res.json(sessions[0]);
});
/**
* Get user prompt by ID
* GET /api/prompt/:id
*/
private handleGetPromptById = this.wrapHandler((req: Request, res: Response): void => {
const id = this.parseIntParam(req, res, 'id');
if (id === null) return;
const store = this.dbManager.getSessionStore();
const prompts = store.getUserPromptsByIds([id]);
if (prompts.length === 0) {
this.notFound(res, `Prompt #${id} not found`);
return;
}
res.json(prompts[0]);
});
/**
* Get database statistics (with worker metadata)
*/
private handleGetStats = this.wrapHandler((req: Request, res: Response): void => {
const db = this.dbManager.getSessionStore().db;
// Read version from package.json
const packageRoot = getPackageRoot();
const packageJsonPath = path.join(packageRoot, 'package.json');
const packageJson = JSON.parse(readFileSync(packageJsonPath, 'utf-8'));
const version = packageJson.version;
// Get database stats
const totalObservations = db.prepare('SELECT COUNT(*) as count FROM observations').get() as { count: number };
const totalSessions = db.prepare('SELECT COUNT(*) as count FROM sdk_sessions').get() as { count: number };
const totalSummaries = db.prepare('SELECT COUNT(*) as count FROM session_summaries').get() as { count: number };
// Get database file size and path
const dbPath = path.join(homedir(), '.claude-mem', 'claude-mem.db');
let dbSize = 0;
if (existsSync(dbPath)) {
dbSize = statSync(dbPath).size;
}
// Worker metadata
const uptime = Math.floor((Date.now() - this.startTime) / 1000);
const activeSessions = this.sessionManager.getActiveSessionCount();
const sseClients = this.sseBroadcaster.getClientCount();
res.json({
worker: {
version,
uptime,
activeSessions,
sseClients,
port: getWorkerPort()
},
database: {
path: dbPath,
size: dbSize,
observations: totalObservations.count,
sessions: totalSessions.count,
summaries: totalSummaries.count
}
});
});
/**
* Get list of distinct projects from observations
* GET /api/projects
*/
private handleGetProjects = this.wrapHandler((req: Request, res: Response): void => {
const db = this.dbManager.getSessionStore().db;
const rows = db.prepare(`
SELECT DISTINCT project
FROM observations
WHERE project IS NOT NULL
GROUP BY project
ORDER BY MAX(created_at_epoch) DESC
`).all() as Array<{ project: string }>;
const projects = rows.map(row => row.project);
res.json({ projects });
});
/**
* Get current processing status
* GET /api/processing-status
*/
private handleGetProcessingStatus = this.wrapHandler((req: Request, res: Response): void => {
const isProcessing = this.sessionManager.isAnySessionProcessing();
const queueDepth = this.sessionManager.getTotalActiveWork(); // Includes queued + actively processing
res.json({ isProcessing, queueDepth });
});
/**
* Set processing status (called by hooks)
* NOTE: This now broadcasts computed status based on active processing (ignores input)
*/
private handleSetProcessing = this.wrapHandler((req: Request, res: Response): void => {
// Broadcast current computed status (ignores manual input)
this.workerService.broadcastProcessingStatus();
const isProcessing = this.sessionManager.isAnySessionProcessing();
const queueDepth = this.sessionManager.getTotalQueueDepth();
const activeSessions = this.sessionManager.getActiveSessionCount();
res.json({ status: 'ok', isProcessing });
});
/**
* Parse pagination parameters from request query
*/
private parsePaginationParams(req: Request): { offset: number; limit: number; project?: string } {
const offset = parseInt(req.query.offset as string, 10) || 0;
const limit = Math.min(parseInt(req.query.limit as string, 10) || 20, 100); // Max 100
const project = req.query.project as string | undefined;
return { offset, limit, project };
}
}