Server-beta: Postgres storage + independent runtime + BullMQ queue (Phases 1–3) (#2351)

* Add server beta runtime foundation

* Address server beta review findings

* Resolve server beta review comments

* Tighten server beta review follow-ups

* Harden server beta auth and search

* Avoid unnecessary FTS rebuilds

* Block scoped keys from creating projects

* Release BullMQ claims best effort on close

* Address server beta review blockers

* Reset BullMQ claims best effort

* Add Postgres observation storage foundation

* feat(server-beta): add independent runtime service

Introduce src/server/runtime/ as a self-contained server-beta runtime
that owns its lifecycle, Postgres bootstrap, and HTTP boundary without
depending on WorkerService.

ServerBetaService wraps the existing Server class, exposes
/healthz and /v1/info with runtime="server-beta", and persists state
to dedicated paths (.server-beta.pid|.port|.runtime.json). The four
boundary managers (queue, generation worker, provider registry, event
broadcaster) are intentionally disabled in this phase and report their
status through /v1/info; later phases activate them.

Adds plans/2026-05-07-finish-bullmq-branch-ship-plan.md to track the
remaining work for this branch.

Phase 2 of plans/2026-05-07-server-beta-independent-bullmq-observation-runtime.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(server-beta): route CLI lifecycle and bundle separate runtime

scripts/build-hooks.js now produces plugin/scripts/server-beta-service.cjs
as a separate Node CJS bundle, alongside the existing worker-service
bundle. The server-beta runtime is now installable independently.

src/npx-cli/commands/server.ts routes start|stop|restart|status to the
server-beta lifecycle instead of the legacy worker. The worker keeps its
own start|stop|restart|status under the worker namespace; the two
runtimes can be operated independently.

src/services/worker-service.ts adds a server-* command parser branch
that delegates to the sibling server-beta-service.cjs bundle so
direct worker-service invocations still route to the right runtime.

tests/npx-cli-server-namespace.test.ts updated to expect server-beta
lifecycle routing.

Includes rebuilt plugin/scripts/*.cjs bundles produced by
build-and-sync.

Phase 2 of plans/2026-05-07-server-beta-independent-bullmq-observation-runtime.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(server-beta): add BullMQ job queue primitives

Introduce src/server/jobs/ as the queue-side primitives that Phase 3 of
the server-beta runtime needs to operate.

types.ts defines a discriminated union over the four job kinds (event,
event-batch, summary, reindex) and maps each to a per-kind BullMQ queue
name and deterministic-ID prefix.

job-id.ts builds deterministic, colon-free BullMQ jobIds from
(kind, team, project, source). The colon ban exists because BullMQ uses
':' as a Redis key separator internally; embedding ':' in jobIds
breaks scan and state lookups.

ServerJobQueue.ts is a thin wrapper over BullMQ Queue + Worker that
enforces autorun:false, default concurrency 1, and an attached error
listener — all per BullMQ docs requirements. Test seams accept queue
and worker factories so unit tests do not need Redis.

outbox.ts publishes through the Postgres ObservationGenerationJob
repository as canonical history. enqueueOutbox writes the row first,
then publishes to BullMQ; if BullMQ throws, the row is transitioned to
failed and a failed event is appended. reconcileOnStartup re-enqueues
queued + processing rows after a restart, replacing terminal BullMQ
jobs that may still be holding the deterministic ID slot. markCompleted
and markFailed wrap transitionStatus and append the matching event row.

Includes 20 unit tests covering deterministic ID stability, colon-free
output, queue lifecycle, error-listener attachment, double-start
refusal, idempotent enqueue, BullMQ failure rollback, startup
reconciliation, max-attempts skipping, and completion / failure /
retry transitions.

Phase 3 commit 1 of plans/2026-05-07-server-beta-independent-bullmq-observation-runtime.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(server-beta): activate queue boundary in runtime service

Wire ActiveServerBetaQueueManager into the server-beta runtime graph.
The active manager owns one ServerJobQueue per generation kind (event,
event-batch, summary, reindex) and surfaces lane metadata through
boundary health.

Selection is opt-in and fail-fast: if CLAUDE_MEM_QUEUE_ENGINE is set to
bullmq the active manager is constructed (and any Redis/config error
throws — no silent fallback to SQLite, per Phase 3 anti-pattern guard).
For any other engine the disabled boundary remains so worker-era and
test setups stay compatible.

Widens ServerBetaBoundaryHealth.status to a discriminated union
('disabled' | 'active' | 'errored') with optional details. The disabled
adapter still emits status='disabled', which keeps the existing
server-beta-service test green.

ServerBetaService receives the manager through a new optional
queueManager field on CreateServerBetaServiceOptions so test graphs
and Phase 4 wiring can inject custom managers.

Adds tests/server/runtime/active-queue-manager.test.ts covering bullmq
guard, active health shape, per-kind queue access, close behavior, and
post-close errored health.

Phase 3 commit 2 of plans/2026-05-07-server-beta-independent-bullmq-observation-runtime.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(server-beta): cap /v1/events/batch at 500 events

Prevents unbounded array DoS surface flagged in PR review.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alex Newman
2026-05-08 01:20:07 -07:00
committed by GitHub
parent 0a43ab7632
commit 36b0929fae
183 changed files with 35709 additions and 2033 deletions
+68
View File
@@ -0,0 +1,68 @@
// SPDX-License-Identifier: Apache-2.0
import type { CreateAgentEvent } from '../../core/schemas/agent-event.js';
import { normalizePlatformSource } from '../../shared/platform-source.js';
export interface ClaudeCodeBasePayload {
contentSessionId: string;
memorySessionId?: string | null;
platformSource?: string | null;
cwd?: string;
agentId?: string;
agentType?: string;
[key: string]: unknown;
}
export interface ClaudeCodeObservationPayload extends ClaudeCodeBasePayload {
tool_name: string;
tool_input?: unknown;
tool_response?: unknown;
tool_use_id?: string;
toolUseId?: string;
}
export function mapClaudeCodeSessionInitToAgentEvent(
projectId: string,
payload: ClaudeCodeBasePayload,
occurredAtEpoch = Date.now(),
): CreateAgentEvent {
return mapClaudeCodePayload(projectId, payload, 'session.init', occurredAtEpoch);
}
export function mapClaudeCodeObservationToAgentEvent(
projectId: string,
payload: ClaudeCodeObservationPayload,
occurredAtEpoch = Date.now(),
): CreateAgentEvent {
return mapClaudeCodePayload(projectId, payload, 'observation.created', occurredAtEpoch);
}
export function mapClaudeCodeSummaryToAgentEvent(
projectId: string,
payload: ClaudeCodeBasePayload,
occurredAtEpoch = Date.now(),
): CreateAgentEvent {
return mapClaudeCodePayload(projectId, payload, 'session.summary', occurredAtEpoch);
}
function mapClaudeCodePayload(
projectId: string,
payload: ClaudeCodeBasePayload,
eventType: string,
occurredAtEpoch: number,
): CreateAgentEvent {
const platformSource = normalizePlatformSource(payload.platformSource);
return {
projectId,
sourceType: 'hook',
eventType,
payload: {
...payload,
platformSource,
toolUseId: payload.toolUseId ?? payload.tool_use_id ?? null,
},
contentSessionId: payload.contentSessionId,
memorySessionId: payload.memorySessionId ?? null,
occurredAtEpoch,
};
}
+42
View File
@@ -0,0 +1,42 @@
// SPDX-License-Identifier: Apache-2.0
export const genericRestEventExamples = {
codexObservation: {
projectId: 'project-id',
sourceType: 'api',
eventType: 'observation.created',
contentSessionId: 'codex-session-id',
payload: {
platformSource: 'codex',
tool_name: 'shell',
cwd: '/workspace/project',
agentId: 'codex-agent-id',
agentType: 'codex',
toolUseId: 'tool-call-id',
tool_input: { command: 'bun test' },
tool_response: { exitCode: 0 },
},
occurredAtEpoch: 1760000000000,
},
opencodeObservation: {
projectId: 'project-id',
sourceType: 'api',
eventType: 'observation.created',
contentSessionId: 'opencode-session-id',
payload: {
platformSource: 'opencode',
tool_name: 'edit',
cwd: '/workspace/project',
toolUseId: 'tool-call-id',
},
occurredAtEpoch: 1760000000000,
},
customMemory: {
projectId: 'project-id',
kind: 'manual',
type: 'note',
title: 'Decision',
narrative: 'Store canonical memory records in SQLite; Redis is queue state only.',
facts: ['SQLite is the source of truth for memories'],
},
} as const;
+32
View File
@@ -0,0 +1,32 @@
// SPDX-License-Identifier: Apache-2.0
import { z } from 'zod';
export const AgentEventSourceTypeSchema = z.enum(['hook', 'worker', 'provider', 'server', 'api']);
export const AgentEventSchema = z.object({
id: z.string().min(1),
projectId: z.string().min(1),
serverSessionId: z.string().min(1).nullable().default(null),
sourceType: AgentEventSourceTypeSchema,
eventType: z.string().min(1),
payload: z.unknown().default({}),
contentSessionId: z.string().min(1).nullable().default(null),
memorySessionId: z.string().min(1).nullable().default(null),
occurredAtEpoch: z.number().int().nonnegative(),
createdAtEpoch: z.number().int().nonnegative()
});
export const CreateAgentEventSchema = AgentEventSchema.omit({
id: true,
createdAtEpoch: true
}).partial({
serverSessionId: true,
payload: true,
contentSessionId: true,
memorySessionId: true
});
export type AgentEventSourceType = z.infer<typeof AgentEventSourceTypeSchema>;
export type AgentEvent = z.infer<typeof AgentEventSchema>;
export type CreateAgentEvent = z.infer<typeof CreateAgentEventSchema>;
+69
View File
@@ -0,0 +1,69 @@
// SPDX-License-Identifier: Apache-2.0
import { z } from 'zod';
export const ApiKeyStatusSchema = z.enum(['active', 'revoked']);
export const AuditActorTypeSchema = z.enum(['user', 'api_key', 'system']);
export const ApiKeySchema = z.object({
id: z.string().min(1),
teamId: z.string().min(1).nullable().default(null),
projectId: z.string().min(1).nullable().default(null),
name: z.string().min(1),
keyHash: z.string().min(1),
prefix: z.string().min(1).nullable().default(null),
scopes: z.array(z.string()).default([]),
status: ApiKeyStatusSchema.default('active'),
lastUsedAtEpoch: z.number().int().nonnegative().nullable().default(null),
expiresAtEpoch: z.number().int().nonnegative().nullable().default(null),
metadata: z.record(z.string(), z.unknown()).default({}),
createdAtEpoch: z.number().int().nonnegative(),
updatedAtEpoch: z.number().int().nonnegative()
});
export const CreateApiKeySchema = ApiKeySchema.omit({
id: true,
status: true,
lastUsedAtEpoch: true,
createdAtEpoch: true,
updatedAtEpoch: true
}).partial({
teamId: true,
projectId: true,
prefix: true,
scopes: true,
expiresAtEpoch: true,
metadata: true
});
export const AuditLogSchema = z.object({
id: z.string().min(1),
teamId: z.string().min(1).nullable().default(null),
projectId: z.string().min(1).nullable().default(null),
actorType: AuditActorTypeSchema,
actorId: z.string().min(1).nullable().default(null),
action: z.string().min(1),
targetType: z.string().min(1).nullable().default(null),
targetId: z.string().min(1).nullable().default(null),
metadata: z.record(z.string(), z.unknown()).default({}),
createdAtEpoch: z.number().int().nonnegative()
});
export const CreateAuditLogSchema = AuditLogSchema.omit({
id: true,
createdAtEpoch: true
}).partial({
teamId: true,
projectId: true,
actorId: true,
targetType: true,
targetId: true,
metadata: true
});
export type ApiKeyStatus = z.infer<typeof ApiKeyStatusSchema>;
export type ApiKey = z.infer<typeof ApiKeySchema>;
export type CreateApiKey = z.infer<typeof CreateApiKeySchema>;
export type AuditActorType = z.infer<typeof AuditActorTypeSchema>;
export type AuditLog = z.infer<typeof AuditLogSchema>;
export type CreateAuditLog = z.infer<typeof CreateAuditLogSchema>;
+15
View File
@@ -0,0 +1,15 @@
// SPDX-License-Identifier: Apache-2.0
import { z } from 'zod';
import { MemoryItemSchema } from './memory-item.js';
export const ContextPackSchema = z.object({
projectId: z.string().min(1),
serverSessionId: z.string().min(1).nullable().default(null),
generatedAtEpoch: z.number().int().nonnegative(),
tokenBudget: z.number().int().positive().nullable().default(null),
items: z.array(MemoryItemSchema).default([]),
metadata: z.record(z.string(), z.unknown()).default({})
});
export type ContextPack = z.infer<typeof ContextPackSchema>;
+9
View File
@@ -0,0 +1,9 @@
// SPDX-License-Identifier: Apache-2.0
export * from './agent-event.js';
export * from './auth.js';
export * from './context-pack.js';
export * from './memory-item.js';
export * from './project.js';
export * from './session.js';
export * from './team.js';
+72
View File
@@ -0,0 +1,72 @@
// SPDX-License-Identifier: Apache-2.0
import { z } from 'zod';
export const MemoryItemKindSchema = z.enum(['observation', 'summary', 'prompt', 'manual']);
export const MemorySourceTypeSchema = z.enum(['observation', 'session_summary', 'user_prompt', 'manual', 'import']);
export const MemoryItemSchema = z.object({
id: z.string().min(1),
projectId: z.string().min(1),
serverSessionId: z.string().min(1).nullable().default(null),
legacyObservationId: z.number().int().positive().nullable().default(null),
kind: MemoryItemKindSchema,
type: z.string().min(1),
title: z.string().min(1).nullable().default(null),
subtitle: z.string().min(1).nullable().default(null),
text: z.string().nullable().default(null),
narrative: z.string().nullable().default(null),
facts: z.array(z.string()).default([]),
concepts: z.array(z.string()).default([]),
filesRead: z.array(z.string()).default([]),
filesModified: z.array(z.string()).default([]),
metadata: z.record(z.string(), z.unknown()).default({}),
createdAtEpoch: z.number().int().nonnegative(),
updatedAtEpoch: z.number().int().nonnegative()
});
export const CreateMemoryItemSchema = MemoryItemSchema.omit({
id: true,
createdAtEpoch: true,
updatedAtEpoch: true
}).partial({
serverSessionId: true,
legacyObservationId: true,
title: true,
subtitle: true,
text: true,
narrative: true,
facts: true,
concepts: true,
filesRead: true,
filesModified: true,
metadata: true
});
export const MemorySourceSchema = z.object({
id: z.string().min(1),
memoryItemId: z.string().min(1),
sourceType: MemorySourceTypeSchema,
legacyTable: z.string().min(1).nullable().default(null),
legacyId: z.number().int().positive().nullable().default(null),
sourceUri: z.string().min(1).nullable().default(null),
metadata: z.record(z.string(), z.unknown()).default({}),
createdAtEpoch: z.number().int().nonnegative()
});
export const CreateMemorySourceSchema = MemorySourceSchema.omit({
id: true,
createdAtEpoch: true
}).partial({
legacyTable: true,
legacyId: true,
sourceUri: true,
metadata: true
});
export type MemoryItemKind = z.infer<typeof MemoryItemKindSchema>;
export type MemoryItem = z.infer<typeof MemoryItemSchema>;
export type CreateMemoryItem = z.infer<typeof CreateMemoryItemSchema>;
export type MemorySourceType = z.infer<typeof MemorySourceTypeSchema>;
export type MemorySource = z.infer<typeof MemorySourceSchema>;
export type CreateMemorySource = z.infer<typeof CreateMemorySourceSchema>;
+26
View File
@@ -0,0 +1,26 @@
// SPDX-License-Identifier: Apache-2.0
import { z } from 'zod';
export const ProjectSchema = z.object({
id: z.string().min(1),
name: z.string().min(1),
slug: z.string().min(1).nullable().default(null),
rootPath: z.string().min(1).nullable().default(null),
metadata: z.record(z.string(), z.unknown()).default({}),
createdAtEpoch: z.number().int().nonnegative(),
updatedAtEpoch: z.number().int().nonnegative()
});
export const CreateProjectSchema = ProjectSchema.omit({
id: true,
createdAtEpoch: true,
updatedAtEpoch: true
}).partial({
slug: true,
rootPath: true,
metadata: true
});
export type Project = z.infer<typeof ProjectSchema>;
export type CreateProject = z.infer<typeof CreateProjectSchema>;
+37
View File
@@ -0,0 +1,37 @@
// SPDX-License-Identifier: Apache-2.0
import { z } from 'zod';
export const ServerSessionStatusSchema = z.enum(['active', 'completed', 'failed']);
export const ServerSessionSchema = z.object({
id: z.string().min(1),
projectId: z.string().min(1),
contentSessionId: z.string().min(1).nullable().default(null),
memorySessionId: z.string().min(1).nullable().default(null),
platformSource: z.string().min(1).default('claude'),
title: z.string().min(1).nullable().default(null),
status: ServerSessionStatusSchema.default('active'),
metadata: z.record(z.string(), z.unknown()).default({}),
startedAtEpoch: z.number().int().nonnegative(),
completedAtEpoch: z.number().int().nonnegative().nullable().default(null),
updatedAtEpoch: z.number().int().nonnegative()
});
export const CreateServerSessionSchema = ServerSessionSchema.omit({
id: true,
startedAtEpoch: true,
status: true,
completedAtEpoch: true,
updatedAtEpoch: true
}).partial({
contentSessionId: true,
memorySessionId: true,
platformSource: true,
title: true,
metadata: true
});
export type ServerSessionStatus = z.infer<typeof ServerSessionStatusSchema>;
export type ServerSession = z.infer<typeof ServerSessionSchema>;
export type CreateServerSession = z.infer<typeof CreateServerSessionSchema>;
+45
View File
@@ -0,0 +1,45 @@
// SPDX-License-Identifier: Apache-2.0
import { z } from 'zod';
export const TeamRoleSchema = z.enum(['owner', 'admin', 'member', 'viewer']);
export const TeamSchema = z.object({
id: z.string().min(1),
name: z.string().min(1),
slug: z.string().min(1).nullable().default(null),
metadata: z.record(z.string(), z.unknown()).default({}),
createdAtEpoch: z.number().int().nonnegative(),
updatedAtEpoch: z.number().int().nonnegative()
});
export const CreateTeamSchema = TeamSchema.omit({
id: true,
createdAtEpoch: true,
updatedAtEpoch: true
}).partial({
slug: true,
metadata: true
});
export const TeamMemberSchema = z.object({
id: z.string().min(1),
teamId: z.string().min(1),
userId: z.string().min(1),
role: TeamRoleSchema,
metadata: z.record(z.string(), z.unknown()).default({}),
createdAtEpoch: z.number().int().nonnegative()
});
export const CreateTeamMemberSchema = TeamMemberSchema.omit({
id: true,
createdAtEpoch: true
}).partial({
metadata: true
});
export type TeamRole = z.infer<typeof TeamRoleSchema>;
export type Team = z.infer<typeof TeamSchema>;
export type CreateTeam = z.infer<typeof CreateTeamSchema>;
export type TeamMember = z.infer<typeof TeamMemberSchema>;
export type CreateTeamMember = z.infer<typeof CreateTeamMemberSchema>;
+38 -8
View File
@@ -619,6 +619,7 @@ function mergeSettings(updates: Record<string, string>): boolean {
type ProviderId = 'claude' | 'gemini' | 'openrouter';
type ClaudeAccessMode = 'subscription' | 'api-key';
type ClaudeApiMode = 'direct' | 'gateway';
type RuntimeId = 'worker' | 'server-beta';
function readRawStoredAuthMethod(): 'subscription' | 'api-key' | 'gateway' | undefined {
try {
@@ -642,6 +643,32 @@ function resolveClaudeAuthMethod(): 'subscription' | 'api-key' | 'gateway' {
return 'subscription';
}
async function promptRuntime(): Promise<RuntimeId> {
if (!isInteractive) {
mergeSettings({ CLAUDE_MEM_RUNTIME: 'worker' });
return 'worker';
}
const selected = await p.select<RuntimeId>({
message: 'Which runtime should claude-mem start after install?',
options: [
{ value: 'worker', label: 'Worker', hint: 'stable compatibility path' },
{ value: 'server-beta', label: 'Server (beta)', hint: 'REST V1, API keys, team-ready storage' },
],
initialValue: 'worker',
});
if (p.isCancel(selected)) {
p.cancel('Installation cancelled.');
process.exit(0);
}
mergeSettings({
CLAUDE_MEM_RUNTIME: selected,
});
return selected;
}
async function promptProvider(options: InstallOptions): Promise<ProviderId> {
const initialProvider = (getSetting('CLAUDE_MEM_PROVIDER') as ProviderId) || 'claude';
@@ -1025,6 +1052,7 @@ export async function runInstallCommand(options: InstallOptions = {}): Promise<v
selectedIDEs = ['claude-code'];
}
const selectedRuntime = await promptRuntime();
const selectedProvider = await promptProvider(options);
if (selectedProvider === 'claude') {
await promptClaudeModel(options);
@@ -1169,7 +1197,7 @@ export async function runInstallCommand(options: InstallOptions = {}): Promise<v
await runTasks([
{
title: 'Starting worker daemon',
title: selectedRuntime === 'server-beta' ? 'Starting server beta daemon' : 'Starting worker daemon',
task: async (message) => {
if (autoStartSkipped) {
return isInteractive
@@ -1180,15 +1208,15 @@ export async function runInstallCommand(options: InstallOptions = {}): Promise<v
const marketplaceScriptPath = join(marketplaceDirectory(), 'plugin', 'scripts', 'worker-service.cjs');
const cacheScriptPath = join(pluginCacheDirectory(version), 'scripts', 'worker-service.cjs');
const scriptPath = existsSync(marketplaceScriptPath) ? marketplaceScriptPath : cacheScriptPath;
message(`Spawning worker on port ${port}...`);
message(`Spawning ${selectedRuntime === 'server-beta' ? 'server beta' : 'worker'} on port ${port}...`);
workerStartResult = await ensureWorkerStarted(port, scriptPath);
switch (workerStartResult) {
case 'ready':
return `Worker ready at http://localhost:${port} ${pc.green('OK')}`;
return `${selectedRuntime === 'server-beta' ? 'Server beta' : 'Worker'} ready at http://localhost:${port} ${pc.green('OK')}`;
case 'warming':
return `Worker starting on port ${port} — finishing in background ${pc.yellow('⏳')}`;
return `${selectedRuntime === 'server-beta' ? 'Server beta' : 'Worker'} starting on port ${port} — finishing in background ${pc.yellow('⏳')}`;
case 'dead':
return `Worker did not start — try \`npx claude-mem start\` manually ${pc.yellow('!')}`;
return `${selectedRuntime === 'server-beta' ? 'Server beta' : 'Worker'} did not start — try \`${selectedRuntime === 'server-beta' ? 'npx claude-mem server start' : 'npx claude-mem start'}\` manually ${pc.yellow('!')}`;
}
},
},
@@ -1256,11 +1284,13 @@ export async function runInstallCommand(options: InstallOptions = {}): Promise<v
const finalWorkerState = workerStartResult as WorkerStartResult;
const workerAlive = finalWorkerState !== 'dead' || workerReady;
const runtimeLabel = selectedRuntime === 'server-beta' ? 'Server beta' : 'Worker';
const runtimeStartCommand = selectedRuntime === 'server-beta' ? 'npx claude-mem server start' : 'npx claude-mem start';
const workerHeadline = autoStartSkipped
? `${pc.yellow('!')} Worker autostart skipped — start it manually with ${pc.bold('npx claude-mem start')}`
? `${pc.yellow('!')} ${runtimeLabel} autostart skipped — start it manually with ${pc.bold(runtimeStartCommand)}`
: workerReady || finalWorkerState === 'ready'
? `${pc.green('✓')} Worker running at ${pc.underline(`http://localhost:${actualPort}`)}`
: `${pc.yellow('⏳')} Worker starting at ${pc.underline(`http://localhost:${actualPort}`)} — give it ~30s, then refresh`;
? `${pc.green('✓')} ${runtimeLabel} running at ${pc.underline(`http://localhost:${actualPort}`)}`
: `${pc.yellow('⏳')} ${runtimeLabel} starting at ${pc.underline(`http://localhost:${actualPort}`)} — give it ~30s, then refresh`;
const nextSteps = autoStartSkipped
? [
workerHeadline,
+51
View File
@@ -29,6 +29,10 @@ function workerServiceScriptPath(): string {
return join(marketplaceDirectory(), 'plugin', 'scripts', 'worker-service.cjs');
}
function serverBetaServiceScriptPath(): string {
return join(marketplaceDirectory(), 'plugin', 'scripts', 'server-beta-service.cjs');
}
function spawnBunWorkerCommand(command: string, extraArgs: string[] = []): void {
ensureInstalledOrExit();
const bunPath = resolveBunOrExit();
@@ -58,6 +62,49 @@ function spawnBunWorkerCommand(command: string, extraArgs: string[] = []): void
});
}
function spawnBunServerBetaCommand(command: string): void {
ensureInstalledOrExit();
const bunPath = resolveBunOrExit();
const serverScript = serverBetaServiceScriptPath();
if (!existsSync(serverScript)) {
console.error(pc.red(`Server beta script not found at: ${serverScript}`));
console.error('The installation may be corrupted. Try: npx claude-mem install');
process.exit(1);
}
const child = spawnHidden(bunPath, [serverScript, command], {
stdio: 'inherit',
cwd: marketplaceDirectory(),
env: process.env,
});
child.on('error', (error) => {
console.error(pc.red(`Failed to start Bun: ${error.message}`));
process.exit(1);
});
child.on('close', (exitCode) => {
process.exit(exitCode ?? 0);
});
}
export function runServerBetaStartCommand(): void {
spawnBunServerBetaCommand('start');
}
export function runServerBetaStopCommand(): void {
spawnBunServerBetaCommand('stop');
}
export function runServerBetaRestartCommand(): void {
spawnBunServerBetaCommand('restart');
}
export function runServerBetaStatusCommand(): void {
spawnBunServerBetaCommand('status');
}
export function runStartCommand(): void {
spawnBunWorkerCommand('start');
}
@@ -74,6 +121,10 @@ export function runStatusCommand(): void {
spawnBunWorkerCommand('status');
}
export function runServerApiKeyCommand(extraArgs: string[] = []): void {
spawnBunWorkerCommand('server', ['api-key', ...extraArgs]);
}
export function runAdoptCommand(extraArgs: string[] = []): void {
ensureInstalledOrExit();
const bunPath = resolveBunOrExit();
+111
View File
@@ -0,0 +1,111 @@
import pc from 'picocolors';
import {
runServerBetaRestartCommand,
runServerBetaStartCommand,
runServerBetaStatusCommand,
runServerBetaStopCommand,
runRestartCommand,
runServerApiKeyCommand,
runStartCommand,
runStatusCommand,
runStopCommand,
} from './runtime.js';
const UNSUPPORTED_SERVER_COMMANDS = new Set([
'logs',
'doctor',
'migrate',
'export',
'import',
]);
function printServerUsage(): void {
console.error(`Usage: ${pc.bold('npx claude-mem server <command>')}`);
console.error('Commands: start, stop, restart, status, logs, doctor, migrate, export, import, api-key create|list|revoke');
}
function failUnsupported(command: string): never {
console.error(pc.red(`Server command not implemented yet: ${command}`));
console.error('This CLI route is reserved for the server runtime, but no backend API exists for it yet.');
process.exit(1);
}
function runWorkerLifecycleCommand(command: string): boolean {
switch (command) {
case 'start':
runStartCommand();
return true;
case 'stop':
runStopCommand();
return true;
case 'restart':
runRestartCommand();
return true;
case 'status':
runStatusCommand();
return true;
default:
return false;
}
}
function runServerBetaLifecycleCommand(command: string): boolean {
switch (command) {
case 'start':
runServerBetaStartCommand();
return true;
case 'stop':
runServerBetaStopCommand();
return true;
case 'restart':
runServerBetaRestartCommand();
return true;
case 'status':
runServerBetaStatusCommand();
return true;
default:
return false;
}
}
export async function runServerCommand(argv: string[] = []): Promise<void> {
const subCommand = argv[0]?.toLowerCase();
if (!subCommand) {
printServerUsage();
process.exit(1);
}
if (UNSUPPORTED_SERVER_COMMANDS.has(subCommand)) {
failUnsupported(`server ${subCommand}`);
}
if (runServerBetaLifecycleCommand(subCommand)) {
return;
}
if (subCommand === 'api-key') {
const apiKeyCommand = argv[1]?.toLowerCase();
if (apiKeyCommand === 'create' || apiKeyCommand === 'list' || apiKeyCommand === 'revoke') {
runServerApiKeyCommand(argv.slice(1));
return;
}
console.error(pc.red(`Unknown server api-key subcommand: ${apiKeyCommand ?? '(none)'}`));
console.error('Usage: npx claude-mem server api-key create|list|revoke');
process.exit(1);
}
console.error(pc.red(`Unknown server command: ${subCommand}`));
printServerUsage();
process.exit(1);
}
export function runWorkerAliasCommand(argv: string[] = []): void {
const subCommand = argv[0]?.toLowerCase();
if (!subCommand || !runWorkerLifecycleCommand(subCommand)) {
console.error(pc.red(`Unknown worker command: ${subCommand ?? '(none)'}`));
console.error('Usage: npx claude-mem worker start|stop|restart|status');
process.exit(1);
}
}
+23
View File
@@ -36,6 +36,17 @@ ${pc.bold('Runtime Commands')} (requires Bun, delegates to installed plugin):
${pc.cyan('npx claude-mem stop')} Stop worker service
${pc.cyan('npx claude-mem restart')} Restart worker service
${pc.cyan('npx claude-mem status')} Show worker status
${pc.cyan('npx claude-mem server start')} Start server service
${pc.cyan('npx claude-mem server stop')} Stop server service
${pc.cyan('npx claude-mem server restart')} Restart server service
${pc.cyan('npx claude-mem server status')} Show server status
${pc.cyan('npx claude-mem server logs')} Show recent server logs
${pc.cyan('npx claude-mem server doctor')} Check server configuration (not yet implemented)
${pc.cyan('npx claude-mem server migrate')} Run server migrations (not yet implemented)
${pc.cyan('npx claude-mem server export')} Export server data (not yet implemented)
${pc.cyan('npx claude-mem server import')} Import server data (not yet implemented)
${pc.cyan('npx claude-mem server api-key create|list|revoke')} Manage API keys (not yet implemented)
${pc.cyan('npx claude-mem worker start|stop|restart|status')} Worker compatibility aliases
${pc.cyan('npx claude-mem search <query>')} Search observations
${pc.cyan('npx claude-mem adopt [--dry-run] [--branch <name>]')} Stamp merged worktrees into parent project
${pc.cyan('npx claude-mem cleanup [--dry-run]')} Run one-time v12.4.3 pollution cleanup (or preview counts)
@@ -139,6 +150,18 @@ async function main(): Promise<void> {
break;
}
case 'server': {
const { runServerCommand } = await import('./commands/server.js');
await runServerCommand(args.slice(1));
break;
}
case 'worker': {
const { runWorkerAliasCommand } = await import('./commands/server.js');
runWorkerAliasCommand(args.slice(1));
break;
}
case 'search': {
const { runSearchCommand } = await import('./commands/runtime.js');
await runSearchCommand(args.slice(1));
+39
View File
@@ -0,0 +1,39 @@
// SPDX-License-Identifier: Apache-2.0
import type { Application } from 'express';
import type { Database } from 'bun:sqlite';
import type { RouteHandler } from '../../services/server/Server.js';
type NodeHandler = ReturnType<typeof import('better-auth/node').toNodeHandler>;
const cachedHandlers = new WeakMap<Database, NodeHandler>();
async function getBetterAuthHandler(database: Database): Promise<NodeHandler> {
const cachedHandler = cachedHandlers.get(database);
if (cachedHandler) {
return cachedHandler;
}
const [{ toNodeHandler }, { createAuth }] = await Promise.all([
import('better-auth/node'),
import('./auth.js'),
]);
const handler = toNodeHandler(createAuth(database));
cachedHandlers.set(database, handler);
return handler;
}
export class BetterAuthRoutes implements RouteHandler {
constructor(private readonly getDatabase: () => Database) {}
setupRoutes(app: Application): void {
app.all('/api/auth/*splat', async (req, res, next) => {
try {
const handler = await getBetterAuthHandler(this.getDatabase());
await handler(req, res);
} catch (error) {
next(error);
}
});
}
}
+118
View File
@@ -0,0 +1,118 @@
// SPDX-License-Identifier: Apache-2.0
import { createHash, randomBytes } from 'crypto';
import { Database } from 'bun:sqlite';
import { AuthRepository, ensureServerStorageSchema } from '../../storage/sqlite/index.js';
import type { ApiKey } from '../../core/schemas/auth.js';
export interface CreatedServerApiKey {
rawKey: string;
record: ApiKey;
}
export interface VerifiedServerApiKey {
record: ApiKey;
teamId: string | null;
projectId: string | null;
scopes: string[];
}
export interface CreateServerApiKeyInput {
name: string;
teamId?: string | null;
projectId?: string | null;
scopes?: string[];
expiresAtEpoch?: number | null;
metadata?: Record<string, unknown>;
}
export function hashServerApiKey(rawKey: string): string {
return createHash('sha256').update(rawKey).digest('hex');
}
export function createRawServerApiKey(): string {
return `cmem_${randomBytes(32).toString('base64url')}`;
}
export function createServerApiKey(db: Database, input: CreateServerApiKeyInput): CreatedServerApiKey {
ensureServerStorageSchema(db);
const rawKey = createRawServerApiKey();
const repo = new AuthRepository(db);
const record = repo.createApiKey({
name: input.name,
teamId: input.teamId ?? null,
projectId: input.projectId ?? null,
keyHash: hashServerApiKey(rawKey),
prefix: rawKey.slice(0, 10),
scopes: input.scopes ?? [],
expiresAtEpoch: input.expiresAtEpoch ?? null,
metadata: input.metadata ?? {},
});
repo.createAuditLog({
teamId: record.teamId,
projectId: record.projectId,
actorType: 'system',
action: 'api_key.create',
targetType: 'api_key',
targetId: record.id,
});
return { rawKey, record };
}
export function verifyServerApiKey(
db: Database,
rawKey: string,
requiredScopes: string[] = [],
): VerifiedServerApiKey | null {
ensureServerStorageSchema(db);
const repo = new AuthRepository(db);
const record = repo.getApiKeyByHash(hashServerApiKey(rawKey));
if (!record || record.status !== 'active') {
return null;
}
if (record.expiresAtEpoch !== null && record.expiresAtEpoch <= Date.now()) {
return null;
}
if (!hasRequiredScopes(record.scopes, requiredScopes)) {
return null;
}
repo.markApiKeyUsed(record.id);
return {
record,
teamId: record.teamId,
projectId: record.projectId,
scopes: record.scopes,
};
}
export function listServerApiKeys(db: Database): ApiKey[] {
ensureServerStorageSchema(db);
return new AuthRepository(db).listApiKeys();
}
export function revokeServerApiKey(db: Database, id: string): ApiKey | null {
ensureServerStorageSchema(db);
const repo = new AuthRepository(db);
const record = repo.revokeApiKey(id);
if (record) {
repo.createAuditLog({
teamId: record.teamId,
projectId: record.projectId,
actorType: 'system',
action: 'api_key.revoke',
targetType: 'api_key',
targetId: record.id,
});
}
return record;
}
function hasRequiredScopes(grantedScopes: string[], requiredScopes: string[]): boolean {
if (requiredScopes.length === 0 || grantedScopes.includes('*')) {
return true;
}
return requiredScopes.every(scope => grantedScopes.includes(scope));
}
+24
View File
@@ -0,0 +1,24 @@
// SPDX-License-Identifier: Apache-2.0
import type { Database } from 'bun:sqlite';
import { betterAuth } from 'better-auth';
import { apiKey } from '@better-auth/api-key';
import { organization } from 'better-auth/plugins';
import { DATA_DIR, ensureDir } from '../../shared/paths.js';
export function createAuth(database: Database) {
ensureDir(DATA_DIR);
return betterAuth({
database,
baseURL: process.env.BETTER_AUTH_URL ?? process.env.CLAUDE_MEM_SERVER_URL ?? 'http://127.0.0.1:37777',
basePath: '/api/auth',
plugins: [
apiKey(),
organization({
teams: {
enabled: true,
},
}),
],
});
}
+215
View File
@@ -0,0 +1,215 @@
// SPDX-License-Identifier: Apache-2.0
import {
Queue,
Worker,
type Job,
type JobsOptions,
type Processor,
type QueueOptions,
type WorkerOptions
} from 'bullmq';
import { logger } from '../../utils/logger.js';
import type { RedisQueueConfig } from '../queue/redis-config.js';
// BullMQ Worker docs: https://docs.bullmq.io/guide/workers
// BullMQ Concurrency: https://docs.bullmq.io/guide/workers/concurrency
// BullMQ Stalled Jobs: https://docs.bullmq.io/guide/jobs/stalled
//
// ServerJobQueue is a thin wrapper around the BullMQ Queue + Worker pair for
// one named queue. It enforces:
// - autorun: false on every Worker (start() is called explicitly)
// - default concurrency: 1 (per-kind concurrency tuning happens later)
// - an attached `error` listener on every Worker (BullMQ docs require this
// to avoid unhandled-error crashes when a job throws)
// Postgres outbox is canonical history; BullMQ is the execution transport
// only. Do not treat completed/failed Worker state as authoritative.
export interface ServerJobCounts {
waiting: number;
active: number;
delayed: number;
failed: number;
completed: number;
}
export interface ServerJobQueueOptions<TPayload> {
name: string;
config: RedisQueueConfig;
concurrency?: number;
lockDurationMs?: number;
defaultJobOptions?: JobsOptions;
// Test seams: allow injecting fakes without touching Redis.
queueFactory?: (name: string, options: QueueOptions) => Pick<
Queue<TPayload>,
'add' | 'getJob' | 'getJobCounts' | 'remove' | 'close'
>;
workerFactory?: (
name: string,
processor: Processor<TPayload> | null,
options: WorkerOptions
) => Pick<Worker<TPayload>, 'on' | 'run' | 'close'>;
}
const DEFAULT_LOCK_DURATION_MS = 5 * 60 * 1000;
export class ServerJobQueue<TPayload extends object = object> {
readonly name: string;
private readonly config: RedisQueueConfig;
private readonly concurrency: number;
private readonly lockDurationMs: number;
private readonly defaultJobOptions: JobsOptions;
private readonly queueFactory?: ServerJobQueueOptions<TPayload>['queueFactory'];
private readonly workerFactory?: ServerJobQueueOptions<TPayload>['workerFactory'];
private queue: ReturnType<NonNullable<ServerJobQueueOptions<TPayload>['queueFactory']>> | Queue<TPayload> | null = null;
private worker: ReturnType<NonNullable<ServerJobQueueOptions<TPayload>['workerFactory']>> | Worker<TPayload> | null = null;
private started = false;
constructor(options: ServerJobQueueOptions<TPayload>) {
this.name = options.name;
this.config = options.config;
this.concurrency = options.concurrency ?? 1;
this.lockDurationMs = options.lockDurationMs ?? DEFAULT_LOCK_DURATION_MS;
this.defaultJobOptions = options.defaultJobOptions ?? {
attempts: 3,
backoff: { type: 'exponential', delay: 5000 },
removeOnComplete: { age: 7 * 24 * 60 * 60, count: 1000 },
removeOnFail: { age: 30 * 24 * 60 * 60, count: 1000 }
};
this.queueFactory = options.queueFactory;
this.workerFactory = options.workerFactory;
}
private getQueue(): NonNullable<typeof this.queue> {
if (this.queue) {
return this.queue;
}
const queueOptions: QueueOptions = {
connection: this.config.connection,
prefix: this.config.prefix,
defaultJobOptions: this.defaultJobOptions
};
this.queue = this.queueFactory
? this.queueFactory(this.name, queueOptions)
: new Queue<TPayload>(this.name, queueOptions);
return this.queue;
}
async add(jobId: string, payload: TPayload, options?: JobsOptions): Promise<void> {
if (jobId.includes(':')) {
throw new Error(`server job ID must not contain ':' (got ${jobId})`);
}
try {
await (this.getQueue().add as (
name: string,
data: TPayload,
opts?: JobsOptions
) => Promise<unknown>)(this.name, payload, {
...this.defaultJobOptions,
...options,
jobId
});
} catch (error) {
throw this.toRedisUnavailableError(error);
}
}
async getJob(jobId: string): Promise<Job<TPayload> | null | undefined> {
try {
return (await this.getQueue().getJob(jobId)) as Job<TPayload> | null | undefined;
} catch (error) {
throw this.toRedisUnavailableError(error);
}
}
async remove(jobId: string): Promise<void> {
try {
await this.getQueue().remove(jobId);
} catch (error) {
throw this.toRedisUnavailableError(error);
}
}
async getCounts(): Promise<ServerJobCounts> {
try {
const counts = await this.getQueue().getJobCounts(
'waiting',
'active',
'delayed',
'failed',
'completed'
);
return {
waiting: counts.waiting ?? 0,
active: counts.active ?? 0,
delayed: counts.delayed ?? 0,
failed: counts.failed ?? 0,
completed: counts.completed ?? 0
};
} catch (error) {
throw this.toRedisUnavailableError(error);
}
}
// BullMQ docs require `worker.on('error', ...)` to avoid unhandled rejections
// when a job throws. We construct the Worker with autorun: false so the
// caller controls startup explicitly via run().
start(processor: Processor<TPayload>): void {
if (this.started) {
throw new Error(`ServerJobQueue ${this.name} is already started`);
}
const workerOptions: WorkerOptions = {
connection: this.config.connection,
prefix: this.config.prefix,
autorun: false,
concurrency: this.concurrency,
lockDuration: this.lockDurationMs
};
const worker = this.workerFactory
? this.workerFactory(this.name, processor, workerOptions)
: new Worker<TPayload>(this.name, processor, workerOptions);
worker.on('error', (error: unknown) => {
logger.warn('QUEUE', `${this.name} worker error`, {
error: error instanceof Error ? error.message : String(error)
});
});
worker.run();
this.worker = worker;
this.started = true;
}
isStarted(): boolean {
return this.started;
}
async close(): Promise<void> {
const errors: Error[] = [];
if (this.worker) {
try {
await this.worker.close();
} catch (error) {
errors.push(error instanceof Error ? error : new Error(String(error)));
}
this.worker = null;
this.started = false;
}
if (this.queue) {
try {
await this.queue.close();
} catch (error) {
errors.push(error instanceof Error ? error : new Error(String(error)));
}
this.queue = null;
}
if (errors.length > 0) {
throw errors[0];
}
}
private toRedisUnavailableError(error: unknown): Error {
const message = error instanceof Error ? error.message : String(error);
return new Error(
`ServerJobQueue ${this.name} requires Redis/Valkey when CLAUDE_MEM_QUEUE_ENGINE=bullmq: ${message}`
);
}
}
+30
View File
@@ -0,0 +1,30 @@
// SPDX-License-Identifier: Apache-2.0
import { createHash } from 'crypto';
import { SERVER_JOB_KIND_PREFIX, type ServerGenerationJobKind } from './types.js';
export interface ServerJobIdParts {
kind: ServerGenerationJobKind;
team_id: string;
project_id: string;
source_type: string;
source_id: string;
}
// SHA-256-derived deterministic IDs avoid Redis key collisions across tenants
// and keep BullMQ jobId deduplication intact across process restarts.
// Format: `${kindPrefix}_${sha256hex}` with NO ':' characters (BullMQ uses ':'
// internally as a key separator; embedding ':' in jobIds causes scan/state
// confusion).
export function buildServerJobId(parts: ServerJobIdParts): string {
const prefix = SERVER_JOB_KIND_PREFIX[parts.kind];
const canonical = JSON.stringify({
kind: parts.kind,
team_id: parts.team_id,
project_id: parts.project_id,
source_type: parts.source_type,
source_id: parts.source_id
});
const digest = createHash('sha256').update(canonical).digest('hex');
return `${prefix}_${digest}`;
}
+295
View File
@@ -0,0 +1,295 @@
// SPDX-License-Identifier: Apache-2.0
import type {
PostgresObservationGenerationJob,
PostgresObservationGenerationJobEventsRepository,
PostgresObservationGenerationJobRepository
} from '../../storage/postgres/generation-jobs.js';
import type { JsonObject } from '../../storage/postgres/utils.js';
import { logger } from '../../utils/logger.js';
import { buildServerJobId } from './job-id.js';
import type { ServerJobQueue } from './ServerJobQueue.js';
import type {
GenerateObservationsForEventJob,
GenerateSessionSummaryJob,
ReindexObservationJob,
ServerGenerationJobKind
} from './types.js';
// Postgres outbox is canonical history; BullMQ is the execution transport.
// Each outbox row corresponds to one observation_generation_jobs row, keyed
// by a deterministic BullMQ jobId so duplicate enqueues collapse on the
// transport side and dedup is enforced again by the row's idempotency_key.
export type SingleSourceJobPayload =
| GenerateObservationsForEventJob
| GenerateSessionSummaryJob
| ReindexObservationJob;
const KIND_TO_JOB_TYPE: Record<SingleSourceJobPayload['kind'], string> = {
event: 'observation_generate_for_event',
summary: 'observation_generate_session_summary',
reindex: 'observation_reindex'
};
export interface OutboxScope {
projectId: string;
teamId: string;
}
export interface EnqueueOutboxRowInput {
payload: SingleSourceJobPayload;
agentEventId?: string | null;
serverSessionId?: string | null;
maxAttempts?: number;
}
// `enqueueOutbox` writes the canonical row first, then publishes to BullMQ.
// If the BullMQ add() throws (for example Redis is unavailable), the row is
// transitioned to `failed` so the next reconciliation pass can resurrect it
// rather than leaving stale `queued` rows that never enter the transport.
export async function enqueueOutbox(
jobRepo: PostgresObservationGenerationJobRepository,
eventsRepo: PostgresObservationGenerationJobEventsRepository,
queue: ServerJobQueue<SingleSourceJobPayload>,
input: EnqueueOutboxRowInput
): Promise<{ row: PostgresObservationGenerationJob; bullmqJobId: string }> {
const { payload } = input;
const bullmqJobId = buildServerJobId({
kind: payload.kind,
team_id: payload.team_id,
project_id: payload.project_id,
source_type: payload.source_type,
source_id: payload.source_id
});
const row = await jobRepo.create({
projectId: payload.project_id,
teamId: payload.team_id,
sourceType: payload.source_type,
sourceId: payload.source_id,
agentEventId: input.agentEventId ?? extractAgentEventId(payload),
serverSessionId: input.serverSessionId ?? extractServerSessionId(payload),
jobType: KIND_TO_JOB_TYPE[payload.kind],
bullmqJobId,
maxAttempts: input.maxAttempts,
payload: payload as unknown as JsonObject
});
await eventsRepo.append({
generationJobId: row.id,
projectId: row.projectId,
teamId: row.teamId,
eventType: 'queued',
statusAfter: row.status,
attempt: row.attempts
});
try {
await queue.add(bullmqJobId, payload);
await eventsRepo.append({
generationJobId: row.id,
projectId: row.projectId,
teamId: row.teamId,
eventType: 'enqueued',
statusAfter: row.status,
attempt: row.attempts
});
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
logger.warn('QUEUE', `failed to publish to BullMQ for job ${row.id}: ${message}`);
await jobRepo.transitionStatus({
id: row.id,
projectId: row.projectId,
teamId: row.teamId,
status: 'failed',
lastError: { message, source: 'bullmq_publish' }
});
await eventsRepo.append({
generationJobId: row.id,
projectId: row.projectId,
teamId: row.teamId,
eventType: 'failed',
statusAfter: 'failed',
attempt: row.attempts,
details: { source: 'bullmq_publish', message }
});
throw error;
}
return { row, bullmqJobId };
}
// `reconcileOnStartup` re-enqueues outbox rows that were left in `queued` or
// `processing` after a crash or restart. For each row we replace any
// terminal BullMQ job that may still be holding the deterministic ID slot
// (BullMQ refuses to re-add a jobId that already exists in `completed` or
// `failed` lists). Reconciliation is a no-op for rows past max_attempts.
export async function reconcileOnStartup(
jobRepo: PostgresObservationGenerationJobRepository,
eventsRepo: PostgresObservationGenerationJobEventsRepository,
queue: ServerJobQueue<SingleSourceJobPayload>,
scope: OutboxScope,
options?: { limit?: number }
): Promise<{ requeued: number; skipped: number }> {
const limit = options?.limit ?? 500;
const queued = await jobRepo.listByStatusForScope({
status: 'queued',
projectId: scope.projectId,
teamId: scope.teamId,
limit
});
const processing = await jobRepo.listByStatusForScope({
status: 'processing',
projectId: scope.projectId,
teamId: scope.teamId,
limit
});
let requeued = 0;
let skipped = 0;
for (const row of [...processing, ...queued]) {
if (row.attempts >= row.maxAttempts) {
skipped += 1;
continue;
}
const bullmqJobId = row.bullmqJobId ?? buildServerJobId(extractIdParts(row));
try {
await queue.remove(bullmqJobId);
} catch (error) {
logger.debug?.('QUEUE', `remove before re-add ignored for ${bullmqJobId}`, {
error: error instanceof Error ? error.message : String(error)
});
}
if (row.status === 'processing') {
await jobRepo.transitionStatus({
id: row.id,
projectId: row.projectId,
teamId: row.teamId,
status: 'queued'
});
await eventsRepo.append({
generationJobId: row.id,
projectId: row.projectId,
teamId: row.teamId,
eventType: 'queued',
statusAfter: 'queued',
attempt: row.attempts,
details: { source: 'reconcile_on_startup' }
});
}
await queue.add(bullmqJobId, row.payload as unknown as SingleSourceJobPayload);
await eventsRepo.append({
generationJobId: row.id,
projectId: row.projectId,
teamId: row.teamId,
eventType: 'enqueued',
statusAfter: 'queued',
attempt: row.attempts,
details: { source: 'reconcile_on_startup' }
});
requeued += 1;
}
return { requeued, skipped };
}
export async function markCompleted(
jobRepo: PostgresObservationGenerationJobRepository,
eventsRepo: PostgresObservationGenerationJobEventsRepository,
input: { id: string; projectId: string; teamId: string; details?: JsonObject }
): Promise<void> {
const updated = await jobRepo.transitionStatus({
id: input.id,
projectId: input.projectId,
teamId: input.teamId,
status: 'completed'
});
if (!updated) {
throw new Error(`generation job ${input.id} not found for scope`);
}
await eventsRepo.append({
generationJobId: updated.id,
projectId: updated.projectId,
teamId: updated.teamId,
eventType: 'completed',
statusAfter: 'completed',
attempt: updated.attempts,
details: input.details ?? {}
});
}
export async function markFailed(
jobRepo: PostgresObservationGenerationJobRepository,
eventsRepo: PostgresObservationGenerationJobEventsRepository,
input: {
id: string;
projectId: string;
teamId: string;
error: { message: string; source?: string };
nextAttemptAt?: Date | null;
}
): Promise<void> {
const status = input.nextAttemptAt ? 'queued' : 'failed';
const updated = await jobRepo.transitionStatus({
id: input.id,
projectId: input.projectId,
teamId: input.teamId,
status,
nextAttemptAt: input.nextAttemptAt ?? null,
lastError: { message: input.error.message, source: input.error.source ?? 'processor' }
});
if (!updated) {
throw new Error(`generation job ${input.id} not found for scope`);
}
await eventsRepo.append({
generationJobId: updated.id,
projectId: updated.projectId,
teamId: updated.teamId,
eventType: status === 'queued' ? 'retry_scheduled' : 'failed',
statusAfter: status,
attempt: updated.attempts,
details: { message: input.error.message, source: input.error.source ?? 'processor' }
});
}
function extractAgentEventId(payload: SingleSourceJobPayload): string | null {
return payload.kind === 'event' ? payload.agent_event_id : null;
}
function extractServerSessionId(payload: SingleSourceJobPayload): string | null {
return payload.kind === 'summary' ? payload.server_session_id : null;
}
function extractIdParts(row: PostgresObservationGenerationJob): {
kind: ServerGenerationJobKind;
team_id: string;
project_id: string;
source_type: string;
source_id: string;
} {
const kind = jobTypeToKind(row.jobType);
return {
kind,
team_id: row.teamId,
project_id: row.projectId,
source_type: row.sourceType,
source_id: row.sourceId
};
}
function jobTypeToKind(jobType: string): ServerGenerationJobKind {
for (const [kind, type] of Object.entries(KIND_TO_JOB_TYPE) as Array<
[SingleSourceJobPayload['kind'], string]
>) {
if (type === jobType) {
return kind;
}
}
throw new Error(`unknown observation generation job_type: ${jobType}`);
}
+59
View File
@@ -0,0 +1,59 @@
// SPDX-License-Identifier: Apache-2.0
import type {
ObservationGenerationJobSourceType,
ObservationGenerationJobStatus
} from '../../storage/postgres/generation-jobs.js';
export type ServerGenerationJobKind = 'event' | 'event-batch' | 'summary' | 'reindex';
export type ServerGenerationJobStatus = ObservationGenerationJobStatus;
export interface ServerGenerationJob {
kind: ServerGenerationJobKind;
team_id: string;
project_id: string;
source_type: ObservationGenerationJobSourceType;
source_id: string;
generation_job_id: string;
}
export interface GenerateObservationsForEventJob extends ServerGenerationJob {
kind: 'event';
agent_event_id: string;
}
export interface GenerateObservationsForEventBatchJob extends ServerGenerationJob {
kind: 'event-batch';
agent_event_ids: string[];
}
export interface GenerateSessionSummaryJob extends ServerGenerationJob {
kind: 'summary';
server_session_id: string;
}
export interface ReindexObservationJob extends ServerGenerationJob {
kind: 'reindex';
observation_id: string;
}
export type ServerGenerationJobPayload =
| GenerateObservationsForEventJob
| GenerateObservationsForEventBatchJob
| GenerateSessionSummaryJob
| ReindexObservationJob;
export const SERVER_JOB_QUEUE_NAMES: Record<ServerGenerationJobKind, string> = {
event: 'server_beta_generate_event',
'event-batch': 'server_beta_generate_event_batch',
summary: 'server_beta_generate_summary',
reindex: 'server_beta_reindex'
};
export const SERVER_JOB_KIND_PREFIX: Record<ServerGenerationJobKind, string> = {
event: 'evt',
'event-batch': 'evtb',
summary: 'sum',
reindex: 'rdx'
};
+12
View File
@@ -0,0 +1,12 @@
// SPDX-License-Identifier: Apache-2.0
export const serverMemoryPrompts = [
{
name: 'record_decision',
description: 'Capture a project decision in Claude-Mem Server memory.',
arguments: [
{ name: 'projectId', description: 'Server project id', required: true },
{ name: 'decision', description: 'Decision text', required: true },
],
},
] as const;
+13
View File
@@ -0,0 +1,13 @@
// SPDX-License-Identifier: Apache-2.0
import { serverMemoryPrompts } from './prompts.js';
import { serverMemoryResources } from './resources.js';
import { serverMemoryTools } from './tools.js';
export function getServerMcpSurface() {
return {
tools: serverMemoryTools,
resources: serverMemoryResources,
prompts: serverMemoryPrompts,
};
}
+16
View File
@@ -0,0 +1,16 @@
// SPDX-License-Identifier: Apache-2.0
export const serverMemoryResources = [
{
uri: 'claude-mem://server/projects',
name: 'Claude-Mem Server Projects',
description: 'Authorized project list exposed by Claude-Mem Server.',
mimeType: 'application/json',
},
{
uri: 'claude-mem://server/memories/recent',
name: 'Recent Claude-Mem Server Memories',
description: 'Recent authorized memory items from the server core.',
mimeType: 'application/json',
},
] as const;
+96
View File
@@ -0,0 +1,96 @@
// SPDX-License-Identifier: Apache-2.0
export interface ServerMcpToolDefinition {
name: string;
description: string;
inputSchema: {
type: 'object';
properties: Record<string, unknown>;
required?: string[];
};
}
export const serverMemoryTools: ServerMcpToolDefinition[] = [
{
name: 'memory_add',
description: 'Add a team-scoped memory item to Claude-Mem Server.',
inputSchema: {
type: 'object',
properties: {
projectId: { type: 'string' },
kind: { type: 'string', enum: ['observation', 'summary', 'prompt', 'manual'] },
type: { type: 'string' },
title: { type: 'string' },
narrative: { type: 'string' },
facts: { type: 'array', items: { type: 'string' } },
},
required: ['projectId', 'kind', 'type'],
},
},
{
name: 'memory_search',
description: 'Search server memory items within the authorized project/team scope.',
inputSchema: {
type: 'object',
properties: {
projectId: { type: 'string' },
query: { type: 'string' },
limit: { type: 'number', minimum: 1, maximum: 100 },
},
required: ['projectId', 'query'],
},
},
{
name: 'memory_context',
description: 'Build a compact context pack from matching server memories.',
inputSchema: {
type: 'object',
properties: {
projectId: { type: 'string' },
query: { type: 'string' },
limit: { type: 'number', minimum: 1, maximum: 50 },
},
required: ['projectId', 'query'],
},
},
{
name: 'memory_forget',
description: 'Forget or tombstone a memory item in the authorized project/team scope.',
inputSchema: {
type: 'object',
properties: {
projectId: { type: 'string' },
memoryId: { type: 'string' },
reason: { type: 'string' },
},
required: ['projectId', 'memoryId'],
},
},
{
name: 'memory_list_recent',
description: 'List recent server memories for an authorized project.',
inputSchema: {
type: 'object',
properties: {
projectId: { type: 'string' },
limit: { type: 'number', minimum: 1, maximum: 100 },
},
required: ['projectId'],
},
},
{
name: 'memory_record_decision',
description: 'Record an architectural or product decision as a server memory.',
inputSchema: {
type: 'object',
properties: {
projectId: { type: 'string' },
title: { type: 'string' },
decision: { type: 'string' },
rationale: { type: 'string' },
consequences: { type: 'array', items: { type: 'string' } },
},
required: ['projectId', 'title', 'decision'],
},
},
];
+125
View File
@@ -0,0 +1,125 @@
// SPDX-License-Identifier: Apache-2.0
import type { Database } from 'bun:sqlite';
import type { NextFunction, Request, RequestHandler, Response } from 'express';
import { verifyServerApiKey } from '../auth/api-key-service.js';
export interface AuthContext {
userId: string | null;
organizationId: string | null;
teamId: string | null;
projectId: string | null;
scopes: string[];
apiKeyId: string | null;
mode: 'api-key' | 'local-dev';
}
declare module 'express-serve-static-core' {
interface Request {
authContext?: AuthContext;
}
}
export interface RequireAuthOptions {
requiredScopes?: string[];
authMode?: string;
allowLocalDevBypass?: boolean;
}
export function requireServerAuth(
getDatabase: () => Database,
options: RequireAuthOptions = {},
): RequestHandler {
return (req: Request, res: Response, next: NextFunction) => {
const authMode = options.authMode ?? process.env.CLAUDE_MEM_AUTH_MODE ?? 'api-key';
const authorization = req.header('authorization') ?? '';
const rawKey = parseBearerToken(authorization);
const allowLocalDevBypass = options.allowLocalDevBypass ?? process.env.CLAUDE_MEM_ALLOW_LOCAL_DEV_BYPASS === '1';
if (
!rawKey
&& authMode === 'local-dev'
&& allowLocalDevBypass
&& isLocalhost(req)
&& hasLoopbackHostHeader(req)
&& !hasForwardedClientHeaders(req)
) {
req.authContext = {
userId: null,
organizationId: null,
teamId: null,
projectId: null,
scopes: ['local-dev'],
apiKeyId: null,
mode: 'local-dev',
};
next();
return;
}
if (!rawKey) {
res.status(401).json({ error: 'Unauthorized', message: 'Missing bearer API key' });
return;
}
const verified = verifyServerApiKey(getDatabase(), rawKey, options.requiredScopes ?? []);
if (!verified) {
res.status(403).json({ error: 'Forbidden', message: 'Invalid API key or insufficient scope' });
return;
}
req.authContext = {
userId: null,
organizationId: null,
teamId: verified.teamId,
projectId: verified.projectId,
scopes: verified.scopes,
apiKeyId: verified.record.id,
mode: 'api-key',
};
next();
};
}
function parseBearerToken(header: string): string | null {
const match = /^Bearer\s+(.+)$/i.exec(header.trim());
return match?.[1]?.trim() || null;
}
function isLocalhost(req: Request): boolean {
const clientIp = req.ip || req.socket.remoteAddress || '';
return clientIp === '127.0.0.1'
|| clientIp === '::1'
|| clientIp === '::ffff:127.0.0.1'
|| clientIp === 'localhost';
}
function hasLoopbackHostHeader(req: Request): boolean {
const host = parseHostWithoutPort(req.header('host') ?? '');
return host === '127.0.0.1'
|| host === 'localhost'
|| host === '::1';
}
function parseHostWithoutPort(rawHost: string): string {
const host = rawHost.trim().toLowerCase();
if (host.startsWith('[')) {
const closeBracketIndex = host.indexOf(']');
return closeBracketIndex === -1 ? host : host.slice(1, closeBracketIndex);
}
const lastColonIndex = host.lastIndexOf(':');
if (lastColonIndex > -1 && /^\d+$/.test(host.slice(lastColonIndex + 1))) {
return host.slice(0, lastColonIndex);
}
return host;
}
function hasForwardedClientHeaders(req: Request): boolean {
return Boolean(
req.header('forwarded')
|| req.header('x-forwarded-for')
|| req.header('x-forwarded-host')
|| req.header('x-real-ip')
);
}
@@ -0,0 +1,547 @@
// SPDX-License-Identifier: Apache-2.0
import { createHash } from 'crypto';
import { EventEmitter } from 'events';
import { Queue, Worker, type Job, type JobType, type QueueOptions, type WorkerOptions } from 'bullmq';
import { Redis } from 'ioredis';
import type { PendingMessage, PendingMessageWithId } from '../../services/worker-types.js';
import type { CreateIteratorOptions } from '../../services/queue/SessionQueueProcessor.js';
import { logger } from '../../utils/logger.js';
import type {
HealthCheckedObservationQueueEngine,
ObservationQueueHealth,
ObservationQueueInspection,
} from './ObservationQueueEngine.js';
import { getRedisQueueConfig, type RedisQueueConfig } from './redis-config.js';
interface BullMqPendingPayload {
sessionDbId: number;
contentSessionId: string;
createdAtEpoch: number;
message: PendingMessage;
}
type BullMqJob = Pick<
Job<BullMqPendingPayload>,
'id' | 'data' | 'moveToCompleted' | 'moveToWait' | 'extendLock' | 'getState'
| 'remove'
>;
type BullMqQueue = Pick<
Queue<BullMqPendingPayload>,
'add' | 'getJob' | 'getJobCounts' | 'getJobs' | 'obliterate' | 'close'
>;
type BullMqWorker = Pick<Worker<BullMqPendingPayload>, 'getNextJob' | 'close'>;
interface RedisHealthClient {
status: string;
connect(): Promise<void>;
ping(): Promise<unknown>;
sadd(key: string, ...members: string[]): Promise<number>;
srem(key: string, ...members: string[]): Promise<number>;
smembers(key: string): Promise<string[]>;
quit(): Promise<unknown>;
disconnect(): void;
}
export interface BullMqObservationQueueEngineOptions {
config?: RedisQueueConfig;
queueFactory?: (name: string, options: QueueOptions) => BullMqQueue;
workerFactory?: (name: string, options: WorkerOptions) => BullMqWorker;
redisFactory?: (config: RedisQueueConfig) => RedisHealthClient;
onMutate?: () => void;
lockDurationMs?: number;
pollIntervalMs?: number;
}
interface SessionRuntime {
queue: BullMqQueue;
worker: BullMqWorker;
events: EventEmitter;
}
interface ClaimedJob {
sessionDbId: number;
job: BullMqJob;
token: string;
lockTimer: ReturnType<typeof setInterval> | null;
}
const QUEUE_JOB_TYPES: JobType[] = ['waiting', 'active', 'delayed', 'prioritized', 'waiting-children'];
const DEFAULT_LOCK_DURATION_MS = 5 * 60 * 1000;
const DEFAULT_POLL_INTERVAL_MS = 250;
export class BullMqObservationQueueEngine
implements HealthCheckedObservationQueueEngine, ObservationQueueInspection {
private readonly config: RedisQueueConfig;
private readonly sessions = new Map<number, SessionRuntime>();
private readonly activeClaims = new Map<number, ClaimedJob>();
private readonly lockDurationMs: number;
private readonly pollIntervalMs: number;
private readonly registryKey: string;
private nextClaimId = 1;
private nextEnqueueId = 1;
private healthClient: RedisHealthClient | null = null;
constructor(private readonly options: BullMqObservationQueueEngineOptions = {}) {
this.config = options.config ?? getRedisQueueConfig();
this.lockDurationMs = options.lockDurationMs ?? DEFAULT_LOCK_DURATION_MS;
this.pollIntervalMs = options.pollIntervalMs ?? DEFAULT_POLL_INTERVAL_MS;
this.registryKey = `${this.config.prefix}:queue_registry:sessions`;
}
async enqueue(sessionDbId: number, contentSessionId: string, message: PendingMessage): Promise<number> {
const runtime = this.getSessionRuntime(sessionDbId);
await this.registerSession(sessionDbId);
const createdAtEpoch = Date.now();
const payload: BullMqPendingPayload = {
sessionDbId,
contentSessionId,
createdAtEpoch,
message,
};
const jobId = getSafeJobId(contentSessionId, message, createdAtEpoch);
const existing = await runtime.queue.getJob(jobId);
if (existing && !await this.isTerminal(existing)) {
return 0;
}
if (existing) {
try {
await existing.remove();
} catch (error) {
throw this.toRedisUnavailableError(error);
}
}
try {
await runtime.queue.add(message.type, payload, {
jobId,
attempts: 1000000,
removeOnComplete: true,
removeOnFail: { age: 24 * 60 * 60, count: 1000 },
});
} catch (error) {
throw this.toRedisUnavailableError(error);
}
runtime.events.emit('message');
this.options.onMutate?.();
return this.nextEnqueueId++;
}
async *createIterator(options: CreateIteratorOptions): AsyncIterableIterator<PendingMessageWithId> {
const {
sessionDbId,
signal,
onIdleTimeout,
idleTimeoutMs = 3 * 60 * 1000,
} = options;
const runtime = this.getSessionRuntime(sessionDbId);
let lastActivityTime = Date.now();
while (!signal.aborted) {
const token = this.createToken(sessionDbId);
let job: BullMqJob | undefined;
try {
job = await runtime.worker.getNextJob(token, { block: false }) as BullMqJob | undefined;
} catch (error) {
throw this.toRedisUnavailableError(error);
}
if (job) {
const claimId = this.nextClaimId++;
this.activeClaims.set(claimId, {
sessionDbId,
job,
token,
lockTimer: this.startLockRenewal(job, token),
});
lastActivityTime = Date.now();
this.options.onMutate?.();
yield {
...job.data.message,
_persistentId: claimId,
_originalTimestamp: job.data.createdAtEpoch,
};
continue;
}
const received = await this.waitForMessage(runtime.events, signal, this.pollIntervalMs);
if (received) {
continue;
}
if (Date.now() - lastActivityTime >= idleTimeoutMs && !signal.aborted) {
onIdleTimeout?.();
return;
}
}
}
async confirmProcessed(messageId: number): Promise<number> {
const claimed = this.activeClaims.get(messageId);
if (!claimed) {
return 0;
}
try {
await claimed.job.moveToCompleted({ ok: true }, claimed.token, false);
} catch (error) {
throw this.toRedisUnavailableError(error);
}
this.finishClaim(messageId, claimed);
await this.unregisterSessionIfEmpty(claimed.sessionDbId);
this.options.onMutate?.();
return 1;
}
async clearPendingForSession(sessionDbId: number): Promise<number> {
const runtime = this.getSessionRuntime(sessionDbId);
const count = await this.getPendingCount(sessionDbId);
try {
await runtime.queue.obliterate({ force: true });
} catch (error) {
throw this.toRedisUnavailableError(error);
}
for (const [claimId, claimed] of Array.from(this.activeClaims.entries())) {
if (claimed.sessionDbId === sessionDbId) {
this.finishClaim(claimId, claimed);
}
}
await this.unregisterSessionIfEmpty(sessionDbId);
if (count > 0) {
runtime.events.emit('message');
this.options.onMutate?.();
}
return count;
}
async resetProcessingToPending(sessionDbId: number): Promise<number> {
let reset = 0;
let resetError: Error | null = null;
for (const [claimId, claimed] of Array.from(this.activeClaims.entries())) {
if (claimed.sessionDbId !== sessionDbId) {
continue;
}
try {
await claimed.job.moveToWait(claimed.token);
} catch (error) {
const normalized = this.toRedisUnavailableError(error);
resetError ??= normalized;
logger.warn('QUEUE', 'BullMQ active claim reset failed', {
sessionDbId,
jobId: claimed.job.id,
error: normalized.message,
});
continue;
}
this.finishClaim(claimId, claimed);
reset++;
}
if (reset > 0) {
this.getSessionRuntime(sessionDbId).events.emit('message');
this.options.onMutate?.();
}
if (resetError) {
throw resetError;
}
return reset;
}
async getPendingCount(sessionDbId: number): Promise<number> {
const counts = await this.getSessionRuntime(sessionDbId).queue.getJobCounts(...QUEUE_JOB_TYPES);
return sumCounts(counts);
}
async getTotalQueueDepth(): Promise<number> {
let total = 0;
const sessionIds = new Set<number>(this.sessions.keys());
for (const sessionDbId of await this.getRegisteredSessionIds()) {
sessionIds.add(sessionDbId);
}
for (const sessionDbId of sessionIds) {
total += await this.getPendingCount(sessionDbId);
}
return total;
}
async peekPendingTypes(sessionDbId: number): Promise<Array<{ message_type: string; tool_name: string | null }>> {
const jobs = await this.getSessionRuntime(sessionDbId).queue.getJobs(QUEUE_JOB_TYPES, 0, -1, true);
return jobs.map(job => ({
message_type: job.data.message.type,
tool_name: job.data.message.tool_name ?? null,
}));
}
async getHealth(): Promise<ObservationQueueHealth> {
try {
const client = this.getHealthClient();
if (client.status === 'wait' || client.status === 'end') {
await client.connect();
}
await client.ping();
return {
engine: 'bullmq',
redis: {
status: 'ok',
mode: this.config.mode,
host: this.config.host,
port: this.config.port,
prefix: this.config.prefix,
},
};
} catch (error) {
return {
engine: 'bullmq',
redis: {
status: 'error',
mode: this.config.mode,
host: this.config.host,
port: this.config.port,
prefix: this.config.prefix,
error: error instanceof Error ? error.message : String(error),
},
};
}
}
async assertHealthy(): Promise<void> {
const health = await this.getHealth();
if (health.redis.status !== 'ok') {
throw new Error(
`CLAUDE_MEM_QUEUE_ENGINE=bullmq requires Redis/Valkey at ${health.redis.host}:${health.redis.port}; ${health.redis.error ?? 'ping failed'}`
);
}
}
async close(): Promise<void> {
let releaseError: Error | null = null;
try {
await this.releaseActiveClaimsToWait();
} catch (error) {
releaseError = error instanceof Error ? error : new Error(String(error));
} finally {
for (const [claimId, claimed] of Array.from(this.activeClaims.entries())) {
this.finishClaim(claimId, claimed);
}
for (const runtime of this.sessions.values()) {
runtime.events.removeAllListeners();
await runtime.worker.close().catch(error => {
logger.warn('QUEUE', 'BullMQ worker close failed', {
error: error instanceof Error ? error.message : String(error),
});
});
await runtime.queue.close().catch(error => {
logger.warn('QUEUE', 'BullMQ queue close failed', {
error: error instanceof Error ? error.message : String(error),
});
});
}
this.sessions.clear();
if (this.healthClient) {
await this.healthClient.quit().catch(() => this.healthClient?.disconnect());
this.healthClient = null;
}
}
if (releaseError) {
throw releaseError;
}
}
private getSessionRuntime(sessionDbId: number): SessionRuntime {
const existing = this.sessions.get(sessionDbId);
if (existing) {
return existing;
}
const name = `claude_mem_session_${sessionDbId}`;
const queueOptions: QueueOptions = {
connection: this.config.connection,
prefix: this.config.prefix,
};
const workerOptions: WorkerOptions = {
connection: this.config.connection,
prefix: this.config.prefix,
autorun: false,
concurrency: 1,
lockDuration: this.lockDurationMs,
};
const runtime: SessionRuntime = {
queue: this.options.queueFactory
? this.options.queueFactory(name, queueOptions)
: new Queue<BullMqPendingPayload>(name, queueOptions),
worker: this.options.workerFactory
? this.options.workerFactory(name, workerOptions)
: new Worker<BullMqPendingPayload>(name, null, workerOptions),
events: new EventEmitter(),
};
this.sessions.set(sessionDbId, runtime);
return runtime;
}
private getHealthClient(): RedisHealthClient {
if (!this.healthClient) {
this.healthClient = this.options.redisFactory
? this.options.redisFactory(this.config)
: new Redis(this.config.connection) as RedisHealthClient;
}
return this.healthClient;
}
private async registerSession(sessionDbId: number): Promise<void> {
try {
await this.getHealthClient().sadd(this.registryKey, String(sessionDbId));
} catch (error) {
throw this.toRedisUnavailableError(error);
}
}
private async unregisterSessionIfEmpty(sessionDbId: number): Promise<void> {
if (await this.getPendingCount(sessionDbId) > 0) {
return;
}
try {
await this.getHealthClient().srem(this.registryKey, String(sessionDbId));
} catch (error) {
throw this.toRedisUnavailableError(error);
}
}
private async getRegisteredSessionIds(): Promise<number[]> {
let rawSessionIds: string[];
try {
rawSessionIds = await this.getHealthClient().smembers(this.registryKey);
} catch (error) {
throw this.toRedisUnavailableError(error);
}
return rawSessionIds
.map(raw => Number.parseInt(raw, 10))
.filter(sessionDbId => Number.isInteger(sessionDbId) && sessionDbId > 0);
}
private async isTerminal(job: BullMqJob): Promise<boolean> {
const state = await job.getState();
return state === 'completed' || state === 'failed' || state === 'unknown';
}
private startLockRenewal(job: BullMqJob, token: string): ReturnType<typeof setInterval> | null {
if (!job.extendLock) {
return null;
}
const interval = setInterval(() => {
job.extendLock(token, this.lockDurationMs).catch(error => {
logger.warn('QUEUE', 'BullMQ job lock renewal failed', {
jobId: job.id,
error: error instanceof Error ? error.message : String(error),
});
});
}, Math.max(1000, Math.floor(this.lockDurationMs / 2)));
return interval;
}
private finishClaim(claimId: number, claimed: ClaimedJob): void {
if (claimed.lockTimer) {
clearInterval(claimed.lockTimer);
}
this.activeClaims.delete(claimId);
}
private async releaseActiveClaimsToWait(): Promise<number> {
let released = 0;
let releaseError: Error | null = null;
for (const [claimId, claimed] of Array.from(this.activeClaims.entries())) {
try {
await claimed.job.moveToWait(claimed.token);
} catch (error) {
const normalized = this.toRedisUnavailableError(error);
releaseError ??= normalized;
logger.warn('QUEUE', 'BullMQ active claim release failed during close', {
sessionDbId: claimed.sessionDbId,
jobId: claimed.job.id,
error: normalized.message,
});
continue;
}
this.finishClaim(claimId, claimed);
released++;
this.sessions.get(claimed.sessionDbId)?.events.emit('message');
}
if (released > 0) {
this.options.onMutate?.();
}
if (releaseError) {
throw releaseError;
}
return released;
}
private waitForMessage(events: EventEmitter, signal: AbortSignal, timeoutMs: number): Promise<boolean> {
return new Promise(resolve => {
let timeout: ReturnType<typeof setTimeout> | undefined;
const cleanup = () => {
if (timeout !== undefined) {
clearTimeout(timeout);
}
events.off('message', onMessage);
signal.removeEventListener('abort', onAbort);
};
const onMessage = () => {
cleanup();
resolve(true);
};
const onAbort = () => {
cleanup();
resolve(false);
};
timeout = setTimeout(() => {
cleanup();
resolve(false);
}, timeoutMs);
events.once('message', onMessage);
signal.addEventListener('abort', onAbort, { once: true });
});
}
private createToken(sessionDbId: number): string {
return `claude-mem-${process.pid}-${sessionDbId}-${Date.now()}-${Math.random().toString(36).slice(2)}`;
}
private toRedisUnavailableError(error: unknown): Error {
const message = error instanceof Error ? error.message : String(error);
return new Error(`BullMQ queue operation failed; Redis/Valkey is required when CLAUDE_MEM_QUEUE_ENGINE=bullmq: ${message}`);
}
}
export function getSafeJobId(contentSessionId: string, message: PendingMessage, createdAtEpoch: number): string {
if (message.type === 'observation') {
if (message.toolUseId) {
return `obs_${sha256(`${contentSessionId}\0${message.toolUseId}`)}`;
}
return `obs_${sha256(`${contentSessionId}\0${createdAtEpoch}\0${stableMessageFingerprint(message)}`)}`;
}
return `sum_${sha256(`${contentSessionId}\0${createdAtEpoch}\0${message.type}`)}`;
}
function stableMessageFingerprint(message: PendingMessage): string {
return JSON.stringify({
type: message.type,
tool_name: message.tool_name ?? null,
tool_input: message.tool_input ?? null,
tool_response: message.tool_response ?? null,
cwd: message.cwd ?? null,
prompt_number: message.prompt_number ?? null,
agentId: message.agentId ?? null,
agentType: message.agentType ?? null,
});
}
function sha256(value: string): string {
return createHash('sha256').update(value).digest('hex');
}
function sumCounts(counts: Record<string, number>): number {
return QUEUE_JOB_TYPES.reduce((sum, type) => sum + (counts[type] ?? 0), 0);
}
+114
View File
@@ -0,0 +1,114 @@
// SPDX-License-Identifier: Apache-2.0
import { EventEmitter } from 'events';
import type { Database } from 'bun:sqlite';
import { SessionQueueProcessor, type CreateIteratorOptions } from '../../services/queue/SessionQueueProcessor.js';
import { PendingMessageStore } from '../../services/sqlite/PendingMessageStore.js';
import type { PendingMessage, PendingMessageWithId } from '../../services/worker-types.js';
export interface ObservationQueueEngine {
enqueue(sessionDbId: number, contentSessionId: string, message: PendingMessage): Promise<number>;
createIterator(options: CreateIteratorOptions): AsyncIterableIterator<PendingMessageWithId>;
confirmProcessed(messageId: number): Promise<number>;
clearPendingForSession(sessionDbId: number): Promise<number>;
resetProcessingToPending(sessionDbId: number): Promise<number>;
getPendingCount(sessionDbId: number): Promise<number>;
getTotalQueueDepth(): Promise<number>;
close(): Promise<void>;
}
export interface ObservationQueueHealth {
engine: 'bullmq';
redis: {
status: 'ok' | 'error';
mode: string;
host: string;
port: number;
prefix: string;
error?: string;
};
}
export interface ObservationQueueInspection {
peekPendingTypes(sessionDbId: number): Promise<Array<{ message_type: string; tool_name: string | null }>>;
}
export type InspectableObservationQueueEngine = ObservationQueueEngine & ObservationQueueInspection;
export type HealthCheckedObservationQueueEngine = InspectableObservationQueueEngine & {
getHealth(): Promise<ObservationQueueHealth>;
assertHealthy(): Promise<void>;
};
export class SqliteObservationQueueEngine implements InspectableObservationQueueEngine {
private readonly store: PendingMessageStore;
private readonly eventsBySession = new Map<number, EventEmitter>();
constructor(db: Database, onMutate?: () => void) {
this.store = new PendingMessageStore(db, onMutate);
}
async enqueue(sessionDbId: number, contentSessionId: string, message: PendingMessage): Promise<number> {
const id = this.store.enqueue(sessionDbId, contentSessionId, message);
if (id > 0) {
this.emit(sessionDbId);
}
return id;
}
createIterator(options: CreateIteratorOptions): AsyncIterableIterator<PendingMessageWithId> {
const processor = new SessionQueueProcessor(this.store, this.getEvents(options.sessionDbId));
return processor.createIterator(options);
}
async confirmProcessed(messageId: number): Promise<number> {
return this.store.confirmProcessed(messageId);
}
async clearPendingForSession(sessionDbId: number): Promise<number> {
const rows = this.store.clearPendingForSession(sessionDbId);
if (rows > 0) {
this.emit(sessionDbId);
}
return rows;
}
async resetProcessingToPending(sessionDbId: number): Promise<number> {
const rows = this.store.resetProcessingToPending(sessionDbId);
if (rows > 0) {
this.emit(sessionDbId);
}
return rows;
}
async getPendingCount(sessionDbId: number): Promise<number> {
return this.store.getPendingCount(sessionDbId);
}
async getTotalQueueDepth(): Promise<number> {
return this.store.getTotalQueueDepth();
}
async peekPendingTypes(sessionDbId: number): Promise<Array<{ message_type: string; tool_name: string | null }>> {
return this.store.peekPendingTypes(sessionDbId);
}
async close(): Promise<void> {
for (const events of this.eventsBySession.values()) {
events.removeAllListeners();
}
this.eventsBySession.clear();
}
private getEvents(sessionDbId: number): EventEmitter {
let events = this.eventsBySession.get(sessionDbId);
if (!events) {
events = new EventEmitter();
this.eventsBySession.set(sessionDbId, events);
}
return events;
}
private emit(sessionDbId: number): void {
this.eventsBySession.get(sessionDbId)?.emit('message');
}
}
+120
View File
@@ -0,0 +1,120 @@
// SPDX-License-Identifier: Apache-2.0
import type { RedisOptions } from 'ioredis';
import { existsSync } from 'fs';
import { SettingsDefaultsManager } from '../../shared/SettingsDefaultsManager.js';
import type { SettingsDefaults } from '../../shared/SettingsDefaultsManager.js';
import { USER_SETTINGS_PATH } from '../../shared/paths.js';
export type ObservationQueueEngineName = 'sqlite' | 'bullmq';
export type RedisMode = 'external' | 'managed' | 'docker';
export interface RedisQueueConfig {
engine: ObservationQueueEngineName;
mode: RedisMode;
url: string | null;
host: string;
port: number;
prefix: string;
connection: RedisOptions;
}
export function getObservationQueueEngineName(): ObservationQueueEngineName {
const raw = getQueueSetting('CLAUDE_MEM_QUEUE_ENGINE').trim().toLowerCase();
if (raw === 'sqlite' || raw === 'bullmq') {
return raw;
}
throw new Error(`Invalid CLAUDE_MEM_QUEUE_ENGINE=${raw}; expected sqlite or bullmq`);
}
export function getRedisQueueConfig(): RedisQueueConfig {
const engine = getObservationQueueEngineName();
const mode = normalizeRedisMode(getQueueSetting('CLAUDE_MEM_REDIS_MODE'));
const url = getQueueSetting('CLAUDE_MEM_REDIS_URL').trim() || null;
const host = getQueueSetting('CLAUDE_MEM_REDIS_HOST').trim() || '127.0.0.1';
const port = parseRedisPort(getQueueSetting('CLAUDE_MEM_REDIS_PORT'));
const prefix = sanitizePrefix(getQueueSetting('CLAUDE_MEM_QUEUE_REDIS_PREFIX'));
const connection = url ? connectionFromUrl(url) : connectionFromHost(host, port);
return {
engine,
mode,
url,
host: url ? describeUrlHost(url).host : host,
port: url ? describeUrlHost(url).port : port,
prefix,
connection,
};
}
function getQueueSetting(key: keyof SettingsDefaults): string {
if (process.env[key] !== undefined) {
return process.env[key]!;
}
if (existsSync(USER_SETTINGS_PATH)) {
return SettingsDefaultsManager.loadFromFile(USER_SETTINGS_PATH)[key];
}
return SettingsDefaultsManager.get(key);
}
function normalizeRedisMode(value: string): RedisMode {
const normalized = value.trim().toLowerCase();
if (normalized === 'external' || normalized === 'managed' || normalized === 'docker') {
return normalized;
}
throw new Error(`Invalid CLAUDE_MEM_REDIS_MODE=${value}; expected external, managed, or docker`);
}
function parseRedisPort(value: string): number {
const port = Number.parseInt(value, 10);
if (!Number.isInteger(port) || port <= 0 || port > 65535) {
throw new Error(`Invalid CLAUDE_MEM_REDIS_PORT=${value}; expected a TCP port`);
}
return port;
}
function sanitizePrefix(value: string): string {
return (value.trim() || 'claude_mem').replace(/[^a-zA-Z0-9_-]/g, '_');
}
function connectionFromHost(host: string, port: number): RedisOptions {
return {
host,
port,
maxRetriesPerRequest: null,
connectTimeout: 1000,
lazyConnect: true,
};
}
function connectionFromUrl(rawUrl: string): RedisOptions {
const parsed = new URL(rawUrl);
if (parsed.protocol !== 'redis:' && parsed.protocol !== 'rediss:') {
throw new Error('CLAUDE_MEM_REDIS_URL must use redis:// or rediss://');
}
const db = parsed.pathname.length > 1
? Number.parseInt(parsed.pathname.slice(1), 10)
: undefined;
if (db !== undefined && (!Number.isInteger(db) || db < 0)) {
throw new Error(`Invalid Redis database in CLAUDE_MEM_REDIS_URL: ${parsed.pathname}`);
}
return {
host: parsed.hostname || '127.0.0.1',
port: parsed.port ? Number.parseInt(parsed.port, 10) : 6379,
username: parsed.username ? decodeURIComponent(parsed.username) : undefined,
password: parsed.password ? decodeURIComponent(parsed.password) : undefined,
db,
tls: parsed.protocol === 'rediss:' ? {} : undefined,
maxRetriesPerRequest: null,
connectTimeout: 1000,
lazyConnect: true,
};
}
function describeUrlHost(rawUrl: string): { host: string; port: number } {
const parsed = new URL(rawUrl);
return {
host: parsed.hostname || '127.0.0.1',
port: parsed.port ? Number.parseInt(parsed.port, 10) : 6379,
};
}
+264
View File
@@ -0,0 +1,264 @@
// SPDX-License-Identifier: Apache-2.0
import type { Application, Request, Response } from 'express';
import type { Database } from 'bun:sqlite';
import { z, type ZodTypeAny } from 'zod';
import type { RouteHandler } from '../../../services/server/Server.js';
import { CreateAgentEventSchema } from '../../../core/schemas/agent-event.js';
import { CreateMemoryItemSchema } from '../../../core/schemas/memory-item.js';
import { CreateProjectSchema } from '../../../core/schemas/project.js';
import { CreateServerSessionSchema } from '../../../core/schemas/session.js';
import {
AgentEventsRepository,
AuthRepository,
MemoryItemsRepository,
ProjectsRepository,
ServerSessionsRepository,
} from '../../../storage/sqlite/index.js';
import { requireServerAuth } from '../../middleware/auth.js';
declare const __DEFAULT_PACKAGE_VERSION__: string;
const BUILT_IN_VERSION = typeof __DEFAULT_PACKAGE_VERSION__ !== 'undefined'
? __DEFAULT_PACKAGE_VERSION__
: 'development';
export interface ServerV1RoutesOptions {
getDatabase: () => Database;
authMode?: string;
runtime?: string;
allowLocalDevBypass?: boolean;
}
export class ServerV1Routes implements RouteHandler {
constructor(private readonly options: ServerV1RoutesOptions) {}
setupRoutes(app: Application): void {
const readAuth = requireServerAuth(this.options.getDatabase, {
authMode: this.options.authMode,
allowLocalDevBypass: this.options.allowLocalDevBypass,
requiredScopes: ['memories:read'],
});
const writeAuth = requireServerAuth(this.options.getDatabase, {
authMode: this.options.authMode,
allowLocalDevBypass: this.options.allowLocalDevBypass,
requiredScopes: ['memories:write'],
});
app.get('/healthz', (_req, res) => {
res.json({ status: 'ok' });
});
app.get('/v1/info', (_req, res) => {
res.json({
name: 'claude-mem-server',
version: BUILT_IN_VERSION,
...(this.options.runtime ? { runtime: this.options.runtime } : {}),
authMode: this.options.authMode ?? process.env.CLAUDE_MEM_AUTH_MODE ?? 'api-key',
});
});
app.get('/v1/projects', readAuth, (req, res) => {
const repo = new ProjectsRepository(this.options.getDatabase());
const projects = req.authContext?.projectId
? [repo.getById(req.authContext.projectId)].filter(project => project !== null)
: repo.list();
res.json({ projects });
this.audit(req, 'projects.list');
});
app.post('/v1/projects', writeAuth, this.handleCreate(CreateProjectSchema, (req, res, body) => {
if (req.authContext?.projectId) {
res.status(403).json({ error: 'Forbidden', message: 'Project-scoped API keys cannot create projects' });
return;
}
const project = new ProjectsRepository(this.options.getDatabase()).create(body);
this.audit(req, 'project.create', project.id);
res.status(201).json({ project });
}));
app.get('/v1/projects/:id', readAuth, (req, res) => {
const id = this.routeParam(req.params.id);
if (!this.ensureProjectAllowed(req, res, id)) return;
const project = new ProjectsRepository(this.options.getDatabase()).getById(id);
if (!project) {
res.status(404).json({ error: 'NotFound', message: 'Project not found' });
return;
}
this.audit(req, 'project.read', project.id);
res.json({ project });
});
app.post('/v1/sessions/start', writeAuth, this.handleCreate(CreateServerSessionSchema, (req, res, body) => {
if (!this.ensureProjectAllowed(req, res, body.projectId)) return;
const session = new ServerSessionsRepository(this.options.getDatabase()).create(body);
this.audit(req, 'session.start', session.id, session.projectId);
res.status(201).json({ session });
}));
app.post('/v1/sessions/:id/end', writeAuth, (req, res) => {
const id = this.routeParam(req.params.id);
const repo = new ServerSessionsRepository(this.options.getDatabase());
const existing = repo.getById(id);
if (!existing) {
res.status(404).json({ error: 'NotFound', message: 'Session not found' });
return;
}
if (!this.ensureProjectAllowed(req, res, existing.projectId)) return;
const session = repo.markCompleted(id);
this.audit(req, 'session.end', id, existing.projectId);
res.json({ session });
});
app.get('/v1/sessions/:id', readAuth, (req, res) => {
const id = this.routeParam(req.params.id);
const session = new ServerSessionsRepository(this.options.getDatabase()).getById(id);
if (!session) {
res.status(404).json({ error: 'NotFound', message: 'Session not found' });
return;
}
if (!this.ensureProjectAllowed(req, res, session.projectId)) return;
this.audit(req, 'session.read', session.id, session.projectId);
res.json({ session });
});
app.post('/v1/events', writeAuth, this.handleCreate(CreateAgentEventSchema, (req, res, body) => {
if (!this.ensureProjectAllowed(req, res, body.projectId)) return;
const event = new AgentEventsRepository(this.options.getDatabase()).create(body);
this.audit(req, 'event.write', event.id, event.projectId);
res.status(201).json({ event });
}));
app.post('/v1/events/batch', writeAuth, this.handleCreate(z.array(CreateAgentEventSchema).min(1).max(500), (req, res, body) => {
for (const event of body) {
if (!this.ensureProjectAllowed(req, res, event.projectId)) return;
}
const db = this.options.getDatabase();
const repo = new AgentEventsRepository(db);
const insertEvents = db.transaction((eventsToCreate: typeof body) => {
return eventsToCreate.map(event => repo.create(event));
});
const events = insertEvents(body);
this.audit(req, 'event.batch_write');
res.status(201).json({ events });
}));
app.get('/v1/events/:id', readAuth, (req, res) => {
const id = this.routeParam(req.params.id);
const event = new AgentEventsRepository(this.options.getDatabase()).getById(id);
if (!event) {
res.status(404).json({ error: 'NotFound', message: 'Event not found' });
return;
}
if (!this.ensureProjectAllowed(req, res, event.projectId)) return;
this.audit(req, 'event.read', event.id, event.projectId);
res.json({ event });
});
app.post('/v1/memories', writeAuth, this.handleCreate(CreateMemoryItemSchema, (req, res, body) => {
if (!this.ensureProjectAllowed(req, res, body.projectId)) return;
const memory = new MemoryItemsRepository(this.options.getDatabase()).create(body);
this.audit(req, 'memory.write', memory.id, memory.projectId);
res.status(201).json({ memory });
}));
app.get('/v1/memories/:id', readAuth, (req, res) => {
const id = this.routeParam(req.params.id);
const memory = new MemoryItemsRepository(this.options.getDatabase()).getById(id);
if (!memory) {
res.status(404).json({ error: 'NotFound', message: 'Memory not found' });
return;
}
if (!this.ensureProjectAllowed(req, res, memory.projectId)) return;
this.audit(req, 'memory.read', memory.id, memory.projectId);
res.json({ memory });
});
app.patch('/v1/memories/:id', writeAuth, this.handleCreate(CreateMemoryItemSchema.partial(), (req, res, body) => {
const id = this.routeParam(req.params.id);
const repo = new MemoryItemsRepository(this.options.getDatabase());
const existing = repo.getById(id);
if (!existing) {
res.status(404).json({ error: 'NotFound', message: 'Memory not found' });
return;
}
if (!this.ensureProjectAllowed(req, res, existing.projectId)) return;
if (body.projectId && body.projectId !== existing.projectId) {
res.status(400).json({ error: 'ValidationError', message: 'projectId cannot be changed' });
return;
}
const memory = repo.update(id, body);
this.audit(req, 'memory.update', id, existing.projectId);
res.json({ memory });
}));
app.post('/v1/search', readAuth, this.handleCreate(z.object({
projectId: z.string().min(1),
query: z.string().min(1),
limit: z.number().int().positive().max(100).optional(),
}), (req, res, body) => {
if (!this.ensureProjectAllowed(req, res, body.projectId)) return;
const memories = new MemoryItemsRepository(this.options.getDatabase()).search(body.projectId, body.query, body.limit ?? 20);
this.audit(req, 'memory.search', null, body.projectId);
res.json({ memories });
}));
app.post('/v1/context', readAuth, this.handleCreate(z.object({
projectId: z.string().min(1),
query: z.string().min(1),
limit: z.number().int().positive().max(50).optional(),
}), (req, res, body) => {
if (!this.ensureProjectAllowed(req, res, body.projectId)) return;
const memories = new MemoryItemsRepository(this.options.getDatabase()).search(body.projectId, body.query, body.limit ?? 10);
this.audit(req, 'memory.context', null, body.projectId);
res.json({ memories, context: memories.map(memory => memory.narrative ?? memory.text ?? memory.title).filter(Boolean).join('\n\n') });
}));
app.get('/v1/audit', readAuth, (req, res) => {
const projectId = String(req.query.projectId ?? '');
if (!projectId) {
res.status(400).json({ error: 'ValidationError', message: 'projectId query parameter is required' });
return;
}
if (!this.ensureProjectAllowed(req, res, projectId)) return;
res.json({ audit: new AuthRepository(this.options.getDatabase()).listAuditLogByProject(projectId) });
});
}
private handleCreate<S extends ZodTypeAny, T = z.infer<S>>(
schema: S,
handler: (req: Request, res: Response, body: T) => void,
) {
return (req: Request, res: Response) => {
const result = schema.safeParse(req.body);
if (!result.success) {
res.status(400).json({ error: 'ValidationError', issues: result.error.issues });
return;
}
handler(req, res, result.data as T);
};
}
private ensureProjectAllowed(req: Request, res: Response, projectId: string): boolean {
if (req.authContext?.projectId && req.authContext.projectId !== projectId) {
res.status(403).json({ error: 'Forbidden', message: 'API key is scoped to a different project' });
return false;
}
return true;
}
private routeParam(value: string | string[]): string {
return Array.isArray(value) ? value[0] ?? '' : value;
}
private audit(req: Request, action: string, targetId: string | null = null, projectId: string | null = null): void {
new AuthRepository(this.options.getDatabase()).createAuditLog({
teamId: req.authContext?.teamId ?? null,
projectId: projectId ?? req.authContext?.projectId ?? null,
actorType: req.authContext?.apiKeyId ? 'api_key' : 'system',
actorId: req.authContext?.apiKeyId ?? null,
action,
targetType: targetId ? action.split('.')[0] : null,
targetId,
});
}
}
@@ -0,0 +1,115 @@
// SPDX-License-Identifier: Apache-2.0
import type { Processor } from 'bullmq';
import { ServerJobQueue } from '../jobs/ServerJobQueue.js';
import {
SERVER_JOB_QUEUE_NAMES,
type ServerGenerationJobKind,
type ServerGenerationJobPayload,
} from '../jobs/types.js';
import type { RedisQueueConfig } from '../queue/redis-config.js';
import { logger } from '../../utils/logger.js';
import type {
ServerBetaBoundaryHealth,
ServerBetaQueueManager,
} from './types.js';
// ActiveServerBetaQueueManager owns one ServerJobQueue per generation kind.
// It is wired in only when CLAUDE_MEM_QUEUE_ENGINE=bullmq is set; otherwise
// create-server-beta-service.ts keeps the disabled adapter in place.
//
// This boundary intentionally does not start any Worker processors here.
// Phase 4+ wires processors that consume the queues, calling
// `start(kind, processor)` once provider generation is ready. Until then,
// the queues exist as transports for `enqueueOutbox` to publish into.
const QUEUE_KINDS: ServerGenerationJobKind[] = ['event', 'event-batch', 'summary', 'reindex'];
export class ActiveServerBetaQueueManager implements ServerBetaQueueManager {
readonly kind = 'queue-manager' as const;
private readonly queues: Map<ServerGenerationJobKind, ServerJobQueue<ServerGenerationJobPayload>>;
private closed = false;
constructor(
private readonly config: RedisQueueConfig,
queues?: Map<ServerGenerationJobKind, ServerJobQueue<ServerGenerationJobPayload>>,
) {
if (config.engine !== 'bullmq') {
throw new Error(
`ActiveServerBetaQueueManager requires CLAUDE_MEM_QUEUE_ENGINE=bullmq (got ${config.engine}); ` +
'do not instantiate when bullmq is not selected.',
);
}
this.queues = queues ?? this.buildQueues(config);
}
getQueue(kind: ServerGenerationJobKind): ServerJobQueue<ServerGenerationJobPayload> {
const queue = this.queues.get(kind);
if (!queue) {
throw new Error(`unknown server generation job kind: ${kind}`);
}
return queue;
}
start(kind: ServerGenerationJobKind, processor: Processor<ServerGenerationJobPayload>): void {
this.getQueue(kind).start(processor);
}
getHealth(): ServerBetaBoundaryHealth {
if (this.closed) {
return { status: 'errored', reason: 'queue-manager closed' };
}
const lanes = QUEUE_KINDS.map((kind) => ({ kind, name: SERVER_JOB_QUEUE_NAMES[kind] }));
return {
status: 'active',
reason: 'BullMQ-backed queue manager wired',
details: {
engine: this.config.engine,
mode: this.config.mode,
host: this.config.host,
port: this.config.port,
prefix: this.config.prefix,
lanes,
},
};
}
async close(): Promise<void> {
if (this.closed) {
return;
}
this.closed = true;
const errors: Error[] = [];
for (const queue of this.queues.values()) {
try {
await queue.close();
} catch (error) {
errors.push(error instanceof Error ? error : new Error(String(error)));
}
}
if (errors.length > 0) {
logger.warn('QUEUE', 'errors closing server-beta queue manager', {
count: errors.length,
first: errors[0]!.message,
});
throw errors[0];
}
}
private buildQueues(
config: RedisQueueConfig,
): Map<ServerGenerationJobKind, ServerJobQueue<ServerGenerationJobPayload>> {
const map = new Map<ServerGenerationJobKind, ServerJobQueue<ServerGenerationJobPayload>>();
for (const kind of QUEUE_KINDS) {
map.set(
kind,
new ServerJobQueue<ServerGenerationJobPayload>({
name: SERVER_JOB_QUEUE_NAMES[kind],
config,
}),
);
}
return map;
}
}
+347
View File
@@ -0,0 +1,347 @@
// SPDX-License-Identifier: Apache-2.0
import type { Application } from 'express';
import { spawn } from 'child_process';
import { existsSync, mkdirSync, readFileSync, rmSync, writeFileSync } from 'fs';
import net from 'net';
import { dirname } from 'path';
import { fileURLToPath } from 'url';
import { Server, type RouteHandler } from '../../services/server/Server.js';
import { paths } from '../../shared/paths.js';
import { logger } from '../../utils/logger.js';
import {
captureProcessStartToken,
verifyPidFileOwnership,
type PidInfo,
} from '../../supervisor/process-registry.js';
import type { ServerBetaServiceGraph } from './types.js';
const SERVER_BETA_RUNTIME = 'server-beta';
const DEFAULT_SERVER_BETA_HOST = '127.0.0.1';
const DEFAULT_SERVER_BETA_PORT = 37877;
export interface ServerBetaServiceOptions {
graph: ServerBetaServiceGraph;
host?: string;
port?: number;
persistRuntimeState?: boolean;
}
export interface ServerBetaRuntimeState {
runtime: typeof SERVER_BETA_RUNTIME;
pid: number;
port: number;
host: string;
startedAt: string;
bootstrap: ServerBetaServiceGraph['postgres']['bootstrap'];
boundaries: {
queueManager: ReturnType<ServerBetaServiceGraph['queueManager']['getHealth']>;
generationWorkerManager: ReturnType<ServerBetaServiceGraph['generationWorkerManager']['getHealth']>;
providerRegistry: ReturnType<ServerBetaServiceGraph['providerRegistry']['getHealth']>;
eventBroadcaster: ReturnType<ServerBetaServiceGraph['eventBroadcaster']['getHealth']>;
};
}
class ServerBetaRuntimeInfoRoutes implements RouteHandler {
constructor(private readonly graph: ServerBetaServiceGraph) {}
setupRoutes(app: Application): void {
app.get('/healthz', (_req, res) => {
res.json({ status: 'ok', runtime: SERVER_BETA_RUNTIME });
});
app.get('/v1/info', (_req, res) => {
res.json({
name: 'claude-mem-server',
runtime: SERVER_BETA_RUNTIME,
authMode: this.graph.authMode,
postgres: {
initialized: this.graph.postgres.bootstrap.initialized,
schemaVersion: this.graph.postgres.bootstrap.schemaVersion,
},
boundaries: {
queueManager: this.graph.queueManager.getHealth(),
generationWorkerManager: this.graph.generationWorkerManager.getHealth(),
providerRegistry: this.graph.providerRegistry.getHealth(),
eventBroadcaster: this.graph.eventBroadcaster.getHealth(),
},
});
});
}
}
export class ServerBetaService {
private readonly graph: ServerBetaServiceGraph;
private readonly host: string;
private readonly requestedPort: number;
private boundPort: number | null = null;
private readonly persistRuntimeState: boolean;
private server: Server | null = null;
private stopping = false;
constructor(options: ServerBetaServiceOptions) {
this.graph = options.graph;
this.host = options.host ?? process.env.CLAUDE_MEM_SERVER_HOST ?? DEFAULT_SERVER_BETA_HOST;
this.requestedPort = options.port ?? getServerBetaPort();
this.persistRuntimeState = options.persistRuntimeState ?? true;
}
async start(): Promise<void> {
if (this.server) {
return;
}
const server = new Server({
getInitializationComplete: () => this.graph.postgres.bootstrap.initialized,
getMcpReady: () => true,
onShutdown: () => this.stop(),
onRestart: async () => {
await this.stop();
await this.start();
},
workerPath: '',
runtime: SERVER_BETA_RUNTIME,
getAiStatus: () => ({
provider: 'disabled',
authMethod: this.graph.authMode,
lastInteraction: null,
}),
});
server.registerRoutes(new ServerBetaRuntimeInfoRoutes(this.graph));
server.finalizeRoutes();
await server.listen(this.requestedPort, this.host);
this.server = server;
this.boundPort = resolveBoundPort(server) ?? this.requestedPort;
if (this.persistRuntimeState) {
writeServerBetaState(this.runtimeState());
}
logger.info('SYSTEM', 'Server beta started', { host: this.host, port: this.boundPort, pid: process.pid });
}
async stop(): Promise<void> {
if (this.stopping) {
return;
}
this.stopping = true;
try {
if (this.server) {
try {
await this.server.close();
} catch (error: unknown) {
if ((error as NodeJS.ErrnoException)?.code !== 'ERR_SERVER_NOT_RUNNING') {
throw error;
}
}
this.server = null;
}
await Promise.all([
this.graph.queueManager.close(),
this.graph.generationWorkerManager.close(),
this.graph.providerRegistry.close(),
this.graph.eventBroadcaster.close(),
]);
await this.graph.postgres.pool.end();
} finally {
if (this.persistRuntimeState) {
removeServerBetaState();
}
this.boundPort = null;
this.stopping = false;
logger.info('SYSTEM', 'Server beta stopped');
}
}
getRuntimeState(): ServerBetaRuntimeState {
return this.runtimeState();
}
private runtimeState(): ServerBetaRuntimeState {
return {
runtime: SERVER_BETA_RUNTIME,
pid: process.pid,
port: this.boundPort ?? this.requestedPort,
host: this.host,
startedAt: new Date().toISOString(),
bootstrap: this.graph.postgres.bootstrap,
boundaries: {
queueManager: this.graph.queueManager.getHealth(),
generationWorkerManager: this.graph.generationWorkerManager.getHealth(),
providerRegistry: this.graph.providerRegistry.getHealth(),
eventBroadcaster: this.graph.eventBroadcaster.getHealth(),
},
};
}
}
function resolveBoundPort(server: Server): number | null {
const address = server.getHttpServer()?.address();
return address && typeof address !== 'string' ? address.port : null;
}
export async function runServerBetaCli(argv: string[] = process.argv.slice(2)): Promise<void> {
const command = argv[0] ?? '--daemon';
const port = getServerBetaPort();
const host = process.env.CLAUDE_MEM_SERVER_HOST ?? DEFAULT_SERVER_BETA_HOST;
switch (command) {
case 'start': {
const existing = readServerBetaPidFile();
if (verifyPidFileOwnership(existing)) {
console.log(JSON.stringify({ status: 'ready', runtime: SERVER_BETA_RUNTIME, pid: existing.pid, port: existing.port }));
return;
}
const daemonPid = spawnServerBetaDaemon(port);
if (daemonPid === undefined) {
console.error('Failed to spawn server beta daemon.');
process.exit(1);
}
console.log(JSON.stringify({ status: 'starting', runtime: SERVER_BETA_RUNTIME, pid: daemonPid, port }));
return;
}
case 'stop': {
const existing = readServerBetaPidFile();
if (!verifyPidFileOwnership(existing)) {
removeServerBetaState();
console.log('Server beta is not running');
return;
}
process.kill(existing.pid, 'SIGTERM');
await waitForPidExit(existing.pid, 5000);
removeServerBetaState();
console.log('Server beta stopped');
return;
}
case 'restart': {
await runServerBetaCli(['stop']);
await runServerBetaCli(['start']);
return;
}
case 'status': {
const state = readServerBetaRuntimeState();
const pidInfo = readServerBetaPidFile();
if (state && verifyPidFileOwnership(pidInfo)) {
console.log('Server beta is running');
console.log(` PID: ${state.pid}`);
console.log(` Port: ${state.port}`);
console.log(` Runtime: ${state.runtime}`);
console.log(` Started: ${state.startedAt}`);
} else {
console.log('Server beta is not running');
}
return;
}
case '--daemon': {
const existing = readServerBetaPidFile();
if (verifyPidFileOwnership(existing) || await isPortInUse(port, host)) {
process.exit(0);
}
const { createServerBetaService } = await import('./create-server-beta-service.js');
const service = await createServerBetaService();
const shutdown = async () => {
await service.stop();
process.exit(0);
};
process.once('SIGTERM', shutdown);
process.once('SIGINT', shutdown);
await service.start();
return;
}
default:
console.error('Usage: server-beta-service start|stop|restart|status');
process.exit(1);
}
}
function getServerBetaPort(): number {
const parsed = Number.parseInt(process.env.CLAUDE_MEM_SERVER_PORT ?? '', 10);
return Number.isInteger(parsed) && parsed > 0 ? parsed : DEFAULT_SERVER_BETA_PORT;
}
function spawnServerBetaDaemon(port: number): number | undefined {
const scriptPath = typeof __filename !== 'undefined' ? __filename : fileURLToPath(import.meta.url);
const child = spawn(process.execPath, [scriptPath, '--daemon'], {
detached: true,
stdio: 'ignore',
env: {
...process.env,
CLAUDE_MEM_SERVER_PORT: String(port),
},
});
child.unref();
return child.pid;
}
function writeServerBetaState(state: ServerBetaRuntimeState): void {
mkdirSync(dirname(paths.serverBetaRuntime()), { recursive: true });
const pidInfo: PidInfo = {
pid: state.pid,
port: state.port,
startedAt: state.startedAt,
startToken: captureProcessStartToken(state.pid) ?? undefined,
};
writeFileSync(paths.serverBetaPid(), JSON.stringify(pidInfo, null, 2));
writeFileSync(paths.serverBetaPort(), `${state.port}\n`);
writeFileSync(paths.serverBetaRuntime(), JSON.stringify(state, null, 2));
}
function readServerBetaPidFile(): PidInfo | null {
if (!existsSync(paths.serverBetaPid())) {
return null;
}
try {
return JSON.parse(readFileSync(paths.serverBetaPid(), 'utf-8')) as PidInfo;
} catch {
return null;
}
}
function readServerBetaRuntimeState(): ServerBetaRuntimeState | null {
if (!existsSync(paths.serverBetaRuntime())) {
return null;
}
try {
return JSON.parse(readFileSync(paths.serverBetaRuntime(), 'utf-8')) as ServerBetaRuntimeState;
} catch {
return null;
}
}
function removeServerBetaState(): void {
rmSync(paths.serverBetaPid(), { force: true });
rmSync(paths.serverBetaPort(), { force: true });
rmSync(paths.serverBetaRuntime(), { force: true });
}
async function isPortInUse(port: number, host: string): Promise<boolean> {
return new Promise(resolve => {
const socket = net.connect({ port, host });
socket.once('connect', () => {
socket.destroy();
resolve(true);
});
socket.once('error', () => resolve(false));
});
}
async function waitForPidExit(pid: number, timeoutMs: number): Promise<void> {
const deadline = Date.now() + timeoutMs;
while (Date.now() < deadline) {
if (!verifyPidFileOwnership({ pid, port: 0, startedAt: '' })) {
return;
}
await new Promise(resolve => setTimeout(resolve, 100));
}
}
if (process.argv[1]?.endsWith('ServerBetaService.ts') || process.argv[1]?.endsWith('server-beta-service.cjs')) {
runServerBetaCli().catch(error => {
console.error(error instanceof Error ? error.message : String(error));
process.exit(1);
});
}
@@ -0,0 +1,93 @@
// SPDX-License-Identifier: Apache-2.0
import { createPostgresStorageRepositories, getSharedPostgresPool, SERVER_BETA_POSTGRES_SCHEMA_VERSION } from '../../storage/postgres/index.js';
import { bootstrapServerBetaPostgresSchema } from '../../storage/postgres/schema.js';
import type { PostgresPool } from '../../storage/postgres/pool.js';
import { getRedisQueueConfig } from '../queue/redis-config.js';
import { ActiveServerBetaQueueManager } from './ActiveServerBetaQueueManager.js';
import { ServerBetaService } from './ServerBetaService.js';
import {
DisabledServerBetaEventBroadcaster,
DisabledServerBetaGenerationWorkerManager,
DisabledServerBetaProviderRegistry,
DisabledServerBetaQueueManager,
type ServerBetaAuthMode,
type ServerBetaBootstrapStatus,
type ServerBetaQueueManager,
type ServerBetaServiceGraph,
} from './types.js';
export interface CreateServerBetaServiceOptions {
pool?: PostgresPool;
authMode?: ServerBetaAuthMode;
bootstrapSchema?: boolean;
queueManager?: ServerBetaQueueManager;
}
export async function createServerBetaService(
options: CreateServerBetaServiceOptions = {},
): Promise<ServerBetaService> {
const pool = options.pool ?? getSharedPostgresPool({ requireDatabaseUrl: true });
const bootstrap = await initializePostgres(pool, options.bootstrapSchema ?? true);
const graph: ServerBetaServiceGraph = {
runtime: 'server-beta',
postgres: {
pool,
bootstrap,
},
authMode: options.authMode ?? parseAuthMode(process.env.CLAUDE_MEM_AUTH_MODE),
queueManager: options.queueManager ?? buildQueueManager(),
generationWorkerManager: new DisabledServerBetaGenerationWorkerManager('Phase 2 boundary only; generation workers are not wired.'),
providerRegistry: new DisabledServerBetaProviderRegistry('Phase 2 boundary only; provider-backed generation is not wired.'),
eventBroadcaster: new DisabledServerBetaEventBroadcaster('Phase 2 boundary only; SSE/event broadcasting is not wired.'),
storage: createPostgresStorageRepositories(pool),
};
return new ServerBetaService({ graph });
}
// Queue manager selection is fail-fast on misconfiguration. If the user
// explicitly opts into BullMQ via CLAUDE_MEM_QUEUE_ENGINE=bullmq we build
// the active manager; any error there throws so the runtime does not
// silently fall back to a disabled queue. Default behavior (sqlite engine
// or no opt-in) keeps the disabled boundary so worker-era runtimes stay
// compatible.
function buildQueueManager(): ServerBetaQueueManager {
const config = getRedisQueueConfig();
if (config.engine !== 'bullmq') {
return new DisabledServerBetaQueueManager(
`Queue engine is "${config.engine}"; set CLAUDE_MEM_QUEUE_ENGINE=bullmq to activate the server-beta queue manager.`,
);
}
return new ActiveServerBetaQueueManager(config);
}
async function initializePostgres(pool: PostgresPool, bootstrapSchema: boolean): Promise<ServerBetaBootstrapStatus> {
if (!bootstrapSchema) {
return { initialized: false, schemaVersion: null, appliedAt: null };
}
await bootstrapServerBetaPostgresSchema(pool);
const result = await pool.query(
`
SELECT version, applied_at
FROM server_beta_schema_migrations
WHERE version = $1
`,
[SERVER_BETA_POSTGRES_SCHEMA_VERSION],
);
const row = result.rows[0] as { version?: number; applied_at?: Date | string } | undefined;
return {
initialized: row?.version === SERVER_BETA_POSTGRES_SCHEMA_VERSION,
schemaVersion: typeof row?.version === 'number' ? row.version : null,
appliedAt: row?.applied_at ? new Date(row.applied_at).toISOString() : null,
};
}
function parseAuthMode(value: string | undefined): ServerBetaAuthMode {
if (value === 'local-dev' || value === 'disabled') {
return value;
}
return 'api-key';
}
+90
View File
@@ -0,0 +1,90 @@
// SPDX-License-Identifier: Apache-2.0
import type { PostgresPool, PostgresStorageRepositories } from '../../storage/postgres/index.js';
export type ServerBetaRuntimeName = 'server-beta';
export type ServerBetaAuthMode = 'api-key' | 'local-dev' | 'disabled';
export type DisabledBoundaryStatus = 'disabled';
export type ServerBetaBoundaryStatus = 'disabled' | 'active' | 'errored';
export interface ServerBetaBootstrapStatus {
initialized: boolean;
schemaVersion: number | null;
appliedAt: string | null;
error?: string;
}
export interface ServerBetaBoundaryHealth {
status: ServerBetaBoundaryStatus;
reason: string;
details?: Record<string, unknown>;
}
export interface ServerBetaQueueManager {
readonly kind: 'queue-manager';
getHealth(): ServerBetaBoundaryHealth;
close(): Promise<void>;
}
export interface ServerBetaGenerationWorkerManager {
readonly kind: 'generation-worker-manager';
getHealth(): ServerBetaBoundaryHealth;
close(): Promise<void>;
}
export interface ServerBetaProviderRegistry {
readonly kind: 'provider-registry';
getHealth(): ServerBetaBoundaryHealth;
close(): Promise<void>;
}
export interface ServerBetaEventBroadcaster {
readonly kind: 'event-broadcaster';
getHealth(): ServerBetaBoundaryHealth;
close(): Promise<void>;
}
export interface ServerBetaServiceGraph {
runtime: ServerBetaRuntimeName;
postgres: {
pool: PostgresPool;
bootstrap: ServerBetaBootstrapStatus;
};
authMode: ServerBetaAuthMode;
queueManager: ServerBetaQueueManager;
generationWorkerManager: ServerBetaGenerationWorkerManager;
providerRegistry: ServerBetaProviderRegistry;
eventBroadcaster: ServerBetaEventBroadcaster;
storage: PostgresStorageRepositories;
}
abstract class DisabledServerBetaBoundary {
abstract readonly kind: ServerBetaQueueManager['kind']
| ServerBetaGenerationWorkerManager['kind']
| ServerBetaProviderRegistry['kind']
| ServerBetaEventBroadcaster['kind'];
constructor(private readonly reason: string) {}
getHealth(): ServerBetaBoundaryHealth {
return { status: 'disabled' as const, reason: this.reason };
}
async close(): Promise<void> {}
}
export class DisabledServerBetaQueueManager extends DisabledServerBetaBoundary implements ServerBetaQueueManager {
readonly kind = 'queue-manager' as const;
}
export class DisabledServerBetaGenerationWorkerManager extends DisabledServerBetaBoundary implements ServerBetaGenerationWorkerManager {
readonly kind = 'generation-worker-manager' as const;
}
export class DisabledServerBetaProviderRegistry extends DisabledServerBetaBoundary implements ServerBetaProviderRegistry {
readonly kind = 'provider-registry' as const;
}
export class DisabledServerBetaEventBroadcaster extends DisabledServerBetaBoundary implements ServerBetaEventBroadcaster {
readonly kind = 'event-broadcaster' as const;
}
@@ -87,10 +87,10 @@ function scanCleanupCounts(dbPath: string): CleanupCounts {
+ (db.prepare(`SELECT COUNT(*) AS n FROM session_summaries WHERE memory_session_id IN (SELECT memory_session_id FROM sdk_sessions WHERE project = ? AND memory_session_id IS NOT NULL)`).get(OBSERVER_SESSIONS_PROJECT) as { n: number }).n;
counts.stuckPendingMessages = (db.prepare(
`SELECT COUNT(*) AS n FROM pending_messages
WHERE status IN ('failed', 'processing')
WHERE status = 'processing'
AND session_db_id IN (
SELECT session_db_id FROM pending_messages
WHERE status IN ('failed', 'processing')
WHERE status = 'processing'
GROUP BY session_db_id
HAVING COUNT(*) >= ?
)`
@@ -222,10 +222,10 @@ function runStuckPendingPurge(db: Database, counts: CleanupCounts): void {
try {
const stuckCount = (db.prepare(
`SELECT COUNT(*) AS n FROM pending_messages
WHERE status IN ('failed', 'processing')
WHERE status = 'processing'
AND session_db_id IN (
SELECT session_db_id FROM pending_messages
WHERE status IN ('failed', 'processing')
WHERE status = 'processing'
GROUP BY session_db_id
HAVING COUNT(*) >= ?
)`
@@ -233,10 +233,10 @@ function runStuckPendingPurge(db: Database, counts: CleanupCounts): void {
db.run(
`DELETE FROM pending_messages
WHERE status IN ('failed', 'processing')
WHERE status = 'processing'
AND session_db_id IN (
SELECT session_db_id FROM pending_messages
WHERE status IN ('failed', 'processing')
WHERE status = 'processing'
GROUP BY session_db_id
HAVING COUNT(*) >= ?
)`,
+47 -7
View File
@@ -9,6 +9,9 @@ export interface CreateIteratorOptions {
sessionDbId: number;
signal: AbortSignal;
onIdleTimeout?: () => void;
idleTimeoutMs?: number;
claimRetryDelayMs?: number;
maxClaimFailures?: number;
}
export class SessionQueueProcessor {
@@ -18,8 +21,16 @@ export class SessionQueueProcessor {
) {}
async *createIterator(options: CreateIteratorOptions): AsyncIterableIterator<PendingMessageWithId> {
const { sessionDbId, signal, onIdleTimeout } = options;
const {
sessionDbId,
signal,
onIdleTimeout,
idleTimeoutMs = IDLE_TIMEOUT_MS,
claimRetryDelayMs = 250,
maxClaimFailures = 3
} = options;
let lastActivityTime = Date.now();
let claimFailures = 0;
while (!signal.aborted) {
let persistentMessage: PersistentPendingMessage | null = null;
@@ -28,18 +39,25 @@ export class SessionQueueProcessor {
} catch (error) {
if (signal.aborted) return;
const normalizedError = error instanceof Error ? error : new Error(String(error));
logger.error('QUEUE', 'Failed to claim next message; ending iterator', { sessionDbId }, normalizedError);
return;
claimFailures++;
logger.error('QUEUE', 'Failed to claim next message', { sessionDbId, claimFailures, maxClaimFailures }, normalizedError);
if (claimFailures >= maxClaimFailures) {
logger.error('QUEUE', 'Claim failure limit reached; ending iterator', { sessionDbId, claimFailures }, normalizedError);
return;
}
await this.waitForDelay(signal, claimRetryDelayMs);
continue;
}
if (persistentMessage) {
claimFailures = 0;
lastActivityTime = Date.now();
yield this.toPendingMessageWithId(persistentMessage);
continue;
}
try {
const idleTimedOut = await this.handleWaitPhase(signal, lastActivityTime, sessionDbId, onIdleTimeout);
const idleTimedOut = await this.handleWaitPhase(signal, lastActivityTime, sessionDbId, idleTimeoutMs, onIdleTimeout);
if (idleTimedOut) return;
lastActivityTime = Date.now();
} catch (error) {
@@ -64,17 +82,18 @@ export class SessionQueueProcessor {
signal: AbortSignal,
lastActivityTime: number,
sessionDbId: number,
idleTimeoutMs: number,
onIdleTimeout?: () => void
): Promise<boolean> {
const receivedMessage = await this.waitForMessage(signal, IDLE_TIMEOUT_MS);
const receivedMessage = await this.waitForMessage(signal, idleTimeoutMs);
if (!receivedMessage && !signal.aborted) {
const idleDuration = Date.now() - lastActivityTime;
if (idleDuration >= IDLE_TIMEOUT_MS) {
if (idleDuration >= idleTimeoutMs) {
logger.info('SESSION', 'Idle timeout reached, triggering abort to kill subprocess', {
sessionDbId,
idleDurationMs: idleDuration,
thresholdMs: IDLE_TIMEOUT_MS
thresholdMs: idleTimeoutMs
});
onIdleTimeout?.();
return true;
@@ -115,4 +134,25 @@ export class SessionQueueProcessor {
timeoutId = setTimeout(onTimeout, timeoutMs);
});
}
private waitForDelay(signal: AbortSignal, delayMs: number): Promise<void> {
return new Promise<void>((resolve) => {
let timeoutId: ReturnType<typeof setTimeout> | undefined;
const cleanup = () => {
if (timeoutId !== undefined) {
clearTimeout(timeoutId);
}
signal.removeEventListener('abort', onAbort);
};
const onAbort = () => {
cleanup();
resolve();
};
timeoutId = setTimeout(() => {
cleanup();
resolve();
}, delayMs);
signal.addEventListener('abort', onAbort, { once: true });
});
}
}
+1
View File
@@ -1,5 +1,6 @@
export {
createCorsMiddleware,
createMiddleware,
requireLocalhost,
summarizeRequestBody
+25 -5
View File
@@ -5,7 +5,7 @@ import * as fs from 'fs';
import path from 'path';
import { ALLOWED_OPERATIONS, ALLOWED_TOPICS } from './allowed-constants.js';
import { logger } from '../../utils/logger.js';
import { createMiddleware, summarizeRequestBody, requireLocalhost } from './Middleware.js';
import { createCorsMiddleware, createMiddleware, summarizeRequestBody, requireLocalhost } from './Middleware.js';
import { errorHandler, notFoundHandler } from './ErrorHandler.js';
import { getSupervisor } from '../../supervisor/index.js';
import { isPidAlive } from '../../supervisor/process-registry.js';
@@ -13,6 +13,7 @@ import { ENV_PREFIXES, ENV_EXACT_MATCHES } from '../../supervisor/env-sanitizer.
import { flushResponseThen } from './flushResponseThen.js';
import { getUptimeSeconds } from '../../shared/uptime.js';
import { globalRateLimitStore } from '../worker/RateLimitStore.js';
import type { ObservationQueueHealth } from '../../server/queue/ObservationQueueEngine.js';
const INSTRUCTIONS_BASE_DIR: string = path.resolve(__dirname, '../skills/mem-search');
const INSTRUCTIONS_OPERATIONS_DIR: string = path.join(INSTRUCTIONS_BASE_DIR, 'operations');
@@ -82,7 +83,10 @@ export interface ServerOptions {
onShutdown: () => Promise<void>;
onRestart: () => Promise<void>;
workerPath: string;
runtime?: string;
getAiStatus: () => AiStatus;
preBodyParserRoutes?: RouteHandler[];
getQueueHealth?: () => ObservationQueueHealth | null | Promise<ObservationQueueHealth | null>;
}
export class Server {
@@ -94,6 +98,8 @@ export class Server {
constructor(options: ServerOptions) {
this.options = options;
this.app = express();
this.setupCors();
this.setupPreBodyParserRoutes();
this.setupMiddleware();
this.setupCoreRoutes();
}
@@ -153,14 +159,27 @@ export class Server {
}
private setupMiddleware(): void {
const middlewares = createMiddleware(summarizeRequestBody);
const middlewares = createMiddleware(summarizeRequestBody, { includeCors: false });
middlewares.forEach(mw => this.app.use(mw));
}
private setupCors(): void {
this.app.use(createCorsMiddleware());
}
private setupPreBodyParserRoutes(): void {
this.options.preBodyParserRoutes?.forEach(handler => handler.setupRoutes(this.app));
}
private setupCoreRoutes(): void {
this.app.get('/api/health', (_req: Request, res: Response) => {
res.status(200).json({
status: 'ok',
this.app.get('/api/health', async (_req: Request, res: Response) => {
const queueHealth = this.options.getQueueHealth
? await this.options.getQueueHealth()
: null;
const queueDegraded = queueHealth?.engine === 'bullmq' && queueHealth.redis.status === 'error';
res.status(queueDegraded ? 503 : 200).json({
status: queueDegraded ? 'degraded' : 'ok',
...(this.options.runtime ? { runtime: this.options.runtime } : {}),
version: BUILT_IN_VERSION,
workerPath: this.options.workerPath,
uptime: getUptimeSeconds(this.startTime),
@@ -172,6 +191,7 @@ export class Server {
mcpReady: this.options.getMcpReady(),
ai: this.options.getAiStatus(),
rateLimits: globalRateLimitStore.getMostRecentByWindow(),
...(queueHealth ? { queue: queueHealth } : {}),
});
});
+42 -3
View File
@@ -57,8 +57,11 @@ export class PendingMessageStore {
message.agentId ?? null
);
this.onMutate?.();
return result.lastInsertRowid as number;
if (result.changes > 0) {
this.onMutate?.();
return result.lastInsertRowid as number;
}
return 0;
}
claimNextMessage(sessionDbId: number): PersistentPendingMessage | null {
@@ -79,7 +82,9 @@ export class PendingMessageStore {
sessionId: sessionDbId
});
}
this.onMutate?.();
if (claimed) {
this.onMutate?.();
}
return claimed;
}
@@ -122,6 +127,40 @@ export class PendingMessageStore {
return result.count;
}
getTotalQueueDepth(): number {
const stmt = this.db.prepare(`
SELECT COUNT(*) as count FROM pending_messages
WHERE status IN ('pending', 'processing')
`);
const result = stmt.get() as { count: number };
return result.count;
}
hasAnyPendingWork(): boolean {
return this.getTotalQueueDepth() > 0;
}
getSessionsWithPendingMessages(): number[] {
const stmt = this.db.prepare(`
SELECT DISTINCT session_db_id FROM pending_messages
WHERE status IN ('pending', 'processing')
ORDER BY session_db_id ASC
`);
return (stmt.all() as Array<{ session_db_id: number }>).map(row => row.session_db_id);
}
confirmProcessed(messageId: number): number {
const stmt = this.db.prepare(`
DELETE FROM pending_messages
WHERE id = ? AND status = 'processing'
`);
const changes = stmt.run(messageId).changes;
if (changes > 0) {
this.onMutate?.();
}
return changes;
}
peekPendingTypes(sessionDbId: number): Array<{ message_type: string; tool_name: string | null }> {
const stmt = this.db.prepare(`
SELECT message_type, tool_name FROM pending_messages
+143 -20
View File
@@ -7,6 +7,7 @@ import {
SchemaVersion
} from '../../../types/database.js';
import { DEFAULT_PLATFORM_SOURCE } from '../../../shared/platform-source.js';
import { ensureServerStorageSchema, SERVER_STORAGE_SCHEMA_VERSION } from '../../../storage/sqlite/schema.js';
export class MigrationRunner {
constructor(private db: Database) {}
@@ -34,6 +35,9 @@ export class MigrationRunner {
this.addObservationsUniqueContentHashIndex();
this.addObservationsMetadataColumn();
this.dropDeadPendingMessagesColumns();
this.dropWorkerPidColumn();
this.createServerOwnedTables();
this.rebuildPendingMessagesForFinalQueueSchema();
}
private initializeSchema(): void {
@@ -422,10 +426,8 @@ export class MigrationRunner {
last_user_message TEXT,
last_assistant_message TEXT,
prompt_number INTEGER,
status TEXT NOT NULL DEFAULT 'pending' CHECK(status IN ('pending', 'processing', 'processed', 'failed')),
retry_count INTEGER NOT NULL DEFAULT 0,
status TEXT NOT NULL DEFAULT 'pending' CHECK(status IN ('pending', 'processing')),
created_at_epoch INTEGER NOT NULL,
completed_at_epoch INTEGER,
FOREIGN KEY (session_db_id) REFERENCES sdk_sessions(id) ON DELETE CASCADE
)
`);
@@ -844,12 +846,8 @@ export class MigrationRunner {
last_assistant_message TEXT,
prompt_number INTEGER,
status TEXT NOT NULL DEFAULT 'pending'
CHECK(status IN ('pending', 'processing', 'processed', 'failed')),
retry_count INTEGER NOT NULL DEFAULT 0,
CHECK(status IN ('pending', 'processing')),
created_at_epoch INTEGER NOT NULL,
failed_at_epoch INTEGER,
completed_at_epoch INTEGER,
worker_pid INTEGER,
agent_type TEXT,
agent_id TEXT,
FOREIGN KEY (session_db_id) REFERENCES sdk_sessions(id) ON DELETE CASCADE
@@ -860,8 +858,7 @@ export class MigrationRunner {
INSERT INTO pending_messages_new (
id, session_db_id, content_session_id, tool_use_id, message_type,
tool_name, tool_input, tool_response, cwd, last_user_message,
last_assistant_message, prompt_number, status, retry_count,
created_at_epoch, failed_at_epoch, completed_at_epoch, worker_pid,
last_assistant_message, prompt_number, status, created_at_epoch,
agent_type, agent_id
)
SELECT
@@ -870,22 +867,19 @@ export class MigrationRunner {
content_session_id,
${has('tool_use_id') ? 'tool_use_id' : 'NULL'},
message_type,
tool_name,
tool_input,
tool_response,
cwd,
${has('tool_name') ? 'tool_name' : 'NULL'},
${has('tool_input') ? 'tool_input' : 'NULL'},
${has('tool_response') ? 'tool_response' : 'NULL'},
${has('cwd') ? 'cwd' : 'NULL'},
${has('last_user_message') ? 'last_user_message' : 'NULL'},
${has('last_assistant_message') ? 'last_assistant_message' : 'NULL'},
${has('prompt_number') ? 'prompt_number' : 'NULL'},
status,
retry_count,
CASE WHEN status = 'processing' THEN 'processing' ELSE 'pending' END,
created_at_epoch,
${has('failed_at_epoch') ? 'failed_at_epoch' : 'NULL'},
${has('completed_at_epoch') ? 'completed_at_epoch' : 'NULL'},
NULL,
${has('agent_type') ? 'agent_type' : 'NULL'},
${has('agent_id') ? 'agent_id' : 'NULL'}
FROM pending_messages
WHERE status IN ('pending', 'processing')
`);
this.db.run('DROP TABLE pending_messages');
@@ -894,7 +888,6 @@ export class MigrationRunner {
this.db.run('CREATE INDEX IF NOT EXISTS idx_pending_messages_session ON pending_messages(session_db_id)');
this.db.run('CREATE INDEX IF NOT EXISTS idx_pending_messages_status ON pending_messages(status)');
this.db.run('CREATE INDEX IF NOT EXISTS idx_pending_messages_claude_session ON pending_messages(content_session_id)');
this.db.run('CREATE INDEX IF NOT EXISTS idx_pending_messages_worker_pid ON pending_messages(worker_pid)');
this.db.run(`
DELETE FROM pending_messages
@@ -991,6 +984,10 @@ export class MigrationRunner {
this.db.run(`DELETE FROM pending_messages WHERE status NOT IN ('pending', 'processing')`);
if (toDrop.includes('worker_pid')) {
this.db.run('DROP INDEX IF EXISTS idx_pending_messages_worker_pid');
}
for (const colName of toDrop) {
try {
this.db.run(`ALTER TABLE pending_messages DROP COLUMN ${colName}`);
@@ -1002,4 +999,130 @@ export class MigrationRunner {
this.db.prepare('INSERT OR IGNORE INTO schema_versions (version, applied_at) VALUES (?, ?)').run(31, new Date().toISOString());
}
private dropWorkerPidColumn(): void {
const applied = this.db.prepare('SELECT version FROM schema_versions WHERE version = ?').get(32) as SchemaVersion | undefined;
if (applied) return;
const cols = this.db.query('PRAGMA table_info(pending_messages)').all() as TableColumnInfo[];
const hasColumn = cols.some(c => c.name === 'worker_pid');
if (hasColumn) {
try {
this.db.run('DROP INDEX IF EXISTS idx_pending_messages_worker_pid');
this.db.run('ALTER TABLE pending_messages DROP COLUMN worker_pid');
logger.debug('DB', 'Dropped worker_pid column and its index from pending_messages');
} catch (error) {
logger.warn('DB', 'Failed to drop worker_pid column from pending_messages', {}, error instanceof Error ? error : new Error(String(error)));
}
}
this.db.prepare('INSERT OR IGNORE INTO schema_versions (version, applied_at) VALUES (?, ?)').run(32, new Date().toISOString());
}
private createServerOwnedTables(): void {
ensureServerStorageSchema(this.db);
this.db.prepare('INSERT OR IGNORE INTO schema_versions (version, applied_at) VALUES (?, ?)').run(
SERVER_STORAGE_SCHEMA_VERSION,
new Date().toISOString()
);
}
private rebuildPendingMessagesForFinalQueueSchema(): void {
const applied = this.db.prepare('SELECT version FROM schema_versions WHERE version = ?').get(34) as SchemaVersion | undefined;
if (applied) return;
const table = this.db.prepare("SELECT sql FROM sqlite_master WHERE type='table' AND name='pending_messages'").get() as { sql: string } | null;
if (!table) {
this.db.prepare('INSERT OR IGNORE INTO schema_versions (version, applied_at) VALUES (?, ?)').run(34, new Date().toISOString());
return;
}
const hasStaleStatusCheck = table.sql.includes("'processed'") || table.sql.includes("'failed'");
const cols = this.db.query('PRAGMA table_info(pending_messages)').all() as TableColumnInfo[];
const colNames = new Set(cols.map(c => c.name));
const hasDeadColumns = ['retry_count', 'failed_at_epoch', 'completed_at_epoch', 'worker_pid'].some(name => colNames.has(name));
if (!hasStaleStatusCheck && !hasDeadColumns) {
this.db.prepare('INSERT OR IGNORE INTO schema_versions (version, applied_at) VALUES (?, ?)').run(34, new Date().toISOString());
return;
}
const has = (name: string) => colNames.has(name);
this.db.run('PRAGMA foreign_keys = OFF');
this.db.run('BEGIN TRANSACTION');
try {
this.db.run('DROP TABLE IF EXISTS pending_messages_final');
this.db.run(`
CREATE TABLE pending_messages_final (
id INTEGER PRIMARY KEY AUTOINCREMENT,
session_db_id INTEGER NOT NULL,
content_session_id TEXT NOT NULL,
tool_use_id TEXT,
message_type TEXT NOT NULL CHECK(message_type IN ('observation', 'summarize')),
tool_name TEXT,
tool_input TEXT,
tool_response TEXT,
cwd TEXT,
last_user_message TEXT,
last_assistant_message TEXT,
prompt_number INTEGER,
status TEXT NOT NULL DEFAULT 'pending' CHECK(status IN ('pending', 'processing')),
created_at_epoch INTEGER NOT NULL,
agent_type TEXT,
agent_id TEXT,
FOREIGN KEY (session_db_id) REFERENCES sdk_sessions(id) ON DELETE CASCADE
)
`);
this.db.run(`
INSERT INTO pending_messages_final (
id, session_db_id, content_session_id, tool_use_id, message_type,
tool_name, tool_input, tool_response, cwd, last_user_message,
last_assistant_message, prompt_number, status, created_at_epoch,
agent_type, agent_id
)
SELECT
id,
session_db_id,
content_session_id,
${has('tool_use_id') ? 'tool_use_id' : 'NULL'},
message_type,
${has('tool_name') ? 'tool_name' : 'NULL'},
${has('tool_input') ? 'tool_input' : 'NULL'},
${has('tool_response') ? 'tool_response' : 'NULL'},
${has('cwd') ? 'cwd' : 'NULL'},
${has('last_user_message') ? 'last_user_message' : 'NULL'},
${has('last_assistant_message') ? 'last_assistant_message' : 'NULL'},
${has('prompt_number') ? 'prompt_number' : 'NULL'},
CASE WHEN status = 'processing' THEN 'processing' ELSE 'pending' END,
created_at_epoch,
${has('agent_type') ? 'agent_type' : 'NULL'},
${has('agent_id') ? 'agent_id' : 'NULL'}
FROM pending_messages
WHERE status IN ('pending', 'processing')
`);
this.db.run('DROP TABLE pending_messages');
this.db.run('ALTER TABLE pending_messages_final RENAME TO pending_messages');
this.db.run('CREATE INDEX IF NOT EXISTS idx_pending_messages_session ON pending_messages(session_db_id)');
this.db.run('CREATE INDEX IF NOT EXISTS idx_pending_messages_status ON pending_messages(status)');
this.db.run('CREATE INDEX IF NOT EXISTS idx_pending_messages_claude_session ON pending_messages(content_session_id)');
this.db.run(`
CREATE UNIQUE INDEX IF NOT EXISTS ux_pending_session_tool
ON pending_messages(content_session_id, tool_use_id)
WHERE tool_use_id IS NOT NULL
`);
this.db.prepare('INSERT OR IGNORE INTO schema_versions (version, applied_at) VALUES (?, ?)').run(34, new Date().toISOString());
this.db.run('COMMIT');
this.db.run('PRAGMA foreign_keys = ON');
} catch (error) {
this.db.run('ROLLBACK');
this.db.run('PRAGMA foreign_keys = ON');
if (error instanceof Error) {
throw error;
}
throw new Error(`Migration 34 failed: ${String(error)}`);
}
}
}
+1 -2
View File
@@ -1,8 +1,7 @@
-- claude-mem SQLite schema
--
-- Authoritative shape of the database after all migrations through
-- runner.ts have been applied (current runner tip = migration 31;
-- SessionStore boot repair records migration 32). Fresh
-- runner.ts have been applied (current tip = migration 34). Fresh
-- databases boot directly into this shape; existing databases reach
-- it via the migration runner.
--
+1 -1
View File
@@ -244,7 +244,7 @@ export class TranscriptEventProcessor {
const toolName = typeof fields.toolName === 'string' ? fields.toolName : undefined;
if (!toolName) return;
const result = ingestObservation({
const result = await ingestObservation({
contentSessionId: session.sessionId,
cwd: session.cwd ?? process.cwd(),
toolName,
+277 -19
View File
@@ -1,9 +1,12 @@
import path from 'path';
import { existsSync } from 'fs';
import { spawn } from 'child_process';
import { Database } from 'bun:sqlite';
import { Client } from '@modelcontextprotocol/sdk/client/index.js';
import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js';
import { getWorkerPort, getWorkerHost } from '../shared/worker-utils.js';
import { DATA_DIR, DB_PATH, ensureDir } from '../shared/paths.js';
import { HOOK_TIMEOUTS } from '../shared/hook-constants.js';
import { SettingsDefaultsManager } from '../shared/SettingsDefaultsManager.js';
import { getAuthMethodDescription } from '../shared/EnvManager.js';
@@ -46,6 +49,13 @@ import { performGracefulShutdown } from './infrastructure/GracefulShutdown.js';
import { adoptMergedWorktrees, adoptMergedWorktreesForAllKnownRepos } from './infrastructure/WorktreeAdoption.js';
import { Server } from './server/Server.js';
import { BetterAuthRoutes } from '../server/auth/BetterAuthRoutes.js';
import {
createServerApiKey,
listServerApiKeys,
revokeServerApiKey,
} from '../server/auth/api-key-service.js';
import { ServerV1Routes } from '../server/routes/v1/ServerV1Routes.js';
import {
updateCursorContextForProject,
@@ -196,6 +206,12 @@ export class WorkerService implements WorkerRef {
: null,
};
},
getQueueHealth: () => this.sessionManager.isBullMqQueueEnabled()
? this.sessionManager.getQueueHealth()
: null,
preBodyParserRoutes: [
new BetterAuthRoutes(() => this.dbManager.getConnection()),
],
});
this.registerRoutes();
@@ -224,7 +240,7 @@ export class WorkerService implements WorkerRef {
next();
});
this.server.app.use('/api', async (req, res, next) => {
this.server.app.use(['/api', '/v1'], async (req, res, next) => {
if (req.path === '/chroma/status' || req.path === '/health' || req.path === '/readiness' || req.path === '/version') {
next();
return;
@@ -253,6 +269,9 @@ export class WorkerService implements WorkerRef {
this.server.registerRoutes(new SettingsRoutes(this.settingsManager));
this.server.registerRoutes(new LogsRoutes());
this.server.registerRoutes(new MemoryRoutes(this.dbManager, 'claude-mem'));
this.server.registerRoutes(new ServerV1Routes({
getDatabase: () => this.dbManager.getConnection(),
}));
}
async start(): Promise<void> {
@@ -260,6 +279,7 @@ export class WorkerService implements WorkerRef {
const host = getWorkerHost();
await startSupervisor();
await this.sessionManager.initializeQueueEngine();
await this.server.listen(port, host);
@@ -705,14 +725,14 @@ export class WorkerService implements WorkerRef {
}
}
this.completionHandler.finalizeSession(sessionDbId);
await this.completionHandler.finalizeSession(sessionDbId);
this.sessionManager.removeSessionImmediate(sessionDbId);
}
private terminateSession(sessionDbId: number, reason: string): void {
private async terminateSession(sessionDbId: number, reason: string): Promise<void> {
logger.info('SYSTEM', 'Session terminated', { sessionId: sessionDbId, reason });
this.completionHandler.finalizeSession(sessionDbId);
await this.completionHandler.finalizeSession(sessionDbId);
this.sessionManager.removeSessionImmediate(sessionDbId);
}
@@ -734,21 +754,23 @@ export class WorkerService implements WorkerRef {
}
broadcastProcessingStatus(): void {
const queueDepth = this.sessionManager.getTotalActiveWork();
const isProcessing = queueDepth > 0;
const activeSessions = this.sessionManager.getActiveSessionCount();
void (async () => {
const queueDepth = await this.sessionManager.getTotalActiveWork();
const isProcessing = queueDepth > 0;
const activeSessions = this.sessionManager.getActiveSessionCount();
logger.info('WORKER', 'Broadcasting processing status', {
isProcessing,
queueDepth,
activeSessions
});
logger.info('WORKER', 'Broadcasting processing status', {
isProcessing,
queueDepth,
activeSessions
});
this.sseBroadcaster.broadcast({
type: 'processing_status',
isProcessing,
queueDepth
});
this.sseBroadcaster.broadcast({
type: 'processing_status',
isProcessing,
queueDepth
});
})();
}
}
@@ -756,11 +778,174 @@ export async function ensureWorkerStarted(port: number): Promise<WorkerStartResu
return ensureWorkerStartedShared(port, __filename);
}
type ParsedWorkerCommand = {
command: string | undefined;
args: string[];
};
function parseWorkerServiceCommand(argv: string[]): ParsedWorkerCommand {
const [rawCommand, maybeSubCommand, ...rest] = argv;
if (rawCommand === 'server') {
const lifecycleCommands = new Set(['start', 'stop', 'restart', 'status']);
if (maybeSubCommand && lifecycleCommands.has(maybeSubCommand)) {
return { command: `server-${maybeSubCommand}`, args: rest };
}
const serverCommands = new Set(['logs', 'doctor', 'migrate', 'export', 'import', 'api-key']);
return {
command: maybeSubCommand && serverCommands.has(maybeSubCommand) ? `server-${maybeSubCommand}` : 'server-help',
args: rest,
};
}
if (rawCommand === 'worker') {
const workerAliases = new Set(['start', 'stop', 'restart', 'status']);
return {
command: maybeSubCommand && workerAliases.has(maybeSubCommand) ? maybeSubCommand : 'worker-help',
args: rest,
};
}
return {
command: rawCommand,
args: maybeSubCommand === undefined ? [] : [maybeSubCommand, ...rest],
};
}
function printServerCommandUnsupported(command: string): never {
console.error(`Server command not implemented yet: ${command}`);
console.error('This worker bundle accepts the CLI route, but no backend API exists for it yet.');
process.exit(1);
}
function printServerCommandHelp(): never {
console.error('Usage: worker-service server <command>');
console.error('Commands: start, stop, restart, status, logs, doctor, migrate, export, import, api-key create|list|revoke');
process.exit(1);
}
function printWorkerAliasHelp(): never {
console.error('Usage: worker-service worker start|stop|restart|status');
process.exit(1);
}
function runServerBetaServiceCli(command: string): void {
const serverBetaScript = path.join(__dirname, 'server-beta-service.cjs');
if (!existsSync(serverBetaScript)) {
console.error(`Server beta script not found at: ${serverBetaScript}`);
console.error('Rebuild or reinstall claude-mem so server-beta-service.cjs is available.');
process.exit(1);
}
const child = spawn(process.execPath, [serverBetaScript, command], {
stdio: 'inherit',
env: process.env,
});
child.on('error', (error) => {
console.error(`Failed to start server beta command: ${error.message}`);
process.exit(1);
});
child.on('close', (exitCode) => {
process.exit(exitCode ?? 0);
});
}
function parseServerApiKeyOptions(args: string[]): Record<string, string> {
const options: Record<string, string> = {};
for (let i = 0; i < args.length; i++) {
const item = args[i];
if (!item.startsWith('--')) {
continue;
}
const key = item.slice(2);
const next = args[i + 1];
if (!next || next.startsWith('--')) {
options[key] = 'true';
continue;
}
options[key] = next;
i++;
}
return options;
}
function openServerCommandDatabase(): Database {
ensureDir(DATA_DIR);
return new Database(DB_PATH, { create: true, readwrite: true });
}
function runServerApiKeyCli(args: string[]): never {
const subCommand = args[0];
const options = parseServerApiKeyOptions(args.slice(1));
const db = openServerCommandDatabase();
try {
if (subCommand === 'create') {
const scopes = (options.scope ?? options.scopes ?? 'memories:read')
.split(',')
.map(scope => scope.trim())
.filter(Boolean);
const created = createServerApiKey(db, {
name: options.name ?? 'server-api-key',
teamId: options.team ?? null,
projectId: options.project ?? null,
scopes,
});
console.log(JSON.stringify({
id: created.record.id,
key: created.rawKey,
name: created.record.name,
teamId: created.record.teamId,
projectId: created.record.projectId,
scopes: created.record.scopes,
}, null, 2));
process.exit(0);
}
if (subCommand === 'list') {
console.log(JSON.stringify(listServerApiKeys(db).map(key => ({
id: key.id,
name: key.name,
prefix: key.prefix,
teamId: key.teamId,
projectId: key.projectId,
scopes: key.scopes,
status: key.status,
lastUsedAtEpoch: key.lastUsedAtEpoch,
expiresAtEpoch: key.expiresAtEpoch,
createdAtEpoch: key.createdAtEpoch,
})), null, 2));
process.exit(0);
}
if (subCommand === 'revoke') {
const id = args[1];
if (!id) {
console.error('Usage: worker-service server api-key revoke <id>');
process.exit(1);
}
const revoked = revokeServerApiKey(db, id);
if (!revoked) {
console.error(`API key not found: ${id}`);
process.exit(1);
}
console.log(JSON.stringify({ id: revoked.id, status: revoked.status }, null, 2));
process.exit(0);
}
console.error(`Unknown server api-key subcommand: ${subCommand ?? '(none)'}`);
console.error('Usage: worker-service server api-key create|list|revoke');
process.exit(1);
} finally {
db.close();
}
}
async function main() {
const command = process.argv[2];
const { command, args: commandArgs } = parseWorkerServiceCommand(process.argv.slice(2));
const hookInitiatedCommands = ['start', 'hook', 'restart', '--daemon'];
if ((hookInitiatedCommands.includes(command) || command === undefined) && isPluginDisabledInClaudeSettings()) {
if ((command === undefined || hookInitiatedCommands.includes(command)) && isPluginDisabledInClaudeSettings()) {
process.exit(0);
}
@@ -822,6 +1007,7 @@ async function main() {
console.log(` PID: ${pidInfo.pid}`);
console.log(` Port: ${pidInfo.port}`);
console.log(` Started: ${pidInfo.startedAt}`);
await printQueueStatusIfBullMq(port);
} else {
console.log('Worker is not running');
}
@@ -829,6 +1015,44 @@ async function main() {
break;
}
case 'server-start':
case 'server-stop':
case 'server-restart':
case 'server-status': {
runServerBetaServiceCli(command.slice('server-'.length));
break;
}
case 'server-logs':
case 'server-doctor':
case 'server-migrate':
case 'server-export':
case 'server-import': {
printServerCommandUnsupported(command.replace('-', ' '));
break;
}
case 'server-api-key': {
const apiKeyCommand = commandArgs[0];
if (apiKeyCommand === 'create' || apiKeyCommand === 'list' || apiKeyCommand === 'revoke') {
runServerApiKeyCli(commandArgs);
}
console.error(`Unknown server api-key subcommand: ${apiKeyCommand ?? '(none)'}`);
console.error('Usage: worker-service server api-key create|list|revoke');
process.exit(1);
break;
}
case 'server-help': {
printServerCommandHelp();
break;
}
case 'worker-help': {
printWorkerAliasHelp();
break;
}
case 'cursor': {
const subcommand = process.argv[3];
const cursorResult = await handleCursorCommand(subcommand, process.argv.slice(4));
@@ -978,6 +1202,40 @@ async function main() {
}
}
async function printQueueStatusIfBullMq(port: number): Promise<void> {
if (SettingsDefaultsManager.get('CLAUDE_MEM_QUEUE_ENGINE').trim().toLowerCase() !== 'bullmq') {
return;
}
try {
const response = await fetch(`http://${getWorkerHost()}:${port}/api/health`);
if (!response.ok) {
console.log(` Queue: BullMQ health unavailable (HTTP ${response.status})`);
return;
}
const body = await response.json() as {
queue?: {
redis?: {
status?: string;
host?: string;
port?: number;
mode?: string;
prefix?: string;
error?: string;
};
};
};
const redis = body.queue?.redis;
if (!redis) {
return;
}
const target = `${redis.host ?? 'unknown'}:${redis.port ?? 'unknown'}`;
const suffix = redis.status === 'ok' ? '' : ` (${redis.error ?? 'unhealthy'})`;
console.log(` Queue: BullMQ Redis ${redis.status ?? 'unknown'} at ${target} [${redis.mode ?? 'external'}, prefix=${redis.prefix ?? 'claude_mem'}]${suffix}`);
} catch (error) {
console.log(` Queue: BullMQ health unavailable (${error instanceof Error ? error.message : String(error)})`);
}
}
const isMainModule = typeof require !== 'undefined' && typeof module !== 'undefined'
? require.main === module || !module.parent || process.env.CLAUDE_MEM_MANAGED === 'true'
: import.meta.url === `file://${process.argv[1]}`
+1
View File
@@ -22,6 +22,7 @@ export interface ActiveSession {
cumulativeInputTokens: number;
cumulativeOutputTokens: number;
earliestPendingTimestamp: number | null;
claimedMessageIds: number[];
conversationHistory: ConversationMessage[];
currentProvider: 'claude' | 'gemini' | 'openrouter' | null;
consecutiveRestarts: number;
+7
View File
@@ -65,6 +65,13 @@ export class DatabaseManager {
return this.chromaSync;
}
getConnection(): Database {
if (!this.db) {
throw new Error('Database not initialized');
}
return this.db;
}
getSessionById(sessionDbId: number): {
id: number;
content_session_id: string;
+102 -58
View File
@@ -1,10 +1,14 @@
import { EventEmitter } from 'events';
import { DatabaseManager } from './DatabaseManager.js';
import { logger } from '../../utils/logger.js';
import type { ActiveSession, PendingMessage, PendingMessageWithId, ObservationData } from '../worker-types.js';
import { PendingMessageStore } from '../sqlite/PendingMessageStore.js';
import { SessionQueueProcessor } from '../queue/SessionQueueProcessor.js';
import {
SqliteObservationQueueEngine,
type HealthCheckedObservationQueueEngine,
type InspectableObservationQueueEngine,
type ObservationQueueHealth
} from '../../server/queue/ObservationQueueEngine.js';
import { BullMqObservationQueueEngine } from '../../server/queue/BullMqObservationQueueEngine.js';
import { getObservationQueueEngineName } from '../../server/queue/redis-config.js';
import { getSdkProcessForSession, ensureSdkProcessExit } from '../../supervisor/process-registry.js';
import { getSupervisor } from '../../supervisor/index.js';
import { RestartGuard } from './RestartGuard.js';
@@ -12,24 +16,55 @@ import { RestartGuard } from './RestartGuard.js';
export class SessionManager {
private dbManager: DatabaseManager;
private sessions: Map<number, ActiveSession> = new Map();
private sessionQueues: Map<number, EventEmitter> = new Map();
private onSessionDeletedCallback?: () => void;
private pendingStore: PendingMessageStore | null = null;
private queueEngine: InspectableObservationQueueEngine | null = null;
private queueEngineName: 'sqlite' | 'bullmq' | null = null;
private onPendingMutate?: () => void;
constructor(dbManager: DatabaseManager) {
this.dbManager = dbManager;
}
private getPendingStore(): PendingMessageStore {
if (!this.pendingStore) {
const sessionStore = this.dbManager.getSessionStore();
this.pendingStore = new PendingMessageStore(
sessionStore.db,
() => this.onPendingMutate?.()
);
private getQueueEngine(): InspectableObservationQueueEngine {
if (!this.queueEngine) {
this.queueEngineName = getObservationQueueEngineName();
if (this.queueEngineName === 'bullmq') {
this.queueEngine = new BullMqObservationQueueEngine({
onMutate: () => this.onPendingMutate?.()
});
} else {
const sessionStore = this.dbManager.getSessionStore();
this.queueEngine = new SqliteObservationQueueEngine(
sessionStore.db,
() => this.onPendingMutate?.()
);
}
}
return this.pendingStore;
return this.queueEngine;
}
async initializeQueueEngine(): Promise<void> {
this.queueEngineName = getObservationQueueEngineName();
if (this.queueEngineName === 'sqlite') {
return;
}
const queue = this.getQueueEngine();
if (isHealthCheckedQueue(queue)) {
await queue.assertHealthy();
await queue.getTotalQueueDepth();
}
}
isBullMqQueueEnabled(): boolean {
return (this.queueEngineName ?? getObservationQueueEngineName()) === 'bullmq';
}
async getQueueHealth(): Promise<ObservationQueueHealth | null> {
const queue = this.getQueueEngine();
if (isHealthCheckedQueue(queue)) {
return queue.getHealth();
}
return null;
}
setOnSessionDeleted(callback: () => void): void {
@@ -134,6 +169,7 @@ export class SessionManager {
cumulativeInputTokens: 0,
cumulativeOutputTokens: 0,
earliestPendingTimestamp: null,
claimedMessageIds: [],
conversationHistory: [], // Initialize empty - will be populated by agents
currentProvider: null, // Will be set when generator starts
consecutiveRestarts: 0, // DEPRECATED: use restartGuard. Kept for logging compat.
@@ -153,9 +189,6 @@ export class SessionManager {
this.sessions.set(sessionDbId, session);
const emitter = new EventEmitter();
this.sessionQueues.set(sessionDbId, emitter);
logger.info('SESSION', 'Session initialized', {
sessionId: sessionDbId,
project: session.project,
@@ -171,7 +204,7 @@ export class SessionManager {
return this.sessions.get(sessionDbId);
}
queueObservation(sessionDbId: number, data: ObservationData): void {
async queueObservation(sessionDbId: number, data: ObservationData): Promise<void> {
let session = this.sessions.get(sessionDbId);
if (!session) {
session = this.initializeSession(sessionDbId);
@@ -190,8 +223,9 @@ export class SessionManager {
};
try {
const messageId = this.getPendingStore().enqueue(sessionDbId, session.contentSessionId, message);
const queueDepth = this.getPendingStore().getPendingCount(sessionDbId);
const queue = this.getQueueEngine();
const messageId = await queue.enqueue(sessionDbId, session.contentSessionId, message);
const queueDepth = await queue.getPendingCount(sessionDbId);
const toolSummary = logger.formatTool(data.tool_name, data.tool_input);
if (messageId === 0) {
logger.debug('QUEUE', `DUP_SUPPRESSED | sessionDbId=${sessionDbId} | type=observation | tool=${toolSummary} | toolUseId=${data.toolUseId ?? 'null'} | depth=${queueDepth}`, {
@@ -212,11 +246,9 @@ export class SessionManager {
throw normalized;
}
const emitter = this.sessionQueues.get(sessionDbId);
emitter?.emit('message');
}
queueSummarize(sessionDbId: number, lastAssistantMessage?: string): void {
async queueSummarize(sessionDbId: number, lastAssistantMessage?: string): Promise<void> {
let session = this.sessions.get(sessionDbId);
if (!session) {
session = this.initializeSession(sessionDbId);
@@ -228,8 +260,9 @@ export class SessionManager {
};
try {
const messageId = this.getPendingStore().enqueue(sessionDbId, session.contentSessionId, message);
const queueDepth = this.getPendingStore().getPendingCount(sessionDbId);
const queue = this.getQueueEngine();
const messageId = await queue.enqueue(sessionDbId, session.contentSessionId, message);
const queueDepth = await queue.getPendingCount(sessionDbId);
if (messageId === 0) {
logger.debug('QUEUE', `DUP_SUPPRESSED | sessionDbId=${sessionDbId} | type=summarize | depth=${queueDepth}`, {
sessionId: sessionDbId
@@ -252,13 +285,32 @@ export class SessionManager {
throw error;
}
const emitter = this.sessionQueues.get(sessionDbId);
emitter?.emit('message');
}
clearPendingForSession(sessionDbId: number): void {
this.getPendingStore().clearPendingForSession(sessionDbId);
this.sessionQueues.get(sessionDbId)?.emit('message');
async clearPendingForSession(sessionDbId: number): Promise<number> {
return await this.getQueueEngine().clearPendingForSession(sessionDbId);
}
async resetProcessingToPending(sessionDbId: number): Promise<number> {
const session = this.sessions.get(sessionDbId);
if (session) {
session.claimedMessageIds = [];
}
return await this.getQueueEngine().resetProcessingToPending(sessionDbId);
}
async confirmClaimedMessages(sessionDbId: number): Promise<number> {
const session = this.sessions.get(sessionDbId);
const claimedIds = session?.claimedMessageIds ?? [];
let confirmed = 0;
for (const messageId of claimedIds) {
confirmed += await this.getQueueEngine().confirmProcessed(messageId);
}
if (session) {
session.claimedMessageIds = [];
session.earliestPendingTimestamp = null;
}
return confirmed;
}
async deleteSession(sessionDbId: number): Promise<void> {
@@ -314,8 +366,6 @@ export class SessionManager {
}
this.sessions.delete(sessionDbId);
this.sessionQueues.delete(sessionDbId);
logger.info('SESSION', 'Session deleted', {
sessionId: sessionDbId,
duration: `${(sessionDuration / 1000).toFixed(1)}s`,
@@ -337,8 +387,6 @@ export class SessionManager {
}
this.sessions.delete(sessionDbId);
this.sessionQueues.delete(sessionDbId);
logger.info('SESSION', 'Session removed from active sessions', {
sessionId: sessionDbId,
project: session.project
@@ -352,31 +400,28 @@ export class SessionManager {
async shutdownAll(): Promise<void> {
const sessionIds = Array.from(this.sessions.keys());
await Promise.all(sessionIds.map(id => this.deleteSession(id)));
await this.queueEngine?.close();
this.queueEngine = null;
}
hasPendingMessages(): boolean {
return this.getTotalQueueDepth() > 0;
async hasPendingMessages(): Promise<boolean> {
return (await this.getTotalQueueDepth()) > 0;
}
getActiveSessionCount(): number {
return this.sessions.size;
}
getTotalQueueDepth(): number {
const stmt = this.dbManager.getSessionStore().db.prepare(`
SELECT COUNT(*) as count FROM pending_messages
WHERE status IN ('pending', 'processing')
`);
const result = stmt.get() as { count: number };
return result.count;
async getTotalQueueDepth(): Promise<number> {
return await this.getQueueEngine().getTotalQueueDepth();
}
getTotalActiveWork(): number {
return this.getTotalQueueDepth();
async getTotalActiveWork(): Promise<number> {
return await this.getTotalQueueDepth();
}
isAnySessionProcessing(): boolean {
return this.getTotalQueueDepth() > 0;
async isAnySessionProcessing(): Promise<boolean> {
return (await this.getTotalQueueDepth()) > 0;
}
async *getMessageIterator(sessionDbId: number): AsyncIterableIterator<PendingMessageWithId> {
@@ -385,16 +430,10 @@ export class SessionManager {
session = this.initializeSession(sessionDbId);
}
const emitter = this.sessionQueues.get(sessionDbId);
if (!emitter) {
throw new Error(`No emitter for session ${sessionDbId}`);
}
const queue = this.getQueueEngine();
await this.resetProcessingToPending(sessionDbId);
this.getPendingStore().resetProcessingToPending(sessionDbId);
const processor = new SessionQueueProcessor(this.getPendingStore(), emitter);
for await (const message of processor.createIterator({
for await (const message of queue.createIterator({
sessionDbId,
signal: session.abortController.signal,
onIdleTimeout: () => {
@@ -404,6 +443,7 @@ export class SessionManager {
session.abortController.abort();
}
})) {
session.claimedMessageIds.push(message._persistentId);
if (session.earliestPendingTimestamp === null) {
session.earliestPendingTimestamp = message._originalTimestamp;
} else {
@@ -416,7 +456,11 @@ export class SessionManager {
}
}
getPendingMessageStore(): PendingMessageStore {
return this.getPendingStore();
getPendingMessageStore(): InspectableObservationQueueEngine {
return this.getQueueEngine();
}
}
function isHealthCheckedQueue(queue: InspectableObservationQueueEngine): queue is HealthCheckedObservationQueueEngine {
return 'getHealth' in queue && 'assertHealthy' in queue;
}
@@ -41,7 +41,7 @@ export async function processAgentResponse(
// Plain-text skip responses are intentionally ignored. Re-queueing them
// creates an observer loop where the same low-signal batch is retried
// until the restart guard fires or the provider quota is exhausted.
sessionManager.clearPendingForSession(session.sessionDbId);
await sessionManager.confirmClaimedMessages(session.sessionDbId);
session.earliestPendingTimestamp = null;
return;
}
@@ -53,7 +53,7 @@ export async function processAgentResponse(
// Reset any claimed-but-undelivered messages back to pending so they don't
// count as "in progress" and trigger a respawn loop while we wait for the
// memory session id to appear. The next generator pass will re-claim them.
sessionManager.getPendingMessageStore().resetProcessingToPending(session.sessionDbId);
await sessionManager.resetProcessingToPending(session.sessionDbId);
return;
}
@@ -99,7 +99,7 @@ export async function processAgentResponse(
session.lastSummaryStored = result.summaryId !== null;
if (summary && (summary.skipped || session.lastSummaryStored)) {
ingestSummary({
await ingestSummary({
kind: 'parsed',
sessionDbId: session.sessionDbId,
messageId: -1,
@@ -108,9 +108,10 @@ export async function processAgentResponse(
});
}
sessionManager.clearPendingForSession(session.sessionDbId);
await sessionManager.confirmClaimedMessages(session.sessionDbId);
session.earliestPendingTimestamp = null;
session.restartGuard?.recordSuccess();
worker?.broadcastProcessingStatus?.();
void notifyTelegram({
observations: labeledObservations,
@@ -182,6 +183,14 @@ async function syncAndBroadcastObservations(
for (const obsId of uniqueObservationIds) {
const observationIndex = result.observationIds.indexOf(obsId);
const obs = observations[observationIndex];
if (!obs) {
logger.warn('DB', `${agentName} storage returned observation id without matching parsed observation`, {
sessionId: session.sessionDbId,
obsId,
observationIndex
});
continue;
}
const chromaStart = Date.now();
dbManager.getChromaSync()?.syncObservation(
@@ -8,4 +8,5 @@ export function cleanupProcessedMessages(
worker: WorkerRef | undefined
): void {
session.earliestPendingTimestamp = null;
worker?.broadcastProcessingStatus?.();
}
+23 -16
View File
@@ -6,26 +6,16 @@ import { getPackageRoot } from '../../../shared/paths.js';
import { logger } from '../../../utils/logger.js';
export function createMiddleware(
summarizeRequestBody: (method: string, path: string, body: any) => string
summarizeRequestBody: (method: string, path: string, body: any) => string,
options: { includeCors?: boolean } = {}
): RequestHandler[] {
const middlewares: RequestHandler[] = [];
middlewares.push(express.json({ limit: '5mb' }));
if (options.includeCors !== false) {
middlewares.push(createCorsMiddleware());
}
middlewares.push(cors({
origin: (origin, callback) => {
if (!origin ||
origin.startsWith('http://localhost:') ||
origin.startsWith('http://127.0.0.1:')) {
callback(null, true);
} else {
callback(new Error('CORS not allowed'));
}
},
methods: ['GET', 'HEAD', 'POST', 'PUT', 'PATCH', 'DELETE'],
allowedHeaders: ['Content-Type', 'X-Requested-With'],
credentials: false
}));
middlewares.push(express.json({ limit: '5mb' }));
middlewares.push((req: Request, res: Response, next: NextFunction) => {
const staticExtensions = ['.html', '.js', '.css', '.svg', '.png', '.jpg', '.jpeg', '.webp', '.woff', '.woff2', '.ttf', '.eot'];
@@ -58,6 +48,23 @@ export function createMiddleware(
return middlewares;
}
export function createCorsMiddleware(): RequestHandler {
return cors({
origin: (origin, callback) => {
if (!origin ||
origin.startsWith('http://localhost:') ||
origin.startsWith('http://127.0.0.1:')) {
callback(null, true);
} else {
callback(new Error('CORS not allowed'));
}
},
methods: ['GET', 'HEAD', 'POST', 'PUT', 'PATCH', 'DELETE'],
allowedHeaders: ['Content-Type', 'Authorization', 'X-Requested-With'],
credentials: false
});
}
export function requireLocalhost(req: Request, res: Response, next: NextFunction): void {
const clientIp = req.ip || req.connection.remoteAddress || '';
const isLocalhost =
@@ -271,15 +271,15 @@ export class DataRoutes extends BaseRouteHandler {
res.json(store.getProjectCatalog());
});
private handleGetProcessingStatus = this.wrapHandler((req: Request, res: Response): void => {
const isProcessing = this.sessionManager.isAnySessionProcessing();
const queueDepth = this.sessionManager.getTotalActiveWork();
private handleGetProcessingStatus = this.wrapHandler(async (req: Request, res: Response): Promise<void> => {
const isProcessing = await this.sessionManager.isAnySessionProcessing();
const queueDepth = await this.sessionManager.getTotalActiveWork();
res.json({ isProcessing, queueDepth });
});
private handleSetProcessing = this.wrapHandler((req: Request, res: Response): void => {
const isProcessing = this.sessionManager.isAnySessionProcessing();
const queueDepth = this.sessionManager.getTotalQueueDepth();
private handleSetProcessing = this.wrapHandler(async (req: Request, res: Response): Promise<void> => {
const isProcessing = await this.sessionManager.isAnySessionProcessing();
const queueDepth = await this.sessionManager.getTotalQueueDepth();
const activeSessions = this.sessionManager.getActiveSessionCount();
res.json({ status: 'ok', isProcessing, queueDepth, activeSessions });
@@ -65,15 +65,15 @@ export class SessionRoutes extends BaseRouteHandler {
return (isGeminiSelected() && isGeminiAvailable()) ? 'gemini' : 'claude';
}
public ensureGeneratorRunning(sessionDbId: number, source: string): void {
public async ensureGeneratorRunning(sessionDbId: number, source: string): Promise<void> {
const session = this.sessionManager.getSession(sessionDbId);
if (!session) return;
const selectedProvider = this.getSelectedProvider();
if (!session.generatorPromise) {
this.applyTierRouting(session);
this.startGeneratorWithProvider(session, selectedProvider, source);
await this.applyTierRouting(session);
await this.startGeneratorWithProvider(session, selectedProvider, source);
return;
}
@@ -89,11 +89,11 @@ export class SessionRoutes extends BaseRouteHandler {
}
}
private startGeneratorWithProvider(
private async startGeneratorWithProvider(
session: ReturnType<typeof this.sessionManager.getSession>,
provider: 'claude' | 'gemini' | 'openrouter',
source: string
): void {
): Promise<void> {
if (!session) return;
if (session.abortController.signal.aborted) {
@@ -107,7 +107,7 @@ export class SessionRoutes extends BaseRouteHandler {
const agentName = provider === 'openrouter' ? 'OpenRouter' : (provider === 'gemini' ? 'Gemini' : 'Claude SDK');
const pendingStore = this.sessionManager.getPendingMessageStore();
const actualQueueDepth = pendingStore.getPendingCount(session.sessionDbId);
const actualQueueDepth = await pendingStore.getPendingCount(session.sessionDbId);
logger.info('SESSION', `Generator auto-starting (${source}) using ${agentName}`, {
sessionId: session.sessionDbId,
@@ -121,7 +121,7 @@ export class SessionRoutes extends BaseRouteHandler {
const myController = session.abortController;
session.generatorPromise = agent.startSession(session, this.workerService)
.catch(error => {
.catch(async error => {
if (myController.signal.aborted) {
logger.debug('HTTP', 'Generator catch: ignoring error after abort', { sessionId: session.sessionDbId });
return;
@@ -145,9 +145,8 @@ export class SessionRoutes extends BaseRouteHandler {
error: errorMsg
}, error);
const pendingStore = this.sessionManager.getPendingMessageStore();
try {
const reset = pendingStore.resetProcessingToPending(session.sessionDbId);
const reset = await this.sessionManager.resetProcessingToPending(session.sessionDbId);
if (reset > 0) {
logger.warn('SESSION', `Reset processing messages after generator error`, {
sessionId: session.sessionDbId,
@@ -168,8 +167,10 @@ export class SessionRoutes extends BaseRouteHandler {
sessionManager: this.sessionManager,
completionHandler: this.completionHandler,
restartGenerator: (s, source) => {
this.applyTierRouting(s);
this.startGeneratorWithProvider(s, this.getSelectedProvider(), source);
void (async () => {
await this.applyTierRouting(s);
await this.startGeneratorWithProvider(s, this.getSelectedProvider(), source);
})();
},
});
});
@@ -222,7 +223,7 @@ export class SessionRoutes extends BaseRouteHandler {
platformSource: z.string().optional(),
}).passthrough();
private handleObservationsByClaudeId = this.wrapHandler((req: Request, res: Response): void => {
private handleObservationsByClaudeId = this.wrapHandler(async (req: Request, res: Response): Promise<void> => {
const {
contentSessionId,
tool_name,
@@ -236,7 +237,7 @@ export class SessionRoutes extends BaseRouteHandler {
toolUseId,
} = req.body;
const result = ingestObservation({
const result = await ingestObservation({
contentSessionId,
toolName: tool_name,
toolInput: tool_input,
@@ -261,7 +262,7 @@ export class SessionRoutes extends BaseRouteHandler {
res.json({ status: 'queued' });
});
private handleSummarizeByClaudeId = this.wrapHandler((req: Request, res: Response): void => {
private handleSummarizeByClaudeId = this.wrapHandler(async (req: Request, res: Response): Promise<void> => {
const { contentSessionId, last_assistant_message, agentId } = req.body;
const platformSource = normalizePlatformSource(req.body.platformSource);
@@ -290,16 +291,16 @@ export class SessionRoutes extends BaseRouteHandler {
const cleanedLastAssistantMessage = last_assistant_message
? stripMemoryTagsFromPrompt(String(last_assistant_message))
: last_assistant_message;
this.sessionManager.queueSummarize(sessionDbId, cleanedLastAssistantMessage);
await this.sessionManager.queueSummarize(sessionDbId, cleanedLastAssistantMessage);
this.ensureGeneratorRunning(sessionDbId, 'summarize');
await this.ensureGeneratorRunning(sessionDbId, 'summarize');
this.eventBroadcaster.broadcastSummarizeQueued();
res.json({ status: 'queued' });
});
private handleStatusByClaudeId = this.wrapHandler((req: Request, res: Response): void => {
private handleStatusByClaudeId = this.wrapHandler(async (req: Request, res: Response): Promise<void> => {
const contentSessionId = req.query.contentSessionId as string;
if (!contentSessionId) {
@@ -316,7 +317,7 @@ export class SessionRoutes extends BaseRouteHandler {
}
const pendingStore = this.sessionManager.getPendingMessageStore();
const queueLength = pendingStore.getPendingCount(sessionDbId);
const queueLength = await pendingStore.getPendingCount(sessionDbId);
res.json({
status: 'active',
@@ -327,7 +328,7 @@ export class SessionRoutes extends BaseRouteHandler {
});
});
private handleSessionInitByClaudeId = this.wrapHandler((req: Request, res: Response): void => {
private handleSessionInitByClaudeId = this.wrapHandler(async (req: Request, res: Response): Promise<void> => {
const { contentSessionId } = req.body;
const project = req.body.project || 'unknown';
@@ -458,7 +459,7 @@ export class SessionRoutes extends BaseRouteHandler {
});
}
this.ensureGeneratorRunning(sessionDbId, 'init');
await this.ensureGeneratorRunning(sessionDbId, 'init');
this.eventBroadcaster.broadcastSessionStarted(sessionDbId, session.project);
} else {
@@ -478,7 +479,7 @@ export class SessionRoutes extends BaseRouteHandler {
'Read', 'Glob', 'Grep', 'LS', 'ListMcpResourcesTool'
]);
private applyTierRouting(session: NonNullable<ReturnType<typeof this.sessionManager.getSession>>): void {
private async applyTierRouting(session: NonNullable<ReturnType<typeof this.sessionManager.getSession>>): Promise<void> {
const settings = SettingsDefaultsManager.loadFromFile(USER_SETTINGS_PATH);
if (settings.CLAUDE_MEM_TIER_ROUTING_ENABLED === 'false') {
session.modelOverride = undefined;
@@ -488,7 +489,7 @@ export class SessionRoutes extends BaseRouteHandler {
session.modelOverride = undefined;
const pendingStore = this.sessionManager.getPendingMessageStore();
const pending = pendingStore.peekPendingTypes(session.sessionDbId);
const pending = await pendingStore.peekPendingTypes(session.sessionDbId);
if (pending.length === 0) {
session.modelOverride = undefined;
@@ -97,12 +97,20 @@ export class ViewerRoutes extends BaseRouteHandler {
timestamp: Date.now()
});
const isProcessing = this.sessionManager.isAnySessionProcessing();
const queueDepth = this.sessionManager.getTotalActiveWork();
this.sseBroadcaster.broadcast({
type: 'processing_status',
isProcessing,
queueDepth
});
void (async () => {
try {
const isProcessing = await this.sessionManager.isAnySessionProcessing();
const queueDepth = await this.sessionManager.getTotalActiveWork();
this.sseBroadcaster.broadcast({
type: 'processing_status',
isProcessing,
queueDepth
});
} catch (error) {
logger.warn('HTTP', 'Failed to broadcast initial processing status', {
error: error instanceof Error ? error.message : String(error),
});
}
})();
});
}
+8 -8
View File
@@ -55,7 +55,7 @@ interface IngestContext {
sessionManager: SessionManager;
dbManager: DatabaseManager;
eventBroadcaster: SessionEventBroadcaster;
ensureGeneratorRunning?: (sessionDbId: number, source: string) => void;
ensureGeneratorRunning?: (sessionDbId: number, source: string) => void | Promise<void>;
}
let ctx: IngestContext | null = null;
@@ -65,7 +65,7 @@ export function setIngestContext(next: IngestContext): void {
}
export function attachIngestGeneratorStarter(
ensureGeneratorRunning: (sessionDbId: number, source: string) => void,
ensureGeneratorRunning: (sessionDbId: number, source: string) => void | Promise<void>,
): void {
requireContext().ensureGeneratorRunning = ensureGeneratorRunning;
}
@@ -94,7 +94,7 @@ export interface ObservationPayload {
toolUseId?: string;
}
export function ingestObservation(payload: ObservationPayload): IngestResult {
export async function ingestObservation(payload: ObservationPayload): Promise<IngestResult> {
const { sessionManager, dbManager, eventBroadcaster, ensureGeneratorRunning } = requireContext();
const platformSource = normalizePlatformSource(payload.platformSource);
@@ -158,7 +158,7 @@ export function ingestObservation(payload: ObservationPayload): IngestResult {
? stripMemoryTagsFromJson(JSON.stringify(payload.toolResponse))
: '{}';
sessionManager.queueObservation(sessionDbId, {
await sessionManager.queueObservation(sessionDbId, {
tool_name: payload.toolName,
tool_input: cleanedToolInput,
tool_response: cleanedToolResponse,
@@ -175,7 +175,7 @@ export function ingestObservation(payload: ObservationPayload): IngestResult {
toolUseId: typeof payload.toolUseId === 'string' ? payload.toolUseId : undefined,
});
ensureGeneratorRunning?.(sessionDbId, 'observation');
await ensureGeneratorRunning?.(sessionDbId, 'observation');
eventBroadcaster.broadcastObservationQueued(sessionDbId);
return { ok: true, sessionDbId };
@@ -229,7 +229,7 @@ export type SummaryPayload =
parsed: ParsedSummary;
};
export function ingestSummary(payload: SummaryPayload): IngestResult {
export async function ingestSummary(payload: SummaryPayload): Promise<IngestResult> {
if (payload.kind === 'queue') {
const { sessionManager, dbManager, ensureGeneratorRunning } = requireContext();
@@ -249,8 +249,8 @@ export function ingestSummary(payload: SummaryPayload): IngestResult {
return { ok: false, reason: message, status: 500 };
}
sessionManager.queueSummarize(sessionDbId, payload.lastAssistantMessage);
ensureGeneratorRunning?.(sessionDbId, 'summarize');
await sessionManager.queueSummarize(sessionDbId, payload.lastAssistantMessage);
await ensureGeneratorRunning?.(sessionDbId, 'summarize');
return { ok: true, sessionDbId };
}
@@ -8,7 +8,7 @@ import { RestartGuard } from '../RestartGuard.js';
export interface GeneratorExitDependencies {
sessionManager: SessionManager;
completionHandler: SessionCompletionHandler;
restartGenerator: (session: ActiveSession, source: string) => void;
restartGenerator: (session: ActiveSession, source: string) => void | Promise<void>;
}
function isHardStopReason(reason: ActiveSession['abortReason']): boolean {
@@ -50,11 +50,11 @@ export async function handleGeneratorExit(
const pendingStore = sessionManager.getPendingMessageStore();
const terminateSession = (logPrefix: string, clearPending: boolean) => {
const terminateSession = async (logPrefix: string, clearPending: boolean) => {
try {
if (clearPending) {
try {
pendingStore.clearPendingForSession(sessionDbId);
await pendingStore.clearPendingForSession(sessionDbId);
} catch (e) {
const normalized = e instanceof Error ? e : new Error(String(e));
logger.error('SESSION', `${logPrefix} pending cleanup failed; continuing finalization`, {
@@ -64,7 +64,7 @@ export async function handleGeneratorExit(
}
}
try {
completionHandler.finalizeSession(sessionDbId);
await completionHandler.finalizeSession(sessionDbId);
} catch (e) {
const normalized = e instanceof Error ? e : new Error(String(e));
logger.error('SESSION', `${logPrefix} finalization failed; forcing in-memory session removal`, {
@@ -82,26 +82,26 @@ export async function handleGeneratorExit(
sessionId: sessionDbId,
reason
});
terminateSession('Hard-stop', true);
await terminateSession('Hard-stop', true);
return;
}
let pendingCount: number;
try {
pendingCount = pendingStore.getPendingCount(sessionDbId);
pendingCount = await pendingStore.getPendingCount(sessionDbId);
} catch (e) {
const normalized = e instanceof Error ? e : new Error(String(e));
logger.error('SESSION', 'Error during recovery pending-count check; aborting to prevent leaks', {
sessionId: sessionDbId
}, normalized);
terminateSession('Recovery abort', true);
await terminateSession('Recovery abort', true);
return;
}
if (pendingCount === 0) {
session.restartGuard?.recordSuccess();
session.consecutiveRestarts = 0;
terminateSession('Natural completion', false);
await terminateSession('Natural completion', false);
return;
}
@@ -120,7 +120,7 @@ export async function handleGeneratorExit(
maxConsecutiveFailures: session.restartGuard.maxConsecutiveFailures,
});
session.consecutiveRestarts = 0;
terminateSession('Restart guard', true);
await terminateSession('Restart guard', true);
return;
}
@@ -145,7 +145,7 @@ export async function handleGeneratorExit(
session.respawnTimer = undefined;
const stillExists = deps.sessionManager.getSession(sessionDbId);
if (stillExists && !stillExists.generatorPromise) {
restartGenerator(stillExists, 'pending-work-restart');
void restartGenerator(stillExists, 'pending-work-restart');
}
}, backoffMs);
}
@@ -11,7 +11,7 @@ export class SessionCompletionHandler {
private dbManager: DatabaseManager
) {}
finalizeSession(sessionDbId: number): void {
async finalizeSession(sessionDbId: number): Promise<void> {
const sessionStore = this.dbManager.getSessionStore();
const row = sessionStore.getSessionById(sessionDbId);
@@ -28,7 +28,7 @@ export class SessionCompletionHandler {
try {
const pendingStore = this.sessionManager.getPendingMessageStore();
const cleared = pendingStore.clearPendingForSession(sessionDbId);
const cleared = await pendingStore.clearPendingForSession(sessionDbId);
if (cleared > 0) {
logger.warn('SESSION', `Cleared ${cleared} orphaned pending messages on session finalize`, {
sessionId: sessionDbId, cleared
@@ -46,7 +46,7 @@ export class SessionCompletionHandler {
}
async completeByDbId(sessionDbId: number): Promise<void> {
this.finalizeSession(sessionDbId);
await this.finalizeSession(sessionDbId);
await this.sessionManager.deleteSession(sessionDbId);
}
+16
View File
@@ -64,6 +64,14 @@ export interface SettingsDefaults {
CLAUDE_MEM_TELEGRAM_CHAT_ID: string;
CLAUDE_MEM_TELEGRAM_TRIGGER_TYPES: string;
CLAUDE_MEM_TELEGRAM_TRIGGER_CONCEPTS: string;
CLAUDE_MEM_QUEUE_ENGINE: string;
CLAUDE_MEM_REDIS_URL: string;
CLAUDE_MEM_REDIS_HOST: string;
CLAUDE_MEM_REDIS_PORT: string;
CLAUDE_MEM_REDIS_MODE: string;
CLAUDE_MEM_QUEUE_REDIS_PREFIX: string;
CLAUDE_MEM_AUTH_MODE: string;
CLAUDE_MEM_RUNTIME: string;
}
export class SettingsDefaultsManager {
@@ -128,6 +136,14 @@ export class SettingsDefaultsManager {
CLAUDE_MEM_TELEGRAM_CHAT_ID: '',
CLAUDE_MEM_TELEGRAM_TRIGGER_TYPES: 'security_alert',
CLAUDE_MEM_TELEGRAM_TRIGGER_CONCEPTS: '',
CLAUDE_MEM_QUEUE_ENGINE: 'sqlite',
CLAUDE_MEM_REDIS_URL: '',
CLAUDE_MEM_REDIS_HOST: '127.0.0.1',
CLAUDE_MEM_REDIS_PORT: '6379',
CLAUDE_MEM_REDIS_MODE: 'external',
CLAUDE_MEM_QUEUE_REDIS_PREFIX: `claude_mem_${process.env.CLAUDE_MEM_WORKER_PORT ?? String(37700 + ((process.getuid?.() ?? 77) % 100))}`,
CLAUDE_MEM_AUTH_MODE: 'api-key',
CLAUDE_MEM_RUNTIME: 'worker',
};
static getAllDefaults(): SettingsDefaults {
+3
View File
@@ -129,6 +129,9 @@ export function createBackupFilename(originalPath: string): string {
export const paths = {
dataDir: () => DATA_DIR,
workerPid: () => join(DATA_DIR, 'worker.pid'),
serverBetaPid: () => join(DATA_DIR, '.server-beta.pid'),
serverBetaPort: () => join(DATA_DIR, '.server-beta.port'),
serverBetaRuntime: () => join(DATA_DIR, '.server-beta.runtime.json'),
settings: () => join(DATA_DIR, 'settings.json'),
database: () => join(DATA_DIR, 'claude-mem.db'),
chroma: () => join(DATA_DIR, 'chroma'),
+186
View File
@@ -0,0 +1,186 @@
// SPDX-License-Identifier: Apache-2.0
import type { JsonObject, JsonValue, PostgresQueryable } from './utils.js';
import {
assertProjectOwnership,
assertSessionOwnership,
canonicalJson,
deterministicKey,
newId,
queryOne,
toEpoch,
toJsonObject
} from './utils.js';
export interface PostgresAgentEvent {
id: string;
projectId: string;
teamId: string;
serverSessionId: string | null;
sourceAdapter: string;
sourceEventId: string | null;
idempotencyKey: string;
eventType: string;
payload: JsonValue;
metadata: JsonObject;
occurredAtEpoch: number;
receivedAtEpoch: number;
createdAtEpoch: number;
}
export interface CreatePostgresAgentEventInput {
id?: string;
projectId: string;
teamId: string;
serverSessionId?: string | null;
sourceAdapter: string;
sourceEventId?: string | null;
eventType: string;
payload?: JsonValue;
metadata?: JsonObject;
occurredAt: Date | string | number;
}
interface AgentEventRow {
id: string;
project_id: string;
team_id: string;
server_session_id: string | null;
source_adapter: string;
source_event_id: string | null;
idempotency_key: string;
event_type: string;
payload: unknown;
metadata: unknown;
occurred_at: Date;
received_at: Date;
created_at: Date;
}
export class PostgresAgentEventsRepository {
constructor(private client: PostgresQueryable) {}
async create(input: CreatePostgresAgentEventInput): Promise<PostgresAgentEvent> {
await assertProjectOwnership(this.client, input.projectId, input.teamId);
if (input.serverSessionId) {
await assertSessionOwnership(this.client, input.serverSessionId, input.projectId, input.teamId);
}
const idempotencyKey = buildAgentEventIdempotencyKey(input);
const row = await queryOne<AgentEventRow>(
this.client,
`
INSERT INTO agent_events (
id, project_id, team_id, server_session_id, source_adapter,
source_event_id, idempotency_key, event_type, payload, metadata, occurred_at
)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9::jsonb, $10::jsonb, $11)
ON CONFLICT (idempotency_key) DO UPDATE SET
metadata = agent_events.metadata || excluded.metadata
RETURNING *
`,
[
input.id ?? newId(),
input.projectId,
input.teamId,
input.serverSessionId ?? null,
input.sourceAdapter,
input.sourceEventId ?? null,
idempotencyKey,
input.eventType,
JSON.stringify(input.payload ?? {}),
JSON.stringify(input.metadata ?? {}),
new Date(input.occurredAt)
]
);
return mapAgentEventRow(row!);
}
async createMany(inputs: CreatePostgresAgentEventInput[]): Promise<PostgresAgentEvent[]> {
const events: PostgresAgentEvent[] = [];
for (const input of inputs) {
events.push(await this.create(input));
}
return events;
}
async getByIdForScope(input: {
id: string;
projectId: string;
teamId: string;
}): Promise<PostgresAgentEvent | null> {
const row = await queryOne<AgentEventRow>(
this.client,
'SELECT * FROM agent_events WHERE id = $1 AND project_id = $2 AND team_id = $3',
[input.id, input.projectId, input.teamId]
);
return row ? mapAgentEventRow(row) : null;
}
async listByProject(input: {
projectId: string;
teamId: string;
serverSessionId?: string | null;
limit?: number;
}): Promise<PostgresAgentEvent[]> {
const result = await this.client.query<AgentEventRow>(
`
SELECT * FROM agent_events
WHERE project_id = $1
AND team_id = $2
AND ($3::text IS NULL OR server_session_id = $3)
ORDER BY occurred_at DESC
LIMIT $4
`,
[input.projectId, input.teamId, input.serverSessionId ?? null, input.limit ?? 100]
);
return result.rows.map(mapAgentEventRow);
}
}
export function buildAgentEventIdempotencyKey(input: {
teamId: string;
projectId: string;
sourceAdapter: string;
sourceEventId?: string | null;
serverSessionId?: string | null;
eventType: string;
occurredAt: Date | string | number;
payload?: JsonValue;
}): string {
if (input.sourceEventId) {
return `agent_event:v1:${deterministicKey([
input.teamId,
input.projectId,
input.sourceAdapter,
input.sourceEventId
])}`;
}
return `agent_event:v1:${deterministicKey([
input.teamId,
input.projectId,
input.sourceAdapter,
input.serverSessionId ?? null,
input.eventType,
new Date(input.occurredAt).toISOString(),
canonicalJson(input.payload ?? {})
])}`;
}
function mapAgentEventRow(row: AgentEventRow): PostgresAgentEvent {
return {
id: row.id,
projectId: row.project_id,
teamId: row.team_id,
serverSessionId: row.server_session_id,
sourceAdapter: row.source_adapter,
sourceEventId: row.source_event_id,
idempotencyKey: row.idempotency_key,
eventType: row.event_type,
payload: row.payload,
metadata: toJsonObject(row.metadata),
occurredAtEpoch: toEpoch(row.occurred_at),
receivedAtEpoch: toEpoch(row.received_at),
createdAtEpoch: toEpoch(row.created_at)
};
}
+168
View File
@@ -0,0 +1,168 @@
// SPDX-License-Identifier: Apache-2.0
import type { JsonObject, PostgresQueryable } from './utils.js';
import { assertProjectOwnership, newId, queryOne, toDate, toEpoch, toJsonArray, toJsonObject } from './utils.js';
export interface PostgresApiKey {
id: string;
keyHash: string;
teamId: string | null;
projectId: string | null;
actorId: string;
scopes: unknown[];
revokedAtEpoch: number | null;
expiresAtEpoch: number | null;
createdAtEpoch: number;
updatedAtEpoch: number;
}
export interface PostgresAuditLog {
id: string;
teamId: string | null;
projectId: string | null;
actorId: string | null;
apiKeyId: string | null;
action: string;
resourceType: string;
resourceId: string | null;
details: JsonObject;
createdAtEpoch: number;
}
interface ApiKeyRow {
id: string;
key_hash: string;
team_id: string | null;
project_id: string | null;
actor_id: string;
scopes: unknown;
revoked_at: Date | null;
expires_at: Date | null;
created_at: Date;
updated_at: Date;
}
interface AuditLogRow {
id: string;
team_id: string | null;
project_id: string | null;
actor_id: string | null;
api_key_id: string | null;
action: string;
resource_type: string;
resource_id: string | null;
details: unknown;
created_at: Date;
}
export class PostgresAuthRepository {
constructor(private client: PostgresQueryable) {}
async createApiKey(input: {
id?: string;
keyHash: string;
teamId?: string | null;
projectId?: string | null;
actorId: string;
scopes?: unknown[];
expiresAt?: Date | null;
}): Promise<PostgresApiKey> {
if (input.projectId && input.teamId) {
await assertProjectOwnership(this.client, input.projectId, input.teamId);
}
const id = input.id ?? newId();
const row = await queryOne<ApiKeyRow>(
this.client,
`
INSERT INTO api_keys (id, key_hash, team_id, project_id, actor_id, scopes, expires_at)
VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7)
RETURNING *
`,
[
id,
input.keyHash,
input.teamId ?? null,
input.projectId ?? null,
input.actorId,
JSON.stringify(input.scopes ?? []),
input.expiresAt ?? null
]
);
return mapApiKeyRow(row!);
}
async createAuditLog(input: {
id?: string;
teamId?: string | null;
projectId?: string | null;
actorId?: string | null;
apiKeyId?: string | null;
action: string;
resourceType: string;
resourceId?: string | null;
details?: JsonObject;
}): Promise<PostgresAuditLog> {
if (input.projectId && input.teamId) {
await assertProjectOwnership(this.client, input.projectId, input.teamId);
}
const id = input.id ?? newId();
const row = await queryOne<AuditLogRow>(
this.client,
`
INSERT INTO audit_log (
id, team_id, project_id, actor_id, api_key_id, action,
resource_type, resource_id, details
)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9::jsonb)
RETURNING *
`,
[
id,
input.teamId ?? null,
input.projectId ?? null,
input.actorId ?? null,
input.apiKeyId ?? null,
input.action,
input.resourceType,
input.resourceId ?? null,
JSON.stringify(input.details ?? {})
]
);
return mapAuditLogRow(row!);
}
async getApiKeyByHash(keyHash: string): Promise<PostgresApiKey | null> {
const row = await queryOne<ApiKeyRow>(this.client, 'SELECT * FROM api_keys WHERE key_hash = $1', [keyHash]);
return row ? mapApiKeyRow(row) : null;
}
}
function mapApiKeyRow(row: ApiKeyRow): PostgresApiKey {
return {
id: row.id,
keyHash: row.key_hash,
teamId: row.team_id,
projectId: row.project_id,
actorId: row.actor_id,
scopes: toJsonArray(row.scopes),
revokedAtEpoch: toDate(row.revoked_at)?.getTime() ?? null,
expiresAtEpoch: toDate(row.expires_at)?.getTime() ?? null,
createdAtEpoch: toEpoch(row.created_at),
updatedAtEpoch: toEpoch(row.updated_at)
};
}
function mapAuditLogRow(row: AuditLogRow): PostgresAuditLog {
return {
id: row.id,
teamId: row.team_id,
projectId: row.project_id,
actorId: row.actor_id,
apiKeyId: row.api_key_id,
action: row.action,
resourceType: row.resource_type,
resourceId: row.resource_id,
details: toJsonObject(row.details),
createdAtEpoch: toEpoch(row.created_at)
};
}
+72
View File
@@ -0,0 +1,72 @@
// SPDX-License-Identifier: Apache-2.0
export interface PostgresConfig {
connectionString: string;
max: number;
idleTimeoutMillis: number;
connectionTimeoutMillis: number;
statementTimeoutMillis: number;
ssl: boolean | { rejectUnauthorized: boolean };
}
export interface ParsePostgresConfigOptions {
env?: NodeJS.ProcessEnv;
requireDatabaseUrl?: boolean;
}
const DEFAULT_POOL_MAX = 10;
const DEFAULT_IDLE_TIMEOUT_MS = 30_000;
const DEFAULT_CONNECTION_TIMEOUT_MS = 5_000;
const DEFAULT_STATEMENT_TIMEOUT_MS = 30_000;
export function getPostgresDatabaseUrl(env: NodeJS.ProcessEnv = process.env): string | null {
return env.CLAUDE_MEM_SERVER_DATABASE_URL || null;
}
export function parsePostgresConfig(options: ParsePostgresConfigOptions = {}): PostgresConfig | null {
const env = options.env ?? process.env;
const connectionString = getPostgresDatabaseUrl(env);
if (!connectionString) {
if (options.requireDatabaseUrl) {
throw new Error('Postgres requires CLAUDE_MEM_SERVER_DATABASE_URL');
}
return null;
}
return {
connectionString,
max: parsePositiveInt(env.CLAUDE_MEM_POSTGRES_POOL_MAX, DEFAULT_POOL_MAX),
idleTimeoutMillis: parsePositiveInt(env.CLAUDE_MEM_POSTGRES_IDLE_TIMEOUT_MS, DEFAULT_IDLE_TIMEOUT_MS),
connectionTimeoutMillis: parsePositiveInt(env.CLAUDE_MEM_POSTGRES_CONNECTION_TIMEOUT_MS, DEFAULT_CONNECTION_TIMEOUT_MS),
statementTimeoutMillis: parsePositiveInt(env.CLAUDE_MEM_POSTGRES_STATEMENT_TIMEOUT_MS, DEFAULT_STATEMENT_TIMEOUT_MS),
ssl: parseSsl(connectionString, env)
};
}
function parsePositiveInt(value: string | undefined, fallback: number): number {
if (!value) {
return fallback;
}
const parsed = Number.parseInt(value, 10);
return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback;
}
function parseSsl(connectionString: string, env: NodeJS.ProcessEnv): boolean | { rejectUnauthorized: boolean } {
if (env.CLAUDE_MEM_POSTGRES_SSL === 'disable' || env.PGSSLMODE === 'disable') {
return false;
}
if (env.CLAUDE_MEM_POSTGRES_SSL === 'require' || env.PGSSLMODE === 'require') {
return { rejectUnauthorized: false };
}
try {
const url = new URL(connectionString);
if (url.searchParams.get('sslmode') === 'require') {
return { rejectUnauthorized: false };
}
} catch {
return false;
}
return false;
}
+457
View File
@@ -0,0 +1,457 @@
// SPDX-License-Identifier: Apache-2.0
import type { JsonObject, PostgresQueryable } from './utils.js';
import {
assertProjectOwnership,
assertSessionOwnership,
deterministicKey,
newId,
queryOne,
toDate,
toEpoch,
toJsonObject
} from './utils.js';
export type ObservationGenerationJobSourceType = 'agent_event' | 'session_summary' | 'observation_reindex';
export type ObservationGenerationJobStatus = 'queued' | 'processing' | 'completed' | 'failed' | 'cancelled';
export type ObservationGenerationJobEventType =
| 'queued'
| 'enqueued'
| 'processing'
| 'retry_scheduled'
| 'completed'
| 'failed'
| 'cancelled';
export interface PostgresObservationGenerationJob {
id: string;
projectId: string;
teamId: string;
agentEventId: string | null;
sourceType: ObservationGenerationJobSourceType;
sourceId: string;
serverSessionId: string | null;
jobType: string;
status: ObservationGenerationJobStatus;
idempotencyKey: string;
bullmqJobId: string | null;
attempts: number;
maxAttempts: number;
nextAttemptAtEpoch: number | null;
lockedAtEpoch: number | null;
lockedBy: string | null;
completedAtEpoch: number | null;
failedAtEpoch: number | null;
cancelledAtEpoch: number | null;
lastError: JsonObject | null;
payload: JsonObject;
createdAtEpoch: number;
updatedAtEpoch: number;
}
export interface PostgresObservationGenerationJobEvent {
id: string;
generationJobId: string;
eventType: ObservationGenerationJobEventType;
statusAfter: ObservationGenerationJobStatus;
attempt: number;
details: JsonObject;
createdAtEpoch: number;
}
interface JobRow {
id: string;
project_id: string;
team_id: string;
agent_event_id: string | null;
source_type: ObservationGenerationJobSourceType;
source_id: string;
server_session_id: string | null;
job_type: string;
status: ObservationGenerationJobStatus;
idempotency_key: string;
bullmq_job_id: string | null;
attempts: number;
max_attempts: number;
next_attempt_at: Date | null;
locked_at: Date | null;
locked_by: string | null;
completed_at: Date | null;
failed_at: Date | null;
cancelled_at: Date | null;
last_error: unknown | null;
payload: unknown;
created_at: Date;
updated_at: Date;
}
interface JobEventRow {
id: string;
generation_job_id: string;
event_type: ObservationGenerationJobEventType;
status_after: ObservationGenerationJobStatus;
attempt: number;
details: unknown;
created_at: Date;
}
export class PostgresObservationGenerationJobRepository {
constructor(private client: PostgresQueryable) {}
async create(input: {
id?: string;
projectId: string;
teamId: string;
sourceType: ObservationGenerationJobSourceType;
sourceId: string;
agentEventId?: string | null;
serverSessionId?: string | null;
jobType: string;
status?: ObservationGenerationJobStatus;
bullmqJobId?: string | null;
maxAttempts?: number;
payload?: JsonObject;
}): Promise<PostgresObservationGenerationJob> {
await this.validateSource(input);
const sourceModel = normalizeSourceModel(input);
const idempotencyKey = buildObservationGenerationJobIdempotencyKey(input);
const row = await queryOne<JobRow>(
this.client,
`
INSERT INTO observation_generation_jobs (
id, project_id, team_id, agent_event_id, source_type, source_id,
server_session_id, job_type, status, idempotency_key, bullmq_job_id,
max_attempts, payload
)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13::jsonb)
ON CONFLICT (idempotency_key) DO UPDATE SET
payload = observation_generation_jobs.payload || excluded.payload,
updated_at = now()
RETURNING *
`,
[
input.id ?? newId(),
input.projectId,
input.teamId,
sourceModel.agentEventId,
input.sourceType,
input.sourceId,
sourceModel.serverSessionId,
input.jobType,
input.status ?? 'queued',
idempotencyKey,
input.bullmqJobId ?? null,
input.maxAttempts ?? 3,
JSON.stringify(input.payload ?? {})
]
);
return mapJobRow(row!);
}
async getByIdForScope(input: {
id: string;
projectId: string;
teamId: string;
}): Promise<PostgresObservationGenerationJob | null> {
const row = await queryOne<JobRow>(
this.client,
'SELECT * FROM observation_generation_jobs WHERE id = $1 AND project_id = $2 AND team_id = $3',
[input.id, input.projectId, input.teamId]
);
return row ? mapJobRow(row) : null;
}
async transitionStatus(input: {
id: string;
projectId: string;
teamId: string;
status: ObservationGenerationJobStatus;
lockedBy?: string | null;
lastError?: JsonObject | null;
nextAttemptAt?: Date | null;
}): Promise<PostgresObservationGenerationJob | null> {
const row = await queryOne<JobRow>(
this.client,
`
UPDATE observation_generation_jobs
SET
status = $2,
attempts = CASE WHEN $2 = 'processing' THEN attempts + 1 ELSE attempts END,
locked_at = CASE WHEN $2 = 'processing' THEN now() ELSE NULL END,
locked_by = CASE WHEN $2 = 'processing' THEN $3 ELSE NULL END,
next_attempt_at = CASE WHEN $2 = 'queued' THEN $4::timestamptz ELSE NULL::timestamptz END,
completed_at = CASE WHEN $2 = 'completed' THEN now() ELSE NULL END,
failed_at = CASE WHEN $2 = 'failed' THEN now() ELSE NULL END,
cancelled_at = CASE WHEN $2 = 'cancelled' THEN now() ELSE NULL END,
last_error = $5::jsonb,
updated_at = now()
WHERE id = $1
AND project_id = $6
AND team_id = $7
AND (
(status = 'queued' AND $2 IN ('processing', 'failed', 'cancelled'))
OR
(status = 'processing' AND $2 IN ('queued', 'completed', 'failed', 'cancelled'))
)
AND ($2 <> 'processing' OR attempts < max_attempts)
AND ($2 <> 'queued' OR attempts < max_attempts)
RETURNING *
`,
[
input.id,
input.status,
input.lockedBy ?? null,
input.nextAttemptAt ?? null,
input.lastError == null ? null : JSON.stringify(input.lastError),
input.projectId,
input.teamId
]
);
if (row) {
return mapJobRow(row);
}
const current = await queryOne<JobRow>(
this.client,
'SELECT * FROM observation_generation_jobs WHERE id = $1 AND project_id = $2 AND team_id = $3',
[input.id, input.projectId, input.teamId]
);
if (!current) {
return null;
}
assertValidJobStatusTransition(mapJobRow(current), input.status);
throw new Error('observation generation job status transition was not applied');
}
async listByStatusForScope(input: {
status: ObservationGenerationJobStatus;
projectId: string;
teamId: string;
limit?: number;
}): Promise<PostgresObservationGenerationJob[]> {
const result = await this.client.query<JobRow>(
`
SELECT * FROM observation_generation_jobs
WHERE status = $1 AND project_id = $2 AND team_id = $3
ORDER BY created_at ASC
LIMIT $4
`,
[input.status, input.projectId, input.teamId, input.limit ?? 100]
);
return result.rows.map(mapJobRow);
}
private async validateSource(input: {
projectId: string;
teamId: string;
sourceType: ObservationGenerationJobSourceType;
sourceId: string;
agentEventId?: string | null;
serverSessionId?: string | null;
}): Promise<void> {
await assertProjectOwnership(this.client, input.projectId, input.teamId);
if (input.sourceType === 'agent_event') {
const eventId = input.agentEventId ?? input.sourceId;
const row = await queryOne<{ id: string; server_session_id: string | null }>(
this.client,
'SELECT id, server_session_id FROM agent_events WHERE id = $1 AND project_id = $2 AND team_id = $3',
[eventId, input.projectId, input.teamId]
);
if (!row || input.sourceId !== eventId) {
throw new Error('agent_event source_id must belong to project_id and team_id');
}
if (input.serverSessionId) {
await assertSessionOwnership(this.client, input.serverSessionId, input.projectId, input.teamId);
if (row.server_session_id && row.server_session_id !== input.serverSessionId) {
throw new Error('server_session_id must match the agent_event server_session_id');
}
}
return;
}
if (input.sourceType === 'session_summary') {
const sessionId = input.serverSessionId ?? input.sourceId;
await assertSessionOwnership(this.client, sessionId, input.projectId, input.teamId);
if (input.sourceId !== sessionId) {
throw new Error('session_summary source_id must equal server_session_id');
}
return;
}
const observation = await queryOne<{ id: string }>(
this.client,
'SELECT id FROM observations WHERE id = $1 AND project_id = $2 AND team_id = $3',
[input.sourceId, input.projectId, input.teamId]
);
if (!observation) {
throw new Error('observation_reindex source_id must belong to project_id and team_id');
}
if (input.serverSessionId) {
await assertSessionOwnership(this.client, input.serverSessionId, input.projectId, input.teamId);
}
}
}
export class PostgresObservationGenerationJobEventsRepository {
constructor(private client: PostgresQueryable) {}
async append(input: {
id?: string;
generationJobId: string;
projectId: string;
teamId: string;
eventType: ObservationGenerationJobEventType;
statusAfter: ObservationGenerationJobStatus;
attempt?: number;
details?: JsonObject;
}): Promise<PostgresObservationGenerationJobEvent> {
const row = await queryOne<JobEventRow>(
this.client,
`
INSERT INTO observation_generation_job_events (
id, generation_job_id, event_type, status_after, attempt, details
)
SELECT $1, jobs.id, $4, $5, $6, $7::jsonb
FROM observation_generation_jobs jobs
WHERE jobs.id = $2
AND jobs.project_id = $3
AND jobs.team_id = $8
RETURNING observation_generation_job_events.*
`,
[
input.id ?? newId(),
input.generationJobId,
input.projectId,
input.eventType,
input.statusAfter,
input.attempt ?? 0,
JSON.stringify(input.details ?? {}),
input.teamId
]
);
if (!row) {
throw new Error('generation_job_id must belong to project_id and team_id');
}
return mapJobEventRow(row!);
}
async listByJobForScope(input: {
generationJobId: string;
projectId: string;
teamId: string;
}): Promise<PostgresObservationGenerationJobEvent[]> {
const result = await this.client.query<JobEventRow>(
`
SELECT events.*
FROM observation_generation_job_events events
INNER JOIN observation_generation_jobs jobs ON jobs.id = events.generation_job_id
WHERE events.generation_job_id = $1 AND jobs.project_id = $2 AND jobs.team_id = $3
ORDER BY events.created_at ASC
`,
[input.generationJobId, input.projectId, input.teamId]
);
return result.rows.map(mapJobEventRow);
}
}
export function buildObservationGenerationJobIdempotencyKey(input: {
teamId: string;
projectId: string;
sourceType: ObservationGenerationJobSourceType;
sourceId: string;
jobType: string;
}): string {
return `observation_generation_job:v1:${deterministicKey([
input.teamId,
input.projectId,
input.sourceType,
input.sourceId,
input.jobType
])}`;
}
function normalizeSourceModel(input: {
sourceType: ObservationGenerationJobSourceType;
sourceId: string;
agentEventId?: string | null;
serverSessionId?: string | null;
}): { agentEventId: string | null; serverSessionId: string | null } {
if (input.sourceType === 'agent_event') {
return { agentEventId: input.agentEventId ?? input.sourceId, serverSessionId: input.serverSessionId ?? null };
}
if (input.sourceType === 'session_summary') {
return { agentEventId: null, serverSessionId: input.serverSessionId ?? input.sourceId };
}
return { agentEventId: null, serverSessionId: input.serverSessionId ?? null };
}
const TERMINAL_JOB_STATUSES = new Set<ObservationGenerationJobStatus>(['completed', 'failed', 'cancelled']);
const ALLOWED_JOB_TRANSITIONS: Record<ObservationGenerationJobStatus, readonly ObservationGenerationJobStatus[]> = {
queued: ['processing', 'failed', 'cancelled'],
processing: ['queued', 'completed', 'failed', 'cancelled'],
completed: [],
failed: [],
cancelled: []
};
function assertValidJobStatusTransition(
current: PostgresObservationGenerationJob,
nextStatus: ObservationGenerationJobStatus
): void {
if (TERMINAL_JOB_STATUSES.has(current.status)) {
throw new Error(`cannot transition observation generation job from terminal status ${current.status}`);
}
if (!ALLOWED_JOB_TRANSITIONS[current.status].includes(nextStatus)) {
throw new Error(`illegal observation generation job transition from ${current.status} to ${nextStatus}`);
}
if (nextStatus === 'processing' && current.attempts >= current.maxAttempts) {
throw new Error('cannot process observation generation job after max_attempts is reached');
}
if (nextStatus === 'queued' && current.attempts >= current.maxAttempts) {
throw new Error('cannot retry observation generation job after max_attempts is reached');
}
}
function mapJobRow(row: JobRow): PostgresObservationGenerationJob {
return {
id: row.id,
projectId: row.project_id,
teamId: row.team_id,
agentEventId: row.agent_event_id,
sourceType: row.source_type,
sourceId: row.source_id,
serverSessionId: row.server_session_id,
jobType: row.job_type,
status: row.status,
idempotencyKey: row.idempotency_key,
bullmqJobId: row.bullmq_job_id,
attempts: row.attempts,
maxAttempts: row.max_attempts,
nextAttemptAtEpoch: toDate(row.next_attempt_at)?.getTime() ?? null,
lockedAtEpoch: toDate(row.locked_at)?.getTime() ?? null,
lockedBy: row.locked_by,
completedAtEpoch: toDate(row.completed_at)?.getTime() ?? null,
failedAtEpoch: toDate(row.failed_at)?.getTime() ?? null,
cancelledAtEpoch: toDate(row.cancelled_at)?.getTime() ?? null,
lastError: row.last_error == null ? null : toJsonObject(row.last_error),
payload: toJsonObject(row.payload),
createdAtEpoch: toEpoch(row.created_at),
updatedAtEpoch: toEpoch(row.updated_at)
};
}
function mapJobEventRow(row: JobEventRow): PostgresObservationGenerationJobEvent {
return {
id: row.id,
generationJobId: row.generation_job_id,
eventType: row.event_type,
statusAfter: row.status_after,
attempt: row.attempt,
details: toJsonObject(row.details),
createdAtEpoch: toEpoch(row.created_at)
};
}
+51
View File
@@ -0,0 +1,51 @@
// SPDX-License-Identifier: Apache-2.0
import type { PostgresQueryable } from './utils.js';
import { PostgresAgentEventsRepository } from './agent-events.js';
import { PostgresAuthRepository } from './auth.js';
import {
PostgresObservationGenerationJobEventsRepository,
PostgresObservationGenerationJobRepository
} from './generation-jobs.js';
import { PostgresObservationRepository, PostgresObservationSourcesRepository } from './observations.js';
import { PostgresProjectsRepository } from './projects.js';
import { PostgresServerSessionsRepository } from './server-sessions.js';
import { PostgresTeamsRepository } from './teams.js';
export * from './agent-events.js';
export * from './auth.js';
export * from './config.js';
export * from './generation-jobs.js';
export * from './observations.js';
export * from './pool.js';
export * from './projects.js';
export * from './schema.js';
export * from './server-sessions.js';
export * from './teams.js';
export type * from './utils.js';
export interface PostgresStorageRepositories {
teams: PostgresTeamsRepository;
projects: PostgresProjectsRepository;
auth: PostgresAuthRepository;
sessions: PostgresServerSessionsRepository;
agentEvents: PostgresAgentEventsRepository;
observations: PostgresObservationRepository;
observationSources: PostgresObservationSourcesRepository;
observationGenerationJobs: PostgresObservationGenerationJobRepository;
observationGenerationJobEvents: PostgresObservationGenerationJobEventsRepository;
}
export function createPostgresStorageRepositories(client: PostgresQueryable): PostgresStorageRepositories {
return {
teams: new PostgresTeamsRepository(client),
projects: new PostgresProjectsRepository(client),
auth: new PostgresAuthRepository(client),
sessions: new PostgresServerSessionsRepository(client),
agentEvents: new PostgresAgentEventsRepository(client),
observations: new PostgresObservationRepository(client),
observationSources: new PostgresObservationSourcesRepository(client),
observationGenerationJobs: new PostgresObservationGenerationJobRepository(client),
observationGenerationJobEvents: new PostgresObservationGenerationJobEventsRepository(client)
};
}
+395
View File
@@ -0,0 +1,395 @@
// SPDX-License-Identifier: Apache-2.0
import type { JsonObject, JsonValue, PostgresQueryable } from './utils.js';
import {
assertProjectOwnership,
assertSessionOwnership,
canonicalJson,
deterministicKey,
newId,
queryOne,
toEpoch,
toJsonObject
} from './utils.js';
export type ObservationSourceType = 'agent_event' | 'session_summary' | 'observation_reindex' | 'manual';
export interface PostgresObservation {
id: string;
projectId: string;
teamId: string;
serverSessionId: string | null;
kind: string;
content: string;
generationKey: string | null;
metadata: JsonObject;
embedding: JsonValue | null;
createdByJobId: string | null;
createdAtEpoch: number;
updatedAtEpoch: number;
}
export interface PostgresObservationSource {
id: string;
observationId: string;
agentEventId: string | null;
generationJobId: string | null;
sourceType: ObservationSourceType;
sourceId: string;
metadata: JsonObject;
createdAtEpoch: number;
}
interface ObservationRow {
id: string;
project_id: string;
team_id: string;
server_session_id: string | null;
kind: string;
content: string;
generation_key: string | null;
metadata: unknown;
embedding: unknown | null;
created_by_job_id: string | null;
created_at: Date;
updated_at: Date;
}
interface ObservationSourceRow {
id: string;
observation_id: string;
agent_event_id: string | null;
generation_job_id: string | null;
source_type: ObservationSourceType;
source_id: string;
metadata: unknown;
created_at: Date;
}
export class PostgresObservationRepository {
constructor(private client: PostgresQueryable) {}
async create(input: {
id?: string;
projectId: string;
teamId: string;
serverSessionId?: string | null;
kind?: string;
content: string;
generationKey?: string | null;
metadata?: JsonObject;
embedding?: JsonValue | null;
createdByJobId?: string | null;
}): Promise<PostgresObservation> {
await assertProjectOwnership(this.client, input.projectId, input.teamId);
if (input.serverSessionId) {
await assertSessionOwnership(this.client, input.serverSessionId, input.projectId, input.teamId);
}
if (input.createdByJobId) {
await assertJobOwnership(this.client, input.createdByJobId, input.projectId, input.teamId);
}
const row = await queryOne<ObservationRow>(
this.client,
`
INSERT INTO observations (
id, project_id, team_id, server_session_id, kind, content,
generation_key, metadata, embedding, created_by_job_id
)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8::jsonb, $9::jsonb, $10)
ON CONFLICT (team_id, project_id, generation_key) WHERE generation_key IS NOT NULL DO UPDATE SET
updated_at = observations.updated_at
RETURNING *
`,
[
input.id ?? newId(),
input.projectId,
input.teamId,
input.serverSessionId ?? null,
input.kind ?? 'observation',
input.content,
input.generationKey ?? null,
JSON.stringify(input.metadata ?? {}),
input.embedding == null ? null : JSON.stringify(input.embedding),
input.createdByJobId ?? null
]
);
return mapObservationRow(row!);
}
async getByIdForScope(input: {
id: string;
projectId: string;
teamId: string;
}): Promise<PostgresObservation | null> {
const row = await queryOne<ObservationRow>(
this.client,
'SELECT * FROM observations WHERE id = $1 AND project_id = $2 AND team_id = $3',
[input.id, input.projectId, input.teamId]
);
return row ? mapObservationRow(row) : null;
}
async listByProject(input: {
projectId: string;
teamId: string;
serverSessionId?: string | null;
limit?: number;
}): Promise<PostgresObservation[]> {
const result = await this.client.query<ObservationRow>(
`
SELECT * FROM observations
WHERE project_id = $1
AND team_id = $2
AND ($3::text IS NULL OR server_session_id = $3)
ORDER BY created_at DESC
LIMIT $4
`,
[input.projectId, input.teamId, input.serverSessionId ?? null, input.limit ?? 100]
);
return result.rows.map(mapObservationRow);
}
async search(input: {
projectId: string;
teamId: string;
query: string;
limit?: number;
}): Promise<PostgresObservation[]> {
const result = await this.client.query<ObservationRow>(
`
SELECT * FROM observations
WHERE project_id = $1
AND team_id = $2
AND content_search @@ websearch_to_tsquery('english', $3)
ORDER BY ts_rank(content_search, websearch_to_tsquery('english', $3)) DESC, updated_at DESC
LIMIT $4
`,
[input.projectId, input.teamId, input.query, input.limit ?? 20]
);
return result.rows.map(mapObservationRow);
}
}
export class PostgresObservationSourcesRepository {
constructor(private client: PostgresQueryable) {}
async addSource(input: {
id?: string;
observationId: string;
projectId: string;
teamId: string;
sourceType: ObservationSourceType;
sourceId: string;
agentEventId?: string | null;
generationJobId?: string | null;
metadata?: JsonObject;
}): Promise<PostgresObservationSource> {
const observation = await queryOne<{ id: string }>(
this.client,
'SELECT id FROM observations WHERE id = $1 AND project_id = $2 AND team_id = $3',
[input.observationId, input.projectId, input.teamId]
);
if (!observation) {
throw new Error('observation_id does not exist');
}
const agentEventId = input.sourceType === 'agent_event'
? input.agentEventId ?? input.sourceId
: null;
if (input.sourceType === 'agent_event') {
if (agentEventId !== input.sourceId) {
throw new Error('agent_event source_id must equal agent_event_id');
}
await assertAgentEventOwnership(this.client, input.sourceId, input.projectId, input.teamId);
} else if (input.sourceType === 'session_summary' && !input.generationJobId) {
await assertSessionOwnership(this.client, input.sourceId, input.projectId, input.teamId);
} else if (input.sourceType === 'observation_reindex' && !input.generationJobId) {
await assertObservationOwnership(this.client, input.sourceId, input.projectId, input.teamId);
}
if (input.generationJobId) {
await assertGenerationJobMatchesSource(this.client, {
generationJobId: input.generationJobId,
projectId: input.projectId,
teamId: input.teamId,
sourceType: input.sourceType,
sourceId: input.sourceId,
agentEventId
});
}
const row = await queryOne<ObservationSourceRow>(
this.client,
`
INSERT INTO observation_sources (
id, observation_id, agent_event_id, generation_job_id,
source_type, source_id, metadata
)
VALUES ($1, $2, $3, $4, $5, $6, $7::jsonb)
ON CONFLICT (observation_id, source_type, source_id) DO UPDATE SET
metadata = observation_sources.metadata || excluded.metadata
RETURNING *
`,
[
input.id ?? newId(),
input.observationId,
agentEventId,
input.generationJobId ?? null,
input.sourceType,
input.sourceId,
JSON.stringify(input.metadata ?? {})
]
);
return mapObservationSourceRow(row!);
}
async listByObservationForScope(input: {
observationId: string;
projectId: string;
teamId: string;
}): Promise<PostgresObservationSource[]> {
const result = await this.client.query<ObservationSourceRow>(
`
SELECT observation_sources.*
FROM observation_sources
INNER JOIN observations
ON observations.id = observation_sources.observation_id
WHERE observation_sources.observation_id = $1
AND observations.project_id = $2
AND observations.team_id = $3
ORDER BY observation_sources.created_at ASC
`,
[input.observationId, input.projectId, input.teamId]
);
return result.rows.map(mapObservationSourceRow);
}
}
export function buildObservationGenerationKey(input: {
generationJobId: string;
parsedObservationIndex: number;
content: string;
}): string {
return `generation:v1:${input.generationJobId}:${input.parsedObservationIndex}:${deterministicKey([
canonicalJson(input.content.trim())
])}`;
}
async function assertJobOwnership(
client: PostgresQueryable,
generationJobId: string,
projectId: string,
teamId: string
): Promise<void> {
const row = await queryOne<{ id: string }>(
client,
'SELECT id FROM observation_generation_jobs WHERE id = $1 AND project_id = $2 AND team_id = $3',
[generationJobId, projectId, teamId]
);
if (!row) {
throw new Error('generation_job_id must belong to project_id and team_id');
}
}
async function assertGenerationJobMatchesSource(
client: PostgresQueryable,
input: {
generationJobId: string;
projectId: string;
teamId: string;
sourceType: ObservationSourceType;
sourceId: string;
agentEventId: string | null;
}
): Promise<void> {
if (input.sourceType === 'manual') {
throw new Error('manual observation sources cannot be linked to a generation_job_id');
}
const row = await queryOne<{
id: string;
source_type: string;
source_id: string;
agent_event_id: string | null;
}>(
client,
`
SELECT id, source_type, source_id, agent_event_id
FROM observation_generation_jobs
WHERE id = $1 AND project_id = $2 AND team_id = $3
`,
[input.generationJobId, input.projectId, input.teamId]
);
if (!row) {
throw new Error('generation_job_id must belong to project_id and team_id');
}
if (row.source_type !== input.sourceType || row.source_id !== input.sourceId) {
throw new Error('generation_job_id source model must match observation source');
}
if (input.sourceType === 'agent_event' && row.agent_event_id !== input.agentEventId) {
throw new Error('generation_job_id agent_event_id must match observation source');
}
}
async function assertAgentEventOwnership(
client: PostgresQueryable,
agentEventId: string,
projectId: string,
teamId: string
): Promise<void> {
const row = await queryOne<{ id: string }>(
client,
'SELECT id FROM agent_events WHERE id = $1 AND project_id = $2 AND team_id = $3',
[agentEventId, projectId, teamId]
);
if (!row) {
throw new Error('agent_event_id must belong to project_id and team_id');
}
}
async function assertObservationOwnership(
client: PostgresQueryable,
observationId: string,
projectId: string,
teamId: string
): Promise<void> {
const row = await queryOne<{ id: string }>(
client,
'SELECT id FROM observations WHERE id = $1 AND project_id = $2 AND team_id = $3',
[observationId, projectId, teamId]
);
if (!row) {
throw new Error('observation_reindex source_id must belong to project_id and team_id');
}
}
function mapObservationRow(row: ObservationRow): PostgresObservation {
return {
id: row.id,
projectId: row.project_id,
teamId: row.team_id,
serverSessionId: row.server_session_id,
kind: row.kind,
content: row.content,
generationKey: row.generation_key,
metadata: toJsonObject(row.metadata),
embedding: row.embedding,
createdByJobId: row.created_by_job_id,
createdAtEpoch: toEpoch(row.created_at),
updatedAtEpoch: toEpoch(row.updated_at)
};
}
function mapObservationSourceRow(row: ObservationSourceRow): PostgresObservationSource {
return {
id: row.id,
observationId: row.observation_id,
agentEventId: row.agent_event_id,
generationJobId: row.generation_job_id,
sourceType: row.source_type,
sourceId: row.source_id,
metadata: toJsonObject(row.metadata),
createdAtEpoch: toEpoch(row.created_at)
};
}
+68
View File
@@ -0,0 +1,68 @@
// SPDX-License-Identifier: Apache-2.0
import pg, { type Pool as PgPool, type PoolClient as PgPoolClient } from 'pg';
import { parsePostgresConfig, type PostgresConfig } from './config.js';
const { Pool } = pg;
export type PostgresPool = PgPool;
export type PostgresPoolClient = PgPoolClient;
let sharedPool: PostgresPool | null = null;
export function createPostgresPool(config: PostgresConfig): PostgresPool {
return new Pool({
connectionString: config.connectionString,
max: config.max,
idleTimeoutMillis: config.idleTimeoutMillis,
connectionTimeoutMillis: config.connectionTimeoutMillis,
statement_timeout: config.statementTimeoutMillis,
ssl: config.ssl
});
}
export function getSharedPostgresPool(options: { requireDatabaseUrl?: boolean } = {}): PostgresPool {
if (sharedPool) {
return sharedPool;
}
const config = parsePostgresConfig({ requireDatabaseUrl: options.requireDatabaseUrl ?? true });
if (!config) {
throw new Error('Postgres requires CLAUDE_MEM_SERVER_DATABASE_URL');
}
sharedPool = createPostgresPool(config);
return sharedPool;
}
export async function checkPostgresHealth(pool: PostgresPool): Promise<boolean> {
try {
await pool.query('SELECT 1');
return true;
} catch {
return false;
}
}
export async function withPostgresTransaction<T>(
pool: PostgresPool,
fn: (client: PostgresPoolClient) => Promise<T>
): Promise<T> {
const client = await pool.connect();
try {
await client.query('BEGIN');
const result = await fn(client);
await client.query('COMMIT');
return result;
} catch (error) {
await client.query('ROLLBACK');
throw error;
} finally {
client.release();
}
}
export async function closePostgresPool(pool: PostgresPool): Promise<void> {
if (pool === sharedPool) {
sharedPool = null;
}
await pool.end();
}
+65
View File
@@ -0,0 +1,65 @@
// SPDX-License-Identifier: Apache-2.0
import type { JsonObject, PostgresQueryable } from './utils.js';
import { newId, queryOne, toEpoch, toJsonObject } from './utils.js';
export interface PostgresProject {
id: string;
teamId: string;
name: string;
metadata: JsonObject;
createdAtEpoch: number;
updatedAtEpoch: number;
}
interface ProjectRow {
id: string;
team_id: string;
name: string;
metadata: unknown;
created_at: Date;
updated_at: Date;
}
export class PostgresProjectsRepository {
constructor(private client: PostgresQueryable) {}
async create(input: {
id?: string;
teamId: string;
name: string;
metadata?: JsonObject;
}): Promise<PostgresProject> {
const id = input.id ?? newId();
const row = await queryOne<ProjectRow>(
this.client,
`
INSERT INTO projects (id, team_id, name, metadata)
VALUES ($1, $2, $3, $4::jsonb)
RETURNING *
`,
[id, input.teamId, input.name, JSON.stringify(input.metadata ?? {})]
);
return mapProjectRow(row!);
}
async getByIdForTeam(id: string, teamId: string): Promise<PostgresProject | null> {
const row = await queryOne<ProjectRow>(
this.client,
'SELECT * FROM projects WHERE id = $1 AND team_id = $2',
[id, teamId]
);
return row ? mapProjectRow(row) : null;
}
}
function mapProjectRow(row: ProjectRow): PostgresProject {
return {
id: row.id,
teamId: row.team_id,
name: row.name,
metadata: toJsonObject(row.metadata),
createdAtEpoch: toEpoch(row.created_at),
updatedAtEpoch: toEpoch(row.updated_at)
};
}
+283
View File
@@ -0,0 +1,283 @@
// SPDX-License-Identifier: Apache-2.0
import type { PostgresQueryable } from './utils.js';
export const SERVER_BETA_POSTGRES_SCHEMA_VERSION = 1;
export const SERVER_BETA_POSTGRES_TABLES = [
'server_beta_schema_migrations',
'teams',
'projects',
'team_members',
'api_keys',
'audit_log',
'server_sessions',
'agent_events',
'observation_generation_jobs',
'observations',
'observation_sources',
'observation_generation_job_events'
] as const;
export async function bootstrapServerBetaPostgresSchema(client: PostgresQueryable): Promise<void> {
if (isPostgresPool(client)) {
const poolClient = await client.connect();
try {
await bootstrapServerBetaPostgresSchema(poolClient);
} finally {
poolClient.release();
}
return;
}
await client.query('BEGIN');
try {
await client.query(PHASE_1_SCHEMA_SQL);
await client.query(
`
INSERT INTO server_beta_schema_migrations (version, description)
VALUES ($1, $2)
ON CONFLICT (version) DO NOTHING
`,
[SERVER_BETA_POSTGRES_SCHEMA_VERSION, 'phase 1 postgres observation storage foundation']
);
await client.query('COMMIT');
} catch (error) {
await client.query('ROLLBACK');
throw error;
}
}
interface PostgresPoolLike extends PostgresQueryable {
connect(): Promise<PostgresQueryable & { release(): void }>;
}
function isPostgresPool(client: PostgresQueryable): client is PostgresPoolLike {
const candidate = client as {
connect?: unknown;
release?: unknown;
totalCount?: unknown;
idleCount?: unknown;
waitingCount?: unknown;
};
return (
typeof candidate.connect === 'function'
&& typeof candidate.release !== 'function'
&& typeof candidate.totalCount === 'number'
&& typeof candidate.idleCount === 'number'
&& typeof candidate.waitingCount === 'number'
);
}
const PHASE_1_SCHEMA_SQL = `
CREATE TABLE IF NOT EXISTS server_beta_schema_migrations (
version INTEGER PRIMARY KEY,
description TEXT NOT NULL,
applied_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS teams (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
metadata JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS projects (
id TEXT PRIMARY KEY,
team_id TEXT NOT NULL REFERENCES teams(id) ON DELETE CASCADE,
name TEXT NOT NULL,
metadata JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (id, team_id)
);
CREATE TABLE IF NOT EXISTS team_members (
team_id TEXT NOT NULL REFERENCES teams(id) ON DELETE CASCADE,
user_id TEXT NOT NULL,
role TEXT NOT NULL,
metadata JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (team_id, user_id)
);
CREATE TABLE IF NOT EXISTS api_keys (
id TEXT PRIMARY KEY,
key_hash TEXT NOT NULL UNIQUE,
team_id TEXT REFERENCES teams(id) ON DELETE CASCADE,
project_id TEXT REFERENCES projects(id) ON DELETE CASCADE,
actor_id TEXT NOT NULL,
scopes JSONB NOT NULL DEFAULT '[]'::jsonb,
revoked_at TIMESTAMPTZ,
expires_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CHECK (project_id IS NULL OR team_id IS NOT NULL),
FOREIGN KEY (project_id, team_id) REFERENCES projects(id, team_id) ON DELETE CASCADE
);
CREATE TABLE IF NOT EXISTS audit_log (
id TEXT PRIMARY KEY,
team_id TEXT REFERENCES teams(id) ON DELETE SET NULL,
project_id TEXT REFERENCES projects(id) ON DELETE SET NULL,
actor_id TEXT,
api_key_id TEXT REFERENCES api_keys(id) ON DELETE SET NULL,
action TEXT NOT NULL,
resource_type TEXT NOT NULL,
resource_id TEXT,
details JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CHECK (project_id IS NULL OR team_id IS NOT NULL),
FOREIGN KEY (project_id, team_id) REFERENCES projects(id, team_id) ON DELETE SET NULL
);
CREATE TABLE IF NOT EXISTS server_sessions (
id TEXT PRIMARY KEY,
project_id TEXT NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
team_id TEXT NOT NULL REFERENCES teams(id) ON DELETE CASCADE,
external_session_id TEXT,
idempotency_key TEXT,
content_session_id TEXT,
agent_id TEXT,
agent_type TEXT,
platform_source TEXT,
generation_status TEXT NOT NULL DEFAULT 'idle',
metadata JSONB NOT NULL DEFAULT '{}'::jsonb,
started_at TIMESTAMPTZ NOT NULL DEFAULT now(),
ended_at TIMESTAMPTZ,
last_generated_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (project_id, external_session_id),
FOREIGN KEY (project_id, team_id) REFERENCES projects(id, team_id) ON DELETE CASCADE
);
CREATE TABLE IF NOT EXISTS agent_events (
id TEXT PRIMARY KEY,
project_id TEXT NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
team_id TEXT NOT NULL REFERENCES teams(id) ON DELETE CASCADE,
server_session_id TEXT REFERENCES server_sessions(id) ON DELETE SET NULL,
source_adapter TEXT NOT NULL,
source_event_id TEXT,
idempotency_key TEXT NOT NULL,
event_type TEXT NOT NULL,
payload JSONB NOT NULL,
metadata JSONB NOT NULL DEFAULT '{}'::jsonb,
occurred_at TIMESTAMPTZ NOT NULL,
received_at TIMESTAMPTZ NOT NULL DEFAULT now(),
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (idempotency_key),
UNIQUE (id, project_id, team_id),
FOREIGN KEY (project_id, team_id) REFERENCES projects(id, team_id) ON DELETE CASCADE
);
CREATE TABLE IF NOT EXISTS observation_generation_jobs (
id TEXT PRIMARY KEY,
project_id TEXT NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
team_id TEXT NOT NULL REFERENCES teams(id) ON DELETE CASCADE,
agent_event_id TEXT REFERENCES agent_events(id) ON DELETE CASCADE,
source_type TEXT NOT NULL CHECK (source_type IN ('agent_event', 'session_summary', 'observation_reindex')),
source_id TEXT NOT NULL,
server_session_id TEXT REFERENCES server_sessions(id) ON DELETE SET NULL,
job_type TEXT NOT NULL,
status TEXT NOT NULL CHECK (status IN ('queued', 'processing', 'completed', 'failed', 'cancelled')),
idempotency_key TEXT NOT NULL UNIQUE,
bullmq_job_id TEXT UNIQUE,
attempts INTEGER NOT NULL DEFAULT 0,
max_attempts INTEGER NOT NULL DEFAULT 3,
next_attempt_at TIMESTAMPTZ,
locked_at TIMESTAMPTZ,
locked_by TEXT,
completed_at TIMESTAMPTZ,
failed_at TIMESTAMPTZ,
cancelled_at TIMESTAMPTZ,
last_error JSONB,
payload JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CHECK (
(source_type = 'agent_event' AND agent_event_id IS NOT NULL AND source_id = agent_event_id)
OR
(source_type = 'session_summary' AND agent_event_id IS NULL AND server_session_id IS NOT NULL AND source_id = server_session_id)
OR
(source_type = 'observation_reindex' AND agent_event_id IS NULL)
),
FOREIGN KEY (agent_event_id, project_id, team_id) REFERENCES agent_events(id, project_id, team_id) ON DELETE CASCADE,
FOREIGN KEY (project_id, team_id) REFERENCES projects(id, team_id) ON DELETE CASCADE
);
CREATE TABLE IF NOT EXISTS observations (
id TEXT PRIMARY KEY,
project_id TEXT NOT NULL REFERENCES projects(id) ON DELETE CASCADE,
team_id TEXT NOT NULL REFERENCES teams(id) ON DELETE CASCADE,
server_session_id TEXT REFERENCES server_sessions(id) ON DELETE SET NULL,
kind TEXT NOT NULL DEFAULT 'observation',
content TEXT NOT NULL,
content_search TSVECTOR GENERATED ALWAYS AS (to_tsvector('english', content)) STORED,
generation_key TEXT,
metadata JSONB NOT NULL DEFAULT '{}'::jsonb,
embedding JSONB,
created_by_job_id TEXT REFERENCES observation_generation_jobs(id) ON DELETE SET NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
FOREIGN KEY (project_id, team_id) REFERENCES projects(id, team_id) ON DELETE CASCADE
);
CREATE TABLE IF NOT EXISTS observation_sources (
id TEXT PRIMARY KEY,
observation_id TEXT NOT NULL REFERENCES observations(id) ON DELETE CASCADE,
agent_event_id TEXT REFERENCES agent_events(id) ON DELETE CASCADE,
generation_job_id TEXT REFERENCES observation_generation_jobs(id) ON DELETE SET NULL,
source_type TEXT NOT NULL CHECK (source_type IN ('agent_event', 'session_summary', 'observation_reindex', 'manual')),
source_id TEXT NOT NULL,
metadata JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (observation_id, source_type, source_id),
UNIQUE (source_type, source_id, generation_job_id, observation_id),
CHECK (
(source_type = 'agent_event' AND agent_event_id IS NOT NULL AND source_id = agent_event_id)
OR
(source_type <> 'agent_event' AND agent_event_id IS NULL)
)
);
CREATE TABLE IF NOT EXISTS observation_generation_job_events (
id TEXT PRIMARY KEY,
generation_job_id TEXT NOT NULL REFERENCES observation_generation_jobs(id) ON DELETE CASCADE,
event_type TEXT NOT NULL CHECK (event_type IN ('queued', 'enqueued', 'processing', 'retry_scheduled', 'completed', 'failed', 'cancelled')),
status_after TEXT NOT NULL CHECK (status_after IN ('queued', 'processing', 'completed', 'failed', 'cancelled')),
attempt INTEGER NOT NULL DEFAULT 0,
details JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS idx_agent_events_project_session ON agent_events(project_id, server_session_id, occurred_at);
ALTER TABLE server_sessions ADD COLUMN IF NOT EXISTS idempotency_key TEXT;
ALTER TABLE observations ADD COLUMN IF NOT EXISTS content_search TSVECTOR GENERATED ALWAYS AS (to_tsvector('english', content)) STORED;
ALTER TABLE observations DROP CONSTRAINT IF EXISTS observations_generation_key_key;
ALTER TABLE observation_generation_jobs DROP CONSTRAINT IF EXISTS observation_generation_jobs_source_type_source_id_job_type_key;
CREATE UNIQUE INDEX IF NOT EXISTS idx_server_sessions_project_idempotency
ON server_sessions(project_id, idempotency_key)
WHERE idempotency_key IS NOT NULL;
CREATE UNIQUE INDEX IF NOT EXISTS idx_observations_generation_key_scope
ON observations(team_id, project_id, generation_key)
WHERE generation_key IS NOT NULL;
CREATE UNIQUE INDEX IF NOT EXISTS idx_observation_jobs_source_scope
ON observation_generation_jobs(team_id, project_id, source_type, source_id, job_type);
CREATE INDEX IF NOT EXISTS idx_projects_team ON projects(team_id, id);
CREATE INDEX IF NOT EXISTS idx_agent_events_team_project ON agent_events(team_id, project_id, occurred_at);
CREATE INDEX IF NOT EXISTS idx_observations_project_session ON observations(project_id, server_session_id, created_at);
CREATE INDEX IF NOT EXISTS idx_observations_team_project ON observations(team_id, project_id, created_at);
CREATE INDEX IF NOT EXISTS idx_observations_content_search ON observations USING GIN (content_search);
CREATE INDEX IF NOT EXISTS idx_observation_sources_event ON observation_sources(agent_event_id);
CREATE INDEX IF NOT EXISTS idx_observation_sources_source ON observation_sources(source_type, source_id);
CREATE INDEX IF NOT EXISTS idx_observation_jobs_status_next_attempt ON observation_generation_jobs(status, next_attempt_at, created_at);
CREATE INDEX IF NOT EXISTS idx_observation_jobs_team_project ON observation_generation_jobs(team_id, project_id, status, created_at);
CREATE INDEX IF NOT EXISTS idx_observation_jobs_event ON observation_generation_jobs(agent_event_id);
CREATE INDEX IF NOT EXISTS idx_observation_jobs_source ON observation_generation_jobs(source_type, source_id);
CREATE INDEX IF NOT EXISTS idx_observation_job_events_job_created ON observation_generation_job_events(generation_job_id, created_at);
CREATE INDEX IF NOT EXISTS idx_audit_log_scope_created ON audit_log(project_id, team_id, created_at);
`;
+186
View File
@@ -0,0 +1,186 @@
// SPDX-License-Identifier: Apache-2.0
import type { JsonObject, PostgresQueryable } from './utils.js';
import { assertProjectOwnership, deterministicKey, newId, queryOne, toDate, toEpoch, toJsonObject } from './utils.js';
export interface PostgresServerSession {
id: string;
projectId: string;
teamId: string;
externalSessionId: string | null;
idempotencyKey: string | null;
contentSessionId: string | null;
agentId: string | null;
agentType: string | null;
platformSource: string | null;
generationStatus: string;
metadata: JsonObject;
startedAtEpoch: number;
endedAtEpoch: number | null;
lastGeneratedAtEpoch: number | null;
createdAtEpoch: number;
updatedAtEpoch: number;
}
interface ServerSessionRow {
id: string;
project_id: string;
team_id: string;
external_session_id: string | null;
idempotency_key: string | null;
content_session_id: string | null;
agent_id: string | null;
agent_type: string | null;
platform_source: string | null;
generation_status: string;
metadata: unknown;
started_at: Date;
ended_at: Date | null;
last_generated_at: Date | null;
created_at: Date;
updated_at: Date;
}
export class PostgresServerSessionsRepository {
constructor(private client: PostgresQueryable) {}
async create(input: {
id?: string;
projectId: string;
teamId: string;
externalSessionId?: string | null;
contentSessionId?: string | null;
agentId?: string | null;
agentType?: string | null;
platformSource?: string | null;
generationStatus?: string;
metadata?: JsonObject;
}): Promise<PostgresServerSession> {
await assertProjectOwnership(this.client, input.projectId, input.teamId);
const id = input.id ?? newId();
const idempotencyKey = buildServerSessionIdempotencyKey(input);
const row = await queryOne<ServerSessionRow>(
this.client,
`
INSERT INTO server_sessions (
id, project_id, team_id, external_session_id, idempotency_key, content_session_id,
agent_id, agent_type, platform_source, generation_status, metadata
)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11::jsonb)
ON CONFLICT (project_id, idempotency_key) WHERE idempotency_key IS NOT NULL DO UPDATE SET
external_session_id = excluded.external_session_id,
content_session_id = excluded.content_session_id,
agent_id = excluded.agent_id,
agent_type = excluded.agent_type,
platform_source = excluded.platform_source,
generation_status = excluded.generation_status,
metadata = excluded.metadata,
updated_at = now()
RETURNING *
`,
[
id,
input.projectId,
input.teamId,
input.externalSessionId ?? null,
idempotencyKey,
input.contentSessionId ?? null,
input.agentId ?? null,
input.agentType ?? null,
input.platformSource ?? null,
input.generationStatus ?? 'idle',
JSON.stringify(input.metadata ?? {})
]
);
return mapServerSessionRow(row!);
}
async getByIdForScope(input: {
id: string;
projectId: string;
teamId: string;
}): Promise<PostgresServerSession | null> {
const row = await queryOne<ServerSessionRow>(
this.client,
'SELECT * FROM server_sessions WHERE id = $1 AND project_id = $2 AND team_id = $3',
[input.id, input.projectId, input.teamId]
);
return row ? mapServerSessionRow(row) : null;
}
async listByProject(projectId: string, teamId: string): Promise<PostgresServerSession[]> {
const result = await this.client.query<ServerSessionRow>(
`
SELECT * FROM server_sessions
WHERE project_id = $1 AND team_id = $2
ORDER BY started_at DESC
`,
[projectId, teamId]
);
return result.rows.map(mapServerSessionRow);
}
}
export function buildServerSessionIdempotencyKey(input: {
projectId: string;
teamId: string;
externalSessionId?: string | null;
contentSessionId?: string | null;
agentId?: string | null;
agentType?: string | null;
platformSource?: string | null;
}): string | null {
if (input.externalSessionId) {
return `server_session:v1:${deterministicKey([
input.teamId,
input.projectId,
'external',
input.externalSessionId
])}`;
}
if (input.contentSessionId) {
return `server_session:v1:${deterministicKey([
input.teamId,
input.projectId,
'content',
input.platformSource ?? null,
input.agentId ?? null,
input.contentSessionId
])}`;
}
if (input.agentId && input.platformSource) {
return `server_session:v1:${deterministicKey([
input.teamId,
input.projectId,
'agent',
input.platformSource,
input.agentId,
input.agentType ?? null
])}`;
}
return null;
}
function mapServerSessionRow(row: ServerSessionRow): PostgresServerSession {
return {
id: row.id,
projectId: row.project_id,
teamId: row.team_id,
externalSessionId: row.external_session_id,
idempotencyKey: row.idempotency_key,
contentSessionId: row.content_session_id,
agentId: row.agent_id,
agentType: row.agent_type,
platformSource: row.platform_source,
generationStatus: row.generation_status,
metadata: toJsonObject(row.metadata),
startedAtEpoch: toEpoch(row.started_at),
endedAtEpoch: toDate(row.ended_at)?.getTime() ?? null,
lastGeneratedAtEpoch: toDate(row.last_generated_at)?.getTime() ?? null,
createdAtEpoch: toEpoch(row.created_at),
updatedAtEpoch: toEpoch(row.updated_at)
};
}
+127
View File
@@ -0,0 +1,127 @@
// SPDX-License-Identifier: Apache-2.0
import type { PostgresQueryable, JsonObject } from './utils.js';
import { newId, queryOne, toEpoch, toJsonObject } from './utils.js';
export type PostgresTeamRole = 'owner' | 'admin' | 'member' | 'viewer';
export interface PostgresTeam {
id: string;
name: string;
metadata: JsonObject;
createdAtEpoch: number;
updatedAtEpoch: number;
}
export interface PostgresTeamMember {
teamId: string;
userId: string;
role: PostgresTeamRole;
metadata: JsonObject;
createdAtEpoch: number;
updatedAtEpoch: number;
}
interface TeamRow {
id: string;
name: string;
metadata: unknown;
created_at: Date;
updated_at: Date;
}
interface TeamMemberRow {
team_id: string;
user_id: string;
role: PostgresTeamRole;
metadata: unknown;
created_at: Date;
updated_at: Date;
}
export class PostgresTeamsRepository {
constructor(private client: PostgresQueryable) {}
async create(input: { id?: string; name: string; metadata?: JsonObject }): Promise<PostgresTeam> {
const id = input.id ?? newId();
const row = await queryOne<TeamRow>(
this.client,
`
INSERT INTO teams (id, name, metadata)
VALUES ($1, $2, $3::jsonb)
RETURNING *
`,
[id, input.name, JSON.stringify(input.metadata ?? {})]
);
return mapTeamRow(row!);
}
async addMember(input: {
teamId: string;
userId: string;
role: PostgresTeamRole;
metadata?: JsonObject;
}): Promise<PostgresTeamMember> {
const row = await queryOne<TeamMemberRow>(
this.client,
`
INSERT INTO team_members (team_id, user_id, role, metadata)
VALUES ($1, $2, $3, $4::jsonb)
ON CONFLICT (team_id, user_id) DO UPDATE SET
role = excluded.role,
metadata = excluded.metadata,
updated_at = now()
RETURNING *
`,
[input.teamId, input.userId, input.role, JSON.stringify(input.metadata ?? {})]
);
return mapTeamMemberRow(row!);
}
async getByIdForUser(input: {
id: string;
userId: string;
}): Promise<PostgresTeam | null> {
const row = await queryOne<TeamRow>(
this.client,
`
SELECT teams.*
FROM teams
INNER JOIN team_members ON team_members.team_id = teams.id
WHERE teams.id = $1 AND team_members.user_id = $2
`,
[input.id, input.userId]
);
return row ? mapTeamRow(row) : null;
}
async getMember(teamId: string, userId: string): Promise<PostgresTeamMember | null> {
const row = await queryOne<TeamMemberRow>(
this.client,
'SELECT * FROM team_members WHERE team_id = $1 AND user_id = $2',
[teamId, userId]
);
return row ? mapTeamMemberRow(row) : null;
}
}
function mapTeamRow(row: TeamRow): PostgresTeam {
return {
id: row.id,
name: row.name,
metadata: toJsonObject(row.metadata),
createdAtEpoch: toEpoch(row.created_at),
updatedAtEpoch: toEpoch(row.updated_at)
};
}
function mapTeamMemberRow(row: TeamMemberRow): PostgresTeamMember {
return {
teamId: row.team_id,
userId: row.user_id,
role: row.role,
metadata: toJsonObject(row.metadata),
createdAtEpoch: toEpoch(row.created_at),
updatedAtEpoch: toEpoch(row.updated_at)
};
}
+107
View File
@@ -0,0 +1,107 @@
// SPDX-License-Identifier: Apache-2.0
import { createHash, randomUUID } from 'crypto';
import type { QueryResult, QueryResultRow } from 'pg';
export type JsonObject = Record<string, unknown>;
export type JsonValue = unknown;
export interface PostgresQueryable {
query<T extends QueryResultRow = QueryResultRow>(text: string, values?: unknown[]): Promise<QueryResult<T>>;
}
export function newId(): string {
return randomUUID();
}
export function toJsonObject(value: unknown): JsonObject {
if (value && typeof value === 'object' && !Array.isArray(value)) {
return value as JsonObject;
}
return {};
}
export function toJsonArray(value: unknown): unknown[] {
return Array.isArray(value) ? value : [];
}
export function toEpoch(value: Date | string | number): number {
if (typeof value === 'number') {
return value;
}
return new Date(value).getTime();
}
export function toDate(value: Date | string | number | null | undefined): Date | null {
if (value == null) {
return null;
}
return value instanceof Date ? value : new Date(value);
}
export async function queryOne<T extends QueryResultRow>(
client: PostgresQueryable,
text: string,
values: unknown[] = []
): Promise<T | null> {
const result = await client.query<T>(text, values);
return result.rows[0] ?? null;
}
export async function assertProjectOwnership(
client: PostgresQueryable,
projectId: string,
teamId: string
): Promise<void> {
const row = await queryOne<{ id: string }>(
client,
'SELECT id FROM projects WHERE id = $1 AND team_id = $2',
[projectId, teamId]
);
if (!row) {
throw new Error('project_id must belong to team_id');
}
}
export async function assertSessionOwnership(
client: PostgresQueryable,
serverSessionId: string,
projectId: string,
teamId: string
): Promise<void> {
const row = await queryOne<{ id: string }>(
client,
'SELECT id FROM server_sessions WHERE id = $1 AND project_id = $2 AND team_id = $3',
[serverSessionId, projectId, teamId]
);
if (!row) {
throw new Error('server_session_id must belong to project_id and team_id');
}
}
export function canonicalJson(value: unknown): string {
return JSON.stringify(sortJson(value));
}
export function deterministicKey(parts: readonly unknown[]): string {
const fingerprint = createHash('sha256')
.update(canonicalJson(parts))
.digest('hex');
return fingerprint;
}
function sortJson(value: unknown): unknown {
if (Array.isArray(value)) {
return value.map(sortJson);
}
if (value && typeof value === 'object') {
const record = value as Record<string, unknown>;
return Object.keys(record)
.sort()
.reduce<Record<string, unknown>>((acc, key) => {
acc[key] = sortJson(record[key]);
return acc;
}, {});
}
return value;
}
+82
View File
@@ -0,0 +1,82 @@
// SPDX-License-Identifier: Apache-2.0
import { randomUUID } from 'crypto';
import { Database } from 'bun:sqlite';
import { AgentEventSchema, CreateAgentEventSchema, type AgentEvent, type AgentEventSourceType, type CreateAgentEvent } from '../../core/schemas/agent-event.js';
import { ensureServerStorageSchema } from './schema.js';
interface AgentEventRow {
id: string;
project_id: string;
server_session_id: string | null;
source_type: AgentEventSourceType;
event_type: string;
payload: string;
content_session_id: string | null;
memory_session_id: string | null;
occurred_at_epoch: number;
created_at_epoch: number;
}
function mapAgentEventRow(row: AgentEventRow): AgentEvent {
return AgentEventSchema.parse({
id: row.id,
projectId: row.project_id,
serverSessionId: row.server_session_id,
sourceType: row.source_type,
eventType: row.event_type,
payload: JSON.parse(row.payload),
contentSessionId: row.content_session_id,
memorySessionId: row.memory_session_id,
occurredAtEpoch: row.occurred_at_epoch,
createdAtEpoch: row.created_at_epoch
});
}
export class AgentEventsRepository {
constructor(private db: Database) {
ensureServerStorageSchema(this.db);
}
create(input: CreateAgentEvent): AgentEvent {
const event = CreateAgentEventSchema.parse(input);
const now = Date.now();
const id = randomUUID();
this.db.prepare(`
INSERT INTO agent_events (
id, project_id, server_session_id, source_type, event_type, payload,
content_session_id, memory_session_id, occurred_at_epoch, created_at_epoch
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`).run(
id,
event.projectId,
event.serverSessionId ?? null,
event.sourceType,
event.eventType,
JSON.stringify(event.payload ?? {}),
event.contentSessionId ?? null,
event.memorySessionId ?? null,
event.occurredAtEpoch,
now
);
return this.getById(id)!;
}
getById(id: string): AgentEvent | null {
const row = this.db.prepare('SELECT * FROM agent_events WHERE id = ?').get(id) as AgentEventRow | null;
return row ? mapAgentEventRow(row) : null;
}
listByProject(projectId: string, limit = 100): AgentEvent[] {
const rows = this.db.prepare(`
SELECT * FROM agent_events
WHERE project_id = ?
ORDER BY occurred_at_epoch DESC
LIMIT ?
`).all(projectId, limit) as AgentEventRow[];
return rows.map(mapAgentEventRow);
}
}
+195
View File
@@ -0,0 +1,195 @@
// SPDX-License-Identifier: Apache-2.0
import { randomUUID } from 'crypto';
import { Database } from 'bun:sqlite';
import {
ApiKeySchema,
AuditLogSchema,
CreateApiKeySchema,
CreateAuditLogSchema,
type ApiKey,
type ApiKeyStatus,
type AuditActorType,
type AuditLog,
type CreateApiKey,
type CreateAuditLog
} from '../../core/schemas/auth.js';
import { ensureServerStorageSchema } from './schema.js';
import { parseJsonArray, parseJsonObject, stringifyJson } from './serde.js';
interface ApiKeyRow {
id: string;
team_id: string | null;
project_id: string | null;
name: string;
key_hash: string;
prefix: string | null;
scopes: string;
status: ApiKeyStatus;
last_used_at_epoch: number | null;
expires_at_epoch: number | null;
metadata: string;
created_at_epoch: number;
updated_at_epoch: number;
}
interface AuditLogRow {
id: string;
team_id: string | null;
project_id: string | null;
actor_type: AuditActorType;
actor_id: string | null;
action: string;
target_type: string | null;
target_id: string | null;
metadata: string;
created_at_epoch: number;
}
function mapApiKeyRow(row: ApiKeyRow): ApiKey {
return ApiKeySchema.parse({
id: row.id,
teamId: row.team_id,
projectId: row.project_id,
name: row.name,
keyHash: row.key_hash,
prefix: row.prefix,
scopes: parseJsonArray(row.scopes),
status: row.status,
lastUsedAtEpoch: row.last_used_at_epoch,
expiresAtEpoch: row.expires_at_epoch,
metadata: parseJsonObject(row.metadata),
createdAtEpoch: row.created_at_epoch,
updatedAtEpoch: row.updated_at_epoch
});
}
function mapAuditLogRow(row: AuditLogRow): AuditLog {
return AuditLogSchema.parse({
id: row.id,
teamId: row.team_id,
projectId: row.project_id,
actorType: row.actor_type,
actorId: row.actor_id,
action: row.action,
targetType: row.target_type,
targetId: row.target_id,
metadata: parseJsonObject(row.metadata),
createdAtEpoch: row.created_at_epoch
});
}
export class AuthRepository {
constructor(private db: Database) {
ensureServerStorageSchema(this.db);
}
createApiKey(input: CreateApiKey): ApiKey {
const key = CreateApiKeySchema.parse(input);
const now = Date.now();
const id = randomUUID();
this.db.prepare(`
INSERT INTO api_keys (
id, team_id, project_id, name, key_hash, prefix, scopes, status,
last_used_at_epoch, expires_at_epoch, metadata, created_at_epoch, updated_at_epoch
)
VALUES (?, ?, ?, ?, ?, ?, ?, 'active', NULL, ?, ?, ?, ?)
`).run(
id,
key.teamId ?? null,
key.projectId ?? null,
key.name,
key.keyHash,
key.prefix ?? null,
stringifyJson(key.scopes ?? []),
key.expiresAtEpoch ?? null,
stringifyJson(key.metadata),
now,
now
);
return this.getApiKeyById(id)!;
}
revokeApiKey(id: string, updatedAtEpoch = Date.now()): ApiKey | null {
this.db.prepare(`
UPDATE api_keys
SET status = 'revoked', updated_at_epoch = ?
WHERE id = ?
`).run(updatedAtEpoch, id);
return this.getApiKeyById(id);
}
markApiKeyUsed(id: string, usedAtEpoch = Date.now()): ApiKey | null {
this.db.prepare(`
UPDATE api_keys
SET last_used_at_epoch = ?, updated_at_epoch = ?
WHERE id = ?
`).run(usedAtEpoch, usedAtEpoch, id);
return this.getApiKeyById(id);
}
createAuditLog(input: CreateAuditLog): AuditLog {
const log = CreateAuditLogSchema.parse(input);
const now = Date.now();
const id = randomUUID();
this.db.prepare(`
INSERT INTO audit_log (
id, team_id, project_id, actor_type, actor_id, action, target_type,
target_id, metadata, created_at_epoch
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`).run(
id,
log.teamId ?? null,
log.projectId ?? null,
log.actorType,
log.actorId ?? null,
log.action,
log.targetType ?? null,
log.targetId ?? null,
stringifyJson(log.metadata),
now
);
return this.getAuditLogById(id)!;
}
getApiKeyById(id: string): ApiKey | null {
const row = this.db.prepare('SELECT * FROM api_keys WHERE id = ?').get(id) as ApiKeyRow | null;
return row ? mapApiKeyRow(row) : null;
}
getApiKeyByHash(keyHash: string): ApiKey | null {
const row = this.db.prepare('SELECT * FROM api_keys WHERE key_hash = ?').get(keyHash) as ApiKeyRow | null;
return row ? mapApiKeyRow(row) : null;
}
listApiKeys(limit = 100): ApiKey[] {
const rows = this.db.prepare(`
SELECT * FROM api_keys
ORDER BY created_at_epoch DESC
LIMIT ?
`).all(limit) as ApiKeyRow[];
return rows.map(mapApiKeyRow);
}
getAuditLogById(id: string): AuditLog | null {
const row = this.db.prepare('SELECT * FROM audit_log WHERE id = ?').get(id) as AuditLogRow | null;
return row ? mapAuditLogRow(row) : null;
}
listAuditLogByProject(projectId: string, limit = 100): AuditLog[] {
const rows = this.db.prepare(`
SELECT * FROM audit_log
WHERE project_id = ?
ORDER BY created_at_epoch DESC
LIMIT ?
`).all(projectId, limit) as AuditLogRow[];
return rows.map(mapAuditLogRow);
}
}
+9
View File
@@ -0,0 +1,9 @@
// SPDX-License-Identifier: Apache-2.0
export * from './agent-events.js';
export * from './auth.js';
export * from './memory-items.js';
export * from './projects.js';
export * from './schema.js';
export * from './server-sessions.js';
export * from './teams.js';
+275
View File
@@ -0,0 +1,275 @@
// SPDX-License-Identifier: Apache-2.0
import { randomUUID } from 'crypto';
import { Database } from 'bun:sqlite';
import {
CreateMemoryItemSchema,
CreateMemorySourceSchema,
MemoryItemSchema,
MemorySourceSchema,
type CreateMemoryItem,
type CreateMemorySource,
type MemoryItem,
type MemoryItemKind,
type MemorySource,
type MemorySourceType
} from '../../core/schemas/memory-item.js';
import { ensureServerStorageSchema } from './schema.js';
import { parseJsonArray, parseJsonObject, stringifyJson } from './serde.js';
interface MemoryItemRow {
id: string;
project_id: string;
server_session_id: string | null;
legacy_observation_id: number | null;
kind: MemoryItemKind;
type: string;
title: string | null;
subtitle: string | null;
text: string | null;
narrative: string | null;
facts: string;
concepts: string;
files_read: string;
files_modified: string;
metadata: string;
created_at_epoch: number;
updated_at_epoch: number;
}
interface MemorySourceRow {
id: string;
memory_item_id: string;
source_type: MemorySourceType;
legacy_table: string | null;
legacy_id: number | null;
source_uri: string | null;
metadata: string;
created_at_epoch: number;
}
function mapMemoryItemRow(row: MemoryItemRow): MemoryItem {
return MemoryItemSchema.parse({
id: row.id,
projectId: row.project_id,
serverSessionId: row.server_session_id,
legacyObservationId: row.legacy_observation_id,
kind: row.kind,
type: row.type,
title: row.title,
subtitle: row.subtitle,
text: row.text,
narrative: row.narrative,
facts: parseJsonArray(row.facts),
concepts: parseJsonArray(row.concepts),
filesRead: parseJsonArray(row.files_read),
filesModified: parseJsonArray(row.files_modified),
metadata: parseJsonObject(row.metadata),
createdAtEpoch: row.created_at_epoch,
updatedAtEpoch: row.updated_at_epoch
});
}
function mapMemorySourceRow(row: MemorySourceRow): MemorySource {
return MemorySourceSchema.parse({
id: row.id,
memoryItemId: row.memory_item_id,
sourceType: row.source_type,
legacyTable: row.legacy_table,
legacyId: row.legacy_id,
sourceUri: row.source_uri,
metadata: parseJsonObject(row.metadata),
createdAtEpoch: row.created_at_epoch
});
}
function buildFtsQuery(query: string): string {
return query
.normalize('NFKC')
.trim()
.split(/\s+/)
.flatMap(token => token.split(/[^\p{L}\p{N}_]+/gu))
.filter(Boolean)
.map(token => `"${token}"`)
.join(' ');
}
export class MemoryItemsRepository {
constructor(private db: Database) {
ensureServerStorageSchema(this.db);
}
create(input: CreateMemoryItem): MemoryItem {
const item = CreateMemoryItemSchema.parse(input);
const now = Date.now();
const id = randomUUID();
this.db.prepare(`
INSERT INTO memory_items (
id, project_id, server_session_id, legacy_observation_id, kind, type,
title, subtitle, text, narrative, facts, concepts, files_read,
files_modified, metadata, created_at_epoch, updated_at_epoch
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`).run(
id,
item.projectId,
item.serverSessionId ?? null,
item.legacyObservationId ?? null,
item.kind,
item.type,
item.title ?? null,
item.subtitle ?? null,
item.text ?? null,
item.narrative ?? null,
stringifyJson(item.facts ?? []),
stringifyJson(item.concepts ?? []),
stringifyJson(item.filesRead ?? []),
stringifyJson(item.filesModified ?? []),
stringifyJson(item.metadata),
now,
now
);
return this.getById(id)!;
}
addSource(input: CreateMemorySource): MemorySource {
const source = CreateMemorySourceSchema.parse(input);
const now = Date.now();
const id = randomUUID();
this.db.prepare(`
INSERT INTO memory_sources (
id, memory_item_id, source_type, legacy_table, legacy_id, source_uri,
metadata, created_at_epoch
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
`).run(
id,
source.memoryItemId,
source.sourceType,
source.legacyTable ?? null,
source.legacyId ?? null,
source.sourceUri ?? null,
stringifyJson(source.metadata),
now
);
return this.getSourceById(id)!;
}
getById(id: string): MemoryItem | null {
const row = this.db.prepare('SELECT * FROM memory_items WHERE id = ?').get(id) as MemoryItemRow | null;
return row ? mapMemoryItemRow(row) : null;
}
getByLegacyObservationId(legacyObservationId: number): MemoryItem | null {
const row = this.db.prepare('SELECT * FROM memory_items WHERE legacy_observation_id = ?').get(legacyObservationId) as MemoryItemRow | null;
return row ? mapMemoryItemRow(row) : null;
}
update(id: string, input: Partial<CreateMemoryItem>): MemoryItem | null {
const existing = this.getById(id);
if (!existing) {
return null;
}
const next = CreateMemoryItemSchema.parse({
projectId: input.projectId ?? existing.projectId,
serverSessionId: input.serverSessionId ?? existing.serverSessionId,
legacyObservationId: input.legacyObservationId ?? existing.legacyObservationId,
kind: input.kind ?? existing.kind,
type: input.type ?? existing.type,
title: input.title ?? existing.title,
subtitle: input.subtitle ?? existing.subtitle,
text: input.text ?? existing.text,
narrative: input.narrative ?? existing.narrative,
facts: input.facts ?? existing.facts,
concepts: input.concepts ?? existing.concepts,
filesRead: input.filesRead ?? existing.filesRead,
filesModified: input.filesModified ?? existing.filesModified,
metadata: input.metadata ?? existing.metadata,
});
const now = Date.now();
this.db.prepare(`
UPDATE memory_items
SET
project_id = ?,
server_session_id = ?,
legacy_observation_id = ?,
kind = ?,
type = ?,
title = ?,
subtitle = ?,
text = ?,
narrative = ?,
facts = ?,
concepts = ?,
files_read = ?,
files_modified = ?,
metadata = ?,
updated_at_epoch = ?
WHERE id = ?
`).run(
next.projectId,
next.serverSessionId ?? null,
next.legacyObservationId ?? null,
next.kind,
next.type,
next.title ?? null,
next.subtitle ?? null,
next.text ?? null,
next.narrative ?? null,
stringifyJson(next.facts ?? []),
stringifyJson(next.concepts ?? []),
stringifyJson(next.filesRead ?? []),
stringifyJson(next.filesModified ?? []),
stringifyJson(next.metadata),
now,
id,
);
return this.getById(id);
}
getSourceById(id: string): MemorySource | null {
const row = this.db.prepare('SELECT * FROM memory_sources WHERE id = ?').get(id) as MemorySourceRow | null;
return row ? mapMemorySourceRow(row) : null;
}
listByProject(projectId: string, limit = 100): MemoryItem[] {
const rows = this.db.prepare(`
SELECT * FROM memory_items
WHERE project_id = ?
ORDER BY created_at_epoch DESC
LIMIT ?
`).all(projectId, limit) as MemoryItemRow[];
return rows.map(mapMemoryItemRow);
}
search(projectId: string, query: string, limit = 20): MemoryItem[] {
const ftsQuery = buildFtsQuery(query);
if (!ftsQuery) return [];
const rows = this.db.prepare(`
SELECT memory_items.*
FROM memory_items
JOIN memory_items_fts ON memory_items_fts.memory_item_id = memory_items.id
WHERE memory_items_fts.project_id = ?
AND memory_items_fts MATCH ?
ORDER BY memory_items.updated_at_epoch DESC
LIMIT ?
`).all(projectId, ftsQuery, limit) as MemoryItemRow[];
return rows.map(mapMemoryItemRow);
}
listSources(memoryItemId: string): MemorySource[] {
const rows = this.db.prepare(`
SELECT * FROM memory_sources
WHERE memory_item_id = ?
ORDER BY created_at_epoch ASC
`).all(memoryItemId) as MemorySourceRow[];
return rows.map(mapMemorySourceRow);
}
}
+90
View File
@@ -0,0 +1,90 @@
// SPDX-License-Identifier: Apache-2.0
import { randomUUID } from 'crypto';
import { Database } from 'bun:sqlite';
import { CreateProjectSchema, ProjectSchema, type CreateProject, type Project } from '../../core/schemas/project.js';
import { ensureServerStorageSchema } from './schema.js';
import { parseJsonObject, stringifyJson } from './serde.js';
interface ProjectRow {
id: string;
name: string;
slug: string | null;
root_path: string | null;
metadata: string;
created_at_epoch: number;
updated_at_epoch: number;
}
function mapProjectRow(row: ProjectRow): Project {
return ProjectSchema.parse({
id: row.id,
name: row.name,
slug: row.slug,
rootPath: row.root_path,
metadata: parseJsonObject(row.metadata),
createdAtEpoch: row.created_at_epoch,
updatedAtEpoch: row.updated_at_epoch
});
}
export class ProjectsRepository {
constructor(private db: Database) {
ensureServerStorageSchema(this.db);
}
create(input: CreateProject): Project {
const project = CreateProjectSchema.parse(input);
const now = Date.now();
const id = randomUUID();
this.db.prepare(`
INSERT INTO projects (id, name, slug, root_path, metadata, created_at_epoch, updated_at_epoch)
VALUES (?, ?, ?, ?, ?, ?, ?)
`).run(
id,
project.name,
project.slug ?? null,
project.rootPath ?? null,
stringifyJson(project.metadata),
now,
now
);
return this.getById(id)!;
}
upsert(input: CreateProject & { id?: string }): Project {
const project = CreateProjectSchema.parse(input);
const now = Date.now();
const id = input.id ?? randomUUID();
this.db.prepare(`
INSERT INTO projects (id, name, slug, root_path, metadata, created_at_epoch, updated_at_epoch)
VALUES (?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(id) DO UPDATE SET
name = excluded.name,
slug = excluded.slug,
root_path = excluded.root_path,
metadata = excluded.metadata,
updated_at_epoch = excluded.updated_at_epoch
`).run(id, project.name, project.slug ?? null, project.rootPath ?? null, stringifyJson(project.metadata), now, now);
return this.getById(id)!;
}
getById(id: string): Project | null {
const row = this.db.prepare('SELECT * FROM projects WHERE id = ?').get(id) as ProjectRow | null;
return row ? mapProjectRow(row) : null;
}
getByRootPath(rootPath: string): Project | null {
const row = this.db.prepare('SELECT * FROM projects WHERE root_path = ?').get(rootPath) as ProjectRow | null;
return row ? mapProjectRow(row) : null;
}
list(): Project[] {
const rows = this.db.prepare('SELECT * FROM projects ORDER BY updated_at_epoch DESC, name ASC').all() as ProjectRow[];
return rows.map(mapProjectRow);
}
}
+305
View File
@@ -0,0 +1,305 @@
// SPDX-License-Identifier: Apache-2.0
import { Database } from 'bun:sqlite';
export const SERVER_STORAGE_SCHEMA_VERSION = 33;
export const SERVER_OWNED_TABLES = [
'projects',
'server_sessions',
'agent_events',
'memory_items',
'memory_sources',
'teams',
'team_members',
'api_keys',
'audit_log'
] as const;
const initializedDatabases = new WeakSet<Database>();
export function ensureServerStorageSchema(db: Database): void {
if (initializedDatabases.has(db)) return;
db.run(`
CREATE TABLE IF NOT EXISTS projects (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
slug TEXT UNIQUE,
root_path TEXT UNIQUE,
metadata TEXT NOT NULL DEFAULT '{}',
created_at_epoch INTEGER NOT NULL,
updated_at_epoch INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS teams (
id TEXT PRIMARY KEY,
name TEXT NOT NULL,
slug TEXT UNIQUE,
metadata TEXT NOT NULL DEFAULT '{}',
created_at_epoch INTEGER NOT NULL,
updated_at_epoch INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS team_members (
id TEXT PRIMARY KEY,
team_id TEXT NOT NULL,
user_id TEXT NOT NULL,
role TEXT NOT NULL CHECK(role IN ('owner', 'admin', 'member', 'viewer')),
metadata TEXT NOT NULL DEFAULT '{}',
created_at_epoch INTEGER NOT NULL,
FOREIGN KEY(team_id) REFERENCES teams(id) ON DELETE CASCADE,
UNIQUE(team_id, user_id)
);
CREATE TABLE IF NOT EXISTS server_sessions (
id TEXT PRIMARY KEY,
project_id TEXT NOT NULL,
content_session_id TEXT,
memory_session_id TEXT,
platform_source TEXT NOT NULL DEFAULT 'claude',
title TEXT,
status TEXT NOT NULL DEFAULT 'active' CHECK(status IN ('active', 'completed', 'failed')),
metadata TEXT NOT NULL DEFAULT '{}',
started_at_epoch INTEGER NOT NULL,
completed_at_epoch INTEGER,
updated_at_epoch INTEGER NOT NULL,
FOREIGN KEY(project_id) REFERENCES projects(id) ON DELETE CASCADE
);
CREATE TABLE IF NOT EXISTS agent_events (
id TEXT PRIMARY KEY,
project_id TEXT NOT NULL,
server_session_id TEXT,
source_type TEXT NOT NULL CHECK(source_type IN ('hook', 'worker', 'provider', 'server', 'api')),
event_type TEXT NOT NULL,
payload TEXT NOT NULL DEFAULT '{}',
content_session_id TEXT,
memory_session_id TEXT,
occurred_at_epoch INTEGER NOT NULL,
created_at_epoch INTEGER NOT NULL,
FOREIGN KEY(project_id) REFERENCES projects(id) ON DELETE CASCADE,
FOREIGN KEY(server_session_id) REFERENCES server_sessions(id) ON DELETE SET NULL
);
CREATE TABLE IF NOT EXISTS memory_items (
id TEXT PRIMARY KEY,
project_id TEXT NOT NULL,
server_session_id TEXT,
legacy_observation_id INTEGER,
kind TEXT NOT NULL CHECK(kind IN ('observation', 'summary', 'prompt', 'manual')),
type TEXT NOT NULL,
title TEXT,
subtitle TEXT,
text TEXT,
narrative TEXT,
facts TEXT NOT NULL DEFAULT '[]',
concepts TEXT NOT NULL DEFAULT '[]',
files_read TEXT NOT NULL DEFAULT '[]',
files_modified TEXT NOT NULL DEFAULT '[]',
metadata TEXT NOT NULL DEFAULT '{}',
created_at_epoch INTEGER NOT NULL,
updated_at_epoch INTEGER NOT NULL,
FOREIGN KEY(project_id) REFERENCES projects(id) ON DELETE CASCADE,
FOREIGN KEY(server_session_id) REFERENCES server_sessions(id) ON DELETE SET NULL
);
CREATE TABLE IF NOT EXISTS memory_sources (
id TEXT PRIMARY KEY,
memory_item_id TEXT NOT NULL,
source_type TEXT NOT NULL CHECK(source_type IN ('observation', 'session_summary', 'user_prompt', 'manual', 'import')),
legacy_table TEXT,
legacy_id INTEGER,
source_uri TEXT,
metadata TEXT NOT NULL DEFAULT '{}',
created_at_epoch INTEGER NOT NULL,
FOREIGN KEY(memory_item_id) REFERENCES memory_items(id) ON DELETE CASCADE
);
CREATE TABLE IF NOT EXISTS api_keys (
id TEXT PRIMARY KEY,
team_id TEXT,
project_id TEXT,
name TEXT NOT NULL,
key_hash TEXT NOT NULL UNIQUE,
prefix TEXT,
scopes TEXT NOT NULL DEFAULT '[]',
status TEXT NOT NULL DEFAULT 'active' CHECK(status IN ('active', 'revoked')),
last_used_at_epoch INTEGER,
expires_at_epoch INTEGER,
metadata TEXT NOT NULL DEFAULT '{}',
created_at_epoch INTEGER NOT NULL,
updated_at_epoch INTEGER NOT NULL,
FOREIGN KEY(team_id) REFERENCES teams(id) ON DELETE CASCADE,
FOREIGN KEY(project_id) REFERENCES projects(id) ON DELETE CASCADE
);
CREATE TABLE IF NOT EXISTS audit_log (
id TEXT PRIMARY KEY,
team_id TEXT,
project_id TEXT,
actor_type TEXT NOT NULL CHECK(actor_type IN ('user', 'api_key', 'system')),
actor_id TEXT,
action TEXT NOT NULL,
target_type TEXT,
target_id TEXT,
metadata TEXT NOT NULL DEFAULT '{}',
created_at_epoch INTEGER NOT NULL,
FOREIGN KEY(team_id) REFERENCES teams(id) ON DELETE SET NULL,
FOREIGN KEY(project_id) REFERENCES projects(id) ON DELETE SET NULL
);
`);
db.run('CREATE INDEX IF NOT EXISTS idx_projects_root_path ON projects(root_path)');
db.run('CREATE INDEX IF NOT EXISTS idx_server_sessions_project ON server_sessions(project_id)');
db.run('CREATE INDEX IF NOT EXISTS idx_server_sessions_content ON server_sessions(content_session_id)');
db.run('CREATE INDEX IF NOT EXISTS idx_server_sessions_memory ON server_sessions(memory_session_id)');
db.run('CREATE INDEX IF NOT EXISTS idx_server_sessions_status ON server_sessions(status)');
db.run('CREATE INDEX IF NOT EXISTS idx_agent_events_project_time ON agent_events(project_id, occurred_at_epoch DESC)');
db.run('CREATE INDEX IF NOT EXISTS idx_agent_events_session_time ON agent_events(server_session_id, occurred_at_epoch DESC)');
db.run('CREATE INDEX IF NOT EXISTS idx_agent_events_type ON agent_events(event_type)');
db.run('CREATE INDEX IF NOT EXISTS idx_memory_items_project_time ON memory_items(project_id, created_at_epoch DESC)');
db.run('CREATE INDEX IF NOT EXISTS idx_memory_items_session_time ON memory_items(server_session_id, created_at_epoch DESC)');
db.run('CREATE INDEX IF NOT EXISTS idx_memory_items_legacy_observation ON memory_items(legacy_observation_id)');
db.run(`
CREATE UNIQUE INDEX IF NOT EXISTS ux_memory_items_legacy_observation
ON memory_items(legacy_observation_id)
WHERE legacy_observation_id IS NOT NULL
`);
db.run('CREATE INDEX IF NOT EXISTS idx_memory_items_kind_type ON memory_items(kind, type)');
db.run(`
CREATE VIRTUAL TABLE IF NOT EXISTS memory_items_fts USING fts5(
memory_item_id UNINDEXED,
project_id UNINDEXED,
title,
subtitle,
text,
narrative,
facts,
concepts,
tokenize='porter unicode61'
)
`);
const memoryItemCount = db.prepare('SELECT COUNT(*) AS count FROM memory_items').get() as { count: number };
const ftsItemCount = db.prepare('SELECT COUNT(*) AS count FROM memory_items_fts').get() as { count: number };
if (memoryItemCount.count !== ftsItemCount.count) {
const rebuildMemoryItemsFts = db.transaction(() => {
db.run('DELETE FROM memory_items_fts');
db.run(`
INSERT INTO memory_items_fts (
memory_item_id, project_id, title, subtitle, text, narrative, facts, concepts
)
SELECT id, project_id, title, subtitle, text, narrative, facts, concepts
FROM memory_items
`);
});
rebuildMemoryItemsFts();
}
db.run('CREATE INDEX IF NOT EXISTS idx_memory_sources_item ON memory_sources(memory_item_id)');
db.run('CREATE INDEX IF NOT EXISTS idx_memory_sources_legacy ON memory_sources(legacy_table, legacy_id)');
db.run(`
CREATE UNIQUE INDEX IF NOT EXISTS ux_memory_sources_legacy_source
ON memory_sources(source_type, legacy_table, legacy_id)
WHERE legacy_table IS NOT NULL AND legacy_id IS NOT NULL
`);
db.run('CREATE INDEX IF NOT EXISTS idx_team_members_team ON team_members(team_id)');
db.run('CREATE INDEX IF NOT EXISTS idx_api_keys_team ON api_keys(team_id)');
db.run('CREATE INDEX IF NOT EXISTS idx_api_keys_project ON api_keys(project_id)');
db.run('CREATE INDEX IF NOT EXISTS idx_api_keys_prefix ON api_keys(prefix)');
db.run('CREATE INDEX IF NOT EXISTS idx_audit_log_team_time ON audit_log(team_id, created_at_epoch DESC)');
db.run('CREATE INDEX IF NOT EXISTS idx_audit_log_project_time ON audit_log(project_id, created_at_epoch DESC)');
db.run('CREATE INDEX IF NOT EXISTS idx_audit_log_actor ON audit_log(actor_type, actor_id)');
db.run(`
CREATE TRIGGER IF NOT EXISTS trg_server_sessions_project_update
BEFORE UPDATE OF project_id ON server_sessions
WHEN EXISTS (
SELECT 1 FROM agent_events
WHERE server_session_id = OLD.id AND project_id <> NEW.project_id
)
OR EXISTS (
SELECT 1 FROM memory_items
WHERE server_session_id = OLD.id AND project_id <> NEW.project_id
)
BEGIN
SELECT RAISE(ABORT, 'server_sessions project_id cannot change while children belong to the previous project');
END;
CREATE TRIGGER IF NOT EXISTS trg_agent_events_session_project_insert
BEFORE INSERT ON agent_events
WHEN NEW.server_session_id IS NOT NULL
AND NOT EXISTS (
SELECT 1 FROM server_sessions
WHERE id = NEW.server_session_id AND project_id = NEW.project_id
)
BEGIN
SELECT RAISE(ABORT, 'agent_events server_session_id must belong to project_id');
END;
CREATE TRIGGER IF NOT EXISTS trg_agent_events_session_project_update
BEFORE UPDATE OF project_id, server_session_id ON agent_events
WHEN NEW.server_session_id IS NOT NULL
AND NOT EXISTS (
SELECT 1 FROM server_sessions
WHERE id = NEW.server_session_id AND project_id = NEW.project_id
)
BEGIN
SELECT RAISE(ABORT, 'agent_events server_session_id must belong to project_id');
END;
CREATE TRIGGER IF NOT EXISTS trg_memory_items_session_project_insert
BEFORE INSERT ON memory_items
WHEN NEW.server_session_id IS NOT NULL
AND NOT EXISTS (
SELECT 1 FROM server_sessions
WHERE id = NEW.server_session_id AND project_id = NEW.project_id
)
BEGIN
SELECT RAISE(ABORT, 'memory_items server_session_id must belong to project_id');
END;
CREATE TRIGGER IF NOT EXISTS trg_memory_items_session_project_update
BEFORE UPDATE OF project_id, server_session_id ON memory_items
WHEN NEW.server_session_id IS NOT NULL
AND NOT EXISTS (
SELECT 1 FROM server_sessions
WHERE id = NEW.server_session_id AND project_id = NEW.project_id
)
BEGIN
SELECT RAISE(ABORT, 'memory_items server_session_id must belong to project_id');
END;
`);
db.run(`
CREATE TRIGGER IF NOT EXISTS trg_memory_items_fts_insert
AFTER INSERT ON memory_items
BEGIN
INSERT INTO memory_items_fts (
memory_item_id, project_id, title, subtitle, text, narrative, facts, concepts
)
VALUES (
new.id, new.project_id, new.title, new.subtitle, new.text, new.narrative, new.facts, new.concepts
);
END;
CREATE TRIGGER IF NOT EXISTS trg_memory_items_fts_update
AFTER UPDATE ON memory_items
BEGIN
DELETE FROM memory_items_fts WHERE memory_item_id = old.id;
INSERT INTO memory_items_fts (
memory_item_id, project_id, title, subtitle, text, narrative, facts, concepts
)
VALUES (
new.id, new.project_id, new.title, new.subtitle, new.text, new.narrative, new.facts, new.concepts
);
END;
CREATE TRIGGER IF NOT EXISTS trg_memory_items_fts_delete
AFTER DELETE ON memory_items
BEGIN
DELETE FROM memory_items_fts WHERE memory_item_id = old.id;
END;
`);
initializedDatabases.add(db);
}
+25
View File
@@ -0,0 +1,25 @@
// SPDX-License-Identifier: Apache-2.0
export function stringifyJson(value: unknown): string {
return JSON.stringify(value ?? {});
}
export function parseJsonObject(value: string | null | undefined): Record<string, unknown> {
if (!value) return {};
try {
const parsed = JSON.parse(value) as unknown;
return parsed && typeof parsed === 'object' && !Array.isArray(parsed) ? parsed as Record<string, unknown> : {};
} catch {
return {};
}
}
export function parseJsonArray(value: string | null | undefined): string[] {
if (!value) return [];
try {
const parsed = JSON.parse(value) as unknown;
return Array.isArray(parsed) ? parsed.filter((item): item is string => typeof item === 'string') : [];
} catch {
return [];
}
}
+96
View File
@@ -0,0 +1,96 @@
// SPDX-License-Identifier: Apache-2.0
import { randomUUID } from 'crypto';
import { Database } from 'bun:sqlite';
import { CreateServerSessionSchema, ServerSessionSchema, type CreateServerSession, type ServerSession, type ServerSessionStatus } from '../../core/schemas/session.js';
import { ensureServerStorageSchema } from './schema.js';
import { parseJsonObject, stringifyJson } from './serde.js';
interface ServerSessionRow {
id: string;
project_id: string;
content_session_id: string | null;
memory_session_id: string | null;
platform_source: string;
title: string | null;
status: ServerSessionStatus;
metadata: string;
started_at_epoch: number;
completed_at_epoch: number | null;
updated_at_epoch: number;
}
function mapServerSessionRow(row: ServerSessionRow): ServerSession {
return ServerSessionSchema.parse({
id: row.id,
projectId: row.project_id,
contentSessionId: row.content_session_id,
memorySessionId: row.memory_session_id,
platformSource: row.platform_source,
title: row.title,
status: row.status,
metadata: parseJsonObject(row.metadata),
startedAtEpoch: row.started_at_epoch,
completedAtEpoch: row.completed_at_epoch,
updatedAtEpoch: row.updated_at_epoch
});
}
export class ServerSessionsRepository {
constructor(private db: Database) {
ensureServerStorageSchema(this.db);
}
create(input: CreateServerSession): ServerSession {
const session = CreateServerSessionSchema.parse(input);
const now = Date.now();
const id = randomUUID();
this.db.prepare(`
INSERT INTO server_sessions (
id, project_id, content_session_id, memory_session_id, platform_source,
title, status, metadata, started_at_epoch, completed_at_epoch, updated_at_epoch
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`).run(
id,
session.projectId,
session.contentSessionId ?? null,
session.memorySessionId ?? null,
session.platformSource ?? 'claude',
session.title ?? null,
'active',
stringifyJson(session.metadata),
now,
null,
now
);
return this.getById(id)!;
}
markCompleted(id: string, completedAtEpoch = Date.now()): ServerSession | null {
this.db.prepare(`
UPDATE server_sessions
SET status = 'completed', completed_at_epoch = ?, updated_at_epoch = ?
WHERE id = ?
`).run(completedAtEpoch, completedAtEpoch, id);
return this.getById(id);
}
getById(id: string): ServerSession | null {
const row = this.db.prepare('SELECT * FROM server_sessions WHERE id = ?').get(id) as ServerSessionRow | null;
return row ? mapServerSessionRow(row) : null;
}
getByMemorySessionId(memorySessionId: string): ServerSession | null {
const row = this.db.prepare('SELECT * FROM server_sessions WHERE memory_session_id = ? ORDER BY started_at_epoch DESC LIMIT 1').get(memorySessionId) as ServerSessionRow | null;
return row ? mapServerSessionRow(row) : null;
}
listByProject(projectId: string): ServerSession[] {
const rows = this.db.prepare('SELECT * FROM server_sessions WHERE project_id = ? ORDER BY started_at_epoch DESC').all(projectId) as ServerSessionRow[];
return rows.map(mapServerSessionRow);
}
}
+97
View File
@@ -0,0 +1,97 @@
// SPDX-License-Identifier: Apache-2.0
import { randomUUID } from 'crypto';
import { Database } from 'bun:sqlite';
import { CreateTeamMemberSchema, CreateTeamSchema, TeamMemberSchema, TeamSchema, type CreateTeam, type CreateTeamMember, type Team, type TeamMember, type TeamRole } from '../../core/schemas/team.js';
import { ensureServerStorageSchema } from './schema.js';
import { parseJsonObject, stringifyJson } from './serde.js';
interface TeamRow {
id: string;
name: string;
slug: string | null;
metadata: string;
created_at_epoch: number;
updated_at_epoch: number;
}
interface TeamMemberRow {
id: string;
team_id: string;
user_id: string;
role: TeamRole;
metadata: string;
created_at_epoch: number;
}
function mapTeamRow(row: TeamRow): Team {
return TeamSchema.parse({
id: row.id,
name: row.name,
slug: row.slug,
metadata: parseJsonObject(row.metadata),
createdAtEpoch: row.created_at_epoch,
updatedAtEpoch: row.updated_at_epoch
});
}
function mapTeamMemberRow(row: TeamMemberRow): TeamMember {
return TeamMemberSchema.parse({
id: row.id,
teamId: row.team_id,
userId: row.user_id,
role: row.role,
metadata: parseJsonObject(row.metadata),
createdAtEpoch: row.created_at_epoch
});
}
export class TeamsRepository {
constructor(private db: Database) {
ensureServerStorageSchema(this.db);
}
create(input: CreateTeam): Team {
const team = CreateTeamSchema.parse(input);
const now = Date.now();
const id = randomUUID();
this.db.prepare(`
INSERT INTO teams (id, name, slug, metadata, created_at_epoch, updated_at_epoch)
VALUES (?, ?, ?, ?, ?, ?)
`).run(id, team.name, team.slug ?? null, stringifyJson(team.metadata), now, now);
return this.getById(id)!;
}
addMember(input: CreateTeamMember): TeamMember {
const member = CreateTeamMemberSchema.parse(input);
const now = Date.now();
const id = randomUUID();
this.db.prepare(`
INSERT INTO team_members (id, team_id, user_id, role, metadata, created_at_epoch)
VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT(team_id, user_id) DO UPDATE SET
role = excluded.role,
metadata = excluded.metadata
`).run(id, member.teamId, member.userId, member.role, stringifyJson(member.metadata), now);
return this.getMember(member.teamId, member.userId)!;
}
getById(id: string): Team | null {
const row = this.db.prepare('SELECT * FROM teams WHERE id = ?').get(id) as TeamRow | null;
return row ? mapTeamRow(row) : null;
}
getMember(teamId: string, userId: string): TeamMember | null {
const row = this.db.prepare('SELECT * FROM team_members WHERE team_id = ? AND user_id = ?').get(teamId, userId) as TeamMemberRow | null;
return row ? mapTeamMemberRow(row) : null;
}
listMembers(teamId: string): TeamMember[] {
const rows = this.db.prepare('SELECT * FROM team_members WHERE team_id = ? ORDER BY created_at_epoch ASC').all(teamId) as TeamMemberRow[];
return rows.map(mapTeamMemberRow);
}
}