Server-beta: Postgres storage + independent runtime + BullMQ queue (Phases 1–3) (#2351)
* Add server beta runtime foundation * Address server beta review findings * Resolve server beta review comments * Tighten server beta review follow-ups * Harden server beta auth and search * Avoid unnecessary FTS rebuilds * Block scoped keys from creating projects * Release BullMQ claims best effort on close * Address server beta review blockers * Reset BullMQ claims best effort * Add Postgres observation storage foundation * feat(server-beta): add independent runtime service Introduce src/server/runtime/ as a self-contained server-beta runtime that owns its lifecycle, Postgres bootstrap, and HTTP boundary without depending on WorkerService. ServerBetaService wraps the existing Server class, exposes /healthz and /v1/info with runtime="server-beta", and persists state to dedicated paths (.server-beta.pid|.port|.runtime.json). The four boundary managers (queue, generation worker, provider registry, event broadcaster) are intentionally disabled in this phase and report their status through /v1/info; later phases activate them. Adds plans/2026-05-07-finish-bullmq-branch-ship-plan.md to track the remaining work for this branch. Phase 2 of plans/2026-05-07-server-beta-independent-bullmq-observation-runtime.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(server-beta): route CLI lifecycle and bundle separate runtime scripts/build-hooks.js now produces plugin/scripts/server-beta-service.cjs as a separate Node CJS bundle, alongside the existing worker-service bundle. The server-beta runtime is now installable independently. src/npx-cli/commands/server.ts routes start|stop|restart|status to the server-beta lifecycle instead of the legacy worker. The worker keeps its own start|stop|restart|status under the worker namespace; the two runtimes can be operated independently. src/services/worker-service.ts adds a server-* command parser branch that delegates to the sibling server-beta-service.cjs bundle so direct worker-service invocations still route to the right runtime. tests/npx-cli-server-namespace.test.ts updated to expect server-beta lifecycle routing. Includes rebuilt plugin/scripts/*.cjs bundles produced by build-and-sync. Phase 2 of plans/2026-05-07-server-beta-independent-bullmq-observation-runtime.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(server-beta): add BullMQ job queue primitives Introduce src/server/jobs/ as the queue-side primitives that Phase 3 of the server-beta runtime needs to operate. types.ts defines a discriminated union over the four job kinds (event, event-batch, summary, reindex) and maps each to a per-kind BullMQ queue name and deterministic-ID prefix. job-id.ts builds deterministic, colon-free BullMQ jobIds from (kind, team, project, source). The colon ban exists because BullMQ uses ':' as a Redis key separator internally; embedding ':' in jobIds breaks scan and state lookups. ServerJobQueue.ts is a thin wrapper over BullMQ Queue + Worker that enforces autorun:false, default concurrency 1, and an attached error listener — all per BullMQ docs requirements. Test seams accept queue and worker factories so unit tests do not need Redis. outbox.ts publishes through the Postgres ObservationGenerationJob repository as canonical history. enqueueOutbox writes the row first, then publishes to BullMQ; if BullMQ throws, the row is transitioned to failed and a failed event is appended. reconcileOnStartup re-enqueues queued + processing rows after a restart, replacing terminal BullMQ jobs that may still be holding the deterministic ID slot. markCompleted and markFailed wrap transitionStatus and append the matching event row. Includes 20 unit tests covering deterministic ID stability, colon-free output, queue lifecycle, error-listener attachment, double-start refusal, idempotent enqueue, BullMQ failure rollback, startup reconciliation, max-attempts skipping, and completion / failure / retry transitions. Phase 3 commit 1 of plans/2026-05-07-server-beta-independent-bullmq-observation-runtime.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(server-beta): activate queue boundary in runtime service Wire ActiveServerBetaQueueManager into the server-beta runtime graph. The active manager owns one ServerJobQueue per generation kind (event, event-batch, summary, reindex) and surfaces lane metadata through boundary health. Selection is opt-in and fail-fast: if CLAUDE_MEM_QUEUE_ENGINE is set to bullmq the active manager is constructed (and any Redis/config error throws — no silent fallback to SQLite, per Phase 3 anti-pattern guard). For any other engine the disabled boundary remains so worker-era and test setups stay compatible. Widens ServerBetaBoundaryHealth.status to a discriminated union ('disabled' | 'active' | 'errored') with optional details. The disabled adapter still emits status='disabled', which keeps the existing server-beta-service test green. ServerBetaService receives the manager through a new optional queueManager field on CreateServerBetaServiceOptions so test graphs and Phase 4 wiring can inject custom managers. Adds tests/server/runtime/active-queue-manager.test.ts covering bullmq guard, active health shape, per-kind queue access, close behavior, and post-close errored health. Phase 3 commit 2 of plans/2026-05-07-server-beta-independent-bullmq-observation-runtime.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(server-beta): cap /v1/events/batch at 500 events Prevents unbounded array DoS surface flagged in PR review. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alex Newman
@@ -619,6 +619,7 @@ function mergeSettings(updates: Record<string, string>): boolean {
|
||||
type ProviderId = 'claude' | 'gemini' | 'openrouter';
|
||||
type ClaudeAccessMode = 'subscription' | 'api-key';
|
||||
type ClaudeApiMode = 'direct' | 'gateway';
|
||||
type RuntimeId = 'worker' | 'server-beta';
|
||||
|
||||
function readRawStoredAuthMethod(): 'subscription' | 'api-key' | 'gateway' | undefined {
|
||||
try {
|
||||
@@ -642,6 +643,32 @@ function resolveClaudeAuthMethod(): 'subscription' | 'api-key' | 'gateway' {
|
||||
return 'subscription';
|
||||
}
|
||||
|
||||
async function promptRuntime(): Promise<RuntimeId> {
|
||||
if (!isInteractive) {
|
||||
mergeSettings({ CLAUDE_MEM_RUNTIME: 'worker' });
|
||||
return 'worker';
|
||||
}
|
||||
|
||||
const selected = await p.select<RuntimeId>({
|
||||
message: 'Which runtime should claude-mem start after install?',
|
||||
options: [
|
||||
{ value: 'worker', label: 'Worker', hint: 'stable compatibility path' },
|
||||
{ value: 'server-beta', label: 'Server (beta)', hint: 'REST V1, API keys, team-ready storage' },
|
||||
],
|
||||
initialValue: 'worker',
|
||||
});
|
||||
|
||||
if (p.isCancel(selected)) {
|
||||
p.cancel('Installation cancelled.');
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
mergeSettings({
|
||||
CLAUDE_MEM_RUNTIME: selected,
|
||||
});
|
||||
return selected;
|
||||
}
|
||||
|
||||
async function promptProvider(options: InstallOptions): Promise<ProviderId> {
|
||||
const initialProvider = (getSetting('CLAUDE_MEM_PROVIDER') as ProviderId) || 'claude';
|
||||
|
||||
@@ -1025,6 +1052,7 @@ export async function runInstallCommand(options: InstallOptions = {}): Promise<v
|
||||
selectedIDEs = ['claude-code'];
|
||||
}
|
||||
|
||||
const selectedRuntime = await promptRuntime();
|
||||
const selectedProvider = await promptProvider(options);
|
||||
if (selectedProvider === 'claude') {
|
||||
await promptClaudeModel(options);
|
||||
@@ -1169,7 +1197,7 @@ export async function runInstallCommand(options: InstallOptions = {}): Promise<v
|
||||
|
||||
await runTasks([
|
||||
{
|
||||
title: 'Starting worker daemon',
|
||||
title: selectedRuntime === 'server-beta' ? 'Starting server beta daemon' : 'Starting worker daemon',
|
||||
task: async (message) => {
|
||||
if (autoStartSkipped) {
|
||||
return isInteractive
|
||||
@@ -1180,15 +1208,15 @@ export async function runInstallCommand(options: InstallOptions = {}): Promise<v
|
||||
const marketplaceScriptPath = join(marketplaceDirectory(), 'plugin', 'scripts', 'worker-service.cjs');
|
||||
const cacheScriptPath = join(pluginCacheDirectory(version), 'scripts', 'worker-service.cjs');
|
||||
const scriptPath = existsSync(marketplaceScriptPath) ? marketplaceScriptPath : cacheScriptPath;
|
||||
message(`Spawning worker on port ${port}...`);
|
||||
message(`Spawning ${selectedRuntime === 'server-beta' ? 'server beta' : 'worker'} on port ${port}...`);
|
||||
workerStartResult = await ensureWorkerStarted(port, scriptPath);
|
||||
switch (workerStartResult) {
|
||||
case 'ready':
|
||||
return `Worker ready at http://localhost:${port} ${pc.green('OK')}`;
|
||||
return `${selectedRuntime === 'server-beta' ? 'Server beta' : 'Worker'} ready at http://localhost:${port} ${pc.green('OK')}`;
|
||||
case 'warming':
|
||||
return `Worker starting on port ${port} — finishing in background ${pc.yellow('⏳')}`;
|
||||
return `${selectedRuntime === 'server-beta' ? 'Server beta' : 'Worker'} starting on port ${port} — finishing in background ${pc.yellow('⏳')}`;
|
||||
case 'dead':
|
||||
return `Worker did not start — try \`npx claude-mem start\` manually ${pc.yellow('!')}`;
|
||||
return `${selectedRuntime === 'server-beta' ? 'Server beta' : 'Worker'} did not start — try \`${selectedRuntime === 'server-beta' ? 'npx claude-mem server start' : 'npx claude-mem start'}\` manually ${pc.yellow('!')}`;
|
||||
}
|
||||
},
|
||||
},
|
||||
@@ -1256,11 +1284,13 @@ export async function runInstallCommand(options: InstallOptions = {}): Promise<v
|
||||
|
||||
const finalWorkerState = workerStartResult as WorkerStartResult;
|
||||
const workerAlive = finalWorkerState !== 'dead' || workerReady;
|
||||
const runtimeLabel = selectedRuntime === 'server-beta' ? 'Server beta' : 'Worker';
|
||||
const runtimeStartCommand = selectedRuntime === 'server-beta' ? 'npx claude-mem server start' : 'npx claude-mem start';
|
||||
const workerHeadline = autoStartSkipped
|
||||
? `${pc.yellow('!')} Worker autostart skipped — start it manually with ${pc.bold('npx claude-mem start')}`
|
||||
? `${pc.yellow('!')} ${runtimeLabel} autostart skipped — start it manually with ${pc.bold(runtimeStartCommand)}`
|
||||
: workerReady || finalWorkerState === 'ready'
|
||||
? `${pc.green('✓')} Worker running at ${pc.underline(`http://localhost:${actualPort}`)}`
|
||||
: `${pc.yellow('⏳')} Worker starting at ${pc.underline(`http://localhost:${actualPort}`)} — give it ~30s, then refresh`;
|
||||
? `${pc.green('✓')} ${runtimeLabel} running at ${pc.underline(`http://localhost:${actualPort}`)}`
|
||||
: `${pc.yellow('⏳')} ${runtimeLabel} starting at ${pc.underline(`http://localhost:${actualPort}`)} — give it ~30s, then refresh`;
|
||||
const nextSteps = autoStartSkipped
|
||||
? [
|
||||
workerHeadline,
|
||||
|
||||
@@ -29,6 +29,10 @@ function workerServiceScriptPath(): string {
|
||||
return join(marketplaceDirectory(), 'plugin', 'scripts', 'worker-service.cjs');
|
||||
}
|
||||
|
||||
function serverBetaServiceScriptPath(): string {
|
||||
return join(marketplaceDirectory(), 'plugin', 'scripts', 'server-beta-service.cjs');
|
||||
}
|
||||
|
||||
function spawnBunWorkerCommand(command: string, extraArgs: string[] = []): void {
|
||||
ensureInstalledOrExit();
|
||||
const bunPath = resolveBunOrExit();
|
||||
@@ -58,6 +62,49 @@ function spawnBunWorkerCommand(command: string, extraArgs: string[] = []): void
|
||||
});
|
||||
}
|
||||
|
||||
function spawnBunServerBetaCommand(command: string): void {
|
||||
ensureInstalledOrExit();
|
||||
const bunPath = resolveBunOrExit();
|
||||
const serverScript = serverBetaServiceScriptPath();
|
||||
|
||||
if (!existsSync(serverScript)) {
|
||||
console.error(pc.red(`Server beta script not found at: ${serverScript}`));
|
||||
console.error('The installation may be corrupted. Try: npx claude-mem install');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const child = spawnHidden(bunPath, [serverScript, command], {
|
||||
stdio: 'inherit',
|
||||
cwd: marketplaceDirectory(),
|
||||
env: process.env,
|
||||
});
|
||||
|
||||
child.on('error', (error) => {
|
||||
console.error(pc.red(`Failed to start Bun: ${error.message}`));
|
||||
process.exit(1);
|
||||
});
|
||||
|
||||
child.on('close', (exitCode) => {
|
||||
process.exit(exitCode ?? 0);
|
||||
});
|
||||
}
|
||||
|
||||
export function runServerBetaStartCommand(): void {
|
||||
spawnBunServerBetaCommand('start');
|
||||
}
|
||||
|
||||
export function runServerBetaStopCommand(): void {
|
||||
spawnBunServerBetaCommand('stop');
|
||||
}
|
||||
|
||||
export function runServerBetaRestartCommand(): void {
|
||||
spawnBunServerBetaCommand('restart');
|
||||
}
|
||||
|
||||
export function runServerBetaStatusCommand(): void {
|
||||
spawnBunServerBetaCommand('status');
|
||||
}
|
||||
|
||||
export function runStartCommand(): void {
|
||||
spawnBunWorkerCommand('start');
|
||||
}
|
||||
@@ -74,6 +121,10 @@ export function runStatusCommand(): void {
|
||||
spawnBunWorkerCommand('status');
|
||||
}
|
||||
|
||||
export function runServerApiKeyCommand(extraArgs: string[] = []): void {
|
||||
spawnBunWorkerCommand('server', ['api-key', ...extraArgs]);
|
||||
}
|
||||
|
||||
export function runAdoptCommand(extraArgs: string[] = []): void {
|
||||
ensureInstalledOrExit();
|
||||
const bunPath = resolveBunOrExit();
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
import pc from 'picocolors';
|
||||
import {
|
||||
runServerBetaRestartCommand,
|
||||
runServerBetaStartCommand,
|
||||
runServerBetaStatusCommand,
|
||||
runServerBetaStopCommand,
|
||||
runRestartCommand,
|
||||
runServerApiKeyCommand,
|
||||
runStartCommand,
|
||||
runStatusCommand,
|
||||
runStopCommand,
|
||||
} from './runtime.js';
|
||||
|
||||
const UNSUPPORTED_SERVER_COMMANDS = new Set([
|
||||
'logs',
|
||||
'doctor',
|
||||
'migrate',
|
||||
'export',
|
||||
'import',
|
||||
]);
|
||||
|
||||
function printServerUsage(): void {
|
||||
console.error(`Usage: ${pc.bold('npx claude-mem server <command>')}`);
|
||||
console.error('Commands: start, stop, restart, status, logs, doctor, migrate, export, import, api-key create|list|revoke');
|
||||
}
|
||||
|
||||
function failUnsupported(command: string): never {
|
||||
console.error(pc.red(`Server command not implemented yet: ${command}`));
|
||||
console.error('This CLI route is reserved for the server runtime, but no backend API exists for it yet.');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
function runWorkerLifecycleCommand(command: string): boolean {
|
||||
switch (command) {
|
||||
case 'start':
|
||||
runStartCommand();
|
||||
return true;
|
||||
case 'stop':
|
||||
runStopCommand();
|
||||
return true;
|
||||
case 'restart':
|
||||
runRestartCommand();
|
||||
return true;
|
||||
case 'status':
|
||||
runStatusCommand();
|
||||
return true;
|
||||
default:
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function runServerBetaLifecycleCommand(command: string): boolean {
|
||||
switch (command) {
|
||||
case 'start':
|
||||
runServerBetaStartCommand();
|
||||
return true;
|
||||
case 'stop':
|
||||
runServerBetaStopCommand();
|
||||
return true;
|
||||
case 'restart':
|
||||
runServerBetaRestartCommand();
|
||||
return true;
|
||||
case 'status':
|
||||
runServerBetaStatusCommand();
|
||||
return true;
|
||||
default:
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
export async function runServerCommand(argv: string[] = []): Promise<void> {
|
||||
const subCommand = argv[0]?.toLowerCase();
|
||||
|
||||
if (!subCommand) {
|
||||
printServerUsage();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (UNSUPPORTED_SERVER_COMMANDS.has(subCommand)) {
|
||||
failUnsupported(`server ${subCommand}`);
|
||||
}
|
||||
|
||||
if (runServerBetaLifecycleCommand(subCommand)) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (subCommand === 'api-key') {
|
||||
const apiKeyCommand = argv[1]?.toLowerCase();
|
||||
if (apiKeyCommand === 'create' || apiKeyCommand === 'list' || apiKeyCommand === 'revoke') {
|
||||
runServerApiKeyCommand(argv.slice(1));
|
||||
return;
|
||||
}
|
||||
console.error(pc.red(`Unknown server api-key subcommand: ${apiKeyCommand ?? '(none)'}`));
|
||||
console.error('Usage: npx claude-mem server api-key create|list|revoke');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.error(pc.red(`Unknown server command: ${subCommand}`));
|
||||
printServerUsage();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
export function runWorkerAliasCommand(argv: string[] = []): void {
|
||||
const subCommand = argv[0]?.toLowerCase();
|
||||
|
||||
if (!subCommand || !runWorkerLifecycleCommand(subCommand)) {
|
||||
console.error(pc.red(`Unknown worker command: ${subCommand ?? '(none)'}`));
|
||||
console.error('Usage: npx claude-mem worker start|stop|restart|status');
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
@@ -36,6 +36,17 @@ ${pc.bold('Runtime Commands')} (requires Bun, delegates to installed plugin):
|
||||
${pc.cyan('npx claude-mem stop')} Stop worker service
|
||||
${pc.cyan('npx claude-mem restart')} Restart worker service
|
||||
${pc.cyan('npx claude-mem status')} Show worker status
|
||||
${pc.cyan('npx claude-mem server start')} Start server service
|
||||
${pc.cyan('npx claude-mem server stop')} Stop server service
|
||||
${pc.cyan('npx claude-mem server restart')} Restart server service
|
||||
${pc.cyan('npx claude-mem server status')} Show server status
|
||||
${pc.cyan('npx claude-mem server logs')} Show recent server logs
|
||||
${pc.cyan('npx claude-mem server doctor')} Check server configuration (not yet implemented)
|
||||
${pc.cyan('npx claude-mem server migrate')} Run server migrations (not yet implemented)
|
||||
${pc.cyan('npx claude-mem server export')} Export server data (not yet implemented)
|
||||
${pc.cyan('npx claude-mem server import')} Import server data (not yet implemented)
|
||||
${pc.cyan('npx claude-mem server api-key create|list|revoke')} Manage API keys (not yet implemented)
|
||||
${pc.cyan('npx claude-mem worker start|stop|restart|status')} Worker compatibility aliases
|
||||
${pc.cyan('npx claude-mem search <query>')} Search observations
|
||||
${pc.cyan('npx claude-mem adopt [--dry-run] [--branch <name>]')} Stamp merged worktrees into parent project
|
||||
${pc.cyan('npx claude-mem cleanup [--dry-run]')} Run one-time v12.4.3 pollution cleanup (or preview counts)
|
||||
@@ -139,6 +150,18 @@ async function main(): Promise<void> {
|
||||
break;
|
||||
}
|
||||
|
||||
case 'server': {
|
||||
const { runServerCommand } = await import('./commands/server.js');
|
||||
await runServerCommand(args.slice(1));
|
||||
break;
|
||||
}
|
||||
|
||||
case 'worker': {
|
||||
const { runWorkerAliasCommand } = await import('./commands/server.js');
|
||||
runWorkerAliasCommand(args.slice(1));
|
||||
break;
|
||||
}
|
||||
|
||||
case 'search': {
|
||||
const { runSearchCommand } = await import('./commands/runtime.js');
|
||||
await runSearchCommand(args.slice(1));
|
||||
|
||||
Reference in New Issue
Block a user