server-beta: Phases 4–13 — event pipeline, generation, MCP, compat, Docker, team audit, observability (#2383)
* feat(server-beta): Phase 4 — Postgres event-to-generation-job pipeline Adds POST /v1/events, /v1/events/batch, GET /v1/jobs/:id, GET /v1/events/:id, and POST /v1/memories on the server-beta runtime, backed by Postgres. - Event row + outbox generation-job row insert in one withPostgresTransaction. - BullMQ enqueue happens after commit; enqueue failure leaves the row queued for Phase 3 startup reconciliation. - ?generate=false skips the outbox; ?wait=true returns queue status only, never observation IDs (provider generation is Phase 5). - Batch pre-validates all event projectIds against api-key scope before any write; mixed-project batches reject 403 with zero side effects. - /v1/memories is a direct insert alias — no generator, no outbox. - Cross-tenant /v1/jobs/:id returns 404 to avoid leaking row existence. - New PostgresAuthMiddleware reads api_keys by SHA-256 hash; populates req.authContext.teamId/projectId; legacy ServerV1Routes (SQLite, used by worker runtime) is left untouched. - Tests: unit suite hardened with stubbed pool.query so route registration is safe; integration tests skip cleanly without CLAUDE_MEM_TEST_POSTGRES_URL. Verification: 87 pass / 1 skip / 0 fail. No new typecheck errors. Required greps for WorkerService and MemoryItemsRepository in src/server/routes/v1 and src/server/runtime return no hits. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(server-beta): Phase 5 — provider observation generator Adds independent provider generation under src/server/generation/ with no worker coupling. Server beta can now generate observations end-to-end: event -> outbox -> BullMQ -> provider -> parser -> persisted observation. - ProviderObservationGenerator orchestrates: lock outbox (queued -> processing), reload agent_event from Postgres (BullMQ payload is advisory only), call provider, hand raw text to processGeneratedResponse, route errors via markGenerationFailed with retryable flag from ServerClassifiedProviderError. - processGeneratedResponse parses with parseAgentXml, persists via PostgresObservationRepository with deterministic generation_key = generation:v1:{job_id}:{index}:{fingerprint}, links via PostgresObservationSourcesRepository, advances outbox status, appends observation_generation_job_events, audits — all in one withPostgresTransaction. Idempotent on retry via UNIQUE constraints. - Three provider adapters under src/server/generation/providers/: Claude, Gemini, OpenRouter. Self-contained — no imports from src/services/worker/*. Worker providers unchanged. - Shared error classification + prompt builder under providers/shared/. Prompt builder strips <private> at the edge; fully-private batches emit <skip_summary /> without billing the provider. - ActiveServerBetaGenerationWorkerManager wires BullMQ Worker via ServerJobQueue.start(...) with concurrency 1 + autorun:false + worker.on('error') per BullMQ docs. - New GET /v1/events/:id/observations on ServerV1PostgresRoutes returns observations linked via observation_sources, team/project scoped. Verification: 104 pass / 4 skip / 0 fail. No typecheck regressions. Anti-pattern greps clean for services/worker imports under src/server, WorkerRef/ActiveSession/SessionStore in src/server/generation. Deferred: ModeManager loading uses a stable fallback observation type list; summary and reindex queue lanes are not yet wired. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(server-beta): Phase 6 — independent server session semantics server_sessions is now the canonical Server beta session model. Sessions are independent of legacy worker ActiveSession state. - PostgresServerSessionRepository extended: findByExternalIdForScope, endSession (idempotent via COALESCE(ended_at, now())), markGenerationStarted/Completed/Failed, listUnprocessedEvents (filters agent_events with completed agent_event jobs). - ServerSessionRuntimeRepository wraps the repo; every method requires explicit team_id + project_id and validates scope via assertProjectOwnership. - SessionGenerationPolicy supports per-event (default), debounce (BullMQ delayed-job replace via getJob+remove+add), and end-of-session. Configured via CLAUDE_MEM_SERVER_SESSION_POLICY and CLAUDE_MEM_SERVER_SESSION_DEBOUNCE_MS env vars; per-team override hooks are exposed on ServerV1PostgresRoutesOptions for future settings layer. - POST /v1/sessions/start (find-or-create on (project_id, external_session_id), GET /v1/sessions/:id (scoped 404), POST /v1/sessions/:id/end (transactional: end + create summary outbox via UNIQUE collapse + enqueue post-commit). Re-ending is fully idempotent. - processSessionSummaryResponse persists summary as kind='summary' observation with the same idempotency model (generation_key + observation_sources UNIQUE). - ProviderObservationGenerator dispatches on source_type: agent_event -> processGeneratedResponse, session_summary -> processSessionSummaryResponse; loadEvents handles session-summary by loading unprocessed events. - ActiveServerBetaGenerationWorkerManager wires summary BullMQ lane alongside event lane (concurrency=1, autorun=false, error listener attached per BullMQ docs). Verification: 110 pass / 6 skip / 0 fail. Net typecheck error count unchanged at 24 (pre-existing, none in Phase 6 files). Anti-pattern greps clean for ActiveSession/SessionStore in src/server/runtime, no worker imports anywhere in src/server. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(server-beta): Phase 7 — hook routing without worker dependency Hooks can now talk directly to server-beta when CLAUDE_MEM_RUNTIME=server-beta is selected, with a clean worker fallback when server-beta is unhealthy. - src/services/hooks/server-beta-client.ts — typed HTTP client for /v1/sessions/start, /v1/events, /v1/sessions/:id/end. Throws ServerBetaClientError with kind classification (missing_api_key, transport, timeout, http_error, invalid_response) and isFallbackEligible helper. Zero imports from services/worker/. - src/services/hooks/runtime-selector.ts — reads CLAUDE_MEM_RUNTIME from settings, returns worker or server-beta context, logs [server-beta-fallback] reason=<code> on every config-time fallback. - src/services/hooks/server-beta-bootstrap.ts — Postgres-backed API key bootstrap. Find-or-creates local-hook-team + local-hook-project, generates cmem_<random> key (SHA-256 hashed), inserts into api_keys with scopes events:write/sessions:write/observations:read/jobs:read. Settings file written with chmod 0600. rotateServerBetaApiKey() wired to a new `claude-mem server keys rotate` command. - src/cli/handlers/{observation,session-init,summarize}.ts — every hook handler tries server-beta first when configured, falls through to the existing worker path on transport/5xx/429/missing-key. One WARN line per fallback. Hook JSON output shape unchanged. - src/shared/SettingsDefaultsManager.ts — three new keys with defaults: CLAUDE_MEM_SERVER_BETA_URL, CLAUDE_MEM_SERVER_BETA_API_KEY, CLAUDE_MEM_SERVER_BETA_PROJECT_ID. - src/npx-cli/commands/install.ts — when installer selects server-beta runtime and CLAUDE_MEM_SERVER_DATABASE_URL is set, bootstraps a local API key automatically. Warns and continues if the DB URL is missing. plugin/scripts/*.cjs bundles rebuilt via npm run build to pick up the new hook handler code path. No plaintext keys in the bundle (verified). Verification: 16 hook unit tests pass; 275 server/storage/services tests pass with 7 pre-existing failures (verified independent of this change via git stash --include-untracked). Build clean. No new typecheck errors in Phase 7 files. Anti-pattern guards verified: - /api/sessions/observations only reached via explicit fallback path - server-beta runtime never starts the worker process - API keys live only in ~/.claude-mem/settings.json (chmod 0600), never in the bundle (grep confirmed) - Worker fallback preserved, observable via single WARN line per call Deferred: semantic context injection (UserPromptSubmit hook) stays worker-only; server-beta does not yet expose /v1/context/semantic. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(server-beta): Phase 8 — MCP backed by server-beta core MCP tools now route through server-beta in server-beta mode while keeping worker-mode search/timeline/get_observations tools fully working. - src/servers/mcp-server.ts — five new observation_* tools registered: observation_add, observation_record_event, observation_search, observation_context, observation_generation_status. Three memory_* compatibility aliases delegate to the canonical handlers. Worker auto-start is gated when selectRuntime() === 'server-beta' so MCP in server-beta mode never spawns the worker. - src/services/hooks/server-beta-client.ts — addObservation, searchObservations, contextObservations, getJobStatus added so MCP shares one transport with hooks (Phase 7). - src/server/routes/v1/ServerV1PostgresRoutes.ts — POST /v1/search and POST /v1/context REST cores backed by PostgresObservationRepository full-text search (GIN tsvector from Phase 1). - Existing memory_search/timeline/get_observations tools call callWorkerAPI unchanged in worker mode; worker tests unaffected. Verification: 39 pass / 4 skip / 0 fail on targeted suite. Pre-existing 7 baseline failures verified independent (git stash). No new typecheck errors. WorkerService grep clean across src/servers/mcp-server.ts and src/server/. Anti-pattern guards verified: - No duplicate generation logic in MCP — observation_record_event hits /v1/events which owns event+outbox+enqueue inside one tx - WorkerService not imported anywhere under MCP server-beta path - No hardcoded worker URLs — all transport via Phase 7 ServerBetaClient - memory_* aliases retained, single handler per pair Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(server-beta): Phase 9 — compatibility adapters without coupling Legacy /api/sessions/observations and /api/sessions/summarize endpoints keep working on server-beta runtime by translating to AgentEvent and session-end calls — no worker code, no route duplication. - src/server/services/IngestEventsService.ts — shared event-ingest path used by both /v1/events and the compat adapter. Owns transactional event row + outbox row + lifecycle log + post-commit BullMQ enqueue, honors Phase 6 SessionGenerationPolicy. - src/server/services/EndSessionService.ts — shared session-end path used by both /v1/sessions/:id/end and the compat adapter. Idempotent ended_at + summary outbox + deterministic summary job id. - src/server/compat/SessionsObservationsAdapter.ts — translates legacy POST /api/sessions/observations payload (Claude Code transcript shape) -> AgentEvent (source_adapter='claude-code-compat', event_type='tool_use') -> IngestEventsService.ingestOne. Resolves contentSessionId to server_sessions via find-or-create. - src/server/compat/SessionsSummarizeAdapter.ts — translates legacy POST /api/sessions/summarize -> EndSessionService.end. Preserves the legacy agentId -> {status:'skipped', reason:'subagent_context'} behavior so existing clients see the same response shape. - src/server/routes/v1/ServerV1PostgresRoutes.ts — refactored to delegate to the new shared services (-203 LoC net) so /v1 and /api compat both call the SAME canonical code path. - src/server/runtime/ServerBetaService.ts — registers both compat adapters alongside ServerV1PostgresRoutes, sharing service instances. - docs/server-beta-parity-map.md — full enumeration of legacy /api/* routes labeled native, adapter, or unsupported (with reasons). Viewer read-path adapters explicitly listed as unsupported pending a future viewer-rewrite phase. Verification: 7 compat tests pass, 6 v1-routes tests still pass (refactor preserved behavior), 4 session-routes tests pass. Pre- existing 16 baseline failures verified independent via git stash. Zero new typecheck errors. Anti-pattern guards verified: - No services/worker/http/routes or WorkerService imports under src/server/compat or src/server/runtime - Compat adapters are thin translators with names ending in *Adapter and a top-of-file comment noting they are legacy compatibility - /v1/* remains the canonical Server beta API; compat adapters call shared services rather than acting as a parallel API Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(server-beta): Phase 10 — Docker stack and deployable runtime Server beta now ships as a Docker stack with no worker process anywhere and a separate horizontal generation worker for scaling. - src/server/runtime/create-server-beta-service.ts — validateServerBetaEnv() fails fast on missing CLAUDE_MEM_SERVER_DATABASE_URL, requires CLAUDE_MEM_QUEUE_ENGINE=bullmq in Docker, rejects CLAUDE_MEM_AUTH_MODE=local-dev and CLAUDE_MEM_ALLOW_LOCAL_DEV_BYPASS inside containers (detected via /.dockerenv or CLAUDE_MEM_DOCKER=1). Adds CLAUDE_MEM_GENERATION_DISABLED so the HTTP service can run generator-free. - src/server/runtime/ServerBetaService.ts — runServerBetaGenerationWorker for the dedicated consumer process; runServerBetaApiKeyCli is a new Postgres-backed `server api-key` command (the legacy worker CLI wrote to SQLite and was invisible to the Postgres runtime); getQueueHealth shim feeds /api/health a consistent ObservationQueueHealth shape. - src/npx-cli/commands/{runtime,server}.ts — `claude-mem server worker start` subcommand that boots only the BullMQ consumer. - docker/claude-mem/{Dockerfile,entrypoint.sh} — entrypoint forces CLAUDE_MEM_DOCKER=1 + CLAUDE_MEM_RUNTIME=server-beta and exposes three modes: server (HTTP only, generation disabled), worker (BullMQ consumer), shell. Worker bundle is no longer the default CMD. - docker-compose.yml — full stack: postgres + valkey + claude-mem-server (HTTP-only) + claude-mem-worker (generation consumer). Wires service-to-service env vars. - scripts/e2e-server-beta-docker.sh + docker/e2e/server-beta-e2e.mjs — E2E now hits /v1/sessions/start, /v1/events?wait=true, /v1/jobs/:id; asserts no worker-service.cjs process anywhere in the stack; one-shot docker compose run --rm verifies local-dev auth is rejected with the expected stderr; restart-and-verify confirms Postgres durability and BullMQ retry idempotency. - docs/server.md — full Phase 10 doc: stack diagram, env table, worker mode, auth-in-Docker policy. - docs/api.md — event generation semantics (wait=true, generationJob). Verification: full Docker E2E PASSED on live daemon (phase1 + phase2 + restart-and-verify + revoked-key + no-worker- process + local-dev-rejected). Unit tests 292 pass / 9 skip / 7 fail (7 fails pre-existing baseline). Zero new typecheck errors. Anti-pattern guards verified: - entrypoint never execs worker-service.cjs; E2E greps prove no worker process anywhere in the stack - validateServerBetaEnv refuses local-dev auth in Docker with explicit remediation message; ALLOW_LOCAL_DEV_BYPASS rejected the same way - Docker requires CLAUDE_MEM_QUEUE_ENGINE=bullmq; in-process queue rejected at startup - claude-mem worker / worker-service / WorkerService greps clean in docker/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(server-beta): Phase 11 — team-aware generation with audit chain Generation jobs now carry team_id/project_id/api_key_id/actor_id/ source_adapter from enqueue through execution; the outbox is reloaded from Postgres before any side effect so BullMQ payload can never act as auth authority. - src/server/jobs/types.ts — ServerGenerationJobPayloadSchema (Zod discriminated union) requires team_id, project_id, generation_job_id, source_adapter, api_key_id, actor_id (nullable), source_type, source_id, plus event_id / server_session_id per kind. assertServerGenerationJobPayload is called at enqueue (outbox.ts) and again at execution boundary. - src/server/services/{IngestEventsService,EndSessionService}.ts + SessionGenerationPolicy.ts — thread identity context (apiKeyId, actorId, sourceAdapter) into both event and summary BullMQ payloads. - src/server/generation/ProviderObservationGenerator.ts — loadCanonicalOutbox loads the outbox row WITHOUT scope filter, then compares candidate.team_id/project_id to payload.team_id/project_id; mismatch -> ServerGenerationScopeViolationError (non-retryable), failed status, generation_job.scope_violation audit. isApiKeyRevoked checks api_keys (revoked_at, expires_at, row missing) before any provider call; revoked -> generation_job.revoked_key audit + non- retryable failure. generation_job.processing audit emitted on lock. - src/server/generation/processGeneratedResponse.ts — generated observations carry team_id/project_id/server_session_id from the reloaded source row (not job payload). observation_sources.metadata records source_adapter, actor_id, api_key_id for traceability. observation.created audit per observation; generation_job.completed audit per terminal transition. All audit rows reference the same generation_job_id in details. - src/server/routes/v1/ServerV1PostgresRoutes.ts — GET /v1/teams/:id/jobs and GET /v1/projects/:id/jobs with SQL-layer scoping (WHERE team_id=$1 [AND project_id=$2] [AND status=$3]); cross-tenant returns 404 to avoid leaking row existence. Pagination via status/limit/offset. audit_log rows for event.received, event.batch_received, observation.read. - src/server/compat/{SessionsObservationsAdapter,SessionsSummarizeAdapter}.ts — propagate apiKeyId and sourceAdapter='claude-code-compat'. Verification: 162 pass / 10 skip / 0 fail. Pre-existing failures in tests/services/queue and tests/services/worker confirmed independent via git stash. Zero new typecheck errors in server-beta files. Required greps: rg "team_id.*req\.body|project_id.*req\.body" src/server -> 0 matches Audit chain integration test passes — generation_job.processing, observation.created, and generation_job.completed audit rows all share the same generation_job_id reference. Anti-pattern guards verified: - BullMQ payload never acts as auth authority — Postgres outbox reload with mismatch check happens before every side effect - team_id / project_id never derived from request body for scope decisions; always req.authContext.teamId / projectId - Application-layer team/project filtering forbidden — listJobsForScope pushes scope into the SQL WHERE clause - Project-scoped key on cross-project /v1/teams/:id/jobs returns 404 - Revoked api keys cause non-retryable failure with audit before any provider call Deferred: a redundant generation_job.queued audit_log row (already covered by observation_generation_job_events lifecycle log per Phase 1 schema split). Compat adapters set actor_id=null but propagate api_key_id which is the canonical reference downstream. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(server-beta): Phase 12 — observability and operations Operators can now inspect, retry, and cancel generation jobs from the CLI; queue lane metrics flow into /api/health and /v1/info; every request gets a stable request_id that flows through HTTP -> audit -> outbox -> generator -> completion log. - src/server/middleware/request-id.ts — honors safe inbound X-Request-Id, mints uuid v4 otherwise. Set on req.requestId and echoed via response header so external traces can correlate. - src/server/jobs/ServerJobQueue.ts — QueueEvents wired with completed, failed, progress, stalled, error listeners; lifecycle counters exposed via observe() API. Logs emitted as [generation] job=<id> source_type=<...> duration=<ms> attempts=<N> reason=<message>. Stalled and error counters survive worker restart. - src/server/jobs/types.ts — ServerGenerationJob payload schema extended with optional request_id; flows through from HTTP into every BullMQ job. - src/server/queue/ObservationQueueEngine.ts — health snapshot now carries per-lane (event, summary) counts via ObservationQueueHealthLaneSnapshot. - src/server/runtime/{ActiveServerBetaQueueManager, ActiveServerBetaGenerationWorkerManager,ServerBetaService}.ts — per-lane getJobCounts feed /api/health and /v1/info; stalled events audit through audit_log with action generation_job.stalled. - src/server/routes/v1/ServerV1PostgresRoutes.ts — GET /v1/jobs (status/source_type/since/limit/offset, scope from api-key, payload stripped unless ?include=payload AND admin scope), POST /v1/jobs/:id/retry (idempotent; queued -> no-op; audit generation_job.retried_by_operator), POST /v1/jobs/:id/cancel (terminal -> no-op; audit generation_job.cancelled_by_operator; generator reload-before-side-effects already prevents double work). - src/server/services/IngestEventsService.ts + SessionGenerationPolicy.ts + ProviderObservationGenerator.ts — request_id propagated end to end. Generator extracts request_id from BullMQ payload and includes it in lock/processing/completion logs and audit details. - src/npx-cli/commands/server-jobs.ts + src/npx-cli/commands/server.ts — `claude-mem server jobs status|failed|retry|cancel`. status compares Postgres outbox counts to BullMQ queue counts and surfaces divergence. failed prints attempts + last_error message. --team and --project filters. Verification: 350 pass / 12 skip / 7 fail (pre-existing baseline, verified independent via git stash). 18 new tests added (request-id middleware, server-jobs CLI seams, jobs list/retry/cancel routes Postgres-gated). Zero new typecheck errors. Anti-pattern guards verified: - agent_events.payload only emitted in /v1/jobs response inside the admin-gated branch (?include=payload + admin scope) — returns 403 otherwise - jobs retry on a queued row is a no-op (no double BullMQ enqueue, no double UPDATE) - Every operator action writes to audit_log with the *_by_operator action and request_id correlation in details - Stalled events audit through generation_job.stalled Sample correlated trace (one request_id end to end): HTTP middleware: req.requestId = 'req-abc' audit event.received: details.requestId = 'req-abc' BullMQ payload: { request_id: 'req-abc', generation_job_id: 'gj_x' } generator lock log: [generation] job locked { jobId, requestId } audit generation_job.processing: details.requestId = 'req-abc' completion log: [generation] job=evt_... duration=1230ms Deferred: live /api/health round-trip integration test (needs Redis); stalled event live integration test (needs Redis); storing request_id on the observations row itself (spec did not require). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(server-beta): add Phase 13 release readiness report Captures the final verification gate: tests (1749 pass, 45 fail all pre-existing baseline, zero regressions), required greps clean, Docker E2E green end-to-end, all 7 exit criteria met, build clean, typecheck unchanged from main. Documents deferred items. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * build(server-beta): rebuild server-beta-service bundle Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(server-beta): address Greptile review on PR #2383 - ProviderObservationGenerator.lockOutbox: skip duplicate worker run when another lock is active instead of returning the row, which previously let two BullMQ workers issue the (paid, rate-limited) external provider call before the persistence-layer terminal-status guard collapsed the duplicate. Reconciliation still recovers from a stale lock on startup or next retry. - docker-compose.yml: require POSTGRES_USER/PASSWORD/DB env vars (no defaults). Stack refuses to start without explicit secrets. Added a header warning that the file must not be deployed unmodified. - e2e-server-beta-docker.sh: export ephemeral test creds for the new required env vars so the Docker E2E driver still runs unattended. - ServerBetaService api-key list: bound query with LIMIT/OFFSET (default 100, max 500) and add optional --team filter to prevent unintentional cross-tenant key metadata disclosure on shared admin hosts. - SessionGenerationPolicy: fix dead `??` fallback for NaN parseInt result; use `||` so DEFAULT_DEBOUNCE_MS actually applies. - ServerV1PostgresRoutes: `?wait=true` now actually waits — polls the outbox row until terminal status (timeout 30s, 100ms interval) on both /v1/events and /v1/events/batch. Returns `waitTimedOut: true` if the cap is hit so callers can re-poll the status endpoints. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(server-beta): address CodeRabbit + Greptile second review on PR #2383 P1 fixes - Operator retry endpoint was re-publishing the Postgres outbox metadata column as the BullMQ payload; the worker's assertServerGenerationJobPayload always rejected it, leaving the row stuck in queued until startup reconciliation. Persist the BullMQ payload on the outbox row at create-time inside IngestEventsService and EndSessionService, then re-enqueue that canonical payload on retry. Major fixes - prompt-builder: escape server_session_id when interpolating into the XML prompt; previously a session id containing `<`, `&`, or quotes could inject XML into the provider input. - ServerJobQueue: route both worker.on('stalled') and the QueueEvents 'stalled' subscriber through a single notifyStalled helper that dedupes by jobId for 30s, so counters.stalled increments once per stall. QueueEvents 'error' now routes through notifyQueueError so it increments counters.errored and runs onError listeners — keeping observability symmetric across both sources. - ServerV1PostgresRoutes: convert PostgresObservationRepository from three dynamic imports to a single static import for consistency. - mcp-server / ServerBetaClient: actually forward the observation_record_event tool's `generate` flag through to the /v1/events endpoint as `?generate=false` instead of voiding it. - server-sessions.markGenerationFailed: guard jsonb_set against a null error payload so the failure path can't null out metadata before the generation_status='failed' write commits. Minor fixes - server-sessions.endSession: keep updated_at stable on repeated calls so the documented idempotency contract holds. - SettingsDefaultsManager + ServerBetaService.getServerBetaPort: derive the server-beta default port from UID (37877 + uid%100), matching the worker port pattern, so two users on the same host don't collide. Docker stacks always pass CLAUDE_MEM_SERVER_PORT explicitly so the containerized deployment is unaffected. - server-session-runtime test: close the pg.Pool in afterAll. - server-beta-release-readiness.md: escape pipes inside table inline code, add `text` language tag to the fenced log block. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(server-beta): address Greptile + CodeRabbit third review on PR #2383 P1 fixes - SessionsObservationsAdapter.resolveServerSession: catch unique-violation (23505) on concurrent compat inserts and re-fetch instead of returning 500. Two compat callers carrying the same contentSessionId can both observe `existing===null` and race on the (project_id, external_session_id) unique constraint; the second now resolves to the raced row instead of dropping the event. - /v1/events/batch: pass `sourceAdapter: null` to ingestBatch so each event's BullMQ payload (and persisted outbox payload column) reflects its own event.sourceAdapter via buildEventBullmqPayload's fallback, rather than stamping the whole batch with the first event's adapter. Minor - server-session-runtime test afterEach: wrap DROP SCHEMA in try/finally so client.release() always runs even if the drop throws. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(test): drop `pool as never` cast — pg.Pool already matches PostgresPool Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(server-beta): retry of completed job now 409s instead of duplicating retryGenerationJob previously fell through to the reset+re-enqueue path when called on a job in `completed` status. The observations index dedupes on (generation_job_id, parsed_observation_index, content) but LLM output is non-deterministic, so a second provider run almost always produced a different content string and bypassed the index, persisting a parallel set of observation rows attributed to the same generation job. Match cancelGenerationJob's 409 guard for completed jobs. failed and cancelled remain valid retry targets. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * build(server-beta): rebuild bundles after rebase onto main Regenerates the three plugin bundles so they reflect the rebased source state. Mechanical rebuild output only — no source changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(server-beta): wrap resolveServerSession in try/catch for structured error response Greptile P1 on PR #2383: resolveServerSession was called before the try/catch in both compat adapters, so Postgres errors during session lookup (timeout, pool exhaustion, etc.) escaped to Express's default error handler and returned HTML/text 500s. Legacy clients calling response.json() would get a parse failure instead of the documented { stored: false, reason: 'internal_error' } (or { status: 'error', reason: 'internal_error' } for the summarize adapter) shape. Move the resolveServerSession call inside the existing try block in both adapters so any failure flows through the structured catch handler. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(server-beta): catch 23505 unique violation in POST /v1/sessions/start Greptile P1 on PR #2383: concurrent requests with the same externalSessionId can both pass the findByExternalIdForScope check, both call repo.create, and the loser hits the (project_id, external_session_id) unique constraint. The handler treated that as an unknown error and returned a 500. Apply the same pattern resolveServerSession already uses: catch error.code '23505' when externalSessionId is set, refetch the row inserted by the winning request, and return 200 with that session. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Alex Newman
@@ -5,6 +5,35 @@ import { logger } from '../../utils/logger.js';
|
||||
import { HOOK_EXIT_CODES } from '../../shared/hook-constants.js';
|
||||
import { shouldTrackProject } from '../../shared/should-track-project.js';
|
||||
import { normalizePlatformSource } from '../../shared/platform-source.js';
|
||||
import { resolveRuntimeContext, logServerBetaFallback } from '../../services/hooks/runtime-selector.js';
|
||||
import { isServerBetaClientError } from '../../services/hooks/server-beta-client.js';
|
||||
|
||||
async function dispatchToWorker(
|
||||
input: NormalizedHookInput,
|
||||
platformSource: string,
|
||||
): Promise<HookResult> {
|
||||
const result = await executeWithWorkerFallback<{ status?: string }>(
|
||||
'/api/sessions/observations',
|
||||
'POST',
|
||||
{
|
||||
contentSessionId: input.sessionId,
|
||||
platformSource,
|
||||
tool_name: input.toolName,
|
||||
tool_input: input.toolInput,
|
||||
tool_response: input.toolResponse,
|
||||
cwd: input.cwd,
|
||||
agentId: input.agentId,
|
||||
agentType: input.agentType,
|
||||
},
|
||||
);
|
||||
|
||||
if (isWorkerFallback(result)) {
|
||||
return { continue: true, suppressOutput: true, exitCode: HOOK_EXIT_CODES.SUCCESS };
|
||||
}
|
||||
|
||||
logger.debug('HOOK', 'Observation sent successfully via worker', { toolName: input.toolName });
|
||||
return { continue: true, suppressOutput: true };
|
||||
}
|
||||
|
||||
export const observationHandler: EventHandler = {
|
||||
async execute(input: NormalizedHookInput): Promise<HookResult> {
|
||||
@@ -28,26 +57,40 @@ export const observationHandler: EventHandler = {
|
||||
return { continue: true, suppressOutput: true };
|
||||
}
|
||||
|
||||
const result = await executeWithWorkerFallback<{ status?: string }>(
|
||||
'/api/sessions/observations',
|
||||
'POST',
|
||||
{
|
||||
contentSessionId: sessionId,
|
||||
platformSource,
|
||||
tool_name: toolName,
|
||||
tool_input: toolInput,
|
||||
tool_response: toolResponse,
|
||||
cwd,
|
||||
agentId: input.agentId,
|
||||
agentType: input.agentType,
|
||||
},
|
||||
);
|
||||
|
||||
if (isWorkerFallback(result)) {
|
||||
return { continue: true, suppressOutput: true, exitCode: HOOK_EXIT_CODES.SUCCESS };
|
||||
const runtime = resolveRuntimeContext();
|
||||
if (runtime.runtime === 'server-beta') {
|
||||
try {
|
||||
await runtime.client.recordEvent({
|
||||
projectId: runtime.projectId,
|
||||
contentSessionId: sessionId,
|
||||
sourceType: 'hook',
|
||||
eventType: 'tool_use',
|
||||
occurredAtEpoch: Date.now(),
|
||||
payload: {
|
||||
tool_name: toolName,
|
||||
tool_input: toolInput,
|
||||
tool_response: toolResponse,
|
||||
cwd,
|
||||
agentId: input.agentId,
|
||||
agentType: input.agentType,
|
||||
platformSource,
|
||||
},
|
||||
});
|
||||
logger.debug('HOOK', 'Observation sent successfully via server-beta', { toolName });
|
||||
return { continue: true, suppressOutput: true };
|
||||
} catch (error: unknown) {
|
||||
if (isServerBetaClientError(error) && error.isFallbackEligible()) {
|
||||
logServerBetaFallback(error.kind, { status: error.status, message: error.message, route: '/v1/events' });
|
||||
// fall through to worker fallback
|
||||
} else {
|
||||
logger.error('HOOK', 'Server beta event failed (non-recoverable)', {
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
});
|
||||
return { continue: true, suppressOutput: true, exitCode: HOOK_EXIT_CODES.SUCCESS };
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
logger.debug('HOOK', 'Observation sent successfully', { toolName });
|
||||
return { continue: true, suppressOutput: true };
|
||||
return dispatchToWorker(input, platformSource);
|
||||
},
|
||||
};
|
||||
|
||||
@@ -8,6 +8,8 @@ import { shouldTrackProject } from '../../shared/should-track-project.js';
|
||||
import { loadFromFileOnce } from '../../shared/hook-settings.js';
|
||||
import { normalizePlatformSource } from '../../shared/platform-source.js';
|
||||
import { isInternalProtocolPayload } from '../../utils/tag-stripping.js';
|
||||
import { resolveRuntimeContext, logServerBetaFallback } from '../../services/hooks/runtime-selector.js';
|
||||
import { isServerBetaClientError } from '../../services/hooks/server-beta-client.js';
|
||||
|
||||
interface SessionInitResponse {
|
||||
sessionDbId: number;
|
||||
@@ -49,6 +51,43 @@ export const sessionInitHandler: EventHandler = {
|
||||
const project = getProjectContext(cwd).primary;
|
||||
const platformSource = normalizePlatformSource(input.platform);
|
||||
|
||||
const runtime = resolveRuntimeContext();
|
||||
if (runtime.runtime === 'server-beta') {
|
||||
try {
|
||||
await runtime.client.startSession({
|
||||
projectId: runtime.projectId,
|
||||
externalSessionId: sessionId,
|
||||
contentSessionId: sessionId,
|
||||
agentId: input.agentId ?? null,
|
||||
agentType: input.agentType ?? null,
|
||||
platformSource,
|
||||
metadata: { project, prompt },
|
||||
});
|
||||
logger.info('HOOK', 'session-init: server-beta session started', {
|
||||
contentSessionId: sessionId,
|
||||
project,
|
||||
});
|
||||
// Server-beta does not currently support the same context-injection
|
||||
// protocol as the worker. Skip semantic injection in server-beta mode
|
||||
// until the server-beta context endpoint exists.
|
||||
return { continue: true, suppressOutput: true };
|
||||
} catch (error: unknown) {
|
||||
if (isServerBetaClientError(error) && error.isFallbackEligible()) {
|
||||
logServerBetaFallback(error.kind, {
|
||||
status: error.status,
|
||||
message: error.message,
|
||||
route: '/v1/sessions/start',
|
||||
});
|
||||
// fall through to worker fallback
|
||||
} else {
|
||||
logger.error('HOOK', 'Server beta session-start failed (non-recoverable)', {
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
});
|
||||
return { continue: true, suppressOutput: true, exitCode: HOOK_EXIT_CODES.SUCCESS };
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
logger.debug('HOOK', 'session-init: Calling /api/sessions/init', { contentSessionId: sessionId, project });
|
||||
|
||||
const initResult = await executeWithWorkerFallback<SessionInitResponse>(
|
||||
|
||||
@@ -7,6 +7,8 @@ import { stripMemoryTagsFromPrompt } from '../../utils/tag-stripping.js';
|
||||
import { HOOK_EXIT_CODES } from '../../shared/hook-constants.js';
|
||||
import { normalizePlatformSource } from '../../shared/platform-source.js';
|
||||
import { shouldTrackProject } from '../../shared/should-track-project.js';
|
||||
import { resolveRuntimeContext, logServerBetaFallback } from '../../services/hooks/runtime-selector.js';
|
||||
import { isServerBetaClientError } from '../../services/hooks/server-beta-client.js';
|
||||
|
||||
export const summarizeHandler: EventHandler = {
|
||||
async execute(input: NormalizedHookInput): Promise<HookResult> {
|
||||
@@ -70,6 +72,53 @@ export const summarizeHandler: EventHandler = {
|
||||
|
||||
const platformSource = normalizePlatformSource(input.platform);
|
||||
|
||||
const runtime = resolveRuntimeContext();
|
||||
if (runtime.runtime === 'server-beta') {
|
||||
try {
|
||||
// Resolve the server_session_id idempotently. /v1/sessions/start is
|
||||
// idempotent on (projectId, externalSessionId) and returns the
|
||||
// existing row when present.
|
||||
const startResult = await runtime.client.startSession({
|
||||
projectId: runtime.projectId,
|
||||
externalSessionId: sessionId,
|
||||
contentSessionId: sessionId,
|
||||
platformSource,
|
||||
});
|
||||
const serverSessionId = startResult.session.id;
|
||||
// Record the last assistant message as an event before closing the
|
||||
// session so it lands in the generation pipeline.
|
||||
await runtime.client.recordEvent({
|
||||
projectId: runtime.projectId,
|
||||
serverSessionId,
|
||||
contentSessionId: sessionId,
|
||||
sourceType: 'hook',
|
||||
eventType: 'assistant_message',
|
||||
occurredAtEpoch: Date.now(),
|
||||
payload: {
|
||||
last_assistant_message: lastAssistantMessage,
|
||||
platformSource,
|
||||
},
|
||||
});
|
||||
await runtime.client.endSession({ sessionId: serverSessionId });
|
||||
logger.debug('HOOK', 'Summary request queued via server-beta');
|
||||
return { continue: true, suppressOutput: true, exitCode: HOOK_EXIT_CODES.SUCCESS };
|
||||
} catch (error: unknown) {
|
||||
if (isServerBetaClientError(error) && error.isFallbackEligible()) {
|
||||
logServerBetaFallback(error.kind, {
|
||||
status: error.status,
|
||||
message: error.message,
|
||||
route: '/v1/sessions/end',
|
||||
});
|
||||
// fall through to worker fallback
|
||||
} else {
|
||||
logger.error('HOOK', 'Server beta summarize failed (non-recoverable)', {
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
});
|
||||
return { continue: true, suppressOutput: true, exitCode: HOOK_EXIT_CODES.SUCCESS };
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const queueResult = await executeWithWorkerFallback<{ status?: string }>(
|
||||
'/api/sessions/summarize',
|
||||
'POST',
|
||||
|
||||
@@ -666,9 +666,45 @@ async function promptRuntime(): Promise<RuntimeId> {
|
||||
mergeSettings({
|
||||
CLAUDE_MEM_RUNTIME: selected,
|
||||
});
|
||||
|
||||
if (selected === 'server-beta') {
|
||||
await maybeBootstrapServerBetaApiKey();
|
||||
}
|
||||
return selected;
|
||||
}
|
||||
|
||||
async function maybeBootstrapServerBetaApiKey(): Promise<void> {
|
||||
// Only attempt if Postgres is configured. Without DATABASE_URL we cannot
|
||||
// reach the api_keys table — the operator must configure the server first
|
||||
// and rerun `claude-mem server keys rotate`.
|
||||
if (!process.env.CLAUDE_MEM_SERVER_DATABASE_URL) {
|
||||
log.warn(
|
||||
'Skipping local hook API key bootstrap: CLAUDE_MEM_SERVER_DATABASE_URL is not set. '
|
||||
+ 'Run `npx claude-mem server keys rotate` after configuring Postgres to provision a key.',
|
||||
);
|
||||
return;
|
||||
}
|
||||
try {
|
||||
const { bootstrapServerBetaApiKey, persistServerBetaSettings } = await import(
|
||||
'../../services/hooks/server-beta-bootstrap.js'
|
||||
);
|
||||
const result = await bootstrapServerBetaApiKey();
|
||||
persistServerBetaSettings(USER_SETTINGS_PATH, {
|
||||
apiKey: result.rawKey,
|
||||
projectId: result.projectId,
|
||||
});
|
||||
log.info(
|
||||
`Provisioned local hook API key (project=${result.projectId.slice(0, 8)}…). `
|
||||
+ 'Settings saved with mode 0600.',
|
||||
);
|
||||
} catch (error: unknown) {
|
||||
log.warn(
|
||||
`Failed to bootstrap server-beta API key: ${error instanceof Error ? error.message : String(error)}. `
|
||||
+ 'Hooks will fall back to the worker until you run `npx claude-mem server keys rotate`.',
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
async function promptProvider(options: InstallOptions): Promise<ProviderId> {
|
||||
const initialProvider = (getSetting('CLAUDE_MEM_PROVIDER') as ProviderId) || 'claude';
|
||||
|
||||
|
||||
@@ -62,7 +62,7 @@ function spawnBunWorkerCommand(command: string, extraArgs: string[] = []): void
|
||||
});
|
||||
}
|
||||
|
||||
function spawnBunServerBetaCommand(command: string): void {
|
||||
function spawnBunServerBetaCommand(command: string, extraArgs: string[] = []): void {
|
||||
ensureInstalledOrExit();
|
||||
const bunPath = resolveBunOrExit();
|
||||
const serverScript = serverBetaServiceScriptPath();
|
||||
@@ -73,7 +73,7 @@ function spawnBunServerBetaCommand(command: string): void {
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const child = spawnHidden(bunPath, [serverScript, command], {
|
||||
const child = spawnHidden(bunPath, [serverScript, command, ...extraArgs], {
|
||||
stdio: 'inherit',
|
||||
cwd: marketplaceDirectory(),
|
||||
env: process.env,
|
||||
@@ -105,6 +105,13 @@ export function runServerBetaStatusCommand(): void {
|
||||
spawnBunServerBetaCommand('status');
|
||||
}
|
||||
|
||||
// Phase 10 — start the BullMQ generation worker (no HTTP). Use this in
|
||||
// Compose to scale generation horizontally while a single (or multiple)
|
||||
// HTTP-only server-beta replicas serve writes/reads.
|
||||
export function runServerBetaWorkerStartCommand(): void {
|
||||
spawnBunServerBetaCommand('worker', ['start']);
|
||||
}
|
||||
|
||||
export function runStartCommand(): void {
|
||||
spawnBunWorkerCommand('start');
|
||||
}
|
||||
|
||||
@@ -0,0 +1,576 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import pc from 'picocolors';
|
||||
import { logger } from '../../utils/logger.js';
|
||||
|
||||
// Phase 12 — `claude-mem server jobs <subcommand>` operator console for the
|
||||
// Postgres-backed observation generation queue. These commands talk DIRECTLY
|
||||
// to Postgres (and BullMQ when configured), bypassing the HTTP API. They MUST
|
||||
// run from a host that can reach the same database the server-beta runtime
|
||||
// uses — set CLAUDE_MEM_SERVER_DATABASE_URL in the shell.
|
||||
//
|
||||
// Anti-pattern guards:
|
||||
// - Operating without --team/--project requires CLAUDE_MEM_SERVER_ADMIN=1
|
||||
// in the env (admin scope). This makes the elevation explicit.
|
||||
// - retry/cancel write to audit_log so every operator action is logged.
|
||||
// - retry is idempotent: a row already in queued status is a no-op.
|
||||
// - cancel sets status to cancelled; the generator's lockOutbox guard
|
||||
// ensures any in-flight delivery aborts before side effects.
|
||||
|
||||
interface ParsedArgs {
|
||||
team: string | null;
|
||||
project: string | null;
|
||||
limit: number;
|
||||
positional: string[];
|
||||
}
|
||||
|
||||
interface JobStatusRow {
|
||||
status: string;
|
||||
count: number;
|
||||
}
|
||||
|
||||
interface FailedJobRow {
|
||||
id: string;
|
||||
source_type: string;
|
||||
source_id: string;
|
||||
attempts: number;
|
||||
failed_at: Date | null;
|
||||
last_error: unknown;
|
||||
team_id: string;
|
||||
project_id: string;
|
||||
}
|
||||
|
||||
const FAILED_DEFAULT_LIMIT = 20;
|
||||
|
||||
export async function runServerJobsCommand(argv: string[]): Promise<void> {
|
||||
const sub = argv[0]?.toLowerCase();
|
||||
const rest = argv.slice(1);
|
||||
if (!sub) {
|
||||
printJobsUsage();
|
||||
process.exit(1);
|
||||
}
|
||||
if (!process.env.CLAUDE_MEM_SERVER_DATABASE_URL) {
|
||||
console.error(pc.red('CLAUDE_MEM_SERVER_DATABASE_URL is required for server jobs commands.'));
|
||||
console.error('Configure Postgres first, then re-run.');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
switch (sub) {
|
||||
case 'status':
|
||||
await runJobsStatus(parseArgs(rest));
|
||||
return;
|
||||
case 'failed':
|
||||
await runJobsFailed(parseArgs(rest));
|
||||
return;
|
||||
case 'retry':
|
||||
await runJobsRetry(parseArgs(rest));
|
||||
return;
|
||||
case 'cancel':
|
||||
await runJobsCancel(parseArgs(rest));
|
||||
return;
|
||||
default:
|
||||
console.error(pc.red(`Unknown server jobs subcommand: ${sub}`));
|
||||
printJobsUsage();
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
function printJobsUsage(): void {
|
||||
console.error(`Usage: ${pc.bold('npx claude-mem server jobs <subcommand>')}`);
|
||||
console.error('Subcommands:');
|
||||
console.error(' status Show queue lane counts (Postgres + BullMQ)');
|
||||
console.error(' failed [--limit N] List failed generation jobs (default 20)');
|
||||
console.error(' retry <id> Re-enqueue a failed/cancelled generation job');
|
||||
console.error(' cancel <id> Cancel a queued/processing generation job');
|
||||
console.error('Filters: --team <id> --project <id> (omit both with CLAUDE_MEM_SERVER_ADMIN=1)');
|
||||
}
|
||||
|
||||
function parseArgs(argv: string[]): ParsedArgs {
|
||||
const out: ParsedArgs = { team: null, project: null, limit: FAILED_DEFAULT_LIMIT, positional: [] };
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const arg = argv[i];
|
||||
if (!arg) continue;
|
||||
if (arg === '--team' || arg === '--project' || arg === '--limit') {
|
||||
const value = argv[i + 1];
|
||||
if (!value) {
|
||||
console.error(pc.red(`Missing value for ${arg}`));
|
||||
process.exit(1);
|
||||
}
|
||||
if (arg === '--team') out.team = value;
|
||||
else if (arg === '--project') out.project = value;
|
||||
else if (arg === '--limit') {
|
||||
const n = Number.parseInt(value, 10);
|
||||
out.limit = Number.isInteger(n) && n > 0 ? n : FAILED_DEFAULT_LIMIT;
|
||||
}
|
||||
i += 1;
|
||||
continue;
|
||||
}
|
||||
if (arg.startsWith('--team=')) { out.team = arg.slice('--team='.length); continue; }
|
||||
if (arg.startsWith('--project=')) { out.project = arg.slice('--project='.length); continue; }
|
||||
if (arg.startsWith('--limit=')) {
|
||||
const n = Number.parseInt(arg.slice('--limit='.length), 10);
|
||||
out.limit = Number.isInteger(n) && n > 0 ? n : FAILED_DEFAULT_LIMIT;
|
||||
continue;
|
||||
}
|
||||
out.positional.push(arg);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// `--team`/`--project` may both be absent only when CLAUDE_MEM_SERVER_ADMIN=1
|
||||
// is set in the env. Without admin we refuse and ask the operator to scope.
|
||||
function requireScope(args: ParsedArgs): { team: string | null; project: string | null } {
|
||||
if (!args.team && !args.project && process.env.CLAUDE_MEM_SERVER_ADMIN !== '1') {
|
||||
console.error(pc.red('Refusing to run unscoped: pass --team <id> and/or --project <id>, or set CLAUDE_MEM_SERVER_ADMIN=1.'));
|
||||
process.exit(1);
|
||||
}
|
||||
return { team: args.team, project: args.project };
|
||||
}
|
||||
|
||||
async function runJobsStatus(args: ParsedArgs): Promise<void> {
|
||||
const scope = requireScope(args);
|
||||
const { pool, releasePool } = await openPool();
|
||||
try {
|
||||
const where: string[] = [];
|
||||
const params: Array<string | number | Date> = [];
|
||||
if (scope.team) { params.push(scope.team); where.push(`team_id = $${params.length}`); }
|
||||
if (scope.project) { params.push(scope.project); where.push(`project_id = $${params.length}`); }
|
||||
const whereClause = where.length > 0 ? `WHERE ${where.join(' AND ')}` : '';
|
||||
|
||||
// Postgres outbox is canonical history.
|
||||
const pgResult = await pool.query(
|
||||
`SELECT status, COUNT(*)::int AS count FROM observation_generation_jobs ${whereClause} GROUP BY status`,
|
||||
params,
|
||||
);
|
||||
const pgCounts: Record<string, number> = {};
|
||||
for (const row of pgResult.rows as JobStatusRow[]) {
|
||||
pgCounts[row.status] = Number(row.count);
|
||||
}
|
||||
|
||||
// BullMQ counts (best effort — missing Redis just shows pg counts).
|
||||
let bullmqCounts: Record<string, { waiting: number; active: number; completed: number; failed: number; delayed: number; stalled: number }> | null = null;
|
||||
try {
|
||||
bullmqCounts = await (testSeams.collectBullmqCounts ?? collectBullmqCounts)();
|
||||
} catch (error) {
|
||||
logger.debug?.('SYSTEM', 'BullMQ counts unavailable', {
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
});
|
||||
}
|
||||
|
||||
const output = {
|
||||
scope: { team: scope.team, project: scope.project },
|
||||
postgres: pgCounts,
|
||||
bullmq: bullmqCounts ?? { unavailable: true },
|
||||
divergence: detectDivergence(pgCounts, bullmqCounts),
|
||||
};
|
||||
console.log(JSON.stringify(output, null, 2));
|
||||
} finally {
|
||||
await releasePool();
|
||||
}
|
||||
}
|
||||
|
||||
async function runJobsFailed(args: ParsedArgs): Promise<void> {
|
||||
const scope = requireScope(args);
|
||||
const { pool, releasePool } = await openPool();
|
||||
try {
|
||||
const where: string[] = [`status = 'failed'`];
|
||||
const params: Array<string | number> = [];
|
||||
if (scope.team) { params.push(scope.team); where.push(`team_id = $${params.length}`); }
|
||||
if (scope.project) { params.push(scope.project); where.push(`project_id = $${params.length}`); }
|
||||
params.push(args.limit);
|
||||
const limitParam = params.length;
|
||||
const result = await pool.query(
|
||||
`
|
||||
SELECT id, source_type, source_id, attempts, failed_at, last_error, team_id, project_id
|
||||
FROM observation_generation_jobs
|
||||
WHERE ${where.join(' AND ')}
|
||||
ORDER BY failed_at DESC NULLS LAST, created_at DESC
|
||||
LIMIT $${limitParam}
|
||||
`,
|
||||
params,
|
||||
);
|
||||
const formatted = (result.rows as FailedJobRow[]).map(row => ({
|
||||
id: row.id,
|
||||
sourceType: row.source_type,
|
||||
sourceId: row.source_id,
|
||||
teamId: row.team_id,
|
||||
projectId: row.project_id,
|
||||
attempts: row.attempts,
|
||||
failedAt: row.failed_at?.toISOString() ?? null,
|
||||
lastError: row.last_error && typeof row.last_error === 'object'
|
||||
? (row.last_error as { message?: string }).message ?? row.last_error
|
||||
: null,
|
||||
}));
|
||||
console.log(JSON.stringify({
|
||||
scope: { team: scope.team, project: scope.project },
|
||||
limit: args.limit,
|
||||
count: formatted.length,
|
||||
failed: formatted,
|
||||
}, null, 2));
|
||||
} finally {
|
||||
await releasePool();
|
||||
}
|
||||
}
|
||||
|
||||
async function runJobsRetry(args: ParsedArgs): Promise<void> {
|
||||
const id = args.positional[0];
|
||||
if (!id) {
|
||||
console.error(pc.red('Usage: server jobs retry <id>'));
|
||||
process.exit(1);
|
||||
}
|
||||
const scope = requireScope(args);
|
||||
const { pool, releasePool } = await openPool();
|
||||
try {
|
||||
// Verify the row exists. Scope-first lookup so admin without --team is
|
||||
// honored, but project/team filters narrow the lookup when present.
|
||||
const lookup = await loadJobScoped(pool, id, scope);
|
||||
if (!lookup) {
|
||||
console.error(pc.red(`Generation job not found: ${id}`));
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (lookup.status === 'queued') {
|
||||
console.log(JSON.stringify({
|
||||
id: lookup.id,
|
||||
action: 'retry',
|
||||
outcome: 'noop_already_queued',
|
||||
retriedCount: extractRetriedCount(lookup.payload),
|
||||
}, null, 2));
|
||||
await writeOperatorAudit(pool, lookup, 'generation_job.retried_by_operator', {
|
||||
outcome: 'noop_already_queued',
|
||||
currentAttempts: lookup.attempts,
|
||||
});
|
||||
return;
|
||||
}
|
||||
if (lookup.status === 'processing') {
|
||||
console.error(pc.red(`Cannot retry an in-flight job. Cancel first or wait. Current status: ${lookup.status}`));
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const newRetriedCount = extractRetriedCount(lookup.payload) + 1;
|
||||
const newPayload = {
|
||||
...(lookup.payload && typeof lookup.payload === 'object' ? lookup.payload as Record<string, unknown> : {}),
|
||||
retried_count: newRetriedCount,
|
||||
last_retried_by: 'cli_operator',
|
||||
};
|
||||
|
||||
const updated = await pool.query(
|
||||
`
|
||||
UPDATE observation_generation_jobs
|
||||
SET status = 'queued',
|
||||
locked_at = NULL,
|
||||
locked_by = NULL,
|
||||
failed_at = NULL,
|
||||
cancelled_at = NULL,
|
||||
completed_at = NULL,
|
||||
last_error = NULL,
|
||||
attempts = LEAST(attempts, max_attempts - 1),
|
||||
payload = $2::jsonb,
|
||||
updated_at = now()
|
||||
WHERE id = $1
|
||||
RETURNING id, status, attempts, bullmq_job_id, source_type
|
||||
`,
|
||||
[id, JSON.stringify(newPayload)],
|
||||
);
|
||||
type UpdatedRetryRow = { id: string; status: string; attempts: number; bullmq_job_id: string | null; source_type: string };
|
||||
const row = (updated.rows as UpdatedRetryRow[])[0];
|
||||
if (!row) {
|
||||
console.error(pc.red('Update returned no rows; the job may have been deleted.'));
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Append lifecycle event row matching the audit chain shape.
|
||||
await pool.query(
|
||||
`INSERT INTO observation_generation_job_events (id, generation_job_id, event_type, status_after, attempt, details)
|
||||
VALUES (gen_random_uuid(), $1, 'queued', 'queued', $2, $3::jsonb)`,
|
||||
[id, row.attempts, JSON.stringify({ source: 'cli_operator_retry', retriedCount: newRetriedCount })],
|
||||
);
|
||||
|
||||
await writeOperatorAudit(pool, lookup, 'generation_job.retried_by_operator', {
|
||||
previousStatus: lookup.status,
|
||||
currentStatus: row.status,
|
||||
retriedCount: newRetriedCount,
|
||||
});
|
||||
|
||||
// Best-effort BullMQ re-publish using the deterministic id.
|
||||
if (row.bullmq_job_id) {
|
||||
try {
|
||||
await (testSeams.republishToBullmq ?? republishToBullmq)(row.source_type, row.bullmq_job_id, newPayload);
|
||||
} catch (error) {
|
||||
logger.warn('SYSTEM', 'BullMQ re-enqueue failed (will reconcile on startup)', {
|
||||
jobId: id,
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
console.log(JSON.stringify({
|
||||
id: row.id,
|
||||
action: 'retry',
|
||||
outcome: 'requeued',
|
||||
retriedCount: newRetriedCount,
|
||||
status: row.status,
|
||||
attempts: row.attempts,
|
||||
}, null, 2));
|
||||
} finally {
|
||||
await releasePool();
|
||||
}
|
||||
}
|
||||
|
||||
async function runJobsCancel(args: ParsedArgs): Promise<void> {
|
||||
const id = args.positional[0];
|
||||
if (!id) {
|
||||
console.error(pc.red('Usage: server jobs cancel <id>'));
|
||||
process.exit(1);
|
||||
}
|
||||
const scope = requireScope(args);
|
||||
const { pool, releasePool } = await openPool();
|
||||
try {
|
||||
const lookup = await loadJobScoped(pool, id, scope);
|
||||
if (!lookup) {
|
||||
console.error(pc.red(`Generation job not found: ${id}`));
|
||||
process.exit(1);
|
||||
}
|
||||
if (lookup.status === 'cancelled') {
|
||||
console.log(JSON.stringify({ id: lookup.id, action: 'cancel', outcome: 'noop_already_cancelled' }, null, 2));
|
||||
await writeOperatorAudit(pool, lookup, 'generation_job.cancelled_by_operator', { outcome: 'noop_already_cancelled' });
|
||||
return;
|
||||
}
|
||||
if (lookup.status === 'completed') {
|
||||
console.error(pc.red('Cannot cancel a completed job.'));
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const updated = await pool.query(
|
||||
`
|
||||
UPDATE observation_generation_jobs
|
||||
SET status = 'cancelled',
|
||||
cancelled_at = now(),
|
||||
updated_at = now()
|
||||
WHERE id = $1
|
||||
RETURNING id, status, bullmq_job_id, source_type
|
||||
`,
|
||||
[id],
|
||||
);
|
||||
type UpdatedCancelRow = { id: string; status: string; bullmq_job_id: string | null; source_type: string };
|
||||
const row = (updated.rows as UpdatedCancelRow[])[0];
|
||||
if (!row) {
|
||||
console.error(pc.red('Update returned no rows.'));
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
await pool.query(
|
||||
`INSERT INTO observation_generation_job_events (id, generation_job_id, event_type, status_after, attempt, details)
|
||||
VALUES (gen_random_uuid(), $1, 'cancelled', 'cancelled', $2, $3::jsonb)`,
|
||||
[id, lookup.attempts, JSON.stringify({ source: 'cli_operator_cancel' })],
|
||||
);
|
||||
|
||||
await writeOperatorAudit(pool, lookup, 'generation_job.cancelled_by_operator', {
|
||||
previousStatus: lookup.status,
|
||||
currentStatus: row.status,
|
||||
});
|
||||
|
||||
// Best-effort BullMQ removal.
|
||||
if (row.bullmq_job_id) {
|
||||
try {
|
||||
await (testSeams.removeFromBullmq ?? removeFromBullmq)(row.source_type, row.bullmq_job_id);
|
||||
} catch (error) {
|
||||
logger.debug?.('SYSTEM', 'BullMQ remove on cancel failed (job may not be in queue)', {
|
||||
jobId: id,
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
console.log(JSON.stringify({
|
||||
id: row.id,
|
||||
action: 'cancel',
|
||||
outcome: 'cancelled',
|
||||
status: row.status,
|
||||
}, null, 2));
|
||||
} finally {
|
||||
await releasePool();
|
||||
}
|
||||
}
|
||||
|
||||
interface JobLookup {
|
||||
id: string;
|
||||
team_id: string;
|
||||
project_id: string;
|
||||
status: string;
|
||||
attempts: number;
|
||||
bullmq_job_id: string | null;
|
||||
source_type: string;
|
||||
payload: Record<string, unknown> | null;
|
||||
}
|
||||
|
||||
async function loadJobScoped(
|
||||
pool: PoolLike,
|
||||
id: string,
|
||||
scope: { team: string | null; project: string | null },
|
||||
): Promise<JobLookup | null> {
|
||||
const where: string[] = ['id = $1'];
|
||||
const params: Array<string> = [id];
|
||||
if (scope.team) { params.push(scope.team); where.push(`team_id = $${params.length}`); }
|
||||
if (scope.project) { params.push(scope.project); where.push(`project_id = $${params.length}`); }
|
||||
const result = await pool.query(
|
||||
`SELECT id, team_id, project_id, status, attempts, bullmq_job_id, source_type, payload
|
||||
FROM observation_generation_jobs
|
||||
WHERE ${where.join(' AND ')}`,
|
||||
params,
|
||||
);
|
||||
const row = (result.rows as JobLookup[])[0];
|
||||
return row ?? null;
|
||||
}
|
||||
|
||||
interface PoolLike {
|
||||
query: (sql: string, params: unknown[]) => Promise<{ rows: unknown[] }>;
|
||||
}
|
||||
|
||||
async function writeOperatorAudit(
|
||||
pool: { query: (sql: string, params: unknown[]) => Promise<unknown> },
|
||||
job: JobLookup,
|
||||
action: string,
|
||||
details: Record<string, unknown>,
|
||||
): Promise<void> {
|
||||
try {
|
||||
await pool.query(
|
||||
`INSERT INTO audit_log (id, team_id, project_id, actor_id, api_key_id, action, resource_type, resource_id, details)
|
||||
VALUES (gen_random_uuid(), $1, $2, NULL, NULL, $3, 'observation_generation_job', $4, $5::jsonb)`,
|
||||
[job.team_id, job.project_id, action, job.id, JSON.stringify({ ...details, source: 'cli_operator' })],
|
||||
);
|
||||
} catch (error) {
|
||||
logger.warn('SYSTEM', 'failed to write operator audit row', {
|
||||
action,
|
||||
jobId: job.id,
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
function extractRetriedCount(payload: Record<string, unknown> | null | undefined): number {
|
||||
if (!payload || typeof payload !== 'object') return 0;
|
||||
const value = (payload as { retried_count?: unknown }).retried_count;
|
||||
return typeof value === 'number' && Number.isFinite(value) && value >= 0 ? Math.floor(value) : 0;
|
||||
}
|
||||
|
||||
function detectDivergence(
|
||||
pg: Record<string, number>,
|
||||
bullmq: Record<string, { waiting: number; active: number; completed: number; failed: number; delayed: number; stalled: number }> | null,
|
||||
): Record<string, unknown> {
|
||||
if (!bullmq) return { reason: 'bullmq_unavailable' };
|
||||
// Sum across lanes for comparison. Postgres counts are per-status; BullMQ
|
||||
// counts are per-state. We compare the obvious two: `failed` and `queued`
|
||||
// (= waiting + delayed). Divergence is informational — Postgres is canonical.
|
||||
const bullmqWaiting = Object.values(bullmq).reduce((a, b) => a + b.waiting + b.delayed, 0);
|
||||
const bullmqFailed = Object.values(bullmq).reduce((a, b) => a + b.failed, 0);
|
||||
const pgQueued = pg.queued ?? 0;
|
||||
const pgFailed = pg.failed ?? 0;
|
||||
const out: Record<string, unknown> = {};
|
||||
if (pgQueued !== bullmqWaiting) {
|
||||
out.queuedMismatch = { postgres: pgQueued, bullmq: bullmqWaiting };
|
||||
}
|
||||
if (pgFailed !== bullmqFailed) {
|
||||
out.failedMismatch = { postgres: pgFailed, bullmq: bullmqFailed };
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// Phase 12 — test seam. Tests can override the pool factory + bullmq access
|
||||
// without resorting to module-level mocks (which leak across Bun test files).
|
||||
// Production callers leave these unset; only `tests/cli/server-jobs.test.ts`
|
||||
// touches them.
|
||||
export interface ServerJobsTestSeams {
|
||||
openPool?: () => Promise<{
|
||||
pool: PoolLike;
|
||||
releasePool: () => Promise<void>;
|
||||
}>;
|
||||
collectBullmqCounts?: () => Promise<Record<string, { waiting: number; active: number; completed: number; failed: number; delayed: number; stalled: number }>>;
|
||||
republishToBullmq?: (sourceType: string, jobId: string, payload: Record<string, unknown>) => Promise<void>;
|
||||
removeFromBullmq?: (sourceType: string, jobId: string) => Promise<void>;
|
||||
}
|
||||
|
||||
let testSeams: ServerJobsTestSeams = {};
|
||||
|
||||
export function __setServerJobsTestSeams(seams: ServerJobsTestSeams): void {
|
||||
testSeams = seams;
|
||||
}
|
||||
|
||||
export function __clearServerJobsTestSeams(): void {
|
||||
testSeams = {};
|
||||
}
|
||||
|
||||
async function openPool(): Promise<{
|
||||
pool: PoolLike;
|
||||
releasePool: () => Promise<void>;
|
||||
}> {
|
||||
if (testSeams.openPool) return testSeams.openPool();
|
||||
const { getSharedPostgresPool } = await import('../../storage/postgres/index.js');
|
||||
const pool = getSharedPostgresPool({ requireDatabaseUrl: true });
|
||||
return {
|
||||
pool: pool as never,
|
||||
releasePool: async () => { /* shared pool tears down on process exit */ },
|
||||
};
|
||||
}
|
||||
|
||||
// BullMQ access. Direct construction avoids importing the runtime, keeping
|
||||
// the CLI fast to boot. Returns counts per known queue name; gracefully
|
||||
// returns null when Redis is unconfigured.
|
||||
async function collectBullmqCounts(): Promise<Record<string, { waiting: number; active: number; completed: number; failed: number; delayed: number; stalled: number }>> {
|
||||
const { getRedisQueueConfig } = await import('../../server/queue/redis-config.js');
|
||||
const { Queue } = await import('bullmq');
|
||||
const config = getRedisQueueConfig();
|
||||
if (config.engine !== 'bullmq') {
|
||||
throw new Error('CLAUDE_MEM_QUEUE_ENGINE is not "bullmq"');
|
||||
}
|
||||
const { SERVER_JOB_QUEUE_NAMES } = await import('../../server/jobs/types.js');
|
||||
const out: Record<string, { waiting: number; active: number; completed: number; failed: number; delayed: number; stalled: number }> = {};
|
||||
for (const [kind, name] of Object.entries(SERVER_JOB_QUEUE_NAMES)) {
|
||||
const queue = new Queue(name, { connection: config.connection, prefix: config.prefix });
|
||||
try {
|
||||
const counts = await queue.getJobCounts('waiting', 'active', 'completed', 'failed', 'delayed');
|
||||
out[kind] = {
|
||||
waiting: Number(counts.waiting ?? 0),
|
||||
active: Number(counts.active ?? 0),
|
||||
completed: Number(counts.completed ?? 0),
|
||||
failed: Number(counts.failed ?? 0),
|
||||
delayed: Number(counts.delayed ?? 0),
|
||||
stalled: 0, // BullMQ rotates the stalled list; runtime tracks it via QueueEvents.
|
||||
};
|
||||
} finally {
|
||||
await queue.close();
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
async function republishToBullmq(sourceType: string, jobId: string, payload: Record<string, unknown>): Promise<void> {
|
||||
const { getRedisQueueConfig } = await import('../../server/queue/redis-config.js');
|
||||
const { Queue } = await import('bullmq');
|
||||
const config = getRedisQueueConfig();
|
||||
if (config.engine !== 'bullmq') return;
|
||||
const { SERVER_JOB_QUEUE_NAMES } = await import('../../server/jobs/types.js');
|
||||
const lane = sourceType === 'session_summary' ? SERVER_JOB_QUEUE_NAMES.summary : SERVER_JOB_QUEUE_NAMES.event;
|
||||
const queue = new Queue(lane, { connection: config.connection, prefix: config.prefix });
|
||||
try {
|
||||
try { await queue.remove(jobId); } catch { /* terminal slot may be missing */ }
|
||||
await queue.add(lane, payload as never, { jobId });
|
||||
} finally {
|
||||
await queue.close();
|
||||
}
|
||||
}
|
||||
|
||||
async function removeFromBullmq(sourceType: string, jobId: string): Promise<void> {
|
||||
const { getRedisQueueConfig } = await import('../../server/queue/redis-config.js');
|
||||
const { Queue } = await import('bullmq');
|
||||
const config = getRedisQueueConfig();
|
||||
if (config.engine !== 'bullmq') return;
|
||||
const { SERVER_JOB_QUEUE_NAMES } = await import('../../server/jobs/types.js');
|
||||
const lane = sourceType === 'session_summary' ? SERVER_JOB_QUEUE_NAMES.summary : SERVER_JOB_QUEUE_NAMES.event;
|
||||
const queue = new Queue(lane, { connection: config.connection, prefix: config.prefix });
|
||||
try {
|
||||
await queue.remove(jobId);
|
||||
} finally {
|
||||
await queue.close();
|
||||
}
|
||||
}
|
||||
@@ -4,6 +4,7 @@ import {
|
||||
runServerBetaStartCommand,
|
||||
runServerBetaStatusCommand,
|
||||
runServerBetaStopCommand,
|
||||
runServerBetaWorkerStartCommand,
|
||||
runRestartCommand,
|
||||
runServerApiKeyCommand,
|
||||
runStartCommand,
|
||||
@@ -21,7 +22,7 @@ const UNSUPPORTED_SERVER_COMMANDS = new Set([
|
||||
|
||||
function printServerUsage(): void {
|
||||
console.error(`Usage: ${pc.bold('npx claude-mem server <command>')}`);
|
||||
console.error('Commands: start, stop, restart, status, logs, doctor, migrate, export, import, api-key create|list|revoke');
|
||||
console.error('Commands: start, stop, restart, status, logs, doctor, migrate, export, import, api-key create|list|revoke, keys rotate, worker start, jobs status|failed|retry|cancel');
|
||||
}
|
||||
|
||||
function failUnsupported(command: string): never {
|
||||
@@ -95,11 +96,101 @@ export async function runServerCommand(argv: string[] = []): Promise<void> {
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (subCommand === 'worker') {
|
||||
const workerCommand = argv[1]?.toLowerCase();
|
||||
if (workerCommand === 'start') {
|
||||
runServerBetaWorkerStartCommand();
|
||||
return;
|
||||
}
|
||||
console.error(pc.red(`Unknown server worker subcommand: ${workerCommand ?? '(none)'}`));
|
||||
console.error('Usage: npx claude-mem server worker start');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (subCommand === 'keys') {
|
||||
const keysCommand = argv[1]?.toLowerCase();
|
||||
if (keysCommand === 'rotate') {
|
||||
await runServerBetaKeysRotateCommand();
|
||||
return;
|
||||
}
|
||||
console.error(pc.red(`Unknown server keys subcommand: ${keysCommand ?? '(none)'}`));
|
||||
console.error('Usage: npx claude-mem server keys rotate');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (subCommand === 'jobs') {
|
||||
// Phase 12 — operator queue console. Uses Postgres (canonical) +
|
||||
// BullMQ (transport) directly. See src/npx-cli/commands/server-jobs.ts.
|
||||
const { runServerJobsCommand } = await import('./server-jobs.js');
|
||||
await runServerJobsCommand(argv.slice(1));
|
||||
return;
|
||||
}
|
||||
|
||||
console.error(pc.red(`Unknown server command: ${subCommand}`));
|
||||
printServerUsage();
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
async function runServerBetaKeysRotateCommand(): Promise<void> {
|
||||
if (!process.env.CLAUDE_MEM_SERVER_DATABASE_URL) {
|
||||
console.error(pc.red('Cannot rotate server-beta API key: CLAUDE_MEM_SERVER_DATABASE_URL is not set.'));
|
||||
console.error('Configure Postgres first, then re-run this command.');
|
||||
process.exit(1);
|
||||
}
|
||||
const { rotateServerBetaApiKey, persistServerBetaSettings } = await import(
|
||||
'../../services/hooks/server-beta-bootstrap.js'
|
||||
);
|
||||
const { SettingsDefaultsManager } = await import('../../shared/SettingsDefaultsManager.js');
|
||||
const { join } = await import('path');
|
||||
const { existsSync, readFileSync } = await import('fs');
|
||||
|
||||
const settingsPath = join(SettingsDefaultsManager.get('CLAUDE_MEM_DATA_DIR'), 'settings.json');
|
||||
let previousApiKeyId: string | null = null;
|
||||
if (existsSync(settingsPath)) {
|
||||
try {
|
||||
const raw = JSON.parse(readFileSync(settingsPath, 'utf-8')) as Record<string, unknown>;
|
||||
const flat = (raw.env && typeof raw.env === 'object' ? raw.env : raw) as Record<string, unknown>;
|
||||
const previousKey = flat.CLAUDE_MEM_SERVER_BETA_API_KEY;
|
||||
if (typeof previousKey === 'string' && previousKey.length > 0) {
|
||||
previousApiKeyId = await lookupApiKeyIdByPlaintext(previousKey);
|
||||
}
|
||||
} catch {
|
||||
// ignore — we'll just generate a new key without revoking the old one
|
||||
}
|
||||
}
|
||||
|
||||
const result = await rotateServerBetaApiKey({ previousApiKeyId });
|
||||
persistServerBetaSettings(settingsPath, {
|
||||
apiKey: result.rawKey,
|
||||
projectId: result.projectId,
|
||||
});
|
||||
console.log(JSON.stringify({
|
||||
rotated: true,
|
||||
apiKeyId: result.apiKeyId,
|
||||
teamId: result.teamId,
|
||||
projectId: result.projectId,
|
||||
settingsPath,
|
||||
}, null, 2));
|
||||
}
|
||||
|
||||
async function lookupApiKeyIdByPlaintext(rawKey: string): Promise<string | null> {
|
||||
const { createPostgresPool } = await import('../../storage/postgres/pool.js');
|
||||
const { parsePostgresConfig } = await import('../../storage/postgres/config.js');
|
||||
const { hashApiKey } = await import('../../services/hooks/server-beta-bootstrap.js');
|
||||
const config = parsePostgresConfig({ requireDatabaseUrl: true });
|
||||
if (!config) return null;
|
||||
const pool = createPostgresPool(config);
|
||||
try {
|
||||
const result = await pool.query<{ id: string }>(
|
||||
'SELECT id FROM api_keys WHERE key_hash = $1 LIMIT 1',
|
||||
[hashApiKey(rawKey)],
|
||||
);
|
||||
return result.rows[0]?.id ?? null;
|
||||
} finally {
|
||||
await pool.end().catch(() => undefined);
|
||||
}
|
||||
}
|
||||
|
||||
export function runWorkerAliasCommand(argv: string[] = []): void {
|
||||
const subCommand = argv[0]?.toLowerCase();
|
||||
|
||||
|
||||
@@ -0,0 +1,212 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
// Legacy compatibility — new clients should use POST /v1/events directly.
|
||||
//
|
||||
// Legacy worker payloads to `/api/sessions/observations` are translated into
|
||||
// the Server beta event/job model and delegated to IngestEventsService. The
|
||||
// adapter never touches worker code, never queues observations directly, and
|
||||
// never uses `src/services/worker/*` types.
|
||||
//
|
||||
// Translation rules:
|
||||
// - `contentSessionId` (Claude Code session UUID) becomes the
|
||||
// `external_session_id` of a Server beta `server_sessions` row, scoped to
|
||||
// the API key's team and project. The session is create-or-found.
|
||||
// - The tool-use shape (tool_name, tool_input, tool_response, tool_use_id)
|
||||
// is mapped to an `agent_event` with sourceAdapter='claude-code-compat',
|
||||
// eventType='tool_use', payload preserves the legacy fields verbatim.
|
||||
// - The API key MUST be project-scoped. Cross-project compat calls return
|
||||
// 400; we never let compat traffic bypass project scope.
|
||||
|
||||
import type { Application, Request, Response } from 'express';
|
||||
import { z } from 'zod';
|
||||
import type { RouteHandler } from '../../services/server/Server.js';
|
||||
import type { PostgresPool } from '../../storage/postgres/pool.js';
|
||||
import { PostgresServerSessionsRepository } from '../../storage/postgres/server-sessions.js';
|
||||
import { logger } from '../../utils/logger.js';
|
||||
import { requirePostgresServerAuth } from '../middleware/postgres-auth.js';
|
||||
import { IngestEventsService } from '../services/IngestEventsService.js';
|
||||
import type { CreatePostgresAgentEventInput } from '../../storage/postgres/agent-events.js';
|
||||
|
||||
const COMPAT_SOURCE_ADAPTER = 'claude-code-compat';
|
||||
const COMPAT_EVENT_TYPE = 'tool_use';
|
||||
|
||||
const observationsSchema = z.object({
|
||||
contentSessionId: z.string().min(1),
|
||||
tool_name: z.string().min(1),
|
||||
tool_input: z.unknown().optional(),
|
||||
tool_response: z.unknown().optional(),
|
||||
cwd: z.string().optional(),
|
||||
agentId: z.string().optional(),
|
||||
agentType: z.string().optional(),
|
||||
platformSource: z.string().optional(),
|
||||
tool_use_id: z.string().optional(),
|
||||
toolUseId: z.string().optional(),
|
||||
}).passthrough();
|
||||
|
||||
export interface SessionsObservationsAdapterOptions {
|
||||
pool: PostgresPool;
|
||||
ingestEvents: IngestEventsService;
|
||||
authMode?: string;
|
||||
allowLocalDevBypass?: boolean;
|
||||
}
|
||||
|
||||
export class SessionsObservationsAdapter implements RouteHandler {
|
||||
constructor(private readonly options: SessionsObservationsAdapterOptions) {}
|
||||
|
||||
setupRoutes(app: Application): void {
|
||||
const writeAuth = requirePostgresServerAuth(this.options.pool, {
|
||||
authMode: this.options.authMode,
|
||||
allowLocalDevBypass: this.options.allowLocalDevBypass,
|
||||
requiredScopes: ['memories:write'],
|
||||
});
|
||||
|
||||
app.post('/api/sessions/observations', writeAuth, this.asyncHandler(async (req, res) => {
|
||||
const parsed = observationsSchema.safeParse(req.body);
|
||||
if (!parsed.success) {
|
||||
res.status(400).json({ error: 'ValidationError', issues: parsed.error.issues });
|
||||
return;
|
||||
}
|
||||
const teamId = req.authContext?.teamId ?? null;
|
||||
const projectId = req.authContext?.projectId ?? null;
|
||||
if (!teamId) {
|
||||
res.status(403).json({ error: 'Forbidden', message: 'API key is not bound to a team' });
|
||||
return;
|
||||
}
|
||||
if (!projectId) {
|
||||
// Compat mode requires a project-scoped key — the legacy payload does
|
||||
// not carry a Server beta projectId, so without scope we cannot place
|
||||
// the row in a tenant-scoped table.
|
||||
res.status(400).json({
|
||||
error: 'BadRequest',
|
||||
message: 'Legacy /api/sessions/observations requires a project-scoped API key',
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
const session = await resolveServerSession({
|
||||
pool: this.options.pool,
|
||||
teamId,
|
||||
projectId,
|
||||
contentSessionId: parsed.data.contentSessionId,
|
||||
platformSource: typeof parsed.data.platformSource === 'string' ? parsed.data.platformSource : null,
|
||||
agentId: typeof parsed.data.agentId === 'string' ? parsed.data.agentId : null,
|
||||
agentType: typeof parsed.data.agentType === 'string' ? parsed.data.agentType : null,
|
||||
});
|
||||
|
||||
const toolUseId = typeof parsed.data.tool_use_id === 'string'
|
||||
? parsed.data.tool_use_id
|
||||
: (typeof parsed.data.toolUseId === 'string' ? parsed.data.toolUseId : null);
|
||||
|
||||
const input: CreatePostgresAgentEventInput = {
|
||||
projectId,
|
||||
teamId,
|
||||
serverSessionId: session.id,
|
||||
sourceAdapter: COMPAT_SOURCE_ADAPTER,
|
||||
sourceEventId: toolUseId,
|
||||
eventType: COMPAT_EVENT_TYPE,
|
||||
payload: {
|
||||
contentSessionId: parsed.data.contentSessionId,
|
||||
tool_name: parsed.data.tool_name,
|
||||
tool_input: parsed.data.tool_input ?? null,
|
||||
tool_response: parsed.data.tool_response ?? null,
|
||||
cwd: parsed.data.cwd ?? null,
|
||||
platformSource: parsed.data.platformSource ?? null,
|
||||
agentId: parsed.data.agentId ?? null,
|
||||
agentType: parsed.data.agentType ?? null,
|
||||
toolUseId,
|
||||
},
|
||||
metadata: { compat: 'sessions/observations' },
|
||||
occurredAt: new Date(),
|
||||
};
|
||||
|
||||
const result = await this.options.ingestEvents.ingestOne(input, {
|
||||
source: 'http_post_api_sessions_observations',
|
||||
apiKeyId: req.authContext?.apiKeyId ?? null,
|
||||
actorId: null,
|
||||
sourceAdapter: COMPAT_SOURCE_ADAPTER,
|
||||
});
|
||||
// Legacy response shape — older clients only check `status`.
|
||||
res.json({
|
||||
status: 'queued',
|
||||
observationCount: 1,
|
||||
sessionId: session.id,
|
||||
serverSessionId: session.id,
|
||||
eventId: result.event.id,
|
||||
generationJobId: result.outbox?.id ?? null,
|
||||
transport: result.enqueueState,
|
||||
});
|
||||
} catch (error) {
|
||||
logger.error('SYSTEM', 'compat observations adapter failed', {
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
contentSessionId: parsed.data.contentSessionId,
|
||||
});
|
||||
res.status(500).json({ stored: false, reason: 'internal_error' });
|
||||
}
|
||||
}));
|
||||
}
|
||||
|
||||
private asyncHandler(fn: (req: Request, res: Response) => Promise<void> | void) {
|
||||
return (req: Request, res: Response, next: (err?: unknown) => void): void => {
|
||||
Promise.resolve(fn(req, res)).catch(next);
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Look up an existing server_session by (project, team, externalSessionId)
|
||||
* or create one if missing. Idempotent: re-issuing for the same content
|
||||
* session returns the existing row.
|
||||
*
|
||||
* Concurrent compat callers can race here — both observe `existing===null`
|
||||
* and both call `repo.create`, where the second will hit one of two unique
|
||||
* constraints (`(project_id, idempotency_key)` covered by ON CONFLICT, or
|
||||
* `(project_id, external_session_id)` which is NOT covered). Catch the
|
||||
* unique-violation and re-fetch so the caller never sees a 500.
|
||||
*/
|
||||
export async function resolveServerSession(input: {
|
||||
pool: PostgresPool;
|
||||
teamId: string;
|
||||
projectId: string;
|
||||
contentSessionId: string;
|
||||
platformSource: string | null;
|
||||
agentId: string | null;
|
||||
agentType: string | null;
|
||||
}): Promise<{ id: string; projectId: string; teamId: string }> {
|
||||
const repo = new PostgresServerSessionsRepository(input.pool);
|
||||
const existing = await repo.findByExternalIdForScope({
|
||||
externalSessionId: input.contentSessionId,
|
||||
projectId: input.projectId,
|
||||
teamId: input.teamId,
|
||||
});
|
||||
if (existing) {
|
||||
return { id: existing.id, projectId: existing.projectId, teamId: existing.teamId };
|
||||
}
|
||||
try {
|
||||
const created = await repo.create({
|
||||
projectId: input.projectId,
|
||||
teamId: input.teamId,
|
||||
externalSessionId: input.contentSessionId,
|
||||
contentSessionId: input.contentSessionId,
|
||||
agentId: input.agentId,
|
||||
agentType: input.agentType,
|
||||
platformSource: input.platformSource,
|
||||
});
|
||||
return { id: created.id, projectId: created.projectId, teamId: created.teamId };
|
||||
} catch (error) {
|
||||
// Postgres unique_violation. A concurrent compat call inserted the row
|
||||
// for this (project, external_session_id) before we could; re-fetch
|
||||
// and return that row instead of bubbling a 500 to the legacy client.
|
||||
if ((error as { code?: string } | null)?.code === '23505') {
|
||||
const racedRow = await repo.findByExternalIdForScope({
|
||||
externalSessionId: input.contentSessionId,
|
||||
projectId: input.projectId,
|
||||
teamId: input.teamId,
|
||||
});
|
||||
if (racedRow) {
|
||||
return { id: racedRow.id, projectId: racedRow.projectId, teamId: racedRow.teamId };
|
||||
}
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
// Legacy compatibility — new clients should use POST /v1/sessions/:id/end directly.
|
||||
//
|
||||
// Translates the legacy `/api/sessions/summarize` request into a call to
|
||||
// EndSessionService. The legacy shape carries `contentSessionId` and an
|
||||
// optional `last_assistant_message`; we resolve the server_session by
|
||||
// (team, project, external_session_id=contentSessionId), then end it.
|
||||
//
|
||||
// Re-summarizing the same session collapses to the same outbox row because
|
||||
// the (team_id, project_id, source_type='session_summary', source_id)
|
||||
// UNIQUE constraint stays in force — exactly the same idempotency guarantee
|
||||
// as `/v1/sessions/:id/end`.
|
||||
|
||||
import type { Application, Request, Response } from 'express';
|
||||
import { z } from 'zod';
|
||||
import type { RouteHandler } from '../../services/server/Server.js';
|
||||
import type { PostgresPool } from '../../storage/postgres/pool.js';
|
||||
import { PostgresServerSessionsRepository } from '../../storage/postgres/server-sessions.js';
|
||||
import { logger } from '../../utils/logger.js';
|
||||
import { requirePostgresServerAuth } from '../middleware/postgres-auth.js';
|
||||
import { EndSessionService } from '../services/EndSessionService.js';
|
||||
import { resolveServerSession } from './SessionsObservationsAdapter.js';
|
||||
|
||||
const summarizeSchema = z.object({
|
||||
contentSessionId: z.string().min(1),
|
||||
last_assistant_message: z.string().optional(),
|
||||
agentId: z.string().optional(),
|
||||
platformSource: z.string().optional(),
|
||||
}).passthrough();
|
||||
|
||||
export interface SessionsSummarizeAdapterOptions {
|
||||
pool: PostgresPool;
|
||||
endSession: EndSessionService;
|
||||
authMode?: string;
|
||||
allowLocalDevBypass?: boolean;
|
||||
}
|
||||
|
||||
export class SessionsSummarizeAdapter implements RouteHandler {
|
||||
constructor(private readonly options: SessionsSummarizeAdapterOptions) {}
|
||||
|
||||
setupRoutes(app: Application): void {
|
||||
const writeAuth = requirePostgresServerAuth(this.options.pool, {
|
||||
authMode: this.options.authMode,
|
||||
allowLocalDevBypass: this.options.allowLocalDevBypass,
|
||||
requiredScopes: ['memories:write'],
|
||||
});
|
||||
|
||||
app.post('/api/sessions/summarize', writeAuth, this.asyncHandler(async (req, res) => {
|
||||
const parsed = summarizeSchema.safeParse(req.body);
|
||||
if (!parsed.success) {
|
||||
res.status(400).json({ error: 'ValidationError', issues: parsed.error.issues });
|
||||
return;
|
||||
}
|
||||
const teamId = req.authContext?.teamId ?? null;
|
||||
const projectId = req.authContext?.projectId ?? null;
|
||||
if (!teamId) {
|
||||
res.status(403).json({ error: 'Forbidden', message: 'API key is not bound to a team' });
|
||||
return;
|
||||
}
|
||||
if (!projectId) {
|
||||
res.status(400).json({
|
||||
error: 'BadRequest',
|
||||
message: 'Legacy /api/sessions/summarize requires a project-scoped API key',
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
// Subagent contexts in legacy code emit summarize calls but the worker
|
||||
// skipped them. We preserve the legacy semantics so existing clients
|
||||
// see the same response shape.
|
||||
if (parsed.data.agentId) {
|
||||
res.json({ status: 'skipped', reason: 'subagent_context' });
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
const session = await resolveServerSession({
|
||||
pool: this.options.pool,
|
||||
teamId,
|
||||
projectId,
|
||||
contentSessionId: parsed.data.contentSessionId,
|
||||
platformSource: typeof parsed.data.platformSource === 'string' ? parsed.data.platformSource : null,
|
||||
agentId: null,
|
||||
agentType: null,
|
||||
});
|
||||
|
||||
const result = await this.options.endSession.end({
|
||||
sessionId: session.id,
|
||||
projectId,
|
||||
teamId,
|
||||
source: 'http_post_api_sessions_summarize',
|
||||
apiKeyId: req.authContext?.apiKeyId ?? null,
|
||||
actorId: null,
|
||||
sourceAdapter: 'claude-code-compat',
|
||||
});
|
||||
if (!result.session) {
|
||||
res.status(404).json({ status: 'not_found', reason: 'session_not_found' });
|
||||
return;
|
||||
}
|
||||
res.json({
|
||||
status: 'queued',
|
||||
sessionId: session.id,
|
||||
serverSessionId: session.id,
|
||||
generationJobId: result.outbox?.id ?? null,
|
||||
transport: result.enqueueState,
|
||||
});
|
||||
} catch (error) {
|
||||
logger.error('SYSTEM', 'compat summarize adapter failed', {
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
contentSessionId: parsed.data.contentSessionId,
|
||||
});
|
||||
res.status(500).json({ status: 'error', reason: 'internal_error' });
|
||||
}
|
||||
}));
|
||||
}
|
||||
|
||||
private asyncHandler(fn: (req: Request, res: Response) => Promise<void> | void) {
|
||||
return (req: Request, res: Response, next: (err?: unknown) => void): void => {
|
||||
Promise.resolve(fn(req, res)).catch(next);
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
// Side-effect import so PostgresServerSessionsRepository symbol is reachable
|
||||
// even when tree-shaking is aggressive in the main bundle.
|
||||
void PostgresServerSessionsRepository;
|
||||
@@ -0,0 +1,538 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import type { Job } from 'bullmq';
|
||||
import { logger } from '../../utils/logger.js';
|
||||
import { PostgresAgentEventsRepository } from '../../storage/postgres/agent-events.js';
|
||||
import { PostgresObservationGenerationJobRepository } from '../../storage/postgres/generation-jobs.js';
|
||||
import { PostgresProjectsRepository } from '../../storage/postgres/projects.js';
|
||||
import { PostgresAuthRepository } from '../../storage/postgres/auth.js';
|
||||
import type { PostgresPool } from '../../storage/postgres/pool.js';
|
||||
import type { PostgresObservationGenerationJob } from '../../storage/postgres/generation-jobs.js';
|
||||
import {
|
||||
assertServerGenerationJobPayload,
|
||||
ServerGenerationJobPayloadValidationError,
|
||||
type ServerGenerationJobPayload,
|
||||
} from '../jobs/types.js';
|
||||
import { ServerClassifiedProviderError } from './providers/shared/error-classification.js';
|
||||
import type { ServerGenerationProvider } from './providers/shared/types.js';
|
||||
import {
|
||||
markGenerationFailed,
|
||||
processGeneratedResponse,
|
||||
processSessionSummaryResponse,
|
||||
type ProcessGeneratedResponseOutcome,
|
||||
} from './processGeneratedResponse.js';
|
||||
import { PostgresServerSessionsRepository } from '../../storage/postgres/server-sessions.js';
|
||||
|
||||
// Phase 11 — sentinel exception class so the worker can distinguish
|
||||
// scope-violation/revoked-key failures from generic processor errors and
|
||||
// audit them under the right action. Marked non-retryable: an attacker who
|
||||
// tampered with a payload should never be retried into the queue.
|
||||
export class ServerGenerationScopeViolationError extends Error {
|
||||
readonly reason: 'scope_mismatch' | 'revoked_key';
|
||||
constructor(reason: 'scope_mismatch' | 'revoked_key', message: string) {
|
||||
super(message);
|
||||
this.reason = reason;
|
||||
}
|
||||
}
|
||||
|
||||
// ProviderObservationGenerator is the BullMQ Worker processor for server-beta
|
||||
// observation generation. It does the following on every job invocation:
|
||||
//
|
||||
// 1. Reload the Postgres outbox row and the source agent_events row.
|
||||
// 2. Lock the outbox by transitioning queued -> processing.
|
||||
// 3. Call the provider with a fully-reloaded ServerGenerationContext.
|
||||
// BullMQ payload data is advisory only.
|
||||
// 4. Hand the raw response to processGeneratedResponse, which persists +
|
||||
// links + advances outbox in one Postgres transaction.
|
||||
// 5. On provider/parse error, route through markGenerationFailed which
|
||||
// decides retry vs final failure based on attempt count + error class.
|
||||
//
|
||||
// Anti-pattern guards verified at the boundary:
|
||||
// - no imports from src/services/worker/*
|
||||
// - no use of WorkerRef / ActiveSession / SessionStore
|
||||
// - no assumption of Claude Code transcript shape
|
||||
|
||||
export interface ProviderObservationGeneratorOptions {
|
||||
pool: PostgresPool;
|
||||
provider: ServerGenerationProvider;
|
||||
workerId?: string;
|
||||
}
|
||||
|
||||
export class ProviderObservationGenerator {
|
||||
constructor(private readonly options: ProviderObservationGeneratorOptions) {}
|
||||
|
||||
/**
|
||||
* Worker entrypoint. Returns a small JSON summary on success so BullMQ's
|
||||
* completed-state telemetry has something to inspect, but Postgres remains
|
||||
* canonical authority.
|
||||
*/
|
||||
async process(
|
||||
job: Job<ServerGenerationJobPayload>,
|
||||
): Promise<{ jobId: string; status: 'completed'; observationCount: number }> {
|
||||
const correlationId = `bullmq:${job.id ?? '?'}`;
|
||||
// Phase 12 — pivot id captured up front so every log line in this
|
||||
// dispatch carries the same identifier whether or not we manage to
|
||||
// load the canonical row. requestId comes from payload (HTTP middleware).
|
||||
const payloadRequestId = (job.data as { request_id?: string | null } | undefined)?.request_id ?? null;
|
||||
|
||||
// Phase 11 — validate the BullMQ payload against the discriminated-union
|
||||
// schema BEFORE doing anything else. A malformed payload (missing
|
||||
// team_id, project_id, generation_job_id, etc.) means the enqueue path
|
||||
// bypassed the boundary contract; we refuse to run it. Throwing surfaces
|
||||
// it on BullMQ's failed list with a clear message.
|
||||
let payload: ServerGenerationJobPayload;
|
||||
try {
|
||||
payload = assertServerGenerationJobPayload(job.data);
|
||||
} catch (error) {
|
||||
if (error instanceof ServerGenerationJobPayloadValidationError) {
|
||||
logger.error('SYSTEM', 'rejecting malformed job payload at execution', {
|
||||
correlationId,
|
||||
issues: error.issues,
|
||||
});
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
|
||||
if (payload.kind !== 'event' && payload.kind !== 'event-batch' && payload.kind !== 'summary') {
|
||||
logger.warn('SYSTEM', 'unsupported job kind for ProviderObservationGenerator', {
|
||||
correlationId,
|
||||
kind: payload.kind,
|
||||
});
|
||||
throw new Error(`unsupported job kind: ${payload.kind}`);
|
||||
}
|
||||
|
||||
// Phase 11 — anti-bypass guard. We MUST NOT trust BullMQ payload data
|
||||
// for tenant scope. Reload the canonical outbox row keyed by id only
|
||||
// (no scope filter), then compare its team_id/project_id to the
|
||||
// payload's. A mismatch indicates payload tampering or a programmer
|
||||
// bug; either way we audit and refuse.
|
||||
const candidate = await this.loadCanonicalOutbox(payload.generation_job_id);
|
||||
if (!candidate) {
|
||||
logger.info('SYSTEM', 'job row not found by id; nothing to do', {
|
||||
correlationId,
|
||||
generationJobId: payload.generation_job_id,
|
||||
});
|
||||
return { jobId: payload.generation_job_id, status: 'completed', observationCount: 0 };
|
||||
}
|
||||
if (candidate.teamId !== payload.team_id || candidate.projectId !== payload.project_id) {
|
||||
const violation = new ServerGenerationScopeViolationError(
|
||||
'scope_mismatch',
|
||||
`BullMQ payload team/project does not match outbox row (jobId=${payload.generation_job_id})`,
|
||||
);
|
||||
await this.auditScopeViolation(payload, candidate, violation, correlationId);
|
||||
// Tag the row as failed so subsequent retries do not pick it up.
|
||||
await markGenerationFailed({
|
||||
pool: this.options.pool,
|
||||
job: candidate,
|
||||
reason: violation.message,
|
||||
classification: 'scope_mismatch',
|
||||
retryable: false,
|
||||
...(this.options.workerId !== undefined ? { workerId: this.options.workerId } : {}),
|
||||
});
|
||||
throw violation;
|
||||
}
|
||||
|
||||
// Phase 11 — revocation check. If the api_key that initiated this job
|
||||
// was revoked between enqueue and execution, do not generate. Audit
|
||||
// and fail without retry.
|
||||
if (payload.api_key_id) {
|
||||
const revoked = await this.isApiKeyRevoked(payload.api_key_id);
|
||||
if (revoked) {
|
||||
const violation = new ServerGenerationScopeViolationError(
|
||||
'revoked_key',
|
||||
`api key ${payload.api_key_id} is revoked; refusing to generate for outbox ${candidate.id}`,
|
||||
);
|
||||
await this.auditRevokedKey(payload, candidate, violation, correlationId);
|
||||
await markGenerationFailed({
|
||||
pool: this.options.pool,
|
||||
job: candidate,
|
||||
reason: violation.message,
|
||||
classification: 'revoked_key',
|
||||
retryable: false,
|
||||
...(this.options.workerId !== undefined ? { workerId: this.options.workerId } : {}),
|
||||
});
|
||||
throw violation;
|
||||
}
|
||||
}
|
||||
|
||||
const fresh = await this.lockOutbox(payload.generation_job_id, payload.team_id, payload.project_id);
|
||||
if (!fresh) {
|
||||
logger.info('SYSTEM', 'job no longer exists or is in terminal status; nothing to do', {
|
||||
correlationId,
|
||||
generationJobId: payload.generation_job_id,
|
||||
});
|
||||
return { jobId: payload.generation_job_id, status: 'completed', observationCount: 0 };
|
||||
}
|
||||
|
||||
// Phase 11 — emit "processing started" audit so we have a row even if
|
||||
// the provider crashes before completion.
|
||||
// Phase 12 — log+audit carry the same job_id / request_id so support
|
||||
// can pivot from BullMQ id -> outbox id -> originating HTTP request.
|
||||
logger.info('SYSTEM', `[generation] job locked for processing`, {
|
||||
correlationId,
|
||||
jobId: fresh.id,
|
||||
bullmqJobId: job.id ?? null,
|
||||
requestId: payloadRequestId,
|
||||
sourceType: fresh.sourceType,
|
||||
attempt: fresh.attempts,
|
||||
});
|
||||
await this.auditEvent({
|
||||
teamId: fresh.teamId,
|
||||
projectId: fresh.projectId,
|
||||
apiKeyId: payload.api_key_id,
|
||||
actorId: payload.actor_id,
|
||||
action: 'generation_job.processing',
|
||||
resourceId: fresh.id,
|
||||
details: {
|
||||
sourceType: fresh.sourceType,
|
||||
sourceId: fresh.sourceId,
|
||||
sourceAdapter: payload.source_adapter,
|
||||
attempt: fresh.attempts,
|
||||
correlationId,
|
||||
requestId: payloadRequestId,
|
||||
},
|
||||
});
|
||||
|
||||
try {
|
||||
const events = await this.loadEvents(fresh, payload);
|
||||
const project = await this.loadProject(fresh);
|
||||
|
||||
const result = await this.options.provider.generate({
|
||||
job: fresh,
|
||||
events,
|
||||
project: {
|
||||
projectId: fresh.projectId,
|
||||
teamId: fresh.teamId,
|
||||
serverSessionId: fresh.serverSessionId,
|
||||
projectName: project?.name ?? null,
|
||||
},
|
||||
});
|
||||
|
||||
const persistInput = {
|
||||
pool: this.options.pool,
|
||||
job: fresh,
|
||||
rawText: result.rawText,
|
||||
modelId: result.modelId,
|
||||
providerLabel: result.providerLabel,
|
||||
// Phase 11 — flow identity context from BullMQ payload into the
|
||||
// persistence layer so observations and audit rows carry the same
|
||||
// generation_job_id reference back through to the original API key.
|
||||
apiKeyId: payload.api_key_id,
|
||||
actorId: payload.actor_id,
|
||||
sourceAdapter: payload.source_adapter,
|
||||
...(this.options.workerId !== undefined ? { workerId: this.options.workerId } : {}),
|
||||
};
|
||||
const outcome: ProcessGeneratedResponseOutcome = fresh.sourceType === 'session_summary'
|
||||
? await processSessionSummaryResponse(persistInput)
|
||||
: await processGeneratedResponse(persistInput);
|
||||
|
||||
if (outcome.kind === 'parse_error') {
|
||||
await markGenerationFailed({
|
||||
pool: this.options.pool,
|
||||
job: fresh,
|
||||
reason: outcome.reason,
|
||||
classification: 'parse_error',
|
||||
retryable: false,
|
||||
...(this.options.workerId !== undefined ? { workerId: this.options.workerId } : {}),
|
||||
});
|
||||
throw new Error(`generation parse error: ${outcome.reason}`);
|
||||
}
|
||||
|
||||
logger.info('SYSTEM', 'generation completed', {
|
||||
correlationId,
|
||||
jobId: outcome.jobId,
|
||||
bullmqJobId: job.id ?? null,
|
||||
requestId: payloadRequestId,
|
||||
observationCount: outcome.observations.length,
|
||||
privateContentDetected: outcome.privateContentDetected,
|
||||
});
|
||||
|
||||
return {
|
||||
jobId: outcome.jobId,
|
||||
status: 'completed',
|
||||
observationCount: outcome.observations.length,
|
||||
};
|
||||
} catch (error) {
|
||||
const classified = error instanceof ServerClassifiedProviderError ? error : null;
|
||||
const retryable = classified
|
||||
? classified.kind === 'transient' || classified.kind === 'rate_limit'
|
||||
: false;
|
||||
await markGenerationFailed({
|
||||
pool: this.options.pool,
|
||||
job: fresh,
|
||||
reason: error instanceof Error ? error.message : String(error),
|
||||
classification: classified?.kind ?? 'unknown',
|
||||
retryable,
|
||||
...(this.options.workerId !== undefined ? { workerId: this.options.workerId } : {}),
|
||||
});
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
// Phase 11 — load the outbox row by id WITHOUT a scope filter so we can
|
||||
// compare its team_id/project_id to the BullMQ payload as a tampering
|
||||
// detector. Authoritative scope decisions still come from this row, NEVER
|
||||
// from the BullMQ payload.
|
||||
private async loadCanonicalOutbox(jobId: string): Promise<PostgresObservationGenerationJob | null> {
|
||||
const result = await this.options.pool.query<{
|
||||
id: string;
|
||||
project_id: string;
|
||||
team_id: string;
|
||||
agent_event_id: string | null;
|
||||
source_type: 'agent_event' | 'session_summary' | 'observation_reindex';
|
||||
source_id: string;
|
||||
server_session_id: string | null;
|
||||
job_type: string;
|
||||
status: 'queued' | 'processing' | 'completed' | 'failed' | 'cancelled';
|
||||
idempotency_key: string;
|
||||
bullmq_job_id: string | null;
|
||||
attempts: number;
|
||||
max_attempts: number;
|
||||
next_attempt_at: Date | null;
|
||||
locked_at: Date | null;
|
||||
locked_by: string | null;
|
||||
completed_at: Date | null;
|
||||
failed_at: Date | null;
|
||||
cancelled_at: Date | null;
|
||||
last_error: unknown;
|
||||
payload: unknown;
|
||||
created_at: Date;
|
||||
updated_at: Date;
|
||||
}>(
|
||||
'SELECT * FROM observation_generation_jobs WHERE id = $1',
|
||||
[jobId],
|
||||
);
|
||||
const row = result.rows[0];
|
||||
if (!row) return null;
|
||||
return {
|
||||
id: row.id,
|
||||
projectId: row.project_id,
|
||||
teamId: row.team_id,
|
||||
agentEventId: row.agent_event_id,
|
||||
sourceType: row.source_type,
|
||||
sourceId: row.source_id,
|
||||
serverSessionId: row.server_session_id,
|
||||
jobType: row.job_type,
|
||||
status: row.status,
|
||||
idempotencyKey: row.idempotency_key,
|
||||
bullmqJobId: row.bullmq_job_id,
|
||||
attempts: row.attempts,
|
||||
maxAttempts: row.max_attempts,
|
||||
nextAttemptAtEpoch: row.next_attempt_at?.getTime() ?? null,
|
||||
lockedAtEpoch: row.locked_at?.getTime() ?? null,
|
||||
lockedBy: row.locked_by,
|
||||
completedAtEpoch: row.completed_at?.getTime() ?? null,
|
||||
failedAtEpoch: row.failed_at?.getTime() ?? null,
|
||||
cancelledAtEpoch: row.cancelled_at?.getTime() ?? null,
|
||||
lastError: row.last_error && typeof row.last_error === 'object'
|
||||
? (row.last_error as Record<string, unknown>)
|
||||
: null,
|
||||
payload: row.payload && typeof row.payload === 'object' && !Array.isArray(row.payload)
|
||||
? (row.payload as Record<string, unknown>)
|
||||
: {},
|
||||
createdAtEpoch: row.created_at.getTime(),
|
||||
updatedAtEpoch: row.updated_at.getTime(),
|
||||
};
|
||||
}
|
||||
|
||||
private async isApiKeyRevoked(apiKeyId: string): Promise<boolean> {
|
||||
const result = await this.options.pool.query<{ revoked_at: Date | null; expires_at: Date | null }>(
|
||||
'SELECT revoked_at, expires_at FROM api_keys WHERE id = $1',
|
||||
[apiKeyId],
|
||||
);
|
||||
const row = result.rows[0];
|
||||
if (!row) {
|
||||
// The key was deleted entirely. Treat as revoked.
|
||||
return true;
|
||||
}
|
||||
if (row.revoked_at) return true;
|
||||
if (row.expires_at && row.expires_at.getTime() <= Date.now()) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
private async auditScopeViolation(
|
||||
payload: ServerGenerationJobPayload,
|
||||
canonical: PostgresObservationGenerationJob,
|
||||
error: ServerGenerationScopeViolationError,
|
||||
correlationId: string,
|
||||
): Promise<void> {
|
||||
logger.error('SYSTEM', 'BullMQ payload scope mismatch — refusing to generate', {
|
||||
correlationId,
|
||||
generationJobId: payload.generation_job_id,
|
||||
payloadTeamId: payload.team_id,
|
||||
payloadProjectId: payload.project_id,
|
||||
canonicalTeamId: canonical.teamId,
|
||||
canonicalProjectId: canonical.projectId,
|
||||
});
|
||||
await this.auditEvent({
|
||||
teamId: canonical.teamId,
|
||||
projectId: canonical.projectId,
|
||||
apiKeyId: payload.api_key_id,
|
||||
actorId: payload.actor_id,
|
||||
action: 'generation_job.scope_violation',
|
||||
resourceId: canonical.id,
|
||||
details: {
|
||||
reason: 'scope_mismatch',
|
||||
message: error.message,
|
||||
payloadTeamId: payload.team_id,
|
||||
payloadProjectId: payload.project_id,
|
||||
canonicalTeamId: canonical.teamId,
|
||||
canonicalProjectId: canonical.projectId,
|
||||
sourceAdapter: payload.source_adapter,
|
||||
correlationId,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
private async auditRevokedKey(
|
||||
payload: ServerGenerationJobPayload,
|
||||
canonical: PostgresObservationGenerationJob,
|
||||
error: ServerGenerationScopeViolationError,
|
||||
correlationId: string,
|
||||
): Promise<void> {
|
||||
logger.warn('SYSTEM', 'api key revoked between enqueue and execute — refusing to generate', {
|
||||
correlationId,
|
||||
generationJobId: payload.generation_job_id,
|
||||
apiKeyId: payload.api_key_id,
|
||||
});
|
||||
await this.auditEvent({
|
||||
teamId: canonical.teamId,
|
||||
projectId: canonical.projectId,
|
||||
apiKeyId: payload.api_key_id,
|
||||
actorId: payload.actor_id,
|
||||
action: 'generation_job.revoked_key',
|
||||
resourceId: canonical.id,
|
||||
details: {
|
||||
reason: 'revoked_key',
|
||||
message: error.message,
|
||||
sourceAdapter: payload.source_adapter,
|
||||
correlationId,
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
private async auditEvent(input: {
|
||||
teamId: string | null;
|
||||
projectId: string | null;
|
||||
apiKeyId: string | null;
|
||||
actorId: string | null;
|
||||
action: string;
|
||||
resourceId: string | null;
|
||||
details?: Record<string, unknown>;
|
||||
}): Promise<void> {
|
||||
try {
|
||||
const repo = new PostgresAuthRepository(this.options.pool);
|
||||
await repo.createAuditLog({
|
||||
teamId: input.teamId,
|
||||
projectId: input.projectId,
|
||||
actorId: input.actorId,
|
||||
apiKeyId: input.apiKeyId,
|
||||
action: input.action,
|
||||
resourceType: 'observation_generation_job',
|
||||
resourceId: input.resourceId,
|
||||
details: input.details ?? {},
|
||||
});
|
||||
} catch (auditError) {
|
||||
logger.warn('SYSTEM', 'audit_log insert failed in ProviderObservationGenerator', {
|
||||
action: input.action,
|
||||
error: auditError instanceof Error ? auditError.message : String(auditError),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
private async lockOutbox(
|
||||
jobId: string,
|
||||
teamId: string,
|
||||
projectId: string,
|
||||
): Promise<PostgresObservationGenerationJob | null> {
|
||||
const repo = new PostgresObservationGenerationJobRepository(this.options.pool);
|
||||
const current = await repo.getByIdForScope({ id: jobId, projectId, teamId });
|
||||
if (!current) {
|
||||
return null;
|
||||
}
|
||||
if (current.status === 'completed' || current.status === 'cancelled' || current.status === 'failed') {
|
||||
return null;
|
||||
}
|
||||
if (current.status === 'processing') {
|
||||
// Another worker holds the lock — most commonly this fires when BullMQ
|
||||
// redelivers a stalled job to a second worker while the first is still
|
||||
// mid-`provider.generate()`. Returning the row here would cause both
|
||||
// workers to issue the (paid, rate-limited) external provider call,
|
||||
// and the persistence-level terminal-status guard only collapses the
|
||||
// duplicate after the call has already happened. Skip instead. If the
|
||||
// first worker truly died, `reconcileOnStartup` (and the next BullMQ
|
||||
// retry) will resurrect the row.
|
||||
logger.info('SYSTEM', 'generation job already in processing; skipping duplicate worker run', {
|
||||
jobId: current.id,
|
||||
lockedBy: current.lockedBy,
|
||||
lockedAtEpoch: current.lockedAtEpoch,
|
||||
attempts: current.attempts,
|
||||
});
|
||||
return null;
|
||||
}
|
||||
const transitioned = await repo.transitionStatus({
|
||||
id: current.id,
|
||||
projectId: current.projectId,
|
||||
teamId: current.teamId,
|
||||
status: 'processing',
|
||||
lockedBy: this.options.workerId ?? 'server-beta-worker',
|
||||
});
|
||||
return transitioned;
|
||||
}
|
||||
|
||||
private async loadEvents(
|
||||
job: PostgresObservationGenerationJob,
|
||||
payload: ServerGenerationJobPayload,
|
||||
): Promise<NonNullable<Awaited<ReturnType<PostgresAgentEventsRepository['getByIdForScope']>>>[]> {
|
||||
const repo = new PostgresAgentEventsRepository(this.options.pool);
|
||||
|
||||
type Event = NonNullable<Awaited<ReturnType<PostgresAgentEventsRepository['getByIdForScope']>>>;
|
||||
|
||||
if (job.sourceType === 'session_summary') {
|
||||
// Summary jobs feed the provider every event tied to the server_session
|
||||
// that hasn't already been collapsed into a completed event-generation
|
||||
// job. The session repo enforces tenant scope inside its WHERE clause.
|
||||
if (!job.serverSessionId) return [];
|
||||
const sessions = new PostgresServerSessionsRepository(this.options.pool);
|
||||
const events = await sessions.listUnprocessedEvents({
|
||||
serverSessionId: job.serverSessionId,
|
||||
projectId: job.projectId,
|
||||
teamId: job.teamId,
|
||||
});
|
||||
return events;
|
||||
}
|
||||
|
||||
if (job.sourceType !== 'agent_event') {
|
||||
return [];
|
||||
}
|
||||
|
||||
if (payload.kind === 'event') {
|
||||
const event = await repo.getByIdForScope({
|
||||
id: payload.agent_event_id,
|
||||
projectId: job.projectId,
|
||||
teamId: job.teamId,
|
||||
});
|
||||
return event ? [event] : [];
|
||||
}
|
||||
|
||||
if (payload.kind === 'event-batch') {
|
||||
const out: Event[] = [];
|
||||
for (const id of payload.agent_event_ids) {
|
||||
const event = await repo.getByIdForScope({
|
||||
id,
|
||||
projectId: job.projectId,
|
||||
teamId: job.teamId,
|
||||
});
|
||||
if (event) out.push(event);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
return [];
|
||||
}
|
||||
|
||||
private async loadProject(job: PostgresObservationGenerationJob) {
|
||||
const repo = new PostgresProjectsRepository(this.options.pool);
|
||||
return await repo.getByIdForTeam(job.projectId, job.teamId);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,539 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import { parseAgentXml, type ParsedObservation, type ParsedSummary } from '../../sdk/parser.js';
|
||||
import { logger } from '../../utils/logger.js';
|
||||
import {
|
||||
PostgresObservationRepository,
|
||||
PostgresObservationSourcesRepository,
|
||||
buildObservationGenerationKey,
|
||||
type PostgresObservation,
|
||||
} from '../../storage/postgres/observations.js';
|
||||
import {
|
||||
PostgresObservationGenerationJobEventsRepository,
|
||||
PostgresObservationGenerationJobRepository,
|
||||
type PostgresObservationGenerationJob,
|
||||
} from '../../storage/postgres/generation-jobs.js';
|
||||
import { PostgresAuthRepository } from '../../storage/postgres/auth.js';
|
||||
import {
|
||||
withPostgresTransaction,
|
||||
type PostgresPool,
|
||||
} from '../../storage/postgres/pool.js';
|
||||
import { stripTags } from '../../utils/tag-stripping.js';
|
||||
|
||||
// processGeneratedResponse owns the full "we got XML from a provider →
|
||||
// persist + link + advance outbox" pipeline. Every side-effect runs inside
|
||||
// a single Postgres transaction so retries are idempotent:
|
||||
//
|
||||
// - observations.generation_key (UNIQUE per team/project) collapses retry
|
||||
// duplicates to a single row.
|
||||
// - observation_sources (UNIQUE on observation_id, source_type, source_id)
|
||||
// collapses duplicate source links.
|
||||
// - observation_generation_jobs.transitionStatus is the lifecycle gate.
|
||||
//
|
||||
// The function NEVER touches worker SessionStore tables, NEVER assumes a
|
||||
// Claude Code transcript shape, and ALWAYS reloads the job before mutating.
|
||||
// BullMQ payload data is advisory; the outbox row is canonical.
|
||||
|
||||
export type ProcessGeneratedResponseOutcome =
|
||||
| {
|
||||
kind: 'completed';
|
||||
jobId: string;
|
||||
observations: PostgresObservation[];
|
||||
privateContentDetected: boolean;
|
||||
}
|
||||
| { kind: 'parse_error'; jobId: string; reason: string };
|
||||
|
||||
export interface ProcessGeneratedResponseInput {
|
||||
pool: PostgresPool;
|
||||
job: PostgresObservationGenerationJob;
|
||||
rawText: string;
|
||||
modelId?: string;
|
||||
providerLabel: string;
|
||||
workerId?: string;
|
||||
// Phase 11 — identity context propagated from the BullMQ payload (and
|
||||
// ultimately the API-key that ingested the source row). Persisted on
|
||||
// observation_sources.metadata for traceability and re-emitted in the
|
||||
// observation.created audit row.
|
||||
apiKeyId?: string | null;
|
||||
actorId?: string | null;
|
||||
sourceAdapter?: string | null;
|
||||
}
|
||||
|
||||
export async function processGeneratedResponse(
|
||||
input: ProcessGeneratedResponseInput,
|
||||
): Promise<ProcessGeneratedResponseOutcome> {
|
||||
const { job, rawText } = input;
|
||||
|
||||
const parsed = parseAgentXml(rawText, job.id);
|
||||
if (!parsed.valid) {
|
||||
return { kind: 'parse_error', jobId: job.id, reason: 'parser rejected response' };
|
||||
}
|
||||
|
||||
// Skip-summary or zero-observation responses are still a success — the
|
||||
// provider explicitly decided there's nothing worth recording (e.g.
|
||||
// privacy-stripped batch). Mark the job completed with no observations.
|
||||
const observationsToWrite = parsed.observations ?? [];
|
||||
const skipped = parsed.summary?.skipped === true;
|
||||
const privateContentDetected = skipped || observationsToWrite.length === 0;
|
||||
|
||||
return await withPostgresTransaction(input.pool, async (client) => {
|
||||
const obsRepo = new PostgresObservationRepository(client);
|
||||
const sourcesRepo = new PostgresObservationSourcesRepository(client);
|
||||
const jobsRepo = new PostgresObservationGenerationJobRepository(client);
|
||||
const eventsLogRepo = new PostgresObservationGenerationJobEventsRepository(client);
|
||||
const auditRepo = new PostgresAuthRepository(client);
|
||||
|
||||
// Reload the job inside the transaction. If it was already completed
|
||||
// by another worker, return its existing observations idempotently.
|
||||
const fresh = await jobsRepo.getByIdForScope({
|
||||
id: job.id,
|
||||
projectId: job.projectId,
|
||||
teamId: job.teamId,
|
||||
});
|
||||
if (!fresh) {
|
||||
throw new Error(`generation job ${job.id} not found in scope`);
|
||||
}
|
||||
if (fresh.status === 'completed' || fresh.status === 'cancelled' || fresh.status === 'failed') {
|
||||
logger.info('SYSTEM', 'generation job already in terminal status; skipping persistence', {
|
||||
jobId: fresh.id,
|
||||
status: fresh.status,
|
||||
});
|
||||
return {
|
||||
kind: 'completed' as const,
|
||||
jobId: fresh.id,
|
||||
observations: [],
|
||||
privateContentDetected,
|
||||
};
|
||||
}
|
||||
|
||||
const persisted: PostgresObservation[] = [];
|
||||
for (let index = 0; index < observationsToWrite.length; index++) {
|
||||
const parsedObservation = observationsToWrite[index]!;
|
||||
const content = renderObservationContent(parsedObservation);
|
||||
if (!content || content.trim().length === 0) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Defense-in-depth: even if the parser slipped a private-tagged
|
||||
// string through, scrub before persisting.
|
||||
const scrubbed = stripTags(content);
|
||||
if (!scrubbed.stripped || scrubbed.stripped.trim().length === 0) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const generationKey = buildObservationGenerationKey({
|
||||
generationJobId: fresh.id,
|
||||
parsedObservationIndex: index,
|
||||
content: scrubbed.stripped,
|
||||
});
|
||||
|
||||
const observation = await obsRepo.create({
|
||||
projectId: fresh.projectId,
|
||||
teamId: fresh.teamId,
|
||||
serverSessionId: fresh.serverSessionId,
|
||||
kind: parsedObservation.type ?? 'observation',
|
||||
content: scrubbed.stripped,
|
||||
generationKey,
|
||||
metadata: {
|
||||
title: parsedObservation.title,
|
||||
subtitle: parsedObservation.subtitle,
|
||||
facts: parsedObservation.facts,
|
||||
narrative: parsedObservation.narrative,
|
||||
concepts: parsedObservation.concepts,
|
||||
files_read: parsedObservation.files_read,
|
||||
files_modified: parsedObservation.files_modified,
|
||||
provider: input.providerLabel,
|
||||
model: input.modelId ?? null,
|
||||
},
|
||||
createdByJobId: fresh.id,
|
||||
});
|
||||
persisted.push(observation);
|
||||
|
||||
await sourcesRepo.addSource({
|
||||
observationId: observation.id,
|
||||
projectId: fresh.projectId,
|
||||
teamId: fresh.teamId,
|
||||
sourceType: fresh.sourceType,
|
||||
sourceId: fresh.sourceId,
|
||||
agentEventId: fresh.agentEventId ?? null,
|
||||
generationJobId: fresh.id,
|
||||
metadata: {
|
||||
provider: input.providerLabel,
|
||||
parsedObservationIndex: index,
|
||||
// Phase 11 — denormalize identity context for traceability so an
|
||||
// operator can answer "which api key produced this observation?"
|
||||
// without joining back through generation_job → outbox → key.
|
||||
source_adapter: input.sourceAdapter ?? null,
|
||||
actor_id: input.actorId ?? null,
|
||||
api_key_id: input.apiKeyId ?? null,
|
||||
},
|
||||
});
|
||||
|
||||
// Phase 11 — audit each generated observation. Using the SAME
|
||||
// generation_job_id reference so the audit chain (event_received →
|
||||
// generation_job.queued → generation_job.processing → observation.
|
||||
// created → observation.read) can be reconstructed.
|
||||
try {
|
||||
await auditRepo.createAuditLog({
|
||||
teamId: fresh.teamId,
|
||||
projectId: fresh.projectId,
|
||||
actorId: input.actorId ?? null,
|
||||
apiKeyId: input.apiKeyId ?? null,
|
||||
action: 'observation.created',
|
||||
resourceType: 'observation',
|
||||
resourceId: observation.id,
|
||||
details: {
|
||||
generationJobId: fresh.id,
|
||||
sourceType: fresh.sourceType,
|
||||
sourceId: fresh.sourceId,
|
||||
provider: input.providerLabel,
|
||||
model: input.modelId ?? null,
|
||||
sourceAdapter: input.sourceAdapter ?? null,
|
||||
parsedObservationIndex: index,
|
||||
},
|
||||
});
|
||||
} catch (auditError) {
|
||||
logger.warn('SYSTEM', 'audit_log observation.created insert failed', {
|
||||
observationId: observation.id,
|
||||
error: auditError instanceof Error ? auditError.message : String(auditError),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Advance outbox status. Phase 1 transitionStatus enforces legal
|
||||
// transitions and tenant scope inside its WHERE clause.
|
||||
await jobsRepo.transitionStatus({
|
||||
id: fresh.id,
|
||||
projectId: fresh.projectId,
|
||||
teamId: fresh.teamId,
|
||||
status: 'completed',
|
||||
});
|
||||
await eventsLogRepo.append({
|
||||
generationJobId: fresh.id,
|
||||
projectId: fresh.projectId,
|
||||
teamId: fresh.teamId,
|
||||
eventType: 'completed',
|
||||
statusAfter: 'completed',
|
||||
attempt: fresh.attempts,
|
||||
details: {
|
||||
provider: input.providerLabel,
|
||||
model: input.modelId ?? null,
|
||||
observationCount: persisted.length,
|
||||
privateContentDetected,
|
||||
workerId: input.workerId ?? null,
|
||||
},
|
||||
});
|
||||
|
||||
// Audit log — best-effort; failure here would already be inside the
|
||||
// transaction so any insert error rolls everything back. We accept
|
||||
// that to keep the pipeline observable end-to-end.
|
||||
try {
|
||||
await auditRepo.createAuditLog({
|
||||
teamId: fresh.teamId,
|
||||
projectId: fresh.projectId,
|
||||
actorId: input.actorId ?? null,
|
||||
apiKeyId: input.apiKeyId ?? null,
|
||||
action: 'generation_job.completed',
|
||||
resourceType: 'observation_generation_job',
|
||||
resourceId: fresh.id,
|
||||
details: {
|
||||
generationJobId: fresh.id,
|
||||
provider: input.providerLabel,
|
||||
model: input.modelId ?? null,
|
||||
observationCount: persisted.length,
|
||||
observationIds: persisted.map(o => o.id),
|
||||
sourceAdapter: input.sourceAdapter ?? null,
|
||||
},
|
||||
});
|
||||
} catch (auditError) {
|
||||
// The audit log table may not have a metadata column on older
|
||||
// schemas; swallow rather than failing generation.
|
||||
logger.warn('SYSTEM', 'audit log insert failed during generation', {
|
||||
jobId: fresh.id,
|
||||
error: auditError instanceof Error ? auditError.message : String(auditError),
|
||||
});
|
||||
}
|
||||
|
||||
return {
|
||||
kind: 'completed' as const,
|
||||
jobId: fresh.id,
|
||||
observations: persisted,
|
||||
privateContentDetected,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
export interface MarkGenerationFailedInput {
|
||||
pool: PostgresPool;
|
||||
job: PostgresObservationGenerationJob;
|
||||
reason: string;
|
||||
classification?: string;
|
||||
retryable: boolean;
|
||||
workerId?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Move a generation job to a non-success terminal state. Used when the
|
||||
* provider returned an error or invalid XML. Retryable failures move the
|
||||
* job back to `queued` so reconciliation can re-enqueue; non-retryable
|
||||
* failures move to `failed`.
|
||||
*/
|
||||
export async function markGenerationFailed(input: MarkGenerationFailedInput): Promise<void> {
|
||||
await withPostgresTransaction(input.pool, async (client) => {
|
||||
const jobsRepo = new PostgresObservationGenerationJobRepository(client);
|
||||
const eventsLogRepo = new PostgresObservationGenerationJobEventsRepository(client);
|
||||
|
||||
const fresh = await jobsRepo.getByIdForScope({
|
||||
id: input.job.id,
|
||||
projectId: input.job.projectId,
|
||||
teamId: input.job.teamId,
|
||||
});
|
||||
if (!fresh || fresh.status === 'completed' || fresh.status === 'cancelled') {
|
||||
return;
|
||||
}
|
||||
|
||||
const canRetry = input.retryable && fresh.attempts < fresh.maxAttempts;
|
||||
const target = canRetry ? 'queued' : 'failed';
|
||||
|
||||
await jobsRepo.transitionStatus({
|
||||
id: fresh.id,
|
||||
projectId: fresh.projectId,
|
||||
teamId: fresh.teamId,
|
||||
status: target,
|
||||
lastError: { reason: input.reason, classification: input.classification ?? null },
|
||||
...(canRetry ? { nextAttemptAt: new Date(Date.now() + retryDelayMs(fresh.attempts)) } : {}),
|
||||
});
|
||||
|
||||
await eventsLogRepo.append({
|
||||
generationJobId: fresh.id,
|
||||
projectId: fresh.projectId,
|
||||
teamId: fresh.teamId,
|
||||
eventType: canRetry ? 'retry_scheduled' : 'failed',
|
||||
statusAfter: target,
|
||||
attempt: fresh.attempts,
|
||||
details: {
|
||||
reason: input.reason,
|
||||
classification: input.classification ?? null,
|
||||
workerId: input.workerId ?? null,
|
||||
},
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Persist a parsed session summary as an observations row with kind='summary'.
|
||||
*
|
||||
* Wraps the same outbox transition / source-link / audit pipeline as
|
||||
* processGeneratedResponse but emits a single 'summary'-kind observation
|
||||
* derived from the summary fields. Idempotency is enforced through the same
|
||||
* `observations.generation_key` UNIQUE index — re-running the summary job
|
||||
* after a restart will collapse to one row.
|
||||
*/
|
||||
export async function processSessionSummaryResponse(
|
||||
input: ProcessGeneratedResponseInput,
|
||||
): Promise<ProcessGeneratedResponseOutcome> {
|
||||
const { job, rawText } = input;
|
||||
|
||||
if (job.sourceType !== 'session_summary') {
|
||||
return { kind: 'parse_error', jobId: job.id, reason: 'session summary processor invoked on non-summary job' };
|
||||
}
|
||||
|
||||
const parsed = parseAgentXml(rawText, job.id);
|
||||
if (!parsed.valid) {
|
||||
return { kind: 'parse_error', jobId: job.id, reason: 'parser rejected summary response' };
|
||||
}
|
||||
|
||||
const summary = parsed.summary ?? null;
|
||||
const skipped = summary?.skipped === true;
|
||||
const summaryContent = summary ? renderSummaryContent(summary) : '';
|
||||
const privateContentDetected = skipped || summaryContent.trim().length === 0;
|
||||
|
||||
return await withPostgresTransaction(input.pool, async (client) => {
|
||||
const obsRepo = new PostgresObservationRepository(client);
|
||||
const sourcesRepo = new PostgresObservationSourcesRepository(client);
|
||||
const jobsRepo = new PostgresObservationGenerationJobRepository(client);
|
||||
const eventsLogRepo = new PostgresObservationGenerationJobEventsRepository(client);
|
||||
const auditRepo = new PostgresAuthRepository(client);
|
||||
|
||||
const fresh = await jobsRepo.getByIdForScope({
|
||||
id: job.id,
|
||||
projectId: job.projectId,
|
||||
teamId: job.teamId,
|
||||
});
|
||||
if (!fresh) {
|
||||
throw new Error(`session summary generation job ${job.id} not found in scope`);
|
||||
}
|
||||
if (fresh.status === 'completed' || fresh.status === 'cancelled' || fresh.status === 'failed') {
|
||||
logger.info('SYSTEM', 'session summary job already in terminal status; skipping persistence', {
|
||||
jobId: fresh.id,
|
||||
status: fresh.status,
|
||||
});
|
||||
return {
|
||||
kind: 'completed' as const,
|
||||
jobId: fresh.id,
|
||||
observations: [],
|
||||
privateContentDetected,
|
||||
};
|
||||
}
|
||||
|
||||
const persisted: PostgresObservation[] = [];
|
||||
if (!privateContentDetected) {
|
||||
const scrubbed = stripTags(summaryContent);
|
||||
const scrubbedContent = scrubbed.stripped ?? '';
|
||||
if (scrubbedContent.trim().length > 0) {
|
||||
const generationKey = buildObservationGenerationKey({
|
||||
generationJobId: fresh.id,
|
||||
parsedObservationIndex: 0,
|
||||
content: scrubbedContent,
|
||||
});
|
||||
const observation = await obsRepo.create({
|
||||
projectId: fresh.projectId,
|
||||
teamId: fresh.teamId,
|
||||
serverSessionId: fresh.serverSessionId,
|
||||
kind: 'summary',
|
||||
content: scrubbedContent,
|
||||
generationKey,
|
||||
metadata: {
|
||||
request: summary?.request ?? null,
|
||||
investigated: summary?.investigated ?? null,
|
||||
learned: summary?.learned ?? null,
|
||||
completed: summary?.completed ?? null,
|
||||
next_steps: summary?.next_steps ?? null,
|
||||
notes: summary?.notes ?? null,
|
||||
provider: input.providerLabel,
|
||||
model: input.modelId ?? null,
|
||||
},
|
||||
createdByJobId: fresh.id,
|
||||
});
|
||||
persisted.push(observation);
|
||||
|
||||
await sourcesRepo.addSource({
|
||||
observationId: observation.id,
|
||||
projectId: fresh.projectId,
|
||||
teamId: fresh.teamId,
|
||||
sourceType: 'session_summary',
|
||||
sourceId: fresh.sourceId,
|
||||
generationJobId: fresh.id,
|
||||
metadata: {
|
||||
provider: input.providerLabel,
|
||||
parsedObservationIndex: 0,
|
||||
source_adapter: input.sourceAdapter ?? null,
|
||||
actor_id: input.actorId ?? null,
|
||||
api_key_id: input.apiKeyId ?? null,
|
||||
},
|
||||
});
|
||||
|
||||
// Phase 11 — observation.created audit for the summary observation.
|
||||
try {
|
||||
await auditRepo.createAuditLog({
|
||||
teamId: fresh.teamId,
|
||||
projectId: fresh.projectId,
|
||||
actorId: input.actorId ?? null,
|
||||
apiKeyId: input.apiKeyId ?? null,
|
||||
action: 'observation.created',
|
||||
resourceType: 'observation',
|
||||
resourceId: observation.id,
|
||||
details: {
|
||||
generationJobId: fresh.id,
|
||||
sourceType: 'session_summary',
|
||||
sourceId: fresh.sourceId,
|
||||
provider: input.providerLabel,
|
||||
model: input.modelId ?? null,
|
||||
sourceAdapter: input.sourceAdapter ?? null,
|
||||
kind: 'summary',
|
||||
},
|
||||
});
|
||||
} catch (auditError) {
|
||||
logger.warn('SYSTEM', 'audit_log observation.created (summary) insert failed', {
|
||||
observationId: observation.id,
|
||||
error: auditError instanceof Error ? auditError.message : String(auditError),
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
await jobsRepo.transitionStatus({
|
||||
id: fresh.id,
|
||||
projectId: fresh.projectId,
|
||||
teamId: fresh.teamId,
|
||||
status: 'completed',
|
||||
});
|
||||
await eventsLogRepo.append({
|
||||
generationJobId: fresh.id,
|
||||
projectId: fresh.projectId,
|
||||
teamId: fresh.teamId,
|
||||
eventType: 'completed',
|
||||
statusAfter: 'completed',
|
||||
attempt: fresh.attempts,
|
||||
details: {
|
||||
provider: input.providerLabel,
|
||||
model: input.modelId ?? null,
|
||||
observationCount: persisted.length,
|
||||
privateContentDetected,
|
||||
workerId: input.workerId ?? null,
|
||||
sourceType: 'session_summary',
|
||||
},
|
||||
});
|
||||
|
||||
try {
|
||||
await auditRepo.createAuditLog({
|
||||
teamId: fresh.teamId,
|
||||
projectId: fresh.projectId,
|
||||
actorId: input.actorId ?? null,
|
||||
apiKeyId: input.apiKeyId ?? null,
|
||||
action: 'generation_job.completed',
|
||||
resourceType: 'observation_generation_job',
|
||||
resourceId: fresh.id,
|
||||
details: {
|
||||
generationJobId: fresh.id,
|
||||
provider: input.providerLabel,
|
||||
model: input.modelId ?? null,
|
||||
observationCount: persisted.length,
|
||||
observationIds: persisted.map(o => o.id),
|
||||
sourceAdapter: input.sourceAdapter ?? null,
|
||||
sourceType: 'session_summary',
|
||||
},
|
||||
});
|
||||
} catch (auditError) {
|
||||
logger.warn('SYSTEM', 'audit log insert failed during summary generation', {
|
||||
jobId: fresh.id,
|
||||
error: auditError instanceof Error ? auditError.message : String(auditError),
|
||||
});
|
||||
}
|
||||
|
||||
return {
|
||||
kind: 'completed' as const,
|
||||
jobId: fresh.id,
|
||||
observations: persisted,
|
||||
privateContentDetected,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
function renderSummaryContent(summary: ParsedSummary): string {
|
||||
const parts: string[] = [];
|
||||
if (summary.request) parts.push(`Request: ${summary.request}`);
|
||||
if (summary.investigated) parts.push(`Investigated: ${summary.investigated}`);
|
||||
if (summary.learned) parts.push(`Learned: ${summary.learned}`);
|
||||
if (summary.completed) parts.push(`Completed: ${summary.completed}`);
|
||||
if (summary.next_steps) parts.push(`Next steps: ${summary.next_steps}`);
|
||||
if (summary.notes) parts.push(`Notes: ${summary.notes}`);
|
||||
return parts.join('\n\n').trim();
|
||||
}
|
||||
|
||||
function renderObservationContent(observation: ParsedObservation): string {
|
||||
const parts: string[] = [];
|
||||
if (observation.title) parts.push(observation.title);
|
||||
if (observation.subtitle) parts.push(observation.subtitle);
|
||||
if (observation.narrative) parts.push(observation.narrative);
|
||||
if (observation.facts && observation.facts.length > 0) {
|
||||
parts.push(observation.facts.map(f => `- ${f}`).join('\n'));
|
||||
}
|
||||
return parts.join('\n\n').trim();
|
||||
}
|
||||
|
||||
function retryDelayMs(attempts: number): number {
|
||||
// Exponential backoff: 5s, 25s, 125s, capped at 10 minutes.
|
||||
const base = 5000 * Math.pow(5, Math.max(0, attempts));
|
||||
return Math.min(base, 10 * 60 * 1000);
|
||||
}
|
||||
@@ -0,0 +1,247 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import { logger } from '../../../utils/logger.js';
|
||||
import {
|
||||
ServerClassifiedProviderError,
|
||||
parseRetryAfterMs,
|
||||
} from './shared/error-classification.js';
|
||||
import { buildServerGenerationPrompt } from './shared/prompt-builder.js';
|
||||
import type {
|
||||
ServerGenerationContext,
|
||||
ServerGenerationProvider,
|
||||
ServerGenerationResult,
|
||||
} from './shared/types.js';
|
||||
|
||||
const ANTHROPIC_API_URL = 'https://api.anthropic.com/v1/messages';
|
||||
const ANTHROPIC_VERSION = '2023-06-01';
|
||||
const DEFAULT_MODEL = 'claude-3-5-sonnet-latest';
|
||||
|
||||
export interface ClaudeObservationProviderOptions {
|
||||
apiKey: string;
|
||||
model?: string;
|
||||
maxOutputTokens?: number;
|
||||
fetchImpl?: typeof fetch;
|
||||
}
|
||||
|
||||
interface AnthropicMessagesResponse {
|
||||
content?: Array<{ type?: string; text?: string }>;
|
||||
usage?: { input_tokens?: number; output_tokens?: number };
|
||||
error?: { type?: string; message?: string };
|
||||
}
|
||||
|
||||
export class ClaudeObservationProvider implements ServerGenerationProvider {
|
||||
readonly providerLabel = 'claude' as const;
|
||||
private readonly apiKey: string;
|
||||
private readonly model: string;
|
||||
private readonly maxOutputTokens: number;
|
||||
private readonly fetchImpl: typeof fetch;
|
||||
|
||||
constructor(options: ClaudeObservationProviderOptions) {
|
||||
if (!options.apiKey) {
|
||||
throw new ServerClassifiedProviderError('Anthropic API key not configured', {
|
||||
kind: 'auth_invalid',
|
||||
cause: new Error('apiKey is required'),
|
||||
});
|
||||
}
|
||||
this.apiKey = options.apiKey;
|
||||
this.model = options.model ?? DEFAULT_MODEL;
|
||||
this.maxOutputTokens = options.maxOutputTokens ?? 4096;
|
||||
this.fetchImpl = options.fetchImpl ?? fetch;
|
||||
}
|
||||
|
||||
async generate(
|
||||
context: ServerGenerationContext,
|
||||
signal?: AbortSignal,
|
||||
): Promise<ServerGenerationResult> {
|
||||
const { prompt, skippedAll } = buildServerGenerationPrompt(context);
|
||||
if (skippedAll) {
|
||||
// All events were scrubbed by privacy stripping. Don't bill the
|
||||
// provider — return a synthetic skip response that parser accepts.
|
||||
return {
|
||||
rawText: '<skip_summary reason="all_events_private" />',
|
||||
providerLabel: this.providerLabel,
|
||||
modelId: this.model,
|
||||
};
|
||||
}
|
||||
|
||||
let response: Response;
|
||||
try {
|
||||
response = await this.fetchImpl(ANTHROPIC_API_URL, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'x-api-key': this.apiKey,
|
||||
'anthropic-version': ANTHROPIC_VERSION,
|
||||
},
|
||||
body: JSON.stringify({
|
||||
model: this.model,
|
||||
max_tokens: this.maxOutputTokens,
|
||||
temperature: 0.3,
|
||||
messages: [{ role: 'user', content: prompt }],
|
||||
}),
|
||||
signal,
|
||||
});
|
||||
} catch (networkError) {
|
||||
throw classifyClaudeServerError({
|
||||
cause: networkError,
|
||||
});
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
const bodyText = await safeReadBody(response);
|
||||
throw classifyClaudeServerError({
|
||||
status: response.status,
|
||||
bodyText,
|
||||
headers: response.headers,
|
||||
cause: new Error(`Anthropic API error: ${response.status} - ${bodyText}`),
|
||||
});
|
||||
}
|
||||
|
||||
let data: AnthropicMessagesResponse;
|
||||
try {
|
||||
data = (await response.json()) as AnthropicMessagesResponse;
|
||||
} catch (parseError) {
|
||||
throw new ServerClassifiedProviderError('Anthropic returned invalid JSON', {
|
||||
kind: 'parse_error',
|
||||
cause: parseError,
|
||||
});
|
||||
}
|
||||
|
||||
if (data.error) {
|
||||
throw classifyClaudeServerError({
|
||||
status: response.status,
|
||||
bodyText: `${data.error.type ?? ''} ${data.error.message ?? ''}`,
|
||||
headers: response.headers,
|
||||
cause: new Error(`Anthropic API error: ${data.error.type} - ${data.error.message}`),
|
||||
});
|
||||
}
|
||||
|
||||
const blocks = Array.isArray(data.content) ? data.content : [];
|
||||
const rawText = blocks
|
||||
.filter(block => block?.type === 'text' && typeof block.text === 'string')
|
||||
.map(block => block.text!)
|
||||
.join('\n')
|
||||
.trim();
|
||||
|
||||
if (!rawText) {
|
||||
logger.warn('SDK', 'Anthropic returned empty content array', {
|
||||
provider: 'claude',
|
||||
model: this.model,
|
||||
});
|
||||
}
|
||||
|
||||
const usage = data.usage ?? {};
|
||||
const tokensUsed =
|
||||
typeof usage.input_tokens === 'number' || typeof usage.output_tokens === 'number'
|
||||
? (usage.input_tokens ?? 0) + (usage.output_tokens ?? 0)
|
||||
: undefined;
|
||||
|
||||
return {
|
||||
rawText,
|
||||
...(tokensUsed !== undefined ? { tokensUsed } : {}),
|
||||
providerLabel: this.providerLabel,
|
||||
modelId: this.model,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
interface ClassifyInput {
|
||||
status?: number;
|
||||
bodyText?: string;
|
||||
headers?: Headers | { get(name: string): string | null };
|
||||
cause: unknown;
|
||||
}
|
||||
|
||||
/**
|
||||
* Anthropic-specific HTTP error classification. Mirrors worker
|
||||
* `classifyClaudeError`, but extracted for server-beta and rebound to
|
||||
* Anthropic Messages REST semantics rather than SDK error classes.
|
||||
*/
|
||||
export function classifyClaudeServerError(input: ClassifyInput): ServerClassifiedProviderError {
|
||||
const status = input.status;
|
||||
const body = input.bodyText ?? '';
|
||||
const lower = body.toLowerCase();
|
||||
const retryAfterMs = input.headers ? parseRetryAfterMs(input.headers.get('retry-after')) : undefined;
|
||||
|
||||
if (lower.includes('overloaded')) {
|
||||
return new ServerClassifiedProviderError(
|
||||
`Anthropic overloaded${status !== undefined ? ` (status ${status})` : ''}`,
|
||||
{ kind: 'transient', cause: input.cause },
|
||||
);
|
||||
}
|
||||
|
||||
if (status === 401 || status === 403 || lower.includes('invalid api key')) {
|
||||
return new ServerClassifiedProviderError(
|
||||
`Anthropic auth invalid${status !== undefined ? ` (status ${status})` : ''}`,
|
||||
{ kind: 'auth_invalid', cause: input.cause },
|
||||
);
|
||||
}
|
||||
|
||||
if (status === 429) {
|
||||
return new ServerClassifiedProviderError('Anthropic rate limit (429)', {
|
||||
kind: 'rate_limit',
|
||||
cause: input.cause,
|
||||
...(retryAfterMs !== undefined ? { retryAfterMs } : {}),
|
||||
});
|
||||
}
|
||||
|
||||
if (lower.includes('quota exceeded')) {
|
||||
return new ServerClassifiedProviderError('Anthropic quota exhausted', {
|
||||
kind: 'quota_exhausted',
|
||||
cause: input.cause,
|
||||
});
|
||||
}
|
||||
|
||||
if (
|
||||
lower.includes('prompt is too long') ||
|
||||
lower.includes('context window') ||
|
||||
lower.includes('max_tokens')
|
||||
) {
|
||||
return new ServerClassifiedProviderError('Anthropic context overflow', {
|
||||
kind: 'unrecoverable',
|
||||
cause: input.cause,
|
||||
});
|
||||
}
|
||||
|
||||
if (status === 529) {
|
||||
return new ServerClassifiedProviderError('Anthropic overloaded (529)', {
|
||||
kind: 'transient',
|
||||
cause: input.cause,
|
||||
});
|
||||
}
|
||||
|
||||
if (status !== undefined && status >= 500 && status < 600) {
|
||||
return new ServerClassifiedProviderError(`Anthropic upstream error (status ${status})`, {
|
||||
kind: 'transient',
|
||||
cause: input.cause,
|
||||
});
|
||||
}
|
||||
|
||||
if (status === 400) {
|
||||
return new ServerClassifiedProviderError('Anthropic bad request (400)', {
|
||||
kind: 'unrecoverable',
|
||||
cause: input.cause,
|
||||
});
|
||||
}
|
||||
|
||||
if (status === undefined) {
|
||||
const message = input.cause instanceof Error ? input.cause.message : String(input.cause);
|
||||
return new ServerClassifiedProviderError(`Anthropic network error: ${message}`, {
|
||||
kind: 'transient',
|
||||
cause: input.cause,
|
||||
});
|
||||
}
|
||||
|
||||
return new ServerClassifiedProviderError(
|
||||
`Anthropic API error: ${status}${body ? ` - ${body.substring(0, 200)}` : ''}`,
|
||||
{ kind: 'unrecoverable', cause: input.cause },
|
||||
);
|
||||
}
|
||||
|
||||
async function safeReadBody(response: Response): Promise<string> {
|
||||
try {
|
||||
return await response.text();
|
||||
} catch {
|
||||
return '';
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,148 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import { logger } from '../../../utils/logger.js';
|
||||
import {
|
||||
ServerClassifiedProviderError,
|
||||
classifyHttpProviderError,
|
||||
parseRetryAfterMs,
|
||||
} from './shared/error-classification.js';
|
||||
import { buildServerGenerationPrompt } from './shared/prompt-builder.js';
|
||||
import type {
|
||||
ServerGenerationContext,
|
||||
ServerGenerationProvider,
|
||||
ServerGenerationResult,
|
||||
} from './shared/types.js';
|
||||
|
||||
const GEMINI_API_URL = 'https://generativelanguage.googleapis.com/v1/models';
|
||||
const DEFAULT_MODEL = 'gemini-2.5-flash';
|
||||
|
||||
export interface GeminiObservationProviderOptions {
|
||||
apiKey: string;
|
||||
model?: string;
|
||||
maxOutputTokens?: number;
|
||||
fetchImpl?: typeof fetch;
|
||||
}
|
||||
|
||||
interface GeminiResponse {
|
||||
candidates?: Array<{
|
||||
content?: { parts?: Array<{ text?: string }> };
|
||||
}>;
|
||||
usageMetadata?: { totalTokenCount?: number };
|
||||
error?: { code?: number; status?: string; message?: string };
|
||||
}
|
||||
|
||||
export class GeminiObservationProvider implements ServerGenerationProvider {
|
||||
readonly providerLabel = 'gemini' as const;
|
||||
private readonly apiKey: string;
|
||||
private readonly model: string;
|
||||
private readonly maxOutputTokens: number;
|
||||
private readonly fetchImpl: typeof fetch;
|
||||
|
||||
constructor(options: GeminiObservationProviderOptions) {
|
||||
if (!options.apiKey) {
|
||||
throw new ServerClassifiedProviderError('Gemini API key not configured', {
|
||||
kind: 'auth_invalid',
|
||||
cause: new Error('apiKey is required'),
|
||||
});
|
||||
}
|
||||
this.apiKey = options.apiKey;
|
||||
this.model = options.model ?? DEFAULT_MODEL;
|
||||
this.maxOutputTokens = options.maxOutputTokens ?? 4096;
|
||||
this.fetchImpl = options.fetchImpl ?? fetch;
|
||||
}
|
||||
|
||||
async generate(
|
||||
context: ServerGenerationContext,
|
||||
signal?: AbortSignal,
|
||||
): Promise<ServerGenerationResult> {
|
||||
const { prompt, skippedAll } = buildServerGenerationPrompt(context);
|
||||
if (skippedAll) {
|
||||
return {
|
||||
rawText: '<skip_summary reason="all_events_private" />',
|
||||
providerLabel: this.providerLabel,
|
||||
modelId: this.model,
|
||||
};
|
||||
}
|
||||
|
||||
const url = `${GEMINI_API_URL}/${encodeURIComponent(this.model)}:generateContent?key=${encodeURIComponent(this.apiKey)}`;
|
||||
|
||||
let response: Response;
|
||||
try {
|
||||
response = await this.fetchImpl(url, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
contents: [{ role: 'user', parts: [{ text: prompt }] }],
|
||||
generationConfig: {
|
||||
temperature: 0.3,
|
||||
maxOutputTokens: this.maxOutputTokens,
|
||||
},
|
||||
}),
|
||||
signal,
|
||||
});
|
||||
} catch (networkError) {
|
||||
throw classifyHttpProviderError({
|
||||
cause: networkError,
|
||||
providerLabel: 'Gemini',
|
||||
});
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
const bodyText = await safeReadBody(response);
|
||||
throw classifyHttpProviderError({
|
||||
status: response.status,
|
||||
bodyText,
|
||||
headers: response.headers,
|
||||
cause: new Error(`Gemini API error: ${response.status} - ${bodyText}`),
|
||||
providerLabel: 'Gemini',
|
||||
});
|
||||
}
|
||||
|
||||
let data: GeminiResponse;
|
||||
try {
|
||||
data = (await response.json()) as GeminiResponse;
|
||||
} catch (parseError) {
|
||||
throw new ServerClassifiedProviderError('Gemini returned invalid JSON', {
|
||||
kind: 'parse_error',
|
||||
cause: parseError,
|
||||
});
|
||||
}
|
||||
|
||||
if (data.error) {
|
||||
throw classifyHttpProviderError({
|
||||
status: response.status,
|
||||
bodyText: `${data.error.status ?? ''} ${data.error.message ?? ''}`,
|
||||
headers: response.headers,
|
||||
cause: new Error(`Gemini API error: ${data.error.status} - ${data.error.message}`),
|
||||
providerLabel: 'Gemini',
|
||||
});
|
||||
}
|
||||
|
||||
const rawText = data.candidates?.[0]?.content?.parts?.[0]?.text?.trim() ?? '';
|
||||
if (!rawText) {
|
||||
logger.warn('SDK', 'Gemini returned empty content', { provider: 'gemini', model: this.model });
|
||||
}
|
||||
|
||||
const tokensUsed = typeof data.usageMetadata?.totalTokenCount === 'number'
|
||||
? data.usageMetadata.totalTokenCount
|
||||
: undefined;
|
||||
|
||||
return {
|
||||
rawText,
|
||||
...(tokensUsed !== undefined ? { tokensUsed } : {}),
|
||||
providerLabel: this.providerLabel,
|
||||
modelId: this.model,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
// Re-export for tests/auditing parity with worker classifier surface.
|
||||
export { parseRetryAfterMs };
|
||||
|
||||
async function safeReadBody(response: Response): Promise<string> {
|
||||
try {
|
||||
return await response.text();
|
||||
} catch {
|
||||
return '';
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,151 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import { logger } from '../../../utils/logger.js';
|
||||
import {
|
||||
ServerClassifiedProviderError,
|
||||
classifyHttpProviderError,
|
||||
} from './shared/error-classification.js';
|
||||
import { buildServerGenerationPrompt } from './shared/prompt-builder.js';
|
||||
import type {
|
||||
ServerGenerationContext,
|
||||
ServerGenerationProvider,
|
||||
ServerGenerationResult,
|
||||
} from './shared/types.js';
|
||||
|
||||
const OPENROUTER_API_URL = 'https://openrouter.ai/api/v1/chat/completions';
|
||||
const DEFAULT_MODEL = 'anthropic/claude-3.5-sonnet';
|
||||
|
||||
export interface OpenRouterObservationProviderOptions {
|
||||
apiKey: string;
|
||||
model?: string;
|
||||
maxOutputTokens?: number;
|
||||
siteUrl?: string;
|
||||
appName?: string;
|
||||
fetchImpl?: typeof fetch;
|
||||
}
|
||||
|
||||
interface OpenRouterResponse {
|
||||
choices?: Array<{ message?: { content?: string } }>;
|
||||
usage?: { total_tokens?: number };
|
||||
error?: { code?: string | number; message?: string };
|
||||
}
|
||||
|
||||
export class OpenRouterObservationProvider implements ServerGenerationProvider {
|
||||
readonly providerLabel = 'openrouter' as const;
|
||||
private readonly apiKey: string;
|
||||
private readonly model: string;
|
||||
private readonly maxOutputTokens: number;
|
||||
private readonly siteUrl: string;
|
||||
private readonly appName: string;
|
||||
private readonly fetchImpl: typeof fetch;
|
||||
|
||||
constructor(options: OpenRouterObservationProviderOptions) {
|
||||
if (!options.apiKey) {
|
||||
throw new ServerClassifiedProviderError('OpenRouter API key not configured', {
|
||||
kind: 'auth_invalid',
|
||||
cause: new Error('apiKey is required'),
|
||||
});
|
||||
}
|
||||
this.apiKey = options.apiKey;
|
||||
this.model = options.model ?? DEFAULT_MODEL;
|
||||
this.maxOutputTokens = options.maxOutputTokens ?? 4096;
|
||||
this.siteUrl = options.siteUrl ?? 'https://github.com/thedotmack/claude-mem';
|
||||
this.appName = options.appName ?? 'claude-mem';
|
||||
this.fetchImpl = options.fetchImpl ?? fetch;
|
||||
}
|
||||
|
||||
async generate(
|
||||
context: ServerGenerationContext,
|
||||
signal?: AbortSignal,
|
||||
): Promise<ServerGenerationResult> {
|
||||
const { prompt, skippedAll } = buildServerGenerationPrompt(context);
|
||||
if (skippedAll) {
|
||||
return {
|
||||
rawText: '<skip_summary reason="all_events_private" />',
|
||||
providerLabel: this.providerLabel,
|
||||
modelId: this.model,
|
||||
};
|
||||
}
|
||||
|
||||
let response: Response;
|
||||
try {
|
||||
response = await this.fetchImpl(OPENROUTER_API_URL, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
Authorization: `Bearer ${this.apiKey}`,
|
||||
'HTTP-Referer': this.siteUrl,
|
||||
'X-Title': this.appName,
|
||||
'Content-Type': 'application/json',
|
||||
},
|
||||
body: JSON.stringify({
|
||||
model: this.model,
|
||||
messages: [{ role: 'user', content: prompt }],
|
||||
temperature: 0.3,
|
||||
max_tokens: this.maxOutputTokens,
|
||||
}),
|
||||
signal,
|
||||
});
|
||||
} catch (networkError) {
|
||||
throw classifyHttpProviderError({
|
||||
cause: networkError,
|
||||
providerLabel: 'OpenRouter',
|
||||
});
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
const bodyText = await safeReadBody(response);
|
||||
throw classifyHttpProviderError({
|
||||
status: response.status,
|
||||
bodyText,
|
||||
headers: response.headers,
|
||||
cause: new Error(`OpenRouter API error: ${response.status} - ${bodyText}`),
|
||||
providerLabel: 'OpenRouter',
|
||||
});
|
||||
}
|
||||
|
||||
let data: OpenRouterResponse;
|
||||
try {
|
||||
data = (await response.json()) as OpenRouterResponse;
|
||||
} catch (parseError) {
|
||||
throw new ServerClassifiedProviderError('OpenRouter returned invalid JSON', {
|
||||
kind: 'parse_error',
|
||||
cause: parseError,
|
||||
});
|
||||
}
|
||||
|
||||
if (data.error) {
|
||||
throw classifyHttpProviderError({
|
||||
status: response.status,
|
||||
bodyText: `${data.error.code ?? ''} ${data.error.message ?? ''}`,
|
||||
headers: response.headers,
|
||||
cause: new Error(`OpenRouter API error: ${data.error.code} - ${data.error.message}`),
|
||||
providerLabel: 'OpenRouter',
|
||||
});
|
||||
}
|
||||
|
||||
const rawText = data.choices?.[0]?.message?.content?.trim() ?? '';
|
||||
if (!rawText) {
|
||||
logger.warn('SDK', 'OpenRouter returned empty content', {
|
||||
provider: 'openrouter',
|
||||
model: this.model,
|
||||
});
|
||||
}
|
||||
|
||||
const tokensUsed = typeof data.usage?.total_tokens === 'number' ? data.usage.total_tokens : undefined;
|
||||
|
||||
return {
|
||||
rawText,
|
||||
...(tokensUsed !== undefined ? { tokensUsed } : {}),
|
||||
providerLabel: this.providerLabel,
|
||||
modelId: this.model,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
async function safeReadBody(response: Response): Promise<string> {
|
||||
try {
|
||||
return await response.text();
|
||||
} catch {
|
||||
return '';
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,136 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
// Server-beta-local copy of the worker provider error classification model.
|
||||
// Phase 5 anti-pattern guard: src/server/* must not import from
|
||||
// src/services/worker/*, so we duplicate the small, stable error model here.
|
||||
// Worker code keeps src/services/worker/provider-errors.ts unchanged.
|
||||
|
||||
export type ServerProviderErrorClass =
|
||||
| 'transient'
|
||||
| 'unrecoverable'
|
||||
| 'rate_limit'
|
||||
| 'quota_exhausted'
|
||||
| 'auth_invalid'
|
||||
| 'parse_error'
|
||||
| (string & {});
|
||||
|
||||
export class ServerClassifiedProviderError extends Error {
|
||||
readonly kind: ServerProviderErrorClass;
|
||||
readonly retryAfterMs?: number;
|
||||
readonly cause: unknown;
|
||||
|
||||
constructor(
|
||||
message: string,
|
||||
opts: {
|
||||
kind: ServerProviderErrorClass;
|
||||
cause: unknown;
|
||||
retryAfterMs?: number;
|
||||
},
|
||||
) {
|
||||
super(message);
|
||||
this.name = 'ServerClassifiedProviderError';
|
||||
this.kind = opts.kind;
|
||||
this.cause = opts.cause;
|
||||
if (opts.retryAfterMs !== undefined) {
|
||||
this.retryAfterMs = opts.retryAfterMs;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export function isServerClassified(err: unknown): err is ServerClassifiedProviderError {
|
||||
return err instanceof ServerClassifiedProviderError;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse Retry-After header (seconds or HTTP-date). Returns ms or undefined.
|
||||
* Behavior intentionally mirrors the worker providers' helper so server
|
||||
* retries match worker retry policy.
|
||||
*/
|
||||
export function parseRetryAfterMs(value: string | null): number | undefined {
|
||||
if (!value) return undefined;
|
||||
const seconds = Number(value);
|
||||
if (!Number.isNaN(seconds) && seconds >= 0) {
|
||||
return Math.floor(seconds * 1000);
|
||||
}
|
||||
const dateMs = Date.parse(value);
|
||||
if (!Number.isNaN(dateMs)) {
|
||||
const delta = dateMs - Date.now();
|
||||
return delta > 0 ? delta : 0;
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
interface ClassifyHttpInput {
|
||||
status?: number;
|
||||
bodyText?: string;
|
||||
headers?: Headers | { get(name: string): string | null };
|
||||
cause: unknown;
|
||||
providerLabel: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Generic HTTP-error → ServerClassifiedProviderError mapping shared by
|
||||
* Gemini and OpenRouter server adapters. Provider-specific overrides (e.g.
|
||||
* Anthropic OverloadedError, Gemini quota body markers) are layered on top
|
||||
* by the per-provider classifier wrappers in this module.
|
||||
*/
|
||||
export function classifyHttpProviderError(input: ClassifyHttpInput): ServerClassifiedProviderError {
|
||||
const { status, providerLabel } = input;
|
||||
const body = input.bodyText ?? '';
|
||||
const lower = body.toLowerCase();
|
||||
const retryAfterMs = input.headers ? parseRetryAfterMs(input.headers.get('retry-after')) : undefined;
|
||||
|
||||
if (
|
||||
lower.includes('quota exceeded') ||
|
||||
lower.includes('insufficient credits') ||
|
||||
lower.includes('insufficient_quota') ||
|
||||
lower.includes('resource_exhausted')
|
||||
) {
|
||||
return new ServerClassifiedProviderError(
|
||||
`${providerLabel} quota exhausted${status !== undefined ? ` (status ${status})` : ''}`,
|
||||
{ kind: 'quota_exhausted', cause: input.cause },
|
||||
);
|
||||
}
|
||||
|
||||
if (status === 429) {
|
||||
return new ServerClassifiedProviderError(`${providerLabel} rate limit (429)`, {
|
||||
kind: 'rate_limit',
|
||||
cause: input.cause,
|
||||
...(retryAfterMs !== undefined ? { retryAfterMs } : {}),
|
||||
});
|
||||
}
|
||||
|
||||
if (status === 401 || status === 403) {
|
||||
return new ServerClassifiedProviderError(`${providerLabel} auth error (status ${status})`, {
|
||||
kind: 'auth_invalid',
|
||||
cause: input.cause,
|
||||
});
|
||||
}
|
||||
|
||||
if (status === 400 || status === 404) {
|
||||
return new ServerClassifiedProviderError(`${providerLabel} bad request (status ${status})`, {
|
||||
kind: 'unrecoverable',
|
||||
cause: input.cause,
|
||||
});
|
||||
}
|
||||
|
||||
if (status !== undefined && status >= 500 && status < 600) {
|
||||
return new ServerClassifiedProviderError(`${providerLabel} upstream error (status ${status})`, {
|
||||
kind: 'transient',
|
||||
cause: input.cause,
|
||||
});
|
||||
}
|
||||
|
||||
if (status === undefined) {
|
||||
const message = input.cause instanceof Error ? input.cause.message : String(input.cause);
|
||||
return new ServerClassifiedProviderError(`${providerLabel} network error: ${message}`, {
|
||||
kind: 'transient',
|
||||
cause: input.cause,
|
||||
});
|
||||
}
|
||||
|
||||
return new ServerClassifiedProviderError(
|
||||
`${providerLabel} API error: ${status}${body ? ` - ${body.substring(0, 200)}` : ''}`,
|
||||
{ kind: 'unrecoverable', cause: input.cause },
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,164 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import { ModeManager } from '../../../../services/domain/ModeManager.js';
|
||||
import type { ModeConfig, ObservationType } from '../../../../services/domain/types.js';
|
||||
import { stripTags } from '../../../../utils/tag-stripping.js';
|
||||
import type { PostgresAgentEvent } from '../../../../storage/postgres/agent-events.js';
|
||||
import type { ServerGenerationContext } from './types.js';
|
||||
|
||||
// Fallback list mirrors the default observation types used by claude-mem
|
||||
// modes. The server-beta prompt does not strictly need a loaded mode file —
|
||||
// the parser accepts any of these as the <type> value — so when no mode is
|
||||
// loaded (tests, fresh installs) we synthesize a minimal type list rather
|
||||
// than throwing.
|
||||
const FALLBACK_OBSERVATION_TYPES: ReadonlyArray<Pick<ObservationType, 'id'>> = [
|
||||
{ id: 'discovery' },
|
||||
{ id: 'progress' },
|
||||
{ id: 'blocker' },
|
||||
{ id: 'decision' },
|
||||
];
|
||||
|
||||
// Build a single-shot generation prompt from a list of AgentEvent records
|
||||
// plus project/session metadata. Output: a user prompt asking the provider
|
||||
// to return one or more <observation> XML blocks (or an empty response if
|
||||
// the batch should be skipped). This is intentionally a single-turn request
|
||||
// — server-beta does NOT use the worker's multi-turn SDK conversation
|
||||
// model. parseAgentXml(...) accepts the response unchanged.
|
||||
//
|
||||
// Privacy: every event payload field passes through `stripTags` (which
|
||||
// removes <private>, <claude-mem-context>, <system-reminder>, etc.) before
|
||||
// being included in the prompt. Privacy enforcement here is belt-and-suspenders
|
||||
// — `processGeneratedResponse` also discards observations that are entirely
|
||||
// derived from privately-tagged inputs.
|
||||
|
||||
export interface BuildServerPromptResult {
|
||||
readonly prompt: string;
|
||||
readonly hadPrivateContent: boolean;
|
||||
readonly skippedAll: boolean;
|
||||
}
|
||||
|
||||
const MAX_PAYLOAD_CHARS = 16 * 1024;
|
||||
|
||||
export function buildServerGenerationPrompt(
|
||||
context: ServerGenerationContext,
|
||||
options: { mode?: ModeConfig } = {},
|
||||
): BuildServerPromptResult {
|
||||
const mode = options.mode ?? loadActiveModeOrFallback();
|
||||
|
||||
let hadPrivateContent = false;
|
||||
let allEventsScrubbedToEmpty = true;
|
||||
const eventBlocks: string[] = [];
|
||||
|
||||
for (const event of context.events) {
|
||||
const block = buildEventBlock(event);
|
||||
if (block.hadPrivate) {
|
||||
hadPrivateContent = true;
|
||||
}
|
||||
if (block.body.length > 0) {
|
||||
allEventsScrubbedToEmpty = false;
|
||||
eventBlocks.push(block.body);
|
||||
}
|
||||
}
|
||||
|
||||
const skippedAll = context.events.length > 0 && allEventsScrubbedToEmpty;
|
||||
|
||||
const sessionTag = context.project.serverSessionId
|
||||
? `\n <server_session_id>${escapeXml(context.project.serverSessionId)}</server_session_id>`
|
||||
: '';
|
||||
const projectTag = context.project.projectName
|
||||
? `\n <project_name>${escapeXml(context.project.projectName)}</project_name>`
|
||||
: '';
|
||||
|
||||
const observationOutputSchema = buildObservationOutputSchema(mode);
|
||||
|
||||
const prompt = [
|
||||
'<server_beta_observation_request>',
|
||||
` <project_id>${escapeXml(context.project.projectId)}</project_id>`,
|
||||
` <team_id>${escapeXml(context.project.teamId)}</team_id>` + sessionTag + projectTag,
|
||||
` <generation_job_id>${escapeXml(context.job.id)}</generation_job_id>`,
|
||||
' <agent_events>',
|
||||
eventBlocks.length > 0 ? eventBlocks.join('\n') : ' <!-- empty after privacy stripping -->',
|
||||
' </agent_events>',
|
||||
'</server_beta_observation_request>',
|
||||
'',
|
||||
'You are observing an agent at work. Return one or more',
|
||||
'<observation>...</observation> XML blocks summarizing durable, useful',
|
||||
'discoveries from the events above. If the events contain nothing worth',
|
||||
'recording (e.g., everything was scrubbed by privacy filters or the',
|
||||
'activity was trivial), return a single self-closing <skip_summary />',
|
||||
'tag and nothing else. Do not include any prose outside the XML.',
|
||||
'',
|
||||
'Schema for each <observation> block:',
|
||||
observationOutputSchema,
|
||||
].join('\n');
|
||||
|
||||
return { prompt, hadPrivateContent, skippedAll };
|
||||
}
|
||||
|
||||
interface EventBlockResult {
|
||||
body: string;
|
||||
hadPrivate: boolean;
|
||||
}
|
||||
|
||||
function buildEventBlock(event: PostgresAgentEvent): EventBlockResult {
|
||||
const rawPayload =
|
||||
typeof event.payload === 'string' ? event.payload : JSON.stringify(event.payload ?? {}, null, 2);
|
||||
|
||||
const stripResult = stripTags(rawPayload);
|
||||
const hadPrivate = (stripResult.counts.private ?? 0) > 0;
|
||||
const truncatedPayload = stripResult.stripped.length > MAX_PAYLOAD_CHARS
|
||||
? stripResult.stripped.slice(0, MAX_PAYLOAD_CHARS) + '\n[...truncated]'
|
||||
: stripResult.stripped;
|
||||
|
||||
if (truncatedPayload.trim().length === 0) {
|
||||
return { body: '', hadPrivate };
|
||||
}
|
||||
|
||||
return {
|
||||
body: [
|
||||
' <agent_event>',
|
||||
` <id>${escapeXml(event.id)}</id>`,
|
||||
` <event_type>${escapeXml(event.eventType)}</event_type>`,
|
||||
` <source_adapter>${escapeXml(event.sourceAdapter)}</source_adapter>`,
|
||||
` <occurred_at>${new Date(event.occurredAtEpoch).toISOString()}</occurred_at>`,
|
||||
' <payload>',
|
||||
escapeXml(truncatedPayload),
|
||||
' </payload>',
|
||||
' </agent_event>',
|
||||
].join('\n'),
|
||||
hadPrivate,
|
||||
};
|
||||
}
|
||||
|
||||
function loadActiveModeOrFallback(): ModeConfig | { observation_types: ReadonlyArray<Pick<ObservationType, 'id'>> } {
|
||||
try {
|
||||
return ModeManager.getInstance().getActiveMode();
|
||||
} catch {
|
||||
return { observation_types: FALLBACK_OBSERVATION_TYPES } as unknown as ModeConfig;
|
||||
}
|
||||
}
|
||||
|
||||
function buildObservationOutputSchema(mode: ModeConfig | { observation_types: ReadonlyArray<Pick<ObservationType, 'id'>> }): string {
|
||||
const types = mode.observation_types.map(t => t.id).join(' | ');
|
||||
return [
|
||||
'<observation>',
|
||||
` <type>[ ${types} ]</type>`,
|
||||
' <title>...</title>',
|
||||
' <subtitle>...</subtitle>',
|
||||
' <facts><fact>...</fact></facts>',
|
||||
' <narrative>...</narrative>',
|
||||
' <concepts><concept>...</concept></concepts>',
|
||||
' <files_read><file>...</file></files_read>',
|
||||
' <files_modified><file>...</file></files_modified>',
|
||||
'</observation>',
|
||||
].join('\n');
|
||||
}
|
||||
|
||||
function escapeXml(text: string): string {
|
||||
return text
|
||||
.replace(/&/g, '&')
|
||||
.replace(/</g, '<')
|
||||
.replace(/>/g, '>')
|
||||
.replace(/"/g, '"')
|
||||
.replace(/'/g, ''');
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import type { PostgresAgentEvent } from '../../../../storage/postgres/agent-events.js';
|
||||
import type { PostgresObservationGenerationJob } from '../../../../storage/postgres/generation-jobs.js';
|
||||
|
||||
// ServerGenerationContext is the input handed to a server provider adapter.
|
||||
// It is reloaded from Postgres on every retry; BullMQ payload is advisory.
|
||||
// Anti-pattern guard: this MUST NOT carry worker session state.
|
||||
export interface ServerGenerationContext {
|
||||
readonly job: PostgresObservationGenerationJob;
|
||||
readonly events: readonly PostgresAgentEvent[];
|
||||
readonly project: {
|
||||
readonly projectId: string;
|
||||
readonly teamId: string;
|
||||
readonly serverSessionId: string | null;
|
||||
readonly projectName?: string | null;
|
||||
};
|
||||
}
|
||||
|
||||
// ServerGenerationResult is the raw provider response (XML accepted by
|
||||
// parseAgentXml). Empty string means provider returned nothing — handled
|
||||
// upstream as a "skip with no observation" outcome by processGeneratedResponse.
|
||||
export interface ServerGenerationResult {
|
||||
readonly rawText: string;
|
||||
readonly tokensUsed?: number;
|
||||
readonly providerLabel: string;
|
||||
readonly modelId?: string;
|
||||
}
|
||||
|
||||
export interface ServerGenerationProvider {
|
||||
readonly providerLabel: 'claude' | 'gemini' | 'openrouter';
|
||||
generate(context: ServerGenerationContext, signal?: AbortSignal): Promise<ServerGenerationResult>;
|
||||
}
|
||||
@@ -2,10 +2,12 @@
|
||||
|
||||
import {
|
||||
Queue,
|
||||
QueueEvents,
|
||||
Worker,
|
||||
type Job,
|
||||
type JobsOptions,
|
||||
type Processor,
|
||||
type QueueEventsOptions,
|
||||
type QueueOptions,
|
||||
type WorkerOptions
|
||||
} from 'bullmq';
|
||||
@@ -33,6 +35,22 @@ export interface ServerJobCounts {
|
||||
completed: number;
|
||||
}
|
||||
|
||||
// Phase 12 — runtime stalled counter. BullMQ doesn't expose a stalled counter
|
||||
// from getJobCounts (the underlying list is rotated on consumption). We keep
|
||||
// a per-process counter that tracks how many distinct stalled events we've
|
||||
// observed since startup. /api/health and /v1/info surface this.
|
||||
export interface ServerJobLifecycleCounters {
|
||||
stalled: number;
|
||||
errored: number;
|
||||
}
|
||||
|
||||
export interface ServerJobObservedListener {
|
||||
onCompleted?: (jobId: string, durationMs: number, returnvalue: unknown) => void;
|
||||
onFailed?: (jobId: string | undefined, attemptsMade: number, reason: string) => void;
|
||||
onStalled?: (jobId: string) => void;
|
||||
onError?: (error: unknown) => void;
|
||||
}
|
||||
|
||||
export interface ServerJobQueueOptions<TPayload> {
|
||||
name: string;
|
||||
config: RedisQueueConfig;
|
||||
@@ -63,7 +81,18 @@ export class ServerJobQueue<TPayload extends object = object> {
|
||||
private readonly workerFactory?: ServerJobQueueOptions<TPayload>['workerFactory'];
|
||||
private queue: ReturnType<NonNullable<ServerJobQueueOptions<TPayload>['queueFactory']>> | Queue<TPayload> | null = null;
|
||||
private worker: ReturnType<NonNullable<ServerJobQueueOptions<TPayload>['workerFactory']>> | Worker<TPayload> | null = null;
|
||||
private queueEvents: QueueEvents | null = null;
|
||||
private started = false;
|
||||
private readonly counters: ServerJobLifecycleCounters = { stalled: 0, errored: 0 };
|
||||
private readonly listeners: ServerJobObservedListener[] = [];
|
||||
private readonly jobStartTimes = new Map<string, number>();
|
||||
// worker.on('stalled') and the QueueEvents 'stalled' subscriber both fire
|
||||
// for the same job — BullMQ's docs explicitly recommend listening on both
|
||||
// for production reliability. To avoid double-counting and double-callback
|
||||
// we record each stalled jobId here for a short TTL and treat the second
|
||||
// signal as an idempotent no-op.
|
||||
private readonly recentlyStalled = new Map<string, NodeJS.Timeout>();
|
||||
private static readonly STALLED_DEDUPE_WINDOW_MS = 30_000;
|
||||
|
||||
constructor(options: ServerJobQueueOptions<TPayload>) {
|
||||
this.name = options.name;
|
||||
@@ -154,6 +183,53 @@ export class ServerJobQueue<TPayload extends object = object> {
|
||||
// BullMQ docs require `worker.on('error', ...)` to avoid unhandled rejections
|
||||
// when a job throws. We construct the Worker with autorun: false so the
|
||||
// caller controls startup explicitly via run().
|
||||
//
|
||||
// Phase 12 — wire `completed`, `failed`, `progress`, `error`, and the
|
||||
// QueueEvents `stalled` listener. Stalled events go through QueueEvents
|
||||
// because BullMQ's docs note rare stalls don't always reach the local
|
||||
// worker.on('stalled') listener; QueueEvents publishes from Redis.
|
||||
// Deduped stalled handler. Counts the stall once even though BullMQ may
|
||||
// surface it via both worker.on('stalled') and QueueEvents 'stalled'.
|
||||
private notifyStalled(jobId: string, source: 'worker' | 'queue-events'): void {
|
||||
if (this.recentlyStalled.has(jobId)) {
|
||||
logger.debug?.('QUEUE', `[generation] job=${jobId} stalled (suppressed duplicate from ${source})`, {
|
||||
queue: this.name,
|
||||
jobId,
|
||||
source,
|
||||
});
|
||||
return;
|
||||
}
|
||||
const timer = setTimeout(() => {
|
||||
this.recentlyStalled.delete(jobId);
|
||||
}, ServerJobQueue.STALLED_DEDUPE_WINDOW_MS);
|
||||
if (typeof (timer as { unref?: () => void }).unref === 'function') {
|
||||
(timer as { unref: () => void }).unref();
|
||||
}
|
||||
this.recentlyStalled.set(jobId, timer);
|
||||
this.counters.stalled += 1;
|
||||
logger.warn('QUEUE', `[generation] job=${jobId} stalled${source === 'queue-events' ? ' (queue-events)' : ''}`, {
|
||||
queue: this.name,
|
||||
jobId,
|
||||
source,
|
||||
});
|
||||
for (const l of this.listeners) {
|
||||
try { l.onStalled?.(jobId); } catch { /* listener errors must not propagate */ }
|
||||
}
|
||||
}
|
||||
|
||||
// Single source of truth for queue-side error accounting. worker errors and
|
||||
// QueueEvents errors both increment counters.errored and notify listeners,
|
||||
// so per-process metrics aren't asymmetric across the two sources.
|
||||
private notifyQueueError(error: unknown, source: 'worker' | 'queue-events'): void {
|
||||
this.counters.errored += 1;
|
||||
logger.warn('QUEUE', `${this.name} ${source} error`, {
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
});
|
||||
for (const l of this.listeners) {
|
||||
try { l.onError?.(error); } catch { /* listener errors must not propagate */ }
|
||||
}
|
||||
}
|
||||
|
||||
start(processor: Processor<TPayload>): void {
|
||||
if (this.started) {
|
||||
throw new Error(`ServerJobQueue ${this.name} is already started`);
|
||||
@@ -168,22 +244,115 @@ export class ServerJobQueue<TPayload extends object = object> {
|
||||
const worker = this.workerFactory
|
||||
? this.workerFactory(this.name, processor, workerOptions)
|
||||
: new Worker<TPayload>(this.name, processor, workerOptions);
|
||||
worker.on('error', (error: unknown) => {
|
||||
logger.warn('QUEUE', `${this.name} worker error`, {
|
||||
error: error instanceof Error ? error.message : String(error)
|
||||
worker.on('error', (error: unknown) => this.notifyQueueError(error, 'worker'));
|
||||
// BullMQ Worker exposes `active`, `completed`, `failed`, `progress`, and
|
||||
// `stalled` events. We attach to all five because the runtime relies on
|
||||
// them for observability (Phase 12).
|
||||
if (typeof (worker as { on?: unknown }).on === 'function') {
|
||||
const w = worker as Worker<TPayload>;
|
||||
w.on('active', (job: Job<TPayload>) => {
|
||||
if (job.id) this.jobStartTimes.set(job.id, Date.now());
|
||||
});
|
||||
});
|
||||
w.on('completed', (job: Job<TPayload>, returnvalue: unknown) => {
|
||||
const startedAt = job.id ? this.jobStartTimes.get(job.id) : undefined;
|
||||
const durationMs = startedAt ? Date.now() - startedAt : 0;
|
||||
if (job.id) this.jobStartTimes.delete(job.id);
|
||||
const sourceType = (job.data as { source_type?: string } | undefined)?.source_type ?? '?';
|
||||
logger.info('QUEUE', `[generation] job=${job.id ?? '?'} source_type=${sourceType} duration=${durationMs}ms`, {
|
||||
queue: this.name,
|
||||
jobId: job.id ?? null,
|
||||
sourceType,
|
||||
durationMs,
|
||||
});
|
||||
for (const l of this.listeners) {
|
||||
try { l.onCompleted?.(job.id ?? '?', durationMs, returnvalue); } catch { /* swallow listener errors only */ }
|
||||
}
|
||||
});
|
||||
w.on('failed', (job: Job<TPayload> | undefined, error: Error) => {
|
||||
if (job?.id) this.jobStartTimes.delete(job.id);
|
||||
const sourceType = (job?.data as { source_type?: string } | undefined)?.source_type ?? '?';
|
||||
const attemptsMade = job?.attemptsMade ?? 0;
|
||||
logger.warn('QUEUE', `[generation] job=${job?.id ?? '?'} source_type=${sourceType} attempts=${attemptsMade} reason=${error.message}`, {
|
||||
queue: this.name,
|
||||
jobId: job?.id ?? null,
|
||||
sourceType,
|
||||
attemptsMade,
|
||||
reason: error.message,
|
||||
});
|
||||
for (const l of this.listeners) {
|
||||
try { l.onFailed?.(job?.id, attemptsMade, error.message); } catch { /* swallow */ }
|
||||
}
|
||||
});
|
||||
w.on('progress', (job: Job<TPayload>, progress: unknown) => {
|
||||
logger.debug?.('QUEUE', `[generation] job=${job.id ?? '?'} progress`, {
|
||||
queue: this.name,
|
||||
jobId: job.id ?? null,
|
||||
progress,
|
||||
});
|
||||
});
|
||||
w.on('stalled', (jobId: string) => this.notifyStalled(jobId, 'worker'));
|
||||
}
|
||||
worker.run();
|
||||
this.worker = worker;
|
||||
|
||||
// QueueEvents subscribes to Redis pub/sub for cross-process events
|
||||
// (BullMQ "Stalled Jobs" docs recommend this for production reliability).
|
||||
// Skip in test/factory mode since the test factory does not provide a
|
||||
// real Redis connection.
|
||||
if (!this.workerFactory) {
|
||||
try {
|
||||
const events = new QueueEvents(this.name, {
|
||||
connection: this.config.connection,
|
||||
prefix: this.config.prefix,
|
||||
} as QueueEventsOptions);
|
||||
events.on('stalled', ({ jobId }: { jobId: string }) => this.notifyStalled(jobId, 'queue-events'));
|
||||
// QueueEvents emits its own 'error' too — surface through the same
|
||||
// counter+listener path as worker errors so observability stays symmetric.
|
||||
events.on('error', (error: Error) => this.notifyQueueError(error, 'queue-events'));
|
||||
this.queueEvents = events;
|
||||
} catch (error) {
|
||||
logger.warn('QUEUE', `${this.name} failed to start QueueEvents listener`, {
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
this.started = true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Phase 12 — register an observer for completed/failed/stalled/error
|
||||
* events. Used by the runtime to surface lifecycle hooks (audit, metrics)
|
||||
* without subclassing. Listeners that throw are isolated.
|
||||
*/
|
||||
observe(listener: ServerJobObservedListener): void {
|
||||
this.listeners.push(listener);
|
||||
}
|
||||
|
||||
/**
|
||||
* Phase 12 — runtime counters for stalled/errored events. waiting/active/
|
||||
* completed/failed/delayed live in `getCounts()` (BullMQ getJobCounts).
|
||||
* Stalled is a per-process counter because BullMQ rotates the underlying
|
||||
* list and there's no reliable count from getJobCounts.
|
||||
*/
|
||||
getLifecycleCounters(): ServerJobLifecycleCounters {
|
||||
return { ...this.counters };
|
||||
}
|
||||
|
||||
isStarted(): boolean {
|
||||
return this.started;
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
const errors: Error[] = [];
|
||||
if (this.queueEvents) {
|
||||
try {
|
||||
await this.queueEvents.close();
|
||||
} catch (error) {
|
||||
errors.push(error instanceof Error ? error : new Error(String(error)));
|
||||
}
|
||||
this.queueEvents = null;
|
||||
}
|
||||
if (this.worker) {
|
||||
try {
|
||||
await this.worker.close();
|
||||
@@ -201,6 +370,10 @@ export class ServerJobQueue<TPayload extends object = object> {
|
||||
}
|
||||
this.queue = null;
|
||||
}
|
||||
for (const timer of this.recentlyStalled.values()) {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
this.recentlyStalled.clear();
|
||||
if (errors.length > 0) {
|
||||
throw errors[0];
|
||||
}
|
||||
|
||||
@@ -9,11 +9,12 @@ import type { JsonObject } from '../../storage/postgres/utils.js';
|
||||
import { logger } from '../../utils/logger.js';
|
||||
import { buildServerJobId } from './job-id.js';
|
||||
import type { ServerJobQueue } from './ServerJobQueue.js';
|
||||
import type {
|
||||
GenerateObservationsForEventJob,
|
||||
GenerateSessionSummaryJob,
|
||||
ReindexObservationJob,
|
||||
ServerGenerationJobKind
|
||||
import {
|
||||
assertServerGenerationJobPayload,
|
||||
type GenerateObservationsForEventJob,
|
||||
type GenerateSessionSummaryJob,
|
||||
type ReindexObservationJob,
|
||||
type ServerGenerationJobKind,
|
||||
} from './types.js';
|
||||
|
||||
// Postgres outbox is canonical history; BullMQ is the execution transport.
|
||||
@@ -86,6 +87,10 @@ export async function enqueueOutbox(
|
||||
});
|
||||
|
||||
try {
|
||||
// Phase 11 — defense in depth. Validate the payload shape at the queue
|
||||
// boundary so a malformed enqueue is rejected synchronously and never
|
||||
// produces a job whose audit trail is missing fields.
|
||||
assertServerGenerationJobPayload(payload);
|
||||
await queue.add(bullmqJobId, payload);
|
||||
await eventsRepo.append({
|
||||
generationJobId: row.id,
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import { z } from 'zod';
|
||||
import type {
|
||||
ObservationGenerationJobSourceType,
|
||||
ObservationGenerationJobStatus
|
||||
@@ -9,6 +10,12 @@ export type ServerGenerationJobKind = 'event' | 'event-batch' | 'summary' | 'rei
|
||||
|
||||
export type ServerGenerationJobStatus = ObservationGenerationJobStatus;
|
||||
|
||||
// Phase 11 — every BullMQ job carries the full team-aware tracing surface so
|
||||
// the worker can audit and scope-check on every retry. team_id and project_id
|
||||
// are advisory: the worker MUST reload the canonical outbox row from Postgres
|
||||
// and compare these fields before any side effect. Treating these as auth
|
||||
// authority would be a bypass — the comparison is a tampering detector, not
|
||||
// the auth gate.
|
||||
export interface ServerGenerationJob {
|
||||
kind: ServerGenerationJobKind;
|
||||
team_id: string;
|
||||
@@ -16,6 +23,18 @@ export interface ServerGenerationJob {
|
||||
source_type: ObservationGenerationJobSourceType;
|
||||
source_id: string;
|
||||
generation_job_id: string;
|
||||
// Identity of the API key that initiated this job at the HTTP boundary.
|
||||
// Reused at execution time to detect revocation between enqueue and run.
|
||||
api_key_id: string | null;
|
||||
// The actor associated with the api key at enqueue time. Audit-only;
|
||||
// never trust this for authz decisions.
|
||||
actor_id: string | null;
|
||||
// Legacy adapter or surface that produced the source row, for routing
|
||||
// and audit (e.g. 'api', 'hooks', 'mcp', 'compat:sessions-observations').
|
||||
source_adapter: string;
|
||||
// Phase 12 — request correlation id, optional but always serialized as a
|
||||
// nullable field so downstream consumers can rely on shape stability.
|
||||
request_id?: string | null;
|
||||
}
|
||||
|
||||
export interface GenerateObservationsForEventJob extends ServerGenerationJob {
|
||||
@@ -57,3 +76,80 @@ export const SERVER_JOB_KIND_PREFIX: Record<ServerGenerationJobKind, string> = {
|
||||
summary: 'sum',
|
||||
reindex: 'rdx'
|
||||
};
|
||||
|
||||
// Phase 11 — Zod schema validates payloads at the queue boundary so a
|
||||
// malformed enqueue is rejected synchronously rather than silently producing
|
||||
// a job the worker can't audit. Required fields here mirror the
|
||||
// ServerGenerationJob interface; a missing team_id, project_id, or
|
||||
// generation_job_id should always be a programmer error caught at enqueue.
|
||||
|
||||
const baseFieldsSchema = z.object({
|
||||
team_id: z.string().min(1, 'team_id is required'),
|
||||
project_id: z.string().min(1, 'project_id is required'),
|
||||
source_type: z.enum(['agent_event', 'session_summary', 'observation_reindex']),
|
||||
source_id: z.string().min(1, 'source_id is required'),
|
||||
generation_job_id: z.string().min(1, 'generation_job_id is required'),
|
||||
// api_key_id and actor_id are nullable to accommodate local-dev/system
|
||||
// enqueues, but the *field* must be present in the payload so audit
|
||||
// records always render the same shape.
|
||||
api_key_id: z.string().min(1).nullable(),
|
||||
actor_id: z.string().min(1).nullable(),
|
||||
source_adapter: z.string().min(1, 'source_adapter is required'),
|
||||
// Phase 12 — request_id is optional in the schema (older jobs predating
|
||||
// this phase have nullable/missing values) but always passes through to
|
||||
// logs and audit when present.
|
||||
request_id: z.string().min(1).nullable().optional(),
|
||||
});
|
||||
|
||||
export const GenerateObservationsForEventJobSchema = baseFieldsSchema.extend({
|
||||
kind: z.literal('event'),
|
||||
agent_event_id: z.string().min(1),
|
||||
});
|
||||
|
||||
export const GenerateObservationsForEventBatchJobSchema = baseFieldsSchema.extend({
|
||||
kind: z.literal('event-batch'),
|
||||
agent_event_ids: z.array(z.string().min(1)).min(1),
|
||||
});
|
||||
|
||||
export const GenerateSessionSummaryJobSchema = baseFieldsSchema.extend({
|
||||
kind: z.literal('summary'),
|
||||
server_session_id: z.string().min(1),
|
||||
});
|
||||
|
||||
export const ReindexObservationJobSchema = baseFieldsSchema.extend({
|
||||
kind: z.literal('reindex'),
|
||||
observation_id: z.string().min(1),
|
||||
});
|
||||
|
||||
export const ServerGenerationJobPayloadSchema = z.discriminatedUnion('kind', [
|
||||
GenerateObservationsForEventJobSchema,
|
||||
GenerateObservationsForEventBatchJobSchema,
|
||||
GenerateSessionSummaryJobSchema,
|
||||
ReindexObservationJobSchema,
|
||||
]);
|
||||
|
||||
export class ServerGenerationJobPayloadValidationError extends Error {
|
||||
readonly issues: z.ZodIssue[];
|
||||
|
||||
constructor(issues: z.ZodIssue[]) {
|
||||
super(`invalid server generation job payload: ${issues.map(i => i.message).join('; ')}`);
|
||||
this.issues = issues;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate a candidate BullMQ payload against the discriminated union and
|
||||
* return a typed payload, or throw `ServerGenerationJobPayloadValidationError`.
|
||||
* Use this at every enqueue site so a malformed payload never enters the
|
||||
* transport — the worker MUST also re-validate from Postgres but defense in
|
||||
* depth is cheap.
|
||||
*/
|
||||
export function assertServerGenerationJobPayload(
|
||||
candidate: unknown,
|
||||
): ServerGenerationJobPayload {
|
||||
const result = ServerGenerationJobPayloadSchema.safeParse(candidate);
|
||||
if (!result.success) {
|
||||
throw new ServerGenerationJobPayloadValidationError(result.error.issues);
|
||||
}
|
||||
return result.data as ServerGenerationJobPayload;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,199 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import { createHash } from 'crypto';
|
||||
import type { NextFunction, Request, RequestHandler, Response } from 'express';
|
||||
import type { PostgresPool } from '../../storage/postgres/pool.js';
|
||||
import type { PostgresApiKey } from '../../storage/postgres/auth.js';
|
||||
import type { AuthContext } from './auth.js';
|
||||
|
||||
// Postgres-backed auth middleware for the server-beta runtime.
|
||||
//
|
||||
// Mirrors src/server/middleware/auth.ts but reads API keys from the Postgres
|
||||
// `api_keys` table instead of bun:sqlite. Phase 4 routes use this so the
|
||||
// runtime depends only on the Postgres pool and Postgres-backed repositories.
|
||||
//
|
||||
// teamId / projectId on req.authContext come straight from the Postgres
|
||||
// api_keys row. Routes use those to scope every read and write.
|
||||
|
||||
export interface PostgresRequireAuthOptions {
|
||||
requiredScopes?: string[];
|
||||
authMode?: string;
|
||||
allowLocalDevBypass?: boolean;
|
||||
// Local-dev fallback team for unauthenticated loopback requests. This is
|
||||
// only used when authMode === 'local-dev' AND allowLocalDevBypass is true
|
||||
// AND the request is on loopback. It must NEVER be used to scope a real
|
||||
// production request.
|
||||
localDevTeamId?: string | null;
|
||||
}
|
||||
|
||||
export function requirePostgresServerAuth(
|
||||
pool: PostgresPool,
|
||||
options: PostgresRequireAuthOptions = {},
|
||||
): RequestHandler {
|
||||
return async (req: Request, res: Response, next: NextFunction) => {
|
||||
try {
|
||||
const authMode = options.authMode ?? process.env.CLAUDE_MEM_AUTH_MODE ?? 'api-key';
|
||||
const authorization = req.header('authorization') ?? '';
|
||||
const rawKey = parseBearerToken(authorization);
|
||||
|
||||
const allowLocalDevBypass = options.allowLocalDevBypass
|
||||
?? process.env.CLAUDE_MEM_ALLOW_LOCAL_DEV_BYPASS === '1';
|
||||
if (
|
||||
!rawKey
|
||||
&& authMode === 'local-dev'
|
||||
&& allowLocalDevBypass
|
||||
&& isLocalhost(req)
|
||||
&& hasLoopbackHostHeader(req)
|
||||
&& !hasForwardedClientHeaders(req)
|
||||
) {
|
||||
const ctx: AuthContext = {
|
||||
userId: null,
|
||||
organizationId: null,
|
||||
teamId: options.localDevTeamId ?? null,
|
||||
projectId: null,
|
||||
scopes: ['local-dev'],
|
||||
apiKeyId: null,
|
||||
mode: 'local-dev',
|
||||
};
|
||||
req.authContext = ctx;
|
||||
next();
|
||||
return;
|
||||
}
|
||||
|
||||
if (!rawKey) {
|
||||
res.status(401).json({ error: 'Unauthorized', message: 'Missing bearer API key' });
|
||||
return;
|
||||
}
|
||||
|
||||
const verified = await verifyPostgresApiKey(pool, rawKey, options.requiredScopes ?? []);
|
||||
if (!verified) {
|
||||
res.status(403).json({ error: 'Forbidden', message: 'Invalid API key or insufficient scope' });
|
||||
return;
|
||||
}
|
||||
|
||||
const ctx: AuthContext = {
|
||||
userId: null,
|
||||
organizationId: null,
|
||||
teamId: verified.teamId,
|
||||
projectId: verified.projectId,
|
||||
scopes: verified.scopes,
|
||||
apiKeyId: verified.apiKeyId,
|
||||
mode: 'api-key',
|
||||
};
|
||||
req.authContext = ctx;
|
||||
next();
|
||||
} catch (error) {
|
||||
next(error);
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
interface VerifiedPostgresApiKey {
|
||||
apiKeyId: string;
|
||||
teamId: string | null;
|
||||
projectId: string | null;
|
||||
scopes: string[];
|
||||
}
|
||||
|
||||
export async function verifyPostgresApiKey(
|
||||
pool: PostgresPool,
|
||||
rawKey: string,
|
||||
requiredScopes: string[],
|
||||
): Promise<VerifiedPostgresApiKey | null> {
|
||||
const keyHash = createHash('sha256').update(rawKey).digest('hex');
|
||||
const result = await pool.query(
|
||||
`
|
||||
SELECT id, team_id, project_id, scopes, revoked_at, expires_at
|
||||
FROM api_keys
|
||||
WHERE key_hash = $1
|
||||
`,
|
||||
[keyHash],
|
||||
);
|
||||
const row = result.rows[0] as Pick<
|
||||
PostgresApiKey,
|
||||
'id' | 'teamId' | 'projectId'
|
||||
> & {
|
||||
id: string;
|
||||
team_id: string | null;
|
||||
project_id: string | null;
|
||||
scopes: unknown;
|
||||
revoked_at: Date | null;
|
||||
expires_at: Date | null;
|
||||
} | undefined;
|
||||
if (!row) {
|
||||
return null;
|
||||
}
|
||||
if (row.revoked_at) {
|
||||
return null;
|
||||
}
|
||||
if (row.expires_at && row.expires_at.getTime() <= Date.now()) {
|
||||
return null;
|
||||
}
|
||||
const scopes = normalizeScopes(row.scopes);
|
||||
if (!hasRequiredScopes(scopes, requiredScopes)) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
apiKeyId: row.id,
|
||||
teamId: row.team_id,
|
||||
projectId: row.project_id,
|
||||
scopes,
|
||||
};
|
||||
}
|
||||
|
||||
function normalizeScopes(value: unknown): string[] {
|
||||
if (!Array.isArray(value)) {
|
||||
return [];
|
||||
}
|
||||
return value.filter((item): item is string => typeof item === 'string');
|
||||
}
|
||||
|
||||
function hasRequiredScopes(grantedScopes: string[], requiredScopes: string[]): boolean {
|
||||
if (requiredScopes.length === 0 || grantedScopes.includes('*')) {
|
||||
return true;
|
||||
}
|
||||
return requiredScopes.every(scope => grantedScopes.includes(scope));
|
||||
}
|
||||
|
||||
function parseBearerToken(header: string): string | null {
|
||||
const match = /^Bearer\s+(.+)$/i.exec(header.trim());
|
||||
return match?.[1]?.trim() || null;
|
||||
}
|
||||
|
||||
function isLocalhost(req: Request): boolean {
|
||||
const clientIp = req.ip || req.socket.remoteAddress || '';
|
||||
return clientIp === '127.0.0.1'
|
||||
|| clientIp === '::1'
|
||||
|| clientIp === '::ffff:127.0.0.1'
|
||||
|| clientIp === 'localhost';
|
||||
}
|
||||
|
||||
function hasLoopbackHostHeader(req: Request): boolean {
|
||||
const host = parseHostWithoutPort(req.header('host') ?? '');
|
||||
return host === '127.0.0.1'
|
||||
|| host === 'localhost'
|
||||
|| host === '::1';
|
||||
}
|
||||
|
||||
function parseHostWithoutPort(rawHost: string): string {
|
||||
const host = rawHost.trim().toLowerCase();
|
||||
if (host.startsWith('[')) {
|
||||
const closeBracketIndex = host.indexOf(']');
|
||||
return closeBracketIndex === -1 ? host : host.slice(1, closeBracketIndex);
|
||||
}
|
||||
|
||||
const lastColonIndex = host.lastIndexOf(':');
|
||||
if (lastColonIndex > -1 && /^\d+$/.test(host.slice(lastColonIndex + 1))) {
|
||||
return host.slice(0, lastColonIndex);
|
||||
}
|
||||
return host;
|
||||
}
|
||||
|
||||
function hasForwardedClientHeaders(req: Request): boolean {
|
||||
return Boolean(
|
||||
req.header('forwarded')
|
||||
|| req.header('x-forwarded-for')
|
||||
|| req.header('x-forwarded-host')
|
||||
|| req.header('x-real-ip'),
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import { randomUUID } from 'crypto';
|
||||
import type { NextFunction, Request, RequestHandler, Response } from 'express';
|
||||
|
||||
// Phase 12 — request_id middleware. Mints a UUID per inbound request and
|
||||
// attaches it to req.requestId so route handlers, ingest services, and
|
||||
// generation jobs can correlate logs back to the original HTTP call. Honors
|
||||
// an inbound `X-Request-Id` header so an upstream load balancer / gateway
|
||||
// can supply the id, but rejects non-conformant values to keep audit rows
|
||||
// clean (UUID v4 OR a small whitelist of [a-zA-Z0-9-_] up to 64 chars).
|
||||
//
|
||||
// Anti-pattern guard: never trust the inbound id for auth — this is purely
|
||||
// an audit/log correlator. Auth still flows through requirePostgresServerAuth.
|
||||
|
||||
const REQUEST_ID_HEADER = 'x-request-id';
|
||||
const REQUEST_ID_MAX_LENGTH = 64;
|
||||
const REQUEST_ID_SAFE_PATTERN = /^[A-Za-z0-9][A-Za-z0-9\-_]{0,63}$/;
|
||||
|
||||
declare module 'express-serve-static-core' {
|
||||
interface Request {
|
||||
requestId?: string;
|
||||
}
|
||||
}
|
||||
|
||||
export function requestIdMiddleware(): RequestHandler {
|
||||
return (req: Request, res: Response, next: NextFunction) => {
|
||||
const inbound = req.header(REQUEST_ID_HEADER);
|
||||
const accepted = inbound && isAcceptableRequestId(inbound) ? inbound : randomUUID();
|
||||
req.requestId = accepted;
|
||||
res.setHeader('X-Request-Id', accepted);
|
||||
next();
|
||||
};
|
||||
}
|
||||
|
||||
export function isAcceptableRequestId(value: string): boolean {
|
||||
if (typeof value !== 'string') return false;
|
||||
if (value.length === 0 || value.length > REQUEST_ID_MAX_LENGTH) return false;
|
||||
return REQUEST_ID_SAFE_PATTERN.test(value);
|
||||
}
|
||||
@@ -17,6 +17,23 @@ export interface ObservationQueueEngine {
|
||||
close(): Promise<void>;
|
||||
}
|
||||
|
||||
// Phase 12 — `lanes` exposes per-queue counts (waiting/active/completed/
|
||||
// failed/delayed/stalled) so deploy probes can monitor saturation per lane.
|
||||
// `unavailable: true` means the sample failed; the health endpoint MUST NOT
|
||||
// 503 just because counts are stale.
|
||||
export interface ObservationQueueHealthLaneSnapshot {
|
||||
kind: string;
|
||||
name: string;
|
||||
waiting: number;
|
||||
active: number;
|
||||
completed: number;
|
||||
failed: number;
|
||||
delayed: number;
|
||||
stalled: number;
|
||||
unavailable: boolean;
|
||||
unavailableReason?: string;
|
||||
}
|
||||
|
||||
export interface ObservationQueueHealth {
|
||||
engine: 'bullmq';
|
||||
redis: {
|
||||
@@ -27,6 +44,7 @@ export interface ObservationQueueHealth {
|
||||
prefix: string;
|
||||
error?: string;
|
||||
};
|
||||
lanes?: ObservationQueueHealthLaneSnapshot[];
|
||||
}
|
||||
|
||||
export interface ObservationQueueInspection {
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,164 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import type { Job } from 'bullmq';
|
||||
import { logger } from '../../utils/logger.js';
|
||||
import { PostgresAuthRepository } from '../../storage/postgres/auth.js';
|
||||
import type { PostgresPool } from '../../storage/postgres/pool.js';
|
||||
import { ProviderObservationGenerator } from '../generation/ProviderObservationGenerator.js';
|
||||
import type { ServerGenerationProvider } from '../generation/providers/shared/types.js';
|
||||
import type { ServerGenerationJobPayload } from '../jobs/types.js';
|
||||
import type { ActiveServerBetaQueueManager } from './ActiveServerBetaQueueManager.js';
|
||||
import type {
|
||||
ServerBetaBoundaryHealth,
|
||||
ServerBetaGenerationWorkerManager,
|
||||
} from './types.js';
|
||||
|
||||
// ActiveServerBetaGenerationWorkerManager wires a BullMQ Worker (per the
|
||||
// 'event' queue) to a ProviderObservationGenerator. Concurrency defaults to
|
||||
// 1 per the plan (line 80–86) so retries observe a single inflight provider
|
||||
// call per server. autorun:false / explicit run() is enforced by
|
||||
// ServerJobQueue.start.
|
||||
//
|
||||
// This class is wired in only when both a queue manager AND a configured
|
||||
// provider are present. create-server-beta-service keeps the disabled
|
||||
// adapter otherwise so server beta can boot without provider credentials.
|
||||
|
||||
export interface ActiveServerBetaGenerationWorkerManagerOptions {
|
||||
pool: PostgresPool;
|
||||
queueManager: ActiveServerBetaQueueManager;
|
||||
provider: ServerGenerationProvider;
|
||||
workerId?: string;
|
||||
// Test seam: replace the generator with a stub.
|
||||
generatorFactory?: (
|
||||
pool: PostgresPool,
|
||||
provider: ServerGenerationProvider,
|
||||
workerId: string,
|
||||
) => ProviderObservationGenerator;
|
||||
}
|
||||
|
||||
export class ActiveServerBetaGenerationWorkerManager implements ServerBetaGenerationWorkerManager {
|
||||
readonly kind = 'generation-worker-manager' as const;
|
||||
private started = false;
|
||||
private closed = false;
|
||||
private readonly generator: ProviderObservationGenerator;
|
||||
private readonly workerId: string;
|
||||
|
||||
constructor(private readonly options: ActiveServerBetaGenerationWorkerManagerOptions) {
|
||||
this.workerId = options.workerId ?? `server-beta-${process.pid}`;
|
||||
this.generator = options.generatorFactory
|
||||
? options.generatorFactory(options.pool, options.provider, this.workerId)
|
||||
: new ProviderObservationGenerator({
|
||||
pool: options.pool,
|
||||
provider: options.provider,
|
||||
workerId: this.workerId,
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Attach BullMQ Worker to the 'event' queue. Per BullMQ docs we use
|
||||
* new Worker(queueName, processor, { concurrency, autorun })
|
||||
* via ServerJobQueue.start(...). Errors are surfaced through the queue
|
||||
* wrapper's worker.on('error', ...) listener.
|
||||
*/
|
||||
start(): void {
|
||||
if (this.started) return;
|
||||
const dispatcher = async (job: Job<ServerGenerationJobPayload>) => {
|
||||
try {
|
||||
return await this.generator.process(job);
|
||||
} catch (error) {
|
||||
logger.warn('SYSTEM', 'observation generator failed', {
|
||||
jobId: job.id,
|
||||
kind: job.data.kind,
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
});
|
||||
throw error;
|
||||
}
|
||||
};
|
||||
this.options.queueManager.start('event', dispatcher);
|
||||
// Phase 6: wire the summary lane alongside the event lane. Concurrency
|
||||
// defaults to 1 per ServerJobQueue config (per the plan), and the same
|
||||
// ProviderObservationGenerator dispatches on job.data.source_type via the
|
||||
// outbox row reload inside lockOutbox+process.
|
||||
this.options.queueManager.start('summary', dispatcher);
|
||||
|
||||
// Phase 12 — audit stalled events directly. Phase 11's audit chain now
|
||||
// covers the operator and provider lifecycle; stalled jobs come from
|
||||
// BullMQ runtime not the HTTP boundary, so we wire them in here. Best-
|
||||
// effort: a missing/unscoped audit MUST NOT crash the worker.
|
||||
for (const lane of ['event', 'summary'] as const) {
|
||||
try {
|
||||
const queue = this.options.queueManager.getQueue(lane);
|
||||
queue.observe({
|
||||
onStalled: (jobId) => {
|
||||
void this.auditStalledJob(jobId, lane);
|
||||
},
|
||||
});
|
||||
} catch (error) {
|
||||
logger.warn('SYSTEM', `failed to wire stalled observer for ${lane} lane`, {
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
this.started = true;
|
||||
}
|
||||
|
||||
// Phase 12 — write a `generation_job.stalled` audit row. We look up the
|
||||
// outbox row by BullMQ jobId (== bullmq_job_id column) so team/project
|
||||
// scope is correct on the audit row even when the original API key
|
||||
// metadata is unavailable (BullMQ retries can outlive a session).
|
||||
private async auditStalledJob(bullmqJobId: string, lane: 'event' | 'summary'): Promise<void> {
|
||||
try {
|
||||
const result = await this.options.pool.query<{
|
||||
id: string;
|
||||
team_id: string | null;
|
||||
project_id: string | null;
|
||||
}>(
|
||||
'SELECT id, team_id, project_id FROM observation_generation_jobs WHERE bullmq_job_id = $1 LIMIT 1',
|
||||
[bullmqJobId],
|
||||
);
|
||||
const row = result.rows[0];
|
||||
if (!row) return;
|
||||
const repo = new PostgresAuthRepository(this.options.pool);
|
||||
await repo.createAuditLog({
|
||||
teamId: row.team_id,
|
||||
projectId: row.project_id,
|
||||
actorId: null,
|
||||
apiKeyId: null,
|
||||
action: 'generation_job.stalled',
|
||||
resourceType: 'observation_generation_job',
|
||||
resourceId: row.id,
|
||||
details: { lane, bullmqJobId },
|
||||
});
|
||||
} catch (error) {
|
||||
logger.warn('SYSTEM', 'failed to audit stalled generation_job', {
|
||||
bullmqJobId,
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
getHealth(): ServerBetaBoundaryHealth {
|
||||
if (this.closed) {
|
||||
return { status: 'errored', reason: 'generation-worker-manager closed' };
|
||||
}
|
||||
return {
|
||||
status: this.started ? 'active' : 'disabled',
|
||||
reason: this.started
|
||||
? 'BullMQ Worker attached to event queue with ProviderObservationGenerator'
|
||||
: 'wired but not started',
|
||||
details: {
|
||||
provider: this.options.provider.providerLabel,
|
||||
workerId: this.workerId,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
if (this.closed) return;
|
||||
this.closed = true;
|
||||
// The underlying Worker is owned by ServerJobQueue.close() (driven by
|
||||
// the queue manager). We do not double-close here; the queue manager's
|
||||
// close cascade handles it.
|
||||
}
|
||||
}
|
||||
@@ -11,6 +11,7 @@ import type { RedisQueueConfig } from '../queue/redis-config.js';
|
||||
import { logger } from '../../utils/logger.js';
|
||||
import type {
|
||||
ServerBetaBoundaryHealth,
|
||||
ServerBetaQueueLaneMetric,
|
||||
ServerBetaQueueManager,
|
||||
} from './types.js';
|
||||
|
||||
@@ -75,6 +76,49 @@ export class ActiveServerBetaQueueManager implements ServerBetaQueueManager {
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Phase 12 — per-lane counts. Returns BullMQ getJobCounts plus the
|
||||
* per-process stalled counter. If Redis is unreachable, the lane is
|
||||
* reported with an `unavailable` flag rather than throwing so /api/health
|
||||
* remains responsive even in partial-failure modes.
|
||||
*/
|
||||
async getLaneMetrics(): Promise<ServerBetaQueueLaneMetric[]> {
|
||||
const out: ServerBetaQueueLaneMetric[] = [];
|
||||
for (const kind of QUEUE_KINDS) {
|
||||
const queue = this.queues.get(kind);
|
||||
if (!queue) continue;
|
||||
const lifecycle = queue.getLifecycleCounters();
|
||||
try {
|
||||
const counts = await queue.getCounts();
|
||||
out.push({
|
||||
kind,
|
||||
name: SERVER_JOB_QUEUE_NAMES[kind],
|
||||
waiting: counts.waiting,
|
||||
active: counts.active,
|
||||
completed: counts.completed,
|
||||
failed: counts.failed,
|
||||
delayed: counts.delayed,
|
||||
stalled: lifecycle.stalled,
|
||||
unavailable: false,
|
||||
});
|
||||
} catch (error) {
|
||||
out.push({
|
||||
kind,
|
||||
name: SERVER_JOB_QUEUE_NAMES[kind],
|
||||
waiting: 0,
|
||||
active: 0,
|
||||
completed: 0,
|
||||
failed: 0,
|
||||
delayed: 0,
|
||||
stalled: lifecycle.stalled,
|
||||
unavailable: true,
|
||||
unavailableReason: error instanceof Error ? error.message : String(error),
|
||||
});
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
if (this.closed) {
|
||||
return;
|
||||
|
||||
@@ -14,7 +14,11 @@ import {
|
||||
verifyPidFileOwnership,
|
||||
type PidInfo,
|
||||
} from '../../supervisor/process-registry.js';
|
||||
import type { ServerBetaServiceGraph } from './types.js';
|
||||
import { ServerV1PostgresRoutes } from '../routes/v1/ServerV1PostgresRoutes.js';
|
||||
import { SessionsObservationsAdapter } from '../compat/SessionsObservationsAdapter.js';
|
||||
import { SessionsSummarizeAdapter } from '../compat/SessionsSummarizeAdapter.js';
|
||||
import { ActiveServerBetaQueueManager } from './ActiveServerBetaQueueManager.js';
|
||||
import type { ServerBetaServiceGraph, ServerBetaQueueLaneMetric } from './types.js';
|
||||
|
||||
const SERVER_BETA_RUNTIME = 'server-beta';
|
||||
const DEFAULT_SERVER_BETA_HOST = '127.0.0.1';
|
||||
@@ -50,7 +54,12 @@ class ServerBetaRuntimeInfoRoutes implements RouteHandler {
|
||||
res.json({ status: 'ok', runtime: SERVER_BETA_RUNTIME });
|
||||
});
|
||||
|
||||
app.get('/v1/info', (_req, res) => {
|
||||
// Phase 12 — `/v1/info` includes per-lane queue metrics so deploy probes
|
||||
// can read waiting/active/completed/failed/delayed/stalled without
|
||||
// hitting `/api/health`. Sampling is best-effort: a Redis blip surfaces
|
||||
// the lane with `unavailable: true` rather than crashing the route.
|
||||
app.get('/v1/info', async (_req, res) => {
|
||||
const queueLanes = await collectQueueLaneMetrics(this.graph);
|
||||
res.json({
|
||||
name: 'claude-mem-server',
|
||||
runtime: SERVER_BETA_RUNTIME,
|
||||
@@ -65,11 +74,28 @@ class ServerBetaRuntimeInfoRoutes implements RouteHandler {
|
||||
providerRegistry: this.graph.providerRegistry.getHealth(),
|
||||
eventBroadcaster: this.graph.eventBroadcaster.getHealth(),
|
||||
},
|
||||
queueLanes,
|
||||
});
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
async function collectQueueLaneMetrics(
|
||||
graph: ServerBetaServiceGraph,
|
||||
): Promise<ServerBetaQueueLaneMetric[]> {
|
||||
const manager = graph.queueManager;
|
||||
if (!(manager instanceof ActiveServerBetaQueueManager)) {
|
||||
return [];
|
||||
}
|
||||
try {
|
||||
return await manager.getLaneMetrics();
|
||||
} catch {
|
||||
// /api/health and /v1/info MUST never throw on a queue blip — surface
|
||||
// empty lanes so the rest of the payload still renders.
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
export class ServerBetaService {
|
||||
private readonly graph: ServerBetaServiceGraph;
|
||||
private readonly host: string;
|
||||
@@ -106,8 +132,73 @@ export class ServerBetaService {
|
||||
authMethod: this.graph.authMode,
|
||||
lastInteraction: null,
|
||||
}),
|
||||
// Phase 10 — surface BullMQ/Valkey health on /api/health so deploy
|
||||
// probes (and the Docker E2E) can confirm the queue engine without
|
||||
// peeking at /v1/info. The queue manager's getHealth() returns its
|
||||
// boundary descriptor; we shape it into the worker-compatible
|
||||
// ObservationQueueHealth schema the Server class expects.
|
||||
// Phase 12 — also include per-lane counts (waiting/active/completed/
|
||||
// failed/delayed/stalled) so deploy probes can monitor saturation.
|
||||
getQueueHealth: async () => {
|
||||
const health = this.graph.queueManager.getHealth();
|
||||
const details = (health.details ?? {}) as Record<string, unknown>;
|
||||
if (health.status !== 'active' || details.engine !== 'bullmq') {
|
||||
return null;
|
||||
}
|
||||
const lanes = await collectQueueLaneMetrics(this.graph);
|
||||
return {
|
||||
engine: 'bullmq' as const,
|
||||
redis: {
|
||||
status: 'ok' as const,
|
||||
mode: String(details.mode ?? 'unknown'),
|
||||
host: String(details.host ?? '127.0.0.1'),
|
||||
port: typeof details.port === 'number' ? details.port : 6379,
|
||||
prefix: String(details.prefix ?? 'claude_mem'),
|
||||
},
|
||||
lanes: lanes.map(lane => ({
|
||||
kind: lane.kind,
|
||||
name: lane.name,
|
||||
waiting: lane.waiting,
|
||||
active: lane.active,
|
||||
completed: lane.completed,
|
||||
failed: lane.failed,
|
||||
delayed: lane.delayed,
|
||||
stalled: lane.stalled,
|
||||
unavailable: lane.unavailable,
|
||||
...(lane.unavailableReason ? { unavailableReason: lane.unavailableReason } : {}),
|
||||
})),
|
||||
};
|
||||
},
|
||||
});
|
||||
server.registerRoutes(new ServerBetaRuntimeInfoRoutes(this.graph));
|
||||
const v1Routes = new ServerV1PostgresRoutes({
|
||||
pool: this.graph.postgres.pool,
|
||||
queueManager: this.graph.queueManager,
|
||||
authMode: this.graph.authMode === 'disabled' ? 'api-key' : this.graph.authMode,
|
||||
runtime: SERVER_BETA_RUNTIME,
|
||||
// Session policy is read inside the routes (default 'per-event' from
|
||||
// resolveSessionGenerationPolicy(), env-overridable via
|
||||
// CLAUDE_MEM_SERVER_SESSION_POLICY). We do not duplicate it here.
|
||||
});
|
||||
server.registerRoutes(v1Routes);
|
||||
|
||||
// Phase 9 — legacy compatibility adapters. These translate the old
|
||||
// `/api/sessions/observations` and `/api/sessions/summarize` worker
|
||||
// routes to the canonical Server beta event/job model. They share the
|
||||
// SAME shared services with /v1/* routes — never duplicate ingest or
|
||||
// session-end logic. New clients should hit /v1/* directly.
|
||||
const compatAuthMode = this.graph.authMode === 'disabled' ? 'api-key' : this.graph.authMode;
|
||||
server.registerRoutes(new SessionsObservationsAdapter({
|
||||
pool: this.graph.postgres.pool,
|
||||
ingestEvents: v1Routes.getIngestEventsService(),
|
||||
authMode: compatAuthMode,
|
||||
}));
|
||||
server.registerRoutes(new SessionsSummarizeAdapter({
|
||||
pool: this.graph.postgres.pool,
|
||||
endSession: v1Routes.getEndSessionService(),
|
||||
authMode: compatAuthMode,
|
||||
}));
|
||||
|
||||
server.finalizeRoutes();
|
||||
|
||||
await server.listen(this.requestedPort, this.host);
|
||||
@@ -184,6 +275,28 @@ export async function runServerBetaCli(argv: string[] = process.argv.slice(2)):
|
||||
const port = getServerBetaPort();
|
||||
const host = process.env.CLAUDE_MEM_SERVER_HOST ?? DEFAULT_SERVER_BETA_HOST;
|
||||
|
||||
// Phase 10: `claude-mem server worker [start|--daemon]` runs the BullMQ
|
||||
// generation worker as a foregrounded process — no HTTP server, no route
|
||||
// registration. In Compose this becomes a separately scaled service.
|
||||
if (command === 'worker') {
|
||||
const sub = (argv[1] ?? '--daemon').toLowerCase();
|
||||
if (sub === 'start' || sub === '--daemon' || sub === 'run') {
|
||||
await runServerBetaGenerationWorker();
|
||||
return;
|
||||
}
|
||||
console.error('Usage: server-beta-service worker start');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// `server api-key create|list|revoke` mirrors the worker-service tooling
|
||||
// but writes to the Postgres `api_keys` table the server-beta runtime
|
||||
// actually reads from. The legacy worker-service CLI talks to SQLite and
|
||||
// would be invisible to this stack.
|
||||
if (command === 'server' && argv[1]?.toLowerCase() === 'api-key') {
|
||||
await runServerBetaApiKeyCli(argv.slice(2));
|
||||
return;
|
||||
}
|
||||
|
||||
switch (command) {
|
||||
case 'start': {
|
||||
const existing = readServerBetaPidFile();
|
||||
@@ -258,9 +371,212 @@ export async function runServerBetaCli(argv: string[] = process.argv.slice(2)):
|
||||
}
|
||||
}
|
||||
|
||||
// Phase 10 — Postgres-backed `server api-key create|list|revoke` CLI. The
|
||||
// legacy `worker-service.cjs server api-key` command talks to SQLite and
|
||||
// is invisible to the server-beta runtime, which reads keys from
|
||||
// Postgres. Use this entrypoint inside Docker / Compose.
|
||||
export async function runServerBetaApiKeyCli(argv: string[]): Promise<void> {
|
||||
const sub = argv[0]?.toLowerCase();
|
||||
const options = parseFlagArgs(argv.slice(1));
|
||||
|
||||
if (!process.env.CLAUDE_MEM_SERVER_DATABASE_URL) {
|
||||
console.error('CLAUDE_MEM_SERVER_DATABASE_URL is required for `server api-key` commands.');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const { getSharedPostgresPool } = await import('../../storage/postgres/index.js');
|
||||
const { PostgresAuthRepository } = await import('../../storage/postgres/auth.js');
|
||||
const { createHash, randomBytes } = await import('crypto');
|
||||
const pool = getSharedPostgresPool({ requireDatabaseUrl: true });
|
||||
const repo = new PostgresAuthRepository(pool);
|
||||
|
||||
try {
|
||||
if (sub === 'create') {
|
||||
const scopes = (options.scope ?? options.scopes ?? 'memories:read')
|
||||
.split(',')
|
||||
.map((scope: string) => scope.trim())
|
||||
.filter(Boolean);
|
||||
// Resolve team/project. If the caller passed --team/--project, honor
|
||||
// them. Otherwise, run the server-beta bootstrap to get-or-create the
|
||||
// local team+project, then create a NEW key against those IDs with
|
||||
// the caller's requested scopes (the bootstrap key uses hook scopes,
|
||||
// which is the wrong default for an arbitrary CLI-issued key).
|
||||
let teamId = options.team ?? null;
|
||||
let projectId = options.project ?? null;
|
||||
if (!teamId || !projectId) {
|
||||
const { bootstrapServerBetaApiKey } = await import('../../services/hooks/server-beta-bootstrap.js');
|
||||
const result = await bootstrapServerBetaApiKey({ pool, closePool: false });
|
||||
teamId = result.teamId;
|
||||
projectId = result.projectId;
|
||||
}
|
||||
const rawKey = `cmem_${randomBytes(24).toString('hex')}`;
|
||||
const keyHash = createHash('sha256').update(rawKey).digest('hex');
|
||||
const created = await repo.createApiKey({
|
||||
keyHash,
|
||||
teamId,
|
||||
projectId,
|
||||
scopes,
|
||||
actorId: 'system:server-beta-cli',
|
||||
});
|
||||
console.log(JSON.stringify({
|
||||
id: created.id,
|
||||
key: rawKey,
|
||||
name: options.name ?? 'server-api-key',
|
||||
teamId,
|
||||
projectId,
|
||||
scopes,
|
||||
}, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
if (sub === 'list') {
|
||||
// Bound the result set to prevent unintentional cross-tenant key
|
||||
// metadata disclosure when an admin runs `api-key list` on a shared
|
||||
// host. Default page is 100; --team filters to a single tenant.
|
||||
const teamFilter = options.team ?? null;
|
||||
const limitArg = Number.parseInt(options.limit ?? '100', 10);
|
||||
const offsetArg = Number.parseInt(options.offset ?? '0', 10);
|
||||
const limit = Number.isFinite(limitArg) && limitArg > 0 && limitArg <= 500
|
||||
? limitArg
|
||||
: 100;
|
||||
const offset = Number.isFinite(offsetArg) && offsetArg >= 0 ? offsetArg : 0;
|
||||
const where = teamFilter ? 'WHERE team_id = $1' : '';
|
||||
const params: unknown[] = teamFilter ? [teamFilter, limit, offset] : [limit, offset];
|
||||
const limitIdx = teamFilter ? 2 : 1;
|
||||
const offsetIdx = teamFilter ? 3 : 2;
|
||||
const result = await pool.query<{
|
||||
id: string;
|
||||
team_id: string | null;
|
||||
project_id: string | null;
|
||||
scopes: unknown;
|
||||
revoked_at: Date | null;
|
||||
expires_at: Date | null;
|
||||
last_used_at: Date | null;
|
||||
created_at: Date;
|
||||
}>(
|
||||
`SELECT id, team_id, project_id, scopes, revoked_at, expires_at, last_used_at, created_at
|
||||
FROM api_keys
|
||||
${where}
|
||||
ORDER BY created_at DESC
|
||||
LIMIT $${limitIdx} OFFSET $${offsetIdx}`,
|
||||
params,
|
||||
);
|
||||
console.log(JSON.stringify({
|
||||
teamId: teamFilter,
|
||||
limit,
|
||||
offset,
|
||||
count: result.rows.length,
|
||||
keys: result.rows.map(row => ({
|
||||
id: row.id,
|
||||
teamId: row.team_id,
|
||||
projectId: row.project_id,
|
||||
scopes: row.scopes,
|
||||
status: row.revoked_at ? 'revoked' : 'active',
|
||||
lastUsedAt: row.last_used_at?.toISOString() ?? null,
|
||||
expiresAt: row.expires_at?.toISOString() ?? null,
|
||||
createdAt: row.created_at.toISOString(),
|
||||
})),
|
||||
}, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
if (sub === 'revoke') {
|
||||
const id = argv[1];
|
||||
if (!id) {
|
||||
console.error('Usage: server-beta-service server api-key revoke <id>');
|
||||
process.exit(1);
|
||||
}
|
||||
const result = await pool.query(
|
||||
`UPDATE api_keys SET revoked_at = now()
|
||||
WHERE id = $1 AND revoked_at IS NULL
|
||||
RETURNING id`,
|
||||
[id],
|
||||
);
|
||||
if (result.rowCount === 0) {
|
||||
console.error(`API key not found or already revoked: ${id}`);
|
||||
process.exit(1);
|
||||
}
|
||||
console.log(JSON.stringify({ id, status: 'revoked' }, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
console.error(`Unknown server api-key subcommand: ${sub ?? '(none)'}`);
|
||||
console.error('Usage: server-beta-service server api-key create|list|revoke');
|
||||
process.exit(1);
|
||||
} finally {
|
||||
// Pool is shared; do not close here. The process will exit and the
|
||||
// pool tears down via the shared module's process exit hook.
|
||||
}
|
||||
}
|
||||
|
||||
function parseFlagArgs(argv: string[]): Record<string, string> {
|
||||
const out: Record<string, string> = {};
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const arg = argv[i];
|
||||
if (!arg) continue;
|
||||
if (arg.startsWith('--')) {
|
||||
const equalsIdx = arg.indexOf('=');
|
||||
if (equalsIdx > -1) {
|
||||
out[arg.slice(2, equalsIdx)] = arg.slice(equalsIdx + 1);
|
||||
} else {
|
||||
out[arg.slice(2)] = argv[i + 1] ?? '';
|
||||
i += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// Phase 10 — generation-worker-only entrypoint. Starts BullMQ workers against
|
||||
// the same Postgres + Valkey/Redis the HTTP server-beta service uses, but
|
||||
// never opens an HTTP listener. In Compose this is a separate, horizontally
|
||||
// scalable service. The HTTP server-beta service should run with
|
||||
// CLAUDE_MEM_GENERATION_DISABLED=true so generation only happens in this
|
||||
// process.
|
||||
export async function runServerBetaGenerationWorker(): Promise<void> {
|
||||
const { validateServerBetaEnv, createServerBetaService } = await import('./create-server-beta-service.js');
|
||||
validateServerBetaEnv();
|
||||
// Build the service WITHOUT starting HTTP. We reuse createServerBetaService
|
||||
// for pool + bootstrap + queue + generation worker wiring, but never call
|
||||
// service.start(). Generation is enabled here even if env says
|
||||
// CLAUDE_MEM_GENERATION_DISABLED, because this IS the generation worker.
|
||||
delete process.env.CLAUDE_MEM_GENERATION_DISABLED;
|
||||
const service = await createServerBetaService();
|
||||
const state = service.getRuntimeState();
|
||||
logger.info('SYSTEM', 'Server beta generation worker started (no HTTP)', {
|
||||
pid: process.pid,
|
||||
queue: state.boundaries.queueManager,
|
||||
generation: state.boundaries.generationWorkerManager,
|
||||
});
|
||||
console.log(JSON.stringify({ status: 'worker-running', runtime: SERVER_BETA_RUNTIME, pid: process.pid }));
|
||||
|
||||
let stopping = false;
|
||||
const shutdown = async () => {
|
||||
if (stopping) return;
|
||||
stopping = true;
|
||||
try {
|
||||
await service.stop();
|
||||
} finally {
|
||||
process.exit(0);
|
||||
}
|
||||
};
|
||||
process.once('SIGTERM', shutdown);
|
||||
process.once('SIGINT', shutdown);
|
||||
|
||||
// Block forever — Workers run in background via BullMQ. Without this the
|
||||
// process would exit and BullMQ jobs would never be consumed.
|
||||
await new Promise<void>(() => {});
|
||||
}
|
||||
|
||||
function getServerBetaPort(): number {
|
||||
const parsed = Number.parseInt(process.env.CLAUDE_MEM_SERVER_PORT ?? '', 10);
|
||||
return Number.isInteger(parsed) && parsed > 0 ? parsed : DEFAULT_SERVER_BETA_PORT;
|
||||
if (Number.isInteger(parsed) && parsed > 0) {
|
||||
return parsed;
|
||||
}
|
||||
// UID-derived default for multi-account isolation: two users on the same
|
||||
// host get distinct ports without explicit configuration. Containerized
|
||||
// deployments always pass CLAUDE_MEM_SERVER_PORT so this branch is local-only.
|
||||
return DEFAULT_SERVER_BETA_PORT + ((process.getuid?.() ?? 77) % 100);
|
||||
}
|
||||
|
||||
function spawnServerBetaDaemon(port: number): number | undefined {
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import {
|
||||
PostgresServerSessionsRepository,
|
||||
type PostgresServerSession,
|
||||
} from '../../storage/postgres/server-sessions.js';
|
||||
import type { PostgresAgentEvent } from '../../storage/postgres/agent-events.js';
|
||||
import type { JsonObject } from '../../storage/postgres/utils.js';
|
||||
import type { PostgresPool } from '../../storage/postgres/pool.js';
|
||||
import type { PostgresQueryable } from '../../storage/postgres/utils.js';
|
||||
|
||||
// ServerSessionRuntimeRepository is the runtime helper layer used by Server
|
||||
// beta routes and generation policies. It is intentionally thin: every method
|
||||
// requires explicit `team_id` + `project_id` and validates scope through the
|
||||
// underlying PostgresServerSessionsRepository (which calls
|
||||
// assertProjectOwnership before any write). It does NOT cache state — every
|
||||
// call hits Postgres so the runtime never trusts in-memory ActiveSession-style
|
||||
// objects, per the Phase 6 anti-pattern guard.
|
||||
|
||||
export interface ServerSessionScope {
|
||||
teamId: string;
|
||||
projectId: string;
|
||||
}
|
||||
|
||||
export interface GetActiveSessionInput extends ServerSessionScope {
|
||||
externalSessionId: string;
|
||||
contentSessionId?: string | null;
|
||||
agentId?: string | null;
|
||||
agentType?: string | null;
|
||||
platformSource?: string | null;
|
||||
metadata?: JsonObject;
|
||||
}
|
||||
|
||||
export interface ServerSessionRuntimeRepositoryOptions {
|
||||
client: PostgresQueryable;
|
||||
}
|
||||
|
||||
export class ServerSessionRuntimeRepository {
|
||||
private readonly repo: PostgresServerSessionsRepository;
|
||||
|
||||
constructor(private readonly options: ServerSessionRuntimeRepositoryOptions) {
|
||||
this.repo = new PostgresServerSessionsRepository(options.client);
|
||||
}
|
||||
|
||||
/**
|
||||
* Find or create the canonical Server beta session row for an external
|
||||
* session id. Idempotent on (project_id, external_session_id).
|
||||
*
|
||||
* Anti-pattern guard: this MUST NOT consult worker `ActiveSession` or any
|
||||
* legacy SessionStore. server_sessions is the canonical model.
|
||||
*/
|
||||
async getActiveSession(input: GetActiveSessionInput): Promise<PostgresServerSession> {
|
||||
const existing = await this.repo.findByExternalIdForScope({
|
||||
externalSessionId: input.externalSessionId,
|
||||
projectId: input.projectId,
|
||||
teamId: input.teamId,
|
||||
});
|
||||
if (existing) {
|
||||
return existing;
|
||||
}
|
||||
return this.repo.create({
|
||||
projectId: input.projectId,
|
||||
teamId: input.teamId,
|
||||
externalSessionId: input.externalSessionId,
|
||||
contentSessionId: input.contentSessionId ?? null,
|
||||
agentId: input.agentId ?? null,
|
||||
agentType: input.agentType ?? null,
|
||||
platformSource: input.platformSource ?? null,
|
||||
metadata: input.metadata ?? {},
|
||||
});
|
||||
}
|
||||
|
||||
async getById(input: { id: string } & ServerSessionScope): Promise<PostgresServerSession | null> {
|
||||
return this.repo.getByIdForScope({
|
||||
id: input.id,
|
||||
projectId: input.projectId,
|
||||
teamId: input.teamId,
|
||||
});
|
||||
}
|
||||
|
||||
async findByExternalId(input: {
|
||||
externalSessionId: string;
|
||||
} & ServerSessionScope): Promise<PostgresServerSession | null> {
|
||||
return this.repo.findByExternalIdForScope({
|
||||
externalSessionId: input.externalSessionId,
|
||||
projectId: input.projectId,
|
||||
teamId: input.teamId,
|
||||
});
|
||||
}
|
||||
|
||||
async listUnprocessedEvents(
|
||||
input: { serverSessionId: string; limit?: number } & ServerSessionScope,
|
||||
): Promise<PostgresAgentEvent[]> {
|
||||
const params: {
|
||||
serverSessionId: string;
|
||||
projectId: string;
|
||||
teamId: string;
|
||||
limit?: number;
|
||||
} = {
|
||||
serverSessionId: input.serverSessionId,
|
||||
projectId: input.projectId,
|
||||
teamId: input.teamId,
|
||||
};
|
||||
if (input.limit !== undefined) {
|
||||
params.limit = input.limit;
|
||||
}
|
||||
return this.repo.listUnprocessedEvents(params);
|
||||
}
|
||||
|
||||
/**
|
||||
* End the session if not already ended. Idempotent — re-ending a session
|
||||
* returns the unchanged row and never creates a duplicate summary job
|
||||
* because the (team_id, project_id, source_type='session_summary',
|
||||
* source_id) UNIQUE constraint on observation_generation_jobs collapses
|
||||
* duplicate enqueue attempts.
|
||||
*/
|
||||
async endSession(
|
||||
input: { id: string } & ServerSessionScope,
|
||||
): Promise<PostgresServerSession | null> {
|
||||
return this.repo.endSession({
|
||||
id: input.id,
|
||||
projectId: input.projectId,
|
||||
teamId: input.teamId,
|
||||
});
|
||||
}
|
||||
|
||||
async markGenerationStarted(
|
||||
input: { id: string } & ServerSessionScope,
|
||||
): Promise<PostgresServerSession | null> {
|
||||
return this.repo.markGenerationStarted({
|
||||
id: input.id,
|
||||
projectId: input.projectId,
|
||||
teamId: input.teamId,
|
||||
});
|
||||
}
|
||||
|
||||
async markGenerationCompleted(
|
||||
input: { id: string } & ServerSessionScope,
|
||||
): Promise<PostgresServerSession | null> {
|
||||
return this.repo.markGenerationCompleted({
|
||||
id: input.id,
|
||||
projectId: input.projectId,
|
||||
teamId: input.teamId,
|
||||
});
|
||||
}
|
||||
|
||||
async markGenerationFailed(
|
||||
input: { id: string; error?: string | null } & ServerSessionScope,
|
||||
): Promise<PostgresServerSession | null> {
|
||||
return this.repo.markGenerationFailed({
|
||||
id: input.id,
|
||||
projectId: input.projectId,
|
||||
teamId: input.teamId,
|
||||
error: input.error ?? null,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
export function createServerSessionRuntimeRepository(
|
||||
pool: PostgresPool,
|
||||
): ServerSessionRuntimeRepository {
|
||||
return new ServerSessionRuntimeRepository({ client: pool });
|
||||
}
|
||||
@@ -0,0 +1,206 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import type { JobsOptions } from 'bullmq';
|
||||
import type {
|
||||
GenerateObservationsForEventJob,
|
||||
GenerateSessionSummaryJob,
|
||||
} from '../jobs/types.js';
|
||||
import { buildServerJobId } from '../jobs/job-id.js';
|
||||
import type { PostgresAgentEvent } from '../../storage/postgres/agent-events.js';
|
||||
import type { PostgresObservationGenerationJob } from '../../storage/postgres/generation-jobs.js';
|
||||
|
||||
// SessionGenerationPolicy decides WHEN to enqueue work for the BullMQ event
|
||||
// and summary lanes. It is configurable via:
|
||||
// - CLAUDE_MEM_SERVER_SESSION_POLICY env var (per-process default)
|
||||
// - per-call override (per-team settings can plug in here later)
|
||||
//
|
||||
// Three policies are supported:
|
||||
// - 'per-event' (default): enqueue immediately on every event POST.
|
||||
// Matches Phase 4/5 behavior.
|
||||
// - 'debounce': enqueue with `delay`; when a new event arrives within
|
||||
// the window, replace the delayed job (deterministic
|
||||
// BullMQ jobId means re-add(jobId, ...) overwrites the
|
||||
// waiting entry, and removeOnComplete/Fail keep things
|
||||
// tidy). Outbox row is canonical so durability is safe.
|
||||
// - 'end-of-session': only enqueue summary jobs at /v1/sessions/:id/end.
|
||||
// Per-event posts skip BullMQ entirely; the outbox row
|
||||
// remains in `queued` state and startup reconciliation
|
||||
// will publish it later (or it can be cancelled).
|
||||
//
|
||||
// Anti-pattern guard: the policy MUST NOT use ActiveSession-style cached
|
||||
// state. Inputs are always reloaded by the caller from Postgres before this
|
||||
// fires.
|
||||
|
||||
export type ServerSessionGenerationPolicy = 'per-event' | 'debounce' | 'end-of-session';
|
||||
|
||||
const DEFAULT_DEBOUNCE_MS = 5000;
|
||||
|
||||
export interface SessionGenerationPolicyOptions {
|
||||
policy?: ServerSessionGenerationPolicy;
|
||||
debounceWindowMs?: number;
|
||||
}
|
||||
|
||||
export function resolveSessionGenerationPolicy(
|
||||
options: SessionGenerationPolicyOptions = {},
|
||||
): { policy: ServerSessionGenerationPolicy; debounceWindowMs: number } {
|
||||
const envPolicy = (process.env.CLAUDE_MEM_SERVER_SESSION_POLICY ?? '').trim().toLowerCase();
|
||||
const policy: ServerSessionGenerationPolicy = options.policy
|
||||
?? (envPolicy === 'debounce' || envPolicy === 'end-of-session' || envPolicy === 'per-event'
|
||||
? envPolicy
|
||||
: 'per-event');
|
||||
const debounceWindowMs = options.debounceWindowMs
|
||||
?? (Number.parseInt(process.env.CLAUDE_MEM_SERVER_SESSION_DEBOUNCE_MS ?? '', 10)
|
||||
|| DEFAULT_DEBOUNCE_MS);
|
||||
return {
|
||||
policy,
|
||||
debounceWindowMs: Number.isFinite(debounceWindowMs) && debounceWindowMs > 0
|
||||
? debounceWindowMs
|
||||
: DEFAULT_DEBOUNCE_MS,
|
||||
};
|
||||
}
|
||||
|
||||
export interface EnqueueEventDecisionInput {
|
||||
event: PostgresAgentEvent;
|
||||
outbox: PostgresObservationGenerationJob;
|
||||
// Phase 11 — identity context captured at HTTP ingest time so the BullMQ
|
||||
// payload carries every audit field. apiKeyId may be null for local-dev
|
||||
// enqueues and `actorId` follows the api key's `actor_id` column.
|
||||
apiKeyId?: string | null;
|
||||
actorId?: string | null;
|
||||
sourceAdapter?: string | null;
|
||||
// Phase 12 — request correlation id minted at the HTTP boundary.
|
||||
requestId?: string | null;
|
||||
}
|
||||
|
||||
export interface EnqueueEventDecision {
|
||||
shouldEnqueue: boolean;
|
||||
jobId: string;
|
||||
payload: GenerateObservationsForEventJob;
|
||||
jobsOptions?: JobsOptions;
|
||||
reason: 'per-event' | 'debounce' | 'end-of-session-skip';
|
||||
}
|
||||
|
||||
export function buildEnqueueEventDecision(
|
||||
input: EnqueueEventDecisionInput,
|
||||
options: SessionGenerationPolicyOptions = {},
|
||||
): EnqueueEventDecision {
|
||||
const resolved = resolveSessionGenerationPolicy(options);
|
||||
const jobId = input.outbox.bullmqJobId ?? buildServerJobId({
|
||||
kind: 'event',
|
||||
team_id: input.event.teamId,
|
||||
project_id: input.event.projectId,
|
||||
source_type: 'agent_event',
|
||||
source_id: input.event.id,
|
||||
});
|
||||
const payload: GenerateObservationsForEventJob = {
|
||||
kind: 'event',
|
||||
team_id: input.outbox.teamId,
|
||||
project_id: input.outbox.projectId,
|
||||
source_type: 'agent_event',
|
||||
source_id: input.event.id,
|
||||
generation_job_id: input.outbox.id,
|
||||
agent_event_id: input.event.id,
|
||||
api_key_id: input.apiKeyId ?? null,
|
||||
actor_id: input.actorId ?? null,
|
||||
source_adapter: input.sourceAdapter ?? input.event.sourceAdapter ?? 'api',
|
||||
request_id: input.requestId ?? null,
|
||||
};
|
||||
|
||||
if (resolved.policy === 'end-of-session') {
|
||||
return { shouldEnqueue: false, jobId, payload, reason: 'end-of-session-skip' };
|
||||
}
|
||||
|
||||
if (resolved.policy === 'debounce') {
|
||||
return {
|
||||
shouldEnqueue: true,
|
||||
jobId,
|
||||
payload,
|
||||
jobsOptions: { delay: resolved.debounceWindowMs },
|
||||
reason: 'debounce',
|
||||
};
|
||||
}
|
||||
|
||||
return { shouldEnqueue: true, jobId, payload, reason: 'per-event' };
|
||||
}
|
||||
|
||||
// Minimal queue surface used by scheduleDebouncedEventJob. Declared as an
|
||||
// interface (instead of `Pick<ServerJobQueue<...>, ...>`) so the parameter
|
||||
// accepts ServerJobQueue<ServerGenerationJobPayload> at the call site without
|
||||
// triggering invariant TPayload type errors. The ServerJobQueue.add signature
|
||||
// is structurally compatible — it requires `payload: TPayload`, and we only
|
||||
// hand in narrowed payloads.
|
||||
export interface DebounceableEventQueue {
|
||||
add(jobId: string, payload: GenerateObservationsForEventJob, options?: JobsOptions): Promise<void>;
|
||||
remove(jobId: string): Promise<void>;
|
||||
getJob(jobId: string): Promise<unknown>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply a debounce decision to a BullMQ queue. If a delayed job already exists
|
||||
* for this deterministic id, BullMQ's `add(jobId, ...)` will be a no-op, so we
|
||||
* proactively remove it first so the new event's delay window starts fresh.
|
||||
*
|
||||
* This implements the "if a new event arrives within window, replace the
|
||||
* delayed job" requirement.
|
||||
*/
|
||||
export async function scheduleDebouncedEventJob(
|
||||
queue: DebounceableEventQueue,
|
||||
decision: EnqueueEventDecision,
|
||||
): Promise<void> {
|
||||
if (!decision.shouldEnqueue) return;
|
||||
if (decision.reason === 'debounce') {
|
||||
try {
|
||||
const existing = await queue.getJob(decision.jobId);
|
||||
if (existing) {
|
||||
await queue.remove(decision.jobId);
|
||||
}
|
||||
} catch {
|
||||
// best-effort; if remove fails because the job already moved to active
|
||||
// we just let `add` no-op or fail through to the caller's error handler
|
||||
}
|
||||
}
|
||||
await queue.add(decision.jobId, decision.payload, decision.jobsOptions);
|
||||
}
|
||||
|
||||
export interface BuildSummaryJobInput {
|
||||
serverSessionId: string;
|
||||
teamId: string;
|
||||
projectId: string;
|
||||
generationJobId: string;
|
||||
// Phase 11 — same identity context the event-payload builder receives.
|
||||
apiKeyId?: string | null;
|
||||
actorId?: string | null;
|
||||
sourceAdapter?: string | null;
|
||||
// Phase 12 — request correlation id flows into the summary lane too.
|
||||
requestId?: string | null;
|
||||
}
|
||||
|
||||
export function buildSummaryJobId(input: {
|
||||
serverSessionId: string;
|
||||
teamId: string;
|
||||
projectId: string;
|
||||
}): string {
|
||||
return buildServerJobId({
|
||||
kind: 'summary',
|
||||
team_id: input.teamId,
|
||||
project_id: input.projectId,
|
||||
source_type: 'session_summary',
|
||||
source_id: input.serverSessionId,
|
||||
});
|
||||
}
|
||||
|
||||
export function buildSummaryJobPayload(input: BuildSummaryJobInput): GenerateSessionSummaryJob {
|
||||
return {
|
||||
kind: 'summary',
|
||||
team_id: input.teamId,
|
||||
project_id: input.projectId,
|
||||
source_type: 'session_summary',
|
||||
source_id: input.serverSessionId,
|
||||
generation_job_id: input.generationJobId,
|
||||
server_session_id: input.serverSessionId,
|
||||
api_key_id: input.apiKeyId ?? null,
|
||||
actor_id: input.actorId ?? null,
|
||||
source_adapter: input.sourceAdapter ?? 'api',
|
||||
request_id: input.requestId ?? null,
|
||||
};
|
||||
}
|
||||
@@ -1,10 +1,17 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
import { existsSync } from 'fs';
|
||||
import { logger } from '../../utils/logger.js';
|
||||
import { createPostgresStorageRepositories, getSharedPostgresPool, SERVER_BETA_POSTGRES_SCHEMA_VERSION } from '../../storage/postgres/index.js';
|
||||
import { bootstrapServerBetaPostgresSchema } from '../../storage/postgres/schema.js';
|
||||
import type { PostgresPool } from '../../storage/postgres/pool.js';
|
||||
import { getRedisQueueConfig } from '../queue/redis-config.js';
|
||||
import { ActiveServerBetaQueueManager } from './ActiveServerBetaQueueManager.js';
|
||||
import { ActiveServerBetaGenerationWorkerManager } from './ActiveServerBetaGenerationWorkerManager.js';
|
||||
import { ClaudeObservationProvider } from '../generation/providers/ClaudeObservationProvider.js';
|
||||
import { GeminiObservationProvider } from '../generation/providers/GeminiObservationProvider.js';
|
||||
import { OpenRouterObservationProvider } from '../generation/providers/OpenRouterObservationProvider.js';
|
||||
import type { ServerGenerationProvider } from '../generation/providers/shared/types.js';
|
||||
import { ServerBetaService } from './ServerBetaService.js';
|
||||
import {
|
||||
DisabledServerBetaEventBroadcaster,
|
||||
@@ -13,6 +20,7 @@ import {
|
||||
DisabledServerBetaQueueManager,
|
||||
type ServerBetaAuthMode,
|
||||
type ServerBetaBootstrapStatus,
|
||||
type ServerBetaGenerationWorkerManager,
|
||||
type ServerBetaQueueManager,
|
||||
type ServerBetaServiceGraph,
|
||||
} from './types.js';
|
||||
@@ -22,13 +30,147 @@ export interface CreateServerBetaServiceOptions {
|
||||
authMode?: ServerBetaAuthMode;
|
||||
bootstrapSchema?: boolean;
|
||||
queueManager?: ServerBetaQueueManager;
|
||||
// Phase 5 seam: tests can inject a fake provider without env config.
|
||||
generationProvider?: ServerGenerationProvider;
|
||||
generationWorkerManager?: ServerBetaGenerationWorkerManager;
|
||||
// Phase 10: when true, skip building the generation worker. Used when the
|
||||
// service is just an HTTP front-end and a separate `server worker` process
|
||||
// consumes the BullMQ queues.
|
||||
generationDisabled?: boolean;
|
||||
// Phase 10: skip env validation (tests). Production code paths always run
|
||||
// validation so misconfiguration fails fast at startup.
|
||||
skipEnvValidation?: boolean;
|
||||
}
|
||||
|
||||
// Phase 10 — env validation. Server beta in Docker requires explicit, complete
|
||||
// configuration. Missing pieces fail fast at startup rather than silently
|
||||
// degrading. Required env when running in Docker:
|
||||
// - CLAUDE_MEM_SERVER_DATABASE_URL (Postgres)
|
||||
// - CLAUDE_MEM_QUEUE_ENGINE=bullmq (no in-memory queue in Docker)
|
||||
// - CLAUDE_MEM_REDIS_URL (BullMQ requires Redis/Valkey)
|
||||
// - CLAUDE_MEM_AUTH_MODE != local-dev (auth must be real in Docker)
|
||||
// `local-dev` bypass is only valid on a developer's loopback; in Docker the
|
||||
// container is reachable via service-to-service networking and exposed ports,
|
||||
// so the loopback assumption is invalid.
|
||||
export interface ServerBetaEnvValidationOptions {
|
||||
env?: NodeJS.ProcessEnv;
|
||||
isDocker?: boolean;
|
||||
}
|
||||
|
||||
export interface ServerBetaEnvValidationResult {
|
||||
isDocker: boolean;
|
||||
runtime: string;
|
||||
authMode: string;
|
||||
queueEngine: string;
|
||||
hasDatabaseUrl: boolean;
|
||||
hasRedisUrl: boolean;
|
||||
}
|
||||
|
||||
export function detectDockerEnvironment(env: NodeJS.ProcessEnv = process.env): boolean {
|
||||
if (env.CLAUDE_MEM_DOCKER === '1' || env.CLAUDE_MEM_DOCKER === 'true') return true;
|
||||
// /.dockerenv is the canonical Docker marker; existsSync is cheap.
|
||||
try {
|
||||
if (existsSync('/.dockerenv')) return true;
|
||||
} catch {
|
||||
// ignore
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
export function validateServerBetaEnv(
|
||||
options: ServerBetaEnvValidationOptions = {},
|
||||
): ServerBetaEnvValidationResult {
|
||||
const env = options.env ?? process.env;
|
||||
const isDocker = options.isDocker ?? detectDockerEnvironment(env);
|
||||
const errors: string[] = [];
|
||||
|
||||
const runtime = (env.CLAUDE_MEM_RUNTIME ?? '').trim();
|
||||
if (!runtime) {
|
||||
// Warn but allow — defaulted to 'worker' upstream; we log a warning so
|
||||
// operators know server-beta is the active runtime here.
|
||||
if (isDocker) {
|
||||
logger.warn('SYSTEM', 'CLAUDE_MEM_RUNTIME unset; server-beta container assumes runtime=server-beta');
|
||||
}
|
||||
} else if (runtime !== 'server-beta' && isDocker) {
|
||||
errors.push(
|
||||
`CLAUDE_MEM_RUNTIME=${runtime} is invalid in Docker; the server-beta image only runs CLAUDE_MEM_RUNTIME=server-beta.`,
|
||||
);
|
||||
}
|
||||
|
||||
const authMode = (env.CLAUDE_MEM_AUTH_MODE ?? 'api-key').trim();
|
||||
if (isDocker) {
|
||||
if (authMode === 'local-dev') {
|
||||
errors.push(
|
||||
'CLAUDE_MEM_AUTH_MODE=local-dev is not allowed in Docker. Set CLAUDE_MEM_AUTH_MODE=api-key and create a key with `claude-mem server api-key create`.',
|
||||
);
|
||||
}
|
||||
if (
|
||||
env.CLAUDE_MEM_ALLOW_LOCAL_DEV_BYPASS === '1'
|
||||
|| env.CLAUDE_MEM_ALLOW_LOCAL_DEV_BYPASS === 'true'
|
||||
) {
|
||||
errors.push(
|
||||
'CLAUDE_MEM_ALLOW_LOCAL_DEV_BYPASS is not allowed in Docker. Loopback bypass cannot be enforced inside a container; remove the variable.',
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const queueEngine = (env.CLAUDE_MEM_QUEUE_ENGINE ?? '').trim().toLowerCase();
|
||||
if (isDocker) {
|
||||
if (!queueEngine) {
|
||||
errors.push('CLAUDE_MEM_QUEUE_ENGINE is required in Docker; set it to "bullmq".');
|
||||
} else if (queueEngine !== 'bullmq') {
|
||||
errors.push(
|
||||
`CLAUDE_MEM_QUEUE_ENGINE=${queueEngine} is not allowed in Docker. Only "bullmq" is supported (no in-process queues across container boundaries).`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const hasDatabaseUrl = Boolean((env.CLAUDE_MEM_SERVER_DATABASE_URL ?? '').trim());
|
||||
if (!hasDatabaseUrl) {
|
||||
errors.push('CLAUDE_MEM_SERVER_DATABASE_URL is required to start server-beta (Postgres connection string).');
|
||||
}
|
||||
|
||||
const hasRedisUrl = Boolean((env.CLAUDE_MEM_REDIS_URL ?? '').trim());
|
||||
if (queueEngine === 'bullmq' && !hasRedisUrl) {
|
||||
errors.push('CLAUDE_MEM_REDIS_URL is required when CLAUDE_MEM_QUEUE_ENGINE=bullmq.');
|
||||
}
|
||||
|
||||
if (errors.length > 0) {
|
||||
const message = [
|
||||
'server-beta startup configuration is invalid:',
|
||||
...errors.map(line => ` - ${line}`),
|
||||
].join('\n');
|
||||
throw new Error(message);
|
||||
}
|
||||
|
||||
return {
|
||||
isDocker,
|
||||
runtime: runtime || 'server-beta',
|
||||
authMode,
|
||||
queueEngine: queueEngine || 'disabled',
|
||||
hasDatabaseUrl,
|
||||
hasRedisUrl,
|
||||
};
|
||||
}
|
||||
|
||||
export async function createServerBetaService(
|
||||
options: CreateServerBetaServiceOptions = {},
|
||||
): Promise<ServerBetaService> {
|
||||
if (!options.skipEnvValidation) {
|
||||
validateServerBetaEnv();
|
||||
}
|
||||
const pool = options.pool ?? getSharedPostgresPool({ requireDatabaseUrl: true });
|
||||
const bootstrap = await initializePostgres(pool, options.bootstrapSchema ?? true);
|
||||
const queueManager = options.queueManager ?? buildQueueManager();
|
||||
const generationDisabled = options.generationDisabled
|
||||
?? (process.env.CLAUDE_MEM_GENERATION_DISABLED === '1'
|
||||
|| process.env.CLAUDE_MEM_GENERATION_DISABLED === 'true');
|
||||
const generationWorkerManager = options.generationWorkerManager
|
||||
?? (generationDisabled
|
||||
? new DisabledServerBetaGenerationWorkerManager(
|
||||
'CLAUDE_MEM_GENERATION_DISABLED is set; this server runs HTTP only. A separate `claude-mem server worker start` process consumes the BullMQ queues.',
|
||||
)
|
||||
: buildGenerationWorkerManager(pool, queueManager, options.generationProvider));
|
||||
const graph: ServerBetaServiceGraph = {
|
||||
runtime: 'server-beta',
|
||||
postgres: {
|
||||
@@ -36,16 +178,74 @@ export async function createServerBetaService(
|
||||
bootstrap,
|
||||
},
|
||||
authMode: options.authMode ?? parseAuthMode(process.env.CLAUDE_MEM_AUTH_MODE),
|
||||
queueManager: options.queueManager ?? buildQueueManager(),
|
||||
generationWorkerManager: new DisabledServerBetaGenerationWorkerManager('Phase 2 boundary only; generation workers are not wired.'),
|
||||
providerRegistry: new DisabledServerBetaProviderRegistry('Phase 2 boundary only; provider-backed generation is not wired.'),
|
||||
queueManager,
|
||||
generationWorkerManager,
|
||||
providerRegistry: new DisabledServerBetaProviderRegistry('Phase 5 keeps the provider registry boundary as inert; per-call providers are owned by the generation worker manager.'),
|
||||
eventBroadcaster: new DisabledServerBetaEventBroadcaster('Phase 2 boundary only; SSE/event broadcasting is not wired.'),
|
||||
storage: createPostgresStorageRepositories(pool),
|
||||
};
|
||||
|
||||
if (generationWorkerManager instanceof ActiveServerBetaGenerationWorkerManager) {
|
||||
generationWorkerManager.start();
|
||||
}
|
||||
|
||||
return new ServerBetaService({ graph });
|
||||
}
|
||||
|
||||
function buildGenerationWorkerManager(
|
||||
pool: PostgresPool,
|
||||
queueManager: ServerBetaQueueManager,
|
||||
injectedProvider?: ServerGenerationProvider,
|
||||
): ServerBetaGenerationWorkerManager {
|
||||
if (!(queueManager instanceof ActiveServerBetaQueueManager)) {
|
||||
return new DisabledServerBetaGenerationWorkerManager(
|
||||
'queue manager is disabled; set CLAUDE_MEM_QUEUE_ENGINE=bullmq to enable provider generation.',
|
||||
);
|
||||
}
|
||||
const provider = injectedProvider ?? buildServerGenerationProviderFromEnv();
|
||||
if (!provider) {
|
||||
return new DisabledServerBetaGenerationWorkerManager(
|
||||
'no server generation provider configured; set CLAUDE_MEM_SERVER_PROVIDER and the matching API key to enable.',
|
||||
);
|
||||
}
|
||||
return new ActiveServerBetaGenerationWorkerManager({
|
||||
pool,
|
||||
queueManager,
|
||||
provider,
|
||||
});
|
||||
}
|
||||
|
||||
function buildServerGenerationProviderFromEnv(): ServerGenerationProvider | null {
|
||||
const provider = (process.env.CLAUDE_MEM_SERVER_PROVIDER ?? '').trim().toLowerCase();
|
||||
if (!provider) return null;
|
||||
try {
|
||||
if (provider === 'claude' || provider === 'anthropic') {
|
||||
const apiKey = process.env.ANTHROPIC_API_KEY ?? process.env.CLAUDE_MEM_ANTHROPIC_API_KEY ?? '';
|
||||
if (!apiKey) return null;
|
||||
const opts: { apiKey: string; model?: string } = { apiKey };
|
||||
if (process.env.CLAUDE_MEM_SERVER_MODEL) opts.model = process.env.CLAUDE_MEM_SERVER_MODEL;
|
||||
return new ClaudeObservationProvider(opts);
|
||||
}
|
||||
if (provider === 'gemini') {
|
||||
const apiKey = process.env.GEMINI_API_KEY ?? process.env.CLAUDE_MEM_GEMINI_API_KEY ?? '';
|
||||
if (!apiKey) return null;
|
||||
const opts: { apiKey: string; model?: string } = { apiKey };
|
||||
if (process.env.CLAUDE_MEM_SERVER_MODEL) opts.model = process.env.CLAUDE_MEM_SERVER_MODEL;
|
||||
return new GeminiObservationProvider(opts);
|
||||
}
|
||||
if (provider === 'openrouter') {
|
||||
const apiKey = process.env.OPENROUTER_API_KEY ?? process.env.CLAUDE_MEM_OPENROUTER_API_KEY ?? '';
|
||||
if (!apiKey) return null;
|
||||
const opts: { apiKey: string; model?: string } = { apiKey };
|
||||
if (process.env.CLAUDE_MEM_SERVER_MODEL) opts.model = process.env.CLAUDE_MEM_SERVER_MODEL;
|
||||
return new OpenRouterObservationProvider(opts);
|
||||
}
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// Queue manager selection is fail-fast on misconfiguration. If the user
|
||||
// explicitly opts into BullMQ via CLAUDE_MEM_QUEUE_ENGINE=bullmq we build
|
||||
// the active manager; any error there throws so the runtime does not
|
||||
|
||||
@@ -20,6 +20,24 @@ export interface ServerBetaBoundaryHealth {
|
||||
details?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
// Phase 12 — per-lane queue metric snapshot. Returned by
|
||||
// ActiveServerBetaQueueManager.getLaneMetrics so /api/health and /v1/info
|
||||
// can publish current waiting/active/completed/failed/delayed/stalled counts
|
||||
// for each generation lane. `unavailable` is set when Redis was unreachable
|
||||
// at sample time so /api/health still responds rather than 500'ing.
|
||||
export interface ServerBetaQueueLaneMetric {
|
||||
kind: string;
|
||||
name: string;
|
||||
waiting: number;
|
||||
active: number;
|
||||
completed: number;
|
||||
failed: number;
|
||||
delayed: number;
|
||||
stalled: number;
|
||||
unavailable: boolean;
|
||||
unavailableReason?: string;
|
||||
}
|
||||
|
||||
export interface ServerBetaQueueManager {
|
||||
readonly kind: 'queue-manager';
|
||||
getHealth(): ServerBetaBoundaryHealth;
|
||||
|
||||
@@ -0,0 +1,155 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
// Shared session-end + summary-job path used by both `/v1/sessions/:id/end`
|
||||
// (canonical) and `src/server/compat/SessionsSummarizeAdapter.ts` (legacy
|
||||
// translator). Both call sites must produce identical Postgres state and
|
||||
// queue effects: ended_at idempotency, exactly one outbox row per session
|
||||
// summary, deterministic BullMQ job id.
|
||||
//
|
||||
// This module MUST NOT import from src/services/worker/* — Phase 9 keeps
|
||||
// the compat shim coupled to Server beta core only.
|
||||
|
||||
import {
|
||||
PostgresObservationGenerationJobEventsRepository,
|
||||
PostgresObservationGenerationJobRepository,
|
||||
type PostgresObservationGenerationJob,
|
||||
} from '../../storage/postgres/generation-jobs.js';
|
||||
import type { PostgresPool } from '../../storage/postgres/pool.js';
|
||||
import { withPostgresTransaction } from '../../storage/postgres/pool.js';
|
||||
import {
|
||||
PostgresServerSessionsRepository,
|
||||
type PostgresServerSession,
|
||||
} from '../../storage/postgres/server-sessions.js';
|
||||
import { logger } from '../../utils/logger.js';
|
||||
import { buildSummaryJobId, buildSummaryJobPayload } from '../runtime/SessionGenerationPolicy.js';
|
||||
import type { GenerateSessionSummaryJob } from '../jobs/types.js';
|
||||
import type { EnqueueOutcome, EventQueueLike } from './IngestEventsService.js';
|
||||
import { newId } from '../../storage/postgres/utils.js';
|
||||
|
||||
const SUMMARY_JOB_TYPE = 'observation_generate_session_summary';
|
||||
|
||||
export interface EndSessionServiceOptions {
|
||||
pool: PostgresPool;
|
||||
resolveSummaryQueue: () => EventQueueLike | null;
|
||||
}
|
||||
|
||||
export interface EndSessionResult {
|
||||
session: PostgresServerSession | null;
|
||||
outbox: PostgresObservationGenerationJob | null;
|
||||
enqueueState: EnqueueOutcome;
|
||||
}
|
||||
|
||||
export interface EndSessionInput {
|
||||
sessionId: string;
|
||||
projectId: string;
|
||||
teamId: string;
|
||||
source?: string;
|
||||
// Phase 11 — identity context propagated into the BullMQ summary payload.
|
||||
apiKeyId?: string | null;
|
||||
actorId?: string | null;
|
||||
sourceAdapter?: string | null;
|
||||
}
|
||||
|
||||
export class EndSessionService {
|
||||
constructor(private readonly options: EndSessionServiceOptions) {}
|
||||
|
||||
async end(input: EndSessionInput): Promise<EndSessionResult> {
|
||||
const source = input.source ?? 'http_post_v1_sessions_end';
|
||||
|
||||
const txResult = await withPostgresTransaction(this.options.pool, async (client) => {
|
||||
const sessionsRepo = new PostgresServerSessionsRepository(client);
|
||||
const ended = await sessionsRepo.endSession({
|
||||
id: input.sessionId,
|
||||
projectId: input.projectId,
|
||||
teamId: input.teamId,
|
||||
});
|
||||
if (!ended) {
|
||||
return {
|
||||
session: null as PostgresServerSession | null,
|
||||
outbox: null as PostgresObservationGenerationJob | null,
|
||||
};
|
||||
}
|
||||
const jobsRepo = new PostgresObservationGenerationJobRepository(client);
|
||||
const eventsLogRepo = new PostgresObservationGenerationJobEventsRepository(client);
|
||||
// Persist the BullMQ payload at create-time so reconciliation and
|
||||
// operator retry can re-enqueue a payload that passes the worker's
|
||||
// assertServerGenerationJobPayload validation.
|
||||
const outboxId = newId();
|
||||
const summaryPayload = buildSummaryJobPayload({
|
||||
serverSessionId: ended.id,
|
||||
teamId: ended.teamId,
|
||||
projectId: ended.projectId,
|
||||
generationJobId: outboxId,
|
||||
apiKeyId: input.apiKeyId ?? null,
|
||||
actorId: input.actorId ?? null,
|
||||
sourceAdapter: input.sourceAdapter ?? null,
|
||||
});
|
||||
const outbox = await jobsRepo.create({
|
||||
id: outboxId,
|
||||
projectId: ended.projectId,
|
||||
teamId: ended.teamId,
|
||||
sourceType: 'session_summary',
|
||||
sourceId: ended.id,
|
||||
serverSessionId: ended.id,
|
||||
jobType: SUMMARY_JOB_TYPE,
|
||||
bullmqJobId: buildSummaryJobId({
|
||||
serverSessionId: ended.id,
|
||||
teamId: ended.teamId,
|
||||
projectId: ended.projectId,
|
||||
}),
|
||||
payload: summaryPayload as unknown as Record<string, unknown>,
|
||||
});
|
||||
await eventsLogRepo.append({
|
||||
generationJobId: outbox.id,
|
||||
projectId: outbox.projectId,
|
||||
teamId: outbox.teamId,
|
||||
eventType: 'queued',
|
||||
statusAfter: outbox.status,
|
||||
attempt: outbox.attempts,
|
||||
details: { source },
|
||||
});
|
||||
return { session: ended, outbox };
|
||||
});
|
||||
|
||||
if (!txResult.session || !txResult.outbox) {
|
||||
return { session: txResult.session, outbox: null, enqueueState: 'skipped' };
|
||||
}
|
||||
const enqueueState = await this.publishSummaryJob(txResult.session.id, txResult.outbox, input);
|
||||
return { session: txResult.session, outbox: txResult.outbox, enqueueState };
|
||||
}
|
||||
|
||||
private async publishSummaryJob(
|
||||
serverSessionId: string,
|
||||
outbox: PostgresObservationGenerationJob,
|
||||
input: EndSessionInput,
|
||||
): Promise<'enqueued' | 'queued_only'> {
|
||||
const queue = this.options.resolveSummaryQueue();
|
||||
if (!queue) {
|
||||
return 'queued_only';
|
||||
}
|
||||
const jobId = outbox.bullmqJobId ?? buildSummaryJobId({
|
||||
serverSessionId,
|
||||
teamId: outbox.teamId,
|
||||
projectId: outbox.projectId,
|
||||
});
|
||||
const payload: GenerateSessionSummaryJob = buildSummaryJobPayload({
|
||||
serverSessionId,
|
||||
teamId: outbox.teamId,
|
||||
projectId: outbox.projectId,
|
||||
generationJobId: outbox.id,
|
||||
apiKeyId: input.apiKeyId ?? null,
|
||||
actorId: input.actorId ?? null,
|
||||
sourceAdapter: input.sourceAdapter ?? null,
|
||||
});
|
||||
try {
|
||||
await queue.add(jobId, payload);
|
||||
return 'enqueued';
|
||||
} catch (error) {
|
||||
logger.warn('SYSTEM', 'failed to publish summary generation job to BullMQ', {
|
||||
outboxId: outbox.id,
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
});
|
||||
return 'queued_only';
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,273 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
|
||||
// Shared event-ingest path used by both `/v1/events` (canonical) and
|
||||
// `src/server/compat/SessionsObservationsAdapter.ts` (legacy translator).
|
||||
// Centralizes the transactional write (event row + outbox row + lifecycle
|
||||
// log) and the post-commit BullMQ enqueue so both call sites apply the
|
||||
// exact same SessionGenerationPolicy and outbox-then-publish guarantees.
|
||||
//
|
||||
// This module MUST NOT import from src/services/worker/* — the whole point
|
||||
// of Phase 9 is to give the compat adapters a translation surface that
|
||||
// reaches Server beta core directly, with no worker-layer detours.
|
||||
|
||||
import type { CreatePostgresAgentEventInput, PostgresAgentEvent } from '../../storage/postgres/agent-events.js';
|
||||
import { PostgresAgentEventsRepository } from '../../storage/postgres/agent-events.js';
|
||||
import {
|
||||
PostgresObservationGenerationJobEventsRepository,
|
||||
PostgresObservationGenerationJobRepository,
|
||||
type PostgresObservationGenerationJob,
|
||||
} from '../../storage/postgres/generation-jobs.js';
|
||||
import type { PostgresPool } from '../../storage/postgres/pool.js';
|
||||
import { withPostgresTransaction } from '../../storage/postgres/pool.js';
|
||||
import { logger } from '../../utils/logger.js';
|
||||
import { buildServerJobId } from '../jobs/job-id.js';
|
||||
import type { GenerateObservationsForEventJob } from '../jobs/types.js';
|
||||
import {
|
||||
buildEnqueueEventDecision,
|
||||
scheduleDebouncedEventJob,
|
||||
type ServerSessionGenerationPolicy,
|
||||
} from '../runtime/SessionGenerationPolicy.js';
|
||||
import { newId } from '../../storage/postgres/utils.js';
|
||||
|
||||
function buildEventBullmqPayload(input: {
|
||||
outboxId: string;
|
||||
event: PostgresAgentEvent;
|
||||
apiKeyId: string | null;
|
||||
actorId: string | null;
|
||||
sourceAdapter: string | null;
|
||||
requestId: string | null;
|
||||
}): GenerateObservationsForEventJob {
|
||||
return {
|
||||
kind: 'event',
|
||||
team_id: input.event.teamId,
|
||||
project_id: input.event.projectId,
|
||||
source_type: 'agent_event',
|
||||
source_id: input.event.id,
|
||||
generation_job_id: input.outboxId,
|
||||
agent_event_id: input.event.id,
|
||||
api_key_id: input.apiKeyId,
|
||||
actor_id: input.actorId,
|
||||
source_adapter: input.sourceAdapter ?? input.event.sourceAdapter ?? 'api',
|
||||
request_id: input.requestId,
|
||||
};
|
||||
}
|
||||
|
||||
const EVENT_JOB_TYPE = 'observation_generate_for_event';
|
||||
|
||||
export type EnqueueOutcome = 'enqueued' | 'queued_only' | 'skipped';
|
||||
|
||||
export interface IngestEventsServiceOptions {
|
||||
pool: PostgresPool;
|
||||
// Lazy queue resolver so the service does not depend on the queue manager
|
||||
// type and tests can swap in a fake. When this returns null, the outbox
|
||||
// row stays `queued` and Phase 3 startup reconciliation will publish it.
|
||||
resolveEventQueue: () => EventQueueLike | null;
|
||||
sessionPolicy?: ServerSessionGenerationPolicy;
|
||||
sessionDebounceWindowMs?: number;
|
||||
}
|
||||
|
||||
export interface EventQueueLike {
|
||||
add(jobId: string, payload: unknown, options?: unknown): Promise<unknown>;
|
||||
}
|
||||
|
||||
export interface IngestEventResult {
|
||||
event: PostgresAgentEvent;
|
||||
outbox: PostgresObservationGenerationJob | null;
|
||||
enqueueState: EnqueueOutcome;
|
||||
}
|
||||
|
||||
export interface IngestEventOptions {
|
||||
generate?: boolean;
|
||||
source?: string;
|
||||
// Phase 11 — identity context that flows from the HTTP auth boundary into
|
||||
// the BullMQ payload and audit log. None of these are auth gates: the
|
||||
// worker reloads and re-validates from Postgres before any side effect.
|
||||
apiKeyId?: string | null;
|
||||
actorId?: string | null;
|
||||
sourceAdapter?: string | null;
|
||||
// Phase 12 — opaque correlation id minted at the HTTP middleware so
|
||||
// generator logs and audit rows can pivot back to the originating request.
|
||||
requestId?: string | null;
|
||||
}
|
||||
|
||||
export class IngestEventsService {
|
||||
constructor(private readonly options: IngestEventsServiceOptions) {}
|
||||
|
||||
async ingestOne(
|
||||
input: CreatePostgresAgentEventInput,
|
||||
opts: IngestEventOptions = {},
|
||||
): Promise<IngestEventResult> {
|
||||
const generate = opts.generate ?? true;
|
||||
const source = opts.source ?? 'http_post_v1_events';
|
||||
|
||||
const txResult = await withPostgresTransaction(this.options.pool, async (client) => {
|
||||
const eventsRepo = new PostgresAgentEventsRepository(client);
|
||||
const inserted = await eventsRepo.create(input);
|
||||
|
||||
if (!generate) {
|
||||
return { event: inserted, outbox: null as PostgresObservationGenerationJob | null };
|
||||
}
|
||||
|
||||
const jobsRepo = new PostgresObservationGenerationJobRepository(client);
|
||||
const eventsLogRepo = new PostgresObservationGenerationJobEventsRepository(client);
|
||||
// Pre-generate the outbox id so we can build the BullMQ payload (which
|
||||
// references generation_job_id) and persist it on the row. Reconciliation
|
||||
// and operator retry rely on this persisted payload to re-enqueue a
|
||||
// payload that passes assertServerGenerationJobPayload at the worker.
|
||||
const outboxId = newId();
|
||||
const bullmqPayload = buildEventBullmqPayload({
|
||||
outboxId,
|
||||
event: inserted,
|
||||
apiKeyId: opts.apiKeyId ?? null,
|
||||
actorId: opts.actorId ?? null,
|
||||
sourceAdapter: opts.sourceAdapter ?? null,
|
||||
requestId: opts.requestId ?? null,
|
||||
});
|
||||
const outbox = await jobsRepo.create({
|
||||
id: outboxId,
|
||||
projectId: inserted.projectId,
|
||||
teamId: inserted.teamId,
|
||||
sourceType: 'agent_event',
|
||||
sourceId: inserted.id,
|
||||
agentEventId: inserted.id,
|
||||
serverSessionId: inserted.serverSessionId,
|
||||
jobType: EVENT_JOB_TYPE,
|
||||
bullmqJobId: buildServerJobId({
|
||||
kind: 'event',
|
||||
team_id: inserted.teamId,
|
||||
project_id: inserted.projectId,
|
||||
source_type: 'agent_event',
|
||||
source_id: inserted.id,
|
||||
}),
|
||||
payload: bullmqPayload as unknown as Record<string, unknown>,
|
||||
});
|
||||
await eventsLogRepo.append({
|
||||
generationJobId: outbox.id,
|
||||
projectId: outbox.projectId,
|
||||
teamId: outbox.teamId,
|
||||
eventType: 'queued',
|
||||
statusAfter: outbox.status,
|
||||
attempt: outbox.attempts,
|
||||
details: { source },
|
||||
});
|
||||
return { event: inserted, outbox };
|
||||
});
|
||||
|
||||
let enqueueState: EnqueueOutcome = 'skipped';
|
||||
if (txResult.outbox) {
|
||||
enqueueState = await this.publishEventJob(txResult.event, txResult.outbox, opts);
|
||||
}
|
||||
return { event: txResult.event, outbox: txResult.outbox, enqueueState };
|
||||
}
|
||||
|
||||
async ingestBatch(
|
||||
inputs: CreatePostgresAgentEventInput[],
|
||||
opts: IngestEventOptions = {},
|
||||
): Promise<IngestEventResult[]> {
|
||||
const generate = opts.generate ?? true;
|
||||
const source = opts.source ?? 'http_post_v1_events_batch';
|
||||
|
||||
const txResults = await withPostgresTransaction(this.options.pool, async (client) => {
|
||||
const eventsRepo = new PostgresAgentEventsRepository(client);
|
||||
const jobsRepo = new PostgresObservationGenerationJobRepository(client);
|
||||
const eventsLogRepo = new PostgresObservationGenerationJobEventsRepository(client);
|
||||
const acc: { event: PostgresAgentEvent; outbox: PostgresObservationGenerationJob | null }[] = [];
|
||||
for (const input of inputs) {
|
||||
const event = await eventsRepo.create(input);
|
||||
if (!generate) {
|
||||
acc.push({ event, outbox: null });
|
||||
continue;
|
||||
}
|
||||
const outboxId = newId();
|
||||
const bullmqPayload = buildEventBullmqPayload({
|
||||
outboxId,
|
||||
event,
|
||||
apiKeyId: opts.apiKeyId ?? null,
|
||||
actorId: opts.actorId ?? null,
|
||||
sourceAdapter: opts.sourceAdapter ?? null,
|
||||
requestId: opts.requestId ?? null,
|
||||
});
|
||||
const outbox = await jobsRepo.create({
|
||||
id: outboxId,
|
||||
projectId: event.projectId,
|
||||
teamId: event.teamId,
|
||||
sourceType: 'agent_event',
|
||||
sourceId: event.id,
|
||||
agentEventId: event.id,
|
||||
serverSessionId: event.serverSessionId,
|
||||
jobType: EVENT_JOB_TYPE,
|
||||
bullmqJobId: buildServerJobId({
|
||||
kind: 'event',
|
||||
team_id: event.teamId,
|
||||
project_id: event.projectId,
|
||||
source_type: 'agent_event',
|
||||
source_id: event.id,
|
||||
}),
|
||||
payload: bullmqPayload as unknown as Record<string, unknown>,
|
||||
});
|
||||
await eventsLogRepo.append({
|
||||
generationJobId: outbox.id,
|
||||
projectId: outbox.projectId,
|
||||
teamId: outbox.teamId,
|
||||
eventType: 'queued',
|
||||
statusAfter: outbox.status,
|
||||
attempt: outbox.attempts,
|
||||
details: { source },
|
||||
});
|
||||
acc.push({ event, outbox });
|
||||
}
|
||||
return acc;
|
||||
});
|
||||
|
||||
return Promise.all(txResults.map(async ({ event, outbox }) => {
|
||||
const enqueueState: EnqueueOutcome = outbox
|
||||
? await this.publishEventJob(event, outbox, opts)
|
||||
: 'skipped';
|
||||
return { event, outbox, enqueueState };
|
||||
}));
|
||||
}
|
||||
|
||||
private async publishEventJob(
|
||||
event: PostgresAgentEvent,
|
||||
outbox: PostgresObservationGenerationJob,
|
||||
opts: IngestEventOptions = {},
|
||||
): Promise<'enqueued' | 'queued_only'> {
|
||||
const queue = this.options.resolveEventQueue();
|
||||
if (!queue) {
|
||||
return 'queued_only';
|
||||
}
|
||||
const policyOptions: { policy?: ServerSessionGenerationPolicy; debounceWindowMs?: number } = {};
|
||||
if (this.options.sessionPolicy !== undefined) {
|
||||
policyOptions.policy = this.options.sessionPolicy;
|
||||
}
|
||||
if (this.options.sessionDebounceWindowMs !== undefined) {
|
||||
policyOptions.debounceWindowMs = this.options.sessionDebounceWindowMs;
|
||||
}
|
||||
const decision = buildEnqueueEventDecision(
|
||||
{
|
||||
event,
|
||||
outbox,
|
||||
apiKeyId: opts.apiKeyId ?? null,
|
||||
actorId: opts.actorId ?? null,
|
||||
sourceAdapter: opts.sourceAdapter ?? event.sourceAdapter ?? null,
|
||||
// Phase 12 — flow request_id into the BullMQ payload so the worker
|
||||
// can emit it in [generation] logs and the audit row.
|
||||
requestId: opts.requestId ?? null,
|
||||
},
|
||||
policyOptions,
|
||||
);
|
||||
if (!decision.shouldEnqueue) {
|
||||
return 'queued_only';
|
||||
}
|
||||
try {
|
||||
await scheduleDebouncedEventJob(queue as never, decision);
|
||||
return 'enqueued';
|
||||
} catch (error) {
|
||||
logger.warn('SYSTEM', 'failed to publish event generation job to BullMQ', {
|
||||
outboxId: outbox.id,
|
||||
error: error instanceof Error ? error.message : String(error),
|
||||
});
|
||||
return 'queued_only';
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -23,6 +23,21 @@ import { existsSync } from 'node:fs';
|
||||
import { dirname, resolve } from 'node:path';
|
||||
import { homedir } from 'node:os';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import {
|
||||
ServerBetaClient,
|
||||
ServerBetaClientError,
|
||||
isServerBetaClientError,
|
||||
type ServerBetaAddObservationRequest,
|
||||
type ServerBetaContextObservationsRequest,
|
||||
type ServerBetaRecordEventRequest,
|
||||
type ServerBetaSearchObservationsRequest,
|
||||
} from '../services/hooks/server-beta-client.js';
|
||||
import {
|
||||
selectRuntime,
|
||||
buildServerBetaContext,
|
||||
type SelectedRuntime,
|
||||
type ServerBetaRuntimeContext,
|
||||
} from '../services/hooks/runtime-selector.js';
|
||||
|
||||
let mcpServerDirResolutionFailed = false;
|
||||
const mcpServerDir = (() => {
|
||||
@@ -150,6 +165,232 @@ async function verifyWorkerConnection(): Promise<boolean> {
|
||||
}
|
||||
}
|
||||
|
||||
// Phase 8 — runtime selection for MCP tools.
|
||||
// In server-beta mode, observation_* tools talk to the server-beta `/v1`
|
||||
// endpoints via the SAME ServerBetaClient hooks use. This guarantees we
|
||||
// share the REST core for writes and searches; we never duplicate the
|
||||
// event-insert + outbox + enqueue logic on the MCP side.
|
||||
//
|
||||
// We deliberately resolve the runtime per-call (cheap; reads cached
|
||||
// settings) so the user can flip CLAUDE_MEM_RUNTIME without restarting
|
||||
// the MCP server.
|
||||
type ServerBetaToolContext = ServerBetaRuntimeContext;
|
||||
|
||||
interface ServerBetaUnavailable {
|
||||
runtime: 'server-beta';
|
||||
available: false;
|
||||
reason: string;
|
||||
}
|
||||
|
||||
interface ServerBetaAvailable extends ServerBetaToolContext {
|
||||
available: true;
|
||||
}
|
||||
|
||||
type ServerBetaResolution = ServerBetaAvailable | ServerBetaUnavailable;
|
||||
|
||||
function resolveServerBetaToolContext(): ServerBetaResolution | null {
|
||||
const runtime: SelectedRuntime = selectRuntime();
|
||||
if (runtime !== 'server-beta') {
|
||||
return null;
|
||||
}
|
||||
const ctx = buildServerBetaContext();
|
||||
if (!ctx) {
|
||||
return {
|
||||
runtime: 'server-beta',
|
||||
available: false,
|
||||
reason: 'server-beta is selected but configuration is incomplete (missing url, api key, or project id)',
|
||||
};
|
||||
}
|
||||
return { ...ctx, available: true };
|
||||
}
|
||||
|
||||
function formatToolError(error: unknown): { content: Array<{ type: 'text'; text: string }>; isError: true } {
|
||||
if (isServerBetaClientError(error)) {
|
||||
return {
|
||||
content: [{
|
||||
type: 'text' as const,
|
||||
text: `Server beta error (${error.kind}${error.status ? ` ${error.status}` : ''}): ${error.message}`,
|
||||
}],
|
||||
isError: true as const,
|
||||
};
|
||||
}
|
||||
return {
|
||||
content: [{
|
||||
type: 'text' as const,
|
||||
text: `Tool error: ${error instanceof Error ? error.message : String(error)}`,
|
||||
}],
|
||||
isError: true as const,
|
||||
};
|
||||
}
|
||||
|
||||
function formatJsonResult(payload: unknown): { content: Array<{ type: 'text'; text: string }> } {
|
||||
return {
|
||||
content: [{
|
||||
type: 'text' as const,
|
||||
text: JSON.stringify(payload, null, 2),
|
||||
}],
|
||||
};
|
||||
}
|
||||
|
||||
function requireServerBetaForObservationTool(toolName: string): ServerBetaAvailable {
|
||||
const resolution = resolveServerBetaToolContext();
|
||||
if (!resolution) {
|
||||
throw new ServerBetaClientError(
|
||||
'transport',
|
||||
`${toolName} requires CLAUDE_MEM_RUNTIME=server-beta. Current runtime is "worker"; use the existing search/timeline/get_observations tools for worker-mode memory access.`,
|
||||
);
|
||||
}
|
||||
if (!resolution.available) {
|
||||
throw new ServerBetaClientError('missing_api_key', `${toolName}: ${resolution.reason}`);
|
||||
}
|
||||
return resolution;
|
||||
}
|
||||
|
||||
interface ObservationAddArgs {
|
||||
projectId?: string;
|
||||
serverSessionId?: string | null;
|
||||
kind?: string;
|
||||
content: string;
|
||||
metadata?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
async function handleObservationAdd(
|
||||
args: ObservationAddArgs,
|
||||
): Promise<{ content: Array<{ type: 'text'; text: string }>; isError?: boolean }> {
|
||||
try {
|
||||
const ctx = requireServerBetaForObservationTool('observation_add');
|
||||
if (typeof args?.content !== 'string' || args.content.trim().length === 0) {
|
||||
throw new Error('observation_add: "content" is required');
|
||||
}
|
||||
const projectId = args.projectId && args.projectId.trim().length > 0 ? args.projectId : ctx.projectId;
|
||||
const request: ServerBetaAddObservationRequest = {
|
||||
projectId,
|
||||
content: args.content,
|
||||
...(args.serverSessionId !== undefined ? { serverSessionId: args.serverSessionId } : {}),
|
||||
...(args.kind !== undefined ? { kind: args.kind } : {}),
|
||||
...(args.metadata !== undefined ? { metadata: args.metadata } : {}),
|
||||
};
|
||||
const response = await ctx.client.addObservation(request);
|
||||
return formatJsonResult(response);
|
||||
} catch (error) {
|
||||
return formatToolError(error);
|
||||
}
|
||||
}
|
||||
|
||||
interface ObservationRecordEventArgs {
|
||||
projectId?: string;
|
||||
serverSessionId?: string | null;
|
||||
contentSessionId?: string | null;
|
||||
memorySessionId?: string | null;
|
||||
sourceType?: 'hook' | 'worker' | 'provider' | 'server' | 'api';
|
||||
eventType: string;
|
||||
payload?: unknown;
|
||||
occurredAtEpoch?: number;
|
||||
generate?: boolean;
|
||||
}
|
||||
|
||||
async function handleObservationRecordEvent(
|
||||
args: ObservationRecordEventArgs,
|
||||
): Promise<{ content: Array<{ type: 'text'; text: string }>; isError?: boolean }> {
|
||||
try {
|
||||
const ctx = requireServerBetaForObservationTool('observation_record_event');
|
||||
if (typeof args?.eventType !== 'string' || args.eventType.trim().length === 0) {
|
||||
throw new Error('observation_record_event: "eventType" is required');
|
||||
}
|
||||
const projectId = args.projectId && args.projectId.trim().length > 0 ? args.projectId : ctx.projectId;
|
||||
const request: ServerBetaRecordEventRequest = {
|
||||
projectId,
|
||||
sourceType: args.sourceType ?? 'api',
|
||||
eventType: args.eventType,
|
||||
occurredAtEpoch: typeof args.occurredAtEpoch === 'number' ? args.occurredAtEpoch : Date.now(),
|
||||
...(args.serverSessionId !== undefined ? { serverSessionId: args.serverSessionId } : {}),
|
||||
...(args.contentSessionId !== undefined ? { contentSessionId: args.contentSessionId } : {}),
|
||||
...(args.memorySessionId !== undefined ? { memorySessionId: args.memorySessionId } : {}),
|
||||
...(args.payload !== undefined ? { payload: args.payload } : {}),
|
||||
...(args.generate !== undefined ? { generate: args.generate } : {}),
|
||||
};
|
||||
const response = await ctx.client.recordEvent(request);
|
||||
return formatJsonResult(response);
|
||||
} catch (error) {
|
||||
return formatToolError(error);
|
||||
}
|
||||
}
|
||||
|
||||
interface ObservationSearchArgs {
|
||||
projectId?: string;
|
||||
query: string;
|
||||
limit?: number;
|
||||
}
|
||||
|
||||
async function handleObservationSearch(
|
||||
args: ObservationSearchArgs,
|
||||
): Promise<{ content: Array<{ type: 'text'; text: string }>; isError?: boolean }> {
|
||||
try {
|
||||
const ctx = requireServerBetaForObservationTool('observation_search');
|
||||
if (typeof args?.query !== 'string' || args.query.trim().length === 0) {
|
||||
throw new Error('observation_search: "query" is required');
|
||||
}
|
||||
const projectId = args.projectId && args.projectId.trim().length > 0 ? args.projectId : ctx.projectId;
|
||||
const request: ServerBetaSearchObservationsRequest = {
|
||||
projectId,
|
||||
query: args.query,
|
||||
...(args.limit !== undefined ? { limit: args.limit } : {}),
|
||||
};
|
||||
const response = await ctx.client.searchObservations(request);
|
||||
return formatJsonResult(response);
|
||||
} catch (error) {
|
||||
return formatToolError(error);
|
||||
}
|
||||
}
|
||||
|
||||
interface ObservationContextArgs {
|
||||
projectId?: string;
|
||||
query: string;
|
||||
limit?: number;
|
||||
}
|
||||
|
||||
async function handleObservationContext(
|
||||
args: ObservationContextArgs,
|
||||
): Promise<{ content: Array<{ type: 'text'; text: string }>; isError?: boolean }> {
|
||||
try {
|
||||
const ctx = requireServerBetaForObservationTool('observation_context');
|
||||
if (typeof args?.query !== 'string' || args.query.trim().length === 0) {
|
||||
throw new Error('observation_context: "query" is required');
|
||||
}
|
||||
const projectId = args.projectId && args.projectId.trim().length > 0 ? args.projectId : ctx.projectId;
|
||||
const request: ServerBetaContextObservationsRequest = {
|
||||
projectId,
|
||||
query: args.query,
|
||||
...(args.limit !== undefined ? { limit: args.limit } : {}),
|
||||
};
|
||||
const response = await ctx.client.contextObservations(request);
|
||||
return formatJsonResult(response);
|
||||
} catch (error) {
|
||||
return formatToolError(error);
|
||||
}
|
||||
}
|
||||
|
||||
interface ObservationGenerationStatusArgs {
|
||||
jobId?: string;
|
||||
job_id?: string;
|
||||
}
|
||||
|
||||
async function handleObservationGenerationStatus(
|
||||
args: ObservationGenerationStatusArgs,
|
||||
): Promise<{ content: Array<{ type: 'text'; text: string }>; isError?: boolean }> {
|
||||
try {
|
||||
const ctx = requireServerBetaForObservationTool('observation_generation_status');
|
||||
const jobId = (args?.jobId ?? args?.job_id ?? '').trim();
|
||||
if (!jobId) {
|
||||
throw new Error('observation_generation_status: "jobId" is required');
|
||||
}
|
||||
const response = await ctx.client.getJobStatus(jobId);
|
||||
return formatJsonResult(response);
|
||||
} catch (error) {
|
||||
return formatToolError(error);
|
||||
}
|
||||
}
|
||||
|
||||
async function ensureWorkerConnection(): Promise<boolean> {
|
||||
if (await verifyWorkerConnection()) {
|
||||
return true;
|
||||
@@ -276,6 +517,156 @@ NEVER fetch full details without filtering first. 10x token savings.`,
|
||||
return await callWorkerAPIPost('/api/observations/batch', args);
|
||||
}
|
||||
},
|
||||
// Phase 8 — observation_* tools backed by server-beta REST core.
|
||||
// These are the canonical names. memory_* tools below are kept as
|
||||
// compatibility aliases that delegate to these handlers, so existing
|
||||
// MCP clients keep working without rewrites. (Plan line 753.)
|
||||
{
|
||||
name: 'observation_add',
|
||||
description: 'Insert a manual observation directly into server-beta storage. Calls /v1/memories — does NOT enqueue generation. Server-beta runtime only. Params: content (required), projectId (optional, falls back to settings), serverSessionId, kind, metadata.',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
projectId: { type: 'string', description: 'Project id (falls back to CLAUDE_MEM_SERVER_BETA_PROJECT_ID)' },
|
||||
serverSessionId: { type: 'string', description: 'Optional server_session_id to attach the observation to' },
|
||||
kind: { type: 'string', description: 'Observation kind (default: manual)' },
|
||||
content: { type: 'string', description: 'Observation content (required)' },
|
||||
metadata: { type: 'object', description: 'Free-form metadata object', additionalProperties: true },
|
||||
},
|
||||
required: ['content'],
|
||||
additionalProperties: false,
|
||||
},
|
||||
handler: async (args: any) => handleObservationAdd(args ?? {}),
|
||||
},
|
||||
{
|
||||
name: 'observation_record_event',
|
||||
description: 'Record an agent event into server-beta. Calls /v1/events — server inserts the event row, the outbox row, and enqueues a generation job atomically. Server-beta runtime only.',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
projectId: { type: 'string' },
|
||||
eventType: { type: 'string', description: 'Event type (required), e.g. PostToolUse, UserPromptSubmit' },
|
||||
sourceType: { type: 'string', enum: ['hook', 'worker', 'provider', 'server', 'api'] },
|
||||
serverSessionId: { type: 'string' },
|
||||
contentSessionId: { type: 'string' },
|
||||
memorySessionId: { type: 'string' },
|
||||
payload: { description: 'Event payload (any JSON value)' },
|
||||
occurredAtEpoch: { type: 'number', description: 'Unix epoch millis (defaults to now)' },
|
||||
generate: { type: 'boolean', description: 'If false, skip generation job (default: true)' },
|
||||
},
|
||||
required: ['eventType'],
|
||||
additionalProperties: false,
|
||||
},
|
||||
handler: async (args: any) => handleObservationRecordEvent(args ?? {}),
|
||||
},
|
||||
{
|
||||
name: 'observation_search',
|
||||
description: 'Full-text search across generated observations using server-beta\'s GIN tsvector index (Phase 1). Calls /v1/search. Server-beta runtime only. Params: query (required), projectId (optional), limit (default 20, max 100).',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
projectId: { type: 'string' },
|
||||
query: { type: 'string', description: 'Search query (required)' },
|
||||
limit: { type: 'number', description: 'Max results (default 20, max 100)' },
|
||||
},
|
||||
required: ['query'],
|
||||
additionalProperties: false,
|
||||
},
|
||||
handler: async (args: any) => handleObservationSearch(args ?? {}),
|
||||
},
|
||||
{
|
||||
name: 'observation_context',
|
||||
description: 'Get top-N relevant observations for context injection. Returns matched observations AND a pre-joined context string suitable for prompt injection. Calls /v1/context. Server-beta runtime only.',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
projectId: { type: 'string' },
|
||||
query: { type: 'string', description: 'Search query (required)' },
|
||||
limit: { type: 'number', description: 'Max observations (default 10, max 50)' },
|
||||
},
|
||||
required: ['query'],
|
||||
additionalProperties: false,
|
||||
},
|
||||
handler: async (args: any) => handleObservationContext(args ?? {}),
|
||||
},
|
||||
{
|
||||
name: 'observation_generation_status',
|
||||
description: 'Look up the status of an observation generation job by id. Calls /v1/jobs/:id. Server-beta runtime only. Returns the same payload as REST.',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
jobId: { type: 'string', description: 'Generation job id (required)' },
|
||||
},
|
||||
required: ['jobId'],
|
||||
additionalProperties: false,
|
||||
},
|
||||
handler: async (args: any) => handleObservationGenerationStatus(args ?? {}),
|
||||
},
|
||||
// Compatibility aliases — keep `memory_*` tool names that pre-existed in
|
||||
// src/server/mcp/tools.ts working for any client that bound to them.
|
||||
// These intentionally delegate to the same observation_* handlers so
|
||||
// there is one code path for MCP write/read against server-beta.
|
||||
{
|
||||
name: 'memory_add',
|
||||
description: 'Compatibility alias for observation_add. Same behavior; same schema modulo the legacy field names.',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
projectId: { type: 'string' },
|
||||
kind: { type: 'string' },
|
||||
content: { type: 'string' },
|
||||
narrative: { type: 'string', description: 'Legacy alias for content; mapped to content if content is missing' },
|
||||
title: { type: 'string', description: 'Legacy field; appended to metadata.title' },
|
||||
metadata: { type: 'object', additionalProperties: true },
|
||||
},
|
||||
required: ['projectId'],
|
||||
additionalProperties: true,
|
||||
},
|
||||
handler: async (args: any) => {
|
||||
// Map legacy fields onto observation_add. `narrative` was the v1
|
||||
// SQLite payload; it is normalized to `content` before forwarding.
|
||||
const merged: ObservationAddArgs = {
|
||||
projectId: args?.projectId,
|
||||
content: args?.content ?? args?.narrative ?? '',
|
||||
kind: args?.kind,
|
||||
metadata: {
|
||||
...(args?.metadata ?? {}),
|
||||
...(args?.title ? { title: args.title } : {}),
|
||||
},
|
||||
};
|
||||
return handleObservationAdd(merged);
|
||||
},
|
||||
},
|
||||
{
|
||||
name: 'memory_search',
|
||||
description: 'Compatibility alias for observation_search. Same FTS path; same /v1/search REST endpoint.',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
projectId: { type: 'string' },
|
||||
query: { type: 'string' },
|
||||
limit: { type: 'number' },
|
||||
},
|
||||
required: ['projectId', 'query'],
|
||||
additionalProperties: true,
|
||||
},
|
||||
handler: async (args: any) => handleObservationSearch(args ?? {}),
|
||||
},
|
||||
{
|
||||
name: 'memory_context',
|
||||
description: 'Compatibility alias for observation_context. Same /v1/context REST endpoint.',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
projectId: { type: 'string' },
|
||||
query: { type: 'string' },
|
||||
limit: { type: 'number' },
|
||||
},
|
||||
required: ['projectId', 'query'],
|
||||
additionalProperties: true,
|
||||
},
|
||||
handler: async (args: any) => handleObservationContext(args ?? {}),
|
||||
},
|
||||
{
|
||||
name: 'smart_search',
|
||||
description: 'Search codebase for symbols, functions, classes using tree-sitter AST parsing. Returns folded structural views with token counts. Use path parameter to scope the search.',
|
||||
@@ -631,6 +1022,14 @@ async function main() {
|
||||
startParentHeartbeat();
|
||||
|
||||
setTimeout(async () => {
|
||||
// Phase 8 — when CLAUDE_MEM_RUNTIME=server-beta, MCP must NOT auto-start
|
||||
// the worker. observation_* tools talk to server-beta directly; the
|
||||
// legacy worker-backed tools (search/timeline/get_observations) will
|
||||
// simply error with a helpful message until the user switches runtime.
|
||||
if (selectRuntime() === 'server-beta') {
|
||||
logger.info('SYSTEM', 'MCP runtime=server-beta — skipping worker auto-start', undefined, {});
|
||||
return;
|
||||
}
|
||||
const workerAvailable = await ensureWorkerConnection();
|
||||
if (!workerAvailable) {
|
||||
logger.error('SYSTEM', 'Worker not available', undefined, {});
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
//
|
||||
// Phase 7 — Runtime selector for hook subcommands.
|
||||
//
|
||||
// Reads `CLAUDE_MEM_RUNTIME` from `~/.claude-mem/settings.json` (via
|
||||
// `loadFromFileOnce`) and decides whether the hook should call the
|
||||
// server-beta /v1 endpoints or fall through to the worker compat path.
|
||||
//
|
||||
// This module deliberately does not import worker code so that hooks
|
||||
// running in server-beta mode can reach the runtime even when no worker
|
||||
// is installed.
|
||||
|
||||
import { loadFromFileOnce } from '../../shared/hook-settings.js';
|
||||
import { logger } from '../../utils/logger.js';
|
||||
import { ServerBetaClient, type ServerBetaClientConfig } from './server-beta-client.js';
|
||||
|
||||
export type SelectedRuntime = 'worker' | 'server-beta';
|
||||
|
||||
export interface ServerBetaRuntimeContext {
|
||||
runtime: 'server-beta';
|
||||
client: ServerBetaClient;
|
||||
projectId: string;
|
||||
serverBaseUrl: string;
|
||||
}
|
||||
|
||||
export interface WorkerRuntimeContext {
|
||||
runtime: 'worker';
|
||||
}
|
||||
|
||||
export type RuntimeContext = ServerBetaRuntimeContext | WorkerRuntimeContext;
|
||||
|
||||
export function selectRuntime(): SelectedRuntime {
|
||||
const settings = loadFromFileOnce();
|
||||
const raw = (settings.CLAUDE_MEM_RUNTIME ?? 'worker').trim().toLowerCase();
|
||||
if (raw === 'server-beta') return 'server-beta';
|
||||
return 'worker';
|
||||
}
|
||||
|
||||
export function buildServerBetaContext(): ServerBetaRuntimeContext | null {
|
||||
const settings = loadFromFileOnce();
|
||||
const serverBaseUrl = (settings.CLAUDE_MEM_SERVER_BETA_URL ?? '').trim();
|
||||
const apiKey = (settings.CLAUDE_MEM_SERVER_BETA_API_KEY ?? '').trim();
|
||||
const projectId = (settings.CLAUDE_MEM_SERVER_BETA_PROJECT_ID ?? '').trim();
|
||||
|
||||
if (!serverBaseUrl) {
|
||||
logger.warn('HOOK', '[server-beta-fallback] reason=missing_base_url');
|
||||
return null;
|
||||
}
|
||||
if (!apiKey) {
|
||||
logger.warn('HOOK', '[server-beta-fallback] reason=missing_api_key');
|
||||
return null;
|
||||
}
|
||||
if (!projectId) {
|
||||
logger.warn('HOOK', '[server-beta-fallback] reason=missing_project_id');
|
||||
return null;
|
||||
}
|
||||
|
||||
const config: ServerBetaClientConfig = {
|
||||
serverBaseUrl,
|
||||
apiKey,
|
||||
};
|
||||
return {
|
||||
runtime: 'server-beta',
|
||||
client: new ServerBetaClient(config),
|
||||
projectId,
|
||||
serverBaseUrl,
|
||||
};
|
||||
}
|
||||
|
||||
export function resolveRuntimeContext(): RuntimeContext {
|
||||
if (selectRuntime() !== 'server-beta') {
|
||||
return { runtime: 'worker' };
|
||||
}
|
||||
const ctx = buildServerBetaContext();
|
||||
if (!ctx) {
|
||||
return { runtime: 'worker' };
|
||||
}
|
||||
return ctx;
|
||||
}
|
||||
|
||||
export function logServerBetaFallback(reason: string, details?: Record<string, unknown>): void {
|
||||
logger.warn('HOOK', `[server-beta-fallback] reason=${reason}`, details ?? {});
|
||||
}
|
||||
@@ -0,0 +1,209 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
//
|
||||
// Phase 7 — Local API key bootstrap for the server-beta runtime.
|
||||
//
|
||||
// When the operator selects `runtime: "server-beta"` during install (or via
|
||||
// the `claude-mem server keys rotate` command), we provision a local hook
|
||||
// API key against the local Postgres so hooks can authenticate to /v1/*.
|
||||
//
|
||||
// Bootstrapping flow:
|
||||
// 1. Connect to Postgres (CLAUDE_MEM_SERVER_DATABASE_URL).
|
||||
// 2. Find or create a "local-hook" team and project so the api_key has
|
||||
// proper tenant scope.
|
||||
// 3. Generate a `cmem_<random>` key, hash with SHA-256, insert into
|
||||
// `api_keys` with the scopes hooks need: events:write, sessions:write,
|
||||
// observations:read, jobs:read.
|
||||
// 4. Persist the plaintext key to ~/.claude-mem/settings.json under
|
||||
// `CLAUDE_MEM_SERVER_BETA_API_KEY`, then chmod that file to 0600 so
|
||||
// only the owner can read it.
|
||||
//
|
||||
// The plaintext key is NEVER written into the generated bundle and never
|
||||
// logged.
|
||||
|
||||
import { createHash, randomBytes } from 'crypto';
|
||||
import { chmodSync, existsSync, readFileSync, writeFileSync, mkdirSync } from 'fs';
|
||||
import { dirname } from 'path';
|
||||
import { createPostgresPool, type PostgresPool } from '../../storage/postgres/pool.js';
|
||||
import { parsePostgresConfig } from '../../storage/postgres/config.js';
|
||||
import { PostgresAuthRepository } from '../../storage/postgres/auth.js';
|
||||
import { PostgresProjectsRepository } from '../../storage/postgres/projects.js';
|
||||
import { PostgresTeamsRepository } from '../../storage/postgres/teams.js';
|
||||
|
||||
const LOCAL_HOOK_TEAM_NAME = 'local-hook-team';
|
||||
const LOCAL_HOOK_PROJECT_NAME = 'local-hook-project';
|
||||
const LOCAL_HOOK_ACTOR_ID = 'system:local-hook-bootstrap';
|
||||
|
||||
export const HOOK_API_KEY_SCOPES: readonly string[] = Object.freeze([
|
||||
'events:write',
|
||||
'sessions:write',
|
||||
'observations:read',
|
||||
'jobs:read',
|
||||
]);
|
||||
|
||||
export interface BootstrapResult {
|
||||
rawKey: string;
|
||||
apiKeyId: string;
|
||||
teamId: string;
|
||||
projectId: string;
|
||||
}
|
||||
|
||||
export interface BootstrapDependencies {
|
||||
pool?: PostgresPool;
|
||||
// For tests: skip pool.end() because the caller owns lifecycle.
|
||||
closePool?: boolean;
|
||||
}
|
||||
|
||||
export async function bootstrapServerBetaApiKey(
|
||||
deps: BootstrapDependencies = {},
|
||||
): Promise<BootstrapResult> {
|
||||
const closePool = deps.closePool ?? deps.pool === undefined;
|
||||
const pool = deps.pool ?? buildPoolFromEnv();
|
||||
|
||||
try {
|
||||
const teamId = await findOrCreateTeam(pool);
|
||||
const projectId = await findOrCreateProject(pool, teamId);
|
||||
|
||||
const rawKey = createRawApiKey();
|
||||
const keyHash = hashApiKey(rawKey);
|
||||
|
||||
const repo = new PostgresAuthRepository(pool);
|
||||
const created = await repo.createApiKey({
|
||||
keyHash,
|
||||
teamId,
|
||||
projectId,
|
||||
actorId: LOCAL_HOOK_ACTOR_ID,
|
||||
scopes: [...HOOK_API_KEY_SCOPES],
|
||||
});
|
||||
await repo.createAuditLog({
|
||||
teamId,
|
||||
projectId,
|
||||
actorId: LOCAL_HOOK_ACTOR_ID,
|
||||
apiKeyId: created.id,
|
||||
action: 'api_key.create',
|
||||
resourceType: 'api_key',
|
||||
resourceId: created.id,
|
||||
details: { source: 'server-beta-bootstrap' },
|
||||
});
|
||||
|
||||
return {
|
||||
rawKey,
|
||||
apiKeyId: created.id,
|
||||
teamId,
|
||||
projectId,
|
||||
};
|
||||
} finally {
|
||||
if (closePool) {
|
||||
await pool.end().catch(() => undefined);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export interface RotateOptions {
|
||||
previousApiKeyId?: string | null;
|
||||
pool?: PostgresPool;
|
||||
}
|
||||
|
||||
export async function rotateServerBetaApiKey(options: RotateOptions = {}): Promise<BootstrapResult> {
|
||||
const closePool = options.pool === undefined;
|
||||
const pool = options.pool ?? buildPoolFromEnv();
|
||||
try {
|
||||
if (options.previousApiKeyId) {
|
||||
await pool.query(
|
||||
`UPDATE api_keys SET revoked_at = now() WHERE id = $1 AND revoked_at IS NULL`,
|
||||
[options.previousApiKeyId],
|
||||
);
|
||||
}
|
||||
return await bootstrapServerBetaApiKey({ pool, closePool: false });
|
||||
} finally {
|
||||
if (closePool) {
|
||||
await pool.end().catch(() => undefined);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export function persistServerBetaSettings(
|
||||
settingsPath: string,
|
||||
values: { apiKey: string; projectId: string; serverBaseUrl?: string },
|
||||
): void {
|
||||
const dir = dirname(settingsPath);
|
||||
if (!existsSync(dir)) {
|
||||
mkdirSync(dir, { recursive: true });
|
||||
}
|
||||
|
||||
let existing: Record<string, unknown> = {};
|
||||
if (existsSync(settingsPath)) {
|
||||
try {
|
||||
existing = JSON.parse(readFileSync(settingsPath, 'utf-8')) as Record<string, unknown>;
|
||||
} catch {
|
||||
existing = {};
|
||||
}
|
||||
}
|
||||
// Settings file format: prefer the flat shape (modern). The migration in
|
||||
// SettingsDefaultsManager.loadFromFile already collapses nested → flat.
|
||||
const flat = (existing.env && typeof existing.env === 'object'
|
||||
? existing.env
|
||||
: existing) as Record<string, unknown>;
|
||||
|
||||
flat.CLAUDE_MEM_SERVER_BETA_API_KEY = values.apiKey;
|
||||
flat.CLAUDE_MEM_SERVER_BETA_PROJECT_ID = values.projectId;
|
||||
if (values.serverBaseUrl) {
|
||||
flat.CLAUDE_MEM_SERVER_BETA_URL = values.serverBaseUrl;
|
||||
}
|
||||
|
||||
writeFileSync(settingsPath, JSON.stringify(flat, null, 2), 'utf-8');
|
||||
// Hooks read this file on every invocation; restrict permissions so other
|
||||
// local users cannot read the API key.
|
||||
try {
|
||||
chmodSync(settingsPath, 0o600);
|
||||
} catch {
|
||||
// Non-POSIX filesystems may reject chmod; settings file remains readable.
|
||||
}
|
||||
}
|
||||
|
||||
export function createRawApiKey(): string {
|
||||
return `cmem_${randomBytes(32).toString('base64url')}`;
|
||||
}
|
||||
|
||||
export function hashApiKey(rawKey: string): string {
|
||||
return createHash('sha256').update(rawKey).digest('hex');
|
||||
}
|
||||
|
||||
async function findOrCreateTeam(pool: PostgresPool): Promise<string> {
|
||||
const existing = await pool.query<{ id: string }>(
|
||||
`SELECT id FROM teams WHERE name = $1 LIMIT 1`,
|
||||
[LOCAL_HOOK_TEAM_NAME],
|
||||
);
|
||||
if (existing.rows[0]) {
|
||||
return existing.rows[0].id;
|
||||
}
|
||||
const repo = new PostgresTeamsRepository(pool);
|
||||
const team = await repo.create({ name: LOCAL_HOOK_TEAM_NAME, metadata: { source: 'local-hook-bootstrap' } });
|
||||
return team.id;
|
||||
}
|
||||
|
||||
async function findOrCreateProject(pool: PostgresPool, teamId: string): Promise<string> {
|
||||
const existing = await pool.query<{ id: string }>(
|
||||
`SELECT id FROM projects WHERE team_id = $1 AND name = $2 LIMIT 1`,
|
||||
[teamId, LOCAL_HOOK_PROJECT_NAME],
|
||||
);
|
||||
if (existing.rows[0]) {
|
||||
return existing.rows[0].id;
|
||||
}
|
||||
const repo = new PostgresProjectsRepository(pool);
|
||||
const project = await repo.create({
|
||||
teamId,
|
||||
name: LOCAL_HOOK_PROJECT_NAME,
|
||||
metadata: { source: 'local-hook-bootstrap' },
|
||||
});
|
||||
return project.id;
|
||||
}
|
||||
|
||||
function buildPoolFromEnv(): PostgresPool {
|
||||
const config = parsePostgresConfig({ requireDatabaseUrl: true });
|
||||
if (!config) {
|
||||
throw new Error(
|
||||
'Cannot bootstrap server-beta API key: CLAUDE_MEM_SERVER_DATABASE_URL is not set.',
|
||||
);
|
||||
}
|
||||
return createPostgresPool(config);
|
||||
}
|
||||
@@ -0,0 +1,400 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
//
|
||||
// Phase 7 — Server beta HTTP client used by hook subcommands when the
|
||||
// installer/setting selects the server-beta runtime. This client speaks
|
||||
// directly to the server-beta runtime's `/v1/*` endpoints. It MUST NOT
|
||||
// import or transitively depend on the worker runtime: the whole point
|
||||
// of phase 7 is that hooks can reach server-beta even when no worker is
|
||||
// running.
|
||||
//
|
||||
// On any transport-class failure (timeout, ECONNREFUSED, 5xx, missing
|
||||
// API key, etc.) callers receive a typed `ServerBetaClientError` so the
|
||||
// hook handler can decide whether to fall back to the worker path.
|
||||
|
||||
import { fetchWithTimeout } from '../../shared/worker-utils.js';
|
||||
import { HOOK_TIMEOUTS, getTimeout } from '../../shared/hook-constants.js';
|
||||
|
||||
const DEFAULT_TIMEOUT_MS = getTimeout(HOOK_TIMEOUTS.API_REQUEST);
|
||||
|
||||
export type ServerBetaClientErrorKind =
|
||||
| 'missing_api_key'
|
||||
| 'transport'
|
||||
| 'timeout'
|
||||
| 'http_error'
|
||||
| 'invalid_response';
|
||||
|
||||
export class ServerBetaClientError extends Error {
|
||||
readonly kind: ServerBetaClientErrorKind;
|
||||
readonly status: number | null;
|
||||
readonly cause?: unknown;
|
||||
|
||||
constructor(kind: ServerBetaClientErrorKind, message: string, options: {
|
||||
status?: number | null;
|
||||
cause?: unknown;
|
||||
} = {}) {
|
||||
super(message);
|
||||
this.name = 'ServerBetaClientError';
|
||||
this.kind = kind;
|
||||
this.status = options.status ?? null;
|
||||
this.cause = options.cause;
|
||||
}
|
||||
|
||||
isFallbackEligible(): boolean {
|
||||
if (this.kind === 'transport' || this.kind === 'timeout' || this.kind === 'missing_api_key') {
|
||||
return true;
|
||||
}
|
||||
if (this.kind === 'http_error') {
|
||||
// 5xx and 429 are transient; fall back. 4xx other than 429 is a real
|
||||
// client bug — surface it via the worker path so it can be observed.
|
||||
if (this.status !== null && this.status >= 500) return true;
|
||||
if (this.status === 429) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
export interface ServerBetaClientConfig {
|
||||
serverBaseUrl: string;
|
||||
apiKey: string;
|
||||
timeoutMs?: number;
|
||||
}
|
||||
|
||||
export interface ServerBetaStartSessionRequest {
|
||||
projectId: string;
|
||||
externalSessionId?: string | null;
|
||||
contentSessionId?: string | null;
|
||||
agentId?: string | null;
|
||||
agentType?: string | null;
|
||||
platformSource?: string | null;
|
||||
metadata?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface ServerBetaStartSessionResponse {
|
||||
session: {
|
||||
id: string;
|
||||
projectId: string;
|
||||
teamId: string;
|
||||
externalSessionId: string | null;
|
||||
contentSessionId: string | null;
|
||||
[key: string]: unknown;
|
||||
};
|
||||
}
|
||||
|
||||
export interface ServerBetaRecordEventRequest {
|
||||
projectId: string;
|
||||
serverSessionId?: string | null;
|
||||
contentSessionId?: string | null;
|
||||
memorySessionId?: string | null;
|
||||
sourceType: 'hook' | 'worker' | 'provider' | 'server' | 'api';
|
||||
eventType: string;
|
||||
payload?: unknown;
|
||||
occurredAtEpoch: number;
|
||||
// When false, the event is recorded but no generation job is enqueued.
|
||||
// Maps to the REST endpoint's `?generate=false` query flag.
|
||||
generate?: boolean;
|
||||
}
|
||||
|
||||
export interface ServerBetaRecordEventResponse {
|
||||
event: {
|
||||
id: string;
|
||||
projectId: string;
|
||||
serverSessionId: string | null;
|
||||
[key: string]: unknown;
|
||||
};
|
||||
generationJob?: {
|
||||
id: string;
|
||||
status: string;
|
||||
[key: string]: unknown;
|
||||
};
|
||||
}
|
||||
|
||||
export interface ServerBetaEndSessionRequest {
|
||||
sessionId: string;
|
||||
}
|
||||
|
||||
export interface ServerBetaEndSessionResponse {
|
||||
session: {
|
||||
id: string;
|
||||
[key: string]: unknown;
|
||||
};
|
||||
generationJob?: {
|
||||
id: string;
|
||||
status: string;
|
||||
[key: string]: unknown;
|
||||
};
|
||||
}
|
||||
|
||||
// Phase 8 — direct/manual observation insertion through `/v1/memories`.
|
||||
// This calls the same Postgres repository path as the REST core, so MCP
|
||||
// and REST never diverge on what counts as a valid observation insert.
|
||||
export interface ServerBetaAddObservationRequest {
|
||||
projectId: string;
|
||||
serverSessionId?: string | null;
|
||||
kind?: string;
|
||||
content: string;
|
||||
metadata?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface ServerBetaAddObservationResponse {
|
||||
memory: {
|
||||
id: string;
|
||||
projectId: string;
|
||||
teamId: string;
|
||||
serverSessionId: string | null;
|
||||
kind: string;
|
||||
content: string;
|
||||
metadata: Record<string, unknown>;
|
||||
[key: string]: unknown;
|
||||
};
|
||||
}
|
||||
|
||||
// Phase 8 — full-text search over generated observations.
|
||||
export interface ServerBetaSearchObservationsRequest {
|
||||
projectId: string;
|
||||
query: string;
|
||||
limit?: number;
|
||||
}
|
||||
|
||||
export interface ServerBetaSearchObservationsResponse {
|
||||
observations: Array<{
|
||||
id: string;
|
||||
projectId: string;
|
||||
content: string;
|
||||
[key: string]: unknown;
|
||||
}>;
|
||||
}
|
||||
|
||||
// Phase 8 — context pack for prompt injection. Server returns both the
|
||||
// matched observations AND a pre-joined `context` string.
|
||||
export interface ServerBetaContextObservationsRequest {
|
||||
projectId: string;
|
||||
query: string;
|
||||
limit?: number;
|
||||
}
|
||||
|
||||
export interface ServerBetaContextObservationsResponse {
|
||||
observations: Array<{
|
||||
id: string;
|
||||
projectId: string;
|
||||
content: string;
|
||||
[key: string]: unknown;
|
||||
}>;
|
||||
context: string;
|
||||
}
|
||||
|
||||
// Phase 8 — generation job status, scoped by api-key team/project.
|
||||
export interface ServerBetaJobStatusResponse {
|
||||
generationJob: {
|
||||
id: string;
|
||||
status: string;
|
||||
[key: string]: unknown;
|
||||
};
|
||||
}
|
||||
|
||||
export class ServerBetaClient {
|
||||
private readonly baseUrl: string;
|
||||
private readonly apiKey: string;
|
||||
private readonly timeoutMs: number;
|
||||
|
||||
constructor(config: ServerBetaClientConfig) {
|
||||
this.baseUrl = stripTrailingSlash(config.serverBaseUrl);
|
||||
this.apiKey = config.apiKey;
|
||||
this.timeoutMs = config.timeoutMs ?? DEFAULT_TIMEOUT_MS;
|
||||
}
|
||||
|
||||
async startSession(input: ServerBetaStartSessionRequest): Promise<ServerBetaStartSessionResponse> {
|
||||
const body = this.buildStartSessionPayload(input);
|
||||
return this.request<ServerBetaStartSessionResponse>('POST', '/v1/sessions/start', body);
|
||||
}
|
||||
|
||||
async recordEvent(input: ServerBetaRecordEventRequest): Promise<ServerBetaRecordEventResponse> {
|
||||
const body = this.buildEventPayload(input);
|
||||
const path = input.generate === false ? '/v1/events?generate=false' : '/v1/events';
|
||||
return this.request<ServerBetaRecordEventResponse>('POST', path, body);
|
||||
}
|
||||
|
||||
async endSession(input: ServerBetaEndSessionRequest): Promise<ServerBetaEndSessionResponse> {
|
||||
if (!input.sessionId) {
|
||||
throw new ServerBetaClientError('invalid_response', 'sessionId is required for endSession');
|
||||
}
|
||||
return this.request<ServerBetaEndSessionResponse>(
|
||||
'POST',
|
||||
`/v1/sessions/${encodeURIComponent(input.sessionId)}/end`,
|
||||
{},
|
||||
);
|
||||
}
|
||||
|
||||
// Phase 8 — direct observation insert (MCP `observation_add`). Calls
|
||||
// `/v1/memories`, which is the canonical write path that MUST NOT enqueue
|
||||
// a generation job. Anti-pattern guard for plan line 770: never duplicate
|
||||
// generation logic in MCP tools.
|
||||
async addObservation(
|
||||
input: ServerBetaAddObservationRequest,
|
||||
): Promise<ServerBetaAddObservationResponse> {
|
||||
return this.request<ServerBetaAddObservationResponse>(
|
||||
'POST',
|
||||
'/v1/memories',
|
||||
this.buildAddObservationPayload(input),
|
||||
);
|
||||
}
|
||||
|
||||
// Phase 8 — MCP `observation_search`. Routes to the FTS-backed REST
|
||||
// endpoint so search ranking and tenant scoping are owned by one place.
|
||||
async searchObservations(
|
||||
input: ServerBetaSearchObservationsRequest,
|
||||
): Promise<ServerBetaSearchObservationsResponse> {
|
||||
return this.request<ServerBetaSearchObservationsResponse>(
|
||||
'POST',
|
||||
'/v1/search',
|
||||
this.buildSearchPayload(input),
|
||||
);
|
||||
}
|
||||
|
||||
// Phase 8 — MCP `observation_context`. Same FTS surface as search, but
|
||||
// returns a pre-joined context string suitable for direct prompt injection.
|
||||
async contextObservations(
|
||||
input: ServerBetaContextObservationsRequest,
|
||||
): Promise<ServerBetaContextObservationsResponse> {
|
||||
return this.request<ServerBetaContextObservationsResponse>(
|
||||
'POST',
|
||||
'/v1/context',
|
||||
this.buildSearchPayload(input),
|
||||
);
|
||||
}
|
||||
|
||||
// Phase 8 — MCP `observation_generation_status`. Server returns the same
|
||||
// payload as `/v1/jobs/:id` so MCP clients and REST clients see identical
|
||||
// job status (including transport state).
|
||||
async getJobStatus(jobId: string): Promise<ServerBetaJobStatusResponse> {
|
||||
if (!jobId) {
|
||||
throw new ServerBetaClientError('invalid_response', 'jobId is required for getJobStatus');
|
||||
}
|
||||
return this.request<ServerBetaJobStatusResponse>(
|
||||
'GET',
|
||||
`/v1/jobs/${encodeURIComponent(jobId)}`,
|
||||
);
|
||||
}
|
||||
|
||||
buildAddObservationPayload(
|
||||
input: ServerBetaAddObservationRequest,
|
||||
): Record<string, unknown> {
|
||||
return {
|
||||
projectId: input.projectId,
|
||||
content: input.content,
|
||||
...(input.serverSessionId !== undefined ? { serverSessionId: input.serverSessionId } : {}),
|
||||
...(input.kind !== undefined ? { kind: input.kind } : {}),
|
||||
...(input.metadata !== undefined ? { metadata: input.metadata } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
buildSearchPayload(
|
||||
input: { projectId: string; query: string; limit?: number },
|
||||
): Record<string, unknown> {
|
||||
return {
|
||||
projectId: input.projectId,
|
||||
query: input.query,
|
||||
...(input.limit !== undefined ? { limit: input.limit } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
buildStartSessionPayload(input: ServerBetaStartSessionRequest): Record<string, unknown> {
|
||||
return {
|
||||
projectId: input.projectId,
|
||||
...(input.externalSessionId !== undefined ? { externalSessionId: input.externalSessionId } : {}),
|
||||
...(input.contentSessionId !== undefined ? { contentSessionId: input.contentSessionId } : {}),
|
||||
...(input.agentId !== undefined ? { agentId: input.agentId } : {}),
|
||||
...(input.agentType !== undefined ? { agentType: input.agentType } : {}),
|
||||
...(input.platformSource !== undefined ? { platformSource: input.platformSource } : {}),
|
||||
...(input.metadata !== undefined ? { metadata: input.metadata } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
buildEventPayload(input: ServerBetaRecordEventRequest): Record<string, unknown> {
|
||||
return {
|
||||
projectId: input.projectId,
|
||||
sourceType: input.sourceType,
|
||||
eventType: input.eventType,
|
||||
occurredAtEpoch: input.occurredAtEpoch,
|
||||
...(input.serverSessionId !== undefined ? { serverSessionId: input.serverSessionId } : {}),
|
||||
...(input.contentSessionId !== undefined ? { contentSessionId: input.contentSessionId } : {}),
|
||||
...(input.memorySessionId !== undefined ? { memorySessionId: input.memorySessionId } : {}),
|
||||
...(input.payload !== undefined ? { payload: input.payload } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
private async request<T>(
|
||||
method: 'GET' | 'POST',
|
||||
path: string,
|
||||
body?: unknown,
|
||||
): Promise<T> {
|
||||
if (!this.apiKey || !this.apiKey.trim()) {
|
||||
throw new ServerBetaClientError(
|
||||
'missing_api_key',
|
||||
'Server beta API key is not configured (CLAUDE_MEM_SERVER_BETA_API_KEY).',
|
||||
);
|
||||
}
|
||||
|
||||
const url = `${this.baseUrl}${path}`;
|
||||
const init: RequestInit = {
|
||||
method,
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
Authorization: `Bearer ${this.apiKey}`,
|
||||
},
|
||||
};
|
||||
if (body !== undefined) {
|
||||
init.body = JSON.stringify(body);
|
||||
}
|
||||
|
||||
let response: Response;
|
||||
try {
|
||||
response = await fetchWithTimeout(url, init, this.timeoutMs);
|
||||
} catch (error: unknown) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
const isTimeout = /timed out|timeout/i.test(message);
|
||||
throw new ServerBetaClientError(
|
||||
isTimeout ? 'timeout' : 'transport',
|
||||
`Server beta ${method} ${path} failed: ${message}`,
|
||||
{ cause: error },
|
||||
);
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
const text = await response.text().catch(() => '');
|
||||
throw new ServerBetaClientError(
|
||||
'http_error',
|
||||
`Server beta ${method} ${path} returned ${response.status}: ${truncate(text, 200)}`,
|
||||
{ status: response.status },
|
||||
);
|
||||
}
|
||||
|
||||
const text = await response.text();
|
||||
if (!text || text.length === 0) {
|
||||
// Endpoints we call always return JSON; a body-less success is unusual
|
||||
// but not fatal — return undefined-shaped object.
|
||||
return {} as T;
|
||||
}
|
||||
try {
|
||||
return JSON.parse(text) as T;
|
||||
} catch (error: unknown) {
|
||||
throw new ServerBetaClientError(
|
||||
'invalid_response',
|
||||
`Server beta ${method} ${path} returned non-JSON response`,
|
||||
{ cause: error },
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export function isServerBetaClientError(error: unknown): error is ServerBetaClientError {
|
||||
return error instanceof ServerBetaClientError;
|
||||
}
|
||||
|
||||
function stripTrailingSlash(url: string): string {
|
||||
return url.replace(/\/+$/, '');
|
||||
}
|
||||
|
||||
function truncate(text: string, max: number): string {
|
||||
if (text.length <= max) return text;
|
||||
return `${text.slice(0, max)}…`;
|
||||
}
|
||||
@@ -72,6 +72,9 @@ export interface SettingsDefaults {
|
||||
CLAUDE_MEM_QUEUE_REDIS_PREFIX: string;
|
||||
CLAUDE_MEM_AUTH_MODE: string;
|
||||
CLAUDE_MEM_RUNTIME: string;
|
||||
CLAUDE_MEM_SERVER_BETA_URL: string;
|
||||
CLAUDE_MEM_SERVER_BETA_API_KEY: string;
|
||||
CLAUDE_MEM_SERVER_BETA_PROJECT_ID: string;
|
||||
}
|
||||
|
||||
export class SettingsDefaultsManager {
|
||||
@@ -144,6 +147,9 @@ export class SettingsDefaultsManager {
|
||||
CLAUDE_MEM_QUEUE_REDIS_PREFIX: `claude_mem_${process.env.CLAUDE_MEM_WORKER_PORT ?? String(37700 + ((process.getuid?.() ?? 77) % 100))}`,
|
||||
CLAUDE_MEM_AUTH_MODE: 'api-key',
|
||||
CLAUDE_MEM_RUNTIME: 'worker',
|
||||
CLAUDE_MEM_SERVER_BETA_URL: `http://127.0.0.1:${process.env.CLAUDE_MEM_SERVER_PORT ?? String(37877 + ((process.getuid?.() ?? 77) % 100))}`, // Default server-beta runtime URL — UID-derived for multi-account isolation
|
||||
CLAUDE_MEM_SERVER_BETA_API_KEY: '', // Local hook API key, populated by installer when runtime=server-beta
|
||||
CLAUDE_MEM_SERVER_BETA_PROJECT_ID: '', // Default Postgres project_id used by hooks when runtime=server-beta
|
||||
};
|
||||
|
||||
static getAllDefaults(): SettingsDefaults {
|
||||
|
||||
@@ -2,6 +2,7 @@
|
||||
|
||||
import type { JsonObject, PostgresQueryable } from './utils.js';
|
||||
import { assertProjectOwnership, deterministicKey, newId, queryOne, toDate, toEpoch, toJsonObject } from './utils.js';
|
||||
import type { PostgresAgentEvent } from './agent-events.js';
|
||||
|
||||
export interface PostgresServerSession {
|
||||
id: string;
|
||||
@@ -119,6 +120,179 @@ export class PostgresServerSessionsRepository {
|
||||
);
|
||||
return result.rows.map(mapServerSessionRow);
|
||||
}
|
||||
|
||||
async findByExternalIdForScope(input: {
|
||||
externalSessionId: string;
|
||||
projectId: string;
|
||||
teamId: string;
|
||||
}): Promise<PostgresServerSession | null> {
|
||||
const row = await queryOne<ServerSessionRow>(
|
||||
this.client,
|
||||
`
|
||||
SELECT * FROM server_sessions
|
||||
WHERE external_session_id = $1 AND project_id = $2 AND team_id = $3
|
||||
`,
|
||||
[input.externalSessionId, input.projectId, input.teamId]
|
||||
);
|
||||
return row ? mapServerSessionRow(row) : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* End a server session by setting `ended_at = now()` if not already set.
|
||||
* Idempotent: if `ended_at` is already populated, returns the row unchanged.
|
||||
* Returns null if no row matches the (id, project_id, team_id) tuple.
|
||||
*/
|
||||
async endSession(input: {
|
||||
id: string;
|
||||
projectId: string;
|
||||
teamId: string;
|
||||
}): Promise<PostgresServerSession | null> {
|
||||
const updated = await queryOne<ServerSessionRow>(
|
||||
this.client,
|
||||
`
|
||||
UPDATE server_sessions
|
||||
SET ended_at = COALESCE(ended_at, now()),
|
||||
updated_at = CASE WHEN ended_at IS NULL THEN now() ELSE updated_at END
|
||||
WHERE id = $1 AND project_id = $2 AND team_id = $3
|
||||
RETURNING *
|
||||
`,
|
||||
[input.id, input.projectId, input.teamId]
|
||||
);
|
||||
return updated ? mapServerSessionRow(updated) : null;
|
||||
}
|
||||
|
||||
async markGenerationStarted(input: {
|
||||
id: string;
|
||||
projectId: string;
|
||||
teamId: string;
|
||||
}): Promise<PostgresServerSession | null> {
|
||||
const updated = await queryOne<ServerSessionRow>(
|
||||
this.client,
|
||||
`
|
||||
UPDATE server_sessions
|
||||
SET generation_status = 'processing', updated_at = now()
|
||||
WHERE id = $1 AND project_id = $2 AND team_id = $3
|
||||
RETURNING *
|
||||
`,
|
||||
[input.id, input.projectId, input.teamId]
|
||||
);
|
||||
return updated ? mapServerSessionRow(updated) : null;
|
||||
}
|
||||
|
||||
async markGenerationCompleted(input: {
|
||||
id: string;
|
||||
projectId: string;
|
||||
teamId: string;
|
||||
}): Promise<PostgresServerSession | null> {
|
||||
const updated = await queryOne<ServerSessionRow>(
|
||||
this.client,
|
||||
`
|
||||
UPDATE server_sessions
|
||||
SET generation_status = 'completed',
|
||||
last_generated_at = now(),
|
||||
updated_at = now()
|
||||
WHERE id = $1 AND project_id = $2 AND team_id = $3
|
||||
RETURNING *
|
||||
`,
|
||||
[input.id, input.projectId, input.teamId]
|
||||
);
|
||||
return updated ? mapServerSessionRow(updated) : null;
|
||||
}
|
||||
|
||||
async markGenerationFailed(input: {
|
||||
id: string;
|
||||
projectId: string;
|
||||
teamId: string;
|
||||
error?: string | null;
|
||||
}): Promise<PostgresServerSession | null> {
|
||||
const updated = await queryOne<ServerSessionRow>(
|
||||
this.client,
|
||||
`
|
||||
UPDATE server_sessions
|
||||
SET generation_status = 'failed',
|
||||
metadata = jsonb_set(
|
||||
COALESCE(metadata, '{}'::jsonb),
|
||||
'{lastGenerationError}',
|
||||
COALESCE(to_jsonb($4::text), 'null'::jsonb),
|
||||
true
|
||||
),
|
||||
updated_at = now()
|
||||
WHERE id = $1 AND project_id = $2 AND team_id = $3
|
||||
RETURNING *
|
||||
`,
|
||||
[input.id, input.projectId, input.teamId, input.error ?? null]
|
||||
);
|
||||
return updated ? mapServerSessionRow(updated) : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* List events tied to this server_session that do NOT yet have a completed
|
||||
* observation_generation_jobs row. Tenant-scoped: rows are filtered by
|
||||
* (project_id, team_id) before any join.
|
||||
*/
|
||||
async listUnprocessedEvents(input: {
|
||||
serverSessionId: string;
|
||||
projectId: string;
|
||||
teamId: string;
|
||||
limit?: number;
|
||||
}): Promise<PostgresAgentEvent[]> {
|
||||
const limit = input.limit ?? 500;
|
||||
const result = await this.client.query<UnprocessedEventRow>(
|
||||
`
|
||||
SELECT e.*
|
||||
FROM agent_events e
|
||||
WHERE e.server_session_id = $1
|
||||
AND e.project_id = $2
|
||||
AND e.team_id = $3
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM observation_generation_jobs j
|
||||
WHERE j.agent_event_id = e.id
|
||||
AND j.project_id = e.project_id
|
||||
AND j.team_id = e.team_id
|
||||
AND j.source_type = 'agent_event'
|
||||
AND j.status = 'completed'
|
||||
)
|
||||
ORDER BY e.occurred_at ASC
|
||||
LIMIT $4
|
||||
`,
|
||||
[input.serverSessionId, input.projectId, input.teamId, limit]
|
||||
);
|
||||
return result.rows.map(mapUnprocessedEventRow);
|
||||
}
|
||||
}
|
||||
|
||||
interface UnprocessedEventRow {
|
||||
id: string;
|
||||
project_id: string;
|
||||
team_id: string;
|
||||
server_session_id: string | null;
|
||||
source_adapter: string;
|
||||
source_event_id: string | null;
|
||||
idempotency_key: string;
|
||||
event_type: string;
|
||||
payload: unknown;
|
||||
metadata: unknown;
|
||||
occurred_at: Date;
|
||||
received_at: Date;
|
||||
created_at: Date;
|
||||
}
|
||||
|
||||
function mapUnprocessedEventRow(row: UnprocessedEventRow): PostgresAgentEvent {
|
||||
return {
|
||||
id: row.id,
|
||||
projectId: row.project_id,
|
||||
teamId: row.team_id,
|
||||
serverSessionId: row.server_session_id,
|
||||
sourceAdapter: row.source_adapter,
|
||||
sourceEventId: row.source_event_id,
|
||||
idempotencyKey: row.idempotency_key,
|
||||
eventType: row.event_type,
|
||||
payload: toJsonObject(row.payload),
|
||||
metadata: toJsonObject(row.metadata),
|
||||
occurredAtEpoch: row.occurred_at.getTime(),
|
||||
receivedAtEpoch: row.received_at.getTime(),
|
||||
createdAtEpoch: row.created_at.getTime()
|
||||
};
|
||||
}
|
||||
|
||||
export function buildServerSessionIdempotencyKey(input: {
|
||||
|
||||
Reference in New Issue
Block a user