claude-mem/plans/2026-05-07-claude-mem-server-apache-bullmq-team-auth.md

# Claude-Mem Server: Apache-2.0, BullMQ, Team Auth Plan

Status: implementation plan  
Date: 2026-05-07  
Primary command: `claude-mem server`  
Target runtime: deployable Docker container plus local compatibility path  
Target license: Apache-2.0 for embeddable/core code

## Executive Decision

Build Claude-Mem Server inside this repo as the canonical next runtime. Keep the current worker as a compatibility shim while moving shared server logic into typed server/core/storage modules.

Use:

- Express 5 initially, because the repo already depends on it and all current routes use `RouteHandler.setupRoutes(app)`.
- BullMQ as the queue engine for deployable server mode.
- Valkey/Redis as the Redis-compatible queue store, with Docker Compose as the first deployable path.
- Better Auth for user/org/team auth and API-key management, after the Express middleware bootstrap is refactored to satisfy Better Auth's handler-order requirements.
- SQLite as the source of truth for memory records in v0.1, with Redis/BullMQ treated as queue state.
- Apache-2.0 for the open core, server, CLI, SDKs, schemas, adapters, MCP tools, tests, examples, and public docs.

Do not build hosted Magic Recall cloud, billing, SSO/SAML/SCIM, enterprise RBAC UI, or cross-customer managed sync in this pass.

## Phase 0: Documentation Discovery

### Local Sources Read

- `/Users/alexnewman/Downloads/claude-mem-handoff-docs/apache-2-plan.md`
- `/Users/alexnewman/Downloads/claude-mem-handoff-docs/claude-mem-server-plan.md`
- `package.json`
- `src/npx-cli/index.ts`
- `src/npx-cli/commands/runtime.ts`
- `src/services/server/Server.ts`
- `src/services/worker-service.ts`
- `src/services/worker/http/middleware.ts`
- `src/services/worker/http/routes/SessionRoutes.ts`
- `src/services/worker/http/routes/SearchRoutes.ts`
- `src/services/worker/http/routes/DataRoutes.ts`
- `src/services/worker/http/routes/MemoryRoutes.ts`
- `src/services/sqlite/PendingMessageStore.ts`
- `src/services/queue/SessionQueueProcessor.ts`
- `src/services/worker/SessionManager.ts`
- `src/services/sqlite/schema.sql`
- `src/services/sqlite/migrations/runner.ts`
- `src/servers/mcp-server.ts`
- `docker/claude-mem/Dockerfile`
- `docker/claude-mem/README.md`
- `plans/2026-05-06-observation-queue-engine-deep-dive.md`
- `plans/2026-05-06-redis-dependency-strategy.md`

### External Docs Read

- BullMQ Queues: https://docs.bullmq.io/guide/queues
- BullMQ Stalled Jobs: https://docs.bullmq.io/guide/jobs/stalled
- BullMQ Job IDs: https://docs.bullmq.io/guide/jobs/job-ids
- Better Auth Express Integration: https://better-auth.com/docs/integrations/express
- Better Auth API Key Plugin: https://better-auth.com/docs/plugins/api-key
- Better Auth Organization Plugin: https://better-auth.com/docs/plugins/organization

### Allowed APIs And Patterns

- Existing route pattern: implement route classes with `setupRoutes(app: express.Application): void`, then register through `Server.registerRoutes(handler)`.
- Existing validation pattern: use `zod` schemas with `validateBody(schema)` from `src/services/worker/http/middleware/validateBody.ts`.
- Existing MCP pattern: add tools to the `tools` array in `src/servers/mcp-server.ts`, with plain JSON Schema `inputSchema` and handlers that call server/core logic.
- BullMQ queue creation: use `new Queue(name, { connection })`, then enqueue jobs with `queue.add(name, data, options)`. BullMQ stores jobs in Redis and workers can pick them up later.
- BullMQ dedupe: use a custom `jobId` or deduplication id for observation dedupe. Custom job IDs are unique per queue and duplicate adds are ignored while the previous job still exists.
- BullMQ stalled-job recovery: active jobs are locked and moved back to waiting or failed if the worker stops renewing the lock.
- Better Auth Express mount: mount `app.all("/api/auth/*splat", toNodeHandler(auth))` before `express.json()` on Express 5. Do not place global `express.json()` before the Better Auth handler.
- Better Auth API keys: use API-key plugin server methods for create, verify, update, delete, list; API keys can carry permissions and org ownership.
- Better Auth org/team support: use organization plugin with `teams: { enabled: true }` and custom project/memory permissions.

### Anti-Pattern Guards

- Do not create a second repo or primary `claude-mem-server` package.
- Do not replace all current worker routes at once.
- Do not put auth routes behind global `express.json()` if using Better Auth.
- Do not make MCP duplicate retrieval/storage logic.
- Do not treat Redis as the memory source of truth.
- Do not use Bee-Queue.
- Do not put sensitive prompt/tool payloads in Redis without a clear retention and redaction policy.
- Do not silently fall back to SQLite when `CLAUDE_MEM_QUEUE_ENGINE=bullmq` is explicitly configured.
- Do not claim team/org memory sync or enterprise SaaS is shipped in v0.1.

## Phase 1: License And Product Boundary

What to implement:

- Replace root `LICENSE` with official Apache License 2.0 text.
- Update `package.json` and nested manifests intended for public/core distribution to `"license": "Apache-2.0"`.
- Add `NOTICE`.
- Add `docs/license.md` and `docs/ip-boundary.md` using the handoff language from `apache-2-plan.md`.
- Update README license language.
- Add scoped SPDX headers to new `src/server`, `src/core`, `src/storage`, `src/sdk`, and `src/adapters` files as they are created.

Documentation references:

- Copy exact license text from https://www.apache.org/licenses/LICENSE-2.0.
- Use scope and commercial boundary from `/Users/alexnewman/Downloads/claude-mem-handoff-docs/apache-2-plan.md`.

Verification:

- `rg -n "AGPL|GNU Affero|Affero|GPL|copyleft|license" .`
- `rg -n "Claude-Mem™|trademark|official Anthropic|endorsed by Anthropic" .`
- `bun test tests/infrastructure/version-consistency.test.ts`
- Human review still required for contributor rights and dependency-license audit.

Anti-pattern guards:

- Do not add trademark claims around `Claude-Mem`.
- Do not move commercial/private features into the Apache-2.0 repo.

## Phase 2: Server Namespace And Compatibility CLI

What to implement:

- Add `src/npx-cli/commands/server.ts`.
- Teach `src/npx-cli/index.ts` to route:
  - `claude-mem server start`
  - `claude-mem server stop`
  - `claude-mem server restart`
  - `claude-mem server status`
  - `claude-mem server doctor`
  - `claude-mem server logs`
  - `claude-mem server migrate`
  - `claude-mem server export`
  - `claude-mem server import`
  - `claude-mem server api-key create|list|revoke`
- Keep existing `start|stop|restart|status` as worker compatibility aliases.
- Add `claude-mem worker start|stop|restart|status` aliases that call the same command implementation as `server`.
- Teach `src/services/worker-service.ts`'s internal command switch to accept the same `server` subcommands where installed plugin scripts invoke the worker bundle directly.

Documentation references:

- Copy process delegation pattern from `src/npx-cli/commands/runtime.ts`.
- Keep help formatting from `src/npx-cli/index.ts`.
- Keep worker script path conventions from `plugin/scripts/worker-service.cjs`.
- Copy worker-service command parsing shape from `src/services/worker-service.ts`.

Verification:

- `bun test tests/install-non-tty.test.ts tests/infrastructure/worker-json-status.test.ts`
- Add CLI parser tests for `server` and `worker` namespaces.
- Manual smoke:
  - `node dist/npx-cli/index.js --help`
  - `node dist/npx-cli/index.js server status`

Anti-pattern guards:

- Do not remove `npx claude-mem install`.
- Do not rename the primary npm binary.

## Phase 3: Server Bootstrap Refactor

What to implement:

- Create `src/server/create-server.ts` as the new composition root.
- Move the generic `Server` shell from `src/services/server/Server.ts` toward `src/server/http-server.ts`, but keep compatibility exports during migration.
- Split middleware registration into ordered buckets:
  - pre-body-parser routes, including Better Auth later;
  - body parser and CORS;
  - request logging and static UI;
  - route registration;
  - not-found/error handlers.
- Fix runtime dependency drift by adding `cors` to production dependencies or removing the runtime import. Current code imports `cors` but only `@types/cors` is declared.
- Update CORS `allowedHeaders` to include `Authorization` before API-key routes ship.

Documentation references:

- Copy existing server lifecycle from `src/services/server/Server.ts`.
- Copy CORS behavior from `src/services/worker/http/middleware.ts`.
- Follow Better Auth Express docs: auth handler before `express.json()`, Express 5 catch-all route uses `*splat`.

Verification:

- `bun test tests/server/server.test.ts tests/worker/middleware/cors-restriction.test.ts`
- Add a regression test proving auth routes are mounted before JSON middleware.
- `npm run typecheck:root`

Anti-pattern guards:

- Do not put global `express.json()` before Better Auth.
- Do not change current default host from `127.0.0.1` without a migration and explicit config.

## Phase 4: Core Contracts And Storage Boundary

What to implement:

- Add shared Zod schemas under `src/core/schemas/`:
  - `agent-event.ts`
  - `memory-item.ts`
  - `context-pack.ts`
  - `project.ts`
  - `session.ts`
  - `team.ts`
  - `auth.ts`
- Add `src/storage/sqlite/` repositories for new server-owned tables:
  - `projects`
  - `server_sessions`
  - `agent_events`
  - `memory_items`
  - `memory_sources`
  - `teams`
  - `team_members`
  - `api_keys` or Better Auth tables
  - `audit_log`
- Keep existing `sdk_sessions`, `observations`, `session_summaries`, `user_prompts`, and `pending_messages` readable during migration.
- Decide and document the translation layer between existing `observations` and new `memory_items`.

Documentation references:

- Use data contracts from `/Users/alexnewman/Downloads/claude-mem-handoff-docs/claude-mem-server-plan.md`.
- Copy repository style from `src/services/sqlite/*` and migration style from `src/services/sqlite/migrations/runner.ts`.

Verification:

- Add migration tests in `tests/services/sqlite/migration-runner.test.ts`.
- Add schema tests for fresh DB and upgraded DB.
- `bun test tests/services/sqlite/ tests/sqlite/`

Anti-pattern guards:

- Do not make Redis the source of truth for memories.
- Do not break existing `observations` search while adding `memory_items`.

## Phase 5: Queue Engine Boundary

What to implement:

- Add `src/server/queue/ObservationQueueEngine.ts` with an interface shaped around current behavior:

```ts
export interface ObservationQueueEngine {
  enqueue(sessionDbId: number, contentSessionId: string, message: PendingMessage): Promise<EnqueueResult>;
  createIterator(sessionDbId: number, signal: AbortSignal, onIdleTimeout?: () => void): AsyncIterableIterator<PendingMessageWithId>;
  clearPendingForSession(sessionDbId: number): Promise<number>;
  resetProcessingToPending(sessionDbId: number): Promise<number>;
  getPendingCount(sessionDbId: number): Promise<number>;
  getTotalQueueDepth(): Promise<number>;
  close(): Promise<void>;
}
```

- Implement `SqliteObservationQueueEngine` by wrapping `PendingMessageStore` and `SessionQueueProcessor`.
- Update `SessionManager` to depend on the interface instead of directly constructing `PendingMessageStore`.
- Clean up schema drift before BullMQ:
  - remove or restore `worker_pid` consistently;
  - reconcile `pending|processing` with stale `processed|failed` references;
  - remove or fix `storeObservationsAndMarkComplete()` dead-code writes.

Documentation references:

- Copy current semantics from `src/services/sqlite/PendingMessageStore.ts`.
- Copy async iterator behavior from `src/services/queue/SessionQueueProcessor.ts`.
- Preserve provider contract consumed by Claude/Gemini/OpenRouter providers through `SessionManager`.

Verification:

- Shared queue contract test suite.
- `bun test tests/services/sqlite/PendingMessageStore.test.ts tests/services/queue/SessionQueueProcessor.test.ts`
- Add tests for dedupe, FIFO, restart reset, idle timeout, and queue depth.

Anti-pattern guards:

- Do not model this as generic stateless jobs only. The current queue is a per-session stream feeding provider generators.
- Do not change `_persistentId` and `_originalTimestamp` semantics.

## Phase 6: BullMQ And Valkey Runtime

What to implement:

- Add dependencies:
  - `bullmq`
  - `ioredis` if BullMQ usage requires direct connection management beyond BullMQ exports.
- Add settings:
  - `CLAUDE_MEM_QUEUE_ENGINE=sqlite|bullmq`
  - `CLAUDE_MEM_REDIS_URL`
  - `CLAUDE_MEM_REDIS_HOST`
  - `CLAUDE_MEM_REDIS_PORT`
  - `CLAUDE_MEM_REDIS_MODE=external|managed|docker`
  - `CLAUDE_MEM_QUEUE_REDIS_PREFIX`
- Add `BullMqObservationQueueEngine`.
- Use one queue per active session at first, with effective concurrency `1`, to preserve per-session FIFO without BullMQ Pro groups.
- Use safe hashed job IDs:
  - observation: `obs_${sha256(contentSessionId + "\0" + toolUseId)}`
  - summarize: `sum_${sha256(contentSessionId + "\0" + createdAtEpoch + "\0" + messageKind)}`
- Store only queue payloads needed to resume processing. Keep memory records in SQLite.
- Add Redis health to `/api/health` and `server status` only when BullMQ is enabled.

Documentation references:

- BullMQ docs: `Queue.add(...)` stores jobs in Redis and workers can process later.
- BullMQ job IDs: duplicate custom IDs are ignored while the prior job still exists.
- BullMQ stalled jobs: active jobs are lock-renewed and moved back or failed when stalled.
- Copy configuration strategy from `plans/2026-05-06-redis-dependency-strategy.md`.

Verification:

- Unit tests with mocked BullMQ queue where possible.
- Integration tests gated by `CLAUDE_MEM_REDIS_URL`.
- Docker Compose test with Valkey:
  - enqueue, kill server, restart, process;
  - duplicate `tool_use_id` suppressed;
  - per-session FIFO;
  - stalled job returns or fails as configured.

Anti-pattern guards:

- Do not use one high-concurrency global queue until same-session ordering is proven.
- Do not silently drop messages if Redis is unavailable.
- Do not use `:` in custom job IDs.

## Phase 7: Auth, Teams, And API Keys

What to implement:

- Add Better Auth after Phase 3 server bootstrap is complete.
- Add auth dependencies:
  - `better-auth`
  - `@better-auth/api-key`
- Create `src/server/auth/auth.ts` with:
  - Better Auth core config;
  - API-key plugin;
  - organization plugin with teams enabled;
  - custom access statements for projects and memories.
- Create `src/server/middleware/auth.ts`:
  - read `Authorization: Bearer <key>`;
  - verify API keys with Better Auth;
  - attach `authContext` containing `userId`, `organizationId`, `teamId`, scopes, and key id;
  - allow localhost unauthenticated reads only if `CLAUDE_MEM_AUTH_MODE=local-dev`.
- Create CLI commands:
  - `claude-mem server api-key create --team <team> --scope memories:read,memories:write`
  - `claude-mem server api-key list`
  - `claude-mem server api-key revoke <id>`
- Add team/project scoping to memory storage and retrieval.
- Add audit rows when memories are served, written, forgotten, imported, or exported.

Documentation references:

- Better Auth API Key plugin supports create, manage, verify, rate limiting, permissions, metadata, custom prefixes, and organization-owned keys.
- Better Auth Organization plugin supports organizations, members, teams, roles, permissions, and `hasPermission`.
- Better Auth Express integration requires the auth handler before body parser.

Verification:

- Auth route tests:
  - unauthenticated write denied;
  - API key with read scope cannot write;
  - revoked key denied;
  - team A key cannot read team B memory;
  - local-only mode does not bind publicly.
- `bun test tests/server/ tests/worker/middleware/`

Anti-pattern guards:

- Do not add SSO/SAML/SCIM in v0.1.
- Do not make API keys plaintext in SQLite. Store only hashed keys and show the raw key once at creation.
- Do not expose memory over LAN by default.

## Phase 8: REST API V1

What to implement:

- Add `src/server/routes/v1/*`:
  - `GET /healthz`
  - `GET /v1/info`
  - `GET /v1/projects`
  - `POST /v1/projects`
  - `GET /v1/projects/:id`
  - `POST /v1/sessions/start`
  - `POST /v1/sessions/:id/end`
  - `GET /v1/sessions/:id`
  - `POST /v1/events`
  - `POST /v1/events/batch`
  - `GET /v1/events/:id`
  - `POST /v1/memories`
  - `GET /v1/memories/:id`
  - `PATCH /v1/memories/:id`
  - `POST /v1/memories/:id/supersede`
  - `POST /v1/forget`
  - `POST /v1/search`
  - `POST /v1/context`
  - `GET /v1/audit`
  - `POST /v1/export`
  - `POST /v1/import`
  - `POST /v1/reindex`
- Keep legacy `/api/*` routes for current hooks, MCP, and viewer.
- Add OpenAPI generation from Zod schemas using existing `zod-to-json-schema`, or add a focused OpenAPI helper only if needed.

Documentation references:

- Copy route class and validation patterns from existing worker route files.
- Copy endpoint list from `claude-mem-server-plan.md`.

Verification:

- Add REST integration tests under `tests/server/v1/`.
- Add OpenAPI snapshot/schema tests.
- Legacy smoke: current `/api/sessions/init`, `/api/sessions/observations`, `/api/search`, `/api/context/inject` still work.

Anti-pattern guards:

- Do not delete `/api/*` compatibility routes in this phase.
- Do not implement MCP as a separate memory stack.

## Phase 9: Adapter Migration

What to implement:

- Add `src/adapters/claude-code/mapper.ts` to map existing hook payloads to `AgentEvent`.
- Add `src/adapters/generic-rest/examples.ts` with Codex/OpenCode/OpenClaw/custom examples.
- Refactor `SessionRoutes` ingestion to call the same event-ingestion service used by `POST /v1/events`.
- Preserve current hook fields:
  - `contentSessionId`
  - `tool_name`
  - `tool_input`
  - `tool_response`
  - `cwd`
  - `agentId`
  - `agentType`
  - `platformSource`
  - `tool_use_id` / `toolUseId`

Documentation references:

- Copy field handling from `src/services/worker/http/routes/SessionRoutes.ts`.
- Copy platform normalization from `src/shared/platform-source.ts`.

Verification:

- Existing hook tests continue passing.
- Add mapper tests for Claude Code, Codex transcript watcher, and generic REST event payloads.

Anti-pattern guards:

- Do not make Claude Code the core data model.
- Do not throw away raw event payloads before redaction/classification decisions are applied.

## Phase 10: MCP Surface On Server Core

What to implement:

- Add `src/server/mcp/tools.ts`, `resources.ts`, `prompts.ts`, and `register.ts`.
- Keep existing `src/servers/mcp-server.ts` as a thin stdio entrypoint.
- Implement tools:
  - `memory_add`
  - `memory_search`
  - `memory_context`
  - `memory_forget`
  - `memory_list_recent`
  - `memory_record_decision`
- Keep existing search/timeline/get-observations tools during migration.

Documentation references:

- Copy low-level SDK usage from `src/servers/mcp-server.ts`.
- Use MCP tool schema tests from `tests/servers/mcp-tool-schemas.test.ts`.

Verification:

- MCP list/call tests for new tools.
- Build guard in `scripts/build-hooks.js` still prevents Bun-only worker code from bloating the MCP bundle.

Anti-pattern guards:

- Do not import Bun-only SQLite/worker internals into the Node MCP bundle.
- Do not bypass auth/team scoping in MCP tools.

## Phase 11: Docker Deployment

What to implement:

- Add `docker/server/Dockerfile` or update `docker/claude-mem/Dockerfile` for server mode.
- Add `docker-compose.yml` with services:
  - `claude-mem-server`
  - `valkey`
  - optional `chroma` if Chroma remains enabled in server profile
- Server container defaults:
  - `CLAUDE_MEM_HOST=0.0.0.0` inside container
  - published port explicitly configured by compose
  - `CLAUDE_MEM_QUEUE_ENGINE=bullmq`
  - `CLAUDE_MEM_REDIS_URL=redis://valkey:6379`
  - persisted `/data/claude-mem`
- Add healthcheck using `GET /healthz`.
- Keep local auth credential mounting patterns from current Docker docs.

Documentation references:

- Copy Bun/uv/Claude Code install style from `docker/claude-mem/Dockerfile`.
- Copy credential handling conventions from `docker/claude-mem/README.md` and `entrypoint.sh`.
- Copy Valkey config guidance from `plans/2026-05-06-redis-dependency-strategy.md`.

Verification:

- `docker compose up --build`
- `curl http://127.0.0.1:<port>/healthz`
- `curl -H "Authorization: Bearer <key>" http://127.0.0.1:<port>/v1/info`
- Kill/restart server container and verify queued events survive.

Anti-pattern guards:

- Do not auto-install Docker Desktop.
- Do not bind public host ports without documented auth.
- Do not run Redis/Valkey without persistence in deployable examples.

## Phase 12: Docs And Migration

What to implement:

- Add:
  - `docs/server.md`
  - `docs/api.md`
  - `docs/adapters.md`
  - `docs/security.md`
  - `docs/docker.md`
  - `docs/migration-worker-to-server.md`
- Update README to introduce Claude-Mem Server first and worker as compatibility language.
- Document:
  - local dev mode;
  - Docker deployment;
  - API-key creation;
  - team/project scoping;
  - generic agent ingestion;
  - queue engine settings;
  - privacy/redaction baseline.

Documentation references:

- Use public docs style from `docs/public/*.mdx`.
- Use handoff docs for product wording and explicit non-goals.

Verification:

- `rg -n "worker service|Worker Service|worker-first" README.md docs`
- Ensure docs still mention compatibility commands where needed.

Anti-pattern guards:

- Do not imply hosted cloud or enterprise features are available.
- Do not call Claude-Mem an official Anthropic project.

## Final Verification Phase

Run:

```sh
npm run typecheck:root
bun test tests/server/ tests/services/queue/ tests/services/sqlite/ tests/servers/
bun test tests/integration/worker-api-endpoints.test.ts
npm run build
docker compose up --build
```

Manual acceptance checklist:

- `npx claude-mem install` still works.
- `claude-mem server start|status|stop` works.
- `claude-mem worker start|status|stop` aliases work.
- Existing Claude Code hooks still write observations.
- Generic REST client can write and search memory.
- MCP tools use the same server/core logic.
- API-key protected writes fail without auth.
- Team-scoped search cannot cross team boundaries.
- BullMQ/Valkey mode survives server container restart.
- SQLite remains canonical source of memory truth.
- Apache-2.0 migration is complete and stale AGPL messaging is removed from public package/docs.

## Suggested Execution Order

1. Phase 1: Apache-2.0 boundary.
2. Phase 2: CLI namespace and aliases.
3. Phase 3: Server bootstrap refactor.
4. Phase 5: Queue boundary and SQLite contract cleanup.
5. Phase 6: BullMQ/Valkey backend.
6. Phase 4: New core/storage contracts.
7. Phase 7: Auth/team/API-key layer.
8. Phase 8: REST V1.
9. Phase 9: Adapter migration.
10. Phase 10: MCP server-core surface.
11. Phase 11: Docker deployment.
12. Phase 12: Docs and migration guide.

The order intentionally moves the middleware and queue boundaries before Better Auth and REST V1. Those two boundaries are the highest-risk coupling points in the current codebase.