feat: Implement new memory processing prompts and XML structures
- Added final finalize prompt for session summary generation with required XML fields. - Introduced recommended prompt flow with structured observation format and hierarchical storage principles. - Created final init prompt for processing tool executions with clear guidelines on when to store observations. - Developed final observation prompt for analyzing tool outputs and generating structured observations. - Migrated old prompt flow to a new system with improved clarity and structured data handling. - Updated parser and storage mechanisms to accommodate new observation formats and fields. - Enhanced documentation for new prompts and their usage in memory processing sessions.
This commit is contained in:
Alex Newman
@@ -0,0 +1,444 @@
|
||||
# Prompt Flow Analysis & Rankings
|
||||
|
||||
## Rating System
|
||||
- ✅ **Smart**: Well-designed, clear purpose, effective
|
||||
- ⚠️ **Problematic**: Has issues but salvageable
|
||||
- ❌ **Stupid**: Poorly designed, confusing, or counterproductive
|
||||
- 🧠 **Context Poison**: Will confuse the AI or create inconsistent behavior
|
||||
- 🔍 **No Clear Purpose**: Exists but unclear why
|
||||
- 🎯 **Clarity Score**: 1-10 (10 = crystal clear, 1 = incomprehensible)
|
||||
|
||||
---
|
||||
|
||||
## Element-by-Element Comparison
|
||||
|
||||
### INIT PROMPTS (Session Start)
|
||||
|
||||
#### CURRENT: "You are a memory processor"
|
||||
```
|
||||
You will PROCESS tool executions during this Claude Code session. Your job is to:
|
||||
1. ANALYZE each tool response for meaningful content
|
||||
2. DECIDE whether it contains something worth storing
|
||||
3. EXTRACT the key insight
|
||||
4. STORE it as an observation in the XML format below
|
||||
|
||||
For MOST meaningful tool outputs, you should generate an observation. Only skip truly routine operations.
|
||||
```
|
||||
|
||||
**Rating**: ❌ **Stupid** + 🧠 **Context Poison**
|
||||
**Clarity**: 3/10
|
||||
|
||||
**Issues**:
|
||||
1. "For MOST" is ambiguous - does that mean 51%? 80%? 95%?
|
||||
2. Creates bias toward over-storage (fear of missing things)
|
||||
3. Contradicts "Only skip truly routine operations" later in prompt
|
||||
4. No clear guidance on what "meaningful" actually means
|
||||
5. "Only skip truly routine" implies almost everything should be stored
|
||||
|
||||
**Why Context Poison**:
|
||||
- Agent will second-guess every decision
|
||||
- Creates inconsistent thresholds across sessions
|
||||
- User gets frustrated with noise
|
||||
|
||||
---
|
||||
|
||||
#### OLD: "You are a semantic memory compressor"
|
||||
```
|
||||
## FIRST: Generate Session Title
|
||||
IMMEDIATELY generate a title and subtitle for this session based on the user request.
|
||||
|
||||
## THEN: Process Tool Responses
|
||||
You will receive a stream of tool responses. For each one:
|
||||
1. ANALYZE: Does this contain information worth remembering?
|
||||
2. DECIDE: Should I store this or skip it?
|
||||
3. EXTRACT: What are the key semantic concepts?
|
||||
4. DECOMPOSE: Break into title + subtitle + atomic facts + narrative
|
||||
5. STORE: Use bash to save the hierarchical memory
|
||||
6. TRACK: Keep count of stored memories (001, 002, 003...)
|
||||
|
||||
# IMPORTANT REMINDERS
|
||||
- Be selective - quality over quantity
|
||||
```
|
||||
|
||||
**Rating**: ⚠️ **Problematic** but contains ✅ **Smart** elements
|
||||
**Clarity**: 6/10
|
||||
|
||||
**Issues**:
|
||||
1. "IMMEDIATELY" vs "THEN" creates ordering confusion
|
||||
2. Session title generation is unclear when it should happen
|
||||
3. Bash tool dependency is fragile
|
||||
4. Manual counter tracking is error-prone
|
||||
|
||||
**Smart Elements**:
|
||||
1. "Quality over quantity" is clear directive
|
||||
2. Hierarchical decomposition gives structure
|
||||
3. Explicit state tracking (counter)
|
||||
4. "Be selective" is unambiguous
|
||||
|
||||
**Verdict**: The philosophy is better (selective, quality-focused), execution is messier (bash commands, ordering confusion)
|
||||
|
||||
---
|
||||
|
||||
### OBSERVATION PROMPTS
|
||||
|
||||
#### CURRENT: buildObservationPrompt
|
||||
```
|
||||
ANALYSIS TASK
|
||||
-------------
|
||||
ANALYZE this tool response and DECIDE: Does it contain something worth storing?
|
||||
|
||||
Most Read, Edit, Grep, Bash, and Write operations contain meaningful content.
|
||||
|
||||
If this contains something worth remembering, output the observation...
|
||||
```
|
||||
|
||||
**Rating**: ❌ **Stupid** + 🧠 **Context Poison**
|
||||
**Clarity**: 4/10
|
||||
|
||||
**Issues**:
|
||||
1. "Most Read, Edit, Grep, Bash, and Write operations contain meaningful content" - Why repeat this per observation?
|
||||
2. Contradicts init prompt's "be selective" guidance
|
||||
3. Creates bias: agent thinks "well, this is a Read, so I should probably store it"
|
||||
4. No guidance on WHAT to extract from the content
|
||||
5. One-sentence observations lose context
|
||||
|
||||
**Why Context Poison**:
|
||||
- Tool-type bias ("It's a Read, so store it") instead of content-based evaluation
|
||||
- Encourages lazy pattern matching instead of semantic analysis
|
||||
- Results in garbage like "Read package.json file" with no insight
|
||||
|
||||
---
|
||||
|
||||
#### OLD: Tool Message
|
||||
```
|
||||
# Tool Response {timeFormatted}
|
||||
|
||||
Tool: {toolName}
|
||||
User Context: "{userPrompt}"
|
||||
|
||||
```
|
||||
{toolResponse}
|
||||
```
|
||||
|
||||
Analyze and store if meaningful.
|
||||
```
|
||||
|
||||
**Rating**: ✅ **Smart** (minimal, non-leading)
|
||||
**Clarity**: 8/10
|
||||
|
||||
**Smart Elements**:
|
||||
1. Doesn't tell agent what to think about tool types
|
||||
2. Trusts agent's judgment from system prompt
|
||||
3. Short and clear
|
||||
4. Includes user context for relevance filtering
|
||||
|
||||
**Issue**:
|
||||
1. Maybe TOO minimal - no reminder of format requirements
|
||||
|
||||
---
|
||||
|
||||
### STORAGE FORMATS
|
||||
|
||||
#### CURRENT: XML Observations
|
||||
```xml
|
||||
<observation>
|
||||
<type>feature</type>
|
||||
<text>Implemented JWT token refresh flow with 7-day expiry</text>
|
||||
</observation>
|
||||
```
|
||||
|
||||
**Rating**: ⚠️ **Problematic**
|
||||
**Clarity**: 7/10
|
||||
|
||||
**Issues**:
|
||||
1. One sentence only - loses narrative context
|
||||
2. Five types (decision, bugfix, feature, refactor, discovery) - are these actually useful categories?
|
||||
3. No file associations
|
||||
4. No concept tagging
|
||||
5. Flat structure - all observations equal weight
|
||||
|
||||
**Smart Elements**:
|
||||
1. Simple to parse
|
||||
2. Structured typing
|
||||
3. Regex-parseable
|
||||
|
||||
---
|
||||
|
||||
#### OLD: Hierarchical Memory (4 levels)
|
||||
```bash
|
||||
--title "SDK Transcript Auto-Cleanup"
|
||||
--subtitle "Automatic deletion of SDK transcripts after completion prevents memory conversations from appearing in UI history"
|
||||
--facts '["stop-streaming.js: Deletes SDK transcript after overview generation", "Path: ~/.claude/projects/{sanitized-cwd}/{sessionId}.jsonl"]'
|
||||
--concepts '["cleanup", "SDK-lifecycle", "UX"]'
|
||||
--files '["hooks/stop-streaming.js"]'
|
||||
```
|
||||
|
||||
**Rating**: ✅ **Smart** (structure) but ❌ **Stupid** (execution via bash)
|
||||
**Clarity**: 8/10 (concept), 3/10 (implementation)
|
||||
|
||||
**Smart Elements**:
|
||||
1. Multiple levels of granularity (title → subtitle → facts → narrative)
|
||||
2. Atomic facts enable precise retrieval
|
||||
3. File associations explicit
|
||||
4. Concept tags for categorization
|
||||
5. Subtitle gives the "why it matters"
|
||||
|
||||
**Stupid Elements**:
|
||||
1. Bash command execution is fragile
|
||||
2. Quote escaping nightmare
|
||||
3. Manual counter tracking
|
||||
4. JSON in bash arguments is error-prone
|
||||
|
||||
**Verdict**: Great data model, terrible implementation
|
||||
|
||||
---
|
||||
|
||||
### SUMMARY/FINALIZE PROMPTS
|
||||
|
||||
#### CURRENT: buildFinalizePrompt (per prompt)
|
||||
```xml
|
||||
<summary>
|
||||
<request>Implement JWT authentication system</request>
|
||||
<investigated>Existing auth middleware, session management</investigated>
|
||||
<learned>Current system uses session cookies; no JWT support</learned>
|
||||
<completed>Implemented JWT token + refresh flow</completed>
|
||||
<next_steps>Add token revocation API endpoint</next_steps>
|
||||
<files_read><file>src/auth.ts</file></files_read>
|
||||
<files_edited><file>src/auth.ts</file></files_edited>
|
||||
<notes>Token secret stored in .env</notes>
|
||||
</summary>
|
||||
```
|
||||
|
||||
**Rating**: ✅ **Smart** (structure) but 🔍 **No Clear Purpose** (frequency)
|
||||
**Clarity**: 9/10
|
||||
|
||||
**Smart Elements**:
|
||||
1. Structured format with clear fields
|
||||
2. Tracks what was learned (semantic value)
|
||||
3. Files read/edited tracked explicitly
|
||||
4. Next steps captured
|
||||
|
||||
**Issues**:
|
||||
1. Generated PER PROMPT - is this too granular?
|
||||
2. Will create many summaries per session
|
||||
3. Unclear how these summaries are used
|
||||
4. No aggregation across prompts
|
||||
|
||||
**Question**: Should this be per-session instead of per-prompt?
|
||||
|
||||
---
|
||||
|
||||
#### OLD: Session Overview (per session)
|
||||
```bash
|
||||
claude-mem store-overview --project "{project}" --session "{sessionId}" --content "2-3 sentence overview"
|
||||
```
|
||||
|
||||
**Rating**: ⚠️ **Problematic**
|
||||
**Clarity**: 5/10
|
||||
|
||||
**Issues**:
|
||||
1. Only 2-3 sentences - very lossy
|
||||
2. No structured fields
|
||||
3. Happens once at end - loses per-prompt context
|
||||
4. Relies on agent's memory of entire session
|
||||
|
||||
**Smart Element**:
|
||||
1. One overview per session (not noisy)
|
||||
|
||||
---
|
||||
|
||||
### DECISION GUIDANCE
|
||||
|
||||
#### CURRENT: What to Store/Skip
|
||||
```
|
||||
Store these:
|
||||
✓ File contents with logic, algorithms, or patterns
|
||||
✓ Search results revealing project structure
|
||||
✓ Build errors or test failures with context
|
||||
...
|
||||
|
||||
Skip these:
|
||||
✗ Simple status checks (git status with no changes)
|
||||
✗ Trivial edits (one-line config changes)
|
||||
...
|
||||
```
|
||||
|
||||
**Rating**: ✅ **Smart**
|
||||
**Clarity**: 8/10
|
||||
|
||||
**Smart Elements**:
|
||||
1. Concrete examples
|
||||
2. Both positive and negative cases
|
||||
3. Action-oriented
|
||||
|
||||
**Issue**:
|
||||
1. Contradicted by "For MOST" and "Most Read, Edit..." statements elsewhere
|
||||
|
||||
---
|
||||
|
||||
#### OLD: What to Store/Skip
|
||||
```
|
||||
Store these:
|
||||
- File contents with logic, algorithms, or patterns
|
||||
- Search results revealing project structure
|
||||
...
|
||||
|
||||
Skip these:
|
||||
- Simple status checks (git status with no changes)
|
||||
- Trivial edits (one-line config changes)
|
||||
- Binary data or noise
|
||||
- Anything without semantic value
|
||||
```
|
||||
|
||||
**Rating**: ✅ **Smart**
|
||||
**Clarity**: 8/10
|
||||
|
||||
**Same as current**, which is good.
|
||||
|
||||
---
|
||||
|
||||
## CRITICAL ISSUES RANKED
|
||||
|
||||
### 1. "For MOST meaningful tool outputs" - 🧠 **CONTEXT POISON #1**
|
||||
**Severity**: CRITICAL
|
||||
**Impact**: Destroys selectivity, fills DB with noise
|
||||
**Fix**: Remove entirely. Replace with: "Be selective. Only store if it reveals important information about the codebase."
|
||||
|
||||
---
|
||||
|
||||
### 2. "Most Read, Edit, Grep, Bash, and Write operations contain meaningful content" - 🧠 **CONTEXT POISON #2**
|
||||
**Severity**: CRITICAL
|
||||
**Impact**: Creates tool-type bias instead of content-based evaluation
|
||||
**Fix**: Remove entirely. It's redundant and harmful.
|
||||
|
||||
---
|
||||
|
||||
### 3. One-sentence observations lose context - ❌ **STUPID**
|
||||
**Severity**: HIGH
|
||||
**Impact**: Can't understand observation without narrative
|
||||
**Fix**: Add narrative field to observations (like old system)
|
||||
|
||||
---
|
||||
|
||||
### 4. No hierarchical structure in current system - ❌ **STUPID**
|
||||
**Severity**: HIGH
|
||||
**Impact**: Can't do granular retrieval (fact-level vs narrative-level)
|
||||
**Fix**: Adopt 4-level hierarchy from old system
|
||||
|
||||
---
|
||||
|
||||
### 5. Bash command execution in old system - ❌ **STUPID**
|
||||
**Severity**: HIGH
|
||||
**Impact**: Fragile, error-prone, quote-escaping nightmare
|
||||
**Fix**: Keep current approach (XML parsing + direct DB writes)
|
||||
|
||||
---
|
||||
|
||||
### 6. Manual memory counter in old system - ⚠️ **PROBLEMATIC**
|
||||
**Severity**: MEDIUM
|
||||
**Impact**: Agent forgets, skips numbers, duplicates
|
||||
**Fix**: Auto-increment in database (current approach)
|
||||
|
||||
---
|
||||
|
||||
### 7. Per-prompt summaries unclear purpose - 🔍 **NO CLEAR PURPOSE**
|
||||
**Severity**: MEDIUM
|
||||
**Impact**: Creates many summaries, unclear how they're used
|
||||
**Fix**: Decide: per-session summary only, or per-prompt with aggregation?
|
||||
|
||||
---
|
||||
|
||||
### 8. Five observation types unclear value - 🔍 **NO CLEAR PURPOSE**
|
||||
**Severity**: LOW
|
||||
**Impact**: Are these categories actually useful for retrieval?
|
||||
**Fix**: Evaluate if types should be: (1) kept as-is, (2) expanded, (3) removed
|
||||
|
||||
---
|
||||
|
||||
## BEST ELEMENTS FROM EACH SYSTEM
|
||||
|
||||
### From OLD System (Keep These)
|
||||
1. ✅ 4-level hierarchy (title → subtitle → facts → narrative)
|
||||
2. ✅ "Be selective - quality over quantity"
|
||||
3. ✅ Atomic facts (50-150 char, self-contained, no pronouns)
|
||||
4. ✅ Concept tagging
|
||||
5. ✅ File associations
|
||||
6. ✅ Minimal observation prompts (don't bias agent)
|
||||
|
||||
### From CURRENT System (Keep These)
|
||||
1. ✅ XML parsing (not bash commands)
|
||||
2. ✅ Auto-increment IDs (not manual counters)
|
||||
3. ✅ Structured summary format (8 fields)
|
||||
4. ✅ Per-prompt tracking
|
||||
5. ✅ Foreign key integrity
|
||||
6. ✅ Typed observations (decision/bugfix/feature/refactor/discovery)
|
||||
|
||||
### From NEITHER System (Add These)
|
||||
1. Clear threshold guidance: "Only store if it reveals important information about the codebase"
|
||||
2. Explicit narrative field in observations
|
||||
3. Vector embeddings for semantic search (current stores in SQLite only)
|
||||
|
||||
---
|
||||
|
||||
## RECOMMENDED HYBRID SYSTEM
|
||||
|
||||
### Storage Format: Hierarchical Observations (XML)
|
||||
```xml
|
||||
<observation>
|
||||
<type>feature</type>
|
||||
<title>JWT Token Refresh Implementation</title>
|
||||
<subtitle>Added 7-day refresh token rotation with Redis storage</subtitle>
|
||||
<facts>
|
||||
<fact>src/auth.ts: refreshToken() generates new JWT with 7-day expiry</fact>
|
||||
<fact>Redis key format: refresh:{userId}:{tokenId} with TTL 604800s</fact>
|
||||
<fact>Old token invalidated on refresh to prevent replay attacks</fact>
|
||||
</facts>
|
||||
<narrative>Implemented JWT refresh token functionality in src/auth.ts. The refreshToken() function validates the old refresh token from Redis, generates a new JWT access token (7-day expiry) and new refresh token, stores the new refresh token in Redis with key format refresh:{userId}:{tokenId} and TTL of 604800 seconds (7 days), and invalidates the old refresh token to prevent replay attacks. This enables long-lived authenticated sessions without requiring users to re-login while maintaining security through token rotation.</narrative>
|
||||
<concepts>
|
||||
<concept>authentication</concept>
|
||||
<concept>security</concept>
|
||||
<concept>session-management</concept>
|
||||
</concepts>
|
||||
<files>
|
||||
<file>src/auth.ts</file>
|
||||
<file>src/middleware/auth.ts</file>
|
||||
</files>
|
||||
</observation>
|
||||
```
|
||||
|
||||
### Guidance: Clear and Unambiguous
|
||||
```
|
||||
Be selective. Only store observations when the tool output reveals important information about:
|
||||
- Architecture or design patterns
|
||||
- Implementation details of features or bug fixes
|
||||
- System state or configuration
|
||||
- Business logic or algorithms
|
||||
|
||||
Skip routine operations like empty git status, simple npm installs, or trivial config changes.
|
||||
|
||||
Each observation should be self-contained and searchable.
|
||||
```
|
||||
|
||||
### Summary: Per-Session (Not Per-Prompt)
|
||||
- Generate ONE summary when session ends
|
||||
- Aggregate all observations from session
|
||||
- Use current structured format (request, investigated, learned, completed, next_steps, files_read, files_edited, notes)
|
||||
|
||||
---
|
||||
|
||||
## FINAL VERDICT
|
||||
|
||||
| Element | Current | Old | Winner |
|
||||
|---------|---------|-----|--------|
|
||||
| **Storage Structure** | Flat one-sentence | 4-level hierarchy | **OLD** |
|
||||
| **Storage Implementation** | XML parsing | Bash commands | **CURRENT** |
|
||||
| **Decision Guidance** | Contradictory | Clear | **OLD** |
|
||||
| **Session Metadata** | None | Title + subtitle | **OLD** |
|
||||
| **Per-Prompt Tracking** | Yes (summaries) | No | **CURRENT** |
|
||||
| **Semantic Search** | No | Yes (ChromaDB) | **OLD** |
|
||||
| **Observation Prompts** | Biased, repetitive | Minimal, clear | **OLD** |
|
||||
| **Auto-Increment IDs** | Yes | No (manual) | **CURRENT** |
|
||||
| **File Associations** | No | Yes | **OLD** |
|
||||
| **Concept Tagging** | No | Yes | **OLD** |
|
||||
|
||||
**Optimal System**: Hybrid - Old system's data model + Current system's implementation approach
|
||||
@@ -0,0 +1,41 @@
|
||||
# Draft Finalize Prompt
|
||||
|
||||
```
|
||||
SESSION ENDING
|
||||
==============
|
||||
This Claude Code session is completing.
|
||||
|
||||
TASK
|
||||
----
|
||||
Review the observations you generated and create a session summary.
|
||||
|
||||
Output this XML:
|
||||
|
||||
```xml
|
||||
<summary>
|
||||
<request>What did the user request?</request>
|
||||
<investigated>What code and systems did you explore?</investigated>
|
||||
<learned>What did you learn about the codebase?</learned>
|
||||
<completed>What was accomplished in this session?</completed>
|
||||
<next_steps>What should be done next?</next_steps>
|
||||
<files_read>
|
||||
<file>src/auth.ts</file>
|
||||
<file>src/middleware/session.ts</file>
|
||||
</files_read>
|
||||
<files_edited>
|
||||
<file>src/auth.ts</file>
|
||||
</files_edited>
|
||||
<notes>Additional insights or context</notes>
|
||||
</summary>
|
||||
```
|
||||
|
||||
REQUIREMENTS
|
||||
------------
|
||||
All 8 fields are required: request, investigated, learned, completed, next_steps, files_read, files_edited, notes
|
||||
|
||||
Files must be wrapped in <file> tags
|
||||
|
||||
If no files were read/edited, use empty tags: <files_read></files_read>
|
||||
|
||||
Focus on semantic insights, not mechanical details.
|
||||
```
|
||||
@@ -0,0 +1,92 @@
|
||||
# Draft Init Prompt
|
||||
|
||||
```
|
||||
You are a memory processor for the "{project}" project.
|
||||
|
||||
SESSION CONTEXT
|
||||
---------------
|
||||
Session ID: {sessionId}
|
||||
User's Goal: {userPrompt}
|
||||
Date: {date}
|
||||
|
||||
YOUR ROLE
|
||||
---------
|
||||
Process tool executions from this Claude Code session and store important observations.
|
||||
|
||||
WHEN TO STORE
|
||||
-------------
|
||||
Store an observation when the tool output reveals significant information about:
|
||||
- Implementation of features or bug fixes
|
||||
- Architecture, design patterns, or system structure
|
||||
- Configuration, environment, or deployment details
|
||||
- Algorithms, business logic, or data flows
|
||||
- Errors, failures, or debugging insights
|
||||
|
||||
WHEN TO SKIP
|
||||
------------
|
||||
Skip routine operations:
|
||||
- Empty status checks (git status with no changes)
|
||||
- Package installations with no errors
|
||||
- Simple file listings
|
||||
- Repetitive operations you've already documented
|
||||
|
||||
OBSERVATION FORMAT
|
||||
------------------
|
||||
Output observations using this XML structure:
|
||||
|
||||
```xml
|
||||
<observation>
|
||||
<type>feature</type>
|
||||
<title>JWT Refresh Token Implementation</title>
|
||||
<subtitle>Added token rotation with Redis storage for secure sessions without re-login</subtitle>
|
||||
<facts>
|
||||
<fact>src/auth.ts: refreshToken() generates new JWT with 7-day expiry</fact>
|
||||
<fact>Redis stores tokens as refresh:{userId}:{tokenId} with 604800s TTL</fact>
|
||||
<fact>Old token invalidated on refresh to prevent replay attacks</fact>
|
||||
</facts>
|
||||
<narrative>Implemented JWT refresh token functionality in src/auth.ts. The refreshToken() function validates the old refresh token from Redis, generates a new JWT access token with 7-day expiry and new refresh token, stores the new refresh token in Redis using the key format refresh:{userId}:{tokenId} with TTL of 604800 seconds, and invalidates the old refresh token to prevent replay attacks. This enables long-lived authenticated sessions without requiring users to re-login while maintaining security through token rotation.</narrative>
|
||||
<concepts>
|
||||
<concept>authentication</concept>
|
||||
<concept>security</concept>
|
||||
<concept>session-management</concept>
|
||||
</concepts>
|
||||
<files>
|
||||
<file>src/auth.ts</file>
|
||||
<file>src/middleware/auth.ts</file>
|
||||
</files>
|
||||
</observation>
|
||||
```
|
||||
|
||||
FIELD REQUIREMENTS
|
||||
------------------
|
||||
|
||||
**type**: One of: decision, bugfix, feature, refactor, discovery
|
||||
|
||||
**title**: 3-8 words capturing the core action
|
||||
Examples: "JWT Refresh Token Implementation", "Database Connection Pool Fix"
|
||||
|
||||
**subtitle**: One sentence (max 24 words) explaining the significance
|
||||
Focus on outcome or benefit
|
||||
Examples: "Added token rotation with Redis storage for secure sessions without re-login"
|
||||
|
||||
**facts**: 3-7 specific, searchable statements (each 50-150 chars)
|
||||
Each fact is ONE piece of information
|
||||
Include filename or component name
|
||||
No pronouns - each fact must stand alone
|
||||
Examples:
|
||||
- "src/auth.ts: refreshToken() generates new JWT with 7-day expiry"
|
||||
- "Redis stores tokens as refresh:{userId}:{tokenId} with 604800s TTL"
|
||||
|
||||
**narrative**: Full explanation (200-400 words)
|
||||
What was done, how it works, why it matters
|
||||
Technical details: files, functions, data structures
|
||||
|
||||
**concepts**: 2-5 broad categories
|
||||
Examples: "authentication", "caching", "error-handling", "performance"
|
||||
|
||||
**files**: All files touched
|
||||
Full paths from project root
|
||||
Examples: "src/auth.ts", "tests/auth.test.ts"
|
||||
|
||||
Ready to process tool executions.
|
||||
```
|
||||
@@ -0,0 +1,21 @@
|
||||
# Draft Observation Prompt
|
||||
|
||||
```
|
||||
TOOL OBSERVATION
|
||||
================
|
||||
Tool: {tool_name}
|
||||
Time: {timestamp}
|
||||
Prompt: {prompt_number}
|
||||
|
||||
Input:
|
||||
{tool_input JSON}
|
||||
|
||||
Output:
|
||||
{tool_output JSON}
|
||||
|
||||
TASK
|
||||
----
|
||||
Analyze this tool output. If it contains significant information about the codebase, generate an observation using the XML format from the init prompt.
|
||||
|
||||
If this is routine or repetitive, you can skip it.
|
||||
```
|
||||
@@ -0,0 +1,286 @@
|
||||
# Current Prompt Flow (SDK System)
|
||||
|
||||
## Architecture Overview
|
||||
- **System**: SDK Agent (persistent HTTP service via PM2)
|
||||
- **Storage**: SQLite (observations + summaries per prompt)
|
||||
- **Hooks**: Context (START), Summary (STOP)
|
||||
|
||||
---
|
||||
|
||||
## Flow Timeline
|
||||
|
||||
### 1. SESSION START (context-hook.js)
|
||||
|
||||
**Trigger**: Claude Code session starts
|
||||
**Hook**: `user-prompt-submit`
|
||||
|
||||
**Actions**:
|
||||
1. Create SDK session in database
|
||||
2. Initialize HTTP worker (if not running)
|
||||
3. Send init request to worker
|
||||
4. Worker starts SDK agent subprocess
|
||||
|
||||
**Init Prompt Sent to SDK**:
|
||||
```
|
||||
You are a memory processor for the "{project}" project.
|
||||
|
||||
SESSION CONTEXT
|
||||
---------------
|
||||
Session ID: {sessionId}
|
||||
User's Goal: {userPrompt}
|
||||
Date: {date}
|
||||
|
||||
YOUR ROLE
|
||||
---------
|
||||
You will PROCESS tool executions during this Claude Code session. Your job is to:
|
||||
|
||||
1. ANALYZE each tool response for meaningful content
|
||||
2. DECIDE whether it contains something worth storing
|
||||
3. EXTRACT the key insight
|
||||
4. STORE it as an observation in the XML format below
|
||||
|
||||
For MOST meaningful tool outputs, you should generate an observation. Only skip truly routine operations.
|
||||
|
||||
WHAT TO STORE
|
||||
--------------
|
||||
Store these:
|
||||
✓ File contents with logic, algorithms, or patterns
|
||||
✓ Search results revealing project structure
|
||||
✓ Build errors or test failures with context
|
||||
✓ Code revealing architecture or design decisions
|
||||
✓ Git diffs with significant changes
|
||||
✓ Command outputs showing system state
|
||||
✓ Bug fixes (e.g., "fixed race condition in auth middleware by adding mutex")
|
||||
✓ New features (e.g., "implemented JWT refresh token flow")
|
||||
✓ Refactorings (e.g., "extracted validation logic into separate service")
|
||||
✓ Discoveries (e.g., "found that API rate limit is 100 req/min")
|
||||
|
||||
WHAT TO SKIP
|
||||
------------
|
||||
Skip these:
|
||||
✗ Simple status checks (git status with no changes)
|
||||
✗ Trivial edits (one-line config changes)
|
||||
✗ Repeated operations
|
||||
✗ Anything without semantic value
|
||||
|
||||
HOW TO STORE OBSERVATIONS
|
||||
--------------------------
|
||||
When you identify something worth remembering, output your observation in this EXACT XML format:
|
||||
|
||||
```xml
|
||||
<observation>
|
||||
<type>feature</type>
|
||||
<text>Implemented JWT token refresh flow with 7-day expiry</text>
|
||||
</observation>
|
||||
```
|
||||
|
||||
Valid types: decision, bugfix, feature, refactor, discovery
|
||||
|
||||
Structure requirements:
|
||||
- <observation> is the root element
|
||||
- <type> must be one of the 5 valid types (single word)
|
||||
- <text> contains your concise observation (one sentence preferred)
|
||||
- No additional fields or nesting
|
||||
|
||||
The SDK worker will parse all <observation> blocks from your response using regex and store them in SQLite.
|
||||
|
||||
You can include your reasoning before or after the observation block, or just output the observation by itself.
|
||||
|
||||
Ready to process tool responses.
|
||||
```
|
||||
|
||||
**SDK Agent State**: Running, waiting for observations
|
||||
|
||||
---
|
||||
|
||||
### 2. TOOL EXECUTION (context-hook.js, continuous)
|
||||
|
||||
**Trigger**: Each tool execution in Claude Code
|
||||
**Hook**: `user-prompt-submit` (captures tool results)
|
||||
|
||||
**Actions**:
|
||||
1. Hook captures tool_name, tool_input, tool_output
|
||||
2. Sends observation to worker HTTP endpoint
|
||||
3. Worker queues observation for SDK agent
|
||||
|
||||
**Observation Prompt Sent to SDK**:
|
||||
```
|
||||
TOOL OBSERVATION
|
||||
================
|
||||
Tool: {tool_name}
|
||||
Time: {timestamp}
|
||||
|
||||
Input:
|
||||
{tool_input JSON}
|
||||
|
||||
Output:
|
||||
{tool_output JSON}
|
||||
|
||||
ANALYSIS TASK
|
||||
-------------
|
||||
ANALYZE this tool response and DECIDE: Does it contain something worth storing?
|
||||
|
||||
Most Read, Edit, Grep, Bash, and Write operations contain meaningful content.
|
||||
|
||||
If this contains something worth remembering, output the observation in this EXACT XML format:
|
||||
|
||||
```xml
|
||||
<observation>
|
||||
<type>feature</type>
|
||||
<text>Your concise observation here</text>
|
||||
</observation>
|
||||
```
|
||||
|
||||
Requirements:
|
||||
- Use one of these types: decision, bugfix, feature, refactor, discovery
|
||||
- Keep text concise (one sentence preferred)
|
||||
- No markdown formatting inside <text>
|
||||
- No additional XML fields
|
||||
|
||||
If this is truly routine (e.g., empty git status), you can skip it. Otherwise, PROCESS and STORE it.
|
||||
```
|
||||
|
||||
**SDK Response**:
|
||||
- Generates 0-N `<observation>` XML blocks
|
||||
- Worker parses and stores in `observations` table
|
||||
- Links to `prompt_number` (increments per user prompt)
|
||||
|
||||
---
|
||||
|
||||
### 3. PROMPT END (summary-hook.js)
|
||||
|
||||
**Trigger**: User prompt completes (stop-streaming event)
|
||||
**Hook**: `stop-streaming`
|
||||
|
||||
**Actions**:
|
||||
1. Send summarize request to worker
|
||||
2. Worker sends finalize prompt to SDK agent
|
||||
|
||||
**Finalize Prompt Sent to SDK**:
|
||||
```
|
||||
SESSION ENDING
|
||||
==============
|
||||
The Claude Code session is finishing.
|
||||
|
||||
FINAL TASK
|
||||
----------
|
||||
1. Review the observations you've stored this session
|
||||
2. Generate a structured summary that answers these questions:
|
||||
- What did user request?
|
||||
- What did you investigate?
|
||||
- What did you learn?
|
||||
- What did you do?
|
||||
- What's next?
|
||||
- Files read
|
||||
- Files edited
|
||||
- Notes
|
||||
|
||||
3. Generate the structured summary and output it in this EXACT XML format:
|
||||
|
||||
```xml
|
||||
<summary>
|
||||
<request>Implement JWT authentication system</request>
|
||||
<investigated>Existing auth middleware, session management, token storage patterns</investigated>
|
||||
<learned>Current system uses session cookies; no JWT support; race condition in middleware</learned>
|
||||
<completed>Implemented JWT token + refresh flow with 7-day expiry; fixed race condition with mutex; added token validation middleware</completed>
|
||||
<next_steps>Add token revocation API endpoint; write integration tests</next_steps>
|
||||
<files_read>
|
||||
<file>src/auth.ts</file>
|
||||
<file>src/middleware/session.ts</file>
|
||||
<file>src/types/user.ts</file>
|
||||
</files_read>
|
||||
<files_edited>
|
||||
<file>src/auth.ts</file>
|
||||
<file>src/middleware/auth.ts</file>
|
||||
<file>src/routes/auth.ts</file>
|
||||
</files_edited>
|
||||
<notes>Token secret stored in .env; refresh tokens use rotation strategy</notes>
|
||||
</summary>
|
||||
```
|
||||
|
||||
Structure requirements:
|
||||
- <summary> is the root element
|
||||
- All 8 child elements are REQUIRED: request, investigated, learned, completed, next_steps, files_read, files_edited, notes
|
||||
- <files_read> and <files_edited> must contain <file> child elements (one per file)
|
||||
- If no files were read/edited, use empty tags: <files_read></files_read>
|
||||
- Text fields can be multiple sentences but avoid markdown formatting
|
||||
- Use underscores in element names: next_steps, files_read, files_edited
|
||||
|
||||
The SDK worker will parse the <summary> block and extract all fields to store in SQLite.
|
||||
|
||||
Generate the summary now in the required XML format.
|
||||
```
|
||||
|
||||
**SDK Response**:
|
||||
- Generates `<summary>` XML block
|
||||
- Worker parses and stores in `session_summaries` table
|
||||
- Links to specific `prompt_number`
|
||||
|
||||
---
|
||||
|
||||
### 4. SESSION END (cleanup-hook.js)
|
||||
|
||||
**Trigger**: Claude Code session ends
|
||||
**Hook**: `session-end`
|
||||
|
||||
**Actions**:
|
||||
1. Mark session as completed
|
||||
2. SDK agent continues running (doesn't terminate)
|
||||
3. Worker stays alive for next session
|
||||
|
||||
---
|
||||
|
||||
## Data Storage
|
||||
|
||||
### Observations Table
|
||||
```sql
|
||||
CREATE TABLE observations (
|
||||
id INTEGER PRIMARY KEY,
|
||||
sdk_session_id TEXT NOT NULL,
|
||||
project TEXT NOT NULL,
|
||||
type TEXT NOT NULL,
|
||||
text TEXT NOT NULL,
|
||||
prompt_number INTEGER NOT NULL,
|
||||
created_at INTEGER NOT NULL,
|
||||
FOREIGN KEY(sdk_session_id) REFERENCES sdk_sessions(sdk_session_id)
|
||||
)
|
||||
```
|
||||
|
||||
### Session Summaries Table
|
||||
```sql
|
||||
CREATE TABLE session_summaries (
|
||||
id INTEGER PRIMARY KEY,
|
||||
sdk_session_id TEXT NOT NULL,
|
||||
project TEXT NOT NULL,
|
||||
request TEXT NOT NULL,
|
||||
investigated TEXT NOT NULL,
|
||||
learned TEXT NOT NULL,
|
||||
completed TEXT NOT NULL,
|
||||
next_steps TEXT NOT NULL,
|
||||
files_read TEXT NOT NULL, -- JSON array
|
||||
files_edited TEXT NOT NULL, -- JSON array
|
||||
notes TEXT NOT NULL,
|
||||
prompt_number INTEGER NOT NULL,
|
||||
created_at INTEGER NOT NULL,
|
||||
FOREIGN KEY(sdk_session_id) REFERENCES sdk_sessions(sdk_session_id)
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Key Characteristics
|
||||
|
||||
### Strengths
|
||||
1. **Persistent SDK agent**: No restart overhead per prompt
|
||||
2. **Structured data**: Typed observations, structured summaries
|
||||
3. **Per-prompt tracking**: `prompt_number` links observations to specific requests
|
||||
4. **Foreign key integrity**: Observations link to sessions via SDK session ID
|
||||
|
||||
### Weaknesses
|
||||
1. **"MOST" ambiguity**: Init prompt says "For MOST meaningful tool outputs" - confusing
|
||||
2. **Observation prompt repetition**: "Most Read, Edit, Grep, Bash, and Write operations contain meaningful content" - contradicts selectivity
|
||||
3. **XML parsing brittleness**: Regex-based XML parsing fragile
|
||||
4. **No narrative context**: Observations are one-sentence only
|
||||
5. **Summary per prompt**: Creates many summaries, unclear if useful
|
||||
6. **No hierarchical organization**: Flat observation list
|
||||
7. **Limited searchability**: Simple text fields, no embedding/vector search
|
||||
@@ -0,0 +1,38 @@
|
||||
# Final Finalize Prompt
|
||||
|
||||
```
|
||||
SESSION ENDING
|
||||
==============
|
||||
This Claude Code session is completing.
|
||||
|
||||
TASK
|
||||
----
|
||||
Review the observations you generated and create a session summary.
|
||||
|
||||
Output this XML:
|
||||
|
||||
```xml
|
||||
<summary>
|
||||
<request>[What did the user request?]</request>
|
||||
<investigated>[What code and systems did you explore?]</investigated>
|
||||
<learned>[What did you learn about the codebase?]</learned>
|
||||
<completed>[What was accomplished in this session?]</completed>
|
||||
<next_steps>[What should be done next?]</next_steps>
|
||||
<files_read>
|
||||
<file>[path/to/file]</file>
|
||||
</files_read>
|
||||
<files_edited>
|
||||
<file>[path/to/file]</file>
|
||||
</files_edited>
|
||||
<notes>[Additional insights or context]</notes>
|
||||
</summary>
|
||||
```
|
||||
|
||||
REQUIREMENTS
|
||||
------------
|
||||
All 8 fields are required: request, investigated, learned, completed, next_steps, files_read, files_edited, notes
|
||||
|
||||
Files must be wrapped in <file> tags
|
||||
|
||||
If no files were read/edited, use empty tags: <files_read></files_read>
|
||||
```
|
||||
@@ -0,0 +1,389 @@
|
||||
# Recommended Prompt Flow (Hybrid System)
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. **Be Selective**: Quality over quantity - only store meaningful insights
|
||||
2. **Hierarchical Storage**: Multiple levels for granular retrieval
|
||||
3. **Clear Guidance**: No ambiguous language like "MOST"
|
||||
4. **Structured Data**: XML format with clear schema
|
||||
5. **Session Tracking**: Title + subtitle per session
|
||||
6. **Per-Prompt Context**: Track which observations came from which user request
|
||||
|
||||
---
|
||||
|
||||
## Flow Timeline
|
||||
|
||||
### 1. SESSION START
|
||||
|
||||
**Trigger**: Claude Code session starts
|
||||
**Hook**: `user-prompt-submit` (context-hook.js)
|
||||
|
||||
**Init Prompt Sent to SDK**:
|
||||
```
|
||||
You are a memory processor for the "{project}" project.
|
||||
|
||||
SESSION CONTEXT
|
||||
---------------
|
||||
Session ID: {sessionId}
|
||||
User's Goal: {userPrompt}
|
||||
Date: {date}
|
||||
|
||||
YOUR ROLE
|
||||
---------
|
||||
Process tool executions from this Claude Code session and store important observations.
|
||||
|
||||
Be selective. Only store observations when the tool output reveals important information about:
|
||||
- Architecture or design patterns
|
||||
- Implementation details of features or bug fixes
|
||||
- System state or configuration
|
||||
- Business logic or algorithms
|
||||
|
||||
Skip routine operations like:
|
||||
- Empty git status checks
|
||||
- Simple npm install output
|
||||
- Trivial config changes
|
||||
- Repetitive operations
|
||||
|
||||
OBSERVATION FORMAT
|
||||
------------------
|
||||
When you identify something worth remembering, output this XML structure:
|
||||
|
||||
```xml
|
||||
<observation>
|
||||
<type>feature</type>
|
||||
<title>Short Title (3-8 words)</title>
|
||||
<subtitle>Concise summary explaining the significance (max 24 words)</subtitle>
|
||||
<facts>
|
||||
<fact>Specific detail 1 (50-150 chars, self-contained)</fact>
|
||||
<fact>Specific detail 2 (50-150 chars, self-contained)</fact>
|
||||
<fact>Specific detail 3 (50-150 chars, self-contained)</fact>
|
||||
</facts>
|
||||
<narrative>Full context: what was done, why it matters, how it works. (200-400 words)</narrative>
|
||||
<concepts>
|
||||
<concept>broad-category-1</concept>
|
||||
<concept>broad-category-2</concept>
|
||||
</concepts>
|
||||
<files>
|
||||
<file>path/to/file1.ts</file>
|
||||
<file>path/to/file2.ts</file>
|
||||
</files>
|
||||
</observation>
|
||||
```
|
||||
|
||||
FIELD REQUIREMENTS
|
||||
------------------
|
||||
|
||||
**Type**: One of: decision, bugfix, feature, refactor, discovery
|
||||
|
||||
**Title**: 3-8 words capturing the core action
|
||||
- Examples: "JWT Refresh Token Implementation", "Race Condition Fix in Auth Middleware"
|
||||
|
||||
**Subtitle**: Max 24 words explaining the significance
|
||||
- Focus on outcome or benefit
|
||||
- Examples: "Added 7-day refresh token rotation with Redis storage for secure long-lived sessions"
|
||||
|
||||
**Facts**: 3-7 atomic facts (50-150 chars each)
|
||||
- Each fact is ONE specific piece of information
|
||||
- Include filename/component when relevant
|
||||
- No pronouns - each fact stands alone
|
||||
- Examples:
|
||||
- "src/auth.ts: refreshToken() generates new JWT with 7-day expiry"
|
||||
- "Redis key format: refresh:{userId}:{tokenId} with TTL 604800s"
|
||||
- "Old token invalidated on refresh to prevent replay attacks"
|
||||
|
||||
**Narrative**: Full story (200-400 words)
|
||||
- What was done
|
||||
- Technical details (files, functions, implementation)
|
||||
- Why it matters
|
||||
|
||||
**Concepts**: 2-5 broad categories for filtering
|
||||
- Examples: "authentication", "caching", "error-handling"
|
||||
|
||||
**Files**: All files touched
|
||||
- Full relative paths from project root
|
||||
|
||||
Ready to process tool executions.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. TOOL EXECUTION
|
||||
|
||||
**Trigger**: Each tool execution
|
||||
**Hook**: `user-prompt-submit` (context-hook.js)
|
||||
|
||||
**Observation Prompt Sent to SDK**:
|
||||
```
|
||||
TOOL OBSERVATION
|
||||
================
|
||||
Tool: {tool_name}
|
||||
Time: {timestamp}
|
||||
Prompt: {prompt_number}
|
||||
|
||||
Input:
|
||||
{tool_input JSON}
|
||||
|
||||
Output:
|
||||
{tool_output JSON}
|
||||
|
||||
Analyze this tool output. If it reveals important information about the codebase, generate an observation using the XML format from the init prompt.
|
||||
```
|
||||
|
||||
**SDK Response Processing**:
|
||||
1. SDK agent analyzes output
|
||||
2. If meaningful, generates `<observation>` XML block
|
||||
3. Worker parses XML and stores in SQLite
|
||||
4. Links to `prompt_number` for per-request tracking
|
||||
|
||||
**Database Schema**:
|
||||
```sql
|
||||
CREATE TABLE observations (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
sdk_session_id TEXT NOT NULL,
|
||||
project TEXT NOT NULL,
|
||||
prompt_number INTEGER NOT NULL,
|
||||
|
||||
-- Core fields
|
||||
type TEXT NOT NULL,
|
||||
title TEXT NOT NULL,
|
||||
subtitle TEXT NOT NULL,
|
||||
narrative TEXT NOT NULL,
|
||||
|
||||
-- Arrays (stored as JSON)
|
||||
facts TEXT NOT NULL, -- JSON array of strings
|
||||
concepts TEXT NOT NULL, -- JSON array of strings
|
||||
files TEXT NOT NULL, -- JSON array of strings
|
||||
|
||||
created_at INTEGER NOT NULL,
|
||||
|
||||
FOREIGN KEY(sdk_session_id) REFERENCES sdk_sessions(sdk_session_id) ON DELETE CASCADE
|
||||
);
|
||||
|
||||
-- Indexes for fast retrieval
|
||||
CREATE INDEX idx_observations_session ON observations(sdk_session_id);
|
||||
CREATE INDEX idx_observations_type ON observations(type);
|
||||
CREATE INDEX idx_observations_prompt ON observations(prompt_number);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. SESSION END
|
||||
|
||||
**Trigger**: Claude Code session ends
|
||||
**Hook**: `session-end` (cleanup-hook.js)
|
||||
|
||||
**Finalize Prompt Sent to SDK**:
|
||||
```
|
||||
SESSION ENDING
|
||||
==============
|
||||
The Claude Code session is completing.
|
||||
|
||||
FINAL TASK
|
||||
----------
|
||||
Review all observations you've generated and create a session summary.
|
||||
|
||||
Output this XML structure:
|
||||
|
||||
```xml
|
||||
<summary>
|
||||
<request>What did the user request?</request>
|
||||
<investigated>What code/systems did you explore?</investigated>
|
||||
<learned>What did you learn about the codebase?</learned>
|
||||
<completed>What was accomplished?</completed>
|
||||
<next_steps>What should happen next?</next_steps>
|
||||
<files_read>
|
||||
<file>path/to/file1.ts</file>
|
||||
<file>path/to/file2.ts</file>
|
||||
</files_read>
|
||||
<files_edited>
|
||||
<file>path/to/file3.ts</file>
|
||||
</files_edited>
|
||||
<notes>Additional context or insights</notes>
|
||||
</summary>
|
||||
```
|
||||
|
||||
Be concise but comprehensive. Focus on semantic insights, not mechanical details.
|
||||
```
|
||||
|
||||
**Database Schema**:
|
||||
```sql
|
||||
CREATE TABLE session_summaries (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
sdk_session_id TEXT NOT NULL,
|
||||
project TEXT NOT NULL,
|
||||
|
||||
request TEXT NOT NULL,
|
||||
investigated TEXT NOT NULL,
|
||||
learned TEXT NOT NULL,
|
||||
completed TEXT NOT NULL,
|
||||
next_steps TEXT NOT NULL,
|
||||
files_read TEXT NOT NULL, -- JSON array
|
||||
files_edited TEXT NOT NULL, -- JSON array
|
||||
notes TEXT NOT NULL,
|
||||
|
||||
created_at INTEGER NOT NULL,
|
||||
|
||||
FOREIGN KEY(sdk_session_id) REFERENCES sdk_sessions(sdk_session_id) ON DELETE CASCADE
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Data Retrieval Patterns
|
||||
|
||||
### Level 1: Session Titles (High-Level Browsing)
|
||||
```sql
|
||||
SELECT
|
||||
sdk_session_id,
|
||||
user_prompt as title,
|
||||
created_at
|
||||
FROM sdk_sessions
|
||||
WHERE project = ?
|
||||
ORDER BY created_at DESC;
|
||||
```
|
||||
|
||||
### Level 2: Session Summaries (Session Overview)
|
||||
```sql
|
||||
SELECT
|
||||
request,
|
||||
completed,
|
||||
next_steps
|
||||
FROM session_summaries
|
||||
WHERE sdk_session_id = ?;
|
||||
```
|
||||
|
||||
### Level 3: Observation Titles (Scannable List)
|
||||
```sql
|
||||
SELECT
|
||||
type,
|
||||
title,
|
||||
subtitle
|
||||
FROM observations
|
||||
WHERE sdk_session_id = ?
|
||||
ORDER BY id;
|
||||
```
|
||||
|
||||
### Level 4: Atomic Facts (Precise Search)
|
||||
```sql
|
||||
SELECT
|
||||
title,
|
||||
facts
|
||||
FROM observations
|
||||
WHERE
|
||||
sdk_session_id = ?
|
||||
AND facts LIKE '%keyword%';
|
||||
```
|
||||
|
||||
### Level 5: Full Narrative (Deep Dive)
|
||||
```sql
|
||||
SELECT
|
||||
title,
|
||||
subtitle,
|
||||
facts,
|
||||
narrative,
|
||||
files
|
||||
FROM observations
|
||||
WHERE id = ?;
|
||||
```
|
||||
|
||||
### By Concept (Category Filter)
|
||||
```sql
|
||||
SELECT
|
||||
title,
|
||||
subtitle,
|
||||
concepts
|
||||
FROM observations
|
||||
WHERE concepts LIKE '%"authentication"%';
|
||||
```
|
||||
|
||||
### By File (File-Based Search)
|
||||
```sql
|
||||
SELECT
|
||||
title,
|
||||
subtitle,
|
||||
files
|
||||
FROM observations
|
||||
WHERE files LIKE '%src/auth.ts%';
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
### Phase 2: Semantic Search
|
||||
- Add vector embeddings for facts and narratives
|
||||
- Store in ChromaDB or similar
|
||||
- Enable similarity search: "Find observations about authentication patterns"
|
||||
|
||||
### Phase 3: Cross-Session Memory
|
||||
- Link related observations across sessions
|
||||
- "Show all JWT-related observations from past 30 days"
|
||||
|
||||
### Phase 4: Session Metadata
|
||||
- Add title/subtitle to sdk_sessions table
|
||||
- Auto-generate from user_prompt or first summary
|
||||
|
||||
---
|
||||
|
||||
## Migration from Current System
|
||||
|
||||
### Step 1: Update Database Schema
|
||||
```sql
|
||||
-- Add new columns to observations table
|
||||
ALTER TABLE observations ADD COLUMN title TEXT;
|
||||
ALTER TABLE observations ADD COLUMN subtitle TEXT;
|
||||
ALTER TABLE observations ADD COLUMN narrative TEXT;
|
||||
ALTER TABLE observations ADD COLUMN facts TEXT;
|
||||
ALTER TABLE observations ADD COLUMN concepts TEXT;
|
||||
ALTER TABLE observations ADD COLUMN files TEXT;
|
||||
|
||||
-- Migrate existing observations (best-effort)
|
||||
UPDATE observations
|
||||
SET
|
||||
title = type || ' - ' || substr(text, 1, 50),
|
||||
subtitle = text,
|
||||
narrative = text,
|
||||
facts = '[]',
|
||||
concepts = '[]',
|
||||
files = '[]'
|
||||
WHERE title IS NULL;
|
||||
```
|
||||
|
||||
### Step 2: Update Prompts
|
||||
- Replace `buildInitPrompt()` with new version (no "MOST")
|
||||
- Replace `buildObservationPrompt()` with new version (no tool-type bias)
|
||||
- Keep `buildFinalizePrompt()` mostly as-is
|
||||
|
||||
### Step 3: Update Parser
|
||||
- Extend `parseObservations()` to extract all new fields
|
||||
- Add `extractFactArray()`, `extractConceptArray()`, `extractFileArray()` helpers
|
||||
- Keep backward compatibility with old one-sentence format
|
||||
|
||||
### Step 4: Update Storage
|
||||
- Modify `HooksDatabase.storeObservation()` to accept all fields
|
||||
- Store arrays as JSON strings
|
||||
|
||||
---
|
||||
|
||||
## Key Improvements Over Current System
|
||||
|
||||
1. ✅ **No "MOST" ambiguity** - Clear "be selective" guidance
|
||||
2. ✅ **No tool-type bias** - Observation prompt doesn't mention tool names
|
||||
3. ✅ **Hierarchical storage** - Title → Subtitle → Facts → Narrative
|
||||
4. ✅ **Atomic facts** - Precise, searchable details
|
||||
5. ✅ **File associations** - Track which files each observation relates to
|
||||
6. ✅ **Concept tagging** - Categorical organization
|
||||
7. ✅ **Rich narratives** - Full context for deep dives
|
||||
8. ✅ **Multiple retrieval levels** - Can search at any granularity
|
||||
|
||||
---
|
||||
|
||||
## Key Improvements Over Old System
|
||||
|
||||
1. ✅ **No bash commands** - XML parsing instead of shell execution
|
||||
2. ✅ **Auto-increment IDs** - No manual counter tracking
|
||||
3. ✅ **Per-prompt tracking** - `prompt_number` links observations to requests
|
||||
4. ✅ **Foreign key integrity** - Automatic cascade deletes
|
||||
5. ✅ **No quote escaping hell** - JSON arrays instead of bash arguments
|
||||
6. ✅ **Structured typing** - Typed observations (decision/bugfix/feature/refactor/discovery)
|
||||
7. ✅ **Session summary at end** - Not just 2-3 sentences, but full structured summary
|
||||
@@ -0,0 +1,91 @@
|
||||
# Final Init Prompt
|
||||
|
||||
```
|
||||
You are a memory processor for the "{project}" project.
|
||||
|
||||
SESSION CONTEXT
|
||||
---------------
|
||||
Session ID: {sessionId}
|
||||
User's Goal: {userPrompt}
|
||||
Date: {date}
|
||||
|
||||
YOUR ROLE
|
||||
---------
|
||||
Process tool executions from this Claude Code session and store observations that contain information worth remembering.
|
||||
|
||||
WHEN TO STORE
|
||||
-------------
|
||||
Store an observation when the tool output contains information worth remembering about:
|
||||
- How things work
|
||||
- Why things exist or were chosen
|
||||
- What changed
|
||||
- Problems and their solutions
|
||||
- Important patterns or gotchas
|
||||
|
||||
WHEN TO SKIP
|
||||
------------
|
||||
Skip routine operations:
|
||||
- Empty status checks
|
||||
- Package installations with no errors
|
||||
- Simple file listings
|
||||
- Repetitive operations you've already documented
|
||||
|
||||
OBSERVATION FORMAT
|
||||
------------------
|
||||
Output observations using this XML structure:
|
||||
|
||||
```xml
|
||||
<observation>
|
||||
<type>change</type>
|
||||
<title>[Short title]</title>
|
||||
<subtitle>[One sentence explanation (max 24 words)]</subtitle>
|
||||
<facts>
|
||||
<fact>[Concise, self-contained statement]</fact>
|
||||
<fact>[Concise, self-contained statement]</fact>
|
||||
<fact>[Concise, self-contained statement]</fact>
|
||||
</facts>
|
||||
<narrative>[Full context: what, how, and why]</narrative>
|
||||
<concepts>
|
||||
<concept>[knowledge-type-category]</concept>
|
||||
<concept>[knowledge-type-category]</concept>
|
||||
</concepts>
|
||||
<files>
|
||||
<file>[path/to/file]</file>
|
||||
<file>[path/to/file]</file>
|
||||
</files>
|
||||
</observation>
|
||||
```
|
||||
|
||||
FIELD REQUIREMENTS
|
||||
------------------
|
||||
|
||||
**type**: One of:
|
||||
- change: modifications to code, config, or documentation
|
||||
- discovery: learning about existing system
|
||||
- decision: choosing an approach and why it was chosen
|
||||
|
||||
**title**: Short title capturing the core action or topic
|
||||
|
||||
**subtitle**: One sentence explanation (max 24 words)
|
||||
|
||||
**facts**: Concise, self-contained statements
|
||||
Each fact is ONE piece of information
|
||||
No pronouns - each fact must stand alone
|
||||
Include specific details: filenames, functions, values
|
||||
|
||||
**narrative**: Full context: what, how, and why
|
||||
What was done, how it works, why it matters
|
||||
|
||||
**concepts**: 2-5 knowledge-type categories:
|
||||
- how-it-works: understanding mechanisms
|
||||
- why-it-exists: purpose or rationale
|
||||
- what-changed: modifications made
|
||||
- problem-solution: issues and their fixes
|
||||
- gotcha: traps or edge cases
|
||||
- pattern: reusable approach
|
||||
- trade-off: pros/cons of a decision
|
||||
|
||||
**files**: All files touched (full paths from project root)
|
||||
|
||||
Ready to process tool executions.
|
||||
```
|
||||
@@ -0,0 +1,17 @@
|
||||
# Final Observation Prompt
|
||||
|
||||
```
|
||||
TOOL OBSERVATION
|
||||
================
|
||||
Tool: {tool_name}
|
||||
Time: {timestamp}
|
||||
Prompt: {prompt_number}
|
||||
|
||||
Input:
|
||||
{tool_input JSON}
|
||||
|
||||
Output:
|
||||
{tool_output JSON}
|
||||
|
||||
Analyze this tool output. If it contains information worth remembering, generate an observation using the XML format.
|
||||
```
|
||||
Reference in New Issue
Block a user