From 05ddf3540c992b977efdfc65d95461beafa53a87 Mon Sep 17 00:00:00 2001
From: Alex Newman <thedotmack@gmail.com>
Date: Sat, 25 Oct 2025 01:59:08 -0400
Subject: [PATCH] Release v4.2.9: Progressive disclosure experimental feature
 announcement
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Documentation Updates:
- Added experimental progressive disclosure context system section to README
- Created EXPERIMENTAL_RELEASE_NOTES.md with comprehensive testing guide
- Created GITHUB_RELEASE_TEMPLATE.md for release announcements
- Updated all version references to 4.2.9 (package.json, marketplace.json, CLAUDE.md, README.md)

Progressive Disclosure Concept:
- Layer 1: Index showing what observations exist with token costs
- Layer 2: On-demand details via MCP search
- Layer 3: Perfect recall via source code access

Purpose:
- Invite users to test feature/context-with-observations branch
- Gather feedback on observation-level context vs summary-only approach
- Validate whether token cost metadata improves Claude's retrieval decisions

Testing Instructions:
- Clear documentation on how to clone, build, and test experimental branch
- Feedback template provided for structured user responses
- GitHub issues encouraged with label 'feedback: progressive-disclosure'

Files Changed:
- README.md (experimental feature section)
- EXPERIMENTAL_RELEASE_NOTES.md (new)
- GITHUB_RELEASE_TEMPLATE.md (new)
- package.json (version bump)
- .claude-plugin/marketplace.json (version bump)
- CLAUDE.md (version history + version number)
- plugin/scripts/*.js (rebuilt)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
---
 .claude-plugin/marketplace.json   |   2 +-
 CLAUDE.md                         |  27 ++-
 EXPERIMENTAL_RELEASE_NOTES.md     | 331 ++++++++++++++++++++++++++++++
 GITHUB_RELEASE_TEMPLATE.md        |  83 ++++++++
 README.md                         |  86 +++++++-
 package.json                      |   2 +-
 plugin/.claude-plugin/plugin.json |   2 +-
 7 files changed, 527 insertions(+), 6 deletions(-)
 create mode 100644 EXPERIMENTAL_RELEASE_NOTES.md
 create mode 100644 GITHUB_RELEASE_TEMPLATE.md

diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json
index ce01ea44..f553d496 100644
--- a/.claude-plugin/marketplace.json
+++ b/.claude-plugin/marketplace.json
@@ -10,7 +10,7 @@
   "plugins": [
     {
       "name": "claude-mem",
-      "version": "4.2.8",
+      "version": "4.2.9",
       "source": "./plugin",
       "description": "Persistent memory system for Claude Code - context compression across sessions"
     }
diff --git a/CLAUDE.md b/CLAUDE.md
index 778a46b5..da07718c 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -4,7 +4,7 @@
 
 Claude-mem is a persistent memory compression system that preserves context across Claude Code sessions. It automatically captures tool usage observations, processes them through the Claude Agent SDK, and makes summaries available to future sessions.
 
-**Current Version**: 4.2.8
+**Current Version**: 4.2.9
 **License**: AGPL-3.0
 **Author**: Alex Newman (@thedotmack)
 
@@ -210,7 +210,30 @@ npm run build && git commit -a -m "Build and update" && git push && cd ~/.claude
 
 ## Version History
 
-### v4.2.8 (Current)
+### v4.2.9 (Current)
+**Breaking Changes**: None (patch version)
+
+**Documentation**:
+- Added experimental progressive disclosure context system documentation
+  - New README section explaining the 3-layer memory retrieval approach
+  - Created `EXPERIMENTAL_RELEASE_NOTES.md` with comprehensive testing guide
+  - Created `GITHUB_RELEASE_TEMPLATE.md` for release announcements
+  - Invites users to test `feature/context-with-observations` branch
+- Progressive disclosure concept: Index (what exists + token costs) → Details (on-demand via MCP) → Perfect recall (source code)
+- Seeks user feedback on observation-level context injection vs current summary-only approach
+
+**Purpose**:
+- Gather real-world feedback before merging experimental context improvements
+- Test whether showing observation index improves Claude's retrieval decisions
+- Validate token cost metadata influences Claude's search behavior
+
+**Files Changed**:
+- `README.md` - Added experimental feature section
+- `EXPERIMENTAL_RELEASE_NOTES.md` - Full testing guide and feedback template
+- `GITHUB_RELEASE_TEMPLATE.md` - Release announcement template
+- Updated all version references to 4.2.9
+
+### v4.2.8
 **Breaking Changes**: None (patch version)
 
 **Critical Bugfix**:
diff --git a/EXPERIMENTAL_RELEASE_NOTES.md b/EXPERIMENTAL_RELEASE_NOTES.md
new file mode 100644
index 00000000..63ee17d2
--- /dev/null
+++ b/EXPERIMENTAL_RELEASE_NOTES.md
@@ -0,0 +1,331 @@
+# Experimental Release: Progressive Disclosure Context System
+
+## 🧪 Branch: `feature/context-with-observations`
+
+**Status:** Seeking user feedback before merging to main
+
+**We'd love your testing and feedback!** This experimental branch reimagines how Claude-Mem presents context at session startup, using a progressive disclosure approach that could significantly improve Claude's ability to leverage past learnings.
+
+---
+
+## What is Progressive Disclosure?
+
+Progressive disclosure is a **layered memory retrieval system** inspired by how humans remember information:
+
+### Layer 1: Index (The "Table of Contents")
+**Frontloaded at session start** - Claude sees:
+- **What exists**: Titles of all recent observations and session summaries
+- **Retrieval cost**: Token counts for each observation
+- **Priority signals**: Type indicators (🔴 critical gotcha, 🟤 architectural decision, 🔵 explanatory)
+
+### Layer 2: Details (On-Demand Retrieval)
+**Retrieved via MCP search** - Claude fetches:
+- Full observation narratives when deeper context is needed
+- Search by concept, file path, type, or keywords
+- Only loads what's relevant to the current task
+
+### Layer 3: Perfect Recall (Source of Truth)
+**Direct code access** - When needed:
+- Read actual source files for implementation details
+- Access original transcripts for exact quotes
+- Full context without compression artifacts
+
+---
+
+## The Problem This Solves
+
+### Current Version (v4.2.x) Limitation
+
+The current context hook shows **only session summaries** at startup:
+
+```markdown
+**Session #312**: Put date/time at end of session titles
+Completed: Added date/time to session list with proper formatting
+Next Steps: Test edge cases with long dates
+```
+
+**Strengths:**
+- ✅ Minimal token overhead (~800 tokens)
+- ✅ Clean, readable summaries
+
+**Weaknesses:**
+- ❌ Claude doesn't know **what** detailed observations exist
+- ❌ Can't make informed decisions about whether to search vs read code
+- ❌ Often re-reads code to understand decisions that were already documented
+
+### Experimental Version Enhancement
+
+The experimental hook shows an **observation index** alongside session summaries:
+
+```markdown
+**src/hooks/context.ts**
+| ID | Time | T | Title | Tokens |
+|----|------|---|-------|--------|
+| #2332 | 1:07 AM | 🔴 | Critical Bugfix: Session ID NULL Constraint | ~201 |
+| #2340 | 1:10 AM | 🟠 | Remove Redundant Summary Section | ~280 |
+| #2344 | 1:34 AM | 🔵 | Added progressive disclosure usage instructions | ~149 |
+```
+
+**Benefits:**
+- ✅ Claude knows **what** learnings exist (titles/types)
+- ✅ Token counts inform **cost-benefit** decisions (fetch ~200 tokens vs re-read 2000-line file)
+- ✅ Progressive disclosure instructions **teach Claude** how to use the system
+- ✅ Type indicators help prioritize (critical gotchas > explanatory notes)
+
+**Trade-offs:**
+- ⚠️ Higher initial token cost (~2,500 tokens vs ~800)
+- ⚠️ More visual noise in the context output
+- ❓ Unknown: Does this actually improve Claude's behavior enough to justify the cost?
+
+---
+
+## What's New in This Branch
+
+### 1. Observation Index Display
+
+Full table view of recent observations grouped by file:
+
+```markdown
+### Oct 25, 2025
+
+**src/hooks/context.ts**
+| ID | Time | T | Title | Tokens |
+|----|------|---|-------|--------|
+| #2296 | 12:12 AM | 🟢 | Session summaries now display date and time | ~141 |
+| #2298 | 12:44 AM | 🔵 | Timeline rendering refactored | ~231 |
+
+**General**
+| ID | Time | T | Title | Tokens |
+|----|------|---|-------|--------|
+| #2301 | 12:50 AM | 🟢 | Development Task Breakdown Created | ~128 |
+```
+
+### 2. Token Cost Metadata
+
+Every observation shows estimated token count:
+- Helps Claude decide: "Is it worth fetching this 500-token explanation, or should I just read the code?"
+- Makes cost-benefit analysis explicit
+
+### 3. Progressive Disclosure Instructions
+
+New guidance section teaches Claude how to use the system:
+
+```markdown
+💡 Progressive Disclosure: This index shows WHAT exists (titles) and retrieval COST (token counts).
+- Use MCP search tools to fetch full observation details on-demand (Layer 2)
+- Prefer searching observations over re-reading code for past decisions and learnings
+- Critical types (🔴 gotcha, 🟤 decision, ⚖️ trade-off) often worth fetching immediately
+```
+
+### 4. Type-Based Priority System
+
+Observations categorized by importance:
+- 🔴 **gotcha** - Critical bugs/blockers (fetch immediately)
+- 🟤 **decision** - Architectural choices (high value)
+- ⚖️ **trade-off** - Design considerations (prevents re-debating)
+- 🟠 **why-it-exists** - Rationale documentation
+- 🟡 **problem-solution** - How issues were solved
+- 🟣 **discovery** - Important learnings
+- 🔵 **how-it-works** - Explanatory/educational
+- 🟢 **what-changed** - Implementation details
+
+---
+
+## Testing Instructions
+
+### Option 1: Quick Test (No Installation)
+
+```bash
+# Clone and checkout experimental branch
+git clone https://github.com/thedotmack/claude-mem.git
+cd claude-mem
+git checkout feature/context-with-observations
+
+# Build the experimental version
+npm install
+npm run build
+
+# Navigate to YOUR project directory
+cd /path/to/your/project
+
+# Run the experimental context hook with full path
+node /path/to/claude-mem/plugin/scripts/context-hook.js
+
+# Example:
+# cd ~/my-app
+# node ~/Downloads/claude-mem/plugin/scripts/context-hook.js
+```
+
+**Important:** The context hook reads from the current working directory (cwd). You must run it from your project's root folder to see context for that specific project.
+
+This shows you the new context format without installing the plugin.
+
+### Option 2: Full Testing (Install Locally)
+
+If you're already using claude-mem and want to test the experimental version:
+
+```bash
+# Navigate to your local claude-mem plugin directory
+cd ~/.claude/plugins/marketplaces/thedotmack
+
+# Checkout experimental branch
+git fetch origin
+git checkout feature/context-with-observations
+
+# Rebuild
+npm install
+npm run build
+
+# Restart Claude Code to see the new context injection
+```
+
+**⚠️ Warning:** This will replace your current context hook. To revert:
+```bash
+git checkout main
+npm run build
+```
+
+---
+
+## What We Want to Know
+
+Please test the experimental branch and share your feedback on these questions:
+
+### 1. Behavioral Impact
+- ✅ **Does Claude use MCP search more effectively?**
+  - Does it fetch observation details more often?
+  - Does it make better decisions about when to search vs read code?
+
+### 2. Token Cost Analysis
+- 💰 **Do token counts influence Claude's retrieval decisions?**
+  - Does Claude reference the token counts when deciding whether to fetch?
+  - Example: "This observation is 500 tokens, so I'll read the code instead"
+
+### 3. Instruction Effectiveness
+- 📖 **Is the progressive disclosure guidance helpful or noisy?**
+  - Does Claude seem to understand the layered retrieval concept?
+  - Do the instructions clutter the context or improve clarity?
+
+### 4. Efficiency Gains
+- 🚀 **Does it reduce redundant code reading?**
+  - Does Claude fetch learnings instead of re-reading entire files?
+  - Overall: Is it faster/smarter despite the higher initial token cost?
+
+### 5. User Experience
+- 👤 **Is the observation table too cluttered?**
+  - Does the table format help or hurt readability?
+  - Would you prefer a different presentation?
+
+---
+
+## How to Provide Feedback
+
+### 📣 GitHub Issues (Please Use This!)
+
+**[→ Click here to open a new issue](https://github.com/thedotmack/claude-mem/issues/new)**
+
+Add the label `feedback: progressive-disclosure` and use this template:
+
+```markdown
+## Progressive Disclosure Feedback
+
+**Branch tested:** feature/context-with-observations
+**Test duration:** [e.g., 2 days, 10 sessions]
+**Project type:** [e.g., TypeScript library, React app, Python backend]
+
+### What worked well:
+- [Your positive observations]
+
+### What didn't work:
+- [Issues or concerns]
+
+### Specific answers:
+1. **Claude's MCP search usage:** [Improved/Same/Worse]
+2. **Token count influence:** [Yes/No/Unclear]
+3. **Instructions helpful:** [Yes/No/Too verbose]
+4. **Code reading reduction:** [Yes/No/Hard to tell]
+5. **Overall impression:** [Worth merging/Needs work/Not useful]
+
+### Additional notes:
+[Any other feedback, screenshots, or examples]
+```
+
+**Why issues?** It keeps all feedback in one searchable place and lets other users see what's being discussed. Please don't hesitate to open an issue - all feedback is valuable, positive or negative!
+
+---
+
+## Next Steps
+
+Based on feedback, we'll decide:
+
+### ✅ If Successful:
+- Merge to `main` branch
+- Release as v4.3.0
+- Make progressive disclosure the default
+- Potentially add verbosity settings (minimal/standard/detailed)
+
+### ⚠️ If Mixed Results:
+- Make it opt-in via settings: `CLAUDE_MEM_VERBOSE_CONTEXT=true`
+- Default to current minimal approach
+- Allow users to choose their preference
+
+### ❌ If Unsuccessful:
+- Keep as experimental branch
+- Continue iterating on the approach
+- May explore alternative presentation formats
+
+---
+
+## Technical Details
+
+### Files Changed
+
+- **src/hooks/context.ts** (lines 227-240)
+  - Added progressive disclosure instructions
+  - Enhanced observation table rendering
+  - Token count display for each observation
+
+### Token Cost Breakdown
+
+**Current version (v4.2.x):**
+- Session summaries only: ~800 tokens
+- 3 sessions × ~250 tokens each
+- Minimal overhead
+
+**Experimental version:**
+- Progressive disclosure instructions: ~150 tokens
+- Observation index: ~2,000 tokens
+  - 50 observations × ~40 tokens per row
+- Session summaries: ~800 tokens
+- **Total: ~2,950 tokens**
+
+**ROI Analysis:**
+- If this prevents even ONE 2,000-token file read per session, it pays for itself
+- If Claude makes smarter retrieval decisions, overall token usage could be lower
+
+---
+
+## Acknowledgments
+
+This experimental feature was inspired by:
+- Anthropic's "Effective context engineering for AI agents" (Sept 2025)
+- Claude Skills' progressive disclosure architecture (Oct 2025)
+- Real-world usage patterns from 200+ GitHub stars in 36 hours
+
+Special thanks to our early adopters for pushing the boundaries of what's possible with persistent memory!
+
+---
+
+## Questions?
+
+- 📖 **Docs:** [docs/](docs/)
+- 🐛 **Issues:** [GitHub Issues](https://github.com/thedotmack/claude-mem/issues)
+- 💬 **Discussion:** [GitHub Discussions](https://github.com/thedotmack/claude-mem/discussions)
+
+---
+
+**Happy Testing!** 🧪
+
+We're excited to hear what you discover with progressive disclosure. This could be a game-changer for how Claude leverages long-term memory, but we need your real-world testing to validate the approach.
+
+— Alex Newman ([@thedotmack](https://github.com/thedotmack))
diff --git a/GITHUB_RELEASE_TEMPLATE.md b/GITHUB_RELEASE_TEMPLATE.md
new file mode 100644
index 00000000..725c9cd0
--- /dev/null
+++ b/GITHUB_RELEASE_TEMPLATE.md
@@ -0,0 +1,83 @@
+# 🧪 Experimental: Progressive Disclosure Context System
+
+> **We'd love your feedback!** Test the new context injection approach and share your experience.
+
+## What is Progressive Disclosure?
+
+A **layered memory retrieval system** that shows Claude:
+1. **Index** (frontloaded): What observations exist + token costs
+2. **Details** (on-demand): Full narratives via MCP search
+3. **Perfect recall**: Source code when needed
+
+**The idea:** Instead of hiding observations completely, show an index so Claude can make informed decisions about what to fetch.
+
+## Try It Out
+
+```bash
+# Clone and build experimental version
+git clone https://github.com/thedotmack/claude-mem.git
+cd claude-mem
+git checkout feature/context-with-observations
+npm install && npm run build
+
+# Navigate to YOUR project and run the hook
+cd /path/to/your/project
+node /path/to/claude-mem/plugin/scripts/context-hook.js
+```
+
+**Important:** Run from your project's root directory to see context for that project.
+
+## What's Different?
+
+**Current (v4.2.x):** Session summaries only (~800 tokens)
+```markdown
+Session #312: Put date/time at end of session titles
+Completed: Added formatting
+Next: Test edge cases
+```
+
+**Experimental:** Observation index + summaries (~2,500 tokens)
+```markdown
+**src/hooks/context.ts**
+| ID | Time | T | Title | Tokens |
+|----|------|---|-------|--------|
+| #2332 | 1:07 AM | 🔴 | Critical Bugfix: Session ID NULL | ~201 |
+| #2340 | 1:10 AM | 🟠 | Remove Redundant Summary Section | ~280 |
+```
+
+Now Claude knows:
+- What learnings exist (without loading them)
+- Cost to fetch details (~200 tokens)
+- Priority (🔴 critical vs 🔵 informational)
+
+## We Want Your Feedback
+
+Test the experimental branch and tell us:
+
+✅ **Does Claude use MCP search more effectively?**
+💰 **Do token counts influence retrieval decisions?**
+📖 **Are the instructions helpful or noisy?**
+🚀 **Does it reduce redundant code reading?**
+
+### 📣 [Please Open a GitHub Issue](https://github.com/thedotmack/claude-mem/issues/new) With Your Experience!
+
+Use the label `feedback: progressive-disclosure` - all feedback is valuable, positive or negative!
+
+## Files Changed
+
+- Updated `README.md` with experimental feature section
+- Enhanced `src/hooks/context.ts` with progressive disclosure instructions
+- New docs: `EXPERIMENTAL_RELEASE_NOTES.md` (full details)
+
+## Next Steps
+
+Based on your feedback:
+- ✅ **If successful:** Merge to main, release as v4.3.0
+- ⚠️ **If mixed:** Make opt-in via settings
+- ❌ **If unsuccessful:** Keep iterating as experimental
+
+---
+
+**Full details:** See [EXPERIMENTAL_RELEASE_NOTES.md](EXPERIMENTAL_RELEASE_NOTES.md)
+
+**Questions?** Join the discussion or open an issue!
diff --git a/README.md b/README.md
index 1de7f171..a514c7e1 100644
--- a/README.md
+++ b/README.md
@@ -17,7 +17,7 @@
     <img src="https://img.shields.io/badge/License-AGPL%203.0-blue.svg" alt="License">
   </a>
   <a href="package.json">
-    <img src="https://img.shields.io/badge/version-4.2.3-green.svg" alt="Version">
+    <img src="https://img.shields.io/badge/version-4.2.9-green.svg" alt="Version">
   </a>
   <a href="package.json">
     <img src="https://img.shields.io/badge/node-%3E%3D18.0.0-brightgreen.svg" alt="Node">
@@ -40,6 +40,90 @@
 
 ---
 
+## 🧪 Experimental Feature: Progressive Disclosure Context
+
+> **We'd love your feedback!** Test the new context injection system in `feature/context-with-observations`
+
+### What is Progressive Disclosure?
+
+Progressive disclosure is a layered approach to memory retrieval that mirrors how humans remember information:
+
+**Layer 1: Index** (Frontloaded at session start)
+- **WHAT** exists: Observation titles and session summaries
+- **COST** of retrieval: Token counts for each observation
+- **TYPE** indicators: Critical (🔴 gotcha, 🟤 decision) vs informational (🔵 how-it-works)
+
+**Layer 2: Details** (Retrieved on-demand via MCP search)
+- Full observation narratives when Claude needs deeper context
+- Search by concept, file, type, or keyword
+
+**Layer 3: Perfect Recall** (Code/transcripts)
+- Source code when implementation details are needed
+- Original transcripts for exact quotes
+
+### The Problem It Solves
+
+**Current version** (v4.2.x): Shows only session-level summaries at startup
+- ✅ Minimal tokens (~800)
+- ❌ Claude doesn't know what detailed observations exist
+- ❌ Often re-reads code to understand past decisions
+
+**Experimental version**: Shows observation index + session summaries
+- ✅ Claude sees WHAT learnings exist without loading full content
+- ✅ Token counts help Claude decide: "fetch details" vs "read code"
+- ✅ Progressive disclosure instructions teach Claude how to use the system
+- ⚠️ Higher token cost (~2,500) but potentially more efficient overall
+
+### How It's Different
+
+The experimental context hook displays observations in a **table format**:
+
+```markdown
+**src/hooks/context.ts**
+| ID | Time | T | Title | Tokens |
+|----|------|---|-------|--------|
+| #2332 | 1:07 AM | 🔴 | Critical Bugfix: Session ID NULL Constraint | ~201 |
+| #2340 | 1:10 AM | 🟠 | Remove Redundant Summary Section | ~280 |
+```
+
+Now Claude knows:
+- A critical bugfix exists about session IDs (~201 tokens to fetch)
+- A design decision exists about summary sections (~280 tokens)
+- Whether to use MCP search or just read the current code
+
+### Try It Out
+
+```bash
+# Clone and checkout the experimental branch
+git clone https://github.com/thedotmack/claude-mem.git
+cd claude-mem
+git checkout feature/context-with-observations
+
+# Build the experimental version
+npm install
+npm run build
+
+# Test the new context hook (from YOUR project directory)
+cd /path/to/your/project
+node /path/to/claude-mem/plugin/scripts/context-hook.js
+
+# Example:
+# cd ~/my-app
+# node ~/claude-mem/plugin/scripts/context-hook.js
+```
+
+**Important:** Run the context hook from your project's root directory to see context specific to that project.
+
+**We want to know:**
+- Does Claude use MCP search more effectively?
+- Do token counts influence retrieval decisions?
+- Is the progressive disclosure guidance helpful or noisy?
+- Does it reduce redundant code reading?
+
+**📣 Share Your Feedback:** [Open a GitHub Issue](https://github.com/thedotmack/claude-mem/issues/new) with your experience! Tag it with `feedback: progressive-disclosure`
+
+---
+
 ## Quick Start
 
 Start a new Claude Code session in the terminal and enter the following commands:
diff --git a/package.json b/package.json
index 488d3e56..15e1a00f 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
 {
   "name": "claude-mem",
-  "version": "4.2.8",
+  "version": "4.2.9",
   "description": "Memory compression system for Claude Code - persist context across sessions",
   "keywords": [
     "claude",
diff --git a/plugin/.claude-plugin/plugin.json b/plugin/.claude-plugin/plugin.json
index b9f2bdf5..e2e51fbb 100644
--- a/plugin/.claude-plugin/plugin.json
+++ b/plugin/.claude-plugin/plugin.json
@@ -1,6 +1,6 @@
 {
   "name": "claude-mem",
-  "version": "4.2.8",
+  "version": "4.2.9",
   "description": "Persistent memory system for Claude Code - seamlessly preserve context across sessions",
   "author": {
     "name": "Alex Newman"