Files
claude-mem/test-results/test-17-all-bugfixes.json
T
Alex Newman c5e68a17c8 refactor: Clean up search architecture, remove experimental contextualize endpoint (#133)
* Refactor code structure for improved readability and maintainability

* Add test results for search API and related functionalities

- Created test result files for various search-related functionalities, including:
  - test-11-search-server-changes.json
  - test-12-context-hook-changes.json
  - test-13-worker-service-changes.json
  - test-14-patterns.json
  - test-15-gotchas.json
  - test-16-discoveries.json
  - test-17-all-bugfixes.json
  - test-18-all-features.json
  - test-19-all-decisions.json
  - test-20-session-search.json
  - test-21-prompt-search.json
  - test-22-decisions-endpoint.json
  - test-23-changes-endpoint.json
  - test-24-how-it-works-endpoint.json
  - test-25-contextualize-endpoint.json
  - test-26-timeline-around-observation.json
  - test-27-multi-param-combo.json
  - test-28-file-type-combo.json

- Each test result file captures specific search failures or outcomes, including issues with undefined properties and successful execution of search queries.
- Enhanced documentation of search architecture and testing strategies, ensuring compliance with established guidelines and improving overall search functionality.

* feat: Enhance unified search API with catch-all parameters and backward compatibility

- Implemented a unified search API at /api/search that accepts catch-all parameters for filtering by type, observation type, concepts, and files.
- Maintained backward compatibility by keeping granular endpoints functional while routing through the same infrastructure.
- Completed comprehensive testing of search capabilities with real-world query scenarios.

fix: Address missing debug output in search API query tests

- Flushed PM2 logs and executed search queries to verify functionality.
- Diagnosed absence of "Raw Chroma" debug messages in worker logs, indicating potential issues with logging or query processing.

refactor: Improve build and deployment pipeline for claude-mem plugin

- Successfully built and synced all hooks and services to the marketplace directory.
- Ensured all dependencies are installed and up-to-date in the deployment location.

feat: Implement hybrid search filters with 90-day recency window

- Enhanced search server to apply a 90-day recency filter to Chroma results before categorizing by document type.

fix: Correct parameter handling in searchUserPrompts method

- Added support for filter-only queries and improved dual-path logic for clarity.

refactor: Rename FTS5 method to clarify fallback status

- Renamed escapeFTS5 to escapeFTS5_fallback_when_chroma_unavailable to indicate its temporary usage.

feat: Introduce contextualize tool for comprehensive project overview

- Added a new tool to fetch recent observations, sessions, and user prompts, providing a quick project overview.

feat: Add semantic shortcut tools for common search patterns

- Implemented 'decisions', 'changes', and 'how_it_works' tools for convenient access to frequently searched observation categories.

feat: Unified timeline tool supports anchor and query modes

- Combined get_context_timeline and get_timeline_by_query into a single interface for timeline exploration.

feat: Unified search tool added to MCP server

- New tool queries all memory types simultaneously, providing combined chronological results for improved search efficiency.

* Refactor search functionality to clarify FTS5 fallback usage

- Updated `worker-service.cjs` to replace FTS5 fallback function with a more descriptive name and improved error handling.
- Enhanced documentation in `SKILL.md` to specify the unified API endpoint and clarify the behavior of the search engine, including the conditions under which FTS5 is used.
- Modified `search-server.ts` to provide clearer logging and descriptions regarding the fallback to FTS5 when UVX/Python is unavailable.
- Renamed and updated the `SessionSearch.ts` methods to reflect the conditions for using FTS5, emphasizing the lack of semantic understanding in fallback scenarios.

* feat: Add ID-based fetch endpoints and simplify mem-search skill

**Problem:**
- Search returns IDs but no way to fetch by ID
- Skill documentation was bloated with too many options
- Claude wasn't using IDs because we didn't tell it how

**Solution:**
1. Added three new HTTP endpoints:
   - GET /api/observation/:id
   - GET /api/session/:id
   - GET /api/prompt/:id

2. Completely rewrote SKILL.md:
   - Stripped complexity down to essentials
   - Clear 3-step prescriptive workflow: Search → Review IDs → Fetch by ID
   - Emphasized ID usage: "The IDs are there for a reason - USE THEM"
   - Removed confusing multi-endpoint documentation
   - Kept only unified search with filters

**Impact:**
- Token efficiency: Claude can now fetch full details only for relevant IDs
- Clarity: One clear workflow instead of 10+ options to choose from
- Usability: IDs are no longer wasted context - they're actionable

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* chore: Move internal docs to private directory

Moved POSTMORTEM and planning docs to ./private to exclude from PR reviews.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: Remove experimental contextualize endpoint

- Removed contextualize MCP tool from search-server (saves ~4KB)
- Disabled FTS5 fallback paths in SessionSearch (now vector-first)
- Cleaned up CLAUDE.md documentation
- Removed contextualize-rewrite-plan.md doc

Rationale:
- Contextualize is better suited as a skill (LLM-powered) than an endpoint
- Search API already provides vector search with configurable limits
- Created issue #132 to track future contextualize skill implementation

Changes:
- src/servers/search-server.ts: Removed contextualize tool definition
- src/services/sqlite/SessionSearch.ts: Disabled FTS5 fallback, added deprecation warnings
- CLAUDE.md: Cleaned up outdated skill documentation
- docs/: Removed contextualize plan document

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: Complete FTS5 cleanup - remove all deprecated search code

This completes the FTS5 cleanup work by removing all commented-out
FTS5 search code while preserving database tables for backward compatibility.

Changes:
- Removed 200+ lines of commented FTS5 search code from SessionSearch.ts
- Removed deprecated degraded_search_query__when_uvx_unavailable method
- Updated all method documentation to clarify vector-first architecture
- Updated class documentation to reflect filter-only query support
- Updated CLAUDE.md to remove FTS5 search references
- Clarified that FTS5 tables exist for backward compatibility only
- Updated "Why SQLite FTS5" section to "Why Vector-First Search"

Database impact: NONE - FTS5 tables remain intact for existing installations

Search architecture:
- ChromaDB: All text-based vector search queries
- SQLite: Filter-only queries (date ranges, metadata, no query text)
- FTS5 tables: Maintained but unused (backward compatibility)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* refactor: Remove all FTS5 fallback execution code from search-server

Completes the FTS5 cleanup by removing all fallback execution paths
that attempted to use FTS5 when ChromaDB was unavailable.

Changes:
- Removed all FTS5 fallback code execution paths
- When ChromaDB fails or is unavailable, return empty results with helpful error messages
- Updated all deprecated tool descriptions (search_observations, search_sessions, search_user_prompts)
- Changed error messages to indicate FTS5 fallback has been removed
- Added installation instructions for UVX/Python when vector search is unavailable
- Updated comments from "hybrid search" to "vector-first search"
- Removed ~100 lines of dead FTS5 fallback code

Database impact: NONE - FTS5 tables remain intact (backward compatibility)

Search behavior when ChromaDB unavailable:
- Text queries: Return empty results with error explaining ChromaDB is required
- Filter-only queries (no text): Continue to work via direct SQLite

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: Address PR 133 review feedback

Critical fixes:
- Remove contextualize endpoint from worker-service (route + handler)
- Fix build script logging to show correct .cjs extension (was .mjs)

Documentation improvements:
- Add comprehensive FTS5 retention rationale documentation
- Include v7.0.0 removal TODO for future cleanup

Testing:
- Build succeeds with correct output logging
- Worker restarts successfully (30th restart)
- Contextualize endpoint properly removed (404 response)
- Search endpoint verified working

This addresses all critical review feedback from PR 133.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-21 18:59:23 -05:00

1 line
8.2 KiB
JSON

[{"type":"text","text":"## Fixed Incorrect Parameter Array in searchUserPrompts FTS5 Path\n*Source: claude-mem://observation/10756*\n\n**Changed params to ftsParams in FTS5 query execution to use correct parameter array with ftsQuery.**\n\nA parameter array mismatch bug was fixed in the searchUserPrompts method's FTS5 code path. The method creates two separate parameter arrays: 'params' for the filter-only path and 'ftsParams' for the FTS5 path. The FTS5 path correctly initialized ftsParams with the escaped query and rebuilt all filter conditions into this new array, but then incorrectly used the 'params' array when executing the SQL query. This would have caused a parameter binding mismatch where the SQL query expected parameters in one order (starting with ftsQuery) but received them in a different order or with missing values. The fix ensures that when the FTS5 path is taken (query text provided), the query uses ftsParams.push() for limit/offset and passes ftsParams to the db.prepare().all() call, maintaining correct parameter alignment throughout the FTS5 execution path.\n\n---\nType: bugfix | Facts: File modified: /Users/alexnewman/Scripts/claude-mem/src/services/sqlite/SessionSearch.ts; Bug was in searchUserPrompts method's FTS5 execution path using wrong parameter array; Changed params.push(limit, offset) to ftsParams.push(limit, offset); Changed this.db.prepare(sql).all(...params) to this.db.prepare(sql).all(...ftsParams); The ftsParams array was created separately for FTS5 path but not being used in query execution; Bug would have caused SQL parameter binding mismatch in FTS5 search path for user prompts | Concepts: problem-solution, what-changed, gotcha | Files: /Users/alexnewman/Scripts/claude-mem/src/services/sqlite/SessionSearch.ts\n\n---\nDate: 11/17/2025, 11:50:17 PM\n\n---\n\n## Diagnosing undefined replace() error in search endpoint\n*Source: claude-mem://observation/10709*\n\n**Sequential thinking initiated to locate source of undefined variable causing replace() method failure in search-server.ts.**\n\nA systematic debugging approach using sequential thinking was initiated to diagnose and fix a runtime error in the search functionality. The error occurs when the replace() method is called on an undefined variable within the search endpoint handler in search-server.ts. Rather than applying a null check workaround, the investigation aims to identify the root cause of why the variable is undefined in the first place. This represents the first thought in a planned 10-step sequential analysis to properly resolve the issue.\n\n---\nType: bugfix | Facts: Error message indicates \"Cannot read properties of undefined (reading 'replace')\" in search code; Sequential thinking process started with thought 1 of 10 planned thoughts; Investigation focuses on query processing in search-server.ts file; The replace() method is being called on an undefined variable in the search endpoint handler; Working directory is /Users/alexnewman/Scripts/claude-mem | Concepts: problem-solution, how-it-works, pattern\n\n---\nDate: 11/17/2025, 10:06:23 PM\n\n---\n\n## Patterns Concept Query Failing with Undefined Replace Error\n*Source: claude-mem://observation/10698*\n\n**Test 14 concept-based query for patterns returns search failure with undefined replace error**\n\nThe concept-based query test for patterns revealed a runtime error in the search API. When querying /api/search with concepts=pattern parameter, the server returns \"Cannot read properties of undefined (reading 'replace')\" instead of observation results. This error pattern typically indicates that a string manipulation operation (replace) is being called on a null or undefined value, likely during query parameter processing or result formatting in search-server.ts. The failure occurs specifically with concept-based filtering, suggesting the concepts parameter handling code path has a bug where an expected string value is missing or undefined. This is a critical issue as concept-based queries are essential for retrieving observations by knowledge category (pattern, gotcha, trade-off, etc.).\n\n---\nType: bugfix | Facts: Test query /api/search?type=observations&concepts=pattern&format=full&limit=5&orderBy=date_desc fails with error \"Cannot read properties of undefined (reading 'replace')\"; Error suggests null or undefined value being passed to string replace operation in search-server.ts; Test file test-results/test-14-patterns.json contains error response instead of observation results | Concepts: problem-solution, gotcha | Files: test-results/test-14-patterns.json\n\n---\nDate: 11/17/2025, 9:53:47 PM\n\n---\n\n## Chroma Internal Error Detection During Testing\n*Source: claude-mem://observation/10669*\n\n**Seven Chroma query failures detected with \"Error finding id\" internal errors requiring investigation despite test suite completion.**\n\nWhile the search test suite reported successful completion of all 28 queries, worker logs reveal that 7 Chroma semantic search queries failed with internal errors. The error pattern indicates Chroma's MCP tool is encountering an \"Error finding id\" issue when querying the 'cm__claude-mem' collection. This manifests as unparseable responses where Chroma returns error text instead of JSON, causing the search server's JSON.parse() to fail. The specific error \"Error executing plan: Internal error: Error finding id\" suggests a data integrity issue within the Chroma vector database, possibly corrupted document IDs or missing metadata. Despite these failures, the search system appears to have gracefully degraded - tests completed and likely fell back to non-semantic results. This issue needs investigation to determine if certain documents have malformed IDs, if the collection needs rebuilding, or if there's a deeper compatibility issue between the Chroma MCP server and the current data schema.\n\n---\nType: bugfix | Facts: Seven instances of \"Failed to parse Chroma response as JSON\" errors found in worker logs; Chroma error message shows: \"Error executing tool chroma_query_documents: Failed to query documents from collection 'cm__claude-mem': Error executing plan: Internal error: Error finding id\"; Errors occurred during test suite execution despite all 28 tests reporting as complete; JSON parsing fails because Chroma returns error text starting with \"Error exec\" instead of valid JSON; Test suite continued execution and completed successfully despite these internal Chroma failures | Concepts: problem-solution, gotcha\n\n---\nDate: 11/17/2025, 9:38:06 PM\n\n---\n\n## Chroma Semantic Search Fix Validated\n*Source: claude-mem://observation/10667*\n\n**Chroma integration now successfully returns 92 semantic matches for worker service query after fixing response parsing.**\n\nThe Chroma semantic search integration was validated as working correctly following the earlier fix for response parsing. A test query searching for \"worker service\" within observations successfully triggered Chroma's semantic search, which identified 92 semantically relevant matches from the vector database. The system correctly limited results to the top 2 matches as requested. The absence of parsing errors in the logs confirms that the fix for handling Chroma's response format is working properly. Both returned observations are contextually relevant to worker service topics - one about build and restart workflows, another about testing phases. This validation confirms that the unified search API can now leverage Chroma's semantic capabilities to find conceptually related content beyond simple keyword matching, significantly enhancing search quality for the testing phase ahead.\n\n---\nType: bugfix | Facts: Test query for \"worker service\" with type=observations and limit=2 executed against localhost:37777/api/search; Chroma returned 92 semantic matches for the query, demonstrating successful semantic search functionality; Search results correctly returned 2 observations: \"Build, Sync, and Worker Restart Executed\" and \"Phase 3 Testing Initiated\"; Worker logs show \"Chroma returned 92 semantic matches\" with no \"Failed to parse\" errors; Response properly formatted with metadata including observation type, date, and claude-mem:// source URIs | Concepts: problem-solution, how-it-works\n\n---\nDate: 11/17/2025, 9:37:40 PM"}]