Enhance observation parsing and querying functionality

- Filter out observation type from concepts array in parseObservations function to ensure types and concepts are treated as separate dimensions. Added logging for removed types.
- Update prompts documentation to clarify that the observation type must not be included in the concepts array.
- Modify search-server to provide clearer guidance on result limits, emphasizing starting with smaller limits to avoid exceeding token limits.
- Refactor SessionSearch methods to accept options for limit, offset, and orderBy parameters, improving flexibility in querying observations by concept and type.

src/sdk/parser.ts

@@ -68,13 +68,25 @@ export function parseObservations(text: string, correlationId?: string): ParsedO
       continue;
     }
 
+    // Filter out type from concepts array (types and concepts are separate dimensions)
+    const cleanedConcepts = concepts.filter(c => c !== type.trim());
+
+    if (cleanedConcepts.length !== concepts.length) {
+      logger.warn('PARSER', 'Removed observation type from concepts array', {
+        correlationId,
+        type: type.trim(),
+        originalConcepts: concepts,
+        cleanedConcepts
+      });
+    }
+
     observations.push({
       type: type.trim(),
       title,
       subtitle,
       facts,
       narrative,
-      concepts,
+      concepts: cleanedConcepts,
       files_read,
       files_modified
     });

src/sdk/prompts.ts

@@ -55,7 +55,7 @@ Output observations using this XML structure:
 <observation>
   <type>[ change | discovery | decision ]</type>
   <!--
-    **type**: One of:
+    **type**: MUST be EXACTLY one of these 3 options (no other values allowed):
       - change: modifications to code, config, or documentation
       - discovery: learning about existing system
       - decision: choosing an approach and why it was chosen
@@ -79,7 +79,7 @@ Output observations using this XML structure:
     <concept>[knowledge-type-category]</concept>
   </concepts>
   <!--
-    **concepts**: 2-5 knowledge-type categories:
+    **concepts**: 2-5 knowledge-type categories. MUST use ONLY these exact keywords:
       - how-it-works: understanding mechanisms
       - why-it-exists: purpose or rationale
       - what-changed: modifications made
@@ -87,6 +87,9 @@ Output observations using this XML structure:
       - gotcha: traps or edge cases
       - pattern: reusable approach
       - trade-off: pros/cons of a decision
+
+    IMPORTANT: Do NOT include the observation type (change/discovery/decision) as a concept.
+    Types and concepts are separate dimensions.
   -->
   <files_read>
     <file>[path/to/file]</file>