claude-mem/docs/context/windows-bun-worker-struggles.md

Windows, Bun, and Worker Service Struggles

A comprehensive chronicle of platform-specific issues, attempted fixes, and architectural decisions.

Executive Summary

The claude-mem project has faced persistent Windows-specific issues centered around three core problems:

1. Console Window Popups: Blank terminal windows appearing when spawning worker and SDK subprocess
2. Zombie Socket Issues: Bun leaving TCP sockets in LISTEN state after termination on Windows
3. Process Management Complexity: Platform-specific spawning logic and reliability issues

These issues have driven multiple PRs, architectural pivots, and significant debate about runtime switching (Bun → Node.js).

---

Timeline of Issues

Issue #209: Windows Worker Startup Failures (Dec 12-13, 2025)

Problem: Worker service failed to start on Windows using PowerShell Start-Process approach.

Symptoms:

• Worker startup attempted via powershell.exe -NoProfile -NonInteractive -Command Start-Process
• Health check retries exhausted (15 attempts over 15 seconds)
• Users left unable to start worker manually

Root Causes:

• Platform-conditional process spawning (PowerShell for Windows, PM2 for Unix)
• PowerShell spawning without -PassThru to capture PID
• Inconsistent process management across platforms

Resolution: Issue was marked as closed, suggesting it was resolved in v7.1.0 through architectural unification with Bun-based ProcessManager using PID file tracking consistently across all platforms.

Status: ✅ Resolved (pre-PR #335)

---

Issue #309 & PR #315: Console Window Popups (Dec 14-15, 2025)

Problem: Blank terminal windows appear when spawning worker processes and SDK subprocesses on Windows.

First Attempted Fix (PR #315): Add windowsHide: true to spawn options

Why It Failed: Node.js bug #21825 - windowsHide: true is ignored when detached: true is also set. Both flags are required:

• detached: true - Needed for background process
• windowsHide: true - Needed to hide window (but doesn't work when detached)

Testing Results (by ToxMox):

• Tested PR #315 on Windows 11
• Confirmed blank terminal windows still appear for both worker and SDK subprocess spawns
• Affects both ProcessManager.ts (worker) and SDKAgent.ts (SDK subprocess)

Working Solution: Use PowerShell's Start-Process with -WindowStyle Hidden flag instead of standard spawn.

Status: ❌ PR #315 closed in favor of more comprehensive solution

---

Bun Zombie Socket Issue (Dec 15, 2025)

Problem: Bun leaves TCP sockets in zombie LISTEN state on Windows after worker termination.

Symptoms:

• Port remains bound even though no process owns it
• OwningProcess shows 0 or dead PID
• New worker instances cannot start due to EADDRINUSE errors
• Happens regardless of termination method (process.exit(), external kill, Ctrl+C)
• Only system reboot clears zombie ports

Upstream Tracking:

• Bun issue #12127
• Bun issue #5774
• Bun issue #8786

Impact: Windows users may need to reboot their systems when worker crashes or is restarted.

Proposed Solution: Switch worker runtime from Bun to Node.js on Windows (or globally).

Status: 🟡 Unresolved - Platform-specific bug in Bun's Windows socket cleanup

---

SDK Subprocess Hang Issue (Dec 15, 2025)

Problem: SDK subprocesses can hang indefinitely, blocking observation processing.

Root Cause: AbortController.abort() does not actually terminate child processes.

Symptoms:

• For-await loop blocks forever waiting for output from hung subprocess
• Observation processing halts
• No recovery mechanism

Solution: Implement watchdog timer that explicitly kills child processes using platform-specific commands:

• Windows: wmic process where ParentProcessId=<pid> delete
• Unix: pkill -P <pid>

Timeout: SDK_QUERY_TIMEOUT_MS set to 2 minutes

Status: ✅ Fixed in PR #335 (watchdog implementation)

---

PR #335: Comprehensive Windows Fix (Dec 15, 2025)

What It Attempted

ToxMox developed a comprehensive PR addressing all Windows issues simultaneously:

1. PowerShell-based spawning to fix popup windows
2. Runtime switch from Bun to Node.js (globally) to fix zombie sockets
3. Queue monitoring system with persistent message queue
4. Watchdog service for stuck message recovery
5. SQLite compatibility layer for Node.js support

Architecture Decisions

ProcessManager Changes:

• Switched from startWithBun() to startWithNode()
• Windows: Uses PowerShell Start-Process -WindowStyle Hidden -PassThru
• Unix: Uses standard spawn() with detached: true
• Captures PID via PowerShell Select-Object -ExpandProperty Id
• Comment states: "Use Node on all platforms (Bun has zombie socket issues on Windows)"

SQLite Compatibility Layer:

• Created sqlite-compat.ts adapter pattern
• Provides bun:sqlite API compatibility via better-sqlite3
• Allows code to work with both Bun and Node.js runtimes

Critical Issues Identified

1. Global vs Platform-Conditional Runtime

The Inconsistency: Code comment explicitly states zombie sockets occur "on Windows", yet solution applies Node.js universally across all platforms.

Questions Raised:

• Why sacrifice Bun's performance on macOS/Linux where no issues documented?
• Platform-specific spawning already implemented - why not platform-specific runtime?
• No documented Bun reliability issues on non-Windows platforms

2. Performance Regressions

better-sqlite3 Blocking:

• Synchronous-only API blocks Node.js event loop during all DB operations
• Contrasts with Bun's async SQLite support
• Affects: enqueue, markProcessing, markProcessed, watchdog checks

Watchdog Polling Overhead:

• Full table scans every 30 seconds even when idle
• Constant database I/O overhead
• No max queue size limits = unbounded growth

Startup Latency:

• Node.js initialization (slower than Bun)
• Native module loading (better-sqlite3)
• Database migrations
• Stuck message scan
• Watchdog initialization
• HTTP server startup

3. Build Dependencies

better-sqlite3 Requirements:

• node-gyp
• Python
• C++ compiler toolchains
• Visual Studio Build Tools (Windows)

Impact:

• Local development machines without build tools fail
• CI/CD pipelines need updated Docker images
• Restricted environments where compilers not permitted
• ARM/M1 Mac compatibility issues

4. Migration Risks

Breaking Changes:

• Automatic database migration adds pending_messages table
• Runtime switch not documented in PR
• Node.js becomes undocumented hard requirement
• No migration guide or rollback procedure

Unanswered Questions:

• What happens to in-flight messages during upgrade?
• Can users safely downgrade?
• Is migration idempotent?

5. Code Quality Issues

Command Injection Risk (ProcessManager.ts:67):

• PowerShell commands use template literal concatenation
• Vulnerable if MARKETPLACE_ROOT or script paths attacker-controlled
• Should use array-based argument passing

Missing Error Handling (WatchdogService.ts:61):

• setInterval callback lacks error handling
• Timer continues running if check() throws
• Creates zombie watchdog scenario

No Queue Size Limits:

• Unbounded database growth if messages accumulate
• Failed messages (exceeding maxRetries) accumulate indefinitely
• Only 24-hour retention for processed messages

---

Assessment and Recommendations

What Was Validated

Legitimate Windows Issues:

• ✅ Console window popups are real (Node.js bug #21825)
• ✅ PowerShell Start-Process solution works
• ✅ Bun zombie socket issue is real and Windows-specific
• ✅ SDK subprocess hang issue is real

What Remains Questionable

Global Runtime Switch:

• ❌ No evidence Bun problematic on macOS/Linux
• ❌ Platform-conditional runtime not considered
• ❌ Performance trade-offs not documented
• ❌ "Windows-only" issue applied globally

Zombie Socket Root Cause:

• 🟡 May be fixable with proper cleanup handlers:
  • Missing server.close() calls before exit
  • Processes killed with SIGKILL before cleanup finishes
  • Missing SIGTERM signal handlers for graceful shutdown
• 🟡 Runtime switch may be unnecessary over-engineering

Salvageable Components

If Extracted into Separate PRs:

1. PowerShell Spawning for Windows Worker

   • Focused PR: "Windows: Use Node.js instead of Bun for worker process"
   • Platform-conditional logic (Node.js on Windows, Bun elsewhere)
   • Independent justification required

2. SQLite Compatibility Layer

   • Well-designed adapter pattern
   • Requires independent justification for Node.js runtime need
   • Should not be bundled with other changes

3. Queue Monitoring UI Concept

   • Valuable visibility into worker state
   • Should build on in-memory state first
   • Remove database persistence requirement initially

4. Watchdog Improvements

   • SDK subprocess timeout handling
   • Evidence of superiority over current approach needed

---

Current Status

Resolved

• ✅ Issue #209: Windows worker startup (v7.1.0)
• ✅ SDK subprocess hang issue (watchdog implementation)

In Progress

• 🔄 PR #339: Windows console popup fix (extracted from PR #335)
• 🔄 PR #338: Queue monitoring system (extracted from PR #335)

Open Questions

• ❓ Should runtime switch be global or Windows-only?
• ❓ Can zombie socket issue be fixed without runtime switch?
• ❓ Is better-sqlite3's synchronous blocking acceptable?
• ❓ Should queue persistence be in-memory first?

---

Lessons Learned

Architectural Principles Violated

YAGNI: Queue persistence, watchdog service, and comprehensive monitoring added without proven need.

Happy Path: Should have started with simplest Windows fix (PowerShell spawning), validated, then added complexity if needed.

Incremental Validation: Bundling multiple architectural changes prevents isolating what actually solves the problem.

What Should Have Happened

1. Phase 1: PowerShell spawning fix for Windows console popups (targeted, testable)
2. Phase 2: Investigate zombie socket root cause (cleanup handlers vs runtime switch)
3. Phase 3: If runtime switch justified, implement as Windows-conditional first
4. Phase 4: Add queue monitoring as optional feature with in-memory state
5. Phase 5: Add persistence only if in-memory insufficient

Key Takeaways

• Windows-specific issues don't justify global architectural changes without clear evidence
• Platform-conditional logic is acceptable when solving platform-specific problems
• Native module dependencies are heavy - avoid unless necessary
• Performance regressions need explicit justification - synchronous blocking, startup latency, polling overhead all impact UX
• Bundle size matters - build tools, compilers, Python are significant requirements

---

References

GitHub Issues:

• #209: Windows worker startup failures
• #309: Console window popups
• #315: windowsHide approach (closed)

PRs:

• #335: Comprehensive Windows fix (under review)
• #338: Queue monitoring system (extracted)
• #339: Windows console popup fix (extracted)

Upstream Bugs:

• Node.js #21825: windowsHide ignored with detached
• Bun #12127, #5774, #8786: Windows zombie sockets

Related Observations:

• #27302: PR #315 windowsHide failure analysis
• #27233: Bun zombie socket discovery
• #27232: Windows background window root cause
• #27286: Runtime switch assessment
• #27283: PowerShell process spawn fix
• #27190: ProcessManager Node.js implementation
• #24532: Issue #209 resolution

---

Last Updated: 2025-12-16 Document Status: Comprehensive review based on memory search through #S3485