feat(session-affinity): add session-sticky routing for multi-account load balancing
When multiple auth credentials are configured, requests from the same
session are now routed to the same credential, improving upstream prompt
cache hit rates and maintaining context continuity.
Core components:
- SessionAffinitySelector: wraps RoundRobin/FillFirst selectors with
session-to-auth binding; automatic failover when bound auth is
unavailable, re-binding via the fallback selector for even distribution
- SessionCache: TTL-based in-memory cache with background cleanup
goroutine, supporting per-session and per-auth invalidation
- StoppableSelector interface: lifecycle hook for selectors holding
resources, called during Manager.StopAutoRefresh()
Session ID extraction priority (extractSessionIDs):
1. metadata.user_id with Claude Code session format (old
user_{hash}_session_{uuid} and new JSON {session_id} format)
2. X-Session-ID header (generic client support)
3. metadata.user_id (non-Claude format, used as-is)
4. conversation_id field
5. Stable FNV hash from system prompt + first user/assistant messages
(fallback for clients with no explicit session ID); returns both a
full hash (primaryID) and a short hash without assistant content
(fallbackID) to inherit bindings from the first turn
Multi-format message hash covers OpenAI messages, Claude system array,
Gemini contents/systemInstruction, and OpenAI Responses API input items
(including inline messages with role but no type field).
Configuration (config.yaml routing section):
- session-affinity: bool (default false)
- session-affinity-ttl: duration string (default "1h")
- claude-code-session-affinity: bool (deprecated, alias for above)
All three fields trigger selector rebuild on config hot reload.
Side effect: Idempotency-Key header is no longer auto-generated with a
random UUID when absent — only forwarded when explicitly provided by the
client, to avoid polluting session hash extraction.
This commit is contained in:
@@ -105,6 +105,13 @@ type Selector interface {
|
||||
Pick(ctx context.Context, provider, model string, opts cliproxyexecutor.Options, auths []*Auth) (*Auth, error)
|
||||
}
|
||||
|
||||
// StoppableSelector is an optional interface for selectors that hold resources.
|
||||
// Selectors that implement this interface will have Stop called during shutdown.
|
||||
type StoppableSelector interface {
|
||||
Selector
|
||||
Stop()
|
||||
}
|
||||
|
||||
// Hook captures lifecycle callbacks for observing auth changes.
|
||||
type Hook interface {
|
||||
// OnAuthRegistered fires when a new auth is registered.
|
||||
@@ -2928,6 +2935,7 @@ func (m *Manager) StartAutoRefresh(parent context.Context, interval time.Duratio
|
||||
}
|
||||
|
||||
// StopAutoRefresh cancels the background refresh loop, if running.
|
||||
// It also stops the selector if it implements StoppableSelector.
|
||||
func (m *Manager) StopAutoRefresh() {
|
||||
m.mu.Lock()
|
||||
cancel := m.refreshCancel
|
||||
@@ -2937,6 +2945,10 @@ func (m *Manager) StopAutoRefresh() {
|
||||
if cancel != nil {
|
||||
cancel()
|
||||
}
|
||||
// Stop selector if it implements StoppableSelector (e.g., SessionAffinitySelector)
|
||||
if stoppable, ok := m.selector.(StoppableSelector); ok {
|
||||
stoppable.Stop()
|
||||
}
|
||||
}
|
||||
|
||||
func (m *Manager) queueRefreshReschedule(authID string) {
|
||||
|
||||
Reference in New Issue
Block a user