GitLab

## Problem

When a proxy is unreachable, token refresh retries up to 4 times with
30s timeout each, causing requests to hang for ~2 minutes before
failing with a generic 502 error. The failed account is not marked,
so subsequent requests keep hitting it.

## Changes

### Proxy connection fast-fail
- Set TCP dial timeout to 5s and TLS handshake timeout to 5s on
  antigravity client, so proxy connectivity issues fail within 5s
  instead of 30s
- Reduce overall HTTP client timeout from 30s to 10s
- Export `IsConnectionError` for service-layer use
- Detect proxy connection errors in `RefreshToken` and return
  immediately with "proxy unavailable" error (no retries)

### Token refresh temp-unschedulable
- Add 8s context timeout for token refresh on request path
- Mark account as temp-unschedulable for 10min when refresh fails
  (both background `TokenRefreshService` and request-path
  `GetAccessToken`)
- Sync temp-unschedulable state to Redis cache for immediate
  scheduler effect
- Inject `TempUnschedCache` into `AntigravityTokenProvider`

### Account failover
- Return `UpstreamFailoverError` on `GetAccessToken` failure in
  `Forward`/`ForwardGemini` to trigger handler-level account switch
  instead of returning 502 directly

### Proxy probe alignment
- Apply same 5s dial/TLS timeout to shared `httpclient` pool
- Reduce proxy probe timeout from 30s to 10s

从上游 /v1/messages 响应头被动采集 5h/7d utilization 并存储到
Account.Extra，页面加载时直接读取本地数据而非调用外部 Usage API。
用户可点击"查询"按钮主动拉取最新数据，主动查询结果自动回写被动缓存。

后端:
- UpdateSessionWindow 合并采集 5h + 7d headers 为单次 DB 写入
- 新增 GetPassiveUsage 从 Extra 构建 UsageInfo (复用 estimateSetupTokenUsage)
- GetUsage 主动查询后 syncActiveToPassive 回写被动缓存
- passive_usage_ 前缀注册为 scheduler-neutral

前端:
- Anthropic 账号 mount/refresh 默认 source=passive
- 新增"被动采样"标签和"查询"按钮 (带 loading 动画)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

When all failover accounts are exhausted, handleFailoverExhausted maps
the upstream status code (e.g. 403) to a client-facing code (e.g. 502)
but did not write the original code to the gin context. This caused ops
error logs to show the mapped code instead of the real upstream code.

Call SetOpsUpstreamError before mapUpstreamError in all failover-
exhausted paths so that ops_error_logger captures the true upstream
status code and message.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Move 529 overload cooldown configuration from config file to admin
settings UI. Adds an enable/disable toggle and configurable cooldown
duration (1-120 min) under /admin/settings gateway tab, stored as
JSON in the settings table.

When disabled, 529 errors are logged but accounts are no longer
paused from scheduling. Falls back to config file value when DB
is unreachable or settingService is nil.

Update model mapping target for claude-haiku-4-5 and
claude-haiku-4-5-20251001 from claude-sonnet-4-5 to claude-sonnet-4-6.
Includes migration script, default constants, and test updates.

Empty text blocks ({"type":"text","text":""}) cause Anthropic upstream to
return 400: "text content blocks must be non-empty". This was not caught
by the existing error detection pattern in isThinkingBlockSignatureError,
nor handled by FilterThinkingBlocksForRetry.

- Add empty text block stripping to FilterThinkingBlocksForRetry
- Fix isThinkingBlockSignatureError to match new Anthropic error format
- Add fast-path byte patterns to avoid unnecessary JSON parsing
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Claude Code v2.1.78 起将 metadata.user_id 从拼接字符串改为 JSON：
旧: user_{hex}_account_{uuid}_session_{uuid}
新: {"device_id":"...","account_uuid":"...","session_id":"..."}

新增集中解析/格式化模块 metadata_userid.go：
- ParseMetadataUserID: 自动识别两种格式，提取 DeviceID/AccountUUID/SessionID
- FormatMetadataUserID: 根据 UA 版本输出对应格式（>= 2.1.78 输出 JSON）
- ExtractCLIVersion: 从 UA 提取版本号，消除与 ClaudeCodeValidator.ExtractVersion 的重复

修改消费者统一使用新模块：
- claude_code_validator: 用 ParseMetadataUserID 替代只匹配旧格式的 userIDPattern
- identity_service: RewriteUserID/WithMasking 增加 fingerprintUA 参数，
  解析用 ParseMetadataUserID，输出用 FormatMetadataUserID（版本感知）
- gateway_service: GenerateSessionHash 用 ParseMetadataUserID 提取 session_id，
  buildOAuthMetadataUserID 用 FormatMetadataUserID 输出版本匹配格式，
  两处 RewriteUserIDWithMasking 调用传入 fp.UserAgent
- account_test_service: generateSessionString 改用 FormatMetadataUserID，
  自动跟随 DefaultHeaders UA 版本

删除三个旧正则: userIDPattern, userIDRegex, sessionIDRegex
统一 hex 匹配为 [a-fA-F0-9]，修复旧 userIDRegex 只匹配小写的不一致

复用 GroupCapacityService，在 admin 分组列表中添加容量列，
显示每个分组的实时并发/会话/RPM 使用量和上限。
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

分组管理列表增强：

1. 今日/累计用量列：
   - 新增独立端点 GET /admin/groups/usage-summary
   - 一次查询返回所有分组的今日费用和累计费用（actual_cost）
   - 前端异步加载后合并显示在分组列表中

2. 账号数区分可用/限流/总量：
   - 将账号数列从单一总量改为 badge 内多行展示
   - 可用: active + schedulable 的账号数（绿色）
   - 限流: rate_limit/overload/temp_unschedulable 的账号数（橙色，无限流时隐藏）
   - 总量: 全部关联账号数
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add a platform filter dropdown to the admin subscriptions view, allowing
filtering subscriptions by platform (Anthropic, OpenAI, Gemini, etc.)
through the group association.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Cover IsValidModelSource/NormalizeModelSource, resolveModelDimensionExpression SQL expressions, invalid model_source 400 responses on both GetModelStats and GetUserBreakdown, upstream_model in scan/insert SQL mock expectations, and updated passthrough/billing test signatures.

Support querying model statistics by 'requested', 'upstream', or 'mapping' dimension. Add resolveModelDimensionExpression for safe SQL expression generation, IsValidModelSource whitelist validator, and NormalizeModelSource fallback. Repository persists and scans upstream_model in all insert/select paths.

Propagate UpstreamModel through ForwardResult and OpenAIForwardResult in Anthropic direct, API-key passthrough, Bedrock, and OpenAI gateway flows. Extract optionalNonEqualStringPtr and optionalTrimmedStringPtr into usage_log_helpers.go. Store upstream_model only when it differs from the requested model.

Also introduces anthropicPassthroughForwardInput struct to reduce parameter count.

- Add compile-time interface assertion for sessionWindowMockRepo
- Fix flaky fallback test by capturing time.Now() before calling UpdateSessionWindow
- Replace stale hardcoded timestamps with dynamic future values
- Add millisecond detection and bounds validation for reset header timestamp
- Use pause/resume pattern for interval in UsageProgressBar to avoid idle timers on large lists
- Fix gofmt comment alignment
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

TestConnection now reuses antigravityRetryLoop instead of a standalone
HTTP loop, gaining credits overages, smart retry, and 429/503 backoff
for free. AccountSwitchError is caught and surfaced as a friendly
message. Also populates RateLimitedModel in TempUnscheduled switch error.

Test fixes:
- Use RATE_LIMIT_EXCEEDED in 503 short-delay test to avoid 60x1s timeout
- Clamp waitDuration=0 instead of 999s to avoid 15s max-wait timeout
- Enhance mockSmartRetryUpstream with repeatLast and body caching

Click on a group name, model name, or endpoint name in the distribution
tables to expand and show per-user usage breakdown (requests, tokens,
actual cost, standard cost).

Backend: new GET /admin/dashboard/user-breakdown API with group_id,
model, endpoint, endpoint_type filters.
Frontend: clickable rows with expand/collapse sub-table in all three
distribution charts.

The 5h window reset time displayed for Setup Token accounts was inaccurate
because UpdateSessionWindow predicted the window end as "current hour + 5h"
instead of reading the actual `anthropic-ratelimit-unified-5h-reset` response
header. This caused the countdown to differ from the official Claude page.

Backend: parse the reset header (Unix timestamp) and use it as the real
window end, falling back to the hour-truncated prediction only when the
header is absent. Also correct stale predictions when a subsequent request
provides the real reset time.

Frontend: add a reactive 60s timer so the reset countdown in
UsageProgressBar ticks down in real-time instead of freezing at the
initial value.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

OAuth upstreams (ChatGPT) reject requests containing role:"system" in
the input array with HTTP 400 "System messages are not allowed". Extract
such items before forwarding and merge their content into the top-level
instructions field, prepending to any existing value.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

POST /backups 和 POST /backups/:id/restore 改为异步：立即返回 HTTP 202，
后台 goroutine 独立执行 pg_dump → gzip → S3 上传，前端每 2s 轮询状态。

后端:
- 新增 StartBackup/StartRestore 方法，后台 goroutine 不依赖 HTTP 连接
- Graceful shutdown 等待活跃操作完成，启动时清理孤立 running 记录
- BackupRecord 新增 progress/restore_status 字段支持进度和恢复状态追踪

前端:
- 创建备份/恢复后轮询 GET /backups/:id 直到完成或失败
- 标签页切换暂停/恢复轮询，组件卸载清理定时器
- 正确处理 409（备份进行中）和轮询超时
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Antigravity streaming handlers were missing the keepalive mechanism
that exists in the standard gateway, causing proxy/CDN idle timeouts
to break connections during long thinking phases (e.g. claude-opus-4-6).
This resulted in truncated responses with missing tool calls.

Add StreamKeepaliveInterval support to all three Antigravity streaming
paths: Claude SSE, Gemini SSE, and upstream passthrough.

Removes hasMeaningfulWindowStats guard so the /usage endpoint consistently
returns WindowStats for both time windows. The frontend now controls
zero-value display filtering at the component level.

UpdateAccount previously required len(input.Extra) > 0, causing explicit
empty payloads (extra:{}) to be silently skipped. Change condition to
input.Extra != nil so clearing quota keys actually persists.

正常终端事件（response.completed 等）退出后连接归还复用，
仅异常路径（读写错误、error 事件、客户端断连）MarkBroken 销毁。

Generate 模式:
- 引入 cleanExit 标记，仅在 isTerminalEvent break 时设置 true
- defer 中根据 cleanExit 决定是否 MarkBroken
- 所有异常路径已在各自分支中提前调用 MarkBroken

Ingress 模式:
- 引入 lastTurnClean 标记，sendAndRelay 正常完成时设为 true
- releaseSessionLease 根据 lastTurnClean 决定是否 MarkBroken
- 错误路径重置 lastTurnClean = false
- 客户端断连后 drain 仍保守 MarkBroken（L2916）

根因：多个用户共享同一 OAuth 账号时，conversation_id/session_id 头
未做用户隔离，导致上游 chatgpt.com 将不同用户的请求关联到同一会话。

HTTP SSE 修复:
- 新增 isolateOpenAISessionID(apiKeyID, raw)，将 API Key ID 混入
  session 标识符（xxhash），确保不同 Key 的用户产生不同上游会话
- buildUpstreamRequest: OAuth 分支先 Del 客户端透传的 session 头，
  再用隔离值覆盖
- buildUpstreamRequestOpenAIPassthrough: 透传路径同样隔离
- ForwardAsAnthropic: Anthropic Messages 兼容路径同步修复
- buildOpenAIWSHeaders: WS 路径的 OAuth session 头同步隔离

- Fix gofmt alignment in admin_service.go and trailing newline in
  antigravity_credits_overages.go
- Suppress errcheck for fmt.Sscanf in client.go GetMinimumAmount

Remove SimulateClaudeMaxEnabled field and related logic from
admin_service.go, and remove applyClaudeMaxCacheBillingPolicyToUsage,
applyClaudeMaxNonStreamingRewrite, setupClaudeMaxStreamingHook calls
from antigravity_gateway_service.go. These symbols are not yet
available in upstream/main.

Replace process-memory sync.Map + per-model runtime state with a single
"AICredits" key in model_rate_limits, making credits exhaustion fully
isomorphic with model-level rate limiting.

Scheduler: rate-limited accounts with overages enabled + credits available
are now scheduled instead of excluded.

Forwarding: when model is rate-limited + credits available, inject credits
proactively without waiting for a 429 round trip.

Storage: credits exhaustion stored as model_rate_limits["AICredits"] with
5h duration, reusing SetModelRateLimit/isRateLimitActiveForKey.

Frontend: show credits_active (yellow ⚡) when model rate-limited but
credits available, credits_exhausted (red) when AICredits key active.

Tests: add unit tests for shouldMarkCreditsExhausted, injectEnabledCreditTypes,
clearCreditsExhausted, and update existing overages tests.

feat: implement resolveCreditsOveragesModelKey function to stabilize model key resolution for credit overages

Introduce OAuthRefreshAPI as the single entry point for all OAuth token
refresh operations, eliminating the race condition where background
refresh and inline refresh could simultaneously use the same
refresh_token (fixes #1035).

Key changes:
- Add OAuthRefreshExecutor interface extending TokenRefresher with CacheKey
- Add OAuthRefreshAPI.RefreshIfNeeded with lock → DB re-read → double-check flow
- Add ProviderRefreshPolicy / BackgroundRefreshPolicy strategy types
- Simplify all 4 TokenProviders to delegate to OAuthRefreshAPI
- Rewrite TokenRefreshService.refreshWithRetry to use unified API path
- Add MergeCredentials and BuildClaudeAccountCredentials helpers
- Add 40 unit tests covering all new and modified code paths

Extend RecordUsageInput and RecordUsageLongContextInput structs with InboundEndpoint and UpstreamEndpoint so that Claude, Gemini, and Sora handlers can record endpoint info alongside OpenAI handlers.

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-opencode

)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>