fix: 订阅白名单多 worker 同步并统一上传上限 500MB

保存 IP 白名单时同步刷新并 Redis 广播 login_* 设置;推送/文件上传与 Nginx 模板对齐为 500MB,避免生产 50m/100m 导致大文件失败。

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Nexus Agent
2026-06-11 17:39:26 +08:00
parent 426bafd169
commit 704764bd7e
15 changed files with 321 additions and 57 deletions
+4 -3
View File
@@ -12,7 +12,8 @@ upstream nexus_api {
# WebSocket log without query string (JWT must not hit access_log)
log_format nexus_ws '$remote_addr - [$time_local] "$request_method $uri" $status $body_bytes_sent';
client_max_body_size 100m;
client_max_body_size 500m;
client_body_timeout 600s;
location /api/ {
proxy_pass http://nexus_api;
@@ -21,8 +22,8 @@ location /api/ {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 300s;
proxy_send_timeout 300s;
proxy_read_timeout 600s;
proxy_send_timeout 600s;
proxy_connect_timeout 10s;
}
+1 -1
View File
@@ -416,7 +416,7 @@ server {
}
# Upload limit
client_max_body_size 100m;
client_max_body_size 500m;
# Well-known for SSL cert validation
location ~ \\.well-known {
+1 -1
View File
@@ -75,5 +75,5 @@ server {
error_log {{LOG_DIR}}/{{SERVER_NAME}}_nexus.error.log;
# ── Upload limit ──
client_max_body_size 100m;
client_max_body_size 500m;
}
+1 -1
View File
@@ -109,5 +109,5 @@ server {
error_log {{LOG_DIR}}/{{SERVER_NAME}}.error.log;
# ── Upload limit ──
client_max_body_size 100m;
client_max_body_size 500m;
}
@@ -0,0 +1,40 @@
# Changelog — 登录 IP 白名单订阅刷新多 Worker 同步与保存竞态修复
**日期**2026-06-11
## 摘要
修复设置页保存订阅 URL 后节点 IP 不立即显示、以及多 uvicorn worker 下各进程白名单内存不一致的问题。保存时同步等待订阅拉取完成并返回新 IP;刷新/保存后通过 Redis `nexus:settings` 广播 `login_*` 键。
## 动机
- 保存后立即 `GET /ip-allowlist` 与异步 `_do_refresh` 竞态,UI 常显示旧/空节点。
- 与 2026-06-08 Telegram 开关同类:仅处理请求的 worker 热更新内存,登录校验读各 worker 本地 `LOGIN_SUBSCRIPTION_IPS`,导致订阅更新后登录时好时坏。
## 涉及文件
- `server/infrastructure/settings_broadcast.py` — 新增 `publish_setting_changes`
- `server/background/ip_allowlist_refresh.py``RefreshResult``do_refresh`、刷新后广播
- `server/api/settings.py` — 保存/切换/手动 IP 路径 await 刷新并广播;保存响应含 `subscription_ips` / `last_refresh`
- `frontend/src/types/api.ts``IpAllowlistSaveResponse`
- `frontend/src/pages/SettingsPage.vue` — 保存后用响应更新节点 chip
- `tests/test_ip_allowlist_sync.py`
## 行为变更
- `POST /settings/ip-allowlist`:有订阅 URL 时**同步**拉取订阅后再返回;响应增加 `subscription_ips``last_refresh``refresh_ok``refresh_error`
- 白名单相关 settings 变更(含 2h 定时刷新)广播至全部 worker。
## 迁移 / 重启
- 无 DB 迁移。
- 需重启/部署后端以加载新逻辑(settings subscriber 已存在,无需额外配置)。
## 验证
```bash
pytest tests/test_ip_allowlist_sync.py tests/test_notify_toggle_sync.py -q
cd frontend && npm run type-check
```
设置页:填订阅 URL → 解析预览 → 保存 → 节点 chip 与「上次刷新」立即更新;多 worker 下任意 worker 处理登录均使用同一 IP 列表。
@@ -0,0 +1,37 @@
# Changelog — 上传/推送体积上限统一 500MB
**日期**2026-06-11
## 摘要
将推送/文件上传的 Nginx `client_max_body_size` 与后端 `MAX_UPLOAD_FILE_SIZE`、下载上限统一为 **500MB**,修复生产 OpenResty 全局 50MB 导致 80MB 推送失败的问题。
## 动机
生产 1Panel OpenResty 全局 `client_max_body_size 50m`,站点未覆盖;仓库模板为 100m,与后端 ZIP 500MB 不一致。用户 80MB 上传在 Nginx 层 413 被拒。
## 涉及文件
- `server/api/sync_v2.py``MAX_UPLOAD_FILE_SIZE`、下载 `MAX_DOWNLOAD_SIZE` → 500MB
- `deploy/nginx_http.conf``deploy/nginx_https.conf``deploy/install.sh`
- `deploy/1panel/openresty-nexus.conf.example` — 500m + 600s 上传超时
- `docs/install-guide.md``docs/design/plans/2026-06-04-1panel-docker-production.md`
## 生产变更
- OpenResty 全局 `nginx.conf``50m``500m`
- `api.synaglobal.vip` 站点 `proxy/root.conf`:增加 `client_max_body_size 500m` 与 proxy/body 600s 超时
## 迁移 / 重启
- 无 DB 迁移;重载 OpenResty/Nginx 后生效
- 后端需部署以应用 `sync_v2.py` 单文件 500MB 校验
## 验证
```bash
pytest tests/test_sync_upload_staging.py -q
nginx -t && openresty -s reload # 生产
```
推送页上传 ≤500MB 单文件/ZIP/批次应通过;>500MB 返回明确 400/413。
@@ -223,7 +223,7 @@ docker compose -f docker/docker-compose.prod.yml exec mysql \
| `/ws/` | WebSocket`Upgrade` + `Connection upgrade`,超时 3600s |
| `/health` | 反代(或仅内网探测) |
| `/app/` | 反代(容器内 StaticFiles |
| `client_max_body_size` | `100m`(上传/推送) |
| `client_max_body_size` | `500m`(上传/推送) |
**WS 日志**:勿记录 query 中的 JWT(示例配置已用 `$uri``$args`)。
+1 -1
View File
@@ -268,7 +268,7 @@ server {
return 404;
}
client_max_body_size 100m;
client_max_body_size 500m;
access_log /var/log/nginx/nexus_access.log;
error_log /var/log/nginx/nexus_error.log;
+13 -3
View File
@@ -442,6 +442,7 @@ import type {
OpsPatrolSyncResponse,
SubscriptionParseResponse,
IpAllowlistSaveRequest,
IpAllowlistSaveResponse,
} from '@/types/api'
import { NOTIFY_TOGGLE_ITEMS } from '@/types/api'
import {
@@ -958,10 +959,19 @@ async function saveAllowlist() {
subscription_url: subscriptionUrl.value.trim(),
enabled: ipAllowlistEnabled.value,
}
await http.post('/settings/ip-allowlist', payload)
snackbar('白名单已保存')
const res = await http.post<IpAllowlistSaveResponse>('/settings/ip-allowlist', payload)
subscriptionIps.value = res.subscription_ips || []
allowlistLastRefresh.value = res.last_refresh || null
allowlistTotalCount.value = new Set([
...(res.subscription_ips || []),
...manual_ips,
]).size
if (res.refresh_ok === false && res.refresh_error) {
snackbar(`白名单已保存,但订阅刷新失败:${res.refresh_error}`, 'warning')
} else {
snackbar('白名单已保存')
}
subParseDialog.value = false
await loadAllowlist()
} catch (e: unknown) {
snackbar(e instanceof Error ? e.message : '保存失败', 'error')
} finally {
+10
View File
@@ -106,6 +106,16 @@ export interface IpAllowlistSaveRequest {
enabled?: boolean
}
export interface IpAllowlistSaveResponse {
success: boolean
manual_count: number
subscription_ips?: string[]
subscription_count?: number
last_refresh?: string | null
refresh_ok?: boolean
refresh_error?: string | null
}
/** File browse response from POST /api/sync/browse */
export interface BrowseResponse {
path?: string
+64 -35
View File
@@ -1321,6 +1321,28 @@ class IpAllowlistSaveRequest(BaseModel):
enabled: Optional[bool] = None # None = don't change
def _allowlist_save_response(*, manual_count: int, refresh, has_subscription: bool) -> dict:
"""Build POST /ip-allowlist response with subscription refresh outcome."""
if not has_subscription:
return {
"success": True,
"manual_count": manual_count,
"subscription_ips": [],
"subscription_count": 0,
"last_refresh": None,
"refresh_ok": True,
}
return {
"success": True,
"manual_count": manual_count,
"subscription_ips": list(refresh.subscription_ips),
"subscription_count": refresh.subscription_count,
"last_refresh": refresh.last_refresh or None,
"refresh_ok": refresh.ok,
"refresh_error": refresh.error if not refresh.ok else None,
}
@router.post("/ip-allowlist", response_model=dict)
async def set_ip_allowlist(
payload: IpAllowlistSaveRequest,
@@ -1330,37 +1352,44 @@ async def set_ip_allowlist(
):
"""Save manual IPs, subscription URL, and/or enabled toggle."""
from server.config import settings as _settings
from server.background.ip_allowlist_refresh import RefreshResult, do_refresh
from server.infrastructure.settings_broadcast import publish_setting_changes
if payload.manual_ips:
_validate_ip_list(payload.manual_ips)
repo = SettingRepositoryImpl(db)
# Toggle enabled/disabled
if payload.enabled is not None:
val = "true" if payload.enabled else "false"
await repo.set("login_allowlist_enabled", val)
_settings.LOGIN_ALLOWLIST_ENABLED = val
# Manual IPs
manual_cleaned = [ip.strip() for ip in payload.manual_ips if ip.strip()]
manual_val = ",".join(manual_cleaned)
await repo.set("login_manual_ips", manual_val)
_settings.LOGIN_MANUAL_IPS = manual_val
changes: dict[str, str] = {"login_manual_ips": manual_val}
if payload.enabled is not None:
val = "true" if payload.enabled else "false"
await repo.set("login_allowlist_enabled", val)
changes["login_allowlist_enabled"] = val
# Subscription URL
sub_url_changed = False
if payload.subscription_url is not None:
sub_url = payload.subscription_url.strip()
await repo.set("login_subscription_url", sub_url)
_settings.LOGIN_SUBSCRIPTION_URL = sub_url
sub_url_changed = bool(sub_url)
changes["login_subscription_url"] = sub_url
# If no subscription URL, rebuild combined from manual only
if not _settings.LOGIN_SUBSCRIPTION_URL:
if payload.subscription_url is not None:
sub_url_now = payload.subscription_url.strip()
else:
sub_url_now = (_settings.LOGIN_SUBSCRIPTION_URL or "").strip()
refresh = RefreshResult(ok=True)
if not sub_url_now:
await repo.set("login_allowed_ips", manual_val)
await repo.set("login_subscription_ips", "")
_settings.LOGIN_ALLOWED_IPS = manual_val
_settings.LOGIN_SUBSCRIPTION_IPS = ""
changes["login_allowed_ips"] = manual_val
changes["login_subscription_ips"] = ""
await publish_setting_changes(changes)
if sub_url_now:
refresh = await do_refresh()
audit_repo = AuditLogRepositoryImpl(db)
from server.domain.models import AuditLog
@@ -1373,13 +1402,11 @@ async def set_ip_allowlist(
ip_address=ip_address,
))
# Trigger refresh if subscription URL is present
if sub_url_changed or _settings.LOGIN_SUBSCRIPTION_URL:
from server.background.ip_allowlist_refresh import _do_refresh
import asyncio
asyncio.create_task(_do_refresh())
return {"success": True, "manual_count": len(manual_cleaned)}
return _allowlist_save_response(
manual_count=len(manual_cleaned),
refresh=refresh,
has_subscription=bool(sub_url_now),
)
@router.post("/ip-allowlist/toggle", response_model=dict)
@@ -1390,12 +1417,12 @@ async def toggle_ip_allowlist(
db: AsyncSession = Depends(get_db),
):
"""Quick toggle for allowlist enforcement (no IP list changes)."""
from server.config import settings as _settings
val = "true" if enabled else "false"
repo = SettingRepositoryImpl(db)
await repo.set("login_allowlist_enabled", val)
_settings.LOGIN_ALLOWLIST_ENABLED = val
from server.infrastructure.settings_broadcast import publish_setting_changes
await publish_setting_changes({"login_allowlist_enabled": val})
audit_repo = AuditLogRepositoryImpl(db)
from server.domain.models import AuditLog
@@ -1426,7 +1453,11 @@ async def add_manual_ips(
val = ",".join(merged)
repo = SettingRepositoryImpl(db)
await repo.set("login_manual_ips", val)
_settings.LOGIN_MANUAL_IPS = val
from server.infrastructure.settings_broadcast import publish_setting_changes
from server.background.ip_allowlist_refresh import do_refresh
await publish_setting_changes({"login_manual_ips": val})
await do_refresh()
# S-07: Audit log
audit_repo = AuditLogRepositoryImpl(db)
@@ -1438,9 +1469,6 @@ async def add_manual_ips(
ip_address=request.client.host if request.client else "",
))
from server.background.ip_allowlist_refresh import _do_refresh
import asyncio
asyncio.create_task(_do_refresh())
return {"success": True, "manual_ips": merged}
@@ -1459,7 +1487,11 @@ async def remove_allowlist_ip(
val = ",".join(remaining)
repo = SettingRepositoryImpl(db)
await repo.set("login_manual_ips", val)
_settings.LOGIN_MANUAL_IPS = val
from server.infrastructure.settings_broadcast import publish_setting_changes
from server.background.ip_allowlist_refresh import do_refresh
await publish_setting_changes({"login_manual_ips": val})
await do_refresh()
# S-07: Audit log
audit_repo = AuditLogRepositoryImpl(db)
@@ -1471,9 +1503,6 @@ async def remove_allowlist_ip(
ip_address=request.client.host if request.client else "",
))
from server.background.ip_allowlist_refresh import _do_refresh
import asyncio
asyncio.create_task(_do_refresh())
return {"success": True, "removed": ip, "remaining": remaining}
+3 -3
View File
@@ -864,7 +864,7 @@ async def file_diff(
# ── File Upload via SFTP ──
MAX_UPLOAD_FILE_SIZE = 104_857_600 # 100 MB
MAX_UPLOAD_FILE_SIZE = 524_288_000 # 500 MB
def _collect_multipart_upload_files(form) -> list:
@@ -1453,8 +1453,8 @@ async def download_file(
await ssh_pool.release(server.id)
raise HTTPException(status_code=400, detail="只能下载文件,不能下载目录")
# H-15: Download file size limit (100MB)
MAX_DOWNLOAD_SIZE = 100 * 1024 * 1024
# H-15: Download file size limit (500MB)
MAX_DOWNLOAD_SIZE = 524_288_000
if hasattr(stat, "size") and stat.size and stat.size > MAX_DOWNLOAD_SIZE:
await ssh_pool.release(server.id)
raise HTTPException(status_code=413, detail=f"文件大小 {stat.size} 字节超过下载限制 100MB")
+37 -8
View File
@@ -8,8 +8,11 @@ Data model:
login_allowlist_enabled — "true" / "false" master switch
"""
from __future__ import annotations
import asyncio
import logging
from dataclasses import dataclass
logger = logging.getLogger("nexus.ip_allowlist_refresh")
@@ -18,18 +21,32 @@ _last_refresh_at: str = ""
_last_subscription_count: int = 0
@dataclass(frozen=True)
class RefreshResult:
ok: bool
subscription_ips: tuple[str, ...] = ()
last_refresh: str = ""
subscription_count: int = 0
error: str | None = None
async def ip_allowlist_refresh_loop():
"""Every 2 hours, re-fetch subscription IPs and rebuild combined allowlist."""
logger.info("IP allowlist refresh loop started (interval=%sh)", REFRESH_INTERVAL // 3600)
# Initial refresh shortly after startup
await asyncio.sleep(5)
await _do_refresh()
await do_refresh()
while True:
await asyncio.sleep(REFRESH_INTERVAL)
await _do_refresh()
await do_refresh()
async def _do_refresh():
async def do_refresh() -> RefreshResult:
"""Fetch subscription, replace subscription IPs, rebuild combined list."""
return await _do_refresh()
async def _do_refresh() -> RefreshResult:
"""Fetch subscription, replace subscription IPs, rebuild combined list."""
global _last_refresh_at, _last_subscription_count
from server.config import settings
@@ -37,7 +54,7 @@ async def _do_refresh():
sub_url = (settings.LOGIN_SUBSCRIPTION_URL or "").strip()
if not sub_url:
return
return RefreshResult(ok=True)
logger.info("Refreshing subscription IPs from: %s", sub_url[:80])
try:
@@ -46,11 +63,11 @@ async def _do_refresh():
resp = await client.get(sub_url, follow_redirects=True)
if resp.status_code != 200:
logger.warning("Subscription returned HTTP %s", resp.status_code)
return
return RefreshResult(ok=False, error=f"订阅链接返回 HTTP {resp.status_code}")
new_sub_ips = parse_subscription_content(resp.text)
except Exception as e:
logger.error("Subscription fetch failed: %s", e)
return
return RefreshResult(ok=False, error=f"拉取订阅失败: {e}")
# Manual IPs (always preserved)
manual_raw = (settings.LOGIN_MANUAL_IPS or "").strip()
@@ -70,16 +87,21 @@ async def _do_refresh():
try:
from server.infrastructure.database.session import AsyncSessionLocal
from server.infrastructure.database.setting_repo import SettingRepositoryImpl
from server.infrastructure.settings_broadcast import publish_setting_changes
async with AsyncSessionLocal() as session:
repo = SettingRepositoryImpl(session)
# Subscription IPs — replaced entirely
await repo.set("login_subscription_ips", sub_val)
# Combined — for auth check
await repo.set("login_allowed_ips", combined_val)
settings.LOGIN_SUBSCRIPTION_IPS = sub_val
settings.LOGIN_ALLOWED_IPS = combined_val
await publish_setting_changes({
"login_subscription_ips": sub_val,
"login_allowed_ips": combined_val,
})
from server.utils.display_time import format_beijing
_last_refresh_at = format_beijing(with_label=False)
@@ -88,8 +110,15 @@ async def _do_refresh():
"Subscription IPs refreshed: %d → combined %d (+ %d manual)",
len(new_sub_ips), len(combined), len(manual_ips),
)
return RefreshResult(
ok=True,
subscription_ips=tuple(new_sub_ips),
last_refresh=_last_refresh_at,
subscription_count=len(new_sub_ips),
)
except Exception as e:
logger.error("Failed to save subscription IPs: %s", e)
return RefreshResult(ok=False, error=f"保存订阅 IP 失败: {e}")
def get_last_refresh_time() -> str:
@@ -38,6 +38,21 @@ async def publish_setting_change(db_key: str, value: str) -> bool:
return False
async def publish_setting_changes(changes: dict[str, str]) -> int:
"""Apply in-memory overrides and broadcast each key to all workers.
Returns the number of keys successfully published.
"""
from server.config import settings
published = 0
for db_key, value in changes.items():
if settings.apply_db_override(db_key, value):
if await publish_setting_change(db_key, value):
published += 1
return published
async def start_settings_subscriber() -> None:
"""Subscribe to nexus:settings — called during app startup (every worker)."""
global _subscriber_task, _redis_subscriber
+93
View File
@@ -0,0 +1,93 @@
"""Tests for login IP allowlist refresh and cross-worker settings broadcast."""
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from server.background.ip_allowlist_refresh import RefreshResult, do_refresh
from server.config import settings
from server.infrastructure.settings_broadcast import publish_setting_changes
@pytest.mark.asyncio
async def test_publish_setting_changes_broadcasts_each_key():
mock_redis = AsyncMock()
mock_redis.publish = AsyncMock(return_value=1)
with patch("server.infrastructure.redis.client.get_redis", return_value=mock_redis):
count = await publish_setting_changes({
"login_manual_ips": "1.2.3.4",
"login_subscription_url": "https://example.com/sub",
})
assert count == 2
assert mock_redis.publish.await_count == 2
assert settings.LOGIN_MANUAL_IPS == "1.2.3.4"
assert settings.LOGIN_SUBSCRIPTION_URL == "https://example.com/sub"
@pytest.mark.asyncio
async def test_do_refresh_publishes_subscription_ips():
settings.LOGIN_SUBSCRIPTION_URL = "https://example.com/sub"
settings.LOGIN_MANUAL_IPS = "10.0.0.1"
settings.LOGIN_SUBSCRIPTION_IPS = ""
settings.LOGIN_ALLOWED_IPS = ""
mock_repo = MagicMock()
mock_repo.set = AsyncMock()
mock_session = MagicMock()
mock_session.__aenter__ = AsyncMock(return_value=mock_session)
mock_session.__aexit__ = AsyncMock(return_value=None)
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.text = "ss://YWVzLTEyODpwbG4@104.194.72.62:443#node1"
mock_client = MagicMock()
mock_client.get = AsyncMock(return_value=mock_response)
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
with (
patch("server.infrastructure.database.session.AsyncSessionLocal", return_value=mock_session),
patch("server.infrastructure.database.setting_repo.SettingRepositoryImpl", return_value=mock_repo),
patch("server.infrastructure.settings_broadcast.publish_setting_changes", new_callable=AsyncMock) as mock_publish,
patch("httpx.AsyncClient", return_value=mock_client),
):
result = await do_refresh()
assert result.ok is True
assert result.subscription_ips == ("104.194.72.62",)
assert result.subscription_count == 1
assert settings.LOGIN_SUBSCRIPTION_IPS == "104.194.72.62"
mock_publish.assert_awaited_once()
published = mock_publish.await_args.args[0]
assert published["login_subscription_ips"] == "104.194.72.62"
assert "10.0.0.1" in published["login_allowed_ips"]
@pytest.mark.asyncio
async def test_do_refresh_without_subscription_url():
settings.LOGIN_SUBSCRIPTION_URL = ""
result = await do_refresh()
assert result.ok is True
assert result.subscription_ips == ()
@pytest.mark.asyncio
async def test_do_refresh_http_error():
settings.LOGIN_SUBSCRIPTION_URL = "https://example.com/sub"
mock_response = MagicMock()
mock_response.status_code = 503
mock_client = MagicMock()
mock_client.get = AsyncMock(return_value=mock_response)
mock_client.__aenter__ = AsyncMock(return_value=mock_client)
mock_client.__aexit__ = AsyncMock(return_value=None)
with patch("httpx.AsyncClient", return_value=mock_client):
result = await do_refresh()
assert result.ok is False
assert result.error is not None
assert "503" in result.error