feat(agent): 401 停心跳、配置重载后重启与升级双文件下发

提取 heartbeat_policy;install/upgrade 同步 heartbeat_policy.py;config reload 恢复心跳协程。

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Nexus Agent
2026-06-08 23:05:47 +08:00
parent 69068e2e39
commit 972456a246
12 changed files with 232 additions and 47 deletions
@@ -0,0 +1,26 @@
# 审计 — Agent 401 停心跳(A-05
**Changelog**: `docs/changelog/2026-06-08-agent-401-stop-heartbeat.md`
## 变更文件
`heartbeat_policy.py` · `agent.py` · `install.sh` · `servers.py` · `server_batch_common.py` · `server_batch_service.py` · `agent_version.py` · `test_agent_heartbeat_stop.py`
## Step 3
| 项 | 结论 |
|----|------|
| 401 / discarded 后 `return` 退出循环 | PASS |
| 5xx 仍退避重试,非静默吞错 | PASS |
| config reload 重启已 done 的心跳任务 | PASS |
| 子机 rollout 依赖 upgrade-agent(文档已写明) | PASS |
## Closure
Gate 7/7 PASS · 待部署 + 子机批量升级验证
## DoD
- [x] `heartbeat_should_stop` 单测
- [x] `get_central_agent_version` 仍读 2.0.0
- [ ] 生产抽样子机 journalctl 401 后无重复心跳(#7 rollout 后)
@@ -0,0 +1,40 @@
# 2026-06-08 — Agent 401 停心跳 + 配置重载后重启(A-05)
## 摘要
边端 Agent 心跳收到中心 **401****discarded** 后永久退出循环;`/config/reload` 写入新 key 后自动重启心跳任务。策略提取为 `heartbeat_policy.py` 并加单测。
## 动机
错误 API key 时旧 Agent 每 60s 无限重试;401 停循环已在 `72d82d7` 落地,但缺单测、缺 key 修复后自动恢复、子机文件未统一 rollout。
## 涉及文件
- `web/agent/heartbeat_policy.py`(新)
- `web/agent/agent.py``AGENT_VERSION` 常量、config reload 重启心跳
- `web/agent/install.sh` — 同步下载 `heartbeat_policy.py`
- `server/application/server_batch_common.py``agent_files_curl_cmds` 升级双文件
- `server/api/servers.py` · `server_batch_service.py` — 升级拉取双文件
- `server/utils/agent_version.py` — 读 `AGENT_VERSION` 常量
- `tests/test_agent_heartbeat_stop.py`
## 迁移 / 重启
- 中心:`/agent/agent.py` 随 API 镜像更新
- **各子机**:须批量 **升级 Agent** 或重装,否则仍跑旧 `agent.py`(见 backlog #7
## 行为
| 中心响应 | Agent |
|----------|--------|
| 401 Invalid API key | 停止心跳,打 error 日志 |
| 200 `status=discarded` | 停止心跳(错误 server_id 等) |
| 5xx / 网络错误 | 指数退避重试(不变) |
| `/config/reload` 新 key | 若心跳已停,重启协程 |
## 验证
```bash
pytest tests/test_agent_heartbeat_stop.py tests/test_agent_version.py -q
bash scripts/local_verify.sh
```
@@ -0,0 +1,30 @@
# 2026-06-08 — Agent 401 停心跳(A-05
## 背景
BL-06 后中心仅接受 per-server `agent_api_key`;错误 key 时心跳 401。边端 Agent 须停止 `while True` 重试,避免日志与无效流量。`72d82d7` 已加 401 `return`,本子任务补齐可测策略、配置重载后重启心跳、子机 rollout 说明。
## 方案
1. `web/agent/heartbeat_policy.py``heartbeat_should_stop(status_code, body)` 纯函数
2. `agent.py` — 401 / discarded 统一走策略;`AGENT_VERSION` 常量 SSOT
3. `/config/reload` 更新 key 后 `_start_heartbeat_task()` 重启已停止的心跳协程
4. 子机需 **批量升级 Agent** 拉取含本逻辑的 `agent.py`backlog #7
## 涉及文件
- `web/agent/heartbeat_policy.py`
- `web/agent/agent.py`
- `server/utils/agent_version.py` — 读 `AGENT_VERSION` 常量
- `tests/test_agent_heartbeat_stop.py`
## 验证
```bash
pytest tests/test_agent_heartbeat_stop.py tests/test_agent_version.py -q
# 子机:故意错 key → journalctl 一条 401 后无新心跳;修正 key + config reload 后恢复
```
## 回滚
恢复 agent.py 内联 401 分支;删除 heartbeat_policy.py
+12 -18
View File
@@ -36,17 +36,11 @@ pushbash scripts/git-push.sh(凭据 deploy/nexus-1panel.secrets.shgitign
- 用户报「server_ids 最多 50 项s after validation, not 101」→ 已修并部署 a401ea5
- 设置页连接池保存 422「Input should be a valid string」→ 已修并部署 ff143c2/632891e
本地未提交 · 下一会话优先
server_batch 僵尸回收 + stuck 扫描(代码已在工作区,未 commit/部署)
- server/background/server_batch_reconcile.py(新
- server/application/services/server_batch_service.py(改)
- server/infrastructure/database/server_batch_job_repo.py(改)
- server/infrastructure/redis/server_batch_store.py(改)
- server/main.py(改:启动 recover + 60s reconcile loop
- tests/test_server_batch_reconcile.py(新)
- docs/changelog/2026-06-08-server-batch-reconcile.md(新)
- docs/design/plans/2026-06-08-server-batch-reconcile.md(新)
→ 流程:补 audit → pytest → commit → push → deploy
2026-06-08 已部署 69068e2
- server_batch 僵尸回收 + stuck 60s 扫描
- sync_logs 30 天 purge(启动首轮 + 24h
- 批量安装 Agent:已装且版本 ≥ 主站则跳过
- 验证:`docs/reports/2026-06-08-full-align-deploy-verification.md`
【用户终验待确认】
- 全选 101 台批量操作(install-agent / detect-path)不再 422
@@ -55,9 +49,9 @@ pushbash scripts/git-push.sh(凭据 deploy/nexus-1panel.secrets.shgitign
- WebSSH 实连 pwd、Telegram 测试发送、推送预览(勿误推生产)
【优先待办 TOP 10】
1. server_batch 僵尸回收 — **本地已实现,待 commit + 部署**(见上
2. sync_logs 30 天自动清理 ☑(本地实现,待部署)
3. Agent 401 停循环A-05 / web/agent/agent.py
1. server_batch 僵尸回收 ☑(69068e2 已部署
2. sync_logs 30 天自动清理 ☑(69068e2 已部署)
3. Agent 401 停循环 ☑(代码+单测;子机 rollout 见 #7
4. Agent upgrade 与 install.sh venv 对齐(D-01
5. ~75 条后端 API 英文 detail 改中文(Pydantic 422 已部分中文化)
6. 更新 AI-HANDOFF + alignment-plan 勾选状态(文档 SSOT
@@ -73,7 +67,7 @@ pushbash scripts/git-push.sh(凭据 deploy/nexus-1panel.secrets.shgitign
【勿重复做 / 已否决】RBAC、移动端、会话录像、审计 CSV 导出、邮件报告、Redis Cluster、独立文件编辑器
【已风险接受】WebSocket JWT queryADR-011)、Agent global API_KEY legacy — risk-acceptance-single-operator.md
【工作区】a401ea5 已 push/deploy;本地勿提交 web/app/assets/* 构建残留、gate_log.jsonl、.cursor/settings.json
【工作区】69068e2 已 push/deploy;本地勿提交 web/app/assets/* 构建残留、gate_log.jsonl、.cursor/settings.json
【文档】Handoff 本文 · 功能指南 nexus-functional-development-guide.md · 验收 development-acceptance-standard.md
```
@@ -83,7 +77,7 @@ pushbash scripts/git-push.sh(凭据 deploy/nexus-1panel.secrets.shgitign
| 项 | 值 |
|----|-----|
| 分支 | `main` @ **`a401ea5`** |
| 分支 | `main` @ **`69068e2`** |
| 生产 URL | `https://api.synaglobal.vip` |
| 运行时 | Docker Compose @ `/opt/nexus`1panel-network |
| SSH | `azureuser@20.24.218.235`(本机 `Host nexus` 常不可用,用 IP + pem |
@@ -120,7 +114,7 @@ pushbash scripts/git-push.sh(凭据 deploy/nexus-1panel.secrets.shgitign
| # | 项 | 状态 |
|---|-----|------|
| 3 | sync_logs 30 天自动 purge | ☑ |
| 4 | Agent 401 停循环(A-05 | |
| 4 | Agent 401 停循环(A-05 | ☑~(待部署+子机升级) |
| 5 | upgrade-agent 与 install.sh 对齐(D-01 | ☐ |
| — | ~75 条后端 API 英文 `detail` 改中文 | ☐ |
| — | ~25 前端英/混排标签 | ☐ |
@@ -158,7 +152,7 @@ pushbash scripts/git-push.sh(凭据 deploy/nexus-1panel.secrets.shgitign
1. server_batch 僵尸任务回收(例 #20 ☑~ **(本地已实现,待部署)**
2. server_batch 启动 stuck 扫描 ☑~ **(同上)**
3. sync_logs 30 天自动 purge ☑
4. Agent 401 停循环(A-05
4. Agent 401 停循环(A-05 ☑~
5. upgrade-agent 与 install.sh 对齐(D-01
6. 生产复核 `push_schedules.target_path` 列(A-02 ~
7. 各子机同步 agent.pyA-05 依赖) ☐
+3 -2
View File
@@ -1724,7 +1724,8 @@ async def upgrade_agent(
detail="NEXUS_API_BASE_URL 未配置,无法构建下载地址",
)
agent_url = f"{base_url}/agent/agent.py"
from server.application.server_batch_common import agent_files_curl_cmds
install_dir = "/opt/nexus-agent"
venv_python = f"{install_dir}/.venv/bin/python"
ssh_user = (server.username or "root").strip()
@@ -1776,7 +1777,7 @@ async def upgrade_agent(
upgrade_cmd = _sudo_wrap(
f"{pip_cmd} "
f"&& {backup_cmd} "
f"&& curl -fsSL {shlex.quote(agent_url)} -o {install_dir}/agent.py "
f"&& {agent_files_curl_cmds(base_url, install_dir)} "
f"&& {kill_port}{systemctl_prefix}systemctl restart nexus-agent "
f"&& sleep 2 "
f"&& {systemctl_prefix}systemctl is-active nexus-agent "
+13 -2
View File
@@ -161,11 +161,21 @@ async def mark_agent_uninstalled(sid: int) -> None:
_REMOTE_AGENT_VERSION_CMD = (
"if [ -f /opt/nexus-agent/agent.py ]; then "
"grep -m1 '\"agent_version\"' /opt/nexus-agent/agent.py 2>/dev/null | "
"sed -n 's/.*\"agent_version\": *\"\\([^\"]*\\)\".*/\\1/p'; fi"
"grep -m1 '^AGENT_VERSION' /opt/nexus-agent/agent.py 2>/dev/null | "
"sed -n 's/.*AGENT_VERSION *= *\"\\([^\"]*\\)\".*/\\1/p'; fi"
)
def agent_files_curl_cmds(base_url: str, install_dir: str = "/opt/nexus-agent") -> str:
"""Download agent.py and heartbeat_policy.py from central /agent/."""
root = base_url.rstrip("/")
return (
f"curl -fsSL {shlex.quote(root + '/agent/agent.py')} -o {install_dir}/agent.py "
f"&& curl -fsSL {shlex.quote(root + '/agent/heartbeat_policy.py')} "
f"-o {install_dir}/heartbeat_policy.py"
)
async def _redis_agent_version(server_id: int) -> str:
from server.infrastructure.redis.client import get_redis
@@ -246,6 +256,7 @@ def result_item_dict(
__all__ = [
"agent_files_curl_cmds",
"agent_install_skip_result",
"batch_server_display_name",
"ensure_agent_api_key",
@@ -8,6 +8,7 @@ import shlex
from typing import Any, Optional
from server.application.server_batch_common import (
agent_files_curl_cmds,
agent_install_skip_result,
batch_server_display_name,
ensure_agent_api_key,
@@ -588,7 +589,6 @@ async def _run_upgrade_agent(live: dict[str, Any]) -> None:
if not server:
return result_item_dict(server_id=sid, server_name=label, success=False, error="服务器不存在")
try:
agent_url = f"{base_url}/agent/agent.py"
install_dir = "/opt/nexus-agent"
venv_python = f"{install_dir}/.venv/bin/python"
ssh_user = (server.username or "root").strip()
@@ -599,7 +599,7 @@ async def _run_upgrade_agent(live: dict[str, Any]) -> None:
f"&& {venv_python} -c 'import sys; v=sys.version_info; sys.exit(0 if v.major==3 and v.minor>=10 else 1)' "
f"&& {venv_python} -m pip install -q fastapi==0.115.6 uvicorn==0.34.0 httpx==0.28.1 psutil python-multipart==0.0.19 "
f"&& cp {install_dir}/agent.py {install_dir}/agent.py.bak "
f"&& curl -fsSL {shlex.quote(agent_url)} -o {install_dir}/agent.py "
f"&& {agent_files_curl_cmds(base_url, install_dir)} "
f"&& {batch_kill_port}{batch_systemctl_prefix}systemctl restart nexus-agent "
f"&& sleep 2 "
f"&& {batch_systemctl_prefix}systemctl is-active nexus-agent "
+4 -4
View File
@@ -7,16 +7,16 @@ from functools import lru_cache
from pathlib import Path
_AGENT_PY = Path(__file__).resolve().parents[2] / "web" / "agent" / "agent.py"
_VERSION_IN_AGENT_RE = re.compile(r'"agent_version":\s*"([^"]+)"')
_AGENT_VERSION_CONST_RE = re.compile(r'^AGENT_VERSION\s*=\s*"([^"]+)"', re.MULTILINE)
@lru_cache(maxsize=1)
def get_central_agent_version() -> str:
"""Read agent_version string shipped at /agent/agent.py (heartbeat payload)."""
"""Read AGENT_VERSION constant from web/agent/agent.py (heartbeat SSOT)."""
text = _AGENT_PY.read_text(encoding="utf-8")
match = _VERSION_IN_AGENT_RE.search(text)
match = _AGENT_VERSION_CONST_RE.search(text)
if not match:
raise RuntimeError(f"agent_version not found in {_AGENT_PY}")
raise RuntimeError(f"AGENT_VERSION not found in {_AGENT_PY}")
return match.group(1)
+36
View File
@@ -0,0 +1,36 @@
"""Agent heartbeat 401 / discarded stop policy (A-05)."""
import sys
from pathlib import Path
import pytest
_AGENT_DIR = Path(__file__).resolve().parents[1] / "web" / "agent"
if str(_AGENT_DIR) not in sys.path:
sys.path.insert(0, str(_AGENT_DIR))
from heartbeat_policy import heartbeat_should_stop # noqa: E402
from server.utils.agent_version import get_central_agent_version # noqa: E402
def test_stop_on_401():
assert heartbeat_should_stop(401) is True
assert heartbeat_should_stop(401, {}) is True
def test_stop_on_discarded():
assert heartbeat_should_stop(200, {"status": "discarded", "reason": "missing server_id"}) is True
def test_continue_on_success():
assert heartbeat_should_stop(200, {"status": "ok"}) is False
def test_continue_on_transient_errors():
assert heartbeat_should_stop(500) is False
assert heartbeat_should_stop(502, None) is False
def test_central_agent_version_matches_constant():
assert get_central_agent_version() == "2.0.0"
+42 -17
View File
@@ -16,6 +16,10 @@ from pathlib import Path
from typing import Optional
import httpx
from heartbeat_policy import heartbeat_should_stop
AGENT_VERSION = "2.0.0"
import psutil
from fastapi import FastAPI, HTTPException, Header, Request, Depends
from fastapi.responses import JSONResponse
@@ -65,15 +69,28 @@ logging.basicConfig(
)
logger = logging.getLogger("nexus-agent")
_heartbeat_task: Optional[asyncio.Task] = None
# --- FastAPI App ---
from contextlib import asynccontextmanager
def _start_heartbeat_task() -> asyncio.Task:
"""Start heartbeat loop (idempotent if previous task still running)."""
global _heartbeat_task
if _heartbeat_task is not None and not _heartbeat_task.done():
return _heartbeat_task
_heartbeat_task = asyncio.create_task(send_heartbeat())
return _heartbeat_task
@asynccontextmanager
async def agent_lifespan(app):
"""Application lifecycle: start heartbeat task on startup, clean up on shutdown."""
global _heartbeat_task
logger.info(f"Nexus Agent starting (server_id={SERVER_ID})")
task = asyncio.create_task(send_heartbeat())
_start_heartbeat_task()
host = config.get("server", {}).get("host", "127.0.0.1")
port = config.get("server", {}).get("port", 8601)
@@ -81,10 +98,12 @@ async def agent_lifespan(app):
yield
task.cancel()
if _heartbeat_task is not None:
_heartbeat_task.cancel()
_heartbeat_task = None
logger.info("Nexus Agent stopped")
app = FastAPI(title="Nexus Agent", version="2.0.0", lifespan=agent_lifespan)
app = FastAPI(title="Nexus Agent", version=AGENT_VERSION, lifespan=agent_lifespan)
def _ip_allowed(client_ip: str) -> bool:
@@ -305,6 +324,10 @@ async def reload_config(data: dict, _auth=Depends(verify_api_key)):
CENTRAL_URL = config.get("central", {}).get("url", CENTRAL_URL)
CENTRAL_API_KEY = config.get("central", {}).get("api_key", CENTRAL_API_KEY)
if CENTRAL_URL and SERVER_ID and CENTRAL_API_KEY:
_start_heartbeat_task()
logger.info("Heartbeat task (re)started after config reload")
return {"status": "ok", "message": "Config updated", "updated_at": now}
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
@@ -377,7 +400,7 @@ async def send_heartbeat():
payload = {
"server_id": SERVER_ID,
"is_online": True,
"agent_version": "2.0.0",
"agent_version": AGENT_VERSION,
"system_info": {
"cpu_usage": cpu,
"mem_usage": mem,
@@ -396,26 +419,28 @@ async def send_heartbeat():
json=payload,
headers={"X-API-Key": CENTRAL_API_KEY},
)
if resp.status_code == 200:
data = resp.json()
if data.get("status") == "discarded":
body = resp.json() if resp.status_code == 200 else None
if heartbeat_should_stop(resp.status_code, body):
if resp.status_code == 401:
logger.error(
"Heartbeat rejected by central (401): invalid API key or "
"server_id=%s not configured with per-server agent_api_key. "
"Stopping heartbeat — reconfigure agent key via admin UI.",
SERVER_ID,
)
else:
reason = (body or {}).get("reason", "unknown")
logger.error(
"Heartbeat discarded by central: %s. "
"This agent may be misconfigured (wrong server_id=%s). "
"Stopping heartbeat to prevent noise.",
data.get("reason", "unknown"), SERVER_ID,
reason,
SERVER_ID,
)
return # Exit heartbeat loop — agent needs reconfiguration
return
if resp.status_code == 200:
_consecutive_failures = 0
logger.debug("Heartbeat sent")
elif resp.status_code == 401:
logger.error(
"Heartbeat rejected by central (401): invalid API key or "
"server_id=%s not configured with per-server agent_api_key. "
"Stopping heartbeat — reconfigure agent key via admin UI.",
SERVER_ID,
)
return
else:
_consecutive_failures += 1
logger.warning(f"Heartbeat failed: {resp.status_code}")
+18
View File
@@ -0,0 +1,18 @@
"""Heartbeat loop stop conditions — shared by agent.py and unit tests."""
from __future__ import annotations
from typing import Any, Mapping, Optional
def heartbeat_should_stop(
status_code: int,
body: Optional[Mapping[str, Any]] = None,
) -> bool:
"""True when central rejected heartbeat permanently (do not retry)."""
if status_code == 401:
return True
if status_code == 200 and isinstance(body, Mapping):
if body.get("status") == "discarded":
return True
return False
+6 -2
View File
@@ -253,13 +253,17 @@ PIP_DEPS="fastapi==${FASTAPI_VER} uvicorn==${UVICORN_VER} httpx==${HTTPX_VER} py
[ -n "$PSUTIL_VER" ] && PIP_DEPS="${PIP_DEPS} psutil==${PSUTIL_VER}" || PIP_DEPS="${PIP_DEPS} psutil"
pip install -q $PIP_DEPS
# 4. Download agent.py
echo "[4/7] Download agent.py..."
# 4. Download agent.py + heartbeat_policy.py
echo "[4/7] Download agent files..."
[ -z "$WEB_URL" ] && WEB_URL="${CENTRAL_URL}"
curl -fsSL "${WEB_URL}/agent/agent.py" -o "$INSTALL_DIR/agent.py" || {
echo "ERROR: Cannot download from $WEB_URL/agent/agent.py"
exit 1
}
curl -fsSL "${WEB_URL}/agent/heartbeat_policy.py" -o "$INSTALL_DIR/heartbeat_policy.py" || {
echo "ERROR: Cannot download from $WEB_URL/agent/heartbeat_policy.py"
exit 1
}
# 5. Config
echo "[5/7] Write config..."