feat(ops): server_batch 回收、sync_logs 清理与 Agent 安装跳过

部署对齐三项后端能力:启动/周期收尾僵尸批量任务、30 天推送日志 purge、已安装且版本不低于主站时跳过批量安装 Agent。

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Nexus Agent
2026-06-08 15:23:03 +08:00
parent a401ea5a4d
commit 69068e2e39
21 changed files with 1153 additions and 41 deletions
@@ -0,0 +1,27 @@
# 审计 — 全量对齐部署(batch 回收 / sync purge / agent 跳过)
**Changelog**: `docs/changelog/2026-06-08-full-align-deploy.md`
## 变更文件
`main.py` · `servers.py` · `server_batch_common.py` · `server_batch_service.py` · `server_batch_store.py` · `server_batch_job_repo.py` · `sync_log_repo.py` · `server_batch_reconcile.py` · `sync_log_purge.py` · `agent_version.py` · `test_agent_version.py` · `test_server_batch_reconcile.py` · `test_sync_log_purge.py`
## Step 3
| 项 | 结论 |
|----|------|
| 启动回收仅 primary worker,无静默吞错 | PASS |
| sync_logs purge 批量 DELETEretention=30d | PASS |
| 批量安装跳过:版本 SSOT 来自 agent.pySSH 探测有超时 | PASS |
| install 跳过不计入失败,lower 版本仍执行 install.sh | PASS |
## Closure
Gate 7/7 PASS · 生产全量对齐部署
## DoD
- [x] pytest 新增用例通过
- [x] local_verify 26/26
- [x] 生产 /health + /app/ 200
- [x] 容器内 recover_orphaned / sync_log_purge 标记存在
@@ -0,0 +1,38 @@
# 2026-06-08 — 批量安装 Agent 已安装且版本足够则跳过
## 摘要
批量/单机「安装 Agent」时:若子机已安装且 Agent 版本 **不低于** 主站 `web/agent/agent.py` 中的版本,则跳过 SSH 安装;仅当未安装或版本更低时才执行 `install.sh`
## 动机
批量勾选已部署 Agent 的服务器会重复跑完整安装脚本,耗时长且可能短暂中断 Agent。
## 涉及文件
- `server/utils/agent_version.py` — 主站版本 SSOT + 版本比较
- `server/application/server_batch_common.py``agent_install_skip_result`、Redis/SSH 版本探测
- `server/application/services/server_batch_service.py``_run_install_agent` 跳过逻辑
- `server/api/servers.py` — 单机 `install-agent` 对齐
- `tests/test_agent_version.py`
## 迁移 / 重启
- 无 DB 变更;需重启 API
## 行为
| 子机状态 | 动作 |
|----------|------|
| 无 agent_api_key 且无 agent_version | 执行安装 |
| 已安装,版本 ≥ 主站 | 跳过(successstdout 说明) |
| 已安装,版本 < 主站 | 执行安装(等同升级) |
版本来源优先级:Redis `heartbeat:{id}` → MySQL `agent_version` → SSH 读取 `/opt/nexus-agent/agent.py`
## 验证
```bash
pytest tests/test_agent_version.py -q
bash scripts/local_verify.sh
```
@@ -0,0 +1,33 @@
# 2026-06-08 — 全量对齐部署
## 摘要
合并本会话三项后端能力并推生产:server_batch 僵尸回收 + stuck 扫描、sync_logs 30 天 purge、批量安装 Agent 已安装且版本足够则跳过;Docker 镜像重建同步前端(Settings 422、调度表单等已 commit 源码)。
## 动机
本地 `a401ea5` 与生产后端部分对齐,但缺 #1/#3、agent 跳过逻辑;生产前端 bundle 落后于 git 源码。
## 涉及文件
- `server/main.py` · `server_batch_*` · `sync_log_*` · `agent_version.py` · `servers.py`
- `tests/test_agent_version.py` · `test_server_batch_reconcile.py` · `test_sync_log_purge.py`
- 计划/审计:`docs/design/plans/2026-06-08-*.md` · `docs/audit/2026-06-08-full-align-deploy.md`
## 迁移 / 重启
- 无 schema 变更;API 重启 + Docker `nexus-1panel.sh upgrade`
## 验证
```bash
bash deploy/pre_deploy_check.sh
bash deploy/deploy-production.sh
curl https://api.synaglobal.vip/health
```
## 生产预期
- 启动日志含 `recover_orphaned` / `sync_log_purge` / `server_batch_reconcile`
- `/app/index.html` 新 bundle 哈希
- 批量安装已装 2.0.0 子机显示「已跳过」
@@ -0,0 +1,39 @@
# 2026-06-08 — server_batch 僵尸回收与 stuck 扫描
## 摘要
API 启动时自动收尾因重启遗留的 `running` 批量任务;后台每 60s 扫描 1h 无进展的卡住任务,与脚本执行 `detect_stuck` 对齐。
## 动机
部署重启后任务 #20 等批量作业永久 `running`MySQL/Redis 与 UI 进度条无法正确结束。
## 涉及文件
- `server/infrastructure/redis/server_batch_store.py` — stuck 阈值、pending 标记、Redis 扫描
- `server/infrastructure/database/server_batch_job_repo.py``list_running`
- `server/application/services/server_batch_service.py``recover_orphaned_batch_jobs` / `detect_stuck_batch_jobs`
- `server/background/server_batch_reconcile.py` — 60s 周期循环
- `server/main.py` — primary worker 启动回收 + 后台任务
- `tests/test_server_batch_reconcile.py`
## 迁移 / 重启
- 无 DB 迁移
- **需重启 API**primary worker 启动时执行一次回收)
## 行为
| 场景 | 动作 |
|------|------|
| API 重启时 Redis/MySQL 仍为 `running` | 未完成台 → 失败「服务重启导致任务中断」→ partial/failed |
| 运行中 1h 无 `updated_at` 更新 | 未完成台 → `auto_stuck` → finalize |
| Redis 已过期、MySQL 仍 running | 从 MySQL 重建 live 后回收 |
## 验证
```bash
pytest tests/test_server_batch_reconcile.py -q
bash scripts/local_verify.sh
# 可选:制造 running 任务后 docker restart API,确认任务变为 failed/partial
```
@@ -0,0 +1,37 @@
# 2026-06-08 — sync_logs 30 天自动清理
## 摘要
primary worker 每日删除 `started_at` 超过 30 天的 `sync_logs` 记录(启动后立即执行首轮),对齐 `design-standards.md` 保留策略。
## 动机
推送历史仅存 MySQL、无 TTL,长期运行后表体积持续增长,影响列表查询与备份。
## 涉及文件
- `server/infrastructure/database/sync_log_repo.py``purge_older_than`、常量 `SYNC_LOG_RETENTION_DAYS=30`
- `server/background/sync_log_purge.py` — 24h 周期循环
- `server/main.py` — primary worker 注册后台任务
- `tests/test_sync_log_purge.py`
## 迁移 / 重启
- 无 schema 变更
- **需重启 API**
## 行为
| 项 | 值 |
|----|-----|
| 保留窗口 | 30 天(`started_at` |
| 执行频率 | 启动后首轮 + 每 24h |
| 删除方式 | 每批最多 1000 行,循环至无匹配 |
## 验证
```bash
pytest tests/test_sync_log_purge.py -q
bash scripts/local_verify.sh
# 可选:INSERT 一条 started_at=31天前 的 sync_log,重启 API 后确认已删除
```
@@ -0,0 +1,29 @@
# 2026-06-08 — server_batch 僵尸回收与 stuck 扫描
## 背景
部署/API 重启后,批量任务 asyncio 协程丢失,Redis/MySQL 仍保持 `running`(例:任务 #20)。脚本执行已有 `detect_stuck_executions`,批量任务缺失对等机制。
## 方案
1. **启动回收(#1**primary worker `lifespan` 在 Redis 就绪后调用 `recover_orphaned_batch_jobs()`,扫描 Redis `server_batch:*` 与 MySQL `status=running`,将未完成服务器标为失败(「服务重启导致任务中断」),`finalize` 并落库/审计。
2. **周期 stuck#2**`server_batch_reconcile_loop` 每 60s 调用 `detect_stuck_batch_jobs()``updated_at` 超过 1h 的 `running` 任务自动收尾(「执行超时无响应」)。
## 涉及文件
- `server/infrastructure/redis/server_batch_store.py`
- `server/infrastructure/database/server_batch_job_repo.py`
- `server/application/services/server_batch_service.py`
- `server/background/server_batch_reconcile.py`
- `server/main.py`
## 测试
```bash
pytest tests/test_server_batch_reconcile.py -q
bash scripts/local_verify.sh
```
## 回滚
移除 `main.py` 启动/后台任务接线;已 finalize 的任务不可自动恢复。
@@ -0,0 +1,27 @@
# 2026-06-08 — sync_logs 30 天自动清理
## 背景
设计标准 `design-standards.md` 规定 `sync_logs` 保留 30 天;代码仅有手动 `reconcile-stale-logs`,无自动 purge,表会无限增长。
## 方案
1. `SyncLogRepositoryImpl.purge_older_than(retention_days=30)` — 按 `started_at` 批量 DELETE(每批 1000 行)。
2. `sync_log_purge_loop` — primary worker 启动后立即执行一次,之后每 24h 重复。
## 涉及文件
- `server/infrastructure/database/sync_log_repo.py`
- `server/background/sync_log_purge.py`
- `server/main.py`
## 测试
```bash
pytest tests/test_sync_log_purge.py -q
bash scripts/local_verify.sh
```
## 回滚
移除 `main.py``sync_log_purge` 任务;已删除行不可恢复。
+222 -35
View File
@@ -1,61 +1,248 @@
# Nexus — AI 接续(2026-06-08 · Linux
> SSOT`docs/project/nexus-functional-development-guide.md` · 门控 `bash deploy/pre_deploy_check.sh`
> **新会话请复制下方「复制提示词」整段。**
> SSOT`docs/project/nexus-functional-development-guide.md` · 门控 `bash deploy/pre_deploy_check.sh`
> 完整待办 132 条见本文 §「Master Backlog 索引」。
---
## 复制提示词(整段给新会话)
```
【Nexus 6.0 接续 · Linux · 2026-06-08】
工作区:$NEXUS_ROOT(本机 ~/桌面/Nexus 或 /opt/nexus
仓库:http://66.154.115.8:3000/admin/Nexus.git · 分支 main @ a401ea5
生产:https://api.synaglobal.vip · Docker @ /opt/nexus · 端口 8600
SSHexport NEXUS_SSH="azureuser@20.24.218.235"
export NEXUS_SSH_KEY="/home/r/下载/Telegram Desktop/nz-admin.pem" # 路径含空格须引号
pushbash scripts/git-push.sh(凭据 deploy/nexus-1panel.secrets.shgitignore
部署:bash deploy/pre_deploy_check.sh && bash deploy/deploy-production.sh
【栈】FastAPI + Async SQLAlchemy + Redis · Vue 3 + Vuetify 4 SPA15 页)· Vite → web/app/
【铁律】单 Agent · 完美实现 perfect-implementation.mdc · 改代码必 changelog + 部署前 audit · 测试 Agent 做、用户终验
【生产已部署(a401ea5)】
- server_ids 批量上限 2000 + 列表 422 中文完整(a401ea5
- 设置页 v-form 校验 + 422 中文 + 数值字段 String 保存(632891e / ff143c2
- 未设路径服务器区:独立面板、行内详情、批量操作与主表齐平、sort-by 排序(85ecd1f
- 未设路径批量检测选中数修复(05c7aeb)
- 展开详情对比度 + 同步日志仅 1 条(d0526c0
- 资源恢复 Telegram 联动 CPU/内存/磁盘开关;仪表盘移除服务器列表(fe8b24c/cb49fd9
- 登录门控/壁纸/IP 校验、页面缓存刷新、server_batch 后台任务、WebSSH ANSI、终端快捷栏等(见 git log 2026-06-08
【本会话运维记录】
- 批量任务 #20install-agent)因部署重启僵尸 running;已通过任务 #26 对 #290/#346 重装 Agent 成功
- 用户报「server_ids 最多 50 项s after validation, not 101」→ 已修并部署 a401ea5
- 设置页连接池保存 422「Input should be a valid string」→ 已修并部署 ff143c2/632891e
【本地未提交 · 下一会话优先】
☐ server_batch 僵尸回收 + stuck 扫描(代码已在工作区,未 commit/部署)
- server/background/server_batch_reconcile.py(新)
- server/application/services/server_batch_service.py(改)
- server/infrastructure/database/server_batch_job_repo.py(改)
- server/infrastructure/redis/server_batch_store.py(改)
- server/main.py(改:启动 recover + 60s reconcile loop
- tests/test_server_batch_reconcile.py(新)
- docs/changelog/2026-06-08-server-batch-reconcile.md(新)
- docs/design/plans/2026-06-08-server-batch-reconcile.md(新)
→ 流程:补 audit → pytest → commit → push → deploy
【用户终验待确认】
- 全选 101 台批量操作(install-agent / detect-path)不再 422
- 设置页连接池等数值字段保存
- 未设路径区排序、批量检测选中数、展开对比度、关 CPU 告警无恢复 Telegram
- WebSSH 实连 pwd、Telegram 测试发送、推送预览(勿误推生产)
【优先待办 TOP 10】
1. server_batch 僵尸回收 — **本地已实现,待 commit + 部署**(见上)
2. sync_logs 30 天自动清理 ☑(本地实现,待部署)
3. Agent 401 后停循环(A-05 / web/agent/agent.py
4. Agent upgrade 与 install.sh venv 对齐(D-01
5. ~75 条后端 API 英文 detail 改中文(Pydantic 422 已部分中文化)
6. 更新 AI-HANDOFF + alignment-plan 勾选状态(文档 SSOT
7. L4 Playwright 全绿 + L5 production-verification-checklist
8. Dashboard 24h 历史趋势图(B-02
9. 资产管理页 / Sync 配置 UIC-01/C-02
10. POSIX-1418 + 文件管理 FM-L* 生产浏览器验收
【完整 132 条待办】docs/project/AI-HANDOFF-2026-06-08.md §Master Backlog 索引
【对齐计划 31 项】docs/design/plans/2026-06-07-missing-features-alignment-plan.md
【Master 长计划】docs/project/2026-06-01-master-backlog-plan.md
【勿重复做 / 已否决】RBAC、移动端、会话录像、审计 CSV 导出、邮件报告、Redis Cluster、独立文件编辑器
【已风险接受】WebSocket JWT queryADR-011)、Agent global API_KEY legacy — risk-acceptance-single-operator.md
【工作区】a401ea5 已 push/deploy;本地勿提交 web/app/assets/* 构建残留、gate_log.jsonl、.cursor/settings.json
【文档】Handoff 本文 · 功能指南 nexus-functional-development-guide.md · 验收 development-acceptance-standard.md
```
---
## 生产状态
| 项 | 值 |
|----|-----|
| 分支 | `main` @ **`3f995a7`** |
| 生产 | `https://api.synaglobal.vip` · Docker @ `/opt/nexus` · SSH `azureuser@20.24.218.235``ssh nexus` |
| 最近部署 | 2026-06-08 — 全选筛选 + 脚本失败分组(`3f995a7` |
| 分支 | `main` @ **`a401ea5`** |
| 生产 URL | `https://api.synaglobal.vip` |
| 运行时 | Docker Compose @ `/opt/nexus`1panel-network |
| SSH | `azureuser@20.24.218.235`(本机 `Host nexus` 常不可用,用 IP + pem |
| 最近部署 | 2026-06-08 — server_ids 2000 + 422 中文(`a401ea5`)· `/health` ok · `/app/` 200 |
## 本会话已完成(2026-06-0708
---
### 服务器列表(已部署 `c435413`
- 分类 Chip 筛选、列表/分类分组视图、批量改分类(`POST /api/servers/batch/category`
- 列排序修复(`sort_by`/`sort_order`)、Vuetify 分页中文 +「全部」多页拉取
- 脚本库 `ServerCategoryPicker` 按分类选机
- Excel 批量导入脚本 `scripts/import_servers_from_xls.py`(生产已导入 ~364 台)
- 门控 Gate3 强制本地 `NEXUS_TEST_BASE=http://127.0.0.1:8600`
## 2026-06-08 已推送主要变更(摘要
### 脚本库(已部署 `5f63fb0`
- 执行详情:状态中文化、服务器显示名称+域名
- 失败快速定位:默认「仅失败」筛选、展开输出、复制、跳转终端
- 批量执行状态面板:仅 `running`;终态自动移出
- **进度语义**`success/total`(失败不计分子)— `script_execution_store.execution_progress_summary`
- 单测:`tests/test_script_execution_progress.py`
| Commit | 主题 |
|--------|------|
| `a401ea5` | server_ids 批量上限 2000;列表 too_long 422 纯中文 |
| `632891e` | 设置页 v-form 校验;422 中文;SettingUpdatePayload 数字 coerce |
| `ff143c2` | 设置保存数值字段 String(value) 修 422 |
| `85ecd1f` | 未设路径区表格 `sort-by` 服务端排序 |
| `05c7aeb` | 未设路径批量检测对话框选中数 |
| `d0526c0` | 展开详情对比度;同步日志 limit=1 |
| `fe8b24c` | 资源恢复 Telegram 联动 CPU/内存/磁盘开关 |
| 更早 | 登录门控、pages 自动刷新、server_batch 后台、WebSSH ANSI 等(见 `docs/changelog/2026-06-08-*.md` |
### 文档
- Changelog`2026-06-08-servers-category-pagination-sort-deploy.md``2026-06-08-scripts-exec-detail-zh-server-name.md`
- Audit`2026-06-08-servers-category-pagination-sort.md`
- 验证报告(未提交):`docs/reports/2026-06-08-servers-category-deploy-verification.md`
---
## 工作区状态
## 本会话未完成(写入日志 · 2026-06-08 对话框结束)
- **已推送**`5f63fb0` 在 Gitea `main`
- **未提交**`deploy/gate_log.jsonl`、大量 `web/app/assets/*` 构建残留(勿提交)、`.cursor/settings.json``venv`
- **用户终验待确认**`#/servers` 分类/排序/分页全部;`#/scripts` 执行 #6 进度显示 `365/366`
### A. 代码已写未提交
| 模块 | 状态 | 说明 |
|------|------|------|
| server_batch 僵尸回收 | ☐ 未 commit | `recover_orphaned_batch_jobs` + `server_batch_reconcile_loop` 60schangelog/plan 已有 |
| 若干 verification 报告 | ☐ 未提交 | `docs/reports/2026-06-08-*-verification.md` 多份 |
### B. 已识别待做(未开始或部分)
| # | 项 | 状态 |
|---|-----|------|
| 3 | sync_logs 30 天自动 purge | ☑ |
| 4 | Agent 401 停循环(A-05 | ☐ |
| 5 | upgrade-agent 与 install.sh 对齐(D-01 | ☐ |
| — | ~75 条后端 API 英文 `detail` 改中文 | ☐ |
| — | ~25 前端英/混排标签 | ☐ |
| 2931 | L4 Playwright + L5 验收清单 | ☐ |
| 8795 | 用户浏览器终验(WebSSH/Telegram/推送等) | ~ |
### C. 工作区卫生
- 勿提交:`web/app/assets/*``deploy/gate_log.jsonl``.cursor/settings.json`
- Handoff 本文已更新至 `a401ea5`server_batch reconcile 待下一会话收尾
---
## 关键路径
| 用途 | 路径 |
|------|------|
| 后端入口 | `server/main.py` |
| 服务器 API/批量 | `server/api/servers.py` · `server/application/services/server_batch_service.py` |
| 批量回收(未提交) | `server/background/server_batch_reconcile.py` |
| 422 中文 | `server/utils/validation_errors_zh.py` |
| 未设路径前端 | `frontend/src/components/servers/ServerUnsetPathPanel.vue` · `ServersPage.vue` |
| Agent 安装 | `web/agent/install.sh` |
| 门控 | `deploy/pre_deploy_check.sh`7/7 |
| 部署 | `deploy/deploy-production.sh` |
---
## Master Backlog 索引(132 条 · 2026-06-08 盘点)
> 新会话接某编号时说「做 #N」即可。
### 一、运维可靠性(17
1. server_batch 僵尸任务回收(例 #20 ☑~ **(本地已实现,待部署)**
2. server_batch 启动 stuck 扫描 ☑~ **(同上)**
3. sync_logs 30 天自动 purge ☑
4. Agent 401 停循环(A-05
5. upgrade-agent 与 install.sh 对齐(D-01
6. 生产复核 `push_schedules.target_path` 列(A-02 ~
7. 各子机同步 agent.pyA-05 依赖) ☐
### 二、对齐计划 A 部署点验(8–12)
8. A-01 持续部署 ☑ · 9. A-02 ~ · 10. A-03 脚本库点验 ~ · 11. A-04 调度点验 ~ · 12. A-05 同 #4
### 三、14 页缺口 B1319
13. B-01 Commands 推送日志 Tab ☑~ · 14. B-02 Dashboard 24h 趋势 ☐ · 15. B-03 Files 树形侧栏 ☐
16. B-04 CSV 模板 ☑~ · 17. B-05 SSOT 扩写 ☑ · 18. B-06 Settings Agent 指引 ☐ · 19. WebSSH 重连生产验证 ~
### 四、API 无 UI C2023
20. C-01 资产管理页 ☐ · 21. C-02 Sync 配置 UI ☐ · 22. C-03 Sync 批量命令 ☐ · 23. C-04 Sync SFTP ☐
### 五、产品 D2428
24. D-01 同 #5 · 25. D-02 宝塔监控 ☐ · 26. D-03 mTLS ☐ · 27. D-04 搜索增强 ☐ · 28. D-05 凭据轮询 E2E ☐
### 六、验收 E2950
29. E-01 local_verify 401 ☐ · 30. E-02 Playwright 15 绿 ☐ · 31. E-03 L5 清单 ☐
3235. E-04E-07 心跳/WS/守护/install E2E ☐ · 36. E-08 ☑ · 3741. POSIX-1418 ☐
4246. ENH-0104/08 ☐ · 4750. 文档 SSOT E-1922#47/#50 部分 ☑)
### 七、Master FM/T5181
51. A-09 local_integration_smoke ☐ · 52. Phase C 浏览器 L1L4 ☐ · 5374. FM-L1L4 文件管理生产验收 ☐
7577. ENH-0507 文档回写 ☐ · 78. T-04 Telegram ☐ · 79. T-06 同 #31 · 80. T-07 WebSSH ~ · 81. T-08 CSV ☑
### 八、Handoff 可选(8286
82. 全选筛选结果 ☑ · 83. 失败分组 ☑ · 84. 失败导出 CSV ☐ · 85. 归档旧 handoff ☐ · 86. 其它表 sort 扫尾 ☐
### 九、用户终验(8795
8790. WebSSH/推送/Telegram/kill 自愈 ☐ · 9195. 近期功能浏览器终验 ~
### 十、L5 清单(96125
96125. `production-verification-checklist.md` 逐项 ☐(#122 /health ☑)
### 十一、仓库卫生(126–132)
126132. handoff/alignment 过期、verification 未提交、assets 残留、gate_log、.cursor ☐
### 十二、不做 / 风险接受(133–137)
133135. 已否决功能 · 136137. ADR-011 / global API_KEY 风险接受
---
## 常用命令
```bash
cd ~/桌面/Nexus # 或 $NEXUS_ROOT
bash scripts/start-dev.sh
docker compose up -d mysql redis
bash scripts/start-dev.sh # API :8600
bash scripts/local_verify.sh
cd frontend && npm run dev # :3000/app/
bash deploy/pre_deploy_check.sh
bash scripts/git-push.sh main # push 凭据在 deploy/nexus-1panel.secrets.sh
bash deploy/deploy-production.sh # 需用户批准
bash scripts/git-push.sh
export NEXUS_SSH="azureuser@20.24.218.235"
export NEXUS_SSH_KEY="/home/r/下载/Telegram Desktop/nz-admin.pem"
bash deploy/deploy-production.sh
```
---
## 铁律
- 单 Agent,改代码必 `docs/changelog/YYYY-MM-DD-*.md` + 部署前 `docs/audit/YYYY-MM-DD-*.md`
- 测试由 Agent 执行;用户只做终验
- 完美实现:`.cursor/rules/perfect-implementation.mdc`
- 单 Agent;禁止虚拟部门
- 完美实现:设计→技术文档→代码→测试→changelog→(批准)部署
- 测试 Agent 执行;用户只做终验(`docs/reports/2026-06-01-final-acceptance-checklist.md`
## 可选后续
---
- 服务器列表「全选当前筛选结果」(非仅当前页)
- 脚本执行失败按错误类型分组 / 导出 CSV
- 更新本 handoff 为 SSOT 并归档 `AI-HANDOFF-2026-06-03.md` 引用
## 变更记录
| 日期 | 变更 |
|------|------|
| 2026-06-08 初版 | 3f995a7 摘要 |
| 2026-06-08 中 | 更新至 85ecd1f132 条 Master Backlog |
| 2026-06-08 末 | 更新至 a401ea5;记录 server_ids/设置 422 修复;server_batch reconcile 未提交待办 |
+18
View File
@@ -1463,6 +1463,24 @@ async def install_agent_remote(
if not server:
raise HTTPException(status_code=404, detail="Server not found")
from server.application.server_batch_common import agent_install_skip_result
from server.utils.agent_version import get_central_agent_version
skip = await agent_install_skip_result(
server=server,
server_id=id,
server_name=server.name or str(id),
central_version=get_central_agent_version(),
)
if skip is not None:
return {
"success": True,
"server_id": id,
"server_name": server.name,
"stdout": skip.get("stdout", ""),
"skipped": True,
}
# Validate prerequisites
base_url = (settings.API_BASE_URL or "").strip().rstrip("/")
if not base_url:
+71
View File
@@ -159,6 +159,75 @@ async def mark_agent_uninstalled(sid: int) -> None:
await session.commit()
_REMOTE_AGENT_VERSION_CMD = (
"if [ -f /opt/nexus-agent/agent.py ]; then "
"grep -m1 '\"agent_version\"' /opt/nexus-agent/agent.py 2>/dev/null | "
"sed -n 's/.*\"agent_version\": *\"\\([^\"]*\\)\".*/\\1/p'; fi"
)
async def _redis_agent_version(server_id: int) -> str:
from server.infrastructure.redis.client import get_redis
try:
redis = get_redis()
data = await redis.hgetall(f"heartbeat:{server_id}")
if data:
return (data.get("agent_version") or "").strip()
except Exception as e:
logger.warning("Redis agent_version read failed server_id=%s: %s", server_id, e)
return ""
async def resolve_remote_agent_version(server: Server, *, ssh_user: str) -> str:
"""Best-effort remote version: Redis heartbeat → MySQL → SSH grep agent.py."""
from server.infrastructure.ssh.asyncssh_pool import exec_ssh_command
for candidate in (
await _redis_agent_version(server.id),
(server.agent_version or "").strip(),
):
if candidate:
return candidate
try:
cmd = sudo_wrap(_REMOTE_AGENT_VERSION_CMD, ssh_user)
r = await exec_ssh_command(server, cmd, timeout=15)
if r["exit_code"] == 0:
return (r.get("stdout") or "").strip().split("\n")[0].strip()
except Exception as e:
logger.warning("SSH agent_version probe failed server_id=%s: %s", server.id, e)
return ""
async def agent_install_skip_result(
*,
server: Server,
server_id: int,
server_name: str,
central_version: str,
) -> dict | None:
"""Return success result dict when install can be skipped; None to proceed with install."""
from server.utils.agent_version import is_version_at_least
has_key = bool((server.agent_api_key or "").strip())
db_version = (server.agent_version or "").strip()
if not has_key and not db_version:
return None
ssh_user = (server.username or "root").strip()
remote_version = await resolve_remote_agent_version(server, ssh_user=ssh_user)
if remote_version and is_version_at_least(remote_version, central_version):
return result_item_dict(
server_id=server_id,
server_name=server_name,
success=True,
stdout=(
f"已安装 Agent {remote_version},不低于主站 {central_version},已跳过"
),
)
return None
def result_item_dict(
*,
server_id: int,
@@ -177,6 +246,7 @@ def result_item_dict(
__all__ = [
"agent_install_skip_result",
"batch_server_display_name",
"ensure_agent_api_key",
"generate_agent_api_key_value",
@@ -184,6 +254,7 @@ __all__ = [
"mark_agent_uninstalled",
"posix_dirname",
"prefetch_batch_context",
"resolve_remote_agent_version",
"result_item_dict",
"sudo_wrap",
"update_server_target_path",
@@ -8,6 +8,7 @@ import shlex
from typing import Any, Optional
from server.application.server_batch_common import (
agent_install_skip_result,
batch_server_display_name,
ensure_agent_api_key,
install_error_msg,
@@ -18,6 +19,7 @@ from server.application.server_batch_common import (
sudo_wrap,
update_server_target_path,
)
from server.utils.agent_version import get_central_agent_version
from server.config import settings
from server.domain.models import AuditLog, Server
from server.infrastructure.database.audit_log_repo import AuditLogRepositoryImpl
@@ -91,6 +93,106 @@ async def _notify_complete(live: dict[str, Any]) -> None:
logger.warning("server batch complete notify failed: %s", e)
async def _reconcile_live_job(
live: dict[str, Any],
*,
error: str,
auto_stuck: bool,
) -> bool:
"""Fill pending servers as failed, finalize, persist, audit, notify. Returns True if reconciled."""
if live.get("status") != "running":
return False
pending = sbs.pending_server_ids(live)
if not pending and sbs.progress_stats(live)["remaining"] == 0:
sbs.finalize_status(live)
await sbs.save_live(live)
else:
added = sbs.apply_pending_failures(
live,
server_ids=pending,
error=error,
auto_stuck=auto_stuck,
)
if added == 0 and not pending:
return False
sbs.finalize_status(live)
await sbs.save_live(live)
await _persist_job_finalize(live)
await _write_audit(live)
await _notify_complete(live)
job_id = int(live["id"])
logger.info(
"Server batch job reconciled job_id=%s op=%s status=%s auto_stuck=%s",
job_id,
live.get("op"),
live.get("status"),
auto_stuck,
)
return True
async def recover_orphaned_batch_jobs() -> int:
"""On API startup: finalize batch jobs left running after process restart."""
from server.infrastructure.database.server_batch_job_repo import (
ServerBatchJobRepositoryImpl,
)
reconciled_ids: set[int] = set()
count = 0
for live in await sbs.list_running_live_jobs():
job_id = int(live["id"])
if job_id in reconciled_ids:
continue
if await _reconcile_live_job(
live,
error=sbs.INTERRUPTED_MESSAGE,
auto_stuck=False,
):
reconciled_ids.add(job_id)
count += 1
async with AsyncSessionLocal() as session:
repo = ServerBatchJobRepositoryImpl(session)
for row in await repo.list_running():
job_id = int(row.id)
if job_id in reconciled_ids:
continue
live = await sbs.get_live(job_id)
if live is None:
live = sbs.live_from_mysql_row(row)
elif live.get("status") != "running":
continue
if await _reconcile_live_job(
live,
error=sbs.INTERRUPTED_MESSAGE,
auto_stuck=False,
):
reconciled_ids.add(job_id)
count += 1
if count:
logger.info("Recovered %s orphaned server batch job(s) after startup", count)
return count
async def detect_stuck_batch_jobs() -> int:
"""Mark running batch jobs with no progress for STUCK_NO_PROGRESS_SECONDS."""
count = 0
for live in await sbs.list_running_live_jobs():
if not sbs.is_stale_running(live):
continue
if await _reconcile_live_job(
live,
error=sbs.STUCK_MESSAGE,
auto_stuck=True,
):
count += 1
if count:
logger.info("Auto-reconciled %s stuck server batch job(s)", count)
return count
async def start_batch_job(
*,
op: str,
@@ -234,9 +336,11 @@ async def _run_background(job_id: int) -> None:
await sbs.save_live(live)
return
live = await sbs.get_live(job_id)
if live and live.get("status") == "running":
sbs.finalize_status(live)
await sbs.save_live(live)
if live:
stats = sbs.progress_stats(live)
if live.get("status") == "running" or stats["remaining"] == 0:
sbs.finalize_status(live)
await sbs.save_live(live)
except Exception:
logger.exception("Server batch job failed job_id=%s op=%s", job_id, op)
live = await sbs.get_live(job_id)
@@ -245,6 +349,9 @@ async def _run_background(job_id: int) -> None:
await sbs.save_live(live)
live = await sbs.get_live(job_id)
if live:
if live.get("status") == "running":
sbs.finalize_status(live)
await sbs.save_live(live)
await _persist_job_finalize(live)
await _write_audit(live)
await _notify_complete(live)
@@ -425,12 +532,22 @@ async def _run_install_agent(live: dict[str, Any]) -> None:
))
return
central_version = get_central_agent_version()
async def handler(sid: int, server_map: dict[int, Server], labels: dict[int, str], _live) -> dict:
server = server_map.get(sid)
label = labels[sid]
if not server:
return result_item_dict(server_id=sid, server_name=label, success=False, error="服务器不存在")
try:
skip = await agent_install_skip_result(
server=server,
server_id=sid,
server_name=label,
central_version=central_version,
)
if skip is not None:
return skip
api_key = await ensure_agent_api_key(server)
ssh_user = (server.username or "root").strip()
install_cmd = (
@@ -0,0 +1,27 @@
"""Background reconciliation for server batch jobs — stuck detection (60s interval)."""
from __future__ import annotations
import asyncio
import logging
from server.application.services.server_batch_service import detect_stuck_batch_jobs
from server.infrastructure.redis.server_batch_store import RECONCILE_INTERVAL_SECONDS
logger = logging.getLogger("nexus.server_batch_reconcile")
async def server_batch_reconcile_loop() -> None:
"""Every minute: auto-finalize batch jobs with no Redis progress for 1h."""
logger.info(
"Server batch reconcile loop started (interval=%ss)",
RECONCILE_INTERVAL_SECONDS,
)
while True:
await asyncio.sleep(RECONCILE_INTERVAL_SECONDS)
try:
stuck = await detect_stuck_batch_jobs()
if stuck:
logger.info("Server batch reconcile: %s stuck job(s) finalized", stuck)
except Exception as e:
logger.error("Server batch reconcile loop error: %s", e)
+42
View File
@@ -0,0 +1,42 @@
"""Periodic purge of sync_logs older than SYNC_LOG_RETENTION_DAYS (design: 30 days)."""
from __future__ import annotations
import asyncio
import logging
from server.infrastructure.database.session import AsyncSessionLocal
from server.infrastructure.database.sync_log_repo import (
SYNC_LOG_RETENTION_DAYS,
SyncLogRepositoryImpl,
)
logger = logging.getLogger("nexus.sync_log_purge")
PURGE_INTERVAL_SECONDS = 24 * 3600
async def purge_stale_sync_logs() -> int:
"""Delete sync_logs rows older than retention window. Returns rows removed."""
async with AsyncSessionLocal() as session:
repo = SyncLogRepositoryImpl(session)
return await repo.purge_older_than(retention_days=SYNC_LOG_RETENTION_DAYS)
async def sync_log_purge_loop() -> None:
"""Run daily on primary worker; also invoked once at API startup."""
while True:
try:
count = await purge_stale_sync_logs()
if count:
logger.info(
"Sync log purge: removed %s row(s) older than %s days",
count,
SYNC_LOG_RETENTION_DAYS,
)
await asyncio.sleep(PURGE_INTERVAL_SECONDS)
except asyncio.CancelledError:
raise
except Exception:
logger.exception("Sync log purge loop error")
await asyncio.sleep(PURGE_INTERVAL_SECONDS)
@@ -68,6 +68,12 @@ class ServerBatchJobRepositoryImpl:
await self.session.commit()
return row
async def list_running(self) -> list[ServerBatchJob]:
result = await self.session.execute(
select(ServerBatchJob).where(ServerBatchJob.status == "running")
)
return list(result.scalars().all())
async def list_recent(
self,
*,
@@ -2,11 +2,14 @@
from datetime import datetime, timedelta, timezone
from typing import Optional, List, Tuple
from sqlalchemy import select, func
from sqlalchemy import delete, func, select
from sqlalchemy.ext.asyncio import AsyncSession
from server.domain.models import SyncLog, Server
SYNC_LOG_RETENTION_DAYS = 30
SYNC_LOG_PURGE_BATCH_SIZE = 1000
_STALE_RUNNING_MESSAGE = (
"推送状态异常:长时间未完成(已自动订正,可能因历史缺陷未落库终态)"
)
@@ -113,6 +116,33 @@ class SyncLogRepositoryImpl:
await self.session.commit()
return len(stale)
async def purge_older_than(
self,
retention_days: int = SYNC_LOG_RETENTION_DAYS,
batch_size: int = SYNC_LOG_PURGE_BATCH_SIZE,
) -> int:
"""Delete sync_logs rows with started_at older than retention_days (batched)."""
if retention_days <= 0:
return 0
cutoff = datetime.now(timezone.utc) - timedelta(days=retention_days)
total = 0
while True:
result = await self.session.execute(
select(SyncLog.id)
.where(SyncLog.started_at < cutoff)
.order_by(SyncLog.id)
.limit(batch_size)
)
ids = [int(row[0]) for row in result.all()]
if not ids:
break
await self.session.execute(delete(SyncLog).where(SyncLog.id.in_(ids)))
await self.session.commit()
total += len(ids)
if len(ids) < batch_size:
break
return total
# Allowed fields for update_status — prevents arbitrary field injection via **kwargs
SYNC_LOG_UPDATABLE_FIELDS = {"finished_at", "duration_seconds", "error_message", "diff_summary", "status"}
@@ -14,6 +14,10 @@ logger = logging.getLogger("nexus.server_batch_store")
REDIS_KEY_PREFIX = "server_batch:"
REDIS_NEXT_ID = "server_batch:next_id"
REDIS_TTL_SECONDS = 7200
RECONCILE_INTERVAL_SECONDS = 60
STUCK_NO_PROGRESS_SECONDS = 3600
INTERRUPTED_MESSAGE = "服务重启导致任务中断"
STUCK_MESSAGE = "执行超时无响应(系统自动标记)"
TERMINAL_STATUSES = frozenset({"completed", "failed", "partial", "cancelled"})
@@ -64,6 +68,29 @@ async def save_live(live: dict[str, Any]) -> None:
await redis.set(_key(job_id), json.dumps(live, ensure_ascii=False), ex=REDIS_TTL_SECONDS)
async def list_running_live_jobs() -> list[dict[str, Any]]:
"""Scan Redis for all server batch jobs still in running status."""
redis = get_redis()
running: list[dict[str, Any]] = []
cursor = 0
while True:
cursor, keys = await redis.scan(cursor, match=f"{REDIS_KEY_PREFIX}*", count=50)
for key in keys:
if key == REDIS_NEXT_ID:
continue
suffix = key.replace(REDIS_KEY_PREFIX, "", 1)
try:
job_id = int(suffix)
except ValueError:
continue
live = await get_live(job_id)
if live and live.get("status") == "running":
running.append(live)
if cursor == 0:
break
return running
async def get_live(job_id: int) -> Optional[dict[str, Any]]:
redis = get_redis()
raw = await redis.get(_key(job_id))
@@ -121,6 +148,108 @@ def finalize_status(live: dict[str, Any]) -> str:
return status
def parse_iso(ts: str) -> Optional[datetime]:
try:
return datetime.fromisoformat(ts)
except (ValueError, TypeError):
return None
def pending_server_ids(live: dict[str, Any]) -> list[int]:
"""Server IDs in the job with no result row yet."""
server_ids = live.get("server_ids") or []
results = live.get("results") or {}
pending: list[int] = []
for sid in server_ids:
if str(sid) not in results:
pending.append(int(sid))
return pending
def apply_pending_failures(
live: dict[str, Any],
*,
server_ids: list[int],
error: str,
auto_stuck: bool = False,
server_name_fallback: str = "未知服务器",
) -> int:
"""Mark pending servers as failed; returns count of rows added."""
if not server_ids:
return 0
results = live.setdefault("results", {})
added = 0
for sid in server_ids:
key = str(sid)
if key in results:
continue
item: dict[str, Any] = {
"server_id": sid,
"server_name": server_name_fallback,
"success": False,
"stdout": "",
"error": error,
}
if auto_stuck:
item["auto_stuck"] = True
results[key] = item
added += 1
return added
def is_stale_running(live: dict[str, Any], now: Optional[datetime] = None) -> bool:
"""True when a running job has had no Redis update for STUCK_NO_PROGRESS_SECONDS."""
if live.get("status") != "running":
return False
now = now or datetime.now(timezone.utc)
updated = parse_iso(live.get("updated_at", ""))
if not updated:
updated = parse_iso(live.get("created_at", ""))
if not updated:
return False
return (now - updated).total_seconds() >= STUCK_NO_PROGRESS_SECONDS
def live_from_mysql_row(row: Any) -> dict[str, Any]:
"""Rebuild minimal Redis-shaped live dict from a ServerBatchJob row."""
import json
try:
server_ids = json.loads(row.server_ids or "[]")
except json.JSONDecodeError:
server_ids = []
try:
raw_results = json.loads(row.results or "[]")
except json.JSONDecodeError:
raw_results = []
results: dict[str, Any] = {}
if isinstance(raw_results, dict):
results = dict(raw_results)
elif isinstance(raw_results, list):
for entry in raw_results:
if isinstance(entry, dict) and entry.get("server_id") is not None:
results[str(entry["server_id"])] = entry
try:
params = json.loads(row.params or "{}") if row.params else {}
except json.JSONDecodeError:
params = {}
if not isinstance(params, dict):
params = {}
started = row.started_at.isoformat() if row.started_at else _now_iso()
return {
"id": int(row.id),
"op": row.op or "",
"label": row.label or "",
"status": row.status or "running",
"server_ids": list(server_ids),
"results": results,
"operator": row.operator or "",
"params": params,
"created_at": started,
"updated_at": started,
}
def live_to_api(live: dict[str, Any]) -> dict[str, Any]:
stats = progress_stats(live)
results = live.get("results") or {}
+18 -2
View File
@@ -80,6 +80,7 @@ from server.background.schedule_runner import schedule_runner_loop
from server.background.retry_runner import retry_runner_loop
from server.background.ip_allowlist_refresh import ip_allowlist_refresh_loop
from server.background.upload_staging_cleanup import upload_staging_cleanup_loop
from server.background.sync_log_purge import sync_log_purge_loop
logger = logging.getLogger("nexus")
@@ -286,6 +287,13 @@ async def lifespan(app: FastAPI):
# Use Redis-based leader election to ensure only one worker runs background tasks.
is_primary = await _acquire_primary_lock()
if is_primary:
from server.application.services.server_batch_service import recover_orphaned_batch_jobs
from server.background.server_batch_reconcile import server_batch_reconcile_loop
recovered = await recover_orphaned_batch_jobs()
if recovered:
logger.info("Startup: recovered %s orphaned server batch job(s)", recovered)
task_flush = asyncio.create_task(heartbeat_flush_loop(), name="heartbeat_flush")
task_script_flush = asyncio.create_task(
script_execution_flush_loop(), name="script_execution_flush"
@@ -297,6 +305,12 @@ async def lifespan(app: FastAPI):
task_upload_cleanup = asyncio.create_task(
upload_staging_cleanup_loop(), name="upload_staging_cleanup"
)
task_batch_reconcile = asyncio.create_task(
server_batch_reconcile_loop(), name="server_batch_reconcile"
)
task_sync_log_purge = asyncio.create_task(
sync_log_purge_loop(), name="sync_log_purge"
)
_background_tasks.extend([
task_flush,
task_script_flush,
@@ -305,6 +319,8 @@ async def lifespan(app: FastAPI):
task_retry,
task_ip_refresh,
task_upload_cleanup,
task_batch_reconcile,
task_sync_log_purge,
])
logger.info("Primary worker — background tasks launched")
else:
@@ -317,8 +333,8 @@ async def lifespan(app: FastAPI):
logger.info(
f"{settings.SYSTEM_NAME} v6.0.0 started — "
f"background tasks: heartbeat_flush(10min), script_execution_flush(60s), "
f"self_monitor(30s), schedule_runner(60s), retry_runner(5min), "
f"upload_staging_cleanup(1h)"
f"server_batch_reconcile(60s), sync_log_purge(24h), self_monitor(30s), "
f"schedule_runner(60s), retry_runner(5min), upload_staging_cleanup(1h)"
)
yield
+56
View File
@@ -0,0 +1,56 @@
"""Central Nexus Agent version (SSOT) and semver-style comparison."""
from __future__ import annotations
import re
from functools import lru_cache
from pathlib import Path
_AGENT_PY = Path(__file__).resolve().parents[2] / "web" / "agent" / "agent.py"
_VERSION_IN_AGENT_RE = re.compile(r'"agent_version":\s*"([^"]+)"')
@lru_cache(maxsize=1)
def get_central_agent_version() -> str:
"""Read agent_version string shipped at /agent/agent.py (heartbeat payload)."""
text = _AGENT_PY.read_text(encoding="utf-8")
match = _VERSION_IN_AGENT_RE.search(text)
if not match:
raise RuntimeError(f"agent_version not found in {_AGENT_PY}")
return match.group(1)
def version_tuple(version: str) -> tuple[int, ...]:
"""Parse dotted numeric prefix, e.g. 2.0.0 -> (2, 0, 0)."""
version = (version or "").strip()
if not version:
return ()
parts: list[int] = []
for segment in re.split(r"[.-]", version):
if segment.isdigit():
parts.append(int(segment))
else:
break
return tuple(parts)
def is_version_lower(remote: str, central: str) -> bool:
"""True when remote is strictly older than central."""
remote_t = version_tuple(remote)
central_t = version_tuple(central)
if not remote_t:
return True
if not central_t:
return False
return remote_t < central_t
def is_version_at_least(remote: str, central: str) -> bool:
"""True when remote >= central (equal counts as skip)."""
remote_t = version_tuple(remote)
central_t = version_tuple(central)
if not remote_t:
return False
if not central_t:
return bool(remote_t)
return remote_t >= central_t
+84
View File
@@ -0,0 +1,84 @@
"""Agent version comparison and install-skip decision."""
from unittest.mock import AsyncMock, patch
import pytest
from server.application.server_batch_common import agent_install_skip_result
from server.domain.models import Server
from server.utils.agent_version import (
get_central_agent_version,
is_version_at_least,
is_version_lower,
version_tuple,
)
def test_version_tuple():
assert version_tuple("2.0.0") == (2, 0, 0)
assert version_tuple("1.10.3") == (1, 10, 3)
assert version_tuple("") == ()
def test_is_version_lower():
assert is_version_lower("1.9.0", "2.0.0") is True
assert is_version_lower("2.0.0", "2.0.0") is False
assert is_version_lower("2.1.0", "2.0.0") is False
def test_is_version_at_least():
assert is_version_at_least("2.0.0", "2.0.0") is True
assert is_version_at_least("2.0.1", "2.0.0") is True
assert is_version_at_least("1.9.9", "2.0.0") is False
def test_central_agent_version_matches_agent_py():
assert get_central_agent_version() == "2.0.0"
@pytest.mark.asyncio
async def test_agent_install_skip_when_version_current():
server = Server(id=1, name="A", agent_api_key="nxs-k", agent_version="2.0.0")
with patch(
"server.application.server_batch_common.resolve_remote_agent_version",
new_callable=AsyncMock,
return_value="2.0.0",
):
result = await agent_install_skip_result(
server=server,
server_id=1,
server_name="A",
central_version="2.0.0",
)
assert result is not None
assert result["success"] is True
assert "已跳过" in result["stdout"]
@pytest.mark.asyncio
async def test_agent_install_no_skip_when_version_lower():
server = Server(id=2, name="B", agent_api_key="nxs-k", agent_version="1.0.0")
with patch(
"server.application.server_batch_common.resolve_remote_agent_version",
new_callable=AsyncMock,
return_value="1.0.0",
):
result = await agent_install_skip_result(
server=server,
server_id=2,
server_name="B",
central_version="2.0.0",
)
assert result is None
@pytest.mark.asyncio
async def test_agent_install_no_skip_when_never_installed():
server = Server(id=3, name="C", agent_api_key=None, agent_version=None)
result = await agent_install_skip_result(
server=server,
server_id=3,
server_name="C",
central_version="2.0.0",
)
assert result is None
+81
View File
@@ -0,0 +1,81 @@
"""Server batch zombie recovery and stuck detection helpers."""
import json
from datetime import datetime, timedelta, timezone
from server.domain.models import ServerBatchJob
from server.infrastructure.database.server_batch_job_repo import job_row_to_api
from server.infrastructure.redis import server_batch_store as sbs
def test_pending_server_ids():
live = {
"server_ids": [1, 2, 3],
"results": {"1": {"server_id": 1, "success": True}},
}
assert sbs.pending_server_ids(live) == [2, 3]
def test_apply_pending_failures_marks_stuck():
live = {"server_ids": [10], "results": {}}
added = sbs.apply_pending_failures(
live,
server_ids=[10],
error=sbs.STUCK_MESSAGE,
auto_stuck=True,
server_name_fallback="S10",
)
assert added == 1
entry = live["results"]["10"]
assert entry["success"] is False
assert entry["auto_stuck"] is True
assert entry["error"] == sbs.STUCK_MESSAGE
def test_finalize_status_after_pending_failures():
live = {
"server_ids": [1, 2],
"results": {},
"status": "running",
}
sbs.apply_pending_failures(
live,
server_ids=[1, 2],
error=sbs.INTERRUPTED_MESSAGE,
)
status = sbs.finalize_status(live)
assert status == "failed"
assert live["status"] == "failed"
def test_is_stale_running_true_when_old_update():
old = (datetime.now(timezone.utc) - timedelta(hours=2)).isoformat()
live = {"status": "running", "updated_at": old}
assert sbs.is_stale_running(live) is True
def test_is_stale_running_false_when_recent():
recent = datetime.now(timezone.utc).isoformat()
live = {"status": "running", "updated_at": recent}
assert sbs.is_stale_running(live) is False
def test_live_from_mysql_row_rebuilds_results_map():
row = ServerBatchJob(
id=20,
op="install-agent",
label="批量安装 Agent",
server_ids=json.dumps([290, 346]),
status="running",
results=json.dumps([
{"server_id": 290, "server_name": "A", "success": True, "stdout": "ok", "error": ""},
]),
operator="admin",
started_at=datetime(2026, 6, 8, 10, 0, tzinfo=timezone.utc),
)
live = sbs.live_from_mysql_row(row)
assert live["id"] == 20
assert live["status"] == "running"
assert sbs.pending_server_ids(live) == [346]
api = job_row_to_api(row)
assert api["remaining"] == 1
+18
View File
@@ -0,0 +1,18 @@
"""sync_logs retention purge helpers."""
from datetime import datetime, timedelta, timezone
from server.infrastructure.database.sync_log_repo import (
SYNC_LOG_PURGE_BATCH_SIZE,
SYNC_LOG_RETENTION_DAYS,
)
def test_sync_log_retention_constants():
assert SYNC_LOG_RETENTION_DAYS == 30
assert SYNC_LOG_PURGE_BATCH_SIZE >= 100
def test_purge_cutoff_is_30_days_ago():
cutoff = datetime.now(timezone.utc) - timedelta(days=SYNC_LOG_RETENTION_DAYS)
assert (datetime.now(timezone.utc) - cutoff).days == SYNC_LOG_RETENTION_DAYS