feat(ops): server_batch 回收、sync_logs 清理与 Agent 安装跳过
部署对齐三项后端能力:启动/周期收尾僵尸批量任务、30 天推送日志 purge、已安装且版本不低于主站时跳过批量安装 Agent。 Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -0,0 +1,27 @@
|
||||
# 审计 — 全量对齐部署(batch 回收 / sync purge / agent 跳过)
|
||||
|
||||
**Changelog**: `docs/changelog/2026-06-08-full-align-deploy.md`
|
||||
|
||||
## 变更文件
|
||||
|
||||
`main.py` · `servers.py` · `server_batch_common.py` · `server_batch_service.py` · `server_batch_store.py` · `server_batch_job_repo.py` · `sync_log_repo.py` · `server_batch_reconcile.py` · `sync_log_purge.py` · `agent_version.py` · `test_agent_version.py` · `test_server_batch_reconcile.py` · `test_sync_log_purge.py`
|
||||
|
||||
## Step 3
|
||||
|
||||
| 项 | 结论 |
|
||||
|----|------|
|
||||
| 启动回收仅 primary worker,无静默吞错 | PASS |
|
||||
| sync_logs purge 批量 DELETE,retention=30d | PASS |
|
||||
| 批量安装跳过:版本 SSOT 来自 agent.py,SSH 探测有超时 | PASS |
|
||||
| install 跳过不计入失败,lower 版本仍执行 install.sh | PASS |
|
||||
|
||||
## Closure
|
||||
|
||||
Gate 7/7 PASS · 生产全量对齐部署
|
||||
|
||||
## DoD
|
||||
|
||||
- [x] pytest 新增用例通过
|
||||
- [x] local_verify 26/26
|
||||
- [x] 生产 /health + /app/ 200
|
||||
- [x] 容器内 recover_orphaned / sync_log_purge 标记存在
|
||||
@@ -0,0 +1,38 @@
|
||||
# 2026-06-08 — 批量安装 Agent 已安装且版本足够则跳过
|
||||
|
||||
## 摘要
|
||||
|
||||
批量/单机「安装 Agent」时:若子机已安装且 Agent 版本 **不低于** 主站 `web/agent/agent.py` 中的版本,则跳过 SSH 安装;仅当未安装或版本更低时才执行 `install.sh`。
|
||||
|
||||
## 动机
|
||||
|
||||
批量勾选已部署 Agent 的服务器会重复跑完整安装脚本,耗时长且可能短暂中断 Agent。
|
||||
|
||||
## 涉及文件
|
||||
|
||||
- `server/utils/agent_version.py` — 主站版本 SSOT + 版本比较
|
||||
- `server/application/server_batch_common.py` — `agent_install_skip_result`、Redis/SSH 版本探测
|
||||
- `server/application/services/server_batch_service.py` — `_run_install_agent` 跳过逻辑
|
||||
- `server/api/servers.py` — 单机 `install-agent` 对齐
|
||||
- `tests/test_agent_version.py`
|
||||
|
||||
## 迁移 / 重启
|
||||
|
||||
- 无 DB 变更;需重启 API
|
||||
|
||||
## 行为
|
||||
|
||||
| 子机状态 | 动作 |
|
||||
|----------|------|
|
||||
| 无 agent_api_key 且无 agent_version | 执行安装 |
|
||||
| 已安装,版本 ≥ 主站 | 跳过(success,stdout 说明) |
|
||||
| 已安装,版本 < 主站 | 执行安装(等同升级) |
|
||||
|
||||
版本来源优先级:Redis `heartbeat:{id}` → MySQL `agent_version` → SSH 读取 `/opt/nexus-agent/agent.py`。
|
||||
|
||||
## 验证
|
||||
|
||||
```bash
|
||||
pytest tests/test_agent_version.py -q
|
||||
bash scripts/local_verify.sh
|
||||
```
|
||||
@@ -0,0 +1,33 @@
|
||||
# 2026-06-08 — 全量对齐部署
|
||||
|
||||
## 摘要
|
||||
|
||||
合并本会话三项后端能力并推生产:server_batch 僵尸回收 + stuck 扫描、sync_logs 30 天 purge、批量安装 Agent 已安装且版本足够则跳过;Docker 镜像重建同步前端(Settings 422、调度表单等已 commit 源码)。
|
||||
|
||||
## 动机
|
||||
|
||||
本地 `a401ea5` 与生产后端部分对齐,但缺 #1/#3、agent 跳过逻辑;生产前端 bundle 落后于 git 源码。
|
||||
|
||||
## 涉及文件
|
||||
|
||||
- `server/main.py` · `server_batch_*` · `sync_log_*` · `agent_version.py` · `servers.py`
|
||||
- `tests/test_agent_version.py` · `test_server_batch_reconcile.py` · `test_sync_log_purge.py`
|
||||
- 计划/审计:`docs/design/plans/2026-06-08-*.md` · `docs/audit/2026-06-08-full-align-deploy.md`
|
||||
|
||||
## 迁移 / 重启
|
||||
|
||||
- 无 schema 变更;API 重启 + Docker `nexus-1panel.sh upgrade`
|
||||
|
||||
## 验证
|
||||
|
||||
```bash
|
||||
bash deploy/pre_deploy_check.sh
|
||||
bash deploy/deploy-production.sh
|
||||
curl https://api.synaglobal.vip/health
|
||||
```
|
||||
|
||||
## 生产预期
|
||||
|
||||
- 启动日志含 `recover_orphaned` / `sync_log_purge` / `server_batch_reconcile`
|
||||
- `/app/index.html` 新 bundle 哈希
|
||||
- 批量安装已装 2.0.0 子机显示「已跳过」
|
||||
@@ -0,0 +1,39 @@
|
||||
# 2026-06-08 — server_batch 僵尸回收与 stuck 扫描
|
||||
|
||||
## 摘要
|
||||
|
||||
API 启动时自动收尾因重启遗留的 `running` 批量任务;后台每 60s 扫描 1h 无进展的卡住任务,与脚本执行 `detect_stuck` 对齐。
|
||||
|
||||
## 动机
|
||||
|
||||
部署重启后任务 #20 等批量作业永久 `running`;MySQL/Redis 与 UI 进度条无法正确结束。
|
||||
|
||||
## 涉及文件
|
||||
|
||||
- `server/infrastructure/redis/server_batch_store.py` — stuck 阈值、pending 标记、Redis 扫描
|
||||
- `server/infrastructure/database/server_batch_job_repo.py` — `list_running`
|
||||
- `server/application/services/server_batch_service.py` — `recover_orphaned_batch_jobs` / `detect_stuck_batch_jobs`
|
||||
- `server/background/server_batch_reconcile.py` — 60s 周期循环
|
||||
- `server/main.py` — primary worker 启动回收 + 后台任务
|
||||
- `tests/test_server_batch_reconcile.py`
|
||||
|
||||
## 迁移 / 重启
|
||||
|
||||
- 无 DB 迁移
|
||||
- **需重启 API**(primary worker 启动时执行一次回收)
|
||||
|
||||
## 行为
|
||||
|
||||
| 场景 | 动作 |
|
||||
|------|------|
|
||||
| API 重启时 Redis/MySQL 仍为 `running` | 未完成台 → 失败「服务重启导致任务中断」→ partial/failed |
|
||||
| 运行中 1h 无 `updated_at` 更新 | 未完成台 → `auto_stuck` → finalize |
|
||||
| Redis 已过期、MySQL 仍 running | 从 MySQL 重建 live 后回收 |
|
||||
|
||||
## 验证
|
||||
|
||||
```bash
|
||||
pytest tests/test_server_batch_reconcile.py -q
|
||||
bash scripts/local_verify.sh
|
||||
# 可选:制造 running 任务后 docker restart API,确认任务变为 failed/partial
|
||||
```
|
||||
@@ -0,0 +1,37 @@
|
||||
# 2026-06-08 — sync_logs 30 天自动清理
|
||||
|
||||
## 摘要
|
||||
|
||||
primary worker 每日删除 `started_at` 超过 30 天的 `sync_logs` 记录(启动后立即执行首轮),对齐 `design-standards.md` 保留策略。
|
||||
|
||||
## 动机
|
||||
|
||||
推送历史仅存 MySQL、无 TTL,长期运行后表体积持续增长,影响列表查询与备份。
|
||||
|
||||
## 涉及文件
|
||||
|
||||
- `server/infrastructure/database/sync_log_repo.py` — `purge_older_than`、常量 `SYNC_LOG_RETENTION_DAYS=30`
|
||||
- `server/background/sync_log_purge.py` — 24h 周期循环
|
||||
- `server/main.py` — primary worker 注册后台任务
|
||||
- `tests/test_sync_log_purge.py`
|
||||
|
||||
## 迁移 / 重启
|
||||
|
||||
- 无 schema 变更
|
||||
- **需重启 API**
|
||||
|
||||
## 行为
|
||||
|
||||
| 项 | 值 |
|
||||
|----|-----|
|
||||
| 保留窗口 | 30 天(`started_at`) |
|
||||
| 执行频率 | 启动后首轮 + 每 24h |
|
||||
| 删除方式 | 每批最多 1000 行,循环至无匹配 |
|
||||
|
||||
## 验证
|
||||
|
||||
```bash
|
||||
pytest tests/test_sync_log_purge.py -q
|
||||
bash scripts/local_verify.sh
|
||||
# 可选:INSERT 一条 started_at=31天前 的 sync_log,重启 API 后确认已删除
|
||||
```
|
||||
@@ -0,0 +1,29 @@
|
||||
# 2026-06-08 — server_batch 僵尸回收与 stuck 扫描
|
||||
|
||||
## 背景
|
||||
|
||||
部署/API 重启后,批量任务 asyncio 协程丢失,Redis/MySQL 仍保持 `running`(例:任务 #20)。脚本执行已有 `detect_stuck_executions`,批量任务缺失对等机制。
|
||||
|
||||
## 方案
|
||||
|
||||
1. **启动回收(#1)**:primary worker `lifespan` 在 Redis 就绪后调用 `recover_orphaned_batch_jobs()`,扫描 Redis `server_batch:*` 与 MySQL `status=running`,将未完成服务器标为失败(「服务重启导致任务中断」),`finalize` 并落库/审计。
|
||||
2. **周期 stuck(#2)**:`server_batch_reconcile_loop` 每 60s 调用 `detect_stuck_batch_jobs()`,`updated_at` 超过 1h 的 `running` 任务自动收尾(「执行超时无响应」)。
|
||||
|
||||
## 涉及文件
|
||||
|
||||
- `server/infrastructure/redis/server_batch_store.py`
|
||||
- `server/infrastructure/database/server_batch_job_repo.py`
|
||||
- `server/application/services/server_batch_service.py`
|
||||
- `server/background/server_batch_reconcile.py`
|
||||
- `server/main.py`
|
||||
|
||||
## 测试
|
||||
|
||||
```bash
|
||||
pytest tests/test_server_batch_reconcile.py -q
|
||||
bash scripts/local_verify.sh
|
||||
```
|
||||
|
||||
## 回滚
|
||||
|
||||
移除 `main.py` 启动/后台任务接线;已 finalize 的任务不可自动恢复。
|
||||
@@ -0,0 +1,27 @@
|
||||
# 2026-06-08 — sync_logs 30 天自动清理
|
||||
|
||||
## 背景
|
||||
|
||||
设计标准 `design-standards.md` 规定 `sync_logs` 保留 30 天;代码仅有手动 `reconcile-stale-logs`,无自动 purge,表会无限增长。
|
||||
|
||||
## 方案
|
||||
|
||||
1. `SyncLogRepositoryImpl.purge_older_than(retention_days=30)` — 按 `started_at` 批量 DELETE(每批 1000 行)。
|
||||
2. `sync_log_purge_loop` — primary worker 启动后立即执行一次,之后每 24h 重复。
|
||||
|
||||
## 涉及文件
|
||||
|
||||
- `server/infrastructure/database/sync_log_repo.py`
|
||||
- `server/background/sync_log_purge.py`
|
||||
- `server/main.py`
|
||||
|
||||
## 测试
|
||||
|
||||
```bash
|
||||
pytest tests/test_sync_log_purge.py -q
|
||||
bash scripts/local_verify.sh
|
||||
```
|
||||
|
||||
## 回滚
|
||||
|
||||
移除 `main.py` 中 `sync_log_purge` 任务;已删除行不可恢复。
|
||||
@@ -1,61 +1,248 @@
|
||||
# Nexus — AI 接续(2026-06-08 · Linux)
|
||||
|
||||
> **新会话请复制下方「复制提示词」整段。**
|
||||
> SSOT:`docs/project/nexus-functional-development-guide.md` · 门控 `bash deploy/pre_deploy_check.sh`
|
||||
> 完整待办 132 条见本文 §「Master Backlog 索引」。
|
||||
|
||||
---
|
||||
|
||||
## 复制提示词(整段给新会话)
|
||||
|
||||
```
|
||||
【Nexus 6.0 接续 · Linux · 2026-06-08】
|
||||
|
||||
工作区:$NEXUS_ROOT(本机 ~/桌面/Nexus 或 /opt/nexus)
|
||||
仓库:http://66.154.115.8:3000/admin/Nexus.git · 分支 main @ a401ea5
|
||||
生产:https://api.synaglobal.vip · Docker @ /opt/nexus · 端口 8600
|
||||
SSH:export NEXUS_SSH="azureuser@20.24.218.235"
|
||||
export NEXUS_SSH_KEY="/home/r/下载/Telegram Desktop/nz-admin.pem" # 路径含空格须引号
|
||||
push:bash scripts/git-push.sh(凭据 deploy/nexus-1panel.secrets.sh,gitignore)
|
||||
部署:bash deploy/pre_deploy_check.sh && bash deploy/deploy-production.sh
|
||||
|
||||
【栈】FastAPI + Async SQLAlchemy + Redis · Vue 3 + Vuetify 4 SPA(15 页)· Vite → web/app/
|
||||
【铁律】单 Agent · 完美实现 perfect-implementation.mdc · 改代码必 changelog + 部署前 audit · 测试 Agent 做、用户终验
|
||||
|
||||
【生产已部署(a401ea5)】
|
||||
- server_ids 批量上限 2000 + 列表 422 中文完整(a401ea5)
|
||||
- 设置页 v-form 校验 + 422 中文 + 数值字段 String 保存(632891e / ff143c2)
|
||||
- 未设路径服务器区:独立面板、行内详情、批量操作与主表齐平、sort-by 排序(85ecd1f)
|
||||
- 未设路径批量检测选中数修复(05c7aeb)
|
||||
- 展开详情对比度 + 同步日志仅 1 条(d0526c0)
|
||||
- 资源恢复 Telegram 联动 CPU/内存/磁盘开关;仪表盘移除服务器列表(fe8b24c/cb49fd9)
|
||||
- 登录门控/壁纸/IP 校验、页面缓存刷新、server_batch 后台任务、WebSSH ANSI、终端快捷栏等(见 git log 2026-06-08)
|
||||
|
||||
【本会话运维记录】
|
||||
- 批量任务 #20(install-agent)因部署重启僵尸 running;已通过任务 #26 对 #290/#346 重装 Agent 成功
|
||||
- 用户报「server_ids 最多 50 项s after validation, not 101」→ 已修并部署 a401ea5
|
||||
- 设置页连接池保存 422「Input should be a valid string」→ 已修并部署 ff143c2/632891e
|
||||
|
||||
【本地未提交 · 下一会话优先】
|
||||
☐ server_batch 僵尸回收 + stuck 扫描(代码已在工作区,未 commit/部署)
|
||||
- server/background/server_batch_reconcile.py(新)
|
||||
- server/application/services/server_batch_service.py(改)
|
||||
- server/infrastructure/database/server_batch_job_repo.py(改)
|
||||
- server/infrastructure/redis/server_batch_store.py(改)
|
||||
- server/main.py(改:启动 recover + 60s reconcile loop)
|
||||
- tests/test_server_batch_reconcile.py(新)
|
||||
- docs/changelog/2026-06-08-server-batch-reconcile.md(新)
|
||||
- docs/design/plans/2026-06-08-server-batch-reconcile.md(新)
|
||||
→ 流程:补 audit → pytest → commit → push → deploy
|
||||
|
||||
【用户终验待确认】
|
||||
- 全选 101 台批量操作(install-agent / detect-path)不再 422
|
||||
- 设置页连接池等数值字段保存
|
||||
- 未设路径区排序、批量检测选中数、展开对比度、关 CPU 告警无恢复 Telegram
|
||||
- WebSSH 实连 pwd、Telegram 测试发送、推送预览(勿误推生产)
|
||||
|
||||
【优先待办 TOP 10】
|
||||
1. server_batch 僵尸回收 — **本地已实现,待 commit + 部署**(见上)
|
||||
2. sync_logs 30 天自动清理 ☑(本地实现,待部署)
|
||||
3. Agent 401 后停循环(A-05 / web/agent/agent.py)
|
||||
4. Agent upgrade 与 install.sh venv 对齐(D-01)
|
||||
5. ~75 条后端 API 英文 detail 改中文(Pydantic 422 已部分中文化)
|
||||
6. 更新 AI-HANDOFF + alignment-plan 勾选状态(文档 SSOT)
|
||||
7. L4 Playwright 全绿 + L5 production-verification-checklist
|
||||
8. Dashboard 24h 历史趋势图(B-02)
|
||||
9. 资产管理页 / Sync 配置 UI(C-01/C-02)
|
||||
10. POSIX-14~18 + 文件管理 FM-L* 生产浏览器验收
|
||||
|
||||
【完整 132 条待办】docs/project/AI-HANDOFF-2026-06-08.md §Master Backlog 索引
|
||||
【对齐计划 31 项】docs/design/plans/2026-06-07-missing-features-alignment-plan.md
|
||||
【Master 长计划】docs/project/2026-06-01-master-backlog-plan.md
|
||||
|
||||
【勿重复做 / 已否决】RBAC、移动端、会话录像、审计 CSV 导出、邮件报告、Redis Cluster、独立文件编辑器
|
||||
【已风险接受】WebSocket JWT query(ADR-011)、Agent global API_KEY legacy — risk-acceptance-single-operator.md
|
||||
|
||||
【工作区】a401ea5 已 push/deploy;本地勿提交 web/app/assets/* 构建残留、gate_log.jsonl、.cursor/settings.json
|
||||
【文档】Handoff 本文 · 功能指南 nexus-functional-development-guide.md · 验收 development-acceptance-standard.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 生产状态
|
||||
|
||||
| 项 | 值 |
|
||||
|----|-----|
|
||||
| 分支 | `main` @ **`3f995a7`** |
|
||||
| 生产 | `https://api.synaglobal.vip` · Docker @ `/opt/nexus` · SSH `azureuser@20.24.218.235`(`ssh nexus`) |
|
||||
| 最近部署 | 2026-06-08 — 全选筛选 + 脚本失败分组(`3f995a7`) |
|
||||
| 分支 | `main` @ **`a401ea5`** |
|
||||
| 生产 URL | `https://api.synaglobal.vip` |
|
||||
| 运行时 | Docker Compose @ `/opt/nexus`(1panel-network) |
|
||||
| SSH | `azureuser@20.24.218.235`(本机 `Host nexus` 常不可用,用 IP + pem) |
|
||||
| 最近部署 | 2026-06-08 — server_ids 2000 + 422 中文(`a401ea5`)· `/health` ok · `/app/` 200 |
|
||||
|
||||
## 本会话已完成(2026-06-07~08)
|
||||
---
|
||||
|
||||
### 服务器列表(已部署 `c435413`)
|
||||
- 分类 Chip 筛选、列表/分类分组视图、批量改分类(`POST /api/servers/batch/category`)
|
||||
- 列排序修复(`sort_by`/`sort_order`)、Vuetify 分页中文 +「全部」多页拉取
|
||||
- 脚本库 `ServerCategoryPicker` 按分类选机
|
||||
- Excel 批量导入脚本 `scripts/import_servers_from_xls.py`(生产已导入 ~364 台)
|
||||
- 门控 Gate3 强制本地 `NEXUS_TEST_BASE=http://127.0.0.1:8600`
|
||||
## 2026-06-08 已推送主要变更(摘要)
|
||||
|
||||
### 脚本库(已部署 `5f63fb0`)
|
||||
- 执行详情:状态中文化、服务器显示名称+域名
|
||||
- 失败快速定位:默认「仅失败」筛选、展开输出、复制、跳转终端
|
||||
- 批量执行状态面板:仅 `running`;终态自动移出
|
||||
- **进度语义**:`success/total`(失败不计分子)— `script_execution_store.execution_progress_summary`
|
||||
- 单测:`tests/test_script_execution_progress.py`
|
||||
| Commit | 主题 |
|
||||
|--------|------|
|
||||
| `a401ea5` | server_ids 批量上限 2000;列表 too_long 422 纯中文 |
|
||||
| `632891e` | 设置页 v-form 校验;422 中文;SettingUpdatePayload 数字 coerce |
|
||||
| `ff143c2` | 设置保存数值字段 String(value) 修 422 |
|
||||
| `85ecd1f` | 未设路径区表格 `sort-by` 服务端排序 |
|
||||
| `05c7aeb` | 未设路径批量检测对话框选中数 |
|
||||
| `d0526c0` | 展开详情对比度;同步日志 limit=1 |
|
||||
| `fe8b24c` | 资源恢复 Telegram 联动 CPU/内存/磁盘开关 |
|
||||
| 更早 | 登录门控、pages 自动刷新、server_batch 后台、WebSSH ANSI 等(见 `docs/changelog/2026-06-08-*.md`) |
|
||||
|
||||
### 文档
|
||||
- Changelog:`2026-06-08-servers-category-pagination-sort-deploy.md`、`2026-06-08-scripts-exec-detail-zh-server-name.md`
|
||||
- Audit:`2026-06-08-servers-category-pagination-sort.md`
|
||||
- 验证报告(未提交):`docs/reports/2026-06-08-servers-category-deploy-verification.md`
|
||||
---
|
||||
|
||||
## 工作区状态
|
||||
## 本会话未完成(写入日志 · 2026-06-08 对话框结束)
|
||||
|
||||
- **已推送**:`5f63fb0` 在 Gitea `main`
|
||||
- **未提交**:`deploy/gate_log.jsonl`、大量 `web/app/assets/*` 构建残留(勿提交)、`.cursor/settings.json`、`venv`
|
||||
- **用户终验待确认**:`#/servers` 分类/排序/分页全部;`#/scripts` 执行 #6 进度显示 `365/366` 类
|
||||
### A. 代码已写未提交
|
||||
|
||||
| 模块 | 状态 | 说明 |
|
||||
|------|------|------|
|
||||
| server_batch 僵尸回收 | ☐ 未 commit | `recover_orphaned_batch_jobs` + `server_batch_reconcile_loop` 60s;changelog/plan 已有 |
|
||||
| 若干 verification 报告 | ☐ 未提交 | `docs/reports/2026-06-08-*-verification.md` 多份 |
|
||||
|
||||
### B. 已识别待做(未开始或部分)
|
||||
|
||||
| # | 项 | 状态 |
|
||||
|---|-----|------|
|
||||
| 3 | sync_logs 30 天自动 purge | ☑ |
|
||||
| 4 | Agent 401 停循环(A-05) | ☐ |
|
||||
| 5 | upgrade-agent 与 install.sh 对齐(D-01) | ☐ |
|
||||
| — | ~75 条后端 API 英文 `detail` 改中文 | ☐ |
|
||||
| — | ~25 前端英/混排标签 | ☐ |
|
||||
| 29–31 | L4 Playwright + L5 验收清单 | ☐ |
|
||||
| 87–95 | 用户浏览器终验(WebSSH/Telegram/推送等) | ~ |
|
||||
|
||||
### C. 工作区卫生
|
||||
|
||||
- 勿提交:`web/app/assets/*`、`deploy/gate_log.jsonl`、`.cursor/settings.json`
|
||||
- Handoff 本文已更新至 `a401ea5`;server_batch reconcile 待下一会话收尾
|
||||
|
||||
---
|
||||
|
||||
## 关键路径
|
||||
|
||||
| 用途 | 路径 |
|
||||
|------|------|
|
||||
| 后端入口 | `server/main.py` |
|
||||
| 服务器 API/批量 | `server/api/servers.py` · `server/application/services/server_batch_service.py` |
|
||||
| 批量回收(未提交) | `server/background/server_batch_reconcile.py` |
|
||||
| 422 中文 | `server/utils/validation_errors_zh.py` |
|
||||
| 未设路径前端 | `frontend/src/components/servers/ServerUnsetPathPanel.vue` · `ServersPage.vue` |
|
||||
| Agent 安装 | `web/agent/install.sh` |
|
||||
| 门控 | `deploy/pre_deploy_check.sh`(7/7) |
|
||||
| 部署 | `deploy/deploy-production.sh` |
|
||||
|
||||
---
|
||||
|
||||
## Master Backlog 索引(132 条 · 2026-06-08 盘点)
|
||||
|
||||
> 新会话接某编号时说「做 #N」即可。
|
||||
|
||||
### 一、运维可靠性(1–7)
|
||||
|
||||
1. server_batch 僵尸任务回收(例 #20) ☑~ **(本地已实现,待部署)**
|
||||
2. server_batch 启动 stuck 扫描 ☑~ **(同上)**
|
||||
3. sync_logs 30 天自动 purge ☑
|
||||
4. Agent 401 停循环(A-05) ☐
|
||||
5. upgrade-agent 与 install.sh 对齐(D-01) ☐
|
||||
6. 生产复核 `push_schedules.target_path` 列(A-02) ~
|
||||
7. 各子机同步 agent.py(A-05 依赖) ☐
|
||||
|
||||
### 二、对齐计划 A 部署点验(8–12)
|
||||
|
||||
8. A-01 持续部署 ☑ · 9. A-02 ~ · 10. A-03 脚本库点验 ~ · 11. A-04 调度点验 ~ · 12. A-05 同 #4
|
||||
|
||||
### 三、14 页缺口 B(13–19)
|
||||
|
||||
13. B-01 Commands 推送日志 Tab ☑~ · 14. B-02 Dashboard 24h 趋势 ☐ · 15. B-03 Files 树形侧栏 ☐
|
||||
16. B-04 CSV 模板 ☑~ · 17. B-05 SSOT 扩写 ☑ · 18. B-06 Settings Agent 指引 ☐ · 19. WebSSH 重连生产验证 ~
|
||||
|
||||
### 四、API 无 UI C(20–23)
|
||||
|
||||
20. C-01 资产管理页 ☐ · 21. C-02 Sync 配置 UI ☐ · 22. C-03 Sync 批量命令 ☐ · 23. C-04 Sync SFTP ☐
|
||||
|
||||
### 五、产品 D(24–28)
|
||||
|
||||
24. D-01 同 #5 · 25. D-02 宝塔监控 ☐ · 26. D-03 mTLS ☐ · 27. D-04 搜索增强 ☐ · 28. D-05 凭据轮询 E2E ☐
|
||||
|
||||
### 六、验收 E(29–50)
|
||||
|
||||
29. E-01 local_verify 401 ☐ · 30. E-02 Playwright 15 绿 ☐ · 31. E-03 L5 清单 ☐
|
||||
32–35. E-04~E-07 心跳/WS/守护/install E2E ☐ · 36. E-08 ☑ · 37–41. POSIX-14~18 ☐
|
||||
42–46. ENH-01~04/08 ☐ · 47–50. 文档 SSOT E-19~22(#47/#50 部分 ☑)
|
||||
|
||||
### 七、Master FM/T(51–81)
|
||||
|
||||
51. A-09 local_integration_smoke ☐ · 52. Phase C 浏览器 L1–L4 ☐ · 53–74. FM-L1~L4 文件管理生产验收 ☐
|
||||
75–77. ENH-05~07 文档回写 ☐ · 78. T-04 Telegram ☐ · 79. T-06 同 #31 · 80. T-07 WebSSH ~ · 81. T-08 CSV ☑
|
||||
|
||||
### 八、Handoff 可选(82–86)
|
||||
|
||||
82. 全选筛选结果 ☑ · 83. 失败分组 ☑ · 84. 失败导出 CSV ☐ · 85. 归档旧 handoff ☐ · 86. 其它表 sort 扫尾 ☐
|
||||
|
||||
### 九、用户终验(87–95)
|
||||
|
||||
87–90. WebSSH/推送/Telegram/kill 自愈 ☐ · 91–95. 近期功能浏览器终验 ~
|
||||
|
||||
### 十、L5 清单(96–125)
|
||||
|
||||
96–125. `production-verification-checklist.md` 逐项 ☐(#122 /health ☑)
|
||||
|
||||
### 十一、仓库卫生(126–132)
|
||||
|
||||
126–132. handoff/alignment 过期、verification 未提交、assets 残留、gate_log、.cursor ☐
|
||||
|
||||
### 十二、不做 / 风险接受(133–137)
|
||||
|
||||
133–135. 已否决功能 · 136–137. ADR-011 / global API_KEY 风险接受
|
||||
|
||||
---
|
||||
|
||||
## 常用命令
|
||||
|
||||
```bash
|
||||
cd ~/桌面/Nexus # 或 $NEXUS_ROOT
|
||||
bash scripts/start-dev.sh
|
||||
docker compose up -d mysql redis
|
||||
bash scripts/start-dev.sh # API :8600
|
||||
bash scripts/local_verify.sh
|
||||
cd frontend && npm run dev # :3000/app/
|
||||
|
||||
bash deploy/pre_deploy_check.sh
|
||||
bash scripts/git-push.sh main # push 凭据在 deploy/nexus-1panel.secrets.sh
|
||||
bash deploy/deploy-production.sh # 需用户批准
|
||||
bash scripts/git-push.sh
|
||||
export NEXUS_SSH="azureuser@20.24.218.235"
|
||||
export NEXUS_SSH_KEY="/home/r/下载/Telegram Desktop/nz-admin.pem"
|
||||
bash deploy/deploy-production.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 铁律
|
||||
|
||||
- 单 Agent,改代码必 `docs/changelog/YYYY-MM-DD-*.md` + 部署前 `docs/audit/YYYY-MM-DD-*.md`
|
||||
- 测试由 Agent 执行;用户只做终验
|
||||
- 完美实现:`.cursor/rules/perfect-implementation.mdc`
|
||||
- 单 Agent;禁止虚拟部门
|
||||
- 完美实现:设计→技术文档→代码→测试→changelog→(批准)部署
|
||||
- 测试 Agent 执行;用户只做终验(`docs/reports/2026-06-01-final-acceptance-checklist.md`)
|
||||
|
||||
## 可选后续
|
||||
---
|
||||
|
||||
- 服务器列表「全选当前筛选结果」(非仅当前页)
|
||||
- 脚本执行失败按错误类型分组 / 导出 CSV
|
||||
- 更新本 handoff 为 SSOT 并归档 `AI-HANDOFF-2026-06-03.md` 引用
|
||||
## 变更记录
|
||||
|
||||
| 日期 | 变更 |
|
||||
|------|------|
|
||||
| 2026-06-08 初版 | 3f995a7 摘要 |
|
||||
| 2026-06-08 中 | 更新至 85ecd1f;132 条 Master Backlog |
|
||||
| 2026-06-08 末 | 更新至 a401ea5;记录 server_ids/设置 422 修复;server_batch reconcile 未提交待办 |
|
||||
|
||||
@@ -1463,6 +1463,24 @@ async def install_agent_remote(
|
||||
if not server:
|
||||
raise HTTPException(status_code=404, detail="Server not found")
|
||||
|
||||
from server.application.server_batch_common import agent_install_skip_result
|
||||
from server.utils.agent_version import get_central_agent_version
|
||||
|
||||
skip = await agent_install_skip_result(
|
||||
server=server,
|
||||
server_id=id,
|
||||
server_name=server.name or str(id),
|
||||
central_version=get_central_agent_version(),
|
||||
)
|
||||
if skip is not None:
|
||||
return {
|
||||
"success": True,
|
||||
"server_id": id,
|
||||
"server_name": server.name,
|
||||
"stdout": skip.get("stdout", ""),
|
||||
"skipped": True,
|
||||
}
|
||||
|
||||
# Validate prerequisites
|
||||
base_url = (settings.API_BASE_URL or "").strip().rstrip("/")
|
||||
if not base_url:
|
||||
|
||||
@@ -159,6 +159,75 @@ async def mark_agent_uninstalled(sid: int) -> None:
|
||||
await session.commit()
|
||||
|
||||
|
||||
_REMOTE_AGENT_VERSION_CMD = (
|
||||
"if [ -f /opt/nexus-agent/agent.py ]; then "
|
||||
"grep -m1 '\"agent_version\"' /opt/nexus-agent/agent.py 2>/dev/null | "
|
||||
"sed -n 's/.*\"agent_version\": *\"\\([^\"]*\\)\".*/\\1/p'; fi"
|
||||
)
|
||||
|
||||
|
||||
async def _redis_agent_version(server_id: int) -> str:
|
||||
from server.infrastructure.redis.client import get_redis
|
||||
|
||||
try:
|
||||
redis = get_redis()
|
||||
data = await redis.hgetall(f"heartbeat:{server_id}")
|
||||
if data:
|
||||
return (data.get("agent_version") or "").strip()
|
||||
except Exception as e:
|
||||
logger.warning("Redis agent_version read failed server_id=%s: %s", server_id, e)
|
||||
return ""
|
||||
|
||||
|
||||
async def resolve_remote_agent_version(server: Server, *, ssh_user: str) -> str:
|
||||
"""Best-effort remote version: Redis heartbeat → MySQL → SSH grep agent.py."""
|
||||
from server.infrastructure.ssh.asyncssh_pool import exec_ssh_command
|
||||
|
||||
for candidate in (
|
||||
await _redis_agent_version(server.id),
|
||||
(server.agent_version or "").strip(),
|
||||
):
|
||||
if candidate:
|
||||
return candidate
|
||||
try:
|
||||
cmd = sudo_wrap(_REMOTE_AGENT_VERSION_CMD, ssh_user)
|
||||
r = await exec_ssh_command(server, cmd, timeout=15)
|
||||
if r["exit_code"] == 0:
|
||||
return (r.get("stdout") or "").strip().split("\n")[0].strip()
|
||||
except Exception as e:
|
||||
logger.warning("SSH agent_version probe failed server_id=%s: %s", server.id, e)
|
||||
return ""
|
||||
|
||||
|
||||
async def agent_install_skip_result(
|
||||
*,
|
||||
server: Server,
|
||||
server_id: int,
|
||||
server_name: str,
|
||||
central_version: str,
|
||||
) -> dict | None:
|
||||
"""Return success result dict when install can be skipped; None to proceed with install."""
|
||||
from server.utils.agent_version import is_version_at_least
|
||||
|
||||
has_key = bool((server.agent_api_key or "").strip())
|
||||
db_version = (server.agent_version or "").strip()
|
||||
if not has_key and not db_version:
|
||||
return None
|
||||
|
||||
ssh_user = (server.username or "root").strip()
|
||||
remote_version = await resolve_remote_agent_version(server, ssh_user=ssh_user)
|
||||
if remote_version and is_version_at_least(remote_version, central_version):
|
||||
return result_item_dict(
|
||||
server_id=server_id,
|
||||
server_name=server_name,
|
||||
success=True,
|
||||
stdout=(
|
||||
f"已安装 Agent {remote_version},不低于主站 {central_version},已跳过"
|
||||
),
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
def result_item_dict(
|
||||
*,
|
||||
server_id: int,
|
||||
@@ -177,6 +246,7 @@ def result_item_dict(
|
||||
|
||||
|
||||
__all__ = [
|
||||
"agent_install_skip_result",
|
||||
"batch_server_display_name",
|
||||
"ensure_agent_api_key",
|
||||
"generate_agent_api_key_value",
|
||||
@@ -184,6 +254,7 @@ __all__ = [
|
||||
"mark_agent_uninstalled",
|
||||
"posix_dirname",
|
||||
"prefetch_batch_context",
|
||||
"resolve_remote_agent_version",
|
||||
"result_item_dict",
|
||||
"sudo_wrap",
|
||||
"update_server_target_path",
|
||||
|
||||
@@ -8,6 +8,7 @@ import shlex
|
||||
from typing import Any, Optional
|
||||
|
||||
from server.application.server_batch_common import (
|
||||
agent_install_skip_result,
|
||||
batch_server_display_name,
|
||||
ensure_agent_api_key,
|
||||
install_error_msg,
|
||||
@@ -18,6 +19,7 @@ from server.application.server_batch_common import (
|
||||
sudo_wrap,
|
||||
update_server_target_path,
|
||||
)
|
||||
from server.utils.agent_version import get_central_agent_version
|
||||
from server.config import settings
|
||||
from server.domain.models import AuditLog, Server
|
||||
from server.infrastructure.database.audit_log_repo import AuditLogRepositoryImpl
|
||||
@@ -91,6 +93,106 @@ async def _notify_complete(live: dict[str, Any]) -> None:
|
||||
logger.warning("server batch complete notify failed: %s", e)
|
||||
|
||||
|
||||
async def _reconcile_live_job(
|
||||
live: dict[str, Any],
|
||||
*,
|
||||
error: str,
|
||||
auto_stuck: bool,
|
||||
) -> bool:
|
||||
"""Fill pending servers as failed, finalize, persist, audit, notify. Returns True if reconciled."""
|
||||
if live.get("status") != "running":
|
||||
return False
|
||||
pending = sbs.pending_server_ids(live)
|
||||
if not pending and sbs.progress_stats(live)["remaining"] == 0:
|
||||
sbs.finalize_status(live)
|
||||
await sbs.save_live(live)
|
||||
else:
|
||||
added = sbs.apply_pending_failures(
|
||||
live,
|
||||
server_ids=pending,
|
||||
error=error,
|
||||
auto_stuck=auto_stuck,
|
||||
)
|
||||
if added == 0 and not pending:
|
||||
return False
|
||||
sbs.finalize_status(live)
|
||||
await sbs.save_live(live)
|
||||
await _persist_job_finalize(live)
|
||||
await _write_audit(live)
|
||||
await _notify_complete(live)
|
||||
job_id = int(live["id"])
|
||||
logger.info(
|
||||
"Server batch job reconciled job_id=%s op=%s status=%s auto_stuck=%s",
|
||||
job_id,
|
||||
live.get("op"),
|
||||
live.get("status"),
|
||||
auto_stuck,
|
||||
)
|
||||
return True
|
||||
|
||||
|
||||
async def recover_orphaned_batch_jobs() -> int:
|
||||
"""On API startup: finalize batch jobs left running after process restart."""
|
||||
from server.infrastructure.database.server_batch_job_repo import (
|
||||
ServerBatchJobRepositoryImpl,
|
||||
)
|
||||
|
||||
reconciled_ids: set[int] = set()
|
||||
count = 0
|
||||
|
||||
for live in await sbs.list_running_live_jobs():
|
||||
job_id = int(live["id"])
|
||||
if job_id in reconciled_ids:
|
||||
continue
|
||||
if await _reconcile_live_job(
|
||||
live,
|
||||
error=sbs.INTERRUPTED_MESSAGE,
|
||||
auto_stuck=False,
|
||||
):
|
||||
reconciled_ids.add(job_id)
|
||||
count += 1
|
||||
|
||||
async with AsyncSessionLocal() as session:
|
||||
repo = ServerBatchJobRepositoryImpl(session)
|
||||
for row in await repo.list_running():
|
||||
job_id = int(row.id)
|
||||
if job_id in reconciled_ids:
|
||||
continue
|
||||
live = await sbs.get_live(job_id)
|
||||
if live is None:
|
||||
live = sbs.live_from_mysql_row(row)
|
||||
elif live.get("status") != "running":
|
||||
continue
|
||||
if await _reconcile_live_job(
|
||||
live,
|
||||
error=sbs.INTERRUPTED_MESSAGE,
|
||||
auto_stuck=False,
|
||||
):
|
||||
reconciled_ids.add(job_id)
|
||||
count += 1
|
||||
|
||||
if count:
|
||||
logger.info("Recovered %s orphaned server batch job(s) after startup", count)
|
||||
return count
|
||||
|
||||
|
||||
async def detect_stuck_batch_jobs() -> int:
|
||||
"""Mark running batch jobs with no progress for STUCK_NO_PROGRESS_SECONDS."""
|
||||
count = 0
|
||||
for live in await sbs.list_running_live_jobs():
|
||||
if not sbs.is_stale_running(live):
|
||||
continue
|
||||
if await _reconcile_live_job(
|
||||
live,
|
||||
error=sbs.STUCK_MESSAGE,
|
||||
auto_stuck=True,
|
||||
):
|
||||
count += 1
|
||||
if count:
|
||||
logger.info("Auto-reconciled %s stuck server batch job(s)", count)
|
||||
return count
|
||||
|
||||
|
||||
async def start_batch_job(
|
||||
*,
|
||||
op: str,
|
||||
@@ -234,9 +336,11 @@ async def _run_background(job_id: int) -> None:
|
||||
await sbs.save_live(live)
|
||||
return
|
||||
live = await sbs.get_live(job_id)
|
||||
if live and live.get("status") == "running":
|
||||
sbs.finalize_status(live)
|
||||
await sbs.save_live(live)
|
||||
if live:
|
||||
stats = sbs.progress_stats(live)
|
||||
if live.get("status") == "running" or stats["remaining"] == 0:
|
||||
sbs.finalize_status(live)
|
||||
await sbs.save_live(live)
|
||||
except Exception:
|
||||
logger.exception("Server batch job failed job_id=%s op=%s", job_id, op)
|
||||
live = await sbs.get_live(job_id)
|
||||
@@ -245,6 +349,9 @@ async def _run_background(job_id: int) -> None:
|
||||
await sbs.save_live(live)
|
||||
live = await sbs.get_live(job_id)
|
||||
if live:
|
||||
if live.get("status") == "running":
|
||||
sbs.finalize_status(live)
|
||||
await sbs.save_live(live)
|
||||
await _persist_job_finalize(live)
|
||||
await _write_audit(live)
|
||||
await _notify_complete(live)
|
||||
@@ -425,12 +532,22 @@ async def _run_install_agent(live: dict[str, Any]) -> None:
|
||||
))
|
||||
return
|
||||
|
||||
central_version = get_central_agent_version()
|
||||
|
||||
async def handler(sid: int, server_map: dict[int, Server], labels: dict[int, str], _live) -> dict:
|
||||
server = server_map.get(sid)
|
||||
label = labels[sid]
|
||||
if not server:
|
||||
return result_item_dict(server_id=sid, server_name=label, success=False, error="服务器不存在")
|
||||
try:
|
||||
skip = await agent_install_skip_result(
|
||||
server=server,
|
||||
server_id=sid,
|
||||
server_name=label,
|
||||
central_version=central_version,
|
||||
)
|
||||
if skip is not None:
|
||||
return skip
|
||||
api_key = await ensure_agent_api_key(server)
|
||||
ssh_user = (server.username or "root").strip()
|
||||
install_cmd = (
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
"""Background reconciliation for server batch jobs — stuck detection (60s interval)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
|
||||
from server.application.services.server_batch_service import detect_stuck_batch_jobs
|
||||
from server.infrastructure.redis.server_batch_store import RECONCILE_INTERVAL_SECONDS
|
||||
|
||||
logger = logging.getLogger("nexus.server_batch_reconcile")
|
||||
|
||||
|
||||
async def server_batch_reconcile_loop() -> None:
|
||||
"""Every minute: auto-finalize batch jobs with no Redis progress for 1h."""
|
||||
logger.info(
|
||||
"Server batch reconcile loop started (interval=%ss)",
|
||||
RECONCILE_INTERVAL_SECONDS,
|
||||
)
|
||||
while True:
|
||||
await asyncio.sleep(RECONCILE_INTERVAL_SECONDS)
|
||||
try:
|
||||
stuck = await detect_stuck_batch_jobs()
|
||||
if stuck:
|
||||
logger.info("Server batch reconcile: %s stuck job(s) finalized", stuck)
|
||||
except Exception as e:
|
||||
logger.error("Server batch reconcile loop error: %s", e)
|
||||
@@ -0,0 +1,42 @@
|
||||
"""Periodic purge of sync_logs older than SYNC_LOG_RETENTION_DAYS (design: 30 days)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
|
||||
from server.infrastructure.database.session import AsyncSessionLocal
|
||||
from server.infrastructure.database.sync_log_repo import (
|
||||
SYNC_LOG_RETENTION_DAYS,
|
||||
SyncLogRepositoryImpl,
|
||||
)
|
||||
|
||||
logger = logging.getLogger("nexus.sync_log_purge")
|
||||
|
||||
PURGE_INTERVAL_SECONDS = 24 * 3600
|
||||
|
||||
|
||||
async def purge_stale_sync_logs() -> int:
|
||||
"""Delete sync_logs rows older than retention window. Returns rows removed."""
|
||||
async with AsyncSessionLocal() as session:
|
||||
repo = SyncLogRepositoryImpl(session)
|
||||
return await repo.purge_older_than(retention_days=SYNC_LOG_RETENTION_DAYS)
|
||||
|
||||
|
||||
async def sync_log_purge_loop() -> None:
|
||||
"""Run daily on primary worker; also invoked once at API startup."""
|
||||
while True:
|
||||
try:
|
||||
count = await purge_stale_sync_logs()
|
||||
if count:
|
||||
logger.info(
|
||||
"Sync log purge: removed %s row(s) older than %s days",
|
||||
count,
|
||||
SYNC_LOG_RETENTION_DAYS,
|
||||
)
|
||||
await asyncio.sleep(PURGE_INTERVAL_SECONDS)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception:
|
||||
logger.exception("Sync log purge loop error")
|
||||
await asyncio.sleep(PURGE_INTERVAL_SECONDS)
|
||||
@@ -68,6 +68,12 @@ class ServerBatchJobRepositoryImpl:
|
||||
await self.session.commit()
|
||||
return row
|
||||
|
||||
async def list_running(self) -> list[ServerBatchJob]:
|
||||
result = await self.session.execute(
|
||||
select(ServerBatchJob).where(ServerBatchJob.status == "running")
|
||||
)
|
||||
return list(result.scalars().all())
|
||||
|
||||
async def list_recent(
|
||||
self,
|
||||
*,
|
||||
|
||||
@@ -2,11 +2,14 @@
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import Optional, List, Tuple
|
||||
from sqlalchemy import select, func
|
||||
from sqlalchemy import delete, func, select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from server.domain.models import SyncLog, Server
|
||||
|
||||
SYNC_LOG_RETENTION_DAYS = 30
|
||||
SYNC_LOG_PURGE_BATCH_SIZE = 1000
|
||||
|
||||
_STALE_RUNNING_MESSAGE = (
|
||||
"推送状态异常:长时间未完成(已自动订正,可能因历史缺陷未落库终态)"
|
||||
)
|
||||
@@ -113,6 +116,33 @@ class SyncLogRepositoryImpl:
|
||||
await self.session.commit()
|
||||
return len(stale)
|
||||
|
||||
async def purge_older_than(
|
||||
self,
|
||||
retention_days: int = SYNC_LOG_RETENTION_DAYS,
|
||||
batch_size: int = SYNC_LOG_PURGE_BATCH_SIZE,
|
||||
) -> int:
|
||||
"""Delete sync_logs rows with started_at older than retention_days (batched)."""
|
||||
if retention_days <= 0:
|
||||
return 0
|
||||
cutoff = datetime.now(timezone.utc) - timedelta(days=retention_days)
|
||||
total = 0
|
||||
while True:
|
||||
result = await self.session.execute(
|
||||
select(SyncLog.id)
|
||||
.where(SyncLog.started_at < cutoff)
|
||||
.order_by(SyncLog.id)
|
||||
.limit(batch_size)
|
||||
)
|
||||
ids = [int(row[0]) for row in result.all()]
|
||||
if not ids:
|
||||
break
|
||||
await self.session.execute(delete(SyncLog).where(SyncLog.id.in_(ids)))
|
||||
await self.session.commit()
|
||||
total += len(ids)
|
||||
if len(ids) < batch_size:
|
||||
break
|
||||
return total
|
||||
|
||||
# Allowed fields for update_status — prevents arbitrary field injection via **kwargs
|
||||
SYNC_LOG_UPDATABLE_FIELDS = {"finished_at", "duration_seconds", "error_message", "diff_summary", "status"}
|
||||
|
||||
|
||||
@@ -14,6 +14,10 @@ logger = logging.getLogger("nexus.server_batch_store")
|
||||
REDIS_KEY_PREFIX = "server_batch:"
|
||||
REDIS_NEXT_ID = "server_batch:next_id"
|
||||
REDIS_TTL_SECONDS = 7200
|
||||
RECONCILE_INTERVAL_SECONDS = 60
|
||||
STUCK_NO_PROGRESS_SECONDS = 3600
|
||||
INTERRUPTED_MESSAGE = "服务重启导致任务中断"
|
||||
STUCK_MESSAGE = "执行超时无响应(系统自动标记)"
|
||||
TERMINAL_STATUSES = frozenset({"completed", "failed", "partial", "cancelled"})
|
||||
|
||||
|
||||
@@ -64,6 +68,29 @@ async def save_live(live: dict[str, Any]) -> None:
|
||||
await redis.set(_key(job_id), json.dumps(live, ensure_ascii=False), ex=REDIS_TTL_SECONDS)
|
||||
|
||||
|
||||
async def list_running_live_jobs() -> list[dict[str, Any]]:
|
||||
"""Scan Redis for all server batch jobs still in running status."""
|
||||
redis = get_redis()
|
||||
running: list[dict[str, Any]] = []
|
||||
cursor = 0
|
||||
while True:
|
||||
cursor, keys = await redis.scan(cursor, match=f"{REDIS_KEY_PREFIX}*", count=50)
|
||||
for key in keys:
|
||||
if key == REDIS_NEXT_ID:
|
||||
continue
|
||||
suffix = key.replace(REDIS_KEY_PREFIX, "", 1)
|
||||
try:
|
||||
job_id = int(suffix)
|
||||
except ValueError:
|
||||
continue
|
||||
live = await get_live(job_id)
|
||||
if live and live.get("status") == "running":
|
||||
running.append(live)
|
||||
if cursor == 0:
|
||||
break
|
||||
return running
|
||||
|
||||
|
||||
async def get_live(job_id: int) -> Optional[dict[str, Any]]:
|
||||
redis = get_redis()
|
||||
raw = await redis.get(_key(job_id))
|
||||
@@ -121,6 +148,108 @@ def finalize_status(live: dict[str, Any]) -> str:
|
||||
return status
|
||||
|
||||
|
||||
def parse_iso(ts: str) -> Optional[datetime]:
|
||||
try:
|
||||
return datetime.fromisoformat(ts)
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def pending_server_ids(live: dict[str, Any]) -> list[int]:
|
||||
"""Server IDs in the job with no result row yet."""
|
||||
server_ids = live.get("server_ids") or []
|
||||
results = live.get("results") or {}
|
||||
pending: list[int] = []
|
||||
for sid in server_ids:
|
||||
if str(sid) not in results:
|
||||
pending.append(int(sid))
|
||||
return pending
|
||||
|
||||
|
||||
def apply_pending_failures(
|
||||
live: dict[str, Any],
|
||||
*,
|
||||
server_ids: list[int],
|
||||
error: str,
|
||||
auto_stuck: bool = False,
|
||||
server_name_fallback: str = "未知服务器",
|
||||
) -> int:
|
||||
"""Mark pending servers as failed; returns count of rows added."""
|
||||
if not server_ids:
|
||||
return 0
|
||||
results = live.setdefault("results", {})
|
||||
added = 0
|
||||
for sid in server_ids:
|
||||
key = str(sid)
|
||||
if key in results:
|
||||
continue
|
||||
item: dict[str, Any] = {
|
||||
"server_id": sid,
|
||||
"server_name": server_name_fallback,
|
||||
"success": False,
|
||||
"stdout": "",
|
||||
"error": error,
|
||||
}
|
||||
if auto_stuck:
|
||||
item["auto_stuck"] = True
|
||||
results[key] = item
|
||||
added += 1
|
||||
return added
|
||||
|
||||
|
||||
def is_stale_running(live: dict[str, Any], now: Optional[datetime] = None) -> bool:
|
||||
"""True when a running job has had no Redis update for STUCK_NO_PROGRESS_SECONDS."""
|
||||
if live.get("status") != "running":
|
||||
return False
|
||||
now = now or datetime.now(timezone.utc)
|
||||
updated = parse_iso(live.get("updated_at", ""))
|
||||
if not updated:
|
||||
updated = parse_iso(live.get("created_at", ""))
|
||||
if not updated:
|
||||
return False
|
||||
return (now - updated).total_seconds() >= STUCK_NO_PROGRESS_SECONDS
|
||||
|
||||
|
||||
def live_from_mysql_row(row: Any) -> dict[str, Any]:
|
||||
"""Rebuild minimal Redis-shaped live dict from a ServerBatchJob row."""
|
||||
import json
|
||||
|
||||
try:
|
||||
server_ids = json.loads(row.server_ids or "[]")
|
||||
except json.JSONDecodeError:
|
||||
server_ids = []
|
||||
try:
|
||||
raw_results = json.loads(row.results or "[]")
|
||||
except json.JSONDecodeError:
|
||||
raw_results = []
|
||||
results: dict[str, Any] = {}
|
||||
if isinstance(raw_results, dict):
|
||||
results = dict(raw_results)
|
||||
elif isinstance(raw_results, list):
|
||||
for entry in raw_results:
|
||||
if isinstance(entry, dict) and entry.get("server_id") is not None:
|
||||
results[str(entry["server_id"])] = entry
|
||||
try:
|
||||
params = json.loads(row.params or "{}") if row.params else {}
|
||||
except json.JSONDecodeError:
|
||||
params = {}
|
||||
if not isinstance(params, dict):
|
||||
params = {}
|
||||
started = row.started_at.isoformat() if row.started_at else _now_iso()
|
||||
return {
|
||||
"id": int(row.id),
|
||||
"op": row.op or "",
|
||||
"label": row.label or "",
|
||||
"status": row.status or "running",
|
||||
"server_ids": list(server_ids),
|
||||
"results": results,
|
||||
"operator": row.operator or "",
|
||||
"params": params,
|
||||
"created_at": started,
|
||||
"updated_at": started,
|
||||
}
|
||||
|
||||
|
||||
def live_to_api(live: dict[str, Any]) -> dict[str, Any]:
|
||||
stats = progress_stats(live)
|
||||
results = live.get("results") or {}
|
||||
|
||||
+18
-2
@@ -80,6 +80,7 @@ from server.background.schedule_runner import schedule_runner_loop
|
||||
from server.background.retry_runner import retry_runner_loop
|
||||
from server.background.ip_allowlist_refresh import ip_allowlist_refresh_loop
|
||||
from server.background.upload_staging_cleanup import upload_staging_cleanup_loop
|
||||
from server.background.sync_log_purge import sync_log_purge_loop
|
||||
|
||||
logger = logging.getLogger("nexus")
|
||||
|
||||
@@ -286,6 +287,13 @@ async def lifespan(app: FastAPI):
|
||||
# Use Redis-based leader election to ensure only one worker runs background tasks.
|
||||
is_primary = await _acquire_primary_lock()
|
||||
if is_primary:
|
||||
from server.application.services.server_batch_service import recover_orphaned_batch_jobs
|
||||
from server.background.server_batch_reconcile import server_batch_reconcile_loop
|
||||
|
||||
recovered = await recover_orphaned_batch_jobs()
|
||||
if recovered:
|
||||
logger.info("Startup: recovered %s orphaned server batch job(s)", recovered)
|
||||
|
||||
task_flush = asyncio.create_task(heartbeat_flush_loop(), name="heartbeat_flush")
|
||||
task_script_flush = asyncio.create_task(
|
||||
script_execution_flush_loop(), name="script_execution_flush"
|
||||
@@ -297,6 +305,12 @@ async def lifespan(app: FastAPI):
|
||||
task_upload_cleanup = asyncio.create_task(
|
||||
upload_staging_cleanup_loop(), name="upload_staging_cleanup"
|
||||
)
|
||||
task_batch_reconcile = asyncio.create_task(
|
||||
server_batch_reconcile_loop(), name="server_batch_reconcile"
|
||||
)
|
||||
task_sync_log_purge = asyncio.create_task(
|
||||
sync_log_purge_loop(), name="sync_log_purge"
|
||||
)
|
||||
_background_tasks.extend([
|
||||
task_flush,
|
||||
task_script_flush,
|
||||
@@ -305,6 +319,8 @@ async def lifespan(app: FastAPI):
|
||||
task_retry,
|
||||
task_ip_refresh,
|
||||
task_upload_cleanup,
|
||||
task_batch_reconcile,
|
||||
task_sync_log_purge,
|
||||
])
|
||||
logger.info("Primary worker — background tasks launched")
|
||||
else:
|
||||
@@ -317,8 +333,8 @@ async def lifespan(app: FastAPI):
|
||||
logger.info(
|
||||
f"{settings.SYSTEM_NAME} v6.0.0 started — "
|
||||
f"background tasks: heartbeat_flush(10min), script_execution_flush(60s), "
|
||||
f"self_monitor(30s), schedule_runner(60s), retry_runner(5min), "
|
||||
f"upload_staging_cleanup(1h)"
|
||||
f"server_batch_reconcile(60s), sync_log_purge(24h), self_monitor(30s), "
|
||||
f"schedule_runner(60s), retry_runner(5min), upload_staging_cleanup(1h)"
|
||||
)
|
||||
|
||||
yield
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
"""Central Nexus Agent version (SSOT) and semver-style comparison."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
|
||||
_AGENT_PY = Path(__file__).resolve().parents[2] / "web" / "agent" / "agent.py"
|
||||
_VERSION_IN_AGENT_RE = re.compile(r'"agent_version":\s*"([^"]+)"')
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def get_central_agent_version() -> str:
|
||||
"""Read agent_version string shipped at /agent/agent.py (heartbeat payload)."""
|
||||
text = _AGENT_PY.read_text(encoding="utf-8")
|
||||
match = _VERSION_IN_AGENT_RE.search(text)
|
||||
if not match:
|
||||
raise RuntimeError(f"agent_version not found in {_AGENT_PY}")
|
||||
return match.group(1)
|
||||
|
||||
|
||||
def version_tuple(version: str) -> tuple[int, ...]:
|
||||
"""Parse dotted numeric prefix, e.g. 2.0.0 -> (2, 0, 0)."""
|
||||
version = (version or "").strip()
|
||||
if not version:
|
||||
return ()
|
||||
parts: list[int] = []
|
||||
for segment in re.split(r"[.-]", version):
|
||||
if segment.isdigit():
|
||||
parts.append(int(segment))
|
||||
else:
|
||||
break
|
||||
return tuple(parts)
|
||||
|
||||
|
||||
def is_version_lower(remote: str, central: str) -> bool:
|
||||
"""True when remote is strictly older than central."""
|
||||
remote_t = version_tuple(remote)
|
||||
central_t = version_tuple(central)
|
||||
if not remote_t:
|
||||
return True
|
||||
if not central_t:
|
||||
return False
|
||||
return remote_t < central_t
|
||||
|
||||
|
||||
def is_version_at_least(remote: str, central: str) -> bool:
|
||||
"""True when remote >= central (equal counts as skip)."""
|
||||
remote_t = version_tuple(remote)
|
||||
central_t = version_tuple(central)
|
||||
if not remote_t:
|
||||
return False
|
||||
if not central_t:
|
||||
return bool(remote_t)
|
||||
return remote_t >= central_t
|
||||
@@ -0,0 +1,84 @@
|
||||
"""Agent version comparison and install-skip decision."""
|
||||
|
||||
from unittest.mock import AsyncMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
from server.application.server_batch_common import agent_install_skip_result
|
||||
from server.domain.models import Server
|
||||
from server.utils.agent_version import (
|
||||
get_central_agent_version,
|
||||
is_version_at_least,
|
||||
is_version_lower,
|
||||
version_tuple,
|
||||
)
|
||||
|
||||
|
||||
def test_version_tuple():
|
||||
assert version_tuple("2.0.0") == (2, 0, 0)
|
||||
assert version_tuple("1.10.3") == (1, 10, 3)
|
||||
assert version_tuple("") == ()
|
||||
|
||||
|
||||
def test_is_version_lower():
|
||||
assert is_version_lower("1.9.0", "2.0.0") is True
|
||||
assert is_version_lower("2.0.0", "2.0.0") is False
|
||||
assert is_version_lower("2.1.0", "2.0.0") is False
|
||||
|
||||
|
||||
def test_is_version_at_least():
|
||||
assert is_version_at_least("2.0.0", "2.0.0") is True
|
||||
assert is_version_at_least("2.0.1", "2.0.0") is True
|
||||
assert is_version_at_least("1.9.9", "2.0.0") is False
|
||||
|
||||
|
||||
def test_central_agent_version_matches_agent_py():
|
||||
assert get_central_agent_version() == "2.0.0"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_agent_install_skip_when_version_current():
|
||||
server = Server(id=1, name="A", agent_api_key="nxs-k", agent_version="2.0.0")
|
||||
with patch(
|
||||
"server.application.server_batch_common.resolve_remote_agent_version",
|
||||
new_callable=AsyncMock,
|
||||
return_value="2.0.0",
|
||||
):
|
||||
result = await agent_install_skip_result(
|
||||
server=server,
|
||||
server_id=1,
|
||||
server_name="A",
|
||||
central_version="2.0.0",
|
||||
)
|
||||
assert result is not None
|
||||
assert result["success"] is True
|
||||
assert "已跳过" in result["stdout"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_agent_install_no_skip_when_version_lower():
|
||||
server = Server(id=2, name="B", agent_api_key="nxs-k", agent_version="1.0.0")
|
||||
with patch(
|
||||
"server.application.server_batch_common.resolve_remote_agent_version",
|
||||
new_callable=AsyncMock,
|
||||
return_value="1.0.0",
|
||||
):
|
||||
result = await agent_install_skip_result(
|
||||
server=server,
|
||||
server_id=2,
|
||||
server_name="B",
|
||||
central_version="2.0.0",
|
||||
)
|
||||
assert result is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_agent_install_no_skip_when_never_installed():
|
||||
server = Server(id=3, name="C", agent_api_key=None, agent_version=None)
|
||||
result = await agent_install_skip_result(
|
||||
server=server,
|
||||
server_id=3,
|
||||
server_name="C",
|
||||
central_version="2.0.0",
|
||||
)
|
||||
assert result is None
|
||||
@@ -0,0 +1,81 @@
|
||||
"""Server batch zombie recovery and stuck detection helpers."""
|
||||
|
||||
import json
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from server.domain.models import ServerBatchJob
|
||||
from server.infrastructure.database.server_batch_job_repo import job_row_to_api
|
||||
from server.infrastructure.redis import server_batch_store as sbs
|
||||
|
||||
|
||||
def test_pending_server_ids():
|
||||
live = {
|
||||
"server_ids": [1, 2, 3],
|
||||
"results": {"1": {"server_id": 1, "success": True}},
|
||||
}
|
||||
assert sbs.pending_server_ids(live) == [2, 3]
|
||||
|
||||
|
||||
def test_apply_pending_failures_marks_stuck():
|
||||
live = {"server_ids": [10], "results": {}}
|
||||
added = sbs.apply_pending_failures(
|
||||
live,
|
||||
server_ids=[10],
|
||||
error=sbs.STUCK_MESSAGE,
|
||||
auto_stuck=True,
|
||||
server_name_fallback="S10",
|
||||
)
|
||||
assert added == 1
|
||||
entry = live["results"]["10"]
|
||||
assert entry["success"] is False
|
||||
assert entry["auto_stuck"] is True
|
||||
assert entry["error"] == sbs.STUCK_MESSAGE
|
||||
|
||||
|
||||
def test_finalize_status_after_pending_failures():
|
||||
live = {
|
||||
"server_ids": [1, 2],
|
||||
"results": {},
|
||||
"status": "running",
|
||||
}
|
||||
sbs.apply_pending_failures(
|
||||
live,
|
||||
server_ids=[1, 2],
|
||||
error=sbs.INTERRUPTED_MESSAGE,
|
||||
)
|
||||
status = sbs.finalize_status(live)
|
||||
assert status == "failed"
|
||||
assert live["status"] == "failed"
|
||||
|
||||
|
||||
def test_is_stale_running_true_when_old_update():
|
||||
old = (datetime.now(timezone.utc) - timedelta(hours=2)).isoformat()
|
||||
live = {"status": "running", "updated_at": old}
|
||||
assert sbs.is_stale_running(live) is True
|
||||
|
||||
|
||||
def test_is_stale_running_false_when_recent():
|
||||
recent = datetime.now(timezone.utc).isoformat()
|
||||
live = {"status": "running", "updated_at": recent}
|
||||
assert sbs.is_stale_running(live) is False
|
||||
|
||||
|
||||
def test_live_from_mysql_row_rebuilds_results_map():
|
||||
row = ServerBatchJob(
|
||||
id=20,
|
||||
op="install-agent",
|
||||
label="批量安装 Agent",
|
||||
server_ids=json.dumps([290, 346]),
|
||||
status="running",
|
||||
results=json.dumps([
|
||||
{"server_id": 290, "server_name": "A", "success": True, "stdout": "ok", "error": ""},
|
||||
]),
|
||||
operator="admin",
|
||||
started_at=datetime(2026, 6, 8, 10, 0, tzinfo=timezone.utc),
|
||||
)
|
||||
live = sbs.live_from_mysql_row(row)
|
||||
assert live["id"] == 20
|
||||
assert live["status"] == "running"
|
||||
assert sbs.pending_server_ids(live) == [346]
|
||||
api = job_row_to_api(row)
|
||||
assert api["remaining"] == 1
|
||||
@@ -0,0 +1,18 @@
|
||||
"""sync_logs retention purge helpers."""
|
||||
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
from server.infrastructure.database.sync_log_repo import (
|
||||
SYNC_LOG_PURGE_BATCH_SIZE,
|
||||
SYNC_LOG_RETENTION_DAYS,
|
||||
)
|
||||
|
||||
|
||||
def test_sync_log_retention_constants():
|
||||
assert SYNC_LOG_RETENTION_DAYS == 30
|
||||
assert SYNC_LOG_PURGE_BATCH_SIZE >= 100
|
||||
|
||||
|
||||
def test_purge_cutoff_is_30_days_ago():
|
||||
cutoff = datetime.now(timezone.utc) - timedelta(days=SYNC_LOG_RETENTION_DAYS)
|
||||
assert (datetime.now(timezone.utc) - cutoff).days == SYNC_LOG_RETENTION_DAYS
|
||||
Reference in New Issue
Block a user