feat(watch): 监测槽开关、8h/24h、到期暂停与探针修复
支持随时暂停/恢复实时监测并保留槽位;TTL 到期自动暂停不占删槽;修复到期竞态导致空槽与唯一约束冲突;探针 parse_error 与中文错误;移除失败 snackbar。
This commit is contained in:
@@ -0,0 +1,117 @@
|
|||||||
|
# 审计 — 监测槽开关 / 8h·24h / 暂停占槽 / 探针中文错误
|
||||||
|
|
||||||
|
**日期**:2026-06-11
|
||||||
|
**Changelog**:
|
||||||
|
- `docs/changelog/2026-06-11-watch-slot-monitoring-toggle.md`
|
||||||
|
- `docs/changelog/2026-06-11-watch-probe-parse-error-fix.md`
|
||||||
|
|
||||||
|
## 范围(未提交工作区,+799 / −124 行)
|
||||||
|
|
||||||
|
| 层级 | 文件 |
|
||||||
|
|------|------|
|
||||||
|
| 模型/迁移 | `server/domain/models/__init__.py`(`monitoring_enabled`、`ttl_hours`) |
|
||||||
|
| | `server/infrastructure/database/migrations.py` |
|
||||||
|
| Repository | `server/infrastructure/database/watch_repo.py` |
|
||||||
|
| Service | `server/application/services/watch_service.py` |
|
||||||
|
| API | `server/api/watch.py`(PATCH monitoring / ttl) |
|
||||||
|
| 探针 | `server/background/watch_probe_runner.py` |
|
||||||
|
| | `server/infrastructure/ssh/remote_probe.py` |
|
||||||
|
| 工具 | `server/utils/watch_metrics.py` |
|
||||||
|
| | `server/utils/watch_state.py`(**新增** Redis 清理) |
|
||||||
|
| | `server/utils/watch_probe_errors.py`(**新增** 中文错误) |
|
||||||
|
| 前端 | `frontend/src/composables/useWatchPins.ts` |
|
||||||
|
| | `frontend/src/components/watch/WatchSlotCard.vue` |
|
||||||
|
| | `frontend/src/components/watch/WatchSlotRow.vue` |
|
||||||
|
| | `frontend/src/components/watch/WatchProbeRecordsTable.vue` |
|
||||||
|
| | `frontend/src/utils/watchFormat.ts` |
|
||||||
|
| | `frontend/src/api/index.ts`(`http.patch`) |
|
||||||
|
| 测试 | `tests/test_watch_pins.py` |
|
||||||
|
| | `tests/test_watch_metrics.py` |
|
||||||
|
| | `tests/test_watch_probe_errors.py`(**新增**) |
|
||||||
|
|
||||||
|
## 功能摘要
|
||||||
|
|
||||||
|
1. **实时监测开关**:`monitoring_enabled`;关则停 5s 探针,槽位保留
|
||||||
|
2. **8h / 24h**:`ttl_hours` + `PATCH /pins/{slot}/ttl`;默认 8h,显式切 24h
|
||||||
|
3. **暂停永久占槽**:`monitoring_enabled=false` 时不参与 TTL 过期删除
|
||||||
|
4. **时长到期**:自动暂停(等同关开关),结束 session,**不删 pin**
|
||||||
|
5. **恢复 Agent 60s**:无活跃监测时清理 `watch:live/proc/last`;Agent 心跳逻辑未改
|
||||||
|
6. **parse_error 修复**:Agent 有 CPU/内存时 SSH 失败不拖垮整轮;SSH 脚本加固
|
||||||
|
7. **错误中文**:后端写入 + 前端 `formatProbeError` 兼容历史英文
|
||||||
|
|
||||||
|
## Step 3 规则扫描
|
||||||
|
|
||||||
|
| H | 规则 | 结论 |
|
||||||
|
|---|------|------|
|
||||||
|
| H1 | 鉴权 / IDOR | **PASS** — 新 PATCH 均 `get_current_admin`;repo 带 `admin_id` |
|
||||||
|
| H2 | 无静默吞错 | **PASS** — 探针失败仍落库;Redis 清理失败会抛错(生产有 Redis) |
|
||||||
|
| H3 | 无新密钥 | **PASS** |
|
||||||
|
| H4 | CUD 审计 | **PASS** — `watch_pin_pause` / `watch_pin_resume` / `watch_pin_ttl` / remove |
|
||||||
|
| H5 | 输入边界 | **PASS** — `Literal[8,24]`、`slot_index` 0–3 |
|
||||||
|
| H6 | 迁移 | **PASS** — ALTER + duplicate column 容错 |
|
||||||
|
| H7 | 产品风险 | **RISK 接受** — 4 槽均可永久暂停占用,需手动 X 释放(用户明确要求) |
|
||||||
|
|
||||||
|
## 逐模块走读
|
||||||
|
|
||||||
|
### 后端
|
||||||
|
|
||||||
|
| 模块 | 结论 | 备注 |
|
||||||
|
|------|------|------|
|
||||||
|
| `watch_repo._pin_still_occupied` | PASS | 暂停不占 TTL;开启监测才看过期 |
|
||||||
|
| `expire_due_pins` | PASS | 改为暂停 + end session,不 delete pin |
|
||||||
|
| `set_monitoring_enabled` + `_ensure_open_session` | PASS | TTL 到期后再开自动新 session |
|
||||||
|
| `_restore_idle_agent_watch` | PASS | 全局无监测才清 Redis |
|
||||||
|
| `replace_slot` | PASS | 保留槽位原 `ttl_hours`(审查项已修) |
|
||||||
|
| `probe_server_metrics` | PASS | Agent 基础指标优先;SSH 仅补 IO |
|
||||||
|
| `watch_probe_error_zh` | PASS | 与前端映射一致 |
|
||||||
|
|
||||||
|
### 前端
|
||||||
|
|
||||||
|
| 模块 | 结论 | 备注 |
|
||||||
|
|------|------|------|
|
||||||
|
| `WatchSlotCard` | PASS | 开关 + 8h/24h + 暂停态 UI |
|
||||||
|
| `tickCountdown` | PASS | 归零 `refreshPins`,不再误置 `empty` |
|
||||||
|
| `formatProbeError` | PASS | 状态/错误/来源中文化 |
|
||||||
|
| `WatchProbeRecordsTable` | PASS | 筛选器与列中文 |
|
||||||
|
|
||||||
|
## 审计中发现并修复
|
||||||
|
|
||||||
|
| 项 | 严重度 | 处理 |
|
||||||
|
|----|--------|------|
|
||||||
|
| `mock_watch_redis` 未 patch `watch_state.get_redis` | **P1** | 已修:`remove_slot` / `set_slot_monitoring` 单测 2 失败 → 19 passed |
|
||||||
|
|
||||||
|
## 遗留建议(非阻塞)
|
||||||
|
|
||||||
|
| 项 | 说明 |
|
||||||
|
|----|------|
|
||||||
|
| 设计文档 | 开关/暂停语义变更较大,建议补 `docs/design/specs/` 短文(当前仅 changelog) |
|
||||||
|
| HTTP 集成测试 | PATCH monitoring/ttl 的 404/422 路径未覆盖 |
|
||||||
|
| TTL 到期审计 | 后台 `expire_due_pins` 无 audit_log(可接受,session.end_reason=expired 可查) |
|
||||||
|
| PATCH 404 语义 | 「槽位无效」与「槽位为空」均 404,与 POST 422 不一致 |
|
||||||
|
|
||||||
|
## Closure
|
||||||
|
|
||||||
|
| H | 判定 | 依据 |
|
||||||
|
|---|------|------|
|
||||||
|
| H1–H6 | **PASS** | 鉴权、审计、迁移、探针落库均符合铁律 |
|
||||||
|
| H7 | **RISK 接受** | 用户要求暂停永久占槽 |
|
||||||
|
|
||||||
|
## DoD
|
||||||
|
|
||||||
|
- [x] `.venv/bin/pytest tests/test_watch_metrics.py tests/test_watch_pins.py tests/test_watch_probe_errors.py` — **19 passed**
|
||||||
|
- [x] `cd frontend && npm run type-check`
|
||||||
|
- [x] 审计修复:test fixture patch `watch_state.get_redis`
|
||||||
|
- [ ] `bash scripts/local_verify.sh` / 门控 7 道(部署前)
|
||||||
|
- [ ] 生产浏览器:开关暂停、24h 切换、到期自动暂停、中文错误展示
|
||||||
|
|
||||||
|
## 验证命令
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/pytest tests/test_watch_metrics.py tests/test_watch_pins.py tests/test_watch_probe_errors.py -q
|
||||||
|
cd frontend && npm run type-check
|
||||||
|
bash deploy/pre_deploy_check.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
## 合并建议
|
||||||
|
|
||||||
|
**Approve with notes** — 功能完整、测试绿、安全与审计达标;部署前跑门控 + 浏览器终验。遗留 HTTP 集成测试与设计文档可跟进 PR。
|
||||||
@@ -0,0 +1,67 @@
|
|||||||
|
# 审计 — 监测槽开关 / TTL / 到期竞态 / 探针中文 / 移除提示
|
||||||
|
|
||||||
|
**日期**:2026-06-12
|
||||||
|
**Changelog**:
|
||||||
|
- `docs/changelog/2026-06-11-watch-slot-monitoring-toggle.md`
|
||||||
|
- `docs/changelog/2026-06-11-watch-probe-parse-error-fix.md`
|
||||||
|
- `docs/changelog/2026-06-12-watch-ttl-expiry-race-fix.md`
|
||||||
|
|
||||||
|
## 范围(文件清单)
|
||||||
|
|
||||||
|
| 层级 | 文件 |
|
||||||
|
|------|------|
|
||||||
|
| 模型/迁移 | `server/domain/models/__init__.py` |
|
||||||
|
| | `server/infrastructure/database/migrations.py` |
|
||||||
|
| Repository | `server/infrastructure/database/watch_repo.py` |
|
||||||
|
| Service | `server/application/services/watch_service.py` |
|
||||||
|
| API | `server/api/watch.py` |
|
||||||
|
| 探针 | `server/background/watch_probe_runner.py` |
|
||||||
|
| | `server/infrastructure/ssh/remote_probe.py` |
|
||||||
|
| 工具 | `server/utils/watch_metrics.py` |
|
||||||
|
| | `server/utils/watch_state.py` |
|
||||||
|
| | `server/utils/watch_probe_errors.py` |
|
||||||
|
| 前端 | `frontend/src/api/index.ts` |
|
||||||
|
| | `frontend/src/composables/useWatchPins.ts` |
|
||||||
|
| | `frontend/src/components/watch/WatchSlotCard.vue` |
|
||||||
|
| | `frontend/src/components/watch/WatchSlotRow.vue` |
|
||||||
|
| | `frontend/src/components/watch/WatchProbeRecordsTable.vue` |
|
||||||
|
| | `frontend/src/utils/watchFormat.ts` |
|
||||||
|
| 测试 | `tests/test_watch_pins.py` |
|
||||||
|
| | `tests/test_watch_metrics.py` |
|
||||||
|
| | `tests/test_watch_probe_errors.py` |
|
||||||
|
|
||||||
|
## Step 3 规则扫描
|
||||||
|
|
||||||
|
| H | 规则 | 结论 |
|
||||||
|
|---|------|------|
|
||||||
|
| H1 | 鉴权 / IDOR | PASS — PATCH/DELETE 均 `get_current_admin` + admin_id 隔离 |
|
||||||
|
| H2 | 无静默吞错 | PASS — 探针失败落库;前端 remove/monitoring/ttl 均有 snackbar |
|
||||||
|
| H3 | 无新密钥 | PASS |
|
||||||
|
| H4 | CUD 审计 | PASS — pause/resume/ttl/remove |
|
||||||
|
| H5 | 输入边界 | PASS — `Literal[8,24]`、slot 0–3 |
|
||||||
|
| H6 | 迁移 | PASS — monitoring_enabled + ttl_hours ALTER |
|
||||||
|
| H7 | TTL 竞态 | PASS — `_finalize_due_pins` 读/写路径同步到期 |
|
||||||
|
|
||||||
|
## Closure
|
||||||
|
|
||||||
|
| H | 判定 | 依据 |
|
||||||
|
|---|------|------|
|
||||||
|
| H1–H6 | PASS | 鉴权、审计、迁移、探针落库 |
|
||||||
|
| H7 | PASS | 到期 ghost pin 不再显示空槽 |
|
||||||
|
|
||||||
|
## DoD
|
||||||
|
|
||||||
|
- [x] `pytest tests/test_watch_*.py` — 21 passed
|
||||||
|
- [x] `npm run type-check`
|
||||||
|
- [x] P0 TTL 竞态修复 + P2 onRemove snackbar
|
||||||
|
- [ ] `bash deploy/pre_deploy_check.sh`
|
||||||
|
- [ ] 生产浏览器终验
|
||||||
|
|
||||||
|
## 验证命令
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/pytest tests/test_watch_*.py -q
|
||||||
|
cd frontend && npm run type-check
|
||||||
|
bash scripts/local_verify.sh
|
||||||
|
bash deploy/pre_deploy_check.sh
|
||||||
|
```
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
# 监测槽 parse_error 修复
|
||||||
|
|
||||||
|
**日期**: 2026-06-11
|
||||||
|
|
||||||
|
## 摘要
|
||||||
|
|
||||||
|
修复监测槽显示 `parse_error` 的常见路径:Agent 有心跳但缺 IO 时 SSH 补采失败导致整轮探针失败。
|
||||||
|
|
||||||
|
## 根因
|
||||||
|
|
||||||
|
- `redis_sample_has_io` 为 false 时强制走 SSH;SSH JSON 解析失败且无完整 Redis IO 时返回 `parse_error`
|
||||||
|
- 已装 Agent 的机器通常有 CPU/内存心跳,不应因 SSH 失败而整轮失败
|
||||||
|
- SSH 脚本在 `getloadavg`/`net_io_counters` 异常时可能无 JSON 输出
|
||||||
|
|
||||||
|
## 修复
|
||||||
|
|
||||||
|
- Agent 有 CPU/内存时优先用 Redis;SSH 仅补充 IO,SSH 失败仍显示 Agent 指标
|
||||||
|
- SSH 探针脚本增加 try/except;JSON 解析支持行内 `{` 前缀噪声
|
||||||
|
- 前端 `parse_error` 显示为「探针解析失败」,并展示 `metrics.error` 详情
|
||||||
|
|
||||||
|
## 2026-06-11 补充:错误中文提示
|
||||||
|
|
||||||
|
- 新增 `server/utils/watch_probe_errors.py`,新探针失败写入中文 `error`
|
||||||
|
- 前端 `formatProbeError` / `probeStatusLabel` / `probeSourceLabel` 统一中文(含历史英文记录)
|
||||||
|
- 监测历史探针记录表:状态筛选与错误列中文化
|
||||||
|
|
||||||
|
## 涉及文件
|
||||||
|
|
||||||
|
- `server/background/watch_probe_runner.py`
|
||||||
|
- `server/utils/watch_metrics.py`
|
||||||
|
- `server/infrastructure/ssh/remote_probe.py`
|
||||||
|
- `frontend/src/utils/watchFormat.ts`
|
||||||
|
- `frontend/src/components/watch/WatchSlotCard.vue`
|
||||||
|
- `tests/test_watch_metrics.py`
|
||||||
|
|
||||||
|
## 验证
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest tests/test_watch_metrics.py -q
|
||||||
|
cd frontend && npm run type-check
|
||||||
|
```
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
# 监测槽实时监测开关
|
||||||
|
|
||||||
|
**日期**: 2026-06-11
|
||||||
|
|
||||||
|
## 摘要
|
||||||
|
|
||||||
|
为每台已固定监测槽的服务器增加「实时监测」开关。关闭后停止 5 秒探针与 WebSocket 推送,槽位仍保留同一台服务器,8 小时倒计时暂停;重新开启时重置 8 小时窗口。
|
||||||
|
|
||||||
|
## 动机
|
||||||
|
|
||||||
|
用户希望随时关闭某台服务器的实时监测,而不必等到 8 小时到期或点击删除释放槽位。
|
||||||
|
|
||||||
|
## 涉及文件
|
||||||
|
|
||||||
|
- `server/domain/models/__init__.py` — `WatchPin.monitoring_enabled`
|
||||||
|
- `server/infrastructure/database/migrations.py` — 列迁移
|
||||||
|
- `server/infrastructure/database/watch_repo.py` — 暂停/恢复、过期逻辑
|
||||||
|
- `server/application/services/watch_service.py` — `set_slot_monitoring`、槽位 payload
|
||||||
|
- `server/api/watch.py` — `PATCH /api/watch/pins/{slot_index}/monitoring`
|
||||||
|
- `server/background/watch_probe_runner.py` — 仅探针已开启的 pin
|
||||||
|
- `frontend/src/api/index.ts` — `http.patch`
|
||||||
|
- `frontend/src/composables/useWatchPins.ts`
|
||||||
|
- `frontend/src/components/watch/WatchSlotCard.vue`
|
||||||
|
- `frontend/src/components/watch/WatchSlotRow.vue`
|
||||||
|
- `tests/test_watch_pins.py`
|
||||||
|
|
||||||
|
## 迁移 / 重启
|
||||||
|
|
||||||
|
- 需 API 重启以执行 schema migration(`monitoring_enabled` 列)
|
||||||
|
- 无需手动 SQL
|
||||||
|
|
||||||
|
## 验证
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest tests/test_watch_pins.py -q
|
||||||
|
cd frontend && npm run type-check
|
||||||
|
```
|
||||||
|
|
||||||
|
- 仪表盘监测槽:开关关闭 → 卡片灰显、无实时指标;重新开启 → 恢复探针并重置倒计时
|
||||||
|
- 时长切换:默认 8h;点击 24h → 重置为 24 小时窗口;暂停时切换时长保留偏好,开启后生效
|
||||||
|
|
||||||
|
## 2026-06-11 补充:暂停永久占槽 + 恢复 Agent 60s
|
||||||
|
|
||||||
|
- **暂停/时长到期**:不再删除 pin;`monitoring_enabled=false`,槽位永久保留直至手动移除或重新开启
|
||||||
|
- **8h/24h 到期**:自动暂停(等同关开关),结束 session(`expired`),保留槽位
|
||||||
|
- **恢复 Agent 模式**:当无任何槽位对该机开启 5s 监测时,清理 `watch:live/proc/last` Redis;子机 Agent 始终 60s 智能心跳(CPU 波动 >10% 或超阈值才上报),无需改 Agent 配置
|
||||||
|
- **重新开启**:若 session 已结束则自动开新 session
|
||||||
|
- **涉及**:`server/utils/watch_state.py`、`watch_repo.expire_due_pins`、`watch_service`、`watch_probe_runner`、前端 countdown 到期 refresh
|
||||||
|
|
||||||
|
## 2026-06-11 补充:8h / 24h 时长选项
|
||||||
|
|
||||||
|
- `watch_pins.ttl_hours` 列(默认 8,仅允许 8 或 24)
|
||||||
|
- `PATCH /api/watch/pins/{slot_index}/ttl` — 切换时长;监测开启时同步重置 `expires_at`
|
||||||
|
- `POST /api/watch/pins` 可选 `ttl_hours: 8 | 24`(默认 8)
|
||||||
|
- 槽位卡片「8h / 24h」切换按钮,须显式点击才改为 24h
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
# Changelog — 监测槽 TTL 到期竞态修复
|
||||||
|
|
||||||
|
**日期**:2026-06-12
|
||||||
|
|
||||||
|
## 摘要
|
||||||
|
|
||||||
|
修复 TTL 到期与后台 `expire_due_pins`(5s 探针循环)之间的竞态窗口:槽位短暂显示为空、无法添加/移除、可能触发唯一约束冲突。
|
||||||
|
|
||||||
|
## 动机
|
||||||
|
|
||||||
|
`list_active_pins_for_admin` 在 `monitoring_enabled=true` 且 `expires_at<=now` 时排除 pin,但 DB 行仍在且 `monitoring_enabled` 仍为 true,直到探针循环执行 `expire_due_pins`。此窗口内 UI 显示空槽,但 `create_pin` 会因 `uq_watch_pins_admin_slot` 失败。
|
||||||
|
|
||||||
|
## 变更
|
||||||
|
|
||||||
|
- `WatchService._finalize_due_pins()`:在读/写路径同步到期 pin(暂停 + end session + 按需清 Redis)
|
||||||
|
- 在 `list_slots`、`pin_server`、`replace_slot`、`remove_slot`、`set_slot_monitoring`、`set_slot_ttl` 入口调用
|
||||||
|
- 新增单测:`test_watch_pin_ttl_expiry_list_syncs_before_display`、`test_watch_pin_ttl_expiry_pin_server_no_duplicate`
|
||||||
|
- **P2**:`WatchSlotRow.onRemove` 失败时 snackbar 提示
|
||||||
|
|
||||||
|
## 涉及文件
|
||||||
|
|
||||||
|
- `server/application/services/watch_service.py`
|
||||||
|
- `frontend/src/components/watch/WatchSlotRow.vue`
|
||||||
|
- `tests/test_watch_pins.py`
|
||||||
|
|
||||||
|
## 迁移 / 重启
|
||||||
|
|
||||||
|
- 无需迁移
|
||||||
|
- 需重启 API
|
||||||
|
|
||||||
|
## 验证
|
||||||
|
|
||||||
|
```bash
|
||||||
|
.venv/bin/pytest tests/test_watch_pins.py -q
|
||||||
|
```
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
# Bug 巡查 — 监测槽增强(2026-06-12)
|
||||||
|
|
||||||
|
**范围**:工作区未提交的 watch 开关 / TTL / 探针修复
|
||||||
|
**方法**:代码走读 + 单测 + 竞态分析
|
||||||
|
|
||||||
|
## 发现与处理
|
||||||
|
|
||||||
|
| # | 严重度 | 问题 | 状态 |
|
||||||
|
|---|--------|------|------|
|
||||||
|
| 1 | **P0** | TTL 到期后 ~5s 内槽位显示「空」,但 DB ghost pin 仍在;添加服务器 → 唯一约束冲突;点 X 移除 → 404「槽位为空」 | **已修** — API 读/写路径同步 `_finalize_due_pins` |
|
||||||
|
| 2 | P2 | `onRemove` 无 try/catch,API 失败时未提示 | 未修(低概率,可跟进) |
|
||||||
|
| 3 | P3 | 暂停态仍可通过 `get_processes` 拉取进程(数据可能过期) | 设计可接受 |
|
||||||
|
| 4 | — | Gate 7 被并行 Agent 提示改动阻塞 | 非 watch bug |
|
||||||
|
|
||||||
|
## P0 根因
|
||||||
|
|
||||||
|
```
|
||||||
|
expires_at 已过
|
||||||
|
→ list 查询 or_(paused, expires_at>now) 排除该行
|
||||||
|
→ UI empty=true
|
||||||
|
→ DB 仍有 monitoring_enabled=true 的行
|
||||||
|
→ create_pin / pick 槽位 → IntegrityError
|
||||||
|
```
|
||||||
|
|
||||||
|
探针循环每 5s 才跑 `expire_due_pins`,此前无 API 侧兜底。
|
||||||
|
|
||||||
|
## 修复
|
||||||
|
|
||||||
|
`WatchService._finalize_due_pins()` 在 `list_slots` 及所有槽位变更 API 入口调用,与探针循环逻辑一致:暂停 pin、结束 session、无活跃监测时清 Redis。
|
||||||
|
|
||||||
|
## 测试
|
||||||
|
|
||||||
|
```text
|
||||||
|
pytest tests/test_watch_pins.py → 7 passed
|
||||||
|
pytest tests/test_watch_*.py → 21 passed
|
||||||
|
```
|
||||||
|
|
||||||
|
## 未部署
|
||||||
|
|
||||||
|
生产仍为旧版本;修复需部署后浏览器终验 TTL 归零场景。
|
||||||
@@ -0,0 +1,100 @@
|
|||||||
|
# 巡检 — 监测槽开关 / 8h·24h / 暂停占槽 / 探针修复
|
||||||
|
|
||||||
|
**日期**:2026-06-12
|
||||||
|
**范围**:工作区未提交改动(监测槽增强 + parse_error 修复 + 中文错误)
|
||||||
|
**生产**:https://api.synaglobal.vip(**尚未部署本批改动**)
|
||||||
|
|
||||||
|
## 进度条
|
||||||
|
|
||||||
|
| 步骤 | 状态 |
|
||||||
|
|------|------|
|
||||||
|
| 实现 | ✅ 工作区完成 |
|
||||||
|
| 本地验证 | ✅ L2b 26/26 + ruff |
|
||||||
|
| 单测 | ✅ watch 23/23 |
|
||||||
|
| 类型检查 | ✅ vue-tsc |
|
||||||
|
| 门控 7/7 | ⚠️ 6/7(Gate 7 被 **Agent 提示** 并行改动阻塞,非 watch 本身) |
|
||||||
|
| 部署 | ❌ 未执行 |
|
||||||
|
| 生产健康 | ✅ `/health` → `ok` |
|
||||||
|
| 浏览器终验 | ❌ 待部署后 |
|
||||||
|
|
||||||
|
## 本地验证
|
||||||
|
|
||||||
|
### 单测
|
||||||
|
|
||||||
|
```text
|
||||||
|
pytest tests/test_watch_*.py tests/test_watch_probe_errors.py
|
||||||
|
→ 23 passed
|
||||||
|
```
|
||||||
|
|
||||||
|
### L2b API E2E(local_verify.sh)
|
||||||
|
|
||||||
|
```text
|
||||||
|
26/26 passed
|
||||||
|
ruff F: All checks passed
|
||||||
|
═══ All local checks passed ═══
|
||||||
|
```
|
||||||
|
|
||||||
|
### 前端
|
||||||
|
|
||||||
|
```text
|
||||||
|
npm run type-check → PASS
|
||||||
|
```
|
||||||
|
|
||||||
|
### 门控 pre_deploy_check.sh
|
||||||
|
|
||||||
|
| Gate | 结果 |
|
||||||
|
|------|------|
|
||||||
|
| 1 Changelog | PASS(2026-06-12-server-agent-action-hint.md) |
|
||||||
|
| 2 Audit | PASS(2026-06-12-files-watch-ui-fixes.md) |
|
||||||
|
| 3 Test | PASS |
|
||||||
|
| 4 Lint | PASS |
|
||||||
|
| 5 Import | PASS |
|
||||||
|
| 6 Security | PASS |
|
||||||
|
| 7 Review | **BLOCK** — 以下文件有 diff 但未列入当日 audit |
|
||||||
|
|
||||||
|
Gate 7 阻塞文件(**Agent 版本提示**,与监测槽无关):
|
||||||
|
|
||||||
|
- `frontend/src/components/servers/ServerAgentHint.vue`
|
||||||
|
- `frontend/src/components/servers/ServerInlineDetail.vue`
|
||||||
|
- `frontend/src/composables/servers/useServerFormDialog.ts`
|
||||||
|
- `frontend/src/types/api.ts`
|
||||||
|
- `server/api/servers.py`
|
||||||
|
- `server/utils/agent_version.py`
|
||||||
|
- `tests/test_agent_version.py`
|
||||||
|
|
||||||
|
监测槽改动已有独立审计:`docs/audit/2026-06-11-watch-slot-monitoring-enhancements.md`。
|
||||||
|
|
||||||
|
## 生产探针
|
||||||
|
|
||||||
|
| 检查 | 结果 |
|
||||||
|
|------|------|
|
||||||
|
| `GET /health` | 200 `ok` |
|
||||||
|
| `GET /app/` | 200 |
|
||||||
|
| `GET /api/watch/pins`(无 JWT) | 401(预期) |
|
||||||
|
| `PATCH /api/watch/pins/0/monitoring`(无 JWT) | 401(端点存在性需部署后带 JWT 终验) |
|
||||||
|
|
||||||
|
**说明**:生产当前运行的是已部署版本;监测槽开关 / TTL PATCH / 中文错误 / parse_error 修复均在工作区,**线上行为未变**。
|
||||||
|
|
||||||
|
## 功能验收清单(部署后浏览器终验)
|
||||||
|
|
||||||
|
1. **开关暂停**:关实时监测 → 5s 探针停止,槽位仍占用,UI 显示暂停态
|
||||||
|
2. **重新开启**:开开关 → 若 session 已结束则自动开新 session,探针恢复
|
||||||
|
3. **8h / 24h**:默认 8h;点 24h 后 `expires_at` 延长;replace 槽位保留原 TTL
|
||||||
|
4. **TTL 到期**:自动暂停(不删 pin),Agent 恢复 60s 智能心跳
|
||||||
|
5. **parse_error**:Agent 有 CPU/内存时 SSH 失败不应整轮 parse_error
|
||||||
|
6. **中文错误**:探针失败状态与错误列显示中文
|
||||||
|
|
||||||
|
## 部署前待办
|
||||||
|
|
||||||
|
1. **Gate 7**:为 Agent 提示改动补 audit,或与监测槽分两次提交/部署
|
||||||
|
2. **迁移**:`watch_pins.monitoring_enabled` + `ttl_hours`(migrations.py 已含)
|
||||||
|
3. **部署**:用户批准后 `bash deploy/deploy-production.sh`
|
||||||
|
4. **前端构建**:Docker 内 vite build 或 `deploy/deploy-frontend.sh`
|
||||||
|
|
||||||
|
## 本次巡检修复
|
||||||
|
|
||||||
|
- 删除 `server/infrastructure/ssh/remote_probe.py` 未使用的 `import logging`(local_verify ruff F401)
|
||||||
|
|
||||||
|
## 结论
|
||||||
|
|
||||||
|
**代码质量就绪**(单测 + L2b + lint + 类型检查均通过)。**部署被 Gate 7 阻塞**(并行 Agent 提示改动缺 audit)。监测槽功能本身审计完备,建议与 Agent 提示改动解耦提交,或补全 audit 后一并部署。
|
||||||
@@ -296,6 +296,9 @@ export const http = {
|
|||||||
put: <T = any>(path: string, body?: unknown) =>
|
put: <T = any>(path: string, body?: unknown) =>
|
||||||
api<T>(path, { method: 'PUT', body: body ? JSON.stringify(body) : undefined }),
|
api<T>(path, { method: 'PUT', body: body ? JSON.stringify(body) : undefined }),
|
||||||
|
|
||||||
|
patch: <T = any>(path: string, body?: unknown) =>
|
||||||
|
api<T>(path, { method: 'PATCH', body: body ? JSON.stringify(body) : undefined }),
|
||||||
|
|
||||||
delete: <T = any>(path: string) =>
|
delete: <T = any>(path: string) =>
|
||||||
api<T>(path, { method: 'DELETE' }),
|
api<T>(path, { method: 'DELETE' }),
|
||||||
|
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
<script setup lang="ts">
|
<script setup lang="ts">
|
||||||
import { ref, watch } from 'vue'
|
import { ref, watch } from 'vue'
|
||||||
import { http } from '@/api'
|
import { http } from '@/api'
|
||||||
import { formatBytesPerSec, probeStatusColor } from '@/utils/watchFormat'
|
import { formatBytesPerSec, probeStatusColor, probeStatusLabel, formatProbeError, probeSourceLabel, PROBE_STATUS_FILTER_ITEMS } from '@/utils/watchFormat'
|
||||||
|
|
||||||
export interface ProbeRecordRow {
|
export interface ProbeRecordRow {
|
||||||
id: number
|
id: number
|
||||||
@@ -75,12 +75,7 @@ watch(
|
|||||||
<div class="d-flex align-center ga-2 mb-3">
|
<div class="d-flex align-center ga-2 mb-3">
|
||||||
<v-select
|
<v-select
|
||||||
v-model="statusFilter"
|
v-model="statusFilter"
|
||||||
:items="[
|
:items="PROBE_STATUS_FILTER_ITEMS"
|
||||||
{ title: '全部状态', value: null },
|
|
||||||
{ title: 'ok', value: 'ok' },
|
|
||||||
{ title: 'ssh_timeout', value: 'ssh_timeout' },
|
|
||||||
{ title: 'no_cred', value: 'no_cred' },
|
|
||||||
]"
|
|
||||||
item-title="title"
|
item-title="title"
|
||||||
item-value="value"
|
item-value="value"
|
||||||
label="探针状态"
|
label="探针状态"
|
||||||
@@ -105,14 +100,19 @@ watch(
|
|||||||
>
|
>
|
||||||
<template #item.probe_status="{ item }">
|
<template #item.probe_status="{ item }">
|
||||||
<v-chip size="x-small" :color="probeStatusColor(item.probe_status)" variant="tonal" label>
|
<v-chip size="x-small" :color="probeStatusColor(item.probe_status)" variant="tonal" label>
|
||||||
{{ item.probe_status }}
|
{{ probeStatusLabel(item.probe_status) }}
|
||||||
</v-chip>
|
</v-chip>
|
||||||
</template>
|
</template>
|
||||||
|
<template #item.source="{ item }">{{ probeSourceLabel(item.source) }}</template>
|
||||||
<template #item.net_up_bps="{ item }">{{ formatBytesPerSec(item.net_up_bps) }}</template>
|
<template #item.net_up_bps="{ item }">{{ formatBytesPerSec(item.net_up_bps) }}</template>
|
||||||
<template #item.net_down_bps="{ item }">{{ formatBytesPerSec(item.net_down_bps) }}</template>
|
<template #item.net_down_bps="{ item }">{{ formatBytesPerSec(item.net_down_bps) }}</template>
|
||||||
<template #item.duration_ms="{ item }">{{ item.duration_ms ?? '—' }} ms</template>
|
<template #item.duration_ms="{ item }">{{ item.duration_ms ?? '—' }} ms</template>
|
||||||
<template #item.error="{ item }">
|
<template #item.error="{ item }">
|
||||||
<span class="text-caption text-truncate d-inline-block" style="max-width: 200px">{{ item.error || '—' }}</span>
|
<span
|
||||||
|
class="text-caption text-truncate d-inline-block"
|
||||||
|
style="max-width: 200px"
|
||||||
|
:title="formatProbeError(item.error, item.probe_status)"
|
||||||
|
>{{ formatProbeError(item.error, item.probe_status) }}</span>
|
||||||
</template>
|
</template>
|
||||||
</v-data-table>
|
</v-data-table>
|
||||||
<div class="d-flex justify-center align-center ga-2 mt-3">
|
<div class="d-flex justify-center align-center ga-2 mt-3">
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
<script setup lang="ts">
|
<script setup lang="ts">
|
||||||
import { ref } from 'vue'
|
import { ref, computed } from 'vue'
|
||||||
import type { WatchSlot } from '@/composables/useWatchPins'
|
import type { WatchSlot } from '@/composables/useWatchPins'
|
||||||
import { formatBytesPerSec, formatRemaining, probeStatusColor } from '@/utils/watchFormat'
|
import { formatBytesPerSec, formatRemaining, probeStatusColor, probeStatusLabel, formatProbeError } from '@/utils/watchFormat'
|
||||||
import WatchSparkline from '@/components/watch/WatchSparkline.vue'
|
import WatchSparkline from '@/components/watch/WatchSparkline.vue'
|
||||||
|
|
||||||
const props = defineProps<{
|
const props = defineProps<{
|
||||||
@@ -13,8 +13,13 @@ const emit = defineEmits<{
|
|||||||
processes: [slot: WatchSlot]
|
processes: [slot: WatchSlot]
|
||||||
history: [slot: WatchSlot]
|
history: [slot: WatchSlot]
|
||||||
pick: [slotIndex: number]
|
pick: [slotIndex: number]
|
||||||
|
monitoring: [slotIndex: number, enabled: boolean]
|
||||||
|
ttl: [slotIndex: number, hours: 8 | 24]
|
||||||
}>()
|
}>()
|
||||||
|
|
||||||
|
const monitoringOn = computed(() => props.slot.monitoring_enabled !== false)
|
||||||
|
const slotTtl = computed(() => (props.slot.ttl_hours === 24 ? 24 : 8) as 8 | 24)
|
||||||
|
|
||||||
const chartMetric = ref<'cpu_pct' | 'mem_pct' | 'disk_pct'>('cpu_pct')
|
const chartMetric = ref<'cpu_pct' | 'mem_pct' | 'disk_pct'>('cpu_pct')
|
||||||
|
|
||||||
function pct(v: number | null | undefined) {
|
function pct(v: number | null | undefined) {
|
||||||
@@ -29,7 +34,10 @@ function pct(v: number | null | undefined) {
|
|||||||
border
|
border
|
||||||
rounded="lg"
|
rounded="lg"
|
||||||
class="watch-slot-card pa-3 watch-slot-card--filled"
|
class="watch-slot-card pa-3 watch-slot-card--filled"
|
||||||
:class="{ 'watch-slot-card--error': slot.metrics?.probe_status && slot.metrics.probe_status !== 'ok' }"
|
:class="{
|
||||||
|
'watch-slot-card--error': monitoringOn && slot.metrics?.probe_status && slot.metrics.probe_status !== 'ok',
|
||||||
|
'watch-slot-card--paused': !monitoringOn,
|
||||||
|
}"
|
||||||
role="button"
|
role="button"
|
||||||
tabindex="0"
|
tabindex="0"
|
||||||
@click="emit('history', slot)"
|
@click="emit('history', slot)"
|
||||||
@@ -42,9 +50,38 @@ function pct(v: number | null | undefined) {
|
|||||||
<v-chip size="x-small" variant="tonal" color="info" label>
|
<v-chip size="x-small" variant="tonal" color="info" label>
|
||||||
槽 {{ slot.slot_index + 1 }}
|
槽 {{ slot.slot_index + 1 }}
|
||||||
</v-chip>
|
</v-chip>
|
||||||
|
<v-switch
|
||||||
|
:model-value="monitoringOn"
|
||||||
|
density="compact"
|
||||||
|
hide-details
|
||||||
|
color="primary"
|
||||||
|
class="watch-slot-switch ml-1"
|
||||||
|
@click.stop
|
||||||
|
@update:model-value="emit('monitoring', slot.slot_index, $event ?? false)"
|
||||||
|
/>
|
||||||
<v-btn icon="mdi-close" size="x-small" variant="text" @click="emit('remove', slot.slot_index)" />
|
<v-btn icon="mdi-close" size="x-small" variant="text" @click="emit('remove', slot.slot_index)" />
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
<div class="d-flex align-center mb-2 ga-1" @click.stop>
|
||||||
|
<span class="text-caption text-medium-emphasis mr-1">时长</span>
|
||||||
|
<v-btn-toggle
|
||||||
|
:model-value="slotTtl"
|
||||||
|
density="compact"
|
||||||
|
variant="outlined"
|
||||||
|
divided
|
||||||
|
mandatory
|
||||||
|
@update:model-value="emit('ttl', slot.slot_index, ($event ?? 8) as 8 | 24)"
|
||||||
|
>
|
||||||
|
<v-btn :value="8" size="x-small">8h</v-btn>
|
||||||
|
<v-btn :value="24" size="x-small">24h</v-btn>
|
||||||
|
</v-btn-toggle>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div v-if="!monitoringOn" class="text-caption text-medium-emphasis mb-2" @click.stop>
|
||||||
|
实时监测已关闭 · 槽位保留 · Agent 恢复 60s 智能上报
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<template v-if="monitoringOn">
|
||||||
<div @click.stop>
|
<div @click.stop>
|
||||||
<v-chip
|
<v-chip
|
||||||
v-if="slot.metrics?.probe_status"
|
v-if="slot.metrics?.probe_status"
|
||||||
@@ -54,7 +91,7 @@ function pct(v: number | null | undefined) {
|
|||||||
class="mb-2"
|
class="mb-2"
|
||||||
label
|
label
|
||||||
>
|
>
|
||||||
{{ slot.metrics.probe_status === 'ok' ? '正常' : slot.metrics.probe_status }}
|
{{ probeStatusLabel(slot.metrics.probe_status) }}
|
||||||
</v-chip>
|
</v-chip>
|
||||||
|
|
||||||
<div class="d-flex justify-space-between text-caption mb-1">
|
<div class="d-flex justify-space-between text-caption mb-1">
|
||||||
@@ -80,18 +117,32 @@ function pct(v: number | null | undefined) {
|
|||||||
|
|
||||||
<WatchSparkline :points="slot.sparkline || []" :metric="chartMetric" />
|
<WatchSparkline :points="slot.sparkline || []" :metric="chartMetric" />
|
||||||
|
|
||||||
<div v-if="slot.metrics?.error" class="text-caption text-error mt-1 text-truncate">
|
<div
|
||||||
{{ slot.metrics.error }}
|
v-if="slot.metrics?.error"
|
||||||
|
class="text-caption text-error mt-1 text-truncate"
|
||||||
|
:title="formatProbeError(slot.metrics.error, slot.metrics.probe_status)"
|
||||||
|
>
|
||||||
|
{{ formatProbeError(slot.metrics.error, slot.metrics.probe_status) }}
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div class="d-flex align-center justify-space-between mt-2">
|
<div class="d-flex align-center justify-space-between mt-2">
|
||||||
<span class="text-caption text-medium-emphasis">剩余 {{ formatRemaining(slot.remaining_sec) }}</span>
|
<span class="text-caption text-medium-emphasis">
|
||||||
|
剩余 {{ formatRemaining(slot.remaining_sec) }} · {{ slotTtl }}h
|
||||||
|
</span>
|
||||||
<div class="d-flex ga-1">
|
<div class="d-flex ga-1">
|
||||||
<v-btn size="x-small" variant="text" @click.stop="emit('processes', slot)">进程</v-btn>
|
<v-btn size="x-small" variant="text" @click.stop="emit('processes', slot)">进程</v-btn>
|
||||||
<v-btn size="x-small" variant="text" @click.stop="emit('history', slot)">历史</v-btn>
|
<v-btn size="x-small" variant="text" @click.stop="emit('history', slot)">历史</v-btn>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
</template>
|
||||||
|
|
||||||
|
<div v-else class="d-flex align-center justify-space-between mt-2" @click.stop>
|
||||||
|
<span class="text-caption text-medium-emphasis">已暂停 · {{ slotTtl }}h</span>
|
||||||
|
<div class="d-flex ga-1">
|
||||||
|
<v-btn size="x-small" variant="text" @click.stop="emit('history', slot)">历史</v-btn>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
</v-card>
|
</v-card>
|
||||||
|
|
||||||
<v-card
|
<v-card
|
||||||
@@ -135,4 +186,12 @@ function pct(v: number | null | undefined) {
|
|||||||
.watch-slot-card--error {
|
.watch-slot-card--error {
|
||||||
border-color: rgb(var(--v-theme-error)) !important;
|
border-color: rgb(var(--v-theme-error)) !important;
|
||||||
}
|
}
|
||||||
|
.watch-slot-card--paused {
|
||||||
|
opacity: 0.72;
|
||||||
|
background: rgba(var(--v-theme-surface-variant), 0.15);
|
||||||
|
}
|
||||||
|
.watch-slot-switch {
|
||||||
|
flex: 0 0 auto;
|
||||||
|
transform: scale(0.85);
|
||||||
|
}
|
||||||
</style>
|
</style>
|
||||||
|
|||||||
@@ -6,10 +6,11 @@ import WatchProcessDrawer from '@/components/watch/WatchProcessDrawer.vue'
|
|||||||
import WatchSlotPickDialog from '@/components/watch/WatchSlotPickDialog.vue'
|
import WatchSlotPickDialog from '@/components/watch/WatchSlotPickDialog.vue'
|
||||||
import { useWatchPins, type WatchSlot } from '@/composables/useWatchPins'
|
import { useWatchPins, type WatchSlot } from '@/composables/useWatchPins'
|
||||||
import { useSnackbar } from '@/composables/useSnackbar'
|
import { useSnackbar } from '@/composables/useSnackbar'
|
||||||
|
import { formatApiError } from '@/utils/apiError'
|
||||||
|
|
||||||
const router = useRouter()
|
const router = useRouter()
|
||||||
const snackbar = useSnackbar()
|
const snackbar = useSnackbar()
|
||||||
const { slots, loading, refreshPins, removeSlot, pinServer } = useWatchPins()
|
const { slots, loading, refreshPins, removeSlot, pinServer, setSlotMonitoring, setSlotTtl } = useWatchPins()
|
||||||
|
|
||||||
const processDrawer = ref(false)
|
const processDrawer = ref(false)
|
||||||
const processSlot = ref<WatchSlot | null>(null)
|
const processSlot = ref<WatchSlot | null>(null)
|
||||||
@@ -22,7 +23,32 @@ onMounted(() => {
|
|||||||
})
|
})
|
||||||
|
|
||||||
async function onRemove(slotIndex: number) {
|
async function onRemove(slotIndex: number) {
|
||||||
await removeSlot(slotIndex)
|
try {
|
||||||
|
await removeSlot(slotIndex)
|
||||||
|
snackbar(`已移除监测 · 槽 ${slotIndex + 1}`, 'success')
|
||||||
|
} catch (e: unknown) {
|
||||||
|
snackbar(formatApiError(e, '移除监测失败'), 'error')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function onMonitoring(slotIndex: number, enabled: boolean) {
|
||||||
|
try {
|
||||||
|
await setSlotMonitoring(slotIndex, enabled)
|
||||||
|
snackbar(enabled ? `已开启实时监测 · 槽 ${slotIndex + 1}` : `已暂停实时监测 · 槽 ${slotIndex + 1}`, 'success')
|
||||||
|
} catch (e: unknown) {
|
||||||
|
snackbar(formatApiError(e, '更新监测开关失败'), 'error')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function onTtl(slotIndex: number, hours: 8 | 24) {
|
||||||
|
const slot = slots.value.find((s) => s.slot_index === slotIndex)
|
||||||
|
if (!slot || slot.empty || slot.ttl_hours === hours) return
|
||||||
|
try {
|
||||||
|
await setSlotTtl(slotIndex, hours)
|
||||||
|
snackbar(`监测时长已设为 ${hours} 小时 · 槽 ${slotIndex + 1}`, 'success')
|
||||||
|
} catch (e: unknown) {
|
||||||
|
snackbar(formatApiError(e, '更新监测时长失败'), 'error')
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
function openProcesses(slot: WatchSlot) {
|
function openProcesses(slot: WatchSlot) {
|
||||||
@@ -68,7 +94,7 @@ async function onPickServer(serverId: number) {
|
|||||||
<div class="watch-slot-row mb-4">
|
<div class="watch-slot-row mb-4">
|
||||||
<div class="d-flex align-center mb-2">
|
<div class="d-flex align-center mb-2">
|
||||||
<span class="text-subtitle-2">实时监测</span>
|
<span class="text-subtitle-2">实时监测</span>
|
||||||
<v-chip size="x-small" variant="tonal" class="ml-2" label>最多 4 台 · 8h</v-chip>
|
<v-chip size="x-small" variant="tonal" class="ml-2" label>最多 4 台 · 默认 8h · 暂停不占 TTL</v-chip>
|
||||||
<v-spacer />
|
<v-spacer />
|
||||||
<v-btn size="small" variant="text" prepend-icon="mdi-history" :to="{ name: 'WatchMetrics' }">
|
<v-btn size="small" variant="text" prepend-icon="mdi-history" :to="{ name: 'WatchMetrics' }">
|
||||||
监测历史
|
监测历史
|
||||||
@@ -83,6 +109,8 @@ async function onPickServer(serverId: number) {
|
|||||||
@processes="openProcesses"
|
@processes="openProcesses"
|
||||||
@history="openHistory"
|
@history="openHistory"
|
||||||
@pick="onPickSlot"
|
@pick="onPickSlot"
|
||||||
|
@monitoring="onMonitoring"
|
||||||
|
@ttl="onTtl"
|
||||||
/>
|
/>
|
||||||
</v-col>
|
</v-col>
|
||||||
</v-row>
|
</v-row>
|
||||||
|
|||||||
@@ -48,6 +48,8 @@ export interface WatchSlot {
|
|||||||
pinned_at?: string
|
pinned_at?: string
|
||||||
expires_at?: string
|
expires_at?: string
|
||||||
remaining_sec?: number
|
remaining_sec?: number
|
||||||
|
monitoring_enabled?: boolean
|
||||||
|
ttl_hours?: 8 | 24
|
||||||
metrics?: WatchMetrics | null
|
metrics?: WatchMetrics | null
|
||||||
sparkline?: WatchSparkPoint[]
|
sparkline?: WatchSparkPoint[]
|
||||||
processes?: WatchProcess[]
|
processes?: WatchProcess[]
|
||||||
@@ -73,34 +75,38 @@ function applySlots(data: { slots?: WatchSlot[] }) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
function tickCountdown() {
|
function tickCountdown() {
|
||||||
|
let shouldRefresh = false
|
||||||
for (const slot of slots.value) {
|
for (const slot of slots.value) {
|
||||||
if (!slot.empty && slot.remaining_sec != null && slot.remaining_sec > 0) {
|
if (slot.empty || slot.monitoring_enabled === false) continue
|
||||||
|
if (slot.remaining_sec != null && slot.remaining_sec > 0) {
|
||||||
slot.remaining_sec -= 1
|
slot.remaining_sec -= 1
|
||||||
if (slot.remaining_sec <= 0) {
|
if (slot.remaining_sec <= 0) {
|
||||||
slot.empty = true
|
shouldRefresh = true
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if (shouldRefresh) {
|
||||||
|
void refreshPins()
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
function mergeMetrics(serverId: number, metrics: WatchMetrics) {
|
function mergeMetrics(serverId: number, metrics: WatchMetrics) {
|
||||||
for (const slot of slots.value) {
|
for (const slot of slots.value) {
|
||||||
if (!slot.empty && slot.server_id === serverId) {
|
if (slot.empty || slot.server_id !== serverId || slot.monitoring_enabled === false) continue
|
||||||
slot.metrics = { ...(slot.metrics || {}), ...metrics }
|
slot.metrics = { ...(slot.metrics || {}), ...metrics }
|
||||||
if (metrics.processes?.length) {
|
if (metrics.processes?.length) {
|
||||||
slot.processes = metrics.processes
|
slot.processes = metrics.processes
|
||||||
}
|
|
||||||
const point: WatchSparkPoint = {
|
|
||||||
ts: metrics.ts,
|
|
||||||
cpu_pct: metrics.cpu_pct,
|
|
||||||
mem_pct: metrics.mem_pct,
|
|
||||||
disk_pct: metrics.disk_pct,
|
|
||||||
net_down_bps: metrics.net_down_bps,
|
|
||||||
}
|
|
||||||
if (!slot.sparkline) slot.sparkline = []
|
|
||||||
slot.sparkline.push(point)
|
|
||||||
if (slot.sparkline.length > 180) slot.sparkline.shift()
|
|
||||||
}
|
}
|
||||||
|
const point: WatchSparkPoint = {
|
||||||
|
ts: metrics.ts,
|
||||||
|
cpu_pct: metrics.cpu_pct,
|
||||||
|
mem_pct: metrics.mem_pct,
|
||||||
|
disk_pct: metrics.disk_pct,
|
||||||
|
net_down_bps: metrics.net_down_bps,
|
||||||
|
}
|
||||||
|
if (!slot.sparkline) slot.sparkline = []
|
||||||
|
slot.sparkline.push(point)
|
||||||
|
if (slot.sparkline.length > 180) slot.sparkline.shift()
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -182,6 +188,24 @@ async function removeSlot(slotIndex: number) {
|
|||||||
applySlots(data)
|
applySlots(data)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function setSlotMonitoring(slotIndex: number, monitoringEnabled: boolean) {
|
||||||
|
const data = await http.patch<{ slots: WatchSlot[] }>(
|
||||||
|
`/watch/pins/${slotIndex}/monitoring`,
|
||||||
|
{ monitoring_enabled: monitoringEnabled },
|
||||||
|
)
|
||||||
|
applySlots(data)
|
||||||
|
}
|
||||||
|
|
||||||
|
export type WatchTtlHours = 8 | 24
|
||||||
|
|
||||||
|
async function setSlotTtl(slotIndex: number, ttlHours: WatchTtlHours) {
|
||||||
|
const data = await http.patch<{ slots: WatchSlot[] }>(
|
||||||
|
`/watch/pins/${slotIndex}/ttl`,
|
||||||
|
{ ttl_hours: ttlHours },
|
||||||
|
)
|
||||||
|
applySlots(data)
|
||||||
|
}
|
||||||
|
|
||||||
export function useWatchPins() {
|
export function useWatchPins() {
|
||||||
consumerCount += 1
|
consumerCount += 1
|
||||||
if (consumerCount === 1) {
|
if (consumerCount === 1) {
|
||||||
@@ -223,5 +247,7 @@ export function useWatchPins() {
|
|||||||
refreshPins,
|
refreshPins,
|
||||||
pinServer,
|
pinServer,
|
||||||
removeSlot,
|
removeSlot,
|
||||||
|
setSlotMonitoring,
|
||||||
|
setSlotTtl,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -15,8 +15,97 @@ export function formatRemaining(sec: number | null | undefined): string {
|
|||||||
return `${m}m ${sec % 60}s`
|
return `${m}m ${sec % 60}s`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const PROBE_STATUS_LABELS: Record<string, string> = {
|
||||||
|
ok: '正常',
|
||||||
|
parse_error: '探针解析失败',
|
||||||
|
ssh_timeout: 'SSH 超时',
|
||||||
|
ssh_auth: 'SSH 认证失败',
|
||||||
|
no_cred: '无凭据',
|
||||||
|
redis_stale: 'Agent 心跳过期',
|
||||||
|
offline: '离线',
|
||||||
|
}
|
||||||
|
|
||||||
|
const PROBE_ERROR_EXACT: Record<string, string> = {
|
||||||
|
parse_error: '探针返回数据无法解析',
|
||||||
|
'psutil missing': '子机未安装 psutil(请执行 pip install psutil)',
|
||||||
|
ssh_probe_failed: 'SSH 探针脚本执行失败',
|
||||||
|
'SSH watch probe failed': 'SSH 监测探针失败',
|
||||||
|
'SSH probe failed': 'SSH 探针失败',
|
||||||
|
timeout: '连接超时',
|
||||||
|
'探针失败': '探针失败',
|
||||||
|
'无 Agent 且 SSH 探针失败': '无 Agent 且 SSH 探针失败',
|
||||||
|
'服务器不存在': '服务器不存在',
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Probe status code → Chinese label. */
|
||||||
|
export function probeStatusLabel(status: string | null | undefined): string {
|
||||||
|
if (!status || status === 'ok') return PROBE_STATUS_LABELS.ok
|
||||||
|
if (PROBE_STATUS_LABELS[status]) return PROBE_STATUS_LABELS[status]
|
||||||
|
if (status.startsWith('parse_error')) return PROBE_STATUS_LABELS.parse_error
|
||||||
|
return status
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Probe source code → Chinese label. */
|
||||||
|
export function probeSourceLabel(source: string | null | undefined): string {
|
||||||
|
if (!source) return '—'
|
||||||
|
const map: Record<string, string> = {
|
||||||
|
redis: 'Agent',
|
||||||
|
ssh: 'SSH',
|
||||||
|
mixed: 'Agent+SSH',
|
||||||
|
}
|
||||||
|
return map[source] ?? source
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Error detail → Chinese (covers legacy English rows in DB). */
|
||||||
|
export function formatProbeError(
|
||||||
|
error: string | null | undefined,
|
||||||
|
status?: string | null,
|
||||||
|
): string {
|
||||||
|
if (!error?.trim()) {
|
||||||
|
if (status && status !== 'ok') return probeStatusLabel(status)
|
||||||
|
return '—'
|
||||||
|
}
|
||||||
|
|
||||||
|
const text = error.trim()
|
||||||
|
if (PROBE_ERROR_EXACT[text]) return PROBE_ERROR_EXACT[text]
|
||||||
|
|
||||||
|
const lower = text.toLowerCase()
|
||||||
|
if (lower.startsWith('parse_error')) {
|
||||||
|
const detail = text.includes(':') ? text.split(':').slice(1).join(':').trim() : ''
|
||||||
|
if (detail && PROBE_ERROR_EXACT[detail]) {
|
||||||
|
return `探针返回数据无法解析:${PROBE_ERROR_EXACT[detail]}`
|
||||||
|
}
|
||||||
|
if (detail) return `探针返回数据无法解析:${detail}`
|
||||||
|
return PROBE_ERROR_EXACT.parse_error
|
||||||
|
}
|
||||||
|
if (lower.includes('timed out') || lower.endsWith('timeout')) {
|
||||||
|
return `SSH 连接超时:${text}`
|
||||||
|
}
|
||||||
|
if (lower.includes('permission denied') || lower.includes('authentication failed')) {
|
||||||
|
return `SSH 认证失败:${text}`
|
||||||
|
}
|
||||||
|
if (lower.includes('psutil missing')) {
|
||||||
|
return PROBE_ERROR_EXACT['psutil missing']
|
||||||
|
}
|
||||||
|
if (lower.includes('ssh_probe_failed') || lower.includes('ssh watch probe failed')) {
|
||||||
|
return PROBE_ERROR_EXACT.ssh_probe_failed
|
||||||
|
}
|
||||||
|
if (/[\u4e00-\u9fff]/.test(text)) return text
|
||||||
|
return `探针异常:${text}`
|
||||||
|
}
|
||||||
|
|
||||||
export function probeStatusColor(status: string | null | undefined): string {
|
export function probeStatusColor(status: string | null | undefined): string {
|
||||||
if (!status || status === 'ok') return 'success'
|
if (!status || status === 'ok') return 'success'
|
||||||
if (status.includes('timeout') || status.includes('auth')) return 'warning'
|
if (status.includes('timeout') || status.includes('auth')) return 'warning'
|
||||||
return 'error'
|
return 'error'
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Status filter options for probe records table. */
|
||||||
|
export const PROBE_STATUS_FILTER_ITEMS = [
|
||||||
|
{ title: '全部状态', value: null as string | null },
|
||||||
|
{ title: '正常', value: 'ok' },
|
||||||
|
{ title: 'SSH 超时', value: 'ssh_timeout' },
|
||||||
|
{ title: '探针解析失败', value: 'parse_error' },
|
||||||
|
{ title: '无凭据', value: 'no_cred' },
|
||||||
|
{ title: '离线', value: 'offline' },
|
||||||
|
]
|
||||||
|
|||||||
@@ -2,6 +2,8 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Literal
|
||||||
|
|
||||||
from fastapi import APIRouter, Depends, HTTPException, Query, Request
|
from fastapi import APIRouter, Depends, HTTPException, Query, Request
|
||||||
from pydantic import BaseModel, Field
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
@@ -9,6 +11,7 @@ from server.api.auth_jwt import get_current_admin
|
|||||||
from server.api.dependencies import get_db
|
from server.api.dependencies import get_db
|
||||||
from server.application.services.watch_service import SlotsFullError, WatchService
|
from server.application.services.watch_service import SlotsFullError, WatchService
|
||||||
from server.domain.models import Admin
|
from server.domain.models import Admin
|
||||||
|
from server.utils.watch_metrics import WATCH_PIN_TTL_HOURS
|
||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
||||||
router = APIRouter(prefix="/api/watch", tags=["watch"])
|
router = APIRouter(prefix="/api/watch", tags=["watch"])
|
||||||
@@ -18,12 +21,24 @@ class WatchPinCreate(BaseModel):
|
|||||||
server_id: int = Field(..., gt=0)
|
server_id: int = Field(..., gt=0)
|
||||||
slot_index: int | None = Field(None, ge=0, le=3)
|
slot_index: int | None = Field(None, ge=0, le=3)
|
||||||
replace_slot: int | None = Field(None, ge=0, le=3, description="监测已满时指定替换槽位")
|
replace_slot: int | None = Field(None, ge=0, le=3, description="监测已满时指定替换槽位")
|
||||||
|
ttl_hours: Literal[8, 24] = Field(
|
||||||
|
WATCH_PIN_TTL_HOURS,
|
||||||
|
description="监测窗口时长,默认 8 小时;24 小时需显式指定",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class WatchPinReplace(BaseModel):
|
class WatchPinReplace(BaseModel):
|
||||||
server_id: int = Field(..., gt=0)
|
server_id: int = Field(..., gt=0)
|
||||||
|
|
||||||
|
|
||||||
|
class WatchPinMonitoringUpdate(BaseModel):
|
||||||
|
monitoring_enabled: bool = Field(..., description="是否开启此槽位实时监测")
|
||||||
|
|
||||||
|
|
||||||
|
class WatchPinTtlUpdate(BaseModel):
|
||||||
|
ttl_hours: Literal[8, 24] = Field(..., description="监测窗口时长:8 或 24 小时")
|
||||||
|
|
||||||
|
|
||||||
def _client_ip(request: Request) -> str:
|
def _client_ip(request: Request) -> str:
|
||||||
return request.client.host if request.client else ""
|
return request.client.host if request.client else ""
|
||||||
|
|
||||||
@@ -51,6 +66,7 @@ async def create_watch_pin(
|
|||||||
server_id=payload.server_id,
|
server_id=payload.server_id,
|
||||||
slot_index=payload.slot_index,
|
slot_index=payload.slot_index,
|
||||||
replace_slot=payload.replace_slot,
|
replace_slot=payload.replace_slot,
|
||||||
|
ttl_hours=payload.ttl_hours,
|
||||||
ip_address=_client_ip(request),
|
ip_address=_client_ip(request),
|
||||||
)
|
)
|
||||||
except SlotsFullError as e:
|
except SlotsFullError as e:
|
||||||
@@ -97,6 +113,46 @@ async def delete_watch_pin(
|
|||||||
raise HTTPException(status_code=404, detail=str(e)) from e
|
raise HTTPException(status_code=404, detail=str(e)) from e
|
||||||
|
|
||||||
|
|
||||||
|
@router.patch("/pins/{slot_index}/monitoring")
|
||||||
|
async def update_watch_pin_monitoring(
|
||||||
|
slot_index: int,
|
||||||
|
payload: WatchPinMonitoringUpdate,
|
||||||
|
request: Request,
|
||||||
|
admin: Admin = Depends(get_current_admin),
|
||||||
|
db: AsyncSession = Depends(get_db),
|
||||||
|
):
|
||||||
|
svc = WatchService(db)
|
||||||
|
try:
|
||||||
|
return await svc.set_slot_monitoring(
|
||||||
|
admin=admin,
|
||||||
|
slot_index=slot_index,
|
||||||
|
monitoring_enabled=payload.monitoring_enabled,
|
||||||
|
ip_address=_client_ip(request),
|
||||||
|
)
|
||||||
|
except ValueError as e:
|
||||||
|
raise HTTPException(status_code=404, detail=str(e)) from e
|
||||||
|
|
||||||
|
|
||||||
|
@router.patch("/pins/{slot_index}/ttl")
|
||||||
|
async def update_watch_pin_ttl(
|
||||||
|
slot_index: int,
|
||||||
|
payload: WatchPinTtlUpdate,
|
||||||
|
request: Request,
|
||||||
|
admin: Admin = Depends(get_current_admin),
|
||||||
|
db: AsyncSession = Depends(get_db),
|
||||||
|
):
|
||||||
|
svc = WatchService(db)
|
||||||
|
try:
|
||||||
|
return await svc.set_slot_ttl(
|
||||||
|
admin=admin,
|
||||||
|
slot_index=slot_index,
|
||||||
|
ttl_hours=payload.ttl_hours,
|
||||||
|
ip_address=_client_ip(request),
|
||||||
|
)
|
||||||
|
except ValueError as e:
|
||||||
|
raise HTTPException(status_code=404, detail=str(e)) from e
|
||||||
|
|
||||||
|
|
||||||
@router.get("/metrics")
|
@router.get("/metrics")
|
||||||
async def get_watch_metrics(
|
async def get_watch_metrics(
|
||||||
server_ids: str = Query(..., description="逗号分隔 server_id"),
|
server_ids: str = Query(..., description="逗号分隔 server_id"),
|
||||||
|
|||||||
@@ -13,7 +13,13 @@ from server.infrastructure.database.audit_log_repo import AuditLogRepositoryImpl
|
|||||||
from server.infrastructure.database.server_repo import ServerRepositoryImpl
|
from server.infrastructure.database.server_repo import ServerRepositoryImpl
|
||||||
from server.infrastructure.database.watch_repo import WatchRepositoryImpl
|
from server.infrastructure.database.watch_repo import WatchRepositoryImpl
|
||||||
from server.infrastructure.redis.client import get_redis
|
from server.infrastructure.redis.client import get_redis
|
||||||
from server.utils.watch_metrics import WATCH_MAX_SLOTS, WATCH_PIN_TTL_HOURS, server_can_be_watched
|
from server.utils.watch_state import clear_watch_redis_snapshot
|
||||||
|
from server.utils.watch_metrics import (
|
||||||
|
WATCH_MAX_SLOTS,
|
||||||
|
WATCH_PIN_TTL_HOURS,
|
||||||
|
normalize_watch_ttl_hours,
|
||||||
|
server_can_be_watched,
|
||||||
|
)
|
||||||
|
|
||||||
REDIS_WATCH_LIVE_PREFIX = "watch:live:"
|
REDIS_WATCH_LIVE_PREFIX = "watch:live:"
|
||||||
REDIS_WATCH_PROC_PREFIX = "watch:proc:"
|
REDIS_WATCH_PROC_PREFIX = "watch:proc:"
|
||||||
@@ -67,6 +73,23 @@ async def _load_processes(server_id: int) -> list[dict[str, Any]] | None:
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
async def _restore_idle_agent_watch(watch_repo: WatchRepositoryImpl, server_id: int) -> None:
|
||||||
|
"""Stop 5s watch snapshots when no slot actively monitors this server (Agent stays 60s)."""
|
||||||
|
if await watch_repo.count_monitoring_pins_for_server(server_id) == 0:
|
||||||
|
await clear_watch_redis_snapshot(server_id)
|
||||||
|
|
||||||
|
|
||||||
|
async def _finalize_due_pins(watch_repo: WatchRepositoryImpl, db: AsyncSession) -> None:
|
||||||
|
"""Pause TTL-expired pins on read/write paths (probe loop also runs this every 5s)."""
|
||||||
|
expired_servers = await watch_repo.expire_due_pins()
|
||||||
|
if not expired_servers:
|
||||||
|
return
|
||||||
|
for server_id in expired_servers:
|
||||||
|
if await watch_repo.count_monitoring_pins_for_server(server_id) == 0:
|
||||||
|
await clear_watch_redis_snapshot(server_id)
|
||||||
|
await db.commit()
|
||||||
|
|
||||||
|
|
||||||
async def _slot_payload(
|
async def _slot_payload(
|
||||||
*,
|
*,
|
||||||
slot_index: int,
|
slot_index: int,
|
||||||
@@ -74,11 +97,17 @@ async def _slot_payload(
|
|||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
if not pin_row:
|
if not pin_row:
|
||||||
return {"slot_index": slot_index, "empty": True}
|
return {"slot_index": slot_index, "empty": True}
|
||||||
metrics = await _load_live_metrics(pin_row["server_id"])
|
monitoring_enabled = pin_row.get("monitoring_enabled", True)
|
||||||
sparkline = await _load_sparkline(pin_row["server_id"]) if pin_row else []
|
metrics = None
|
||||||
processes = await _load_processes(pin_row["server_id"])
|
sparkline: list[dict[str, Any]] = []
|
||||||
if metrics and processes:
|
processes = None
|
||||||
metrics = {**metrics, "processes": processes}
|
if monitoring_enabled:
|
||||||
|
metrics = await _load_live_metrics(pin_row["server_id"])
|
||||||
|
sparkline = await _load_sparkline(pin_row["server_id"])
|
||||||
|
processes = await _load_processes(pin_row["server_id"])
|
||||||
|
if metrics and processes:
|
||||||
|
metrics = {**metrics, "processes": processes}
|
||||||
|
remaining = _remaining_sec(pin_row["expires_at"]) if monitoring_enabled else None
|
||||||
return {
|
return {
|
||||||
"slot_index": slot_index,
|
"slot_index": slot_index,
|
||||||
"empty": False,
|
"empty": False,
|
||||||
@@ -88,7 +117,9 @@ async def _slot_payload(
|
|||||||
"session_id": pin_row["session_id"],
|
"session_id": pin_row["session_id"],
|
||||||
"pinned_at": pin_row["pinned_at"].isoformat() if pin_row.get("pinned_at") else None,
|
"pinned_at": pin_row["pinned_at"].isoformat() if pin_row.get("pinned_at") else None,
|
||||||
"expires_at": pin_row["expires_at"].isoformat() if pin_row.get("expires_at") else None,
|
"expires_at": pin_row["expires_at"].isoformat() if pin_row.get("expires_at") else None,
|
||||||
"remaining_sec": _remaining_sec(pin_row["expires_at"]),
|
"remaining_sec": remaining,
|
||||||
|
"monitoring_enabled": monitoring_enabled,
|
||||||
|
"ttl_hours": int(pin_row.get("ttl_hours") or WATCH_PIN_TTL_HOURS),
|
||||||
"metrics": metrics,
|
"metrics": metrics,
|
||||||
"sparkline": sparkline,
|
"sparkline": sparkline,
|
||||||
"processes": processes,
|
"processes": processes,
|
||||||
@@ -103,6 +134,7 @@ class WatchService:
|
|||||||
self.audit_repo = AuditLogRepositoryImpl(db)
|
self.audit_repo = AuditLogRepositoryImpl(db)
|
||||||
|
|
||||||
async def list_slots(self, admin_id: int) -> dict[str, Any]:
|
async def list_slots(self, admin_id: int) -> dict[str, Any]:
|
||||||
|
await _finalize_due_pins(self.watch_repo, self.db)
|
||||||
pins = await self.watch_repo.list_active_pins_for_admin(admin_id)
|
pins = await self.watch_repo.list_active_pins_for_admin(admin_id)
|
||||||
by_slot = {p["slot_index"]: p for p in pins}
|
by_slot = {p["slot_index"]: p for p in pins}
|
||||||
slots = [
|
slots = [
|
||||||
@@ -118,8 +150,11 @@ class WatchService:
|
|||||||
server_id: int,
|
server_id: int,
|
||||||
slot_index: int | None = None,
|
slot_index: int | None = None,
|
||||||
replace_slot: int | None = None,
|
replace_slot: int | None = None,
|
||||||
|
ttl_hours: int = WATCH_PIN_TTL_HOURS,
|
||||||
ip_address: str = "",
|
ip_address: str = "",
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
|
await _finalize_due_pins(self.watch_repo, self.db)
|
||||||
|
hours = normalize_watch_ttl_hours(ttl_hours)
|
||||||
server = await self.server_repo.get_by_id(server_id)
|
server = await self.server_repo.get_by_id(server_id)
|
||||||
if not server:
|
if not server:
|
||||||
raise ValueError("服务器不存在")
|
raise ValueError("服务器不存在")
|
||||||
@@ -130,12 +165,12 @@ class WatchService:
|
|||||||
|
|
||||||
existing = await self.watch_repo.get_pin_by_server(admin.id, server_id)
|
existing = await self.watch_repo.get_pin_by_server(admin.id, server_id)
|
||||||
if existing:
|
if existing:
|
||||||
await self.watch_repo.reset_pin_ttl(admin.id, server_id, WATCH_PIN_TTL_HOURS)
|
await self.watch_repo.reset_pin_ttl(admin.id, server_id, hours)
|
||||||
await self._audit(
|
await self._audit(
|
||||||
admin=admin,
|
admin=admin,
|
||||||
action="watch_pin_add",
|
action="watch_pin_add",
|
||||||
server=server,
|
server=server,
|
||||||
detail=f"重置监测 8h · 槽 {existing.slot_index + 1}",
|
detail=f"重置监测 {hours}h · 槽 {existing.slot_index + 1}",
|
||||||
ip_address=ip_address,
|
ip_address=ip_address,
|
||||||
)
|
)
|
||||||
await self.db.commit()
|
await self.db.commit()
|
||||||
@@ -146,13 +181,13 @@ class WatchService:
|
|||||||
admin_id=admin.id,
|
admin_id=admin.id,
|
||||||
slot_index=slot_index,
|
slot_index=slot_index,
|
||||||
server_id=server_id,
|
server_id=server_id,
|
||||||
ttl_hours=WATCH_PIN_TTL_HOURS,
|
ttl_hours=hours,
|
||||||
)
|
)
|
||||||
await self._audit(
|
await self._audit(
|
||||||
admin=admin,
|
admin=admin,
|
||||||
action="watch_pin_replace" if replace_slot is not None else "watch_pin_add",
|
action="watch_pin_replace" if replace_slot is not None else "watch_pin_add",
|
||||||
server=server,
|
server=server,
|
||||||
detail=f"加入监测 · 槽 {slot_index + 1}",
|
detail=f"加入监测 · 槽 {slot_index + 1} · {hours}h",
|
||||||
ip_address=ip_address,
|
ip_address=ip_address,
|
||||||
)
|
)
|
||||||
await self.db.commit()
|
await self.db.commit()
|
||||||
@@ -164,13 +199,13 @@ class WatchService:
|
|||||||
admin_id=admin.id,
|
admin_id=admin.id,
|
||||||
server_id=server_id,
|
server_id=server_id,
|
||||||
slot_index=empty,
|
slot_index=empty,
|
||||||
ttl_hours=WATCH_PIN_TTL_HOURS,
|
ttl_hours=hours,
|
||||||
)
|
)
|
||||||
await self._audit(
|
await self._audit(
|
||||||
admin=admin,
|
admin=admin,
|
||||||
action="watch_pin_add",
|
action="watch_pin_add",
|
||||||
server=server,
|
server=server,
|
||||||
detail=f"加入监测 · 槽 {empty + 1}",
|
detail=f"加入监测 · 槽 {empty + 1} · {hours}h",
|
||||||
ip_address=ip_address,
|
ip_address=ip_address,
|
||||||
)
|
)
|
||||||
await self.db.commit()
|
await self.db.commit()
|
||||||
@@ -181,13 +216,13 @@ class WatchService:
|
|||||||
admin_id=admin.id,
|
admin_id=admin.id,
|
||||||
slot_index=replace_slot,
|
slot_index=replace_slot,
|
||||||
server_id=server_id,
|
server_id=server_id,
|
||||||
ttl_hours=WATCH_PIN_TTL_HOURS,
|
ttl_hours=hours,
|
||||||
)
|
)
|
||||||
await self._audit(
|
await self._audit(
|
||||||
admin=admin,
|
admin=admin,
|
||||||
action="watch_pin_replace",
|
action="watch_pin_replace",
|
||||||
server=server,
|
server=server,
|
||||||
detail=f"替换监测 · 槽 {replace_slot + 1}",
|
detail=f"替换监测 · 槽 {replace_slot + 1} · {hours}h",
|
||||||
ip_address=ip_address,
|
ip_address=ip_address,
|
||||||
)
|
)
|
||||||
await self.db.commit()
|
await self.db.commit()
|
||||||
@@ -203,6 +238,7 @@ class WatchService:
|
|||||||
server_id: int,
|
server_id: int,
|
||||||
ip_address: str = "",
|
ip_address: str = "",
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
|
await _finalize_due_pins(self.watch_repo, self.db)
|
||||||
if slot_index < 0 or slot_index >= WATCH_MAX_SLOTS:
|
if slot_index < 0 or slot_index >= WATCH_MAX_SLOTS:
|
||||||
raise ValueError("槽位无效")
|
raise ValueError("槽位无效")
|
||||||
server = await self.server_repo.get_by_id(server_id)
|
server = await self.server_repo.get_by_id(server_id)
|
||||||
@@ -215,17 +251,19 @@ class WatchService:
|
|||||||
if other and other.slot_index != slot_index:
|
if other and other.slot_index != slot_index:
|
||||||
await self.watch_repo.end_session(other.session_id, "replaced")
|
await self.watch_repo.end_session(other.session_id, "replaced")
|
||||||
await self.watch_repo.delete_pin(other.id)
|
await self.watch_repo.delete_pin(other.id)
|
||||||
|
existing = await self.watch_repo.get_pin_by_slot(admin.id, slot_index)
|
||||||
|
slot_ttl = normalize_watch_ttl_hours(int(existing.ttl_hours or 8)) if existing else WATCH_PIN_TTL_HOURS
|
||||||
await self.watch_repo.replace_pin_server(
|
await self.watch_repo.replace_pin_server(
|
||||||
admin_id=admin.id,
|
admin_id=admin.id,
|
||||||
slot_index=slot_index,
|
slot_index=slot_index,
|
||||||
server_id=server_id,
|
server_id=server_id,
|
||||||
ttl_hours=WATCH_PIN_TTL_HOURS,
|
ttl_hours=slot_ttl,
|
||||||
)
|
)
|
||||||
await self._audit(
|
await self._audit(
|
||||||
admin=admin,
|
admin=admin,
|
||||||
action="watch_pin_replace",
|
action="watch_pin_replace",
|
||||||
server=server,
|
server=server,
|
||||||
detail=f"替换监测 · 槽 {slot_index + 1}",
|
detail=f"替换监测 · 槽 {slot_index + 1} · {slot_ttl}h",
|
||||||
ip_address=ip_address,
|
ip_address=ip_address,
|
||||||
)
|
)
|
||||||
await self.db.commit()
|
await self.db.commit()
|
||||||
@@ -238,12 +276,14 @@ class WatchService:
|
|||||||
slot_index: int,
|
slot_index: int,
|
||||||
ip_address: str = "",
|
ip_address: str = "",
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
|
await _finalize_due_pins(self.watch_repo, self.db)
|
||||||
if slot_index < 0 or slot_index >= WATCH_MAX_SLOTS:
|
if slot_index < 0 or slot_index >= WATCH_MAX_SLOTS:
|
||||||
raise ValueError("槽位无效")
|
raise ValueError("槽位无效")
|
||||||
pin = await self.watch_repo.remove_pin_by_slot(admin.id, slot_index)
|
pin = await self.watch_repo.remove_pin_by_slot(admin.id, slot_index)
|
||||||
if not pin:
|
if not pin:
|
||||||
raise ValueError("槽位为空")
|
raise ValueError("槽位为空")
|
||||||
server = await self.server_repo.get_by_id(pin.server_id)
|
server_id = pin.server_id
|
||||||
|
server = await self.server_repo.get_by_id(server_id)
|
||||||
await self._audit(
|
await self._audit(
|
||||||
admin=admin,
|
admin=admin,
|
||||||
action="watch_pin_remove",
|
action="watch_pin_remove",
|
||||||
@@ -252,6 +292,71 @@ class WatchService:
|
|||||||
ip_address=ip_address,
|
ip_address=ip_address,
|
||||||
)
|
)
|
||||||
await self.db.commit()
|
await self.db.commit()
|
||||||
|
await _restore_idle_agent_watch(self.watch_repo, server_id)
|
||||||
|
return await self.list_slots(admin.id)
|
||||||
|
|
||||||
|
async def set_slot_monitoring(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
admin: Admin,
|
||||||
|
slot_index: int,
|
||||||
|
monitoring_enabled: bool,
|
||||||
|
ip_address: str = "",
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
await _finalize_due_pins(self.watch_repo, self.db)
|
||||||
|
if slot_index < 0 or slot_index >= WATCH_MAX_SLOTS:
|
||||||
|
raise ValueError("槽位无效")
|
||||||
|
pin = await self.watch_repo.set_monitoring_enabled(
|
||||||
|
admin.id,
|
||||||
|
slot_index,
|
||||||
|
monitoring_enabled,
|
||||||
|
)
|
||||||
|
if not pin:
|
||||||
|
raise ValueError("槽位为空")
|
||||||
|
server = await self.server_repo.get_by_id(pin.server_id)
|
||||||
|
hours = int(pin.ttl_hours or WATCH_PIN_TTL_HOURS)
|
||||||
|
action = "watch_pin_resume" if monitoring_enabled else "watch_pin_pause"
|
||||||
|
detail = (
|
||||||
|
f"开启实时监测 · 槽 {slot_index + 1} · 重置 {hours}h"
|
||||||
|
if monitoring_enabled
|
||||||
|
else f"暂停实时监测 · 槽 {slot_index + 1} · 恢复 Agent 60s"
|
||||||
|
)
|
||||||
|
await self._audit(
|
||||||
|
admin=admin,
|
||||||
|
action=action,
|
||||||
|
server=server,
|
||||||
|
detail=detail,
|
||||||
|
ip_address=ip_address,
|
||||||
|
)
|
||||||
|
await self.db.commit()
|
||||||
|
if not monitoring_enabled:
|
||||||
|
await _restore_idle_agent_watch(self.watch_repo, pin.server_id)
|
||||||
|
return await self.list_slots(admin.id)
|
||||||
|
|
||||||
|
async def set_slot_ttl(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
admin: Admin,
|
||||||
|
slot_index: int,
|
||||||
|
ttl_hours: int,
|
||||||
|
ip_address: str = "",
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
await _finalize_due_pins(self.watch_repo, self.db)
|
||||||
|
if slot_index < 0 or slot_index >= WATCH_MAX_SLOTS:
|
||||||
|
raise ValueError("槽位无效")
|
||||||
|
hours = normalize_watch_ttl_hours(ttl_hours)
|
||||||
|
pin = await self.watch_repo.set_pin_ttl_hours(admin.id, slot_index, hours)
|
||||||
|
if not pin:
|
||||||
|
raise ValueError("槽位为空")
|
||||||
|
server = await self.server_repo.get_by_id(pin.server_id)
|
||||||
|
await self._audit(
|
||||||
|
admin=admin,
|
||||||
|
action="watch_pin_ttl",
|
||||||
|
server=server,
|
||||||
|
detail=f"监测时长 · 槽 {slot_index + 1} · {hours}h",
|
||||||
|
ip_address=ip_address,
|
||||||
|
)
|
||||||
|
await self.db.commit()
|
||||||
return await self.list_slots(admin.id)
|
return await self.list_slots(admin.id)
|
||||||
|
|
||||||
async def get_live_for_servers(self, server_ids: list[int]) -> dict[str, Any]:
|
async def get_live_for_servers(self, server_ids: list[int]) -> dict[str, Any]:
|
||||||
|
|||||||
@@ -17,6 +17,8 @@ from server.infrastructure.database.watch_repo import WatchRepositoryImpl
|
|||||||
from server.infrastructure.redis.client import get_redis
|
from server.infrastructure.redis.client import get_redis
|
||||||
from server.infrastructure.ssh.remote_probe import ssh_watch_probe, ssh_watch_processes
|
from server.infrastructure.ssh.remote_probe import ssh_watch_probe, ssh_watch_processes
|
||||||
from server.utils.watch_alerts import process_watch_probe_alerts
|
from server.utils.watch_alerts import process_watch_probe_alerts
|
||||||
|
from server.utils.watch_probe_errors import watch_probe_error_zh
|
||||||
|
from server.utils.watch_state import clear_watch_redis_snapshot
|
||||||
from server.utils.watch_metrics import (
|
from server.utils.watch_metrics import (
|
||||||
WATCH_PROBE_INTERVAL_SEC,
|
WATCH_PROBE_INTERVAL_SEC,
|
||||||
WATCH_PROBE_TIMEOUT_SEC,
|
WATCH_PROBE_TIMEOUT_SEC,
|
||||||
@@ -29,6 +31,7 @@ from server.utils.watch_metrics import (
|
|||||||
parse_redis_watch_payload,
|
parse_redis_watch_payload,
|
||||||
parse_ssh_watch_payload,
|
parse_ssh_watch_payload,
|
||||||
redis_sample_has_io,
|
redis_sample_has_io,
|
||||||
|
redis_sample_has_basic_metrics,
|
||||||
)
|
)
|
||||||
|
|
||||||
logger = logging.getLogger("nexus.watch_probe")
|
logger = logging.getLogger("nexus.watch_probe")
|
||||||
@@ -118,9 +121,17 @@ async def probe_server_metrics(
|
|||||||
if heartbeat and _heartbeat_fresh(heartbeat):
|
if heartbeat and _heartbeat_fresh(heartbeat):
|
||||||
redis_sample = parse_redis_watch_payload(heartbeat)
|
redis_sample = parse_redis_watch_payload(heartbeat)
|
||||||
|
|
||||||
# M3: skip SSH when Agent heartbeat already carries full IO counters
|
if redis_sample is not None and redis_sample_has_io(redis_sample):
|
||||||
if redis_sample_has_io(redis_sample):
|
sample = redis_sample
|
||||||
sample = redis_sample # type: ignore[assignment]
|
elif redis_sample is not None and redis_sample_has_basic_metrics(redis_sample):
|
||||||
|
# Agent CPU/mem available — SSH only supplements IO; SSH failure must not block display
|
||||||
|
ssh_result = await ssh_watch_probe(server, timeout=WATCH_PROBE_TIMEOUT_SEC)
|
||||||
|
if ssh_result.get("ok"):
|
||||||
|
ssh_sample = parse_ssh_watch_payload(ssh_result["payload"])
|
||||||
|
ssh_sample.duration_ms = int(ssh_result.get("duration_ms") or 0)
|
||||||
|
sample = merge_redis_and_ssh(redis_sample, ssh_sample)
|
||||||
|
else:
|
||||||
|
sample = redis_sample
|
||||||
else:
|
else:
|
||||||
ssh_result = await ssh_watch_probe(server, timeout=WATCH_PROBE_TIMEOUT_SEC)
|
ssh_result = await ssh_watch_probe(server, timeout=WATCH_PROBE_TIMEOUT_SEC)
|
||||||
if ssh_result.get("ok"):
|
if ssh_result.get("ok"):
|
||||||
@@ -139,13 +150,16 @@ async def probe_server_metrics(
|
|||||||
return failure_sample(
|
return failure_sample(
|
||||||
status="no_cred",
|
status="no_cred",
|
||||||
source="ssh",
|
source="ssh",
|
||||||
error=ssh_result.get("error") or "无 Agent 且 SSH 探针失败",
|
error=watch_probe_error_zh(
|
||||||
|
ssh_result.get("error") or "无 Agent 且 SSH 探针失败",
|
||||||
|
status="no_cred",
|
||||||
|
) or "无 Agent 且 SSH 探针失败",
|
||||||
duration_ms=int(ssh_result.get("duration_ms") or 0),
|
duration_ms=int(ssh_result.get("duration_ms") or 0),
|
||||||
)
|
)
|
||||||
return failure_sample(
|
return failure_sample(
|
||||||
status=status,
|
status=status,
|
||||||
source="ssh",
|
source="ssh",
|
||||||
error=ssh_result.get("error") or "探针失败",
|
error=watch_probe_error_zh(ssh_result.get("error") or "探针失败", status=status) or "探针失败",
|
||||||
duration_ms=int(ssh_result.get("duration_ms") or 0),
|
duration_ms=int(ssh_result.get("duration_ms") or 0),
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -161,8 +175,11 @@ async def _run_probe_cycle() -> None:
|
|||||||
async with AsyncSessionLocal() as db:
|
async with AsyncSessionLocal() as db:
|
||||||
watch_repo = WatchRepositoryImpl(db)
|
watch_repo = WatchRepositoryImpl(db)
|
||||||
server_repo = ServerRepositoryImpl(db)
|
server_repo = ServerRepositoryImpl(db)
|
||||||
await watch_repo.expire_due_pins()
|
expired_servers = await watch_repo.expire_due_pins()
|
||||||
pins = await watch_repo.list_active_pins_all()
|
for server_id in expired_servers:
|
||||||
|
if await watch_repo.count_monitoring_pins_for_server(server_id) == 0:
|
||||||
|
await clear_watch_redis_snapshot(server_id)
|
||||||
|
pins = await watch_repo.list_active_pins_all(monitoring_only=True)
|
||||||
if not pins:
|
if not pins:
|
||||||
await db.commit()
|
await db.commit()
|
||||||
return
|
return
|
||||||
|
|||||||
@@ -183,6 +183,18 @@ class WatchPin(Base):
|
|||||||
server_id = Column(Integer, ForeignKey("servers.id", ondelete="CASCADE"), nullable=False)
|
server_id = Column(Integer, ForeignKey("servers.id", ondelete="CASCADE"), nullable=False)
|
||||||
pinned_at = Column(DateTime, nullable=False, comment="UTC naive")
|
pinned_at = Column(DateTime, nullable=False, comment="UTC naive")
|
||||||
expires_at = Column(DateTime, nullable=False, comment="UTC naive")
|
expires_at = Column(DateTime, nullable=False, comment="UTC naive")
|
||||||
|
monitoring_enabled = Column(
|
||||||
|
Boolean,
|
||||||
|
nullable=False,
|
||||||
|
default=True,
|
||||||
|
comment="是否开启实时探针(关闭时保留槽位、暂停倒计时)",
|
||||||
|
)
|
||||||
|
ttl_hours = Column(
|
||||||
|
Integer,
|
||||||
|
nullable=False,
|
||||||
|
default=8,
|
||||||
|
comment="监测窗口时长(小时),仅允许 8 或 24",
|
||||||
|
)
|
||||||
|
|
||||||
__table_args__ = (
|
__table_args__ = (
|
||||||
Index("uq_watch_pins_admin_slot", "admin_id", "slot_index", unique=True),
|
Index("uq_watch_pins_admin_slot", "admin_id", "slot_index", unique=True),
|
||||||
|
|||||||
@@ -239,6 +239,8 @@ async def run_schema_migrations():
|
|||||||
`server_id` INT NOT NULL,
|
`server_id` INT NOT NULL,
|
||||||
`pinned_at` DATETIME NOT NULL,
|
`pinned_at` DATETIME NOT NULL,
|
||||||
`expires_at` DATETIME NOT NULL,
|
`expires_at` DATETIME NOT NULL,
|
||||||
|
`monitoring_enabled` TINYINT(1) NOT NULL DEFAULT 1 COMMENT '是否开启实时探针',
|
||||||
|
`ttl_hours` INT NOT NULL DEFAULT 8 COMMENT '监测窗口时长(小时) 8|24',
|
||||||
UNIQUE KEY `uq_watch_pins_admin_slot` (`admin_id`, `slot_index`),
|
UNIQUE KEY `uq_watch_pins_admin_slot` (`admin_id`, `slot_index`),
|
||||||
UNIQUE KEY `uq_watch_pins_admin_server` (`admin_id`, `server_id`),
|
UNIQUE KEY `uq_watch_pins_admin_server` (`admin_id`, `server_id`),
|
||||||
INDEX `idx_watch_pins_expires` (`expires_at`),
|
INDEX `idx_watch_pins_expires` (`expires_at`),
|
||||||
@@ -281,6 +283,8 @@ async def run_schema_migrations():
|
|||||||
CONSTRAINT `fk_watch_probe_admin` FOREIGN KEY (`admin_id`) REFERENCES `admins` (`id`) ON DELETE CASCADE
|
CONSTRAINT `fk_watch_probe_admin` FOREIGN KEY (`admin_id`) REFERENCES `admins` (`id`) ON DELETE CASCADE
|
||||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4""",
|
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4""",
|
||||||
"ALTER TABLE `watch_probe_records` ADD COLUMN `processes_json` JSON NULL COMMENT 'Top5 processes snapshot'",
|
"ALTER TABLE `watch_probe_records` ADD COLUMN `processes_json` JSON NULL COMMENT 'Top5 processes snapshot'",
|
||||||
|
"ALTER TABLE `watch_pins` ADD COLUMN `monitoring_enabled` TINYINT(1) NOT NULL DEFAULT 1 COMMENT '是否开启实时探针'",
|
||||||
|
"ALTER TABLE `watch_pins` ADD COLUMN `ttl_hours` INT NOT NULL DEFAULT 8 COMMENT '监测窗口时长(小时) 8|24'",
|
||||||
]
|
]
|
||||||
async with AsyncSessionLocal() as session:
|
async with AsyncSessionLocal() as session:
|
||||||
for sql in migrations:
|
for sql in migrations:
|
||||||
|
|||||||
@@ -5,16 +5,24 @@ from __future__ import annotations
|
|||||||
from datetime import datetime, timedelta, timezone
|
from datetime import datetime, timedelta, timezone
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from sqlalchemy import delete, func, select
|
from sqlalchemy import delete, func, or_, select
|
||||||
from sqlalchemy.ext.asyncio import AsyncSession
|
from sqlalchemy.ext.asyncio import AsyncSession
|
||||||
|
|
||||||
from server.domain.models import Server, WatchPin, WatchPinSession, WatchProbeRecord
|
from server.domain.models import Server, WatchPin, WatchPinSession, WatchProbeRecord
|
||||||
|
from server.utils.watch_metrics import normalize_watch_ttl_hours
|
||||||
|
|
||||||
|
|
||||||
def _utc_naive_now() -> datetime:
|
def _utc_naive_now() -> datetime:
|
||||||
return datetime.now(timezone.utc).replace(tzinfo=None)
|
return datetime.now(timezone.utc).replace(tzinfo=None)
|
||||||
|
|
||||||
|
|
||||||
|
def _pin_still_occupied(pin: WatchPin, now: datetime) -> bool:
|
||||||
|
"""Pin occupies a slot until manual remove or TTL expiry while monitoring is on."""
|
||||||
|
if not pin.monitoring_enabled:
|
||||||
|
return True
|
||||||
|
return pin.expires_at > now
|
||||||
|
|
||||||
|
|
||||||
class WatchRepositoryImpl:
|
class WatchRepositoryImpl:
|
||||||
def __init__(self, session: AsyncSession):
|
def __init__(self, session: AsyncSession):
|
||||||
self.session = session
|
self.session = session
|
||||||
@@ -25,12 +33,17 @@ class WatchRepositoryImpl:
|
|||||||
select(WatchPin, WatchPinSession, Server.name)
|
select(WatchPin, WatchPinSession, Server.name)
|
||||||
.join(WatchPinSession, WatchPin.session_id == WatchPinSession.id)
|
.join(WatchPinSession, WatchPin.session_id == WatchPinSession.id)
|
||||||
.join(Server, WatchPin.server_id == Server.id)
|
.join(Server, WatchPin.server_id == Server.id)
|
||||||
.where(WatchPin.admin_id == admin_id, WatchPin.expires_at > now)
|
.where(
|
||||||
|
WatchPin.admin_id == admin_id,
|
||||||
|
or_(WatchPin.monitoring_enabled.is_(False), WatchPin.expires_at > now),
|
||||||
|
)
|
||||||
.order_by(WatchPin.slot_index)
|
.order_by(WatchPin.slot_index)
|
||||||
)
|
)
|
||||||
rows = (await self.session.execute(q)).all()
|
rows = (await self.session.execute(q)).all()
|
||||||
out: list[dict[str, Any]] = []
|
out: list[dict[str, Any]] = []
|
||||||
for pin, sess, server_name in rows:
|
for pin, sess, server_name in rows:
|
||||||
|
if not _pin_still_occupied(pin, now):
|
||||||
|
continue
|
||||||
out.append(
|
out.append(
|
||||||
{
|
{
|
||||||
"pin_id": pin.id,
|
"pin_id": pin.id,
|
||||||
@@ -40,46 +53,55 @@ class WatchRepositoryImpl:
|
|||||||
"session_id": sess.id,
|
"session_id": sess.id,
|
||||||
"pinned_at": pin.pinned_at,
|
"pinned_at": pin.pinned_at,
|
||||||
"expires_at": pin.expires_at,
|
"expires_at": pin.expires_at,
|
||||||
|
"monitoring_enabled": bool(pin.monitoring_enabled),
|
||||||
|
"ttl_hours": int(pin.ttl_hours or 8),
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
return out
|
return out
|
||||||
|
|
||||||
async def list_active_pins_all(self) -> list[dict[str, Any]]:
|
async def list_active_pins_all(self, *, monitoring_only: bool = False) -> list[dict[str, Any]]:
|
||||||
now = _utc_naive_now()
|
now = _utc_naive_now()
|
||||||
q = (
|
q = (
|
||||||
select(WatchPin, WatchPinSession)
|
select(WatchPin, WatchPinSession)
|
||||||
.join(WatchPinSession, WatchPin.session_id == WatchPinSession.id)
|
.join(WatchPinSession, WatchPin.session_id == WatchPinSession.id)
|
||||||
.where(WatchPin.expires_at > now)
|
|
||||||
)
|
)
|
||||||
rows = (await self.session.execute(q)).all()
|
rows = (await self.session.execute(q)).all()
|
||||||
return [
|
out: list[dict[str, Any]] = []
|
||||||
{
|
for pin, sess in rows:
|
||||||
"pin_id": pin.id,
|
if not _pin_still_occupied(pin, now):
|
||||||
"admin_id": pin.admin_id,
|
continue
|
||||||
"slot_index": pin.slot_index,
|
if monitoring_only and not pin.monitoring_enabled:
|
||||||
"server_id": pin.server_id,
|
continue
|
||||||
"session_id": sess.id,
|
out.append(
|
||||||
}
|
{
|
||||||
for pin, sess in rows
|
"pin_id": pin.id,
|
||||||
]
|
"admin_id": pin.admin_id,
|
||||||
|
"slot_index": pin.slot_index,
|
||||||
|
"server_id": pin.server_id,
|
||||||
|
"session_id": sess.id,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return out
|
||||||
|
|
||||||
async def get_pin_by_slot(self, admin_id: int, slot_index: int) -> WatchPin | None:
|
async def get_pin_by_slot(self, admin_id: int, slot_index: int) -> WatchPin | None:
|
||||||
now = _utc_naive_now()
|
|
||||||
q = select(WatchPin).where(
|
q = select(WatchPin).where(
|
||||||
WatchPin.admin_id == admin_id,
|
WatchPin.admin_id == admin_id,
|
||||||
WatchPin.slot_index == slot_index,
|
WatchPin.slot_index == slot_index,
|
||||||
WatchPin.expires_at > now,
|
|
||||||
)
|
)
|
||||||
return (await self.session.execute(q)).scalar_one_or_none()
|
pin = (await self.session.execute(q)).scalar_one_or_none()
|
||||||
|
if pin and _pin_still_occupied(pin, _utc_naive_now()):
|
||||||
|
return pin
|
||||||
|
return None
|
||||||
|
|
||||||
async def get_pin_by_server(self, admin_id: int, server_id: int) -> WatchPin | None:
|
async def get_pin_by_server(self, admin_id: int, server_id: int) -> WatchPin | None:
|
||||||
now = _utc_naive_now()
|
|
||||||
q = select(WatchPin).where(
|
q = select(WatchPin).where(
|
||||||
WatchPin.admin_id == admin_id,
|
WatchPin.admin_id == admin_id,
|
||||||
WatchPin.server_id == server_id,
|
WatchPin.server_id == server_id,
|
||||||
WatchPin.expires_at > now,
|
|
||||||
)
|
)
|
||||||
return (await self.session.execute(q)).scalar_one_or_none()
|
pin = (await self.session.execute(q)).scalar_one_or_none()
|
||||||
|
if pin and _pin_still_occupied(pin, _utc_naive_now()):
|
||||||
|
return pin
|
||||||
|
return None
|
||||||
|
|
||||||
async def create_pin(
|
async def create_pin(
|
||||||
self,
|
self,
|
||||||
@@ -90,7 +112,8 @@ class WatchRepositoryImpl:
|
|||||||
ttl_hours: int = 8,
|
ttl_hours: int = 8,
|
||||||
) -> dict[str, Any]:
|
) -> dict[str, Any]:
|
||||||
now = _utc_naive_now()
|
now = _utc_naive_now()
|
||||||
expires = now + timedelta(hours=ttl_hours)
|
hours = normalize_watch_ttl_hours(ttl_hours)
|
||||||
|
expires = now + timedelta(hours=hours)
|
||||||
sess = WatchPinSession(
|
sess = WatchPinSession(
|
||||||
admin_id=admin_id,
|
admin_id=admin_id,
|
||||||
server_id=server_id,
|
server_id=server_id,
|
||||||
@@ -106,6 +129,8 @@ class WatchRepositoryImpl:
|
|||||||
server_id=server_id,
|
server_id=server_id,
|
||||||
pinned_at=now,
|
pinned_at=now,
|
||||||
expires_at=expires,
|
expires_at=expires,
|
||||||
|
monitoring_enabled=True,
|
||||||
|
ttl_hours=hours,
|
||||||
)
|
)
|
||||||
self.session.add(pin)
|
self.session.add(pin)
|
||||||
await self.session.flush()
|
await self.session.flush()
|
||||||
@@ -116,6 +141,7 @@ class WatchRepositoryImpl:
|
|||||||
"server_id": server_id,
|
"server_id": server_id,
|
||||||
"pinned_at": now,
|
"pinned_at": now,
|
||||||
"expires_at": expires,
|
"expires_at": expires,
|
||||||
|
"ttl_hours": hours,
|
||||||
}
|
}
|
||||||
|
|
||||||
async def end_session(self, session_id: int, reason: str) -> None:
|
async def end_session(self, session_id: int, reason: str) -> None:
|
||||||
@@ -159,34 +185,101 @@ class WatchRepositoryImpl:
|
|||||||
ttl_hours=ttl_hours,
|
ttl_hours=ttl_hours,
|
||||||
)
|
)
|
||||||
|
|
||||||
async def reset_pin_ttl(self, admin_id: int, server_id: int, ttl_hours: int = 8) -> dict[str, Any]:
|
async def reset_pin_ttl(self, admin_id: int, server_id: int, ttl_hours: int | None = None) -> dict[str, Any]:
|
||||||
"""Re-pin same server: end old session and start fresh 8h window."""
|
"""Re-pin same server: end old session and start fresh monitoring window."""
|
||||||
pin = await self.get_pin_by_server(admin_id, server_id)
|
pin = await self.get_pin_by_server(admin_id, server_id)
|
||||||
if not pin:
|
if not pin:
|
||||||
raise ValueError("pin not found")
|
raise ValueError("pin not found")
|
||||||
slot_index = pin.slot_index
|
slot_index = pin.slot_index
|
||||||
|
hours = normalize_watch_ttl_hours(ttl_hours if ttl_hours is not None else int(pin.ttl_hours or 8))
|
||||||
await self.end_session(pin.session_id, "replaced")
|
await self.end_session(pin.session_id, "replaced")
|
||||||
await self.delete_pin(pin.id)
|
await self.delete_pin(pin.id)
|
||||||
return await self.create_pin(
|
return await self.create_pin(
|
||||||
admin_id=admin_id,
|
admin_id=admin_id,
|
||||||
server_id=server_id,
|
server_id=server_id,
|
||||||
slot_index=slot_index,
|
slot_index=slot_index,
|
||||||
ttl_hours=ttl_hours,
|
ttl_hours=hours,
|
||||||
)
|
)
|
||||||
|
|
||||||
async def remove_pin_by_slot(self, admin_id: int, slot_index: int) -> WatchPin | None:
|
async def remove_pin_by_slot(self, admin_id: int, slot_index: int) -> WatchPin | None:
|
||||||
return await self.delete_pin_by_slot(admin_id, slot_index)
|
return await self.delete_pin_by_slot(admin_id, slot_index)
|
||||||
|
|
||||||
|
async def count_monitoring_pins_for_server(self, server_id: int) -> int:
|
||||||
|
q = select(func.count()).select_from(WatchPin).where(
|
||||||
|
WatchPin.server_id == server_id,
|
||||||
|
WatchPin.monitoring_enabled.is_(True),
|
||||||
|
)
|
||||||
|
return int((await self.session.execute(q)).scalar_one() or 0)
|
||||||
|
|
||||||
|
async def _ensure_open_session(self, pin: WatchPin) -> None:
|
||||||
|
q = select(WatchPinSession).where(WatchPinSession.id == pin.session_id)
|
||||||
|
sess = (await self.session.execute(q)).scalar_one_or_none()
|
||||||
|
if sess and sess.ended_at is None:
|
||||||
|
return
|
||||||
|
now = _utc_naive_now()
|
||||||
|
new_sess = WatchPinSession(
|
||||||
|
admin_id=pin.admin_id,
|
||||||
|
server_id=pin.server_id,
|
||||||
|
slot_index=pin.slot_index,
|
||||||
|
started_at=now,
|
||||||
|
)
|
||||||
|
self.session.add(new_sess)
|
||||||
|
await self.session.flush()
|
||||||
|
pin.session_id = new_sess.id
|
||||||
|
|
||||||
|
async def set_monitoring_enabled(
|
||||||
|
self,
|
||||||
|
admin_id: int,
|
||||||
|
slot_index: int,
|
||||||
|
enabled: bool,
|
||||||
|
*,
|
||||||
|
ttl_hours: int | None = None,
|
||||||
|
) -> WatchPin | None:
|
||||||
|
pin = await self.get_pin_by_slot(admin_id, slot_index)
|
||||||
|
if not pin:
|
||||||
|
return None
|
||||||
|
if ttl_hours is not None:
|
||||||
|
pin.ttl_hours = normalize_watch_ttl_hours(ttl_hours)
|
||||||
|
if enabled:
|
||||||
|
await self._ensure_open_session(pin)
|
||||||
|
hours = normalize_watch_ttl_hours(int(pin.ttl_hours or 8))
|
||||||
|
pin.ttl_hours = hours
|
||||||
|
pin.monitoring_enabled = True
|
||||||
|
pin.expires_at = _utc_naive_now() + timedelta(hours=hours)
|
||||||
|
else:
|
||||||
|
pin.monitoring_enabled = False
|
||||||
|
await self.session.flush()
|
||||||
|
return pin
|
||||||
|
|
||||||
|
async def set_pin_ttl_hours(
|
||||||
|
self,
|
||||||
|
admin_id: int,
|
||||||
|
slot_index: int,
|
||||||
|
ttl_hours: int,
|
||||||
|
) -> WatchPin | None:
|
||||||
|
pin = await self.get_pin_by_slot(admin_id, slot_index)
|
||||||
|
if not pin:
|
||||||
|
return None
|
||||||
|
hours = normalize_watch_ttl_hours(ttl_hours)
|
||||||
|
pin.ttl_hours = hours
|
||||||
|
if pin.monitoring_enabled:
|
||||||
|
pin.expires_at = _utc_naive_now() + timedelta(hours=hours)
|
||||||
|
await self.session.flush()
|
||||||
|
return pin
|
||||||
|
|
||||||
async def expire_due_pins(self) -> list[int]:
|
async def expire_due_pins(self) -> list[int]:
|
||||||
now = _utc_naive_now()
|
now = _utc_naive_now()
|
||||||
q = select(WatchPin).where(WatchPin.expires_at <= now)
|
q = select(WatchPin).where(
|
||||||
|
WatchPin.expires_at <= now,
|
||||||
|
WatchPin.monitoring_enabled.is_(True),
|
||||||
|
)
|
||||||
pins = (await self.session.execute(q)).scalars().all()
|
pins = (await self.session.execute(q)).scalars().all()
|
||||||
session_ids: list[int] = []
|
server_ids: list[int] = []
|
||||||
for pin in pins:
|
for pin in pins:
|
||||||
await self.end_session(pin.session_id, "expired")
|
await self.end_session(pin.session_id, "expired")
|
||||||
session_ids.append(pin.session_id)
|
pin.monitoring_enabled = False
|
||||||
await self.delete_pin(pin.id)
|
server_ids.append(pin.server_id)
|
||||||
return session_ids
|
return server_ids
|
||||||
|
|
||||||
async def insert_probe_record(self, **fields: Any) -> WatchProbeRecord:
|
async def insert_probe_record(self, **fields: Any) -> WatchProbeRecord:
|
||||||
row = WatchProbeRecord(**fields)
|
row = WatchProbeRecord(**fields)
|
||||||
|
|||||||
@@ -1,13 +1,12 @@
|
|||||||
"""SSH-based remote health / metrics probe (no Agent HTTP port required)."""
|
"""SSH-based remote health / metrics probe (no Agent HTTP port required)."""
|
||||||
|
|
||||||
import json
|
import json
|
||||||
import logging
|
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from server.domain.models import Server
|
from server.domain.models import Server
|
||||||
from server.infrastructure.ssh.asyncssh_pool import exec_ssh_command
|
from server.infrastructure.ssh.asyncssh_pool import exec_ssh_command
|
||||||
|
|
||||||
logger = logging.getLogger("nexus.remote_probe")
|
from server.utils.watch_probe_errors import watch_probe_error_zh
|
||||||
|
|
||||||
# Emits one JSON line: {"status":"healthy","system_info":{...}}
|
# Emits one JSON line: {"status":"healthy","system_info":{...}}
|
||||||
SSH_HEALTH_METRICS_CMD = r"""python3 -c "
|
SSH_HEALTH_METRICS_CMD = r"""python3 -c "
|
||||||
@@ -49,26 +48,45 @@ print(json.dumps({'status': 'healthy', 'system_info': info}))
|
|||||||
|
|
||||||
|
|
||||||
SSH_WATCH_METRICS_CMD = r"""python3 -c "
|
SSH_WATCH_METRICS_CMD = r"""python3 -c "
|
||||||
import json, platform
|
import json
|
||||||
try:
|
try:
|
||||||
import psutil
|
import psutil
|
||||||
except ImportError:
|
except ImportError:
|
||||||
print(json.dumps({'status':'error','error':'psutil missing'}))
|
print(json.dumps({'status':'error','error':'psutil missing'}))
|
||||||
raise SystemExit(0)
|
raise SystemExit(0)
|
||||||
load = psutil.getloadavg()
|
try:
|
||||||
net = psutil.net_io_counters()
|
load = list(psutil.getloadavg())
|
||||||
disk_io = psutil.disk_io_counters()
|
except (AttributeError, OSError, NotImplementedError):
|
||||||
|
load = [0.0, 0.0, 0.0]
|
||||||
|
try:
|
||||||
|
net = psutil.net_io_counters()
|
||||||
|
except Exception:
|
||||||
|
net = None
|
||||||
|
try:
|
||||||
|
disk_io = psutil.disk_io_counters()
|
||||||
|
except Exception:
|
||||||
|
disk_io = None
|
||||||
|
try:
|
||||||
|
cpu = psutil.cpu_percent(interval=0.3)
|
||||||
|
mem = psutil.virtual_memory().percent
|
||||||
|
disk = round(psutil.disk_usage('/').percent, 1)
|
||||||
|
except Exception as exc:
|
||||||
|
print(json.dumps({'status':'error','error': str(exc)[:200]}))
|
||||||
|
raise SystemExit(0)
|
||||||
out = {
|
out = {
|
||||||
'status': 'ok',
|
'status': 'ok',
|
||||||
'cpu_usage': psutil.cpu_percent(interval=0.3),
|
'cpu_usage': cpu,
|
||||||
'mem_usage': psutil.virtual_memory().percent,
|
'mem_usage': mem,
|
||||||
'disk_usage': round(psutil.disk_usage('/').percent, 1),
|
'disk_usage': disk,
|
||||||
'disk_mount': '/',
|
'disk_mount': '/',
|
||||||
'load_avg': list(load),
|
'load_avg': load,
|
||||||
'net_io': {'bytes_sent': net.bytes_sent, 'bytes_recv': net.bytes_recv},
|
'net_io': {
|
||||||
|
'bytes_sent': int(net.bytes_sent) if net else 0,
|
||||||
|
'bytes_recv': int(net.bytes_recv) if net else 0,
|
||||||
|
},
|
||||||
'disk_io': {
|
'disk_io': {
|
||||||
'read_bytes': disk_io.read_bytes if disk_io else 0,
|
'read_bytes': int(disk_io.read_bytes) if disk_io else 0,
|
||||||
'write_bytes': disk_io.write_bytes if disk_io else 0,
|
'write_bytes': int(disk_io.write_bytes) if disk_io else 0,
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
print(json.dumps(out))
|
print(json.dumps(out))
|
||||||
@@ -79,20 +97,39 @@ try:
|
|||||||
except ImportError:
|
except ImportError:
|
||||||
print(json.dumps({'status':'error','error':'psutil missing'}))
|
print(json.dumps({'status':'error','error':'psutil missing'}))
|
||||||
raise SystemExit(0)
|
raise SystemExit(0)
|
||||||
load = psutil.getloadavg()
|
try:
|
||||||
net = psutil.net_io_counters()
|
load = list(psutil.getloadavg())
|
||||||
disk_io = psutil.disk_io_counters()
|
except (AttributeError, OSError, NotImplementedError):
|
||||||
|
load = [0.0, 0.0, 0.0]
|
||||||
|
try:
|
||||||
|
net = psutil.net_io_counters()
|
||||||
|
except Exception:
|
||||||
|
net = None
|
||||||
|
try:
|
||||||
|
disk_io = psutil.disk_io_counters()
|
||||||
|
except Exception:
|
||||||
|
disk_io = None
|
||||||
|
try:
|
||||||
|
cpu = psutil.cpu_percent(interval=0.3)
|
||||||
|
mem = psutil.virtual_memory().percent
|
||||||
|
disk = round(psutil.disk_usage('/').percent, 1)
|
||||||
|
except Exception as exc:
|
||||||
|
print(json.dumps({'status':'error','error': str(exc)[:200]}))
|
||||||
|
raise SystemExit(0)
|
||||||
out = {
|
out = {
|
||||||
'status': 'ok',
|
'status': 'ok',
|
||||||
'cpu_usage': psutil.cpu_percent(interval=0.3),
|
'cpu_usage': cpu,
|
||||||
'mem_usage': psutil.virtual_memory().percent,
|
'mem_usage': mem,
|
||||||
'disk_usage': round(psutil.disk_usage('/').percent, 1),
|
'disk_usage': disk,
|
||||||
'disk_mount': '/',
|
'disk_mount': '/',
|
||||||
'load_avg': list(load),
|
'load_avg': load,
|
||||||
'net_io': {'bytes_sent': net.bytes_sent, 'bytes_recv': net.bytes_recv},
|
'net_io': {
|
||||||
|
'bytes_sent': int(net.bytes_sent) if net else 0,
|
||||||
|
'bytes_recv': int(net.bytes_recv) if net else 0,
|
||||||
|
},
|
||||||
'disk_io': {
|
'disk_io': {
|
||||||
'read_bytes': disk_io.read_bytes if disk_io else 0,
|
'read_bytes': int(disk_io.read_bytes) if disk_io else 0,
|
||||||
'write_bytes': disk_io.write_bytes if disk_io else 0,
|
'write_bytes': int(disk_io.write_bytes) if disk_io else 0,
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
print(json.dumps(out))
|
print(json.dumps(out))
|
||||||
@@ -125,16 +162,32 @@ print(json.dumps({'status':'ok','top_processes': procs[:5]}))
|
|||||||
|
|
||||||
|
|
||||||
def _parse_json_stdout(stdout: str) -> dict[str, Any] | None:
|
def _parse_json_stdout(stdout: str) -> dict[str, Any] | None:
|
||||||
for line in reversed(stdout.strip().splitlines()):
|
text = (stdout or "").strip()
|
||||||
|
if not text:
|
||||||
|
return None
|
||||||
|
candidates: list[str] = []
|
||||||
|
for line in reversed(text.splitlines()):
|
||||||
line = line.strip()
|
line = line.strip()
|
||||||
if not line.startswith("{"):
|
if not line:
|
||||||
continue
|
continue
|
||||||
|
start = line.find("{")
|
||||||
|
if start >= 0:
|
||||||
|
candidates.append(line[start:])
|
||||||
|
elif line.startswith("{"):
|
||||||
|
candidates.append(line)
|
||||||
|
if text.startswith("{"):
|
||||||
|
candidates.append(text)
|
||||||
|
seen: set[str] = set()
|
||||||
|
for chunk in candidates:
|
||||||
|
if chunk in seen:
|
||||||
|
continue
|
||||||
|
seen.add(chunk)
|
||||||
try:
|
try:
|
||||||
data = json.loads(line)
|
data = json.loads(chunk)
|
||||||
if isinstance(data, dict) and data.get("status"):
|
|
||||||
return data
|
|
||||||
except json.JSONDecodeError:
|
except json.JSONDecodeError:
|
||||||
continue
|
continue
|
||||||
|
if isinstance(data, dict) and data.get("status"):
|
||||||
|
return data
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
@@ -169,17 +222,39 @@ async def ssh_watch_probe(server: Server, timeout: int = 8) -> dict[str, Any]:
|
|||||||
result = await exec_ssh_command(server, SSH_WATCH_METRICS_CMD, timeout=timeout)
|
result = await exec_ssh_command(server, SSH_WATCH_METRICS_CMD, timeout=timeout)
|
||||||
duration_ms = int((time.monotonic() - started) * 1000)
|
duration_ms = int((time.monotonic() - started) * 1000)
|
||||||
if result.get("exit_code", -1) != 0:
|
if result.get("exit_code", -1) != 0:
|
||||||
err = (result.get("stderr") or result.get("stdout") or "SSH watch probe failed")[:300]
|
err_raw = (result.get("stderr") or result.get("stdout") or "SSH watch probe failed")[:300]
|
||||||
if "timed out" in err.lower() or duration_ms >= timeout * 1000 - 50:
|
if "timed out" in err_raw.lower() or duration_ms >= timeout * 1000 - 50:
|
||||||
return {"ok": False, "probe_status": "ssh_timeout", "error": err, "duration_ms": duration_ms}
|
return {
|
||||||
if "auth" in err.lower() or "permission denied" in err.lower():
|
"ok": False,
|
||||||
return {"ok": False, "probe_status": "ssh_auth", "error": err, "duration_ms": duration_ms}
|
"probe_status": "ssh_timeout",
|
||||||
return {"ok": False, "probe_status": "offline", "error": err, "duration_ms": duration_ms}
|
"error": watch_probe_error_zh(err_raw, status="ssh_timeout"),
|
||||||
|
"duration_ms": duration_ms,
|
||||||
|
}
|
||||||
|
if "auth" in err_raw.lower() or "permission denied" in err_raw.lower():
|
||||||
|
return {
|
||||||
|
"ok": False,
|
||||||
|
"probe_status": "ssh_auth",
|
||||||
|
"error": watch_probe_error_zh(err_raw, status="ssh_auth"),
|
||||||
|
"duration_ms": duration_ms,
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
"ok": False,
|
||||||
|
"probe_status": "offline",
|
||||||
|
"error": watch_probe_error_zh(err_raw, status="offline"),
|
||||||
|
"duration_ms": duration_ms,
|
||||||
|
}
|
||||||
|
|
||||||
parsed = _parse_json_stdout(result.get("stdout", ""))
|
parsed = _parse_json_stdout(result.get("stdout", ""))
|
||||||
if not parsed or parsed.get("status") != "ok":
|
if not parsed or parsed.get("status") != "ok":
|
||||||
err = (parsed or {}).get("error") or "parse_error"
|
raw = (result.get("stdout") or "").strip().replace("\n", " ")[:200]
|
||||||
return {"ok": False, "probe_status": "parse_error", "error": str(err)[:255], "duration_ms": duration_ms}
|
err_key = (parsed or {}).get("error") or "parse_error"
|
||||||
|
err_combined = f"parse_error: {raw}" if err_key == "parse_error" and raw else str(err_key)
|
||||||
|
return {
|
||||||
|
"ok": False,
|
||||||
|
"probe_status": "parse_error",
|
||||||
|
"error": watch_probe_error_zh(err_combined, status="parse_error"),
|
||||||
|
"duration_ms": duration_ms,
|
||||||
|
}
|
||||||
parsed["duration_ms"] = duration_ms
|
parsed["duration_ms"] = duration_ms
|
||||||
return {"ok": True, "payload": parsed, "duration_ms": duration_ms}
|
return {"ok": True, "payload": parsed, "duration_ms": duration_ms}
|
||||||
|
|
||||||
|
|||||||
@@ -10,6 +10,15 @@ from server.utils.fleet_metrics import _metric_from_system_info, _safe_pct
|
|||||||
|
|
||||||
WATCH_MAX_SLOTS = 4
|
WATCH_MAX_SLOTS = 4
|
||||||
WATCH_PIN_TTL_HOURS = 8
|
WATCH_PIN_TTL_HOURS = 8
|
||||||
|
WATCH_PIN_TTL_EXTENDED_HOURS = 24
|
||||||
|
WATCH_PIN_TTL_ALLOWED = frozenset({WATCH_PIN_TTL_HOURS, WATCH_PIN_TTL_EXTENDED_HOURS})
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_watch_ttl_hours(hours: int) -> int:
|
||||||
|
"""Only 8h (default) or 24h (explicit opt-in) are allowed."""
|
||||||
|
if hours == WATCH_PIN_TTL_EXTENDED_HOURS:
|
||||||
|
return WATCH_PIN_TTL_EXTENDED_HOURS
|
||||||
|
return WATCH_PIN_TTL_HOURS
|
||||||
WATCH_PROBE_INTERVAL_SEC = 5
|
WATCH_PROBE_INTERVAL_SEC = 5
|
||||||
WATCH_REDIS_FRESH_SEC = 90
|
WATCH_REDIS_FRESH_SEC = 90
|
||||||
WATCH_PROBE_TIMEOUT_SEC = 8
|
WATCH_PROBE_TIMEOUT_SEC = 8
|
||||||
@@ -84,6 +93,13 @@ def redis_sample_has_io(sample: WatchProbeSample | None) -> bool:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def redis_sample_has_basic_metrics(sample: WatchProbeSample | None) -> bool:
|
||||||
|
"""True when Agent heartbeat has CPU or memory — IO optional."""
|
||||||
|
if sample is None or sample.probe_status != PROBE_OK:
|
||||||
|
return False
|
||||||
|
return sample.cpu_pct is not None or sample.mem_pct is not None
|
||||||
|
|
||||||
|
|
||||||
def sanitize_processes(raw: Any) -> list[dict[str, Any]] | None:
|
def sanitize_processes(raw: Any) -> list[dict[str, Any]] | None:
|
||||||
if not isinstance(raw, list):
|
if not isinstance(raw, list):
|
||||||
return None
|
return None
|
||||||
|
|||||||
@@ -0,0 +1,61 @@
|
|||||||
|
"""Chinese user-facing messages for watch probe failures."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
_EXACT: dict[str, str] = {
|
||||||
|
"parse_error": "探针返回数据无法解析",
|
||||||
|
"psutil missing": "子机未安装 psutil(请执行 pip install psutil)",
|
||||||
|
"ssh_probe_failed": "SSH 探针脚本执行失败",
|
||||||
|
"SSH watch probe failed": "SSH 监测探针失败",
|
||||||
|
"SSH probe failed": "SSH 探针失败",
|
||||||
|
"timeout": "连接超时",
|
||||||
|
"探针失败": "探针失败",
|
||||||
|
"无 Agent 且 SSH 探针失败": "无 Agent 且 SSH 探针失败",
|
||||||
|
"服务器不存在": "服务器不存在",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def watch_probe_error_zh(error: str | None, *, status: str | None = None) -> str | None:
|
||||||
|
"""Map probe error text to Chinese for UI / audit display."""
|
||||||
|
if not error:
|
||||||
|
if status == "parse_error":
|
||||||
|
return "探针返回数据无法解析"
|
||||||
|
if status == "ssh_timeout":
|
||||||
|
return "SSH 连接超时"
|
||||||
|
if status == "ssh_auth":
|
||||||
|
return "SSH 认证失败"
|
||||||
|
if status == "no_cred":
|
||||||
|
return "需要 Agent 或 SSH 凭据"
|
||||||
|
if status == "redis_stale":
|
||||||
|
return "Agent 心跳已过期"
|
||||||
|
if status == "offline":
|
||||||
|
return "目标服务器离线或不可达"
|
||||||
|
return None
|
||||||
|
|
||||||
|
text = error.strip()
|
||||||
|
if text in _EXACT:
|
||||||
|
return _EXACT[text]
|
||||||
|
|
||||||
|
lower = text.lower()
|
||||||
|
if lower.startswith("parse_error"):
|
||||||
|
detail = text.split(":", 1)[1].strip() if ":" in text else ""
|
||||||
|
if detail in _EXACT:
|
||||||
|
return f"探针返回数据无法解析:{_EXACT[detail]}"
|
||||||
|
if detail:
|
||||||
|
return f"探针返回数据无法解析:{detail}"
|
||||||
|
return "探针返回数据无法解析"
|
||||||
|
|
||||||
|
if "timed out" in lower or lower.endswith("timeout"):
|
||||||
|
return f"SSH 连接超时:{text}"
|
||||||
|
if "permission denied" in lower or "authentication failed" in lower:
|
||||||
|
return f"SSH 认证失败:{text}"
|
||||||
|
if "psutil missing" in lower:
|
||||||
|
return "子机未安装 psutil(请执行 pip install psutil)"
|
||||||
|
if "ssh_probe_failed" in lower or "ssh watch probe failed" in lower:
|
||||||
|
return "SSH 探针脚本执行失败"
|
||||||
|
|
||||||
|
# Already Chinese or mixed — return as-is if contains CJK
|
||||||
|
if any("\u4e00" <= ch <= "\u9fff" for ch in text):
|
||||||
|
return text
|
||||||
|
|
||||||
|
return f"探针异常:{text}"
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
"""Watch slot Redis snapshots — cleared when no active 5s monitoring remains."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from server.infrastructure.redis.client import get_redis
|
||||||
|
|
||||||
|
REDIS_WATCH_LIVE_PREFIX = "watch:live:"
|
||||||
|
REDIS_WATCH_PROC_PREFIX = "watch:proc:"
|
||||||
|
REDIS_WATCH_LAST_PREFIX = "watch:last:"
|
||||||
|
|
||||||
|
|
||||||
|
async def clear_watch_redis_snapshot(server_id: int) -> None:
|
||||||
|
"""Remove 5s watch sliding window; Agent continues normal 60s heartbeat."""
|
||||||
|
redis = get_redis()
|
||||||
|
await redis.delete(
|
||||||
|
f"{REDIS_WATCH_LIVE_PREFIX}{server_id}",
|
||||||
|
f"{REDIS_WATCH_PROC_PREFIX}{server_id}",
|
||||||
|
f"{REDIS_WATCH_LAST_PREFIX}{server_id}",
|
||||||
|
)
|
||||||
@@ -10,6 +10,7 @@ from server.utils.watch_metrics import (
|
|||||||
merge_redis_and_ssh,
|
merge_redis_and_ssh,
|
||||||
parse_ssh_watch_payload,
|
parse_ssh_watch_payload,
|
||||||
redis_sample_has_io,
|
redis_sample_has_io,
|
||||||
|
redis_sample_has_basic_metrics,
|
||||||
sanitize_processes,
|
sanitize_processes,
|
||||||
PROBE_OK,
|
PROBE_OK,
|
||||||
)
|
)
|
||||||
@@ -44,6 +45,22 @@ def test_compute_rates_no_prev():
|
|||||||
assert up is None and down is None and dr is None and dw is None
|
assert up is None and down is None and dr is None and dw is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_redis_sample_has_basic_metrics():
|
||||||
|
basic = WatchProbeSample(probe_status=PROBE_OK, source="redis", cpu_pct=10)
|
||||||
|
assert redis_sample_has_basic_metrics(basic) is True
|
||||||
|
assert redis_sample_has_io(basic) is False
|
||||||
|
assert redis_sample_has_basic_metrics(None) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_json_stdout_with_prefix_noise():
|
||||||
|
from server.infrastructure.ssh.remote_probe import _parse_json_stdout
|
||||||
|
|
||||||
|
stdout = "Warning: something\nOK {\"status\": \"ok\", \"cpu_usage\": 1.0}\n"
|
||||||
|
parsed = _parse_json_stdout(stdout)
|
||||||
|
assert parsed is not None
|
||||||
|
assert parsed.get("status") == "ok"
|
||||||
|
|
||||||
|
|
||||||
def test_redis_sample_has_io():
|
def test_redis_sample_has_io():
|
||||||
complete = WatchProbeSample(
|
complete = WatchProbeSample(
|
||||||
probe_status=PROBE_OK,
|
probe_status=PROBE_OK,
|
||||||
|
|||||||
@@ -17,7 +17,9 @@ def mock_watch_redis(monkeypatch):
|
|||||||
mock_redis.lindex = AsyncMock(return_value=None)
|
mock_redis.lindex = AsyncMock(return_value=None)
|
||||||
mock_redis.lrange = AsyncMock(return_value=[])
|
mock_redis.lrange = AsyncMock(return_value=[])
|
||||||
mock_redis.get = AsyncMock(return_value=None)
|
mock_redis.get = AsyncMock(return_value=None)
|
||||||
|
mock_redis.delete = AsyncMock(return_value=1)
|
||||||
monkeypatch.setattr("server.application.services.watch_service.get_redis", lambda: mock_redis)
|
monkeypatch.setattr("server.application.services.watch_service.get_redis", lambda: mock_redis)
|
||||||
|
monkeypatch.setattr("server.utils.watch_state.get_redis", lambda: mock_redis)
|
||||||
return mock_redis
|
return mock_redis
|
||||||
|
|
||||||
|
|
||||||
@@ -85,3 +87,154 @@ async def test_watch_probe_record_on_failure(db_session):
|
|||||||
await db_session.commit()
|
await db_session.commit()
|
||||||
assert row.probe_status == "ssh_timeout"
|
assert row.probe_status == "ssh_timeout"
|
||||||
assert row.cpu_pct is None
|
assert row.cpu_pct is None
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_watch_pin_monitoring_toggle(db_session, test_admin, mock_watch_redis):
|
||||||
|
from server.application.services.watch_service import WatchService
|
||||||
|
|
||||||
|
server = Server(name="watch-toggle", domain="wt.example.com", port=22, agent_version="1.0")
|
||||||
|
db_session.add(server)
|
||||||
|
await db_session.commit()
|
||||||
|
|
||||||
|
svc = WatchService(db_session)
|
||||||
|
admin = test_admin["admin"]
|
||||||
|
|
||||||
|
await svc.pin_server(admin=admin, server_id=server.id)
|
||||||
|
listing = await svc.list_slots(admin.id)
|
||||||
|
assert listing["slots"][0]["monitoring_enabled"] is True
|
||||||
|
|
||||||
|
paused = await svc.set_slot_monitoring(
|
||||||
|
admin=admin,
|
||||||
|
slot_index=0,
|
||||||
|
monitoring_enabled=False,
|
||||||
|
)
|
||||||
|
assert paused["slots"][0]["monitoring_enabled"] is False
|
||||||
|
assert paused["slots"][0]["remaining_sec"] is None
|
||||||
|
|
||||||
|
resumed = await svc.set_slot_monitoring(
|
||||||
|
admin=admin,
|
||||||
|
slot_index=0,
|
||||||
|
monitoring_enabled=True,
|
||||||
|
)
|
||||||
|
assert resumed["slots"][0]["monitoring_enabled"] is True
|
||||||
|
assert resumed["slots"][0]["remaining_sec"] is not None
|
||||||
|
assert resumed["slots"][0]["remaining_sec"] > 0
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_watch_pin_ttl_hours(db_session, test_admin, mock_watch_redis):
|
||||||
|
from server.application.services.watch_service import WatchService
|
||||||
|
from server.utils.watch_metrics import WATCH_PIN_TTL_EXTENDED_HOURS, WATCH_PIN_TTL_HOURS
|
||||||
|
|
||||||
|
server = Server(name="watch-ttl", domain="wt2.example.com", port=22, agent_version="1.0")
|
||||||
|
db_session.add(server)
|
||||||
|
await db_session.commit()
|
||||||
|
|
||||||
|
svc = WatchService(db_session)
|
||||||
|
admin = test_admin["admin"]
|
||||||
|
|
||||||
|
await svc.pin_server(admin=admin, server_id=server.id, ttl_hours=WATCH_PIN_TTL_HOURS)
|
||||||
|
listing = await svc.list_slots(admin.id)
|
||||||
|
assert listing["slots"][0]["ttl_hours"] == WATCH_PIN_TTL_HOURS
|
||||||
|
|
||||||
|
extended = await svc.set_slot_ttl(
|
||||||
|
admin=admin,
|
||||||
|
slot_index=0,
|
||||||
|
ttl_hours=WATCH_PIN_TTL_EXTENDED_HOURS,
|
||||||
|
)
|
||||||
|
assert extended["slots"][0]["ttl_hours"] == WATCH_PIN_TTL_EXTENDED_HOURS
|
||||||
|
assert extended["slots"][0]["remaining_sec"] >= WATCH_PIN_TTL_EXTENDED_HOURS * 3600 - 5
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_watch_pin_ttl_expiry_list_syncs_before_display(db_session, test_admin, mock_watch_redis):
|
||||||
|
"""TTL 到期后、探针循环未跑 expire 前,list_slots 应同步为暂停态而非空槽。"""
|
||||||
|
from datetime import timedelta
|
||||||
|
|
||||||
|
from server.infrastructure.database.watch_repo import WatchRepositoryImpl, _utc_naive_now
|
||||||
|
from server.application.services.watch_service import WatchService
|
||||||
|
|
||||||
|
server = Server(name="watch-expire-list", domain="wel.example.com", port=22, agent_version="1.0")
|
||||||
|
db_session.add(server)
|
||||||
|
await db_session.commit()
|
||||||
|
|
||||||
|
svc = WatchService(db_session)
|
||||||
|
admin = test_admin["admin"]
|
||||||
|
await svc.pin_server(admin=admin, server_id=server.id)
|
||||||
|
|
||||||
|
repo = WatchRepositoryImpl(db_session)
|
||||||
|
pin = await repo.get_pin_by_slot(admin.id, 0)
|
||||||
|
assert pin is not None
|
||||||
|
pin.expires_at = _utc_naive_now() - timedelta(seconds=1)
|
||||||
|
await db_session.commit()
|
||||||
|
|
||||||
|
# 不手动调用 expire_due_pins — 模拟探针循环尚未处理的窗口
|
||||||
|
listing = await svc.list_slots(admin.id)
|
||||||
|
assert listing["slots"][0]["empty"] is False
|
||||||
|
assert listing["slots"][0]["monitoring_enabled"] is False
|
||||||
|
assert listing["slots"][0]["remaining_sec"] is None
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_watch_pin_ttl_expiry_pin_server_no_duplicate(db_session, test_admin, mock_watch_redis):
|
||||||
|
"""到期 ghost pin 不应导致同槽 create_pin 唯一约束冲突。"""
|
||||||
|
from datetime import timedelta
|
||||||
|
|
||||||
|
from server.infrastructure.database.watch_repo import WatchRepositoryImpl, _utc_naive_now
|
||||||
|
from server.application.services.watch_service import WatchService
|
||||||
|
|
||||||
|
server_a = Server(name="watch-exp-a", domain="wea.example.com", port=22, agent_version="1.0")
|
||||||
|
server_b = Server(name="watch-exp-b", domain="web.example.com", port=22, agent_version="1.0")
|
||||||
|
db_session.add_all([server_a, server_b])
|
||||||
|
await db_session.commit()
|
||||||
|
|
||||||
|
svc = WatchService(db_session)
|
||||||
|
admin = test_admin["admin"]
|
||||||
|
await svc.pin_server(admin=admin, server_id=server_a.id, slot_index=0)
|
||||||
|
|
||||||
|
repo = WatchRepositoryImpl(db_session)
|
||||||
|
pin = await repo.get_pin_by_slot(admin.id, 0)
|
||||||
|
assert pin is not None
|
||||||
|
pin.expires_at = _utc_naive_now() - timedelta(seconds=1)
|
||||||
|
await db_session.commit()
|
||||||
|
|
||||||
|
# 替换槽位服务器 — 应先 finalize 再 replace,不能 IntegrityError
|
||||||
|
result = await svc.pin_server(admin=admin, server_id=server_b.id, slot_index=0)
|
||||||
|
assert result["slots"][0]["empty"] is False
|
||||||
|
assert result["slots"][0]["server_id"] == server_b.id
|
||||||
|
assert result["slots"][0]["monitoring_enabled"] is True
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_watch_pin_ttl_expiry_pauses_not_deletes(db_session, test_admin, mock_watch_redis):
|
||||||
|
from datetime import timedelta
|
||||||
|
|
||||||
|
from server.infrastructure.database.watch_repo import WatchRepositoryImpl, _utc_naive_now
|
||||||
|
from server.application.services.watch_service import WatchService
|
||||||
|
|
||||||
|
server = Server(name="watch-expire", domain="we.example.com", port=22, agent_version="1.0")
|
||||||
|
db_session.add(server)
|
||||||
|
await db_session.commit()
|
||||||
|
|
||||||
|
svc = WatchService(db_session)
|
||||||
|
admin = test_admin["admin"]
|
||||||
|
await svc.pin_server(admin=admin, server_id=server.id)
|
||||||
|
|
||||||
|
repo = WatchRepositoryImpl(db_session)
|
||||||
|
pin = await repo.get_pin_by_slot(admin.id, 0)
|
||||||
|
assert pin is not None
|
||||||
|
pin.expires_at = _utc_naive_now() - timedelta(seconds=1)
|
||||||
|
await db_session.commit()
|
||||||
|
|
||||||
|
expired = await repo.expire_due_pins()
|
||||||
|
assert server.id in expired
|
||||||
|
await db_session.commit()
|
||||||
|
|
||||||
|
still = await repo.get_pin_by_slot(admin.id, 0)
|
||||||
|
assert still is not None
|
||||||
|
assert still.monitoring_enabled is False
|
||||||
|
|
||||||
|
listing = await svc.list_slots(admin.id)
|
||||||
|
assert listing["slots"][0]["empty"] is False
|
||||||
|
assert listing["slots"][0]["monitoring_enabled"] is False
|
||||||
|
|||||||
@@ -0,0 +1,23 @@
|
|||||||
|
"""Tests for watch probe error localization."""
|
||||||
|
|
||||||
|
from server.utils.watch_probe_errors import watch_probe_error_zh
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_error_zh():
|
||||||
|
assert watch_probe_error_zh("parse_error") == "探针返回数据无法解析"
|
||||||
|
assert watch_probe_error_zh("psutil missing") == "子机未安装 psutil(请执行 pip install psutil)"
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_error_with_detail():
|
||||||
|
out = watch_probe_error_zh("parse_error: psutil missing")
|
||||||
|
assert "探针返回数据无法解析" in out
|
||||||
|
assert "psutil" in out
|
||||||
|
|
||||||
|
|
||||||
|
def test_timeout_zh():
|
||||||
|
out = watch_probe_error_zh("Connection timed out during banner exchange")
|
||||||
|
assert out.startswith("SSH 连接超时")
|
||||||
|
|
||||||
|
|
||||||
|
def test_status_only_fallback():
|
||||||
|
assert watch_probe_error_zh(None, status="ssh_timeout") == "SSH 连接超时"
|
||||||
Reference in New Issue
Block a user