15 Commits

Author SHA1 Message Date
Maxim
72869dcf5f Separate network faults from software drops and fix CPU temperature history 2026-10-07 22:02:08 +03:00
Maxim
1296d052a5 Run guest updates outside agent sandbox 2026-08-21 22:09:35 +03:00
Maxim
f764195ba5 Clarify process CPU and improve Docker actions 2026-08-21 21:58:00 +03:00
Maxim
f6498a2fc1 Add backup maintenance and guest update controls 2026-08-21 21:15:52 +03:00
Maxim
714c29c449 Fix NVMe self-tests and mobile layout 2026-08-21 19:17:30 +03:00
Maxim
21b4f528e0 Add agent operations history and manual links 2026-08-07 22:24:44 +03:00
Maxim
c50b4cf986 Improve agent problem visibility 2026-08-07 21:36:38 +03:00
Maxim
338d53cb2f Add guest root cause diagnostics 2026-08-07 21:23:05 +03:00
Maxim
a2525b53f0 Classify and enrich guest services 2026-08-06 23:24:43 +03:00
Maxim
331f0ee39a Add detailed guest agent diagnostics 2026-08-06 21:12:42 +03:00
Maxim
12aba383f1 Add secure automatic agent updates 2026-08-06 21:01:54 +03:00
Maxim
c618eb018c Expand guest agent system monitoring 2026-08-06 20:48:27 +03:00
Maxim
f1ff15e344 Add read-only guest monitoring agents 2026-08-06 20:24:13 +03:00
Maxim
5401b5b0f2 Add maintenance windows for planned work 2026-08-06 19:35:33 +03:00
Maxim
c01b4b6630 Fix service monitor connection leak 2026-08-05 20:04:29 +03:00
30 changed files with 3465 additions and 137 deletions

107
README.md
View File

@@ -32,7 +32,7 @@ ssh root@192.168.0.10
```bash
apt-get update
apt-get install -y curl ca-certificates tar smartmontools iproute2 iputils-ping
apt-get install -y curl ca-certificates tar smartmontools iproute2 iputils-ping ethtool
cd /tmp
curl -fL \
@@ -322,6 +322,78 @@ apcaccess status
- `GET /api/events` — поток метрик SSE с интервалом две секунды.
- `GET/POST/PUT/DELETE /api/services` — управление проверками сервисов.
- `GET /api/services/history` — семидневная история доступности и отклика.
- `GET/PUT/DELETE /api/settings/maintenance` — состояние, запуск и досрочное завершение плановых работ.
- `GET/DELETE /api/agents` — подключённые гостевые агенты.
- `POST /api/agents/enrollment-token` — одноразовый токен регистрации на 15 минут.
## Maintenance-режим
Во вкладке `Настройки → Maintenance` можно на заданное время отключить все
алерты либо выбрать конкретные сервисы и VM/LXC. Подавленные события не
отправляются по почте и не отображаются как активные, но сохраняются в истории
с причиной плановых работ. После окончания окна обычный мониторинг включается
автоматически.
## Агенты VM и LXC
Read-only агент показывает состояние Linux внутри VM/LXC: CPU, load average,
память, корневой раздел, uptime, ОС, ядро, IP-адреса, systemd-службы, доступные
APT-обновления и необходимость перезагрузки. Для Docker отображаются контейнеры,
health-check, число перезапусков, CPU и память. Агент сам подключается к Dashboard
каждые 10 секунд; открывать входящий порт внутри гостевой системы не нужно.
В подробностях агента systemd-службы разделяются на системные, установленные
дополнительно и локальные unit-файлы. Для каждой службы показываются пакет и путь
unit-файла, время запуска, число перезапусков, последний exit code, память и
накопленное CPU-время, когда соответствующий accounting доступен в systemd.
Расширенная диагностика агента включает последние сообщения проблемных служб,
`Requires`/`After`, PID, команду запуска и слушающие порты; выявляет частые
рестарты, OOM-kill и процессы в D-state. Отдельно отображаются top процессов по
CPU и RAM с накопленным I/O, swap, PSI pressure, все постоянные файловые системы,
их заполнение и использование inode.
Панель «Диагностика homelab» и карта сопоставляют известные IP и порты с VM/LXC,
агентом, systemd-службой или Docker-контейнером и настроенной HTTP/TCP-проверкой.
Связи по PID и порту считаются подтверждёнными; совпадения только по названию
помечаются как предположение.
Для сложных reverse proxy и Docker-сетей в управлении сервисами можно сохранить
ручную связь `сервис → агент/VMID → systemd/Docker/процесс → порт`. Она имеет
приоритет над автоматическим определением и используется для группировки
связанных уведомлений по первопричине.
Dashboard хранит семь дней истории CPU, RAM, swap, PSI и файловых систем каждого
агента. Из подробностей агента доступны только заранее разрешённые действия:
перезапуск известной systemd-службы или Docker-контейнера, `reset-failed` и
обновление списка пакетов. Агент не принимает произвольные shell-команды; каждое
действие требует подтверждения в браузере и сохраняется в журнале управления.
Откройте `Настройки → Агенты`, укажите адрес Dashboard, доступный из VM/LXC, и
VMID. Кнопка создаст готовую команду установки. Одноразовый токен действует 15
минут и после регистрации заменяется индивидуальным секретом агента.
После однократной установки агент обновляется автоматически вслед за Dashboard.
Центральный сервер отдаёт агенту собственный бинарник и SHA-256; агент проверяет
контрольную сумму, атомарно заменяет файл и перезапускается через systemd. Если
агент был удалён из Dashboard, повторная команда с новым токеном автоматически
выполнит новую регистрацию.
В первой версии агент не выполняет команды и не перезапускает службы или
контейнеры. Для чтения Docker ему требуется доступ к `/var/run/docker.sock`,
поэтому служба устанавливается от root. Сам Dashboard пока не имеет
авторизации: используйте локальную сеть или HTTPS через доверенный reverse
proxy и не публикуйте агентские API напрямую в интернет.
Полностью удалить агент из VM/LXC:
```bash
curl -fsSL https://git.myown.center/maxim/ProxmoxDash/raw/branch/main/scripts/uninstall-agent.sh | sh
```
Оформление, пороги, почта и обновления Dashboard находятся в обычных настройках.
Рабочие инструменты — сервисы, агенты, плановые работы и UPS-сценарий — вынесены
в отдельный раздел `Управление` в шапке Dashboard.
## Сборка
@@ -329,3 +401,36 @@ apcaccess status
go test ./...
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o proxmox-cpu-dashboard .
```
## Сетевые ошибки и температура CPU (v0.19.3)
RX/TX drops отображаются отдельно как информационные счётчики. Их рост сам по
себе не вызывает аварийный алерт: в том числе LLDP на Linux bridge. Настройки
Keenetic, LLDP и group_fwd_mask менять не требуется. Для VM учитываются tap,
для LXC — veth; fwbr/fwpr/fwln остаются в группе интерфейсов хоста.
Новые rx_errors/tx_errors, CRC/frame/FIFO/missed, carrier/link-down вызывают
критический алерт. Sysfs позволяет отделить rx_missed_errors от RX drops,
которые в /proc/net/dev объединены. Для Atlantic дополнительно читаются
InErrors, InDroppedDma, AllocFails, SkbAllocFails и Restarts через ethtool -S.
Установите `ethtool` на существующем хосте для этой дополнительной диагностики.
При отсутствии утилиты остаются стандартные счётчики sysfs. Счётчики ошибок
могут пересекаться: их сумма — прирост счётчиков, а не число уникальных пакетов.
После перезапуска и сброса счётчиков старые значения не создают новые алерты.
Карточка CPU и новые точки истории показывают максимум доступных датчиков CPU,
а не среднее Tctl/Tdie/CCD или ядер. На AMD Tctl может включать управляющее
смещение; подписи и значения всех датчиков доступны в подробностях CPU.
История снимается раз в минуту, а на диапазоне 7 дней усредняется по 10 минутам,
поэтому короткие пики между измерениями не гарантированно попадут на график.
Старые точки до обновления остаются средними: исходные значения датчиков в них
не сохранены. Масштаб графика расширяется для значений выше 100 °C.
Релиз Linux amd64 собирается с номером версии и без CGO:
```bash
./scripts/build-release.sh v0.19.3
```
Скрипт запускает тесты и создаёт в dist бинарник, SHA-256 и установочный архив,
совместимые со встроенным обновлением из Gitea Releases.

1044
agent.go Normal file

File diff suppressed because it is too large Load Diff

346
agent_actions.go Normal file
View File

@@ -0,0 +1,346 @@
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net"
"net/http"
"os/exec"
"path/filepath"
"regexp"
"strings"
"time"
)
type AgentCommand struct {
ID int64 `json:"id"`
Action string `json:"action"`
Target string `json:"target,omitempty"`
}
type AgentActionResult struct {
ID int64 `json:"id"`
Success bool `json:"success"`
Message string `json:"message"`
}
type AgentAction struct {
ID int64 `json:"id"`
AgentID string `json:"agentId"`
AgentName string `json:"agentName"`
Action string `json:"action"`
Target string `json:"target"`
RequestedAt int64 `json:"requestedAt"`
StartedAt *int64 `json:"startedAt,omitempty"`
CompletedAt *int64 `json:"completedAt,omitempty"`
Status string `json:"status"`
Result string `json:"result"`
}
var safeUnitName = regexp.MustCompile(`^[A-Za-z0-9_.@:-]+\.service$`)
var safeContainerName = regexp.MustCompile(`^[A-Za-z0-9_.-]{1,128}$`)
func (s *Store) QueueAgentAction(agentID, action, target string) (AgentAction, error) {
action, target = strings.TrimSpace(action), strings.TrimSpace(target)
var raw, name string
if err := s.db.QueryRow(`SELECT report_json,name FROM agents WHERE id=?`, agentID).Scan(&raw, &name); err != nil {
return AgentAction{}, fmt.Errorf("агент не найден")
}
var report AgentReport
_ = json.Unmarshal([]byte(raw), &report)
valid := false
switch action {
case "systemd-restart", "systemd-reset-failed":
if safeUnitName.MatchString(target) {
for _, service := range report.Services {
if service.Name == target {
valid = true
break
}
}
}
case "docker-restart", "docker-check", "docker-update", "docker-remove":
if safeContainerName.MatchString(target) {
for _, container := range report.Containers {
if (container.ID == target || container.Name == target) && (action != "docker-remove" || container.State != "running") {
valid = true
break
}
}
}
case "updates-refresh", "system-update", "system-reboot":
valid = target == ""
}
if !valid {
return AgentAction{}, fmt.Errorf("действие или цель не входят в разрешённый список")
}
now := time.Now().Unix()
result, err := s.db.Exec(`INSERT INTO agent_actions(agent_id,action,target,requested_at,status) VALUES(?,?,?,?, 'queued')`, agentID, action, target, now)
if err != nil {
return AgentAction{}, err
}
id, _ := result.LastInsertId()
return AgentAction{ID: id, AgentID: agentID, AgentName: name, Action: action, Target: target, RequestedAt: now, Status: "queued"}, nil
}
func (s *Store) DispatchAgentActions(agentID string) ([]AgentCommand, error) {
tx, err := s.db.Begin()
if err != nil {
return nil, err
}
defer tx.Rollback()
// Обновление пакетов и Docker-образов может быть долгим. Не выдаём одну
// команду повторно, пока предыдущий запуск ещё имеет шанс завершиться.
_, _ = tx.Exec(`UPDATE agent_actions SET status='queued',started_at=NULL WHERE agent_id=? AND status='running' AND started_at<?`, agentID, time.Now().Add(-45*time.Minute).Unix())
rows, err := tx.Query(`SELECT id,action,target FROM agent_actions WHERE agent_id=? AND status='queued' ORDER BY id LIMIT 3`, agentID)
if err != nil {
return nil, err
}
var commands []AgentCommand
for rows.Next() {
var c AgentCommand
if err = rows.Scan(&c.ID, &c.Action, &c.Target); err != nil {
rows.Close()
return nil, err
}
commands = append(commands, c)
}
rows.Close()
now := time.Now().Unix()
for _, c := range commands {
if _, err = tx.Exec(`UPDATE agent_actions SET status='running',started_at=? WHERE id=?`, now, c.ID); err != nil {
return nil, err
}
}
return commands, tx.Commit()
}
func (s *Store) CompleteAgentActions(agentID string, results []AgentActionResult) {
now := time.Now().Unix()
for _, result := range results {
status := "failed"
if result.Success {
status = "completed"
}
message := result.Message
if len(message) > 4000 {
message = message[:4000]
}
_, _ = s.db.Exec(`UPDATE agent_actions SET status=?,result=?,completed_at=? WHERE id=? AND agent_id=? AND status='running'`, status, message, now, result.ID, agentID)
}
}
func (s *Store) AgentActions(limit int) ([]AgentAction, error) {
if limit < 1 || limit > 200 {
limit = 100
}
rows, err := s.db.Query(`SELECT a.id,a.agent_id,g.name,a.action,a.target,a.requested_at,a.started_at,a.completed_at,a.status,a.result FROM agent_actions a JOIN agents g ON g.id=a.agent_id ORDER BY a.id DESC LIMIT ?`, limit)
if err != nil {
return nil, err
}
defer rows.Close()
var out []AgentAction
for rows.Next() {
var v AgentAction
if err = rows.Scan(&v.ID, &v.AgentID, &v.AgentName, &v.Action, &v.Target, &v.RequestedAt, &v.StartedAt, &v.CompletedAt, &v.Status, &v.Result); err != nil {
return nil, err
}
out = append(out, v)
}
return out, rows.Err()
}
func executeAgentCommand(command AgentCommand) AgentActionResult {
result := AgentActionResult{ID: command.ID}
var output string
switch command.Action {
case "systemd-restart":
output = runAgentCombinedCommand(30*time.Second, "systemctl", "restart", command.Target)
case "systemd-reset-failed":
output = runAgentCombinedCommand(15*time.Second, "systemctl", "reset-failed", command.Target)
case "docker-restart":
output = restartAgentContainer(command.Target)
case "docker-update":
output = updateAgentContainer(command.Target, true)
case "docker-check":
output = updateAgentContainer(command.Target, false)
case "docker-remove":
output = removeAgentContainer(command.Target)
case "system-update":
output = updateAgentSystem()
case "system-reboot":
output = scheduleAgentReboot()
case "updates-refresh":
guestInventory.Lock()
guestInventory.value.CollectedAt = time.Time{}
guestInventory.Unlock()
result.Success = true
result.Message = "Проверка обновлений поставлена в очередь"
return result
default:
result.Message = "Команда не разрешена"
return result
}
result.Success = !strings.HasPrefix(output, "ERROR: ")
result.Message = strings.TrimSpace(output)
if result.Message == "" && result.Success {
result.Message = "Выполнено успешно"
}
return result
}
func removeAgentContainer(target string) string {
transport := &http.Transport{DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
return (&net.Dialer{}).DialContext(ctx, "unix", "/var/run/docker.sock")
}}
defer transport.CloseIdleConnections()
request, _ := http.NewRequest(http.MethodDelete, "http://docker/containers/"+target+"?v=false&force=false", nil)
response, err := (&http.Client{Transport: transport, Timeout: 20 * time.Second}).Do(request)
if err != nil {
return "ERROR: " + err.Error()
}
defer response.Body.Close()
if response.StatusCode != http.StatusNoContent {
message, _ := io.ReadAll(io.LimitReader(response.Body, 2048))
return fmt.Sprintf("ERROR: Docker вернул %s: %s", response.Status, strings.TrimSpace(string(message)))
}
return "Остановленный контейнер удалён; volumes сохранены"
}
func updateAgentSystem() string {
aptGet, err := exec.LookPath("apt-get")
if err != nil {
return "ERROR: apt-get не найден; обновление поддерживается для Debian/Ubuntu"
}
if output := runPrivilegedTransient(5*time.Minute, "update-lists", aptGet, "update"); strings.HasPrefix(output, "ERROR: ") {
return explainSystemUpdateError(output)
}
message := runPrivilegedTransient(30*time.Minute, "upgrade", aptGet, "-y", "upgrade")
if strings.HasPrefix(message, "ERROR: ") {
return explainSystemUpdateError(message)
}
guestInventory.Lock()
guestInventory.value.CollectedAt = time.Time{}
guestInventory.Unlock()
return "Обновления системы установлены; список пакетов будет обновлён в следующем отчёте агента"
}
func explainSystemUpdateError(message string) string {
if strings.Contains(strings.ToLower(message), "read-only file system") {
return "ERROR: файловая система агента доступна только для чтения. Обновите Dashboard и агент до v0.19.2 или новее, затем повторите операцию."
}
if len(message) > 1800 {
message = message[len(message)-1800:]
}
return message
}
func runPrivilegedTransient(timeout time.Duration, suffix, name string, args ...string) string {
systemdRun, err := exec.LookPath("systemd-run")
if err != nil {
return "ERROR: systemd-run не найден; невозможно безопасно выйти из read-only sandbox агента"
}
unit := fmt.Sprintf("proxmox-dashboard-%s-%d", suffix, time.Now().UnixNano())
commandArgs := []string{"--quiet", "--wait", "--collect", "--pipe", "--unit=" + unit, "--setenv=DEBIAN_FRONTEND=noninteractive", name}
commandArgs = append(commandArgs, args...)
return runAgentCombinedCommand(timeout, systemdRun, commandArgs...)
}
func scheduleAgentReboot() string {
systemdRun, err := exec.LookPath("systemd-run")
if err != nil {
return "ERROR: systemd-run не найден"
}
systemctl, err := exec.LookPath("systemctl")
if err != nil {
return "ERROR: systemctl не найден"
}
unit := fmt.Sprintf("proxmox-dashboard-reboot-%d", time.Now().UnixNano())
output := runAgentCombinedCommand(15*time.Second, systemdRun, "--quiet", "--collect", "--unit="+unit, "--on-active=20s", systemctl, "reboot")
if strings.HasPrefix(output, "ERROR: ") {
return output
}
return "Перезагрузка запланирована через 20 секунд; кратковременный offline агента ожидаем"
}
func updateAgentContainer(target string, recreate bool) string {
transport := &http.Transport{DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
return (&net.Dialer{}).DialContext(ctx, "unix", "/var/run/docker.sock")
}}
defer transport.CloseIdleConnections()
client := &http.Client{Transport: transport, Timeout: 10 * time.Second}
response, err := client.Get("http://docker/containers/" + target + "/json")
if err != nil {
return "ERROR: " + err.Error()
}
defer response.Body.Close()
var detail struct {
Config struct {
Labels map[string]string `json:"Labels"`
} `json:"Config"`
}
if response.StatusCode != http.StatusOK || json.NewDecoder(response.Body).Decode(&detail) != nil {
return "ERROR: не удалось получить Compose-конфигурацию контейнера"
}
labels := detail.Config.Labels
service := labels["com.docker.compose.service"]
workdir := labels["com.docker.compose.project.working_dir"]
files := strings.Split(labels["com.docker.compose.project.config_files"], ",")
if service == "" || workdir == "" || len(files) == 0 || !filepath.IsAbs(workdir) || !filepath.IsAbs(files[0]) {
return "ERROR: безопасное обновление доступно только для Docker Compose"
}
if _, err = exec.LookPath("docker"); err != nil {
return "ERROR: Docker CLI не найден"
}
commands := [][]string{{"compose", "-f", files[0], "pull", service}}
if recreate {
commands = append(commands, []string{"compose", "-f", files[0], "up", "-d", "--no-deps", service})
}
for _, args := range commands {
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Minute)
command := exec.CommandContext(ctx, "docker", args...)
command.Dir = workdir
output, commandErr := command.CombinedOutput()
cancel()
if commandErr != nil {
return "ERROR: " + commandErr.Error() + ": " + strings.TrimSpace(string(output))
}
}
if recreate {
return "Образ загружен, Compose-контейнер пересоздан"
}
return "Проверка завершена: образ загружен; если digest изменился, карточка покажет доступное обновление"
}
func runAgentCombinedCommand(timeout time.Duration, name string, args ...string) string {
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
output, err := exec.CommandContext(ctx, name, args...).CombinedOutput()
message := strings.TrimSpace(string(output))
if err != nil {
return "ERROR: " + err.Error() + ": " + message
}
return message
}
func restartAgentContainer(target string) string {
transport := &http.Transport{DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
return (&net.Dialer{}).DialContext(ctx, "unix", "/var/run/docker.sock")
}}
defer transport.CloseIdleConnections()
client := &http.Client{Transport: transport, Timeout: 35 * time.Second}
request, _ := http.NewRequest(http.MethodPost, "http://docker/containers/"+target+"/restart?t=10", bytes.NewReader(nil))
response, err := client.Do(request)
if err != nil {
return "ERROR: " + err.Error()
}
defer response.Body.Close()
body, _ := io.ReadAll(io.LimitReader(response.Body, 4096))
if response.StatusCode >= 300 {
return "ERROR: Docker вернул " + response.Status + ": " + strings.TrimSpace(string(body))
}
return "Контейнер перезапущен"
}

77
agent_actions_test.go Normal file
View File

@@ -0,0 +1,77 @@
package main
import (
"encoding/json"
"strings"
"testing"
"time"
)
func TestAgentActionAllowlistAndLifecycle(t *testing.T) {
store, err := openStore(":memory:")
if err != nil {
t.Fatal(err)
}
defer store.Close()
report, _ := json.Marshal(AgentReport{Services: []AgentService{{Name: "gitea.service"}}, Containers: []AgentContainer{{ID: "abc123", Name: "gitea", State: "running"}, {ID: "old123", Name: "old", State: "exited"}}})
_, err = store.db.Exec(`INSERT INTO agents(id,name,secret_hash,enrolled_at,last_seen,report_json) VALUES('a','agent','x',?,?,?)`, time.Now().Unix(), time.Now().Unix(), string(report))
if err != nil {
t.Fatal(err)
}
if _, err = store.QueueAgentAction("a", "systemd-restart", "../../bad.service"); err == nil {
t.Fatal("unsafe unit accepted")
}
if _, err = store.QueueAgentAction("a", "docker-update", "gitea"); err != nil {
t.Fatalf("known Docker container rejected: %v", err)
}
if _, err = store.QueueAgentAction("a", "system-update", ""); err != nil {
t.Fatalf("system update rejected: %v", err)
}
if _, err = store.QueueAgentAction("a", "docker-remove", "gitea"); err == nil {
t.Fatal("running/unknown-state container accepted for removal")
}
action, err := store.QueueAgentAction("a", "docker-remove", "old")
if err != nil {
t.Fatalf("stopped container rejected for removal: %v", err)
}
commands, err := store.DispatchAgentActions("a")
if err != nil || len(commands) != 3 || commands[2].ID != action.ID {
t.Fatalf("commands=%#v err=%v", commands, err)
}
store.CompleteAgentActions("a", []AgentActionResult{{ID: action.ID, Success: true, Message: "ok"}})
items, err := store.AgentActions(10)
if err != nil || len(items) != 3 || items[0].Status != "completed" {
t.Fatalf("items=%#v err=%v", items, err)
}
if _, err = store.QueueAgentAction("a", "system-reboot", ""); err != nil {
t.Fatalf("system reboot rejected: %v", err)
}
if _, err = store.QueueAgentAction("a", "system-reboot", "now"); err == nil {
t.Fatal("system reboot with unexpected target accepted")
}
}
func TestManualServiceLinkGroupsOfflineAlerts(t *testing.T) {
metrics := dashboardMetrics{Agents: []ManagedAgent{{ID: "a", Name: "app", VMID: 105, Online: false}}, ServiceLinks: []ServiceLink{{ServiceID: 7, AgentID: "a", VMID: 105}}, Services: ServicesMetrics{Services: []ServiceStatus{{MonitoredService: MonitoredService{ID: 7, Name: "Gitea"}, Endpoints: []ServiceEndpointStatus{{Kind: "local", Up: false}}}}}}
alerts := []Alert{{ID: "agent-offline-a-105"}, {ID: "service-7-local"}, {ID: "other"}}
grouped := groupLinkedAlerts(metrics, alerts)
if len(grouped) != 2 {
t.Fatalf("grouped=%#v", grouped)
}
found := false
for _, alert := range grouped {
if alert.ID == "group-agent-offline-a-105" {
found = true
}
}
if !found {
t.Fatalf("group alert missing: %#v", grouped)
}
}
func TestExplainReadOnlySystemUpdateError(t *testing.T) {
message := explainSystemUpdateError("ERROR: apt failed: Read-only file system")
if !strings.Contains(message, "v0.19.2") {
t.Fatalf("read-only failure is not actionable: %s", message)
}
}

336
agent_diagnostics.go Normal file
View File

@@ -0,0 +1,336 @@
package main
import (
"bufio"
"encoding/json"
"net"
"os"
"path/filepath"
"regexp"
"runtime"
"sort"
"strconv"
"strings"
"sync"
"syscall"
"time"
)
type AgentLog struct {
Time int64 `json:"time"`
Message string `json:"message"`
}
type AgentProcess struct {
PID int `json:"pid"`
Name string `json:"name"`
State string `json:"state"`
CPUPercent float64 `json:"cpuPercent"`
CPUCores float64 `json:"cpuCores"`
HostPercent float64 `json:"hostPercent"`
MemoryBytes uint64 `json:"memoryBytes"`
ReadBytes uint64 `json:"readBytes"`
WriteBytes uint64 `json:"writeBytes"`
Elapsed int64 `json:"elapsedSeconds"`
Command string `json:"command"`
}
type AgentFilesystem struct {
Device string `json:"device"`
Mountpoint string `json:"mountpoint"`
Type string `json:"type"`
TotalBytes uint64 `json:"totalBytes"`
UsedBytes uint64 `json:"usedBytes"`
UsedPercent float64 `json:"usedPercent"`
InodesTotal uint64 `json:"inodesTotal"`
InodesUsed uint64 `json:"inodesUsed"`
InodesPercent float64 `json:"inodesPercent"`
}
type AgentListenPort struct {
Protocol string `json:"protocol"`
Address string `json:"address"`
Port int `json:"port"`
PID int `json:"pid,omitempty"`
Process string `json:"process,omitempty"`
}
type AgentPressureValue struct {
Avg10 float64 `json:"avg10"`
Avg60 float64 `json:"avg60"`
Avg300 float64 `json:"avg300"`
}
type AgentPressure struct {
CPU AgentPressureValue `json:"cpu"`
Memory AgentPressureValue `json:"memory"`
IO AgentPressureValue `json:"io"`
}
var agentOOMCache struct {
sync.Mutex
value []AgentLog
at time.Time
}
func compactSystemdExec(value string) string {
if pathAt := strings.Index(value, "path="); pathAt >= 0 {
value = value[pathAt+5:]
}
if end := strings.IndexAny(value, " ;}"); end > 0 {
value = value[:end]
}
if len(value) > 240 {
value = value[:240]
}
return strings.TrimSpace(value)
}
func limitedFields(value string, limit int) []string {
fields := strings.Fields(value)
if len(fields) > limit {
fields = fields[:limit]
}
return fields
}
func appendUniqueInt(items []int, value int) []int {
for _, item := range items {
if item == value {
return items
}
}
return append(items, value)
}
func collectAgentJournal(unit string, limit int, since string) []AgentLog {
args := []string{"--no-pager", "-o", "json", "--reverse", "-n", strconv.Itoa(limit), "--since", since}
if unit != "" {
args = append(args, "-u", unit)
}
output := runAgentCommand(6*time.Second, "journalctl", args...)
result := []AgentLog{}
for _, line := range strings.Split(output, "\n") {
var entry struct {
Message any `json:"MESSAGE"`
Timestamp string `json:"__REALTIME_TIMESTAMP"`
}
if json.Unmarshal([]byte(line), &entry) != nil {
continue
}
message, ok := entry.Message.(string)
if !ok || strings.TrimSpace(message) == "" {
continue
}
microseconds, _ := strconv.ParseInt(entry.Timestamp, 10, 64)
result = append(result, AgentLog{Time: microseconds / 1_000_000, Message: message})
}
return result
}
func cachedAgentOOMKills() []AgentLog {
agentOOMCache.Lock()
defer agentOOMCache.Unlock()
if time.Since(agentOOMCache.at) < time.Minute {
return agentOOMCache.value
}
output := runAgentCommand(6*time.Second, "journalctl", "-k", "--since", "24 hours ago", "--reverse", "--no-pager", "-o", "json", "-g", "Out of memory|Killed process")
result := []AgentLog{}
for _, line := range strings.Split(output, "\n") {
var entry struct {
Message any `json:"MESSAGE"`
Timestamp string `json:"__REALTIME_TIMESTAMP"`
}
if json.Unmarshal([]byte(line), &entry) != nil {
continue
}
message, ok := entry.Message.(string)
if !ok {
continue
}
microseconds, _ := strconv.ParseInt(entry.Timestamp, 10, 64)
result = append(result, AgentLog{Time: microseconds / 1_000_000, Message: message})
if len(result) >= 20 {
break
}
}
agentOOMCache.value, agentOOMCache.at = result, time.Now()
return result
}
func collectAgentProcesses() []AgentProcess {
result := []AgentProcess{}
seen := map[int]bool{}
for _, sortBy := range []string{"-pcpu", "-rss"} {
output := runAgentCommand(5*time.Second, "ps", "-eo", "pid=,comm=,state=,pcpu=,rss=,etimes=,args=", "--sort="+sortBy)
added := 0
for _, line := range strings.Split(output, "\n") {
fields := strings.Fields(line)
if len(fields) < 7 {
continue
}
pid, err := strconv.Atoi(fields[0])
if err != nil || seen[pid] {
continue
}
seen[pid] = true
cpu, _ := strconv.ParseFloat(fields[3], 64)
rss, _ := strconv.ParseUint(fields[4], 10, 64)
elapsed, _ := strconv.ParseInt(fields[5], 10, 64)
cpuCount := runtime.NumCPU()
if cpuCount < 1 {
cpuCount = 1
}
item := AgentProcess{PID: pid, Name: fields[1], State: fields[2], CPUPercent: cpu, CPUCores: cpu / 100, HostPercent: cpu / float64(cpuCount), MemoryBytes: rss * 1024, Elapsed: elapsed, Command: strings.Join(fields[6:], " ")}
item.ReadBytes, item.WriteBytes = readAgentProcessIO(pid)
result = append(result, item)
added++
if added >= 12 || len(result) >= 24 {
break
}
}
}
return result
}
func readAgentProcessIO(pid int) (uint64, uint64) {
data, err := os.ReadFile(filepath.Join("/proc", strconv.Itoa(pid), "io"))
if err != nil {
return 0, 0
}
values := map[string]uint64{}
for _, line := range strings.Split(string(data), "\n") {
fields := strings.Fields(line)
if len(fields) != 2 {
continue
}
values[strings.TrimSuffix(fields[0], ":")], _ = strconv.ParseUint(fields[1], 10, 64)
}
return values["read_bytes"], values["write_bytes"]
}
func collectAgentFilesystems() []AgentFilesystem {
file, err := os.Open("/proc/self/mounts")
if err != nil {
return nil
}
defer file.Close()
excluded := map[string]bool{"proc": true, "sysfs": true, "devtmpfs": true, "devpts": true, "tmpfs": true, "cgroup": true, "cgroup2": true, "overlay": true, "squashfs": true, "nsfs": true, "tracefs": true, "debugfs": true, "securityfs": true, "pstore": true, "mqueue": true, "hugetlbfs": true, "fusectl": true, "configfs": true, "autofs": true, "rpc_pipefs": true}
seen := map[string]bool{}
result := []AgentFilesystem{}
scanner := bufio.NewScanner(file)
for scanner.Scan() {
fields := strings.Fields(scanner.Text())
if len(fields) < 3 || excluded[fields[2]] || seen[fields[1]] {
continue
}
mountpoint := strings.ReplaceAll(fields[1], `\040`, " ")
var stat syscall.Statfs_t
if syscall.Statfs(mountpoint, &stat) != nil || stat.Blocks == 0 {
continue
}
seen[mountpoint] = true
total := stat.Blocks * uint64(stat.Bsize)
free := stat.Bavail * uint64(stat.Bsize)
used := total - free
item := AgentFilesystem{Device: fields[0], Mountpoint: mountpoint, Type: fields[2], TotalBytes: total, UsedBytes: used, UsedPercent: float64(used) / float64(total) * 100, InodesTotal: stat.Files}
if stat.Files > stat.Ffree {
item.InodesUsed = stat.Files - stat.Ffree
item.InodesPercent = float64(item.InodesUsed) / float64(stat.Files) * 100
}
result = append(result, item)
if len(result) >= 30 {
break
}
}
sort.Slice(result, func(i, j int) bool { return result[i].Mountpoint < result[j].Mountpoint })
return result
}
var ssPIDPattern = regexp.MustCompile(`pid=([0-9]+)`)
var ssNamePattern = regexp.MustCompile(`users:\(\(\"([^\"]+)\"`)
func collectAgentListenPorts() []AgentListenPort {
output := runAgentCommand(5*time.Second, "ss", "-H", "-lntup")
result := []AgentListenPort{}
seen := map[string]bool{}
for _, line := range strings.Split(output, "\n") {
fields := strings.Fields(line)
if len(fields) < 5 {
continue
}
protocol := strings.ToLower(fields[0])
local := ""
for _, field := range fields[1:] {
if strings.Contains(field, ":") && !strings.Contains(field, "users:") {
local = field
break
}
}
if local == "" {
continue
}
address, portText, err := net.SplitHostPort(local)
if err != nil {
idx := strings.LastIndex(local, ":")
if idx < 0 {
continue
}
address, portText = local[:idx], local[idx+1:]
}
port, _ := strconv.Atoi(portText)
if port <= 0 {
continue
}
pid := 0
if match := ssPIDPattern.FindStringSubmatch(line); len(match) > 1 {
pid, _ = strconv.Atoi(match[1])
}
process := ""
if match := ssNamePattern.FindStringSubmatch(line); len(match) > 1 {
process = match[1]
}
key := protocol + "|" + address + "|" + portText + "|" + strconv.Itoa(pid)
if seen[key] {
continue
}
seen[key] = true
result = append(result, AgentListenPort{Protocol: protocol, Address: address, Port: port, PID: pid, Process: process})
}
sort.Slice(result, func(i, j int) bool {
if result[i].Port == result[j].Port {
return result[i].Protocol < result[j].Protocol
}
return result[i].Port < result[j].Port
})
return result
}
func collectAgentPressure() AgentPressure {
return AgentPressure{CPU: readPressure("/proc/pressure/cpu"), Memory: readPressure("/proc/pressure/memory"), IO: readPressure("/proc/pressure/io")}
}
func readPressure(path string) AgentPressureValue {
data, err := os.ReadFile(path)
if err != nil {
return AgentPressureValue{}
}
line := strings.SplitN(string(data), "\n", 2)[0]
value := AgentPressureValue{}
for _, field := range strings.Fields(line) {
key, raw, ok := strings.Cut(field, "=")
if !ok {
continue
}
number, _ := strconv.ParseFloat(raw, 64)
switch key {
case "avg10":
value.Avg10 = number
case "avg60":
value.Avg60 = number
case "avg300":
value.Avg300 = number
}
}
return value
}

152
agent_test.go Normal file
View File

@@ -0,0 +1,152 @@
package main
import (
"os"
"testing"
"time"
)
func TestAgentEnrollmentTokenIsSingleUse(t *testing.T) {
store, err := openStore(":memory:")
if err != nil {
t.Fatal(err)
}
defer store.Close()
token, err := store.CreateAgentEnrollmentToken()
if err != nil {
t.Fatal(err)
}
credentials, err := store.EnrollAgent(agentEnrollmentRequest{Token: token, Name: "docker-01", Hostname: "vm105", VMID: 105})
if err != nil {
t.Fatal(err)
}
if credentials.ID == "" || credentials.Secret == "" {
t.Fatal("credentials were not generated")
}
if _, err = store.EnrollAgent(agentEnrollmentRequest{Token: token, Hostname: "duplicate"}); err == nil {
t.Fatal("token was accepted twice")
}
}
func TestCurrentAgentUpdateManifestMatchesExecutable(t *testing.T) {
manifest, path, err := currentAgentUpdate()
if err != nil {
t.Fatal(err)
}
if manifest.URL != "/api/agent/binary" || len(manifest.SHA256) != 64 {
t.Fatalf("unexpected manifest: %#v", manifest)
}
if _, err = os.Stat(path); err != nil {
t.Fatal(err)
}
}
func TestAgentOnlyInstallsNewerVersion(t *testing.T) {
if !newerAgentVersion("v0.14.0", "v0.13.0") {
t.Fatal("new version was not detected")
}
if newerAgentVersion("v0.12.0", "v0.13.0") {
t.Fatal("downgrade was allowed")
}
if newerAgentVersion("v0.13.0", "v0.13.0") {
t.Fatal("same version was offered")
}
}
func TestShortDockerDigest(t *testing.T) {
if got := shortDockerDigest("sha256:1234567890abcdef"); got != "1234567890ab" {
t.Fatalf("unexpected short digest: %q", got)
}
}
func TestParseAgentIssues(t *testing.T) {
issues := parseAgentIssues(`{"MESSAGE":"disk error","_SYSTEMD_UNIT":"worker.service","__REALTIME_TIMESTAMP":"1722960000000000"}` + "\n" + `{"MESSAGE":"network error","__REALTIME_TIMESTAMP":"1722960001000000"}`)
if len(issues) != 2 || issues[0].Unit != "worker.service" || issues[1].Unit != "system" || issues[0].Time != 1722960000 {
t.Fatalf("unexpected issues: %#v", issues)
}
}
func TestClassifyAgentService(t *testing.T) {
cases := []struct{ path, pkg, priority, want string }{{"/lib/systemd/system/cron.service", "cron", "important", "system"}, {"/lib/systemd/system/jellyfin.service", "jellyfin", "optional", "added"}, {"/etc/systemd/system/my.service", "", "", "local"}, {"/run/systemd/system/transient.service", "", "", "runtime"}}
for _, tc := range cases {
if got := classifyAgentService(tc.path, tc.pkg, tc.priority); got != tc.want {
t.Errorf("%s: got %s want %s", tc.path, got, tc.want)
}
}
}
func TestAgentServiceDiagnosticsHelpers(t *testing.T) {
if got := compactSystemdExec(`{ path=/usr/bin/example ; argv[]=/usr/bin/example --serve ; }`); got != "/usr/bin/example" {
t.Fatalf("unexpected executable: %q", got)
}
fields := limitedFields("a b c d", 2)
if len(fields) != 2 || fields[0] != "a" || fields[1] != "b" {
t.Fatalf("unexpected limited fields: %#v", fields)
}
ports := appendUniqueInt([]int{80}, 80)
ports = appendUniqueInt(ports, 443)
if len(ports) != 2 || ports[1] != 443 {
t.Fatalf("unexpected ports: %#v", ports)
}
}
func TestAgentReportAuthenticationAndOfflineState(t *testing.T) {
store, err := openStore(":memory:")
if err != nil {
t.Fatal(err)
}
defer store.Close()
token, _ := store.CreateAgentEnrollmentToken()
credentials, _ := store.EnrollAgent(agentEnrollmentRequest{Token: token, Name: "docker", Hostname: "vm", VMID: 101})
report := AgentReport{Hostname: "vm", Version: "test", DockerAvailable: true, Containers: []AgentContainer{{ID: "abc", Name: "web", State: "running"}}}
if err = store.SaveAgentReport(credentials.ID, "wrong", "127.0.0.1", report); err == nil {
t.Fatal("invalid secret accepted")
}
if err = store.SaveAgentReport(credentials.ID, credentials.Secret, "127.0.0.1", report); err != nil {
t.Fatal(err)
}
agents, err := store.Agents()
if err != nil {
t.Fatal(err)
}
if len(agents) != 1 || !agents[0].Online || len(agents[0].Report.Containers) != 1 {
t.Fatalf("unexpected agents: %#v", agents)
}
_, _ = store.db.Exec(`UPDATE agents SET last_seen=? WHERE id=?`, time.Now().Add(-time.Minute).Unix(), credentials.ID)
agents, _ = store.Agents()
if agents[0].Online {
t.Fatal("stale agent is online")
}
}
func TestAgentProblemsBecomeMaintenanceAwareAlerts(t *testing.T) {
metrics := dashboardMetrics{Agents: []ManagedAgent{{ID: "node", Name: "app", VMID: 105, Online: true, Report: AgentReport{
Services: []AgentService{{Name: "postgres.service", State: "failed", SubState: "failed"}},
Containers: []AgentContainer{{ID: "dead", Name: "web", State: "exited", Status: "Exited (1)"}}, DockerAvailable: true,
}}}}
alerts := evaluateAlerts(metrics, defaultThresholds())
window := MaintenanceWindow{Active: true, VMIDs: []int{105}}
active, planned := splitMaintenanceAlerts(alerts, window)
if len(planned) != 2 || len(active) != 0 {
t.Fatalf("active=%d planned=%d alerts=%#v", len(active), len(planned), alerts)
}
}
func TestAgentDiagnosticsBecomeAlerts(t *testing.T) {
metrics := dashboardMetrics{Agents: []ManagedAgent{{ID: "node", Name: "app", VMID: 105, Online: true, Report: AgentReport{
Services: []AgentService{{Name: "web.service", State: "active", Flapping: true, Restarts: 8, StartedAt: time.Now().Unix()}},
Filesystems: []AgentFilesystem{{Mountpoint: "/data", TotalBytes: 1000, UsedBytes: 950, UsedPercent: 95}},
}}}}
alerts := evaluateAlerts(metrics, defaultThresholds())
wanted := map[string]bool{"agent-flapping-node-web.service-105": false, "agent-filesystem-node-/data-105": false}
for _, alert := range alerts {
if _, ok := wanted[alert.ID]; ok {
wanted[alert.ID] = true
}
}
for id, found := range wanted {
if !found {
t.Fatalf("missing alert %s in %#v", id, alerts)
}
}
}

View File

@@ -6,16 +6,18 @@ import (
)
type AlertEvent struct {
EventID int64 `json:"eventId"`
ID string `json:"id"`
Severity string `json:"severity"`
Source string `json:"source"`
Title string `json:"title"`
Message string `json:"message"`
FirstSeen int64 `json:"firstSeen"`
LastSeen int64 `json:"lastSeen"`
ResolvedAt *int64 `json:"resolvedAt,omitempty"`
AcknowledgedAt *int64 `json:"acknowledgedAt,omitempty"`
EventID int64 `json:"eventId"`
ID string `json:"id"`
Severity string `json:"severity"`
Source string `json:"source"`
Title string `json:"title"`
Message string `json:"message"`
FirstSeen int64 `json:"firstSeen"`
LastSeen int64 `json:"lastSeen"`
ResolvedAt *int64 `json:"resolvedAt,omitempty"`
AcknowledgedAt *int64 `json:"acknowledgedAt,omitempty"`
Planned bool `json:"planned"`
MaintenanceReason string `json:"maintenanceReason,omitempty"`
}
type RecoveryEvent struct {
@@ -24,13 +26,17 @@ type RecoveryEvent struct {
}
func (s *Store) SyncAlerts(alerts []Alert) ([]Alert, error) {
return s.SyncAlertsWithMaintenance(alerts, nil, "")
}
func (s *Store) SyncAlertsWithMaintenance(alerts, plannedAlerts []Alert, maintenanceReason string) ([]Alert, error) {
tx, err := s.db.Begin()
if err != nil {
return alerts, err
}
defer tx.Rollback()
now := time.Now().Unix()
rows, err := tx.Query(`SELECT id,alert_key FROM alert_events WHERE resolved_at IS NULL`)
rows, err := tx.Query(`SELECT id,alert_key,planned FROM alert_events WHERE resolved_at IS NULL`)
if err != nil {
return alerts, err
}
@@ -38,28 +44,43 @@ func (s *Store) SyncAlerts(alerts []Alert) ([]Alert, error) {
for rows.Next() {
var id int64
var key string
if err := rows.Scan(&id, &key); err != nil {
var planned bool
if err := rows.Scan(&id, &key, &planned); err != nil {
rows.Close()
return alerts, err
}
active[key] = id
active[alertEventKey(key, planned)] = id
}
rows.Close()
seen := map[string]bool{}
for _, alert := range alerts {
seen[alert.ID] = true
if id, ok := active[alert.ID]; ok {
_, err = tx.Exec(`UPDATE alert_events SET severity=?,source=?,title=?,message=?,last_seen=? WHERE id=?`, alert.Severity, alert.Source, alert.Title, alert.Message, now, id)
} else {
result, insertErr := tx.Exec(`INSERT INTO alert_events(alert_key,severity,source,title,message,first_seen,last_seen) VALUES(?,?,?,?,?,?,?)`, alert.ID, alert.Severity, alert.Source, alert.Title, alert.Message, now, now)
if insertErr != nil {
return alerts, insertErr
syncGroup := func(group []Alert, planned bool) error {
for _, alert := range group {
key := alertEventKey(alert.ID, planned)
seen[key] = true
reason := ""
if planned {
reason = maintenanceReason
}
if id, ok := active[key]; ok {
_, err = tx.Exec(`UPDATE alert_events SET severity=?,source=?,title=?,message=?,last_seen=?,maintenance_reason=? WHERE id=?`, alert.Severity, alert.Source, alert.Title, alert.Message, now, reason, id)
} else {
result, insertErr := tx.Exec(`INSERT INTO alert_events(alert_key,severity,source,title,message,first_seen,last_seen,planned,maintenance_reason) VALUES(?,?,?,?,?,?,?,?,?)`, alert.ID, alert.Severity, alert.Source, alert.Title, alert.Message, now, now, planned, reason)
if insertErr != nil {
return insertErr
}
active[key], _ = result.LastInsertId()
}
if err != nil {
return err
}
active[alert.ID], _ = result.LastInsertId()
}
if err != nil {
return alerts, err
}
return nil
}
if err = syncGroup(alerts, false); err != nil {
return alerts, err
}
if err = syncGroup(plannedAlerts, true); err != nil {
return alerts, err
}
for key, id := range active {
if !seen[key] {
@@ -74,7 +95,7 @@ func (s *Store) SyncAlerts(alerts []Alert) ([]Alert, error) {
if err = tx.Commit(); err != nil {
return alerts, err
}
rows, err = s.db.Query(`SELECT id,alert_key,severity,source,title,message,first_seen,last_seen FROM alert_events WHERE resolved_at IS NULL AND acknowledged_at IS NULL ORDER BY CASE severity WHEN 'critical' THEN 0 WHEN 'warning' THEN 1 ELSE 2 END,first_seen DESC`)
rows, err = s.db.Query(`SELECT id,alert_key,severity,source,title,message,first_seen,last_seen FROM alert_events WHERE resolved_at IS NULL AND acknowledged_at IS NULL AND planned=0 ORDER BY CASE severity WHEN 'critical' THEN 0 WHEN 'warning' THEN 1 ELSE 2 END,first_seen DESC`)
if err != nil {
return alerts, err
}
@@ -90,16 +111,35 @@ func (s *Store) SyncAlerts(alerts []Alert) ([]Alert, error) {
return visible, rows.Err()
}
func alertEventKey(id string, planned bool) string {
if planned {
return id + "\x00planned"
}
return id + "\x00active"
}
func (s *Store) AcknowledgeAlert(eventID int64) error {
_, err := s.db.Exec(`UPDATE alert_events SET acknowledged_at=? WHERE id=? AND acknowledged_at IS NULL`, time.Now().Unix(), eventID)
return err
}
func (s *Store) AlertHistory(limit int) ([]AlertEvent, error) {
return s.AlertHistoryBetween(limit, 0, 0)
}
func (s *Store) AlertHistoryBetween(limit int, from, to int64) ([]AlertEvent, error) {
if limit < 1 || limit > 500 {
limit = 200
}
rows, err := s.db.Query(`SELECT id,alert_key,severity,source,title,message,first_seen,last_seen,resolved_at,acknowledged_at FROM alert_events ORDER BY first_seen DESC LIMIT ?`, limit)
query := `SELECT id,alert_key,severity,source,title,message,first_seen,last_seen,resolved_at,acknowledged_at,planned,maintenance_reason FROM alert_events`
args := []any{}
if from > 0 && to > from {
query += ` WHERE first_seen>=? AND first_seen<?`
args = append(args, from, to)
}
query += ` ORDER BY first_seen DESC LIMIT ?`
args = append(args, limit)
rows, err := s.db.Query(query, args...)
if err != nil {
return nil, err
}
@@ -108,7 +148,7 @@ func (s *Store) AlertHistory(limit int) ([]AlertEvent, error) {
for rows.Next() {
var e AlertEvent
var resolved, acked sql.NullInt64
if err = rows.Scan(&e.EventID, &e.ID, &e.Severity, &e.Source, &e.Title, &e.Message, &e.FirstSeen, &e.LastSeen, &resolved, &acked); err != nil {
if err = rows.Scan(&e.EventID, &e.ID, &e.Severity, &e.Source, &e.Title, &e.Message, &e.FirstSeen, &e.LastSeen, &resolved, &acked, &e.Planned, &e.MaintenanceReason); err != nil {
return nil, err
}
if resolved.Valid {
@@ -123,7 +163,7 @@ func (s *Store) AlertHistory(limit int) ([]AlertEvent, error) {
}
func (s *Store) PendingEmailAlerts() ([]Alert, error) {
rows, err := s.db.Query(`SELECT id,alert_key,severity,source,title,message FROM alert_events WHERE resolved_at IS NULL AND acknowledged_at IS NULL AND email_sent_at IS NULL ORDER BY first_seen`)
rows, err := s.db.Query(`SELECT id,alert_key,severity,source,title,message FROM alert_events WHERE resolved_at IS NULL AND acknowledged_at IS NULL AND email_sent_at IS NULL AND planned=0 ORDER BY first_seen`)
if err != nil {
return nil, err
}
@@ -155,7 +195,7 @@ func (s *Store) MarkAlertsEmailed(alerts []Alert) error {
}
func (s *Store) PendingImmediateAlerts() ([]Alert, error) {
rows, err := s.db.Query(`SELECT id,alert_key,severity,source,title,message FROM alert_events WHERE resolved_at IS NULL AND acknowledged_at IS NULL AND email_sent_at IS NULL AND (severity='critical' OR alert_key='ups-battery') ORDER BY first_seen`)
rows, err := s.db.Query(`SELECT id,alert_key,severity,source,title,message FROM alert_events WHERE resolved_at IS NULL AND acknowledged_at IS NULL AND email_sent_at IS NULL AND planned=0 AND (severity='critical' OR alert_key='ups-battery') ORDER BY first_seen`)
if err != nil {
return nil, err
}
@@ -171,7 +211,7 @@ func (s *Store) PendingImmediateAlerts() ([]Alert, error) {
return out, rows.Err()
}
func (s *Store) PendingRecoveries() ([]RecoveryEvent, error) {
rows, err := s.db.Query(`SELECT id,alert_key,severity,source,title,message FROM alert_events WHERE resolved_at IS NOT NULL AND email_sent_at IS NOT NULL AND recovery_sent_at IS NULL ORDER BY resolved_at`)
rows, err := s.db.Query(`SELECT id,alert_key,severity,source,title,message FROM alert_events WHERE resolved_at IS NOT NULL AND email_sent_at IS NOT NULL AND recovery_sent_at IS NULL AND planned=0 ORDER BY resolved_at`)
if err != nil {
return nil, err
}

View File

@@ -74,7 +74,10 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
if disk.SMARTStatus == "Ошибка" {
add("disk-smart-"+disk.Name, "critical", "Диски", disk.Model+": ошибка SMART", joinReasons(disk.AttentionReasons, "SMART сообщает о неисправности диска."))
} else if disk.SMARTStatus == "Требует внимания" {
add("disk-smart-"+disk.Name, "warning", "Диски", disk.Model+": SMART требует внимания", joinReasons(disk.AttentionReasons, "Проверьте SMART-показатели диска."))
reasons := withoutReason(disk.AttentionReasons, "SMART self-test ещё ни разу не запускался")
if len(reasons) > 0 {
add("disk-smart-"+disk.Name, "warning", "Диски", disk.Model+": SMART требует внимания", joinReasons(reasons, "Проверьте SMART-показатели диска."))
}
}
if disk.Temperature != nil {
limit := thresholds.DiskTempWarning
@@ -197,6 +200,50 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
add("guests-stopped", "info", "VM/LXC", "Есть остановленные гости", fmt.Sprintf("Остановлено VM/LXC: %d. Это уведомление информационное.", stopped))
}
}
for _, agent := range metrics.Agents {
vmSuffix := fmt.Sprintf("-%d", agent.VMID)
if !agent.Online {
add("agent-offline-"+agent.ID+vmSuffix, "critical", "Агенты", agent.Name+": агент не отвечает", fmt.Sprintf("Последний heartbeat: %s назад. VMID %d.", time.Since(time.Unix(agent.LastSeen, 0)).Round(time.Second), agent.VMID))
continue
}
if agent.Report.DockerAvailable {
for _, container := range agent.Report.Containers {
if container.State != "running" {
add("agent-container-stopped-"+agent.ID+"-"+container.ID+vmSuffix, "warning", "Docker", agent.Name+": контейнер "+container.Name+" остановлен", fmt.Sprintf("%s · VMID %d.", container.Status, agent.VMID))
} else if container.Health == "unhealthy" {
add("agent-container-health-"+agent.ID+"-"+container.ID+vmSuffix, "critical", "Docker", agent.Name+": контейнер "+container.Name+" unhealthy", fmt.Sprintf("Docker health-check не проходит · VMID %d.", agent.VMID))
}
}
}
for _, service := range agent.Report.Services {
if service.State == "failed" || service.SubState == "failed" {
add("agent-systemd-"+agent.ID+"-"+service.Name+vmSuffix, "critical", "Агенты", agent.Name+": упала служба "+service.Name, fmt.Sprintf("%s · VMID %d.", firstNonEmpty(service.Description, service.SubState), agent.VMID))
} else if service.Flapping {
add("agent-flapping-"+agent.ID+"-"+service.Name+vmSuffix, "warning", "Агенты", agent.Name+": служба постоянно перезапускается", fmt.Sprintf("%s: %d рестартов, последний запуск %s · VMID %d.", service.Name, service.Restarts, time.Unix(service.StartedAt, 0).Format("15:04:05"), agent.VMID))
}
}
for _, filesystem := range agent.Report.Filesystems {
if filesystem.UsedPercent >= 90 {
add("agent-filesystem-"+agent.ID+"-"+filesystem.Mountpoint+vmSuffix, "critical", "Агенты", agent.Name+": заканчивается место "+filesystem.Mountpoint, fmt.Sprintf("Заполнено %.1f%%, свободно %.1f GB · VMID %d.", filesystem.UsedPercent, float64(filesystem.TotalBytes-filesystem.UsedBytes)/1073741824, agent.VMID))
}
if filesystem.InodesPercent >= 90 {
add("agent-inodes-"+agent.ID+"-"+filesystem.Mountpoint+vmSuffix, "critical", "Агенты", agent.Name+": заканчиваются inode "+filesystem.Mountpoint, fmt.Sprintf("Использовано %.1f%% inode · VMID %d.", filesystem.InodesPercent, agent.VMID))
}
}
if len(agent.Report.OOMKills) > 0 {
latest := agent.Report.OOMKills[0]
add("agent-oom-"+agent.ID+vmSuffix, "critical", "Агенты", agent.Name+": ядро завершало процессы из-за нехватки памяти", fmt.Sprintf("%s · %s · VMID %d.", time.Unix(latest.Time, 0).Format("02.01 15:04"), latest.Message, agent.VMID))
}
if agent.Report.RebootRequired {
add("agent-reboot-"+agent.ID+vmSuffix, "info", "Агенты", agent.Name+": требуется перезагрузка", fmt.Sprintf("Обновления системы запросили перезагрузку · VMID %d.", agent.VMID))
}
if agent.Report.MemoryTotal > 0 && float64(agent.Report.MemoryUsed)/float64(agent.Report.MemoryTotal)*100 >= 95 {
add("agent-memory-"+agent.ID+vmSuffix, "warning", "Агенты", agent.Name+": почти закончилась память", fmt.Sprintf("Используется %.1f%% RAM · VMID %d.", float64(agent.Report.MemoryUsed)/float64(agent.Report.MemoryTotal)*100, agent.VMID))
}
if agent.Report.RootTotal > 0 && float64(agent.Report.RootUsed)/float64(agent.Report.RootTotal)*100 >= 90 {
add("agent-storage-"+agent.ID+vmSuffix, "critical", "Агенты", agent.Name+": мало места на системном разделе", fmt.Sprintf("Раздел / заполнен на %.1f%% · VMID %d.", float64(agent.Report.RootUsed)/float64(agent.Report.RootTotal)*100, agent.VMID))
}
}
for _, service := range metrics.Services.Services {
for _, endpoint := range service.Endpoints {
if !endpoint.Up {
@@ -216,10 +263,13 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
}
var networkErrors uint64
for _, iface := range metrics.Network.Interfaces {
networkErrors += iface.ReceiveErrors + iface.TransmitErrors + iface.ReceiveDrops + iface.TransmitDrops
networkErrors += iface.NewErrors + iface.NewFaults
if iface.LinkDown {
add("network-link-"+iface.Name, "critical", "Сеть", iface.Name+": нет линка", "Используемый интерфейс потерял carrier или перешёл в down.")
}
}
if networkErrors > 0 {
add("network-errors", "warning", "Сеть", "Сетевые ошибки и потери пакетов", fmt.Sprintf("Суммарный счётчик интерфейсов: %d.", networkErrors))
add("network-errors", "critical", "Сеть", "Появились новые сетевые ошибки", fmt.Sprintf("Прирост счётчиков ошибок: %d (счётчики могут пересекаться). Software RX/TX drops показаны отдельно и не создают этот алерт.", networkErrors))
}
if !metrics.Network.Health.CheckedAt.IsZero() && !metrics.Network.Health.GatewayUp {
add("network-gateway", "critical", "Сеть", "Недоступен локальный шлюз", firstNonEmpty(metrics.Network.Health.Gateway, "Default gateway не определён."))
@@ -239,20 +289,10 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
for _, name := range metrics.Network.Health.DisappearedInterfaces {
add("network-interface-missing-"+name, "warning", "Сеть", "Исчез сетевой интерфейс", name+" больше не присутствует в /proc/net/dev.")
}
for _, iface := range metrics.Network.Interfaces {
if iface.Kind == "Physical" && !iface.Carrier && (iface.Master != "" || iface.ReceiveBytes+iface.TransmitBytes > 0) {
add("network-link-"+iface.Name, "warning", "Сеть", iface.Name+": нет carrier", "Физический линк не поднят.")
}
if (iface.Kind == "Bridge" || iface.Kind == "Bond" || iface.Kind == "VLAN") && iface.State == "down" {
add("network-state-"+iface.Name, "warning", "Сеть", iface.Name+": интерфейс выключен", iface.Kind+" находится в состоянии down.")
}
}
if !metrics.Monitor.LastSuccessfulCollection.IsZero() && time.Since(metrics.Monitor.LastSuccessfulCollection) > 30*time.Second {
add("monitor-stale", "critical", "Dashboard", "Данные мониторинга устарели", "Последний успешный сбор был "+time.Since(metrics.Monitor.LastSuccessfulCollection).Round(time.Second).String()+" назад.")
}
if metrics.Monitor.SlowCollector {
add("monitor-slow", "warning", "Dashboard", "Сбор метрик выполняется слишком долго", fmt.Sprintf("Последний цикл занял %d ms.", metrics.Monitor.CollectionDurationMs))
}
if metrics.Monitor.HistoryWriteError != "" {
add("monitor-history", "critical", "Dashboard", "Ошибка записи истории", metrics.Monitor.HistoryWriteError)
}
@@ -345,3 +385,13 @@ func joinReasons(reasons []string, fallback string) string {
}
return strings.Join(reasons, " · ")
}
func withoutReason(reasons []string, excluded string) []string {
filtered := make([]string, 0, len(reasons))
for _, reason := range reasons {
if reason != excluded {
filtered = append(filtered, reason)
}
}
return filtered
}

View File

@@ -16,3 +16,26 @@ func TestEvaluateAlerts(t *testing.T) {
t.Fatalf("неверная сортировка уведомлений: %+v", alerts)
}
}
func TestSlowCollectionDoesNotCreateAlert(t *testing.T) {
metrics := dashboardMetrics{Monitor: MonitorHealth{SlowCollector: true, CollectionDurationMs: 12308}}
for _, alert := range evaluateAlerts(metrics, defaultThresholds()) {
if alert.ID == "monitor-slow" {
t.Fatalf("медленный цикл не должен создавать уведомление: %+v", alert)
}
}
}
func TestNeverRunSelfTestCreatesOneAlert(t *testing.T) {
metrics := dashboardMetrics{Disks: []DiskMetrics{{
Name: "nvme0n1",
Model: "Test NVMe",
SMARTStatus: "Требует внимания",
SelfTestNeverRun: true,
AttentionReasons: []string{"SMART self-test ещё ни разу не запускался"},
}}}
alerts := evaluateAlerts(metrics, defaultThresholds())
if len(alerts) != 1 || alerts[0].ID != "disk-selftest-nvme0n1" {
t.Fatalf("ожидалось одно точное уведомление о self-test, получено: %+v", alerts)
}
}

38
cpu.go
View File

@@ -3,6 +3,7 @@ package main
import (
"bufio"
"fmt"
"math"
"os"
"path/filepath"
"strconv"
@@ -33,14 +34,16 @@ type CPUStatic struct {
type CPUMetrics struct {
CPUStatic
UsagePercent float64 `json:"usagePercent"`
FrequencyMHz float64 `json:"frequencyMhz"`
Load1 float64 `json:"load1"`
Load5 float64 `json:"load5"`
Load15 float64 `json:"load15"`
Uptime uint64 `json:"uptimeSeconds"`
Temperatures []Temperature `json:"temperatures"`
CollectedAt time.Time `json:"collectedAt"`
UsagePercent float64 `json:"usagePercent"`
FrequencyMHz float64 `json:"frequencyMhz"`
Load1 float64 `json:"load1"`
Load5 float64 `json:"load5"`
Load15 float64 `json:"load15"`
Uptime uint64 `json:"uptimeSeconds"`
TemperatureCelsius *float64 `json:"temperatureCelsius"`
Temperatures []Temperature `json:"temperatures"`
TopProcesses []AgentProcess `json:"topProcesses"`
CollectedAt time.Time `json:"collectedAt"`
}
type Temperature struct {
@@ -89,10 +92,11 @@ func (c *cpuCollector) collect() (CPUMetrics, error) {
load1, load5, load15, _ := readLoadAverage()
uptime, _ := readUptime()
temperatures := readTemperatures()
return CPUMetrics{
CPUStatic: c.static, UsagePercent: usage, FrequencyMHz: frequency,
Load1: load1, Load5: load5, Load15: load15, Uptime: uptime,
Temperatures: readTemperatures(), CollectedAt: time.Now(),
Temperatures: temperatures, TemperatureCelsius: cpuTemperature(temperatures), TopProcesses: collectAgentProcesses(), CollectedAt: time.Now(),
}, nil
}
@@ -284,3 +288,19 @@ func isCPUTemperatureChip(name string) bool {
return false
}
}
// cpuTemperature is the hottest CPU sensor, shared by live and historical views.
// Do not average Tctl/Tdie with cooler cores or CCDs.
func cpuTemperature(temperatures []Temperature) *float64 {
var hottest *float64
for _, t := range temperatures {
if math.IsNaN(t.Celsius) || math.IsInf(t.Celsius, 0) || t.Celsius <= 0 {
continue
}
if hottest == nil || t.Celsius > *hottest {
value := t.Celsius
hottest = &value
}
}
return hottest
}

24
cpu_temperature_test.go Normal file
View File

@@ -0,0 +1,24 @@
package main
import (
"math"
"testing"
)
func TestCPUTemperatureSharedWithHistory(t *testing.T) {
temperatures := []Temperature{{Label: "k10temp · Tctl", Celsius: 62}, {Label: "k10temp · Tccd1", Celsius: 38}, {Label: "k10temp · Tccd2", Celsius: 41}}
live := cpuTemperature(temperatures)
point := historyPointFromMetrics(dashboardMetrics{CPU: CPUMetrics{Temperatures: temperatures}})
if live == nil || *live != 62 || point.CPUTemperature != *live {
t.Fatalf("live/history disagree: %v, %+v", live, point)
}
}
func TestCPUTemperatureInvalidSensors(t *testing.T) {
if cpuTemperature([]Temperature{{Celsius: math.NaN()}, {Celsius: math.Inf(1)}, {Celsius: 0}}) != nil {
t.Fatal("invalid temperature selected")
}
if cpuTemperature(nil) != nil {
t.Fatal("missing sensor should stay missing")
}
}

View File

@@ -391,6 +391,17 @@ type smartctlJSON struct {
} `json:"table"`
} `json:"standard"`
} `json:"ata_smart_self_test_log"`
NVMeSelfTestLog struct {
Table []struct {
SelfTestCode struct {
String string `json:"string"`
} `json:"self_test_code"`
SelfTestResult struct {
String string `json:"string"`
} `json:"self_test_result"`
PowerOnHours uint64 `json:"power_on_hours"`
} `json:"table"`
} `json:"nvme_self_test_log"`
NVMeHealth *struct {
Temperature float64 `json:"temperature"`
PercentageUsed float64 `json:"percentage_used"`
@@ -410,6 +421,10 @@ func applySMART(disk *DiskMetrics) {
if len(output) == 0 {
return
}
applySMARTJSON(disk, output)
}
func applySMARTJSON(disk *DiskMetrics, output []byte) {
var smart smartctlJSON
if json.Unmarshal(output, &smart) != nil {
return
@@ -451,12 +466,20 @@ func applySMART(disk *DiskMetrics) {
disk.AttentionReasons = append(disk.AttentionReasons, "Ошибки целостности носителя: "+strconv.FormatUint(smart.NVMeHealth.MediaErrors, 10))
}
}
if len(smart.ATASelfTestLog.Standard.Table) > 0 {
if len(smart.NVMeSelfTestLog.Table) > 0 {
last := smart.NVMeSelfTestLog.Table[0]
description := strings.TrimSpace(last.SelfTestCode.String)
result := firstNonEmpty(strings.TrimSpace(last.SelfTestResult.String), "Результат неизвестен")
if description != "" {
disk.LastSelfTest = description + " · " + result
} else {
disk.LastSelfTest = result
}
setLastSelfTestAt(disk, last.PowerOnHours)
} else if len(smart.ATASelfTestLog.Standard.Table) > 0 {
last := smart.ATASelfTestLog.Standard.Table[0]
disk.LastSelfTest = firstNonEmpty(last.Status.String, "Результат неизвестен")
if disk.PowerOnHours != nil && *disk.PowerOnHours >= last.LifetimeHours {
disk.LastSelfTestAt = time.Now().Add(-time.Duration(*disk.PowerOnHours-last.LifetimeHours) * time.Hour).Format(time.RFC3339)
}
setLastSelfTestAt(disk, last.LifetimeHours)
} else {
disk.SelfTestNeverRun = true
disk.LastSelfTest = "Никогда не запускался"
@@ -469,6 +492,12 @@ func applySMART(disk *DiskMetrics) {
}
}
func setLastSelfTestAt(disk *DiskMetrics, testPowerOnHours uint64) {
if disk.PowerOnHours != nil && *disk.PowerOnHours >= testPowerOnHours {
disk.LastSelfTestAt = time.Now().Add(-time.Duration(*disk.PowerOnHours-testPowerOnHours) * time.Hour).Format(time.RFC3339)
}
}
func applyATAAttributes(disk *DiskMetrics, smart smartctlJSON) {
wearNames := map[string]bool{
"Percent_Lifetime_Remain": true, "SSD_Life_Left": true,

View File

@@ -6,8 +6,44 @@ import (
"path/filepath"
"strings"
"testing"
"time"
)
func TestApplySMARTJSONReadsNVMeSelfTestHistory(t *testing.T) {
disk := DiskMetrics{Path: "/dev/nvme0n1"}
output := []byte(`{
"model_name":"ADATA LEGEND 900",
"smart_status":{"passed":true},
"power_on_time":{"hours":960},
"nvme_self_test_log":{
"current_self_test_operation":{"value":0,"string":"No self-test in progress"},
"table":[{
"self_test_code":{"value":1,"string":"Short"},
"self_test_result":{"value":0,"string":"Completed without error"},
"power_on_hours":959
}]
}
}`)
before := time.Now().Add(-2 * time.Hour)
applySMARTJSON(&disk, output)
after := time.Now()
if disk.SelfTestNeverRun {
t.Fatal("завершённый NVMe self-test ошибочно отмечен как никогда не запускавшийся")
}
if disk.LastSelfTest != "Short · Completed without error" {
t.Fatalf("неожиданный результат self-test: %q", disk.LastSelfTest)
}
parsed, err := time.Parse(time.RFC3339, disk.LastSelfTestAt)
if err != nil || parsed.Before(before) || parsed.After(after) {
t.Fatalf("неверно вычислено время self-test: %q, err=%v", disk.LastSelfTestAt, err)
}
if len(disk.AttentionReasons) != 0 || disk.SMARTStatus != "Исправен" {
t.Fatalf("исправный NVMe требует внимания: %+v", disk)
}
}
func TestIsPhysicalDiskName(t *testing.T) {
tests := map[string]bool{
"sda": true, "sdb": true, "nvme0n1": true,

View File

@@ -64,6 +64,30 @@ func (s *Store) AddEntityMetrics(metrics dashboardMetrics) {
s.AddEntityHistory("backup-duration", "VMID "+task.ID, ts, &duration, nil)
}
}
for _, agent := range metrics.Agents {
if !agent.Online {
continue
}
name := agent.ID + " · " + agent.Name
cpu := agent.Report.CPUPercent
memory := float64(0)
if agent.Report.MemoryTotal > 0 {
memory = float64(agent.Report.MemoryUsed) / float64(agent.Report.MemoryTotal) * 100
}
swap := float64(0)
if agent.Report.SwapTotal > 0 {
swap = float64(agent.Report.SwapUsed) / float64(agent.Report.SwapTotal) * 100
}
s.AddEntityHistory("agent-resource", name, ts, &cpu, &memory)
s.AddEntityHistory("agent-swap", name, ts, &swap, nil)
pressure := agent.Report.Pressure.IO.Avg10
s.AddEntityHistory("agent-pressure", name, ts, &pressure, nil)
for _, filesystem := range agent.Report.Filesystems {
used := filesystem.UsedPercent
inodes := filesystem.InodesPercent
s.AddEntityHistory("agent-filesystem", name+" · "+filesystem.Mountpoint, ts, &used, &inodes)
}
}
s.PruneEntityHistory()
}
@@ -75,12 +99,10 @@ func historyPointFromMetrics(metrics dashboardMetrics) HistoryPoint {
NetworkTransmit: metrics.Network.TransmitPerSec, UPSCharge: metrics.UPS.ChargePercent,
UPSLoad: metrics.UPS.LoadPercent,
}
if len(metrics.CPU.Temperatures) > 0 {
for _, temperature := range metrics.CPU.Temperatures {
point.CPUTemperature += temperature.Celsius
}
point.CPUTemperature /= float64(len(metrics.CPU.Temperatures))
if temperature := cpuTemperature(metrics.CPU.Temperatures); temperature != nil {
point.CPUTemperature = *temperature
}
var zfsSize, zfsUsed uint64
for _, pool := range metrics.ZFS.Pools {
zfsSize += pool.SizeBytes

View File

@@ -8,7 +8,7 @@ func TestHistoryPointFromMetrics(t *testing.T) {
Guests: GuestsMetrics{Guests: []GuestMetrics{{Status: "running", CPU: .5, MaxCPU: 4, MemoryBytes: 50, MaxMemory: 100}}},
}
point := historyPointFromMetrics(metrics)
if point.CPUTemperature != 50 || point.GuestsCPU != 50 || point.GuestsMemory != 50 {
if point.CPUTemperature != 60 || point.GuestsCPU != 50 || point.GuestsMemory != 50 {
t.Fatalf("неожиданная историческая точка: %+v", point)
}
}

12
main.go
View File

@@ -20,11 +20,23 @@ var version = "dev"
func main() {
showVersion := flag.Bool("version", false, "show version")
agentMode := flag.Bool("agent", false, "run as a read-only guest agent")
agentServer := flag.String("agent-server", "", "dashboard URL used by the guest agent")
agentEnrollToken := flag.String("agent-enroll-token", "", "one-time enrollment token")
agentName := flag.String("agent-name", "", "agent display name")
agentVMID := flag.Int("agent-vmid", 0, "Proxmox VM/LXC ID")
agentConfig := flag.String("agent-config", "/var/lib/proxmox-dashboard-agent/agent.json", "agent configuration path")
flag.Parse()
if *showVersion {
fmt.Println(version)
return
}
if *agentMode {
if err := runGuestAgent(agentOptions{Server: *agentServer, EnrollmentToken: *agentEnrollToken, Name: *agentName, VMID: *agentVMID, ConfigPath: *agentConfig}); err != nil {
log.Fatalf("Ошибка агента: %v", err)
}
return
}
address := os.Getenv("DASHBOARD_ADDR")
if address == "" {

170
maintenance_window.go Normal file
View File

@@ -0,0 +1,170 @@
package main
import (
"encoding/json"
"fmt"
"strconv"
"strings"
"time"
)
type MaintenanceWindow struct {
Enabled bool `json:"enabled"`
Active bool `json:"active"`
StartedAt int64 `json:"startedAt"`
EndsAt int64 `json:"endsAt"`
Reason string `json:"reason"`
AllAlerts bool `json:"allAlerts"`
ServiceIDs []int64 `json:"serviceIds"`
VMIDs []int `json:"vmids"`
Automatic bool `json:"automatic"`
Source string `json:"source,omitempty"`
}
func automaticBackupMaintenance(activity ProxmoxActivity, links []ServiceLink, now time.Time) MaintenanceWindow {
vmids := []int{}
backupRunning := false
for _, task := range activity.Tasks {
if task.Type != "vzdump" || task.EndTime > 0 || (task.Status != "" && !strings.EqualFold(task.Status, "running")) {
continue
}
backupRunning = true
if vmid := backupTaskVMID(task); vmid > 0 {
vmids = append(vmids, vmid)
}
}
vmids = uniqueInt(vmids)
if !backupRunning {
return MaintenanceWindow{}
}
selected := map[int]bool{}
for _, vmid := range vmids {
selected[vmid] = true
}
serviceIDs := []int64{}
for _, link := range links {
if selected[link.VMID] {
serviceIDs = append(serviceIDs, link.ServiceID)
}
}
return MaintenanceWindow{Enabled: true, Active: true, StartedAt: now.Unix(), EndsAt: now.Add(2 * time.Minute).Unix(), Reason: "Автоматически: выполняется backup VM/LXC", VMIDs: vmids, ServiceIDs: uniqueInt64(serviceIDs), AllAlerts: len(vmids) == 0, Automatic: true, Source: "backup"}
}
func backupTaskVMID(task ProxmoxTask) int {
if vmid, err := strconv.Atoi(strings.TrimSpace(task.ID)); err == nil && vmid > 0 {
return vmid
}
parts := strings.Split(task.UPID, ":")
for i, part := range parts {
if part == "vzdump" && i+1 < len(parts) {
vmid, _ := strconv.Atoi(parts[i+1])
return vmid
}
}
return 0
}
func (s *Store) MaintenanceWindow() MaintenanceWindow {
var value string
var window MaintenanceWindow
if s.db.QueryRow(`SELECT value FROM settings WHERE key='maintenance_window'`).Scan(&value) == nil {
_ = json.Unmarshal([]byte(value), &window)
}
now := time.Now().Unix()
window.Active = window.Enabled && window.StartedAt <= now && window.EndsAt > now
if window.Enabled && window.EndsAt <= now {
window.Enabled = false
data, _ := json.Marshal(window)
_, _ = s.db.Exec(`INSERT INTO settings(key,value) VALUES('maintenance_window',?) ON CONFLICT(key) DO UPDATE SET value=excluded.value`, string(data))
}
return window
}
func (s *Store) SaveMaintenanceWindow(window MaintenanceWindow) (MaintenanceWindow, error) {
now := time.Now()
window.Reason = strings.TrimSpace(window.Reason)
if window.Reason == "" {
return window, fmt.Errorf("укажите причину плановых работ")
}
if window.EndsAt <= now.Unix() || window.EndsAt > now.Add(30*24*time.Hour).Unix() {
return window, fmt.Errorf("окончание должно быть в будущем, максимум через 30 дней")
}
if !window.AllAlerts && len(window.ServiceIDs) == 0 && len(window.VMIDs) == 0 {
return window, fmt.Errorf("выберите сервис, VM/LXC или все алерты")
}
window.Enabled = true
window.Active = true
window.StartedAt = now.Unix()
window.ServiceIDs = uniqueInt64(window.ServiceIDs)
window.VMIDs = uniqueInt(window.VMIDs)
data, _ := json.Marshal(window)
_, err := s.db.Exec(`INSERT INTO settings(key,value) VALUES('maintenance_window',?) ON CONFLICT(key) DO UPDATE SET value=excluded.value`, string(data))
return window, err
}
func (s *Store) StopMaintenanceWindow() (MaintenanceWindow, error) {
window := s.MaintenanceWindow()
window.Enabled = false
window.Active = false
window.EndsAt = time.Now().Unix()
data, _ := json.Marshal(window)
_, err := s.db.Exec(`INSERT INTO settings(key,value) VALUES('maintenance_window',?) ON CONFLICT(key) DO UPDATE SET value=excluded.value`, string(data))
return window, err
}
func splitMaintenanceAlerts(alerts []Alert, window MaintenanceWindow) (active, planned []Alert) {
if !window.Active {
return alerts, nil
}
for _, alert := range alerts {
if maintenanceMatchesAlert(window, alert) {
planned = append(planned, alert)
} else {
active = append(active, alert)
}
}
return active, planned
}
func maintenanceMatchesAlert(window MaintenanceWindow, alert Alert) bool {
if window.AllAlerts {
return true
}
for _, id := range window.ServiceIDs {
value := strconv.FormatInt(id, 10)
if strings.HasPrefix(alert.ID, "service-"+value+"-") || strings.HasPrefix(alert.ID, "service-tls-"+value+"-") {
return true
}
}
for _, vmid := range window.VMIDs {
value := strconv.Itoa(vmid)
if strings.HasSuffix(alert.ID, "-"+value) || strings.Contains(alert.Message, "VMID "+value) {
return true
}
}
return false
}
func uniqueInt64(values []int64) []int64 {
seen := map[int64]bool{}
result := make([]int64, 0, len(values))
for _, value := range values {
if value > 0 && !seen[value] {
seen[value] = true
result = append(result, value)
}
}
return result
}
func uniqueInt(values []int) []int {
seen := map[int]bool{}
result := make([]int, 0, len(values))
for _, value := range values {
if value > 0 && !seen[value] {
seen[value] = true
result = append(result, value)
}
}
return result
}

View File

@@ -0,0 +1,75 @@
package main
import (
"testing"
"time"
)
func TestMaintenanceFiltersSelectedTargets(t *testing.T) {
window := MaintenanceWindow{Active: true, ServiceIDs: []int64{7}, VMIDs: []int{105}}
alerts := []Alert{
{ID: "service-7-local"},
{ID: "service-tls-8-public"},
{ID: "backup-old-105"},
{ID: "task-failed-x", Message: "VMID 105, status error"},
{ID: "cpu-usage"},
}
active, planned := splitMaintenanceAlerts(alerts, window)
if len(active) != 2 || len(planned) != 3 {
t.Fatalf("неверное разделение: active=%+v planned=%+v", active, planned)
}
}
func TestMaintenanceExpiresAutomatically(t *testing.T) {
store, err := openStore(":memory:")
if err != nil {
t.Fatal(err)
}
defer store.Close()
window, err := store.SaveMaintenanceWindow(MaintenanceWindow{EndsAt: time.Now().Add(time.Hour).Unix(), Reason: "Обновление", AllAlerts: true})
if err != nil || !window.Active {
t.Fatalf("режим не включён: %+v %v", window, err)
}
if _, err = store.db.Exec(`UPDATE settings SET value=? WHERE key='maintenance_window'`, `{"enabled":true,"startedAt":1,"endsAt":2,"reason":"test","allAlerts":true}`); err != nil {
t.Fatal(err)
}
if expired := store.MaintenanceWindow(); expired.Active || expired.Enabled {
t.Fatalf("просроченный режим остался активным: %+v", expired)
}
}
func TestRunningBackupCreatesScopedAutomaticMaintenance(t *testing.T) {
now := time.Now()
window := automaticBackupMaintenance(ProxmoxActivity{Tasks: []ProxmoxTask{{Type: "vzdump", ID: "105"}, {Type: "vzdump", ID: "106", EndTime: now.Unix()}}}, []ServiceLink{{ServiceID: 7, VMID: 105}, {ServiceID: 8, VMID: 999}}, now)
if !window.Active || !window.Automatic || window.Source != "backup" || len(window.VMIDs) != 1 || window.VMIDs[0] != 105 || len(window.ServiceIDs) != 1 || window.ServiceIDs[0] != 7 {
t.Fatalf("неверное автоматическое окно: %+v", window)
}
}
func TestRunningBackupWithoutVMIDSuppressesAllAlerts(t *testing.T) {
window := automaticBackupMaintenance(ProxmoxActivity{Tasks: []ProxmoxTask{{Type: "vzdump", UPID: "unknown"}}}, nil, time.Now())
if !window.Active || !window.AllAlerts {
t.Fatalf("backup с неизвестной целью должен безопасно подавить алерты: %+v", window)
}
}
func TestPlannedAlertOnlyAppearsInHistory(t *testing.T) {
store, err := openStore(":memory:")
if err != nil {
t.Fatal(err)
}
defer store.Close()
alert := Alert{ID: "service-7-local", Severity: "critical", Source: "Сервисы", Title: "Недоступен"}
visible, err := store.SyncAlertsWithMaintenance(nil, []Alert{alert}, "Обновление")
if err != nil || len(visible) != 0 {
t.Fatalf("плановый алерт стал активным: %+v %v", visible, err)
}
history, err := store.AlertHistory(10)
if err != nil || len(history) != 1 || !history[0].Planned || history[0].MaintenanceReason != "Обновление" {
t.Fatalf("плановое событие не сохранено: %+v %v", history, err)
}
pending, err := store.PendingEmailAlerts()
if err != nil || len(pending) != 0 {
t.Fatalf("плановое событие попало в почту: %+v %v", pending, err)
}
}

View File

@@ -3,6 +3,7 @@ package main
import (
"bufio"
"context"
"fmt"
"io"
"net"
"net/http"
@@ -36,27 +37,37 @@ type NetworkHealth struct {
ExternalIPChanged bool `json:"externalIpChanged"`
DisappearedInterfaces []string `json:"disappearedInterfaces"`
CheckedAt time.Time `json:"checkedAt"`
Summary string `json:"summary"`
LikelyCause string `json:"likelyCause"`
RecommendedAction string `json:"recommendedAction"`
}
type NetworkInterface struct {
Name string `json:"name"`
ReceiveBytes uint64 `json:"receiveBytes"`
TransmitBytes uint64 `json:"transmitBytes"`
ReceiveErrors uint64 `json:"receiveErrors"`
TransmitErrors uint64 `json:"transmitErrors"`
ReceiveDrops uint64 `json:"receiveDrops"`
TransmitDrops uint64 `json:"transmitDrops"`
ReceivePerSec float64 `json:"receivePerSec"`
TransmitPerSec float64 `json:"transmitPerSec"`
VMID int `json:"vmid,omitempty"`
GuestName string `json:"guestName,omitempty"`
GuestType string `json:"guestType,omitempty"`
Kind string `json:"kind"`
State string `json:"state"`
Carrier bool `json:"carrier"`
SpeedMbps int `json:"speedMbps,omitempty"`
Duplex string `json:"duplex,omitempty"`
Master string `json:"master,omitempty"`
Name string `json:"name"`
ReceiveBytes uint64 `json:"receiveBytes"`
TransmitBytes uint64 `json:"transmitBytes"`
ReceiveErrors uint64 `json:"receiveErrors"`
TransmitErrors uint64 `json:"transmitErrors"`
ReceiveDrops uint64 `json:"receiveDrops"`
TransmitDrops uint64 `json:"transmitDrops"`
NewErrors uint64 `json:"newErrors"`
NewReceiveDrops uint64 `json:"newReceiveDrops"`
NewTransmitDrops uint64 `json:"newTransmitDrops"`
FaultCounters map[string]uint64 `json:"faultCounters,omitempty"`
NewFaults uint64 `json:"newFaults"`
LinkDown bool `json:"linkDown"`
NewDrops uint64 `json:"newDrops"`
ReceivePerSec float64 `json:"receivePerSec"`
TransmitPerSec float64 `json:"transmitPerSec"`
VMID int `json:"vmid,omitempty"`
GuestName string `json:"guestName,omitempty"`
GuestType string `json:"guestType,omitempty"`
Kind string `json:"kind"`
State string `json:"state"`
Carrier bool `json:"carrier"`
SpeedMbps int `json:"speedMbps,omitempty"`
Duplex string `json:"duplex,omitempty"`
Master string `json:"master,omitempty"`
}
type networkSnapshot struct {
@@ -77,13 +88,14 @@ type networkCollector struct {
}
func newNetworkCollector(path string) (*networkCollector, error) {
snapshot, _, _, err := readNetworkSnapshot(path)
snapshot, _, details, err := readNetworkSnapshot(path)
if err != nil {
return nil, err
}
_, _, details, _ := readNetworkSnapshot(path)
previous := map[string]NetworkInterface{}
for _, v := range details {
decorateInterface(&v)
collectInterfaceFaults(&v)
previous[v.Name] = v
}
known := map[string]bool{}
@@ -110,10 +122,11 @@ func (c *networkCollector) collect() (NetworkMetrics, error) {
for i := range details {
v := &details[i]
decorateInterface(v)
if old, ok := c.previousInterfaces[v.Name]; ok && elapsed > 0 && v.ReceiveBytes >= old.ReceiveBytes && v.TransmitBytes >= old.TransmitBytes {
v.ReceivePerSec = float64(v.ReceiveBytes-old.ReceiveBytes) / elapsed
v.TransmitPerSec = float64(v.TransmitBytes-old.TransmitBytes) / elapsed
collectInterfaceFaults(v)
if old, ok := c.previousInterfaces[v.Name]; ok {
updateInterfaceDeltas(v, old, elapsed)
}
v.LinkDown = interfaceLinkDown(*v)
}
c.previousInterfaces = map[string]NetworkInterface{}
for _, v := range details {
@@ -138,12 +151,41 @@ func (c *networkCollector) collect() (NetworkMetrics, error) {
}
}
c.knownInterfaces = currentNames
explainNetworkHealth(&c.health, details)
return NetworkMetrics{
Interface: strings.Join(interfaceNames, ", "), Interfaces: details, ReceiveBytes: current.received,
TransmitBytes: current.sent, ReceivePerSec: receivedRate, TransmitPerSec: sentRate, Health: c.health,
}, nil
}
func explainNetworkHealth(health *NetworkHealth, interfaces []NetworkInterface) {
health.Summary, health.LikelyCause, health.RecommendedAction = "Сеть работает штатно", "Проблем не обнаружено", "Действий не требуется"
switch {
case health.Gateway == "":
health.Summary, health.LikelyCause, health.RecommendedAction = "Не найден маршрут в локальную сеть", "На хосте отсутствует default gateway", "Проверьте адрес шлюза и конфигурацию bridge/VLAN"
case !health.GatewayUp:
health.Summary, health.LikelyCause, health.RecommendedAction = "Хост не видит локальный шлюз", "Проблема между Proxmox и роутером: линк, bridge, bond или VLAN", "Проверьте кабель, carrier и настройки vmbr/bond/VLAN"
case health.PacketLoss > 0:
health.Summary = fmt.Sprintf("Нестабильная локальная сеть: потеряно %.1f%% пакетов", health.PacketLoss)
health.LikelyCause, health.RecommendedAction = "Перегрузка или ошибки линка между Proxmox и шлюзом", "Проверьте аппаратные ошибки, кабель, порт коммутатора и duplex"
case !health.DNSUp && health.InternetUp:
health.Summary, health.LikelyCause, health.RecommendedAction = "Интернет доступен, но DNS не отвечает", "Проблема DNS-сервера или /etc/resolv.conf", "Проверьте настроенные DNS и попробуйте разрешить домен вручную"
case health.DNSUp && !health.InternetUp:
health.Summary, health.LikelyCause, health.RecommendedAction = "Локальная сеть и DNS работают, но внешняя проверка недоступна", "Проблема выше роутера, блокировка HTTPS или недоступность провайдера", "Проверьте маршрут наружу, firewall и доступ к внешним адресам"
}
for _, v := range interfaces {
if v.LinkDown {
health.Summary, health.LikelyCause, health.RecommendedAction = v.Name+": линк не поднят", "Нет carrier либо интерфейс административно выключен", "Проверьте кабель, порт коммутатора и состояние интерфейса"
return
}
if v.NewErrors+v.NewFaults > 0 {
health.Summary = fmt.Sprintf("%s: за последний интервал появилось %d изменений счётчиков сетевых ошибок", v.Name, v.NewErrors+v.NewFaults)
health.LikelyCause, health.RecommendedAction = "Счётчики интерфейса сейчас растут из-за перегрузки, кабеля, MTU или duplex", "Проверьте скорость и duplex, кабель, порт коммутатора и MTU на обоих концах"
return
}
}
}
func decorateInterface(v *NetworkInterface) {
base := filepath.Join("/sys/class/net", v.Name)
v.State = readTrimmed(filepath.Join(base, "operstate"))
@@ -154,6 +196,10 @@ func decorateInterface(v *NetworkInterface) {
v.Master = filepath.Base(target)
}
switch {
case strings.HasPrefix(v.Name, "fwbr"):
v.Kind = "Firewall bridge"
case strings.HasPrefix(v.Name, "fwpr") || strings.HasPrefix(v.Name, "fwln"):
v.Kind = "Firewall link"
case strings.HasPrefix(v.Name, "vmbr"):
v.Kind = "Bridge"
case strings.HasPrefix(v.Name, "bond"):
@@ -232,7 +278,7 @@ func readNetworkSnapshot(path string) (networkSnapshot, []string, []NetworkInter
continue
}
fields := strings.Fields(values)
if len(fields) < 9 {
if len(fields) < 16 {
continue
}
rx, errRX := strconv.ParseUint(fields[0], 10, 64)

103
network_faults.go Normal file
View File

@@ -0,0 +1,103 @@
package main
import (
"context"
"os/exec"
"path/filepath"
"strconv"
"strings"
"time"
)
func counterDelta(current, previous uint64) uint64 {
if current < previous {
return 0
}
return current - previous
}
func updateInterfaceDeltas(v *NetworkInterface, old NetworkInterface, elapsed float64) {
if elapsed <= 0 {
return
}
v.ReceivePerSec = float64(counterDelta(v.ReceiveBytes, old.ReceiveBytes)) / elapsed
v.TransmitPerSec = float64(counterDelta(v.TransmitBytes, old.TransmitBytes)) / elapsed
v.NewErrors = counterDelta(v.ReceiveErrors, old.ReceiveErrors) + counterDelta(v.TransmitErrors, old.TransmitErrors)
v.NewReceiveDrops = counterDelta(v.ReceiveDrops, old.ReceiveDrops)
v.NewTransmitDrops = counterDelta(v.TransmitDrops, old.TransmitDrops)
v.NewDrops = v.NewReceiveDrops + v.NewTransmitDrops
for name, value := range v.FaultCounters {
if previous, ok := old.FaultCounters[name]; ok {
v.NewFaults += counterDelta(value, previous)
}
}
// A transition matters even if the interface has not carried traffic yet.
v.LinkDown = old.Carrier && !v.Carrier
}
func interfaceLinkDown(v NetworkInterface) bool {
if v.Kind == "Firewall bridge" || v.Kind == "Firewall link" {
return false
}
if v.LinkDown {
return true
}
used := v.Master != "" || v.ReceiveBytes+v.TransmitBytes > 0
if v.Kind == "Physical" {
return used && (!v.Carrier || v.State == "down")
}
return used && v.State == "down"
}
func collectInterfaceFaults(v *NetworkInterface) {
base := filepath.Join("/sys/class/net", v.Name)
// /proc/net/dev folds missed packets into RX drop. Sysfs keeps software
// drops separate from rx_missed_errors, which must still trigger an alert.
if n, err := strconv.ParseUint(readTrimmed(filepath.Join(base, "statistics/rx_dropped")), 10, 64); err == nil {
v.ReceiveDrops = n
}
v.FaultCounters = map[string]uint64{}
for _, name := range []string{"rx_crc_errors", "rx_frame_errors", "rx_fifo_errors", "rx_missed_errors", "tx_fifo_errors", "tx_carrier_errors", "carrier_down_count"} {
path := filepath.Join(base, "statistics", name)
if name == "carrier_down_count" {
path = filepath.Join(base, name)
}
if n, err := strconv.ParseUint(readTrimmed(path), 10, 64); err == nil {
v.FaultCounters[name] = n
}
}
// Only Atlantic exposes the driver-specific names below. No configuration
// changes, and a missing ethtool is not itself a network failure.
driver, err := filepath.EvalSymlinks(filepath.Join(base, "device/driver"))
if err != nil || filepath.Base(driver) != "atlantic" {
return
}
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
if out, err := exec.CommandContext(ctx, "ethtool", "-S", v.Name).Output(); err == nil {
for k, n := range parseAtlanticFaults(string(out)) {
v.FaultCounters["atlantic:"+k] = n
}
}
}
func parseAtlanticFaults(output string) map[string]uint64 {
result := map[string]uint64{}
for _, line := range strings.Split(output, "\n") {
key, value, ok := strings.Cut(strings.TrimSpace(line), ":")
if !ok {
continue
}
fields := strings.Fields(key)
if len(fields) == 0 {
continue
}
switch fields[len(fields)-1] {
case "InErrors", "InDroppedDma", "AllocFails", "SkbAllocFails", "Restarts":
if n, err := strconv.ParseUint(strings.TrimSpace(value), 10, 64); err == nil {
result[key] = n
}
}
}
return result
}

67
network_faults_test.go Normal file
View File

@@ -0,0 +1,67 @@
package main
import (
"strings"
"testing"
)
func TestNetworkDropsDoNotAlert(t *testing.T) {
for _, name := range []string{"enp6s0", "vmbr0", "fwbr100i0", "tap100i0"} {
v := NetworkInterface{Name: name, Carrier: true, State: "up", NewDrops: 15, NewReceiveDrops: 15}
metrics := dashboardMetrics{Network: NetworkMetrics{Interfaces: []NetworkInterface{v}}}
for _, a := range evaluateAlerts(metrics, AlertThresholds{}) {
if a.ID == "network-errors" {
t.Fatalf("%s: drops triggered alert", name)
}
}
v.NewErrors = 1
metrics.Network.Interfaces[0] = v
found := false
for _, a := range evaluateAlerts(metrics, AlertThresholds{}) {
if a.ID == "network-errors" {
found = a.Severity == "critical"
}
}
if !found {
t.Fatalf("%s: errors did not trigger critical alert", name)
}
}
}
func TestNetworkHardwareFaultAndLinkAlerts(t *testing.T) {
for _, v := range []NetworkInterface{{Name: "eth0", NewFaults: 1}, {Name: "eth0", LinkDown: true}} {
found := false
for _, a := range evaluateAlerts(dashboardMetrics{Network: NetworkMetrics{Interfaces: []NetworkInterface{v}}}, AlertThresholds{}) {
if strings.HasPrefix(a.ID, "network-") && a.Severity == "critical" {
found = true
}
}
if !found {
t.Fatalf("no critical alert for %+v", v)
}
}
}
func TestNetworkCounterResetsAndFaultBaselines(t *testing.T) {
old := NetworkInterface{ReceiveBytes: 100, ReceiveErrors: 20, TransmitErrors: 4, ReceiveDrops: 8, FaultCounters: map[string]uint64{"rx_missed_errors": 10, "atlantic:InErrors": 9}, Carrier: true}
next := NetworkInterface{ReceiveBytes: 1, ReceiveErrors: 1, TransmitErrors: 6, ReceiveDrops: 10, FaultCounters: map[string]uint64{"rx_missed_errors": 12, "atlantic:InErrors": 1, "new": 100}, Carrier: false}
updateInterfaceDeltas(&next, old, 2)
if next.NewErrors != 2 || next.NewFaults != 2 || next.NewReceiveDrops != 2 || next.ReceivePerSec != 0 || !next.LinkDown {
t.Fatalf("bad deltas: %+v", next)
}
}
func TestAtlanticFaults(t *testing.T) {
got := parseAtlanticFaults("NIC statistics:\n InErrors: 2\n InDroppedDma: 3\n Queue[0] AllocFails: 4\n Queue[1] SkbAllocFails: 5\n Queue[0] Restarts: 6\n Queue[0] InErrors: 7\n InPackets: 100\n Queue[0] XdpDrop: 50\n InErrorsBad: nope")
if len(got) != 6 || got["InDroppedDma"] != 3 || got["Queue[0] Restarts"] != 6 {
t.Fatalf("bad counters: %v", got)
}
}
func TestFirewallNotAttributedToGuest(t *testing.T) {
n := NetworkMetrics{Interfaces: []NetworkInterface{{Name: "fwbr100i0", ReceiveDrops: 100}, {Name: "tap100i0"}}}
decorateNetworkInterfaces(&n, GuestsMetrics{})
if n.Interfaces[0].VMID != 0 || n.Interfaces[1].VMID != 100 {
t.Fatalf("bad VM attribution: %+v", n)
}
}

View File

@@ -3,6 +3,7 @@ package main
import (
"os"
"path/filepath"
"strings"
"testing"
)
@@ -33,8 +34,28 @@ func TestReadNetworkSnapshot(t *testing.T) {
}
}
func TestExplainNetworkHealthDistinguishesDNSFailure(t *testing.T) {
health := NetworkHealth{Gateway: "192.168.0.1", GatewayUp: true, InternetUp: true, DNSUp: false}
explainNetworkHealth(&health, nil)
if !strings.Contains(health.Summary, "DNS") || health.RecommendedAction == "" {
t.Fatalf("непонятная диагностика: %+v", health)
}
}
func TestExplainNetworkHealthIgnoresOldInterfaceCounters(t *testing.T) {
health := NetworkHealth{Gateway: "192.168.0.1", GatewayUp: true, InternetUp: true, DNSUp: true}
explainNetworkHealth(&health, []NetworkInterface{{Name: "eno1", Carrier: true, State: "up", ReceiveDrops: 42}})
if health.Summary != "Сеть работает штатно" {
t.Fatalf("старый накопленный счётчик вызвал текущую тревогу: %+v", health)
}
explainNetworkHealth(&health, []NetworkInterface{{Name: "eno1", Carrier: true, State: "up", ReceiveDrops: 43, NewDrops: 1}})
if health.Summary != "Сеть работает штатно" {
t.Fatalf("software drops вызвали тревогу: %+v", health)
}
}
func TestInterfaceVMID(t *testing.T) {
for name, want := range map[string]int{"tap100i0": 100, "veth108i0": 108, "fwbr113i0": 113, "vmbr0": 0, "enp6s0": 0} {
for name, want := range map[string]int{"tap100i0": 100, "veth108i0": 108, "fwbr113i0": 0, "vmbr0": 0, "enp6s0": 0} {
if got := interfaceVMID(name); got != want {
t.Fatalf("%s: got %d want %d", name, got, want)
}

16
scripts/build-release.sh Executable file
View File

@@ -0,0 +1,16 @@
#!/bin/sh
set -eu
version=${1:?Usage: build-release.sh vX.Y.Z}
cd "$(dirname "$0")/.."
go test ./...
mkdir -p dist
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags "-s -w -X main.version=$version" -o dist/proxmox-dashboard-linux-amd64 .
(cd dist && shasum -a 256 proxmox-dashboard-linux-amd64 > proxmox-dashboard-linux-amd64.sha256)
stage=$(mktemp -d)
trap 'rm -rf "$stage"' EXIT INT TERM
package="$stage/proxmox-dashboard-install"
mkdir -p "$package/dist" "$package/scripts"
cp dist/proxmox-dashboard-linux-amd64 dist/proxmox-dashboard-linux-amd64.sha256 "$package/dist/"
cp scripts/install.sh scripts/install-agent.sh scripts/uninstall-agent.sh scripts/proxmox-dashboard-update "$package/scripts/"
cp ./*.service ./*.timer ./*.path README.md "$package/"
tar -czf dist/proxmox-dashboard-install-amd64.tar.gz -C "$stage" proxmox-dashboard-install

65
scripts/install-agent.sh Executable file
View File

@@ -0,0 +1,65 @@
#!/bin/sh
set -eu
if [ "$(id -u)" -ne 0 ]; then
echo "Запустите команду установки от root." >&2
exit 1
fi
if [ -z "${DASHBOARD_URL:-}" ] || [ -z "${ENROLL_TOKEN:-}" ]; then
echo "Не заданы DASHBOARD_URL и ENROLL_TOKEN. Создайте команду в настройках Dashboard." >&2
exit 1
fi
GITEA_URL=${GITEA_URL:-https://git.myown.center}
GITEA_REPOSITORY=${GITEA_REPOSITORY:-maxim/ProxmoxDash}
VMID=${VMID:-0}
AGENT_NAME=${AGENT_NAME:-$(hostname)}
release_json=$(curl -fsSL "$GITEA_URL/api/v1/repos/$GITEA_REPOSITORY/releases/latest")
download_url=$(printf '%s' "$release_json" | sed -n 's/.*"browser_download_url":"\([^"]*\/proxmox-dashboard-linux-amd64\)".*/\1/p' | head -n 1)
if [ -z "$download_url" ]; then
echo "В последнем релизе не найден Linux-бинарник." >&2
exit 1
fi
install -d -m 0700 /var/lib/proxmox-dashboard-agent
temporary_binary=$(mktemp /tmp/proxmox-dashboard-agent.XXXXXX)
trap 'rm -f "$temporary_binary"' EXIT
curl -fsSL "$download_url" -o "$temporary_binary"
install -m 0755 "$temporary_binary" /var/lib/proxmox-dashboard-agent/proxmox-dashboard-agent
umask 077
{
printf 'DASHBOARD_URL=%s\n' "$DASHBOARD_URL"
printf 'ENROLL_TOKEN=%s\n' "$ENROLL_TOKEN"
printf 'AGENT_NAME=%s\n' "$AGENT_NAME"
printf 'VMID=%s\n' "$VMID"
} > /etc/proxmox-dashboard-agent.env
cat > /etc/systemd/system/proxmox-dashboard-agent.service <<'UNIT'
[Unit]
Description=Proxmox Dashboard read-only guest agent
After=network-online.target docker.service
Wants=network-online.target
[Service]
Type=simple
EnvironmentFile=/etc/proxmox-dashboard-agent.env
ExecStart=/var/lib/proxmox-dashboard-agent/proxmox-dashboard-agent --agent --agent-server ${DASHBOARD_URL} --agent-enroll-token ${ENROLL_TOKEN} --agent-name ${AGENT_NAME} --agent-vmid ${VMID}
Restart=always
RestartSec=5
NoNewPrivileges=true
ProtectHome=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/var/lib/proxmox-dashboard-agent
[Install]
WantedBy=multi-user.target
UNIT
systemctl daemon-reload
systemctl enable proxmox-dashboard-agent.service
systemctl restart proxmox-dashboard-agent.service
rm -f /usr/local/bin/proxmox-dashboard-agent
echo "Агент установлен. Через несколько секунд он появится в Dashboard."

16
scripts/uninstall-agent.sh Executable file
View File

@@ -0,0 +1,16 @@
#!/bin/sh
set -eu
if [ "$(id -u)" -ne 0 ]; then
echo "Запустите удаление от root." >&2
exit 1
fi
systemctl disable --now proxmox-dashboard-agent.service 2>/dev/null || true
rm -f /etc/systemd/system/proxmox-dashboard-agent.service
rm -f /etc/proxmox-dashboard-agent.env
rm -f /usr/local/bin/proxmox-dashboard-agent
rm -rf /var/lib/proxmox-dashboard-agent
systemctl daemon-reload
systemctl reset-failed proxmox-dashboard-agent.service 2>/dev/null || true
echo "Агент и его локальные учётные данные удалены."

110
service_links.go Normal file
View File

@@ -0,0 +1,110 @@
package main
import (
"fmt"
"sort"
"strings"
)
type ServiceLink struct {
ServiceID int64 `json:"serviceId"`
AgentID string `json:"agentId"`
VMID int `json:"vmid"`
ComponentType string `json:"componentType"`
ComponentName string `json:"componentName"`
Port int `json:"port"`
}
func (s *Store) ServiceLinks() ([]ServiceLink, error) {
rows, err := s.db.Query(`SELECT service_id,agent_id,vmid,component_type,component_name,port FROM service_links ORDER BY service_id`)
if err != nil {
return nil, err
}
defer rows.Close()
var out []ServiceLink
for rows.Next() {
var v ServiceLink
if err = rows.Scan(&v.ServiceID, &v.AgentID, &v.VMID, &v.ComponentType, &v.ComponentName, &v.Port); err != nil {
return nil, err
}
out = append(out, v)
}
return out, rows.Err()
}
func (s *Store) SaveServiceLink(v ServiceLink) (ServiceLink, error) {
v.AgentID = strings.TrimSpace(v.AgentID)
v.ComponentType = strings.TrimSpace(v.ComponentType)
v.ComponentName = strings.TrimSpace(v.ComponentName)
if v.ServiceID <= 0 {
return v, fmt.Errorf("выберите сервис")
}
if v.AgentID == "" && v.VMID <= 0 {
return v, fmt.Errorf("выберите агент или VMID")
}
if v.ComponentType != "" && v.ComponentType != "systemd" && v.ComponentType != "docker" && v.ComponentType != "process" {
return v, fmt.Errorf("неизвестный тип компонента")
}
if v.Port < 0 || v.Port > 65535 {
return v, fmt.Errorf("некорректный порт")
}
_, err := s.db.Exec(`INSERT INTO service_links(service_id,agent_id,vmid,component_type,component_name,port) VALUES(?,?,?,?,?,?) ON CONFLICT(service_id) DO UPDATE SET agent_id=excluded.agent_id,vmid=excluded.vmid,component_type=excluded.component_type,component_name=excluded.component_name,port=excluded.port`, v.ServiceID, v.AgentID, v.VMID, v.ComponentType, v.ComponentName, v.Port)
return v, err
}
func (s *Store) DeleteServiceLink(serviceID int64) error {
_, err := s.db.Exec(`DELETE FROM service_links WHERE service_id=?`, serviceID)
return err
}
func groupLinkedAlerts(metrics dashboardMetrics, alerts []Alert) []Alert {
if len(metrics.ServiceLinks) == 0 {
return alerts
}
agentOffline := map[string]ManagedAgent{}
for _, agent := range metrics.Agents {
if !agent.Online {
agentOffline[agent.ID] = agent
}
}
affected := map[string][]string{}
remove := map[string]bool{}
for _, link := range metrics.ServiceLinks {
agent, offline := agentOffline[link.AgentID]
if !offline {
continue
}
for _, service := range metrics.Services.Services {
if service.ID != link.ServiceID {
continue
}
for _, endpoint := range service.Endpoints {
if !endpoint.Up {
key := fmt.Sprintf("service-%d-%s", service.ID, endpoint.Kind)
remove[key] = true
affected[agent.ID] = append(affected[agent.ID], service.Name)
}
}
}
}
if len(affected) == 0 {
return alerts
}
out := make([]Alert, 0, len(alerts))
for _, alert := range alerts {
skip := remove[alert.ID]
for agentID := range affected {
if alert.ID == "agent-offline-"+agentID+fmt.Sprintf("-%d", agentOffline[agentID].VMID) {
skip = true
}
}
if !skip {
out = append(out, alert)
}
}
for agentID, names := range affected {
sort.Strings(names)
agent := agentOffline[agentID]
out = append(out, Alert{ID: "group-agent-offline-" + agentID + fmt.Sprintf("-%d", agent.VMID), Severity: "critical", Source: "Корреляция", Title: agent.Name + ": агент недоступен, связанные сервисы не отвечают", Message: fmt.Sprintf("VMID %d. Затронуто сервисов: %d — %s. Вероятная первопричина: недоступна гостевая система или агент.", agent.VMID, len(names), strings.Join(names, ", "))})
}
return out
}

View File

@@ -74,6 +74,20 @@ type ServiceHistoryPoint struct {
LatencyMS float64 `json:"latencyMs"`
}
var serviceHTTPTransport = &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: (&net.Dialer{Timeout: 10 * time.Second, KeepAlive: 30 * time.Second}).DialContext,
ForceAttemptHTTP2: true,
MaxIdleConns: 32,
MaxIdleConnsPerHost: 2,
IdleConnTimeout: 30 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
ExpectContinueTimeout: time.Second,
TLSClientConfig: &tls.Config{InsecureSkipVerify: true, MinVersion: tls.VersionTLS12},
}
var serviceHTTPClient = &http.Client{Transport: serviceHTTPTransport}
func (s *Store) Services() ([]MonitoredService, error) {
rows, err := s.db.Query(`SELECT id,name,url,local_address,check_type,expected_code,expected_text,timeout_seconds,enabled FROM services ORDER BY id`)
if err != nil {
@@ -296,14 +310,12 @@ func checkServiceEndpoint(v MonitoredService, kind, label, target string) Servic
st.Error = strings.TrimSpace(string(out))
}
default:
tr := &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true, MinVersion: tls.VersionTLS12}}
client := &http.Client{Transport: tr, Timeout: time.Duration(v.TimeoutSeconds) * time.Second}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, target, nil)
if err != nil {
st.Error = err.Error()
break
}
resp, err := client.Do(req)
resp, err := serviceHTTPClient.Do(req)
if err != nil {
st.Error = err.Error()
} else {

View File

@@ -1,6 +1,9 @@
package main
import "testing"
import (
"testing"
"time"
)
func TestServiceCRUD(t *testing.T) {
store, err := openStore(":memory:")
@@ -38,3 +41,15 @@ func TestValidateServiceChecksBothHTTPAddresses(t *testing.T) {
t.Fatal(err)
}
}
func TestHTTPServiceTransportHasBoundedIdlePool(t *testing.T) {
if serviceHTTPClient.Transport != serviceHTTPTransport {
t.Fatal("проверки сервисов должны использовать общий HTTP transport")
}
if serviceHTTPTransport.IdleConnTimeout < time.Second {
t.Fatal("idle-соединения должны автоматически закрываться")
}
if serviceHTTPTransport.MaxIdleConnsPerHost < 1 || serviceHTTPTransport.MaxIdleConnsPerHost > 4 {
t.Fatalf("небезопасный размер пула на хост: %d", serviceHTTPTransport.MaxIdleConnsPerHost)
}
}

View File

@@ -138,6 +138,11 @@ func openStore(path string) (*Store, error) {
`CREATE INDEX IF NOT EXISTS service_checks_service_ts_idx ON service_checks(service_id,ts)`,
`CREATE TABLE IF NOT EXISTS entity_history (kind TEXT NOT NULL,name TEXT NOT NULL,ts INTEGER NOT NULL,value1 REAL,value2 REAL,PRIMARY KEY(kind,name,ts))`,
`CREATE INDEX IF NOT EXISTS entity_history_kind_ts_idx ON entity_history(kind,ts)`,
`CREATE TABLE IF NOT EXISTS agent_enrollment_tokens (token_hash TEXT PRIMARY KEY, created_at INTEGER NOT NULL, expires_at INTEGER NOT NULL, used_at INTEGER)`,
`CREATE TABLE IF NOT EXISTS agents (id TEXT PRIMARY KEY, name TEXT NOT NULL, vmid INTEGER NOT NULL DEFAULT 0, hostname TEXT NOT NULL DEFAULT '', machine_id TEXT NOT NULL DEFAULT '', secret_hash TEXT NOT NULL, enrolled_at INTEGER NOT NULL, last_seen INTEGER NOT NULL, version TEXT NOT NULL DEFAULT '', report_json TEXT NOT NULL DEFAULT '{}', remote_address TEXT NOT NULL DEFAULT '')`,
`CREATE TABLE IF NOT EXISTS agent_actions (id INTEGER PRIMARY KEY AUTOINCREMENT, agent_id TEXT NOT NULL, action TEXT NOT NULL, target TEXT NOT NULL DEFAULT '', requested_at INTEGER NOT NULL, started_at INTEGER, completed_at INTEGER, status TEXT NOT NULL DEFAULT 'queued', result TEXT NOT NULL DEFAULT '', FOREIGN KEY(agent_id) REFERENCES agents(id) ON DELETE CASCADE)`,
`CREATE INDEX IF NOT EXISTS agent_actions_agent_status_idx ON agent_actions(agent_id,status)`,
`CREATE TABLE IF NOT EXISTS service_links (service_id INTEGER PRIMARY KEY, agent_id TEXT NOT NULL DEFAULT '', vmid INTEGER NOT NULL DEFAULT 0, component_type TEXT NOT NULL DEFAULT '', component_name TEXT NOT NULL DEFAULT '', port INTEGER NOT NULL DEFAULT 0, FOREIGN KEY(service_id) REFERENCES services(id) ON DELETE CASCADE)`,
}
for _, statement := range statements {
if _, err := db.Exec(statement); err != nil {
@@ -147,6 +152,9 @@ func openStore(path string) (*Store, error) {
}
// Older databases get this column without requiring a separate migration tool.
_, _ = db.Exec(`ALTER TABLE alert_events ADD COLUMN recovery_sent_at INTEGER`)
_, _ = db.Exec(`ALTER TABLE alert_events ADD COLUMN planned INTEGER NOT NULL DEFAULT 0`)
_, _ = db.Exec(`ALTER TABLE alert_events ADD COLUMN maintenance_reason TEXT NOT NULL DEFAULT ''`)
_, _ = db.Exec(`DELETE FROM alert_events WHERE alert_key='monitor-slow'`)
_, _ = db.Exec(`ALTER TABLE service_checks ADD COLUMN endpoint TEXT NOT NULL DEFAULT 'public'`)
_, _ = db.Exec(`PRAGMA foreign_keys=ON`)
return &Store{db: db, path: path}, nil
@@ -230,7 +238,7 @@ func (s *Store) SaveThresholds(value AlertThresholds) error {
}
func defaultEmailSettings() EmailSettings {
return EmailSettings{Port: 587, IntervalMinutes: 60, Severities: []string{"critical", "warning"}, Sources: []string{"CPU", "Память", "Storage", "Диски", "ZFS", "UPS", "VM/LXC", "Proxmox", "Dashboard", "Сервисы", "Система", "Сеть"}}
return EmailSettings{Port: 587, IntervalMinutes: 60, Severities: []string{"critical", "warning"}, Sources: []string{"CPU", "Память", "Storage", "Диски", "ZFS", "UPS", "VM/LXC", "Proxmox", "Dashboard", "Сервисы", "Система", "Сеть", "Агенты", "Docker"}}
}
func (s *Store) EmailSettings(includePassword bool) EmailSettings {

372
web.go

File diff suppressed because one or more lines are too long