Add guest root cause diagnostics

README.md

@@ -3,8 +3,139 @@
 Локальная веб-панель CPU для Proxmox. Данные читаются напрямую из `/proc` и
 `/sys`, без Proxmox API. Интерфейс обновляется каждые две секунды.
 
+## Установка для новичка — скопируйте команды целиком
+
+### Что потребуется
+
+- сервер Proxmox VE с доступом в интернет или к `git.myown.center`;
+- вход в консоль Proxmox под пользователем `root`;
+- IP-адрес Proxmox в локальной сети;
+- свободный TCP-порт `9105` между вашим компьютером и Proxmox.
+
+Go, Node.js, Docker и отдельную базу данных устанавливать не нужно. Панель
+поставляется готовым Linux-бинарником, а SQLite встроен внутрь.
+
+### 1. Подключитесь к Proxmox
+
+На своём компьютере откройте Terminal и выполните команду, заменив IP на адрес
+своего Proxmox:
+
+```bash
+ssh root@192.168.0.10
+```
+
+Введите пароль `root`. Следующие команды выполняйте уже внутри Proxmox.
+
+### 2. Установите панель
+
+Скопируйте весь блок и вставьте его в консоль Proxmox:
+
+```bash
+apt-get update
+apt-get install -y curl ca-certificates tar smartmontools iproute2 iputils-ping
+
+cd /tmp
+curl -fL \
+  https://git.myown.center/maxim/ProxmoxDash/releases/download/latest/proxmox-dashboard-install-amd64.tar.gz \
+  -o proxmox-dashboard-install-amd64.tar.gz
+
+tar -xzf proxmox-dashboard-install-amd64.tar.gz
+cd proxmox-dashboard-install
+UPDATE_MODE=notify ./scripts/install.sh
+```
+
+Готово. Установщик сам:
+
+- положит бинарник в `/usr/local/bin`;
+- создаст локальную базу настроек и истории;
+- установит systemd-службу;
+- запустит панель;
+- настроит ежедневную проверку новых версий;
+- сохранит предыдущий бинарник при будущем обновлении.
+
+### 3. Откройте панель
+
+В браузере на своём компьютере откройте:
+
+```text
+http://192.168.0.10:9105
+```
+
+Замените `192.168.0.10` на IP своего Proxmox. Узнать адрес можно командой:
+
+```bash
+hostname -I
+```
+
+### 4. Проверьте, что всё работает
+
+```bash
+systemctl --no-pager --full status proxmox-cpu-dashboard.service
+curl -I http://127.0.0.1:9105
+```
+
+В статусе службы должно быть `active (running)`, а локальная проверка должна
+вернуть HTTP-ответ `200 OK`.
+
+### Как обновлять
+
+В режиме `notify`, установленном командами выше, откройте в панели
+`Настройки → Обновления` и нажмите `Проверить и установить обновление`.
+
+Если нужны полностью автоматические обновления, повторно запустите установщик:
+
+```bash
+cd /tmp/proxmox-dashboard-install
+UPDATE_MODE=auto ./scripts/install.sh
+```
+
+### Если репозиторий станет приватным
+
+Создайте в Gitea токен только для чтения репозитория и передайте его
+установщику. Вместо текста `ВАШ_ТОКЕН` вставьте настоящий токен:
+
+```bash
+cd /tmp/proxmox-dashboard-install
+GITEA_TOKEN='ВАШ_ТОКЕН' UPDATE_MODE=notify ./scripts/install.sh
+```
+
+Токен сохраняется на Proxmox с правами `0600` и не показывается в браузере.
+
+### Если страница не открывается
+
+Выполните на Proxmox:
+
+```bash
+systemctl restart proxmox-cpu-dashboard.service
+systemctl --no-pager --full status proxmox-cpu-dashboard.service
+journalctl -u proxmox-cpu-dashboard.service -n 100 --no-pager
+ss -lntp | grep 9105
+```
+
+Проверьте, что используете правильный IP, порт `9105` доступен из локальной
+сети, а firewall Proxmox не блокирует подключение. Панель пока не имеет
+авторизации — не публикуйте порт `9105` напрямую в интернет.
+
+---
+
 Панель показывает:
 
+- изменение критичных SMART-счётчиков, результаты self-test и исчезновение дисков;
+- realtime read/write IOPS, MB/s, latency, utilization и глубину очереди каждого диска;
+- недельную историю IOPS/utilization и алерты при устойчивом I/O-давлении;
+- вероятный источник I/O: VM/LXC, процесс хоста или активная backup/snapshot/migration-задача;
+- отдельные пороги HDD, SATA SSD и NVMe с исключениями по имени или серийному номеру;
+- недельные baseline-аномалии IOPS, latency, температуры, RAM, ZFS и длительности backup;
+- корреляцию связанных событий в одну причинно-следственную цепочку;
+- время первого появления и последнего наблюдения каждого уведомления;
+- автоматический короткий SMART self-test раз в неделю для дисков без свежего теста;
+- datasets, vdev-классы, ошибки read/write/checksum, scrub и ARC ZFS;
+- доступность шлюза, DNS и интернета, packet loss, latency и внешний IP;
+- тип, состояние, скорость, duplex и master сетевых интерфейсов;
+- состояние связи с NUT/apcupsd, возраст батареи и расчётную автономность UPS;
+- настраиваемый сценарий корректного выключения VM/LXC и Proxmox с безопасным dry-run;
+- самоконтроль сборщика, записи истории, свободного места базы и systemd watchdog;
+
 - модель и текущую загрузку CPU;
 - количество сокетов, физических ядер и логических потоков;
 - среднюю текущую частоту;
@@ -191,6 +322,67 @@ apcaccess status
 - `GET /api/events` — поток метрик SSE с интервалом две секунды.
 - `GET/POST/PUT/DELETE /api/services` — управление проверками сервисов.
 - `GET /api/services/history` — семидневная история доступности и отклика.
+- `GET/PUT/DELETE /api/settings/maintenance` — состояние, запуск и досрочное завершение плановых работ.
+- `GET/DELETE /api/agents` — подключённые гостевые агенты.
+- `POST /api/agents/enrollment-token` — одноразовый токен регистрации на 15 минут.
+
+## Maintenance-режим
+
+Во вкладке `Настройки → Maintenance` можно на заданное время отключить все
+алерты либо выбрать конкретные сервисы и VM/LXC. Подавленные события не
+отправляются по почте и не отображаются как активные, но сохраняются в истории
+с причиной плановых работ. После окончания окна обычный мониторинг включается
+автоматически.
+
+## Агенты VM и LXC
+
+Read-only агент показывает состояние Linux внутри VM/LXC: CPU, load average,
+память, корневой раздел, uptime, ОС, ядро, IP-адреса, systemd-службы, доступные
+APT-обновления и необходимость перезагрузки. Для Docker отображаются контейнеры,
+health-check, число перезапусков, CPU и память. Агент сам подключается к Dashboard
+каждые 10 секунд; открывать входящий порт внутри гостевой системы не нужно.
+
+В подробностях агента systemd-службы разделяются на системные, установленные
+дополнительно и локальные unit-файлы. Для каждой службы показываются пакет и путь
+unit-файла, время запуска, число перезапусков, последний exit code, память и
+накопленное CPU-время, когда соответствующий accounting доступен в systemd.
+
+Расширенная диагностика агента включает последние сообщения проблемных служб,
+`Requires`/`After`, PID, команду запуска и слушающие порты; выявляет частые
+рестарты, OOM-kill и процессы в D-state. Отдельно отображаются top процессов по
+CPU и RAM с накопленным I/O, swap, PSI pressure, все постоянные файловые системы,
+их заполнение и использование inode.
+
+Панель «Диагностика homelab» и карта сопоставляют известные IP и порты с VM/LXC,
+агентом, systemd-службой или Docker-контейнером и настроенной HTTP/TCP-проверкой.
+Связи по PID и порту считаются подтверждёнными; совпадения только по названию
+помечаются как предположение.
+
+Откройте `Настройки → Агенты`, укажите адрес Dashboard, доступный из VM/LXC, и
+VMID. Кнопка создаст готовую команду установки. Одноразовый токен действует 15
+минут и после регистрации заменяется индивидуальным секретом агента.
+
+После однократной установки агент обновляется автоматически вслед за Dashboard.
+Центральный сервер отдаёт агенту собственный бинарник и SHA-256; агент проверяет
+контрольную сумму, атомарно заменяет файл и перезапускается через systemd. Если
+агент был удалён из Dashboard, повторная команда с новым токеном автоматически
+выполнит новую регистрацию.
+
+В первой версии агент не выполняет команды и не перезапускает службы или
+контейнеры. Для чтения Docker ему требуется доступ к `/var/run/docker.sock`,
+поэтому служба устанавливается от root. Сам Dashboard пока не имеет
+авторизации: используйте локальную сеть или HTTPS через доверенный reverse
+proxy и не публикуйте агентские API напрямую в интернет.
+
+Полностью удалить агент из VM/LXC:
+
+```bash
+curl -fsSL https://git.myown.center/maxim/ProxmoxDash/raw/branch/main/scripts/uninstall-agent.sh | sh
+```
+
+Оформление, пороги, почта и обновления Dashboard находятся в обычных настройках.
+Рабочие инструменты — сервисы, агенты, плановые работы и UPS-сценарий — вынесены
+в отдельный раздел `Управление` в шапке Dashboard.
 
 ## Сборка
 

activity.go

@@ -57,7 +57,7 @@ func (c *activityCollector) collect(guests GuestsMetrics) ProxmoxActivity {
 	value := c.value
 	c.mu.RUnlock()
 	c.mu.Lock()
-	if !c.refreshing && time.Since(time.Unix(value.CheckedAt, 0)) > 10*time.Minute {
+	if !c.refreshing && time.Since(time.Unix(value.CheckedAt, 0)) > 15*time.Second {
 		c.refreshing = true
 		go c.refreshWithGuests(guests)
 	}

agent.go

@@ -0,0 +1,957 @@
+package main
+
+import (
+	"bytes"
+	"context"
+	"crypto/hmac"
+	"crypto/rand"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net"
+	"net/http"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"runtime"
+	"strconv"
+	"strings"
+	"sync"
+	"syscall"
+	"time"
+)
+
+const agentOfflineAfter = 45 * time.Second
+
+type agentOptions struct {
+	Server, EnrollmentToken, Name, ConfigPath string
+	VMID                                      int
+}
+
+type agentConfig struct {
+	Server string `json:"server"`
+	ID     string `json:"id"`
+	Secret string `json:"secret"`
+	Name   string `json:"name"`
+	VMID   int    `json:"vmid"`
+}
+
+type AgentContainer struct {
+	ID          string   `json:"id"`
+	Name        string   `json:"name"`
+	Image       string   `json:"image"`
+	State       string   `json:"state"`
+	Status      string   `json:"status"`
+	Health      string   `json:"health,omitempty"`
+	Restarts    int64    `json:"restarts"`
+	CPUPercent  float64  `json:"cpuPercent"`
+	MemoryUsage uint64   `json:"memoryUsage"`
+	Ports       []string `json:"ports,omitempty"`
+}
+
+type AgentInterface struct {
+	Name      string   `json:"name"`
+	Addresses []string `json:"addresses"`
+}
+
+type AgentService struct {
+	Name        string     `json:"name"`
+	State       string     `json:"state"`
+	SubState    string     `json:"subState"`
+	Description string     `json:"description"`
+	Origin      string     `json:"origin"`
+	UnitPath    string     `json:"unitPath,omitempty"`
+	Package     string     `json:"package,omitempty"`
+	Restarts    uint64     `json:"restarts"`
+	ExitCode    int        `json:"exitCode"`
+	StartedAt   int64      `json:"startedAt,omitempty"`
+	MemoryBytes uint64     `json:"memoryBytes,omitempty"`
+	CPUSeconds  float64    `json:"cpuSeconds,omitempty"`
+	MainPID     int        `json:"mainPid,omitempty"`
+	ExecStart   string     `json:"execStart,omitempty"`
+	Requires    []string   `json:"requires,omitempty"`
+	After       []string   `json:"after,omitempty"`
+	ListenPorts []int      `json:"listenPorts,omitempty"`
+	Logs        []AgentLog `json:"logs,omitempty"`
+	Flapping    bool       `json:"flapping"`
+}
+
+type AgentIssue struct {
+	Time    int64  `json:"time"`
+	Unit    string `json:"unit"`
+	Message string `json:"message"`
+}
+
+type AgentReport struct {
+	Hostname         string            `json:"hostname"`
+	MachineID        string            `json:"machineId"`
+	Version          string            `json:"version"`
+	OS               string            `json:"os"`
+	Arch             string            `json:"arch"`
+	OSName           string            `json:"osName"`
+	Kernel           string            `json:"kernel"`
+	UptimeSeconds    int64             `json:"uptimeSeconds"`
+	Load1            float64           `json:"load1"`
+	CPUPercent       float64           `json:"cpuPercent"`
+	MemoryTotal      uint64            `json:"memoryTotal"`
+	MemoryUsed       uint64            `json:"memoryUsed"`
+	SwapTotal        uint64            `json:"swapTotal"`
+	SwapUsed         uint64            `json:"swapUsed"`
+	RootTotal        uint64            `json:"rootTotal"`
+	RootUsed         uint64            `json:"rootUsed"`
+	DockerAvailable  bool              `json:"dockerAvailable"`
+	DockerError      string            `json:"dockerError,omitempty"`
+	Containers       []AgentContainer  `json:"containers"`
+	Interfaces       []AgentInterface  `json:"interfaces"`
+	Services         []AgentService    `json:"services"`
+	Issues           []AgentIssue      `json:"issues"`
+	Processes        []AgentProcess    `json:"processes"`
+	Filesystems      []AgentFilesystem `json:"filesystems"`
+	ListenPorts      []AgentListenPort `json:"listenPorts"`
+	Pressure         AgentPressure     `json:"pressure"`
+	OOMKills         []AgentLog        `json:"oomKills"`
+	UpdatesAvailable int               `json:"updatesAvailable"`
+	RebootRequired   bool              `json:"rebootRequired"`
+	InventoryAt      int64             `json:"inventoryAt"`
+	CollectedAt      int64             `json:"collectedAt"`
+}
+
+type agentInventory struct {
+	OSName           string
+	Kernel           string
+	Services         []AgentService
+	UpdatesAvailable int
+	RebootRequired   bool
+	CollectedAt      time.Time
+}
+
+type AgentUpdateManifest struct {
+	Version string `json:"version"`
+	SHA256  string `json:"sha256"`
+	URL     string `json:"url"`
+}
+
+type agentReportResponse struct {
+	Accepted bool                 `json:"accepted"`
+	Update   *AgentUpdateManifest `json:"update,omitempty"`
+}
+
+var guestInventory struct {
+	sync.Mutex
+	value agentInventory
+}
+
+var guestIssues struct {
+	sync.Mutex
+	value       []AgentIssue
+	collectedAt time.Time
+}
+
+var agentBinaryInfo struct {
+	sync.Once
+	manifest AgentUpdateManifest
+	path     string
+	err      error
+}
+
+func currentAgentUpdate() (AgentUpdateManifest, string, error) {
+	agentBinaryInfo.Do(func() {
+		agentBinaryInfo.path, agentBinaryInfo.err = os.Executable()
+		if agentBinaryInfo.err != nil {
+			return
+		}
+		file, err := os.Open(agentBinaryInfo.path)
+		if err != nil {
+			agentBinaryInfo.err = err
+			return
+		}
+		defer file.Close()
+		hash := sha256.New()
+		if _, err = io.Copy(hash, file); err != nil {
+			agentBinaryInfo.err = err
+			return
+		}
+		agentBinaryInfo.manifest = AgentUpdateManifest{Version: version, SHA256: hex.EncodeToString(hash.Sum(nil)), URL: "/api/agent/binary"}
+	})
+	return agentBinaryInfo.manifest, agentBinaryInfo.path, agentBinaryInfo.err
+}
+
+type ManagedAgent struct {
+	ID            string      `json:"id"`
+	Name          string      `json:"name"`
+	VMID          int         `json:"vmid"`
+	Hostname      string      `json:"hostname"`
+	Version       string      `json:"version"`
+	LastSeen      int64       `json:"lastSeen"`
+	EnrolledAt    int64       `json:"enrolledAt"`
+	Online        bool        `json:"online"`
+	RemoteAddress string      `json:"remoteAddress"`
+	Report        AgentReport `json:"report"`
+}
+
+type agentEnrollmentRequest struct {
+	Token, Name, Hostname, MachineID string
+	VMID                             int
+}
+
+func randomHex(bytesCount int) (string, error) {
+	b := make([]byte, bytesCount)
+	if _, err := rand.Read(b); err != nil {
+		return "", err
+	}
+	return hex.EncodeToString(b), nil
+}
+
+func tokenHash(value string) string {
+	sum := sha256.Sum256([]byte(value))
+	return hex.EncodeToString(sum[:])
+}
+
+func (s *Store) CreateAgentEnrollmentToken() (string, error) {
+	token, err := randomHex(24)
+	if err != nil {
+		return "", err
+	}
+	_, err = s.db.Exec(`INSERT INTO agent_enrollment_tokens(token_hash,created_at,expires_at) VALUES(?,?,?)`, tokenHash(token), time.Now().Unix(), time.Now().Add(15*time.Minute).Unix())
+	return token, err
+}
+
+func (s *Store) EnrollAgent(req agentEnrollmentRequest) (agentConfig, error) {
+	tx, err := s.db.Begin()
+	if err != nil {
+		return agentConfig{}, err
+	}
+	defer tx.Rollback()
+	var expires int64
+	if err = tx.QueryRow(`SELECT expires_at FROM agent_enrollment_tokens WHERE token_hash=? AND used_at IS NULL`, tokenHash(req.Token)).Scan(&expires); err != nil || expires < time.Now().Unix() {
+		return agentConfig{}, errors.New("одноразовый токен недействителен или истёк")
+	}
+	id, err := randomHex(12)
+	if err != nil {
+		return agentConfig{}, err
+	}
+	secret, err := randomHex(32)
+	if err != nil {
+		return agentConfig{}, err
+	}
+	name := strings.TrimSpace(req.Name)
+	if name == "" {
+		name = req.Hostname
+	}
+	now := time.Now().Unix()
+	if _, err = tx.Exec(`INSERT INTO agents(id,name,vmid,hostname,machine_id,secret_hash,enrolled_at,last_seen,report_json,remote_address) VALUES(?,?,?,?,?,?,?,?,?,?)`, id, name, req.VMID, req.Hostname, req.MachineID, tokenHash(secret), now, now, `{}`, ""); err != nil {
+		return agentConfig{}, err
+	}
+	if _, err = tx.Exec(`UPDATE agent_enrollment_tokens SET used_at=? WHERE token_hash=?`, now, tokenHash(req.Token)); err != nil {
+		return agentConfig{}, err
+	}
+	if err = tx.Commit(); err != nil {
+		return agentConfig{}, err
+	}
+	return agentConfig{ID: id, Secret: secret, Name: name, VMID: req.VMID}, nil
+}
+
+func (s *Store) SaveAgentReport(id, secret, remote string, report AgentReport) error {
+	if !s.AuthenticateAgent(id, secret) {
+		return errors.New("агент не авторизован")
+	}
+	data, _ := json.Marshal(report)
+	_, err := s.db.Exec(`UPDATE agents SET hostname=?,version=?,last_seen=?,report_json=?,remote_address=? WHERE id=?`, report.Hostname, report.Version, time.Now().Unix(), string(data), remote, id)
+	return err
+}
+
+func (s *Store) AuthenticateAgent(id, secret string) bool {
+	var expected string
+	return s.db.QueryRow(`SELECT secret_hash FROM agents WHERE id=?`, id).Scan(&expected) == nil && hmac.Equal([]byte(expected), []byte(tokenHash(secret)))
+}
+
+func (s *Store) Agents() ([]ManagedAgent, error) {
+	rows, err := s.db.Query(`SELECT id,name,vmid,hostname,version,enrolled_at,last_seen,report_json,remote_address FROM agents ORDER BY name`)
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+	result := []ManagedAgent{}
+	for rows.Next() {
+		var a ManagedAgent
+		var raw string
+		if err = rows.Scan(&a.ID, &a.Name, &a.VMID, &a.Hostname, &a.Version, &a.EnrolledAt, &a.LastSeen, &raw, &a.RemoteAddress); err != nil {
+			return nil, err
+		}
+		_ = json.Unmarshal([]byte(raw), &a.Report)
+		a.Online = time.Since(time.Unix(a.LastSeen, 0)) < agentOfflineAfter
+		result = append(result, a)
+	}
+	return result, rows.Err()
+}
+
+func (s *Store) DeleteAgent(id string) error {
+	_, err := s.db.Exec(`DELETE FROM agents WHERE id=?`, id)
+	return err
+}
+
+func runGuestAgent(opts agentOptions) error {
+	config, err := loadAgentConfig(opts.ConfigPath)
+	if err != nil && !os.IsNotExist(err) {
+		return err
+	}
+	if opts.Server != "" {
+		config.Server = strings.TrimRight(opts.Server, "/")
+	}
+	if opts.Name != "" {
+		config.Name = opts.Name
+	}
+	if opts.VMID > 0 {
+		config.VMID = opts.VMID
+	}
+	if config.Server == "" {
+		return errors.New("укажите --agent-server")
+	}
+	if config.ID == "" {
+		if opts.EnrollmentToken == "" {
+			return errors.New("для первого запуска нужен --agent-enroll-token")
+		}
+		config, err = enrollRemote(config, opts.EnrollmentToken)
+		if err != nil {
+			return err
+		}
+		if err = saveAgentConfig(opts.ConfigPath, config); err != nil {
+			return err
+		}
+	}
+	client := &http.Client{Timeout: 12 * time.Second}
+	for {
+		report := collectAgentReport()
+		data, _ := json.Marshal(report)
+		req, _ := http.NewRequest(http.MethodPost, config.Server+"/api/agent/report", bytes.NewReader(data))
+		req.Header.Set("Content-Type", "application/json")
+		req.Header.Set("Authorization", "Bearer "+config.ID+"."+config.Secret)
+		response, requestErr := client.Do(req)
+		if requestErr == nil {
+			responseData, _ := io.ReadAll(io.LimitReader(response.Body, 64<<10))
+			response.Body.Close()
+			if response.StatusCode == http.StatusUnauthorized && opts.EnrollmentToken != "" {
+				fresh, enrollmentErr := enrollRemote(config, opts.EnrollmentToken)
+				if enrollmentErr == nil {
+					config = fresh
+					if saveErr := saveAgentConfig(opts.ConfigPath, config); saveErr != nil {
+						return saveErr
+					}
+					continue
+				}
+				requestErr = fmt.Errorf("повторная регистрация: %v", enrollmentErr)
+			}
+			if response.StatusCode >= 300 {
+				if requestErr == nil {
+					requestErr = fmt.Errorf("сервер вернул %s", response.Status)
+				}
+			}
+			if response.StatusCode == http.StatusOK {
+				var result agentReportResponse
+				if json.Unmarshal(responseData, &result) == nil && result.Update != nil && newerAgentVersion(result.Update.Version, version) {
+					if updateErr := applyAgentUpdate(client, config, *result.Update); updateErr != nil {
+						requestErr = fmt.Errorf("автообновление: %v", updateErr)
+					} else {
+						return nil
+					}
+				}
+			}
+		}
+		if requestErr != nil {
+			fmt.Fprintf(os.Stderr, "agent: %v\n", requestErr)
+		}
+		time.Sleep(10 * time.Second)
+	}
+}
+
+func newerAgentVersion(candidate, current string) bool {
+	parse := func(value string) [3]int {
+		value = strings.TrimPrefix(strings.TrimSpace(value), "v")
+		value = strings.SplitN(value, "-", 2)[0]
+		parts := strings.Split(value, ".")
+		var result [3]int
+		for i := 0; i < len(parts) && i < 3; i++ {
+			result[i], _ = strconv.Atoi(parts[i])
+		}
+		return result
+	}
+	if candidate == "" || current == "" || current == "dev" {
+		return false
+	}
+	next, installed := parse(candidate), parse(current)
+	for i := 0; i < 3; i++ {
+		if next[i] > installed[i] {
+			return true
+		}
+		if next[i] < installed[i] {
+			return false
+		}
+	}
+	return false
+}
+
+func applyAgentUpdate(client *http.Client, config agentConfig, update AgentUpdateManifest) error {
+	if !strings.HasPrefix(update.URL, "/api/agent/") || len(update.SHA256) != 64 {
+		return errors.New("сервер вернул некорректный манифест")
+	}
+	request, err := http.NewRequest(http.MethodGet, config.Server+update.URL, nil)
+	if err != nil {
+		return err
+	}
+	request.Header.Set("Authorization", "Bearer "+config.ID+"."+config.Secret)
+	updateClient := *client
+	updateClient.Timeout = 2 * time.Minute
+	response, err := updateClient.Do(request)
+	if err != nil {
+		return err
+	}
+	defer response.Body.Close()
+	if response.StatusCode != http.StatusOK {
+		return fmt.Errorf("загрузка вернула %s", response.Status)
+	}
+	executable, err := os.Executable()
+	if err != nil {
+		return err
+	}
+	temporary := executable + ".update"
+	_ = os.Remove(temporary)
+	file, err := os.OpenFile(temporary, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0755)
+	if err != nil {
+		return err
+	}
+	keep := false
+	defer func() {
+		file.Close()
+		if !keep {
+			_ = os.Remove(temporary)
+		}
+	}()
+	hash := sha256.New()
+	written, err := io.Copy(io.MultiWriter(file, hash), io.LimitReader(response.Body, (64<<20)+1))
+	if err != nil {
+		return err
+	}
+	if written > 64<<20 {
+		return errors.New("бинарник превышает допустимый размер")
+	}
+	if err = file.Sync(); err != nil {
+		return err
+	}
+	if err = file.Close(); err != nil {
+		return err
+	}
+	actual := hex.EncodeToString(hash.Sum(nil))
+	if !hmac.Equal([]byte(actual), []byte(strings.ToLower(update.SHA256))) {
+		return errors.New("SHA-256 обновления не совпал")
+	}
+	if err = os.Chmod(temporary, 0755); err != nil {
+		return err
+	}
+	if err = os.Rename(temporary, executable); err != nil {
+		return err
+	}
+	keep = true
+	return nil
+}
+
+func enrollRemote(config agentConfig, token string) (agentConfig, error) {
+	hostname, _ := os.Hostname()
+	machineID, _ := os.ReadFile("/etc/machine-id")
+	payload := agentEnrollmentRequest{Token: token, Name: config.Name, VMID: config.VMID, Hostname: hostname, MachineID: strings.TrimSpace(string(machineID))}
+	data, _ := json.Marshal(payload)
+	response, err := http.Post(config.Server+"/api/agent/enroll", "application/json", bytes.NewReader(data))
+	if err != nil {
+		return config, err
+	}
+	defer response.Body.Close()
+	if response.StatusCode != http.StatusOK {
+		message, _ := io.ReadAll(io.LimitReader(response.Body, 4096))
+		return config, fmt.Errorf("регистрация: %s", strings.TrimSpace(string(message)))
+	}
+	var credentials agentConfig
+	if err = json.NewDecoder(response.Body).Decode(&credentials); err != nil {
+		return config, err
+	}
+	credentials.Server = config.Server
+	return credentials, nil
+}
+
+func loadAgentConfig(path string) (agentConfig, error) {
+	data, err := os.ReadFile(path)
+	if err != nil {
+		return agentConfig{}, err
+	}
+	var c agentConfig
+	err = json.Unmarshal(data, &c)
+	return c, err
+}
+func saveAgentConfig(path string, c agentConfig) error {
+	if err := os.MkdirAll(filepath.Dir(path), 0700); err != nil {
+		return err
+	}
+	data, _ := json.MarshalIndent(c, "", "  ")
+	return os.WriteFile(path, data, 0600)
+}
+
+func collectAgentReport() AgentReport {
+	hostname, _ := os.Hostname()
+	machineID, _ := os.ReadFile("/etc/machine-id")
+	report := AgentReport{Hostname: hostname, MachineID: strings.TrimSpace(string(machineID)), Version: version, OS: runtime.GOOS, Arch: runtime.GOARCH, CollectedAt: time.Now().Unix()}
+	if data, err := os.ReadFile("/proc/uptime"); err == nil {
+		fields := strings.Fields(string(data))
+		if len(fields) > 0 {
+			value, _ := strconv.ParseFloat(fields[0], 64)
+			report.UptimeSeconds = int64(value)
+		}
+	}
+	if data, err := os.ReadFile("/proc/loadavg"); err == nil {
+		fields := strings.Fields(string(data))
+		if len(fields) > 0 {
+			report.Load1, _ = strconv.ParseFloat(fields[0], 64)
+		}
+	}
+	report.CPUPercent = sampleAgentCPU()
+	report.Interfaces = collectAgentInterfaces()
+	report.Processes = collectAgentProcesses()
+	report.Filesystems = collectAgentFilesystems()
+	report.ListenPorts = collectAgentListenPorts()
+	report.Pressure = collectAgentPressure()
+	report.OOMKills = cachedAgentOOMKills()
+	if data, err := os.ReadFile("/proc/meminfo"); err == nil {
+		values := map[string]uint64{}
+		for _, line := range strings.Split(string(data), "\n") {
+			f := strings.Fields(line)
+			if len(f) >= 2 {
+				v, _ := strconv.ParseUint(f[1], 10, 64)
+				values[strings.TrimSuffix(f[0], ":")] = v * 1024
+			}
+		}
+		report.MemoryTotal = values["MemTotal"]
+		available := values["MemAvailable"]
+		if report.MemoryTotal > available {
+			report.MemoryUsed = report.MemoryTotal - available
+		}
+		report.SwapTotal = values["SwapTotal"]
+		if report.SwapTotal > values["SwapFree"] {
+			report.SwapUsed = report.SwapTotal - values["SwapFree"]
+		}
+	}
+	var stat syscall.Statfs_t
+	if syscall.Statfs("/", &stat) == nil {
+		report.RootTotal = stat.Blocks * uint64(stat.Bsize)
+		free := stat.Bavail * uint64(stat.Bsize)
+		if report.RootTotal > free {
+			report.RootUsed = report.RootTotal - free
+		}
+	}
+	report.Containers, report.DockerError = collectDockerContainers()
+	report.DockerAvailable = report.DockerError == ""
+	inventory := cachedAgentInventory()
+	report.OSName, report.Kernel, report.Services = inventory.OSName, inventory.Kernel, inventory.Services
+	report.Issues = cachedAgentIssues()
+	report.UpdatesAvailable, report.RebootRequired, report.InventoryAt = inventory.UpdatesAvailable, inventory.RebootRequired, inventory.CollectedAt.Unix()
+	return report
+}
+
+func collectAgentInterfaces() []AgentInterface {
+	interfaces, err := net.Interfaces()
+	if err != nil {
+		return nil
+	}
+	result := []AgentInterface{}
+	for _, iface := range interfaces {
+		if iface.Flags&net.FlagLoopback != 0 {
+			continue
+		}
+		addresses, _ := iface.Addrs()
+		item := AgentInterface{Name: iface.Name}
+		for _, address := range addresses {
+			value := address.String()
+			if host, _, err := net.ParseCIDR(value); err == nil {
+				value = host.String()
+			}
+			item.Addresses = append(item.Addresses, value)
+		}
+		if len(item.Addresses) > 0 {
+			result = append(result, item)
+		}
+	}
+	return result
+}
+
+func cachedAgentInventory() agentInventory {
+	guestInventory.Lock()
+	defer guestInventory.Unlock()
+	if time.Since(guestInventory.value.CollectedAt) < 15*time.Minute {
+		return guestInventory.value
+	}
+	value := agentInventory{CollectedAt: time.Now()}
+	value.OSName = readOSPrettyName()
+	value.Kernel = strings.TrimSpace(runAgentCommand(3*time.Second, "uname", "-r"))
+	value.Services = collectAgentServices()
+	value.UpdatesAvailable = countAgentUpdates()
+	_, err := os.Stat("/var/run/reboot-required")
+	value.RebootRequired = err == nil
+	guestInventory.value = value
+	return value
+}
+
+func cachedAgentIssues() []AgentIssue {
+	guestIssues.Lock()
+	defer guestIssues.Unlock()
+	if time.Since(guestIssues.collectedAt) >= time.Minute {
+		guestIssues.value = collectAgentIssues()
+		guestIssues.collectedAt = time.Now()
+	}
+	return guestIssues.value
+}
+
+func readOSPrettyName() string {
+	data, err := os.ReadFile("/etc/os-release")
+	if err != nil {
+		return runtime.GOOS
+	}
+	for _, line := range strings.Split(string(data), "\n") {
+		if strings.HasPrefix(line, "PRETTY_NAME=") {
+			return strings.Trim(strings.TrimPrefix(line, "PRETTY_NAME="), "\"")
+		}
+	}
+	return runtime.GOOS
+}
+
+func runAgentCommand(timeout time.Duration, name string, args ...string) string {
+	ctx, cancel := context.WithTimeout(context.Background(), timeout)
+	defer cancel()
+	output, err := exec.CommandContext(ctx, name, args...).Output()
+	if err != nil && len(output) == 0 {
+		return ""
+	}
+	return string(output)
+}
+
+func collectAgentServices() []AgentService {
+	output := runAgentCommand(5*time.Second, "systemctl", "list-units", "--type=service", "--state=running,failed", "--no-legend", "--plain", "--no-pager")
+	result := []AgentService{}
+	for _, line := range strings.Split(output, "\n") {
+		fields := strings.Fields(line)
+		offset := 0
+		if len(fields) > 0 && fields[0] == "●" {
+			offset = 1
+		}
+		if len(fields) < offset+4 {
+			continue
+		}
+		description := ""
+		if len(fields) > offset+4 {
+			description = strings.Join(fields[offset+4:], " ")
+		}
+		result = append(result, AgentService{Name: fields[offset], State: fields[offset+2], SubState: fields[offset+3], Description: description})
+		if len(result) >= 300 {
+			break
+		}
+	}
+	return enrichAgentServices(result)
+}
+
+func enrichAgentServices(services []AgentService) []AgentService {
+	if len(services) == 0 {
+		return services
+	}
+	args := []string{"show", "--no-pager", "--property=Id", "--property=FragmentPath", "--property=NRestarts", "--property=ExecMainStatus", "--property=ActiveEnterTimestampUSec", "--property=MemoryCurrent", "--property=CPUUsageNSec", "--property=MainPID", "--property=ExecStart", "--property=Requires", "--property=After"}
+	for _, service := range services {
+		args = append(args, service.Name)
+	}
+	show := runAgentCommand(8*time.Second, "systemctl", args...)
+	properties := map[string]map[string]string{}
+	paths := map[string]string{}
+	for _, record := range strings.Split(show, "\n\n") {
+		values := map[string]string{}
+		for _, line := range strings.Split(record, "\n") {
+			key, value, ok := strings.Cut(line, "=")
+			if ok {
+				values[key] = value
+			}
+		}
+		if id := values["Id"]; id != "" {
+			properties[id] = values
+			paths[id] = values["FragmentPath"]
+		}
+	}
+	pathArgs := []string{"-S"}
+	for _, service := range services {
+		if path := paths[service.Name]; path != "" {
+			pathArgs = append(pathArgs, path)
+		}
+	}
+	owners := map[string]string{}
+	packages := []string{}
+	if len(pathArgs) > 1 {
+		output := runAgentCommand(8*time.Second, "dpkg-query", pathArgs...)
+		for _, line := range strings.Split(output, "\n") {
+			left, path, ok := strings.Cut(line, ": ")
+			if !ok {
+				continue
+			}
+			pkg := strings.TrimSpace(strings.Split(left, ",")[0])
+			path = strings.TrimSpace(path)
+			if pkg != "" && path != "" {
+				owners[path] = pkg
+				packages = append(packages, pkg)
+			}
+		}
+	}
+	priorities := map[string]string{}
+	if len(packages) > 0 {
+		packageArgs := []string{"-W", "-f=${binary:Package}\t${Priority}\n"}
+		packageArgs = append(packageArgs, packages...)
+		output := runAgentCommand(8*time.Second, "dpkg-query", packageArgs...)
+		for _, line := range strings.Split(output, "\n") {
+			fields := strings.Fields(line)
+			if len(fields) >= 2 {
+				priorities[fields[0]] = fields[1]
+			}
+		}
+	}
+	for i := range services {
+		service := &services[i]
+		service.UnitPath = paths[service.Name]
+		service.Package = owners[service.UnitPath]
+		service.Origin = classifyAgentService(service.UnitPath, service.Package, priorities[service.Package])
+		values := properties[service.Name]
+		service.Restarts, _ = strconv.ParseUint(values["NRestarts"], 10, 64)
+		exit, _ := strconv.ParseInt(values["ExecMainStatus"], 10, 32)
+		service.ExitCode = int(exit)
+		started, _ := strconv.ParseInt(values["ActiveEnterTimestampUSec"], 10, 64)
+		service.StartedAt = started / 1_000_000
+		service.MemoryBytes, _ = strconv.ParseUint(values["MemoryCurrent"], 10, 64)
+		cpu, _ := strconv.ParseUint(values["CPUUsageNSec"], 10, 64)
+		service.CPUSeconds = float64(cpu) / 1_000_000_000
+		pid, _ := strconv.Atoi(values["MainPID"])
+		service.MainPID = pid
+		service.ExecStart = compactSystemdExec(values["ExecStart"])
+		service.Requires = limitedFields(values["Requires"], 12)
+		service.After = limitedFields(values["After"], 12)
+		service.Flapping = service.Restarts >= 5 && service.StartedAt > time.Now().Add(-10*time.Minute).Unix()
+		if service.State == "failed" || service.SubState == "failed" || service.Flapping {
+			service.Logs = collectAgentJournal(service.Name, 8, "-2h")
+		}
+	}
+	ports := collectAgentListenPorts()
+	for i := range services {
+		for _, port := range ports {
+			if services[i].MainPID > 0 && port.PID == services[i].MainPID {
+				services[i].ListenPorts = appendUniqueInt(services[i].ListenPorts, port.Port)
+			}
+		}
+	}
+	return services
+}
+
+func classifyAgentService(path, pkg, priority string) string {
+	switch {
+	case strings.HasPrefix(path, "/etc/systemd/") || strings.HasPrefix(path, "/usr/local/"):
+		return "local"
+	case priority == "required" || priority == "important" || priority == "standard":
+		return "system"
+	case pkg != "":
+		return "added"
+	case strings.HasPrefix(path, "/run/systemd/"):
+		return "runtime"
+	default:
+		return "system"
+	}
+}
+
+func collectAgentIssues() []AgentIssue {
+	output := runAgentCommand(8*time.Second, "journalctl", "-p", "err..alert", "--since", "24 hours ago", "--reverse", "-n", "50", "--no-pager", "-o", "json")
+	return parseAgentIssues(output)
+}
+
+func parseAgentIssues(output string) []AgentIssue {
+	result := []AgentIssue{}
+	for _, line := range strings.Split(output, "\n") {
+		var entry struct {
+			Message   any    `json:"MESSAGE"`
+			Unit      string `json:"_SYSTEMD_UNIT"`
+			Timestamp string `json:"__REALTIME_TIMESTAMP"`
+		}
+		if json.Unmarshal([]byte(line), &entry) != nil {
+			continue
+		}
+		message, ok := entry.Message.(string)
+		if !ok || strings.TrimSpace(message) == "" {
+			continue
+		}
+		microseconds, _ := strconv.ParseInt(entry.Timestamp, 10, 64)
+		unit := entry.Unit
+		if unit == "" {
+			unit = "system"
+		}
+		result = append(result, AgentIssue{Time: microseconds / 1_000_000, Unit: unit, Message: message})
+	}
+	return result
+}
+
+func countAgentUpdates() int {
+	output := runAgentCommand(45*time.Second, "apt-get", "-s", "-o", "Debug::NoLocking=1", "upgrade")
+	count := 0
+	for _, line := range strings.Split(output, "\n") {
+		if strings.HasPrefix(line, "Inst ") {
+			count++
+		}
+	}
+	return count
+}
+
+func sampleAgentCPU() float64 {
+	total1, idle1 := readAgentCPU()
+	time.Sleep(100 * time.Millisecond)
+	total2, idle2 := readAgentCPU()
+	if total2 <= total1 {
+		return 0
+	}
+	totalDelta, idleDelta := total2-total1, idle2-idle1
+	if idleDelta > totalDelta {
+		return 0
+	}
+	return float64(totalDelta-idleDelta) / float64(totalDelta) * 100
+}
+
+func readAgentCPU() (uint64, uint64) {
+	data, err := os.ReadFile("/proc/stat")
+	if err != nil {
+		return 0, 0
+	}
+	line := strings.SplitN(string(data), "\n", 2)[0]
+	fields := strings.Fields(line)
+	if len(fields) < 5 {
+		return 0, 0
+	}
+	var total uint64
+	values := make([]uint64, 0, len(fields)-1)
+	for _, field := range fields[1:] {
+		v, _ := strconv.ParseUint(field, 10, 64)
+		values = append(values, v)
+		total += v
+	}
+	idle := values[3]
+	if len(values) > 4 {
+		idle += values[4]
+	}
+	return total, idle
+}
+
+func collectDockerContainers() ([]AgentContainer, string) {
+	if _, err := os.Stat("/var/run/docker.sock"); err != nil {
+		return []AgentContainer{}, "Docker socket недоступен"
+	}
+	transport := &http.Transport{DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
+		return (&net.Dialer{}).DialContext(ctx, "unix", "/var/run/docker.sock")
+	}}
+	defer transport.CloseIdleConnections()
+	client := &http.Client{Transport: transport, Timeout: 5 * time.Second}
+	response, err := client.Get("http://docker/containers/json?all=1")
+	if err != nil {
+		return nil, err.Error()
+	}
+	defer response.Body.Close()
+	if response.StatusCode != 200 {
+		return nil, response.Status
+	}
+	var raw []struct {
+		ID, Image, State, Status string
+		Names                    []string
+		Ports                    []struct {
+			IP                      string
+			PrivatePort, PublicPort int
+			Type                    string
+		}
+	}
+	if err = json.NewDecoder(response.Body).Decode(&raw); err != nil {
+		return nil, err.Error()
+	}
+	result := make([]AgentContainer, 0, len(raw))
+	for _, v := range raw {
+		name := strings.TrimPrefix(firstString(v.Names), "/")
+		item := AgentContainer{ID: shortID(v.ID), Name: name, Image: v.Image, State: v.State, Status: v.Status}
+		for _, p := range v.Ports {
+			if p.PublicPort > 0 {
+				item.Ports = append(item.Ports, fmt.Sprintf("%d:%d/%s", p.PublicPort, p.PrivatePort, p.Type))
+			}
+		}
+		inspect, inspectErr := client.Get("http://docker/containers/" + v.ID + "/json")
+		if inspectErr == nil {
+			var detail struct {
+				RestartCount int64
+				State        struct{ Health *struct{ Status string } }
+			}
+			if json.NewDecoder(inspect.Body).Decode(&detail) == nil {
+				item.Restarts = detail.RestartCount
+				if detail.State.Health != nil {
+					item.Health = detail.State.Health.Status
+				}
+			}
+			inspect.Body.Close()
+		}
+		statsResponse, statsErr := client.Get("http://docker/containers/" + v.ID + "/stats?stream=false")
+		if statsErr == nil {
+			var stats struct {
+				CPUStats struct {
+					CPUUsage struct {
+						TotalUsage uint64 `json:"total_usage"`
+					} `json:"cpu_usage"`
+					SystemCPUUsage uint64 `json:"system_cpu_usage"`
+					OnlineCPUs     uint64 `json:"online_cpus"`
+				} `json:"cpu_stats"`
+				PreCPUStats struct {
+					CPUUsage struct {
+						TotalUsage uint64 `json:"total_usage"`
+					} `json:"cpu_usage"`
+					SystemCPUUsage uint64 `json:"system_cpu_usage"`
+				} `json:"precpu_stats"`
+				MemoryStats struct {
+					Usage uint64            `json:"usage"`
+					Stats map[string]uint64 `json:"stats"`
+				} `json:"memory_stats"`
+			}
+			if json.NewDecoder(statsResponse.Body).Decode(&stats) == nil {
+				cpuDelta, systemDelta := uint64(0), uint64(0)
+				if stats.CPUStats.CPUUsage.TotalUsage >= stats.PreCPUStats.CPUUsage.TotalUsage {
+					cpuDelta = stats.CPUStats.CPUUsage.TotalUsage - stats.PreCPUStats.CPUUsage.TotalUsage
+				}
+				if stats.CPUStats.SystemCPUUsage >= stats.PreCPUStats.SystemCPUUsage {
+					systemDelta = stats.CPUStats.SystemCPUUsage - stats.PreCPUStats.SystemCPUUsage
+				}
+				cpus := stats.CPUStats.OnlineCPUs
+				if cpus == 0 {
+					cpus = 1
+				}
+				if systemDelta > 0 {
+					item.CPUPercent = float64(cpuDelta) / float64(systemDelta) * float64(cpus) * 100
+				}
+				item.MemoryUsage = stats.MemoryStats.Usage
+				if cache := stats.MemoryStats.Stats["inactive_file"]; item.MemoryUsage > cache {
+					item.MemoryUsage -= cache
+				}
+			}
+			statsResponse.Body.Close()
+		}
+		result = append(result, item)
+	}
+	return result, ""
+}
+func firstString(v []string) string {
+	if len(v) > 0 {
+		return v[0]
+	}
+	return ""
+}
+func shortID(v string) string {
+	if len(v) > 12 {
+		return v[:12]
+	}
+	return v
+}

agent_diagnostics.go

@@ -0,0 +1,329 @@
+package main
+
+import (
+	"bufio"
+	"encoding/json"
+	"net"
+	"os"
+	"path/filepath"
+	"regexp"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"syscall"
+	"time"
+)
+
+type AgentLog struct {
+	Time    int64  `json:"time"`
+	Message string `json:"message"`
+}
+
+type AgentProcess struct {
+	PID         int     `json:"pid"`
+	Name        string  `json:"name"`
+	State       string  `json:"state"`
+	CPUPercent  float64 `json:"cpuPercent"`
+	MemoryBytes uint64  `json:"memoryBytes"`
+	ReadBytes   uint64  `json:"readBytes"`
+	WriteBytes  uint64  `json:"writeBytes"`
+	Elapsed     int64   `json:"elapsedSeconds"`
+	Command     string  `json:"command"`
+}
+
+type AgentFilesystem struct {
+	Device        string  `json:"device"`
+	Mountpoint    string  `json:"mountpoint"`
+	Type          string  `json:"type"`
+	TotalBytes    uint64  `json:"totalBytes"`
+	UsedBytes     uint64  `json:"usedBytes"`
+	UsedPercent   float64 `json:"usedPercent"`
+	InodesTotal   uint64  `json:"inodesTotal"`
+	InodesUsed    uint64  `json:"inodesUsed"`
+	InodesPercent float64 `json:"inodesPercent"`
+}
+
+type AgentListenPort struct {
+	Protocol string `json:"protocol"`
+	Address  string `json:"address"`
+	Port     int    `json:"port"`
+	PID      int    `json:"pid,omitempty"`
+	Process  string `json:"process,omitempty"`
+}
+
+type AgentPressureValue struct {
+	Avg10  float64 `json:"avg10"`
+	Avg60  float64 `json:"avg60"`
+	Avg300 float64 `json:"avg300"`
+}
+
+type AgentPressure struct {
+	CPU    AgentPressureValue `json:"cpu"`
+	Memory AgentPressureValue `json:"memory"`
+	IO     AgentPressureValue `json:"io"`
+}
+
+var agentOOMCache struct {
+	sync.Mutex
+	value []AgentLog
+	at    time.Time
+}
+
+func compactSystemdExec(value string) string {
+	if pathAt := strings.Index(value, "path="); pathAt >= 0 {
+		value = value[pathAt+5:]
+	}
+	if end := strings.IndexAny(value, " ;}"); end > 0 {
+		value = value[:end]
+	}
+	if len(value) > 240 {
+		value = value[:240]
+	}
+	return strings.TrimSpace(value)
+}
+
+func limitedFields(value string, limit int) []string {
+	fields := strings.Fields(value)
+	if len(fields) > limit {
+		fields = fields[:limit]
+	}
+	return fields
+}
+
+func appendUniqueInt(items []int, value int) []int {
+	for _, item := range items {
+		if item == value {
+			return items
+		}
+	}
+	return append(items, value)
+}
+
+func collectAgentJournal(unit string, limit int, since string) []AgentLog {
+	args := []string{"--no-pager", "-o", "json", "--reverse", "-n", strconv.Itoa(limit), "--since", since}
+	if unit != "" {
+		args = append(args, "-u", unit)
+	}
+	output := runAgentCommand(6*time.Second, "journalctl", args...)
+	result := []AgentLog{}
+	for _, line := range strings.Split(output, "\n") {
+		var entry struct {
+			Message   any    `json:"MESSAGE"`
+			Timestamp string `json:"__REALTIME_TIMESTAMP"`
+		}
+		if json.Unmarshal([]byte(line), &entry) != nil {
+			continue
+		}
+		message, ok := entry.Message.(string)
+		if !ok || strings.TrimSpace(message) == "" {
+			continue
+		}
+		microseconds, _ := strconv.ParseInt(entry.Timestamp, 10, 64)
+		result = append(result, AgentLog{Time: microseconds / 1_000_000, Message: message})
+	}
+	return result
+}
+
+func cachedAgentOOMKills() []AgentLog {
+	agentOOMCache.Lock()
+	defer agentOOMCache.Unlock()
+	if time.Since(agentOOMCache.at) < time.Minute {
+		return agentOOMCache.value
+	}
+	output := runAgentCommand(6*time.Second, "journalctl", "-k", "--since", "24 hours ago", "--reverse", "--no-pager", "-o", "json", "-g", "Out of memory|Killed process")
+	result := []AgentLog{}
+	for _, line := range strings.Split(output, "\n") {
+		var entry struct {
+			Message   any    `json:"MESSAGE"`
+			Timestamp string `json:"__REALTIME_TIMESTAMP"`
+		}
+		if json.Unmarshal([]byte(line), &entry) != nil {
+			continue
+		}
+		message, ok := entry.Message.(string)
+		if !ok {
+			continue
+		}
+		microseconds, _ := strconv.ParseInt(entry.Timestamp, 10, 64)
+		result = append(result, AgentLog{Time: microseconds / 1_000_000, Message: message})
+		if len(result) >= 20 {
+			break
+		}
+	}
+	agentOOMCache.value, agentOOMCache.at = result, time.Now()
+	return result
+}
+
+func collectAgentProcesses() []AgentProcess {
+	result := []AgentProcess{}
+	seen := map[int]bool{}
+	for _, sortBy := range []string{"-pcpu", "-rss"} {
+		output := runAgentCommand(5*time.Second, "ps", "-eo", "pid=,comm=,state=,pcpu=,rss=,etimes=,args=", "--sort="+sortBy)
+		added := 0
+		for _, line := range strings.Split(output, "\n") {
+			fields := strings.Fields(line)
+			if len(fields) < 7 {
+				continue
+			}
+			pid, err := strconv.Atoi(fields[0])
+			if err != nil || seen[pid] {
+				continue
+			}
+			seen[pid] = true
+			cpu, _ := strconv.ParseFloat(fields[3], 64)
+			rss, _ := strconv.ParseUint(fields[4], 10, 64)
+			elapsed, _ := strconv.ParseInt(fields[5], 10, 64)
+			item := AgentProcess{PID: pid, Name: fields[1], State: fields[2], CPUPercent: cpu, MemoryBytes: rss * 1024, Elapsed: elapsed, Command: strings.Join(fields[6:], " ")}
+			item.ReadBytes, item.WriteBytes = readAgentProcessIO(pid)
+			result = append(result, item)
+			added++
+			if added >= 12 || len(result) >= 24 {
+				break
+			}
+		}
+	}
+	return result
+}
+
+func readAgentProcessIO(pid int) (uint64, uint64) {
+	data, err := os.ReadFile(filepath.Join("/proc", strconv.Itoa(pid), "io"))
+	if err != nil {
+		return 0, 0
+	}
+	values := map[string]uint64{}
+	for _, line := range strings.Split(string(data), "\n") {
+		fields := strings.Fields(line)
+		if len(fields) != 2 {
+			continue
+		}
+		values[strings.TrimSuffix(fields[0], ":")], _ = strconv.ParseUint(fields[1], 10, 64)
+	}
+	return values["read_bytes"], values["write_bytes"]
+}
+
+func collectAgentFilesystems() []AgentFilesystem {
+	file, err := os.Open("/proc/self/mounts")
+	if err != nil {
+		return nil
+	}
+	defer file.Close()
+	excluded := map[string]bool{"proc": true, "sysfs": true, "devtmpfs": true, "devpts": true, "tmpfs": true, "cgroup": true, "cgroup2": true, "overlay": true, "squashfs": true, "nsfs": true, "tracefs": true, "debugfs": true, "securityfs": true, "pstore": true, "mqueue": true, "hugetlbfs": true, "fusectl": true, "configfs": true, "autofs": true, "rpc_pipefs": true}
+	seen := map[string]bool{}
+	result := []AgentFilesystem{}
+	scanner := bufio.NewScanner(file)
+	for scanner.Scan() {
+		fields := strings.Fields(scanner.Text())
+		if len(fields) < 3 || excluded[fields[2]] || seen[fields[1]] {
+			continue
+		}
+		mountpoint := strings.ReplaceAll(fields[1], `\040`, " ")
+		var stat syscall.Statfs_t
+		if syscall.Statfs(mountpoint, &stat) != nil || stat.Blocks == 0 {
+			continue
+		}
+		seen[mountpoint] = true
+		total := stat.Blocks * uint64(stat.Bsize)
+		free := stat.Bavail * uint64(stat.Bsize)
+		used := total - free
+		item := AgentFilesystem{Device: fields[0], Mountpoint: mountpoint, Type: fields[2], TotalBytes: total, UsedBytes: used, UsedPercent: float64(used) / float64(total) * 100, InodesTotal: stat.Files}
+		if stat.Files > stat.Ffree {
+			item.InodesUsed = stat.Files - stat.Ffree
+			item.InodesPercent = float64(item.InodesUsed) / float64(stat.Files) * 100
+		}
+		result = append(result, item)
+		if len(result) >= 30 {
+			break
+		}
+	}
+	sort.Slice(result, func(i, j int) bool { return result[i].Mountpoint < result[j].Mountpoint })
+	return result
+}
+
+var ssPIDPattern = regexp.MustCompile(`pid=([0-9]+)`)
+var ssNamePattern = regexp.MustCompile(`users:\(\(\"([^\"]+)\"`)
+
+func collectAgentListenPorts() []AgentListenPort {
+	output := runAgentCommand(5*time.Second, "ss", "-H", "-lntup")
+	result := []AgentListenPort{}
+	seen := map[string]bool{}
+	for _, line := range strings.Split(output, "\n") {
+		fields := strings.Fields(line)
+		if len(fields) < 5 {
+			continue
+		}
+		protocol := strings.ToLower(fields[0])
+		local := ""
+		for _, field := range fields[1:] {
+			if strings.Contains(field, ":") && !strings.Contains(field, "users:") {
+				local = field
+				break
+			}
+		}
+		if local == "" {
+			continue
+		}
+		address, portText, err := net.SplitHostPort(local)
+		if err != nil {
+			idx := strings.LastIndex(local, ":")
+			if idx < 0 {
+				continue
+			}
+			address, portText = local[:idx], local[idx+1:]
+		}
+		port, _ := strconv.Atoi(portText)
+		if port <= 0 {
+			continue
+		}
+		pid := 0
+		if match := ssPIDPattern.FindStringSubmatch(line); len(match) > 1 {
+			pid, _ = strconv.Atoi(match[1])
+		}
+		process := ""
+		if match := ssNamePattern.FindStringSubmatch(line); len(match) > 1 {
+			process = match[1]
+		}
+		key := protocol + "|" + address + "|" + portText + "|" + strconv.Itoa(pid)
+		if seen[key] {
+			continue
+		}
+		seen[key] = true
+		result = append(result, AgentListenPort{Protocol: protocol, Address: address, Port: port, PID: pid, Process: process})
+	}
+	sort.Slice(result, func(i, j int) bool {
+		if result[i].Port == result[j].Port {
+			return result[i].Protocol < result[j].Protocol
+		}
+		return result[i].Port < result[j].Port
+	})
+	return result
+}
+
+func collectAgentPressure() AgentPressure {
+	return AgentPressure{CPU: readPressure("/proc/pressure/cpu"), Memory: readPressure("/proc/pressure/memory"), IO: readPressure("/proc/pressure/io")}
+}
+
+func readPressure(path string) AgentPressureValue {
+	data, err := os.ReadFile(path)
+	if err != nil {
+		return AgentPressureValue{}
+	}
+	line := strings.SplitN(string(data), "\n", 2)[0]
+	value := AgentPressureValue{}
+	for _, field := range strings.Fields(line) {
+		key, raw, ok := strings.Cut(field, "=")
+		if !ok {
+			continue
+		}
+		number, _ := strconv.ParseFloat(raw, 64)
+		switch key {
+		case "avg10":
+			value.Avg10 = number
+		case "avg60":
+			value.Avg60 = number
+		case "avg300":
+			value.Avg300 = number
+		}
+	}
+	return value
+}

agent_test.go

@@ -0,0 +1,146 @@
+package main
+
+import (
+	"os"
+	"testing"
+	"time"
+)
+
+func TestAgentEnrollmentTokenIsSingleUse(t *testing.T) {
+	store, err := openStore(":memory:")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer store.Close()
+	token, err := store.CreateAgentEnrollmentToken()
+	if err != nil {
+		t.Fatal(err)
+	}
+	credentials, err := store.EnrollAgent(agentEnrollmentRequest{Token: token, Name: "docker-01", Hostname: "vm105", VMID: 105})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if credentials.ID == "" || credentials.Secret == "" {
+		t.Fatal("credentials were not generated")
+	}
+	if _, err = store.EnrollAgent(agentEnrollmentRequest{Token: token, Hostname: "duplicate"}); err == nil {
+		t.Fatal("token was accepted twice")
+	}
+}
+
+func TestCurrentAgentUpdateManifestMatchesExecutable(t *testing.T) {
+	manifest, path, err := currentAgentUpdate()
+	if err != nil {
+		t.Fatal(err)
+	}
+	if manifest.URL != "/api/agent/binary" || len(manifest.SHA256) != 64 {
+		t.Fatalf("unexpected manifest: %#v", manifest)
+	}
+	if _, err = os.Stat(path); err != nil {
+		t.Fatal(err)
+	}
+}
+
+func TestAgentOnlyInstallsNewerVersion(t *testing.T) {
+	if !newerAgentVersion("v0.14.0", "v0.13.0") {
+		t.Fatal("new version was not detected")
+	}
+	if newerAgentVersion("v0.12.0", "v0.13.0") {
+		t.Fatal("downgrade was allowed")
+	}
+	if newerAgentVersion("v0.13.0", "v0.13.0") {
+		t.Fatal("same version was offered")
+	}
+}
+
+func TestParseAgentIssues(t *testing.T) {
+	issues := parseAgentIssues(`{"MESSAGE":"disk error","_SYSTEMD_UNIT":"worker.service","__REALTIME_TIMESTAMP":"1722960000000000"}` + "\n" + `{"MESSAGE":"network error","__REALTIME_TIMESTAMP":"1722960001000000"}`)
+	if len(issues) != 2 || issues[0].Unit != "worker.service" || issues[1].Unit != "system" || issues[0].Time != 1722960000 {
+		t.Fatalf("unexpected issues: %#v", issues)
+	}
+}
+
+func TestClassifyAgentService(t *testing.T) {
+	cases := []struct{ path, pkg, priority, want string }{{"/lib/systemd/system/cron.service", "cron", "important", "system"}, {"/lib/systemd/system/jellyfin.service", "jellyfin", "optional", "added"}, {"/etc/systemd/system/my.service", "", "", "local"}, {"/run/systemd/system/transient.service", "", "", "runtime"}}
+	for _, tc := range cases {
+		if got := classifyAgentService(tc.path, tc.pkg, tc.priority); got != tc.want {
+			t.Errorf("%s: got %s want %s", tc.path, got, tc.want)
+		}
+	}
+}
+
+func TestAgentServiceDiagnosticsHelpers(t *testing.T) {
+	if got := compactSystemdExec(`{ path=/usr/bin/example ; argv[]=/usr/bin/example --serve ; }`); got != "/usr/bin/example" {
+		t.Fatalf("unexpected executable: %q", got)
+	}
+	fields := limitedFields("a b c d", 2)
+	if len(fields) != 2 || fields[0] != "a" || fields[1] != "b" {
+		t.Fatalf("unexpected limited fields: %#v", fields)
+	}
+	ports := appendUniqueInt([]int{80}, 80)
+	ports = appendUniqueInt(ports, 443)
+	if len(ports) != 2 || ports[1] != 443 {
+		t.Fatalf("unexpected ports: %#v", ports)
+	}
+}
+
+func TestAgentReportAuthenticationAndOfflineState(t *testing.T) {
+	store, err := openStore(":memory:")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer store.Close()
+	token, _ := store.CreateAgentEnrollmentToken()
+	credentials, _ := store.EnrollAgent(agentEnrollmentRequest{Token: token, Name: "docker", Hostname: "vm", VMID: 101})
+	report := AgentReport{Hostname: "vm", Version: "test", DockerAvailable: true, Containers: []AgentContainer{{ID: "abc", Name: "web", State: "running"}}}
+	if err = store.SaveAgentReport(credentials.ID, "wrong", "127.0.0.1", report); err == nil {
+		t.Fatal("invalid secret accepted")
+	}
+	if err = store.SaveAgentReport(credentials.ID, credentials.Secret, "127.0.0.1", report); err != nil {
+		t.Fatal(err)
+	}
+	agents, err := store.Agents()
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(agents) != 1 || !agents[0].Online || len(agents[0].Report.Containers) != 1 {
+		t.Fatalf("unexpected agents: %#v", agents)
+	}
+	_, _ = store.db.Exec(`UPDATE agents SET last_seen=? WHERE id=?`, time.Now().Add(-time.Minute).Unix(), credentials.ID)
+	agents, _ = store.Agents()
+	if agents[0].Online {
+		t.Fatal("stale agent is online")
+	}
+}
+
+func TestAgentProblemsBecomeMaintenanceAwareAlerts(t *testing.T) {
+	metrics := dashboardMetrics{Agents: []ManagedAgent{{ID: "node", Name: "app", VMID: 105, Online: true, Report: AgentReport{
+		Services:   []AgentService{{Name: "postgres.service", State: "failed", SubState: "failed"}},
+		Containers: []AgentContainer{{ID: "dead", Name: "web", State: "exited", Status: "Exited (1)"}}, DockerAvailable: true,
+	}}}}
+	alerts := evaluateAlerts(metrics, defaultThresholds())
+	window := MaintenanceWindow{Active: true, VMIDs: []int{105}}
+	active, planned := splitMaintenanceAlerts(alerts, window)
+	if len(planned) != 2 || len(active) != 0 {
+		t.Fatalf("active=%d planned=%d alerts=%#v", len(active), len(planned), alerts)
+	}
+}
+
+func TestAgentDiagnosticsBecomeAlerts(t *testing.T) {
+	metrics := dashboardMetrics{Agents: []ManagedAgent{{ID: "node", Name: "app", VMID: 105, Online: true, Report: AgentReport{
+		Services:    []AgentService{{Name: "web.service", State: "active", Flapping: true, Restarts: 8, StartedAt: time.Now().Unix()}},
+		Filesystems: []AgentFilesystem{{Mountpoint: "/data", TotalBytes: 1000, UsedBytes: 950, UsedPercent: 95}},
+	}}}}
+	alerts := evaluateAlerts(metrics, defaultThresholds())
+	wanted := map[string]bool{"agent-flapping-node-web.service-105": false, "agent-filesystem-node-/data-105": false}
+	for _, alert := range alerts {
+		if _, ok := wanted[alert.ID]; ok {
+			wanted[alert.ID] = true
+		}
+	}
+	for id, found := range wanted {
+		if !found {
+			t.Fatalf("missing alert %s in %#v", id, alerts)
+		}
+	}
+}

alert_events.go

@@ -16,6 +16,8 @@ type AlertEvent struct {
 	LastSeen          int64  `json:"lastSeen"`
 	ResolvedAt        *int64 `json:"resolvedAt,omitempty"`
 	AcknowledgedAt    *int64 `json:"acknowledgedAt,omitempty"`
+	Planned           bool   `json:"planned"`
+	MaintenanceReason string `json:"maintenanceReason,omitempty"`
 }
 
 type RecoveryEvent struct {
@@ -24,13 +26,17 @@ type RecoveryEvent struct {
 }
 
 func (s *Store) SyncAlerts(alerts []Alert) ([]Alert, error) {
+	return s.SyncAlertsWithMaintenance(alerts, nil, "")
+}
+
+func (s *Store) SyncAlertsWithMaintenance(alerts, plannedAlerts []Alert, maintenanceReason string) ([]Alert, error) {
 	tx, err := s.db.Begin()
 	if err != nil {
 		return alerts, err
 	}
 	defer tx.Rollback()
 	now := time.Now().Unix()
-	rows, err := tx.Query(`SELECT id,alert_key FROM alert_events WHERE resolved_at IS NULL`)
+	rows, err := tx.Query(`SELECT id,alert_key,planned FROM alert_events WHERE resolved_at IS NULL`)
 	if err != nil {
 		return alerts, err
 	}
@@ -38,28 +44,43 @@ func (s *Store) SyncAlerts(alerts []Alert) ([]Alert, error) {
 	for rows.Next() {
 		var id int64
 		var key string
-		if err := rows.Scan(&id, &key); err != nil {
+		var planned bool
+		if err := rows.Scan(&id, &key, &planned); err != nil {
 			rows.Close()
 			return alerts, err
 		}
-		active[key] = id
+		active[alertEventKey(key, planned)] = id
 	}
 	rows.Close()
 	seen := map[string]bool{}
-	for _, alert := range alerts {
-		seen[alert.ID] = true
-		if id, ok := active[alert.ID]; ok {
-			_, err = tx.Exec(`UPDATE alert_events SET severity=?,source=?,title=?,message=?,last_seen=? WHERE id=?`, alert.Severity, alert.Source, alert.Title, alert.Message, now, id)
-		} else {
-			result, insertErr := tx.Exec(`INSERT INTO alert_events(alert_key,severity,source,title,message,first_seen,last_seen) VALUES(?,?,?,?,?,?,?)`, alert.ID, alert.Severity, alert.Source, alert.Title, alert.Message, now, now)
-			if insertErr != nil {
-				return alerts, insertErr
+	syncGroup := func(group []Alert, planned bool) error {
+		for _, alert := range group {
+			key := alertEventKey(alert.ID, planned)
+			seen[key] = true
+			reason := ""
+			if planned {
+				reason = maintenanceReason
 			}
-			active[alert.ID], _ = result.LastInsertId()
+			if id, ok := active[key]; ok {
+				_, err = tx.Exec(`UPDATE alert_events SET severity=?,source=?,title=?,message=?,last_seen=?,maintenance_reason=? WHERE id=?`, alert.Severity, alert.Source, alert.Title, alert.Message, now, reason, id)
+			} else {
+				result, insertErr := tx.Exec(`INSERT INTO alert_events(alert_key,severity,source,title,message,first_seen,last_seen,planned,maintenance_reason) VALUES(?,?,?,?,?,?,?,?,?)`, alert.ID, alert.Severity, alert.Source, alert.Title, alert.Message, now, now, planned, reason)
+				if insertErr != nil {
+					return insertErr
+				}
+				active[key], _ = result.LastInsertId()
 			}
 			if err != nil {
+				return err
+			}
+		}
+		return nil
+	}
+	if err = syncGroup(alerts, false); err != nil {
 		return alerts, err
 	}
+	if err = syncGroup(plannedAlerts, true); err != nil {
+		return alerts, err
 	}
 	for key, id := range active {
 		if !seen[key] {
@@ -74,7 +95,7 @@ func (s *Store) SyncAlerts(alerts []Alert) ([]Alert, error) {
 	if err = tx.Commit(); err != nil {
 		return alerts, err
 	}
-	rows, err = s.db.Query(`SELECT id,alert_key,severity,source,title,message FROM alert_events WHERE resolved_at IS NULL AND acknowledged_at IS NULL ORDER BY CASE severity WHEN 'critical' THEN 0 WHEN 'warning' THEN 1 ELSE 2 END,first_seen DESC`)
+	rows, err = s.db.Query(`SELECT id,alert_key,severity,source,title,message,first_seen,last_seen FROM alert_events WHERE resolved_at IS NULL AND acknowledged_at IS NULL AND planned=0 ORDER BY CASE severity WHEN 'critical' THEN 0 WHEN 'warning' THEN 1 ELSE 2 END,first_seen DESC`)
 	if err != nil {
 		return alerts, err
 	}
@@ -82,7 +103,7 @@ func (s *Store) SyncAlerts(alerts []Alert) ([]Alert, error) {
 	var visible []Alert
 	for rows.Next() {
 		var a Alert
-		if err = rows.Scan(&a.EventID, &a.ID, &a.Severity, &a.Source, &a.Title, &a.Message); err != nil {
+		if err = rows.Scan(&a.EventID, &a.ID, &a.Severity, &a.Source, &a.Title, &a.Message, &a.FirstSeen, &a.LastSeen); err != nil {
 			return alerts, err
 		}
 		visible = append(visible, a)
@@ -90,6 +111,13 @@ func (s *Store) SyncAlerts(alerts []Alert) ([]Alert, error) {
 	return visible, rows.Err()
 }
 
+func alertEventKey(id string, planned bool) string {
+	if planned {
+		return id + "\x00planned"
+	}
+	return id + "\x00active"
+}
+
 func (s *Store) AcknowledgeAlert(eventID int64) error {
 	_, err := s.db.Exec(`UPDATE alert_events SET acknowledged_at=? WHERE id=? AND acknowledged_at IS NULL`, time.Now().Unix(), eventID)
 	return err
@@ -99,7 +127,7 @@ func (s *Store) AlertHistory(limit int) ([]AlertEvent, error) {
 	if limit < 1 || limit > 500 {
 		limit = 200
 	}
-	rows, err := s.db.Query(`SELECT id,alert_key,severity,source,title,message,first_seen,last_seen,resolved_at,acknowledged_at FROM alert_events ORDER BY first_seen DESC LIMIT ?`, limit)
+	rows, err := s.db.Query(`SELECT id,alert_key,severity,source,title,message,first_seen,last_seen,resolved_at,acknowledged_at,planned,maintenance_reason FROM alert_events ORDER BY first_seen DESC LIMIT ?`, limit)
 	if err != nil {
 		return nil, err
 	}
@@ -108,7 +136,7 @@ func (s *Store) AlertHistory(limit int) ([]AlertEvent, error) {
 	for rows.Next() {
 		var e AlertEvent
 		var resolved, acked sql.NullInt64
-		if err = rows.Scan(&e.EventID, &e.ID, &e.Severity, &e.Source, &e.Title, &e.Message, &e.FirstSeen, &e.LastSeen, &resolved, &acked); err != nil {
+		if err = rows.Scan(&e.EventID, &e.ID, &e.Severity, &e.Source, &e.Title, &e.Message, &e.FirstSeen, &e.LastSeen, &resolved, &acked, &e.Planned, &e.MaintenanceReason); err != nil {
 			return nil, err
 		}
 		if resolved.Valid {
@@ -123,7 +151,7 @@ func (s *Store) AlertHistory(limit int) ([]AlertEvent, error) {
 }
 
 func (s *Store) PendingEmailAlerts() ([]Alert, error) {
-	rows, err := s.db.Query(`SELECT id,alert_key,severity,source,title,message FROM alert_events WHERE resolved_at IS NULL AND acknowledged_at IS NULL AND email_sent_at IS NULL ORDER BY first_seen`)
+	rows, err := s.db.Query(`SELECT id,alert_key,severity,source,title,message FROM alert_events WHERE resolved_at IS NULL AND acknowledged_at IS NULL AND email_sent_at IS NULL AND planned=0 ORDER BY first_seen`)
 	if err != nil {
 		return nil, err
 	}
@@ -155,7 +183,7 @@ func (s *Store) MarkAlertsEmailed(alerts []Alert) error {
 }
 
 func (s *Store) PendingImmediateAlerts() ([]Alert, error) {
-	rows, err := s.db.Query(`SELECT id,alert_key,severity,source,title,message FROM alert_events WHERE resolved_at IS NULL AND acknowledged_at IS NULL AND email_sent_at IS NULL AND (severity='critical' OR alert_key='ups-battery') ORDER BY first_seen`)
+	rows, err := s.db.Query(`SELECT id,alert_key,severity,source,title,message FROM alert_events WHERE resolved_at IS NULL AND acknowledged_at IS NULL AND email_sent_at IS NULL AND planned=0 AND (severity='critical' OR alert_key='ups-battery') ORDER BY first_seen`)
 	if err != nil {
 		return nil, err
 	}
@@ -171,7 +199,7 @@ func (s *Store) PendingImmediateAlerts() ([]Alert, error) {
 	return out, rows.Err()
 }
 func (s *Store) PendingRecoveries() ([]RecoveryEvent, error) {
-	rows, err := s.db.Query(`SELECT id,alert_key,severity,source,title,message FROM alert_events WHERE resolved_at IS NOT NULL AND email_sent_at IS NOT NULL AND recovery_sent_at IS NULL ORDER BY resolved_at`)
+	rows, err := s.db.Query(`SELECT id,alert_key,severity,source,title,message FROM alert_events WHERE resolved_at IS NOT NULL AND email_sent_at IS NOT NULL AND recovery_sent_at IS NULL AND planned=0 ORDER BY resolved_at`)
 	if err != nil {
 		return nil, err
 	}

alert_events_test.go

@@ -10,7 +10,7 @@ func TestAlertLifecycle(t *testing.T) {
 	defer store.Close()
 	alert := Alert{ID: "disk", Severity: "warning", Source: "Диски", Title: "SMART", Message: "Проверьте диск"}
 	visible, err := store.SyncAlerts([]Alert{alert})
-	if err != nil || len(visible) != 1 || visible[0].EventID == 0 {
+	if err != nil || len(visible) != 1 || visible[0].EventID == 0 || visible[0].FirstSeen == 0 || visible[0].LastSeen == 0 {
 		t.Fatalf("unexpected new alert: %+v %v", visible, err)
 	}
 	firstID := visible[0].EventID

alerts.go

@@ -14,6 +14,8 @@ type Alert struct {
 	Source    string `json:"source"`
 	Title     string `json:"title"`
 	Message   string `json:"message"`
+	FirstSeen int64  `json:"firstSeen,omitempty"`
+	LastSeen  int64  `json:"lastSeen,omitempty"`
 }
 
 func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Alert {
@@ -61,6 +63,14 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
 		add("root-storage", "warning", "Storage", "Мало места на системном разделе", fmt.Sprintf("Раздел / заполнен на %.1f%%.", metrics.Storage.UsagePercent))
 	}
 	for _, disk := range metrics.Disks {
+		diskExcluded := false
+		for _, value := range strings.Split(thresholds.DiskIOExcluded, ",") {
+			value = strings.TrimSpace(value)
+			if value != "" && (strings.EqualFold(value, disk.Name) || strings.EqualFold(value, disk.Serial)) {
+				diskExcluded = true
+				break
+			}
+		}
 		if disk.SMARTStatus == "Ошибка" {
 			add("disk-smart-"+disk.Name, "critical", "Диски", disk.Model+": ошибка SMART", joinReasons(disk.AttentionReasons, "SMART сообщает о неисправности диска."))
 		} else if disk.SMARTStatus == "Требует внимания" {
@@ -75,6 +85,26 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
 				add("disk-temp-"+disk.Name, "warning", "Диски", disk.Model+": высокая температура", fmt.Sprintf("Температура %.1f °C.", *disk.Temperature))
 			}
 		}
+		if len(disk.CounterChanges) > 0 {
+			add("disk-growth-"+disk.Name, "critical", "Диски", disk.Model+": ухудшились SMART-счётчики", strings.Join(disk.CounterChanges, " · "))
+		}
+		if !diskExcluded && disk.IOPressureSeconds >= thresholds.DiskIODuration {
+			severity := "warning"
+			if disk.Utilization >= 98 || disk.LatencyMs >= 100 {
+				severity = "critical"
+			}
+			cause := disk.IOCause
+			if cause == "" {
+				cause = "источник пока не определён"
+			}
+			add("disk-io-pressure-"+disk.Name, severity, "Диски", disk.Model+": длительная I/O-нагрузка", fmt.Sprintf("%s занят на %.1f%%; вероятная причина: %s (уверенность %d%%). %.0f IOPS, %.1f MB/s, latency %.1f ms, очередь %.1f.", disk.Name, disk.Utilization, cause, disk.IOConfidence, disk.ReadIOPS+disk.WriteIOPS, (disk.ReadBytesPerSec+disk.WriteBytesPerSec)/1048576, disk.LatencyMs, disk.AverageQueue))
+		}
+		if disk.SelfTestNeverRun {
+			add("disk-selftest-"+disk.Name, "warning", "Диски", disk.Model+": нет завершённого SMART self-test", "Короткий тест будет автоматически запущен; проверьте его результат после завершения.")
+		}
+	}
+	for _, missing := range metrics.MissingDisks {
+		add("disk-missing-"+missing, "critical", "Диски", "Физический диск исчез", missing+" больше не обнаруживается в /sys/block.")
 	}
 
 	for _, pool := range metrics.ZFS.Pools {
@@ -89,6 +119,25 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
 		if strings.Contains(strings.ToLower(pool.Scan), "in progress") {
 			add("zfs-scan-"+pool.Name, "info", "ZFS", pool.Name+": выполняется scrub/resilver", pool.Scan)
 		}
+		if pool.ReadErrors+pool.WriteErrors+pool.ChecksumErrors > 0 {
+			add("zfs-vdev-errors-"+pool.Name, "critical", "ZFS", pool.Name+": ошибки чтения/записи/checksum", fmt.Sprintf("read=%d, write=%d, checksum=%d.", pool.ReadErrors, pool.WriteErrors, pool.ChecksumErrors))
+		}
+		if len(pool.ErrorChanges) > 0 {
+			add("zfs-new-errors-"+pool.Name, "critical", "ZFS", pool.Name+": появились новые ошибки", strings.Join(pool.ErrorChanges, " · "))
+		}
+		if pool.ScrubTooOld {
+			add("zfs-scrub-old-"+pool.Name, "warning", "ZFS", pool.Name+": scrub просрочен", "Scrub не выполнялся более 35 дней либо ни разу не запускался.")
+		}
+	}
+	for _, dataset := range metrics.ZFS.Datasets {
+		if dataset.CapacityPercent >= 90 {
+			add("zfs-dataset-"+dataset.Name, "critical", "ZFS", dataset.Name+": dataset почти заполнен", fmt.Sprintf("Заполнение %.1f%%, доступно %.1f GB.", dataset.CapacityPercent, float64(dataset.AvailableBytes)/1073741824))
+		} else if dataset.CapacityPercent >= 80 {
+			add("zfs-dataset-"+dataset.Name, "warning", "ZFS", dataset.Name+": мало свободного места", fmt.Sprintf("Заполнение %.1f%%.", dataset.CapacityPercent))
+		}
+	}
+	if metrics.ZFS.ARC.Available && metrics.ZFS.ARC.Hits+metrics.ZFS.ARC.Misses > 10000 && metrics.ZFS.ARC.HitRatio < 70 {
+		add("zfs-arc-hit", "warning", "ZFS", "Низкий ARC hit ratio", fmt.Sprintf("Попадания ARC: %.1f%%.", metrics.ZFS.ARC.HitRatio))
 	}
 
 	if metrics.UPS.Available {
@@ -101,6 +150,14 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
 		if metrics.UPS.ChargePercent != nil && *metrics.UPS.ChargePercent < thresholds.UPSChargeCritical {
 			add("ups-charge", "critical", "UPS", "Низкий заряд батареи", fmt.Sprintf("Осталось %.1f%%.", *metrics.UPS.ChargePercent))
 		}
+		if metrics.UPS.ReplaceBattery {
+			add("ups-replace-battery", "critical", "UPS", "UPS требует заменить батарею", "NUT/APC сообщает флаг Replace Battery.")
+		}
+		if metrics.UPS.BatteryAgeDays > 3*365 {
+			add("ups-battery-age", "warning", "UPS", "Батарея UPS старше трёх лет", fmt.Sprintf("Возраст батареи примерно %d дней.", metrics.UPS.BatteryAgeDays))
+		}
+	} else if metrics.UPS.Source != "" {
+		add("ups-communication", "warning", "UPS", "Нет связи с UPS", "NUT и apcupsd не вернули данные. Проверьте USB/сетевое подключение и службу UPS.")
 	}
 
 	if metrics.Backups.Available && metrics.Guests.Available {
@@ -140,6 +197,50 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
 			add("guests-stopped", "info", "VM/LXC", "Есть остановленные гости", fmt.Sprintf("Остановлено VM/LXC: %d. Это уведомление информационное.", stopped))
 		}
 	}
+	for _, agent := range metrics.Agents {
+		vmSuffix := fmt.Sprintf("-%d", agent.VMID)
+		if !agent.Online {
+			add("agent-offline-"+agent.ID+vmSuffix, "critical", "Агенты", agent.Name+": агент не отвечает", fmt.Sprintf("Последний heartbeat: %s назад. VMID %d.", time.Since(time.Unix(agent.LastSeen, 0)).Round(time.Second), agent.VMID))
+			continue
+		}
+		if agent.Report.DockerAvailable {
+			for _, container := range agent.Report.Containers {
+				if container.State != "running" {
+					add("agent-container-stopped-"+agent.ID+"-"+container.ID+vmSuffix, "warning", "Docker", agent.Name+": контейнер "+container.Name+" остановлен", fmt.Sprintf("%s · VMID %d.", container.Status, agent.VMID))
+				} else if container.Health == "unhealthy" {
+					add("agent-container-health-"+agent.ID+"-"+container.ID+vmSuffix, "critical", "Docker", agent.Name+": контейнер "+container.Name+" unhealthy", fmt.Sprintf("Docker health-check не проходит · VMID %d.", agent.VMID))
+				}
+			}
+		}
+		for _, service := range agent.Report.Services {
+			if service.State == "failed" || service.SubState == "failed" {
+				add("agent-systemd-"+agent.ID+"-"+service.Name+vmSuffix, "critical", "Агенты", agent.Name+": упала служба "+service.Name, fmt.Sprintf("%s · VMID %d.", firstNonEmpty(service.Description, service.SubState), agent.VMID))
+			} else if service.Flapping {
+				add("agent-flapping-"+agent.ID+"-"+service.Name+vmSuffix, "warning", "Агенты", agent.Name+": служба постоянно перезапускается", fmt.Sprintf("%s: %d рестартов, последний запуск %s · VMID %d.", service.Name, service.Restarts, time.Unix(service.StartedAt, 0).Format("15:04:05"), agent.VMID))
+			}
+		}
+		for _, filesystem := range agent.Report.Filesystems {
+			if filesystem.UsedPercent >= 90 {
+				add("agent-filesystem-"+agent.ID+"-"+filesystem.Mountpoint+vmSuffix, "critical", "Агенты", agent.Name+": заканчивается место "+filesystem.Mountpoint, fmt.Sprintf("Заполнено %.1f%%, свободно %.1f GB · VMID %d.", filesystem.UsedPercent, float64(filesystem.TotalBytes-filesystem.UsedBytes)/1073741824, agent.VMID))
+			}
+			if filesystem.InodesPercent >= 90 {
+				add("agent-inodes-"+agent.ID+"-"+filesystem.Mountpoint+vmSuffix, "critical", "Агенты", agent.Name+": заканчиваются inode "+filesystem.Mountpoint, fmt.Sprintf("Использовано %.1f%% inode · VMID %d.", filesystem.InodesPercent, agent.VMID))
+			}
+		}
+		if len(agent.Report.OOMKills) > 0 {
+			latest := agent.Report.OOMKills[0]
+			add("agent-oom-"+agent.ID+vmSuffix, "critical", "Агенты", agent.Name+": ядро завершало процессы из-за нехватки памяти", fmt.Sprintf("%s · %s · VMID %d.", time.Unix(latest.Time, 0).Format("02.01 15:04"), latest.Message, agent.VMID))
+		}
+		if agent.Report.RebootRequired {
+			add("agent-reboot-"+agent.ID+vmSuffix, "info", "Агенты", agent.Name+": требуется перезагрузка", fmt.Sprintf("Обновления системы запросили перезагрузку · VMID %d.", agent.VMID))
+		}
+		if agent.Report.MemoryTotal > 0 && float64(agent.Report.MemoryUsed)/float64(agent.Report.MemoryTotal)*100 >= 95 {
+			add("agent-memory-"+agent.ID+vmSuffix, "warning", "Агенты", agent.Name+": почти закончилась память", fmt.Sprintf("Используется %.1f%% RAM · VMID %d.", float64(agent.Report.MemoryUsed)/float64(agent.Report.MemoryTotal)*100, agent.VMID))
+		}
+		if agent.Report.RootTotal > 0 && float64(agent.Report.RootUsed)/float64(agent.Report.RootTotal)*100 >= 90 {
+			add("agent-storage-"+agent.ID+vmSuffix, "critical", "Агенты", agent.Name+": мало места на системном разделе", fmt.Sprintf("Раздел / заполнен на %.1f%% · VMID %d.", float64(agent.Report.RootUsed)/float64(agent.Report.RootTotal)*100, agent.VMID))
+		}
+	}
 	for _, service := range metrics.Services.Services {
 		for _, endpoint := range service.Endpoints {
 			if !endpoint.Up {
@@ -164,6 +265,41 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
 	if networkErrors > 0 {
 		add("network-errors", "warning", "Сеть", "Сетевые ошибки и потери пакетов", fmt.Sprintf("Суммарный счётчик интерфейсов: %d.", networkErrors))
 	}
+	if !metrics.Network.Health.CheckedAt.IsZero() && !metrics.Network.Health.GatewayUp {
+		add("network-gateway", "critical", "Сеть", "Недоступен локальный шлюз", firstNonEmpty(metrics.Network.Health.Gateway, "Default gateway не определён."))
+	}
+	if !metrics.Network.Health.CheckedAt.IsZero() && !metrics.Network.Health.DNSUp {
+		add("network-dns", "critical", "Сеть", "Не работает DNS", "Не удалось разрешить deb.debian.org.")
+	}
+	if !metrics.Network.Health.CheckedAt.IsZero() && !metrics.Network.Health.InternetUp {
+		add("network-internet", "warning", "Сеть", "Нет доступа в интернет", "Внешняя проверка IP не выполнена.")
+	}
+	if metrics.Network.Health.PacketLoss > 0 {
+		add("network-loss", "warning", "Сеть", "Потери пакетов до шлюза", fmt.Sprintf("Потери %.1f%%, задержка %.1f ms.", metrics.Network.Health.PacketLoss, metrics.Network.Health.LatencyMs))
+	}
+	if metrics.Network.Health.ExternalIPChanged {
+		add("network-public-ip", "info", "Сеть", "Изменился внешний IP", "Новый адрес: "+metrics.Network.Health.ExternalIP)
+	}
+	for _, name := range metrics.Network.Health.DisappearedInterfaces {
+		add("network-interface-missing-"+name, "warning", "Сеть", "Исчез сетевой интерфейс", name+" больше не присутствует в /proc/net/dev.")
+	}
+	for _, iface := range metrics.Network.Interfaces {
+		if iface.Kind == "Physical" && !iface.Carrier && (iface.Master != "" || iface.ReceiveBytes+iface.TransmitBytes > 0) {
+			add("network-link-"+iface.Name, "warning", "Сеть", iface.Name+": нет carrier", "Физический линк не поднят.")
+		}
+		if (iface.Kind == "Bridge" || iface.Kind == "Bond" || iface.Kind == "VLAN") && iface.State == "down" {
+			add("network-state-"+iface.Name, "warning", "Сеть", iface.Name+": интерфейс выключен", iface.Kind+" находится в состоянии down.")
+		}
+	}
+	if !metrics.Monitor.LastSuccessfulCollection.IsZero() && time.Since(metrics.Monitor.LastSuccessfulCollection) > 30*time.Second {
+		add("monitor-stale", "critical", "Dashboard", "Данные мониторинга устарели", "Последний успешный сбор был "+time.Since(metrics.Monitor.LastSuccessfulCollection).Round(time.Second).String()+" назад.")
+	}
+	if metrics.Monitor.HistoryWriteError != "" {
+		add("monitor-history", "critical", "Dashboard", "Ошибка записи истории", metrics.Monitor.HistoryWriteError)
+	}
+	if metrics.Monitor.DatabaseFreeBytes > 0 && metrics.Monitor.DatabaseFreeBytes < 1024*1024*1024 {
+		add("monitor-db-space", "warning", "Dashboard", "Мало места для базы истории", fmt.Sprintf("На разделе базы свободно %.1f GB.", float64(metrics.Monitor.DatabaseFreeBytes)/1073741824))
+	}
 	oldSnapshots := 0
 	for _, snapshot := range metrics.Activity.Snapshots {
 		if snapshot.AgeSeconds > 30*24*3600 {
@@ -185,6 +321,9 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
 			add("task-failed-"+task.UPID, severity, "Proxmox", task.Type+": задача завершилась ошибкой", fmt.Sprintf("VMID %s, пользователь %s, статус %s.", task.ID, task.User, task.Status))
 		}
 	}
+	for _, trend := range metrics.Trends {
+		add(trend.ID, trend.Severity, trend.Source, trend.Title, trend.Message)
+	}
 
 	priority := map[string]int{"critical": 0, "warning": 1, "info": 2}
 	sort.SliceStable(alerts, func(i, j int) bool {
@@ -196,6 +335,51 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
 	return alerts
 }
 
+func correlateAlerts(metrics dashboardMetrics, alerts []Alert) []Alert {
+	pressureAlerts := map[string]bool{}
+	for _, a := range alerts {
+		if strings.HasPrefix(a.ID, "disk-io-pressure-") {
+			pressureAlerts[strings.TrimPrefix(a.ID, "disk-io-pressure-")] = true
+		}
+	}
+	var pressure []DiskMetrics
+	for _, d := range metrics.Disks {
+		if pressureAlerts[d.Name] && d.IOCause != "" {
+			pressure = append(pressure, d)
+		}
+	}
+	if len(pressure) == 0 {
+		return alerts
+	}
+	var down []string
+	for _, service := range metrics.Services.Services {
+		for _, endpoint := range service.Endpoints {
+			if !endpoint.Up {
+				down = append(down, service.Name)
+				break
+			}
+		}
+	}
+	d := pressure[0]
+	chain := d.IOCause + " → " + d.Name + fmt.Sprintf(" util %.1f%% → latency %.1f ms", d.Utilization, d.LatencyMs)
+	if len(down) > 0 {
+		chain += " → недоступны/медленны сервисы: " + strings.Join(down, ", ")
+	}
+	severity := "warning"
+	if len(down) > 0 || d.Utilization >= 98 {
+		severity = "critical"
+	}
+	correlated := Alert{ID: "correlation-io-" + d.Name, Severity: severity, Source: "Корреляция", Title: "Обнаружена связанная цепочка I/O-событий", Message: chain + fmt.Sprintf(". Уверенность причины %d%%.", d.IOConfidence)}
+	filtered := []Alert{correlated}
+	for _, a := range alerts {
+		if strings.HasPrefix(a.ID, "disk-io-pressure-") || strings.HasPrefix(a.ID, "service-") {
+			continue
+		}
+		filtered = append(filtered, a)
+	}
+	return filtered
+}
+
 func joinReasons(reasons []string, fallback string) string {
 	if len(reasons) == 0 {
 		return fallback

alerts_test.go

@@ -16,3 +16,12 @@ func TestEvaluateAlerts(t *testing.T) {
 		t.Fatalf("неверная сортировка уведомлений: %+v", alerts)
 	}
 }
+
+func TestSlowCollectionDoesNotCreateAlert(t *testing.T) {
+	metrics := dashboardMetrics{Monitor: MonitorHealth{SlowCollector: true, CollectionDurationMs: 12308}}
+	for _, alert := range evaluateAlerts(metrics, defaultThresholds()) {
+		if alert.ID == "monitor-slow" {
+			t.Fatalf("медленный цикл не должен создавать уведомление: %+v", alert)
+		}
+	}
+}

disks.go

@@ -25,28 +25,207 @@ type DiskMetrics struct {
 	SMARTStatus       string       `json:"smartStatus"`
 	SMARTAvailable    bool         `json:"smartAvailable"`
 	AttentionReasons  []string     `json:"attentionReasons"`
+	SMARTHistory      []string     `json:"smartHistory"`
 	Temperature       *float64     `json:"temperatureCelsius"`
 	PowerOnHours      *uint64      `json:"powerOnHours"`
 	WearUsedPercent   *float64     `json:"wearUsedPercent"`
+	Reallocated       uint64       `json:"reallocated"`
+	Pending           uint64       `json:"pending"`
+	Uncorrectable     uint64       `json:"uncorrectable"`
+	CRCErrors         uint64       `json:"crcErrors"`
+	MediaErrors       uint64       `json:"mediaErrors"`
+	SMARTErrorCount   uint64       `json:"smartErrorCount"`
+	CounterChanges    []string     `json:"counterChanges"`
+	LastSelfTest      string       `json:"lastSelfTest"`
+	LastSelfTestAt    string       `json:"lastSelfTestAt"`
+	SelfTestNeverRun  bool         `json:"selfTestNeverRun"`
+	ShortTestStarted  bool         `json:"shortTestStarted"`
+	ReadIOPS          float64      `json:"readIops"`
+	WriteIOPS         float64      `json:"writeIops"`
+	ReadBytesPerSec   float64      `json:"readBytesPerSec"`
+	WriteBytesPerSec  float64      `json:"writeBytesPerSec"`
+	ReadLatencyMs     float64      `json:"readLatencyMs"`
+	WriteLatencyMs    float64      `json:"writeLatencyMs"`
+	LatencyMs         float64      `json:"latencyMs"`
+	Utilization       float64      `json:"utilizationPercent"`
+	QueueDepth        uint64       `json:"queueDepth"`
+	AverageQueue      float64      `json:"averageQueue"`
+	IOPressureSeconds float64      `json:"ioPressureSeconds"`
+	IOCause           string       `json:"ioCause"`
+	IOCauseDetail     string       `json:"ioCauseDetail"`
+	IOConfidence      int          `json:"ioConfidence"`
+	TopConsumers      []IOConsumer `json:"topConsumers"`
 }
 
+type diskIOSnapshot struct{ reads, readSectors, readMS, writes, writeSectors, writeMS, inFlight, ioMS, weightedMS uint64 }
+
 type diskCollector struct {
 	mu              sync.Mutex
 	cached          []DiskMetrics
 	updatedAt       time.Time
+	previous        map[string]DiskMetrics
+	missing         []string
+	lastTests       map[string]time.Time
+	ioPrevious      map[string]diskIOSnapshot
+	ioUpdatedAt     time.Time
+	ioPressureSince map[string]time.Time
 }
 
-func (c *diskCollector) collect() []DiskMetrics {
+func (c *diskCollector) collect(thresholds AlertThresholds) []DiskMetrics {
 	c.mu.Lock()
 	defer c.mu.Unlock()
 	if time.Since(c.updatedAt) < time.Minute && c.cached != nil {
+		c.applyDiskIO(c.cached, thresholds)
 		return c.cached
 	}
-	c.cached = readPhysicalDisks()
+	current := readPhysicalDisks()
+	if c.previous == nil {
+		c.previous = map[string]DiskMetrics{}
+		c.lastTests = map[string]time.Time{}
+	}
+	seen := map[string]bool{}
+	c.missing = nil
+	for i := range current {
+		disk := &current[i]
+		key := disk.Serial
+		if key == "" || key == "Не указан" {
+			key = disk.Path
+		}
+		seen[key] = true
+		if old, ok := c.previous[key]; ok {
+			disk.CounterChanges = diskCounterChanges(old, *disk)
+			disk.AttentionReasons = append(disk.AttentionReasons, disk.CounterChanges...)
+		}
+		if disk.SMARTAvailable && (disk.SelfTestNeverRun || selfTestOlderThanWeek(disk.LastSelfTestAt)) && time.Since(c.lastTests[key]) >= 7*24*time.Hour {
+			disk.ShortTestStarted = startShortSMARTTest(disk.Path)
+			if disk.ShortTestStarted {
+				c.lastTests[key] = time.Now()
+			}
+		}
+		c.previous[key] = *disk
+	}
+	for key, disk := range c.previous {
+		if !seen[key] {
+			c.missing = append(c.missing, disk.Model+" ("+disk.Serial+")")
+		}
+	}
+	c.cached = current
+	c.applyDiskIO(c.cached, thresholds)
 	c.updatedAt = time.Now()
 	return c.cached
 }
 
+func (c *diskCollector) applyDiskIO(disks []DiskMetrics, thresholds AlertThresholds) {
+	now := time.Now()
+	snapshots := readDiskIOStats("/proc/diskstats")
+	if c.ioPrevious == nil {
+		c.ioPrevious = map[string]diskIOSnapshot{}
+		c.ioPressureSince = map[string]time.Time{}
+	}
+	elapsed := now.Sub(c.ioUpdatedAt).Seconds()
+	for i := range disks {
+		disk := &disks[i]
+		disk.ReadIOPS, disk.WriteIOPS, disk.ReadBytesPerSec, disk.WriteBytesPerSec = 0, 0, 0, 0
+		disk.ReadLatencyMs, disk.WriteLatencyMs, disk.LatencyMs, disk.Utilization, disk.AverageQueue, disk.IOPressureSeconds = 0, 0, 0, 0, 0, 0
+		current, ok := snapshots[disk.Name]
+		if !ok {
+			continue
+		}
+		disk.QueueDepth = current.inFlight
+		if old, found := c.ioPrevious[disk.Name]; found && elapsed > 0 && diskIOCountersMonotonic(old, current) {
+			readOps := current.reads - old.reads
+			writeOps := current.writes - old.writes
+			disk.ReadIOPS = float64(readOps) / elapsed
+			disk.WriteIOPS = float64(writeOps) / elapsed
+			disk.ReadBytesPerSec = float64(current.readSectors-old.readSectors) * 512 / elapsed
+			disk.WriteBytesPerSec = float64(current.writeSectors-old.writeSectors) * 512 / elapsed
+			if readOps > 0 {
+				disk.ReadLatencyMs = float64(current.readMS-old.readMS) / float64(readOps)
+			}
+			if writeOps > 0 {
+				disk.WriteLatencyMs = float64(current.writeMS-old.writeMS) / float64(writeOps)
+			}
+			if readOps+writeOps > 0 {
+				disk.LatencyMs = float64(current.readMS-old.readMS+current.writeMS-old.writeMS) / float64(readOps+writeOps)
+			}
+			disk.Utilization = min(float64(current.ioMS-old.ioMS)/(elapsed*10), 100)
+			disk.AverageQueue = float64(current.weightedMS-old.weightedMS) / (elapsed * 1000)
+			utilLimit, latencyLimit, queueLimit := thresholds.DiskSSDUtil, thresholds.DiskSSDLatency, thresholds.DiskSSDQueue
+			if disk.Type == "HDD" {
+				utilLimit, latencyLimit, queueLimit = thresholds.DiskHDDUtil, thresholds.DiskHDDLatency, thresholds.DiskHDDQueue
+			} else if strings.Contains(disk.Type, "NVMe") {
+				utilLimit, latencyLimit, queueLimit = thresholds.DiskNVMeUtil, thresholds.DiskNVMeLatency, thresholds.DiskNVMeQueue
+			}
+			pressured := disk.Utilization >= utilLimit || (disk.LatencyMs >= latencyLimit && disk.ReadIOPS+disk.WriteIOPS >= 1) || disk.AverageQueue >= queueLimit
+			if pressured {
+				if c.ioPressureSince[disk.Name].IsZero() {
+					c.ioPressureSince[disk.Name] = now
+				}
+				disk.IOPressureSeconds = now.Sub(c.ioPressureSince[disk.Name]).Seconds()
+			} else {
+				delete(c.ioPressureSince, disk.Name)
+			}
+		}
+		c.ioPrevious[disk.Name] = current
+	}
+	c.ioUpdatedAt = now
+}
+
+func diskIOCountersMonotonic(old, current diskIOSnapshot) bool {
+	return current.reads >= old.reads && current.readSectors >= old.readSectors && current.readMS >= old.readMS && current.writes >= old.writes && current.writeSectors >= old.writeSectors && current.writeMS >= old.writeMS && current.ioMS >= old.ioMS && current.weightedMS >= old.weightedMS
+}
+
+func readDiskIOStats(path string) map[string]diskIOSnapshot {
+	data, err := os.ReadFile(path)
+	if err != nil {
+		return nil
+	}
+	result := map[string]diskIOSnapshot{}
+	for _, line := range strings.Split(string(data), "\n") {
+		f := strings.Fields(line)
+		if len(f) < 14 || !isPhysicalDiskName(f[2]) {
+			continue
+		}
+		values := make([]uint64, 11)
+		for i := 0; i < 11; i++ {
+			values[i], _ = strconv.ParseUint(f[i+3], 10, 64)
+		}
+		result[f[2]] = diskIOSnapshot{reads: values[0], readSectors: values[2], readMS: values[3], writes: values[4], writeSectors: values[6], writeMS: values[7], inFlight: values[8], ioMS: values[9], weightedMS: values[10]}
+	}
+	return result
+}
+
+func (c *diskCollector) missingDisks() []string {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	return append([]string(nil), c.missing...)
+}
+
+func diskCounterChanges(old, current DiskMetrics) []string {
+	checks := []struct {
+		label    string
+		old, now uint64
+	}{{"Reallocated вырос", old.Reallocated, current.Reallocated}, {"Pending вырос", old.Pending, current.Pending}, {"Uncorrectable вырос", old.Uncorrectable, current.Uncorrectable}, {"CRC-ошибки выросли", old.CRCErrors, current.CRCErrors}, {"Media errors выросли", old.MediaErrors, current.MediaErrors}, {"Ошибки SMART-журнала выросли", old.SMARTErrorCount, current.SMARTErrorCount}}
+	var changes []string
+	for _, check := range checks {
+		if check.now > check.old {
+			changes = append(changes, check.label+": "+strconv.FormatUint(check.old, 10)+" → "+strconv.FormatUint(check.now, 10))
+		}
+	}
+	return changes
+}
+
+func selfTestOlderThanWeek(value string) bool {
+	when, err := time.Parse(time.RFC3339, value)
+	return err == nil && time.Since(when) > 7*24*time.Hour
+}
+
+func startShortSMARTTest(path string) bool {
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+	return exec.CommandContext(ctx, "smartctl", "-t", "short", path).Run() == nil
+}
+
 func readPhysicalDisks() []DiskMetrics {
 	entries, _ := os.ReadDir("/sys/block")
 	usage := readDiskUsage()
@@ -202,6 +381,16 @@ type smartctlJSON struct {
 			Count int `json:"count"`
 		} `json:"summary"`
 	} `json:"ata_smart_error_log"`
+	ATASelfTestLog struct {
+		Standard struct {
+			Table []struct {
+				Status struct {
+					String string `json:"string"`
+				} `json:"status"`
+				LifetimeHours uint64 `json:"lifetime_hours"`
+			} `json:"table"`
+		} `json:"standard"`
+	} `json:"ata_smart_self_test_log"`
 	NVMeHealth *struct {
 		Temperature     float64 `json:"temperature"`
 		PercentageUsed  float64 `json:"percentage_used"`
@@ -254,6 +443,7 @@ func applySMART(disk *DiskMetrics) {
 	}
 	if smart.NVMeHealth != nil {
 		disk.WearUsedPercent = &smart.NVMeHealth.PercentageUsed
+		disk.MediaErrors = smart.NVMeHealth.MediaErrors
 		if smart.NVMeHealth.CriticalWarning != 0 {
 			disk.AttentionReasons = append(disk.AttentionReasons, "NVMe сообщает критическое предупреждение")
 		}
@@ -261,6 +451,17 @@ func applySMART(disk *DiskMetrics) {
 			disk.AttentionReasons = append(disk.AttentionReasons, "Ошибки целостности носителя: "+strconv.FormatUint(smart.NVMeHealth.MediaErrors, 10))
 		}
 	}
+	if len(smart.ATASelfTestLog.Standard.Table) > 0 {
+		last := smart.ATASelfTestLog.Standard.Table[0]
+		disk.LastSelfTest = firstNonEmpty(last.Status.String, "Результат неизвестен")
+		if disk.PowerOnHours != nil && *disk.PowerOnHours >= last.LifetimeHours {
+			disk.LastSelfTestAt = time.Now().Add(-time.Duration(*disk.PowerOnHours-last.LifetimeHours) * time.Hour).Format(time.RFC3339)
+		}
+	} else {
+		disk.SelfTestNeverRun = true
+		disk.LastSelfTest = "Никогда не запускался"
+		disk.AttentionReasons = append(disk.AttentionReasons, "SMART self-test ещё ни разу не запускался")
+	}
 	applyATAAttributes(disk, smart)
 	applySMARTExitStatus(disk, smart.Smartctl.ExitStatus)
 	if len(disk.AttentionReasons) > 0 && disk.SMARTStatus == "Исправен" {
@@ -275,20 +476,37 @@ func applyATAAttributes(disk *DiskMetrics, smart smartctlJSON) {
 		"Wear_Leveling_Count": true,
 	}
 	for _, attribute := range smart.ATAAttributes.Table {
+		switch attribute.Name {
+		case "Reallocated_Sector_Ct", "Reallocated_Event_Count":
+			disk.Reallocated += uint64(max(attribute.Raw.Value, 0))
+		case "Current_Pending_Sector":
+			disk.Pending = uint64(max(attribute.Raw.Value, 0))
+		case "Offline_Uncorrectable", "Reported_Uncorrect":
+			disk.Uncorrectable += uint64(max(attribute.Raw.Value, 0))
+		case "UDMA_CRC_Error_Count", "Interface_CRC_Error_Count":
+			disk.CRCErrors += uint64(max(attribute.Raw.Value, 0))
+		}
 		if wearNames[attribute.Name] && attribute.Value >= 0 && attribute.Value <= 100 {
 			wear := float64(100 - attribute.Value)
 			disk.WearUsedPercent = &wear
 		}
-		if attribute.WhenFailed != "" && attribute.WhenFailed != "-" {
+		if strings.EqualFold(attribute.WhenFailed, "FAILING_NOW") {
 			reason := attribute.Name + " вышел за порог"
 			if attribute.Raw.String != "" {
 				reason += " (" + attribute.Raw.String + ")"
 			}
 			disk.AttentionReasons = append(disk.AttentionReasons, reason)
+		} else if attribute.WhenFailed != "" && attribute.WhenFailed != "-" {
+			reason := attribute.Name + " ранее выходил за порог"
+			if attribute.Raw.String != "" {
+				reason += " (" + attribute.Raw.String + ")"
+			}
+			disk.SMARTHistory = append(disk.SMARTHistory, reason)
 		}
 	}
 	if smart.ATAErrorLog.Summary.Count > 0 {
-		disk.AttentionReasons = append(disk.AttentionReasons, "Ошибок в SMART-журнале: "+strconv.Itoa(smart.ATAErrorLog.Summary.Count))
+		disk.SMARTErrorCount = uint64(smart.ATAErrorLog.Summary.Count)
+		disk.SMARTHistory = append(disk.SMARTHistory, "Записей в SMART-журнале: "+strconv.Itoa(smart.ATAErrorLog.Summary.Count))
 	}
 }
 
@@ -302,12 +520,12 @@ func applySMARTExitStatus(disk *DiskMetrics, status int) {
 		disk.AttentionReasons = append(disk.AttentionReasons, "Критический SMART-атрибут достиг порога")
 	}
 	if status&32 != 0 {
-		disk.AttentionReasons = append(disk.AttentionReasons, "SMART-атрибут ранее находился ниже порога")
+		disk.SMARTHistory = append(disk.SMARTHistory, "SMART-атрибут ранее находился ниже порога")
 	}
 	if status&64 != 0 {
-		disk.AttentionReasons = append(disk.AttentionReasons, "В журнале SMART обнаружены ошибки")
+		disk.SMARTHistory = append(disk.SMARTHistory, "В журнале SMART есть старые записи об ошибках")
 	}
 	if status&128 != 0 {
-		disk.AttentionReasons = append(disk.AttentionReasons, "В журнале самотестирования есть ошибки")
+		disk.SMARTHistory = append(disk.SMARTHistory, "В журнале самотестирования есть старые ошибки")
 	}
 }

disks_test.go

@@ -2,6 +2,9 @@ package main
 
 import (
 	"encoding/json"
+	"os"
+	"path/filepath"
+	"strings"
 	"testing"
 )
 
@@ -17,6 +20,22 @@ func TestIsPhysicalDiskName(t *testing.T) {
 	}
 }
 
+func TestReadDiskIOStats(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "diskstats")
+	data := "   8       0 sda 100 2 300 40 50 3 700 80 4 90 120 0 0 0 0 0 0\n   7       0 loop0 999 0 999 0 999 0 999 0 0 0 0\n"
+	if err := os.WriteFile(path, []byte(data), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	stats := readDiskIOStats(path)
+	disk, ok := stats["sda"]
+	if !ok || disk.reads != 100 || disk.readSectors != 300 || disk.writes != 50 || disk.writeSectors != 700 || disk.inFlight != 4 || disk.ioMS != 90 || disk.weightedMS != 120 {
+		t.Fatalf("unexpected stats: %+v", disk)
+	}
+	if _, exists := stats["loop0"]; exists {
+		t.Fatal("virtual device must be ignored")
+	}
+}
+
 func TestSumFilesystemUsage(t *testing.T) {
 	device := lsblkDevice{Children: []lsblkDevice{
 		{FSUsed: json.RawMessage(`100`), FSAvail: json.RawMessage(`300`)},
@@ -50,3 +69,37 @@ func TestApplyATAWearAndReason(t *testing.T) {
 		t.Fatalf("причина предупреждения не добавлена: %+v", disk)
 	}
 }
+
+func TestHistoricalATAFailureDoesNotRequireAttention(t *testing.T) {
+	var smart smartctlJSON
+	attribute := struct {
+		Name       string `json:"name"`
+		Value      int    `json:"value"`
+		Threshold  int    `json:"thresh"`
+		WhenFailed string `json:"when_failed"`
+		Raw        struct {
+			String string `json:"string"`
+			Value  int64  `json:"value"`
+		} `json:"raw"`
+	}{Name: "Airflow_Temperature_Cel", WhenFailed: "In_the_past"}
+	attribute.Raw.String = "46 (Min/Max 26/47 #15)"
+	smart.ATAAttributes.Table = append(smart.ATAAttributes.Table, attribute)
+	disk := DiskMetrics{}
+	applyATAAttributes(&disk, smart)
+	applySMARTExitStatus(&disk, 32|64|128)
+	if len(disk.AttentionReasons) != 0 {
+		t.Fatalf("историческая ошибка стала активной: %+v", disk.AttentionReasons)
+	}
+	if len(disk.SMARTHistory) != 4 {
+		t.Fatalf("история SMART потеряна: %+v", disk.SMARTHistory)
+	}
+}
+
+func TestSMARTErrorLogGrowthRequiresAttention(t *testing.T) {
+	old := DiskMetrics{SMARTErrorCount: 2}
+	current := DiskMetrics{SMARTErrorCount: 3}
+	changes := diskCounterChanges(old, current)
+	if len(changes) != 1 || !strings.Contains(changes[0], "2 → 3") {
+		t.Fatalf("рост журнала не обнаружен: %+v", changes)
+	}
+}

history.go

@@ -44,6 +44,11 @@ func (s *Store) AddEntityMetrics(metrics dashboardMetrics) {
 		s.AddEntityHistory("guest", fmt.Sprintf("%d · %s", g.VMID, g.Name), ts, &cpu, &mem)
 	}
 	for _, d := range metrics.Disks {
+		iops := d.ReadIOPS + d.WriteIOPS
+		util := d.Utilization
+		s.AddEntityHistory("disk-io", d.Name+" · "+d.Model, ts, &iops, &util)
+		latency := d.LatencyMs
+		s.AddEntityHistory("disk-latency", d.Name+" · "+d.Model, ts, &latency, nil)
 		if d.Temperature == nil {
 			continue
 		}
@@ -53,6 +58,12 @@ func (s *Store) AddEntityMetrics(metrics dashboardMetrics) {
 		}
 		s.AddEntityHistory("disk", d.Name+" · "+d.Model, ts, d.Temperature, wear)
 	}
+	for _, task := range metrics.Activity.Tasks {
+		if task.Type == "vzdump" && task.Duration > 0 && task.EndTime >= time.Now().Add(-15*time.Minute).Unix() {
+			duration := float64(task.Duration)
+			s.AddEntityHistory("backup-duration", "VMID "+task.ID, ts, &duration, nil)
+		}
+	}
 	s.PruneEntityHistory()
 }
 

io_attribution.go

@@ -0,0 +1,178 @@
+package main
+
+import (
+	"os"
+	"path/filepath"
+	"regexp"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+)
+
+type IOConsumer struct {
+	Kind             string  `json:"kind"`
+	Name             string  `json:"name"`
+	VMID             int     `json:"vmid,omitempty"`
+	ReadBytesPerSec  float64 `json:"readBytesPerSec"`
+	WriteBytesPerSec float64 `json:"writeBytesPerSec"`
+	SharePercent     float64 `json:"sharePercent"`
+}
+type processIOSnapshot struct {
+	read, write           uint64
+	name, cmdline, cgroup string
+}
+type ioAttributionCollector struct {
+	mu        sync.Mutex
+	previous  map[int]processIOSnapshot
+	updatedAt time.Time
+	cached    []IOConsumer
+}
+
+var qemuIDPattern = regexp.MustCompile(`(?:^|\s)-id\s+(\d+)`)
+var lxcIDPattern = regexp.MustCompile(`(?:lxc(?:\.payload)?[./-]|pve-container@)(\d+)`)
+
+func (c *ioAttributionCollector) collect(guests GuestsMetrics) []IOConsumer {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	now := time.Now()
+	current := readProcessIO()
+	elapsed := now.Sub(c.updatedAt).Seconds()
+	if c.previous == nil || elapsed <= 0 {
+		c.previous = current
+		c.updatedAt = now
+		return nil
+	}
+	names := map[int]string{}
+	for _, g := range guests.Guests {
+		names[g.VMID] = g.Name
+	}
+	grouped := map[string]*IOConsumer{}
+	var total float64
+	for pid, snapshot := range current {
+		old, ok := c.previous[pid]
+		if !ok || snapshot.read < old.read || snapshot.write < old.write {
+			continue
+		}
+		read := float64(snapshot.read-old.read) / elapsed
+		write := float64(snapshot.write-old.write) / elapsed
+		if read+write < 1024 {
+			continue
+		}
+		kind, name, vmid := classifyIOProcess(snapshot, names)
+		key := kind + ":" + strconv.Itoa(vmid) + ":" + name
+		consumer := grouped[key]
+		if consumer == nil {
+			consumer = &IOConsumer{Kind: kind, Name: name, VMID: vmid}
+			grouped[key] = consumer
+		}
+		consumer.ReadBytesPerSec += read
+		consumer.WriteBytesPerSec += write
+		total += read + write
+	}
+	var result []IOConsumer
+	for _, v := range grouped {
+		if total > 0 {
+			v.SharePercent = (v.ReadBytesPerSec + v.WriteBytesPerSec) / total * 100
+		}
+		result = append(result, *v)
+	}
+	sort.Slice(result, func(i, j int) bool {
+		return result[i].ReadBytesPerSec+result[i].WriteBytesPerSec > result[j].ReadBytesPerSec+result[j].WriteBytesPerSec
+	})
+	if len(result) > 8 {
+		result = result[:8]
+	}
+	c.previous = current
+	c.updatedAt = now
+	c.cached = result
+	return result
+}
+
+func classifyIOProcess(p processIOSnapshot, names map[int]string) (kind, name string, vmid int) {
+	if match := qemuIDPattern.FindStringSubmatch(p.cmdline); len(match) > 1 {
+		vmid, _ = strconv.Atoi(match[1])
+		return "VM", firstNonEmpty(names[vmid], "VM "+match[1]), vmid
+	}
+	if match := lxcIDPattern.FindStringSubmatch(p.cgroup); len(match) > 1 {
+		vmid, _ = strconv.Atoi(match[1])
+		return "LXC", firstNonEmpty(names[vmid], "LXC "+match[1]), vmid
+	}
+	return "Процесс", firstNonEmpty(p.name, "PID"), 0
+}
+
+func readProcessIO() map[int]processIOSnapshot {
+	result := map[int]processIOSnapshot{}
+	entries, _ := filepath.Glob("/proc/[0-9]*/io")
+	for _, path := range entries {
+		pid, _ := strconv.Atoi(filepath.Base(filepath.Dir(path)))
+		data, err := os.ReadFile(path)
+		if err != nil {
+			continue
+		}
+		var item processIOSnapshot
+		for _, line := range strings.Split(string(data), "\n") {
+			key, value, ok := strings.Cut(line, ":")
+			if !ok {
+				continue
+			}
+			n, _ := strconv.ParseUint(strings.TrimSpace(value), 10, 64)
+			if key == "read_bytes" {
+				item.read = n
+			} else if key == "write_bytes" {
+				item.write = n
+			}
+		}
+		comm, _ := os.ReadFile(filepath.Join(filepath.Dir(path), "comm"))
+		cmd, _ := os.ReadFile(filepath.Join(filepath.Dir(path), "cmdline"))
+		cg, _ := os.ReadFile(filepath.Join(filepath.Dir(path), "cgroup"))
+		item.name = strings.TrimSpace(string(comm))
+		item.cmdline = strings.ReplaceAll(string(cmd), "\x00", " ")
+		item.cgroup = string(cg)
+		result[pid] = item
+	}
+	return result
+}
+
+func attributeDiskIO(metrics *dashboardMetrics, consumers []IOConsumer) {
+	active := activeIOOperation(metrics.Activity.Tasks)
+	for i := range metrics.Disks {
+		disk := &metrics.Disks[i]
+		disk.TopConsumers = append([]IOConsumer(nil), consumers...)
+		if active != "" {
+			disk.IOCause = active
+			disk.IOCauseDetail = "Активная задача Proxmox совпадает по времени с дисковой нагрузкой"
+			disk.IOConfidence = 75
+		} else if len(consumers) > 0 {
+			top := consumers[0]
+			disk.IOCause = top.Kind + " · " + top.Name
+			disk.IOCauseDetail = "Крупнейший потребитель host I/O: " + strconv.FormatFloat(top.SharePercent, 'f', 1, 64) + "%"
+			disk.IOConfidence = 55
+		}
+	}
+}
+
+func activeIOOperation(tasks []ProxmoxTask) string {
+	for _, task := range tasks {
+		if task.EndTime != 0 && task.Status != "" {
+			continue
+		}
+		label := ""
+		switch task.Type {
+		case "vzdump":
+			label = "Backup"
+		case "qmigrate":
+			label = "Миграция"
+		case "qmsnapshot", "vzsnapshot":
+			label = "Snapshot"
+		}
+		if label != "" {
+			if task.ID != "" {
+				label += " VM/LXC " + task.ID
+			}
+			return label
+		}
+	}
+	return ""
+}

io_attribution_test.go

@@ -0,0 +1,33 @@
+package main
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestClassifyQEMUAndLXCIO(t *testing.T) {
+	kind, name, vmid := classifyIOProcess(processIOSnapshot{name: "qemu", cmdline: "qemu-system-x86_64 -id 105 -name guest=test"}, map[int]string{105: "Gitea"})
+	if kind != "VM" || name != "Gitea" || vmid != 105 {
+		t.Fatalf("unexpected qemu attribution: %s %s %d", kind, name, vmid)
+	}
+	kind, name, vmid = classifyIOProcess(processIOSnapshot{name: "worker", cgroup: "0::/lxc.payload.220/ns"}, map[int]string{220: "db"})
+	if kind != "LXC" || name != "db" || vmid != 220 {
+		t.Fatalf("unexpected lxc attribution: %s %s %d", kind, name, vmid)
+	}
+}
+
+func TestActiveIOOperation(t *testing.T) {
+	value := activeIOOperation([]ProxmoxTask{{Type: "vzdump", ID: "105", Status: "", EndTime: 0}})
+	if !strings.Contains(value, "Backup") || !strings.Contains(value, "105") {
+		t.Fatalf("unexpected operation: %s", value)
+	}
+}
+
+func TestCorrelateAlerts(t *testing.T) {
+	metrics := dashboardMetrics{Disks: []DiskMetrics{{Name: "sda", IOCause: "Backup VM/LXC 105", IOConfidence: 90, Utilization: 98, LatencyMs: 80}}, Services: ServicesMetrics{Services: []ServiceStatus{{MonitoredService: MonitoredService{Name: "Gitea"}, Endpoints: []ServiceEndpointStatus{{Up: false}}}}}}
+	alerts := []Alert{{ID: "disk-io-pressure-sda", Severity: "critical"}, {ID: "service-1-public", Severity: "critical"}, {ID: "other", Severity: "warning"}}
+	result := correlateAlerts(metrics, alerts)
+	if len(result) != 2 || result[0].Source != "Корреляция" || !strings.Contains(result[0].Message, "Gitea") {
+		t.Fatalf("unexpected correlation: %+v", result)
+	}
+}

main.go

@@ -20,11 +20,23 @@ var version = "dev"
 
 func main() {
 	showVersion := flag.Bool("version", false, "show version")
+	agentMode := flag.Bool("agent", false, "run as a read-only guest agent")
+	agentServer := flag.String("agent-server", "", "dashboard URL used by the guest agent")
+	agentEnrollToken := flag.String("agent-enroll-token", "", "one-time enrollment token")
+	agentName := flag.String("agent-name", "", "agent display name")
+	agentVMID := flag.Int("agent-vmid", 0, "Proxmox VM/LXC ID")
+	agentConfig := flag.String("agent-config", "/var/lib/proxmox-dashboard-agent/agent.json", "agent configuration path")
 	flag.Parse()
 	if *showVersion {
 		fmt.Println(version)
 		return
 	}
+	if *agentMode {
+		if err := runGuestAgent(agentOptions{Server: *agentServer, EnrollmentToken: *agentEnrollToken, Name: *agentName, VMID: *agentVMID, ConfigPath: *agentConfig}); err != nil {
+			log.Fatalf("Ошибка агента: %v", err)
+		}
+		return
+	}
 
 	address := os.Getenv("DASHBOARD_ADDR")
 	if address == "" {
@@ -50,6 +62,7 @@ func main() {
 		Handler:           routes(collector, store),
 		ReadHeaderTimeout: 5 * time.Second,
 	}
+	runSystemdWatchdog()
 
 	go func() {
 		log.Printf("Proxmox Dashboard %s запущен на http://0.0.0.0%s", version, address)

maintenance_window.go

@@ -0,0 +1,125 @@
+package main
+
+import (
+	"encoding/json"
+	"fmt"
+	"strconv"
+	"strings"
+	"time"
+)
+
+type MaintenanceWindow struct {
+	Enabled    bool    `json:"enabled"`
+	Active     bool    `json:"active"`
+	StartedAt  int64   `json:"startedAt"`
+	EndsAt     int64   `json:"endsAt"`
+	Reason     string  `json:"reason"`
+	AllAlerts  bool    `json:"allAlerts"`
+	ServiceIDs []int64 `json:"serviceIds"`
+	VMIDs      []int   `json:"vmids"`
+}
+
+func (s *Store) MaintenanceWindow() MaintenanceWindow {
+	var value string
+	var window MaintenanceWindow
+	if s.db.QueryRow(`SELECT value FROM settings WHERE key='maintenance_window'`).Scan(&value) == nil {
+		_ = json.Unmarshal([]byte(value), &window)
+	}
+	now := time.Now().Unix()
+	window.Active = window.Enabled && window.StartedAt <= now && window.EndsAt > now
+	if window.Enabled && window.EndsAt <= now {
+		window.Enabled = false
+		data, _ := json.Marshal(window)
+		_, _ = s.db.Exec(`INSERT INTO settings(key,value) VALUES('maintenance_window',?) ON CONFLICT(key) DO UPDATE SET value=excluded.value`, string(data))
+	}
+	return window
+}
+
+func (s *Store) SaveMaintenanceWindow(window MaintenanceWindow) (MaintenanceWindow, error) {
+	now := time.Now()
+	window.Reason = strings.TrimSpace(window.Reason)
+	if window.Reason == "" {
+		return window, fmt.Errorf("укажите причину плановых работ")
+	}
+	if window.EndsAt <= now.Unix() || window.EndsAt > now.Add(30*24*time.Hour).Unix() {
+		return window, fmt.Errorf("окончание должно быть в будущем, максимум через 30 дней")
+	}
+	if !window.AllAlerts && len(window.ServiceIDs) == 0 && len(window.VMIDs) == 0 {
+		return window, fmt.Errorf("выберите сервис, VM/LXC или все алерты")
+	}
+	window.Enabled = true
+	window.Active = true
+	window.StartedAt = now.Unix()
+	window.ServiceIDs = uniqueInt64(window.ServiceIDs)
+	window.VMIDs = uniqueInt(window.VMIDs)
+	data, _ := json.Marshal(window)
+	_, err := s.db.Exec(`INSERT INTO settings(key,value) VALUES('maintenance_window',?) ON CONFLICT(key) DO UPDATE SET value=excluded.value`, string(data))
+	return window, err
+}
+
+func (s *Store) StopMaintenanceWindow() (MaintenanceWindow, error) {
+	window := s.MaintenanceWindow()
+	window.Enabled = false
+	window.Active = false
+	window.EndsAt = time.Now().Unix()
+	data, _ := json.Marshal(window)
+	_, err := s.db.Exec(`INSERT INTO settings(key,value) VALUES('maintenance_window',?) ON CONFLICT(key) DO UPDATE SET value=excluded.value`, string(data))
+	return window, err
+}
+
+func splitMaintenanceAlerts(alerts []Alert, window MaintenanceWindow) (active, planned []Alert) {
+	if !window.Active {
+		return alerts, nil
+	}
+	for _, alert := range alerts {
+		if maintenanceMatchesAlert(window, alert) {
+			planned = append(planned, alert)
+		} else {
+			active = append(active, alert)
+		}
+	}
+	return active, planned
+}
+
+func maintenanceMatchesAlert(window MaintenanceWindow, alert Alert) bool {
+	if window.AllAlerts {
+		return true
+	}
+	for _, id := range window.ServiceIDs {
+		value := strconv.FormatInt(id, 10)
+		if strings.HasPrefix(alert.ID, "service-"+value+"-") || strings.HasPrefix(alert.ID, "service-tls-"+value+"-") {
+			return true
+		}
+	}
+	for _, vmid := range window.VMIDs {
+		value := strconv.Itoa(vmid)
+		if strings.HasSuffix(alert.ID, "-"+value) || strings.Contains(alert.Message, "VMID "+value) {
+			return true
+		}
+	}
+	return false
+}
+
+func uniqueInt64(values []int64) []int64 {
+	seen := map[int64]bool{}
+	result := make([]int64, 0, len(values))
+	for _, value := range values {
+		if value > 0 && !seen[value] {
+			seen[value] = true
+			result = append(result, value)
+		}
+	}
+	return result
+}
+
+func uniqueInt(values []int) []int {
+	seen := map[int]bool{}
+	result := make([]int, 0, len(values))
+	for _, value := range values {
+		if value > 0 && !seen[value] {
+			seen[value] = true
+			result = append(result, value)
+		}
+	}
+	return result
+}

maintenance_window_test.go

@@ -0,0 +1,60 @@
+package main
+
+import (
+	"testing"
+	"time"
+)
+
+func TestMaintenanceFiltersSelectedTargets(t *testing.T) {
+	window := MaintenanceWindow{Active: true, ServiceIDs: []int64{7}, VMIDs: []int{105}}
+	alerts := []Alert{
+		{ID: "service-7-local"},
+		{ID: "service-tls-8-public"},
+		{ID: "backup-old-105"},
+		{ID: "task-failed-x", Message: "VMID 105, status error"},
+		{ID: "cpu-usage"},
+	}
+	active, planned := splitMaintenanceAlerts(alerts, window)
+	if len(active) != 2 || len(planned) != 3 {
+		t.Fatalf("неверное разделение: active=%+v planned=%+v", active, planned)
+	}
+}
+
+func TestMaintenanceExpiresAutomatically(t *testing.T) {
+	store, err := openStore(":memory:")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer store.Close()
+	window, err := store.SaveMaintenanceWindow(MaintenanceWindow{EndsAt: time.Now().Add(time.Hour).Unix(), Reason: "Обновление", AllAlerts: true})
+	if err != nil || !window.Active {
+		t.Fatalf("режим не включён: %+v %v", window, err)
+	}
+	if _, err = store.db.Exec(`UPDATE settings SET value=? WHERE key='maintenance_window'`, `{"enabled":true,"startedAt":1,"endsAt":2,"reason":"test","allAlerts":true}`); err != nil {
+		t.Fatal(err)
+	}
+	if expired := store.MaintenanceWindow(); expired.Active || expired.Enabled {
+		t.Fatalf("просроченный режим остался активным: %+v", expired)
+	}
+}
+
+func TestPlannedAlertOnlyAppearsInHistory(t *testing.T) {
+	store, err := openStore(":memory:")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer store.Close()
+	alert := Alert{ID: "service-7-local", Severity: "critical", Source: "Сервисы", Title: "Недоступен"}
+	visible, err := store.SyncAlertsWithMaintenance(nil, []Alert{alert}, "Обновление")
+	if err != nil || len(visible) != 0 {
+		t.Fatalf("плановый алерт стал активным: %+v %v", visible, err)
+	}
+	history, err := store.AlertHistory(10)
+	if err != nil || len(history) != 1 || !history[0].Planned || history[0].MaintenanceReason != "Обновление" {
+		t.Fatalf("плановое событие не сохранено: %+v %v", history, err)
+	}
+	pending, err := store.PendingEmailAlerts()
+	if err != nil || len(pending) != 0 {
+		t.Fatalf("плановое событие попало в почту: %+v %v", pending, err)
+	}
+}

network.go

@@ -2,7 +2,13 @@ package main
 
 import (
 	"bufio"
+	"context"
+	"io"
+	"net"
+	"net/http"
 	"os"
+	"os/exec"
+	"path/filepath"
 	"strconv"
 	"strings"
 	"sync"
@@ -16,6 +22,20 @@ type NetworkMetrics struct {
 	ReceivePerSec  float64            `json:"receivePerSec"`
 	TransmitPerSec float64            `json:"transmitPerSec"`
 	Interfaces     []NetworkInterface `json:"interfaces"`
+	Health         NetworkHealth      `json:"health"`
+}
+
+type NetworkHealth struct {
+	Gateway               string    `json:"gateway"`
+	GatewayUp             bool      `json:"gatewayUp"`
+	DNSUp                 bool      `json:"dnsUp"`
+	InternetUp            bool      `json:"internetUp"`
+	PacketLoss            float64   `json:"packetLoss"`
+	LatencyMs             float64   `json:"latencyMs"`
+	ExternalIP            string    `json:"externalIp"`
+	ExternalIPChanged     bool      `json:"externalIpChanged"`
+	DisappearedInterfaces []string  `json:"disappearedInterfaces"`
+	CheckedAt             time.Time `json:"checkedAt"`
 }
 
 type NetworkInterface struct {
@@ -31,6 +51,12 @@ type NetworkInterface struct {
 	VMID           int     `json:"vmid,omitempty"`
 	GuestName      string  `json:"guestName,omitempty"`
 	GuestType      string  `json:"guestType,omitempty"`
+	Kind           string  `json:"kind"`
+	State          string  `json:"state"`
+	Carrier        bool    `json:"carrier"`
+	SpeedMbps      int     `json:"speedMbps,omitempty"`
+	Duplex         string  `json:"duplex,omitempty"`
+	Master         string  `json:"master,omitempty"`
 }
 
 type networkSnapshot struct {
@@ -44,6 +70,10 @@ type networkCollector struct {
 	path               string
 	previous           networkSnapshot
 	previousInterfaces map[string]NetworkInterface
+	health             NetworkHealth
+	healthUpdatedAt    time.Time
+	externalIP         string
+	knownInterfaces    map[string]bool
 }
 
 func newNetworkCollector(path string) (*networkCollector, error) {
@@ -56,7 +86,11 @@ func newNetworkCollector(path string) (*networkCollector, error) {
 	for _, v := range details {
 		previous[v.Name] = v
 	}
-	return &networkCollector{path: path, previous: snapshot, previousInterfaces: previous}, nil
+	known := map[string]bool{}
+	for _, v := range details {
+		known[v.Name] = true
+	}
+	return &networkCollector{path: path, previous: snapshot, previousInterfaces: previous, knownInterfaces: known}, nil
 }
 
 func (c *networkCollector) collect() (NetworkMetrics, error) {
@@ -75,6 +109,7 @@ func (c *networkCollector) collect() (NetworkMetrics, error) {
 	}
 	for i := range details {
 		v := &details[i]
+		decorateInterface(v)
 		if old, ok := c.previousInterfaces[v.Name]; ok && elapsed > 0 && v.ReceiveBytes >= old.ReceiveBytes && v.TransmitBytes >= old.TransmitBytes {
 			v.ReceivePerSec = float64(v.ReceiveBytes-old.ReceiveBytes) / elapsed
 			v.TransmitPerSec = float64(v.TransmitBytes-old.TransmitBytes) / elapsed
@@ -85,12 +120,100 @@ func (c *networkCollector) collect() (NetworkMetrics, error) {
 		c.previousInterfaces[v.Name] = v
 	}
 	c.previous = current
+	if time.Since(c.healthUpdatedAt) > 30*time.Second {
+		c.health = checkNetworkHealth(c.externalIP)
+		if c.health.ExternalIP != "" {
+			c.externalIP = c.health.ExternalIP
+		}
+		c.healthUpdatedAt = time.Now()
+	}
+	currentNames := map[string]bool{}
+	for _, v := range details {
+		currentNames[v.Name] = true
+	}
+	c.health.DisappearedInterfaces = nil
+	for name := range c.knownInterfaces {
+		if !currentNames[name] {
+			c.health.DisappearedInterfaces = append(c.health.DisappearedInterfaces, name)
+		}
+	}
+	c.knownInterfaces = currentNames
 	return NetworkMetrics{
 		Interface: strings.Join(interfaceNames, ", "), Interfaces: details, ReceiveBytes: current.received,
-		TransmitBytes: current.sent, ReceivePerSec: receivedRate, TransmitPerSec: sentRate,
+		TransmitBytes: current.sent, ReceivePerSec: receivedRate, TransmitPerSec: sentRate, Health: c.health,
 	}, nil
 }
 
+func decorateInterface(v *NetworkInterface) {
+	base := filepath.Join("/sys/class/net", v.Name)
+	v.State = readTrimmed(filepath.Join(base, "operstate"))
+	v.Carrier = readTrimmed(filepath.Join(base, "carrier")) == "1"
+	v.SpeedMbps, _ = strconv.Atoi(readTrimmed(filepath.Join(base, "speed")))
+	v.Duplex = readTrimmed(filepath.Join(base, "duplex"))
+	if target, err := filepath.EvalSymlinks(filepath.Join(base, "master")); err == nil {
+		v.Master = filepath.Base(target)
+	}
+	switch {
+	case strings.HasPrefix(v.Name, "vmbr"):
+		v.Kind = "Bridge"
+	case strings.HasPrefix(v.Name, "bond"):
+		v.Kind = "Bond"
+	case strings.Contains(v.Name, "."):
+		v.Kind = "VLAN"
+	case strings.HasPrefix(v.Name, "tap") || strings.HasPrefix(v.Name, "veth"):
+		v.Kind = "Guest"
+	default:
+		v.Kind = "Physical"
+	}
+}
+
+func checkNetworkHealth(previousIP string) NetworkHealth {
+	result := NetworkHealth{CheckedAt: time.Now()}
+	ctx, cancel := context.WithTimeout(context.Background(), 6*time.Second)
+	defer cancel()
+	if out, err := exec.CommandContext(ctx, "ip", "route", "show", "default").Output(); err == nil {
+		f := strings.Fields(string(out))
+		for i := range f {
+			if f[i] == "via" && i+1 < len(f) {
+				result.Gateway = f[i+1]
+				break
+			}
+		}
+	}
+	if result.Gateway != "" {
+		out, _ := exec.CommandContext(ctx, "ping", "-c", "3", "-W", "1", result.Gateway).CombinedOutput()
+		text := string(out)
+		result.GatewayUp = strings.Contains(text, "0% packet loss")
+		if i := strings.Index(text, "packet loss"); i > 0 {
+			start := strings.LastIndex(text[:i], " ")
+			if start >= 0 {
+				result.PacketLoss, _ = strconv.ParseFloat(strings.TrimSuffix(strings.TrimSpace(text[start:i]), "%"), 64)
+			}
+		}
+		if i := strings.Index(text, "min/avg/max"); i >= 0 {
+			if eq := strings.Index(text[i:], "="); eq >= 0 {
+				parts := strings.Split(strings.Fields(text[i+eq+1:])[0], "/")
+				if len(parts) > 1 {
+					result.LatencyMs, _ = strconv.ParseFloat(parts[1], 64)
+				}
+			}
+		}
+	}
+	resolver := net.Resolver{}
+	_, err := resolver.LookupHost(ctx, "deb.debian.org")
+	result.DNSUp = err == nil
+	client := http.Client{Timeout: 4 * time.Second}
+	request, _ := http.NewRequestWithContext(ctx, http.MethodGet, "https://api.ipify.org", nil)
+	if response, err := client.Do(request); err == nil {
+		defer response.Body.Close()
+		body, _ := io.ReadAll(io.LimitReader(response.Body, 128))
+		result.ExternalIP = strings.TrimSpace(string(body))
+		result.InternetUp = response.StatusCode == 200 && net.ParseIP(result.ExternalIP) != nil
+	}
+	result.ExternalIPChanged = previousIP != "" && result.ExternalIP != "" && previousIP != result.ExternalIP
+	return result
+}
+
 func readNetworkSnapshot(path string) (networkSnapshot, []string, []NetworkInterface, error) {
 	file, err := os.Open(path)
 	if err != nil {

proxmox-cpu-dashboard.service

@@ -4,7 +4,7 @@ After=network-online.target
 Wants=network-online.target
 
 [Service]
-Type=simple
+Type=notify
 ExecStart=/usr/local/bin/proxmox-cpu-dashboard
 Environment=DASHBOARD_ADDR=:9105
 Environment=DASHBOARD_DB_PATH=/var/lib/proxmox-dashboard/dashboard.db
@@ -12,6 +12,8 @@ StateDirectory=proxmox-dashboard
 StateDirectoryMode=0750
 Restart=on-failure
 RestartSec=3
+WatchdogSec=30
+TimeoutStartSec=20
 NoNewPrivileges=true
 ProtectHome=true
 PrivateTmp=true

reliability.go

@@ -0,0 +1,244 @@
+package main
+
+import (
+	"context"
+	"fmt"
+	"log"
+	"net"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"sort"
+	"strings"
+	"sync"
+	"syscall"
+	"time"
+)
+
+func notifySystemd(message string) bool {
+	address := os.Getenv("NOTIFY_SOCKET")
+	if address == "" {
+		return false
+	}
+	if strings.HasPrefix(address, "@") {
+		address = "\x00" + address[1:]
+	}
+	connection, err := net.DialUnix("unixgram", nil, &net.UnixAddr{Name: address, Net: "unixgram"})
+	if err != nil {
+		return false
+	}
+	defer connection.Close()
+	_, err = connection.Write([]byte(message))
+	return err == nil
+}
+func runSystemdWatchdog() {
+	if !notifySystemd("READY=1\nSTATUS=Dashboard collects Proxmox metrics") {
+		return
+	}
+	go func() {
+		ticker := time.NewTicker(10 * time.Second)
+		defer ticker.Stop()
+		for range ticker.C {
+			notifySystemd("WATCHDOG=1")
+		}
+	}()
+}
+
+type MonitorHealth struct {
+	LastSuccessfulCollection time.Time `json:"lastSuccessfulCollection"`
+	LastCollectionError      string    `json:"lastCollectionError"`
+	CollectionDurationMs     int64     `json:"collectionDurationMs"`
+	SlowCollector            bool      `json:"slowCollector"`
+	LastHistoryWrite         time.Time `json:"lastHistoryWrite"`
+	HistoryWriteError        string    `json:"historyWriteError"`
+	DatabaseFreeBytes        uint64    `json:"databaseFreeBytes"`
+	DatabasePath             string    `json:"databasePath"`
+	WatchdogEnabled          bool      `json:"watchdogEnabled"`
+}
+
+type monitorTracker struct {
+	mu          sync.RWMutex
+	lastSuccess time.Time
+	lastError   string
+	duration    time.Duration
+}
+
+func runCollectionWatchdog(monitor *monitorTracker) {
+	started := time.Now()
+	ticker := time.NewTicker(15 * time.Second)
+	defer ticker.Stop()
+	for range ticker.C {
+		monitor.mu.RLock()
+		last := monitor.lastSuccess
+		monitor.mu.RUnlock()
+		if time.Since(started) < 2*time.Minute {
+			continue
+		}
+		if last.IsZero() || time.Since(last) > 90*time.Second {
+			log.Printf("Встроенный watchdog: сбор метрик не завершался более 90 секунд; перезапускаем процесс")
+			os.Exit(1)
+		}
+	}
+}
+
+func (m *monitorTracker) record(start time.Time, err error) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	m.duration = time.Since(start)
+	if err != nil {
+		m.lastError = err.Error()
+		return
+	}
+	m.lastSuccess = time.Now()
+	m.lastError = ""
+}
+
+func (m *monitorTracker) snapshot(store *Store) MonitorHealth {
+	m.mu.RLock()
+	result := MonitorHealth{LastSuccessfulCollection: m.lastSuccess, LastCollectionError: m.lastError, CollectionDurationMs: m.duration.Milliseconds(), SlowCollector: m.duration > 10*time.Second, DatabasePath: store.path, WatchdogEnabled: os.Getenv("NOTIFY_SOCKET") != ""}
+	m.mu.RUnlock()
+	result.LastHistoryWrite, result.HistoryWriteError = store.historyHealth()
+	path := store.path
+	if path == ":memory:" {
+		path = "/tmp"
+	} else {
+		path = filepath.Dir(path)
+	}
+	var stat syscall.Statfs_t
+	if syscall.Statfs(path, &stat) == nil {
+		result.DatabaseFreeBytes = stat.Bavail * uint64(stat.Bsize)
+	}
+	return result
+}
+
+type UPSShutdownSettings struct {
+	Enabled        bool    `json:"enabled"`
+	ChargePercent  float64 `json:"chargePercent"`
+	RuntimeSeconds float64 `json:"runtimeSeconds"`
+	GraceSeconds   int     `json:"graceSeconds"`
+	GuestOrder     []int   `json:"guestOrder"`
+}
+
+func defaultUPSShutdownSettings() UPSShutdownSettings {
+	return UPSShutdownSettings{ChargePercent: 15, RuntimeSeconds: 600, GraceSeconds: 120}
+}
+func (s *Store) UPSShutdownSettings() UPSShutdownSettings {
+	value := defaultUPSShutdownSettings()
+	s.loadJSONSetting("ups_shutdown", &value)
+	return value
+}
+func (s *Store) SaveUPSShutdownSettings(value UPSShutdownSettings) error {
+	if value.ChargePercent < 1 || value.ChargePercent > 80 || value.RuntimeSeconds < 60 || value.RuntimeSeconds > 7200 || value.GraceSeconds < 30 || value.GraceSeconds > 1800 {
+		return fmt.Errorf("проверьте пороги UPS и задержку")
+	}
+	return s.saveJSONSetting("ups_shutdown", value)
+}
+
+type ShutdownStep struct {
+	Order  int    `json:"order"`
+	Target string `json:"target"`
+	Action string `json:"action"`
+	Detail string `json:"detail"`
+}
+
+func buildShutdownDryRun(settings UPSShutdownSettings, guests GuestsMetrics) []ShutdownStep {
+	byID := map[int]GuestMetrics{}
+	for _, g := range guests.Guests {
+		if g.Template == 0 && g.Status == "running" {
+			byID[g.VMID] = g
+		}
+	}
+	var ordered []GuestMetrics
+	for _, id := range settings.GuestOrder {
+		if g, ok := byID[id]; ok {
+			ordered = append(ordered, g)
+			delete(byID, id)
+		}
+	}
+	var rest []GuestMetrics
+	for _, g := range byID {
+		rest = append(rest, g)
+	}
+	sort.Slice(rest, func(i, j int) bool { return rest[i].VMID < rest[j].VMID })
+	ordered = append(ordered, rest...)
+	steps := []ShutdownStep{{Order: 1, Target: "UPS", Action: "Ожидание порога", Detail: fmt.Sprintf("заряд ≤ %.0f%% или автономность ≤ %.0f сек.", settings.ChargePercent, settings.RuntimeSeconds)}}
+	for _, g := range ordered {
+		kind := "LXC"
+		if g.Type == "qemu" {
+			kind = "VM"
+		}
+		steps = append(steps, ShutdownStep{Order: len(steps) + 1, Target: fmt.Sprintf("%s %d · %s", kind, g.VMID, g.Name), Action: "Корректное завершение", Detail: fmt.Sprintf("ожидание до %d сек.", settings.GraceSeconds)})
+	}
+	steps = append(steps, ShutdownStep{Order: len(steps) + 1, Target: "Proxmox host", Action: "Завершение работы", Detail: "только после остановки VM/LXC"})
+	return steps
+}
+
+func runUPSShutdownController(store *Store, ups *upsCollector, guests *guestCollector) {
+	ticker := time.NewTicker(10 * time.Second)
+	defer ticker.Stop()
+	triggered := false
+	for range ticker.C {
+		settings := store.UPSShutdownSettings()
+		metrics := ups.collect()
+		onBattery := strings.Contains(metrics.Status, "OB") || metrics.Status == "ONBATT"
+		if !settings.Enabled || !metrics.Available || !onBattery {
+			if metrics.Available && !onBattery {
+				triggered = false
+			}
+			continue
+		}
+		lowCharge := metrics.ChargePercent != nil && *metrics.ChargePercent <= settings.ChargePercent
+		lowRuntime := metrics.RuntimeSeconds != nil && *metrics.RuntimeSeconds <= settings.RuntimeSeconds
+		if triggered || (!lowCharge && !lowRuntime) {
+			continue
+		}
+		triggered = true
+		log.Printf("UPS достиг порога безопасного выключения; запускаем остановку гостей")
+		go executeUPSShutdown(settings, guests.collect())
+	}
+}
+
+func executeUPSShutdown(settings UPSShutdownSettings, guests GuestsMetrics) {
+	steps := buildShutdownDryRun(settings, guests)
+	_ = steps
+	byID := map[int]GuestMetrics{}
+	for _, g := range guests.Guests {
+		if g.Template == 0 && g.Status == "running" {
+			byID[g.VMID] = g
+		}
+	}
+	var order []int
+	for _, id := range settings.GuestOrder {
+		if _, ok := byID[id]; ok {
+			order = append(order, id)
+			delete(byID, id)
+		}
+	}
+	var remaining []int
+	for id := range byID {
+		remaining = append(remaining, id)
+	}
+	sort.Ints(remaining)
+	order = append(order, remaining...)
+	for _, id := range order {
+		g := GuestMetrics{}
+		for _, candidate := range guests.Guests {
+			if candidate.VMID == id {
+				g = candidate
+				break
+			}
+		}
+		command := "pct"
+		if g.Type == "qemu" {
+			command = "qm"
+		}
+		ctx, cancel := context.WithTimeout(context.Background(), time.Duration(settings.GraceSeconds+30)*time.Second)
+		err := exec.CommandContext(ctx, command, "shutdown", fmt.Sprint(id), "--timeout", fmt.Sprint(settings.GraceSeconds)).Run()
+		cancel()
+		if err != nil {
+			log.Printf("UPS shutdown: не удалось остановить VMID %d: %v", id, err)
+		}
+	}
+	log.Printf("UPS shutdown: гости обработаны, выключаем Proxmox host")
+	_ = exec.Command("shutdown", "-h", "now").Run()
+}

reliability_test.go

@@ -0,0 +1,38 @@
+package main
+
+import (
+	"strings"
+	"testing"
+	"time"
+)
+
+func TestDiskCounterChanges(t *testing.T) {
+	old := DiskMetrics{Reallocated: 1, Pending: 2, CRCErrors: 4}
+	current := DiskMetrics{Reallocated: 2, Pending: 2, CRCErrors: 7}
+	changes := strings.Join(diskCounterChanges(old, current), " ")
+	if !strings.Contains(changes, "Reallocated") || !strings.Contains(changes, "CRC") || strings.Contains(changes, "Pending") {
+		t.Fatalf("unexpected counter changes: %s", changes)
+	}
+}
+
+func TestParseScrub(t *testing.T) {
+	status, at, duration, old := parseScrub("scrub repaired 0B in 01:12:03 with 0 errors on Fri Jan  2 03:04:05 2026")
+	if status != "Завершён" || at == "" || duration != "01:12:03" || !old {
+		t.Fatalf("unexpected scrub: %q %q %q %v", status, at, duration, old)
+	}
+}
+
+func TestShutdownDryRunUsesConfiguredOrder(t *testing.T) {
+	settings := UPSShutdownSettings{ChargePercent: 15, RuntimeSeconds: 600, GraceSeconds: 120, GuestOrder: []int{105}}
+	guests := GuestsMetrics{Guests: []GuestMetrics{{VMID: 101, Name: "one", Type: "qemu", Status: "running"}, {VMID: 105, Name: "five", Type: "lxc", Status: "running"}}}
+	steps := buildShutdownDryRun(settings, guests)
+	if len(steps) != 4 || !strings.Contains(steps[1].Target, "105") || !strings.Contains(steps[2].Target, "101") || steps[3].Target != "Proxmox host" {
+		t.Fatalf("unexpected plan: %+v", steps)
+	}
+}
+
+func TestSelfTestOlderThanWeek(t *testing.T) {
+	if !selfTestOlderThanWeek(time.Now().Add(-8*24*time.Hour).Format(time.RFC3339)) || selfTestOlderThanWeek(time.Now().Format(time.RFC3339)) {
+		t.Fatal("self-test age evaluation is incorrect")
+	}
+}

scripts/install-agent.sh

@@ -0,0 +1,65 @@
+#!/bin/sh
+set -eu
+
+if [ "$(id -u)" -ne 0 ]; then
+  echo "Запустите команду установки от root." >&2
+  exit 1
+fi
+if [ -z "${DASHBOARD_URL:-}" ] || [ -z "${ENROLL_TOKEN:-}" ]; then
+  echo "Не заданы DASHBOARD_URL и ENROLL_TOKEN. Создайте команду в настройках Dashboard." >&2
+  exit 1
+fi
+
+GITEA_URL=${GITEA_URL:-https://git.myown.center}
+GITEA_REPOSITORY=${GITEA_REPOSITORY:-maxim/ProxmoxDash}
+VMID=${VMID:-0}
+AGENT_NAME=${AGENT_NAME:-$(hostname)}
+
+release_json=$(curl -fsSL "$GITEA_URL/api/v1/repos/$GITEA_REPOSITORY/releases/latest")
+download_url=$(printf '%s' "$release_json" | sed -n 's/.*"browser_download_url":"\([^"]*\/proxmox-dashboard-linux-amd64\)".*/\1/p' | head -n 1)
+if [ -z "$download_url" ]; then
+  echo "В последнем релизе не найден Linux-бинарник." >&2
+  exit 1
+fi
+
+install -d -m 0700 /var/lib/proxmox-dashboard-agent
+temporary_binary=$(mktemp /tmp/proxmox-dashboard-agent.XXXXXX)
+trap 'rm -f "$temporary_binary"' EXIT
+curl -fsSL "$download_url" -o "$temporary_binary"
+install -m 0755 "$temporary_binary" /var/lib/proxmox-dashboard-agent/proxmox-dashboard-agent
+
+umask 077
+{
+  printf 'DASHBOARD_URL=%s\n' "$DASHBOARD_URL"
+  printf 'ENROLL_TOKEN=%s\n' "$ENROLL_TOKEN"
+  printf 'AGENT_NAME=%s\n' "$AGENT_NAME"
+  printf 'VMID=%s\n' "$VMID"
+} > /etc/proxmox-dashboard-agent.env
+
+cat > /etc/systemd/system/proxmox-dashboard-agent.service <<'UNIT'
+[Unit]
+Description=Proxmox Dashboard read-only guest agent
+After=network-online.target docker.service
+Wants=network-online.target
+
+[Service]
+Type=simple
+EnvironmentFile=/etc/proxmox-dashboard-agent.env
+ExecStart=/var/lib/proxmox-dashboard-agent/proxmox-dashboard-agent --agent --agent-server ${DASHBOARD_URL} --agent-enroll-token ${ENROLL_TOKEN} --agent-name ${AGENT_NAME} --agent-vmid ${VMID}
+Restart=always
+RestartSec=5
+NoNewPrivileges=true
+ProtectHome=true
+PrivateTmp=true
+ProtectSystem=strict
+ReadWritePaths=/var/lib/proxmox-dashboard-agent
+
+[Install]
+WantedBy=multi-user.target
+UNIT
+
+systemctl daemon-reload
+systemctl enable proxmox-dashboard-agent.service
+systemctl restart proxmox-dashboard-agent.service
+rm -f /usr/local/bin/proxmox-dashboard-agent
+echo "Агент установлен. Через несколько секунд он появится в Dashboard."

scripts/uninstall-agent.sh

@@ -0,0 +1,16 @@
+#!/bin/sh
+set -eu
+
+if [ "$(id -u)" -ne 0 ]; then
+  echo "Запустите удаление от root." >&2
+  exit 1
+fi
+
+systemctl disable --now proxmox-dashboard-agent.service 2>/dev/null || true
+rm -f /etc/systemd/system/proxmox-dashboard-agent.service
+rm -f /etc/proxmox-dashboard-agent.env
+rm -f /usr/local/bin/proxmox-dashboard-agent
+rm -rf /var/lib/proxmox-dashboard-agent
+systemctl daemon-reload
+systemctl reset-failed proxmox-dashboard-agent.service 2>/dev/null || true
+echo "Агент и его локальные учётные данные удалены."

services.go

@@ -74,6 +74,20 @@ type ServiceHistoryPoint struct {
 	LatencyMS float64 `json:"latencyMs"`
 }
 
+var serviceHTTPTransport = &http.Transport{
+	Proxy:                 http.ProxyFromEnvironment,
+	DialContext:           (&net.Dialer{Timeout: 10 * time.Second, KeepAlive: 30 * time.Second}).DialContext,
+	ForceAttemptHTTP2:     true,
+	MaxIdleConns:          32,
+	MaxIdleConnsPerHost:   2,
+	IdleConnTimeout:       30 * time.Second,
+	TLSHandshakeTimeout:   10 * time.Second,
+	ExpectContinueTimeout: time.Second,
+	TLSClientConfig:       &tls.Config{InsecureSkipVerify: true, MinVersion: tls.VersionTLS12},
+}
+
+var serviceHTTPClient = &http.Client{Transport: serviceHTTPTransport}
+
 func (s *Store) Services() ([]MonitoredService, error) {
 	rows, err := s.db.Query(`SELECT id,name,url,local_address,check_type,expected_code,expected_text,timeout_seconds,enabled FROM services ORDER BY id`)
 	if err != nil {
@@ -296,14 +310,12 @@ func checkServiceEndpoint(v MonitoredService, kind, label, target string) Servic
 			st.Error = strings.TrimSpace(string(out))
 		}
 	default:
-		tr := &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true, MinVersion: tls.VersionTLS12}}
-		client := &http.Client{Transport: tr, Timeout: time.Duration(v.TimeoutSeconds) * time.Second}
 		req, err := http.NewRequestWithContext(ctx, http.MethodGet, target, nil)
 		if err != nil {
 			st.Error = err.Error()
 			break
 		}
-		resp, err := client.Do(req)
+		resp, err := serviceHTTPClient.Do(req)
 		if err != nil {
 			st.Error = err.Error()
 		} else {

services_test.go

@@ -1,6 +1,9 @@
 package main
 
-import "testing"
+import (
+	"testing"
+	"time"
+)
 
 func TestServiceCRUD(t *testing.T) {
 	store, err := openStore(":memory:")
@@ -38,3 +41,15 @@ func TestValidateServiceChecksBothHTTPAddresses(t *testing.T) {
 		t.Fatal(err)
 	}
 }
+
+func TestHTTPServiceTransportHasBoundedIdlePool(t *testing.T) {
+	if serviceHTTPClient.Transport != serviceHTTPTransport {
+		t.Fatal("проверки сервисов должны использовать общий HTTP transport")
+	}
+	if serviceHTTPTransport.IdleConnTimeout < time.Second {
+		t.Fatal("idle-соединения должны автоматически закрываться")
+	}
+	if serviceHTTPTransport.MaxIdleConnsPerHost < 1 || serviceHTTPTransport.MaxIdleConnsPerHost > 4 {
+		t.Fatalf("небезопасный размер пула на хост: %d", serviceHTTPTransport.MaxIdleConnsPerHost)
+	}
+}

store.go

@@ -7,6 +7,7 @@ import (
 	"os"
 	"path/filepath"
 	"strings"
+	"sync"
 	"time"
 
 	_ "modernc.org/sqlite"
@@ -30,6 +31,17 @@ type AlertThresholds struct {
 	NVMeTempWarning   float64 `json:"nvmeTempWarning"`
 	UPSChargeCritical float64 `json:"upsChargeCritical"`
 	BackupMaxAgeHours float64 `json:"backupMaxAgeHours"`
+	DiskHDDUtil       float64 `json:"diskHddUtil"`
+	DiskHDDLatency    float64 `json:"diskHddLatency"`
+	DiskHDDQueue      float64 `json:"diskHddQueue"`
+	DiskSSDUtil       float64 `json:"diskSsdUtil"`
+	DiskSSDLatency    float64 `json:"diskSsdLatency"`
+	DiskSSDQueue      float64 `json:"diskSsdQueue"`
+	DiskNVMeUtil      float64 `json:"diskNvmeUtil"`
+	DiskNVMeLatency   float64 `json:"diskNvmeLatency"`
+	DiskNVMeQueue     float64 `json:"diskNvmeQueue"`
+	DiskIODuration    float64 `json:"diskIoDuration"`
+	DiskIOExcluded    string  `json:"diskIoExcluded"`
 }
 
 func defaultThresholds() AlertThresholds {
@@ -39,6 +51,7 @@ func defaultThresholds() AlertThresholds {
 		StorageWarning: 85, StorageCritical: 95, ZFSWarning: 80, ZFSCritical: 90,
 		DiskTempWarning: 60, NVMeTempWarning: 75, UPSChargeCritical: 30,
 		BackupMaxAgeHours: 48,
+		DiskHDDUtil:       90, DiskHDDLatency: 50, DiskHDDQueue: 4, DiskSSDUtil: 90, DiskSSDLatency: 20, DiskSSDQueue: 4, DiskNVMeUtil: 95, DiskNVMeLatency: 10, DiskNVMeQueue: 8, DiskIODuration: 60,
 	}
 }
 
@@ -58,7 +71,13 @@ type HistoryPoint struct {
 	GuestsMemory    float64  `json:"guestsMemory"`
 }
 
-type Store struct{ db *sql.DB }
+type Store struct {
+	db               *sql.DB
+	path             string
+	healthMu         sync.RWMutex
+	lastHistoryAt    time.Time
+	lastHistoryError string
+}
 
 type EmailSettings struct {
 	Enabled         bool     `json:"enabled"`
@@ -119,6 +138,8 @@ func openStore(path string) (*Store, error) {
 		`CREATE INDEX IF NOT EXISTS service_checks_service_ts_idx ON service_checks(service_id,ts)`,
 		`CREATE TABLE IF NOT EXISTS entity_history (kind TEXT NOT NULL,name TEXT NOT NULL,ts INTEGER NOT NULL,value1 REAL,value2 REAL,PRIMARY KEY(kind,name,ts))`,
 		`CREATE INDEX IF NOT EXISTS entity_history_kind_ts_idx ON entity_history(kind,ts)`,
+		`CREATE TABLE IF NOT EXISTS agent_enrollment_tokens (token_hash TEXT PRIMARY KEY, created_at INTEGER NOT NULL, expires_at INTEGER NOT NULL, used_at INTEGER)`,
+		`CREATE TABLE IF NOT EXISTS agents (id TEXT PRIMARY KEY, name TEXT NOT NULL, vmid INTEGER NOT NULL DEFAULT 0, hostname TEXT NOT NULL DEFAULT '', machine_id TEXT NOT NULL DEFAULT '', secret_hash TEXT NOT NULL, enrolled_at INTEGER NOT NULL, last_seen INTEGER NOT NULL, version TEXT NOT NULL DEFAULT '', report_json TEXT NOT NULL DEFAULT '{}', remote_address TEXT NOT NULL DEFAULT '')`,
 	}
 	for _, statement := range statements {
 		if _, err := db.Exec(statement); err != nil {
@@ -128,9 +149,12 @@ func openStore(path string) (*Store, error) {
 	}
 	// Older databases get this column without requiring a separate migration tool.
 	_, _ = db.Exec(`ALTER TABLE alert_events ADD COLUMN recovery_sent_at INTEGER`)
+	_, _ = db.Exec(`ALTER TABLE alert_events ADD COLUMN planned INTEGER NOT NULL DEFAULT 0`)
+	_, _ = db.Exec(`ALTER TABLE alert_events ADD COLUMN maintenance_reason TEXT NOT NULL DEFAULT ''`)
+	_, _ = db.Exec(`DELETE FROM alert_events WHERE alert_key='monitor-slow'`)
 	_, _ = db.Exec(`ALTER TABLE service_checks ADD COLUMN endpoint TEXT NOT NULL DEFAULT 'public'`)
 	_, _ = db.Exec(`PRAGMA foreign_keys=ON`)
-	return &Store{db: db}, nil
+	return &Store{db: db, path: path}, nil
 }
 
 type EntityHistoryPoint struct {
@@ -211,7 +235,7 @@ func (s *Store) SaveThresholds(value AlertThresholds) error {
 }
 
 func defaultEmailSettings() EmailSettings {
-	return EmailSettings{Port: 587, IntervalMinutes: 60, Severities: []string{"critical", "warning"}, Sources: []string{"CPU", "Память", "Storage", "Диски", "ZFS", "UPS", "VM/LXC", "Proxmox", "Dashboard", "Сервисы", "Система", "Сеть"}}
+	return EmailSettings{Port: 587, IntervalMinutes: 60, Severities: []string{"critical", "warning"}, Sources: []string{"CPU", "Память", "Storage", "Диски", "ZFS", "UPS", "VM/LXC", "Proxmox", "Dashboard", "Сервисы", "Система", "Сеть", "Агенты", "Docker"}}
 }
 
 func (s *Store) EmailSettings(includePassword bool) EmailSettings {
@@ -278,6 +302,19 @@ func validateThresholds(t AlertThresholds) error {
 	if t.BackupMaxAgeHours < 1 || t.BackupMaxAgeHours > 24*365 {
 		return fmt.Errorf("возраст backup должен быть от 1 до 8760 часов")
 	}
+	for _, value := range []float64{t.DiskHDDUtil, t.DiskSSDUtil, t.DiskNVMeUtil} {
+		if value < 1 || value > 100 {
+			return fmt.Errorf("пороги utilization должны быть от 1 до 100")
+		}
+	}
+	for _, value := range []float64{t.DiskHDDLatency, t.DiskSSDLatency, t.DiskNVMeLatency, t.DiskHDDQueue, t.DiskSSDQueue, t.DiskNVMeQueue} {
+		if value < 0 {
+			return fmt.Errorf("пороги latency и очереди не могут быть отрицательными")
+		}
+	}
+	if t.DiskIODuration < 5 || t.DiskIODuration > 3600 {
+		return fmt.Errorf("длительность I/O-проблемы должна быть от 5 до 3600 секунд")
+	}
 	return nil
 }
 
@@ -285,9 +322,45 @@ func (s *Store) AddHistory(point HistoryPoint) error {
 	_, err := s.db.Exec(`INSERT OR REPLACE INTO history(ts,cpu_usage,cpu_temp,memory_usage,swap_usage,root_usage,network_rx,network_tx,ups_charge,ups_load,zfs_usage,guests_cpu,guests_memory) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?)`,
 		point.Timestamp, point.CPUUsage, point.CPUTemperature, point.MemoryUsage, point.SwapUsage, point.RootUsage, point.NetworkReceive, point.NetworkTransmit, point.UPSCharge, point.UPSLoad, point.ZFSUsage, point.GuestsCPU, point.GuestsMemory)
 	if err != nil {
+		s.setHistoryHealth(time.Time{}, err)
 		return err
 	}
 	_, err = s.db.Exec(`DELETE FROM history WHERE ts < ?`, time.Now().Add(-historyRetention).Unix())
+	s.setHistoryHealth(time.Now(), err)
+	return err
+}
+
+func (s *Store) setHistoryHealth(at time.Time, err error) {
+	s.healthMu.Lock()
+	defer s.healthMu.Unlock()
+	if !at.IsZero() {
+		s.lastHistoryAt = at
+	}
+	if err != nil {
+		s.lastHistoryError = err.Error()
+	} else {
+		s.lastHistoryError = ""
+	}
+}
+func (s *Store) historyHealth() (time.Time, string) {
+	s.healthMu.RLock()
+	defer s.healthMu.RUnlock()
+	return s.lastHistoryAt, s.lastHistoryError
+}
+
+func (s *Store) loadJSONSetting(key string, target any) bool {
+	var value string
+	if s.db.QueryRow(`SELECT value FROM settings WHERE key=?`, key).Scan(&value) != nil {
+		return false
+	}
+	return json.Unmarshal([]byte(value), target) == nil
+}
+func (s *Store) saveJSONSetting(key string, value any) error {
+	data, err := json.Marshal(value)
+	if err != nil {
+		return err
+	}
+	_, err = s.db.Exec(`INSERT INTO settings(key,value) VALUES(?,?) ON CONFLICT(key) DO UPDATE SET value=excluded.value`, key, string(data))
 	return err
 }
 

trends.go

@@ -0,0 +1,107 @@
+package main
+
+import (
+	"database/sql"
+	"fmt"
+	"sync"
+	"time"
+)
+
+type entityAverage struct {
+	Value1, Value2 float64
+	Count          int
+}
+type TrendAnomaly struct {
+	ID       string `json:"id"`
+	Severity string `json:"severity"`
+	Source   string `json:"source"`
+	Title    string `json:"title"`
+	Message  string `json:"message"`
+}
+type trendCollector struct {
+	mu        sync.Mutex
+	updatedAt time.Time
+	cached    []TrendAnomaly
+}
+
+func (s *Store) entityAverages(kind string) map[string]entityAverage {
+	return s.entityAveragesSince(kind, 7*24*time.Hour)
+}
+func (s *Store) entityAveragesSince(kind string, duration time.Duration) map[string]entityAverage {
+	rows, err := s.db.Query(`SELECT name,AVG(value1),AVG(value2),COUNT(*) FROM entity_history WHERE kind=? AND ts>=? GROUP BY name`, kind, time.Now().Add(-duration).Unix())
+	if err != nil {
+		return nil
+	}
+	defer rows.Close()
+	result := map[string]entityAverage{}
+	for rows.Next() {
+		var name string
+		var v1, v2 sql.NullFloat64
+		var count int
+		if rows.Scan(&name, &v1, &v2, &count) == nil {
+			result[name] = entityAverage{Value1: v1.Float64, Value2: v2.Float64, Count: count}
+		}
+	}
+	return result
+}
+
+func (c *trendCollector) collect(metrics dashboardMetrics, store *Store) []TrendAnomaly {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	if time.Since(c.updatedAt) < time.Minute {
+		return c.cached
+	}
+	var out []TrendAnomaly
+	iops := store.entityAverages("disk-io")
+	latency := store.entityAverages("disk-latency")
+	latencyDay := store.entityAveragesSince("disk-latency", 24*time.Hour)
+	temps := store.entityAverages("disk")
+	for _, d := range metrics.Disks {
+		name := d.Name + " · " + d.Model
+		currentIOPS := d.ReadIOPS + d.WriteIOPS
+		if base := iops[name]; base.Count >= 30 && base.Value1 >= 1 && currentIOPS >= base.Value1*5 {
+			out = append(out, TrendAnomaly{"disk-iops-" + d.Name, "warning", "Аномалии", d.Model + ": IOPS резко выше обычного", fmt.Sprintf("Сейчас %.0f IOPS, недельный baseline %.0f — рост в %.1f раза.", currentIOPS, base.Value1, currentIOPS/base.Value1)})
+		}
+		if base := latency[name]; base.Count >= 30 && base.Value1 > 0 && d.LatencyMs >= base.Value1*2 && d.LatencyMs-base.Value1 >= 10 {
+			out = append(out, TrendAnomaly{"disk-latency-" + d.Name, "warning", "Аномалии", d.Model + ": latency ухудшилась", fmt.Sprintf("Сейчас %.1f ms при baseline %.1f ms.", d.LatencyMs, base.Value1)})
+		}
+		if week, day := latency[name], latencyDay[name]; week.Count >= 100 && day.Count >= 30 && week.Value1 > 0 && day.Value1 >= week.Value1*1.5 && day.Value1-week.Value1 >= 5 {
+			out = append(out, TrendAnomaly{"disk-latency-degrading-" + d.Name, "warning", "Аномалии", d.Model + ": latency постепенно ухудшается", fmt.Sprintf("Среднее за сутки %.1f ms против недельного %.1f ms.", day.Value1, week.Value1)})
+		}
+		if d.Temperature != nil {
+			if base := temps[name]; base.Count >= 30 && *d.Temperature >= base.Value1+10 {
+				out = append(out, TrendAnomaly{"disk-temp-trend-" + d.Name, "warning", "Аномалии", d.Model + ": температура выше обычной", fmt.Sprintf("Сейчас %.1f °C, baseline %.1f °C.", *d.Temperature, base.Value1)})
+			}
+		}
+	}
+	guestAvg := store.entityAverages("guest")
+	for _, g := range metrics.Guests.Guests {
+		if g.MaxMemory == 0 {
+			continue
+		}
+		name := fmt.Sprintf("%d · %s", g.VMID, g.Name)
+		current := float64(g.MemoryBytes) / float64(g.MaxMemory) * 100
+		if base := guestAvg[name]; base.Count >= 30 && current >= base.Value2*1.5 && current-base.Value2 >= 15 {
+			out = append(out, TrendAnomaly{fmt.Sprintf("guest-memory-%d", g.VMID), "warning", "Аномалии", g.Name + ": память выше обычной", fmt.Sprintf("Сейчас %.1f%%, baseline %.1f%%.", current, base.Value2)})
+		}
+	}
+	zfsAverage := store.entityAverages("zfs")
+	for _, pool := range metrics.ZFS.Pools {
+		if base := zfsAverage[pool.Name]; base.Count >= 30 && pool.CapacityPercent-base.Value1 >= 5 {
+			out = append(out, TrendAnomaly{"zfs-growth-" + pool.Name, "warning", "Аномалии", pool.Name + ": заполнение выше недельного уровня", fmt.Sprintf("Сейчас %.1f%%, недельный baseline %.1f%%.", pool.CapacityPercent, base.Value1)})
+		}
+	}
+	backupAverage := store.entityAverages("backup-duration")
+	for _, task := range metrics.Activity.Tasks {
+		if task.Type != "vzdump" || task.Duration <= 0 {
+			continue
+		}
+		name := "VMID " + task.ID
+		if base := backupAverage[name]; base.Count >= 3 && float64(task.Duration) >= base.Value1*1.5 && float64(task.Duration)-base.Value1 >= 60 {
+			out = append(out, TrendAnomaly{"backup-duration-" + task.ID, "warning", "Аномалии", "Backup " + name + " выполнялся дольше обычного", fmt.Sprintf("Последняя длительность %s, baseline %s.", time.Duration(task.Duration)*time.Second, time.Duration(base.Value1)*time.Second)})
+		}
+	}
+	c.cached = out
+	c.updatedAt = time.Now()
+	return out
+}

ups.go

@@ -24,6 +24,12 @@ type UPSMetrics struct {
 	InputFrequency    *float64 `json:"inputFrequency"`
 	BatteryVoltage    *float64 `json:"batteryVoltage"`
 	Temperature       *float64 `json:"temperatureCelsius"`
+	CommunicationOK   bool     `json:"communicationOk"`
+	BatteryDate       string   `json:"batteryDate"`
+	BatteryAgeDays    int      `json:"batteryAgeDays,omitempty"`
+	ReplaceBattery    bool     `json:"replaceBattery"`
+	LowBatteryCharge  *float64 `json:"lowBatteryCharge,omitempty"`
+	LowBatteryRuntime *float64 `json:"lowBatteryRuntime,omitempty"`
 }
 
 type upsCollector struct {
@@ -42,9 +48,15 @@ func (c *upsCollector) collect() UPSMetrics {
 		c.cached = metrics
 	} else if metrics, ok := readAPCUPSD(); ok {
 		c.cached = metrics
+	} else {
+		if c.cached.Available || c.cached.Source != "" {
+			c.cached.Available = false
+			c.cached.CommunicationOK = false
+			c.cached.StatusLabel = "Нет связи"
 		} else {
 			c.cached = UPSMetrics{}
 		}
+	}
 	c.updatedAt = time.Now()
 	return c.cached
 }
@@ -85,6 +97,7 @@ func readAPCUPSD() (UPSMetrics, bool) {
 		"ups.load": raw["LOADPCT"], "input.voltage": raw["LINEV"],
 		"output.voltage": raw["OUTPUTV"], "input.frequency": raw["LINEFREQ"],
 		"battery.voltage": raw["BATTV"], "ups.temperature": raw["ITEMP"],
+		"battery.date": raw["BATTDATE"], "battery.charge.low": raw["MBATTCHG"], "battery.runtime.low": minutesToSeconds(raw["MINTIMEL"]),
 	}
 	return upsFromValues("apcupsd", raw["UPSNAME"], values), true
 }
@@ -102,14 +115,24 @@ func parseColonValues(output string) map[string]string {
 
 func upsFromValues(source, name string, values map[string]string) UPSMetrics {
 	status := values["ups.status"]
-	return UPSMetrics{
-		Available: true, Source: source, Name: name, Model: values["device.model"],
+	metrics := UPSMetrics{
+		Available: true, CommunicationOK: true, Source: source, Name: name, Model: values["device.model"],
 		Status: status, StatusLabel: upsStatusLabel(status),
 		ChargePercent: numberPointer(values["battery.charge"]), RuntimeSeconds: numberPointer(values["battery.runtime"]),
 		LoadPercent: numberPointer(values["ups.load"]), InputVoltage: numberPointer(values["input.voltage"]),
 		OutputVoltage: numberPointer(values["output.voltage"]), InputFrequency: numberPointer(values["input.frequency"]),
 		BatteryVoltage: numberPointer(values["battery.voltage"]), Temperature: numberPointer(firstNonEmpty(values["ups.temperature"], values["battery.temperature"])),
+		BatteryDate: firstNonEmpty(values["battery.date"], values["battery.mfr.date"]), ReplaceBattery: strings.Contains(status, "RB"), LowBatteryCharge: numberPointer(values["battery.charge.low"]), LowBatteryRuntime: numberPointer(values["battery.runtime.low"]),
 	}
+	if metrics.BatteryDate != "" {
+		for _, layout := range []string{"2006/01/02", "2006-01-02", "01/02/2006", "02/01/2006"} {
+			if date, err := time.Parse(layout, metrics.BatteryDate); err == nil {
+				metrics.BatteryAgeDays = int(time.Since(date).Hours() / 24)
+				break
+			}
+		}
+	}
+	return metrics
 }
 
 func numberPointer(value string) *float64 {

zfs.go

@@ -2,7 +2,9 @@ package main
 
 import (
 	"context"
+	"os"
 	"os/exec"
+	"regexp"
 	"strconv"
 	"strings"
 	"sync"
@@ -21,17 +23,55 @@ type ZFSPoolMetrics struct {
 	Scan             string    `json:"scan"`
 	Errors           string    `json:"errors"`
 	AttentionReasons []string  `json:"attentionReasons"`
+	ReadErrors       uint64    `json:"readErrors"`
+	WriteErrors      uint64    `json:"writeErrors"`
+	ChecksumErrors   uint64    `json:"checksumErrors"`
+	VDevs            []ZFSVDev `json:"vdevs"`
+	ScrubStatus      string    `json:"scrubStatus"`
+	LastScrubAt      string    `json:"lastScrubAt"`
+	ScrubDuration    string    `json:"scrubDuration"`
+	ScrubTooOld      bool      `json:"scrubTooOld"`
+	ErrorChanges     []string  `json:"errorChanges"`
+}
+
+type ZFSVDev struct {
+	Name     string `json:"name"`
+	Class    string `json:"class"`
+	State    string `json:"state"`
+	Read     uint64 `json:"read"`
+	Write    uint64 `json:"write"`
+	Checksum uint64 `json:"checksum"`
+}
+type ZFSDataset struct {
+	Name            string  `json:"name"`
+	Pool            string  `json:"pool"`
+	UsedBytes       uint64  `json:"usedBytes"`
+	AvailableBytes  uint64  `json:"availableBytes"`
+	ReferencedBytes uint64  `json:"referencedBytes"`
+	Mountpoint      string  `json:"mountpoint"`
+	CapacityPercent float64 `json:"capacityPercent"`
+}
+type ZFSARC struct {
+	Available   bool    `json:"available"`
+	SizeBytes   uint64  `json:"sizeBytes"`
+	TargetBytes uint64  `json:"targetBytes"`
+	HitRatio    float64 `json:"hitRatio"`
+	Hits        uint64  `json:"hits"`
+	Misses      uint64  `json:"misses"`
 }
 
 type ZFSMetrics struct {
 	Available bool             `json:"available"`
 	Pools     []ZFSPoolMetrics `json:"pools"`
+	Datasets  []ZFSDataset     `json:"datasets"`
+	ARC       ZFSARC           `json:"arc"`
 }
 
 type zfsCollector struct {
 	mu        sync.Mutex
 	cached    ZFSMetrics
 	updatedAt time.Time
+	previous  map[string]ZFSPoolMetrics
 }
 
 func (c *zfsCollector) collect() ZFSMetrics {
@@ -40,7 +80,26 @@ func (c *zfsCollector) collect() ZFSMetrics {
 	if time.Since(c.updatedAt) < 30*time.Second {
 		return c.cached
 	}
-	c.cached = readZFSPools()
+	current := readZFSPools()
+	if c.previous == nil {
+		c.previous = map[string]ZFSPoolMetrics{}
+	}
+	for i := range current.Pools {
+		pool := &current.Pools[i]
+		if old, ok := c.previous[pool.Name]; ok {
+			checks := []struct {
+				label       string
+				before, now uint64
+			}{{"read", old.ReadErrors, pool.ReadErrors}, {"write", old.WriteErrors, pool.WriteErrors}, {"checksum", old.ChecksumErrors, pool.ChecksumErrors}}
+			for _, check := range checks {
+				if check.now > check.before {
+					pool.ErrorChanges = append(pool.ErrorChanges, check.label+": "+strconv.FormatUint(check.before, 10)+" → "+strconv.FormatUint(check.now, 10))
+				}
+			}
+		}
+		c.previous[pool.Name] = *pool
+	}
+	c.cached = current
 	c.updatedAt = time.Now()
 	return c.cached
 }
@@ -67,15 +126,35 @@ func readZFSPools() ZFSMetrics {
 		pool.FreeBytes, _ = strconv.ParseUint(fields[3], 10, 64)
 		pool.Fragmentation = parsePercent(fields[4])
 		pool.CapacityPercent = parsePercent(fields[5])
-		pool.Scan, pool.Errors = readZFSStatus(pool.Name)
+		pool.Scan, pool.Errors, pool.VDevs = readZFSStatus(pool.Name)
+		for _, vdev := range pool.VDevs {
+			pool.ReadErrors += vdev.Read
+			pool.WriteErrors += vdev.Write
+			pool.ChecksumErrors += vdev.Checksum
+			if vdev.State != "ONLINE" {
+				pool.AttentionReasons = append(pool.AttentionReasons, vdev.Class+" "+vdev.Name+": "+vdev.State)
+			}
+		}
+		pool.ScrubStatus, pool.LastScrubAt, pool.ScrubDuration, pool.ScrubTooOld = parseScrub(pool.Scan)
 		if pool.Health != "ONLINE" {
 			pool.AttentionReasons = append(pool.AttentionReasons, "Состояние пула: "+pool.Health)
 		}
 		if pool.Errors != "" && !strings.Contains(pool.Errors, "No known data errors") {
 			pool.AttentionReasons = append(pool.AttentionReasons, pool.Errors)
 		}
+		if pool.ReadErrors+pool.WriteErrors+pool.ChecksumErrors > 0 {
+			pool.AttentionReasons = append(pool.AttentionReasons, "Ошибки vdev: read="+strconv.FormatUint(pool.ReadErrors, 10)+", write="+strconv.FormatUint(pool.WriteErrors, 10)+", checksum="+strconv.FormatUint(pool.ChecksumErrors, 10))
+		}
+		if pool.ScrubTooOld {
+			pool.AttentionReasons = append(pool.AttentionReasons, "Scrub не выполнялся более 35 дней")
+		}
+		if pool.CapacityPercent >= 80 {
+			pool.AttentionReasons = append(pool.AttentionReasons, "Мало свободного места для безопасной CoW-работы ZFS")
+		}
 		result.Pools = append(result.Pools, pool)
 	}
+	result.Datasets = readZFSDatasets()
+	result.ARC = readZFSARC()
 	return result
 }
 
@@ -84,11 +163,13 @@ func parsePercent(value string) float64 {
 	return parsed
 }
 
-func readZFSStatus(pool string) (string, string) {
+func readZFSStatus(pool string) (string, string, []ZFSVDev) {
 	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
 	defer cancel()
 	output, _ := exec.CommandContext(ctx, "zpool", "status", pool).Output()
-	var scan, errors string
+	var scan, errors, currentClass string
+	var vdevs []ZFSVDev
+	inConfig := false
 	for _, line := range strings.Split(string(output), "\n") {
 		trimmed := strings.TrimSpace(line)
 		if strings.HasPrefix(trimmed, "scan:") {
@@ -97,6 +178,98 @@ func readZFSStatus(pool string) (string, string) {
 		if strings.HasPrefix(trimmed, "errors:") {
 			errors = strings.TrimSpace(strings.TrimPrefix(trimmed, "errors:"))
 		}
+		if strings.HasPrefix(trimmed, "config:") {
+			inConfig = true
+			continue
 		}
-	return scan, errors
+		if !inConfig || trimmed == "" || strings.HasPrefix(trimmed, "NAME") {
+			continue
+		}
+		fields := strings.Fields(trimmed)
+		if len(fields) == 1 && (fields[0] == "logs" || fields[0] == "cache" || fields[0] == "spares" || fields[0] == "special") {
+			currentClass = fields[0]
+			continue
+		}
+		if len(fields) >= 5 && (fields[1] == "ONLINE" || fields[1] == "DEGRADED" || fields[1] == "OFFLINE" || fields[1] == "UNAVAIL" || fields[1] == "FAULTED" || fields[1] == "REMOVED") {
+			read, _ := strconv.ParseUint(fields[2], 10, 64)
+			write, _ := strconv.ParseUint(fields[3], 10, 64)
+			checksum, _ := strconv.ParseUint(fields[4], 10, 64)
+			class := currentClass
+			if class == "" {
+				class = "data"
+			}
+			vdevs = append(vdevs, ZFSVDev{Name: fields[0], Class: class, State: fields[1], Read: read, Write: write, Checksum: checksum})
+		}
+	}
+	return scan, errors, vdevs
+}
+
+var scrubDatePattern = regexp.MustCompile(`(?:on|since) (\w{3} \w{3}\s+\d+ \d\d:\d\d:\d\d \d{4})`)
+
+func parseScrub(scan string) (status, lastAt, duration string, tooOld bool) {
+	lower := strings.ToLower(scan)
+	status = scan
+	if strings.Contains(lower, "in progress") {
+		status = "Выполняется"
+	} else if strings.Contains(lower, "repaired") {
+		status = "Завершён"
+	} else if strings.Contains(lower, "none requested") || scan == "" {
+		status = "Никогда не запускался"
+		return status, "", "", true
+	}
+	if match := scrubDatePattern.FindStringSubmatch(scan); len(match) > 1 {
+		if t, err := time.Parse("Mon Jan 2 15:04:05 2006", match[1]); err == nil {
+			lastAt = t.Format(time.RFC3339)
+			tooOld = time.Since(t) > 35*24*time.Hour
+		}
+	}
+	if parts := strings.Split(scan, " in "); len(parts) > 1 {
+		duration = strings.Fields(parts[1])[0]
+	}
+	return
+}
+
+func readZFSDatasets() []ZFSDataset {
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+	out, err := exec.CommandContext(ctx, "zfs", "list", "-H", "-p", "-o", "name,used,avail,refer,mountpoint").Output()
+	if err != nil {
+		return nil
+	}
+	var result []ZFSDataset
+	for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
+		f := strings.Fields(line)
+		if len(f) < 5 {
+			continue
+		}
+		used, _ := strconv.ParseUint(f[1], 10, 64)
+		avail, _ := strconv.ParseUint(f[2], 10, 64)
+		refer, _ := strconv.ParseUint(f[3], 10, 64)
+		pct := 0.0
+		if used+avail > 0 {
+			pct = float64(used) / float64(used+avail) * 100
+		}
+		result = append(result, ZFSDataset{Name: f[0], Pool: strings.Split(f[0], "/")[0], UsedBytes: used, AvailableBytes: avail, ReferencedBytes: refer, Mountpoint: strings.Join(f[4:], " "), CapacityPercent: pct})
+	}
+	return result
+}
+
+func readZFSARC() ZFSARC {
+	data, err := os.ReadFile("/proc/spl/kstat/zfs/arcstats")
+	if err != nil {
+		return ZFSARC{}
+	}
+	values := map[string]uint64{}
+	for _, line := range strings.Split(string(data), "\n") {
+		f := strings.Fields(line)
+		if len(f) >= 3 {
+			values[f[0]], _ = strconv.ParseUint(f[2], 10, 64)
+		}
+	}
+	hits, misses := values["hits"], values["misses"]
+	ratio := 0.0
+	if hits+misses > 0 {
+		ratio = float64(hits) / float64(hits+misses) * 100
+	}
+	return ZFSARC{Available: true, SizeBytes: values["size"], TargetBytes: values["c"], HitRatio: ratio, Hits: hits, Misses: misses}
 }