Automatically install missing conntrack for enabled network collector

README.md

@@ -3,8 +3,139 @@
 Локальная веб-панель CPU для Proxmox. Данные читаются напрямую из `/proc` и
 `/sys`, без Proxmox API. Интерфейс обновляется каждые две секунды.
 
+## Установка для новичка — скопируйте команды целиком
+
+### Что потребуется
+
+- сервер Proxmox VE с доступом в интернет или к `git.myown.center`;
+- вход в консоль Proxmox под пользователем `root`;
+- IP-адрес Proxmox в локальной сети;
+- свободный TCP-порт `9105` между вашим компьютером и Proxmox.
+
+Go, Node.js, Docker и отдельную базу данных устанавливать не нужно. Панель
+поставляется готовым Linux-бинарником, а SQLite встроен внутрь.
+
+### 1. Подключитесь к Proxmox
+
+На своём компьютере откройте Terminal и выполните команду, заменив IP на адрес
+своего Proxmox:
+
+```bash
+ssh root@192.168.0.10
+```
+
+Введите пароль `root`. Следующие команды выполняйте уже внутри Proxmox.
+
+### 2. Установите панель
+
+Скопируйте весь блок и вставьте его в консоль Proxmox:
+
+```bash
+apt-get update
+apt-get install -y curl ca-certificates tar smartmontools iproute2 iputils-ping ethtool
+
+cd /tmp
+curl -fL \
+  https://git.myown.center/maxim/ProxmoxDash/releases/download/latest/proxmox-dashboard-install-amd64.tar.gz \
+  -o proxmox-dashboard-install-amd64.tar.gz
+
+tar -xzf proxmox-dashboard-install-amd64.tar.gz
+cd proxmox-dashboard-install
+UPDATE_MODE=notify ./scripts/install.sh
+```
+
+Готово. Установщик сам:
+
+- положит бинарник в `/usr/local/bin`;
+- создаст локальную базу настроек и истории;
+- установит systemd-службу;
+- запустит панель;
+- настроит ежедневную проверку новых версий;
+- сохранит предыдущий бинарник при будущем обновлении.
+
+### 3. Откройте панель
+
+В браузере на своём компьютере откройте:
+
+```text
+http://192.168.0.10:9105
+```
+
+Замените `192.168.0.10` на IP своего Proxmox. Узнать адрес можно командой:
+
+```bash
+hostname -I
+```
+
+### 4. Проверьте, что всё работает
+
+```bash
+systemctl --no-pager --full status proxmox-cpu-dashboard.service
+curl -I http://127.0.0.1:9105
+```
+
+В статусе службы должно быть `active (running)`, а локальная проверка должна
+вернуть HTTP-ответ `200 OK`.
+
+### Как обновлять
+
+В режиме `notify`, установленном командами выше, откройте в панели
+`Настройки → Обновления` и нажмите `Проверить и установить обновление`.
+
+Если нужны полностью автоматические обновления, повторно запустите установщик:
+
+```bash
+cd /tmp/proxmox-dashboard-install
+UPDATE_MODE=auto ./scripts/install.sh
+```
+
+### Если репозиторий станет приватным
+
+Создайте в Gitea токен только для чтения репозитория и передайте его
+установщику. Вместо текста `ВАШ_ТОКЕН` вставьте настоящий токен:
+
+```bash
+cd /tmp/proxmox-dashboard-install
+GITEA_TOKEN='ВАШ_ТОКЕН' UPDATE_MODE=notify ./scripts/install.sh
+```
+
+Токен сохраняется на Proxmox с правами `0600` и не показывается в браузере.
+
+### Если страница не открывается
+
+Выполните на Proxmox:
+
+```bash
+systemctl restart proxmox-cpu-dashboard.service
+systemctl --no-pager --full status proxmox-cpu-dashboard.service
+journalctl -u proxmox-cpu-dashboard.service -n 100 --no-pager
+ss -lntp | grep 9105
+```
+
+Проверьте, что используете правильный IP, порт `9105` доступен из локальной
+сети, а firewall Proxmox не блокирует подключение. Панель пока не имеет
+авторизации — не публикуйте порт `9105` напрямую в интернет.
+
+---
+
 Панель показывает:
 
+- изменение критичных SMART-счётчиков, результаты self-test и исчезновение дисков;
+- realtime read/write IOPS, MB/s, latency, utilization и глубину очереди каждого диска;
+- недельную историю IOPS/utilization и алерты при устойчивом I/O-давлении;
+- вероятный источник I/O: VM/LXC, процесс хоста или активная backup/snapshot/migration-задача;
+- отдельные пороги HDD, SATA SSD и NVMe с исключениями по имени или серийному номеру;
+- недельные baseline-аномалии IOPS, latency, температуры, RAM, ZFS и длительности backup;
+- корреляцию связанных событий в одну причинно-следственную цепочку;
+- время первого появления и последнего наблюдения каждого уведомления;
+- автоматический короткий SMART self-test раз в неделю для дисков без свежего теста;
+- datasets, vdev-классы, ошибки read/write/checksum, scrub и ARC ZFS;
+- доступность шлюза, DNS и интернета, packet loss, latency и внешний IP;
+- тип, состояние, скорость, duplex и master сетевых интерфейсов;
+- состояние связи с NUT/apcupsd, возраст батареи и расчётную автономность UPS;
+- настраиваемый сценарий корректного выключения VM/LXC и Proxmox с безопасным dry-run;
+- самоконтроль сборщика, записи истории, свободного места базы и systemd watchdog;
+
 - модель и текущую загрузку CPU;
 - количество сокетов, физических ядер и логических потоков;
 - среднюю текущую частоту;
@@ -19,15 +150,51 @@
 - состояние, заполнение, фрагментацию и scrub/resilver ZFS-пулов;
 - состояние UPS через NUT или apcupsd, заряд, автономность, нагрузку и напряжение.
 - состояние VM и LXC, CPU, память, выделенные ядра и uptime.
+- IP-адреса VM через QEMU Guest Agent, автозапуск, storage, заполнение диска и
+  суммарный сетевой трафик гостей;
 - последнюю найденную резервную копию каждой VM/LXC, storage, размер и возраст.
+- версию Proxmox, запущенное и последнее установленное ядро, доступные
+  обновления, необходимость перезагрузки, последнее обновление системы,
+  ближайшие backup и ZFS scrub.
+- управляемый мониторинг HTTP/HTTPS, TCP и ping для homelab-сервисов: отклик,
+  ожидаемый код и текст, TLS-сертификат, uptime за сутки и неделю;
+- ошибки systemd и ядра за 24 часа, упавшие службы, ошибки и drops каждого
+  сетевого интерфейса.
+- snapshots VM/LXC с возрастом, описанием и доступным размером;
+- последние задачи Proxmox: backup, start/stop, snapshot, migration, пользователь,
+  длительность и результат;
+- сравнение размера backup с предыдущей копией и ручную безопасную проверку
+  читаемости локального архива или конфигурации PBS snapshot;
+- карту homelab: Proxmox, VM/LXC, сервисы и используемые storage.
 
 Через кнопку настроек можно скрывать неиспользуемые карточки и переключать
-светлую или тёмную тему. Выбор сохраняется локально в браузере.
+светлую или тёмную тему. Карточки можно переставлять перетаскиванием или
+стрелками в настройках. Видимость, порядок и тема сохраняются локально в
+браузере.
 
 Центр уведомлений объединяет критические события, предупреждения и
 информационные сообщения по CPU, памяти, storage, SMART, ZFS, UPS, VM/LXC и
 резервным копиям. Критические события всегда отображаются первыми.
 
+Во вкладке `Настройки → Почта` настраиваются SMTP-сервер, получатель и интервал
+агрегированной сводки. Можно отдельно выбрать уровни важности и источники
+событий — например, полностью исключить уведомления о резервных копиях. Там же
+доступна отправка тестового письма. Поддерживаются STARTTLS (обычно порт 587) и
+SMTP поверх TLS (порт 465).
+
+Тема письма кодируется по MIME и корректно отображает кириллицу. Сам дайджест
+оформлен как HTML-письмо и содержит только новые неподтверждённые события — уже
+отправленные или принятые события повторно не рассылаются. В центре уведомлений
+есть вкладки активных событий и 30-дневной истории. Кнопка `Принято` скрывает
+событие из активных, не удаляя его из журнала; повторное возникновение проблемы
+создаёт новое событие.
+
+Критические события отправляются сразу, не ожидая очередной сводки. К ним
+относятся недоступность сервиса, отказ SMART/ZFS/systemd, критический перегрев и
+низкий заряд UPS. Переход UPS на батарею также отправляется немедленно. После
+устранения уже отправленной проблемы приходит отдельное уведомление о
+восстановлении.
+
 Пороги уведомлений сохраняются в SQLite и настраиваются через вкладку
 `Настройки → Уведомления`. Панель раз в минуту сохраняет историю CPU,
 температуры, памяти, swap, сети, storage, ZFS, UPS и гостей. Данные старше семи
@@ -44,6 +211,13 @@
 Готовые версии для Linux публикуются в разделе Releases репозитория. Бинарник
 не требует установки Go или системной библиотеки SQLite.
 
+Проверка обновлений выполняется отдельным systemd-таймером раз в сутки. Статус
+виден во вкладке `Настройки → Обновления` и в центре уведомлений. Доступны два
+режима: `notify` только сообщает о новой версии, `auto` скачивает её, сохраняет
+предыдущий бинарник как резервный и перезапускает панель. В режиме `notify`
+обновление можно запустить кнопкой `Проверить и установить обновление` в
+настройках. Запрос выполняет отдельная системная служба.
+
 На главном экране находятся компактные карточки CPU, памяти, диска и сети.
 Нажатие на карточку открывает подробные показатели.
 
@@ -56,6 +230,32 @@ chmod +x proxmox-cpu-dashboard-linux-amd64
 ./proxmox-cpu-dashboard-linux-amd64
 ```
 
+## Установка на сервер и обновления
+
+Скачайте из Releases архив `proxmox-dashboard-install-amd64.tar.gz`, распакуйте
+его на Proxmox и запустите от root:
+
+```bash
+tar -xzf proxmox-dashboard-install-amd64.tar.gz
+cd proxmox-dashboard-install
+UPDATE_MODE=notify ./scripts/install.sh
+```
+
+Для полностью автоматических обновлений используйте `UPDATE_MODE=auto`.
+Приватному репозиторию нужен read-only API-токен Gitea:
+
+```bash
+GITEA_TOKEN='read-only-token' UPDATE_MODE=auto ./scripts/install.sh
+```
+
+Токен сохраняется с правами `0600` в `/etc/proxmox-dashboard/update.env`.
+Проверить обновления вручную можно командой:
+
+```bash
+systemctl start proxmox-dashboard-update.service
+systemctl status proxmox-dashboard-update.service
+```
+
 Откройте:
 
 ```text
@@ -120,6 +320,80 @@ apcaccess status
 - `GET /api/cpu` — один JSON-снимок.
 - `GET /api/system` — единый JSON-снимок CPU и памяти.
 - `GET /api/events` — поток метрик SSE с интервалом две секунды.
+- `GET/POST/PUT/DELETE /api/services` — управление проверками сервисов.
+- `GET /api/services/history` — семидневная история доступности и отклика.
+- `GET/PUT/DELETE /api/settings/maintenance` — состояние, запуск и досрочное завершение плановых работ.
+- `GET/DELETE /api/agents` — подключённые гостевые агенты.
+- `POST /api/agents/enrollment-token` — одноразовый токен регистрации на 15 минут.
+
+## Maintenance-режим
+
+Во вкладке `Настройки → Maintenance` можно на заданное время отключить все
+алерты либо выбрать конкретные сервисы и VM/LXC. Подавленные события не
+отправляются по почте и не отображаются как активные, но сохраняются в истории
+с причиной плановых работ. После окончания окна обычный мониторинг включается
+автоматически.
+
+## Агенты VM и LXC
+
+Read-only агент показывает состояние Linux внутри VM/LXC: CPU, load average,
+память, корневой раздел, uptime, ОС, ядро, IP-адреса, systemd-службы, доступные
+APT-обновления и необходимость перезагрузки. Для Docker отображаются контейнеры,
+health-check, число перезапусков, CPU и память. Агент сам подключается к Dashboard
+каждые 10 секунд; открывать входящий порт внутри гостевой системы не нужно.
+
+В подробностях агента systemd-службы разделяются на системные, установленные
+дополнительно и локальные unit-файлы. Для каждой службы показываются пакет и путь
+unit-файла, время запуска, число перезапусков, последний exit code, память и
+накопленное CPU-время, когда соответствующий accounting доступен в systemd.
+
+Расширенная диагностика агента включает последние сообщения проблемных служб,
+`Requires`/`After`, PID, команду запуска и слушающие порты; выявляет частые
+рестарты, OOM-kill и процессы в D-state. Отдельно отображаются top процессов по
+CPU и RAM с накопленным I/O, swap, PSI pressure, все постоянные файловые системы,
+их заполнение и использование inode.
+
+Панель «Диагностика homelab» и карта сопоставляют известные IP и порты с VM/LXC,
+агентом, systemd-службой или Docker-контейнером и настроенной HTTP/TCP-проверкой.
+Связи по PID и порту считаются подтверждёнными; совпадения только по названию
+помечаются как предположение.
+
+Для сложных reverse proxy и Docker-сетей в управлении сервисами можно сохранить
+ручную связь `сервис → агент/VMID → systemd/Docker/процесс → порт`. Она имеет
+приоритет над автоматическим определением и используется для группировки
+связанных уведомлений по первопричине.
+
+Dashboard хранит семь дней истории CPU, RAM, swap, PSI и файловых систем каждого
+агента. Из подробностей агента доступны только заранее разрешённые действия:
+перезапуск известной systemd-службы или Docker-контейнера, `reset-failed` и
+обновление списка пакетов. Агент не принимает произвольные shell-команды; каждое
+действие требует подтверждения в браузере и сохраняется в журнале управления.
+
+Откройте `Настройки → Агенты`, укажите адрес Dashboard, доступный из VM/LXC, и
+VMID. Кнопка создаст готовую команду установки. Одноразовый токен действует 15
+минут и после регистрации заменяется индивидуальным секретом агента.
+
+После однократной установки агент обновляется автоматически вслед за Dashboard.
+Центральный сервер отдаёт агенту собственный бинарник и SHA-256; агент проверяет
+контрольную сумму, атомарно заменяет файл и перезапускается через systemd. Если
+агент был удалён из Dashboard, повторная команда с новым токеном автоматически
+выполнит новую регистрацию.
+
+В первой версии агент не выполняет команды и не перезапускает службы или
+контейнеры. Для чтения Docker ему требуется доступ к `/var/run/docker.sock`,
+поэтому служба устанавливается от root. Сам Dashboard пока не имеет
+авторизации: используйте локальную сеть или HTTPS через доверенный reverse
+proxy и не публикуйте агентские API напрямую в интернет.
+
+Полностью удалить агент из VM/LXC:
+
+```bash
+curl -fsSL https://git.myown.center/maxim/ProxmoxDash/raw/branch/main/scripts/uninstall-agent.sh | sh
+```
+
+Оформление, пороги, почта и обновления Dashboard находятся в обычных настройках.
+Рабочие инструменты — сервисы, агенты, плановые работы и UPS-сценарий — вынесены
+в отдельный раздел `Управление` в шапке Dashboard.
 
 ## Сборка
 
@@ -127,3 +401,163 @@ apcaccess status
 go test ./...
 GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o proxmox-cpu-dashboard .
 ```
+
+## Сетевые ошибки и температура CPU (v0.19.3)
+
+RX/TX drops отображаются отдельно как информационные счётчики. Их рост сам по
+себе не вызывает аварийный алерт: в том числе LLDP на Linux bridge. Настройки
+Keenetic, LLDP и group_fwd_mask менять не требуется. Для VM учитываются tap,
+для LXC — veth; fwbr/fwpr/fwln остаются в группе интерфейсов хоста.
+
+Новые rx_errors/tx_errors, CRC/frame/FIFO/missed, carrier/link-down вызывают
+критический алерт. Sysfs позволяет отделить rx_missed_errors от RX drops,
+которые в /proc/net/dev объединены. Для Atlantic дополнительно читаются
+InErrors, InDroppedDma, AllocFails, SkbAllocFails и Restarts через ethtool -S.
+Установите `ethtool` на существующем хосте для этой дополнительной диагностики.
+При отсутствии утилиты остаются стандартные счётчики sysfs. Счётчики ошибок
+могут пересекаться: их сумма — прирост счётчиков, а не число уникальных пакетов.
+После перезапуска и сброса счётчиков старые значения не создают новые алерты.
+
+Карточка CPU и новые точки истории показывают максимум доступных датчиков CPU,
+а не среднее Tctl/Tdie/CCD или ядер. На AMD Tctl может включать управляющее
+смещение; подписи и значения всех датчиков доступны в подробностях CPU.
+История снимается раз в минуту, а на диапазоне 7 дней усредняется по 10 минутам,
+поэтому короткие пики между измерениями не гарантированно попадут на график.
+Старые точки до обновления остаются средними: исходные значения датчиков в них
+не сохранены. Масштаб графика расширяется для значений выше 100 °C.
+
+Релиз Linux amd64 собирается с номером версии и без CGO:
+
+```bash
+./scripts/build-release.sh v0.19.3
+```
+
+Скрипт запускает тесты и создаёт в dist бинарник, SHA-256 и установочный архив,
+совместимые со встроенным обновлением из Gitea Releases.
+
+## Карта трафика NPM + CrowdSec (v0.20.0)
+
+С v0.20.1 страница помещается в высоту окна: карта и лента в центре,
+счётчики сверху, график и рейтинги снизу. На телефоне доступны вкладки
+«Карта / События / Статистика». Последние 100 событий листаются по страницам;
+нажатие на строку открывает полные подробности.
+
+Откройте **Настройки → Карта трафика**, включите страницу и выберите
+агент VM/LXC, где установлены Nginx Proxy Manager и CrowdSec. Например,
+`nginxproxymanager · VMID 102`. Необязательно выберите страну сервера для
+конечной точки линий на карте. Сохраните настройки: в шапке появится ссылка
+«Карта трафика», страница также доступна по `/traffic`.
+
+Функция **по умолчанию выключена**. Только выбранный агент получает команду
+включить сбор. При выключении или смене источника старый агент останавливает
+сбор после следующего успешного обычного отчёта; сервер сразу прекращает
+приём его событий. Выключенная страница и её API недоступны. Это глобальная
+настройка экземпляра Dashboard. Карта не создаёт общих алертов или писем.
+
+Агент автоматически обновляется вместе с Dashboard; для карты требуется
+v0.20.0 или новее. Он читает стандартные NPM access-логи из
+`/opt/npm/data/logs/*_access.log`, включая несжатые файлы ротации. Другой каталог
+задаётся переменной окружения службы `DASHBOARD_NPM_LOG_DIR`; значение `off`
+полностью запрещает сбор на конкретном агенте. Порт CrowdSec наружу открывать
+и новые API-ключи создавать не нужно: используется локальная команда
+`cscli alerts list --since 24h --limit 50 -o json` с существующими учётными
+данными машины. Права на логи и запуск cscli нужны службе агента.
+
+- Голубой: все записанные HTTP-запросы, включая обычные и локальные обращения.
+- Оранжевый: реальные (не simulated) срабатывания сценариев CrowdSec.
+- Розовый: решения CrowdSec, например ban или captcha. Это **не** счётчик
+  фактически отброшенных пакетов и не список действующих сейчас банов.
+- HTTP 401/403/502 сами по себе не считаются атакой или блокировкой.
+
+Карта показывает страны источников, живую ленту последних 100 событий,
+счётчики за последние 24 часа, уникальные IP HTTP-клиентов, график по минутам
+и средний req/s за последние 60 секунд. Есть фильтры типа события, домена и
+внешних IP, пауза и полный экран. Локальные адреса видны в ленте без страны.
+Все счётчики следуют выбранным фильтрам. Уникальный IP не равен человеку.
+
+Первое включение начинает чтение с конца логов, без импорта старых HTTP-запросов.
+Обычно обновление занимает несколько секунд; Nginx записывает запрос после
+его завершения. Это не карта открытых TCP-соединений. Позиции чтения и очередь
+до 500 событий сохраняются в `traffic-state.json` рядом с конфигом агента;
+повторы после обрыва связи устраняются на сервере. При переполнении очереди
+чтение ждёт отправки. Если за время длительного простоя логи уже сжаты или
+удалены, восстановить эти строки нельзя. Размер очереди чтения и ошибки
+парсинга отображаются на самой странице, без глобальных уведомлений.
+
+CrowdSec опрашивается каждые 15 секунд, до 50 последних alerts за 24 часа;
+при достижении лимита страница предупреждает о возможной неполноте истории.
+Связанный домен берётся из метаданных события, только если он однозначен.
+Новый ban не рисуется как новый запрос. Для наблюдения реально отброшенных
+firewall пакетов нужен отдельный источник: в этой версии его нет.
+
+В SQLite хранится до 24 часов и 200 000 событий; старые строки удаляются при
+приёме новых данных. Данные каждого выбранного агента фильтруются отдельно.
+Query string и fragment URL отбрасываются на агенте и повторно на сервере;
+заголовки, cookie и тела запросов не передаются. IP, домен и путь сохраняются.
+
+GeoIP работает локально: DB-IP Country Lite за октябрь 2026 встроена в бинарник,
+исходящие запросы с IP посетителей не выполняются. Координаты на карте — условные
+точки стран Natural Earth, а не точное местоположение посетителя. Дата базы
+видна внизу карты. Обновление базы требует нового релиза; автоматического
+скачивания нет. Лицензии и происхождение данных: `assets/NOTICE.md`.
+
+Локальный предпросмотр с явно обозначенными искусственными данными:
+
+```bash
+DASHBOARD_TRAFFIC_PREVIEW=1 go test -run '^TestTrafficPreview$' -timeout 30m
+# http://127.0.0.1:19105/traffic
+```
+
+## Сетевая активность TCP/UDP/ICMP (v0.21.0)
+
+В **Настройки → Карта трафика** включите дополнительный сбор TCP/UDP/ICMP.
+Он выполняется только выбранным агентом v0.21.0+; на странице карты появится
+кнопка **TCP / UDP**, открывающая `/network`. Основной выключатель карты
+отключает оба сборщика. Сетевой режим по умолчанию выключен и не создаёт
+общих алертов/писем.
+
+На Debian/Ubuntu агент **сам проверяет и устанавливает conntrack**, когда
+включён сетевой режим. Выполняются `apt-get update` и установка только
+`conntrack` с зависимостями (`--no-install-recommends --no-remove`), без
+обновления всей системы. Установка выполняется от root через отдельную
+systemd-службу, как существующие операции обновления агента, и не требует
+отключать его read-only sandbox. Начатая пакетная операция завершается даже
+при выключении сбора. Ход/ошибка видны в диагностике; повтор после ошибки —
+не чаще раза в 30 минут работы процесса. После перезапуска агента проверка
+выполняется заново. Если conntrack уже установлен, apt-get не запускается.
+
+На других системах, без root или systemd/apt-get, выводится инструкция для
+ручной установки. Для счётчиков CrowdSec требуется доступный `nft`; он
+обычно уже установлен вместе с nftables-bouncer и автоматически не ставится.
+
+Агенту нужны root/CAP_NET_ADMIN в сетевом namespace хоста. Установщик агента
+запускает службу от root. Чтение: `/proc/net/nf_conntrack`, если доступен,
+иначе `conntrack -L -f ipv4/ipv6 -o extended`. Ничего устанавливать на хост
+самого Dashboard не требуется. Firewall, NAT и sysctl сборщик не меняет.
+
+Страница показывает состояния TCP, UDP/ICMP-потоки, обе пары адресов до/после
+NAT (в том числе Docker), локальную GeoIP, топ инициаторов и портов, фильтры,
+поиск и детали потока. Список листается по страницам; переход на следующую
+фиксирует снимок, «К свежим» возвращает live. Для телефона — вкладки.
+
+Это **снимки таблицы примерно каждые 5 секунд**, не полный журнал соединений:
+короткие потоки между снимками могут быть пропущены. До 5 000 записей на
+снимок, 8 MiB вывода команды и таймаут 4 секунды; ограничения/ошибки видны в
+диагностике. Сохраняется только последний снимок каждого агента. Инициатор
+не обязательно внешний клиент: NPM → upstream — отдельный поток. Порт не
+доказывает протокол приложения; UDP ASSURED не означает TCP handshake.
+
+Байты/пакеты доступны только при включённом учёте conntrack для конкретных
+потоков. Это накопленные значения за жизнь потока, а не трафик за 24 часа.
+При отсутствии учёта интерфейс показывает «нет учёта». При необходимости
+администратор может включить `net.netfilter.nf_conntrack_acct=1`; изменение
+применяется к новым потокам и автоматически не выполняется.
+
+Firewall читается раз в 30 секунд командой `nft -j list ruleset`. Поддержаны
+анонимные счётчики непосредственно перед финальным `drop/reject` в таблице
+или цепочке с `crowdsec` в имени. Остальные счётчики не выдаются за блокировки.
+Это сумма пакетов/байтов выбранных правил с момента их создания, независимо
+от фильтров потоков; при пересоздании правил она может уменьшиться.
+Для iptables/ipset-only, именованных счётчиков и правил без счётчиков выводится
+«недоступно», а не ноль. Отброшенный пакет до подтверждения conntrack может
+отсутствовать в таблице потоков. Содержимое пакетов не перехватывается.

activity.go

@@ -0,0 +1,189 @@
+package main
+
+import (
+	"encoding/json"
+	"fmt"
+	"os"
+	"os/exec"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+)
+
+type ProxmoxTask struct {
+	UPID      string `json:"upid"`
+	Node      string `json:"node"`
+	Type      string `json:"type"`
+	ID        string `json:"id"`
+	User      string `json:"user"`
+	Status    string `json:"status"`
+	StartTime int64  `json:"starttime"`
+	EndTime   int64  `json:"endtime"`
+	Duration  int64  `json:"duration"`
+}
+type GuestSnapshot struct {
+	VMID        int     `json:"vmid"`
+	GuestName   string  `json:"guestName"`
+	GuestType   string  `json:"guestType"`
+	Name        string  `json:"name"`
+	Description string  `json:"description"`
+	CreatedAt   int64   `json:"createdAt"`
+	Running     bool    `json:"running"`
+	Parent      string  `json:"parent,omitempty"`
+	SizeBytes   *uint64 `json:"sizeBytes,omitempty"`
+	AgeSeconds  int64   `json:"ageSeconds"`
+}
+type ProxmoxActivity struct {
+	Available bool            `json:"available"`
+	Tasks     []ProxmoxTask   `json:"tasks"`
+	Snapshots []GuestSnapshot `json:"snapshots"`
+	CheckedAt int64           `json:"checkedAt"`
+}
+type activityCollector struct {
+	mu         sync.RWMutex
+	value      ProxmoxActivity
+	refreshing bool
+}
+
+func newActivityCollector() *activityCollector {
+	c := &activityCollector{refreshing: true}
+	go c.refresh()
+	return c
+}
+func (c *activityCollector) collect(guests GuestsMetrics) ProxmoxActivity {
+	c.mu.RLock()
+	value := c.value
+	c.mu.RUnlock()
+	c.mu.Lock()
+	if !c.refreshing && time.Since(time.Unix(value.CheckedAt, 0)) > 15*time.Second {
+		c.refreshing = true
+		go c.refreshWithGuests(guests)
+	}
+	c.mu.Unlock()
+	return value
+}
+func (c *activityCollector) refresh() { c.refreshWithGuests(readGuests()) }
+func (c *activityCollector) refreshWithGuests(guests GuestsMetrics) {
+	value := readProxmoxActivity(guests)
+	c.mu.Lock()
+	c.value = value
+	c.refreshing = false
+	c.mu.Unlock()
+}
+func readProxmoxActivity(guests GuestsMetrics) ProxmoxActivity {
+	result := ProxmoxActivity{CheckedAt: time.Now().Unix()}
+	if _, err := exec.LookPath("pvesh"); err != nil {
+		return result
+	}
+	node, err := os.Hostname()
+	if err != nil {
+		return result
+	}
+	result.Available = true
+	out := commandOutput(8*time.Second, "pvesh", "get", "/nodes/"+node+"/tasks", "--limit", "100", "--output-format", "json")
+	if out != "" && json.Unmarshal([]byte(out), &result.Tasks) == nil {
+		for i := range result.Tasks {
+			if result.Tasks[i].EndTime > result.Tasks[i].StartTime {
+				result.Tasks[i].Duration = result.Tasks[i].EndTime - result.Tasks[i].StartTime
+			}
+		}
+	}
+	for _, guest := range guests.Guests {
+		if guest.Template != 0 {
+			continue
+		}
+		kind := "qemu"
+		label := "VM"
+		if guest.Type != "qemu" {
+			kind = "lxc"
+			label = "LXC"
+		}
+		out := commandOutput(5*time.Second, "pvesh", "get", fmt.Sprintf("/nodes/%s/%s/%d/snapshot", guest.Node, kind, guest.VMID), "--output-format", "json")
+		if out == "" {
+			continue
+		}
+		var raw []struct {
+			Name        string `json:"name"`
+			Description string `json:"description"`
+			SnapTime    int64  `json:"snaptime"`
+			Running     int    `json:"running"`
+			Parent      string `json:"parent"`
+			VMState     any    `json:"vmstate"`
+		}
+		if json.Unmarshal([]byte(out), &raw) != nil {
+			continue
+		}
+		for _, s := range raw {
+			if s.Name == "current" {
+				continue
+			}
+			item := GuestSnapshot{VMID: guest.VMID, GuestName: guest.Name, GuestType: label, Name: s.Name, Description: s.Description, CreatedAt: s.SnapTime, Running: s.Running != 0, Parent: s.Parent}
+			if item.CreatedAt > 0 {
+				item.AgeSeconds = time.Now().Unix() - item.CreatedAt
+			}
+			if size := uintValue(s.VMState); size > 1 {
+				item.SizeBytes = &size
+			}
+			result.Snapshots = append(result.Snapshots, item)
+		}
+	}
+	sort.Slice(result.Snapshots, func(i, j int) bool { return result.Snapshots[i].CreatedAt > result.Snapshots[j].CreatedAt })
+	return result
+}
+func uintValue(v any) uint64 {
+	switch n := v.(type) {
+	case float64:
+		return uint64(n)
+	case string:
+		value, _ := strconv.ParseUint(n, 10, 64)
+		return value
+	}
+	return 0
+}
+
+type BackupCheckResult struct {
+	OK        bool   `json:"ok"`
+	Level     string `json:"level"`
+	Message   string `json:"message"`
+	CheckedAt int64  `json:"checkedAt"`
+}
+
+func verifyBackup(volid string) BackupCheckResult {
+	result := BackupCheckResult{CheckedAt: time.Now().Unix(), Level: "metadata"}
+	if volid == "" || strings.ContainsAny(volid, "\r\n") {
+		result.Message = "Некорректный идентификатор backup"
+		return result
+	}
+	out, err := exec.Command("pvesm", "path", volid).CombinedOutput()
+	if err != nil {
+		result.Message = "Backup не найден в подключённом storage: " + strings.TrimSpace(string(out))
+		return result
+	}
+	path := strings.TrimSpace(string(out))
+	if strings.HasPrefix(path, "pbs://") {
+		ctxOutput := commandOutput(20*time.Second, "pvesm", "extractconfig", volid)
+		if ctxOutput == "" {
+			result.Message = "Snapshot найден в PBS, но конфигурацию для тестового чтения получить не удалось"
+			return result
+		}
+		result.OK = true
+		result.Level = "pbs-metadata"
+		result.Message = "PBS snapshot и его конфигурация успешно прочитаны. Полное восстановление VM не запускалось."
+		return result
+	}
+	info, err := os.Stat(path)
+	if err != nil {
+		result.Message = "Метаданные доступны, но локальный файл не открыт: " + err.Error()
+		return result
+	}
+	if info.Size() == 0 {
+		result.Message = "Файл backup пуст"
+		return result
+	}
+	result.OK = true
+	result.Level = "readability"
+	result.Message = fmt.Sprintf("Файл доступен для чтения, размер %d байт. Полное тестовое восстановление не запускалось.", info.Size())
+	return result
+}

activity_test.go

@@ -0,0 +1,9 @@
+package main
+
+import "testing"
+
+func TestUintValue(t *testing.T) {
+	if uintValue("42") != 42 || uintValue(float64(7)) != 7 || uintValue(nil) != 0 {
+		t.Fatal("unexpected numeric conversion")
+	}
+}

agent_actions.go

@@ -0,0 +1,346 @@
+package main
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"fmt"
+	"io"
+	"net"
+	"net/http"
+	"os/exec"
+	"path/filepath"
+	"regexp"
+	"strings"
+	"time"
+)
+
+type AgentCommand struct {
+	ID     int64  `json:"id"`
+	Action string `json:"action"`
+	Target string `json:"target,omitempty"`
+}
+
+type AgentActionResult struct {
+	ID      int64  `json:"id"`
+	Success bool   `json:"success"`
+	Message string `json:"message"`
+}
+
+type AgentAction struct {
+	ID          int64  `json:"id"`
+	AgentID     string `json:"agentId"`
+	AgentName   string `json:"agentName"`
+	Action      string `json:"action"`
+	Target      string `json:"target"`
+	RequestedAt int64  `json:"requestedAt"`
+	StartedAt   *int64 `json:"startedAt,omitempty"`
+	CompletedAt *int64 `json:"completedAt,omitempty"`
+	Status      string `json:"status"`
+	Result      string `json:"result"`
+}
+
+var safeUnitName = regexp.MustCompile(`^[A-Za-z0-9_.@:-]+\.service$`)
+var safeContainerName = regexp.MustCompile(`^[A-Za-z0-9_.-]{1,128}$`)
+
+func (s *Store) QueueAgentAction(agentID, action, target string) (AgentAction, error) {
+	action, target = strings.TrimSpace(action), strings.TrimSpace(target)
+	var raw, name string
+	if err := s.db.QueryRow(`SELECT report_json,name FROM agents WHERE id=?`, agentID).Scan(&raw, &name); err != nil {
+		return AgentAction{}, fmt.Errorf("агент не найден")
+	}
+	var report AgentReport
+	_ = json.Unmarshal([]byte(raw), &report)
+	valid := false
+	switch action {
+	case "systemd-restart", "systemd-reset-failed":
+		if safeUnitName.MatchString(target) {
+			for _, service := range report.Services {
+				if service.Name == target {
+					valid = true
+					break
+				}
+			}
+		}
+	case "docker-restart", "docker-check", "docker-update", "docker-remove":
+		if safeContainerName.MatchString(target) {
+			for _, container := range report.Containers {
+				if (container.ID == target || container.Name == target) && (action != "docker-remove" || container.State != "running") {
+					valid = true
+					break
+				}
+			}
+		}
+	case "updates-refresh", "system-update", "system-reboot":
+		valid = target == ""
+	}
+	if !valid {
+		return AgentAction{}, fmt.Errorf("действие или цель не входят в разрешённый список")
+	}
+	now := time.Now().Unix()
+	result, err := s.db.Exec(`INSERT INTO agent_actions(agent_id,action,target,requested_at,status) VALUES(?,?,?,?, 'queued')`, agentID, action, target, now)
+	if err != nil {
+		return AgentAction{}, err
+	}
+	id, _ := result.LastInsertId()
+	return AgentAction{ID: id, AgentID: agentID, AgentName: name, Action: action, Target: target, RequestedAt: now, Status: "queued"}, nil
+}
+
+func (s *Store) DispatchAgentActions(agentID string) ([]AgentCommand, error) {
+	tx, err := s.db.Begin()
+	if err != nil {
+		return nil, err
+	}
+	defer tx.Rollback()
+	// Обновление пакетов и Docker-образов может быть долгим. Не выдаём одну
+	// команду повторно, пока предыдущий запуск ещё имеет шанс завершиться.
+	_, _ = tx.Exec(`UPDATE agent_actions SET status='queued',started_at=NULL WHERE agent_id=? AND status='running' AND started_at<?`, agentID, time.Now().Add(-45*time.Minute).Unix())
+	rows, err := tx.Query(`SELECT id,action,target FROM agent_actions WHERE agent_id=? AND status='queued' ORDER BY id LIMIT 3`, agentID)
+	if err != nil {
+		return nil, err
+	}
+	var commands []AgentCommand
+	for rows.Next() {
+		var c AgentCommand
+		if err = rows.Scan(&c.ID, &c.Action, &c.Target); err != nil {
+			rows.Close()
+			return nil, err
+		}
+		commands = append(commands, c)
+	}
+	rows.Close()
+	now := time.Now().Unix()
+	for _, c := range commands {
+		if _, err = tx.Exec(`UPDATE agent_actions SET status='running',started_at=? WHERE id=?`, now, c.ID); err != nil {
+			return nil, err
+		}
+	}
+	return commands, tx.Commit()
+}
+
+func (s *Store) CompleteAgentActions(agentID string, results []AgentActionResult) {
+	now := time.Now().Unix()
+	for _, result := range results {
+		status := "failed"
+		if result.Success {
+			status = "completed"
+		}
+		message := result.Message
+		if len(message) > 4000 {
+			message = message[:4000]
+		}
+		_, _ = s.db.Exec(`UPDATE agent_actions SET status=?,result=?,completed_at=? WHERE id=? AND agent_id=? AND status='running'`, status, message, now, result.ID, agentID)
+	}
+}
+
+func (s *Store) AgentActions(limit int) ([]AgentAction, error) {
+	if limit < 1 || limit > 200 {
+		limit = 100
+	}
+	rows, err := s.db.Query(`SELECT a.id,a.agent_id,g.name,a.action,a.target,a.requested_at,a.started_at,a.completed_at,a.status,a.result FROM agent_actions a JOIN agents g ON g.id=a.agent_id ORDER BY a.id DESC LIMIT ?`, limit)
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+	var out []AgentAction
+	for rows.Next() {
+		var v AgentAction
+		if err = rows.Scan(&v.ID, &v.AgentID, &v.AgentName, &v.Action, &v.Target, &v.RequestedAt, &v.StartedAt, &v.CompletedAt, &v.Status, &v.Result); err != nil {
+			return nil, err
+		}
+		out = append(out, v)
+	}
+	return out, rows.Err()
+}
+
+func executeAgentCommand(command AgentCommand) AgentActionResult {
+	result := AgentActionResult{ID: command.ID}
+	var output string
+	switch command.Action {
+	case "systemd-restart":
+		output = runAgentCombinedCommand(30*time.Second, "systemctl", "restart", command.Target)
+	case "systemd-reset-failed":
+		output = runAgentCombinedCommand(15*time.Second, "systemctl", "reset-failed", command.Target)
+	case "docker-restart":
+		output = restartAgentContainer(command.Target)
+	case "docker-update":
+		output = updateAgentContainer(command.Target, true)
+	case "docker-check":
+		output = updateAgentContainer(command.Target, false)
+	case "docker-remove":
+		output = removeAgentContainer(command.Target)
+	case "system-update":
+		output = updateAgentSystem()
+	case "system-reboot":
+		output = scheduleAgentReboot()
+	case "updates-refresh":
+		guestInventory.Lock()
+		guestInventory.value.CollectedAt = time.Time{}
+		guestInventory.Unlock()
+		result.Success = true
+		result.Message = "Проверка обновлений поставлена в очередь"
+		return result
+	default:
+		result.Message = "Команда не разрешена"
+		return result
+	}
+	result.Success = !strings.HasPrefix(output, "ERROR: ")
+	result.Message = strings.TrimSpace(output)
+	if result.Message == "" && result.Success {
+		result.Message = "Выполнено успешно"
+	}
+	return result
+}
+
+func removeAgentContainer(target string) string {
+	transport := &http.Transport{DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
+		return (&net.Dialer{}).DialContext(ctx, "unix", "/var/run/docker.sock")
+	}}
+	defer transport.CloseIdleConnections()
+	request, _ := http.NewRequest(http.MethodDelete, "http://docker/containers/"+target+"?v=false&force=false", nil)
+	response, err := (&http.Client{Transport: transport, Timeout: 20 * time.Second}).Do(request)
+	if err != nil {
+		return "ERROR: " + err.Error()
+	}
+	defer response.Body.Close()
+	if response.StatusCode != http.StatusNoContent {
+		message, _ := io.ReadAll(io.LimitReader(response.Body, 2048))
+		return fmt.Sprintf("ERROR: Docker вернул %s: %s", response.Status, strings.TrimSpace(string(message)))
+	}
+	return "Остановленный контейнер удалён; volumes сохранены"
+}
+
+func updateAgentSystem() string {
+	aptGet, err := exec.LookPath("apt-get")
+	if err != nil {
+		return "ERROR: apt-get не найден; обновление поддерживается для Debian/Ubuntu"
+	}
+	if output := runPrivilegedTransient(5*time.Minute, "update-lists", aptGet, "update"); strings.HasPrefix(output, "ERROR: ") {
+		return explainSystemUpdateError(output)
+	}
+	message := runPrivilegedTransient(30*time.Minute, "upgrade", aptGet, "-y", "upgrade")
+	if strings.HasPrefix(message, "ERROR: ") {
+		return explainSystemUpdateError(message)
+	}
+	guestInventory.Lock()
+	guestInventory.value.CollectedAt = time.Time{}
+	guestInventory.Unlock()
+	return "Обновления системы установлены; список пакетов будет обновлён в следующем отчёте агента"
+}
+
+func explainSystemUpdateError(message string) string {
+	if strings.Contains(strings.ToLower(message), "read-only file system") {
+		return "ERROR: файловая система агента доступна только для чтения. Обновите Dashboard и агент до v0.19.2 или новее, затем повторите операцию."
+	}
+	if len(message) > 1800 {
+		message = message[len(message)-1800:]
+	}
+	return message
+}
+
+func runPrivilegedTransient(timeout time.Duration, suffix, name string, args ...string) string {
+	systemdRun, err := exec.LookPath("systemd-run")
+	if err != nil {
+		return "ERROR: systemd-run не найден; невозможно безопасно выйти из read-only sandbox агента"
+	}
+	unit := fmt.Sprintf("proxmox-dashboard-%s-%d", suffix, time.Now().UnixNano())
+	commandArgs := []string{"--quiet", "--wait", "--collect", "--pipe", "--unit=" + unit, "--setenv=DEBIAN_FRONTEND=noninteractive", name}
+	commandArgs = append(commandArgs, args...)
+	return runAgentCombinedCommand(timeout, systemdRun, commandArgs...)
+}
+
+func scheduleAgentReboot() string {
+	systemdRun, err := exec.LookPath("systemd-run")
+	if err != nil {
+		return "ERROR: systemd-run не найден"
+	}
+	systemctl, err := exec.LookPath("systemctl")
+	if err != nil {
+		return "ERROR: systemctl не найден"
+	}
+	unit := fmt.Sprintf("proxmox-dashboard-reboot-%d", time.Now().UnixNano())
+	output := runAgentCombinedCommand(15*time.Second, systemdRun, "--quiet", "--collect", "--unit="+unit, "--on-active=20s", systemctl, "reboot")
+	if strings.HasPrefix(output, "ERROR: ") {
+		return output
+	}
+	return "Перезагрузка запланирована через 20 секунд; кратковременный offline агента ожидаем"
+}
+
+func updateAgentContainer(target string, recreate bool) string {
+	transport := &http.Transport{DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
+		return (&net.Dialer{}).DialContext(ctx, "unix", "/var/run/docker.sock")
+	}}
+	defer transport.CloseIdleConnections()
+	client := &http.Client{Transport: transport, Timeout: 10 * time.Second}
+	response, err := client.Get("http://docker/containers/" + target + "/json")
+	if err != nil {
+		return "ERROR: " + err.Error()
+	}
+	defer response.Body.Close()
+	var detail struct {
+		Config struct {
+			Labels map[string]string `json:"Labels"`
+		} `json:"Config"`
+	}
+	if response.StatusCode != http.StatusOK || json.NewDecoder(response.Body).Decode(&detail) != nil {
+		return "ERROR: не удалось получить Compose-конфигурацию контейнера"
+	}
+	labels := detail.Config.Labels
+	service := labels["com.docker.compose.service"]
+	workdir := labels["com.docker.compose.project.working_dir"]
+	files := strings.Split(labels["com.docker.compose.project.config_files"], ",")
+	if service == "" || workdir == "" || len(files) == 0 || !filepath.IsAbs(workdir) || !filepath.IsAbs(files[0]) {
+		return "ERROR: безопасное обновление доступно только для Docker Compose"
+	}
+	if _, err = exec.LookPath("docker"); err != nil {
+		return "ERROR: Docker CLI не найден"
+	}
+	commands := [][]string{{"compose", "-f", files[0], "pull", service}}
+	if recreate {
+		commands = append(commands, []string{"compose", "-f", files[0], "up", "-d", "--no-deps", service})
+	}
+	for _, args := range commands {
+		ctx, cancel := context.WithTimeout(context.Background(), 15*time.Minute)
+		command := exec.CommandContext(ctx, "docker", args...)
+		command.Dir = workdir
+		output, commandErr := command.CombinedOutput()
+		cancel()
+		if commandErr != nil {
+			return "ERROR: " + commandErr.Error() + ": " + strings.TrimSpace(string(output))
+		}
+	}
+	if recreate {
+		return "Образ загружен, Compose-контейнер пересоздан"
+	}
+	return "Проверка завершена: образ загружен; если digest изменился, карточка покажет доступное обновление"
+}
+
+func runAgentCombinedCommand(timeout time.Duration, name string, args ...string) string {
+	ctx, cancel := context.WithTimeout(context.Background(), timeout)
+	defer cancel()
+	output, err := exec.CommandContext(ctx, name, args...).CombinedOutput()
+	message := strings.TrimSpace(string(output))
+	if err != nil {
+		return "ERROR: " + err.Error() + ": " + message
+	}
+	return message
+}
+
+func restartAgentContainer(target string) string {
+	transport := &http.Transport{DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
+		return (&net.Dialer{}).DialContext(ctx, "unix", "/var/run/docker.sock")
+	}}
+	defer transport.CloseIdleConnections()
+	client := &http.Client{Transport: transport, Timeout: 35 * time.Second}
+	request, _ := http.NewRequest(http.MethodPost, "http://docker/containers/"+target+"/restart?t=10", bytes.NewReader(nil))
+	response, err := client.Do(request)
+	if err != nil {
+		return "ERROR: " + err.Error()
+	}
+	defer response.Body.Close()
+	body, _ := io.ReadAll(io.LimitReader(response.Body, 4096))
+	if response.StatusCode >= 300 {
+		return "ERROR: Docker вернул " + response.Status + ": " + strings.TrimSpace(string(body))
+	}
+	return "Контейнер перезапущен"
+}

agent_actions_test.go

@@ -0,0 +1,77 @@
+package main
+
+import (
+	"encoding/json"
+	"strings"
+	"testing"
+	"time"
+)
+
+func TestAgentActionAllowlistAndLifecycle(t *testing.T) {
+	store, err := openStore(":memory:")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer store.Close()
+	report, _ := json.Marshal(AgentReport{Services: []AgentService{{Name: "gitea.service"}}, Containers: []AgentContainer{{ID: "abc123", Name: "gitea", State: "running"}, {ID: "old123", Name: "old", State: "exited"}}})
+	_, err = store.db.Exec(`INSERT INTO agents(id,name,secret_hash,enrolled_at,last_seen,report_json) VALUES('a','agent','x',?,?,?)`, time.Now().Unix(), time.Now().Unix(), string(report))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if _, err = store.QueueAgentAction("a", "systemd-restart", "../../bad.service"); err == nil {
+		t.Fatal("unsafe unit accepted")
+	}
+	if _, err = store.QueueAgentAction("a", "docker-update", "gitea"); err != nil {
+		t.Fatalf("known Docker container rejected: %v", err)
+	}
+	if _, err = store.QueueAgentAction("a", "system-update", ""); err != nil {
+		t.Fatalf("system update rejected: %v", err)
+	}
+	if _, err = store.QueueAgentAction("a", "docker-remove", "gitea"); err == nil {
+		t.Fatal("running/unknown-state container accepted for removal")
+	}
+	action, err := store.QueueAgentAction("a", "docker-remove", "old")
+	if err != nil {
+		t.Fatalf("stopped container rejected for removal: %v", err)
+	}
+	commands, err := store.DispatchAgentActions("a")
+	if err != nil || len(commands) != 3 || commands[2].ID != action.ID {
+		t.Fatalf("commands=%#v err=%v", commands, err)
+	}
+	store.CompleteAgentActions("a", []AgentActionResult{{ID: action.ID, Success: true, Message: "ok"}})
+	items, err := store.AgentActions(10)
+	if err != nil || len(items) != 3 || items[0].Status != "completed" {
+		t.Fatalf("items=%#v err=%v", items, err)
+	}
+	if _, err = store.QueueAgentAction("a", "system-reboot", ""); err != nil {
+		t.Fatalf("system reboot rejected: %v", err)
+	}
+	if _, err = store.QueueAgentAction("a", "system-reboot", "now"); err == nil {
+		t.Fatal("system reboot with unexpected target accepted")
+	}
+}
+
+func TestManualServiceLinkGroupsOfflineAlerts(t *testing.T) {
+	metrics := dashboardMetrics{Agents: []ManagedAgent{{ID: "a", Name: "app", VMID: 105, Online: false}}, ServiceLinks: []ServiceLink{{ServiceID: 7, AgentID: "a", VMID: 105}}, Services: ServicesMetrics{Services: []ServiceStatus{{MonitoredService: MonitoredService{ID: 7, Name: "Gitea"}, Endpoints: []ServiceEndpointStatus{{Kind: "local", Up: false}}}}}}
+	alerts := []Alert{{ID: "agent-offline-a-105"}, {ID: "service-7-local"}, {ID: "other"}}
+	grouped := groupLinkedAlerts(metrics, alerts)
+	if len(grouped) != 2 {
+		t.Fatalf("grouped=%#v", grouped)
+	}
+	found := false
+	for _, alert := range grouped {
+		if alert.ID == "group-agent-offline-a-105" {
+			found = true
+		}
+	}
+	if !found {
+		t.Fatalf("group alert missing: %#v", grouped)
+	}
+}
+
+func TestExplainReadOnlySystemUpdateError(t *testing.T) {
+	message := explainSystemUpdateError("ERROR: apt failed: Read-only file system")
+	if !strings.Contains(message, "v0.19.2") {
+		t.Fatalf("read-only failure is not actionable: %s", message)
+	}
+}

agent_diagnostics.go

@@ -0,0 +1,336 @@
+package main
+
+import (
+	"bufio"
+	"encoding/json"
+	"net"
+	"os"
+	"path/filepath"
+	"regexp"
+	"runtime"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"syscall"
+	"time"
+)
+
+type AgentLog struct {
+	Time    int64  `json:"time"`
+	Message string `json:"message"`
+}
+
+type AgentProcess struct {
+	PID         int     `json:"pid"`
+	Name        string  `json:"name"`
+	State       string  `json:"state"`
+	CPUPercent  float64 `json:"cpuPercent"`
+	CPUCores    float64 `json:"cpuCores"`
+	HostPercent float64 `json:"hostPercent"`
+	MemoryBytes uint64  `json:"memoryBytes"`
+	ReadBytes   uint64  `json:"readBytes"`
+	WriteBytes  uint64  `json:"writeBytes"`
+	Elapsed     int64   `json:"elapsedSeconds"`
+	Command     string  `json:"command"`
+}
+
+type AgentFilesystem struct {
+	Device        string  `json:"device"`
+	Mountpoint    string  `json:"mountpoint"`
+	Type          string  `json:"type"`
+	TotalBytes    uint64  `json:"totalBytes"`
+	UsedBytes     uint64  `json:"usedBytes"`
+	UsedPercent   float64 `json:"usedPercent"`
+	InodesTotal   uint64  `json:"inodesTotal"`
+	InodesUsed    uint64  `json:"inodesUsed"`
+	InodesPercent float64 `json:"inodesPercent"`
+}
+
+type AgentListenPort struct {
+	Protocol string `json:"protocol"`
+	Address  string `json:"address"`
+	Port     int    `json:"port"`
+	PID      int    `json:"pid,omitempty"`
+	Process  string `json:"process,omitempty"`
+}
+
+type AgentPressureValue struct {
+	Avg10  float64 `json:"avg10"`
+	Avg60  float64 `json:"avg60"`
+	Avg300 float64 `json:"avg300"`
+}
+
+type AgentPressure struct {
+	CPU    AgentPressureValue `json:"cpu"`
+	Memory AgentPressureValue `json:"memory"`
+	IO     AgentPressureValue `json:"io"`
+}
+
+var agentOOMCache struct {
+	sync.Mutex
+	value []AgentLog
+	at    time.Time
+}
+
+func compactSystemdExec(value string) string {
+	if pathAt := strings.Index(value, "path="); pathAt >= 0 {
+		value = value[pathAt+5:]
+	}
+	if end := strings.IndexAny(value, " ;}"); end > 0 {
+		value = value[:end]
+	}
+	if len(value) > 240 {
+		value = value[:240]
+	}
+	return strings.TrimSpace(value)
+}
+
+func limitedFields(value string, limit int) []string {
+	fields := strings.Fields(value)
+	if len(fields) > limit {
+		fields = fields[:limit]
+	}
+	return fields
+}
+
+func appendUniqueInt(items []int, value int) []int {
+	for _, item := range items {
+		if item == value {
+			return items
+		}
+	}
+	return append(items, value)
+}
+
+func collectAgentJournal(unit string, limit int, since string) []AgentLog {
+	args := []string{"--no-pager", "-o", "json", "--reverse", "-n", strconv.Itoa(limit), "--since", since}
+	if unit != "" {
+		args = append(args, "-u", unit)
+	}
+	output := runAgentCommand(6*time.Second, "journalctl", args...)
+	result := []AgentLog{}
+	for _, line := range strings.Split(output, "\n") {
+		var entry struct {
+			Message   any    `json:"MESSAGE"`
+			Timestamp string `json:"__REALTIME_TIMESTAMP"`
+		}
+		if json.Unmarshal([]byte(line), &entry) != nil {
+			continue
+		}
+		message, ok := entry.Message.(string)
+		if !ok || strings.TrimSpace(message) == "" {
+			continue
+		}
+		microseconds, _ := strconv.ParseInt(entry.Timestamp, 10, 64)
+		result = append(result, AgentLog{Time: microseconds / 1_000_000, Message: message})
+	}
+	return result
+}
+
+func cachedAgentOOMKills() []AgentLog {
+	agentOOMCache.Lock()
+	defer agentOOMCache.Unlock()
+	if time.Since(agentOOMCache.at) < time.Minute {
+		return agentOOMCache.value
+	}
+	output := runAgentCommand(6*time.Second, "journalctl", "-k", "--since", "24 hours ago", "--reverse", "--no-pager", "-o", "json", "-g", "Out of memory|Killed process")
+	result := []AgentLog{}
+	for _, line := range strings.Split(output, "\n") {
+		var entry struct {
+			Message   any    `json:"MESSAGE"`
+			Timestamp string `json:"__REALTIME_TIMESTAMP"`
+		}
+		if json.Unmarshal([]byte(line), &entry) != nil {
+			continue
+		}
+		message, ok := entry.Message.(string)
+		if !ok {
+			continue
+		}
+		microseconds, _ := strconv.ParseInt(entry.Timestamp, 10, 64)
+		result = append(result, AgentLog{Time: microseconds / 1_000_000, Message: message})
+		if len(result) >= 20 {
+			break
+		}
+	}
+	agentOOMCache.value, agentOOMCache.at = result, time.Now()
+	return result
+}
+
+func collectAgentProcesses() []AgentProcess {
+	result := []AgentProcess{}
+	seen := map[int]bool{}
+	for _, sortBy := range []string{"-pcpu", "-rss"} {
+		output := runAgentCommand(5*time.Second, "ps", "-eo", "pid=,comm=,state=,pcpu=,rss=,etimes=,args=", "--sort="+sortBy)
+		added := 0
+		for _, line := range strings.Split(output, "\n") {
+			fields := strings.Fields(line)
+			if len(fields) < 7 {
+				continue
+			}
+			pid, err := strconv.Atoi(fields[0])
+			if err != nil || seen[pid] {
+				continue
+			}
+			seen[pid] = true
+			cpu, _ := strconv.ParseFloat(fields[3], 64)
+			rss, _ := strconv.ParseUint(fields[4], 10, 64)
+			elapsed, _ := strconv.ParseInt(fields[5], 10, 64)
+			cpuCount := runtime.NumCPU()
+			if cpuCount < 1 {
+				cpuCount = 1
+			}
+			item := AgentProcess{PID: pid, Name: fields[1], State: fields[2], CPUPercent: cpu, CPUCores: cpu / 100, HostPercent: cpu / float64(cpuCount), MemoryBytes: rss * 1024, Elapsed: elapsed, Command: strings.Join(fields[6:], " ")}
+			item.ReadBytes, item.WriteBytes = readAgentProcessIO(pid)
+			result = append(result, item)
+			added++
+			if added >= 12 || len(result) >= 24 {
+				break
+			}
+		}
+	}
+	return result
+}
+
+func readAgentProcessIO(pid int) (uint64, uint64) {
+	data, err := os.ReadFile(filepath.Join("/proc", strconv.Itoa(pid), "io"))
+	if err != nil {
+		return 0, 0
+	}
+	values := map[string]uint64{}
+	for _, line := range strings.Split(string(data), "\n") {
+		fields := strings.Fields(line)
+		if len(fields) != 2 {
+			continue
+		}
+		values[strings.TrimSuffix(fields[0], ":")], _ = strconv.ParseUint(fields[1], 10, 64)
+	}
+	return values["read_bytes"], values["write_bytes"]
+}
+
+func collectAgentFilesystems() []AgentFilesystem {
+	file, err := os.Open("/proc/self/mounts")
+	if err != nil {
+		return nil
+	}
+	defer file.Close()
+	excluded := map[string]bool{"proc": true, "sysfs": true, "devtmpfs": true, "devpts": true, "tmpfs": true, "cgroup": true, "cgroup2": true, "overlay": true, "squashfs": true, "nsfs": true, "tracefs": true, "debugfs": true, "securityfs": true, "pstore": true, "mqueue": true, "hugetlbfs": true, "fusectl": true, "configfs": true, "autofs": true, "rpc_pipefs": true}
+	seen := map[string]bool{}
+	result := []AgentFilesystem{}
+	scanner := bufio.NewScanner(file)
+	for scanner.Scan() {
+		fields := strings.Fields(scanner.Text())
+		if len(fields) < 3 || excluded[fields[2]] || seen[fields[1]] {
+			continue
+		}
+		mountpoint := strings.ReplaceAll(fields[1], `\040`, " ")
+		var stat syscall.Statfs_t
+		if syscall.Statfs(mountpoint, &stat) != nil || stat.Blocks == 0 {
+			continue
+		}
+		seen[mountpoint] = true
+		total := stat.Blocks * uint64(stat.Bsize)
+		free := stat.Bavail * uint64(stat.Bsize)
+		used := total - free
+		item := AgentFilesystem{Device: fields[0], Mountpoint: mountpoint, Type: fields[2], TotalBytes: total, UsedBytes: used, UsedPercent: float64(used) / float64(total) * 100, InodesTotal: stat.Files}
+		if stat.Files > stat.Ffree {
+			item.InodesUsed = stat.Files - stat.Ffree
+			item.InodesPercent = float64(item.InodesUsed) / float64(stat.Files) * 100
+		}
+		result = append(result, item)
+		if len(result) >= 30 {
+			break
+		}
+	}
+	sort.Slice(result, func(i, j int) bool { return result[i].Mountpoint < result[j].Mountpoint })
+	return result
+}
+
+var ssPIDPattern = regexp.MustCompile(`pid=([0-9]+)`)
+var ssNamePattern = regexp.MustCompile(`users:\(\(\"([^\"]+)\"`)
+
+func collectAgentListenPorts() []AgentListenPort {
+	output := runAgentCommand(5*time.Second, "ss", "-H", "-lntup")
+	result := []AgentListenPort{}
+	seen := map[string]bool{}
+	for _, line := range strings.Split(output, "\n") {
+		fields := strings.Fields(line)
+		if len(fields) < 5 {
+			continue
+		}
+		protocol := strings.ToLower(fields[0])
+		local := ""
+		for _, field := range fields[1:] {
+			if strings.Contains(field, ":") && !strings.Contains(field, "users:") {
+				local = field
+				break
+			}
+		}
+		if local == "" {
+			continue
+		}
+		address, portText, err := net.SplitHostPort(local)
+		if err != nil {
+			idx := strings.LastIndex(local, ":")
+			if idx < 0 {
+				continue
+			}
+			address, portText = local[:idx], local[idx+1:]
+		}
+		port, _ := strconv.Atoi(portText)
+		if port <= 0 {
+			continue
+		}
+		pid := 0
+		if match := ssPIDPattern.FindStringSubmatch(line); len(match) > 1 {
+			pid, _ = strconv.Atoi(match[1])
+		}
+		process := ""
+		if match := ssNamePattern.FindStringSubmatch(line); len(match) > 1 {
+			process = match[1]
+		}
+		key := protocol + "|" + address + "|" + portText + "|" + strconv.Itoa(pid)
+		if seen[key] {
+			continue
+		}
+		seen[key] = true
+		result = append(result, AgentListenPort{Protocol: protocol, Address: address, Port: port, PID: pid, Process: process})
+	}
+	sort.Slice(result, func(i, j int) bool {
+		if result[i].Port == result[j].Port {
+			return result[i].Protocol < result[j].Protocol
+		}
+		return result[i].Port < result[j].Port
+	})
+	return result
+}
+
+func collectAgentPressure() AgentPressure {
+	return AgentPressure{CPU: readPressure("/proc/pressure/cpu"), Memory: readPressure("/proc/pressure/memory"), IO: readPressure("/proc/pressure/io")}
+}
+
+func readPressure(path string) AgentPressureValue {
+	data, err := os.ReadFile(path)
+	if err != nil {
+		return AgentPressureValue{}
+	}
+	line := strings.SplitN(string(data), "\n", 2)[0]
+	value := AgentPressureValue{}
+	for _, field := range strings.Fields(line) {
+		key, raw, ok := strings.Cut(field, "=")
+		if !ok {
+			continue
+		}
+		number, _ := strconv.ParseFloat(raw, 64)
+		switch key {
+		case "avg10":
+			value.Avg10 = number
+		case "avg60":
+			value.Avg60 = number
+		case "avg300":
+			value.Avg300 = number
+		}
+	}
+	return value
+}

agent_network.go

@@ -0,0 +1,271 @@
+package main
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net"
+	"net/http"
+	"os"
+	"os/exec"
+	"runtime"
+	"strconv"
+	"strings"
+	"time"
+)
+
+// Bounded command output and deadline keep large conntrack tables from exhausting the agent.
+type networkBuffer struct {
+	bytes.Buffer
+	limit int
+}
+
+func (b *networkBuffer) Write(p []byte) (int, error) {
+	if len(p) > b.limit-b.Len() {
+		return 0, errors.New("output limit exceeded")
+	}
+	return b.Buffer.Write(p)
+}
+func networkCommand(ctx context.Context, name string, args ...string) ([]byte, error) {
+	ctx, cancel := context.WithTimeout(ctx, 4*time.Second)
+	defer cancel()
+	cmd := exec.CommandContext(ctx, name, args...)
+	out := &networkBuffer{limit: 8 << 20}
+	stderr := &networkBuffer{limit: 4096}
+	cmd.Stdout = out
+	cmd.Stderr = stderr
+	if err := cmd.Run(); err != nil {
+		return nil, fmt.Errorf("%s: %v %s", name, err, trafficText(stderr.String(), 240))
+	}
+	return out.Bytes(), nil
+}
+func parseNetworkFlow(line string) (NetworkFlow, bool) {
+	var f NetworkFlow
+	fields := strings.Fields(line)
+	protoIndex := -1
+	for i, v := range fields {
+		if v == "tcp" || v == "udp" || v == "icmp" || v == "icmpv6" {
+			f.Protocol = v
+			protoIndex = i
+			break
+		}
+	}
+	if protoIndex < 0 {
+		return f, false
+	}
+	values := map[string][]string{}
+	for _, v := range fields[protoIndex+1:] {
+		if k, val, ok := strings.Cut(v, "="); ok {
+			values[k] = append(values[k], val)
+		}
+	}
+	get := func(k string, i int) string {
+		if len(values[k]) > i {
+			return values[k][i]
+		}
+		return ""
+	}
+	integer := func(k string, i int) int { n, _ := strconv.Atoi(get(k, i)); return n }
+	number := func(k string, i int) uint64 { n, _ := strconv.ParseUint(get(k, i), 10, 64); return n }
+	f.Source = get("src", 0)
+	f.Destination = get("dst", 0)
+	f.ReplySource = get("src", 1)
+	f.ReplyDestination = get("dst", 1)
+	if net.ParseIP(f.Source) == nil || net.ParseIP(f.Destination) == nil || net.ParseIP(f.ReplySource) == nil || net.ParseIP(f.ReplyDestination) == nil {
+		return f, false
+	}
+	f.SourcePort = integer("sport", 0)
+	f.DestinationPort = integer("dport", 0)
+	f.ReplySourcePort = integer("sport", 1)
+	f.ReplyDestinationPort = integer("dport", 1)
+	f.State = "TRACKED"
+	if f.Protocol == "tcp" && len(fields) > protoIndex+3 && !strings.Contains(fields[protoIndex+3], "=") {
+		f.State = fields[protoIndex+3]
+	}
+	if strings.Contains(line, "[UNREPLIED]") {
+		f.State = "UNREPLIED"
+		if f.Protocol == "tcp" && len(fields) > protoIndex+3 {
+			f.State = fields[protoIndex+3]
+		}
+	}
+	if f.Protocol == "udp" && strings.Contains(line, "[ASSURED]") {
+		f.State = "ASSURED"
+	}
+	f.Accounting = len(values["bytes"]) == 2 && len(values["packets"]) == 2
+	f.Bytes = number("bytes", 0)
+	f.ReplyBytes = number("bytes", 1)
+	f.Packets = number("packets", 0)
+	f.ReplyPackets = number("packets", 1)
+	f.Zone = get("zone", 0)
+	return f, true
+}
+func appendNetworkFlows(snapshot *NetworkSnapshot, raw []byte) {
+	for _, line := range strings.Split(string(raw), "\n") {
+		if strings.TrimSpace(line) == "" {
+			continue
+		}
+		f, ok := parseNetworkFlow(line)
+		if !ok {
+			if strings.Contains(line, "src=") {
+				snapshot.ParseErrors++
+			}
+			continue
+		}
+		if len(snapshot.Flows) >= 5000 {
+			snapshot.Limited = true
+			continue
+		}
+		snapshot.Flows = append(snapshot.Flows, f)
+	}
+}
+
+// Only counters immediately before a terminal drop/reject in a CrowdSec table/chain
+// are reported as blocked packets. General chain and set counters are not summed.
+func parseFirewallCounters(raw []byte) ([]FirewallCounter, error) {
+	var doc struct {
+		NFTables []struct {
+			Rule *struct {
+				Family string                       `json:"family"`
+				Table  string                       `json:"table"`
+				Chain  string                       `json:"chain"`
+				Handle int                          `json:"handle"`
+				Expr   []map[string]json.RawMessage `json:"expr"`
+			} `json:"rule"`
+		} `json:"nftables"`
+	}
+	if err := json.Unmarshal(raw, &doc); err != nil {
+		return nil, err
+	}
+	result := []FirewallCounter{}
+	for _, item := range doc.NFTables {
+		r := item.Rule
+		if r == nil || !strings.Contains(strings.ToLower(r.Table+" "+r.Chain), "crowdsec") {
+			continue
+		}
+		if len(r.Expr) < 2 {
+			continue
+		}
+		last := r.Expr[len(r.Expr)-1]
+		_, drop := last["drop"]
+		_, reject := last["reject"]
+		if !drop && !reject {
+			continue
+		}
+		counter, ok := r.Expr[len(r.Expr)-2]["counter"]
+		if !ok {
+			continue
+		}
+		var c struct {
+			Packets uint64 `json:"packets"`
+			Bytes   uint64 `json:"bytes"`
+		}
+		if json.Unmarshal(counter, &c) != nil || len(counter) == 0 || counter[0] != '{' {
+			continue
+		}
+		result = append(result, FirewallCounter{Name: fmt.Sprintf("%s/%s/%s #%d", r.Family, r.Table, r.Chain, r.Handle), Packets: c.Packets, Bytes: c.Bytes})
+		if len(result) == 128 {
+			break
+		}
+	}
+	return result, nil
+}
+func collectNetwork(ctx context.Context) NetworkSnapshot {
+	s := NetworkSnapshot{At: time.Now().Unix(), Flows: []NetworkFlow{}}
+	if runtime.GOOS != "linux" {
+		s.Error = "Сбор conntrack поддерживается только на Linux"
+		return s
+	}
+	// Proc contains both address families when available. Modern kernels often expose only netlink.
+	for _, path := range []string{"/proc/net/nf_conntrack", "/proc/net/ip_conntrack"} {
+		f, err := os.Open(path)
+		if err != nil {
+			continue
+		}
+		raw, err := io.ReadAll(io.LimitReader(f, (8<<20)+1))
+		f.Close()
+		if err != nil {
+			s.Error = err.Error()
+			return s
+		}
+		if len(raw) > 8<<20 {
+			s.Error = "Таблица conntrack превышает лимит чтения 8 MiB"
+			return s
+		}
+		appendNetworkFlows(&s, raw)
+		return s
+	}
+	for _, family := range []string{"ipv4", "ipv6"} {
+		raw, err := networkCommand(ctx, "conntrack", "-L", "-f", family, "-o", "extended")
+		if err != nil {
+			s.Error += "Conntrack " + family + ": нужен пакет conntrack и права root/CAP_NET_ADMIN. " + err.Error() + "; "
+			continue
+		}
+		appendNetworkFlows(&s, raw)
+	}
+	return s
+}
+func startAgentNetwork(config agentConfig) func() {
+	ctx, cancel := context.WithCancel(context.Background())
+	done := make(chan struct{})
+	go func() {
+		defer close(done)
+		client := &http.Client{Timeout: 10 * time.Second}
+		ticker := time.NewTicker(5 * time.Second)
+		defer ticker.Stop()
+		var firewall []FirewallCounter
+		var firewallAt int64
+		var firewallErr string
+		lastFirewall := time.Time{}
+		for {
+			if ctx.Err() != nil {
+				return
+			}
+			snapshot := NetworkSnapshot{At: time.Now().Unix(), Flows: []NetworkFlow{}}
+			if message := ensureNetworkDependency(); message != "" {
+				snapshot.Error = message
+			} else {
+				snapshot = collectNetwork(ctx)
+			}
+			if time.Since(lastFirewall) >= 30*time.Second {
+				lastFirewall = time.Now()
+				raw, err := networkCommand(ctx, "nft", "-j", "list", "ruleset")
+				firewall = nil
+				firewallAt = 0
+				firewallErr = ""
+				if err == nil {
+					firewall, err = parseFirewallCounters(raw)
+				}
+				if err != nil {
+					firewallErr = "Счётчики nftables недоступны: " + err.Error()
+				} else if len(firewall) == 0 {
+					firewallErr = "В правилах CrowdSec нет поддерживаемых счётчиков перед drop/reject. Это не означает отсутствие блокировок."
+				} else {
+					firewallAt = time.Now().Unix()
+				}
+			}
+			snapshot.Firewall = firewall
+			snapshot.FirewallAt = firewallAt
+			snapshot.FirewallError = firewallErr
+			data, _ := json.Marshal(snapshot)
+			req, err := http.NewRequestWithContext(ctx, "POST", config.Server+"/api/agent/network", bytes.NewReader(data))
+			if err != nil {
+				return
+			}
+			req.Header.Set("Authorization", "Bearer "+config.ID+"."+config.Secret)
+			req.Header.Set("Content-Type", "application/json")
+			if res, err := client.Do(req); err == nil {
+				io.Copy(io.Discard, io.LimitReader(res.Body, 4096))
+				res.Body.Close()
+			}
+			select {
+			case <-ctx.Done():
+				return
+			case <-ticker.C:
+			}
+		}
+	}()
+	return func() { cancel(); <-done }
+}

agent_network_setup.go

@@ -0,0 +1,89 @@
+package main
+
+import (
+	"fmt"
+	"os"
+	"os/exec"
+	"runtime"
+	"strings"
+	"sync"
+	"time"
+)
+
+// Package work runs in a transient systemd unit, as do existing agent update
+// actions. It must not inherit ProtectSystem=strict from the reporting agent.
+// A started package transaction is allowed to finish if collection is disabled.
+type networkDependencyState struct {
+	sync.Mutex
+	running     bool
+	nextAttempt time.Time
+	message     string
+}
+
+var networkDependency networkDependencyState
+
+func (d *networkDependencyState) ensure(now time.Time, present bool, install func() error) string {
+	d.Lock()
+	defer d.Unlock()
+	if present {
+		return ""
+	}
+	if d.running {
+		return "Автоустановка conntrack… Сбор начнётся после установки пакета."
+	}
+	if now.Before(d.nextAttempt) {
+		return d.message + " Повторная попытка не раньше " + d.nextAttempt.Format("15:04:05") + " (время агента)."
+	}
+	d.running = true
+	d.nextAttempt = now.Add(30 * time.Minute)
+	go func() {
+		err := install()
+		d.Lock()
+		defer d.Unlock()
+		d.running = false
+		if err != nil {
+			d.message = "Не удалось установить conntrack: " + trafficText(err.Error(), 400)
+		} else {
+			d.message = "Установка завершена; ожидается появление conntrack в PATH"
+		}
+	}()
+	return "Автоустановка conntrack… Сбор начнётся после установки пакета."
+}
+func ensureNetworkDependency() string {
+	if runtime.GOOS != "linux" {
+		return "Сбор conntrack поддерживается только на Linux"
+	}
+	_, err := exec.LookPath("conntrack")
+	return networkDependency.ensure(time.Now(), err == nil, installNetworkDependency)
+}
+func installNetworkDependency() error {
+	if os.Geteuid() != 0 {
+		return fmt.Errorf("для автоустановки нужны права root")
+	}
+	apt, err := exec.LookPath("apt-get")
+	if err != nil {
+		return fmt.Errorf("автоустановка поддерживается на Debian/Ubuntu с apt-get; установите conntrack вручную")
+	}
+	run := func(suffix string, args ...string) error {
+		out := runPrivilegedTransient(10*time.Minute, suffix, apt, args...)
+		if strings.HasPrefix(out, "ERROR: ") {
+			return fmt.Errorf("%s", explainSystemUpdateError(out))
+		}
+		return nil
+	}
+	return installConntrackPackage(run, func() bool { _, err := exec.LookPath("conntrack"); return err == nil })
+}
+func installConntrackPackage(run func(string, ...string) error, present func() bool) error {
+	// No upgrade, autoremove, firewall changes, or runtime sysctl changes.
+	options := []string{"-o", "DPkg::Lock::Timeout=30", "-o", "Acquire::Retries=1", "-o", "Acquire::http::Timeout=20", "-o", "Acquire::https::Timeout=20"}
+	if err := run("conntrack-lists", append(append([]string{}, options...), "update")...); err != nil {
+		return err
+	}
+	if err := run("conntrack-install", append(append([]string{}, options...), "-y", "--no-install-recommends", "--no-remove", "install", "conntrack")...); err != nil {
+		return err
+	}
+	if !present() {
+		return fmt.Errorf("apt-get завершился, но conntrack не найден в PATH")
+	}
+	return nil
+}

agent_test.go

@@ -0,0 +1,152 @@
+package main
+
+import (
+	"os"
+	"testing"
+	"time"
+)
+
+func TestAgentEnrollmentTokenIsSingleUse(t *testing.T) {
+	store, err := openStore(":memory:")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer store.Close()
+	token, err := store.CreateAgentEnrollmentToken()
+	if err != nil {
+		t.Fatal(err)
+	}
+	credentials, err := store.EnrollAgent(agentEnrollmentRequest{Token: token, Name: "docker-01", Hostname: "vm105", VMID: 105})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if credentials.ID == "" || credentials.Secret == "" {
+		t.Fatal("credentials were not generated")
+	}
+	if _, err = store.EnrollAgent(agentEnrollmentRequest{Token: token, Hostname: "duplicate"}); err == nil {
+		t.Fatal("token was accepted twice")
+	}
+}
+
+func TestCurrentAgentUpdateManifestMatchesExecutable(t *testing.T) {
+	manifest, path, err := currentAgentUpdate()
+	if err != nil {
+		t.Fatal(err)
+	}
+	if manifest.URL != "/api/agent/binary" || len(manifest.SHA256) != 64 {
+		t.Fatalf("unexpected manifest: %#v", manifest)
+	}
+	if _, err = os.Stat(path); err != nil {
+		t.Fatal(err)
+	}
+}
+
+func TestAgentOnlyInstallsNewerVersion(t *testing.T) {
+	if !newerAgentVersion("v0.14.0", "v0.13.0") {
+		t.Fatal("new version was not detected")
+	}
+	if newerAgentVersion("v0.12.0", "v0.13.0") {
+		t.Fatal("downgrade was allowed")
+	}
+	if newerAgentVersion("v0.13.0", "v0.13.0") {
+		t.Fatal("same version was offered")
+	}
+}
+
+func TestShortDockerDigest(t *testing.T) {
+	if got := shortDockerDigest("sha256:1234567890abcdef"); got != "1234567890ab" {
+		t.Fatalf("unexpected short digest: %q", got)
+	}
+}
+
+func TestParseAgentIssues(t *testing.T) {
+	issues := parseAgentIssues(`{"MESSAGE":"disk error","_SYSTEMD_UNIT":"worker.service","__REALTIME_TIMESTAMP":"1722960000000000"}` + "\n" + `{"MESSAGE":"network error","__REALTIME_TIMESTAMP":"1722960001000000"}`)
+	if len(issues) != 2 || issues[0].Unit != "worker.service" || issues[1].Unit != "system" || issues[0].Time != 1722960000 {
+		t.Fatalf("unexpected issues: %#v", issues)
+	}
+}
+
+func TestClassifyAgentService(t *testing.T) {
+	cases := []struct{ path, pkg, priority, want string }{{"/lib/systemd/system/cron.service", "cron", "important", "system"}, {"/lib/systemd/system/jellyfin.service", "jellyfin", "optional", "added"}, {"/etc/systemd/system/my.service", "", "", "local"}, {"/run/systemd/system/transient.service", "", "", "runtime"}}
+	for _, tc := range cases {
+		if got := classifyAgentService(tc.path, tc.pkg, tc.priority); got != tc.want {
+			t.Errorf("%s: got %s want %s", tc.path, got, tc.want)
+		}
+	}
+}
+
+func TestAgentServiceDiagnosticsHelpers(t *testing.T) {
+	if got := compactSystemdExec(`{ path=/usr/bin/example ; argv[]=/usr/bin/example --serve ; }`); got != "/usr/bin/example" {
+		t.Fatalf("unexpected executable: %q", got)
+	}
+	fields := limitedFields("a b c d", 2)
+	if len(fields) != 2 || fields[0] != "a" || fields[1] != "b" {
+		t.Fatalf("unexpected limited fields: %#v", fields)
+	}
+	ports := appendUniqueInt([]int{80}, 80)
+	ports = appendUniqueInt(ports, 443)
+	if len(ports) != 2 || ports[1] != 443 {
+		t.Fatalf("unexpected ports: %#v", ports)
+	}
+}
+
+func TestAgentReportAuthenticationAndOfflineState(t *testing.T) {
+	store, err := openStore(":memory:")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer store.Close()
+	token, _ := store.CreateAgentEnrollmentToken()
+	credentials, _ := store.EnrollAgent(agentEnrollmentRequest{Token: token, Name: "docker", Hostname: "vm", VMID: 101})
+	report := AgentReport{Hostname: "vm", Version: "test", DockerAvailable: true, Containers: []AgentContainer{{ID: "abc", Name: "web", State: "running"}}}
+	if err = store.SaveAgentReport(credentials.ID, "wrong", "127.0.0.1", report); err == nil {
+		t.Fatal("invalid secret accepted")
+	}
+	if err = store.SaveAgentReport(credentials.ID, credentials.Secret, "127.0.0.1", report); err != nil {
+		t.Fatal(err)
+	}
+	agents, err := store.Agents()
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(agents) != 1 || !agents[0].Online || len(agents[0].Report.Containers) != 1 {
+		t.Fatalf("unexpected agents: %#v", agents)
+	}
+	_, _ = store.db.Exec(`UPDATE agents SET last_seen=? WHERE id=?`, time.Now().Add(-time.Minute).Unix(), credentials.ID)
+	agents, _ = store.Agents()
+	if agents[0].Online {
+		t.Fatal("stale agent is online")
+	}
+}
+
+func TestAgentProblemsBecomeMaintenanceAwareAlerts(t *testing.T) {
+	metrics := dashboardMetrics{Agents: []ManagedAgent{{ID: "node", Name: "app", VMID: 105, Online: true, Report: AgentReport{
+		Services:   []AgentService{{Name: "postgres.service", State: "failed", SubState: "failed"}},
+		Containers: []AgentContainer{{ID: "dead", Name: "web", State: "exited", Status: "Exited (1)"}}, DockerAvailable: true,
+	}}}}
+	alerts := evaluateAlerts(metrics, defaultThresholds())
+	window := MaintenanceWindow{Active: true, VMIDs: []int{105}}
+	active, planned := splitMaintenanceAlerts(alerts, window)
+	if len(planned) != 2 || len(active) != 0 {
+		t.Fatalf("active=%d planned=%d alerts=%#v", len(active), len(planned), alerts)
+	}
+}
+
+func TestAgentDiagnosticsBecomeAlerts(t *testing.T) {
+	metrics := dashboardMetrics{Agents: []ManagedAgent{{ID: "node", Name: "app", VMID: 105, Online: true, Report: AgentReport{
+		Services:    []AgentService{{Name: "web.service", State: "active", Flapping: true, Restarts: 8, StartedAt: time.Now().Unix()}},
+		Filesystems: []AgentFilesystem{{Mountpoint: "/data", TotalBytes: 1000, UsedBytes: 950, UsedPercent: 95}},
+	}}}}
+	alerts := evaluateAlerts(metrics, defaultThresholds())
+	wanted := map[string]bool{"agent-flapping-node-web.service-105": false, "agent-filesystem-node-/data-105": false}
+	for _, alert := range alerts {
+		if _, ok := wanted[alert.ID]; ok {
+			wanted[alert.ID] = true
+		}
+	}
+	for id, found := range wanted {
+		if !found {
+			t.Fatalf("missing alert %s in %#v", id, alerts)
+		}
+	}
+}

agent_traffic.go

@@ -0,0 +1,387 @@
+package main
+
+import (
+	"bufio"
+	"bytes"
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net"
+	"net/http"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"regexp"
+	"sort"
+	"strconv"
+	"strings"
+	"syscall"
+	"time"
+)
+
+var npmTrafficPattern = regexp.MustCompile(`^\[([^]]+)\] .*? (\d{3}) - ([A-Z]+) (?:https?|HTTP) (\S+) "([^"]*)" \[Client ([^]]+)\]`)
+
+func parseNPMTraffic(line string) (TrafficEvent, bool) {
+	m := npmTrafficPattern.FindStringSubmatch(line)
+	if m == nil {
+		return TrafficEvent{}, false
+	}
+	ip := net.ParseIP(m[6])
+	if ip == nil {
+		return TrafficEvent{}, false
+	}
+	ts, err := time.Parse("02/Jan/2006:15:04:05 -0700", m[1])
+	if err != nil {
+		return TrafficEvent{}, false
+	}
+	status, _ := strconv.Atoi(m[2])
+	return TrafficEvent{Time: ts.Unix(), Kind: "request", IP: ip.String(), Domain: trafficText(m[4], 253), Method: m[3], Path: trafficPath(m[5]), Status: status}, true
+}
+
+type trafficCursor struct {
+	Offset     int64 `json:"offset"`
+	Generation int64 `json:"generation"`
+	Seen       int64 `json:"seen"`
+}
+type trafficAgentState struct {
+	BacklogBytes int64                    `json:"backlogBytes"`
+	Initialized  bool                     `json:"initialized"`
+	StartedAt    int64                    `json:"startedAt"`
+	Cursors      map[string]trafficCursor `json:"cursors"`
+	Pending      []TrafficEvent           `json:"pending"`
+	ParseErrors  uint64                   `json:"parseErrors"`
+	CrowdSeen    map[string]int64         `json:"crowdSeen"`
+}
+
+func (state *trafficAgentState) readLogs(dir string) (int, error) {
+	state.BacklogBytes = 0
+	paths, err := filepath.Glob(filepath.Join(dir, "*_access.log*"))
+	if err != nil {
+		return 0, err
+	}
+	sort.Strings(paths)
+	files := 0
+	budget := int64(1 << 20)
+	now := time.Now().Unix()
+	var firstErr error
+	for _, path := range paths {
+		if strings.HasSuffix(path, ".gz") {
+			continue
+		}
+		f, err := os.Open(path)
+		if err != nil {
+			firstErr = err
+			continue
+		}
+		info, err := f.Stat()
+		if err != nil || !info.Mode().IsRegular() {
+			f.Close()
+			continue
+		}
+		st, ok := info.Sys().(*syscall.Stat_t)
+		if !ok {
+			f.Close()
+			continue
+		}
+		active := strings.HasSuffix(path, "_access.log")
+		if active {
+			files++
+		}
+		key := fmt.Sprintf("%d:%d", st.Dev, st.Ino)
+		cursor, known := state.Cursors[key]
+		if !known {
+			if !state.Initialized || !active {
+				cursor.Offset = info.Size()
+			}
+			cursor.Generation = now
+		}
+		cursor.Seen = now
+		if info.Size() < cursor.Offset {
+			cursor.Offset = 0
+			cursor.Generation++
+		}
+		if len(state.Pending) < 400 && budget > 0 {
+			_, err = f.Seek(cursor.Offset, io.SeekStart)
+			if err == nil {
+				reader := bufio.NewReaderSize(io.LimitReader(f, budget), 64<<10)
+				for len(state.Pending) < 400 && budget > 0 {
+					line, readErr := reader.ReadString('\n')
+					// Keep an unfinished line for the next poll. Bound malformed huge lines.
+					if readErr != nil {
+						if len(line) >= 64<<10 {
+							cursor.Offset += int64(len(line))
+							state.ParseErrors++
+						}
+						break
+					}
+					offset := cursor.Offset
+					cursor.Offset += int64(len(line))
+					budget -= int64(len(line))
+					if len(line) > 64<<10 {
+						state.ParseErrors++
+						continue
+					}
+					e, ok := parseNPMTraffic(line)
+					if !ok {
+						state.ParseErrors++
+						continue
+					}
+					if e.Time < now-86400 {
+						continue
+					}
+					e.ID = fmt.Sprintf("npm:%s:%d:%d", key, cursor.Generation, offset)
+					state.Pending = append(state.Pending, e)
+				}
+			} else {
+				firstErr = err
+			}
+		}
+		if info.Size() > cursor.Offset {
+			state.BacklogBytes += info.Size() - cursor.Offset
+		}
+		state.Cursors[key] = cursor
+		f.Close()
+	}
+	for key, cursor := range state.Cursors {
+		if cursor.Seen < now-7*86400 {
+			delete(state.Cursors, key)
+		}
+	}
+	if files > 0 {
+		state.Initialized = true
+	}
+	if firstErr != nil {
+		return files, errors.New("Не удалось прочитать часть access-логов NPM")
+	}
+	if files == 0 {
+		return files, errors.New("Access-логи NPM не найдены")
+	}
+	return files, nil
+}
+
+type crowdTrafficAlert struct {
+	ID        int64  `json:"id"`
+	UUID      string `json:"uuid"`
+	CreatedAt string `json:"created_at"`
+	Scenario  string `json:"scenario"`
+	Simulated bool   `json:"simulated"`
+	Source    struct {
+		IP string `json:"ip"`
+	} `json:"source"`
+	Events []struct {
+		Meta []struct {
+			Key   string `json:"key"`
+			Value string `json:"value"`
+		} `json:"meta"`
+	} `json:"events"`
+	Decisions []struct {
+		ID        int64  `json:"id"`
+		Type      string `json:"type"`
+		Value     string `json:"value"`
+		Simulated bool   `json:"simulated"`
+	} `json:"decisions"`
+}
+
+func parseCrowdTraffic(data []byte) ([]TrafficEvent, error) {
+	var alerts []crowdTrafficAlert
+	if err := json.Unmarshal(data, &alerts); err != nil {
+		return nil, err
+	}
+	out := []TrafficEvent{}
+	for _, a := range alerts {
+		if a.Simulated || net.ParseIP(a.Source.IP) == nil {
+			continue
+		}
+		ts, err := time.Parse(time.RFC3339Nano, a.CreatedAt)
+		if err != nil {
+			continue
+		}
+		id := a.UUID
+		if id == "" {
+			id = fmt.Sprintf("%d:%d", a.ID, ts.Unix())
+		}
+		// An alert may span multiple domains. Do not invent a one-to-one link.
+		domains := map[string]bool{}
+		for _, event := range a.Events {
+			for _, m := range event.Meta {
+				if m.Key == "target_fqdn" {
+					domains[m.Value] = true
+				}
+			}
+		}
+		domain := ""
+		if len(domains) == 1 {
+			for d := range domains {
+				domain = d
+			}
+		}
+		out = append(out, TrafficEvent{ID: "cs-alert:" + id, Time: ts.Unix(), Kind: "alert", IP: a.Source.IP, Domain: domain, Reason: a.Scenario})
+		for _, d := range a.Decisions {
+			if !d.Simulated && net.ParseIP(d.Value) != nil {
+				out = append(out, TrafficEvent{ID: fmt.Sprintf("cs-decision:%s:%d", id, d.ID), Time: ts.Unix(), Kind: "decision", IP: d.Value, Domain: domain, Reason: d.Type + ": " + a.Scenario})
+			}
+		}
+	}
+	return out, nil
+}
+
+// cscli uses the existing local machine credentials. No new bouncer key or
+// public LAPI port is needed, and only the read-only alerts list is executed.
+func collectCrowdTraffic(ctx context.Context) ([]TrafficEvent, bool, error) {
+	ctx, cancel := context.WithTimeout(ctx, 6*time.Second)
+	defer cancel()
+	cmd := exec.CommandContext(ctx, "cscli", "alerts", "list", "--since", "24h", "--limit", "50", "-o", "json")
+	var output limitedTrafficOutput
+	cmd.Stdout = &output
+	if err := cmd.Run(); err != nil {
+		return nil, false, errors.New("cscli alerts list недоступен; проверьте права агента и локальную LAPI")
+	}
+	events, err := parseCrowdTraffic(output.Bytes())
+	if err != nil {
+		return nil, false, errors.New("Не удалось разобрать JSON CrowdSec")
+	}
+	var raw []json.RawMessage
+	json.Unmarshal(output.Bytes(), &raw)
+	return events, len(raw) >= 50, nil
+}
+
+type limitedTrafficOutput struct{ bytes.Buffer }
+
+func (b *limitedTrafficOutput) Write(p []byte) (int, error) {
+	if b.Len()+len(p) > 4<<20 {
+		return 0, errors.New("CrowdSec response too large")
+	}
+	return b.Buffer.Write(p)
+}
+
+func saveTrafficAgentState(path string, state trafficAgentState) error {
+	data, err := json.Marshal(state)
+	if err != nil {
+		return err
+	}
+	f, err := os.CreateTemp(filepath.Dir(path), ".traffic-")
+	if err != nil {
+		return err
+	}
+	name := f.Name()
+	defer os.Remove(name)
+	if err = f.Chmod(0600); err == nil {
+		_, err = f.Write(data)
+	}
+	if err == nil {
+		err = f.Sync()
+	}
+	closeErr := f.Close()
+	if err != nil {
+		return err
+	}
+	if closeErr != nil {
+		return closeErr
+	}
+	return os.Rename(name, path)
+}
+func runAgentTraffic(ctx context.Context, config agentConfig, stateDir string) {
+	dir := os.Getenv("DASHBOARD_NPM_LOG_DIR")
+	if dir == "off" {
+		return
+	}
+	if dir == "" {
+		dir = "/opt/npm/data/logs"
+	}
+
+	statePath := filepath.Join(stateDir, "traffic-state.json")
+	state := trafficAgentState{StartedAt: time.Now().Unix(), Cursors: map[string]trafficCursor{}, CrowdSeen: map[string]int64{}}
+	if data, err := os.ReadFile(statePath); err == nil {
+		if json.Unmarshal(data, &state) != nil {
+			fmt.Fprintln(os.Stderr, "traffic: invalid checkpoint; collector stopped")
+			return
+		}
+	} else if !os.IsNotExist(err) {
+		fmt.Fprintln(os.Stderr, "traffic: checkpoint unreadable")
+		return
+	}
+	if state.Cursors == nil {
+		state.Cursors = map[string]trafficCursor{}
+	}
+	if state.CrowdSeen == nil {
+		state.CrowdSeen = map[string]int64{}
+	}
+	client := &http.Client{Timeout: 10 * time.Second}
+	lastCrowd := time.Time{}
+	crowdAt := int64(0)
+	crowdError := ""
+	crowdLimited := false
+	ticker := time.NewTicker(2 * time.Second)
+	defer ticker.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		default:
+		}
+		files, logErr := state.readLogs(dir)
+		if len(state.Pending) < 400 && time.Since(lastCrowd) >= 15*time.Second {
+			events, limited, err := collectCrowdTraffic(ctx)
+			crowdLimited = limited
+			lastCrowd = time.Now()
+			if err != nil {
+				crowdError = err.Error()
+			} else {
+				crowdError = ""
+				crowdAt = time.Now().Unix()
+				for _, e := range events {
+					if _, seen := state.CrowdSeen[e.ID]; !seen && len(state.Pending) < 500 {
+						state.Pending = append(state.Pending, e)
+						state.CrowdSeen[e.ID] = e.Time
+					}
+				}
+				for id, ts := range state.CrowdSeen {
+					if ts < time.Now().Unix()-86400 {
+						delete(state.CrowdSeen, id)
+					}
+				}
+			}
+		}
+		// Persist the pending batch AND offsets before sending. A retry/restart may
+		// resend events; the server's (agent,event_id) unique key makes it idempotent.
+		if err := saveTrafficAgentState(statePath, state); err != nil {
+			fmt.Fprintln(os.Stderr, "traffic: checkpoint write failed; collector stopped")
+			return
+		}
+		batch := TrafficBatch{BacklogBytes: state.BacklogBytes, CrowdSecLimited: crowdLimited, Events: state.Pending, Files: files, ParseErrors: state.ParseErrors, CrowdSecAt: crowdAt, CrowdSecError: crowdError, StartedAt: state.StartedAt}
+		if logErr != nil {
+			batch.LogError = logErr.Error()
+		}
+		data, _ := json.Marshal(batch)
+		req, err := http.NewRequestWithContext(ctx, "POST", config.Server+"/api/agent/traffic", bytes.NewReader(data))
+		if err != nil {
+			return
+		}
+		req.Header.Set("Authorization", "Bearer "+config.ID+"."+config.Secret)
+		req.Header.Set("Content-Type", "application/json")
+		if res, err := client.Do(req); err == nil {
+			io.Copy(io.Discard, io.LimitReader(res.Body, 4096))
+			res.Body.Close()
+			if res.StatusCode == 200 {
+				state.Pending = nil
+				if saveTrafficAgentState(statePath, state) != nil {
+					return
+				}
+			}
+		}
+		select {
+		case <-ctx.Done():
+			return
+		case <-ticker.C:
+		}
+	}
+}
+
+func startAgentTraffic(config agentConfig, stateDir string) func() {
+	ctx, cancel := context.WithCancel(context.Background())
+	done := make(chan struct{})
+	go func() { defer close(done); runAgentTraffic(ctx, config, stateDir) }()
+	return func() { cancel(); <-done }
+}

alert_events.go

@@ -0,0 +1,242 @@
+package main
+
+import (
+	"database/sql"
+	"time"
+)
+
+type AlertEvent struct {
+	EventID           int64  `json:"eventId"`
+	ID                string `json:"id"`
+	Severity          string `json:"severity"`
+	Source            string `json:"source"`
+	Title             string `json:"title"`
+	Message           string `json:"message"`
+	FirstSeen         int64  `json:"firstSeen"`
+	LastSeen          int64  `json:"lastSeen"`
+	ResolvedAt        *int64 `json:"resolvedAt,omitempty"`
+	AcknowledgedAt    *int64 `json:"acknowledgedAt,omitempty"`
+	Planned           bool   `json:"planned"`
+	MaintenanceReason string `json:"maintenanceReason,omitempty"`
+}
+
+type RecoveryEvent struct {
+	EventID                              int64
+	ID, Severity, Source, Title, Message string
+}
+
+func (s *Store) SyncAlerts(alerts []Alert) ([]Alert, error) {
+	return s.SyncAlertsWithMaintenance(alerts, nil, "")
+}
+
+func (s *Store) SyncAlertsWithMaintenance(alerts, plannedAlerts []Alert, maintenanceReason string) ([]Alert, error) {
+	tx, err := s.db.Begin()
+	if err != nil {
+		return alerts, err
+	}
+	defer tx.Rollback()
+	now := time.Now().Unix()
+	rows, err := tx.Query(`SELECT id,alert_key,planned FROM alert_events WHERE resolved_at IS NULL`)
+	if err != nil {
+		return alerts, err
+	}
+	active := map[string]int64{}
+	for rows.Next() {
+		var id int64
+		var key string
+		var planned bool
+		if err := rows.Scan(&id, &key, &planned); err != nil {
+			rows.Close()
+			return alerts, err
+		}
+		active[alertEventKey(key, planned)] = id
+	}
+	rows.Close()
+	seen := map[string]bool{}
+	syncGroup := func(group []Alert, planned bool) error {
+		for _, alert := range group {
+			key := alertEventKey(alert.ID, planned)
+			seen[key] = true
+			reason := ""
+			if planned {
+				reason = maintenanceReason
+			}
+			if id, ok := active[key]; ok {
+				_, err = tx.Exec(`UPDATE alert_events SET severity=?,source=?,title=?,message=?,last_seen=?,maintenance_reason=? WHERE id=?`, alert.Severity, alert.Source, alert.Title, alert.Message, now, reason, id)
+			} else {
+				result, insertErr := tx.Exec(`INSERT INTO alert_events(alert_key,severity,source,title,message,first_seen,last_seen,planned,maintenance_reason) VALUES(?,?,?,?,?,?,?,?,?)`, alert.ID, alert.Severity, alert.Source, alert.Title, alert.Message, now, now, planned, reason)
+				if insertErr != nil {
+					return insertErr
+				}
+				active[key], _ = result.LastInsertId()
+			}
+			if err != nil {
+				return err
+			}
+		}
+		return nil
+	}
+	if err = syncGroup(alerts, false); err != nil {
+		return alerts, err
+	}
+	if err = syncGroup(plannedAlerts, true); err != nil {
+		return alerts, err
+	}
+	for key, id := range active {
+		if !seen[key] {
+			if _, err = tx.Exec(`UPDATE alert_events SET resolved_at=? WHERE id=?`, now, id); err != nil {
+				return alerts, err
+			}
+		}
+	}
+	if _, err = tx.Exec(`DELETE FROM alert_events WHERE (resolved_at IS NOT NULL OR acknowledged_at IS NOT NULL) AND last_seen < ?`, now-int64(30*24*time.Hour/time.Second)); err != nil {
+		return alerts, err
+	}
+	if err = tx.Commit(); err != nil {
+		return alerts, err
+	}
+	rows, err = s.db.Query(`SELECT id,alert_key,severity,source,title,message,first_seen,last_seen FROM alert_events WHERE resolved_at IS NULL AND acknowledged_at IS NULL AND planned=0 ORDER BY CASE severity WHEN 'critical' THEN 0 WHEN 'warning' THEN 1 ELSE 2 END,first_seen DESC`)
+	if err != nil {
+		return alerts, err
+	}
+	defer rows.Close()
+	var visible []Alert
+	for rows.Next() {
+		var a Alert
+		if err = rows.Scan(&a.EventID, &a.ID, &a.Severity, &a.Source, &a.Title, &a.Message, &a.FirstSeen, &a.LastSeen); err != nil {
+			return alerts, err
+		}
+		visible = append(visible, a)
+	}
+	return visible, rows.Err()
+}
+
+func alertEventKey(id string, planned bool) string {
+	if planned {
+		return id + "\x00planned"
+	}
+	return id + "\x00active"
+}
+
+func (s *Store) AcknowledgeAlert(eventID int64) error {
+	_, err := s.db.Exec(`UPDATE alert_events SET acknowledged_at=? WHERE id=? AND acknowledged_at IS NULL`, time.Now().Unix(), eventID)
+	return err
+}
+
+func (s *Store) AlertHistory(limit int) ([]AlertEvent, error) {
+	return s.AlertHistoryBetween(limit, 0, 0)
+}
+
+func (s *Store) AlertHistoryBetween(limit int, from, to int64) ([]AlertEvent, error) {
+	if limit < 1 || limit > 500 {
+		limit = 200
+	}
+	query := `SELECT id,alert_key,severity,source,title,message,first_seen,last_seen,resolved_at,acknowledged_at,planned,maintenance_reason FROM alert_events`
+	args := []any{}
+	if from > 0 && to > from {
+		query += ` WHERE first_seen>=? AND first_seen<?`
+		args = append(args, from, to)
+	}
+	query += ` ORDER BY first_seen DESC LIMIT ?`
+	args = append(args, limit)
+	rows, err := s.db.Query(query, args...)
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+	var result []AlertEvent
+	for rows.Next() {
+		var e AlertEvent
+		var resolved, acked sql.NullInt64
+		if err = rows.Scan(&e.EventID, &e.ID, &e.Severity, &e.Source, &e.Title, &e.Message, &e.FirstSeen, &e.LastSeen, &resolved, &acked, &e.Planned, &e.MaintenanceReason); err != nil {
+			return nil, err
+		}
+		if resolved.Valid {
+			e.ResolvedAt = &resolved.Int64
+		}
+		if acked.Valid {
+			e.AcknowledgedAt = &acked.Int64
+		}
+		result = append(result, e)
+	}
+	return result, rows.Err()
+}
+
+func (s *Store) PendingEmailAlerts() ([]Alert, error) {
+	rows, err := s.db.Query(`SELECT id,alert_key,severity,source,title,message FROM alert_events WHERE resolved_at IS NULL AND acknowledged_at IS NULL AND email_sent_at IS NULL AND planned=0 ORDER BY first_seen`)
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+	var result []Alert
+	for rows.Next() {
+		var a Alert
+		if err = rows.Scan(&a.EventID, &a.ID, &a.Severity, &a.Source, &a.Title, &a.Message); err != nil {
+			return nil, err
+		}
+		result = append(result, a)
+	}
+	return result, rows.Err()
+}
+
+func (s *Store) MarkAlertsEmailed(alerts []Alert) error {
+	tx, err := s.db.Begin()
+	if err != nil {
+		return err
+	}
+	defer tx.Rollback()
+	now := time.Now().Unix()
+	for _, a := range alerts {
+		if _, err = tx.Exec(`UPDATE alert_events SET email_sent_at=? WHERE id=?`, now, a.EventID); err != nil {
+			return err
+		}
+	}
+	return tx.Commit()
+}
+
+func (s *Store) PendingImmediateAlerts() ([]Alert, error) {
+	rows, err := s.db.Query(`SELECT id,alert_key,severity,source,title,message FROM alert_events WHERE resolved_at IS NULL AND acknowledged_at IS NULL AND email_sent_at IS NULL AND planned=0 AND (severity='critical' OR alert_key='ups-battery') ORDER BY first_seen`)
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+	var out []Alert
+	for rows.Next() {
+		var a Alert
+		if err = rows.Scan(&a.EventID, &a.ID, &a.Severity, &a.Source, &a.Title, &a.Message); err != nil {
+			return nil, err
+		}
+		out = append(out, a)
+	}
+	return out, rows.Err()
+}
+func (s *Store) PendingRecoveries() ([]RecoveryEvent, error) {
+	rows, err := s.db.Query(`SELECT id,alert_key,severity,source,title,message FROM alert_events WHERE resolved_at IS NOT NULL AND email_sent_at IS NOT NULL AND recovery_sent_at IS NULL AND planned=0 ORDER BY resolved_at`)
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+	var out []RecoveryEvent
+	for rows.Next() {
+		var v RecoveryEvent
+		if err = rows.Scan(&v.EventID, &v.ID, &v.Severity, &v.Source, &v.Title, &v.Message); err != nil {
+			return nil, err
+		}
+		out = append(out, v)
+	}
+	return out, rows.Err()
+}
+func (s *Store) MarkRecoveriesEmailed(items []RecoveryEvent) error {
+	tx, err := s.db.Begin()
+	if err != nil {
+		return err
+	}
+	defer tx.Rollback()
+	now := time.Now().Unix()
+	for _, v := range items {
+		if _, err = tx.Exec(`UPDATE alert_events SET recovery_sent_at=? WHERE id=?`, now, v.EventID); err != nil {
+			return err
+		}
+	}
+	return tx.Commit()
+}

alert_events_test.go

@@ -0,0 +1,58 @@
+package main
+
+import "testing"
+
+func TestAlertLifecycle(t *testing.T) {
+	store, err := openStore(":memory:")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer store.Close()
+	alert := Alert{ID: "disk", Severity: "warning", Source: "Диски", Title: "SMART", Message: "Проверьте диск"}
+	visible, err := store.SyncAlerts([]Alert{alert})
+	if err != nil || len(visible) != 1 || visible[0].EventID == 0 || visible[0].FirstSeen == 0 || visible[0].LastSeen == 0 {
+		t.Fatalf("unexpected new alert: %+v %v", visible, err)
+	}
+	firstID := visible[0].EventID
+	if err = store.AcknowledgeAlert(firstID); err != nil {
+		t.Fatal(err)
+	}
+	visible, err = store.SyncAlerts([]Alert{alert})
+	if err != nil || len(visible) != 0 {
+		t.Fatalf("acknowledged alert became visible: %+v %v", visible, err)
+	}
+	if _, err = store.SyncAlerts(nil); err != nil {
+		t.Fatal(err)
+	}
+	visible, err = store.SyncAlerts([]Alert{alert})
+	if err != nil || len(visible) != 1 || visible[0].EventID == firstID {
+		t.Fatalf("recurring alert was not new: %+v %v", visible, err)
+	}
+	history, err := store.AlertHistory(20)
+	if err != nil || len(history) != 2 {
+		t.Fatalf("unexpected history: %+v %v", history, err)
+	}
+}
+
+func TestPendingEmailAlerts(t *testing.T) {
+	store, err := openStore(":memory:")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer store.Close()
+	visible, err := store.SyncAlerts([]Alert{{ID: "cpu", Severity: "critical", Source: "CPU", Title: "Load"}})
+	if err != nil {
+		t.Fatal(err)
+	}
+	pending, err := store.PendingEmailAlerts()
+	if err != nil || len(pending) != 1 {
+		t.Fatalf("unexpected pending: %+v %v", pending, err)
+	}
+	if err = store.MarkAlertsEmailed(visible); err != nil {
+		t.Fatal(err)
+	}
+	pending, err = store.PendingEmailAlerts()
+	if err != nil || len(pending) != 0 {
+		t.Fatalf("emailed alert repeated: %+v %v", pending, err)
+	}
+}

alerts.go

@@ -8,11 +8,14 @@ import (
 )
 
 type Alert struct {
-	ID       string `json:"id"`
-	Severity string `json:"severity"`
-	Source   string `json:"source"`
-	Title    string `json:"title"`
-	Message  string `json:"message"`
+	EventID   int64  `json:"eventId,omitempty"`
+	ID        string `json:"id"`
+	Severity  string `json:"severity"`
+	Source    string `json:"source"`
+	Title     string `json:"title"`
+	Message   string `json:"message"`
+	FirstSeen int64  `json:"firstSeen,omitempty"`
+	LastSeen  int64  `json:"lastSeen,omitempty"`
 }
 
 func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Alert {
@@ -20,6 +23,17 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
 	add := func(id, severity, source, title, message string) {
 		alerts = append(alerts, Alert{ID: id, Severity: severity, Source: source, Title: title, Message: message})
 	}
+	if metrics.Update.State == "available" {
+		add("dashboard-update", "info", "Dashboard", "Доступно обновление "+metrics.Update.AvailableVersion, "Режим обновлений: только уведомлять. Установите обновление с сервера или включите автоматический режим.")
+	} else if metrics.Update.State == "error" {
+		add("dashboard-update-error", "warning", "Dashboard", "Не удалось проверить обновления", metrics.Update.Message)
+	}
+	if metrics.Maintenance.RebootRequired || metrics.Maintenance.NewKernelAvailable {
+		add("maintenance-reboot", "warning", "Proxmox", "Требуется перезагрузка хоста", "Установлено новое ядро или система запросила перезагрузку.")
+	}
+	if metrics.Maintenance.AvailableUpdates > 0 {
+		add("maintenance-updates", "info", "Proxmox", "Доступны обновления системы", fmt.Sprintf("Пакетов для обновления: %d.", metrics.Maintenance.AvailableUpdates))
+	}
 
 	if metrics.CPU.UsagePercent >= thresholds.CPUCritical {
 		add("cpu-usage", "critical", "CPU", "Очень высокая загрузка CPU", fmt.Sprintf("Текущая загрузка %.1f%%.", metrics.CPU.UsagePercent))
@@ -49,10 +63,21 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
 		add("root-storage", "warning", "Storage", "Мало места на системном разделе", fmt.Sprintf("Раздел / заполнен на %.1f%%.", metrics.Storage.UsagePercent))
 	}
 	for _, disk := range metrics.Disks {
+		diskExcluded := false
+		for _, value := range strings.Split(thresholds.DiskIOExcluded, ",") {
+			value = strings.TrimSpace(value)
+			if value != "" && (strings.EqualFold(value, disk.Name) || strings.EqualFold(value, disk.Serial)) {
+				diskExcluded = true
+				break
+			}
+		}
 		if disk.SMARTStatus == "Ошибка" {
 			add("disk-smart-"+disk.Name, "critical", "Диски", disk.Model+": ошибка SMART", joinReasons(disk.AttentionReasons, "SMART сообщает о неисправности диска."))
 		} else if disk.SMARTStatus == "Требует внимания" {
-			add("disk-smart-"+disk.Name, "warning", "Диски", disk.Model+": SMART требует внимания", joinReasons(disk.AttentionReasons, "Проверьте SMART-показатели диска."))
+			reasons := withoutReason(disk.AttentionReasons, "SMART self-test ещё ни разу не запускался")
+			if len(reasons) > 0 {
+				add("disk-smart-"+disk.Name, "warning", "Диски", disk.Model+": SMART требует внимания", joinReasons(reasons, "Проверьте SMART-показатели диска."))
+			}
 		}
 		if disk.Temperature != nil {
 			limit := thresholds.DiskTempWarning
@@ -63,6 +88,26 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
 				add("disk-temp-"+disk.Name, "warning", "Диски", disk.Model+": высокая температура", fmt.Sprintf("Температура %.1f °C.", *disk.Temperature))
 			}
 		}
+		if len(disk.CounterChanges) > 0 {
+			add("disk-growth-"+disk.Name, "critical", "Диски", disk.Model+": ухудшились SMART-счётчики", strings.Join(disk.CounterChanges, " · "))
+		}
+		if !diskExcluded && disk.IOPressureSeconds >= thresholds.DiskIODuration {
+			severity := "warning"
+			if disk.Utilization >= 98 || disk.LatencyMs >= 100 {
+				severity = "critical"
+			}
+			cause := disk.IOCause
+			if cause == "" {
+				cause = "источник пока не определён"
+			}
+			add("disk-io-pressure-"+disk.Name, severity, "Диски", disk.Model+": длительная I/O-нагрузка", fmt.Sprintf("%s занят на %.1f%%; вероятная причина: %s (уверенность %d%%). %.0f IOPS, %.1f MB/s, latency %.1f ms, очередь %.1f.", disk.Name, disk.Utilization, cause, disk.IOConfidence, disk.ReadIOPS+disk.WriteIOPS, (disk.ReadBytesPerSec+disk.WriteBytesPerSec)/1048576, disk.LatencyMs, disk.AverageQueue))
+		}
+		if disk.SelfTestNeverRun {
+			add("disk-selftest-"+disk.Name, "warning", "Диски", disk.Model+": нет завершённого SMART self-test", "Короткий тест будет автоматически запущен; проверьте его результат после завершения.")
+		}
+	}
+	for _, missing := range metrics.MissingDisks {
+		add("disk-missing-"+missing, "critical", "Диски", "Физический диск исчез", missing+" больше не обнаруживается в /sys/block.")
 	}
 
 	for _, pool := range metrics.ZFS.Pools {
@@ -77,6 +122,25 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
 		if strings.Contains(strings.ToLower(pool.Scan), "in progress") {
 			add("zfs-scan-"+pool.Name, "info", "ZFS", pool.Name+": выполняется scrub/resilver", pool.Scan)
 		}
+		if pool.ReadErrors+pool.WriteErrors+pool.ChecksumErrors > 0 {
+			add("zfs-vdev-errors-"+pool.Name, "critical", "ZFS", pool.Name+": ошибки чтения/записи/checksum", fmt.Sprintf("read=%d, write=%d, checksum=%d.", pool.ReadErrors, pool.WriteErrors, pool.ChecksumErrors))
+		}
+		if len(pool.ErrorChanges) > 0 {
+			add("zfs-new-errors-"+pool.Name, "critical", "ZFS", pool.Name+": появились новые ошибки", strings.Join(pool.ErrorChanges, " · "))
+		}
+		if pool.ScrubTooOld {
+			add("zfs-scrub-old-"+pool.Name, "warning", "ZFS", pool.Name+": scrub просрочен", "Scrub не выполнялся более 35 дней либо ни разу не запускался.")
+		}
+	}
+	for _, dataset := range metrics.ZFS.Datasets {
+		if dataset.CapacityPercent >= 90 {
+			add("zfs-dataset-"+dataset.Name, "critical", "ZFS", dataset.Name+": dataset почти заполнен", fmt.Sprintf("Заполнение %.1f%%, доступно %.1f GB.", dataset.CapacityPercent, float64(dataset.AvailableBytes)/1073741824))
+		} else if dataset.CapacityPercent >= 80 {
+			add("zfs-dataset-"+dataset.Name, "warning", "ZFS", dataset.Name+": мало свободного места", fmt.Sprintf("Заполнение %.1f%%.", dataset.CapacityPercent))
+		}
+	}
+	if metrics.ZFS.ARC.Available && metrics.ZFS.ARC.Hits+metrics.ZFS.ARC.Misses > 10000 && metrics.ZFS.ARC.HitRatio < 70 {
+		add("zfs-arc-hit", "warning", "ZFS", "Низкий ARC hit ratio", fmt.Sprintf("Попадания ARC: %.1f%%.", metrics.ZFS.ARC.HitRatio))
 	}
 
 	if metrics.UPS.Available {
@@ -89,6 +153,14 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
 		if metrics.UPS.ChargePercent != nil && *metrics.UPS.ChargePercent < thresholds.UPSChargeCritical {
 			add("ups-charge", "critical", "UPS", "Низкий заряд батареи", fmt.Sprintf("Осталось %.1f%%.", *metrics.UPS.ChargePercent))
 		}
+		if metrics.UPS.ReplaceBattery {
+			add("ups-replace-battery", "critical", "UPS", "UPS требует заменить батарею", "NUT/APC сообщает флаг Replace Battery.")
+		}
+		if metrics.UPS.BatteryAgeDays > 3*365 {
+			add("ups-battery-age", "warning", "UPS", "Батарея UPS старше трёх лет", fmt.Sprintf("Возраст батареи примерно %d дней.", metrics.UPS.BatteryAgeDays))
+		}
+	} else if metrics.UPS.Source != "" {
+		add("ups-communication", "warning", "UPS", "Нет связи с UPS", "NUT и apcupsd не вернули данные. Проверьте USB/сетевое подключение и службу UPS.")
 	}
 
 	if metrics.Backups.Available && metrics.Guests.Available {
@@ -128,6 +200,129 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
 			add("guests-stopped", "info", "VM/LXC", "Есть остановленные гости", fmt.Sprintf("Остановлено VM/LXC: %d. Это уведомление информационное.", stopped))
 		}
 	}
+	for _, agent := range metrics.Agents {
+		vmSuffix := fmt.Sprintf("-%d", agent.VMID)
+		if !agent.Online {
+			add("agent-offline-"+agent.ID+vmSuffix, "critical", "Агенты", agent.Name+": агент не отвечает", fmt.Sprintf("Последний heartbeat: %s назад. VMID %d.", time.Since(time.Unix(agent.LastSeen, 0)).Round(time.Second), agent.VMID))
+			continue
+		}
+		if agent.Report.DockerAvailable {
+			for _, container := range agent.Report.Containers {
+				if container.State != "running" {
+					add("agent-container-stopped-"+agent.ID+"-"+container.ID+vmSuffix, "warning", "Docker", agent.Name+": контейнер "+container.Name+" остановлен", fmt.Sprintf("%s · VMID %d.", container.Status, agent.VMID))
+				} else if container.Health == "unhealthy" {
+					add("agent-container-health-"+agent.ID+"-"+container.ID+vmSuffix, "critical", "Docker", agent.Name+": контейнер "+container.Name+" unhealthy", fmt.Sprintf("Docker health-check не проходит · VMID %d.", agent.VMID))
+				}
+			}
+		}
+		for _, service := range agent.Report.Services {
+			if service.State == "failed" || service.SubState == "failed" {
+				add("agent-systemd-"+agent.ID+"-"+service.Name+vmSuffix, "critical", "Агенты", agent.Name+": упала служба "+service.Name, fmt.Sprintf("%s · VMID %d.", firstNonEmpty(service.Description, service.SubState), agent.VMID))
+			} else if service.Flapping {
+				add("agent-flapping-"+agent.ID+"-"+service.Name+vmSuffix, "warning", "Агенты", agent.Name+": служба постоянно перезапускается", fmt.Sprintf("%s: %d рестартов, последний запуск %s · VMID %d.", service.Name, service.Restarts, time.Unix(service.StartedAt, 0).Format("15:04:05"), agent.VMID))
+			}
+		}
+		for _, filesystem := range agent.Report.Filesystems {
+			if filesystem.UsedPercent >= 90 {
+				add("agent-filesystem-"+agent.ID+"-"+filesystem.Mountpoint+vmSuffix, "critical", "Агенты", agent.Name+": заканчивается место "+filesystem.Mountpoint, fmt.Sprintf("Заполнено %.1f%%, свободно %.1f GB · VMID %d.", filesystem.UsedPercent, float64(filesystem.TotalBytes-filesystem.UsedBytes)/1073741824, agent.VMID))
+			}
+			if filesystem.InodesPercent >= 90 {
+				add("agent-inodes-"+agent.ID+"-"+filesystem.Mountpoint+vmSuffix, "critical", "Агенты", agent.Name+": заканчиваются inode "+filesystem.Mountpoint, fmt.Sprintf("Использовано %.1f%% inode · VMID %d.", filesystem.InodesPercent, agent.VMID))
+			}
+		}
+		if len(agent.Report.OOMKills) > 0 {
+			latest := agent.Report.OOMKills[0]
+			add("agent-oom-"+agent.ID+vmSuffix, "critical", "Агенты", agent.Name+": ядро завершало процессы из-за нехватки памяти", fmt.Sprintf("%s · %s · VMID %d.", time.Unix(latest.Time, 0).Format("02.01 15:04"), latest.Message, agent.VMID))
+		}
+		if agent.Report.RebootRequired {
+			add("agent-reboot-"+agent.ID+vmSuffix, "info", "Агенты", agent.Name+": требуется перезагрузка", fmt.Sprintf("Обновления системы запросили перезагрузку · VMID %d.", agent.VMID))
+		}
+		if agent.Report.MemoryTotal > 0 && float64(agent.Report.MemoryUsed)/float64(agent.Report.MemoryTotal)*100 >= 95 {
+			add("agent-memory-"+agent.ID+vmSuffix, "warning", "Агенты", agent.Name+": почти закончилась память", fmt.Sprintf("Используется %.1f%% RAM · VMID %d.", float64(agent.Report.MemoryUsed)/float64(agent.Report.MemoryTotal)*100, agent.VMID))
+		}
+		if agent.Report.RootTotal > 0 && float64(agent.Report.RootUsed)/float64(agent.Report.RootTotal)*100 >= 90 {
+			add("agent-storage-"+agent.ID+vmSuffix, "critical", "Агенты", agent.Name+": мало места на системном разделе", fmt.Sprintf("Раздел / заполнен на %.1f%% · VMID %d.", float64(agent.Report.RootUsed)/float64(agent.Report.RootTotal)*100, agent.VMID))
+		}
+	}
+	for _, service := range metrics.Services.Services {
+		for _, endpoint := range service.Endpoints {
+			if !endpoint.Up {
+				add(fmt.Sprintf("service-%d-%s", service.ID, endpoint.Kind), "critical", "Сервисы", service.Name+": недоступен "+strings.ToLower(endpoint.Label), endpoint.Error)
+			} else if endpoint.TLSDays != nil && *endpoint.TLSDays < 0 {
+				add(fmt.Sprintf("service-tls-%d-%s", service.ID, endpoint.Kind), "critical", "Сервисы", service.Name+": сертификат истёк", fmt.Sprintf("%s: истёк %d дн. назад.", endpoint.Label, -*endpoint.TLSDays))
+			} else if endpoint.TLSDays != nil && *endpoint.TLSDays < 14 {
+				add(fmt.Sprintf("service-tls-%d-%s", service.ID, endpoint.Kind), "warning", "Сервисы", service.Name+": сертификат скоро истечёт", fmt.Sprintf("%s: осталось %d дн.", endpoint.Label, *endpoint.TLSDays))
+			}
+		}
+	}
+	if metrics.SystemHealth.FailedUnits > 0 {
+		add("systemd-failed", "critical", "Система", "Есть упавшие systemd-службы", fmt.Sprintf("Не запущено служб: %d.", metrics.SystemHealth.FailedUnits))
+	}
+	if metrics.SystemHealth.KernelErrors > 0 {
+		add("kernel-errors", "warning", "Система", "Ошибки ядра за 24 часа", fmt.Sprintf("Найдено сообщений: %d.", metrics.SystemHealth.KernelErrors))
+	}
+	var networkErrors uint64
+	for _, iface := range metrics.Network.Interfaces {
+		networkErrors += iface.NewErrors + iface.NewFaults
+		if iface.LinkDown {
+			add("network-link-"+iface.Name, "critical", "Сеть", iface.Name+": нет линка", "Используемый интерфейс потерял carrier или перешёл в down.")
+		}
+	}
+	if networkErrors > 0 {
+		add("network-errors", "critical", "Сеть", "Появились новые сетевые ошибки", fmt.Sprintf("Прирост счётчиков ошибок: %d (счётчики могут пересекаться). Software RX/TX drops показаны отдельно и не создают этот алерт.", networkErrors))
+	}
+	if !metrics.Network.Health.CheckedAt.IsZero() && !metrics.Network.Health.GatewayUp {
+		add("network-gateway", "critical", "Сеть", "Недоступен локальный шлюз", firstNonEmpty(metrics.Network.Health.Gateway, "Default gateway не определён."))
+	}
+	if !metrics.Network.Health.CheckedAt.IsZero() && !metrics.Network.Health.DNSUp {
+		add("network-dns", "critical", "Сеть", "Не работает DNS", "Не удалось разрешить deb.debian.org.")
+	}
+	if !metrics.Network.Health.CheckedAt.IsZero() && !metrics.Network.Health.InternetUp {
+		add("network-internet", "warning", "Сеть", "Нет доступа в интернет", "Внешняя проверка IP не выполнена.")
+	}
+	if metrics.Network.Health.PacketLoss > 0 {
+		add("network-loss", "warning", "Сеть", "Потери пакетов до шлюза", fmt.Sprintf("Потери %.1f%%, задержка %.1f ms.", metrics.Network.Health.PacketLoss, metrics.Network.Health.LatencyMs))
+	}
+	if metrics.Network.Health.ExternalIPChanged {
+		add("network-public-ip", "info", "Сеть", "Изменился внешний IP", "Новый адрес: "+metrics.Network.Health.ExternalIP)
+	}
+	for _, name := range metrics.Network.Health.DisappearedInterfaces {
+		add("network-interface-missing-"+name, "warning", "Сеть", "Исчез сетевой интерфейс", name+" больше не присутствует в /proc/net/dev.")
+	}
+
+	if !metrics.Monitor.LastSuccessfulCollection.IsZero() && time.Since(metrics.Monitor.LastSuccessfulCollection) > 30*time.Second {
+		add("monitor-stale", "critical", "Dashboard", "Данные мониторинга устарели", "Последний успешный сбор был "+time.Since(metrics.Monitor.LastSuccessfulCollection).Round(time.Second).String()+" назад.")
+	}
+	if metrics.Monitor.HistoryWriteError != "" {
+		add("monitor-history", "critical", "Dashboard", "Ошибка записи истории", metrics.Monitor.HistoryWriteError)
+	}
+	if metrics.Monitor.DatabaseFreeBytes > 0 && metrics.Monitor.DatabaseFreeBytes < 1024*1024*1024 {
+		add("monitor-db-space", "warning", "Dashboard", "Мало места для базы истории", fmt.Sprintf("На разделе базы свободно %.1f GB.", float64(metrics.Monitor.DatabaseFreeBytes)/1073741824))
+	}
+	oldSnapshots := 0
+	for _, snapshot := range metrics.Activity.Snapshots {
+		if snapshot.AgeSeconds > 30*24*3600 {
+			oldSnapshots++
+		}
+	}
+	if oldSnapshots > 0 {
+		add("old-snapshots", "warning", "VM/LXC", "Есть старые snapshots", fmt.Sprintf("Снимков старше 30 дней: %d.", oldSnapshots))
+	}
+	for _, task := range metrics.Activity.Tasks {
+		if task.StartTime < time.Now().Add(-24*time.Hour).Unix() {
+			continue
+		}
+		if task.Status != "" && task.Status != "OK" {
+			severity := "warning"
+			if task.Type == "vzdump" {
+				severity = "critical"
+			}
+			add("task-failed-"+task.UPID, severity, "Proxmox", task.Type+": задача завершилась ошибкой", fmt.Sprintf("VMID %s, пользователь %s, статус %s.", task.ID, task.User, task.Status))
+		}
+	}
+	for _, trend := range metrics.Trends {
+		add(trend.ID, trend.Severity, trend.Source, trend.Title, trend.Message)
+	}
 
 	priority := map[string]int{"critical": 0, "warning": 1, "info": 2}
 	sort.SliceStable(alerts, func(i, j int) bool {
@@ -139,9 +334,64 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
 	return alerts
 }
 
+func correlateAlerts(metrics dashboardMetrics, alerts []Alert) []Alert {
+	pressureAlerts := map[string]bool{}
+	for _, a := range alerts {
+		if strings.HasPrefix(a.ID, "disk-io-pressure-") {
+			pressureAlerts[strings.TrimPrefix(a.ID, "disk-io-pressure-")] = true
+		}
+	}
+	var pressure []DiskMetrics
+	for _, d := range metrics.Disks {
+		if pressureAlerts[d.Name] && d.IOCause != "" {
+			pressure = append(pressure, d)
+		}
+	}
+	if len(pressure) == 0 {
+		return alerts
+	}
+	var down []string
+	for _, service := range metrics.Services.Services {
+		for _, endpoint := range service.Endpoints {
+			if !endpoint.Up {
+				down = append(down, service.Name)
+				break
+			}
+		}
+	}
+	d := pressure[0]
+	chain := d.IOCause + " → " + d.Name + fmt.Sprintf(" util %.1f%% → latency %.1f ms", d.Utilization, d.LatencyMs)
+	if len(down) > 0 {
+		chain += " → недоступны/медленны сервисы: " + strings.Join(down, ", ")
+	}
+	severity := "warning"
+	if len(down) > 0 || d.Utilization >= 98 {
+		severity = "critical"
+	}
+	correlated := Alert{ID: "correlation-io-" + d.Name, Severity: severity, Source: "Корреляция", Title: "Обнаружена связанная цепочка I/O-событий", Message: chain + fmt.Sprintf(". Уверенность причины %d%%.", d.IOConfidence)}
+	filtered := []Alert{correlated}
+	for _, a := range alerts {
+		if strings.HasPrefix(a.ID, "disk-io-pressure-") || strings.HasPrefix(a.ID, "service-") {
+			continue
+		}
+		filtered = append(filtered, a)
+	}
+	return filtered
+}
+
 func joinReasons(reasons []string, fallback string) string {
 	if len(reasons) == 0 {
 		return fallback
 	}
 	return strings.Join(reasons, " · ")
 }
+
+func withoutReason(reasons []string, excluded string) []string {
+	filtered := make([]string, 0, len(reasons))
+	for _, reason := range reasons {
+		if reason != excluded {
+			filtered = append(filtered, reason)
+		}
+	}
+	return filtered
+}

alerts_test.go

@@ -16,3 +16,26 @@ func TestEvaluateAlerts(t *testing.T) {
 		t.Fatalf("неверная сортировка уведомлений: %+v", alerts)
 	}
 }
+
+func TestSlowCollectionDoesNotCreateAlert(t *testing.T) {
+	metrics := dashboardMetrics{Monitor: MonitorHealth{SlowCollector: true, CollectionDurationMs: 12308}}
+	for _, alert := range evaluateAlerts(metrics, defaultThresholds()) {
+		if alert.ID == "monitor-slow" {
+			t.Fatalf("медленный цикл не должен создавать уведомление: %+v", alert)
+		}
+	}
+}
+
+func TestNeverRunSelfTestCreatesOneAlert(t *testing.T) {
+	metrics := dashboardMetrics{Disks: []DiskMetrics{{
+		Name:             "nvme0n1",
+		Model:            "Test NVMe",
+		SMARTStatus:      "Требует внимания",
+		SelfTestNeverRun: true,
+		AttentionReasons: []string{"SMART self-test ещё ни разу не запускался"},
+	}}}
+	alerts := evaluateAlerts(metrics, defaultThresholds())
+	if len(alerts) != 1 || alerts[0].ID != "disk-selftest-nvme0n1" {
+		t.Fatalf("ожидалось одно точное уведомление о self-test, получено: %+v", alerts)
+	}
+}

assets/NOTICE.md

@@ -0,0 +1,14 @@
+# Map data
+
+`world.json` is derived from Natural Earth 1:110m admin-0 countries (coordinates
+rounded to two decimal places; only country codes, labels and boundaries kept).
+Natural Earth data is public domain: https://www.naturalearthdata.com/about/terms-of-use/
+Source: https://github.com/nvkelso/natural-earth-vector/blob/master/geojson/ne_110m_admin_0_countries.geojson
+
+`dbip-country-lite-2026-10.mmdb.gz`: IP Geolocation by DB-IP, October 2026.
+Source: https://download.db-ip.com/free/dbip-country-lite-2026-10.mmdb.gz
+License: Creative Commons Attribution 4.0 International (CC BY 4.0).
+https://creativecommons.org/licenses/by/4.0/
+https://db-ip.com/db/download/ip-to-country-lite
+The database is unmodified. The traffic page displays the required DB-IP link.
+Countries are approximate; plotted positions are country labels, not exact IP locations.

assets/network.html

@@ -0,0 +1,39 @@
+<!doctype html><html lang="ru"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Сетевая активность · Proxmox Dashboard</title><style>
+:root{color-scheme:dark;--bg:#090e15;--panel:#111b26;--line:#293846;--muted:#92a8ba;--lime:#c0ee86;--blue:#70dcec}*{box-sizing:border-box}html,body{margin:0;height:100%;overflow:hidden}body{background:radial-gradient(ellipse at 30% 30%,#142738,transparent 65%),var(--bg);color:#e9f0f6;font:13px system-ui,sans-serif;padding:22px;display:grid;grid-template-rows:54px 32px 82px 36px minmax(0,1fr) 22px;gap:12px}header{display:flex;align-items:center;justify-content:space-between;gap:12px}h1{font-size:26px;letter-spacing:-1px;margin:0}small,.muted{color:var(--muted)}.eyebrow{font-size:9px;letter-spacing:2px;color:var(--lime)}nav{display:flex;gap:15px;align-items:center}a{color:var(--blue);text-decoration:none}button,input,select{font:inherit;color:inherit;background:#182430;border:1px solid var(--line);padding:7px 10px;border-radius:8px;min-width:0}button{cursor:pointer}button:disabled{opacity:.35;cursor:default}button:hover{border-color:var(--blue)}:focus-visible{outline:2px solid var(--blue);outline-offset:2px}.status{display:flex;align-items:center;gap:10px;color:var(--muted);min-width:0}.status span{overflow:hidden;white-space:nowrap;text-overflow:ellipsis}.badge{font-size:10px;color:var(--lime);border:1px solid #496036;padding:5px 8px;border-radius:20px;white-space:nowrap}.badge.warn{color:#ffbd80;border-color:#875d36}.metrics{display:grid;grid-template-columns:repeat(4,1fr);gap:12px}.metric,.panel{border:1px solid var(--line);background:#101a24d9;border-radius:14px;min-width:0;min-height:0;overflow:hidden}.metric{padding:12px 16px}.metric small{font-size:11px}.metric b{display:block;font-size:28px;line-height:1.3;letter-spacing:-1px}.filters{display:flex;gap:8px}.filters input{flex:1}.filters label{display:flex;align-items:center;gap:6px;font-size:11px;color:var(--muted)}.filters label input{accent-color:var(--lime)}main{display:grid;grid-template-columns:minmax(0,1fr) 290px;gap:12px;min-height:0}.flows{display:grid;grid-template-rows:40px 27px minmax(0,1fr) 38px;padding:0 14px}.heading{display:flex;align-items:center;justify-content:space-between;gap:8px}h2{font-size:12px;font-weight:600;margin:0}.table-head,.flow{display:grid;grid-template-columns:minmax(0,1.25fr) minmax(0,1.25fr) 80px 108px 95px;gap:12px;align-items:center}.table-head{color:var(--muted);font-size:10px}.flow{height:55px;border-top:1px solid #21303d;cursor:pointer;font:12px ui-monospace,monospace}.flow:hover{background:#20344566}.flow>div{min-width:0}.flow strong{font-weight:500;color:var(--blue);display:block;white-space:nowrap;overflow:hidden;text-overflow:ellipsis}.flow small{font:10px system-ui;display:block;margin-top:4px;white-space:nowrap;overflow:hidden;text-overflow:ellipsis}.state{font-size:10px}.state.ok{color:var(--lime)}.state.pending{color:#ffc085}#rows{min-height:0;overflow:hidden}.pager{display:flex;align-items:center;gap:12px;color:var(--muted);font-size:11px;border-top:1px solid var(--line)}.pager button{padding:3px 8px}.pager #resume{margin-left:auto;color:var(--lime)}aside{display:grid;grid-template-rows:1fr 1fr 1fr;gap:12px;min-height:0}aside .panel{padding:12px 14px}aside h2{margin-bottom:9px}.rank{display:flex;justify-content:space-between;gap:10px;font-size:11px;line-height:22px}.rank span{overflow:hidden;text-overflow:ellipsis;white-space:nowrap}.rank b{font-weight:500;color:var(--blue)}.block-value{font-size:24px;color:#c4a9ff;margin:8px 0}.help{font-size:10px;color:var(--muted);line-height:1.5}.empty{color:var(--muted);padding:20px 0;line-height:1.7;font-size:12px}.footer{display:flex;justify-content:space-between;font-size:10px;color:var(--muted);gap:10px}.link-button{background:none;border:0;padding:0;color:var(--blue);font-size:11px}dialog{width:min(650px,calc(100vw - 24px));max-height:85dvh;overflow:auto;border:1px solid #455969;border-radius:18px;background:#141f2b;color:#e9f0f6;padding:22px}dialog::backdrop{background:#03080ccc;backdrop-filter:blur(5px)}dialog .heading{margin-bottom:16px}dialog h2{font-size:19px}.detail{padding:10px 0;border-top:1px solid var(--line);overflow-wrap:anywhere}.detail small{display:block;margin-bottom:4px}dialog p{line-height:1.7}#tabs{display:none}.filters #port{width:110px}
+@media(max-width:1100px){main{grid-template-columns:minmax(0,1fr) 220px}.table-head,.flow{grid-template-columns:1fr 1fr 55px 83px;gap:8px}.flow .volume,.table-head .volume{display:none}.flow{font-size:11px}nav{gap:9px}body{padding:14px}aside .panel{padding:9px}aside .rank{line-height:19px}}
+@media(max-height:700px) and (min-width:761px){body{padding:12px;gap:8px;grid-template-rows:40px 26px 64px 32px minmax(0,1fr) 18px}.metric{padding:8px 12px}.metric b{font-size:22px}.flow{height:46px}aside{grid-template-rows:1fr 1fr 1fr;gap:6px}aside .rank{line-height:17px}.block-value{font-size:20px;margin:3px 0}aside h2{margin-bottom:4px}.help{font-size:9px}}
+@media(max-width:760px){body{padding:10px;gap:8px;grid-template-rows:45px 30px 104px auto 28px minmax(0,1fr) 18px}h1{font-size:18px;white-space:nowrap}.eyebrow{font-size:7px}nav a{font-size:0}nav a:before{content:"HTTP ↗";font-size:11px}nav .dashboard{display:none}.metrics{grid-template-columns:1fr 1fr;gap:6px}.metric{padding:6px 10px}.metric small{font-size:9px}.metric b{font-size:22px}.filters{display:grid;grid-template-columns:1fr 100px;gap:5px}.filters label{grid-column:2}.filters input{grid-column:1;grid-row:2}.filters select{font-size:11px;padding:6px}.filters #port{width:100%}#tabs{display:flex;gap:6px}#tabs button{flex:1;padding:4px;font-size:11px}#tabs button.active{border-color:var(--blue)}main{display:block}.flows{height:100%;padding:0 10px;grid-template-rows:34px 24px minmax(0,1fr) 36px}aside{display:none;height:100%;grid-template-columns:1fr 1fr;grid-template-rows:minmax(0,1fr) 105px}aside .firewall{grid-column:1/-1}body[data-tab=stats] .flows{display:none}body[data-tab=stats] aside{display:grid}.flow,.table-head{grid-template-columns:1fr 1fr 60px}.flow .protocol,.table-head .protocol{display:none}.flow{height:59px;font-size:10px}.state{font-size:9px}.flow small{font-size:9px}.status{font-size:10px}.status button{font-size:10px}.footer{font-size:8px}.footer .link-button{font-size:9px}aside .rank{line-height:19px}.block-value{display:inline-block;font-size:20px;margin:0 10px 0 0}.help{font-size:9px}.firewall h2{margin-bottom:4px}}
+</style></head><body data-tab="flows">
+<header><div><div class="eyebrow">PROXMOX / NETWORK OBSERVATORY</div><h1>Сетевая активность<span style="color:var(--lime)">.</span></h1></div><nav><a href="/traffic">HTTP / CrowdSec ↗</a><a class="dashboard" href="/">Dashboard</a><button id="pause">Пауза</button></nav></header>
+<div class="status"><b id="live" class="badge warn">ОЖИДАНИЕ</b><span id="source">Подключение к агенту…</span><button class="link-button" id="statusDetails">Диагностика ⓘ</button></div>
+<section class="metrics"><div class="metric"><small>TCP · ESTABLISHED</small><b id="established">—</b></div><div class="metric"><small>TCP · SYN_SENT / SYN_RECV</small><b id="pending">—</b></div><div class="metric"><small>UDP · отслеживаемые потоки</small><b id="udp">—</b></div><div class="metric"><small>ICMP / ICMPv6 · записи</small><b id="icmp">—</b></div></section>
+<div class="filters"><select id="protocol" aria-label="Протокол"><option value="">Все протоколы</option><option value="tcp">TCP</option><option value="udp">UDP</option><option value="icmp">ICMP / ICMPv6</option></select><select id="port" aria-label="Порт назначения"><option value="">Все порты</option></select><input id="search" placeholder="Найти IP или состояние…" aria-label="Поиск IP или состояния"><label><input id="external" type="checkbox">Внешние IP</label></div>
+<nav id="tabs"><button class="active" data-tab="flows">Потоки</button><button data-tab="stats">Статистика</button></nav>
+<main><section class="panel flows"><div class="heading"><h2>Текущая таблица conntrack</h2><small id="count">—</small></div><div class="table-head"><span>Инициатор</span><span>Назначение / NAT</span><span class="protocol">Протокол</span><span>Состояние</span><span class="volume">Байты → / ←</span></div><div id="rows"><div class="empty">Ожидание первого снимка…</div></div><div class="pager"><button id="prev" aria-label="Предыдущие потоки">←</button><span id="page">—</span><button id="next" aria-label="Следующие потоки">→</button><button id="resume">К свежим ↗</button></div></section>
+<aside><section class="panel"><h2>Топ инициаторов · потоки</h2><div id="topIPs"></div></section><section class="panel"><h2>Топ портов назначения</h2><div id="topPorts"></div></section><section class="panel firewall"><div class="heading"><h2>CrowdSec · firewall</h2><button class="link-button" id="firewallDetails">ⓘ</button></div><div id="blocked" class="block-value">—</div><div class="help" id="firewallNote">Ожидание счётчиков</div></section></aside></main>
+<footer class="footer"><span>Снимки ~5 с · короткие потоки могут быть пропущены</span><button class="link-button" id="about">Как читать данные ⓘ</button><a href="/#traffic-settings">Настройки ↗</a></footer>
+<dialog id="dialog"><div class="heading"><h2 id="title"></h2><button id="close" aria-label="Закрыть">×</button></div><div id="details"></div></dialog>
+<script>
+const el=id=>document.getElementById(id),safe=v=>String(v??'').replace(/[&<>"']/g,c=>({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;',"'":'&#39;'}[c])),num=v=>Number(v||0).toLocaleString('ru-RU'),bytes=n=>{n=Number(n||0);return n>=1073741824?(n/1073741824).toFixed(1)+' GiB':n>=1048576?(n/1048576).toFixed(1)+' MiB':n>=1024?(n/1024).toFixed(1)+' KiB':n+' B'},endpoint=(ip,p)=>p?(ip.includes(':')?'['+ip+']':ip)+':'+p:ip;
+const names=new Intl.DisplayNames(['ru'],{type:'region'}),country=c=>c==='LAN'?'Локальная сеть':c&&/^[A-Z]{2}$/.test(c)?names.of(c):'Без геолокации';
+let snapshot=null,frozen=null,page=0,visible=[],paused=false,busy=false,networkError='';
+const show=(title,html)=>{el('title').textContent=title;el('details').innerHTML=html;el('dialog').showModal()},fields=rows=>rows.map(([k,v])=>'<div class="detail"><small>'+safe(k)+'</small>'+safe(v)+'</div>').join('');
+function filterFlows(){const p=el('protocol').value,port=el('port').value,q=el('search').value.trim().toLowerCase();return ((frozen||snapshot)?.flows||[]).filter(f=>(!p||(p==='icmp'?f.protocol.startsWith('icmp'):f.protocol===p))&&(!port||String(f.destinationPort)===port)&&(!el('external').checked||f.external)&&(!q||[f.source,f.destination,f.replySource,f.replyDestination,f.state].join(' ').toLowerCase().includes(q)))}
+function rank(id,items){const counts=new Map();items.forEach(v=>counts.set(v,(counts.get(v)||0)+1));const h=el(id).parentElement.clientHeight;const limit=Math.max(1,Math.min(5,Math.floor((h-44)/22)));el(id).innerHTML=[...counts].sort((a,b)=>b[1]-a[1]||a[0].localeCompare(b[0])).slice(0,limit).map(([k,v])=>'<div class="rank"><span title="'+safe(k)+'">'+safe(k)+'</span><b>'+num(v)+'</b></div>').join('')||'<div class="help">Нет потоков</div>'}
+function render(){if(!snapshot)return;const data=frozen||snapshot,items=filterFlows();el('established').textContent=num(items.filter(f=>f.protocol==='tcp'&&f.state==='ESTABLISHED').length);el('pending').textContent=num(items.filter(f=>f.protocol==='tcp'&&['SYN_SENT','SYN_RECV'].includes(f.state)).length);el('udp').textContent=num(items.filter(f=>f.protocol==='udp').length);el('icmp').textContent=num(items.filter(f=>f.protocol.startsWith('icmp')).length);
+ if(!data.receivedAt||(data.error&&!(data.flows||[]).length)){['established','pending','udp','icmp'].forEach(id=>el(id).textContent='—')}
+ const fresh=snapshot.receivedAt&&Date.now()/1000-snapshot.receivedAt<25;el('live').textContent=networkError?'НЕТ СВЯЗИ':paused?'ПАУЗА':frozen?'СНИМОК':!fresh?'НЕТ ДАННЫХ':snapshot.error||snapshot.limited||snapshot.parseErrors?'НЕПОЛНЫЕ ДАННЫЕ':'● LIVE';el('live').classList.toggle('warn',!!networkError||!fresh||paused||!!snapshot.error||snapshot.limited||!!snapshot.parseErrors);el('source').textContent=(snapshot.agentName||'Агент')+' · '+(snapshot.at?new Date(data.at*1000).toLocaleTimeString('ru-RU'):'ожидание')+(snapshot.error?' · '+snapshot.error:snapshot.limited?' · лимит 5 000 потоков':'');
+ const rowHeight=innerWidth<=760?59:innerHeight<=700?46:55,count=Math.max(1,Math.floor(el('rows').clientHeight/rowHeight)),pages=Math.max(1,Math.ceil(items.length/count));page=Math.min(page,pages-1);visible=items.slice(page*count,(page+1)*count);el('count').textContent=num(items.length)+' / '+num(data.flows?.length)+' · '+(frozen?'снимок':'live');
+ el('rows').innerHTML=visible.map((f,i)=>{const nat=f.destination!==f.replySource||f.destinationPort!==f.replySourcePort;return '<article class="flow" role="button" tabindex="0" data-flow="'+i+'" aria-label="Поток '+safe(endpoint(f.source,f.sourcePort))+'"><div><strong>'+safe(endpoint(f.source,f.sourcePort))+'</strong><small>'+safe(country(f.country))+'</small></div><div><strong>'+safe(endpoint(f.destination,f.destinationPort))+'</strong><small>'+safe(nat?'NAT → '+endpoint(f.replySource,f.replySourcePort):country(f.destinationCountry))+'</small></div><div class="protocol">'+safe(f.protocol.toUpperCase())+'</div><div class="state '+(f.state==='ESTABLISHED'?'ok':f.state.includes('SYN')?'pending':'')+'">'+safe(f.state)+'<small>'+safe(f.protocol.toUpperCase())+'</small></div><div class="volume">'+(f.accounting?bytes(f.bytes)+'<small>'+bytes(f.replyBytes)+'</small>':'<small>нет учёта</small>')+'</div></article>'}).join('')||'<div class="empty">'+safe(snapshot.error||'Нет потоков по выбранным фильтрам')+'</div>';
+ el('page').textContent=(page+1)+' / '+pages;el('prev').disabled=page===0;el('next').disabled=page>=pages-1;rank('topIPs',items.map(f=>f.source));rank('topPorts',items.filter(f=>f.destinationPort).map(f=>f.protocol.toUpperCase()+' / '+f.destinationPort));
+ const counters=snapshot.firewall||[];el('blocked').textContent=counters.length?num(counters.reduce((a,c)=>a+c.packets,0))+' пак.':'—';el('firewallNote').textContent=counters.length?'С начала жизни счётчиков · '+bytes(counters.reduce((a,c)=>a+c.bytes,0))+' · '+new Date(snapshot.firewallAt*1000).toLocaleTimeString('ru-RU'):snapshot.firewallError||'Счётчики ещё не получены';
+}
+function openFlow(target){const row=target.closest('[data-flow]');if(!row)return;const f=visible[+row.dataset.flow];if(!f)return;show('Поток '+f.protocol.toUpperCase(),fields([['Инициатор',endpoint(f.source,f.sourcePort)+' · '+country(f.country)],['Исходное назначение',endpoint(f.destination,f.destinationPort)],['Обратное направление (NAT)',endpoint(f.replySource,f.replySourcePort)+' → '+endpoint(f.replyDestination,f.replyDestinationPort)],['Состояние / зона',f.state+' / '+(f.zone||'0')],['Инициатор → назначение',f.accounting?bytes(f.bytes)+' · '+num(f.packets)+' пакетов':'Учёт пакетов/байтов не включён для этого потока'],['Ответное направление',f.accounting?bytes(f.replyBytes)+' · '+num(f.replyPackets)+' пакетов':'Нет данных'],['Время снимка',new Date((frozen||snapshot).at*1000).toLocaleString('ru-RU')]]))}
+el('rows').onclick=e=>openFlow(e.target);el('rows').onkeydown=e=>{if(e.key==='Enter'||e.key===' '){e.preventDefault();openFlow(e.target)}};el('close').onclick=()=>el('dialog').close();
+el('statusDetails').onclick=()=>show('Диагностика агента',fields([['Сбор',snapshot?.error||'Ошибок conntrack не сообщено'],['Связь',networkError||'Последний ответ получен'],['Получен сервером',snapshot?.receivedAt?new Date(snapshot.receivedAt*1000).toLocaleString('ru-RU'):'Ещё не получен'],['Ограничения',snapshot?.limited?'Достигнут лимит 5 000 записей; статистика частичная':'До 5 000 записей в снимке'],['Нераспознанные строки',num(snapshot?.parseErrors)],['Firewall',snapshot?.firewallError||'Поддерживаемые счётчики получены']])+ '<p>На Debian/Ubuntu агент автоматически устанавливает отсутствующий <code>conntrack</code> через apt-get и systemd. Службе нужны root; для чтения таблицы — root/CAP_NET_ADMIN. Ошибка установки повторяется не чаще раза в 30 минут. Для firewall нужен <code>nft</code>. При выключенном учёте conntrack байты недоступны; sysctl и правила автоматически не изменяются.</p>');
+el('firewallDetails').onclick=()=>show('Счётчики блокировок',fields((snapshot?.firewall||[]).map(c=>[c.name,num(c.packets)+' пакетов · '+bytes(c.bytes)]))+'<p>'+safe(snapshot?.firewallError||'Накопительные счётчики правил CrowdSec непосредственно перед drop/reject. Сбрасываются при пересоздании правил. Не зависят от фильтров списка потоков; не показывают количество людей или HTTP-запросов.')+'</p>');
+el('about').onclick=()=>show('Как читать сетевые данные','<p>Это текущие отслеживаемые потоки TCP, UDP, ICMP и ICMPv6, а не журнал всех пакетов. Снимки делаются примерно раз в 5 секунд: короткие потоки могут быть пропущены. TIME_WAIT и другие состояния закрытия остаются в таблице до истечения таймера.</p><p>Инициатор и назначение — исходное направление conntrack. Обратная пара адресов показывает трансляцию NAT, включая Docker. Соединение NPM с upstream — отдельный поток; оно не считается ещё одним посетителем.</p><p>UDP ASSURED — отметка conntrack, не TCP-соединение. Порт 443 сам по себе не доказывает HTTPS или QUIC. SYN_SENT/SYN_RECV не означают атаку.</p><p>Байты и пакеты накоплены за жизнь конкретного потока, если учёт был включён при его создании. Длительность соединения и точный поток новых подключений по этим снимкам не вычисляются.</p><p>Фильтры влияют на карточки и топы. Firewall показывает отдельные накопительные счётчики без фильтров. Приватные и локальные IP не геолокируются. Геолокация: <a href="https://db-ip.com">DB-IP</a>, CC BY 4.0.</p>');
+function reset(){frozen=null;page=0;render()};['protocol','port','external'].forEach(id=>el(id).onchange=reset);el('search').oninput=reset;el('next').onclick=()=>{if(!frozen)frozen=snapshot;page++;render()};el('prev').onclick=()=>{page--;render()};el('resume').onclick=reset;el('pause').onclick=()=>{paused=!paused;el('pause').textContent=paused?'Продолжить':'Пауза';render();if(!paused)poll()};document.querySelectorAll('[data-tab]').forEach(b=>{if(b.tagName==='BUTTON')b.onclick=()=>{document.body.dataset.tab=b.dataset.tab;document.querySelectorAll('#tabs button').forEach(t=>t.classList.toggle('active',t===b));render()}});window.addEventListener('resize',render);
+async function poll(){if(paused||busy||document.hidden)return;busy=true;try{const response=await fetch('/api/network');if(response.status===404){location.replace('/traffic');return}if(!response.ok)throw Error('HTTP '+response.status);snapshot=await response.json();networkError='';const current=el('port').value,ports=[...new Set((snapshot.flows||[]).map(f=>f.destinationPort).filter(Boolean))];if(current&&!ports.includes(+current))ports.push(+current);ports.sort((a,b)=>a-b);el('port').innerHTML='<option value="">Все порты</option>'+ports.map(p=>'<option value="'+p+'">'+p+'</option>').join('');el('port').value=current;render()}catch(e){networkError='Не удалось обновить: '+e.message;el('source').textContent=networkError;el('live').textContent='НЕТ СВЯЗИ';el('live').classList.add('warn')}finally{busy=false}}
+poll();setInterval(poll,5000);document.addEventListener('visibilitychange',()=>{if(!document.hidden)poll()});
+</script></body></html>

backups.go

@@ -12,12 +12,15 @@ import (
 )
 
 type BackupItem struct {
-	VMID      int    `json:"vmid"`
-	Storage   string `json:"storage"`
-	VolumeID  string `json:"volid"`
-	Format    string `json:"format"`
-	SizeBytes uint64 `json:"size"`
-	CreatedAt int64  `json:"ctime"`
+	VMID              int      `json:"vmid"`
+	Storage           string   `json:"storage"`
+	VolumeID          string   `json:"volid"`
+	Format            string   `json:"format"`
+	SizeBytes         uint64   `json:"size"`
+	CreatedAt         int64    `json:"ctime"`
+	PreviousSizeBytes uint64   `json:"previousSize,omitempty"`
+	SizeChangePercent *float64 `json:"sizeChangePercent,omitempty"`
+	PreviousCreatedAt int64    `json:"previousCtime,omitempty"`
 }
 
 type BackupMetrics struct {
@@ -89,17 +92,25 @@ func readBackups() BackupMetrics {
 }
 
 func latestBackups(items []BackupItem) []BackupItem {
-	latest := make(map[int]BackupItem)
+	byGuest := make(map[int][]BackupItem)
 	for _, item := range items {
 		if item.VMID == 0 {
 			continue
 		}
-		if previous, ok := latest[item.VMID]; !ok || item.CreatedAt > previous.CreatedAt {
-			latest[item.VMID] = item
-		}
+		byGuest[item.VMID] = append(byGuest[item.VMID], item)
 	}
-	result := make([]BackupItem, 0, len(latest))
-	for _, item := range latest {
+	result := make([]BackupItem, 0, len(byGuest))
+	for _, history := range byGuest {
+		sort.Slice(history, func(i, j int) bool { return history[i].CreatedAt > history[j].CreatedAt })
+		item := history[0]
+		if len(history) > 1 {
+			item.PreviousSizeBytes = history[1].SizeBytes
+			item.PreviousCreatedAt = history[1].CreatedAt
+			if history[1].SizeBytes > 0 {
+				change := (float64(item.SizeBytes) - float64(history[1].SizeBytes)) / float64(history[1].SizeBytes) * 100
+				item.SizeChangePercent = &change
+			}
+		}
 		result = append(result, item)
 	}
 	sort.Slice(result, func(i, j int) bool { return result[i].VMID < result[j].VMID })

backups_test.go

@@ -12,4 +12,15 @@ func TestLatestBackups(t *testing.T) {
 	if len(latest) != 2 || latest[0].Storage != "new" {
 		t.Fatalf("неожиданный список последних копий: %+v", latest)
 	}
+	if latest[0].PreviousSizeBytes != 0 {
+		t.Fatalf("unexpected previous size: %+v", latest[0])
+	}
+}
+
+func TestLatestBackupsComparesSize(t *testing.T) {
+	items := []BackupItem{{VMID: 100, CreatedAt: 20, SizeBytes: 120}, {VMID: 100, CreatedAt: 10, SizeBytes: 100}}
+	latest := latestBackups(items)
+	if len(latest) != 1 || latest[0].SizeChangePercent == nil || *latest[0].SizeChangePercent != 20 {
+		t.Fatalf("unexpected comparison: %+v", latest)
+	}
 }

cpu.go

@@ -3,6 +3,7 @@ package main
 import (
 	"bufio"
 	"fmt"
+	"math"
 	"os"
 	"path/filepath"
 	"strconv"
@@ -33,14 +34,16 @@ type CPUStatic struct {
 
 type CPUMetrics struct {
 	CPUStatic
-	UsagePercent float64       `json:"usagePercent"`
-	FrequencyMHz float64       `json:"frequencyMhz"`
-	Load1        float64       `json:"load1"`
-	Load5        float64       `json:"load5"`
-	Load15       float64       `json:"load15"`
-	Uptime       uint64        `json:"uptimeSeconds"`
-	Temperatures []Temperature `json:"temperatures"`
-	CollectedAt  time.Time     `json:"collectedAt"`
+	UsagePercent       float64        `json:"usagePercent"`
+	FrequencyMHz       float64        `json:"frequencyMhz"`
+	Load1              float64        `json:"load1"`
+	Load5              float64        `json:"load5"`
+	Load15             float64        `json:"load15"`
+	Uptime             uint64         `json:"uptimeSeconds"`
+	TemperatureCelsius *float64       `json:"temperatureCelsius"`
+	Temperatures       []Temperature  `json:"temperatures"`
+	TopProcesses       []AgentProcess `json:"topProcesses"`
+	CollectedAt        time.Time      `json:"collectedAt"`
 }
 
 type Temperature struct {
@@ -89,10 +92,11 @@ func (c *cpuCollector) collect() (CPUMetrics, error) {
 	load1, load5, load15, _ := readLoadAverage()
 	uptime, _ := readUptime()
 
+	temperatures := readTemperatures()
 	return CPUMetrics{
 		CPUStatic: c.static, UsagePercent: usage, FrequencyMHz: frequency,
 		Load1: load1, Load5: load5, Load15: load15, Uptime: uptime,
-		Temperatures: readTemperatures(), CollectedAt: time.Now(),
+		Temperatures: temperatures, TemperatureCelsius: cpuTemperature(temperatures), TopProcesses: collectAgentProcesses(), CollectedAt: time.Now(),
 	}, nil
 }
 
@@ -284,3 +288,19 @@ func isCPUTemperatureChip(name string) bool {
 		return false
 	}
 }
+
+// cpuTemperature is the hottest CPU sensor, shared by live and historical views.
+// Do not average Tctl/Tdie with cooler cores or CCDs.
+func cpuTemperature(temperatures []Temperature) *float64 {
+	var hottest *float64
+	for _, t := range temperatures {
+		if math.IsNaN(t.Celsius) || math.IsInf(t.Celsius, 0) || t.Celsius <= 0 {
+			continue
+		}
+		if hottest == nil || t.Celsius > *hottest {
+			value := t.Celsius
+			hottest = &value
+		}
+	}
+	return hottest
+}

cpu_temperature_test.go

@@ -0,0 +1,24 @@
+package main
+
+import (
+	"math"
+	"testing"
+)
+
+func TestCPUTemperatureSharedWithHistory(t *testing.T) {
+	temperatures := []Temperature{{Label: "k10temp · Tctl", Celsius: 62}, {Label: "k10temp · Tccd1", Celsius: 38}, {Label: "k10temp · Tccd2", Celsius: 41}}
+	live := cpuTemperature(temperatures)
+	point := historyPointFromMetrics(dashboardMetrics{CPU: CPUMetrics{Temperatures: temperatures}})
+	if live == nil || *live != 62 || point.CPUTemperature != *live {
+		t.Fatalf("live/history disagree: %v, %+v", live, point)
+	}
+}
+
+func TestCPUTemperatureInvalidSensors(t *testing.T) {
+	if cpuTemperature([]Temperature{{Celsius: math.NaN()}, {Celsius: math.Inf(1)}, {Celsius: 0}}) != nil {
+		t.Fatal("invalid temperature selected")
+	}
+	if cpuTemperature(nil) != nil {
+		t.Fatal("missing sensor should stay missing")
+	}
+}

disks.go

@@ -13,40 +13,219 @@ import (
 )
 
 type DiskMetrics struct {
-	Name             string   `json:"name"`
-	Path             string   `json:"path"`
-	Model            string   `json:"model"`
-	Serial           string   `json:"serial"`
-	Type             string   `json:"type"`
-	Protocol         string   `json:"protocol"`
-	SizeBytes        uint64   `json:"sizeBytes"`
-	UsedBytes        *uint64  `json:"usedBytes"`
-	UsagePercent     *float64 `json:"usagePercent"`
-	SMARTStatus      string   `json:"smartStatus"`
-	SMARTAvailable   bool     `json:"smartAvailable"`
-	AttentionReasons []string `json:"attentionReasons"`
-	Temperature      *float64 `json:"temperatureCelsius"`
-	PowerOnHours     *uint64  `json:"powerOnHours"`
-	WearUsedPercent  *float64 `json:"wearUsedPercent"`
+	Name              string       `json:"name"`
+	Path              string       `json:"path"`
+	Model             string       `json:"model"`
+	Serial            string       `json:"serial"`
+	Type              string       `json:"type"`
+	Protocol          string       `json:"protocol"`
+	SizeBytes         uint64       `json:"sizeBytes"`
+	UsedBytes         *uint64      `json:"usedBytes"`
+	UsagePercent      *float64     `json:"usagePercent"`
+	SMARTStatus       string       `json:"smartStatus"`
+	SMARTAvailable    bool         `json:"smartAvailable"`
+	AttentionReasons  []string     `json:"attentionReasons"`
+	SMARTHistory      []string     `json:"smartHistory"`
+	Temperature       *float64     `json:"temperatureCelsius"`
+	PowerOnHours      *uint64      `json:"powerOnHours"`
+	WearUsedPercent   *float64     `json:"wearUsedPercent"`
+	Reallocated       uint64       `json:"reallocated"`
+	Pending           uint64       `json:"pending"`
+	Uncorrectable     uint64       `json:"uncorrectable"`
+	CRCErrors         uint64       `json:"crcErrors"`
+	MediaErrors       uint64       `json:"mediaErrors"`
+	SMARTErrorCount   uint64       `json:"smartErrorCount"`
+	CounterChanges    []string     `json:"counterChanges"`
+	LastSelfTest      string       `json:"lastSelfTest"`
+	LastSelfTestAt    string       `json:"lastSelfTestAt"`
+	SelfTestNeverRun  bool         `json:"selfTestNeverRun"`
+	ShortTestStarted  bool         `json:"shortTestStarted"`
+	ReadIOPS          float64      `json:"readIops"`
+	WriteIOPS         float64      `json:"writeIops"`
+	ReadBytesPerSec   float64      `json:"readBytesPerSec"`
+	WriteBytesPerSec  float64      `json:"writeBytesPerSec"`
+	ReadLatencyMs     float64      `json:"readLatencyMs"`
+	WriteLatencyMs    float64      `json:"writeLatencyMs"`
+	LatencyMs         float64      `json:"latencyMs"`
+	Utilization       float64      `json:"utilizationPercent"`
+	QueueDepth        uint64       `json:"queueDepth"`
+	AverageQueue      float64      `json:"averageQueue"`
+	IOPressureSeconds float64      `json:"ioPressureSeconds"`
+	IOCause           string       `json:"ioCause"`
+	IOCauseDetail     string       `json:"ioCauseDetail"`
+	IOConfidence      int          `json:"ioConfidence"`
+	TopConsumers      []IOConsumer `json:"topConsumers"`
 }
 
+type diskIOSnapshot struct{ reads, readSectors, readMS, writes, writeSectors, writeMS, inFlight, ioMS, weightedMS uint64 }
+
 type diskCollector struct {
-	mu        sync.Mutex
-	cached    []DiskMetrics
-	updatedAt time.Time
+	mu              sync.Mutex
+	cached          []DiskMetrics
+	updatedAt       time.Time
+	previous        map[string]DiskMetrics
+	missing         []string
+	lastTests       map[string]time.Time
+	ioPrevious      map[string]diskIOSnapshot
+	ioUpdatedAt     time.Time
+	ioPressureSince map[string]time.Time
 }
 
-func (c *diskCollector) collect() []DiskMetrics {
+func (c *diskCollector) collect(thresholds AlertThresholds) []DiskMetrics {
 	c.mu.Lock()
 	defer c.mu.Unlock()
 	if time.Since(c.updatedAt) < time.Minute && c.cached != nil {
+		c.applyDiskIO(c.cached, thresholds)
 		return c.cached
 	}
-	c.cached = readPhysicalDisks()
+	current := readPhysicalDisks()
+	if c.previous == nil {
+		c.previous = map[string]DiskMetrics{}
+		c.lastTests = map[string]time.Time{}
+	}
+	seen := map[string]bool{}
+	c.missing = nil
+	for i := range current {
+		disk := &current[i]
+		key := disk.Serial
+		if key == "" || key == "Не указан" {
+			key = disk.Path
+		}
+		seen[key] = true
+		if old, ok := c.previous[key]; ok {
+			disk.CounterChanges = diskCounterChanges(old, *disk)
+			disk.AttentionReasons = append(disk.AttentionReasons, disk.CounterChanges...)
+		}
+		if disk.SMARTAvailable && (disk.SelfTestNeverRun || selfTestOlderThanWeek(disk.LastSelfTestAt)) && time.Since(c.lastTests[key]) >= 7*24*time.Hour {
+			disk.ShortTestStarted = startShortSMARTTest(disk.Path)
+			if disk.ShortTestStarted {
+				c.lastTests[key] = time.Now()
+			}
+		}
+		c.previous[key] = *disk
+	}
+	for key, disk := range c.previous {
+		if !seen[key] {
+			c.missing = append(c.missing, disk.Model+" ("+disk.Serial+")")
+		}
+	}
+	c.cached = current
+	c.applyDiskIO(c.cached, thresholds)
 	c.updatedAt = time.Now()
 	return c.cached
 }
 
+func (c *diskCollector) applyDiskIO(disks []DiskMetrics, thresholds AlertThresholds) {
+	now := time.Now()
+	snapshots := readDiskIOStats("/proc/diskstats")
+	if c.ioPrevious == nil {
+		c.ioPrevious = map[string]diskIOSnapshot{}
+		c.ioPressureSince = map[string]time.Time{}
+	}
+	elapsed := now.Sub(c.ioUpdatedAt).Seconds()
+	for i := range disks {
+		disk := &disks[i]
+		disk.ReadIOPS, disk.WriteIOPS, disk.ReadBytesPerSec, disk.WriteBytesPerSec = 0, 0, 0, 0
+		disk.ReadLatencyMs, disk.WriteLatencyMs, disk.LatencyMs, disk.Utilization, disk.AverageQueue, disk.IOPressureSeconds = 0, 0, 0, 0, 0, 0
+		current, ok := snapshots[disk.Name]
+		if !ok {
+			continue
+		}
+		disk.QueueDepth = current.inFlight
+		if old, found := c.ioPrevious[disk.Name]; found && elapsed > 0 && diskIOCountersMonotonic(old, current) {
+			readOps := current.reads - old.reads
+			writeOps := current.writes - old.writes
+			disk.ReadIOPS = float64(readOps) / elapsed
+			disk.WriteIOPS = float64(writeOps) / elapsed
+			disk.ReadBytesPerSec = float64(current.readSectors-old.readSectors) * 512 / elapsed
+			disk.WriteBytesPerSec = float64(current.writeSectors-old.writeSectors) * 512 / elapsed
+			if readOps > 0 {
+				disk.ReadLatencyMs = float64(current.readMS-old.readMS) / float64(readOps)
+			}
+			if writeOps > 0 {
+				disk.WriteLatencyMs = float64(current.writeMS-old.writeMS) / float64(writeOps)
+			}
+			if readOps+writeOps > 0 {
+				disk.LatencyMs = float64(current.readMS-old.readMS+current.writeMS-old.writeMS) / float64(readOps+writeOps)
+			}
+			disk.Utilization = min(float64(current.ioMS-old.ioMS)/(elapsed*10), 100)
+			disk.AverageQueue = float64(current.weightedMS-old.weightedMS) / (elapsed * 1000)
+			utilLimit, latencyLimit, queueLimit := thresholds.DiskSSDUtil, thresholds.DiskSSDLatency, thresholds.DiskSSDQueue
+			if disk.Type == "HDD" {
+				utilLimit, latencyLimit, queueLimit = thresholds.DiskHDDUtil, thresholds.DiskHDDLatency, thresholds.DiskHDDQueue
+			} else if strings.Contains(disk.Type, "NVMe") {
+				utilLimit, latencyLimit, queueLimit = thresholds.DiskNVMeUtil, thresholds.DiskNVMeLatency, thresholds.DiskNVMeQueue
+			}
+			pressured := disk.Utilization >= utilLimit || (disk.LatencyMs >= latencyLimit && disk.ReadIOPS+disk.WriteIOPS >= 1) || disk.AverageQueue >= queueLimit
+			if pressured {
+				if c.ioPressureSince[disk.Name].IsZero() {
+					c.ioPressureSince[disk.Name] = now
+				}
+				disk.IOPressureSeconds = now.Sub(c.ioPressureSince[disk.Name]).Seconds()
+			} else {
+				delete(c.ioPressureSince, disk.Name)
+			}
+		}
+		c.ioPrevious[disk.Name] = current
+	}
+	c.ioUpdatedAt = now
+}
+
+func diskIOCountersMonotonic(old, current diskIOSnapshot) bool {
+	return current.reads >= old.reads && current.readSectors >= old.readSectors && current.readMS >= old.readMS && current.writes >= old.writes && current.writeSectors >= old.writeSectors && current.writeMS >= old.writeMS && current.ioMS >= old.ioMS && current.weightedMS >= old.weightedMS
+}
+
+func readDiskIOStats(path string) map[string]diskIOSnapshot {
+	data, err := os.ReadFile(path)
+	if err != nil {
+		return nil
+	}
+	result := map[string]diskIOSnapshot{}
+	for _, line := range strings.Split(string(data), "\n") {
+		f := strings.Fields(line)
+		if len(f) < 14 || !isPhysicalDiskName(f[2]) {
+			continue
+		}
+		values := make([]uint64, 11)
+		for i := 0; i < 11; i++ {
+			values[i], _ = strconv.ParseUint(f[i+3], 10, 64)
+		}
+		result[f[2]] = diskIOSnapshot{reads: values[0], readSectors: values[2], readMS: values[3], writes: values[4], writeSectors: values[6], writeMS: values[7], inFlight: values[8], ioMS: values[9], weightedMS: values[10]}
+	}
+	return result
+}
+
+func (c *diskCollector) missingDisks() []string {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	return append([]string(nil), c.missing...)
+}
+
+func diskCounterChanges(old, current DiskMetrics) []string {
+	checks := []struct {
+		label    string
+		old, now uint64
+	}{{"Reallocated вырос", old.Reallocated, current.Reallocated}, {"Pending вырос", old.Pending, current.Pending}, {"Uncorrectable вырос", old.Uncorrectable, current.Uncorrectable}, {"CRC-ошибки выросли", old.CRCErrors, current.CRCErrors}, {"Media errors выросли", old.MediaErrors, current.MediaErrors}, {"Ошибки SMART-журнала выросли", old.SMARTErrorCount, current.SMARTErrorCount}}
+	var changes []string
+	for _, check := range checks {
+		if check.now > check.old {
+			changes = append(changes, check.label+": "+strconv.FormatUint(check.old, 10)+" → "+strconv.FormatUint(check.now, 10))
+		}
+	}
+	return changes
+}
+
+func selfTestOlderThanWeek(value string) bool {
+	when, err := time.Parse(time.RFC3339, value)
+	return err == nil && time.Since(when) > 7*24*time.Hour
+}
+
+func startShortSMARTTest(path string) bool {
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+	return exec.CommandContext(ctx, "smartctl", "-t", "short", path).Run() == nil
+}
+
 func readPhysicalDisks() []DiskMetrics {
 	entries, _ := os.ReadDir("/sys/block")
 	usage := readDiskUsage()
@@ -202,6 +381,27 @@ type smartctlJSON struct {
 			Count int `json:"count"`
 		} `json:"summary"`
 	} `json:"ata_smart_error_log"`
+	ATASelfTestLog struct {
+		Standard struct {
+			Table []struct {
+				Status struct {
+					String string `json:"string"`
+				} `json:"status"`
+				LifetimeHours uint64 `json:"lifetime_hours"`
+			} `json:"table"`
+		} `json:"standard"`
+	} `json:"ata_smart_self_test_log"`
+	NVMeSelfTestLog struct {
+		Table []struct {
+			SelfTestCode struct {
+				String string `json:"string"`
+			} `json:"self_test_code"`
+			SelfTestResult struct {
+				String string `json:"string"`
+			} `json:"self_test_result"`
+			PowerOnHours uint64 `json:"power_on_hours"`
+		} `json:"table"`
+	} `json:"nvme_self_test_log"`
 	NVMeHealth *struct {
 		Temperature     float64 `json:"temperature"`
 		PercentageUsed  float64 `json:"percentage_used"`
@@ -221,6 +421,10 @@ func applySMART(disk *DiskMetrics) {
 	if len(output) == 0 {
 		return
 	}
+	applySMARTJSON(disk, output)
+}
+
+func applySMARTJSON(disk *DiskMetrics, output []byte) {
 	var smart smartctlJSON
 	if json.Unmarshal(output, &smart) != nil {
 		return
@@ -254,6 +458,7 @@ func applySMART(disk *DiskMetrics) {
 	}
 	if smart.NVMeHealth != nil {
 		disk.WearUsedPercent = &smart.NVMeHealth.PercentageUsed
+		disk.MediaErrors = smart.NVMeHealth.MediaErrors
 		if smart.NVMeHealth.CriticalWarning != 0 {
 			disk.AttentionReasons = append(disk.AttentionReasons, "NVMe сообщает критическое предупреждение")
 		}
@@ -261,6 +466,25 @@ func applySMART(disk *DiskMetrics) {
 			disk.AttentionReasons = append(disk.AttentionReasons, "Ошибки целостности носителя: "+strconv.FormatUint(smart.NVMeHealth.MediaErrors, 10))
 		}
 	}
+	if len(smart.NVMeSelfTestLog.Table) > 0 {
+		last := smart.NVMeSelfTestLog.Table[0]
+		description := strings.TrimSpace(last.SelfTestCode.String)
+		result := firstNonEmpty(strings.TrimSpace(last.SelfTestResult.String), "Результат неизвестен")
+		if description != "" {
+			disk.LastSelfTest = description + " · " + result
+		} else {
+			disk.LastSelfTest = result
+		}
+		setLastSelfTestAt(disk, last.PowerOnHours)
+	} else if len(smart.ATASelfTestLog.Standard.Table) > 0 {
+		last := smart.ATASelfTestLog.Standard.Table[0]
+		disk.LastSelfTest = firstNonEmpty(last.Status.String, "Результат неизвестен")
+		setLastSelfTestAt(disk, last.LifetimeHours)
+	} else {
+		disk.SelfTestNeverRun = true
+		disk.LastSelfTest = "Никогда не запускался"
+		disk.AttentionReasons = append(disk.AttentionReasons, "SMART self-test ещё ни разу не запускался")
+	}
 	applyATAAttributes(disk, smart)
 	applySMARTExitStatus(disk, smart.Smartctl.ExitStatus)
 	if len(disk.AttentionReasons) > 0 && disk.SMARTStatus == "Исправен" {
@@ -268,6 +492,12 @@ func applySMART(disk *DiskMetrics) {
 	}
 }
 
+func setLastSelfTestAt(disk *DiskMetrics, testPowerOnHours uint64) {
+	if disk.PowerOnHours != nil && *disk.PowerOnHours >= testPowerOnHours {
+		disk.LastSelfTestAt = time.Now().Add(-time.Duration(*disk.PowerOnHours-testPowerOnHours) * time.Hour).Format(time.RFC3339)
+	}
+}
+
 func applyATAAttributes(disk *DiskMetrics, smart smartctlJSON) {
 	wearNames := map[string]bool{
 		"Percent_Lifetime_Remain": true, "SSD_Life_Left": true,
@@ -275,20 +505,37 @@ func applyATAAttributes(disk *DiskMetrics, smart smartctlJSON) {
 		"Wear_Leveling_Count": true,
 	}
 	for _, attribute := range smart.ATAAttributes.Table {
+		switch attribute.Name {
+		case "Reallocated_Sector_Ct", "Reallocated_Event_Count":
+			disk.Reallocated += uint64(max(attribute.Raw.Value, 0))
+		case "Current_Pending_Sector":
+			disk.Pending = uint64(max(attribute.Raw.Value, 0))
+		case "Offline_Uncorrectable", "Reported_Uncorrect":
+			disk.Uncorrectable += uint64(max(attribute.Raw.Value, 0))
+		case "UDMA_CRC_Error_Count", "Interface_CRC_Error_Count":
+			disk.CRCErrors += uint64(max(attribute.Raw.Value, 0))
+		}
 		if wearNames[attribute.Name] && attribute.Value >= 0 && attribute.Value <= 100 {
 			wear := float64(100 - attribute.Value)
 			disk.WearUsedPercent = &wear
 		}
-		if attribute.WhenFailed != "" && attribute.WhenFailed != "-" {
+		if strings.EqualFold(attribute.WhenFailed, "FAILING_NOW") {
 			reason := attribute.Name + " вышел за порог"
 			if attribute.Raw.String != "" {
 				reason += " (" + attribute.Raw.String + ")"
 			}
 			disk.AttentionReasons = append(disk.AttentionReasons, reason)
+		} else if attribute.WhenFailed != "" && attribute.WhenFailed != "-" {
+			reason := attribute.Name + " ранее выходил за порог"
+			if attribute.Raw.String != "" {
+				reason += " (" + attribute.Raw.String + ")"
+			}
+			disk.SMARTHistory = append(disk.SMARTHistory, reason)
 		}
 	}
 	if smart.ATAErrorLog.Summary.Count > 0 {
-		disk.AttentionReasons = append(disk.AttentionReasons, "Ошибок в SMART-журнале: "+strconv.Itoa(smart.ATAErrorLog.Summary.Count))
+		disk.SMARTErrorCount = uint64(smart.ATAErrorLog.Summary.Count)
+		disk.SMARTHistory = append(disk.SMARTHistory, "Записей в SMART-журнале: "+strconv.Itoa(smart.ATAErrorLog.Summary.Count))
 	}
 }
 
@@ -302,12 +549,12 @@ func applySMARTExitStatus(disk *DiskMetrics, status int) {
 		disk.AttentionReasons = append(disk.AttentionReasons, "Критический SMART-атрибут достиг порога")
 	}
 	if status&32 != 0 {
-		disk.AttentionReasons = append(disk.AttentionReasons, "SMART-атрибут ранее находился ниже порога")
+		disk.SMARTHistory = append(disk.SMARTHistory, "SMART-атрибут ранее находился ниже порога")
 	}
 	if status&64 != 0 {
-		disk.AttentionReasons = append(disk.AttentionReasons, "В журнале SMART обнаружены ошибки")
+		disk.SMARTHistory = append(disk.SMARTHistory, "В журнале SMART есть старые записи об ошибках")
 	}
 	if status&128 != 0 {
-		disk.AttentionReasons = append(disk.AttentionReasons, "В журнале самотестирования есть ошибки")
+		disk.SMARTHistory = append(disk.SMARTHistory, "В журнале самотестирования есть старые ошибки")
 	}
 }

disks_test.go

@@ -2,9 +2,48 @@ package main
 
 import (
 	"encoding/json"
+	"os"
+	"path/filepath"
+	"strings"
 	"testing"
+	"time"
 )
 
+func TestApplySMARTJSONReadsNVMeSelfTestHistory(t *testing.T) {
+	disk := DiskMetrics{Path: "/dev/nvme0n1"}
+	output := []byte(`{
+		"model_name":"ADATA LEGEND 900",
+		"smart_status":{"passed":true},
+		"power_on_time":{"hours":960},
+		"nvme_self_test_log":{
+			"current_self_test_operation":{"value":0,"string":"No self-test in progress"},
+			"table":[{
+				"self_test_code":{"value":1,"string":"Short"},
+				"self_test_result":{"value":0,"string":"Completed without error"},
+				"power_on_hours":959
+			}]
+		}
+	}`)
+
+	before := time.Now().Add(-2 * time.Hour)
+	applySMARTJSON(&disk, output)
+	after := time.Now()
+
+	if disk.SelfTestNeverRun {
+		t.Fatal("завершённый NVMe self-test ошибочно отмечен как никогда не запускавшийся")
+	}
+	if disk.LastSelfTest != "Short · Completed without error" {
+		t.Fatalf("неожиданный результат self-test: %q", disk.LastSelfTest)
+	}
+	parsed, err := time.Parse(time.RFC3339, disk.LastSelfTestAt)
+	if err != nil || parsed.Before(before) || parsed.After(after) {
+		t.Fatalf("неверно вычислено время self-test: %q, err=%v", disk.LastSelfTestAt, err)
+	}
+	if len(disk.AttentionReasons) != 0 || disk.SMARTStatus != "Исправен" {
+		t.Fatalf("исправный NVMe требует внимания: %+v", disk)
+	}
+}
+
 func TestIsPhysicalDiskName(t *testing.T) {
 	tests := map[string]bool{
 		"sda": true, "sdb": true, "nvme0n1": true,
@@ -17,6 +56,22 @@ func TestIsPhysicalDiskName(t *testing.T) {
 	}
 }
 
+func TestReadDiskIOStats(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "diskstats")
+	data := "   8       0 sda 100 2 300 40 50 3 700 80 4 90 120 0 0 0 0 0 0\n   7       0 loop0 999 0 999 0 999 0 999 0 0 0 0\n"
+	if err := os.WriteFile(path, []byte(data), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	stats := readDiskIOStats(path)
+	disk, ok := stats["sda"]
+	if !ok || disk.reads != 100 || disk.readSectors != 300 || disk.writes != 50 || disk.writeSectors != 700 || disk.inFlight != 4 || disk.ioMS != 90 || disk.weightedMS != 120 {
+		t.Fatalf("unexpected stats: %+v", disk)
+	}
+	if _, exists := stats["loop0"]; exists {
+		t.Fatal("virtual device must be ignored")
+	}
+}
+
 func TestSumFilesystemUsage(t *testing.T) {
 	device := lsblkDevice{Children: []lsblkDevice{
 		{FSUsed: json.RawMessage(`100`), FSAvail: json.RawMessage(`300`)},
@@ -50,3 +105,37 @@ func TestApplyATAWearAndReason(t *testing.T) {
 		t.Fatalf("причина предупреждения не добавлена: %+v", disk)
 	}
 }
+
+func TestHistoricalATAFailureDoesNotRequireAttention(t *testing.T) {
+	var smart smartctlJSON
+	attribute := struct {
+		Name       string `json:"name"`
+		Value      int    `json:"value"`
+		Threshold  int    `json:"thresh"`
+		WhenFailed string `json:"when_failed"`
+		Raw        struct {
+			String string `json:"string"`
+			Value  int64  `json:"value"`
+		} `json:"raw"`
+	}{Name: "Airflow_Temperature_Cel", WhenFailed: "In_the_past"}
+	attribute.Raw.String = "46 (Min/Max 26/47 #15)"
+	smart.ATAAttributes.Table = append(smart.ATAAttributes.Table, attribute)
+	disk := DiskMetrics{}
+	applyATAAttributes(&disk, smart)
+	applySMARTExitStatus(&disk, 32|64|128)
+	if len(disk.AttentionReasons) != 0 {
+		t.Fatalf("историческая ошибка стала активной: %+v", disk.AttentionReasons)
+	}
+	if len(disk.SMARTHistory) != 4 {
+		t.Fatalf("история SMART потеряна: %+v", disk.SMARTHistory)
+	}
+}
+
+func TestSMARTErrorLogGrowthRequiresAttention(t *testing.T) {
+	old := DiskMetrics{SMARTErrorCount: 2}
+	current := DiskMetrics{SMARTErrorCount: 3}
+	changes := diskCounterChanges(old, current)
+	if len(changes) != 1 || !strings.Contains(changes[0], "2 → 3") {
+		t.Fatalf("рост журнала не обнаружен: %+v", changes)
+	}
+}

email.go

@@ -0,0 +1,193 @@
+package main
+
+import (
+	"crypto/tls"
+	"fmt"
+	"html"
+	"log"
+	"mime"
+	"net"
+	"net/smtp"
+	"sort"
+	"strings"
+	"time"
+)
+
+func filterEmailAlerts(alerts []Alert, settings EmailSettings) []Alert {
+	allowedSeverity, allowedSource := map[string]bool{}, map[string]bool{}
+	for _, value := range settings.Severities {
+		allowedSeverity[value] = true
+	}
+	for _, value := range settings.Sources {
+		allowedSource[value] = true
+	}
+	var result []Alert
+	for _, alert := range alerts {
+		if allowedSeverity[alert.Severity] && allowedSource[alert.Source] {
+			result = append(result, alert)
+		}
+	}
+	return result
+}
+
+func sendAlertEmail(settings EmailSettings, alerts []Alert, test bool) error {
+	subject := "Proxmox Dashboard: сводка уведомлений"
+	body := `<div style="padding:32px;background:#0b1117;color:#edf3f7;font-family:Arial,sans-serif"><div style="max-width:680px;margin:auto"><div style="color:#55d58a;font-size:12px;letter-spacing:2px">PROXMOX DASHBOARD</div><h1 style="margin:10px 0 24px">Сводка событий</h1>`
+	if test {
+		subject = "Proxmox Dashboard: тест почты"
+		body += `<div style="padding:22px;border:1px solid #2b3944;border-radius:14px;background:#111a22"><b style="font-size:18px">Почта настроена</b><p style="color:#9aabb7;margin-bottom:0">Тестовое сообщение успешно отправлено вашим сервером.</p></div>`
+	}
+	if len(alerts) > 0 {
+		groups := map[string][]Alert{}
+		for _, alert := range alerts {
+			groups[alert.Severity] = append(groups[alert.Severity], alert)
+		}
+		var sections []string
+		for _, severity := range []string{"critical", "warning", "info"} {
+			items := groups[severity]
+			if len(items) == 0 {
+				continue
+			}
+			title := map[string]string{"critical": "Критические", "warning": "Предупреждения", "info": "Информация"}[severity]
+			color := map[string]string{"critical": "#ff665c", "warning": "#ffb35c", "info": "#6ba8ff"}[severity]
+			sort.Slice(items, func(i, j int) bool { return items[i].Title < items[j].Title })
+			cards := ""
+			for _, item := range items {
+				cards += fmt.Sprintf(`<div style="margin:10px 0;padding:16px 18px;border:1px solid #2b3944;border-left:4px solid %s;border-radius:10px;background:#111a22"><div style="color:#8fa0ac;font-size:11px;text-transform:uppercase">%s</div><div style="font-weight:bold;font-size:16px;margin:5px 0">%s</div><div style="color:#aab7c0;font-size:14px">%s</div></div>`, color, html.EscapeString(item.Source), html.EscapeString(item.Title), html.EscapeString(item.Message))
+			}
+			sections = append(sections, fmt.Sprintf(`<h2 style="font-size:14px;color:%s;margin:24px 0 8px">%s · %d</h2>%s`, color, title, len(items), cards))
+		}
+		body += strings.Join(sections, "")
+		subject = fmt.Sprintf("Proxmox Dashboard: событий — %d", len(alerts))
+	}
+	body += `<div style="color:#687985;font-size:12px;margin-top:28px">События собраны напрямую с Proxmox-хоста. Подтверждённые события повторно не отправляются.</div></div></div>`
+	message := []byte("From: " + settings.From + "\r\nTo: " + settings.To + "\r\nSubject: " + mime.QEncoding.Encode("UTF-8", subject) + "\r\nMIME-Version: 1.0\r\nContent-Type: text/html; charset=UTF-8\r\nContent-Transfer-Encoding: 8bit\r\n\r\n" + body + "\r\n")
+	address := net.JoinHostPort(settings.Host, fmt.Sprint(settings.Port))
+	tlsConfig := &tls.Config{ServerName: settings.Host, MinVersion: tls.VersionTLS12, InsecureSkipVerify: settings.SkipTLSVerify} //nolint:gosec -- explicit private-network option
+	var client *smtp.Client
+	if settings.Port == 465 {
+		connection, err := tls.Dial("tcp", address, tlsConfig)
+		if err != nil {
+			return err
+		}
+		client, err = smtp.NewClient(connection, settings.Host)
+		if err != nil {
+			return err
+		}
+	} else {
+		var err error
+		client, err = smtp.Dial(address)
+		if err != nil {
+			return err
+		}
+		if ok, _ := client.Extension("STARTTLS"); ok {
+			if err = client.StartTLS(tlsConfig); err != nil {
+				return err
+			}
+		}
+	}
+	defer client.Close()
+	if settings.Username != "" {
+		if err := client.Auth(smtp.PlainAuth("", settings.Username, settings.Password, settings.Host)); err != nil {
+			return err
+		}
+	}
+	if err := client.Mail(settings.From); err != nil {
+		return err
+	}
+	if err := client.Rcpt(settings.To); err != nil {
+		return err
+	}
+	writer, err := client.Data()
+	if err != nil {
+		return err
+	}
+	if _, err = writer.Write(message); err != nil {
+		return err
+	}
+	if err = writer.Close(); err != nil {
+		return err
+	}
+	return client.Quit()
+}
+
+func sendRecoveryEmail(settings EmailSettings, items []RecoveryEvent) error {
+	alerts := make([]Alert, 0, len(items))
+	for _, v := range items {
+		alerts = append(alerts, Alert{ID: v.ID, Severity: "info", Source: v.Source, Title: "Восстановлено: " + v.Title, Message: "Проблема больше не наблюдается."})
+	}
+	return sendAlertEmail(settings, alerts, false)
+}
+
+func runImmediateNotifier(collect func() (dashboardMetrics, error), store *Store) {
+	ticker := time.NewTicker(5 * time.Second)
+	defer ticker.Stop()
+	for range ticker.C {
+		settings := store.EmailSettings(true)
+		if !settings.Enabled {
+			continue
+		}
+		if _, err := collect(); err != nil {
+			continue
+		}
+		urgent, err := store.PendingImmediateAlerts()
+		if err == nil && len(urgent) > 0 {
+			urgent = filterEmailAlerts(urgent, settings)
+			if len(urgent) > 0 && sendAlertEmail(settings, urgent, false) == nil {
+				_ = store.MarkAlertsEmailed(urgent)
+			}
+		}
+		recoveries, err := store.PendingRecoveries()
+		if err == nil && len(recoveries) > 0 {
+			var filtered []RecoveryEvent
+			for _, v := range recoveries {
+				if contains(settings.Sources, v.Source) {
+					filtered = append(filtered, v)
+				}
+			}
+			if len(filtered) > 0 && sendRecoveryEmail(settings, filtered) == nil {
+				_ = store.MarkRecoveriesEmailed(filtered)
+			}
+		}
+	}
+}
+func contains(items []string, want string) bool {
+	for _, v := range items {
+		if v == want {
+			return true
+		}
+	}
+	return false
+}
+
+func runEmailNotifier(collect func() (dashboardMetrics, error), store *Store) {
+	ticker := time.NewTicker(time.Minute)
+	defer ticker.Stop()
+	for range ticker.C {
+		settings := store.EmailSettings(true)
+		if !settings.Enabled || time.Since(store.LastEmailAt()) < time.Duration(settings.IntervalMinutes)*time.Minute {
+			continue
+		}
+		_, err := collect()
+		if err != nil {
+			continue
+		}
+		pending, err := store.PendingEmailAlerts()
+		if err != nil {
+			continue
+		}
+		alerts := filterEmailAlerts(pending, settings)
+		if len(alerts) == 0 {
+			continue
+		}
+		if err := sendAlertEmail(settings, alerts, false); err != nil {
+			log.Printf("Не удалось отправить почтовую сводку: %v", err)
+			continue
+		}
+		if err := store.MarkAlertsEmailed(alerts); err != nil {
+			log.Printf("Не удалось отметить отправленные события: %v", err)
+			continue
+		}
+		store.SetLastEmailAt(time.Now())
+	}
+}

email_test.go

@@ -0,0 +1,40 @@
+package main
+
+import "testing"
+
+func TestFilterEmailAlerts(t *testing.T) {
+	alerts := []Alert{
+		{ID: "cpu", Severity: "critical", Source: "CPU"},
+		{ID: "backup", Severity: "critical", Source: "Backups"},
+		{ID: "info", Severity: "info", Source: "CPU"},
+	}
+	settings := EmailSettings{Severities: []string{"critical"}, Sources: []string{"CPU"}}
+	filtered := filterEmailAlerts(alerts, settings)
+	if len(filtered) != 1 || filtered[0].ID != "cpu" {
+		t.Fatalf("unexpected filtered alerts: %+v", filtered)
+	}
+}
+
+func TestEmailPasswordIsPreserved(t *testing.T) {
+	store, err := openStore(":memory:")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer store.Close()
+	settings := defaultEmailSettings()
+	settings.Host, settings.From, settings.To, settings.Password = "smtp.example.com", "dash@example.com", "admin@example.com", "secret"
+	if err := store.SaveEmailSettings(settings); err != nil {
+		t.Fatal(err)
+	}
+	settings.Password = ""
+	settings.IntervalMinutes = 120
+	if err := store.SaveEmailSettings(settings); err != nil {
+		t.Fatal(err)
+	}
+	if got := store.EmailSettings(true); got.Password != "secret" || got.IntervalMinutes != 120 {
+		t.Fatalf("unexpected settings: %+v", got)
+	}
+	if got := store.EmailSettings(false); got.Password != "" || !got.HasPassword {
+		t.Fatalf("password leaked or marker lost: %+v", got)
+	}
+}

go.mod

@@ -2,7 +2,10 @@ module proxmox-host-info
 
 go 1.25.0
 
-require modernc.org/sqlite v1.54.0
+require (
+	github.com/oschwald/maxminddb-golang v1.13.1
+	modernc.org/sqlite v1.54.0
+)
 
 require (
 	github.com/dustin/go-humanize v1.0.1 // indirect

go.sum

@@ -1,3 +1,5 @@
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
 github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
 github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e h1:ijClszYn+mADRFY17kjQEVQ1XRhq2/JR1M3sGqeJoxs=
@@ -10,8 +12,14 @@ github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWE
 github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
 github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
 github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
+github.com/oschwald/maxminddb-golang v1.13.1 h1:G3wwjdN9JmIK2o/ermkHM+98oX5fS+k5MbwsmL4MRQE=
+github.com/oschwald/maxminddb-golang v1.13.1/go.mod h1:K4pgV9N/GcK694KSTmVSDTODk4IsCNThNdTmnaBZ/F8=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
 github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
+github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
+github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
 golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ=
 golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
 golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
@@ -21,6 +29,8 @@ golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw=
 golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
 golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
 golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
+gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 modernc.org/cc/v4 v4.29.0 h1:CXgwL8cvxmyzBQZzbSl/6xFtMCryb6u8IOqDci39cgc=
 modernc.org/cc/v4 v4.29.0/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI=
 modernc.org/ccgo/v4 v4.34.6 h1:sBgfIwyN0TQ9C5hwIeuqyeAKyMWnbvj2fvpF4L11uzU=

guests.go

@@ -3,24 +3,35 @@ package main
 import (
 	"context"
 	"encoding/json"
+	"fmt"
 	"os/exec"
 	"sort"
+	"strconv"
+	"strings"
 	"sync"
 	"time"
 )
 
 type GuestMetrics struct {
-	VMID        int     `json:"vmid"`
-	Name        string  `json:"name"`
-	Type        string  `json:"type"`
-	Node        string  `json:"node"`
-	Status      string  `json:"status"`
-	CPU         float64 `json:"cpu"`
-	MaxCPU      float64 `json:"maxcpu"`
-	MemoryBytes uint64  `json:"mem"`
-	MaxMemory   uint64  `json:"maxmem"`
-	Uptime      uint64  `json:"uptime"`
-	Template    int     `json:"template"`
+	VMID        int      `json:"vmid"`
+	Name        string   `json:"name"`
+	Type        string   `json:"type"`
+	Node        string   `json:"node"`
+	Status      string   `json:"status"`
+	CPU         float64  `json:"cpu"`
+	MaxCPU      float64  `json:"maxcpu"`
+	MemoryBytes uint64   `json:"mem"`
+	MaxMemory   uint64   `json:"maxmem"`
+	Uptime      uint64   `json:"uptime"`
+	Template    int      `json:"template"`
+	DiskBytes   uint64   `json:"disk"`
+	MaxDisk     uint64   `json:"maxdisk"`
+	NetInBytes  uint64   `json:"netin"`
+	NetOutBytes uint64   `json:"netout"`
+	OnBoot      bool     `json:"onboot"`
+	Agent       bool     `json:"agent"`
+	IPAddresses []string `json:"ipAddresses"`
+	Storage     []string `json:"storage"`
 }
 
 type GuestsMetrics struct {
@@ -29,9 +40,17 @@ type GuestsMetrics struct {
 }
 
 type guestCollector struct {
-	mu        sync.Mutex
-	cached    GuestsMetrics
-	updatedAt time.Time
+	mu                sync.Mutex
+	cached            GuestsMetrics
+	updatedAt         time.Time
+	details           map[int]guestDetails
+	detailsUpdatedAt  time.Time
+	detailsRefreshing bool
+}
+
+type guestDetails struct {
+	OnBoot, Agent        bool
+	IPAddresses, Storage []string
 }
 
 func (c *guestCollector) collect() GuestsMetrics {
@@ -41,10 +60,143 @@ func (c *guestCollector) collect() GuestsMetrics {
 		return c.cached
 	}
 	c.cached = readGuests()
+	if time.Since(c.detailsUpdatedAt) > time.Minute && !c.detailsRefreshing {
+		c.detailsRefreshing = true
+		guests := append([]GuestMetrics(nil), c.cached.Guests...)
+		go func() {
+			details := readGuestDetails(guests)
+			c.mu.Lock()
+			c.details = details
+			c.detailsUpdatedAt = time.Now()
+			c.detailsRefreshing = false
+			c.mu.Unlock()
+		}()
+	}
+	for i := range c.cached.Guests {
+		if detail, ok := c.details[c.cached.Guests[i].VMID]; ok {
+			c.cached.Guests[i].OnBoot = detail.OnBoot
+			c.cached.Guests[i].Agent = detail.Agent
+			c.cached.Guests[i].IPAddresses = detail.IPAddresses
+			c.cached.Guests[i].Storage = detail.Storage
+		}
+	}
 	c.updatedAt = time.Now()
 	return c.cached
 }
 
+func readGuestDetails(guests []GuestMetrics) map[int]guestDetails {
+	result := map[int]guestDetails{}
+	for _, guest := range guests {
+		if guest.Template != 0 {
+			continue
+		}
+		kind := guest.Type
+		if kind != "qemu" {
+			kind = "lxc"
+		}
+		data := commandOutput(5*time.Second, "pvesh", "get", "/nodes/"+guest.Node+"/"+kind+"/"+strconv.Itoa(guest.VMID)+"/config", "--output-format", "json")
+		var config map[string]any
+		if json.Unmarshal([]byte(data), &config) != nil {
+			continue
+		}
+		detail := guestDetails{}
+		detail.OnBoot = numberValue(config["onboot"]) == 1
+		detail.Agent = agentEnabled(config["agent"])
+		storageSet := map[string]bool{}
+		ipSet := map[string]bool{}
+		for key, value := range config {
+			s := fmt.Sprint(value)
+			if key == "rootfs" || strings.HasPrefix(key, "scsi") || strings.HasPrefix(key, "sata") || strings.HasPrefix(key, "virtio") {
+				if name := storageName(s); name != "" {
+					storageSet[name] = true
+				}
+			}
+			if strings.HasPrefix(key, "net") || strings.HasPrefix(key, "ipconfig") {
+				for _, ip := range ipsFromConfig(s) {
+					ipSet[ip] = true
+				}
+			}
+		}
+		if guest.Status == "running" && kind == "qemu" {
+			agent := commandOutput(5*time.Second, "pvesh", "get", "/nodes/"+guest.Node+"/qemu/"+strconv.Itoa(guest.VMID)+"/agent/network-get-interfaces", "--output-format", "json")
+			for _, ip := range ipsFromAgent(agent) {
+				ipSet[ip] = true
+			}
+			if agent != "" {
+				detail.Agent = true
+			}
+		}
+		for value := range storageSet {
+			detail.Storage = append(detail.Storage, value)
+		}
+		for value := range ipSet {
+			detail.IPAddresses = append(detail.IPAddresses, value)
+		}
+		sort.Strings(detail.Storage)
+		sort.Strings(detail.IPAddresses)
+		result[guest.VMID] = detail
+	}
+	return result
+}
+
+func numberValue(value any) float64 {
+	switch v := value.(type) {
+	case float64:
+		return v
+	case string:
+		n, _ := strconv.ParseFloat(v, 64)
+		return n
+	}
+	return 0
+}
+func agentEnabled(value any) bool {
+	s := strings.ToLower(fmt.Sprint(value))
+	return s == "1" || strings.Contains(s, "enabled=1")
+}
+func storageName(value string) string {
+	first := strings.SplitN(value, ",", 2)[0]
+	parts := strings.SplitN(first, ":", 2)
+	if len(parts) == 2 {
+		return parts[0]
+	}
+	return ""
+}
+func ipsFromConfig(value string) []string {
+	var result []string
+	for _, part := range strings.Split(value, ",") {
+		part = strings.TrimSpace(part)
+		if strings.HasPrefix(part, "ip=") || strings.HasPrefix(part, "ip6=") {
+			ip := strings.SplitN(part, "=", 2)[1]
+			if ip != "dhcp" && ip != "auto" {
+				result = append(result, strings.SplitN(ip, "/", 2)[0])
+			}
+		}
+	}
+	return result
+}
+func ipsFromAgent(data string) []string {
+	var payload struct {
+		Result []struct {
+			IPAddresses []struct {
+				Address string `json:"ip-address"`
+				Type    string `json:"ip-address-type"`
+			} `json:"ip-addresses"`
+		} `json:"result"`
+	}
+	if json.Unmarshal([]byte(data), &payload) != nil {
+		return nil
+	}
+	var result []string
+	for _, iface := range payload.Result {
+		for _, ip := range iface.IPAddresses {
+			if ip.Address != "127.0.0.1" && ip.Address != "::1" && !strings.HasPrefix(ip.Address, "fe80:") {
+				result = append(result, ip.Address)
+			}
+		}
+	}
+	return result
+}
+
 func readGuests() GuestsMetrics {
 	if _, err := exec.LookPath("pvesh"); err != nil {
 		return GuestsMetrics{}

guests_test.go

@@ -15,3 +15,16 @@ func TestGuestJSON(t *testing.T) {
 		t.Fatalf("неожиданные данные гостя: %+v", guest)
 	}
 }
+
+func TestGuestConfigHelpers(t *testing.T) {
+	if got := storageName("local-zfs:vm-101-disk-0,size=32G"); got != "local-zfs" {
+		t.Fatalf("storage: %s", got)
+	}
+	ips := ipsFromConfig("name=eth0,bridge=vmbr0,ip=192.168.0.20/24,ip6=auto")
+	if len(ips) != 1 || ips[0] != "192.168.0.20" {
+		t.Fatalf("ips: %+v", ips)
+	}
+	if !agentEnabled("enabled=1,fstrim_cloned_disks=1") {
+		t.Fatal("agent should be enabled")
+	}
+}

history.go

@@ -1,6 +1,7 @@
 package main
 
 import (
+	"fmt"
 	"log"
 	"time"
 )
@@ -15,6 +16,7 @@ func runHistoryCollector(collect func() (dashboardMetrics, error), store *Store)
 		if err := store.AddHistory(historyPointFromMetrics(metrics)); err != nil {
 			log.Printf("Не удалось сохранить исторические метрики: %v", err)
 		}
+		store.AddEntityMetrics(metrics)
 	}
 	record()
 	ticker := time.NewTicker(time.Minute)
@@ -24,6 +26,71 @@ func runHistoryCollector(collect func() (dashboardMetrics, error), store *Store)
 	}
 }
 
+func (s *Store) AddEntityMetrics(metrics dashboardMetrics) {
+	ts := time.Now().Unix()
+	for _, p := range metrics.ZFS.Pools {
+		v := p.CapacityPercent
+		s.AddEntityHistory("zfs", p.Name, ts, &v, nil)
+	}
+	for _, g := range metrics.Guests.Guests {
+		if g.Template != 0 {
+			continue
+		}
+		cpu := g.CPU * 100
+		var mem float64
+		if g.MaxMemory > 0 {
+			mem = float64(g.MemoryBytes) / float64(g.MaxMemory) * 100
+		}
+		s.AddEntityHistory("guest", fmt.Sprintf("%d · %s", g.VMID, g.Name), ts, &cpu, &mem)
+	}
+	for _, d := range metrics.Disks {
+		iops := d.ReadIOPS + d.WriteIOPS
+		util := d.Utilization
+		s.AddEntityHistory("disk-io", d.Name+" · "+d.Model, ts, &iops, &util)
+		latency := d.LatencyMs
+		s.AddEntityHistory("disk-latency", d.Name+" · "+d.Model, ts, &latency, nil)
+		if d.Temperature == nil {
+			continue
+		}
+		var wear *float64
+		if d.WearUsedPercent != nil {
+			wear = d.WearUsedPercent
+		}
+		s.AddEntityHistory("disk", d.Name+" · "+d.Model, ts, d.Temperature, wear)
+	}
+	for _, task := range metrics.Activity.Tasks {
+		if task.Type == "vzdump" && task.Duration > 0 && task.EndTime >= time.Now().Add(-15*time.Minute).Unix() {
+			duration := float64(task.Duration)
+			s.AddEntityHistory("backup-duration", "VMID "+task.ID, ts, &duration, nil)
+		}
+	}
+	for _, agent := range metrics.Agents {
+		if !agent.Online {
+			continue
+		}
+		name := agent.ID + " · " + agent.Name
+		cpu := agent.Report.CPUPercent
+		memory := float64(0)
+		if agent.Report.MemoryTotal > 0 {
+			memory = float64(agent.Report.MemoryUsed) / float64(agent.Report.MemoryTotal) * 100
+		}
+		swap := float64(0)
+		if agent.Report.SwapTotal > 0 {
+			swap = float64(agent.Report.SwapUsed) / float64(agent.Report.SwapTotal) * 100
+		}
+		s.AddEntityHistory("agent-resource", name, ts, &cpu, &memory)
+		s.AddEntityHistory("agent-swap", name, ts, &swap, nil)
+		pressure := agent.Report.Pressure.IO.Avg10
+		s.AddEntityHistory("agent-pressure", name, ts, &pressure, nil)
+		for _, filesystem := range agent.Report.Filesystems {
+			used := filesystem.UsedPercent
+			inodes := filesystem.InodesPercent
+			s.AddEntityHistory("agent-filesystem", name+" · "+filesystem.Mountpoint, ts, &used, &inodes)
+		}
+	}
+	s.PruneEntityHistory()
+}
+
 func historyPointFromMetrics(metrics dashboardMetrics) HistoryPoint {
 	point := HistoryPoint{
 		Timestamp: time.Now().Unix(), CPUUsage: metrics.CPU.UsagePercent,
@@ -32,12 +99,10 @@ func historyPointFromMetrics(metrics dashboardMetrics) HistoryPoint {
 		NetworkTransmit: metrics.Network.TransmitPerSec, UPSCharge: metrics.UPS.ChargePercent,
 		UPSLoad: metrics.UPS.LoadPercent,
 	}
-	if len(metrics.CPU.Temperatures) > 0 {
-		for _, temperature := range metrics.CPU.Temperatures {
-			point.CPUTemperature += temperature.Celsius
-		}
-		point.CPUTemperature /= float64(len(metrics.CPU.Temperatures))
+	if temperature := cpuTemperature(metrics.CPU.Temperatures); temperature != nil {
+		point.CPUTemperature = *temperature
 	}
+
 	var zfsSize, zfsUsed uint64
 	for _, pool := range metrics.ZFS.Pools {
 		zfsSize += pool.SizeBytes

history_test.go

@@ -8,7 +8,7 @@ func TestHistoryPointFromMetrics(t *testing.T) {
 		Guests: GuestsMetrics{Guests: []GuestMetrics{{Status: "running", CPU: .5, MaxCPU: 4, MemoryBytes: 50, MaxMemory: 100}}},
 	}
 	point := historyPointFromMetrics(metrics)
-	if point.CPUTemperature != 50 || point.GuestsCPU != 50 || point.GuestsMemory != 50 {
+	if point.CPUTemperature != 60 || point.GuestsCPU != 50 || point.GuestsMemory != 50 {
 		t.Fatalf("неожиданная историческая точка: %+v", point)
 	}
 }

io_attribution.go

@@ -0,0 +1,178 @@
+package main
+
+import (
+	"os"
+	"path/filepath"
+	"regexp"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+)
+
+type IOConsumer struct {
+	Kind             string  `json:"kind"`
+	Name             string  `json:"name"`
+	VMID             int     `json:"vmid,omitempty"`
+	ReadBytesPerSec  float64 `json:"readBytesPerSec"`
+	WriteBytesPerSec float64 `json:"writeBytesPerSec"`
+	SharePercent     float64 `json:"sharePercent"`
+}
+type processIOSnapshot struct {
+	read, write           uint64
+	name, cmdline, cgroup string
+}
+type ioAttributionCollector struct {
+	mu        sync.Mutex
+	previous  map[int]processIOSnapshot
+	updatedAt time.Time
+	cached    []IOConsumer
+}
+
+var qemuIDPattern = regexp.MustCompile(`(?:^|\s)-id\s+(\d+)`)
+var lxcIDPattern = regexp.MustCompile(`(?:lxc(?:\.payload)?[./-]|pve-container@)(\d+)`)
+
+func (c *ioAttributionCollector) collect(guests GuestsMetrics) []IOConsumer {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	now := time.Now()
+	current := readProcessIO()
+	elapsed := now.Sub(c.updatedAt).Seconds()
+	if c.previous == nil || elapsed <= 0 {
+		c.previous = current
+		c.updatedAt = now
+		return nil
+	}
+	names := map[int]string{}
+	for _, g := range guests.Guests {
+		names[g.VMID] = g.Name
+	}
+	grouped := map[string]*IOConsumer{}
+	var total float64
+	for pid, snapshot := range current {
+		old, ok := c.previous[pid]
+		if !ok || snapshot.read < old.read || snapshot.write < old.write {
+			continue
+		}
+		read := float64(snapshot.read-old.read) / elapsed
+		write := float64(snapshot.write-old.write) / elapsed
+		if read+write < 1024 {
+			continue
+		}
+		kind, name, vmid := classifyIOProcess(snapshot, names)
+		key := kind + ":" + strconv.Itoa(vmid) + ":" + name
+		consumer := grouped[key]
+		if consumer == nil {
+			consumer = &IOConsumer{Kind: kind, Name: name, VMID: vmid}
+			grouped[key] = consumer
+		}
+		consumer.ReadBytesPerSec += read
+		consumer.WriteBytesPerSec += write
+		total += read + write
+	}
+	var result []IOConsumer
+	for _, v := range grouped {
+		if total > 0 {
+			v.SharePercent = (v.ReadBytesPerSec + v.WriteBytesPerSec) / total * 100
+		}
+		result = append(result, *v)
+	}
+	sort.Slice(result, func(i, j int) bool {
+		return result[i].ReadBytesPerSec+result[i].WriteBytesPerSec > result[j].ReadBytesPerSec+result[j].WriteBytesPerSec
+	})
+	if len(result) > 8 {
+		result = result[:8]
+	}
+	c.previous = current
+	c.updatedAt = now
+	c.cached = result
+	return result
+}
+
+func classifyIOProcess(p processIOSnapshot, names map[int]string) (kind, name string, vmid int) {
+	if match := qemuIDPattern.FindStringSubmatch(p.cmdline); len(match) > 1 {
+		vmid, _ = strconv.Atoi(match[1])
+		return "VM", firstNonEmpty(names[vmid], "VM "+match[1]), vmid
+	}
+	if match := lxcIDPattern.FindStringSubmatch(p.cgroup); len(match) > 1 {
+		vmid, _ = strconv.Atoi(match[1])
+		return "LXC", firstNonEmpty(names[vmid], "LXC "+match[1]), vmid
+	}
+	return "Процесс", firstNonEmpty(p.name, "PID"), 0
+}
+
+func readProcessIO() map[int]processIOSnapshot {
+	result := map[int]processIOSnapshot{}
+	entries, _ := filepath.Glob("/proc/[0-9]*/io")
+	for _, path := range entries {
+		pid, _ := strconv.Atoi(filepath.Base(filepath.Dir(path)))
+		data, err := os.ReadFile(path)
+		if err != nil {
+			continue
+		}
+		var item processIOSnapshot
+		for _, line := range strings.Split(string(data), "\n") {
+			key, value, ok := strings.Cut(line, ":")
+			if !ok {
+				continue
+			}
+			n, _ := strconv.ParseUint(strings.TrimSpace(value), 10, 64)
+			if key == "read_bytes" {
+				item.read = n
+			} else if key == "write_bytes" {
+				item.write = n
+			}
+		}
+		comm, _ := os.ReadFile(filepath.Join(filepath.Dir(path), "comm"))
+		cmd, _ := os.ReadFile(filepath.Join(filepath.Dir(path), "cmdline"))
+		cg, _ := os.ReadFile(filepath.Join(filepath.Dir(path), "cgroup"))
+		item.name = strings.TrimSpace(string(comm))
+		item.cmdline = strings.ReplaceAll(string(cmd), "\x00", " ")
+		item.cgroup = string(cg)
+		result[pid] = item
+	}
+	return result
+}
+
+func attributeDiskIO(metrics *dashboardMetrics, consumers []IOConsumer) {
+	active := activeIOOperation(metrics.Activity.Tasks)
+	for i := range metrics.Disks {
+		disk := &metrics.Disks[i]
+		disk.TopConsumers = append([]IOConsumer(nil), consumers...)
+		if active != "" {
+			disk.IOCause = active
+			disk.IOCauseDetail = "Активная задача Proxmox совпадает по времени с дисковой нагрузкой"
+			disk.IOConfidence = 75
+		} else if len(consumers) > 0 {
+			top := consumers[0]
+			disk.IOCause = top.Kind + " · " + top.Name
+			disk.IOCauseDetail = "Крупнейший потребитель host I/O: " + strconv.FormatFloat(top.SharePercent, 'f', 1, 64) + "%"
+			disk.IOConfidence = 55
+		}
+	}
+}
+
+func activeIOOperation(tasks []ProxmoxTask) string {
+	for _, task := range tasks {
+		if task.EndTime != 0 && task.Status != "" {
+			continue
+		}
+		label := ""
+		switch task.Type {
+		case "vzdump":
+			label = "Backup"
+		case "qmigrate":
+			label = "Миграция"
+		case "qmsnapshot", "vzsnapshot":
+			label = "Snapshot"
+		}
+		if label != "" {
+			if task.ID != "" {
+				label += " VM/LXC " + task.ID
+			}
+			return label
+		}
+	}
+	return ""
+}

io_attribution_test.go

@@ -0,0 +1,33 @@
+package main
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestClassifyQEMUAndLXCIO(t *testing.T) {
+	kind, name, vmid := classifyIOProcess(processIOSnapshot{name: "qemu", cmdline: "qemu-system-x86_64 -id 105 -name guest=test"}, map[int]string{105: "Gitea"})
+	if kind != "VM" || name != "Gitea" || vmid != 105 {
+		t.Fatalf("unexpected qemu attribution: %s %s %d", kind, name, vmid)
+	}
+	kind, name, vmid = classifyIOProcess(processIOSnapshot{name: "worker", cgroup: "0::/lxc.payload.220/ns"}, map[int]string{220: "db"})
+	if kind != "LXC" || name != "db" || vmid != 220 {
+		t.Fatalf("unexpected lxc attribution: %s %s %d", kind, name, vmid)
+	}
+}
+
+func TestActiveIOOperation(t *testing.T) {
+	value := activeIOOperation([]ProxmoxTask{{Type: "vzdump", ID: "105", Status: "", EndTime: 0}})
+	if !strings.Contains(value, "Backup") || !strings.Contains(value, "105") {
+		t.Fatalf("unexpected operation: %s", value)
+	}
+}
+
+func TestCorrelateAlerts(t *testing.T) {
+	metrics := dashboardMetrics{Disks: []DiskMetrics{{Name: "sda", IOCause: "Backup VM/LXC 105", IOConfidence: 90, Utilization: 98, LatencyMs: 80}}, Services: ServicesMetrics{Services: []ServiceStatus{{MonitoredService: MonitoredService{Name: "Gitea"}, Endpoints: []ServiceEndpointStatus{{Up: false}}}}}}
+	alerts := []Alert{{ID: "disk-io-pressure-sda", Severity: "critical"}, {ID: "service-1-public", Severity: "critical"}, {ID: "other", Severity: "warning"}}
+	result := correlateAlerts(metrics, alerts)
+	if len(result) != 2 || result[0].Source != "Корреляция" || !strings.Contains(result[0].Message, "Gitea") {
+		t.Fatalf("unexpected correlation: %+v", result)
+	}
+}

main.go

@@ -3,6 +3,7 @@ package main
 import (
 	"context"
 	"errors"
+	"flag"
 	"fmt"
 	"log"
 	"net/http"
@@ -18,6 +19,25 @@ const defaultDatabasePath = "/var/lib/proxmox-dashboard/dashboard.db"
 var version = "dev"
 
 func main() {
+	showVersion := flag.Bool("version", false, "show version")
+	agentMode := flag.Bool("agent", false, "run as a read-only guest agent")
+	agentServer := flag.String("agent-server", "", "dashboard URL used by the guest agent")
+	agentEnrollToken := flag.String("agent-enroll-token", "", "one-time enrollment token")
+	agentName := flag.String("agent-name", "", "agent display name")
+	agentVMID := flag.Int("agent-vmid", 0, "Proxmox VM/LXC ID")
+	agentConfig := flag.String("agent-config", "/var/lib/proxmox-dashboard-agent/agent.json", "agent configuration path")
+	flag.Parse()
+	if *showVersion {
+		fmt.Println(version)
+		return
+	}
+	if *agentMode {
+		if err := runGuestAgent(agentOptions{Server: *agentServer, EnrollmentToken: *agentEnrollToken, Name: *agentName, VMID: *agentVMID, ConfigPath: *agentConfig}); err != nil {
+			log.Fatalf("Ошибка агента: %v", err)
+		}
+		return
+	}
+
 	address := os.Getenv("DASHBOARD_ADDR")
 	if address == "" {
 		address = defaultAddress
@@ -42,6 +62,7 @@ func main() {
 		Handler:           routes(collector, store),
 		ReadHeaderTimeout: 5 * time.Second,
 	}
+	runSystemdWatchdog()
 
 	go func() {
 		log.Printf("Proxmox Dashboard %s запущен на http://0.0.0.0%s", version, address)

maintenance.go

@@ -0,0 +1,264 @@
+package main
+
+import (
+	"bufio"
+	"compress/gzip"
+	"context"
+	"encoding/json"
+	"io"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"regexp"
+	"sort"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+)
+
+type MaintenanceMetrics struct {
+	Available             bool      `json:"available"`
+	ProxmoxVersion        string    `json:"proxmoxVersion"`
+	KernelVersion         string    `json:"kernelVersion"`
+	LatestInstalledKernel string    `json:"latestInstalledKernel"`
+	NewKernelAvailable    bool      `json:"newKernelAvailable"`
+	RebootRequired        bool      `json:"rebootRequired"`
+	AvailableUpdates      int       `json:"availableUpdates"`
+	LastSystemUpdate      time.Time `json:"lastSystemUpdate"`
+	NextBackup            string    `json:"nextBackup"`
+	NextScrub             string    `json:"nextScrub"`
+	CheckedAt             time.Time `json:"checkedAt"`
+}
+
+type maintenanceCollector struct {
+	mu         sync.Mutex
+	cached     MaintenanceMetrics
+	updatedAt  time.Time
+	refreshing bool
+}
+
+func (c *maintenanceCollector) collect() MaintenanceMetrics {
+	c.mu.Lock()
+	if time.Since(c.updatedAt) >= 15*time.Minute && !c.refreshing {
+		c.refreshing = true
+		go func() {
+			metrics := readMaintenance()
+			c.mu.Lock()
+			c.cached = metrics
+			c.updatedAt = time.Now()
+			c.refreshing = false
+			c.mu.Unlock()
+		}()
+	}
+	result := c.cached
+	c.mu.Unlock()
+	return result
+}
+
+func commandOutput(timeout time.Duration, name string, args ...string) string {
+	ctx, cancel := context.WithTimeout(context.Background(), timeout)
+	defer cancel()
+	command := exec.CommandContext(ctx, name, args...)
+	command.Env = append(os.Environ(), "LC_ALL=C")
+	output, err := command.Output()
+	if err != nil {
+		return ""
+	}
+	return strings.TrimSpace(string(output))
+}
+
+func readMaintenance() MaintenanceMetrics {
+	result := MaintenanceMetrics{Available: true, CheckedAt: time.Now()}
+	result.ProxmoxVersion = firstLine(commandOutput(5*time.Second, "pveversion"))
+	result.KernelVersion = commandOutput(3*time.Second, "uname", "-r")
+	result.RebootRequired = fileExists("/var/run/reboot-required")
+	apt := commandOutput(45*time.Second, "apt-get", "-s", "-o", "Debug::NoLocking=1", "upgrade")
+	for _, line := range strings.Split(apt, "\n") {
+		if strings.HasPrefix(line, "Inst ") {
+			result.AvailableUpdates++
+		}
+	}
+	result.LastSystemUpdate = lastAPTUpdate("/var/log/apt/history.log")
+	installed := commandOutput(8*time.Second, "dpkg-query", "-W", "-f=${Package} ${Version}\\n", "pve-kernel-*", "proxmox-kernel-*")
+	result.LatestInstalledKernel = newestKernel(installed)
+	result.NewKernelAvailable = result.LatestInstalledKernel != "" && !strings.Contains(result.KernelVersion, result.LatestInstalledKernel)
+	result.NextBackup = nextBackupSchedule()
+	result.NextScrub = nextScrubTimer(commandOutput(8*time.Second, "systemctl", "list-timers", "--all", "--no-legend", "--plain"))
+	if result.NextScrub == "" {
+		result.NextScrub = nextZFSScrubCron("/etc/cron.d/zfsutils-linux", time.Now())
+	}
+	return result
+}
+
+func firstLine(value string) string {
+	if i := strings.IndexByte(value, '\n'); i >= 0 {
+		return value[:i]
+	}
+	return value
+}
+func fileExists(path string) bool { _, err := os.Stat(path); return err == nil }
+
+func lastAPTUpdate(path string) time.Time {
+	file, err := os.Open(path)
+	if err == nil {
+		defer file.Close()
+		if result := scanAPTUpdate(file); !result.IsZero() {
+			return result
+		}
+	}
+	files, _ := filepath.Glob(path + ".*.gz")
+	sort.Slice(files, func(i, j int) bool {
+		a, _ := os.Stat(files[i])
+		b, _ := os.Stat(files[j])
+		return a.ModTime().After(b.ModTime())
+	})
+	for _, name := range files {
+		compressed, err := os.Open(name)
+		if err != nil {
+			continue
+		}
+		reader, err := gzip.NewReader(compressed)
+		if err != nil {
+			compressed.Close()
+			continue
+		}
+		result := scanAPTUpdate(reader)
+		reader.Close()
+		compressed.Close()
+		if !result.IsZero() {
+			return result
+		}
+	}
+	return time.Time{}
+}
+func scanAPTUpdate(reader io.Reader) time.Time {
+	scanner := bufio.NewScanner(reader)
+	var result time.Time
+	for scanner.Scan() {
+		line := strings.TrimSpace(scanner.Text())
+		if strings.HasPrefix(line, "End-Date:") {
+			value := strings.TrimSpace(strings.TrimPrefix(line, "End-Date:"))
+			if parsed, err := time.ParseInLocation("2006-01-02  15:04:05", value, time.Local); err == nil {
+				result = parsed
+			}
+		}
+	}
+	return result
+}
+
+var versionNumbers = regexp.MustCompile(`\d+`)
+
+func newestKernel(output string) string {
+	var versions []string
+	for _, line := range strings.Split(output, "\n") {
+		fields := strings.Fields(line)
+		if len(fields) > 1 && !strings.Contains(fields[0], "helper") {
+			versions = append(versions, fields[1])
+		}
+	}
+	sort.Slice(versions, func(i, j int) bool { return compareNumericVersion(versions[i], versions[j]) > 0 })
+	if len(versions) > 0 {
+		return versions[0]
+	}
+	return ""
+}
+func compareNumericVersion(a, b string) int {
+	aa, bb := versionNumbers.FindAllString(a, -1), versionNumbers.FindAllString(b, -1)
+	for i := 0; i < len(aa) || i < len(bb); i++ {
+		av, bv := "0", "0"
+		if i < len(aa) {
+			av = aa[i]
+		}
+		if i < len(bb) {
+			bv = bb[i]
+		}
+		if len(av) != len(bv) {
+			if len(av) > len(bv) {
+				return 1
+			}
+			return -1
+		}
+		if av != bv {
+			if av > bv {
+				return 1
+			}
+			return -1
+		}
+	}
+	return 0
+}
+
+func nextBackupSchedule() string {
+	data := commandOutput(8*time.Second, "pvesh", "get", "/cluster/backup", "--output-format", "json")
+	if data == "" {
+		return ""
+	}
+	var jobs []struct {
+		Enabled  *int   `json:"enabled"`
+		Schedule string `json:"schedule"`
+	}
+	if json.Unmarshal([]byte(data), &jobs) != nil {
+		return ""
+	}
+	var next []string
+	for _, job := range jobs {
+		if (job.Enabled != nil && *job.Enabled == 0) || job.Schedule == "" {
+			continue
+		}
+		calendar := commandOutput(5*time.Second, "systemd-analyze", "calendar", "--iterations=1", job.Schedule)
+		for _, line := range strings.Split(calendar, "\n") {
+			if strings.Contains(line, "Next elapse:") {
+				trimmed := strings.TrimSpace(line)
+				next = append(next, strings.TrimSpace(strings.TrimPrefix(trimmed, "Next elapse:")))
+			}
+		}
+	}
+	sort.Strings(next)
+	if len(next) > 0 {
+		return next[0]
+	}
+	return ""
+}
+func nextScrubTimer(output string) string {
+	for _, line := range strings.Split(output, "\n") {
+		if strings.Contains(strings.ToLower(line), "zfs") && strings.Contains(strings.ToLower(line), "scrub") {
+			fields := strings.Fields(line)
+			if len(fields) >= 4 {
+				return strings.Join(fields[:4], " ")
+			}
+		}
+	}
+	return ""
+}
+
+func nextZFSScrubCron(path string, now time.Time) string {
+	data, err := os.ReadFile(path)
+	if err != nil {
+		return ""
+	}
+	for _, line := range strings.Split(string(data), "\n") {
+		if !strings.Contains(line, "/usr/lib/zfs-linux/scrub") {
+			continue
+		}
+		fields := strings.Fields(line)
+		if len(fields) < 5 {
+			return ""
+		}
+		minute, _ := strconv.Atoi(fields[0])
+		hour, _ := strconv.Atoi(fields[1])
+		rangeParts := strings.SplitN(fields[2], "-", 2)
+		if len(rangeParts) != 2 {
+			return ""
+		}
+		from, _ := strconv.Atoi(rangeParts[0])
+		to, _ := strconv.Atoi(rangeParts[1])
+		for offset := 0; offset < 70; offset++ {
+			day := time.Date(now.Year(), now.Month(), now.Day()+offset, hour, minute, 0, 0, now.Location())
+			if day.After(now) && day.Day() >= from && day.Day() <= to && day.Weekday() == time.Sunday {
+				return day.Format("Mon 2006-01-02 15:04 MST")
+			}
+		}
+	}
+	return ""
+}

maintenance_test.go

@@ -0,0 +1,33 @@
+package main
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+	"time"
+)
+
+func TestNewestKernel(t *testing.T) {
+	output := "proxmox-kernel-6.8 6.8.12-5\nproxmox-kernel-6.14 6.14.8-2"
+	if got := newestKernel(output); got != "6.14.8-2" {
+		t.Fatalf("unexpected newest kernel: %s", got)
+	}
+}
+
+func TestNextZFSScrubCron(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "zfs")
+	if err := os.WriteFile(path, []byte("24 0 8-14 * * root /usr/lib/zfs-linux/scrub\n"), 0600); err != nil {
+		t.Fatal(err)
+	}
+	now := time.Date(2026, 8, 1, 12, 0, 0, 0, time.UTC)
+	if got := nextZFSScrubCron(path, now); got != "Sun 2026-08-09 00:24 UTC" {
+		t.Fatalf("unexpected cron scrub: %s", got)
+	}
+}
+
+func TestNextScrubTimer(t *testing.T) {
+	line := "Sun 2026-08-02 00:24:00 MSK 1 day left zfs-scrub-weekly@tank.timer"
+	if got := nextScrubTimer(line); got != "Sun 2026-08-02 00:24:00 MSK" {
+		t.Fatalf("unexpected scrub: %s", got)
+	}
+}

maintenance_window.go

@@ -0,0 +1,170 @@
+package main
+
+import (
+	"encoding/json"
+	"fmt"
+	"strconv"
+	"strings"
+	"time"
+)
+
+type MaintenanceWindow struct {
+	Enabled    bool    `json:"enabled"`
+	Active     bool    `json:"active"`
+	StartedAt  int64   `json:"startedAt"`
+	EndsAt     int64   `json:"endsAt"`
+	Reason     string  `json:"reason"`
+	AllAlerts  bool    `json:"allAlerts"`
+	ServiceIDs []int64 `json:"serviceIds"`
+	VMIDs      []int   `json:"vmids"`
+	Automatic  bool    `json:"automatic"`
+	Source     string  `json:"source,omitempty"`
+}
+
+func automaticBackupMaintenance(activity ProxmoxActivity, links []ServiceLink, now time.Time) MaintenanceWindow {
+	vmids := []int{}
+	backupRunning := false
+	for _, task := range activity.Tasks {
+		if task.Type != "vzdump" || task.EndTime > 0 || (task.Status != "" && !strings.EqualFold(task.Status, "running")) {
+			continue
+		}
+		backupRunning = true
+		if vmid := backupTaskVMID(task); vmid > 0 {
+			vmids = append(vmids, vmid)
+		}
+	}
+	vmids = uniqueInt(vmids)
+	if !backupRunning {
+		return MaintenanceWindow{}
+	}
+	selected := map[int]bool{}
+	for _, vmid := range vmids {
+		selected[vmid] = true
+	}
+	serviceIDs := []int64{}
+	for _, link := range links {
+		if selected[link.VMID] {
+			serviceIDs = append(serviceIDs, link.ServiceID)
+		}
+	}
+	return MaintenanceWindow{Enabled: true, Active: true, StartedAt: now.Unix(), EndsAt: now.Add(2 * time.Minute).Unix(), Reason: "Автоматически: выполняется backup VM/LXC", VMIDs: vmids, ServiceIDs: uniqueInt64(serviceIDs), AllAlerts: len(vmids) == 0, Automatic: true, Source: "backup"}
+}
+
+func backupTaskVMID(task ProxmoxTask) int {
+	if vmid, err := strconv.Atoi(strings.TrimSpace(task.ID)); err == nil && vmid > 0 {
+		return vmid
+	}
+	parts := strings.Split(task.UPID, ":")
+	for i, part := range parts {
+		if part == "vzdump" && i+1 < len(parts) {
+			vmid, _ := strconv.Atoi(parts[i+1])
+			return vmid
+		}
+	}
+	return 0
+}
+
+func (s *Store) MaintenanceWindow() MaintenanceWindow {
+	var value string
+	var window MaintenanceWindow
+	if s.db.QueryRow(`SELECT value FROM settings WHERE key='maintenance_window'`).Scan(&value) == nil {
+		_ = json.Unmarshal([]byte(value), &window)
+	}
+	now := time.Now().Unix()
+	window.Active = window.Enabled && window.StartedAt <= now && window.EndsAt > now
+	if window.Enabled && window.EndsAt <= now {
+		window.Enabled = false
+		data, _ := json.Marshal(window)
+		_, _ = s.db.Exec(`INSERT INTO settings(key,value) VALUES('maintenance_window',?) ON CONFLICT(key) DO UPDATE SET value=excluded.value`, string(data))
+	}
+	return window
+}
+
+func (s *Store) SaveMaintenanceWindow(window MaintenanceWindow) (MaintenanceWindow, error) {
+	now := time.Now()
+	window.Reason = strings.TrimSpace(window.Reason)
+	if window.Reason == "" {
+		return window, fmt.Errorf("укажите причину плановых работ")
+	}
+	if window.EndsAt <= now.Unix() || window.EndsAt > now.Add(30*24*time.Hour).Unix() {
+		return window, fmt.Errorf("окончание должно быть в будущем, максимум через 30 дней")
+	}
+	if !window.AllAlerts && len(window.ServiceIDs) == 0 && len(window.VMIDs) == 0 {
+		return window, fmt.Errorf("выберите сервис, VM/LXC или все алерты")
+	}
+	window.Enabled = true
+	window.Active = true
+	window.StartedAt = now.Unix()
+	window.ServiceIDs = uniqueInt64(window.ServiceIDs)
+	window.VMIDs = uniqueInt(window.VMIDs)
+	data, _ := json.Marshal(window)
+	_, err := s.db.Exec(`INSERT INTO settings(key,value) VALUES('maintenance_window',?) ON CONFLICT(key) DO UPDATE SET value=excluded.value`, string(data))
+	return window, err
+}
+
+func (s *Store) StopMaintenanceWindow() (MaintenanceWindow, error) {
+	window := s.MaintenanceWindow()
+	window.Enabled = false
+	window.Active = false
+	window.EndsAt = time.Now().Unix()
+	data, _ := json.Marshal(window)
+	_, err := s.db.Exec(`INSERT INTO settings(key,value) VALUES('maintenance_window',?) ON CONFLICT(key) DO UPDATE SET value=excluded.value`, string(data))
+	return window, err
+}
+
+func splitMaintenanceAlerts(alerts []Alert, window MaintenanceWindow) (active, planned []Alert) {
+	if !window.Active {
+		return alerts, nil
+	}
+	for _, alert := range alerts {
+		if maintenanceMatchesAlert(window, alert) {
+			planned = append(planned, alert)
+		} else {
+			active = append(active, alert)
+		}
+	}
+	return active, planned
+}
+
+func maintenanceMatchesAlert(window MaintenanceWindow, alert Alert) bool {
+	if window.AllAlerts {
+		return true
+	}
+	for _, id := range window.ServiceIDs {
+		value := strconv.FormatInt(id, 10)
+		if strings.HasPrefix(alert.ID, "service-"+value+"-") || strings.HasPrefix(alert.ID, "service-tls-"+value+"-") {
+			return true
+		}
+	}
+	for _, vmid := range window.VMIDs {
+		value := strconv.Itoa(vmid)
+		if strings.HasSuffix(alert.ID, "-"+value) || strings.Contains(alert.Message, "VMID "+value) {
+			return true
+		}
+	}
+	return false
+}
+
+func uniqueInt64(values []int64) []int64 {
+	seen := map[int64]bool{}
+	result := make([]int64, 0, len(values))
+	for _, value := range values {
+		if value > 0 && !seen[value] {
+			seen[value] = true
+			result = append(result, value)
+		}
+	}
+	return result
+}
+
+func uniqueInt(values []int) []int {
+	seen := map[int]bool{}
+	result := make([]int, 0, len(values))
+	for _, value := range values {
+		if value > 0 && !seen[value] {
+			seen[value] = true
+			result = append(result, value)
+		}
+	}
+	return result
+}

maintenance_window_test.go

@@ -0,0 +1,75 @@
+package main
+
+import (
+	"testing"
+	"time"
+)
+
+func TestMaintenanceFiltersSelectedTargets(t *testing.T) {
+	window := MaintenanceWindow{Active: true, ServiceIDs: []int64{7}, VMIDs: []int{105}}
+	alerts := []Alert{
+		{ID: "service-7-local"},
+		{ID: "service-tls-8-public"},
+		{ID: "backup-old-105"},
+		{ID: "task-failed-x", Message: "VMID 105, status error"},
+		{ID: "cpu-usage"},
+	}
+	active, planned := splitMaintenanceAlerts(alerts, window)
+	if len(active) != 2 || len(planned) != 3 {
+		t.Fatalf("неверное разделение: active=%+v planned=%+v", active, planned)
+	}
+}
+
+func TestMaintenanceExpiresAutomatically(t *testing.T) {
+	store, err := openStore(":memory:")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer store.Close()
+	window, err := store.SaveMaintenanceWindow(MaintenanceWindow{EndsAt: time.Now().Add(time.Hour).Unix(), Reason: "Обновление", AllAlerts: true})
+	if err != nil || !window.Active {
+		t.Fatalf("режим не включён: %+v %v", window, err)
+	}
+	if _, err = store.db.Exec(`UPDATE settings SET value=? WHERE key='maintenance_window'`, `{"enabled":true,"startedAt":1,"endsAt":2,"reason":"test","allAlerts":true}`); err != nil {
+		t.Fatal(err)
+	}
+	if expired := store.MaintenanceWindow(); expired.Active || expired.Enabled {
+		t.Fatalf("просроченный режим остался активным: %+v", expired)
+	}
+}
+
+func TestRunningBackupCreatesScopedAutomaticMaintenance(t *testing.T) {
+	now := time.Now()
+	window := automaticBackupMaintenance(ProxmoxActivity{Tasks: []ProxmoxTask{{Type: "vzdump", ID: "105"}, {Type: "vzdump", ID: "106", EndTime: now.Unix()}}}, []ServiceLink{{ServiceID: 7, VMID: 105}, {ServiceID: 8, VMID: 999}}, now)
+	if !window.Active || !window.Automatic || window.Source != "backup" || len(window.VMIDs) != 1 || window.VMIDs[0] != 105 || len(window.ServiceIDs) != 1 || window.ServiceIDs[0] != 7 {
+		t.Fatalf("неверное автоматическое окно: %+v", window)
+	}
+}
+
+func TestRunningBackupWithoutVMIDSuppressesAllAlerts(t *testing.T) {
+	window := automaticBackupMaintenance(ProxmoxActivity{Tasks: []ProxmoxTask{{Type: "vzdump", UPID: "unknown"}}}, nil, time.Now())
+	if !window.Active || !window.AllAlerts {
+		t.Fatalf("backup с неизвестной целью должен безопасно подавить алерты: %+v", window)
+	}
+}
+
+func TestPlannedAlertOnlyAppearsInHistory(t *testing.T) {
+	store, err := openStore(":memory:")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer store.Close()
+	alert := Alert{ID: "service-7-local", Severity: "critical", Source: "Сервисы", Title: "Недоступен"}
+	visible, err := store.SyncAlertsWithMaintenance(nil, []Alert{alert}, "Обновление")
+	if err != nil || len(visible) != 0 {
+		t.Fatalf("плановый алерт стал активным: %+v %v", visible, err)
+	}
+	history, err := store.AlertHistory(10)
+	if err != nil || len(history) != 1 || !history[0].Planned || history[0].MaintenanceReason != "Обновление" {
+		t.Fatalf("плановое событие не сохранено: %+v %v", history, err)
+	}
+	pending, err := store.PendingEmailAlerts()
+	if err != nil || len(pending) != 0 {
+		t.Fatalf("плановое событие попало в почту: %+v %v", pending, err)
+	}
+}

network.go

@@ -2,7 +2,14 @@ package main
 
 import (
 	"bufio"
+	"context"
+	"fmt"
+	"io"
+	"net"
+	"net/http"
 	"os"
+	"os/exec"
+	"path/filepath"
 	"strconv"
 	"strings"
 	"sync"
@@ -10,11 +17,57 @@ import (
 )
 
 type NetworkMetrics struct {
-	Interface      string  `json:"interface"`
-	ReceiveBytes   uint64  `json:"receiveBytes"`
-	TransmitBytes  uint64  `json:"transmitBytes"`
-	ReceivePerSec  float64 `json:"receivePerSec"`
-	TransmitPerSec float64 `json:"transmitPerSec"`
+	Interface      string             `json:"interface"`
+	ReceiveBytes   uint64             `json:"receiveBytes"`
+	TransmitBytes  uint64             `json:"transmitBytes"`
+	ReceivePerSec  float64            `json:"receivePerSec"`
+	TransmitPerSec float64            `json:"transmitPerSec"`
+	Interfaces     []NetworkInterface `json:"interfaces"`
+	Health         NetworkHealth      `json:"health"`
+}
+
+type NetworkHealth struct {
+	Gateway               string    `json:"gateway"`
+	GatewayUp             bool      `json:"gatewayUp"`
+	DNSUp                 bool      `json:"dnsUp"`
+	InternetUp            bool      `json:"internetUp"`
+	PacketLoss            float64   `json:"packetLoss"`
+	LatencyMs             float64   `json:"latencyMs"`
+	ExternalIP            string    `json:"externalIp"`
+	ExternalIPChanged     bool      `json:"externalIpChanged"`
+	DisappearedInterfaces []string  `json:"disappearedInterfaces"`
+	CheckedAt             time.Time `json:"checkedAt"`
+	Summary               string    `json:"summary"`
+	LikelyCause           string    `json:"likelyCause"`
+	RecommendedAction     string    `json:"recommendedAction"`
+}
+
+type NetworkInterface struct {
+	Name             string            `json:"name"`
+	ReceiveBytes     uint64            `json:"receiveBytes"`
+	TransmitBytes    uint64            `json:"transmitBytes"`
+	ReceiveErrors    uint64            `json:"receiveErrors"`
+	TransmitErrors   uint64            `json:"transmitErrors"`
+	ReceiveDrops     uint64            `json:"receiveDrops"`
+	TransmitDrops    uint64            `json:"transmitDrops"`
+	NewErrors        uint64            `json:"newErrors"`
+	NewReceiveDrops  uint64            `json:"newReceiveDrops"`
+	NewTransmitDrops uint64            `json:"newTransmitDrops"`
+	FaultCounters    map[string]uint64 `json:"faultCounters,omitempty"`
+	NewFaults        uint64            `json:"newFaults"`
+	LinkDown         bool              `json:"linkDown"`
+	NewDrops         uint64            `json:"newDrops"`
+	ReceivePerSec    float64           `json:"receivePerSec"`
+	TransmitPerSec   float64           `json:"transmitPerSec"`
+	VMID             int               `json:"vmid,omitempty"`
+	GuestName        string            `json:"guestName,omitempty"`
+	GuestType        string            `json:"guestType,omitempty"`
+	Kind             string            `json:"kind"`
+	State            string            `json:"state"`
+	Carrier          bool              `json:"carrier"`
+	SpeedMbps        int               `json:"speedMbps,omitempty"`
+	Duplex           string            `json:"duplex,omitempty"`
+	Master           string            `json:"master,omitempty"`
 }
 
 type networkSnapshot struct {
@@ -24,24 +77,39 @@ type networkSnapshot struct {
 }
 
 type networkCollector struct {
-	mu       sync.Mutex
-	path     string
-	previous networkSnapshot
+	mu                 sync.Mutex
+	path               string
+	previous           networkSnapshot
+	previousInterfaces map[string]NetworkInterface
+	health             NetworkHealth
+	healthUpdatedAt    time.Time
+	externalIP         string
+	knownInterfaces    map[string]bool
 }
 
 func newNetworkCollector(path string) (*networkCollector, error) {
-	snapshot, _, err := readNetworkSnapshot(path)
+	snapshot, _, details, err := readNetworkSnapshot(path)
 	if err != nil {
 		return nil, err
 	}
-	return &networkCollector{path: path, previous: snapshot}, nil
+	previous := map[string]NetworkInterface{}
+	for _, v := range details {
+		decorateInterface(&v)
+		collectInterfaceFaults(&v)
+		previous[v.Name] = v
+	}
+	known := map[string]bool{}
+	for _, v := range details {
+		known[v.Name] = true
+	}
+	return &networkCollector{path: path, previous: snapshot, previousInterfaces: previous, knownInterfaces: known}, nil
 }
 
 func (c *networkCollector) collect() (NetworkMetrics, error) {
 	c.mu.Lock()
 	defer c.mu.Unlock()
 
-	current, interfaces, err := readNetworkSnapshot(c.path)
+	current, interfaceNames, details, err := readNetworkSnapshot(c.path)
 	if err != nil {
 		return NetworkMetrics{}, err
 	}
@@ -51,22 +119,157 @@ func (c *networkCollector) collect() (NetworkMetrics, error) {
 		receivedRate = float64(current.received-c.previous.received) / elapsed
 		sentRate = float64(current.sent-c.previous.sent) / elapsed
 	}
+	for i := range details {
+		v := &details[i]
+		decorateInterface(v)
+		collectInterfaceFaults(v)
+		if old, ok := c.previousInterfaces[v.Name]; ok {
+			updateInterfaceDeltas(v, old, elapsed)
+		}
+		v.LinkDown = interfaceLinkDown(*v)
+	}
+	c.previousInterfaces = map[string]NetworkInterface{}
+	for _, v := range details {
+		c.previousInterfaces[v.Name] = v
+	}
 	c.previous = current
+	if time.Since(c.healthUpdatedAt) > 30*time.Second {
+		c.health = checkNetworkHealth(c.externalIP)
+		if c.health.ExternalIP != "" {
+			c.externalIP = c.health.ExternalIP
+		}
+		c.healthUpdatedAt = time.Now()
+	}
+	currentNames := map[string]bool{}
+	for _, v := range details {
+		currentNames[v.Name] = true
+	}
+	c.health.DisappearedInterfaces = nil
+	for name := range c.knownInterfaces {
+		if !currentNames[name] {
+			c.health.DisappearedInterfaces = append(c.health.DisappearedInterfaces, name)
+		}
+	}
+	c.knownInterfaces = currentNames
+	explainNetworkHealth(&c.health, details)
 	return NetworkMetrics{
-		Interface: strings.Join(interfaces, ", "), ReceiveBytes: current.received,
-		TransmitBytes: current.sent, ReceivePerSec: receivedRate, TransmitPerSec: sentRate,
+		Interface: strings.Join(interfaceNames, ", "), Interfaces: details, ReceiveBytes: current.received,
+		TransmitBytes: current.sent, ReceivePerSec: receivedRate, TransmitPerSec: sentRate, Health: c.health,
 	}, nil
 }
 
-func readNetworkSnapshot(path string) (networkSnapshot, []string, error) {
+func explainNetworkHealth(health *NetworkHealth, interfaces []NetworkInterface) {
+	health.Summary, health.LikelyCause, health.RecommendedAction = "Сеть работает штатно", "Проблем не обнаружено", "Действий не требуется"
+	switch {
+	case health.Gateway == "":
+		health.Summary, health.LikelyCause, health.RecommendedAction = "Не найден маршрут в локальную сеть", "На хосте отсутствует default gateway", "Проверьте адрес шлюза и конфигурацию bridge/VLAN"
+	case !health.GatewayUp:
+		health.Summary, health.LikelyCause, health.RecommendedAction = "Хост не видит локальный шлюз", "Проблема между Proxmox и роутером: линк, bridge, bond или VLAN", "Проверьте кабель, carrier и настройки vmbr/bond/VLAN"
+	case health.PacketLoss > 0:
+		health.Summary = fmt.Sprintf("Нестабильная локальная сеть: потеряно %.1f%% пакетов", health.PacketLoss)
+		health.LikelyCause, health.RecommendedAction = "Перегрузка или ошибки линка между Proxmox и шлюзом", "Проверьте аппаратные ошибки, кабель, порт коммутатора и duplex"
+	case !health.DNSUp && health.InternetUp:
+		health.Summary, health.LikelyCause, health.RecommendedAction = "Интернет доступен, но DNS не отвечает", "Проблема DNS-сервера или /etc/resolv.conf", "Проверьте настроенные DNS и попробуйте разрешить домен вручную"
+	case health.DNSUp && !health.InternetUp:
+		health.Summary, health.LikelyCause, health.RecommendedAction = "Локальная сеть и DNS работают, но внешняя проверка недоступна", "Проблема выше роутера, блокировка HTTPS или недоступность провайдера", "Проверьте маршрут наружу, firewall и доступ к внешним адресам"
+	}
+	for _, v := range interfaces {
+		if v.LinkDown {
+			health.Summary, health.LikelyCause, health.RecommendedAction = v.Name+": линк не поднят", "Нет carrier либо интерфейс административно выключен", "Проверьте кабель, порт коммутатора и состояние интерфейса"
+			return
+		}
+		if v.NewErrors+v.NewFaults > 0 {
+			health.Summary = fmt.Sprintf("%s: за последний интервал появилось %d изменений счётчиков сетевых ошибок", v.Name, v.NewErrors+v.NewFaults)
+			health.LikelyCause, health.RecommendedAction = "Счётчики интерфейса сейчас растут из-за перегрузки, кабеля, MTU или duplex", "Проверьте скорость и duplex, кабель, порт коммутатора и MTU на обоих концах"
+			return
+		}
+	}
+}
+
+func decorateInterface(v *NetworkInterface) {
+	base := filepath.Join("/sys/class/net", v.Name)
+	v.State = readTrimmed(filepath.Join(base, "operstate"))
+	v.Carrier = readTrimmed(filepath.Join(base, "carrier")) == "1"
+	v.SpeedMbps, _ = strconv.Atoi(readTrimmed(filepath.Join(base, "speed")))
+	v.Duplex = readTrimmed(filepath.Join(base, "duplex"))
+	if target, err := filepath.EvalSymlinks(filepath.Join(base, "master")); err == nil {
+		v.Master = filepath.Base(target)
+	}
+	switch {
+	case strings.HasPrefix(v.Name, "fwbr"):
+		v.Kind = "Firewall bridge"
+	case strings.HasPrefix(v.Name, "fwpr") || strings.HasPrefix(v.Name, "fwln"):
+		v.Kind = "Firewall link"
+	case strings.HasPrefix(v.Name, "vmbr"):
+		v.Kind = "Bridge"
+	case strings.HasPrefix(v.Name, "bond"):
+		v.Kind = "Bond"
+	case strings.Contains(v.Name, "."):
+		v.Kind = "VLAN"
+	case strings.HasPrefix(v.Name, "tap") || strings.HasPrefix(v.Name, "veth"):
+		v.Kind = "Guest"
+	default:
+		v.Kind = "Physical"
+	}
+}
+
+func checkNetworkHealth(previousIP string) NetworkHealth {
+	result := NetworkHealth{CheckedAt: time.Now()}
+	ctx, cancel := context.WithTimeout(context.Background(), 6*time.Second)
+	defer cancel()
+	if out, err := exec.CommandContext(ctx, "ip", "route", "show", "default").Output(); err == nil {
+		f := strings.Fields(string(out))
+		for i := range f {
+			if f[i] == "via" && i+1 < len(f) {
+				result.Gateway = f[i+1]
+				break
+			}
+		}
+	}
+	if result.Gateway != "" {
+		out, _ := exec.CommandContext(ctx, "ping", "-c", "3", "-W", "1", result.Gateway).CombinedOutput()
+		text := string(out)
+		result.GatewayUp = strings.Contains(text, "0% packet loss")
+		if i := strings.Index(text, "packet loss"); i > 0 {
+			start := strings.LastIndex(text[:i], " ")
+			if start >= 0 {
+				result.PacketLoss, _ = strconv.ParseFloat(strings.TrimSuffix(strings.TrimSpace(text[start:i]), "%"), 64)
+			}
+		}
+		if i := strings.Index(text, "min/avg/max"); i >= 0 {
+			if eq := strings.Index(text[i:], "="); eq >= 0 {
+				parts := strings.Split(strings.Fields(text[i+eq+1:])[0], "/")
+				if len(parts) > 1 {
+					result.LatencyMs, _ = strconv.ParseFloat(parts[1], 64)
+				}
+			}
+		}
+	}
+	resolver := net.Resolver{}
+	_, err := resolver.LookupHost(ctx, "deb.debian.org")
+	result.DNSUp = err == nil
+	client := http.Client{Timeout: 4 * time.Second}
+	request, _ := http.NewRequestWithContext(ctx, http.MethodGet, "https://api.ipify.org", nil)
+	if response, err := client.Do(request); err == nil {
+		defer response.Body.Close()
+		body, _ := io.ReadAll(io.LimitReader(response.Body, 128))
+		result.ExternalIP = strings.TrimSpace(string(body))
+		result.InternetUp = response.StatusCode == 200 && net.ParseIP(result.ExternalIP) != nil
+	}
+	result.ExternalIPChanged = previousIP != "" && result.ExternalIP != "" && previousIP != result.ExternalIP
+	return result
+}
+
+func readNetworkSnapshot(path string) (networkSnapshot, []string, []NetworkInterface, error) {
 	file, err := os.Open(path)
 	if err != nil {
-		return networkSnapshot{}, nil, err
+		return networkSnapshot{}, nil, nil, err
 	}
 	defer file.Close()
 
 	var received, sent uint64
 	var interfaces []string
+	var details []NetworkInterface
 	scanner := bufio.NewScanner(file)
 	for scanner.Scan() {
 		line := strings.TrimSpace(scanner.Text())
@@ -75,7 +278,7 @@ func readNetworkSnapshot(path string) (networkSnapshot, []string, error) {
 			continue
 		}
 		fields := strings.Fields(values)
-		if len(fields) < 9 {
+		if len(fields) < 16 {
 			continue
 		}
 		rx, errRX := strconv.ParseUint(fields[0], 10, 64)
@@ -84,7 +287,12 @@ func readNetworkSnapshot(path string) (networkSnapshot, []string, error) {
 			received += rx
 			sent += tx
 			interfaces = append(interfaces, strings.TrimSpace(name))
+			rxErr, _ := strconv.ParseUint(fields[2], 10, 64)
+			rxDrop, _ := strconv.ParseUint(fields[3], 10, 64)
+			txErr, _ := strconv.ParseUint(fields[10], 10, 64)
+			txDrop, _ := strconv.ParseUint(fields[11], 10, 64)
+			details = append(details, NetworkInterface{Name: strings.TrimSpace(name), ReceiveBytes: rx, TransmitBytes: tx, ReceiveErrors: rxErr, ReceiveDrops: rxDrop, TransmitErrors: txErr, TransmitDrops: txDrop})
 		}
 	}
-	return networkSnapshot{at: time.Now(), received: received, sent: sent}, interfaces, scanner.Err()
+	return networkSnapshot{at: time.Now(), received: received, sent: sent}, interfaces, details, scanner.Err()
 }

network_faults.go

@@ -0,0 +1,103 @@
+package main
+
+import (
+	"context"
+	"os/exec"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"time"
+)
+
+func counterDelta(current, previous uint64) uint64 {
+	if current < previous {
+		return 0
+	}
+	return current - previous
+}
+
+func updateInterfaceDeltas(v *NetworkInterface, old NetworkInterface, elapsed float64) {
+	if elapsed <= 0 {
+		return
+	}
+	v.ReceivePerSec = float64(counterDelta(v.ReceiveBytes, old.ReceiveBytes)) / elapsed
+	v.TransmitPerSec = float64(counterDelta(v.TransmitBytes, old.TransmitBytes)) / elapsed
+	v.NewErrors = counterDelta(v.ReceiveErrors, old.ReceiveErrors) + counterDelta(v.TransmitErrors, old.TransmitErrors)
+	v.NewReceiveDrops = counterDelta(v.ReceiveDrops, old.ReceiveDrops)
+	v.NewTransmitDrops = counterDelta(v.TransmitDrops, old.TransmitDrops)
+	v.NewDrops = v.NewReceiveDrops + v.NewTransmitDrops
+	for name, value := range v.FaultCounters {
+		if previous, ok := old.FaultCounters[name]; ok {
+			v.NewFaults += counterDelta(value, previous)
+		}
+	}
+	// A transition matters even if the interface has not carried traffic yet.
+	v.LinkDown = old.Carrier && !v.Carrier
+}
+
+func interfaceLinkDown(v NetworkInterface) bool {
+	if v.Kind == "Firewall bridge" || v.Kind == "Firewall link" {
+		return false
+	}
+	if v.LinkDown {
+		return true
+	}
+	used := v.Master != "" || v.ReceiveBytes+v.TransmitBytes > 0
+	if v.Kind == "Physical" {
+		return used && (!v.Carrier || v.State == "down")
+	}
+	return used && v.State == "down"
+}
+
+func collectInterfaceFaults(v *NetworkInterface) {
+	base := filepath.Join("/sys/class/net", v.Name)
+	// /proc/net/dev folds missed packets into RX drop. Sysfs keeps software
+	// drops separate from rx_missed_errors, which must still trigger an alert.
+	if n, err := strconv.ParseUint(readTrimmed(filepath.Join(base, "statistics/rx_dropped")), 10, 64); err == nil {
+		v.ReceiveDrops = n
+	}
+	v.FaultCounters = map[string]uint64{}
+	for _, name := range []string{"rx_crc_errors", "rx_frame_errors", "rx_fifo_errors", "rx_missed_errors", "tx_fifo_errors", "tx_carrier_errors", "carrier_down_count"} {
+		path := filepath.Join(base, "statistics", name)
+		if name == "carrier_down_count" {
+			path = filepath.Join(base, name)
+		}
+		if n, err := strconv.ParseUint(readTrimmed(path), 10, 64); err == nil {
+			v.FaultCounters[name] = n
+		}
+	}
+	// Only Atlantic exposes the driver-specific names below. No configuration
+	// changes, and a missing ethtool is not itself a network failure.
+	driver, err := filepath.EvalSymlinks(filepath.Join(base, "device/driver"))
+	if err != nil || filepath.Base(driver) != "atlantic" {
+		return
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), time.Second)
+	defer cancel()
+	if out, err := exec.CommandContext(ctx, "ethtool", "-S", v.Name).Output(); err == nil {
+		for k, n := range parseAtlanticFaults(string(out)) {
+			v.FaultCounters["atlantic:"+k] = n
+		}
+	}
+}
+
+func parseAtlanticFaults(output string) map[string]uint64 {
+	result := map[string]uint64{}
+	for _, line := range strings.Split(output, "\n") {
+		key, value, ok := strings.Cut(strings.TrimSpace(line), ":")
+		if !ok {
+			continue
+		}
+		fields := strings.Fields(key)
+		if len(fields) == 0 {
+			continue
+		}
+		switch fields[len(fields)-1] {
+		case "InErrors", "InDroppedDma", "AllocFails", "SkbAllocFails", "Restarts":
+			if n, err := strconv.ParseUint(strings.TrimSpace(value), 10, 64); err == nil {
+				result[key] = n
+			}
+		}
+	}
+	return result
+}

network_faults_test.go

@@ -0,0 +1,67 @@
+package main
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestNetworkDropsDoNotAlert(t *testing.T) {
+	for _, name := range []string{"enp6s0", "vmbr0", "fwbr100i0", "tap100i0"} {
+		v := NetworkInterface{Name: name, Carrier: true, State: "up", NewDrops: 15, NewReceiveDrops: 15}
+		metrics := dashboardMetrics{Network: NetworkMetrics{Interfaces: []NetworkInterface{v}}}
+		for _, a := range evaluateAlerts(metrics, AlertThresholds{}) {
+			if a.ID == "network-errors" {
+				t.Fatalf("%s: drops triggered alert", name)
+			}
+		}
+		v.NewErrors = 1
+		metrics.Network.Interfaces[0] = v
+		found := false
+		for _, a := range evaluateAlerts(metrics, AlertThresholds{}) {
+			if a.ID == "network-errors" {
+				found = a.Severity == "critical"
+			}
+		}
+		if !found {
+			t.Fatalf("%s: errors did not trigger critical alert", name)
+		}
+	}
+}
+
+func TestNetworkHardwareFaultAndLinkAlerts(t *testing.T) {
+	for _, v := range []NetworkInterface{{Name: "eth0", NewFaults: 1}, {Name: "eth0", LinkDown: true}} {
+		found := false
+		for _, a := range evaluateAlerts(dashboardMetrics{Network: NetworkMetrics{Interfaces: []NetworkInterface{v}}}, AlertThresholds{}) {
+			if strings.HasPrefix(a.ID, "network-") && a.Severity == "critical" {
+				found = true
+			}
+		}
+		if !found {
+			t.Fatalf("no critical alert for %+v", v)
+		}
+	}
+}
+
+func TestNetworkCounterResetsAndFaultBaselines(t *testing.T) {
+	old := NetworkInterface{ReceiveBytes: 100, ReceiveErrors: 20, TransmitErrors: 4, ReceiveDrops: 8, FaultCounters: map[string]uint64{"rx_missed_errors": 10, "atlantic:InErrors": 9}, Carrier: true}
+	next := NetworkInterface{ReceiveBytes: 1, ReceiveErrors: 1, TransmitErrors: 6, ReceiveDrops: 10, FaultCounters: map[string]uint64{"rx_missed_errors": 12, "atlantic:InErrors": 1, "new": 100}, Carrier: false}
+	updateInterfaceDeltas(&next, old, 2)
+	if next.NewErrors != 2 || next.NewFaults != 2 || next.NewReceiveDrops != 2 || next.ReceivePerSec != 0 || !next.LinkDown {
+		t.Fatalf("bad deltas: %+v", next)
+	}
+}
+
+func TestAtlanticFaults(t *testing.T) {
+	got := parseAtlanticFaults("NIC statistics:\n InErrors: 2\n InDroppedDma: 3\n Queue[0] AllocFails: 4\n Queue[1] SkbAllocFails: 5\n Queue[0] Restarts: 6\n Queue[0] InErrors: 7\n InPackets: 100\n Queue[0] XdpDrop: 50\n InErrorsBad: nope")
+	if len(got) != 6 || got["InDroppedDma"] != 3 || got["Queue[0] Restarts"] != 6 {
+		t.Fatalf("bad counters: %v", got)
+	}
+}
+
+func TestFirewallNotAttributedToGuest(t *testing.T) {
+	n := NetworkMetrics{Interfaces: []NetworkInterface{{Name: "fwbr100i0", ReceiveDrops: 100}, {Name: "tap100i0"}}}
+	decorateNetworkInterfaces(&n, GuestsMetrics{})
+	if n.Interfaces[0].VMID != 0 || n.Interfaces[1].VMID != 100 {
+		t.Fatalf("bad VM attribution: %+v", n)
+	}
+}

network_flows.go

@@ -0,0 +1,170 @@
+package main
+
+import (
+	"database/sql"
+	_ "embed"
+	"encoding/json"
+	"errors"
+	"io"
+	"net"
+	"net/http"
+	"time"
+)
+
+//go:embed assets/network.html
+var networkHTML string
+
+// Original and reply tuples describe one conntrack entry, not two connections.
+type NetworkFlow struct {
+	External             bool   `json:"external"`
+	Protocol             string `json:"protocol"`
+	Source               string `json:"source"`
+	Destination          string `json:"destination"`
+	SourcePort           int    `json:"sourcePort"`
+	DestinationPort      int    `json:"destinationPort"`
+	ReplySource          string `json:"replySource"`
+	ReplyDestination     string `json:"replyDestination"`
+	ReplySourcePort      int    `json:"replySourcePort"`
+	ReplyDestinationPort int    `json:"replyDestinationPort"`
+	State                string `json:"state"`
+	Country              string `json:"country"`
+	DestinationCountry   string `json:"destinationCountry"`
+	Packets              uint64 `json:"packets"`
+	ReplyPackets         uint64 `json:"replyPackets"`
+	Bytes                uint64 `json:"bytes"`
+	ReplyBytes           uint64 `json:"replyBytes"`
+	Accounting           bool   `json:"accounting"`
+	Zone                 string `json:"zone,omitempty"`
+}
+type FirewallCounter struct {
+	Name    string `json:"name"`
+	Packets uint64 `json:"packets"`
+	Bytes   uint64 `json:"bytes"`
+}
+type NetworkSnapshot struct {
+	At            int64             `json:"at"`
+	ReceivedAt    int64             `json:"receivedAt"`
+	Flows         []NetworkFlow     `json:"flows"`
+	Error         string            `json:"error,omitempty"`
+	Limited       bool              `json:"limited"`
+	ParseErrors   int               `json:"parseErrors"`
+	Firewall      []FirewallCounter `json:"firewall"`
+	FirewallAt    int64             `json:"firewallAt"`
+	FirewallError string            `json:"firewallError,omitempty"`
+	AgentName     string            `json:"agentName"`
+}
+
+func networkExternalIP(value string) bool {
+	ip := net.ParseIP(value)
+	if ip == nil || !ip.IsGlobalUnicast() || ip.IsPrivate() || ip.IsLoopback() || ip.IsLinkLocalUnicast() {
+		return false
+	}
+	if v4 := ip.To4(); v4 != nil && v4[0] == 100 && v4[1] >= 64 && v4[1] <= 127 {
+		return false
+	}
+	return true
+}
+func (s *Store) SaveNetwork(agent string, snapshot NetworkSnapshot) error {
+	if len(snapshot.Flows) > 5000 || len(snapshot.Firewall) > 128 {
+		return errors.New("network snapshot too large")
+	}
+	if snapshot.At < time.Now().Unix()-120 || snapshot.At > time.Now().Unix()+60 {
+		return errors.New("invalid snapshot time")
+	}
+	for i := range snapshot.Flows {
+		f := &snapshot.Flows[i]
+		if net.ParseIP(f.Source) == nil || net.ParseIP(f.Destination) == nil || net.ParseIP(f.ReplySource) == nil || net.ParseIP(f.ReplyDestination) == nil {
+			return errors.New("invalid network address")
+		}
+		for _, p := range []int{f.SourcePort, f.DestinationPort, f.ReplySourcePort, f.ReplyDestinationPort} {
+			if p < 0 || p > 65535 {
+				return errors.New("invalid network port")
+			}
+		}
+		if f.Protocol != "tcp" && f.Protocol != "udp" && f.Protocol != "icmp" && f.Protocol != "icmpv6" {
+			return errors.New("invalid protocol")
+		}
+		f.State = trafficText(f.State, 32)
+		f.Zone = trafficText(f.Zone, 16)
+		f.External = networkExternalIP(f.Source) || networkExternalIP(f.Destination)
+		f.Country = lookupTrafficCountry(f.Source)
+		f.DestinationCountry = lookupTrafficCountry(f.Destination)
+	}
+	snapshot.Error = trafficText(snapshot.Error, 600)
+	snapshot.FirewallError = trafficText(snapshot.FirewallError, 600)
+	for i := range snapshot.Firewall {
+		snapshot.Firewall[i].Name = trafficText(snapshot.Firewall[i].Name, 200)
+	}
+	snapshot.ReceivedAt = time.Now().Unix()
+	snapshot.AgentName = ""
+	raw, err := json.Marshal(snapshot)
+	if err != nil {
+		return err
+	}
+	_, err = s.db.Exec(`INSERT INTO network_sources(agent,snapshot) VALUES(?,?) ON CONFLICT(agent) DO UPDATE SET snapshot=excluded.snapshot`, agent, string(raw))
+	return err
+}
+func networkRoutes(mux *http.ServeMux, s *Store) {
+	enabled := func() bool { c := s.TrafficSettings(); return c.Enabled && c.NetworkEnabled }
+	mux.HandleFunc("/network", func(w http.ResponseWriter, r *http.Request) {
+		if !enabled() {
+			http.Error(w, "Включите TCP/UDP в настройках карты трафика", 404)
+			return
+		}
+		w.Header().Set("Content-Type", "text/html; charset=utf-8")
+		w.Write([]byte(networkHTML))
+	})
+	mux.HandleFunc("/api/network", func(w http.ResponseWriter, r *http.Request) {
+		if r.Method != "GET" {
+			w.WriteHeader(405)
+			return
+		}
+		if !enabled() {
+			http.Error(w, "Сбор сети отключён", 404)
+			return
+		}
+		var raw, name string
+		err := s.db.QueryRow(`SELECT n.snapshot,a.name FROM network_sources n JOIN agents a ON a.id=n.agent WHERE n.agent=?`, s.TrafficSettings().AgentID).Scan(&raw, &name)
+		if err != nil && err != sql.ErrNoRows {
+			writeJSON(w, nil, err)
+			return
+		}
+		if err != nil {
+			writeJSON(w, NetworkSnapshot{Flows: []NetworkFlow{}, Error: "Ожидание агента v0.21.0 или новее"}, nil)
+			return
+		}
+		var snapshot NetworkSnapshot
+		if err = json.Unmarshal([]byte(raw), &snapshot); err != nil {
+			writeJSON(w, nil, err)
+			return
+		}
+		snapshot.AgentName = name
+		writeJSON(w, snapshot, nil)
+	})
+	mux.HandleFunc("/api/agent/network", func(w http.ResponseWriter, r *http.Request) {
+		if r.Method != "POST" {
+			w.WriteHeader(405)
+			return
+		}
+		id, secret, ok := agentCredentials(r)
+		if !ok || !s.AuthenticateAgent(id, secret) {
+			http.Error(w, "Unauthorized", 401)
+			return
+		}
+		c := s.TrafficSettings()
+		if !c.Enabled || !c.NetworkEnabled || c.AgentID != id {
+			http.Error(w, "Источник отключён", 403)
+			return
+		}
+		var snapshot NetworkSnapshot
+		if json.NewDecoder(io.LimitReader(r.Body, 4<<20)).Decode(&snapshot) != nil {
+			http.Error(w, "Invalid snapshot", 400)
+			return
+		}
+		if err := s.SaveNetwork(id, snapshot); err != nil {
+			http.Error(w, err.Error(), 400)
+			return
+		}
+		writeJSON(w, map[string]bool{"accepted": true}, nil)
+	})
+}

network_flows_test.go

@@ -0,0 +1,189 @@
+package main
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+	"time"
+)
+
+const networkNATLine = "ipv4 2 tcp 6 431999 ESTABLISHED src=8.8.8.8 dst=192.168.0.12 sport=51000 dport=443 packets=12 bytes=1500 src=172.18.0.2 dst=8.8.8.8 sport=443 dport=51000 packets=16 bytes=2500 [ASSURED] mark=0 zone=2 use=1"
+
+func TestNetworkConntrackTuples(t *testing.T) {
+	f, ok := parseNetworkFlow(networkNATLine)
+	if !ok || f.Source != "8.8.8.8" || f.Destination != "192.168.0.12" || f.ReplySource != "172.18.0.2" || f.SourcePort != 51000 || f.ReplySourcePort != 443 || f.State != "ESTABLISHED" || !f.Accounting || f.Bytes != 1500 || f.ReplyBytes != 2500 || f.Zone != "2" {
+		t.Fatalf("NAT tuple corrupted: %+v", f)
+	}
+	udp, ok := parseNetworkFlow("ipv6 10 udp 17 29 src=2001:4860:4860::8888 dst=fd00::2 sport=51000 dport=53 [UNREPLIED] src=fd00::2 dst=2001:4860:4860::8888 sport=53 dport=51000 mark=0 use=1")
+	if !ok || udp.State != "UNREPLIED" || udp.Accounting || udp.Protocol != "udp" {
+		t.Fatalf("UDP: %+v", udp)
+	}
+	icmp, ok := parseNetworkFlow("icmp 1 29 src=8.8.8.8 dst=192.168.0.12 type=8 code=0 id=3 src=192.168.0.12 dst=8.8.8.8 type=0 code=0 id=3 use=1")
+	if !ok || icmp.SourcePort != 0 || icmp.State != "TRACKED" {
+		t.Fatalf("ICMP: %+v", icmp)
+	}
+	if _, ok := parseNetworkFlow("tcp 6 20 SYN_SENT src=not-an-ip dst=192.168.0.12"); ok {
+		t.Fatal("invalid accepted")
+	}
+}
+func TestNetworkSnapshotLimit(t *testing.T) {
+	var s NetworkSnapshot
+	appendNetworkFlows(&s, []byte(strings.Repeat(networkNATLine+"\n", 5001)))
+	if len(s.Flows) != 5000 || !s.Limited {
+		t.Fatalf("limit not enforced %d", len(s.Flows))
+	}
+	b := networkBuffer{limit: 2}
+	if _, err := b.Write([]byte("123")); err == nil || b.Len() != 0 {
+		t.Fatal("output unbounded")
+	}
+}
+func TestFirewallOnlyConfirmedDropCounters(t *testing.T) {
+	raw := `{"nftables":[{"rule":{"family":"ip","table":"crowdsec","chain":"crowdsec-chain","handle":3,"expr":[{"match":{}},{"counter":{"packets":10,"bytes":1000}},{"drop":null}]}},{"rule":{"table":"crowdsec","expr":[{"counter":{"packets":99}},{"accept":null}]}},{"rule":{"table":"other","expr":[{"counter":{"packets":99}},{"drop":null}]}},{"rule":{"table":"crowdsec","expr":[{"counter":{"packets":99}},{"match":{}},{"drop":null}]}},{"rule":{"table":"crowdsec","expr":[{"counter":"named-counter"},{"drop":null}]}}]}`
+	counters, err := parseFirewallCounters([]byte(raw))
+	if err != nil || len(counters) != 1 || counters[0].Packets != 10 || counters[0].Bytes != 1000 {
+		t.Fatalf("invented blocks: %+v %v", counters, err)
+	}
+	if _, err := parseFirewallCounters([]byte("invalid")); err == nil {
+		t.Fatal("invalid nft JSON accepted")
+	}
+}
+func TestNetworkRouteAuthorizationAndSourceSelection(t *testing.T) {
+	s, a := trafficFixture(t)
+	s.SaveTrafficSettings(TrafficSettings{Enabled: true, AgentID: a.ID})
+	mux := http.NewServeMux()
+	trafficRoutes(mux, s)
+	flow, _ := parseNetworkFlow(networkNATLine)
+	snapshot := NetworkSnapshot{At: time.Now().Unix(), Flows: []NetworkFlow{flow}}
+	raw, _ := json.Marshal(snapshot)
+	post := func(token string) int {
+		r := httptest.NewRequest("POST", "/api/agent/network", bytes.NewReader(raw))
+		r.Header.Set("Authorization", "Bearer "+token)
+		w := httptest.NewRecorder()
+		mux.ServeHTTP(w, r)
+		return w.Code
+	}
+	if got := post(a.ID + "." + a.Secret); got != 403 {
+		t.Fatalf("disabled accepts data: %d", got)
+	}
+	s.SaveTrafficSettings(TrafficSettings{Enabled: true, NetworkEnabled: true, AgentID: a.ID})
+	if got := post("wrong.token"); got != 401 {
+		t.Fatalf("unauthenticated: %d", got)
+	}
+	if got := post(a.ID + "." + a.Secret); got != 200 {
+		t.Fatalf("valid: %d", got)
+	}
+	w := httptest.NewRecorder()
+	mux.ServeHTTP(w, httptest.NewRequest("GET", "/api/network", nil))
+	var got NetworkSnapshot
+	json.Unmarshal(w.Body.Bytes(), &got)
+	if w.Code != 200 || len(got.Flows) != 1 || got.Flows[0].Country == "" || got.ReceivedAt == 0 {
+		t.Fatalf("bad snapshot: %s", w.Body.String())
+	}
+	token, _ := s.CreateAgentEnrollmentToken()
+	other, _ := s.EnrollAgent(agentEnrollmentRequest{Token: token, Name: "other"})
+	s.SaveTrafficSettings(TrafficSettings{Enabled: true, NetworkEnabled: true, AgentID: other.ID})
+	if code := post(a.ID + "." + a.Secret); code != 403 {
+		t.Fatalf("old source accepted: %d", code)
+	}
+	w = httptest.NewRecorder()
+	mux.ServeHTTP(w, httptest.NewRequest("GET", "/api/network", nil))
+	json.Unmarshal(w.Body.Bytes(), &got)
+	if len(got.Flows) != 0 {
+		t.Fatal("old source data leaked")
+	}
+	s.SaveTrafficSettings(TrafficSettings{Enabled: false, NetworkEnabled: true, AgentID: a.ID})
+	for _, path := range []string{"/network", "/api/network"} {
+		w = httptest.NewRecorder()
+		mux.ServeHTTP(w, httptest.NewRequest("GET", path, nil))
+		if w.Code != 404 {
+			t.Fatalf("disabled page: %s", path)
+		}
+	}
+}
+func TestNetworkRejectBadSnapshots(t *testing.T) {
+	s, a := trafficFixture(t)
+	if err := s.SaveNetwork(a.ID, NetworkSnapshot{At: time.Now().Unix() - 300}); err == nil {
+		t.Fatal("stale timestamp")
+	}
+	f, _ := parseNetworkFlow(networkNATLine)
+	f.SourcePort = 99999
+	if err := s.SaveNetwork(a.ID, NetworkSnapshot{At: time.Now().Unix(), Flows: []NetworkFlow{f}}); err == nil {
+		t.Fatal("invalid port")
+	}
+}
+func TestNetworkCommandCancellation(t *testing.T) {
+	ctx, cancel := context.WithCancel(context.Background())
+	cancel()
+	if _, err := networkCommand(ctx, "conntrack", "-L"); err == nil {
+		t.Fatal("cancelled command ran")
+	}
+}
+
+func TestNetworkExternalAddresses(t *testing.T) {
+	for _, ip := range []string{"192.168.1.1", "127.0.0.1", "100.64.0.1", "224.0.0.1", "ff02::1", "fe80::1", "fd00::1", "::"} {
+		if networkExternalIP(ip) {
+			t.Fatalf("not external: %s", ip)
+		}
+	}
+	for _, ip := range []string{"8.8.8.8", "2001:4860:4860::8888"} {
+		if !networkExternalIP(ip) {
+			t.Fatalf("external: %s", ip)
+		}
+	}
+}
+
+func TestConntrackInstallerOnlyInstallsRequestedPackage(t *testing.T) {
+	var calls [][]string
+	err := installConntrackPackage(func(name string, args ...string) error {
+		calls = append(calls, append([]string{name}, args...))
+		return nil
+	}, func() bool { return true })
+	if err != nil || len(calls) != 2 {
+		t.Fatalf("installer: %v %v", calls, err)
+	}
+	if calls[0][len(calls[0])-1] != "update" || strings.Join(calls[1][len(calls[1])-5:], " ") != "-y --no-install-recommends --no-remove install conntrack" {
+		t.Fatalf("unexpected apt operation: %v", calls)
+	}
+	calls = nil
+	err = installConntrackPackage(func(name string, args ...string) error { calls = append(calls, args); return fmt.Errorf("apt locked") }, func() bool { return true })
+	if err == nil || len(calls) != 1 {
+		t.Fatal("continued after failed apt update")
+	}
+}
+func TestConntrackDependencySingleInstallAndBackoff(t *testing.T) {
+	var d networkDependencyState
+	now := time.Now()
+	started := make(chan struct{})
+	finish := make(chan struct{})
+	installer := func() error { close(started); <-finish; return fmt.Errorf("offline") }
+	if got := d.ensure(now, true, installer); got != "" {
+		t.Fatal("installed tool should be skipped")
+	}
+	d.ensure(now, false, installer)
+	<-started
+	// A second call must not start a second installer (which would panic closing started).
+	d.ensure(now, false, installer)
+	close(finish)
+	deadline := time.Now().Add(time.Second)
+	for {
+		d.Lock()
+		running := d.running
+		d.Unlock()
+		if !running {
+			break
+		}
+		if time.Now().After(deadline) {
+			t.Fatal("installer stuck")
+		}
+		time.Sleep(time.Millisecond)
+	}
+	got := d.ensure(now.Add(time.Minute), false, installer)
+	if !strings.Contains(got, "offline") || !strings.Contains(got, "Повторная") {
+		t.Fatalf("backoff missing: %s", got)
+	}
+}

network_test.go

@@ -3,6 +3,7 @@ package main
 import (
 	"os"
 	"path/filepath"
+	"strings"
 	"testing"
 )
 
@@ -18,7 +19,7 @@ func TestReadNetworkSnapshot(t *testing.T) {
 		t.Fatal(err)
 	}
 
-	snapshot, interfaces, err := readNetworkSnapshot(path)
+	snapshot, interfaces, details, err := readNetworkSnapshot(path)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -28,4 +29,35 @@ func TestReadNetworkSnapshot(t *testing.T) {
 	if len(interfaces) != 2 {
 		t.Fatalf("неожиданный список интерфейсов: %v", interfaces)
 	}
+	if len(details) != 2 {
+		t.Fatalf("unexpected interface details: %+v", details)
+	}
+}
+
+func TestExplainNetworkHealthDistinguishesDNSFailure(t *testing.T) {
+	health := NetworkHealth{Gateway: "192.168.0.1", GatewayUp: true, InternetUp: true, DNSUp: false}
+	explainNetworkHealth(&health, nil)
+	if !strings.Contains(health.Summary, "DNS") || health.RecommendedAction == "" {
+		t.Fatalf("непонятная диагностика: %+v", health)
+	}
+}
+
+func TestExplainNetworkHealthIgnoresOldInterfaceCounters(t *testing.T) {
+	health := NetworkHealth{Gateway: "192.168.0.1", GatewayUp: true, InternetUp: true, DNSUp: true}
+	explainNetworkHealth(&health, []NetworkInterface{{Name: "eno1", Carrier: true, State: "up", ReceiveDrops: 42}})
+	if health.Summary != "Сеть работает штатно" {
+		t.Fatalf("старый накопленный счётчик вызвал текущую тревогу: %+v", health)
+	}
+	explainNetworkHealth(&health, []NetworkInterface{{Name: "eno1", Carrier: true, State: "up", ReceiveDrops: 43, NewDrops: 1}})
+	if health.Summary != "Сеть работает штатно" {
+		t.Fatalf("software drops вызвали тревогу: %+v", health)
+	}
+}
+
+func TestInterfaceVMID(t *testing.T) {
+	for name, want := range map[string]int{"tap100i0": 100, "veth108i0": 108, "fwbr113i0": 0, "vmbr0": 0, "enp6s0": 0} {
+		if got := interfaceVMID(name); got != want {
+			t.Fatalf("%s: got %d want %d", name, got, want)
+		}
+	}
 }

proxmox-cpu-dashboard.service

@@ -4,7 +4,7 @@ After=network-online.target
 Wants=network-online.target
 
 [Service]
-Type=simple
+Type=notify
 ExecStart=/usr/local/bin/proxmox-cpu-dashboard
 Environment=DASHBOARD_ADDR=:9105
 Environment=DASHBOARD_DB_PATH=/var/lib/proxmox-dashboard/dashboard.db
@@ -12,6 +12,8 @@ StateDirectory=proxmox-dashboard
 StateDirectoryMode=0750
 Restart=on-failure
 RestartSec=3
+WatchdogSec=30
+TimeoutStartSec=20
 NoNewPrivileges=true
 ProtectHome=true
 PrivateTmp=true

proxmox-dashboard-update.path

@@ -0,0 +1,9 @@
+[Unit]
+Description=Watch for manual Proxmox Dashboard update requests
+
+[Path]
+PathExists=/var/lib/proxmox-dashboard/update-request
+Unit=proxmox-dashboard-update.service
+
+[Install]
+WantedBy=multi-user.target

proxmox-dashboard-update.service

@@ -0,0 +1,10 @@
+[Unit]
+Description=Check and install Proxmox Dashboard updates
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/sbin/proxmox-dashboard-update
+Nice=10
+IOSchedulingClass=idle

proxmox-dashboard-update.timer

@@ -0,0 +1,11 @@
+[Unit]
+Description=Check Proxmox Dashboard updates daily
+
+[Timer]
+OnBootSec=10min
+OnUnitActiveSec=24h
+RandomizedDelaySec=30min
+Persistent=true
+
+[Install]
+WantedBy=timers.target

reliability.go

@@ -0,0 +1,244 @@
+package main
+
+import (
+	"context"
+	"fmt"
+	"log"
+	"net"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"sort"
+	"strings"
+	"sync"
+	"syscall"
+	"time"
+)
+
+func notifySystemd(message string) bool {
+	address := os.Getenv("NOTIFY_SOCKET")
+	if address == "" {
+		return false
+	}
+	if strings.HasPrefix(address, "@") {
+		address = "\x00" + address[1:]
+	}
+	connection, err := net.DialUnix("unixgram", nil, &net.UnixAddr{Name: address, Net: "unixgram"})
+	if err != nil {
+		return false
+	}
+	defer connection.Close()
+	_, err = connection.Write([]byte(message))
+	return err == nil
+}
+func runSystemdWatchdog() {
+	if !notifySystemd("READY=1\nSTATUS=Dashboard collects Proxmox metrics") {
+		return
+	}
+	go func() {
+		ticker := time.NewTicker(10 * time.Second)
+		defer ticker.Stop()
+		for range ticker.C {
+			notifySystemd("WATCHDOG=1")
+		}
+	}()
+}
+
+type MonitorHealth struct {
+	LastSuccessfulCollection time.Time `json:"lastSuccessfulCollection"`
+	LastCollectionError      string    `json:"lastCollectionError"`
+	CollectionDurationMs     int64     `json:"collectionDurationMs"`
+	SlowCollector            bool      `json:"slowCollector"`
+	LastHistoryWrite         time.Time `json:"lastHistoryWrite"`
+	HistoryWriteError        string    `json:"historyWriteError"`
+	DatabaseFreeBytes        uint64    `json:"databaseFreeBytes"`
+	DatabasePath             string    `json:"databasePath"`
+	WatchdogEnabled          bool      `json:"watchdogEnabled"`
+}
+
+type monitorTracker struct {
+	mu          sync.RWMutex
+	lastSuccess time.Time
+	lastError   string
+	duration    time.Duration
+}
+
+func runCollectionWatchdog(monitor *monitorTracker) {
+	started := time.Now()
+	ticker := time.NewTicker(15 * time.Second)
+	defer ticker.Stop()
+	for range ticker.C {
+		monitor.mu.RLock()
+		last := monitor.lastSuccess
+		monitor.mu.RUnlock()
+		if time.Since(started) < 2*time.Minute {
+			continue
+		}
+		if last.IsZero() || time.Since(last) > 90*time.Second {
+			log.Printf("Встроенный watchdog: сбор метрик не завершался более 90 секунд; перезапускаем процесс")
+			os.Exit(1)
+		}
+	}
+}
+
+func (m *monitorTracker) record(start time.Time, err error) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	m.duration = time.Since(start)
+	if err != nil {
+		m.lastError = err.Error()
+		return
+	}
+	m.lastSuccess = time.Now()
+	m.lastError = ""
+}
+
+func (m *monitorTracker) snapshot(store *Store) MonitorHealth {
+	m.mu.RLock()
+	result := MonitorHealth{LastSuccessfulCollection: m.lastSuccess, LastCollectionError: m.lastError, CollectionDurationMs: m.duration.Milliseconds(), SlowCollector: m.duration > 10*time.Second, DatabasePath: store.path, WatchdogEnabled: os.Getenv("NOTIFY_SOCKET") != ""}
+	m.mu.RUnlock()
+	result.LastHistoryWrite, result.HistoryWriteError = store.historyHealth()
+	path := store.path
+	if path == ":memory:" {
+		path = "/tmp"
+	} else {
+		path = filepath.Dir(path)
+	}
+	var stat syscall.Statfs_t
+	if syscall.Statfs(path, &stat) == nil {
+		result.DatabaseFreeBytes = stat.Bavail * uint64(stat.Bsize)
+	}
+	return result
+}
+
+type UPSShutdownSettings struct {
+	Enabled        bool    `json:"enabled"`
+	ChargePercent  float64 `json:"chargePercent"`
+	RuntimeSeconds float64 `json:"runtimeSeconds"`
+	GraceSeconds   int     `json:"graceSeconds"`
+	GuestOrder     []int   `json:"guestOrder"`
+}
+
+func defaultUPSShutdownSettings() UPSShutdownSettings {
+	return UPSShutdownSettings{ChargePercent: 15, RuntimeSeconds: 600, GraceSeconds: 120}
+}
+func (s *Store) UPSShutdownSettings() UPSShutdownSettings {
+	value := defaultUPSShutdownSettings()
+	s.loadJSONSetting("ups_shutdown", &value)
+	return value
+}
+func (s *Store) SaveUPSShutdownSettings(value UPSShutdownSettings) error {
+	if value.ChargePercent < 1 || value.ChargePercent > 80 || value.RuntimeSeconds < 60 || value.RuntimeSeconds > 7200 || value.GraceSeconds < 30 || value.GraceSeconds > 1800 {
+		return fmt.Errorf("проверьте пороги UPS и задержку")
+	}
+	return s.saveJSONSetting("ups_shutdown", value)
+}
+
+type ShutdownStep struct {
+	Order  int    `json:"order"`
+	Target string `json:"target"`
+	Action string `json:"action"`
+	Detail string `json:"detail"`
+}
+
+func buildShutdownDryRun(settings UPSShutdownSettings, guests GuestsMetrics) []ShutdownStep {
+	byID := map[int]GuestMetrics{}
+	for _, g := range guests.Guests {
+		if g.Template == 0 && g.Status == "running" {
+			byID[g.VMID] = g
+		}
+	}
+	var ordered []GuestMetrics
+	for _, id := range settings.GuestOrder {
+		if g, ok := byID[id]; ok {
+			ordered = append(ordered, g)
+			delete(byID, id)
+		}
+	}
+	var rest []GuestMetrics
+	for _, g := range byID {
+		rest = append(rest, g)
+	}
+	sort.Slice(rest, func(i, j int) bool { return rest[i].VMID < rest[j].VMID })
+	ordered = append(ordered, rest...)
+	steps := []ShutdownStep{{Order: 1, Target: "UPS", Action: "Ожидание порога", Detail: fmt.Sprintf("заряд ≤ %.0f%% или автономность ≤ %.0f сек.", settings.ChargePercent, settings.RuntimeSeconds)}}
+	for _, g := range ordered {
+		kind := "LXC"
+		if g.Type == "qemu" {
+			kind = "VM"
+		}
+		steps = append(steps, ShutdownStep{Order: len(steps) + 1, Target: fmt.Sprintf("%s %d · %s", kind, g.VMID, g.Name), Action: "Корректное завершение", Detail: fmt.Sprintf("ожидание до %d сек.", settings.GraceSeconds)})
+	}
+	steps = append(steps, ShutdownStep{Order: len(steps) + 1, Target: "Proxmox host", Action: "Завершение работы", Detail: "только после остановки VM/LXC"})
+	return steps
+}
+
+func runUPSShutdownController(store *Store, ups *upsCollector, guests *guestCollector) {
+	ticker := time.NewTicker(10 * time.Second)
+	defer ticker.Stop()
+	triggered := false
+	for range ticker.C {
+		settings := store.UPSShutdownSettings()
+		metrics := ups.collect()
+		onBattery := strings.Contains(metrics.Status, "OB") || metrics.Status == "ONBATT"
+		if !settings.Enabled || !metrics.Available || !onBattery {
+			if metrics.Available && !onBattery {
+				triggered = false
+			}
+			continue
+		}
+		lowCharge := metrics.ChargePercent != nil && *metrics.ChargePercent <= settings.ChargePercent
+		lowRuntime := metrics.RuntimeSeconds != nil && *metrics.RuntimeSeconds <= settings.RuntimeSeconds
+		if triggered || (!lowCharge && !lowRuntime) {
+			continue
+		}
+		triggered = true
+		log.Printf("UPS достиг порога безопасного выключения; запускаем остановку гостей")
+		go executeUPSShutdown(settings, guests.collect())
+	}
+}
+
+func executeUPSShutdown(settings UPSShutdownSettings, guests GuestsMetrics) {
+	steps := buildShutdownDryRun(settings, guests)
+	_ = steps
+	byID := map[int]GuestMetrics{}
+	for _, g := range guests.Guests {
+		if g.Template == 0 && g.Status == "running" {
+			byID[g.VMID] = g
+		}
+	}
+	var order []int
+	for _, id := range settings.GuestOrder {
+		if _, ok := byID[id]; ok {
+			order = append(order, id)
+			delete(byID, id)
+		}
+	}
+	var remaining []int
+	for id := range byID {
+		remaining = append(remaining, id)
+	}
+	sort.Ints(remaining)
+	order = append(order, remaining...)
+	for _, id := range order {
+		g := GuestMetrics{}
+		for _, candidate := range guests.Guests {
+			if candidate.VMID == id {
+				g = candidate
+				break
+			}
+		}
+		command := "pct"
+		if g.Type == "qemu" {
+			command = "qm"
+		}
+		ctx, cancel := context.WithTimeout(context.Background(), time.Duration(settings.GraceSeconds+30)*time.Second)
+		err := exec.CommandContext(ctx, command, "shutdown", fmt.Sprint(id), "--timeout", fmt.Sprint(settings.GraceSeconds)).Run()
+		cancel()
+		if err != nil {
+			log.Printf("UPS shutdown: не удалось остановить VMID %d: %v", id, err)
+		}
+	}
+	log.Printf("UPS shutdown: гости обработаны, выключаем Proxmox host")
+	_ = exec.Command("shutdown", "-h", "now").Run()
+}

reliability_test.go

@@ -0,0 +1,38 @@
+package main
+
+import (
+	"strings"
+	"testing"
+	"time"
+)
+
+func TestDiskCounterChanges(t *testing.T) {
+	old := DiskMetrics{Reallocated: 1, Pending: 2, CRCErrors: 4}
+	current := DiskMetrics{Reallocated: 2, Pending: 2, CRCErrors: 7}
+	changes := strings.Join(diskCounterChanges(old, current), " ")
+	if !strings.Contains(changes, "Reallocated") || !strings.Contains(changes, "CRC") || strings.Contains(changes, "Pending") {
+		t.Fatalf("unexpected counter changes: %s", changes)
+	}
+}
+
+func TestParseScrub(t *testing.T) {
+	status, at, duration, old := parseScrub("scrub repaired 0B in 01:12:03 with 0 errors on Fri Jan  2 03:04:05 2026")
+	if status != "Завершён" || at == "" || duration != "01:12:03" || !old {
+		t.Fatalf("unexpected scrub: %q %q %q %v", status, at, duration, old)
+	}
+}
+
+func TestShutdownDryRunUsesConfiguredOrder(t *testing.T) {
+	settings := UPSShutdownSettings{ChargePercent: 15, RuntimeSeconds: 600, GraceSeconds: 120, GuestOrder: []int{105}}
+	guests := GuestsMetrics{Guests: []GuestMetrics{{VMID: 101, Name: "one", Type: "qemu", Status: "running"}, {VMID: 105, Name: "five", Type: "lxc", Status: "running"}}}
+	steps := buildShutdownDryRun(settings, guests)
+	if len(steps) != 4 || !strings.Contains(steps[1].Target, "105") || !strings.Contains(steps[2].Target, "101") || steps[3].Target != "Proxmox host" {
+		t.Fatalf("unexpected plan: %+v", steps)
+	}
+}
+
+func TestSelfTestOlderThanWeek(t *testing.T) {
+	if !selfTestOlderThanWeek(time.Now().Add(-8*24*time.Hour).Format(time.RFC3339)) || selfTestOlderThanWeek(time.Now().Format(time.RFC3339)) {
+		t.Fatal("self-test age evaluation is incorrect")
+	}
+}

scripts/build-release.sh

@@ -0,0 +1,17 @@
+#!/bin/sh
+set -eu
+version=${1:?Usage: build-release.sh vX.Y.Z}
+cd "$(dirname "$0")/.."
+go test ./...
+mkdir -p dist
+GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags "-s -w -X main.version=$version" -o dist/proxmox-dashboard-linux-amd64 .
+(cd dist && shasum -a 256 proxmox-dashboard-linux-amd64 > proxmox-dashboard-linux-amd64.sha256)
+stage=$(mktemp -d)
+trap 'rm -rf "$stage"' EXIT INT TERM
+package="$stage/proxmox-dashboard-install"
+mkdir -p "$package/dist" "$package/scripts"
+cp dist/proxmox-dashboard-linux-amd64 dist/proxmox-dashboard-linux-amd64.sha256 "$package/dist/"
+cp scripts/install.sh scripts/install-agent.sh scripts/uninstall-agent.sh scripts/proxmox-dashboard-update "$package/scripts/"
+cp assets/NOTICE.md "$package/MAP-DATA-NOTICE.md"
+cp ./*.service ./*.timer ./*.path README.md "$package/"
+tar -czf dist/proxmox-dashboard-install-amd64.tar.gz -C "$stage" proxmox-dashboard-install

scripts/install-agent.sh

@@ -0,0 +1,65 @@
+#!/bin/sh
+set -eu
+
+if [ "$(id -u)" -ne 0 ]; then
+  echo "Запустите команду установки от root." >&2
+  exit 1
+fi
+if [ -z "${DASHBOARD_URL:-}" ] || [ -z "${ENROLL_TOKEN:-}" ]; then
+  echo "Не заданы DASHBOARD_URL и ENROLL_TOKEN. Создайте команду в настройках Dashboard." >&2
+  exit 1
+fi
+
+GITEA_URL=${GITEA_URL:-https://git.myown.center}
+GITEA_REPOSITORY=${GITEA_REPOSITORY:-maxim/ProxmoxDash}
+VMID=${VMID:-0}
+AGENT_NAME=${AGENT_NAME:-$(hostname)}
+
+release_json=$(curl -fsSL "$GITEA_URL/api/v1/repos/$GITEA_REPOSITORY/releases/latest")
+download_url=$(printf '%s' "$release_json" | sed -n 's/.*"browser_download_url":"\([^"]*\/proxmox-dashboard-linux-amd64\)".*/\1/p' | head -n 1)
+if [ -z "$download_url" ]; then
+  echo "В последнем релизе не найден Linux-бинарник." >&2
+  exit 1
+fi
+
+install -d -m 0700 /var/lib/proxmox-dashboard-agent
+temporary_binary=$(mktemp /tmp/proxmox-dashboard-agent.XXXXXX)
+trap 'rm -f "$temporary_binary"' EXIT
+curl -fsSL "$download_url" -o "$temporary_binary"
+install -m 0755 "$temporary_binary" /var/lib/proxmox-dashboard-agent/proxmox-dashboard-agent
+
+umask 077
+{
+  printf 'DASHBOARD_URL=%s\n' "$DASHBOARD_URL"
+  printf 'ENROLL_TOKEN=%s\n' "$ENROLL_TOKEN"
+  printf 'AGENT_NAME=%s\n' "$AGENT_NAME"
+  printf 'VMID=%s\n' "$VMID"
+} > /etc/proxmox-dashboard-agent.env
+
+cat > /etc/systemd/system/proxmox-dashboard-agent.service <<'UNIT'
+[Unit]
+Description=Proxmox Dashboard read-only guest agent
+After=network-online.target docker.service
+Wants=network-online.target
+
+[Service]
+Type=simple
+EnvironmentFile=/etc/proxmox-dashboard-agent.env
+ExecStart=/var/lib/proxmox-dashboard-agent/proxmox-dashboard-agent --agent --agent-server ${DASHBOARD_URL} --agent-enroll-token ${ENROLL_TOKEN} --agent-name ${AGENT_NAME} --agent-vmid ${VMID}
+Restart=always
+RestartSec=5
+NoNewPrivileges=true
+ProtectHome=true
+PrivateTmp=true
+ProtectSystem=strict
+ReadWritePaths=/var/lib/proxmox-dashboard-agent
+
+[Install]
+WantedBy=multi-user.target
+UNIT
+
+systemctl daemon-reload
+systemctl enable proxmox-dashboard-agent.service
+systemctl restart proxmox-dashboard-agent.service
+rm -f /usr/local/bin/proxmox-dashboard-agent
+echo "Агент установлен. Через несколько секунд он появится в Dashboard."

scripts/install.sh

@@ -0,0 +1,37 @@
+#!/bin/sh
+set -eu
+
+if [ "$(id -u)" -ne 0 ]; then
+  echo "Запустите установщик от root." >&2
+  exit 1
+fi
+
+SOURCE_DIR=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
+UPDATE_MODE=${UPDATE_MODE:-notify}
+GITEA_URL=${GITEA_URL:-https://git.myown.center}
+GITEA_REPOSITORY=${GITEA_REPOSITORY:-maxim/ProxmoxDash}
+GITEA_TOKEN=${GITEA_TOKEN:-}
+
+install -d -m 0750 /etc/proxmox-dashboard /var/lib/proxmox-dashboard
+install -m 0755 "$SOURCE_DIR/dist/proxmox-dashboard-linux-amd64" /usr/local/bin/proxmox-cpu-dashboard
+install -m 0755 "$SOURCE_DIR/scripts/proxmox-dashboard-update" /usr/local/sbin/proxmox-dashboard-update
+install -m 0644 "$SOURCE_DIR/proxmox-cpu-dashboard.service" /etc/systemd/system/proxmox-cpu-dashboard.service
+install -m 0644 "$SOURCE_DIR/proxmox-dashboard-update.service" /etc/systemd/system/proxmox-dashboard-update.service
+install -m 0644 "$SOURCE_DIR/proxmox-dashboard-update.timer" /etc/systemd/system/proxmox-dashboard-update.timer
+install -m 0644 "$SOURCE_DIR/proxmox-dashboard-update.path" /etc/systemd/system/proxmox-dashboard-update.path
+
+umask 077
+{
+  printf 'GITEA_URL=%s\n' "$GITEA_URL"
+  printf 'GITEA_REPOSITORY=%s\n' "$GITEA_REPOSITORY"
+  printf 'GITEA_TOKEN=%s\n' "$GITEA_TOKEN"
+  printf 'UPDATE_MODE=%s\n' "$UPDATE_MODE"
+} > /etc/proxmox-dashboard/update.env
+
+systemctl daemon-reload
+systemctl enable proxmox-cpu-dashboard.service
+systemctl restart proxmox-cpu-dashboard.service
+systemctl enable --now proxmox-dashboard-update.timer
+systemctl enable --now proxmox-dashboard-update.path
+systemctl start proxmox-dashboard-update.service
+echo "Proxmox Dashboard установлен. Откройте http://$(hostname -I | awk '{print $1}'):9105"

scripts/proxmox-dashboard-update

@@ -0,0 +1,111 @@
+#!/bin/sh
+set -eu
+
+CONFIG_FILE=${DASHBOARD_UPDATE_CONFIG:-/etc/proxmox-dashboard/update.env}
+STATUS_FILE=${DASHBOARD_UPDATE_STATUS_PATH:-/var/lib/proxmox-dashboard/update-status.json}
+BINARY_PATH=${DASHBOARD_BINARY_PATH:-/usr/local/bin/proxmox-cpu-dashboard}
+REQUEST_FILE=${DASHBOARD_UPDATE_REQUEST_PATH:-/var/lib/proxmox-dashboard/update-request}
+
+if [ -f "$CONFIG_FILE" ]; then
+  # shellcheck disable=SC1090
+  . "$CONFIG_FILE"
+fi
+
+GITEA_URL=${GITEA_URL:-https://git.myown.center}
+GITEA_REPOSITORY=${GITEA_REPOSITORY:-maxim/ProxmoxDash}
+GITEA_TOKEN=${GITEA_TOKEN:-}
+UPDATE_MODE=${UPDATE_MODE:-notify}
+ASSET_NAME=${ASSET_NAME:-proxmox-dashboard-linux-amd64}
+manual_request=false
+if [ -f "$REQUEST_FILE" ]; then
+  manual_request=true
+  rm -f "$REQUEST_FILE"
+fi
+
+mkdir -p "$(dirname "$STATUS_FILE")"
+tmp_dir=$(mktemp -d /tmp/proxmox-dashboard-update.XXXXXX)
+trap 'rm -rf "$tmp_dir"' EXIT INT TERM
+
+auth_header=""
+if [ -n "$GITEA_TOKEN" ]; then
+  auth_header="Authorization: token $GITEA_TOKEN"
+fi
+
+api_get() {
+  if [ -n "$auth_header" ]; then
+    curl --fail --silent --show-error --location --header "$auth_header" "$1"
+  else
+    curl --fail --silent --show-error --location "$1"
+  fi
+}
+
+write_status() {
+  python3 - "$STATUS_FILE" "$1" "$2" "$3" "$4" "$5" "$6" <<'PY'
+import datetime, json, os, sys, tempfile
+path, current, available, state, mode, message, release_url = sys.argv[1:]
+payload = {
+    "currentVersion": current,
+    "availableVersion": available,
+    "state": state,
+    "mode": mode,
+    "message": message,
+    "releaseUrl": release_url,
+    "checkedAt": datetime.datetime.now(datetime.timezone.utc).isoformat(),
+}
+fd, temporary = tempfile.mkstemp(dir=os.path.dirname(path), prefix=".update-status-")
+with os.fdopen(fd, "w") as stream:
+    json.dump(payload, stream)
+os.chmod(temporary, 0o644)
+os.replace(temporary, path)
+PY
+}
+
+current_version=$($BINARY_PATH -version 2>/dev/null || printf 'dev')
+release_json=$(api_get "$GITEA_URL/api/v1/repos/$GITEA_REPOSITORY/releases/latest")
+printf '%s' "$release_json" > "$tmp_dir/release.json"
+release_data=$(python3 - "$tmp_dir/release.json" "$ASSET_NAME" <<'PY'
+import json, sys
+release = json.load(open(sys.argv[1]))
+asset_name = sys.argv[2]
+assets = {item.get("name"): item.get("browser_download_url") for item in release.get("assets", [])}
+binary = assets.get(asset_name, "")
+checksum = assets.get(asset_name + ".sha256", "")
+print(release.get("tag_name", ""))
+print(release.get("html_url", ""))
+print(binary)
+print(checksum)
+PY
+)
+latest_version=$(printf '%s\n' "$release_data" | sed -n '1p')
+release_url=$(printf '%s\n' "$release_data" | sed -n '2p')
+binary_url=$(printf '%s\n' "$release_data" | sed -n '3p')
+checksum_url=$(printf '%s\n' "$release_data" | sed -n '4p')
+
+if [ -z "$latest_version" ] || [ -z "$binary_url" ]; then
+  write_status "$current_version" "$latest_version" error "$UPDATE_MODE" "В свежем релизе не найден бинарник $ASSET_NAME." "$release_url"
+  exit 1
+fi
+
+newest=$(printf '%s\n%s\n' "$current_version" "$latest_version" | sort -V | tail -n 1)
+if [ "$current_version" = "$latest_version" ] || [ "$newest" = "$current_version" ]; then
+  write_status "$current_version" "$latest_version" current "$UPDATE_MODE" "Установлена свежая версия." "$release_url"
+  exit 0
+fi
+
+if [ "$UPDATE_MODE" != "auto" ] && [ "$manual_request" != "true" ]; then
+  write_status "$current_version" "$latest_version" available "$UPDATE_MODE" "Доступна новая версия $latest_version." "$release_url"
+  exit 0
+fi
+
+write_status "$current_version" "$latest_version" installing "$UPDATE_MODE" "Скачиваем и проверяем $latest_version." "$release_url"
+api_get "$binary_url" > "$tmp_dir/$ASSET_NAME"
+if [ -n "$checksum_url" ]; then
+  api_get "$checksum_url" > "$tmp_dir/$ASSET_NAME.sha256"
+  (cd "$tmp_dir" && sha256sum -c "$ASSET_NAME.sha256")
+fi
+chmod 0755 "$tmp_dir/$ASSET_NAME"
+file "$tmp_dir/$ASSET_NAME" | grep -q 'ELF 64-bit'
+cp -p "$BINARY_PATH" "$BINARY_PATH.previous"
+install -m 0755 "$tmp_dir/$ASSET_NAME" "$BINARY_PATH"
+write_status "$latest_version" "$latest_version" updated "$UPDATE_MODE" "Обновление $latest_version установлено." "$release_url"
+systemctl restart proxmox-cpu-dashboard.service

scripts/uninstall-agent.sh

@@ -0,0 +1,16 @@
+#!/bin/sh
+set -eu
+
+if [ "$(id -u)" -ne 0 ]; then
+  echo "Запустите удаление от root." >&2
+  exit 1
+fi
+
+systemctl disable --now proxmox-dashboard-agent.service 2>/dev/null || true
+rm -f /etc/systemd/system/proxmox-dashboard-agent.service
+rm -f /etc/proxmox-dashboard-agent.env
+rm -f /usr/local/bin/proxmox-dashboard-agent
+rm -rf /var/lib/proxmox-dashboard-agent
+systemctl daemon-reload
+systemctl reset-failed proxmox-dashboard-agent.service 2>/dev/null || true
+echo "Агент и его локальные учётные данные удалены."

service_links.go

@@ -0,0 +1,110 @@
+package main
+
+import (
+	"fmt"
+	"sort"
+	"strings"
+)
+
+type ServiceLink struct {
+	ServiceID     int64  `json:"serviceId"`
+	AgentID       string `json:"agentId"`
+	VMID          int    `json:"vmid"`
+	ComponentType string `json:"componentType"`
+	ComponentName string `json:"componentName"`
+	Port          int    `json:"port"`
+}
+
+func (s *Store) ServiceLinks() ([]ServiceLink, error) {
+	rows, err := s.db.Query(`SELECT service_id,agent_id,vmid,component_type,component_name,port FROM service_links ORDER BY service_id`)
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+	var out []ServiceLink
+	for rows.Next() {
+		var v ServiceLink
+		if err = rows.Scan(&v.ServiceID, &v.AgentID, &v.VMID, &v.ComponentType, &v.ComponentName, &v.Port); err != nil {
+			return nil, err
+		}
+		out = append(out, v)
+	}
+	return out, rows.Err()
+}
+
+func (s *Store) SaveServiceLink(v ServiceLink) (ServiceLink, error) {
+	v.AgentID = strings.TrimSpace(v.AgentID)
+	v.ComponentType = strings.TrimSpace(v.ComponentType)
+	v.ComponentName = strings.TrimSpace(v.ComponentName)
+	if v.ServiceID <= 0 {
+		return v, fmt.Errorf("выберите сервис")
+	}
+	if v.AgentID == "" && v.VMID <= 0 {
+		return v, fmt.Errorf("выберите агент или VMID")
+	}
+	if v.ComponentType != "" && v.ComponentType != "systemd" && v.ComponentType != "docker" && v.ComponentType != "process" {
+		return v, fmt.Errorf("неизвестный тип компонента")
+	}
+	if v.Port < 0 || v.Port > 65535 {
+		return v, fmt.Errorf("некорректный порт")
+	}
+	_, err := s.db.Exec(`INSERT INTO service_links(service_id,agent_id,vmid,component_type,component_name,port) VALUES(?,?,?,?,?,?) ON CONFLICT(service_id) DO UPDATE SET agent_id=excluded.agent_id,vmid=excluded.vmid,component_type=excluded.component_type,component_name=excluded.component_name,port=excluded.port`, v.ServiceID, v.AgentID, v.VMID, v.ComponentType, v.ComponentName, v.Port)
+	return v, err
+}
+func (s *Store) DeleteServiceLink(serviceID int64) error {
+	_, err := s.db.Exec(`DELETE FROM service_links WHERE service_id=?`, serviceID)
+	return err
+}
+
+func groupLinkedAlerts(metrics dashboardMetrics, alerts []Alert) []Alert {
+	if len(metrics.ServiceLinks) == 0 {
+		return alerts
+	}
+	agentOffline := map[string]ManagedAgent{}
+	for _, agent := range metrics.Agents {
+		if !agent.Online {
+			agentOffline[agent.ID] = agent
+		}
+	}
+	affected := map[string][]string{}
+	remove := map[string]bool{}
+	for _, link := range metrics.ServiceLinks {
+		agent, offline := agentOffline[link.AgentID]
+		if !offline {
+			continue
+		}
+		for _, service := range metrics.Services.Services {
+			if service.ID != link.ServiceID {
+				continue
+			}
+			for _, endpoint := range service.Endpoints {
+				if !endpoint.Up {
+					key := fmt.Sprintf("service-%d-%s", service.ID, endpoint.Kind)
+					remove[key] = true
+					affected[agent.ID] = append(affected[agent.ID], service.Name)
+				}
+			}
+		}
+	}
+	if len(affected) == 0 {
+		return alerts
+	}
+	out := make([]Alert, 0, len(alerts))
+	for _, alert := range alerts {
+		skip := remove[alert.ID]
+		for agentID := range affected {
+			if alert.ID == "agent-offline-"+agentID+fmt.Sprintf("-%d", agentOffline[agentID].VMID) {
+				skip = true
+			}
+		}
+		if !skip {
+			out = append(out, alert)
+		}
+	}
+	for agentID, names := range affected {
+		sort.Strings(names)
+		agent := agentOffline[agentID]
+		out = append(out, Alert{ID: "group-agent-offline-" + agentID + fmt.Sprintf("-%d", agent.VMID), Severity: "critical", Source: "Корреляция", Title: agent.Name + ": агент недоступен, связанные сервисы не отвечают", Message: fmt.Sprintf("VMID %d. Затронуто сервисов: %d — %s. Вероятная первопричина: недоступна гостевая система или агент.", agent.VMID, len(names), strings.Join(names, ", "))})
+	}
+	return out
+}

services.go

@@ -0,0 +1,353 @@
+package main
+
+import (
+	"context"
+	"crypto/tls"
+	"database/sql"
+	"fmt"
+	"io"
+	"net"
+	"net/http"
+	"net/url"
+	"os/exec"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+)
+
+type MonitoredService struct {
+	ID             int64  `json:"id"`
+	Name           string `json:"name"`
+	URL            string `json:"url"`
+	LocalAddress   string `json:"localAddress"`
+	CheckType      string `json:"checkType"`
+	ExpectedCode   int    `json:"expectedCode"`
+	ExpectedText   string `json:"expectedText"`
+	TimeoutSeconds int    `json:"timeoutSeconds"`
+	Enabled        bool   `json:"enabled"`
+}
+
+type ServiceStatus struct {
+	MonitoredService
+	Endpoints  []ServiceEndpointStatus `json:"endpoints"`
+	Up         bool                    `json:"up"`
+	CheckedAt  int64                   `json:"checkedAt"`
+	LatencyMS  float64                 `json:"latencyMs"`
+	StatusCode int                     `json:"statusCode"`
+	Error      string                  `json:"error,omitempty"`
+	TLSExpiry  *int64                  `json:"tlsExpiry,omitempty"`
+	TLSDays    *int                    `json:"tlsDays,omitempty"`
+	UptimeDay  float64                 `json:"uptimeDay"`
+	UptimeWeek float64                 `json:"uptimeWeek"`
+}
+
+type ServiceEndpointStatus struct {
+	Kind             string   `json:"kind"`
+	Label            string   `json:"label"`
+	Address          string   `json:"address"`
+	Up               bool     `json:"up"`
+	CheckedAt        int64    `json:"checkedAt"`
+	LatencyMS        float64  `json:"latencyMs"`
+	StatusCode       int      `json:"statusCode"`
+	Error            string   `json:"error,omitempty"`
+	TLSExpiry        *int64   `json:"tlsExpiry,omitempty"`
+	TLSDays          *int     `json:"tlsDays,omitempty"`
+	TLSSubject       string   `json:"tlsSubject,omitempty"`
+	TLSIssuer        string   `json:"tlsIssuer,omitempty"`
+	TLSDNSNames      []string `json:"tlsDnsNames,omitempty"`
+	TLSHostnameValid bool     `json:"tlsHostnameValid"`
+	TLSHostnameError string   `json:"tlsHostnameError,omitempty"`
+	UptimeDay        float64  `json:"uptimeDay"`
+	UptimeWeek       float64  `json:"uptimeWeek"`
+}
+
+type ServicesMetrics struct {
+	Available bool            `json:"available"`
+	Up        int             `json:"up"`
+	Total     int             `json:"total"`
+	Services  []ServiceStatus `json:"services"`
+}
+type ServiceHistoryPoint struct {
+	Timestamp int64   `json:"timestamp"`
+	Up        bool    `json:"up"`
+	LatencyMS float64 `json:"latencyMs"`
+}
+
+var serviceHTTPTransport = &http.Transport{
+	Proxy:                 http.ProxyFromEnvironment,
+	DialContext:           (&net.Dialer{Timeout: 10 * time.Second, KeepAlive: 30 * time.Second}).DialContext,
+	ForceAttemptHTTP2:     true,
+	MaxIdleConns:          32,
+	MaxIdleConnsPerHost:   2,
+	IdleConnTimeout:       30 * time.Second,
+	TLSHandshakeTimeout:   10 * time.Second,
+	ExpectContinueTimeout: time.Second,
+	TLSClientConfig:       &tls.Config{InsecureSkipVerify: true, MinVersion: tls.VersionTLS12},
+}
+
+var serviceHTTPClient = &http.Client{Transport: serviceHTTPTransport}
+
+func (s *Store) Services() ([]MonitoredService, error) {
+	rows, err := s.db.Query(`SELECT id,name,url,local_address,check_type,expected_code,expected_text,timeout_seconds,enabled FROM services ORDER BY id`)
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+	var result []MonitoredService
+	for rows.Next() {
+		var v MonitoredService
+		if err = rows.Scan(&v.ID, &v.Name, &v.URL, &v.LocalAddress, &v.CheckType, &v.ExpectedCode, &v.ExpectedText, &v.TimeoutSeconds, &v.Enabled); err != nil {
+			return nil, err
+		}
+		result = append(result, v)
+	}
+	return result, rows.Err()
+}
+
+func validateService(v MonitoredService) error {
+	v.Name = strings.TrimSpace(v.Name)
+	v.URL = strings.TrimSpace(v.URL)
+	v.LocalAddress = strings.TrimSpace(v.LocalAddress)
+	if v.Name == "" || (v.URL == "" && v.LocalAddress == "") {
+		return fmt.Errorf("укажите название и хотя бы один адрес")
+	}
+	if v.CheckType != "http" && v.CheckType != "tcp" && v.CheckType != "ping" {
+		return fmt.Errorf("неизвестный тип проверки")
+	}
+	if v.TimeoutSeconds < 1 || v.TimeoutSeconds > 30 {
+		return fmt.Errorf("таймаут должен быть от 1 до 30 секунд")
+	}
+	if v.CheckType == "http" {
+		for _, target := range []string{v.URL, v.LocalAddress} {
+			if target == "" {
+				continue
+			}
+			u, err := url.Parse(target)
+			if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
+				return fmt.Errorf("оба адреса должны начинаться с http:// или https://")
+			}
+		}
+	}
+	return nil
+}
+
+func (s *Store) SaveService(v MonitoredService) (MonitoredService, error) {
+	if err := validateService(v); err != nil {
+		return v, err
+	}
+	if v.ExpectedCode == 0 {
+		v.ExpectedCode = 200
+	}
+	if v.ID == 0 {
+		r, err := s.db.Exec(`INSERT INTO services(name,url,local_address,check_type,expected_code,expected_text,timeout_seconds,enabled) VALUES(?,?,?,?,?,?,?,?)`, v.Name, v.URL, v.LocalAddress, v.CheckType, v.ExpectedCode, v.ExpectedText, v.TimeoutSeconds, v.Enabled)
+		if err != nil {
+			return v, err
+		}
+		v.ID, _ = r.LastInsertId()
+	} else {
+		_, err := s.db.Exec(`UPDATE services SET name=?,url=?,local_address=?,check_type=?,expected_code=?,expected_text=?,timeout_seconds=?,enabled=? WHERE id=?`, v.Name, v.URL, v.LocalAddress, v.CheckType, v.ExpectedCode, v.ExpectedText, v.TimeoutSeconds, v.Enabled, v.ID)
+		if err != nil {
+			return v, err
+		}
+	}
+	return v, nil
+}
+func (s *Store) DeleteService(id int64) error {
+	_, err := s.db.Exec(`DELETE FROM services WHERE id=?`, id)
+	return err
+}
+func (s *Store) addServiceCheck(id int64, endpoint string, v ServiceEndpointStatus) {
+	var expiry any
+	if v.TLSExpiry != nil {
+		expiry = *v.TLSExpiry
+	}
+	_, _ = s.db.Exec(`INSERT INTO service_checks(service_id,ts,up,latency_ms,status_code,error,tls_expiry,endpoint) VALUES(?,?,?,?,?,?,?,?)`, id, v.CheckedAt, v.Up, v.LatencyMS, v.StatusCode, v.Error, expiry, endpoint)
+	_, _ = s.db.Exec(`DELETE FROM service_checks WHERE ts<?`, time.Now().Add(-historyRetention).Unix())
+}
+func (s *Store) serviceUptime(id int64, endpoint string, since time.Time) float64 {
+	var value sql.NullFloat64
+	_ = s.db.QueryRow(`SELECT AVG(up)*100 FROM service_checks WHERE service_id=? AND endpoint=? AND ts>=?`, id, endpoint, since.Unix()).Scan(&value)
+	if value.Valid {
+		return value.Float64
+	}
+	return 0
+}
+func (s *Store) ServiceHistory(id int64, hours int) ([]ServiceHistoryPoint, error) {
+	if hours < 1 {
+		hours = 24
+	}
+	if hours > 168 {
+		hours = 168
+	}
+	rows, err := s.db.Query(`SELECT ts,up,latency_ms FROM service_checks WHERE service_id=? AND ts>=? ORDER BY ts`, id, time.Now().Add(-time.Duration(hours)*time.Hour).Unix())
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+	var out []ServiceHistoryPoint
+	for rows.Next() {
+		var p ServiceHistoryPoint
+		if err = rows.Scan(&p.Timestamp, &p.Up, &p.LatencyMS); err != nil {
+			return nil, err
+		}
+		out = append(out, p)
+	}
+	return out, rows.Err()
+}
+
+type serviceCollector struct {
+	mu      sync.RWMutex
+	metrics ServicesMetrics
+	store   *Store
+}
+
+func newServiceCollector(store *Store) *serviceCollector {
+	c := &serviceCollector{store: store}
+	go c.run()
+	return c
+}
+func (c *serviceCollector) collect() ServicesMetrics {
+	c.mu.RLock()
+	defer c.mu.RUnlock()
+	return c.metrics
+}
+func (c *serviceCollector) run() {
+	c.checkAll()
+	ticker := time.NewTicker(15 * time.Second)
+	defer ticker.Stop()
+	for range ticker.C {
+		c.checkAll()
+	}
+}
+func (c *serviceCollector) checkAll() {
+	items, err := c.store.Services()
+	if err != nil {
+		return
+	}
+	m := ServicesMetrics{Available: true}
+	for _, v := range items {
+		if !v.Enabled {
+			continue
+		}
+		st := checkService(v)
+		for i := range st.Endpoints {
+			ep := &st.Endpoints[i]
+			ep.UptimeDay = c.store.serviceUptime(v.ID, ep.Kind, time.Now().Add(-24*time.Hour))
+			ep.UptimeWeek = c.store.serviceUptime(v.ID, ep.Kind, time.Now().Add(-7*24*time.Hour))
+			c.store.addServiceCheck(v.ID, ep.Kind, *ep)
+		}
+		if len(st.Endpoints) > 0 {
+			st.UptimeDay = st.Endpoints[0].UptimeDay
+			st.UptimeWeek = st.Endpoints[0].UptimeWeek
+		}
+		m.Total++
+		if st.Up {
+			m.Up++
+		}
+		m.Services = append(m.Services, st)
+	}
+	c.mu.Lock()
+	c.metrics = m
+	c.mu.Unlock()
+}
+
+func checkService(v MonitoredService) ServiceStatus {
+	st := ServiceStatus{MonitoredService: v, CheckedAt: time.Now().Unix(), Up: true}
+	if v.URL != "" {
+		st.Endpoints = append(st.Endpoints, checkServiceEndpoint(v, "public", "Внешний адрес", v.URL))
+	}
+	if v.LocalAddress != "" {
+		st.Endpoints = append(st.Endpoints, checkServiceEndpoint(v, "local", "Локальный адрес", v.LocalAddress))
+	}
+	for _, ep := range st.Endpoints {
+		if !ep.Up {
+			st.Up = false
+		}
+		st.LatencyMS += ep.LatencyMS
+		if st.StatusCode == 0 {
+			st.StatusCode = ep.StatusCode
+		}
+		if st.Error == "" && ep.Error != "" {
+			st.Error = ep.Label + ": " + ep.Error
+		}
+	}
+	if len(st.Endpoints) > 0 {
+		st.LatencyMS /= float64(len(st.Endpoints))
+		ep := st.Endpoints[0]
+		st.TLSExpiry = ep.TLSExpiry
+		st.TLSDays = ep.TLSDays
+	}
+	return st
+}
+
+func checkServiceEndpoint(v MonitoredService, kind, label, target string) ServiceEndpointStatus {
+	st := ServiceEndpointStatus{Kind: kind, Label: label, Address: target, CheckedAt: time.Now().Unix()}
+	start := time.Now()
+	ctx, cancel := context.WithTimeout(context.Background(), time.Duration(v.TimeoutSeconds)*time.Second)
+	defer cancel()
+	switch v.CheckType {
+	case "tcp":
+		d := net.Dialer{}
+		conn, err := d.DialContext(ctx, "tcp", target)
+		if err == nil {
+			st.Up = true
+			conn.Close()
+		} else {
+			st.Error = err.Error()
+		}
+	case "ping":
+		host := strings.TrimPrefix(strings.TrimPrefix(target, "https://"), "http://")
+		host = strings.Split(host, "/")[0]
+		host = strings.Split(host, ":")[0]
+		arg := "-W"
+		timeout := strconv.Itoa(v.TimeoutSeconds)
+		if _, err := exec.LookPath("ping"); err != nil {
+			st.Error = "ping не установлен"
+		} else if out, err := exec.CommandContext(ctx, "ping", "-c", "1", arg, timeout, host).CombinedOutput(); err == nil {
+			st.Up = true
+		} else {
+			st.Error = strings.TrimSpace(string(out))
+		}
+	default:
+		req, err := http.NewRequestWithContext(ctx, http.MethodGet, target, nil)
+		if err != nil {
+			st.Error = err.Error()
+			break
+		}
+		resp, err := serviceHTTPClient.Do(req)
+		if err != nil {
+			st.Error = err.Error()
+		} else {
+			st.StatusCode = resp.StatusCode
+			body, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
+			resp.Body.Close()
+			st.Up = resp.StatusCode == v.ExpectedCode
+			if st.Up && v.ExpectedText != "" && !strings.Contains(string(body), v.ExpectedText) {
+				st.Up = false
+				st.Error = "ожидаемый текст не найден"
+			}
+			if !st.Up && st.Error == "" {
+				st.Error = fmt.Sprintf("HTTP %d, ожидался %d", resp.StatusCode, v.ExpectedCode)
+			}
+			if resp.TLS != nil && len(resp.TLS.PeerCertificates) > 0 {
+				cert := resp.TLS.PeerCertificates[0]
+				expiry := cert.NotAfter.Unix()
+				days := int(time.Until(cert.NotAfter).Hours() / 24)
+				st.TLSExpiry = &expiry
+				st.TLSDays = &days
+				st.TLSSubject = cert.Subject.CommonName
+				st.TLSIssuer = cert.Issuer.CommonName
+				st.TLSDNSNames = cert.DNSNames
+				host := req.URL.Hostname()
+				if err := cert.VerifyHostname(host); err == nil {
+					st.TLSHostnameValid = true
+				} else {
+					st.TLSHostnameError = err.Error()
+				}
+			}
+		}
+	}
+	st.LatencyMS = float64(time.Since(start).Microseconds()) / 1000
+	return st
+}

services_test.go

@@ -0,0 +1,55 @@
+package main
+
+import (
+	"testing"
+	"time"
+)
+
+func TestServiceCRUD(t *testing.T) {
+	store, err := openStore(":memory:")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer store.Close()
+	v, err := store.SaveService(MonitoredService{Name: "Test", URL: "https://example.test", CheckType: "http", ExpectedCode: 204, TimeoutSeconds: 2, Enabled: true})
+	if err != nil || v.ID == 0 {
+		t.Fatalf("save: %+v %v", v, err)
+	}
+	items, err := store.Services()
+	if err != nil || len(items) != 1 {
+		t.Fatalf("list: %+v %v", items, err)
+	}
+	if err := store.DeleteService(v.ID); err != nil {
+		t.Fatal(err)
+	}
+}
+
+func TestValidateServiceAllowsLocalAddress(t *testing.T) {
+	v := MonitoredService{Name: "Host", LocalAddress: "192.168.0.12", CheckType: "ping", TimeoutSeconds: 5}
+	if err := validateService(v); err != nil {
+		t.Fatal(err)
+	}
+}
+
+func TestValidateServiceChecksBothHTTPAddresses(t *testing.T) {
+	v := MonitoredService{Name: "Gitea", URL: "https://git.example.test", LocalAddress: "192.168.0.6:3000", CheckType: "http", TimeoutSeconds: 5}
+	if err := validateService(v); err == nil {
+		t.Fatal("expected invalid local HTTP address")
+	}
+	v.LocalAddress = "https://192.168.0.6:3000"
+	if err := validateService(v); err != nil {
+		t.Fatal(err)
+	}
+}
+
+func TestHTTPServiceTransportHasBoundedIdlePool(t *testing.T) {
+	if serviceHTTPClient.Transport != serviceHTTPTransport {
+		t.Fatal("проверки сервисов должны использовать общий HTTP transport")
+	}
+	if serviceHTTPTransport.IdleConnTimeout < time.Second {
+		t.Fatal("idle-соединения должны автоматически закрываться")
+	}
+	if serviceHTTPTransport.MaxIdleConnsPerHost < 1 || serviceHTTPTransport.MaxIdleConnsPerHost > 4 {
+		t.Fatalf("небезопасный размер пула на хост: %d", serviceHTTPTransport.MaxIdleConnsPerHost)
+	}
+}

store.go

@@ -6,6 +6,8 @@ import (
 	"fmt"
 	"os"
 	"path/filepath"
+	"strings"
+	"sync"
 	"time"
 
 	_ "modernc.org/sqlite"
@@ -29,6 +31,17 @@ type AlertThresholds struct {
 	NVMeTempWarning   float64 `json:"nvmeTempWarning"`
 	UPSChargeCritical float64 `json:"upsChargeCritical"`
 	BackupMaxAgeHours float64 `json:"backupMaxAgeHours"`
+	DiskHDDUtil       float64 `json:"diskHddUtil"`
+	DiskHDDLatency    float64 `json:"diskHddLatency"`
+	DiskHDDQueue      float64 `json:"diskHddQueue"`
+	DiskSSDUtil       float64 `json:"diskSsdUtil"`
+	DiskSSDLatency    float64 `json:"diskSsdLatency"`
+	DiskSSDQueue      float64 `json:"diskSsdQueue"`
+	DiskNVMeUtil      float64 `json:"diskNvmeUtil"`
+	DiskNVMeLatency   float64 `json:"diskNvmeLatency"`
+	DiskNVMeQueue     float64 `json:"diskNvmeQueue"`
+	DiskIODuration    float64 `json:"diskIoDuration"`
+	DiskIOExcluded    string  `json:"diskIoExcluded"`
 }
 
 func defaultThresholds() AlertThresholds {
@@ -38,6 +51,7 @@ func defaultThresholds() AlertThresholds {
 		StorageWarning: 85, StorageCritical: 95, ZFSWarning: 80, ZFSCritical: 90,
 		DiskTempWarning: 60, NVMeTempWarning: 75, UPSChargeCritical: 30,
 		BackupMaxAgeHours: 48,
+		DiskHDDUtil:       90, DiskHDDLatency: 50, DiskHDDQueue: 4, DiskSSDUtil: 90, DiskSSDLatency: 20, DiskSSDQueue: 4, DiskNVMeUtil: 95, DiskNVMeLatency: 10, DiskNVMeQueue: 8, DiskIODuration: 60,
 	}
 }
 
@@ -57,7 +71,28 @@ type HistoryPoint struct {
 	GuestsMemory    float64  `json:"guestsMemory"`
 }
 
-type Store struct{ db *sql.DB }
+type Store struct {
+	db               *sql.DB
+	path             string
+	healthMu         sync.RWMutex
+	lastHistoryAt    time.Time
+	lastHistoryError string
+}
+
+type EmailSettings struct {
+	Enabled         bool     `json:"enabled"`
+	Host            string   `json:"host"`
+	Port            int      `json:"port"`
+	Username        string   `json:"username"`
+	Password        string   `json:"password,omitempty"`
+	HasPassword     bool     `json:"hasPassword"`
+	SkipTLSVerify   bool     `json:"skipTlsVerify"`
+	From            string   `json:"from"`
+	To              string   `json:"to"`
+	IntervalMinutes int      `json:"intervalMinutes"`
+	Severities      []string `json:"severities"`
+	Sources         []string `json:"sources"`
+}
 
 func openStore(path string) (*Store, error) {
 	if path != ":memory:" {
@@ -71,6 +106,7 @@ func openStore(path string) (*Store, error) {
 	}
 	db.SetMaxOpenConns(1)
 	statements := []string{
+		trafficSchema,
 		`PRAGMA journal_mode=WAL`,
 		`PRAGMA busy_timeout=5000`,
 		`CREATE TABLE IF NOT EXISTS settings (key TEXT PRIMARY KEY, value TEXT NOT NULL)`,
@@ -81,6 +117,33 @@ func openStore(path string) (*Store, error) {
 			zfs_usage REAL NOT NULL, guests_cpu REAL NOT NULL, guests_memory REAL NOT NULL
 		)`,
 		`CREATE INDEX IF NOT EXISTS history_ts_idx ON history(ts)`,
+		`CREATE TABLE IF NOT EXISTS alert_events (
+			id INTEGER PRIMARY KEY AUTOINCREMENT, alert_key TEXT NOT NULL, severity TEXT NOT NULL,
+			source TEXT NOT NULL, title TEXT NOT NULL, message TEXT NOT NULL,
+			first_seen INTEGER NOT NULL, last_seen INTEGER NOT NULL, resolved_at INTEGER,
+			acknowledged_at INTEGER, email_sent_at INTEGER
+		)`,
+		`CREATE INDEX IF NOT EXISTS alert_events_active_idx ON alert_events(alert_key,resolved_at)`,
+		`CREATE TABLE IF NOT EXISTS services (
+			id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT NOT NULL, url TEXT NOT NULL,
+			local_address TEXT NOT NULL DEFAULT '', check_type TEXT NOT NULL DEFAULT 'http',
+			expected_code INTEGER NOT NULL DEFAULT 200, expected_text TEXT NOT NULL DEFAULT '',
+			timeout_seconds INTEGER NOT NULL DEFAULT 5, enabled INTEGER NOT NULL DEFAULT 1
+		)`,
+		`CREATE TABLE IF NOT EXISTS service_checks (
+			id INTEGER PRIMARY KEY AUTOINCREMENT, service_id INTEGER NOT NULL, ts INTEGER NOT NULL,
+			up INTEGER NOT NULL, latency_ms REAL NOT NULL, status_code INTEGER NOT NULL DEFAULT 0,
+			error TEXT NOT NULL DEFAULT '', tls_expiry INTEGER,
+			FOREIGN KEY(service_id) REFERENCES services(id) ON DELETE CASCADE
+		)`,
+		`CREATE INDEX IF NOT EXISTS service_checks_service_ts_idx ON service_checks(service_id,ts)`,
+		`CREATE TABLE IF NOT EXISTS entity_history (kind TEXT NOT NULL,name TEXT NOT NULL,ts INTEGER NOT NULL,value1 REAL,value2 REAL,PRIMARY KEY(kind,name,ts))`,
+		`CREATE INDEX IF NOT EXISTS entity_history_kind_ts_idx ON entity_history(kind,ts)`,
+		`CREATE TABLE IF NOT EXISTS agent_enrollment_tokens (token_hash TEXT PRIMARY KEY, created_at INTEGER NOT NULL, expires_at INTEGER NOT NULL, used_at INTEGER)`,
+		`CREATE TABLE IF NOT EXISTS agents (id TEXT PRIMARY KEY, name TEXT NOT NULL, vmid INTEGER NOT NULL DEFAULT 0, hostname TEXT NOT NULL DEFAULT '', machine_id TEXT NOT NULL DEFAULT '', secret_hash TEXT NOT NULL, enrolled_at INTEGER NOT NULL, last_seen INTEGER NOT NULL, version TEXT NOT NULL DEFAULT '', report_json TEXT NOT NULL DEFAULT '{}', remote_address TEXT NOT NULL DEFAULT '')`,
+		`CREATE TABLE IF NOT EXISTS agent_actions (id INTEGER PRIMARY KEY AUTOINCREMENT, agent_id TEXT NOT NULL, action TEXT NOT NULL, target TEXT NOT NULL DEFAULT '', requested_at INTEGER NOT NULL, started_at INTEGER, completed_at INTEGER, status TEXT NOT NULL DEFAULT 'queued', result TEXT NOT NULL DEFAULT '', FOREIGN KEY(agent_id) REFERENCES agents(id) ON DELETE CASCADE)`,
+		`CREATE INDEX IF NOT EXISTS agent_actions_agent_status_idx ON agent_actions(agent_id,status)`,
+		`CREATE TABLE IF NOT EXISTS service_links (service_id INTEGER PRIMARY KEY, agent_id TEXT NOT NULL DEFAULT '', vmid INTEGER NOT NULL DEFAULT 0, component_type TEXT NOT NULL DEFAULT '', component_name TEXT NOT NULL DEFAULT '', port INTEGER NOT NULL DEFAULT 0, FOREIGN KEY(service_id) REFERENCES services(id) ON DELETE CASCADE)`,
 	}
 	for _, statement := range statements {
 		if _, err := db.Exec(statement); err != nil {
@@ -88,7 +151,68 @@ func openStore(path string) (*Store, error) {
 			return nil, err
 		}
 	}
-	return &Store{db: db}, nil
+	// Older databases get this column without requiring a separate migration tool.
+	_, _ = db.Exec(`ALTER TABLE alert_events ADD COLUMN recovery_sent_at INTEGER`)
+	_, _ = db.Exec(`ALTER TABLE alert_events ADD COLUMN planned INTEGER NOT NULL DEFAULT 0`)
+	_, _ = db.Exec(`ALTER TABLE alert_events ADD COLUMN maintenance_reason TEXT NOT NULL DEFAULT ''`)
+	_, _ = db.Exec(`DELETE FROM alert_events WHERE alert_key='monitor-slow'`)
+	_, _ = db.Exec(`ALTER TABLE service_checks ADD COLUMN endpoint TEXT NOT NULL DEFAULT 'public'`)
+	_, _ = db.Exec(`PRAGMA foreign_keys=ON`)
+	return &Store{db: db, path: path}, nil
+}
+
+type EntityHistoryPoint struct {
+	Timestamp int64    `json:"timestamp"`
+	Value1    *float64 `json:"value1,omitempty"`
+	Value2    *float64 `json:"value2,omitempty"`
+}
+type EntityHistorySeries struct {
+	Name   string               `json:"name"`
+	Points []EntityHistoryPoint `json:"points"`
+}
+
+func (s *Store) AddEntityHistory(kind, name string, ts int64, v1, v2 *float64) {
+	_, _ = s.db.Exec(`INSERT OR REPLACE INTO entity_history(kind,name,ts,value1,value2) VALUES(?,?,?,?,?)`, kind, name, ts, v1, v2)
+}
+func (s *Store) EntityHistory(kind string, hours int) ([]EntityHistorySeries, error) {
+	if hours < 1 {
+		hours = 24
+	}
+	if hours > 168 {
+		hours = 168
+	}
+	rows, err := s.db.Query(`SELECT name,ts,value1,value2 FROM entity_history WHERE kind=? AND ts>=? ORDER BY name,ts`, kind, time.Now().Add(-time.Duration(hours)*time.Hour).Unix())
+	if err != nil {
+		return nil, err
+	}
+	defer rows.Close()
+	var out []EntityHistorySeries
+	index := map[string]int{}
+	for rows.Next() {
+		var name string
+		var p EntityHistoryPoint
+		var v1, v2 sql.NullFloat64
+		if err = rows.Scan(&name, &p.Timestamp, &v1, &v2); err != nil {
+			return nil, err
+		}
+		if v1.Valid {
+			p.Value1 = &v1.Float64
+		}
+		if v2.Valid {
+			p.Value2 = &v2.Float64
+		}
+		i, ok := index[name]
+		if !ok {
+			i = len(out)
+			index[name] = i
+			out = append(out, EntityHistorySeries{Name: name})
+		}
+		out[i].Points = append(out[i].Points, p)
+	}
+	return out, rows.Err()
+}
+func (s *Store) PruneEntityHistory() {
+	_, _ = s.db.Exec(`DELETE FROM entity_history WHERE ts<?`, time.Now().Add(-historyRetention).Unix())
 }
 
 func (s *Store) Close() error { return s.db.Close() }
@@ -114,6 +238,55 @@ func (s *Store) SaveThresholds(value AlertThresholds) error {
 	return err
 }
 
+func defaultEmailSettings() EmailSettings {
+	return EmailSettings{Port: 587, IntervalMinutes: 60, Severities: []string{"critical", "warning"}, Sources: []string{"CPU", "Память", "Storage", "Диски", "ZFS", "UPS", "VM/LXC", "Proxmox", "Dashboard", "Сервисы", "Система", "Сеть", "Агенты", "Docker"}}
+}
+
+func (s *Store) EmailSettings(includePassword bool) EmailSettings {
+	settings := defaultEmailSettings()
+	var value string
+	if s.db.QueryRow(`SELECT value FROM settings WHERE key='email_notifications'`).Scan(&value) == nil {
+		_ = json.Unmarshal([]byte(value), &settings)
+	}
+	settings.HasPassword = settings.Password != ""
+	if !includePassword {
+		settings.Password = ""
+	}
+	return settings
+}
+
+func (s *Store) SaveEmailSettings(settings EmailSettings) error {
+	if settings.Port < 1 || settings.Port > 65535 || settings.IntervalMinutes < 5 || settings.IntervalMinutes > 10080 {
+		return fmt.Errorf("проверьте порт и интервал (от 5 минут до 7 дней)")
+	}
+	if settings.Enabled && (settings.Host == "" || settings.From == "" || settings.To == "") {
+		return fmt.Errorf("для отправки нужны SMTP-сервер, отправитель и получатель")
+	}
+	if strings.ContainsAny(settings.Host+settings.From+settings.To+settings.Username, "\r\n") {
+		return fmt.Errorf("поля почты содержат недопустимые символы")
+	}
+	if settings.Password == "" {
+		settings.Password = s.EmailSettings(true).Password
+	}
+	settings.HasPassword = settings.Password != ""
+	data, _ := json.Marshal(settings)
+	_, err := s.db.Exec(`INSERT INTO settings(key,value) VALUES('email_notifications',?) ON CONFLICT(key) DO UPDATE SET value=excluded.value`, string(data))
+	return err
+}
+
+func (s *Store) LastEmailAt() time.Time {
+	var value string
+	if s.db.QueryRow(`SELECT value FROM settings WHERE key='email_last_sent'`).Scan(&value) != nil {
+		return time.Time{}
+	}
+	result, _ := time.Parse(time.RFC3339, value)
+	return result
+}
+
+func (s *Store) SetLastEmailAt(value time.Time) {
+	_, _ = s.db.Exec(`INSERT INTO settings(key,value) VALUES('email_last_sent',?) ON CONFLICT(key) DO UPDATE SET value=excluded.value`, value.Format(time.RFC3339))
+}
+
 func validateThresholds(t AlertThresholds) error {
 	percentages := []float64{t.CPUWarning, t.CPUCritical, t.MemoryWarning, t.MemoryCritical, t.SwapWarning, t.StorageWarning, t.StorageCritical, t.ZFSWarning, t.ZFSCritical, t.UPSChargeCritical}
 	for _, value := range percentages {
@@ -133,6 +306,19 @@ func validateThresholds(t AlertThresholds) error {
 	if t.BackupMaxAgeHours < 1 || t.BackupMaxAgeHours > 24*365 {
 		return fmt.Errorf("возраст backup должен быть от 1 до 8760 часов")
 	}
+	for _, value := range []float64{t.DiskHDDUtil, t.DiskSSDUtil, t.DiskNVMeUtil} {
+		if value < 1 || value > 100 {
+			return fmt.Errorf("пороги utilization должны быть от 1 до 100")
+		}
+	}
+	for _, value := range []float64{t.DiskHDDLatency, t.DiskSSDLatency, t.DiskNVMeLatency, t.DiskHDDQueue, t.DiskSSDQueue, t.DiskNVMeQueue} {
+		if value < 0 {
+			return fmt.Errorf("пороги latency и очереди не могут быть отрицательными")
+		}
+	}
+	if t.DiskIODuration < 5 || t.DiskIODuration > 3600 {
+		return fmt.Errorf("длительность I/O-проблемы должна быть от 5 до 3600 секунд")
+	}
 	return nil
 }
 
@@ -140,9 +326,45 @@ func (s *Store) AddHistory(point HistoryPoint) error {
 	_, err := s.db.Exec(`INSERT OR REPLACE INTO history(ts,cpu_usage,cpu_temp,memory_usage,swap_usage,root_usage,network_rx,network_tx,ups_charge,ups_load,zfs_usage,guests_cpu,guests_memory) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?)`,
 		point.Timestamp, point.CPUUsage, point.CPUTemperature, point.MemoryUsage, point.SwapUsage, point.RootUsage, point.NetworkReceive, point.NetworkTransmit, point.UPSCharge, point.UPSLoad, point.ZFSUsage, point.GuestsCPU, point.GuestsMemory)
 	if err != nil {
+		s.setHistoryHealth(time.Time{}, err)
 		return err
 	}
 	_, err = s.db.Exec(`DELETE FROM history WHERE ts < ?`, time.Now().Add(-historyRetention).Unix())
+	s.setHistoryHealth(time.Now(), err)
+	return err
+}
+
+func (s *Store) setHistoryHealth(at time.Time, err error) {
+	s.healthMu.Lock()
+	defer s.healthMu.Unlock()
+	if !at.IsZero() {
+		s.lastHistoryAt = at
+	}
+	if err != nil {
+		s.lastHistoryError = err.Error()
+	} else {
+		s.lastHistoryError = ""
+	}
+}
+func (s *Store) historyHealth() (time.Time, string) {
+	s.healthMu.RLock()
+	defer s.healthMu.RUnlock()
+	return s.lastHistoryAt, s.lastHistoryError
+}
+
+func (s *Store) loadJSONSetting(key string, target any) bool {
+	var value string
+	if s.db.QueryRow(`SELECT value FROM settings WHERE key=?`, key).Scan(&value) != nil {
+		return false
+	}
+	return json.Unmarshal([]byte(value), target) == nil
+}
+func (s *Store) saveJSONSetting(key string, value any) error {
+	data, err := json.Marshal(value)
+	if err != nil {
+		return err
+	}
+	_, err = s.db.Exec(`INSERT INTO settings(key,value) VALUES(?,?) ON CONFLICT(key) DO UPDATE SET value=excluded.value`, key, string(data))
 	return err
 }
 

store_test.go

@@ -29,3 +29,17 @@ func TestStoreSettingsAndHistory(t *testing.T) {
 		t.Fatalf("история не сохранилась: points=%+v err=%v", points, err)
 	}
 }
+
+func TestEntityHistory(t *testing.T) {
+	store, err := openStore(":memory:")
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer store.Close()
+	value := 42.5
+	store.AddEntityHistory("zfs", "rpool", time.Now().Unix(), &value, nil)
+	series, err := store.EntityHistory("zfs", 24)
+	if err != nil || len(series) != 1 || len(series[0].Points) != 1 || series[0].Name != "rpool" {
+		t.Fatalf("unexpected entity history: %+v %v", series, err)
+	}
+}

system_health.go

@@ -0,0 +1,116 @@
+package main
+
+import (
+	"os/exec"
+	"regexp"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+)
+
+type SystemIssue struct {
+	Time     int64  `json:"time"`
+	Unit     string `json:"unit"`
+	Priority string `json:"priority"`
+	Message  string `json:"message"`
+	Count    int    `json:"count"`
+}
+type SystemHealthMetrics struct {
+	Available      bool            `json:"available"`
+	FailedUnits    int             `json:"failedUnits"`
+	KernelErrors   int             `json:"kernelErrors"`
+	SystemErrors   int             `json:"systemErrors"`
+	Issues         []SystemIssue   `json:"issues"`
+	FailedServices []FailedService `json:"failedServices"`
+	CheckedAt      int64           `json:"checkedAt"`
+}
+type FailedService struct {
+	Unit        string `json:"unit"`
+	Load        string `json:"load"`
+	Active      string `json:"active"`
+	State       string `json:"state"`
+	Description string `json:"description"`
+}
+type systemHealthCollector struct {
+	mu    sync.RWMutex
+	value SystemHealthMetrics
+}
+
+func newSystemHealthCollector() *systemHealthCollector {
+	c := &systemHealthCollector{}
+	go c.run()
+	return c
+}
+func (c *systemHealthCollector) collect() SystemHealthMetrics {
+	c.mu.RLock()
+	defer c.mu.RUnlock()
+	return c.value
+}
+func (c *systemHealthCollector) run() {
+	c.refresh()
+	t := time.NewTicker(time.Minute)
+	defer t.Stop()
+	for range t.C {
+		c.refresh()
+	}
+}
+func (c *systemHealthCollector) refresh() {
+	m := SystemHealthMetrics{CheckedAt: time.Now().Unix()}
+	if _, err := exec.LookPath("journalctl"); err != nil {
+		c.mu.Lock()
+		c.value = m
+		c.mu.Unlock()
+		return
+	}
+	m.Available = true
+	if out, err := exec.Command("systemctl", "--failed", "--no-legend", "--plain").Output(); err == nil {
+		for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
+			fields := strings.Fields(line)
+			if len(fields) < 4 {
+				continue
+			}
+			unit := strings.TrimPrefix(fields[0], "●")
+			offset := 1
+			if unit == "" && len(fields) > 4 {
+				unit = fields[1]
+				offset = 2
+			}
+			description := ""
+			if len(fields) > offset+3 {
+				description = strings.Join(fields[offset+3:], " ")
+			}
+			m.FailedServices = append(m.FailedServices, FailedService{Unit: unit, Load: fields[offset], Active: fields[offset+1], State: fields[offset+2], Description: description})
+			m.FailedUnits++
+		}
+	}
+	out, _ := exec.Command("journalctl", "--since", "24 hours ago", "-p", "0..3", "--no-pager", "-o", "short-unix", "-n", "100").Output()
+	pidPattern := regexp.MustCompile(`\[[0-9]+\]`)
+	grouped := map[string]int{}
+	for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
+		f := strings.Fields(line)
+		if len(f) < 2 {
+			continue
+		}
+		raw := strings.TrimSuffix(f[0], ":")
+		sec, _ := strconv.ParseFloat(raw, 64)
+		unit := strings.TrimSuffix(f[1], ":")
+		message := strings.Join(f[2:], " ")
+		normalized := pidPattern.ReplaceAllString(message, "[*]")
+		key := unit + "\x00" + normalized
+		if index, ok := grouped[key]; ok {
+			m.Issues[index].Count++
+			m.Issues[index].Time = int64(sec)
+		} else {
+			grouped[key] = len(m.Issues)
+			m.Issues = append(m.Issues, SystemIssue{Time: int64(sec), Unit: unit, Priority: "error", Message: normalized, Count: 1})
+		}
+		m.SystemErrors++
+		if strings.Contains(strings.ToLower(unit), "kernel") {
+			m.KernelErrors++
+		}
+	}
+	c.mu.Lock()
+	c.value = m
+	c.mu.Unlock()
+}

traffic.go

@@ -0,0 +1,300 @@
+package main
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net"
+	"net/http"
+	"strings"
+	"time"
+)
+
+type TrafficEvent struct {
+	ID       string `json:"id"`
+	Sequence int64  `json:"sequence,omitempty"`
+	Time     int64  `json:"time"`
+	Kind     string `json:"kind"`
+	IP       string `json:"ip"`
+	Domain   string `json:"domain,omitempty"`
+	Method   string `json:"method,omitempty"`
+	Path     string `json:"path,omitempty"`
+	Status   int    `json:"status,omitempty"`
+	Reason   string `json:"reason,omitempty"`
+	Country  string `json:"country,omitempty"`
+	Agent    string `json:"agent,omitempty"`
+}
+
+type TrafficBatch struct {
+	Events          []TrafficEvent `json:"events"`
+	Files           int            `json:"files"`
+	BacklogBytes    int64          `json:"backlogBytes"`
+	CrowdSecLimited bool           `json:"crowdsecLimited"`
+	ParseErrors     uint64         `json:"parseErrors"`
+	LogError        string         `json:"logError,omitempty"`
+	CrowdSecError   string         `json:"crowdsecError,omitempty"`
+	CrowdSecAt      int64          `json:"crowdsecAt"`
+	StartedAt       int64          `json:"startedAt"`
+}
+
+type TrafficSource struct {
+	Agent    string       `json:"agent"`
+	Name     string       `json:"name"`
+	LastSeen int64        `json:"lastSeen"`
+	Status   TrafficBatch `json:"status"`
+}
+
+type TrafficCount struct {
+	Name  string `json:"name"`
+	Count int64  `json:"count"`
+}
+
+type TrafficView struct {
+	Events    []TrafficEvent  `json:"events"`
+	Sources   []TrafficSource `json:"sources"`
+	Countries []TrafficCount  `json:"countries"`
+	Domains   []TrafficCount  `json:"domains"`
+	IPs       []TrafficCount  `json:"ips"`
+	Requests  int64           `json:"requests"`
+	UniqueIPs int64           `json:"uniqueIps"`
+	Alerts    int64           `json:"alerts"`
+	Decisions int64           `json:"decisions"`
+	QPS       float64         `json:"qps"`
+	Series    []TrafficCount  `json:"series"`
+	GeoReady  bool            `json:"geoReady"`
+	GeoDate   string          `json:"geoDate"`
+	Now       int64           `json:"now"`
+	Settings  TrafficSettings `json:"settings"`
+	Limited   bool            `json:"limited"`
+}
+
+const trafficSchema = `CREATE TABLE IF NOT EXISTS traffic_events (
+ seq INTEGER PRIMARY KEY AUTOINCREMENT, agent TEXT NOT NULL, event_id TEXT NOT NULL,
+ ts INTEGER NOT NULL, kind TEXT NOT NULL, ip TEXT NOT NULL, domain TEXT NOT NULL,
+ method TEXT NOT NULL, path TEXT NOT NULL, status INTEGER NOT NULL, reason TEXT NOT NULL,
+ country TEXT NOT NULL, UNIQUE(agent,event_id));
+ CREATE INDEX IF NOT EXISTS traffic_ts ON traffic_events(ts);
+ CREATE INDEX IF NOT EXISTS traffic_filter ON traffic_events(domain,kind,ts);
+ CREATE TABLE IF NOT EXISTS network_sources(agent TEXT PRIMARY KEY, snapshot TEXT NOT NULL);
+ CREATE TABLE IF NOT EXISTS traffic_sources(agent TEXT PRIMARY KEY, last_seen INTEGER NOT NULL, status_json TEXT NOT NULL);`
+
+func (s *Store) SaveTraffic(agent string, batch TrafficBatch) error {
+	if len(batch.Events) > 500 {
+		return errors.New("too many traffic events")
+	}
+	now := time.Now().Unix()
+	tx, err := s.db.Begin()
+	if err != nil {
+		return err
+	}
+	defer tx.Rollback()
+	stmt, err := tx.Prepare(`INSERT OR IGNORE INTO traffic_events(agent,event_id,ts,kind,ip,domain,method,path,status,reason,country) VALUES(?,?,?,?,?,?,?,?,?,?,?)`)
+	if err != nil {
+		return err
+	}
+	defer stmt.Close()
+	for _, e := range batch.Events {
+		if e.Kind != "request" && e.Kind != "alert" && e.Kind != "decision" {
+			return errors.New("invalid traffic kind")
+		}
+		if net.ParseIP(e.IP) == nil || len(e.ID) == 0 || len(e.ID) > 160 {
+			return errors.New("invalid traffic event")
+		}
+		if e.Time < now-86400 || e.Time > now+300 {
+			continue
+		}
+		country := lookupTrafficCountry(e.IP)
+		_, err = stmt.Exec(agent, e.ID, e.Time, e.Kind, e.IP, trafficText(e.Domain, 253), trafficText(e.Method, 16), trafficPath(e.Path), e.Status, trafficText(e.Reason, 256), country)
+		if err != nil {
+			return err
+		}
+	}
+	batch.Events = nil
+	raw, err := json.Marshal(batch)
+	if err != nil {
+		return err
+	}
+	if _, err = tx.Exec(`INSERT INTO traffic_sources(agent,last_seen,status_json) VALUES(?,?,?) ON CONFLICT(agent) DO UPDATE SET last_seen=excluded.last_seen,status_json=excluded.status_json`, agent, now, string(raw)); err != nil {
+		return err
+	}
+	// Bound both age and total rows, including a high-traffic installation.
+	if _, err = tx.Exec(`DELETE FROM traffic_events WHERE ts<? OR seq <= COALESCE((SELECT seq FROM traffic_events ORDER BY seq DESC LIMIT 1 OFFSET 200000),0)`, now-86400); err != nil {
+		return err
+	}
+	return tx.Commit()
+}
+
+func trafficText(s string, max int) string {
+	s = strings.Map(func(r rune) rune {
+		if r < 32 || r == 127 {
+			return -1
+		}
+		return r
+	}, s)
+	r := []rune(s)
+	if len(r) > max {
+		return string(r[:max])
+	}
+	return s
+}
+func trafficPath(s string) string {
+	s = strings.SplitN(s, "?", 2)[0]
+	s = strings.SplitN(s, "#", 2)[0]
+	return trafficText(s, 256)
+}
+
+func (s *Store) TrafficView(domain, kind string, external bool) (TrafficView, error) {
+	now := time.Now().Unix()
+	v := TrafficView{Events: []TrafficEvent{}, Sources: []TrafficSource{}, Countries: []TrafficCount{}, Domains: []TrafficCount{}, IPs: []TrafficCount{}, Series: []TrafficCount{}, Now: now, GeoReady: trafficGeoReady(), GeoDate: trafficGeoDate}
+	v.Settings = s.TrafficSettings()
+	where := ` WHERE ts>=? AND agent=?`
+	args := []any{now - 86400, v.Settings.AgentID}
+	if external {
+		where += ` AND country!='LAN'`
+	}
+	if domain != "" {
+		where += ` AND domain=?`
+		args = append(args, domain)
+	}
+	if kind != "" {
+		where += ` AND kind=?`
+		args = append(args, kind)
+	}
+	err := s.db.QueryRow(`SELECT COALESCE(SUM(kind='request'),0),COUNT(DISTINCT CASE WHEN kind='request' THEN ip END),COALESCE(SUM(kind='alert'),0),COALESCE(SUM(kind='decision'),0),COALESCE(SUM(kind='request' AND ts>=?),0)/60.0 FROM traffic_events`+where, append([]any{now - 60}, args...)...).Scan(&v.Requests, &v.UniqueIPs, &v.Alerts, &v.Decisions, &v.QPS)
+	if err != nil {
+		return v, err
+	}
+	rows, err := s.db.Query(`SELECT seq,event_id,ts,kind,ip,domain,method,path,status,reason,country,agent FROM traffic_events`+where+` ORDER BY ts DESC,seq DESC LIMIT 100`, args...)
+	if err != nil {
+		return v, err
+	}
+	for rows.Next() {
+		var e TrafficEvent
+		if err = rows.Scan(&e.Sequence, &e.ID, &e.Time, &e.Kind, &e.IP, &e.Domain, &e.Method, &e.Path, &e.Status, &e.Reason, &e.Country, &e.Agent); err != nil {
+			rows.Close()
+			return v, err
+		}
+		v.Events = append(v.Events, e)
+	}
+	err = rows.Err()
+	rows.Close()
+	if err != nil {
+		return v, err
+	}
+	for _, q := range []struct {
+		column string
+		out    *[]TrafficCount
+	}{{"country", &v.Countries}, {"domain", &v.Domains}, {"ip", &v.IPs}} {
+		r, e := s.db.Query(`SELECT `+q.column+`,COUNT(*) FROM traffic_events`+where+` GROUP BY `+q.column+` ORDER BY COUNT(*) DESC LIMIT 50`, args...)
+		if e != nil {
+			return v, e
+		}
+		for r.Next() {
+			var c TrafficCount
+			if e = r.Scan(&c.Name, &c.Count); e != nil {
+				r.Close()
+				return v, e
+			}
+			*q.out = append(*q.out, c)
+		}
+		e = r.Err()
+		r.Close()
+		if e != nil {
+			return v, e
+		}
+	}
+	r, err := s.db.Query(`SELECT CAST((ts/60)*60 AS TEXT),COUNT(*) FROM traffic_events`+where+` AND kind='request' AND ts>=? GROUP BY ts/60 ORDER BY ts/60`, append(args, now-3600)...)
+	if err != nil {
+		return v, err
+	}
+	for r.Next() {
+		var c TrafficCount
+		if err = r.Scan(&c.Name, &c.Count); err != nil {
+			r.Close()
+			return v, err
+		}
+		v.Series = append(v.Series, c)
+	}
+	err = r.Err()
+	r.Close()
+	if err != nil {
+		return v, err
+	}
+	r, err = s.db.Query(`SELECT t.agent,COALESCE(a.name,t.agent),t.last_seen,t.status_json FROM traffic_sources t LEFT JOIN agents a ON a.id=t.agent WHERE t.agent=? ORDER BY t.last_seen DESC`, v.Settings.AgentID)
+	if err != nil {
+		return v, err
+	}
+	for r.Next() {
+		var src TrafficSource
+		var raw string
+		if err = r.Scan(&src.Agent, &src.Name, &src.LastSeen, &raw); err != nil {
+			r.Close()
+			return v, err
+		}
+		json.Unmarshal([]byte(raw), &src.Status)
+		v.Sources = append(v.Sources, src)
+	}
+	err = r.Err()
+	r.Close()
+	var count int64
+	s.db.QueryRow(`SELECT COUNT(*) FROM traffic_events`).Scan(&count)
+	v.Limited = count >= 200000
+	return v, err
+}
+
+func trafficRoutes(mux *http.ServeMux, s *Store) {
+	trafficSettingsRoute(mux, s)
+	networkRoutes(mux, s)
+	mux.HandleFunc("/traffic", func(w http.ResponseWriter, r *http.Request) {
+		if !s.TrafficSettings().Enabled {
+			http.Error(w, "Карта трафика отключена. Включите её в настройках Dashboard.", 404)
+			return
+		}
+		w.Header().Set("Content-Type", "text/html; charset=utf-8")
+		w.Write([]byte(trafficHTML))
+	})
+	mux.HandleFunc("/api/traffic", func(w http.ResponseWriter, r *http.Request) {
+		if r.Method != "GET" {
+			w.WriteHeader(405)
+			return
+		}
+		if !s.TrafficSettings().Enabled {
+			http.Error(w, "Карта отключена", 404)
+			return
+		}
+		view, err := s.TrafficView(r.URL.Query().Get("domain"), r.URL.Query().Get("kind"), r.URL.Query().Get("external") == "1")
+		writeJSON(w, view, err)
+	})
+	mux.HandleFunc("/api/traffic/world", func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "application/json")
+		w.Header().Set("Cache-Control", "public, max-age=86400")
+		w.Write(trafficWorld)
+	})
+	mux.HandleFunc("/api/agent/traffic", func(w http.ResponseWriter, r *http.Request) {
+		if r.Method != "POST" {
+			w.WriteHeader(405)
+			return
+		}
+		id, secret, ok := agentCredentials(r)
+		if !ok || !s.AuthenticateAgent(id, secret) {
+			http.Error(w, "Unauthorized", 401)
+			return
+		}
+		settings := s.TrafficSettings()
+		if !settings.Enabled || settings.AgentID != id {
+			http.Error(w, "Источник отключён", 403)
+			return
+		}
+		var batch TrafficBatch
+		if json.NewDecoder(io.LimitReader(r.Body, 1<<20)).Decode(&batch) != nil {
+			http.Error(w, "Invalid traffic batch", 400)
+			return
+		}
+		if err := s.SaveTraffic(id, batch); err != nil {
+			http.Error(w, fmt.Sprint(err), 500)
+			return
+		}
+		writeJSON(w, map[string]bool{"accepted": true}, nil)
+	})
+}

traffic_geo.go

@@ -0,0 +1,65 @@
+package main
+
+import (
+	"bytes"
+	"compress/gzip"
+	_ "embed"
+	"io"
+	"net"
+	"sync"
+
+	"github.com/oschwald/maxminddb-golang"
+)
+
+//go:embed assets/world.json
+var trafficWorld []byte
+
+//go:embed assets/dbip-country-lite-2026-10.mmdb.gz
+var trafficGeoCompressed []byte
+
+const trafficGeoDate = "2026-10"
+
+var trafficGeo struct {
+	sync.Once
+	db *maxminddb.Reader
+}
+
+func trafficGeoReady() bool {
+	trafficGeo.Do(func() {
+		r, err := gzip.NewReader(bytes.NewReader(trafficGeoCompressed))
+		if err != nil {
+			return
+		}
+		defer r.Close()
+		data, err := io.ReadAll(io.LimitReader(r, 32<<20))
+		if err != nil {
+			return
+		}
+		trafficGeo.db, _ = maxminddb.FromBytes(data)
+	})
+	return trafficGeo.db != nil
+}
+func lookupTrafficCountry(ipText string) string {
+	ip := net.ParseIP(ipText)
+	if ip == nil {
+		return ""
+	}
+	if ip.IsPrivate() || ip.IsLoopback() || ip.IsLinkLocalUnicast() {
+		return "LAN"
+	}
+	if !ip.IsGlobalUnicast() {
+		return ""
+	}
+	if !trafficGeoReady() {
+		return ""
+	}
+	var record struct {
+		Country struct {
+			Code string `maxminddb:"iso_code"`
+		} `maxminddb:"country"`
+	}
+	if trafficGeo.db.Lookup(ip, &record) != nil {
+		return ""
+	}
+	return record.Country.Code
+}

traffic_preview_test.go

@@ -0,0 +1,78 @@
+package main
+
+import (
+	"fmt"
+	"net/http"
+	"os"
+	"strings"
+	"testing"
+	"time"
+)
+
+// Opt-in local visual fixture; never included in production binaries.
+func TestTrafficPreview(t *testing.T) {
+	if os.Getenv("DASHBOARD_TRAFFIC_PREVIEW") != "1" {
+		t.Skip("manual visual preview")
+	}
+	s, a := trafficFixture(t)
+	s.SaveTrafficSettings(TrafficSettings{Enabled: true, NetworkEnabled: true, AgentID: a.ID, ServerCountry: "RU"})
+	mux := http.NewServeMux()
+	trafficRoutes(mux, s)
+	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
+		w.Header().Set("Content-Type", "text/html; charset=utf-8")
+		w.Write([]byte(dashboardHTML))
+	})
+	mux.HandleFunc("/api/agents", func(w http.ResponseWriter, r *http.Request) { items, _ := s.Agents(); writeJSON(w, items, nil) })
+	go func() {
+		i := 0
+		for {
+			i++
+			events := []TrafficEvent{}
+			for j, ip := range []string{"8.8.8.8", "1.1.1.1", "20.194.96.114", "192.168.0.10", "77.83.39.94", "51.158.145.102", "123.56.127.23"} {
+				kind := "request"
+				if j == 4 {
+					kind = "alert"
+				}
+				if j == 5 {
+					kind = "decision"
+				}
+				events = append(events, TrafficEvent{ID: fmt.Sprintf("preview-%d-%d", i, j), Time: time.Now().Unix(), Kind: kind, IP: ip, Domain: "demo.example", Method: "GET", Path: "/demo", Status: 200, Reason: "ДЕМО · crowdsecurity/http-probing"})
+			}
+			flows := []NetworkFlow{}
+			for k := 0; k < 32; k++ {
+				f, _ := parseNetworkFlow(networkNATLine)
+				f.SourcePort = 51000 + k
+				f.ReplyDestinationPort = f.SourcePort
+				f.Source = []string{"8.8.8.8", "1.1.1.1", "20.194.96.114", "192.168.0.10", "51.158.145.102", "2001:4860:4860::8888"}[k%6]
+				f.ReplyDestination = f.Source
+				if k%3 == 0 {
+					f.Protocol = "udp"
+					f.State = "ASSURED"
+					f.DestinationPort = 53
+					f.ReplySourcePort = 53
+				}
+				if k%5 == 0 && f.Protocol == "tcp" {
+					f.State = "SYN_SENT"
+				}
+				flows = append(flows, f)
+			}
+			s.SaveNetwork(a.ID, NetworkSnapshot{At: time.Now().Unix(), Flows: flows, FirewallAt: time.Now().Unix(), Firewall: []FirewallCounter{{Name: "DEMO ip/crowdsec/drop #3", Packets: 12345, Bytes: 654321}}})
+			s.SaveTraffic(a.ID, TrafficBatch{Events: events, Files: 22, StartedAt: time.Now().Add(-time.Hour).Unix(), CrowdSecAt: time.Now().Unix()})
+			time.Sleep(3 * time.Second)
+		}
+	}()
+	handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		if r.URL.Path == "/network" {
+			w.Header().Set("Content-Type", "text/html; charset=utf-8")
+			w.Write([]byte(strings.Replace(networkHTML, "PROXMOX / NETWORK OBSERVATORY", "ДЕМО / ИСКУССТВЕННЫЕ ДАННЫЕ", 1)))
+			return
+		}
+		if r.URL.Path == "/traffic" {
+			w.Header().Set("Content-Type", "text/html; charset=utf-8")
+			w.Write([]byte(strings.Replace(trafficHTML, "PROXMOX / LIVE OBSERVATORY", "ДЕМО / ИСКУССТВЕННЫЕ ДАННЫЕ", 1)))
+			return
+		}
+		mux.ServeHTTP(w, r)
+	})
+	t.Fatal(http.ListenAndServe("127.0.0.1:19105", handler))
+}

traffic_settings.go

@@ -0,0 +1,71 @@
+package main
+
+import (
+	"encoding/json"
+	"errors"
+	"io"
+	"net/http"
+	"strings"
+)
+
+type TrafficSettings struct {
+	NetworkEnabled bool   `json:"networkEnabled"`
+	Enabled        bool   `json:"enabled"`
+	AgentID        string `json:"agentId"`
+	ServerCountry  string `json:"serverCountry"`
+}
+
+func (s *Store) TrafficSettings() TrafficSettings {
+	var c TrafficSettings
+	s.loadJSONSetting("traffic", &c)
+	return c
+}
+func (s *Store) SaveTrafficSettings(c TrafficSettings) (TrafficSettings, error) {
+	c.AgentID = strings.TrimSpace(c.AgentID)
+	c.ServerCountry = strings.ToUpper(strings.TrimSpace(c.ServerCountry))
+	if c.Enabled {
+		var count int
+		if s.db.QueryRow(`SELECT COUNT(*) FROM agents WHERE id=?`, c.AgentID).Scan(&count) != nil || count != 1 {
+			return c, errors.New("Выберите зарегистрированный агент")
+		}
+	}
+	if c.ServerCountry != "" {
+		var countries []struct {
+			Code string `json:"code"`
+		}
+		json.Unmarshal(trafficWorld, &countries)
+		found := false
+		for _, country := range countries {
+			if country.Code == c.ServerCountry {
+				found = true
+				break
+			}
+		}
+		if !found {
+			return c, errors.New("Неизвестная страна сервера")
+		}
+	}
+	return c, s.saveJSONSetting("traffic", c)
+}
+func trafficSettingsRoute(mux *http.ServeMux, s *Store) {
+	mux.HandleFunc("/api/settings/traffic", func(w http.ResponseWriter, r *http.Request) {
+		switch r.Method {
+		case "GET":
+			writeJSON(w, s.TrafficSettings(), nil)
+		case "PUT":
+			var c TrafficSettings
+			if json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&c) != nil {
+				http.Error(w, "Некорректные настройки", 400)
+				return
+			}
+			value, err := s.SaveTrafficSettings(c)
+			if err != nil {
+				http.Error(w, err.Error(), 400)
+				return
+			}
+			writeJSON(w, value, nil)
+		default:
+			w.WriteHeader(405)
+		}
+	})
+}

traffic_test.go

@@ -0,0 +1,217 @@
+package main
+
+import (
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+func trafficFixture(t *testing.T) (*Store, agentConfig) {
+	t.Helper()
+	s, err := openStore(":memory:")
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Cleanup(func() { s.Close() })
+	token, err := s.CreateAgentEnrollmentToken()
+	if err != nil {
+		t.Fatal(err)
+	}
+	a, err := s.EnrollAgent(agentEnrollmentRequest{Token: token, Name: "NPM test", VMID: 102})
+	if err != nil {
+		t.Fatal(err)
+	}
+	return s, a
+}
+func npmTestLine(ip, path string) string {
+	return fmt.Sprintf("[%s] - - 301 - GET https example.test \"%s\" [Client %s] [Length 166] [Gzip -] [Sent-to 192.168.0.9] \"-\" \"-\"\n", time.Now().UTC().Format("02/Jan/2006:15:04:05 -0700"), path, ip)
+}
+func TestNPMTrafficFormats(t *testing.T) {
+	for _, status := range []string{"- 200 200", "- - 301", "- 502 502", "- - 401"} {
+		line := strings.Replace(npmTestLine("2001:4860:4860::8888", "/login?token=secret#fragment"), "- - 301", status, 1)
+		e, ok := parseNPMTraffic(line)
+		if !ok || e.Kind != "request" || e.Path != "/login" || e.Domain != "example.test" || e.IP != "2001:4860:4860::8888" {
+			t.Fatalf("bad parse: %+v %v", e, ok)
+		}
+	}
+	if _, ok := parseNPMTraffic("broken"); ok {
+		t.Fatal("accepted invalid log")
+	}
+}
+func TestCrowdTrafficDoesNotInventBlockedRequests(t *testing.T) {
+	raw := `[{"id":12,"uuid":"alert-uuid","created_at":"2026-10-09T16:16:43Z","source":{"ip":"20.194.96.114"},"scenario":"crowdsecurity/http-admin-interface-probing","events":[{"meta":[{"key":"target_fqdn","value":"example.test"}]}],"decisions":[{"id":42,"value":"20.194.96.114","type":"ban","simulated":false},{"id":43,"value":"1.1.1.1","type":"ban","simulated":true}]}]`
+	events, err := parseCrowdTraffic([]byte(raw))
+	if err != nil || len(events) != 2 {
+		t.Fatalf("%+v %v", events, err)
+	}
+	if events[0].Kind != "alert" || events[1].Kind != "decision" || events[1].Domain != "example.test" {
+		t.Fatalf("bad categories %+v", events)
+	}
+	simulated := strings.Replace(raw, `"id":12`, `"simulated":true,"id":12`, 1)
+	events, err = parseCrowdTraffic([]byte(simulated))
+	if err != nil || len(events) != 0 {
+		t.Fatal("simulation counted as live")
+	}
+}
+func TestTrafficLogCheckpointRotationAndPartialLines(t *testing.T) {
+	dir := t.TempDir()
+	path := filepath.Join(dir, "proxy-host-1_access.log")
+	os.WriteFile(path, []byte(npmTestLine("1.1.1.1", "/old")), 0600)
+	state := trafficAgentState{Cursors: map[string]trafficCursor{}}
+	if _, err := state.readLogs(dir); err != nil {
+		t.Fatal(err)
+	}
+	if len(state.Pending) != 0 {
+		t.Fatal("imported old logs on first activation")
+	}
+	appendFile := func(path, s string) {
+		f, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, 0600)
+		if err != nil {
+			t.Fatal(err)
+		}
+		f.WriteString(s)
+		f.Close()
+	}
+	line := npmTestLine("8.8.8.8", "/new")
+	appendFile(path, line[:len(line)/2])
+	state.readLogs(dir)
+	if len(state.Pending) != 0 {
+		t.Fatal("partial line emitted")
+	}
+	appendFile(path, line[len(line)/2:])
+	state.readLogs(dir)
+	if len(state.Pending) != 1 {
+		t.Fatal("complete line missing")
+	}
+	id := state.Pending[0].ID
+	state.Pending = nil
+	checkpoint := filepath.Join(dir, "state.json")
+	if err := saveTrafficAgentState(checkpoint, state); err != nil {
+		t.Fatal(err)
+	}
+	raw, _ := os.ReadFile(checkpoint)
+	var restored trafficAgentState
+	json.Unmarshal(raw, &restored)
+	os.Rename(path, path+".1")
+	appendFile(path+".1", npmTestLine("9.9.9.9", "/rotated"))
+	os.WriteFile(path, []byte(npmTestLine("1.0.0.1", "/fresh")), 0600)
+	restored.readLogs(dir)
+	if len(restored.Pending) != 2 {
+		t.Fatalf("rotation lost events %+v", restored)
+	}
+	for _, e := range restored.Pending {
+		if e.ID == id {
+			t.Fatal("duplicate id")
+		}
+	}
+	restored.Pending = nil
+	restored.readLogs(dir)
+	if len(restored.Pending) != 0 {
+		t.Fatal("re-read same lines")
+	}
+}
+func TestTrafficStoreDedupFiltersAndSanitization(t *testing.T) {
+	s, a := trafficFixture(t)
+	if s.TrafficSettings().Enabled {
+		t.Fatal("enabled by default")
+	}
+	if _, err := s.SaveTrafficSettings(TrafficSettings{Enabled: true, AgentID: a.ID}); err != nil {
+		t.Fatal(err)
+	}
+	now := time.Now().Unix()
+	batch := TrafficBatch{Files: 22, Events: []TrafficEvent{{ID: "1", Time: now, Kind: "request", IP: "1.1.1.1", Domain: "example.test", Path: "/login?secret=x", Status: 502}, {ID: "2", Time: now, Kind: "request", IP: "192.168.0.10", Domain: "example.test", Status: 200}, {ID: "3", Time: now, Kind: "alert", IP: "1.1.1.1", Reason: "probe"}, {ID: "4", Time: now, Kind: "decision", IP: "1.1.1.1", Reason: "ban"}}}
+	for i := 0; i < 2; i++ {
+		if err := s.SaveTraffic(a.ID, batch); err != nil {
+			t.Fatal(err)
+		}
+	}
+	v, err := s.TrafficView("", "", false)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if v.Requests != 2 || v.UniqueIPs != 2 || v.Alerts != 1 || v.Decisions != 1 || len(v.Events) != 4 {
+		t.Fatalf("wrong counts: %+v", v)
+	}
+	for _, e := range v.Events {
+		if strings.Contains(e.Path, "secret") {
+			t.Fatal("secret saved")
+		}
+	}
+	v, err = s.TrafficView("example.test", "request", true)
+	if err != nil || v.Requests != 1 {
+		t.Fatalf("bad filter %+v %v", v, err)
+	}
+	batch.Events = []TrafficEvent{{ID: "invalid", Kind: "blocked", IP: "1.1.1.1", Time: now}}
+	if s.SaveTraffic(a.ID, batch) == nil {
+		t.Fatal("accepted invented block event")
+	}
+}
+func TestTrafficAPIOptInAndSourceAuthorization(t *testing.T) {
+	s, a := trafficFixture(t)
+	mux := http.NewServeMux()
+	trafficRoutes(mux, s)
+	get := func(path string) int {
+		r := httptest.NewRecorder()
+		mux.ServeHTTP(r, httptest.NewRequest("GET", path, nil))
+		return r.Code
+	}
+	if get("/traffic") != 404 || get("/api/traffic") != 404 {
+		t.Fatal("disabled page exposed")
+	}
+	send := func(secret string) int {
+		r := httptest.NewRecorder()
+		req := httptest.NewRequest("POST", "/api/agent/traffic", strings.NewReader(`{"events":[]}`))
+		req.Header.Set("Authorization", "Bearer "+a.ID+"."+secret)
+		mux.ServeHTTP(r, req)
+		return r.Code
+	}
+	if send("wrong") != 401 || send(a.Secret) != 403 {
+		t.Fatal("disabled/unauthorized ingestion accepted")
+	}
+	if _, err := s.SaveTrafficSettings(TrafficSettings{Enabled: true, AgentID: "missing"}); err == nil {
+		t.Fatal("unknown agent accepted")
+	}
+	s.SaveTrafficSettings(TrafficSettings{Enabled: true, AgentID: a.ID, ServerCountry: "RU"})
+	if send(a.Secret) != 200 || get("/traffic") != 200 {
+		t.Fatal("selected agent not working")
+	}
+	token, _ := s.CreateAgentEnrollmentToken()
+	other, _ := s.EnrollAgent(agentEnrollmentRequest{Token: token, Name: "Other"})
+	s.SaveTrafficSettings(TrafficSettings{Enabled: true, AgentID: other.ID})
+	if send(a.Secret) != 403 {
+		t.Fatal("old source still accepted")
+	}
+	s.SaveTrafficSettings(TrafficSettings{Enabled: false, AgentID: other.ID})
+	if get("/api/traffic") != 404 {
+		t.Fatal("disable failed")
+	}
+}
+func TestTrafficGeoDatabase(t *testing.T) {
+	if !trafficGeoReady() {
+		t.Fatal("embedded database invalid")
+	}
+	if lookupTrafficCountry("192.168.0.10") != "LAN" {
+		t.Fatal("LAN mapped to country")
+	}
+	if c := lookupTrafficCountry("8.8.8.8"); len(c) != 2 {
+		t.Fatalf("bad public IP geo: %q", c)
+	}
+}
+func TestTrafficCheckpointKeepsPendingUntilAcknowledged(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "state.json")
+	s := trafficAgentState{Pending: []TrafficEvent{{ID: "retry-me", IP: "1.1.1.1"}}, Cursors: map[string]trafficCursor{"file": {Offset: 100}}}
+	if err := saveTrafficAgentState(path, s); err != nil {
+		t.Fatal(err)
+	}
+	raw, _ := os.ReadFile(path)
+	if !bytes.Contains(raw, []byte("retry-me")) {
+		t.Fatal("pending batch lost")
+	}
+}

traffic_web.go

@@ -0,0 +1,6 @@
+package main
+
+import _ "embed"
+
+//go:embed assets/traffic.html
+var trafficHTML string

trends.go

@@ -0,0 +1,107 @@
+package main
+
+import (
+	"database/sql"
+	"fmt"
+	"sync"
+	"time"
+)
+
+type entityAverage struct {
+	Value1, Value2 float64
+	Count          int
+}
+type TrendAnomaly struct {
+	ID       string `json:"id"`
+	Severity string `json:"severity"`
+	Source   string `json:"source"`
+	Title    string `json:"title"`
+	Message  string `json:"message"`
+}
+type trendCollector struct {
+	mu        sync.Mutex
+	updatedAt time.Time
+	cached    []TrendAnomaly
+}
+
+func (s *Store) entityAverages(kind string) map[string]entityAverage {
+	return s.entityAveragesSince(kind, 7*24*time.Hour)
+}
+func (s *Store) entityAveragesSince(kind string, duration time.Duration) map[string]entityAverage {
+	rows, err := s.db.Query(`SELECT name,AVG(value1),AVG(value2),COUNT(*) FROM entity_history WHERE kind=? AND ts>=? GROUP BY name`, kind, time.Now().Add(-duration).Unix())
+	if err != nil {
+		return nil
+	}
+	defer rows.Close()
+	result := map[string]entityAverage{}
+	for rows.Next() {
+		var name string
+		var v1, v2 sql.NullFloat64
+		var count int
+		if rows.Scan(&name, &v1, &v2, &count) == nil {
+			result[name] = entityAverage{Value1: v1.Float64, Value2: v2.Float64, Count: count}
+		}
+	}
+	return result
+}
+
+func (c *trendCollector) collect(metrics dashboardMetrics, store *Store) []TrendAnomaly {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	if time.Since(c.updatedAt) < time.Minute {
+		return c.cached
+	}
+	var out []TrendAnomaly
+	iops := store.entityAverages("disk-io")
+	latency := store.entityAverages("disk-latency")
+	latencyDay := store.entityAveragesSince("disk-latency", 24*time.Hour)
+	temps := store.entityAverages("disk")
+	for _, d := range metrics.Disks {
+		name := d.Name + " · " + d.Model
+		currentIOPS := d.ReadIOPS + d.WriteIOPS
+		if base := iops[name]; base.Count >= 30 && base.Value1 >= 1 && currentIOPS >= base.Value1*5 {
+			out = append(out, TrendAnomaly{"disk-iops-" + d.Name, "warning", "Аномалии", d.Model + ": IOPS резко выше обычного", fmt.Sprintf("Сейчас %.0f IOPS, недельный baseline %.0f — рост в %.1f раза.", currentIOPS, base.Value1, currentIOPS/base.Value1)})
+		}
+		if base := latency[name]; base.Count >= 30 && base.Value1 > 0 && d.LatencyMs >= base.Value1*2 && d.LatencyMs-base.Value1 >= 10 {
+			out = append(out, TrendAnomaly{"disk-latency-" + d.Name, "warning", "Аномалии", d.Model + ": latency ухудшилась", fmt.Sprintf("Сейчас %.1f ms при baseline %.1f ms.", d.LatencyMs, base.Value1)})
+		}
+		if week, day := latency[name], latencyDay[name]; week.Count >= 100 && day.Count >= 30 && week.Value1 > 0 && day.Value1 >= week.Value1*1.5 && day.Value1-week.Value1 >= 5 {
+			out = append(out, TrendAnomaly{"disk-latency-degrading-" + d.Name, "warning", "Аномалии", d.Model + ": latency постепенно ухудшается", fmt.Sprintf("Среднее за сутки %.1f ms против недельного %.1f ms.", day.Value1, week.Value1)})
+		}
+		if d.Temperature != nil {
+			if base := temps[name]; base.Count >= 30 && *d.Temperature >= base.Value1+10 {
+				out = append(out, TrendAnomaly{"disk-temp-trend-" + d.Name, "warning", "Аномалии", d.Model + ": температура выше обычной", fmt.Sprintf("Сейчас %.1f °C, baseline %.1f °C.", *d.Temperature, base.Value1)})
+			}
+		}
+	}
+	guestAvg := store.entityAverages("guest")
+	for _, g := range metrics.Guests.Guests {
+		if g.MaxMemory == 0 {
+			continue
+		}
+		name := fmt.Sprintf("%d · %s", g.VMID, g.Name)
+		current := float64(g.MemoryBytes) / float64(g.MaxMemory) * 100
+		if base := guestAvg[name]; base.Count >= 30 && current >= base.Value2*1.5 && current-base.Value2 >= 15 {
+			out = append(out, TrendAnomaly{fmt.Sprintf("guest-memory-%d", g.VMID), "warning", "Аномалии", g.Name + ": память выше обычной", fmt.Sprintf("Сейчас %.1f%%, baseline %.1f%%.", current, base.Value2)})
+		}
+	}
+	zfsAverage := store.entityAverages("zfs")
+	for _, pool := range metrics.ZFS.Pools {
+		if base := zfsAverage[pool.Name]; base.Count >= 30 && pool.CapacityPercent-base.Value1 >= 5 {
+			out = append(out, TrendAnomaly{"zfs-growth-" + pool.Name, "warning", "Аномалии", pool.Name + ": заполнение выше недельного уровня", fmt.Sprintf("Сейчас %.1f%%, недельный baseline %.1f%%.", pool.CapacityPercent, base.Value1)})
+		}
+	}
+	backupAverage := store.entityAverages("backup-duration")
+	for _, task := range metrics.Activity.Tasks {
+		if task.Type != "vzdump" || task.Duration <= 0 {
+			continue
+		}
+		name := "VMID " + task.ID
+		if base := backupAverage[name]; base.Count >= 3 && float64(task.Duration) >= base.Value1*1.5 && float64(task.Duration)-base.Value1 >= 60 {
+			out = append(out, TrendAnomaly{"backup-duration-" + task.ID, "warning", "Аномалии", "Backup " + name + " выполнялся дольше обычного", fmt.Sprintf("Последняя длительность %s, baseline %s.", time.Duration(task.Duration)*time.Second, time.Duration(base.Value1)*time.Second)})
+		}
+	}
+	c.cached = out
+	c.updatedAt = time.Now()
+	return out
+}

update.go

@@ -0,0 +1,77 @@
+package main
+
+import (
+	"encoding/json"
+	"fmt"
+	"os"
+	"path/filepath"
+	"time"
+)
+
+const defaultUpdateStatusPath = "/var/lib/proxmox-dashboard/update-status.json"
+const defaultUpdateRequestPath = "/var/lib/proxmox-dashboard/update-request"
+
+type UpdateStatus struct {
+	CurrentVersion   string    `json:"currentVersion"`
+	AvailableVersion string    `json:"availableVersion"`
+	State            string    `json:"state"`
+	Mode             string    `json:"mode"`
+	Message          string    `json:"message"`
+	ReleaseURL       string    `json:"releaseUrl"`
+	CheckedAt        time.Time `json:"checkedAt"`
+}
+
+func readUpdateStatus() UpdateStatus {
+	status := UpdateStatus{CurrentVersion: version, State: "unknown", Mode: "notify"}
+	path := os.Getenv("DASHBOARD_UPDATE_STATUS_PATH")
+	if path == "" {
+		path = defaultUpdateStatusPath
+	}
+	data, err := os.ReadFile(path)
+	if err != nil || json.Unmarshal(data, &status) != nil {
+		return status
+	}
+	status.CurrentVersion = version
+	return status
+}
+
+func requestUpdate() (UpdateStatus, error) {
+	requestPath := os.Getenv("DASHBOARD_UPDATE_REQUEST_PATH")
+	if requestPath == "" {
+		requestPath = defaultUpdateRequestPath
+	}
+	if err := os.MkdirAll(filepath.Dir(requestPath), 0o750); err != nil {
+		return UpdateStatus{}, err
+	}
+	status := readUpdateStatus()
+	status.State = "queued"
+	status.Message = "Обновление поставлено в очередь."
+	status.CheckedAt = time.Now()
+	statusPath := os.Getenv("DASHBOARD_UPDATE_STATUS_PATH")
+	if statusPath == "" {
+		statusPath = defaultUpdateStatusPath
+	}
+	data, _ := json.Marshal(status)
+	temporary, err := os.CreateTemp(filepath.Dir(statusPath), ".update-status-")
+	if err != nil {
+		return UpdateStatus{}, err
+	}
+	temporaryPath := temporary.Name()
+	defer os.Remove(temporaryPath)
+	if _, err = temporary.Write(data); err == nil {
+		err = temporary.Chmod(0o644)
+	}
+	if closeErr := temporary.Close(); err == nil {
+		err = closeErr
+	}
+	if err == nil {
+		err = os.Rename(temporaryPath, statusPath)
+	}
+	if err != nil {
+		return UpdateStatus{}, fmt.Errorf("не удалось сохранить запрос обновления: %w", err)
+	}
+	if err := os.WriteFile(requestPath, []byte(time.Now().Format(time.RFC3339Nano)), 0o640); err != nil {
+		return UpdateStatus{}, err
+	}
+	return status, nil
+}

update_test.go

@@ -0,0 +1,45 @@
+package main
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+)
+
+func TestReadUpdateStatus(t *testing.T) {
+	path := filepath.Join(t.TempDir(), "status.json")
+	if err := os.WriteFile(path, []byte(`{"currentVersion":"old","availableVersion":"v2.0.0","state":"available","mode":"notify","message":"Доступно обновление"}`), 0o600); err != nil {
+		t.Fatal(err)
+	}
+	t.Setenv("DASHBOARD_UPDATE_STATUS_PATH", path)
+	status := readUpdateStatus()
+	if status.CurrentVersion != version || status.AvailableVersion != "v2.0.0" || status.State != "available" {
+		t.Fatalf("unexpected update status: %+v", status)
+	}
+}
+
+func TestReadUpdateStatusWithoutFile(t *testing.T) {
+	t.Setenv("DASHBOARD_UPDATE_STATUS_PATH", filepath.Join(t.TempDir(), "missing.json"))
+	status := readUpdateStatus()
+	if status.CurrentVersion != version || status.State != "unknown" {
+		t.Fatalf("unexpected default update status: %+v", status)
+	}
+}
+
+func TestRequestUpdate(t *testing.T) {
+	directory := t.TempDir()
+	statusPath := filepath.Join(directory, "status.json")
+	requestPath := filepath.Join(directory, "request")
+	t.Setenv("DASHBOARD_UPDATE_STATUS_PATH", statusPath)
+	t.Setenv("DASHBOARD_UPDATE_REQUEST_PATH", requestPath)
+	status, err := requestUpdate()
+	if err != nil {
+		t.Fatal(err)
+	}
+	if status.State != "queued" {
+		t.Fatalf("unexpected queued status: %+v", status)
+	}
+	if _, err := os.Stat(requestPath); err != nil {
+		t.Fatalf("update request was not created: %v", err)
+	}
+}

ups.go

@@ -10,20 +10,26 @@ import (
 )
 
 type UPSMetrics struct {
-	Available      bool     `json:"available"`
-	Source         string   `json:"source"`
-	Name           string   `json:"name"`
-	Model          string   `json:"model"`
-	Status         string   `json:"status"`
-	StatusLabel    string   `json:"statusLabel"`
-	ChargePercent  *float64 `json:"chargePercent"`
-	RuntimeSeconds *float64 `json:"runtimeSeconds"`
-	LoadPercent    *float64 `json:"loadPercent"`
-	InputVoltage   *float64 `json:"inputVoltage"`
-	OutputVoltage  *float64 `json:"outputVoltage"`
-	InputFrequency *float64 `json:"inputFrequency"`
-	BatteryVoltage *float64 `json:"batteryVoltage"`
-	Temperature    *float64 `json:"temperatureCelsius"`
+	Available         bool     `json:"available"`
+	Source            string   `json:"source"`
+	Name              string   `json:"name"`
+	Model             string   `json:"model"`
+	Status            string   `json:"status"`
+	StatusLabel       string   `json:"statusLabel"`
+	ChargePercent     *float64 `json:"chargePercent"`
+	RuntimeSeconds    *float64 `json:"runtimeSeconds"`
+	LoadPercent       *float64 `json:"loadPercent"`
+	InputVoltage      *float64 `json:"inputVoltage"`
+	OutputVoltage     *float64 `json:"outputVoltage"`
+	InputFrequency    *float64 `json:"inputFrequency"`
+	BatteryVoltage    *float64 `json:"batteryVoltage"`
+	Temperature       *float64 `json:"temperatureCelsius"`
+	CommunicationOK   bool     `json:"communicationOk"`
+	BatteryDate       string   `json:"batteryDate"`
+	BatteryAgeDays    int      `json:"batteryAgeDays,omitempty"`
+	ReplaceBattery    bool     `json:"replaceBattery"`
+	LowBatteryCharge  *float64 `json:"lowBatteryCharge,omitempty"`
+	LowBatteryRuntime *float64 `json:"lowBatteryRuntime,omitempty"`
 }
 
 type upsCollector struct {
@@ -43,7 +49,13 @@ func (c *upsCollector) collect() UPSMetrics {
 	} else if metrics, ok := readAPCUPSD(); ok {
 		c.cached = metrics
 	} else {
-		c.cached = UPSMetrics{}
+		if c.cached.Available || c.cached.Source != "" {
+			c.cached.Available = false
+			c.cached.CommunicationOK = false
+			c.cached.StatusLabel = "Нет связи"
+		} else {
+			c.cached = UPSMetrics{}
+		}
 	}
 	c.updatedAt = time.Now()
 	return c.cached
@@ -85,6 +97,7 @@ func readAPCUPSD() (UPSMetrics, bool) {
 		"ups.load": raw["LOADPCT"], "input.voltage": raw["LINEV"],
 		"output.voltage": raw["OUTPUTV"], "input.frequency": raw["LINEFREQ"],
 		"battery.voltage": raw["BATTV"], "ups.temperature": raw["ITEMP"],
+		"battery.date": raw["BATTDATE"], "battery.charge.low": raw["MBATTCHG"], "battery.runtime.low": minutesToSeconds(raw["MINTIMEL"]),
 	}
 	return upsFromValues("apcupsd", raw["UPSNAME"], values), true
 }
@@ -102,14 +115,24 @@ func parseColonValues(output string) map[string]string {
 
 func upsFromValues(source, name string, values map[string]string) UPSMetrics {
 	status := values["ups.status"]
-	return UPSMetrics{
-		Available: true, Source: source, Name: name, Model: values["device.model"],
+	metrics := UPSMetrics{
+		Available: true, CommunicationOK: true, Source: source, Name: name, Model: values["device.model"],
 		Status: status, StatusLabel: upsStatusLabel(status),
 		ChargePercent: numberPointer(values["battery.charge"]), RuntimeSeconds: numberPointer(values["battery.runtime"]),
 		LoadPercent: numberPointer(values["ups.load"]), InputVoltage: numberPointer(values["input.voltage"]),
 		OutputVoltage: numberPointer(values["output.voltage"]), InputFrequency: numberPointer(values["input.frequency"]),
 		BatteryVoltage: numberPointer(values["battery.voltage"]), Temperature: numberPointer(firstNonEmpty(values["ups.temperature"], values["battery.temperature"])),
+		BatteryDate: firstNonEmpty(values["battery.date"], values["battery.mfr.date"]), ReplaceBattery: strings.Contains(status, "RB"), LowBatteryCharge: numberPointer(values["battery.charge.low"]), LowBatteryRuntime: numberPointer(values["battery.runtime.low"]),
 	}
+	if metrics.BatteryDate != "" {
+		for _, layout := range []string{"2006/01/02", "2006-01-02", "01/02/2006", "02/01/2006"} {
+			if date, err := time.Parse(layout, metrics.BatteryDate); err == nil {
+				metrics.BatteryAgeDays = int(time.Since(date).Hours() / 24)
+				break
+			}
+		}
+	}
+	return metrics
 }
 
 func numberPointer(value string) *float64 {

zfs.go

@@ -2,7 +2,9 @@ package main
 
 import (
 	"context"
+	"os"
 	"os/exec"
+	"regexp"
 	"strconv"
 	"strings"
 	"sync"
@@ -10,28 +12,66 @@ import (
 )
 
 type ZFSPoolMetrics struct {
-	Name             string   `json:"name"`
-	Health           string   `json:"health"`
-	SizeBytes        uint64   `json:"sizeBytes"`
-	AllocatedBytes   uint64   `json:"allocatedBytes"`
-	FreeBytes        uint64   `json:"freeBytes"`
-	CapacityPercent  float64  `json:"capacityPercent"`
-	Fragmentation    float64  `json:"fragmentationPercent"`
-	DedupRatio       string   `json:"dedupRatio"`
-	Scan             string   `json:"scan"`
-	Errors           string   `json:"errors"`
-	AttentionReasons []string `json:"attentionReasons"`
+	Name             string    `json:"name"`
+	Health           string    `json:"health"`
+	SizeBytes        uint64    `json:"sizeBytes"`
+	AllocatedBytes   uint64    `json:"allocatedBytes"`
+	FreeBytes        uint64    `json:"freeBytes"`
+	CapacityPercent  float64   `json:"capacityPercent"`
+	Fragmentation    float64   `json:"fragmentationPercent"`
+	DedupRatio       string    `json:"dedupRatio"`
+	Scan             string    `json:"scan"`
+	Errors           string    `json:"errors"`
+	AttentionReasons []string  `json:"attentionReasons"`
+	ReadErrors       uint64    `json:"readErrors"`
+	WriteErrors      uint64    `json:"writeErrors"`
+	ChecksumErrors   uint64    `json:"checksumErrors"`
+	VDevs            []ZFSVDev `json:"vdevs"`
+	ScrubStatus      string    `json:"scrubStatus"`
+	LastScrubAt      string    `json:"lastScrubAt"`
+	ScrubDuration    string    `json:"scrubDuration"`
+	ScrubTooOld      bool      `json:"scrubTooOld"`
+	ErrorChanges     []string  `json:"errorChanges"`
+}
+
+type ZFSVDev struct {
+	Name     string `json:"name"`
+	Class    string `json:"class"`
+	State    string `json:"state"`
+	Read     uint64 `json:"read"`
+	Write    uint64 `json:"write"`
+	Checksum uint64 `json:"checksum"`
+}
+type ZFSDataset struct {
+	Name            string  `json:"name"`
+	Pool            string  `json:"pool"`
+	UsedBytes       uint64  `json:"usedBytes"`
+	AvailableBytes  uint64  `json:"availableBytes"`
+	ReferencedBytes uint64  `json:"referencedBytes"`
+	Mountpoint      string  `json:"mountpoint"`
+	CapacityPercent float64 `json:"capacityPercent"`
+}
+type ZFSARC struct {
+	Available   bool    `json:"available"`
+	SizeBytes   uint64  `json:"sizeBytes"`
+	TargetBytes uint64  `json:"targetBytes"`
+	HitRatio    float64 `json:"hitRatio"`
+	Hits        uint64  `json:"hits"`
+	Misses      uint64  `json:"misses"`
 }
 
 type ZFSMetrics struct {
 	Available bool             `json:"available"`
 	Pools     []ZFSPoolMetrics `json:"pools"`
+	Datasets  []ZFSDataset     `json:"datasets"`
+	ARC       ZFSARC           `json:"arc"`
 }
 
 type zfsCollector struct {
 	mu        sync.Mutex
 	cached    ZFSMetrics
 	updatedAt time.Time
+	previous  map[string]ZFSPoolMetrics
 }
 
 func (c *zfsCollector) collect() ZFSMetrics {
@@ -40,7 +80,26 @@ func (c *zfsCollector) collect() ZFSMetrics {
 	if time.Since(c.updatedAt) < 30*time.Second {
 		return c.cached
 	}
-	c.cached = readZFSPools()
+	current := readZFSPools()
+	if c.previous == nil {
+		c.previous = map[string]ZFSPoolMetrics{}
+	}
+	for i := range current.Pools {
+		pool := &current.Pools[i]
+		if old, ok := c.previous[pool.Name]; ok {
+			checks := []struct {
+				label       string
+				before, now uint64
+			}{{"read", old.ReadErrors, pool.ReadErrors}, {"write", old.WriteErrors, pool.WriteErrors}, {"checksum", old.ChecksumErrors, pool.ChecksumErrors}}
+			for _, check := range checks {
+				if check.now > check.before {
+					pool.ErrorChanges = append(pool.ErrorChanges, check.label+": "+strconv.FormatUint(check.before, 10)+" → "+strconv.FormatUint(check.now, 10))
+				}
+			}
+		}
+		c.previous[pool.Name] = *pool
+	}
+	c.cached = current
 	c.updatedAt = time.Now()
 	return c.cached
 }
@@ -67,15 +126,35 @@ func readZFSPools() ZFSMetrics {
 		pool.FreeBytes, _ = strconv.ParseUint(fields[3], 10, 64)
 		pool.Fragmentation = parsePercent(fields[4])
 		pool.CapacityPercent = parsePercent(fields[5])
-		pool.Scan, pool.Errors = readZFSStatus(pool.Name)
+		pool.Scan, pool.Errors, pool.VDevs = readZFSStatus(pool.Name)
+		for _, vdev := range pool.VDevs {
+			pool.ReadErrors += vdev.Read
+			pool.WriteErrors += vdev.Write
+			pool.ChecksumErrors += vdev.Checksum
+			if vdev.State != "ONLINE" {
+				pool.AttentionReasons = append(pool.AttentionReasons, vdev.Class+" "+vdev.Name+": "+vdev.State)
+			}
+		}
+		pool.ScrubStatus, pool.LastScrubAt, pool.ScrubDuration, pool.ScrubTooOld = parseScrub(pool.Scan)
 		if pool.Health != "ONLINE" {
 			pool.AttentionReasons = append(pool.AttentionReasons, "Состояние пула: "+pool.Health)
 		}
 		if pool.Errors != "" && !strings.Contains(pool.Errors, "No known data errors") {
 			pool.AttentionReasons = append(pool.AttentionReasons, pool.Errors)
 		}
+		if pool.ReadErrors+pool.WriteErrors+pool.ChecksumErrors > 0 {
+			pool.AttentionReasons = append(pool.AttentionReasons, "Ошибки vdev: read="+strconv.FormatUint(pool.ReadErrors, 10)+", write="+strconv.FormatUint(pool.WriteErrors, 10)+", checksum="+strconv.FormatUint(pool.ChecksumErrors, 10))
+		}
+		if pool.ScrubTooOld {
+			pool.AttentionReasons = append(pool.AttentionReasons, "Scrub не выполнялся более 35 дней")
+		}
+		if pool.CapacityPercent >= 80 {
+			pool.AttentionReasons = append(pool.AttentionReasons, "Мало свободного места для безопасной CoW-работы ZFS")
+		}
 		result.Pools = append(result.Pools, pool)
 	}
+	result.Datasets = readZFSDatasets()
+	result.ARC = readZFSARC()
 	return result
 }
 
@@ -84,11 +163,13 @@ func parsePercent(value string) float64 {
 	return parsed
 }
 
-func readZFSStatus(pool string) (string, string) {
+func readZFSStatus(pool string) (string, string, []ZFSVDev) {
 	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
 	defer cancel()
 	output, _ := exec.CommandContext(ctx, "zpool", "status", pool).Output()
-	var scan, errors string
+	var scan, errors, currentClass string
+	var vdevs []ZFSVDev
+	inConfig := false
 	for _, line := range strings.Split(string(output), "\n") {
 		trimmed := strings.TrimSpace(line)
 		if strings.HasPrefix(trimmed, "scan:") {
@@ -97,6 +178,98 @@ func readZFSStatus(pool string) (string, string) {
 		if strings.HasPrefix(trimmed, "errors:") {
 			errors = strings.TrimSpace(strings.TrimPrefix(trimmed, "errors:"))
 		}
+		if strings.HasPrefix(trimmed, "config:") {
+			inConfig = true
+			continue
+		}
+		if !inConfig || trimmed == "" || strings.HasPrefix(trimmed, "NAME") {
+			continue
+		}
+		fields := strings.Fields(trimmed)
+		if len(fields) == 1 && (fields[0] == "logs" || fields[0] == "cache" || fields[0] == "spares" || fields[0] == "special") {
+			currentClass = fields[0]
+			continue
+		}
+		if len(fields) >= 5 && (fields[1] == "ONLINE" || fields[1] == "DEGRADED" || fields[1] == "OFFLINE" || fields[1] == "UNAVAIL" || fields[1] == "FAULTED" || fields[1] == "REMOVED") {
+			read, _ := strconv.ParseUint(fields[2], 10, 64)
+			write, _ := strconv.ParseUint(fields[3], 10, 64)
+			checksum, _ := strconv.ParseUint(fields[4], 10, 64)
+			class := currentClass
+			if class == "" {
+				class = "data"
+			}
+			vdevs = append(vdevs, ZFSVDev{Name: fields[0], Class: class, State: fields[1], Read: read, Write: write, Checksum: checksum})
+		}
 	}
-	return scan, errors
+	return scan, errors, vdevs
+}
+
+var scrubDatePattern = regexp.MustCompile(`(?:on|since) (\w{3} \w{3}\s+\d+ \d\d:\d\d:\d\d \d{4})`)
+
+func parseScrub(scan string) (status, lastAt, duration string, tooOld bool) {
+	lower := strings.ToLower(scan)
+	status = scan
+	if strings.Contains(lower, "in progress") {
+		status = "Выполняется"
+	} else if strings.Contains(lower, "repaired") {
+		status = "Завершён"
+	} else if strings.Contains(lower, "none requested") || scan == "" {
+		status = "Никогда не запускался"
+		return status, "", "", true
+	}
+	if match := scrubDatePattern.FindStringSubmatch(scan); len(match) > 1 {
+		if t, err := time.Parse("Mon Jan 2 15:04:05 2006", match[1]); err == nil {
+			lastAt = t.Format(time.RFC3339)
+			tooOld = time.Since(t) > 35*24*time.Hour
+		}
+	}
+	if parts := strings.Split(scan, " in "); len(parts) > 1 {
+		duration = strings.Fields(parts[1])[0]
+	}
+	return
+}
+
+func readZFSDatasets() []ZFSDataset {
+	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
+	defer cancel()
+	out, err := exec.CommandContext(ctx, "zfs", "list", "-H", "-p", "-o", "name,used,avail,refer,mountpoint").Output()
+	if err != nil {
+		return nil
+	}
+	var result []ZFSDataset
+	for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
+		f := strings.Fields(line)
+		if len(f) < 5 {
+			continue
+		}
+		used, _ := strconv.ParseUint(f[1], 10, 64)
+		avail, _ := strconv.ParseUint(f[2], 10, 64)
+		refer, _ := strconv.ParseUint(f[3], 10, 64)
+		pct := 0.0
+		if used+avail > 0 {
+			pct = float64(used) / float64(used+avail) * 100
+		}
+		result = append(result, ZFSDataset{Name: f[0], Pool: strings.Split(f[0], "/")[0], UsedBytes: used, AvailableBytes: avail, ReferencedBytes: refer, Mountpoint: strings.Join(f[4:], " "), CapacityPercent: pct})
+	}
+	return result
+}
+
+func readZFSARC() ZFSARC {
+	data, err := os.ReadFile("/proc/spl/kstat/zfs/arcstats")
+	if err != nil {
+		return ZFSARC{}
+	}
+	values := map[string]uint64{}
+	for _, line := range strings.Split(string(data), "\n") {
+		f := strings.Fields(line)
+		if len(f) >= 3 {
+			values[f[0]], _ = strconv.ParseUint(f[2], 10, 64)
+		}
+	}
+	hits, misses := values["hits"], values["misses"]
+	ratio := 0.0
+	if hits+misses > 0 {
+		ratio = float64(hits) / float64(hits+misses) * 100
+	}
+	return ZFSARC{Available: true, SizeBytes: values["size"], TargetBytes: values["c"], HitRatio: ratio, Hits: hits, Misses: misses}
 }