23 Commits

Author SHA1 Message Date
Maxim
f1ff15e344 Add read-only guest monitoring agents 2026-08-06 20:24:13 +03:00
Maxim
5401b5b0f2 Add maintenance windows for planned work 2026-08-06 19:35:33 +03:00
Maxim
c01b4b6630 Fix service monitor connection leak 2026-08-05 20:04:29 +03:00
Maxim
a14001b373 Treat historical SMART failures as resolved 2026-08-03 21:17:15 +03:00
Maxim
05450bc949 Improve system health sorting and dashboard spacing 2026-08-03 21:09:30 +03:00
Maxim
8241fba6f4 Add IO attribution anomaly detection and alert correlation 2026-08-03 18:02:13 +03:00
Maxim
98afa2517b Add per-disk IO performance monitoring 2026-08-03 17:32:28 +03:00
Maxim
96af2faeb1 Add binary-compatible collection watchdog 2026-08-01 17:03:15 +03:00
Maxim
a570537e64 Add infrastructure reliability monitoring 2026-08-01 16:58:25 +03:00
Maxim
f6ac7eeb1a Fix task filters and improve homelab topology 2026-08-01 14:09:13 +03:00
Maxim
b21377337b Add beginner installation guide 2026-08-01 13:57:25 +03:00
Maxim
f1356fec37 Add backup diagnostics snapshots tasks and homelab map 2026-08-01 13:50:58 +03:00
Maxim
92607f62d5 Add per-resource history and guest network groups 2026-08-01 12:43:05 +03:00
Maxim
d3fbbf2a6b Improve service checks and system health details 2026-08-01 12:21:26 +03:00
Maxim
980f69cf8f Add homelab service and system health monitoring 2026-08-01 12:02:09 +03:00
Maxim
ad8972320d Expand guest metrics and add Proxmox maintenance overview 2026-08-01 11:38:03 +03:00
Maxim
5491be2e0c Add acknowledged alert history and HTML email digests 2026-08-01 11:00:16 +03:00
Maxim
9e907f8455 Allow optional SMTP TLS verification bypass 2026-07-31 23:31:22 +03:00
maxim
a987667233 Add configurable email notification digests 2026-07-31 23:01:46 +03:00
maxim
58ec47b188 Add one-click dashboard updates 2026-07-31 22:32:20 +03:00
maxim
2485eae120 Restart dashboard during repeated installation 2026-07-31 22:21:18 +03:00
maxim
3f881a59e5 Add managed Gitea updates and server installer 2026-07-31 22:16:51 +03:00
maxim
22f40f3830 Improve historical charts with tabs and tooltips 2026-07-31 22:05:04 +03:00
47 changed files with 5445 additions and 158 deletions

236
README.md
View File

@@ -3,8 +3,139 @@
Локальная веб-панель CPU для Proxmox. Данные читаются напрямую из `/proc` и
`/sys`, без Proxmox API. Интерфейс обновляется каждые две секунды.
## Установка для новичка — скопируйте команды целиком
### Что потребуется
- сервер Proxmox VE с доступом в интернет или к `git.myown.center`;
- вход в консоль Proxmox под пользователем `root`;
- IP-адрес Proxmox в локальной сети;
- свободный TCP-порт `9105` между вашим компьютером и Proxmox.
Go, Node.js, Docker и отдельную базу данных устанавливать не нужно. Панель
поставляется готовым Linux-бинарником, а SQLite встроен внутрь.
### 1. Подключитесь к Proxmox
На своём компьютере откройте Terminal и выполните команду, заменив IP на адрес
своего Proxmox:
```bash
ssh root@192.168.0.10
```
Введите пароль `root`. Следующие команды выполняйте уже внутри Proxmox.
### 2. Установите панель
Скопируйте весь блок и вставьте его в консоль Proxmox:
```bash
apt-get update
apt-get install -y curl ca-certificates tar smartmontools iproute2 iputils-ping
cd /tmp
curl -fL \
https://git.myown.center/maxim/ProxmoxDash/releases/download/latest/proxmox-dashboard-install-amd64.tar.gz \
-o proxmox-dashboard-install-amd64.tar.gz
tar -xzf proxmox-dashboard-install-amd64.tar.gz
cd proxmox-dashboard-install
UPDATE_MODE=notify ./scripts/install.sh
```
Готово. Установщик сам:
- положит бинарник в `/usr/local/bin`;
- создаст локальную базу настроек и истории;
- установит systemd-службу;
- запустит панель;
- настроит ежедневную проверку новых версий;
- сохранит предыдущий бинарник при будущем обновлении.
### 3. Откройте панель
В браузере на своём компьютере откройте:
```text
http://192.168.0.10:9105
```
Замените `192.168.0.10` на IP своего Proxmox. Узнать адрес можно командой:
```bash
hostname -I
```
### 4. Проверьте, что всё работает
```bash
systemctl --no-pager --full status proxmox-cpu-dashboard.service
curl -I http://127.0.0.1:9105
```
В статусе службы должно быть `active (running)`, а локальная проверка должна
вернуть HTTP-ответ `200 OK`.
### Как обновлять
В режиме `notify`, установленном командами выше, откройте в панели
`Настройки → Обновления` и нажмите `Проверить и установить обновление`.
Если нужны полностью автоматические обновления, повторно запустите установщик:
```bash
cd /tmp/proxmox-dashboard-install
UPDATE_MODE=auto ./scripts/install.sh
```
### Если репозиторий станет приватным
Создайте в Gitea токен только для чтения репозитория и передайте его
установщику. Вместо текста `ВАШ_ТОКЕН` вставьте настоящий токен:
```bash
cd /tmp/proxmox-dashboard-install
GITEA_TOKEN='ВАШ_ТОКЕН' UPDATE_MODE=notify ./scripts/install.sh
```
Токен сохраняется на Proxmox с правами `0600` и не показывается в браузере.
### Если страница не открывается
Выполните на Proxmox:
```bash
systemctl restart proxmox-cpu-dashboard.service
systemctl --no-pager --full status proxmox-cpu-dashboard.service
journalctl -u proxmox-cpu-dashboard.service -n 100 --no-pager
ss -lntp | grep 9105
```
Проверьте, что используете правильный IP, порт `9105` доступен из локальной
сети, а firewall Proxmox не блокирует подключение. Панель пока не имеет
авторизации — не публикуйте порт `9105` напрямую в интернет.
---
Панель показывает:
- изменение критичных SMART-счётчиков, результаты self-test и исчезновение дисков;
- realtime read/write IOPS, MB/s, latency, utilization и глубину очереди каждого диска;
- недельную историю IOPS/utilization и алерты при устойчивом I/O-давлении;
- вероятный источник I/O: VM/LXC, процесс хоста или активная backup/snapshot/migration-задача;
- отдельные пороги HDD, SATA SSD и NVMe с исключениями по имени или серийному номеру;
- недельные baseline-аномалии IOPS, latency, температуры, RAM, ZFS и длительности backup;
- корреляцию связанных событий в одну причинно-следственную цепочку;
- время первого появления и последнего наблюдения каждого уведомления;
- автоматический короткий SMART self-test раз в неделю для дисков без свежего теста;
- datasets, vdev-классы, ошибки read/write/checksum, scrub и ARC ZFS;
- доступность шлюза, DNS и интернета, packet loss, latency и внешний IP;
- тип, состояние, скорость, duplex и master сетевых интерфейсов;
- состояние связи с NUT/apcupsd, возраст батареи и расчётную автономность UPS;
- настраиваемый сценарий корректного выключения VM/LXC и Proxmox с безопасным dry-run;
- самоконтроль сборщика, записи истории, свободного места базы и systemd watchdog;
- модель и текущую загрузку CPU;
- количество сокетов, физических ядер и логических потоков;
- среднюю текущую частоту;
@@ -19,19 +150,57 @@
- состояние, заполнение, фрагментацию и scrub/resilver ZFS-пулов;
- состояние UPS через NUT или apcupsd, заряд, автономность, нагрузку и напряжение.
- состояние VM и LXC, CPU, память, выделенные ядра и uptime.
- IP-адреса VM через QEMU Guest Agent, автозапуск, storage, заполнение диска и
суммарный сетевой трафик гостей;
- последнюю найденную резервную копию каждой VM/LXC, storage, размер и возраст.
- версию Proxmox, запущенное и последнее установленное ядро, доступные
обновления, необходимость перезагрузки, последнее обновление системы,
ближайшие backup и ZFS scrub.
- управляемый мониторинг HTTP/HTTPS, TCP и ping для homelab-сервисов: отклик,
ожидаемый код и текст, TLS-сертификат, uptime за сутки и неделю;
- ошибки systemd и ядра за 24 часа, упавшие службы, ошибки и drops каждого
сетевого интерфейса.
- snapshots VM/LXC с возрастом, описанием и доступным размером;
- последние задачи Proxmox: backup, start/stop, snapshot, migration, пользователь,
длительность и результат;
- сравнение размера backup с предыдущей копией и ручную безопасную проверку
читаемости локального архива или конфигурации PBS snapshot;
- карту homelab: Proxmox, VM/LXC, сервисы и используемые storage.
Через кнопку настроек можно скрывать неиспользуемые карточки и переключать
светлую или тёмную тему. Выбор сохраняется локально в браузере.
светлую или тёмную тему. Карточки можно переставлять перетаскиванием или
стрелками в настройках. Видимость, порядок и тема сохраняются локально в
браузере.
Центр уведомлений объединяет критические события, предупреждения и
информационные сообщения по CPU, памяти, storage, SMART, ZFS, UPS, VM/LXC и
резервным копиям. Критические события всегда отображаются первыми.
Во вкладке `Настройки → Почта` настраиваются SMTP-сервер, получатель и интервал
агрегированной сводки. Можно отдельно выбрать уровни важности и источники
событий — например, полностью исключить уведомления о резервных копиях. Там же
доступна отправка тестового письма. Поддерживаются STARTTLS (обычно порт 587) и
SMTP поверх TLS (порт 465).
Тема письма кодируется по MIME и корректно отображает кириллицу. Сам дайджест
оформлен как HTML-письмо и содержит только новые неподтверждённые события — уже
отправленные или принятые события повторно не рассылаются. В центре уведомлений
есть вкладки активных событий и 30-дневной истории. Кнопка `Принято` скрывает
событие из активных, не удаляя его из журнала; повторное возникновение проблемы
создаёт новое событие.
Критические события отправляются сразу, не ожидая очередной сводки. К ним
относятся недоступность сервиса, отказ SMART/ZFS/systemd, критический перегрев и
низкий заряд UPS. Переход UPS на батарею также отправляется немедленно. После
устранения уже отправленной проблемы приходит отдельное уведомление о
восстановлении.
Пороги уведомлений сохраняются в SQLite и настраиваются через вкладку
`Настройки → Уведомления`. Панель раз в минуту сохраняет историю CPU,
температуры, памяти, swap, сети, storage, ZFS, UPS и гостей. Данные старше семи
дней удаляются автоматически. Графики находятся внутри подробных окон карточек.
дней удаляются автоматически. В подробных окнах карточек текущие показатели и
история разделены на вкладки. Для графиков можно выбрать последний час, 24 часа
или 7 дней, а точные значения и время отображаются при наведении.
База данных по умолчанию находится здесь:
@@ -42,6 +211,13 @@
Готовые версии для Linux публикуются в разделе Releases репозитория. Бинарник
не требует установки Go или системной библиотеки SQLite.
Проверка обновлений выполняется отдельным systemd-таймером раз в сутки. Статус
виден во вкладке `Настройки → Обновления` и в центре уведомлений. Доступны два
режима: `notify` только сообщает о новой версии, `auto` скачивает её, сохраняет
предыдущий бинарник как резервный и перезапускает панель. В режиме `notify`
обновление можно запустить кнопкой `Проверить и установить обновление` в
настройках. Запрос выполняет отдельная системная служба.
На главном экране находятся компактные карточки CPU, памяти, диска и сети.
Нажатие на карточку открывает подробные показатели.
@@ -54,6 +230,32 @@ chmod +x proxmox-cpu-dashboard-linux-amd64
./proxmox-cpu-dashboard-linux-amd64
```
## Установка на сервер и обновления
Скачайте из Releases архив `proxmox-dashboard-install-amd64.tar.gz`, распакуйте
его на Proxmox и запустите от root:
```bash
tar -xzf proxmox-dashboard-install-amd64.tar.gz
cd proxmox-dashboard-install
UPDATE_MODE=notify ./scripts/install.sh
```
Для полностью автоматических обновлений используйте `UPDATE_MODE=auto`.
Приватному репозиторию нужен read-only API-токен Gitea:
```bash
GITEA_TOKEN='read-only-token' UPDATE_MODE=auto ./scripts/install.sh
```
Токен сохраняется с правами `0600` в `/etc/proxmox-dashboard/update.env`.
Проверить обновления вручную можно командой:
```bash
systemctl start proxmox-dashboard-update.service
systemctl status proxmox-dashboard-update.service
```
Откройте:
```text
@@ -118,6 +320,36 @@ apcaccess status
- `GET /api/cpu` — один JSON-снимок.
- `GET /api/system` — единый JSON-снимок CPU и памяти.
- `GET /api/events` — поток метрик SSE с интервалом две секунды.
- `GET/POST/PUT/DELETE /api/services` — управление проверками сервисов.
- `GET /api/services/history` — семидневная история доступности и отклика.
- `GET/PUT/DELETE /api/settings/maintenance` — состояние, запуск и досрочное завершение плановых работ.
- `GET/DELETE /api/agents` — подключённые гостевые агенты.
- `POST /api/agents/enrollment-token` — одноразовый токен регистрации на 15 минут.
## Maintenance-режим
Во вкладке `Настройки → Maintenance` можно на заданное время отключить все
алерты либо выбрать конкретные сервисы и VM/LXC. Подавленные события не
отправляются по почте и не отображаются как активные, но сохраняются в истории
с причиной плановых работ. После окончания окна обычный мониторинг включается
автоматически.
## Агенты VM и LXC
Read-only агент показывает состояние Linux внутри VM/LXC: load average, память,
корневой раздел, uptime, версию агента, Docker-контейнеры, health-check и число
перезапусков. Агент сам подключается к Dashboard каждые 10 секунд; открывать
входящий порт внутри гостевой системы не нужно.
Откройте `Настройки → Агенты`, укажите адрес Dashboard, доступный из VM/LXC, и
VMID. Кнопка создаст готовую команду установки. Одноразовый токен действует 15
минут и после регистрации заменяется индивидуальным секретом агента.
В первой версии агент не выполняет команды и не перезапускает службы или
контейнеры. Для чтения Docker ему требуется доступ к `/var/run/docker.sock`,
поэтому служба устанавливается от root. Сам Dashboard пока не имеет
авторизации: используйте локальную сеть или HTTPS через доверенный reverse
proxy и не публикуйте агентские API напрямую в интернет.
## Сборка

189
activity.go Normal file
View File

@@ -0,0 +1,189 @@
package main
import (
"encoding/json"
"fmt"
"os"
"os/exec"
"sort"
"strconv"
"strings"
"sync"
"time"
)
type ProxmoxTask struct {
UPID string `json:"upid"`
Node string `json:"node"`
Type string `json:"type"`
ID string `json:"id"`
User string `json:"user"`
Status string `json:"status"`
StartTime int64 `json:"starttime"`
EndTime int64 `json:"endtime"`
Duration int64 `json:"duration"`
}
type GuestSnapshot struct {
VMID int `json:"vmid"`
GuestName string `json:"guestName"`
GuestType string `json:"guestType"`
Name string `json:"name"`
Description string `json:"description"`
CreatedAt int64 `json:"createdAt"`
Running bool `json:"running"`
Parent string `json:"parent,omitempty"`
SizeBytes *uint64 `json:"sizeBytes,omitempty"`
AgeSeconds int64 `json:"ageSeconds"`
}
type ProxmoxActivity struct {
Available bool `json:"available"`
Tasks []ProxmoxTask `json:"tasks"`
Snapshots []GuestSnapshot `json:"snapshots"`
CheckedAt int64 `json:"checkedAt"`
}
type activityCollector struct {
mu sync.RWMutex
value ProxmoxActivity
refreshing bool
}
func newActivityCollector() *activityCollector {
c := &activityCollector{refreshing: true}
go c.refresh()
return c
}
func (c *activityCollector) collect(guests GuestsMetrics) ProxmoxActivity {
c.mu.RLock()
value := c.value
c.mu.RUnlock()
c.mu.Lock()
if !c.refreshing && time.Since(time.Unix(value.CheckedAt, 0)) > 15*time.Second {
c.refreshing = true
go c.refreshWithGuests(guests)
}
c.mu.Unlock()
return value
}
func (c *activityCollector) refresh() { c.refreshWithGuests(readGuests()) }
func (c *activityCollector) refreshWithGuests(guests GuestsMetrics) {
value := readProxmoxActivity(guests)
c.mu.Lock()
c.value = value
c.refreshing = false
c.mu.Unlock()
}
func readProxmoxActivity(guests GuestsMetrics) ProxmoxActivity {
result := ProxmoxActivity{CheckedAt: time.Now().Unix()}
if _, err := exec.LookPath("pvesh"); err != nil {
return result
}
node, err := os.Hostname()
if err != nil {
return result
}
result.Available = true
out := commandOutput(8*time.Second, "pvesh", "get", "/nodes/"+node+"/tasks", "--limit", "100", "--output-format", "json")
if out != "" && json.Unmarshal([]byte(out), &result.Tasks) == nil {
for i := range result.Tasks {
if result.Tasks[i].EndTime > result.Tasks[i].StartTime {
result.Tasks[i].Duration = result.Tasks[i].EndTime - result.Tasks[i].StartTime
}
}
}
for _, guest := range guests.Guests {
if guest.Template != 0 {
continue
}
kind := "qemu"
label := "VM"
if guest.Type != "qemu" {
kind = "lxc"
label = "LXC"
}
out := commandOutput(5*time.Second, "pvesh", "get", fmt.Sprintf("/nodes/%s/%s/%d/snapshot", guest.Node, kind, guest.VMID), "--output-format", "json")
if out == "" {
continue
}
var raw []struct {
Name string `json:"name"`
Description string `json:"description"`
SnapTime int64 `json:"snaptime"`
Running int `json:"running"`
Parent string `json:"parent"`
VMState any `json:"vmstate"`
}
if json.Unmarshal([]byte(out), &raw) != nil {
continue
}
for _, s := range raw {
if s.Name == "current" {
continue
}
item := GuestSnapshot{VMID: guest.VMID, GuestName: guest.Name, GuestType: label, Name: s.Name, Description: s.Description, CreatedAt: s.SnapTime, Running: s.Running != 0, Parent: s.Parent}
if item.CreatedAt > 0 {
item.AgeSeconds = time.Now().Unix() - item.CreatedAt
}
if size := uintValue(s.VMState); size > 1 {
item.SizeBytes = &size
}
result.Snapshots = append(result.Snapshots, item)
}
}
sort.Slice(result.Snapshots, func(i, j int) bool { return result.Snapshots[i].CreatedAt > result.Snapshots[j].CreatedAt })
return result
}
func uintValue(v any) uint64 {
switch n := v.(type) {
case float64:
return uint64(n)
case string:
value, _ := strconv.ParseUint(n, 10, 64)
return value
}
return 0
}
type BackupCheckResult struct {
OK bool `json:"ok"`
Level string `json:"level"`
Message string `json:"message"`
CheckedAt int64 `json:"checkedAt"`
}
func verifyBackup(volid string) BackupCheckResult {
result := BackupCheckResult{CheckedAt: time.Now().Unix(), Level: "metadata"}
if volid == "" || strings.ContainsAny(volid, "\r\n") {
result.Message = "Некорректный идентификатор backup"
return result
}
out, err := exec.Command("pvesm", "path", volid).CombinedOutput()
if err != nil {
result.Message = "Backup не найден в подключённом storage: " + strings.TrimSpace(string(out))
return result
}
path := strings.TrimSpace(string(out))
if strings.HasPrefix(path, "pbs://") {
ctxOutput := commandOutput(20*time.Second, "pvesm", "extractconfig", volid)
if ctxOutput == "" {
result.Message = "Snapshot найден в PBS, но конфигурацию для тестового чтения получить не удалось"
return result
}
result.OK = true
result.Level = "pbs-metadata"
result.Message = "PBS snapshot и его конфигурация успешно прочитаны. Полное восстановление VM не запускалось."
return result
}
info, err := os.Stat(path)
if err != nil {
result.Message = "Метаданные доступны, но локальный файл не открыт: " + err.Error()
return result
}
if info.Size() == 0 {
result.Message = "Файл backup пуст"
return result
}
result.OK = true
result.Level = "readability"
result.Message = fmt.Sprintf("Файл доступен для чтения, размер %d байт. Полное тестовое восстановление не запускалось.", info.Size())
return result
}

9
activity_test.go Normal file
View File

@@ -0,0 +1,9 @@
package main
import "testing"
func TestUintValue(t *testing.T) {
if uintValue("42") != 42 || uintValue(float64(7)) != 7 || uintValue(nil) != 0 {
t.Fatal("unexpected numeric conversion")
}
}

457
agent.go Normal file
View File

@@ -0,0 +1,457 @@
package main
import (
"bytes"
"context"
"crypto/hmac"
"crypto/rand"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"net"
"net/http"
"os"
"path/filepath"
"runtime"
"strconv"
"strings"
"syscall"
"time"
)
const agentOfflineAfter = 45 * time.Second
type agentOptions struct {
Server, EnrollmentToken, Name, ConfigPath string
VMID int
}
type agentConfig struct {
Server string `json:"server"`
ID string `json:"id"`
Secret string `json:"secret"`
Name string `json:"name"`
VMID int `json:"vmid"`
}
type AgentContainer struct {
ID string `json:"id"`
Name string `json:"name"`
Image string `json:"image"`
State string `json:"state"`
Status string `json:"status"`
Health string `json:"health,omitempty"`
Restarts int64 `json:"restarts"`
CPUPercent float64 `json:"cpuPercent"`
MemoryUsage uint64 `json:"memoryUsage"`
Ports []string `json:"ports,omitempty"`
}
type AgentReport struct {
Hostname string `json:"hostname"`
MachineID string `json:"machineId"`
Version string `json:"version"`
OS string `json:"os"`
Arch string `json:"arch"`
UptimeSeconds int64 `json:"uptimeSeconds"`
Load1 float64 `json:"load1"`
CPUPercent float64 `json:"cpuPercent"`
MemoryTotal uint64 `json:"memoryTotal"`
MemoryUsed uint64 `json:"memoryUsed"`
RootTotal uint64 `json:"rootTotal"`
RootUsed uint64 `json:"rootUsed"`
DockerAvailable bool `json:"dockerAvailable"`
DockerError string `json:"dockerError,omitempty"`
Containers []AgentContainer `json:"containers"`
CollectedAt int64 `json:"collectedAt"`
}
type ManagedAgent struct {
ID string `json:"id"`
Name string `json:"name"`
VMID int `json:"vmid"`
Hostname string `json:"hostname"`
Version string `json:"version"`
LastSeen int64 `json:"lastSeen"`
EnrolledAt int64 `json:"enrolledAt"`
Online bool `json:"online"`
RemoteAddress string `json:"remoteAddress"`
Report AgentReport `json:"report"`
}
type agentEnrollmentRequest struct {
Token, Name, Hostname, MachineID string
VMID int
}
func randomHex(bytesCount int) (string, error) {
b := make([]byte, bytesCount)
if _, err := rand.Read(b); err != nil {
return "", err
}
return hex.EncodeToString(b), nil
}
func tokenHash(value string) string {
sum := sha256.Sum256([]byte(value))
return hex.EncodeToString(sum[:])
}
func (s *Store) CreateAgentEnrollmentToken() (string, error) {
token, err := randomHex(24)
if err != nil {
return "", err
}
_, err = s.db.Exec(`INSERT INTO agent_enrollment_tokens(token_hash,created_at,expires_at) VALUES(?,?,?)`, tokenHash(token), time.Now().Unix(), time.Now().Add(15*time.Minute).Unix())
return token, err
}
func (s *Store) EnrollAgent(req agentEnrollmentRequest) (agentConfig, error) {
tx, err := s.db.Begin()
if err != nil {
return agentConfig{}, err
}
defer tx.Rollback()
var expires int64
if err = tx.QueryRow(`SELECT expires_at FROM agent_enrollment_tokens WHERE token_hash=? AND used_at IS NULL`, tokenHash(req.Token)).Scan(&expires); err != nil || expires < time.Now().Unix() {
return agentConfig{}, errors.New("одноразовый токен недействителен или истёк")
}
id, err := randomHex(12)
if err != nil {
return agentConfig{}, err
}
secret, err := randomHex(32)
if err != nil {
return agentConfig{}, err
}
name := strings.TrimSpace(req.Name)
if name == "" {
name = req.Hostname
}
now := time.Now().Unix()
if _, err = tx.Exec(`INSERT INTO agents(id,name,vmid,hostname,machine_id,secret_hash,enrolled_at,last_seen,report_json,remote_address) VALUES(?,?,?,?,?,?,?,?,?,?)`, id, name, req.VMID, req.Hostname, req.MachineID, tokenHash(secret), now, now, `{}`, ""); err != nil {
return agentConfig{}, err
}
if _, err = tx.Exec(`UPDATE agent_enrollment_tokens SET used_at=? WHERE token_hash=?`, now, tokenHash(req.Token)); err != nil {
return agentConfig{}, err
}
if err = tx.Commit(); err != nil {
return agentConfig{}, err
}
return agentConfig{ID: id, Secret: secret, Name: name, VMID: req.VMID}, nil
}
func (s *Store) SaveAgentReport(id, secret, remote string, report AgentReport) error {
var expected string
if s.db.QueryRow(`SELECT secret_hash FROM agents WHERE id=?`, id).Scan(&expected) != nil || !hmac.Equal([]byte(expected), []byte(tokenHash(secret))) {
return errors.New("агент не авторизован")
}
data, _ := json.Marshal(report)
_, err := s.db.Exec(`UPDATE agents SET hostname=?,version=?,last_seen=?,report_json=?,remote_address=? WHERE id=?`, report.Hostname, report.Version, time.Now().Unix(), string(data), remote, id)
return err
}
func (s *Store) Agents() ([]ManagedAgent, error) {
rows, err := s.db.Query(`SELECT id,name,vmid,hostname,version,enrolled_at,last_seen,report_json,remote_address FROM agents ORDER BY name`)
if err != nil {
return nil, err
}
defer rows.Close()
result := []ManagedAgent{}
for rows.Next() {
var a ManagedAgent
var raw string
if err = rows.Scan(&a.ID, &a.Name, &a.VMID, &a.Hostname, &a.Version, &a.EnrolledAt, &a.LastSeen, &raw, &a.RemoteAddress); err != nil {
return nil, err
}
_ = json.Unmarshal([]byte(raw), &a.Report)
a.Online = time.Since(time.Unix(a.LastSeen, 0)) < agentOfflineAfter
result = append(result, a)
}
return result, rows.Err()
}
func (s *Store) DeleteAgent(id string) error {
_, err := s.db.Exec(`DELETE FROM agents WHERE id=?`, id)
return err
}
func runGuestAgent(opts agentOptions) error {
config, err := loadAgentConfig(opts.ConfigPath)
if err != nil && !os.IsNotExist(err) {
return err
}
if opts.Server != "" {
config.Server = strings.TrimRight(opts.Server, "/")
}
if opts.Name != "" {
config.Name = opts.Name
}
if opts.VMID > 0 {
config.VMID = opts.VMID
}
if config.Server == "" {
return errors.New("укажите --agent-server")
}
if config.ID == "" {
if opts.EnrollmentToken == "" {
return errors.New("для первого запуска нужен --agent-enroll-token")
}
config, err = enrollRemote(config, opts.EnrollmentToken)
if err != nil {
return err
}
if err = saveAgentConfig(opts.ConfigPath, config); err != nil {
return err
}
}
client := &http.Client{Timeout: 12 * time.Second}
for {
report := collectAgentReport()
data, _ := json.Marshal(report)
req, _ := http.NewRequest(http.MethodPost, config.Server+"/api/agent/report", bytes.NewReader(data))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "Bearer "+config.ID+"."+config.Secret)
response, requestErr := client.Do(req)
if requestErr == nil {
io.Copy(io.Discard, response.Body)
response.Body.Close()
if response.StatusCode >= 300 {
requestErr = fmt.Errorf("сервер вернул %s", response.Status)
}
}
if requestErr != nil {
fmt.Fprintf(os.Stderr, "agent: %v\n", requestErr)
}
time.Sleep(10 * time.Second)
}
}
func enrollRemote(config agentConfig, token string) (agentConfig, error) {
hostname, _ := os.Hostname()
machineID, _ := os.ReadFile("/etc/machine-id")
payload := agentEnrollmentRequest{Token: token, Name: config.Name, VMID: config.VMID, Hostname: hostname, MachineID: strings.TrimSpace(string(machineID))}
data, _ := json.Marshal(payload)
response, err := http.Post(config.Server+"/api/agent/enroll", "application/json", bytes.NewReader(data))
if err != nil {
return config, err
}
defer response.Body.Close()
if response.StatusCode != http.StatusOK {
message, _ := io.ReadAll(io.LimitReader(response.Body, 4096))
return config, fmt.Errorf("регистрация: %s", strings.TrimSpace(string(message)))
}
var credentials agentConfig
if err = json.NewDecoder(response.Body).Decode(&credentials); err != nil {
return config, err
}
credentials.Server = config.Server
return credentials, nil
}
func loadAgentConfig(path string) (agentConfig, error) {
data, err := os.ReadFile(path)
if err != nil {
return agentConfig{}, err
}
var c agentConfig
err = json.Unmarshal(data, &c)
return c, err
}
func saveAgentConfig(path string, c agentConfig) error {
if err := os.MkdirAll(filepath.Dir(path), 0700); err != nil {
return err
}
data, _ := json.MarshalIndent(c, "", " ")
return os.WriteFile(path, data, 0600)
}
func collectAgentReport() AgentReport {
hostname, _ := os.Hostname()
machineID, _ := os.ReadFile("/etc/machine-id")
report := AgentReport{Hostname: hostname, MachineID: strings.TrimSpace(string(machineID)), Version: version, OS: runtime.GOOS, Arch: runtime.GOARCH, CollectedAt: time.Now().Unix()}
if data, err := os.ReadFile("/proc/uptime"); err == nil {
fields := strings.Fields(string(data))
if len(fields) > 0 {
value, _ := strconv.ParseFloat(fields[0], 64)
report.UptimeSeconds = int64(value)
}
}
if data, err := os.ReadFile("/proc/loadavg"); err == nil {
fields := strings.Fields(string(data))
if len(fields) > 0 {
report.Load1, _ = strconv.ParseFloat(fields[0], 64)
}
}
report.CPUPercent = sampleAgentCPU()
if data, err := os.ReadFile("/proc/meminfo"); err == nil {
values := map[string]uint64{}
for _, line := range strings.Split(string(data), "\n") {
f := strings.Fields(line)
if len(f) >= 2 {
v, _ := strconv.ParseUint(f[1], 10, 64)
values[strings.TrimSuffix(f[0], ":")] = v * 1024
}
}
report.MemoryTotal = values["MemTotal"]
available := values["MemAvailable"]
if report.MemoryTotal > available {
report.MemoryUsed = report.MemoryTotal - available
}
}
var stat syscall.Statfs_t
if syscall.Statfs("/", &stat) == nil {
report.RootTotal = stat.Blocks * uint64(stat.Bsize)
free := stat.Bavail * uint64(stat.Bsize)
if report.RootTotal > free {
report.RootUsed = report.RootTotal - free
}
}
report.Containers, report.DockerError = collectDockerContainers()
report.DockerAvailable = report.DockerError == ""
return report
}
func sampleAgentCPU() float64 {
total1, idle1 := readAgentCPU()
time.Sleep(100 * time.Millisecond)
total2, idle2 := readAgentCPU()
if total2 <= total1 {
return 0
}
totalDelta, idleDelta := total2-total1, idle2-idle1
return float64(totalDelta-idleDelta) / float64(totalDelta) * 100
}
func readAgentCPU() (uint64, uint64) {
data, err := os.ReadFile("/proc/stat")
if err != nil {
return 0, 0
}
line := strings.SplitN(string(data), "\n", 2)[0]
fields := strings.Fields(line)
if len(fields) < 5 {
return 0, 0
}
var total uint64
values := make([]uint64, 0, len(fields)-1)
for _, field := range fields[1:] {
v, _ := strconv.ParseUint(field, 10, 64)
values = append(values, v)
total += v
}
idle := values[3]
if len(values) > 4 {
idle += values[4]
}
return total, idle
}
func collectDockerContainers() ([]AgentContainer, string) {
if _, err := os.Stat("/var/run/docker.sock"); err != nil {
return []AgentContainer{}, "Docker socket недоступен"
}
transport := &http.Transport{DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
return (&net.Dialer{}).DialContext(ctx, "unix", "/var/run/docker.sock")
}}
client := &http.Client{Transport: transport, Timeout: 5 * time.Second}
response, err := client.Get("http://docker/containers/json?all=1")
if err != nil {
return nil, err.Error()
}
defer response.Body.Close()
if response.StatusCode != 200 {
return nil, response.Status
}
var raw []struct {
ID, Image, State, Status string
Names []string
Ports []struct {
IP string
PrivatePort, PublicPort int
Type string
}
}
if err = json.NewDecoder(response.Body).Decode(&raw); err != nil {
return nil, err.Error()
}
result := make([]AgentContainer, 0, len(raw))
for _, v := range raw {
name := strings.TrimPrefix(firstString(v.Names), "/")
item := AgentContainer{ID: shortID(v.ID), Name: name, Image: v.Image, State: v.State, Status: v.Status}
for _, p := range v.Ports {
if p.PublicPort > 0 {
item.Ports = append(item.Ports, fmt.Sprintf("%d:%d/%s", p.PublicPort, p.PrivatePort, p.Type))
}
}
inspect, inspectErr := client.Get("http://docker/containers/" + v.ID + "/json")
if inspectErr == nil {
var detail struct {
RestartCount int64
State struct{ Health *struct{ Status string } }
}
if json.NewDecoder(inspect.Body).Decode(&detail) == nil {
item.Restarts = detail.RestartCount
if detail.State.Health != nil {
item.Health = detail.State.Health.Status
}
}
inspect.Body.Close()
}
statsResponse, statsErr := client.Get("http://docker/containers/" + v.ID + "/stats?stream=false")
if statsErr == nil {
var stats struct {
CPUStats struct {
CPUUsage struct {
TotalUsage uint64 `json:"total_usage"`
} `json:"cpu_usage"`
SystemCPUUsage uint64 `json:"system_cpu_usage"`
OnlineCPUs uint64 `json:"online_cpus"`
} `json:"cpu_stats"`
PreCPUStats struct {
CPUUsage struct {
TotalUsage uint64 `json:"total_usage"`
} `json:"cpu_usage"`
SystemCPUUsage uint64 `json:"system_cpu_usage"`
} `json:"precpu_stats"`
MemoryStats struct {
Usage uint64 `json:"usage"`
Stats map[string]uint64 `json:"stats"`
} `json:"memory_stats"`
}
if json.NewDecoder(statsResponse.Body).Decode(&stats) == nil {
cpuDelta := stats.CPUStats.CPUUsage.TotalUsage - stats.PreCPUStats.CPUUsage.TotalUsage
systemDelta := stats.CPUStats.SystemCPUUsage - stats.PreCPUStats.SystemCPUUsage
cpus := stats.CPUStats.OnlineCPUs
if cpus == 0 {
cpus = 1
}
if systemDelta > 0 {
item.CPUPercent = float64(cpuDelta) / float64(systemDelta) * float64(cpus) * 100
}
item.MemoryUsage = stats.MemoryStats.Usage
if cache := stats.MemoryStats.Stats["inactive_file"]; item.MemoryUsage > cache {
item.MemoryUsage -= cache
}
}
statsResponse.Body.Close()
}
result = append(result, item)
}
return result, ""
}
func firstString(v []string) string {
if len(v) > 0 {
return v[0]
}
return ""
}
func shortID(v string) string {
if len(v) > 12 {
return v[:12]
}
return v
}

57
agent_test.go Normal file
View File

@@ -0,0 +1,57 @@
package main
import (
"testing"
"time"
)
func TestAgentEnrollmentTokenIsSingleUse(t *testing.T) {
store, err := openStore(":memory:")
if err != nil {
t.Fatal(err)
}
defer store.Close()
token, err := store.CreateAgentEnrollmentToken()
if err != nil {
t.Fatal(err)
}
credentials, err := store.EnrollAgent(agentEnrollmentRequest{Token: token, Name: "docker-01", Hostname: "vm105", VMID: 105})
if err != nil {
t.Fatal(err)
}
if credentials.ID == "" || credentials.Secret == "" {
t.Fatal("credentials were not generated")
}
if _, err = store.EnrollAgent(agentEnrollmentRequest{Token: token, Hostname: "duplicate"}); err == nil {
t.Fatal("token was accepted twice")
}
}
func TestAgentReportAuthenticationAndOfflineState(t *testing.T) {
store, err := openStore(":memory:")
if err != nil {
t.Fatal(err)
}
defer store.Close()
token, _ := store.CreateAgentEnrollmentToken()
credentials, _ := store.EnrollAgent(agentEnrollmentRequest{Token: token, Name: "docker", Hostname: "vm", VMID: 101})
report := AgentReport{Hostname: "vm", Version: "test", DockerAvailable: true, Containers: []AgentContainer{{ID: "abc", Name: "web", State: "running"}}}
if err = store.SaveAgentReport(credentials.ID, "wrong", "127.0.0.1", report); err == nil {
t.Fatal("invalid secret accepted")
}
if err = store.SaveAgentReport(credentials.ID, credentials.Secret, "127.0.0.1", report); err != nil {
t.Fatal(err)
}
agents, err := store.Agents()
if err != nil {
t.Fatal(err)
}
if len(agents) != 1 || !agents[0].Online || len(agents[0].Report.Containers) != 1 {
t.Fatalf("unexpected agents: %#v", agents)
}
_, _ = store.db.Exec(`UPDATE agents SET last_seen=? WHERE id=?`, time.Now().Add(-time.Minute).Unix(), credentials.ID)
agents, _ = store.Agents()
if agents[0].Online {
t.Fatal("stale agent is online")
}
}

230
alert_events.go Normal file
View File

@@ -0,0 +1,230 @@
package main
import (
"database/sql"
"time"
)
type AlertEvent struct {
EventID int64 `json:"eventId"`
ID string `json:"id"`
Severity string `json:"severity"`
Source string `json:"source"`
Title string `json:"title"`
Message string `json:"message"`
FirstSeen int64 `json:"firstSeen"`
LastSeen int64 `json:"lastSeen"`
ResolvedAt *int64 `json:"resolvedAt,omitempty"`
AcknowledgedAt *int64 `json:"acknowledgedAt,omitempty"`
Planned bool `json:"planned"`
MaintenanceReason string `json:"maintenanceReason,omitempty"`
}
type RecoveryEvent struct {
EventID int64
ID, Severity, Source, Title, Message string
}
func (s *Store) SyncAlerts(alerts []Alert) ([]Alert, error) {
return s.SyncAlertsWithMaintenance(alerts, nil, "")
}
func (s *Store) SyncAlertsWithMaintenance(alerts, plannedAlerts []Alert, maintenanceReason string) ([]Alert, error) {
tx, err := s.db.Begin()
if err != nil {
return alerts, err
}
defer tx.Rollback()
now := time.Now().Unix()
rows, err := tx.Query(`SELECT id,alert_key,planned FROM alert_events WHERE resolved_at IS NULL`)
if err != nil {
return alerts, err
}
active := map[string]int64{}
for rows.Next() {
var id int64
var key string
var planned bool
if err := rows.Scan(&id, &key, &planned); err != nil {
rows.Close()
return alerts, err
}
active[alertEventKey(key, planned)] = id
}
rows.Close()
seen := map[string]bool{}
syncGroup := func(group []Alert, planned bool) error {
for _, alert := range group {
key := alertEventKey(alert.ID, planned)
seen[key] = true
reason := ""
if planned {
reason = maintenanceReason
}
if id, ok := active[key]; ok {
_, err = tx.Exec(`UPDATE alert_events SET severity=?,source=?,title=?,message=?,last_seen=?,maintenance_reason=? WHERE id=?`, alert.Severity, alert.Source, alert.Title, alert.Message, now, reason, id)
} else {
result, insertErr := tx.Exec(`INSERT INTO alert_events(alert_key,severity,source,title,message,first_seen,last_seen,planned,maintenance_reason) VALUES(?,?,?,?,?,?,?,?,?)`, alert.ID, alert.Severity, alert.Source, alert.Title, alert.Message, now, now, planned, reason)
if insertErr != nil {
return insertErr
}
active[key], _ = result.LastInsertId()
}
if err != nil {
return err
}
}
return nil
}
if err = syncGroup(alerts, false); err != nil {
return alerts, err
}
if err = syncGroup(plannedAlerts, true); err != nil {
return alerts, err
}
for key, id := range active {
if !seen[key] {
if _, err = tx.Exec(`UPDATE alert_events SET resolved_at=? WHERE id=?`, now, id); err != nil {
return alerts, err
}
}
}
if _, err = tx.Exec(`DELETE FROM alert_events WHERE (resolved_at IS NOT NULL OR acknowledged_at IS NOT NULL) AND last_seen < ?`, now-int64(30*24*time.Hour/time.Second)); err != nil {
return alerts, err
}
if err = tx.Commit(); err != nil {
return alerts, err
}
rows, err = s.db.Query(`SELECT id,alert_key,severity,source,title,message,first_seen,last_seen FROM alert_events WHERE resolved_at IS NULL AND acknowledged_at IS NULL AND planned=0 ORDER BY CASE severity WHEN 'critical' THEN 0 WHEN 'warning' THEN 1 ELSE 2 END,first_seen DESC`)
if err != nil {
return alerts, err
}
defer rows.Close()
var visible []Alert
for rows.Next() {
var a Alert
if err = rows.Scan(&a.EventID, &a.ID, &a.Severity, &a.Source, &a.Title, &a.Message, &a.FirstSeen, &a.LastSeen); err != nil {
return alerts, err
}
visible = append(visible, a)
}
return visible, rows.Err()
}
func alertEventKey(id string, planned bool) string {
if planned {
return id + "\x00planned"
}
return id + "\x00active"
}
func (s *Store) AcknowledgeAlert(eventID int64) error {
_, err := s.db.Exec(`UPDATE alert_events SET acknowledged_at=? WHERE id=? AND acknowledged_at IS NULL`, time.Now().Unix(), eventID)
return err
}
func (s *Store) AlertHistory(limit int) ([]AlertEvent, error) {
if limit < 1 || limit > 500 {
limit = 200
}
rows, err := s.db.Query(`SELECT id,alert_key,severity,source,title,message,first_seen,last_seen,resolved_at,acknowledged_at,planned,maintenance_reason FROM alert_events ORDER BY first_seen DESC LIMIT ?`, limit)
if err != nil {
return nil, err
}
defer rows.Close()
var result []AlertEvent
for rows.Next() {
var e AlertEvent
var resolved, acked sql.NullInt64
if err = rows.Scan(&e.EventID, &e.ID, &e.Severity, &e.Source, &e.Title, &e.Message, &e.FirstSeen, &e.LastSeen, &resolved, &acked, &e.Planned, &e.MaintenanceReason); err != nil {
return nil, err
}
if resolved.Valid {
e.ResolvedAt = &resolved.Int64
}
if acked.Valid {
e.AcknowledgedAt = &acked.Int64
}
result = append(result, e)
}
return result, rows.Err()
}
func (s *Store) PendingEmailAlerts() ([]Alert, error) {
rows, err := s.db.Query(`SELECT id,alert_key,severity,source,title,message FROM alert_events WHERE resolved_at IS NULL AND acknowledged_at IS NULL AND email_sent_at IS NULL AND planned=0 ORDER BY first_seen`)
if err != nil {
return nil, err
}
defer rows.Close()
var result []Alert
for rows.Next() {
var a Alert
if err = rows.Scan(&a.EventID, &a.ID, &a.Severity, &a.Source, &a.Title, &a.Message); err != nil {
return nil, err
}
result = append(result, a)
}
return result, rows.Err()
}
func (s *Store) MarkAlertsEmailed(alerts []Alert) error {
tx, err := s.db.Begin()
if err != nil {
return err
}
defer tx.Rollback()
now := time.Now().Unix()
for _, a := range alerts {
if _, err = tx.Exec(`UPDATE alert_events SET email_sent_at=? WHERE id=?`, now, a.EventID); err != nil {
return err
}
}
return tx.Commit()
}
func (s *Store) PendingImmediateAlerts() ([]Alert, error) {
rows, err := s.db.Query(`SELECT id,alert_key,severity,source,title,message FROM alert_events WHERE resolved_at IS NULL AND acknowledged_at IS NULL AND email_sent_at IS NULL AND planned=0 AND (severity='critical' OR alert_key='ups-battery') ORDER BY first_seen`)
if err != nil {
return nil, err
}
defer rows.Close()
var out []Alert
for rows.Next() {
var a Alert
if err = rows.Scan(&a.EventID, &a.ID, &a.Severity, &a.Source, &a.Title, &a.Message); err != nil {
return nil, err
}
out = append(out, a)
}
return out, rows.Err()
}
func (s *Store) PendingRecoveries() ([]RecoveryEvent, error) {
rows, err := s.db.Query(`SELECT id,alert_key,severity,source,title,message FROM alert_events WHERE resolved_at IS NOT NULL AND email_sent_at IS NOT NULL AND recovery_sent_at IS NULL AND planned=0 ORDER BY resolved_at`)
if err != nil {
return nil, err
}
defer rows.Close()
var out []RecoveryEvent
for rows.Next() {
var v RecoveryEvent
if err = rows.Scan(&v.EventID, &v.ID, &v.Severity, &v.Source, &v.Title, &v.Message); err != nil {
return nil, err
}
out = append(out, v)
}
return out, rows.Err()
}
func (s *Store) MarkRecoveriesEmailed(items []RecoveryEvent) error {
tx, err := s.db.Begin()
if err != nil {
return err
}
defer tx.Rollback()
now := time.Now().Unix()
for _, v := range items {
if _, err = tx.Exec(`UPDATE alert_events SET recovery_sent_at=? WHERE id=?`, now, v.EventID); err != nil {
return err
}
}
return tx.Commit()
}

58
alert_events_test.go Normal file
View File

@@ -0,0 +1,58 @@
package main
import "testing"
func TestAlertLifecycle(t *testing.T) {
store, err := openStore(":memory:")
if err != nil {
t.Fatal(err)
}
defer store.Close()
alert := Alert{ID: "disk", Severity: "warning", Source: "Диски", Title: "SMART", Message: "Проверьте диск"}
visible, err := store.SyncAlerts([]Alert{alert})
if err != nil || len(visible) != 1 || visible[0].EventID == 0 || visible[0].FirstSeen == 0 || visible[0].LastSeen == 0 {
t.Fatalf("unexpected new alert: %+v %v", visible, err)
}
firstID := visible[0].EventID
if err = store.AcknowledgeAlert(firstID); err != nil {
t.Fatal(err)
}
visible, err = store.SyncAlerts([]Alert{alert})
if err != nil || len(visible) != 0 {
t.Fatalf("acknowledged alert became visible: %+v %v", visible, err)
}
if _, err = store.SyncAlerts(nil); err != nil {
t.Fatal(err)
}
visible, err = store.SyncAlerts([]Alert{alert})
if err != nil || len(visible) != 1 || visible[0].EventID == firstID {
t.Fatalf("recurring alert was not new: %+v %v", visible, err)
}
history, err := store.AlertHistory(20)
if err != nil || len(history) != 2 {
t.Fatalf("unexpected history: %+v %v", history, err)
}
}
func TestPendingEmailAlerts(t *testing.T) {
store, err := openStore(":memory:")
if err != nil {
t.Fatal(err)
}
defer store.Close()
visible, err := store.SyncAlerts([]Alert{{ID: "cpu", Severity: "critical", Source: "CPU", Title: "Load"}})
if err != nil {
t.Fatal(err)
}
pending, err := store.PendingEmailAlerts()
if err != nil || len(pending) != 1 {
t.Fatalf("unexpected pending: %+v %v", pending, err)
}
if err = store.MarkAlertsEmailed(visible); err != nil {
t.Fatal(err)
}
pending, err = store.PendingEmailAlerts()
if err != nil || len(pending) != 0 {
t.Fatalf("emailed alert repeated: %+v %v", pending, err)
}
}

223
alerts.go
View File

@@ -8,11 +8,14 @@ import (
)
type Alert struct {
ID string `json:"id"`
Severity string `json:"severity"`
Source string `json:"source"`
Title string `json:"title"`
Message string `json:"message"`
EventID int64 `json:"eventId,omitempty"`
ID string `json:"id"`
Severity string `json:"severity"`
Source string `json:"source"`
Title string `json:"title"`
Message string `json:"message"`
FirstSeen int64 `json:"firstSeen,omitempty"`
LastSeen int64 `json:"lastSeen,omitempty"`
}
func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Alert {
@@ -20,6 +23,17 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
add := func(id, severity, source, title, message string) {
alerts = append(alerts, Alert{ID: id, Severity: severity, Source: source, Title: title, Message: message})
}
if metrics.Update.State == "available" {
add("dashboard-update", "info", "Dashboard", "Доступно обновление "+metrics.Update.AvailableVersion, "Режим обновлений: только уведомлять. Установите обновление с сервера или включите автоматический режим.")
} else if metrics.Update.State == "error" {
add("dashboard-update-error", "warning", "Dashboard", "Не удалось проверить обновления", metrics.Update.Message)
}
if metrics.Maintenance.RebootRequired || metrics.Maintenance.NewKernelAvailable {
add("maintenance-reboot", "warning", "Proxmox", "Требуется перезагрузка хоста", "Установлено новое ядро или система запросила перезагрузку.")
}
if metrics.Maintenance.AvailableUpdates > 0 {
add("maintenance-updates", "info", "Proxmox", "Доступны обновления системы", fmt.Sprintf("Пакетов для обновления: %d.", metrics.Maintenance.AvailableUpdates))
}
if metrics.CPU.UsagePercent >= thresholds.CPUCritical {
add("cpu-usage", "critical", "CPU", "Очень высокая загрузка CPU", fmt.Sprintf("Текущая загрузка %.1f%%.", metrics.CPU.UsagePercent))
@@ -49,6 +63,14 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
add("root-storage", "warning", "Storage", "Мало места на системном разделе", fmt.Sprintf("Раздел / заполнен на %.1f%%.", metrics.Storage.UsagePercent))
}
for _, disk := range metrics.Disks {
diskExcluded := false
for _, value := range strings.Split(thresholds.DiskIOExcluded, ",") {
value = strings.TrimSpace(value)
if value != "" && (strings.EqualFold(value, disk.Name) || strings.EqualFold(value, disk.Serial)) {
diskExcluded = true
break
}
}
if disk.SMARTStatus == "Ошибка" {
add("disk-smart-"+disk.Name, "critical", "Диски", disk.Model+": ошибка SMART", joinReasons(disk.AttentionReasons, "SMART сообщает о неисправности диска."))
} else if disk.SMARTStatus == "Требует внимания" {
@@ -63,6 +85,26 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
add("disk-temp-"+disk.Name, "warning", "Диски", disk.Model+": высокая температура", fmt.Sprintf("Температура %.1f °C.", *disk.Temperature))
}
}
if len(disk.CounterChanges) > 0 {
add("disk-growth-"+disk.Name, "critical", "Диски", disk.Model+": ухудшились SMART-счётчики", strings.Join(disk.CounterChanges, " · "))
}
if !diskExcluded && disk.IOPressureSeconds >= thresholds.DiskIODuration {
severity := "warning"
if disk.Utilization >= 98 || disk.LatencyMs >= 100 {
severity = "critical"
}
cause := disk.IOCause
if cause == "" {
cause = "источник пока не определён"
}
add("disk-io-pressure-"+disk.Name, severity, "Диски", disk.Model+": длительная I/O-нагрузка", fmt.Sprintf("%s занят на %.1f%%; вероятная причина: %s (уверенность %d%%). %.0f IOPS, %.1f MB/s, latency %.1f ms, очередь %.1f.", disk.Name, disk.Utilization, cause, disk.IOConfidence, disk.ReadIOPS+disk.WriteIOPS, (disk.ReadBytesPerSec+disk.WriteBytesPerSec)/1048576, disk.LatencyMs, disk.AverageQueue))
}
if disk.SelfTestNeverRun {
add("disk-selftest-"+disk.Name, "warning", "Диски", disk.Model+": нет завершённого SMART self-test", "Короткий тест будет автоматически запущен; проверьте его результат после завершения.")
}
}
for _, missing := range metrics.MissingDisks {
add("disk-missing-"+missing, "critical", "Диски", "Физический диск исчез", missing+" больше не обнаруживается в /sys/block.")
}
for _, pool := range metrics.ZFS.Pools {
@@ -77,6 +119,25 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
if strings.Contains(strings.ToLower(pool.Scan), "in progress") {
add("zfs-scan-"+pool.Name, "info", "ZFS", pool.Name+": выполняется scrub/resilver", pool.Scan)
}
if pool.ReadErrors+pool.WriteErrors+pool.ChecksumErrors > 0 {
add("zfs-vdev-errors-"+pool.Name, "critical", "ZFS", pool.Name+": ошибки чтения/записи/checksum", fmt.Sprintf("read=%d, write=%d, checksum=%d.", pool.ReadErrors, pool.WriteErrors, pool.ChecksumErrors))
}
if len(pool.ErrorChanges) > 0 {
add("zfs-new-errors-"+pool.Name, "critical", "ZFS", pool.Name+": появились новые ошибки", strings.Join(pool.ErrorChanges, " · "))
}
if pool.ScrubTooOld {
add("zfs-scrub-old-"+pool.Name, "warning", "ZFS", pool.Name+": scrub просрочен", "Scrub не выполнялся более 35 дней либо ни разу не запускался.")
}
}
for _, dataset := range metrics.ZFS.Datasets {
if dataset.CapacityPercent >= 90 {
add("zfs-dataset-"+dataset.Name, "critical", "ZFS", dataset.Name+": dataset почти заполнен", fmt.Sprintf("Заполнение %.1f%%, доступно %.1f GB.", dataset.CapacityPercent, float64(dataset.AvailableBytes)/1073741824))
} else if dataset.CapacityPercent >= 80 {
add("zfs-dataset-"+dataset.Name, "warning", "ZFS", dataset.Name+": мало свободного места", fmt.Sprintf("Заполнение %.1f%%.", dataset.CapacityPercent))
}
}
if metrics.ZFS.ARC.Available && metrics.ZFS.ARC.Hits+metrics.ZFS.ARC.Misses > 10000 && metrics.ZFS.ARC.HitRatio < 70 {
add("zfs-arc-hit", "warning", "ZFS", "Низкий ARC hit ratio", fmt.Sprintf("Попадания ARC: %.1f%%.", metrics.ZFS.ARC.HitRatio))
}
if metrics.UPS.Available {
@@ -89,6 +150,14 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
if metrics.UPS.ChargePercent != nil && *metrics.UPS.ChargePercent < thresholds.UPSChargeCritical {
add("ups-charge", "critical", "UPS", "Низкий заряд батареи", fmt.Sprintf("Осталось %.1f%%.", *metrics.UPS.ChargePercent))
}
if metrics.UPS.ReplaceBattery {
add("ups-replace-battery", "critical", "UPS", "UPS требует заменить батарею", "NUT/APC сообщает флаг Replace Battery.")
}
if metrics.UPS.BatteryAgeDays > 3*365 {
add("ups-battery-age", "warning", "UPS", "Батарея UPS старше трёх лет", fmt.Sprintf("Возраст батареи примерно %d дней.", metrics.UPS.BatteryAgeDays))
}
} else if metrics.UPS.Source != "" {
add("ups-communication", "warning", "UPS", "Нет связи с UPS", "NUT и apcupsd не вернули данные. Проверьте USB/сетевое подключение и службу UPS.")
}
if metrics.Backups.Available && metrics.Guests.Available {
@@ -128,6 +197,105 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
add("guests-stopped", "info", "VM/LXC", "Есть остановленные гости", fmt.Sprintf("Остановлено VM/LXC: %d. Это уведомление информационное.", stopped))
}
}
for _, agent := range metrics.Agents {
vmSuffix := fmt.Sprintf("-%d", agent.VMID)
if !agent.Online {
add("agent-offline-"+agent.ID+vmSuffix, "critical", "Агенты", agent.Name+": агент не отвечает", fmt.Sprintf("Последний heartbeat: %s назад. VMID %d.", time.Since(time.Unix(agent.LastSeen, 0)).Round(time.Second), agent.VMID))
continue
}
if agent.Report.DockerAvailable {
for _, container := range agent.Report.Containers {
if container.State != "running" {
add("agent-container-stopped-"+agent.ID+"-"+container.ID+vmSuffix, "warning", "Docker", agent.Name+": контейнер "+container.Name+" остановлен", fmt.Sprintf("%s · VMID %d.", container.Status, agent.VMID))
} else if container.Health == "unhealthy" {
add("agent-container-health-"+agent.ID+"-"+container.ID+vmSuffix, "critical", "Docker", agent.Name+": контейнер "+container.Name+" unhealthy", fmt.Sprintf("Docker health-check не проходит · VMID %d.", agent.VMID))
}
}
}
}
for _, service := range metrics.Services.Services {
for _, endpoint := range service.Endpoints {
if !endpoint.Up {
add(fmt.Sprintf("service-%d-%s", service.ID, endpoint.Kind), "critical", "Сервисы", service.Name+": недоступен "+strings.ToLower(endpoint.Label), endpoint.Error)
} else if endpoint.TLSDays != nil && *endpoint.TLSDays < 0 {
add(fmt.Sprintf("service-tls-%d-%s", service.ID, endpoint.Kind), "critical", "Сервисы", service.Name+": сертификат истёк", fmt.Sprintf("%s: истёк %d дн. назад.", endpoint.Label, -*endpoint.TLSDays))
} else if endpoint.TLSDays != nil && *endpoint.TLSDays < 14 {
add(fmt.Sprintf("service-tls-%d-%s", service.ID, endpoint.Kind), "warning", "Сервисы", service.Name+": сертификат скоро истечёт", fmt.Sprintf("%s: осталось %d дн.", endpoint.Label, *endpoint.TLSDays))
}
}
}
if metrics.SystemHealth.FailedUnits > 0 {
add("systemd-failed", "critical", "Система", "Есть упавшие systemd-службы", fmt.Sprintf("Не запущено служб: %d.", metrics.SystemHealth.FailedUnits))
}
if metrics.SystemHealth.KernelErrors > 0 {
add("kernel-errors", "warning", "Система", "Ошибки ядра за 24 часа", fmt.Sprintf("Найдено сообщений: %d.", metrics.SystemHealth.KernelErrors))
}
var networkErrors uint64
for _, iface := range metrics.Network.Interfaces {
networkErrors += iface.ReceiveErrors + iface.TransmitErrors + iface.ReceiveDrops + iface.TransmitDrops
}
if networkErrors > 0 {
add("network-errors", "warning", "Сеть", "Сетевые ошибки и потери пакетов", fmt.Sprintf("Суммарный счётчик интерфейсов: %d.", networkErrors))
}
if !metrics.Network.Health.CheckedAt.IsZero() && !metrics.Network.Health.GatewayUp {
add("network-gateway", "critical", "Сеть", "Недоступен локальный шлюз", firstNonEmpty(metrics.Network.Health.Gateway, "Default gateway не определён."))
}
if !metrics.Network.Health.CheckedAt.IsZero() && !metrics.Network.Health.DNSUp {
add("network-dns", "critical", "Сеть", "Не работает DNS", "Не удалось разрешить deb.debian.org.")
}
if !metrics.Network.Health.CheckedAt.IsZero() && !metrics.Network.Health.InternetUp {
add("network-internet", "warning", "Сеть", "Нет доступа в интернет", "Внешняя проверка IP не выполнена.")
}
if metrics.Network.Health.PacketLoss > 0 {
add("network-loss", "warning", "Сеть", "Потери пакетов до шлюза", fmt.Sprintf("Потери %.1f%%, задержка %.1f ms.", metrics.Network.Health.PacketLoss, metrics.Network.Health.LatencyMs))
}
if metrics.Network.Health.ExternalIPChanged {
add("network-public-ip", "info", "Сеть", "Изменился внешний IP", "Новый адрес: "+metrics.Network.Health.ExternalIP)
}
for _, name := range metrics.Network.Health.DisappearedInterfaces {
add("network-interface-missing-"+name, "warning", "Сеть", "Исчез сетевой интерфейс", name+" больше не присутствует в /proc/net/dev.")
}
for _, iface := range metrics.Network.Interfaces {
if iface.Kind == "Physical" && !iface.Carrier && (iface.Master != "" || iface.ReceiveBytes+iface.TransmitBytes > 0) {
add("network-link-"+iface.Name, "warning", "Сеть", iface.Name+": нет carrier", "Физический линк не поднят.")
}
if (iface.Kind == "Bridge" || iface.Kind == "Bond" || iface.Kind == "VLAN") && iface.State == "down" {
add("network-state-"+iface.Name, "warning", "Сеть", iface.Name+": интерфейс выключен", iface.Kind+" находится в состоянии down.")
}
}
if !metrics.Monitor.LastSuccessfulCollection.IsZero() && time.Since(metrics.Monitor.LastSuccessfulCollection) > 30*time.Second {
add("monitor-stale", "critical", "Dashboard", "Данные мониторинга устарели", "Последний успешный сбор был "+time.Since(metrics.Monitor.LastSuccessfulCollection).Round(time.Second).String()+" назад.")
}
if metrics.Monitor.HistoryWriteError != "" {
add("monitor-history", "critical", "Dashboard", "Ошибка записи истории", metrics.Monitor.HistoryWriteError)
}
if metrics.Monitor.DatabaseFreeBytes > 0 && metrics.Monitor.DatabaseFreeBytes < 1024*1024*1024 {
add("monitor-db-space", "warning", "Dashboard", "Мало места для базы истории", fmt.Sprintf("На разделе базы свободно %.1f GB.", float64(metrics.Monitor.DatabaseFreeBytes)/1073741824))
}
oldSnapshots := 0
for _, snapshot := range metrics.Activity.Snapshots {
if snapshot.AgeSeconds > 30*24*3600 {
oldSnapshots++
}
}
if oldSnapshots > 0 {
add("old-snapshots", "warning", "VM/LXC", "Есть старые snapshots", fmt.Sprintf("Снимков старше 30 дней: %d.", oldSnapshots))
}
for _, task := range metrics.Activity.Tasks {
if task.StartTime < time.Now().Add(-24*time.Hour).Unix() {
continue
}
if task.Status != "" && task.Status != "OK" {
severity := "warning"
if task.Type == "vzdump" {
severity = "critical"
}
add("task-failed-"+task.UPID, severity, "Proxmox", task.Type+": задача завершилась ошибкой", fmt.Sprintf("VMID %s, пользователь %s, статус %s.", task.ID, task.User, task.Status))
}
}
for _, trend := range metrics.Trends {
add(trend.ID, trend.Severity, trend.Source, trend.Title, trend.Message)
}
priority := map[string]int{"critical": 0, "warning": 1, "info": 2}
sort.SliceStable(alerts, func(i, j int) bool {
@@ -139,6 +307,51 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
return alerts
}
func correlateAlerts(metrics dashboardMetrics, alerts []Alert) []Alert {
pressureAlerts := map[string]bool{}
for _, a := range alerts {
if strings.HasPrefix(a.ID, "disk-io-pressure-") {
pressureAlerts[strings.TrimPrefix(a.ID, "disk-io-pressure-")] = true
}
}
var pressure []DiskMetrics
for _, d := range metrics.Disks {
if pressureAlerts[d.Name] && d.IOCause != "" {
pressure = append(pressure, d)
}
}
if len(pressure) == 0 {
return alerts
}
var down []string
for _, service := range metrics.Services.Services {
for _, endpoint := range service.Endpoints {
if !endpoint.Up {
down = append(down, service.Name)
break
}
}
}
d := pressure[0]
chain := d.IOCause + " → " + d.Name + fmt.Sprintf(" util %.1f%% → latency %.1f ms", d.Utilization, d.LatencyMs)
if len(down) > 0 {
chain += " → недоступны/медленны сервисы: " + strings.Join(down, ", ")
}
severity := "warning"
if len(down) > 0 || d.Utilization >= 98 {
severity = "critical"
}
correlated := Alert{ID: "correlation-io-" + d.Name, Severity: severity, Source: "Корреляция", Title: "Обнаружена связанная цепочка I/O-событий", Message: chain + fmt.Sprintf(". Уверенность причины %d%%.", d.IOConfidence)}
filtered := []Alert{correlated}
for _, a := range alerts {
if strings.HasPrefix(a.ID, "disk-io-pressure-") || strings.HasPrefix(a.ID, "service-") {
continue
}
filtered = append(filtered, a)
}
return filtered
}
func joinReasons(reasons []string, fallback string) string {
if len(reasons) == 0 {
return fallback

View File

@@ -16,3 +16,12 @@ func TestEvaluateAlerts(t *testing.T) {
t.Fatalf("неверная сортировка уведомлений: %+v", alerts)
}
}
func TestSlowCollectionDoesNotCreateAlert(t *testing.T) {
metrics := dashboardMetrics{Monitor: MonitorHealth{SlowCollector: true, CollectionDurationMs: 12308}}
for _, alert := range evaluateAlerts(metrics, defaultThresholds()) {
if alert.ID == "monitor-slow" {
t.Fatalf("медленный цикл не должен создавать уведомление: %+v", alert)
}
}
}

View File

@@ -12,12 +12,15 @@ import (
)
type BackupItem struct {
VMID int `json:"vmid"`
Storage string `json:"storage"`
VolumeID string `json:"volid"`
Format string `json:"format"`
SizeBytes uint64 `json:"size"`
CreatedAt int64 `json:"ctime"`
VMID int `json:"vmid"`
Storage string `json:"storage"`
VolumeID string `json:"volid"`
Format string `json:"format"`
SizeBytes uint64 `json:"size"`
CreatedAt int64 `json:"ctime"`
PreviousSizeBytes uint64 `json:"previousSize,omitempty"`
SizeChangePercent *float64 `json:"sizeChangePercent,omitempty"`
PreviousCreatedAt int64 `json:"previousCtime,omitempty"`
}
type BackupMetrics struct {
@@ -89,17 +92,25 @@ func readBackups() BackupMetrics {
}
func latestBackups(items []BackupItem) []BackupItem {
latest := make(map[int]BackupItem)
byGuest := make(map[int][]BackupItem)
for _, item := range items {
if item.VMID == 0 {
continue
}
if previous, ok := latest[item.VMID]; !ok || item.CreatedAt > previous.CreatedAt {
latest[item.VMID] = item
}
byGuest[item.VMID] = append(byGuest[item.VMID], item)
}
result := make([]BackupItem, 0, len(latest))
for _, item := range latest {
result := make([]BackupItem, 0, len(byGuest))
for _, history := range byGuest {
sort.Slice(history, func(i, j int) bool { return history[i].CreatedAt > history[j].CreatedAt })
item := history[0]
if len(history) > 1 {
item.PreviousSizeBytes = history[1].SizeBytes
item.PreviousCreatedAt = history[1].CreatedAt
if history[1].SizeBytes > 0 {
change := (float64(item.SizeBytes) - float64(history[1].SizeBytes)) / float64(history[1].SizeBytes) * 100
item.SizeChangePercent = &change
}
}
result = append(result, item)
}
sort.Slice(result, func(i, j int) bool { return result[i].VMID < result[j].VMID })

View File

@@ -12,4 +12,15 @@ func TestLatestBackups(t *testing.T) {
if len(latest) != 2 || latest[0].Storage != "new" {
t.Fatalf("неожиданный список последних копий: %+v", latest)
}
if latest[0].PreviousSizeBytes != 0 {
t.Fatalf("unexpected previous size: %+v", latest[0])
}
}
func TestLatestBackupsComparesSize(t *testing.T) {
items := []BackupItem{{VMID: 100, CreatedAt: 20, SizeBytes: 120}, {VMID: 100, CreatedAt: 10, SizeBytes: 100}}
latest := latestBackups(items)
if len(latest) != 1 || latest[0].SizeChangePercent == nil || *latest[0].SizeChangePercent != 20 {
t.Fatalf("unexpected comparison: %+v", latest)
}
}

268
disks.go
View File

@@ -13,40 +13,219 @@ import (
)
type DiskMetrics struct {
Name string `json:"name"`
Path string `json:"path"`
Model string `json:"model"`
Serial string `json:"serial"`
Type string `json:"type"`
Protocol string `json:"protocol"`
SizeBytes uint64 `json:"sizeBytes"`
UsedBytes *uint64 `json:"usedBytes"`
UsagePercent *float64 `json:"usagePercent"`
SMARTStatus string `json:"smartStatus"`
SMARTAvailable bool `json:"smartAvailable"`
AttentionReasons []string `json:"attentionReasons"`
Temperature *float64 `json:"temperatureCelsius"`
PowerOnHours *uint64 `json:"powerOnHours"`
WearUsedPercent *float64 `json:"wearUsedPercent"`
Name string `json:"name"`
Path string `json:"path"`
Model string `json:"model"`
Serial string `json:"serial"`
Type string `json:"type"`
Protocol string `json:"protocol"`
SizeBytes uint64 `json:"sizeBytes"`
UsedBytes *uint64 `json:"usedBytes"`
UsagePercent *float64 `json:"usagePercent"`
SMARTStatus string `json:"smartStatus"`
SMARTAvailable bool `json:"smartAvailable"`
AttentionReasons []string `json:"attentionReasons"`
SMARTHistory []string `json:"smartHistory"`
Temperature *float64 `json:"temperatureCelsius"`
PowerOnHours *uint64 `json:"powerOnHours"`
WearUsedPercent *float64 `json:"wearUsedPercent"`
Reallocated uint64 `json:"reallocated"`
Pending uint64 `json:"pending"`
Uncorrectable uint64 `json:"uncorrectable"`
CRCErrors uint64 `json:"crcErrors"`
MediaErrors uint64 `json:"mediaErrors"`
SMARTErrorCount uint64 `json:"smartErrorCount"`
CounterChanges []string `json:"counterChanges"`
LastSelfTest string `json:"lastSelfTest"`
LastSelfTestAt string `json:"lastSelfTestAt"`
SelfTestNeverRun bool `json:"selfTestNeverRun"`
ShortTestStarted bool `json:"shortTestStarted"`
ReadIOPS float64 `json:"readIops"`
WriteIOPS float64 `json:"writeIops"`
ReadBytesPerSec float64 `json:"readBytesPerSec"`
WriteBytesPerSec float64 `json:"writeBytesPerSec"`
ReadLatencyMs float64 `json:"readLatencyMs"`
WriteLatencyMs float64 `json:"writeLatencyMs"`
LatencyMs float64 `json:"latencyMs"`
Utilization float64 `json:"utilizationPercent"`
QueueDepth uint64 `json:"queueDepth"`
AverageQueue float64 `json:"averageQueue"`
IOPressureSeconds float64 `json:"ioPressureSeconds"`
IOCause string `json:"ioCause"`
IOCauseDetail string `json:"ioCauseDetail"`
IOConfidence int `json:"ioConfidence"`
TopConsumers []IOConsumer `json:"topConsumers"`
}
type diskIOSnapshot struct{ reads, readSectors, readMS, writes, writeSectors, writeMS, inFlight, ioMS, weightedMS uint64 }
type diskCollector struct {
mu sync.Mutex
cached []DiskMetrics
updatedAt time.Time
mu sync.Mutex
cached []DiskMetrics
updatedAt time.Time
previous map[string]DiskMetrics
missing []string
lastTests map[string]time.Time
ioPrevious map[string]diskIOSnapshot
ioUpdatedAt time.Time
ioPressureSince map[string]time.Time
}
func (c *diskCollector) collect() []DiskMetrics {
func (c *diskCollector) collect(thresholds AlertThresholds) []DiskMetrics {
c.mu.Lock()
defer c.mu.Unlock()
if time.Since(c.updatedAt) < time.Minute && c.cached != nil {
c.applyDiskIO(c.cached, thresholds)
return c.cached
}
c.cached = readPhysicalDisks()
current := readPhysicalDisks()
if c.previous == nil {
c.previous = map[string]DiskMetrics{}
c.lastTests = map[string]time.Time{}
}
seen := map[string]bool{}
c.missing = nil
for i := range current {
disk := &current[i]
key := disk.Serial
if key == "" || key == "Не указан" {
key = disk.Path
}
seen[key] = true
if old, ok := c.previous[key]; ok {
disk.CounterChanges = diskCounterChanges(old, *disk)
disk.AttentionReasons = append(disk.AttentionReasons, disk.CounterChanges...)
}
if disk.SMARTAvailable && (disk.SelfTestNeverRun || selfTestOlderThanWeek(disk.LastSelfTestAt)) && time.Since(c.lastTests[key]) >= 7*24*time.Hour {
disk.ShortTestStarted = startShortSMARTTest(disk.Path)
if disk.ShortTestStarted {
c.lastTests[key] = time.Now()
}
}
c.previous[key] = *disk
}
for key, disk := range c.previous {
if !seen[key] {
c.missing = append(c.missing, disk.Model+" ("+disk.Serial+")")
}
}
c.cached = current
c.applyDiskIO(c.cached, thresholds)
c.updatedAt = time.Now()
return c.cached
}
func (c *diskCollector) applyDiskIO(disks []DiskMetrics, thresholds AlertThresholds) {
now := time.Now()
snapshots := readDiskIOStats("/proc/diskstats")
if c.ioPrevious == nil {
c.ioPrevious = map[string]diskIOSnapshot{}
c.ioPressureSince = map[string]time.Time{}
}
elapsed := now.Sub(c.ioUpdatedAt).Seconds()
for i := range disks {
disk := &disks[i]
disk.ReadIOPS, disk.WriteIOPS, disk.ReadBytesPerSec, disk.WriteBytesPerSec = 0, 0, 0, 0
disk.ReadLatencyMs, disk.WriteLatencyMs, disk.LatencyMs, disk.Utilization, disk.AverageQueue, disk.IOPressureSeconds = 0, 0, 0, 0, 0, 0
current, ok := snapshots[disk.Name]
if !ok {
continue
}
disk.QueueDepth = current.inFlight
if old, found := c.ioPrevious[disk.Name]; found && elapsed > 0 && diskIOCountersMonotonic(old, current) {
readOps := current.reads - old.reads
writeOps := current.writes - old.writes
disk.ReadIOPS = float64(readOps) / elapsed
disk.WriteIOPS = float64(writeOps) / elapsed
disk.ReadBytesPerSec = float64(current.readSectors-old.readSectors) * 512 / elapsed
disk.WriteBytesPerSec = float64(current.writeSectors-old.writeSectors) * 512 / elapsed
if readOps > 0 {
disk.ReadLatencyMs = float64(current.readMS-old.readMS) / float64(readOps)
}
if writeOps > 0 {
disk.WriteLatencyMs = float64(current.writeMS-old.writeMS) / float64(writeOps)
}
if readOps+writeOps > 0 {
disk.LatencyMs = float64(current.readMS-old.readMS+current.writeMS-old.writeMS) / float64(readOps+writeOps)
}
disk.Utilization = min(float64(current.ioMS-old.ioMS)/(elapsed*10), 100)
disk.AverageQueue = float64(current.weightedMS-old.weightedMS) / (elapsed * 1000)
utilLimit, latencyLimit, queueLimit := thresholds.DiskSSDUtil, thresholds.DiskSSDLatency, thresholds.DiskSSDQueue
if disk.Type == "HDD" {
utilLimit, latencyLimit, queueLimit = thresholds.DiskHDDUtil, thresholds.DiskHDDLatency, thresholds.DiskHDDQueue
} else if strings.Contains(disk.Type, "NVMe") {
utilLimit, latencyLimit, queueLimit = thresholds.DiskNVMeUtil, thresholds.DiskNVMeLatency, thresholds.DiskNVMeQueue
}
pressured := disk.Utilization >= utilLimit || (disk.LatencyMs >= latencyLimit && disk.ReadIOPS+disk.WriteIOPS >= 1) || disk.AverageQueue >= queueLimit
if pressured {
if c.ioPressureSince[disk.Name].IsZero() {
c.ioPressureSince[disk.Name] = now
}
disk.IOPressureSeconds = now.Sub(c.ioPressureSince[disk.Name]).Seconds()
} else {
delete(c.ioPressureSince, disk.Name)
}
}
c.ioPrevious[disk.Name] = current
}
c.ioUpdatedAt = now
}
func diskIOCountersMonotonic(old, current diskIOSnapshot) bool {
return current.reads >= old.reads && current.readSectors >= old.readSectors && current.readMS >= old.readMS && current.writes >= old.writes && current.writeSectors >= old.writeSectors && current.writeMS >= old.writeMS && current.ioMS >= old.ioMS && current.weightedMS >= old.weightedMS
}
func readDiskIOStats(path string) map[string]diskIOSnapshot {
data, err := os.ReadFile(path)
if err != nil {
return nil
}
result := map[string]diskIOSnapshot{}
for _, line := range strings.Split(string(data), "\n") {
f := strings.Fields(line)
if len(f) < 14 || !isPhysicalDiskName(f[2]) {
continue
}
values := make([]uint64, 11)
for i := 0; i < 11; i++ {
values[i], _ = strconv.ParseUint(f[i+3], 10, 64)
}
result[f[2]] = diskIOSnapshot{reads: values[0], readSectors: values[2], readMS: values[3], writes: values[4], writeSectors: values[6], writeMS: values[7], inFlight: values[8], ioMS: values[9], weightedMS: values[10]}
}
return result
}
func (c *diskCollector) missingDisks() []string {
c.mu.Lock()
defer c.mu.Unlock()
return append([]string(nil), c.missing...)
}
func diskCounterChanges(old, current DiskMetrics) []string {
checks := []struct {
label string
old, now uint64
}{{"Reallocated вырос", old.Reallocated, current.Reallocated}, {"Pending вырос", old.Pending, current.Pending}, {"Uncorrectable вырос", old.Uncorrectable, current.Uncorrectable}, {"CRC-ошибки выросли", old.CRCErrors, current.CRCErrors}, {"Media errors выросли", old.MediaErrors, current.MediaErrors}, {"Ошибки SMART-журнала выросли", old.SMARTErrorCount, current.SMARTErrorCount}}
var changes []string
for _, check := range checks {
if check.now > check.old {
changes = append(changes, check.label+": "+strconv.FormatUint(check.old, 10)+" → "+strconv.FormatUint(check.now, 10))
}
}
return changes
}
func selfTestOlderThanWeek(value string) bool {
when, err := time.Parse(time.RFC3339, value)
return err == nil && time.Since(when) > 7*24*time.Hour
}
func startShortSMARTTest(path string) bool {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
return exec.CommandContext(ctx, "smartctl", "-t", "short", path).Run() == nil
}
func readPhysicalDisks() []DiskMetrics {
entries, _ := os.ReadDir("/sys/block")
usage := readDiskUsage()
@@ -202,6 +381,16 @@ type smartctlJSON struct {
Count int `json:"count"`
} `json:"summary"`
} `json:"ata_smart_error_log"`
ATASelfTestLog struct {
Standard struct {
Table []struct {
Status struct {
String string `json:"string"`
} `json:"status"`
LifetimeHours uint64 `json:"lifetime_hours"`
} `json:"table"`
} `json:"standard"`
} `json:"ata_smart_self_test_log"`
NVMeHealth *struct {
Temperature float64 `json:"temperature"`
PercentageUsed float64 `json:"percentage_used"`
@@ -254,6 +443,7 @@ func applySMART(disk *DiskMetrics) {
}
if smart.NVMeHealth != nil {
disk.WearUsedPercent = &smart.NVMeHealth.PercentageUsed
disk.MediaErrors = smart.NVMeHealth.MediaErrors
if smart.NVMeHealth.CriticalWarning != 0 {
disk.AttentionReasons = append(disk.AttentionReasons, "NVMe сообщает критическое предупреждение")
}
@@ -261,6 +451,17 @@ func applySMART(disk *DiskMetrics) {
disk.AttentionReasons = append(disk.AttentionReasons, "Ошибки целостности носителя: "+strconv.FormatUint(smart.NVMeHealth.MediaErrors, 10))
}
}
if len(smart.ATASelfTestLog.Standard.Table) > 0 {
last := smart.ATASelfTestLog.Standard.Table[0]
disk.LastSelfTest = firstNonEmpty(last.Status.String, "Результат неизвестен")
if disk.PowerOnHours != nil && *disk.PowerOnHours >= last.LifetimeHours {
disk.LastSelfTestAt = time.Now().Add(-time.Duration(*disk.PowerOnHours-last.LifetimeHours) * time.Hour).Format(time.RFC3339)
}
} else {
disk.SelfTestNeverRun = true
disk.LastSelfTest = "Никогда не запускался"
disk.AttentionReasons = append(disk.AttentionReasons, "SMART self-test ещё ни разу не запускался")
}
applyATAAttributes(disk, smart)
applySMARTExitStatus(disk, smart.Smartctl.ExitStatus)
if len(disk.AttentionReasons) > 0 && disk.SMARTStatus == "Исправен" {
@@ -275,20 +476,37 @@ func applyATAAttributes(disk *DiskMetrics, smart smartctlJSON) {
"Wear_Leveling_Count": true,
}
for _, attribute := range smart.ATAAttributes.Table {
switch attribute.Name {
case "Reallocated_Sector_Ct", "Reallocated_Event_Count":
disk.Reallocated += uint64(max(attribute.Raw.Value, 0))
case "Current_Pending_Sector":
disk.Pending = uint64(max(attribute.Raw.Value, 0))
case "Offline_Uncorrectable", "Reported_Uncorrect":
disk.Uncorrectable += uint64(max(attribute.Raw.Value, 0))
case "UDMA_CRC_Error_Count", "Interface_CRC_Error_Count":
disk.CRCErrors += uint64(max(attribute.Raw.Value, 0))
}
if wearNames[attribute.Name] && attribute.Value >= 0 && attribute.Value <= 100 {
wear := float64(100 - attribute.Value)
disk.WearUsedPercent = &wear
}
if attribute.WhenFailed != "" && attribute.WhenFailed != "-" {
if strings.EqualFold(attribute.WhenFailed, "FAILING_NOW") {
reason := attribute.Name + " вышел за порог"
if attribute.Raw.String != "" {
reason += " (" + attribute.Raw.String + ")"
}
disk.AttentionReasons = append(disk.AttentionReasons, reason)
} else if attribute.WhenFailed != "" && attribute.WhenFailed != "-" {
reason := attribute.Name + " ранее выходил за порог"
if attribute.Raw.String != "" {
reason += " (" + attribute.Raw.String + ")"
}
disk.SMARTHistory = append(disk.SMARTHistory, reason)
}
}
if smart.ATAErrorLog.Summary.Count > 0 {
disk.AttentionReasons = append(disk.AttentionReasons, "Ошибок в SMART-журнале: "+strconv.Itoa(smart.ATAErrorLog.Summary.Count))
disk.SMARTErrorCount = uint64(smart.ATAErrorLog.Summary.Count)
disk.SMARTHistory = append(disk.SMARTHistory, "Записей в SMART-журнале: "+strconv.Itoa(smart.ATAErrorLog.Summary.Count))
}
}
@@ -302,12 +520,12 @@ func applySMARTExitStatus(disk *DiskMetrics, status int) {
disk.AttentionReasons = append(disk.AttentionReasons, "Критический SMART-атрибут достиг порога")
}
if status&32 != 0 {
disk.AttentionReasons = append(disk.AttentionReasons, "SMART-атрибут ранее находился ниже порога")
disk.SMARTHistory = append(disk.SMARTHistory, "SMART-атрибут ранее находился ниже порога")
}
if status&64 != 0 {
disk.AttentionReasons = append(disk.AttentionReasons, "В журнале SMART обнаружены ошибки")
disk.SMARTHistory = append(disk.SMARTHistory, "В журнале SMART есть старые записи об ошибках")
}
if status&128 != 0 {
disk.AttentionReasons = append(disk.AttentionReasons, "В журнале самотестирования есть ошибки")
disk.SMARTHistory = append(disk.SMARTHistory, "В журнале самотестирования есть старые ошибки")
}
}

View File

@@ -2,6 +2,9 @@ package main
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
)
@@ -17,6 +20,22 @@ func TestIsPhysicalDiskName(t *testing.T) {
}
}
func TestReadDiskIOStats(t *testing.T) {
path := filepath.Join(t.TempDir(), "diskstats")
data := " 8 0 sda 100 2 300 40 50 3 700 80 4 90 120 0 0 0 0 0 0\n 7 0 loop0 999 0 999 0 999 0 999 0 0 0 0\n"
if err := os.WriteFile(path, []byte(data), 0o600); err != nil {
t.Fatal(err)
}
stats := readDiskIOStats(path)
disk, ok := stats["sda"]
if !ok || disk.reads != 100 || disk.readSectors != 300 || disk.writes != 50 || disk.writeSectors != 700 || disk.inFlight != 4 || disk.ioMS != 90 || disk.weightedMS != 120 {
t.Fatalf("unexpected stats: %+v", disk)
}
if _, exists := stats["loop0"]; exists {
t.Fatal("virtual device must be ignored")
}
}
func TestSumFilesystemUsage(t *testing.T) {
device := lsblkDevice{Children: []lsblkDevice{
{FSUsed: json.RawMessage(`100`), FSAvail: json.RawMessage(`300`)},
@@ -50,3 +69,37 @@ func TestApplyATAWearAndReason(t *testing.T) {
t.Fatalf("причина предупреждения не добавлена: %+v", disk)
}
}
func TestHistoricalATAFailureDoesNotRequireAttention(t *testing.T) {
var smart smartctlJSON
attribute := struct {
Name string `json:"name"`
Value int `json:"value"`
Threshold int `json:"thresh"`
WhenFailed string `json:"when_failed"`
Raw struct {
String string `json:"string"`
Value int64 `json:"value"`
} `json:"raw"`
}{Name: "Airflow_Temperature_Cel", WhenFailed: "In_the_past"}
attribute.Raw.String = "46 (Min/Max 26/47 #15)"
smart.ATAAttributes.Table = append(smart.ATAAttributes.Table, attribute)
disk := DiskMetrics{}
applyATAAttributes(&disk, smart)
applySMARTExitStatus(&disk, 32|64|128)
if len(disk.AttentionReasons) != 0 {
t.Fatalf("историческая ошибка стала активной: %+v", disk.AttentionReasons)
}
if len(disk.SMARTHistory) != 4 {
t.Fatalf("история SMART потеряна: %+v", disk.SMARTHistory)
}
}
func TestSMARTErrorLogGrowthRequiresAttention(t *testing.T) {
old := DiskMetrics{SMARTErrorCount: 2}
current := DiskMetrics{SMARTErrorCount: 3}
changes := diskCounterChanges(old, current)
if len(changes) != 1 || !strings.Contains(changes[0], "2 → 3") {
t.Fatalf("рост журнала не обнаружен: %+v", changes)
}
}

193
email.go Normal file
View File

@@ -0,0 +1,193 @@
package main
import (
"crypto/tls"
"fmt"
"html"
"log"
"mime"
"net"
"net/smtp"
"sort"
"strings"
"time"
)
func filterEmailAlerts(alerts []Alert, settings EmailSettings) []Alert {
allowedSeverity, allowedSource := map[string]bool{}, map[string]bool{}
for _, value := range settings.Severities {
allowedSeverity[value] = true
}
for _, value := range settings.Sources {
allowedSource[value] = true
}
var result []Alert
for _, alert := range alerts {
if allowedSeverity[alert.Severity] && allowedSource[alert.Source] {
result = append(result, alert)
}
}
return result
}
func sendAlertEmail(settings EmailSettings, alerts []Alert, test bool) error {
subject := "Proxmox Dashboard: сводка уведомлений"
body := `<div style="padding:32px;background:#0b1117;color:#edf3f7;font-family:Arial,sans-serif"><div style="max-width:680px;margin:auto"><div style="color:#55d58a;font-size:12px;letter-spacing:2px">PROXMOX DASHBOARD</div><h1 style="margin:10px 0 24px">Сводка событий</h1>`
if test {
subject = "Proxmox Dashboard: тест почты"
body += `<div style="padding:22px;border:1px solid #2b3944;border-radius:14px;background:#111a22"><b style="font-size:18px">Почта настроена</b><p style="color:#9aabb7;margin-bottom:0">Тестовое сообщение успешно отправлено вашим сервером.</p></div>`
}
if len(alerts) > 0 {
groups := map[string][]Alert{}
for _, alert := range alerts {
groups[alert.Severity] = append(groups[alert.Severity], alert)
}
var sections []string
for _, severity := range []string{"critical", "warning", "info"} {
items := groups[severity]
if len(items) == 0 {
continue
}
title := map[string]string{"critical": "Критические", "warning": "Предупреждения", "info": "Информация"}[severity]
color := map[string]string{"critical": "#ff665c", "warning": "#ffb35c", "info": "#6ba8ff"}[severity]
sort.Slice(items, func(i, j int) bool { return items[i].Title < items[j].Title })
cards := ""
for _, item := range items {
cards += fmt.Sprintf(`<div style="margin:10px 0;padding:16px 18px;border:1px solid #2b3944;border-left:4px solid %s;border-radius:10px;background:#111a22"><div style="color:#8fa0ac;font-size:11px;text-transform:uppercase">%s</div><div style="font-weight:bold;font-size:16px;margin:5px 0">%s</div><div style="color:#aab7c0;font-size:14px">%s</div></div>`, color, html.EscapeString(item.Source), html.EscapeString(item.Title), html.EscapeString(item.Message))
}
sections = append(sections, fmt.Sprintf(`<h2 style="font-size:14px;color:%s;margin:24px 0 8px">%s · %d</h2>%s`, color, title, len(items), cards))
}
body += strings.Join(sections, "")
subject = fmt.Sprintf("Proxmox Dashboard: событий — %d", len(alerts))
}
body += `<div style="color:#687985;font-size:12px;margin-top:28px">События собраны напрямую с Proxmox-хоста. Подтверждённые события повторно не отправляются.</div></div></div>`
message := []byte("From: " + settings.From + "\r\nTo: " + settings.To + "\r\nSubject: " + mime.QEncoding.Encode("UTF-8", subject) + "\r\nMIME-Version: 1.0\r\nContent-Type: text/html; charset=UTF-8\r\nContent-Transfer-Encoding: 8bit\r\n\r\n" + body + "\r\n")
address := net.JoinHostPort(settings.Host, fmt.Sprint(settings.Port))
tlsConfig := &tls.Config{ServerName: settings.Host, MinVersion: tls.VersionTLS12, InsecureSkipVerify: settings.SkipTLSVerify} //nolint:gosec -- explicit private-network option
var client *smtp.Client
if settings.Port == 465 {
connection, err := tls.Dial("tcp", address, tlsConfig)
if err != nil {
return err
}
client, err = smtp.NewClient(connection, settings.Host)
if err != nil {
return err
}
} else {
var err error
client, err = smtp.Dial(address)
if err != nil {
return err
}
if ok, _ := client.Extension("STARTTLS"); ok {
if err = client.StartTLS(tlsConfig); err != nil {
return err
}
}
}
defer client.Close()
if settings.Username != "" {
if err := client.Auth(smtp.PlainAuth("", settings.Username, settings.Password, settings.Host)); err != nil {
return err
}
}
if err := client.Mail(settings.From); err != nil {
return err
}
if err := client.Rcpt(settings.To); err != nil {
return err
}
writer, err := client.Data()
if err != nil {
return err
}
if _, err = writer.Write(message); err != nil {
return err
}
if err = writer.Close(); err != nil {
return err
}
return client.Quit()
}
func sendRecoveryEmail(settings EmailSettings, items []RecoveryEvent) error {
alerts := make([]Alert, 0, len(items))
for _, v := range items {
alerts = append(alerts, Alert{ID: v.ID, Severity: "info", Source: v.Source, Title: "Восстановлено: " + v.Title, Message: "Проблема больше не наблюдается."})
}
return sendAlertEmail(settings, alerts, false)
}
func runImmediateNotifier(collect func() (dashboardMetrics, error), store *Store) {
ticker := time.NewTicker(5 * time.Second)
defer ticker.Stop()
for range ticker.C {
settings := store.EmailSettings(true)
if !settings.Enabled {
continue
}
if _, err := collect(); err != nil {
continue
}
urgent, err := store.PendingImmediateAlerts()
if err == nil && len(urgent) > 0 {
urgent = filterEmailAlerts(urgent, settings)
if len(urgent) > 0 && sendAlertEmail(settings, urgent, false) == nil {
_ = store.MarkAlertsEmailed(urgent)
}
}
recoveries, err := store.PendingRecoveries()
if err == nil && len(recoveries) > 0 {
var filtered []RecoveryEvent
for _, v := range recoveries {
if contains(settings.Sources, v.Source) {
filtered = append(filtered, v)
}
}
if len(filtered) > 0 && sendRecoveryEmail(settings, filtered) == nil {
_ = store.MarkRecoveriesEmailed(filtered)
}
}
}
}
func contains(items []string, want string) bool {
for _, v := range items {
if v == want {
return true
}
}
return false
}
func runEmailNotifier(collect func() (dashboardMetrics, error), store *Store) {
ticker := time.NewTicker(time.Minute)
defer ticker.Stop()
for range ticker.C {
settings := store.EmailSettings(true)
if !settings.Enabled || time.Since(store.LastEmailAt()) < time.Duration(settings.IntervalMinutes)*time.Minute {
continue
}
_, err := collect()
if err != nil {
continue
}
pending, err := store.PendingEmailAlerts()
if err != nil {
continue
}
alerts := filterEmailAlerts(pending, settings)
if len(alerts) == 0 {
continue
}
if err := sendAlertEmail(settings, alerts, false); err != nil {
log.Printf("Не удалось отправить почтовую сводку: %v", err)
continue
}
if err := store.MarkAlertsEmailed(alerts); err != nil {
log.Printf("Не удалось отметить отправленные события: %v", err)
continue
}
store.SetLastEmailAt(time.Now())
}
}

40
email_test.go Normal file
View File

@@ -0,0 +1,40 @@
package main
import "testing"
func TestFilterEmailAlerts(t *testing.T) {
alerts := []Alert{
{ID: "cpu", Severity: "critical", Source: "CPU"},
{ID: "backup", Severity: "critical", Source: "Backups"},
{ID: "info", Severity: "info", Source: "CPU"},
}
settings := EmailSettings{Severities: []string{"critical"}, Sources: []string{"CPU"}}
filtered := filterEmailAlerts(alerts, settings)
if len(filtered) != 1 || filtered[0].ID != "cpu" {
t.Fatalf("unexpected filtered alerts: %+v", filtered)
}
}
func TestEmailPasswordIsPreserved(t *testing.T) {
store, err := openStore(":memory:")
if err != nil {
t.Fatal(err)
}
defer store.Close()
settings := defaultEmailSettings()
settings.Host, settings.From, settings.To, settings.Password = "smtp.example.com", "dash@example.com", "admin@example.com", "secret"
if err := store.SaveEmailSettings(settings); err != nil {
t.Fatal(err)
}
settings.Password = ""
settings.IntervalMinutes = 120
if err := store.SaveEmailSettings(settings); err != nil {
t.Fatal(err)
}
if got := store.EmailSettings(true); got.Password != "secret" || got.IntervalMinutes != 120 {
t.Fatalf("unexpected settings: %+v", got)
}
if got := store.EmailSettings(false); got.Password != "" || !got.HasPassword {
t.Fatalf("password leaked or marker lost: %+v", got)
}
}

180
guests.go
View File

@@ -3,24 +3,35 @@ package main
import (
"context"
"encoding/json"
"fmt"
"os/exec"
"sort"
"strconv"
"strings"
"sync"
"time"
)
type GuestMetrics struct {
VMID int `json:"vmid"`
Name string `json:"name"`
Type string `json:"type"`
Node string `json:"node"`
Status string `json:"status"`
CPU float64 `json:"cpu"`
MaxCPU float64 `json:"maxcpu"`
MemoryBytes uint64 `json:"mem"`
MaxMemory uint64 `json:"maxmem"`
Uptime uint64 `json:"uptime"`
Template int `json:"template"`
VMID int `json:"vmid"`
Name string `json:"name"`
Type string `json:"type"`
Node string `json:"node"`
Status string `json:"status"`
CPU float64 `json:"cpu"`
MaxCPU float64 `json:"maxcpu"`
MemoryBytes uint64 `json:"mem"`
MaxMemory uint64 `json:"maxmem"`
Uptime uint64 `json:"uptime"`
Template int `json:"template"`
DiskBytes uint64 `json:"disk"`
MaxDisk uint64 `json:"maxdisk"`
NetInBytes uint64 `json:"netin"`
NetOutBytes uint64 `json:"netout"`
OnBoot bool `json:"onboot"`
Agent bool `json:"agent"`
IPAddresses []string `json:"ipAddresses"`
Storage []string `json:"storage"`
}
type GuestsMetrics struct {
@@ -29,9 +40,17 @@ type GuestsMetrics struct {
}
type guestCollector struct {
mu sync.Mutex
cached GuestsMetrics
updatedAt time.Time
mu sync.Mutex
cached GuestsMetrics
updatedAt time.Time
details map[int]guestDetails
detailsUpdatedAt time.Time
detailsRefreshing bool
}
type guestDetails struct {
OnBoot, Agent bool
IPAddresses, Storage []string
}
func (c *guestCollector) collect() GuestsMetrics {
@@ -41,10 +60,143 @@ func (c *guestCollector) collect() GuestsMetrics {
return c.cached
}
c.cached = readGuests()
if time.Since(c.detailsUpdatedAt) > time.Minute && !c.detailsRefreshing {
c.detailsRefreshing = true
guests := append([]GuestMetrics(nil), c.cached.Guests...)
go func() {
details := readGuestDetails(guests)
c.mu.Lock()
c.details = details
c.detailsUpdatedAt = time.Now()
c.detailsRefreshing = false
c.mu.Unlock()
}()
}
for i := range c.cached.Guests {
if detail, ok := c.details[c.cached.Guests[i].VMID]; ok {
c.cached.Guests[i].OnBoot = detail.OnBoot
c.cached.Guests[i].Agent = detail.Agent
c.cached.Guests[i].IPAddresses = detail.IPAddresses
c.cached.Guests[i].Storage = detail.Storage
}
}
c.updatedAt = time.Now()
return c.cached
}
func readGuestDetails(guests []GuestMetrics) map[int]guestDetails {
result := map[int]guestDetails{}
for _, guest := range guests {
if guest.Template != 0 {
continue
}
kind := guest.Type
if kind != "qemu" {
kind = "lxc"
}
data := commandOutput(5*time.Second, "pvesh", "get", "/nodes/"+guest.Node+"/"+kind+"/"+strconv.Itoa(guest.VMID)+"/config", "--output-format", "json")
var config map[string]any
if json.Unmarshal([]byte(data), &config) != nil {
continue
}
detail := guestDetails{}
detail.OnBoot = numberValue(config["onboot"]) == 1
detail.Agent = agentEnabled(config["agent"])
storageSet := map[string]bool{}
ipSet := map[string]bool{}
for key, value := range config {
s := fmt.Sprint(value)
if key == "rootfs" || strings.HasPrefix(key, "scsi") || strings.HasPrefix(key, "sata") || strings.HasPrefix(key, "virtio") {
if name := storageName(s); name != "" {
storageSet[name] = true
}
}
if strings.HasPrefix(key, "net") || strings.HasPrefix(key, "ipconfig") {
for _, ip := range ipsFromConfig(s) {
ipSet[ip] = true
}
}
}
if guest.Status == "running" && kind == "qemu" {
agent := commandOutput(5*time.Second, "pvesh", "get", "/nodes/"+guest.Node+"/qemu/"+strconv.Itoa(guest.VMID)+"/agent/network-get-interfaces", "--output-format", "json")
for _, ip := range ipsFromAgent(agent) {
ipSet[ip] = true
}
if agent != "" {
detail.Agent = true
}
}
for value := range storageSet {
detail.Storage = append(detail.Storage, value)
}
for value := range ipSet {
detail.IPAddresses = append(detail.IPAddresses, value)
}
sort.Strings(detail.Storage)
sort.Strings(detail.IPAddresses)
result[guest.VMID] = detail
}
return result
}
func numberValue(value any) float64 {
switch v := value.(type) {
case float64:
return v
case string:
n, _ := strconv.ParseFloat(v, 64)
return n
}
return 0
}
func agentEnabled(value any) bool {
s := strings.ToLower(fmt.Sprint(value))
return s == "1" || strings.Contains(s, "enabled=1")
}
func storageName(value string) string {
first := strings.SplitN(value, ",", 2)[0]
parts := strings.SplitN(first, ":", 2)
if len(parts) == 2 {
return parts[0]
}
return ""
}
func ipsFromConfig(value string) []string {
var result []string
for _, part := range strings.Split(value, ",") {
part = strings.TrimSpace(part)
if strings.HasPrefix(part, "ip=") || strings.HasPrefix(part, "ip6=") {
ip := strings.SplitN(part, "=", 2)[1]
if ip != "dhcp" && ip != "auto" {
result = append(result, strings.SplitN(ip, "/", 2)[0])
}
}
}
return result
}
func ipsFromAgent(data string) []string {
var payload struct {
Result []struct {
IPAddresses []struct {
Address string `json:"ip-address"`
Type string `json:"ip-address-type"`
} `json:"ip-addresses"`
} `json:"result"`
}
if json.Unmarshal([]byte(data), &payload) != nil {
return nil
}
var result []string
for _, iface := range payload.Result {
for _, ip := range iface.IPAddresses {
if ip.Address != "127.0.0.1" && ip.Address != "::1" && !strings.HasPrefix(ip.Address, "fe80:") {
result = append(result, ip.Address)
}
}
}
return result
}
func readGuests() GuestsMetrics {
if _, err := exec.LookPath("pvesh"); err != nil {
return GuestsMetrics{}

View File

@@ -15,3 +15,16 @@ func TestGuestJSON(t *testing.T) {
t.Fatalf("неожиданные данные гостя: %+v", guest)
}
}
func TestGuestConfigHelpers(t *testing.T) {
if got := storageName("local-zfs:vm-101-disk-0,size=32G"); got != "local-zfs" {
t.Fatalf("storage: %s", got)
}
ips := ipsFromConfig("name=eth0,bridge=vmbr0,ip=192.168.0.20/24,ip6=auto")
if len(ips) != 1 || ips[0] != "192.168.0.20" {
t.Fatalf("ips: %+v", ips)
}
if !agentEnabled("enabled=1,fstrim_cloned_disks=1") {
t.Fatal("agent should be enabled")
}
}

View File

@@ -1,6 +1,7 @@
package main
import (
"fmt"
"log"
"time"
)
@@ -15,6 +16,7 @@ func runHistoryCollector(collect func() (dashboardMetrics, error), store *Store)
if err := store.AddHistory(historyPointFromMetrics(metrics)); err != nil {
log.Printf("Не удалось сохранить исторические метрики: %v", err)
}
store.AddEntityMetrics(metrics)
}
record()
ticker := time.NewTicker(time.Minute)
@@ -24,6 +26,47 @@ func runHistoryCollector(collect func() (dashboardMetrics, error), store *Store)
}
}
func (s *Store) AddEntityMetrics(metrics dashboardMetrics) {
ts := time.Now().Unix()
for _, p := range metrics.ZFS.Pools {
v := p.CapacityPercent
s.AddEntityHistory("zfs", p.Name, ts, &v, nil)
}
for _, g := range metrics.Guests.Guests {
if g.Template != 0 {
continue
}
cpu := g.CPU * 100
var mem float64
if g.MaxMemory > 0 {
mem = float64(g.MemoryBytes) / float64(g.MaxMemory) * 100
}
s.AddEntityHistory("guest", fmt.Sprintf("%d · %s", g.VMID, g.Name), ts, &cpu, &mem)
}
for _, d := range metrics.Disks {
iops := d.ReadIOPS + d.WriteIOPS
util := d.Utilization
s.AddEntityHistory("disk-io", d.Name+" · "+d.Model, ts, &iops, &util)
latency := d.LatencyMs
s.AddEntityHistory("disk-latency", d.Name+" · "+d.Model, ts, &latency, nil)
if d.Temperature == nil {
continue
}
var wear *float64
if d.WearUsedPercent != nil {
wear = d.WearUsedPercent
}
s.AddEntityHistory("disk", d.Name+" · "+d.Model, ts, d.Temperature, wear)
}
for _, task := range metrics.Activity.Tasks {
if task.Type == "vzdump" && task.Duration > 0 && task.EndTime >= time.Now().Add(-15*time.Minute).Unix() {
duration := float64(task.Duration)
s.AddEntityHistory("backup-duration", "VMID "+task.ID, ts, &duration, nil)
}
}
s.PruneEntityHistory()
}
func historyPointFromMetrics(metrics dashboardMetrics) HistoryPoint {
point := HistoryPoint{
Timestamp: time.Now().Unix(), CPUUsage: metrics.CPU.UsagePercent,

178
io_attribution.go Normal file
View File

@@ -0,0 +1,178 @@
package main
import (
"os"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"sync"
"time"
)
type IOConsumer struct {
Kind string `json:"kind"`
Name string `json:"name"`
VMID int `json:"vmid,omitempty"`
ReadBytesPerSec float64 `json:"readBytesPerSec"`
WriteBytesPerSec float64 `json:"writeBytesPerSec"`
SharePercent float64 `json:"sharePercent"`
}
type processIOSnapshot struct {
read, write uint64
name, cmdline, cgroup string
}
type ioAttributionCollector struct {
mu sync.Mutex
previous map[int]processIOSnapshot
updatedAt time.Time
cached []IOConsumer
}
var qemuIDPattern = regexp.MustCompile(`(?:^|\s)-id\s+(\d+)`)
var lxcIDPattern = regexp.MustCompile(`(?:lxc(?:\.payload)?[./-]|pve-container@)(\d+)`)
func (c *ioAttributionCollector) collect(guests GuestsMetrics) []IOConsumer {
c.mu.Lock()
defer c.mu.Unlock()
now := time.Now()
current := readProcessIO()
elapsed := now.Sub(c.updatedAt).Seconds()
if c.previous == nil || elapsed <= 0 {
c.previous = current
c.updatedAt = now
return nil
}
names := map[int]string{}
for _, g := range guests.Guests {
names[g.VMID] = g.Name
}
grouped := map[string]*IOConsumer{}
var total float64
for pid, snapshot := range current {
old, ok := c.previous[pid]
if !ok || snapshot.read < old.read || snapshot.write < old.write {
continue
}
read := float64(snapshot.read-old.read) / elapsed
write := float64(snapshot.write-old.write) / elapsed
if read+write < 1024 {
continue
}
kind, name, vmid := classifyIOProcess(snapshot, names)
key := kind + ":" + strconv.Itoa(vmid) + ":" + name
consumer := grouped[key]
if consumer == nil {
consumer = &IOConsumer{Kind: kind, Name: name, VMID: vmid}
grouped[key] = consumer
}
consumer.ReadBytesPerSec += read
consumer.WriteBytesPerSec += write
total += read + write
}
var result []IOConsumer
for _, v := range grouped {
if total > 0 {
v.SharePercent = (v.ReadBytesPerSec + v.WriteBytesPerSec) / total * 100
}
result = append(result, *v)
}
sort.Slice(result, func(i, j int) bool {
return result[i].ReadBytesPerSec+result[i].WriteBytesPerSec > result[j].ReadBytesPerSec+result[j].WriteBytesPerSec
})
if len(result) > 8 {
result = result[:8]
}
c.previous = current
c.updatedAt = now
c.cached = result
return result
}
func classifyIOProcess(p processIOSnapshot, names map[int]string) (kind, name string, vmid int) {
if match := qemuIDPattern.FindStringSubmatch(p.cmdline); len(match) > 1 {
vmid, _ = strconv.Atoi(match[1])
return "VM", firstNonEmpty(names[vmid], "VM "+match[1]), vmid
}
if match := lxcIDPattern.FindStringSubmatch(p.cgroup); len(match) > 1 {
vmid, _ = strconv.Atoi(match[1])
return "LXC", firstNonEmpty(names[vmid], "LXC "+match[1]), vmid
}
return "Процесс", firstNonEmpty(p.name, "PID"), 0
}
func readProcessIO() map[int]processIOSnapshot {
result := map[int]processIOSnapshot{}
entries, _ := filepath.Glob("/proc/[0-9]*/io")
for _, path := range entries {
pid, _ := strconv.Atoi(filepath.Base(filepath.Dir(path)))
data, err := os.ReadFile(path)
if err != nil {
continue
}
var item processIOSnapshot
for _, line := range strings.Split(string(data), "\n") {
key, value, ok := strings.Cut(line, ":")
if !ok {
continue
}
n, _ := strconv.ParseUint(strings.TrimSpace(value), 10, 64)
if key == "read_bytes" {
item.read = n
} else if key == "write_bytes" {
item.write = n
}
}
comm, _ := os.ReadFile(filepath.Join(filepath.Dir(path), "comm"))
cmd, _ := os.ReadFile(filepath.Join(filepath.Dir(path), "cmdline"))
cg, _ := os.ReadFile(filepath.Join(filepath.Dir(path), "cgroup"))
item.name = strings.TrimSpace(string(comm))
item.cmdline = strings.ReplaceAll(string(cmd), "\x00", " ")
item.cgroup = string(cg)
result[pid] = item
}
return result
}
func attributeDiskIO(metrics *dashboardMetrics, consumers []IOConsumer) {
active := activeIOOperation(metrics.Activity.Tasks)
for i := range metrics.Disks {
disk := &metrics.Disks[i]
disk.TopConsumers = append([]IOConsumer(nil), consumers...)
if active != "" {
disk.IOCause = active
disk.IOCauseDetail = "Активная задача Proxmox совпадает по времени с дисковой нагрузкой"
disk.IOConfidence = 75
} else if len(consumers) > 0 {
top := consumers[0]
disk.IOCause = top.Kind + " · " + top.Name
disk.IOCauseDetail = "Крупнейший потребитель host I/O: " + strconv.FormatFloat(top.SharePercent, 'f', 1, 64) + "%"
disk.IOConfidence = 55
}
}
}
func activeIOOperation(tasks []ProxmoxTask) string {
for _, task := range tasks {
if task.EndTime != 0 && task.Status != "" {
continue
}
label := ""
switch task.Type {
case "vzdump":
label = "Backup"
case "qmigrate":
label = "Миграция"
case "qmsnapshot", "vzsnapshot":
label = "Snapshot"
}
if label != "" {
if task.ID != "" {
label += " VM/LXC " + task.ID
}
return label
}
}
return ""
}

33
io_attribution_test.go Normal file
View File

@@ -0,0 +1,33 @@
package main
import (
"strings"
"testing"
)
func TestClassifyQEMUAndLXCIO(t *testing.T) {
kind, name, vmid := classifyIOProcess(processIOSnapshot{name: "qemu", cmdline: "qemu-system-x86_64 -id 105 -name guest=test"}, map[int]string{105: "Gitea"})
if kind != "VM" || name != "Gitea" || vmid != 105 {
t.Fatalf("unexpected qemu attribution: %s %s %d", kind, name, vmid)
}
kind, name, vmid = classifyIOProcess(processIOSnapshot{name: "worker", cgroup: "0::/lxc.payload.220/ns"}, map[int]string{220: "db"})
if kind != "LXC" || name != "db" || vmid != 220 {
t.Fatalf("unexpected lxc attribution: %s %s %d", kind, name, vmid)
}
}
func TestActiveIOOperation(t *testing.T) {
value := activeIOOperation([]ProxmoxTask{{Type: "vzdump", ID: "105", Status: "", EndTime: 0}})
if !strings.Contains(value, "Backup") || !strings.Contains(value, "105") {
t.Fatalf("unexpected operation: %s", value)
}
}
func TestCorrelateAlerts(t *testing.T) {
metrics := dashboardMetrics{Disks: []DiskMetrics{{Name: "sda", IOCause: "Backup VM/LXC 105", IOConfidence: 90, Utilization: 98, LatencyMs: 80}}, Services: ServicesMetrics{Services: []ServiceStatus{{MonitoredService: MonitoredService{Name: "Gitea"}, Endpoints: []ServiceEndpointStatus{{Up: false}}}}}}
alerts := []Alert{{ID: "disk-io-pressure-sda", Severity: "critical"}, {ID: "service-1-public", Severity: "critical"}, {ID: "other", Severity: "warning"}}
result := correlateAlerts(metrics, alerts)
if len(result) != 2 || result[0].Source != "Корреляция" || !strings.Contains(result[0].Message, "Gitea") {
t.Fatalf("unexpected correlation: %+v", result)
}
}

21
main.go
View File

@@ -3,6 +3,7 @@ package main
import (
"context"
"errors"
"flag"
"fmt"
"log"
"net/http"
@@ -18,6 +19,25 @@ const defaultDatabasePath = "/var/lib/proxmox-dashboard/dashboard.db"
var version = "dev"
func main() {
showVersion := flag.Bool("version", false, "show version")
agentMode := flag.Bool("agent", false, "run as a read-only guest agent")
agentServer := flag.String("agent-server", "", "dashboard URL used by the guest agent")
agentEnrollToken := flag.String("agent-enroll-token", "", "one-time enrollment token")
agentName := flag.String("agent-name", "", "agent display name")
agentVMID := flag.Int("agent-vmid", 0, "Proxmox VM/LXC ID")
agentConfig := flag.String("agent-config", "/var/lib/proxmox-dashboard-agent/agent.json", "agent configuration path")
flag.Parse()
if *showVersion {
fmt.Println(version)
return
}
if *agentMode {
if err := runGuestAgent(agentOptions{Server: *agentServer, EnrollmentToken: *agentEnrollToken, Name: *agentName, VMID: *agentVMID, ConfigPath: *agentConfig}); err != nil {
log.Fatalf("Ошибка агента: %v", err)
}
return
}
address := os.Getenv("DASHBOARD_ADDR")
if address == "" {
address = defaultAddress
@@ -42,6 +62,7 @@ func main() {
Handler: routes(collector, store),
ReadHeaderTimeout: 5 * time.Second,
}
runSystemdWatchdog()
go func() {
log.Printf("Proxmox Dashboard %s запущен на http://0.0.0.0%s", version, address)

264
maintenance.go Normal file
View File

@@ -0,0 +1,264 @@
package main
import (
"bufio"
"compress/gzip"
"context"
"encoding/json"
"io"
"os"
"os/exec"
"path/filepath"
"regexp"
"sort"
"strconv"
"strings"
"sync"
"time"
)
type MaintenanceMetrics struct {
Available bool `json:"available"`
ProxmoxVersion string `json:"proxmoxVersion"`
KernelVersion string `json:"kernelVersion"`
LatestInstalledKernel string `json:"latestInstalledKernel"`
NewKernelAvailable bool `json:"newKernelAvailable"`
RebootRequired bool `json:"rebootRequired"`
AvailableUpdates int `json:"availableUpdates"`
LastSystemUpdate time.Time `json:"lastSystemUpdate"`
NextBackup string `json:"nextBackup"`
NextScrub string `json:"nextScrub"`
CheckedAt time.Time `json:"checkedAt"`
}
type maintenanceCollector struct {
mu sync.Mutex
cached MaintenanceMetrics
updatedAt time.Time
refreshing bool
}
func (c *maintenanceCollector) collect() MaintenanceMetrics {
c.mu.Lock()
if time.Since(c.updatedAt) >= 15*time.Minute && !c.refreshing {
c.refreshing = true
go func() {
metrics := readMaintenance()
c.mu.Lock()
c.cached = metrics
c.updatedAt = time.Now()
c.refreshing = false
c.mu.Unlock()
}()
}
result := c.cached
c.mu.Unlock()
return result
}
func commandOutput(timeout time.Duration, name string, args ...string) string {
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
command := exec.CommandContext(ctx, name, args...)
command.Env = append(os.Environ(), "LC_ALL=C")
output, err := command.Output()
if err != nil {
return ""
}
return strings.TrimSpace(string(output))
}
func readMaintenance() MaintenanceMetrics {
result := MaintenanceMetrics{Available: true, CheckedAt: time.Now()}
result.ProxmoxVersion = firstLine(commandOutput(5*time.Second, "pveversion"))
result.KernelVersion = commandOutput(3*time.Second, "uname", "-r")
result.RebootRequired = fileExists("/var/run/reboot-required")
apt := commandOutput(45*time.Second, "apt-get", "-s", "-o", "Debug::NoLocking=1", "upgrade")
for _, line := range strings.Split(apt, "\n") {
if strings.HasPrefix(line, "Inst ") {
result.AvailableUpdates++
}
}
result.LastSystemUpdate = lastAPTUpdate("/var/log/apt/history.log")
installed := commandOutput(8*time.Second, "dpkg-query", "-W", "-f=${Package} ${Version}\\n", "pve-kernel-*", "proxmox-kernel-*")
result.LatestInstalledKernel = newestKernel(installed)
result.NewKernelAvailable = result.LatestInstalledKernel != "" && !strings.Contains(result.KernelVersion, result.LatestInstalledKernel)
result.NextBackup = nextBackupSchedule()
result.NextScrub = nextScrubTimer(commandOutput(8*time.Second, "systemctl", "list-timers", "--all", "--no-legend", "--plain"))
if result.NextScrub == "" {
result.NextScrub = nextZFSScrubCron("/etc/cron.d/zfsutils-linux", time.Now())
}
return result
}
func firstLine(value string) string {
if i := strings.IndexByte(value, '\n'); i >= 0 {
return value[:i]
}
return value
}
func fileExists(path string) bool { _, err := os.Stat(path); return err == nil }
func lastAPTUpdate(path string) time.Time {
file, err := os.Open(path)
if err == nil {
defer file.Close()
if result := scanAPTUpdate(file); !result.IsZero() {
return result
}
}
files, _ := filepath.Glob(path + ".*.gz")
sort.Slice(files, func(i, j int) bool {
a, _ := os.Stat(files[i])
b, _ := os.Stat(files[j])
return a.ModTime().After(b.ModTime())
})
for _, name := range files {
compressed, err := os.Open(name)
if err != nil {
continue
}
reader, err := gzip.NewReader(compressed)
if err != nil {
compressed.Close()
continue
}
result := scanAPTUpdate(reader)
reader.Close()
compressed.Close()
if !result.IsZero() {
return result
}
}
return time.Time{}
}
func scanAPTUpdate(reader io.Reader) time.Time {
scanner := bufio.NewScanner(reader)
var result time.Time
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
if strings.HasPrefix(line, "End-Date:") {
value := strings.TrimSpace(strings.TrimPrefix(line, "End-Date:"))
if parsed, err := time.ParseInLocation("2006-01-02 15:04:05", value, time.Local); err == nil {
result = parsed
}
}
}
return result
}
var versionNumbers = regexp.MustCompile(`\d+`)
func newestKernel(output string) string {
var versions []string
for _, line := range strings.Split(output, "\n") {
fields := strings.Fields(line)
if len(fields) > 1 && !strings.Contains(fields[0], "helper") {
versions = append(versions, fields[1])
}
}
sort.Slice(versions, func(i, j int) bool { return compareNumericVersion(versions[i], versions[j]) > 0 })
if len(versions) > 0 {
return versions[0]
}
return ""
}
func compareNumericVersion(a, b string) int {
aa, bb := versionNumbers.FindAllString(a, -1), versionNumbers.FindAllString(b, -1)
for i := 0; i < len(aa) || i < len(bb); i++ {
av, bv := "0", "0"
if i < len(aa) {
av = aa[i]
}
if i < len(bb) {
bv = bb[i]
}
if len(av) != len(bv) {
if len(av) > len(bv) {
return 1
}
return -1
}
if av != bv {
if av > bv {
return 1
}
return -1
}
}
return 0
}
func nextBackupSchedule() string {
data := commandOutput(8*time.Second, "pvesh", "get", "/cluster/backup", "--output-format", "json")
if data == "" {
return ""
}
var jobs []struct {
Enabled *int `json:"enabled"`
Schedule string `json:"schedule"`
}
if json.Unmarshal([]byte(data), &jobs) != nil {
return ""
}
var next []string
for _, job := range jobs {
if (job.Enabled != nil && *job.Enabled == 0) || job.Schedule == "" {
continue
}
calendar := commandOutput(5*time.Second, "systemd-analyze", "calendar", "--iterations=1", job.Schedule)
for _, line := range strings.Split(calendar, "\n") {
if strings.Contains(line, "Next elapse:") {
trimmed := strings.TrimSpace(line)
next = append(next, strings.TrimSpace(strings.TrimPrefix(trimmed, "Next elapse:")))
}
}
}
sort.Strings(next)
if len(next) > 0 {
return next[0]
}
return ""
}
func nextScrubTimer(output string) string {
for _, line := range strings.Split(output, "\n") {
if strings.Contains(strings.ToLower(line), "zfs") && strings.Contains(strings.ToLower(line), "scrub") {
fields := strings.Fields(line)
if len(fields) >= 4 {
return strings.Join(fields[:4], " ")
}
}
}
return ""
}
func nextZFSScrubCron(path string, now time.Time) string {
data, err := os.ReadFile(path)
if err != nil {
return ""
}
for _, line := range strings.Split(string(data), "\n") {
if !strings.Contains(line, "/usr/lib/zfs-linux/scrub") {
continue
}
fields := strings.Fields(line)
if len(fields) < 5 {
return ""
}
minute, _ := strconv.Atoi(fields[0])
hour, _ := strconv.Atoi(fields[1])
rangeParts := strings.SplitN(fields[2], "-", 2)
if len(rangeParts) != 2 {
return ""
}
from, _ := strconv.Atoi(rangeParts[0])
to, _ := strconv.Atoi(rangeParts[1])
for offset := 0; offset < 70; offset++ {
day := time.Date(now.Year(), now.Month(), now.Day()+offset, hour, minute, 0, 0, now.Location())
if day.After(now) && day.Day() >= from && day.Day() <= to && day.Weekday() == time.Sunday {
return day.Format("Mon 2006-01-02 15:04 MST")
}
}
}
return ""
}

33
maintenance_test.go Normal file
View File

@@ -0,0 +1,33 @@
package main
import (
"os"
"path/filepath"
"testing"
"time"
)
func TestNewestKernel(t *testing.T) {
output := "proxmox-kernel-6.8 6.8.12-5\nproxmox-kernel-6.14 6.14.8-2"
if got := newestKernel(output); got != "6.14.8-2" {
t.Fatalf("unexpected newest kernel: %s", got)
}
}
func TestNextZFSScrubCron(t *testing.T) {
path := filepath.Join(t.TempDir(), "zfs")
if err := os.WriteFile(path, []byte("24 0 8-14 * * root /usr/lib/zfs-linux/scrub\n"), 0600); err != nil {
t.Fatal(err)
}
now := time.Date(2026, 8, 1, 12, 0, 0, 0, time.UTC)
if got := nextZFSScrubCron(path, now); got != "Sun 2026-08-09 00:24 UTC" {
t.Fatalf("unexpected cron scrub: %s", got)
}
}
func TestNextScrubTimer(t *testing.T) {
line := "Sun 2026-08-02 00:24:00 MSK 1 day left zfs-scrub-weekly@tank.timer"
if got := nextScrubTimer(line); got != "Sun 2026-08-02 00:24:00 MSK" {
t.Fatalf("unexpected scrub: %s", got)
}
}

125
maintenance_window.go Normal file
View File

@@ -0,0 +1,125 @@
package main
import (
"encoding/json"
"fmt"
"strconv"
"strings"
"time"
)
type MaintenanceWindow struct {
Enabled bool `json:"enabled"`
Active bool `json:"active"`
StartedAt int64 `json:"startedAt"`
EndsAt int64 `json:"endsAt"`
Reason string `json:"reason"`
AllAlerts bool `json:"allAlerts"`
ServiceIDs []int64 `json:"serviceIds"`
VMIDs []int `json:"vmids"`
}
func (s *Store) MaintenanceWindow() MaintenanceWindow {
var value string
var window MaintenanceWindow
if s.db.QueryRow(`SELECT value FROM settings WHERE key='maintenance_window'`).Scan(&value) == nil {
_ = json.Unmarshal([]byte(value), &window)
}
now := time.Now().Unix()
window.Active = window.Enabled && window.StartedAt <= now && window.EndsAt > now
if window.Enabled && window.EndsAt <= now {
window.Enabled = false
data, _ := json.Marshal(window)
_, _ = s.db.Exec(`INSERT INTO settings(key,value) VALUES('maintenance_window',?) ON CONFLICT(key) DO UPDATE SET value=excluded.value`, string(data))
}
return window
}
func (s *Store) SaveMaintenanceWindow(window MaintenanceWindow) (MaintenanceWindow, error) {
now := time.Now()
window.Reason = strings.TrimSpace(window.Reason)
if window.Reason == "" {
return window, fmt.Errorf("укажите причину плановых работ")
}
if window.EndsAt <= now.Unix() || window.EndsAt > now.Add(30*24*time.Hour).Unix() {
return window, fmt.Errorf("окончание должно быть в будущем, максимум через 30 дней")
}
if !window.AllAlerts && len(window.ServiceIDs) == 0 && len(window.VMIDs) == 0 {
return window, fmt.Errorf("выберите сервис, VM/LXC или все алерты")
}
window.Enabled = true
window.Active = true
window.StartedAt = now.Unix()
window.ServiceIDs = uniqueInt64(window.ServiceIDs)
window.VMIDs = uniqueInt(window.VMIDs)
data, _ := json.Marshal(window)
_, err := s.db.Exec(`INSERT INTO settings(key,value) VALUES('maintenance_window',?) ON CONFLICT(key) DO UPDATE SET value=excluded.value`, string(data))
return window, err
}
func (s *Store) StopMaintenanceWindow() (MaintenanceWindow, error) {
window := s.MaintenanceWindow()
window.Enabled = false
window.Active = false
window.EndsAt = time.Now().Unix()
data, _ := json.Marshal(window)
_, err := s.db.Exec(`INSERT INTO settings(key,value) VALUES('maintenance_window',?) ON CONFLICT(key) DO UPDATE SET value=excluded.value`, string(data))
return window, err
}
func splitMaintenanceAlerts(alerts []Alert, window MaintenanceWindow) (active, planned []Alert) {
if !window.Active {
return alerts, nil
}
for _, alert := range alerts {
if maintenanceMatchesAlert(window, alert) {
planned = append(planned, alert)
} else {
active = append(active, alert)
}
}
return active, planned
}
func maintenanceMatchesAlert(window MaintenanceWindow, alert Alert) bool {
if window.AllAlerts {
return true
}
for _, id := range window.ServiceIDs {
value := strconv.FormatInt(id, 10)
if strings.HasPrefix(alert.ID, "service-"+value+"-") || strings.HasPrefix(alert.ID, "service-tls-"+value+"-") {
return true
}
}
for _, vmid := range window.VMIDs {
value := strconv.Itoa(vmid)
if strings.HasSuffix(alert.ID, "-"+value) || strings.Contains(alert.Message, "VMID "+value) {
return true
}
}
return false
}
func uniqueInt64(values []int64) []int64 {
seen := map[int64]bool{}
result := make([]int64, 0, len(values))
for _, value := range values {
if value > 0 && !seen[value] {
seen[value] = true
result = append(result, value)
}
}
return result
}
func uniqueInt(values []int) []int {
seen := map[int]bool{}
result := make([]int, 0, len(values))
for _, value := range values {
if value > 0 && !seen[value] {
seen[value] = true
result = append(result, value)
}
}
return result
}

View File

@@ -0,0 +1,60 @@
package main
import (
"testing"
"time"
)
func TestMaintenanceFiltersSelectedTargets(t *testing.T) {
window := MaintenanceWindow{Active: true, ServiceIDs: []int64{7}, VMIDs: []int{105}}
alerts := []Alert{
{ID: "service-7-local"},
{ID: "service-tls-8-public"},
{ID: "backup-old-105"},
{ID: "task-failed-x", Message: "VMID 105, status error"},
{ID: "cpu-usage"},
}
active, planned := splitMaintenanceAlerts(alerts, window)
if len(active) != 2 || len(planned) != 3 {
t.Fatalf("неверное разделение: active=%+v planned=%+v", active, planned)
}
}
func TestMaintenanceExpiresAutomatically(t *testing.T) {
store, err := openStore(":memory:")
if err != nil {
t.Fatal(err)
}
defer store.Close()
window, err := store.SaveMaintenanceWindow(MaintenanceWindow{EndsAt: time.Now().Add(time.Hour).Unix(), Reason: "Обновление", AllAlerts: true})
if err != nil || !window.Active {
t.Fatalf("режим не включён: %+v %v", window, err)
}
if _, err = store.db.Exec(`UPDATE settings SET value=? WHERE key='maintenance_window'`, `{"enabled":true,"startedAt":1,"endsAt":2,"reason":"test","allAlerts":true}`); err != nil {
t.Fatal(err)
}
if expired := store.MaintenanceWindow(); expired.Active || expired.Enabled {
t.Fatalf("просроченный режим остался активным: %+v", expired)
}
}
func TestPlannedAlertOnlyAppearsInHistory(t *testing.T) {
store, err := openStore(":memory:")
if err != nil {
t.Fatal(err)
}
defer store.Close()
alert := Alert{ID: "service-7-local", Severity: "critical", Source: "Сервисы", Title: "Недоступен"}
visible, err := store.SyncAlertsWithMaintenance(nil, []Alert{alert}, "Обновление")
if err != nil || len(visible) != 0 {
t.Fatalf("плановый алерт стал активным: %+v %v", visible, err)
}
history, err := store.AlertHistory(10)
if err != nil || len(history) != 1 || !history[0].Planned || history[0].MaintenanceReason != "Обновление" {
t.Fatalf("плановое событие не сохранено: %+v %v", history, err)
}
pending, err := store.PendingEmailAlerts()
if err != nil || len(pending) != 0 {
t.Fatalf("плановое событие попало в почту: %+v %v", pending, err)
}
}

View File

@@ -2,7 +2,13 @@ package main
import (
"bufio"
"context"
"io"
"net"
"net/http"
"os"
"os/exec"
"path/filepath"
"strconv"
"strings"
"sync"
@@ -10,11 +16,47 @@ import (
)
type NetworkMetrics struct {
Interface string `json:"interface"`
Interface string `json:"interface"`
ReceiveBytes uint64 `json:"receiveBytes"`
TransmitBytes uint64 `json:"transmitBytes"`
ReceivePerSec float64 `json:"receivePerSec"`
TransmitPerSec float64 `json:"transmitPerSec"`
Interfaces []NetworkInterface `json:"interfaces"`
Health NetworkHealth `json:"health"`
}
type NetworkHealth struct {
Gateway string `json:"gateway"`
GatewayUp bool `json:"gatewayUp"`
DNSUp bool `json:"dnsUp"`
InternetUp bool `json:"internetUp"`
PacketLoss float64 `json:"packetLoss"`
LatencyMs float64 `json:"latencyMs"`
ExternalIP string `json:"externalIp"`
ExternalIPChanged bool `json:"externalIpChanged"`
DisappearedInterfaces []string `json:"disappearedInterfaces"`
CheckedAt time.Time `json:"checkedAt"`
}
type NetworkInterface struct {
Name string `json:"name"`
ReceiveBytes uint64 `json:"receiveBytes"`
TransmitBytes uint64 `json:"transmitBytes"`
ReceiveErrors uint64 `json:"receiveErrors"`
TransmitErrors uint64 `json:"transmitErrors"`
ReceiveDrops uint64 `json:"receiveDrops"`
TransmitDrops uint64 `json:"transmitDrops"`
ReceivePerSec float64 `json:"receivePerSec"`
TransmitPerSec float64 `json:"transmitPerSec"`
VMID int `json:"vmid,omitempty"`
GuestName string `json:"guestName,omitempty"`
GuestType string `json:"guestType,omitempty"`
Kind string `json:"kind"`
State string `json:"state"`
Carrier bool `json:"carrier"`
SpeedMbps int `json:"speedMbps,omitempty"`
Duplex string `json:"duplex,omitempty"`
Master string `json:"master,omitempty"`
}
type networkSnapshot struct {
@@ -24,24 +66,38 @@ type networkSnapshot struct {
}
type networkCollector struct {
mu sync.Mutex
path string
previous networkSnapshot
mu sync.Mutex
path string
previous networkSnapshot
previousInterfaces map[string]NetworkInterface
health NetworkHealth
healthUpdatedAt time.Time
externalIP string
knownInterfaces map[string]bool
}
func newNetworkCollector(path string) (*networkCollector, error) {
snapshot, _, err := readNetworkSnapshot(path)
snapshot, _, _, err := readNetworkSnapshot(path)
if err != nil {
return nil, err
}
return &networkCollector{path: path, previous: snapshot}, nil
_, _, details, _ := readNetworkSnapshot(path)
previous := map[string]NetworkInterface{}
for _, v := range details {
previous[v.Name] = v
}
known := map[string]bool{}
for _, v := range details {
known[v.Name] = true
}
return &networkCollector{path: path, previous: snapshot, previousInterfaces: previous, knownInterfaces: known}, nil
}
func (c *networkCollector) collect() (NetworkMetrics, error) {
c.mu.Lock()
defer c.mu.Unlock()
current, interfaces, err := readNetworkSnapshot(c.path)
current, interfaceNames, details, err := readNetworkSnapshot(c.path)
if err != nil {
return NetworkMetrics{}, err
}
@@ -51,22 +107,123 @@ func (c *networkCollector) collect() (NetworkMetrics, error) {
receivedRate = float64(current.received-c.previous.received) / elapsed
sentRate = float64(current.sent-c.previous.sent) / elapsed
}
for i := range details {
v := &details[i]
decorateInterface(v)
if old, ok := c.previousInterfaces[v.Name]; ok && elapsed > 0 && v.ReceiveBytes >= old.ReceiveBytes && v.TransmitBytes >= old.TransmitBytes {
v.ReceivePerSec = float64(v.ReceiveBytes-old.ReceiveBytes) / elapsed
v.TransmitPerSec = float64(v.TransmitBytes-old.TransmitBytes) / elapsed
}
}
c.previousInterfaces = map[string]NetworkInterface{}
for _, v := range details {
c.previousInterfaces[v.Name] = v
}
c.previous = current
if time.Since(c.healthUpdatedAt) > 30*time.Second {
c.health = checkNetworkHealth(c.externalIP)
if c.health.ExternalIP != "" {
c.externalIP = c.health.ExternalIP
}
c.healthUpdatedAt = time.Now()
}
currentNames := map[string]bool{}
for _, v := range details {
currentNames[v.Name] = true
}
c.health.DisappearedInterfaces = nil
for name := range c.knownInterfaces {
if !currentNames[name] {
c.health.DisappearedInterfaces = append(c.health.DisappearedInterfaces, name)
}
}
c.knownInterfaces = currentNames
return NetworkMetrics{
Interface: strings.Join(interfaces, ", "), ReceiveBytes: current.received,
TransmitBytes: current.sent, ReceivePerSec: receivedRate, TransmitPerSec: sentRate,
Interface: strings.Join(interfaceNames, ", "), Interfaces: details, ReceiveBytes: current.received,
TransmitBytes: current.sent, ReceivePerSec: receivedRate, TransmitPerSec: sentRate, Health: c.health,
}, nil
}
func readNetworkSnapshot(path string) (networkSnapshot, []string, error) {
func decorateInterface(v *NetworkInterface) {
base := filepath.Join("/sys/class/net", v.Name)
v.State = readTrimmed(filepath.Join(base, "operstate"))
v.Carrier = readTrimmed(filepath.Join(base, "carrier")) == "1"
v.SpeedMbps, _ = strconv.Atoi(readTrimmed(filepath.Join(base, "speed")))
v.Duplex = readTrimmed(filepath.Join(base, "duplex"))
if target, err := filepath.EvalSymlinks(filepath.Join(base, "master")); err == nil {
v.Master = filepath.Base(target)
}
switch {
case strings.HasPrefix(v.Name, "vmbr"):
v.Kind = "Bridge"
case strings.HasPrefix(v.Name, "bond"):
v.Kind = "Bond"
case strings.Contains(v.Name, "."):
v.Kind = "VLAN"
case strings.HasPrefix(v.Name, "tap") || strings.HasPrefix(v.Name, "veth"):
v.Kind = "Guest"
default:
v.Kind = "Physical"
}
}
func checkNetworkHealth(previousIP string) NetworkHealth {
result := NetworkHealth{CheckedAt: time.Now()}
ctx, cancel := context.WithTimeout(context.Background(), 6*time.Second)
defer cancel()
if out, err := exec.CommandContext(ctx, "ip", "route", "show", "default").Output(); err == nil {
f := strings.Fields(string(out))
for i := range f {
if f[i] == "via" && i+1 < len(f) {
result.Gateway = f[i+1]
break
}
}
}
if result.Gateway != "" {
out, _ := exec.CommandContext(ctx, "ping", "-c", "3", "-W", "1", result.Gateway).CombinedOutput()
text := string(out)
result.GatewayUp = strings.Contains(text, "0% packet loss")
if i := strings.Index(text, "packet loss"); i > 0 {
start := strings.LastIndex(text[:i], " ")
if start >= 0 {
result.PacketLoss, _ = strconv.ParseFloat(strings.TrimSuffix(strings.TrimSpace(text[start:i]), "%"), 64)
}
}
if i := strings.Index(text, "min/avg/max"); i >= 0 {
if eq := strings.Index(text[i:], "="); eq >= 0 {
parts := strings.Split(strings.Fields(text[i+eq+1:])[0], "/")
if len(parts) > 1 {
result.LatencyMs, _ = strconv.ParseFloat(parts[1], 64)
}
}
}
}
resolver := net.Resolver{}
_, err := resolver.LookupHost(ctx, "deb.debian.org")
result.DNSUp = err == nil
client := http.Client{Timeout: 4 * time.Second}
request, _ := http.NewRequestWithContext(ctx, http.MethodGet, "https://api.ipify.org", nil)
if response, err := client.Do(request); err == nil {
defer response.Body.Close()
body, _ := io.ReadAll(io.LimitReader(response.Body, 128))
result.ExternalIP = strings.TrimSpace(string(body))
result.InternetUp = response.StatusCode == 200 && net.ParseIP(result.ExternalIP) != nil
}
result.ExternalIPChanged = previousIP != "" && result.ExternalIP != "" && previousIP != result.ExternalIP
return result
}
func readNetworkSnapshot(path string) (networkSnapshot, []string, []NetworkInterface, error) {
file, err := os.Open(path)
if err != nil {
return networkSnapshot{}, nil, err
return networkSnapshot{}, nil, nil, err
}
defer file.Close()
var received, sent uint64
var interfaces []string
var details []NetworkInterface
scanner := bufio.NewScanner(file)
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
@@ -84,7 +241,12 @@ func readNetworkSnapshot(path string) (networkSnapshot, []string, error) {
received += rx
sent += tx
interfaces = append(interfaces, strings.TrimSpace(name))
rxErr, _ := strconv.ParseUint(fields[2], 10, 64)
rxDrop, _ := strconv.ParseUint(fields[3], 10, 64)
txErr, _ := strconv.ParseUint(fields[10], 10, 64)
txDrop, _ := strconv.ParseUint(fields[11], 10, 64)
details = append(details, NetworkInterface{Name: strings.TrimSpace(name), ReceiveBytes: rx, TransmitBytes: tx, ReceiveErrors: rxErr, ReceiveDrops: rxDrop, TransmitErrors: txErr, TransmitDrops: txDrop})
}
}
return networkSnapshot{at: time.Now(), received: received, sent: sent}, interfaces, scanner.Err()
return networkSnapshot{at: time.Now(), received: received, sent: sent}, interfaces, details, scanner.Err()
}

View File

@@ -18,7 +18,7 @@ func TestReadNetworkSnapshot(t *testing.T) {
t.Fatal(err)
}
snapshot, interfaces, err := readNetworkSnapshot(path)
snapshot, interfaces, details, err := readNetworkSnapshot(path)
if err != nil {
t.Fatal(err)
}
@@ -28,4 +28,15 @@ func TestReadNetworkSnapshot(t *testing.T) {
if len(interfaces) != 2 {
t.Fatalf("неожиданный список интерфейсов: %v", interfaces)
}
if len(details) != 2 {
t.Fatalf("unexpected interface details: %+v", details)
}
}
func TestInterfaceVMID(t *testing.T) {
for name, want := range map[string]int{"tap100i0": 100, "veth108i0": 108, "fwbr113i0": 113, "vmbr0": 0, "enp6s0": 0} {
if got := interfaceVMID(name); got != want {
t.Fatalf("%s: got %d want %d", name, got, want)
}
}
}

View File

@@ -4,7 +4,7 @@ After=network-online.target
Wants=network-online.target
[Service]
Type=simple
Type=notify
ExecStart=/usr/local/bin/proxmox-cpu-dashboard
Environment=DASHBOARD_ADDR=:9105
Environment=DASHBOARD_DB_PATH=/var/lib/proxmox-dashboard/dashboard.db
@@ -12,6 +12,8 @@ StateDirectory=proxmox-dashboard
StateDirectoryMode=0750
Restart=on-failure
RestartSec=3
WatchdogSec=30
TimeoutStartSec=20
NoNewPrivileges=true
ProtectHome=true
PrivateTmp=true

View File

@@ -0,0 +1,9 @@
[Unit]
Description=Watch for manual Proxmox Dashboard update requests
[Path]
PathExists=/var/lib/proxmox-dashboard/update-request
Unit=proxmox-dashboard-update.service
[Install]
WantedBy=multi-user.target

View File

@@ -0,0 +1,10 @@
[Unit]
Description=Check and install Proxmox Dashboard updates
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/proxmox-dashboard-update
Nice=10
IOSchedulingClass=idle

View File

@@ -0,0 +1,11 @@
[Unit]
Description=Check Proxmox Dashboard updates daily
[Timer]
OnBootSec=10min
OnUnitActiveSec=24h
RandomizedDelaySec=30min
Persistent=true
[Install]
WantedBy=timers.target

244
reliability.go Normal file
View File

@@ -0,0 +1,244 @@
package main
import (
"context"
"fmt"
"log"
"net"
"os"
"os/exec"
"path/filepath"
"sort"
"strings"
"sync"
"syscall"
"time"
)
func notifySystemd(message string) bool {
address := os.Getenv("NOTIFY_SOCKET")
if address == "" {
return false
}
if strings.HasPrefix(address, "@") {
address = "\x00" + address[1:]
}
connection, err := net.DialUnix("unixgram", nil, &net.UnixAddr{Name: address, Net: "unixgram"})
if err != nil {
return false
}
defer connection.Close()
_, err = connection.Write([]byte(message))
return err == nil
}
func runSystemdWatchdog() {
if !notifySystemd("READY=1\nSTATUS=Dashboard collects Proxmox metrics") {
return
}
go func() {
ticker := time.NewTicker(10 * time.Second)
defer ticker.Stop()
for range ticker.C {
notifySystemd("WATCHDOG=1")
}
}()
}
type MonitorHealth struct {
LastSuccessfulCollection time.Time `json:"lastSuccessfulCollection"`
LastCollectionError string `json:"lastCollectionError"`
CollectionDurationMs int64 `json:"collectionDurationMs"`
SlowCollector bool `json:"slowCollector"`
LastHistoryWrite time.Time `json:"lastHistoryWrite"`
HistoryWriteError string `json:"historyWriteError"`
DatabaseFreeBytes uint64 `json:"databaseFreeBytes"`
DatabasePath string `json:"databasePath"`
WatchdogEnabled bool `json:"watchdogEnabled"`
}
type monitorTracker struct {
mu sync.RWMutex
lastSuccess time.Time
lastError string
duration time.Duration
}
func runCollectionWatchdog(monitor *monitorTracker) {
started := time.Now()
ticker := time.NewTicker(15 * time.Second)
defer ticker.Stop()
for range ticker.C {
monitor.mu.RLock()
last := monitor.lastSuccess
monitor.mu.RUnlock()
if time.Since(started) < 2*time.Minute {
continue
}
if last.IsZero() || time.Since(last) > 90*time.Second {
log.Printf("Встроенный watchdog: сбор метрик не завершался более 90 секунд; перезапускаем процесс")
os.Exit(1)
}
}
}
func (m *monitorTracker) record(start time.Time, err error) {
m.mu.Lock()
defer m.mu.Unlock()
m.duration = time.Since(start)
if err != nil {
m.lastError = err.Error()
return
}
m.lastSuccess = time.Now()
m.lastError = ""
}
func (m *monitorTracker) snapshot(store *Store) MonitorHealth {
m.mu.RLock()
result := MonitorHealth{LastSuccessfulCollection: m.lastSuccess, LastCollectionError: m.lastError, CollectionDurationMs: m.duration.Milliseconds(), SlowCollector: m.duration > 10*time.Second, DatabasePath: store.path, WatchdogEnabled: os.Getenv("NOTIFY_SOCKET") != ""}
m.mu.RUnlock()
result.LastHistoryWrite, result.HistoryWriteError = store.historyHealth()
path := store.path
if path == ":memory:" {
path = "/tmp"
} else {
path = filepath.Dir(path)
}
var stat syscall.Statfs_t
if syscall.Statfs(path, &stat) == nil {
result.DatabaseFreeBytes = stat.Bavail * uint64(stat.Bsize)
}
return result
}
type UPSShutdownSettings struct {
Enabled bool `json:"enabled"`
ChargePercent float64 `json:"chargePercent"`
RuntimeSeconds float64 `json:"runtimeSeconds"`
GraceSeconds int `json:"graceSeconds"`
GuestOrder []int `json:"guestOrder"`
}
func defaultUPSShutdownSettings() UPSShutdownSettings {
return UPSShutdownSettings{ChargePercent: 15, RuntimeSeconds: 600, GraceSeconds: 120}
}
func (s *Store) UPSShutdownSettings() UPSShutdownSettings {
value := defaultUPSShutdownSettings()
s.loadJSONSetting("ups_shutdown", &value)
return value
}
func (s *Store) SaveUPSShutdownSettings(value UPSShutdownSettings) error {
if value.ChargePercent < 1 || value.ChargePercent > 80 || value.RuntimeSeconds < 60 || value.RuntimeSeconds > 7200 || value.GraceSeconds < 30 || value.GraceSeconds > 1800 {
return fmt.Errorf("проверьте пороги UPS и задержку")
}
return s.saveJSONSetting("ups_shutdown", value)
}
type ShutdownStep struct {
Order int `json:"order"`
Target string `json:"target"`
Action string `json:"action"`
Detail string `json:"detail"`
}
func buildShutdownDryRun(settings UPSShutdownSettings, guests GuestsMetrics) []ShutdownStep {
byID := map[int]GuestMetrics{}
for _, g := range guests.Guests {
if g.Template == 0 && g.Status == "running" {
byID[g.VMID] = g
}
}
var ordered []GuestMetrics
for _, id := range settings.GuestOrder {
if g, ok := byID[id]; ok {
ordered = append(ordered, g)
delete(byID, id)
}
}
var rest []GuestMetrics
for _, g := range byID {
rest = append(rest, g)
}
sort.Slice(rest, func(i, j int) bool { return rest[i].VMID < rest[j].VMID })
ordered = append(ordered, rest...)
steps := []ShutdownStep{{Order: 1, Target: "UPS", Action: "Ожидание порога", Detail: fmt.Sprintf("заряд ≤ %.0f%% или автономность ≤ %.0f сек.", settings.ChargePercent, settings.RuntimeSeconds)}}
for _, g := range ordered {
kind := "LXC"
if g.Type == "qemu" {
kind = "VM"
}
steps = append(steps, ShutdownStep{Order: len(steps) + 1, Target: fmt.Sprintf("%s %d · %s", kind, g.VMID, g.Name), Action: "Корректное завершение", Detail: fmt.Sprintf("ожидание до %d сек.", settings.GraceSeconds)})
}
steps = append(steps, ShutdownStep{Order: len(steps) + 1, Target: "Proxmox host", Action: "Завершение работы", Detail: "только после остановки VM/LXC"})
return steps
}
func runUPSShutdownController(store *Store, ups *upsCollector, guests *guestCollector) {
ticker := time.NewTicker(10 * time.Second)
defer ticker.Stop()
triggered := false
for range ticker.C {
settings := store.UPSShutdownSettings()
metrics := ups.collect()
onBattery := strings.Contains(metrics.Status, "OB") || metrics.Status == "ONBATT"
if !settings.Enabled || !metrics.Available || !onBattery {
if metrics.Available && !onBattery {
triggered = false
}
continue
}
lowCharge := metrics.ChargePercent != nil && *metrics.ChargePercent <= settings.ChargePercent
lowRuntime := metrics.RuntimeSeconds != nil && *metrics.RuntimeSeconds <= settings.RuntimeSeconds
if triggered || (!lowCharge && !lowRuntime) {
continue
}
triggered = true
log.Printf("UPS достиг порога безопасного выключения; запускаем остановку гостей")
go executeUPSShutdown(settings, guests.collect())
}
}
func executeUPSShutdown(settings UPSShutdownSettings, guests GuestsMetrics) {
steps := buildShutdownDryRun(settings, guests)
_ = steps
byID := map[int]GuestMetrics{}
for _, g := range guests.Guests {
if g.Template == 0 && g.Status == "running" {
byID[g.VMID] = g
}
}
var order []int
for _, id := range settings.GuestOrder {
if _, ok := byID[id]; ok {
order = append(order, id)
delete(byID, id)
}
}
var remaining []int
for id := range byID {
remaining = append(remaining, id)
}
sort.Ints(remaining)
order = append(order, remaining...)
for _, id := range order {
g := GuestMetrics{}
for _, candidate := range guests.Guests {
if candidate.VMID == id {
g = candidate
break
}
}
command := "pct"
if g.Type == "qemu" {
command = "qm"
}
ctx, cancel := context.WithTimeout(context.Background(), time.Duration(settings.GraceSeconds+30)*time.Second)
err := exec.CommandContext(ctx, command, "shutdown", fmt.Sprint(id), "--timeout", fmt.Sprint(settings.GraceSeconds)).Run()
cancel()
if err != nil {
log.Printf("UPS shutdown: не удалось остановить VMID %d: %v", id, err)
}
}
log.Printf("UPS shutdown: гости обработаны, выключаем Proxmox host")
_ = exec.Command("shutdown", "-h", "now").Run()
}

38
reliability_test.go Normal file
View File

@@ -0,0 +1,38 @@
package main
import (
"strings"
"testing"
"time"
)
func TestDiskCounterChanges(t *testing.T) {
old := DiskMetrics{Reallocated: 1, Pending: 2, CRCErrors: 4}
current := DiskMetrics{Reallocated: 2, Pending: 2, CRCErrors: 7}
changes := strings.Join(diskCounterChanges(old, current), " ")
if !strings.Contains(changes, "Reallocated") || !strings.Contains(changes, "CRC") || strings.Contains(changes, "Pending") {
t.Fatalf("unexpected counter changes: %s", changes)
}
}
func TestParseScrub(t *testing.T) {
status, at, duration, old := parseScrub("scrub repaired 0B in 01:12:03 with 0 errors on Fri Jan 2 03:04:05 2026")
if status != "Завершён" || at == "" || duration != "01:12:03" || !old {
t.Fatalf("unexpected scrub: %q %q %q %v", status, at, duration, old)
}
}
func TestShutdownDryRunUsesConfiguredOrder(t *testing.T) {
settings := UPSShutdownSettings{ChargePercent: 15, RuntimeSeconds: 600, GraceSeconds: 120, GuestOrder: []int{105}}
guests := GuestsMetrics{Guests: []GuestMetrics{{VMID: 101, Name: "one", Type: "qemu", Status: "running"}, {VMID: 105, Name: "five", Type: "lxc", Status: "running"}}}
steps := buildShutdownDryRun(settings, guests)
if len(steps) != 4 || !strings.Contains(steps[1].Target, "105") || !strings.Contains(steps[2].Target, "101") || steps[3].Target != "Proxmox host" {
t.Fatalf("unexpected plan: %+v", steps)
}
}
func TestSelfTestOlderThanWeek(t *testing.T) {
if !selfTestOlderThanWeek(time.Now().Add(-8*24*time.Hour).Format(time.RFC3339)) || selfTestOlderThanWeek(time.Now().Format(time.RFC3339)) {
t.Fatal("self-test age evaluation is incorrect")
}
}

63
scripts/install-agent.sh Executable file
View File

@@ -0,0 +1,63 @@
#!/bin/sh
set -eu
if [ "$(id -u)" -ne 0 ]; then
echo "Запустите команду установки от root." >&2
exit 1
fi
if [ -z "${DASHBOARD_URL:-}" ] || [ -z "${ENROLL_TOKEN:-}" ]; then
echo "Не заданы DASHBOARD_URL и ENROLL_TOKEN. Создайте команду в настройках Dashboard." >&2
exit 1
fi
GITEA_URL=${GITEA_URL:-https://git.myown.center}
GITEA_REPOSITORY=${GITEA_REPOSITORY:-maxim/ProxmoxDash}
VMID=${VMID:-0}
AGENT_NAME=${AGENT_NAME:-$(hostname)}
release_json=$(curl -fsSL "$GITEA_URL/api/v1/repos/$GITEA_REPOSITORY/releases/latest")
download_url=$(printf '%s' "$release_json" | sed -n 's/.*"browser_download_url":"\([^"]*\/proxmox-dashboard-linux-amd64\)".*/\1/p' | head -n 1)
if [ -z "$download_url" ]; then
echo "В последнем релизе не найден Linux-бинарник." >&2
exit 1
fi
install -d -m 0700 /var/lib/proxmox-dashboard-agent
temporary_binary=$(mktemp /tmp/proxmox-dashboard-agent.XXXXXX)
trap 'rm -f "$temporary_binary"' EXIT
curl -fsSL "$download_url" -o "$temporary_binary"
install -m 0755 "$temporary_binary" /usr/local/bin/proxmox-dashboard-agent
umask 077
{
printf 'DASHBOARD_URL=%s\n' "$DASHBOARD_URL"
printf 'ENROLL_TOKEN=%s\n' "$ENROLL_TOKEN"
printf 'AGENT_NAME=%s\n' "$AGENT_NAME"
printf 'VMID=%s\n' "$VMID"
} > /etc/proxmox-dashboard-agent.env
cat > /etc/systemd/system/proxmox-dashboard-agent.service <<'UNIT'
[Unit]
Description=Proxmox Dashboard read-only guest agent
After=network-online.target docker.service
Wants=network-online.target
[Service]
Type=simple
EnvironmentFile=/etc/proxmox-dashboard-agent.env
ExecStart=/usr/local/bin/proxmox-dashboard-agent --agent --agent-server ${DASHBOARD_URL} --agent-enroll-token ${ENROLL_TOKEN} --agent-name ${AGENT_NAME} --agent-vmid ${VMID}
Restart=always
RestartSec=5
NoNewPrivileges=true
ProtectHome=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/var/lib/proxmox-dashboard-agent
[Install]
WantedBy=multi-user.target
UNIT
systemctl daemon-reload
systemctl enable --now proxmox-dashboard-agent.service
echo "Агент установлен. Через несколько секунд он появится в Dashboard."

37
scripts/install.sh Executable file
View File

@@ -0,0 +1,37 @@
#!/bin/sh
set -eu
if [ "$(id -u)" -ne 0 ]; then
echo "Запустите установщик от root." >&2
exit 1
fi
SOURCE_DIR=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
UPDATE_MODE=${UPDATE_MODE:-notify}
GITEA_URL=${GITEA_URL:-https://git.myown.center}
GITEA_REPOSITORY=${GITEA_REPOSITORY:-maxim/ProxmoxDash}
GITEA_TOKEN=${GITEA_TOKEN:-}
install -d -m 0750 /etc/proxmox-dashboard /var/lib/proxmox-dashboard
install -m 0755 "$SOURCE_DIR/dist/proxmox-dashboard-linux-amd64" /usr/local/bin/proxmox-cpu-dashboard
install -m 0755 "$SOURCE_DIR/scripts/proxmox-dashboard-update" /usr/local/sbin/proxmox-dashboard-update
install -m 0644 "$SOURCE_DIR/proxmox-cpu-dashboard.service" /etc/systemd/system/proxmox-cpu-dashboard.service
install -m 0644 "$SOURCE_DIR/proxmox-dashboard-update.service" /etc/systemd/system/proxmox-dashboard-update.service
install -m 0644 "$SOURCE_DIR/proxmox-dashboard-update.timer" /etc/systemd/system/proxmox-dashboard-update.timer
install -m 0644 "$SOURCE_DIR/proxmox-dashboard-update.path" /etc/systemd/system/proxmox-dashboard-update.path
umask 077
{
printf 'GITEA_URL=%s\n' "$GITEA_URL"
printf 'GITEA_REPOSITORY=%s\n' "$GITEA_REPOSITORY"
printf 'GITEA_TOKEN=%s\n' "$GITEA_TOKEN"
printf 'UPDATE_MODE=%s\n' "$UPDATE_MODE"
} > /etc/proxmox-dashboard/update.env
systemctl daemon-reload
systemctl enable proxmox-cpu-dashboard.service
systemctl restart proxmox-cpu-dashboard.service
systemctl enable --now proxmox-dashboard-update.timer
systemctl enable --now proxmox-dashboard-update.path
systemctl start proxmox-dashboard-update.service
echo "Proxmox Dashboard установлен. Откройте http://$(hostname -I | awk '{print $1}'):9105"

111
scripts/proxmox-dashboard-update Executable file
View File

@@ -0,0 +1,111 @@
#!/bin/sh
set -eu
CONFIG_FILE=${DASHBOARD_UPDATE_CONFIG:-/etc/proxmox-dashboard/update.env}
STATUS_FILE=${DASHBOARD_UPDATE_STATUS_PATH:-/var/lib/proxmox-dashboard/update-status.json}
BINARY_PATH=${DASHBOARD_BINARY_PATH:-/usr/local/bin/proxmox-cpu-dashboard}
REQUEST_FILE=${DASHBOARD_UPDATE_REQUEST_PATH:-/var/lib/proxmox-dashboard/update-request}
if [ -f "$CONFIG_FILE" ]; then
# shellcheck disable=SC1090
. "$CONFIG_FILE"
fi
GITEA_URL=${GITEA_URL:-https://git.myown.center}
GITEA_REPOSITORY=${GITEA_REPOSITORY:-maxim/ProxmoxDash}
GITEA_TOKEN=${GITEA_TOKEN:-}
UPDATE_MODE=${UPDATE_MODE:-notify}
ASSET_NAME=${ASSET_NAME:-proxmox-dashboard-linux-amd64}
manual_request=false
if [ -f "$REQUEST_FILE" ]; then
manual_request=true
rm -f "$REQUEST_FILE"
fi
mkdir -p "$(dirname "$STATUS_FILE")"
tmp_dir=$(mktemp -d /tmp/proxmox-dashboard-update.XXXXXX)
trap 'rm -rf "$tmp_dir"' EXIT INT TERM
auth_header=""
if [ -n "$GITEA_TOKEN" ]; then
auth_header="Authorization: token $GITEA_TOKEN"
fi
api_get() {
if [ -n "$auth_header" ]; then
curl --fail --silent --show-error --location --header "$auth_header" "$1"
else
curl --fail --silent --show-error --location "$1"
fi
}
write_status() {
python3 - "$STATUS_FILE" "$1" "$2" "$3" "$4" "$5" "$6" <<'PY'
import datetime, json, os, sys, tempfile
path, current, available, state, mode, message, release_url = sys.argv[1:]
payload = {
"currentVersion": current,
"availableVersion": available,
"state": state,
"mode": mode,
"message": message,
"releaseUrl": release_url,
"checkedAt": datetime.datetime.now(datetime.timezone.utc).isoformat(),
}
fd, temporary = tempfile.mkstemp(dir=os.path.dirname(path), prefix=".update-status-")
with os.fdopen(fd, "w") as stream:
json.dump(payload, stream)
os.chmod(temporary, 0o644)
os.replace(temporary, path)
PY
}
current_version=$($BINARY_PATH -version 2>/dev/null || printf 'dev')
release_json=$(api_get "$GITEA_URL/api/v1/repos/$GITEA_REPOSITORY/releases/latest")
printf '%s' "$release_json" > "$tmp_dir/release.json"
release_data=$(python3 - "$tmp_dir/release.json" "$ASSET_NAME" <<'PY'
import json, sys
release = json.load(open(sys.argv[1]))
asset_name = sys.argv[2]
assets = {item.get("name"): item.get("browser_download_url") for item in release.get("assets", [])}
binary = assets.get(asset_name, "")
checksum = assets.get(asset_name + ".sha256", "")
print(release.get("tag_name", ""))
print(release.get("html_url", ""))
print(binary)
print(checksum)
PY
)
latest_version=$(printf '%s\n' "$release_data" | sed -n '1p')
release_url=$(printf '%s\n' "$release_data" | sed -n '2p')
binary_url=$(printf '%s\n' "$release_data" | sed -n '3p')
checksum_url=$(printf '%s\n' "$release_data" | sed -n '4p')
if [ -z "$latest_version" ] || [ -z "$binary_url" ]; then
write_status "$current_version" "$latest_version" error "$UPDATE_MODE" "В свежем релизе не найден бинарник $ASSET_NAME." "$release_url"
exit 1
fi
newest=$(printf '%s\n%s\n' "$current_version" "$latest_version" | sort -V | tail -n 1)
if [ "$current_version" = "$latest_version" ] || [ "$newest" = "$current_version" ]; then
write_status "$current_version" "$latest_version" current "$UPDATE_MODE" "Установлена свежая версия." "$release_url"
exit 0
fi
if [ "$UPDATE_MODE" != "auto" ] && [ "$manual_request" != "true" ]; then
write_status "$current_version" "$latest_version" available "$UPDATE_MODE" "Доступна новая версия $latest_version." "$release_url"
exit 0
fi
write_status "$current_version" "$latest_version" installing "$UPDATE_MODE" "Скачиваем и проверяем $latest_version." "$release_url"
api_get "$binary_url" > "$tmp_dir/$ASSET_NAME"
if [ -n "$checksum_url" ]; then
api_get "$checksum_url" > "$tmp_dir/$ASSET_NAME.sha256"
(cd "$tmp_dir" && sha256sum -c "$ASSET_NAME.sha256")
fi
chmod 0755 "$tmp_dir/$ASSET_NAME"
file "$tmp_dir/$ASSET_NAME" | grep -q 'ELF 64-bit'
cp -p "$BINARY_PATH" "$BINARY_PATH.previous"
install -m 0755 "$tmp_dir/$ASSET_NAME" "$BINARY_PATH"
write_status "$latest_version" "$latest_version" updated "$UPDATE_MODE" "Обновление $latest_version установлено." "$release_url"
systemctl restart proxmox-cpu-dashboard.service

353
services.go Normal file
View File

@@ -0,0 +1,353 @@
package main
import (
"context"
"crypto/tls"
"database/sql"
"fmt"
"io"
"net"
"net/http"
"net/url"
"os/exec"
"strconv"
"strings"
"sync"
"time"
)
type MonitoredService struct {
ID int64 `json:"id"`
Name string `json:"name"`
URL string `json:"url"`
LocalAddress string `json:"localAddress"`
CheckType string `json:"checkType"`
ExpectedCode int `json:"expectedCode"`
ExpectedText string `json:"expectedText"`
TimeoutSeconds int `json:"timeoutSeconds"`
Enabled bool `json:"enabled"`
}
type ServiceStatus struct {
MonitoredService
Endpoints []ServiceEndpointStatus `json:"endpoints"`
Up bool `json:"up"`
CheckedAt int64 `json:"checkedAt"`
LatencyMS float64 `json:"latencyMs"`
StatusCode int `json:"statusCode"`
Error string `json:"error,omitempty"`
TLSExpiry *int64 `json:"tlsExpiry,omitempty"`
TLSDays *int `json:"tlsDays,omitempty"`
UptimeDay float64 `json:"uptimeDay"`
UptimeWeek float64 `json:"uptimeWeek"`
}
type ServiceEndpointStatus struct {
Kind string `json:"kind"`
Label string `json:"label"`
Address string `json:"address"`
Up bool `json:"up"`
CheckedAt int64 `json:"checkedAt"`
LatencyMS float64 `json:"latencyMs"`
StatusCode int `json:"statusCode"`
Error string `json:"error,omitempty"`
TLSExpiry *int64 `json:"tlsExpiry,omitempty"`
TLSDays *int `json:"tlsDays,omitempty"`
TLSSubject string `json:"tlsSubject,omitempty"`
TLSIssuer string `json:"tlsIssuer,omitempty"`
TLSDNSNames []string `json:"tlsDnsNames,omitempty"`
TLSHostnameValid bool `json:"tlsHostnameValid"`
TLSHostnameError string `json:"tlsHostnameError,omitempty"`
UptimeDay float64 `json:"uptimeDay"`
UptimeWeek float64 `json:"uptimeWeek"`
}
type ServicesMetrics struct {
Available bool `json:"available"`
Up int `json:"up"`
Total int `json:"total"`
Services []ServiceStatus `json:"services"`
}
type ServiceHistoryPoint struct {
Timestamp int64 `json:"timestamp"`
Up bool `json:"up"`
LatencyMS float64 `json:"latencyMs"`
}
var serviceHTTPTransport = &http.Transport{
Proxy: http.ProxyFromEnvironment,
DialContext: (&net.Dialer{Timeout: 10 * time.Second, KeepAlive: 30 * time.Second}).DialContext,
ForceAttemptHTTP2: true,
MaxIdleConns: 32,
MaxIdleConnsPerHost: 2,
IdleConnTimeout: 30 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
ExpectContinueTimeout: time.Second,
TLSClientConfig: &tls.Config{InsecureSkipVerify: true, MinVersion: tls.VersionTLS12},
}
var serviceHTTPClient = &http.Client{Transport: serviceHTTPTransport}
func (s *Store) Services() ([]MonitoredService, error) {
rows, err := s.db.Query(`SELECT id,name,url,local_address,check_type,expected_code,expected_text,timeout_seconds,enabled FROM services ORDER BY id`)
if err != nil {
return nil, err
}
defer rows.Close()
var result []MonitoredService
for rows.Next() {
var v MonitoredService
if err = rows.Scan(&v.ID, &v.Name, &v.URL, &v.LocalAddress, &v.CheckType, &v.ExpectedCode, &v.ExpectedText, &v.TimeoutSeconds, &v.Enabled); err != nil {
return nil, err
}
result = append(result, v)
}
return result, rows.Err()
}
func validateService(v MonitoredService) error {
v.Name = strings.TrimSpace(v.Name)
v.URL = strings.TrimSpace(v.URL)
v.LocalAddress = strings.TrimSpace(v.LocalAddress)
if v.Name == "" || (v.URL == "" && v.LocalAddress == "") {
return fmt.Errorf("укажите название и хотя бы один адрес")
}
if v.CheckType != "http" && v.CheckType != "tcp" && v.CheckType != "ping" {
return fmt.Errorf("неизвестный тип проверки")
}
if v.TimeoutSeconds < 1 || v.TimeoutSeconds > 30 {
return fmt.Errorf("таймаут должен быть от 1 до 30 секунд")
}
if v.CheckType == "http" {
for _, target := range []string{v.URL, v.LocalAddress} {
if target == "" {
continue
}
u, err := url.Parse(target)
if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
return fmt.Errorf("оба адреса должны начинаться с http:// или https://")
}
}
}
return nil
}
func (s *Store) SaveService(v MonitoredService) (MonitoredService, error) {
if err := validateService(v); err != nil {
return v, err
}
if v.ExpectedCode == 0 {
v.ExpectedCode = 200
}
if v.ID == 0 {
r, err := s.db.Exec(`INSERT INTO services(name,url,local_address,check_type,expected_code,expected_text,timeout_seconds,enabled) VALUES(?,?,?,?,?,?,?,?)`, v.Name, v.URL, v.LocalAddress, v.CheckType, v.ExpectedCode, v.ExpectedText, v.TimeoutSeconds, v.Enabled)
if err != nil {
return v, err
}
v.ID, _ = r.LastInsertId()
} else {
_, err := s.db.Exec(`UPDATE services SET name=?,url=?,local_address=?,check_type=?,expected_code=?,expected_text=?,timeout_seconds=?,enabled=? WHERE id=?`, v.Name, v.URL, v.LocalAddress, v.CheckType, v.ExpectedCode, v.ExpectedText, v.TimeoutSeconds, v.Enabled, v.ID)
if err != nil {
return v, err
}
}
return v, nil
}
func (s *Store) DeleteService(id int64) error {
_, err := s.db.Exec(`DELETE FROM services WHERE id=?`, id)
return err
}
func (s *Store) addServiceCheck(id int64, endpoint string, v ServiceEndpointStatus) {
var expiry any
if v.TLSExpiry != nil {
expiry = *v.TLSExpiry
}
_, _ = s.db.Exec(`INSERT INTO service_checks(service_id,ts,up,latency_ms,status_code,error,tls_expiry,endpoint) VALUES(?,?,?,?,?,?,?,?)`, id, v.CheckedAt, v.Up, v.LatencyMS, v.StatusCode, v.Error, expiry, endpoint)
_, _ = s.db.Exec(`DELETE FROM service_checks WHERE ts<?`, time.Now().Add(-historyRetention).Unix())
}
func (s *Store) serviceUptime(id int64, endpoint string, since time.Time) float64 {
var value sql.NullFloat64
_ = s.db.QueryRow(`SELECT AVG(up)*100 FROM service_checks WHERE service_id=? AND endpoint=? AND ts>=?`, id, endpoint, since.Unix()).Scan(&value)
if value.Valid {
return value.Float64
}
return 0
}
func (s *Store) ServiceHistory(id int64, hours int) ([]ServiceHistoryPoint, error) {
if hours < 1 {
hours = 24
}
if hours > 168 {
hours = 168
}
rows, err := s.db.Query(`SELECT ts,up,latency_ms FROM service_checks WHERE service_id=? AND ts>=? ORDER BY ts`, id, time.Now().Add(-time.Duration(hours)*time.Hour).Unix())
if err != nil {
return nil, err
}
defer rows.Close()
var out []ServiceHistoryPoint
for rows.Next() {
var p ServiceHistoryPoint
if err = rows.Scan(&p.Timestamp, &p.Up, &p.LatencyMS); err != nil {
return nil, err
}
out = append(out, p)
}
return out, rows.Err()
}
type serviceCollector struct {
mu sync.RWMutex
metrics ServicesMetrics
store *Store
}
func newServiceCollector(store *Store) *serviceCollector {
c := &serviceCollector{store: store}
go c.run()
return c
}
func (c *serviceCollector) collect() ServicesMetrics {
c.mu.RLock()
defer c.mu.RUnlock()
return c.metrics
}
func (c *serviceCollector) run() {
c.checkAll()
ticker := time.NewTicker(15 * time.Second)
defer ticker.Stop()
for range ticker.C {
c.checkAll()
}
}
func (c *serviceCollector) checkAll() {
items, err := c.store.Services()
if err != nil {
return
}
m := ServicesMetrics{Available: true}
for _, v := range items {
if !v.Enabled {
continue
}
st := checkService(v)
for i := range st.Endpoints {
ep := &st.Endpoints[i]
ep.UptimeDay = c.store.serviceUptime(v.ID, ep.Kind, time.Now().Add(-24*time.Hour))
ep.UptimeWeek = c.store.serviceUptime(v.ID, ep.Kind, time.Now().Add(-7*24*time.Hour))
c.store.addServiceCheck(v.ID, ep.Kind, *ep)
}
if len(st.Endpoints) > 0 {
st.UptimeDay = st.Endpoints[0].UptimeDay
st.UptimeWeek = st.Endpoints[0].UptimeWeek
}
m.Total++
if st.Up {
m.Up++
}
m.Services = append(m.Services, st)
}
c.mu.Lock()
c.metrics = m
c.mu.Unlock()
}
func checkService(v MonitoredService) ServiceStatus {
st := ServiceStatus{MonitoredService: v, CheckedAt: time.Now().Unix(), Up: true}
if v.URL != "" {
st.Endpoints = append(st.Endpoints, checkServiceEndpoint(v, "public", "Внешний адрес", v.URL))
}
if v.LocalAddress != "" {
st.Endpoints = append(st.Endpoints, checkServiceEndpoint(v, "local", "Локальный адрес", v.LocalAddress))
}
for _, ep := range st.Endpoints {
if !ep.Up {
st.Up = false
}
st.LatencyMS += ep.LatencyMS
if st.StatusCode == 0 {
st.StatusCode = ep.StatusCode
}
if st.Error == "" && ep.Error != "" {
st.Error = ep.Label + ": " + ep.Error
}
}
if len(st.Endpoints) > 0 {
st.LatencyMS /= float64(len(st.Endpoints))
ep := st.Endpoints[0]
st.TLSExpiry = ep.TLSExpiry
st.TLSDays = ep.TLSDays
}
return st
}
func checkServiceEndpoint(v MonitoredService, kind, label, target string) ServiceEndpointStatus {
st := ServiceEndpointStatus{Kind: kind, Label: label, Address: target, CheckedAt: time.Now().Unix()}
start := time.Now()
ctx, cancel := context.WithTimeout(context.Background(), time.Duration(v.TimeoutSeconds)*time.Second)
defer cancel()
switch v.CheckType {
case "tcp":
d := net.Dialer{}
conn, err := d.DialContext(ctx, "tcp", target)
if err == nil {
st.Up = true
conn.Close()
} else {
st.Error = err.Error()
}
case "ping":
host := strings.TrimPrefix(strings.TrimPrefix(target, "https://"), "http://")
host = strings.Split(host, "/")[0]
host = strings.Split(host, ":")[0]
arg := "-W"
timeout := strconv.Itoa(v.TimeoutSeconds)
if _, err := exec.LookPath("ping"); err != nil {
st.Error = "ping не установлен"
} else if out, err := exec.CommandContext(ctx, "ping", "-c", "1", arg, timeout, host).CombinedOutput(); err == nil {
st.Up = true
} else {
st.Error = strings.TrimSpace(string(out))
}
default:
req, err := http.NewRequestWithContext(ctx, http.MethodGet, target, nil)
if err != nil {
st.Error = err.Error()
break
}
resp, err := serviceHTTPClient.Do(req)
if err != nil {
st.Error = err.Error()
} else {
st.StatusCode = resp.StatusCode
body, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
resp.Body.Close()
st.Up = resp.StatusCode == v.ExpectedCode
if st.Up && v.ExpectedText != "" && !strings.Contains(string(body), v.ExpectedText) {
st.Up = false
st.Error = "ожидаемый текст не найден"
}
if !st.Up && st.Error == "" {
st.Error = fmt.Sprintf("HTTP %d, ожидался %d", resp.StatusCode, v.ExpectedCode)
}
if resp.TLS != nil && len(resp.TLS.PeerCertificates) > 0 {
cert := resp.TLS.PeerCertificates[0]
expiry := cert.NotAfter.Unix()
days := int(time.Until(cert.NotAfter).Hours() / 24)
st.TLSExpiry = &expiry
st.TLSDays = &days
st.TLSSubject = cert.Subject.CommonName
st.TLSIssuer = cert.Issuer.CommonName
st.TLSDNSNames = cert.DNSNames
host := req.URL.Hostname()
if err := cert.VerifyHostname(host); err == nil {
st.TLSHostnameValid = true
} else {
st.TLSHostnameError = err.Error()
}
}
}
}
st.LatencyMS = float64(time.Since(start).Microseconds()) / 1000
return st
}

55
services_test.go Normal file
View File

@@ -0,0 +1,55 @@
package main
import (
"testing"
"time"
)
func TestServiceCRUD(t *testing.T) {
store, err := openStore(":memory:")
if err != nil {
t.Fatal(err)
}
defer store.Close()
v, err := store.SaveService(MonitoredService{Name: "Test", URL: "https://example.test", CheckType: "http", ExpectedCode: 204, TimeoutSeconds: 2, Enabled: true})
if err != nil || v.ID == 0 {
t.Fatalf("save: %+v %v", v, err)
}
items, err := store.Services()
if err != nil || len(items) != 1 {
t.Fatalf("list: %+v %v", items, err)
}
if err := store.DeleteService(v.ID); err != nil {
t.Fatal(err)
}
}
func TestValidateServiceAllowsLocalAddress(t *testing.T) {
v := MonitoredService{Name: "Host", LocalAddress: "192.168.0.12", CheckType: "ping", TimeoutSeconds: 5}
if err := validateService(v); err != nil {
t.Fatal(err)
}
}
func TestValidateServiceChecksBothHTTPAddresses(t *testing.T) {
v := MonitoredService{Name: "Gitea", URL: "https://git.example.test", LocalAddress: "192.168.0.6:3000", CheckType: "http", TimeoutSeconds: 5}
if err := validateService(v); err == nil {
t.Fatal("expected invalid local HTTP address")
}
v.LocalAddress = "https://192.168.0.6:3000"
if err := validateService(v); err != nil {
t.Fatal(err)
}
}
func TestHTTPServiceTransportHasBoundedIdlePool(t *testing.T) {
if serviceHTTPClient.Transport != serviceHTTPTransport {
t.Fatal("проверки сервисов должны использовать общий HTTP transport")
}
if serviceHTTPTransport.IdleConnTimeout < time.Second {
t.Fatal("idle-соединения должны автоматически закрываться")
}
if serviceHTTPTransport.MaxIdleConnsPerHost < 1 || serviceHTTPTransport.MaxIdleConnsPerHost > 4 {
t.Fatalf("небезопасный размер пула на хост: %d", serviceHTTPTransport.MaxIdleConnsPerHost)
}
}

222
store.go
View File

@@ -6,6 +6,8 @@ import (
"fmt"
"os"
"path/filepath"
"strings"
"sync"
"time"
_ "modernc.org/sqlite"
@@ -29,6 +31,17 @@ type AlertThresholds struct {
NVMeTempWarning float64 `json:"nvmeTempWarning"`
UPSChargeCritical float64 `json:"upsChargeCritical"`
BackupMaxAgeHours float64 `json:"backupMaxAgeHours"`
DiskHDDUtil float64 `json:"diskHddUtil"`
DiskHDDLatency float64 `json:"diskHddLatency"`
DiskHDDQueue float64 `json:"diskHddQueue"`
DiskSSDUtil float64 `json:"diskSsdUtil"`
DiskSSDLatency float64 `json:"diskSsdLatency"`
DiskSSDQueue float64 `json:"diskSsdQueue"`
DiskNVMeUtil float64 `json:"diskNvmeUtil"`
DiskNVMeLatency float64 `json:"diskNvmeLatency"`
DiskNVMeQueue float64 `json:"diskNvmeQueue"`
DiskIODuration float64 `json:"diskIoDuration"`
DiskIOExcluded string `json:"diskIoExcluded"`
}
func defaultThresholds() AlertThresholds {
@@ -38,6 +51,7 @@ func defaultThresholds() AlertThresholds {
StorageWarning: 85, StorageCritical: 95, ZFSWarning: 80, ZFSCritical: 90,
DiskTempWarning: 60, NVMeTempWarning: 75, UPSChargeCritical: 30,
BackupMaxAgeHours: 48,
DiskHDDUtil: 90, DiskHDDLatency: 50, DiskHDDQueue: 4, DiskSSDUtil: 90, DiskSSDLatency: 20, DiskSSDQueue: 4, DiskNVMeUtil: 95, DiskNVMeLatency: 10, DiskNVMeQueue: 8, DiskIODuration: 60,
}
}
@@ -57,7 +71,28 @@ type HistoryPoint struct {
GuestsMemory float64 `json:"guestsMemory"`
}
type Store struct{ db *sql.DB }
type Store struct {
db *sql.DB
path string
healthMu sync.RWMutex
lastHistoryAt time.Time
lastHistoryError string
}
type EmailSettings struct {
Enabled bool `json:"enabled"`
Host string `json:"host"`
Port int `json:"port"`
Username string `json:"username"`
Password string `json:"password,omitempty"`
HasPassword bool `json:"hasPassword"`
SkipTLSVerify bool `json:"skipTlsVerify"`
From string `json:"from"`
To string `json:"to"`
IntervalMinutes int `json:"intervalMinutes"`
Severities []string `json:"severities"`
Sources []string `json:"sources"`
}
func openStore(path string) (*Store, error) {
if path != ":memory:" {
@@ -81,6 +116,30 @@ func openStore(path string) (*Store, error) {
zfs_usage REAL NOT NULL, guests_cpu REAL NOT NULL, guests_memory REAL NOT NULL
)`,
`CREATE INDEX IF NOT EXISTS history_ts_idx ON history(ts)`,
`CREATE TABLE IF NOT EXISTS alert_events (
id INTEGER PRIMARY KEY AUTOINCREMENT, alert_key TEXT NOT NULL, severity TEXT NOT NULL,
source TEXT NOT NULL, title TEXT NOT NULL, message TEXT NOT NULL,
first_seen INTEGER NOT NULL, last_seen INTEGER NOT NULL, resolved_at INTEGER,
acknowledged_at INTEGER, email_sent_at INTEGER
)`,
`CREATE INDEX IF NOT EXISTS alert_events_active_idx ON alert_events(alert_key,resolved_at)`,
`CREATE TABLE IF NOT EXISTS services (
id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT NOT NULL, url TEXT NOT NULL,
local_address TEXT NOT NULL DEFAULT '', check_type TEXT NOT NULL DEFAULT 'http',
expected_code INTEGER NOT NULL DEFAULT 200, expected_text TEXT NOT NULL DEFAULT '',
timeout_seconds INTEGER NOT NULL DEFAULT 5, enabled INTEGER NOT NULL DEFAULT 1
)`,
`CREATE TABLE IF NOT EXISTS service_checks (
id INTEGER PRIMARY KEY AUTOINCREMENT, service_id INTEGER NOT NULL, ts INTEGER NOT NULL,
up INTEGER NOT NULL, latency_ms REAL NOT NULL, status_code INTEGER NOT NULL DEFAULT 0,
error TEXT NOT NULL DEFAULT '', tls_expiry INTEGER,
FOREIGN KEY(service_id) REFERENCES services(id) ON DELETE CASCADE
)`,
`CREATE INDEX IF NOT EXISTS service_checks_service_ts_idx ON service_checks(service_id,ts)`,
`CREATE TABLE IF NOT EXISTS entity_history (kind TEXT NOT NULL,name TEXT NOT NULL,ts INTEGER NOT NULL,value1 REAL,value2 REAL,PRIMARY KEY(kind,name,ts))`,
`CREATE INDEX IF NOT EXISTS entity_history_kind_ts_idx ON entity_history(kind,ts)`,
`CREATE TABLE IF NOT EXISTS agent_enrollment_tokens (token_hash TEXT PRIMARY KEY, created_at INTEGER NOT NULL, expires_at INTEGER NOT NULL, used_at INTEGER)`,
`CREATE TABLE IF NOT EXISTS agents (id TEXT PRIMARY KEY, name TEXT NOT NULL, vmid INTEGER NOT NULL DEFAULT 0, hostname TEXT NOT NULL DEFAULT '', machine_id TEXT NOT NULL DEFAULT '', secret_hash TEXT NOT NULL, enrolled_at INTEGER NOT NULL, last_seen INTEGER NOT NULL, version TEXT NOT NULL DEFAULT '', report_json TEXT NOT NULL DEFAULT '{}', remote_address TEXT NOT NULL DEFAULT '')`,
}
for _, statement := range statements {
if _, err := db.Exec(statement); err != nil {
@@ -88,7 +147,68 @@ func openStore(path string) (*Store, error) {
return nil, err
}
}
return &Store{db: db}, nil
// Older databases get this column without requiring a separate migration tool.
_, _ = db.Exec(`ALTER TABLE alert_events ADD COLUMN recovery_sent_at INTEGER`)
_, _ = db.Exec(`ALTER TABLE alert_events ADD COLUMN planned INTEGER NOT NULL DEFAULT 0`)
_, _ = db.Exec(`ALTER TABLE alert_events ADD COLUMN maintenance_reason TEXT NOT NULL DEFAULT ''`)
_, _ = db.Exec(`DELETE FROM alert_events WHERE alert_key='monitor-slow'`)
_, _ = db.Exec(`ALTER TABLE service_checks ADD COLUMN endpoint TEXT NOT NULL DEFAULT 'public'`)
_, _ = db.Exec(`PRAGMA foreign_keys=ON`)
return &Store{db: db, path: path}, nil
}
type EntityHistoryPoint struct {
Timestamp int64 `json:"timestamp"`
Value1 *float64 `json:"value1,omitempty"`
Value2 *float64 `json:"value2,omitempty"`
}
type EntityHistorySeries struct {
Name string `json:"name"`
Points []EntityHistoryPoint `json:"points"`
}
func (s *Store) AddEntityHistory(kind, name string, ts int64, v1, v2 *float64) {
_, _ = s.db.Exec(`INSERT OR REPLACE INTO entity_history(kind,name,ts,value1,value2) VALUES(?,?,?,?,?)`, kind, name, ts, v1, v2)
}
func (s *Store) EntityHistory(kind string, hours int) ([]EntityHistorySeries, error) {
if hours < 1 {
hours = 24
}
if hours > 168 {
hours = 168
}
rows, err := s.db.Query(`SELECT name,ts,value1,value2 FROM entity_history WHERE kind=? AND ts>=? ORDER BY name,ts`, kind, time.Now().Add(-time.Duration(hours)*time.Hour).Unix())
if err != nil {
return nil, err
}
defer rows.Close()
var out []EntityHistorySeries
index := map[string]int{}
for rows.Next() {
var name string
var p EntityHistoryPoint
var v1, v2 sql.NullFloat64
if err = rows.Scan(&name, &p.Timestamp, &v1, &v2); err != nil {
return nil, err
}
if v1.Valid {
p.Value1 = &v1.Float64
}
if v2.Valid {
p.Value2 = &v2.Float64
}
i, ok := index[name]
if !ok {
i = len(out)
index[name] = i
out = append(out, EntityHistorySeries{Name: name})
}
out[i].Points = append(out[i].Points, p)
}
return out, rows.Err()
}
func (s *Store) PruneEntityHistory() {
_, _ = s.db.Exec(`DELETE FROM entity_history WHERE ts<?`, time.Now().Add(-historyRetention).Unix())
}
func (s *Store) Close() error { return s.db.Close() }
@@ -114,6 +234,55 @@ func (s *Store) SaveThresholds(value AlertThresholds) error {
return err
}
func defaultEmailSettings() EmailSettings {
return EmailSettings{Port: 587, IntervalMinutes: 60, Severities: []string{"critical", "warning"}, Sources: []string{"CPU", "Память", "Storage", "Диски", "ZFS", "UPS", "VM/LXC", "Proxmox", "Dashboard", "Сервисы", "Система", "Сеть", "Агенты", "Docker"}}
}
func (s *Store) EmailSettings(includePassword bool) EmailSettings {
settings := defaultEmailSettings()
var value string
if s.db.QueryRow(`SELECT value FROM settings WHERE key='email_notifications'`).Scan(&value) == nil {
_ = json.Unmarshal([]byte(value), &settings)
}
settings.HasPassword = settings.Password != ""
if !includePassword {
settings.Password = ""
}
return settings
}
func (s *Store) SaveEmailSettings(settings EmailSettings) error {
if settings.Port < 1 || settings.Port > 65535 || settings.IntervalMinutes < 5 || settings.IntervalMinutes > 10080 {
return fmt.Errorf("проверьте порт и интервал (от 5 минут до 7 дней)")
}
if settings.Enabled && (settings.Host == "" || settings.From == "" || settings.To == "") {
return fmt.Errorf("для отправки нужны SMTP-сервер, отправитель и получатель")
}
if strings.ContainsAny(settings.Host+settings.From+settings.To+settings.Username, "\r\n") {
return fmt.Errorf("поля почты содержат недопустимые символы")
}
if settings.Password == "" {
settings.Password = s.EmailSettings(true).Password
}
settings.HasPassword = settings.Password != ""
data, _ := json.Marshal(settings)
_, err := s.db.Exec(`INSERT INTO settings(key,value) VALUES('email_notifications',?) ON CONFLICT(key) DO UPDATE SET value=excluded.value`, string(data))
return err
}
func (s *Store) LastEmailAt() time.Time {
var value string
if s.db.QueryRow(`SELECT value FROM settings WHERE key='email_last_sent'`).Scan(&value) != nil {
return time.Time{}
}
result, _ := time.Parse(time.RFC3339, value)
return result
}
func (s *Store) SetLastEmailAt(value time.Time) {
_, _ = s.db.Exec(`INSERT INTO settings(key,value) VALUES('email_last_sent',?) ON CONFLICT(key) DO UPDATE SET value=excluded.value`, value.Format(time.RFC3339))
}
func validateThresholds(t AlertThresholds) error {
percentages := []float64{t.CPUWarning, t.CPUCritical, t.MemoryWarning, t.MemoryCritical, t.SwapWarning, t.StorageWarning, t.StorageCritical, t.ZFSWarning, t.ZFSCritical, t.UPSChargeCritical}
for _, value := range percentages {
@@ -133,6 +302,19 @@ func validateThresholds(t AlertThresholds) error {
if t.BackupMaxAgeHours < 1 || t.BackupMaxAgeHours > 24*365 {
return fmt.Errorf("возраст backup должен быть от 1 до 8760 часов")
}
for _, value := range []float64{t.DiskHDDUtil, t.DiskSSDUtil, t.DiskNVMeUtil} {
if value < 1 || value > 100 {
return fmt.Errorf("пороги utilization должны быть от 1 до 100")
}
}
for _, value := range []float64{t.DiskHDDLatency, t.DiskSSDLatency, t.DiskNVMeLatency, t.DiskHDDQueue, t.DiskSSDQueue, t.DiskNVMeQueue} {
if value < 0 {
return fmt.Errorf("пороги latency и очереди не могут быть отрицательными")
}
}
if t.DiskIODuration < 5 || t.DiskIODuration > 3600 {
return fmt.Errorf("длительность I/O-проблемы должна быть от 5 до 3600 секунд")
}
return nil
}
@@ -140,9 +322,45 @@ func (s *Store) AddHistory(point HistoryPoint) error {
_, err := s.db.Exec(`INSERT OR REPLACE INTO history(ts,cpu_usage,cpu_temp,memory_usage,swap_usage,root_usage,network_rx,network_tx,ups_charge,ups_load,zfs_usage,guests_cpu,guests_memory) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?)`,
point.Timestamp, point.CPUUsage, point.CPUTemperature, point.MemoryUsage, point.SwapUsage, point.RootUsage, point.NetworkReceive, point.NetworkTransmit, point.UPSCharge, point.UPSLoad, point.ZFSUsage, point.GuestsCPU, point.GuestsMemory)
if err != nil {
s.setHistoryHealth(time.Time{}, err)
return err
}
_, err = s.db.Exec(`DELETE FROM history WHERE ts < ?`, time.Now().Add(-historyRetention).Unix())
s.setHistoryHealth(time.Now(), err)
return err
}
func (s *Store) setHistoryHealth(at time.Time, err error) {
s.healthMu.Lock()
defer s.healthMu.Unlock()
if !at.IsZero() {
s.lastHistoryAt = at
}
if err != nil {
s.lastHistoryError = err.Error()
} else {
s.lastHistoryError = ""
}
}
func (s *Store) historyHealth() (time.Time, string) {
s.healthMu.RLock()
defer s.healthMu.RUnlock()
return s.lastHistoryAt, s.lastHistoryError
}
func (s *Store) loadJSONSetting(key string, target any) bool {
var value string
if s.db.QueryRow(`SELECT value FROM settings WHERE key=?`, key).Scan(&value) != nil {
return false
}
return json.Unmarshal([]byte(value), target) == nil
}
func (s *Store) saveJSONSetting(key string, value any) error {
data, err := json.Marshal(value)
if err != nil {
return err
}
_, err = s.db.Exec(`INSERT INTO settings(key,value) VALUES(?,?) ON CONFLICT(key) DO UPDATE SET value=excluded.value`, key, string(data))
return err
}

View File

@@ -29,3 +29,17 @@ func TestStoreSettingsAndHistory(t *testing.T) {
t.Fatalf("история не сохранилась: points=%+v err=%v", points, err)
}
}
func TestEntityHistory(t *testing.T) {
store, err := openStore(":memory:")
if err != nil {
t.Fatal(err)
}
defer store.Close()
value := 42.5
store.AddEntityHistory("zfs", "rpool", time.Now().Unix(), &value, nil)
series, err := store.EntityHistory("zfs", 24)
if err != nil || len(series) != 1 || len(series[0].Points) != 1 || series[0].Name != "rpool" {
t.Fatalf("unexpected entity history: %+v %v", series, err)
}
}

116
system_health.go Normal file
View File

@@ -0,0 +1,116 @@
package main
import (
"os/exec"
"regexp"
"strconv"
"strings"
"sync"
"time"
)
type SystemIssue struct {
Time int64 `json:"time"`
Unit string `json:"unit"`
Priority string `json:"priority"`
Message string `json:"message"`
Count int `json:"count"`
}
type SystemHealthMetrics struct {
Available bool `json:"available"`
FailedUnits int `json:"failedUnits"`
KernelErrors int `json:"kernelErrors"`
SystemErrors int `json:"systemErrors"`
Issues []SystemIssue `json:"issues"`
FailedServices []FailedService `json:"failedServices"`
CheckedAt int64 `json:"checkedAt"`
}
type FailedService struct {
Unit string `json:"unit"`
Load string `json:"load"`
Active string `json:"active"`
State string `json:"state"`
Description string `json:"description"`
}
type systemHealthCollector struct {
mu sync.RWMutex
value SystemHealthMetrics
}
func newSystemHealthCollector() *systemHealthCollector {
c := &systemHealthCollector{}
go c.run()
return c
}
func (c *systemHealthCollector) collect() SystemHealthMetrics {
c.mu.RLock()
defer c.mu.RUnlock()
return c.value
}
func (c *systemHealthCollector) run() {
c.refresh()
t := time.NewTicker(time.Minute)
defer t.Stop()
for range t.C {
c.refresh()
}
}
func (c *systemHealthCollector) refresh() {
m := SystemHealthMetrics{CheckedAt: time.Now().Unix()}
if _, err := exec.LookPath("journalctl"); err != nil {
c.mu.Lock()
c.value = m
c.mu.Unlock()
return
}
m.Available = true
if out, err := exec.Command("systemctl", "--failed", "--no-legend", "--plain").Output(); err == nil {
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
fields := strings.Fields(line)
if len(fields) < 4 {
continue
}
unit := strings.TrimPrefix(fields[0], "●")
offset := 1
if unit == "" && len(fields) > 4 {
unit = fields[1]
offset = 2
}
description := ""
if len(fields) > offset+3 {
description = strings.Join(fields[offset+3:], " ")
}
m.FailedServices = append(m.FailedServices, FailedService{Unit: unit, Load: fields[offset], Active: fields[offset+1], State: fields[offset+2], Description: description})
m.FailedUnits++
}
}
out, _ := exec.Command("journalctl", "--since", "24 hours ago", "-p", "0..3", "--no-pager", "-o", "short-unix", "-n", "100").Output()
pidPattern := regexp.MustCompile(`\[[0-9]+\]`)
grouped := map[string]int{}
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
f := strings.Fields(line)
if len(f) < 2 {
continue
}
raw := strings.TrimSuffix(f[0], ":")
sec, _ := strconv.ParseFloat(raw, 64)
unit := strings.TrimSuffix(f[1], ":")
message := strings.Join(f[2:], " ")
normalized := pidPattern.ReplaceAllString(message, "[*]")
key := unit + "\x00" + normalized
if index, ok := grouped[key]; ok {
m.Issues[index].Count++
m.Issues[index].Time = int64(sec)
} else {
grouped[key] = len(m.Issues)
m.Issues = append(m.Issues, SystemIssue{Time: int64(sec), Unit: unit, Priority: "error", Message: normalized, Count: 1})
}
m.SystemErrors++
if strings.Contains(strings.ToLower(unit), "kernel") {
m.KernelErrors++
}
}
c.mu.Lock()
c.value = m
c.mu.Unlock()
}

107
trends.go Normal file
View File

@@ -0,0 +1,107 @@
package main
import (
"database/sql"
"fmt"
"sync"
"time"
)
type entityAverage struct {
Value1, Value2 float64
Count int
}
type TrendAnomaly struct {
ID string `json:"id"`
Severity string `json:"severity"`
Source string `json:"source"`
Title string `json:"title"`
Message string `json:"message"`
}
type trendCollector struct {
mu sync.Mutex
updatedAt time.Time
cached []TrendAnomaly
}
func (s *Store) entityAverages(kind string) map[string]entityAverage {
return s.entityAveragesSince(kind, 7*24*time.Hour)
}
func (s *Store) entityAveragesSince(kind string, duration time.Duration) map[string]entityAverage {
rows, err := s.db.Query(`SELECT name,AVG(value1),AVG(value2),COUNT(*) FROM entity_history WHERE kind=? AND ts>=? GROUP BY name`, kind, time.Now().Add(-duration).Unix())
if err != nil {
return nil
}
defer rows.Close()
result := map[string]entityAverage{}
for rows.Next() {
var name string
var v1, v2 sql.NullFloat64
var count int
if rows.Scan(&name, &v1, &v2, &count) == nil {
result[name] = entityAverage{Value1: v1.Float64, Value2: v2.Float64, Count: count}
}
}
return result
}
func (c *trendCollector) collect(metrics dashboardMetrics, store *Store) []TrendAnomaly {
c.mu.Lock()
defer c.mu.Unlock()
if time.Since(c.updatedAt) < time.Minute {
return c.cached
}
var out []TrendAnomaly
iops := store.entityAverages("disk-io")
latency := store.entityAverages("disk-latency")
latencyDay := store.entityAveragesSince("disk-latency", 24*time.Hour)
temps := store.entityAverages("disk")
for _, d := range metrics.Disks {
name := d.Name + " · " + d.Model
currentIOPS := d.ReadIOPS + d.WriteIOPS
if base := iops[name]; base.Count >= 30 && base.Value1 >= 1 && currentIOPS >= base.Value1*5 {
out = append(out, TrendAnomaly{"disk-iops-" + d.Name, "warning", "Аномалии", d.Model + ": IOPS резко выше обычного", fmt.Sprintf("Сейчас %.0f IOPS, недельный baseline %.0f — рост в %.1f раза.", currentIOPS, base.Value1, currentIOPS/base.Value1)})
}
if base := latency[name]; base.Count >= 30 && base.Value1 > 0 && d.LatencyMs >= base.Value1*2 && d.LatencyMs-base.Value1 >= 10 {
out = append(out, TrendAnomaly{"disk-latency-" + d.Name, "warning", "Аномалии", d.Model + ": latency ухудшилась", fmt.Sprintf("Сейчас %.1f ms при baseline %.1f ms.", d.LatencyMs, base.Value1)})
}
if week, day := latency[name], latencyDay[name]; week.Count >= 100 && day.Count >= 30 && week.Value1 > 0 && day.Value1 >= week.Value1*1.5 && day.Value1-week.Value1 >= 5 {
out = append(out, TrendAnomaly{"disk-latency-degrading-" + d.Name, "warning", "Аномалии", d.Model + ": latency постепенно ухудшается", fmt.Sprintf("Среднее за сутки %.1f ms против недельного %.1f ms.", day.Value1, week.Value1)})
}
if d.Temperature != nil {
if base := temps[name]; base.Count >= 30 && *d.Temperature >= base.Value1+10 {
out = append(out, TrendAnomaly{"disk-temp-trend-" + d.Name, "warning", "Аномалии", d.Model + ": температура выше обычной", fmt.Sprintf("Сейчас %.1f °C, baseline %.1f °C.", *d.Temperature, base.Value1)})
}
}
}
guestAvg := store.entityAverages("guest")
for _, g := range metrics.Guests.Guests {
if g.MaxMemory == 0 {
continue
}
name := fmt.Sprintf("%d · %s", g.VMID, g.Name)
current := float64(g.MemoryBytes) / float64(g.MaxMemory) * 100
if base := guestAvg[name]; base.Count >= 30 && current >= base.Value2*1.5 && current-base.Value2 >= 15 {
out = append(out, TrendAnomaly{fmt.Sprintf("guest-memory-%d", g.VMID), "warning", "Аномалии", g.Name + ": память выше обычной", fmt.Sprintf("Сейчас %.1f%%, baseline %.1f%%.", current, base.Value2)})
}
}
zfsAverage := store.entityAverages("zfs")
for _, pool := range metrics.ZFS.Pools {
if base := zfsAverage[pool.Name]; base.Count >= 30 && pool.CapacityPercent-base.Value1 >= 5 {
out = append(out, TrendAnomaly{"zfs-growth-" + pool.Name, "warning", "Аномалии", pool.Name + ": заполнение выше недельного уровня", fmt.Sprintf("Сейчас %.1f%%, недельный baseline %.1f%%.", pool.CapacityPercent, base.Value1)})
}
}
backupAverage := store.entityAverages("backup-duration")
for _, task := range metrics.Activity.Tasks {
if task.Type != "vzdump" || task.Duration <= 0 {
continue
}
name := "VMID " + task.ID
if base := backupAverage[name]; base.Count >= 3 && float64(task.Duration) >= base.Value1*1.5 && float64(task.Duration)-base.Value1 >= 60 {
out = append(out, TrendAnomaly{"backup-duration-" + task.ID, "warning", "Аномалии", "Backup " + name + " выполнялся дольше обычного", fmt.Sprintf("Последняя длительность %s, baseline %s.", time.Duration(task.Duration)*time.Second, time.Duration(base.Value1)*time.Second)})
}
}
c.cached = out
c.updatedAt = time.Now()
return out
}

77
update.go Normal file
View File

@@ -0,0 +1,77 @@
package main
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"time"
)
const defaultUpdateStatusPath = "/var/lib/proxmox-dashboard/update-status.json"
const defaultUpdateRequestPath = "/var/lib/proxmox-dashboard/update-request"
type UpdateStatus struct {
CurrentVersion string `json:"currentVersion"`
AvailableVersion string `json:"availableVersion"`
State string `json:"state"`
Mode string `json:"mode"`
Message string `json:"message"`
ReleaseURL string `json:"releaseUrl"`
CheckedAt time.Time `json:"checkedAt"`
}
func readUpdateStatus() UpdateStatus {
status := UpdateStatus{CurrentVersion: version, State: "unknown", Mode: "notify"}
path := os.Getenv("DASHBOARD_UPDATE_STATUS_PATH")
if path == "" {
path = defaultUpdateStatusPath
}
data, err := os.ReadFile(path)
if err != nil || json.Unmarshal(data, &status) != nil {
return status
}
status.CurrentVersion = version
return status
}
func requestUpdate() (UpdateStatus, error) {
requestPath := os.Getenv("DASHBOARD_UPDATE_REQUEST_PATH")
if requestPath == "" {
requestPath = defaultUpdateRequestPath
}
if err := os.MkdirAll(filepath.Dir(requestPath), 0o750); err != nil {
return UpdateStatus{}, err
}
status := readUpdateStatus()
status.State = "queued"
status.Message = "Обновление поставлено в очередь."
status.CheckedAt = time.Now()
statusPath := os.Getenv("DASHBOARD_UPDATE_STATUS_PATH")
if statusPath == "" {
statusPath = defaultUpdateStatusPath
}
data, _ := json.Marshal(status)
temporary, err := os.CreateTemp(filepath.Dir(statusPath), ".update-status-")
if err != nil {
return UpdateStatus{}, err
}
temporaryPath := temporary.Name()
defer os.Remove(temporaryPath)
if _, err = temporary.Write(data); err == nil {
err = temporary.Chmod(0o644)
}
if closeErr := temporary.Close(); err == nil {
err = closeErr
}
if err == nil {
err = os.Rename(temporaryPath, statusPath)
}
if err != nil {
return UpdateStatus{}, fmt.Errorf("не удалось сохранить запрос обновления: %w", err)
}
if err := os.WriteFile(requestPath, []byte(time.Now().Format(time.RFC3339Nano)), 0o640); err != nil {
return UpdateStatus{}, err
}
return status, nil
}

45
update_test.go Normal file
View File

@@ -0,0 +1,45 @@
package main
import (
"os"
"path/filepath"
"testing"
)
func TestReadUpdateStatus(t *testing.T) {
path := filepath.Join(t.TempDir(), "status.json")
if err := os.WriteFile(path, []byte(`{"currentVersion":"old","availableVersion":"v2.0.0","state":"available","mode":"notify","message":"Доступно обновление"}`), 0o600); err != nil {
t.Fatal(err)
}
t.Setenv("DASHBOARD_UPDATE_STATUS_PATH", path)
status := readUpdateStatus()
if status.CurrentVersion != version || status.AvailableVersion != "v2.0.0" || status.State != "available" {
t.Fatalf("unexpected update status: %+v", status)
}
}
func TestReadUpdateStatusWithoutFile(t *testing.T) {
t.Setenv("DASHBOARD_UPDATE_STATUS_PATH", filepath.Join(t.TempDir(), "missing.json"))
status := readUpdateStatus()
if status.CurrentVersion != version || status.State != "unknown" {
t.Fatalf("unexpected default update status: %+v", status)
}
}
func TestRequestUpdate(t *testing.T) {
directory := t.TempDir()
statusPath := filepath.Join(directory, "status.json")
requestPath := filepath.Join(directory, "request")
t.Setenv("DASHBOARD_UPDATE_STATUS_PATH", statusPath)
t.Setenv("DASHBOARD_UPDATE_REQUEST_PATH", requestPath)
status, err := requestUpdate()
if err != nil {
t.Fatal(err)
}
if status.State != "queued" {
t.Fatalf("unexpected queued status: %+v", status)
}
if _, err := os.Stat(requestPath); err != nil {
t.Fatalf("update request was not created: %v", err)
}
}

57
ups.go
View File

@@ -10,20 +10,26 @@ import (
)
type UPSMetrics struct {
Available bool `json:"available"`
Source string `json:"source"`
Name string `json:"name"`
Model string `json:"model"`
Status string `json:"status"`
StatusLabel string `json:"statusLabel"`
ChargePercent *float64 `json:"chargePercent"`
RuntimeSeconds *float64 `json:"runtimeSeconds"`
LoadPercent *float64 `json:"loadPercent"`
InputVoltage *float64 `json:"inputVoltage"`
OutputVoltage *float64 `json:"outputVoltage"`
InputFrequency *float64 `json:"inputFrequency"`
BatteryVoltage *float64 `json:"batteryVoltage"`
Temperature *float64 `json:"temperatureCelsius"`
Available bool `json:"available"`
Source string `json:"source"`
Name string `json:"name"`
Model string `json:"model"`
Status string `json:"status"`
StatusLabel string `json:"statusLabel"`
ChargePercent *float64 `json:"chargePercent"`
RuntimeSeconds *float64 `json:"runtimeSeconds"`
LoadPercent *float64 `json:"loadPercent"`
InputVoltage *float64 `json:"inputVoltage"`
OutputVoltage *float64 `json:"outputVoltage"`
InputFrequency *float64 `json:"inputFrequency"`
BatteryVoltage *float64 `json:"batteryVoltage"`
Temperature *float64 `json:"temperatureCelsius"`
CommunicationOK bool `json:"communicationOk"`
BatteryDate string `json:"batteryDate"`
BatteryAgeDays int `json:"batteryAgeDays,omitempty"`
ReplaceBattery bool `json:"replaceBattery"`
LowBatteryCharge *float64 `json:"lowBatteryCharge,omitempty"`
LowBatteryRuntime *float64 `json:"lowBatteryRuntime,omitempty"`
}
type upsCollector struct {
@@ -43,7 +49,13 @@ func (c *upsCollector) collect() UPSMetrics {
} else if metrics, ok := readAPCUPSD(); ok {
c.cached = metrics
} else {
c.cached = UPSMetrics{}
if c.cached.Available || c.cached.Source != "" {
c.cached.Available = false
c.cached.CommunicationOK = false
c.cached.StatusLabel = "Нет связи"
} else {
c.cached = UPSMetrics{}
}
}
c.updatedAt = time.Now()
return c.cached
@@ -85,6 +97,7 @@ func readAPCUPSD() (UPSMetrics, bool) {
"ups.load": raw["LOADPCT"], "input.voltage": raw["LINEV"],
"output.voltage": raw["OUTPUTV"], "input.frequency": raw["LINEFREQ"],
"battery.voltage": raw["BATTV"], "ups.temperature": raw["ITEMP"],
"battery.date": raw["BATTDATE"], "battery.charge.low": raw["MBATTCHG"], "battery.runtime.low": minutesToSeconds(raw["MINTIMEL"]),
}
return upsFromValues("apcupsd", raw["UPSNAME"], values), true
}
@@ -102,14 +115,24 @@ func parseColonValues(output string) map[string]string {
func upsFromValues(source, name string, values map[string]string) UPSMetrics {
status := values["ups.status"]
return UPSMetrics{
Available: true, Source: source, Name: name, Model: values["device.model"],
metrics := UPSMetrics{
Available: true, CommunicationOK: true, Source: source, Name: name, Model: values["device.model"],
Status: status, StatusLabel: upsStatusLabel(status),
ChargePercent: numberPointer(values["battery.charge"]), RuntimeSeconds: numberPointer(values["battery.runtime"]),
LoadPercent: numberPointer(values["ups.load"]), InputVoltage: numberPointer(values["input.voltage"]),
OutputVoltage: numberPointer(values["output.voltage"]), InputFrequency: numberPointer(values["input.frequency"]),
BatteryVoltage: numberPointer(values["battery.voltage"]), Temperature: numberPointer(firstNonEmpty(values["ups.temperature"], values["battery.temperature"])),
BatteryDate: firstNonEmpty(values["battery.date"], values["battery.mfr.date"]), ReplaceBattery: strings.Contains(status, "RB"), LowBatteryCharge: numberPointer(values["battery.charge.low"]), LowBatteryRuntime: numberPointer(values["battery.runtime.low"]),
}
if metrics.BatteryDate != "" {
for _, layout := range []string{"2006/01/02", "2006-01-02", "01/02/2006", "02/01/2006"} {
if date, err := time.Parse(layout, metrics.BatteryDate); err == nil {
metrics.BatteryAgeDays = int(time.Since(date).Hours() / 24)
break
}
}
}
return metrics
}
func numberPointer(value string) *float64 {

608
web.go
View File

@@ -6,24 +6,36 @@ import (
"io"
"net/http"
"strconv"
"strings"
"time"
)
type dashboardMetrics struct {
CPU CPUMetrics `json:"cpu"`
Memory MemoryMetrics `json:"memory"`
Storage StorageMetrics `json:"storage"`
Disks []DiskMetrics `json:"disks"`
Network NetworkMetrics `json:"network"`
ZFS ZFSMetrics `json:"zfs"`
UPS UPSMetrics `json:"ups"`
Guests GuestsMetrics `json:"guests"`
Backups BackupMetrics `json:"backups"`
Alerts []Alert `json:"alerts"`
Timestamp time.Time `json:"timestamp"`
CPU CPUMetrics `json:"cpu"`
Memory MemoryMetrics `json:"memory"`
Storage StorageMetrics `json:"storage"`
Disks []DiskMetrics `json:"disks"`
MissingDisks []string `json:"missingDisks"`
Network NetworkMetrics `json:"network"`
ZFS ZFSMetrics `json:"zfs"`
UPS UPSMetrics `json:"ups"`
Guests GuestsMetrics `json:"guests"`
Backups BackupMetrics `json:"backups"`
Alerts []Alert `json:"alerts"`
Update UpdateStatus `json:"update"`
Maintenance MaintenanceMetrics `json:"maintenance"`
MaintenanceWindow MaintenanceWindow `json:"maintenanceWindow"`
Services ServicesMetrics `json:"services"`
SystemHealth SystemHealthMetrics `json:"systemHealth"`
Activity ProxmoxActivity `json:"activity"`
Monitor MonitorHealth `json:"monitor"`
UPSShutdown UPSShutdownSettings `json:"upsShutdown"`
Trends []TrendAnomaly `json:"trends"`
Agents []ManagedAgent `json:"agents"`
Timestamp time.Time `json:"timestamp"`
}
func collectDashboard(cpuCollector *cpuCollector, networkCollector *networkCollector, diskCollector *diskCollector, zfsCollector *zfsCollector, upsCollector *upsCollector, guestCollector *guestCollector, backupCollector *backupCollector, store *Store) (dashboardMetrics, error) {
func collectDashboard(cpuCollector *cpuCollector, networkCollector *networkCollector, diskCollector *diskCollector, zfsCollector *zfsCollector, upsCollector *upsCollector, guestCollector *guestCollector, backupCollector *backupCollector, maintenanceCollector *maintenanceCollector, serviceCollector *serviceCollector, systemHealth *systemHealthCollector, activity *activityCollector, ioAttribution *ioAttributionCollector, trends *trendCollector, store *Store, monitor *monitorTracker) (dashboardMetrics, error) {
cpu, err := cpuCollector.collect()
if err != nil {
return dashboardMetrics{}, err
@@ -40,16 +52,72 @@ func collectDashboard(cpuCollector *cpuCollector, networkCollector *networkColle
if err != nil {
return dashboardMetrics{}, err
}
thresholds := store.Thresholds()
metrics := dashboardMetrics{
CPU: cpu, Memory: memory, Storage: storage,
Disks: diskCollector.collect(), Network: network,
Disks: diskCollector.collect(thresholds), Network: network,
ZFS: zfsCollector.collect(), UPS: upsCollector.collect(),
Guests: guestCollector.collect(), Backups: backupCollector.collect(), Timestamp: time.Now(),
Guests: guestCollector.collect(), Backups: backupCollector.collect(), Maintenance: maintenanceCollector.collect(), Services: serviceCollector.collect(), SystemHealth: systemHealth.collect(), Update: readUpdateStatus(), Timestamp: time.Now(),
}
metrics.MissingDisks = diskCollector.missingDisks()
metrics.Monitor = monitor.snapshot(store)
metrics.UPSShutdown = store.UPSShutdownSettings()
metrics.Activity = activity.collect(metrics.Guests)
attributeDiskIO(&metrics, ioAttribution.collect(metrics.Guests))
metrics.Trends = trends.collect(metrics, store)
metrics.Agents, _ = store.Agents()
decorateNetworkInterfaces(&metrics.Network, metrics.Guests)
metrics.MaintenanceWindow = store.MaintenanceWindow()
allAlerts := correlateAlerts(metrics, evaluateAlerts(metrics, thresholds))
activeAlerts, plannedAlerts := splitMaintenanceAlerts(allAlerts, metrics.MaintenanceWindow)
metrics.Alerts, err = store.SyncAlertsWithMaintenance(activeAlerts, plannedAlerts, metrics.MaintenanceWindow.Reason)
if err != nil {
return dashboardMetrics{}, err
}
metrics.Alerts = evaluateAlerts(metrics, store.Thresholds())
return metrics, nil
}
func decorateNetworkInterfaces(network *NetworkMetrics, guests GuestsMetrics) {
names := map[int]GuestMetrics{}
for _, g := range guests.Guests {
names[g.VMID] = g
}
for i := range network.Interfaces {
v := &network.Interfaces[i]
vmid := interfaceVMID(v.Name)
if vmid == 0 {
continue
}
v.VMID = vmid
if g, ok := names[vmid]; ok {
v.GuestName = g.Name
if g.Type == "qemu" {
v.GuestType = "VM"
} else {
v.GuestType = "LXC"
}
}
}
}
func interfaceVMID(name string) int {
prefixes := []string{"tap", "veth", "fwbr", "fwpr", "fwln"}
for _, prefix := range prefixes {
if !strings.HasPrefix(name, prefix) {
continue
}
rest := strings.TrimPrefix(name, prefix)
end := 0
for end < len(rest) && rest[end] >= '0' && rest[end] <= '9' {
end++
}
if end > 0 {
id, _ := strconv.Atoi(rest[:end])
return id
}
}
return 0
}
func routes(collector *cpuCollector, store *Store) http.Handler {
mux := http.NewServeMux()
network, networkErr := newNetworkCollector("/proc/net/dev")
@@ -58,10 +126,24 @@ func routes(collector *cpuCollector, store *Store) http.Handler {
ups := &upsCollector{}
guests := &guestCollector{}
backups := &backupCollector{}
maintenance := &maintenanceCollector{}
services := newServiceCollector(store)
systemHealth := newSystemHealthCollector()
activity := newActivityCollector()
ioAttribution := &ioAttributionCollector{}
trends := &trendCollector{}
monitor := &monitorTracker{}
go runCollectionWatchdog(monitor)
go runUPSShutdownController(store, ups, guests)
collect := func() (dashboardMetrics, error) {
return collectDashboard(collector, network, disks, zfs, ups, guests, backups, store)
started := time.Now()
metrics, err := collectDashboard(collector, network, disks, zfs, ups, guests, backups, maintenance, services, systemHealth, activity, ioAttribution, trends, store, monitor)
monitor.record(started, err)
return metrics, err
}
go runHistoryCollector(collect, store)
go runEmailNotifier(collect, store)
go runImmediateNotifier(collect, store)
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/" {
http.NotFound(w, r)
@@ -75,6 +157,15 @@ func routes(collector *cpuCollector, store *Store) http.Handler {
writeJSON(w, metrics, err)
})
mux.HandleFunc("/api/version", func(w http.ResponseWriter, _ *http.Request) { writeJSON(w, map[string]string{"version": version}, nil) })
mux.HandleFunc("/api/update/status", func(w http.ResponseWriter, _ *http.Request) { writeJSON(w, readUpdateStatus(), nil) })
mux.HandleFunc("/api/update/request", func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
w.WriteHeader(http.StatusMethodNotAllowed)
return
}
status, err := requestUpdate()
writeJSON(w, status, err)
})
mux.HandleFunc("/api/system", func(w http.ResponseWriter, _ *http.Request) {
if networkErr != nil {
writeJSON(w, nil, networkErr)
@@ -102,11 +193,221 @@ func routes(collector *cpuCollector, store *Store) http.Handler {
w.WriteHeader(http.StatusMethodNotAllowed)
}
})
mux.HandleFunc("/api/settings/maintenance", func(w http.ResponseWriter, r *http.Request) {
switch r.Method {
case http.MethodGet:
writeJSON(w, store.MaintenanceWindow(), nil)
case http.MethodPut:
var window MaintenanceWindow
if err := json.NewDecoder(io.LimitReader(r.Body, 64<<10)).Decode(&window); err != nil {
http.Error(w, "Некорректные настройки", http.StatusBadRequest)
return
}
value, err := store.SaveMaintenanceWindow(window)
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
writeJSON(w, value, nil)
case http.MethodDelete:
value, err := store.StopMaintenanceWindow()
writeJSON(w, value, err)
default:
w.WriteHeader(http.StatusMethodNotAllowed)
}
})
mux.HandleFunc("/api/agents", func(w http.ResponseWriter, r *http.Request) {
switch r.Method {
case http.MethodGet:
agents, err := store.Agents()
writeJSON(w, agents, err)
case http.MethodDelete:
id := strings.TrimSpace(r.URL.Query().Get("id"))
if id == "" {
http.Error(w, "Не указан агент", http.StatusBadRequest)
return
}
writeJSON(w, map[string]bool{"deleted": true}, store.DeleteAgent(id))
default:
w.WriteHeader(http.StatusMethodNotAllowed)
}
})
mux.HandleFunc("/api/agents/enrollment-token", func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
w.WriteHeader(http.StatusMethodNotAllowed)
return
}
token, err := store.CreateAgentEnrollmentToken()
writeJSON(w, map[string]any{"token": token, "expiresInSeconds": 900}, err)
})
mux.HandleFunc("/api/agent/enroll", func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
w.WriteHeader(http.StatusMethodNotAllowed)
return
}
var request agentEnrollmentRequest
if json.NewDecoder(io.LimitReader(r.Body, 16<<10)).Decode(&request) != nil || strings.TrimSpace(request.Token) == "" {
http.Error(w, "Некорректная регистрация", http.StatusBadRequest)
return
}
credentials, err := store.EnrollAgent(request)
if err != nil {
http.Error(w, err.Error(), http.StatusUnauthorized)
return
}
writeJSON(w, credentials, nil)
})
mux.HandleFunc("/api/agent/report", func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
w.WriteHeader(http.StatusMethodNotAllowed)
return
}
credentials := strings.TrimPrefix(r.Header.Get("Authorization"), "Bearer ")
id, secret, ok := strings.Cut(credentials, ".")
if !ok || id == "" || secret == "" {
http.Error(w, "Агент не авторизован", http.StatusUnauthorized)
return
}
var report AgentReport
if json.NewDecoder(io.LimitReader(r.Body, 2<<20)).Decode(&report) != nil {
http.Error(w, "Некорректный отчёт", http.StatusBadRequest)
return
}
if err := store.SaveAgentReport(id, secret, r.RemoteAddr, report); err != nil {
http.Error(w, err.Error(), http.StatusUnauthorized)
return
}
writeJSON(w, map[string]bool{"accepted": true}, nil)
})
mux.HandleFunc("/api/alerts/history", func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet {
w.WriteHeader(http.StatusMethodNotAllowed)
return
}
events, err := store.AlertHistory(200)
writeJSON(w, events, err)
})
mux.HandleFunc("/api/alerts/acknowledge", func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
w.WriteHeader(http.StatusMethodNotAllowed)
return
}
var request struct {
EventID int64 `json:"eventId"`
}
if err := json.NewDecoder(io.LimitReader(r.Body, 1024)).Decode(&request); err != nil || request.EventID < 1 {
http.Error(w, "Некорректное событие", http.StatusBadRequest)
return
}
writeJSON(w, map[string]bool{"acknowledged": true}, store.AcknowledgeAlert(request.EventID))
})
mux.HandleFunc("/api/settings/email", func(w http.ResponseWriter, r *http.Request) {
switch r.Method {
case http.MethodGet:
writeJSON(w, store.EmailSettings(false), nil)
case http.MethodPut:
var settings EmailSettings
if err := json.NewDecoder(io.LimitReader(r.Body, 64<<10)).Decode(&settings); err != nil {
http.Error(w, "Некорректные настройки", http.StatusBadRequest)
return
}
if err := store.SaveEmailSettings(settings); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
writeJSON(w, store.EmailSettings(false), nil)
default:
w.WriteHeader(http.StatusMethodNotAllowed)
}
})
mux.HandleFunc("/api/settings/email/test", func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
w.WriteHeader(http.StatusMethodNotAllowed)
return
}
settings := store.EmailSettings(true)
writeJSON(w, map[string]bool{"sent": true}, sendAlertEmail(settings, nil, true))
})
mux.HandleFunc("/api/settings/ups-shutdown", func(w http.ResponseWriter, r *http.Request) {
switch r.Method {
case http.MethodGet:
writeJSON(w, store.UPSShutdownSettings(), nil)
case http.MethodPut:
var settings UPSShutdownSettings
if json.NewDecoder(io.LimitReader(r.Body, 16<<10)).Decode(&settings) != nil {
http.Error(w, "Некорректные настройки", http.StatusBadRequest)
return
}
if err := store.SaveUPSShutdownSettings(settings); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
writeJSON(w, settings, nil)
default:
w.WriteHeader(http.StatusMethodNotAllowed)
}
})
mux.HandleFunc("/api/ups/shutdown-dry-run", func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
w.WriteHeader(http.StatusMethodNotAllowed)
return
}
writeJSON(w, buildShutdownDryRun(store.UPSShutdownSettings(), guests.collect()), nil)
})
mux.HandleFunc("/api/history", func(w http.ResponseWriter, r *http.Request) {
hours, _ := strconv.Atoi(r.URL.Query().Get("hours"))
points, err := store.History(hours)
writeJSON(w, points, err)
})
mux.HandleFunc("/api/entity-history", func(w http.ResponseWriter, r *http.Request) {
hours, _ := strconv.Atoi(r.URL.Query().Get("hours"))
series, err := store.EntityHistory(r.URL.Query().Get("kind"), hours)
writeJSON(w, series, err)
})
mux.HandleFunc("/api/services", func(w http.ResponseWriter, r *http.Request) {
switch r.Method {
case http.MethodGet:
items, err := store.Services()
writeJSON(w, items, err)
case http.MethodPost, http.MethodPut:
var v MonitoredService
if err := json.NewDecoder(io.LimitReader(r.Body, 32<<10)).Decode(&v); err != nil {
http.Error(w, "Некорректные данные", 400)
return
}
saved, err := store.SaveService(v)
writeJSON(w, saved, err)
case http.MethodDelete:
id, _ := strconv.ParseInt(r.URL.Query().Get("id"), 10, 64)
if id < 1 {
http.Error(w, "Некорректный сервис", 400)
return
}
writeJSON(w, map[string]bool{"deleted": true}, store.DeleteService(id))
default:
w.WriteHeader(http.StatusMethodNotAllowed)
}
})
mux.HandleFunc("/api/services/history", func(w http.ResponseWriter, r *http.Request) {
id, _ := strconv.ParseInt(r.URL.Query().Get("id"), 10, 64)
hours, _ := strconv.Atoi(r.URL.Query().Get("hours"))
points, err := store.ServiceHistory(id, hours)
writeJSON(w, points, err)
})
mux.HandleFunc("/api/backups/verify", func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
w.WriteHeader(http.StatusMethodNotAllowed)
return
}
var request struct {
VolumeID string `json:"volid"`
}
if json.NewDecoder(io.LimitReader(r.Body, 8<<10)).Decode(&request) != nil {
http.Error(w, "Некорректный backup", http.StatusBadRequest)
return
}
writeJSON(w, verifyBackup(request.VolumeID), nil)
})
mux.HandleFunc("/api/events", func(w http.ResponseWriter, r *http.Request) {
flusher, ok := w.(http.Flusher)
if !ok {
@@ -163,7 +464,7 @@ button{font:inherit}main{width:min(1060px,calc(100% - 32px));margin:auto;padding
header{display:flex;align-items:end;justify-content:space-between;margin-bottom:22px}.eyebrow{color:var(--green);font:700 11px/1 monospace;letter-spacing:.16em;text-transform:uppercase}
h1{font-size:clamp(28px,5vw,44px);letter-spacing:-.045em;margin:9px 0 0}.header-actions,.status{display:flex;align-items:center;gap:10px}.status{color:var(--muted);font-size:13px}.dot{width:8px;height:8px;border-radius:50%;background:#f39a56}.dot.live{background:var(--green);box-shadow:0 0 15px var(--green)}.settings-button{width:38px;height:38px;border:1px solid var(--line);border-radius:12px;background:var(--panel);color:var(--text);cursor:pointer;font-size:18px}.settings-button:hover{border-color:var(--muted)}
.cards{display:grid;grid-template-columns:1fr 1fr;grid-auto-rows:1fr;gap:16px}.card{appearance:none;display:flex;flex-direction:column;text-align:left;color:inherit;width:100%;padding:0;background:linear-gradient(145deg,rgba(18,28,36,.98),rgba(12,19,25,.98));border:1px solid var(--line);border-radius:20px;overflow:hidden;cursor:pointer;transition:transform .18s,border-color .18s,box-shadow .18s}.card:hover{transform:translateY(-2px);border-color:#3d4e5c;box-shadow:0 20px 55px rgba(0,0,0,.22)}.card:focus-visible{outline:2px solid var(--green);outline-offset:3px}
.card-head{display:flex;justify-content:space-between;align-items:flex-start;gap:20px;min-height:106px;padding:23px 24px 18px}.card-head>div:first-child{min-width:0}.label{color:var(--muted);font-size:11px;text-transform:uppercase;letter-spacing:.1em}.name{font-size:17px;font-weight:650;margin-top:6px;white-space:nowrap;overflow:hidden;text-overflow:ellipsis;max-width:350px}.percent{flex:0 0 auto;font:700 35px/.95 ui-monospace,monospace;letter-spacing:-.07em}.percent small{font-size:16px;color:var(--green)}
.card-head{display:flex;justify-content:space-between;align-items:flex-start;gap:20px;min-height:106px;padding:23px 24px 18px}.card-head>div:first-child{min-width:0}.label{color:var(--muted);font-size:11px;text-transform:uppercase;letter-spacing:.1em}.name{font-size:17px;font-weight:650;margin-top:6px;white-space:nowrap;overflow:hidden;text-overflow:ellipsis;max-width:350px}.percent{display:inline-flex;align-items:baseline;gap:.22em;flex:0 0 auto;font:700 35px/.95 ui-monospace,monospace;letter-spacing:-.04em}.percent small{font-size:16px;color:var(--green);letter-spacing:-.02em}
.bar{height:5px;background:#222e37;margin:0 24px;overflow:hidden;border-radius:20px}.fill{height:100%;width:0;background:linear-gradient(90deg,var(--green),#afe36d);transition:width .5s ease}.memory .fill{background:linear-gradient(90deg,var(--violet),#72b9ff)}
.storage .fill{background:linear-gradient(90deg,#ffb35c,#ff755c)}.network .fill{background:linear-gradient(90deg,#43c6d9,#65ddaa)}.zfs .fill{background:linear-gradient(90deg,#46c0ff,#6577ff)}.ups .fill{background:linear-gradient(90deg,#d5e85a,#55d58a)}.guests .fill{background:linear-gradient(90deg,#ff78b8,#9d8cff)}.backups .fill{background:linear-gradient(90deg,#ffcf5c,#55d58a)}.alert-center{grid-column:1/-1}.alert-center .fill{background:linear-gradient(90deg,#ffb35c,#ff665c)}
.quick{display:grid;grid-template-columns:1fr 1fr;margin-top:19px;border-top:1px solid var(--line)}.quick div{padding:16px 24px}.quick div+div{border-left:1px solid var(--line)}.quick strong{display:block;font:650 20px/1.2 ui-monospace,monospace;margin-top:5px}.hint{color:#657581;font-size:11px;margin:auto 24px 18px;padding-top:14px}
@@ -171,15 +472,24 @@ dialog{width:min(760px,calc(100% - 28px));max-height:calc(100vh - 32px);padding:
.modal-head{position:sticky;top:0;z-index:2;display:flex;align-items:flex-start;justify-content:space-between;padding:26px 28px;background:rgba(16,24,32,.94);backdrop-filter:blur(14px);border-bottom:1px solid var(--line)}.modal-head h2{font-size:24px;letter-spacing:-.03em;margin:5px 0 0}.close{width:38px;height:38px;border-radius:50%;border:1px solid var(--line);background:#17222b;color:var(--text);font-size:22px;cursor:pointer}.detail-grid{display:grid;grid-template-columns:repeat(3,1fr)}.detail{padding:22px 27px;border-right:1px solid var(--line);border-bottom:1px solid var(--line)}.detail:nth-child(3n){border-right:0}.detail strong{display:block;font:650 24px/1.2 ui-monospace,monospace;margin-top:7px}.wide{padding:24px 28px}.temps{display:flex;flex-wrap:wrap;gap:9px;margin-top:12px}.temp{padding:7px 11px;border:1px solid var(--line);border-radius:99px}.temp b{color:var(--green);margin-left:7px}.empty{color:var(--muted);font-size:13px}
.disk-list{display:grid;gap:12px;padding:16px}.disk-item{padding:19px;border:1px solid var(--line);border-radius:15px;background:#0c141a}.disk-top{display:flex;justify-content:space-between;gap:20px}.disk-model{font-size:17px;font-weight:650}.disk-meta{color:var(--muted);font:12px/1.5 ui-monospace,monospace;margin-top:4px;overflow-wrap:anywhere}.badge{position:relative;height:max-content;padding:5px 9px;border:1px solid #2d6245;border-radius:99px;color:var(--green);font-size:11px;white-space:nowrap}.badge.warn{color:#ffb35c;border-color:#6d4c2c}.badge.bad{color:#ff755c;border-color:#71362f}.badge[data-reason]:hover::after,.badge[data-reason]:focus::after{content:attr(data-reason);position:absolute;z-index:5;right:0;bottom:calc(100% + 10px);width:280px;padding:11px 13px;border:1px solid #4c5963;border-radius:10px;background:#1a242c;color:var(--text);font:12px/1.45 Inter,system-ui,sans-serif;white-space:normal;box-shadow:0 14px 35px #0008}.disk-stats{display:grid;grid-template-columns:repeat(6,1fr);gap:14px;margin-top:18px}.disk-stat span{display:block;color:var(--muted);font-size:10px;text-transform:uppercase;letter-spacing:.08em}.disk-stat b{display:block;margin-top:5px;font:600 14px ui-monospace,monospace}
.settings-tabs{display:flex;gap:6px;padding:12px 26px 0}.settings-tab{padding:9px 13px;border:1px solid transparent;border-radius:10px;background:transparent;color:var(--muted);cursor:pointer}.settings-tab.active{background:var(--panel);border-color:var(--line);color:var(--text)}.settings-body{padding:22px 26px 28px}.settings-panel[hidden]{display:none}.settings-group+.settings-group{margin-top:25px}.theme-options{display:grid;grid-template-columns:1fr 1fr;gap:10px;margin-top:11px}.theme-option{padding:13px;border:1px solid var(--line);border-radius:12px;background:var(--panel);color:var(--text);cursor:pointer}.theme-option.active{border-color:var(--green);box-shadow:inset 0 0 0 1px var(--green)}.setting-list{display:grid;grid-template-columns:1fr 1fr;gap:8px;margin-top:11px}.setting-item{display:flex;align-items:center;gap:10px;padding:11px 12px;border:1px solid var(--line);border-radius:11px}.setting-item input{accent-color:var(--green);width:17px;height:17px}.threshold-grid{display:grid;grid-template-columns:1fr 1fr;gap:10px}.threshold-item{display:flex;align-items:center;justify-content:space-between;gap:15px;padding:11px 12px;border:1px solid var(--line);border-radius:11px}.threshold-item span{font-size:13px}.threshold-item input{width:78px;padding:7px 8px;border:1px solid var(--line);border-radius:8px;background:var(--bg);color:var(--text);font:600 14px ui-monospace,monospace}.save-settings{width:100%;margin-top:16px;padding:12px;border:0;border-radius:11px;background:var(--green);color:#07120c;font-weight:700;cursor:pointer}.settings-note{color:var(--muted);font-size:12px;margin-top:10px}.card[hidden]{display:none}
.chart-section{padding:22px 25px;border-top:1px solid var(--line)}.chart-head{display:flex;justify-content:space-between;align-items:center;margin-bottom:12px}.chart-head span{color:var(--muted);font-size:12px}.metric-chart{display:block;width:100%;height:180px;border-radius:12px;background:rgba(100,130,150,.05)}
.detail-tabs{display:flex;gap:6px;padding:12px 26px;border-bottom:1px solid var(--line)}.detail-tab{padding:9px 13px;border:1px solid transparent;border-radius:10px;background:transparent;color:var(--muted);cursor:pointer}.detail-tab.active{background:var(--panel);border-color:var(--line);color:var(--text)}.detail-panel[hidden]{display:none}.chart-section{position:relative;padding:22px 25px}.chart-head{display:flex;justify-content:space-between;align-items:center;gap:16px;margin-bottom:12px}.range-switch{display:flex;gap:4px;padding:3px;border:1px solid var(--line);border-radius:10px}.range-button{padding:5px 9px;border:0;border-radius:7px;background:transparent;color:var(--muted);font-size:11px;cursor:pointer}.range-button.active{background:var(--panel);color:var(--text);box-shadow:0 1px 5px #0003}.metric-chart{display:block;width:100%;height:200px;border-radius:12px;background:rgba(100,130,150,.05);cursor:crosshair}.chart-tooltip{display:none;position:absolute;z-index:4;min-width:165px;padding:10px 12px;border:1px solid #40505c;border-radius:10px;background:#17222b;color:var(--text);font-size:12px;line-height:1.55;pointer-events:none;box-shadow:0 12px 32px #0008}.chart-tooltip.visible{display:block}.chart-tooltip-time{color:var(--muted);margin-bottom:4px}.chart-tooltip-value{display:flex;align-items:center;justify-content:space-between;gap:18px}.chart-tooltip-value i{width:7px;height:7px;border-radius:50%;margin-right:7px;display:inline-block}
.alert-list{display:grid;gap:10px;padding:16px}.alert-item{display:grid;grid-template-columns:auto 1fr;gap:13px;padding:17px 18px;border:1px solid var(--line);border-radius:14px;background:#0c141a}.alert-icon{width:10px;height:10px;border-radius:50%;margin-top:6px;background:#6ba8ff;box-shadow:0 0 14px currentColor}.alert-item.warning .alert-icon{background:#ffb35c}.alert-item.critical .alert-icon{background:#ff665c}.alert-title{font-weight:650}.alert-source{color:var(--muted);font-size:11px;text-transform:uppercase;letter-spacing:.08em;margin-bottom:4px}.alert-message{color:var(--muted);font-size:13px;margin-top:4px}.all-good{padding:45px 24px;text-align:center}.all-good strong{display:block;font-size:22px;margin-bottom:7px}.all-good span{color:var(--muted)}
html[data-theme="light"] body{background:radial-gradient(circle at 78% -8%,#cde9da 0,transparent 34%),var(--bg)}html[data-theme="light"] .card{background:linear-gradient(145deg,#fff,#f7f9fa)}html[data-theme="light"] dialog{background:#fff;box-shadow:0 30px 80px #4e64752b}html[data-theme="light"] .modal-head{background:rgba(255,255,255,.94)}html[data-theme="light"] .close{background:#f1f4f6}html[data-theme="light"] .disk-item,html[data-theme="light"] .alert-item{background:#f7f9fa}html[data-theme="light"] .bar{background:#e2e8ec}html[data-theme="light"] dialog::backdrop{background:rgba(30,45,55,.35)}html[data-theme="light"] .badge[data-reason]:hover::after,html[data-theme="light"] .badge[data-reason]:focus::after{background:#fff}
html[data-theme="light"] body{background:radial-gradient(circle at 78% -8%,#cde9da 0,transparent 34%),var(--bg)}html[data-theme="light"] .card{background:linear-gradient(145deg,#fff,#f7f9fa)}html[data-theme="light"] dialog{background:#fff;box-shadow:0 30px 80px #4e64752b}html[data-theme="light"] .modal-head{background:rgba(255,255,255,.94)}html[data-theme="light"] .close{background:#f1f4f6}html[data-theme="light"] .disk-item,html[data-theme="light"] .alert-item{background:#f7f9fa}html[data-theme="light"] .bar{background:#e2e8ec}html[data-theme="light"] dialog::backdrop{background:rgba(30,45,55,.35)}html[data-theme="light"] .badge[data-reason]:hover::after,html[data-theme="light"] .badge[data-reason]:focus::after,html[data-theme="light"] .chart-tooltip{background:#fff;box-shadow:0 12px 32px #4e647533}
footer{color:#657581;font-size:12px;margin-top:17px}
@media(max-width:700px){main{padding:28px 0}.cards{grid-template-columns:1fr}header{align-items:flex-start;flex-direction:column;gap:14px}.detail-grid{grid-template-columns:repeat(2,1fr)}.detail:nth-child(3n){border-right:1px solid var(--line)}.detail:nth-child(2n){border-right:0}.name{max-width:220px}.disk-stats,.setting-list,.threshold-grid{grid-template-columns:repeat(2,1fr)}}
.maintenance-banner{display:flex;align-items:center;justify-content:space-between;gap:16px;margin:-7px 0 18px;padding:12px 16px;border:1px solid #6d4c2c;border-radius:13px;background:rgba(255,179,92,.08);color:#ffb35c}.maintenance-banner[hidden]{display:none}.maintenance-banner strong{color:var(--text)}.maintenance-banner small{display:block;color:var(--muted);margin-top:2px}.maintenance-targets{display:grid;grid-template-columns:1fr 1fr;gap:8px;margin-top:10px;max-height:240px;overflow:auto}.maintenance-targets .setting-item{background:var(--panel)}
@media(max-width:700px){main{padding:28px 0}.cards{grid-template-columns:1fr}header{align-items:flex-start;flex-direction:column;gap:14px}.detail-grid{grid-template-columns:repeat(2,1fr)}.detail:nth-child(3n){border-right:1px solid var(--line)}.detail:nth-child(2n){border-right:0}.name{max-width:220px}.disk-stats,.setting-list,.threshold-grid{grid-template-columns:repeat(2,1fr)}.service-form,.maintenance-grid{grid-template-columns:1fr}.maintenance-grid .detail{border-right:0}.chart-head{align-items:flex-start;flex-direction:column}.detail-tabs,.settings-tabs,.system-tabs{overflow-x:auto}}
.save-settings:disabled{cursor:wait;opacity:.55}[data-settings-panel="email"] .threshold-item input{width:min(210px,52vw)}.ack-alert{margin-top:11px;padding:6px 10px;border:1px solid var(--line);border-radius:8px;background:var(--panel);color:var(--text);font-size:11px;cursor:pointer}.history-state{display:inline-block;margin-top:8px;color:var(--muted);font-size:11px}.alert-panel[hidden]{display:none}
.maintenance .fill{background:linear-gradient(90deg,#55d58a,#43c6d9)}.card-order{display:grid;gap:7px;margin-top:11px}.order-item{display:flex;align-items:center;gap:10px;padding:9px 10px;border:1px solid var(--line);border-radius:10px;background:var(--panel)}.order-item.dragging{opacity:.45}.order-handle{cursor:grab;color:var(--muted);font-size:17px}.order-item span:nth-child(2){flex:1}.order-button{width:30px;height:28px;border:1px solid var(--line);border-radius:7px;background:var(--bg);color:var(--text);cursor:pointer}
.services .fill{background:linear-gradient(90deg,#55d58a,#43c6d9)}.system-health .fill{background:linear-gradient(90deg,#6ba8ff,#9d8cff)}.service-form{display:grid;grid-template-columns:1fr 1fr;gap:12px}.field{display:grid;gap:6px;min-width:0}.field>span{color:var(--muted);font-size:11px;text-transform:uppercase;letter-spacing:.07em}.field small{color:var(--muted);font-size:11px}.field input,.field select{width:100%;min-width:0;padding:11px 12px;border:1px solid var(--line);border-radius:9px;background:var(--bg);color:var(--text)}.check-field{display:flex;align-items:center;gap:10px;padding:12px;border:1px solid var(--line);border-radius:9px}.check-field input{width:18px;height:18px;accent-color:var(--green)}.service-actions{display:flex;flex-wrap:wrap;gap:7px;margin-top:10px}.service-actions button,.open-address{padding:7px 10px;border:1px solid var(--line);border-radius:8px;background:var(--panel);color:var(--text);cursor:pointer;text-decoration:none}.status-up{color:var(--green)}.status-down{color:#ff665c}.issue-message{overflow-wrap:anywhere;line-height:1.6;margin-top:12px}.full-span{grid-column:1/-1}.endpoint-list{display:grid;gap:10px;margin-top:16px}.endpoint{padding:14px;border:1px solid var(--line);border-radius:12px;background:rgba(100,130,150,.04)}.endpoint-head{display:flex;align-items:flex-start;justify-content:space-between;gap:12px}.endpoint-address{color:var(--muted);font:12px ui-monospace,monospace;overflow-wrap:anywhere}.certificate{margin-top:11px;padding-top:11px;border-top:1px solid var(--line);color:var(--muted);font-size:12px}.certificate b{color:var(--text)}.maintenance-grid .detail{min-width:0}.maintenance-grid .detail strong{font-size:clamp(15px,2.2vw,21px);overflow-wrap:anywhere;word-break:break-word}.system-tabs{display:flex;align-items:center;gap:6px;padding:12px 26px;border-bottom:1px solid var(--line)}.system-panel[hidden]{display:none}.system-sort{margin-left:auto;padding:8px 11px;border:1px solid var(--line);border-radius:9px;background:var(--panel);color:var(--text);cursor:pointer;white-space:nowrap}
.snapshots .fill{background:linear-gradient(90deg,#9d8cff,#ff78b8)}.tasks .fill{background:linear-gradient(90deg,#43c6d9,#6ba8ff)}.homelab-map .fill{background:linear-gradient(90deg,#55d58a,#9d8cff)}.map-board{display:grid;grid-template-columns:170px 1fr 1fr 1fr;gap:14px;padding:18px;min-width:680px}.map-column{display:grid;align-content:start;gap:9px}.map-column-title{color:var(--muted);font-size:10px;text-transform:uppercase;letter-spacing:.1em}.map-node{padding:12px;border:1px solid var(--line);border-radius:12px;background:var(--panel)}.map-node.good{border-color:#2d6245}.map-node.bad{border-color:#71362f}.map-node.warn{border-color:#6d4c2c}.map-node strong{display:block}.map-node small{display:block;color:var(--muted);margin-top:3px}.map-scroll{overflow-x:auto}.backup-check{margin-top:9px;color:var(--muted);font-size:12px}.task-filter{display:flex;gap:6px;padding:12px 16px;overflow-x:auto}
.topology-board{padding:18px;min-width:620px}.topology-host{margin-bottom:14px}.topology-grid{display:grid;grid-template-columns:repeat(2,minmax(270px,1fr));gap:12px}.topology-group{padding:14px;border:1px solid var(--line);border-radius:15px;background:rgba(100,130,150,.04)}.topology-group-head{display:flex;justify-content:space-between;gap:10px;padding-bottom:10px;border-bottom:1px solid var(--line)}.topology-section{margin-top:11px}.topology-section-title{color:var(--muted);font-size:10px;text-transform:uppercase;letter-spacing:.08em;margin-bottom:6px}.topology-items{display:flex;flex-wrap:wrap;gap:6px}.topology-chip{padding:6px 9px;border:1px solid var(--line);border-radius:9px;background:var(--panel);font-size:12px}.topology-chip.good{border-color:#2d6245}.topology-chip.bad{border-color:#71362f}.topology-unassigned{margin-top:14px}@media(max-width:700px){.topology-grid{grid-template-columns:1fr}}
.monitor .fill{background:linear-gradient(90deg,#55d58a,#43c6d9)}.shutdown-plan{display:grid;gap:8px;margin-top:12px}.shutdown-step{padding:10px 12px;border:1px solid var(--line);border-radius:10px}.infra-status{display:grid;grid-template-columns:repeat(4,1fr);gap:8px;margin:14px 0}.infra-status>div{padding:11px;border:1px solid var(--line);border-radius:10px}.infra-status b{display:block;margin-top:4px}@media(max-width:700px){.infra-status{grid-template-columns:1fr 1fr}}
.agents .fill{background:linear-gradient(90deg,#55d58a,#6ba8ff)}.agent-command{display:block;width:100%;padding:12px;border:1px solid var(--line);border-radius:10px;background:var(--bg);color:var(--text);font:12px/1.5 ui-monospace,monospace;overflow-wrap:anywhere;white-space:pre-wrap}.agent-containers{display:flex;flex-wrap:wrap;gap:7px;margin-top:13px}
</style>
</head>
<body><main>
<header><div><div class="eyebrow">Proxmox host telemetry</div><h1>Host overview</h1></div><div class="header-actions"><div class="status"><i class="dot" id="dot"></i><span id="status">Подключение…</span></div><button class="settings-button" id="settingsButton" aria-label="Настройки">⚙</button></div></header>
<div class="maintenance-banner" id="maintenanceBanner" hidden><div><strong>Плановые работы</strong><small id="maintenanceBannerReason"></small></div><span id="maintenanceBannerEnds"></span></div>
<section class="cards">
<button class="card alert-center" data-card="alerts" data-open="alertsDialog" aria-label="Открыть центр уведомлений">
<div class="card-head"><div><div class="label">Notification center</div><div class="name" id="alertHeadline">Проверка состояния…</div></div><div class="percent"><span id="alertCount">—</span><small> событий</small></div></div>
@@ -214,7 +524,7 @@ footer{color:#657581;font-size:12px;margin-top:17px}
<button class="card zfs" data-card="zfs" data-open="zfsDialog" aria-label="Открыть подробности ZFS">
<div class="card-head"><div><div class="label">ZFS</div><div class="name">Пулы хранения</div></div><div class="percent"><span id="zfsCount">—</span><small> пулов</small></div></div>
<div class="bar"><div class="fill" id="zfsBar"></div></div>
<div class="quick"><div><span class="label">Состояние</span><strong id="zfsHealth">—</strong></div><div><span class="label">Занято</span><strong id="zfsUsed">—</strong></div></div>
<div class="quick"><div><span class="label">Все пулы</span><strong id="zfsHealth">—</strong></div><div><span class="label">Суммарно занято</span><strong id="zfsUsed">—</strong></div></div>
<div class="hint">Нажмите, чтобы посмотреть подробности →</div>
</button>
<button class="card ups" data-card="ups" data-open="upsDialog" aria-label="Открыть подробности ИБП">
@@ -223,25 +533,67 @@ footer{color:#657581;font-size:12px;margin-top:17px}
<div class="quick"><div><span class="label">Состояние</span><strong id="upsStatus">—</strong></div><div><span class="label">Автономность</span><strong id="upsRuntime">—</strong></div></div>
<div class="hint">Нажмите, чтобы посмотреть подробности →</div>
</button>
<button class="card monitor" data-card="monitor" data-open="monitorDialog" aria-label="Открыть самоконтроль дашборда">
<div class="card-head"><div><div class="label">Monitoring health</div><div class="name">Самоконтроль</div></div><div class="percent"><span id="monitorState">—</span></div></div><div class="bar"><div class="fill" id="monitorBar"></div></div>
<div class="quick"><div><span class="label">Последний сбор</span><strong id="monitorLast">—</strong></div><div><span class="label">База</span><strong id="monitorDB">—</strong></div></div><div class="hint">Watchdog, история и свежесть данных →</div>
</button>
<button class="card guests" data-card="guests" data-open="guestsDialog" aria-label="Открыть подробности VM и LXC">
<div class="card-head"><div><div class="label">Virtualization</div><div class="name">VM и LXC</div></div><div class="percent"><span id="guestRunning">—</span><small id="guestTotal"> / —</small></div></div>
<div class="bar"><div class="fill" id="guestBar"></div></div>
<div class="quick"><div><span class="label">CPU гостей</span><strong id="guestCPU">—</strong></div><div><span class="label">Память гостей</span><strong id="guestMemory">—</strong></div></div>
<div class="quick"><div><span class="label">CPU VM и LXC</span><strong id="guestCPU">—</strong></div><div><span class="label">Память VM и LXC</span><strong id="guestMemory">—</strong></div></div>
<div class="hint">Нажмите, чтобы посмотреть подробности →</div>
</button>
<button class="card agents" data-card="agents" data-open="agentsDialog" aria-label="Открыть агенты VM и LXC">
<div class="card-head"><div><div class="label">Guest telemetry</div><div class="name">Агенты VM и LXC</div></div><div class="percent"><span id="agentOnline">—</span><small id="agentTotal"> / —</small></div></div>
<div class="bar"><div class="fill" id="agentBar"></div></div>
<div class="quick"><div><span class="label">Docker-контейнеры</span><strong id="agentContainers">—</strong></div><div><span class="label">Проблемы</span><strong id="agentProblems">—</strong></div></div>
<div class="hint">Системные метрики и Docker внутри VM/LXC →</div>
</button>
<button class="card maintenance" data-card="maintenance" data-open="maintenanceDialog" aria-label="Открыть сведения Proxmox">
<div class="card-head"><div><div class="label">Proxmox</div><div class="name" id="maintenanceVersion">Обслуживание хоста</div></div><div class="percent"><span id="maintenanceUpdates">—</span><small> updates</small></div></div>
<div class="bar"><div class="fill" id="maintenanceBar"></div></div>
<div class="quick"><div><span class="label">Перезагрузка</span><strong id="maintenanceReboot">—</strong></div><div><span class="label">Ядро</span><strong id="maintenanceKernel">—</strong></div></div>
<div class="hint">Нажмите, чтобы посмотреть обслуживание →</div>
</button>
<button class="card backups" data-card="backups" data-open="backupsDialog" aria-label="Открыть подробности резервных копий">
<div class="card-head"><div><div class="label">Backups</div><div class="name">Резервные копии</div></div><div class="percent"><span id="backupProtected">—</span><small id="backupTotal"> / —</small></div></div>
<div class="bar"><div class="fill" id="backupBar"></div></div>
<div class="quick"><div><span class="label" id="backupFreshLabel">Свежие · 48 часов</span><strong id="backupFresh">—</strong></div><div><span class="label">Без копии</span><strong id="backupMissing">—</strong></div></div>
<div class="hint">Нажмите, чтобы посмотреть подробности →</div>
</button>
<button class="card services" data-card="services" data-open="servicesDialog" aria-label="Открыть мониторинг сервисов">
<div class="card-head"><div><div class="label">Homelab services</div><div class="name">Сервисы</div></div><div class="percent"><span id="serviceUp">—</span><small id="serviceTotal"> / —</small></div></div>
<div class="bar"><div class="fill" id="serviceBar"></div></div>
<div class="quick"><div><span class="label">Доступность · сутки</span><strong id="serviceUptime">—</strong></div><div><span class="label">Средний отклик</span><strong id="serviceLatency">—</strong></div></div>
<div class="hint">Нажмите, чтобы управлять проверками →</div>
</button>
<button class="card system-health" data-card="system" data-open="systemDialog" aria-label="Открыть системные события">
<div class="card-head"><div><div class="label">System health</div><div class="name">Система и ядро</div></div><div class="percent"><span id="systemIssues">—</span><small> ошибок</small></div></div>
<div class="bar"><div class="fill" id="systemBar"></div></div>
<div class="quick"><div><span class="label">Упавшие службы</span><strong id="failedUnits">—</strong></div><div><span class="label">Ошибки ядра</span><strong id="kernelErrors">—</strong></div></div>
<div class="hint">Нажмите, чтобы посмотреть журнал →</div>
</button>
<button class="card snapshots" data-card="snapshots" data-open="snapshotsDialog" aria-label="Открыть snapshots">
<div class="card-head"><div><div class="label">Snapshots</div><div class="name">Снимки VM и LXC</div></div><div class="percent"><span id="snapshotCount">—</span><small> шт.</small></div></div><div class="bar"><div class="fill" id="snapshotBar"></div></div>
<div class="quick"><div><span class="label">Старше 30 дней</span><strong id="oldSnapshots">—</strong></div><div><span class="label">VM/LXC со снимками</span><strong id="snapshotGuests">—</strong></div></div><div class="hint">Нажмите, чтобы посмотреть снимки →</div>
</button>
<button class="card tasks" data-card="tasks" data-open="tasksDialog" aria-label="Открыть задачи Proxmox">
<div class="card-head"><div><div class="label">Proxmox activity</div><div class="name">События и задачи</div></div><div class="percent"><span id="taskCount">—</span><small> задач</small></div></div><div class="bar"><div class="fill" id="taskBar"></div></div>
<div class="quick"><div><span class="label">Ошибки</span><strong id="failedTasks">—</strong></div><div><span class="label">Последняя операция</span><strong id="lastTask">—</strong></div></div><div class="hint">Нажмите, чтобы посмотреть журнал →</div>
</button>
<button class="card homelab-map" data-card="map" data-open="mapDialog" aria-label="Открыть карту homelab">
<div class="card-head"><div><div class="label">Topology</div><div class="name">Карта homelab</div></div><div class="percent"><span id="mapNodes">—</span><small> узлов</small></div></div><div class="bar"><div class="fill" id="mapBar"></div></div>
<div class="quick"><div><span class="label">VM / LXC</span><strong id="mapGuests">—</strong></div><div><span class="label">Сервисы / storage</span><strong id="mapResources">—</strong></div></div><div class="hint">Proxmox → VM/LXC → сервисы → storage →</div>
</button>
</section>
<footer>Обновление каждые 2 секунды · данные читаются напрямую с хоста</footer>
</main>
<dialog id="alertsDialog">
<div class="modal-head"><div><div class="label">Notification center</div><h2>Центр уведомлений</h2></div><button class="close" aria-label="Закрыть">×</button></div>
<div class="alert-list" id="alertList"><span class="empty">Проверка состояния системы…</span></div>
<div class="settings-tabs"><button class="settings-tab active" data-alert-tab="active">Активные</button><button class="settings-tab" data-alert-tab="history">История</button></div>
<div class="alert-list alert-panel" data-alert-panel="active" id="alertList"><span class="empty">Проверка состояния системы…</span></div>
<div class="alert-list alert-panel" data-alert-panel="history" id="alertHistory" hidden><span class="empty">Загрузка истории…</span></div>
</dialog>
<dialog id="cpuDialog">
@@ -271,8 +623,8 @@ footer{color:#657581;font-size:12px;margin-top:17px}
<div class="detail"><span class="label">Используется</span><strong id="dDiskUsed">—</strong></div><div class="detail"><span class="label">Свободно</span><strong id="dDiskFree">—</strong></div><div class="detail"><span class="label">Общий объём</span><strong id="dDiskTotal">—</strong></div>
<div class="detail"><span class="label">Заполнение</span><strong id="dDiskPercent">—</strong></div><div class="detail"><span class="label">Точка монтирования</span><strong>/</strong></div>
</div>
<div class="disk-list" id="diskList"><span class="empty">Поиск накопителей…</span></div>
<div class="chart-section"><div class="chart-head"><strong>Заполнение системного раздела</strong><span>Последние 7 дней</span></div><canvas class="metric-chart" id="storageChart"></canvas></div>
<div class="disk-list" id="diskList"><span class="empty">Поиск накопителей…</span></div><div class="wide"><span class="label">Расшифровка источников I/O</span><div class="disk-list" id="ioAttributionList"><span class="empty">Накопление данных…</span></div></div>
<div class="chart-section"><div class="chart-head"><strong>Температура и I/O физических дисков</strong><span>Отдельно по каждому диску</span></div><div class="label">Температура</div><canvas class="metric-chart" id="storageChart"></canvas><div class="label">IOPS</div><canvas class="metric-chart" id="diskIOChart"></canvas><div class="label">Utilization</div><canvas class="metric-chart" id="diskUtilChart"></canvas></div>
</dialog>
<dialog id="networkDialog">
@@ -281,13 +633,15 @@ footer{color:#657581;font-size:12px;margin-top:17px}
<div class="detail"><span class="label">Получение сейчас</span><strong id="dNetDown">—</strong></div><div class="detail"><span class="label">Передача сейчас</span><strong id="dNetUp">—</strong></div><div class="detail"><span class="label">Суммарная скорость</span><strong id="dNetTotal">—</strong></div>
<div class="detail"><span class="label">Получено с запуска</span><strong id="dNetReceived">—</strong></div><div class="detail"><span class="label">Передано с запуска</span><strong id="dNetSent">—</strong></div><div class="detail"><span class="label">Интерфейсы</span><strong id="dNetInterfaces">—</strong></div>
</div>
<div class="wide"><span class="label">Доступность инфраструктуры</span><div class="infra-status"><div><span>Шлюз</span><b id="dNetGateway">—</b></div><div><span>DNS</span><b id="dNetDNS">—</b></div><div><span>Интернет</span><b id="dNetInternet">—</b></div><div><span>Внешний IP</span><b id="dNetExternalIP">—</b></div></div></div>
<div class="disk-list" id="networkInterfacesList"></div>
<div class="chart-section"><div class="chart-head"><strong>Входящий и исходящий трафик</strong><span>Последние 7 дней</span></div><canvas class="metric-chart" id="networkChart"></canvas></div>
</dialog>
<dialog id="zfsDialog">
<div class="modal-head"><div><div class="label">ZFS details</div><h2>Пулы хранения</h2></div><button class="close" aria-label="Закрыть">×</button></div>
<div class="disk-list" id="zfsList"><span class="empty">Поиск ZFS-пулов…</span></div>
<div class="chart-section"><div class="chart-head"><strong>Заполнение ZFS</strong><span>Последние 7 дней</span></div><canvas class="metric-chart" id="zfsChart"></canvas></div>
<div class="wide"><span class="label">ARC</span><div class="infra-status"><div><span>Размер</span><b id="zfsArcSize">—</b></div><div><span>Цель</span><b id="zfsArcTarget">—</b></div><div><span>Hit ratio</span><b id="zfsArcHit">—</b></div><div><span>Datasets</span><b id="zfsDatasetCount">—</b></div></div></div><div class="disk-list" id="zfsList"><span class="empty">Поиск ZFS-пулов…</span></div>
<div class="chart-section"><div class="chart-head"><strong>Заполнение каждого ZFS-пула</strong><span>Отдельно по каждому пулу</span></div><canvas class="metric-chart" id="zfsChart"></canvas></div>
</dialog>
<dialog id="upsDialog">
@@ -296,15 +650,18 @@ footer{color:#657581;font-size:12px;margin-top:17px}
<div class="detail"><span class="label">Состояние</span><strong id="dUpsStatus">—</strong></div><div class="detail"><span class="label">Заряд</span><strong id="dUpsCharge">—</strong></div><div class="detail"><span class="label">Автономность</span><strong id="dUpsRuntime">—</strong></div>
<div class="detail"><span class="label">Нагрузка</span><strong id="dUpsLoad">—</strong></div><div class="detail"><span class="label">Входное напряжение</span><strong id="dUpsInput">—</strong></div><div class="detail"><span class="label">Выходное напряжение</span><strong id="dUpsOutput">—</strong></div>
<div class="detail"><span class="label">Частота</span><strong id="dUpsFrequency">—</strong></div><div class="detail"><span class="label">Батарея</span><strong id="dUpsBattery">—</strong></div><div class="detail"><span class="label">Температура</span><strong id="dUpsTemp">—</strong></div>
<div class="detail"><span class="label">Связь с UPS</span><strong id="dUpsCommunication">—</strong></div><div class="detail"><span class="label">Дата батареи</span><strong id="dUpsBatteryDate">—</strong></div><div class="detail"><span class="label">Возраст батареи</span><strong id="dUpsBatteryAge">—</strong></div>
</div>
<div class="wide"><span class="label">Источник данных</span><div class="temps"><span class="temp" id="dUpsSource">Не настроен</span></div></div>
<div class="chart-section"><div class="chart-head"><strong>Заряд и нагрузка UPS</strong><span>Последние 7 дней</span></div><canvas class="metric-chart" id="upsChart"></canvas></div>
</dialog>
<dialog id="monitorDialog"><div class="modal-head"><div><div class="label">Dashboard health</div><h2>Самоконтроль мониторинга</h2></div><button class="close" aria-label="Закрыть">×</button></div><div class="detail-grid"><div class="detail"><span class="label">Последний успешный сбор</span><strong id="dMonitorLast">—</strong></div><div class="detail"><span class="label">Длительность цикла</span><strong id="dMonitorDuration">—</strong></div><div class="detail"><span class="label">Последняя запись истории</span><strong id="dMonitorHistory">—</strong></div><div class="detail"><span class="label">Свободно для базы</span><strong id="dMonitorDB">—</strong></div><div class="detail"><span class="label">Systemd watchdog</span><strong id="dMonitorWatchdog">—</strong></div><div class="detail"><span class="label">Ошибка</span><strong id="dMonitorError">—</strong></div></div></dialog>
<dialog id="guestsDialog">
<div class="modal-head"><div><div class="label">Virtualization details</div><h2>VM и LXC</h2></div><button class="close" aria-label="Закрыть">×</button></div>
<div class="disk-list" id="guestList"><span class="empty">Получение списка гостей…</span></div>
<div class="chart-section"><div class="chart-head"><strong>Суммарные CPU и RAM гостей</strong><span>Последние 7 дней</span></div><canvas class="metric-chart" id="guestsChart"></canvas></div>
<div class="disk-list" id="guestList"><span class="empty">Получение списка VM и LXC…</span></div>
<div class="chart-section"><div class="chart-head"><strong>CPU каждой VM и LXC</strong><span>Отдельно по каждой VM и LXC</span></div><canvas class="metric-chart" id="guestsChart"></canvas></div>
</dialog>
<dialog id="backupsDialog">
@@ -312,19 +669,51 @@ footer{color:#657581;font-size:12px;margin-top:17px}
<div class="disk-list" id="backupList"><span class="empty">Получение списка резервных копий…</span></div>
</dialog>
<dialog id="maintenanceDialog">
<div class="modal-head"><div><div class="label">Proxmox maintenance</div><h2>Система и обслуживание</h2></div><button class="close" aria-label="Закрыть">×</button></div>
<div class="detail-grid maintenance-grid">
<div class="detail"><span class="label">Proxmox</span><strong id="dPveVersion">—</strong></div><div class="detail"><span class="label">Запущенное ядро</span><strong id="dKernel">—</strong></div><div class="detail"><span class="label">Новое ядро</span><strong id="dLatestKernel">—</strong></div>
<div class="detail"><span class="label">Обновления</span><strong id="dUpdates">—</strong></div><div class="detail"><span class="label">Перезагрузка</span><strong id="dReboot">—</strong></div><div class="detail"><span class="label">Последнее обновление</span><strong id="dLastUpdate">—</strong></div>
</div>
<div class="wide"><span class="label">Ближайшие задания</span><div class="disk-stats"><div class="disk-stat"><span>Backup</span><b id="dNextBackup">—</b></div><div class="disk-stat"><span>ZFS scrub</span><b id="dNextScrub">—</b></div><div class="disk-stat"><span>Проверено</span><b id="dMaintenanceChecked">—</b></div></div></div>
</dialog>
<dialog id="servicesDialog">
<div class="modal-head"><div><div class="label">Homelab monitoring</div><h2>Сервисы</h2></div><button class="close" aria-label="Закрыть">×</button></div>
<div class="disk-list" id="serviceList"><span class="empty">Добавьте первую проверку в настройках.</span></div>
</dialog>
<dialog id="openServiceDialog">
<div class="modal-head"><div><div class="label">Открыть сервис</div><h2 id="openServiceTitle">Выберите адрес</h2></div><button class="close" aria-label="Закрыть">×</button></div>
<div class="wide"><div class="settings-note" style="margin:0 0 12px">У сервиса указаны два адреса. Выберите, через какой открыть страницу.</div><div class="theme-options" id="openServiceChoices"></div></div>
</dialog>
<dialog id="systemDialog">
<div class="modal-head"><div><div class="label">System health</div><h2>Ошибки системы и ядра</h2></div><button class="close" aria-label="Закрыть">×</button></div>
<div class="detail-grid"><div class="detail"><span class="label">Ошибки · 24 часа</span><strong id="dSystemErrors">—</strong></div><div class="detail"><span class="label">Ядро</span><strong id="dKernelErrors">—</strong></div><div class="detail"><span class="label">Упавшие службы</span><strong id="dFailedUnits">—</strong></div></div>
<div class="system-tabs"><button class="detail-tab active" data-system-tab="issues">Ошибки журнала</button><button class="detail-tab" data-system-tab="services">Упавшие службы</button><button class="system-sort" id="systemSort" type="button" aria-label="Сначала показаны свежие события">Сначала свежие ↓</button></div>
<div class="disk-list system-panel" data-system-panel="issues" id="issueList"><span class="empty">Системных ошибок нет.</span></div>
<div class="disk-list system-panel" data-system-panel="services" id="failedServiceList" hidden><span class="empty">Упавших служб нет.</span></div>
</dialog>
<dialog id="snapshotsDialog"><div class="modal-head"><div><div class="label">Snapshots</div><h2>Снимки VM и LXC</h2></div><button class="close" aria-label="Закрыть">×</button></div><div class="disk-list" id="snapshotList"><span class="empty">Получение snapshots…</span></div></dialog>
<dialog id="tasksDialog"><div class="modal-head"><div><div class="label">Proxmox activity</div><h2>События и задачи</h2></div><button class="close" aria-label="Закрыть">×</button></div><div class="task-filter"><button class="detail-tab active" data-task-filter="all">Все</button><button class="detail-tab" data-task-filter="vzdump">Backup</button><button class="detail-tab" data-task-filter="qm">VM/LXC</button><button class="detail-tab" data-task-filter="error">Ошибки</button></div><div class="disk-list" id="taskList"><span class="empty">Получение задач…</span></div></dialog>
<dialog id="mapDialog"><div class="modal-head"><div><div class="label">Homelab topology</div><h2>Карта homelab</h2></div><button class="close" aria-label="Закрыть">×</button></div><div class="map-scroll"><div class="topology-board" id="mapBoard"></div></div><div class="wide settings-note">Порт от локального адреса отбрасывается. Полученный IP сравнивается с адресами VM/LXC из Proxmox. Storage берётся из конфигурации каждой VM и LXC.</div></dialog>
<dialog id="agentsDialog"><div class="modal-head"><div><div class="label">Guest telemetry</div><h2>Агенты VM и LXC</h2></div><button class="close" aria-label="Закрыть">×</button></div><div class="disk-list" id="agentList"><span class="empty">Подключённые агенты появятся здесь.</span></div></dialog>
<dialog id="settingsDialog">
<div class="modal-head"><div><div class="label">Dashboard</div><h2>Настройки</h2></div><button class="close" aria-label="Закрыть">×</button></div>
<div class="settings-tabs"><button class="settings-tab active" data-settings-tab="appearance">Интерфейс</button><button class="settings-tab" data-settings-tab="alerts">Уведомления</button></div>
<div class="settings-tabs"><button class="settings-tab active" data-settings-tab="appearance">Интерфейс</button><button class="settings-tab" data-settings-tab="alerts">Пороги</button><button class="settings-tab" data-settings-tab="maintenance-mode">Maintenance</button><button class="settings-tab" data-settings-tab="services">Сервисы</button><button class="settings-tab" data-settings-tab="agents">Агенты</button><button class="settings-tab" data-settings-tab="ups">UPS shutdown</button><button class="settings-tab" data-settings-tab="email">Почта</button><button class="settings-tab" data-settings-tab="updates">Обновления</button></div>
<div class="settings-body settings-panel" data-settings-panel="appearance">
<div class="settings-group"><div class="label">Тема оформления</div><div class="theme-options"><button class="theme-option" data-theme-choice="dark">Тёмная</button><button class="theme-option" data-theme-choice="light">Светлая</button></div></div>
<div class="settings-group"><div class="label">Видимые карточки</div><div class="setting-list">
<label class="setting-item"><input type="checkbox" data-card-toggle="alerts"> Центр уведомлений</label>
<label class="setting-item"><input type="checkbox" data-card-toggle="cpu"> CPU</label><label class="setting-item"><input type="checkbox" data-card-toggle="memory"> Память</label>
<label class="setting-item"><input type="checkbox" data-card-toggle="storage"> Диски</label><label class="setting-item"><input type="checkbox" data-card-toggle="network"> Сеть</label>
<label class="setting-item"><input type="checkbox" data-card-toggle="zfs"> ZFS</label><label class="setting-item"><input type="checkbox" data-card-toggle="ups"> UPS</label>
<label class="setting-item"><input type="checkbox" data-card-toggle="guests"> VM и LXC</label><label class="setting-item"><input type="checkbox" data-card-toggle="backups"> Резервные копии</label>
<label class="setting-item"><input type="checkbox" data-card-toggle="zfs"> ZFS</label><label class="setting-item"><input type="checkbox" data-card-toggle="ups"> UPS</label><label class="setting-item"><input type="checkbox" data-card-toggle="monitor"> Самоконтроль</label>
<label class="setting-item"><input type="checkbox" data-card-toggle="guests"> VM и LXC</label><label class="setting-item"><input type="checkbox" data-card-toggle="agents"> Агенты VM и LXC</label><label class="setting-item"><input type="checkbox" data-card-toggle="maintenance"> Proxmox</label><label class="setting-item"><input type="checkbox" data-card-toggle="backups"> Резервные копии</label><label class="setting-item"><input type="checkbox" data-card-toggle="services"> Сервисы</label><label class="setting-item"><input type="checkbox" data-card-toggle="system"> Система</label><label class="setting-item"><input type="checkbox" data-card-toggle="snapshots"> Snapshots</label><label class="setting-item"><input type="checkbox" data-card-toggle="tasks"> Задачи Proxmox</label><label class="setting-item"><input type="checkbox" data-card-toggle="map"> Карта homelab</label>
</div></div>
<div class="settings-group"><div class="label">Порядок карточек</div><div class="settings-note">Перетаскивайте строки или используйте стрелки.</div><div class="card-order" id="cardOrder"></div></div>
</div>
<div class="settings-body settings-panel" data-settings-panel="ups" hidden><div class="settings-group"><div class="label">Безопасное выключение при работе от батареи</div><div class="settings-note">Автоматическое выключение по умолчанию отключено. Dry-run ничего не останавливает и показывает будущий порядок.</div><div class="threshold-grid"><label class="setting-item"><input type="checkbox" id="upsShutdownEnabled"> Разрешить сценарий</label><label class="threshold-item"><span>Заряд, %</span><input type="number" id="upsShutdownCharge"></label><label class="threshold-item"><span>Автономность, сек.</span><input type="number" id="upsShutdownRuntime"></label><label class="threshold-item"><span>Ожидание VM/LXC, сек.</span><input type="number" id="upsShutdownGrace"></label><label class="field full-span"><span>Порядок VMID через запятую</span><input id="upsShutdownOrder" placeholder="101, 105, 220"><small>Неуказанные запущенные VM и LXC добавятся после них по VMID.</small></label></div><button class="save-settings" id="saveUPSShutdown">Сохранить сценарий</button><button class="theme-option" id="dryRunUPSShutdown" style="margin-top:10px;width:100%">Показать dry-run</button><div class="shutdown-plan" id="upsShutdownPlan"></div><div class="settings-note" id="upsShutdownStatus"></div></div></div>
<div class="settings-body settings-panel" data-settings-panel="alerts" hidden>
<div class="threshold-grid">
<label class="threshold-item"><span>CPU · предупреждение, %</span><input type="number" data-threshold="cpuWarning"></label><label class="threshold-item"><span>CPU · критично, %</span><input type="number" data-threshold="cpuCritical"></label>
@@ -335,9 +724,54 @@ footer{color:#657581;font-size:12px;margin-top:17px}
<label class="threshold-item"><span>ZFS · критично, %</span><input type="number" data-threshold="zfsCritical"></label><label class="threshold-item"><span>Температура HDD/SSD</span><input type="number" data-threshold="diskTempWarning"></label>
<label class="threshold-item"><span>Температура NVMe</span><input type="number" data-threshold="nvmeTempWarning"></label><label class="threshold-item"><span>UPS заряд · критично, %</span><input type="number" data-threshold="upsChargeCritical"></label>
<label class="threshold-item"><span>Максимальный возраст backup, ч</span><input type="number" data-threshold="backupMaxAgeHours"></label>
<label class="threshold-item"><span>HDD utilization, %</span><input type="number" data-threshold="diskHddUtil"></label><label class="threshold-item"><span>HDD latency, ms</span><input type="number" data-threshold="diskHddLatency"></label><label class="threshold-item"><span>HDD очередь</span><input type="number" data-threshold="diskHddQueue"></label>
<label class="threshold-item"><span>SATA SSD utilization, %</span><input type="number" data-threshold="diskSsdUtil"></label><label class="threshold-item"><span>SATA SSD latency, ms</span><input type="number" data-threshold="diskSsdLatency"></label><label class="threshold-item"><span>SATA SSD очередь</span><input type="number" data-threshold="diskSsdQueue"></label>
<label class="threshold-item"><span>NVMe utilization, %</span><input type="number" data-threshold="diskNvmeUtil"></label><label class="threshold-item"><span>NVMe latency, ms</span><input type="number" data-threshold="diskNvmeLatency"></label><label class="threshold-item"><span>NVMe очередь</span><input type="number" data-threshold="diskNvmeQueue"></label>
<label class="threshold-item"><span>Минимальная длительность, сек.</span><input type="number" data-threshold="diskIoDuration"></label><label class="field full-span"><span>Исключить диски из I/O-алертов</span><input type="text" data-threshold="diskIoExcluded" placeholder="sda, SERIAL123"><small>Имена или серийные номера через запятую.</small></label>
</div>
<button class="save-settings" id="saveThresholds">Сохранить пороги</button><div class="settings-note" id="thresholdStatus">Настройки хранятся на сервере и применяются для всех браузеров.</div>
</div>
<div class="settings-body settings-panel" data-settings-panel="maintenance-mode" hidden>
<div class="settings-group"><div class="label">Окно плановых работ</div><div class="settings-note">Связанные сбои останутся в истории с пометкой «Плановые работы», но не попадут в активные алерты и письма.</div></div>
<div class="threshold-grid" style="margin-top:12px"><label class="field full-span"><span>Причина</span><input id="maintenanceReason" placeholder="Например, обновление Vaultwarden"></label><label class="threshold-item"><span>Продолжительность, минут</span><input id="maintenanceDuration" type="number" min="5" max="43200" value="60"></label><label class="setting-item"><input type="checkbox" id="maintenanceAll"> Отключить все алерты</label></div>
<div class="settings-group"><div class="label">Сервисы</div><div class="maintenance-targets" id="maintenanceServices"><span class="empty">Загрузка сервисов…</span></div></div>
<div class="settings-group"><div class="label">VM и LXC</div><div class="maintenance-targets" id="maintenanceGuests"><span class="empty">Данные VM/LXC ещё не получены.</span></div></div>
<button class="save-settings" id="startMaintenance">Включить Maintenance</button><button class="theme-option" id="stopMaintenance" style="margin-top:9px;width:100%" disabled>Завершить плановые работы</button><div class="settings-note" id="maintenanceStatus">Режим выключен.</div>
</div>
<div class="settings-body settings-panel" data-settings-panel="services" hidden>
<div class="settings-group"><div class="label">Новая или изменяемая проверка</div><div class="service-form">
<input type="hidden" id="serviceId"><label class="field"><span>Название сервиса</span><input id="serviceName" placeholder="Например, Gitea"></label><label class="field"><span>Способ проверки</span><select id="serviceType"><option value="http">Открыть HTTP / HTTPS</option><option value="tcp">Подключиться к TCP-порту</option><option value="ping">Проверить ping</option></select></label>
<label class="field"><span>Внешний веб-адрес</span><input id="serviceURL" placeholder="https://git.myown.center"><small>Проверяется отдельно через DNS-имя.</small></label><label class="field"><span>Локальный адрес</span><input id="serviceLocal" placeholder="https://192.168.0.6:3000"><small>Проверяется отдельно из сети Proxmox.</small></label>
<label class="field"><span>Ожидаемый HTTP-код</span><input id="serviceCode" type="number" value="200"><small>Обычно 200 означает, что страница открылась успешно.</small></label><label class="field"><span>Текст, который должен быть на странице</span><input id="serviceText" placeholder="Необязательно"><small>Например, Gitea. Пустое поле отключает проверку текста.</small></label>
<label class="field"><span>Сколько ждать ответа</span><input id="serviceTimeout" type="number" value="5" min="1" max="30"><small>Таймаут в секундах, обычно достаточно 5.</small></label><label class="check-field"><input id="serviceEnabled" type="checkbox" checked><span><b>Проверка включена</b><small style="display:block">Сервис проверяется каждые 15 секунд.</small></span></label>
</div><button class="save-settings" id="saveService">Сохранить сервис</button><div class="settings-note" id="serviceStatus">Проверка выполняется каждые 15 секунд, история хранится 7 дней.</div></div>
<div class="settings-group"><div class="label">Добавленные сервисы</div><div class="disk-list" id="serviceSettingsList"><span class="empty">Список пуст.</span></div></div>
</div>
<div class="settings-body settings-panel" data-settings-panel="agents" hidden>
<div class="settings-group"><div class="label">Подключить новую VM или LXC</div><div class="settings-note">Токен одноразовый и действует 15 минут. Агент только читает системные метрики и Docker — команд управления в этой версии нет.</div><div class="threshold-grid" style="margin-top:12px"><label class="field"><span>Адрес Dashboard из VM/LXC</span><input id="agentServerURL" placeholder="https://dashboard.example.lan"></label><label class="field"><span>VMID · необязательно</span><input id="agentVMID" type="number" min="0" placeholder="105"></label></div><button class="save-settings" id="createAgentToken">Создать команду установки</button></div>
<div class="settings-group" id="agentInstallBlock" hidden><div class="label">Выполните внутри VM/LXC от root</div><code class="agent-command" id="agentInstallCommand"></code><div class="settings-note">Для Docker-метрик служба получает доступ к /var/run/docker.sock. Сам агент не принимает входящие подключения.</div></div>
<div class="settings-group"><div class="label">Подключённые агенты</div><div id="agentSettingsList" class="disk-list" style="padding:12px 0 0"><span class="empty">Агентов пока нет.</span></div></div>
</div>
<div class="settings-body settings-panel" data-settings-panel="email" hidden>
<div class="settings-group"><div class="setting-list"><label class="setting-item"><input type="checkbox" id="emailEnabled"> Включить почтовые дайджесты</label><label class="setting-item"><input type="checkbox" id="emailSkipTLS"> Не проверять TLS-сертификат</label></div><div class="settings-note">Только для доверенного сервера во внутренней сети.</div></div>
<div class="settings-group"><div class="label">SMTP и получатель</div><div class="threshold-grid">
<label class="threshold-item"><span>SMTP-сервер</span><input id="emailHost" type="text"></label><label class="threshold-item"><span>Порт</span><input id="emailPort" type="number"></label>
<label class="threshold-item"><span>Логин</span><input id="emailUsername" type="text"></label><label class="threshold-item"><span>Пароль</span><input id="emailPassword" type="password" placeholder="Не изменять"></label>
<label class="threshold-item"><span>Отправитель</span><input id="emailFrom" type="email"></label><label class="threshold-item"><span>Получатель</span><input id="emailTo" type="email"></label>
<label class="threshold-item"><span>Интервал, минут</span><input id="emailInterval" type="number" min="5"></label>
</div></div>
<div class="settings-group"><div class="label">Важность</div><div class="setting-list"><label class="setting-item"><input type="checkbox" data-email-severity="critical"> Критические</label><label class="setting-item"><input type="checkbox" data-email-severity="warning"> Предупреждения</label><label class="setting-item"><input type="checkbox" data-email-severity="info"> Информация</label></div></div>
<div class="settings-group"><div class="label">Какие разделы отправлять</div><div class="setting-list">
<label class="setting-item"><input type="checkbox" data-email-source="CPU"> CPU</label><label class="setting-item"><input type="checkbox" data-email-source="Память"> Память</label><label class="setting-item"><input type="checkbox" data-email-source="Storage"> Storage</label><label class="setting-item"><input type="checkbox" data-email-source="Диски"> Диски</label><label class="setting-item"><input type="checkbox" data-email-source="ZFS"> ZFS</label><label class="setting-item"><input type="checkbox" data-email-source="UPS"> UPS</label><label class="setting-item"><input type="checkbox" data-email-source="VM/LXC"> VM и LXC</label><label class="setting-item"><input type="checkbox" data-email-source="Агенты"> Агенты</label><label class="setting-item"><input type="checkbox" data-email-source="Docker"> Docker</label><label class="setting-item"><input type="checkbox" data-email-source="Proxmox"> Proxmox</label><label class="setting-item"><input type="checkbox" data-email-source="Backups"> Резервные копии</label><label class="setting-item"><input type="checkbox" data-email-source="Dashboard"> Dashboard</label><label class="setting-item"><input type="checkbox" data-email-source="Сервисы"> Сервисы</label><label class="setting-item"><input type="checkbox" data-email-source="Система"> Система</label><label class="setting-item"><input type="checkbox" data-email-source="Сеть"> Сеть</label>
</div></div>
<button class="save-settings" id="saveEmail">Сохранить настройки почты</button><button class="theme-option" id="testEmail" style="width:100%;margin-top:9px">Отправить тестовое письмо</button><div class="settings-note" id="emailStatus">Пароль хранится в локальной базе сервера и не возвращается в браузер.</div>
</div>
<div class="settings-body settings-panel" data-settings-panel="updates" hidden>
<div class="detail-grid">
<div class="detail"><span class="label">Установлена</span><strong id="updateCurrent">—</strong></div><div class="detail"><span class="label">Доступна</span><strong id="updateAvailable">—</strong></div><div class="detail"><span class="label">Режим</span><strong id="updateMode">—</strong></div>
</div>
<div class="wide"><div class="label">Состояние обновлений</div><div class="temps"><span class="temp" id="updateMessage">Проверка ещё не выполнялась</span></div><button class="save-settings" id="installUpdate">Проверить и установить обновление</button><div class="settings-note" id="updateChecked">Проверка запускается автоматически раз в сутки.</div></div>
</div>
</dialog>
<script>
@@ -347,37 +781,109 @@ const rate=n=>size(n)+'/s';
const safe=v=>String(v??'').replace(/[&<>"']/g,c=>({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;',"'":'&#39;'}[c]));
const uptime=s=>{const d=Math.floor(s/86400),h=Math.floor(s%86400/3600),m=Math.floor(s%3600/60);return d?d+'д '+h+'ч':h?h+'ч '+m+'м':m+'м'};
let alertSettings={backupMaxAgeHours:48};
function render(x){const c=x.cpu,m=x.memory,s=x.storage,n=x.network,z=x.zfs||{available:false,pools:[]},u=x.ups||{available:false},g=x.guests||{available:false,guests:[]},b=x.backups||{available:false,items:[]},alerts=x.alerts||[],disks=x.disks||[],avg=c.temperatures.length?c.temperatures.reduce((a,t)=>a+t.celsius,0)/c.temperatures.length:null;
let maintenanceWindow={},maintenanceGuests=[],maintenanceTargetSignature='';
let lastDashboardEvent=0;
let currentServiceStatuses=[];
let currentSystemHealth={issues:[],failedServices:[]},systemSortNewest=true;
function renderSystemHealth(sh){currentSystemHealth=sh||{issues:[],failedServices:[]};const issues=[...(sh.issues||[])].sort((a,b)=>systemSortNewest?b.time-a.time:a.time-b.time);el('issueList').innerHTML=issues.length?issues.map(v=>'<article class="disk-item"><div class="disk-top"><div><div class="disk-model">'+safe(v.unit)+'</div><div class="disk-meta">'+new Date(v.time*1000).toLocaleString('ru-RU')+'</div></div><span class="badge bad">'+(v.count>1?'Повторений: '+v.count:'1 раз')+'</span></div><div class="settings-note issue-message">'+safe(v.message)+'</div></article>').join(''):'<span class="empty">Критических сообщений за 24 часа нет.</span>';el('failedServiceList').innerHTML=(sh.failedServices||[]).length?sh.failedServices.map(v=>'<article class="disk-item"><div class="disk-top"><div><div class="disk-model">'+safe(v.unit)+'</div><div class="disk-meta">'+safe(v.description||'Описание отсутствует')+'</div></div><span class="badge bad">'+safe(v.state)+'</span></div><div class="disk-stats"><div class="disk-stat"><span>Загрузка</span><b>'+safe(v.load)+'</b></div><div class="disk-stat"><span>Состояние</span><b>'+safe(v.active)+'</b></div><div class="disk-stat"><span>Подсостояние</span><b>'+safe(v.state)+'</b></div></div></article>').join(''):'<span class="empty">Упавших systemd-служб нет.</span>';el('systemSort').textContent=systemSortNewest?'Сначала свежие ↓':'Сначала старые ↑';el('systemSort').setAttribute('aria-label',systemSortNewest?'Сначала показаны свежие события':'Сначала показаны старые события')}
function endpointHTML(v){const tls=v.tlsDays==null?'Нет TLS':v.tlsDays<0?'Сертификат истёк':v.tlsDays+' дн. до окончания',cert=v.tlsSubject?'<div class="certificate"><div>Сертификат: <b>'+safe(v.tlsSubject)+'</b></div><div>Домены: <b>'+safe((v.tlsDnsNames||[]).join(', ')||'не указаны')+'</b></div><div>Издатель: <b>'+safe(v.tlsIssuer||'—')+'</b></div><div>Для этого адреса: <b class="'+(v.tlsHostnameValid?'status-up':'status-down')+'">'+(v.tlsHostnameValid?'подходит':'не подходит')+'</b></div></div>':'';return '<div class="endpoint"><div class="endpoint-head"><div><b>'+safe(v.label)+'</b><div class="endpoint-address">'+safe(v.address)+'</div></div><span class="badge '+(v.up?'':'bad')+'">'+(v.up?'Доступен':'Недоступен')+'</span></div><div class="disk-stats"><div class="disk-stat"><span>Отклик</span><b>'+fixed(v.latencyMs)+' ms</b></div><div class="disk-stat"><span>HTTP</span><b>'+(v.statusCode||'—')+'</b></div><div class="disk-stat"><span>TLS</span><b>'+tls+'</b></div><div class="disk-stat"><span>Uptime · сутки</span><b>'+fixed(v.uptimeDay)+'%</b></div><div class="disk-stat"><span>Uptime · неделя</span><b>'+fixed(v.uptimeWeek)+'%</b></div></div>'+cert+(v.error?'<div class="settings-note status-down">'+safe(v.error)+'</div>':'')+'</div>'}
function renderServices(svc){const items=svc.services||[];currentServiceStatuses=items;const avg=items.length?items.reduce((a,v)=>a+v.uptimeDay,0)/items.length:0,live=items.filter(v=>v.up),latency=live.length?live.reduce((a,v)=>a+v.latencyMs,0)/live.length:0;el('serviceUp').textContent=svc.up||0;el('serviceTotal').textContent=' / '+(svc.total||0);el('serviceBar').style.width=(svc.total?svc.up/svc.total*100:0)+'%';el('serviceUptime').textContent=svc.total?fixed(avg)+'%':'—';el('serviceLatency').textContent=svc.total?fixed(latency)+' ms':'—';el('serviceList').innerHTML=items.length?items.map(v=>'<article class="disk-item"><div class="disk-top"><div><div class="disk-model">'+safe(v.name)+'</div><div class="disk-meta">'+(v.endpoints||[]).length+' '+((v.endpoints||[]).length===1?'адрес':'адреса')+' проверяются независимо</div></div><span class="badge '+(v.up?'':'bad')+'">'+(v.up?'Всё доступно':'Есть проблема')+'</span></div><div class="endpoint-list">'+(v.endpoints||[]).map(endpointHTML).join('')+'</div><div class="service-actions"><button data-service-open="'+v.id+'">Открыть сервис ↗</button></div></article>').join(''):'<span class="empty">Добавьте первую проверку в настройках.</span>'}
function renderMaintenanceTargets(){const signature=JSON.stringify([configuredServices.map(v=>[v.id,v.name]),maintenanceGuests.map(v=>[v.vmid,v.name,v.type]),maintenanceWindow.serviceIds||[],maintenanceWindow.vmids||[]]);if(signature===maintenanceTargetSignature)return;maintenanceTargetSignature=signature;const serviceIDs=new Set(maintenanceWindow.serviceIds||[]),vmids=new Set(maintenanceWindow.vmids||[]);el('maintenanceServices').innerHTML=configuredServices.length?configuredServices.map(v=>'<label class="setting-item"><input type="checkbox" data-maintenance-service="'+v.id+'" '+(serviceIDs.has(v.id)?'checked':'')+'> '+safe(v.name)+'</label>').join(''):'<span class="empty">Сервисы не добавлены.</span>';el('maintenanceGuests').innerHTML=maintenanceGuests.length?maintenanceGuests.map(v=>'<label class="setting-item"><input type="checkbox" data-maintenance-vm="'+v.vmid+'" '+(vmids.has(v.vmid)?'checked':'')+'> '+safe(v.name||('VMID '+v.vmid))+' · '+(v.type==='qemu'?'VM':'LXC')+'</label>').join(''):'<span class="empty">VM/LXC не найдены.</span>';toggleMaintenanceTargets()}
function toggleMaintenanceTargets(){const disabled=el('maintenanceAll').checked;document.querySelectorAll('[data-maintenance-service],[data-maintenance-vm]').forEach(v=>v.disabled=disabled)}
function applyMaintenanceState(window){maintenanceWindow=window||{};const active=!!maintenanceWindow.active;el('maintenanceBanner').hidden=!active;el('maintenanceBannerReason').textContent=maintenanceWindow.reason||'';el('maintenanceBannerEnds').textContent=active?'до '+new Date(maintenanceWindow.endsAt*1000).toLocaleString('ru-RU'):'';el('stopMaintenance').disabled=!active;el('maintenanceStatus').textContent=active?'Активен до '+new Date(maintenanceWindow.endsAt*1000).toLocaleString('ru-RU')+' · '+maintenanceWindow.reason:'Режим выключен.';if(active){el('maintenanceReason').value=maintenanceWindow.reason||'';el('maintenanceDuration').value=Math.max(5,Math.ceil((maintenanceWindow.endsAt-Date.now()/1000)/60));el('maintenanceAll').checked=!!maintenanceWindow.allAlerts}renderMaintenanceTargets()}
function renderNetworkGroups(interfaces){const groups=new Map();interfaces.forEach(v=>{const key=v.vmid?'guest-'+v.vmid:'host',group=groups.get(key)||{vmid:v.vmid,name:v.vmid?(v.guestName||('VMID '+v.vmid)):'Интерфейсы хоста',type:v.guestType||'Host',items:[]};group.items.push(v);groups.set(key,group)});el('networkInterfacesList').innerHTML=[...groups.values()].map(g=>{const primary=g.vmid?g.items.filter(v=>v.name.startsWith('tap')||v.name.startsWith('veth')):g.items,rx=primary.reduce((a,v)=>a+v.receivePerSec,0),tx=primary.reduce((a,v)=>a+v.transmitPerSec,0),errors=g.items.reduce((a,v)=>a+v.receiveErrors+v.transmitErrors+v.receiveDrops+v.transmitDrops,0),down=primary.some(v=>v.kind==='Physical'&&!v.carrier&&(v.master||v.receiveBytes+v.transmitBytes>0));return '<article class="disk-item"><div class="disk-top"><div><div class="disk-model">'+safe(g.name)+'</div><div class="disk-meta">'+(g.vmid?'VMID '+g.vmid+' · ':'')+safe(g.type)+' · '+safe(g.items.map(v=>v.name+' ['+(v.kind||'Interface')+(v.speedMbps?' · '+v.speedMbps+'Mb/s':'')+(v.duplex?' · '+v.duplex:'')+(v.master?' → '+v.master:'')+']').join(', '))+'</div></div><span class="badge '+(errors||down?'warn':'')+'"'+(errors||down?' tabindex="0" data-reason="'+safe(down?'Нет carrier на используемом физическом интерфейсе':'Счётчики errors/drops больше нуля')+'"':'')+'>'+(down?'Нет линка':errors?'Есть потери':'Без ошибок')+'</span></div><div class="disk-stats"><div class="disk-stat"><span>Сейчас ↓</span><b>'+rate(rx)+'</b></div><div class="disk-stat"><span>Сейчас ↑</span><b>'+rate(tx)+'</b></div><div class="disk-stat"><span>Получено</span><b>'+size(primary.reduce((a,v)=>a+v.receiveBytes,0))+'</b></div><div class="disk-stat"><span>Передано</span><b>'+size(primary.reduce((a,v)=>a+v.transmitBytes,0))+'</b></div><div class="disk-stat"><span>Ошибки / drops</span><b>'+errors+'</b></div></div></article>'}).join('')}
function renderIOAttribution(disks){const active=disks.filter(d=>d.ioCause||(d.topConsumers||[]).length);el('ioAttributionList').innerHTML=active.length?active.map(d=>'<article class="disk-item"><div class="disk-top"><div><div class="disk-model">'+safe(d.name)+' · '+safe(d.ioCause||'Источник не определён')+'</div><div class="disk-meta">'+safe(d.ioCauseDetail||'Собираем статистику процессов')+'</div></div><span class="badge '+(d.ioConfidence>=80?'':'warn')+'" tabindex="0" data-reason="Уверенность учитывает совпадение активной задачи Proxmox и долю I/O процессов">'+(d.ioConfidence||0)+'%</span></div><div class="topology-items">'+(d.topConsumers||[]).slice(0,5).map(v=>'<span class="topology-chip">'+safe(v.kind)+' · '+safe(v.name)+' · '+fixed(v.sharePercent)+'% · ↓'+rate(v.readBytesPerSec)+' ↑'+rate(v.writeBytesPerSec)+'</span>').join('')+'</div></article>').join(''):'<span class="empty">Для атрибуции нужно несколько realtime-циклов.</span>'}
const taskLabel=type=>({vzdump:'Backup',qmstart:'Запуск VM',qmstop:'Остановка VM',qmshutdown:'Выключение VM',qmsnapshot:'Snapshot VM',vzstart:'Запуск LXC',vzstop:'Остановка LXC',vzsnapshot:'Snapshot LXC',qmigrate:'Миграция',aptupdate:'Обновление пакетов'})[type]||type;
function explainWarningBadges(){el('snapshotList').querySelectorAll('.badge.warn').forEach(v=>{v.tabIndex=0;v.dataset.reason='Snapshot старше 30 дней. Проверьте, нужен ли он: старые снимки могут занимать место в storage.'});el('taskList').querySelectorAll('.badge.bad').forEach(v=>{v.tabIndex=0;v.dataset.reason='Задача Proxmox завершилась не со статусом OK. Откройте её журнал в интерфейсе Proxmox для подробностей.'})}
let currentTasks=[],currentTaskFilter='all';
function renderTasks(tasks,filter='all'){currentTasks=tasks;const shown=tasks.filter(v=>filter==='all'||filter==='error'&&v.status!=='OK'||filter==='qm'&&(/^(qm|vz)/.test(v.type))||v.type===filter);el('taskList').innerHTML=shown.length?shown.map(v=>'<article class="disk-item"><div class="disk-top"><div><div class="disk-model">'+safe(taskLabel(v.type))+(v.id?' · '+safe(v.id):'')+'</div><div class="disk-meta">'+new Date(v.starttime*1000).toLocaleString('ru-RU')+' · '+safe(v.user||'—')+'</div></div><span class="badge '+(v.status==='OK'?'':'bad')+'">'+safe(v.status||'Выполняется')+'</span></div><div class="disk-stats"><div class="disk-stat"><span>Длительность</span><b>'+uptime(v.duration||Math.max(0,Date.now()/1000-v.starttime))+'</b></div><div class="disk-stat"><span>Узел</span><b>'+safe(v.node)+'</b></div><div class="disk-stat"><span>VMID</span><b>'+safe(v.id||'—')+'</b></div></div></article>').join(''):'<span class="empty">Задач этого типа нет.</span>'}
function renderActivity(activity,guests,services,zfs){const snapshots=activity.snapshots||[],tasks=activity.tasks||[],old=snapshots.filter(v=>v.ageSeconds>30*86400),guestCount=new Set(snapshots.map(v=>v.vmid)).size,failed=tasks.filter(v=>v.status&&v.status!=='OK');el('snapshotCount').textContent=snapshots.length;el('oldSnapshots').textContent=old.length;el('snapshotGuests').textContent=guestCount;el('snapshotBar').style.width=Math.min(old.length*20,100)+'%';el('snapshotList').innerHTML=snapshots.length?snapshots.map(v=>'<article class="disk-item"><div class="disk-top"><div><div class="disk-model">'+safe(v.guestName||('VMID '+v.vmid))+' · '+safe(v.name)+'</div><div class="disk-meta">VMID '+v.vmid+' · '+safe(v.guestType)+(v.createdAt?' · '+new Date(v.createdAt*1000).toLocaleString('ru-RU'):'')+'</div></div><span class="badge '+(v.ageSeconds>30*86400?'warn':'')+'">'+(v.ageSeconds?uptime(v.ageSeconds):'Без даты')+'</span></div><div class="disk-stats"><div class="disk-stat"><span>Размер</span><b>'+(v.sizeBytes?size(v.sizeBytes):'Storage не сообщает')+'</b></div><div class="disk-stat"><span>Состояние VM</span><b>'+(v.running?'Запущена':'Остановлена')+'</b></div><div class="disk-stat"><span>Описание</span><b>'+safe(v.description||'—')+'</b></div></div></article>').join(''):'<span class="empty">Snapshots не найдены.</span>';el('taskCount').textContent=tasks.length;el('failedTasks').textContent=failed.length;el('lastTask').textContent=tasks.length?taskLabel(tasks[0].type):'—';el('taskBar').style.width=Math.min(failed.length*20,100)+'%';renderTasks(tasks,currentTaskFilter);renderHomelabMap(guests,services,zfs)}
function topologyHost(address){if(!address)return '';try{return new URL(address.includes('://')?address:'http://'+address).hostname.replace(/^\[|\]$/g,'')}catch(e){return address.replace(/^[a-z]+:\/\//i,'').split('/')[0].replace(/:\d+$/,'')}}
function renderHomelabMap(guests,services,zfs){const guestItems=(guests.guests||[]).filter(v=>!v.template),serviceItems=services.services||[],pools=zfs.pools||[],serviceOwners=new Map();serviceItems.forEach(s=>{const host=topologyHost(s.localAddress||'');const owner=guestItems.find(g=>(g.ipAddresses||[]).some(ip=>ip===host));serviceOwners.set(s.id,owner?.vmid||0)});const serviceChip=s=>{const healthy=(s.endpoints||[]).every(v=>v.up);return '<span class="topology-chip '+(healthy?'good':'bad')+'"'+(!healthy?' tabindex="0" data-reason="Один или несколько адресов сервиса недоступны"':'')+'>'+safe(s.name)+'</span>'},storageChip=name=>{const pool=pools.find(p=>p.name===name),healthy=!pool||pool.health==='ONLINE';return '<span class="topology-chip '+(healthy?'good':'bad')+'">'+safe(name)+(pool?' · '+fixed(pool.capacityPercent)+'%':'')+'</span>'};const groups=guestItems.map(g=>{const owned=serviceItems.filter(s=>serviceOwners.get(s.id)===g.vmid),storage=[...new Set(g.storage||[])],running=g.status==='running';return '<section class="topology-group"><div class="topology-group-head"><div><strong>'+safe(g.name||('VMID '+g.vmid))+'</strong><div class="disk-meta">VMID '+g.vmid+' · '+safe(g.type==='qemu'?'VM':'LXC')+' · '+safe((g.ipAddresses||[]).join(', ')||'IP не получен')+'</div></div><span class="badge '+(running?'':'warn')+'"'+(!running?' tabindex="0" data-reason="VM/LXC сейчас остановлена"':'')+'>'+(running?'Работает':'Остановлена')+'</span></div><div class="topology-section"><div class="topology-section-title">Сервисы</div><div class="topology-items">'+(owned.length?owned.map(serviceChip).join(''):'<span class="empty">Не сопоставлены</span>')+'</div></div><div class="topology-section"><div class="topology-section-title">Storage</div><div class="topology-items">'+(storage.length?storage.map(storageChip).join(''):'<span class="empty">Не определён</span>')+'</div></div></section>'}).join(''),unassigned=serviceItems.filter(s=>!serviceOwners.get(s.id)),usedStorage=new Set(guestItems.flatMap(g=>g.storage||[])),unassignedPools=pools.filter(p=>!usedStorage.has(p.name));el('mapBoard').innerHTML='<div class="topology-host map-node good"><strong>Proxmox host</strong><small>'+safe(location.hostname)+' · '+guestItems.length+' VM/LXC</small></div><div class="topology-grid">'+groups+'</div>'+((unassigned.length||unassignedPools.length)?'<section class="topology-group topology-unassigned"><div class="topology-group-head"><strong>Без автоматической привязки</strong><span class="badge warn" tabindex="0" data-reason="Для сервиса не найден совпадающий IP VM/LXC либо storage не используется в конфигурации VM/LXC">Проверьте связи</span></div><div class="topology-section"><div class="topology-section-title">Сервисы</div><div class="topology-items">'+unassigned.map(serviceChip).join('')+'</div></div><div class="topology-section"><div class="topology-section-title">Остальные ZFS-пулы</div><div class="topology-items">'+unassignedPools.map(p=>storageChip(p.name)).join('')+'</div></div></section>':'');const storage=new Set([...usedStorage,...pools.map(p=>p.name)]);el('mapGuests').textContent=guestItems.length;el('mapResources').textContent=serviceItems.length+' / '+storage.size;el('mapNodes').textContent=1+guestItems.length+serviceItems.length+storage.size;el('mapBar').style.width='100%'}
function agentHTML(a,settings=false){const r=a.report||{},containers=r.containers||[],memoryPercent=r.memoryTotal?r.memoryUsed/r.memoryTotal*100:0,diskPercent=r.rootTotal?r.rootUsed/r.rootTotal*100:0,problems=containers.filter(v=>v.state!=='running'||v.health==='unhealthy');return '<article class="disk-item"><div class="disk-top"><div><div class="disk-model">'+safe(a.name||a.hostname)+'</div><div class="disk-meta">'+safe(a.hostname)+' · VMID '+(a.vmid||'не указан')+' · агент '+safe(a.version||'—')+' · '+(a.online?'heartbeat сейчас':'последний '+new Date(a.lastSeen*1000).toLocaleString('ru-RU'))+'</div></div><span class="badge '+(a.online?'':'bad')+'">'+(a.online?'Онлайн':'Не отвечает')+'</span></div><div class="disk-stats"><div class="disk-stat"><span>CPU / Load</span><b>'+fixed(r.cpuPercent||0)+'% / '+fixed(r.load1||0)+'</b></div><div class="disk-stat"><span>Память</span><b>'+size(r.memoryUsed)+' · '+fixed(memoryPercent)+'%</b></div><div class="disk-stat"><span>Раздел /</span><b>'+size(r.rootUsed)+' · '+fixed(diskPercent)+'%</b></div><div class="disk-stat"><span>Uptime</span><b>'+uptime(r.uptimeSeconds||0)+'</b></div><div class="disk-stat"><span>Docker</span><b>'+(r.dockerAvailable?containers.length+' контейнеров':safe(r.dockerError||'Нет данных'))+'</b></div><div class="disk-stat"><span>Проблемы</span><b>'+problems.length+'</b></div></div><div class="agent-containers">'+(containers.length?containers.map(v=>'<span class="topology-chip '+(v.state==='running'&&v.health!=='unhealthy'?'good':'bad')+'">'+safe(v.name)+' · '+safe(v.health||v.state)+' · CPU '+fixed(v.cpuPercent||0)+'% · RAM '+size(v.memoryUsage||0)+' · restarts '+v.restarts+'</span>').join(''):'<span class="empty">Docker-контейнеры не обнаружены</span>')+'</div>'+(settings?'<div class="service-actions"><button data-agent-delete="'+safe(a.id)+'">Удалить агент</button></div>':'')+'</article>'}
function renderAgents(items){const agents=items||[],online=agents.filter(v=>v.online),containers=agents.flatMap(v=>(v.report&&v.report.containers)||[]),problems=(agents.length-online.length)+containers.filter(v=>v.state!=='running'||v.health==='unhealthy').length;el('agentOnline').textContent=online.length;el('agentTotal').textContent=' / '+agents.length;el('agentBar').style.width=(agents.length?online.length/agents.length*100:0)+'%';el('agentContainers').textContent=containers.length;el('agentProblems').textContent=problems;el('agentList').innerHTML=agents.length?agents.map(v=>agentHTML(v)).join(''):'<span class="empty">Агенты пока не подключены. Создайте команду установки в настройках.</span>';el('agentSettingsList').innerHTML=agents.length?agents.map(v=>agentHTML(v,true)).join(''):'<span class="empty">Агентов пока нет.</span>'}
function render(x){const c=x.cpu,m=x.memory,s=x.storage,n=x.network,z=x.zfs||{available:false,pools:[]},u=x.ups||{available:false},g=x.guests||{available:false,guests:[]},b=x.backups||{available:false,items:[]},mt=x.maintenance||{},alerts=x.alerts||[],disks=x.disks||[],avg=c.temperatures.length?c.temperatures.reduce((a,t)=>a+t.celsius,0)/c.temperatures.length:null;
lastDashboardEvent=Date.now();
maintenanceGuests=(g.guests||[]).filter(v=>!v.template);applyMaintenanceState(x.maintenanceWindow||{});
renderAgents(x.agents||[]);
const update=x.update||{},updating=['queued','installing'].includes(update.state);el('updateCurrent').textContent=update.currentVersion||'—';el('updateAvailable').textContent=update.availableVersion||'—';el('updateMode').textContent=update.mode==='auto'?'Автоматически':'Уведомлять';el('updateMessage').textContent=update.message||'Проверка ещё не выполнялась';el('updateChecked').textContent=update.checkedAt?'Последняя проверка: '+new Date(update.checkedAt).toLocaleString('ru-RU'):'Проверка запускается автоматически раз в сутки.';el('installUpdate').disabled=updating;el('installUpdate').textContent=updating?'Обновление выполняется…':'Проверить и установить обновление';
const critical=alerts.filter(a=>a.severity==='critical').length,warnings=alerts.filter(a=>a.severity==='warning').length,infos=alerts.filter(a=>a.severity==='info').length;el('alertCount').textContent=alerts.length;el('alertCritical').textContent=critical;el('alertWarnings').textContent=warnings;el('alertHeadline').textContent=critical?'Требуется срочное внимание':warnings?'Есть предупреждения':infos?'Есть информационные события':'Система в норме';el('alertBar').style.width=(critical?100:warnings?65:infos?30:0)+'%';
el('alertList').innerHTML=alerts.length?alerts.map(a=>'<article class="alert-item '+safe(a.severity)+'"><i class="alert-icon"></i><div><div class="alert-source">'+safe(a.source)+' · '+(a.severity==='critical'?'Критическое':a.severity==='warning'?'Предупреждение':'Информация')+'</div><div class="alert-title">'+safe(a.title)+'</div><div class="alert-message">'+safe(a.message)+'</div></div></article>').join(''):'<div class="all-good"><strong>Всё в порядке</strong><span>Активных предупреждений сейчас нет.</span></div>';
el('alertList').innerHTML=alerts.length?alerts.map(a=>'<article class="alert-item '+safe(a.severity)+'"><i class="alert-icon"></i><div><div class="alert-source">'+safe(a.source)+' · '+(a.severity==='critical'?'Критическое':a.severity==='warning'?'Предупреждение':'Информация')+(a.firstSeen?' · С '+new Date(a.firstSeen*1000).toLocaleString('ru-RU'):'')+'</div><div class="alert-title">'+safe(a.title)+'</div><div class="alert-message">'+safe(a.message)+'</div><button class="ack-alert" data-ack="'+a.eventId+'">Принято</button></div></article>').join(''):'<div class="all-good"><strong>Всё в порядке</strong><span>Новых неподтверждённых событий нет.</span></div>';
el('cpuName').textContent=el('cpuDetailName').textContent=c.model;el('cpuUsage').textContent=fixed(c.usagePercent);el('cpuBar').style.width=Math.min(c.usagePercent,100)+'%';el('cpuTemp').textContent=avg===null?'Нет датчика':fixed(avg)+' °C';el('cpuTopology').textContent=c.physicalCores+' / '+c.logicalCpus;
el('dCpuUsage').textContent=fixed(c.usagePercent)+'%';el('dSockets').textContent=c.sockets;el('dCores').textContent=c.physicalCores;el('dThreads').textContent=c.logicalCpus;el('dFreq').textContent=(c.frequencyMhz/1000).toFixed(2)+' GHz';el('dUptime').textContent=uptime(c.uptimeSeconds);el('dLoad1').textContent=fixed(c.load1);el('dLoad5').textContent=fixed(c.load5);el('dLoad15').textContent=fixed(c.load15);el('dTemps').innerHTML=c.temperatures.length?c.temperatures.map(t=>'<span class="temp">'+t.label+'<b>'+fixed(t.celsius)+'°C</b></span>').join(''):'<span class="empty">Датчики температуры не найдены</span>';
el('memUsage').textContent=fixed(m.usagePercent);el('memBar').style.width=Math.min(m.usagePercent,100)+'%';el('memUsed').textContent=size(m.usedBytes);el('memAvailable').textContent=size(m.availableBytes);el('dMemUsed').textContent=size(m.usedBytes);el('dMemAvailable').textContent=size(m.availableBytes);el('dMemTotal').textContent=size(m.totalBytes);el('dMemCache').textContent=size(m.cachedBytes);el('dMemBuffers').textContent=size(m.buffersBytes);el('dMemPercent').textContent=fixed(m.usagePercent)+'%';el('dSwapUsed').textContent=size(m.swapUsedBytes);el('dSwapTotal').textContent=size(m.swapTotalBytes);el('dSwapPercent').textContent=fixed(m.swapPercent)+'%';
const failed=disks.filter(d=>d.smartStatus==='Ошибка').length,attention=disks.filter(d=>d.smartStatus==='Требует внимания').length,available=disks.filter(d=>d.smartAvailable).length;let health=failed?'Ошибка':attention?'Внимание':available===disks.length&&disks.length?'Исправны':available?'Частично':'Нет данных';
el('diskCount').textContent=disks.length;el('diskCountLabel').textContent=disks.length===1?' диск':' дисков';el('diskHealth').textContent=health;el('diskUsage').textContent=fixed(s.usagePercent)+'%';el('diskBar').style.width=Math.min(s.usagePercent,100)+'%';el('dDiskUsed').textContent=size(s.usedBytes);el('dDiskFree').textContent=size(s.freeBytes);el('dDiskTotal').textContent=size(s.totalBytes);el('dDiskPercent').textContent=fixed(s.usagePercent)+'%';
el('diskList').innerHTML=disks.length?disks.map(d=>{const badge=d.smartStatus==='Ошибка'?'bad':d.smartStatus==='Исправен'?'':'warn',reason=(d.attentionReasons||[]).join(' · '),tip=reason?' tabindex="0" data-reason="'+safe(reason)+'"':'';return '<article class="disk-item"><div class="disk-top"><div><div class="disk-model">'+safe(d.model)+'</div><div class="disk-meta">'+safe(d.path)+' · SN: '+safe(d.serial)+'</div></div><span class="badge '+badge+'"'+tip+'>'+safe(d.smartStatus)+'</span></div><div class="disk-stats"><div class="disk-stat"><span>Тип</span><b>'+safe(d.type)+'</b></div><div class="disk-stat"><span>Объём</span><b>'+size(d.sizeBytes)+'</b></div><div class="disk-stat"><span>Занято</span><b>'+(d.usedBytes==null?'—':size(d.usedBytes)+' · '+fixed(d.usagePercent)+'%')+'</b></div><div class="disk-stat"><span>Температура</span><b>'+(d.temperatureCelsius==null?'—':fixed(d.temperatureCelsius)+' °C')+'</b></div><div class="disk-stat"><span>Наработка</span><b>'+(d.powerOnHours==null?'—':d.powerOnHours+' ч')+'</b></div><div class="disk-stat"><span>Износ</span><b>'+(d.wearUsedPercent==null?'—':fixed(d.wearUsedPercent)+'%')+'</b></div></div></article>'}).join(''):'<span class="empty">Физические накопители не найдены</span>';
const totalRate=n.receivePerSec+n.transmitPerSec;el('netInterfaces').textContent=n.interface||'Сетевые интерфейсы';el('netTotal').textContent=(totalRate/1048576).toFixed(2);el('netBar').style.width=Math.min(totalRate/125000000*100,100)+'%';el('netDown').textContent=rate(n.receivePerSec);el('netUp').textContent=rate(n.transmitPerSec);el('dNetDown').textContent=rate(n.receivePerSec);el('dNetUp').textContent=rate(n.transmitPerSec);el('dNetTotal').textContent=rate(totalRate);el('dNetReceived').textContent=size(n.receiveBytes);el('dNetSent').textContent=size(n.transmitBytes);el('dNetInterfaces').textContent=n.interface||'—';
const pools=z.pools||[],zTotal=pools.reduce((a,p)=>a+p.sizeBytes,0),zUsed=pools.reduce((a,p)=>a+p.allocatedBytes,0),zPercent=zTotal?zUsed/zTotal*100:0,zBad=pools.filter(p=>p.health!=='ONLINE').length;el('zfsCount').textContent=pools.length;el('zfsHealth').textContent=!z.available?'Не найден':zBad?'Внимание':pools.length?'Исправны':'Нет пулов';el('zfsUsed').textContent=zTotal?size(zUsed)+' · '+fixed(zPercent)+'%':'—';el('zfsBar').style.width=Math.min(zPercent,100)+'%';
el('zfsList').innerHTML=pools.length?pools.map(p=>{const bad=p.health!=='ONLINE',reason=(p.attentionReasons||[]).join(' · '),tip=reason?' tabindex="0" data-reason="'+safe(reason)+'"':'';return '<article class="disk-item"><div class="disk-top"><div><div class="disk-model">'+safe(p.name)+'</div><div class="disk-meta">'+safe(p.scan||'Данные scrub отсутствуют')+'</div></div><span class="badge '+(bad?'bad':'')+'"'+tip+'>'+safe(p.health)+'</span></div><div class="disk-stats"><div class="disk-stat"><span>Объём</span><b>'+size(p.sizeBytes)+'</b></div><div class="disk-stat"><span>Занято</span><b>'+size(p.allocatedBytes)+'</b></div><div class="disk-stat"><span>Свободно</span><b>'+size(p.freeBytes)+'</b></div><div class="disk-stat"><span>Заполнение</span><b>'+fixed(p.capacityPercent)+'%</b></div><div class="disk-stat"><span>Фрагментация</span><b>'+fixed(p.fragmentationPercent)+'%</b></div><div class="disk-stat"><span>Dedup</span><b>'+safe(p.dedupRatio)+'</b></div></div></article>'}).join(''):'<span class="empty">'+(z.available?'ZFS-пулы не найдены':'Команда zpool недоступна')+'</span>';
const val=(v,suffix='')=>v==null?'—':fixed(v)+suffix;el('upsName').textContent=u.available?(u.model||u.name||'Источник бесперебойного питания'):'UPS не настроен';el('upsCharge').textContent=u.chargePercent==null?'—':fixed(u.chargePercent);el('upsBar').style.width=u.chargePercent==null?'0%':Math.min(u.chargePercent,100)+'%';el('upsStatus').textContent=u.available?u.statusLabel:'Не найден';el('upsRuntime').textContent=u.runtimeSeconds==null?'—':uptime(u.runtimeSeconds);el('dUpsName').textContent=u.model||u.name||'Источник бесперебойного питания';el('dUpsStatus').textContent=u.available?u.statusLabel:'Не настроен';el('dUpsCharge').textContent=val(u.chargePercent,'%');el('dUpsRuntime').textContent=u.runtimeSeconds==null?'—':uptime(u.runtimeSeconds);el('dUpsLoad').textContent=val(u.loadPercent,'%');el('dUpsInput').textContent=val(u.inputVoltage,' V');el('dUpsOutput').textContent=val(u.outputVoltage,' V');el('dUpsFrequency').textContent=val(u.inputFrequency,' Hz');el('dUpsBattery').textContent=val(u.batteryVoltage,' V');el('dUpsTemp').textContent=val(u.temperatureCelsius,' °C');el('dUpsSource').textContent=u.available?(u.source+' · '+(u.name||'UPS')):'Установите и настройте NUT или apcupsd';
el('diskList').innerHTML=disks.length?disks.map(d=>{const ioWarn=d.ioPressureSeconds>=60,badge=d.smartStatus==='Ошибка'?'bad':d.smartStatus==='Исправен'?(ioWarn?'warn':''):'warn',reasons=[...(d.attentionReasons||[]),...(ioWarn?['Длительная I/O-нагрузка: '+fixed(d.ioPressureSeconds)+' сек.']:[])],reason=reasons.join(' · '),tip=reason?' tabindex="0" data-reason="'+safe(reason)+'"':'';return '<article class="disk-item"><div class="disk-top"><div><div class="disk-model">'+safe(d.model)+'</div><div class="disk-meta">'+safe(d.path)+' · SN: '+safe(d.serial)+'</div></div><span class="badge '+badge+'"'+tip+'>'+(ioWarn?'Высокий I/O':safe(d.smartStatus))+'</span></div><div class="disk-stats"><div class="disk-stat"><span>Тип / объём</span><b>'+safe(d.type)+' · '+size(d.sizeBytes)+'</b></div><div class="disk-stat"><span>Занято</span><b>'+(d.usedBytes==null?'—':size(d.usedBytes)+' · '+fixed(d.usagePercent)+'%')+'</b></div><div class="disk-stat"><span>Температура / износ</span><b>'+(d.temperatureCelsius==null?'—':fixed(d.temperatureCelsius)+' °C')+' · '+(d.wearUsedPercent==null?'—':fixed(d.wearUsedPercent)+'%')+'</b></div><div class="disk-stat"><span>IOPS · чтение / запись</span><b>'+fixed(d.readIops)+' / '+fixed(d.writeIops)+'</b></div><div class="disk-stat"><span>Скорость · чтение / запись</span><b>'+rate(d.readBytesPerSec)+' / '+rate(d.writeBytesPerSec)+'</b></div><div class="disk-stat"><span>Latency · чтение / запись</span><b>'+fixed(d.readLatencyMs)+' / '+fixed(d.writeLatencyMs)+' ms</b></div><div class="disk-stat"><span>Utilization / очередь</span><b>'+fixed(d.utilizationPercent)+'% / '+fixed(d.averageQueue)+' ('+d.queueDepth+' сейчас)</b></div><div class="disk-stat"><span>Reallocated / Pending</span><b>'+d.reallocated+' / '+d.pending+'</b></div><div class="disk-stat"><span>Uncorrectable / CRC</span><b>'+d.uncorrectable+' / '+d.crcErrors+'</b></div><div class="disk-stat"><span>Последний self-test</span><b>'+safe(d.lastSelfTest||'—')+(d.lastSelfTestAt?' · '+new Date(d.lastSelfTestAt).toLocaleDateString('ru-RU'):'')+(d.shortTestStarted?' · запущен новый':'')+'</b></div></div></article>'}).join(''):'<span class="empty">Физические накопители не найдены</span>';
const totalRate=n.receivePerSec+n.transmitPerSec,nh=n.health||{};el('netInterfaces').textContent=n.interface||'Сетевые интерфейсы';el('netTotal').textContent=(totalRate/1048576).toFixed(2);el('netBar').style.width=Math.min(totalRate/125000000*100,100)+'%';el('netDown').textContent=rate(n.receivePerSec);el('netUp').textContent=rate(n.transmitPerSec);el('dNetDown').textContent=rate(n.receivePerSec);el('dNetUp').textContent=rate(n.transmitPerSec);el('dNetTotal').textContent=rate(totalRate);el('dNetReceived').textContent=size(n.receiveBytes);el('dNetSent').textContent=size(n.transmitBytes);el('dNetInterfaces').textContent=n.interface||'—';el('dNetGateway').textContent=(nh.gateway||'Не определён')+' · '+(nh.gatewayUp?'доступен':'нет ответа')+(nh.latencyMs?' · '+fixed(nh.latencyMs)+' ms':'');el('dNetDNS').textContent=nh.dnsUp?'Работает':'Ошибка';el('dNetInternet').textContent=nh.internetUp?'Доступен':'Недоступен';el('dNetExternalIP').textContent=(nh.externalIp||'—')+(nh.externalIpChanged?' · изменился':'');
const pools=z.pools||[],zTotal=pools.reduce((a,p)=>a+p.sizeBytes,0),zUsed=pools.reduce((a,p)=>a+p.allocatedBytes,0),zPercent=zTotal?zUsed/zTotal*100:0,zBad=pools.filter(p=>p.health!=='ONLINE').length;el('zfsCount').textContent=pools.length;el('zfsHealth').textContent=!z.available?'Не найден':zBad?zBad+' требуют внимания':pools.length+' ONLINE';el('zfsUsed').textContent=zTotal?size(zUsed)+' / '+size(zTotal)+' · '+fixed(zPercent)+'%':'—';el('zfsBar').style.width=Math.min(zPercent,100)+'%';
const datasets=z.datasets||[],arc=z.arc||{};el('zfsArcSize').textContent=arc.available?size(arc.sizeBytes):'—';el('zfsArcTarget').textContent=arc.available?size(arc.targetBytes):'—';el('zfsArcHit').textContent=arc.available?fixed(arc.hitRatio)+'%':'—';el('zfsDatasetCount').textContent=datasets.length;el('zfsList').innerHTML=pools.length?pools.map(p=>{const bad=p.health!=='ONLINE'||p.readErrors+p.writeErrors+p.checksumErrors>0,reason=(p.attentionReasons||[]).join(' · '),tip=reason?' tabindex="0" data-reason="'+safe(reason)+'"':'',poolDatasets=datasets.filter(d=>d.pool===p.name),vdevs=p.vdevs||[];return '<article class="disk-item"><div class="disk-top"><div><div class="disk-model">'+safe(p.name)+'</div><div class="disk-meta">Scrub: '+safe(p.scrubStatus||p.scan||'нет данных')+(p.lastScrubAt?' · '+new Date(p.lastScrubAt).toLocaleString('ru-RU'):'')+(p.scrubDuration?' · '+safe(p.scrubDuration):'')+'</div></div><span class="badge '+(bad?'bad':p.scrubTooOld?'warn':'')+'"'+tip+'>'+safe(p.health)+'</span></div><div class="disk-stats"><div class="disk-stat"><span>Объём / свободно</span><b>'+size(p.sizeBytes)+' / '+size(p.freeBytes)+'</b></div><div class="disk-stat"><span>Заполнение</span><b>'+fixed(p.capacityPercent)+'%</b></div><div class="disk-stat"><span>Read / write / checksum</span><b>'+p.readErrors+' / '+p.writeErrors+' / '+p.checksumErrors+'</b></div></div><div class="topology-section"><div class="topology-section-title">Vdev · data / special / cache / log</div><div class="topology-items">'+(vdevs.length?vdevs.map(v=>'<span class="topology-chip '+(v.state==='ONLINE'?'good':'bad')+'">'+safe(v.class)+' · '+safe(v.name)+' · '+safe(v.state)+' · '+v.read+'/'+v.write+'/'+v.checksum+'</span>').join(''):'<span class="empty">Нет данных</span>')+'</div></div><div class="topology-section"><div class="topology-section-title">Datasets</div><div class="topology-items">'+(poolDatasets.length?poolDatasets.map(d=>'<span class="topology-chip '+(d.capacityPercent>=80?'bad':'good')+'">'+safe(d.name)+' · '+fixed(d.capacityPercent)+'% · '+size(d.usedBytes)+'</span>').join(''):'<span class="empty">Нет datasets</span>')+'</div></div></article>'}).join(''):'<span class="empty">'+(z.available?'ZFS-пулы не найдены':'Команда zpool недоступна')+'</span>';
const val=(v,suffix='')=>v==null?'—':fixed(v)+suffix;el('upsName').textContent=u.available?(u.model||u.name||'Источник бесперебойного питания'):'UPS не настроен';el('upsCharge').textContent=u.chargePercent==null?'—':fixed(u.chargePercent);el('upsBar').style.width=u.chargePercent==null?'0%':Math.min(u.chargePercent,100)+'%';el('upsStatus').textContent=u.available?u.statusLabel:'Не найден';el('upsRuntime').textContent=u.runtimeSeconds==null?'—':uptime(u.runtimeSeconds);el('dUpsName').textContent=u.model||u.name||'Источник бесперебойного питания';el('dUpsStatus').textContent=u.available?u.statusLabel:'Не настроен';el('dUpsCharge').textContent=val(u.chargePercent,'%');el('dUpsRuntime').textContent=u.runtimeSeconds==null?'—':uptime(u.runtimeSeconds);el('dUpsLoad').textContent=val(u.loadPercent,'%');el('dUpsInput').textContent=val(u.inputVoltage,' V');el('dUpsOutput').textContent=val(u.outputVoltage,' V');el('dUpsFrequency').textContent=val(u.inputFrequency,' Hz');el('dUpsBattery').textContent=val(u.batteryVoltage,' V');el('dUpsTemp').textContent=val(u.temperatureCelsius,' °C');el('dUpsCommunication').textContent=u.communicationOk?'Связь установлена':'Нет связи';el('dUpsBatteryDate').textContent=u.batteryDate||'Не сообщает';el('dUpsBatteryAge').textContent=u.batteryAgeDays?u.batteryAgeDays+' дней':'—';el('dUpsSource').textContent=u.available?(u.source+' · '+(u.name||'UPS')):'Установите и настройте NUT или apcupsd';
const mh=x.monitor||{},last=mh.lastSuccessfulCollection?new Date(mh.lastSuccessfulCollection):null,stale=!last||Date.now()-last.getTime()>30000,monitorProblem=stale||mh.lastCollectionError||mh.historyWriteError||mh.slowCollector;el('monitorState').textContent=monitorProblem?'Внимание':'OK';el('monitorBar').style.width=monitorProblem?'100%':'12%';el('monitorLast').textContent=last?Math.max(0,Math.round((Date.now()-last.getTime())/1000))+' сек. назад':'—';el('monitorDB').textContent=size(mh.databaseFreeBytes);el('dMonitorLast').textContent=last?last.toLocaleString('ru-RU'):'Ещё не завершён';el('dMonitorDuration').textContent=(mh.collectionDurationMs||0)+' ms';el('dMonitorHistory').textContent=mh.lastHistoryWrite?new Date(mh.lastHistoryWrite).toLocaleString('ru-RU'):'Ещё не записывалась';el('dMonitorDB').textContent=size(mh.databaseFreeBytes);el('dMonitorWatchdog').textContent=mh.watchdogEnabled?'Включён · 30 сек.':'Не активен';el('dMonitorError').textContent=mh.lastCollectionError||mh.historyWriteError||'Нет';
const guests=(g.guests||[]).filter(v=>!v.template),running=guests.filter(v=>v.status==='running'),guestCores=running.reduce((a,v)=>a+v.maxcpu,0),guestCPU=guestCores?running.reduce((a,v)=>a+v.cpu*v.maxcpu,0)/guestCores*100:0,guestMem=running.reduce((a,v)=>a+v.mem,0),guestMaxMem=running.reduce((a,v)=>a+v.maxmem,0);el('guestRunning').textContent=running.length;el('guestTotal').textContent=' / '+guests.length;el('guestBar').style.width=guests.length?(running.length/guests.length*100)+'%':'0%';el('guestCPU').textContent=g.available?fixed(guestCPU)+'%':'Не найдено';el('guestMemory').textContent=g.available?(size(guestMem)+' / '+size(guestMaxMem)):'—';
el('guestList').innerHTML=guests.length?guests.map(v=>{const live=v.status==='running',type=v.type==='qemu'?'VM':'LXC',memPercent=v.maxmem?v.mem/v.maxmem*100:0;return '<article class="disk-item"><div class="disk-top"><div><div class="disk-model">'+safe(v.name||('Guest '+v.vmid))+'</div><div class="disk-meta">VMID '+v.vmid+' · '+type+' · '+safe(v.node)+'</div></div><span class="badge '+(live?'':'warn')+'">'+(live?'Запущен':'Остановлен')+'</span></div><div class="disk-stats"><div class="disk-stat"><span>Тип</span><b>'+type+'</b></div><div class="disk-stat"><span>CPU</span><b>'+(live?fixed(v.cpu*100)+'%':'—')+'</b></div><div class="disk-stat"><span>Ядра</span><b>'+v.maxcpu+'</b></div><div class="disk-stat"><span>Память</span><b>'+(live?size(v.mem)+' · '+fixed(memPercent)+'%':size(v.maxmem))+'</b></div><div class="disk-stat"><span>Лимит RAM</span><b>'+size(v.maxmem)+'</b></div><div class="disk-stat"><span>Uptime</span><b>'+(live?uptime(v.uptime):'—')+'</b></div></div></article>'}).join(''):'<span class="empty">'+(g.available?'VM и LXC не найдены':'Команда pvesh недоступна')+'</span>';
el('guestList').innerHTML=guests.length?guests.map(v=>{const live=v.status==='running',type=v.type==='qemu'?'VM':'LXC',memPercent=v.maxmem?v.mem/v.maxmem*100:0,diskPercent=v.maxdisk?v.disk/v.maxdisk*100:0,ips=(v.ipAddresses||[]).join(', ')||'IP не найден';return '<article class="disk-item"><div class="disk-top"><div><div class="disk-model">'+safe(v.name||('Guest '+v.vmid))+'</div><div class="disk-meta">VMID '+v.vmid+' · '+type+' · '+safe(v.node)+' · '+safe(ips)+'</div></div><span class="badge '+(live?'':'warn')+'">'+(live?'Запущен':'Остановлен')+'</span></div><div class="disk-stats"><div class="disk-stat"><span>CPU</span><b>'+(live?fixed(v.cpu*100)+'%':'—')+' / '+v.maxcpu+'</b></div><div class="disk-stat"><span>Память</span><b>'+(live?size(v.mem)+' · '+fixed(memPercent)+'%':size(v.maxmem))+'</b></div><div class="disk-stat"><span>Диск</span><b>'+(v.maxdisk?size(v.disk)+' · '+fixed(diskPercent)+'%':'—')+'</b></div><div class="disk-stat"><span>Сеть</span><b>'+size(v.netin)+' · ↑ '+size(v.netout)+'</b></div><div class="disk-stat"><span>Guest Agent</span><b>'+(v.agent?'Доступен':'Нет')+'</b></div><div class="disk-stat"><span>Автозапуск</span><b>'+(v.onboot?'Да':'Нет')+'</b></div><div class="disk-stat"><span>Storage</span><b>'+safe((v.storage||[]).join(', ')||'—')+'</b></div><div class="disk-stat"><span>Uptime</span><b>'+(live?uptime(v.uptime):'—')+'</b></div></div></article>'}).join(''):'<span class="empty">'+(g.available?'VM и LXC не найдены':'Команда pvesh недоступна')+'</span>';
const pveShort=(mt.proxmoxVersion||'Proxmox VE').split('(')[0].trim();el('maintenanceVersion').textContent=pveShort;el('maintenanceUpdates').textContent=mt.availableUpdates??'—';el('maintenanceBar').style.width=Math.min((mt.availableUpdates||0)*5,100)+'%';el('maintenanceReboot').textContent=(mt.rebootRequired||mt.newKernelAvailable)?'Требуется':'Не требуется';el('maintenanceKernel').textContent=mt.kernelVersion||'—';el('dPveVersion').textContent=pveShort;el('dPveVersion').title=mt.proxmoxVersion||'';el('dKernel').textContent=mt.kernelVersion||'—';el('dLatestKernel').textContent=mt.latestInstalledKernel||'—';el('dUpdates').textContent=(mt.availableUpdates??'—')+' пакетов';el('dReboot').textContent=(mt.rebootRequired||mt.newKernelAvailable)?'Требуется':'Не требуется';el('dLastUpdate').textContent=mt.lastSystemUpdate?new Date(mt.lastSystemUpdate).toLocaleString('ru-RU'):'—';el('dNextBackup').textContent=mt.nextBackup||'Расписание не найдено';el('dNextScrub').textContent=mt.nextScrub||'Таймер не найден';el('dMaintenanceChecked').textContent=mt.checkedAt?new Date(mt.checkedAt).toLocaleString('ru-RU'):'—';
const backupLimitHours=alertSettings.backupMaxAgeHours||48,backupLimit=backupLimitHours*3600,backupMap=new Map((b.items||[]).map(item=>[item.vmid,item])),now=Date.now()/1000,protectedGuests=guests.filter(v=>backupMap.has(v.vmid)),freshGuests=protectedGuests.filter(v=>now-backupMap.get(v.vmid).ctime<=backupLimit),missing=guests.length-protectedGuests.length;el('backupFreshLabel').textContent='Свежие · '+backupLimitHours+' часов';el('backupProtected').textContent=protectedGuests.length;el('backupTotal').textContent=' / '+guests.length;el('backupBar').style.width=guests.length?(protectedGuests.length/guests.length*100)+'%':'0%';el('backupFresh').textContent=b.available?freshGuests.length:'Не найдено';el('backupMissing').textContent=b.available?missing:'—';
el('backupList').innerHTML=guests.length?guests.map(v=>{const item=backupMap.get(v.vmid),type=v.type==='qemu'?'VM':'LXC';if(!item)return '<article class="disk-item"><div class="disk-top"><div><div class="disk-model">'+safe(v.name||('Guest '+v.vmid))+'</div><div class="disk-meta">VMID '+v.vmid+' · '+type+'</div></div><span class="badge bad">Копии нет</span></div></article>';const age=now-item.ctime,fresh=age<=backupLimit,date=new Date(item.ctime*1000).toLocaleString('ru-RU');return '<article class="disk-item"><div class="disk-top"><div><div class="disk-model">'+safe(v.name||('Guest '+v.vmid))+'</div><div class="disk-meta">VMID '+v.vmid+' · '+type+' · '+date+'</div></div><span class="badge '+(fresh?'':'warn')+'">'+(fresh?'Актуальна':'Старше '+backupLimitHours+' ч')+'</span></div><div class="disk-stats"><div class="disk-stat"><span>Хранилище</span><b>'+safe(item.storage)+'</b></div><div class="disk-stat"><span>Размер</span><b>'+size(item.size)+'</b></div><div class="disk-stat"><span>Формат</span><b>'+safe(item.format)+'</b></div><div class="disk-stat"><span>Возраст</span><b>'+uptime(age)+'</b></div><div class="disk-stat"><span>Создана</span><b>'+date+'</b></div><div class="disk-stat"><span>VMID</span><b>'+v.vmid+'</b></div></div></article>'}).join(''):'<span class="empty">'+(b.available?'VM и LXC не найдены':'Данные резервных копий недоступны')+'</span>';
el('backupList').innerHTML=guests.length?guests.map(v=>{const item=backupMap.get(v.vmid),type=v.type==='qemu'?'VM':'LXC';if(!item)return '<article class="disk-item"><div class="disk-top"><div><div class="disk-model">'+safe(v.name||('Guest '+v.vmid))+'</div><div class="disk-meta">VMID '+v.vmid+' · '+type+'</div></div><span class="badge bad">Копии нет</span></div></article>';const age=now-item.ctime,fresh=age<=backupLimit,date=new Date(item.ctime*1000).toLocaleString('ru-RU'),task=(x.activity?.tasks||[]).find(t=>t.type==='vzdump'&&String(t.id)===String(v.vmid)),change=item.sizeChangePercent==null?'Нет предыдущей':(item.sizeChangePercent>=0?'+':'')+fixed(item.sizeChangePercent)+'%';return '<article class="disk-item"><div class="disk-top"><div><div class="disk-model">'+safe(v.name||('Guest '+v.vmid))+'</div><div class="disk-meta">VMID '+v.vmid+' · '+type+' · '+date+'</div></div><span class="badge '+(fresh?'':'warn')+'">'+(fresh?'Актуальна':'Старше '+backupLimitHours+' ч')+'</span></div><div class="disk-stats"><div class="disk-stat"><span>Хранилище</span><b>'+safe(item.storage)+'</b></div><div class="disk-stat"><span>Размер</span><b>'+size(item.size)+'</b></div><div class="disk-stat"><span>К предыдущей</span><b>'+change+'</b></div><div class="disk-stat"><span>Последнее задание</span><b>'+(task?safe(task.status)+' · '+uptime(task.duration):'Общее задание')+'</b></div><div class="disk-stat"><span>Возраст</span><b>'+uptime(age)+'</b></div><div class="disk-stat"><span>Формат</span><b>'+safe(item.format)+'</b></div></div><div class="service-actions"><button data-verify-backup="'+safe(item.volid)+'">Проверить читаемость</button></div><div class="backup-check" data-backup-result="'+safe(item.volid)+'"></div></article>'}).join(''):'<span class="empty">'+(b.available?'VM и LXC не найдены':'Данные резервных копий недоступны')+'</span>';
const svc=x.services||{services:[],up:0,total:0};renderServices(svc);
const sh=x.systemHealth||{},issueCount=sh.systemErrors||0;el('systemIssues').textContent=issueCount;el('systemBar').style.width=Math.min(issueCount*10,100)+'%';el('failedUnits').textContent=sh.failedUnits??'—';el('kernelErrors').textContent=sh.kernelErrors??'—';el('dSystemErrors').textContent=issueCount;el('dKernelErrors').textContent=sh.kernelErrors||0;el('dFailedUnits').textContent=sh.failedUnits||0;renderSystemHealth(sh);
renderNetworkGroups(n.interfaces||[]);
renderActivity(x.activity||{},g,svc,z);renderIOAttribution(disks);explainWarningBadges();
el('dot').classList.add('live');el('status').textContent='Данные поступают';
}
let historyData=null,historyLoadedAt=0;
async function loadHistory(){if(historyData&&Date.now()-historyLoadedAt<60000)return historyData;const response=await fetch('/api/history?hours=168');if(!response.ok)throw new Error('history');historyData=await response.json()||[];historyLoadedAt=Date.now();return historyData}
function drawChart(id,series,fixedMax=100){const canvas=el(id);if(!canvas||!canvas.offsetWidth)return;const points=historyData||[],dpr=window.devicePixelRatio||1,w=canvas.offsetWidth,h=180;canvas.width=w*dpr;canvas.height=h*dpr;const ctx=canvas.getContext('2d');ctx.scale(dpr,dpr);const styles=getComputedStyle(document.documentElement),text=styles.getPropertyValue('--muted').trim(),line=styles.getPropertyValue('--line').trim();ctx.clearRect(0,0,w,h);ctx.font='11px system-ui';ctx.fillStyle=text;if(points.length<2){ctx.textAlign='center';ctx.fillText('История появится после нескольких минут работы',w/2,h/2);return}const values=series.flatMap(s=>points.map(p=>s.value(p)).filter(v=>v!=null&&Number.isFinite(v))),max=fixedMax||Math.max(...values,1)*1.1,min=0,pad={l:38,r:12,t:30,b:22},pw=w-pad.l-pad.r,ph=h-pad.t-pad.b;ctx.strokeStyle=line;ctx.lineWidth=1;for(let i=0;i<=4;i++){const y=pad.t+ph*i/4;ctx.beginPath();ctx.moveTo(pad.l,y);ctx.lineTo(w-pad.r,y);ctx.stroke();ctx.fillStyle=text;ctx.textAlign='right';ctx.fillText((max*(1-i/4)).toFixed(max>100?0:0),pad.l-7,y+4)}series.forEach((s,index)=>{ctx.strokeStyle=s.color;ctx.lineWidth=2;ctx.beginPath();let started=false;points.forEach((p,i)=>{const v=s.value(p);if(v==null||!Number.isFinite(v))return;const x=pad.l+pw*i/(points.length-1),y=pad.t+ph*(1-(v-min)/(max-min||1));if(!started){ctx.moveTo(x,y);started=true}else ctx.lineTo(x,y)});ctx.stroke();ctx.fillStyle=s.color;ctx.textAlign='left';ctx.fillText(s.label,pad.l+index*125,15)});ctx.fillStyle=text;ctx.textAlign='left';ctx.fillText('7 дней назад',pad.l,h-5);ctx.textAlign='right';ctx.fillText('сейчас',w-pad.r,h-5)}
function drawAllCharts(){drawChart('cpuChart',[{label:'CPU %',color:'#55d58a',value:p=>p.cpuUsage},{label:'Температура °C',color:'#ff9b5c',value:p=>p.cpuTemperature}]);drawChart('memoryChart',[{label:'RAM %',color:'#9d8cff',value:p=>p.memoryUsage},{label:'Swap %',color:'#ff78b8',value:p=>p.swapUsage}]);drawChart('storageChart',[{label:'Раздел / %',color:'#ff9b5c',value:p=>p.rootUsage}]);drawChart('networkChart',[{label:'Приём MB/s',color:'#43c6d9',value:p=>p.networkReceive/1048576},{label:'Передача MB/s',color:'#65ddaa',value:p=>p.networkTransmit/1048576}],0);drawChart('zfsChart',[{label:'ZFS %',color:'#6577ff',value:p=>p.zfsUsage}]);drawChart('upsChart',[{label:'Заряд %',color:'#55d58a',value:p=>p.upsCharge},{label:'Нагрузка %',color:'#d5e85a',value:p=>p.upsLoad}]);drawChart('guestsChart',[{label:'CPU %',color:'#ff78b8',value:p=>p.guestsCpu},{label:'RAM %',color:'#9d8cff',value:p=>p.guestsMemory}])}
document.querySelectorAll('[data-open]').forEach(b=>b.onclick=()=>{const dialog=el(b.dataset.open);dialog.showModal();if(dialog.querySelector('canvas'))setTimeout(()=>loadHistory().then(drawAllCharts).catch(()=>{}),50)});
const percent=v=>fixed(v)+'%',temperature=v=>fixed(v)+' °C',megabytes=v=>fixed(v)+' MB/s';
const chartConfigs={cpuChart:{series:[{label:'CPU',color:'#55d58a',value:p=>p.cpuUsage,format:percent},{label:'Температура',color:'#ff9b5c',value:p=>p.cpuTemperature,format:temperature}]},memoryChart:{series:[{label:'RAM',color:'#9d8cff',value:p=>p.memoryUsage,format:percent},{label:'Swap',color:'#ff78b8',value:p=>p.swapUsage,format:percent}]},storageChart:{series:[{label:'Раздел /',color:'#ff9b5c',value:p=>p.rootUsage,format:percent}]},networkChart:{max:0,series:[{label:'Приём',color:'#43c6d9',value:p=>p.networkReceive/1048576,format:megabytes},{label:'Передача',color:'#65ddaa',value:p=>p.networkTransmit/1048576,format:megabytes}]},zfsChart:{series:[{label:'ZFS',color:'#6577ff',value:p=>p.zfsUsage,format:percent}]},upsChart:{series:[{label:'Заряд',color:'#55d58a',value:p=>p.upsCharge,format:percent},{label:'Нагрузка',color:'#d5e85a',value:p=>p.upsLoad,format:percent}]},guestsChart:{series:[{label:'CPU',color:'#ff78b8',value:p=>p.guestsCpu,format:percent},{label:'RAM',color:'#9d8cff',value:p=>p.guestsMemory,format:percent}]}};
const historyCache=new Map();
async function loadHistory(hours){const cached=historyCache.get(hours);if(cached&&Date.now()-cached.loadedAt<60000)return cached.points;const response=await fetch('/api/history?hours='+hours);if(!response.ok)throw new Error('history');const points=await response.json()||[];historyCache.set(hours,{points,loadedAt:Date.now()});return points}
const entityChartKinds={storageChart:{kind:'disk',format:temperature},diskIOChart:{kind:'disk-io',format:v=>fixed(v)+' IOPS',dynamic:true},diskUtilChart:{kind:'disk-io',format:percent,value:'value2'},zfsChart:{kind:'zfs',format:percent},guestsChart:{kind:'guest',format:percent}},chartColors=['#55d58a','#6ba8ff','#ff9b5c','#9d8cff','#ff78b8','#43c6d9','#d5e85a','#ff665c','#6577ff','#65ddaa'];
async function loadEntityChart(id,hours){const definition=entityChartKinds[id],response=await fetch('/api/entity-history?kind='+definition.kind+'&hours='+hours);if(!response.ok)throw new Error('entity history');const data=await response.json()||[],timestamps=new Map(),series=data.map((item,index)=>{const key='entity'+index;item.points.forEach(p=>{const point=timestamps.get(p.timestamp)||{timestamp:p.timestamp};point[key]=p[definition.value||'value1'];timestamps.set(p.timestamp,point)});return {label:item.name,color:chartColors[index%chartColors.length],value:p=>p[key],format:definition.format}});chartConfigs[id]={series,max:definition.dynamic?0:undefined};return [...timestamps.values()].sort((a,b)=>a.timestamp-b.timestamp)}
function drawChart(id,points){const canvas=el(id),config=chartConfigs[id];if(!canvas||!config||!canvas.offsetWidth)return;const series=config.series,dpr=window.devicePixelRatio||1,w=canvas.offsetWidth,h=200;canvas.width=w*dpr;canvas.height=h*dpr;const ctx=canvas.getContext('2d');ctx.scale(dpr,dpr);const styles=getComputedStyle(document.documentElement),text=styles.getPropertyValue('--muted').trim(),line=styles.getPropertyValue('--line').trim();ctx.clearRect(0,0,w,h);ctx.font='11px system-ui';ctx.fillStyle=text;if(points.length<2){ctx.textAlign='center';ctx.fillText('История появится после нескольких минут работы',w/2,h/2);canvas._chartState=null;return}const values=series.flatMap(s=>points.map(p=>s.value(p)).filter(v=>v!=null&&Number.isFinite(v))),max=config.max===0?Math.max(...values,1)*1.1:100,pad={l:38,r:12,t:30,b:25},pw=w-pad.l-pad.r,ph=h-pad.t-pad.b;ctx.strokeStyle=line;ctx.lineWidth=1;for(let i=0;i<=4;i++){const y=pad.t+ph*i/4;ctx.beginPath();ctx.moveTo(pad.l,y);ctx.lineTo(w-pad.r,y);ctx.stroke();ctx.fillStyle=text;ctx.textAlign='right';ctx.fillText((max*(1-i/4)).toFixed(max<10?1:0),pad.l-7,y+4)}series.forEach((s,index)=>{ctx.strokeStyle=s.color;ctx.lineWidth=2;ctx.beginPath();let started=false;points.forEach((p,i)=>{const v=s.value(p);if(v==null||!Number.isFinite(v))return;const x=pad.l+pw*i/(points.length-1),y=pad.t+ph*(1-v/(max||1));if(!started){ctx.moveTo(x,y);started=true}else ctx.lineTo(x,y)});ctx.stroke();ctx.fillStyle=s.color;ctx.textAlign='left';ctx.fillText(s.label,pad.l+index*130,15)});const first=new Date(points[0].timestamp*1000),last=new Date(points[points.length-1].timestamp*1000),timeOptions={hour:'2-digit',minute:'2-digit'};ctx.fillStyle=text;ctx.textAlign='left';ctx.fillText(first.toLocaleString('ru-RU',timeOptions),pad.l,h-6);ctx.textAlign='right';ctx.fillText(last.toLocaleString('ru-RU',timeOptions),w-pad.r,h-6);canvas._chartState={points,series,pad,pw,w}}
function setupChartHover(canvas){canvas.addEventListener('mousemove',event=>{const state=canvas._chartState,section=canvas.closest('.chart-section'),tooltip=section&&section.querySelector('.chart-tooltip');if(!state||!tooltip)return;const rect=canvas.getBoundingClientRect(),x=event.clientX-rect.left,index=Math.max(0,Math.min(state.points.length-1,Math.round((x-state.pad.l)/state.pw*(state.points.length-1)))),point=state.points[index],date=new Date(point.timestamp*1000);tooltip.innerHTML='<div class="chart-tooltip-time">'+date.toLocaleString('ru-RU')+'</div>'+state.series.map(s=>{const value=s.value(point),shown=value==null||!Number.isFinite(value)?'—':s.format(value);return '<div class="chart-tooltip-value"><span><i style="background:'+s.color+'"></i>'+s.label+'</span><strong>'+shown+'</strong></div>'}).join('');tooltip.classList.add('visible');const sectionRect=section.getBoundingClientRect(),left=Math.min(event.clientX-sectionRect.left+12,section.clientWidth-tooltip.offsetWidth-10);tooltip.style.left=Math.max(10,left)+'px';tooltip.style.top=Math.max(55,event.clientY-sectionRect.top-tooltip.offsetHeight-12)+'px'});canvas.addEventListener('mouseleave',()=>{const tooltip=canvas.closest('.chart-section').querySelector('.chart-tooltip');tooltip.classList.remove('visible')})}
async function showHistory(section){const hours=Number(section.dataset.hours||1);await Promise.all([...section.querySelectorAll('canvas')].map(async canvas=>{try{drawChart(canvas.id,entityChartKinds[canvas.id]?await loadEntityChart(canvas.id,hours):await loadHistory(hours))}catch(e){drawChart(canvas.id,[])}}))}
function setupDetailTabs(){document.querySelectorAll('dialog').forEach(dialog=>{const chart=dialog.querySelector('.chart-section');if(!chart)return;const head=dialog.querySelector('.modal-head'),tabs=document.createElement('div'),realtime=document.createElement('div');tabs.className='detail-tabs';tabs.innerHTML='<button class="detail-tab active" data-detail-tab="realtime">Реальное время</button><button class="detail-tab" data-detail-tab="history">История</button>';realtime.className='detail-panel';realtime.dataset.detailPanel='realtime';while(head.nextElementSibling&&head.nextElementSibling!==chart)realtime.appendChild(head.nextElementSibling);head.after(tabs);tabs.after(realtime);chart.classList.add('detail-panel');chart.dataset.detailPanel='history';chart.dataset.hours='1';chart.hidden=true;const title=chart.querySelector('.chart-head strong'),chartHead=chart.querySelector('.chart-head');chartHead.innerHTML='';chartHead.appendChild(title);const ranges=document.createElement('div');ranges.className='range-switch';ranges.innerHTML='<button class="range-button active" data-hours="1">1 час</button><button class="range-button" data-hours="24">24 часа</button><button class="range-button" data-hours="168">7 дней</button>';chartHead.appendChild(ranges);const tooltip=document.createElement('div');tooltip.className='chart-tooltip';chart.appendChild(tooltip);chart.querySelectorAll('canvas').forEach(setupChartHover);tabs.querySelectorAll('[data-detail-tab]').forEach(tab=>tab.onclick=()=>{tabs.querySelectorAll('.detail-tab').forEach(t=>t.classList.toggle('active',t===tab));realtime.hidden=tab.dataset.detailTab!=='realtime';chart.hidden=tab.dataset.detailTab!=='history';if(!chart.hidden)setTimeout(()=>showHistory(chart),20)});ranges.querySelectorAll('[data-hours]').forEach(button=>button.onclick=()=>{chart.dataset.hours=button.dataset.hours;ranges.querySelectorAll('.range-button').forEach(b=>b.classList.toggle('active',b===button));showHistory(chart)})})}
setupDetailTabs();
document.querySelectorAll('[data-open]').forEach(b=>b.onclick=()=>el(b.dataset.open).showModal());
document.querySelectorAll('dialog').forEach(d=>{d.querySelector('.close').onclick=()=>d.close();d.onclick=e=>{if(e.target===d)d.close()}});
const cardNames=['alerts','cpu','memory','storage','network','zfs','ups','guests','backups'];let hiddenCards=[];try{hiddenCards=JSON.parse(localStorage.getItem('proxmox-hidden-cards')||'[]')}catch(e){}
function applyCardSettings(){cardNames.forEach(name=>{const visible=!hiddenCards.includes(name),card=document.querySelector('[data-card="'+name+'"]'),toggle=document.querySelector('[data-card-toggle="'+name+'"]');if(card)card.hidden=!visible;if(toggle)toggle.checked=visible})}
document.querySelectorAll('[data-card-toggle]').forEach(toggle=>toggle.onchange=()=>{const name=toggle.dataset.cardToggle;hiddenCards=toggle.checked?hiddenCards.filter(v=>v!==name):[...new Set([...hiddenCards,name])];localStorage.setItem('proxmox-hidden-cards',JSON.stringify(hiddenCards));applyCardSettings()});applyCardSettings();
function setTheme(theme){document.documentElement.dataset.theme=theme;localStorage.setItem('proxmox-theme',theme);document.querySelectorAll('[data-theme-choice]').forEach(b=>b.classList.toggle('active',b.dataset.themeChoice===theme));drawAllCharts()}document.querySelectorAll('[data-theme-choice]').forEach(b=>b.onclick=()=>setTheme(b.dataset.themeChoice));setTheme(document.documentElement.dataset.theme);el('settingsButton').onclick=()=>el('settingsDialog').showModal();
document.querySelectorAll('[data-system-tab]').forEach(tab=>tab.onclick=()=>{document.querySelectorAll('[data-system-tab]').forEach(v=>v.classList.toggle('active',v===tab));document.querySelectorAll('[data-system-panel]').forEach(v=>v.hidden=v.dataset.systemPanel!==tab.dataset.systemTab)});
el('systemSort').onclick=()=>{systemSortNewest=!systemSortNewest;renderSystemHealth(currentSystemHealth)};
el('serviceList').onclick=event=>{const button=event.target.closest('[data-service-open]');if(!button)return;const service=currentServiceStatuses.find(v=>v.id===Number(button.dataset.serviceOpen));if(!service)return;const choices=(service.endpoints||[]).filter(v=>/^https?:\/\//.test(v.address));if(choices.length===1){window.open(choices[0].address,'_blank','noopener');return}if(!choices.length)return;el('openServiceTitle').textContent=service.name;el('openServiceChoices').innerHTML=choices.map(v=>'<a class="theme-option" href="'+safe(v.address)+'" target="_blank" rel="noopener">'+safe(v.label)+'<small style="display:block;color:var(--muted);margin-top:4px">'+safe(v.address)+'</small></a>').join('');el('openServiceDialog').showModal()};
document.querySelectorAll('[data-task-filter]').forEach(button=>button.onclick=()=>{currentTaskFilter=button.dataset.taskFilter;document.querySelectorAll('[data-task-filter]').forEach(v=>v.classList.toggle('active',v===button));renderTasks(currentTasks,currentTaskFilter);explainWarningBadges()});
el('backupList').onclick=async event=>{const button=event.target.closest('[data-verify-backup]');if(!button)return;const target=el('backupList').querySelector('[data-backup-result="'+CSS.escape(button.dataset.verifyBackup)+'"]');button.disabled=true;target.textContent='Проверяем метаданные и читаемость…';try{const response=await fetch('/api/backups/verify',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({volid:button.dataset.verifyBackup})}),result=await response.json();target.textContent=(result.ok?'✓ ':'⚠ ')+result.message;target.className='backup-check '+(result.ok?'status-up':'status-down')}catch(e){target.textContent='Не удалось выполнить проверку.'}finally{button.disabled=false}};
el('alertList').onclick=async event=>{const button=event.target.closest('[data-ack]');if(!button)return;button.disabled=true;const response=await fetch('/api/alerts/acknowledge',{method:'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify({eventId:Number(button.dataset.ack)})});if(response.ok)button.closest('.alert-item').remove();else button.disabled=false};
async function loadAlertHistory(){const response=await fetch('/api/alerts/history');if(!response.ok)return;const events=await response.json()||[];el('alertHistory').innerHTML=events.length?events.map(a=>{const state=a.planned?'Плановые работы · '+safe(a.maintenanceReason||'Причина не указана')+(a.resolvedAt?' · завершено '+new Date(a.resolvedAt*1000).toLocaleString('ru-RU'):''):a.acknowledgedAt?'Принято · '+new Date(a.acknowledgedAt*1000).toLocaleString('ru-RU'):a.resolvedAt?'Завершено · '+new Date(a.resolvedAt*1000).toLocaleString('ru-RU'):'Активно';return '<article class="alert-item '+safe(a.severity)+'"><i class="alert-icon"></i><div><div class="alert-source">'+safe(a.source)+' · '+new Date(a.firstSeen*1000).toLocaleString('ru-RU')+'</div><div class="alert-title">'+safe(a.title)+'</div><div class="alert-message">'+safe(a.message)+'</div><span class="history-state">'+state+'</span></div></article>'}).join(''):'<div class="all-good"><strong>История пока пуста</strong><span>Новые события появятся здесь автоматически.</span></div>'}
document.querySelectorAll('[data-alert-tab]').forEach(tab=>tab.onclick=()=>{document.querySelectorAll('[data-alert-tab]').forEach(t=>t.classList.toggle('active',t===tab));document.querySelectorAll('[data-alert-panel]').forEach(p=>p.hidden=p.dataset.alertPanel!==tab.dataset.alertTab);if(tab.dataset.alertTab==='history')loadAlertHistory()});
const cardNames=['alerts','cpu','memory','storage','network','zfs','ups','monitor','guests','agents','maintenance','backups','services','system','snapshots','tasks','map'],cardLabels={alerts:'Центр уведомлений',cpu:'CPU',memory:'Память',storage:'Диски',network:'Сеть',zfs:'ZFS',ups:'UPS',monitor:'Самоконтроль',guests:'VM и LXC',agents:'Агенты VM и LXC',maintenance:'Proxmox',backups:'Резервные копии',services:'Сервисы',system:'Система и ядро',snapshots:'Snapshots',tasks:'Задачи Proxmox',map:'Карта homelab'};let hiddenCards=[],cardOrder=[];try{hiddenCards=JSON.parse(localStorage.getItem('proxmox-hidden-cards')||'[]');cardOrder=JSON.parse(localStorage.getItem('proxmox-card-order')||'[]')}catch(e){}cardOrder=[...cardOrder.filter(x=>cardNames.includes(x)),...cardNames.filter(x=>!cardOrder.includes(x))];
function renderCardOrder(){el('cardOrder').innerHTML=cardOrder.map(name=>'<div class="order-item" draggable="true" data-order-card="'+name+'"><span class="order-handle">⠿</span><span>'+cardLabels[name]+'</span><button class="order-button" data-move="up" aria-label="Выше">↑</button><button class="order-button" data-move="down" aria-label="Ниже">↓</button></div>').join('')}
function saveCardOrder(){localStorage.setItem('proxmox-card-order',JSON.stringify(cardOrder));applyCardSettings();renderCardOrder()}
function applyCardSettings(){const container=document.querySelector('.cards');cardOrder.forEach(name=>{const card=document.querySelector('[data-card="'+name+'"]');if(card)container.appendChild(card)});cardNames.forEach(name=>{const visible=!hiddenCards.includes(name),card=document.querySelector('[data-card="'+name+'"]'),toggle=document.querySelector('[data-card-toggle="'+name+'"]');if(card)card.hidden=!visible;if(toggle)toggle.checked=visible})}
document.querySelectorAll('[data-card-toggle]').forEach(toggle=>toggle.onchange=()=>{const name=toggle.dataset.cardToggle;hiddenCards=toggle.checked?hiddenCards.filter(v=>v!==name):[...new Set([...hiddenCards,name])];localStorage.setItem('proxmox-hidden-cards',JSON.stringify(hiddenCards));applyCardSettings()});renderCardOrder();applyCardSettings();
el('cardOrder').onclick=event=>{const button=event.target.closest('[data-move]'),item=event.target.closest('[data-order-card]');if(!button||!item)return;const index=cardOrder.indexOf(item.dataset.orderCard),next=button.dataset.move==='up'?index-1:index+1;if(next<0||next>=cardOrder.length)return;[cardOrder[index],cardOrder[next]]=[cardOrder[next],cardOrder[index]];saveCardOrder()};let draggedCard='';el('cardOrder').addEventListener('dragstart',event=>{const item=event.target.closest('[data-order-card]');if(!item)return;draggedCard=item.dataset.orderCard;item.classList.add('dragging')});el('cardOrder').addEventListener('dragend',event=>{event.target.closest('[data-order-card]')?.classList.remove('dragging');draggedCard=''});el('cardOrder').addEventListener('dragover',event=>event.preventDefault());el('cardOrder').addEventListener('drop',event=>{event.preventDefault();const target=event.target.closest('[data-order-card]');if(!target||!draggedCard||target.dataset.orderCard===draggedCard)return;const from=cardOrder.indexOf(draggedCard),to=cardOrder.indexOf(target.dataset.orderCard);cardOrder.splice(from,1);cardOrder.splice(to,0,draggedCard);saveCardOrder()});
function setTheme(theme){document.documentElement.dataset.theme=theme;localStorage.setItem('proxmox-theme',theme);document.querySelectorAll('[data-theme-choice]').forEach(b=>b.classList.toggle('active',b.dataset.themeChoice===theme));document.querySelectorAll('.chart-section:not([hidden])').forEach(showHistory)}document.querySelectorAll('[data-theme-choice]').forEach(b=>b.onclick=()=>setTheme(b.dataset.themeChoice));setTheme(document.documentElement.dataset.theme);el('settingsButton').onclick=()=>el('settingsDialog').showModal();
document.querySelectorAll('[data-settings-tab]').forEach(tab=>tab.onclick=()=>{document.querySelectorAll('[data-settings-tab]').forEach(t=>t.classList.toggle('active',t===tab));document.querySelectorAll('[data-settings-panel]').forEach(p=>p.hidden=p.dataset.settingsPanel!==tab.dataset.settingsTab)});
el('agentServerURL').value=location.origin;
el('createAgentToken').onclick=async()=>{const button=el('createAgentToken'),server=el('agentServerURL').value.trim().replace(/\/$/,''),vmid=Number(el('agentVMID').value)||0;if(!/^https?:\/\//.test(server)){alert('Укажите полный адрес Dashboard, доступный из VM/LXC.');return}button.disabled=true;button.textContent='Создаём токен…';try{const response=await fetch('/api/agents/enrollment-token',{method:'POST'});if(!response.ok)throw new Error(await response.text());const value=await response.json(),quote=v=>"'"+String(v).replaceAll("'","'\"'\"'")+"'";el('agentInstallCommand').textContent='curl -fsSL https://git.myown.center/maxim/ProxmoxDash/raw/branch/main/scripts/install-agent.sh | DASHBOARD_URL='+quote(server)+' ENROLL_TOKEN='+quote(value.token)+' VMID='+quote(vmid)+' sh';el('agentInstallBlock').hidden=false}catch(e){alert('Не удалось создать токен: '+e.message)}finally{button.disabled=false;button.textContent='Создать команду установки'}};
el('agentSettingsList').onclick=async event=>{const button=event.target.closest('[data-agent-delete]');if(!button||!confirm('Удалить агент? Для повторного подключения потребуется новый токен.'))return;button.disabled=true;const response=await fetch('/api/agents?id='+encodeURIComponent(button.dataset.agentDelete),{method:'DELETE'});if(response.ok)button.closest('.disk-item').remove();else{button.disabled=false;alert(await response.text())}};
async function loadThresholds(){const response=await fetch('/api/settings/alerts');if(!response.ok)return;alertSettings=await response.json();document.querySelectorAll('[data-threshold]').forEach(input=>input.value=alertSettings[input.dataset.threshold]??'')}
el('saveThresholds').onclick=async()=>{const values={};document.querySelectorAll('[data-threshold]').forEach(input=>values[input.dataset.threshold]=Number(input.value));const status=el('thresholdStatus');status.textContent='Сохранение…';const response=await fetch('/api/settings/alerts',{method:'PUT',headers:{'Content-Type':'application/json'},body:JSON.stringify(values)});if(response.ok){alertSettings=await response.json();status.textContent='Пороги сохранены и уже применяются.'}else{status.textContent=await response.text()||'Не удалось сохранить настройки.'}};loadThresholds().catch(()=>{});
el('saveThresholds').onclick=async()=>{const values={};document.querySelectorAll('[data-threshold]').forEach(input=>values[input.dataset.threshold]=input.dataset.threshold==='diskIoExcluded'?input.value:Number(input.value));const status=el('thresholdStatus');status.textContent='Сохранение…';const response=await fetch('/api/settings/alerts',{method:'PUT',headers:{'Content-Type':'application/json'},body:JSON.stringify(values)});if(response.ok){alertSettings=await response.json();status.textContent='Пороги сохранены и уже применяются.'}else{status.textContent=await response.text()||'Не удалось сохранить настройки.'}};loadThresholds().catch(()=>{});
let configuredServices=[];
async function loadServices(){const response=await fetch('/api/services');if(!response.ok)return;configuredServices=await response.json()||[];el('serviceSettingsList').innerHTML=configuredServices.length?configuredServices.map(v=>'<article class="disk-item"><div class="disk-top"><div><div class="disk-model">'+safe(v.name)+'</div><div class="disk-meta">'+safe([v.url&&('Внешний: '+v.url),v.localAddress&&('Локальный: '+v.localAddress)].filter(Boolean).join(' · '))+'</div></div><span class="badge '+(v.enabled?'':'warn')+'">'+(v.enabled?'Включён':'Выключен')+'</span></div><div class="service-actions"><button data-service-edit="'+v.id+'">Изменить</button><button data-service-delete="'+v.id+'">Удалить</button></div></article>').join(''):'<span class="empty">Список пуст.</span>'}
function resetServiceForm(){el('serviceId').value='';el('serviceName').value='';el('serviceType').value='http';el('serviceURL').value='';el('serviceLocal').value='';el('serviceCode').value=200;el('serviceText').value='';el('serviceTimeout').value=5;el('serviceEnabled').checked=true}
el('saveService').onclick=async()=>{const value={id:Number(el('serviceId').value)||0,name:el('serviceName').value.trim(),checkType:el('serviceType').value,url:el('serviceURL').value.trim(),localAddress:el('serviceLocal').value.trim(),expectedCode:Number(el('serviceCode').value)||200,expectedText:el('serviceText').value,timeoutSeconds:Number(el('serviceTimeout').value)||5,enabled:el('serviceEnabled').checked},status=el('serviceStatus');status.textContent='Сохранение…';const response=await fetch('/api/services',{method:value.id?'PUT':'POST',headers:{'Content-Type':'application/json'},body:JSON.stringify(value)});status.textContent=response.ok?'Сервис сохранён. Первая проверка появится в течение 15 секунд.':await response.text();if(response.ok){resetServiceForm();loadServices()}};
el('serviceSettingsList').onclick=async event=>{const edit=event.target.closest('[data-service-edit]'),del=event.target.closest('[data-service-delete]');if(edit){const v=configuredServices.find(x=>x.id===Number(edit.dataset.serviceEdit));if(!v)return;el('serviceId').value=v.id;el('serviceName').value=v.name;el('serviceType').value=v.checkType;el('serviceURL').value=v.url;el('serviceLocal').value=v.localAddress||'';el('serviceCode').value=v.expectedCode;el('serviceText').value=v.expectedText||'';el('serviceTimeout').value=v.timeoutSeconds;el('serviceEnabled').checked=v.enabled;return}if(del&&confirm('Удалить сервис и его историю?')){const response=await fetch('/api/services?id='+del.dataset.serviceDelete,{method:'DELETE'});if(response.ok)loadServices()}};loadServices().then(()=>{maintenanceTargetSignature='';renderMaintenanceTargets()}).catch(()=>{});
el('maintenanceAll').onchange=toggleMaintenanceTargets;
async function loadMaintenance(){const response=await fetch('/api/settings/maintenance');if(!response.ok)return;applyMaintenanceState(await response.json())}
el('startMaintenance').onclick=async()=>{const duration=Number(el('maintenanceDuration').value),value={endsAt:Math.floor(Date.now()/1000)+duration*60,reason:el('maintenanceReason').value.trim(),allAlerts:el('maintenanceAll').checked,serviceIds:[...document.querySelectorAll('[data-maintenance-service]:checked')].map(v=>Number(v.dataset.maintenanceService)),vmids:[...document.querySelectorAll('[data-maintenance-vm]:checked')].map(v=>Number(v.dataset.maintenanceVm))},status=el('maintenanceStatus');status.textContent='Включаем режим…';const response=await fetch('/api/settings/maintenance',{method:'PUT',headers:{'Content-Type':'application/json'},body:JSON.stringify(value)});if(response.ok){maintenanceTargetSignature='';applyMaintenanceState(await response.json())}else status.textContent=await response.text()};
el('stopMaintenance').onclick=async()=>{const response=await fetch('/api/settings/maintenance',{method:'DELETE'});if(response.ok){maintenanceTargetSignature='';applyMaintenanceState(await response.json())}else el('maintenanceStatus').textContent=await response.text()};loadMaintenance().catch(()=>{});
function upsShutdownForm(){return {enabled:el('upsShutdownEnabled').checked,chargePercent:Number(el('upsShutdownCharge').value),runtimeSeconds:Number(el('upsShutdownRuntime').value),graceSeconds:Number(el('upsShutdownGrace').value),guestOrder:el('upsShutdownOrder').value.split(',').map(v=>Number(v.trim())).filter(v=>v>0)}}
async function loadUPSShutdown(){const response=await fetch('/api/settings/ups-shutdown');if(!response.ok)return;const s=await response.json();el('upsShutdownEnabled').checked=!!s.enabled;el('upsShutdownCharge').value=s.chargePercent||15;el('upsShutdownRuntime').value=s.runtimeSeconds||600;el('upsShutdownGrace').value=s.graceSeconds||120;el('upsShutdownOrder').value=(s.guestOrder||[]).join(', ')}
el('saveUPSShutdown').onclick=async()=>{const status=el('upsShutdownStatus');status.textContent='Сохранение…';const response=await fetch('/api/settings/ups-shutdown',{method:'PUT',headers:{'Content-Type':'application/json'},body:JSON.stringify(upsShutdownForm())});status.textContent=response.ok?(upsShutdownForm().enabled?'Сценарий сохранён и разрешён.':'Сценарий сохранён, автоматическое выполнение отключено.'):await response.text()};
el('dryRunUPSShutdown').onclick=async()=>{const saved=await fetch('/api/settings/ups-shutdown',{method:'PUT',headers:{'Content-Type':'application/json'},body:JSON.stringify(upsShutdownForm())});if(!saved.ok){el('upsShutdownStatus').textContent=await saved.text();return}const response=await fetch('/api/ups/shutdown-dry-run',{method:'POST'}),steps=await response.json();el('upsShutdownPlan').innerHTML=(steps||[]).map(v=>'<div class="shutdown-step"><b>'+v.order+'. '+safe(v.target)+'</b><div class="disk-meta">'+safe(v.action)+' · '+safe(v.detail)+'</div></div>').join('');el('upsShutdownStatus').textContent='Dry-run завершён: ни одна VM и хост не были остановлены.'};loadUPSShutdown().catch(()=>{});
function emailForm(){return {enabled:el('emailEnabled').checked,skipTlsVerify:el('emailSkipTLS').checked,host:el('emailHost').value.trim(),port:Number(el('emailPort').value),username:el('emailUsername').value.trim(),password:el('emailPassword').value,from:el('emailFrom').value.trim(),to:el('emailTo').value.trim(),intervalMinutes:Number(el('emailInterval').value),severities:[...document.querySelectorAll('[data-email-severity]:checked')].map(x=>x.dataset.emailSeverity),sources:[...document.querySelectorAll('[data-email-source]:checked')].map(x=>x.dataset.emailSource)}}
async function loadEmail(){const response=await fetch('/api/settings/email');if(!response.ok)return;const s=await response.json();el('emailEnabled').checked=s.enabled;el('emailSkipTLS').checked=!!s.skipTlsVerify;el('emailHost').value=s.host||'';el('emailPort').value=s.port||587;el('emailUsername').value=s.username||'';el('emailPassword').value='';el('emailPassword').placeholder=s.hasPassword?'Сохранён · не изменять':'Пароль';el('emailFrom').value=s.from||'';el('emailTo').value=s.to||'';el('emailInterval').value=s.intervalMinutes||60;document.querySelectorAll('[data-email-severity]').forEach(x=>x.checked=(s.severities||[]).includes(x.dataset.emailSeverity));document.querySelectorAll('[data-email-source]').forEach(x=>x.checked=(s.sources||[]).includes(x.dataset.emailSource))}
el('saveEmail').onclick=async()=>{const status=el('emailStatus');status.textContent='Сохранение…';const response=await fetch('/api/settings/email',{method:'PUT',headers:{'Content-Type':'application/json'},body:JSON.stringify(emailForm())});status.textContent=response.ok?'Настройки сохранены.':await response.text();if(response.ok)loadEmail()};
el('testEmail').onclick=async()=>{const status=el('emailStatus');status.textContent='Отправляем тестовое письмо…';const saved=await fetch('/api/settings/email',{method:'PUT',headers:{'Content-Type':'application/json'},body:JSON.stringify(emailForm())});if(!saved.ok){status.textContent=await saved.text();return}const response=await fetch('/api/settings/email/test',{method:'POST'});status.textContent=response.ok?'Тестовое письмо отправлено.':await response.text()};loadEmail().catch(()=>{});
el('installUpdate').onclick=async()=>{const button=el('installUpdate');button.disabled=true;button.textContent='Ставим в очередь…';el('updateMessage').textContent='Создаём запрос на обновление…';try{const response=await fetch('/api/update/request',{method:'POST'});if(!response.ok)throw new Error(await response.text());const status=await response.json();el('updateMessage').textContent=status.message;button.textContent='Обновление выполняется…'}catch(e){button.disabled=false;button.textContent='Попробовать ещё раз';el('updateMessage').textContent='Не удалось запустить обновление.'}};
const events=new EventSource('/api/events');events.onmessage=e=>render(JSON.parse(e.data));events.onerror=()=>{el('dot').classList.remove('live');el('status').textContent='Переподключение…'};
setInterval(()=>{if(lastDashboardEvent&&Date.now()-lastDashboardEvent>15000){el('dot').classList.remove('live');el('status').textContent='Данные устарели · переподключение…';el('monitorState').textContent='STALE';el('monitorBar').style.width='100%'}},3000);
</script></body></html>`

205
zfs.go
View File

@@ -2,7 +2,9 @@ package main
import (
"context"
"os"
"os/exec"
"regexp"
"strconv"
"strings"
"sync"
@@ -10,28 +12,66 @@ import (
)
type ZFSPoolMetrics struct {
Name string `json:"name"`
Health string `json:"health"`
SizeBytes uint64 `json:"sizeBytes"`
AllocatedBytes uint64 `json:"allocatedBytes"`
FreeBytes uint64 `json:"freeBytes"`
CapacityPercent float64 `json:"capacityPercent"`
Fragmentation float64 `json:"fragmentationPercent"`
DedupRatio string `json:"dedupRatio"`
Scan string `json:"scan"`
Errors string `json:"errors"`
AttentionReasons []string `json:"attentionReasons"`
Name string `json:"name"`
Health string `json:"health"`
SizeBytes uint64 `json:"sizeBytes"`
AllocatedBytes uint64 `json:"allocatedBytes"`
FreeBytes uint64 `json:"freeBytes"`
CapacityPercent float64 `json:"capacityPercent"`
Fragmentation float64 `json:"fragmentationPercent"`
DedupRatio string `json:"dedupRatio"`
Scan string `json:"scan"`
Errors string `json:"errors"`
AttentionReasons []string `json:"attentionReasons"`
ReadErrors uint64 `json:"readErrors"`
WriteErrors uint64 `json:"writeErrors"`
ChecksumErrors uint64 `json:"checksumErrors"`
VDevs []ZFSVDev `json:"vdevs"`
ScrubStatus string `json:"scrubStatus"`
LastScrubAt string `json:"lastScrubAt"`
ScrubDuration string `json:"scrubDuration"`
ScrubTooOld bool `json:"scrubTooOld"`
ErrorChanges []string `json:"errorChanges"`
}
type ZFSVDev struct {
Name string `json:"name"`
Class string `json:"class"`
State string `json:"state"`
Read uint64 `json:"read"`
Write uint64 `json:"write"`
Checksum uint64 `json:"checksum"`
}
type ZFSDataset struct {
Name string `json:"name"`
Pool string `json:"pool"`
UsedBytes uint64 `json:"usedBytes"`
AvailableBytes uint64 `json:"availableBytes"`
ReferencedBytes uint64 `json:"referencedBytes"`
Mountpoint string `json:"mountpoint"`
CapacityPercent float64 `json:"capacityPercent"`
}
type ZFSARC struct {
Available bool `json:"available"`
SizeBytes uint64 `json:"sizeBytes"`
TargetBytes uint64 `json:"targetBytes"`
HitRatio float64 `json:"hitRatio"`
Hits uint64 `json:"hits"`
Misses uint64 `json:"misses"`
}
type ZFSMetrics struct {
Available bool `json:"available"`
Pools []ZFSPoolMetrics `json:"pools"`
Datasets []ZFSDataset `json:"datasets"`
ARC ZFSARC `json:"arc"`
}
type zfsCollector struct {
mu sync.Mutex
cached ZFSMetrics
updatedAt time.Time
previous map[string]ZFSPoolMetrics
}
func (c *zfsCollector) collect() ZFSMetrics {
@@ -40,7 +80,26 @@ func (c *zfsCollector) collect() ZFSMetrics {
if time.Since(c.updatedAt) < 30*time.Second {
return c.cached
}
c.cached = readZFSPools()
current := readZFSPools()
if c.previous == nil {
c.previous = map[string]ZFSPoolMetrics{}
}
for i := range current.Pools {
pool := &current.Pools[i]
if old, ok := c.previous[pool.Name]; ok {
checks := []struct {
label string
before, now uint64
}{{"read", old.ReadErrors, pool.ReadErrors}, {"write", old.WriteErrors, pool.WriteErrors}, {"checksum", old.ChecksumErrors, pool.ChecksumErrors}}
for _, check := range checks {
if check.now > check.before {
pool.ErrorChanges = append(pool.ErrorChanges, check.label+": "+strconv.FormatUint(check.before, 10)+" → "+strconv.FormatUint(check.now, 10))
}
}
}
c.previous[pool.Name] = *pool
}
c.cached = current
c.updatedAt = time.Now()
return c.cached
}
@@ -67,15 +126,35 @@ func readZFSPools() ZFSMetrics {
pool.FreeBytes, _ = strconv.ParseUint(fields[3], 10, 64)
pool.Fragmentation = parsePercent(fields[4])
pool.CapacityPercent = parsePercent(fields[5])
pool.Scan, pool.Errors = readZFSStatus(pool.Name)
pool.Scan, pool.Errors, pool.VDevs = readZFSStatus(pool.Name)
for _, vdev := range pool.VDevs {
pool.ReadErrors += vdev.Read
pool.WriteErrors += vdev.Write
pool.ChecksumErrors += vdev.Checksum
if vdev.State != "ONLINE" {
pool.AttentionReasons = append(pool.AttentionReasons, vdev.Class+" "+vdev.Name+": "+vdev.State)
}
}
pool.ScrubStatus, pool.LastScrubAt, pool.ScrubDuration, pool.ScrubTooOld = parseScrub(pool.Scan)
if pool.Health != "ONLINE" {
pool.AttentionReasons = append(pool.AttentionReasons, "Состояние пула: "+pool.Health)
}
if pool.Errors != "" && !strings.Contains(pool.Errors, "No known data errors") {
pool.AttentionReasons = append(pool.AttentionReasons, pool.Errors)
}
if pool.ReadErrors+pool.WriteErrors+pool.ChecksumErrors > 0 {
pool.AttentionReasons = append(pool.AttentionReasons, "Ошибки vdev: read="+strconv.FormatUint(pool.ReadErrors, 10)+", write="+strconv.FormatUint(pool.WriteErrors, 10)+", checksum="+strconv.FormatUint(pool.ChecksumErrors, 10))
}
if pool.ScrubTooOld {
pool.AttentionReasons = append(pool.AttentionReasons, "Scrub не выполнялся более 35 дней")
}
if pool.CapacityPercent >= 80 {
pool.AttentionReasons = append(pool.AttentionReasons, "Мало свободного места для безопасной CoW-работы ZFS")
}
result.Pools = append(result.Pools, pool)
}
result.Datasets = readZFSDatasets()
result.ARC = readZFSARC()
return result
}
@@ -84,11 +163,13 @@ func parsePercent(value string) float64 {
return parsed
}
func readZFSStatus(pool string) (string, string) {
func readZFSStatus(pool string) (string, string, []ZFSVDev) {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
output, _ := exec.CommandContext(ctx, "zpool", "status", pool).Output()
var scan, errors string
var scan, errors, currentClass string
var vdevs []ZFSVDev
inConfig := false
for _, line := range strings.Split(string(output), "\n") {
trimmed := strings.TrimSpace(line)
if strings.HasPrefix(trimmed, "scan:") {
@@ -97,6 +178,98 @@ func readZFSStatus(pool string) (string, string) {
if strings.HasPrefix(trimmed, "errors:") {
errors = strings.TrimSpace(strings.TrimPrefix(trimmed, "errors:"))
}
if strings.HasPrefix(trimmed, "config:") {
inConfig = true
continue
}
if !inConfig || trimmed == "" || strings.HasPrefix(trimmed, "NAME") {
continue
}
fields := strings.Fields(trimmed)
if len(fields) == 1 && (fields[0] == "logs" || fields[0] == "cache" || fields[0] == "spares" || fields[0] == "special") {
currentClass = fields[0]
continue
}
if len(fields) >= 5 && (fields[1] == "ONLINE" || fields[1] == "DEGRADED" || fields[1] == "OFFLINE" || fields[1] == "UNAVAIL" || fields[1] == "FAULTED" || fields[1] == "REMOVED") {
read, _ := strconv.ParseUint(fields[2], 10, 64)
write, _ := strconv.ParseUint(fields[3], 10, 64)
checksum, _ := strconv.ParseUint(fields[4], 10, 64)
class := currentClass
if class == "" {
class = "data"
}
vdevs = append(vdevs, ZFSVDev{Name: fields[0], Class: class, State: fields[1], Read: read, Write: write, Checksum: checksum})
}
}
return scan, errors
return scan, errors, vdevs
}
var scrubDatePattern = regexp.MustCompile(`(?:on|since) (\w{3} \w{3}\s+\d+ \d\d:\d\d:\d\d \d{4})`)
func parseScrub(scan string) (status, lastAt, duration string, tooOld bool) {
lower := strings.ToLower(scan)
status = scan
if strings.Contains(lower, "in progress") {
status = "Выполняется"
} else if strings.Contains(lower, "repaired") {
status = "Завершён"
} else if strings.Contains(lower, "none requested") || scan == "" {
status = "Никогда не запускался"
return status, "", "", true
}
if match := scrubDatePattern.FindStringSubmatch(scan); len(match) > 1 {
if t, err := time.Parse("Mon Jan 2 15:04:05 2006", match[1]); err == nil {
lastAt = t.Format(time.RFC3339)
tooOld = time.Since(t) > 35*24*time.Hour
}
}
if parts := strings.Split(scan, " in "); len(parts) > 1 {
duration = strings.Fields(parts[1])[0]
}
return
}
func readZFSDatasets() []ZFSDataset {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
out, err := exec.CommandContext(ctx, "zfs", "list", "-H", "-p", "-o", "name,used,avail,refer,mountpoint").Output()
if err != nil {
return nil
}
var result []ZFSDataset
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
f := strings.Fields(line)
if len(f) < 5 {
continue
}
used, _ := strconv.ParseUint(f[1], 10, 64)
avail, _ := strconv.ParseUint(f[2], 10, 64)
refer, _ := strconv.ParseUint(f[3], 10, 64)
pct := 0.0
if used+avail > 0 {
pct = float64(used) / float64(used+avail) * 100
}
result = append(result, ZFSDataset{Name: f[0], Pool: strings.Split(f[0], "/")[0], UsedBytes: used, AvailableBytes: avail, ReferencedBytes: refer, Mountpoint: strings.Join(f[4:], " "), CapacityPercent: pct})
}
return result
}
func readZFSARC() ZFSARC {
data, err := os.ReadFile("/proc/spl/kstat/zfs/arcstats")
if err != nil {
return ZFSARC{}
}
values := map[string]uint64{}
for _, line := range strings.Split(string(data), "\n") {
f := strings.Fields(line)
if len(f) >= 3 {
values[f[0]], _ = strconv.ParseUint(f[2], 10, 64)
}
}
hits, misses := values["hits"], values["misses"]
ratio := 0.0
if hits+misses > 0 {
ratio = float64(hits) / float64(hits+misses) * 100
}
return ZFSARC{Available: true, SizeBytes: values["size"], TargetBytes: values["c"], HitRatio: ratio, Hits: hits, Misses: misses}
}