Compare commits
11 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f7b347212e | ||
|
|
72869dcf5f | ||
|
|
1296d052a5 | ||
|
|
f764195ba5 | ||
|
|
f6498a2fc1 | ||
|
|
714c29c449 | ||
|
|
21b4f528e0 | ||
|
|
c50b4cf986 | ||
|
|
338d53cb2f | ||
|
|
a2525b53f0 | ||
|
|
331f0ee39a |
130
README.md
130
README.md
@@ -32,7 +32,7 @@ ssh root@192.168.0.10
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
apt-get update
|
apt-get update
|
||||||
apt-get install -y curl ca-certificates tar smartmontools iproute2 iputils-ping
|
apt-get install -y curl ca-certificates tar smartmontools iproute2 iputils-ping ethtool
|
||||||
|
|
||||||
cd /tmp
|
cd /tmp
|
||||||
curl -fL \
|
curl -fL \
|
||||||
@@ -342,6 +342,33 @@ APT-обновления и необходимость перезагрузки.
|
|||||||
health-check, число перезапусков, CPU и память. Агент сам подключается к Dashboard
|
health-check, число перезапусков, CPU и память. Агент сам подключается к Dashboard
|
||||||
каждые 10 секунд; открывать входящий порт внутри гостевой системы не нужно.
|
каждые 10 секунд; открывать входящий порт внутри гостевой системы не нужно.
|
||||||
|
|
||||||
|
В подробностях агента systemd-службы разделяются на системные, установленные
|
||||||
|
дополнительно и локальные unit-файлы. Для каждой службы показываются пакет и путь
|
||||||
|
unit-файла, время запуска, число перезапусков, последний exit code, память и
|
||||||
|
накопленное CPU-время, когда соответствующий accounting доступен в systemd.
|
||||||
|
|
||||||
|
Расширенная диагностика агента включает последние сообщения проблемных служб,
|
||||||
|
`Requires`/`After`, PID, команду запуска и слушающие порты; выявляет частые
|
||||||
|
рестарты, OOM-kill и процессы в D-state. Отдельно отображаются top процессов по
|
||||||
|
CPU и RAM с накопленным I/O, swap, PSI pressure, все постоянные файловые системы,
|
||||||
|
их заполнение и использование inode.
|
||||||
|
|
||||||
|
Панель «Диагностика homelab» и карта сопоставляют известные IP и порты с VM/LXC,
|
||||||
|
агентом, systemd-службой или Docker-контейнером и настроенной HTTP/TCP-проверкой.
|
||||||
|
Связи по PID и порту считаются подтверждёнными; совпадения только по названию
|
||||||
|
помечаются как предположение.
|
||||||
|
|
||||||
|
Для сложных reverse proxy и Docker-сетей в управлении сервисами можно сохранить
|
||||||
|
ручную связь `сервис → агент/VMID → systemd/Docker/процесс → порт`. Она имеет
|
||||||
|
приоритет над автоматическим определением и используется для группировки
|
||||||
|
связанных уведомлений по первопричине.
|
||||||
|
|
||||||
|
Dashboard хранит семь дней истории CPU, RAM, swap, PSI и файловых систем каждого
|
||||||
|
агента. Из подробностей агента доступны только заранее разрешённые действия:
|
||||||
|
перезапуск известной systemd-службы или Docker-контейнера, `reset-failed` и
|
||||||
|
обновление списка пакетов. Агент не принимает произвольные shell-команды; каждое
|
||||||
|
действие требует подтверждения в браузере и сохраняется в журнале управления.
|
||||||
|
|
||||||
Откройте `Настройки → Агенты`, укажите адрес Dashboard, доступный из VM/LXC, и
|
Откройте `Настройки → Агенты`, укажите адрес Dashboard, доступный из VM/LXC, и
|
||||||
VMID. Кнопка создаст готовую команду установки. Одноразовый токен действует 15
|
VMID. Кнопка создаст готовую команду установки. Одноразовый токен действует 15
|
||||||
минут и после регистрации заменяется индивидуальным секретом агента.
|
минут и после регистрации заменяется индивидуальным секретом агента.
|
||||||
@@ -374,3 +401,104 @@ curl -fsSL https://git.myown.center/maxim/ProxmoxDash/raw/branch/main/scripts/un
|
|||||||
go test ./...
|
go test ./...
|
||||||
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o proxmox-cpu-dashboard .
|
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o proxmox-cpu-dashboard .
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Сетевые ошибки и температура CPU (v0.19.3)
|
||||||
|
|
||||||
|
RX/TX drops отображаются отдельно как информационные счётчики. Их рост сам по
|
||||||
|
себе не вызывает аварийный алерт: в том числе LLDP на Linux bridge. Настройки
|
||||||
|
Keenetic, LLDP и group_fwd_mask менять не требуется. Для VM учитываются tap,
|
||||||
|
для LXC — veth; fwbr/fwpr/fwln остаются в группе интерфейсов хоста.
|
||||||
|
|
||||||
|
Новые rx_errors/tx_errors, CRC/frame/FIFO/missed, carrier/link-down вызывают
|
||||||
|
критический алерт. Sysfs позволяет отделить rx_missed_errors от RX drops,
|
||||||
|
которые в /proc/net/dev объединены. Для Atlantic дополнительно читаются
|
||||||
|
InErrors, InDroppedDma, AllocFails, SkbAllocFails и Restarts через ethtool -S.
|
||||||
|
Установите `ethtool` на существующем хосте для этой дополнительной диагностики.
|
||||||
|
При отсутствии утилиты остаются стандартные счётчики sysfs. Счётчики ошибок
|
||||||
|
могут пересекаться: их сумма — прирост счётчиков, а не число уникальных пакетов.
|
||||||
|
После перезапуска и сброса счётчиков старые значения не создают новые алерты.
|
||||||
|
|
||||||
|
Карточка CPU и новые точки истории показывают максимум доступных датчиков CPU,
|
||||||
|
а не среднее Tctl/Tdie/CCD или ядер. На AMD Tctl может включать управляющее
|
||||||
|
смещение; подписи и значения всех датчиков доступны в подробностях CPU.
|
||||||
|
История снимается раз в минуту, а на диапазоне 7 дней усредняется по 10 минутам,
|
||||||
|
поэтому короткие пики между измерениями не гарантированно попадут на график.
|
||||||
|
Старые точки до обновления остаются средними: исходные значения датчиков в них
|
||||||
|
не сохранены. Масштаб графика расширяется для значений выше 100 °C.
|
||||||
|
|
||||||
|
Релиз Linux amd64 собирается с номером версии и без CGO:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./scripts/build-release.sh v0.19.3
|
||||||
|
```
|
||||||
|
|
||||||
|
Скрипт запускает тесты и создаёт в dist бинарник, SHA-256 и установочный архив,
|
||||||
|
совместимые со встроенным обновлением из Gitea Releases.
|
||||||
|
|
||||||
|
## Карта трафика NPM + CrowdSec (v0.20.0)
|
||||||
|
|
||||||
|
Откройте **Настройки → Карта трафика**, включите страницу и выберите
|
||||||
|
агент VM/LXC, где установлены Nginx Proxy Manager и CrowdSec. Например,
|
||||||
|
`nginxproxymanager · VMID 102`. Необязательно выберите страну сервера для
|
||||||
|
конечной точки линий на карте. Сохраните настройки: в шапке появится ссылка
|
||||||
|
«Карта трафика», страница также доступна по `/traffic`.
|
||||||
|
|
||||||
|
Функция **по умолчанию выключена**. Только выбранный агент получает команду
|
||||||
|
включить сбор. При выключении или смене источника старый агент останавливает
|
||||||
|
сбор после следующего успешного обычного отчёта; сервер сразу прекращает
|
||||||
|
приём его событий. Выключенная страница и её API недоступны. Это глобальная
|
||||||
|
настройка экземпляра Dashboard. Карта не создаёт общих алертов или писем.
|
||||||
|
|
||||||
|
Агент автоматически обновляется вместе с Dashboard; для карты требуется
|
||||||
|
v0.20.0 или новее. Он читает стандартные NPM access-логи из
|
||||||
|
`/opt/npm/data/logs/*_access.log`, включая несжатые файлы ротации. Другой каталог
|
||||||
|
задаётся переменной окружения службы `DASHBOARD_NPM_LOG_DIR`; значение `off`
|
||||||
|
полностью запрещает сбор на конкретном агенте. Порт CrowdSec наружу открывать
|
||||||
|
и новые API-ключи создавать не нужно: используется локальная команда
|
||||||
|
`cscli alerts list --since 24h --limit 50 -o json` с существующими учётными
|
||||||
|
данными машины. Права на логи и запуск cscli нужны службе агента.
|
||||||
|
|
||||||
|
- Голубой: все записанные HTTP-запросы, включая обычные и локальные обращения.
|
||||||
|
- Оранжевый: реальные (не simulated) срабатывания сценариев CrowdSec.
|
||||||
|
- Розовый: решения CrowdSec, например ban или captcha. Это **не** счётчик
|
||||||
|
фактически отброшенных пакетов и не список действующих сейчас банов.
|
||||||
|
- HTTP 401/403/502 сами по себе не считаются атакой или блокировкой.
|
||||||
|
|
||||||
|
Карта показывает страны источников, живую ленту последних 100 событий,
|
||||||
|
счётчики за последние 24 часа, уникальные IP HTTP-клиентов, график по минутам
|
||||||
|
и средний req/s за последние 60 секунд. Есть фильтры типа события, домена и
|
||||||
|
внешних IP, пауза и полный экран. Локальные адреса видны в ленте без страны.
|
||||||
|
Все счётчики следуют выбранным фильтрам. Уникальный IP не равен человеку.
|
||||||
|
|
||||||
|
Первое включение начинает чтение с конца логов, без импорта старых HTTP-запросов.
|
||||||
|
Обычно обновление занимает несколько секунд; Nginx записывает запрос после
|
||||||
|
его завершения. Это не карта открытых TCP-соединений. Позиции чтения и очередь
|
||||||
|
до 500 событий сохраняются в `traffic-state.json` рядом с конфигом агента;
|
||||||
|
повторы после обрыва связи устраняются на сервере. При переполнении очереди
|
||||||
|
чтение ждёт отправки. Если за время длительного простоя логи уже сжаты или
|
||||||
|
удалены, восстановить эти строки нельзя. Размер очереди чтения и ошибки
|
||||||
|
парсинга отображаются на самой странице, без глобальных уведомлений.
|
||||||
|
|
||||||
|
CrowdSec опрашивается каждые 15 секунд, до 50 последних alerts за 24 часа;
|
||||||
|
при достижении лимита страница предупреждает о возможной неполноте истории.
|
||||||
|
Связанный домен берётся из метаданных события, только если он однозначен.
|
||||||
|
Новый ban не рисуется как новый запрос. Для наблюдения реально отброшенных
|
||||||
|
firewall пакетов нужен отдельный источник: в этой версии его нет.
|
||||||
|
|
||||||
|
В SQLite хранится до 24 часов и 200 000 событий; старые строки удаляются при
|
||||||
|
приёме новых данных. Данные каждого выбранного агента фильтруются отдельно.
|
||||||
|
Query string и fragment URL отбрасываются на агенте и повторно на сервере;
|
||||||
|
заголовки, cookie и тела запросов не передаются. IP, домен и путь сохраняются.
|
||||||
|
|
||||||
|
GeoIP работает локально: DB-IP Country Lite за октябрь 2026 встроена в бинарник,
|
||||||
|
исходящие запросы с IP посетителей не выполняются. Координаты на карте — условные
|
||||||
|
точки стран Natural Earth, а не точное местоположение посетителя. Дата базы
|
||||||
|
видна внизу карты. Обновление базы требует нового релиза; автоматического
|
||||||
|
скачивания нет. Лицензии и происхождение данных: `assets/NOTICE.md`.
|
||||||
|
|
||||||
|
Локальный предпросмотр с явно обозначенными искусственными данными:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
DASHBOARD_TRAFFIC_PREVIEW=1 go test -run '^TestTrafficPreview$' -timeout 30m
|
||||||
|
# http://127.0.0.1:19105/traffic
|
||||||
|
```
|
||||||
|
|||||||
308
agent.go
308
agent.go
@@ -13,6 +13,7 @@ import (
|
|||||||
"io"
|
"io"
|
||||||
"net"
|
"net"
|
||||||
"net/http"
|
"net/http"
|
||||||
|
"net/url"
|
||||||
"os"
|
"os"
|
||||||
"os/exec"
|
"os/exec"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
@@ -50,6 +51,12 @@ type AgentContainer struct {
|
|||||||
CPUPercent float64 `json:"cpuPercent"`
|
CPUPercent float64 `json:"cpuPercent"`
|
||||||
MemoryUsage uint64 `json:"memoryUsage"`
|
MemoryUsage uint64 `json:"memoryUsage"`
|
||||||
Ports []string `json:"ports,omitempty"`
|
Ports []string `json:"ports,omitempty"`
|
||||||
|
Compose bool `json:"compose"`
|
||||||
|
UpdateAvailable bool `json:"updateAvailable"`
|
||||||
|
CurrentVersion string `json:"currentVersion,omitempty"`
|
||||||
|
CurrentDigest string `json:"currentDigest,omitempty"`
|
||||||
|
AvailableVersion string `json:"availableVersion,omitempty"`
|
||||||
|
AvailableDigest string `json:"availableDigest,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type AgentInterface struct {
|
type AgentInterface struct {
|
||||||
@@ -62,6 +69,27 @@ type AgentService struct {
|
|||||||
State string `json:"state"`
|
State string `json:"state"`
|
||||||
SubState string `json:"subState"`
|
SubState string `json:"subState"`
|
||||||
Description string `json:"description"`
|
Description string `json:"description"`
|
||||||
|
Origin string `json:"origin"`
|
||||||
|
UnitPath string `json:"unitPath,omitempty"`
|
||||||
|
Package string `json:"package,omitempty"`
|
||||||
|
Restarts uint64 `json:"restarts"`
|
||||||
|
ExitCode int `json:"exitCode"`
|
||||||
|
StartedAt int64 `json:"startedAt,omitempty"`
|
||||||
|
MemoryBytes uint64 `json:"memoryBytes,omitempty"`
|
||||||
|
CPUSeconds float64 `json:"cpuSeconds,omitempty"`
|
||||||
|
MainPID int `json:"mainPid,omitempty"`
|
||||||
|
ExecStart string `json:"execStart,omitempty"`
|
||||||
|
Requires []string `json:"requires,omitempty"`
|
||||||
|
After []string `json:"after,omitempty"`
|
||||||
|
ListenPorts []int `json:"listenPorts,omitempty"`
|
||||||
|
Logs []AgentLog `json:"logs,omitempty"`
|
||||||
|
Flapping bool `json:"flapping"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type AgentIssue struct {
|
||||||
|
Time int64 `json:"time"`
|
||||||
|
Unit string `json:"unit"`
|
||||||
|
Message string `json:"message"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type AgentReport struct {
|
type AgentReport struct {
|
||||||
@@ -75,8 +103,11 @@ type AgentReport struct {
|
|||||||
UptimeSeconds int64 `json:"uptimeSeconds"`
|
UptimeSeconds int64 `json:"uptimeSeconds"`
|
||||||
Load1 float64 `json:"load1"`
|
Load1 float64 `json:"load1"`
|
||||||
CPUPercent float64 `json:"cpuPercent"`
|
CPUPercent float64 `json:"cpuPercent"`
|
||||||
|
CPUCount int `json:"cpuCount"`
|
||||||
MemoryTotal uint64 `json:"memoryTotal"`
|
MemoryTotal uint64 `json:"memoryTotal"`
|
||||||
MemoryUsed uint64 `json:"memoryUsed"`
|
MemoryUsed uint64 `json:"memoryUsed"`
|
||||||
|
SwapTotal uint64 `json:"swapTotal"`
|
||||||
|
SwapUsed uint64 `json:"swapUsed"`
|
||||||
RootTotal uint64 `json:"rootTotal"`
|
RootTotal uint64 `json:"rootTotal"`
|
||||||
RootUsed uint64 `json:"rootUsed"`
|
RootUsed uint64 `json:"rootUsed"`
|
||||||
DockerAvailable bool `json:"dockerAvailable"`
|
DockerAvailable bool `json:"dockerAvailable"`
|
||||||
@@ -84,10 +115,17 @@ type AgentReport struct {
|
|||||||
Containers []AgentContainer `json:"containers"`
|
Containers []AgentContainer `json:"containers"`
|
||||||
Interfaces []AgentInterface `json:"interfaces"`
|
Interfaces []AgentInterface `json:"interfaces"`
|
||||||
Services []AgentService `json:"services"`
|
Services []AgentService `json:"services"`
|
||||||
|
Issues []AgentIssue `json:"issues"`
|
||||||
|
Processes []AgentProcess `json:"processes"`
|
||||||
|
Filesystems []AgentFilesystem `json:"filesystems"`
|
||||||
|
ListenPorts []AgentListenPort `json:"listenPorts"`
|
||||||
|
Pressure AgentPressure `json:"pressure"`
|
||||||
|
OOMKills []AgentLog `json:"oomKills"`
|
||||||
UpdatesAvailable int `json:"updatesAvailable"`
|
UpdatesAvailable int `json:"updatesAvailable"`
|
||||||
RebootRequired bool `json:"rebootRequired"`
|
RebootRequired bool `json:"rebootRequired"`
|
||||||
InventoryAt int64 `json:"inventoryAt"`
|
InventoryAt int64 `json:"inventoryAt"`
|
||||||
CollectedAt int64 `json:"collectedAt"`
|
CollectedAt int64 `json:"collectedAt"`
|
||||||
|
ActionResults []AgentActionResult `json:"actionResults,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type agentInventory struct {
|
type agentInventory struct {
|
||||||
@@ -107,7 +145,9 @@ type AgentUpdateManifest struct {
|
|||||||
|
|
||||||
type agentReportResponse struct {
|
type agentReportResponse struct {
|
||||||
Accepted bool `json:"accepted"`
|
Accepted bool `json:"accepted"`
|
||||||
|
TrafficEnabled bool `json:"trafficEnabled"`
|
||||||
Update *AgentUpdateManifest `json:"update,omitempty"`
|
Update *AgentUpdateManifest `json:"update,omitempty"`
|
||||||
|
Commands []AgentCommand `json:"commands,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
var guestInventory struct {
|
var guestInventory struct {
|
||||||
@@ -115,6 +155,17 @@ var guestInventory struct {
|
|||||||
value agentInventory
|
value agentInventory
|
||||||
}
|
}
|
||||||
|
|
||||||
|
var guestIssues struct {
|
||||||
|
sync.Mutex
|
||||||
|
value []AgentIssue
|
||||||
|
collectedAt time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
var guestActionResults struct {
|
||||||
|
sync.Mutex
|
||||||
|
value []AgentActionResult
|
||||||
|
}
|
||||||
|
|
||||||
var agentBinaryInfo struct {
|
var agentBinaryInfo struct {
|
||||||
sync.Once
|
sync.Once
|
||||||
manifest AgentUpdateManifest
|
manifest AgentUpdateManifest
|
||||||
@@ -223,6 +274,8 @@ func (s *Store) SaveAgentReport(id, secret, remote string, report AgentReport) e
|
|||||||
if !s.AuthenticateAgent(id, secret) {
|
if !s.AuthenticateAgent(id, secret) {
|
||||||
return errors.New("агент не авторизован")
|
return errors.New("агент не авторизован")
|
||||||
}
|
}
|
||||||
|
s.CompleteAgentActions(id, report.ActionResults)
|
||||||
|
report.ActionResults = nil
|
||||||
data, _ := json.Marshal(report)
|
data, _ := json.Marshal(report)
|
||||||
_, err := s.db.Exec(`UPDATE agents SET hostname=?,version=?,last_seen=?,report_json=?,remote_address=? WHERE id=?`, report.Hostname, report.Version, time.Now().Unix(), string(data), remote, id)
|
_, err := s.db.Exec(`UPDATE agents SET hostname=?,version=?,last_seen=?,report_json=?,remote_address=? WHERE id=?`, report.Hostname, report.Version, time.Now().Unix(), string(data), remote, id)
|
||||||
return err
|
return err
|
||||||
@@ -287,9 +340,21 @@ func runGuestAgent(opts agentOptions) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
var trafficStop func()
|
||||||
|
stopTraffic := func() {
|
||||||
|
if trafficStop != nil {
|
||||||
|
trafficStop()
|
||||||
|
trafficStop = nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
defer stopTraffic()
|
||||||
|
|
||||||
client := &http.Client{Timeout: 12 * time.Second}
|
client := &http.Client{Timeout: 12 * time.Second}
|
||||||
for {
|
for {
|
||||||
report := collectAgentReport()
|
report := collectAgentReport()
|
||||||
|
guestActionResults.Lock()
|
||||||
|
report.ActionResults = append([]AgentActionResult(nil), guestActionResults.value...)
|
||||||
|
guestActionResults.Unlock()
|
||||||
data, _ := json.Marshal(report)
|
data, _ := json.Marshal(report)
|
||||||
req, _ := http.NewRequest(http.MethodPost, config.Server+"/api/agent/report", bytes.NewReader(data))
|
req, _ := http.NewRequest(http.MethodPost, config.Server+"/api/agent/report", bytes.NewReader(data))
|
||||||
req.Header.Set("Content-Type", "application/json")
|
req.Header.Set("Content-Type", "application/json")
|
||||||
@@ -301,6 +366,7 @@ func runGuestAgent(opts agentOptions) error {
|
|||||||
if response.StatusCode == http.StatusUnauthorized && opts.EnrollmentToken != "" {
|
if response.StatusCode == http.StatusUnauthorized && opts.EnrollmentToken != "" {
|
||||||
fresh, enrollmentErr := enrollRemote(config, opts.EnrollmentToken)
|
fresh, enrollmentErr := enrollRemote(config, opts.EnrollmentToken)
|
||||||
if enrollmentErr == nil {
|
if enrollmentErr == nil {
|
||||||
|
stopTraffic()
|
||||||
config = fresh
|
config = fresh
|
||||||
if saveErr := saveAgentConfig(opts.ConfigPath, config); saveErr != nil {
|
if saveErr := saveAgentConfig(opts.ConfigPath, config); saveErr != nil {
|
||||||
return saveErr
|
return saveErr
|
||||||
@@ -316,7 +382,29 @@ func runGuestAgent(opts agentOptions) error {
|
|||||||
}
|
}
|
||||||
if response.StatusCode == http.StatusOK {
|
if response.StatusCode == http.StatusOK {
|
||||||
var result agentReportResponse
|
var result agentReportResponse
|
||||||
if json.Unmarshal(responseData, &result) == nil && result.Update != nil && newerAgentVersion(result.Update.Version, version) {
|
if json.Unmarshal(responseData, &result) == nil {
|
||||||
|
if !result.TrafficEnabled {
|
||||||
|
stopTraffic()
|
||||||
|
} else if trafficStop == nil {
|
||||||
|
trafficStop = startAgentTraffic(config, filepath.Dir(opts.ConfigPath))
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(report.ActionResults) > 0 {
|
||||||
|
guestActionResults.Lock()
|
||||||
|
guestActionResults.value = nil
|
||||||
|
guestActionResults.Unlock()
|
||||||
|
}
|
||||||
|
if len(result.Commands) > 0 {
|
||||||
|
executed := make([]AgentActionResult, 0, len(result.Commands))
|
||||||
|
for _, command := range result.Commands {
|
||||||
|
executed = append(executed, executeAgentCommand(command))
|
||||||
|
}
|
||||||
|
guestActionResults.Lock()
|
||||||
|
guestActionResults.value = append(guestActionResults.value, executed...)
|
||||||
|
guestActionResults.Unlock()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if result.Update != nil && newerAgentVersion(result.Update.Version, version) {
|
||||||
if updateErr := applyAgentUpdate(client, config, *result.Update); updateErr != nil {
|
if updateErr := applyAgentUpdate(client, config, *result.Update); updateErr != nil {
|
||||||
requestErr = fmt.Errorf("автообновление: %v", updateErr)
|
requestErr = fmt.Errorf("автообновление: %v", updateErr)
|
||||||
} else {
|
} else {
|
||||||
@@ -328,7 +416,9 @@ func runGuestAgent(opts agentOptions) error {
|
|||||||
if requestErr != nil {
|
if requestErr != nil {
|
||||||
fmt.Fprintf(os.Stderr, "agent: %v\n", requestErr)
|
fmt.Fprintf(os.Stderr, "agent: %v\n", requestErr)
|
||||||
}
|
}
|
||||||
time.Sleep(10 * time.Second)
|
// Короткий интервал нужен для интерактивных команд из панели. Тяжёлый
|
||||||
|
// системный инвентарь отдельно кэшируется и не запускается каждые 3 секунды.
|
||||||
|
time.Sleep(3 * time.Second)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -464,7 +554,7 @@ func saveAgentConfig(path string, c agentConfig) error {
|
|||||||
func collectAgentReport() AgentReport {
|
func collectAgentReport() AgentReport {
|
||||||
hostname, _ := os.Hostname()
|
hostname, _ := os.Hostname()
|
||||||
machineID, _ := os.ReadFile("/etc/machine-id")
|
machineID, _ := os.ReadFile("/etc/machine-id")
|
||||||
report := AgentReport{Hostname: hostname, MachineID: strings.TrimSpace(string(machineID)), Version: version, OS: runtime.GOOS, Arch: runtime.GOARCH, CollectedAt: time.Now().Unix()}
|
report := AgentReport{Hostname: hostname, MachineID: strings.TrimSpace(string(machineID)), Version: version, OS: runtime.GOOS, Arch: runtime.GOARCH, CPUCount: runtime.NumCPU(), CollectedAt: time.Now().Unix()}
|
||||||
if data, err := os.ReadFile("/proc/uptime"); err == nil {
|
if data, err := os.ReadFile("/proc/uptime"); err == nil {
|
||||||
fields := strings.Fields(string(data))
|
fields := strings.Fields(string(data))
|
||||||
if len(fields) > 0 {
|
if len(fields) > 0 {
|
||||||
@@ -480,6 +570,11 @@ func collectAgentReport() AgentReport {
|
|||||||
}
|
}
|
||||||
report.CPUPercent = sampleAgentCPU()
|
report.CPUPercent = sampleAgentCPU()
|
||||||
report.Interfaces = collectAgentInterfaces()
|
report.Interfaces = collectAgentInterfaces()
|
||||||
|
report.Processes = collectAgentProcesses()
|
||||||
|
report.Filesystems = collectAgentFilesystems()
|
||||||
|
report.ListenPorts = collectAgentListenPorts()
|
||||||
|
report.Pressure = collectAgentPressure()
|
||||||
|
report.OOMKills = cachedAgentOOMKills()
|
||||||
if data, err := os.ReadFile("/proc/meminfo"); err == nil {
|
if data, err := os.ReadFile("/proc/meminfo"); err == nil {
|
||||||
values := map[string]uint64{}
|
values := map[string]uint64{}
|
||||||
for _, line := range strings.Split(string(data), "\n") {
|
for _, line := range strings.Split(string(data), "\n") {
|
||||||
@@ -494,6 +589,10 @@ func collectAgentReport() AgentReport {
|
|||||||
if report.MemoryTotal > available {
|
if report.MemoryTotal > available {
|
||||||
report.MemoryUsed = report.MemoryTotal - available
|
report.MemoryUsed = report.MemoryTotal - available
|
||||||
}
|
}
|
||||||
|
report.SwapTotal = values["SwapTotal"]
|
||||||
|
if report.SwapTotal > values["SwapFree"] {
|
||||||
|
report.SwapUsed = report.SwapTotal - values["SwapFree"]
|
||||||
|
}
|
||||||
}
|
}
|
||||||
var stat syscall.Statfs_t
|
var stat syscall.Statfs_t
|
||||||
if syscall.Statfs("/", &stat) == nil {
|
if syscall.Statfs("/", &stat) == nil {
|
||||||
@@ -507,6 +606,7 @@ func collectAgentReport() AgentReport {
|
|||||||
report.DockerAvailable = report.DockerError == ""
|
report.DockerAvailable = report.DockerError == ""
|
||||||
inventory := cachedAgentInventory()
|
inventory := cachedAgentInventory()
|
||||||
report.OSName, report.Kernel, report.Services = inventory.OSName, inventory.Kernel, inventory.Services
|
report.OSName, report.Kernel, report.Services = inventory.OSName, inventory.Kernel, inventory.Services
|
||||||
|
report.Issues = cachedAgentIssues()
|
||||||
report.UpdatesAvailable, report.RebootRequired, report.InventoryAt = inventory.UpdatesAvailable, inventory.RebootRequired, inventory.CollectedAt.Unix()
|
report.UpdatesAvailable, report.RebootRequired, report.InventoryAt = inventory.UpdatesAvailable, inventory.RebootRequired, inventory.CollectedAt.Unix()
|
||||||
return report
|
return report
|
||||||
}
|
}
|
||||||
@@ -554,6 +654,16 @@ func cachedAgentInventory() agentInventory {
|
|||||||
return value
|
return value
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func cachedAgentIssues() []AgentIssue {
|
||||||
|
guestIssues.Lock()
|
||||||
|
defer guestIssues.Unlock()
|
||||||
|
if time.Since(guestIssues.collectedAt) >= time.Minute {
|
||||||
|
guestIssues.value = collectAgentIssues()
|
||||||
|
guestIssues.collectedAt = time.Now()
|
||||||
|
}
|
||||||
|
return guestIssues.value
|
||||||
|
}
|
||||||
|
|
||||||
func readOSPrettyName() string {
|
func readOSPrettyName() string {
|
||||||
data, err := os.ReadFile("/etc/os-release")
|
data, err := os.ReadFile("/etc/os-release")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -571,7 +681,7 @@ func runAgentCommand(timeout time.Duration, name string, args ...string) string
|
|||||||
ctx, cancel := context.WithTimeout(context.Background(), timeout)
|
ctx, cancel := context.WithTimeout(context.Background(), timeout)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
output, err := exec.CommandContext(ctx, name, args...).Output()
|
output, err := exec.CommandContext(ctx, name, args...).Output()
|
||||||
if err != nil {
|
if err != nil && len(output) == 0 {
|
||||||
return ""
|
return ""
|
||||||
}
|
}
|
||||||
return string(output)
|
return string(output)
|
||||||
@@ -594,10 +704,149 @@ func collectAgentServices() []AgentService {
|
|||||||
description = strings.Join(fields[offset+4:], " ")
|
description = strings.Join(fields[offset+4:], " ")
|
||||||
}
|
}
|
||||||
result = append(result, AgentService{Name: fields[offset], State: fields[offset+2], SubState: fields[offset+3], Description: description})
|
result = append(result, AgentService{Name: fields[offset], State: fields[offset+2], SubState: fields[offset+3], Description: description})
|
||||||
if len(result) >= 150 {
|
if len(result) >= 300 {
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
return enrichAgentServices(result)
|
||||||
|
}
|
||||||
|
|
||||||
|
func enrichAgentServices(services []AgentService) []AgentService {
|
||||||
|
if len(services) == 0 {
|
||||||
|
return services
|
||||||
|
}
|
||||||
|
args := []string{"show", "--no-pager", "--property=Id", "--property=FragmentPath", "--property=NRestarts", "--property=ExecMainStatus", "--property=ActiveEnterTimestampUSec", "--property=MemoryCurrent", "--property=CPUUsageNSec", "--property=MainPID", "--property=ExecStart", "--property=Requires", "--property=After"}
|
||||||
|
for _, service := range services {
|
||||||
|
args = append(args, service.Name)
|
||||||
|
}
|
||||||
|
show := runAgentCommand(8*time.Second, "systemctl", args...)
|
||||||
|
properties := map[string]map[string]string{}
|
||||||
|
paths := map[string]string{}
|
||||||
|
for _, record := range strings.Split(show, "\n\n") {
|
||||||
|
values := map[string]string{}
|
||||||
|
for _, line := range strings.Split(record, "\n") {
|
||||||
|
key, value, ok := strings.Cut(line, "=")
|
||||||
|
if ok {
|
||||||
|
values[key] = value
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if id := values["Id"]; id != "" {
|
||||||
|
properties[id] = values
|
||||||
|
paths[id] = values["FragmentPath"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
pathArgs := []string{"-S"}
|
||||||
|
for _, service := range services {
|
||||||
|
if path := paths[service.Name]; path != "" {
|
||||||
|
pathArgs = append(pathArgs, path)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
owners := map[string]string{}
|
||||||
|
packages := []string{}
|
||||||
|
if len(pathArgs) > 1 {
|
||||||
|
output := runAgentCommand(8*time.Second, "dpkg-query", pathArgs...)
|
||||||
|
for _, line := range strings.Split(output, "\n") {
|
||||||
|
left, path, ok := strings.Cut(line, ": ")
|
||||||
|
if !ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
pkg := strings.TrimSpace(strings.Split(left, ",")[0])
|
||||||
|
path = strings.TrimSpace(path)
|
||||||
|
if pkg != "" && path != "" {
|
||||||
|
owners[path] = pkg
|
||||||
|
packages = append(packages, pkg)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
priorities := map[string]string{}
|
||||||
|
if len(packages) > 0 {
|
||||||
|
packageArgs := []string{"-W", "-f=${binary:Package}\t${Priority}\n"}
|
||||||
|
packageArgs = append(packageArgs, packages...)
|
||||||
|
output := runAgentCommand(8*time.Second, "dpkg-query", packageArgs...)
|
||||||
|
for _, line := range strings.Split(output, "\n") {
|
||||||
|
fields := strings.Fields(line)
|
||||||
|
if len(fields) >= 2 {
|
||||||
|
priorities[fields[0]] = fields[1]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for i := range services {
|
||||||
|
service := &services[i]
|
||||||
|
service.UnitPath = paths[service.Name]
|
||||||
|
service.Package = owners[service.UnitPath]
|
||||||
|
service.Origin = classifyAgentService(service.UnitPath, service.Package, priorities[service.Package])
|
||||||
|
values := properties[service.Name]
|
||||||
|
service.Restarts, _ = strconv.ParseUint(values["NRestarts"], 10, 64)
|
||||||
|
exit, _ := strconv.ParseInt(values["ExecMainStatus"], 10, 32)
|
||||||
|
service.ExitCode = int(exit)
|
||||||
|
started, _ := strconv.ParseInt(values["ActiveEnterTimestampUSec"], 10, 64)
|
||||||
|
service.StartedAt = started / 1_000_000
|
||||||
|
service.MemoryBytes, _ = strconv.ParseUint(values["MemoryCurrent"], 10, 64)
|
||||||
|
cpu, _ := strconv.ParseUint(values["CPUUsageNSec"], 10, 64)
|
||||||
|
service.CPUSeconds = float64(cpu) / 1_000_000_000
|
||||||
|
pid, _ := strconv.Atoi(values["MainPID"])
|
||||||
|
service.MainPID = pid
|
||||||
|
service.ExecStart = compactSystemdExec(values["ExecStart"])
|
||||||
|
service.Requires = limitedFields(values["Requires"], 12)
|
||||||
|
service.After = limitedFields(values["After"], 12)
|
||||||
|
service.Flapping = service.Restarts >= 5 && service.StartedAt > time.Now().Add(-10*time.Minute).Unix()
|
||||||
|
if service.State == "failed" || service.SubState == "failed" || service.Flapping {
|
||||||
|
service.Logs = collectAgentJournal(service.Name, 8, "-2h")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
ports := collectAgentListenPorts()
|
||||||
|
for i := range services {
|
||||||
|
for _, port := range ports {
|
||||||
|
if services[i].MainPID > 0 && port.PID == services[i].MainPID {
|
||||||
|
services[i].ListenPorts = appendUniqueInt(services[i].ListenPorts, port.Port)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return services
|
||||||
|
}
|
||||||
|
|
||||||
|
func classifyAgentService(path, pkg, priority string) string {
|
||||||
|
switch {
|
||||||
|
case strings.HasPrefix(path, "/etc/systemd/") || strings.HasPrefix(path, "/usr/local/"):
|
||||||
|
return "local"
|
||||||
|
case priority == "required" || priority == "important" || priority == "standard":
|
||||||
|
return "system"
|
||||||
|
case pkg != "":
|
||||||
|
return "added"
|
||||||
|
case strings.HasPrefix(path, "/run/systemd/"):
|
||||||
|
return "runtime"
|
||||||
|
default:
|
||||||
|
return "system"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func collectAgentIssues() []AgentIssue {
|
||||||
|
output := runAgentCommand(8*time.Second, "journalctl", "-p", "err..alert", "--since", "24 hours ago", "--reverse", "-n", "50", "--no-pager", "-o", "json")
|
||||||
|
return parseAgentIssues(output)
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseAgentIssues(output string) []AgentIssue {
|
||||||
|
result := []AgentIssue{}
|
||||||
|
for _, line := range strings.Split(output, "\n") {
|
||||||
|
var entry struct {
|
||||||
|
Message any `json:"MESSAGE"`
|
||||||
|
Unit string `json:"_SYSTEMD_UNIT"`
|
||||||
|
Timestamp string `json:"__REALTIME_TIMESTAMP"`
|
||||||
|
}
|
||||||
|
if json.Unmarshal([]byte(line), &entry) != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
message, ok := entry.Message.(string)
|
||||||
|
if !ok || strings.TrimSpace(message) == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
microseconds, _ := strconv.ParseInt(entry.Timestamp, 10, 64)
|
||||||
|
unit := entry.Unit
|
||||||
|
if unit == "" {
|
||||||
|
unit = "system"
|
||||||
|
}
|
||||||
|
result = append(result, AgentIssue{Time: microseconds / 1_000_000, Unit: unit, Message: message})
|
||||||
|
}
|
||||||
return result
|
return result
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -691,14 +940,23 @@ func collectDockerContainers() ([]AgentContainer, string) {
|
|||||||
inspect, inspectErr := client.Get("http://docker/containers/" + v.ID + "/json")
|
inspect, inspectErr := client.Get("http://docker/containers/" + v.ID + "/json")
|
||||||
if inspectErr == nil {
|
if inspectErr == nil {
|
||||||
var detail struct {
|
var detail struct {
|
||||||
|
Image string
|
||||||
RestartCount int64
|
RestartCount int64
|
||||||
State struct{ Health *struct{ Status string } }
|
State struct{ Health *struct{ Status string } }
|
||||||
|
Config struct {
|
||||||
|
Labels map[string]string
|
||||||
|
}
|
||||||
}
|
}
|
||||||
if json.NewDecoder(inspect.Body).Decode(&detail) == nil {
|
if json.NewDecoder(inspect.Body).Decode(&detail) == nil {
|
||||||
item.Restarts = detail.RestartCount
|
item.Restarts = detail.RestartCount
|
||||||
if detail.State.Health != nil {
|
if detail.State.Health != nil {
|
||||||
item.Health = detail.State.Health.Status
|
item.Health = detail.State.Health.Status
|
||||||
}
|
}
|
||||||
|
item.Compose = detail.Config.Labels["com.docker.compose.service"] != ""
|
||||||
|
item.CurrentVersion, item.CurrentDigest, _ = inspectDockerImage(client, detail.Image, v.Image)
|
||||||
|
var availableID string
|
||||||
|
item.AvailableVersion, item.AvailableDigest, availableID = inspectDockerImage(client, v.Image, v.Image)
|
||||||
|
item.UpdateAvailable = detail.Image != "" && availableID != "" && detail.Image != availableID
|
||||||
}
|
}
|
||||||
inspect.Body.Close()
|
inspect.Body.Close()
|
||||||
}
|
}
|
||||||
@@ -749,6 +1007,46 @@ func collectDockerContainers() ([]AgentContainer, string) {
|
|||||||
}
|
}
|
||||||
return result, ""
|
return result, ""
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func inspectDockerImage(client *http.Client, reference, fallback string) (string, string, string) {
|
||||||
|
if reference == "" {
|
||||||
|
return "", "", ""
|
||||||
|
}
|
||||||
|
response, err := client.Get("http://docker/images/" + url.PathEscape(reference) + "/json")
|
||||||
|
if err != nil {
|
||||||
|
return "", "", ""
|
||||||
|
}
|
||||||
|
defer response.Body.Close()
|
||||||
|
var detail struct {
|
||||||
|
ID string
|
||||||
|
RepoTags []string
|
||||||
|
RepoDigests []string
|
||||||
|
Config struct{ Labels map[string]string }
|
||||||
|
}
|
||||||
|
if response.StatusCode != http.StatusOK || json.NewDecoder(response.Body).Decode(&detail) != nil {
|
||||||
|
return "", "", ""
|
||||||
|
}
|
||||||
|
version := firstNonEmpty(detail.Config.Labels["org.opencontainers.image.version"], detail.Config.Labels["version"], firstString(detail.RepoTags), fallback, shortDockerDigest(detail.ID))
|
||||||
|
digest := ""
|
||||||
|
if len(detail.RepoDigests) > 0 {
|
||||||
|
digest = detail.RepoDigests[0]
|
||||||
|
if _, value, found := strings.Cut(digest, "@"); found {
|
||||||
|
digest = value
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if digest == "" {
|
||||||
|
digest = detail.ID
|
||||||
|
}
|
||||||
|
return version, shortDockerDigest(digest), detail.ID
|
||||||
|
}
|
||||||
|
|
||||||
|
func shortDockerDigest(value string) string {
|
||||||
|
value = strings.TrimPrefix(value, "sha256:")
|
||||||
|
if len(value) > 12 {
|
||||||
|
value = value[:12]
|
||||||
|
}
|
||||||
|
return value
|
||||||
|
}
|
||||||
func firstString(v []string) string {
|
func firstString(v []string) string {
|
||||||
if len(v) > 0 {
|
if len(v) > 0 {
|
||||||
return v[0]
|
return v[0]
|
||||||
|
|||||||
346
agent_actions.go
Normal file
346
agent_actions.go
Normal file
@@ -0,0 +1,346 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
type AgentCommand struct {
|
||||||
|
ID int64 `json:"id"`
|
||||||
|
Action string `json:"action"`
|
||||||
|
Target string `json:"target,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type AgentActionResult struct {
|
||||||
|
ID int64 `json:"id"`
|
||||||
|
Success bool `json:"success"`
|
||||||
|
Message string `json:"message"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type AgentAction struct {
|
||||||
|
ID int64 `json:"id"`
|
||||||
|
AgentID string `json:"agentId"`
|
||||||
|
AgentName string `json:"agentName"`
|
||||||
|
Action string `json:"action"`
|
||||||
|
Target string `json:"target"`
|
||||||
|
RequestedAt int64 `json:"requestedAt"`
|
||||||
|
StartedAt *int64 `json:"startedAt,omitempty"`
|
||||||
|
CompletedAt *int64 `json:"completedAt,omitempty"`
|
||||||
|
Status string `json:"status"`
|
||||||
|
Result string `json:"result"`
|
||||||
|
}
|
||||||
|
|
||||||
|
var safeUnitName = regexp.MustCompile(`^[A-Za-z0-9_.@:-]+\.service$`)
|
||||||
|
var safeContainerName = regexp.MustCompile(`^[A-Za-z0-9_.-]{1,128}$`)
|
||||||
|
|
||||||
|
func (s *Store) QueueAgentAction(agentID, action, target string) (AgentAction, error) {
|
||||||
|
action, target = strings.TrimSpace(action), strings.TrimSpace(target)
|
||||||
|
var raw, name string
|
||||||
|
if err := s.db.QueryRow(`SELECT report_json,name FROM agents WHERE id=?`, agentID).Scan(&raw, &name); err != nil {
|
||||||
|
return AgentAction{}, fmt.Errorf("агент не найден")
|
||||||
|
}
|
||||||
|
var report AgentReport
|
||||||
|
_ = json.Unmarshal([]byte(raw), &report)
|
||||||
|
valid := false
|
||||||
|
switch action {
|
||||||
|
case "systemd-restart", "systemd-reset-failed":
|
||||||
|
if safeUnitName.MatchString(target) {
|
||||||
|
for _, service := range report.Services {
|
||||||
|
if service.Name == target {
|
||||||
|
valid = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
case "docker-restart", "docker-check", "docker-update", "docker-remove":
|
||||||
|
if safeContainerName.MatchString(target) {
|
||||||
|
for _, container := range report.Containers {
|
||||||
|
if (container.ID == target || container.Name == target) && (action != "docker-remove" || container.State != "running") {
|
||||||
|
valid = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
case "updates-refresh", "system-update", "system-reboot":
|
||||||
|
valid = target == ""
|
||||||
|
}
|
||||||
|
if !valid {
|
||||||
|
return AgentAction{}, fmt.Errorf("действие или цель не входят в разрешённый список")
|
||||||
|
}
|
||||||
|
now := time.Now().Unix()
|
||||||
|
result, err := s.db.Exec(`INSERT INTO agent_actions(agent_id,action,target,requested_at,status) VALUES(?,?,?,?, 'queued')`, agentID, action, target, now)
|
||||||
|
if err != nil {
|
||||||
|
return AgentAction{}, err
|
||||||
|
}
|
||||||
|
id, _ := result.LastInsertId()
|
||||||
|
return AgentAction{ID: id, AgentID: agentID, AgentName: name, Action: action, Target: target, RequestedAt: now, Status: "queued"}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Store) DispatchAgentActions(agentID string) ([]AgentCommand, error) {
|
||||||
|
tx, err := s.db.Begin()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer tx.Rollback()
|
||||||
|
// Обновление пакетов и Docker-образов может быть долгим. Не выдаём одну
|
||||||
|
// команду повторно, пока предыдущий запуск ещё имеет шанс завершиться.
|
||||||
|
_, _ = tx.Exec(`UPDATE agent_actions SET status='queued',started_at=NULL WHERE agent_id=? AND status='running' AND started_at<?`, agentID, time.Now().Add(-45*time.Minute).Unix())
|
||||||
|
rows, err := tx.Query(`SELECT id,action,target FROM agent_actions WHERE agent_id=? AND status='queued' ORDER BY id LIMIT 3`, agentID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
var commands []AgentCommand
|
||||||
|
for rows.Next() {
|
||||||
|
var c AgentCommand
|
||||||
|
if err = rows.Scan(&c.ID, &c.Action, &c.Target); err != nil {
|
||||||
|
rows.Close()
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
commands = append(commands, c)
|
||||||
|
}
|
||||||
|
rows.Close()
|
||||||
|
now := time.Now().Unix()
|
||||||
|
for _, c := range commands {
|
||||||
|
if _, err = tx.Exec(`UPDATE agent_actions SET status='running',started_at=? WHERE id=?`, now, c.ID); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return commands, tx.Commit()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Store) CompleteAgentActions(agentID string, results []AgentActionResult) {
|
||||||
|
now := time.Now().Unix()
|
||||||
|
for _, result := range results {
|
||||||
|
status := "failed"
|
||||||
|
if result.Success {
|
||||||
|
status = "completed"
|
||||||
|
}
|
||||||
|
message := result.Message
|
||||||
|
if len(message) > 4000 {
|
||||||
|
message = message[:4000]
|
||||||
|
}
|
||||||
|
_, _ = s.db.Exec(`UPDATE agent_actions SET status=?,result=?,completed_at=? WHERE id=? AND agent_id=? AND status='running'`, status, message, now, result.ID, agentID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Store) AgentActions(limit int) ([]AgentAction, error) {
|
||||||
|
if limit < 1 || limit > 200 {
|
||||||
|
limit = 100
|
||||||
|
}
|
||||||
|
rows, err := s.db.Query(`SELECT a.id,a.agent_id,g.name,a.action,a.target,a.requested_at,a.started_at,a.completed_at,a.status,a.result FROM agent_actions a JOIN agents g ON g.id=a.agent_id ORDER BY a.id DESC LIMIT ?`, limit)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
var out []AgentAction
|
||||||
|
for rows.Next() {
|
||||||
|
var v AgentAction
|
||||||
|
if err = rows.Scan(&v.ID, &v.AgentID, &v.AgentName, &v.Action, &v.Target, &v.RequestedAt, &v.StartedAt, &v.CompletedAt, &v.Status, &v.Result); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out = append(out, v)
|
||||||
|
}
|
||||||
|
return out, rows.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
func executeAgentCommand(command AgentCommand) AgentActionResult {
|
||||||
|
result := AgentActionResult{ID: command.ID}
|
||||||
|
var output string
|
||||||
|
switch command.Action {
|
||||||
|
case "systemd-restart":
|
||||||
|
output = runAgentCombinedCommand(30*time.Second, "systemctl", "restart", command.Target)
|
||||||
|
case "systemd-reset-failed":
|
||||||
|
output = runAgentCombinedCommand(15*time.Second, "systemctl", "reset-failed", command.Target)
|
||||||
|
case "docker-restart":
|
||||||
|
output = restartAgentContainer(command.Target)
|
||||||
|
case "docker-update":
|
||||||
|
output = updateAgentContainer(command.Target, true)
|
||||||
|
case "docker-check":
|
||||||
|
output = updateAgentContainer(command.Target, false)
|
||||||
|
case "docker-remove":
|
||||||
|
output = removeAgentContainer(command.Target)
|
||||||
|
case "system-update":
|
||||||
|
output = updateAgentSystem()
|
||||||
|
case "system-reboot":
|
||||||
|
output = scheduleAgentReboot()
|
||||||
|
case "updates-refresh":
|
||||||
|
guestInventory.Lock()
|
||||||
|
guestInventory.value.CollectedAt = time.Time{}
|
||||||
|
guestInventory.Unlock()
|
||||||
|
result.Success = true
|
||||||
|
result.Message = "Проверка обновлений поставлена в очередь"
|
||||||
|
return result
|
||||||
|
default:
|
||||||
|
result.Message = "Команда не разрешена"
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
result.Success = !strings.HasPrefix(output, "ERROR: ")
|
||||||
|
result.Message = strings.TrimSpace(output)
|
||||||
|
if result.Message == "" && result.Success {
|
||||||
|
result.Message = "Выполнено успешно"
|
||||||
|
}
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
func removeAgentContainer(target string) string {
|
||||||
|
transport := &http.Transport{DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
|
||||||
|
return (&net.Dialer{}).DialContext(ctx, "unix", "/var/run/docker.sock")
|
||||||
|
}}
|
||||||
|
defer transport.CloseIdleConnections()
|
||||||
|
request, _ := http.NewRequest(http.MethodDelete, "http://docker/containers/"+target+"?v=false&force=false", nil)
|
||||||
|
response, err := (&http.Client{Transport: transport, Timeout: 20 * time.Second}).Do(request)
|
||||||
|
if err != nil {
|
||||||
|
return "ERROR: " + err.Error()
|
||||||
|
}
|
||||||
|
defer response.Body.Close()
|
||||||
|
if response.StatusCode != http.StatusNoContent {
|
||||||
|
message, _ := io.ReadAll(io.LimitReader(response.Body, 2048))
|
||||||
|
return fmt.Sprintf("ERROR: Docker вернул %s: %s", response.Status, strings.TrimSpace(string(message)))
|
||||||
|
}
|
||||||
|
return "Остановленный контейнер удалён; volumes сохранены"
|
||||||
|
}
|
||||||
|
|
||||||
|
func updateAgentSystem() string {
|
||||||
|
aptGet, err := exec.LookPath("apt-get")
|
||||||
|
if err != nil {
|
||||||
|
return "ERROR: apt-get не найден; обновление поддерживается для Debian/Ubuntu"
|
||||||
|
}
|
||||||
|
if output := runPrivilegedTransient(5*time.Minute, "update-lists", aptGet, "update"); strings.HasPrefix(output, "ERROR: ") {
|
||||||
|
return explainSystemUpdateError(output)
|
||||||
|
}
|
||||||
|
message := runPrivilegedTransient(30*time.Minute, "upgrade", aptGet, "-y", "upgrade")
|
||||||
|
if strings.HasPrefix(message, "ERROR: ") {
|
||||||
|
return explainSystemUpdateError(message)
|
||||||
|
}
|
||||||
|
guestInventory.Lock()
|
||||||
|
guestInventory.value.CollectedAt = time.Time{}
|
||||||
|
guestInventory.Unlock()
|
||||||
|
return "Обновления системы установлены; список пакетов будет обновлён в следующем отчёте агента"
|
||||||
|
}
|
||||||
|
|
||||||
|
func explainSystemUpdateError(message string) string {
|
||||||
|
if strings.Contains(strings.ToLower(message), "read-only file system") {
|
||||||
|
return "ERROR: файловая система агента доступна только для чтения. Обновите Dashboard и агент до v0.19.2 или новее, затем повторите операцию."
|
||||||
|
}
|
||||||
|
if len(message) > 1800 {
|
||||||
|
message = message[len(message)-1800:]
|
||||||
|
}
|
||||||
|
return message
|
||||||
|
}
|
||||||
|
|
||||||
|
func runPrivilegedTransient(timeout time.Duration, suffix, name string, args ...string) string {
|
||||||
|
systemdRun, err := exec.LookPath("systemd-run")
|
||||||
|
if err != nil {
|
||||||
|
return "ERROR: systemd-run не найден; невозможно безопасно выйти из read-only sandbox агента"
|
||||||
|
}
|
||||||
|
unit := fmt.Sprintf("proxmox-dashboard-%s-%d", suffix, time.Now().UnixNano())
|
||||||
|
commandArgs := []string{"--quiet", "--wait", "--collect", "--pipe", "--unit=" + unit, "--setenv=DEBIAN_FRONTEND=noninteractive", name}
|
||||||
|
commandArgs = append(commandArgs, args...)
|
||||||
|
return runAgentCombinedCommand(timeout, systemdRun, commandArgs...)
|
||||||
|
}
|
||||||
|
|
||||||
|
func scheduleAgentReboot() string {
|
||||||
|
systemdRun, err := exec.LookPath("systemd-run")
|
||||||
|
if err != nil {
|
||||||
|
return "ERROR: systemd-run не найден"
|
||||||
|
}
|
||||||
|
systemctl, err := exec.LookPath("systemctl")
|
||||||
|
if err != nil {
|
||||||
|
return "ERROR: systemctl не найден"
|
||||||
|
}
|
||||||
|
unit := fmt.Sprintf("proxmox-dashboard-reboot-%d", time.Now().UnixNano())
|
||||||
|
output := runAgentCombinedCommand(15*time.Second, systemdRun, "--quiet", "--collect", "--unit="+unit, "--on-active=20s", systemctl, "reboot")
|
||||||
|
if strings.HasPrefix(output, "ERROR: ") {
|
||||||
|
return output
|
||||||
|
}
|
||||||
|
return "Перезагрузка запланирована через 20 секунд; кратковременный offline агента ожидаем"
|
||||||
|
}
|
||||||
|
|
||||||
|
func updateAgentContainer(target string, recreate bool) string {
|
||||||
|
transport := &http.Transport{DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
|
||||||
|
return (&net.Dialer{}).DialContext(ctx, "unix", "/var/run/docker.sock")
|
||||||
|
}}
|
||||||
|
defer transport.CloseIdleConnections()
|
||||||
|
client := &http.Client{Transport: transport, Timeout: 10 * time.Second}
|
||||||
|
response, err := client.Get("http://docker/containers/" + target + "/json")
|
||||||
|
if err != nil {
|
||||||
|
return "ERROR: " + err.Error()
|
||||||
|
}
|
||||||
|
defer response.Body.Close()
|
||||||
|
var detail struct {
|
||||||
|
Config struct {
|
||||||
|
Labels map[string]string `json:"Labels"`
|
||||||
|
} `json:"Config"`
|
||||||
|
}
|
||||||
|
if response.StatusCode != http.StatusOK || json.NewDecoder(response.Body).Decode(&detail) != nil {
|
||||||
|
return "ERROR: не удалось получить Compose-конфигурацию контейнера"
|
||||||
|
}
|
||||||
|
labels := detail.Config.Labels
|
||||||
|
service := labels["com.docker.compose.service"]
|
||||||
|
workdir := labels["com.docker.compose.project.working_dir"]
|
||||||
|
files := strings.Split(labels["com.docker.compose.project.config_files"], ",")
|
||||||
|
if service == "" || workdir == "" || len(files) == 0 || !filepath.IsAbs(workdir) || !filepath.IsAbs(files[0]) {
|
||||||
|
return "ERROR: безопасное обновление доступно только для Docker Compose"
|
||||||
|
}
|
||||||
|
if _, err = exec.LookPath("docker"); err != nil {
|
||||||
|
return "ERROR: Docker CLI не найден"
|
||||||
|
}
|
||||||
|
commands := [][]string{{"compose", "-f", files[0], "pull", service}}
|
||||||
|
if recreate {
|
||||||
|
commands = append(commands, []string{"compose", "-f", files[0], "up", "-d", "--no-deps", service})
|
||||||
|
}
|
||||||
|
for _, args := range commands {
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Minute)
|
||||||
|
command := exec.CommandContext(ctx, "docker", args...)
|
||||||
|
command.Dir = workdir
|
||||||
|
output, commandErr := command.CombinedOutput()
|
||||||
|
cancel()
|
||||||
|
if commandErr != nil {
|
||||||
|
return "ERROR: " + commandErr.Error() + ": " + strings.TrimSpace(string(output))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if recreate {
|
||||||
|
return "Образ загружен, Compose-контейнер пересоздан"
|
||||||
|
}
|
||||||
|
return "Проверка завершена: образ загружен; если digest изменился, карточка покажет доступное обновление"
|
||||||
|
}
|
||||||
|
|
||||||
|
func runAgentCombinedCommand(timeout time.Duration, name string, args ...string) string {
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), timeout)
|
||||||
|
defer cancel()
|
||||||
|
output, err := exec.CommandContext(ctx, name, args...).CombinedOutput()
|
||||||
|
message := strings.TrimSpace(string(output))
|
||||||
|
if err != nil {
|
||||||
|
return "ERROR: " + err.Error() + ": " + message
|
||||||
|
}
|
||||||
|
return message
|
||||||
|
}
|
||||||
|
|
||||||
|
func restartAgentContainer(target string) string {
|
||||||
|
transport := &http.Transport{DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
|
||||||
|
return (&net.Dialer{}).DialContext(ctx, "unix", "/var/run/docker.sock")
|
||||||
|
}}
|
||||||
|
defer transport.CloseIdleConnections()
|
||||||
|
client := &http.Client{Transport: transport, Timeout: 35 * time.Second}
|
||||||
|
request, _ := http.NewRequest(http.MethodPost, "http://docker/containers/"+target+"/restart?t=10", bytes.NewReader(nil))
|
||||||
|
response, err := client.Do(request)
|
||||||
|
if err != nil {
|
||||||
|
return "ERROR: " + err.Error()
|
||||||
|
}
|
||||||
|
defer response.Body.Close()
|
||||||
|
body, _ := io.ReadAll(io.LimitReader(response.Body, 4096))
|
||||||
|
if response.StatusCode >= 300 {
|
||||||
|
return "ERROR: Docker вернул " + response.Status + ": " + strings.TrimSpace(string(body))
|
||||||
|
}
|
||||||
|
return "Контейнер перезапущен"
|
||||||
|
}
|
||||||
77
agent_actions_test.go
Normal file
77
agent_actions_test.go
Normal file
@@ -0,0 +1,77 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestAgentActionAllowlistAndLifecycle(t *testing.T) {
|
||||||
|
store, err := openStore(":memory:")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer store.Close()
|
||||||
|
report, _ := json.Marshal(AgentReport{Services: []AgentService{{Name: "gitea.service"}}, Containers: []AgentContainer{{ID: "abc123", Name: "gitea", State: "running"}, {ID: "old123", Name: "old", State: "exited"}}})
|
||||||
|
_, err = store.db.Exec(`INSERT INTO agents(id,name,secret_hash,enrolled_at,last_seen,report_json) VALUES('a','agent','x',?,?,?)`, time.Now().Unix(), time.Now().Unix(), string(report))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err = store.QueueAgentAction("a", "systemd-restart", "../../bad.service"); err == nil {
|
||||||
|
t.Fatal("unsafe unit accepted")
|
||||||
|
}
|
||||||
|
if _, err = store.QueueAgentAction("a", "docker-update", "gitea"); err != nil {
|
||||||
|
t.Fatalf("known Docker container rejected: %v", err)
|
||||||
|
}
|
||||||
|
if _, err = store.QueueAgentAction("a", "system-update", ""); err != nil {
|
||||||
|
t.Fatalf("system update rejected: %v", err)
|
||||||
|
}
|
||||||
|
if _, err = store.QueueAgentAction("a", "docker-remove", "gitea"); err == nil {
|
||||||
|
t.Fatal("running/unknown-state container accepted for removal")
|
||||||
|
}
|
||||||
|
action, err := store.QueueAgentAction("a", "docker-remove", "old")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("stopped container rejected for removal: %v", err)
|
||||||
|
}
|
||||||
|
commands, err := store.DispatchAgentActions("a")
|
||||||
|
if err != nil || len(commands) != 3 || commands[2].ID != action.ID {
|
||||||
|
t.Fatalf("commands=%#v err=%v", commands, err)
|
||||||
|
}
|
||||||
|
store.CompleteAgentActions("a", []AgentActionResult{{ID: action.ID, Success: true, Message: "ok"}})
|
||||||
|
items, err := store.AgentActions(10)
|
||||||
|
if err != nil || len(items) != 3 || items[0].Status != "completed" {
|
||||||
|
t.Fatalf("items=%#v err=%v", items, err)
|
||||||
|
}
|
||||||
|
if _, err = store.QueueAgentAction("a", "system-reboot", ""); err != nil {
|
||||||
|
t.Fatalf("system reboot rejected: %v", err)
|
||||||
|
}
|
||||||
|
if _, err = store.QueueAgentAction("a", "system-reboot", "now"); err == nil {
|
||||||
|
t.Fatal("system reboot with unexpected target accepted")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestManualServiceLinkGroupsOfflineAlerts(t *testing.T) {
|
||||||
|
metrics := dashboardMetrics{Agents: []ManagedAgent{{ID: "a", Name: "app", VMID: 105, Online: false}}, ServiceLinks: []ServiceLink{{ServiceID: 7, AgentID: "a", VMID: 105}}, Services: ServicesMetrics{Services: []ServiceStatus{{MonitoredService: MonitoredService{ID: 7, Name: "Gitea"}, Endpoints: []ServiceEndpointStatus{{Kind: "local", Up: false}}}}}}
|
||||||
|
alerts := []Alert{{ID: "agent-offline-a-105"}, {ID: "service-7-local"}, {ID: "other"}}
|
||||||
|
grouped := groupLinkedAlerts(metrics, alerts)
|
||||||
|
if len(grouped) != 2 {
|
||||||
|
t.Fatalf("grouped=%#v", grouped)
|
||||||
|
}
|
||||||
|
found := false
|
||||||
|
for _, alert := range grouped {
|
||||||
|
if alert.ID == "group-agent-offline-a-105" {
|
||||||
|
found = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
t.Fatalf("group alert missing: %#v", grouped)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExplainReadOnlySystemUpdateError(t *testing.T) {
|
||||||
|
message := explainSystemUpdateError("ERROR: apt failed: Read-only file system")
|
||||||
|
if !strings.Contains(message, "v0.19.2") {
|
||||||
|
t.Fatalf("read-only failure is not actionable: %s", message)
|
||||||
|
}
|
||||||
|
}
|
||||||
336
agent_diagnostics.go
Normal file
336
agent_diagnostics.go
Normal file
@@ -0,0 +1,336 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"encoding/json"
|
||||||
|
"net"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
|
"runtime"
|
||||||
|
"sort"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"syscall"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
type AgentLog struct {
|
||||||
|
Time int64 `json:"time"`
|
||||||
|
Message string `json:"message"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type AgentProcess struct {
|
||||||
|
PID int `json:"pid"`
|
||||||
|
Name string `json:"name"`
|
||||||
|
State string `json:"state"`
|
||||||
|
CPUPercent float64 `json:"cpuPercent"`
|
||||||
|
CPUCores float64 `json:"cpuCores"`
|
||||||
|
HostPercent float64 `json:"hostPercent"`
|
||||||
|
MemoryBytes uint64 `json:"memoryBytes"`
|
||||||
|
ReadBytes uint64 `json:"readBytes"`
|
||||||
|
WriteBytes uint64 `json:"writeBytes"`
|
||||||
|
Elapsed int64 `json:"elapsedSeconds"`
|
||||||
|
Command string `json:"command"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type AgentFilesystem struct {
|
||||||
|
Device string `json:"device"`
|
||||||
|
Mountpoint string `json:"mountpoint"`
|
||||||
|
Type string `json:"type"`
|
||||||
|
TotalBytes uint64 `json:"totalBytes"`
|
||||||
|
UsedBytes uint64 `json:"usedBytes"`
|
||||||
|
UsedPercent float64 `json:"usedPercent"`
|
||||||
|
InodesTotal uint64 `json:"inodesTotal"`
|
||||||
|
InodesUsed uint64 `json:"inodesUsed"`
|
||||||
|
InodesPercent float64 `json:"inodesPercent"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type AgentListenPort struct {
|
||||||
|
Protocol string `json:"protocol"`
|
||||||
|
Address string `json:"address"`
|
||||||
|
Port int `json:"port"`
|
||||||
|
PID int `json:"pid,omitempty"`
|
||||||
|
Process string `json:"process,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type AgentPressureValue struct {
|
||||||
|
Avg10 float64 `json:"avg10"`
|
||||||
|
Avg60 float64 `json:"avg60"`
|
||||||
|
Avg300 float64 `json:"avg300"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type AgentPressure struct {
|
||||||
|
CPU AgentPressureValue `json:"cpu"`
|
||||||
|
Memory AgentPressureValue `json:"memory"`
|
||||||
|
IO AgentPressureValue `json:"io"`
|
||||||
|
}
|
||||||
|
|
||||||
|
var agentOOMCache struct {
|
||||||
|
sync.Mutex
|
||||||
|
value []AgentLog
|
||||||
|
at time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
func compactSystemdExec(value string) string {
|
||||||
|
if pathAt := strings.Index(value, "path="); pathAt >= 0 {
|
||||||
|
value = value[pathAt+5:]
|
||||||
|
}
|
||||||
|
if end := strings.IndexAny(value, " ;}"); end > 0 {
|
||||||
|
value = value[:end]
|
||||||
|
}
|
||||||
|
if len(value) > 240 {
|
||||||
|
value = value[:240]
|
||||||
|
}
|
||||||
|
return strings.TrimSpace(value)
|
||||||
|
}
|
||||||
|
|
||||||
|
func limitedFields(value string, limit int) []string {
|
||||||
|
fields := strings.Fields(value)
|
||||||
|
if len(fields) > limit {
|
||||||
|
fields = fields[:limit]
|
||||||
|
}
|
||||||
|
return fields
|
||||||
|
}
|
||||||
|
|
||||||
|
func appendUniqueInt(items []int, value int) []int {
|
||||||
|
for _, item := range items {
|
||||||
|
if item == value {
|
||||||
|
return items
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return append(items, value)
|
||||||
|
}
|
||||||
|
|
||||||
|
func collectAgentJournal(unit string, limit int, since string) []AgentLog {
|
||||||
|
args := []string{"--no-pager", "-o", "json", "--reverse", "-n", strconv.Itoa(limit), "--since", since}
|
||||||
|
if unit != "" {
|
||||||
|
args = append(args, "-u", unit)
|
||||||
|
}
|
||||||
|
output := runAgentCommand(6*time.Second, "journalctl", args...)
|
||||||
|
result := []AgentLog{}
|
||||||
|
for _, line := range strings.Split(output, "\n") {
|
||||||
|
var entry struct {
|
||||||
|
Message any `json:"MESSAGE"`
|
||||||
|
Timestamp string `json:"__REALTIME_TIMESTAMP"`
|
||||||
|
}
|
||||||
|
if json.Unmarshal([]byte(line), &entry) != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
message, ok := entry.Message.(string)
|
||||||
|
if !ok || strings.TrimSpace(message) == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
microseconds, _ := strconv.ParseInt(entry.Timestamp, 10, 64)
|
||||||
|
result = append(result, AgentLog{Time: microseconds / 1_000_000, Message: message})
|
||||||
|
}
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
func cachedAgentOOMKills() []AgentLog {
|
||||||
|
agentOOMCache.Lock()
|
||||||
|
defer agentOOMCache.Unlock()
|
||||||
|
if time.Since(agentOOMCache.at) < time.Minute {
|
||||||
|
return agentOOMCache.value
|
||||||
|
}
|
||||||
|
output := runAgentCommand(6*time.Second, "journalctl", "-k", "--since", "24 hours ago", "--reverse", "--no-pager", "-o", "json", "-g", "Out of memory|Killed process")
|
||||||
|
result := []AgentLog{}
|
||||||
|
for _, line := range strings.Split(output, "\n") {
|
||||||
|
var entry struct {
|
||||||
|
Message any `json:"MESSAGE"`
|
||||||
|
Timestamp string `json:"__REALTIME_TIMESTAMP"`
|
||||||
|
}
|
||||||
|
if json.Unmarshal([]byte(line), &entry) != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
message, ok := entry.Message.(string)
|
||||||
|
if !ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
microseconds, _ := strconv.ParseInt(entry.Timestamp, 10, 64)
|
||||||
|
result = append(result, AgentLog{Time: microseconds / 1_000_000, Message: message})
|
||||||
|
if len(result) >= 20 {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
agentOOMCache.value, agentOOMCache.at = result, time.Now()
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
func collectAgentProcesses() []AgentProcess {
|
||||||
|
result := []AgentProcess{}
|
||||||
|
seen := map[int]bool{}
|
||||||
|
for _, sortBy := range []string{"-pcpu", "-rss"} {
|
||||||
|
output := runAgentCommand(5*time.Second, "ps", "-eo", "pid=,comm=,state=,pcpu=,rss=,etimes=,args=", "--sort="+sortBy)
|
||||||
|
added := 0
|
||||||
|
for _, line := range strings.Split(output, "\n") {
|
||||||
|
fields := strings.Fields(line)
|
||||||
|
if len(fields) < 7 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
pid, err := strconv.Atoi(fields[0])
|
||||||
|
if err != nil || seen[pid] {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen[pid] = true
|
||||||
|
cpu, _ := strconv.ParseFloat(fields[3], 64)
|
||||||
|
rss, _ := strconv.ParseUint(fields[4], 10, 64)
|
||||||
|
elapsed, _ := strconv.ParseInt(fields[5], 10, 64)
|
||||||
|
cpuCount := runtime.NumCPU()
|
||||||
|
if cpuCount < 1 {
|
||||||
|
cpuCount = 1
|
||||||
|
}
|
||||||
|
item := AgentProcess{PID: pid, Name: fields[1], State: fields[2], CPUPercent: cpu, CPUCores: cpu / 100, HostPercent: cpu / float64(cpuCount), MemoryBytes: rss * 1024, Elapsed: elapsed, Command: strings.Join(fields[6:], " ")}
|
||||||
|
item.ReadBytes, item.WriteBytes = readAgentProcessIO(pid)
|
||||||
|
result = append(result, item)
|
||||||
|
added++
|
||||||
|
if added >= 12 || len(result) >= 24 {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
func readAgentProcessIO(pid int) (uint64, uint64) {
|
||||||
|
data, err := os.ReadFile(filepath.Join("/proc", strconv.Itoa(pid), "io"))
|
||||||
|
if err != nil {
|
||||||
|
return 0, 0
|
||||||
|
}
|
||||||
|
values := map[string]uint64{}
|
||||||
|
for _, line := range strings.Split(string(data), "\n") {
|
||||||
|
fields := strings.Fields(line)
|
||||||
|
if len(fields) != 2 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
values[strings.TrimSuffix(fields[0], ":")], _ = strconv.ParseUint(fields[1], 10, 64)
|
||||||
|
}
|
||||||
|
return values["read_bytes"], values["write_bytes"]
|
||||||
|
}
|
||||||
|
|
||||||
|
func collectAgentFilesystems() []AgentFilesystem {
|
||||||
|
file, err := os.Open("/proc/self/mounts")
|
||||||
|
if err != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
defer file.Close()
|
||||||
|
excluded := map[string]bool{"proc": true, "sysfs": true, "devtmpfs": true, "devpts": true, "tmpfs": true, "cgroup": true, "cgroup2": true, "overlay": true, "squashfs": true, "nsfs": true, "tracefs": true, "debugfs": true, "securityfs": true, "pstore": true, "mqueue": true, "hugetlbfs": true, "fusectl": true, "configfs": true, "autofs": true, "rpc_pipefs": true}
|
||||||
|
seen := map[string]bool{}
|
||||||
|
result := []AgentFilesystem{}
|
||||||
|
scanner := bufio.NewScanner(file)
|
||||||
|
for scanner.Scan() {
|
||||||
|
fields := strings.Fields(scanner.Text())
|
||||||
|
if len(fields) < 3 || excluded[fields[2]] || seen[fields[1]] {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
mountpoint := strings.ReplaceAll(fields[1], `\040`, " ")
|
||||||
|
var stat syscall.Statfs_t
|
||||||
|
if syscall.Statfs(mountpoint, &stat) != nil || stat.Blocks == 0 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen[mountpoint] = true
|
||||||
|
total := stat.Blocks * uint64(stat.Bsize)
|
||||||
|
free := stat.Bavail * uint64(stat.Bsize)
|
||||||
|
used := total - free
|
||||||
|
item := AgentFilesystem{Device: fields[0], Mountpoint: mountpoint, Type: fields[2], TotalBytes: total, UsedBytes: used, UsedPercent: float64(used) / float64(total) * 100, InodesTotal: stat.Files}
|
||||||
|
if stat.Files > stat.Ffree {
|
||||||
|
item.InodesUsed = stat.Files - stat.Ffree
|
||||||
|
item.InodesPercent = float64(item.InodesUsed) / float64(stat.Files) * 100
|
||||||
|
}
|
||||||
|
result = append(result, item)
|
||||||
|
if len(result) >= 30 {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sort.Slice(result, func(i, j int) bool { return result[i].Mountpoint < result[j].Mountpoint })
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
var ssPIDPattern = regexp.MustCompile(`pid=([0-9]+)`)
|
||||||
|
var ssNamePattern = regexp.MustCompile(`users:\(\(\"([^\"]+)\"`)
|
||||||
|
|
||||||
|
func collectAgentListenPorts() []AgentListenPort {
|
||||||
|
output := runAgentCommand(5*time.Second, "ss", "-H", "-lntup")
|
||||||
|
result := []AgentListenPort{}
|
||||||
|
seen := map[string]bool{}
|
||||||
|
for _, line := range strings.Split(output, "\n") {
|
||||||
|
fields := strings.Fields(line)
|
||||||
|
if len(fields) < 5 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
protocol := strings.ToLower(fields[0])
|
||||||
|
local := ""
|
||||||
|
for _, field := range fields[1:] {
|
||||||
|
if strings.Contains(field, ":") && !strings.Contains(field, "users:") {
|
||||||
|
local = field
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if local == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
address, portText, err := net.SplitHostPort(local)
|
||||||
|
if err != nil {
|
||||||
|
idx := strings.LastIndex(local, ":")
|
||||||
|
if idx < 0 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
address, portText = local[:idx], local[idx+1:]
|
||||||
|
}
|
||||||
|
port, _ := strconv.Atoi(portText)
|
||||||
|
if port <= 0 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
pid := 0
|
||||||
|
if match := ssPIDPattern.FindStringSubmatch(line); len(match) > 1 {
|
||||||
|
pid, _ = strconv.Atoi(match[1])
|
||||||
|
}
|
||||||
|
process := ""
|
||||||
|
if match := ssNamePattern.FindStringSubmatch(line); len(match) > 1 {
|
||||||
|
process = match[1]
|
||||||
|
}
|
||||||
|
key := protocol + "|" + address + "|" + portText + "|" + strconv.Itoa(pid)
|
||||||
|
if seen[key] {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen[key] = true
|
||||||
|
result = append(result, AgentListenPort{Protocol: protocol, Address: address, Port: port, PID: pid, Process: process})
|
||||||
|
}
|
||||||
|
sort.Slice(result, func(i, j int) bool {
|
||||||
|
if result[i].Port == result[j].Port {
|
||||||
|
return result[i].Protocol < result[j].Protocol
|
||||||
|
}
|
||||||
|
return result[i].Port < result[j].Port
|
||||||
|
})
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
func collectAgentPressure() AgentPressure {
|
||||||
|
return AgentPressure{CPU: readPressure("/proc/pressure/cpu"), Memory: readPressure("/proc/pressure/memory"), IO: readPressure("/proc/pressure/io")}
|
||||||
|
}
|
||||||
|
|
||||||
|
func readPressure(path string) AgentPressureValue {
|
||||||
|
data, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
return AgentPressureValue{}
|
||||||
|
}
|
||||||
|
line := strings.SplitN(string(data), "\n", 2)[0]
|
||||||
|
value := AgentPressureValue{}
|
||||||
|
for _, field := range strings.Fields(line) {
|
||||||
|
key, raw, ok := strings.Cut(field, "=")
|
||||||
|
if !ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
number, _ := strconv.ParseFloat(raw, 64)
|
||||||
|
switch key {
|
||||||
|
case "avg10":
|
||||||
|
value.Avg10 = number
|
||||||
|
case "avg60":
|
||||||
|
value.Avg60 = number
|
||||||
|
case "avg300":
|
||||||
|
value.Avg300 = number
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return value
|
||||||
|
}
|
||||||
@@ -53,6 +53,43 @@ func TestAgentOnlyInstallsNewerVersion(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestShortDockerDigest(t *testing.T) {
|
||||||
|
if got := shortDockerDigest("sha256:1234567890abcdef"); got != "1234567890ab" {
|
||||||
|
t.Fatalf("unexpected short digest: %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseAgentIssues(t *testing.T) {
|
||||||
|
issues := parseAgentIssues(`{"MESSAGE":"disk error","_SYSTEMD_UNIT":"worker.service","__REALTIME_TIMESTAMP":"1722960000000000"}` + "\n" + `{"MESSAGE":"network error","__REALTIME_TIMESTAMP":"1722960001000000"}`)
|
||||||
|
if len(issues) != 2 || issues[0].Unit != "worker.service" || issues[1].Unit != "system" || issues[0].Time != 1722960000 {
|
||||||
|
t.Fatalf("unexpected issues: %#v", issues)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestClassifyAgentService(t *testing.T) {
|
||||||
|
cases := []struct{ path, pkg, priority, want string }{{"/lib/systemd/system/cron.service", "cron", "important", "system"}, {"/lib/systemd/system/jellyfin.service", "jellyfin", "optional", "added"}, {"/etc/systemd/system/my.service", "", "", "local"}, {"/run/systemd/system/transient.service", "", "", "runtime"}}
|
||||||
|
for _, tc := range cases {
|
||||||
|
if got := classifyAgentService(tc.path, tc.pkg, tc.priority); got != tc.want {
|
||||||
|
t.Errorf("%s: got %s want %s", tc.path, got, tc.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAgentServiceDiagnosticsHelpers(t *testing.T) {
|
||||||
|
if got := compactSystemdExec(`{ path=/usr/bin/example ; argv[]=/usr/bin/example --serve ; }`); got != "/usr/bin/example" {
|
||||||
|
t.Fatalf("unexpected executable: %q", got)
|
||||||
|
}
|
||||||
|
fields := limitedFields("a b c d", 2)
|
||||||
|
if len(fields) != 2 || fields[0] != "a" || fields[1] != "b" {
|
||||||
|
t.Fatalf("unexpected limited fields: %#v", fields)
|
||||||
|
}
|
||||||
|
ports := appendUniqueInt([]int{80}, 80)
|
||||||
|
ports = appendUniqueInt(ports, 443)
|
||||||
|
if len(ports) != 2 || ports[1] != 443 {
|
||||||
|
t.Fatalf("unexpected ports: %#v", ports)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestAgentReportAuthenticationAndOfflineState(t *testing.T) {
|
func TestAgentReportAuthenticationAndOfflineState(t *testing.T) {
|
||||||
store, err := openStore(":memory:")
|
store, err := openStore(":memory:")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -94,3 +131,22 @@ func TestAgentProblemsBecomeMaintenanceAwareAlerts(t *testing.T) {
|
|||||||
t.Fatalf("active=%d planned=%d alerts=%#v", len(active), len(planned), alerts)
|
t.Fatalf("active=%d planned=%d alerts=%#v", len(active), len(planned), alerts)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestAgentDiagnosticsBecomeAlerts(t *testing.T) {
|
||||||
|
metrics := dashboardMetrics{Agents: []ManagedAgent{{ID: "node", Name: "app", VMID: 105, Online: true, Report: AgentReport{
|
||||||
|
Services: []AgentService{{Name: "web.service", State: "active", Flapping: true, Restarts: 8, StartedAt: time.Now().Unix()}},
|
||||||
|
Filesystems: []AgentFilesystem{{Mountpoint: "/data", TotalBytes: 1000, UsedBytes: 950, UsedPercent: 95}},
|
||||||
|
}}}}
|
||||||
|
alerts := evaluateAlerts(metrics, defaultThresholds())
|
||||||
|
wanted := map[string]bool{"agent-flapping-node-web.service-105": false, "agent-filesystem-node-/data-105": false}
|
||||||
|
for _, alert := range alerts {
|
||||||
|
if _, ok := wanted[alert.ID]; ok {
|
||||||
|
wanted[alert.ID] = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for id, found := range wanted {
|
||||||
|
if !found {
|
||||||
|
t.Fatalf("missing alert %s in %#v", id, alerts)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
387
agent_traffic.go
Normal file
387
agent_traffic.go
Normal file
@@ -0,0 +1,387 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
|
"sort"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"syscall"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
var npmTrafficPattern = regexp.MustCompile(`^\[([^]]+)\] .*? (\d{3}) - ([A-Z]+) (?:https?|HTTP) (\S+) "([^"]*)" \[Client ([^]]+)\]`)
|
||||||
|
|
||||||
|
func parseNPMTraffic(line string) (TrafficEvent, bool) {
|
||||||
|
m := npmTrafficPattern.FindStringSubmatch(line)
|
||||||
|
if m == nil {
|
||||||
|
return TrafficEvent{}, false
|
||||||
|
}
|
||||||
|
ip := net.ParseIP(m[6])
|
||||||
|
if ip == nil {
|
||||||
|
return TrafficEvent{}, false
|
||||||
|
}
|
||||||
|
ts, err := time.Parse("02/Jan/2006:15:04:05 -0700", m[1])
|
||||||
|
if err != nil {
|
||||||
|
return TrafficEvent{}, false
|
||||||
|
}
|
||||||
|
status, _ := strconv.Atoi(m[2])
|
||||||
|
return TrafficEvent{Time: ts.Unix(), Kind: "request", IP: ip.String(), Domain: trafficText(m[4], 253), Method: m[3], Path: trafficPath(m[5]), Status: status}, true
|
||||||
|
}
|
||||||
|
|
||||||
|
type trafficCursor struct {
|
||||||
|
Offset int64 `json:"offset"`
|
||||||
|
Generation int64 `json:"generation"`
|
||||||
|
Seen int64 `json:"seen"`
|
||||||
|
}
|
||||||
|
type trafficAgentState struct {
|
||||||
|
BacklogBytes int64 `json:"backlogBytes"`
|
||||||
|
Initialized bool `json:"initialized"`
|
||||||
|
StartedAt int64 `json:"startedAt"`
|
||||||
|
Cursors map[string]trafficCursor `json:"cursors"`
|
||||||
|
Pending []TrafficEvent `json:"pending"`
|
||||||
|
ParseErrors uint64 `json:"parseErrors"`
|
||||||
|
CrowdSeen map[string]int64 `json:"crowdSeen"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func (state *trafficAgentState) readLogs(dir string) (int, error) {
|
||||||
|
state.BacklogBytes = 0
|
||||||
|
paths, err := filepath.Glob(filepath.Join(dir, "*_access.log*"))
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
sort.Strings(paths)
|
||||||
|
files := 0
|
||||||
|
budget := int64(1 << 20)
|
||||||
|
now := time.Now().Unix()
|
||||||
|
var firstErr error
|
||||||
|
for _, path := range paths {
|
||||||
|
if strings.HasSuffix(path, ".gz") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
f, err := os.Open(path)
|
||||||
|
if err != nil {
|
||||||
|
firstErr = err
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
info, err := f.Stat()
|
||||||
|
if err != nil || !info.Mode().IsRegular() {
|
||||||
|
f.Close()
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
st, ok := info.Sys().(*syscall.Stat_t)
|
||||||
|
if !ok {
|
||||||
|
f.Close()
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
active := strings.HasSuffix(path, "_access.log")
|
||||||
|
if active {
|
||||||
|
files++
|
||||||
|
}
|
||||||
|
key := fmt.Sprintf("%d:%d", st.Dev, st.Ino)
|
||||||
|
cursor, known := state.Cursors[key]
|
||||||
|
if !known {
|
||||||
|
if !state.Initialized || !active {
|
||||||
|
cursor.Offset = info.Size()
|
||||||
|
}
|
||||||
|
cursor.Generation = now
|
||||||
|
}
|
||||||
|
cursor.Seen = now
|
||||||
|
if info.Size() < cursor.Offset {
|
||||||
|
cursor.Offset = 0
|
||||||
|
cursor.Generation++
|
||||||
|
}
|
||||||
|
if len(state.Pending) < 400 && budget > 0 {
|
||||||
|
_, err = f.Seek(cursor.Offset, io.SeekStart)
|
||||||
|
if err == nil {
|
||||||
|
reader := bufio.NewReaderSize(io.LimitReader(f, budget), 64<<10)
|
||||||
|
for len(state.Pending) < 400 && budget > 0 {
|
||||||
|
line, readErr := reader.ReadString('\n')
|
||||||
|
// Keep an unfinished line for the next poll. Bound malformed huge lines.
|
||||||
|
if readErr != nil {
|
||||||
|
if len(line) >= 64<<10 {
|
||||||
|
cursor.Offset += int64(len(line))
|
||||||
|
state.ParseErrors++
|
||||||
|
}
|
||||||
|
break
|
||||||
|
}
|
||||||
|
offset := cursor.Offset
|
||||||
|
cursor.Offset += int64(len(line))
|
||||||
|
budget -= int64(len(line))
|
||||||
|
if len(line) > 64<<10 {
|
||||||
|
state.ParseErrors++
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
e, ok := parseNPMTraffic(line)
|
||||||
|
if !ok {
|
||||||
|
state.ParseErrors++
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if e.Time < now-86400 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
e.ID = fmt.Sprintf("npm:%s:%d:%d", key, cursor.Generation, offset)
|
||||||
|
state.Pending = append(state.Pending, e)
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
firstErr = err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if info.Size() > cursor.Offset {
|
||||||
|
state.BacklogBytes += info.Size() - cursor.Offset
|
||||||
|
}
|
||||||
|
state.Cursors[key] = cursor
|
||||||
|
f.Close()
|
||||||
|
}
|
||||||
|
for key, cursor := range state.Cursors {
|
||||||
|
if cursor.Seen < now-7*86400 {
|
||||||
|
delete(state.Cursors, key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if files > 0 {
|
||||||
|
state.Initialized = true
|
||||||
|
}
|
||||||
|
if firstErr != nil {
|
||||||
|
return files, errors.New("Не удалось прочитать часть access-логов NPM")
|
||||||
|
}
|
||||||
|
if files == 0 {
|
||||||
|
return files, errors.New("Access-логи NPM не найдены")
|
||||||
|
}
|
||||||
|
return files, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type crowdTrafficAlert struct {
|
||||||
|
ID int64 `json:"id"`
|
||||||
|
UUID string `json:"uuid"`
|
||||||
|
CreatedAt string `json:"created_at"`
|
||||||
|
Scenario string `json:"scenario"`
|
||||||
|
Simulated bool `json:"simulated"`
|
||||||
|
Source struct {
|
||||||
|
IP string `json:"ip"`
|
||||||
|
} `json:"source"`
|
||||||
|
Events []struct {
|
||||||
|
Meta []struct {
|
||||||
|
Key string `json:"key"`
|
||||||
|
Value string `json:"value"`
|
||||||
|
} `json:"meta"`
|
||||||
|
} `json:"events"`
|
||||||
|
Decisions []struct {
|
||||||
|
ID int64 `json:"id"`
|
||||||
|
Type string `json:"type"`
|
||||||
|
Value string `json:"value"`
|
||||||
|
Simulated bool `json:"simulated"`
|
||||||
|
} `json:"decisions"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseCrowdTraffic(data []byte) ([]TrafficEvent, error) {
|
||||||
|
var alerts []crowdTrafficAlert
|
||||||
|
if err := json.Unmarshal(data, &alerts); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out := []TrafficEvent{}
|
||||||
|
for _, a := range alerts {
|
||||||
|
if a.Simulated || net.ParseIP(a.Source.IP) == nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
ts, err := time.Parse(time.RFC3339Nano, a.CreatedAt)
|
||||||
|
if err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
id := a.UUID
|
||||||
|
if id == "" {
|
||||||
|
id = fmt.Sprintf("%d:%d", a.ID, ts.Unix())
|
||||||
|
}
|
||||||
|
// An alert may span multiple domains. Do not invent a one-to-one link.
|
||||||
|
domains := map[string]bool{}
|
||||||
|
for _, event := range a.Events {
|
||||||
|
for _, m := range event.Meta {
|
||||||
|
if m.Key == "target_fqdn" {
|
||||||
|
domains[m.Value] = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
domain := ""
|
||||||
|
if len(domains) == 1 {
|
||||||
|
for d := range domains {
|
||||||
|
domain = d
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out = append(out, TrafficEvent{ID: "cs-alert:" + id, Time: ts.Unix(), Kind: "alert", IP: a.Source.IP, Domain: domain, Reason: a.Scenario})
|
||||||
|
for _, d := range a.Decisions {
|
||||||
|
if !d.Simulated && net.ParseIP(d.Value) != nil {
|
||||||
|
out = append(out, TrafficEvent{ID: fmt.Sprintf("cs-decision:%s:%d", id, d.ID), Time: ts.Unix(), Kind: "decision", IP: d.Value, Domain: domain, Reason: d.Type + ": " + a.Scenario})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// cscli uses the existing local machine credentials. No new bouncer key or
|
||||||
|
// public LAPI port is needed, and only the read-only alerts list is executed.
|
||||||
|
func collectCrowdTraffic(ctx context.Context) ([]TrafficEvent, bool, error) {
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, 6*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
cmd := exec.CommandContext(ctx, "cscli", "alerts", "list", "--since", "24h", "--limit", "50", "-o", "json")
|
||||||
|
var output limitedTrafficOutput
|
||||||
|
cmd.Stdout = &output
|
||||||
|
if err := cmd.Run(); err != nil {
|
||||||
|
return nil, false, errors.New("cscli alerts list недоступен; проверьте права агента и локальную LAPI")
|
||||||
|
}
|
||||||
|
events, err := parseCrowdTraffic(output.Bytes())
|
||||||
|
if err != nil {
|
||||||
|
return nil, false, errors.New("Не удалось разобрать JSON CrowdSec")
|
||||||
|
}
|
||||||
|
var raw []json.RawMessage
|
||||||
|
json.Unmarshal(output.Bytes(), &raw)
|
||||||
|
return events, len(raw) >= 50, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type limitedTrafficOutput struct{ bytes.Buffer }
|
||||||
|
|
||||||
|
func (b *limitedTrafficOutput) Write(p []byte) (int, error) {
|
||||||
|
if b.Len()+len(p) > 4<<20 {
|
||||||
|
return 0, errors.New("CrowdSec response too large")
|
||||||
|
}
|
||||||
|
return b.Buffer.Write(p)
|
||||||
|
}
|
||||||
|
|
||||||
|
func saveTrafficAgentState(path string, state trafficAgentState) error {
|
||||||
|
data, err := json.Marshal(state)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
f, err := os.CreateTemp(filepath.Dir(path), ".traffic-")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
name := f.Name()
|
||||||
|
defer os.Remove(name)
|
||||||
|
if err = f.Chmod(0600); err == nil {
|
||||||
|
_, err = f.Write(data)
|
||||||
|
}
|
||||||
|
if err == nil {
|
||||||
|
err = f.Sync()
|
||||||
|
}
|
||||||
|
closeErr := f.Close()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if closeErr != nil {
|
||||||
|
return closeErr
|
||||||
|
}
|
||||||
|
return os.Rename(name, path)
|
||||||
|
}
|
||||||
|
func runAgentTraffic(ctx context.Context, config agentConfig, stateDir string) {
|
||||||
|
dir := os.Getenv("DASHBOARD_NPM_LOG_DIR")
|
||||||
|
if dir == "off" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if dir == "" {
|
||||||
|
dir = "/opt/npm/data/logs"
|
||||||
|
}
|
||||||
|
|
||||||
|
statePath := filepath.Join(stateDir, "traffic-state.json")
|
||||||
|
state := trafficAgentState{StartedAt: time.Now().Unix(), Cursors: map[string]trafficCursor{}, CrowdSeen: map[string]int64{}}
|
||||||
|
if data, err := os.ReadFile(statePath); err == nil {
|
||||||
|
if json.Unmarshal(data, &state) != nil {
|
||||||
|
fmt.Fprintln(os.Stderr, "traffic: invalid checkpoint; collector stopped")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
} else if !os.IsNotExist(err) {
|
||||||
|
fmt.Fprintln(os.Stderr, "traffic: checkpoint unreadable")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if state.Cursors == nil {
|
||||||
|
state.Cursors = map[string]trafficCursor{}
|
||||||
|
}
|
||||||
|
if state.CrowdSeen == nil {
|
||||||
|
state.CrowdSeen = map[string]int64{}
|
||||||
|
}
|
||||||
|
client := &http.Client{Timeout: 10 * time.Second}
|
||||||
|
lastCrowd := time.Time{}
|
||||||
|
crowdAt := int64(0)
|
||||||
|
crowdError := ""
|
||||||
|
crowdLimited := false
|
||||||
|
ticker := time.NewTicker(2 * time.Second)
|
||||||
|
defer ticker.Stop()
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
files, logErr := state.readLogs(dir)
|
||||||
|
if len(state.Pending) < 400 && time.Since(lastCrowd) >= 15*time.Second {
|
||||||
|
events, limited, err := collectCrowdTraffic(ctx)
|
||||||
|
crowdLimited = limited
|
||||||
|
lastCrowd = time.Now()
|
||||||
|
if err != nil {
|
||||||
|
crowdError = err.Error()
|
||||||
|
} else {
|
||||||
|
crowdError = ""
|
||||||
|
crowdAt = time.Now().Unix()
|
||||||
|
for _, e := range events {
|
||||||
|
if _, seen := state.CrowdSeen[e.ID]; !seen && len(state.Pending) < 500 {
|
||||||
|
state.Pending = append(state.Pending, e)
|
||||||
|
state.CrowdSeen[e.ID] = e.Time
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for id, ts := range state.CrowdSeen {
|
||||||
|
if ts < time.Now().Unix()-86400 {
|
||||||
|
delete(state.CrowdSeen, id)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Persist the pending batch AND offsets before sending. A retry/restart may
|
||||||
|
// resend events; the server's (agent,event_id) unique key makes it idempotent.
|
||||||
|
if err := saveTrafficAgentState(statePath, state); err != nil {
|
||||||
|
fmt.Fprintln(os.Stderr, "traffic: checkpoint write failed; collector stopped")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
batch := TrafficBatch{BacklogBytes: state.BacklogBytes, CrowdSecLimited: crowdLimited, Events: state.Pending, Files: files, ParseErrors: state.ParseErrors, CrowdSecAt: crowdAt, CrowdSecError: crowdError, StartedAt: state.StartedAt}
|
||||||
|
if logErr != nil {
|
||||||
|
batch.LogError = logErr.Error()
|
||||||
|
}
|
||||||
|
data, _ := json.Marshal(batch)
|
||||||
|
req, err := http.NewRequestWithContext(ctx, "POST", config.Server+"/api/agent/traffic", bytes.NewReader(data))
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
req.Header.Set("Authorization", "Bearer "+config.ID+"."+config.Secret)
|
||||||
|
req.Header.Set("Content-Type", "application/json")
|
||||||
|
if res, err := client.Do(req); err == nil {
|
||||||
|
io.Copy(io.Discard, io.LimitReader(res.Body, 4096))
|
||||||
|
res.Body.Close()
|
||||||
|
if res.StatusCode == 200 {
|
||||||
|
state.Pending = nil
|
||||||
|
if saveTrafficAgentState(statePath, state) != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case <-ticker.C:
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func startAgentTraffic(config agentConfig, stateDir string) func() {
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
done := make(chan struct{})
|
||||||
|
go func() { defer close(done); runAgentTraffic(ctx, config, stateDir) }()
|
||||||
|
return func() { cancel(); <-done }
|
||||||
|
}
|
||||||
@@ -124,10 +124,22 @@ func (s *Store) AcknowledgeAlert(eventID int64) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (s *Store) AlertHistory(limit int) ([]AlertEvent, error) {
|
func (s *Store) AlertHistory(limit int) ([]AlertEvent, error) {
|
||||||
|
return s.AlertHistoryBetween(limit, 0, 0)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Store) AlertHistoryBetween(limit int, from, to int64) ([]AlertEvent, error) {
|
||||||
if limit < 1 || limit > 500 {
|
if limit < 1 || limit > 500 {
|
||||||
limit = 200
|
limit = 200
|
||||||
}
|
}
|
||||||
rows, err := s.db.Query(`SELECT id,alert_key,severity,source,title,message,first_seen,last_seen,resolved_at,acknowledged_at,planned,maintenance_reason FROM alert_events ORDER BY first_seen DESC LIMIT ?`, limit)
|
query := `SELECT id,alert_key,severity,source,title,message,first_seen,last_seen,resolved_at,acknowledged_at,planned,maintenance_reason FROM alert_events`
|
||||||
|
args := []any{}
|
||||||
|
if from > 0 && to > from {
|
||||||
|
query += ` WHERE first_seen>=? AND first_seen<?`
|
||||||
|
args = append(args, from, to)
|
||||||
|
}
|
||||||
|
query += ` ORDER BY first_seen DESC LIMIT ?`
|
||||||
|
args = append(args, limit)
|
||||||
|
rows, err := s.db.Query(query, args...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|||||||
45
alerts.go
45
alerts.go
@@ -74,7 +74,10 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
|
|||||||
if disk.SMARTStatus == "Ошибка" {
|
if disk.SMARTStatus == "Ошибка" {
|
||||||
add("disk-smart-"+disk.Name, "critical", "Диски", disk.Model+": ошибка SMART", joinReasons(disk.AttentionReasons, "SMART сообщает о неисправности диска."))
|
add("disk-smart-"+disk.Name, "critical", "Диски", disk.Model+": ошибка SMART", joinReasons(disk.AttentionReasons, "SMART сообщает о неисправности диска."))
|
||||||
} else if disk.SMARTStatus == "Требует внимания" {
|
} else if disk.SMARTStatus == "Требует внимания" {
|
||||||
add("disk-smart-"+disk.Name, "warning", "Диски", disk.Model+": SMART требует внимания", joinReasons(disk.AttentionReasons, "Проверьте SMART-показатели диска."))
|
reasons := withoutReason(disk.AttentionReasons, "SMART self-test ещё ни разу не запускался")
|
||||||
|
if len(reasons) > 0 {
|
||||||
|
add("disk-smart-"+disk.Name, "warning", "Диски", disk.Model+": SMART требует внимания", joinReasons(reasons, "Проверьте SMART-показатели диска."))
|
||||||
|
}
|
||||||
}
|
}
|
||||||
if disk.Temperature != nil {
|
if disk.Temperature != nil {
|
||||||
limit := thresholds.DiskTempWarning
|
limit := thresholds.DiskTempWarning
|
||||||
@@ -215,8 +218,22 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
|
|||||||
for _, service := range agent.Report.Services {
|
for _, service := range agent.Report.Services {
|
||||||
if service.State == "failed" || service.SubState == "failed" {
|
if service.State == "failed" || service.SubState == "failed" {
|
||||||
add("agent-systemd-"+agent.ID+"-"+service.Name+vmSuffix, "critical", "Агенты", agent.Name+": упала служба "+service.Name, fmt.Sprintf("%s · VMID %d.", firstNonEmpty(service.Description, service.SubState), agent.VMID))
|
add("agent-systemd-"+agent.ID+"-"+service.Name+vmSuffix, "critical", "Агенты", agent.Name+": упала служба "+service.Name, fmt.Sprintf("%s · VMID %d.", firstNonEmpty(service.Description, service.SubState), agent.VMID))
|
||||||
|
} else if service.Flapping {
|
||||||
|
add("agent-flapping-"+agent.ID+"-"+service.Name+vmSuffix, "warning", "Агенты", agent.Name+": служба постоянно перезапускается", fmt.Sprintf("%s: %d рестартов, последний запуск %s · VMID %d.", service.Name, service.Restarts, time.Unix(service.StartedAt, 0).Format("15:04:05"), agent.VMID))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
for _, filesystem := range agent.Report.Filesystems {
|
||||||
|
if filesystem.UsedPercent >= 90 {
|
||||||
|
add("agent-filesystem-"+agent.ID+"-"+filesystem.Mountpoint+vmSuffix, "critical", "Агенты", agent.Name+": заканчивается место "+filesystem.Mountpoint, fmt.Sprintf("Заполнено %.1f%%, свободно %.1f GB · VMID %d.", filesystem.UsedPercent, float64(filesystem.TotalBytes-filesystem.UsedBytes)/1073741824, agent.VMID))
|
||||||
|
}
|
||||||
|
if filesystem.InodesPercent >= 90 {
|
||||||
|
add("agent-inodes-"+agent.ID+"-"+filesystem.Mountpoint+vmSuffix, "critical", "Агенты", agent.Name+": заканчиваются inode "+filesystem.Mountpoint, fmt.Sprintf("Использовано %.1f%% inode · VMID %d.", filesystem.InodesPercent, agent.VMID))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(agent.Report.OOMKills) > 0 {
|
||||||
|
latest := agent.Report.OOMKills[0]
|
||||||
|
add("agent-oom-"+agent.ID+vmSuffix, "critical", "Агенты", agent.Name+": ядро завершало процессы из-за нехватки памяти", fmt.Sprintf("%s · %s · VMID %d.", time.Unix(latest.Time, 0).Format("02.01 15:04"), latest.Message, agent.VMID))
|
||||||
|
}
|
||||||
if agent.Report.RebootRequired {
|
if agent.Report.RebootRequired {
|
||||||
add("agent-reboot-"+agent.ID+vmSuffix, "info", "Агенты", agent.Name+": требуется перезагрузка", fmt.Sprintf("Обновления системы запросили перезагрузку · VMID %d.", agent.VMID))
|
add("agent-reboot-"+agent.ID+vmSuffix, "info", "Агенты", agent.Name+": требуется перезагрузка", fmt.Sprintf("Обновления системы запросили перезагрузку · VMID %d.", agent.VMID))
|
||||||
}
|
}
|
||||||
@@ -246,10 +263,13 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
|
|||||||
}
|
}
|
||||||
var networkErrors uint64
|
var networkErrors uint64
|
||||||
for _, iface := range metrics.Network.Interfaces {
|
for _, iface := range metrics.Network.Interfaces {
|
||||||
networkErrors += iface.ReceiveErrors + iface.TransmitErrors + iface.ReceiveDrops + iface.TransmitDrops
|
networkErrors += iface.NewErrors + iface.NewFaults
|
||||||
|
if iface.LinkDown {
|
||||||
|
add("network-link-"+iface.Name, "critical", "Сеть", iface.Name+": нет линка", "Используемый интерфейс потерял carrier или перешёл в down.")
|
||||||
|
}
|
||||||
}
|
}
|
||||||
if networkErrors > 0 {
|
if networkErrors > 0 {
|
||||||
add("network-errors", "warning", "Сеть", "Сетевые ошибки и потери пакетов", fmt.Sprintf("Суммарный счётчик интерфейсов: %d.", networkErrors))
|
add("network-errors", "critical", "Сеть", "Появились новые сетевые ошибки", fmt.Sprintf("Прирост счётчиков ошибок: %d (счётчики могут пересекаться). Software RX/TX drops показаны отдельно и не создают этот алерт.", networkErrors))
|
||||||
}
|
}
|
||||||
if !metrics.Network.Health.CheckedAt.IsZero() && !metrics.Network.Health.GatewayUp {
|
if !metrics.Network.Health.CheckedAt.IsZero() && !metrics.Network.Health.GatewayUp {
|
||||||
add("network-gateway", "critical", "Сеть", "Недоступен локальный шлюз", firstNonEmpty(metrics.Network.Health.Gateway, "Default gateway не определён."))
|
add("network-gateway", "critical", "Сеть", "Недоступен локальный шлюз", firstNonEmpty(metrics.Network.Health.Gateway, "Default gateway не определён."))
|
||||||
@@ -269,14 +289,7 @@ func evaluateAlerts(metrics dashboardMetrics, thresholds AlertThresholds) []Aler
|
|||||||
for _, name := range metrics.Network.Health.DisappearedInterfaces {
|
for _, name := range metrics.Network.Health.DisappearedInterfaces {
|
||||||
add("network-interface-missing-"+name, "warning", "Сеть", "Исчез сетевой интерфейс", name+" больше не присутствует в /proc/net/dev.")
|
add("network-interface-missing-"+name, "warning", "Сеть", "Исчез сетевой интерфейс", name+" больше не присутствует в /proc/net/dev.")
|
||||||
}
|
}
|
||||||
for _, iface := range metrics.Network.Interfaces {
|
|
||||||
if iface.Kind == "Physical" && !iface.Carrier && (iface.Master != "" || iface.ReceiveBytes+iface.TransmitBytes > 0) {
|
|
||||||
add("network-link-"+iface.Name, "warning", "Сеть", iface.Name+": нет carrier", "Физический линк не поднят.")
|
|
||||||
}
|
|
||||||
if (iface.Kind == "Bridge" || iface.Kind == "Bond" || iface.Kind == "VLAN") && iface.State == "down" {
|
|
||||||
add("network-state-"+iface.Name, "warning", "Сеть", iface.Name+": интерфейс выключен", iface.Kind+" находится в состоянии down.")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if !metrics.Monitor.LastSuccessfulCollection.IsZero() && time.Since(metrics.Monitor.LastSuccessfulCollection) > 30*time.Second {
|
if !metrics.Monitor.LastSuccessfulCollection.IsZero() && time.Since(metrics.Monitor.LastSuccessfulCollection) > 30*time.Second {
|
||||||
add("monitor-stale", "critical", "Dashboard", "Данные мониторинга устарели", "Последний успешный сбор был "+time.Since(metrics.Monitor.LastSuccessfulCollection).Round(time.Second).String()+" назад.")
|
add("monitor-stale", "critical", "Dashboard", "Данные мониторинга устарели", "Последний успешный сбор был "+time.Since(metrics.Monitor.LastSuccessfulCollection).Round(time.Second).String()+" назад.")
|
||||||
}
|
}
|
||||||
@@ -372,3 +385,13 @@ func joinReasons(reasons []string, fallback string) string {
|
|||||||
}
|
}
|
||||||
return strings.Join(reasons, " · ")
|
return strings.Join(reasons, " · ")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func withoutReason(reasons []string, excluded string) []string {
|
||||||
|
filtered := make([]string, 0, len(reasons))
|
||||||
|
for _, reason := range reasons {
|
||||||
|
if reason != excluded {
|
||||||
|
filtered = append(filtered, reason)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return filtered
|
||||||
|
}
|
||||||
|
|||||||
@@ -25,3 +25,17 @@ func TestSlowCollectionDoesNotCreateAlert(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestNeverRunSelfTestCreatesOneAlert(t *testing.T) {
|
||||||
|
metrics := dashboardMetrics{Disks: []DiskMetrics{{
|
||||||
|
Name: "nvme0n1",
|
||||||
|
Model: "Test NVMe",
|
||||||
|
SMARTStatus: "Требует внимания",
|
||||||
|
SelfTestNeverRun: true,
|
||||||
|
AttentionReasons: []string{"SMART self-test ещё ни разу не запускался"},
|
||||||
|
}}}
|
||||||
|
alerts := evaluateAlerts(metrics, defaultThresholds())
|
||||||
|
if len(alerts) != 1 || alerts[0].ID != "disk-selftest-nvme0n1" {
|
||||||
|
t.Fatalf("ожидалось одно точное уведомление о self-test, получено: %+v", alerts)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
14
assets/NOTICE.md
Normal file
14
assets/NOTICE.md
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
# Map data
|
||||||
|
|
||||||
|
`world.json` is derived from Natural Earth 1:110m admin-0 countries (coordinates
|
||||||
|
rounded to two decimal places; only country codes, labels and boundaries kept).
|
||||||
|
Natural Earth data is public domain: https://www.naturalearthdata.com/about/terms-of-use/
|
||||||
|
Source: https://github.com/nvkelso/natural-earth-vector/blob/master/geojson/ne_110m_admin_0_countries.geojson
|
||||||
|
|
||||||
|
`dbip-country-lite-2026-10.mmdb.gz`: IP Geolocation by DB-IP, October 2026.
|
||||||
|
Source: https://download.db-ip.com/free/dbip-country-lite-2026-10.mmdb.gz
|
||||||
|
License: Creative Commons Attribution 4.0 International (CC BY 4.0).
|
||||||
|
https://creativecommons.org/licenses/by/4.0/
|
||||||
|
https://db-ip.com/db/download/ip-to-country-lite
|
||||||
|
The database is unmodified. The traffic page displays the required DB-IP link.
|
||||||
|
Countries are approximate; plotted positions are country labels, not exact IP locations.
|
||||||
BIN
assets/dbip-country-lite-2026-10.mmdb.gz
Normal file
BIN
assets/dbip-country-lite-2026-10.mmdb.gz
Normal file
Binary file not shown.
47
assets/traffic.html
Normal file
47
assets/traffic.html
Normal file
@@ -0,0 +1,47 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="ru"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><title>Карта трафика · Proxmox Dashboard</title>
|
||||||
|
<style>
|
||||||
|
:root{color-scheme:dark;--bg:#070a13;--panel:#101525;--line:#222a40;--text:#eef2ff;--muted:#8490ab;--blue:#55c9ff;--orange:#ffb05f;--pink:#fa79b7}*{box-sizing:border-box}body{margin:0;background:radial-gradient(ellipse at 48% 40%,#10172a 0,transparent 62%),var(--bg);color:var(--text);font:14px/1.5 system-ui,sans-serif}a{color:var(--blue);text-decoration:none}button,select{font:inherit;color:var(--text);background:#141b2d;border:1px solid #303c59;border-radius:9px;padding:9px 12px}button{cursor:pointer}button:hover,select:hover{border-color:var(--blue)}button.active{background:#18334c;border-color:#55c9ff}header{padding:28px 32px 14px;display:flex;justify-content:space-between;align-items:center;gap:20px}.eyebrow{color:var(--blue);font-size:10px;letter-spacing:.2em;font-weight:700}h1{font-size:29px;letter-spacing:-.04em;margin:5px 0}header nav{display:flex;gap:15px;align-items:center}.live{font-size:11px;padding:5px 10px;border:1px solid #294c4a;border-radius:30px;color:#73ddb2}.live.stale{color:var(--orange);border-color:#6a4a2a}.toolbar{padding:12px 32px;display:flex;align-items:center;flex-wrap:wrap;gap:8px}.toolbar select{max-width:270px}.toolbar label{margin-left:8px;color:var(--muted)}.dashboard{display:grid;grid-template-columns:205px minmax(0,1fr) 330px;gap:18px;margin:12px 32px}.panel{background:#101525d9;border:1px solid var(--line);border-radius:16px;padding:18px;min-width:0}.panel h2{font-size:14px;font-weight:550;margin:0 0 16px}.metric{padding:17px 0;border-bottom:1px solid var(--line)}.metric:first-child{padding-top:0}.metric:last-child{border:0}.metric small{color:var(--muted);display:block;font-size:11px}.metric b{display:block;font-size:32px;letter-spacing:-.04em;margin:4px 0}.metric.request b{color:var(--blue)}.metric.alert b{color:var(--orange)}.metric.decision b{color:var(--pink)}.aside{display:flex;flex-direction:column;gap:16px}.ranking{display:flex;justify-content:space-between;gap:8px;padding:7px 0;color:#bac7de;font-size:12px}.ranking span{overflow:hidden;text-overflow:ellipsis;white-space:nowrap}.ranking b{font:12px ui-monospace,monospace;color:var(--blue)}.map-panel{position:relative;min-height:490px;display:flex;align-items:center;flex-direction:column;justify-content:center}.map-panel canvas{display:block;width:100%;aspect-ratio:1.85;touch-action:pan-y}.map-heading{position:absolute;top:8px;left:16px;right:16px;display:flex;justify-content:space-between;gap:12px;color:var(--muted);font-size:11px}.map-foot{width:100%;padding:16px;color:var(--muted);font-size:11px;display:flex;justify-content:space-between;gap:12px}.legend{display:flex;flex-wrap:wrap;gap:14px}.dot{display:inline-block;width:7px;height:7px;border-radius:50%;background:var(--blue);margin-right:6px}.dot.alert{background:var(--orange)}.dot.decision{background:var(--pink)}.feed{max-height:670px;overflow:auto}.event{padding:14px 0;border-top:1px solid var(--line)}.event-head{display:flex;justify-content:space-between;gap:9px;align-items:baseline}.event-head b{font:12px ui-monospace,monospace}.event-head time{font-size:10px;color:var(--muted)}.event .meta{color:#b0bdd4;font-size:11px;margin-top:5px;overflow-wrap:anywhere}.event .path{font:11px/1.6 ui-monospace,monospace;color:var(--muted);margin-top:5px;overflow-wrap:anywhere}.kind{font-size:10px;color:var(--blue)}.kind.alert{color:var(--orange)}.kind.decision{color:var(--pink)}.empty{color:var(--muted);font-size:13px;padding:24px 0}.bottom{margin:18px 32px;display:grid;grid-template-columns:minmax(0,1fr) 330px;gap:18px}.chart{width:100%;height:100px;display:block}.note{font-size:11px;color:var(--muted);overflow-wrap:anywhere}.status-box{margin:0 32px;padding:10px 0;color:var(--muted);font-size:12px}.status-box.error{color:var(--orange)}footer{margin:25px 32px;font-size:11px;color:var(--muted);display:flex;gap:18px;flex-wrap:wrap}.tooltip{position:fixed;pointer-events:none;display:none;padding:10px 14px;border:1px solid #475578;border-radius:9px;background:#101725f5;font-size:12px;z-index:3}.now{font:13px ui-monospace,monospace;color:var(--muted)}@media(min-width:1700px){.dashboard{grid-template-columns:230px minmax(0,1fr) 360px}.map-panel{min-height:680px}.feed{max-height:780px}}@media(max-width:1100px){.dashboard{grid-template-columns:165px minmax(0,1fr)}.feed-panel{grid-column:1/-1}.feed{max-height:330px}.map-panel{min-height:380px}.bottom{grid-template-columns:1fr}}@media(max-width:650px){header{padding:20px 16px;align-items:flex-start;flex-direction:column}h1{font-size:25px}.toolbar{padding:0 16px}.dashboard{display:flex;flex-direction:column;margin:18px 16px}.aside{display:contents}.metrics{display:grid;grid-template-columns:1fr 1fr;gap:12px}.metric,.metric:first-child{padding:0;border:0}.metric b{font-size:27px}.rank-panel{order:3}.map-panel{min-height:260px;order:1}.feed-panel{order:2}.bottom,footer{margin:16px}.status-box{margin:0 16px}.map-heading{position:static;padding:0 0 10px}.map-foot{padding:8px 0;font-size:10px}.legend{gap:8px}.now{display:none}}
|
||||||
|
</style></head><body>
|
||||||
|
<header><div><div class="eyebrow">PROXMOX DASHBOARD / TRAFFIC OBSERVATORY</div><h1>Мир обращается к вашим сервисам</h1><span class="note">HTTP-запросы и события CrowdSec · последние 24 часа</span></div><nav><span id="live" class="live stale">Подключение…</span><span class="now" id="clock"></span><a href="/">← Dashboard</a><button id="fullscreen" aria-label="Полный экран">⛶</button></nav></header>
|
||||||
|
<div class="toolbar"><button class="active" data-kind="">Весь трафик</button><button data-kind="request">HTTP-запросы</button><button data-kind="alert">Срабатывания</button><button data-kind="decision">Решения CrowdSec</button><select id="domain" aria-label="Фильтр по домену"><option value="">Все домены</option></select><label><input type="checkbox" id="external"> Только внешние IP</label><button id="pause">Пауза</button></div>
|
||||||
|
<div class="status-box" id="status" role="status">Получаем данные выбранного агента…</div>
|
||||||
|
<section class="dashboard">
|
||||||
|
<aside class="aside"><div class="panel metrics"><div class="metric"><small>УНИКАЛЬНЫХ IP · HTTP</small><b id="unique">—</b></div><div class="metric request"><small>HTTP-ЗАПРОСОВ</small><b id="requests">—</b></div><div class="metric alert"><small>СРАБАТЫВАНИЙ CROWDSEC</small><b id="alerts">—</b></div><div class="metric decision"><small>РЕШЕНИЙ CROWDSEC</small><b id="decisions">—</b></div></div><div class="panel rank-panel"><h2>Страны · события</h2><div id="countries"></div></div></aside>
|
||||||
|
<div class="map-panel"><div class="map-heading"><span>ИСТОЧНИКИ ЗАПРОСОВ</span><span id="sourceName">—</span></div><canvas id="world" role="img" aria-label="Карта стран источников; те же данные доступны в списке стран и ленте событий"></canvas><div class="map-foot"><div class="legend"><span><i class="dot"></i>Запрос</span><span><i class="dot alert"></i>Срабатывание</span><span><i class="dot decision"></i>Решение</span></div><span id="unmapped">Без геолокации: —</span></div><div class="note">Точки обозначают страны, а не точное местоположение людей.</div></div>
|
||||||
|
<aside class="panel feed-panel"><h2>Лента событий <span class="note">· последние 100</span></h2><div class="feed" id="feed"><div class="empty">Ожидание первого события…</div></div></aside>
|
||||||
|
</section>
|
||||||
|
<section class="bottom"><div class="panel"><div style="display:flex;justify-content:space-between"><h2>HTTP-запросы в минуту · последний час</h2><span class="note"><b id="qps">—</b> req/s · среднее за 60 сек.</span></div><canvas class="chart" id="chart" role="img" aria-label="Количество HTTP-запросов за каждую минуту последнего часа"></canvas></div><div class="panel"><h2>IP · события</h2><div id="ips"></div></div></section>
|
||||||
|
<footer><span>Решение о бане ≠ подтверждённый заблокированный запрос. HTTP 401/403/502 не означают блокировку CrowdSec.</span><span>GeoIP: <a href="https://db-ip.com" target="_blank" rel="noopener">IP Geolocation by DB-IP</a> · <span id="geoDate"></span> · <a href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a></span><span>Карта: <a href="https://www.naturalearthdata.com/">Natural Earth</a></span><a href="/#traffic-settings">Настройки источника</a></footer>
|
||||||
|
<div id="tooltip" class="tooltip"></div>
|
||||||
|
<script>
|
||||||
|
const el=id=>document.getElementById(id),safe=v=>String(v??'').replace(/[&<>"']/g,c=>({'&':'&','<':'<','>':'>','"':'"',"'":'''}[c])),num=n=>Number(n||0).toLocaleString('ru-RU');
|
||||||
|
let world=[],countryMap=new Map(),view=null,kind='',paused=false,busy=false,pulses=[],seen=new Set(),hitPoints=[],baseMap=null,lastPoll=0;
|
||||||
|
const colors={request:'#55c9ff',alert:'#ffb05f',decision:'#fa79b7'},kindLabels={request:'HTTP-запрос',alert:'Срабатывание',decision:'Решение CrowdSec'};
|
||||||
|
const countryNames=new Intl.DisplayNames(['ru'],{type:'region'});const countryName=code=>code==='LAN'?'Локальная сеть':countryMap.get(code)?.name||(code&&/^[A-Z]{2}$/.test(code)?countryNames.of(code):'Не определена');
|
||||||
|
function project(lon,lat,w,h){return [(lon+180)/360*w,(85-lat)/170*h]}
|
||||||
|
function mapBackground(w,h){const off=document.createElement('canvas');off.width=w;off.height=h;const ctx=off.getContext('2d'),mask=document.createElement('canvas');mask.width=w;mask.height=h;const m=mask.getContext('2d');m.fillStyle='#fff';world.forEach(c=>{const polygons=c.geometry.type==='Polygon'?[c.geometry.coordinates]:c.geometry.coordinates;polygons.forEach(poly=>{m.beginPath();poly.forEach(ring=>{ring.forEach(([lon,lat],i)=>{const [x,y]=project(lon,lat,w,h);if(i===0)m.moveTo(x,y);else m.lineTo(x,y)});m.closePath()});m.fill('evenodd')})});const pixels=m.getImageData(0,0,w,h).data;const step=Math.max(4,Math.round(w/210));ctx.fillStyle='#2a3355';for(let y=0,row=0;y<h;y+=step,row++){for(let x=(row%2)*step/2;x<w;x+=step){if(pixels[(Math.floor(y)*w+Math.floor(x))*4+3]){ctx.beginPath();ctx.arc(x,y,step*.32,0,Math.PI*2);ctx.fill()}}}return off}
|
||||||
|
function drawMap(now){const canvas=el('world'),w=Math.max(1,Math.floor(canvas.clientWidth)),h=Math.round(w/1.85),dpr=Math.min(devicePixelRatio||1,2);if(canvas.width!==Math.round(w*dpr)||canvas.height!==Math.round(h*dpr)){canvas.width=Math.round(w*dpr);canvas.height=Math.round(h*dpr);baseMap=mapBackground(w,h)}const ctx=canvas.getContext('2d');ctx.setTransform(dpr,0,0,dpr,0,0);ctx.clearRect(0,0,w,h);if(baseMap)ctx.drawImage(baseMap,0,0);hitPoints=[];
|
||||||
|
if(view){const destination=countryMap.get(view.settings.serverCountry),end=destination?project(destination.lon,destination.lat,w,h):[w*.82,h*.88];
|
||||||
|
for(const c of view.countries){const country=countryMap.get(c.name);if(!country)continue;const [x,y]=project(country.lon,country.lat,w,h);ctx.fillStyle='#55c9ff';ctx.globalAlpha=.75;ctx.beginPath();ctx.arc(x,y,2+Math.min(5,Math.log2(c.count+1)),0,Math.PI*2);ctx.fill();hitPoints.push({x,y,text:country.name+' · '+num(c.count)+' событий'})}ctx.globalAlpha=1;
|
||||||
|
pulses=pulses.filter(p=>now-p.start<3200);for(const p of pulses){const c=countryMap.get(p.country);if(!c)continue;const start=project(c.lon,c.lat,w,h),t=Math.min(1,(now-p.start)/2400),control=[(start[0]+end[0])/2,(start[1]+end[1])/2-Math.min(h*.3,Math.abs(start[0]-end[0])*.3+30)],fade=Math.min(1,(3200-(now-p.start))/700);ctx.strokeStyle=colors[p.kind];ctx.globalAlpha=.25*fade;ctx.lineWidth=1;ctx.beginPath();ctx.moveTo(...start);ctx.quadraticCurveTo(...control,...end);ctx.stroke();const x=(1-t)**2*start[0]+2*(1-t)*t*control[0]+t*t*end[0],y=(1-t)**2*start[1]+2*(1-t)*t*control[1]+t*t*end[1];ctx.fillStyle=colors[p.kind];ctx.globalAlpha=fade;ctx.shadowColor=colors[p.kind];ctx.shadowBlur=12;ctx.beginPath();ctx.arc(x,y,2.5,0,Math.PI*2);ctx.fill();ctx.shadowBlur=0}
|
||||||
|
ctx.globalAlpha=1;ctx.strokeStyle='#c8b5ff';ctx.lineWidth=1.5;ctx.beginPath();ctx.arc(...end,7,0,Math.PI*2);ctx.stroke();ctx.fillStyle='#e7dcff';ctx.beginPath();ctx.arc(...end,3,0,Math.PI*2);ctx.fill();ctx.font='11px system-ui';ctx.textAlign='center';ctx.fillText(destination?'NPM · '+destination.name:'NPM · геопозиция не задана',end[0],end[1]+22);
|
||||||
|
}
|
||||||
|
if(!document.hidden&&!paused)requestAnimationFrame(drawMap)
|
||||||
|
}
|
||||||
|
function ranking(id,items,label){el(id).innerHTML=items.length?items.slice(0,7).map(c=>'<div class="ranking"><span>'+safe(label(c.name))+'</span><b>'+num(c.count)+'</b></div>').join(''):'<div class="note">Пока нет событий</div>'}
|
||||||
|
function drawChart(){const c=el('chart'),w=c.clientWidth,h=100,dpr=devicePixelRatio||1;c.width=w*dpr;c.height=h*dpr;const ctx=c.getContext('2d');ctx.scale(dpr,dpr);if(!view)return;const buckets=new Map(view.series.map(v=>[Number(v.name),v.count])),end=Math.floor(view.now/60)*60,values=Array.from({length:60},(_,i)=>buckets.get(end-(59-i)*60)||0),max=Math.max(1,...values);ctx.strokeStyle='#26314b';ctx.beginPath();ctx.moveTo(0,82);ctx.lineTo(w,82);ctx.stroke();ctx.fillStyle='#55c9ff';values.forEach((v,i)=>{const x=i*w/60,height=v/max*65;ctx.fillRect(x,82-height,Math.max(1,w/60-3),height)});ctx.fillStyle='#8490ab';ctx.font='10px system-ui';ctx.fillText('−60 мин',0,98);ctx.textAlign='right';ctx.fillText('сейчас',w,98);ctx.fillText('макс. '+num(max)+'/мин',w,12)}
|
||||||
|
function render(data){view=data;el('unique').textContent=num(data.uniqueIps);el('requests').textContent=num(data.requests);el('alerts').textContent=num(data.alerts);el('decisions').textContent=num(data.decisions);el('qps').textContent=Number(data.qps).toFixed(2);el('geoDate').textContent=data.geoDate;ranking('countries',data.countries,countryName);ranking('ips',data.ips,v=>v);
|
||||||
|
const src=data.sources[0],fresh=src&&data.now-src.lastSeen<20;el('live').textContent=paused?'ПАУЗА':fresh?'● В ЭФИРЕ':'ОЖИДАНИЕ АГЕНТА';el('live').classList.toggle('stale',!fresh||paused);el('sourceName').textContent=src?src.name:'Источник ещё не прислал данные';
|
||||||
|
const messages=[];if(!src)messages.push('Выбранный агент ещё не прислал данные. Нужен агент v0.20.0 или новее.');else{messages.push(src.name+' · '+src.status.files+' логов · сбор с '+new Date(src.status.startedAt*1000).toLocaleString('ru-RU'));if(!fresh)messages.push('Источник не обновляется; показана сохранённая история');if(src.status.logError)messages.push(src.status.logError);if(src.status.crowdsecError)messages.push('CrowdSec: '+src.status.crowdsecError);if(src.status.backlogBytes)messages.push('Ожидает чтения: '+num(src.status.backlogBytes)+' байт');if(src.status.crowdsecLimited)messages.push('CrowdSec: получены последние 50 срабатываний; более ранняя история может быть неполной');if(src.status.parseErrors)messages.push('Нераспознанных строк: '+src.status.parseErrors)}if(!data.geoReady)messages.push('База GeoIP недоступна');if(data.limited)messages.push('Достигнут лимит 200 000 событий: часть суточной истории удалена');el('status').textContent=messages.join(' · ');
|
||||||
|
el('status').classList.toggle('error',!fresh||!!src?.status.logError||!!src?.status.crowdsecError||data.limited);
|
||||||
|
const current=el('domain').value;if(!current&&!kind){el('domain').innerHTML='<option value="">Все домены</option>'+data.domains.filter(c=>c.name).map(c=>'<option value="'+safe(c.name)+'">'+safe(c.name)+'</option>').join('')}
|
||||||
|
const unmapped=data.countries.filter(c=>!countryMap.has(c.name)).reduce((n,c)=>n+c.count,0);el('unmapped').textContent='Локальные / без геолокации: '+num(unmapped);
|
||||||
|
el('feed').innerHTML=data.events.length?data.events.map(e=>'<article class="event"><div class="event-head"><b>'+safe(e.ip)+'</b><time>'+new Date(e.time*1000).toLocaleTimeString('ru-RU')+'</time></div><div class="meta">'+safe(countryName(e.country))+' · <span class="kind '+safe(e.kind)+'">'+kindLabels[e.kind]+'</span></div><div class="path">'+safe(e.kind==='request'?e.method+' '+e.domain+e.path+' · HTTP '+e.status:(e.domain?e.domain+' · ':'')+e.reason)+'</div></article>').join(''):'<div class="empty">Нет событий по выбранному фильтру. Новые HTTP-запросы появятся после записи в access-лог.</div>';
|
||||||
|
for(const e of data.events){if(!seen.has(e.sequence)&&e.time>=data.now-15&&pulses.length<70)pulses.push({country:e.country,kind:e.kind,start:performance.now()});seen.add(e.sequence)}if(seen.size>5000)seen=new Set(data.events.map(e=>e.sequence));drawChart();
|
||||||
|
}
|
||||||
|
async function poll(){if(paused||busy||document.hidden)return;busy=true;try{const q=new URLSearchParams({domain:el('domain').value,kind,external:el('external').checked?'1':'0'}),r=await fetch('/api/traffic?'+q);if(!r.ok){if(r.status===404){view=null;pulses=[];location.replace('/');return;}throw Error(r.status===404?'Карта отключена':'Не удалось обновить данные')}render(await r.json());lastPoll=Date.now()}catch(e){el('live').textContent='НЕТ ОБНОВЛЕНИЙ';el('live').classList.add('stale');el('status').textContent=e.message;el('status').classList.add('error')}finally{busy=false}}
|
||||||
|
document.querySelectorAll('[data-kind]').forEach(b=>b.onclick=()=>{kind=b.dataset.kind;document.querySelectorAll('[data-kind]').forEach(x=>x.classList.toggle('active',x===b));pulses=[];poll()});el('domain').onchange=()=>{pulses=[];poll()};el('external').onchange=()=>{pulses=[];poll()};el('pause').onclick=()=>{paused=!paused;el('pause').textContent=paused?'Продолжить':'Пауза';el('live').textContent=paused?'ПАУЗА':'Подключение…';if(!paused){poll();requestAnimationFrame(drawMap)}};el('fullscreen').onclick=()=>{if(document.fullscreenElement)document.exitFullscreen();else document.documentElement.requestFullscreen().catch(()=>{})};window.addEventListener('resize',drawChart);document.addEventListener('visibilitychange',()=>{if(!document.hidden&&!paused){poll();requestAnimationFrame(drawMap)}});
|
||||||
|
el('world').addEventListener('pointermove',event=>{const rect=event.currentTarget.getBoundingClientRect(),x=event.clientX-rect.left,y=event.clientY-rect.top,point=hitPoints.find(p=>Math.hypot(x-p.x,y-p.y)<14),tip=el('tooltip');tip.style.display=point?'block':'none';if(point){tip.textContent=point.text;tip.style.left=Math.min(innerWidth-240,event.clientX+12)+'px';tip.style.top=event.clientY+15+'px'}});el('world').addEventListener('pointerleave',()=>el('tooltip').style.display='none');
|
||||||
|
(async()=>{try{const r=await fetch('/api/traffic/world');if(!r.ok)throw Error();world=await r.json();countryMap=new Map(world.map(c=>[c.code,c]));requestAnimationFrame(drawMap);await poll();setInterval(poll,2000)}catch(e){el('status').textContent='Не удалось загрузить карту. Обновите страницу.'}})();setInterval(()=>{el('clock').textContent=new Date().toLocaleTimeString('ru-RU')},1000);
|
||||||
|
</script></body></html>
|
||||||
1
assets/world.json
Normal file
1
assets/world.json
Normal file
File diff suppressed because one or more lines are too long
22
cpu.go
22
cpu.go
@@ -3,6 +3,7 @@ package main
|
|||||||
import (
|
import (
|
||||||
"bufio"
|
"bufio"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"math"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strconv"
|
"strconv"
|
||||||
@@ -39,7 +40,9 @@ type CPUMetrics struct {
|
|||||||
Load5 float64 `json:"load5"`
|
Load5 float64 `json:"load5"`
|
||||||
Load15 float64 `json:"load15"`
|
Load15 float64 `json:"load15"`
|
||||||
Uptime uint64 `json:"uptimeSeconds"`
|
Uptime uint64 `json:"uptimeSeconds"`
|
||||||
|
TemperatureCelsius *float64 `json:"temperatureCelsius"`
|
||||||
Temperatures []Temperature `json:"temperatures"`
|
Temperatures []Temperature `json:"temperatures"`
|
||||||
|
TopProcesses []AgentProcess `json:"topProcesses"`
|
||||||
CollectedAt time.Time `json:"collectedAt"`
|
CollectedAt time.Time `json:"collectedAt"`
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -89,10 +92,11 @@ func (c *cpuCollector) collect() (CPUMetrics, error) {
|
|||||||
load1, load5, load15, _ := readLoadAverage()
|
load1, load5, load15, _ := readLoadAverage()
|
||||||
uptime, _ := readUptime()
|
uptime, _ := readUptime()
|
||||||
|
|
||||||
|
temperatures := readTemperatures()
|
||||||
return CPUMetrics{
|
return CPUMetrics{
|
||||||
CPUStatic: c.static, UsagePercent: usage, FrequencyMHz: frequency,
|
CPUStatic: c.static, UsagePercent: usage, FrequencyMHz: frequency,
|
||||||
Load1: load1, Load5: load5, Load15: load15, Uptime: uptime,
|
Load1: load1, Load5: load5, Load15: load15, Uptime: uptime,
|
||||||
Temperatures: readTemperatures(), CollectedAt: time.Now(),
|
Temperatures: temperatures, TemperatureCelsius: cpuTemperature(temperatures), TopProcesses: collectAgentProcesses(), CollectedAt: time.Now(),
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -284,3 +288,19 @@ func isCPUTemperatureChip(name string) bool {
|
|||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// cpuTemperature is the hottest CPU sensor, shared by live and historical views.
|
||||||
|
// Do not average Tctl/Tdie with cooler cores or CCDs.
|
||||||
|
func cpuTemperature(temperatures []Temperature) *float64 {
|
||||||
|
var hottest *float64
|
||||||
|
for _, t := range temperatures {
|
||||||
|
if math.IsNaN(t.Celsius) || math.IsInf(t.Celsius, 0) || t.Celsius <= 0 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if hottest == nil || t.Celsius > *hottest {
|
||||||
|
value := t.Celsius
|
||||||
|
hottest = &value
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return hottest
|
||||||
|
}
|
||||||
|
|||||||
24
cpu_temperature_test.go
Normal file
24
cpu_temperature_test.go
Normal file
@@ -0,0 +1,24 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"math"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestCPUTemperatureSharedWithHistory(t *testing.T) {
|
||||||
|
temperatures := []Temperature{{Label: "k10temp · Tctl", Celsius: 62}, {Label: "k10temp · Tccd1", Celsius: 38}, {Label: "k10temp · Tccd2", Celsius: 41}}
|
||||||
|
live := cpuTemperature(temperatures)
|
||||||
|
point := historyPointFromMetrics(dashboardMetrics{CPU: CPUMetrics{Temperatures: temperatures}})
|
||||||
|
if live == nil || *live != 62 || point.CPUTemperature != *live {
|
||||||
|
t.Fatalf("live/history disagree: %v, %+v", live, point)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCPUTemperatureInvalidSensors(t *testing.T) {
|
||||||
|
if cpuTemperature([]Temperature{{Celsius: math.NaN()}, {Celsius: math.Inf(1)}, {Celsius: 0}}) != nil {
|
||||||
|
t.Fatal("invalid temperature selected")
|
||||||
|
}
|
||||||
|
if cpuTemperature(nil) != nil {
|
||||||
|
t.Fatal("missing sensor should stay missing")
|
||||||
|
}
|
||||||
|
}
|
||||||
37
disks.go
37
disks.go
@@ -391,6 +391,17 @@ type smartctlJSON struct {
|
|||||||
} `json:"table"`
|
} `json:"table"`
|
||||||
} `json:"standard"`
|
} `json:"standard"`
|
||||||
} `json:"ata_smart_self_test_log"`
|
} `json:"ata_smart_self_test_log"`
|
||||||
|
NVMeSelfTestLog struct {
|
||||||
|
Table []struct {
|
||||||
|
SelfTestCode struct {
|
||||||
|
String string `json:"string"`
|
||||||
|
} `json:"self_test_code"`
|
||||||
|
SelfTestResult struct {
|
||||||
|
String string `json:"string"`
|
||||||
|
} `json:"self_test_result"`
|
||||||
|
PowerOnHours uint64 `json:"power_on_hours"`
|
||||||
|
} `json:"table"`
|
||||||
|
} `json:"nvme_self_test_log"`
|
||||||
NVMeHealth *struct {
|
NVMeHealth *struct {
|
||||||
Temperature float64 `json:"temperature"`
|
Temperature float64 `json:"temperature"`
|
||||||
PercentageUsed float64 `json:"percentage_used"`
|
PercentageUsed float64 `json:"percentage_used"`
|
||||||
@@ -410,6 +421,10 @@ func applySMART(disk *DiskMetrics) {
|
|||||||
if len(output) == 0 {
|
if len(output) == 0 {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
applySMARTJSON(disk, output)
|
||||||
|
}
|
||||||
|
|
||||||
|
func applySMARTJSON(disk *DiskMetrics, output []byte) {
|
||||||
var smart smartctlJSON
|
var smart smartctlJSON
|
||||||
if json.Unmarshal(output, &smart) != nil {
|
if json.Unmarshal(output, &smart) != nil {
|
||||||
return
|
return
|
||||||
@@ -451,12 +466,20 @@ func applySMART(disk *DiskMetrics) {
|
|||||||
disk.AttentionReasons = append(disk.AttentionReasons, "Ошибки целостности носителя: "+strconv.FormatUint(smart.NVMeHealth.MediaErrors, 10))
|
disk.AttentionReasons = append(disk.AttentionReasons, "Ошибки целостности носителя: "+strconv.FormatUint(smart.NVMeHealth.MediaErrors, 10))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if len(smart.ATASelfTestLog.Standard.Table) > 0 {
|
if len(smart.NVMeSelfTestLog.Table) > 0 {
|
||||||
|
last := smart.NVMeSelfTestLog.Table[0]
|
||||||
|
description := strings.TrimSpace(last.SelfTestCode.String)
|
||||||
|
result := firstNonEmpty(strings.TrimSpace(last.SelfTestResult.String), "Результат неизвестен")
|
||||||
|
if description != "" {
|
||||||
|
disk.LastSelfTest = description + " · " + result
|
||||||
|
} else {
|
||||||
|
disk.LastSelfTest = result
|
||||||
|
}
|
||||||
|
setLastSelfTestAt(disk, last.PowerOnHours)
|
||||||
|
} else if len(smart.ATASelfTestLog.Standard.Table) > 0 {
|
||||||
last := smart.ATASelfTestLog.Standard.Table[0]
|
last := smart.ATASelfTestLog.Standard.Table[0]
|
||||||
disk.LastSelfTest = firstNonEmpty(last.Status.String, "Результат неизвестен")
|
disk.LastSelfTest = firstNonEmpty(last.Status.String, "Результат неизвестен")
|
||||||
if disk.PowerOnHours != nil && *disk.PowerOnHours >= last.LifetimeHours {
|
setLastSelfTestAt(disk, last.LifetimeHours)
|
||||||
disk.LastSelfTestAt = time.Now().Add(-time.Duration(*disk.PowerOnHours-last.LifetimeHours) * time.Hour).Format(time.RFC3339)
|
|
||||||
}
|
|
||||||
} else {
|
} else {
|
||||||
disk.SelfTestNeverRun = true
|
disk.SelfTestNeverRun = true
|
||||||
disk.LastSelfTest = "Никогда не запускался"
|
disk.LastSelfTest = "Никогда не запускался"
|
||||||
@@ -469,6 +492,12 @@ func applySMART(disk *DiskMetrics) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func setLastSelfTestAt(disk *DiskMetrics, testPowerOnHours uint64) {
|
||||||
|
if disk.PowerOnHours != nil && *disk.PowerOnHours >= testPowerOnHours {
|
||||||
|
disk.LastSelfTestAt = time.Now().Add(-time.Duration(*disk.PowerOnHours-testPowerOnHours) * time.Hour).Format(time.RFC3339)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func applyATAAttributes(disk *DiskMetrics, smart smartctlJSON) {
|
func applyATAAttributes(disk *DiskMetrics, smart smartctlJSON) {
|
||||||
wearNames := map[string]bool{
|
wearNames := map[string]bool{
|
||||||
"Percent_Lifetime_Remain": true, "SSD_Life_Left": true,
|
"Percent_Lifetime_Remain": true, "SSD_Life_Left": true,
|
||||||
|
|||||||
@@ -6,8 +6,44 @@ import (
|
|||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
func TestApplySMARTJSONReadsNVMeSelfTestHistory(t *testing.T) {
|
||||||
|
disk := DiskMetrics{Path: "/dev/nvme0n1"}
|
||||||
|
output := []byte(`{
|
||||||
|
"model_name":"ADATA LEGEND 900",
|
||||||
|
"smart_status":{"passed":true},
|
||||||
|
"power_on_time":{"hours":960},
|
||||||
|
"nvme_self_test_log":{
|
||||||
|
"current_self_test_operation":{"value":0,"string":"No self-test in progress"},
|
||||||
|
"table":[{
|
||||||
|
"self_test_code":{"value":1,"string":"Short"},
|
||||||
|
"self_test_result":{"value":0,"string":"Completed without error"},
|
||||||
|
"power_on_hours":959
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
}`)
|
||||||
|
|
||||||
|
before := time.Now().Add(-2 * time.Hour)
|
||||||
|
applySMARTJSON(&disk, output)
|
||||||
|
after := time.Now()
|
||||||
|
|
||||||
|
if disk.SelfTestNeverRun {
|
||||||
|
t.Fatal("завершённый NVMe self-test ошибочно отмечен как никогда не запускавшийся")
|
||||||
|
}
|
||||||
|
if disk.LastSelfTest != "Short · Completed without error" {
|
||||||
|
t.Fatalf("неожиданный результат self-test: %q", disk.LastSelfTest)
|
||||||
|
}
|
||||||
|
parsed, err := time.Parse(time.RFC3339, disk.LastSelfTestAt)
|
||||||
|
if err != nil || parsed.Before(before) || parsed.After(after) {
|
||||||
|
t.Fatalf("неверно вычислено время self-test: %q, err=%v", disk.LastSelfTestAt, err)
|
||||||
|
}
|
||||||
|
if len(disk.AttentionReasons) != 0 || disk.SMARTStatus != "Исправен" {
|
||||||
|
t.Fatalf("исправный NVMe требует внимания: %+v", disk)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestIsPhysicalDiskName(t *testing.T) {
|
func TestIsPhysicalDiskName(t *testing.T) {
|
||||||
tests := map[string]bool{
|
tests := map[string]bool{
|
||||||
"sda": true, "sdb": true, "nvme0n1": true,
|
"sda": true, "sdb": true, "nvme0n1": true,
|
||||||
|
|||||||
5
go.mod
5
go.mod
@@ -2,7 +2,10 @@ module proxmox-host-info
|
|||||||
|
|
||||||
go 1.25.0
|
go 1.25.0
|
||||||
|
|
||||||
require modernc.org/sqlite v1.54.0
|
require (
|
||||||
|
github.com/oschwald/maxminddb-golang v1.13.1
|
||||||
|
modernc.org/sqlite v1.54.0
|
||||||
|
)
|
||||||
|
|
||||||
require (
|
require (
|
||||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||||
|
|||||||
10
go.sum
10
go.sum
@@ -1,3 +1,5 @@
|
|||||||
|
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||||
|
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
||||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||||
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e h1:ijClszYn+mADRFY17kjQEVQ1XRhq2/JR1M3sGqeJoxs=
|
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e h1:ijClszYn+mADRFY17kjQEVQ1XRhq2/JR1M3sGqeJoxs=
|
||||||
@@ -10,8 +12,14 @@ github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWE
|
|||||||
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
|
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
|
||||||
github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
|
github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
|
||||||
github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
|
github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
|
||||||
|
github.com/oschwald/maxminddb-golang v1.13.1 h1:G3wwjdN9JmIK2o/ermkHM+98oX5fS+k5MbwsmL4MRQE=
|
||||||
|
github.com/oschwald/maxminddb-golang v1.13.1/go.mod h1:K4pgV9N/GcK694KSTmVSDTODk4IsCNThNdTmnaBZ/F8=
|
||||||
|
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||||
|
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
|
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
|
||||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
|
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
|
||||||
|
github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
|
||||||
|
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
|
||||||
golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ=
|
golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ=
|
||||||
golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
|
golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
|
||||||
golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
|
golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
|
||||||
@@ -21,6 +29,8 @@ golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw=
|
|||||||
golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||||
golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
|
golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
|
||||||
golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
|
golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
|
||||||
|
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||||
|
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||||
modernc.org/cc/v4 v4.29.0 h1:CXgwL8cvxmyzBQZzbSl/6xFtMCryb6u8IOqDci39cgc=
|
modernc.org/cc/v4 v4.29.0 h1:CXgwL8cvxmyzBQZzbSl/6xFtMCryb6u8IOqDci39cgc=
|
||||||
modernc.org/cc/v4 v4.29.0/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI=
|
modernc.org/cc/v4 v4.29.0/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI=
|
||||||
modernc.org/ccgo/v4 v4.34.6 h1:sBgfIwyN0TQ9C5hwIeuqyeAKyMWnbvj2fvpF4L11uzU=
|
modernc.org/ccgo/v4 v4.34.6 h1:sBgfIwyN0TQ9C5hwIeuqyeAKyMWnbvj2fvpF4L11uzU=
|
||||||
|
|||||||
32
history.go
32
history.go
@@ -64,6 +64,30 @@ func (s *Store) AddEntityMetrics(metrics dashboardMetrics) {
|
|||||||
s.AddEntityHistory("backup-duration", "VMID "+task.ID, ts, &duration, nil)
|
s.AddEntityHistory("backup-duration", "VMID "+task.ID, ts, &duration, nil)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
for _, agent := range metrics.Agents {
|
||||||
|
if !agent.Online {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
name := agent.ID + " · " + agent.Name
|
||||||
|
cpu := agent.Report.CPUPercent
|
||||||
|
memory := float64(0)
|
||||||
|
if agent.Report.MemoryTotal > 0 {
|
||||||
|
memory = float64(agent.Report.MemoryUsed) / float64(agent.Report.MemoryTotal) * 100
|
||||||
|
}
|
||||||
|
swap := float64(0)
|
||||||
|
if agent.Report.SwapTotal > 0 {
|
||||||
|
swap = float64(agent.Report.SwapUsed) / float64(agent.Report.SwapTotal) * 100
|
||||||
|
}
|
||||||
|
s.AddEntityHistory("agent-resource", name, ts, &cpu, &memory)
|
||||||
|
s.AddEntityHistory("agent-swap", name, ts, &swap, nil)
|
||||||
|
pressure := agent.Report.Pressure.IO.Avg10
|
||||||
|
s.AddEntityHistory("agent-pressure", name, ts, &pressure, nil)
|
||||||
|
for _, filesystem := range agent.Report.Filesystems {
|
||||||
|
used := filesystem.UsedPercent
|
||||||
|
inodes := filesystem.InodesPercent
|
||||||
|
s.AddEntityHistory("agent-filesystem", name+" · "+filesystem.Mountpoint, ts, &used, &inodes)
|
||||||
|
}
|
||||||
|
}
|
||||||
s.PruneEntityHistory()
|
s.PruneEntityHistory()
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -75,12 +99,10 @@ func historyPointFromMetrics(metrics dashboardMetrics) HistoryPoint {
|
|||||||
NetworkTransmit: metrics.Network.TransmitPerSec, UPSCharge: metrics.UPS.ChargePercent,
|
NetworkTransmit: metrics.Network.TransmitPerSec, UPSCharge: metrics.UPS.ChargePercent,
|
||||||
UPSLoad: metrics.UPS.LoadPercent,
|
UPSLoad: metrics.UPS.LoadPercent,
|
||||||
}
|
}
|
||||||
if len(metrics.CPU.Temperatures) > 0 {
|
if temperature := cpuTemperature(metrics.CPU.Temperatures); temperature != nil {
|
||||||
for _, temperature := range metrics.CPU.Temperatures {
|
point.CPUTemperature = *temperature
|
||||||
point.CPUTemperature += temperature.Celsius
|
|
||||||
}
|
|
||||||
point.CPUTemperature /= float64(len(metrics.CPU.Temperatures))
|
|
||||||
}
|
}
|
||||||
|
|
||||||
var zfsSize, zfsUsed uint64
|
var zfsSize, zfsUsed uint64
|
||||||
for _, pool := range metrics.ZFS.Pools {
|
for _, pool := range metrics.ZFS.Pools {
|
||||||
zfsSize += pool.SizeBytes
|
zfsSize += pool.SizeBytes
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ func TestHistoryPointFromMetrics(t *testing.T) {
|
|||||||
Guests: GuestsMetrics{Guests: []GuestMetrics{{Status: "running", CPU: .5, MaxCPU: 4, MemoryBytes: 50, MaxMemory: 100}}},
|
Guests: GuestsMetrics{Guests: []GuestMetrics{{Status: "running", CPU: .5, MaxCPU: 4, MemoryBytes: 50, MaxMemory: 100}}},
|
||||||
}
|
}
|
||||||
point := historyPointFromMetrics(metrics)
|
point := historyPointFromMetrics(metrics)
|
||||||
if point.CPUTemperature != 50 || point.GuestsCPU != 50 || point.GuestsMemory != 50 {
|
if point.CPUTemperature != 60 || point.GuestsCPU != 50 || point.GuestsMemory != 50 {
|
||||||
t.Fatalf("неожиданная историческая точка: %+v", point)
|
t.Fatalf("неожиданная историческая точка: %+v", point)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -17,6 +17,51 @@ type MaintenanceWindow struct {
|
|||||||
AllAlerts bool `json:"allAlerts"`
|
AllAlerts bool `json:"allAlerts"`
|
||||||
ServiceIDs []int64 `json:"serviceIds"`
|
ServiceIDs []int64 `json:"serviceIds"`
|
||||||
VMIDs []int `json:"vmids"`
|
VMIDs []int `json:"vmids"`
|
||||||
|
Automatic bool `json:"automatic"`
|
||||||
|
Source string `json:"source,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func automaticBackupMaintenance(activity ProxmoxActivity, links []ServiceLink, now time.Time) MaintenanceWindow {
|
||||||
|
vmids := []int{}
|
||||||
|
backupRunning := false
|
||||||
|
for _, task := range activity.Tasks {
|
||||||
|
if task.Type != "vzdump" || task.EndTime > 0 || (task.Status != "" && !strings.EqualFold(task.Status, "running")) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
backupRunning = true
|
||||||
|
if vmid := backupTaskVMID(task); vmid > 0 {
|
||||||
|
vmids = append(vmids, vmid)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
vmids = uniqueInt(vmids)
|
||||||
|
if !backupRunning {
|
||||||
|
return MaintenanceWindow{}
|
||||||
|
}
|
||||||
|
selected := map[int]bool{}
|
||||||
|
for _, vmid := range vmids {
|
||||||
|
selected[vmid] = true
|
||||||
|
}
|
||||||
|
serviceIDs := []int64{}
|
||||||
|
for _, link := range links {
|
||||||
|
if selected[link.VMID] {
|
||||||
|
serviceIDs = append(serviceIDs, link.ServiceID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return MaintenanceWindow{Enabled: true, Active: true, StartedAt: now.Unix(), EndsAt: now.Add(2 * time.Minute).Unix(), Reason: "Автоматически: выполняется backup VM/LXC", VMIDs: vmids, ServiceIDs: uniqueInt64(serviceIDs), AllAlerts: len(vmids) == 0, Automatic: true, Source: "backup"}
|
||||||
|
}
|
||||||
|
|
||||||
|
func backupTaskVMID(task ProxmoxTask) int {
|
||||||
|
if vmid, err := strconv.Atoi(strings.TrimSpace(task.ID)); err == nil && vmid > 0 {
|
||||||
|
return vmid
|
||||||
|
}
|
||||||
|
parts := strings.Split(task.UPID, ":")
|
||||||
|
for i, part := range parts {
|
||||||
|
if part == "vzdump" && i+1 < len(parts) {
|
||||||
|
vmid, _ := strconv.Atoi(parts[i+1])
|
||||||
|
return vmid
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return 0
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *Store) MaintenanceWindow() MaintenanceWindow {
|
func (s *Store) MaintenanceWindow() MaintenanceWindow {
|
||||||
|
|||||||
@@ -38,6 +38,21 @@ func TestMaintenanceExpiresAutomatically(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunningBackupCreatesScopedAutomaticMaintenance(t *testing.T) {
|
||||||
|
now := time.Now()
|
||||||
|
window := automaticBackupMaintenance(ProxmoxActivity{Tasks: []ProxmoxTask{{Type: "vzdump", ID: "105"}, {Type: "vzdump", ID: "106", EndTime: now.Unix()}}}, []ServiceLink{{ServiceID: 7, VMID: 105}, {ServiceID: 8, VMID: 999}}, now)
|
||||||
|
if !window.Active || !window.Automatic || window.Source != "backup" || len(window.VMIDs) != 1 || window.VMIDs[0] != 105 || len(window.ServiceIDs) != 1 || window.ServiceIDs[0] != 7 {
|
||||||
|
t.Fatalf("неверное автоматическое окно: %+v", window)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunningBackupWithoutVMIDSuppressesAllAlerts(t *testing.T) {
|
||||||
|
window := automaticBackupMaintenance(ProxmoxActivity{Tasks: []ProxmoxTask{{Type: "vzdump", UPID: "unknown"}}}, nil, time.Now())
|
||||||
|
if !window.Active || !window.AllAlerts {
|
||||||
|
t.Fatalf("backup с неизвестной целью должен безопасно подавить алерты: %+v", window)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestPlannedAlertOnlyAppearsInHistory(t *testing.T) {
|
func TestPlannedAlertOnlyAppearsInHistory(t *testing.T) {
|
||||||
store, err := openStore(":memory:")
|
store, err := openStore(":memory:")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
58
network.go
58
network.go
@@ -3,6 +3,7 @@ package main
|
|||||||
import (
|
import (
|
||||||
"bufio"
|
"bufio"
|
||||||
"context"
|
"context"
|
||||||
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"net"
|
"net"
|
||||||
"net/http"
|
"net/http"
|
||||||
@@ -36,6 +37,9 @@ type NetworkHealth struct {
|
|||||||
ExternalIPChanged bool `json:"externalIpChanged"`
|
ExternalIPChanged bool `json:"externalIpChanged"`
|
||||||
DisappearedInterfaces []string `json:"disappearedInterfaces"`
|
DisappearedInterfaces []string `json:"disappearedInterfaces"`
|
||||||
CheckedAt time.Time `json:"checkedAt"`
|
CheckedAt time.Time `json:"checkedAt"`
|
||||||
|
Summary string `json:"summary"`
|
||||||
|
LikelyCause string `json:"likelyCause"`
|
||||||
|
RecommendedAction string `json:"recommendedAction"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type NetworkInterface struct {
|
type NetworkInterface struct {
|
||||||
@@ -46,6 +50,13 @@ type NetworkInterface struct {
|
|||||||
TransmitErrors uint64 `json:"transmitErrors"`
|
TransmitErrors uint64 `json:"transmitErrors"`
|
||||||
ReceiveDrops uint64 `json:"receiveDrops"`
|
ReceiveDrops uint64 `json:"receiveDrops"`
|
||||||
TransmitDrops uint64 `json:"transmitDrops"`
|
TransmitDrops uint64 `json:"transmitDrops"`
|
||||||
|
NewErrors uint64 `json:"newErrors"`
|
||||||
|
NewReceiveDrops uint64 `json:"newReceiveDrops"`
|
||||||
|
NewTransmitDrops uint64 `json:"newTransmitDrops"`
|
||||||
|
FaultCounters map[string]uint64 `json:"faultCounters,omitempty"`
|
||||||
|
NewFaults uint64 `json:"newFaults"`
|
||||||
|
LinkDown bool `json:"linkDown"`
|
||||||
|
NewDrops uint64 `json:"newDrops"`
|
||||||
ReceivePerSec float64 `json:"receivePerSec"`
|
ReceivePerSec float64 `json:"receivePerSec"`
|
||||||
TransmitPerSec float64 `json:"transmitPerSec"`
|
TransmitPerSec float64 `json:"transmitPerSec"`
|
||||||
VMID int `json:"vmid,omitempty"`
|
VMID int `json:"vmid,omitempty"`
|
||||||
@@ -77,13 +88,14 @@ type networkCollector struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func newNetworkCollector(path string) (*networkCollector, error) {
|
func newNetworkCollector(path string) (*networkCollector, error) {
|
||||||
snapshot, _, _, err := readNetworkSnapshot(path)
|
snapshot, _, details, err := readNetworkSnapshot(path)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
_, _, details, _ := readNetworkSnapshot(path)
|
|
||||||
previous := map[string]NetworkInterface{}
|
previous := map[string]NetworkInterface{}
|
||||||
for _, v := range details {
|
for _, v := range details {
|
||||||
|
decorateInterface(&v)
|
||||||
|
collectInterfaceFaults(&v)
|
||||||
previous[v.Name] = v
|
previous[v.Name] = v
|
||||||
}
|
}
|
||||||
known := map[string]bool{}
|
known := map[string]bool{}
|
||||||
@@ -110,10 +122,11 @@ func (c *networkCollector) collect() (NetworkMetrics, error) {
|
|||||||
for i := range details {
|
for i := range details {
|
||||||
v := &details[i]
|
v := &details[i]
|
||||||
decorateInterface(v)
|
decorateInterface(v)
|
||||||
if old, ok := c.previousInterfaces[v.Name]; ok && elapsed > 0 && v.ReceiveBytes >= old.ReceiveBytes && v.TransmitBytes >= old.TransmitBytes {
|
collectInterfaceFaults(v)
|
||||||
v.ReceivePerSec = float64(v.ReceiveBytes-old.ReceiveBytes) / elapsed
|
if old, ok := c.previousInterfaces[v.Name]; ok {
|
||||||
v.TransmitPerSec = float64(v.TransmitBytes-old.TransmitBytes) / elapsed
|
updateInterfaceDeltas(v, old, elapsed)
|
||||||
}
|
}
|
||||||
|
v.LinkDown = interfaceLinkDown(*v)
|
||||||
}
|
}
|
||||||
c.previousInterfaces = map[string]NetworkInterface{}
|
c.previousInterfaces = map[string]NetworkInterface{}
|
||||||
for _, v := range details {
|
for _, v := range details {
|
||||||
@@ -138,12 +151,41 @@ func (c *networkCollector) collect() (NetworkMetrics, error) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
c.knownInterfaces = currentNames
|
c.knownInterfaces = currentNames
|
||||||
|
explainNetworkHealth(&c.health, details)
|
||||||
return NetworkMetrics{
|
return NetworkMetrics{
|
||||||
Interface: strings.Join(interfaceNames, ", "), Interfaces: details, ReceiveBytes: current.received,
|
Interface: strings.Join(interfaceNames, ", "), Interfaces: details, ReceiveBytes: current.received,
|
||||||
TransmitBytes: current.sent, ReceivePerSec: receivedRate, TransmitPerSec: sentRate, Health: c.health,
|
TransmitBytes: current.sent, ReceivePerSec: receivedRate, TransmitPerSec: sentRate, Health: c.health,
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func explainNetworkHealth(health *NetworkHealth, interfaces []NetworkInterface) {
|
||||||
|
health.Summary, health.LikelyCause, health.RecommendedAction = "Сеть работает штатно", "Проблем не обнаружено", "Действий не требуется"
|
||||||
|
switch {
|
||||||
|
case health.Gateway == "":
|
||||||
|
health.Summary, health.LikelyCause, health.RecommendedAction = "Не найден маршрут в локальную сеть", "На хосте отсутствует default gateway", "Проверьте адрес шлюза и конфигурацию bridge/VLAN"
|
||||||
|
case !health.GatewayUp:
|
||||||
|
health.Summary, health.LikelyCause, health.RecommendedAction = "Хост не видит локальный шлюз", "Проблема между Proxmox и роутером: линк, bridge, bond или VLAN", "Проверьте кабель, carrier и настройки vmbr/bond/VLAN"
|
||||||
|
case health.PacketLoss > 0:
|
||||||
|
health.Summary = fmt.Sprintf("Нестабильная локальная сеть: потеряно %.1f%% пакетов", health.PacketLoss)
|
||||||
|
health.LikelyCause, health.RecommendedAction = "Перегрузка или ошибки линка между Proxmox и шлюзом", "Проверьте аппаратные ошибки, кабель, порт коммутатора и duplex"
|
||||||
|
case !health.DNSUp && health.InternetUp:
|
||||||
|
health.Summary, health.LikelyCause, health.RecommendedAction = "Интернет доступен, но DNS не отвечает", "Проблема DNS-сервера или /etc/resolv.conf", "Проверьте настроенные DNS и попробуйте разрешить домен вручную"
|
||||||
|
case health.DNSUp && !health.InternetUp:
|
||||||
|
health.Summary, health.LikelyCause, health.RecommendedAction = "Локальная сеть и DNS работают, но внешняя проверка недоступна", "Проблема выше роутера, блокировка HTTPS или недоступность провайдера", "Проверьте маршрут наружу, firewall и доступ к внешним адресам"
|
||||||
|
}
|
||||||
|
for _, v := range interfaces {
|
||||||
|
if v.LinkDown {
|
||||||
|
health.Summary, health.LikelyCause, health.RecommendedAction = v.Name+": линк не поднят", "Нет carrier либо интерфейс административно выключен", "Проверьте кабель, порт коммутатора и состояние интерфейса"
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if v.NewErrors+v.NewFaults > 0 {
|
||||||
|
health.Summary = fmt.Sprintf("%s: за последний интервал появилось %d изменений счётчиков сетевых ошибок", v.Name, v.NewErrors+v.NewFaults)
|
||||||
|
health.LikelyCause, health.RecommendedAction = "Счётчики интерфейса сейчас растут из-за перегрузки, кабеля, MTU или duplex", "Проверьте скорость и duplex, кабель, порт коммутатора и MTU на обоих концах"
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func decorateInterface(v *NetworkInterface) {
|
func decorateInterface(v *NetworkInterface) {
|
||||||
base := filepath.Join("/sys/class/net", v.Name)
|
base := filepath.Join("/sys/class/net", v.Name)
|
||||||
v.State = readTrimmed(filepath.Join(base, "operstate"))
|
v.State = readTrimmed(filepath.Join(base, "operstate"))
|
||||||
@@ -154,6 +196,10 @@ func decorateInterface(v *NetworkInterface) {
|
|||||||
v.Master = filepath.Base(target)
|
v.Master = filepath.Base(target)
|
||||||
}
|
}
|
||||||
switch {
|
switch {
|
||||||
|
case strings.HasPrefix(v.Name, "fwbr"):
|
||||||
|
v.Kind = "Firewall bridge"
|
||||||
|
case strings.HasPrefix(v.Name, "fwpr") || strings.HasPrefix(v.Name, "fwln"):
|
||||||
|
v.Kind = "Firewall link"
|
||||||
case strings.HasPrefix(v.Name, "vmbr"):
|
case strings.HasPrefix(v.Name, "vmbr"):
|
||||||
v.Kind = "Bridge"
|
v.Kind = "Bridge"
|
||||||
case strings.HasPrefix(v.Name, "bond"):
|
case strings.HasPrefix(v.Name, "bond"):
|
||||||
@@ -232,7 +278,7 @@ func readNetworkSnapshot(path string) (networkSnapshot, []string, []NetworkInter
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
fields := strings.Fields(values)
|
fields := strings.Fields(values)
|
||||||
if len(fields) < 9 {
|
if len(fields) < 16 {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
rx, errRX := strconv.ParseUint(fields[0], 10, 64)
|
rx, errRX := strconv.ParseUint(fields[0], 10, 64)
|
||||||
|
|||||||
103
network_faults.go
Normal file
103
network_faults.go
Normal file
@@ -0,0 +1,103 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func counterDelta(current, previous uint64) uint64 {
|
||||||
|
if current < previous {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
return current - previous
|
||||||
|
}
|
||||||
|
|
||||||
|
func updateInterfaceDeltas(v *NetworkInterface, old NetworkInterface, elapsed float64) {
|
||||||
|
if elapsed <= 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
v.ReceivePerSec = float64(counterDelta(v.ReceiveBytes, old.ReceiveBytes)) / elapsed
|
||||||
|
v.TransmitPerSec = float64(counterDelta(v.TransmitBytes, old.TransmitBytes)) / elapsed
|
||||||
|
v.NewErrors = counterDelta(v.ReceiveErrors, old.ReceiveErrors) + counterDelta(v.TransmitErrors, old.TransmitErrors)
|
||||||
|
v.NewReceiveDrops = counterDelta(v.ReceiveDrops, old.ReceiveDrops)
|
||||||
|
v.NewTransmitDrops = counterDelta(v.TransmitDrops, old.TransmitDrops)
|
||||||
|
v.NewDrops = v.NewReceiveDrops + v.NewTransmitDrops
|
||||||
|
for name, value := range v.FaultCounters {
|
||||||
|
if previous, ok := old.FaultCounters[name]; ok {
|
||||||
|
v.NewFaults += counterDelta(value, previous)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// A transition matters even if the interface has not carried traffic yet.
|
||||||
|
v.LinkDown = old.Carrier && !v.Carrier
|
||||||
|
}
|
||||||
|
|
||||||
|
func interfaceLinkDown(v NetworkInterface) bool {
|
||||||
|
if v.Kind == "Firewall bridge" || v.Kind == "Firewall link" {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
if v.LinkDown {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
used := v.Master != "" || v.ReceiveBytes+v.TransmitBytes > 0
|
||||||
|
if v.Kind == "Physical" {
|
||||||
|
return used && (!v.Carrier || v.State == "down")
|
||||||
|
}
|
||||||
|
return used && v.State == "down"
|
||||||
|
}
|
||||||
|
|
||||||
|
func collectInterfaceFaults(v *NetworkInterface) {
|
||||||
|
base := filepath.Join("/sys/class/net", v.Name)
|
||||||
|
// /proc/net/dev folds missed packets into RX drop. Sysfs keeps software
|
||||||
|
// drops separate from rx_missed_errors, which must still trigger an alert.
|
||||||
|
if n, err := strconv.ParseUint(readTrimmed(filepath.Join(base, "statistics/rx_dropped")), 10, 64); err == nil {
|
||||||
|
v.ReceiveDrops = n
|
||||||
|
}
|
||||||
|
v.FaultCounters = map[string]uint64{}
|
||||||
|
for _, name := range []string{"rx_crc_errors", "rx_frame_errors", "rx_fifo_errors", "rx_missed_errors", "tx_fifo_errors", "tx_carrier_errors", "carrier_down_count"} {
|
||||||
|
path := filepath.Join(base, "statistics", name)
|
||||||
|
if name == "carrier_down_count" {
|
||||||
|
path = filepath.Join(base, name)
|
||||||
|
}
|
||||||
|
if n, err := strconv.ParseUint(readTrimmed(path), 10, 64); err == nil {
|
||||||
|
v.FaultCounters[name] = n
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Only Atlantic exposes the driver-specific names below. No configuration
|
||||||
|
// changes, and a missing ethtool is not itself a network failure.
|
||||||
|
driver, err := filepath.EvalSymlinks(filepath.Join(base, "device/driver"))
|
||||||
|
if err != nil || filepath.Base(driver) != "atlantic" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
|
||||||
|
defer cancel()
|
||||||
|
if out, err := exec.CommandContext(ctx, "ethtool", "-S", v.Name).Output(); err == nil {
|
||||||
|
for k, n := range parseAtlanticFaults(string(out)) {
|
||||||
|
v.FaultCounters["atlantic:"+k] = n
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseAtlanticFaults(output string) map[string]uint64 {
|
||||||
|
result := map[string]uint64{}
|
||||||
|
for _, line := range strings.Split(output, "\n") {
|
||||||
|
key, value, ok := strings.Cut(strings.TrimSpace(line), ":")
|
||||||
|
if !ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
fields := strings.Fields(key)
|
||||||
|
if len(fields) == 0 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
switch fields[len(fields)-1] {
|
||||||
|
case "InErrors", "InDroppedDma", "AllocFails", "SkbAllocFails", "Restarts":
|
||||||
|
if n, err := strconv.ParseUint(strings.TrimSpace(value), 10, 64); err == nil {
|
||||||
|
result[key] = n
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return result
|
||||||
|
}
|
||||||
67
network_faults_test.go
Normal file
67
network_faults_test.go
Normal file
@@ -0,0 +1,67 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestNetworkDropsDoNotAlert(t *testing.T) {
|
||||||
|
for _, name := range []string{"enp6s0", "vmbr0", "fwbr100i0", "tap100i0"} {
|
||||||
|
v := NetworkInterface{Name: name, Carrier: true, State: "up", NewDrops: 15, NewReceiveDrops: 15}
|
||||||
|
metrics := dashboardMetrics{Network: NetworkMetrics{Interfaces: []NetworkInterface{v}}}
|
||||||
|
for _, a := range evaluateAlerts(metrics, AlertThresholds{}) {
|
||||||
|
if a.ID == "network-errors" {
|
||||||
|
t.Fatalf("%s: drops triggered alert", name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
v.NewErrors = 1
|
||||||
|
metrics.Network.Interfaces[0] = v
|
||||||
|
found := false
|
||||||
|
for _, a := range evaluateAlerts(metrics, AlertThresholds{}) {
|
||||||
|
if a.ID == "network-errors" {
|
||||||
|
found = a.Severity == "critical"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
t.Fatalf("%s: errors did not trigger critical alert", name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNetworkHardwareFaultAndLinkAlerts(t *testing.T) {
|
||||||
|
for _, v := range []NetworkInterface{{Name: "eth0", NewFaults: 1}, {Name: "eth0", LinkDown: true}} {
|
||||||
|
found := false
|
||||||
|
for _, a := range evaluateAlerts(dashboardMetrics{Network: NetworkMetrics{Interfaces: []NetworkInterface{v}}}, AlertThresholds{}) {
|
||||||
|
if strings.HasPrefix(a.ID, "network-") && a.Severity == "critical" {
|
||||||
|
found = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
t.Fatalf("no critical alert for %+v", v)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNetworkCounterResetsAndFaultBaselines(t *testing.T) {
|
||||||
|
old := NetworkInterface{ReceiveBytes: 100, ReceiveErrors: 20, TransmitErrors: 4, ReceiveDrops: 8, FaultCounters: map[string]uint64{"rx_missed_errors": 10, "atlantic:InErrors": 9}, Carrier: true}
|
||||||
|
next := NetworkInterface{ReceiveBytes: 1, ReceiveErrors: 1, TransmitErrors: 6, ReceiveDrops: 10, FaultCounters: map[string]uint64{"rx_missed_errors": 12, "atlantic:InErrors": 1, "new": 100}, Carrier: false}
|
||||||
|
updateInterfaceDeltas(&next, old, 2)
|
||||||
|
if next.NewErrors != 2 || next.NewFaults != 2 || next.NewReceiveDrops != 2 || next.ReceivePerSec != 0 || !next.LinkDown {
|
||||||
|
t.Fatalf("bad deltas: %+v", next)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAtlanticFaults(t *testing.T) {
|
||||||
|
got := parseAtlanticFaults("NIC statistics:\n InErrors: 2\n InDroppedDma: 3\n Queue[0] AllocFails: 4\n Queue[1] SkbAllocFails: 5\n Queue[0] Restarts: 6\n Queue[0] InErrors: 7\n InPackets: 100\n Queue[0] XdpDrop: 50\n InErrorsBad: nope")
|
||||||
|
if len(got) != 6 || got["InDroppedDma"] != 3 || got["Queue[0] Restarts"] != 6 {
|
||||||
|
t.Fatalf("bad counters: %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFirewallNotAttributedToGuest(t *testing.T) {
|
||||||
|
n := NetworkMetrics{Interfaces: []NetworkInterface{{Name: "fwbr100i0", ReceiveDrops: 100}, {Name: "tap100i0"}}}
|
||||||
|
decorateNetworkInterfaces(&n, GuestsMetrics{})
|
||||||
|
if n.Interfaces[0].VMID != 0 || n.Interfaces[1].VMID != 100 {
|
||||||
|
t.Fatalf("bad VM attribution: %+v", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -3,6 +3,7 @@ package main
|
|||||||
import (
|
import (
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -33,8 +34,28 @@ func TestReadNetworkSnapshot(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestExplainNetworkHealthDistinguishesDNSFailure(t *testing.T) {
|
||||||
|
health := NetworkHealth{Gateway: "192.168.0.1", GatewayUp: true, InternetUp: true, DNSUp: false}
|
||||||
|
explainNetworkHealth(&health, nil)
|
||||||
|
if !strings.Contains(health.Summary, "DNS") || health.RecommendedAction == "" {
|
||||||
|
t.Fatalf("непонятная диагностика: %+v", health)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExplainNetworkHealthIgnoresOldInterfaceCounters(t *testing.T) {
|
||||||
|
health := NetworkHealth{Gateway: "192.168.0.1", GatewayUp: true, InternetUp: true, DNSUp: true}
|
||||||
|
explainNetworkHealth(&health, []NetworkInterface{{Name: "eno1", Carrier: true, State: "up", ReceiveDrops: 42}})
|
||||||
|
if health.Summary != "Сеть работает штатно" {
|
||||||
|
t.Fatalf("старый накопленный счётчик вызвал текущую тревогу: %+v", health)
|
||||||
|
}
|
||||||
|
explainNetworkHealth(&health, []NetworkInterface{{Name: "eno1", Carrier: true, State: "up", ReceiveDrops: 43, NewDrops: 1}})
|
||||||
|
if health.Summary != "Сеть работает штатно" {
|
||||||
|
t.Fatalf("software drops вызвали тревогу: %+v", health)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestInterfaceVMID(t *testing.T) {
|
func TestInterfaceVMID(t *testing.T) {
|
||||||
for name, want := range map[string]int{"tap100i0": 100, "veth108i0": 108, "fwbr113i0": 113, "vmbr0": 0, "enp6s0": 0} {
|
for name, want := range map[string]int{"tap100i0": 100, "veth108i0": 108, "fwbr113i0": 0, "vmbr0": 0, "enp6s0": 0} {
|
||||||
if got := interfaceVMID(name); got != want {
|
if got := interfaceVMID(name); got != want {
|
||||||
t.Fatalf("%s: got %d want %d", name, got, want)
|
t.Fatalf("%s: got %d want %d", name, got, want)
|
||||||
}
|
}
|
||||||
|
|||||||
17
scripts/build-release.sh
Executable file
17
scripts/build-release.sh
Executable file
@@ -0,0 +1,17 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
set -eu
|
||||||
|
version=${1:?Usage: build-release.sh vX.Y.Z}
|
||||||
|
cd "$(dirname "$0")/.."
|
||||||
|
go test ./...
|
||||||
|
mkdir -p dist
|
||||||
|
GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -trimpath -ldflags "-s -w -X main.version=$version" -o dist/proxmox-dashboard-linux-amd64 .
|
||||||
|
(cd dist && shasum -a 256 proxmox-dashboard-linux-amd64 > proxmox-dashboard-linux-amd64.sha256)
|
||||||
|
stage=$(mktemp -d)
|
||||||
|
trap 'rm -rf "$stage"' EXIT INT TERM
|
||||||
|
package="$stage/proxmox-dashboard-install"
|
||||||
|
mkdir -p "$package/dist" "$package/scripts"
|
||||||
|
cp dist/proxmox-dashboard-linux-amd64 dist/proxmox-dashboard-linux-amd64.sha256 "$package/dist/"
|
||||||
|
cp scripts/install.sh scripts/install-agent.sh scripts/uninstall-agent.sh scripts/proxmox-dashboard-update "$package/scripts/"
|
||||||
|
cp assets/NOTICE.md "$package/MAP-DATA-NOTICE.md"
|
||||||
|
cp ./*.service ./*.timer ./*.path README.md "$package/"
|
||||||
|
tar -czf dist/proxmox-dashboard-install-amd64.tar.gz -C "$stage" proxmox-dashboard-install
|
||||||
110
service_links.go
Normal file
110
service_links.go
Normal file
@@ -0,0 +1,110 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
type ServiceLink struct {
|
||||||
|
ServiceID int64 `json:"serviceId"`
|
||||||
|
AgentID string `json:"agentId"`
|
||||||
|
VMID int `json:"vmid"`
|
||||||
|
ComponentType string `json:"componentType"`
|
||||||
|
ComponentName string `json:"componentName"`
|
||||||
|
Port int `json:"port"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Store) ServiceLinks() ([]ServiceLink, error) {
|
||||||
|
rows, err := s.db.Query(`SELECT service_id,agent_id,vmid,component_type,component_name,port FROM service_links ORDER BY service_id`)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
var out []ServiceLink
|
||||||
|
for rows.Next() {
|
||||||
|
var v ServiceLink
|
||||||
|
if err = rows.Scan(&v.ServiceID, &v.AgentID, &v.VMID, &v.ComponentType, &v.ComponentName, &v.Port); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out = append(out, v)
|
||||||
|
}
|
||||||
|
return out, rows.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Store) SaveServiceLink(v ServiceLink) (ServiceLink, error) {
|
||||||
|
v.AgentID = strings.TrimSpace(v.AgentID)
|
||||||
|
v.ComponentType = strings.TrimSpace(v.ComponentType)
|
||||||
|
v.ComponentName = strings.TrimSpace(v.ComponentName)
|
||||||
|
if v.ServiceID <= 0 {
|
||||||
|
return v, fmt.Errorf("выберите сервис")
|
||||||
|
}
|
||||||
|
if v.AgentID == "" && v.VMID <= 0 {
|
||||||
|
return v, fmt.Errorf("выберите агент или VMID")
|
||||||
|
}
|
||||||
|
if v.ComponentType != "" && v.ComponentType != "systemd" && v.ComponentType != "docker" && v.ComponentType != "process" {
|
||||||
|
return v, fmt.Errorf("неизвестный тип компонента")
|
||||||
|
}
|
||||||
|
if v.Port < 0 || v.Port > 65535 {
|
||||||
|
return v, fmt.Errorf("некорректный порт")
|
||||||
|
}
|
||||||
|
_, err := s.db.Exec(`INSERT INTO service_links(service_id,agent_id,vmid,component_type,component_name,port) VALUES(?,?,?,?,?,?) ON CONFLICT(service_id) DO UPDATE SET agent_id=excluded.agent_id,vmid=excluded.vmid,component_type=excluded.component_type,component_name=excluded.component_name,port=excluded.port`, v.ServiceID, v.AgentID, v.VMID, v.ComponentType, v.ComponentName, v.Port)
|
||||||
|
return v, err
|
||||||
|
}
|
||||||
|
func (s *Store) DeleteServiceLink(serviceID int64) error {
|
||||||
|
_, err := s.db.Exec(`DELETE FROM service_links WHERE service_id=?`, serviceID)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
func groupLinkedAlerts(metrics dashboardMetrics, alerts []Alert) []Alert {
|
||||||
|
if len(metrics.ServiceLinks) == 0 {
|
||||||
|
return alerts
|
||||||
|
}
|
||||||
|
agentOffline := map[string]ManagedAgent{}
|
||||||
|
for _, agent := range metrics.Agents {
|
||||||
|
if !agent.Online {
|
||||||
|
agentOffline[agent.ID] = agent
|
||||||
|
}
|
||||||
|
}
|
||||||
|
affected := map[string][]string{}
|
||||||
|
remove := map[string]bool{}
|
||||||
|
for _, link := range metrics.ServiceLinks {
|
||||||
|
agent, offline := agentOffline[link.AgentID]
|
||||||
|
if !offline {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
for _, service := range metrics.Services.Services {
|
||||||
|
if service.ID != link.ServiceID {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
for _, endpoint := range service.Endpoints {
|
||||||
|
if !endpoint.Up {
|
||||||
|
key := fmt.Sprintf("service-%d-%s", service.ID, endpoint.Kind)
|
||||||
|
remove[key] = true
|
||||||
|
affected[agent.ID] = append(affected[agent.ID], service.Name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(affected) == 0 {
|
||||||
|
return alerts
|
||||||
|
}
|
||||||
|
out := make([]Alert, 0, len(alerts))
|
||||||
|
for _, alert := range alerts {
|
||||||
|
skip := remove[alert.ID]
|
||||||
|
for agentID := range affected {
|
||||||
|
if alert.ID == "agent-offline-"+agentID+fmt.Sprintf("-%d", agentOffline[agentID].VMID) {
|
||||||
|
skip = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !skip {
|
||||||
|
out = append(out, alert)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for agentID, names := range affected {
|
||||||
|
sort.Strings(names)
|
||||||
|
agent := agentOffline[agentID]
|
||||||
|
out = append(out, Alert{ID: "group-agent-offline-" + agentID + fmt.Sprintf("-%d", agent.VMID), Severity: "critical", Source: "Корреляция", Title: agent.Name + ": агент недоступен, связанные сервисы не отвечают", Message: fmt.Sprintf("VMID %d. Затронуто сервисов: %d — %s. Вероятная первопричина: недоступна гостевая система или агент.", agent.VMID, len(names), strings.Join(names, ", "))})
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
4
store.go
4
store.go
@@ -106,6 +106,7 @@ func openStore(path string) (*Store, error) {
|
|||||||
}
|
}
|
||||||
db.SetMaxOpenConns(1)
|
db.SetMaxOpenConns(1)
|
||||||
statements := []string{
|
statements := []string{
|
||||||
|
trafficSchema,
|
||||||
`PRAGMA journal_mode=WAL`,
|
`PRAGMA journal_mode=WAL`,
|
||||||
`PRAGMA busy_timeout=5000`,
|
`PRAGMA busy_timeout=5000`,
|
||||||
`CREATE TABLE IF NOT EXISTS settings (key TEXT PRIMARY KEY, value TEXT NOT NULL)`,
|
`CREATE TABLE IF NOT EXISTS settings (key TEXT PRIMARY KEY, value TEXT NOT NULL)`,
|
||||||
@@ -140,6 +141,9 @@ func openStore(path string) (*Store, error) {
|
|||||||
`CREATE INDEX IF NOT EXISTS entity_history_kind_ts_idx ON entity_history(kind,ts)`,
|
`CREATE INDEX IF NOT EXISTS entity_history_kind_ts_idx ON entity_history(kind,ts)`,
|
||||||
`CREATE TABLE IF NOT EXISTS agent_enrollment_tokens (token_hash TEXT PRIMARY KEY, created_at INTEGER NOT NULL, expires_at INTEGER NOT NULL, used_at INTEGER)`,
|
`CREATE TABLE IF NOT EXISTS agent_enrollment_tokens (token_hash TEXT PRIMARY KEY, created_at INTEGER NOT NULL, expires_at INTEGER NOT NULL, used_at INTEGER)`,
|
||||||
`CREATE TABLE IF NOT EXISTS agents (id TEXT PRIMARY KEY, name TEXT NOT NULL, vmid INTEGER NOT NULL DEFAULT 0, hostname TEXT NOT NULL DEFAULT '', machine_id TEXT NOT NULL DEFAULT '', secret_hash TEXT NOT NULL, enrolled_at INTEGER NOT NULL, last_seen INTEGER NOT NULL, version TEXT NOT NULL DEFAULT '', report_json TEXT NOT NULL DEFAULT '{}', remote_address TEXT NOT NULL DEFAULT '')`,
|
`CREATE TABLE IF NOT EXISTS agents (id TEXT PRIMARY KEY, name TEXT NOT NULL, vmid INTEGER NOT NULL DEFAULT 0, hostname TEXT NOT NULL DEFAULT '', machine_id TEXT NOT NULL DEFAULT '', secret_hash TEXT NOT NULL, enrolled_at INTEGER NOT NULL, last_seen INTEGER NOT NULL, version TEXT NOT NULL DEFAULT '', report_json TEXT NOT NULL DEFAULT '{}', remote_address TEXT NOT NULL DEFAULT '')`,
|
||||||
|
`CREATE TABLE IF NOT EXISTS agent_actions (id INTEGER PRIMARY KEY AUTOINCREMENT, agent_id TEXT NOT NULL, action TEXT NOT NULL, target TEXT NOT NULL DEFAULT '', requested_at INTEGER NOT NULL, started_at INTEGER, completed_at INTEGER, status TEXT NOT NULL DEFAULT 'queued', result TEXT NOT NULL DEFAULT '', FOREIGN KEY(agent_id) REFERENCES agents(id) ON DELETE CASCADE)`,
|
||||||
|
`CREATE INDEX IF NOT EXISTS agent_actions_agent_status_idx ON agent_actions(agent_id,status)`,
|
||||||
|
`CREATE TABLE IF NOT EXISTS service_links (service_id INTEGER PRIMARY KEY, agent_id TEXT NOT NULL DEFAULT '', vmid INTEGER NOT NULL DEFAULT 0, component_type TEXT NOT NULL DEFAULT '', component_name TEXT NOT NULL DEFAULT '', port INTEGER NOT NULL DEFAULT 0, FOREIGN KEY(service_id) REFERENCES services(id) ON DELETE CASCADE)`,
|
||||||
}
|
}
|
||||||
for _, statement := range statements {
|
for _, statement := range statements {
|
||||||
if _, err := db.Exec(statement); err != nil {
|
if _, err := db.Exec(statement); err != nil {
|
||||||
|
|||||||
298
traffic.go
Normal file
298
traffic.go
Normal file
@@ -0,0 +1,298 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
type TrafficEvent struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
Sequence int64 `json:"sequence,omitempty"`
|
||||||
|
Time int64 `json:"time"`
|
||||||
|
Kind string `json:"kind"`
|
||||||
|
IP string `json:"ip"`
|
||||||
|
Domain string `json:"domain,omitempty"`
|
||||||
|
Method string `json:"method,omitempty"`
|
||||||
|
Path string `json:"path,omitempty"`
|
||||||
|
Status int `json:"status,omitempty"`
|
||||||
|
Reason string `json:"reason,omitempty"`
|
||||||
|
Country string `json:"country,omitempty"`
|
||||||
|
Agent string `json:"agent,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type TrafficBatch struct {
|
||||||
|
Events []TrafficEvent `json:"events"`
|
||||||
|
Files int `json:"files"`
|
||||||
|
BacklogBytes int64 `json:"backlogBytes"`
|
||||||
|
CrowdSecLimited bool `json:"crowdsecLimited"`
|
||||||
|
ParseErrors uint64 `json:"parseErrors"`
|
||||||
|
LogError string `json:"logError,omitempty"`
|
||||||
|
CrowdSecError string `json:"crowdsecError,omitempty"`
|
||||||
|
CrowdSecAt int64 `json:"crowdsecAt"`
|
||||||
|
StartedAt int64 `json:"startedAt"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type TrafficSource struct {
|
||||||
|
Agent string `json:"agent"`
|
||||||
|
Name string `json:"name"`
|
||||||
|
LastSeen int64 `json:"lastSeen"`
|
||||||
|
Status TrafficBatch `json:"status"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type TrafficCount struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
Count int64 `json:"count"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type TrafficView struct {
|
||||||
|
Events []TrafficEvent `json:"events"`
|
||||||
|
Sources []TrafficSource `json:"sources"`
|
||||||
|
Countries []TrafficCount `json:"countries"`
|
||||||
|
Domains []TrafficCount `json:"domains"`
|
||||||
|
IPs []TrafficCount `json:"ips"`
|
||||||
|
Requests int64 `json:"requests"`
|
||||||
|
UniqueIPs int64 `json:"uniqueIps"`
|
||||||
|
Alerts int64 `json:"alerts"`
|
||||||
|
Decisions int64 `json:"decisions"`
|
||||||
|
QPS float64 `json:"qps"`
|
||||||
|
Series []TrafficCount `json:"series"`
|
||||||
|
GeoReady bool `json:"geoReady"`
|
||||||
|
GeoDate string `json:"geoDate"`
|
||||||
|
Now int64 `json:"now"`
|
||||||
|
Settings TrafficSettings `json:"settings"`
|
||||||
|
Limited bool `json:"limited"`
|
||||||
|
}
|
||||||
|
|
||||||
|
const trafficSchema = `CREATE TABLE IF NOT EXISTS traffic_events (
|
||||||
|
seq INTEGER PRIMARY KEY AUTOINCREMENT, agent TEXT NOT NULL, event_id TEXT NOT NULL,
|
||||||
|
ts INTEGER NOT NULL, kind TEXT NOT NULL, ip TEXT NOT NULL, domain TEXT NOT NULL,
|
||||||
|
method TEXT NOT NULL, path TEXT NOT NULL, status INTEGER NOT NULL, reason TEXT NOT NULL,
|
||||||
|
country TEXT NOT NULL, UNIQUE(agent,event_id));
|
||||||
|
CREATE INDEX IF NOT EXISTS traffic_ts ON traffic_events(ts);
|
||||||
|
CREATE INDEX IF NOT EXISTS traffic_filter ON traffic_events(domain,kind,ts);
|
||||||
|
CREATE TABLE IF NOT EXISTS traffic_sources(agent TEXT PRIMARY KEY, last_seen INTEGER NOT NULL, status_json TEXT NOT NULL);`
|
||||||
|
|
||||||
|
func (s *Store) SaveTraffic(agent string, batch TrafficBatch) error {
|
||||||
|
if len(batch.Events) > 500 {
|
||||||
|
return errors.New("too many traffic events")
|
||||||
|
}
|
||||||
|
now := time.Now().Unix()
|
||||||
|
tx, err := s.db.Begin()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer tx.Rollback()
|
||||||
|
stmt, err := tx.Prepare(`INSERT OR IGNORE INTO traffic_events(agent,event_id,ts,kind,ip,domain,method,path,status,reason,country) VALUES(?,?,?,?,?,?,?,?,?,?,?)`)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer stmt.Close()
|
||||||
|
for _, e := range batch.Events {
|
||||||
|
if e.Kind != "request" && e.Kind != "alert" && e.Kind != "decision" {
|
||||||
|
return errors.New("invalid traffic kind")
|
||||||
|
}
|
||||||
|
if net.ParseIP(e.IP) == nil || len(e.ID) == 0 || len(e.ID) > 160 {
|
||||||
|
return errors.New("invalid traffic event")
|
||||||
|
}
|
||||||
|
if e.Time < now-86400 || e.Time > now+300 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
country := lookupTrafficCountry(e.IP)
|
||||||
|
_, err = stmt.Exec(agent, e.ID, e.Time, e.Kind, e.IP, trafficText(e.Domain, 253), trafficText(e.Method, 16), trafficPath(e.Path), e.Status, trafficText(e.Reason, 256), country)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
batch.Events = nil
|
||||||
|
raw, err := json.Marshal(batch)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if _, err = tx.Exec(`INSERT INTO traffic_sources(agent,last_seen,status_json) VALUES(?,?,?) ON CONFLICT(agent) DO UPDATE SET last_seen=excluded.last_seen,status_json=excluded.status_json`, agent, now, string(raw)); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
// Bound both age and total rows, including a high-traffic installation.
|
||||||
|
if _, err = tx.Exec(`DELETE FROM traffic_events WHERE ts<? OR seq <= COALESCE((SELECT seq FROM traffic_events ORDER BY seq DESC LIMIT 1 OFFSET 200000),0)`, now-86400); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return tx.Commit()
|
||||||
|
}
|
||||||
|
|
||||||
|
func trafficText(s string, max int) string {
|
||||||
|
s = strings.Map(func(r rune) rune {
|
||||||
|
if r < 32 || r == 127 {
|
||||||
|
return -1
|
||||||
|
}
|
||||||
|
return r
|
||||||
|
}, s)
|
||||||
|
r := []rune(s)
|
||||||
|
if len(r) > max {
|
||||||
|
return string(r[:max])
|
||||||
|
}
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
func trafficPath(s string) string {
|
||||||
|
s = strings.SplitN(s, "?", 2)[0]
|
||||||
|
s = strings.SplitN(s, "#", 2)[0]
|
||||||
|
return trafficText(s, 256)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Store) TrafficView(domain, kind string, external bool) (TrafficView, error) {
|
||||||
|
now := time.Now().Unix()
|
||||||
|
v := TrafficView{Events: []TrafficEvent{}, Sources: []TrafficSource{}, Countries: []TrafficCount{}, Domains: []TrafficCount{}, IPs: []TrafficCount{}, Series: []TrafficCount{}, Now: now, GeoReady: trafficGeoReady(), GeoDate: trafficGeoDate}
|
||||||
|
v.Settings = s.TrafficSettings()
|
||||||
|
where := ` WHERE ts>=? AND agent=?`
|
||||||
|
args := []any{now - 86400, v.Settings.AgentID}
|
||||||
|
if external {
|
||||||
|
where += ` AND country!='LAN'`
|
||||||
|
}
|
||||||
|
if domain != "" {
|
||||||
|
where += ` AND domain=?`
|
||||||
|
args = append(args, domain)
|
||||||
|
}
|
||||||
|
if kind != "" {
|
||||||
|
where += ` AND kind=?`
|
||||||
|
args = append(args, kind)
|
||||||
|
}
|
||||||
|
err := s.db.QueryRow(`SELECT COALESCE(SUM(kind='request'),0),COUNT(DISTINCT CASE WHEN kind='request' THEN ip END),COALESCE(SUM(kind='alert'),0),COALESCE(SUM(kind='decision'),0),COALESCE(SUM(kind='request' AND ts>=?),0)/60.0 FROM traffic_events`+where, append([]any{now - 60}, args...)...).Scan(&v.Requests, &v.UniqueIPs, &v.Alerts, &v.Decisions, &v.QPS)
|
||||||
|
if err != nil {
|
||||||
|
return v, err
|
||||||
|
}
|
||||||
|
rows, err := s.db.Query(`SELECT seq,event_id,ts,kind,ip,domain,method,path,status,reason,country,agent FROM traffic_events`+where+` ORDER BY ts DESC,seq DESC LIMIT 100`, args...)
|
||||||
|
if err != nil {
|
||||||
|
return v, err
|
||||||
|
}
|
||||||
|
for rows.Next() {
|
||||||
|
var e TrafficEvent
|
||||||
|
if err = rows.Scan(&e.Sequence, &e.ID, &e.Time, &e.Kind, &e.IP, &e.Domain, &e.Method, &e.Path, &e.Status, &e.Reason, &e.Country, &e.Agent); err != nil {
|
||||||
|
rows.Close()
|
||||||
|
return v, err
|
||||||
|
}
|
||||||
|
v.Events = append(v.Events, e)
|
||||||
|
}
|
||||||
|
err = rows.Err()
|
||||||
|
rows.Close()
|
||||||
|
if err != nil {
|
||||||
|
return v, err
|
||||||
|
}
|
||||||
|
for _, q := range []struct {
|
||||||
|
column string
|
||||||
|
out *[]TrafficCount
|
||||||
|
}{{"country", &v.Countries}, {"domain", &v.Domains}, {"ip", &v.IPs}} {
|
||||||
|
r, e := s.db.Query(`SELECT `+q.column+`,COUNT(*) FROM traffic_events`+where+` GROUP BY `+q.column+` ORDER BY COUNT(*) DESC LIMIT 50`, args...)
|
||||||
|
if e != nil {
|
||||||
|
return v, e
|
||||||
|
}
|
||||||
|
for r.Next() {
|
||||||
|
var c TrafficCount
|
||||||
|
if e = r.Scan(&c.Name, &c.Count); e != nil {
|
||||||
|
r.Close()
|
||||||
|
return v, e
|
||||||
|
}
|
||||||
|
*q.out = append(*q.out, c)
|
||||||
|
}
|
||||||
|
e = r.Err()
|
||||||
|
r.Close()
|
||||||
|
if e != nil {
|
||||||
|
return v, e
|
||||||
|
}
|
||||||
|
}
|
||||||
|
r, err := s.db.Query(`SELECT CAST((ts/60)*60 AS TEXT),COUNT(*) FROM traffic_events`+where+` AND kind='request' AND ts>=? GROUP BY ts/60 ORDER BY ts/60`, append(args, now-3600)...)
|
||||||
|
if err != nil {
|
||||||
|
return v, err
|
||||||
|
}
|
||||||
|
for r.Next() {
|
||||||
|
var c TrafficCount
|
||||||
|
if err = r.Scan(&c.Name, &c.Count); err != nil {
|
||||||
|
r.Close()
|
||||||
|
return v, err
|
||||||
|
}
|
||||||
|
v.Series = append(v.Series, c)
|
||||||
|
}
|
||||||
|
err = r.Err()
|
||||||
|
r.Close()
|
||||||
|
if err != nil {
|
||||||
|
return v, err
|
||||||
|
}
|
||||||
|
r, err = s.db.Query(`SELECT t.agent,COALESCE(a.name,t.agent),t.last_seen,t.status_json FROM traffic_sources t LEFT JOIN agents a ON a.id=t.agent WHERE t.agent=? ORDER BY t.last_seen DESC`, v.Settings.AgentID)
|
||||||
|
if err != nil {
|
||||||
|
return v, err
|
||||||
|
}
|
||||||
|
for r.Next() {
|
||||||
|
var src TrafficSource
|
||||||
|
var raw string
|
||||||
|
if err = r.Scan(&src.Agent, &src.Name, &src.LastSeen, &raw); err != nil {
|
||||||
|
r.Close()
|
||||||
|
return v, err
|
||||||
|
}
|
||||||
|
json.Unmarshal([]byte(raw), &src.Status)
|
||||||
|
v.Sources = append(v.Sources, src)
|
||||||
|
}
|
||||||
|
err = r.Err()
|
||||||
|
r.Close()
|
||||||
|
var count int64
|
||||||
|
s.db.QueryRow(`SELECT COUNT(*) FROM traffic_events`).Scan(&count)
|
||||||
|
v.Limited = count >= 200000
|
||||||
|
return v, err
|
||||||
|
}
|
||||||
|
|
||||||
|
func trafficRoutes(mux *http.ServeMux, s *Store) {
|
||||||
|
trafficSettingsRoute(mux, s)
|
||||||
|
mux.HandleFunc("/traffic", func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if !s.TrafficSettings().Enabled {
|
||||||
|
http.Error(w, "Карта трафика отключена. Включите её в настройках Dashboard.", 404)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||||
|
w.Write([]byte(trafficHTML))
|
||||||
|
})
|
||||||
|
mux.HandleFunc("/api/traffic", func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if r.Method != "GET" {
|
||||||
|
w.WriteHeader(405)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if !s.TrafficSettings().Enabled {
|
||||||
|
http.Error(w, "Карта отключена", 404)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
view, err := s.TrafficView(r.URL.Query().Get("domain"), r.URL.Query().Get("kind"), r.URL.Query().Get("external") == "1")
|
||||||
|
writeJSON(w, view, err)
|
||||||
|
})
|
||||||
|
mux.HandleFunc("/api/traffic/world", func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
w.Header().Set("Cache-Control", "public, max-age=86400")
|
||||||
|
w.Write(trafficWorld)
|
||||||
|
})
|
||||||
|
mux.HandleFunc("/api/agent/traffic", func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if r.Method != "POST" {
|
||||||
|
w.WriteHeader(405)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
id, secret, ok := agentCredentials(r)
|
||||||
|
if !ok || !s.AuthenticateAgent(id, secret) {
|
||||||
|
http.Error(w, "Unauthorized", 401)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
settings := s.TrafficSettings()
|
||||||
|
if !settings.Enabled || settings.AgentID != id {
|
||||||
|
http.Error(w, "Источник отключён", 403)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var batch TrafficBatch
|
||||||
|
if json.NewDecoder(io.LimitReader(r.Body, 1<<20)).Decode(&batch) != nil {
|
||||||
|
http.Error(w, "Invalid traffic batch", 400)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := s.SaveTraffic(id, batch); err != nil {
|
||||||
|
http.Error(w, fmt.Sprint(err), 500)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
writeJSON(w, map[string]bool{"accepted": true}, nil)
|
||||||
|
})
|
||||||
|
}
|
||||||
65
traffic_geo.go
Normal file
65
traffic_geo.go
Normal file
@@ -0,0 +1,65 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"compress/gzip"
|
||||||
|
_ "embed"
|
||||||
|
"io"
|
||||||
|
"net"
|
||||||
|
"sync"
|
||||||
|
|
||||||
|
"github.com/oschwald/maxminddb-golang"
|
||||||
|
)
|
||||||
|
|
||||||
|
//go:embed assets/world.json
|
||||||
|
var trafficWorld []byte
|
||||||
|
|
||||||
|
//go:embed assets/dbip-country-lite-2026-10.mmdb.gz
|
||||||
|
var trafficGeoCompressed []byte
|
||||||
|
|
||||||
|
const trafficGeoDate = "2026-10"
|
||||||
|
|
||||||
|
var trafficGeo struct {
|
||||||
|
sync.Once
|
||||||
|
db *maxminddb.Reader
|
||||||
|
}
|
||||||
|
|
||||||
|
func trafficGeoReady() bool {
|
||||||
|
trafficGeo.Do(func() {
|
||||||
|
r, err := gzip.NewReader(bytes.NewReader(trafficGeoCompressed))
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer r.Close()
|
||||||
|
data, err := io.ReadAll(io.LimitReader(r, 32<<20))
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
trafficGeo.db, _ = maxminddb.FromBytes(data)
|
||||||
|
})
|
||||||
|
return trafficGeo.db != nil
|
||||||
|
}
|
||||||
|
func lookupTrafficCountry(ipText string) string {
|
||||||
|
ip := net.ParseIP(ipText)
|
||||||
|
if ip == nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
if ip.IsPrivate() || ip.IsLoopback() || ip.IsLinkLocalUnicast() {
|
||||||
|
return "LAN"
|
||||||
|
}
|
||||||
|
if !ip.IsGlobalUnicast() {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
if !trafficGeoReady() {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
var record struct {
|
||||||
|
Country struct {
|
||||||
|
Code string `maxminddb:"iso_code"`
|
||||||
|
} `maxminddb:"country"`
|
||||||
|
}
|
||||||
|
if trafficGeo.db.Lookup(ip, &record) != nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return record.Country.Code
|
||||||
|
}
|
||||||
54
traffic_preview_test.go
Normal file
54
traffic_preview_test.go
Normal file
@@ -0,0 +1,54 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Opt-in local visual fixture; never included in production binaries.
|
||||||
|
func TestTrafficPreview(t *testing.T) {
|
||||||
|
if os.Getenv("DASHBOARD_TRAFFIC_PREVIEW") != "1" {
|
||||||
|
t.Skip("manual visual preview")
|
||||||
|
}
|
||||||
|
s, a := trafficFixture(t)
|
||||||
|
s.SaveTrafficSettings(TrafficSettings{Enabled: true, AgentID: a.ID, ServerCountry: "RU"})
|
||||||
|
mux := http.NewServeMux()
|
||||||
|
trafficRoutes(mux, s)
|
||||||
|
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||||
|
w.Write([]byte(dashboardHTML))
|
||||||
|
})
|
||||||
|
mux.HandleFunc("/api/agents", func(w http.ResponseWriter, r *http.Request) { items, _ := s.Agents(); writeJSON(w, items, nil) })
|
||||||
|
go func() {
|
||||||
|
i := 0
|
||||||
|
for {
|
||||||
|
i++
|
||||||
|
events := []TrafficEvent{}
|
||||||
|
for j, ip := range []string{"8.8.8.8", "1.1.1.1", "20.194.96.114", "192.168.0.10", "77.83.39.94", "51.158.145.102", "123.56.127.23"} {
|
||||||
|
kind := "request"
|
||||||
|
if j == 4 {
|
||||||
|
kind = "alert"
|
||||||
|
}
|
||||||
|
if j == 5 {
|
||||||
|
kind = "decision"
|
||||||
|
}
|
||||||
|
events = append(events, TrafficEvent{ID: fmt.Sprintf("preview-%d-%d", i, j), Time: time.Now().Unix(), Kind: kind, IP: ip, Domain: "demo.example", Method: "GET", Path: "/demo", Status: 200, Reason: "ДЕМО · crowdsecurity/http-probing"})
|
||||||
|
}
|
||||||
|
s.SaveTraffic(a.ID, TrafficBatch{Events: events, Files: 22, StartedAt: time.Now().Add(-time.Hour).Unix(), CrowdSecAt: time.Now().Unix()})
|
||||||
|
time.Sleep(3 * time.Second)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
if r.URL.Path == "/traffic" {
|
||||||
|
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||||
|
w.Write([]byte(strings.Replace(trafficHTML, "HTTP-запросы и события CrowdSec · последние 24 часа", "ДЕМОНСТРАЦИЯ · искусственные данные для проверки интерфейса", 1)))
|
||||||
|
return
|
||||||
|
}
|
||||||
|
mux.ServeHTTP(w, r)
|
||||||
|
})
|
||||||
|
t.Fatal(http.ListenAndServe("127.0.0.1:19105", handler))
|
||||||
|
}
|
||||||
70
traffic_settings.go
Normal file
70
traffic_settings.go
Normal file
@@ -0,0 +1,70 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
type TrafficSettings struct {
|
||||||
|
Enabled bool `json:"enabled"`
|
||||||
|
AgentID string `json:"agentId"`
|
||||||
|
ServerCountry string `json:"serverCountry"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Store) TrafficSettings() TrafficSettings {
|
||||||
|
var c TrafficSettings
|
||||||
|
s.loadJSONSetting("traffic", &c)
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
func (s *Store) SaveTrafficSettings(c TrafficSettings) (TrafficSettings, error) {
|
||||||
|
c.AgentID = strings.TrimSpace(c.AgentID)
|
||||||
|
c.ServerCountry = strings.ToUpper(strings.TrimSpace(c.ServerCountry))
|
||||||
|
if c.Enabled {
|
||||||
|
var count int
|
||||||
|
if s.db.QueryRow(`SELECT COUNT(*) FROM agents WHERE id=?`, c.AgentID).Scan(&count) != nil || count != 1 {
|
||||||
|
return c, errors.New("Выберите зарегистрированный агент")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if c.ServerCountry != "" {
|
||||||
|
var countries []struct {
|
||||||
|
Code string `json:"code"`
|
||||||
|
}
|
||||||
|
json.Unmarshal(trafficWorld, &countries)
|
||||||
|
found := false
|
||||||
|
for _, country := range countries {
|
||||||
|
if country.Code == c.ServerCountry {
|
||||||
|
found = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
return c, errors.New("Неизвестная страна сервера")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return c, s.saveJSONSetting("traffic", c)
|
||||||
|
}
|
||||||
|
func trafficSettingsRoute(mux *http.ServeMux, s *Store) {
|
||||||
|
mux.HandleFunc("/api/settings/traffic", func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
switch r.Method {
|
||||||
|
case "GET":
|
||||||
|
writeJSON(w, s.TrafficSettings(), nil)
|
||||||
|
case "PUT":
|
||||||
|
var c TrafficSettings
|
||||||
|
if json.NewDecoder(io.LimitReader(r.Body, 4096)).Decode(&c) != nil {
|
||||||
|
http.Error(w, "Некорректные настройки", 400)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
value, err := s.SaveTrafficSettings(c)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, err.Error(), 400)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
writeJSON(w, value, nil)
|
||||||
|
default:
|
||||||
|
w.WriteHeader(405)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
217
traffic_test.go
Normal file
217
traffic_test.go
Normal file
@@ -0,0 +1,217 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func trafficFixture(t *testing.T) (*Store, agentConfig) {
|
||||||
|
t.Helper()
|
||||||
|
s, err := openStore(":memory:")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() { s.Close() })
|
||||||
|
token, err := s.CreateAgentEnrollmentToken()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
a, err := s.EnrollAgent(agentEnrollmentRequest{Token: token, Name: "NPM test", VMID: 102})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return s, a
|
||||||
|
}
|
||||||
|
func npmTestLine(ip, path string) string {
|
||||||
|
return fmt.Sprintf("[%s] - - 301 - GET https example.test \"%s\" [Client %s] [Length 166] [Gzip -] [Sent-to 192.168.0.9] \"-\" \"-\"\n", time.Now().UTC().Format("02/Jan/2006:15:04:05 -0700"), path, ip)
|
||||||
|
}
|
||||||
|
func TestNPMTrafficFormats(t *testing.T) {
|
||||||
|
for _, status := range []string{"- 200 200", "- - 301", "- 502 502", "- - 401"} {
|
||||||
|
line := strings.Replace(npmTestLine("2001:4860:4860::8888", "/login?token=secret#fragment"), "- - 301", status, 1)
|
||||||
|
e, ok := parseNPMTraffic(line)
|
||||||
|
if !ok || e.Kind != "request" || e.Path != "/login" || e.Domain != "example.test" || e.IP != "2001:4860:4860::8888" {
|
||||||
|
t.Fatalf("bad parse: %+v %v", e, ok)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if _, ok := parseNPMTraffic("broken"); ok {
|
||||||
|
t.Fatal("accepted invalid log")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
func TestCrowdTrafficDoesNotInventBlockedRequests(t *testing.T) {
|
||||||
|
raw := `[{"id":12,"uuid":"alert-uuid","created_at":"2026-10-09T16:16:43Z","source":{"ip":"20.194.96.114"},"scenario":"crowdsecurity/http-admin-interface-probing","events":[{"meta":[{"key":"target_fqdn","value":"example.test"}]}],"decisions":[{"id":42,"value":"20.194.96.114","type":"ban","simulated":false},{"id":43,"value":"1.1.1.1","type":"ban","simulated":true}]}]`
|
||||||
|
events, err := parseCrowdTraffic([]byte(raw))
|
||||||
|
if err != nil || len(events) != 2 {
|
||||||
|
t.Fatalf("%+v %v", events, err)
|
||||||
|
}
|
||||||
|
if events[0].Kind != "alert" || events[1].Kind != "decision" || events[1].Domain != "example.test" {
|
||||||
|
t.Fatalf("bad categories %+v", events)
|
||||||
|
}
|
||||||
|
simulated := strings.Replace(raw, `"id":12`, `"simulated":true,"id":12`, 1)
|
||||||
|
events, err = parseCrowdTraffic([]byte(simulated))
|
||||||
|
if err != nil || len(events) != 0 {
|
||||||
|
t.Fatal("simulation counted as live")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
func TestTrafficLogCheckpointRotationAndPartialLines(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
path := filepath.Join(dir, "proxy-host-1_access.log")
|
||||||
|
os.WriteFile(path, []byte(npmTestLine("1.1.1.1", "/old")), 0600)
|
||||||
|
state := trafficAgentState{Cursors: map[string]trafficCursor{}}
|
||||||
|
if _, err := state.readLogs(dir); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if len(state.Pending) != 0 {
|
||||||
|
t.Fatal("imported old logs on first activation")
|
||||||
|
}
|
||||||
|
appendFile := func(path, s string) {
|
||||||
|
f, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, 0600)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
f.WriteString(s)
|
||||||
|
f.Close()
|
||||||
|
}
|
||||||
|
line := npmTestLine("8.8.8.8", "/new")
|
||||||
|
appendFile(path, line[:len(line)/2])
|
||||||
|
state.readLogs(dir)
|
||||||
|
if len(state.Pending) != 0 {
|
||||||
|
t.Fatal("partial line emitted")
|
||||||
|
}
|
||||||
|
appendFile(path, line[len(line)/2:])
|
||||||
|
state.readLogs(dir)
|
||||||
|
if len(state.Pending) != 1 {
|
||||||
|
t.Fatal("complete line missing")
|
||||||
|
}
|
||||||
|
id := state.Pending[0].ID
|
||||||
|
state.Pending = nil
|
||||||
|
checkpoint := filepath.Join(dir, "state.json")
|
||||||
|
if err := saveTrafficAgentState(checkpoint, state); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
raw, _ := os.ReadFile(checkpoint)
|
||||||
|
var restored trafficAgentState
|
||||||
|
json.Unmarshal(raw, &restored)
|
||||||
|
os.Rename(path, path+".1")
|
||||||
|
appendFile(path+".1", npmTestLine("9.9.9.9", "/rotated"))
|
||||||
|
os.WriteFile(path, []byte(npmTestLine("1.0.0.1", "/fresh")), 0600)
|
||||||
|
restored.readLogs(dir)
|
||||||
|
if len(restored.Pending) != 2 {
|
||||||
|
t.Fatalf("rotation lost events %+v", restored)
|
||||||
|
}
|
||||||
|
for _, e := range restored.Pending {
|
||||||
|
if e.ID == id {
|
||||||
|
t.Fatal("duplicate id")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
restored.Pending = nil
|
||||||
|
restored.readLogs(dir)
|
||||||
|
if len(restored.Pending) != 0 {
|
||||||
|
t.Fatal("re-read same lines")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
func TestTrafficStoreDedupFiltersAndSanitization(t *testing.T) {
|
||||||
|
s, a := trafficFixture(t)
|
||||||
|
if s.TrafficSettings().Enabled {
|
||||||
|
t.Fatal("enabled by default")
|
||||||
|
}
|
||||||
|
if _, err := s.SaveTrafficSettings(TrafficSettings{Enabled: true, AgentID: a.ID}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
now := time.Now().Unix()
|
||||||
|
batch := TrafficBatch{Files: 22, Events: []TrafficEvent{{ID: "1", Time: now, Kind: "request", IP: "1.1.1.1", Domain: "example.test", Path: "/login?secret=x", Status: 502}, {ID: "2", Time: now, Kind: "request", IP: "192.168.0.10", Domain: "example.test", Status: 200}, {ID: "3", Time: now, Kind: "alert", IP: "1.1.1.1", Reason: "probe"}, {ID: "4", Time: now, Kind: "decision", IP: "1.1.1.1", Reason: "ban"}}}
|
||||||
|
for i := 0; i < 2; i++ {
|
||||||
|
if err := s.SaveTraffic(a.ID, batch); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
v, err := s.TrafficView("", "", false)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if v.Requests != 2 || v.UniqueIPs != 2 || v.Alerts != 1 || v.Decisions != 1 || len(v.Events) != 4 {
|
||||||
|
t.Fatalf("wrong counts: %+v", v)
|
||||||
|
}
|
||||||
|
for _, e := range v.Events {
|
||||||
|
if strings.Contains(e.Path, "secret") {
|
||||||
|
t.Fatal("secret saved")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
v, err = s.TrafficView("example.test", "request", true)
|
||||||
|
if err != nil || v.Requests != 1 {
|
||||||
|
t.Fatalf("bad filter %+v %v", v, err)
|
||||||
|
}
|
||||||
|
batch.Events = []TrafficEvent{{ID: "invalid", Kind: "blocked", IP: "1.1.1.1", Time: now}}
|
||||||
|
if s.SaveTraffic(a.ID, batch) == nil {
|
||||||
|
t.Fatal("accepted invented block event")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
func TestTrafficAPIOptInAndSourceAuthorization(t *testing.T) {
|
||||||
|
s, a := trafficFixture(t)
|
||||||
|
mux := http.NewServeMux()
|
||||||
|
trafficRoutes(mux, s)
|
||||||
|
get := func(path string) int {
|
||||||
|
r := httptest.NewRecorder()
|
||||||
|
mux.ServeHTTP(r, httptest.NewRequest("GET", path, nil))
|
||||||
|
return r.Code
|
||||||
|
}
|
||||||
|
if get("/traffic") != 404 || get("/api/traffic") != 404 {
|
||||||
|
t.Fatal("disabled page exposed")
|
||||||
|
}
|
||||||
|
send := func(secret string) int {
|
||||||
|
r := httptest.NewRecorder()
|
||||||
|
req := httptest.NewRequest("POST", "/api/agent/traffic", strings.NewReader(`{"events":[]}`))
|
||||||
|
req.Header.Set("Authorization", "Bearer "+a.ID+"."+secret)
|
||||||
|
mux.ServeHTTP(r, req)
|
||||||
|
return r.Code
|
||||||
|
}
|
||||||
|
if send("wrong") != 401 || send(a.Secret) != 403 {
|
||||||
|
t.Fatal("disabled/unauthorized ingestion accepted")
|
||||||
|
}
|
||||||
|
if _, err := s.SaveTrafficSettings(TrafficSettings{Enabled: true, AgentID: "missing"}); err == nil {
|
||||||
|
t.Fatal("unknown agent accepted")
|
||||||
|
}
|
||||||
|
s.SaveTrafficSettings(TrafficSettings{Enabled: true, AgentID: a.ID, ServerCountry: "RU"})
|
||||||
|
if send(a.Secret) != 200 || get("/traffic") != 200 {
|
||||||
|
t.Fatal("selected agent not working")
|
||||||
|
}
|
||||||
|
token, _ := s.CreateAgentEnrollmentToken()
|
||||||
|
other, _ := s.EnrollAgent(agentEnrollmentRequest{Token: token, Name: "Other"})
|
||||||
|
s.SaveTrafficSettings(TrafficSettings{Enabled: true, AgentID: other.ID})
|
||||||
|
if send(a.Secret) != 403 {
|
||||||
|
t.Fatal("old source still accepted")
|
||||||
|
}
|
||||||
|
s.SaveTrafficSettings(TrafficSettings{Enabled: false, AgentID: other.ID})
|
||||||
|
if get("/api/traffic") != 404 {
|
||||||
|
t.Fatal("disable failed")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
func TestTrafficGeoDatabase(t *testing.T) {
|
||||||
|
if !trafficGeoReady() {
|
||||||
|
t.Fatal("embedded database invalid")
|
||||||
|
}
|
||||||
|
if lookupTrafficCountry("192.168.0.10") != "LAN" {
|
||||||
|
t.Fatal("LAN mapped to country")
|
||||||
|
}
|
||||||
|
if c := lookupTrafficCountry("8.8.8.8"); len(c) != 2 {
|
||||||
|
t.Fatalf("bad public IP geo: %q", c)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
func TestTrafficCheckpointKeepsPendingUntilAcknowledged(t *testing.T) {
|
||||||
|
path := filepath.Join(t.TempDir(), "state.json")
|
||||||
|
s := trafficAgentState{Pending: []TrafficEvent{{ID: "retry-me", IP: "1.1.1.1"}}, Cursors: map[string]trafficCursor{"file": {Offset: 100}}}
|
||||||
|
if err := saveTrafficAgentState(path, s); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
raw, _ := os.ReadFile(path)
|
||||||
|
if !bytes.Contains(raw, []byte("retry-me")) {
|
||||||
|
t.Fatal("pending batch lost")
|
||||||
|
}
|
||||||
|
}
|
||||||
6
traffic_web.go
Normal file
6
traffic_web.go
Normal file
@@ -0,0 +1,6 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import _ "embed"
|
||||||
|
|
||||||
|
//go:embed assets/traffic.html
|
||||||
|
var trafficHTML string
|
||||||
Reference in New Issue
Block a user