diff --git a/INSTALL.md b/INSTALL.md index 6e1e4c1..cc0f109 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -230,32 +230,80 @@ Answer the prompts (callsign, e-mail, password, admin level). Levels: Log in on the web page with that callsign. The **Admin** menu appears. -## 9. Live data API — what this repository does NOT contain +## 9. Live data backend: svxdr The dashboard (node list, map, monitor, last-heard, trunks) fetches JSON and Server-Sent Events from `/svxportal/api/...`, which nginx proxies to -`127.0.0.1:8098`. That backend service is **not included** here. Without it -the account/admin/talkgroup pages work, but the front page's node list, map, -monitor, last-heard and trunks views simply stay **empty**. There is no error -banner: the API helper returns null on a failed fetch and the page keeps its -empty tables. Check the browser console (404 on `/svxportal/api/status`) if -you are unsure. +`127.0.0.1:8098`. That backend is **svxdr**, included in `svxdr/`: one Go +binary with four modules switched on or off per box in `/etc/svxlink/svxdr.json`. -Endpoints the UI expects: +| Module | What it does | A plain reflector needs it? | +|--------|--------------|-----------------------------| +| `proxy` | reads the reflector's own `/status`, caches it, serves it enriched on 8090 | yes | +| `nodelist` | builds the node/lastheard/monitor/trunk data from the proxy + Redis, serves `/api/*` on 8098 | yes | +| `astmeta` | AllStar/Asterisk AMI talker metadata for USRP bridges | no, leave `false` | +| `xlink` | relays talker events to an MQTT broker | no, leave `false` | + +Without svxdr the account/admin/talkgroup pages work, but the front page's +live views stay **empty** with no error banner (404 on `/svxportal/api/status` +in the browser console). + +### Build + +```bash +sudo apt install golang-go # Go 1.24 or newer +cd svxdr && CGO_ENABLED=0 go build -trimpath -o svxdr . && cd .. +sudo install -m0755 svxdr/svxdr /usr/local/bin/svxdr +``` + +### Configure + +```bash +sudo cp install/svxdr.json.example /etc/svxlink/svxdr.json +sudo chmod 640 /etc/svxlink/svxdr.json +sudoedit /etc/svxlink/svxdr.json +``` + +Set at least: + +- `redis.socket` and `redis.password` -> the same Redis as `auth/config.php` (step 3). +- `proxy.reflector_status_url` -> your reflector's `HTTP_SRV_PORT` status URL. + With `HTTP_SRV_PORT=8099` in `svxreflector.conf` (as in + `install/svxreflector.conf.example`) the default `http://127.0.0.1:8099/status` is right. +- `nodelist.reflector_status_url` stays on the proxy (`http://127.0.0.1:8090/status`). +- All `CHANGEME` tokens: any random string; the ingest tokens only matter if a + remote box posts talker metadata to you. +- `nodelist.federation.seeds`: `/status` URLs of other reflectors you want in the + map that your own trunks do not reach. Empty = only what the crawl finds. + `aggregate_url` empty disables the gap detector. + +The port layout must match the nginx config: reflector 8099 -> proxy 8090 -> +nodelist 8098 -> nginx `/svxportal/api/`. + +### Run + +```bash +sudo cp install/systemd/svxdr.service /etc/systemd/system/svxdr.service +sudo systemctl daemon-reload +sudo systemctl enable --now svxdr +systemctl status svxdr --no-pager +curl -s http://127.0.0.1:8098/healthz # -> ok +curl -s http://127.0.0.1:8098/api/status | head -c 300 +``` + +Config changes are picked up with `sudo systemctl reload svxdr` (SIGHUP), no +restart needed. `journalctl -u svxdr -f` shows what it polls and serves. + +Endpoints the UI uses: | Path | Used by | |------|---------| -| `/svxportal/api/status` | node table, topology | +| `/svxportal/api/status` | node table, topology, CTCSS | | `/svxportal/api/stream` | live updates (SSE) | | `/svxportal/api/lastheard` | last-heard list | | `/svxportal/api/monitor-nodes` | monitor view | | `/svxportal/api/local-trunks` | trunks view | -You need to provide something that serves these, for example a small service -that reads your reflector's `HTTP_SRV_PORT` status endpoint -([GeuReflector](https://github.com/audric/GeuReflector) serves `/status` there) -and republishes it. The static reflector status page in `help.html` describes the fields. - ## 10. Reflector admin features (optional, need sudo helpers) The pages **Admin → Reflector**, **Certificates**, **Node control** and **MQTT** diff --git a/README.md b/README.md index 2a01a2a..7610d05 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,9 @@ verification, and admin pages for accounts, node certificates, node control and talkgroup requests. Static HTML/JS (Bootstrap from jsdelivr) plus a small PHP backend that keeps -its data in Redis. No build step. +its data in Redis, and **svxdr**, a Go daemon that turns the reflector's +`/status` into the live node, lastheard, monitor and trunk data the dashboard +shows (`svxdr/`, one `go build`). - **Reflector:** this portal is used with [GeuReflector](https://github.com/audric/GeuReflector) (SvxReflector with reflector-to-reflector trunking). Its status endpoint @@ -15,10 +17,10 @@ its data in Redis. No build step. - **Config templates:** [`install/`](install/) - **User help:** `help.html` (served at `/help.html`) -This repository contains the web application only. The live-data API it -consumes (`/svxportal/api/...`) and the sudo helper scripts used by the -reflector admin pages are site-specific and not included; INSTALL.md -sections 9 and 10 describe what they must provide. +Not included: the sudo helper scripts the reflector admin pages call +(`svxreflector-pty`, `svxreflector-users`, `svxreflector-systemctl`, +`svx-mqtt-provision`). They are specific to how each reflector box is set up; +INSTALL.md section 10 describes what they must do. ## License diff --git a/install/nginx-svxportal.conf b/install/nginx-svxportal.conf index 796a70f..d37daec 100644 --- a/install/nginx-svxportal.conf +++ b/install/nginx-svxportal.conf @@ -85,8 +85,8 @@ server { location / { try_files $uri $uri/ =404; } # Live data API (/svxportal/api/status, /stream, /lastheard, /monitor-nodes, - # /local-trunks). The web UI expects a JSON/SSE backend here on 127.0.0.1:8098. - # That backend is NOT part of this repository — see INSTALL.md, section 8. + # /local-trunks), served by svxdr's nodelist module on 127.0.0.1:8098. + # See INSTALL.md, section 9. location ^~ /svxportal/api/ { proxy_pass http://127.0.0.1:8098/api/; proxy_http_version 1.1; diff --git a/install/svxdr.json.example b/install/svxdr.json.example new file mode 100644 index 0000000..12a370e --- /dev/null +++ b/install/svxdr.json.example @@ -0,0 +1,101 @@ +{ + "modules": { + "astmeta": false, + "nodelist": true, + "proxy": true, + "xlink": false + }, + "redis": { + "socket": "/run/redis/redis.sock", + "password": "CHANGEME", + "db": 0, + "status_key": "/registry/svxlink/status/json", + "status_hash_key": "/registry/svxlink/status/hash", + "pubsub_chan": "/registry/svxlink/status/update", + "nodes_online_key": "/registry/svxlink/nodes/online", + "node_key_prefix": "/registry/svxlink/node", + "node_connect_chan": "/registry/svxlink/node/connect", + "node_disconnect_chan": "/registry/svxlink/node/disconnect", + "node_update_chan": "/registry/svxlink/node/update", + "lastheard_key": "/registry/svxlink/lastheard" + }, + "trunk": { + "satellite_id": "", + "host": "127.0.0.1", + "port": 5403 + }, + "lastheard": { + "bridge_location_format": "{system} via {node} / {talker}", + "bridge_location_format_no_system": "via {node} / {talker}" + }, + "nodes": [], + "astmeta": { + "ami_host": "127.0.0.1", + "ami_port": "5038", + "ami_user": "astmeta", + "ami_secret": "CHANGEME", + "ignore_nodes": [ + "1999", + "1998" + ], + "poll_interval": "1s", + "hang_time": "2s", + "network": "AllStar", + "usrp_target": "127.0.0.1:4001", + "format": "info", + "refresh_interval": "10s", + "lookup_enabled": true, + "lookup_url": "http://allmondb.allstarlink.org/", + "cache_ttl": "24h", + "mqtt_enabled": false, + "nodelist_url": "", + "nodelist_token": "CHANGEME", + "nodelist_peers": [] + }, + "nodelist": { + "listen_addr": "127.0.0.1:8098", + "reflector_status_url": "http://127.0.0.1:8090/status", + "poll_interval": "1s", + "reflector_log_path": "/var/log/svxreflector", + "network_json_path": "/var/www/html/svxportal/network.json", + "astdb_url": "http://allmondb.allstarlink.org/", + "talker_meta_ttl": "30s", + "talker_ingest_token": "CHANGEME", + "allstar_nodes_ttl": "30m", + "external": { + "pull_scheme": "https", + "pull_path": "/svx_externalpull.php", + "pull_interval": "30s", + "status_ports": [], + "status_path": "/status", + "proxy_path": "/proxy_status.php", + "detail_path": "/geu/status/", + "max_refs": 100, + "stale_ttl": "5m" + }, + "federation": { + "seeds": [], + "aggregate_url": "", + "aggregate_interval": "5m" + } + }, + "proxy": { + "listen": "127.0.0.1:8090", + "reflector_status_url": "http://127.0.0.1:8099/status", + "cache_ttl": "200ms", + "fetch_timeout": "5s", + "meta_ttl": "30s", + "ingest_token": "CHANGEME", + "upstream_status_url": "", + "forward_talker_url": "", + "sync_interval": "2s", + "remote_ttl": "60s" + }, + "xlink": { + "broker": "", + "user": "", + "pass": "CHANGEME", + "topic": "callmgr/xlink/talker", + "origin": "" + } +} diff --git a/install/systemd/svxdr.service b/install/systemd/svxdr.service new file mode 100644 index 0000000..c875c0c --- /dev/null +++ b/install/systemd/svxdr.service @@ -0,0 +1,24 @@ +# svxdr — live-data daemon for SvxPortal. Install: +# sudo cp install/systemd/svxdr.service /etc/systemd/system/ && sudo systemctl daemon-reload && sudo systemctl enable --now svxdr +# Build and configure per INSTALL.md section 9. +[Unit] +Description=svxdr — merged svx daemon (astmeta + nodelist + proxy + xlink) +After=network-online.target redis-server.service +Wants=network-online.target + +[Service] +Type=simple +# One binary, one config; which subsystems run is decided by "modules" in the +# JSON (or override with flags, e.g. ExecStart=... --nodelist --proxy). +ExecStart=/usr/local/bin/svxdr --config /etc/svxlink/svxdr.json +Restart=on-failure +RestartSec=3 +# Reload config in place (SIGHUP) without dropping the process. +ExecReload=/bin/kill -HUP $MAINPID +# Match the file/socket permissions the old services used (redis socket, logs, +# /var/www network.json). Adjust User/Group to whatever nodelist ran as. +#User=root +LimitNOFILE=65536 + +[Install] +WantedBy=multi-user.target diff --git a/svxdr/astmeta/astmeta.go b/svxdr/astmeta/astmeta.go new file mode 100644 index 0000000..f7b2db1 --- /dev/null +++ b/svxdr/astmeta/astmeta.go @@ -0,0 +1,634 @@ +// svxdr — live-data daemon for SvxPortal (nodelist + proxy + astmeta + xlink) +// Copyright (C) 2026 SA7AUX (Björn) +// SPDX-License-Identifier: AGPL-3.0-or-later +// +// This program is free software: you can redistribute it and/or modify it under +// the terms of the GNU Affero General Public License as published by the Free +// Software Foundation, either version 3 of the License, or (at your option) any +// later version. See the LICENSE file for details. + +// Package astmeta is the svxdr astmeta subsystem — AllStar talker metadata, +// folded in from the standalone svx-web/astmeta. It polls AMI (rpt xnode) for +// keyed remote nodes, resolves callsign/system, feeds svxlink UsrpLogic (USRP), +// optionally MQTT, and POSTs talker-meta to the nodelist/proxy (kept for the +// REMOTE case, e.g. astmeta on the allstar box; when co-located a later phase +// shares the store directly). Logic preserved exactly, including the fan-out +// allstar remap. Config now comes from svxdr.json (astmeta + nodes + redis). +package astmeta + +import ( + "bufio" + "bytes" + "context" + "encoding/binary" + "encoding/json" + "fmt" + "io" + "log" + "net" + "net/http" + "os" + "regexp" + "strconv" + "strings" + "sync" + "time" + + mqtt "github.com/eclipse/paho.mqtt.golang" + "github.com/redis/go-redis/v9" + + "svxportal/svxdr/config" +) + +// settings holds the astmeta-relevant config, derived from svxdr.json. +type settings struct { + AmiHost, AmiPort, AmiUser, AmiSecret string + IgnoreNodes map[string]bool + PollInterval, HangTime time.Duration + Network string + + Target, Format, SpaceReplace string + RefreshInterval time.Duration + + LookupEnabled bool + LookupURL, LookupFile string + CacheTTL time.Duration + RedisCache bool + RedisPrefix string + Redis config.Redis + + MqttEnabled bool + MqttBroker, MqttUser, MqttPass, MqttPrefix string + + NodelistURL, NodelistToken string + NodelistPeers []metaTarget + Via string + + // per-node, from cfg.Nodes + WatchNodes []string + NodeTg map[string]string + NodeVia map[string]string + NodeAllstar map[string]string + Names map[string]string + ownNodes map[string]bool +} + +func (s *settings) tgFor(wn string) string { return s.NodeTg[wn] } +func (s *settings) viaFor(wn string) string { + if v := s.NodeVia[wn]; v != "" { + return v + } + return s.Via +} + +func newSettings(c *config.Config) *settings { + a := c.Astmeta + s := &settings{ + AmiHost: orDef(a.AmiHost, "127.0.0.1"), AmiPort: orDef(a.AmiPort, "5038"), + AmiUser: a.AmiUser, AmiSecret: a.AmiSecret, + IgnoreNodes: map[string]bool{}, + PollInterval: orDur(a.PollInterval.D(), time.Second), + HangTime: a.HangTime.D(), + Network: orDef(a.Network, "AllStar"), + Target: a.UsrpTarget, // empty = USRP sink disabled (newSender returns nil) + Format: orDef(a.Format, "info"), + SpaceReplace: orDef(a.SpaceReplace, "-"), + RefreshInterval: a.RefreshInterval.D(), + LookupEnabled: a.LookupEnabled, + LookupURL: a.LookupURL, LookupFile: a.LookupFile, + CacheTTL: orDur(a.CacheTTL.D(), 24*time.Hour), + RedisCache: a.RedisCache, + RedisPrefix: orDef(a.RedisCachePrefix, "astmeta:node:"), + Redis: c.Redis, + MqttEnabled: a.MqttEnabled, MqttBroker: a.MqttBroker, MqttUser: a.MqttUser, + MqttPass: a.MqttPass, MqttPrefix: a.MqttPrefix, + NodelistURL: a.NodelistURL, NodelistToken: a.NodelistToken, + NodeTg: map[string]string{}, + NodeVia: map[string]string{}, + NodeAllstar: map[string]string{}, + Names: map[string]string{}, + ownNodes: map[string]bool{}, + } + s.Via = "" // global via fallback is unused now that nodes carry their own via + for _, n := range a.IgnoreNodes { + if n = strings.TrimSpace(n); n != "" { + s.IgnoreNodes[n] = true + } + } + for _, p := range a.NodelistPeers { + if p.URL != "" { + s.NodelistPeers = append(s.NodelistPeers, metaTarget{URL: p.URL, Token: p.Token}) + } + } + // per-node bindings from nodes[] (the source of truth) + for _, e := range c.Nodes { + if !e.IsEnabled() || e.SvxNode == 0 { + continue + } + wn := strconv.Itoa(e.SvxNode) + s.WatchNodes = append(s.WatchNodes, wn) + s.ownNodes[wn] = true + if e.TG != 0 { + s.NodeTg[wn] = strconv.Itoa(e.TG) + } + via := e.Via + if via == "" { + via = e.Callsign + } + if via != "" { + s.NodeVia[wn] = via + } + if e.Allstar != 0 { + s.NodeAllstar[wn] = strconv.Itoa(e.Allstar) + if e.System != "" { + s.Names[strconv.Itoa(e.Allstar)] = e.System + } + } + } + return s +} + +func orDef(v, d string) string { + if v == "" { + return d + } + return v +} +func orDur(v, d time.Duration) time.Duration { + if v <= 0 { + return d + } + return v +} +func sleepCtx(ctx context.Context, d time.Duration) bool { + t := time.NewTimer(d) + defer t.Stop() + select { + case <-ctx.Done(): + return false + case <-t.C: + return true + } +} + +// ---------------------------------------------------------------- AMI -------- + +type ami struct { + s *settings + conn net.Conn + rd *bufio.Reader +} + +func (a *ami) connect() error { + conn, err := net.DialTimeout("tcp", net.JoinHostPort(a.s.AmiHost, a.s.AmiPort), 5*time.Second) + if err != nil { + return err + } + a.conn = conn + a.rd = bufio.NewReader(conn) + a.conn.SetReadDeadline(time.Now().Add(5 * time.Second)) + if _, err := a.rd.ReadString('\n'); err != nil { + conn.Close() + return fmt.Errorf("ami banner: %w", err) + } + resp, err := a.request(fmt.Sprintf( + "Action: Login\r\nUsername: %s\r\nSecret: %s\r\nEvents: off\r\n\r\n", + a.s.AmiUser, a.s.AmiSecret)) + if err != nil { + conn.Close() + return err + } + if !strings.Contains(resp, "Success") { + conn.Close() + return fmt.Errorf("ami login failed: %s", strings.TrimSpace(resp)) + } + return nil +} + +func (a *ami) request(action string) (string, error) { + a.conn.SetWriteDeadline(time.Now().Add(5 * time.Second)) + if _, err := io.WriteString(a.conn, action); err != nil { + return "", err + } + var b strings.Builder + follows := false + for { + a.conn.SetReadDeadline(time.Now().Add(5 * time.Second)) + line, err := a.rd.ReadString('\n') + if err != nil { + return "", err + } + b.WriteString(line) + t := strings.TrimRight(line, "\r\n") + if strings.HasPrefix(t, "Response: Follows") { + follows = true + } + if follows { + if strings.Contains(t, "--END COMMAND--") { + follows = false + } + continue + } + if t == "" { + return b.String(), nil + } + } +} + +func (a *ami) alinks(node string) ([]string, error) { + resp, err := a.request(fmt.Sprintf("Action: Command\r\nCommand: rpt xnode %s\r\n\r\n", node)) + if err != nil { + return nil, err + } + for _, line := range strings.Split(resp, "\n") { + line = strings.TrimPrefix(strings.TrimSpace(line), "Output: ") + if v, ok := strings.CutPrefix(line, "RPT_ALINKS="); ok { + parts := strings.Split(strings.TrimSpace(v), ",") + if len(parts) > 1 { + return parts[1:], nil + } + return nil, nil + } + } + return nil, fmt.Errorf("no RPT_ALINKS in xnode output for %s", node) +} + +// ------------------------------------------------------------- resolver ------ + +var astdbLine = regexp.MustCompile(`^([0-9]+)\|([^|]*)\|([^|]*)\|(.*)$`) + +type resolver struct { + s *settings + ctx context.Context + rdb *redis.Client + mu sync.RWMutex + astdb map[string][2]string + last time.Time +} + +func newResolver(ctx context.Context, s *settings) *resolver { + r := &resolver{s: s, ctx: ctx, astdb: map[string][2]string{}} + if s.RedisCache { + opt := &redis.Options{Password: s.Redis.Password, DB: s.Redis.DB} + if s.Redis.Socket != "" { + opt.Network, opt.Addr = "unix", s.Redis.Socket + } else { + opt.Addr = orDef(s.Redis.Addr, "127.0.0.1:6379") + } + r.rdb = redis.NewClient(opt) + } + return r +} + +func (r *resolver) refresh() { + if !r.s.LookupEnabled || time.Since(r.last) < r.s.CacheTTL { + return + } + var rd io.ReadCloser + var err error + switch { + case r.s.LookupFile != "": + rd, err = os.Open(r.s.LookupFile) + case r.s.LookupURL != "": + var resp *http.Response + cl := &http.Client{Timeout: 60 * time.Second} + resp, err = cl.Get(r.s.LookupURL) + if err == nil { + rd = resp.Body + } + default: + r.last = time.Now() + return + } + if err != nil { + log.Printf("astmeta lookup: refresh failed: %v", err) + r.last = time.Now().Add(-r.s.CacheTTL + 5*time.Minute) + return + } + defer rd.Close() + m := map[string][2]string{} + sc := bufio.NewScanner(rd) + sc.Buffer(make([]byte, 1024*1024), 1024*1024) + for sc.Scan() { + if f := astdbLine.FindStringSubmatch(sc.Text()); f != nil { + m[f[1]] = [2]string{f[2], f[3]} + } + } + if len(m) > 0 { + r.mu.Lock() + r.astdb = m + r.mu.Unlock() + log.Printf("astmeta lookup: astdb loaded, %d nodes", len(m)) + } + r.last = time.Now() +} + +func (r *resolver) Resolve(node string) (call, system string) { + system = r.s.Names[node] + call = node + if r.rdb != nil { + if v, err := r.rdb.Get(r.ctx, r.s.RedisPrefix+node).Result(); err == nil && v != "" { + return v, system + } + } + r.refresh() + r.mu.RLock() + e, ok := r.astdb[node] + r.mu.RUnlock() + if ok && e[0] != "" { + if r.rdb != nil { + r.rdb.Set(r.ctx, r.s.RedisPrefix+node, e[0], r.s.CacheTTL) + } + call = e[0] + } + return call, system +} + +// ---------------------------------------------------------------- sender ----- + +type sender struct { + s *settings + conn net.Conn + seq uint32 +} + +func newSender(s *settings) (*sender, error) { + if s.Target == "" { + return nil, nil // USRP sink optional + } + conn, err := net.Dial("udp", s.Target) + if err != nil { + return nil, err + } + return &sender{s: s, conn: conn}, nil +} + +func (s *sender) header(utype uint32) []byte { + h := make([]byte, 32) + copy(h[0:4], "USRP") + binary.LittleEndian.PutUint32(h[4:8], s.seq) + binary.LittleEndian.PutUint32(h[20:24], utype) + s.seq++ + return h +} + +func (s *sender) SendInfo(call, node, tg string) error { + payload := map[string]any{ + "ab": map[string]string{"version": "astmeta"}, + "digital": map[string]string{ + "gw": node, "rpt": node, "tg": tg, "ts": "1", "cc": "1", + "call": call, "ambe_mode": "USRP", + }, + "last_tune": tg, + } + j, err := json.Marshal(payload) + if err != nil { + return err + } + frame := append(s.header(2), []byte("INFO:")...) + frame = append(frame, j...) + _, err = s.conn.Write(frame) + return err +} + +func (s *sender) SendTlv(call, node, tg string) error { + if len(call) > 8 { + call = call[:8] + } + tgn, _ := strconv.Atoi(tg) + nodeNum, _ := strconv.Atoi(node) + body := make([]byte, 2+3+4+3+1+1+306) + body[0] = 0x08 + body[1] = byte(len(call) + 13) + binary.LittleEndian.PutUint32(body[5:9], uint32(nodeNum)) + body[9] = byte(tgn >> 16) + body[10] = byte(tgn >> 8) + body[11] = byte(tgn) + body[12] = 1 + body[13] = 1 + copy(body[14:], call) + frame := append(s.header(2), body...) + _, err := s.conn.Write(frame) + return err +} + +func (s *sender) Send(call, node, tg string) error { + call = strings.Join(strings.Fields(call), s.s.SpaceReplace) + if s.s.Format == "tlv" { + return s.SendTlv(call, node, tg) + } + return s.SendInfo(call, node, tg) +} + +// ------------------------------------------------------------- publisher ----- + +type publisher struct { + s *settings + cl mqtt.Client +} + +func newPublisher(s *settings) *publisher { + if !s.MqttEnabled { + return nil + } + opts := mqtt.NewClientOptions(). + AddBroker(s.MqttBroker).SetClientID("svxdr-astmeta"). + SetUsername(s.MqttUser).SetPassword(s.MqttPass). + SetAutoReconnect(true).SetConnectRetry(true). + SetConnectRetryInterval(5 * time.Second) + cl := mqtt.NewClient(opts) + if t := cl.Connect(); t.Wait() && t.Error() != nil { + log.Printf("astmeta mqtt: connect failed: %v (will keep retrying)", t.Error()) + } + return &publisher{s: s, cl: cl} +} + +func (p *publisher) PublishMeta(call, node, system, tg, via, network string) { + topic := strings.TrimRight(p.s.MqttPrefix, "/") + "/talker/" + tg + "/meta" + j, _ := json.Marshal(map[string]any{ + "callsign": call, "node": node, "system": system, "via": via, + "network": network, "ts": time.Now().UnixMilli(), + }) + p.cl.Publish(topic, 0, false, j) +} + +// --------------------------------------------------------------- poster ------ + +type metaTarget struct{ URL, Token string } + +type poster struct { + targets []metaTarget + cl *http.Client +} + +func newPoster(s *settings) *poster { + var targets []metaTarget + if s.NodelistURL != "" && s.NodelistToken != "" { + targets = append(targets, metaTarget{URL: s.NodelistURL, Token: s.NodelistToken}) + } + targets = append(targets, s.NodelistPeers...) + if len(targets) == 0 { + return nil + } + return &poster{targets: targets, cl: &http.Client{Timeout: 5 * time.Second}} +} + +func (p *poster) PostMeta(call, node, system, event string, tg int, via, network string) { + if tg <= 0 { + return + } + body, _ := json.Marshal(map[string]any{ + "tg": tg, "event": event, "callsign": call, "node": node, + "system": system, "via": via, "network": network, + }) + for _, t := range p.targets { + req, err := http.NewRequest(http.MethodPost, t.URL, bytes.NewReader(body)) + if err != nil { + continue + } + req.Header.Set("Content-Type", "application/json") + if t.Token != "" { + req.Header.Set("X-Ingest-Token", t.Token) + } + resp, err := p.cl.Do(req) + if err != nil { + log.Printf("astmeta nodelist: post to %s failed: %v", t.URL, err) + continue + } + resp.Body.Close() + if resp.StatusCode >= 300 { + log.Printf("astmeta nodelist: post %s to %s returned HTTP %d", event, t.URL, resp.StatusCode) + } + } +} + +// ------------------------------------------------------------------ Run ------ + +var alinkEntry = regexp.MustCompile(`^([0-9]+)([A-Z]+)$`) + +// Run polls AMI for every watched node and emits talker metadata until ctx ends. +func Run(ctx context.Context, c *config.Config) error { + s := newSettings(c) + if len(s.WatchNodes) == 0 { + return fmt.Errorf("astmeta: no nodes to watch (nodes[] empty)") + } + if s.AmiUser == "" || s.AmiSecret == "" { + return fmt.Errorf("astmeta: ami_user/ami_secret required") + } + resolver := newResolver(ctx, s) + snd, err := newSender(s) + if err != nil { + return fmt.Errorf("astmeta usrp: %w", err) + } + pub := newPublisher(s) + post := newPoster(s) + log.Printf("astmeta: watching %v via AMI %s:%s (usrp=%v mqtt=%v post=%v)", + s.WatchNodes, s.AmiHost, s.AmiPort, snd != nil, pub != nil, post != nil) + + a := &ami{s: s} + keyed := map[string]bool{} + unkeyedAt := map[string]time.Time{} + lastSent := map[string]time.Time{} + + for { + if ctx.Err() != nil { + return nil + } + if a.conn == nil { + if err := a.connect(); err != nil { + log.Printf("astmeta ami: %v — retrying in 10s", err) + if !sleepCtx(ctx, 10*time.Second) { + return nil + } + continue + } + log.Printf("astmeta ami: connected") + } + if !sleepCtx(ctx, s.PollInterval) { + return nil + } + + seen := map[string]bool{} + fail := false + for _, wn := range s.WatchNodes { + tg := s.tgFor(wn) + via := s.viaFor(wn) + tgNum, _ := strconv.Atoi(tg) + + entries, err := a.alinks(wn) + if err != nil { + log.Printf("astmeta ami: %v — reconnecting", err) + a.conn.Close() + a.conn = nil + fail = true + break + } + for _, e := range entries { + f := alinkEntry.FindStringSubmatch(e) + if f == nil { + continue + } + node, flags := f[1], f[2] + if s.IgnoreNodes[node] { + continue + } + reportNode := node + if static := s.NodeAllstar[wn]; static != "" && node != static && s.ownNodes[node] { + reportNode = static + } + k := wn + "|" + node + isKeyed := strings.HasSuffix(flags, "K") + seen[k] = true + refresh := s.RefreshInterval > 0 && time.Since(lastSent[k]) >= s.RefreshInterval + if isKeyed { + delete(unkeyedAt, k) + if !keyed[k] || refresh { + call, system := resolver.Resolve(reportNode) + display := system + if display == "" { + display = call + } + if snd != nil { + if err := snd.Send(display, reportNode, tg); err != nil { + log.Printf("astmeta usrp: send failed: %v", err) + } + } + if !keyed[k] { + log.Printf("astmeta talker start: node %s call %s system %q tg %s via %s", + reportNode, call, system, tg, via) + } + if pub != nil { + pub.PublishMeta(call, reportNode, system, tg, via, s.Network) + } + if post != nil { + post.PostMeta(call, reportNode, system, "start", tgNum, via, s.Network) + } + lastSent[k] = time.Now() + } + keyed[k] = true + } else if keyed[k] { + if _, pending := unkeyedAt[k]; !pending { + unkeyedAt[k] = time.Now() + } + if s.HangTime <= 0 || time.Since(unkeyedAt[k]) >= s.HangTime { + log.Printf("astmeta talker stop: node %s tg %s", reportNode, tg) + if post != nil { + post.PostMeta("", reportNode, "", "stop", tgNum, via, s.Network) + } + keyed[k] = false + delete(unkeyedAt, k) + } + } + } + } + if !fail { + for k := range keyed { + if !seen[k] { + delete(keyed, k) + delete(lastSent, k) + delete(unkeyedAt, k) + } + } + } + } +} diff --git a/svxdr/config/config.go b/svxdr/config/config.go new file mode 100644 index 0000000..6d9f1cb --- /dev/null +++ b/svxdr/config/config.go @@ -0,0 +1,412 @@ +// svxdr — live-data daemon for SvxPortal (nodelist + proxy + astmeta + xlink) +// Copyright (C) 2026 SA7AUX (Björn) +// SPDX-License-Identifier: AGPL-3.0-or-later +// +// This program is free software: you can redistribute it and/or modify it under +// the terms of the GNU Affero General Public License as published by the Free +// Software Foundation, either version 3 of the License, or (at your option) any +// later version. See the LICENSE file for details. + +// Package config is the ONE config for svxdr — the merged svx daemon +// (astmeta + nodelist + proxy + xlink). Lives at /etc/svxlink/svxdr.json. +// +// One file, sectioned per subsystem plus shared sections (redis, nodes, trunk, +// lastheard). Read at start and on reload (SIGHUP / file-watch) and held in +// memory. `modules` toggles which subsystems run on this box, so the same +// binary runs all of them on speed/aldmr and only astmeta on the allstar box. +// +// Secrets carry an @@PLACEHOLDER@@ that deploy-remote.sh --conf fills from the +// live file per box, so the structure is one file but secrets differ per host. +package config + +import ( + "bufio" + "encoding/json" + "os" + "strconv" + "strings" + "time" +) + +// DefaultPath is where the merged config lives. +const DefaultPath = "/etc/svxlink/svxdr.json" + +// Config is the whole svxdr.json. +type Config struct { + Modules Modules `json:"modules"` // per-box on/off + Redis Redis `json:"redis"` // shared + Nodes []Node `json:"nodes"` // shared bridge map + Trunk Trunk `json:"trunk"` // shared chan_svx_sat trunk + Lastheard Lastheard `json:"lastheard"` // shared render config + Astmeta Astmeta `json:"astmeta"` + Nodelist Nodelist `json:"nodelist"` + Proxy Proxy `json:"proxy"` + Xlink Xlink `json:"xlink"` + Tglist Tglist `json:"tglist"` // shared canonical TG->name list source +} + +// Modules toggles which subsystems this box runs. Unset (all false) is treated +// as "run nothing" — a box MUST opt in, so a fresh/empty file never silently +// starts services. +type Modules struct { + Astmeta bool `json:"astmeta"` + Nodelist bool `json:"nodelist"` + Proxy bool `json:"proxy"` + Xlink bool `json:"xlink"` +} + +// Redis — shared connection (as today). +type Redis struct { + Socket string `json:"socket"` // /run/redis/redis.sock + Addr string `json:"addr"` // host:port (if not socket) + Password string `json:"password"` // @@PLACEHOLDER@@ per box + DB int `json:"db"` + // key namespace (kept configurable, defaults mirror today's env) + StatusKey string `json:"status_key"` + StatusHashKey string `json:"status_hash_key"` + PubsubChan string `json:"pubsub_chan"` + NodesOnlineKey string `json:"nodes_online_key"` + NodeKeyPrefix string `json:"node_key_prefix"` + NodeConnectChan string `json:"node_connect_chan"` + NodeDisconnectChan string `json:"node_disconnect_chan"` + NodeUpdateChan string `json:"node_update_chan"` + LastheardKey string `json:"lastheard_key"` +} + +// Node — one AllStar↔reflector bridge node (the nodes.json bridge map). +type Node struct { + SvxNode int `json:"svx_node"` + TG int `json:"tg"` + Allstar int `json:"allstar"` + System string `json:"system"` + Via string `json:"via"` + Callsign string `json:"callsign"` + Location string `json:"location,omitempty"` + // Group = the shared reflector login these per-TG nodes share in the /status + // data (e.g. "KT4ROY-ASL", "KT4ROY-DMR"). Config-driven: any group works the + // same, nothing about ASL/DMR/etc is hardcoded in the web. Nodes with a Group + // are rendered per-TG under that group's trunk. + Group string `json:"group,omitempty"` + // Mode = free-text type shown for this node (AllStar, DMR, YSF, SVX, or anything + // you write). Config wins; empty falls back to the data-derived type. No hardcode. + Mode string `json:"mode,omitempty"` + Enabled *bool `json:"enabled,omitempty"` // nil/true = enabled +} + +func (n Node) IsEnabled() bool { return n.Enabled == nil || *n.Enabled } + +// Trunk — the shared chan_svx_sat trunk (for the Asterisk generator). +type Trunk struct { + SatelliteID string `json:"satellite_id"` + Host string `json:"host"` + Port int `json:"port"` + // ConfPath = chan_svx_sat's svx_sat.conf. Last Heard reads the per-node + // reflector name (base callsign + suffix) straight from it, so the name has + // ONE source and is never duplicated here. Optional; defaults to + // /etc/asterisk/svx_sat.conf. + ConfPath string `json:"conf_path"` +} + +// Lastheard — how bridge/AllStar rows render. +type Lastheard struct { + BridgeLocationFormat string `json:"bridge_location_format"` + BridgeLocationFormatNoSystem string `json:"bridge_location_format_no_system"` + // reflector_names is derived from nodes[] (via/callsign/allstar -> system) + // but an explicit map here can still add names nodes[] doesn't cover. + ReflectorNames map[string]string `json:"reflector_names,omitempty"` +} + +// Astmeta — AllStar AMI poller settings (was /etc/astmeta.conf, minus the +// per-node bindings which now live in nodes[]/[names]). +type Astmeta struct { + AmiHost string `json:"ami_host"` + AmiPort string `json:"ami_port"` + AmiUser string `json:"ami_user"` + AmiSecret string `json:"ami_secret"` // @@PLACEHOLDER@@ + IgnoreNodes []string `json:"ignore_nodes"` + PollInterval Duration `json:"poll_interval"` + HangTime Duration `json:"hang_time"` + Network string `json:"network"` // ingress mode label (default AllStar) + // USRP sink (svxlink UsrpLogic), MQTT publish, lookup — kept as today. + UsrpTarget string `json:"usrp_target"` + Format string `json:"format"` + SpaceReplace string `json:"space_replace"` + RefreshInterval Duration `json:"refresh_interval"` + LookupEnabled bool `json:"lookup_enabled"` + LookupURL string `json:"lookup_url"` + LookupFile string `json:"lookup_file"` + CacheTTL Duration `json:"cache_ttl"` + // resolver node->callsign cache in the shared redis (astmeta.conf [redis]). + RedisCache bool `json:"redis_cache"` + RedisCachePrefix string `json:"redis_cache_prefix"` + MqttEnabled bool `json:"mqtt_enabled"` + MqttBroker string `json:"mqtt_broker"` + MqttUser string `json:"mqtt_user"` + MqttPass string `json:"mqtt_pass"` // @@PLACEHOLDER@@ + MqttPrefix string `json:"mqtt_prefix"` + // /api/talker-meta targets (local nodelist/proxy handled in-memory when + // co-located; these are for a REMOTE svxdr, e.g. astmeta on the allstar box). + NodelistURL string `json:"nodelist_url"` + NodelistToken string `json:"nodelist_token"` // @@PLACEHOLDER@@ + NodelistPeers []NodeMetaURL `json:"nodelist_peers"` +} + +type NodeMetaURL struct { + URL string `json:"url"` + Token string `json:"token"` +} + +// Nodelist — reflector status crawl + lastheard + talker-meta API. +type Nodelist struct { + ListenAddr string `json:"listen_addr"` + ReflectorStatus string `json:"reflector_status_url"` + MonitorStatusURL string `json:"monitor_status_url"` + Debug bool `json:"debug"` + PollInterval Duration `json:"poll_interval"` + ReflectorLogPath string `json:"reflector_log_path"` + NetworkJSONPath string `json:"network_json_path"` + AstdbURL string `json:"astdb_url"` + TalkerMetaTTL Duration `json:"talker_meta_ttl"` + TalkerIngestToken string `json:"talker_ingest_token"` // @@PLACEHOLDER@@ + AllstarNodesTTL Duration `json:"allstar_nodes_ttl"` + NodeOnlineTTL Duration `json:"node_online_ttl"` + NodeTTL Duration `json:"node_ttl"` + // nodelist's own MQTT publish of reflector status (MQTT_* env). + MqttBroker string `json:"mqtt_broker"` + MqttUser string `json:"mqtt_user"` + MqttPass string `json:"mqtt_pass"` + MqttPrefix string `json:"mqtt_prefix"` + // How often a REMOTE Monitor->Nodes read hits the master status. Kept gentle so + // a shared hub is never hammered (local reads stay fast). + MonitorInterval Duration `json:"monitor_interval"` + // external reflector federation (the "which other reflectors we crawl" list) + External External `json:"external"` + Federation Federation `json:"federation"` +} + +type External struct { + PullURL string `json:"pull_url"` + PullScheme string `json:"pull_scheme"` + PullPort string `json:"pull_port"` + PullPath string `json:"pull_path"` + PullInterval Duration `json:"pull_interval"` + StatusPorts []string `json:"status_ports"` + StatusPath string `json:"status_path"` + ProxyPath string `json:"proxy_path"` + DetailPath string `json:"detail_path"` + MaxRefs int `json:"max_refs"` + StaleTTL Duration `json:"stale_ttl"` +} + +type Federation struct { + Seeds []string `json:"seeds"` + AggregateURL string `json:"aggregate_url"` + AggregateEvery Duration `json:"aggregate_interval"` +} + +// Proxy — enriched status proxy. +type Proxy struct { + Listen string `json:"listen"` + UpstreamURL string `json:"reflector_status_url"` + CacheTTL Duration `json:"cache_ttl"` + FetchTimeout Duration `json:"fetch_timeout"` + MetaTTL Duration `json:"meta_ttl"` + IngestToken string `json:"ingest_token"` // @@PLACEHOLDER@@ + UpstreamPeer string `json:"upstream_status_url"` + ForwardTalker string `json:"forward_talker_url"` + SyncInterval Duration `json:"sync_interval"` + RemoteTTL Duration `json:"remote_ttl"` +} + +// Xlink — mesh talker publish over MQTT (was the XLINK_* env in nodelist). +type Xlink struct { + Broker string `json:"broker"` + User string `json:"user"` + Pass string `json:"pass"` // @@PLACEHOLDER@@ + Topic string `json:"topic"` + Origin string `json:"origin"` +} + +// Tglist — the shared canonical TG -> name/system list, edited in svx-web and +// served at /tglist.php. Every program points its own URL here (per-box +// override, no hardcoding). Empty URL = feature off: fall back to the per-box +// nodes[].system. The list is cached to disk so a fetch failure keeps the last +// good copy (consumers stay independent of the list host being reachable). +type Tglist struct { + File string `json:"file"` // local JSON file (SVX source of truth, /etc/svxlink/tglist.json) — read directly, no HTTP + URL string `json:"url"` // alternative: fetch over HTTP (e.g. https://host/tglist.php). File wins if both set. + RefreshInterval Duration `json:"refresh_interval"` // default 15m + CacheFile string `json:"cache_file"` // disk fallback for the URL path (default /var/lib/svxdr/tglist.json) +} + +// Duration is a time.Duration that (un)marshals as a Go duration string ("2s"). +type Duration time.Duration + +func (d Duration) D() time.Duration { return time.Duration(d) } + +func (d *Duration) UnmarshalJSON(b []byte) error { + var s string + if err := json.Unmarshal(b, &s); err != nil { + // also accept a bare number = seconds + var n float64 + if err2 := json.Unmarshal(b, &n); err2 != nil { + return err + } + *d = Duration(time.Duration(n * float64(time.Second))) + return nil + } + if s == "" { + *d = 0 + return nil + } + v, err := time.ParseDuration(s) + if err != nil { + return err + } + *d = Duration(v) + return nil +} + +func (d Duration) MarshalJSON() ([]byte, error) { + return json.Marshal(time.Duration(d).String()) +} + +// Load reads and parses the config file. +func Load(path string) (*Config, error) { + b, err := os.ReadFile(path) + if err != nil { + return nil, err + } + var c Config + if err := json.Unmarshal(b, &c); err != nil { + return nil, err + } + // AllStar/sat-node config lives in chan_svx_sat's svx_sat.conf (ONE file for + // everything AllStar). Parse it into Node entries and merge, so nodes[] in + // svxdr.json only needs the non-AllStar nodes (e.g. the DMR side). Every + // downstream subsystem (astmeta/nodelist/proxy/web) then reads one source. + c.Nodes = append(c.Nodes, loadSatNodes(c.Trunk.ConfPath)...) + return &c, nil +} + +// loadSatNodes reads chan_svx_sat's svx_sat.conf and turns each node section into +// a Node: SvxNode = the section id, TG/Allstar/System from its keys, and the +// display name (Via = Callsign) computed exactly like chan_svx_sat's svx_eff_cs +// (bare trunk/node callsign, node designator stripped at the first '-', plus the +// per-node suffix). Group = the trunk's satellite_id. Missing file -> nil. +func loadSatNodes(confPath string) []Node { + if confPath == "" { + confPath = "/etc/asterisk/svx_sat.conf" + } + f, err := os.Open(confPath) + if err != nil { + return nil + } + defer f.Close() + + trunkCS := map[string]string{} // trunk name -> callsign + trunkSat := map[string]string{} // trunk name -> satellite_id + type nc struct { + trunk, callsign, suffix, system string + tg, allstar int + } + nodes := map[string]*nc{} + order := []string{} + cur, curTrunk := "", "" + + sc := bufio.NewScanner(f) + for sc.Scan() { + line := strings.TrimSpace(sc.Text()) + if line == "" || strings.HasPrefix(line, ";") || strings.HasPrefix(line, "#") { + continue + } + if strings.HasPrefix(line, "[") && strings.HasSuffix(line, "]") { + sec := strings.TrimSpace(line[1 : len(line)-1]) + if strings.HasPrefix(sec, "trunk:") { + curTrunk, cur = strings.TrimPrefix(sec, "trunk:"), "" + } else { + curTrunk, cur = "", sec + if nodes[cur] == nil { + nodes[cur] = &nc{} + order = append(order, cur) + } + } + continue + } + eq := strings.IndexByte(line, '=') + if eq < 0 { + continue + } + k := strings.TrimSpace(line[:eq]) + v := strings.TrimSpace(line[eq+1:]) + switch { + case curTrunk != "": + switch k { + case "callsign": + trunkCS[curTrunk] = v + case "satellite_id": + trunkSat[curTrunk] = v + } + case cur != "": + switch k { + case "trunk": + nodes[cur].trunk = v + case "callsign": + nodes[cur].callsign = v + case "suffix": + nodes[cur].suffix = v + case "system": + nodes[cur].system = v + case "tg": + nodes[cur].tg, _ = strconv.Atoi(v) + case "allstar": + nodes[cur].allstar, _ = strconv.Atoi(v) + } + } + } + + var out []Node + for _, sec := range order { + n := nodes[sec] + if n.tg == 0 { + continue + } + cs := n.callsign + if cs == "" { + cs = trunkCS[n.trunk] + } + if cs == "" { + continue + } + name := satEffCS(cs, n.suffix) + svxNode, _ := strconv.Atoi(sec) + group := trunkSat[n.trunk] + if group == "" { + group = trunkCS[n.trunk] + } + out = append(out, Node{ + SvxNode: svxNode, TG: n.tg, Allstar: n.allstar, + System: n.system, Via: name, Callsign: name, + Group: group, Mode: "AllStar", + }) + } + return out +} + +// satEffCS mirrors chan_svx_sat's svx_eff_cs: bare callsign (strip node +// designator at the first '-') + suffix; an alnum-leading suffix gets an auto "-". +// Empty suffix -> the callsign unchanged. +func satEffCS(callsign, suffix string) string { + if suffix == "" { + return callsign + } + base := callsign + if i := strings.IndexByte(base, '-'); i >= 0 { + base = base[:i] + } + c := suffix[0] + if (c >= '0' && c <= '9') || (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') { + return base + "-" + suffix + } + return base + suffix +} diff --git a/svxdr/go.mod b/svxdr/go.mod new file mode 100644 index 0000000..b79a674 --- /dev/null +++ b/svxdr/go.mod @@ -0,0 +1,16 @@ +module svxportal/svxdr + +go 1.24.4 + +require ( + github.com/eclipse/paho.mqtt.golang v1.5.1 + github.com/redis/go-redis/v9 v9.17.2 +) + +require ( + github.com/cespare/xxhash/v2 v2.3.0 // indirect + github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect + github.com/gorilla/websocket v1.5.3 // indirect + golang.org/x/net v0.44.0 // indirect + golang.org/x/sync v0.17.0 // indirect +) diff --git a/svxdr/go.sum b/svxdr/go.sum new file mode 100644 index 0000000..bae5ba0 --- /dev/null +++ b/svxdr/go.sum @@ -0,0 +1,18 @@ +github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs= +github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c= +github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA= +github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0= +github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= +github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78= +github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc= +github.com/eclipse/paho.mqtt.golang v1.5.1 h1:/VSOv3oDLlpqR2Epjn1Q7b2bSTplJIeV2ISgCl2W7nE= +github.com/eclipse/paho.mqtt.golang v1.5.1/go.mod h1:1/yJCneuyOoCOzKSsOTUc0AJfpsItBGWvYpBLimhArU= +github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg= +github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= +github.com/redis/go-redis/v9 v9.17.2 h1:P2EGsA4qVIM3Pp+aPocCJ7DguDHhqrXNhVcEp4ViluI= +github.com/redis/go-redis/v9 v9.17.2/go.mod h1:u410H11HMLoB+TP67dz8rL9s6QW2j76l0//kSOd3370= +golang.org/x/net v0.44.0 h1:evd8IRDyfNBMBTTY5XRF1vaZlD+EmWx6x8PkhR04H/I= +golang.org/x/net v0.44.0/go.mod h1:ECOoLqd5U3Lhyeyo/QDCEVQ4sNgYsqvCZ722XogGieY= +golang.org/x/sync v0.17.0 h1:l60nONMj9l5drqw6jlhIELNv9I0A4OFgRsG9k2oT9Ug= +golang.org/x/sync v0.17.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI= diff --git a/svxdr/main.go b/svxdr/main.go new file mode 100644 index 0000000..ca0de0a --- /dev/null +++ b/svxdr/main.go @@ -0,0 +1,132 @@ +// svxdr — live-data daemon for SvxPortal (nodelist + proxy + astmeta + xlink) +// Copyright (C) 2026 SA7AUX (Björn) +// SPDX-License-Identifier: AGPL-3.0-or-later +// +// This program is free software: you can redistribute it and/or modify it under +// the terms of the GNU Affero General Public License as published by the Free +// Software Foundation, either version 3 of the License, or (at your option) any +// later version. See the LICENSE file for details. + +// svxdr — the merged svx daemon: astmeta + nodelist + proxy + xlink in ONE +// process, each a subsystem goroutine, toggled per box by /etc/svxlink/svxdr.json. +// +// One binary, one config. Runs all subsystems on speed/aldmr; only astmeta on +// the allstar box. Co-located subsystems share memory (no localhost HTTP hop); +// the HTTP talker-meta ingest stays for a REMOTE astmeta feeding this box. +// +// Phase 1 scaffold: config + module toggles + lifecycle (start / SIGHUP reload / +// graceful stop). Subsystems are stubs here and are folded in one at a time +// (proxy, then astmeta, then nodelist+xlink) so the existing standalone +// binaries keep running until cutover. +package main + +import ( + "context" + "flag" + "log" + "os" + "os/signal" + "sync" + "syscall" + + "svxportal/svxdr/astmeta" + "svxportal/svxdr/config" + "svxportal/svxdr/nodelist" + "svxportal/svxdr/proxy" +) + +// module is one subsystem. Run blocks until ctx is cancelled; it should return +// nil on clean shutdown. reload is called (best-effort) when the config file +// changes so a running module can pick up new settings without a restart. +type module struct { + name string + enabled func(*config.Config) bool + run func(ctx context.Context, cfg *config.Config) error +} + +var modules = []module{ + {"proxy", func(c *config.Config) bool { return c.Modules.Proxy }, proxy.Run}, + {"astmeta", func(c *config.Config) bool { return c.Modules.Astmeta }, astmeta.Run}, + {"nodelist", func(c *config.Config) bool { return c.Modules.Nodelist }, nodelist.Run}, + // xlink is not a separate process — it runs inside nodelist, gated by + // modules.xlink in svxdr.json (see nodelist/svxdrcfg.go). +} + +func main() { + log.SetFlags(log.LstdFlags) + path := flag.String("config", "", "path to svxdr.json (default "+config.DefaultPath+" or $SVXDR_CONFIG)") + fAstmeta := flag.Bool("astmeta", false, "run astmeta (overrides modules in config)") + fNodelist := flag.Bool("nodelist", false, "run nodelist (overrides modules in config)") + fProxy := flag.Bool("proxy", false, "run proxy (overrides modules in config)") + fXlink := flag.Bool("xlink", false, "run xlink inside nodelist (overrides modules in config)") + flag.Parse() + cfgPath := *path + if cfgPath == "" { + if p := os.Getenv("SVXDR_CONFIG"); p != "" { + cfgPath = p + } else { + cfgPath = config.DefaultPath + } + } + + cfg, err := config.Load(cfgPath) + if err != nil { + log.Fatalf("svxdr: config %s: %v", cfgPath, err) + } + + // If ANY module flag is passed, the flags fully define the set to run on this + // box (config "modules" is overridden) — handy for per-box units and testing. + if *fAstmeta || *fNodelist || *fProxy || *fXlink { + cfg.Modules = config.Modules{Astmeta: *fAstmeta, Nodelist: *fNodelist, Proxy: *fProxy, Xlink: *fXlink} + log.Printf("svxdr: module set overridden by flags: astmeta=%v nodelist=%v proxy=%v xlink=%v", + *fAstmeta, *fNodelist, *fProxy, *fXlink) + } + + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) + defer stop() + + // SIGHUP -> reload config from disk and hand the new config to modules. + hup := make(chan os.Signal, 1) + signal.Notify(hup, syscall.SIGHUP) + + var wg sync.WaitGroup + started := 0 + for _, m := range modules { + if !m.enabled(cfg) { + continue + } + started++ + wg.Add(1) + go func(m module) { + defer wg.Done() + log.Printf("svxdr: starting module %q", m.name) + if err := m.run(ctx, cfg); err != nil { + log.Printf("svxdr: module %q exited: %v", m.name, err) + } + }(m) + } + if started == 0 { + log.Printf("svxdr: no modules enabled in %s (set \"modules\": {...}) — idle", cfgPath) + } else { + log.Printf("svxdr: %d module(s) running from %s", started, cfgPath) + } + + // lifecycle loop: reload on SIGHUP, exit on signal/context. + for { + select { + case <-ctx.Done(): + log.Printf("svxdr: shutting down") + wg.Wait() + return + case <-hup: + nc, err := config.Load(cfgPath) + if err != nil { + log.Printf("svxdr: reload failed, keeping current config: %v", err) + continue + } + cfg = nc + log.Printf("svxdr: config reloaded from %s (modules pick up changes on next cycle)", cfgPath) + // NOTE: per-module live reload is wired as each subsystem is folded in. + } + } +} diff --git a/svxdr/nodelist/local_trunks.go b/svxdr/nodelist/local_trunks.go new file mode 100644 index 0000000..109fc34 --- /dev/null +++ b/svxdr/nodelist/local_trunks.go @@ -0,0 +1,299 @@ +// svxdr — live-data daemon for SvxPortal (nodelist + proxy + astmeta + xlink) +// Copyright (C) 2026 SA7AUX (Björn) +// SPDX-License-Identifier: AGPL-3.0-or-later +// +// This program is free software: you can redistribute it and/or modify it under +// the terms of the GNU Affero General Public License as published by the Free +// Software Foundation, either version 3 of the License, or (at your option) any +// later version. See the LICENSE file for details. + +package nodelist + +// Monitor → Trunks lists ONLY the trunks THIS reflector is configured to connect +// to — read straight from the local svxreflector.conf. Two config dialects exist +// in the wild and both are supported: +// +// 1) newer GEU fork: +// [ReflectorTrunk] +// Peers=SE-AUX +// [TrunkPeer#SE-AUX] +// Host=reflector.reflector-network.se +// +// 2) older fork: +// [TRUNK_US_SE] +// HOST=reflector.reflector-network.se +// PORT=5302 +// REMOTE_PREFIX=240 +// +// The configured trunk NAME is matched against the LOCAL reflector's /status to +// fill online/offline + node count. The host is used only for that match and is +// never sent to the client (host/IP must never be exposed). + +import ( + "bufio" + "encoding/json" + "net/http" + "os" + "sort" + "strings" + "sync" + "time" +) + +// svxreflectorConf is declared in main.go (default /etc/svxlink/svxreflector.conf). + +// satTG is one talkgroup carried by a satellite trunk (per-TG label + tg number). +type satTG struct { + Name string `json:"name"` + TG string `json:"tg"` +} + +type localTrunk struct { + Name string `json:"name"` + Connected bool `json:"connected"` + Nodes int `json:"nodes"` + TGs []satTG `json:"tgs,omitempty"` // satellite trunks: the TGs it carries +} + +type confTrunk struct { + name string + hosts []string // normalized, for status matching only — never exposed +} + +// normTrunkHost strips scheme, path and port and lowercases a host. +func normTrunkHost(h string) string { + h = strings.ToLower(strings.TrimSpace(h)) + if i := strings.Index(h, "://"); i >= 0 { + h = h[i+3:] + } + if i := strings.IndexByte(h, '/'); i >= 0 { + h = h[:i] + } + if i := strings.LastIndexByte(h, ':'); i >= 0 { + h = h[:i] + } + return strings.Trim(h, "[]") +} + +// parseConfTrunks reads svxreflector.conf and returns the configured trunks +// (both dialects). Order is preserved; duplicates by name are dropped. +func parseConfTrunks(path string) []confTrunk { + f, err := os.Open(path) + if err != nil { + return nil + } + defer f.Close() + + type section struct { + name string + kv map[string]string + } + var order []*section + byName := map[string]*section{} + var cur *section + sc := bufio.NewScanner(f) + for sc.Scan() { + line := strings.TrimSpace(sc.Text()) + if line == "" || strings.HasPrefix(line, "#") || strings.HasPrefix(line, ";") { + continue + } + if strings.HasPrefix(line, "[") && strings.HasSuffix(line, "]") { + cur = §ion{name: line[1 : len(line)-1], kv: map[string]string{}} + order = append(order, cur) + byName[cur.name] = cur + continue + } + if cur == nil { + continue + } + if i := strings.IndexByte(line, '='); i > 0 { + cur.kv[strings.ToUpper(strings.TrimSpace(line[:i]))] = strings.TrimSpace(line[i+1:]) + } + } + + var out []confTrunk + seen := map[string]bool{} + add := func(name, hostCSV string) { + name = strings.TrimSpace(name) + if name == "" || seen[name] { + return + } + seen[name] = true + var hosts []string + for _, h := range strings.Split(hostCSV, ",") { + if hh := normTrunkHost(h); hh != "" { + hosts = append(hosts, hh) + } + } + out = append(out, confTrunk{name: name, hosts: hosts}) + } + + // Dialect 1: [ReflectorTrunk] Peers=A,B -> [TrunkPeer#A] Host=... + if rt := byName["ReflectorTrunk"]; rt != nil { + for _, p := range strings.Split(rt.kv["PEERS"], ",") { + p = strings.TrimSpace(p) + if p == "" { + continue + } + host := "" + if peer := byName["TrunkPeer#"+p]; peer != nil { + host = peer.kv["HOST"] + } + add(p, host) + } + } + // Dialect 2: [TRUNK_xxx] HOST=... + for _, s := range order { + if strings.HasPrefix(strings.ToUpper(s.name), "TRUNK_") { + add(s.name, s.kv["HOST"]) + } + } + return out +} + +var localTrunkState = struct { + mu sync.RWMutex + data []localTrunk + src string +}{data: []localTrunk{}} + +// buildLocalTrunks parses the conf and matches each trunk against the LOCAL +// reflector's /status (by name, then host) for online + node count. +func buildLocalTrunks(client *http.Client) { + conf := parseConfTrunks(svxreflectorConf) + + type liveT struct { + connected bool + nodes int + host string + } + live := map[string]liveT{} + // Satellite trunks (chan_svx_sat) live status, keyed by reflector sat id. + type liveSat struct { + connected bool + nodes int + } + liveSats := map[string]liveSat{} + if resp, err := client.Get(reflectorURL); err == nil { + func() { + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return + } + var raw struct { + Trunks map[string]struct { + Host string `json:"host"` + Connected bool `json:"connected"` + Nodes []json.RawMessage `json:"nodes"` + } `json:"trunks"` + TrunksUDP []struct { + Name string `json:"name"` + Status bool `json:"status"` + Host string `json:"host"` + } `json:"trunks_UDP"` + Satellites map[string]struct { + Authenticated bool `json:"authenticated"` + Nodes []json.RawMessage `json:"nodes"` + } `json:"satellites"` + } + if json.NewDecoder(resp.Body).Decode(&raw) == nil { + for n, t := range raw.Trunks { + live[n] = liveT{connected: t.Connected, nodes: len(t.Nodes), host: normTrunkHost(t.Host)} + } + for id, s := range raw.Satellites { + liveSats[id] = liveSat{connected: s.Authenticated, nodes: len(s.Nodes)} + } + // trunks_UDP peers are named like "TrunkPeer#SE-AUX"; index them + // both raw and with the prefix stripped so a conf name matches. + for _, t := range raw.TrunksUDP { + v := liveT{connected: t.Status, host: normTrunkHost(t.Host)} + live[t.Name] = v + if s := strings.TrimPrefix(t.Name, "TrunkPeer#"); s != t.Name { + live[s] = v + } + } + } + }() + } + + out := make([]localTrunk, 0, len(conf)) + for _, c := range conf { + lt := localTrunk{Name: c.name} + if v, ok := live[c.name]; ok { // exact name match — most reliable + lt.Connected, lt.Nodes = v.connected, v.nodes + } else { + for _, v := range live { // fall back to host match + if v.host == "" { + continue + } + for _, h := range c.hosts { + if v.host == h || strings.Contains(v.host, h) || strings.Contains(h, v.host) { + lt.Connected, lt.Nodes = v.connected, v.nodes + } + } + } + } + out = append(out, lt) + } + + // Satellite trunks (chan_svx_sat): ONE trunk per satellite, named by the + // satellite id (e.g. KT4ROY-ASL). All chan_svx_sat entries that share the same + // satellite_id are the SAME trunk carrying N talkgroups — they are grouped, not + // listed as one-trunk-per-TG. The per-TG labels (callsign + tg) are attached as + // TGs so the UI can tell the talkgroups apart under the single trunk name. + // Fully dynamic: the satellite set is the union of the configured [SATELLITE] + // ids in svxreflector.conf and any live-authenticated sat in the reflector + // /status, so new satellites (Chicago, …) appear automatically and offline ones + // still show. Secrets from [SATELLITE] are never read/exposed — only the id. + satSet := map[string]bool{} + for id := range liveSats { + satSet[id] = true + } + for _, id := range satelliteIDsFromReflectorConf(svxreflectorConf) { + satSet[id] = true + } + for _, id := range satTrunkIDs() { // configured satellites always show, even offline + satSet[id] = true + } + if len(satSet) > 0 { + ids := make([]string, 0, len(satSet)) + for id := range satSet { + ids = append(ids, id) + } + sort.Strings(ids) + for _, id := range ids { + s := liveSats[id] // zero value (offline, 0 nodes) if not live + out = append(out, localTrunk{ + Name: id, // trunk name = satellite id (KT4ROY-ASL) + Connected: s.connected, + Nodes: s.nodes, + TGs: satTGsForTrunk(id), // TGs + per-TG names straight from svxdr.json + }) + } + } + + localTrunkState.mu.Lock() + localTrunkState.data = out + localTrunkState.src = svxreflectorConf + localTrunkState.mu.Unlock() +} + +func localTrunksStart(mux *http.ServeMux) { + client := &http.Client{Timeout: 8 * time.Second} + buildLocalTrunks(client) + go func() { + t := time.NewTicker(10 * time.Second) + defer t.Stop() + for range t.C { + buildLocalTrunks(client) + } + }() + + mux.HandleFunc("/api/local-trunks", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json; charset=utf-8") + localTrunkState.mu.RLock() + b, _ := json.Marshal(map[string]interface{}{"trunks": localTrunkState.data, "source": localTrunkState.src}) + localTrunkState.mu.RUnlock() + writeMaybeGzip(w, r, b) + }) +} diff --git a/svxdr/nodelist/monitor_nodes.go b/svxdr/nodelist/monitor_nodes.go new file mode 100644 index 0000000..de55af1 --- /dev/null +++ b/svxdr/nodelist/monitor_nodes.go @@ -0,0 +1,445 @@ +// svxdr — live-data daemon for SvxPortal (nodelist + proxy + astmeta + xlink) +// Copyright (C) 2026 SA7AUX (Björn) +// SPDX-License-Identifier: AGPL-3.0-or-later +// +// This program is free software: you can redistribute it and/or modify it under +// the terms of the GNU Affero General Public License as published by the Free +// Software Foundation, either version 3 of the License, or (at your option) any +// later version. See the LICENSE file for details. + +package nodelist + +// Monitor Nodes, the simple way: read ONE reflector /status endpoint that +// already carries the whole picture (local nodes + every trunk with its nodes) +// and reshape it into {local, trunks:[{name,host,nodes}]}. No recursive crawl. +// +// Which URL is read is per-box config MONITOR_STATUS_URL (falls back to +// REFLECTOR_STATUS_URL): +// speed → https://reflector.sm2ampr.net/geu/status/ (the master's full view) +// aldmr → http://127.0.0.1:8080/status (its own reflector) + +import ( + "encoding/json" + "io" + "net/http" + "regexp" + "sort" + "strconv" + "strings" + "sync" + "time" + + "svxportal/svxdr/statusnorm" +) + +var monitorStatusURL = env("MONITOR_STATUS_URL", reflectorURL) + +// monitorInterval is how often REMOTE status reads (a Monitor->Nodes URL or a +// trunk_status topology URL that points at another server) run. Gentle by default +// so a shared hub is never hammered; override via svxdr.json nodelist.monitor_interval. +// A LOCAL read (127.0.0.1) is free and refreshes fast — see localReadInterval. +var monitorInterval = 60 * time.Second + +// localReadInterval: reading our OWN reflector costs nothing, so do it often. +var localReadInterval = 5 * time.Second + +// isLocalURL reports whether a status URL points at this host (loopback). Local +// reads may be frequent; remote reads must be throttled to monitorInterval. +func isLocalURL(u string) bool { + return strings.Contains(u, "127.0.0.1") || strings.Contains(u, "localhost") || strings.Contains(u, "[::1]") +} + +// readInterval picks the fast (local) or gentle (remote) cadence for a URL. +func readInterval(u string) time.Duration { + if isLocalURL(u) { + return localReadInterval + } + return monitorInterval +} + +type monNode struct { + Callsign string `json:"callsign"` + TG string `json:"tg,omitempty"` + NodeClass string `json:"nodeClass,omitempty"` + Mode string `json:"mode,omitempty"` + Talker bool `json:"talker,omitempty"` +} + +// uniquifyNames makes every node name in the view unique WITHOUT ever touching the +// Local (top-level) nodes — those are the reflector's own map keys, exactly what the +// Reflector clients view shows, so renaming them would make the two views disagree. +// Only trunk/satellite array nodes (which have no key and never appear in Reflector +// clients) are numbered, and only enough to stay clear of every name already taken +// (Local keys + earlier array nodes): SA7AUX -> SA7AUX-1, SA7AUX-2, … +func uniquifyNames(out *monStatus) { + taken := map[string]bool{} + for _, n := range out.Local { // canonical, never renamed + taken[n.Callsign] = true + } + next := map[string]int{} + for i := range out.Trunks { + for j := range out.Trunks[i].Nodes { + base := out.Trunks[i].Nodes[j].Callsign + if !taken[base] { + taken[base] = true + continue + } + for { + next[base]++ + cand := base + "-" + strconv.Itoa(next[base]) + if !taken[cand] { + taken[cand] = true + out.Trunks[i].Nodes[j].Callsign = cand + break + } + } + } + } +} + +type monTrunk struct { + Name string `json:"name"` + Host string `json:"host,omitempty"` + Connected bool `json:"connected"` + Nodes []monNode `json:"nodes"` +} + +type monStatus struct { + Local []monNode `json:"local"` + Trunks []monTrunk `json:"trunks"` + Source string `json:"source"` + UpdatedAt string `json:"updated_at"` +} + +var monState = struct { + mu sync.RWMutex + data monStatus +}{} + +// rawNode is one node as it appears either as a value in the top-level "nodes" +// object (keyed by callsign) or as an item in a trunk's "nodes" array. +type rawNode struct { + Callsign string `json:"callsign"` + SW string `json:"sw"` + TG json.Number `json:"tg"` + NodeClass string `json:"nodeClass"` + RestrictedTG bool `json:"restrictedTG"` + IsTalker bool `json:"isTalker"` + System string `json:"system"` // per-node link/type from the data (e.g. "DMR", "DMR/YSF", "DMR/Hytera") + Reflector string `json:"Reflector"` // which reflector this node is on (master status carries it) + MonitoredTGs []json.Number `json:"monitoredTGs"` + Links []struct { + LocalTg json.Number `json:"localTg"` + } `json:"links"` +} + +// masterReflectorName picks a label for the monitored master's own nodes purely +// from the DATA (the Reflector field the nodes report) — no dependency on the +// local conf format, so every box derives it the same way. +func masterReflectorName(nodes map[string]json.RawMessage) string { + count := map[string]int{} + for _, raw := range nodes { + var r struct { + Reflector string `json:"Reflector"` + } + if json.Unmarshal(raw, &r) == nil && strings.TrimSpace(r.Reflector) != "" { + count[r.Reflector]++ + } + } + best, n := "", 0 + for k, c := range count { + if c > n { + best, n = k, c + } + } + return best +} + +// monModeOf: real type from the software string (svx nodes carry no "mode"). +func monModeOf(sw string) string { + s := strings.ToLower(sw) + switch { + case strings.Contains(s, "chan_pbx"): + return "PBX" + case strings.Contains(s, "chan_svx_sat"): + return "AllStar" + case strings.Contains(s, "chan_svx"): + return "AllStar" + case strings.Contains(s, "svxlink"): + return "SVX" + case strings.Contains(s, "mmdvm"), strings.Contains(s, "brandmeister"), strings.Contains(s, "dmr"): + return "DMR" + case strings.Contains(s, "ysf"): + return "YSF" + case strings.Contains(s, "nxdn"): + return "NXDN" + case strings.Contains(s, "m17"): + return "M17" + case sw != "": + return sw + default: + return "" + } +} + +// A bridge node just links systems together (chan_svx asterisk, a DMR/trunk +// bridge, or an -AC/-DMR/-T suffix) — not a real user/radio client, so we never +// show it. chan_svx_sat IS a real AllStar node over the satellite trunk and is +// kept. Mirrors the TG Monitor's isBridge so both views show only real users. +var bridgeSuffix = regexp.MustCompile(`(?i)-(AC|DMR|T)$`) + +func isBridgeNode(callsign, sw, nodeClass string) bool { + s := strings.ToLower(sw) + // AllStar radio-network peers ARE shown — the links to svx/dmr/ysf/nxdn/m17 + // are exactly what the user wants visible under their node/trunk. chan_svx and + // chan_svx_sat both count. Only pure non-radio infrastructure stays hidden. + if strings.Contains(s, "chan_svx") || strings.Contains(s, "chan_pbx") { // chan_svx + chan_svx_sat + chan_pbx_svx (PBX) + return false + } + nc := strings.ToLower(nodeClass) + switch nc { + case "dmr", "ysf", "nxdn", "m17": + return false // radio-network peer — show + case "xlx", "mumble", "stream", "echolink", "sip", "zello": + return true // non-radio / D-Star infra — hide + } + // sw-based radio bridges (DMR_/YSF_/NXDN_/M17_Bridge) → show; the remaining + // "*bridge" (XLX_/Mumble_/Janus Bridge) is non-radio infra → hide. + if strings.Contains(s, "dmr") || strings.Contains(s, "ysf") || + strings.Contains(s, "nxdn") || strings.Contains(s, "m17") { + return false + } + if strings.Contains(s, "bridge") { + return true + } + // pure infrastructure gateway callsigns are never real peers. + u := strings.ToUpper(callsign) + if strings.HasPrefix(u, "SIP") || strings.HasPrefix(u, "DCS") || + strings.HasPrefix(u, "XLX") || strings.Contains(u, "MUMBLE") { + return true + } + return false // default: show (real users + radio peers) +} + +func toMonNode(callsign string, raw json.RawMessage) (monNode, bool) { + var r rawNode + _ = json.Unmarshal(raw, &r) + // Display name = the node's KEY (its registered identity — exactly what the + // Reflector clients view shows via the /status map key), so the two views can + // never disagree on a name. Trunk/satellite array nodes have no key -> use the + // callsign field (which statusnorm may have set to the per-TG group name). + cs := callsign + if cs == "" { + cs = r.Callsign + } + // A configured node (in nodes[]) is an intentional own node — always show it, + // even if its nodeClass would otherwise read as infra (e.g. a DMR gateway as + // "Stream"). Everything else keeps the bridge/infra filter. + if !isConfiguredName(cs) && isBridgeNode(cs, r.SW, r.NodeClass) { + return monNode{}, false // bridge, not a real user — hide it + } + // TG: the node's own tg, else its first monitored/linked TG. + tg := "" + if s := r.TG.String(); s != "" && s != "0" { + tg = s + } else if len(r.MonitoredTGs) > 0 { + tg = r.MonitoredTGs[0].String() + } else if len(r.Links) > 0 { + tg = r.Links[0].LocalTg.String() + } + if r.RestrictedTG && tg != "" { + tg = "*" + tg + } + // Per-TG name: several sat nodes share ONE reflector login (KT4ROY-ASL), so + // under a trunk they'd all show that same callsign. If svxdr.json configures a + // per-TG via for this node's TG, show THAT (KT4ROY-ASL1..4) — the trunk's own + // name is unaffected. One place (svxdr.json) drives it. + // Type/mode, in priority order (no hardcoding of any mode): + // 1) config nodes[].mode (explicit override, keyed by display name) + // 2) the node's own "system" field from the data (DMR, DMR/YSF, DMR/Hytera…) + // 3) type derived from the software string + mode := modeForName(cs) + if mode == "" { + mode = r.System + } + if mode == "" { + mode = monModeOf(r.SW) + } + return monNode{Callsign: cs, TG: tg, NodeClass: r.NodeClass, Mode: mode, Talker: r.IsTalker}, true +} + +// ownLocalNodes reads OUR OWN reflector (reflectorURL) and returns its top-level +// nodes — the ones genuinely local to THIS server. Used to keep Local accurate +// when MONITOR_STATUS_URL points at a foreign master. +func ownLocalNodes(client *http.Client) ([]monNode, bool) { + resp, err := client.Get(reflectorURL) + if err != nil { + return nil, false + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return nil, false + } + var raw struct { + Nodes map[string]json.RawMessage `json:"nodes"` + } + if json.NewDecoder(resp.Body).Decode(&raw) != nil { + return nil, false + } + var out []monNode + for cs, rn := range raw.Nodes { + if mn, ok := toMonNode(cs, rn); ok { + out = append(out, mn) + } + } + return out, true +} + +func buildMonitor(client *http.Client, generated string) (monStatus, bool) { + resp, err := client.Get(monitorStatusURL) + if err != nil { + return monStatus{}, false + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return monStatus{}, false + } + body, err := io.ReadAll(resp.Body) + if err != nil { + return monStatus{}, false + } + // Normalize at ingest: sat-login node callsigns -> per-TG name. Covers the + // case where monitorStatusURL is a foreign master (e.g. speed reads the SE hub + // where aldmr's sat nodes federate in). A local read via the proxy is already + // normalized; re-normalizing is a harmless no-op. + body = statusnorm.Normalize(body, renameSatNode) + var raw struct { + Nodes map[string]json.RawMessage `json:"nodes"` + Trunks map[string]struct { + Host string `json:"host"` + Connected bool `json:"connected"` + Nodes []json.RawMessage `json:"nodes"` + } `json:"trunks"` + // Satellite trunks (chan_svx_sat): each id (e.g. KT4ROY-ASL) carries N + // AllStar nodes. They live here, NOT in top-level nodes, so Monitor→Nodes + // must read them too or the sat nodes are invisible. + Satellites map[string]struct { + Authenticated bool `json:"authenticated"` + Nodes []json.RawMessage `json:"nodes"` + } `json:"satellites"` + } + if json.Unmarshal(body, &raw) != nil { + return monStatus{}, false + } + + out := monStatus{Source: monitorStatusURL, UpdatedAt: generated} + + // Local = only OUR reflector's own nodes. Whenever MONITOR_STATUS_URL is a + // REMOTE master (not our own reflector), ALWAYS split: read our OWN reflector for + // the local set and put the master's own top-level nodes under a section instead. + // This must behave the SAME on every box regardless of its svxreflector.conf + // dialect — so the split is decided purely by "is the master remote", and the + // section label is derived from the DATA (masterReflectorName) when the conf + // doesn't name the uplink trunk. No per-box special cases. + doSplit := reflectorURL != monitorStatusURL + masterLabel := "" + if doSplit { + if masterLabel = masterReflectorName(raw.Nodes); masterLabel == "" { + masterLabel = "uplink" + } + } + selfCalls := map[string]bool{} + if doSplit { + if selfNodes, ok := ownLocalNodes(client); ok { + for _, mn := range selfNodes { + out.Local = append(out.Local, mn) + selfCalls[strings.ToUpper(mn.Callsign)] = true + } + } + } + + var foreignTop []monNode + for cs, rn := range raw.Nodes { + mn, ok := toMonNode(cs, rn) + if !ok { + continue + } + if !doSplit { + out.Local = append(out.Local, mn) + } else if !selfCalls[strings.ToUpper(mn.Callsign)] { + foreignTop = append(foreignTop, mn) // the master's own node → its trunk section + } + } + sort.Slice(out.Local, func(i, j int) bool { return out.Local[i].Callsign < out.Local[j].Callsign }) + + // The master's own nodes become one trunk section (we reach it over a trunk). + if len(foreignTop) > 0 { + sort.Slice(foreignTop, func(i, j int) bool { return foreignTop[i].Callsign < foreignTop[j].Callsign }) + out.Trunks = append(out.Trunks, monTrunk{Name: masterLabel, Connected: true, Nodes: foreignTop}) + } + + for name, t := range raw.Trunks { + mt := monTrunk{Name: name, Host: t.Host, Connected: t.Connected} + for _, rn := range t.Nodes { + if mn, ok := toMonNode("", rn); ok { + mt.Nodes = append(mt.Nodes, mn) + } + } + sort.Slice(mt.Nodes, func(i, j int) bool { return mt.Nodes[i].Callsign < mt.Nodes[j].Callsign }) + out.Trunks = append(out.Trunks, mt) + } + + // Satellite trunks: one section per satellite id (KT4ROY-ASL), its AllStar + // nodes (KT4ROY-ASL1..4 via toMonNode's per-TG rename) shown under it. + for id, s := range raw.Satellites { + mt := monTrunk{Name: id, Connected: s.Authenticated} + for _, rn := range s.Nodes { + if mn, ok := toMonNode("", rn); ok { + mt.Nodes = append(mt.Nodes, mn) + } + } + sort.Slice(mt.Nodes, func(i, j int) bool { return mt.Nodes[i].Callsign < mt.Nodes[j].Callsign }) + out.Trunks = append(out.Trunks, mt) + } + // Connected trunks first, then by name — stable, tidy sections. + sort.Slice(out.Trunks, func(i, j int) bool { + if out.Trunks[i].Connected != out.Trunks[j].Connected { + return out.Trunks[i].Connected + } + return out.Trunks[i].Name < out.Trunks[j].Name + }) + // Make every node name unique across the whole view (backend, before the web). + uniquifyNames(&out) + return out, true +} + +func monitorNodesStart(mux *http.ServeMux) { + client := &http.Client{Timeout: 12 * time.Second} + + refresh := func() { + // stamp with a wall-clock time computed off the request (Date.now is + // unavailable in workflow scripts, but this is normal server code). + if next, ok := buildMonitor(client, time.Now().UTC().Format(time.RFC3339)); ok { + monState.mu.Lock() + monState.data = next + monState.mu.Unlock() + } + } + refresh() + go func() { + // Local reflector reads are free (fast); a remote master URL is throttled. + t := time.NewTicker(readInterval(monitorStatusURL)) + defer t.Stop() + for range t.C { + refresh() + } + }() + + mux.HandleFunc("/api/monitor-nodes", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json; charset=utf-8") + monState.mu.RLock() + b, _ := json.Marshal(monState.data) + monState.mu.RUnlock() + writeMaybeGzip(w, r, b) + }) +} diff --git a/svxdr/nodelist/nodelist.go b/svxdr/nodelist/nodelist.go new file mode 100644 index 0000000..889c8de --- /dev/null +++ b/svxdr/nodelist/nodelist.go @@ -0,0 +1,4315 @@ +// svxdr — live-data daemon for SvxPortal (nodelist + proxy + astmeta + xlink) +// Copyright (C) 2026 SA7AUX (Björn) +// SPDX-License-Identifier: AGPL-3.0-or-later +// +// This program is free software: you can redistribute it and/or modify it under +// the terms of the GNU Affero General Public License as published by the Free +// Software Foundation, either version 3 of the License, or (at your option) any +// later version. See the LICENSE file for details. + +// Package nodelist is the svxdr nodelist subsystem — the reflector status crawl, +// lastheard, talker-meta API, external federation and xlink, folded in from the +// standalone svx-web/nodelist. The logic is preserved verbatim; only config +// wiring changed: env-var globals are overridden by applyConfig() from +// svxdr.json, and main() became Run(ctx, cfg). See svxdrcfg.go. +package nodelist + +import ( + "bufio" + "compress/gzip" + "context" + "crypto/sha256" + "crypto/subtle" + "crypto/tls" + "encoding/hex" + "encoding/json" + "fmt" + "io" + "log" + "net" + "net/http" + "net/url" + "os" + "path/filepath" + "regexp" + "sort" + "strconv" + "strings" + "sync" + "time" + + mqtt "github.com/eclipse/paho.mqtt.golang" + "github.com/redis/go-redis/v9" + + "svxportal/svxdr/config" +) + +var ( + ctx = context.Background() + + listenAddr = env("LISTEN_ADDR", ":8098") + reflectorURL = env("REFLECTOR_STATUS_URL", "http://127.0.0.1:8090/status") + pollInterval = envDur("POLL_INTERVAL", 1*time.Second) + + redisSock = env("REDIS_SOCKET", "/run/redis/redis.sock") + redisPass = env("REDIS_PASSWORD", "") + + statusKey = env("REDIS_STATUS_KEY", "/registry/svxlink/status/json") + statusHashKey = env("REDIS_STATUS_HASH_KEY", "/registry/svxlink/status/hash") + pubsubChan = env("REDIS_PUBSUB_CHAN", "/registry/svxlink/status/update") + + nodesOnlineKey = env("REDIS_NODES_ONLINE_KEY", "/registry/svxlink/nodes/online") + nodeKeyPrefix = env("REDIS_NODE_KEY_PREFIX", "/registry/svxlink/node") + nodeConnectChan = env("REDIS_NODE_CONNECT_CHAN", "/registry/svxlink/node/connect") + nodeDisconnectChan = env("REDIS_NODE_DISCONNECT_CHAN", "/registry/svxlink/node/disconnect") + nodeUpdateChan = env("REDIS_NODE_UPDATE_CHAN", "/registry/svxlink/node/update") + + lastheardKey = env("REDIS_LASTHEARD_KEY", "/registry/svxlink/lastheard") + lastheardWindowSec = int64(24 * 60 * 60) + reflectorLogPath = env("REFLECTOR_LOG_PATH", "/var/log/svxreflector") + + nodeOnlineTTL = envDur("NODE_ONLINE_TTL", 0) + nodeTTL = envDur("NODE_TTL", 0) + + // How long a talker-meta entry (POST /api/talker-meta) stays valid + // without refresh (astmeta re-injects USRP INFO every ~10s while the + // talker is keyed, which re-fires the TCL hook that posts here). + talkerMetaTTL = envDur("TALKER_META_TTL", 30*time.Second) + + // Shared secret for POST /api/talker-meta (the svxlink TCL hook on the + // USRP bridge). Empty = endpoint disabled. + ingestToken = env("TALKER_INGEST_TOKEN", "") + + // How long a bridge-heard station stays in the /api/allstar-nodes list + // after its last keyup. + allstarNodesTTL = envDur("ALLSTAR_NODES_TTL", 30*time.Minute) + + // MQTT (reflector publishes the full picture incl. trunk-side nodes here). + mqttBroker = env("MQTT_BROKER", "") // e.g. tcp://127.0.0.1:1883 + mqttUser = env("MQTT_USERNAME", "") + mqttPass = env("MQTT_PASSWORD", "") + mqttPrefix = env("MQTT_TOPIC_PREFIX", "") // e.g. svxreflector/sa7aux + + // Cross-link talker mesh (SVX side). Publishes local SVX talkers to the mesh + // and injects DMR/AllStar talkers into the SVX lastheard. The DMR side owns + // the TG maps + translation, so this side only ever deals in its own svx_tg. + xlinkEnabled = envBool("XLINK_ENABLED", false) + xlinkBroker = env("XLINK_BROKER", "") // tcp://host:port + xlinkUser = env("XLINK_USER", "") + xlinkPass = env("XLINK_PASS", "") + xlinkTopic = env("XLINK_TOPIC", "callmgr/xlink/talker") + xlinkOrigin = env("XLINK_ORIGIN", "") // defaults to svx@ + + // External network pull: upstream aggregate of every reflector in the + // network (sources + their nodes tagged _source + all network trunks). + // The uplink host is derived from svxreflector.conf at runtime — NO address + // is baked into the binary. Scheme/port/path are env-tunable because the + // topology HTTP port varies per system/trunk; port is never assumed. + svxreflectorConf = env("SVXREFLECTOR_CONF", "/etc/svxlink/svxreflector.conf") + // chan_svx_sat config: maps each ENABLED satellite trunk's satellite_id + // (e.g. sat403872) to its friendly callsign (e.g. KT4ROY-SAT1). Disabled + // (commented) sections are absent, so only enabled sat trunks are named/shown. + svxSatConf = env("SVX_SAT_CONF", "/etc/asterisk/svx_sat.conf") + externalPullURL = env("EXTERNAL_PULL_URL", "") // optional full override; else derived from conf + externalPullScheme = env("EXTERNAL_PULL_SCHEME", "https") + externalPullPort = env("EXTERNAL_PULL_PORT", "") // topology (svx_externalpull.php): empty = scheme default (443/80) + externalPullPath = env("EXTERNAL_PULL_PATH", "/svx_externalpull.php") + externalPullInterval = envDur("EXTERNAL_PULL_INTERVAL", 60*time.Second) // gentle: spread crawl, ~1/min + + // Per-ref /status crawl. Each reflector's own /status gives its nodes AND its + // trunks, whose hosts branch out to more reflectors — so we crawl the graph. + externalStatusPorts = env("EXTERNAL_STATUS_PORTS", "") // per-server /status ports (comma array) to try beyond 443/80 — spec in /etc/default/nodelist; NO port baked in (svxref is often 8080 but owners vary, e.g. 8090) + externalStatusPath = env("EXTERNAL_STATUS_PATH", "/status") + externalProxyPath = env("EXTERNAL_PROXY_PATH", "/proxy_status.php") // dashboard's real per-ref status (nodes+trunks+trunks_UDP+routing) + externalDetailPath = env("EXTERNAL_DETAIL_PATH", "/geu/status/") // merged roster with full per-node detail (location/freq/sw), enriches route nodes + externalMaxRefs = envInt("EXTERNAL_MAX_REFS", 100) // crawl bound (cycle-safe), 0 = unlimited + externalStaleTTL = envDur("EXTERNAL_STALE_TTL", 5*time.Minute) // keep a reflector's last-good trunks this long when its probe times out — smooths flicker; 0 disables retention + + // Federation seeds: reflector /status URLs that are NOT reachable via our own + // trunk topology — they don't advertise themselves in the mesh, so the crawl + // never finds them on its own (e.g. the Swedish Reflector-network core). Seed + // them explicitly; the recursive crawl then reads each one's nodes and follows + // its trunks like any other reflector. Comma-separated full /status URLs, + // set in the site config (FEDERATION_SEEDS env / svxdr.json federation.seeds). + // Empty = no seeding. There is deliberately NO built-in list: seeds are + // site-specific runtime config and must never be hardcoded here. + federationSeeds = env("FEDERATION_SEEDS", "") // no built-in seeds: configure per site, empty = no seeding + + // Fas 2: federation gap detector. Periodically fetch the network-wide + // aggregate (sa2blv's reflector-network) and diff it against what our own + // recursive crawl covers, so we can see which reflectors/nodes we are missing + // and add them to FEDERATION_SEEDS. Read-only — never merged into the roster + // (our own fresh crawl/MQTT data stays the single source of truth). Empty URL + // disables the detector. + federationAggURL = env("FEDERATION_AGGREGATE_URL", "") // site config; empty disables the detector + federationAggInterval = envDur("FEDERATION_AGGREGATE_INTERVAL", 5*time.Minute) + + debug = envBool("DEBUG", false) +) + +// mqttState holds the latest node lists published per source on MQTT +// (/nodes/local and /nodes/). Keyed src -> callsign -> raw. +var mqttState = struct { + mu sync.RWMutex + bySrc map[string]map[string]json.RawMessage +}{bySrc: map[string]map[string]json.RawMessage{}} + +// mergeMqttNodes returns local nodes (authoritative) overlaid on every MQTT +// source's nodes, so trunk-side nodes (SE-EASTCOAST, SM-RO, …) appear too. +func mergeMqttNodes(local map[string]json.RawMessage) map[string]json.RawMessage { + merged := map[string]json.RawMessage{} + mqttState.mu.RLock() + for src, m := range mqttState.bySrc { + if src == "local" { + continue // HTTP /status is authoritative for local nodes + } + for cs, raw := range m { + merged[cs] = raw + } + } + mqttState.mu.RUnlock() + for cs, raw := range local { + merged[cs] = raw + } + return merged +} + +// onNodesMsg handles a /nodes/ message: {"nodes":[{callsign,...},...]}. +func onNodesMsg(topic string, payload []byte) { + src := topic[strings.LastIndex(topic, "/")+1:] + var p struct { + Nodes []json.RawMessage `json:"nodes"` + } + if err := json.Unmarshal(payload, &p); err != nil { + return + } + m := map[string]json.RawMessage{} + for _, raw := range p.Nodes { + var h struct { + Callsign string `json:"callsign"` + } + if json.Unmarshal(raw, &h) != nil || h.Callsign == "" { + continue + } + stored := raw + // Tag trunk-sourced nodes so the UI can tell them apart from local + // clients (only local clients can be blocked via NODE BLOCK). + if src != "local" { + var obj map[string]json.RawMessage + if json.Unmarshal(raw, &obj) == nil { + if sj, err := json.Marshal(src); err == nil { + obj["remoteTrunk"] = sj + if nb, err := json.Marshal(obj); err == nil { + stored = nb + } + } + } + } + m[h.Callsign] = stored + } + mqttState.mu.Lock() + mqttState.bySrc[src] = m + mqttState.mu.Unlock() + if debug { + log.Printf("DEBUG mqtt nodes src=%s count=%d", src, len(m)) + } +} + +func startMqtt() { + defer func() { + if r := recover(); r != nil { + log.Printf("mqtt goroutine panic: %v", r) + } + }() + + topic := strings.TrimRight(mqttPrefix, "/") + "/nodes/+" + opts := mqtt.NewClientOptions(). + AddBroker(mqttBroker). + SetClientID("nodelist"). + SetUsername(mqttUser). + SetPassword(mqttPass). + SetAutoReconnect(true). + SetConnectRetry(true). + SetConnectRetryInterval(5 * time.Second). + SetOnConnectHandler(func(c mqtt.Client) { + if t := c.Subscribe(topic, 0, func(_ mqtt.Client, msg mqtt.Message) { + onNodesMsg(msg.Topic(), msg.Payload()) + }); t.Wait() && t.Error() != nil { + log.Printf("mqtt subscribe failed: %v", t.Error()) + } else { + log.Printf("mqtt subscribed: %s", topic) + } + }). + SetConnectionLostHandler(func(_ mqtt.Client, err error) { + log.Printf("mqtt connection lost: %v", err) + }) + + c := mqtt.NewClient(opts) + if t := c.Connect(); t.Wait() && t.Error() != nil { + log.Printf("mqtt connect failed: %v (broker=%s)", t.Error(), mqttBroker) + } +} + +// ---------------------------------------------------------- cross-link mesh -- + +// xlinkMsg is one talker announcement on the cross-link topic. tg is always in +// the RECEIVER's numbering (the DMR side translates before publishing), so this +// side only ever deals in its own svx_tg. +type xlinkMsg struct { + Origin string `json:"origin"` + Network string `json:"network"` // dmr | svx | allstar (the SIDE, for routing/loop-guard) + Callsign string `json:"callsign"` + System string `json:"system"` // system/reflector name (e.g. East Coast Reflector), distinct from callsign + Trunk string `json:"trunk"` // trunk/link the talker enters through (e.g. KT4ROY-SAT1) → "callsign via trunk" + Node string `json:"node"` // source node/id (e.g. AllStar node 45225) → DMR src_id + TG int `json:"tg"` + Backend string `json:"backend"` // display backend: AllStar / SVXLink / trunk … + Event string `json:"event"` // start | end + TS int64 `json:"ts"` +} + +var xlinkClient mqtt.Client + +// xlinkStart connects the SVX side to the cross-link mesh (if enabled) and +// subscribes for peer talkers to inject into the SVX lastheard. +func xlinkStart(rdb *redis.Client) { + if !xlinkEnabled || xlinkBroker == "" { + return + } + if xlinkOrigin == "" { + h, _ := os.Hostname() + xlinkOrigin = "svx@" + h + } + opts := mqtt.NewClientOptions(). + AddBroker(xlinkBroker). + SetClientID(fmt.Sprintf("nodelist-xlink-%d", time.Now().UnixNano())). + SetUsername(xlinkUser). + SetPassword(xlinkPass). + SetAutoReconnect(true). + SetConnectRetry(true). + SetConnectRetryInterval(5 * time.Second). + SetOnConnectHandler(func(c mqtt.Client) { + if t := c.Subscribe(xlinkTopic, 0, func(_ mqtt.Client, m mqtt.Message) { + xlinkOnMessage(rdb, m.Payload()) + }); t.Wait() && t.Error() != nil { + log.Printf("xlink subscribe failed: %v", t.Error()) + } else { + log.Printf("xlink subscribed: %s", xlinkTopic) + } + }) + xlinkClient = mqtt.NewClient(opts) + xlinkClient.Connect() + log.Printf("xlink: mesh enabled broker=%s topic=%s origin=%s", xlinkBroker, xlinkTopic, xlinkOrigin) +} + +// xlinkPublishTalker announces a local SVX talker to the mesh (own numbering). +// The DMR side gates on its TG map, so publishing every talker is harmless. +func xlinkPublishTalker(callsign, system, trunk, node string, tg int, backend, event string, ts int64) { + if xlinkClient == nil || tg <= 0 { + return + } + if backend == "" { + backend = "SVXLink" + } + b, _ := json.Marshal(xlinkMsg{Origin: xlinkOrigin, Network: "svx", + Callsign: callsign, System: system, Trunk: trunk, Node: node, TG: tg, Backend: backend, Event: event, TS: ts}) + xlinkClient.Publish(xlinkTopic, 0, false, b) +} + +// xlinkOnMessage injects a peer (DMR/AllStar) talker into the SVX lastheard. +// Own-origin and svx-origin messages are ignored (loop guard); injected rows go +// straight to Redis (not the reflector log), so they are never re-published. +func xlinkOnMessage(rdb *redis.Client, payload []byte) { + var m xlinkMsg + if json.Unmarshal(payload, &m) != nil { + return + } + if m.Origin == xlinkOrigin || m.Network == "svx" || m.Network == "" { + return + } + if m.Event != "start" || m.TG <= 0 { + return // show on start; the DMR side is authoritative on gating/TG + } + ts := m.TS + if ts == 0 { + ts = time.Now().Unix() + } + backend := m.Backend + if backend == "" { + backend = "DMR" + } + writeLastheardEvent(rdb, lastheardEvent{ + Ts: ts, Callsign: m.Callsign, TG: m.TG, Duration: 1, + Source: "crosslink", Network: backend, + // Location = the TG's reflector name (e.g. 314093 -> "WHITELIGHTNING"), so a + // cross-linked DMR talker gets the same Location label as an AllStar talker + // on that TG. From nodes[].system (svx_sat.conf), via locForTG's fallback. + Location: locForTG(m.TG), + }) +} + +// extTrunkRef is one trunk as owned by a specific server: its name, the host on +// the other end, and whether it's up. Keeping ownership lets the topology graph +// draw the real server->server links instead of hanging everything on center. +type extTrunkRef struct { + Name string `json:"name"` + Host string `json:"host"` + Connected bool `json:"connected"` + Kind string `json:"kind"` // "trunk" | "udp" | "satellite" +} + +// extSource is one reflector (server) in the network: the nodes connected to it +// and the trunks it owns. IP/HTTPPort/Aliases feed the admin info box. +type extSource struct { + Tag string `json:"tag"` + URL string `json:"url"` + OK bool `json:"ok"` + Version string `json:"version,omitempty"` + Mode string `json:"mode,omitempty"` + IP string `json:"ip,omitempty"` + HTTPPort string `json:"http_port,omitempty"` + Aliases []string `json:"aliases,omitempty"` // other hostnames resolving to the same IP + Nodes map[string]json.RawMessage `json:"nodes"` + Trunks []extTrunkRef `json:"trunks"` + Routing []routeEntry `json:"-"` // this ref's Routing_table (callsign -> trunk IP), for node attribution + Detail map[string]json.RawMessage `json:"-"` // merged-roster per-node detail, to enrich route nodes +} + +// externalState holds the latest network-wide picture pulled from +// EXTERNAL_PULL_URL: sources (each with its grouped nodes, sorted by tag) and +// the raw trunks map covering the whole network. +var externalState = struct { + mu sync.RWMutex + sources []extSource // sorted by Tag + trunks json.RawMessage // raw {name: trunkobj} map + routing map[string]string // trunk IP -> callsign (from reflectors' Routing_table) + generated string + lastOKAt time.Time + lastErr string + // Network-wide bridge directory: every crawled reflector's OWN bridges + // (its /api/bridge-nodes), unioned. Lets any box show "connected to + // (node)" for a bridge that lives on another server, derived + // purely from the topology it already crawls — no per-box config. + bridgeNodes map[string]bridgeInfo // UPPER(via) -> {node, system} +}{} + +// pullSource is one reflector as listed in the topology (svx_externalpull.php). +// Its URL is the owner-published status endpoint we read directly for nodes. +type pullSource struct { + Tag string `json:"tag"` + URL string `json:"url"` + OK bool `json:"ok"` + Version string `json:"version"` + Mode string `json:"mode"` +} + +// externalPull mirrors the fields we consume from svx_externalpull.php. It is +// used purely as the TOPOLOGY (the list of reflectors + the network trunk list); +// per-ref nodes are read directly from each reflector's own /status, not taken +// from this aggregate. +type externalPull struct { + Generated string `json:"generated"` + Sources []pullSource `json:"sources"` + Trunks json.RawMessage `json:"trunks"` +} + +// uplinkHostsFromConf parses svxreflector.conf and returns the trunk-peer hosts +// that form this reflector's uplink(s): +// +// [ReflectorTrunk] Peers=SE-AUX,... -> [TrunkPeer#SE-AUX] Host= +// +// It also handles the twin2 form where each trunk is its own [TRUNK_*] section +// with the peer HOST inline. No address is baked into the binary; the uplink +// follows the config. Note the peer's Port= is the trunk protocol port +// (5500/5302), NOT the HTTP topology port, so it is deliberately ignored here. +func uplinkHostsFromConf(path string) []string { + f, err := os.Open(path) + if err != nil { + return nil + } + defer f.Close() + + var peers []string + hostByPeer := map[string]string{} + var directHosts []string // twin2 [TRUNK_*] sections carry the peer host inline + curPeer := "" // current [TrunkPeer#] section + inTrunk := false // inside [ReflectorTrunk] + inTrunkSec := false // inside a twin2 [TRUNK_*] section + + sc := bufio.NewScanner(f) + for sc.Scan() { + line := strings.TrimSpace(sc.Text()) + if line == "" || strings.HasPrefix(line, "#") { + continue + } + if strings.HasPrefix(line, "[") && strings.HasSuffix(line, "]") { + sec := line[1 : len(line)-1] + inTrunk = strings.EqualFold(sec, "ReflectorTrunk") + inTrunkSec = strings.HasPrefix(strings.ToUpper(sec), "TRUNK_") + curPeer = "" + if i := strings.IndexByte(sec, '#'); i >= 0 && strings.EqualFold(sec[:i], "TrunkPeer") { + curPeer = sec[i+1:] + } + continue + } + eq := strings.IndexByte(line, '=') + if eq < 0 { + continue + } + key := strings.TrimSpace(line[:eq]) + val := strings.TrimSpace(line[eq+1:]) + switch { + case inTrunk && strings.EqualFold(key, "Peers"): + for _, p := range strings.Split(val, ",") { + if p = strings.TrimSpace(p); p != "" { + peers = append(peers, p) + } + } + case curPeer != "" && strings.EqualFold(key, "Host"): + hostByPeer[curPeer] = val + case inTrunkSec && strings.EqualFold(key, "Host"): + directHosts = append(directHosts, val) + } + } + + var hosts []string + seen := map[string]struct{}{} + add := func(h string) { + if h == "" { + return + } + if _, ok := seen[h]; ok { + return + } + seen[h] = struct{}{} + hosts = append(hosts, h) + } + for _, p := range peers { + add(hostByPeer[p]) + } + for _, h := range directHosts { + add(h) + } + return hosts +} + +// satTGsFromConf parses chan_svx_sat's config (svx_sat.conf) and groups every +// ENABLED entry by its satellite_id, returning the talkgroups (per-TG callsign + +// tg) each satellite carries. Several entries sharing a satellite_id are the SAME +// trunk with several TGs, so they are grouped — not collapsed to the last callsign. +func satTGsFromConf(path string) map[string][]satTG { + out := map[string][]satTG{} + f, err := os.Open(path) + if err != nil { + return out + } + defer f.Close() + + var satID, call, tg string + flush := func() { + if satID != "" { + out[satID] = append(out[satID], satTG{Name: call, TG: tg}) + } + satID, call, tg = "", "", "" + } + sc := bufio.NewScanner(f) + for sc.Scan() { + line := strings.TrimSpace(sc.Text()) + if line == "" || strings.HasPrefix(line, ";") || strings.HasPrefix(line, "#") { + continue + } + if strings.HasPrefix(line, "[") && strings.HasSuffix(line, "]") { + flush() + continue + } + eq := strings.IndexByte(line, '=') + if eq < 0 { + continue + } + switch strings.ToLower(strings.TrimSpace(line[:eq])) { + case "satellite_id": + satID = strings.TrimSpace(line[eq+1:]) + case "callsign": + call = strings.TrimSpace(line[eq+1:]) + case "tg": + tg = strings.TrimSpace(line[eq+1:]) + } + } + flush() + return out +} + +// satelliteIDsFromReflectorConf parses svxreflector.conf's [SATELLITE] section and +// returns the configured satellite ids (the in each SECRET_=… line). Only +// the id is read — the secret value is never touched or exposed. Lets the Trunks +// view list every configured satellite (online or offline) fully dynamically. +func satelliteIDsFromReflectorConf(path string) []string { + f, err := os.Open(path) + if err != nil { + return nil + } + defer f.Close() + + var ids []string + seen := map[string]bool{} + inSat := false + sc := bufio.NewScanner(f) + for sc.Scan() { + line := strings.TrimSpace(sc.Text()) + if line == "" || strings.HasPrefix(line, "#") || strings.HasPrefix(line, ";") { + continue + } + if strings.HasPrefix(line, "[") && strings.HasSuffix(line, "]") { + inSat = strings.EqualFold(line, "[SATELLITE]") + continue + } + if !inSat { + continue + } + eq := strings.IndexByte(line, '=') + if eq < 0 { + continue + } + key := strings.TrimSpace(line[:eq]) + if len(key) > 7 && strings.EqualFold(key[:7], "SECRET_") { + if id := key[7:]; id != "" && !seen[id] { + seen[id] = true + ids = append(ids, id) + } + } + } + return ids +} + +// satNamesFromConf parses chan_svx_sat's config (svx_sat.conf) and returns a map +// from each ENABLED satellite trunk's satellite_id (e.g. "sat403872") to its +// friendly callsign (e.g. "KT4ROY-SAT1"). Commented-out (disabled) sections and +// keys are skipped, so only enabled sat trunks are surfaced. Empty if unreadable. +func satNamesFromConf(path string) map[string]string { + out := map[string]string{} + f, err := os.Open(path) + if err != nil { + return out + } + defer f.Close() + + var satID, call string + flush := func() { + if satID != "" && call != "" { + out[satID] = call + } + satID, call = "", "" + } + sc := bufio.NewScanner(f) + for sc.Scan() { + line := strings.TrimSpace(sc.Text()) + if line == "" || strings.HasPrefix(line, ";") || strings.HasPrefix(line, "#") { + continue // blank or disabled/commented → skip + } + if strings.HasPrefix(line, "[") && strings.HasSuffix(line, "]") { + flush() // section boundary — commit the previous section + continue + } + eq := strings.IndexByte(line, '=') + if eq < 0 { + continue + } + switch strings.ToLower(strings.TrimSpace(line[:eq])) { + case "satellite_id": + satID = strings.TrimSpace(line[eq+1:]) + case "callsign": + call = strings.TrimSpace(line[eq+1:]) + } + } + flush() + return out +} + +// externalPullURLs returns the topology URL(s) to fetch. EXTERNAL_PULL_URL wins +// if set; otherwise one URL per uplink host derived from the conf. Neither host +// nor port is ever baked in — port comes from EXTERNAL_PULL_PORT (empty = scheme +// default) because it varies per system/trunk. +func externalPullURLs() []string { + if externalPullURL != "" { + return []string{externalPullURL} + } + hosts := uplinkHostsFromConf(svxreflectorConf) + scheme := externalPullScheme + if scheme == "" { + scheme = "https" + } + path := externalPullPath + if !strings.HasPrefix(path, "/") { + path = "/" + path + } + urls := make([]string, 0, len(hosts)) + for _, h := range hosts { + host := h + if externalPullPort != "" { + host = net.JoinHostPort(h, externalPullPort) + } + urls = append(urls, fmt.Sprintf("%s://%s%s", scheme, host, path)) + } + return urls +} + +// fetchExternalPull GETs and parses one topology endpoint (svx_externalpull.php). +func fetchExternalPull(client *http.Client, u string) (externalPull, error) { + var p externalPull + req, err := http.NewRequest(http.MethodGet, u, nil) + if err != nil { + return p, err + } + req.Header.Set("Accept", "application/json") + resp, err := client.Do(req) + if err != nil { + return p, err + } + defer resp.Body.Close() + if resp.StatusCode != 200 { + _, _ = io.Copy(io.Discard, resp.Body) + return p, fmt.Errorf("http %d", resp.StatusCode) + } + body, err := io.ReadAll(resp.Body) + if err != nil { + return p, err + } + if err := json.Unmarshal(body, &p); err != nil { + return p, fmt.Errorf("parse: %w", err) + } + return p, nil +} + +// hostFromURL extracts the host (no port) from a URL, "" on failure. +func hostFromURL(raw string) string { + if raw == "" { + return "" + } + u, err := url.Parse(raw) + if err != nil { + return "" + } + return u.Hostname() +} + +// normalizeHost lowercases and strips scheme/port for dedupe keying. +func normalizeHost(h string) string { + h = strings.TrimSpace(strings.ToLower(h)) + if h == "" { + return "" + } + if strings.Contains(h, "://") { + if hh := hostFromURL(h); hh != "" { + return strings.ToLower(hh) + } + } + if host, _, err := net.SplitHostPort(h); err == nil { + return host + } + return h +} + +// resolveCache memoises hostname -> primary IP so the same host isn't looked up +// twice within a crawl. +var ( + resolveMu sync.Mutex + resolveCache = map[string]string{} +) + +// refCacheEntry is one reflector's last SUCCESSFUL crawl contribution, retained so +// a single timed-out probe (or a briefly-unreachable subtree) doesn't erase its +// trunks. Keyed by hostKey — the same dedup key the crawl uses for `visited`. +type refCacheEntry struct { + src extSource // the reflector's extSource incl. its owned trunk refs + trunks map[string]json.RawMessage // raw trunk/udp/satellite blobs it contributed (name -> raw) + routing []routeEntry // its Routing_table entries (trunk IP -> callsign) + at time.Time // when this reflector was last read OK +} + +// refCache holds last-known-good per reflector so crawlNetwork can resurrect a +// reflector whose probe failed this cycle, until it's been unreachable for +// externalStaleTTL. Touched only in the single-threaded fold, never by probe +// goroutines; the mutex is belt-and-suspenders. +var ( + refCacheMu sync.Mutex + refCache = map[string]refCacheEntry{} +) + +// putRefCache stores a reflector's just-read good contribution, stamped now. +func putRefCache(key string, e refCacheEntry) { + e.at = time.Now() + refCacheMu.Lock() + refCache[key] = e + refCacheMu.Unlock() +} + +// freshRefCache returns a reflector's retained contribution if present and read +// within externalStaleTTL; stale entries are pruned so the map stays bounded. +func freshRefCache(key string) (refCacheEntry, bool) { + if key == "" || externalStaleTTL <= 0 { + return refCacheEntry{}, false + } + refCacheMu.Lock() + defer refCacheMu.Unlock() + e, ok := refCache[key] + if !ok { + return refCacheEntry{}, false + } + if time.Since(e.at) > externalStaleTTL { + delete(refCache, key) + return refCacheEntry{}, false + } + return e, true +} + +// hostKey returns a dedup key for a host: its resolved IP when possible (so two +// hostnames for the same physical reflector — e.g. sm2ampr.net and +// reflector-network.se both 44.5.25.40 — collapse to one), else the bare host. +func hostKey(host string) string { + h := normalizeHost(host) + if h == "" { + return "" + } + resolveMu.Lock() + if v, ok := resolveCache[h]; ok { + resolveMu.Unlock() + return v + } + resolveMu.Unlock() + + key := h + if ips, err := net.LookupHost(h); err == nil && len(ips) > 0 { + sort.Strings(ips) + key = ips[0] + } + resolveMu.Lock() + resolveCache[h] = key + resolveMu.Unlock() + return key +} + +// isIPStr reports whether h is a bare IP literal. +func isIPStr(h string) bool { + return net.ParseIP(strings.Trim(h, "[]")) != nil +} + +// trunkHost reads the "host" field from a trunk status object. +func trunkHost(raw json.RawMessage) string { + var t struct { + Host string `json:"host"` + } + if json.Unmarshal(raw, &t) != nil { + return "" + } + return strings.TrimSpace(t.Host) +} + +// trunkConnected reads the "connected" flag from a trunk object. +func trunkConnected(raw json.RawMessage) bool { + var t struct { + Connected bool `json:"connected"` + } + _ = json.Unmarshal(raw, &t) + return t.Connected +} + +func ensureSlash(p string) string { + if !strings.HasPrefix(p, "/") { + return "/" + p + } + return p +} + +// statusCandidates returns the ordered status URLs to try for a reflector. The +// dashboard's proxy_status.php is tried FIRST — it exposes the real, full status +// (nodes + trunks + trunks_UDP + Routing_table); a raw /status can be empty on +// hub reflectors. The HTTP port varies per owner and isn't published, so we try +// 443 (https) and 80 (http) — common behind a proxy — plus each configured extra +// port (EXTERNAL_STATUS_PORTS, spec'd per server — no port assumed), with proxy + /status. +func statusCandidates(host, publishedURL string, ports []string) []string { + h := host + if h == "" { + h = hostFromURL(publishedURL) + } + proxy := ensureSlash(externalProxyPath) + stat := ensureSlash(externalStatusPath) + + var c []string + if h != "" { + c = append(c, "https://"+h+proxy) // proxy_status.php over 443 first + } + if publishedURL != "" { + c = append(c, publishedURL) + } + if h != "" { + // 443 (https) and 80 (http) — common behind nginx/apache + c = append(c, "https://"+h+stat, "http://"+h+proxy, "http://"+h+stat) + // ports learned from the network (each svxref's http_port) + configured seeds + for _, p := range ports { + p = strings.TrimSpace(p) + if p == "" { + continue + } + hp := net.JoinHostPort(h, p) + c = append(c, "http://"+hp+proxy, "http://"+hp+stat) + } + } + return c +} + +// refStatus is one reflector's own status: its nodes, its named trunks, its UDP +// trunk peers, and the HTTP port it reports for itself (http_port) — the crawl +// learns real ports from the network instead of guessing. +type routeEntry struct { + Callsign string + Trunk string // the trunk IP this callsign is reachable via + TG int + MonitoredTGs []int +} + +type refStatus struct { + Nodes map[string]json.RawMessage + Trunks map[string]json.RawMessage + TrunksUDP []json.RawMessage + Satellites map[string]json.RawMessage // id -> full satellite obj (incl. its nodes[]) + HTTPPort string + Version string + Mode string + Routing []routeEntry + Detail map[string]json.RawMessage // full per-node detail from the merged roster (enriches route nodes) +} + +// jsonToPort reads http_port whether emitted as a string ("8085") or a number. +func jsonToPort(raw json.RawMessage) string { + var s string + if json.Unmarshal(raw, &s) == nil { + return strings.TrimSpace(s) + } + var n json.Number + if json.Unmarshal(raw, &n) == nil { + return n.String() + } + return "" +} + +// parseNodes accepts a reflector's "nodes" either as an object keyed by callsign +// or as an array of node objects (different reflectors/versions emit both), and +// returns a callsign-keyed map. +func parseNodes(raw json.RawMessage) map[string]json.RawMessage { + var m map[string]json.RawMessage + if json.Unmarshal(raw, &m) == nil { + return m + } + var arr []json.RawMessage + if json.Unmarshal(raw, &arr) == nil { + out := make(map[string]json.RawMessage, len(arr)) + for _, r := range arr { + var h struct { + Callsign string `json:"callsign"` + } + if json.Unmarshal(r, &h) == nil && h.Callsign != "" { + out[h.Callsign] = r + } + } + return out + } + return map[string]json.RawMessage{} +} + +// parseRefStatus parses a /status body into nodes + trunks. Requires a top-level +// "nodes" key so a stray 200 (HTML/redirect page) is rejected. +func parseRefStatus(body []byte) (refStatus, bool) { + var top map[string]json.RawMessage + if json.Unmarshal(body, &top) != nil { + return refStatus{}, false + } + if _, ok := top["nodes"]; !ok { + return refStatus{}, false + } + rs := refStatus{Nodes: parseNodes(top["nodes"]), Trunks: map[string]json.RawMessage{}} + if raw, ok := top["trunks"]; ok { + var tr map[string]json.RawMessage + if json.Unmarshal(raw, &tr) == nil { + rs.Trunks = tr + } + } + if raw, ok := top["trunks_UDP"]; ok { + var arr []json.RawMessage + if json.Unmarshal(raw, &arr) == nil { + rs.TrunksUDP = arr + } + } + if raw, ok := top["satellites"]; ok { + var m map[string]json.RawMessage + if json.Unmarshal(raw, &m) == nil { + rs.Satellites = m + } + } + if raw, ok := top["http_port"]; ok { + rs.HTTPPort = jsonToPort(raw) + } + if raw, ok := top["version"]; ok { + _ = json.Unmarshal(raw, &rs.Version) + } + if raw, ok := top["mode"]; ok { + _ = json.Unmarshal(raw, &rs.Mode) + } + if raw, ok := top["Routing_table"]; ok { + var arr []struct { + Callsign string `json:"callsign"` + Trunk string `json:"trunk"` + TG int `json:"tg"` + TGMonitor []int `json:"tg_monitor"` + } + if json.Unmarshal(raw, &arr) == nil { + for _, e := range arr { + if e.Callsign != "" && e.Trunk != "" { + rs.Routing = append(rs.Routing, routeEntry{ + Callsign: e.Callsign, Trunk: strings.TrimSpace(e.Trunk), + TG: e.TG, MonitoredTGs: e.TGMonitor, + }) + } + } + } + } + return rs, true +} + +// normalizeUDPTrunk turns a trunks_UDP entry into a uniform trunk object: it +// maps status->connected so the same renderer works for named + UDP trunks, and +// returns the peer name and host for dedup/branching. +func normalizeUDPTrunk(raw json.RawMessage) (name string, out json.RawMessage, host string) { + var m map[string]json.RawMessage + if json.Unmarshal(raw, &m) != nil { + return "", raw, "" + } + if b, ok := m["name"]; ok { + _ = json.Unmarshal(b, &name) + } + if b, ok := m["host"]; ok { + _ = json.Unmarshal(b, &host) + } + if st, ok := m["status"]; ok { + if _, has := m["connected"]; !has { + m["connected"] = st + } + } + if b, err := json.Marshal(m); err == nil { + out = b + } else { + out = raw + } + return name, out, strings.TrimSpace(host) +} + +// readRefStatus tries each candidate URL until one returns a parseable /status. +// fetchDetailRoster fetches the reflector's MERGED node roster (full per-node +// detail: location, freqs, sw, sysop) from externalDetailPath derived off the +// working status URL (same scheme+host, different path). proxy_status.php only +// lists a UDP-peer member's callsign in Routing_table, so this fills in detail. +// Best-effort; returns nil on any failure (route nodes then stay sparse). +func fetchDetailRoster(client *http.Client, usedURL string) map[string]json.RawMessage { + if externalDetailPath == "" || usedURL == "" { + return nil + } + u, err := url.Parse(usedURL) + if err != nil { + return nil + } + u.Path = ensureSlash(externalDetailPath) + u.RawQuery = "" + req, err := http.NewRequest(http.MethodGet, u.String(), nil) + if err != nil { + return nil + } + req.Header.Set("Accept", "application/json") + resp, err := client.Do(req) + if err != nil { + return nil + } + body, rerr := io.ReadAll(io.LimitReader(resp.Body, 16<<20)) + resp.Body.Close() + if rerr != nil || resp.StatusCode != 200 { + return nil + } + var top map[string]json.RawMessage + if json.Unmarshal(body, &top) != nil { + return nil + } + nm := parseNodes(top["nodes"]) + if len(nm) == 0 { + return nil + } + return nm +} + +func readRefStatus(client *http.Client, host, publishedURL string, ports []string) (rs refStatus, usedURL string, ok bool) { + for _, u := range statusCandidates(host, publishedURL, ports) { + req, err := http.NewRequest(http.MethodGet, u, nil) + if err != nil { + continue + } + req.Header.Set("Accept", "application/json") + resp, err := client.Do(req) + if err != nil { + continue + } + body, rerr := io.ReadAll(io.LimitReader(resp.Body, 16<<20)) + resp.Body.Close() + if rerr != nil || resp.StatusCode != 200 { + continue + } + if r, parsed := parseRefStatus(body); parsed { + // A routing table means members reachable via UDP peers whose detail the + // primary endpoint omits — pull the merged roster to enrich them. + if len(r.Routing) > 0 { + if d := fetchDetailRoster(client, u); d != nil { + r.Detail = d + } + } + return r, u, true + } + } + return refStatus{Nodes: map[string]json.RawMessage{}}, "", false +} + +type crawlSeed struct{ tag, host, url string } + +// crawlNetwork discovers reflectors from the uplink topology, then reads each +// reflector's own proxy_status.php for its nodes AND its trunks, following those +// trunks' hosts out to further reflectors. Breadth-first by level with bounded +// concurrency; cycle-safe (visited hosts), bounded (EXTERNAL_MAX_REFS, depth), +// best-effort (unreachable refs listed OK=false, bare-IP trunk hosts skipped). +func crawlNetwork(client *http.Client) (sources []extSource, trunks map[string]json.RawMessage, routing map[string]string, generated string, err error) { + topoURLs := externalPullURLs() + + trunks = map[string]json.RawMessage{} + routing = map[string]string{} + visited := map[string]struct{}{} + // Enabled satellite trunks, satellite_id -> friendly callsign (KT4ROY-SAT1), + // read from this box's chan_svx_sat config. Used to show the sat trunk under + // its real name instead of the raw reflector id (sat403872). + satNames := satNamesFromConf(svxSatConf) + hostsByKey := map[string]map[string]struct{}{} // IP key -> hostnames seen for it + var frontier []crawlSeed + lastErr := "" + topoOK := false + + recordHost := func(h string) { + h = normalizeHost(h) + if h == "" || isIPStr(h) { + return + } + k := hostKey(h) + if k == "" { + return + } + if hostsByKey[k] == nil { + hostsByKey[k] = map[string]struct{}{} + } + hostsByKey[k][h] = struct{}{} + } + addTrunk := func(name string, raw json.RawMessage) { + if name == "" { + return + } + if _, dup := trunks[name]; !dup { + trunks[name] = raw + } + } + enqueue := func(next *[]crawlSeed, s crawlSeed) { + recordHost(s.host) + recordHost(hostFromURL(s.url)) + key := hostKey(s.host) + if key == "" { + key = hostKey(hostFromURL(s.url)) + } + if key == "" { + return + } + if _, seen := visited[key]; seen { + return + } + *next = append(*next, s) + } + // replayCached folds a reflector's retained last-good contribution into this + // cycle's result set (routing, raw trunks, source) and re-seeds its trunk hosts + // so the subtree it fronts is still followed. Used when its probe timed out. + replayCached := func(e refCacheEntry, next *[]crawlSeed) { + for _, en := range e.routing { + if _, ok := routing[en.Trunk]; !ok { + routing[en.Trunk] = en.Callsign + } + } + for name, raw := range e.trunks { + addTrunk(name, raw) + } + for _, t := range e.src.Trunks { + if t.Host != "" { + enqueue(next, crawlSeed{tag: t.Host, host: t.Host}) + } + } + sources = append(sources, e.src) + } + + // Seed from the local reflector's OWN live status: every link it reports — + // TCP trunks and UDP trunk peers alike — carries the peer host in runtime + // data. No trunk mechanism is derived from config-format knowledge here; the + // reflector itself is the source of truth for what it is linked to. + if body, _, ferr := fetchReflectorStatus(reflectorURL); ferr == nil { + if rs, ok := parseRefStatus(body); ok { + for _, raw := range rs.Trunks { + if h := trunkHost(raw); h != "" { + enqueue(&frontier, crawlSeed{tag: h, host: h}) + } + } + for _, raw := range rs.TrunksUDP { + if _, _, h := normalizeUDPTrunk(raw); h != "" { + enqueue(&frontier, crawlSeed{tag: h, host: h}) + } + } + } + } + + // Seed the crawl with the configured federation reflectors (the Swedish core + // etc). They are not in our trunk topology, so without this the crawl never + // reaches them; once seeded, the recursive crawl reads each one's /status + // (its nodes) and follows its trunks like any other reflector. + for _, su := range strings.Split(federationSeeds, ",") { + if su = strings.TrimSpace(su); su != "" { + enqueue(&frontier, crawlSeed{tag: su, host: hostFromURL(su), url: su}) + } + } + + if len(topoURLs) == 0 && len(frontier) == 0 { + return nil, nil, nil, "", fmt.Errorf("no uplink derived from %s or %s", svxreflectorConf, reflectorURL) + } + + // Seed from the topology aggregate: the uplink host itself, its sources + // (owner-published url), and its trunks (recorded + branched). + for _, tu := range topoURLs { + p, e := fetchExternalPull(client, tu) + if e != nil { + lastErr = e.Error() + continue + } + topoOK = true + if p.Generated != "" { + generated = p.Generated + } + if uh := hostFromURL(tu); uh != "" { + enqueue(&frontier, crawlSeed{tag: uh, host: uh}) + } + for _, s := range p.Sources { + enqueue(&frontier, crawlSeed{tag: s.Tag, host: hostFromURL(s.URL), url: s.URL}) + } + var tr map[string]json.RawMessage + if len(p.Trunks) > 0 && json.Unmarshal(p.Trunks, &tr) == nil { + for name, raw := range tr { + addTrunk(name, raw) + if h := trunkHost(raw); h != "" { + enqueue(&frontier, crawlSeed{tag: h, host: h}) + } + } + } + } + if !topoOK && len(frontier) == 0 { + return nil, nil, nil, "", fmt.Errorf("topology unreachable: %s", lastErr) + } + + // Ports to try per host: seeded from the per-server config array (empty by + // default — nothing baked in), plus our own reflector's port, then grown with + // every real http_port a reached svxref reports (data-driven, no assumption). + learnedPorts := map[string]struct{}{} + for _, p := range strings.Split(externalStatusPorts, ",") { + if p = strings.TrimSpace(p); p != "" { + learnedPorts[p] = struct{}{} + } + } + // The port our OWN reflector serves /status on (from REFLECTOR_STATUS_URL) + // is a natural candidate for peers running the same software. + if u, uerr := url.Parse(reflectorURL); uerr == nil { + if p := u.Port(); p != "" { + learnedPorts[p] = struct{}{} + } + } + + const maxDepth = 4 + for depth := 0; depth < maxDepth && len(frontier) > 0; depth++ { + ports := make([]string, 0, len(learnedPorts)) + for p := range learnedPorts { + ports = append(ports, p) + } + sort.Strings(ports) + // Claim this level's hosts (dedup vs visited, respect the ref cap). + var batch []crawlSeed + for _, s := range frontier { + key := hostKey(s.host) + if key == "" { + key = hostKey(hostFromURL(s.url)) + } + if key == "" { + continue + } + if _, seen := visited[key]; seen { + continue + } + if externalMaxRefs > 0 && len(visited) >= externalMaxRefs { + log.Printf("external crawl: hit EXTERNAL_MAX_REFS=%d — some refs not read", externalMaxRefs) + break + } + visited[key] = struct{}{} + batch = append(batch, s) + } + if len(batch) == 0 { + break + } + + // Read the whole level concurrently (bounded). + type readResult struct { + seed crawlSeed + src extSource + rs refStatus + } + results := make([]readResult, len(batch)) + sem := make(chan struct{}, 10) + var wg sync.WaitGroup + for i, s := range batch { + wg.Add(1) + go func(i int, s crawlSeed) { + defer wg.Done() + sem <- struct{}{} + defer func() { <-sem }() + rs, usedURL, ok := readRefStatus(client, s.host, s.url, ports) + results[i] = readResult{ + seed: s, + src: extSource{Tag: s.tag, URL: usedURL, OK: ok, Version: rs.Version, Mode: rs.Mode, Nodes: rs.Nodes, Routing: rs.Routing, Detail: rs.Detail}, + rs: rs, + } + }(i, s) + } + wg.Wait() + + // Fold results single-threaded: record trunks (global + per owning server), + // then branch to the next level. + var next []crawlSeed + for _, r := range results { + // Learn this reflector's real HTTP port for the next levels. + if r.rs.HTTPPort != "" { + learnedPorts[r.rs.HTTPPort] = struct{}{} + } + key := hostKey(r.seed.host) + if key == "" { + key = hostKey(hostFromURL(r.seed.url)) + } + // Probe failed this cycle: rather than dropping this reflector (and all + // its trunks) — the flicker cause — fall back to its retained last-good + // contribution if still fresh, and keep branching from it. + if !r.src.OK { + if e, ok := freshRefCache(key); ok { + replayCached(e, &next) + continue + } + // no fresh cache — record the empty source as before + } + // Accumulate the routing table (trunk IP -> callsign) for self/identity tagging. + for _, e := range r.rs.Routing { + if _, ok := routing[e.Trunk]; !ok { + routing[e.Trunk] = e.Callsign + } + } + src := r.src + src.HTTPPort = r.rs.HTTPPort + src.IP = hostKey(r.seed.host) + if src.IP == "" { + src.IP = hostKey(hostFromURL(r.seed.url)) + } + if !isIPStr(src.IP) { + src.IP = "" // unresolved host — no IP to show + } + owned := []extTrunkRef{} // never nil → never serialized as JSON null + contrib := map[string]json.RawMessage{} // this ref's own raw blobs, kept for retention + for name, raw := range r.rs.Trunks { + addTrunk(name, raw) + contrib[name] = raw + h := trunkHost(raw) + owned = append(owned, extTrunkRef{Name: name, Host: h, Connected: trunkConnected(raw), Kind: "trunk"}) + if h != "" { + enqueue(&next, crawlSeed{tag: h, host: h}) + } + } + for _, raw := range r.rs.TrunksUDP { + name, norm, h := normalizeUDPTrunk(raw) + if name == "" { + continue + } + addTrunk(name, norm) + contrib[name] = norm + owned = append(owned, extTrunkRef{Name: name, Host: h, Connected: trunkConnected(norm), Kind: "udp"}) + if h != "" { + enqueue(&next, crawlSeed{tag: h, host: h}) + } + } + for id, sraw := range r.rs.Satellites { + var meta struct { + Authenticated bool `json:"authenticated"` + } + _ = json.Unmarshal(sraw, &meta) + // Show the ENABLED sat trunk under its real callsign (KT4ROY-SAT1) + // from this box's chan_svx_sat config, not the raw reflector id + // (sat403872). Not in the config → keep the raw id as a fallback. + name := satNames[id] + if name == "" { + name = id + } + addTrunk(name, sraw) // so the flat list captures the satellite's nodes[] + contrib[name] = sraw + owned = append(owned, extTrunkRef{Name: name, Connected: meta.Authenticated, Kind: "satellite"}) + } + src.Trunks = owned + sources = append(sources, src) + // Retain this good read so a future timed-out cycle can resurrect it. + if r.src.OK && key != "" && externalStaleTTL > 0 { + putRefCache(key, refCacheEntry{src: src, trunks: contrib, routing: r.rs.Routing}) + } + } + frontier = next + } + + // Resurrection pass: reflectors reachable recently but not crawled AT ALL this + // cycle (their only path in went through a reflector that timed out, so they + // were never enqueued) keep their retained last-good trunks until the staleness + // window expires. Reflectors already handled above are in `visited` → skipped, + // so nothing is duplicated. + if externalStaleTTL > 0 { + refCacheMu.Lock() + var revive []refCacheEntry + for k, e := range refCache { + if time.Since(e.at) > externalStaleTTL { + delete(refCache, k) // prune so the cache stays bounded + continue + } + if _, seen := visited[k]; seen { + continue + } + revive = append(revive, e) + } + refCacheMu.Unlock() + var discard []crawlSeed // last-known picture — do not branch further + for _, e := range revive { + replayCached(e, &discard) + } + } + + // Attach alias hostnames: every other hostname that resolves to this ref's IP + // (e.g. reflector-network.se also answers as sm2ampr.net). Done after the crawl + // so all hostnames have been recorded. + for i := range sources { + key := sources[i].IP + if key == "" { + key = hostKey(sources[i].Tag) + } + primary := normalizeHost(sources[i].Tag) + for h := range hostsByKey[key] { + if h != primary { + sources[i].Aliases = append(sources[i].Aliases, h) + } + } + sort.Strings(sources[i].Aliases) + } + + // Keep ALL discovered reflectors (online + firewalled/offline). The offline + // ones must stay so Topology can show them; Monitor Nodes filters to online + // in the frontend. + sort.Slice(sources, func(i, j int) bool { return sources[i].Tag < sources[j].Tag }) + return sources, trunks, routing, generated, nil +} + +// bridgeHTTP fetches other reflectors' /api/bridge-nodes during the crawl. Skips +// TLS verification: this is a best-effort read of a public bridge directory +// across many independently-operated servers (varied/self-signed certs). +var bridgeHTTP = &http.Client{ + Timeout: 2500 * time.Millisecond, + Transport: &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}, +} + +// crawlBridgeNodes fetches each crawled reflector's OWN /api/bridge-nodes and +// unions them into a network-wide bridge directory. A host that doesn't run +// svxportal (no endpoint / 404 / timeout) is simply skipped. Every host comes +// from the topology the crawl already produced — nothing is hard-coded. +func crawlBridgeNodes(sources []extSource) map[string]bridgeInfo { + out := map[string]bridgeInfo{} + seen := map[string]bool{} + var wg sync.WaitGroup + var mu sync.Mutex + for _, s := range sources { + host := hostFromURL(s.URL) // status URL is often empty here... + if host == "" { + host = s.IP // ...so fall back to the reflector IP (topology-derived) + } + if host == "" || seen[host] { + continue + } + seen[host] = true + hp := host + if strings.Contains(host, ":") { + hp = "[" + host + "]" // bracket IPv6 literals + } + wg.Add(1) + go func(hp string) { // parallel: whole crawl bounded by one client timeout + defer wg.Done() + resp, err := bridgeHTTP.Get("https://" + hp + "/svxportal/api/bridge-nodes") + if err != nil { + return + } + body, _ := io.ReadAll(io.LimitReader(resp.Body, 256<<10)) + resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return + } + var m map[string]bridgeInfo + if json.Unmarshal(body, &m) != nil { + return + } + mu.Lock() + for via, bi := range m { + if bi.Node != "" { + out[strings.ToUpper(via)] = bi + } + } + mu.Unlock() + }(hp) + } + wg.Wait() + return out +} + +// externalPuller periodically crawls the network (topology → per-ref /status → +// follow trunks) and stores the result in externalState. Runs as its own +// goroutine, off the status-serving hot path. Re-derives the uplink from the +// conf each cycle so a peer change is followed without a restart. +func externalPuller() { + defer func() { + if r := recover(); r != nil { + log.Printf("externalPuller panic: %v", r) + } + }() + + client := &http.Client{Timeout: 2500 * time.Millisecond} // per request; crawl is parallel + best-effort, off the hot path. Higher = fewer transient timeouts (last-good retention covers the rest) + loggedNoUplink := false + + pullOnce := func() { + if len(externalPullURLs()) == 0 { + if !loggedNoUplink { + log.Printf("external crawl: no uplink host in %s and no EXTERNAL_PULL_URL set; disabled", svxreflectorConf) + loggedNoUplink = true + } + return + } + loggedNoUplink = false + + sources, trunks, routing, generated, err := crawlNetwork(client) + if err != nil { + setExternalErr(err.Error()) + return + } + + trunksRaw, _ := json.Marshal(trunks) + totalNodes := 0 + for _, s := range sources { + totalNodes += len(s.Nodes) + } + + // Piggyback on the crawl: fetch each discovered reflector's OWN bridge + // directory and union it. Hosts come from the topology → nothing is + // hard-coded. Best-effort; keep last-good on an empty round. + bridges := crawlBridgeNodes(sources) + + externalState.mu.Lock() + externalState.sources = sources + externalState.trunks = trunksRaw + externalState.routing = routing + externalState.generated = generated + externalState.lastOKAt = time.Now() + externalState.lastErr = "" + if len(bridges) > 0 { + externalState.bridgeNodes = bridges + } + externalState.mu.Unlock() + + if debug { + log.Printf("DEBUG external crawl: refs=%d nodes=%d trunks=%d", len(sources), totalNodes, len(trunks)) + } + } + + pullOnce() + t := time.NewTicker(externalPullInterval) + defer t.Stop() + for range t.C { + pullOnce() + } +} + +func setExternalErr(msg string) { + externalState.mu.Lock() + externalState.lastErr = msg + externalState.mu.Unlock() + if debug { + log.Printf("DEBUG external pull failed: %s (url=%s)", msg, externalPullURL) + } +} + +// tglistPuller keeps the canonical TG->system map fresh from the shared tglist +// (svx-web /tglist.php). Mirrors externalPuller: fetch on a gentle ticker, keep +// last-good on failure, cache to disk so names survive a restart before the +// first successful fetch. URL empty = feature off — the per-box nodes[].system +// stays the only source, so nothing changes until a box opts in via config. +func tglistPuller(ctx context.Context, c *config.Config) { + defer func() { + if r := recover(); r != nil { + log.Printf("tglistPuller panic: %v", r) + } + }() + + file := strings.TrimSpace(c.Tglist.File) + url := strings.TrimSpace(c.Tglist.URL) + if file == "" && url == "" { + log.Printf("tglist: no file/url set; Location uses per-box nodes[].system only") + return + } + cacheFile := strings.TrimSpace(c.Tglist.CacheFile) + if cacheFile == "" { + cacheFile = "/var/lib/svxdr/tglist.json" + } + interval := c.Tglist.RefreshInterval.D() + if interval <= 0 { + interval = 15 * time.Minute + } + client := &http.Client{Timeout: 4 * time.Second} + + // Parse a /tglist.php body into the TG->system map (system, else name). + apply := func(body []byte) int { + var doc struct { + TGs []struct { + TG int `json:"tg"` + Name string `json:"name"` + System string `json:"system"` + } `json:"tgs"` + } + if err := json.Unmarshal(body, &doc); err != nil { + return -1 + } + m := make(map[int]string, len(doc.TGs)) + for _, e := range doc.TGs { + if e.TG <= 0 { + continue + } + s := strings.TrimSpace(e.System) + if s == "" { + s = strings.TrimSpace(e.Name) + } + if s != "" { + m[e.TG] = s + } + } + setTglistSystem(m) + return len(m) + } + + // Cold start: seed from the disk cache so names are present even before the + // first fetch (and if the list host is unreachable at boot). + if b, err := os.ReadFile(cacheFile); err == nil && len(b) > 0 { + if n := apply(b); n >= 0 { + log.Printf("tglist: seeded %d TG names from cache %s", n, cacheFile) + } + } + + pullOnce := func() { + // SVX source of truth is a local JSON file — read it directly, no HTTP. + if file != "" { + body, err := os.ReadFile(file) + if err != nil { + if debug { + log.Printf("DEBUG tglist read failed: %v (file=%s)", err, file) + } + return + } + if n := apply(body); n >= 0 && debug { + log.Printf("DEBUG tglist: %d TG names from %s", n, file) + } + return + } + req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) + if err != nil { + return + } + resp, err := client.Do(req) + if err != nil { + if debug { + log.Printf("DEBUG tglist fetch failed: %v (url=%s)", err, url) + } + return + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + if debug { + log.Printf("DEBUG tglist fetch status %d (url=%s)", resp.StatusCode, url) + } + return + } + body, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20)) + if err != nil { + return + } + n := apply(body) + if n < 0 { + return // parse error: keep last-good + } + if err := os.MkdirAll(filepath.Dir(cacheFile), 0o755); err == nil { + _ = os.WriteFile(cacheFile, body, 0o644) + } + if debug { + log.Printf("DEBUG tglist: %d TG names from %s", n, url) + } + } + + pullOnce() + t := time.NewTicker(interval) + defer t.Stop() + for { + select { + case <-ctx.Done(): + return + case <-t.C: + pullOnce() + } + } +} + +// ---- Complete network structure (nodelist collects everything; the website +// picks what it needs). Every slice is always allocated so nothing serializes as +// JSON null. Nothing is stripped: online + offline reflectors are all kept, tagged. + +type netNode struct { + Callsign string `json:"callsign"` + TG int `json:"tg"` + RestrictedTG bool `json:"restrictedTG"` + Location string `json:"location"` + NodeClass string `json:"nodeClass"` + SW string `json:"sw"` + Sysop string `json:"sysop"` + MonitoredTGs []int `json:"monitoredTGs"` + IsTalker bool `json:"isTalker"` + RxFreq string `json:"rxFreq"` + TxFreq string `json:"txFreq"` + RxCtcss string `json:"rxCtcss"` // node's REPORT_CTCSS (the subtone it transmits) + TxCtcss string `json:"txCtcss"` // access tone selecting the current TG +} + +type netTrunkRef struct { + Name string `json:"name"` + Host string `json:"host"` + Connected bool `json:"connected"` + Kind string `json:"kind"` +} + +type netReflector struct { + Type string `json:"type"` // "local" | "reflector" + Self bool `json:"self"` + Uplink bool `json:"uplink"` + Name string `json:"name"` + Hostname string `json:"hostname"` + IP string `json:"ip"` + Port string `json:"port"` + Online bool `json:"online"` // up in the network (HTTP-read OR a trunk to it is connected) + Firewalled bool `json:"firewalled"` // online via trunk but HTTP status unreachable (no node detail) + Version string `json:"version"` + Mode string `json:"mode"` + Aliases []string `json:"aliases"` + Nodes []netNode `json:"nodes"` + Trunks []netTrunkRef `json:"trunks"` +} + +type netTrunk struct { + Name string `json:"name"` + Owner string `json:"owner"` + Host string `json:"host"` + Connected bool `json:"connected"` + NodeCount int `json:"node_count"` + Kind string `json:"kind"` + Nodes []netNode `json:"nodes"` // callsigns this trunk carries (the "external" node list) +} + +type networkDoc struct { + Generated string `json:"generated"` + Reflectors []netReflector `json:"reflectors"` + Trunks []netTrunk `json:"trunks"` +} + +var ( + networkMu sync.RWMutex + networkJSON = []byte(`{"generated":"","reflectors":[],"trunks":[]}`) + lastNetworkHash string + networkJSONPath = env("NETWORK_JSON_PATH", "/var/www/html/svxportal/network.json") +) + +// writeMaybeGzip writes b with a 200, gzip-compressing when the client accepts it. +// The status/network blobs are large (~900 KB) and served to every browser on each +// update, so gzip (~6x smaller) is a major bandwidth win. The caller sets Content-Type. +func writeMaybeGzip(w http.ResponseWriter, r *http.Request, b []byte) { + if strings.Contains(r.Header.Get("Accept-Encoding"), "gzip") { + w.Header().Set("Content-Encoding", "gzip") + w.Header().Set("Vary", "Accept-Encoding") + w.WriteHeader(200) + gz := gzip.NewWriter(w) + _, _ = gz.Write(b) + _ = gz.Close() + return + } + w.WriteHeader(200) + _, _ = w.Write(b) +} + +// rawScalarStr returns a JSON scalar (string or number) as a trimmed string. +func rawScalarStr(r json.RawMessage) string { + if len(r) == 0 { + return "" + } + var s string + if json.Unmarshal(r, &s) == nil { + return strings.TrimSpace(s) + } + return strings.TrimSpace(strings.Trim(string(r), `"`)) +} + +// toNetNode extracts the fields the website needs from a raw node object. +func toNetNode(callsign string, raw json.RawMessage) netNode { + var n struct { + Callsign string `json:"callsign"` + TG int `json:"tg"` + RestrictedTG bool `json:"restrictedTG"` + NodeLocation string `json:"nodeLocation"` + NodeClass string `json:"nodeClass"` + SW string `json:"sw"` + SwVer string `json:"swVer"` + Sysop string `json:"sysop"` + MonitoredTGs []int `json:"monitoredTGs"` + IsTalker bool `json:"isTalker"` + RXFREQ json.RawMessage `json:"RXFREQ"` + TXFREQ json.RawMessage `json:"TXFREQ"` + ReportCtcss json.RawMessage `json:"reportCtcss"` + ToneToTG map[string]int `json:"toneToTalkgroup"` + QTH []struct { + Name string `json:"name"` + RX map[string]struct { + Freq json.RawMessage `json:"freq"` + } `json:"rx"` + TX map[string]struct { + Freq json.RawMessage `json:"freq"` + } `json:"tx"` + } `json:"qth"` + } + _ = json.Unmarshal(raw, &n) + cs := callsign + if cs == "" { + cs = n.Callsign + } + loc := n.NodeLocation + if loc == "" { + for _, q := range n.QTH { + if q.Name != "" { + loc = q.Name + break + } + } + } + // Frequencies: top-level RXFREQ/TXFREQ, else the first qth rx/tx freq. + rxFreq, txFreq := rawScalarStr(n.RXFREQ), rawScalarStr(n.TXFREQ) + if rxFreq == "" || txFreq == "" { + for _, q := range n.QTH { + if rxFreq == "" { + for _, p := range q.RX { + if f := rawScalarStr(p.Freq); f != "" { + rxFreq = f + break + } + } + } + if txFreq == "" { + for _, p := range q.TX { + if f := rawScalarStr(p.Freq); f != "" { + txFreq = f + break + } + } + } + if rxFreq != "" && txFreq != "" { + break + } + } + } + // Subtones: RX = the node's REPORT_CTCSS; TX = the access tone selecting the + // current TG (from toneToTalkgroup). + rxCtcss := rawScalarStr(n.ReportCtcss) + txCtcss := "" + if n.TG != 0 { + for tone, tg := range n.ToneToTG { + if tg == n.TG { + txCtcss = tone + break + } + } + } + sw := strings.TrimSpace(n.SW + " " + n.SwVer) + mt := n.MonitoredTGs + if mt == nil { + mt = []int{} + } + return netNode{Callsign: cs, TG: n.TG, RestrictedTG: n.RestrictedTG, Location: loc, + NodeClass: n.NodeClass, SW: sw, Sysop: n.Sysop, MonitoredTGs: mt, IsTalker: n.IsTalker, + RxFreq: rxFreq, TxFreq: txFreq, RxCtcss: rxCtcss, TxCtcss: txCtcss} +} + +// aliasesNorm returns the normalized-host form of each alias. +func aliasesNorm(aliases []string) []string { + out := make([]string, 0, len(aliases)) + for _, a := range aliases { + out = append(out, normalizeHost(a)) + } + return out +} + +// localReflectorMeta reads the local reflector's name (SERVER_CERT ORG) and the +// identifiers (hostname/IP) that mark a crawled ref as "self". +func localReflectorMeta() (name string, selfKeys map[string]struct{}) { + selfKeys = map[string]struct{}{} + name = "LOCAL" + f, err := os.Open(svxreflectorConf) + if err != nil { + return + } + defer f.Close() + sc := bufio.NewScanner(f) + inCert := false + for sc.Scan() { + line := strings.TrimSpace(sc.Text()) + if strings.HasPrefix(line, "[") && strings.HasSuffix(line, "]") { + inCert = strings.EqualFold(line[1:len(line)-1], "SERVER_CERT") + continue + } + if !inCert { + continue + } + i := strings.IndexByte(line, '=') + if i < 0 { + continue + } + k := strings.ToUpper(strings.TrimSpace(line[:i])) + v := strings.TrimSpace(line[i+1:]) + switch k { + case "ORG": + if v != "" { + name = v + } + case "COMMON_NAME": + if v != "" { + selfKeys[normalizeHost(v)] = struct{}{} + if ip := hostKey(v); ip != "" { + selfKeys[ip] = struct{}{} + } + } + case "SUBJECT_ALT_NAME": + for _, part := range strings.Split(v, ",") { + part = strings.TrimSpace(part) + up := strings.ToUpper(part) + if strings.HasPrefix(up, "DNS:") { + h := normalizeHost(part[4:]) + selfKeys[h] = struct{}{} + if ip := hostKey(h); ip != "" { + selfKeys[ip] = struct{}{} + } + } else if strings.HasPrefix(up, "IP:") { + selfKeys[strings.TrimSpace(part[3:])] = struct{}{} + } + } + } + } + return +} + +// buildNetworkDoc assembles the complete structure from the local reflector +// status (body) + the crawl (externalState). Strips nothing; tags everything. +func buildNetworkDoc(body []byte) networkDoc { + externalState.mu.RLock() + sources := externalState.sources + rawTrunks := externalState.trunks + routing := externalState.routing + generated := externalState.generated + externalState.mu.RUnlock() + + doc := networkDoc{Generated: generated, Reflectors: []netReflector{}, Trunks: []netTrunk{}} + + localName, selfKeys := localReflectorMeta() + uplinks := map[string]struct{}{} + for _, h := range uplinkHostsFromConf(svxreflectorConf) { + uplinks[normalizeHost(h)] = struct{}{} + } + + // Local reflector entry (SA7AUX): nodes without remoteTrunk + local trunks. + var top map[string]json.RawMessage + _ = json.Unmarshal(body, &top) + localNodes := []netNode{} + if raw, ok := top["nodes"]; ok { + var m map[string]json.RawMessage + if json.Unmarshal(raw, &m) == nil { + for cs, nr := range m { + var chk struct { + RemoteTrunk json.RawMessage `json:"remoteTrunk"` + } + _ = json.Unmarshal(nr, &chk) + if len(chk.RemoteTrunk) > 0 { + continue + } + localNodes = append(localNodes, toNetNode(cs, nr)) + } + } + } + sort.Slice(localNodes, func(i, j int) bool { return localNodes[i].Callsign < localNodes[j].Callsign }) + localTrunks := []netTrunkRef{} + if raw, ok := top["trunks"]; ok { + var m map[string]json.RawMessage + if json.Unmarshal(raw, &m) == nil { + for name, tr := range m { + localTrunks = append(localTrunks, netTrunkRef{Name: name, Host: trunkHost(tr), Connected: trunkConnected(tr), Kind: "trunk"}) + } + } + } + if raw, ok := top["trunks_UDP"]; ok { + var arr []json.RawMessage + if json.Unmarshal(raw, &arr) == nil { + for _, tr := range arr { + name, norm, h := normalizeUDPTrunk(tr) + if name == "" { + continue + } + localTrunks = append(localTrunks, netTrunkRef{Name: name, Host: h, Connected: trunkConnected(norm), Kind: "udp"}) + } + } + } + if raw, ok := top["satellites"]; ok { + var m map[string]struct { + Authenticated bool `json:"authenticated"` + } + if json.Unmarshal(raw, &m) == nil { + satNames := satNamesFromConf(svxSatConf) + for id, sv := range m { + // Real callsign (KT4ROY-SAT1) from chan_svx_sat, not the raw id. + name := satNames[id] + if name == "" { + name = id + } + localTrunks = append(localTrunks, netTrunkRef{Name: name, Connected: sv.Authenticated, Kind: "satellite"}) + } + } + } + doc.Reflectors = append(doc.Reflectors, netReflector{ + Type: "local", Self: true, Name: localName, Online: true, + Aliases: []string{}, Nodes: localNodes, Trunks: localTrunks, + }) + + // Parse the merged trunk map once: for node counts (flat list below) and to + // know which hosts have a connected trunk (up in the network even if their + // HTTP status is firewalled from here). + var trmap map[string]json.RawMessage + _ = json.Unmarshal(rawTrunks, &trmap) + if trmap == nil { + trmap = map[string]json.RawMessage{} + } + // The local reflector's own /status carries each trunk's peer nodes (pushed via + // the reflector's trunk protocol) — take them straight from there. We poll NO + // ONE for network info; the local status is the single source. + if bt, ok := top["trunks"]; ok { + var bm map[string]json.RawMessage + if json.Unmarshal(bt, &bm) == nil { + for name, raw := range bm { + if _, dup := trmap[name]; !dup { + trmap[name] = raw + } + } + } + } + // Satellite trunks: their nodes live in the body's "satellites" block, keyed by + // sat id; index them under the same name localTrunks uses so their nodes show. + if bs, ok := top["satellites"]; ok { + var sm map[string]json.RawMessage + if json.Unmarshal(bs, &sm) == nil { + satNames := satNamesFromConf(svxSatConf) + for id, raw := range sm { + name := satNames[id] + if name == "" { + name = id + } + if _, dup := trmap[name]; !dup { + trmap[name] = raw + } + } + } + } + connectedHosts := map[string]struct{}{} + for _, tr := range trmap { + var m struct { + Host string `json:"host"` + Connected bool `json:"connected"` + } + if json.Unmarshal(tr, &m) == nil && m.Connected && m.Host != "" { + connectedHosts[normalizeHost(m.Host)] = struct{}{} + } + } + + // Route-based node attribution (mirrors SvxReflectorDashboard's + // merge_trunk_status_nodes): every crawled reflector's Routing_table maps a + // callsign to the trunk IP it is reachable via. Group by IP so trunks and + // reflectors whose own node list is unreadable (firewalled UDP peers) still + // show their members. Deduped per IP; our own callsign is never added (no + // self-dupe). + // Full per-node detail (location/freq/sw/sysop) pulled from reflectors' merged + // rosters — used to enrich route nodes the Routing_table only names sparsely. + detailByCallsign := map[string]json.RawMessage{} + for _, s := range sources { + for cs, nr := range s.Detail { + if _, ok := detailByCallsign[cs]; !ok { + detailByCallsign[cs] = nr + } + } + } + routeByIP := map[string][]netNode{} + routeSeen := map[string]map[string]struct{}{} + for _, s := range sources { + for _, e := range s.Routing { + ip := normalizeHost(e.Trunk) + if ip == "" || e.Callsign == "" { + continue + } + if localName != "" && strings.EqualFold(e.Callsign, localName) { + continue + } + if routeSeen[ip] == nil { + routeSeen[ip] = map[string]struct{}{} + } + if _, dup := routeSeen[ip][e.Callsign]; dup { + continue + } + routeSeen[ip][e.Callsign] = struct{}{} + // Prefer the full-detail node from the merged roster; fall back to the + // sparse Routing_table fields (callsign + tg + monitored TGs). + if draw, ok := detailByCallsign[e.Callsign]; ok { + routeByIP[ip] = append(routeByIP[ip], toNetNode(e.Callsign, draw)) + continue + } + mon := e.MonitoredTGs + if mon == nil { + mon = []int{} + } + routeByIP[ip] = append(routeByIP[ip], netNode{Callsign: e.Callsign, TG: e.TG, MonitoredTGs: mon}) + } + } + + // Crawled reflectors — all of them, tagged (nothing dropped). + for _, s := range sources { + nodes := []netNode{} + for cs, nr := range s.Nodes { + nodes = append(nodes, toNetNode(cs, nr)) + } + // Fill members we can't read directly (firewalled peers) from the routing + // tables of reflectors that DO see this one — matched by IP / tag. Deduped + // by callsign so a directly-read node is never duplicated. + seenN := map[string]struct{}{} + for _, n := range nodes { + seenN[n.Callsign] = struct{}{} + } + for _, key := range []string{normalizeHost(s.IP), normalizeHost(s.Tag)} { + if key == "" { + continue + } + for _, rn := range routeByIP[key] { + if _, ok := seenN[rn.Callsign]; ok { + continue + } + seenN[rn.Callsign] = struct{}{} + nodes = append(nodes, rn) + } + } + sort.Slice(nodes, func(i, j int) bool { return nodes[i].Callsign < nodes[j].Callsign }) + trunks := []netTrunkRef{} + for _, t := range s.Trunks { + trunks = append(trunks, netTrunkRef{Name: t.Name, Host: t.Host, Connected: t.Connected, Kind: t.Kind}) + } + aliases := s.Aliases + if aliases == nil { + aliases = []string{} + } + _, isSelf := selfKeys[s.IP] + if !isSelf { + _, isSelf = selfKeys[normalizeHost(s.Tag)] + } + // Routing table: this ref's trunk IP maps to our own callsign → it's us. + if !isSelf && localName != "" { + if strings.EqualFold(routing[s.IP], localName) || strings.EqualFold(routing[normalizeHost(s.Tag)], localName) { + isSelf = true + } + } + _, isUplink := uplinks[normalizeHost(s.Tag)] + if !isUplink { + _, isUplink = uplinks[normalizeHost(hostFromURL(s.URL))] + } + hostname := hostFromURL(s.URL) + if hostname == "" && !isIPStr(s.Tag) { + hostname = s.Tag + } + // Online = HTTP-read OR a trunk to this reflector is connected. If it's up + // only via a connected trunk (HTTP unreachable), it's firewalled. + online := s.OK + firewalled := false + if !online { + for _, k := range append([]string{normalizeHost(s.Tag), normalizeHost(hostname), normalizeHost(s.IP)}, aliasesNorm(aliases)...) { + if k == "" { + continue + } + if _, ok := connectedHosts[k]; ok { + online = true + firewalled = true + break + } + } + } + doc.Reflectors = append(doc.Reflectors, netReflector{ + Type: "reflector", Self: isSelf, Uplink: isUplink, + Name: s.Tag, Hostname: hostname, IP: s.IP, Port: s.HTTPPort, + Online: online, Firewalled: firewalled, Version: s.Version, Mode: s.Mode, Aliases: aliases, + Nodes: nodes, Trunks: trunks, + }) + } + + // Flat list of ALL links in the network — trunks, UDP peers AND satellites — + // built from the reflectors' own link lists (deduped by name), with owner + + // node count. This includes satellites, which the merged trunk map omits. + seenT := map[string]struct{}{} + for _, r := range doc.Reflectors { + for _, t := range r.Trunks { + if _, ok := seenT[t.Name]; ok { + continue + } + seenT[t.Name] = struct{}{} + tnodes := []netNode{} + if raw, ok := trmap[t.Name]; ok { + var m struct { + Nodes []json.RawMessage `json:"nodes"` + } + if json.Unmarshal(raw, &m) == nil { + for _, nr := range m.Nodes { + tnodes = append(tnodes, toNetNode("", nr)) + } + } + } + // A UDP peer exposes no nodes[] inline; fill its members from the owning + // reflector's Routing_table (callsigns reachable via its host IP). Deduped. + seenTN := map[string]struct{}{} + for _, n := range tnodes { + seenTN[n.Callsign] = struct{}{} + } + for _, rn := range routeByIP[normalizeHost(t.Host)] { + if _, ok := seenTN[rn.Callsign]; ok { + continue + } + seenTN[rn.Callsign] = struct{}{} + tnodes = append(tnodes, rn) + } + sort.Slice(tnodes, func(i, j int) bool { return tnodes[i].Callsign < tnodes[j].Callsign }) + doc.Trunks = append(doc.Trunks, netTrunk{ + Name: t.Name, Owner: r.Name, Host: t.Host, + Connected: t.Connected, NodeCount: len(tnodes), Kind: t.Kind, Nodes: tnodes, + }) + } + } + sort.Slice(doc.Trunks, func(i, j int) bool { return doc.Trunks[i].Name < doc.Trunks[j].Name }) + return doc +} + +// writeNetworkFile atomically writes b to networkJSONPath (temp + rename) when it +// changed, so external fetchers never see a half-written file. +func writeNetworkFile(b []byte) { + if networkJSONPath == "" { + return + } + dir := filepath.Dir(networkJSONPath) + tmp, err := os.CreateTemp(dir, ".network-*.json") + if err != nil { + return + } + tmpName := tmp.Name() + if _, err := tmp.Write(b); err != nil { + tmp.Close() + os.Remove(tmpName) + return + } + tmp.Close() + if err := os.Rename(tmpName, networkJSONPath); err != nil { + os.Remove(tmpName) + } +} + +// injectExternal adds ext_sources and network_trunks (from externalState) as +// top-level fields to the served status JSON. If no external data is available +// yet, body is returned unchanged so local nodes/trunks still go out. +func injectExternal(body []byte) []byte { + // Build the complete network structure and publish it (served at /api/network + // and written to networkJSONPath for other systems to fetch). + doc := buildNetworkDoc(body) + nb, nerr := json.Marshal(doc) + if nerr == nil { + networkMu.Lock() + networkJSON = nb + networkMu.Unlock() + if h := hash(nb); h != lastNetworkHash { + lastNetworkHash = h + writeNetworkFile(nb) + } + } + + var top map[string]json.RawMessage + if json.Unmarshal(body, &top) != nil { + return body + } + // The complete structure the frontend reads. The old flat ext_sources / + // network_trunks fields (unused by the frontend) are intentionally NOT emitted + // — they were ~66% of the served blob. Other systems fetch /api/network. + if nerr == nil { + top["network"] = nb + } + if bb, err := json.Marshal(top); err == nil { + return bb + } + return body +} + +const ( + heartbeatKey = "/registry/svxlink/nodelist/heartbeat" +) + +type statusEnvelope struct { + Nodes map[string]json.RawMessage `json:"nodes"` + Trunks map[string]trunkStatus `json:"trunks"` + Satellites map[string]satelliteStatus `json:"satellites"` +} + +// satelliteStatus mirrors a satellite trunk (chan_svx_sat / chan_pbx_svx). Its +// nodes[] carry a nodeClass so a PBX gateway talker can be shown as +// " via " rather than the raw gateway node name. +type satelliteStatus struct { + Nodes []struct { + Callsign string `json:"callsign"` + TG int `json:"tg"` + IsTalker bool `json:"isTalker"` + NodeClass string `json:"nodeClass"` + NodeLocation string `json:"nodeLocation"` + SW string `json:"sw"` + } `json:"nodes"` +} + +// trunkStatus mirrors the per-trunk JSON emitted by the reflector +// (SatelliteLink/TrunkLink statusJson): active_talkers maps a TG to the real +// remote talker callsign, and nodes[] lists partner nodes with isTalker. +type trunkStatus struct { + ActiveTalkers map[string]string `json:"active_talkers"` + Nodes []struct { + Callsign string `json:"callsign"` + TG int `json:"tg"` + IsTalker bool `json:"isTalker"` + } `json:"nodes"` +} + +type nodeInfo struct { + ToneToTalkgroup map[string]int `json:"toneToTalkgroup"` + QTH []struct { + Name string `json:"name"` + RX map[string]struct { + Freq float64 `json:"freq"` + Name string `json:"name"` + Active bool `json:"active"` + Siglev float64 `json:"siglev"` + } `json:"rx"` + TX map[string]struct { + Freq float64 `json:"freq"` + } `json:"tx"` + } `json:"qth"` + + RX float64 `json:"rx"` + TX float64 `json:"tx"` + RxFreq float64 `json:"rxFreq"` + TxFreq float64 `json:"txFreq"` + + IsTalker bool `json:"isTalker"` + TG int `json:"tg"` + NodeLocation string `json:"nodeLocation"` + Sysop string `json:"sysop"` +} + +// live holds the real-callsign lookup maps maintained by the poller and read +// by the log tailer: +// - tgTalker: TG -> real remote talker callsign (from trunks[].active_talkers) +// - nodeSysop: node-name -> operator callsign (from nodes[].sysop) +// - tgMeta: TG -> talker meta published on MQTT (talker//meta) by an +// external source such as astmeta (AllStar bridge talker id) +var live = struct { + mu sync.RWMutex + tgTalker map[int]string + tgTalkerAt map[int]int64 // TG -> unix ts tgTalker[tg] last had a value (sticky window) + nodeSysop map[string]string + tgMeta map[int]talkerMeta + asNodes map[string]allstarNode + viaNode map[string]bridgeInfo // UPPER(via callsign) -> bridge target, long-lived + tgLoc map[int]string // TG -> gateway nodeLocation (e.g. SA7AUX-PBX), for the Location column + tgMode map[int]string // TG -> Backend/mode of the gateway node (e.g. PBX), for the Backend column +}{tgTalker: map[int]string{}, tgTalkerAt: map[int]int64{}, nodeSysop: map[string]string{}, + tgMeta: map[int]talkerMeta{}, asNodes: map[string]allstarNode{}, + viaNode: map[string]bridgeInfo{}, tgLoc: map[int]string{}, tgMode: map[int]string{}} + +// locForTG returns the gateway nodeLocation currently mapped to a TG (empty if none). +func locForTG(tg int) string { + // FROZEN DESIGN: the tglist is the SOLE source of the Location shown in + // Lastheard. No live gateway location, no astmeta, no nodes[].system — every + // talker on a TG shows exactly the tglist name for that TG (blank if the TG + // is not in the list), so the display is identical on every box. + return systemForTG(tg) +} + +// modeForTG returns the gateway Backend/mode currently mapped to a TG (empty if none). +func modeForTG(tg int) string { + live.mu.RLock() + defer live.mu.RUnlock() + return live.tgMode[tg] +} + +// bridgeInfo is what a bridge login (via) speaks through: the AllStar node it is +// connected to and that node's system name (e.g. 45192 / "East Coast Reflector"). +// Long-lived + redis-persisted so the web can show "connected to " and +// the connected-nodes button on the bridge row even when it is idle. +type bridgeInfo struct { + Node string `json:"node"` + System string `json:"system"` +} + +// allstarNode is one station/system heard through the bridge via the +// talker-meta ingest. Kept ALLSTAR_NODES_TTL from the last keyup and +// published in reflector-status "nodes" format on /api/allstar-nodes. +type allstarNode struct { + Callsign string + Node string + System string + Via string + TG int + At time.Time +} + +// talkerMeta is one talker-meta entry (fed by POST /api/talker-meta): who is +// really talking on the TG and from which node/system. Generic across +// sources: the AllStar bridge fills it via the svxlink TCL hook (node = +// AllStar node number), a DMR/NXDN bridge can fill the same fields (node = +// DMR id, resolved against the radioid tables in redis). +type talkerMeta struct { + Callsign string `json:"callsign"` + Node string `json:"node"` + System string `json:"system"` + Via string `json:"via"` // reflector callsign the bridge speaks through + Network string `json:"network"` // ingress mode: AllStar / DMR / DMR/YSF … + At time.Time +} + +// talkerMetaFor returns the fresh talker meta for a TG, if any. logName is +// the talker name as it appears in the reflector log: when the meta carries +// a "via" callsign it only applies to that talker, so lingering meta can +// never mislabel an unrelated station on the same TG. +func talkerMetaFor(tg int, logName string) (talkerMeta, bool) { + live.mu.RLock() + defer live.mu.RUnlock() + m, ok := live.tgMeta[tg] + if !ok || m.Callsign == "" || time.Since(m.At) > talkerMetaTTL { + return talkerMeta{}, false + } + // The meta's via identifies the talker's bridge. Prefer the per-TG via + // configured in svxdr.json (nodes[].via) so the callsign/via is defined in + // ONE place and need NOT equal the reflector-log talker name (svx_sat.conf + // callsign). Only when no per-TG via is configured do we fall back to the + // old log-name match (backward compat for boxes without a nodes[] config). + if anyViaForTG(tg) { + if m.Via != "" && !viaConfiguredForTG(tg, m.Via) { + return talkerMeta{}, false + } + } else if m.Via != "" && m.Via != logName { + return talkerMeta{}, false + } + return m, true +} + +// resolveTalker maps a node/trunk name (as it appears in the reflector log) to +// the real operator callsign, returning the trunk/node name separately when it +// differs so the UI can show both. For local nodes, fresh talker meta +// (e.g. the AllStar system behind a USRP bridge) wins over the sysop mapping. +func resolveTalker(nodeName string, tg int, isTrunk bool) (callsign, trunk string) { + callsign = nodeName + live.mu.RLock() + if isTrunk { + if c := live.tgTalker[tg]; c != "" { + callsign = c + } + } else if m, ok := live.tgMeta[tg]; ok && m.Callsign != "" && + time.Since(m.At) <= talkerMetaTTL && + (m.Via == "" || m.Via == nodeName) { + callsign = m.Callsign + } else if s := live.nodeSysop[nodeName]; s != "" { + callsign = s + } + live.mu.RUnlock() + if callsign != nodeName { + trunk = nodeName + } + return +} + +func isDigits(s string) bool { + if s == "" { + return false + } + for _, r := range s { + if r < '0' || r > '9' { + return false + } + } + return true +} + +// isLoopbackAddr reports whether a net/http RemoteAddr ("ip:port") is loopback +// (127.0.0.0/8 or ::1). Used to let same-box bridges POST talker-meta untokened. +func isLoopbackAddr(remote string) bool { + host, _, err := net.SplitHostPort(remote) + if err != nil { + host = remote + } + ip := net.ParseIP(host) + return ip != nil && ip.IsLoopback() +} + +// injectTalkerMeta adds a top-level talker_meta object (tg -> fresh meta) to +// the served status JSON so the web can show the real talker behind a bridge +// node. The web applies the same via-gating as lastheard: the meta only +// labels the node whose callsign matches via. +func injectTalkerMeta(body []byte) []byte { + type metaOut struct { + Callsign string `json:"callsign"` + Node string `json:"node"` + System string `json:"system"` + Via string `json:"via"` + } + out := map[string]metaOut{} + bridge := map[string]bridgeInfo{} // via callsign -> {node, system} (for the + // web's per-bridge "connected to" + button) + // Network-wide bridges first (crawled from every reflector's own + // /api/bridge-nodes), so any box shows a bridge that lives on another + // server; the LOCAL map overrides on conflict (it is authoritative). + externalState.mu.RLock() + for via, bi := range externalState.bridgeNodes { + bridge[via] = bi + } + externalState.mu.RUnlock() + live.mu.RLock() + for tg, m := range live.tgMeta { + if m.Callsign == "" || time.Since(m.At) > talkerMetaTTL { + continue + } + out[strconv.Itoa(tg)] = metaOut{m.Callsign, m.Node, m.System, m.Via} + } + for via, bi := range live.viaNode { + bridge[via] = bi + } + live.mu.RUnlock() + if len(out) == 0 && len(bridge) == 0 { + return body + } + var top map[string]json.RawMessage + if json.Unmarshal(body, &top) != nil { + return body + } + if len(out) > 0 { + if b, err := json.Marshal(out); err == nil { + top["talker_meta"] = b + } + } + if len(bridge) > 0 { + if b, err := json.Marshal(bridge); err == nil { + top["bridge_nodes"] = b + } + } + bb, err := json.Marshal(top) + if err != nil { + return body + } + return bb +} + +type talkerSession struct { + StartTs int64 + TG int + Location string + RxName string + Siglev float64 +} + +type lastheardEvent struct { + Ts int64 `json:"ts"` + Callsign string `json:"callsign"` + Trunk string `json:"trunk,omitempty"` + TG int `json:"tg"` + Duration int64 `json:"duration_s"` + Location string `json:"location,omitempty"` + RxName string `json:"rx_name,omitempty"` + Siglev float64 `json:"siglev,omitempty"` + Source string `json:"source"` + // Source-network identity from talker meta (astmeta for AllStar, the + // DMR bridge later): node/ID number and human-readable system name. + // Empty for plain svxlink traffic — the UI shows them as blank columns. + Node string `json:"node,omitempty"` + System string `json:"system,omitempty"` + // Ingress mode: HOW the talker enters — "AllStar", "SVXLink", "trunk", + // and later "DMR", "DMR/YSF", "DMR/OB"… from the callmgr bridge. The web + // shows this as the Mode column (replacing the old System name). Always + // set: a row with no meta defaults to "trunk" or "SVXLink" by source. + Network string `json:"network,omitempty"` +} + +type runtimeState struct { + mu sync.RWMutex + + StartedAt time.Time + + LastFetchOKAt time.Time + LastFetchURL string + LastFetchBytes int + LastFetchErr string + + LastParsedNodes int + LastStatusHash string + + LastRedisOKAt time.Time + LastRedisErr string +} + +var st = &runtimeState{ + StartedAt: time.Now(), +} + +// ---- Fas 2: federation gap detector ------------------------------------------ +// Compares the network-wide aggregate (FEDERATION_AGGREGATE_URL, sa2blv's +// reflector-network) against what our OWN recursive crawl + MQTT actually cover. +// Purely diagnostic: it names the reflectors/callsigns we are NOT seeing so they +// can be added to FEDERATION_SEEDS. Nothing here is merged into the served roster +// — our own fresh data stays authoritative. + +var federationGap = struct { + mu sync.RWMutex + updatedAt time.Time + aggTotal int + haveTotal int + missingCS int + missingRefs map[string]int // reflector -> count of its nodes we do not cover + err string +}{missingRefs: map[string]int{}} + +// ourCoveredCallsigns returns the set of callsigns we already see, from the +// recursive crawl (every reflector's /status nodes) and every MQTT source. +func ourCoveredCallsigns() map[string]struct{} { + have := map[string]struct{}{} + externalState.mu.RLock() + for _, s := range externalState.sources { + for cs := range s.Nodes { + have[cs] = struct{}{} + } + } + externalState.mu.RUnlock() + mqttState.mu.RLock() + for _, m := range mqttState.bySrc { + for cs := range m { + have[cs] = struct{}{} + } + } + mqttState.mu.RUnlock() + return have +} + +func setFederationGapErr(msg string) { + federationGap.mu.Lock() + federationGap.err = msg + federationGap.updatedAt = time.Now() + federationGap.mu.Unlock() + if debug { + log.Printf("DEBUG federation gap fetch failed: %s", msg) + } +} + +func federationGapOnce(client *http.Client) { + resp, err := client.Get(federationAggURL) + if err != nil { + setFederationGapErr(err.Error()) + return + } + body, _ := io.ReadAll(io.LimitReader(resp.Body, 16<<20)) + resp.Body.Close() + if resp.StatusCode != http.StatusOK { + setFederationGapErr(fmt.Sprintf("HTTP %d", resp.StatusCode)) + return + } + // Aggregate is an object keyed by callsign; we only need each node's reflector. + var agg map[string]struct { + Reflector string `json:"reflector"` + } + if json.Unmarshal(body, &agg) != nil { + setFederationGapErr("aggregate is not a callsign->node object") + return + } + + // Feed the aggregate into the All-Nodes flat layer so Monitor Nodes covers + // the whole network (real reflector names), not just our own crawl. + am := make(map[string]flatNode, len(agg)) + for cs, n := range agg { + r := strings.TrimSpace(n.Reflector) + if r == "" { + continue + } + am[strings.ToUpper(cs)] = flatNode{Callsign: cs, Ref: r} + } + setAggNodes(am) + + have := ourCoveredCallsigns() + missingRefs := map[string]int{} + missing := 0 + for cs, n := range agg { + if _, ok := have[cs]; ok { + continue + } + missing++ + ref := strings.TrimSpace(n.Reflector) + if ref == "" { + ref = "(unknown)" + } + missingRefs[ref]++ + } + + federationGap.mu.Lock() + federationGap.updatedAt = time.Now() + federationGap.aggTotal = len(agg) + federationGap.haveTotal = len(have) + federationGap.missingCS = missing + federationGap.missingRefs = missingRefs + federationGap.err = "" + federationGap.mu.Unlock() + + if missing > 0 { + type kv struct { + ref string + n int + } + top := make([]kv, 0, len(missingRefs)) + for r, n := range missingRefs { + top = append(top, kv{r, n}) + } + sort.Slice(top, func(i, j int) bool { return top[i].n > top[j].n }) + parts := make([]string, 0, len(top)) + for i, t := range top { + if i >= 12 { + parts = append(parts, fmt.Sprintf("…+%d more", len(top)-i)) + break + } + parts = append(parts, fmt.Sprintf("%s(%d)", t.ref, t.n)) + } + log.Printf("federation gap: %d/%d aggregate callsigns not covered — missing reflectors: %s (add to FEDERATION_SEEDS)", + missing, len(agg), strings.Join(parts, ", ")) + } else if debug { + log.Printf("federation gap: 0 — we cover all %d aggregate callsigns", len(agg)) + } +} + +func federationGapLoop() { + if federationAggURL == "" { + log.Printf("federation gap detector disabled (FEDERATION_AGGREGATE_URL empty)") + return + } + client := &http.Client{Timeout: 20 * time.Second} + time.Sleep(30 * time.Second) // let the crawl populate before the first diff + federationGapOnce(client) + t := time.NewTicker(federationAggInterval) + defer t.Stop() + for range t.C { + federationGapOnce(client) + } +} + +// Run is the nodelist subsystem entrypoint for svxdr. It applies config from +// svxdr.json over the defaults, starts the crawl/lastheard/federation/xlink +// goroutines and the HTTP server, and returns when ctx is cancelled. +func Run(rctx context.Context, c *config.Config) error { + applyConfig(c) + rdb := redis.NewClient(&redis.Options{ + Network: "unix", + Addr: redisSock, + Password: redisPass, + }) + + if err := rdb.Ping(ctx).Err(); err != nil { + return fmt.Errorf("nodelist redis ping failed: %v (sock=%s)", err, redisSock) + } + + // Restore the persisted via->node map (bridge login -> AllStar node) so the + // web's per-bridge "connected nodes" button works immediately after a + // restart, before any fresh talker meta arrives. + if vn, err := rdb.HGetAll(ctx, "nodelist:vianode").Result(); err == nil && len(vn) > 0 { + live.mu.Lock() + for via, v := range vn { + node, system, _ := strings.Cut(v, "|") // stored as "node|system" + live.viaNode[strings.ToUpper(via)] = bridgeInfo{Node: node, System: system} + } + live.mu.Unlock() + log.Printf("restored %d via->node bridge mappings", len(vn)) + } + + // Location column render config now comes from svxdr.json (lastheard formats + // + reflector names derived from nodes[]), not a separate lastheard.json. + applyRenderFromConfig(c) + + go poller(rdb) + go logTailer(rdb, reflectorLogPath) + xlinkStart(rdb) + if mqttBroker != "" && mqttPrefix != "" { + go startMqtt() + } else { + log.Printf("mqtt disabled (set MQTT_BROKER and MQTT_TOPIC_PREFIX to enable)") + } + // Topology: crawl the network from our uplink (derived from svxreflector.conf) + // so the graph shows every reflector AND the trunk links between them. Each + // reflector's OWN /status carries the real peer hosts that draw the edges — an + // aggregated hub status does not, so nothing else can reproduce the links. + // Runs on a gentle interval (EXTERNAL_PULL_INTERVAL, default 60s) with last-good + // retention, so it is spread out and never hammers any single server. + go externalPuller() + + // Canonical TG->name list (svx-web /tglist.php). No-op until tglist.url is set + // in config; when set, it feeds systemForTG so Location renders the same name + // on every box that points at the same list. + go tglistPuller(rctx, c) + + mux := http.NewServeMux() + + mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "text/plain; charset=utf-8") + w.WriteHeader(200) + _, _ = w.Write([]byte("ok\n")) + }) + + mux.HandleFunc("/api/status", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json; charset=utf-8") + b, err := rdb.Get(ctx, statusKey).Bytes() + if err != nil { + http.Error(w, `{"msg":"no status yet"}`, http.StatusServiceUnavailable) + return + } + writeMaybeGzip(w, r, b) + }) + + // Fas 2: federation gap report — which reflectors/callsigns the network-wide + // aggregate has that our own crawl does not cover yet. Diagnostic only. + mux.HandleFunc("/api/gaps", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json; charset=utf-8") + federationGap.mu.RLock() + out := map[string]any{ + "updatedAt": federationGap.updatedAt, + "aggregateURL": federationAggURL, + "aggregateTotal": federationGap.aggTotal, + "weCover": federationGap.haveTotal, + "missingCallsigns": federationGap.missingCS, + "missingReflectors": federationGap.missingRefs, + "error": federationGap.err, + } + federationGap.mu.RUnlock() + b, _ := json.Marshal(out) + writeMaybeGzip(w, r, b) + }) + + // This box's OWN bridges (via -> {node, system}), for other reflectors to + // crawl and union into their network-wide bridge directory. Returns only + // the LOCAL map (never the crawled union) so crawling can't amplify/loop. + mux.HandleFunc("/api/bridge-nodes", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json; charset=utf-8") + out := map[string]bridgeInfo{} + live.mu.RLock() + for via, bi := range live.viaNode { + out[via] = bi + } + live.mu.RUnlock() + b, _ := json.Marshal(out) + writeMaybeGzip(w, r, b) + }) + + // The complete network structure on its own, for other systems to fetch. + mux.HandleFunc("/api/network", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json; charset=utf-8") + networkMu.RLock() + b := networkJSON + networkMu.RUnlock() + writeMaybeGzip(w, r, b) + }) + + mux.HandleFunc("/api/stream", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "text/event-stream; charset=utf-8") + w.Header().Set("Cache-Control", "no-cache") + w.Header().Set("Connection", "keep-alive") + + flusher, ok := w.(http.Flusher) + if !ok { + http.Error(w, "SSE not supported", 500) + return + } + + sub := rdb.Subscribe(ctx, pubsubChan) + defer sub.Close() + + _, _ = io.WriteString(w, "event: update\ndata: 1\n\n") + flusher.Flush() + + pingT := time.NewTicker(25 * time.Second) + defer pingT.Stop() + + ch := sub.Channel() + + for { + select { + case <-r.Context().Done(): + return + case <-pingT.C: + _, _ = io.WriteString(w, "event: ping\ndata: 1\n\n") + flusher.Flush() + case <-ch: + _, _ = io.WriteString(w, "event: update\ndata: 1\n\n") + flusher.Flush() + } + } + }) + + mux.HandleFunc("/api/debug", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json; charset=utf-8") + resp := map[string]any{ + "listen": listenAddr, + "reflectorURL": reflectorURL, + "pollInterval": pollInterval.String(), + "redisSock": redisSock, + "statusKey": statusKey, + "nodePrefix": nodeKeyPrefix, + "onlineTTL": effectiveOnlineTTL().String(), + "debug": debug, + } + + // Current talker meta (MQTT or ingest fed), with age so the freshness + // gating can be judged without knowing the TTL. + type metaDump struct { + talkerMeta + AgeSec int64 `json:"age_s"` + } + metas := map[string]metaDump{} + live.mu.RLock() + for tg, m := range live.tgMeta { + metas[strconv.Itoa(tg)] = metaDump{m, int64(time.Since(m.At).Seconds())} + } + live.mu.RUnlock() + resp["talkerMeta"] = metas + + st.mu.RLock() + resp["startedAt"] = st.StartedAt.Format(time.RFC3339) + resp["lastFetchOKAt"] = tOrEmpty(st.LastFetchOKAt) + resp["lastFetchURL"] = st.LastFetchURL + resp["lastFetchBytes"] = st.LastFetchBytes + resp["lastFetchErr"] = st.LastFetchErr + resp["lastParsedNodes"] = st.LastParsedNodes + resp["lastStatusHash"] = st.LastStatusHash + resp["lastRedisOKAt"] = tOrEmpty(st.LastRedisOKAt) + resp["lastRedisErr"] = st.LastRedisErr + st.mu.RUnlock() + + externalState.mu.RLock() + resp["externalPullURLs"] = externalPullURLs() // derived from conf (or override) + resp["externalConf"] = svxreflectorConf + resp["externalSources"] = len(externalState.sources) + resp["externalGenerated"] = externalState.generated + resp["externalLastOKAt"] = tOrEmpty(externalState.lastOKAt) + resp["externalLastErr"] = externalState.lastErr + externalState.mu.RUnlock() + + _ = json.NewEncoder(w).Encode(resp) + }) + + mux.HandleFunc("/api/lastheard", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json; charset=utf-8") + + limit := 200 + if s := r.URL.Query().Get("limit"); s != "" { + var n int + if _, err := fmt.Sscanf(s, "%d", &n); err == nil && n > 0 && n <= 1000 { + limit = n + } + } + + now := time.Now().Unix() + minTs := now - lastheardWindowSec + + vals, err := rdb.ZRevRangeByScore(ctx, lastheardKey, &redis.ZRangeBy{ + Min: fmt.Sprintf("%d", minTs), + Max: "+inf", + Offset: 0, + Count: int64(limit), + }).Result() + if err != nil { + http.Error(w, `{"msg":"redis error"}`, 500) + return + } + + out := make([]json.RawMessage, 0, len(vals)) + for _, v := range vals { + out = append(out, json.RawMessage(v)) + } + _ = json.NewEncoder(w).Encode(map[string]any{ + "window_s": lastheardWindowSec, + "count": len(out), + "events": out, + }) + }) + + mux.HandleFunc("/api/nodekeys", func(w http.ResponseWriter, r *http.Request) { + call := strings.TrimSpace(r.URL.Query().Get("call")) + if call == "" { + http.Error(w, `{"msg":"missing ?call="}`, 400) + return + } + + keys := []string{ + statusHashKey, + statusKey, + fmt.Sprintf("%s/%s/online", nodeKeyPrefix, call), + fmt.Sprintf("%s/%s/hash", nodeKeyPrefix, call), + fmt.Sprintf("%s/%s/json", nodeKeyPrefix, call), + fmt.Sprintf("%s/%s/rx", nodeKeyPrefix, call), + fmt.Sprintf("%s/%s/tx", nodeKeyPrefix, call), + fmt.Sprintf("%s/%s/toneToTalkgroup", nodeKeyPrefix, call), + heartbeatKey, + } + + out := map[string]any{"call": call, "keys": map[string]any{}} + m := out["keys"].(map[string]any) + + for _, k := range keys { + v, err := rdb.Get(ctx, k).Result() + if err != nil { + m[k] = nil + } else { + m[k] = v + } + } + + w.Header().Set("Content-Type", "application/json; charset=utf-8") + _ = json.NewEncoder(w).Encode(out) + }) + + // Talker metadata pushed from the USRP bridge's svxlink TCL hook + // (usrp_stationdata_received) over HTTPS. Feeds the same live.tgMeta + // store as the MQTT talker//meta leg, so via-gating and TTL apply + // unchanged. A bare numeric id is resolved to a callsign via the + // radioid tables callmgr2 keeps in redis (DMR first, then NXDN). + mux.HandleFunc("/api/talker-meta", func(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodPost { + http.Error(w, `{"msg":"POST only"}`, http.StatusMethodNotAllowed) + return + } + // Zero-config: en process på SAMMA burk (callmgr-satelliten, lokal astmeta) + // får POSTa utan token — så en DMR/SVX-brygga bara funkar utan konfig. + // Fjärranrop kräver X-Ingest-Token som förr (och att ett token är satt). + if !isLoopbackAddr(r.RemoteAddr) { + if ingestToken == "" { + http.Error(w, `{"msg":"ingest disabled"}`, http.StatusNotFound) + return + } + tok := r.Header.Get("X-Ingest-Token") + if subtle.ConstantTimeCompare([]byte(tok), []byte(ingestToken)) != 1 { + http.Error(w, `{"msg":"bad token"}`, http.StatusUnauthorized) + return + } + } + var in struct { + TG int `json:"tg"` + Event string `json:"event"` + Callsign string `json:"callsign"` + Node string `json:"node"` + System string `json:"system"` + Via string `json:"via"` + Network string `json:"network"` // ingress mode (AllStar / DMR / DMR/YSF …) + } + body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 4096)) + if err != nil || json.Unmarshal(body, &in) != nil || in.TG <= 0 { + http.Error(w, `{"msg":"bad request"}`, http.StatusBadRequest) + return + } + w.Header().Set("Content-Type", "application/json; charset=utf-8") + if in.Event == "stop" { + // Intentionally no clear: sub-second re-keys would race it + // (same design as astmeta). The TTL ages the entry out. + _, _ = w.Write([]byte(`{"ok":true}`)) + return + } + cs := strings.TrimSpace(in.Callsign) + node := strings.TrimSpace(in.Node) + if (cs == "" || cs == "N0CALL" || isDigits(cs)) && isDigits(node) { + for _, k := range []string{ + "/registry/ext/dmr/db/users/id/" + node, + "/registry/ext/nxdn/db/users/id/" + node, + } { + if v, err := rdb.HGet(ctx, k, "callsign").Result(); err == nil && v != "" { + cs = v + break + } + } + } + if cs == "" { + http.Error(w, `{"msg":"no callsign"}`, http.StatusBadRequest) + return + } + m := talkerMeta{Callsign: cs, Node: node, + System: strings.TrimSpace(in.System), Via: strings.TrimSpace(in.Via), + Network: strings.TrimSpace(in.Network), At: time.Now()} + live.mu.Lock() + live.tgMeta[in.TG] = m + // Track the station for /api/allstar-nodes and prune expired entries. + live.asNodes[cs] = allstarNode{Callsign: cs, Node: node, + System: m.System, Via: m.Via, TG: in.TG, At: m.At} + for k, v := range live.asNodes { + if time.Since(v.At) > allstarNodesTTL { + delete(live.asNodes, k) + } + } + // Long-lived via->node map: which AllStar node a bridge login speaks + // through, so the web can offer a "connected nodes" button on that + // bridge row even when idle (no fresh talker meta). Derived, never + // hard-coded, and persisted so it survives a restart. + if m.Via != "" && node != "" { + via := strings.ToUpper(m.Via) + bi := bridgeInfo{Node: node, System: m.System} + if live.viaNode[via] != bi { + live.viaNode[via] = bi + if rdb != nil { + rdb.HSet(ctx, "nodelist:vianode", via, node+"|"+m.System) + } + } + } + live.mu.Unlock() + if debug { + log.Printf("DEBUG ingest talker meta tg=%d callsign=%q node=%q via=%q", + in.TG, m.Callsign, m.Node, m.Via) + } + _, _ = w.Write([]byte(`{"ok":true}`)) + }) + + // The bridge-heard stations as a reflector-status style node list: + // every station seen via the talker-meta ingest within + // ALLSTAR_NODES_TTL, keyed by callsign. Location comes from the + // radioid tables in redis when the station's id is known. + mux.HandleFunc("/api/allstar-nodes", func(w http.ResponseWriter, r *http.Request) { + type nodeOut struct { + Hidden bool `json:"hidden"` + IsTalker bool `json:"isTalker"` + MonitoredTGs []int `json:"monitoredTGs"` + NodeClass string `json:"nodeClass"` + NodeLocation string `json:"nodeLocation"` + RestrictedTG bool `json:"restrictedTG"` + SW string `json:"sw"` + Sysop string `json:"sysop"` + TG int `json:"tg"` + Node string `json:"node,omitempty"` + Via string `json:"via,omitempty"` + LastHeard int64 `json:"last_heard"` + } + out := map[string]nodeOut{} + now := time.Now() + live.mu.RLock() + for cs, n := range live.asNodes { + if now.Sub(n.At) > allstarNodesTTL { + continue + } + m, ok := live.tgMeta[n.TG] + isTalker := ok && m.Callsign == cs && now.Sub(m.At) <= talkerMetaTTL + loc := "via " + n.Via + if n.Via == "" { + loc = n.System + } + out[cs] = nodeOut{IsTalker: isTalker, MonitoredTGs: []int{n.TG}, + NodeClass: "AllStar", NodeLocation: loc, SW: "AllStarLink", + Sysop: cs, TG: n.TG, Node: n.Node, Via: n.Via, + LastHeard: n.At.Unix()} + } + live.mu.RUnlock() + // Location from the radioid tables — outside the lock (redis I/O). + for cs, n := range out { + if !isDigits(n.Node) { + continue + } + for _, k := range []string{ + "/registry/ext/dmr/db/users/id/" + n.Node, + "/registry/ext/nxdn/db/users/id/" + n.Node, + } { + vals, err := rdb.HMGet(ctx, k, "city", "state", "country").Result() + if err != nil { + continue + } + parts := []string{} + for _, v := range vals { + if s, ok := v.(string); ok && s != "" { + parts = append(parts, s) + } + } + if len(parts) > 0 { + n.NodeLocation = strings.Join(parts, ", ") + out[cs] = n + break + } + } + } + resp := map[string]any{ + "mode": "bridge", + "ttl_s": int(allstarNodesTTL.Seconds()), + "nodes": out, + } + w.Header().Set("Content-Type", "application/json; charset=utf-8") + _ = json.NewEncoder(w).Encode(resp) + }) + + // /api/remote-nodes?node= — ADDITIVE read-only endpoint: the nodes + // currently linked to an AllStar node, from stats.allstarlink.org, reshaped + // to {node, callsign, location, keyed}. Server-side fetch, short cache. Does + // not touch any existing response shape, so all current readers are safe. + mux.HandleFunc("/api/remote-nodes", func(w http.ResponseWriter, r *http.Request) { + node := strings.TrimSpace(r.URL.Query().Get("node")) + if !isDigits(node) { + http.Error(w, `{"msg":"node query param (digits) required"}`, http.StatusBadRequest) + return + } + resp, err := remoteNodes(node) + if err != nil { + http.Error(w, `{"msg":"upstream fetch failed"}`, http.StatusBadGateway) + return + } + w.Header().Set("Content-Type", "application/json; charset=utf-8") + _ = json.NewEncoder(w).Encode(resp) + }) + + // All-Nodes flat layer: /api/nodes (full) + /api/nodes-stream (per-node deltas). + nodeFlatStart(mux, rdb) + + // Monitor Nodes (single-endpoint): /api/monitor-nodes — Local + per-trunk. + monitorNodesStart(mux) + + // Monitor Trunks: /api/local-trunks — ONLY the trunks this reflector is + // configured to connect to, read from the local svxreflector.conf. + localTrunksStart(mux) + + srv := &http.Server{ + Addr: listenAddr, + Handler: securityHeaders(mux), + ReadHeaderTimeout: 5 * time.Second, + } + + log.Printf("nodelist listen=%s reflector=%s poll=%s redis=unix:%s statusKey=%s nodePrefix=%s onlineTTL=%s debug=%v", + listenAddr, reflectorURL, pollInterval, redisSock, statusKey, nodeKeyPrefix, effectiveOnlineTTL(), debug) + + go func() { + <-rctx.Done() + sc, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + _ = srv.Shutdown(sc) + }() + if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed { + return err + } + return nil +} + +func tOrEmpty(t time.Time) string { + if t.IsZero() { + return "" + } + return t.Format(time.RFC3339) +} + +func effectiveOnlineTTL() time.Duration { + onlineTTL := nodeOnlineTTL + if onlineTTL <= 0 { + onlineTTL = nodeTTL + } + if onlineTTL <= 0 { + onlineTTL = maxDur(15*time.Second, 3*pollInterval) + } + return onlineTTL +} + +// ---- /api/remote-nodes: AllStarLink stats proxy (additive, read-only) -------- + +var ( + remoteHTTP = &http.Client{Timeout: 8 * time.Second} + remoteMu sync.Mutex + remoteCache = map[string]remoteEntry{} +) + +const remoteTTL = 8 * time.Second + +type remoteEntry struct { + at time.Time + resp *remoteResp +} + +type remoteNode struct { + Node string `json:"node"` + Callsign string `json:"callsign"` + Location string `json:"location"` + Keyed bool `json:"keyed"` +} + +type remoteResp struct { + Node string `json:"node"` // the AllStar node we queried + Count int `json:"count"` + Nodes []remoteNode `json:"nodes"` + FetchedAt int64 `json:"fetched_at"` +} + +// asStats mirrors only the fields we consume from stats.allstarlink.org. +// The connected nodes are node-number strings under stats.data.links; this +// endpoint carries no callsign/location (resolved separately, TODO astdb). +type asStats struct { + Stats struct { + Data struct { + Links []string `json:"links"` + } `json:"data"` + } `json:"stats"` +} + +// remoteNodes fetches (and briefly caches) the linked-node list for an AllStar +// node from the public stats API, reshaped for the UI. +func remoteNodes(node string) (*remoteResp, error) { + remoteMu.Lock() + if e, ok := remoteCache[node]; ok && time.Since(e.at) < remoteTTL { + remoteMu.Unlock() + return e.resp, nil + } + remoteMu.Unlock() + + req, err := http.NewRequest(http.MethodGet, "https://stats.allstarlink.org/api/stats/"+node, nil) + if err != nil { + return nil, err + } + req.Header.Set("Accept", "application/json") + hr, err := remoteHTTP.Do(req) + if err != nil { + return nil, err + } + defer hr.Body.Close() + if hr.StatusCode >= 300 { + return nil, fmt.Errorf("stats HTTP %d", hr.StatusCode) + } + var s asStats + if err := json.NewDecoder(io.LimitReader(hr.Body, 8<<20)).Decode(&s); err != nil { + return nil, err + } + + ensureAstdb() // background-loads the node->callsign db if stale (non-blocking) + resp := &remoteResp{Node: node, FetchedAt: time.Now().Unix()} + for _, ln := range s.Stats.Data.Links { + n := strings.TrimSpace(ln) + if n == "" { + continue + } + cs, loc := astdbLookup(n) + resp.Nodes = append(resp.Nodes, remoteNode{Node: n, Callsign: cs, Location: loc}) + } + resp.Count = len(resp.Nodes) + + remoteMu.Lock() + remoteCache[node] = remoteEntry{at: time.Now(), resp: resp} + remoteMu.Unlock() + return resp, nil +} + +// ---- astdb: AllStar node -> callsign/location (background-loaded, cached) ---- + +var ( + astdbURL = env("ASTDB_URL", "http://allmondb.allstarlink.org/") // node|call|desc|loc + astdbLine = regexp.MustCompile(`^([0-9]+)\|([^|]*)\|([^|]*)\|(.*)$`) + astdbMu sync.RWMutex + astdbMap = map[string][2]string{} // node -> {callsign, location} + astdbAt time.Time + astdbBusy bool +) + +const astdbTTL = 6 * time.Hour + +// ensureAstdb starts a background reload when the cache is stale. Non-blocking: +// the caller uses whatever is currently loaded (empty until the first load +// finishes), so a request never waits on the (large) allmondb download. +func ensureAstdb() { + astdbMu.Lock() + stale := time.Since(astdbAt) >= astdbTTL + if astdbBusy || !stale { + astdbMu.Unlock() + return + } + astdbBusy = true + astdbMu.Unlock() + go loadAstdb() +} + +func loadAstdb() { + defer func() { astdbMu.Lock(); astdbBusy = false; astdbMu.Unlock() }() + if astdbURL == "" { + return + } + hr, err := remoteHTTP.Get(astdbURL) + if err != nil { + log.Printf("astdb: fetch failed: %v", err) + return + } + defer hr.Body.Close() + m := map[string][2]string{} + sc := bufio.NewScanner(io.LimitReader(hr.Body, 64<<20)) + sc.Buffer(make([]byte, 1<<20), 1<<20) + for sc.Scan() { + if f := astdbLine.FindStringSubmatch(sc.Text()); f != nil { + m[f[1]] = [2]string{strings.TrimSpace(f[2]), strings.TrimSpace(f[4])} + } + } + if len(m) > 0 { + astdbMu.Lock() + astdbMap = m + astdbAt = time.Now() + astdbMu.Unlock() + log.Printf("astdb: loaded %d nodes", len(m)) + } +} + +func astdbLookup(node string) (callsign, location string) { + astdbMu.RLock() + defer astdbMu.RUnlock() + if v, ok := astdbMap[node]; ok { + return v[0], v[1] + } + return "", "" +} + +func maxDur(a, b time.Duration) time.Duration { + if a > b { + return a + } + return b +} + +func poller(rdb *redis.Client) { + defer func() { + if r := recover(); r != nil { + log.Printf("poller panic: %v (will NOT recover)", r) + os.Exit(2) + } + }() + + t := time.NewTicker(pollInterval) + defer t.Stop() + + onlineTTL := effectiveOnlineTTL() + + nodeOnlineKey := func(cs string) string { + return fmt.Sprintf("%s/%s/online", nodeKeyPrefix, cs) + } + + prevOnline := map[string]struct{}{} + if arr, err := rdb.SMembers(ctx, nodesOnlineKey).Result(); err == nil { + for _, cs := range arr { + prevOnline[cs] = struct{}{} + } + } + + var lastFetchErrLog time.Time + + for range t.C { + body, usedURL, err := fetchReflectorStatus(reflectorURL) + if err != nil { + st.mu.Lock() + st.LastFetchErr = err.Error() + st.mu.Unlock() + + if debug { + log.Printf("DEBUG fetch failed: %v (url=%s)", err, reflectorURL) + } + if time.Since(lastFetchErrLog) > 30*time.Second { + log.Printf("reflector fetch failed: %v (url=%s)", err, reflectorURL) + lastFetchErrLog = time.Now() + } + continue + } + + st.mu.Lock() + st.LastFetchOKAt = time.Now() + st.LastFetchURL = usedURL + st.LastFetchBytes = len(body) + st.LastFetchErr = "" + st.mu.Unlock() + + var env statusEnvelope + if err := json.Unmarshal(body, &env); err != nil { + st.mu.Lock() + st.LastFetchErr = "parse: " + err.Error() + st.mu.Unlock() + log.Printf("status json parse failed: %v (url=%s)", err, usedURL) + continue + } + if env.Nodes == nil { + env.Nodes = map[string]json.RawMessage{} + } + + st.mu.Lock() + st.LastParsedNodes = len(env.Nodes) + st.mu.Unlock() + + // Refresh the real-callsign lookup maps for the log tailer. + newTgTalker := map[int]string{} + newTgTalkerAt := map[int]int64{} + newTgLoc := map[int]string{} + newTgMode := map[int]string{} + crawlNow := time.Now().Unix() + for _, tr := range env.Trunks { + for tgs, cs := range tr.ActiveTalkers { + if cs == "" { + continue + } + if tg, e := strconv.Atoi(tgs); e == nil && tg > 0 { + newTgTalker[tg] = cs + newTgTalkerAt[tg] = crawlNow + } + } + for _, n := range tr.Nodes { + if n.IsTalker && n.Callsign != "" && n.TG > 0 { + newTgTalker[n.TG] = n.Callsign + newTgTalkerAt[n.TG] = crawlNow + } + } + } + // Satellite trunks (chan_svx_sat / chan_pbx_svx) carry their talkers in a + // separate section, not env.Trunks. The node's callsign is the real caller + // (SA7AUX); its gateway label rides in nodeLocation (SA7AUX-PBX) -> the + // Location column, like an AllStar node's node_info.json. + for _, sat := range env.Satellites { + for _, n := range sat.Nodes { + if !n.IsTalker || n.Callsign == "" || n.TG <= 0 { + continue + } + newTgTalker[n.TG] = n.Callsign + newTgTalkerAt[n.TG] = crawlNow + if n.NodeLocation != "" { + newTgLoc[n.TG] = n.NodeLocation + } + if m := monModeOf(n.SW); m != "" { + newTgMode[n.TG] = m + } + } + } + newNodeSysop := map[string]string{} + for cs, raw := range env.Nodes { + if sy := extractSysop(raw); sy != "" { + newNodeSysop[cs] = sy + } + } + // Sticky: a talker-start from the log tail can race ahead of this crawl, + // so a TG the crawl momentarily lost (talker still active, roster lagging) + // would drop back to the raw trunk name. Carry a very-recent value forward + // (short 15s window) so resolveTalker keeps the real callsign across the gap. + const stickyWindow = int64(15) + live.mu.Lock() + for tg, cs := range live.tgTalker { + if _, ok := newTgTalker[tg]; ok { + continue + } + if at := live.tgTalkerAt[tg]; at > 0 && crawlNow-at <= stickyWindow { + newTgTalker[tg] = cs + newTgTalkerAt[tg] = at + } + } + live.tgTalker = newTgTalker + live.tgTalkerAt = newTgTalkerAt + live.tgLoc = newTgLoc + live.tgMode = newTgMode + live.nodeSysop = newNodeSysop + live.mu.Unlock() + + nowOnline := map[string]struct{}{} + for cs := range env.Nodes { + nowOnline[cs] = struct{}{} + } + added, removed := diffSets(prevOnline, nowOnline) + + pipe := rdb.Pipeline() + + pipe.Set(ctx, heartbeatKey, fmt.Sprint(time.Now().Unix()), 5*time.Second) + + // Merge MQTT-sourced trunk nodes into the served status so the + // dashboard's node views show trunk-side nodes + their isTalker state. + storeBody := body + if merged := mergeMqttNodes(env.Nodes); len(merged) > len(env.Nodes) { + var top map[string]json.RawMessage + if json.Unmarshal(body, &top) == nil { + if nb, err := json.Marshal(merged); err == nil { + top["nodes"] = nb + if bb, err := json.Marshal(top); err == nil { + storeBody = bb + } + } + } + } + + // Inject the network-wide picture (ext_sources + network_trunks) so the + // Monitor views can show every reflector's nodes and all trunks. + storeBody = injectExternal(storeBody) + + // Fresh bridge talker meta for the "Talking now" strip. + storeBody = injectTalkerMeta(storeBody) + + h := hash(storeBody) + pipe.Set(ctx, statusKey, storeBody, 0) + pipe.Set(ctx, statusHashKey, h, 0) + pipe.Publish(ctx, pubsubChan, "1") + + st.mu.Lock() + st.LastStatusHash = h + st.mu.Unlock() + + if len(added) > 0 { + members := make([]interface{}, 0, len(added)) + for _, cs := range added { + members = append(members, cs) + } + pipe.SAdd(ctx, nodesOnlineKey, members...) + } + if len(removed) > 0 { + members := make([]interface{}, 0, len(removed)) + for _, cs := range removed { + members = append(members, cs) + } + pipe.SRem(ctx, nodesOnlineKey, members...) + } + + for cs, raw := range env.Nodes { + pipe.Set(ctx, nodeOnlineKey(cs), "1", onlineTTL) + + nh := hash(raw) + pipe.Set(ctx, fmt.Sprintf("%s/%s/hash", nodeKeyPrefix, cs), nh, 0) + + // IMPORTANT: go-redis can't marshal json.RawMessage directly; cast to []byte. + pipe.Set(ctx, fmt.Sprintf("%s/%s/json", nodeKeyPrefix, cs), []byte(raw), 0) + + rx, tx := extractRxTx(raw) + pipe.Set(ctx, fmt.Sprintf("%s/%s/rx", nodeKeyPrefix, cs), fmt.Sprintf("%.5f", rx), 0) + pipe.Set(ctx, fmt.Sprintf("%s/%s/tx", nodeKeyPrefix, cs), fmt.Sprintf("%.5f", tx), 0) + + tones := extractToneToTG(raw) + if tones != nil && len(tones) > 0 { + b, _ := json.Marshal(tones) + pipe.Set(ctx, fmt.Sprintf("%s/%s/toneToTalkgroup", nodeKeyPrefix, cs), b, 0) + } else { + pipe.Del(ctx, fmt.Sprintf("%s/%s/toneToTalkgroup", nodeKeyPrefix, cs)) + } + } + + for _, cs := range added { + pipe.Publish(ctx, nodeConnectChan, cs) + pipe.Publish(ctx, nodeUpdateChan, "connect:"+cs) + } + for _, cs := range removed { + pipe.Publish(ctx, nodeDisconnectChan, cs) + pipe.Publish(ctx, nodeUpdateChan, "disconnect:"+cs) + pipe.Del(ctx, nodeOnlineKey(cs)) + } + + // lastheard events come from the reflector-log tailer (logTailer goroutine) + + if _, execErr := pipe.Exec(ctx); execErr != nil { + st.mu.Lock() + st.LastRedisErr = execErr.Error() + st.mu.Unlock() + log.Printf("redis exec failed: %v", execErr) + continue + } + + st.mu.Lock() + st.LastRedisOKAt = time.Now() + st.LastRedisErr = "" + st.mu.Unlock() + + if debug { + log.Printf("DEBUG wrote: nodes=%d added=%d removed=%d", len(env.Nodes), len(added), len(removed)) + } + + prevOnline = nowOnline + } +} + +func extractRxTx(raw json.RawMessage) (rx float64, tx float64) { + var ni nodeInfo + if err := json.Unmarshal(raw, &ni); err != nil { + return 0, 0 + } + + if ni.RxFreq > 0 { + rx = ni.RxFreq + } else if ni.RX > 0 { + rx = ni.RX + } + if ni.TxFreq > 0 { + tx = ni.TxFreq + } else if ni.TX > 0 { + tx = ni.TX + } + + if (rx <= 0 || tx <= 0) && len(ni.QTH) > 0 { + q := ni.QTH[0] + if rx <= 0 { + for _, v := range q.RX { + if v.Freq > 0 { + rx = v.Freq + break + } + } + } + if tx <= 0 { + for _, v := range q.TX { + if v.Freq > 0 { + tx = v.Freq + break + } + } + } + } + + if rx > 0 && tx <= 0 { + tx = rx + } + return +} + +func extractToneToTG(raw json.RawMessage) map[string]int { + var ni nodeInfo + if err := json.Unmarshal(raw, &ni); err != nil { + return nil + } + return ni.ToneToTalkgroup +} + +func extractSysop(raw json.RawMessage) string { + var ni nodeInfo + if err := json.Unmarshal(raw, &ni); err != nil { + return "" + } + return strings.TrimSpace(ni.Sysop) +} + +var ( + // Matches both a local/node talker ("... KT4ROY-A: Talker start on TG #N") + // and a trunk/satellite-delivered one ("... [trunk] SA7AUX: Trunk talker + // start on TG #N"). The optional "Trunk " group tags the source. + talkerLineRe = regexp.MustCompile(`^(.+?): (Trunk )?[Tt]alker (start|stop) on TG #(\d+)\s*$`) + logDateRe = regexp.MustCompile(`^(\w{3}\s+\d{1,2}\s+\w{3}\s+\d{4}\s+\d{2}:\d{2}:\d{2}\s+(?:AM|PM))\s+[A-Z]{2,5}$`) +) + +type talkerLogEvent struct { + Ts int64 + Callsign string + TG int + IsStop bool + IsTrunk bool +} + +func parseTalkerLogLine(line string) (ev talkerLogEvent, ok bool) { + m := talkerLineRe.FindStringSubmatch(line) + if m == nil { + return + } + prefix := m[1] + + idx := strings.LastIndex(prefix, ": ") + if idx < 0 { + return + } + datePart := prefix[:idx] + tail := prefix[idx+2:] + + ev.Ts = parseReflectorDate(datePart) + if ev.Ts == 0 { + ev.Ts = time.Now().Unix() + } + ev.IsTrunk = m[2] != "" || strings.Contains(tail, "-> Trunk") + fields := strings.Fields(tail) + // Newer reflector builds prefix log lines with severity/component tags + // ("[info] [client] KT4ROY-AS: Talker start ..."); skip any such tags so + // the callsign is found in both the old and the new log format. + for len(fields) > 0 && strings.HasPrefix(fields[0], "[") && strings.HasSuffix(fields[0], "]") { + fields = fields[1:] + } + if len(fields) == 0 { + return + } + ev.Callsign = strings.TrimRight(fields[0], ":,") + ev.IsStop = m[3] == "stop" + tg, err := strconv.Atoi(m[4]) + if err != nil { + return + } + ev.TG = tg + ok = true + return +} + +func parseReflectorDate(s string) int64 { + s = strings.TrimSpace(s) + m := logDateRe.FindStringSubmatch(s) + if m == nil { + t, err := time.ParseInLocation("Mon 2 Jan 2006 03:04:05 PM MST", s, time.Local) + if err == nil { + return t.Unix() + } + return 0 + } + t, err := time.ParseInLocation("Mon 2 Jan 2006 03:04:05 PM", m[1], time.Local) + if err != nil { + return 0 + } + return t.Unix() +} + +type liveTalker struct { + StartTs int64 + TG int + IsTrunk bool + Callsign string // resolved real callsign, captured at talker-start + Trunk string // node/trunk name when it differs from the callsign + Node string // source-network node/id (AllStar node, DMR id), from meta + System string // source system name (e.g. "East Coast Reflector"), from meta + Via string // reflector callsign the bridge speaks through, from meta + Network string // ingress mode: AllStar / SVXLink / trunk / DMR / DMR/YSF … + LogName string // raw talker name from the reflector log (for via-match) +} + +// lastheardRenderCfg controls how bridge/AllStar rows render in lastheard, +// editable at runtime via a JSON file (hot-reloaded) — NO recompile, nothing +// hardcoded. Path: LASTHEARD_RENDER_JSON env, default /etc/nodelist/lastheard.json. +// {system}/{node}/{talker} in the format strings are filled dynamically. +// reflector_names gives a friendly System name to reflectors astmeta doesn't +// name (e.g. "ASL-dodropin" -> "Do Drop In"), keyed by the bridge/log name. +type lastheardRenderCfg struct { + BridgeLocationFormat string `json:"bridge_location_format"` + BridgeLocationFormatNoSystem string `json:"bridge_location_format_no_system"` + ReflectorNames map[string]string `json:"reflector_names"` +} + +var defaultRenderCfg = lastheardRenderCfg{ + BridgeLocationFormat: "{system} via {node} / {talker}", + BridgeLocationFormatNoSystem: "via {node} / {talker}", + ReflectorNames: map[string]string{}, +} + +var ( + renderCfgMu sync.RWMutex + renderCfg = defaultRenderCfg +) + +func renderConfig() lastheardRenderCfg { + renderCfgMu.RLock() + defer renderCfgMu.RUnlock() + return renderCfg +} + +// loadRenderConfig merges the JSON file over the built-in defaults, so the +// binary works with no file and a partial file only overrides what it sets. +func loadRenderConfig(path string) { + cfg := defaultRenderCfg + cfg.ReflectorNames = map[string]string{} + if b, err := os.ReadFile(path); err == nil { + var fc lastheardRenderCfg + if jerr := json.Unmarshal(b, &fc); jerr != nil { + log.Printf("lastheard render json %s: %v (keeping current)", path, jerr) + return + } + if strings.TrimSpace(fc.BridgeLocationFormat) != "" { + cfg.BridgeLocationFormat = fc.BridgeLocationFormat + } + if strings.TrimSpace(fc.BridgeLocationFormatNoSystem) != "" { + cfg.BridgeLocationFormatNoSystem = fc.BridgeLocationFormatNoSystem + } + for k, v := range fc.ReflectorNames { + cfg.ReflectorNames[k] = v + } + } + // nodes.json (shared bridge map) merged on top: it is the source of truth + // for reflector names + Location formats. Absent file = no-op. + applyNodesRenderConfig(&cfg, nodesJSONPath()) + renderCfgMu.Lock() + renderCfg = cfg + renderCfgMu.Unlock() +} + +// watchRenderConfig hot-reloads whenever lastheard.json OR nodes.json changes, +// so editing the shared bridge map updates the Location column with no restart. +func watchRenderConfig(path string) { + var last, lastNodes time.Time + for { + reload := false + if fi, err := os.Stat(path); err == nil && fi.ModTime() != last { + last = fi.ModTime() + reload = true + } + if fi, err := os.Stat(nodesJSONPath()); err == nil && fi.ModTime() != lastNodes { + lastNodes = fi.ModTime() + reload = true + } + if reload { + loadRenderConfig(path) + } + time.Sleep(3 * time.Second) + } +} + +// bridgeIdentity splits a talker into the (Callsign, Location) shown in +// lastheard, uniform for PBX and AllStar (approved layout). PBX / plain svxlink: +// Callsign = lt.Callsign (sip-tg / real callsign), Location = locForTG +// (SA7AUX-PBX / blank). AllStar/bridge (meta System known): the BRIDGE +// (lt.Via, e.g. KT4ROY-AS) is the Callsign, and Location carries +// " via / " (e.g. "East Coast Reflector via 45225 / W2ECR"). +// The AllStar node number is dynamic from astmeta (lt.Node), never hardcoded. +func bridgeIdentity(lt liveTalker, tg int) (callsign, location string) { + callsign = lt.Callsign + location = locForTG(tg) + bridge := lt.Via + if bridge == "" { + bridge = lt.Trunk + } + // A bridge/AllStar talker has a bridge node (Via/Trunk) AND meta (Node or + // System) — the reflector name may be absent (e.g. ASL-dodropin). Plain + // svxlink / PBX carry no bridge node -> untouched. + if bridge == "" || (lt.Node == "" && lt.System == "") { + return callsign, location + } + // FROZEN DESIGN: Location is the tglist name ONLY (already set above via + // locForTG). We no longer build " via / " from live + // astmeta — the tglist is the sole source, so the row reads identically on + // every box. The bridge is still the Callsign (who spoke). + callsign = bridge + return callsign, location +} + +func writeLastheardEvent(rdb *redis.Client, ev lastheardEvent) { + b, err := json.Marshal(ev) + if err != nil { + return + } + pipe := rdb.Pipeline() + pipe.ZAdd(ctx, lastheardKey, redis.Z{Score: float64(ev.Ts), Member: string(b)}) + cutoff := time.Now().Unix() - lastheardWindowSec + pipe.ZRemRangeByScore(ctx, lastheardKey, "-inf", fmt.Sprintf("(%d", cutoff)) + if _, err := pipe.Exec(ctx); err != nil { + log.Printf("lastheard write failed: %v", err) + } +} + +func sourceLabel(isTrunk bool) string { + if isTrunk { + return "trunk" + } + return "local" +} + +// defaultNetwork is the ingress mode when no talker meta enriches the row: +// a trunk-delivered talker enters over a reflector↔reflector trunk, everything +// else is a plain svxlink client. A bridge (AllStar via astmeta, later DMR via +// callmgr) overrides this with its own network string from the talker meta. +func defaultNetwork(isTrunk bool) string { + if isTrunk { + return "trunk" + } + return "SVXLink" +} + +func logTailer(rdb *redis.Client, path string) { + defer func() { + if r := recover(); r != nil { + log.Printf("logTailer panic: %v", r) + } + }() + + active := map[string]liveTalker{} // key: callsign|tg + keyOf := func(cs string, tg int) string { return fmt.Sprintf("%s|%d", cs, tg) } + + mkLive := func(ev talkerLogEvent) liveTalker { + cs, tn := resolveTalker(ev.Callsign, ev.TG, ev.IsTrunk) + lt := liveTalker{StartTs: ev.Ts, TG: ev.TG, IsTrunk: ev.IsTrunk, + Callsign: cs, Trunk: tn, LogName: ev.Callsign, + Network: defaultNetwork(ev.IsTrunk)} + // Meta applies regardless of source (locally bridged traffic here, + // trunk-delivered on the partner reflector) — gated by via-match. + if m, ok := talkerMetaFor(ev.TG, ev.Callsign); ok { + lt.Callsign, lt.Node, lt.System, lt.Via = m.Callsign, m.Node, m.System, m.Via + if m.Network != "" { + lt.Network = m.Network + } + if lt.Trunk == "" && ev.Callsign != m.Callsign { + lt.Trunk = ev.Callsign + } + } + // Gateway-node Backend (PBX etc.) wins over the generic "trunk" ingress. + if md := modeForTG(ev.TG); md != "" { + lt.Network = md + } + return lt + } + + emit := func(start liveTalker, tg int, stopTs int64) { + // The meta often arrives just after talker-start (astmeta polls AMI + // about once a second), so short keyups miss it in mkLive. Re-check + // at stop — the meta is still fresh at this point — so those rows + // get the real source identity instead of the sysop fallback. + if start.Node == "" { + if m, ok := talkerMetaFor(tg, start.LogName); ok { + if start.Trunk == "" && start.Callsign != m.Callsign { + start.Trunk = start.Callsign + } + start.Callsign, start.Node, start.System, start.Via = m.Callsign, m.Node, m.System, m.Via + if m.Network != "" { + start.Network = m.Network + } + } + } + dur := stopTs - start.StartTs + if dur < 1 { + dur = 1 + } + // Re-apply the gateway Backend at stop: tgMode may not have been populated + // yet when mkLive ran at talker-start (crawl lag on short keyups). + if md := modeForTG(tg); md != "" { + start.Network = md + } + // Location column, symmetric for PBX and AllStar: the gateway/reflector + // descriptor lives here, the station stays in Callsign. PBX gets its + // gateway nodeLocation via locForTG (e.g. SA7AUX-PBX); a bridge/AllStar + // talker (no TG->location map) gets " via " (e.g. + // "East Coast Reflector via KT4ROY-AS") so it renders like the PBX rows. + // Callsign + Location split (uniform PBX/AllStar). See bridgeIdentity. + csOut, loc := bridgeIdentity(start, tg) + writeLastheardEvent(rdb, lastheardEvent{ + Ts: start.StartTs, + Callsign: csOut, + Trunk: start.Trunk, + TG: tg, + Duration: dur, + Location: loc, + Source: sourceLabel(start.IsTrunk), + Node: start.Node, + System: start.System, + Network: start.Network, + }) + } + + // 1) Backfill: scan whole file, replay events into ZSET (last 24h only). + now0 := time.Now().Unix() + // Timestamps already persisted in the lastheard set, so a restart's backfill + // does not re-emit (duplicate) rows the previous run already wrote live — a + // live row may be enriched ("X via X-PBX") while the raw backfill cannot + // enrich (tgTalker is empty for historical events), producing a second row. + existing := map[int64]bool{} + if zs, zerr := rdb.ZRangeWithScores(ctx, lastheardKey, 0, -1).Result(); zerr == nil { + for _, z := range zs { + existing[int64(z.Score)] = true + } + } + if f, err := os.Open(path); err == nil { + sc := bufio.NewScanner(f) + sc.Buffer(make([]byte, 64*1024), 1024*1024) + for sc.Scan() { + ev, ok := parseTalkerLogLine(sc.Text()) + if !ok { + continue + } + if ev.Ts < now0-lastheardWindowSec { + if !ev.IsStop { + continue + } + delete(active, keyOf(ev.Callsign, ev.TG)) + continue + } + k := keyOf(ev.Callsign, ev.TG) + if !ev.IsStop { + active[k] = mkLive(ev) + } else if st, was := active[k]; was { + if !existing[st.StartTs] { // already persisted by a previous run → don't duplicate + emit(st, ev.TG, ev.Ts) + } + delete(active, k) + } + } + f.Close() + log.Printf("logTailer: backfill done, %d in-flight talkers", len(active)) + } else { + log.Printf("logTailer: cannot open %s for backfill: %v", path, err) + } + + // 2) Tail mode: follow new lines, handle rotation. + for { + f, err := os.Open(path) + if err != nil { + time.Sleep(2 * time.Second) + continue + } + if _, err := f.Seek(0, io.SeekEnd); err != nil { + f.Close() + time.Sleep(2 * time.Second) + continue + } + + fi0, _ := f.Stat() + reader := bufio.NewReader(f) + + for { + line, err := reader.ReadString('\n') + if err == nil { + line = strings.TrimRight(line, "\r\n") + if ev, ok := parseTalkerLogLine(line); ok { + k := keyOf(ev.Callsign, ev.TG) + if !ev.IsStop { + lt := mkLive(ev) + active[k] = lt + // Live only (not backfill): announce the SVX talker to the mesh. + xlinkPublishTalker(lt.Callsign, lt.System, lt.Trunk, lt.Node, ev.TG, lt.Network, "start", ev.Ts) + } else if st, was := active[k]; was { + xlinkPublishTalker(st.Callsign, st.System, st.Trunk, st.Node, ev.TG, st.Network, "stop", ev.Ts) + emit(st, ev.TG, ev.Ts) + delete(active, k) + } else { + lt := mkLive(ev) + cs3, loc3 := bridgeIdentity(lt, ev.TG) + writeLastheardEvent(rdb, lastheardEvent{ + Ts: ev.Ts, + Callsign: cs3, + Trunk: lt.Trunk, + TG: ev.TG, + Duration: 1, + Location: loc3, + Source: sourceLabel(ev.IsTrunk), + Node: lt.Node, + System: lt.System, + Network: lt.Network, + }) + } + } + continue + } + if err == io.EOF { + time.Sleep(300 * time.Millisecond) + fi1, sErr := os.Stat(path) + if sErr == nil && fi0 != nil { + if !os.SameFile(fi0, fi1) || fi1.Size() < fi0.Size() { + f.Close() + break + } + } + continue + } + log.Printf("logTailer read error: %v", err) + f.Close() + break + } + time.Sleep(500 * time.Millisecond) + } +} + +func extractTalkerInfo(raw json.RawMessage) (talking bool, tg int, location, rxName string, siglev float64) { + var ni nodeInfo + if err := json.Unmarshal(raw, &ni); err != nil { + return + } + talking = ni.IsTalker + tg = ni.TG + location = ni.NodeLocation + + if len(ni.QTH) > 0 { + for _, q := range ni.QTH { + if location == "" && q.Name != "" { + location = q.Name + } + for _, rx := range q.RX { + if rx.Active { + if rx.Name != "" { + rxName = rx.Name + if q.Name != "" { + rxName = rx.Name + " " + q.Name + } + } + siglev = rx.Siglev + return + } + } + } + } + return +} + +func diffSets(prev, now map[string]struct{}) (added []string, removed []string) { + for k := range now { + if _, ok := prev[k]; !ok { + added = append(added, k) + } + } + for k := range prev { + if _, ok := now[k]; !ok { + removed = append(removed, k) + } + } + sort.Strings(added) + sort.Strings(removed) + return +} + +func fetchReflectorStatus(url string) ([]byte, string, error) { + try := func(u string) ([]byte, string, error) { + b, err := fetch(u) + if err != nil { + return nil, u, err + } + if !json.Valid(b) { + return nil, u, fmt.Errorf("not json (http 200)") + } + var m map[string]json.RawMessage + if err := json.Unmarshal(b, &m); err != nil { + return nil, u, err + } + if _, ok := m["nodes"]; !ok { + return nil, u, fmt.Errorf("missing top-level 'nodes'") + } + return b, u, nil + } + + body, used, err := try(url) + if err == nil { + return body, used, nil + } + + alt := altStatusURL(url) + if alt == "" || alt == url { + return nil, used, err + } + return try(alt) +} + +func altStatusURL(url string) string { + if strings.HasSuffix(url, "/status") { + return url + "/json" + } + if strings.HasSuffix(url, "/status/json") { + return strings.TrimSuffix(url, "/json") + } + if strings.HasSuffix(url, "/status/") { + return strings.TrimSuffix(url, "/") + "/json" + } + if strings.HasSuffix(url, "/status/json/") { + alt := strings.TrimSuffix(url, "/") + alt = strings.TrimSuffix(alt, "/json") + return alt + } + return "" +} + +func fetch(url string) ([]byte, error) { + c := &http.Client{Timeout: 2 * time.Second} + resp, err := c.Get(url) + if err != nil { + return nil, err + } + defer resp.Body.Close() + + if resp.StatusCode != 200 { + _, _ = io.Copy(io.Discard, resp.Body) + return nil, fmt.Errorf("http %d", resp.StatusCode) + } + return io.ReadAll(resp.Body) +} + +func hash(b []byte) string { + sum := sha256.Sum256(b) + return hex.EncodeToString(sum[:]) +} + +func securityHeaders(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("X-Content-Type-Options", "nosniff") + w.Header().Set("X-Frame-Options", "DENY") + w.Header().Set("Referrer-Policy", "no-referrer") + next.ServeHTTP(w, r) + }) +} + +func env(k, def string) string { + v := os.Getenv(k) + if v == "" { + return def + } + return v +} + +func envBool(k string, def bool) bool { + v := strings.TrimSpace(strings.ToLower(os.Getenv(k))) + if v == "" { + return def + } + switch v { + case "1", "true", "yes", "y", "on": + return true + case "0", "false", "no", "n", "off": + return false + default: + return def + } +} + +func envDur(k string, def time.Duration) time.Duration { + v := os.Getenv(k) + if v == "" { + return def + } + d, err := time.ParseDuration(v) + if err != nil { + return def + } + return d +} + +func envInt(k string, def int) int { + v := os.Getenv(k) + if v == "" { + return def + } + n, err := strconv.Atoi(strings.TrimSpace(v)) + if err != nil { + return def + } + return n +} diff --git a/svxdr/nodelist/nodes.go b/svxdr/nodelist/nodes.go new file mode 100644 index 0000000..a19273b --- /dev/null +++ b/svxdr/nodelist/nodes.go @@ -0,0 +1,83 @@ +// svxdr — live-data daemon for SvxPortal (nodelist + proxy + astmeta + xlink) +// Copyright (C) 2026 SA7AUX (Björn) +// SPDX-License-Identifier: AGPL-3.0-or-later +// +// This program is free software: you can redistribute it and/or modify it under +// the terms of the GNU Affero General Public License as published by the Free +// Software Foundation, either version 3 of the License, or (at your option) any +// later version. See the LICENSE file for details. + +package nodelist + +// nodes.json — shared bridge map (see svx-web/nodes.README.md). nodelist takes +// the friendly reflector names (System) and the Location format strings from it, +// so the Location column ("{system} via {node} / {talker}") is driven by the +// same one file that configures astmeta and Asterisk. Merged ON TOP of the +// existing lastheard.json, so a box with only lastheard.json is unchanged. + +import ( + "encoding/json" + "os" + "strconv" + "strings" +) + +type nodesFileNL struct { + Lastheard struct { + BridgeLocationFormat string `json:"bridge_location_format"` + BridgeLocationFormatNoSystem string `json:"bridge_location_format_no_system"` + } `json:"lastheard"` + Nodes []struct { + Allstar int `json:"allstar"` + System string `json:"system"` + Via string `json:"via"` + Callsign string `json:"callsign"` + Enabled *bool `json:"enabled"` + } `json:"nodes"` +} + +// applyNodesRenderConfig merges nodes.json's reflector names + Location formats +// over cfg. A missing/invalid file is a no-op (cfg keeps the lastheard.json / +// default values). ReflectorNames is keyed the way bridgeIdentity looks it up: +// by the bridge label (via/callsign) and, as a fallback, the far AllStar node. +func applyNodesRenderConfig(cfg *lastheardRenderCfg, path string) { + b, err := os.ReadFile(path) + if err != nil { + return + } + var nf nodesFileNL + if json.Unmarshal(b, &nf) != nil { + return + } + if s := strings.TrimSpace(nf.Lastheard.BridgeLocationFormat); s != "" { + cfg.BridgeLocationFormat = s + } + if s := strings.TrimSpace(nf.Lastheard.BridgeLocationFormatNoSystem); s != "" { + cfg.BridgeLocationFormatNoSystem = s + } + if cfg.ReflectorNames == nil { + cfg.ReflectorNames = map[string]string{} + } + for _, e := range nf.Nodes { + if (e.Enabled != nil && !*e.Enabled) || e.System == "" { + continue + } + if e.Via != "" { + cfg.ReflectorNames[e.Via] = e.System + } + if e.Callsign != "" { + cfg.ReflectorNames[e.Callsign] = e.System + } + if e.Allstar != 0 { + cfg.ReflectorNames[strconv.Itoa(e.Allstar)] = e.System + } + } +} + +// nodesJSONPath returns the nodes.json path (NODES_JSON env, default /etc/nodes.json). +func nodesJSONPath() string { + if p := os.Getenv("NODES_JSON"); p != "" { + return p + } + return "/etc/nodes.json" +} diff --git a/svxdr/nodelist/nodes_flat.go b/svxdr/nodelist/nodes_flat.go new file mode 100644 index 0000000..6ba953b --- /dev/null +++ b/svxdr/nodelist/nodes_flat.go @@ -0,0 +1,289 @@ +// svxdr — live-data daemon for SvxPortal (nodelist + proxy + astmeta + xlink) +// Copyright (C) 2026 SA7AUX (Björn) +// SPDX-License-Identifier: AGPL-3.0-or-later +// +// This program is free software: you can redistribute it and/or modify it under +// the terms of the GNU Affero General Public License as published by the Free +// Software Foundation, either version 3 of the License, or (at your option) any +// later version. See the LICENSE file for details. + +package nodelist + +// All-Nodes flat layer: one total, deduped node list across the whole crawled +// network (federation via externalState.sources). Each node is attributed to +// the REAL reflector it is connected to (source tag; this box's own host is +// shown as the reflector's real name, never an IP). Mesh dupes collapse to one +// owning ref. Instead of re-shipping the whole list, this holds state +// server-side and streams only per-node deltas (added/changed/removed) over SSE. + +import ( + "encoding/json" + "fmt" + "net" + "net/http" + "sort" + "strings" + "sync" + "time" + + "github.com/redis/go-redis/v9" +) + +// Federation aggregate (reflector-network.com/api/nodes): callsign -> node with +// the REAL reflector name. Populated by the gap loop; used here to (a) cover the +// whole network, not just what our own crawl reaches, and (b) give real ref +// names instead of a bare IP. UPPER(callsign) keyed. +var aggNodes = struct { + mu sync.RWMutex + m map[string]flatNode +}{m: map[string]flatNode{}} + +func setAggNodes(m map[string]flatNode) { + aggNodes.mu.Lock() + aggNodes.m = m + aggNodes.mu.Unlock() +} + +type flatNode struct { + Callsign string `json:"callsign"` + Ref string `json:"ref"` + TG string `json:"tg"` +} + +var flatState = struct { + mu sync.RWMutex + nodes map[string]flatNode // keyed by UPPER(callsign) + ver uint64 +}{nodes: map[string]flatNode{}} + +// --- SSE subscriber hub ---------------------------------------------------- +type flatSub struct{ ch chan []byte } + +var flatHub = struct { + mu sync.Mutex + subs map[*flatSub]struct{} +}{subs: map[*flatSub]struct{}{}} + +func flatSubscribe() *flatSub { + s := &flatSub{ch: make(chan []byte, 8)} + flatHub.mu.Lock() + flatHub.subs[s] = struct{}{} + flatHub.mu.Unlock() + return s +} +func flatUnsubscribe(s *flatSub) { + flatHub.mu.Lock() + delete(flatHub.subs, s) + flatHub.mu.Unlock() +} +func flatBroadcast(msg []byte) { + flatHub.mu.Lock() + for s := range flatHub.subs { + select { + case s.ch <- msg: + default: // slow client — drop; it will re-sync on next full snapshot/reconnect + } + } + flatHub.mu.Unlock() +} + +// tgOf extracts the "tg" (with "*" prefix for restrictedTG) from a node's JSON. +func tgOf(raw json.RawMessage) string { + var n struct { + TG json.Number `json:"tg"` + RestrictedTG bool `json:"restrictedTG"` + } + if json.Unmarshal(raw, &n) != nil { + return "" + } + s := n.TG.String() + if s == "" { + return "" + } + if n.RestrictedTG { + return "*" + s + } + return s +} + +var localRefName string // real name of THIS reflector (from svxreflector.conf), never an IP + +// buildFlat assembles the deduped map straight from the federation crawl. Each +// node is attributed to the REAL reflector it is connected to (source.Tag). +// First ref by tag wins (sources are tag-sorted → deterministic). A source +// whose tag is this box's own host is renamed to the reflector's real name so +// the ref column is never "127.0.0.1"/an IP. No local-status overlay — that mix +// carried trunk-side (remote) nodes and mis-tagged them as local. +func buildFlat() map[string]flatNode { + out := make(map[string]flatNode, 256) + selfHost := hostFromURL(reflectorURL) + externalState.mu.RLock() + for _, s := range externalState.sources { + ref := s.Tag + // Never expose a raw http(s)://host:port/status URL as the ref — those + // are FEDERATION_SEEDS whose tag is the seed URL. Show only the host. + if strings.HasPrefix(ref, "http://") || strings.HasPrefix(ref, "https://") { + ref = hostFromURL(ref) + } + if (hostFromURL(s.URL) == selfHost || hostFromURL(s.Tag) == selfHost) && localRefName != "" { + ref = localRefName + } + // Bare IP → prefer a hostname alias (a name resolving to the same IP). + if net.ParseIP(ref) != nil { + for _, a := range s.Aliases { + if a != "" && net.ParseIP(a) == nil { + ref = a + break + } + } + } + for cs, raw := range s.Nodes { + key := strings.ToUpper(cs) + if _, seen := out[key]; seen { + continue + } + out[key] = flatNode{Callsign: cs, Ref: ref, TG: tgOf(raw)} + } + } + externalState.mu.RUnlock() + + // Cover the WHOLE network (not just what our own crawl reaches): fold in the + // federation aggregate. Crawl wins for nodes we see live (fresh TG); the + // aggregate adds everyone else AND supplies a real reflector name where the + // crawl only had a bare IP. + aggNodes.mu.RLock() + for k, an := range aggNodes.m { + if ex, ok := out[k]; ok { + if an.Ref != "" && net.ParseIP(ex.Ref) != nil { + ex.Ref = an.Ref + out[k] = ex + } + continue + } + out[k] = flatNode{Callsign: an.Callsign, Ref: an.Ref} + } + aggNodes.mu.RUnlock() + return out +} + +// rebuildFlat rebuilds, diffs against the current state and — only when the set +// actually changed — bumps the version and streams the delta to SSE clients. +func rebuildFlat() { + next := buildFlat() + + flatState.mu.Lock() + prev := flatState.nodes + var added, changed []flatNode + var removed []string + for k, nv := range next { + if pv, ok := prev[k]; !ok { + added = append(added, nv) + } else if pv.Ref != nv.Ref || pv.TG != nv.TG { + changed = append(changed, nv) + } + } + for k, pv := range prev { + if _, ok := next[k]; !ok { + removed = append(removed, pv.Callsign) + } + } + if len(added) == 0 && len(changed) == 0 && len(removed) == 0 { + flatState.mu.Unlock() + return + } + flatState.nodes = next + flatState.ver++ + ver := flatState.ver + flatState.mu.Unlock() + + payload, _ := json.Marshal(struct { + Added []flatNode `json:"added"` + Changed []flatNode `json:"changed"` + Removed []string `json:"removed"` + Ver uint64 `json:"v"` + }{added, changed, removed, ver}) + flatBroadcast([]byte("event: nodes-delta\ndata: " + string(payload) + "\n\n")) +} + +// flatSnapshotJSON returns the full current list (for /api/nodes and the SSE +// initial frame), sorted by callsign. +func flatSnapshotJSON() ([]byte, uint64) { + flatState.mu.RLock() + rows := make([]flatNode, 0, len(flatState.nodes)) + for _, v := range flatState.nodes { + rows = append(rows, v) + } + ver := flatState.ver + flatState.mu.RUnlock() + sort.Slice(rows, func(i, j int) bool { return rows[i].Callsign < rows[j].Callsign }) + b, _ := json.Marshal(struct { + Nodes []flatNode `json:"nodes"` + Ver uint64 `json:"v"` + }{rows, ver}) + return b, ver +} + +// nodeFlatStart wires the endpoints and the update driver. Called from main() +// once rdb exists. The driver rebuilds on every pubsubChan tick (instant local +// changes) and on a slow safety ticker (catches federation crawl updates). +func nodeFlatStart(mux *http.ServeMux, rdb *redis.Client) { + localRefName, _ = localReflectorMeta() // real reflector name for the "local" tag + rebuildFlat() // prime + + mux.HandleFunc("/api/nodes", func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json; charset=utf-8") + b, _ := flatSnapshotJSON() + writeMaybeGzip(w, r, b) + }) + + mux.HandleFunc("/api/nodes-stream", func(w http.ResponseWriter, r *http.Request) { + flusher, ok := w.(http.Flusher) + if !ok { + http.Error(w, "SSE not supported", 500) + return + } + w.Header().Set("Content-Type", "text/event-stream; charset=utf-8") + w.Header().Set("Cache-Control", "no-cache") + w.Header().Set("Connection", "keep-alive") + + // initial full snapshot + snap, _ := flatSnapshotJSON() + fmt.Fprintf(w, "event: nodes-full\ndata: %s\n\n", snap) + flusher.Flush() + + sub := flatSubscribe() + defer flatUnsubscribe(sub) + ping := time.NewTicker(25 * time.Second) + defer ping.Stop() + + for { + select { + case <-r.Context().Done(): + return + case <-ping.C: + fmt.Fprint(w, "event: ping\ndata: 1\n\n") + flusher.Flush() + case msg := <-sub.ch: + w.Write(msg) + flusher.Flush() + } + } + }) + + // Driver: instant on pubsub, plus a safety ticker for crawl-side changes. + go func() { + sub := rdb.Subscribe(ctx, pubsubChan) + defer sub.Close() + ch := sub.Channel() + tick := time.NewTicker(3 * time.Second) + defer tick.Stop() + for { + select { + case <-ch: + rebuildFlat() + case <-tick.C: + rebuildFlat() + } + } + }() +} diff --git a/svxdr/nodelist/svxdrcfg.go b/svxdr/nodelist/svxdrcfg.go new file mode 100644 index 0000000..ce4835b --- /dev/null +++ b/svxdr/nodelist/svxdrcfg.go @@ -0,0 +1,366 @@ +// svxdr — live-data daemon for SvxPortal (nodelist + proxy + astmeta + xlink) +// Copyright (C) 2026 SA7AUX (Björn) +// SPDX-License-Identifier: AGPL-3.0-or-later +// +// This program is free software: you can redistribute it and/or modify it under +// the terms of the GNU Affero General Public License as published by the Free +// Software Foundation, either version 3 of the License, or (at your option) any +// later version. See the LICENSE file for details. + +package nodelist + +// svxdr config wiring. applyConfig overrides the package's env-derived globals +// with values from svxdr.json (a set field wins; an unset field keeps the +// built-in default). applyRenderFromConfig builds the lastheard render config +// (Location formats + reflector names) from svxdr.json instead of a separate +// lastheard.json / nodes.json. Nothing else in nodelist changed. + +import ( + "strconv" + "strings" + "sync" + "time" + + "svxportal/svxdr/config" +) + +// Per-TG expected via (from svxdr.json nodes[].via). talkerMetaFor matches the +// incoming meta against THIS instead of the reflector-log talker name, so the +// callsign/via is defined in ONE place (svxdr.json) and svx_sat.conf's callsign +// no longer has to equal it. +var ( + tgViaMu sync.RWMutex + groupTgVia = map[string]map[int]string{} // group login -> tg -> via (per group, so groups may share a TG) + tgViaAny = map[int]map[string]bool{} // tg -> set of every configured via (for talkerMetaFor) +) + +func setTgVia(gv map[string]map[int]string, any map[int]map[string]bool) { + tgViaMu.Lock() + groupTgVia, tgViaAny = gv, any + tgViaMu.Unlock() +} +func viaFor(login string, tg int) string { + tgViaMu.RLock() + defer tgViaMu.RUnlock() + if m := groupTgVia[login]; m != nil { + return m[tg] + } + return "" +} +func anyViaForTG(tg int) bool { + tgViaMu.RLock() + defer tgViaMu.RUnlock() + return len(tgViaAny[tg]) > 0 +} +func viaConfiguredForTG(tg int, via string) bool { + tgViaMu.RLock() + defer tgViaMu.RUnlock() + return tgViaAny[tg][via] +} + +// Per-node mode/type (from nodes[].mode), keyed by the node's display name (via or +// callsign). Config wins over the data-derived type; empty = fall back to data. +var ( + modeByNameMu sync.RWMutex + modeByNameMap = map[string]string{} +) + +func setModeByName(m map[string]string) { modeByNameMu.Lock(); modeByNameMap = m; modeByNameMu.Unlock() } +func modeForName(name string) string { + modeByNameMu.RLock() + defer modeByNameMu.RUnlock() + return modeByNameMap[name] +} + +// Per-TG reflector/system name (from nodes[].system), so ANY talker on that TG — +// AllStar, DMR or a bare reflector client — falls back to the same Location label +// (e.g. TG 314093 -> "WHITELIGHTNING"). Used by locForTG when there is no live +// gateway location for the TG. +var ( + tgSystemMu sync.RWMutex + tgSystemMap = map[int]string{} +) + +func setTgSystem(m map[int]string) { tgSystemMu.Lock(); tgSystemMap = m; tgSystemMu.Unlock() } + +// Canonical TG->system name fetched from the shared tglist (svx-web /tglist.php). +// FROZEN DESIGN: the tglist is the SOLE source of TG names/Location shown in +// Lastheard — there is NO fallback (not nodes[].system, not live gateway/astmeta +// data). A TG absent from the tglist has no name, by design; the list is edited +// on the master and every box syncs it, so the display is identical everywhere. +var ( + tglistSystemMu sync.RWMutex + tglistSystemMap = map[int]string{} +) + +func setTglistSystem(m map[int]string) { tglistSystemMu.Lock(); tglistSystemMap = m; tglistSystemMu.Unlock() } + +func systemForTG(tg int) string { + tglistSystemMu.RLock() + defer tglistSystemMu.RUnlock() + return tglistSystemMap[tg] +} + +// Configured node display names (every nodes[] via/callsign). A configured node is +// an intentional own node and is never hidden by the bridge/infra filter. +var ( + confNamesMu sync.RWMutex + confNamesSet = map[string]bool{} +) + +func setConfiguredNames(m map[string]bool) { confNamesMu.Lock(); confNamesSet = m; confNamesMu.Unlock() } +func isConfiguredName(name string) bool { + confNamesMu.RLock() + defer confNamesMu.RUnlock() + return confNamesSet[name] +} + +// satLogins = the shared reflector login(s) our sat nodes use (= trunk +// satellite_id, e.g. KT4ROY-ASL). The per-TG rename applies ONLY to a node +// whose callsign IS such a login — so a FOREIGN node that merely monitors the +// same TG (e.g. SA7AUX on 3149050) is never mislabeled KT4ROY-ASL2. +var ( + satLoginsMu sync.RWMutex + satLoginsSet = map[string]bool{} +) + +func setSatLogins(m map[string]bool) { satLoginsMu.Lock(); satLoginsSet = m; satLoginsMu.Unlock() } +func isSatLogin(cs string) bool { + satLoginsMu.RLock() + defer satLoginsMu.RUnlock() + return satLoginsSet[cs] +} + +// renameSatNode is THE rule for statusnorm.Normalize: a node whose login (its +// callsign for trunk/sat nodes, its key for top-level nodes) is a configured group +// -> that group's per-TG name. Keyed by (group, tg), so two different nodes (e.g. an +// ASL one and a DMR one) can share a TG and each keeps its own group's name. +func renameSatNode(login string, tg int) string { + // DISABLED (2026-08-25): the sat nodes now announce network-unique names at + // the source (callsign + per-TG suffix, e.g. KT4ROY-ASL-A93 / -050), so the + // bare shared login no longer appears and this per-TG rewrite is redundant — + // and we must NOT re-rewrite the already-unique source names. Always keep the + // source name; duplicate-safety still lives in uniquifyNames (view-only). + _ = login + _ = tg + return "" +} + +// satTrunkTGs = the satellite trunk's talkgroups straight from svxdr.json nodes[] +// (grouped under the trunk satellite_id), names = the per-TG via. ONE source; used +// by Monitor->Trunks so it needs neither svx_sat.conf nor an override. +var ( + satTrunkTGsMu sync.RWMutex + satTrunkTGs = map[string][]satTG{} +) + +func setSatTrunkTGs(m map[string][]satTG) { satTrunkTGsMu.Lock(); satTrunkTGs = m; satTrunkTGsMu.Unlock() } +func satTGsForTrunk(id string) []satTG { + satTrunkTGsMu.RLock() + defer satTrunkTGsMu.RUnlock() + return satTrunkTGs[id] +} +func satTrunkIDs() []string { + satTrunkTGsMu.RLock() + defer satTrunkTGsMu.RUnlock() + ids := make([]string, 0, len(satTrunkTGs)) + for id := range satTrunkTGs { + ids = append(ids, id) + } + return ids +} + +func setStr(dst *string, v string) { + if v != "" { + *dst = v + } +} +func setDur(dst *time.Duration, v config.Duration) { + if v.D() > 0 { + *dst = v.D() + } +} + +func applyConfig(c *config.Config) { + n := c.Nodelist + setStr(&listenAddr, n.ListenAddr) + setStr(&reflectorURL, n.ReflectorStatus) + // monitor status defaults to the (now-resolved) reflector URL unless set. + monitorStatusURL = reflectorURL + setStr(&monitorStatusURL, n.MonitorStatusURL) + debug = n.Debug + setDur(&pollInterval, n.PollInterval) + setStr(&reflectorLogPath, n.ReflectorLogPath) + setStr(&networkJSONPath, n.NetworkJSONPath) + setStr(&astdbURL, n.AstdbURL) + setDur(&talkerMetaTTL, n.TalkerMetaTTL) + setStr(&ingestToken, n.TalkerIngestToken) + setDur(&allstarNodesTTL, n.AllstarNodesTTL) + setDur(&nodeOnlineTTL, n.NodeOnlineTTL) + setDur(&nodeTTL, n.NodeTTL) + setStr(&mqttBroker, n.MqttBroker) + setStr(&mqttUser, n.MqttUser) + setStr(&mqttPass, n.MqttPass) + setStr(&mqttPrefix, n.MqttPrefix) + + e := n.External + setStr(&externalPullURL, e.PullURL) + setStr(&externalPullScheme, e.PullScheme) + setStr(&externalPullPort, e.PullPort) + setStr(&externalPullPath, e.PullPath) + setDur(&externalPullInterval, e.PullInterval) + if len(e.StatusPorts) > 0 { + externalStatusPorts = strings.Join(e.StatusPorts, ",") + } + setStr(&externalStatusPath, e.StatusPath) + setStr(&externalProxyPath, e.ProxyPath) + setStr(&externalDetailPath, e.DetailPath) + if e.MaxRefs > 0 { + externalMaxRefs = e.MaxRefs + } + setDur(&externalStaleTTL, e.StaleTTL) + + setDur(&monitorInterval, n.MonitorInterval) + + f := n.Federation + if len(f.Seeds) > 0 { + federationSeeds = strings.Join(f.Seeds, ",") + } + setStr(&federationAggURL, f.AggregateURL) + setDur(&federationAggInterval, f.AggregateEvery) + + r := c.Redis + setStr(&redisSock, r.Socket) + setStr(&redisPass, r.Password) + setStr(&statusKey, r.StatusKey) + setStr(&statusHashKey, r.StatusHashKey) + setStr(&pubsubChan, r.PubsubChan) + setStr(&nodesOnlineKey, r.NodesOnlineKey) + setStr(&nodeKeyPrefix, r.NodeKeyPrefix) + setStr(&nodeConnectChan, r.NodeConnectChan) + setStr(&nodeDisconnectChan, r.NodeDisconnectChan) + setStr(&nodeUpdateChan, r.NodeUpdateChan) + setStr(&lastheardKey, r.LastheardKey) + + // xlink only when this box has the module enabled. + xlinkEnabled = c.Modules.Xlink + if c.Modules.Xlink { + x := c.Xlink + setStr(&xlinkBroker, x.Broker) + setStr(&xlinkUser, x.User) + setStr(&xlinkPass, x.Pass) + setStr(&xlinkTopic, x.Topic) + setStr(&xlinkOrigin, x.Origin) + } +} + +// applyRenderFromConfig sets the lastheard render config (Location formats + +// reflector names) from svxdr.json. reflector_names is keyed the way +// bridgeIdentity looks it up: bridge label (via/callsign) and far AllStar node. +func applyRenderFromConfig(c *config.Config) { + rc := defaultRenderCfg + rc.ReflectorNames = map[string]string{} + if s := strings.TrimSpace(c.Lastheard.BridgeLocationFormat); s != "" { + rc.BridgeLocationFormat = s + } + if s := strings.TrimSpace(c.Lastheard.BridgeLocationFormatNoSystem); s != "" { + rc.BridgeLocationFormatNoSystem = s + } + for k, v := range c.Lastheard.ReflectorNames { + if v != "" { + rc.ReflectorNames[k] = v + } + } + // Skip via/callsign labels that several nodes share with DIFFERENT systems + // (e.g. a uniform trunk callsign KT4ROY-ASL across East Coast + WHITELIGHTNING): + // keying reflector_names by them collapses and then overrides astmeta's + // correct per-TG System. Only add unambiguous labels. + const conflict = "\x00CONFLICT" + label := map[string]string{} + mark := func(k, sys string) { + if k == "" || sys == "" { + return + } + if prev, ok := label[k]; ok && prev != sys { + label[k] = conflict + } else if !ok { + label[k] = sys + } + } + for _, e := range c.Nodes { + if !e.IsEnabled() { + continue + } + mark(e.Via, e.System) + mark(e.Callsign, e.System) + } + for _, e := range c.Nodes { + if !e.IsEnabled() || e.System == "" { + continue + } + if e.Via != "" && label[e.Via] != conflict { + rc.ReflectorNames[e.Via] = e.System + } + if e.Callsign != "" && label[e.Callsign] != conflict { + rc.ReflectorNames[e.Callsign] = e.System + } + if e.Allstar != 0 { + rc.ReflectorNames[strconv.Itoa(e.Allstar)] = e.System + } + } + // ONE pass over nodes[] builds everything, purely config-driven, group-aware: + // gv[group][tg]=via per-(group,TG) rename — two nodes (ASL + DMR) may share a TG + // any[tg]={via,...} every configured via for a TG — talkerMetaFor match + // logins/groupTGs the group trunks (satLogins + per-TG names) + // modeMap[name]=mode free-text type per node + // NOTHING about ASL/DMR/etc is hardcoded — add a group/mode in config and it + // renders the same as any other. + gv := map[string]map[int]string{} + any := map[int]map[string]bool{} + logins := map[string]bool{} + groupTGs := map[string][]satTG{} + modeMap := map[string]string{} + confNames := map[string]bool{} + tgSys := map[int]string{} + for _, e := range c.Nodes { + if !e.IsEnabled() || e.TG == 0 { + continue + } + if e.System != "" { + tgSys[e.TG] = e.System // TG -> reflector name, Location fallback for any talker on this TG + } + name := e.Via + if name == "" { + name = e.Callsign + } + if name == "" { + continue + } + confNames[name] = true + if any[e.TG] == nil { + any[e.TG] = map[string]bool{} + } + any[e.TG][name] = true + if e.Mode != "" { + modeMap[name] = e.Mode + } + if e.Group != "" { + logins[e.Group] = true + if gv[e.Group] == nil { + gv[e.Group] = map[int]string{} + } + gv[e.Group][e.TG] = name + groupTGs[e.Group] = append(groupTGs[e.Group], satTG{Name: name, TG: strconv.Itoa(e.TG)}) + } + } + setTgVia(gv, any) + setSatLogins(logins) + setSatTrunkTGs(groupTGs) + setModeByName(modeMap) + setConfiguredNames(confNames) + setTgSystem(tgSys) + + renderCfgMu.Lock() + renderCfg = rc + renderCfgMu.Unlock() +} diff --git a/svxdr/proxy/proxy.go b/svxdr/proxy/proxy.go new file mode 100644 index 0000000..cb06fd0 --- /dev/null +++ b/svxdr/proxy/proxy.go @@ -0,0 +1,416 @@ +// svxdr — live-data daemon for SvxPortal (nodelist + proxy + astmeta + xlink) +// Copyright (C) 2026 SA7AUX (Björn) +// SPDX-License-Identifier: AGPL-3.0-or-later +// +// This program is free software: you can redistribute it and/or modify it under +// the terms of the GNU Affero General Public License as published by the Free +// Software Foundation, either version 3 of the License, or (at your option) any +// later version. See the LICENSE file for details. + +// Package proxy is the svxdr proxy subsystem — the enriched SvxReflector status +// proxy, folded in from the standalone svx-web/proxy. Behaviour is preserved +// exactly (byte-identical status enrichment, same /status, /api/talker-meta, +// /healthz, same meta-sync). Config now comes from svxdr.json's "proxy" section +// instead of env, and Run is ctx-aware for graceful shutdown. +package proxy + +import ( + "bytes" + "context" + "encoding/json" + "io" + "log" + "net/http" + "strconv" + "strings" + "sync" + "time" + + "svxportal/svxdr/config" + "svxportal/svxdr/statusnorm" +) + +type talkerMeta struct { + Callsign string `json:"callsign,omitempty"` + Node string `json:"node,omitempty"` + System string `json:"system,omitempty"` + Via string `json:"via,omitempty"` + Network string `json:"network,omitempty"` + at time.Time +} + +// Server holds the proxy's config-derived settings and in-memory state. Instance +// -scoped (no package globals) so a future in-memory hand-off from the astmeta +// module can share one store, and so reload can rebuild cleanly. +type Server struct { + listen string + upstreamURL string + cacheTTL time.Duration + metaTTL time.Duration + ingestToken string + upstreamPeer string + forwardTalker string + syncInterval time.Duration + remoteTTL time.Duration + httpClient *http.Client + tgVia map[int]string // TG -> per-TG display callsign (svxdr.json nodes[]) + satLogins map[string]bool // shared sat logins (trunk satellite_id) + + upstream struct { + mu sync.RWMutex + data []byte + at time.Time + } + meta struct { + mu sync.RWMutex + byTG map[int]talkerMeta + } + remote struct { + mu sync.RWMutex + sats json.RawMessage + at time.Time + } +} + +// Run starts the proxy HTTP server and meta-sync loop, blocking until ctx is +// cancelled, then shutting the server down gracefully. +func Run(ctx context.Context, c *config.Config) error { + p := c.Proxy + s := &Server{ + listen: orDefault(p.Listen, ":8099"), + upstreamURL: orDefault(p.UpstreamURL, "http://127.0.0.1:8080/status"), + cacheTTL: orDur(p.CacheTTL.D(), 200*time.Millisecond), + metaTTL: orDur(p.MetaTTL.D(), 30*time.Second), + ingestToken: p.IngestToken, + upstreamPeer: p.UpstreamPeer, + forwardTalker: p.ForwardTalker, + syncInterval: orDur(p.SyncInterval.D(), 2*time.Second), + remoteTTL: orDur(p.RemoteTTL.D(), 60*time.Second), + } + s.meta.byTG = map[int]talkerMeta{} + s.httpClient = &http.Client{Timeout: orDur(p.FetchTimeout.D(), 5*time.Second)} + + // Per-TG display callsign: sat nodes share one reflector login (KT4ROY-ASL); + // override the callsign shown in /status with the per-TG name (KT4ROY-ASL1..4) + // from svxdr.json. The Asterisk login is NOT touched — this is display only. + s.tgVia = map[int]string{} + s.satLogins = map[string]bool{} + if c.Trunk.SatelliteID != "" { + s.satLogins[c.Trunk.SatelliteID] = true + } + for _, n := range c.Nodes { + if !n.IsEnabled() || n.TG == 0 { + continue + } + via := n.Via + if via == "" { + via = n.Callsign + } + if via != "" { + s.tgVia[n.TG] = via + } + } + + mux := http.NewServeMux() + mux.HandleFunc("/status", s.handleStatus) + mux.HandleFunc("/api/talker-meta", s.handleTalkerMeta) + mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte("ok\n")) }) + + go s.metaSyncLoop(ctx) + + srv := &http.Server{Addr: s.listen, Handler: mux, ReadHeaderTimeout: 5 * time.Second} + go func() { + <-ctx.Done() + sc, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + _ = srv.Shutdown(sc) + }() + log.Printf("proxy: listen=%s upstream=%s cache=%s metaTTL=%s", s.listen, s.upstreamURL, s.cacheTTL, s.metaTTL) + if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed { + return err + } + return nil +} + +func orDefault(v, def string) string { + if v == "" { + return def + } + return v +} +func orDur(v, def time.Duration) time.Duration { + if v <= 0 { + return def + } + return v +} + +// ---- upstream status cache ------------------------------------------------- + +func (s *Server) fetchStatus() ([]byte, error) { + resp, err := s.httpClient.Get(s.upstreamURL) + if err != nil { + return nil, err + } + defer resp.Body.Close() + return io.ReadAll(io.LimitReader(resp.Body, 8<<20)) +} + +func (s *Server) cachedStatus() ([]byte, error) { + s.upstream.mu.RLock() + if s.upstream.data != nil && time.Since(s.upstream.at) < s.cacheTTL { + d := s.upstream.data + s.upstream.mu.RUnlock() + return d, nil + } + s.upstream.mu.RUnlock() + + d, err := s.fetchStatus() + if err != nil { + s.upstream.mu.RLock() + stale := s.upstream.data + s.upstream.mu.RUnlock() + if stale != nil { + return stale, nil + } + return nil, err + } + // ONE normalization, at ingest (cache fill — at most once per cacheTTL, not + // per request): sat-login node callsigns become their per-TG name. Everything + // that reads this proxy (its own /status AND nodelist over it) is then correct + // with no per-view special-casing. + d = statusnorm.Normalize(d, s.renameNode) + s.upstream.mu.Lock() + s.upstream.data = d + s.upstream.at = time.Now() + s.upstream.mu.Unlock() + return d, nil +} + +// renameNode is the ONE rule: a node whose callsign IS a shared sat login and +// whose TG has a per-TG name gets that name; everything else is left alone. +func (s *Server) renameNode(callsign string, tg int) string { + // DISABLED (2026-08-25): sat nodes now advertise network-unique names at the + // source (callsign + per-TG suffix, e.g. KT4ROY-ASL-A93), so the bare shared + // login no longer appears and this per-TG rewrite is redundant. Never rewrite + // the already-unique source name; dupe-safety stays in the nodelist view. + _ = callsign + _ = tg + return "" +} + +// ---- meta store ------------------------------------------------------------ + +func (s *Server) setMeta(tg int, m talkerMeta) { + m.at = time.Now() + s.meta.mu.Lock() + s.meta.byTG[tg] = m + s.meta.mu.Unlock() +} + +func (s *Server) freshMeta() map[string]talkerMeta { + out := map[string]talkerMeta{} + s.meta.mu.RLock() + for tg, m := range s.meta.byTG { + if time.Since(m.at) <= s.metaTTL { + out[strconv.Itoa(tg)] = m + } + } + s.meta.mu.RUnlock() + return out +} + +func (s *Server) setRemoteSatellites(raw json.RawMessage) { + s.remote.mu.Lock() + s.remote.sats = raw + s.remote.at = time.Now() + s.remote.mu.Unlock() +} + +func (s *Server) freshRemoteSatellites() json.RawMessage { + s.remote.mu.RLock() + defer s.remote.mu.RUnlock() + if s.remote.sats != nil && time.Since(s.remote.at) <= s.remoteTTL { + return s.remote.sats + } + return nil +} + +// ---- meta sync ------------------------------------------------------------- + +type metaEntry struct { + Callsign string `json:"callsign"` + Node string `json:"node"` + System string `json:"system"` + Via string `json:"via"` + Network string `json:"network"` +} + +func (s *Server) metaSyncLoop(ctx context.Context) { + if s.upstreamPeer == "" { + return + } + log.Printf("proxy meta-sync: upstream=%s forward=%s interval=%s", s.upstreamPeer, s.forwardTalker, s.syncInterval) + t := time.NewTicker(s.syncInterval) + defer t.Stop() + for { + select { + case <-ctx.Done(): + return + case <-t.C: + } + resp, err := s.httpClient.Get(s.upstreamPeer) + if err != nil { + continue + } + body, _ := io.ReadAll(io.LimitReader(resp.Body, 8<<20)) + resp.Body.Close() + var top struct { + TalkerMeta map[string]metaEntry `json:"talker_meta"` + Satellites json.RawMessage `json:"satellites"` + } + if json.Unmarshal(body, &top) != nil { + continue + } + if len(top.Satellites) > 0 && !bytes.Equal(bytes.TrimSpace(top.Satellites), []byte("null")) { + s.setRemoteSatellites(top.Satellites) + } + for tgStr, m := range top.TalkerMeta { + tg, err := strconv.Atoi(tgStr) + if err != nil || tg <= 0 { + continue + } + s.setMeta(tg, talkerMeta{Callsign: m.Callsign, Node: m.Node, System: m.System, Via: m.Via, Network: m.Network}) + if s.forwardTalker != "" { + s.forwardMeta(tg, m) + } + } + } +} + +func (s *Server) forwardMeta(tg int, m metaEntry) { + body, _ := json.Marshal(map[string]any{ + "tg": tg, "event": "start", + "callsign": m.Callsign, "node": m.Node, "system": m.System, + "via": m.Via, "network": "", + }) + req, err := http.NewRequest(http.MethodPost, s.forwardTalker, bytes.NewReader(body)) + if err != nil { + return + } + req.Header.Set("Content-Type", "application/json") + resp, err := s.httpClient.Do(req) + if err != nil { + return + } + resp.Body.Close() +} + +// ---- enrichment (byte-exact, preserved from svx-web/proxy) ------------------ + +func (s *Server) enrich(raw []byte) []byte { + extras := s.buildExtras(raw) + if len(extras) == 0 { + return raw + } + i := 0 + for i < len(raw) && raw[i] != '{' { + i++ + } + if i >= len(raw) { + return raw + } + j := i + 1 + for j < len(raw) && (raw[j] == ' ' || raw[j] == '\t' || raw[j] == '\n' || raw[j] == '\r') { + j++ + } + empty := j < len(raw) && raw[j] == '}' + + out := make([]byte, 0, len(raw)+len(extras)+1) + out = append(out, raw[:i+1]...) + out = append(out, extras...) + if !empty { + out = append(out, ',') + } + out = append(out, raw[i+1:]...) + return out +} + +func (s *Server) buildExtras(raw []byte) []byte { + var frags [][]byte + if fm := s.freshMeta(); len(fm) > 0 { + if b, err := json.Marshal(fm); err == nil { + frags = append(frags, append([]byte(`"talker_meta":`), b...)) + } + } + if sats := s.freshRemoteSatellites(); sats != nil && !bytes.Contains(raw, []byte(`"satellites"`)) { + frags = append(frags, append([]byte(`"satellites":`), sats...)) + } + if len(frags) == 0 { + return nil + } + return bytes.Join(frags, []byte(",")) +} + +// ---- HTTP ------------------------------------------------------------------ + +func isLoopback(addr string) bool { + host := addr + if i := strings.LastIndex(addr, ":"); i > 0 { + host = addr[:i] + } + host = strings.Trim(host, "[]") + return host == "127.0.0.1" || host == "::1" || host == "localhost" +} + +func (s *Server) handleStatus(w http.ResponseWriter, r *http.Request) { + raw, err := s.cachedStatus() + if err != nil { + http.Error(w, `{"error":"upstream unavailable"}`, http.StatusBadGateway) + return + } + w.Header().Set("Content-Type", "application/json; charset=utf-8") + w.Header().Set("Cache-Control", "no-cache") + w.Header().Set("Access-Control-Allow-Origin", "*") + _, _ = w.Write(s.enrich(raw)) +} + +func (s *Server) handleTalkerMeta(w http.ResponseWriter, r *http.Request) { + if r.Method != http.MethodPost { + http.Error(w, `{"msg":"POST only"}`, http.StatusMethodNotAllowed) + return + } + if !isLoopback(r.RemoteAddr) { + if s.ingestToken == "" || r.Header.Get("X-Ingest-Token") != s.ingestToken { + http.Error(w, `{"msg":"unauthorized"}`, http.StatusUnauthorized) + return + } + } + var in struct { + TG int `json:"tg"` + Event string `json:"event"` + Callsign string `json:"callsign"` + Node string `json:"node"` + System string `json:"system"` + Via string `json:"via"` + Network string `json:"network"` + } + body, err := io.ReadAll(io.LimitReader(r.Body, 4096)) + if err != nil || json.Unmarshal(body, &in) != nil || in.TG <= 0 { + http.Error(w, `{"msg":"bad request"}`, http.StatusBadRequest) + return + } + w.Header().Set("Content-Type", "application/json; charset=utf-8") + if in.Event == "stop" { + _, _ = w.Write([]byte(`{"ok":true}`)) + return + } + s.setMeta(in.TG, talkerMeta{ + Callsign: strings.TrimSpace(in.Callsign), + Node: strings.TrimSpace(in.Node), + System: strings.TrimSpace(in.System), + Via: strings.TrimSpace(in.Via), + Network: strings.TrimSpace(in.Network), + }) + _, _ = w.Write([]byte(`{"ok":true}`)) +} diff --git a/svxdr/statusnorm/statusnorm.go b/svxdr/statusnorm/statusnorm.go new file mode 100644 index 0000000..227318d --- /dev/null +++ b/svxdr/statusnorm/statusnorm.go @@ -0,0 +1,142 @@ +// svxdr — live-data daemon for SvxPortal (nodelist + proxy + astmeta + xlink) +// Copyright (C) 2026 SA7AUX (Björn) +// SPDX-License-Identifier: AGPL-3.0-or-later +// +// This program is free software: you can redistribute it and/or modify it under +// the terms of the GNU Affero General Public License as published by the Free +// Software Foundation, either version 3 of the License, or (at your option) any +// later version. See the LICENSE file for details. + +// Package statusnorm holds the ONE rule that turns a reflector's shared sat-login +// (e.g. KT4ROY-ASL, used by all its TG nodes) into the per-TG display callsign +// (KT4ROY-ASL1..4). It is applied ONCE where a status is ingested (the proxy's +// cache fill, and the federation crawl), so every consumer downstream reads +// already-correct data — no per-view special-casing anywhere. +package statusnorm + +import "encoding/json" + +// Normalize rewrites the callsign of every node in trunks[].nodes and +// satellites[].nodes: rename(callsign, tg) returns the replacement, or "" to keep +// it. All other data is preserved. Parse failure or no change -> raw unchanged. +func Normalize(raw []byte, rename func(callsign string, tg int) string) []byte { + var top map[string]json.RawMessage + if json.Unmarshal(raw, &top) != nil { + return raw + } + changed := false + + // Top-level nodes[]: a group can present as a plain node (e.g. a DMR gateway), + // not only under a trunk/satellite. Keyed by the node's login/identity, so rename + // by that KEY (its callsign field may be the operator, not the login). + if blk, ok := top["nodes"]; ok { + var nm map[string]json.RawMessage + if json.Unmarshal(blk, &nm) == nil { + blkChanged := false + for id, nRaw := range nm { + var n map[string]json.RawMessage + if json.Unmarshal(nRaw, &n) != nil { + continue + } + cs, tg := callsignTG(n) + repl := rename(id, tg) // match on the node's KEY (its login) + if repl == "" || repl == cs { + continue + } + b, _ := json.Marshal(repl) + n["callsign"] = b + if enc, err := json.Marshal(n); err == nil { + nm[id] = enc + blkChanged = true + } + } + if blkChanged { + if enc, err := json.Marshal(nm); err == nil { + top["nodes"] = enc + changed = true + } + } + } + } + + for _, key := range []string{"trunks", "satellites"} { + blk, ok := top[key] + if !ok { + continue + } + var group map[string]json.RawMessage + if json.Unmarshal(blk, &group) != nil { + continue + } + blkChanged := false + for id, objRaw := range group { + var obj map[string]json.RawMessage + if json.Unmarshal(objRaw, &obj) != nil { + continue + } + nodesRaw, ok := obj["nodes"] + if !ok { + continue + } + var nodes []json.RawMessage + if json.Unmarshal(nodesRaw, &nodes) != nil { + continue + } + nodesChanged := false + for i, nRaw := range nodes { + var n map[string]json.RawMessage + if json.Unmarshal(nRaw, &n) != nil { + continue + } + cs, tg := callsignTG(n) + if cs == "" { + continue + } + repl := rename(cs, tg) + if repl == "" || repl == cs { + continue + } + b, _ := json.Marshal(repl) + n["callsign"] = b + if enc, err := json.Marshal(n); err == nil { + nodes[i] = enc + nodesChanged = true + } + } + if nodesChanged { + if enc, err := json.Marshal(nodes); err == nil { + obj["nodes"] = enc + if oenc, err := json.Marshal(obj); err == nil { + group[id] = oenc + blkChanged = true + } + } + } + } + if blkChanged { + if enc, err := json.Marshal(group); err == nil { + top[key] = enc + changed = true + } + } + } + if !changed { + return raw + } + if enc, err := json.Marshal(top); err == nil { + return enc + } + return raw +} + +func callsignTG(n map[string]json.RawMessage) (string, int) { + var cs string + if v, ok := n["callsign"]; ok { + _ = json.Unmarshal(v, &cs) + } + var tg int + if v, ok := n["tg"]; ok { + _ = json.Unmarshal(v, &tg) + } + return cs, tg +}