#collector — counters, snapshots and ingestion
The collector turns what the host is doing into numbers: interfaces and peers, nginx requests, mail deliveries. It is not a cube and has no application to build. It owns three independent submodules, each with its own raw snapshot, its own database and its own daemon:
| Submodule | Reads | Snapshot | Database |
|---|---|---|---|
net | /proc/net/dev, ss | net.json | net.db |
nginx | access_json.log, stub status | nginx.json | nginx.db |
mail | Postfix mail.log, hook lines | mail.json | mail.db |
Collection and ingestion are deliberately separate. A daemon writes a JSON snapshot on its own timer and never waits on SQLite; a keeper job folds that snapshot into the database afterwards. The raw file stays on disk, so what the collector saw can always be inspected without querying anything.
aegis collector --helpaegis collector statusThe group has no banner of its own: a bare aegis collector is an argparse error (exit 2), and --help is the discovery path. The commands are setup, status, collect, ingest, and the same three verbs under net, nginx and mail.
#How one pass works
collect reads the source and writes <raw_dir>/<name>.json atomically.The snapshot replaces the previous one.
ingest reads that snapshot, inserts the rows, advances the state table andrecords last_ingest.
interval seconds; a keeper job repeats step 2on the ingestion interval.
A failure in either half is reported and does not stop the other submodules.
#Platform-wide commands
#aegis collector setup
Create the database of every enabled submodule and apply its migrations. It is idempotent: an up-to-date database is reported as such, and files from the two older layouts (a flat <name>.db, then storage/collector/<name>.db) are adopted on first use rather than recreated.
aegis collector setup| Argument / option | Default | Meaning |
|---|---|---|
| — | — | none; the command takes no arguments or options |
aegis collector setupaegis collector setup && aegis collector status✔ net: /aegis/database/storage/net/net.db (up to date); nginx: /aegis/database/storage/nginx/nginx.db (up to date); mail: /aegis/database/storage/mail/mail.db (up to date)An exception while creating a database prints the message with a ✖ marker and exits 1.
#aegis collector status
Print the whole collector on one screen: settings, daemon state, raw snapshot age and size, each database with its last ingest, and the data sections the submodules provide. Values come from the status files and from the SQLite databases, so the report cannot disagree with the data it describes.
aegis collector status --limit <n>| Option | Default | Meaning |
|---|---|---|
--limit | 10 | rows in the "top peers", "ports", domains and hooks sections |
aegis collector statusaegis collector status --limit 5Collector raw dir /aegis/data/collector enabled net, nginx, mail interval net 30s (ingest 30s) interval nginx 30s (ingest 30s) interval mail 30s (ingest 30s) aegis-collector-net running (pid 8123) cycles 1842 (errors 0) started 2026-11-16T21:00:03+00:00 last cycle 2026-11-17T06:42:01+00:00 interval 30s aegis-collector-nginx running (pid 8124) cycles 1840 (errors 0) started 2026-11-16T21:00:03+00:00 last cycle 2026-11-17T06:42:01+00:00 interval 30s aegis-collector-mail running (pid 8125) cycles 1839 (errors 2) started 2026-11-16T21:00:03+00:00 last cycle 2026-11-17T06:42:01+00:00 interval 30s Raw files net 184.2 kB 12s old /aegis/data/collector/net.json nginx 42.6 kB 12s old /aegis/data/collector/nginx.json mail 9.1 kB 12s old /aegis/data/collector/mail.json net.db 1.2 MB (9 tables) /aegis/database/storage/net/net.db last ingest 2026-11-17T06:42:00+00:00 last collect 2026-11-17T06:41:58+00:00 nginx.db 412.0 kB (7 tables) /aegis/database/storage/nginx/nginx.db last ingest 2026-11-17T06:42:00+00:00 last collect 2026-11-17T06:41:58+00:00 mail.db 84.3 kB (6 tables) /aegis/database/storage/mail/mail.db last ingest 2026-11-17T06:42:00+00:00 last collect 2026-11-17T06:41:58+00:00 Interfaces name state mtu rx tx rx/s tx/s eth0 up 1500 12.4 GB 3.1 GB 1.2 MB/s 84.0 kB/s lo unknown 65536 214.0 MB 214.0 MB 12.0 kB/s 12.0 kB/s Top peers (by rx rate, 2) ip conns rx tx rx/s tx/s ports 203.0.113.24 18 284.1 MB 12.2 MB 84.0 kB/s 9.1 kB/s 443, 80 198.51.100.7 4 18.4 MB 2.1 MB 12.0 kB/s 1.2 kB/s 443 Ports (2) proto port listen conns rx tx tcp 443 yes 18 284.1 MB 12.2 MB tcp 80 yes 4 18.4 MB 2.1 MB nginx (bucket 2026-11-17T06:41:00+00:00) requests 184 bytes 12.4 MB status 171 2xx, 6 3xx, 7 4xx, 0 5xx connections active max 12 (reading 1, writing 2, waiting 9) mail (bucket 2026-11-17T06:41:00+00:00) deliveries 42 sent, 1 deferred, 0 bounced, 0 rejected example.com 38 sent 0 deferred 0 bounced hook queue out 42 runs 0 failed avg 12 ms max 88 msA daemon that never started shows a line instead of counters:
aegis-collector-mail not started (no status file)A leftover status file from a crashed process is reported honestly:
aegis-collector-net dead (stale status file)status always exits 0. A missing database, a missing snapshot or a section with no rows simply shortens the report; nginx and mail sections appear only when their submodule is enabled and their database has a bucket.
#aegis collector collect
Run one collection pass for every submodule, in the order net, nginx, mail. Each result is printed on its own line; a failing submodule prints an error and the rest still run. The exit code is 1 when any submodule failed.
aegis collector collect| Argument / option | Default | Meaning |
|---|---|---|
| — | — | none; the command takes no arguments or options |
aegis collector collectaegis collector collect && aegis collector ingest✔ net: 4 interfaces, 23 peers, 12 ports✔ nginx: 184 requests, 65126400 bytes, 184 stored✔ mail: 42 deliveries, 57 events, 3 hook runsA submodule that raises prints its name and the error on stderr, marked ✖, and makes the command exit 1; the remaining submodules still run.
collect writes every submodule it is asked for, enabled or not. The enabled flag decides what the keeper supervises, not what an explicit command may run.
#aegis collector ingest
Fold every snapshot into its database, in the same order and with the same per-submodule error handling. An ingest with no snapshot yet is not an error for the report, but the submodule returns its own message.
aegis collector ingest| Argument / option | Default | Meaning |
|---|---|---|
| — | — | none; the command takes no arguments or options |
aegis collector ingestaegis collector ingest && aegis collector status✔ net: 4 interfaces, 23 peers, 12 ports✔ nginx: 184 requests, 184 stored✔ mail: 42 deliveries, 57 eventsA submodule that cannot write its database prints its name and the error, marked ✖, and the exit code is 1.
#The net collector
net reads /proc/net/dev for cumulative interface counters and the socket table for peers and service ports. It uses ss -tin because that is the only source of per-socket bytes_sent / bytes_received without a kernel module; when ss is absent it falls back to /proc/net/tcp* and /proc/net/udp*, which give counts but no bytes.
Every pass replaces the snapshot atomically, including one row per socket so the ingest can account traffic that disappears when a socket closes.
#aegis collector net daemon
Run the net collector in the foreground, one pass every collector.net.interval seconds. The keeper runs exactly this command as the service aegis-collector-net; start it by hand only when debugging.
aegis collector net daemon| Argument / option | Default | Meaning |
|---|---|---|
| — | — | none; the command takes no arguments or options |
aegis collector net daemonaegis collector net daemon &2026-11-17T06:40:00+00:00 aegis-collector-net started, every 30s2026-11-17T06:40:00+00:00 4 interfaces, 23 peers, 12 ports in 0.09s2026-11-17T06:40:30+00:00 4 interfaces, 23 peers, 12 ports in 0.08sThe loop is drift-free, never stacks missed cycles and handles signals:
| Signal | Effect |
|---|---|
SIGTERM, SIGINT | finish the current cycle, remove the pid file, exit 0 |
SIGHUP | re-read the configuration and apply a new interval |
A second copy refuses to start, logs already running (pid 8123) and exits 1. The pid file is run/collector/net.pid; the status file and the log are described under Files and paths.
#aegis collector net collect
Run one net pass and print what it found. This is the same code the daemon calls, so it is the quickest way to prove the host data is readable.
aegis collector net collect| Argument / option | Default | Meaning |
|---|---|---|
| — | — | none; the command takes no arguments or options |
aegis collector net collectaegis collector net collect && aegis collector net ingest✔ net: 4 interfaces, 23 peers, 12 ports#aegis collector net ingest
Read net.json and write its rows into net.db: interfaces with their rates, peers with connection counts and traffic, and service ports. The pass also stores the last collect and ingest stamps in the database state table.
aegis collector net ingest| Argument / option | Default | Meaning |
|---|---|---|
| — | — | none; the command takes no arguments or options |
aegis collector net ingestaegis collector net ingest && aegis collector status --limit 5✔ net: 4 interfaces, 23 peers, 12 ports#The nginx collector
nginx reads access_json.log from the byte offset it stored last time, so a pass counts only the requests that happened since the previous one. Rotation is detected by inode, size and the first line of the file, which also survives copytruncate. The nginx stub status page supplies the live connection counts that a log cannot show.
Each parsed request is also copied to a per-cycle spool file, then renamed to .jsonl; the ingest reads whole files only, so a request is never stored twice or cut in half.
#aegis collector nginx daemon
Run the nginx collector in the foreground, every collector.nginx.interval seconds. The keeper supervises it as aegis-collector-nginx.
aegis collector nginx daemon| Argument / option | Default | Meaning |
|---|---|---|
| — | — | none; the command takes no arguments or options |
aegis collector nginx daemonaegis collector nginx daemon &2026-11-17T06:40:00+00:00 aegis-collector-nginx started, every 30s2026-11-17T06:40:00+00:00 184 requests, 65126400 bytes, 184 stored in 0.04sThe signal handling, the pid file and the refusal of a second copy are the same as the net daemon.
#aegis collector nginx collect
Run one nginx pass. With no access log yet (nginx installed but not serving, or access_json.log not created) the counters are zero and the pass still writes a snapshot.
aegis collector nginx collect| Argument / option | Default | Meaning |
|---|---|---|
| — | — | none; the command takes no arguments or options |
aegis collector nginx collectaegis collector nginx collect && aegis collector nginx ingest✔ nginx: 184 requests, 65126400 bytes, 184 stored✔ nginx: 0 requests, 0 bytes, 0 stored#aegis collector nginx ingest
Write the snapshot's counters and the spooled requests into nginx.db.
aegis collector nginx ingest| Argument / option | Default | Meaning |
|---|---|---|
| — | — | none; the command takes no arguments or options |
aegis collector nginx ingestaegis collector nginx ingest && aegis collector status✔ nginx: 184 requests, 184 stored#The mail collector
mail reads Postfix's mail.log from the last offset. Delivery lines carry status=sent|deferred|bounced|rejected, the lines around them carry the sender, size and SMTP client, and the platform's own hook runner writes aegis-hook name=… direction=… status=… ms=… lines. Per-message events go to a spool file the ingest reads, exactly like nginx requests.
#aegis collector mail daemon
Run the mail collector in the foreground. The keeper supervises it as aegis-collector-mail.
aegis collector mail daemon| Argument / option | Default | Meaning |
|---|---|---|
| — | — | none; the command takes no arguments or options |
aegis collector mail daemonaegis collector mail daemon &2026-11-17T06:40:00+00:00 aegis-collector-mail started, every 30s2026-11-17T06:40:00+00:00 42 deliveries, 57 events, 3 hook runs in 0.02s#aegis collector mail collect
Read new lines from mail.log and write the snapshot plus the event spool.
aegis collector mail collect| Argument / option | Default | Meaning |
|---|---|---|
| — | — | none; the command takes no arguments or options |
aegis collector mail collectaegis collector mail collect && aegis collector mail ingest✔ mail: 42 deliveries, 57 events, 3 hook runsThe Postfix log path comes from postfix.log_file and defaults to /aegis/logs/postfix/mail.log.
#aegis collector mail ingest
Write the delivery counters, per-domain totals, hook runs and message events into mail.db.
aegis collector mail ingest| Argument / option | Default | Meaning |
|---|---|---|
| — | — | none; the command takes no arguments or options |
aegis collector mail ingestaegis collector mail ingest && aegis collector status✔ mail: 42 deliveries, 57 events#Files and paths
/aegis/storage/logs/collector/<name>.json snapshot/aegis/storage/logs/collector/<name>.status.json pid, cycles, last error/aegis/database/storage/<name>/<name>.db SQLite database/aegis/database/migrations/core/<name>/*.sql migrations/aegis/run/collector/<name>.pid pid file/aegis/logs/collector/<name>.log one line per cycleThe paths follow aegis.root; on this platform it is /aegis. The snapshot directory is collector.raw_dir when set (absolute or relative to the root), and the default above otherwise. A database left by an older layout is moved into place on first use, so an upgraded host needs no manual step.
#Configuration
| Key | Default | Meaning |
|---|---|---|
collector.raw_dir | storage/logs/collector | where snapshots are written |
collector.interval | 60 | fallback collection interval, seconds |
collector.ingest_interval | collection interval | fallback ingestion interval |
collector.keep_days | 0 (keep everything) | fallback retention for append-only rows |
collector.net.enabled | true | enable the net submodule |
collector.nginx.enabled | true | enable the nginx submodule |
collector.mail.enabled | true | enable the mail submodule |
collector.<name>.interval | collector.interval | collection interval for one submodule |
collector.<name>.ingest_interval | its collection interval | ingestion interval for one submodule |
collector.net.geoip_ttl_hours | 48 | how long one cached GeoIP answer is reused; 0 looks it up again |
collector.<name>.keep_days prunes the append-only tables of that submodule on every ingest (request rows, mail events, rollups, cached GeoIP answers); 0 disables pruning for that submodule. The global collector.keep_days is the fallback, and collector.<name>.enabled decides which submodules the keeper supervises. status prints the nginx and mail data sections only when their enabled key is explicitly true, so set all three in config/aegis.toml when you want the report to match what runs.
Set collector.<name>.ingest_interval explicitly when the ingestion rate matters: when neither it nor a collection interval is configured, the keeper registers the ingest job with a 10-second fallback.
The keeper registers one continuous daemon per enabled submodule and one collector-<name>-ingest job per enabled submodule:
aegis keeper servicesaegis keeper cron#Checking it is alive
aegis collector status # daemon state, snapshot age, last ingestaegis collector net collect # can the source be read at all?aegis check # collector rows in the platform health report| Symptom | Likely cause |
|---|---|
not started (no status file) | the keeper never started the daemon, or it is disabled |
dead (stale status file) | the process died; the status file is from the previous run |
| a snapshot age that keeps growing | the daemon is hung or the iterator stopped |
no collector database yet | aegis collector setup has not run |
| nginx counters stay at zero | access_json.log is missing, or its format is not JSON |
There is nothing destructive in this group: no command deletes a database or a snapshot. collect replaces the current snapshot, which is its purpose.