#Keeper — process supervision
Aegis does not use systemd. keeper starts the daemons, keeps them alive, and owns their pid files and logs.
#A service definition
Service( name="bind", command=("/aegis/apps/dns/bind/9.21/sbin/named", "-c", "/aegis/config/bind/named.conf", "-u", "root"), pid_file=Path("/aegis/run/bind/named.pid"), log_file=Path("/aegis/logs/bind/named.log"), environment={"LD_LIBRARY_PATH": "/aegis/apps/dns/bind/9.21/lib64:…"},)Cubes build it; the keeper never guesses paths:
class Bind(Cube): def service(self) -> keeper.Service: ...registry.services(config) walks the cubes on disk and collects every service(), so a new cube becomes supervised without touching the keeper.
#Operations
keeper.start(service) # detached, own session, stdout/stderr to the logkeeper.stop(service) # SIGTERM, then SIGKILL after 15 skeeper.restart(service)keeper.status(service) # "running (pid 123)" / "stopped"keeper.running(service) # boolstart refuses to run a second copy: it reads the pid file and checks the process is alive. A daemon that writes its own pid is waited for up to 10 seconds; if it exits first, the error points at the log file — unless a live pid appeared meanwhile, which means another start won the race. That is not a crash, and the answer says so (already running (pid 123)) instead of triggering a backoff.
When service.user is set, start hands the log file to that account: the keeper creates it as root, and a daemon that opens its own log path (redis, supervisord) would otherwise die with Can't open the log file: Permission denied.
#The loop
aegis keeper daemon # foregroundaegis keeper start # background, same loopEvery 10 seconds it starts any registered service that is not running, restarts one that stopped answering, and writes one line per action:
2026-11-15 14:35:56 keeper started (pid 679986)2026-11-15 14:36:07 bind: started (pid 680051)2026-11-17 15:12:04 aegis-tasker: no heartbeat, restarted (killed (pid 801564), started (pid 802341))Each service is handled in its own try, so a cube that raises while being inspected cannot abandon the rest of the tick. A service that dies within 60 seconds of its own start is treated as a crash loop, not as a restart: the next attempt waits 10 s, then 30 s, then 60 s, then every 5 minutes, so one broken daemon cannot fill the log or the process table.
SIGTERM/SIGINT end the loop cleanly and remove <run>/keeper.pid.
#Boot
aegis keeper enable # /etc/init.d/aegis-keeper + update-rc.d defaultsaegis keeper disableThe init script is a thin wrapper around aegis keeper start|stop|restart|status, so nothing depends on systemd. When update-rc.d is unavailable the script is still written and the message says so.
#Services without a pid file
Most daemons write their own pid file. Two do not (vsftpd, beanstalkd), and Postfix's master writes it in an unexpected place; a service can therefore declare writes_pid=False, in which case the keeper records the pid of the process it started instead of waiting ten seconds for a file that will never appear.
#Heartbeats
A pid only proves that a process exists. A daemon that deadlocks — a worker pool that never returns, a lock taken twice — keeps its pid and looks healthy to every check the keeper had.
A service can therefore declare a heartbeat file it rewrites while its loop runs:
Service( name="aegis-tasker", …, heartbeat=run_dir / "tasker.heartbeat", heartbeat_timeout=90.0,)The keeper treats a running service with a heartbeat older than its timeout as hung and restarts it — stop, then start — with the same backoff as a crash. A missing file is not stale: a daemon that has just started may not have written one yet, and restarting then would loop.
The tasker uses this: it beats once per queue poll (about a second), so a frozen tasker is noticed within 90 seconds instead of staying alive forever.
#One writer per pid file
Two parties could write a pid: the keeper and the daemon. For a service with writes_pid=True the daemon owns the file, and the keeper only reads it. That is the default and what every daemon should do.
The second half of the problem is the lock that keeps a single instance. It must not live in the pid file: the keeper removes a stale pid file before starting, and a lock on a removed file is a lock on an inode nobody else will open, so a second daemon would happily lock a fresh file and run beside the first. A daemon that needs exclusivity keeps it in its own file that is never removed (<run>/tasker.lock) and publishes the holder's pid where the keeper expects it.
#Scheduled jobs
The same loop also runs periodic work. Jobs are registered in code, exactly like services: a cube with periodic work exposes jobs() and returns Job objects from app.modules.keeper.cron.
Job name="ssl-renew" every=21600 # seconds run=lambda config: cube.renew help="Renew certificates that are close to expiry"The scheduler stores only state — when each job last ran and what it answered — in <run>/keeper-cron.json. A job runs when now - last >= every, so a keeper restart never re-runs everything at once, and an interval is honoured to the second even though the loop ticks every 10 s.
A failing job is logged and swallowed: one broken handler must not stop the supervision loop.
aegis keeper cron # every job with interval, last run, resultaegis keeper cron-run ssl-renew # run one job now, due or not#ssl-renew
The first registered job, and the reason the scheduler exists:
[certbot] renew_days left (30 by default);certbot renew — the DNS-01 challenge is answered through BIND with theTSIG key, so no port has to be reachable;
nginx, postfix and dovecot, which hold the files open.It checks every [certbot] renew_interval seconds (6 hours by default). With nothing close to expiry it answers nothing to renew and changes nothing.
#Log rotation
A daemon keeps its log file open for its whole life, and most of the daemons the platform supervises have no signal that reopens it. Rotation is therefore copytruncate: the file is copied to .1, then truncated, and the writer never notices. No service is restarted, and no per-daemon hook exists to forget.
The job logs-rotate (every [logs] interval, 5 minutes by default) walks three sources, all already known to the platform:
[aegis] logs_dir — keeper, panel, AI, tasker, pulse, bird, collector;/aegis/logs — the applications: nginx, PHP, MariaDB, PostgreSQL, mail,FTP, certbot.
Every *.log found is classified by its name and directory, and each class has its own limit and generation count:
| Class | Recognised by | Default |
|---|---|---|
access | access in the file name | 32 MiB, 7 generations |
error | error in the file name | 16 MiB, 5 |
mail | postfix, dovecot, mail.log | 64 MiB, 10 |
service | everything else | 8 MiB, 4 |
A file below its limit is only stat-ed: nothing is read and nothing is opened. .1 stays plain text so it opens without unpacking, the generations from .2 up are gzipped, and the one past keep is deleted, so the disk cost of a log has a ceiling.
aegis keeper logs # file, class, size, limit, generationsaegis keeper logs rotate # rotate now, ignoring the intervalaegis keeper logs show access_json # the policy and generations of one log[logs]interval = 300compress = truedelaycompress = true[logs.classes.mail]size_mb = 64keep = 10The interval and the enabled flag are overridden in aegis.db (cron_jobs), like every job; the sizes and the counts stay in config/aegis.toml.
A log the collector follows (access_json.log) survives rotation: the collector remembers the inode, the offset and the first line, so a truncation restarts the read at zero instead of losing or duplicating requests.
#Files
| Path | Content |
|---|---|
<run>/keeper.pid | pid of the loop |
<logs>/keeper.log | supervision and job log |
<run>/keeper-cron.json | last run and result of every job |
<run>/<app>/<daemon>.pid | pid of each service |
<run>/<daemon>.heartbeat | liveness of a service that declares one |
<run>/<daemon>.lock | single-instance lock, when the daemon uses one |
<logs>/<app>/… | service output |
#Adding a service
service() method;aegis keeper services — it appears immediately;aegis keeper start (or wait for the loop) starts it.#Adding a job
jobs() method returning Job objects;aegis keeper cron — it appears immediately;with aegis keeper cron-run <name>.
After a deploy, restart the keeper. The daemon imports the cube modules once and keeps them for its whole life, so new code (a new service, a new job, a fixed path) only takes effect after
aegis keeper restart._stopwaits for the old process to leave before the new one starts — without that wait the fresh daemon would inherit the stale modules.
#Known limitation
worker_connections 8192 exceeds the default ulimit -n 1024 for nginx; the keeper currently inherits the caller's limits. Raising limits per service (resource.setrlimit before spawn) is on the roadmap.