Aegis documentation

#keeper — services, jobs and logs

The keeper is the platform's process supervisor. It starts every registered daemon, keeps its pid file and log, restarts what died or hung, runs the scheduled jobs and rotates the logs. Aegis does not use systemd: the keeper itself is started by a SysV init script at boot.

Cubes provide service definitions; the keeper never guesses a path. A cube with a daemon implements service(), a cube with periodic work implements jobs(), and both become supervised as soon as the code is on disk.

Output is human-formatted throughout: these commands print their own lines and tables and never consult the global --json flag. Exit codes follow Conventions: 0 success, 1 failure, 2 usage, 130 Ctrl+C.

#How supervision works

  • The loop ticks every 10 seconds. Each tick starts every registered service
  • that is not running, then hands each due job to a pool of four worker threads.

  • A pid file is the source of truth for a service. PermissionError while
  • probing a pid means the process exists but belongs to another account, so it is never reported as stopped.

  • A service that writes its own pid file is waited for up to 10 seconds. A
  • daemon that forks and exits gets a 2.5-second grace period to write the file as the new process. A service with writes_pid=False has the keeper's own child pid recorded instead.

  • A service that dies within 60 seconds of its own start is treated as a crash
  • loop. The next attempt waits 0, 10, 30, 60 and then 300 seconds between consecutive failures, so a broken daemon cannot fill the log or the process table. Staying up for 60 seconds clears the penalty.

  • A service may declare a heartbeat file. A live pid whose heartbeat is older
  • than its timeout is treated as hung and restarted. A missing heartbeat file is not stale: a daemon that has just started may not have written one yet.

  • Every service is handled in its own try. A cube that raises while being
  • inspected cannot abandon the rest of the tick.

  • stop sends SIGTERM, waits up to 15 seconds, then sends SIGKILL.
  • The process renames itself to aegis-keeper, so ps, top and pgrep
  • show the supervisor instead of another python3 -m aegis.

    #Daemon control

    #aegis keeper start

    Starts the keeper in the background. If a live keeper already owns <run>/keeper.pid, nothing is started and the running pid is reported.

    bash
    aegis keeper start
    Argument / optionDefaultMeaning
    ——no arguments or options

    The child runs bin/aegis keeper daemon, appends its output to <logs>/keeper.log and claims the pid file. The command returns as soon as the process is spawned; a later tick brings the services up.

    Examples:

    bash
    aegis keeper startaegis keeper start && aegis keeper services
    text
    ✔ keeper started (pid 679986)

    Already running:

    text
    ℹ keeper already running (pid 679986)

    #aegis keeper stop

    Destructive. Stops the supervision loop. The supervised services keep running, but nothing restarts them until the keeper is started again.

    Asks the keeper to exit with SIGTERM and waits until the pid file is gone. The wait is deliberate: a restart that let the old process live a moment longer would bring up a keeper that still holds the previous cube modules.

    bash
    aegis keeper stop
    Argument / optionDefaultMeaning
    ——no arguments or options

    Examples:

    bash
    aegis keeper stopaegis keeper stop && aegis keeper status
    text
    ℹ keeper stopped (pid 679986)

    Not running:

    text
    ℹ keeper not running

    If the process ignores SIGTERM for 15 seconds:

    text
    ✖ keeper did not stop (pid 679986)

    The command exits 1 in that case and does not escalate to SIGKILL.

    #aegis keeper restart

    Destructive. Stops the loop, waits for the old process to leave, then starts a fresh one. Services are unsupervised during the gap.

    bash
    aegis keeper restart
    Argument / optionDefaultMeaning
    ——no arguments or options

    This is what a deploy needs: the daemon imports the cube modules once and keeps them for its whole life, so new code, a new service or a fixed path only takes effect after a restart. If the stop fails, the command exits 1 and does not start a second keeper.

    Examples:

    bash
    aegis keeper restartaegis keeper restart && aegis keeper services
    text
    ℹ keeper stopped (pid 679986)✔ keeper started (pid 681204)

    When the old loop was already gone:

    text
    ℹ keeper not running✔ keeper started (pid 681204)

    #aegis keeper status

    Reports whether the keeper is alive. Exits 0 when running and 1 when stopped, which makes it usable in a script.

    bash
    aegis keeper status
    Argument / optionDefaultMeaning
    ——no arguments or options

    Examples:

    bash
    aegis keeper statusaegis keeper status || aegis keeper start
    text
    ✔ keeper is running (pid 679986)

    Stopped:

    text
    ℹ keeper is stopped

    #aegis keeper daemon

    Runs the supervision loop in the foreground. This is the process that keeper start and the boot script launch.

    bash
    aegis keeper daemon
    Argument / optionDefaultMeaning
    ——no arguments or options

    The loop writes its own lines to <logs>/keeper.log, not to the terminal, so a foreground run looks quiet. It refuses to start a second copy: if the pid file names a live process, it logs already running (pid N) and exits 0. SIGTERM and SIGINT end it cleanly; the pid file is removed only while it still holds this process's pid.

    Examples:

    bash
    aegis keeper daemoncd /etc/aegis && ./bin/aegis keeper daemon

    The log then contains:

    text
    2026-11-17 10:35:56 keeper started (pid 679986, tick 10s, 8 services)2026-11-17 10:36:07 bind: started (pid 680051)2026-11-17 10:40:12 aegis-tasker: no heartbeat, restarted (killed (pid 801564), started (pid 802341))2026-11-17 10:41:02 keeper stopped

    #aegis keeper enable

    Writes /etc/init.d/aegis-keeper and registers it for the default runlevels with update-rc.d. The script forwards start, stop, restart and status to aegis keeper; no unit files and no systemd are involved.

    bash
    aegis keeper enable
    Argument / optionDefaultMeaning
    ——no arguments or options

    Writing /etc/init.d requires root. If update-rc.d is missing or fails, the script is still written and the message says so:

    text
    ✔ script written, update-rc.d failed: update-rc.d: not found

    Examples:

    bash
    aegis keeper enableaegis keeper enable && /etc/init.d/aegis-keeper status
    text
    ✔ enabled through /etc/init.d/aegis-keeper

    #aegis keeper disable

    Destructive. Unregisters the boot hook and deletes /etc/init.d/aegis-keeper. It does not stop a running keeper, and it does not touch the services.

    bash
    aegis keeper disable
    Argument / optionDefaultMeaning
    ——no arguments or options

    Examples:

    bash
    aegis keeper disableaegis keeper disable && aegis keeper stop
    text
    ✔ disabled and removed /etc/init.d/aegis-keeper

    #Services

    #aegis keeper services

    Lists every service the keeper would supervise, with its current state and pid file.

    bash
    aegis keeper services
    Argument / optionDefaultMeaning
    ——no arguments or options

    The list is sorted by service name. It contains every installed cube that provides service(), plus the collector services, the tasker, the AI daemon, pulse and Bird when those modules are enabled. A stopped service may be started by the next tick; the table shows the state at the moment of the call.

    Examples:

    bash
    aegis keeper servicesaegis keeper services | grep stopped
    text
    service        state               pid file─────────────────────────────────────────────────────────aegis-tasker   running (pid 8123)  /aegis/run/tasker/tasker.pidbind           stopped             /aegis/run/bind/named.pidnginx          running (pid 8044)  /aegis/run/nginx/nginx.pid

    With no cube installed:

    text
    ℹ no services registered

    #Scheduled jobs

    #aegis keeper cron

    Shows every enabled job with its effective interval, last run and last result.

    bash
    aegis keeper cron
    Argument / optionDefaultMeaning
    ——no arguments or options

    Job definitions live in code. The effective interval, the enabled flag and the overlap guard live in aegis.db (cron_jobs); the panel edits the same rows. Before listing, the registry inserts jobs it has not seen and marks jobs that disappeared from the code as orphaned instead of deleting them. A disabled job is left out of this table.

    Examples:

    bash
    aegis keeper cronaegis keeper cron | grep logs-rotate
    text
    job           every  last run             result──────────────────────────────────────────────────────────────logs-rotate   5m     2026-11-17 10:35:00  rotated 2 logs (48.3M)ssl-renew     6h     2026-11-17 08:00:00  nothing to renew

    A job that never ran shows never and -:

    text
    job           every  last run  result──────────────────────────────────────ssl-renew     6h     never     -

    #aegis keeper cron-run

    Runs one scheduled job immediately, whether its interval has elapsed or not.

    bash
    aegis keeper cron-run NAME
    ArgumentDefaultMeaning
    NAMErequiredjob name as shown by aegis keeper cron

    The run goes through the scheduler, so it takes the job's lock and is recorded in cron_logs like a scheduled run. Overlap protection is on by default: a second run while the first still holds the lock answers previous run is still in progress. A job that is disabled is not in the enabled list, so it is reported as unknown.

    Examples:

    bash
    aegis keeper cron-run logs-rotateaegis keeper cron-run ssl-renew
    text
    ✔ logs-rotate: nothing to rotate

    Failure or an unknown name exits 1:

    text
    ✖ unknown job: ssl-reneww

    #Log rotation

    #aegis keeper logs

    Lists the logs the rotation job manages, with the policy that applies to each one. This is the default verb when no verb is given.

    bash
    aegis keeper logs VERB NAME
    ArgumentDefaultMeaning
    VERBlistlist, rotate or show
    NAMEnonepart of a log path; required by show

    The verb is a positional argument, not a subcommand, so aegis keeper logs, aegis keeper logs list and aegis keeper logs show nginx all work. rotate and show are described below.

    The listing uses stat only, so it never opens a file a daemon is writing. Rows are sorted by total disk cost, largest first. The file column is shortened relative to the log root it lives under.

    Examples:

    bash
    aegis keeper logsaegis keeper logs list
    text
    file                    class    size    limit  gens  disk─────────────────────────────────────────────────────────────nginx/access.log        access   12.4M   32.0M  3     51.2Mnginx/error.log         error    1.2M    16.0M  2     2.4Mkeeper.log              service  512K    8.0M   1     1.0Mℹ 3 logs, 54.6M on disk

    With no *.log under any root:

    text
    ℹ no log files

    #aegis keeper logs rotate

    Runs the logs-rotate job now, through the scheduler, so it takes the job's lock and records the run. Files below their limit are left alone.

    bash
    aegis keeper logs rotate
    Argument / optionDefaultMeaning
    ——no arguments or options

    Examples:

    bash
    aegis keeper logs rotateaegis keeper logs rotate && aegis keeper logs
    text
    ✔ rotated 2 logs (48.3M)

    Nothing reached its limit:

    text
    ✔ nothing to rotate

    A disabled job or a run that still holds the lock is reported as an error and exits 1:

    text
    ✖ logs-rotate: previous run is still in progress

    #aegis keeper logs show

    Shows the policy of every log whose path contains the given text, and the generations that already exist.

    bash
    aegis keeper logs show NAME
    ArgumentDefaultMeaning
    NAMErequiredcase-insensitive part of a log path, for example access_json

    The match is a plain substring of the full path, so a directory name works too.

    Examples:

    bash
    aegis keeper logs show access_jsonaegis keeper logs show nginx
    text
    log               /aegis/logs/nginx/access_json.logclass             accesssize              12.4Mrotate at         32.0Mkeep              7compress          yesdelaycompress     yesgeneration        size──────────────────────────access_json.log.1 12.1Maccess_json.log.2.gz 3.4M

    No name or no match exits 1:

    text
    ✖ show needs a part of the path, for example access_json✖ no log matches access-json

    #How rotation works

    The job is registered in code as logs-rotate. It runs every [logs] interval seconds (300 by default, never below 10); the enabled flag and an overridden interval live in aegis.db like every other job.

    It discovers *.log files under three kinds of root, without following symlinks:

  • the log directory of every supervised service;
  • [aegis] logs_dir — keeper, panel, tasker, AI, pulse, bird, collector;
  • the application log tree (/aegis/logs) — nginx, PHP, databases, mail, FTP,
  • certbot.

    Each file is classified by its name and path, and each class has its own limit and generation count:

    ClassRecognised byDefault
    accessaccess in the file name32 MiB, 7 generations
    errorerror in the file name16 MiB, 5
    mailpostfix, dovecot, or a name starting with mail.64 MiB, 10
    serviceeverything else8 MiB, 4

    The method is copytruncate, because a supervised daemon keeps its log open for its whole life and most have no signal that reopens it. When a file reaches its limit:

  • the oldest generation is dropped and the others are renumbered up;
  • the log is copied to .1 through a .part file, then the original is
  • truncated — a crash during the copy leaves the log untouched;

  • with compress, generations are gzipped. With delaycompress, .1 stays
  • plain so it opens without unpacking, and .2 upward are .gz. Without delaycompress, compression starts at .1.

    The generation past keep is deleted, so the disk cost of a log has a ceiling. One unreadable file does not stop the others; the result reports the rotated count and the failed count separately.

    Relevant configuration:

    toml
    [logs]interval = 300compress = truedelaycompress = true[logs.classes.mail]size_mb = 64keep = 10

    #Files

    PathContent
    <run>/keeper.pidpid of the supervision loop
    <logs>/keeper.logsupervision and job log
    database/storage/aegis/aegis.dbcron_jobs (schedule, locks) and cron_logs (history)
    <run>/<app>/<daemon>.pidpid of each service
    <run>/<daemon>.heartbeatliveness of a service that declares one
    <run>/<daemon>.locksingle-instance lock, when the daemon uses one
    <logs>/<app>/…service output

    cron_logs keeps the newest 2000 rows; older runs are pruned as new ones arrive. A job lock is considered stale after one hour, and a live run renews it well before that, so a process that died without releasing the lock cannot block the job forever.

    #Operational notes

  • After a deploy, run aegis keeper restart. The daemon does not reload cube
  • modules.

  • keeper stop and keeper disable do not stop the supervised services. Use
  • aegis restart for a full restart of the keeper, every service and the panel.

  • A service declared with writes_pid=False still gets a pid file: the keeper
  • writes the pid of the process it started.

  • A stale pid file is removed before a start; a live pid that belongs to another
  • account is reported as running, not stopped.