GrepBoard/Manuals
47 manuals ·

Find why a systemd service is failed

·Ubuntu 24.04 · systemd 255 (reproduced with transient units)·6 steps

Read the Active and status lines of systemctl status, check the log from just before the failure, and fix 203/EXEC and "start request repeated too quickly".

When a service shows failed, look at status, then journalctl, then systemctl cat. The output below comes from transient units (systemd-run) that were made to fail on purpose; the unit name is shown as myapp and paths as /srv/app.

[1] Run systemctl status to see how the service died. Active: gives the state and reason (Result: exit-code), Process: gives the command and its exit code (code=exited, status=3), and Main PID: is the last process that ran.

console
# systemctl status myapp --no-pager× myapp.service - /bin/sh -c "echo \"myapp: cannot open /srv/app/config.yml\" >&2; exit 3"     Active: failed (Result: exit-code) since Mon 2026-10-19 12:16:54 KST; 1s ago    Process: 47127 ExecStart=/bin/sh -c echo "myapp: cannot open /srv/app/config.yml" >&2; exit 3 (code=exited, status=3)   Main PID: 47127 (code=exited, status=3)

[2] Read the program's own error with journalctl. -u selects the unit, -b this boot, -n 50 the last 50 lines, and --no-pager prints without a pager.

console
# journalctl -u myapp -b --no-pager -n 50Oct 19 12:16:54 server1 sh[47127]: myapp: cannot open /srv/app/config.ymlOct 19 12:16:54 server1 systemd[1]: myapp.service: Main process exited, code=exited, status=3/NOTIMPLEMENTEDOct 19 12:16:54 server1 systemd[1]: myapp.service: Failed with result 'exit-code'.

The sh[47127]: line is the message the program wrote itself, which is the cause.

[3] Show the unit that is actually in effect with systemctl cat. It prints the unit file and its drop-ins (myapp.service.d/) with their file paths.

console
# systemctl cat myapp --no-pager# /run/systemd/transient/myapp.service[Service]ExecStart=ExecStart="/bin/sh" "-c" "echo \"myapp: cannot open /srv/app/config.yml\" >&2; exit 3"

[4] A common cause is status=203/EXEC. The systemd.exec manual (Process Exit Codes) says the execve(2) call failed, "most likely this is caused by a missing or non-accessible executable file". Below, the file exists but the interpreter on its first line (#!/usr/bin/python4) does not.

console
# systemctl status myapp --no-pager    Process: 47581 ExecStart=/srv/app/bin/myapp (code=exited, status=203/EXEC)   Main PID: 47581 (code=exited, status=203/EXEC) # journalctl -u myapp -b --no-pager -n 50Oct 19 12:16:59 server1 (myapp)[47581]: myapp.service: Failed to execute /srv/app/bin/myapp: No such file or directory

[5] A service with restarts enabled that keeps dying ends with start request repeated too quickly. Below is a unit with Restart=always, limited to 3 starts in 10 seconds.

console
# journalctl -u myapp -b --no-pager -n 4Oct 19 12:17:05 server1 systemd[1]: myapp.service: Scheduled restart job, restart counter is at 3.Oct 19 12:17:05 server1 systemd[1]: myapp.service: Start request repeated too quickly.Oct 19 12:17:05 server1 systemd[1]: Failed to start myapp.service - /bin/sh -c "echo \"myapp: bind: address already in use\" >&2; exit 1".

That line is only the result. The cause is higher up in the log, in the program's own line (myapp: bind: address already in use). The default limit is StartLimitIntervalSec=10s and StartLimitBurst=5; once it is exceeded, the unit is not started again until reset-failed.

[6] Fix the cause. If you edited the unit file, run daemon-reload to make systemd re-read it, then reset-failed (it resets the failed state) to clear the failure record and the start limit, then start the service.

/etc/systemd/system/myapp.service
[Unit]Description=myapp [Service]ExecStart=/srv/app/bin/myapp --config /srv/app/config.ymlRestart=on-failure [Install]WantedBy=multi-user.target
console
# systemctl daemon-reload# systemctl reset-failed myapp# systemctl restart myapp

Analyze your syslog · journal log in the browser ↗ — nothing is uploaded.

#systemd#systemctl#journalctl