Run at Startup (without rc.local)

A question that comes up quite frequently in Pi Wars circles is “how do I make my robot program run at start-up?” Someone asked me that directly at the Pi Wars Miniconference - and I gave the same answer I always do: “Use systemd”.

In itself, that not a very useful answer, because systemd is a large and varied project with lots of tools and functionality. It’s just a bit too complex to describe over voice without context. However, with the help of examples, it really is quite straightforward to make a simple systemd service. There’s also all kinds of bells and whistles you can try out too if you’re feeling adventurous!

If you just want a quick fix, then either jump to the “tl; dr” section below, or head over to the page on raspberrypi.org. But if you want to know more about what you’re doing, read on!

Not rc.local

Many of the tutorials you will see online mention rc.local as a way to run things at boot. This is a practice which I really wish would end!

rc.local is an archaic way to run one-shot things at boot-time. Even before systemd was around, there were much better ways to run “services” than rc.local.

From my perspective, there’s several major problems with rc.local (all relevant in the context of a Pi Wars robot):

  1. If something you run in rc.local gets stuck, then your system will hang during boot
  2. By default, things run from rc.local can’t output anywhere; so you can’t see any errors or other output
  3. There’s no “lifetime management”. rc.local runs once and once only.
    • If you want to run it again, you have to reboot.
    • If you want to stop something which was started by rc.local, you need to find its process manually and kill it.

systemd

Nowadays, for better or for worse (better, in my opinion), your Linux-based system will be running systemd unless you’ve explicitly avoided it.

systemd is the “system daemon”. It’s not a single thing - it’s a whole collection of programs which handle services, networking, logging, containers and other things. It got a lot of bad press from people who preferred the “old” way of doing things, but that’s an argument you can go and find somewhere else on the internet.

For the purposes of starting programs at boot, we only care about one of its concepts: services.

systemd service

A systemd.service is simply a program or operation which should be run under the control of systemd. Why do it under systemd?

  • It can track dependencies, and make sure that e.g. your program only runs once network or GUI is available
  • It can run your program as any user you like, with any environment you like
  • It can restart your program if it crashes or exists
  • If you need multiple programs to run, systemd can handle the ordering and restart them all if one of them fails.
  • It can capture the output of your program into its logging system
  • It’s easy to stop, start, and restart your program

All a service is to systemd is a .service file, which should be placed in /etc/systemd/system1. A very basic systemd service would look like this:

[Unit]
Description=My first systemd service

[Service]
ExecStart=/usr/bin/echo "Hello systemd world!"

The easiest “gotcha!” in writing systemd services is that the environment which they run in is by default quite “empty” - for example $PATH isn’t set, which means you have to use the full path to your command.

If you save that as /etc/systemd/system/hello.service then you can start it with the systemctl command, which is how we control systemd:

$ sudo systemctl start hello.service

systemd will run it, then you can check the output with journalctl, which stores all the system’s log messages (-e jumps to the end of the log):

$ journalctl -e
[lots of log messages]
Nov 24 17:02:15 eva systemd[919]: Started My first systemd service.
Nov 24 17:02:15 eva echo[23731]: Hello systemd world!
Nov 24 17:02:15 eva systemd[919]: hello.service: Succeeded.

If you wanted to run this service on boot, then we need to add a new section telling systemd how to “Install” this service:

[Unit]
Description=My first systemd service

[Service]
ExecStart=/usr/bin/echo "Hello systemd world!"

[Install]
WantedBy=multi-user.target

If you modify a service file, you need to tell systemd to reload it (normally it loads them all during system boot, and so won’t notice your changes until you reboot) with systemctl daemon-reload; then, we can enable our service to make it run at boot:

$ sudo systemctl daemon-reload
$ sudo systemctl enable hello.service
Created symlink /etc/systemd/system/multi-user.target.wants/hello.service → /etc/systemd/system/hello.service.

The multi-user.target tells systemd that our service should be considered part of the system when it’s running in “multi-user” mode. This name comes from the old days of computing, but it basically means “normal operation”.

Now, if you reboot, you can look in the journal for the service getting run. (-b0 means “only show messages from this boot, -b-1 would be from the previous boot):

$ journalctl -b0
... lots of log messages
Nov 24 17:25:01 eva systemd[919]: Started My first systemd service.
Nov 24 17:25:01 eva echo[23731]: Hello systemd world!
Nov 24 17:25:01 eva systemd[919]: hello.service: Succeeded.
... lots of log messages

We can actually do better, and only show the messages from our service with --unit=:

$ journalctl -b0 --unit=hello.service
Nov 24 17:25:01 eva systemd[919]: Started My first systemd service.
Nov 24 17:25:01 eva echo[23731]: Hello systemd world!
Nov 24 17:25:01 eva systemd[919]: hello.service: Succeeded.

Stopping and restarting

The simple service we just wrote runs a single command which exits immediately. However, the program for running a robot should continue running forever. systemd has classifications for these different Types of services.

For one whose program keeps running, we should use Type=exec. systemd will consider an exec service running once it has successfully started the program, and ended if the program exits.

To test this out, let’s create a very simple python program which just prints a number every second:

#!/usr/bin/python3
import time

i = 0
while True:
   print(str(i))
   i +=1
   time.sleep(1.0)

And a systemd service to run it. There’s one little detail here which is important: the -u option passed to python3 makes sure that the output is unbuffered, and so shows up in the journal immediately. As before, full paths are needed:

[Unit]
Description=Python counter

[Service]
Type=exec
ExecStart=/usr/bin/python3 -u /home/pi/counter.py

Now, if we start the service we will see its output in the journal:

$ sudo systemctl start counter.service
$ sleep 5
$ sudo journalctl -b0 --unit=counter.service
Nov 24 22:35:43 eva systemd[919]: Starting Python counter...
Nov 24 22:35:43 eva systemd[919]: Started Python counter.
Nov 24 22:35:44 eva python3[18030]: 0
Nov 24 22:35:45 eva python3[18030]: 1
Nov 24 22:35:46 eva python3[18030]: 2
Nov 24 22:35:47 eva python3[18030]: 3
Nov 24 22:35:48 eva python3[18030]: 4

We can stop the program with sudo systemctl stop counter.service and restart (stop then start again) it with sudo systemctl restart counter.service - no more looking for PIDs to kill!

$ sudo systemctl restart counter.service
$ sleep 3
$ sudo systemctl stop counter.service
$ sudo journalctl -b0 --unit=counter.service

tl;dr

The too-long, didn’t read summary is:

  1. Create a /etc/systemd/system/robot.service file, containing:

    [Unit]
    Description=Robot
    
    [Service]
    Type=exec
    ExecStart=/usr/bin/python3 -u /home/pi/my-robot-script.py
    
    [Install]
    WantedBy=multi-user.target
    
  2. Make it run at boot with sudo systemctl enable robot.service

  3. Start/stop/restart it with sudo systemctl {stop,start,restart} robot.service

  4. Prevent it from running at boot with sudo systemctl disable robot.service

  5. View the output with sudo journalctl --unit=robot.service

Other useful options

systemd has literally hundreds of options for controlling how services are run, all documented in great detail in the various manual pages.

Here are some of the options which I think are most useful for something like a Pi Wars robot:

Automatic restarting

If your robot code crashes (or even if it exits normally), you might want it to start again automatically without having to reboot.

systemd has some useful settings for this, which should go in the [Service] section:

[Service]
Restart=always
RestartSec=5s
  • Restart=always: Whenever the program exits, start it again
  • Restart=on-failure: If the program exits with a non-zero exit code, then restart it (e.g. exit(1) -> restart, exit(0) -> don’t restart). This will also restart if your program crashes.
  • RestartSec=5s: You can add a delay before systemd restarts the program. This can be helpful to stop an infinite very fast loop if your program crashes every time it starts!

Set-up or clean-up scripts

As well as the ExecStart= command for running the program, you can add ExecStartPre= and ExecStartPost= which can be used to execute commands before or after the main program is started.

This might do something like delete a file or create a directory before starting the main script.

However, this cannot be used to run other programs which are part of the robot. Any programs run from ExecStartPre/ExecStartPost will be terminated by systemd before the next service is run.

If you have multiple programs which form part of your robot, see the next section on dependencies.

Dependencies between services

On Mini Mouse, I have two programs making up the robot:

  • yapidh is a C program driving the stepper motors
  • bot is the main robot program

bot can’t work properly unless yapidh is running first, systemd lets you describe this kind of dependency in the unit files.

There’s several different ways to describe dependencies, each being subtly different. If you’re planning to make use of any of them, I strongly recommend you read the unit man page.

  1. You can say that one service should be started Before= or After= another service
  • systemd uses these to decide which order to start services in, but only if they were already being started!
  1. You can say that one service Requires= or is RequiredBy= another one
  • If a service Requires= another, then when the service is started, the one(s) specified in Requires= will be started too.
  • If any of the services in Requires= is explicitly stopped, then this one will be stopped, too.
  • RequiredBy= is the same, but in the opposite direction.
  1. You can say a service Wants= or is WantedBy= another one
  • Wants= is like Requires= but weaker. If one of the “wanted” services fails, it won’t cause the Want-er to fail.
  1. PartOf= will make this service restart when the one it’s PartOf is restarted.

For Mini Mouse, I make both components have Restart=always, and add a Requires= and PartOf= dependency from bot to yapidh. That gives the desired behaviour:

  • yapidh is always started when bot is started
  • If either program crashes or exits, it will be restarted
  • If yapidh is restarted, then bot is restarted first

Also note that yapidh doesn’t need to be enable-d, because it gets started by bot’s Requires=yapidh.service.

Here are the .service files:

bot.service

[Unit]
Description=Bot service
Requires=yapidh.service
After=yapidh.service
PartOf=yapidh.service

[Service]
Type=exec
ExecStart=/home/pi/go/src/github.com/usedbytes/mini_mouse/mini_mouse-bot/bot
Restart=always
RestartSec=10s

[Install]
WantedBy=multi-user.target

yapidh.service

[Unit]
Description=Yapidh service

[Service]
Type=exec
Nice=-20
CPUSchedulingPolicy=rr
CPUSchedulingPriority=99
ExecStart=/home/pi/Programming/yapidh/yapidh

If I force yapidh to crash, we can see the following sequence in the logs after running sudo systemctl start bot:

Nov 24 22:19:46 eva systemd[919]: Starting Yapidh service...
Nov 24 22:19:46 eva systemd[919]: Started Yapidh service.
Nov 24 22:19:46 eva systemd[919]: Starting Bot service...
Nov 24 22:19:46 eva systemd[919]: Started Bot service.
Nov 24 22:19:49 eva systemd[919]: yapidh.service: Main process exited, code=exited, status=1/FAILURE
Nov 24 22:19:49 eva systemd[919]: yapidh.service: Failed with result 'exit-code'.
Nov 24 22:19:59 eva systemd[919]: yapidh.service: Service RestartSec=10s expired, scheduling restart.
Nov 24 22:19:59 eva systemd[919]: yapidh.service: Scheduled restart job, restart counter is at 1.
Nov 24 22:19:59 eva systemd[919]: Stopping Bot service...
Nov 24 22:19:59 eva systemd[919]: bot.service: Main process exited, code=killed, status=15/TERM
Nov 24 22:19:59 eva systemd[919]: bot.service: Succeeded.
Nov 24 22:19:59 eva systemd[919]: Stopped Bot service.
Nov 24 22:19:59 eva systemd[919]: Stopped Yapidh service.
Nov 24 22:19:59 eva systemd[919]: Starting Yapidh service...
Nov 24 22:19:59 eva systemd[919]: Started Yapidh service.
Nov 24 22:19:59 eva systemd[919]: Starting Bot service...
Nov 24 22:19:59 eva systemd[919]: Started Bot service.

We see that bot forces yapidh to start. Then yapidh fails, and 10 seconds later is restarted - which triggers a restart of bot too.

Further Reading

There’s plenty more that you can learn about systemd - and the best way (although a little dry) is to read the documentation:


  1. systemd stores services in a number of places. /lib/systemd is usually used by the distribution, and /etc/systemd for things added by the end user or system administrator (you!) ↩︎