191

I'd like to startup an Apache Spark cluster after boot using the following command:

sudo ./path/to/spark/sbin/start-all.sh

Then run this command when the system prepares to reboot/shutdown:

sudo ./path/to/spark/sbin/stop-all.sh

How can I get started? Is there a basic template I can build on?

I've tried to use an extremely simple (file: /lib/systemd/system/spark.service):

[Unit]
Description=Spark service

[Service]
ExecStart=sudo ./path/to/spark/sbin/start-all.sh

Which doesn't work.

Anthon
  • 277
macourtney7
  • 2,817
  • Have a look at: https://wiki.ubuntu.com/SystemdForUpstartUsers –  May 26 '17 at 09:03
  • 1
    Hi @WillemK, I had looked at this page already. This issue I found is I can't just replace exec with ExecStart=. Plus, I haven't used upstart before. – macourtney7 May 26 '17 at 09:07
  • 2
    The dot before the path of your script looks extremely suspicious. – Andrea Lazzarotto May 26 '17 at 09:09
  • @AndreaLazzarotto I think OP is trying to run the script the way OP would in the terminal hence the .... – George Udosen May 26 '17 at 09:25
  • Hi @AndreaLazzarotto, this is correct. Apologies for any confusion caused. – macourtney7 May 26 '17 at 10:28
  • @George yes I think so too, however using relative paths in services could lead to errors due to the different PATH. I suggest to use absolute paths instead. – Andrea Lazzarotto May 26 '17 at 10:58
  • @AndreaLazzarotto, I've been trying to use an absolute path but as you can see in my response to the answer given I'm getting an absolute path error. The .sh is located at /usr/lib/spark-2.1/sbin/start-all.sh and this is what I've supplied. Is this not the absolute path? – macourtney7 May 26 '17 at 11:08
  • @macourtney7 see my updated and the link to a working tutorial with more comprehensive options for systemd – George Udosen May 26 '17 at 11:17

3 Answers3

266

Your .service file should look like this:

[Unit]
Description=Spark service

[Service]
ExecStart=/path/to/spark/sbin/start-all.sh


[Install]
WantedBy=multi-user.target

Now, take a few more steps to enable and use the .service file:

  1. Place it in /etc/systemd/system folder with a name like myfirst.service.

  2. Make sure that your script is executable with:

     chmod u+x /path/to/spark/sbin/start-all.sh

  3. Start it:

     sudo systemctl start myfirst

  4. Enable it to run at boot:

     sudo systemctl enable myfirst

  5. Stop it:

     sudo systemctl stop myfirst

Notes

  1. You don't need to launch Spark with sudo in your service, as the default service user is already root.

  2. Look at the links below for more systemd options.

Moreover

Now what we have above is just rudimentary, here is a complete setup for spark:

[Unit]
Description=Apache Spark Master and Slave Servers
After=network.target
After=systemd-user-sessions.service
After=network-online.target

[Service]
User=spark
Type=forking
ExecStart=/opt/spark-1.6.1-bin-hadoop2.6/sbin/start-all.sh
ExecStop=/opt/spark-1.6.1-bin-hadoop2.6/sbin/stop-all.sh
TimeoutSec=30
Restart=on-failure
RestartSec=30
StartLimitInterval=350
StartLimitBurst=10


[Install]
WantedBy=multi-user.target

To setup the service:

sudo systemctl start spark.service
sudo systemctl stop spark.service
sudo systemctl enable spark.service

Further reading

Please read through the following links. Spark is a complex setup, so you should understand how it integrates with Ubuntu's init service.

Roj
  • 105
  • 5
George Udosen
  • 36,677
  • 1
    Thanks for this, I've created a file based on what you suggested. Upon running sudo systemctl start spark is receive the following error: Failed to start spark.service: Unit spark.service is not loaded properly: Invalid argument. See system logs and 'systemctl status spark.service' for details. – macourtney7 May 26 '17 at 10:24
  • The main part of systemctl status spark.service is as follows: Executable path is not absolute and spark.service: Service lacks both ExecStart= and ExecStop= setting. Refusing. – macourtney7 May 26 '17 at 10:27
  • The issues are 1) Spark binary path (should replace what we have in the service file) is needed, 2) Spark has a shut down command what is it. 3) Did you go through the links I gave you. I don't use spark so supply them – George Udosen May 26 '17 at 11:05
  • @GeorgeUdosen Thanks for your answer, my question is how can I run spark under a specific command after rebooting?The question is here https://askubuntu.com/questions/979498/how-to-start-a-zookeeper-daemon-after-booting-under-specific-user-in-ubuntu-serv – Soheil Pourbafrani Dec 13 '17 at 06:21
  • Please advise that user and administration units should now be put in /etc/systemd/system and not in /lib/systemd/system. systemclt enable myservice will not work in /lib/systemd. @see https://unix.stackexchange.com/questions/206315/whats-the-difference-between-usr-lib-systemd-system-and-etc-systemd-system – Gabriel Glenn Jan 08 '20 at 12:19
6

Copy-paste this into a terminal (as root) to create /root/boot.sh and run it on boot:

bootscript=/root/boot.sh
servicename=customboot

cat > $bootscript <<EOF
#!/usr/bin/env bash
echo "$bootscript ran at \$(date)!" > /tmp/it-works
EOF

chmod +x $bootscript

cat > /etc/systemd/system/$servicename.service <<EOF
[Service]
ExecStart=$bootscript
[Install]
WantedBy=default.target
EOF

systemctl enable $servicename

To modify the parameters, for example to use a different $bootscript, just set that variable manually and skip that line when copying the commands.

After running the commands, you can edit /root/boot.sh using your favorite editor, and it will run on next boot. You can also immediately run it by using:

systemctl start $servicename
Luc
  • 993
  • 1
    I'm a bit confused by systemd docs, but shouldn't it be Type=oneshot RemainAfterExit=yes or systemd will consider the task inactive unless the custom script leaves some processes running. – Peter Lamberg Jul 27 '19 at 14:38
  • @PeterLamberg I tried reading systemd docs too and yet here we both are ;). I remember they weren't very clear, but the answer I posted works for me on multiple systems (I revisit this page every now and then when I need it again). Do you mean that, because it's considered 'inactive', every successive 'start' call will re-run the script? Because I would consider that as expected for a shell script. I'd find it weird if I had to 'stop' something that isn't actually running before I can start it again. – Luc Jul 27 '19 at 18:53
1

Adding startup items in systemd is complex and cumbersome. To make this more convenient, I have wirte a tool add_service which provides a simple way to quickly add startup item in systemd.

Install:

pip3 install add_service

Usage:

python -m add_service [shell_file/cmd] [user (default `whoami`)]

Examples:

python -m add_service ssh_nat.sh  # by default user is `whoami`
python -m add_service "`which python3` -m http.server 80" root
Yang
  • 111
  • Is this your script? If so, please disclose that in the answer, both here and in https://unix.stackexchange.com/a/687324/70524 – muru Jan 22 '22 at 10:46